Most of My AI Product Is Not AI

In my earlier post about Voomero, I wrote about why I started it and said the product would tell its own story over time. This is the first part of that story, although it is not the part I expected to write about.

Voomero takes a property listing and produces a video. This is one of its features, and the one I want to focus on here. When people hear that, they often assume a model generates the video and most of my work is figuring out the right prompt.

But planning one video starts with two model calls. One writes about fifteen lines of text. The other turns a short query into an embedding so we can search a music catalogue. Additional calls may be needed if the text needs revision.

Everything else is ordinary code.

I want to explain what that code does, because I think it is one of the more interesting parts of building an AI product, and it gets much less attention than the model itself.

This is an example of a property video made with Voomero:

What a Plan Actually Is

The planner receives a property listing and the real estate agent's preferences: the style, length, formats, whether they want narration, and a color palette. It uses those inputs to produce a document describing the whole video. The rendering engine then turns that document into frames.

For each video, the planner has to decide:

• which sections, or beats, the video has, and in what order
• which photos each beat shows, and in what order
• which opening layout presents the first photo
• which text arrangement, called a lockup, sits on top of it, and which effects animate its lines
• which montage shows the rooms together
• which showcase shows them one at a time, and where the caption sits on each slide
• how the map presents the neighborhood, including the map tiles and marker shape
• which layout presents the property details, and which facts fit
• which price frame closes the video
• which typefaces to use
• which music track to use, and which part of it to play
• which transition, color grade or CSS treatment, photo entrance, and backdrop to use
• how long every beat lasts, in frames

Each of those comes with smaller choices: camera movement, board material, arrangement, cropping, text placement, and shadow strength.

Throughout the plan, the planner has to choose from a set of options. These are the creative decisions I wanted the software to handle for the user.

Why I Did Not Give That to a Model

My first idea was to describe the listing and the catalogue to a model, then ask it to return its choices as JSON. That would have been enough for a demo.

But I decided against it for four reasons.

First, I need to reproduce the result. An agent might generate a video, like it, and come back the next day asking for the same video but shorter. If I ask a model to create the plan again, the same request can produce a different result. I need a way to refer to a specific version, both for the agent and for myself when I need to reproduce a bug.

Second, I need to measure how the planner behaves. Across fifty different creative directions, how many opening layouts does it actually use? With a model making the decisions, every test run costs money and time, and the results can change between runs. With code, I can test fifty directions in a few seconds and get the same results every time.

Third, the rendering engine already has exact measurements for its components. It knows how much of a frame a text column occupies, how bright a photo becomes after a layout applies its color grade, and how long a caption stays readable before it scrolls away. Those measurements live next to the components that produce them. I did not want a model guessing at things the code already knows.

But the most important reason is that I want control over the creative decisions. Voomero exists so an agent does not have to make dozens of them. If I hand those decisions to a model, it becomes harder to understand why it chose something and how to improve it. When a video looks wrong, I want to find the rule responsible and change it.

That is why these decisions live in code. But how do I turn creative preferences into rules the planner can use?

Scoring Everything Against the Same Brief

If I choose the layout, typeface, and transition using separate rules, each choice might make sense on its own. But that does not mean they will work well together.

Instead, every part of the catalogue describes itself in the same way. It lists the tones it fits, such as cinematic, elegant, minimal, and playful. It also describes qualities such as drama, ornateness, warmth, energy, and pace. In the code, these qualities are called axes.

The planner describes each beat using those same tones and values. That description is the brief. It comes from the style the agent chose, the photos in that beat, the moods found during photo analysis, and the colors in the palette.

The agent's chosen style has substantially more influence than the inferred signals. Those signals act as secondary adjustments: they can refine the ranking, but they should not override the preference the user actually selected.

The same scoring function evaluates every candidate against that brief. It considers how well the candidate matches the tone and how closely its qualities fit the intended result. It only compares axes the candidate defines, so a layout with no warmth value is not penalized for leaving it out.

Some qualities also need to be judged differently depending on the direction of the mismatch. For ornateness, being simpler than requested is more acceptable than being too decorative.

A simple component can still fit into a decorative video. But a heavily decorated component can make a video feel too busy when the user asked for something simple.

Why I Do Not Always Pick the Highest Score

Once I could score the options, I started by choosing the highest score. It took me a while to notice the problem with that.

Two listings with the same style tend to produce similar briefs. That means the ranking barely changes, and the planner keeps choosing the same opening layout for cinematic videos.

I measured this across a hundred runs covering five styles, seven mood sets, and four palettes. Then I compared it with choosing from a group of the best matches, which I call a band:

The music had even less variety. The search query combines the style with a few predefined terms, so the closest match depends mostly on the style:

With five styles, we were only playing five tracks from the whole catalogue.

So I use the same rule throughout the planner: score all the options, keep a small high-scoring band, then deterministically select one from that group. The scoring keeps the choices relevant, while the selection allows more of the catalogue to be used. In conceptual pseudocode:

const ranked = rank(candidates, brief)
const band = strongestMatches(ranked)
const choice = selectDeterministically(band, key)

The Draw Is Not Random

The draw uses no random number generator. It hashes stable listing information, the creative brief, an explicit seed, and a namespace for the decision being made. The same inputs produce the same hash, so the choice can be repeated. Conceptually, it looks like this:

const key = hash(
  stableListingSignals + creativeBrief + seed + decisionNamespace
)

The inputs need a consistent representation so equivalent requests produce the same key.

The seed lets the agent get different versions from the same listing and preferences. Saving it with the inputs makes a version reproducible. With the same inputs and seed, the result is the same on any machine.

The namespace separates the choices. Selecting a layout and selecting music are different decisions, even when they use the same listing and brief. Each decision type gets an independent namespace so those choices do not move together simply because they share inputs.

Only information that should affect a creative decision belongs in its key. For example, producing a vertical and a landscape version should preserve the same creative direction. A change in output format should not accidentally change the opening layout.

A Bug the Existing Tests Did Not Catch

For a while, the planner was producing less variety than I expected, but there were no errors and the tests were passing.

To select a candidate, the planner takes the remainder after dividing the hash by the size of the candidate group. I was using FNV-1a, a common, fast string hash. But its low bits depend only on the low bits of the input characters. When the group size is a power of two, the remainder depends only on the lowest bits of the hash.

Different inputs could still produce the same few remainders, which meant they selected the same few options.

I found the problem by measuring how many different options the planner selected. Sixty different seeds for one listing reached only three opening layouts and three lockups, leaving other strong candidates unused. Typeface selection kept resolving the same way across all sixty runs, even when other candidates had equal scores.

The plans were valid and the videos rendered correctly. Each choice was a good match, but the choices kept repeating. Changing the seed was not producing the expected variety because the changes in the hash were not reaching the low bits used by the remainder operation.

The fix was five lines: the Murmur3 finalizer. It mixes the accumulated hash value so changes in the higher bits also affect the low bits before we take the remainder:

export function hash(value: string): number {
  let result = 2166136261
 
  for (let index = 0; index < value.length; index += 1) {
    result ^= value.charCodeAt(index)
    result = Math.imul(result, 16777619)
  }
 
  result ^= result >>> 16
  result = Math.imul(result, 0x85ebca6b)
  result ^= result >>> 13
  result = Math.imul(result, 0xc2b2ae35)
  result ^= result >>> 16
 
  return result >>> 0
}

The same sixty seeds now reach every slot of every band.

This reminded me that repeatable results are only part of the requirement. I also need to check how the choices are distributed. A planner can run without errors and still produce much less variety than intended. The tests need to measure that behavior as well as whether a plan is valid.

That is extra work that comes with making the planner deterministic, and I still think it is worth doing.

Where the Model Fits

The planner uses a model for two tasks. Both involve language.

The first is writing the title, subtitle, and captions. These need to describe the specific property, so the planner cannot simply choose them from the catalogue.

Even here, most of the work happens in code. Before calling the model, the planner has already chosen the layout, so it can describe the available space and how the text will be presented. The model needs to write for that context.

The model also receives grounded information about the property. Its claims need to come from that information, rather than filling gaps with plausible details.

Every response is checked against factual, spatial, and layout constraints. A sentence can sound good and still be unusable because it makes an unsupported claim or does not fit the layout. If the result fails those checks, the system can retry or fall back to deterministic copy built from known facts. The model's response is a candidate that needs validation before it becomes part of the video.

The second task is finding music. Each track in the catalogue has a written description of its mood and use. The planner builds a short query in the same language, converts it into an embedding, and searches for similar descriptions. Apart from generating the text, this is the only part of planning that needs a network call.

The search results then go through the same process as the other choices: keep a band of the best matches and choose one using the seeded hash.

What This Means in Practice

Two model calls make the cost of planning a video very low. But the main benefit for me is being able to understand and improve the result.

If an agent tells me a video feels too busy, I can reproduce it from its seed and read the recorded reason for each choice. I can find the rule that allowed three decorative components into the same composition, change it, and test a hundred listings to see what else changed. That takes minutes. If a model made those decisions, I would not have the same direct way to find and change the responsible rule.

It also lets the planner keep working when something is unavailable. If the music catalogue is missing, it plans a silent video. If the embedding model is unavailable, it chooses a track without ranking the matches. If the text model refuses or returns unusable output, it uses sentences built from the stored facts. If the component registry is empty, it uses one known working variant from each family.

The planner handles these cases without throwing an error. A video with a less suitable track is still something the agent can use. A video that never gets planned is not.

One piece of feedback on a Voomero video brought this back to the problem I wanted to solve. Translated from Portuguese:

"I think this video really catches the buyer's attention and perfectly captures the property's potential."

They also asked if they could use it to promote the property and how much this kind of service would cost. For me, their interest in using it mattered more than the compliment. They could see how it would help them market the property.

In the Voomero post, I wrote that AI was never the goal, just the tool that made the product possible. Building the planner has helped me explain what that means in practice.

AI made this product possible because a model can write about a property and understand a photograph. Building those capabilities into this product was not practical five years ago. But the code around them decides how the video comes together, and that is where almost all of my work goes.

I think the same is true of many AI products being built right now. Most of the attention goes to the model, but much of the engineering is in the software that turns its output into something people can actually use.