A Hammer Looking for a Nail

In my last post, I wrote about why most of Voomero is ordinary code and why I only use a model for the parts that need language. But while building it, I also found myself doing something that went against that approach. I found an AI tool I liked and started looking for a place to use it.

The tool is JEV, from a company called TypeSafe. It entered early access in the middle of September. Almost as soon as I read about it, I wanted to use it in Voomero.

I spent time testing it, and the results were good. But when I looked at what the product actually needed, I could not justify adding it yet.

Why I Wanted to Use JEV

With most language models, you send a prompt and get text back. Then your code has to check whether that text is something the product can use.

JEV works differently. You give it text and a set of typed questions. A question can ask for a yes or no, a choice from a list of options, or a rating on a scale. Each answer comes back as a number instead of generated text.

In one of my first calls, I asked whether an opening title gives someone a reason to be interested in the whole property or simply names a room:

{
  "model": "jev-1.13.0",
  "state": {
    "title": "Bright arrival",
    "core_claim": "A bright, modern 2-bedroom home in Marina del Rey with a balcony, a terrace and a garden"
  },
  "questions": {
    "is_hook": {
      "type": "noul",
      "instructions": "Does the title state the core claim as a benefit of the whole property, rather than naming a room, a feature or what one photo shows?",
      "criteria": {
        "true": "A benefit or way of living the whole property offers",
        "false": "A room label, a feature name or a caption of one photo"
      }
    }
  }
}

It returned a probability of 0.4 that the title was a hook. The call took about 700 milliseconds and cost a fraction of a cent.

I had spent months keeping decisions in code wherever a model was not needed. A model that answers specific questions with typed values seemed to fit that approach well. It was fast, the cost was low, and the probability gave me a way to decide whether to act on an answer.

The part I had not worked out was where the product actually needed it.

Why I Left It Out of the Planner

Before building the latest version of the planner, I reviewed how it should work and considered JEV as part of that process. I decided to leave it out.

Most of what the planner knows about a property comes from photos, while JEV only reads text. The creative decisions already belong to the scoring code, and I did not want to give those decisions to a model. There was no clear step in the planner that needed a judge.

I agreed with the decision, although I was still disappointed. I liked the tool and wanted to find a useful place for it.

Checking the Generated Text

Once the new planner was working, I came back to JEV. The planner uses a model to write the title, subtitle, and captions on each slide. Before that text becomes part of a video, code checks it for problems such as lines that are too long, numbers that are not in the listing, and exact repetition.

Those checks can catch specific mistakes, but they cannot judge what a sentence means.

This seemed like a useful place for JEV. I chose three questions that ordinary code cannot answer well:

1. Is the opening title a hook? "Indoor-outdoor flow" sells the property. "Terrace dining" just names a spot.
2. Does a line claim something the evidence does not support? For example, a line mentioning a view that no photo shows.
3. Do two lines make the same point? "Alfresco corner" and "Dining on terrace" in the same video say the same thing twice.

The example below lets you try the first question with real titles from Voomero videos. I sent each title to JEV three times using the request above. The numbers are the answers it returned.

Before deciding whether to add it, I wanted to measure how well it handled all three questions.

How I Tested It

I compared three approaches using the same questions:

• JEV
• GPT-5.4 mini, a small regular OpenAI model, given exactly the same text as JEV
• the simple rules I already had, such as "a short title that names a room is probably weak"

I collected titles, lines, and pairs of lines from real Voomero videos of one property. I asked each judge to evaluate every item three times so I could also measure whether its answers changed between calls.

The first run found almost no problems in the generated text. I thought I needed harder examples, so I built extra test cases from real lines. I used slide captions as opening titles, checked a caption against photos of a different room, and paired a first draft with its rewrite.

That gave me 121 items in total: 20 titles, 60 lines, and 41 pairs. Of those, 27 were test cases I had built myself.

To measure the results, I needed a reference answer for each item. Labelling everything myself would have taken about an hour. Instead, I used two strong vision models from different companies, Claude Sonnet 5 and GPT-5.4. Both could see the actual photos, which the judges could not. I only counted an item when both models agreed on its label.

I chose models from different companies because models from the same family tend to share blind spots. I did not want to rely on one family's judgment as the correct answer. The whole labelling process cost about two dollars.

What the Results Showed

I used F1 to compare the results. It is a score between 0 and 1 that measures how well a judge catches real mistakes while avoiding false alarms.

JEV had the highest score on two of the three questions. But the bigger difference was in speed and consistency:

The probabilities changed slightly between calls. In the example above, "Bright arrival" returned 0.43, 0.40, and 0.42. But across my test, those changes never moved an answer from yes to no or the other way around.

The probability was also useful for deciding which answers I could act on. When JEV returned at least 0.8 or at most 0.2, it was correct every time in this test. Depending on the question, that covered about half to three quarters of the items. I could use those confident answers in the product and ignore the rest.

At this point, I was already thinking about how to add it to the planner.

How Often Did the Problem Actually Happen?

Then I separated the real generated text from the test cases I had built. As before, I only counted items where both labellers agreed.

Most of the mistakes the judges caught came from the examples I had created myself. The model writing the video text had already stopped making most of them.

Shortly before running these tests, I had changed how the planner asks for the opening title. It now explains what the whole video is trying to say and asks the model to write a title around that message. Before the change, I had seen titles like "Bright arrival". After it, I did not find a single weak title. The planner also gives each part of the video its own message, which leaves fewer reasons for the model to repeat the same point.

When the first run found almost nothing wrong, I treated that as a reason to make the test harder. But it was already telling me something useful about the product. The mistakes I wanted JEV to catch were barely happening.

I was so focused on measuring the judge that I skipped measuring the problem.

Why I Did Not Add It Anyway

I still considered keeping JEV as an extra check. It was fast, cheap, and consistent, so adding it seemed like a small cost even if it only caught an occasional mistake.

But when I worked through the numbers, the false alarms surprised me. A checker will sometimes flag a correct line as a mistake. When real mistakes are rare, those incorrect flags can outnumber the real ones, even when the checker performs well overall.

The example below shows how that changes. Move the slider to see what happens to the flags as mistakes become more or less common:

For my test cases, almost every flag is a real mistake, which explains why the results looked so good. For the real generated text, fewer than half are. Each false alarm would send a correct line back to the model for a rewrite.

There is also more to the cost than the price of a call. A checker that finds a problem in one or two lines out of a hundred still adds a network call to every video. It adds another service that can be slow or unavailable, another reason to rewrite the text, and a dependency on a product that has only just entered early access. These are costs I would have to consider with any service. I need enough benefit to justify them.

JEV's text-only input was another limitation. Whether a sentence is supported often depends on what is visible in a photo.

I saw this in the tests. Both text judges flagged "sliding door to fresh air" as unsupported because the written description of the photo did not mention a door. But the model that wrote the caption had seen the photo itself. The judges were missing information that the writing model had used, so they could not settle whether the caption was correct.

What I Would Do Differently

JEV did what it promised in these tests. It was fast, cheap, and more consistent than the regular model I compared it with. But Voomero did not have enough of the problem I wanted it to solve.

I should have started by counting how often the problem happened in real videos. If I had checked the opening titles first, I would have known in ten minutes that there was very little for a judge to do.

I also need to be more careful about why I create test cases. Harder examples are useful when I want to measure what a tool can do. But when I am deciding whether the product needs that tool, examples I created to give it work can make the answer look better than it is.

The size of the test matters too. This was 121 items from one listing, all in English, with labels produced by models rather than people. I would not use these results as a general benchmark for JEV. They helped me make a specific decision about what Voomero needed at that point.

The Hammer Is Still on My Desk

I still want to use JEV, and there is another part of Voomero where it might fit. When agents set up a video, they can write free-form notes such as "please don't show the price" or "focus on the garden and keep it short". Today, those notes do not affect the video because nothing converts them into settings the planner understands.

To use those notes, I need answers to specific questions. Does the agent want to hide the price? Which topics should the video focus on? Did they ask for a different length? These are typed questions about a short piece of text, which is exactly what JEV is built for. There is also a problem in the product today: the planner cannot use preferences that agents write in their own words.

I have not tested this properly yet, although I could not resist trying a few notes. I wrote the examples below myself and sent each one to JEV once. These are the answers it returned:

It handled the two Portuguese notes as well as the English ones, which surprised me because the model works best in English. But it was less certain about whether "keep it short" or "make it snappy" meant the agent wanted a shorter video. If I only acted on confident answers, as I would in the product, those requests would be ignored.

That uncertainty is something I need to understand before relying on it. I do not see it as a failure, but seven notes I wrote myself tell me very little about how it would work with real requests.

This time, I want to start by measuring how many agents actually write a note and what they ask for. I still like the tool. I just need to understand the problem before deciding to use it.