AI at scale in enterprises: part 2

Last year I wrote about agents at scale in enterprises. It’s still very valid, except for its name. Today I’d call it AI at scale in enterprises. Agents… I’m not even sure they fit enterprises. Let me try to explain.

Last year we spent two months here in San Francisco. After one year executing, I just came back for one more month, this time by myself, while the team stayed in Brazil handling operations and the client load.

We played the whole playbook

Throughout these last 18 months we played a lot with agents. We built our own AI Gateway, the CLI harness, the agent sandbox, the platform and the app building infrastructure, so anyone can build their own apps. We played the whole playbook, and I think we did really well.

The managed agent is really great. It works well in its own ephemeral Linux sandbox, and the user manages its context, bounded by our security and distribution model. Any user can build their own app and deploy it. Users are really using it, building their apps and being happy.

But the intelligence is still limited to building apps, not operations. Like we always believed, finance and accounting can’t be 99.9%. It’s either 100% right or it’s wrong, except for some judgments that require interpretation, such as insights or structuring a chart of accounts. Even flagging a suspicious transaction can be done with deterministic rules.

0.99 × 0.99 × 0.99

In that sense, the current LLM paradigm simply does not fit.

An LLM works by predicting the next token. Agents run that in a loop and, sometimes, long-running processes run in an outer loop. Each of those steps carries an error. Even with the perfect context and the perfect harness, you’re still multiplying 0.99 × 0.99 × 0.99. And chaining these responses, handing off to another agent, makes things worse.

That’s the same idea we presented 18 months ago, and it didn’t change, even with the models evolving A LOT. This simply does not fit large-scale operations, and companies keep getting deeper and deeper into a paradigm that simply does not work there. It’s like drinking seawater expecting to kill your thirst.

Kind of works

In the meantime we had some success using AI with structured outputs and events, for a couple of very specific things.

One is the account discovery algorithm. We use Gemini Flash Lite to extract the stable parts of a transaction description, then query the indexed base to discover account candidates for that transaction. That worked, somehow. It was a very long journey to get to an algorithm that works with thousands of transactions loaded at once.

We also added structured outputs to decide if an uploaded file is an invoice, a receipt or a statement, and hand off its extraction to the proper parser. Those parsers use Gemini Flash structured outputs, plus an agentic context engineering process that runs automatically on user corrections: it adjusts the prompts, saves them in properties and uses them for the next parsing. Self-improving, getting more “intelligent” over time.

That kind of works, but it’s really slow, expensive and inaccurate. For one big client, Brain, we had to create specific OCR parsers to make the process predictable and move forward. Even choosing the proper parser is a decision point, a big IF statement over lots of variables, and some of those variables are in the very document that needs to be parsed (argh!).

That pain got me really skeptical about the whole AI and agents thing. We even doubled down on our own CLI agent, the harness and all the infrastructure for people to build and deploy apps. Yet I didn’t see a light at the end of the tunnel for AI being used in operations, not just in code.

Until September 15th.

The light at the end of the tunnel

That day TypeSafe released Jev, and Diogo Almeida, its founder, explained a new class of model, outside the LLM paradigm. It’s a model trained for judgments, meant to be used by systems. A single call makes multiple judgments, in a tiny fraction of the time and a tiny fraction of the cost. It’s like a single next-token prediction that returns a series of judgments, with a confidence level on each.

Suddenly that light at the end of the tunnel started to appear!

I quickly started playing with it, and it was really promising. Way faster, cheaper, more precise and more predictable than LLM structured outputs. Report is one of the things I built with it.

Then many others started appearing. Cloudflare built Clef, OpenAI released its Decisions API on top of Luna, and others are cooking right now. Some look more like a structured output around an LLM, similar to what we already had, but with a similar spec, which I understand will soon evolve into better calibrated judgments.

And then the hard part came… how could I test them in so many different places? Our apps were already full of Jev calls, and testing another model meant changing every one of them. I could use the AI SDK’s experimental Decisions, but it would be yet another coupling to a specific implementation.

A very smart IF statement

So I added it to our AI gateway, behind a standard interface, so I could safely integrate it across the codebase and change the underlying model without breaking any client.

At the beginning I called it “Decisions”. I wanted to use the TypeSafe TypeScript client, so I wrapped it in a TypeSafe-compatible spec and added a couple of adapters, to use and test different models without changing my code. Similar to what we already did with the Open Responses spec.

This turned out to be on another level of importance than Open Responses. LLM interactions are usually limited to a chat or an agent, so the point of coupling is limited. But small atomic judgments spread across the whole system. It’s like a very smart IF statement. It can go anywhere a judgment is needed.

And things kept unfolding during this month in SF. Cloudflare came out with Clairic, a model focused on extraction, not decisions, and on November 11 Google came out with Gemini Decision 1.

I went to The Pragmatic Engineer Unplugged, where Diogo was on stage. He is really passionate when explaining his point of view, and things started to get clearer for me. The broad category is not decision or classification models. Well, a model can specialize in those, but the paradigm is broader. In a single call, you can classify, extract data, forecast or predict things. That’s when I started to understand Diogo’s vision.

So we embraced it, and refactored our AI gateway and primitives around System One models. From all my reflections, that’s the closest definition of this new category of models.

Open System One

At first I extracted the spec into its own internal repo. At some point it struck me that the most important thing now is an open spec, so we can keep integrating it into our systems and shield our adoption from the rush and the hype. So I moved the repo to its own org and published it as the Open System One spec.

I hate this “Open xxx” everything, but it was the closest thing I found to say “an open spec for System One models”. “Open System One Responses” was too big, and still had the “Open” in it anyway.

I don’t expect, or intend, to make it the canonical spec every lab and provider will follow. I just wanted to:

1. Build our own infrastructure as if such a spec already existed, so we don’t end up coupled to a specific provider, can test many of them, and streamline our own adoption.
2. Genuinely express the problem, so maybe some day an official spec emerges and we can shut this one down.

Back to operations

For the first time, I see a real possibility of using AI in operations consistently, at scale and in a really serious way, driven by events and consistent, intelligent judgments.

We will double down on rolling that out to our clients over the next months, and see how it goes.

Software is so back.

Find me on X at @maelcaldas.

← All posts