// signals
Most LLM features fail the same way. Someone wires a model to the front of the flow, lets it decide what happens, and then can’t explain, or reproduce, why a given input produced a given answer. Re-run it, get a different result. I build them the other way around: the model goes last.
The decision you can’t let a model make
MeetKlay is a marketing consultancy’s diagnosis tool: a client engagement I rebuilt off Squarespace onto a custom PHP and SQLite stack. A visitor answers 22 questions across 6 stages, and the tool routes them to one of three engagement paths: Advisory for a clarity problem, Blueprint for a structural one, Strategic Oversight for drift. That routing is a commercial decision. It decides what the consultancy pitches and what the visitor is told they need. It can’t wobble between runs, and it can’t depend on how a language model felt about the prose that day.
Pass 0 decides. Everything after it explains.
Before a single model call, MeetKlay runs Pass 0: deterministic scoring. It’s a faithful port of the consultancy’s own calculator (a threshold engine, 6 routing rules plus 1 override, 20-plus variables), and it runs in roughly no time. Pass 0 picks the path.
Only then does the Claude pipeline start. An initial diagnosis pass. An optional brand-research pass that can use web search, hard-capped at 8 calls and skipped entirely when there’s nothing to look up. Then a synthesis pass that writes the final, personalised diagnosis, grounded in the respondent’s own words. It runs on claude-sonnet-4-6, called by a direct POST to the Messages API (no SDK, no proxy) in 35 to 115 seconds per submission.
Here’s the line that matters: the pass that writes what the client actually reads can’t move the recommendation. The model has real range (it reads freeform answers and public brand signals and writes something specific instead of a template), but the path is already set before it opens its mouth. The one place the model is allowed to challenge the score is a single early pass, logged and bounded on purpose. It is not free rein.
The receipt: replay before you publish
The hard part isn’t the first build. It’s keeping determinism alive after a non-engineer starts editing the thing. MeetKlay ships a CMS that lets someone with no code edit questions, options, branching, and the scoring weights themselves: exactly where reliability usually dies.
So publishing isn’t a save button. It runs 8 invariants, then the scoring test suite, then replays the last 50 submissions through the new configuration to flag any recommendation that would flip, and only then swaps the config atomically, with a .bak rollback if anything looks wrong. At the 0.3.0 publish check, I replayed 13 historical production submissions through the regenerated scoring. Zero recommendation flips. That’s the whole point of putting determinism first: you can prove the decision layer is stable, because it’s math, not vibes.
Determinism where it’s trusted, the model where it helps
None of this is anti-LLM. The model does the thing only it can do: turn 22 raw answers into a diagnosis that sounds like it was written for one person, because it was. What it doesn’t do is make the decision the client is paying for. Put the model where it adds range. Put determinism where it has to be trusted. Keep a hard line between them, and ask the model to be right about nothing more than the words.
If you can’t replay your LLM feature and get the same decision every time, you don’t have a feature: you have a demo. Making the model go second is what turns the demo into something you can ship.