How do you know
the agent is right?

A wrong answer arrives looking exactly like a right one

Ask what the exposure is on overdue loans and you get a number and a chart. If the agent joined the wrong table or quietly dropped a filter, you still get a number and a chart. There is no stack trace, no error, nothing to notice. The failure mode is plausibility, and a bank acting on it would never know.

Retrieval is fuzzy. Relationships are not.

Which tables matter for “how many customers are overdue” is a judgment call, so that is retrieval's job: M-Schema documents in a vector store, searched semantically. Which tables can legally join is not a judgment call at all. It is a fact, it is cheap to compute, and it is complete.

So the foreign-key graph is never retrieved, it is stated, as the only relationships that exist, and the same graph is loaded through a single function so the prompt and the validator can never disagree about what the schema is.

Use retrieval where the answer is uncertain. Use the schema where it is knowable.

The known gap: a table that bridges two others is named in neither the question nor the retrieved set, so it arrives as a relationship without a column list. The literature calls this structural blindness. One hop of graph expansion is the fix, and it is on the list.

Checks for queries that run but do not mean what they look like

Generated SQL is parsed with sqlglot and its scope tree walked rather than pattern-matched over the text, so quoted identifiers, USING(…) joins and CTE scoping all resolve correctly, so an alias reused inside a subquery cannot be mistaken for the same alias outside it.

A join between tables with no foreign-key relationship is rejected before execution. The rejection goes back to the generator as a message it can act on, and the loop is bounded at three attempts, after which the agent says so rather than trying forever.

The model never does arithmetic

The synthesis prompt shows the model a sample of the rows. So any figure describing the whole result set is computed in code and handed over with instructions to quote it verbatim and derive nothing else.

Asking a model to total a 40-row sample of a 500-row result is how a confidently wrong naira figure reaches a manager.

Identifier columns are skipped, because the sum of a column of loan ids is a number but it is not a fact about anything.

The test suite cannot reach this

Pytest tells you the code still does what it did yesterday. It cannot tell you whether the agent still answers correctly, because that depends on the prompt, the schema documents in the vector store, the strictness of the SQL validator and the model itself, none of which the test suite touches. Every one of those can change without a single test going red.

So the suite runs real questions through the real graph, and scores the answers.

The hard part is comparing results, not strings

Two completely different queries can both be right. Comparing generated SQL to a reference string would fail almost everything correct, so the harness runs both and compares what came back: ordering enforced only where the gold query actually ordered, columns matched rather than assumed positional, and values normalised so a decimal and an integer of the same amount agree. Most of the scoring code is this, not the plumbing.

Some cases exist to be refused

The 34 cases are not all lookups. There are aggregations, joins, multi-hop joins, negation, temporal questions and rankings, and then the categories that matter most: relationship traps, refusals and safety. A question the agent should decline is as much a test as one it should answer, and an agent that cheerfully answers everything has not passed, it has failed in a way the other cases cannot see.

A suite is only useful if people believe it

A rate limit or a provider outage is reported as broken, never as failed. Scoring a quota blip as a regression is how a suite loses its credibility. Someone spends an afternoon hunting a prompt bug that was never there, and after that nobody trusts a red run.

For the same reason the headline number is not the pass rate. You save a run you were happy with as the baseline and compare against it, and what the comparison prints is the list of cases that went from passing to failing. The direction of travel is the thing worth acting on, not the score.

The corrections are not trusted either

When someone flags a wrong answer, the correction does not go into retrieval. It waits. A corrected question/SQL pair is admitted as a worked example only once it has passed the same gauntlet generated SQL passes: read-only, the same relationship checks, and it must actually execute.

A pair that cannot run is a plausible-looking lie with a manager’s name on it.

And it earns its place only if it teaches something the schema cannot, like “out on the streets” meaning status = 'ACTIVE'. Anything a competent analyst could derive from the schema alone just costs a retrieval slot. The distrust is symmetric: the model is not trusted, and neither is the person correcting it.