Can you trust an LLM to do calculations?

Not as a rule. A language model predicts text, and a number in that text is a prediction like any other word. It will be right often enough to look reliable and wrong often enough to matter, with the same confidence either way. Let the model propose, describe and reason, and let deterministic code compute and verify. Every number a reader sees should come from code that can be re-run.

How do you stop an AI agent from hallucinating numbers in a report?

Do not let numbers travel through the model. Give it a fixed briefing built by code, make it return assumptions in a strict schema that a loader rejects when the shape or the range is wrong, recompute every derived figure independently, and fail the whole build on any inconsistency. The model's job is judgement about which assumptions are reasonable, with a stated reason for each.

What is a guardrail for an AI agent?

A check outside the model that decides what the agent may read, what shape its output must take, and whether the result is accepted. Examples: a deterministic context pack instead of free browsing, a schema on every write, a second implementation that recomputes the result, and a verification step that reads the artefact back before calling the run a success.

Why check the output file instead of the exit code?

Because an agent runner can exit cleanly having written nothing useful: a usage limit was hit, a transient error was swallowed, the model declined. Exit code zero says the process ended, not that the work was done. Read the artefact back and check that it changed in the way the task required.

Should the analysis still be reviewed by a person?

For anything that moves money, yes, and the review is easier when the machine has done the checking. The agent's output shows the alternatives it rejected and why, every number has a source, and the second computation has already flagged any disagreement. The person reviews the judgement, not the arithmetic.

Does this approach work outside finance?

It is the same rule wherever a document has to be both written and right: proposals, reports, plans, quotes. Let the model draft and argue, let code compute and check. We use the identical pattern for a meal planner: the model proposes meals and estimates their macros, a plain script checks every day against the targets and sends back a specific critique, bounded at two revisions.

The model never does the arithmetic: letting an AI agent write your analysis without trusting its numbers

Twice a day a job on our machine refreshes the earnings calendar for thirty-two companies, pulls any new results release, and, where a company's written thesis is older than its latest results, sends the thesis to an analyst agent for review. The agent rewrites the valuation assumptions and the narrative. Nobody watches it do this. The output feeds our own portfolio.

Every reader of that sentence has the same question, and it is the right one: how do you let a language model write financial analysis when language models make up numbers? The answer is the rule in the title. The model never does the arithmetic. This post is what that rule looks like when it is built rather than stated.

The short version

A number in a model's output is a prediction, like every other word. Let the model propose and reason; let code compute and verify.
Fix what goes in. The agent reads a briefing built by code, the same briefing every time, not whatever it finds.
Put a schema on what comes out. Assumptions return in a strict shape that the loader rejects when it is wrong.
Compute twice. A second implementation, from different inputs, recomputes the result and disagrees out loud.
Exit code zero is not success. Read the artefact back before calling the run done.

flowchart LR
  B["Briefing<br/>built by code"] --> A["Agent<br/>proposes assumptions,<br/>writes the thesis"]
  A --> S{"Schema<br/>check"}
  S -->|rejected| A
  S -->|accepted| W["Workbook<br/>computes the value"]
  W --> R["Offline recomputation<br/>from the same assumptions"]
  R --> D{"Agree?"}
  D -->|no| X["Build fails, loudly"]
  D -->|yes| V["Validator scans every tab"]
  V --> T["Thesis, every number sourced"]

What the system does

For each company on the watchlist, a pipeline pulls the financial statements, ratios, estimates, peers, dividends and macro rates from four sources and builds a twenty-tab workbook: the statements, the company's own operating metrics, and four valuation methods rolled into one composite fair value. The workbook is a spreadsheet with formulas. It computes; nothing else does.

The agent's job is the part a spreadsheet cannot do: decide what the assumptions should be. The discount rate. The growth path. How much weight each valuation method deserves for this particular company, and why. Then write the thesis: what the market believes, what we believe, where the two differ, and what would change our mind.

That is judgement, and judgement is what language models are good at. Arithmetic is not.

Rule 1. Fix what goes in

The first design decision was to stop the agent browsing. An agent that reads the raw data files, queries the sheet through a tool, and wanders through a folder of past notes produces a different analysis every time, and half the variance is which files it happened to open.

So the agent's entry point is a script that builds a briefing: one read of the live workbook, a few local files, rendered as a couple of thousand words of structured text. The current valuation, the consensus, the trend in the fundamentals, the company's own metrics, the assumptions currently in force and their history. The same briefing shape, every company, every run.

Two things follow. The run is reproducible, because the input is. And the agent is told, in its instructions, that no number may appear in its output that did not come from the briefing, the data files, or a cited source. Training-data recall is not a source.

flowchart TB
  subgraph IN["What the agent may read"]
    W["Workbook snapshot<br/>one read, ~20 ranges"]
    F["Local data files<br/>statements, metrics, estimates"]
    H["Assumption history<br/>what was set before, and why"]
  end
  IN --> BR["Briefing script<br/>renders ~2,000 words"]
  BR --> AG["Agent"]
  NO["Free browsing<br/>tool queries<br/>training-data recall"] -. "not available" .-> AG

Rule 2. Put a schema on what comes out

The agent does not edit the workbook. It writes one small file of assumptions, and a loader reads that file into the workbook build.

The file has a fixed shape. The valuation methods it may weight are an enumerated set. The discount-rate inputs have names. The scenario it points at must be one of five. A weight that does not sum, a key nobody defined, a source that does not exist: the loader rejects the file with a message that says which field and why. The agent then fixes its output, or fails, but the workbook never sees a malformed assumption.

The schema is where the model's freedom ends. Inside it, the agent can argue for a 9.3% discount rate over 8.5%, and it must say why, showing the alternatives it rejected: the trailing actual, the five-year range, the consensus, the peer range. Outside it, nothing gets through.

Rule 3. Compute twice

A workbook full of formulas can be beautiful and wrong. Ours was, once: a market-cap cell fed by a live data function returned a figure about fifty-nine times too large, and every per-share number downstream inherited it. Nothing showed an error. Every cell computed. The validator that scans for formula errors found none, because there were none.

The fix is a second implementation. A script rebuilds the main valuation from the assumption file and the data files alone, with no spreadsheet and no live data, using the same method written independently. It is never the source of a quoted number; the workbook is. Its permanent job is to be the second opinion, run across every company after every build, and to disagree out loud when the two computations part.

The general rule is older than software: a check that shares the inputs of the thing it checks is a consistency test, not a verification. Verification needs a path the candidate did not touch.

Rule 4. Exit code zero is not success

The agent runs headless, under a timeout, called by the scheduled job. And an agent runner can finish cleanly having done nothing useful. A usage limit was reached and it printed a polite line. A transient error was caught and swallowed. The model declined the task in prose. In every case the process exits with success.

So the job does not trust the exit code. After each review it reads the thesis file back and checks its own "last updated" marker against the date of the results it was meant to cover. Only then is the review counted as done. A run that exited cleanly and changed nothing is reported as a failure, by name, in the digest.

sequenceDiagram
  participant J as Scheduled job
  participant C as Calendar
  participant A as Agent (headless, timed)
  participant T as Thesis file
  participant D as Digest
  J->>C: refresh last and next results date
  J->>J: which theses predate their latest results?
  J->>A: review this company
  A-->>J: exit 0
  J->>T: read the "last updated" marker
  alt marker on or after the results date
    J->>D: reviewed
  else unchanged
    J->>D: failed, by name
  end

Rule 5. Ground it in what moved the stock

Fundamentals tell you what a company said. They do not tell you what happened to the stock afterwards, who took a stake, or who is calling for the chief executive's job.

We learned this on one company where two thorough passes, a full write-up and an adversarial review of it, both missed that a well-known investor had built a 4.7% stake and was publicly demanding a change of leadership. It was the single largest driver of the price that quarter, and the thesis had attributed the move to the market forgiving a weak result. A separate daily job of ours recaps eighteen finance channels into a dated archive, and the archive had the stake, two days after it happened.

The fix was not a better model. It was a rule: before finishing any thesis, the agent must search that archive for the company and read what it finds. The recaps are leads, not facts; anything material is checked against the filings before it enters the workbook. But a dated, sized position from a named investor is worth more as a bull-or-bear data point than a consensus price target, and the model will not know to look unless the environment makes it.

What this costs

More scripts than model calls. The briefing builder, the loader with its schema, the offline recomputation, the validator, the read-back check, the archive search: each is a small program with one job, and together they are most of the code. The agent's instructions are long, but they are mostly about stance: calibration over advocacy, show the alternatives you rejected, "fair value is below the price" is a legitimate conclusion.

What it buys is an analysis you can defend. Every number traces to a source. Two independent computations agree. The thesis changed when it should have, and the job says so when it did not. A person reviews the judgement and skips the arithmetic, because the arithmetic was never the model's to get wrong.

The same rule, everywhere

We use the identical pattern for a meal planner that reads a training log and a recipe corpus: the model proposes a week of meals and estimates the macros, a plain script checks every day against the targets and sends back a specific critique, and the loop is bounded at two revisions. The domain could not be further from equity valuation. The rule is the same, because the failure is the same.

Wherever a document has to be both written and right, split the work. Let the model draft, argue and choose. Let code compute, check and refuse.

What we do with this

This is the shape of most of the AI work we build for clients: not a chatbot over a document store, but an agent with a fixed view of the data, a schema on everything it writes, a second computation behind every figure, and a job that checks the work landed. If you have a report, a plan or a set of numbers you would like an agent to draft without ever trusting its arithmetic, get in touch, or see how we build and run software end to end.