The model is never allowed to tell you a number.
A market-analysis product for Indian equities, where the most important decision was what the language model was not permitted to do.
Fluent, confident, and occasionally wrong about arithmetic.
Most products in this category ask a language model to look at market data and say what it thinks. That's the wrong shape for a system whose output people put money behind.
The model is convincing whether it's right or not, and there's no reliable way for a reader — or for me — to tell which sentence to trust. A hallucinated price level isn't a UX defect here. It's a loss.
Prompting can reduce the odds of a wrong number. It can't remove the possibility.
So I stopped trying to make the model reliable and changed what it was allowed to touch.
Split the system in two, and put the boundary where the numbers are.
A deterministic, rule-based scoring engine became the system of record. It produces every score, level and trigger a user can act on, and it can be audited line by line. The language model sits strictly downstream, receives those numbers as input, and may produce prose only.
Two paths leave the engine. Only one carries numbers, and it never passes through the model. The worst case for the other path is plainer language — never a wrong figure.
Three more decisions from the same principle.
A validator on every figure
Each number in generated prose is checked against the record it came from. Mismatch, it retries once. Still wrong, it falls back to a deterministic template. There is no path from a hallucinated figure to a user.
A human approves every weight change
Nightly calibration computes Spearman rank information coefficient per component. It can propose a change to the scoring weights. It cannot make one — a person signs off against the evidence.
The user sets the thresholds
Telegram alerts fire at 2% on the portfolio and 2.5% on a single position — defaults the user can move. The product doesn't decide what's worth your attention.
Written for someone who isn't a quant.
233 stocks scored every night. Each gets four component scores, a forecast band across four horizons, and a written explanation of why it's moving — with a watch level naming what would invalidate the setup.
When the explanation layer can't run, the product says so in plain language and shows the base rates alone. It does not quietly present statistics as though they were analysis. Graceful degradation is easy to design and almost never shipped, because it means admitting in the interface that something didn't work.
It publishes its own hit rate, even when that's near a coin flip.
Every forecast band carries the historical win rate behind it and the sample size it came from — 45.7% over 6,921 observations at T+3, and similar through the longer horizons. Every early signal carries the same for its own type.
Those numbers sit close to a coin flip, and they're on the page, in front of the user, because the calibration loop is telling me the truth about what this system does and doesn't yet know.
It's also why the product isn't priced. I'm running a forward test before anyone pays.
Most products here would bury that. It's the number I trust most on the whole platform.
Every signal gets marked to a real outcome.
Paper trading runs on ₹30,00,000 of virtual capital against live NSE prices, benchmarked against the NIFTY 50 from the same start date. Track-record and behavioural logging capture every signal as it fires and attach its real forward outcome once matured.
Positions carry a written read in the same plain language as everything else — "Quiet. Barely moved, on no news and below-average volume" — with the specific flags behind it. Not a terminal readout.
Six things that have to hold before a number reaches a user.
Most of these existed before I had a name for them. Writing them down as a set is what turned a collection of defensive habits into something I could hand to a reviewer — and what showed me which two were missing.
The model cannot produce a number
Figures are computed by the scoring engine and passed in. The model writes prose around them. There is no code path where a generated token becomes a displayed price.
Every figure is validated against source
Numbers appearing in generated text are checked back against the record they came from. A mismatch retries once, then falls through to a deterministic template.
A human approves every weight change
Nightly calibration can propose a change to the scoring weights. It cannot apply one. Nothing about how this system scores moves without a person signing off against the evidence.
A defined degraded state
When the model is unavailable the numbers still render and the page says the written commentary is off. Never a blank screen, never a silent gap where an explanation should be.
Bounded blast radius on untrusted text
News headlines reach the model. It cannot emit a figure, place an order, or call a tool — so the worst a crafted headline achieves is bad prose, not a bad trade. Containment rather than filtering.
A cost ceiling per user
All thirteen AI workloads write a per-call cost log that reconciles against the provider's own billing. Spend is a product metric with an owner, not a monthly surprise.
Two are missing, and I would rather name them than have you find them. There is no measured groundedness rate on the claims in generated prose — only on the figures. And I have never run an adversarial test to see what a deliberately crafted headline could make the model write.
I cut the running cost by three quarters and the output got better.
I'd been treating the model bill as one number at the end of the month. So I instrumented every workload to write a per-call cost log — every explanation, every forecast, every nightly job, tagged and priced.
Most of the spend wasn't users. It was scheduled pre-generation running against the entire stock universe every night, whether anyone looked at those stocks or not.
Same coverage, same schedule, better output — because most of what I removed was work nobody had asked for.
Per-workload instrumentation is what made the reduction findable. One monthly figure would have hidden it entirely.
I had my own measurement audited. It was wrong in three ways.
Publishing a hit rate only means something if the number behind it is computed correctly. So I ran a structured audit across the scoring and calibration code. It came back with nine findings. Three of them changed what I am willing to claim.
The information coefficient was pooled across dates
Rank correlation was computed over every stock-date at once. Pooled that way it largely measures whether the market moved together — not whether my scores ranked stocks against each other, which is the only thing it was there to test. It has to be computed within each date and averaged across dates, with a standard error.
Every stored IC value is void. I don't quote them, and I say so before I'm asked.Win rates were being published at n = 1
Buckets were ranked by raw win rate, which meant the top of the list was, by construction, the least-evidenced bucket on it. One lucky observation outranked four hundred honest ones — and the top of the list is exactly where a user looks first.
Gate at n ≥ 30, rank by a lower confidence bound rather than the raw rate. Scoped, not yet shipped.It was in-sample. There was no holdout.
Bucket win rates included each signal's own resolved outcome, so "this bucket wins 68%" was a description of the past wearing the costume of a forecast. A random split wouldn't fix it either — returns are autocorrelated, so the split has to be chronological.
Walk-forward split, fit and holdout reported separately. Scoped, not yet shipped.For most of this build, the numbers I was proudest of were the ones least entitled to be believed.
The headline figure survives: 45.7% at T+3 is an aggregate across every realised outcome, not a bucket anyone selected, so findings 02 and 03 don't reach it. It's the per-bucket win rates I've stopped quoting. Holding that distinction is the actual work — a product that reports on itself needs its self-reporting audited harder than anything it reports on.
I built the measurement fifth. It should have been first.
I snapshot outcomes after the fact rather than at the instant a signal fires, which means I can't cleanly attribute a result to the exact model version and weights that produced it. That gap now blocks the backtest engine entirely — everything downstream is sequenced behind fixing it.
Fixing it first would have cost a week. Fixing it fifth cost me a quarter of measurement I can't reconstruct. It's the clearest lesson I've taken from this build: in a system that learns from outcomes, the outcome schema is not a later problem.