Assurance is the engineering that lets promises be kept: testing, evaluation, review, monitoring and release control. More assurance is not always better. Its level should follow two things: how severe the consequences of an error are, and how hard the result is to verify. A market summary and an automatically placed order need assurance that differs by orders of magnitude. AI systems raise two new problems for assurance. The same input can produce different outputs, so correctness cannot be judged from a single test; it needs statistical validation on a sufficiently large evaluation set. And a system that performs well on common cases may fail on rare but critical ones, so evaluation must cover those cases specifically. After launch, markets and users change, so assurance has to continue as ongoing monitoring and staged release. Finally, assurance itself has a cost. Too much of it slows iteration, and its level should be proportionate to the risk.
The functional safety standard for automotive electronics sets the safety integrity level from three factors: the severity of harm, the frequency of exposure, and whether the driver can control the situation (International Organization for Standardization, 2018). A financial agent can borrow the same approach and set the level of assurance along two dimensions: how severe the consequences of an error are, and how hard the result is to verify. A function with light consequences that is easy to verify, such as a market summary with its sources attached, needs only spot checks. A function with heavy consequences that is hard to verify, such as executing trades automatically within the granted authority, needs formal constraints, checks on every action at the execution layer, and human review. Most functions lie between the two.
Assurance level: consequence × difficulty of verification
Across is how severe the consequences of an error are; down is how hard the result is to verify. Each cell shows its assurance level. Tap a function to see the assurance it needs.
In traditional software the same input always produces the same output, so one passing test shows that a path is correct. An AI agent is different: ask the same question twice and the answers may differ; one correct answer does not mean it is always correct. Judging its error rate requires a sufficiently large set of evaluation questions and a count of the results. The statistical "rule of three" gives an intuitive threshold: if no error occurs in n independent trials, then with 95% confidence the true error rate is no more than about 3 ÷ n (Hanley and Lippman-Hand, 1983). To say with confidence that the error rate is below 1%, at least about 300 questions must all be answered correctly.
How many questions before you can say the error rate is below 1%
Drag the number of evaluation questions and the number of errors among them, and see the highest the true error rate could be at 95% confidence.
If an evaluation set is drawn in proportion to actual use, the great majority of questions are common, simple cases. Overall accuracy can be high while failures on rare cases stay hidden. In finance the most critical cases are precisely the rare ones: the day of expiry, sharp market swings, trading suspensions and restructurings, missing data. Evaluation should be stratified by case, with accuracy computed separately for rare but critical cases and a higher bar set for them. It should also actively seek inputs that make the system fail. Ganguli and colleagues (2022) organized large-scale "red teaming" to find inputs that lead models to produce harmful outputs, and released nearly 40,000 records of these attacks.
Evaluation at launch only shows how the system performed on the data of that time. Afterwards, markets enter new states, users ask new questions, and data sources change their formats. Sculley and colleagues (2015) called such problems, which accumulate over time in machine learning systems, hidden technical debt: a system's performance can quietly degrade without any change to the code. Key metrics therefore need continuous monitoring, for example the rate of missing data, the rate of requests for instructions, the share of outputs corrected by users, and the rate of missed alerts. When a metric crosses a preset threshold, an alert fires automatically, and if necessary the system rolls back to the previous version or to a more conservative mode.
Even a new version that has passed every evaluation can reveal problems after launch. The standard practice in site reliability engineering is staged release: first let a small share of traffic use the new version, watch the key metrics, and widen the release step by step once no problem appears (Beyer and colleagues, 2016). A defect the evaluation missed then affects only a few users, and can be found and withdrawn before the release widens. For functions that affect trades and money, the first stage can run without taking effect: the new version makes its decisions but does not execute them, and they are compared with the old version's decisions.
The stronger the assurance, the slower and more costly the launch. Boehm and Turner (2004) argued that the rigour of a process should be adjusted to risk: agility where risk is low, discipline where risk is high. Choosing a level of assurance can be seen as a trade-off. With too little assurance, the expected loss from defects is high. With too much, the cost of review and waiting is high, iteration slows, and users wait a long time for improvements. The optimal level differs for functions of different risk; assuring every function to the highest standard is as wrong as assuring every function to the lowest.
How much assurance is right
The stronger the assurance, the smaller the expected loss from defects and the higher the cost of review and waiting. Choose a function's risk level and see where total cost is lowest (illustrative).
Grade each function by consequence and difficulty of verification; validate with statistical and stratified evaluation; after launch, monitor continuously and release in stages; and keep the level of assurance proportionate to the risk.
Spot checks, evaluation sets, stratified evaluation and review, formal constraints and checks on everything, allocated by level.
Set the number of questions by the error rate to be shown; even with 0 errors, you need at least 3 ÷ target error rate questions.
Expiry days, sharp swings, suspensions and missing data must each meet the bar on their own.
Set alerts and rollback on key metrics; let new versions go to a small group first; do not slow low-risk functions with the highest standard.