Product × Finance × AI: Financial Products in the Age of AIPart Three Deriving a product中文
Chapter 23

Agents make mistakesLimits, confirmation and compensation

The agreement is clearly written and the process is visible, yet the scribe will still make mistakes. It may deposit grain meant for the house account into the retirement account, execute "sell 2 portions" as "sell 20 portions", or keep the agreement perfectly and still misjudge. Some mistakes can be corrected the next day; others, once made, cannot be undone. No single line of defence is foolproof. The only option is to layer defences so that the chance of one error passing through all of them becomes very small. The scribe is fast and its actions are tightly linked, so an error can spread quickly before anyone notices it; it must therefore be slowed down at key points. The limit on each step determines the worst possible loss, and operations that cannot be undone require his own confirmation. When something goes wrong, who pays and how much must be written into the agreement in advance. After a mistake, the cause must also be found and written back into the agreement.

1. The scribe makes mistakes

The scribe's mistakes come in three kinds, and they differ in nature. The first is an execution error: the agreement says to sell 2 portions and it sells 20; the grain should go into the house account and it goes into the retirement account. This is the scribe's own fault. The second is overstepping: it takes risks outside the boundaries of the agreement, or makes trades that conflict with his interests. This is a breach of the agreement. The third is a misjudgement: it keeps the agreement completely, shifts the share of ship shares back before a storm as the agreement says, and the storm comes anyway. This is no one's fault; it is a risk the agreement itself has accepted.

The consequences of the three kinds should fall on different people. Execution errors and overstepping are paid for by the party that deploys the scribe, the operator. A misjudgement is borne by him, as the agreement provides, and he knows in advance that such cases can occur. If no one can tell which kind a mistake is, compensation cannot even be discussed. This is one reason the previous step required the scribe to keep "what was known at the time" and "which clause it relied on".

There is one new constraint: agents make mistakes, and some consequences cannot be undone.

Today's counterpart. Investor protection funds provide cover when a brokerage fails and client assets are missing, but they do not cover falls in the value of securities. The Securities Investor Protection Corporation in the United States covers each account up to $500,000, including up to $250,000 in cash, and states explicitly that market losses are not covered.

Three kinds of mistake, three kinds of consequence

Select a kind of mistake to see an example, whose fault it is, who bears the consequences, and how the records tell it apart.

Illustrative: the examples are fictional.

2. Layered defences

For an error to cause a loss, it must first pass through several checkpoints: the boundaries of the agreement, automatic checks before execution, the limit on each step, confirmation of key operations, and reconciliation afterwards. Every checkpoint has holes: the boundaries are incompletely written, the checks miss something, the limit is set too high, he does not read carefully when confirming, the reconciliation sample misses the error. As long as the holes in different checkpoints are not in the same place, it is hard for one error to pass through all of them at once. But if several checkpoints share the same hole, for example if they all rely on one set of the scribe's own data, they will fail together.

Today's counterpart. The Surgical Safety Checklist published by the World Health Organization in 2008 divides an operation into three checkpoints: before anaesthesia, before skin incision, and before the patient leaves the operating room. At each checkpoint the team confirms every item before continuing.

Five lines of defence

To cause a loss, an error must pass through five lines of defence. Drag the share of errors each line misses, then tick "several lines share one set of data", and see how many of 10,000 errors pass through every line.

Illustrative: the five lines of defence, in order, are the boundaries of the agreement, automatic checks before execution, the limit on each step, confirmation of key operations, and reconciliation afterwards (Reason's Swiss cheese model).

3. Fast and tightly linked

When people make mistakes, they usually make them slowly, and others have time to notice. When the scribe makes a mistake, it can repeat it a thousand times in a second. Each of its steps also drives the next: a sale triggers a rebalancing, the rebalancing triggers a purchase, and the purchase changes the proportions again. Before anyone notices, an error can spread rapidly along this chain. The more tightly the parts of a system are connected and the faster they move, the harder it is to stop an error midway; only where the system is looser and slower does an error have a chance of being caught.

A 2013 order of the U.S. Securities and Exchange Commission describes an incident. On 1 August 2012, while a U.S. market maker was deploying new trading software, one of its eight servers was not updated, and a piece of old code that had been unused for years was triggered again. In 45 minutes it sent millions of orders into the market, executed more than 4 million trades in 154 stocks, and ultimately lost more than $460 million. The order found that the firm lacked adequate controls to limit the risks of market access; it was the first enforcement case under that rule.

Today's counterpart. The market access rule requires brokers to set credit and capital thresholds before orders enter the market, so as to block erroneous orders.

Fast and tight: no time to stop the error

A faulty scribe sends 100 erroneous orders per second. Drag the time it takes to discover the error, then turn on two ways of slowing down, and see how many erroneous orders go out.

Illustrative: each erroneous order causes an average loss of 0.01 portions. In the 2012 incident at a U.S. market maker, the error lasted about 45 minutes (U.S. Securities and Exchange Commission, 2013); the time unit in this figure is minutes.

4. Limits determine the worst loss

Since mistakes are unavoidable, the first task is to work out how bad the worst case can be. The most grain a single mistake can move depends on how much that step is allowed to move. The most that mistakes can accumulate in one day depends on the daily total limit. Set limits on every step, every day and every account, for example no more than 2% of household wealth per step and no more than 5% cumulatively per day, and the worst loss changes from "everything he owns" to an amount known in advance. The limits need not be very low; they only need to match the amount the task actually requires. If he invests 2 portions each month, there is no need to allow a purchase of 20 portions at once.

Today's counterpart. Bank cards and online payments usually carry per-transaction and daily limits, and amounts above the limit require additional verification.

Limits determine the worst loss

The scribe makes 20 mistakes a year. The losses vary in size, and now and then one is very large. Add three layers of limits one by one, and see the total loss for the year and the worst single loss.

Illustrative: the losses from the twenty mistakes are expressed as a percentage of household wealth. Most are below 1%, with the occasional very large one; the 13th and 14th mistakes occur on the same day. The vertical axis is drawn up to 30%.

5. Confirmation and slowing down

Operations that cannot be undone, such as moving grain out of the granary, drawing on the house money early, or changing the agreement, require his own confirmation. Yet by the twentieth confirmation he no longer reads what he is confirming. For confirmation to work, it must be rare and carry weight: require it only for operations that cannot be undone, and when asking, set out the consequences of the step rather than offering only an "Agree" button. For very large amounts, wait for a period after confirmation before executing, leaving room for him and the scribe to change their minds.

Today's counterpart. A 2016 notice of the People's Bank of China provides that when an individual transfers money to another person through an automated teller machine, the issuing bank carries out the transfer only 24 hours after accepting it, and during this period the individual may ask to cancel it.

Confirmations should be rare and carry weight

In a year there are 4 operations that cannot be undone, and 1 of them is in fact wrong. Switch between two ways of confirming, and see how many times a year he is asked to confirm, whether he reads carefully, and whether that one error is stopped.

Illustrative: in a year he can read only about 12 confirmations carefully. When he does read carefully, a confirmation that sets out the consequences lets him spot the error 90% of the time, while one with only an "Agree" button does so only 30% of the time. During a one-day delay, the scribe's anomaly checks can stop half of the errors he missed (the value of keeping options open described by Arrow and Fisher, and by Henry).

6. Who pays

Limits and confirmation make errors smaller and rarer, but they cannot make them disappear. When something goes wrong, the rules for compensation must be written in advance. Execution errors and overstepping are compensated in full by the operator: as long as the records show that the scribe breached the agreement or executed incorrectly, he is not required to prove who was at fault. Misjudgements are borne by him under the agreement, but when the agreement is made the scribe must tell him how large such losses could be. The operator insures against these payouts, turning occasional large compensation payments into a predictable annual cost.

Today's counterpart. The Financial Industry Regulatory Authority in the United States provides an arbitration procedure for disputes between investors and brokers.

Who pays

Four kinds of loss, four places they fall. Select a row to see the basis and conditions for compensation.

Illustrative: the compensation rules are a product design derived in this book, not a restatement of any current regulation; the row "Broker fails, assets missing" follows the scope of protection of the Securities Investor Protection Corporation.

7. After the mistake

Paying compensation does not close the matter. The same mistake will happen again unless someone finds out why it passed through every line of defence: was something left out of the agreement, was a limit set too high, or were the consequences not set out clearly enough at confirmation? Once the cause is found, what needs to change is often not the scribe but the agreement and the defences themselves. The change must also be written into the records so that he knows why the agreement changed. Some mistakes can still be reversed shortly after they occur: a clearly erroneous trade can be cancelled on application within a set time.

Today's counterpart. FINRA rules allow trades with clearly erroneous terms, such as price or quantity, to be reviewed and, where the conditions are met, cancelled.

Six steps after a mistake

After grain meant for the house account was deposited into the retirement account by mistake, what the scribe and the operator do, in order. Select a step to see the details.

Illustrative: the example is fictional.

8. Ten years behind the defences: Civilization Simulator, version 23

The simulator unlocks "limits, confirmation and compensation". Over ten years the scribe makes two kinds of mistakes, execution errors and overstepping; misjudgements are market risk accepted by the agreement and are not counted. You can turn on or off the per-step and per-day limits, confirmation and delay for irreversible operations, and the operator's compensation, and adjust how often the scribe makes mistakes. Then see the losses caused by mistakes over ten years, the part he bears himself, the extra fees he pays for this protection, and how many times he is asked to confirm.

Civilization Simulator · Chapter 23: ten years behind the defences

Over ten years the scribe makes two kinds of mistakes, execution errors and overstepping, and some of them occur in operations that cannot be undone. Turn three rules on or off and adjust how often mistakes occur, and see the losses caused over ten years, the part he absorbs himself, and what he pays for protection in confirmations and fees.

Product: what to promiseProduct: mandate and authorityProduct: process records and verifiable evidenceProduct: limits, confirmation and compensationProduct: the path to trust · Ch. 24

Illustrative model: the figure shows expected values over ten years, with losses as a percentage of household wealth. At the baseline frequency, there are 2 execution errors a year; 98% of them lose about 0.2%, and the rest about 10%. Overstepping occurs 0.3 times a year, with an average loss of 2%. With limits, a single execution error loses no more than 2%, and losses from overstepping are halved. 20% of both kinds of mistake occur in operations that cannot be undone, and confirmation with delay stops 80% of these. When the operator compensates, all losses from execution errors and overstepping are paid, and the cost of compensation and insurance is added to his annual fees at 1.2 times. Misjudgements are market risk accepted by the agreement and are not shown.

9. What this step leaves open

Defences, limits and compensation keep the consequences of mistakes under control, but several questions remain unresolved:

  1. At the very beginning. A scribe that has just opened for business has no record. On what basis should he begin to trust it?
  2. How much trust. Once a record of keeping the agreement builds up, how far and how fast can the authority be widened?
  3. Trust after a mistake. After one mistake, does trust fall to zero, or does it only step back one level?
  4. The whole product. Promises, the agreement, records, defences and compensation: what kind of product do they make together?

What this step leaves open

Four questions that remain once the consequences of mistakes are under control. Select a question to see the assumption it hides.

Behind the defencesLimits, confirmation and compensation

10. Implications for products

For a wealth-management product built around an agent, the fourth part to grow is "limits, confirmation and compensation". Three implications follow. First, allocate consequences by kind of error: execution errors and overstepping are compensated by the operator, which insures against them; misjudgements are borne by the user under the agreement, and the user is told at the outset how large they could be; the records determine which kind each error belongs to. Second, layer the defences and rely on none of them alone: the boundaries of the agreement, checks before execution, per-step and per-day limits, confirmation and delay for irreversible operations, and reconciliation afterwards, with the lines of defence kept as independent of one another as possible. Third, deliberately slow down where things move fast: limits and delays let errors be caught before they spread; after a mistake, find the cause, change the agreement and the defences, and tell the user why they changed.

Chapter summary: agents make mistakes

Four conclusions, each tied to one mechanism in this chapter.

Three kinds of mistakeConsequences allocated by kind

The operator pays for execution errors and overstepping; the user bears misjudgements under the agreement; the records decide which kind each one is.

Swiss cheese modelLayered, independent defences

No line of defence is foolproof, and lines that share a hole fail together.

Normal accidentsDeliberately slow down where things move fast

Limits, automatic stops on anomalies and delays catch errors before they spread.

Rerun: irreversibility and vicarious liabilityFew, weighty confirmations; compensation written in advance

Confirm only operations that cannot be undone, and set out their consequences; the operator insures, and compensation does not depend on the user proving anything.

References