Product × Finance × AI: Financial Products in the Age of AIPart Three Deriving a product中文
Chapter 22

The user cannot see the processProcess records and verifiable evidence

A year has passed. The villager opens the ledger and sees that the retirement account is up 10%. Is this the result of the scribe keeping to the agreement, or did it take risks outside the agreement and happen to win? Conversely, if the account were down 10%, would that be a bad run of years or a breach by the scribe? From results alone he cannot tell. For him to see the process, the records must be verifiable; otherwise they are only the scribe's own account. Verification takes time, so not every entry can be checked. Records must be layered so that he can understand them though he looks only a few times a year. They must state what was known at the time, so that decisions are not judged afterwards by their outcomes. And every step must state which clause of the agreement it relies on.

1. Results alone cannot tell good from bad

A 10% gain in a year may mean the scribe kept the agreement and the years were good; it may also mean it quietly put the house money into ship shares and happened to catch a good season for the fleet. A 10% loss may mean a storm, or it may mean a breach. The swings of good and bad years are far larger than the scribe's influence, so judged by results alone, good and bad scribes are almost indistinguishable over a year or two. A principal who cannot tell them apart can only extend trust at the average level: good scribes receive less trust than they deserve, and bad ones are overrated.

There is one new constraint: the user cannot see the agent's process, and information is asymmetric.

Today's counterpart. A fund's one-year performance depends largely on that year's market; the same return may come from very different levels of risk.

Results alone cannot tell good from bad

Four of twenty scribes breach the agreement by taking extra risk outside it. Each dot is one scribe's result. Drag the number of years observed and see whether results separate those who keep the agreement from those who breach it.

Scribe keeping the agreementScribe in breach

Illustrative: scribes who keep the agreement earn 5% a year on average, each with volatility of 10%; those in breach take more risk, with an average return of 6% and volatility of 22%; all of them also share the same run of years. The horizontal axis is the annualized return over the observation period.

2. Verifiable records

Suppose each scribe can choose whether to make its records public, and public records are kept by a party unconnected with it and cannot be altered. Work through what happens. The scribes that keep the agreement most strictly will certainly disclose, because disclosure sets them apart. Among those that remain undisclosed, the better ones will also disclose; otherwise principals will lump them together with worse ones. Round after round, as long as public records can be verified and disclosure costs little, almost every scribe discloses, and not disclosing becomes bad news in itself.

Experiments support this reasoning only in part. In the experiments of Jin, Luca and Martin (2021), receivers of information were often not sceptical enough about non-disclosure, so senders did not always disclose; full disclosure emerged only gradually when receivers were told after each round what had been withheld. For verifiable records to work, the product must also help the user read "no record" as a signal.

Today's counterpart. Listed companies must publish audited financial reports at regular intervals; refusing or delaying disclosure in itself puts investors on alert.

Silence becomes bad news

Ten scribes score from 1 to 10 on how well they keep the agreement, and each decides whether to make its verifiable records public. The principal treats every scribe that does not disclose as having their average score. Click "Next round" to see who discloses.

Illustrative: a scribe discloses when its own score exceeds the principal's estimate for those who do not disclose (the unravelling process of Grossman, 1981, and Milgrom, 1981). When the principal is "not sceptical enough", it always treats non-disclosers as a 7, matching the insufficient scepticism of receivers in the experiments of Jin, Luca and Martin (2021).

3. Verification has a cost

That records can be verified does not mean every entry must be. Checking several hundred actions a year one by one would take a great deal of his time, and the labour of another scribe as well. Work through where verification should be spent. When results meet the agreement and every indicator is within its boundaries, a detailed check yields little; when results are poor or a condition in the agreement has been triggered, a detailed check yields the most. Add unannounced random spot checks in ordinary times, and the scribe cannot predict which entry will be examined, so the temptation to breach the agreement shrinks.

Today's counterpart. A financial audit does not examine every transaction. It sets priorities by risk and then tests samples; the International Standards on Auditing contain specific provisions on audit sampling.

Where to spend verification

The scribe carries out 300 actions a year, 6 of them breaches. Most breaches occur where results are poor or a trigger fires, but some hide among good results. Switch among three ways of checking and compare the effort spent with the breaches found.

Illustrative: of the 300 actions, 30 had poor results or set off a trigger, and 4 breaches hide among them; the other 270 hide 2 breaches. Each detailed check takes about 6 minutes. With random spot checks, the scribe does not know which action will be examined (Townsend's 1979 idea of verifying only when results are poor, plus random spot checks).

4. Records he can understand: layering

The scribe carries out several hundred actions a year, each one recorded, but he looks only a few times a year for half an hour each time. Putting the full record in front of him is the same as giving him nothing. Work out how to arrange it so he can understand it. The top layer says in one sentence whether each account has moved closer to or further from its target over the period and whether any boundary has been crossed. The second layer lists a few key points: the decisions that were large, irreversible or set off a trigger. Only the bottom layer holds the complete entry-by-entry record, for him or whoever checks on his behalf to examine in depth. He can stop at any layer.

Ben-Shahar and Schneider (2014) examined the effects of mandated disclosure systematically and concluded that it usually fails: most people skip the disclosures and lack the ability to analyse them, while legislators keep adding disclosure requirements. The amount disclosed is not in itself the answer.

Today's counterpart. In 2019 the US Securities and Exchange Commission required broker-dealers and investment advisers to give retail clients a relationship summary of no more than two pages, or no more than four pages for firms offering both kinds of service.

A record in three layers

One year's records arranged in three layers. Select a layer to view it; any sentence in the first layer can be traced to its basis in the layer below.

Illustrative: the record entries are invented examples; "Full record" shows only the first few entries of 312 in total.

5. What was known at the time

Looking back after a storm, the scribe's purchase of ship shares three days before it seems foolish; yet three days before, nobody knew the storm was coming. Judge decisions by outcomes learned afterwards, and good decisions are blamed for bad luck while bad ones are praised for good luck. To judge fairly, the record cannot say only what was done; it must also say what was known at the time: which news had been seen, which clause of the agreement called for the action, and how large the risk was estimated to be then.

Fischhoff's (1975) experiments found that people who know an outcome overestimate how likely it was beforehand, and are unaware of this influence.

Today's counterpart. Trade records usually capture the time of the order, the price and the market conditions at that moment, so that later review can reconstruct the situation at the instant the order was placed.

Seen afterwards, and seen at the time

Three days before a storm, the scribe bought some ship shares as the agreement required. Switch between two ways of looking and judge whether the step was sound.

Illustrative: the record entries are invented examples.

6. Why this was done

An entry reading "Sold 2 ship shares" tells him nothing about whether it was right. Written as "Sold 2 ship shares: the retirement account's ship-share proportion had risen to 86% (agreed 80%, a deviation of more than five percentage points); brought back per clause 3 of the agreement", it lets him see at a glance whether the step kept to the agreement. When every step states which clause it relies on and what the figures were at the time, the record turns from a running log into evidence that can be checked item by item. A step for which no basis can be given is precisely the one he should question.

Today's counterpart. The EU's General Data Protection Regulation provides that, for decisions based solely on automated processing, the data subject is entitled to meaningful information about the logic involved, as well as the significance and the envisaged consequences of such processing.

A running log, or evidence

The same four entries, written two ways. Switch between them and see whether he can tell at a glance if each step kept to the agreement.

Illustrative: the record entries are invented examples.

7. A visible process: Civilization Simulator, version 22

The simulator unlocks "process records and verifiable evidence". Over ten years the scribe occasionally breaches the agreement: it crosses a boundary, takes risks outside the agreement, or makes trades that conflict with his interests. You can choose how records are kept: no records, a summary written by the scribe itself, or layered records kept by a third party and open to verification. Then choose how they are checked: never, every entry, or detailed checks when results are poor plus random spot checks. See, over the ten years, how many breaches are caught, how much of his time is spent, and whether he mistakes a good scribe for a bad one.

Civilization Simulator · Chapter 22: a visible process

Over ten years the scribe occasionally breaches the agreement; there are also bad runs of years in which results are poor through no fault of its own. Choose how records are kept and how they are checked, set how much time he can spend on checking each year, and see the breaches found, the time he spends, and how often he wrongly blames a scribe that kept the agreement.

Product: what to promiseProduct: mandate and authorityProduct: process records and verifiable evidenceProduct: limits, confirmation and compensation · Ch. 23
Record
Check

Illustrative model: the figure shows expected values over ten years. The scribe breaches the agreement 0.6 times a year on average, a third of these hidden among good results; there is a 30% chance each year of a bad run. Without records, a breach can be noticed only when results are poor (20%), and 30% of bad runs are mistaken for breaches. A summary written by the scribe leaves out breaches, so only 30% are found, and mistaken blame falls to 15%. Layered records kept by a third party state the information and basis at the time, so every breach that is checked is found and there is no mistaken blame. Checking every entry takes 30 hours a year and smart checking about 6 hours; when this exceeds his time, only a corresponding share can be checked. With random spot checks, the number of breaches halves.

8. What this step leaves open

Once the process can be seen and verified, several problems remain:

  1. After a breach is found. If the scribe is found to have crossed a boundary, who bears the loss, and how is it compensated?
  2. Errors within the boundaries. If the scribe keeps the agreement fully but misjudges, whose consequences are they?
  3. Irreversible actions. Some mistakes cannot be undone however clearly they are recorded. How can they be stopped before they happen?
  4. A new scribe. A scribe that has just opened for business has no record. On what basis can he begin to trust it?

What this step leaves open

Four problems that remain once the process can be seen and verified. Select one to see the assumption it hides.

A visible processVerifiable layered records

9. Implications for products

For a wealth-management product built around an agent, the third part that has to grow is "process records and verifiable evidence". Three implications follow. First, records should be kept by a party unconnected with the agent, be unalterable, and be open to the user by default; verifiable records make agents that keep their agreements willing to be seen, and turn "no record" into a signal. Second, records should be layered by the user's attention: one sentence on how far each goal is and whether any boundary was crossed, then the key points, and only then everything; each layer must be traceable to the one below. Third, every step should state what was known at the time, which clause of the agreement it relies on, and what the figures were then; verification takes place in three situations, poor results, a triggered condition and random selection, with the agent doing most of the checking for the user, who looks only at conclusions and exceptions.

Chapter summary: the user cannot see the process

Four conclusions, each tied to one mechanism in this chapter.

UnravellingVerifiable records make silence bad news

Records kept by a third party and unalterable make agents that keep their agreements willing to be seen.

Costly verificationDetailed checks on poor results or triggers, spot checks otherwise

The agent does most of the checking for the user, who looks only at conclusions and exceptions.

Rerun: attentionOne sentence, key points, full record

Each layer is traceable to the one below, so a summary cannot hide bad news.

At the time, and on what basisRecord what was known then and which clause applied

This guards against hindsight; a step with no stated basis is exactly the one to question.

References