Product × Finance × AI: Financial Products in the Age of AIPart Two Changing one parameter中文
Chapter 15

Who guarantees an agent's promisesCompensation, reputation, records and authority

A scribe can buy and sell ship shares for a household, adjust its plans, and even draw grain out of the granary. It will make mistakes, and others may exploit it. Yet when it errs, punishing it achieves nothing: it has no property from which to pay compensation and no long-built name to lose, and it can start over under a new name. Of the three devices that grew out of lending, neither collateral nor reputation can attach to it, and a third party's judgment can take nothing from it. The only device left is to hold the people behind it responsible. So the arrangements must change. Whoever deploys a scribe bears liability for compensation and insures against it; every step the scribe takes leaves a record kept by someone else; the scribe receives only the least authority its task requires, and the assets stay with a custodian who has nothing to do with it.

1. A scribe has nothing to lose

A promise is credible because the party who breaks it loses something: future cooperation, the property pledged as collateral, or the grain a judgment orders it to pay. A scribe acting for a principal must also make promises: to buy and sell according to the agreed plan, not to exceed its authority, and not to hand the principal's grain to anyone else. But it has no property, and it does not care about the future.

Consider what punishment does. Someone who manages grain for others and must make good any loss in full will check his work carefully. But if all his property amounts to only a small part of the loss, a judgment ordering full compensation can take only what he has. For him, the worst outcome is losing that small amount of property, so he decides how much care to take according to that amount, not according to the loss the principal might suffer. The less property he has, the less he cares. Nor will he insure against other people's losses of his own accord, because he would pay the premium while the principal received the benefit. A scribe is the extreme case of this reasoning: its property is zero.

Today's counterpart. The product liability directive the European Union adopted in 2024 explicitly brings software, including artificial intelligence systems, within the definition of "product". When defective software causes harm, the injured party need only prove the defect and the causal link, not that the manufacturer was at fault.

Those who cannot pay will not be very careful

One mistake costs the principal 100 portions. The grain manager can spend more effort checking to reduce mistakes, but checking has a cost. Drag his property and see how much effort he spends checking and how much the principal expects to lose each year; then make whoever deploys him liable and insured.

Illustrative: the probability of a mistake is 30% × e^(−checking), and each unit of checking costs 4 portions. The grain manager chooses the level of checking that minimizes "cost of checking + probability of a mistake × the amount he actually has to pay", where the amount he actually has to pay is the smaller of the loss and his property (a simplification of Shavell, 1986). Vertical axis: the principal's expected loss per year, in portions.

2. Starting over under a new name

Reputation binds because a record of default follows the one who defaulted: fail to repay this time, and you cannot borrow next time. A scribe, however, can change its name at any moment. When bad reviews pile up under one name, it drops that name and takes on work under a new one. Every "newcomer" a principal meets may be an old hand that has just changed its name.

All a principal can do is treat every newcomer with extra caution: hand it only small matters at first, and entrust larger matters only after it has worked for a while. This shrinks the gain from changing names, but honest newcomers must also accept a period of worse treatment. The cheaper names are, the longer that period lasts. To make reputation work again, names must become expensive: reputation should attach not to the scribe but to the person who deploys it and who cannot change identity at will.

The effect of the price of identity on cooperation was later studied directly. Friedman and Resnick (2001) analysed online settings where anyone can register a new name at will. When names are cheap, cooperation can be sustained only by making newcomers accept worse treatment at first, and this itself carries a social cost. The remedies they discuss include charging entry fees and using identities that cannot be changed.

Today's counterpart. App and plug-in platforms usually require developers to register under their real names, and ratings and penalties are recorded against the developer's account, not only against a particular app.

When names are cheap, reputation cannot attach

Ten scribes take on work for twenty rounds in a row, and three of them are unreliable. Each row is a scribe and each cell is a round. Switch among three situations and see how often principals are cheated and how much honest newcomers lose.

Normal workCheated the principalSmall jobs only (newcomer under caution)ExcludedChanged name

Illustrative: an unreliable scribe cheats with a probability of one half in each round, and cheating is recorded; when names are cheap, it changes its name immediately after cheating. When newcomers are treated with caution, any new name takes only small jobs for its first four rounds; a small job pays a quarter of the normal rate, and the loss from cheating is also only a quarter. When identity cannot be changed, one instance of cheating means exclusion (following the idea in Friedman and Resnick, 2001). Two further honest scribes arrive as newcomers in rounds 6 and 12.

3. Holding the people behind it responsible

A scribe cannot afford to pay compensation, but whoever deploys it can: that person has property, knows what the scribe has done, and can decide whether to spend more effort checking it. If the deployer must pay for the scribe's mistakes, he will count the expected compensation as part of his own costs and keep adding checks until the cost of one more check equals the compensation it saves. He will also insure of his own accord, turning occasional large payouts into a fixed annual premium, and build the premium into the fees he charges. Liability for compensation thus falls on someone who can both afford to pay and is in a position to control the scribe.

Today's counterpart. The "manufacturers" covered by the EU product liability directive include developers of software and providers of artificial intelligence systems.

Whoever pays compensation does the checking

A scribe deployed by one operator works for forty households each year. Switch who bears the loss from mistakes, and see how much effort the operator is willing to spend checking the scribe, how many mistakes occur, and how much principals pay in extra fees and lose without compensation each year.

Illustrative: the scribe handles 50 matters a year for each household; each has a 2% × e^(−checking) probability of a mistake, and each mistake costs 1 portion on average. Each unit of checking costs the operator 10 portions, and the operator chooses the level of checking that minimizes "cost of checking + the amount it has to pay". When it insures, the premium equals the expected compensation and, together with the cost of checking, is built into the fees charged to households (following the idea in Sykes, 1984). Amounts are in portions.

4. Every step leaves a record

To hold the people behind a scribe responsible, one must first be able to tell whether a loss came from the scribe's error or from a bad run of years. When a person does the work, the principal sees only the result. When a scribe does it, every step can be written down: what it read, what it based its judgment on, and what instruction it gave and when. With such a record, checking changes from guesswork into comparison: another scribe can examine each transaction and determine which one broke the agreement.

The record itself must also be trustworthy. If the scribe keeps its own record, it can rewrite it. Like the certificates for ship shares, the record must be entrusted to someone with no connection to the scribe and must not be alterable once written. What the principal can see is then no longer just the result but the process, and the cost of constraining the agent falls sharply.

Today's counterpart. In 2012 the U.S. Securities and Exchange Commission adopted a rule requiring a consolidated audit trail, which records the full life of every order in U.S. markets, from origination, routing, modification and cancellation to execution, and identifies the brokers that handled it.

Who keeps the record

In one month a scribe did twelve things for a household, two of which broke the agreement. Switch among three situations and see what the principal can find out.

    Illustrative: the twelve items and the two breaches are a made-up example. "Kept by a third party" means that once written, the record is handed to a party with no connection to the scribe, and no party can alter it.

    5. Grant only the authority needed

    A record can settle responsibility after the fact, but it cannot undo a loss that has already occurred. The largest loss a scribe can cause with a single mistake depends on what it is permitted to do. If it can use all of a principal's grain, one mistake could ruin a household; if it can buy or sell only a small portion at a time, the worst case is the loss of a small portion.

    This yields a rule for assigning authority: give the scribe only the least authority its task requires. A scribe that keeps accounts for someone needs only to read the accounts, not to place orders. A scribe that makes regular purchases for someone needs only to buy within a limit; it does not need to sell, still less to transfer grain to anyone else. Irreversible operations, such as moving grain out of the granary or changing the authorization itself, must be confirmed by the principal in person. The assets remain with a custodian who has no connection to the scribe; the scribe can only issue instructions and cannot take anything away.

    Today's counterpart. The market access rule adopted by the U.S. Securities and Exchange Commission in 2010 requires brokers to check customer orders against preset credit and capital thresholds before the orders reach the market and to block erroneous orders, and these controls must be under the broker's direct and exclusive control.

    Grant only the authority needed

    First choose the scribe's task, then switch among three ways of granting authority, and see what it is allowed to do and the largest loss a single mistake could cause.

    Worst loss from one mistake

    Illustrative: a household has 100 portions of property in all; the limit for monthly purchases is 2 portions, and adjusting to plan can move at most 10 portions at a time. The worst loss is the largest amount the scribe can use in a single mistake; operations that need confirmation cannot be completed by the scribe alone.

    6. Guaranteed scribes: Civilization Simulator, version 15

    The simulator unlocks "Guaranteeing agents". Forty households hand their ship shares to scribes for management. The scribes are deployed by ten operators, some of whom are unreliable: their scribes make more mistakes, and they do not pay when something goes wrong. You can separately switch on or off operator liability with insurance, records kept by a third party, and authority limits with confirmation, and adjust the share of unreliable operators. Then watch how many households are willing to delegate over twenty years, and how much of the loss caused by scribes' mistakes goes uncompensated.

    Civilization Simulator · Chapter 15: guaranteed scribes

    Forty households hand their ship shares to scribes for management, and the scribes are deployed by ten operators. Switch three rules on or off separately and adjust the share of unreliable operators, then watch how many households are willing to delegate over twenty years and how much of the loss from mistakes goes uncompensated.

    Regulation & custodyParameter: price of cognitive labourPersonal agentGuaranteeing agentsAnalysts for all · Ch. 16
    Households willing to delegate

    Illustrative model: a scribe does 20 things a year for each household. Each task done by a reliable operator's scribe has a 2.5% probability of a mistake, and by an unreliable operator's scribe 5%; when operators are liable they check more closely, and the probability of a mistake is halved. Each mistake costs 1.5% of the household's property, or at most 0.2% when there are authority limits. Without checkable records, principals find only 30% of mistakes; with records, they find 95%. Of the mistakes found, all are compensated when operators are liable and insured; otherwise reliable operators, to protect their names, compensate only part (20% without records, half with records), and unreliable operators compensate nothing. At first 80% of households are willing to delegate; whenever there are uncompensated losses, that share falls, and it recovers slowly in calm years.

    7. What this step leaves open

    Guarantees and records make scribes trustworthy, but several questions remain unresolved:

    1. The cost of the guarantee. Liability, insurance and records all cost money, and these costs are built into fees. Will the threshold for delegation rise again as a result?
    2. The keeper of the record. The record must be kept by someone with no connection to the scribe. Who constrains that keeper?
    3. A scribe for everyone. If every household has a scribe read the reports and pick ship shares for it, how will information advantage change?
    4. Scribes that resemble each other. Many scribes come from the same few operators. Will they make the same decision at the same moment?

    What this step leaves open

    Four questions that remain unresolved once guarantees and records have made scribes trustworthy. Click a question to see the assumption it hides.

    Guaranteed scribesLiability, records and authority

    8. Implications for products

    An agent product that can place orders for users must first answer the same question as a lending product: why should anyone believe it will keep its promises? Three implications follow. First, the agent itself cannot provide a guarantee; the guarantee comes from the institution that operates it. That institution's capital, liability and insurance are what users can rely on, and the product should state this clearly. Second, the process should leave a record that is kept by a third party and cannot be altered, so that users and regulators can check each transaction; a process that can be checked is the precondition for assigning responsibility. Third, authority should be graded by the size of the matter and whether it can be undone, and the assets should stay with an independent custodian; the agent can only issue instructions, and irreversible operations are confirmed by the user in person.

    Chapter summary: who guarantees an agent's promises

    Four conclusions, each corresponding to one mechanism in the chapter.

    Judgment-proofThose who cannot pay will not be very careful

    A scribe has no property, so punishing it is pointless, and it will not insure of its own accord.

    Reputation and vicarious liabilityGuaranteed by the people behind it

    Names can be changed at will, so reputation must attach to the operator; compensation is borne by the operator, who can afford to pay and is in a position to control the scribe.

    RecordsKept by a third party, unalterable

    With a record of every transaction, checking changes from guesswork into comparison, and responsibility can be assigned.

    Least privilegeGrant only the authority needed

    Authority is graded by size and by whether an action can be undone, and the assets stay with an independent custodian.

    References