The third term of the return equation is information advantage: only by knowing more than others can one earn more than others. The ceiling on this term is set first by prices themselves. Prices are hard to predict not because markets are chaotic, but precisely because participants are all trying to predict them: once a change becomes foreseeable, trading pulls it forward into today's price, and what is left for tomorrow is only new information that cannot be foreseen today. Excess returns from analysing public information are steadily eroded by competition.
In 1959 Roberts of the University of Chicago ran a simple experiment: he generated "prices" week by week from random numbers and drew them as price charts. The resulting charts were almost indistinguishable from real stock price charts, and they even showed the various "patterns" that technical analysis talks about.
Which one is a real stock price?
Of the four price charts below, pick the one that looks as if it came from a real stock.
What the experiment shows is not that "stock prices are random numbers", but something else: from the look of a chart alone, a patterned series cannot be told apart from a purely random one. Formations that look like patterns appear in large numbers in random series as well.
In 1900 the French mathematician Bachelier proposed in his doctoral thesis, Théorie de la spéculation, that price movements could be regarded as random. In 1965 Samuelson explained why prices should behave this way. Suppose everyone knows for certain that a stock will rise 10% next week. No one is willing to sell today at the old price, while many people want to buy, so the price rises today until next week's rise no longer exists. Foreseeable changes are pulled forward to today by trading.
This leads to a seemingly paradoxical conclusion: the harder participants try to predict prices, and the more able they are, the harder prices become to predict. The randomness of prices is not evidence of market failure but the result of competition. The price change left for tomorrow can only come from new information that appears tomorrow, and new information is by definition unforeseeable today.
This inference can be tested: if the predictable part has already been traded away, today's price change should contain no information about tomorrow's. In 1953 the statistician Kendall analysed a British index of industrial share prices, together with the price series of cotton in New York and wheat in Chicago, and found that the correlation between price changes in adjacent weeks was close to zero. In 1965 Fama tested the daily returns of the 30 stocks in the Dow Jones Industrial Average and reached the same conclusion. In 1966 Fama and Blume tested a variety of filter rules of the "buy after a rise, sell after a fall" kind; after trading costs, none of them consistently beat buying and holding.
Does today’s move tell you about tomorrow’s?
Each dot is one day: the horizontal axis is that day’s change, the vertical axis the next day’s. If the cloud slopes up to the right, today could be used to predict tomorrow.
With evidence this consistent, why do so many people still believe that price charts contain patterns? In 1971 Tversky and Kahneman pointed out that people expect even very short random sequences to show the regularities of the whole population, the so-called "law of small numbers". Research on the perception of randomness has further found that people picture randomness as frequent alternation between heads and tails, with few long runs. Real random sequences are not like that: they contain runs far longer than intuition suggests. In 100 coin tosses, the longest run of the same side is usually six or seven; on seeing such a run, people tend to assume there is a cause behind it.
True randomness has longer runs than intuition expects
Toss a coin 100 times: green is heads (up), grey is tails (down). The longest run of the same side is outlined in orange.
This intuitive bias explains why chart patterns, short-term "winning and losing streaks" and backtests on small samples so easily lead people to believe they have found a pattern.
A strict random walk does not hold exactly. In 1988 Lo and MacKinlay found small positive correlations in the weekly returns of US stock portfolios; in 1993 Jegadeesh and Titman found that stocks that had done well over the past 3 to 12 months went on outperforming poor performers for the following months, the phenomenon known as "momentum". Academics have since found more than a hundred such "anomalies".
But this predictability often weakens quickly once it is discovered. In 2016 McLean and Pontiff examined 97 published return predictors: compared with the original papers' sample periods, returns were on average 26% lower outside the sample period and 58% lower after publication. Once predictability is made public, trading erodes it, which is exactly the mechanism described above at work. What remains is partly compensation for risk and partly so small that it is hard to profit from after trading costs.
100 → 74The fall out of sample: some “patterns” held only by chance in a particular sample, a bias produced by data mining.
74 → 42The further fall after publication: once investors know, they trade on it, and trading erodes the predictable part.
The difficulty of predicting prices determines what financial products should not promise.
Treat claims of “consistently predicting rises and falls” with scepticism first, including claims that come from AI.
Short-term chart patterns and streaks of rises or falls are mostly normal features of random sequences.
AI is very good at finding plausible-looking patterns in noise; backtests and patterns must be tested on data not used to build the model.