The efficient market hypothesis: why it is so hard to beat the market

In depth8 min read · 12 sources

The efficient market hypothesis states that securities prices reflect available information quickly and largely in full. If that is true, the market cannot be beaten systematically using publicly known information, at least not after costs. The hypothesis is one of the most thoroughly researched ideas in finance, and the answer from the research is more nuanced than “yes” or “no”.

What the hypothesis actually claims

Fama (1970) organised the idea in an influential review article. A market is called efficient if prices “fully reflect” the available information. Depending on what counts as available, there are three forms:

It is important to note what the hypothesis does not say: it does not claim that prices are always “right”, that there are no bubbles or that nobody ever beats the market. It only says that deviations cannot be exploited systematically and predictably. According to the hypothesis, anyone who beats the market was either lucky or bore a risk that is rewarded.

A well-known paradox goes back to Sanford Grossman and Joseph Stiglitz: if markets were perfectly efficient, it would not pay to gather and analyse information. Then nobody would do it, and prices could no longer reflect the information. Markets therefore have to be just inefficient enough for analysts' work to pay off, but no more.

The arithmetic of active management

The strongest argument for index funds does not need the efficient market hypothesis at all. Sharpe (1991) showed with simple arithmetic: all investors together hold the market. Passive investors earn the market return before costs, so active investors as a group must also earn the market return before costs. After costs, active investors fall short on average, because they pay more.

French (2008) estimated the cost of active investing in the US stock market: on average over the years 1980 to 2006, investors spent 0.67% of the total market value every year searching for superior returns. A typical investor could have increased their return by 67 basis points a year by investing passively in the market.

The case for efficient markets

• Funds rarely beat the market over the long run. Fama and French (2010) show that the aggregate portfolio of active US equity funds is close to the market and that the high costs are borne in full by investors. Simulations suggest that only a few funds earn enough to cover their costs.
• Success is hardly persistent. Carhart (1997) finds that the persistence of fund performance can largely be explained by costs and the momentum effect.
• Prices react quickly. New public information is usually priced in within a short time; if you only read about it in the newspaper, you are too late.
• Forecasts fail out of sample. Welch and Goyal (2008) tested the well-known variables for predicting equity returns: over 30 years, most of them predicted poorly both in and out of sample and would not have helped an investor with market timing.

The case against: anomalies and overreaction

Shiller (1980) compared share prices with the present value of the dividends actually paid later. This “ex post rational” price follows a very smooth path, while actual prices fluctuate far more. A simple model of efficient markets only fits this if you assume that expectations about dividends fluctuate much more than dividends ever did.

Then there are anomalies: patterns that should not exist according to simple models. The best known is momentum: Jegadeesh and Titman (1993) showed that stocks that had done well over the past 3 to 12 months continued to outperform the losers in the following months; part of the lead reverses in the subsequent years. Other examples are the value effect, the size effect and the low-volatility anomaly.

Whether an anomaly reveals inefficiency or a risk premium often cannot be decided: every test of market efficiency is at the same time a test of a model of what return is appropriate for a given risk. Fama called this the joint hypothesis problem.

Behavioural finance: why people deviate from the rational model

The efficient market hypothesis does not require every investor to be rational, only that errors cancel out or are exploited by arbitrageurs. Behavioural finance, however, documents systematic errors that do not cancel out. With prospect theory, Kahneman and Tversky (1979) described how people weight losses more heavily than gains of the same size and perceive probabilities in a distorted way. Using tens of thousands of brokerage accounts, Barber and Odean (2000) showed that private investors trade too much and lose return as a result. They explain this pattern with overconfidence.

Such errors do not automatically create profit opportunities for others: if arbitrage is risky or expensive, mispricing can persist for a long time. For the individual investor, the most important consequence is therefore not to exploit other people's mistakes but to avoid your own, for example with fixed rules for your savings plan and rebalancing.

Do anomalies disappear once they are known?

McLean and Pontiff (2016) carried out a direct test. They examined 97 variables that, according to published research, predict stock returns. Outside the original sample, the strategies' returns were 26% lower, and after publication 58% lower. The first figure points to chance findings from data mining; the difference of 32% suggests that investors trade on the published patterns and thereby weaken them.

This fits a market that is not perfect but adaptive: inefficiencies exist, are discovered and then shrink. For private investors who only learn about anomalies from books, there is usually little left, all the more so after costs.

For more on how many discoveries are genuine effects, see the article backtest overfitting.

For advanced readers: luck or skill?

How often does a fund manager beat the market by pure chance? A simple thought experiment: 1,000 managers toss a coin every year; heads means “beats the market”.

The statistics behind this is the binomial distribution. Research takes the same approach, only with real returns: Fama and French (2010) compare the distribution of the estimated alphas of all funds with distributions generated by bootstrap under the assumption that no fund has genuine skill. Only if the real results in the tails differ clearly from the simulated ones does this point to skill. Before costs, they find evidence of genuine outperformance and underperformance in the extreme tails; after costs, hardly any.

What this means for investors

• Costs are certain, alpha is not: low costs are the only return you know in advance.
• Diversify broadly: if you cannot beat the market, you should at least hold all of it (see world portfolio).
• Choose factors deliberately: value, size or momentum can be captured in a targeted and inexpensive way. Whether you see the premium as compensation for risk or as mispricing, it can fail to materialise for a long time.
• Test your own ideas honestly: a strategy that only works in a backtest is usually based on overfitting. Out-of-sample testing and realistic costs are a must.

Malkiel (2003) sums up the debate as follows: markets are more efficient and less predictable than many recent academic papers suggest.

How Portfolio Lab calculates it

Portfolio Lab makes no claim about whether markets are efficient, but it gives you tools to test strategies yourself: Tactical allocation tests moving averages, momentum rotation, dual momentum and other rules historically; Optimisation can run on a rolling, out-of-sample basis; and Factors shows whether a fund's alpha remains once factors are taken into account. The tactical models calculate their signals using only prices up to the respective month-end and implement them at that month's closing price: with no look-ahead, but also with no execution delay. Details are in the tactical allocation methodology.

This article explains a method; it is not investment or tax advice. Worked examples are simplified and rounded. All terms in the glossary.