Technical analysis, chart patterns and candlesticks from a quant perspective

In depth11 min read · 15 sources

Technical analysis, also known as charting, reads from price charts and trading volumes where a price will go next: trends, support levels, chart patterns such as the head and shoulders, Japanese candlesticks and indicators such as RSI or MACD. From a quant perspective, each of these rules is a hypothesis that has to be stated precisely and tested with costs and against lucky hits. Here you will see how to do that and what has come out of it so far.

Basic concepts: trend, support, resistance

An uptrend is a sequence of higher highs and higher lows, a downtrend the opposite. Moving averages smooth the price so that the trend becomes visible; the 200-day line is well known. Support is a price zone in which chartists expect buying, such as an earlier low; resistance is a zone in which they expect selling. If the price breaks through resistance, this is called a breakout. Rising volume is regarded as confirmation. These concepts are set out in this form in textbooks such as Murphy (1999).

This only becomes testable with a precise rule. Instead of “the price breaks out”, the rule then reads, for example: “The closing price is above the highest closing price of the last 50 trading days; the purchase is made at the next day's closing price.” A rule has to be worded like this, or similarly, before it can be tested. Brock et al. (1992), for instance, tested breakouts from trading ranges.

Classic chart patterns

• Head and shoulders: three highs, the middle one the highest. The line through the two lows in between is called the neckline; if the price falls below it, this is regarded as a trend reversal to the downside. The inverse pattern is regarded as a bottom.
• Double top and double bottom: two highs or lows at a similar level, shaped like an M or a W on the chart.
• Triangles: the upper and lower trendlines converge (ascending, descending or symmetrical). A breakout in one direction is expected.
• Flags and pennants: a short, shallow countermove after a steep rise or fall. A continuation of the previous trend is expected.

Lo et al. (2000) name the fundamental problem right at the start: whether a price chart shows a geometric shape is a matter of perception. Such shapes, the authors write, are

They therefore detect patterns automatically: a nonparametric kernel regression smooths the price, and the highs and lows of the smoothed line are combined into patterns according to fixed rules. After all, even an experienced chartist tends to see, in hindsight, the patterns that worked (hindsight bias). And a head and shoulders pattern is only complete with the right shoulder and the break of the neckline. If you draw the pattern in earlier, you are using knowledge from the future.

Candlesticks: doji, hammer, engulfing, morning star

A candlestick shows the open, high, low and close for one day. The body runs from the open to the close (light or green when the price rises, dark or red when it falls), and the wicks extend to the high and the low. According to tradition, the method comes from the Japanese rice trade. Some well-known patterns:

• Doji: open and close almost the same, so the body is just a line. Regarded as a sign of indecision.
• Hammer: small body at the top, long lower wick, usually at least twice as long as the body, after a decline. Regarded as a possible reversal to the upside.
• Engulfing: the body of the second candle engulfs that of the first and runs in the opposite direction (rising as bullish engulfing, falling as bearish engulfing).
• Morning star: three candles, namely a large dark one, a small one gapping below it, then a large light one that reaches deep into the body of the first. Counterpart: evening star.

Formalised, a doji might look like this: the absolute value of the close minus the open is at most one tenth of the range from high to low. Such thresholds are set, not derived. Every other threshold is one more variant tested.

Indicators: RSI, MACD, Bollinger Bands, momentum

• RSI: the relative strength index of Wilder (1978) relates the average gains and losses of the last 14 periods to each other. Above 70, a price is considered overbought; below 30, oversold.
• MACD: the MACD is the difference between two exponential moving averages, usually over 12 and 26 periods. Its own smoothing over 9 periods is called the signal line; crossovers are regarded as signals.
• Bollinger Bands: Bollinger Bands usually lie two standard deviations above and below a 20-period moving average. They widen when volatility rises.
• Momentum: today's price divided by the price n periods ago, minus 1. As time-series momentum, this is the best-documented form of the “trend” idea: Moskowitz et al. (2012) found it in all 58 futures examined on equity indices, currencies, commodities and bonds (trend following).

Many indicators measure the same thing in different ways: how far has the price risen recently, and how strongly does it fluctuate? So if you use RSI, MACD and momentum together, you have fewer independent signals than it seems.

How to test technical analysis quantitatively

1. Formalise: define and write down the rule, the parameters and the assets in advance, before the results are in.
2. Choose the timing honestly: a signal from today's closing price can be traded at the next price at the earliest.
3. Benchmark: buy and hold, compared at the same level of risk. Brock et al. (1992) compared their rules by bootstrap with random models: random walk, AR(1), GARCH-M and EGARCH.
4. Costs: bid-ask spread, fees, market impact and taxes. The higher the turnover, the more they matter.
5. Data snooping: every rule tested counts. The Reality Check of White (2000) checks whether the best rule is more than luck among many; the SPA test of Hansen (2005) has more power and is less sensitive to poor variants.
6. Out of sample: later years, other markets, other assets.

What the research shows

• Moving averages and breakouts: Brock et al. (1992) tested these two simple rules on the Dow Jones from 1897 to 1986 and found strong support: after buy signals, returns were higher and less volatile than after sell signals, and after sell signals they were even negative. This is inconsistent with the four random models.
• Correcting for data snooping: Sullivan et al. (1999) applied the Reality Check to a universe expanded to thousands of rules and 100 years of daily Dow Jones data. For the original period, the best rule survived the correction; in the subsequent years from 1987 to 1996, its advantage did not repeat itself.
• Costs: Bajgrowicz and Scaillet (2012) tested technical rules on the Dow Jones from 1897 to 2011 using the false discovery rate. Even low trading costs completely wiped out the excess return in the sample, and the rules that would turn out best in future could not be picked in advance.
• Chart patterns: for US stocks from 1962 to 1996, Lo et al. (2000) compared the returns after patterns such as head and shoulders or double bottoms with the ordinary distribution. Several patterns provided additional information and might have some practical value. This is a finding about distributions, not a trading strategy after costs.
• Candlesticks: for three-day patterns in S&P 500 stocks from 1992 to 1996, Caginalp and Laurent (1998) found statistically significant out-of-sample predictive power and a gain of almost 1% over a two-day holding period. Marshall et al. (2006), by contrast, used a bootstrap of opening, high, low and closing prices for stocks in the Dow Jones Industrial Average and found no added value.
• Overview: among 95 modern studies, Park and Irwin (2007) counted 56 with positive, 20 with negative and 19 with mixed results. Technical rules generated profits in many speculative markets, at least until the early 1990s; early studies found them in currency and futures markets, not in stock markets. Most tests, however, suffer from data snooping, the selection of rules after the fact and risks and costs that are hard to estimate.
• Equity premium: according to Neely et al. (2014), technical indicators predict the US equity premium in and out of sample as well as or better than macro variables. They tend to pick up the typical decline in the premium near business cycle peaks, while macro variables tend to pick up the rise near troughs; together they work best.

For advanced readers: why patterns might work and why they disappear

There are explanations from behavioural economics. In the model of Hong and Stein (1999), news spreads only slowly. Prices therefore initially underreact, and trend followers can profit from this until, in the long run, they push the price too far. That is consistent with momentum over months and reversal over years. Caginalp and Laurent (1998) interpret their results as an indication that traders are influenced by the price path itself.

Then there is the self-fulfilling prophecy: if many market participants watch the same line, such as a round price level or the 200-day line, orders cluster there, and the price really does react in the short term. This is an assessment, not an established measurement. The effect fades as soon as too many want to trade earlier. Chance is just as likely: if you look for patterns, you will find them in random series too, and the efficient market hypothesis in its weak form (Fama 1970) predicts exactly that.

With patterns, there is a further problem: they are rare. A complete head and shoulders pattern is a rare event for any individual stock, and the observations also overlap over the holding period. Few independent cases mean large standard errors. Even a genuine effect can then hardly be separated from chance. More on these pitfalls under backtest overfitting.

Limits and common mistakes

• Hindsight: on the finished chart, patterns are obvious; in real time, they are not. The right shoulder and the breakout are only established later.
• Confirmation instead of testing: if you collect examples in which a pattern worked, you overlook the false signals (confirmation bias).
• Unrealistic execution: trading at the closing price of the signal day, without spread and without market impact, is too optimistic.
• Many parameters: window lengths, thresholds, timeframes and assets multiply the number of tests.
• Survivors: if you test patterns on today's index members, you leave out bankruptcies and relegated stocks (survivorship bias).
• Taxes: frequent trading realises gains early and forgoes the tax deferral of buying and holding.

This article evaluates methods and is not investment advice; it recommends neither trading software nor signal services.

How Portfolio Lab calculates it

The Tactical page tests formalised trend rules on a monthly basis: the moving average for each asset (default 10 months, selectable from 2 to 24; invested above the average, otherwise in the money market or a fallback asset), dual momentum (default 12 months, with absolute momentum against the money market) and relative strength over averaged windows of 1, 3, 6, 9 and 12 months, plus target volatility, adaptive allocation and a valuation model. The signals use only prices up to the end of the respective month. Trades are executed at the closing price of the same month-end or, optionally, one month later; trading costs per order can be set from 0 to 2% and apply equally to the model and to buy and hold, but taxes do not. The page counts the allocation changes (methodology, momentum strategies).

Portfolio Lab does not calculate candlestick patterns, chart formations, RSI, MACD, Bollinger Bands or daily signals. The Tactical page does not offer a correction for data snooping either. If you try out many variants, you should apply the hurdles from the example above yourself.

This article explains a method; it is not investment or tax advice. Worked examples are simplified and rounded. All terms in the glossary.