Regime methodology: how the hidden Markov model works

Advanced17 min read · updated 26 October 2026 · 13 sources

Portfolio Lab's hidden Markov model (HMM) splits the weekly or monthly returns of a price series into two to four regimes, each with its own mean and volatility, from “Calm” to “Turbulent”. The Regime page uses it to show, in hindsight, when each regime was likely; the tactical model Regime (HMM) uses the same calculation as a signal, without look-ahead. This methodology describes what the code computes: model, estimation, filtering and smoothing, metrics, model selection, walk-forward and limitations. The idea and the research are covered in the article Regime models.

The model: hidden regimes, normally distributed returns

The model follows Hamilton (1989) and the presentation in Rabiner (1989): a hidden regime jumps from period to period according to a Markov chain. Which regime comes next depends only on the current one. Within each regime, the return is normally distributed, with its own mean and its own volatility. Only the return is observed; the model infers the regime as a probability (hidden Markov model).

• Log returns in percent: 100 × ln(P_t / P_t−1). Log returns can be added up over time; a mean of 0.25% per week is 13% a year.
• Weekly or monthly: the weekly close is the last trading day up to Friday, the monthly close the last trading day of the month, each with its actual date. Over ten years, weekly returns give 520 observations, monthly returns only 120.
• Completed periods only: the current week or month only counts once its regular last trading day (Friday, or the last business day of the month) is in the price data or the period is over, for example after a Friday holiday. Otherwise half a week would be treated like a full one.
• One series: the computational core can also estimate several series with a full covariance matrix per regime. Both pages use a single series: the selected price series on the Regime page, the base portfolio or a ticker on the Tactical page.

Estimation with the Baum-Welch algorithm

The code estimates the parameters π, A, μ and σ by maximum likelihood using the Baum-Welch algorithm of Baum et al. (1970), a special case of the EM algorithm of Dempster et al. (1977). Each iteration has two steps:

1. E-step: using the current parameters, the forward-backward algorithm computes for each period the probability γ_t(k) of each regime and, for each pair of consecutive periods, the probability ξ_t(i, j) of a switch from i to j.
2. M-step: new parameters as weighted averages, namely the initial distribution π = γ_1, the transitions a_ij = Σ ξ_t(i, j) / Σ γ_t(i), and the mean and variance of each regime with the weights γ_t(k).

To keep the probabilities from drifting toward zero over thousands of periods, the algorithm works in scaled form, as in Rabiner (1989): the forward quantities are normalized to sum to 1 in every period, and the logarithms of the normalization factors add up to the log-likelihood. The code also divides the densities of each period by their maximum; this factor, too, enters log L.

• Stopping rule: when log L rises by less than 10⁻⁶ in an iteration, or after 300 iterations at the latest. If the estimate shown has not converged by then, the Regime page says so in a note.
• Five starting values, fixed seed: EM only finds a local maximum. The first start assigns the periods by rolling volatility (centered window of 9 periods, sample data only): with two regimes, the calmer half goes into regime 0 and the more volatile half into regime 1. The other four draw random boundaries between the 15th and 95th percentiles of this volatility and blend each assignment half and half with random weights from a Dirichlet distribution. The random number generator always starts with the seed 1989, so the same data always produce the same model.
• Starting parameters: one M-step from this assignment. The code counts the transitions from consecutive periods, adding 1 to each cell of the matrix; the initial distribution is uniform.
• All starts at once: the starts run side by side until each one stops. The one with the highest log-likelihood wins.
• Floors: 10⁻⁴ of the series' total variance is added to the variance of each regime. A regime's volatility is therefore at least 1% of the total volatility; without this floor, a regime could explain a single value with a volatility close to zero and drive the likelihood to infinity. The code raises entries of π and A to at least 10⁻¹².
• Minimum data: the estimation needs at least 2K + 2 returns, the Regime page 100, and the tactical model 260 weekly or 120 monthly returns for its first estimate.

Ordering and naming the regimes

For the likelihood, it makes no difference which regime carries which number (label switching). After each estimation, the code therefore sorts the regimes by variance in ascending order (regime 0 is always the calmest) and assigns names according to the number of regimes:

The names only describe the volatility ranking. “Turbulent” does not mean crisis or recession, and the mean of a regime can have either sign.

Sorting alone does not always solve label switching: if two regimes are close in volatility, a new estimate can swap their order even though the regimes have barely changed. The code therefore checks three things and displays them:

• Matching with the previous estimate (Tactical): at each annual re-estimation, the code looks for the assignment of new to old regimes with the smallest sum of Bhattacharyya distances, across all K! assignments. If it differs from the sorting, the Regime tab reports a role swap with its date. The names still follow volatility: “Turbulent” remains the most volatile regime, which is what the trading rule relies on.
• Gap between volatilities: each estimate comes with the smallest ratio of the volatilities of adjacent regimes. Below 1.2, the Regime page and the Tactical page point out that the assignment of names is uncertain.
• Starting values at the same optimum: the code counts how many starting values lead to the same solution, i.e. to the same regimes (Bhattacharyya distance per regime at most 0.02) and the same probabilities of staying (at most 5 percentage points apart); the initial distribution does not count. If at most half of them reach the best solution, the likelihood has several peaks, and the page says so.

On the computational side, the code runs the forward and backward recursions in blocks of 32 periods as matrix products, for all starting values at once. Mathematically, this is the ordinary forward-backward algorithm; the tests compare it with a simple loop over all periods. Because the block boundaries do not depend on the length of the series, the filter for the first t returns equals the filter of the whole series at that point, up to rounding errors (below 10⁻¹²). A later return does not change any earlier filtered probability.

Filtered, smoothed, Viterbi: what the page shows

A single estimate yields three views of the regimes. They differ in which returns they use:

The filtered probability is the Hamilton filter from the formula above: prediction with A, weighting with the densities, normalization. The smoothed probability γ_t comes from the forward-backward algorithm; Kim (1994) described it for Markov-switching models (filtered and smoothed probability). The Viterbi algorithm of Viterbi (1967) uses dynamic programming in log space to find the single sequence of regimes with the highest joint probability.

Smoothing contains look-ahead: the probability for week t also uses the returns of the weeks that follow. If an unremarkable week is followed by a slump, the smoothed curve already colors that week as more turbulent, even though nobody could have known it at the time. The chart below shows this. The Regime page adds a second kind of look-ahead: the parameters, too, come from the full history, so even the filtered curve there is a look back. The page says so in a note; only the tactical model computes without either kind of look-ahead (look-ahead bias).

In the price chart on a logarithmic axis, the page colors each period by the regime with the highest smoothed probability. “Current regime” is the regime with the highest filtered probability in the last completed period. “Since (Viterbi)” gives the closing date of the first period in the last contiguous segment of the Viterbi path, together with its length; if the path ends in a regime other than the current one, the name of that regime appears instead of the length.

Worked example: one step of the Hamilton filter

Two regimes with weekly returns; the figures are chosen freely. Calm (regime 0): mean 0.25%, volatility 1.5% per week, a₀₀ = 0.98. Turbulent (regime 1): mean −0.4%, volatility 4.0%, a₁₁ = 0.90. After the previous week, the filtered probability of “Turbulent” was 10%. Now a week brings a log return of −3%.

Metrics per regime

The “Regimes at a glance” table annualizes each regime from its estimated parameters, with n = 52 periods for weekly and n = 12 for monthly returns:

• Mean and vol p.a.: mean and volatility of the log returns in the regime, scaled up by n or √n. The geometric return is the performance that a full year in this regime would produce.
• Share of time: average of the smoothed probabilities over all periods of the series.
• Expected duration: a regime ends in each period with probability 1 − a_kk. The duration is therefore geometrically distributed with a mean of 1 / (1 − a_kk) periods. The “Expected duration” tile shows it for the current regime.
• Today: filtered probability of the last completed period.
• Stationary distribution: the long-run share of each regime implied by A alone (π A = π). The code computes it as well, but the table shows the share of time. The two need not agree: the stationary distribution comes from the model, the share of time from the actual series.
• Transition matrix: row = regime in this period, column = regime in the next, in percent.

Model selection: log L, AIC, BIC and Auto

For each series, the Regime page estimates four models with one to four regimes, those with several regimes each from five starting values. More regimes almost always raise the log-likelihood; information criteria subtract a penalty per parameter to compensate. The count includes K − 1 initial probabilities, K(K − 1) transition probabilities and, per regime, d means and d(d + 1)/2 variances and covariances. For a single series (d = 1), that makes 2, 7, 14 and 23 parameters for one to four regimes.

From eight returns upwards, ln n is greater than 2: the BIC therefore penalizes each parameter more heavily than the AIC, with 6.91 instead of 2 for 1,000 weekly returns. Schwarz (1978) derived it as a large-sample approximation to Bayesian model selection. With Auto, the page shows the model with the smallest BIC among two, three and four regimes; in a tie, the smaller number wins. If the model with just one regime has the smallest BIC, a note says that the BIC prefers a single regime. The page still shows the best model with at least two. The “Model selection” table marks the model shown and the one with the smallest BIC.

An example with 1,000 weekly returns (ln 1,000 = 6.908) and freely chosen values for log L:

The AIC would be smallest with three regimes, the BIC with two. The third regime improves log L by 10, i.e. −2 log L by 20; in return, it adds 7 parameters, which the AIC penalizes with 14 and the BIC with 7 × 6.908 = 48.4. Auto therefore shows two regimes. The usual likelihood ratio test does not help with this choice, because the parameters of an additional regime are not even identified under the null hypothesis (regime models).

Tactical: the regime model without look-ahead

The tactical model Regime (HMM) runs the same estimation walk-forward: at each month-end, it uses only returns up to that date and parameters from an estimate that extends at most to that date.

1. Signal: the return of the base portfolio (with the base weights, reset to the weights at each month-end, from daily prices in the base currency) or of a separate ticker that only supplies the signal and is not held. The page loads its prices in addition; if they start later than those of the portfolio, the period is shortened.
2. First estimate at the first month-end with at least 260 weekly returns (five years) or 120 monthly returns (ten years) since the start of the common prices in the selected period, with five starting values.
3. Re-estimation at every 12th month-end thereafter, i.e. annually, with all returns up to that point (expanding window). The starting values are the parameters of the last estimate (warm start) and a fresh start based on rolling volatility; the higher likelihood wins.
4. Only filtering in between: the probability at month-end t is the Hamilton filter over all returns up to t, with the parameters of the last estimate up to t. At month-end, the last completed week counts; the days between it and the month-end only enter the following month.
5. P(Turbulent) is the filtered probability of the regime with the highest volatility. With three regimes, “Normal” counts like “Calm”.
6. Rule: with Threshold (default), the entire base portfolio moves into the fallback asset when P(Turbulent) is above the threshold (default 50%, selectable from 30 to 90%); otherwise the base weights apply. With Share by probability, 1 − P(Turbulent) is held in the base portfolio and the rest in the fallback asset.
7. Execution, costs and comparison as for all tactical models: at the closing price of the same month-end or one month later, optionally with trading costs per order, compared with buying and holding the base weights from the first execution, i.e. only from the first estimate onwards (Tactical methodology).
8. Monthly prices only: if the median gap between two prices is more than 20 days, as with asset classes that only have monthly values, the model uses monthly instead of weekly returns and says so in a note.

The Regime tab in the results shows P(Turbulent) at each month-end (under the Threshold rule, with the threshold line), the regimes at the last month-end with mean, volatility, expected duration and probability from the estimate valid at that time, the transition matrix, and the number and dates of the estimates. Notes give the source of the signal and the first estimate with its number of returns.

Data and settings of the Regime page

• Prices from the app's price cache in trading currency or index points, without conversion (unlike the Tactical page, which calculates in the base currency).
• At least 100 returns; otherwise the page states the available number and the minimum.
• Monthly values only: if the median gap between two of the last 260 prices is more than 20 days, the page uses monthly returns and says so.
• Cache: the four estimates per series, frequency and start year, and the response per number of regimes, for at most 12 hours and only until midnight, because the next day brings new closing prices.

Limitations

• Normal distribution per regime: within a regime, the return has no fat tails, no skewness and no volatility dynamics of its own; heavy tails arise only from the mixture of regimes. Using ten subseries of around 17,000 daily S&P 500 returns, Rydén et al. (1998) show that an HMM built from normal distributions reproduces most stylized facts well. The estimated parameters, however, sometimes varied considerably from one subseries to the next.
• Fixed parameters: within one estimate, μ, σ and A apply to the whole series. Nystrup et al. (2017) let the parameters drift over time; this way, even a model with two regimes reproduces the long memory of squared daily returns, which until then had been the hardest feature for an HMM to capture. Portfolio Lab adapts the parameters only through the annual re-estimation on the Tactical page.
• Parameter uncertainty: the pages show point estimates without standard errors. With few switches, the transition probabilities in particular, and with them the expected durations, are uncertain.
• Number of regimes: the number of parameters grows quadratically with K; according to Nystrup, Madsen and Lindström (2015), this is why such models for financial returns rarely have more than two states. AIC and BIC can contradict each other, and the Tactical page only allows two or three regimes.
• Label switching: sorting by volatility determines which regime is called “Turbulent”. If two regimes are close in volatility, a re-estimation can swap their roles; the pages report this (matching, gap, starting values), but it cannot be fixed. Stephens (2000) shows for Bayesian mixture models that such artificial ordering constraints do not solve the problem in general. If you see frequent swaps or narrow gaps, you are better off choosing fewer regimes.
• Local maxima: five starting values reduce the risk of landing on a poor local maximum, but do not rule it out.
• Frequent trading with the proportional rule: 1 − P(Turbulent) changes every month, so the portfolio is reallocated every month, with costs and, in a real brokerage account, taxes. The threshold trades less often, but all or nothing. In Nystrup et al. (2018), a penalty on frequent trading made the approach more robust, and an optimization using forecasts from the HMM beat a fixed decision rule like this one.
• Statistical, not economic regimes: Calm and Turbulent are volatility ranks of a series, not phases of the business cycle. Rule-based macro regimes are described in the Macro methodology.
• Hindsight on the Regime page: smoothed probabilities, the Viterbi path and parameters from the full history change with every new period, even for weeks far in the past. The page is an analysis tool, not investment advice.

This page describes how Portfolio Lab calculates; past results are not a forecast. All terms in the glossary.