Interrupted Time Series (ITS)

What are Interrupted Time Series (ITS)?

An interrupted time series follows one outcome, measured at regular intervals such as daily orders, for a stretch before an intervention and a stretch after it. Wikipedia describes the method as tracking a long period before and after a point of intervention, with effects judged by changes in the level and slope of the series. A tutorial in the International Journal of Epidemiology calls the pre-existing trend, continued unchanged, the counterfactual: the comparison against which any change after the intervention is read.

The usual analysis is segmented regression: a line is fitted to the series before the intervention and is allowed to jump, change slope, or both afterwards. The same tutorial advises deciding the expected shape of the effect before looking at the outcome data, because choosing it afterwards raises the chance of a false finding.

The design rests on an assumption stated in a BMJ tutorial: without the intervention, the pre-intervention trend would have continued unchanged and no external factors systematically affected the series. Other events at the same time, seasonality and changes in how the data are recorded can each be mistaken for an effect. A control series that the intervention did not touch can address part of this. There is no fixed minimum number of data points; power depends on variability, effect size and seasonality.

The name is used loosely. Huntington-Klein notes that it can refer to any event study that accounts for a before-event trend, but in practice usually means either an event study built on time series econometrics or a regression that fits one line before the event and another after it.

For illustration, suppose daily orders were 100 on the day before a change to the free-shipping threshold and had been rising by 1 order a week. Four weeks later the pre-change trend projects 100 + 4 = 104 orders. If the store records 120, the estimated effect that day is 120 - 104 = 16 orders, where a plain comparison with the old level would show 120 - 100 = 20. If a promotion also started that week, the 16 cannot be split between the shipping change and the promotion.

Why Interrupted Time Series (ITS) matter for ecommerce

Some marketing changes reach the whole store at once: a new price, a shipping threshold, a checkout change, an always-on channel launched without a holdout. An epidemiology tutorial notes that interventions already rolled out to a whole population have no control group, which is the situation here, and a BMJ tutorial adds that a plain pre-post comparison fails to account for trends under way before the intervention. ITS adds the earlier trend as the baseline. It cannot separate the change from other events in the same weeks, such as a promotion, an email send or a stock-out, and the epidemiology tutorial names missing control for such events among the main threats to validity. A control series, as in Google's CausalImpact, addresses part of this.

How to use Interrupted Time Series (ITS)

1. Fix the intervention date and write down the effect you expect before opening the results: a one-time jump in level, a change in slope, or an effect that starts after a lag. Choosing the shape after seeing the data raises the chance of a false finding.
2. Export the outcome from Shopify reports to a spreadsheet as an evenly spaced series, such as daily or weekly orders or net sales, covering a stretch before the date and a stretch after it. There is no fixed minimum number of points, so treat short series and small expected effects with extra caution. Check that tracking or reporting settings did not change in the same week.
3. Plot the series and mark the date. Look for trend, weekly and yearly seasonality, outliers and one-off events such as a sale or a stock-out.
4. Fit a segmented regression in a spreadsheet or statistics package: the outcome on time, an after-the-change indicator and time since the change. Add terms for seasonality and for the dates of known promotions.
5. Check the errors. Time series are often autocorrelated, which can make spurious effects look significant, so inspect the residuals and use standard errors that allow for autocorrelation. Then re-run with a fake change date earlier in the series and with a different seasonality adjustment.
6. Read the result. Pass: the level or slope change holds up across specifications, no fake date produces a similar effect, the calendar shows no other change in that window, and a control series the change could not affect stays flat. Fail: the estimate moves with the model, a fake date looks as large, or something else changed the same week. Then report a possible effect, not a measured one.

Formula

Y = b0 + b1 T + b2 After + b3 (T - T0) x After + e

Where:
- Y: the outcome in period T, such as orders per day
- T: time, counted in days or weeks from the start of the series
- T0: the period in which the intervention happened
- After: zero before the intervention, one from then on
- b0: the starting level
- b1: the trend before the intervention
- b2: the jump in level at the intervention
- b3: the change in slope after the intervention
- e: the error term

If the pre-intervention trend would have continued unchanged, b2 and b3 measure the departure from it.

Common mistakes

1. Ignoring what else happened in the same weeks. Promotions, email sends, price moves, stock-outs and tracking changes get credited to the intervention. The epidemiology tutorial names missing control for such concurrent events among the main threats to validity.
2. Choosing the effect shape after seeing the data. The same tutorial discourages it because it raises the likelihood of detecting an effect from random fluctuation.
3. Leaving seasonality in the series. If peak weeks are unevenly spread before and after the change, the result can be biased, especially in a short series.
4. Trusting ordinary significance tests on autocorrelated data. Huntington-Klein notes that such data make a statistically significant effect very likely even when the event had none, so a randomly chosen fake event date can come out significant far more often than it should. The epidemiology tutorial adds that seasonality often explains much of the autocorrelation, but says it should always be assessed.
5. Stretching the after-period too long. Predictions from the earlier trend get worse the farther they run from the event and other influences creep in, so Huntington-Klein advises a fairly short post-event window unless nothing else is going on.

Frequently asked questions

• What is interrupted time series analysis in simple terms?

  It follows one metric, such as daily orders, before and after a dated change and asks whether the level or the trend shifted by more than the earlier trend predicts. The earlier trend, projected forward, is the stand-in for what would have happened without the change.
• How many data points do I need for an interrupted time series?

  There is no fixed minimum. A tutorial in the International Journal of Epidemiology says power depends on variability, effect size, seasonality and how the points are spread before and after, and that few time points or small expected effects call for caution.
• How is interrupted time series different from an A/B test?

  An A/B test randomizes who is exposed, so a comparison group is built in. ITS has no comparison group: it compares the series with its own earlier trend, so anything else that changed at the same time is mixed into the estimate.
• Is interrupted time series the same as an event study or CausalImpact?

  They are related, not identical. Huntington-Klein notes the term is used loosely, usually meaning an event study built on time series econometrics or a segmented regression. CausalImpact builds the counterfactual from control time series that the intervention did not affect.

Further reading

• Interrupted time series regression for the evaluation of public health interventions: a tutorial · International Journal of Epidemiology
• Regression based quasi-experimental approach when randomisation is not an option: interrupted time series analysis · BMJ
• Event Studies · Huntington-Klein, The Effect
• CausalImpact: an R package for causal inference using Bayesian structural time-series models · Google
• Inferring causal impact using Bayesian structural time-series models · Brodersen et al., arXiv

Related terms

• ExperimentsExperiments are scientific procedures that test hypotheses or demonstrate facts. In marketing, experiments like A/B…
• Immortal Time BiasImmortal Time Bias occurs in observational studies when the follow-up period includes time where the outcome cannot…
• Interaction EffectAn Interaction Effect occurs when one variable's effect on an outcome depends on another variable's level.
• InterventionAn Intervention is an action taken to produce a change in an outcome.
• Quasi-ExperimentA quasi-experiment estimates the causal impact of an intervention without random assignment. It applies when random…
• Time Series AnalysisTime Series Analysis analyzes data points collected over consistent intervals of time. It is used for forecasting…

Keep reading

Go deeper: Causal attribution, explained.

• Best Causal Inference Tools for Marketing Attribution
• How Causal Inference Reveals Which Channels Actually Drive Sales
• Causal Inference for Marketing Teams: A Non-Technical Guide
• What Is Causal Inference and Why Every Marketer Needs It
• Causal Inference: The Future of Marketing Attribution
• How Causal inference modeling Solves cookie deprecation impact for Beauty brand founders refining Meta Ads spend

All glossary terms · More in Causal Inference

Have an audience of ecommerce operators? Earn 20% recurring commission by recommending the €99 read.