Benchmarkzero shot time series forecasting

We benchmarked TimesFM zero-shot against Prophet. Here is where each one wins.

Pierre N. · Founder, VecTime Cloud21 August 2026 · 9 min read

On 300 series from the M4 competition dataset, TimesFM 2.5 forecasting zero-shot — no fitting, no tuning, no per-series model — had a lower mean and median error than both Prophet at its defaults and seasonal-naive, at every frequency we tested. It also lost to seasonal-naive on 20, 32 and 34 series out of 100. The largest gap between the two methods was not point accuracy at all: asked for an 80% interval, Prophet delivered 44.8% coverage on Monthly data, and TimesFM delivered 75.8%.Everything below is reproducible. The script, the summary and every per-series row are linked at the end, and the sections you would use to argue against us — the losses, the setup Prophet was denied, the metrics that disagree with each other — are in the middle rather than the footnotes.

A disclosure worth reading firstWe sell a hosted TimesFM API, so we are the last people whose benchmark you should take on trust. That is exactly why the per-series CSV is downloadable rather than summarised, why seasonal-naive — the baseline that is embarrassingly hard to beat — is in every table, and why the count of series where TimesFM lost is a section rather than a sentence. Disagree with it by rerunning it.

What we ran

M4(opens in new tab) is a public competition dataset with its own train/test split, its own horizons and its own seasonal periods. Those are not ours to choose: changing them would make these numbers incomparable to every published M4 result, which is most of what they are worth. We sampled 100 series each from Hourly (horizon 48, seasonal period 24), Weekly (13, 1) and Monthly (18, 12) with a seeded RNG, so the same command draws the same series on your machine and ours.

Method	How it was run
TimesFM 2.5 (200M)	Zero-shot and batched, with no fitting step of any kind. Served over POST /v1/forecast on a free-tier key, so history was capped at the free plan's 1,024-point context window.
Prophet 1.1.6	Fitted per series, default settings, interval_width=0.8 to match the q10–q90 band. No holidays, no regressors — see the section on that below, because it matters.
seasonal-naive	Repeat the last seasonal cycle. On Weekly, where M4 sets the seasonal period to 1, this degenerates to “next week equals this week” and is still hard to beat.

Scores are sMAPE and MASE as M4 defines them, plus one M4 does not report: coverage, the share of held-out points that landed inside the interval each method promised. Prophet's default interval is 80% and the q10–q90 band is also 80%, so the comparison is like for like. Everything ran on an M1 Pro, on CPU.

The results

Lower is better for sMAPE and MASE, and both columns read as mean / median. For coverage, 80% is the target rather than a maximum — above it is conservative, below it is overconfident.

Frequency	Method	sMAPE	MASE	Coverage
Hourly	TimesFM (zero-shot)	9.095 / 5.225	0.741 / 0.566	82.5%
Hourly	seasonal-naive	16.146 / 6.854	1.277 / 1.204	—
Hourly	Prophet	21.233 / 12.672	1.840 / 1.255	65.0%
Weekly	TimesFM (zero-shot)	6.297 / 3.779	1.905 / 1.152	86.8%
Weekly	seasonal-naive	9.544 / 5.256	2.569 / 1.830	—
Weekly	Prophet	21.019 / 11.253	7.012 / 2.538	58.5%
Monthly	TimesFM (zero-shot)	15.074 / 8.714	1.017 / 0.793	75.8%
Monthly	seasonal-naive	16.781 / 10.357	1.224 / 0.990	—
Monthly	Prophet	23.661 / 11.244	1.349 / 1.085	44.8%

TimesFM (zero-shot)seasonal-naiveProphet (defaults)

The same first column, drawn: median sMAPE per frequency, where a shorter bar is a better forecast. The gap that matters is not TimesFM against Prophet — it is how close plain seasonal-naive stays to both.sMAPE and MASE disagree often, and that is information rather than a bug. sMAPE is scale-free but penalises over- and under-forecasting asymmetrically; MASE compares against the seasonal-naive error on the training portion of the same series, so it answers “is this better than the trivial method” directly. Weekly is the clearest case: TimesFM has by far the best sMAPE at 6.297, and a MASE of 1.905 — better than both baselines, and still worse than the in-sample naive error on the average series.

Where TimesFM loses

A table of means is the least interesting way to read a benchmark, because a mean can win while losing on a third of the individual cases. Per series, by MASE — “naive” is seasonal-naive:

Frequency	Lost to naive	Lost to Prophet	Beat both
Hourly	20 / 100	8 / 100	78 / 100
Weekly	32 / 100	17 / 100	59 / 100
Monthly	34 / 100	34 / 100	48 / 100

beaten by seasonal-naivebeaten by Prophet

Out of 100 series per frequency. The averages above say TimesFM wins everywhere; this says that on a third of monthly series the thing you would have got for free beat it. Both are true, and only one of them is on most vendors' comparison pages.Monthly is the honest result here, and it is the frequency most business series actually live at. TimesFM is beaten by both baselines on 16 of 100 Monthly series, wins outright on 48, and its mean MASE advantage over plain seasonal-naive is 1.017 against 1.224. That is a real improvement and it is not a large one. If your data is monthly, treat this as a reason to run the comparison on your own series rather than a reason to skip it.

The bigger gap is the interval, not the point

Point error is what benchmarks report and it is rarely what breaks a plan. If you size inventory from a P90, or alert when a metric leaves its expected band, what you depend on is the interval meaning what it says. Both methods were asked for 80%.

TimesFM q10–q90Prophet 80% interval

Prophet was overconfident at every frequency, and on Monthly data fewer than half the actual values fell inside the range it promised. TimesFM was slightly wide on Hourly and Weekly, slightly optimistic on Monthly, and near nominal throughout. The same pattern holds series by series rather than only in the average, counting the series whose own interval covered at least 80% of the held-out points:

Frequency	TimesFM ≥ 80%	Prophet ≥ 80%
Hourly	62 / 100	25 / 100
Weekly	75 / 100	39 / 100
Monthly	53 / 100	21 / 100

Seasonal-naive has no interval at all, which is why its coverage cell is blank rather than zero. That is its own kind of answer: the cheapest method gives you a number and no way to size the risk around it.

Prophet's mean is not Prophet's typical

On Weekly data Prophet's mean MASE is 7.012 and its median is 2.538. The mean is dragged by a handful of catastrophic fits — the worst single series scored 158.6, against a worst case of 12.1 for TimesFM on the same set. Quoting only the mean would misrepresent Prophet as far worse than it usually is, which is why both columns are in the table above. The failure mode is real and worth knowing about, but it is a tail, not a typical case.

When Prophet is still the right call

This benchmark ran Prophet in the setup that suits it least, and that is a limitation of the benchmark rather than a verdict on the tool.

You know something about the future. Prophet is designed to be handed holidays, promotions and other known future events as regressors. It was given none here. On a series driven by events you can enumerate, a Prophet model with those events supplied should be expected to do better than it does in this table — and TimesFM, in this setup, cannot use them at all.You need the decomposition, not just the forecast. Prophet hands back separable trend, seasonality and holiday components. A foundation model hands back a distribution over what happens next and no account of why.Your series is well served by something simpler. Seasonal-naive beat TimesFM on a third of Monthly series. Whatever you pick, benchmark it against repeating the last cycle before you believe it — on strongly periodic data that is a genuinely hard baseline, and it costs nothing to run.

What would move these numbers

A longer context. The free tier caps history at 1,024 points. M4's Hourly series run to 960, so they fit, but a larger window would give the model more to work with on longer histories.A tuned Prophet. These are defaults: no tuned seasonality, no holidays, no regressors, no cross-validated changepoint prior.More series. 100 per frequency is enough to see the shape and not enough to split hairs. The script takes --series 500.

Run it yourself

The script downloads the M4 files itself and uv(opens in new tab) installs the dependencies from the pins in its header. Those pins are load-bearing: cmdstanpy 1.3 breaks Prophet 1.1.6's backend loader and pandas 3 breaks its internal reindexing, both per series and both caught, so without the pins Prophet quietly becomes a column of NaN — a benchmark in which the competitor never ran and which looks like a clean sweep.

bash
# The hosted path: this script, a free-tier key, nothing else.
# Create a key at https://vectime.cloud/app/keys — no card required.
curl -O https://vectime.cloud/benchmarks/run.py
VECTIME_API_KEY=vct_live_… uv run run.py \
  --freq hourly weekly monthly --series 100 --seed 7

The script also has an in-process mode that loads the weights on your own machine and needs no account at all; it imports our inference wrapper, so it runs from a checkout of the service rather than from this file alone. We cross-checked the two paths on M4 Hourly and they agree: identical summary table at the precision quoted here, the same 20-of-100 loss rate against seasonal-naive, and a largest per-series divergence of 1.7e-4 sMAPE.

run.py summary.csv per-series.csv (900 rows)

The per-series file is the one to check first. Every row is one series, one method, one score — which is what you need to see the distribution rather than trust a mean.

Run it on your own series.

If you want to try this without hosting a model, VecTime Cloud's free tier is 300 forecasts a month and needs no card.

Start free Read the API docs

Pierre N.Founder, VecTime CloudBuilds VecTime Cloud — the Go API and its metering, the Python inference service, and the benchmarks published here, including the runs that go against us.