Signal Over Noise: Benchmarking Time Series Models with sktime

Time series models, especially forecasters and classifiers, play an important role in AI driven decision-making across a wide range of industries, including retail, finance, energy, and healthcare. In recent years, the number of available models for time series forecasting and classification has exploded, ranging from classical statistical techniques to deep learning and foundation models. But how do we evaluate which model is best suited for our specific use cases? Do the latest AI models really live up to their hype, or do they succumb to the old and mighty statistical models? Let’s find out together.

In this talk, I will demonstrate benchmarking time series estimators with sktime. sktime is among the leading time-series frameworks, openly governed by the users and developers and is open-sourced and free to use. It has the largest model zoo with interfaces to all major time series models including the ones from the recent research conference appearances.

Benchmarking with sktime can be performed across a combination of all the estimators and tasks. Estimators include all the models, and tasks can further be a combination of different cross-validation strategies, datasets, and scoring metrics.

Audience can expect to learn:

• Why benchmarking is essential

• The unique challenges of benchmarking and how sktime's consistent and composable API makes benchmarking easy, an example run

• Insights from recent benchmarks across different domains, and how they challenge common modeling assumptions