Benchmarking can be hard, here's why we're still creating a new one for OCaml

Software and hardware are continuously evolving. New innovations & improvements help bring us better performance gains and sometimes things work out like magic. This is also true for programming languages like Java and OCaml. And yet, while developing our research ideas and trying to argue about performance, its hard to comprehend what we’re seeing: Is my benchmark now faster because of what we did, or was it another previous unrelated change showing up now? Is it better caching or was it the garbage collector being triggered? How can we be sure. There is, thus, a need to get reliable feedback while building up our systems and often benchmark suites can be key in guiding us.

In this talk, we’ll try to visualize how benchmarking can be hard, what to look out for and with helpful scenarios all along the way. We will briefly look into how this may show up in real life in the context of improving OCaml: an industrial strength functional programming language. Finally, we describe some of the ideas, old & new, we’re thinking about to help us grow, quantify, and maintain(!) a new benchmark suite to aid us in bringing more cutting edge research to our users without sacrificing on legacy code nor slowing down production workloads.

• Why do we care about benchmarks while developing and why these can be hard

• Ways to build reliable feedback from benchmarks and maintaining them as systems grow

• A quick intro to OCaml that hopefully piques your interest to learn about it more :)