Closing this following our policy that we don't keep old wishes open that nobody is actively planning to work on.

One good way of doing performance regression tests would be setting up tests
that try to reproduce the results from some of our old papers (e.g. the LM-cut
paper, the merge-and-shrink paper, the ECAI 2010 landmarks paper) and complain
if performance gets too much worse than what was reported there. To keep runtime
small, we could limit the experiments to those runs which were actually
successfully solved in those experiments.