Another 6 years have passed :-)

I stumbled over this project today, https://github.com/marketplace/actions/continuous-benchmark, which looks like a nice and easy way for tracking performance of GitHub projects.

OK, this one has been languishing again. :-)

After close to six years of "this would be nice to have", I'm marking mark it as
"resolved" in the sense of "it's not on the TODO list". Everybody is very
welcome to bring this idea to live at any time, but I think we shouldn't keep
(too many) things on the tracker that are not bugs and that we are not actively
pursuing.

OK, no more discussion here. Based on Erez's and Jendrik's thoughts, let's
retarget this one. I'm unassigning this from me and will remove the 1.0 keyword.
If anyone wants to take charge of this one, feel free to step up. :-)

I agree with Erez. Ideally, this could use the codespeed system 
(https://github.com/tobami/codespeed/) used by http://speed.pypy.org/ for 
graphing performance over time.

This would be nice to have, doesn't require too much familiarity with the 
internals of the planner, and is not urgent. I think this might make a good 
project for an undergraduate student.

No activity here for a long time, and I don't really know how we should proceed
here since this is a bit fuzzy. Two options, please cast your vote if interested:

A. We can close this without doing anything further.

B. We can convert this into something that allows us to track certain
performance indicators over time as a sanity test for wide-scope planner
changes. We already have a large experiment for such purposes, but I think we
would benefit from additionally having something smaller that just generates a
few numbers that can be tracked as a graph over time. For example, I would be
quite interested in seeing the number of problems solved by the LAMA-2011 or A*
+ LM-Cut configurations over time to see when we mess up something in terms of
performance. We could pick one or a few indicators of this kind for each of our
papers whose main experiments can be reproduced from Fast Downward's master
repository and/or for our past IPC configurations.

One thing that needs to be compared is blind search against the old seq-opt-base
code.

Also see issue38.

We need to make sure that the performance of the various search configurations
is at least comparable to what we reported in the various papers.