Buildbot is set up now => resolved.

I generated such a support dir, called "misc".

At the moment it contains the buildbot config, the uncrustify config and a
script I wrote a while ago to check our #include guard convention. BTW, I fixed
that a while ago across all source files, but it's now messed up again in quite
a few:

helmert@alfons:~/planning/trunk/downward/search$
../../misc/check-include-guard-convention.py *.h */*.h | grep ^False
False iterated_search.h #ifndef ITERATED_SEARCH_H_
False search_progress.h #ifndef SEARCHPROGRESS_H_
False learning/maximum_heuristic.h #ifndef MAXIMUM_HEURISTIC_H
False learning/PDB_state_space_sample.h #ifndef PDBSTATESPACESAMPLE_H_
False learning/probe_state_space_sample.h #ifndef PROBESTATESPACESAMPLE_H_
False learning/selective_max_heuristic.h #ifndef SELECTIVE_MAX_HEURISTIC_H_
False learning/state_space_sample.h #ifndef STATESPACESAMPLE_H_

Erez, can you fix that? Both the names of the guards should be different and the
#endif lines should be different. (See our coding conventions.)

I like the support dir idea.

I'd like to have it less visible too, but I'd prefer to have it in some
subdirectory instead of as a hidden file. Maybe a subdir scripts/buildbot/?
Other suggestions welcome. The cfg file for uncrustify should also be stored
away somewhere; maybe introduce a new "support" directory or similar for both?

Is it possible to make the buildbot configuration (in the repository) a hidden
file? And would anybody object?

OK, running the master is no problem of course.

The buildbot master does not perform any builds, that's what build slaves are 
for. Really.
The buildbot master just keeps track of what needs to be done, and has a very 
nice web interface. I can still run the build slave on my local machine.

Also, the trigger on update is easy to set up, there are instructions on how to 
do that (add something the the update hook of SVN)

Hmm... problem with my previous message? Anyway, here it is again:

I'd rather not have it run on my local desktop due to the load this produces.
(The wiki, repositories and issue tracker are not so demanding, but builds and
stuff like that are.) Do you have a machine at Technion that might be used? We
could also use the gkigrid, but I'm not sure how happy the rest of the group
would be about that.

Instead of polling, I think it's better to be triggered on update. That's not
hard to setup, although we might wait with that until the switch to Mercurial
which will probably happen not too far in the future.

I've got a builtbot (master & slave) running on my local machine.
It does the following:
A. Poll the repository every 10 minutes
B. Upon detecting a change:
  1. a 5 minute no-change timer for incremental build is triggered
  2. a 10 minute no-change timer for a clean build is triggered
C. Nightly test at 3:00 every night
D. Weekly test at 8:00 on Saturday

The tests use Jendrik's scripts. We should discuss exactly what to run, but it's 
easy enough to configure, and it also collects data (for now total time and 
expanded states, but this can also change).

TODO:
1. Run the buildbot master on alfons, so it can be publicly available (Malte - 
can you open non-standard ports?)
2. Decide on the exact configurations and problems that will run nightly/weekly
3. Find some nice way to display the collected results, and possibly have alerts 
on them

Malte - please install buildbot, and I will provide the necessary configuration
files

Here's what I think we should do - please let me know if you have any comments:
1. Quick (incremental) build - will run every time someone commits code, and
send an immediate e-mail if the build breaks.
2. Full build - will run every night. This will allow us to keep track of
warning counts, compilation time, etc.
3. Quick test - will run every night. A test suite which consists of 1 or 2
problems from each domain, which can be solved in up to 10 seconds, using
different configurations (which should cover different search algorithms). 
We will keep track of expanded states and time, and this will allow us to detect
significant changes in search behavior.
4. Performance test - will run every weekend. Like the quick test, but a few
more problems, which take up to about a minute each.

Basically we can define more test suites, but I think this should catch most
problems quickly enough.

We'll start with creating an automatic build that will report whenever anyone 
checks in something that breaks the build (using buildbot).
Then we'll see how we can perform automatic unit testing and automatic performance  
testing.

Sounds good. Personally, I'd separate this into two issues:

 1. Create an automated test suite. (Building the planner would be the
    most basic part of the test.)
 2. Have an automated way of running the test suite.

For the second part, maybe something like BuildBot
(http://en.wikipedia.org/wiki/BuildBot) could be used.

This issue was suggested by Michael, and has 2 items:

1. Create a nightly build, with appropriate e-mail to anyone who breaks the code
2. Create a nightly test suite, to be run after each nightly build, to make sure
we don't introduce any bugs into the code. This would require running many
planner configurations on many problems, so we should think about how to do this
more efficiently.