All tests pass now. For technical reasons Erez' buildslave is commented out in 
the buildmaster config currently. We can discuss reconnecting your buildslave via 
mail.

The buildbot is up and running again, but we are still testing things.

Jendrik, I think the next thing to test is that the nightly/weekly tests with
lab work as expected. On the admin side, I don't think anything more needs to be
done at the moment, so you can test things via the buildbot web interface and by
making commits to the repository. Feel free to commit directly to master-default
when this is necessary to test something out, but if it's a change that breaks
the planner in some way, it should be backed out again soonish.

On closer inspection, it looks like we lost the buildbot connector when we moved
the Fast Downward repository to Basel. It would be needless effort to fix this
before moving the buildbot to Basel; let's do both at the same time. Will look
into it.

@Erez: nothing to do on your side at the moment.

> Also, of course there's no way to test the buildbot while the repository is
> frozen...

Ah, not true, the nightly and weekly tests should run regardless, I think.

Will do, but we need a build slave to test this. Erez's notebook has been idle
for the last year, although it seems to be connected and has been talking to the
buildbot. Any idea what is going on there and what we can do to change this?

I'll restart the buildmaster to see if that makes a difference, but I don't
really think it should, since it has been restarted a few times since then.
Erez, is there a way you can force some kind of reinitialization?

Also, of course there's no way to test the buildbot while the repository is
frozen...

I have translated the configs and suites and selected the attributes that I 
think we should check. I also set the baseline and added some documentation on 
how to run the experiment.

We still have to update master.cfg and probably README and update-buildmaster-
config. Can you do that, Malte?

configs and suites: I suggest we start with the ones we used before. I can
"translate" the configs if you give me the old parameter strings.

attributes: how about all attributes?

baseline: let's use the current version of default

Please document in a text file in the buildbot directory how to change all of
the above, in particular how to define a new baseline. More generally, the
textfile should contain enough information for us to maintain the buildbot in
the future. I can help with adding stuff to it.

I have adapted the experiment to lab and uploaded the branch to my bitbucket 
repo. Malte, you should have received an invitation.

What's left to do (once the updated exp is merged) is specifying the baseline, 
configs, suites and attributes in buildbot-exp.py and setting up the 
infrastructure. Can you do that, Malte?

Indeed the web interface says that the buildbot is connected. However, it didn't
do anything since August 18 last year. Maybe a problem on the server side? I
guess it's not worth fixing now if we're going to migrate soon anyway.

Regarding buildbot upgrades: I would suggest upgrading to whatever version is
shipped with the machine we're migrating to. Are you aware of any significant
changes we need to know about as users?

My buildslave seems to think everything is fine, and it's connected to 
buildbot.fast-downward.org. 
Anyway, I would be happy to help any way I can.
We might also consider moving to a newer version of buildbot - we were a 
generation behind when we started this, a long time ago.

We have revisited the buildbot topic recently. Jendrik and I will work towards
migrating everything offer to lab and to our server infrastructure in Basel.
(Erez, it seems that your build-slave is no longer set up. I assume this is
intentional?)

This issue looks relevant, so I'm using this comment to bring it back to
everybody's attention.

Pushed to downward-fixes  branch  issue133lab  (I didn't want to mess around 
with reopening old branched, it looked like it was going to cause some 
problems).

I just rediscovered the   misc/update-buildmaster-config  script to update the 
buildbot master configuration.

Ok, I'm convinced ;) I added the tag v1.0 to the current tip.

I very strongly recommend against using whatever is the current revision in our
test code. Working with a moving target is really difficult if you're not
intimately involved in the development of the tool -- for instance, I know
several people who gave up using the new-scripts (including me at some point)
because the behaviour changed often and without notice. It is a really a
tremendous pain; I cannot emphasize this enough.

Of course I could do that, but I think lab bugfixes can reach the tests 
more easily if we don't use a tagged revision. Of course that is only 
true for complete experiments, i.e. the ones that use the same lab 
revision for preprocessing, search and reports. Once we have a system in 
place that caches results, we should use tags.

The script looks good. You could inherit from RelativeReport and just 
overwrite the _get_table method, but that's only cosmetic.

The script looks good, I'll have a closer look later. You can pull the 
lab package from bitbucket/jendrikseipp/lab.

In the past we've been burned by incompatible changes in the scripts more than
once. Hence, I suggest that Jendrik tags a version of lab -- version 1.0 or
whatever -- and we use an explicit tag in our test code.

(That tag could be set in a configuration file inside our repository so that we
can change it without restarting the buildbots, but that's advanced stuff and
less critical.)

I've got a performance comparison script using the new lab package ready.
It was a lot easier to write than the old comparison script, which means Jendrik 
did a great job on the lab package.

I want to incorporate this into the buildbot (for nightly/weekly tests), but we 
first need to decide whether to simply "install" the current version of lab 
somewhere on the buildslave, or to update the lab package every time we run 
this. If we want to update it, it would be good to create a publicly accessible 
repository (like http://hg.fast-downward.org) to pull from.

Any thoughts?

Nice, if you have any questions while porting, don't hesitate to ask. 
Make sure you look at the "examples" directory to see how an experiment 
is written, then just call your script with "./your-script.py all" and 
all steps will be added to the grid with each step waiting for the 
previous one to finish first.

I can do it.

On Tue, May 1, 2012 at 10:02 PM, Malte Helmert <
downward.issues@googlemail.com> wrote:

>
> Malte Helmert <malte.helmert@unibas.ch> added the comment:
>
> > I added a "continuous" mode for the lab/downward packages. We can now run
> > experiments without user interaction on the grid.
>
> Excellent!
>
> > It would probably make sense to port the daily/weekly tests to the new
> lab
> > package.
>
> What would need to be done for that, and who can do it?
>
> _______________________________________________________
> Fast Downward issue tracker <downward.issues@gmail.com>
> <http://issues.fast-downward.org/issue133>
> _______________________________________________________
>

-- 

--------------------------------------------------------------
"Adventure is just bad planning."
    Roald Amundsen
    Norwegian Arctic & Antarctic explorer
    (1872 - 1928)
--------------------------------------------------------------

> I added a "continuous" mode for the lab/downward packages. We can now run
> experiments without user interaction on the grid.

Excellent!

> It would probably make sense to port the daily/weekly tests to the new lab 
> package.

What would need to be done for that, and who can do it?

I added a "continuous" mode for the lab/downward packages. We can now run 
experiments without user interaction on the grid.

It would probably make sense to port the daily/weekly tests to the new lab 
package.

I changed the call for fetching preprocessing results to ./resultfetcher.py in 
./run-fd-test.

I didn't use
"downward-resultfetcher.py --dest dir-of-the-props-file" to combine the results, 
because it says that the directory is an evaluation directory, and asks for a 
confirmation. Even if I answer yes, something still doesn't work right (probably 
because I change the names of the configs in the properties file).

Thanks for reminding me.
I'm working on it now, should be done soon.

What's the status on this?

line 22: You should use resultfetcher.py to get the preprocess files.
downward-resultfetcher.py tries to parse the files, while
resultfetcher.py only copies them.

To combine different properties files you should probably use
"downward-resultfetcher.py --dest dir-of-the-props-file". This will also
trigger recalculation of each run's quality.
> run-fd-test-grid     (script to run experiments on a grid)
same as above
> fd-test-compare      (script to compare results to baseline)
> template.q           (grid job template)
> run-fd-test-grid-all (script to run satisficing and optimal suites on grid)
looks good.

I got rid of comments out code.

The relevant files to review are:
run-fd-test          (script to run experiments on a single machine locally)
run-fd-test-grid     (script to run experiments on a grid)
fd-test-compare      (script to compare results to baseline)
template.q           (grid job template)
run-fd-test-grid-all (script to run satisficing and optimal suites on grid)

> Rietveld is giving me trouble - it's trying to upload a lot of files that are 
> not under source control, and it's taking forever.

I noticed too that all hg commands are a bit slow due to this. Note that every
subdirectory in new-scripts starting with exp-... is ignored (due to .hgignore)
and will be skipped quickly.

> However, since there aren't any actual code changes here (only shell scripts 
> which call the new-scripts python scripts), I'm not sure it's necessary.

I thought I saw some changes, but maybe I was not comparing to the closest
version. But either way, I think a code review would be good for this -- I tried
to follow what is exactly is going on but had some difficulty, and I think it'd
be good for Jendrik to have a look at the code, too. If you could clean it up a
bit first (getting rid of all the commented-out code), that'd be good, too.

Sorry, I guess the commits from habakuk have the wrong user name.
Rietveld is giving me trouble - it's trying to upload a lot of files that are 
not under source control, and it's taking forever.
However, since there aren't any actual code changes here (only shell scripts 
which call the new-scripts python scripts), I'm not sure it's necessary.
If you still want rietveld, I'll try to find a way around the useless uploads.

I started having a look at the code. Two comments so far:

- Commits should not be made with username "Fast Downward
  <downward@domain.invalid>."
- Can you open a rietveld issue for this?

Works for me. It'd still be good to open a new issue for this, though -- over
there, we could track which kinds of messages it would be good to send out. Some
of them would be a bit of work.

I think we can simply create another google group (like fast-
downward@googlegroups.com), which would be specifically meant for automated 
notifications.
Something like  fast-downward-notifications@googlegroups.com
and then anyone (or maybe just anyone we approve) could subscribe to this list.
If everyone's okay with this, I'll set this up.

> Maybe we should set up some permanent fixed address to send all e-mails to?

I've been thinking for a while that it'd be good to have a mailing list or some
such which could get some automated updates such as

1. new issues added to the tracker
2. regular status reports on the tracker (number of open issues, issues
   opened/closed recently, etc.)
3. buildbot failures
4. possibly commit messages

Other projects have this, and I think especially point 1. could be good for
letting people find out about new bugs reported etc. that they might be
interested in. Right now, new issue reports are only read by me unless the
submitter manually sets the nosy list, and I'm usually very conservative in
bothering other people about new issues since I don't want them to get the idea
that the work is forced down their throat.

If anyone would work on setting something like this up, it could also be a
natural recipient for these performance reports.

If either of you thinks this might make some sense, feel free to open a new
issue and/or take the discussion to the internal mailing list.

Sure.
For now, if you want to receive the e-mails, just change in  run-fd-test-grid  $USER@gmail.com to your e-mail address.

Thanks! I'll give it a whirl in the context of issue214. (If you don't hear from
me again about this by the end of next week, feel free to ping again.)

Actually, the e-mail is now sent to a non-existing address.
Maybe we should set up some permanent fixed address to send all e-mails to?

ssh downward@habakuk.informatik.uni-freiburg.de
cd erez/downward-fixes/new-scripts/
./run-fd-test-grid-all 

This should run the experiment, compare the results to the baseline (from the 
experiment I just finished running), and send the results via e-mail (to me, but 
that's easy enough to change).

How do I trigger an experiment, and how do I compare it to an older experiment?

As far as I'm concerned, this is ready.

I think it's just a matter of creating the files with the CONFIGS/SUITES for 
optimal and satisficing.
I'll do that soon.

Hi everyone, is this issue stuck? What's the next action that needs to be done
to get this moving?

Not currently. Experiments are always cartesian products of planner versions,
configurations and benchmarks. We were already discussing changing this
behaviour w.r.t. planner versions vs. configurations, so maybe it's worth
rethinking this more generally. For now, I'd just suggest generating two
experiments, though.

The nightly/weekly tests will run on the build slave (my desktop PC).

About the full tests on the grid - except for the portfolios, is there any way to 
create one experiment so that the optimal configs will run on the optimal domains 
and the satisficing configs will run on the satisficing domains?

The grid-based stuff should *not* run automatically every day or week, but only
on request. I'm not opposed to daily or weekly tests in general, but they should
be run locally. We should avoid spamming the grid (which right now we only have
access to as a courtesy, and where other people can get quite unhappy if it's
blocked by "unnecessary" experiments). (I think I wrote this earlier in the
email discussion, but I'm not sure if I wrote this before on the tracker.)

Regarding what is run in nightly/weekly tests, I have no preferences at the
moment. Regarding full tests, my suggestion would be

- all optimal IPC configs on the "OPTIMAL" suite
- all satisficing IPC configs on the "ALL" suite

(We really should have an "OPTIMAL" and a "SATISFICING" suite...)

I think the scripts don't currently support the IPC configs that use portfolios,
but Jendrik mentioned today that he had a fix for that. Maybe the two of you
could get in touch over this?

Using python for sending the mail seems to work.
I think this is almost ready to be merged - we just need to define our test 
suites for nightly/weekly/full (configs and problems).

Which script are you talking about? How and where does it try to send mail?
Does it produce any output?

If you do this from Python rather than bash and use
"smtp.informatik.uni-freiburg.de" instead of "localhost" in the SMTP constructor
call, you might get more verbose error reporting. Be sure to print the results
of the call to sendmail; "{}" means success if I remember the documentation
correctly.

The mail command seems to work (even when I do it from qlogin), but the actual 
script does not send me an e-mail.
Any ideas?

I tried from habakuk and it works with either localhost or
smtp.informatik.uni-freiburg.de as the SMTP server, but only if I use a valid
email address as the sender. downward@... is not valid, but this one works:

$ ssh downward@habakuk.informatik.uni-freiburg.de
[...]
downward@habakuk:~> python
[...]
Python 2.6 (r26:66714, Mar 30 2010, 00:30:21) 
[GCC 4.3.2 [gcc-4_3-branch revision 141291]] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from smtplib import SMTP
>>> connection = SMTP("localhost")
>>> connection.sendmail(from_addr="helmert@informatik.uni-freiburg.de",
to_addrs="malte.helmert@unibas.ch", msg="Hello from downward@habakuk")
{}
>>> [Ctrl-D]

Using downward@... as the result address gives no obvious error message, but the
email doesn't seem to arrive. Using smtp.informatik.uni-freiburg.de as the
server instead, we get a proper error message when using the downward@... from
address.

You can achieve the same effect with "mail" and the -r option to set the sender:

downward@habakuk:~> mail -r helmert@informatik.uni-freiburg.de
malte.helmert@unibas.ch
Subject: Foo
Bar
.

(There should be no line break before malte.helmert@unibas.ch; not sure how this
will be displayed in the tracker.) The "-r" argument is the from address, which
should be a valid email address like the one given. The following argument is
the to address.

I can't seem to send an e-mail from habakuk (I didn't try the grid computers 
yet). I'm trying to send the e-mails through the  localhost  SMTP server - is 
there some other server I should use?

Whatever works for you; we'll have to try it out in practice to say what works best.

Generally speaking, I often find it useful to share the HTML links to the
results. So if the results are easier to present in HTML than plain-text, one
option would be to copy to a location that is accessible from the web and send
an email with a link to the URL or something like that.

Okay, it took a bit more tweaking, but it works now.
The only question now is how to report the results?

The only idea I could think of is to send an e-mail, but I would welcome other 
suggestions.

Thanks, I think it worked (have to wait for the grid :-)

I have added the two flags --only-main-script and --no-main-script now. 
This should enable two-stage experiment creation.

That sounds risky. 
I don't know how the grid works exactly, but moving around files which have jobs 
pointing to them sounds like a bad idea.

You could try to mv the .q file before creating the experiment and 
moving it back after.

Ah, I see. I forgot the context.

I guess if the experiment is created in two stages, the experiment creation code
should be smart enough to see that this is what is going on, and the second
stage shouldn't consider this an "overwrite" but a completion of an already
started activity.

Yes, deleting (and even overwriting) the job file once it has been submitted is
dangerous. Could give us stale NFS handles and stuff like this.

But if I delete the whole directory then the .q file that's in there (with a job 
waiting to run it) will be deleted.
Won't that cause some problems?

Just delete the whole directory before running the script?
I think that's what "overwrite it" does anyway, right?

I think I know what the problem is.
When I create the search experiment, I get the following:

The directory "/home/downward/erez/downward-fixes/new-scripts/erez-exp" is not 
empty, do you want to overwrite it? (Y/N): 

and then nothing happens.
Would it be possible to add an overwrite flag?

Hmm, unclear, especially since the directory into which it can't chdir has been
cut off. :-) But I assume it's /home/downward/erez/downward-fixes/new-scripts/exp
 which does exist and is also properly accessible from the grid.

I think I'd need more info. Can you repeat the attempt (after cleaning up) and
send a complete shell transcript (what you typed and what you got for steps 1.-4.?

(The error-ed job should be removed from the queue with "qdel JOB_ID".)

I tried doing the following:
1. Create the search experiment skeleton (using the --build-main-script-only 
flag to downward_experiments.py)
2. Create a grid job to build the actual search experiment
3. Submit the grid job to build the search experiment
4. Submit the job to run the search experiment (the file created in step 1), 
making it wait until the job in step 3 has completed.

However, the last job does not start, giving the error below.
Any ideas?

downward@habakuk:~/erez/downward-fixes/new-scripts> qstat -j 16689
==============================================================
job_number:                 16689
exec_file:                  job_scripts/16689
submission_time:            Tue Aug 30 09:50:50 2011
owner:                      downward
uid:                        6933
group:                      ki-hiwi
gid:                        1280
sge_o_home:                 /home/downward
sge_o_log_name:             downward
sge_o_path:                 /opt/sge6-2/bin/lx24-
x86:/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/X11R6/bin:/usr/games:/opt/kde
3/bin:/usr/lib/mit/bin:/usr/lib/mit/sbin
sge_o_shell:                /bin/bash
sge_o_workdir:              /home/downward/erez/downward-fixes/new-scripts/exp
sge_o_host:                 habakuk
account:                    sge
cwd:                        /home/downward/erez/downward-fixes/new-scripts/exp
stderr_path_list:           NONE:NONE:exp.err
mail_list:                  downward@habakuk.informatik.uni-freiburg.de
notify:                     FALSE
job_name:                   exp.q
stdout_path_list:           NONE:NONE:exp.log
jobshare:                   0
hard_queue_list:            opteron_core.q
shell_list:                 NONE:/bin/bash
env_list:                   
job_args:                   -hold_jid,16688
script_file:                exp.q
job-array tasks:            1-14:1
error reason    1:          08/30/2011 09:50:57 [6933:20779]: error: can't chdir 
to /home/downward/erez/downward-fixes/new-scrip
error reason    2:          08/30/2011 09:50:58 [6933:28569]: error: can't chdir 
to /home/downward/erez/downward-fixes/new-scrip
error reason    3:          08/30/2011 09:50:57 [6933:32047]: error: can't chdir 
to /home/downward/erez/downward-fixes/new-scrip
error reason    4:          08/30/2011 09:50:57 [6933:21519]: error: can't chdir 
to /home/downward/erez/downward-fixes/new-scrip
error reason    5:          08/30/2011 09:50:57 [6933:21528]: error: can't chdir 
to /home/downward/erez/downward-fixes/new-scrip
error reason    6:          08/30/2011 09:50:57 [6933:20788]: error: can't chdir 
to /home/downward/erez/downward-fixes/new-scrip
error reason    7:          08/30/2011 09:50:58 [6933:28578]: error: can't chdir 
to /home/downward/erez/downward-fixes/new-scrip
error reason    8:          08/30/2011 09:50:57 [6933:32056]: error: can't chdir 
to /home/downward/erez/downward-fixes/new-scrip
error reason    9:          08/30/2011 09:50:57 [6933:21532]: error: can't chdir 
to /home/downward/erez/downward-fixes/new-scrip
error reason   10:          08/30/2011 09:50:58 [6933:28582]: error: can't chdir 
to /home/downward/erez/downward-fixes/new-scrip
error reason   11:          08/30/2011 09:50:57 [6933:20792]: error: can't chdir 
to /home/downward/erez/downward-fixes/new-scrip
error reason   12:          08/30/2011 09:50:57 [6933:32060]: error: can't chdir 
to /home/downward/erez/downward-fixes/new-scrip
error reason   13:          08/30/2011 09:50:57 [6933:21536]: error: can't chdir 
to /home/downward/erez/downward-fixes/new-scrip
error reason   14:          08/30/2011 09:50:57 [6933:20796]: error: can't chdir 
to /home/downward/erez/downward-fixes/new-scrip
scheduling info:            (Collecting of scheduler job information is turned 
off)

I got a little sidetracked with the multi-landmarks thing.
I'll try Jenrik's new flag, and see if it helps.

Erez.

On Mon, Aug 29, 2011 at 1:32 PM, Malte Helmert <
downward.issues@googlemail.com> wrote:

>
> Malte Helmert <malte.helmert@unibas.ch> added the comment:
>
> Erez, any progress on this?
>
> If not, are you stuck on something we can help with, or is it more a
> question of
> available time?
>
> _______________________________________________________
> Fast Downward issue tracker <downward.issues@gmail.com>
> <http://issues.fast-downward.org/issue133>
> _______________________________________________________
>

-- 

--------------------------------------------------------------
"Adventure is just bad planning."
    Roald Amundsen
    Norwegian Arctic & Antarctic explorer
    (1872 - 1928)
--------------------------------------------------------------

Erez, any progress on this?

If not, are you stuck on something we can help with, or is it more a question of
available time?

The parameter is there. Can you try it? 

I only added one parameter --build-main-script-only, because it doesn't matter if 
the script is later overwritten again.

Ok, I'll add the required parameter.

One option might be to use the --compact parameter. If used, the 
preprocessed files are not copied, but only referenced. This makes for 
much smaller experiments.

The properties files would be wrong, but also apart from this: that way lies
madness. Not updating files that need to be updated, not deleting files that
should be deleted, using wrong versions of files, etc. Besides, who says we
already have an old experiment with the right configurations and task sets lying
around?

I think it'd be a good idea to think about ways of making a slimmer experiment
setup, and I have some ideas (which involve moving more of the work to the
actual running script), but I don't think recycling is the answer.

I think we can make this even simpler, if we add a flag to the experiment telling 
the script to delete the old results (run.log, run.err, driver.log, driver.err, 
...) before running the search. 
Then we could simply re-run the same experiment over and over (replacing the 
binaries before each run), and collect and store the results afterward.

That's why I suggested the separation. The *.q file should be generatable
quickly on habakuk, without generating all the support files. Something like

habakuk $ cat > prepare-job.q
./downward_experiments.py -s $SUITE -c $CONFIGS --path $EXPNAME gkigrid
--do-everything-except-building-the-job-file
^D
habakuk $ qsub -q opteron_core.q prepare-job.q
Submitted job 12345.
habakuk $ ./downward_experiments.py -s $SUITE -c $CONFIGS --path $EXPNAME
gkigrid --only-build-the-job-file
habakuk $ cd $EXPNAME && qsub -q opteron_core.q job.q --but-please-wait-for 12345

As far as I know, the *.q file only needs to know a few settings (memory, time
limit) and the number of jobs to be created, so I think it should be possible to
generate it quite quickly.

Slight problem - I can't _create_ a job for running the search experiment (which 
would need to call   qsub search_exp.q) before the job to build the experiment 
finishes (since this is the job that create the search_exp.q file).

Malte's proposal should work. The preprocessed files land in 
preprocessed-files/WORK-WORK and will be fetched by any search 
experiment that also uses those "revisions" for the preprocessing.

>That would require some new options for the experiment building that would allow
>us to do*only*  the *.q file building or*only*  everything else.

I am not sure how you want to split up the experiment building. A simple 
.q file containing
./downward_experiments.py -s $SUITE -c $CONFIGS --path $EXPNAME gkigrid
should be sufficient.

> The --complete flag will run the translator and preprocessor for each 
> configuration, so it's not good here.

I think it's good for the cases where we want to do "whole-planner" experiments
(e.g. our pre-IPC sanity test), but these are indeed rare. Otherwise we should
go for the preprocessed preprocessed tasks. :-)

The --complete flag will run the translator and preprocessor for each 
configuration, so it's not good here.

However, creating all jobs in advance sounds like a good idea - I'll do that.

Some thoughts:

1) I think we shouldn't run a separate preprocess and search experiment as part
of this automated test. We should either work with already made preprocessed
data (the common use case, when we only changed the search code, to minimize
translator/preprocessor noise) or use a "complete experiment" that integrates
everything into one. I haven't used these complete experiments so far, but I
think they exist. ;-) In any case, I think the default should be to grab
already-available preprocessed data.

2) To submit a job, we only really need the *.q file, and as far as I know that
should be very quick to produce. So one way to do things if we want to do the
actual experiment-directory-building-and-analysis on the grid would be:

On habakuk:
1. Create a job for building the search experiment; submit it.
2. Create a job for running the search experiment; submit it, making it wait for 1.
3. Create a job for running the result-fetcher; submit it, making it wait for 2.
4. Create a job for doing the evaluation; submit it, making it wait for 3.

That would require some new options for the experiment building that would allow
us to do *only* the *.q file building or *only* everything else.

(Am I making myself clear?)

The idea is to do this:
1. (on habakuk) Create the preprocessing experiment, submit it to the grid
2. (on habakuk) Create the post-preprocessing job, submit it to the grid to 
start after (1)
3. (post-preprocessing job, on the grid) Create the search experiment, submit it 
to the grid
4. (post-preprocessing job, on the grid) Create the post-search job, submit it 
to the grid to start after (3)
5. (post-search job, on the grid) Collect and analyze results from search 
experiment

Can you send a brief description of the flow of information you had in mind? Who
triggers what where and how, which jobs are involved, etc.?

There's another problem - it appears I can't submit a job to the grid from a 
grid computer. I get the following message:
Unable to run job: denied: host "gkigrid5.informatik.privat" is no submit host.

One option is to submit the job from habakuk using ssh, but when I try to do 
this, it asks me for a password.
Any ideas?

Great, I'll try my scripts again tomorrow.

hg will be installed on the gkigrid machines today. The OS on those machines
comes with a rather old version (1.0.2), but that should be fine since it's the
same version as on habakuk. The OS will likely be upgraded in March/April.

In that case, I'll wait until hg is installed on the grid, and see if it works 
after that.

Working on it.

> Will --complete run preprocessing once,  or for each configuration?
It will run it for each configuration separately.

Of course, if we construct the experiments on the grid without hg, we can only 
use the working copy and not e.g. compare two revisions. So I would vote for 
having hg installed on the grid.

I'd prefer to avoid heavy lifting on habakuk, though. It's old, slow, and
everybody in the group suffers if it's put under load. If there's a way to do
more of the heavy lifting directly on the grid, that would be better. Jendrik,
any thoughts to share?

We can probably get around this if we can ssh from the grid to habakuk (so
that we can run the build experiments on habakuk, and then submit the actual
running back to the grid).
I'll try this with qlogin next week and see how it goes.

Erez.

On Thu, Aug 11, 2011 at 10:55 PM, Malte Helmert <
downward.issues@googlemail.com> wrote:

>
> Malte Helmert <malte.helmert@unibas.ch> added the comment:
>
> hg is indeed not installed on the grid:
>
> $ qlogin -q opteron_core.q
> local configuration habakuk.informatik.uni-freiburg.de not defined - using
> global configuration
> Your job 16602 ("QLOGIN") has been submitted
> waiting for interactive job to be scheduled ...
> Your interactive job 16602 has been successfully scheduled.
> Establishing builtin session to host gkigrid6.informatik.privat ...
> helmert@gkigrid6:~> hg
>
> Das Programm 'hg' kann im folgenden Paket gefunden werden:
>  * mercurial [ Pfad: /usr/bin/hg, Repository: zypp (repo-oss) ]
>
> Zum installieren versuchen Sie: sudo zypper install mercurial
>
> -bash: hg: command not found
>
> =============================================
>
> In general, you can use qlogin with the appropriate queue to get an
> interactive
> session on the respective queue, assuming that it's not busy. (Right now
> it's not.)
>
> If we can work around this, that would be great. If after evaluating all
> options
> we decide that we want hg on the grid, we can probably ask Uli to install
> it,
> though.
>
> _______________________________________________________
> Fast Downward issue tracker <downward.issues@gmail.com>
> <http://issues.fast-downward.org/issue133>
> _______________________________________________________
>

-- 

--------------------------------------------------------------
"Adventure is just bad planning."
    Roald Amundsen
    Norwegian Arctic & Antarctic explorer
    (1872 - 1928)
--------------------------------------------------------------

hg is indeed not installed on the grid:

$ qlogin -q opteron_core.q
local configuration habakuk.informatik.uni-freiburg.de not defined - using
global configuration
Your job 16602 ("QLOGIN") has been submitted
waiting for interactive job to be scheduled ...
Your interactive job 16602 has been successfully scheduled.
Establishing builtin session to host gkigrid6.informatik.privat ...
helmert@gkigrid6:~> hg

Das Programm 'hg' kann im folgenden Paket gefunden werden:
  * mercurial [ Pfad: /usr/bin/hg, Repository: zypp (repo-oss) ]

Zum installieren versuchen Sie: sudo zypper install mercurial

-bash: hg: command not found

=============================================

In general, you can use qlogin with the appropriate queue to get an interactive
session on the respective queue, assuming that it's not busy. (Right now it's not.)

If we can work around this, that would be great. If after evaluating all options
we decide that we want hg on the grid, we can probably ask Uli to install it,
though.

I want to run a preprocessing experiment, then a
search experiment, and then run an analysis.

Will --complete run preprocessing once,  or for each configuration?
On Aug 11, 2011 10:19 PM, "Jendrik" <downward.issues@googlemail.com> wrote:
>
> Jendrik <jendrik.seipp@mars.uni-freiburg.de> added the comment:
>
> That is probably the case. Can you outline what you're trying to
> achieve? Maybe I can come up with a solution that does not involve
> letting the grid build experiments.
>
> _______________________________________________________
> Fast Downward issue tracker <downward.issues@gmail.com>
> <http://issues.fast-downward.org/issue133>
> _______________________________________________________

That is probably the case. Can you outline what you're trying to 
achieve? Maybe I can come up with a solution that does not involve 
letting the grid build experiments.

Is it possible that HG is not installed on gkigrid2?
On Aug 11, 2011 9:50 PM, "Jendrik" <downward.issues@googlemail.com> wrote:
>
> Jendrik <jendrik.seipp@mars.uni-freiburg.de> added the comment:
>
> I'm not 100% sure what is going wrong and what you're trying to achieve.
I'll try to give you some input nonetheless :)
>
> 1) The error you describe in msg1558 arises because the program fails to
call 'hg log -r %s --template {node|short}' % rev
> This probably means that hg cannot be found which would be odd, because it
is installed on habakuk. How can this be reproduced? Do you
> think this is an error in the scripts?
>
> 2) Integrated experiments: You can create a combined preprocess and search
experiment by passing --complete to downward_experiments.py
>
> _______________________________________________________
> Fast Downward issue tracker <downward.issues@gmail.com>
> <http://issues.fast-downward.org/issue133>
> _______________________________________________________

I'm not 100% sure what is going wrong and what you're trying to achieve. I'll try to give you some input nonetheless :)

1) The error you describe in msg1558 arises because the program fails to call 'hg log -r %s --template {node|short}' % rev
   This probably means that hg cannot be found which would be odd, because it is installed on habakuk. How can this be reproduced? Do you                    
think this is an error in the scripts?

2) Integrated experiments: You can create a combined preprocess and search experiment by passing --complete to downward_experiments.py

This is probably more in Jendrik's territory (he said hopefully ;-)).

Can you send precise instructions on how he can reproduce this (shell transcript)?

The directory is the same one, and it's even the same user.
It might be some environment variable, but I don't know.
 On Aug 11, 2011 5:31 PM, "Malte Helmert" <downward.issues@googlemail.com>
wrote:
>
> Malte Helmert <malte.helmert@unibas.ch> added the comment:
>
> PS: Regarding first running the preprocessor and then the search code:
>
> 1) There is a way to run an integrated experiment. Jendrik knows more
about that.
>
> 2) For most of the "acceptance test" experiments I had in mind, e.g. the
one for
> Andrew's state-space patch, it's preferable to reuse old preprocessed data
to
> reduce noise.
>
> _______________________________________________________
> Fast Downward issue tracker <downward.issues@gmail.com>
> <http://issues.fast-downward.org/issue133>
> _______________________________________________________

PS: Regarding first running the preprocessor and then the search code:

1) There is a way to run an integrated experiment. Jendrik knows more about that.

2) For most of the "acceptance test" experiments I had in mind, e.g. the one for
Andrew's state-space patch, it's preferable to reuse old preprocessed data to
reduce noise.

These machines have the same NFS mounts. One possible guess is that it's an
issue with the working directory from which things are run, so it might be
useful to log that near the start of the job. You might also want to log which
command is attempted to run here.

I've made some progress with running this on the grid, but I've run into some 
problems.
What I do is run the preprocessing experiment on the grid, with a job that waits 
until preprocessing is finished. The second job builds the actual search 
experiment.
The problem is that the ./downward_experiments.py call in the second job gives 
an error. The only difference I've seen is that it doesn't run on habakuk, but 
on some grid computer (gkigrid2, last I checked). The same command runs fine on 
habakuk.
Does either of you have any ideas?

The error is:
Traceback (most recent call last):
  File "./downward_experiments.py", line 387, in <module>
    build_experiment(combinations)
  File "./downward_experiments.py", line 378, in build_experiment
    exp = DownwardExperiment(combinations)
  File "./downward_experiments.py", line 203, in __init__
    self.make_runs()
  File "./downward_experiments.py", line 302, in make_runs
    self._make_search_runs()
  File "./downward_experiments.py", line 326, in _make_search_runs
    run = DownwardRun(self, translator, preprocessor, planner, prob)
  File "./downward_experiments.py", line 79, in __init__
    self.set_properties()
  File "./downward_experiments.py", line 89, in set_properties
    self.set_property('translator_parent', self.translator.parent_rev)
  File "/home/downward/erez/downward-fixes/new-scripts/checkouts.py", line 187, 
in parent_rev
    self.parent = tools.run_command(cmd)
  File "/home/downward/erez/downward-fixes/new-scripts/tools.py", line 185, in 
run_command
    p = subprocess.Popen(cmd, stdout=subprocess.PIPE, env=env)
  File "/usr/lib64/python2.6/subprocess.py", line 595, in __init__
    errread, errwrite)
  File "/usr/lib64/python2.6/subprocess.py", line 1106, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory

On Thu, Aug 11, 2011 at 12:17 PM, Malte Helmert <
downward.issues@googlemail.com> wrote:

Sure it can wait.

I'll look into it.

> _______________________________________________________
> Fast Downward issue tracker <downward.issues@gmail.com>
> <http://issues.fast-downward.org/issue133>
> _______________________________________________________
>

-- 

--------------------------------------------------------------
"Adventure is just bad planning."
    Roald Amundsen
    Norwegian Arctic & Antarctic explorer
    (1872 - 1928)
--------------------------------------------------------------

> Currently the line that checks initial_h_value is commented out - once this
> is fixed I can add it back.

Is it OK if I wait with the merging until this is fixed? Every merge is work;
not terribly much, but it adds up.

> To do regression tests on the grid, I will need to use the option Malte 
> mentioned, that runs something once all the jobs are finished. 
> Is this something that needs to be in the python code, or can I specify this 
> externally from the command line somehow?

That should be a separate grid job, since it should only be run after all grid
tasks (i.e. "sub-jobs") have been run. So I'd do the following:

$ qsub [grid job that does the actual work]
$ qsub [grid job that does the analysis, with options to defer until the
        job that does the actual work is finished]

The deferring part can be done with the "-hold_jid" option. Basically, you can
say that job 2 should be put on hold until job 1 is done. I've only done this
once (to defer a search experiment until some preprocess experiment was done),
so don't remember all details. "man 1 qsub" will tell you. Once you've figured
this out, it would be useful information to put on the GkiGrid wiki page.

The nightly/weekly test code is ready to be merged.
There's still one problem with the initial_h_value attribute -
I get this message:
2011-08-11 10:49:05,995 ERROR    The following attributes are not present in the 
dataset: ['initial_h_value']

Currently the line that checks initial_h_value is commented out - once this is 
fixed I can add it back.

To do regression tests on the grid, I will need to use the option Malte 
mentioned, that runs something once all the jobs are finished. 
Is this something that needs to be in the python code, or can I specify this 
externally from the command line somehow?

> Malte - can you merge this for now, so that nightly/weekly tests are back
> online?

Done, but I haven't closed the branch yet. Let me know when I should do the
final merge.

> Adapting the script to run on the grid should be easy enough (just change
> local to gkigrid in the call to ./downward_experiment.py, right?), so it's
> just a matter of running the second part of the script asynchronously.

One note on that: It's possible to submit a second job after the first job with
a dependency, so that it is only started once all tasks of the first job are
completed. This can be used to run the analysis, send out a ping or do whatever
other activity is necessary after the actual planner runs are completed.

Both issues have been resolved.

Two small things:
1. I get this error with the -rel report

AttributeError: 'RelativeReport' object has no attribute 'change'
Traceback (most recent call last):
  File "./downward-reports.py", line 429, in <module>
    report.write()
  File "/home/batman/hg/downward-fixes/new-scripts/reports.py", line 168, in 
write
    self.write_to_disk(self.build())
  File "/home/batman/hg/downward-fixes/new-scripts/reports.py", line 177, in 
build
    text = self.get_text()
  File "./downward-reports.py", line 96, in get_text
    table = self._get_table(attribute)
  File "./downward-reports.py", line 276, in _get_table
    logging.info('No changes above %d%% for "%s"' % (self.change,

2. When it works, I still get an INFO line saying "creating report" even though 
I have -l WARNING set. This is a problem, since I check for empty output.

Yes it is ;) There are now the two parameters --rel-change and --abs-change

Nightly/Weekly test scripts now work again (we still might want to discuss what 
problems/configs we want to test nightly/weekly).
Also - there might be some noise in the reports, since they check for a relative 
change of over 5%, so run time changing from 0.11 to 0.12 looks like an error.

Jendrik - is it possible to add a minimum absolute change parameter?

Adapting the script to run on the grid should be easy enough (just change local 
to gkigrid in the call to ./downward_experiment.py, right?), so it's just a 
matter of running the second part of the script asynchronously.
I'll look into how to do that soon, but I might need some support from Jendrik 
in adding a "post-experiment" hook to experiments.

Malte - can you merge this for now, so that nightly/weekly tests are back 
online?

For the record, let me paste some stuff from an email to Erez:

Here's a set of requirements to get started:

1) It should be easy to define metrics. Each metric is defined by
  1) A planner configuration (e.g. seq-opt-fdss-1 with 30 min timeout
     and 2 GB memory limit)
  2) A bunch of inputs (e.g. all IPC 2008 optimal tasks)
  3) A scoring function (e.g. IPC score)
and defines a single number. If multiple metrics can be computed from
the same data (e.g. because they differ only on the scoring
function, or the input set is a subset of another input set for which we
have data), then the existing data should be used rather than running
the planner again.

For starters, let's just consider one metric for each of our IPC 2011
systems, using the IPC 2008 inputs (seq-sat or seq-opt as appropriate)
and IPC score as the scoring function. (There's one problem here
because that metric is w.r.t. a reference quality which might change
over time, but let's maybe ignore that problem for now and assume we
know the reference quality for each task.)

2) It should be possible to send off a single command that says "I
want to compute metric X on the current revision" and starts the
respective computation. Since I'm thinking of something that can take
a week or so to compute, it should be started asynchronously. Once
it's done, there should be some kind of notification that the data is
available.

All data should be stored permanently in some kind of central
repository and not be recomputed if already available (unless
explicitly cleared). The raw experiment data should be stored
away somewhere, too, although we will have to think about what exactly
to keep since this the complete raw data will grow unreasonably.

3) It should be possible to generate a simple HTML table or LaTeX
report that tabulates the value of a certain metric or a certain set
of metrics across all revisions for which we have data. We probably
want some metadata for revisions (nicknames, dates, associated issues)
so that the table is interpretable.

The tests should be adapted to the new scripts.
We should also do this in a way that allows running a full suite of tests (on the 
grid) from the command line (and store the results by version)

Merged.

Jendrik, if you have any comments on Erez's usage of the new_scripts (if some
things could be simplified etc.), let us know.

This is in a branch called 'issue133new'.
(hg gave me a hard time with the old issue133 branch)

I'll look into it next week.

Is this fixed now? I remember that I couldn't merge this at the time because it
was intertwined with the problem class code, but I don't remember if we fixed it
some other way.

If not, can you create a new branch issue133 off the default branch so that I
can merge this? (It shouldn't be a problem if a branch by the same name already
exists elsewhere except that I have to be a bit careful when pulling the changes.)

In issue133 branch in my repository

The daily and weekly tests currently fail, but are still showing green lights,
probably because no exit codes are propagated by the scripts. We should

 1) fix the scripts so that they show errors, and then
 2) fix the errors.