Issue131

Title scripts to run translator + preprocessor separately from search
Priority feature Status resolved
Superseder Flag unexpected failures for new-scripts experiments
View: 137
Nosy List jendrik, malte
Assigned To jendrik Keywords
Optional summary

Created on 2010-10-16.20:26:47 by malte, last changed by jendrik.

Messages
msg629 (view) Author: jendrik Date: 2010-10-29.22:29:37
Those files are now included automatically.
msg626 (view) Author: malte Date: 2010-10-29.19:40:39
Testing the tracker. Please ignore.
msg625 (view) Author: malte Date: 2010-10-29.19:20:32
Sorry for reopening, but I have to revise my "Sounds good to me" comment:

The preprocessed-problems directories should not just contain the "output"
files, but also the files generated by the translator: output.sas, test.groups,
all.groups. The all.groups file is actually needed by some of the planner
configs (landmark-count heuristic), test.groups is very useful for debugging,
and output.sas is sometimes somewhat useful for debugging.
msg624 (view) Author: jendrik Date: 2010-10-29.19:17:36
Closing the issue.
msg623 (view) Author: jendrik Date: 2010-10-29.19:16:58
Separate experiments are now the default behaviour. Opened issue137 for tracking 
the failure flagging.
msg567 (view) Author: malte Date: 2010-10-17.14:20:14
Sounds good to me.

> What type of error handling do you prefer when a *search component* experiment
> is run and no preprocessed files are found?

I'd go for a warning message at the point the experiment is created.

If the experiment is run despite the warnings, the runs should fail with some
meaningful error message.

Regarding error messages, I wonder what our current status is regarding expected
vs. unexpected failures of the planner. "Expected failures" are when the planner
fails to solve a problems because it times our or runs out of memory;
"unexpected failures" are everything else (segmentation faults, missing inputs,
bugs in the scripts that call the planner).

Unexpected failures should always be flagged so that they're not overlooked when
analyzing experimental results.
msg566 (view) Author: jendrik Date: 2010-10-17.03:19:38
I have thought about the issue a bit and came up with the following stuff:

The preprocessed files will be put in new-scripts/preprocessed-problems/DOMAIN/PROBLEM dirs. Those dirs will contain an "output" file and a file containing the revision number of the translator 
and preprocessor. Is this good? Should there be only one dir per domain?

What type of error handling do you prefer when a *search component* experiment is run and no preprocessed files are found?

- Automatic preprocessing
- Display of an error message
- No error message (Running the experiment will crash)
msg564 (view) Author: malte Date: 2010-10-16.20:26:47
In most cases, we want to run the translator/preprocessor only once, and then do
separate experiments using the search component only (on the already
preprocessed inputs).
History
Date User Action Args
2010-10-29 22:29:37jendriksetstatus: chatting -> resolved
messages: + msg629
2010-10-29 19:41:45maltesetsuperseder: + Flag unexpected failures for new-scripts experiments
2010-10-29 19:40:39maltesetsuperseder: - Flag unexpected failures for new-scripts experiments
messages: + msg626
2010-10-29 19:20:32maltesetstatus: resolved -> chatting
messages: + msg625
2010-10-29 19:17:36jendriksetstatus: chatting -> resolved
messages: + msg624
2010-10-29 19:16:59jendriksetsuperseder: + Flag unexpected failures for new-scripts experiments
messages: + msg623
2010-10-17 14:20:14maltesetmessages: + msg567
2010-10-17 03:19:38jendriksetmessages: + msg566
2010-10-16 20:26:47maltecreate