Issue137

Title Flag unexpected failures for new-scripts experiments
Priority feature Status resolved
Superseder Nosy List florian, jendrik, malte
Assigned To jendrik Keywords
Optional summary

Created on 2010-10-29.19:15:46 by jendrik, last changed by jendrik.

Messages
msg2406 (view) Author: jendrik Date: 2013-03-22.16:07:48
Now it is ;)
msg2405 (view) Author: malte Date: 2013-03-22.15:54:43
Excellent! Is this documented?
msg2404 (view) Author: jendrik Date: 2013-03-22.15:49:35
The AbsoluteReport in the lab package now flags all unexplained errors (also 
invalid solutions) in a separate table. Additionally, warnings are raised already 
when the results are fetched from the experiment directory. I'll close this 
issue, feel free to reopen in case something's missing.
msg1215 (view) Author: jendrik Date: 2011-01-22.19:13:29
Note from issue197: Flag invalid plans too.
msg755 (view) Author: malte Date: 2010-11-25.15:16:18
> MemoryErrors and timeouts are caught better now, but are not added to the
> properties file since they are expected errors. Is that the desired
> behaviour?

Errors in the translator or preprocessor should always be considered errors (if
nothing else, they are serious performance issues). For the search code, running
out of memory or time should indeed be considered an expected failure.
msg754 (view) Author: jendrik Date: 2010-11-25.14:59:52
I have looked into this and implemented this functionality in the run-template.
Errors are now added to the properties file for each of the three parts 
(pre/post-processing, main command), so segmentation faults or missing inputs 
can be seen in the reports. MemoryErrors and timeouts are caught better now, but 
are not added to the properties file since they are expected errors. Is that the 
desired behaviour? 
Errors in the translator cause the preprocessor not to be run now.

Although the errors are present in the reports, we should probably think about a 
better way to present them. At the moment they are just another table.
msg622 (view) Author: jendrik Date: 2010-10-29.19:15:46
Quoting Malte in issue131:

Regarding error messages, I wonder what our current status is regarding expected
vs. unexpected failures of the planner. "Expected failures" are when the planner
fails to solve a problems because it times our or runs out of memory;
"unexpected failures" are everything else (segmentation faults, missing inputs,
bugs in the scripts that call the planner).

Unexpected failures should always be flagged so that they're not overlooked when
analyzing experimental results.
History
Date User Action Args
2013-03-22 16:07:48jendriksetstatus: chatting -> resolved
messages: + msg2406
2013-03-22 15:54:43maltesetstatus: resolved -> chatting
messages: + msg2405
2013-03-22 15:49:35jendriksetstatus: chatting -> resolved
nosy: + florian
messages: + msg2404
2011-01-22 19:13:30jendriksetmessages: + msg1215
2010-11-25 15:16:18maltesetmessages: + msg755
2010-11-25 14:59:53jendriksetmessages: + msg754
2010-10-29 19:41:45maltelinkissue131 superseder
2010-10-29 19:40:39malteunlinkissue131 superseder
2010-10-29 19:16:59jendriklinkissue131 superseder
2010-10-29 19:15:46jendrikcreate