Issue932

Title update or remove stone soup scripts
Priority wish Status resolved
Superseder Nosy List jendrik, malte, silvan
Assigned To jendrik Keywords
Optional summary

Created on 2019-10-08.19:26:38 by jendrik, last changed by jendrik.

Messages
msg9015 (view) Author: jendrik Date: 2019-10-16.14:04:18
Merged.
msg9003 (view) Author: malte Date: 2019-10-09.17:19:53
Looks good, including the changelog entry.
msg9002 (view) Author: jendrik Date: 2019-10-09.14:48:53
Done.

I prepared a pull request that removes the Stone Soup files from the Fast Downward repo: https://bitbucket.org/jendrikseipp/downward/pull-requests/142
msg9001 (view) Author: malte Date: 2019-10-09.14:00:38
Feel free!
msg9000 (view) Author: jendrik Date: 2019-10-09.14:00:23
We could put the old scripts (and maybe even the new scripts) into a new repository. Shall I create a repo at https://bitbucket.org/aibasel/stonesoup ?
msg8999 (view) Author: malte Date: 2019-10-09.13:49:12
It's not really a specific portfolio, it's the "best" (modulo the greediness of the algorithm) portfolio for the given input planner configurations and input benchmark results. So I see utility in this, people can use this in a similar way that they use things like SMAC. But I certainly agree that this isn't something that has to be in this repository. It would be nice to have it accessible somewhere.
msg8998 (view) Author: silvan Date: 2019-10-09.12:52:37
I don't see a reason why we should have scripts to generate a specific Fast Downward portfolio in the code base. Regarding the point that such scripts serve as documentation for how to build stone soup portfolios - isn't the paper https://ai.dmi.unibas.ch/papers/helmert-et-al-ipc2011.pdf good enough for this purpose? 

I'm in favor of entirely removing the scripts.
msg8993 (view) Author: malte Date: 2019-10-08.19:28:57
I don't like the first option very much.

If newer Stone Stoup scripts are somehow discoverable (e.g. via a Zenodo link from a Stone Soup planner abstracts or something along those lines), then I think they don't necessarily need to be in the Fast Downward repository, although I also don't see much harm in having them there if it isn't a massive number of files.
msg8992 (view) Author: jendrik Date: 2019-10-08.19:26:38
This came up during issue931 (https://bitbucket.org/jendrikseipp/downward/pull-requests/139/issue931/activity#comment-119686443). 

The repo contains the scripts that were used to generate the IPC 2011 stone soup portfolios under misc/stonesoup/. There are two scripts that are basically identical except that one uses coverage scores (0 or 1 for optimal planning) and the other uses IPC scores (between 0 and 1 for satisficing planning).

The scripts expect Lab properties files in a very old format (configobj) and therefore can't be used for new experiment data. However, it serves as documentation for how to build stone soup portfolios. The question is whether that documentation should live in the Fast Downward repository. For IPC 2018 I ported one of the scripts to the new JSON properties format. Also, my version uses numpy for faster processing.

We are unsure about how best to proceed here. We see the following options:

* Keep misc/stonesoup/ as it is.
* Remove misc/stonesoup/ (and possibly move it somewhere else).
* Remove misc/stonesoup/* and add the new stone soup script instead.

What do you think?
History
Date User Action Args
2019-10-16 14:04:18jendriksetstatus: reviewing -> resolved
messages: + msg9015
2019-10-09 17:19:53maltesetmessages: + msg9003
2019-10-09 14:48:53jendriksetstatus: chatting -> reviewing
assignedto: jendrik
messages: + msg9002
2019-10-09 14:00:38maltesetmessages: + msg9001
2019-10-09 14:00:23jendriksetmessages: + msg9000
2019-10-09 13:49:12maltesetmessages: + msg8999
2019-10-09 12:52:37silvansetnosy: + silvan
messages: + msg8998
2019-10-08 19:28:57maltesetstatus: unread -> chatting
messages: + msg8993
2019-10-08 19:26:38jendrikcreate