Performance looks good and no more wall-clock timeouts for unsolvable mystery 
problems (due to the portfolio never giving up trying to find a solution).

Merged.

Sounds good, please test and merge if performance looks OK.

I think the best solution is to restore the old behaviour (run last_config only after all 
configs have run). The patch would therefore only change one thing, namely it would just 
consider successful configs in the next round instead of all configs.

I'm not sure I understand the details of this; I'm not sure how the code worked
before, so it's hard to comment on the suggested changes. Maybe it's worth
discussing this briefly between the two of us and Gabi, given that Gabi
implemented the satisficing FDSS.

> If final_config is given but not final_config_builder, 
> run it after the first solution has been found, to 
> mirror the behaviour of portfolios with final_config_builder. 
> Previously, all remaining configs were run before the 
> last_config was started.

This change significantly reduces the plan quality for FDSS-1 (the only portfolio that uses the final_config parameter):
Switching to last_config after a solution has been found produces worse plans. IPC score with final_config after first solution: 
1884.20, IPC score with final_config only after all configs have been run: 1963.72 (suite has 2252 problems, the score is calculated 
from the <=4 plan costs found by FDSS-{1,2} on the default and issue branch).

Two solutions come to mind: 

- Restore the old behaviour and run last_config only after all configs have run.
- Remove final_config from FDSS-1 (and possibly from portfolio.py altogether).

I vote for the first solution although it is inconsistent with the behaviour in the case last_config_builder is given.

In case we choose solution 1, we have to decide if we want to start final_config directly after the last regular config or if we want 
to run the set of successful configs again first. These might perform better with lower g_bounds.

Sounds good!

> - When config X fails do not run it again in the next round.

This should be documented in the portfolio user documentation (which needs to be
added if it doesn't exist :-)) because it's something the user must be aware of
when using randomized algorithms.

> It seems this difference is due to noise.

"It seems" is a bit vague. :-) We need clear evidence for this before we can merge.

I took a stab at this now. There are multiple changes in the proposed branch at bitbucket:

- When config X fails do not run it again in the next round.
- Only rerun the same config with real costs if we can really change the cost handling (this is not necessarily the case for custom 
portfolios).
- If final_config is given but not final_config_builder, run it after the first solution has been found, to mirror the behaviour of 
portfolios with final_config_builder. Previously, all remaining configs were run before the last_config was started.

To facilitate these changes I refactored the code.

A preliminary experiment with a search_time limit of 3min shows that the number of solved tasks stays the same for FDSS-2 and decreases by 
3 for FDSS-1. It seems this difference is due to noise.

> Currently the N configs of a sat portfolio are run in round-robin fashion
> until the timeout occurs. We should stop a portfolio with N configurations if
> N consecutive configs find no additional plan to save time.

We can be stricter: as soon as config X fails to find a better plan, remove X
from future consideration. If it fails with bound B, it should fail with a lower
bound B'.

> For some configurations it may even make sense to stop directly after they
> haven't found any better plan.

"Most" configurations, I would say.

Currently the N configs of a sat portfolio are run in round-robin fashion until 
the timeout occurs. We should stop a portfolio with N configurations if N 
consecutive configs find no additional plan to save time.

For some configurations it may even make sense to stop directly after they 
haven't found any better plan.