I added the documentation and merged the branch into master.

It would be nice to have something like your msg2764 (but with the updated
rules) documented somewhere, since this is something a user might want to know.
A block comment in the code is enough, in my opinion, but they shouldn't have to
read the actual code. (I didn't check if we already have such a comment.)

Otherwise, feel free to merge. Florian already looked at it, so I don't need to
review it.

Looks good to me.

I opened a pull request on bitbucket with a small patch:

In case we have only a single kind of unexpected exit code, let the portfolio 
return it. By returning e.g. EXIT_INPUT_ERROR instead of EXIT_CRITICAL_ERROR it 
becomes easier to find the source of an error.

I think we should still implement part 2) from below:

2) If no config in the portfolio supports the task, return EXIT_UNSUPPORTED 
instead of EXIT_CRITICAL_ERROR.

OK, that should indeed not pass silently. Closing this issue then.

I prefer the current behaviour because of "errors should never pass silently". 

Running a planner on a problem that it doesn't support is a user error of which
the user should be made aware. This is still true if it's only one configuration
in a portfolio -- presumably, the user went to some effort to pick this
particular portfolio, and if it turns out that they're running on a domain where
some configurations don't make sense, they would have picked a different
portfolio or different timings.

For example, if I have a portfolio which first runs planner A for 100 seconds
and then planner B for 1700 seconds, I'd like to know if I'm running this on a
problem not supported by planner B.

My last comment is a little messy. Let me clarify and add some numbers for easier referencing:

1) At the moment the overall portfolio returncode is EXIT_CRITICAL_ERROR if one of the configs does not support the task. I think we 
should include EXIT_UNSUPPORTED in the list of expected error codes and return EXIT_SOLVED if at least one config solves the task even 
if other configs don't support it.

2) If no config supports the task, I think we should return EXIT_UNSUPPORTED.

3) When EXIT_UNSUPPORTED is an expected returncode we have to decide what the overall returncode should be for the following 
combinations:

[EXIT_UNSOLVED_INCOMPLETE, EXIT_UNSUPPORTED, ...]
[EXIT_TIMEOUT, EXIT_UNSUPPORTED, ...]
[EXIT_OUT_OF_MEMORY, EXIT_UNSUPPORTED, ...]
[EXIT_OUT_OF_MEMORY, EXIT_TIMEOUT, EXIT_UNSUPPORTED, ...]

I propose removing all EXIT_UNSUPPORTED returncodes from the list of returncodes and calculating the overall returncode as before. 
(This solution would render 1) obsolete).

At the moment the overall portfolio returncode is calculated from the individual planner returncodes in the following way:

[EXIT_SOLVED, EXIT_UNSUPPORTED, ...] -> EXIT_CRITICAL_ERROR (correct: EXIT_SOLVED)

I think we should include EXIT_UNSUPPORTED in the list of expected error codes. Which exit code should the portfolio return for 

[EXIT_TIMEOUT, EXIT_UNSUPPORTED, ...]
[EXIT_OUT_OF_MEMORY, EXIT_UNSUPPORTED, ...]
[EXIT_OUT_OF_MEMORY, EXIT_TIMEOUT, EXIT_UNSUPPORTED, ...]

BTW: Is the following calculation correct?

[EXIT_UNSOLVED_INCOMPLETE, EXIT_TIMEOUT, ...] -> EXIT_UNSOLVED_INCOMPLETE