I have added portfolio support now. The python portfolios must reside in the planner's directory and can be called by 
doing:

./downward --portfolio myportfolio.py --plan-file sas_plan < output

The portfolio format can be seen e.g. in downward-seq-sat-fdss-1.py

The scripts have also been adapted. You can see a portfolio experiment example in test-exp.sh

I have thought a little about this issue and wanted to ask if it's really necessary to 
let the main downward script call the portfolios (i.e. ./downward ... --portfolio XXX).

Wouldn't it be more intuitive to just implement portfolios like downward-seq-sat-fdss-
1.py and call them directly?

Another option would be to give seq_sat_portfolio.py the commandline option "--
inputfile XXX". XXX would be the path to a file in the format of downward-seq-sat-fdss-
1.py

(The same applies for the optimal versions of the scripts and the portfolio module)

I don't think that the savings are worth the more complicated (and more
error-prone) code, so I would prefer to leave the portfolio functionality in the
scripts rather than making it part of the core planner. (And as long as we have
not released a version 1.0, I am anyway strictly against adding new
functionality to the core planner.)

Right now there are many practical concerns with handling this planner-internally.

Memory overflows are one; another is that we'd need to be able to interrupt the
code at many points, not just during search. For example:

 - LM construction can time out
 - M&S heuristic construction can time out
 - a single heuristic evaluation of h^m can time out

So we'd need lots of places where we check the timeouts *and* a way to
gracefully bail out without losing memory resources at all these checkpoints. So
we'd have to deal with partially constructed heuristics objects etc. It's
possible, but it won't be easy to get right and will be quite a bit of work.

So right now I think just killing and restarting is a more practical solution.
Of course that means reparsing etc., and that's not great, but it's also usually
not terrible.

Another consideration is that if we'd do all this planner-internally, we should
delay constructing heuristics etc. until they are really needed, and discard
them as soon as they are no longer needed. For example, right now it's not
possible to do something like this:

 - construct heuristic h1 (say, an LM heuristic)
 - do a search with h1
 - release the memory associated with h1
 - construct heuristic h2 (say, a M&S heuristic)
 - do a search with h2
 - release the memory associated with h2

(OK, we can make it possible in some cases where we don't need to predefine
heuristics with --heuristic, but in many cases we must do that, e.g. to use a
heuristic both for its estimates and for preferred operators.)

If we want to handle things like this planner-internally, we would need a way to
limit the lifetime of declared heuristics, landmark graphs and similar objects.

Sorry to butt in like this, but it seems to me like we don't need portfolios. 
Instead, we need to be able to specify timeouts on a few things (search, 
heuristic initialization, ...). 
Then we would be able to support sequential portfolios just by using iterated 
search with timeouts. This also saves us the planner initialization time 
(reading the SAS file), and opens the door to more complex behavior (A* + 
heuristic which changes after some time, without re-expanding everything).
Of course, there's still the problem of handling memory overflow, but I think we 
can guard against that in the code.

Any thoughts?

I see. In principle yes, but it would be good to avoid ever more convoluted
indirections in the calling scripts, so I'd prefer to have the planning script
only accept one new parameter, the actual portfolio, rather than passing a
portfolio *and* a portfolio runner script.

We might need some additional options about how to select the anytime behavour,
but I wouldn't want to pass a complete python script like we do currently.
Something like

$ ./downward < output --anytime-portfolio XXX
or
$ ./downward < output --simple-portfolio XXX
would be nicer, I think.

Or alternatively the "anytime"/"simple" choice could be part of XXX and we'd
simply have
$ ./downward < output --portfolio XXX

I wouldn't mind if XXX used Python syntax, but it should only contain the data
definition of what currently is in e.g. downward-seq-opt-fdss-1.py, not the code
bits.

Probably this was too naive, but I thought the modules seq_opt_portfolio.py and 
seq_sat_portfolio.py could be used to run any kind of scripts that have the form 
of downward-seq-opt-fdss-1.py.

I don't think I understand, especially the second part.
Have you had a look at how the portfolios currently work, especially in the
satisficing case, where there's some complicated interactions with anytime
search? How does your suggestion relate to this? Can you give a usage example
and explain what goes on internally for this example?

How about this approach:

- the downward script gets two new parameters:
   - a python portfolio file
   - a python script that calls the planner repeatedly with the configs 
in the portfolio file

Not sure where the most natural place for this is, but I like to think of the
scripts as a convenient way to run experiments rather than something that *must*
be used in order to run some version of Fast Downward. So that probably means
that a better place to live would be in the src/ hierarchy (rather than
new-scripts). So I guess my answer is: in the downward script (which itself
needs cleaning up).

Any thoughts on this? Do you think this is woth supporting in the long run? Or 
should this be hacked into the downward script if it is really needed?

Recent IPCs have shown that there might be some interest in running portfolio 
planners...

What do you think would be the best way to go about this? 
- Adapt seq_{sat,opt}_portfolio.py?
- Add CallGroup class that has Call items (with individual timeouts) and a 
global timeout?

Personally I think the second option is preferrable.