The numbers did not change much in between the two experiments. I merged and 
pushed the branch. Will send a short notice to the dev list now.

If the results are as before, feel free to merge.

I have made the suggested changes and conducted a new experiment. The results are 
attached and the code review has been updated:
https://bitbucket.org/jendrikseipp/downward/pull-request/16/issue392-
succ_mode/diff

Reviewed.

No need to keep the old experiments if the new ones work out fine.

I have made your suggested changes and updated the code review. Can you have a 
look, please? 

Experiments: Do we want to keep the old experiments or should I just create two 
new ones for the LAMA configs with the new common_setup.py?

Thanks, Jendrik!

I would say there are few strong trends in the experiments. The experimental
results for the LAMA-like configs have a trend towards using no randomization,
but that's not really true for the FF-like configs, where the overall winner (in
terms of quality) is a shuffled config. There are also no clear general trends;
it depends quite a bit on the domain.

Given this, I think it's good to keep options available here, so that it's
possible to choose whatever is good for the given domain.

The question then is what the default should be. All other things being equal,
"original" would seem a logical default since that's what the other search
algorithms use. If we add up the quality scores in the two LAMA experiments, it
also performs better than our current default. So I would suggest making
"original" the default in the future.

Regarding "shuffled_preferred_first": to me, randomizing the order and ordering
preferred operators first are two orthogonal things, and I think it's cleanest
and simplest to avoid "multiplying out" the choices in an enum. Rather, I think
we should have two boolean options:

- randomize_successors (default: false)
- preferred_successors_first (default: false)

The documentation should make clear that randomization happens first (if
enabled), and after that preferred successors are moved to the front (if enabled).

I checked the code, and if I didn't overlook something, this would also suggest
a simplified implementation of the form:

if (randomize_successors) {
    [shuffle operators]
}

if (preferred_successors_first {
    ...
}

Can you modify the code along these lines? I don't think we need another
experiment now, but I'd like to have a repeated experiment to be on the safe
side just before the final merge (i.e. once code review is complete). I think
it's enough to only repeat the lama-unit and lama-nonunit experiments. If we do
a repeated experiment, I think it might also be good to use the most recent
version of our IssueExperiment class.

Going over the old messages, the remaining TODOs are to open an issue for the
future cleaner implementation of this feature as part of a general successor
modification functionality that could also be used for partial-order reduction
and that should also be part of eager search (msg2796 and msg2985).

Code: I have renamed "succ_mode" to "succ_order" and added a new ordering "shuffled_pref_first" 
that orders preferred operators before other operators and shuffles both lists individually. 
The intended effect was to obtain better anytime behaviour through randomization while keeping 
the preferred operators in the front of the list. Since it doesn't yield better results than 
the other three orderings we can probably remove the shuffled_pref_first ordering again. What 
do you think?

Review: I opened a pull-request at 
https://bitbucket.org/jendrikseipp/downward/pull-request/16

Experiments: I have "fixed" and extended the original experiment script and also added two 
other experiments for the unitcost and non-unitcost versions of LAMA (iterated). The results 
are attached.

Pushed to ssh://hg@bitbucket.org/malte/downward.

Can you take over the issue? The main thing to do is to fix the experiment
script. (The problem with the current one will be obvious from looking at it.)
Also, if we want to retain this options, we should probably set a TODO somewhere
that a similar option ought to exist in the eager search code.

Could you please make the branch for this issue available somewhere? I would like 
to test the influence of the operator order on the performance of the uniform 
portfolio.

PS: in a more modular implementation, this functionality could be implemented
with the same mechanism as partial-order reduction. In both cases, we take the
vector of applicable operators and return a new modified vector. (This vector
may be pruned, reordered, or both.) This could also be used to implement pruning
based on preferred operators.

Already done and evaluated, but the experiment isn't clean enough yet, so I'll
wait a bit until posting results unless someone is in a hurry. The short summary
is that "pref_first" has highest coverage with LAMA-2011 overall, but "original"
is also better in quite a few domains. "shuffled" tends to be worse.

I can expose this option to the command line, but I don't have the resources to 
run experiments. Maybe someone in Basel can do that?

The lazy search algorithm has an internal option "succ_mode" for influencing the
order in which applicable operators are considered:

- pref_first: in the order generated by the successor generator, except
  that all preferred operators come before all non-preferred operators
- original: in the order generated by the successor generator
- shuffled: in random order

This option is currently set to "pref_first" and cannot be modified from the
command line. It should be settable from the command line, and the various
choices should be evaluated.