This is part of issue727.

We want to allow all evaluators to subscribe to notifications about the initial 
state and state transitions.

Changes merged.

Looks good to merge.

Good idea. I added it to the agenda.

Yes, the memory usage is in kB. The results look good to me. Small changes in
memory usage like the ones in this experiment are normal.

I was surprised to see out-of-memory warnings from slurm in the saticficing
experiment though. It looks like some jobs used close to our slurm limit of 4GB
before they were killed. This should never happen since we set a ulimit of 2GB.
This could be problematic since the memory limit of slurm doesn't always work
and will sometimes not be enforced. The tasks that use over 4GB could then take
away memory from other tasks on the same node. But I guess this issue is not the
right place to discuss this. Jendrik, should we add this to the agenda for the
next meeting?

I have (finally!) uploaded some experiment results here:

http://ai.cs.unibas.ch/_tmp_files/frances/issue724-v1-sat-issue724-base-issue724-v1-compare.html
http://ai.cs.unibas.ch/_tmp_files/frances/issue724-v1-opt-issue724-base-issue724-v1-compare.html

They include configurations for:
* lama-first
* ehc(ff)
* A* + blind heur.
* A* + lm-cut
* seq-opt-bjolp

The only significant differences seem to be in memory usage (which I assume are given in KB).
I don't have a clear intuition of the source of these differences, although they are perhaps small enough to ignore them?

Thanks Malte, sounds good. I have drafted an experiments setup for this, but 
probably best will be to steal yet 5 more minutes tomorrow to Jendrik/Florian to 
check everything is correct, and then we're good to run the experiments.

I'm done commenting on bitbucket. There are some issues to do with heuristic
caching and path-dependent heuristics that we should discuss (also with Salomé),
but I think we are ready for experiments either way.

I think the experiments should exercise all search algorithms that were changed
and both path-dependent heuristics ("regular" lmcount and synergy lmcount). If
we include lama-first, lama and seq-opt-bjolp, we have coverage of all
heuristics, I think. Choose whatever else makes sense to get coverage of all
search algorithms (e.g. something like EHC with the FF heuristic).

I'm happy with the code now :-) If you want, you could already prepare the Lab 
tests until Malte has time to review the code. One configuration that we should 
definitely test is lama-first.

I left some comments on bitbucket.

Sure, I thought the repo was public, should be now.

I'll have a look at the code. Could you please give the group access to your 
repository?

Incidentally, I was going to tackle the small issue mentioned in the "TODO" 
here:

https://bitbucket.org/gfrances/downward/pull-
requests/1/issue724/diff#Lsrc/search/evaluator.hT43

namely, that `notify_state_transition` returns a bool which is not used 
anywhere (I confirmed this). Would you rather open a new issue / deal with this 
separately?

I have created a first PR with the changes we discussed in person here:

https://bitbucket.org/gfrances/downward/pull-requests/1/issue724/diff

Added a cross reference to issue727.

My suggested answers to Manuel's questions: we should have a registration
mechanism similar to the current "get_involved_heuristics" method (but with a
new name) that recurses through the open lists and evaluators in order to let
them subscribe to notifications about the initial state and transitions.

This registration mechanism should make sure that the same evaluator is only
asked once whether it wants to register, even if it is reached multiple times.
(The same should be true for open lists, but currently it's not possible to have
an open list be used in multiple contexts.)

All evaluators should be able to subscribe, not just heuristics. In particular,
this mechanism might turn out to be useful for implementing g values and "real g
values".

> If you look into use cases you could also check if we can use an OperatorID,
> OperatorProxy, or just remove the parameter.

Even if we don't currently use the operator parameter, I don't think we should
remove it. There are path-dependent heuristics other than the ones used in our
current codebase (for example in Salomé's LTL code), and it doesn't seem
unreasonable that a path-dependent heuristic should receive information about
operators associated with transitions.

A related thing we should look at is what kind of information the notification
should provide. In issue725 I tried to reduce the number of GlobalOperator* and
notify_state_transition is one of them. It currently takes two GlobalStates and
one GlobalOperator*. But the GlobalOperator* is never used (only forwarded). If
you look into use cases you could also check if we can use an OperatorID,
OperatorProxy, or just remove the parameter.

A path dependent heuristic can subscribe itself to be notified about initial
state and state transitions that are reached during search.

Currently, FD uses subscriptions and notifications as follows:
- All evaluators can subscribe but only heuristics will be notified.
- All heuristics subscribe themselves but not each heuristic requires a
notification.

The current use of subscriptions and notifications raises following questions:
- Should evaluators also be notified?
- If yes, should an evaluator notify its sub-evaluators or should sub-evaluators
be directly notified, independent of the evaluator (e.g. SumEvaluator)?
- Which evaluators/heuristics require notification?

We want to evaluate different use cases (e.g. consider notification under
reopening of states) and look at their requirements. Based on those insights, we
wish to define which evaluators can subscribe, how they subscribe, and how they
are notified about the visited search space (e.g. transitions etc.).