Issue1165

Title Get rid of assumption that landmarks don't overlap outside of factories.
Priority wish Status resolved
Superseder Nosy List clemens, jendrik, malte, salome
Assigned To Keywords
Optional summary
Part of issue257.

Created on 2025-01-22.17:14:22 by clemens, last changed by clemens.

Summary
Part of issue257.
Messages
msg11766 (view) Author: clemens Date: 2025-02-04.08:31:45
I've moved the experiments linked in msg11745 to a more permanent place: https://ai.dmi.unibas.ch/_experiments/ai/downward/issue1165/
msg11754 (view) Author: clemens Date: 2025-01-30.12:11:54
I've merged the changes, thanks for your help everyone!
msg11751 (view) Author: clemens Date: 2025-01-29.15:39:42
Yes exactly, I omitted h^m because anything we do regarding preferred operators should not have much effect on them. Sure, with m=2 there are also some simple landmarks for which we could theoretically compute preferred operators, but as far as I know we don't usually do that, so I didn't test it. Would you like to see data of this kind? I can set up another experiment if you think that makes a difference.

The new code doesn't change anything regarding the semantics of when operators are preferred. We essentially just precompute which landmarks contain a certain atom -- information that was previously gathered from the landmark graph which keeps track of this because it makes sure that landmarks don't overlap. Since we want to move towards overlapping landmarks, I want to get rid of the assumption that we can get this information from the landmark graph.
msg11750 (view) Author: malte Date: 2025-01-29.14:43:24
For the experiments, is there a reason not to include the h^m landmarks? (Maybe I overlooked something.) The h^2 landmarks in particular tend to stretch the time and memory resources of landmark computation and landmark graph storage the most. Are they not considered because they can't currently be used for preferred operators?

I don't think I know enough about the context of this to really comment meaningfully. 

Based just on the messages below:

Whether an atom achieves a landmark in general depends on the state. For atom landmarks and disjunctive landmarks, we know which atoms make sure the formula is true, but not whether they transition from false to true. For conjunctive landmarks, we don't even know if a landmark will be true after achieving an atom without knowing the context.

With conditional effects, there are additional complications regarding whether an atom is true after applying an action because the effects of the actions are context-dependent.

If the new code isn't more problematic regarding these points than the old code, feel free to merge.
msg11749 (view) Author: salome Date: 2025-01-29.11:54:43
I looked at the pull request and the results. Aside from the minor remarks I made in the pull request all seems good to me.
msg11748 (view) Author: clemens Date: 2025-01-24.13:25:52
I have now also opened the pull request. It would be great if somebody could find the time to take a look! https://github.com/aibasel/downward/pull/242
msg11747 (view) Author: clemens Date: 2025-01-24.12:44:23
I have now also implemented (2) and it looks to me that it doesn't have a significant impact on performance (as expected):
https://ai.dmi.unibas.ch/_experiments/ai/downward/issue1165/data/issue1165-v1-opt-cegar-eval/issue1165-v1-opt-cegar-issue1165-base-issue1165-v1-compare.html
msg11745 (view) Author: clemens Date: 2025-01-22.17:14:22
In issue257 we want to allow overlapping landmarks. Many parts of the code assume that each atom (aka FactPair) occurs in at most one landmark. Landmark factories make sure this is the case (unless we use conjunctive landmarks in lm_hm), but it is an unreasonable restriction on this level. If a heuristic thinks it benefits from having non-overlapping landmarks, it should deal with this on its own, because other heuristics might benefit from overlaps instead (see for example Büchner, Christen and Eriksson, SoCS 2024).

While landmark factories are the main place where this assumption is used, it is not the only place. To make things a little bit easier to digest, I suggest to deal with all places outside the landmark factories first and get rid of this assumption there. After this is done, we can get back to the bigger issue of landmark factories using this assumption.

So this issue wants to reimplement all parts of the code that currently use that at most one landmark exists for a given atom. I've identified two places where this happens:
(1) In landmark heuristics to compute preferred operators, LandmarkHeuristic::generate_preferred_operators(...) to be precise.
(2) In the code for Cartesian abstractions to diversify abstraction collections based on landmarks.

For (1), I already have a suggestion for a work-around. Instead of computing on demand whether a certain fact achieves a future landmark, we precompute which landmarks are achieved by any given fact (could be more than one in theory, but with the assumption underlying the issue it's at most one for now) and checking whether any of them is future during search.

In an experiment, I've compared the difference between the two implementations and it seems minor to me. It includes reference runs where no preferred operators are used and the difference between the two code versions seems to be bigger in these reference runs (even though the implementation makes no difference for those) than in the runs that actually underly changes, so maybe everything is just noise again. In the lama-first-pref runs, there's not much of a change at all, except a slight decrease (~2) in time_scores (maybe due to the precomputation). An even larger decrease (~3) occurs in the reference runs lama-first, though. In contrast, the time_scores improve by ~6 for the lm_zg-pref runs and stay the same in its reference lm_zg. In the lama anytime configuration, the costs are slightly worse over all, but looking at the domain-wise comparison, it is sometimes better and sometimes worse. Again, the reference runs yield a larger decrease in performance regarding the final costs.

Full details can be found here:
https://ai.dmi.unibas.ch/_experiments/buechner/downward/untangle-lm-prefops/data/untangle-lm-prefops-v1-sat-eval/untangle-lm-prefops-v1-sat-3c8a65b5d-a2d652541-compare.html
https://ai.dmi.unibas.ch/_experiments/buechner/downward/untangle-lm-prefops/data/untangle-lm-prefops-v1-anytime-eval/untangle-lm-prefops-v1-anytime-3c8a65b5d-a2d652541-compare.html

Regarding (2), since I'm not really familiar with that code, I've had a brief discussion with Jendrik. It seems that the assumption is not at all important there and it should be easy to get rid of it. I'll try to do so now. @Jendrik: Do you have a suggestion which search configuration I should run to make sure nothing bad happens in this change?

Before I open a pull request, I would be interested to hear whether the results I already have for (1) sound concerning to anybody. If not, and it actually turns out that (2) is easy to fix, I hope we can soon move on from this issue.
History
Date User Action Args
2025-02-04 08:31:45clemenssetmessages: + msg11766
2025-01-30 12:11:54clemenssetstatus: in-progress -> resolved
messages: + msg11754
2025-01-29 15:39:43clemenssetmessages: + msg11751
2025-01-29 14:43:24maltesetmessages: + msg11750
2025-01-29 11:54:43salomesetmessages: + msg11749
2025-01-24 13:25:52clemenssetmessages: + msg11748
2025-01-24 12:44:23clemenssetmessages: + msg11747
2025-01-22 17:14:22clemenscreate