Related issues:
* issue526 (consistent use of shared pointers for plugins)
* issue740 (configurable output file of translator)
* issue509 (introduces the task class, in particular issue700
            will get rid of a lot of global variables)
* issue1081 (moving error handling to constructors)
* issue1082 (not using options in component constructors)

In the last sprint we worked on issue1081 and closed the last related issue of 
this one.
The next step would be to rebase the prototype branch on the current main.

The next steps we want to take are the following ones:
- There are a lot of changes in the diff of the pull request that origin from a merge which mostly removed the options objects, which were taken by many features in their constructors, and replaced them by the objects acutally needed. The keep the diff small, the part of that diff that doesn't contribute to the prototype should be removed in the short term. In the long term we need the features to behave and their constructors to look the same as described in the registry so that we can user the inofrmation to generate pybind11 code and export them all. At the moment there are many features that aren't exported because of that reason.
- Since there is no information about which member functions of the categories we want to export, we still need to manually provide that information. The current prototype takes it in the file src/search/pybindings/functions.txt. We want to change that and provide the information directly within the code e.g. in the category plugins.
- There are some hacks in the code that we want to take out or find a better solution for. These contain:
	- in the file src/search/pybindings/definitions_for_pybind11.h there is the function read_task
	- in the file src/search/tasks/root_task.h there is the function get_root_taskt
	- the question which types and functions we also need to export with the library besides the classes held by the registry and how we want to do that automatically. This will surely contain the return types of the member functions of the categories. Indeed we don't export all types held in the registry. The current prototype only supports the export of features and categories (and some other types). Related to this is the fact that we still extract information of the raw rergistry and it would be more preferable to extract it from the registry only. (Don't forget to remove the corresponding getter in the raw registry.)
- We need to find a solution for the exported type no_transform which is of type abstract task in the C++ side and which is intended to provide the task which was read out of the .sas file, but cannot be exported yet because it is abstract.
- All automatically generated files, which include at the moment the pybind11 code and a list of the headers that have to be included in the pybind11 code, should be automatically removed after the build or be placed in the build folder so that they are ignored by git.
- The class Type and its derived classes (e.g. EnumType) have the member function get_fully_qualified_name which is defined in a useful way only for some of the derived classes. This needs to be done and with it the member variable std::string fully_qualified_name should be added to Type and the function Feature::get_constructor_signature should be adapted. The functions set_fully_qualified_name that are members of EnumType and FeatureType should be revomed and the fully_qualified_name should be provided directly in the constructor.
- We want to add the documentation of the exported types. The information for that are also found in the registry.
- Currently the two builds are done independently by invoking two times cmake and make. We want to merge both such that we need to call them both once. We want to do that by having only one main method which, depending on the flags set, proceeds as usual or generates the pybind11 code by invoking a function which gets the location of the generated pybind11 code as argument.
	- Currently in the first build, cmake writes down all the files that are included in the compilation and that are stored in ${PLANNER_SOURCES} into a file and the executable then parses that file and extracts from that the includes for the pybind11 code. We want to find a better solution without writing down the content of ${PLANNER_SOURCES} and parsing it out again.
	- With add_custom_command() we want to add a custom command in cmake that depends on our project (in this case the target downward which was already created with add_executable()) and which outputs the generated pybind11 code. Then we can use this one again to create our library e.g. with pybind11_add_module().

The current prototype for the python library works as follows and is based on pybind11. Most of the information needed to create the library is being extracted from the registry class which is also used for the generation of the wiki. The library is made with two builds. The first build creates an executable which, when invoked, automaically generates pybind11 code using the mentionned information of the registry. In the second build the generated pybind11 file is compiled itself and the python library is created. We still cannot export all the features since their signature and their behaviour is not up to date with the information contained in the registry, which renders it not possible yet to automatically create the pybind11 code for them.

The current prototype can be found here: https://github.com/Emanuele-Tirendi/downward/tree/pybind11-code-generation. A diff to a not up to date version of the main branch, but in which one can easily see the important changes, can be seen in the following pull request: https://github.com/Emanuele-Tirendi/downward/pull/5/files#diff-c33a8df534cc282d3c38333b29abdf59232bdc3f774b72b149e8640019986006

I created issue1081 for moving the error handling into the constructors
and issue1082 for introducing the constructors with explicit arguments.

We started looking into a Python interface during the current sprint, so let me dig up this issue again.

Gabi prepared a prototype solution for a Python binding with pybind11 that supports running a search with a heuristic implemented in Python.
https://github.com/aibasel/downward/pull/152

We also discussed some points about object creation, in particular where the boundary between the option parser, the Python interface, and the library code should be. I'm copying some notes on the results of that discussion:

* Among other things, we discussed how to automate the manual steps in the pybind11 code that replicate information that is already present in the code (about variable names, types, default values, documentation). We said that such automation is not something we need to solve immediately; it's not a blocker in the sense that if we couldn't automate these things and needed to specify them multiple times, we would still want to do the C++/Python integration.

* But we also noticed that the introspection capabilities we already have for creating the wiki documentation should be adaptable with reasonable effort to do the bulk of the pybind11 binding code. Basically, in the same way that we can ask the planner to generate the wiki documentation, we could ask it to generate the pybind11 code.

* This also answers the question whether we want the "Python library mode" to be the only build mode or if we want to support others (e.g. C++ library or C++ executable). We need a C++ library or executable with this idea to generate the pybind11 code without having a bootstrapping problem.

* We want constructors without options objects in the builders and the library interface eventually. We discussed the option of having two constructors (one taking individual arguments, one taking an option object and delegating to the other) and the alternative solution of having only one constructor and unpacking the options object in the feature class. The current plan is to go with the second option and move towards the structure we have for some of the search engines where the feature and plugin code is in a file separate from the implementation.

* We also discussed how errors and warning were handled during the construction of builders: We want to move most/all of the error checking into the constructors, so we don't have to duplicate code for checking for objects created by the parser and objects created through the Python interface. This means we have to throw exceptions from the constructors and catch them in the parser, so we can report them with the parser context. Warnings during parsing do not fit well in this setup, but the two warnings we have so far are not that important and can be dropped.

I added the related issues to the summary (and added issue509) where it's easier
to keep track of them and possible to update the list.

Also, FYI: if you just write issueXYZ for an issue number XYZ, the tracker will
create links and tool tips for you.

Just for the (mostly my own) record, I'm listing here a few issues the closing of which will be good for this project. I'll keep updating this list:

* http://issues.fast-downward.org/issue526 (having shared_ptr used consistently in those objects that we want to expose to the Python module, such as the heuristic, would simplify memory ownership issues)
* http://issues.fast-downward.org/issue740 (not a big thing, but will make the Python module initialization code slightly cleaner)

Sure, could be helpful as a starting / comparison point.

We developed an interface from PSVN to Fast Downward as part of a project once.
In that interface, PSVN used Fast Downward as a library. It wasn't super
efficient because the changes to the task class were not ready then.

But if it helps I can dig up the source code and the documentation from the project.

(Moved from summary field)

We'd like to offer the user the possibility of interacting "programmatically" 
with the planer. This is a high-level issue / wishlist of features that could 
be 
related to this goal, and which could include, for instance,

(1) Offering a Python interface (i.e. creating one or more Python modules with 
well-designed interfaces that bind with the C++ library) for creating planning 
problems programmatically. 

(2) Offering a Python / C++ interface for easily specifying the options of the 
different plugins, in a way alleviating the need for complex command-line 
option 
parsing.  

(3) Offering the possibility of subclassing certain interfaces in order to 
specify (in a possibly domain-dependent manner) the desired search behaviour, 
or 
additional control information.

(4) Offering the possibility of using symbols whose denotation is defined 
through 
C++ / Python functions.

(5) ...


Good examples/inspiration for some/all of the above points could include for 
instance:
(A) The Gecode CSP solver (http://www.gecode.org/doc-latest/MPG.pdf), which 
beyond the ability to process problems specified in declarative languages such 
as 
flatzinc, offers different ways to declare CSP problems by extending the 
appropriate C++ classes, etc., as well as to control the solving process.

(B) The z3 SMT solver, which offers an elegant Python API to declare SMT 
problems, instantiate solvers with different options, and solve the problems 
(simple example: 
https://github.com/Z3Prover/z3/blob/master/examples/python/socrates.py)