TODO:

* replace "python" by "python3" in shebang lines (excluding experiments)
* remove compatibility imports for Python 2.7
* Choose minimum supported Python 3 version
* Update public and internal documentation
* Make entry for CHANGES.md
* Add an experiment with python3 and discuss in experiments README
* Simplify shell-escaping in driver/util.py
* Release scripts need to be tested
* Remove buildbot for Ubuntu 16.04

Yes, unfortunately only 2 Haslums though :-)

Awesome, thanks, Cedric! (And of course everyone else that helped!) So that's some Haslums earned?

The issue branch is merged and all the buildbots are working (again).

Updated the CHANGES.md according to your suggestions.

Thanks! I don't think it covers enough at the moment. We don't just drop support for Python 2 but also for Python 3.2-3.5. And I think a change log entry for a change of this kind should be a bit more constructive (what *do* we support?) rather than just saying what breaks. I would also include a rationale for breaking things. Remember that the change log is our way of providing release notes.

Something like "Raise minimum supported Python version to 3.6" as the basic description and then the longer text:

Fast Downward now requires Python 3.6 or newer; support for Python 2.7 and Python 3.2-3.5 has been dropped. The main reason for this change is Python 2 reaching its end of support on January 1, 2020. See https://python3statement.org/ for more background.

Updated TODOs.

Also added en entry for CHANGES.md and enforce the use of Python 3 in the experiment for issue899.

> I think the experiment already compares Python 2.7 (base) against Python 3.6 (v1).

This is the case. I ran 2 separate experiments and then fetched them together.

Please ignore my last comment. I forgot that Cedric fetched the results from two experiments.

Judging by the logs however, I think only Python 3 was run.

That would be good but wasn't clear to me. (Doesn't a single lab experiment always use a single Python version because lab uses its own Python version for Fast Downward?)

I think the experiment already compares Python 2.7 (base) against Python 3.6 (v1).

Experimental results look good, but I'd be even more interested in a comparison between the different Python versions, i.e., the one we use currently and the one we will use after this issue. Or did we already use Python 3.6 before?

If it's difficult to have them in the same experiment, I don't think it's a problem to compare results from two different experiments in this case (and hope that grid noise won't be too dissimilar).

Installing Python from source is easy, but the way I interpreted our discussion was that by requiring Python 3.6 we effectively drop support for 16.04. So I would go with removing the buildbot, but I don't feel strongly about it.

If we consider it our guideline to support the two most recent LTS versions, we are only predating the abandoning of 16.04 by a few months. I don't think this is critical. People can use Fast Downward 19.06, it is not that we introduced revolutionary changes since then. (Soplex, but people who want to use LPs in Fast Downward need a much higher pain threshold anyway.)

Changing the python version to 3.6 is a problem for the ubuntu 16.04 buildbot. Since xenial has no python 3.6. I think the simplest solution would be to remove the buildbot for xenial, but this clashes with the policy that we want to support the 2 latest LTS version for ubuntu. I did an offline poll for this and 4 people support this solution and 2 people think we should keep the buildbot.

@Malte: What is your opinion? We could also update the python version on the xenial buildbot to 3.6, but since this is not in an official repository, I don't think it makes sense to say that we support xenial.

Updated TODOs.

I checked the release scripts and they seam to work.

Here are the experiments for the translator:

https://ai.dmi.unibas.ch/_tmp_files/geissmann/issue939-absolutereport.html
https://ai.dmi.unibas.ch/_tmp_files/geissmann/issue939-comparativereport.html
https://ai.dmi.unibas.ch/_tmp_files/geissmann/issue939-different_output_sas.csv

The base version runs with python2.7 and the v1 version runs with python3.6.

The unexplained errors come from lab. Version 5.0 of lab should fix this and I re-run the experiments, so I'm not sure why this error persisted.

Updated TODOs.

I addressed the comment on the pull request.

We preferred raising the minimum version to 3.6 now rather than to 3.5 now and to 3.6 in four months.

> After some offline polling and discussion, we decided to use Python 3.6
> as the minimum supported Python version.

What was the reason to go with 3.6 over 3.5? When we talked you said 3.5 was the
default version on the oldest Ubuntu version we still support.

> Does lab's Python version affect the Python version that is used to run Fast Downward?

Yes, it does.

After some offline polling and discussion, we decided to use Python 3.6 as the minimum supported Python version. I updated the public docs accordingly.

I addressed all of the things in msg9054 in second part, except shell-escaping.

Added unaddressed items to TODOs.

I'm done with my bitbucket comments.

Some additional general comments:

- Some work may be needed on the buildbot side.

- Buildbot changes need to be tested.

- Release scripts need to be tested.

- I am not confident about the changes related to caught exceptions, and I think we basically don't test these. Not sure what to do about this, but wanted to mention it.

- I think we want a before/after translator experiment that looks at whether the output.sas files have changed and at runtime and memory. Apologies if that is a hassle. (But then, it would be great for the future if it weren't a hassle. ;-))

- We don't want to convert the old experiments, but I think it would be nice to have at least one python3 experiments and discuss the issue in the experiments README, with a pointer to a python3 experiment. Does lab's Python version affect the Python version that is used to run Fast Downward?

I started reviewing this a bit, not just by looking at the diff but also by grepping through the code for things that look like they might need changing.

Firstly, we need an entry for CHANGES.md.

Secondly, these are some things that look like they need changing:

./src/translate/tests/test_scripts.py:        assert subprocess.check_call(["python", filename, DOMAIN, PROBLEM], cwd=folder) == 0

=> python3

./src/translate/translate.py:def python_version_supported():
./src/translate/translate.py:if not python_version_supported():
./src/translate/translate.py:    sys.exit("Error: Translator only supports Python >= 2.7 and Python >= 3.2.")

=> function python_version_supported and its error message need updating

./driver/util.py:# Shell-escaping code taken from Python's shlex.quote (missing in Python < 3.3).

=> This sounds like it can be simplified in Python >= 3.3.

./driver/run_components.py:# TODO: We might want to turn translate into a module and call it with "python -m translate".

=> python3

./misc/tests/test-translator.py:    env["PYTHONHASHSEED"] = "random"

=> This is the default (now). Are we worried about this being set
externally? I wouldn't be. Who controls the external environment can
already pretty much do what they want.

./misc/tests/test-translator.py:            r".* command line string: .*/python\d(.\d)?"]:

=> Change this to only match python3 versions?

./misc/tox.ini:  bash -c "python misc/autodoc/autodoc.py --dry-run > /dev/null"

=> Better to use python3 here ("explicit is better than implicit")?

./misc/release/templates/_Vagrantfile.tpl:        python
./misc/release/templates/_Dockerfile.tpl:    python
./misc/release/templates/_Dockerfile.tpl:    python  \

=> These three need to be "python3" or perhaps a more specific version.

> I don't think we should support an end-of-life Python version like 3.4.

I am not convinced. Supporting RHEL is a benefit. As far as I know, they do their own maintenance past the usual end of life. I've had several past scenarios (the grid I had access to in Freiburg, the one I ran IPC 2008 on) where not supporting a current RHEL would have been a deal-breaker for running the planner. I'm not saying we should absolutely support it, but I would prefer to support it if the trade-off regarding Python features is acceptable.

Python 3.6 introduced f-strings (https://docs.python.org/3/whatsnew/3.6.html#pep-498-formatted-string-literals), which I really like. Other than that, I don't think Python 3.6 introduced anything terribly useful for us.

I don't think we should support an end-of-life Python version like 3.4. Also,  supporting all officially supported Python versions sounds like a good idea to me.

Update TODOs.

Replaced shebang lines.

Python 3.6 is the maximum requirement I would consider because it's Ubuntu 18.04's default python3 version.

But I think Python 3.6 may be problematic for some some people. I get Python 3.4 on the grid when I run python3. I realize I can probably get a different version in some way (with an appropriate module), but I've often been on evaluation platforms where going further than the default was very difficult. Computational grids etc. often run things like RHEL which move very slowly. Do we get much benefit from requiring Python 3.6 instead? What specifically?

I see that Python 3.4 has reached its end of life, and it looks like the oldest Python version still within its end of life is Python 3.5. I see this as an argument in favour of 3.5.

In summary, the versions I'd consider are 3.4, 3.5 and 3.6. I would be in favour of the oldest one if we don't have to sacrifice anything significant for it. If not, I'd perhaps be in favour of 3.5 because that's still within its service life (if I read correctly).

Before we can update the public documentation, we need to decide on the minimum supported Python version. I think we should require Python >= 3.6.

Internal docs are updated.

> The driver calls the translator with the interpreter that was used
> to call the  driver.

Ah, I was surprised to hear that, but the code agrees with you. :-)

Update TODOs.

The driver calls the translator with the interpreter that was used to call the driver.

I updated the summary with your TODOs and some additional ones.

Correct me if I'm wrong, but won't this still use Python 2.7 for the translator?

Besides this, I think if we change to Python 3, we should also do this for "our" parts of the repository, not just the public scripts. Otherwise, we rely on unsupported Python on our side, which isn't great either.

As a first test, I tried:

    grep '#.*python' **/*.py

which found many things ("**" requires certain shell features to be enabled to recurse into all subdirectories).

We don't need to change the occurrences in "experiments" (but should make sure to use python3 there in the future, too), but I would closely look at all others hits.

We should also look at the documentation, including our internal documentation of how to set up the grid on the DMI wiki (if that still lives there). Also buildbot setup etc.

Here is the pull request https://bitbucket.org/cgeissmann/downward/pull-requests/11

Because python2 is no longer supported in 2020 (https://www.python.org/doc/sunset-python-2/), we would like to drop support for python2 as well and change to python3 instead.

The plan is to change the executable files to use python3 as well as the tests.

@Malte: We discussed this before in the sprint meeting, is it okay for you like this? Do you have any remarks?