Thanks, I merged and pushed. I'm closing this issue but if any problems show up during the release that is planned for next week, we can reopen it or open a new one depending on the problem.

Looks good to me.

I generated all 4 VMs with vagrant ({19.06, 19.12} x {original files, new files modified to pull from the test repo}). I then mounted the disks of the new and old version and ran meld on them.

For 19.06 the differences were:
* mercurial/git/ca-certificates installation
* autogenerated ssh keys
* ssl certificates
* some timestamps
* machine ID
* apt package cache, dpkg status
* a copy of the vagrant script used to initialize the VM
* In the Fast Downward directory, the build logs have one difference in the name of a temporary directory created by CMake but are otherwise identical. In particular, all of the *.o files are identical. The created downward binary is different, though. I assume this is caused by different paths and timestamps. There are no other differences besides the obvious stuff ignored by meld (.hg vs .git, .hgtags, .hgignore).

On 19.12 the differences are similar but also show that the old VM installed both Python 2 and Python 3, while the new one only installed Python 3. My guess is that Python 2 was installed as a dependency of Mercurial. I tried running the planner in the new VM and saw no problem.

I think all of this is fine and the issue is ready to merge now. Do you agree Malte?

Thanks, Guillem! I don't think those diffs are a concern.

I had a look at the release-workflow wiki page and at the PR, it all looks good to me. Perhaps the only very minor change I'd do to the 
wiki page is to separate the instructions for updating the changelog and those for updating the readme.md copyright years / list of 
contributors into two different bullet points for further emphasis on the latter, as they are not really related.

I'd to that myself but just saw my wiki account is disabled... I'll try to fix that with Malte, but in the meantime I think this is 
good to merge.

I ran for 19.12 as well, conclusions are the same. 
Just for the sake of completeness, my previous message was not fully precise, 
there's a diff in the "exports" of both images, but the differences concern only 
the names of the images I'm exporting (which get saved as some form of metadata), 
not the contents of the Docker layers.

Thanks, Guillem! Can you do the same for 19.12?

I can have a go at the Vagrantfile, but for what it's worth, complete binary reproducibility is less of a goal for these. They serve a different use case. But we should test that they work. :-)

I just built locally two different Docker images for the 19.06 release, one from the current recipes pulling 
from hg.fast-downward.ord, the other one from the new recipe by Florian, pulling from github.com. As Malte said,  
I couldn't push from github.com/aibasel/downward, of course, I did push from github.com/aibasel/downward-test-
repo: I think for our purposes this difference is not important.

Both images have the same image ID when you enlist them ("docker images"); I'd assume the image ID is some hash 
of the contents, so that's good news. I exported both images to a tarball ("docker save ..."), and there's no 
diff between both tarballs. Summing up, I'd be more confident now in re-publishing the Docker and Singularity 
recipes without re-publishing the binary images themselves. 

Could someone else test the vagrantfile?

I guess we don't need to republish the containers if we are confident they are the same, but I would be reluctant to put the recipes into the repository untested. For example, before your latest change, the 19.06 recipe wouldn't have worked because it didn't install Python 2, right?

So I'd be happier if they were tested in some form. If we can verify that the docker containers are the same before/after, that would be perfect. Otherwise, I think we should do some basic tests that they are "equivalent" before/after. I assume the Singularity recipes have not changed, or have they? (I don't think they use either hg or git.)

Similarly, I think we should test the Vagrantfiles if we change them.

Of course, there's a chicken-and-egg problem given that the recipes depend on the aibasel/downward github repository. So we might not be able to test this right now, but then I think we should do some tests after merging.

> Is it acceptable practice to change the release container recipes after the
> fact? If yes, I think we should also rebuild and republish these containers?

I don't know what the best practice is but the original recipes are no longer useful if hg.fast-downward.org is no longer reachable, so I think it makes sense to update them.

I would not expect any difference when rebuilding the containers as the only thing that changes is the place where the source code comes from. The differences in the installed packages (git + ca-certificates instead of mercurial) doesn't make a difference in Docker because we do a multi-stage build. The difference is limited to the first stage and the final stage only copies the planner binaries and Python files from the first stage, which should be identical. Singularity is build based on Docker, so there shouldn't be a problem there as well. And the vagrant VM is not something we distribute in compiled form, so we could not rebuild it.

Of course, I don't have a problem with rebuilding the containers anyway but I think only you have the passwords to do so.

I had a brief look and didn't spot any problems.

My previous comment replied to msg9495. Regarding the changed recipes for the historical releases, is it acceptable practice to change the release container recipes after the fact? If yes, I think we should also rebuild and republish these containers?

I'm not bothered by these two points, but thanks for pointing them out.

> I assume the existing recipes are all copied verbatim from the other repository?

No, they are created from the updated templates, so they pull from github instead of hg.fast-downward.org. Now that I looked at the differences again, I saw that we changed the template between 19.06 and 19.12 to use Python 3 instead of Python 2. I'll change that back to Python 2 for the 19.06 release.

One thing to note is that the conversion script turns our two existing releases into option B (msg9383), while our workflow uses option D (msg9384). I think this is fine, because with bugfix releases, the release branch would diverge from main anyway.

Also, for our existing releases, the commits only create the tags and change the version numbers. The actual recipe files for the old releases are added by the pull request, so they come later in the history. The downside of this is that after updating to release-19.06.0 there are no recipe files in /misc/releases. I also think this is acceptable.

I did a superficial review; I'm happy to defer testing the nuts and bolts when we do the actual release. (And I see you already did some testing.) I assume the existing recipes are all copied verbatim from the other repository? Here they of course just show up as added code.

If Jendrik and/or Guillem sign this off, I'm happy to see it merged.

I created a pull request for this on Github:
https://github.com/aibasel/florian-downward/pull/1/files

The updated workflow is on the wiki
http://www.fast-downward.org/ForDevelopers/ReleaseWorkflow

I think this is ready for review.

I moved the notes to issue975 so they don't get lost.

> We are still considering moving more information from the wiki to github later
> on. Some problems that currently stopped us from embracing github releases more
> were:

Two more:

- github's tarballs/zip files are automatically generated as a snapshot of the tagged version, but we would like to filter out things like the experiments directory. Marking things as excluded for release appears to be possible in principle in git, and github would presumably honor this configuration, but we haven't looked into it yet.

- It would be good not to have two separate authoritative sources for our releases, so depending on how completely we move the release info over to github, we need to do some work to migrate the existing releases.

If anybody is interested in pushing the suggestions of moving releases to github, I suggest opening an issue and collecting these points there.


BTW, a possible solution to the 19.06.0 issue is to have the tag be named 19.06. We said this would clash with the branch name, and it does. But we could use separate naming conventions for release branches and released version tags, for example "release-19.06" for the branch and "v19.06" for the tag. (Just an example of a possible convention.)

After some offline discussion, we decided to use the "releases" feature on github to some degree. For now, we want to have a very short description of the release that consists mostly of a link to the release's page on our wiki (e.g., to http://www.fast-downward.org/Releases/19.12).

We are still considering moving more information from the wiki to github later on. Some problems that currently stopped us from embracing github releases more were:
- The releases page is cluttered with all non-release tags.
- The automatically generated tarball is named after the tag, which does not match
  our convention for normal (non-bugfix) releases: it uses 19.06.0 instead of 19.06.
- Having the data on github instead of our servers makes us more reliant on them.
- Having the full description of a release on the release page means that only one
  the latest release is visible (the next is on page 2).

I think the same can be said for most other issues related to the hg -> git conversion, like the hg cleanup issue, the git conversion issue and others. (Perhaps some of these *were* marker urgent/critical, I actually changed some issues this week.)

It's been a long time since we discussed these priorities, but basically the idea was that urgent or critical issues would be ones that prevent people from using the planner right now, such as a broken compile in the current release. An issue that can wait in principle until the next release wouldn't be urgent or critical under this rule (even if the next release is scheduled to happen soon). We can rediscuss what the priorities mean, of course.

My reasoning for this was that as long as we do nothing about this issue, we cannot publish Fast Downward at the place where we decided it should live. I'm fine with changing the priority.

I don't agree with the definition of "critical", perhaps we can discuss this in one of the post-morning meetings.

If it helps simplify things, we are all ears. :-)

Only mildly related to this issue, but it came up to my mind that now it is relatively straightforward (easy as 
in "change one line in the Singularity recipe") to build the singularity image from the local Docker registry 
instead of the docker hub, so that we could build both images locally with no dependency wrt the hub.
Of course then we'll want to push the image to the hub anyway, so this is not a huge improvement, but still 
looks slightly more robust and efficient to build everything locally. I could look into this today as well if 
necessary.

I like option D in particular, because then everything directly related to the release is in the commit that is also tagged as the release. That also works nicely with './fast-downward.py --version'. I'll look into updating the release scripts tomorrow.

As far as I know, the automatic builds of singularity hub haven't worked for any of our past releases, so sacrificing them doesn't sound like a big deal.

I prefer the msg9383 tree (option B) over the msg9382 tree (option A). What about these simpler variants:

Option C:

* (4) Update version number to 19.12+.
* (3) Update version number to 19.12.
      (tagged release-19.12.0 and tip of branch release-19.12)
* (2) Commit container recipes for 19.12
* (1) older commits on default

Same as yours, but with a linear history.


Option D:

* (3) Update version number to 19.12+.
* (2) Commit container recipes for 19.12 and update version number to 19.12.
      (tagged release-19.12.0 and tip of branch release-19.12)
* (1) older commits on default

AKA the simplest possible thing.


One (small?) advantage of options C and D is that the release versions are also covered by our continuous integration (currently buildbot), which AFAIK doesn't test side branches.

Maybe an alternative would be to switch off the automatic builds of singularityhub and create this tree:

* (M3'') Update version number to 19.12+.
|
| * (R1'') Update version number to 19.12.
| |       (tagged release-19.12.0 and tip of branch release-19.12)
|/
* (M2'') Commit container recipes for 19.12
* (M1'') older commits on default

One thing I noticed while looking at the converted repository is the way the release branches look like. Our Mercurial workflow generates the following revision tree (for example release 19.12):

* (M2) Update version number to 19.12+.
|
| * (R3) Tag (R2) as release-19.12.0 (tip of open branch release-19.12)
| * (R2) Update version number to 19.12. (tagged release-19.12.0)
| * (R1) Create branch release-19.12.
|/
* (M1) older commits on default
* ...

The conversion script skips the empty commit (R1) that just creates the branch, which is fine, I think. But it keeps the commit (R3) that does nothing. For future releases, would we have just a single commit on the release branch? 

And where would the commit for the container files go? The dependency is a bit tricky there: the docker recipe pulls from the official repository and expects the release tag to exist, so we have to publish the release tag before building the docker container. The singularity script is built automatically when we push the recipe and it expects the docker image to be at dockerhub.

The workflow that mirrors our old one would end up with this tree. We'd push after creating R1' and M2', then build the docker image, publish it, then add all container files and push again.

* (M3') Commit container recipes for 19.12
* (M2') Update version number to 19.12+.
|
| * (R1') Update version number to 19.12.
| |       (tagged release-19.12.0 and tip of branch release-19.12)
|/
* (M1') older commits on default
* ...

The downside would be that the branch 19.12. would not contain the recipe files in misc/releases/19.12 because they are not added in this branch.

we decided to rename misc/release to misc/releases and have the previous file structure in that directory (e.g., misc/releases/latest/Singularity).

The scripts have to be adapted to this setting and we have to find a new place for the information in the README.