Issue974

Title Merge the repository for container recipes into the main repository
Priority feature Status resolved
Superseder Nosy List florian, guillem, jendrik, malte
Assigned To florian Keywords infrastructure
Optional summary

Created on 2020-07-01.12:38:22 by florian, last changed by florian.

Messages
msg9522 (view) Author: florian Date: 2020-07-07.14:13:32
Thanks, I merged and pushed. I'm closing this issue but if any problems show up during the release that is planned for next week, we can reopen it or open a new one depending on the problem.
msg9521 (view) Author: malte Date: 2020-07-07.12:52:22
Looks good to me.
msg9520 (view) Author: florian Date: 2020-07-07.12:49:53
I generated all 4 VMs with vagrant ({19.06, 19.12} x {original files, new files modified to pull from the test repo}). I then mounted the disks of the new and old version and ran meld on them.

For 19.06 the differences were:
* mercurial/git/ca-certificates installation
* autogenerated ssh keys
* ssl certificates
* some timestamps
* machine ID
* apt package cache, dpkg status
* a copy of the vagrant script used to initialize the VM
* In the Fast Downward directory, the build logs have one difference in the name of a temporary directory created by CMake but are otherwise identical. In particular, all of the *.o files are identical. The created downward binary is different, though. I assume this is caused by different paths and timestamps. There are no other differences besides the obvious stuff ignored by meld (.hg vs .git, .hgtags, .hgignore).

On 19.12 the differences are similar but also show that the old VM installed both Python 2 and Python 3, while the new one only installed Python 3. My guess is that Python 2 was installed as a dependency of Mercurial. I tried running the planner in the new VM and saw no problem.

I think all of this is fine and the issue is ready to merge now. Do you agree Malte?
msg9517 (view) Author: malte Date: 2020-07-07.11:56:41
Thanks, Guillem! I don't think those diffs are a concern.
msg9516 (view) Author: guillem Date: 2020-07-07.11:46:30
I had a look at the release-workflow wiki page and at the PR, it all looks good to me. Perhaps the only very minor change I'd do to the 
wiki page is to separate the instructions for updating the changelog and those for updating the readme.md copyright years / list of 
contributors into two different bullet points for further emphasis on the latter, as they are not really related.

I'd to that myself but just saw my wiki account is disabled... I'll try to fix that with Malte, but in the meantime I think this is 
good to merge.
msg9514 (view) Author: guillem Date: 2020-07-07.10:38:43
I ran for 19.12 as well, conclusions are the same. 
Just for the sake of completeness, my previous message was not fully precise, 
there's a diff in the "exports" of both images, but the differences concern only 
the names of the images I'm exporting (which get saved as some form of metadata), 
not the contents of the Docker layers.
msg9513 (view) Author: malte Date: 2020-07-07.09:54:47
Thanks, Guillem! Can you do the same for 19.12?

I can have a go at the Vagrantfile, but for what it's worth, complete binary reproducibility is less of a goal for these. They serve a different use case. But we should test that they work. :-)
msg9512 (view) Author: guillem Date: 2020-07-07.09:48:39
I just built locally two different Docker images for the 19.06 release, one from the current recipes pulling 
from hg.fast-downward.ord, the other one from the new recipe by Florian, pulling from github.com. As Malte said,  
I couldn't push from github.com/aibasel/downward, of course, I did push from github.com/aibasel/downward-test-
repo: I think for our purposes this difference is not important.

Both images have the same image ID when you enlist them ("docker images"); I'd assume the image ID is some hash 
of the contents, so that's good news. I exported both images to a tarball ("docker save ..."), and there's no 
diff between both tarballs. Summing up, I'd be more confident now in re-publishing the Docker and Singularity 
recipes without re-publishing the binary images themselves. 

Could someone else test the vagrantfile?
msg9507 (view) Author: malte Date: 2020-07-06.22:18:10
I guess we don't need to republish the containers if we are confident they are the same, but I would be reluctant to put the recipes into the repository untested. For example, before your latest change, the 19.06 recipe wouldn't have worked because it didn't install Python 2, right?

So I'd be happier if they were tested in some form. If we can verify that the docker containers are the same before/after, that would be perfect. Otherwise, I think we should do some basic tests that they are "equivalent" before/after. I assume the Singularity recipes have not changed, or have they? (I don't think they use either hg or git.)

Similarly, I think we should test the Vagrantfiles if we change them.

Of course, there's a chicken-and-egg problem given that the recipes depend on the aibasel/downward github repository. So we might not be able to test this right now, but then I think we should do some tests after merging.
msg9503 (view) Author: florian Date: 2020-07-06.19:46:17
> Is it acceptable practice to change the release container recipes after the
> fact? If yes, I think we should also rebuild and republish these containers?

I don't know what the best practice is but the original recipes are no longer useful if hg.fast-downward.org is no longer reachable, so I think it makes sense to update them.

I would not expect any difference when rebuilding the containers as the only thing that changes is the place where the source code comes from. The differences in the installed packages (git + ca-certificates instead of mercurial) doesn't make a difference in Docker because we do a multi-stage build. The difference is limited to the first stage and the final stage only copies the planner binaries and Python files from the first stage, which should be identical. Singularity is build based on Docker, so there shouldn't be a problem there as well. And the vagrant VM is not something we distribute in compiled form, so we could not rebuild it.

Of course, I don't have a problem with rebuilding the containers anyway but I think only you have the passwords to do so.
msg9501 (view) Author: jendrik Date: 2020-07-06.19:31:07
I had a brief look and didn't spot any problems.
msg9498 (view) Author: malte Date: 2020-07-06.19:09:36
My previous comment replied to msg9495. Regarding the changed recipes for the historical releases, is it acceptable practice to change the release container recipes after the fact? If yes, I think we should also rebuild and republish these containers?
msg9497 (view) Author: malte Date: 2020-07-06.19:06:42
I'm not bothered by these two points, but thanks for pointing them out.
msg9496 (view) Author: florian Date: 2020-07-06.19:04:25
> I assume the existing recipes are all copied verbatim from the other repository?

No, they are created from the updated templates, so they pull from github instead of hg.fast-downward.org. Now that I looked at the differences again, I saw that we changed the template between 19.06 and 19.12 to use Python 3 instead of Python 2. I'll change that back to Python 2 for the 19.06 release.
msg9495 (view) Author: florian Date: 2020-07-06.18:58:23
One thing to note is that the conversion script turns our two existing releases into option B (msg9383), while our workflow uses option D (msg9384). I think this is fine, because with bugfix releases, the release branch would diverge from main anyway.

Also, for our existing releases, the commits only create the tags and change the version numbers. The actual recipe files for the old releases are added by the pull request, so they come later in the history. The downside of this is that after updating to release-19.06.0 there are no recipe files in /misc/releases. I also think this is acceptable.
msg9494 (view) Author: malte Date: 2020-07-06.18:55:46
I did a superficial review; I'm happy to defer testing the nuts and bolts when we do the actual release. (And I see you already did some testing.) I assume the existing recipes are all copied verbatim from the other repository? Here they of course just show up as added code.

If Jendrik and/or Guillem sign this off, I'm happy to see it merged.
msg9493 (view) Author: florian Date: 2020-07-06.18:43:15
I created a pull request for this on Github:
https://github.com/aibasel/florian-downward/pull/1/files

The updated workflow is on the wiki
http://www.fast-downward.org/ForDevelopers/ReleaseWorkflow

I think this is ready for review.
msg9487 (view) Author: jendrik Date: 2020-07-06.17:01:56
I moved the notes to issue975 so they don't get lost.
msg9483 (view) Author: malte Date: 2020-07-06.16:15:00
> We are still considering moving more information from the wiki to github later
> on. Some problems that currently stopped us from embracing github releases more
> were:

Two more:

- github's tarballs/zip files are automatically generated as a snapshot of the tagged version, but we would like to filter out things like the experiments directory. Marking things as excluded for release appears to be possible in principle in git, and github would presumably honor this configuration, but we haven't looked into it yet.

- It would be good not to have two separate authoritative sources for our releases, so depending on how completely we move the release info over to github, we need to do some work to migrate the existing releases.

If anybody is interested in pushing the suggestions of moving releases to github, I suggest opening an issue and collecting these points there.


BTW, a possible solution to the 19.06.0 issue is to have the tag be named 19.06. We said this would clash with the branch name, and it does. But we could use separate naming conventions for release branches and released version tags, for example "release-19.06" for the branch and "v19.06" for the tag. (Just an example of a possible convention.)
msg9482 (view) Author: florian Date: 2020-07-06.15:47:44
After some offline discussion, we decided to use the "releases" feature on github to some degree. For now, we want to have a very short description of the release that consists mostly of a link to the release's page on our wiki (e.g., to http://www.fast-downward.org/Releases/19.12).

We are still considering moving more information from the wiki to github later on. Some problems that currently stopped us from embracing github releases more were:
- The releases page is cluttered with all non-release tags.
- The automatically generated tarball is named after the tag, which does not match
  our convention for normal (non-bugfix) releases: it uses 19.06.0 instead of 19.06.
- Having the data on github instead of our servers makes us more reliant on them.
- Having the full description of a release on the release page means that only one
  the latest release is visible (the next is on page 2).
msg9427 (view) Author: malte Date: 2020-07-03.00:39:50
I think the same can be said for most other issues related to the hg -> git conversion, like the hg cleanup issue, the git conversion issue and others. (Perhaps some of these *were* marker urgent/critical, I actually changed some issues this week.)

It's been a long time since we discussed these priorities, but basically the idea was that urgent or critical issues would be ones that prevent people from using the planner right now, such as a broken compile in the current release. An issue that can wait in principle until the next release wouldn't be urgent or critical under this rule (even if the next release is scheduled to happen soon). We can rediscuss what the priorities mean, of course.
msg9425 (view) Author: florian Date: 2020-07-03.00:35:25
My reasoning for this was that as long as we do nothing about this issue, we cannot publish Fast Downward at the place where we decided it should live. I'm fine with changing the priority.
msg9422 (view) Author: malte Date: 2020-07-02.19:40:53
I don't agree with the definition of "critical", perhaps we can discuss this in one of the post-morning meetings.
msg9406 (view) Author: malte Date: 2020-07-02.16:34:49
If it helps simplify things, we are all ears. :-)
msg9398 (view) Author: gfrances Date: 2020-07-02.09:16:58
Only mildly related to this issue, but it came up to my mind that now it is relatively straightforward (easy as 
in "change one line in the Singularity recipe") to build the singularity image from the local Docker registry 
instead of the docker hub, so that we could build both images locally with no dependency wrt the hub.
Of course then we'll want to push the image to the hub anyway, so this is not a huge improvement, but still 
looks slightly more robust and efficient to build everything locally. I could look into this today as well if 
necessary.
msg9387 (view) Author: florian Date: 2020-07-01.19:37:16
I like option D in particular, because then everything directly related to the release is in the commit that is also tagged as the release. That also works nicely with './fast-downward.py --version'. I'll look into updating the release scripts tomorrow.
msg9384 (view) Author: malte Date: 2020-07-01.18:36:12
As far as I know, the automatic builds of singularity hub haven't worked for any of our past releases, so sacrificing them doesn't sound like a big deal.

I prefer the msg9383 tree (option B) over the msg9382 tree (option A). What about these simpler variants:

Option C:

* (4) Update version number to 19.12+.
* (3) Update version number to 19.12.
      (tagged release-19.12.0 and tip of branch release-19.12)
* (2) Commit container recipes for 19.12
* (1) older commits on default

Same as yours, but with a linear history.


Option D:

* (3) Update version number to 19.12+.
* (2) Commit container recipes for 19.12 and update version number to 19.12.
      (tagged release-19.12.0 and tip of branch release-19.12)
* (1) older commits on default

AKA the simplest possible thing.


One (small?) advantage of options C and D is that the release versions are also covered by our continuous integration (currently buildbot), which AFAIK doesn't test side branches.
msg9383 (view) Author: florian Date: 2020-07-01.18:13:25
Maybe an alternative would be to switch off the automatic builds of singularityhub and create this tree:

* (M3'') Update version number to 19.12+.
|
| * (R1'') Update version number to 19.12.
| |       (tagged release-19.12.0 and tip of branch release-19.12)
|/
* (M2'') Commit container recipes for 19.12
* (M1'') older commits on default
msg9382 (view) Author: florian Date: 2020-07-01.18:11:51
One thing I noticed while looking at the converted repository is the way the release branches look like. Our Mercurial workflow generates the following revision tree (for example release 19.12):

* (M2) Update version number to 19.12+.
|
| * (R3) Tag (R2) as release-19.12.0 (tip of open branch release-19.12)
| * (R2) Update version number to 19.12. (tagged release-19.12.0)
| * (R1) Create branch release-19.12.
|/
* (M1) older commits on default
* ...

The conversion script skips the empty commit (R1) that just creates the branch, which is fine, I think. But it keeps the commit (R3) that does nothing. For future releases, would we have just a single commit on the release branch? 

And where would the commit for the container files go? The dependency is a bit tricky there: the docker recipe pulls from the official repository and expects the release tag to exist, so we have to publish the release tag before building the docker container. The singularity script is built automatically when we push the recipe and it expects the docker image to be at dockerhub.

The workflow that mirrors our old one would end up with this tree. We'd push after creating R1' and M2', then build the docker image, publish it, then add all container files and push again.

* (M3') Commit container recipes for 19.12
* (M2') Update version number to 19.12+.
|
| * (R1') Update version number to 19.12.
| |       (tagged release-19.12.0 and tip of branch release-19.12)
|/
* (M1') older commits on default
* ...

The downside would be that the branch 19.12. would not contain the recipe files in misc/releases/19.12 because they are not added in this branch.
msg9377 (view) Author: florian Date: 2020-07-01.12:38:21
we decided to rename misc/release to misc/releases and have the previous file structure in that directory (e.g., misc/releases/latest/Singularity).

The scripts have to be adapted to this setting and we have to find a new place for the information in the README.
History
Date User Action Args
2020-07-07 14:13:32floriansetstatus: reviewing -> resolved
messages: + msg9522
2020-07-07 12:52:22maltesetmessages: + msg9521
2020-07-07 12:49:53floriansetmessages: + msg9520
2020-07-07 11:56:41maltesetmessages: + msg9517
2020-07-07 11:46:30guillemsetmessages: + msg9516
2020-07-07 10:38:43guillemsetmessages: + msg9514
2020-07-07 09:54:47maltesetmessages: + msg9513
2020-07-07 09:48:39guillemsetmessages: + msg9512
2020-07-06 22:18:10maltesetmessages: + msg9507
2020-07-06 19:46:18floriansetmessages: + msg9503
2020-07-06 19:31:07jendriksetmessages: + msg9501
2020-07-06 19:09:36maltesetmessages: + msg9498
2020-07-06 19:06:42maltesetmessages: + msg9497
2020-07-06 19:04:25floriansetmessages: + msg9496
2020-07-06 18:58:23floriansetmessages: + msg9495
2020-07-06 18:55:46maltesetmessages: + msg9494
2020-07-06 18:43:15floriansetstatus: chatting -> reviewing
assignedto: florian
messages: + msg9493
2020-07-06 17:01:56jendriksetmessages: + msg9487
2020-07-06 16:15:00maltesetmessages: + msg9483
2020-07-06 15:59:22jendriksetpriority: critical -> feature
2020-07-06 15:47:44floriansetpriority: wish -> critical
messages: + msg9482
2020-07-03 00:39:51maltesetmessages: + msg9427
2020-07-03 00:35:25floriansetpriority: critical -> wish
messages: + msg9425
2020-07-02 19:40:53maltesetnosy: + guillem, - gfrances
messages: + msg9422
2020-07-02 17:23:11floriansetpriority: meta -> critical
keyword: + infrastructure
2020-07-02 16:34:49maltesetmessages: + msg9406
2020-07-02 09:16:58gfrancessetnosy: + gfrances
messages: + msg9398
2020-07-01 19:37:16floriansetmessages: + msg9387
2020-07-01 18:36:12maltesetmessages: + msg9384
2020-07-01 18:13:25floriansetmessages: + msg9383
2020-07-01 18:11:51floriansetstatus: unread -> chatting
messages: + msg9382
2020-07-01 12:38:22floriancreate