Issue923

Title Create LP containers for releases
Priority feature Status chatting
Superseder Nosy List florian, guillem, jendrik, malte
Assigned To Keywords
Optional summary
TODO:
- ~~clean up Dockerfile~~
- merge Dockerfiles?
- ~~create Singularity file~~
- ~~create Vagrantfile~~
- ~~include files in release workflow and scripts~~
- document use in wiki
- decide version naming
- test with CPLEX 12.9

Created on 2019-06-12.13:46:26 by florian, last changed by malte.

Summary
TODO:
- clean up Dockerfile
- merge Dockerfiles?
- create Singularity file
- create Vagrantfile
- include files in release workflow and scripts
- document use in wiki
- decide version naming
- test with CPLEX 12.9
Messages
msg8922 (view) Author: malte Date: 2019-06-26.14:01:57
CPLEX 12.8 and CPLEX 12.9 can now be downloaded from https://ibm.com/academic. 

The page needs an account, and when I got to the final download page, the
default "Download Directory" didn't work for me. But after switching to "http",
I could successfully download a CPLEX 12.9 installer. I didn't test it yet.
msg8921 (view) Author: florian Date: 2019-06-24.18:52:02
I continued working on this and adapted the scripts so they now create the
following containers:

- A Docker container without LP support based on template "_Dockerfile.tpl".
  It is tagged as "aibasel/downward:19.06" and "aibasel/downward:latest".
  Running a config that requires an LP solver, will stop the planner nicely with
  our error message ("you have to compile with LP support ...").
- A Docker container with LP support based on template "_Dockerfile_CPLEX.tpl".
  It is tagged as "aibasel/downward:19.06-CPLEX" and
  "aibasel/downward:latest-CPLEX".
  To run it, a CPLEX installation has to to be mounted
     docker run -v $CPLEX:/workspace/cplex aibasel/downward:19.07-CPLEX ...
  When the container is started without mounting CPLEX (even on blind search),
  the planner crashes with the error message:
      ... libcplex1280.so: cannot open shared object file:
          No such file or directory...
- Two Singularity containers based on the two Docker containers. Since the
  Singularity containers only differ in the tag of the docker image they use as a
  base, they both use the same template "_Singularity.tpl".
- A Vagrantfile to set up a VM with Vagrant. It will use a CPLEX installation
  file if one is specified in the correct environment variable at the time the
  VM is provisioned. Otherwise the VM will not install OSI and the planner will
  not support LP configurations. I used a single file here instead of two for
  LP/no-LP builds because we don't actually want to build two different
  containers/VMs as we do for Docker.

The open questions from my point of view are:
- Do you all agree on the version numbers for CPLEX containers?
- Should we combine the two Docker templates into one or split the Vagrant
template in two?
msg8901 (view) Author: malte Date: 2019-06-14.11:51:23
After talking to Michael: the IBM Academic Initiative website is currently in
transition. The old one doesn't work any more and seems to be discontinued.
There is a new work-in-progress one at

    https://ibm.biz/academic.

For the CPLEX entry, navigate to "Technology"/"See All"/"Data Science". However,
the CPLEX entry currently says "Coming soon". Michael said that it might become
available in a week or so (no promises).

In the meantime, it looks like there is no path towards getting a free academic
version of CPLEX. Once academic CPLEX is available again, I would like our
documentation and published containers to apply to the version of CPLEX that we
want to recommend for use. This may be the current version (12.9.x) or the one
we currently use (12.8.x). My thoughts on the version to recommend are:

1) Only recommend a version that IBM makes available. (We have to wait and see
if they will only make the latest version available or also older ones.)

2) If possible, recommend the most recent version because it's likely to be
around for longer.

3) Only recommend a version that we have tested and are reasonably happy with.
(If 12.9.x crashes the planner or has terrible performance, don't recommend it.)

My recommendation would be not to publish LP containers or update our public
documentation until we can get a current version of CPLEX again because I
wouldn't want us to publish something that becomes obsolete immediately. While
academic CPLEX is unavailable, the utility of such containers is also limited.
We may have to revisit this if the "one week" prediction turns into much more.

If someone wants to continue working on the containers in the meantime, of
course feel free to, but I would advocate against releasing them while there is
a high chance it might become out of date within a week.

Once we can access the new CPLEX version, I think we should benchmark it, like
we have done with other CPLEX upgrades in the past. If someone wants to already
prepare such a benchmark, that could be useful, too.

Some useful information on CPLEX version history and release dates here:

https://www-01.ibm.com/software/support/lifecycleapp/PLCDetail.wss?from=spf&synkey=U455580K80410V36
msg8895 (view) Author: malte Date: 2019-06-13.13:14:02
Regarding the Vagrantfile, I'm a bit stuck because I can't follow the LP
download instructions. Something seems to be broken on the IBM side. I contacted
Michael Katz about this; he offered to help with things like these earlier.

It would be possible to proceed with an earlier download by Florian. The issues
with that are that it doesn't really simulate what we want to simulate (the
"normal" experience of someone trying to do this), and there might be wasted
work if things have changed with whatever is the currently available CPLEX
download. But it would allow us to make progress while we're waiting for
something to happen on the other side.
msg8883 (view) Author: florian Date: 2019-06-12.15:27:21
I created a branch for this issue and added the Dockerfile. This is *not ready
for review yet*, I just wanted a place to keep the file until we have time to
work on this.
https://bitbucket.org/FlorianPommerening/downward-issues/pull-requests/58

We could also decide to merge the two Dockerfiles into one and have it compile
with an LP solver if it is present in the directory and without one otherwise
(we do something similar in our Buildbot worker containers). It would make the
file more complicated but on the other hand we would only have one file to
maintain instead of two. On the container side, it wouldn't make a difference as
we need two containers either way.

I added some TODOs to the summary.
msg8879 (view) Author: florian Date: 2019-06-12.13:46:26
We currently offer only containers without LP support in a release. The main
issue is that we are not allowed to package and distribute CPLEX in a container. 
This issue is about creating LP-enabled containers (Docker, Singularity and
Vagrant), include them in the release scripts, and update the documentation.

We discussed a solution that works for Docker and will likely transfer to
Singularity: in the first stage of our multi-stage build, we copy in the CPLEX
shared library, and compile OSI and the planner with it. In the second stage, we
then copy OSI and the planner (but not the shared library). Since we only
publish the second stage, CPLEX does not leave our computers. Users can then run
the container by mounting their CPLEX installation in the correct path. This
requires using the same CPLEX version and using separate containers for Fast
Downward with and without LP support.

For Vagrant, the solution should be straight-forward: we set up the provision
script so it looks for the CPLEX installation script in the current directory.
If it is present, we install CPLEX and OSI, otherwise, we don't. The
instructions on the Wiki should be enough. We also do something similar in the
Dockerfiles of our buildbot workers.
History
Date User Action Args
2019-06-26 14:01:57maltesetmessages: + msg8922
2019-06-24 18:52:02floriansetmessages: + msg8921
summary: TODO: - clean up Dockerfile - merge Dockerfiles? - create Singularity file - create Vagrantfile - include files in release workflow and scripts - document use in wiki -> TODO: - ~~clean up Dockerfile~~ - merge Dockerfiles? - ~~create Singularity file~~ - ~~create Vagrantfile~~ - ~~include files in release workflow and scripts~~ - document use in wiki - decide version naming - test with CPLEX 12.9
2019-06-14 11:51:23maltesetmessages: + msg8901
2019-06-13 13:14:02maltesetmessages: + msg8895
2019-06-12 15:27:21floriansetstatus: unread -> chatting
messages: + msg8883
summary: TODO: - clean up Dockerfile - merge Dockerfiles? - create Singularity file - create Vagrantfile - include files in release workflow and scripts - document use in wiki
2019-06-12 13:46:26floriancreate