Title Create LP containers for releases
Priority feature Status chatting
Superseder Nosy List florian, guillem, jendrik, malte, mkatz
Assigned To Keywords
Optional summary
- ~~clean up Dockerfile~~
- merge Dockerfiles?
- ~~create Singularity file~~
- ~~create Vagrantfile~~
- ~~include files in release workflow and scripts~~
- document use in wiki
- decide version naming
- test with CPLEX 12.9

Created on 2019-06-12.13:46:26 by florian, last changed by jendrik.

- clean up Dockerfile
- merge Dockerfiles?
- create Singularity file
- create Vagrantfile
- include files in release workflow and scripts
- document use in wiki
- decide version naming
- test with CPLEX 12.9
msg9475 (view) Author: jendrik Date: 2020-07-04.17:41:23
My ZIB contact said that "it should be possible" to include SoPlex and SCIP in our containers and that they "would be able to work with [us] to get this done". I suggest to send them our container setup and ask whether they're fine with it, like we did for the GitHub actions.
msg9474 (view) Author: malte Date: 2020-07-04.17:35:12
I don't know if Florian's update is still current after the recent discussion with ZIB, but if it is: I don't think we could easily distribute a container under the ZIB license. We don't currently have a way to track the information they require, and it would also preclude automated setups that we would like to be able to offer (e.g. just pulling from singularity-hub).
msg9354 (view) Author: florian Date: 2020-06-19.12:15:08
I'm adding Michael to the nosy list as he wrote to me about creating containers with packaged LP/MIP solvers.

Since the last time this issue was updated, we discussed the option of including SoPlex instead of CPLEX in a container. Since SoPlex is only an LP solver, using the SCIP Optimization Suite that includes SoPlex and the MIP solver SCIP would be an option. Both SoPlex and the SCIP Optimization Suite are distributed under the ZIB license:

I had a closer look at the license and it looks like it allows redistribution as long as the requirements in point 3 of the license are met.

3a) If I understand this point correctly, the container would have to be under the ZIB license but the license of Fast Downward (without SCIP) would be unaffected. For us, this would mean that we offer Fast Downward (as part of this container) under an additional license that allows relatively unrestricted use in a non-commercial and academic setting.

3b) If we modify any files of SoPlex/SCIP we have to keep track of the changes.

3c) We would have to keep track of access to the container in the same way that the people at ZIB do. They are asking for Name, Email, Country, Institution (optional) and City (optional) and require users to confirm that they the rights to use the software (see Interestingly, they also offer a dockerized version of SCIP, where they don't track access in this way:

3d) The container has to include the source code. I'm not sure if this refers to only the sources of SCIP/SoPlex or also the ones of Fast Downward.
msg8922 (view) Author: malte Date: 2019-06-26.14:01:57
CPLEX 12.8 and CPLEX 12.9 can now be downloaded from 

The page needs an account, and when I got to the final download page, the
default "Download Directory" didn't work for me. But after switching to "http",
I could successfully download a CPLEX 12.9 installer. I didn't test it yet.
msg8921 (view) Author: florian Date: 2019-06-24.18:52:02
I continued working on this and adapted the scripts so they now create the
following containers:

- A Docker container without LP support based on template "_Dockerfile.tpl".
  It is tagged as "aibasel/downward:19.06" and "aibasel/downward:latest".
  Running a config that requires an LP solver, will stop the planner nicely with
  our error message ("you have to compile with LP support ...").
- A Docker container with LP support based on template "_Dockerfile_CPLEX.tpl".
  It is tagged as "aibasel/downward:19.06-CPLEX" and
  To run it, a CPLEX installation has to to be mounted
     docker run -v $CPLEX:/workspace/cplex aibasel/downward:19.07-CPLEX ...
  When the container is started without mounting CPLEX (even on blind search),
  the planner crashes with the error message:
      ... cannot open shared object file:
          No such file or directory...
- Two Singularity containers based on the two Docker containers. Since the
  Singularity containers only differ in the tag of the docker image they use as a
  base, they both use the same template "_Singularity.tpl".
- A Vagrantfile to set up a VM with Vagrant. It will use a CPLEX installation
  file if one is specified in the correct environment variable at the time the
  VM is provisioned. Otherwise the VM will not install OSI and the planner will
  not support LP configurations. I used a single file here instead of two for
  LP/no-LP builds because we don't actually want to build two different
  containers/VMs as we do for Docker.

The open questions from my point of view are:
- Do you all agree on the version numbers for CPLEX containers?
- Should we combine the two Docker templates into one or split the Vagrant
template in two?
msg8901 (view) Author: malte Date: 2019-06-14.11:51:23
After talking to Michael: the IBM Academic Initiative website is currently in
transition. The old one doesn't work any more and seems to be discontinued.
There is a new work-in-progress one at

For the CPLEX entry, navigate to "Technology"/"See All"/"Data Science". However,
the CPLEX entry currently says "Coming soon". Michael said that it might become
available in a week or so (no promises).

In the meantime, it looks like there is no path towards getting a free academic
version of CPLEX. Once academic CPLEX is available again, I would like our
documentation and published containers to apply to the version of CPLEX that we
want to recommend for use. This may be the current version (12.9.x) or the one
we currently use (12.8.x). My thoughts on the version to recommend are:

1) Only recommend a version that IBM makes available. (We have to wait and see
if they will only make the latest version available or also older ones.)

2) If possible, recommend the most recent version because it's likely to be
around for longer.

3) Only recommend a version that we have tested and are reasonably happy with.
(If 12.9.x crashes the planner or has terrible performance, don't recommend it.)

My recommendation would be not to publish LP containers or update our public
documentation until we can get a current version of CPLEX again because I
wouldn't want us to publish something that becomes obsolete immediately. While
academic CPLEX is unavailable, the utility of such containers is also limited.
We may have to revisit this if the "one week" prediction turns into much more.

If someone wants to continue working on the containers in the meantime, of
course feel free to, but I would advocate against releasing them while there is
a high chance it might become out of date within a week.

Once we can access the new CPLEX version, I think we should benchmark it, like
we have done with other CPLEX upgrades in the past. If someone wants to already
prepare such a benchmark, that could be useful, too.

Some useful information on CPLEX version history and release dates here:
msg8895 (view) Author: malte Date: 2019-06-13.13:14:02
Regarding the Vagrantfile, I'm a bit stuck because I can't follow the LP
download instructions. Something seems to be broken on the IBM side. I contacted
Michael Katz about this; he offered to help with things like these earlier.

It would be possible to proceed with an earlier download by Florian. The issues
with that are that it doesn't really simulate what we want to simulate (the
"normal" experience of someone trying to do this), and there might be wasted
work if things have changed with whatever is the currently available CPLEX
download. But it would allow us to make progress while we're waiting for
something to happen on the other side.
msg8883 (view) Author: florian Date: 2019-06-12.15:27:21
I created a branch for this issue and added the Dockerfile. This is *not ready
for review yet*, I just wanted a place to keep the file until we have time to
work on this.

We could also decide to merge the two Dockerfiles into one and have it compile
with an LP solver if it is present in the directory and without one otherwise
(we do something similar in our Buildbot worker containers). It would make the
file more complicated but on the other hand we would only have one file to
maintain instead of two. On the container side, it wouldn't make a difference as
we need two containers either way.

I added some TODOs to the summary.
msg8879 (view) Author: florian Date: 2019-06-12.13:46:26
We currently offer only containers without LP support in a release. The main
issue is that we are not allowed to package and distribute CPLEX in a container. 
This issue is about creating LP-enabled containers (Docker, Singularity and
Vagrant), include them in the release scripts, and update the documentation.

We discussed a solution that works for Docker and will likely transfer to
Singularity: in the first stage of our multi-stage build, we copy in the CPLEX
shared library, and compile OSI and the planner with it. In the second stage, we
then copy OSI and the planner (but not the shared library). Since we only
publish the second stage, CPLEX does not leave our computers. Users can then run
the container by mounting their CPLEX installation in the correct path. This
requires using the same CPLEX version and using separate containers for Fast
Downward with and without LP support.

For Vagrant, the solution should be straight-forward: we set up the provision
script so it looks for the CPLEX installation script in the current directory.
If it is present, we install CPLEX and OSI, otherwise, we don't. The
instructions on the Wiki should be enough. We also do something similar in the
Dockerfiles of our buildbot workers.
Date User Action Args
2020-07-04 17:41:23jendriksetmessages: + msg9475
2020-07-04 17:35:13maltesetmessages: + msg9474
2020-06-19 12:15:08floriansetnosy: + mkatz
messages: + msg9354
2019-06-26 14:01:57maltesetmessages: + msg8922
2019-06-24 18:52:02floriansetmessages: + msg8921
summary: TODO: - clean up Dockerfile - merge Dockerfiles? - create Singularity file - create Vagrantfile - include files in release workflow and scripts - document use in wiki -> TODO: - ~~clean up Dockerfile~~ - merge Dockerfiles? - ~~create Singularity file~~ - ~~create Vagrantfile~~ - ~~include files in release workflow and scripts~~ - document use in wiki - decide version naming - test with CPLEX 12.9
2019-06-14 11:51:23maltesetmessages: + msg8901
2019-06-13 13:14:02maltesetmessages: + msg8895
2019-06-12 15:27:21floriansetstatus: unread -> chatting
messages: + msg8883
summary: TODO: - clean up Dockerfile - merge Dockerfiles? - create Singularity file - create Vagrantfile - include files in release workflow and scripts - document use in wiki
2019-06-12 13:46:26floriancreate