TODO:
- clean up Dockerfile
- merge Dockerfiles?
- create Singularity file
- create Vagrantfile
- include files in release workflow and scripts
- document use in wiki
- decide version naming
- test with CPLEX 12.9

The decision against having a separate CPLEX Dockerfile closed the remaining open TODOs in the summary, so I think this issue can be resolved. I created issue979 for the work towards supporting CPLEX in a release container.

> I have locally commented out the cplex variant for the release. I will report
> back when the release is done, probably not before tomorrow.

I have removed the CPLEX-related parts of the Dockerfiles (also affecting Singularity) in a commit directly on the main branch to avoid confusion with multiple different container versions. If we want to revive these, we still have them in the history.

Vagrant still builds with CPLEX support if the library is provided.

From my side, this can be marked as resolved (I would like to have a follow-up issue for containers that support CPLEX via dynamic loading if we don't have that yet). I'll leave it to Florian to mark this as resolved in case he wants to continue the discussion and update/close the TODO list for the issue.

I've merged the version of the code in the pull request and am currently running it for the 20.06 release. I'm keeping this open for now so that we can wrap up the discussion and decide on the next steps.

I have locally commented out the cplex variant for the release. I will report back when the release is done, probably not before tomorrow.

I've also looked a little bit into dynamic linking, so that we can stick with a single version but support commercial solvers in the future. Apparently OSI supports dynamically linking to commercial solvers including CPLEX and Gurobi via the "lazylpsolverlibs" library. It's mentioned in the README of the project, i.e., on its github landing page:

    https://github.com/coin-or/Osi

From how they describe it, this is mainly intended for packaging, so perhaps this is already enabled in the OSI version that ships with current Ubuntus. I couldn't easily find out since which OSI version this is available, or how the Ubuntu 20.04 version is configured. (I'm still running Ubuntu 18.04.) In any case, this may require going through an OSI version upgrade or worse, so perhaps it won't be trivial. But at least it's something OSI tries to explicitly support.

FWIW, I'm also attaching the script I used to test the ways bash options can be build sequentially. It helped me understand the problems I saw in the Vagranfile.

If you want to try out the attached Dockerfile, replace the non-existent tag release-99.99.0 with whatever version you want to build and replace the line

RUN ./misc/releases/install-lp-solvers.sh --cplex --soplex

with 

ADD install-lp-solvers.sh .
RUN ./install-lp-solvers.sh --cplex --soplex

You then need the two LP installers and the attached script in the same directory as the Dockerfile when you call "docker build".

I made it work with bash arrays. I think this nicely expresses which of the strings belong together. Much better than putting everything into one string and then splitting it later on, hoping that the quotation marks prevent splitting at the wrong place. So I think the pull request is now complete and ready for review.

Side note: while working on the solution for Vagrant, I tried putting everything into one script and that started looking very nice. The common code from the two Dockerfiles and the Vagrantfile would be in one place there, the container recipes would looks cleaner and the logic for deciding what to install would also be in one place. Unfortunately there are two roadblocks. First we cannot set variables from inside the script because the way we set variables is different from container to container. This is manageable by only doing the environment variable configuration in the container and doing the rest in the script. The other roadblock is docker-specific, I think. OSI complained about not finding the CPLEX header at a specific location but when I looked there in the finished container, the file was there. I suspect that this has something to do with accessing the layer we are currently building but I have not looked much further. I attached  the current state of the script and will add a Dockerfile that uses it, in case we want to pursue this further later on.

I don't have a great suggestion, so anything that works is fine. I may be able to look at it later, but having a working (even if perhaps not so pretty) solution now would be great. Prettification is optional.

I tried with \\" and it indeed showed up correctly when I printed the variable with echo. But in the place where I used the variable (./configure ... $OPTIONS), the script then complained about an unrecognized option -lm which is part after the space in the variable. So it ignored the inner quotation marks.

I also tried adding more quotes to make the option string contain \". This meant that I had to write \\\\\\" in the place where I add to the string. Vagrant turns this into \\\" before giving it to the shell which then interprets it as a backslash followed by a quotation mark. Still, this didn't change anything, the error now was 'unrecognized option -lm"' instead of 'unrecognized option -lm'. In both cases the inner quotes are ignored.

I'll try the workaround next, unless someone has a better idea.

Maybe a workaround would be to add a script misc/releases/install-available-lp-solvers.sh and just call it here. We could even make it more general and use it for the docker containers as well to remove the duplication there.

...and if this is indeed the problem, I would try doubling the escaping backslashes, i.e. '\=' => '\\='.

I think you can do something like "vagrant provision" to redo the provisioning steps in an existing VM. Perhaps that can reduce the turnaround time. I have no complete answer, but when I type this:

X=""hi there""

on my (bash) shell prompt, I get essentially the same error message as you about program "there" not being found. So perhaps someone (Vagrant?) is interpreting the backslash escapes before the shell sees them. If '\"' becomes '"', your line becomes that line.

Debugging this is a bit painful because the VM has to be created/destroyed every time. I found a difference to a normal shell that is probably the cause of this. The following script prints 
>> "hi there" << 
as expected:

=====
#!/bin/sh
X="\"hi there\""
echo ">>> $X <<<"
=====

However, if I put the last two lines in the Vagrantfile, I get this output:

    default: /tmp/vagrant-shell: line 2: there: command not found
    default: >>>  <<<

I don't fully understand this. The "hi" is ignored and is neither executed as a command nor added to the variable. Also, the quotes after "there" do not show up as unmatched quotes.

Thanks, Florian! Accumulating options like this should work fine at least in bash. I'm not sure what shell Vagrant uses for this. One thing I noticed between the two updates to OSI_CONFIG_OPTIONS for CPLEX and SoPlex is that this doesn't include quotes:

     --with-cplex-incdir=$DOWNWARD_CPLEX_ROOT/include/ilcplex

and this does:

     --with-soplex-incdir="$DOWNWARD_SOPLEX_ROOT/include"

Not sure that this should make a difference, though. Both paths are always the same and don't include spaces.

I'm fine either way. A CPLEX container would be nice to have but if the setup is more complicated than compiling the planner locally, I don't see many people using it. Currently, I think the effort is reasonable: To use the container you have to install CPLEX and mount the correct directory when you run the container.  At least this saves you the non-trivial OSI installation. But of course more problems could be lurking there.

Anyway, I prepared a pull request with the modified templates and release scripts. It contains the current plan (the first part of msg9619). If we want to leave out CPLEX that should not be a large change. 

https://github.com/aibasel/downward/pull/7/files

The Vagrant file is not yet working but I didn't want this to hold up the review.
I think the issue is how I collect options for OSI by adding them to a string (X="$X new-option") and then using the options as $X. I remember that this is problematic but don't know the correct way of doing this. Here is the place in the file:
https://github.com/aibasel/downward/pull/7/files#diff-755b0e080574d43cd6b78e4484d815aeR54

Thanks, Florian! I would say that's not too large, so we could go with this proposal:

- 20.06 => with SoPlex
- latest => alias for 20.06
- 20.06-cplex => variant of 20.06 with SoPlex + CPLEX

Alternatively, we could consider *not* releasing 20.06-cplex to avoid having a short-lived "special" version. People could still use CPLEX from the tarball as always. We would then try to improve the code so that the default can include CPLEX support in 20.12 while still working for people without CPLEX installed.

Of course there are other possible alternatives.

Thoughts?

Thanks Guillem, that was exactly what I was looking for yesterday. It worked out fine once I found out that a tag is required and the container name is not enough.

> If the compression in the Singularity archive is proportional, we are looking 
> at something like 36 MB vs. 47 MB for the Singularity images

Good guess:

36M no LP
42M CPLEX
46M SoPlex
46M CPLEX+SoPlex

Hi Guillem, that would be interesting to see. I'd love to see the four numbers that Florian mentioned also for Singularity containers. For me, the Singularity containers are the more important of the two.

Thanks, Florian! I think 98 MB vs. 128 MB is not trivial, but also not substantial enough to introduce the complication of having many different versions.

If the compression in the Singularity archive is proportional, we are looking at something like 36 MB vs. 47 MB for the Singularity images, and I think the larger size is fine. (It would be a bit more unfortunate if the LP stuff turned out to be less compressible, but I see no a priori reason why that should be the case.)

So my suggestion would be to go with the two options that include SoPlex, and work towards having a single container again for 20.12.

Just a quick note - in case you're interested in testing the Singularity 
images, checking size, etc., without uploading anything to the Docker Hub, that 
is doable (I think only for Singularity 3 and above, but not sure). Just need 
to change a couple of lines in the Singularity file, and declare the base 
docker image as in here:

https://github.com/AI-Planning/planutils/blob/master/environments/singularity-
builder/setup.sh#L93

That should pull from the local docker daemon.

As expected, adding CPLEX doesn't influence the container size a lot (because nothing is actually added except for some OSI libraries). SoPlex increases the container size by ~30MB:

no LP solver: 98.4MB
CPLEX only:    110MB
SoPlex only:   128MB
SoPlex+CPLEX:  128MB

I could not check the size of the Singularity containers because they can only be generated from docker images if the docker image is on Dockerhub. I expect them to be much smaller, because they use squashfs to store the files. For example, the current release has a size of 36MB. Compared to the 98MB of the docker container that is quite a good compression.

Sounds good. I'll prepare the different docker files and measure their size. I have to leave now, so I wont finish it today but it shouldn't be too much work.

Commenting on the specific options:

Regarding D), I don't see us do the necessary development right now.

Regarding A) vs. B), I have no preference.

Comparing A vs. C, I think there are still choices left to resolve, or I don't understand all the details.

In option A), does the "LP" option require both SoPlex and CPLEX? What if the user wants to use LPs but doesn't have CPLEX? They cannot use the no-LP build because it doesn't use LPs, and they cannot use the LP build because it links against CPLEX, so they're stuck.

If we want to support CPLEX and SoPlex in principle, without the D-ish option I think a "SoPlex, but no CPLEX" option is needed.

There are four possible combinations:

X1: no LP solver support compiled in
X2: only SoPlex support compiled in
X3: only CPlex support compiled in
X4: support for both compiled in

I think X2 is valuable because it supports LPs out of the box and requires no special user setup or license to use. Given X2, the only utility of X1 would lie in its size if it's substantially smaller. I'd be inclined to omit X1 unless there is a really substantial difference in size.

Similarly, X3 has limited value compared to X4 unless it is substantially smaller.


So ideally (and I don't know how much work this is), I would like to see the container sizes for all four options, then hopefully rule out X1 and X3 and go with X2 and X4.

I don't think we necessarily need multiple "latest" variants. It's fine that if people want a "special" version, they select a specific version number, especially if we're trying to unify to a single container again in future versions (which means a symbolic alias like "latest-LP" would no longer be needed). 

For example, we could go with this:

- 20.06 => with SoPlex
- latest => alias for 20.06
- 20.06-cplex => variant of 20.06 with SoPlex + CPLEX

But I'd like to see first if we're fine with having no non-LP version.

> Let's get back to the question on the number of container files. To clarify, the
> question is about the number of container files, not containers. If we install
> CPLEX as a shared library and compile Fast Downward against it, the planner will
> not start if it cannot find the CPLEX library at runtime, even it in blind
> search.

This doesn't imply that there has to be more than one container. There are at least two conceptually straightforward approaches of supporting with-CPLEX and without-CPLEX from the same container: have multiple executables within the container, or make the single executable smarter, so that it does not unconditionally require the shared libraries but only binds to them at runtime if they are available. In the medium term (not for the upcoming release), I think it would be good to move towards a single container.


This implies that whatever multiple-container strategy we follow for the upcoming release will be fairly temporary, to gather experience with all this, but hopefully not the solution we will use for 20.12. Hence I wouldn't worry about maintainability of one vs. multiple container files. We should use whatever is simplest now.

Let's get back to the question on the number of container files. To clarify, the question is about the number of container files, not containers. If we install CPLEX as a shared library and compile Fast Downward against it, the planner will not start if it cannot find the CPLEX library at runtime, even it in blind search.

I was just wondering if one template file that can be used to create both containers would be easier to maintain than two separate template files where one is more or less a subset of the other.

I see four possible options:
A) Containers "19.06", "19.06-LP", "latest", "latest-LP", created by two different templates.
B) Containers "19.06", "19.06-LP", "latest", "latest-LP", created by a single template compiled in different settings.
C) We could ship the release with SoPlex only. This would only need a single container, since we could link statically against SoPlex.
D) We could implement the advanced dynamic linking you mentioned so the CPLEX library is only loaded on demand. This way we could also have a single release container with both SoPlex support and support for CPLEX if it is mounted at runtime.

Thanks, that worked! :-)

I tried exactly that before, but stupidly used the wrong search box in step 5 (I used the box at the top right), and that found the same software, but with a different access path where I would have needed to pay or authenticate. I guess that search box takes you outside of the Academic Initiative context.

For those trying the same thing, let me add to Florian's instructions that you need to switch the tab from "Download Director" to "http" in step 7 in order for the download button to do anything, or at least that was my experience.

I'm trying to re-construct how I got to the download. I'm logged in already, so the steps might be different for me. Can you test if this also works for you:

1. Log in at ibm.com/academic
2. From the menu "Technology" select "Data Science"
3. Scroll down, find "ILOG CPLEX Optimization Studio" under "Software" and click on "Download v12.10"
4. In this view, click on "Search for Software" on the left
5. Enter "cplex 12.9" in the search box
6. The search results should contain "IBM ILOG CPLEX Optimization Studio Multiplatform Multilingual 12.9 eAssembly (CJ4Z5ML)"
7. Select the version for "Linux x86-64" and download it.

It is also possible to put "CJ4Z5ML" into the field "Part numbers" after step 3 to go directly to 6 but of course that only works if you already know that this is the magic product code.

An easy-to-take but wrong path is to click on the hamburger menu on ibm.com/academic and select anything under products. This will bring you to a page where you can also search for and find CPLEX 12.9 but it is the general IBM page about it, not the academic page. If you found the download this way, you will see the download link but if you click on it, you are forwarded to a page that will forward your request to "the person who manages access for your company".

Sticking to 12.9 works for me, but then I need help downloading 12.9.

I managed to get to the Academic Initiative download of 12.10, but not 12.9, at least not a page where I was able to download the free academic edition. Any advice? I did search via various interface and found some relevant-looking things, but ultimately nothing that helped.


Not sure when we should expect 12.11. These are the release months of the past CPLEX releases according to wikipedia:

12.10    December, 2019
12.9     March, 2019
12.8     December, 2017
12.7     November, 2016
12.6.2   June, 2015
12.6     December, 2013

No really obvious pattern.

The tests for 12.9 took some time, if I remember correctly. If we plan to use this for the release, I'd stick to 12.9 for now. Also, I don't know about IBMs release schedule but I wouldn't be surprised if version 12.11 was released in the next few months. I would wait for that until we update our recommendation and use 12.9 for the release now.

Thanks! Would it make sense to test 12.10 now?

Actually, we should probably use CPLEX 12.9 instead of 12.8 as this is the version we currently recommend on the the wiki. It also satisfies both criteria: we did the performance tests in issue925 and it is available in the same way as 12.8. We didn't test the latest version 12.10 yet.

> If possible, we should use a CPLEX version that is currently available and that 
> has been performance-tested by us. Does 12.8 satisfy these criteria? Any Osi 
> version that works for us is fine given that this is an implementation detail
> of the container.

Yes, 12.8 is tested and it is available. Although, finding old versions on the IBM website is a bit tricky: you have to explicitly search for the old version, and on the page of the current version, there is no way to get to older versions directly.

Hi Florian, if I recall correctly, you mentioned that adding SoPlex support to the containers would be easy. Do I remember this correctly? Do we already have a recipe somewhere?

Also, do you have or can you create a github pull request for the CPLEX container support? It would be great if we could include containers with LP support in the upcoming release, and I can put what available time I have towards that.

In msg8883 you wrote that this is not ready for review yet. Does your update in msg8921 imply that it is now ready for review?

Regarding the two questions:

Version numbers: do you mean the Osi/CPLEX versions? If possible, we should use a CPLEX version that is currently available and that has been performance-tested by us. Does 12.8 satisfy these criteria? Any Osi version that works for us is fine given that this is an implementation detail of the container.

I cannot currently comment fully on the question whether we should have one or multiple containers. All other things being equal, I would prefer a single container and a single Vagrantfile. But it also depends on the size of the containers and on how many complications the LP builds imply for people that don't need the LPs. With some more advanced dynamic linking features, we could have a planner build that supports a given LP solver if it's available at runtime, with no need for different builds. I wouldn't advocate doing such more advanced things for our upcoming release, but I think it could be an interesting hiwi task. Can probably be done in a weekend (famous last words).

Just to be clear: I don't know if anybody already prepared the recipes to include  SoPlex in the containers.

Regarding the email exchange: you should already have received the relevant mail yesterday.

Hi Jendrik, we should do this now, given that we want to release next week. Do you want to do this? If you would prefer me to do this, can you forward me your email exchange with them, so that I can reference it?

My ZIB contact said that "it should be possible" to include SoPlex and SCIP in our containers and that they "would be able to work with [us] to get this done". I suggest to send them our container setup and ask whether they're fine with it, like we did for the GitHub actions.

I don't know if Florian's update is still current after the recent discussion with ZIB, but if it is: I don't think we could easily distribute a container under the ZIB license. We don't currently have a way to track the information they require, and it would also preclude automated setups that we would like to be able to offer (e.g. just pulling from singularity-hub).

I'm adding Michael to the nosy list as he wrote to me about creating containers with packaged LP/MIP solvers.

Since the last time this issue was updated, we discussed the option of including SoPlex instead of CPLEX in a container. Since SoPlex is only an LP solver, using the SCIP Optimization Suite that includes SoPlex and the MIP solver SCIP would be an option. Both SoPlex and the SCIP Optimization Suite are distributed under the ZIB license: https://scipopt.org/academic.txt

I had a closer look at the license and it looks like it allows redistribution as long as the requirements in point 3 of the license are met.

3a) If I understand this point correctly, the container would have to be under the ZIB license but the license of Fast Downward (without SCIP) would be unaffected. For us, this would mean that we offer Fast Downward (as part of this container) under an additional license that allows relatively unrestricted use in a non-commercial and academic setting.

3b) If we modify any files of SoPlex/SCIP we have to keep track of the changes.

3c) We would have to keep track of access to the container in the same way that the people at ZIB do. They are asking for Name, Email, Country, Institution (optional) and City (optional) and require users to confirm that they the rights to use the software (see https://www.scipopt.org/download.php?fname=scipoptsuite-7.0.0.tgz). Interestingly, they also offer a dockerized version of SCIP, where they don't track access in this way: http://www.pokutta.com/blog/pages/scip/scip-teaching.html

3d) The container has to include the source code. I'm not sure if this refers to only the sources of SCIP/SoPlex or also the ones of Fast Downward.

CPLEX 12.8 and CPLEX 12.9 can now be downloaded from https://ibm.com/academic. 

The page needs an account, and when I got to the final download page, the
default "Download Directory" didn't work for me. But after switching to "http",
I could successfully download a CPLEX 12.9 installer. I didn't test it yet.

I continued working on this and adapted the scripts so they now create the
following containers:

- A Docker container without LP support based on template "_Dockerfile.tpl".
  It is tagged as "aibasel/downward:19.06" and "aibasel/downward:latest".
  Running a config that requires an LP solver, will stop the planner nicely with
  our error message ("you have to compile with LP support ...").
- A Docker container with LP support based on template "_Dockerfile_CPLEX.tpl".
  It is tagged as "aibasel/downward:19.06-CPLEX" and
  "aibasel/downward:latest-CPLEX".
  To run it, a CPLEX installation has to to be mounted
     docker run -v $CPLEX:/workspace/cplex aibasel/downward:19.07-CPLEX ...
  When the container is started without mounting CPLEX (even on blind search),
  the planner crashes with the error message:
      ... libcplex1280.so: cannot open shared object file:
          No such file or directory...
- Two Singularity containers based on the two Docker containers. Since the
  Singularity containers only differ in the tag of the docker image they use as a
  base, they both use the same template "_Singularity.tpl".
- A Vagrantfile to set up a VM with Vagrant. It will use a CPLEX installation
  file if one is specified in the correct environment variable at the time the
  VM is provisioned. Otherwise the VM will not install OSI and the planner will
  not support LP configurations. I used a single file here instead of two for
  LP/no-LP builds because we don't actually want to build two different
  containers/VMs as we do for Docker.

The open questions from my point of view are:
- Do you all agree on the version numbers for CPLEX containers?
- Should we combine the two Docker templates into one or split the Vagrant
template in two?

After talking to Michael: the IBM Academic Initiative website is currently in
transition. The old one doesn't work any more and seems to be discontinued.
There is a new work-in-progress one at

    https://ibm.biz/academic.

For the CPLEX entry, navigate to "Technology"/"See All"/"Data Science". However,
the CPLEX entry currently says "Coming soon". Michael said that it might become
available in a week or so (no promises).

In the meantime, it looks like there is no path towards getting a free academic
version of CPLEX. Once academic CPLEX is available again, I would like our
documentation and published containers to apply to the version of CPLEX that we
want to recommend for use. This may be the current version (12.9.x) or the one
we currently use (12.8.x). My thoughts on the version to recommend are:

1) Only recommend a version that IBM makes available. (We have to wait and see
if they will only make the latest version available or also older ones.)

2) If possible, recommend the most recent version because it's likely to be
around for longer.

3) Only recommend a version that we have tested and are reasonably happy with.
(If 12.9.x crashes the planner or has terrible performance, don't recommend it.)

My recommendation would be not to publish LP containers or update our public
documentation until we can get a current version of CPLEX again because I
wouldn't want us to publish something that becomes obsolete immediately. While
academic CPLEX is unavailable, the utility of such containers is also limited.
We may have to revisit this if the "one week" prediction turns into much more.

If someone wants to continue working on the containers in the meantime, of
course feel free to, but I would advocate against releasing them while there is
a high chance it might become out of date within a week.

Once we can access the new CPLEX version, I think we should benchmark it, like
we have done with other CPLEX upgrades in the past. If someone wants to already
prepare such a benchmark, that could be useful, too.

Some useful information on CPLEX version history and release dates here:

https://www-01.ibm.com/software/support/lifecycleapp/PLCDetail.wss?from=spf&synkey=U455580K80410V36

Regarding the Vagrantfile, I'm a bit stuck because I can't follow the LP
download instructions. Something seems to be broken on the IBM side. I contacted
Michael Katz about this; he offered to help with things like these earlier.

It would be possible to proceed with an earlier download by Florian. The issues
with that are that it doesn't really simulate what we want to simulate (the
"normal" experience of someone trying to do this), and there might be wasted
work if things have changed with whatever is the currently available CPLEX
download. But it would allow us to make progress while we're waiting for
something to happen on the other side.

I created a branch for this issue and added the Dockerfile. This is *not ready
for review yet*, I just wanted a place to keep the file until we have time to
work on this.
https://bitbucket.org/FlorianPommerening/downward-issues/pull-requests/58

We could also decide to merge the two Dockerfiles into one and have it compile
with an LP solver if it is present in the directory and without one otherwise
(we do something similar in our Buildbot worker containers). It would make the
file more complicated but on the other hand we would only have one file to
maintain instead of two. On the container side, it wouldn't make a difference as
we need two containers either way.

I added some TODOs to the summary.

We currently offer only containers without LP support in a release. The main
issue is that we are not allowed to package and distribute CPLEX in a container. 
This issue is about creating LP-enabled containers (Docker, Singularity and
Vagrant), include them in the release scripts, and update the documentation.

We discussed a solution that works for Docker and will likely transfer to
Singularity: in the first stage of our multi-stage build, we copy in the CPLEX
shared library, and compile OSI and the planner with it. In the second stage, we
then copy OSI and the planner (but not the shared library). Since we only
publish the second stage, CPLEX does not leave our computers. Users can then run
the container by mounting their CPLEX installation in the correct path. This
requires using the same CPLEX version and using separate containers for Fast
Downward with and without LP support.

For Vagrant, the solution should be straight-forward: we set up the provision
script so it looks for the CPLEX installation script in the current directory.
If it is present, we install CPLEX and OSI, otherwise, we don't. The
instructions on the Wiki should be enough. We also do something similar in the
Dockerfiles of our buildbot workers.