Title Support for SoPlex
Priority feature Status resolved
Superseder Nosy List florian, guillem, jendrik, malte
Assigned To florian Keywords
Optional summary

Created on 2017-11-30.13:31:57 by florian, last changed by malte.

File name Uploaded Type Edit Remove
main.cpp florian, 2019-06-28.23:40:31 text/x-c++src
msg8955 (view) Author: malte Date: 2019-08-08.14:55:45
I think a downward-dev announcement makes sense, and this should also be mentioned in the change log.
msg8953 (view) Author: florian Date: 2019-08-08.14:00:44
The instructions should be up to date now and Silvan just tested them on the
grid and locally.

One problem for users with existing installations of OSI is the error we had on
the buildbot: if they don't have zlib on their system but OSI is installed,
CMake will still try to include zlib and fail. Installing zlib or reinstalling
OSI according to our updated instructions fixes this. Should we send an
announcement to downward-dev about this?
msg8952 (view) Author: malte Date: 2019-08-07.16:34:43
Great! Don't forget to also change the non-Soplex LP install instructions, as these change too with the new zlib dependency, right?
msg8951 (view) Author: florian Date: 2019-08-07.16:14:19
I made the change in the change log and merged. 

The buildbot now fails because OSI is compiled without zlib there. I'll install
it on the build slaves and will try the build again.

I'll also update the documentation in the wiki with instructions how to install
msg8948 (view) Author: malte Date: 2019-08-07.11:11:23
Left a small comment on bitbucket, otherwise looks good to merge.
msg8945 (view) Author: florian Date: 2019-08-05.23:05:05
I just saw that this issue is still open but I think the last commit handled all
remaining open points. Can I merge this?

By the way: the OSI team responded to the issue and suggested a fix. It will
probably end up in a future version of the interface but our code now works with
both the fixed and the unfixed version so it doesn't matter so much for us.
msg8942 (view) Author: florian Date: 2019-06-29.14:09:43
Thanks for the review. I made the changes you suggested and added a change log
msg8941 (view) Author: malte Date: 2019-06-29.13:27:21
I left some comments; I tried to understand the 90 new lines of CMake as well as
possible, but I know very little about it, so that isn't very thoroughly reviewed.

This needs an entry to the change log. It should include a brief mention of
relative performance. If we want to get fancy, we can discuss the different
solvers on a wiki page. If not, mentioning the experiments associated with this
issue will do. (This low-tech solution is probably the better one; for one, the
issue won't get changed over time, at most added to.)
msg8940 (view) Author: guillem Date: 2019-06-29.12:49:27
Great job, Florian :-)
msg8939 (view) Author: florian Date: 2019-06-29.11:22:04
If someone wants to review the changes before the merge, they are very minor and
can still be found at:
msg8938 (view) Author: florian Date: 2019-06-29.11:19:48
It could belong to both OSI or SoPlex. If the SoPlex API requires the sense to
be set after loading, then OSI should hide that difference. I reported a bug
with OSI. As far as I know the teams of both projects are both at ZIB and may
even overlap, so they can pass it along if necessary.

The experiments with the fix finished over night and the results look
impressive. SoPlex outperforms CPLEX in terms of coverage (-1 with potential
heuristics, +4 with operator counting, +19 with lmcount). While the results vary
by domain and configuration a lot, it seems like a viable alternative.

I think this is ready to be merged.
msg8937 (view) Author: malte Date: 2019-06-29.00:16:30
Thanks, Florian! This may be worth reporting a bug with the SoPlex people. (Or
with OSI, if OSI is involved.)

We now also have a nice experiment illustrating that maximizing instead of
minimizing potential heuristic values is a good idea. :-)
msg8936 (view) Author: florian Date: 2019-06-28.23:40:31
For the record: I'm attaching the test program I used to analyze this behavior.
msg8935 (view) Author: florian Date: 2019-06-28.23:39:19
I finally figured out what was going on. We set the objective sense (i.e.,
whether we are maximizing or minimizing the LP) before loading the problem. The
CPLEX interface handles this correctly but the SoPlex interface silently ignores
it. This meant that the LPs that we want to maximize (potentials and LM-count)
were minimized instead. As a result, we got a very expensive blind heuristic.

Changing the code, so the objective sense is set after loading the problem
should fix the issue. I started an experiment to confirm. I forgot to set the
silent flag, so the "unexplained errors" will appear in this experiment as well.
msg8932 (view) Author: malte Date: 2019-06-28.17:49:49
Very interesting. It looks like CPLEX and SOPLEX give different heuristic
values, based on "expansions until last jump" and similar statistics. Should
they? Perhaps the potential heuristic configurations might use different
heuristics because these depend on which of several equally good solutions we
get from the LP solver, but the third and I think also the second configuration
should be fully deterministic, shouldn't it?

Is there a way to get rid of these "unexplained errors" in future experiments?
msg8929 (view) Author: florian Date: 2019-06-28.01:10:42
Since we switched to native builds, this issue became a bit simpler. We also now
have a compatible zlib module on the grid, so I could run some tests. The
results are not at all what I expected. It seems like while SoPlex is slower
than CPLEX on average, CPLEX is extremely slow in some cases.
I didn't look into the results in more detail yet, maybe there are other
influences. For example, I'm comparing against CPLEX 12.9 which we have not
evaluated yet (see issue925, experiments are still running).
msg6689 (view) Author: malte Date: 2017-12-02.14:55:47
PS: In the past, it was quite common for us to not be able to compile on the
experiment platform due to version issues. That's why we had static builds in
the first place: to build on our own machines, then execute on the experiment
platform. (I don't know if that still works.)
msg6688 (view) Author: malte Date: 2017-12-02.14:53:06
Using a different compiler version for the library and for the planner is
something that I'd recommend against anyway. It's likely to not work, and
possible to lead to things breaking in very subtle ways that manifest in random
crashes etc.
msg6687 (view) Author: florian Date: 2017-12-02.14:30:09
It's much harder than I thought to get this running on the grid. SoPlex requires
a CMake version that only comes in two modules on the grid and these modules are
only compatible with modules for certain g++ versions. Using a newer compiler
version for the libraries makes them incompatible with the old compiler, so we
cannot use the compiler old compiler for the planner. Finding a compatible zlib
version on the grid is an additional problem. I'll stop working on this for
today but next time I'll try the whole process without zlib. I'll also try to
look into the issue with the newer compiler version.
msg6686 (view) Author: florian Date: 2017-12-02.13:25:47
There was a small error in msg6683: The option is called "--with-soplex-incdir"
not "--with-soplex-inc-dir".

For the record, here is how I installed osi with cplex/soplex support on the
grid it required a few extra hoops to jump through with loading the correct
modules and configuring OSI to work with them:

module purge
module load GCC/5.4.0-2.26.lua
module load CMake/3.4.3-foss-2016b.lua 
module load GMP/6.1.0-intel-2017.01.lua

# set in .zshrc
export DOWNWARD_COIN_ROOT64=~/local/opt/coin-0.107.9-64-cplex1251-soplex301
export DOWNWARD_SOPLEX_ROOT64=~/local/opt/soplex-3.0.1-64

tar xvf soplex-3.0.1.tgz
cd soplex-3.0.1
mkdir build64
cd build64
make -j20
make install
cd ../..

tar xvf Osi-0.107.9.tgz
cd Osi-0.107.9
./configure CC="gcc"  CFLAGS="-m64 -pthread -Wno-long-long" \
            CXX="g++" \
            CXXFLAGS="-m64 -pthread -Wno-long-long \
                      -DTHREADLOCAL='' \
                      -D__STDC_CONSTANT_MACROS \
                      -D__STDC_LIMIT_MACROS" \
            LDFLAGS="-L$DOWNWARD_CPLEX_ROOT64/lib/x86-64_sles10_4.1/static_pic \
                     -L$DOWNWARD_SOPLEX_ROOT64/lib" \
            --without-lapack --enable-static=yes \
            --prefix="$DOWNWARD_COIN_ROOT64" \
            --with-cplex-incdir=$DOWNWARD_CPLEX_ROOT64/include/ilcplex \
            --with-cplex-lib="-lcplex -lm" \
            --with-soplex-incdir=$DOWNWARD_SOPLEX_ROOT64/include \
make -j20
make install

# Repeat OSI installation with these variables for CPLEX 12.7.1 (change
x86-64_sles10_4.1 to x86-64_linux)
export DOWNWARD_COIN_ROOT64=~/local/opt/coin-0.107.9-64-cplex1271-soplex301
export DOWNWARD_CPLEX_ROOT64=~/local/opt/cplex-12.7.1-64/cplex
msg6685 (view) Author: malte Date: 2017-12-02.00:57:30
> Sounds good. I think the zlib issue is a question of whether it is easier to
> document how to compile OSI and SoPlex without zlib or to document how to
> install zlib so that CMake finds it.

Agreed. Given that zlib is a very common library, I would expect the latter to
be easier.

> Looks like this is working. I prepared a pull request for it:

The patch looks good to me.

If you think it is useful, I can try this out too, but I suggest doing this
after it has been merged and our wiki has been updated. That way we also get to
test the documentation. In that case, please remind me that I wanted to do this
once the documentation is ready.

> What kind of experiments should we run with this? Is A*/h^seq on the optimal
> benchmarks enough?

You know best which kinds of configurations exercise the LP code.
msg6684 (view) Author: florian Date: 2017-12-01.22:33:43
Looks like this is working. I prepared a pull request for it:

I can currently compile it on my laptop with {release, debug} x {32, 64} as a
static build. I cannot try a dynamic build because I cannot compile SoPlex as a
shared library (but that also doesn't work for CPLEX, so I think we can ignore
it). I didn't try it on Mac or Windows yet, but I can give that a try tomorrow.

What kind of experiments should we run with this? Is A*/h^seq on the optimal
benchmarks enough?
I ran one blocks world instance for testing and the results was closer to CPLEX
than I expected (for the 2260950 evaluations CPLEX took 158.172s and SoPlex
msg6683 (view) Author: florian Date: 2017-12-01.19:29:53
Sounds good. I think the zlib issue is a question of whether it is easier to
document how to compile OSI and SoPlex without zlib or to document how to
install zlib so that CMake finds it. I think both are OK but I tend towards the
second option currently (I'll have a clearer picture when this issue is closer
to completion).

Also just to keep track of what I did:

# Download and extract SoPlex 3.0.1 code
# Compile 64 bit SoPlex:
mkdir build64
cd build64
cmake -DCMAKE_INSTALL_PREFIX="/opt/soplex-3.0.1-64" ..
sudo make install
cd ..

# Compile 32 bit SoPlex:
sudo apt install zlib1g-dev:i386
sudo apt install libgmp-dev:i386 (important for the performance of SoPlex but
should work without)
mkdir build32
cd build32
cmake -DCMAKE_INSTALL_PREFIX="/opt/soplex-3.0.1-32" ..
sudo make install

# Set up environment variables in bashrc (still had similar variables for CPLEX
from earlier)
export DOWNWARD_SOPLEX_ROOT32=/opt/soplex-3.0.1-32
export DOWNWARD_SOPLEX_ROOT64=/opt/soplex-3.0.1-64

# Download and extract OSI 0.107.9
# Compile 64 bit version of OSI
./configure CC="gcc"  CFLAGS="-m64 -pthread -Wno-long-long" \
            CXX="g++" CXXFLAGS="-m64 -pthread -Wno-long-long" \
            LDFLAGS="-L$DOWNWARD_CPLEX_ROOT64/lib/x86-64_linux/static_pic \
                     -L$DOWNWARD_SOPLEX_ROOT64/lib" \
            --without-lapack --enable-static=yes \
            --prefix="/opt/osi-0.107.9-cplex1263-soplex301-64" \
            --with-cplex-incdir=$DOWNWARD_CPLEX_ROOT64/include/ilcplex \
            --with-cplex-lib="-lcplex -lm" \
            --with-soplex-inc-dir=/opt/soplex-3.0.1-64/include \
sudo make install

# Compile 32 bit version of OSI
make distclean
./configure CC="gcc"  CFLAGS="-m32 -pthread -Wno-long-long" \
            CXX="g++" CXXFLAGS="-m32 -pthread -Wno-long-long" \
            LDFLAGS="-L$DOWNWARD_CPLEX_ROOT32/lib/x86_linux/static_pic \
                     -L$DOWNWARD_SOPLEX_ROOT32/lib" \
            --without-lapack --enable-static=yes \
            --prefix="/opt/osi-0.107.9-cplex1263-soplex301-32" \
            --with-cplex-incdir=$DOWNWARD_CPLEX_ROOT32/include/ilcplex \
            --with-cplex-lib="-lcplex -lm" \
            --with-soplex-inc-dir=/opt/soplex-3.0.1-32/include \
sudo make install

# Set up environment variables in bashrc:
export DOWNWARD_COIN_ROOT32=/opt/osi-0.107.9-cplex1263-soplex301-32
export DOWNWARD_COIN_ROOT64=/opt/osi-0.107.9-cplex1263-soplex301-64
msg6682 (view) Author: malte Date: 2017-12-01.18:57:34
I don't think we were planning to use zlib for compressed planner input. I think
for us it makes much more sense to handle compression at the driver layer. But
of course we can have zlib as a dependency for some subset of CPLEX/OSI/SoPlex
if that makes our life easier.

I wouldn't worry too much about 32-bit support at this stage. If we integrate
SoPlex only for 64-bit builts, that is fine.
msg6681 (view) Author: florian Date: 2017-12-01.18:43:50
The installation of SoPlex that comes with SCIP has zlib support. For CPLEX and
OSI this is also the default but so far our build instructions explicitly
disable this. In SoPlex the zlib support can also be disabled but that would
mean a more complex installation of SoPlex. 

Since we already talked about accepting compressed input to the planner I
thought it might be interesting to see how complicated it is to link the planner
with zlib so I changed the CMake files so they now look for zlib if an LP solver
is used. This change would mean that we can remove the "--disable-zlib" line
from the LP build instructions.

Linking with zlib was no problem on a 64bit build but it was getting complicated
for the 32bit version. On Ubuntu several packages provide a 32bit version of the
library in different locations. The (only) one that worked for me was
zlib1g-dev:i386. For the other packages, CMake didn't find the library, for
example because there was only a symlink and none called

On the non-CMake front of this issue, I added options to use the solver to our
interface and everything went relatively smooth. Currently, the solver still
outputs log messages for every solved LP that I'll try to disable next.

I'll also try to compile SoPlex from scratch in 32 and 64 bit to see if both
compiles work. Although, if the 32bit build doesn't work, I'd suggest to treat
it as unsupported since we want to get rid of it anyway.
msg6654 (view) Author: malte Date: 2017-11-30.13:38:33
Sounds great.
msg6653 (view) Author: florian Date: 2017-11-30.13:31:57
SoPlex ( is a fast open-source free-for-academic-use LP
solver included in SCIP. It should work behind our existing interface through
OSI. I'd like to add support for it here.
Date User Action Args
2019-08-08 14:55:45maltesetmessages: + msg8955
2019-08-08 14:00:44floriansetmessages: + msg8953
2019-08-07 16:34:43maltesetmessages: + msg8952
2019-08-07 16:14:19floriansetstatus: chatting -> resolved
messages: + msg8951
2019-08-07 11:11:23maltesetmessages: + msg8948
2019-08-05 23:05:05floriansetmessages: + msg8945
2019-06-29 14:09:43floriansetmessages: + msg8942
2019-06-29 13:27:21maltesetmessages: + msg8941
2019-06-29 12:49:28guillemsetmessages: + msg8940
2019-06-29 11:22:04floriansetmessages: + msg8939
2019-06-29 11:19:48floriansetmessages: + msg8938
2019-06-29 00:16:31maltesetmessages: + msg8937
2019-06-28 23:40:31floriansetfiles: + main.cpp
messages: + msg8936
2019-06-28 23:39:19floriansetmessages: + msg8935
2019-06-28 17:49:49maltesetmessages: + msg8932
2019-06-28 01:10:42floriansetmessages: + msg8929
2017-12-07 19:17:20guillemsetnosy: + guillem
2017-12-02 14:55:47maltesetmessages: + msg6689
2017-12-02 14:53:06maltesetmessages: + msg6688
2017-12-02 14:30:09floriansetmessages: + msg6687
2017-12-02 13:25:47floriansetmessages: + msg6686
2017-12-02 00:57:30maltesetmessages: + msg6685
2017-12-01 22:33:43floriansetmessages: + msg6684
2017-12-01 19:29:53floriansetmessages: + msg6683
2017-12-01 18:57:34maltesetmessages: + msg6682
2017-12-01 18:43:50floriansetmessages: + msg6681
2017-11-30 13:38:33maltesetstatus: unread -> chatting
messages: + msg6654
2017-11-30 13:31:57floriancreate