Issue752

Title Support for SoPlex
Priority feature Status chatting
Superseder Nosy List florian, guillem, jendrik, malte
Assigned To florian Keywords
Optional summary

Created on 2017-11-30.13:31:57 by florian, last changed by guillem.

Messages
msg6689 (view) Author: malte Date: 2017-12-02.14:55:47
PS: In the past, it was quite common for us to not be able to compile on the
experiment platform due to version issues. That's why we had static builds in
the first place: to build on our own machines, then execute on the experiment
platform. (I don't know if that still works.)
msg6688 (view) Author: malte Date: 2017-12-02.14:53:06
Using a different compiler version for the library and for the planner is
something that I'd recommend against anyway. It's likely to not work, and
possible to lead to things breaking in very subtle ways that manifest in random
crashes etc.
msg6687 (view) Author: florian Date: 2017-12-02.14:30:09
It's much harder than I thought to get this running on the grid. SoPlex requires
a CMake version that only comes in two modules on the grid and these modules are
only compatible with modules for certain g++ versions. Using a newer compiler
version for the libraries makes them incompatible with the old compiler, so we
cannot use the compiler old compiler for the planner. Finding a compatible zlib
version on the grid is an additional problem. I'll stop working on this for
today but next time I'll try the whole process without zlib. I'll also try to
look into the issue with the newer compiler version.
msg6686 (view) Author: florian Date: 2017-12-02.13:25:47
There was a small error in msg6683: The option is called "--with-soplex-incdir"
not "--with-soplex-inc-dir".


For the record, here is how I installed osi with cplex/soplex support on the
grid it required a few extra hoops to jump through with loading the correct
modules and configuring OSI to work with them:

module purge
module load GCC/5.4.0-2.26.lua
module load CMake/3.4.3-foss-2016b.lua 
module load GMP/6.1.0-intel-2017.01.lua

# set in .zshrc
export DOWNWARD_COIN_ROOT64=~/local/opt/coin-0.107.9-64-cplex1251-soplex301
export
DOWNWARD_CPLEX_ROOT64=~/local/opt/ibm/ILOG/CPLEX_Enterprise_Server1251/CPLEX_Studio/cplex
export DOWNWARD_SOPLEX_ROOT64=~/local/opt/soplex-3.0.1-64

tar xvf soplex-3.0.1.tgz
cd soplex-3.0.1
mkdir build64
cd build64
cmake -DCMAKE_INSTALL_PREFIX="$DOWNWARD_SOPLEX_ROOT64" -DGMP_DIR="$EBROOTGMP" ..
make -j20
make install
cd ../..

wget http://www.coin-or.org/download/source/Osi/Osi-0.107.9.tgz
tar xvf Osi-0.107.9.tgz
cd Osi-0.107.9
./configure CC="gcc"  CFLAGS="-m64 -pthread -Wno-long-long" \
            CXX="g++" \
            CXXFLAGS="-m64 -pthread -Wno-long-long \
                      -DTHREADLOCAL='' \
                      -D__STDC_CONSTANT_MACROS \
                      -D__STDC_LIMIT_MACROS" \
            LDFLAGS="-L$DOWNWARD_CPLEX_ROOT64/lib/x86-64_sles10_4.1/static_pic \
                     -L$DOWNWARD_SOPLEX_ROOT64/lib" \
            --without-lapack --enable-static=yes \
            --prefix="$DOWNWARD_COIN_ROOT64" \
            --with-cplex-incdir=$DOWNWARD_CPLEX_ROOT64/include/ilcplex \
            --with-cplex-lib="-lcplex -lm" \
            --with-soplex-incdir=$DOWNWARD_SOPLEX_ROOT64/include \
            --with-soplex-lib=-lsoplex 
make -j20
make install

# Repeat OSI installation with these variables for CPLEX 12.7.1 (change
x86-64_sles10_4.1 to x86-64_linux)
export DOWNWARD_COIN_ROOT64=~/local/opt/coin-0.107.9-64-cplex1271-soplex301
export DOWNWARD_CPLEX_ROOT64=~/local/opt/cplex-12.7.1-64/cplex
msg6685 (view) Author: malte Date: 2017-12-02.00:57:30
> Sounds good. I think the zlib issue is a question of whether it is easier to
> document how to compile OSI and SoPlex without zlib or to document how to
> install zlib so that CMake finds it.

Agreed. Given that zlib is a very common library, I would expect the latter to
be easier.

> Looks like this is working. I prepared a pull request for it:
>
https://bitbucket.org/FlorianPommerening/downward-issues/pull-requests/40/issue752/diff

The patch looks good to me.

If you think it is useful, I can try this out too, but I suggest doing this
after it has been merged and our wiki has been updated. That way we also get to
test the documentation. In that case, please remind me that I wanted to do this
once the documentation is ready.

> What kind of experiments should we run with this? Is A*/h^seq on the optimal
> benchmarks enough?

You know best which kinds of configurations exercise the LP code.
msg6684 (view) Author: florian Date: 2017-12-01.22:33:43
Looks like this is working. I prepared a pull request for it:
https://bitbucket.org/FlorianPommerening/downward-issues/pull-requests/40/issue752/diff

I can currently compile it on my laptop with {release, debug} x {32, 64} as a
static build. I cannot try a dynamic build because I cannot compile SoPlex as a
shared library (but that also doesn't work for CPLEX, so I think we can ignore
it). I didn't try it on Mac or Windows yet, but I can give that a try tomorrow.

What kind of experiments should we run with this? Is A*/h^seq on the optimal
benchmarks enough?
I ran one blocks world instance for testing and the results was closer to CPLEX
than I expected (for the 2260950 evaluations CPLEX took 158.172s and SoPlex
205.312s).
msg6683 (view) Author: florian Date: 2017-12-01.19:29:53
Sounds good. I think the zlib issue is a question of whether it is easier to
document how to compile OSI and SoPlex without zlib or to document how to
install zlib so that CMake finds it. I think both are OK but I tend towards the
second option currently (I'll have a clearer picture when this issue is closer
to completion).

Also just to keep track of what I did:

# Download and extract SoPlex 3.0.1 code
# Compile 64 bit SoPlex:
mkdir build64
cd build64
cmake -DCMAKE_INSTALL_PREFIX="/opt/soplex-3.0.1-64" ..
make
sudo make install
cd ..

# Compile 32 bit SoPlex:
sudo apt install zlib1g-dev:i386
sudo apt install libgmp-dev:i386 (important for the performance of SoPlex but
should work without)
mkdir build32
cd build32
cmake -DCMAKE_INSTALL_PREFIX="/opt/soplex-3.0.1-32" ..
make
sudo make install

# Set up environment variables in bashrc (still had similar variables for CPLEX
from earlier)
export DOWNWARD_SOPLEX_ROOT32=/opt/soplex-3.0.1-32
export DOWNWARD_SOPLEX_ROOT64=/opt/soplex-3.0.1-64

# Download and extract OSI 0.107.9
# Compile 64 bit version of OSI
./configure CC="gcc"  CFLAGS="-m64 -pthread -Wno-long-long" \
            CXX="g++" CXXFLAGS="-m64 -pthread -Wno-long-long" \
            LDFLAGS="-L$DOWNWARD_CPLEX_ROOT64/lib/x86-64_linux/static_pic \
                     -L$DOWNWARD_SOPLEX_ROOT64/lib" \
            --without-lapack --enable-static=yes \
            --prefix="/opt/osi-0.107.9-cplex1263-soplex301-64" \
            --with-cplex-incdir=$DOWNWARD_CPLEX_ROOT64/include/ilcplex \
            --with-cplex-lib="-lcplex -lm" \
            --with-soplex-inc-dir=/opt/soplex-3.0.1-64/include \
            --with-soplex-lib="-lsoplex" 
make
sudo make install

# Compile 32 bit version of OSI
make distclean
./configure CC="gcc"  CFLAGS="-m32 -pthread -Wno-long-long" \
            CXX="g++" CXXFLAGS="-m32 -pthread -Wno-long-long" \
            LDFLAGS="-L$DOWNWARD_CPLEX_ROOT32/lib/x86_linux/static_pic \
                     -L$DOWNWARD_SOPLEX_ROOT32/lib" \
            --without-lapack --enable-static=yes \
            --prefix="/opt/osi-0.107.9-cplex1263-soplex301-32" \
            --with-cplex-incdir=$DOWNWARD_CPLEX_ROOT32/include/ilcplex \
            --with-cplex-lib="-lcplex -lm" \
            --with-soplex-inc-dir=/opt/soplex-3.0.1-32/include \
            --with-soplex-lib="-lsoplex" 
make
sudo make install

# Set up environment variables in bashrc:
export DOWNWARD_COIN_ROOT32=/opt/osi-0.107.9-cplex1263-soplex301-32
export DOWNWARD_COIN_ROOT64=/opt/osi-0.107.9-cplex1263-soplex301-64
msg6682 (view) Author: malte Date: 2017-12-01.18:57:34
I don't think we were planning to use zlib for compressed planner input. I think
for us it makes much more sense to handle compression at the driver layer. But
of course we can have zlib as a dependency for some subset of CPLEX/OSI/SoPlex
if that makes our life easier.

I wouldn't worry too much about 32-bit support at this stage. If we integrate
SoPlex only for 64-bit builts, that is fine.
msg6681 (view) Author: florian Date: 2017-12-01.18:43:50
The installation of SoPlex that comes with SCIP has zlib support. For CPLEX and
OSI this is also the default but so far our build instructions explicitly
disable this. In SoPlex the zlib support can also be disabled but that would
mean a more complex installation of SoPlex. 

Since we already talked about accepting compressed input to the planner I
thought it might be interesting to see how complicated it is to link the planner
with zlib so I changed the CMake files so they now look for zlib if an LP solver
is used. This change would mean that we can remove the "--disable-zlib" line
from the LP build instructions.

Linking with zlib was no problem on a 64bit build but it was getting complicated
for the 32bit version. On Ubuntu several packages provide a 32bit version of the
library in different locations. The (only) one that worked for me was
zlib1g-dev:i386. For the other packages, CMake didn't find the library, for
example because there was only a libz.so.1 symlink and none called libz.so.

On the non-CMake front of this issue, I added options to use the solver to our
interface and everything went relatively smooth. Currently, the solver still
outputs log messages for every solved LP that I'll try to disable next.

I'll also try to compile SoPlex from scratch in 32 and 64 bit to see if both
compiles work. Although, if the 32bit build doesn't work, I'd suggest to treat
it as unsupported since we want to get rid of it anyway.
msg6654 (view) Author: malte Date: 2017-11-30.13:38:33
Sounds great.
msg6653 (view) Author: florian Date: 2017-11-30.13:31:57
SoPlex (http://soplex.zib.de/) is a fast open-source free-for-academic-use LP
solver included in SCIP. It should work behind our existing interface through
OSI. I'd like to add support for it here.
History
Date User Action Args
2017-12-07 19:17:20guillemsetnosy: + guillem
2017-12-02 14:55:47maltesetmessages: + msg6689
2017-12-02 14:53:06maltesetmessages: + msg6688
2017-12-02 14:30:09floriansetmessages: + msg6687
2017-12-02 13:25:47floriansetmessages: + msg6686
2017-12-02 00:57:30maltesetmessages: + msg6685
2017-12-01 22:33:43floriansetmessages: + msg6684
2017-12-01 19:29:53floriansetmessages: + msg6683
2017-12-01 18:57:34maltesetmessages: + msg6682
2017-12-01 18:43:50floriansetmessages: + msg6681
2017-11-30 13:38:33maltesetstatus: unread -> chatting
messages: + msg6654
2017-11-30 13:31:57floriancreate