In an experiment for the IPC 2014 I translated some additional domains that are 
currently not part of the benchmark suite in the FD repo. Most of the domains have 
conditional effects and the files tend to be bigger than the standard benchmark 
files. The experiment confirms that pypy translates tasks faster when the 
translation with CPython takes more than ~10 seconds, but is slower on tasks that 
can be translated in under 10 seconds. For the tasks in this experiment CPython has 
(very) roughly half of pypy's memory requirements.

I think it's interesting to see which speedup factors pypy can achieve for tasks 
that take long to translate (e.g. 30s vs. 119s for one task), but the pypy startup 
time is non-negligible for short-running translations. Additionally, CPython is much 
more common, probably better maintained and has a lower memory footprint. Therefore, 
I think we should stay with CPython (for now).

Should we mark this one as resolved?

I ran a more complete experiment on all domains in the benchmark directory (I forgot airport-adl though) comparing 
cpython 2.7 and pypy 2.0. From the attached results it becomes clear that pypy is only faster than cpython when 
cpython takes more than ~5 seconds. However, there are ~8 domains where pypy is much faster than cpython. Two of 
these domains are fully grounded and therefore a better alternative to using pypy for them is to use a dedicated 
invariant synthesis algorithm. Overall, it seems we should stick with cpython until pypy becomes faster even for 
short running translations.

Another option could be numba (http://numba.pydata.org/). Although it's mainly written for 
speeding up numpy code, it supposedly also makes plain python code faster.

A small example:

from numba import autojit
@autojit
def func():
    for i in range(10**8):
        i*i*i*i
func()

Python: 39.5s
Python+numba (1st run compiles): 3.0s
Python+numba (later runs use cached compilation): 2.2s

Running pypy on the grid seems to be very complicated due to missing and old libraries, so 
I ran a smaller experiment on my laptop with the IPC11 satisficing suite.

PyPy 2.1+ (trunk) has a lower sum of absolute runtime, but needs more time relatively. In 
addition PyPy needs 72% more memory than CPython 2.7 on average. PyPy runs faster on all 
tasks for which CPython needs at least 6 seconds (except for woodworking). The detailed 
results are attached.

Interesting! We need more data for this to be conclusive, though.

Different domains have bottlenecks in different places in the translator. For
PSR-Large, the time for processing axioms will take a significant fraction of
time, whereas in domains like Satellite and Scanalyzer it will be zero.

So if you do more experiments, it would be good to collect separate data for the
different bottlenecks of the translator, and also to collect peak memory statistics.

If I run PSR-Large #50 on my notebook, it seems to start swapping, so
memory-related issues become very important. I just did a quick test, and it
took 182 seconds CPU time and 300 seconds real-time.

It seems that PyPy is indeed faster than CPython for bigger problems (based on very small 
experiments). The following runs were performed on my Thinkpad X230 (dual-core) with an SSD.

psr-large/p50-s219-n100-l3-f30.pddl:
CPython 2.7:      135.9 secs
CPython 3.3:      143.3 secs
PyPy 1.9:          58.1 secs
PyPy 2.1+(trunk):  54.0 secs

I will probably run a bigger experiment when time allows.

Cython is yet another option.

Just out of curiosity I reran this with newer versions (on the new laptop with 
SSD):

CPython: 0.2 secs
PyPy:    1.0 secs

The mypy project (http://www.mypy-lang.org/) might also be interesting for us. 
It let's users annotate parts of the python sourcecode with type annotations to 
allow for efficient compilation to C code.

Since we're not really using many of the more dynamic aspects of Python, it may
be worth trying to compile parts or all of the translator with Shed Skin or
RPython. Been on my list forever, but I never had enough time. If anyone's
interested in trying this out, I think there should be some potential.

Out of curiosity I have translated some gripper problems with CPython2.6 and 
PyPy 1.5 to see if PyPy performs better on this task. It seems however that the 
code runs better on CPython:

gripper:prob20.pddl
- CPython: 0.67 CPU secs
- PyPy: 2.57 CPU secs