The current setup of the Github actions is summarized here: http://www.fast-downward.org/ForDevelopers/GithubActions

Remove the description of workflows from summary (it is documented on the wiki).

I added an entry for in the change log.

From my side, I don't know if there is anything else to do. If there is, please let me know. Otherwise, I'll close this issue on Monday before we start the next sprint.

I documented our current setup of Github actions here: http://www.fast-downward.org/ForDevelopers/GithubActions

> I still don't fully understand what gets executed when in the presence of forks etc. This is probably easier to discuss in person.

Each commit pushed to the main repo triggers a build including LP solvers.

Commits pushed to forks within a pull request against the main repo also trigger builds on the main repo, however, these builds only compile with LP solvers if the secrets are set up on the fork.

We can discuss further in person. Let's coordinate via email.

It's not so much about protecting access to the CPLEX binary files as it is about protecting running of CPLEX by triggering builds that link against CPLEX. I still don't fully understand what gets executed when in the presence of forks etc. This is probably easier to discuss in person.

I merged the issue and pushed, so our repo now runs Github actions on push and pull requests.

I would like to come back to a point we discussed in issue971:

S: Malte said that when running CPLEX in Github actions, we should make sure that everyone with admin access on the Github repo is registered in the IBM academic initiative. 
M: Indeed, that was a requirement by IBM, and of course it also makes sense.

Does this really make sense? I think that everyone who creates a pull request against the aibasel/downward repo and adds a commit to the pr now triggers a Github action. At the same time, admins of the repo *cannot* find out the CPLEX binary just from having admin access. The URL is stored as a secret of the repo which can only be removed or overwritten, but not seen.


Jendrik also brought up the point that it is very likely that all of us who have a fork of the main repo and push the latest version there will automatically enable the Github actions. However, since the forks don't have the secrets setup, the actions won't work. I also don't think that they should run on every fork.

After lots of trial and error, I went with the simple solution of using git reset against main and recommitting everything in a single commit (instead of using git rebase or whatever other solution).

We now have a single pull request.

Yes, we should squash everything at the end.

BTW, I think it would be nice to squash everything into one commit for this issue.

I opened issue1001 as a follow-up issue of this one, where we want to discuss replacing the builbot by Github actions.

I updated the summary to describe our current Github action workflows and added a last TODO.

We decided to have four workflows:
- ubuntu 18.04, running with the default GCC and clang versions and with Python 3.6.
- ubuntu 20.04, running with the default GCC, the latest GCC, the oldest and latest clang versions, and with Python 3.6 and 3.8 (compiling only with Python 3.6; testing only with default GCC)
- the latest macOS version as provided by Github actions
- the latest Windows as provided by Github actions

The ubuntu workflows compile with CPLEX and SoPlex. macOS and Windows compile without LP solvers and only run the translator and search tests.

I agree that we should create a new issue for the remaining todos.

The items in the summary of this issue go further than what the title suggests. Since we decided to not directly replace any buildbot functionality by Github actions immediately, I don't think we should have all of the remaining TODOs be blockers for this issue. I suggest to open a new issue "use Github actions instead of buildbot for continuous integration" or similar and to move the remaining TODOS currently listed here to the new issue. This includes:
- setup nightly and weekly tests
- setup regression tests
- run CPLEX and SoPlex on MacOS and Windows workflows
- run Gurobi on all workflows

Testing the planner on windows is working and can be reviewed (see PR).

> Is it sufficient to test the compilation with a single python version?
> Is it sufficient to test the search with a single compiler version?

In the discussion today, we decided that it is sufficient.

> Is it sufficient to test the compilation with a single python version?
> Is it sufficient to test the search with a single compiler version?

I'd say yes to both of these questions.

After a lot of debugging (not using the tool, which didn't run out of the box), the linux workflow is again working.

To progress in the sprint Silvan and I discussed the compiler/python/LP version.

In line with msg9730 we decided to test only the oldest version fast-downward AND the test system support, as well as the newest version on the test system.

For Ubuntu 20.04 this means:
  - Compiler: gcc-7, gcc-10, clang-3.9, clang-11
  - Python: 3.6, 3.9

For Mac and Windows we have to test what is possible.

For the compilation respectively search tests, we wondered if we really have to test for all compiler and python versions. 
Is it sufficient to test the compilation with a single python version?
Is it sufficient to test the search with a single compiler version?

I just stumbled over a program that allows to run GitHub Actions locally, which could be very useful for working on this issue and debugging GitHub Actions quickly: https://github.com/nektos/act

Add note about CONTRIBUTING.md to summary.

We discussed this live today and decided that we want to integrate the GitHub Actions more or less as they're currently implemented. They're not meant to replace the Buildbot, at least for now. Instead the goal is to let them highlight style and compilation problems to contributors early in the reviewing process and let reviewers focus on the design of the code.

We agreed that we don't want to test all combinations of compilers and Python versions on all platforms. Instead, we want to test at most the oldest and newest supported versions of the C++ compilers and Python versions. Which versions to test on which platform is still up for debate and also depends on how easy they can be set up, I think.

I added them.

> If I'm not overlooking anything, the main remaining unknown for this issue is
> whether we can compile the planner with CPLEX on Windows automatically (i.e.,
> without clicking through installers).  I added a TODO for the corresponding
> subissue.

Silvan's msg9097 lists some other TODOs. If they are not yet done, can they be merged with the TODOs in the summary?

No need to mark Mac OS things that the buildbot currently doesn't do as TODOs, but it would be nice to keep track of them in the summary as optional future tasks (but not blockers for moving to github actions).

Is anyone (with some Windows expertise) interested in looking into compiling the planner with CPLEX on Windows for Github Actions?

If I'm not overlooking anything, the main remaining unknown for this issue is whether we can compile the planner with CPLEX on Windows automatically (i.e., without clicking through installers).  I added a TODO for the corresponding subissue.

Update pull request.

Thanks, I forwarded the relevant bits to the ZIB guys.

@Jendrik, the soplex action works, including redacting all output related to URLs. The code contains lots of duplication and should really be considered as a prototype.

> Is that visible to you, too? 

Yes it is. Also, for the final location of the installer, we should remember to pick a place that sounds less "temporary".

One problem I just stumbled across is that if using the secret URL with wget, the actions fails with an error message giving away the URL. Is that visible to you, too? 
https://github.com/silvansievers/issue490/runs/833765241?check_suite_focus=true

(Let me know if you need to be added to the repo.)

Update repo for pull request (with latest default).

I see no immediate issues with the CPLEX license that would prevent us from using a cloud-based buildbot replacement as long as access is limited to members of our research group. But I suggest you speak to Florian (who is not on this issue) for a second opinion and also look into this yourself.

Some relevant references:
https://my15.digitalexperience.ibm.com/b73a5759-c6a6-4033-ab6b-d9d4f9a6d65b/dxsites/151914d1-03d2-48fe-97d9-d21166848e65/faqs/agreement
https://my15.digitalexperience.ibm.com/b73a5759-c6a6-4033-ab6b-d9d4f9a6d65b/dxsites/151914d1-03d2-48fe-97d9-d21166848e65/faqs/guidelines
https://my15.digitalexperience.ibm.com/b73a5759-c6a6-4033-ab6b-d9d4f9a6d65b/dxsites/151914d1-03d2-48fe-97d9-d21166848e65/faqs/faqs

Current status of github actions:
- ubuntu: tests run for different compilers, python versions, including CPLEX
  TODOs: 
  - find out if and how we can legally run CPLEX.
  - nightly and weekly tests
- mac: those tests run by the buildbot run: exit codes and standard configs
  TODOs: CPLEX? valgrind for all code tests?
- windows: compilation runs (without CPLEX)
  TODOs: CPLEX? run some tests?

> Cedric, are you still working on the Windows yaml file? I saw that compilation works, but the planner is not yet tested. Also, it would be great if you could look into adding CPLEX also for the Windows builds.

Not today. I can look into it tomorrow.

Building with different Python versions also has its benefits, because it tests build.py with different versions. However, I agree that we should try to only build the planner for each revision once (or twice if we count the debug build). It seems this is possible by using artifacts:

https://help.github.com/en/actions/automating-your-workflow-with-github-actions/persisting-workflow-data-using-artifacts
https://stackoverflow.com/a/57877438/1277298

Patrick and I discussed this a bit further and I'll try to come up with a solution using matrix builds. We currently don't see an easy way of avoiding to compile the planner again for all different python versions.

Cedric, are you still working on the Windows yaml file? I saw that compilation works, but the planner is not yet tested. Also, it would be great if you could look into adding CPLEX also for the Windows builds.

There are multiple ways how to select which tox environments are run (https://tox.readthedocs.io/en/latest/config.html#conf-envlist). If you use a matrix setup, it's better to select only a single Python version. If the matrix build only provides a single Python version, you can use "tox -e py,style". Otherwise, you can probably set TOXENV=py36,style for the matrix build for Python 3.6.

What are each scenario (nighly, weekly?)
If the tests differ only in the python version used, I would prefer to have only
a single configuration file (if we run different tests in different scenarios,
those can be in different configuration files).
About the issue that only one python version succeeds, does specifying to tox to
run only a certain python version help? (e.g. tox -e p36 if your matrix is in
the step for python 3.6)?

The ubuntu setup also works (tested gcc, but not clang, python 3.6 and 3.7, and with CPLEX). I think we should now discuss how to test the different scenarios. In the current version, I used a matrix build to test different python versions, but then the generic test "tox" will fail on each build because only one python version succeeds. Should we address this with the already planned task of having separate tox configs for each scenario? This would however mean that we need to do the matrix setup there, too...

I left some comments there.

The MacOS tests now run successfully.

Inspecting your code more closely, the problem is that the code needs to call "pytest" instead of "python". Nonetheless, I recommend to use the official way of obtaining Python 3.8.

It seems that using homebrew is not the recommended way of installing Python versions for Github Actions. In case you haven't tried that already, 

here is a list of available versions:
https://help.github.com/en/actions/automating-your-workflow-with-github-actions/software-installed-on-github-hosted-runners#python-2

and here is how to install them: https://github.com/marketplace/actions/setup-python-environment

I'd choose Python 3.8 since this is the version that people will most likely want to install on their machines.

I tried to get the MacOS Yaml file done, but failed because "python3.7 <filename>" just doesn't do anything.

I tried to give access to aibasel, but this didn't work, so I only invited Jendrik and Patrick as "collaborators" so far. Any ideas on why this does not work are appreciated. Otherwise, we could also first try to have separate tox configurations, because running "tox" worked when I tried it.

Hi all, just include me in the nosy list for all issues. (Same for Jendrik, but he was already included.)

Attached is the current version of the yml file for the GitHub Actions (for
testing on ubuntu)