Part of issue1001.

Done. For reference: I went to the "Daily Tests" workflow page and clicked on the "..." symbol for each workflow run and then selected "delete workflow run". After all runs were gone, the whole workflow disappeared as well.

> I'll close the pull request.
The workflow "daily" still shows on the aibasel repository. I think it would be good to get rid of it. Could you please look into this? Maybe you have to also remove the commits of your fork or remove/disable the workflow manually from the aibasel repo.

> Commit f2dc7fda5 fixed the ".hg" directory name problem. The Buildbot
> tests are now failing for other reasons.

It looks like this is because the tests no longer include a build step. This was changed during the last sprint, and I remember that someone commented that this would break certain parts of the buildbots. If it only affects tests we want to disable anyway, there is not necessarily something we need to do about it.

OK, let's drop the tests. I'll close the pull request.

Commit f2dc7fda5 fixed the ".hg" directory name problem. The Buildbot tests are now failing for other reasons.

> In my opinion this only makes sense if we have something that we would like to
> test that takes so long that we don't want to do it on every push (translating
> one task per domain might qualify, I don't know).

I don't think it should take long. Right now the tests fail, I think because the test code mistakes the ".hg" directory within the benchmark repository for a directory containing a planning domain.

I don't have a strong opinion, but like Silvan my recommendation would be to drop these tests.

For what it's worth, if we check if the translator was modified, we could run these as part of the regular tests. Translator changes are rare, I don't think it would be a problem. (If the tests run too long because some domain for some reason has a huge first task, I'd just adapt the test.) But I think the better solution would be to keep things simple and not have these tests at all.

> I'd like to limit this issue to whether we want to keep the nightly translator tests (in their current status) or whether we should drop them.

I'm in favor of dropping them. If we keep them, I agree with Florian's suggestion of only running the tests if necessary. We could even go further and check if the translator was was modified.

I think it could make sense to just abort the tests if the revision didn't change, i.e., to run it once per "day that had pushes". In my opinion this only makes sense if we have something that we would like to test that takes so long that we don't want to do it on every push (translating one task per domain might qualify, I don't know).

Sorry, didn't mean to change the title.

How often is the translator modified? I assume far less than every day. Since the tasks are static, nothing about the tested objects changes, or am I missing something? I don't understand why this should be tested daily.

I'd like to limit this issue to whether we want to keep the nightly translator tests (in their current status) or whether we should drop them.

I think a better way to check for determinism would be to compare the result to the result obtained on the previous day or a reference result.

Is this issue just for the translator tests or should we also discuss what other tests make sense to run once a day?

Yes, the translator tests check that the output is deterministic. Of course the proposed daily action would also check that all first tasks from the downward-benchmarks repo can actually be translated. Three times :-)

I think checking for determinism has value, but I'm fine with dropping the daily test.

You can find the code for the new workflow here: https://github.com/aibasel/downward/pull/31

To me, this test doesn't sound useful. What do we test with this? That the translator is deterministic?

Is this a useful test? We shouldn't run it just because we always used to run it.

The Buildbot currently runs two nightly/weekly tests: regression tests and "medium" translator tests. Both tests are the same for the nightly and weekly variant. We agreed that the regression tests in their current form don't have much value and that we want a better solution for tracking performance regressions in the long term. The "medium" translator tests run the first task of each domain three times and check that the output is the same.

I suggest to run the "medium" translator tests daily on GitHub.