proposal for regression testing

[HansVRP]

Idea for starting to include regression benchmarks.

@JanssenBrm I would also need info on how to best expose it such that we can keep a log on the service catalogue

[algorithm-services-catalogue]

🔍 Catalogue's Preview Site Deployed

Your changes have been deployed to the preview site:

🔗 Preview URL: https://esa-apex.github.io/apex-algorithms-catalogue-web/pr-preview/pr-490/

This preview will be updated automatically when you push new changes to your PR.

[HansVRP]

@JanssenBrm @VictorVerhaert ready to check. I have opted for a more adaptive benchmark where we look at the average and the std. Depending on the nr of successful runs the benchmark becomes more determinantal

[HansVRP]

@JanssenBrm @JeroenVerstraelen @VictorVerhaert all feedback is welcome

[VictorVerhaert]

Two small optional comment aimed at trying to prevent false fails. One more question: could you try and run it using github actions and see how it behaves in practice?
Otherwise the pr looks clean

[HansVRP]

seems to be working well:

https://jenkins.vgt.vito.be/job/openEO/job/openeo-apex-benchmarks-handpicked-run/61/

[HansVRP]

ended up making some changes seeing the behavior on jenkins.

So I did add explicit time limitations on data to use and I made cost the sole gating value which can determine a failure. All other values were to volatile, especially for these small benchmarks which hover around 4 credits.

[HansVRP]

(blocked by #547)

[HansVRP]

@JanssenBrm @soxofaan Could the two of you also take a look at this proposal of regression testing?

In general I look into the merged parquet for the last X months and per usage metric/cost get all valid metrics for succesful runs.

Based on that I calculate the median and the MAD and convert it into a standard deviation.

For now the test will fail if the measured cost > median +3.5 x MAD which seemed sensible for some tests I have ran.

The other metrics fluctuate quite a lot, so for those I only log warnings in case of a failure

[soxofaan]

short on time here, so I could only give this a quick look
I'm a bit worried about the design here: if I understand correctly, you are adding hundreds of lines of logic in the main test_run_benchmark code path, involving (S3) io, pandas manipulations, which will make the main code path (already known to be brittle and trigger happy), even more brittle and hard to debug.
Can't the performance regression be added as a separate post-processing tool run, working on test suite output?

[HansVRP]

@soxofaan most of the code indeed relates to getting a good statistic out of this merger parquet file which can be used to determine there occurred regression or not. Ideally it would be cached somewhere locally such that we would not need to calculate the baseline on runtime each time.

am I correct that this is what you are proposing as well?

[HansVRP]

what I can do is move the baseline calculation to a separate github workflow and trigger it weekly to recompute the baseline.

in the actual test phase we would read in said file from S3 and check for regression?

[soxofaan]

no I'm not talking about caching (I'm not sure there is even something useful to do with caching in a test_run_benchmark context),
What I mean is to keep the pytest ... test_run_benchmark run/workflow to the bare essentials (for stability and to-the-point results)
and do all the other analysis, trend watching, ... in separate tools or workflows (to allow experimentation, iteration, interactivity)

[HansVRP]

makes sense; I'll split both workflows up, let the regression trigger weekly and create separate github issues!