Potential Issues with running of tests

[AlexanderKaran]

Feedback from NextJS Team:

Each framework runs in a separate matrix VM, so comparisons include between-runner variance.

[AlexanderKaran]

Currently, we have an EC2 instance for each framework, which runs each test repetition. For example, if an install test is run five times, we run it five times on that EC2 instance.

Depot provides consistent instance types, and tests have already proven stable, with tiny, tiny variances between runs.

However, this could cause an issue if one instance is faulty. We could run all tests on one, but then we have to be super, super careful to clean up.

The best solution is either to leave it as is and explain this in the methodology. As the tests get re-run a lot, or we run each test repetion on its own EC2 and randomise the order of the tests and frameworks running them.

[AlexanderKaran]

Another option as well could be to leave tests as they are and re-run the whole set a few times and also run tests regularly on a schedule

[AlexanderKaran]

Also now added this: #412

Which I wonder means we don't even really need to do anything here. If we pick up a big variance between tests or a big change between a tests current and new variance in the PR which merges the stats we just run again?

[43081j]

i think so. its probably fine for now and we can re-visit later if we need to.

depot seems stable so far and we can just re-run whenever its not.

[AlexanderKaran]

@43081j Just so we are on the same page, I won't make changes to how the tests run, but I will:

• Publish some tables in the methodology pages that include:
  • Variance between all runs of a test
  • Variance between a full test suite

I'll also add a section that flags whether the swing has changed too much in a PR that updates the stats, and finally ensure the tests run at least twice a week at a random time.

Later, we can work on turning the variance into a confidence score; the lower it is, the better.