Skip to content

CircleCI tests improvement #2023

@Didayolo

Description

@Didayolo

E2E tests (#2042)

  • E2E tests are failing on Django 4.2.0 pull request (Django to 4.2.0 #1959). We're remove it them but need to add them again.

  • E2E tests are randomly failing (~ 1 time out of 4).

Podman test

More E2E tests

Add to the E2E tests:

  • A Codabench results submission (Update E2E tests #2302)
  • Another Codabench code submission, typically Mini-AutoML (Update E2E tests #2302)
  • A failing submission (Update E2E tests #2302)
  • Several submissions made to the same benchmark
  • Add a "ingestion during scoring" test
  • Multi-task tests are disabled on CI
  • Test submission on custom queue
  • All tests on codalab/codalab-legacy:py312? It is much lighter

E2E test files:
https://github.com/codalab/codabench/tree/develop/tests/test_files/competitions
https://github.com/codalab/codabench/tree/develop/tests/test_files/submissions

The test:
https://github.com/codalab/codabench/blob/develop/tests/test_submission.py

The templates:
https://github.com/codalab/competition-examples/

Improve E2E

This wait we do not wait in case of failure, and we have more informative tests.

Metadata

Metadata

Assignees

No one assigned

    Labels

    EnhancementFeature suggestions and improvementsPost-itInternal ideas

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions