Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 22 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -78,6 +78,28 @@ jobs:
format: go-coverprofile
version: v5.19.0

mocha-compatibility:
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
mocha: ["8.4.0", "9.2.2", "10.8.2", "11.7.6"]

steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1

- name: Set up Go
uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v5
with:
go-version: "1.26.3"

- name: Test Mocha adapter
run: |
mocha_dir="${RUNNER_TEMP}/mocha-${{ matrix.mocha }}"
npm install --prefix "${mocha_dir}" "mocha@${{ matrix.mocha }}"
DDTEST_MOCHA_NODE_MODULES="${mocha_dir}/node_modules" \
go test -v ./internal/framework -run TestMochaAdapterIntegration

lint:
runs-on: ubuntu-latest
steps:
Expand Down
27 changes: 22 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ Currently supported:

- Ruby with RSpec or Minitest.
- Python with pytest.
- JavaScript with Jest or Vitest.
- JavaScript with Jest, Mocha, or Vitest.

## Prerequisites

Expand All @@ -23,8 +23,9 @@ Minimum supported library and runtime requirements:

- Ruby requires the `datadog-ci` gem **1.31.0** or higher.
- Python requires the `ddtrace` package **4.11.0** or higher and `pytest`.
- JavaScript requires the `dd-trace` package **5.111.0** or higher, Node.js, and
Jest or Vitest 1.6 or higher.
- JavaScript requires the `dd-trace` package **5.111.0** or higher and Node.js.
Mocha support requires Mocha 8 or higher; Vitest support requires Vitest 1.6
or higher.

For instructions on setting up Test Optimization, see the [Datadog Test Optimization documentation](https://docs.datadoghq.com/tests/setup/).

Expand Down Expand Up @@ -89,6 +90,16 @@ ddtest plan \
--max-parallelism 32
```

For JavaScript/Mocha:

```bash
ddtest plan \
--platform javascript \
--framework mocha \
--min-parallelism 8 \
--max-parallelism 32
```

This prepares the plan and writes it to `.testoptimization/` folder for later reuse.
Copy `.testoptimization/` to any CI job that runs `ddtest run` or reads DDTest's
plan file lists. For the full file layout and formats, see
Expand Down Expand Up @@ -122,6 +133,12 @@ For JavaScript/Vitest:
ddtest run --platform javascript --framework vitest
```

For JavaScript/Mocha:

```bash
ddtest run --platform javascript --framework mocha
```

For CI-node mode, worker environment variables, custom commands, and
parallelism details, see [Running DDTest](docs/running.md).

Expand All @@ -130,8 +147,8 @@ parallelism details, see [Running DDTest](docs/running.md).
| CLI flag | What it does |
| --- | --- |
| `--platform` | Language/platform. Currently supported: `ruby`, `python`, `javascript`. |
| `--framework` | Test framework. Currently supported: `rspec`, `minitest`, `pytest`, `jest`, `vitest`. |
| `--command` | Override the default base command for supported framework modes. Currently used by RSpec and Minitest run/discovery, and Jest and Vitest run/discovery. For pytest, use `PYTEST_ADDOPTS` for pytest flags. |
| `--framework` | Test framework. Currently supported: `rspec`, `minitest`, `pytest`, `jest`, `mocha`, `vitest`. |
| `--command` | Override the default base command for supported framework modes. Currently used by RSpec and Minitest run/discovery, and Jest, Mocha, and Vitest run/discovery. For pytest, use `PYTEST_ADDOPTS` for pytest flags. |
| `--min-parallelism` | Minimum CI node or worker count DDTest considers when planning. |
| `--max-parallelism` | Maximum CI node or worker count DDTest considers when planning. |
| `--target-time` | Target wall time DDTest tries to satisfy when selecting parallelism. |
Expand Down
12 changes: 12 additions & 0 deletions docs/best_practices.md
Original file line number Diff line number Diff line change
Expand Up @@ -179,6 +179,18 @@ DDTest passes the Vitest arguments to its config-aware discovery API. DDTest
appends the selected test files during execution. Do not include test files or a
`--` separator in the command.

## Mocha Support

Use a command that invokes Mocha directly when passing framework flags:

```bash
ddtest run --platform javascript --framework mocha --command "pnpm exec mocha --parallel"
```

Do not include test files or a `--` separator. DDTest reads Mocha's effective
configuration for discovery and replaces configured `spec` inputs with the
files assigned to each worker during execution.

## Minitest Support In Non-Rails Projects

We use `bundle exec rake test` command when we don't detect `rails` command to
Expand Down
2 changes: 1 addition & 1 deletion docs/layout.md
Original file line number Diff line number Diff line change
Expand Up @@ -237,7 +237,7 @@ Fields:
| --- | --- |
| `name` | Test name reported by the framework. |
| `suite` | Test suite name reported by the framework. |
| `module` | Framework module name, such as `rspec`, `minitest`, `pytest`, `jest`, or `vitest`. |
| `module` | Framework module name, such as `rspec`, `minitest`, `pytest`, `jest`, `mocha`, or `vitest`. |
| `parameters` | Serialized test parameters. |
| `suiteSourceFile` | Source file containing the suite. |

Expand Down
43 changes: 41 additions & 2 deletions docs/running.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,12 @@ For JavaScript/Vitest:
ddtest run --platform javascript --framework vitest
```

For JavaScript/Mocha:

```bash
ddtest run --platform javascript --framework mocha
```

On one CI node, the default `--min-parallelism` and `--max-parallelism` equal
the available physical CPU core count, so DDTest can start one worker per
physical core without defaulting to one worker per hyperthread.
Expand Down Expand Up @@ -69,6 +75,12 @@ For JavaScript/Vitest:
ddtest run --platform javascript --framework vitest --ci-node <CI_NODE_INDEX>
```

For JavaScript/Mocha:

```bash
ddtest run --platform javascript --framework mocha --ci-node <CI_NODE_INDEX>
```

In CI-node mode, DDTest uses one local worker by default so database and other
per-worker resources stay easy to isolate. To fan out within each CI node, set
`--ci-node-workers` to a positive integer, or use `--ci-node-workers ncpu` to
Expand Down Expand Up @@ -102,8 +114,8 @@ starting each worker.

Use `--command` to override the framework's default base test command where
supported. DDTest currently applies this override to RSpec run and full
discovery, Minitest run and full discovery, and Jest and Vitest run and file
discovery:
discovery, Minitest run and full discovery, and Jest, Mocha, and Vitest run and
file discovery:

```bash
ddtest run --platform ruby --framework rspec --command "bundle exec rspec --profile"
Expand All @@ -116,6 +128,14 @@ and `--runTestsByPath <files>` during execution:
ddtest run --platform javascript --framework jest --command "pnpm jest --runInBand"
```

For JavaScript/Mocha, the command must invoke Mocha directly. DDTest loads its
effective configuration, discovers files without loading test modules, and
replaces configured `spec` entries with each worker's assigned files:

```bash
ddtest run --platform javascript --framework mocha --command "pnpm exec mocha --parallel"
```

For JavaScript/Vitest, the command must invoke Vitest directly. During planning,
DDTest uses `list --filesOnly --json` on Vitest 2.0 and newer and the config-aware
discovery API on Vitest 1.6. It appends selected files during execution:
Expand Down Expand Up @@ -175,6 +195,25 @@ as Jest's `--testMatch`.
DDTest prepends `-r dd-trace/ci/init` to `NODE_OPTIONS` for worker processes
unless `NODE_OPTIONS` already loads `dd-trace/ci/init`.

## Mocha Discovery And Instrumentation

DDTest supports Mocha 8 and newer. It uses Mocha's own option loader and file
collector, so discovery honors `.mocharc.*`, the `mocha` property in
`package.json`, `MOCHA_OPTIONS` on versions that support it, `spec`,
`extension`, `recursive`, `ignore`, and `sort` without loading test modules or
running hooks. Files configured with `--file` are treated as shared setup and
are loaded by every worker rather than being partitioned.

Mocha normally adds positional files to configured `spec` patterns. During a
DDTest run, the adapter replaces that merged list with the worker's assigned
files while preserving the rest of the effective Mocha configuration. This
prevents every worker from running the entire configured suite.

DDTest uses the local `node_modules/.bin/mocha` when present and otherwise
expects Mocha to be resolvable from the current project. Discovery removes
`-r dd-trace/ci/init` from `NODE_OPTIONS`; test runs retain it for Test
Optimization instrumentation.

## Vitest Discovery And Instrumentation

For JavaScript/Vitest 2.0 or higher, DDTest discovers test files with Vitest's
Expand Down
6 changes: 3 additions & 3 deletions docs/settings.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,16 +6,16 @@ CLI flags take precedence over environment variables.
| CLI flag | Environment variable | Env alias | Default | What it does |
| --- | --- | --- | ---: | --- |
| `--platform` | `DD_TEST_OPTIMIZATION_RUNNER_PLATFORM` | | `ruby` | Language/platform. Currently supported: `ruby`, `python`, `javascript`. |
| `--framework` | `DD_TEST_OPTIMIZATION_RUNNER_FRAMEWORK` | | `rspec` | Test framework. Currently supported: `rspec`, `minitest`, `pytest`, `jest`, `vitest`. |
| `--command` | `DD_TEST_OPTIMIZATION_RUNNER_COMMAND` | | `""` | Override the default base test command for supported framework modes. Currently used by RSpec and Minitest run/discovery, and Jest and Vitest run/discovery; pytest ignores it. DDTest appends selected tests and framework-specific flags. For pytest, use `PYTEST_ADDOPTS` for pytest flags. |
| `--framework` | `DD_TEST_OPTIMIZATION_RUNNER_FRAMEWORK` | | `rspec` | Test framework. Currently supported: `rspec`, `minitest`, `pytest`, `jest`, `mocha`, `vitest`. |
| `--command` | `DD_TEST_OPTIMIZATION_RUNNER_COMMAND` | | `""` | Override the default base test command for supported framework modes. Currently used by RSpec and Minitest run/discovery, and Jest, Mocha, and Vitest run/discovery; pytest ignores it. DDTest appends selected tests and framework-specific flags. For pytest, use `PYTEST_ADDOPTS` for pytest flags. |
| `--min-parallelism` | `DD_TEST_OPTIMIZATION_RUNNER_MIN_PARALLELISM` | | physical CPU count | Minimum count DDTest considers when planning. Interpret it as CI nodes in CI-node mode, or workers in a single-node run. |
| `--max-parallelism` | `DD_TEST_OPTIMIZATION_RUNNER_MAX_PARALLELISM` | | physical CPU count | Maximum count DDTest considers when planning. Interpret it as CI nodes in CI-node mode, or workers in a single-node run. |
| `--ci-job-overhead` | `DD_TEST_OPTIMIZATION_RUNNER_CI_JOB_OVERHEAD` | | `25s` | Modeled overhead for adding one more CI node. Accepts durations such as `25s`, `1m`, `1500ms`, or `0s` to disable this bias. Increase it to use fewer CI nodes; decrease it to prefer faster wall time. |
| `--target-time` | `DD_TEST_OPTIMIZATION_RUNNER_TARGET_TIME` | | `0s` | Target wall time for the selected split. Accepts durations such as `10m`, `300s`, `1500ms`, or `0s` to disable the target. DDTest first considers splits at or below this wall time; if none are possible within the min/max parallelism range, it warns and selects the split with the lowest expected wall time, ignoring CI job overhead, to get as close as possible to the target. |
| `--ci-node` | `DD_TEST_OPTIMIZATION_RUNNER_CI_NODE` | | `-1` (off) | Restrict this run to files assigned to CI node **N** (0-indexed). |
| `--ci-node-workers` | `DD_TEST_OPTIMIZATION_RUNNER_CI_NODE_WORKERS` | | `1` | Number of workers to start on this CI node. Use a positive integer, or `ncpu` to use the node's available physical CPU cores. |
| `--worker-env` | `DD_TEST_OPTIMIZATION_RUNNER_WORKER_ENV` | | `""` | Template env vars per worker: `--worker-env "DATABASE_NAME_TEST=app_test{{nodeIndex}}_{{workerIndex}}"`. `{{nodeIndex}}` is the CI node index (`0` for single-node runs); `{{workerIndex}}` is the worker process index within that CI node. |
| `--tests-location` | `DD_TEST_OPTIMIZATION_RUNNER_TESTS_LOCATION` | `KNAPSACK_PRO_TEST_FILE_PATTERN` | `""` | Custom glob pattern to discover test files, such as `--tests-location "custom/spec/**/*_spec.rb"`, `--tests-location "tests/**/*_test.py"`, or `--tests-location "packages/**/__tests__/**/*.test.ts"`. Defaults to `spec/**/*_spec.rb` for RSpec, `test/**/*_test.rb` for Minitest, pytest config or `**/{test_*,*_test}.py` for pytest, and each JavaScript framework's configured/default test matching for Jest and Vitest. |
| `--tests-location` | `DD_TEST_OPTIMIZATION_RUNNER_TESTS_LOCATION` | `KNAPSACK_PRO_TEST_FILE_PATTERN` | `""` | Custom glob pattern to discover test files, such as `--tests-location "custom/spec/**/*_spec.rb"`, `--tests-location "tests/**/*_test.py"`, or `--tests-location "packages/**/__tests__/**/*.test.ts"`. Defaults to `spec/**/*_spec.rb` for RSpec, `test/**/*_test.rb` for Minitest, pytest config or `**/{test_*,*_test}.py` for pytest, and each JavaScript framework's configured/default test matching for Jest, Mocha, and Vitest. |
| `--tests-exclude-pattern` | `DD_TEST_OPTIMIZATION_RUNNER_TESTS_EXCLUDE_PATTERN` | `KNAPSACK_PRO_TEST_FILE_EXCLUDE_PATTERN` | `""` | Glob pattern to exclude test files from discovery, such as `--tests-exclude-pattern "spec/system/**/*_spec.rb"`. |
| `--test-discovery-cache` | `DD_TEST_OPTIMIZATION_RUNNER_TEST_DISCOVERY_CACHE` | | `""` | Path to a restored test discovery cache file. DDTest imports it before planning and refreshes the internal discovery cache after successful full discovery. |
| `--force-full-test-discovery` | `DD_TEST_OPTIMIZATION_RUNNER_FORCE_FULL_TEST_DISCOVERY` | | `false` | Force full test discovery when the framework supports it, including in suite-level skipping mode. |
Expand Down
12 changes: 12 additions & 0 deletions docs/third-party-runners.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,18 @@ if [ -s .testoptimization/runner/test-files.txt ]; then
fi
```

## Mocha

When another runner consumes DDTest's Mocha file list, load Test Optimization
initialization before invoking Mocha:

```bash
export NODE_OPTIONS="-r dd-trace/ci/init${NODE_OPTIONS:+ $NODE_OPTIONS}"
if [ -s .testoptimization/runner/test-files.txt ]; then
xargs ./node_modules/.bin/mocha < .testoptimization/runner/test-files.txt
fi
```

## Custom Runners

Read `.testoptimization/runner/test-files.txt` when your runner should handle
Expand Down
Loading
Loading