Build/Test Tools: Publish PHPUnit timing metrics to CodeVitals by lancewillett · Pull Request #13083 · WordPress/wordpress-develop

lancewillett · 2026-08-16T21:32:07Z

Trac ticket: https://core.trac.wordpress.org/ticket/65887

What this changes

The canonical PHPUnit job publishes six aggregate timing metrics to the existing public CodeVitals dashboard after every push to trunk:

PHP 8.5;
MySQL 8.4;
single site;
default domain and cache configuration.

The publisher reads the existing JUnit report immediately after the main test suite, before later test-group runs replace the file. It streams the XML, creates a small JSON payload, and sends no test names or per-test records.

This complements #13070. That pull request preserves the raw JUnit report for detailed analysis; this one adds the visible trend layer.

Metrics

Key	Display name	Unit	Default
`phpunit-suite-time`	PHPUnit main suite time	s	Visible
`phpunit-p95-test-time`	PHPUnit test time (p95)	ms	Other
`phpunit-p99-test-time`	PHPUnit test time (p99)	ms	Other
`phpunit-max-test-time`	PHPUnit maximum test time	ms	Other
`phpunit-tests-over-500ms`	PHPUnit tests over 500 ms	tests	Other
`phpunit-tests-over-1s`	PHPUnit tests over 1 s	tests	Other

Keeping one metric visible by default avoids crowding the existing dashboard. The other five remain available from the Other selector for diagnosis.

The definitions must exist in CodeVitals before the first submission. Suggested priorities are 100–105 in the order above, with a regression noise floor of 0 until trunk runs establish a baseline.

The workflow also verifies that CodeVitals accepted all six metrics. A missing definition fails the step instead of silently dropping data.

How to validate

After the first trunk push following merge, open:

https://codevitals.run/public/WordPress/wordpress-develop/metrics

The PHPUnit main suite time card should appear. Select it to view the trend by commit. Use Other to add the percentile, maximum, and slow-test-count metrics.

The parser can be checked locally with:

npm run test:php -- --filter Tests_Includes_JUnit_Timing_Metrics

A previously captured CI report containing 30,871 timed test cases produced:

suite: 281.19193 seconds;
p95: 49.723 milliseconds;
p99: 100.294 milliseconds;
maximum: 3,446.888 milliseconds;
over 500 milliseconds: 31 tests;
over 1 second: 19 tests.

An authenticated CodeVitals submission remains a pre-merge validation item. The token stays in the existing CODEVITALS_PROJECT_TOKEN Actions secret and is never available to local or pull-request runs.

Testing

Focused PHPUnit tests passed: 4 tests, 6 assertions.
PHP syntax checks and WordPress PHP coding standards passed.
actionlint passed for both changed workflow files.
The command-line publisher returned all six numeric metrics from both focused and full CI JUnit reports.
The CodeVitals endpoint returned the expected authentication error for an invalid test token.

github-actions · 2026-08-16T21:34:55Z

Test using WordPress Playground

The changes in this pull request can previewed and tested using a WordPress Playground instance.

WordPress Playground is an experimental project that creates a full WordPress instance entirely within the browser.

Some things to be aware of

All changes will be lost when closing a tab with a Playground instance.
All changes will be lost when refreshing the page.
A fresh instance is created each time the link below is clicked.
Every time this pull request is updated, a new ZIP file containing all changes is created. If changes are not reflected in the Playground instance,
it's possible that the most recent build failed, or has not completed. Check the list of workflow runs to be sure.

For more details about these limitations and more, check out the Limitations page in the WordPress Playground documentation.

Test this pull request with WordPress Playground.

lancewillett · 2026-08-16T21:53:37Z

@johnbillion Thanks for suggesting CodeVitals on https://core.trac.wordpress.org/ticket/65887#comment:4.

CodeVitals requires GitHub repository admin access to manage metric definitions. Could you add these six definitions for WordPress/wordpress-develop?

Key	Display name	Unit	Priority	Visible by default
`phpunit-suite-time`	PHPUnit main suite time	s	100	Yes
`phpunit-p95-test-time`	PHPUnit test time (p95)	ms	101	No
`phpunit-p99-test-time`	PHPUnit test time (p99)	ms	102	No
`phpunit-max-test-time`	PHPUnit maximum test time	ms	103	No
`phpunit-tests-over-500ms`	PHPUnit tests over 500 ms	tests	104	No
`phpunit-tests-over-1s`	PHPUnit tests over 1 s	tests	105	No

The service silently drops unknown metric keys. This PR verifies that all six were accepted, so the scheduled publishing step will fail clearly if any definition is missing.

Once these exist, I can complete the authenticated submission and verify the public dashboard: https://codevitals.run/public/WordPress/wordpress-develop/metrics

github-actions · 2026-08-16T21:53:43Z

The following accounts have interacted with this PR and/or linked issues. I will continue to update these lists as activity occurs. You can also manually ask me to refresh this list by adding the props-bot label.

Unlinked Accounts

The following contributors have not linked their GitHub and WordPress.org accounts: @lance.willett@a8c.com.

Contributors, please read how to link your accounts to ensure your work is properly credited in WordPress releases.

Core Committers: Use this line as a base for the props when committing in SVN:

Props lancewillett, desrosj, adrianmoldovanwp.

To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook.

desrosj

Some small questions and change requests.

lancewillett · 2026-08-17T01:01:45Z

Thanks @desrosj -- fixed pushed bfd9637

is there any reason why we can not submit test results for every commit?

I originally limited this to weekly runs for storage and consistency, but updated it to publish after every push to trunk based on your feedback.

PR runs remain excluded because their branch-specific results are less comparable—and would create a LOT more storage that isn't that helpful for stats. At least, not now.

lancewillett · 2026-08-17T01:06:04Z

@adimoldovan Curious your take on the approach. Is there a better way to store the test run results and display it in a dashboard somewhere?

Copilot

Pull request overview

This PR adds a small pipeline to extract aggregate timing statistics from the PHPUnit JUnit report produced in CI and publish those aggregates to the existing public CodeVitals dashboard for trunk pushes, enabling trend tracking without exposing per-test details.

Changes:

Add an XML-streaming parser (WP_PHPUnit_Timing_Metrics) to compute suite time, p95/p99/max test time, and slow-test counts from a JUnit report.
Add a CLI publisher (prepare-timing-results.php) that produces a compact JSON payload for CodeVitals ingestion.
Extend the canonical PHPUnit CI job to publish and validate the six metrics after the main suite run.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
`tests/phpunit/includes/class-wp-phpunit-timing-metrics.php`	Implements JUnit XML parsing and aggregate timing metric extraction.
`tests/phpunit/prepare-timing-results.php`	CLI wrapper to generate the CodeVitals JSON payload from a JUnit file plus commit metadata.
`tests/phpunit/tests/includes/junitTimingMetrics.php`	Adds PHPUnit coverage for timing extraction logic and edge cases.
`.github/workflows/reusable-phpunit-tests-v3.yml`	Publishes timing metrics to CodeVitals from the canonical `trunk` push job and validates acceptance.
`.github/workflows/phpunit-tests.yml`	Passes the CodeVitals token secret through to the reusable PHPUnit workflow.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

desrosj · 2026-08-17T23:09:32Z

+		$file                    = tempnam( sys_get_temp_dir(), 'junit-timing-' );
+		$this->temporary_files[] = $file;
+		$suite_time_attribute    = null === $suite_time ? '' : sprintf( ' time="%s"', $suite_time );


I think that this makes sense. Any objection to this @lancewillett?

Good call. Pushed a follow-up (d450580) that fails the test explicitly when tempnam() returns false, so it can't reach unlink() in teardown.

desrosj

I think this is worth trying. There's one outstanding Copilot comment I think is worth including.

adimoldovan · 2026-08-18T11:48:59Z

@adimoldovan Curious your take on the approach. Is there a better way to store the test run results and display it in a dashboard somewhere?

@lancewillett The approach works if the goal is to store the trend. If what we want is signal to optimize slow tests, none of the six aggregates can name a test. #13070 already gives us time, file, and line per case, so publishing the top slow tests or those over a threshold as Github annotations would get us there. That could be a good signal in PRs too, so an author can spot a slow test they added.

…ics test. If tempnam() returns false, fail the test explicitly instead of storing false and emitting an unlink() warning during teardown. See #65887.

lancewillett · 2026-08-21T00:43:36Z

@adimoldovan Thanks for your feedback. Opened #13212 for flagging the slow tests by name using annotations.

lancewillett added 2 commits August 16, 2026 14:30

Build/Test Tools: Publish scheduled PHPUnit timing metrics

930525d

Build/Test Tools: Verify CodeVitals accepts timing metrics

7c4f94b

lancewillett marked this pull request as ready for review August 16, 2026 21:53

lancewillett requested review from desrosj and johnbillion and removed request for johnbillion August 16, 2026 21:53

desrosj requested changes Aug 16, 2026

View reviewed changes

Comment thread tests/phpunit/tests/includes/junitTimingMetrics.php Outdated

Comment thread .github/workflows/reusable-phpunit-tests-v3.yml Outdated

Comment thread tests/phpunit/prepare-timing-results.php

desrosj mentioned this pull request Aug 16, 2026

Build/Test Tools: Preserve PHPUnit timing results for use in metrics #13070

Open

Build/Test Tools: Address PHPUnit metrics review

bfd9637

lancewillett changed the title ~~Build/Test Tools: Publish scheduled PHPUnit timing metrics to CodeVitals~~ Build/Test Tools: Publish PHPUnit timing metrics to CodeVitals Aug 17, 2026

lancewillett requested review from adimoldovan and desrosj August 17, 2026 01:01

desrosj requested a lite review from Copilot August 17, 2026 22:58

Copilot started reviewing on behalf of desrosj August 17, 2026 22:59 View session

Merge branch 'trunk' into feat/65887-publish-phpunit-metrics

92f21b4

Copilot AI reviewed Aug 17, 2026

View reviewed changes

desrosj approved these changes Aug 17, 2026

View reviewed changes

lancewillett mentioned this pull request Aug 20, 2026

Build/Test Tools: Flag slow PHPUnit tests with annotations #13212

Open

Build/Test Tools: Guard against a failed tempnam() in the timing-metr…

d450580

…ics test. If tempnam() returns false, fail the test explicitly instead of storing false and emitting an unlink() warning during teardown. See #65887.

Uh oh!

Conversation

lancewillett commented Aug 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What this changes

Metrics

How to validate

Testing

Uh oh!

github-actions Bot commented Aug 16, 2026

Test using WordPress Playground

Some things to be aware of

Uh oh!

lancewillett commented Aug 16, 2026

Uh oh!

github-actions Bot commented Aug 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Unlinked Accounts

Uh oh!

desrosj left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

lancewillett commented Aug 17, 2026

Uh oh!

lancewillett commented Aug 17, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

desrosj Aug 17, 2026

Choose a reason for hiding this comment

Uh oh!

lancewillett Aug 21, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

desrosj left a comment

Choose a reason for hiding this comment

Uh oh!

adimoldovan commented Aug 18, 2026

Uh oh!

lancewillett commented Aug 21, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

lancewillett commented Aug 16, 2026 •

edited

Loading

github-actions Bot commented Aug 16, 2026 •

edited

Loading