feat(projectconfig): accept [tests] and [test-groups] schema in config by bhagyapathak · Pull Request #229 · microsoft/azure-linux-dev-tools

bhagyapathak · 2026-06-04T10:10:19Z

Adds a parse-only schema for declaring tests and reusable test groups in project TOML, plus matching per-component and per-image test reference lists.

Copilot

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Adds support for a “new-shape” test configuration schema by introducing first-class project-level test definitions and groups, and enabling images/components to reference them.

Changes:

Introduces TestDefinition, TestGroup, TestRef, and ComponentTestsConfig types for the new schema.
Extends ConfigFile, ImageTestsConfig, and ComponentConfig to carry new test/group references.
Updates fingerprint decision test to exclude component test-selection metadata from build inputs.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
internal/projectconfig/tests.go	Adds the new schema types for tests, groups, and references.
internal/projectconfig/configfile.go	Adds top-level `[tests]` and `[test-groups]` maps to the config shape.
internal/projectconfig/image.go	Allows images to reference tests/groups via `ImageTestsConfig.Tests`.
internal/projectconfig/component.go	Allows components to reference tests/groups and deep-copies the new field.
internal/projectconfig/fingerprint_test.go	Excludes `ComponentConfig.Tests` from fingerprint build inputs.

Copilot

Pull request overview

Copilot reviewed 12 out of 12 changed files in this pull request and generated 4 comments.

Copilot

Pull request overview

Copilot reviewed 12 out of 12 changed files in this pull request and generated 3 comments.

Copilot

Pull request overview

Copilot reviewed 12 out of 12 changed files in this pull request and generated 5 comments.

Copilot

Pull request overview

Copilot reviewed 17 out of 17 changed files in this pull request and generated 3 comments.

Copilot

Pull request overview

Copilot reviewed 18 out of 18 changed files in this pull request and generated 4 comments.

+func (TestRef) JSONSchemaExtend(schema *jsonschema.Schema) {
+	// Exactly one of name|group: encoded as oneOf with `required` on each and
+	// `not` excluding the other, which forbids both `{}` and `{name, group}`.
+	schema.OneOf = []*jsonschema.Schema{
+		{Required: []string{"name"}, Not: &jsonschema.Schema{Required: []string{"group"}}},
+		{Required: []string{"group"}, Not: &jsonschema.Schema{Required: []string{"name"}}},
+	}


liunan-ms

The new shape seems a superset of tmt, lisa, and pytest, other fields like required-capabilities, reusable groups, and per-component references. Is this intended as the next-generation replacement for test-suites? If yes, is there a migration path designed?

reubeno · 2026-06-24T15:33:12Z

Thanks for your patience -- I went through this in detail yesterday; lots of goodness here. I've created a branch based on the work that also adds some additional proposed changes. I'll be sending it out during my day today.

Some of my main comments/questions:

We need to unify the test-suites and test elements one way or the other.
There's some additional validation for the test references that I think will be helpful to catch typos/dangling references.
Can we better constrain the test kinds?
There are some opportunities for additional unit testing of some of the new code that I think will be worthwhile. Copilot can definitely help with some of it.
Nested test groups will bring some complexity; how do we imagine they'll be used?

I'll follow up with more details later today.

…g files

Copilot

Pull request overview

Copilot reviewed 19 out of 19 changed files in this pull request and generated 7 comments.

+        "kind": {
+          "items": {
+            "type": "string"
+          },
+          "type": "array",
+          "title": "Kind",
+          "description": "Test kind hints (e.g. functional or performance)"
+        },


Copilot

Pull request overview

Copilot reviewed 19 out of 19 changed files in this pull request and generated 3 comments.

+	// Type identifies the framework/runner. Required, and constrained to the
+	// closed enum in the schema tag at the schema layer. The loader still
+	// accepts unknown values permissively; the resolver is the source of truth.
+	Type string `toml:"type" json:"type" jsonschema:"required,title=Type,description=Test framework type,enum=pytest,enum=lisa,enum=tmt"`


+// TestRef is a reference to either a test (by name) or another group (by name).
+// Exactly one of Name or Group should be set; semantic validation is the resolver's
+// responsibility.


+	if err := validateTestDefinitions(f.Tests); err != nil {
+		return err
+	}
+
+	if err := validateNewTestReferences(f); err != nil {
+		return err
+	}


bhagyapathak · 2026-07-02T15:33:35Z

The new shape seems a superset of tmt, lisa, and pytest, other fields like required-capabilities, reusable groups, and per-component references. Is this intended as the next-generation replacement for test-suites? If yes, is there a migration path designed?

Thank you @liunan-ms for pointing this out.
Yes, that is the intent.
The new tests/test-groups model is designed as the next-generation superset of test-suites: it keeps framework-specific configs (pytest/lisa/tmt) while adding capabilities like required-capabilities, reusable groups, and per-component/per-image references. I am working on migration plan, will update it in some time.

bhagyapathak · 2026-07-02T15:55:43Z

Thanks for your patience -- I went through this in detail yesterday; lots of goodness here. I've created a branch based on the work that also adds some additional proposed changes. I'll be sending it out during my day today.

Some of my main comments/questions:

We need to unify the test-suites and test elements one way or the other.

There's some additional validation for the test references that I think will be helpful to catch typos/dangling references.

Can we better constrain the test kinds?

There are some opportunities for additional unit testing of some of the new code that I think will be worthwhile. Copilot can definitely help with some of it.

Nested test groups will bring some complexity; how do we imagine they'll be used?

I'll follow up with more details later today.

Hi @reubeno
Thanks for the detailed feedback and for taking the time to review this so thoroughly.

For this iteration, I have already tightened a few areas:

Added load-time validation for test references to catch typos and dangling references earlier.
Added restrictions around nested test groups to avoid introducing unnecessary complexity right now.
Constrained test kinds to a tighter allowed set.

One important migration concern is ControlTower compatibility. Today, ControlTower consumes testSuites and derives TestSuiteDetails from that contract. If we remove those fields before ControlTower is updated, we risk missing test metadata and skipping validation task wiring.

To avoid breakage, I propose this rollout:

Make tests canonical internally only.
Continue accepting legacy test-suites input and normalize to canonical tests immediately after parse.
Continue emitting compatibility suite-shaped metadata projected from canonical tests until ControlTower migrates.
Migrate ControlTower to consume canonical tests and test-groups
Remove compatibility projection only after ControlTower rollout is complete.

I agree with your points on unifying test-suites and tests strengthening validation and expanding unit coverage. I will incorporate those as part of the migration work.

I am looking forward to your suggestions on the migration.
cc @liunan-ms For migration plan

… with 2 updates (microsoft#259) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…bot-github-actions group across 1 directory (microsoft#260) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…directories with 1 update (microsoft#265) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…crosoft#212) Co-authored-by: Nan Liu <108544011+liunan-ms@users.noreply.github.com>

Co-authored-by: Nan Liu <108544011+liunan-ms@users.noreply.github.com>

…ration (microsoft#261) Co-authored-by: Antonio Salinas <asalinas@microsoft.com>

…ommits and enum upstream-status - Rename OverlayCategoryBackportDistGit -> OverlayCategoryUpstreamBackport (value 'backport-dist-git' -> 'upstream-backport'). Widens the category to cover Fedora dist-git and native OSS backports alike. - Replace BugRef with reusable URLRef (URL-carrying struct with room for tracker/source-specific fields). Metadata.Commits is now []URLRef instead of []string; Metadata.Bugs is []URLRef. - Replace tri-state Upstreamable *bool with typed OverlayUpstreamStatus enum (upstreamed, upstreamable, needs-upstream-hook, inapplicable); empty string = not-yet-assessed. - Regenerate schema and scenario snapshots; update overlays.md user reference to match the new field names, enum values, and TOML examples.

…karound Broaden the category to cover any overlay that is explicitly intended to be dropped once an upstream or environmental fix lands, including (but not limited to) workarounds for dependencies not yet imported into AZL. Also broaden the azl-branding-policy to cover fixes for upstream code that hard-codes Fedora identity strings

Add an optional [component-groups.<name>.metadata] block reusing the existing OverlayMetadata schema (required category plus commits, bugs, and upstreamable). Metadata is documentation only and does not affect how group members are resolved or built. Validation is wired into ConfigFile.Validate. Regenerate JSON schema; update reference docs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…PI drift The component-groups metadata PR was based on the pre-rework OverlayMetadata API and landed after that rework, leaving main red: - loader_test.go referenced removed identifiers (`OverlayCategoryBackportDistGit`, `BugRef`, `Upstreamable`) and inline TOML using the old `upstreamable` field. Migrated it to `OverlayCategoryUpstreamBackport`, `URLRef`, and `upstream-status`. - component-groups.md linked to a nonexistent `#bug-references` anchor in overlays.md. Updated to the actual `#url-references` heading. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…crosoft#267) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Co-authored-by: Antonio Salinas <asalinas@microsoft.com>

…g files

Copilot

Pull request overview

Copilot reviewed 114 out of 117 changed files in this pull request and generated 7 comments.

+	for _, ref := range refs {
+		switch {


+// findRuff locates the ruff executable on the PATH. ruff is a Python tool and is not managed via the Go
+// tools modules, so developers must install it themselves (the VS Code ruff extension bundles a copy that
+// is not exposed on the PATH).
+func findRuff() (string, error) {
+	ruffPath, err := exec.LookPath("ruff")
+	if err != nil {
+		return "", fmt.Errorf("%w: install it with 'pip install ruff' or see "+
+			"https://docs.astral.sh/ruff/installation/: %w", ErrRuffNotFound, err)
+	}
+
+	return ruffPath, nil
+}


+// findPyright locates the pyright executable on the PATH. Like ruff, pyright is a Python tool that is not
+// managed via the Go tools modules, so developers must install it themselves.
+func findPyright() (string, error) {
+	pyrightPath, err := exec.LookPath("pyright")
+	if err != nil {
+		return "", fmt.Errorf("%w: install it with 'pip install pyright' or see "+
+			"https://microsoft.github.io/pyright/#/installation: %w", ErrPyrightNotFnd, err)
+	}
+
+	return pyrightPath, nil
+}


+    permissions:
+      contents: read
+    steps:
+      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2


+// mergeTests merges individual test definitions from a loaded config file into the
+// resolved config. Duplicate test names are not allowed.
+func mergeTests(resolvedCfg *ProjectConfig, loadedCfg *ConfigFile) error {
+	for testName, testDef := range loadedCfg.Tests {
+		if _, ok := resolvedCfg.Tests[testName]; ok {
+			return fmt.Errorf("%w: test %#q", ErrDuplicateTestSuites, testName)
+		}
+
+		resolvedCfg.Tests[testName] = testDef.WithAbsolutePaths(loadedCfg.dir)
+	}
+
+	return nil
+}
+
+// mergeTestGroups merges named test groups from a loaded config file into the
+// resolved config. Duplicate group names are not allowed.
+func mergeTestGroups(resolvedCfg *ProjectConfig, loadedCfg *ConfigFile) error {
+	for groupName, group := range loadedCfg.TestGroups {
+		if _, ok := resolvedCfg.TestGroups[groupName]; ok {
+			return fmt.Errorf("%w: test group %#q", ErrDuplicateTestSuites, groupName)
+		}


+		if cfg != nil {
+			resolvedTests, err := cfg.ResolveComponentTests(comp.GetConfig())
+			if err != nil {
+				return nil, fmt.Errorf("failed to resolve tests for component %q:\n%w", comp.GetName(), err)
+			}


+// Changelog generates a draft changelog section for the next release from the Conventional Commit
+// history and prepends it to 'CHANGELOG.md', using git-cliff and the repo's 'cliff.toml'.
+//
+// The output is a *draft*: review and curate it into user-facing notes before releasing. See
+// docs/developer/how-to/releasing.md.
+//
+// git-cliff must be installed and on PATH; this target does not install it. The same target runs
+// locally and in CI (CI just installs git-cliff first), so there is a single changelog flow.
+func Changelog() error {


Copilot AI review requested due to automatic review settings June 4, 2026 10:10

bhagyapathak changed the title ~~feat(projectconfig): accept [tests] and [test-groups] schema in confi…~~ feat(projectconfig): accept [tests] and [test-groups] schema in config Jun 4, 2026

Copilot AI reviewed Jun 4, 2026

View reviewed changes

Comment thread internal/projectconfig/configfile.go Outdated

Comment thread internal/projectconfig/image.go

Comment thread internal/projectconfig/configfile.go Outdated

Copilot AI review requested due to automatic review settings June 4, 2026 16:03

Copilot AI reviewed Jun 4, 2026

View reviewed changes

Comment thread schemas/azldev.schema.json

Comment thread schemas/azldev.schema.json

Comment thread internal/projectconfig/tests.go

Comment thread docs/user/reference/config/tests.md

bhagyapathak force-pushed the bhagya/azldev-test-schema branch from f0e30ee to b44cb81 Compare June 5, 2026 13:57

Copilot AI review requested due to automatic review settings June 5, 2026 15:37

bhagyapathak force-pushed the bhagya/azldev-test-schema branch from b44cb81 to f7dbf0e Compare June 5, 2026 15:37

Copilot AI reviewed Jun 5, 2026

View reviewed changes

Comment thread internal/projectconfig/image.go

Comment thread internal/projectconfig/configfile.go

Comment thread internal/projectconfig/tests.go Outdated

bhagyapathak force-pushed the bhagya/azldev-test-schema branch from f7dbf0e to c01da5d Compare June 8, 2026 11:17

Copilot AI review requested due to automatic review settings June 8, 2026 11:28

bhagyapathak force-pushed the bhagya/azldev-test-schema branch from c01da5d to bd478bd Compare June 8, 2026 11:28

Copilot AI reviewed Jun 8, 2026

View reviewed changes

Comment thread schemas/azldev.schema.json

Comment thread schemas/azldev.schema.json

Comment thread schemas/azldev.schema.json

Comment thread internal/projectconfig/component.go Outdated

Comment thread docs/user/reference/config/tests.md

bhagyapathak force-pushed the bhagya/azldev-test-schema branch from bd478bd to fafc0c1 Compare June 8, 2026 11:41

Copilot AI review requested due to automatic review settings June 8, 2026 15:26

bhagyapathak force-pushed the bhagya/azldev-test-schema branch from fafc0c1 to f1e6eb3 Compare June 8, 2026 15:26

Copilot AI reviewed Jun 8, 2026

View reviewed changes

Comment thread scenario/__snapshots__/TestSnapshots_config_generate-schema_stdout_1.snap

Comment thread internal/projectconfig/tests.go

Comment thread internal/projectconfig/testsuite_test.go Outdated

bhagyapathak force-pushed the bhagya/azldev-test-schema branch from f1e6eb3 to 86e4307 Compare June 8, 2026 16:34

Copilot AI review requested due to automatic review settings June 9, 2026 07:20

bhagyapathak force-pushed the bhagya/azldev-test-schema branch from 86e4307 to 9064645 Compare June 9, 2026 07:20

Copilot AI reviewed Jun 9, 2026

View reviewed changes

christopherco requested a review from liunan-ms June 18, 2026 05:27

liunan-ms reviewed Jun 18, 2026

View reviewed changes

Comment thread internal/projectconfig/tests.go

feat(projectconfig): accept [tests] and [test-groups] schema in confi…

8ab6361

…g files

bhagyapathak force-pushed the bhagya/azldev-test-schema branch from 9064645 to 8ab6361 Compare July 1, 2026 11:54

Copilot AI review requested due to automatic review settings July 2, 2026 14:31

Copilot started reviewing on behalf of bhagyapathak July 2, 2026 14:31 View session

Copilot AI reviewed Jul 2, 2026

View reviewed changes

Strengthen test metadata validation and reference integrity checks

c5fb905

bhagyapathak force-pushed the bhagya/azldev-test-schema branch from 7693b03 to c5fb905 Compare July 2, 2026 15:00

bhagyapathak requested a review from Copilot July 2, 2026 15:06

Copilot started reviewing on behalf of bhagyapathak July 2, 2026 15:06 View session

Copilot AI reviewed Jul 2, 2026

View reviewed changes

dmcilvaney and others added 18 commits July 15, 2026 16:20

chore(release): add release tools to mage (microsoft#254)

23e5447

build(deps): bump the dependabot-gomod-minor group across 1 directory…

5a03dd5

… with 2 updates (microsoft#259) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

build(deps): bump actions/checkout from 6.0.3 to 7.0.0 in the dependa…

33e0b5c

…bot-github-actions group across 1 directory (microsoft#260) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

build(deps): bump golang.org/x/net from 0.54.0 to 0.55.0 (microsoft#266)

b7a1318

Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

build(deps): bump the dependabot-gomod-tools-security group across 2 …

53c1af5

…directories with 1 update (microsoft#265) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

feat(cli): add history command to show changes made to components (mi…

ae390fc

…crosoft#212) Co-authored-by: Nan Liu <108544011+liunan-ms@users.noreply.github.com>

feat(overlays): handle empty sections explicitly (microsoft#208)

5917bdf

Co-authored-by: Nan Liu <108544011+liunan-ms@users.noreply.github.com>

feat: in verbose mode have mock print all output (microsoft#231)

8ce989c

feat(source-files): add custom origin type for mock-based source gene…

c83c8c8

…ration (microsoft#261) Co-authored-by: Antonio Salinas <asalinas@microsoft.com>

ci: add python linters to cover future python code (microsoft#225)

c6e7441

build(deps): bump the dependabot-gomod-minor group with 3 updates (mi…

8925fdb

…crosoft#267) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

feat(source-files): generative source dependencies (microsoft#271)

4aa3d00

Co-authored-by: Antonio Salinas <asalinas@microsoft.com>

fix(mcp): fix issues with the MCP server mode (microsoft#272)

af8c411

feat(projectconfig): accept [tests] and [test-groups] schema in confi…

6188a64

…g files

Copilot AI review requested due to automatic review settings July 17, 2026 13:00

Copilot started reviewing on behalf of bhagyapathak July 17, 2026 13:01 View session

Copilot AI reviewed Jul 17, 2026

View reviewed changes

Uh oh!

Conversation

bhagyapathak commented Jun 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

liunan-ms left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

reubeno commented Jun 24, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

bhagyapathak commented Jul 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bhagyapathak commented Jul 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

bhagyapathak commented Jun 4, 2026 •

edited

Loading

bhagyapathak commented Jul 2, 2026 •

edited

Loading

bhagyapathak commented Jul 2, 2026 •

edited

Loading