AWS DevOps Agent Release Testing Action

A GitHub Action that triggers the AWS DevOps Release Testing Agent after deployment. It automatically runs UI or API tests against your application and reports results as a GitHub Check Run on your commit.

Choose Your Path

	Quick Test	Full Setup
Best for	Fast CI/CD validation, one-off tests, trying out the agent	Persistent config, authenticated testing, team-shared profiles
Setup time	~2 minutes	~10 minutes
Authentication	Guest mode only	Guest + authenticated (credentials, storage state)
Configuration	Inline in workflow file	Saved in Operator App, reusable across workflows
Additional options	Target URL, agent type, test requirement, API spec file	All of the above + personas, credentials (with MFA/TOTP)

---

Common Setup (Required for Both Paths)

Step 1: Enable Release Testing in Your Agent Space

1. Navigate to the AWS DevOps Agent Console
2. Select your Agent Space
3. Go to Capability Providers and add GitHub (install the GitHub App to your organization)

Step 2: Add Your GitHub Repository as a Pipeline Source

1. In your Agent Space, go to Pipeline Sources
2. Click Add source → select GitHub
3. Select your repository and confirm the association

Step 3: Create a Webhook Endpoint

1. In the Agent Space, go to Settings → Webhooks
2. Create a new webhook endpoint
3. Note down:
   • Webhook URL
   • Webhook Secret

Step 4: Configure GitHub Repository Secrets

In your GitHub repository, go to Settings → Secrets and variables → Actions → Repository secrets and add:

Secret	Description
DEVOPS_AGENT_WEBHOOK_URL	The webhook URL from Step 3
DEVOPS_AGENT_WEBHOOK_SECRET	The webhook signing secret from Step 3

---

Quick Test (Profileless)

After completing the Common Setup, you can trigger tests directly with just a target URL — no test profile needed.

UI Testing

name: Release Tests
on:
  push:
    branches: [main]

permissions:
  contents: read
  pull-requests: read

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - name: Trigger Release Tests
        uses: aws-actions/devops-agent-qa@v1
        with:
          webhook-url: ${{ secrets.DEVOPS_AGENT_WEBHOOK_URL }}
          webhook-secret: ${{ secrets.DEVOPS_AGENT_WEBHOOK_SECRET }}
          target-url: https://my-staging-app.example.com
          agent-type: ui
          test-requirement: "Verify the checkout flow after payment refactor"  # optional
        env:
          GITHUB_TOKEN: ${{ github.token }}

API Testing

      - name: Trigger API Tests
        uses: aws-actions/devops-agent-qa@v1
        with:
          webhook-url: ${{ secrets.DEVOPS_AGENT_WEBHOOK_URL }}
          webhook-secret: ${{ secrets.DEVOPS_AGENT_WEBHOOK_SECRET }}
          target-url: https://my-api.example.com
          agent-type: api
          api-spec-file: ./openapi.yaml   # optional
          test-requirement: "Verify the checkout flow after payment refactor"  # optional
        env:
          GITHUB_TOKEN: ${{ github.token }}

Limitations

• Guest mode only — no authenticated testing. If you need login flows, use a test profile.
• API specs must fit in the webhook payload — for specs exceeding ~1MB, use a test profile instead.

---

Full Setup (with Test Profile)

Test profiles let you save and reuse configuration across runs. They support options not available in profileless mode: authenticated testing with saved credentials and multi-factor auth (email, MFA/TOTP), multiple user personas, and persistent configuration your team can share.

After completing the Common Setup, follow these additional steps:

Step 5: Create a Test Profile

1. In the Agent Space Operator App, navigate to Test Profiles
2. Click Add test profile
3. Configure your test profile:
   • Target URL — the application to test
   • Agent type — UI or API
   • Personas — guest, authenticated, or both
   • Credentials — Secrets Manager ARN for authenticated testing
   • API spec — OpenAPI specification (for API agent)
4. Save and note down the Test Profile ID (starts with ki-)

Step 6: Add the Action to Your Workflow

Add the release testing step to your workflow (e.g., .github/workflows/release-tests.yml):

name: Release Tests

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

permissions:
  contents: read
  pull-requests: read

jobs:
  # Add any job before (e.g. build, deploy) ...

  test:
    runs-on: ubuntu-latest
    steps:

      - name: Trigger Release Tests
        uses: aws-actions/devops-agent-qa@v1
        with:
          webhook-url: ${{ secrets.DEVOPS_AGENT_WEBHOOK_URL }}
          webhook-secret: ${{ secrets.DEVOPS_AGENT_WEBHOOK_SECRET }}
          test-profile-id: ki-abc123def456
          test-requirement: "Focus on the new payment integration"  # optional
        env:
          GITHUB_TOKEN: ${{ github.token }}

  # ... or any job after

---

Inputs

Input	Required	Description
webhook-url	Yes	The webhook URL from your Agent Space
webhook-secret	Yes	The webhook signing secret for HMAC-SHA256 authentication
test-profile-id	Conditional	The test profile ID to trigger (provide this OR target-url + agent-type)
target-url	Conditional	Target application URL for profileless execution (requires agent-type)
agent-type	Conditional	Agent type: ui or api (requires target-url)
api-spec-file	No	Path to OpenAPI spec file in repo (optional for api agent type; if omitted, spec is generated from code)
test-requirement	No	Optional focus area for testing (e.g., "verify login flow after auth refactor")

Validation: Provide either test-profile-id OR (target-url + agent-type), not both.

Permissions

The workflow must declare these permissions:

Permission	Reason
contents: read	Required for actions/checkout in private repos
pull-requests: read	Resolve PR number from merge commit SHA

How It Works

1. Your workflow triggers (e.g., after deployment)
2. The action signs and sends a webhook to your Agent Space
3. The Agent Space creates a Check Run (in_progress) on the commit, which appears as a pending check on your PR or commit
4. The Release Testing Agent picks up the task and runs tests against your application
5. Results are reported back as a GitHub Check Run (pass/fail with a detailed summary)

You can view the full execution details (timeline, test cases, screenshots for UI tests) in the Operator App linked from your Agent Space.

Security

See CONTRIBUTING for more information.

License

This project is licensed under the Apache-2.0 License.