Agent Persona Exploration - 2026-04-30

[github-actions[bot]]

Overview

Systematic exploration of how the agentic-workflows custom agent responds to workflow creation requests from 4 software worker personas. Tested 3 of 4 scenarios successfully (QA scenario timed out). Findings are based on agent responses to Backend Engineer, DevOps Engineer, and Product Manager scenarios.

Summary

• Scenarios Tested: 3 (of 4 planned; QA scenario timed out)
• Average Quality Score: ~4.3/5.0
• Run: §25189218263

Key Findings

• ✅ The agent consistently produces well-structured, production-quality YAML frontmatter with appropriate engine, permissions, and tool selections
• ✅ Minimal permissions (e.g. contents: read + only necessary write permissions) are applied by default — good security hygiene
• ✅ The agent correctly separates MCP-only patterns (no network: allowed needed when everything goes through GitHub MCP) from patterns that need external HTTP
• ✅ Concurrency groups and path filters were proactively suggested for the PR-triggered scenario — demonstrates awareness of run efficiency
• ⚠️ The agent doesn't always use the latest on: schedule shorthand (e.g., schedule: weekly-monday) — it sometimes falls back to cron syntax, which is valid but less readable

Top Patterns Observed

1. Engine: copilot recommended for all scenarios — appropriate for structured analysis and report generation
2. Triggers: pull_request (with paths: filter) for PR automation; workflow_run for deployment monitoring; schedule for periodic digests
3. Tools: github MCP with scoped toolsets — never over-provisioned (e.g., only [pull_requests, repos] for a PR reviewer)
4. Security: checkout: false suggested when diff is accessible via MCP (avoids unnecessary repo clone)
5. Safe outputs: safe-outputs: create-issue: true used for incident creation; correct discussions: write permission for digest posting

View Scenario Scores

Scenario 1 — Backend Engineer: DB Migration PR Reviewer

Dimension	Score	Notes
Trigger appropriateness	5	pull_request + paths: filter is exactly right
Tool selection	5	pull_requests + repos toolsets, no extras
Security practices	5	checkout: false, minimal permissions, concurrency cancel
Prompt clarity	5	Detailed 4-step analysis with severity tiers (🔴🟡🟢)
Completeness	4	Missing fork safety (roles:) but noted as optional add-on
Average	4.8

Scenario 2 — DevOps Engineer: Deployment Failure Monitor

Dimension	Score	Notes
Trigger appropriateness	5	workflow_run with named deployment workflows is correct
Tool selection	5	actions + issues toolsets; safe-outputs: create-issue
Security practices	4	Good; could note that log content is untrusted input
Prompt clarity	5	Excellent multi-step with dedup logic (search before create)
Completeness	4	Label pre-creation reminder is a nice touch
Average	4.6

Scenario 4 — Product Manager: Weekly Digest

Dimension	Score	Notes
Trigger appropriateness	5	schedule: weekly-monday + workflow_dispatch fallback
Tool selection	5	default + discussions toolsets; discussions: write permission
Security practices	4	Clean, no network exposure; min-integrity: approved noted
Prompt clarity	5	Excellent tone guidance for non-technical stakeholders
Completeness	4	Quiet-week handling explicitly addressed
Average	4.6

View Areas for Improvement

• Fork safety not default: The DB migration reviewer response noted roles: [write, maintainer, admin] only as an optional add-on. For PR automation, fork safety should be the default recommendation with an opt-out note, since external PRs can trigger workflows with write permissions.

• Prompt injection awareness: The deployment monitor scenario correctly used safe-outputs but didn't explicitly flag that workflow log content is untrusted input. A reminder to avoid echoing raw log lines into issue titles would strengthen security guidance.

• Toolset availability assumptions: The weekly digest assumed a discussions toolset exists without checking whether it's enabled in the repo. A validation step in the prompt (or a note in documentation) would prevent silent failures.

• QA scenario timeout: The coverage analysis scenario caused the agent to exceed token budget, suggesting complex multi-artifact scenarios may need explicit scope-limiting guidance in the prompt template.

Recommendations

1. Strengthen fork safety defaults in PR automation guidance — Update .github/aw/create-agentic-workflow.md to recommend roles: [write, maintainer, admin] by default for pull_request triggered workflows, with a note on how to opt out for open-source repos that want to include external contributors.

2. Add untrusted-input callout for log/issue content — Enhance .github/aw/github-agentic-workflows.md with a short security note: when workflow prompts consume external content (issue bodies, log output, PR descriptions), remind authors to sanitize or scope what gets written to GitHub (use safe-outputs for writes based on that content).

3. Document toolset availability checks — Add a pattern to the workflow authoring guide showing how to gracefully handle missing GitHub Discussion categories (e.g., fallback from announcements → general), since this is a common first-run failure point.

References:

• §25189218263

Generated by Agent Persona Explorer · ● 2.5M · ◷