vscode(feat): Add VS Code extension E2E tests and CI pipeline by lambrianmsft · Pull Request #8884 · Azure/LogicAppsUX

lambrianmsft · 2026-03-05T02:30:14Z

ExTester UI tests: workspace creation, designer actions, inline JavaScript, stateless variables, designer view extensions, keyboard navigation, data mapper
Workspace conversion tests: dialog dismiss, create workspace from legacy project, multiple designers simultaneously, right-click add workflow, subfolder prompt
Shared helper modules: helpers.ts, designerHelpers.ts, runHelpers.ts
GitHub Actions workflow for CI (vscode-e2e.yml)
CLI integration tests: workspace conversion, project outside workspace
Documentation: SKILL.md, CLAUDE.md updates, copilot-instructions
Product fix: silentAuth support in getAuthorizationToken.ts
createWorkspace validation improvements

ADO #31054994: Full coverage of all 15 steps (workspace requirement)

Commit Type

feature - New functionality
fix - Bug fix
refactor - Code restructuring without behavior change
perf - Performance improvement
docs - Documentation update
test - Test-related changes
chore - Maintenance/tooling

Risk Level

Low - Minor changes, limited scope
Medium - Moderate changes, some user impact
High - Major changes, significant user/system impact

What & Why

Impact of Change

Users:
There are no user facing changes.
Developers:
Developers will have better visibility on how their changes affect the vscode experience.
System:
Test architecture is now set up for vscode webviews so new functionality can have e2e tests written for them.

Test Plan

Unit tests added/updated
E2E tests added/updated
Manual testing completed
Tested in:

Contributors

Screenshots/Videos

github-actions · 2026-03-05T02:34:12Z

❌ PR Validation Error

An error occurred while validating your PR. Please try again later or contact the maintainers.

Error: API request failed with status 504

Copilot

Pull request overview

Adds end-to-end test coverage for the Logic Apps VS Code extension (both UI-driven ExTester/Selenium suites and extension-host CLI-style tests), plus a CI workflow to execute them, while also tightening Create Workspace validation and enabling optional silent auth for automation scenarios.

Changes:

Introduces extensive ExTester UI E2E suites (workspace creation/conversion + designer lifecycle scenarios) with shared helpers and documentation.
Adds a GitHub Actions workflow to run the VS Code extension E2E pipeline and upload screenshots as artifacts.
Improves Create Workspace validation (notably for function name/namespace) and adds silentAuth support in token acquisition.

Reviewed changes

Copilot reviewed 53 out of 56 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
apps/vs-code-react/src/app/createWorkspace/validation/helper.ts	Adds C#-safe function name regex + stronger function field validation messages.
apps/vs-code-react/src/app/createWorkspace/validation/test/helper.test.ts	Adds Vitest coverage for new validation behaviors.
apps/vs-code-react/src/app/createWorkspace/steps/dotNetFrameworkStep.tsx	Revalidates function namespace/name on state changes.
apps/vs-code-react/src/app/createWorkspace/createWorkspace.tsx	Uses namespace regex for navigation gating during wizard progression.
apps/vs-code-designer/tsconfig.e2e.json	Adds TypeScript config for extension-host E2E tests output.
apps/vs-code-designer/src/test/ui/workspaceManifest.ts	Shared manifest format + cleanup helpers for cross-phase UI tests.
apps/vs-code-designer/src/test/ui/workspaceConversionYes.test.ts	UI test validating “Open Workspace” prompt and accepting it.
apps/vs-code-designer/src/test/ui/workspaceConversionSubfolder.test.ts	UI test validating prompt when opening a logic app subfolder and dismissing it.
apps/vs-code-designer/src/test/ui/workspaceConversionNo.test.ts	UI test validating prompt dismissal (“No”/escape/close) and VS Code remains usable.
apps/vs-code-designer/src/test/ui/workspaceConversionCreate.test.ts	UI test creating workspace from a legacy project and verifying source is untouched.
apps/vs-code-designer/src/test/ui/test-settings.json	Disables updates/telemetry/trust prompts to stabilize UI automation.
apps/vs-code-designer/src/test/ui/statelessVariables.test.ts	UI E2E around variable actions + debug/run verification.
apps/vs-code-designer/src/test/ui/runHelpers.ts	Shared debug/overview/run-history helper utilities for UI tests.
apps/vs-code-designer/src/test/ui/run-clean.ps1	Local helper script to clean processes and run UI tests.
apps/vs-code-designer/src/test/ui/pin-ext-dev-path.js	Preload hook preventing ExTester from clobbering `EXTENSION_DEV_PATH`.
apps/vs-code-designer/src/test/ui/keyboardNavigation.test.ts	UI E2E for canvas keyboard navigation.
apps/vs-code-designer/src/test/ui/inlineJavascript.test.ts	UI E2E for inline JavaScript action + run verification.
apps/vs-code-designer/src/test/ui/helpers.ts	Shared UI test helpers (screenshots, dismissal, activity bar/tree helpers).
apps/vs-code-designer/src/test/ui/designerViewExtended.test.ts	UI E2E for parallel branches + run-after settings.
apps/vs-code-designer/src/test/ui/dataMapper.test.ts	UI E2E smoke coverage for Data Mapper integration points.
apps/vs-code-designer/src/test/ui/WORKSPACE_CONVERSION_TEST_PLAN.md	Documents mapping from ADO steps → test files/phases.
apps/vs-code-designer/src/test/ui/README.md	Updates UI E2E documentation (phases, build/run, gotchas).
apps/vs-code-designer/src/test/e2e/runTest.ts	Adds a “runner entry point” placeholder for extension-host tests.
apps/vs-code-designer/src/test/e2e/integration/workflow.test.ts	Adds extension-host integration tests around workflow file access.
apps/vs-code-designer/src/test/e2e/integration/projectOutsideWorkspace.test.ts	Adds extension-host integration tests for workspace/project prompt behavior.
apps/vs-code-designer/src/test/e2e/integration/designerOpens.test.ts	Adds extension-host tests simulating webview panel creation/caching/protocol.
apps/vs-code-designer/src/test/e2e/integration/designer.test.ts	Adds basic extension-host webview panel API tests.
apps/vs-code-designer/src/test/e2e/integration/createWorkspace.test.ts	Adds extension-host tests that validate expected workspace structure.
apps/vs-code-designer/src/test/e2e/extension.test.ts	Adds basic extension-host activation tests.
apps/vs-code-designer/src/test/e2e/commands.test.ts	Adds extension-host command enumeration/smoke tests.
apps/vs-code-designer/src/test/e2e/README.md	Documents CLI-based E2E structure and usage.
apps/vs-code-designer/src/app/utils/codeless/getAuthorizationToken.ts	Adds `silentAuth` option support to avoid interactive auth prompts in automation.
apps/vs-code-designer/e2e/test-workspace/package.json	Adds a fixture workspace for extension-host E2E runs.
apps/vs-code-designer/e2e/test-workspace/Workflows/TestWorkflow/workflow.json	Adds a sample workflow fixture.
apps/vs-code-designer/e2e/test-workspace/.vscode/settings.json	Adds fixture workspace VS Code settings.
apps/vs-code-designer/e2e/test-workspace/.vscode/extensions.json	Adds fixture workspace extension recommendations.
apps/vs-code-designer/CLAUDE.md	Documents E2E test phases/helpers and authoring rules.
apps/vs-code-designer/.vscode/tasks.json	Adds VS Code tasks for compiling E2E tests / watch.
apps/vs-code-designer/.vscode/launch.json	Adds launch configs for extension + tests.
apps/vs-code-designer/.vscode-test.mjs	Adds `@vscode/test-cli` configuration profiles.
apps/vs-code-designer/.github/copilot-skills/vscode-e2e-testing.md	Adds a “skill” doc describing E2E conventions and architecture.
CLAUDE.md	Links repo-level guidance to VS Code extension E2E docs/tools.
.gitignore	Ignores additional E2E artifacts (.vscode-test, test-extensions).
.github/workflows/vscode-e2e.yml	Adds CI job to build extension and run VS Code E2E tests under Xvfb.
.github/copilot-instructions.md	Adds E2E run/build guidance + formatting/linting expectations.

Comments suppressed due to low confidence (1)

apps/vs-code-designer/src/app/utils/codeless/getAuthorizationToken.ts:22

When silentAuth is enabled, getSessionFromVSCode(..., { silent: true }) can return undefined (per the comment). In that case getAuthorizationToken() returns Bearer undefined, which is almost certainly not a valid token and will cause confusing downstream failures. Handle the no-session case explicitly (e.g., throw a localized error, or fall back to createIfNone: true depending on desired behavior).

export async function getAuthData(tenantId?: string): Promise<AuthenticationSession> {
  // When silentAuth is enabled (e.g. in automated test environments),
  // use { silent: true } to avoid showing the "wants to sign in" dialog.
  // This returns undefined if no cached session exists, instead of prompting.
  const silentAuth = vscode.workspace.getConfiguration('azureLogicAppsStandard').get<boolean>('silentAuth', false);
  if (silentAuth) {
    return await getSessionFromVSCode(undefined, tenantId, { silent: true });
  }
  return await getSessionFromVSCode(undefined, tenantId, { createIfNone: true });
}

export async function getAuthorizationToken(tenantId?: string): Promise<string> {
  const authData = await getAuthData(tenantId);
  return `Bearer ${authData?.accessToken}`;
}

apps/vs-code-designer/src/test/ui/workspaceConversionYes.test.ts

apps/vs-code-designer/src/test/ui/run-clean.ps1

apps/vs-code-designer/.vscode/launch.json

apps/vs-code-react/src/app/createWorkspace/createWorkspace.tsx

apps/vs-code-designer/.vscode/tasks.json

apps/vs-code-designer/.vscode-test.mjs

apps/vs-code-react/src/app/createWorkspace/steps/dotNetFrameworkStep.tsx

- ExTester UI tests: workspace creation, designer actions, inline JavaScript, stateless variables, designer view extensions, keyboard navigation, data mapper - Workspace conversion tests: dialog dismiss, create workspace from legacy project, multiple designers simultaneously, right-click add workflow, subfolder prompt - Shared helper modules: helpers.ts, designerHelpers.ts, runHelpers.ts - GitHub Actions workflow for CI (vscode-e2e.yml) - CLI integration tests: workspace conversion, project outside workspace - Documentation: SKILL.md, CLAUDE.md updates, copilot-instructions - Product fix: silentAuth support in getAuthorizationToken.ts - createWorkspace validation improvements ADO #31054994: Full coverage of all 15 steps (workspace requirement)

github-actions · 2026-03-05T02:52:41Z

❌ PR Validation Error

An error occurred while validating your PR. Please try again later or contact the maintainers.

Error: API request failed with status 500

ExTester's page-objects use '.editor-instance' CSS selector which doesn't exist in VS Code 1.102.0 (the version CI downloaded with 'max'). Pin to 1.108.0 which is known to work with ExTester 8.21.0. This fixes all 13 Phase 4.1 workspace creation test failures on CI.

…onflict)

github-actions · 2026-03-05T03:16:45Z

❌ PR Validation Error

An error occurred while validating your PR. Please try again later or contact the maintainers.

Error: API request failed with status 504

…tSettings, conversion assertions - Add writeTestSettings() function for per-phase VS Code settings control - Disable autoStartDesignTime for conversion tests (4.8a-4.8e) - Add JS-based QuickPick dismissal for Azure connector wizard prompts - Fix waitForDesignerWebviewTab to skip command palette detection - Fix designerActions local waitForDependencyValidation timeout - Relax conversion file assertion: only check for removed files - Replace this.skip() with assert.fail() across all test files - Add retry logic for legacy project cleanup in 4.8b

github-actions · 2026-03-05T21:46:56Z

❌ PR Validation Error

An error occurred while validating your PR. Please try again later or contact the maintainers.

Error: API request failed with status 504

…fallback - Phase 4.8b: enable validateDependencies so extension activates for dialog - Phase 4.8c: restore full design-time settings (needs designer) - Phase 4.8d/4.8e: restore conversion settings after 4.8c - Manifest lookups: fall back to Stateless when Stateful unavailable - clearAndType: scrollIntoView + JS click fallback for overlay/iframe issues - wsDir/appDir lookups: fall back to any standard entry (not just Stateful)

github-actions · 2026-03-05T23:57:24Z

❌ PR Validation Error

An error occurred while validating your PR. Please try again later or contact the maintainers.

Error: API request failed with status 500

- clearAndType: use VSBrowser.instance.driver (not describe-scoped var) - executeOpenDesignerCommand: 10 retries × 5s delay (was 5 × 3s) - ALL conversion tests: validateDependencies=true (extension must activate) - conversiononly mode: same fix for validateDependencies

github-actions · 2026-03-06T00:44:52Z

❌ PR Validation Error

An error occurred while validating your PR. Please try again later or contact the maintainers.

Error: API request failed with status 504

Root cause: opening a .code-workspace file restarts the extension host. On CI, the extension re-downloads/validates dependencies (NodeJs, FuncCoreTools, DotNetSDK) which takes 30-120s. Without waiting for this second activation cycle, 'Open Designer' is never found because the extension hasn't finished registering its commands. Fix: openWorkspaceFileInSession now calls waitForDependencyValidation after the workspace switch, polling up to 300s for the extension to re-activate. Applied to both designerHelpers.ts (shared) and designerActions.test.ts (local).

Root cause: each phase launches VS Code with the workspace as a startup resource, but then openDesignerForEntry re-opens the same workspace via openWorkspaceFileInSession. This triggers an unnecessary extension host restart + dependency re-validation (60s+ on CI each time). Fix: check VS Code's window title before switching. If the workspace is already open, skip the switch and go directly to opening workflow.json. Also increase CI job timeout from 60→90 min as safety margin.

github-actions · 2026-03-06T02:08:27Z

❌ PR Validation Error

An error occurred while validating your PR. Please try again later or contact the maintainers.

Error: API request failed with status 500

Root cause: waitForDependencyValidation(300s) in openWorkspaceFileInSession created dangling Promises that outlived Mocha test timeouts. These Promises continued polling the dead WebDriver session after the browser shut down, causing 'invalid session id' errors to leak into the next phase. Fix: - Remove waitForDependencyValidation from openWorkspaceFileInSession entirely - Use simple sleep(5s) for workspace settle instead of 300s blocking poll - Increase executeOpenDesignerCommand to 20 retries × 10s = 200s - Extension activation wait now happens at command-palette level, not at workspace-switch level (no dangling Promises)

github-actions · 2026-03-06T03:32:09Z

❌ PR Validation Error

An error occurred while validating your PR. Please try again later or contact the maintainers.

Error: API request failed with status 504

ROOT CAUSE: prepareFreshSession used Windows-only PowerShell to kill lingering VS Code processes. On Linux CI, this silently failed, leaving the Phase 4.1 VS Code window running. When Phase 4.2+ launched a NEW VS Code, ExTester's 'code -r wsFilePath' sent the workspace-open command to the OLD window. The new window stayed bare (no folder), which is why all screenshots showed empty VS Code with no extension behavior. FIX: - Use pkill on Linux/Mac to kill test-resources VS Code processes - Also kill lingering chromedriver processes between phases - Keep PowerShell for Windows compatibility - Add step-by-step screenshots to standard workspace creation test - Add failure screenshots in afterEach hook - Revert excessive retry counts back to 5x3s (root cause was stale processes, not timing)

github-actions · 2026-03-06T03:59:42Z

❌ PR Validation Error

An error occurred while validating your PR. Please try again later or contact the maintainers.

Error: API request failed with status 500

Root causes fixed: 1. switchToWebviewFrame: 10 retries x 3s (was 5x2s) for cold CI webview render 2. openDesignerForEntry: uses Explorer right-click instead of command palette - No timing dependency on command registration after workspace switch - Context menu available as soon as extension loads - Mirrors real user behavior 3. clearAndType: add VSBrowser import for JS-click fallback on rules engine 4. waitForDependencyValidation: added to before hooks of 4 test files (inlineJavascript, statelessVariables, designerViewExtended, keyboardNavigation) 5. getPhase2Resources: aligned fallback with test files (prefer standard+Stateful → standard → manifest[0])

Root cause analysis: code -r (openResources) works intermittently on CI. Phases 4.2, 4.4, 4.6 succeed while 4.3, 4.5, 4.8c fail — alternating pattern suggests race condition with stale IPC socket files from killed processes. Fix: - Clean up *.sock files in settings dir after killing processes - Increase post-kill wait from 3s to 5s for full process exit - Revert appDir startup resource change (would trigger conversion prompt) - Keep wsFilePath for startup resources

github-actions · 2026-03-06T22:01:51Z

❌ PR Validation Error

An error occurred while validating your PR. Please try again later or contact the maintainers.

Error: API request failed with status 500

Adds diagnostic logging to understand why openResources (code -r) doesn't work on Linux CI: - VS Code title before/after openResources - Explorer sidebar row count and contents - Settings dir file listing (looking for IPC sockets) - Startup resource existence checks (file vs directory) This is a diagnostic-only change to capture evidence for the next fix.

…age, added scrolling to better capture any error messages that appear

github-actions · 2026-03-25T21:27:04Z

❌ PR Validation Error

An error occurred while validating your PR. Please try again later or contact the maintainers.

Error: API request failed with status 500

github-actions · 2026-03-25T21:59:58Z

❌ PR Validation Error

An error occurred while validating your PR. Please try again later or contact the maintainers.

Error: API request failed with status 504

…ests, and update shared asset resolver to prevent asset-copying with unit testing

…le from pr coverage check

…le from pr coverage check in the pr-coverage.yml

github-actions · 2026-03-27T02:59:20Z

❌ PR Validation Error

An error occurred while validating your PR. Please try again later or contact the maintainers.

Error: API request failed with status 504

apps/vs-code-designer/src/__test__/devContainerIntegration.test.ts

github-actions · 2026-03-27T22:52:34Z

❌ PR Validation Error

An error occurred while validating your PR. Please try again later or contact the maintainers.

Error: API request failed with status 500

github-actions · 2026-03-27T22:52:35Z

❌ PR Validation Error

An error occurred while validating your PR. Please try again later or contact the maintainers.

Error: API request failed with status 500

github-actions · 2026-03-27T22:52:36Z

❌ PR Validation Error

An error occurred while validating your PR. Please try again later or contact the maintainers.

Error: API request failed with status 500

Copilot AI review requested due to automatic review settings March 5, 2026 02:30

Copilot started reviewing on behalf of lambrianmsft March 5, 2026 02:32 View session

Copilot AI reviewed Mar 5, 2026

View reviewed changes

lambrianmsft force-pushed the lambrian/vscode_e2e_testing branch from 10cb7db to 672161e Compare March 5, 2026 02:47

lambrianmsft force-pushed the lambrian/vscode_e2e_testing branch from 672161e to 515f663 Compare March 5, 2026 02:48

lambrianmsft added 2 commits March 4, 2026 18:59

Merge main into lambrian/vscode_e2e_testing (resolve pnpm-lock.yaml c…

4a722e3

…onflict)

github-actions bot mentioned this pull request Mar 5, 2026

[repo-status] Daily Repo Status — March 5, 2026 #8885

Closed

lambrianmsft added 2 commits March 5, 2026 17:02

github-actions bot mentioned this pull request Mar 6, 2026

[repo-status] Daily Repo Status — March 6, 2026 #8891

Closed

github-actions bot mentioned this pull request Mar 25, 2026

[repo-status] Daily Repo Status — March 25, 2026 #8963

Closed

lambrianmsft added 2 commits March 25, 2026 09:55

Removed missing intl, updated for additional validation message cover…

a94fc1f

…age, added scrolling to better capture any error messages that appear

Merge branch 'main' into lambrian/vscode_e2e_testing

a37466c

fix corrupted pnpm lock file

f3070e0

correction CodeQL Incomplete string escaping or encoding error

38554aa

github-actions bot mentioned this pull request Mar 26, 2026

[repo-status] Daily Repo Status — March 26, 2026 #8967

Closed

lambrianmsft added 3 commits March 26, 2026 17:49

Removed the onboarding/runtime dependency cycle, cleaned up related t…

9857a63

…ests, and update shared asset resolver to prevent asset-copying with unit testing

Exclude the src/test/e2e, src/test/ui, and the authorization token fi…

873cda7

…le from pr coverage check

Exclude the src/test/e2e, src/test/ui, and the authorization token fi…

aba5b17

…le from pr coverage check in the pr-coverage.yml

github-actions bot mentioned this pull request Mar 27, 2026

[repo-status] Daily Repo Status — March 27, 2026 #8973

Closed

andrew-eldridge reviewed Mar 27, 2026

View reviewed changes

apps/vs-code-designer/src/__test__/devContainerIntegration.test.ts Show resolved Hide resolved

andrew-eldridge approved these changes Mar 27, 2026

View reviewed changes

lambrianmsft merged commit 2e486cd into Azure:main Mar 27, 2026
14 of 15 checks passed

lambrianmsft added risk:medium Medium risk change with potential impact risk:low Low risk change with minimal impact and removed risk:medium Medium risk change with potential impact labels Mar 27, 2026

This was referenced Apr 6, 2026

[repo-status] Daily Repo Status — April 6, 2026 #9016

Closed

[repo-status] Daily Repo Status — April 7, 2026 #9021

Closed

Conversation

lambrianmsft commented Mar 5, 2026

Commit Type

Risk Level

What & Why

Impact of Change

Test Plan

Contributors

Screenshots/Videos

Uh oh!

github-actions bot commented Mar 5, 2026

❌ PR Validation Error

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Mar 5, 2026

❌ PR Validation Error

Uh oh!

github-actions bot commented Mar 5, 2026

❌ PR Validation Error

Uh oh!

github-actions bot commented Mar 5, 2026

❌ PR Validation Error

Uh oh!

github-actions bot commented Mar 5, 2026

❌ PR Validation Error

Uh oh!

github-actions bot commented Mar 6, 2026

❌ PR Validation Error

Uh oh!

github-actions bot commented Mar 6, 2026

❌ PR Validation Error

Uh oh!

github-actions bot commented Mar 6, 2026

❌ PR Validation Error

Uh oh!

github-actions bot commented Mar 6, 2026

❌ PR Validation Error

Uh oh!

github-actions bot commented Mar 6, 2026

❌ PR Validation Error

Uh oh!

github-actions bot commented Mar 25, 2026

❌ PR Validation Error

Uh oh!

github-actions bot commented Mar 25, 2026

❌ PR Validation Error

Uh oh!

github-actions bot commented Mar 27, 2026

❌ PR Validation Error

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Mar 27, 2026

❌ PR Validation Error

Uh oh!

github-actions bot commented Mar 27, 2026

❌ PR Validation Error

Uh oh!

github-actions bot commented Mar 27, 2026

❌ PR Validation Error

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!