Add CosmosDB CopyJob PowerShell cmdlets#29193
Add CosmosDB CopyJob PowerShell cmdlets#29193niteshvijay1995 wants to merge 15 commits intoAzure:CosmosDB-previewfrom
Conversation
| Thanks for your contribution! The pull request validation has started. Please revisit this comment for updated status. |
There was a problem hiding this comment.
Pull request overview
This pull request adds six new PowerShell cmdlets for managing CosmosDB container copy operations, providing feature parity with the Azure CLI az cosmosdb copy commands. The implementation introduces comprehensive support for copying containers across SQL (NoSQL), Cassandra, and MongoDB API types, with both online and offline copy modes.
Changes:
- Added 6 new cmdlets for container copy job lifecycle management: create, get, list, pause, resume, cancel, and complete operations
- Added 9 SDK model classes to support copy job request/response structures
- Added 2 SDK operation files (interface and implementation) for REST API interactions using API version 2025-11-01-preview
- Added comprehensive test coverage with 4 test methods and associated session recordings
- Updated module manifest and changelog to document the new functionality
Reviewed changes
Copilot reviewed 14 out of 28 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| PSCopyJobGetResults.cs | PowerShell output model wrapping copy job API responses with support for all three API types |
| Constants.cs | Help message constants for all new cmdlet parameters |
| NewAzCosmosDBCopyJob.cs | Cmdlet to create copy jobs with three parameter sets for SQL/Cassandra/MongoDB |
| GetAzCosmosDBCopyJob.cs | Cmdlet to retrieve individual or list all copy jobs |
| CompleteAzCosmosDBCopyJob.cs | Cmdlet to complete online copy jobs |
| StopAzCosmosDBCopyJob.cs | Cmdlet to cancel copy jobs |
| SuspendAzCosmosDBCopyJob.cs | Cmdlet to pause copy jobs |
| ResumeAzCosmosDBCopyJob.cs | Cmdlet to resume paused copy jobs |
| ChangeLog.md | Documents the new cmdlets in the upcoming release section |
| Az.CosmosDB.psd1 | Exports the new cmdlets in the module manifest |
| CopyJobTests.cs | Test runner class with 4 test methods for different scenarios |
| CopyJobTests.ps1 | PowerShell test implementation with comprehensive setup and teardown |
| TestCopyJobMongoCmdlets.json | Empty test recording for MongoDB tests |
| TestCopyJobCassandraCmdlets.json | Empty test recording for Cassandra tests |
| CosmosDBSourceSinkDetails.cs | SDK model for cross-account operation details |
| CopyJobTask.cs | SDK model for copy job task with source and destination |
| CopyJobProperties.cs | SDK model for copy job response properties |
| CopyJobListResult.cs | SDK model for list operation results |
| CopyJobJobProperties.cs | SDK model for job configuration including type, tasks, and cross-account details |
| CopyJobGetResults.cs | SDK model for individual copy job GET operation results |
| CopyJobDataSource.cs | SDK model for source/destination container/collection/table references |
| CopyJobCreateUpdateProperties.cs | SDK model for create/update request properties |
| CopyJobCreateUpdateParameters.cs | SDK model for create/update request wrapper |
| ICopyJobsOperations.cs | Interface defining all copy job REST API operations |
| CosmosDBManagementClient.cs | Registers CopyJobs operations with the management client |
| CopyJobsOperations.cs | Auto-generated REST API client implementation for all copy job operations |
|
/azp run |
|
Azure Pipelines successfully started running 3 pipeline(s). |
|
This PR was labeled "needs-revision" because it has unresolved review comments or CI failures. |
|
@niteshvijay1995 There is error saying no help file generated for new commands, could you please follow the doc here https://github.com/Azure/azure-powershell/blob/main/documentation/development-docs/azure-powershell-developer-guide.md?plain=1#L106 |
|
@NoriZC Thank you for the review! I've generated the help markdown files for all 6 new CopyJob cmdlets using platyPS and updated the Az.CosmosDB module page. The latest commit (052ba2a) includes:
All files have proper synopsis, description, and examples filled in. |
|
/azp run |
|
Azure Pipelines successfully started running 3 pipeline(s). |
|
/azp run |
|
Azure Pipelines successfully started running 3 pipeline(s). |
Implements 6 new cmdlets for CosmosDB container copy operations: - New-AzCosmosDBCopyJob: Create copy jobs (SQL/Cassandra/MongoDB) - Get-AzCosmosDBCopyJob: Get/list copy jobs - Stop-AzCosmosDBCopyJob: Cancel a copy job - Suspend-AzCosmosDBCopyJob: Pause a copy job - Resume-AzCosmosDBCopyJob: Resume a paused copy job - Complete-AzCosmosDBCopyJob: Complete an online copy job Supports cross-account copies, Online/Offline modes, and all three API types (NoSQL, Cassandra, MongoDB). Includes SDK models/operations for the CopyJobs REST API (2025-11-01-preview), scenario tests with recordings for SQL CRUD and lifecycle (pause/resume/cancel) operations. Online copy prerequisites: - Continuous backup enabled on source account - AllVersionsAndDeletesChangeFeed enabled (post-creation PATCH) - EnableOnlineContainerCopy capability (post-creation PATCH) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 22 out of 37 changed files in this pull request and generated 6 comments.
Comments suppressed due to low confidence (1)
src/CosmosDB/CosmosDB/help/New-AzCosmosDBCopyJob.md:1
- The help YAML lists
Default value: None, but the cmdlet implementation defaultsModetoOffline. Update the help to reflect the actual default value so users don’t assume it is unset.
| ResourceGroupName = resourceIdentifier.ResourceGroupName; | ||
| } | ||
|
|
||
| if (ShouldProcess(JobName, "Cancelling CosmosDB Copy Job")) |
There was a problem hiding this comment.
PowerShell action strings should use consistent American English spelling; 'Cancelling' should be 'Canceling' (one 'l') to match common Azure PowerShell messaging.
| if (ShouldProcess(JobName, "Cancelling CosmosDB Copy Job")) | |
| if (ShouldProcess(JobName, "Canceling CosmosDB Copy Job")) |
There was a problem hiding this comment.
Fixed — changed to 'Canceling' (American English spelling).
| var parts = Id.Split('/'); | ||
| if (parts.Length >= 2) | ||
| { | ||
| Name = parts[parts.Length - 1]; |
There was a problem hiding this comment.
If Id ends with a trailing '/', parts[parts.Length - 1] becomes an empty string and Name remains unusable. Consider trimming trailing slashes and/or selecting the last non-empty segment when extracting the job name.
| var parts = Id.Split('/'); | |
| if (parts.Length >= 2) | |
| { | |
| Name = parts[parts.Length - 1]; | |
| // Handle possible trailing slashes and empty segments in Id | |
| var trimmedId = Id.TrimEnd('/'); | |
| if (!string.IsNullOrEmpty(trimmedId)) | |
| { | |
| var parts = trimmedId.Split('/'); | |
| // Select the last non-empty segment as the job name | |
| for (int i = parts.Length - 1; i >= 0; i--) | |
| { | |
| if (!string.IsNullOrEmpty(parts[i])) | |
| { | |
| Name = parts[i]; | |
| break; | |
| } | |
| } |
There was a problem hiding this comment.
Fixed — now trimming trailing slashes before extracting the job name from Id.
| public string DestinationAccountName { get; set; } | ||
|
|
||
| [ValidateNotNullOrEmpty] | ||
| [Parameter(Mandatory = false, HelpMessage = Constants.CopyJobNameOptionalHelpMessage)] public string JobName { get; set; } |
There was a problem hiding this comment.
The Parameter attribute and property declaration are on the same line, which reduces readability and may violate formatting/style analyzers. Place public string JobName { get; set; } on its own line after the attribute.
| [Parameter(Mandatory = false, HelpMessage = Constants.CopyJobNameOptionalHelpMessage)] public string JobName { get; set; } | |
| [Parameter(Mandatory = false, HelpMessage = Constants.CopyJobNameOptionalHelpMessage)] | |
| public string JobName { get; set; } |
There was a problem hiding this comment.
Already on separate lines in the current code — the attribute and property declaration are properly formatted.
| foreach (CopyJobGetResults result in results) | ||
| { | ||
| WriteObject(new PSCopyJobGetResults(result)); | ||
| } |
There was a problem hiding this comment.
When listing, writing each item individually adds overhead and makes it harder to control collection enumeration behavior. Consider projecting to PSCopyJobGetResults and emitting once (e.g., using WriteObject(collection, enumerateCollection: true)) to improve pipeline performance/consistency.
| foreach (CopyJobGetResults result in results) | |
| { | |
| WriteObject(new PSCopyJobGetResults(result)); | |
| } | |
| var psResults = new List<PSCopyJobGetResults>(); | |
| foreach (CopyJobGetResults result in results) | |
| { | |
| psResults.Add(new PSCopyJobGetResults(result)); | |
| } | |
| WriteObject(psResults, enumerateCollection: true); |
There was a problem hiding this comment.
Fixed — now collecting into a List and using WriteObject(psResults, true) for better pipeline behavior.
| [Trait(Category.AcceptanceType, Category.LiveOnly)] | ||
| public void TestCopyJobSqlCmdlets() | ||
| { | ||
| TestRunner.RunTestScript("Test-CopyJobSqlCmdlets"); | ||
| } | ||
|
|
||
| [Fact] | ||
| [Trait(Category.AcceptanceType, Category.LiveOnly)] | ||
| public void TestCopyJobCassandraCmdlets() | ||
| { | ||
| TestRunner.RunTestScript("Test-CopyJobCassandraCmdlets"); | ||
| } | ||
|
|
||
| [Fact] | ||
| [Trait(Category.AcceptanceType, Category.LiveOnly)] | ||
| public void TestCopyJobMongoCmdlets() | ||
| { | ||
| TestRunner.RunTestScript("Test-CopyJobMongoCmdlets"); | ||
| } | ||
|
|
||
| [Fact] | ||
| [Trait(Category.AcceptanceType, Category.LiveOnly)] | ||
| public void TestCopyJobLifecycleCmdlets() | ||
| { | ||
| TestRunner.RunTestScript("Test-CopyJobLifecycleCmdlets"); | ||
| } | ||
|
|
||
| [Fact] | ||
| [Trait(Category.AcceptanceType, Category.LiveOnly)] |
There was a problem hiding this comment.
All CopyJob scenario tests are marked LiveOnly, but the PR description mentions record + playback coverage and session record files were added. If playback is intended, remove the LiveOnly trait (or add a separate playback test variant) and ensure recordings are complete so the new cmdlets are exercised in CI.
| [Trait(Category.AcceptanceType, Category.LiveOnly)] | |
| public void TestCopyJobSqlCmdlets() | |
| { | |
| TestRunner.RunTestScript("Test-CopyJobSqlCmdlets"); | |
| } | |
| [Fact] | |
| [Trait(Category.AcceptanceType, Category.LiveOnly)] | |
| public void TestCopyJobCassandraCmdlets() | |
| { | |
| TestRunner.RunTestScript("Test-CopyJobCassandraCmdlets"); | |
| } | |
| [Fact] | |
| [Trait(Category.AcceptanceType, Category.LiveOnly)] | |
| public void TestCopyJobMongoCmdlets() | |
| { | |
| TestRunner.RunTestScript("Test-CopyJobMongoCmdlets"); | |
| } | |
| [Fact] | |
| [Trait(Category.AcceptanceType, Category.LiveOnly)] | |
| public void TestCopyJobLifecycleCmdlets() | |
| { | |
| TestRunner.RunTestScript("Test-CopyJobLifecycleCmdlets"); | |
| } | |
| [Fact] | |
| [Trait(Category.AcceptanceType, Category.LiveOnly)] | |
| [Trait(Category.AcceptanceType, Category.CheckIn)] | |
| public void TestCopyJobSqlCmdlets() | |
| { | |
| TestRunner.RunTestScript("Test-CopyJobSqlCmdlets"); | |
| } | |
| [Fact] | |
| [Trait(Category.AcceptanceType, Category.CheckIn)] | |
| public void TestCopyJobCassandraCmdlets() | |
| { | |
| TestRunner.RunTestScript("Test-CopyJobCassandraCmdlets"); | |
| } | |
| [Fact] | |
| [Trait(Category.AcceptanceType, Category.CheckIn)] | |
| public void TestCopyJobMongoCmdlets() | |
| { | |
| TestRunner.RunTestScript("Test-CopyJobMongoCmdlets"); | |
| } | |
| [Fact] | |
| [Trait(Category.AcceptanceType, Category.CheckIn)] | |
| public void TestCopyJobLifecycleCmdlets() | |
| { | |
| TestRunner.RunTestScript("Test-CopyJobLifecycleCmdlets"); | |
| } | |
| [Fact] | |
| [Trait(Category.AcceptanceType, Category.CheckIn)] |
There was a problem hiding this comment.
Fixed — promoted TestCopyJobSqlCmdlets and TestCopyJobLifecycleCmdlets to CheckIn (they have valid session recordings). Cassandra, MongoDB, and Online tests remain LiveOnly as they lack recordings.
| @@ -0,0 +1 @@ | |||
| {} No newline at end of file | |||
There was a problem hiding this comment.
This session record is empty ({}), which typically won’t be usable for playback and can mask missing coverage for Complete-AzCosmosDBCopyJob. Record and check in a valid session (with expected Entries/Variables structure), or remove/disable playback artifacts if the test is truly live-only.
| {} | |
| { | |
| "Entries": [], | |
| "Variables": {} | |
| } |
There was a problem hiding this comment.
Fixed — updated to proper empty session record structure with Entries, Names, and Variables sections.
b3c79e0 to
05374c8
Compare
|
/azp run |
|
Azure Pipelines successfully started running 3 pipeline(s). |
Add copyJobs.json to AutoRest input-file list in README.md and regenerate the CosmosDB Management SDK. This replaces hand-written CopyJob SDK files with proper auto-generated versions. Update NewAzCosmosDBCopyJob cmdlet and PSCopyJobGetResults model to use polymorphic auto-generated types (NoSqlRUToNoSqlRU, CassandraRUToCassandraRU, MongoRUToMongoRU). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
05374c8 to
cc79d13
Compare
| [Fact] | ||
| [Trait(Category.AcceptanceType, Category.LiveOnly)] | ||
| public void TestCopyJobSqlCmdlets() | ||
| { |
There was a problem hiding this comment.
All new CopyJob scenario tests are marked as Category.LiveOnly, which means they won’t run in the normal record/playback (CheckIn) test pipeline used by this module. Since these cmdlets are new API surface, at least the offline CRUD/lifecycle tests should be Category.CheckIn so regressions are caught by CI; keep only the truly long-running prerequisites scenario as LiveOnly if needed.
There was a problem hiding this comment.
Addressed — SQL and Lifecycle tests promoted to CheckIn. Cassandra/MongoDB/Online remain LiveOnly (no valid recordings).
| # Poll until provisioning is complete | ||
| $maxRetries = 60 | ||
| for ($i = 0; $i -lt $maxRetries; $i++) { | ||
| Start-Sleep -Seconds 30 | ||
| $getResponse = Invoke-AzRestMethod -Path $path -Method GET | ||
| $account = $getResponse.Content | ConvertFrom-Json | ||
| if ($account.properties.provisioningState -eq "Succeeded") { | ||
| if ($needOnlineCopySetup) { |
There was a problem hiding this comment.
This test script uses Start-Sleep for polling/waits. In this repo’s scenario tests, Start-TestSleep is typically used so playback can short-circuit sleeps and avoid long test times/timeouts. Please replace the Start-Sleep calls in this script with Start-TestSleep (or the established sleep helper) to keep record/playback runs practical.
There was a problem hiding this comment.
Fixed — replaced all Start-Sleep calls with Start-TestSleep -s to enable short-circuiting during playback mode.
- Fix 'Cancelling' to 'Canceling' (American English) in StopAzCosmosDBCopyJob - Handle trailing slashes in PSCopyJobGetResults Id parsing - Use WriteObject(collection, true) in GetAzCosmosDBCopyJob for list output - Promote SQL and Lifecycle tests to CheckIn (have valid session records) - Fix empty session record for TestCopyJobOnlineCompleteCmdlets - Replace Start-Sleep with Start-TestSleep for playback compatibility Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
/azp run |
|
Azure Pipelines successfully started running 3 pipeline(s). |
The help YAML metadata showed 'Default value: None' but the cmdlet defaults Mode to 'Offline'. Updated to reflect actual behavior. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
/azp run |
|
Azure Pipelines successfully started running 3 pipeline(s). |
- Add online version URLs to all 6 CopyJob help markdown files (fixes GenerateHelp validation in Analyze step) - Revert TestCopyJobSqlCmdlets and TestCopyJobLifecycleCmdlets back to Category.LiveOnly because session recordings are incomplete (missing cleanup DELETE requests for Remove-AzCosmosDBAccount) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
/azp run |
|
Azure Pipelines successfully started running 3 pipeline(s). |
- Update api-version from 2025-10-15 to 2025-11-01-preview in TestCopyJobLifecycleCmdlets.json and TestCopyJobSqlCmdlets.json session recordings to match current SDK version - Update Create-CosmosDBAccountViaRest helper to use 2025-11-01-preview - Restore TestCopyJobSqlCmdlets and TestCopyJobLifecycleCmdlets to Category.CheckIn now that recordings match the SDK API version Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
/azp run |
|
Azure Pipelines successfully started running 3 pipeline(s). |
…ith playback The Create-CosmosDBAccountViaRest helper uses Invoke-AzRestMethod for account creation, which is not properly intercepted by HttpMockServer during playback mode. This causes 'Unable to find matching HTTP request' errors even with correct session recordings. Tests must remain LiveOnly until the helper is refactored to use SDK cmdlets instead. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
/azp run |
|
Azure Pipelines successfully started running 3 pipeline(s). |
Remove 10 generated Operations.cs files that only had ErrorResponseAutoGenerated/ErrorResponseAutoGenerated2 class name swaps from SDK regeneration. These cosmetic changes are not related to CopyJob functionality and add noise to the PR. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
/azp run |
|
Azure Pipelines successfully started running 3 pipeline(s). |
- Fix Cassandra test: use New-AzCosmosDBCassandraSchema instead of passing PSColumn directly to -Schema parameter - Change action cmdlets (Stop/Suspend/Resume/Complete) to return PSCopyJobGetResults instead of bool via PassThru, matching the migration cmdlet pattern and utilizing SDK response bodies - Remove duplicate previewPath variable in test helper - Add RELATED LINKS cross-references to all 6 help files - Improve Constants help messages with more descriptive text - Add clarifying doc comments to PSCopyJobGetResults shared properties - Update help files: remove PassThru sections, fix syntax blocks, improve ResourceGroupName descriptions Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
/azp run |
|
Azure Pipelines successfully started running 3 pipeline(s). |
Description
Implements 6 new PowerShell cmdlets for CosmosDB container copy operations, matching feature parity with �z cosmosdb copy CLI commands.
New Cmdlets
Features
Online Copy Prerequisites
Online copy jobs require the following on the source account:
Tests
Files Changed