From 61f2e4af5a67cc8b7d40992b608b23631cf2579a Mon Sep 17 00:00:00 2001 From: Sean Kane Date: Mon, 6 Jul 2026 17:11:57 -0600 Subject: [PATCH 1/3] [DRAFT] SAA operator commands documentation --- .../activities/activity-operations.mdx | 266 +++++++++++++++++- .../activities/standalone-activity.mdx | 10 +- 2 files changed, 269 insertions(+), 7 deletions(-) diff --git a/docs/encyclopedia/activities/activity-operations.mdx b/docs/encyclopedia/activities/activity-operations.mdx index 5a3a05c008..c990d51523 100644 --- a/docs/encyclopedia/activities/activity-operations.mdx +++ b/docs/encyclopedia/activities/activity-operations.mdx @@ -22,6 +22,7 @@ This page discusses the following: - [Unpause](#unpause) - [Reset](#reset) - [Update Options](#update-options) +- [Standalone Activity operator commands](#standalone-activity-operator-commands) - [Observability](#observability) Activity Operations are deliberate actions you perform on a specific [Activity Execution](/activity-execution), as @@ -30,8 +31,8 @@ opposed to lifecycle behaviors like [retries](/encyclopedia/retry-policies) and You can perform Activity Operations through the [CLI](/cli/command-reference/activity), the UI, or directly via the gRPC API. Not all operations are available in all interfaces yet - see the [summary table](#operations-summary) for current -support. Activity Operations don't apply to [Local Activities](/local-activity) or -[Standalone Activities](/standalone-activity). +support. Activity Operations don't apply to [Local Activities](/local-activity). For Standalone Activities, see +[Standalone Activity operator commands](#standalone-activity-operator-commands). :::note Public Preview @@ -125,7 +126,7 @@ but they must be Unpaused separately. - **Pause operates on individual Activities by ID within a single Workflow.** Unlike Unpause, Reset, and Update Options, there's no `--query` flag. To pause multiple Activities, issue separate commands for each Activity ID. -- **No Namespace-wide query for Paused Activities.** You must know the Workflow Id. See [Observability](#observability). +- **No Namespace-wide query for Paused Activities.** You must know the Workflow ID. See [Observability](#observability). ## Unpause {/* #unpause */} @@ -314,6 +315,263 @@ options, including Retry Policy, Task Queue, and bulk mode via `--query`. - **Update Options is CLI and gRPC only.** It's not available in the UI. +## Standalone Activity operator commands {/* #standalone-activity-operator-commands */} + +Standalone Activity operator commands use the same Activity operation concepts as Workflow Activities, but they target a +top-level [Standalone Activity](/standalone-activity) instead of an Activity scheduled by a Workflow. + +- To target a Standalone Activity, identify the Activity by Activity ID. +- If the Activity ID has multiple runs, use the Activity Run ID to target one run. +- Do not provide a Workflow ID when you are targeting a Standalone Activity. +- These commands operate on one Standalone Activity Execution at a time. +- Workflow Run Timeout and Workflow Execution Timeout don't apply because a Standalone Activity is not part of a + Workflow Execution. +- Standalone Activities still use Activity timeouts: Schedule-To-Close, Schedule-To-Start, Start-To-Close, and + Heartbeat. +- Schedule-To-Close is the overall Activity Execution deadline. It keeps running while the Activity is Paused. +- Start-To-Close and Heartbeat timeouts apply only while a Worker is running an attempt. +- If the Activity has a Start Delay, the first dispatch can't happen before `ScheduleTime + StartDelay`. +- Start Delay also affects the Schedule-To-Close and Schedule-To-Start deadlines for the first dispatch. +- Start Delay stops affecting dispatch after the first Worker has picked up the Activity. +- If an operation reissues a dispatch task, the task is scheduled at the later of the operation time, any remaining + Start Delay, and any preserved retry backoff. +- A cancel request can still be accepted while Pause or Reset is pending; after cancellation is pending, cancellation + takes precedence over operator commands. +- Terminal Activity states can't be Paused, Unpaused, Reset, or updated. +- The commands return `NotFound` if the targeted Standalone Activity Execution doesn't exist. + +### Pause a Standalone Activity {/* #pause-standalone-activity */} + +Pause prevents the Temporal Service from dispatching another attempt for a Standalone Activity. + +- If the Activity is SCHEDULED: + - Pause moves it to PAUSED. + - Pause invalidates any pending dispatch task. + - No Worker receives it until it is Unpaused or Reset without `keep_paused`. +- If the Activity is STARTED: + - Pause moves it to PAUSE_REQUESTED. + - The running Worker keeps its task token. + - Pause doesn't invalidate Start-To-Close or Heartbeat timeout tasks. + - If the Activity heartbeats: + - The next heartbeat response reports that the Activity is paused. + - If the Activity completes successfully: + - The Activity completes normally. + - If the Activity fails retryably: + - The Activity moves to PAUSED instead of dispatching the next retry. + - If the Activity fails non-retryably: + - The Activity fails instead of moving to PAUSED. + - If the Activity reaches Start-To-Close Timeout: + - The retry path consumes the pause request and the Activity moves to PAUSED when retries remain. + - If the Activity reaches Heartbeat Timeout: + - The retry path consumes the pause request and the Activity moves to PAUSED when retries remain. +- When a pause request is consumed by a failed or timed-out attempt, the attempt count is incremented before the + Activity lands in PAUSED. +- Pause doesn't stop the Schedule-To-Close Timeout. +- If the Activity is PAUSED: + - It can time out with Schedule-To-Close Timeout. + TODO: list all combinations and iteractions for pause and SAA timeouts +- Pause is rejected: + - If cancellation is already pending. + TODO: List out all scenarios where a pause will be rejected + TODO: list out all scenarios where a cancellation is rejected + - If the Activity is already Paused, unless the request is a duplicate with the same request ID. + - If reset is already pending. +- Pause records pause metadata such as identity, reason, request ID, and pause time. + +### Unpause a Standalone Activity {/* #unpause-standalone-activity */} + +Unpause resumes a Standalone Activity that is PAUSED or has a pending pause request. + +- If the Activity is PAUSED: + - Unpause: + - moves it back to SCHEDULED. + - creates a new dispatch task. + - creates a new Schedule-To-Start timeout task when Schedule-To-Start Timeout is set. + - If retry backoff is still in the future: + - Unpause honors the remaining retry backoff by default. + - If the retry backoff came from a Worker-provided `NextRetryDelay`: + - Unpause honors that retry delay by default. + - If retry backoff has already elapsed: + - Unpause can dispatch immediately. + - Unpause doesn't clear the current retry interval unless `reset_attempts` is set. + - If `reset_attempts` is set: + - Unpause: + - resets the attempt count to 1. + - clears the current retry interval. + - If `reset_heartbeat` is set: + - Unpause clears the stored heartbeat time and heartbeat details. + - If `jitter` is set: + - Unpause delays dispatch by a random duration within the jitter window. + - If the Activity still has an unexpired Start Delay and no Worker has picked it up yet: + - Unpause honors the Start Delay. +- If the Activity is PAUSE_REQUESTED: + - Unpause moves it back to STARTED. + - The running Worker keeps its task token. + - Unpause doesn't schedule a new attempt. + - Unpause doesn't invalidate the current Start-To-Close or Heartbeat timeout tasks. + - If cancellation is requested before Unpause: + - Unpause is a no-op. +- If the Activity is RESET_REQUESTED because Reset with `keep_paused` was requested while the Activity was + PAUSE_REQUESTED: + - Unpause clears the `keep_paused` intent but keeps the reset pending. + - The running Worker continues to see the pending reset on heartbeat, but no longer sees the Activity as paused. + - The reset lands in SCHEDULED instead of PAUSED when the running attempt yields. +- Unpausing an Activity that is not Paused is a no-op. +- Unpausing a terminal Activity is rejected. + +### Reset a Standalone Activity {/* #reset-standalone-activity */} + +Reset clears retry state and schedules, or requests, attempt 1 for a Standalone Activity. + +- If the Activity is SCHEDULED: + - Reset moves it to SCHEDULED at attempt 1. + - If the Activity is in retry backoff: + - Reset discards the retry backoff. + - Reset creates a new dispatch task. + - Reset creates a new Schedule-To-Start timeout task when Schedule-To-Start Timeout is set. + - If the Activity has never been picked up and still has an unexpired Start Delay: + - Reset honors the Start Delay. + - If `jitter` is set: + - Reset delays dispatch by a random duration within the jitter window. +- If the Activity is PAUSED and `keep_paused` is false: + - Reset moves it to SCHEDULED at attempt 1. + - Reset unpauses the Activity. +- If the Activity is PAUSED and `keep_paused` is true: + - Reset leaves it PAUSED at attempt 1. + - No new dispatch task is created until a later Unpause. +- If the Activity is STARTED: + - Reset moves it to RESET_REQUESTED. + - Reset is deferred until the running attempt yields. + - The running Worker keeps its task token. + - The next heartbeat response reports that the Activity was reset. + - If the Activity completes successfully: + - The Activity completes normally and the reset doesn't apply. + - If the Activity fails: + - The reset is applied and the next attempt is attempt 1. + - If the Activity fails non-retryably: + - The reset is still applied and the next attempt is attempt 1. + - If the Activity has no retries remaining: + - The reset is still applied and the next attempt is attempt 1. + - If the Activity reaches Start-To-Close Timeout: + - The reset is applied and the Activity moves to SCHEDULED at attempt 1. + - If the Activity reaches Heartbeat Timeout: + - The reset is applied and the Activity moves to SCHEDULED at attempt 1. +- If the Activity is PAUSE_REQUESTED and `keep_paused` is false: + - Reset clears the pause request. + - Reset is deferred until the running attempt yields. + - The Activity dispatches attempt 1 when the reset is applied. +- If the Activity is PAUSE_REQUESTED and `keep_paused` is true: + - Reset is deferred until the running attempt yields. + - The Activity lands in PAUSED at attempt 1 when the reset is applied. + - The next heartbeat response reports both that the Activity was reset and that the Activity is paused. +- Reset doesn't stop the Schedule-To-Close Timeout. +- Reset doesn't create a new Schedule-To-Close clock. +- Schedule-To-Close continues to use the Activity Execution's first dispatch time. +- After a reset dispatches a new attempt, the attempt's scheduled time reports the reset attempt's dispatch time, not + the original first-attempt dispatch time. +- Reset discards retry backoff, but it still respects any remaining Start Delay before the first Worker pickup. +- If `reset_heartbeat` is set: + - Reset clears stored heartbeat time and heartbeat details. + - If the Activity is non-running: + - Heartbeat state is cleared immediately. + - If the Activity is running: + - Heartbeat state is cleared when the reset is applied to the next attempt. +- If `reset_heartbeat` is not set: + - Heartbeat details are preserved. +- If `restore_original_options` is set: + - Reset restores the options from when the Standalone Activity was created. + - Reset can restore Task Queue, Schedule-To-Close Timeout, Schedule-To-Start Timeout, Start-To-Close Timeout, + Heartbeat Timeout, Retry Policy, Priority, and Start Delay. + - Subfield updates made before Reset don't change the saved original options. + - Start Delay is restored only if the Activity has not started its first attempt. + - If the Activity is non-running: + - The restore is applied immediately. + - If the Activity is running: + - The restore is deferred until the reset is applied to the next attempt. + - Restored timeouts don't shorten or otherwise disturb the in-flight attempt. + - If the Activity completes successfully after the reset is requested: + - The Activity completes and the restored options are never applied. +- Reset is rejected: + - If cancellation is already pending. + - If another reset is already pending. + - If the Activity is terminal. + +### Update Standalone Activity options {/* #update-standalone-activity-options */} + +Update Options changes a Standalone Activity's stored runtime options. + +- Update Options can change: + - Task Queue name. + - Schedule-To-Close Timeout. + - Schedule-To-Start Timeout. + - Start-To-Close Timeout. + - Heartbeat Timeout. + - Retry Policy. + - Retry Policy initial interval. + - Retry Policy backoff coefficient. + - Retry Policy maximum interval. + - Retry Policy maximum attempts. + - Priority. + - Priority Key. + - Fairness Key. + - Fairness Weight. + - Start Delay while the Activity is still in its delay window. +- Update Options requires an update mask unless `restore_original` is used. +- If `restore_original` is set: + - `restore_original` can't be combined with an update mask. + - `restore_original` restores the options from when the Standalone Activity was created. + - `restore_original` is applied immediately to the active Standalone Activity Execution. + - `restore_original` restores Start Delay only if the Activity has not started its first attempt. +- Updated timeout values are validated and normalized after the update. +- If Schedule-To-Close Timeout is updated: + - Update Options reissues the Schedule-To-Close timeout task at the new deadline. + - Stale Schedule-To-Close timeout tasks from before the latest options update are ignored. + - If Schedule-To-Close Timeout is disabled: + - Previously scheduled Schedule-To-Close timeout tasks are ignored. + - Shortening Schedule-To-Close Timeout can cause the Activity to time out sooner. + - Extending Schedule-To-Close Timeout can allow future retries that would otherwise be blocked by the Activity + deadline. +- If Start-To-Close Timeout or Heartbeat Timeout is updated for a running attempt: + - Update Options reissues the Start-To-Close timeout task for that attempt. + - Update Options reissues the Heartbeat timeout task for that attempt. + - Updated running-attempt timers are anchored to the current attempt's start time or last heartbeat time. + - Updating Start-To-Close or Heartbeat Timeout can affect the current running attempt. +- If Task Queue is updated: + - The change affects future dispatches, not a task already picked up by a Worker. +- If Retry Policy is updated: + - Retry Policy subfield updates are merged with the existing Retry Policy and then validated. + - The update is rejected if the merged Retry Policy is invalid. + - Update Options recalculates the current retry interval only when the Activity is SCHEDULED or PAUSED, is waiting in + retry backoff, and the current retry interval was derived from the Retry Policy. + - Update Options doesn't recalculate a Worker-provided `NextRetryDelay` that is already scheduled. +- If an unrelated field is updated: + - Update Options doesn't recalculate the current retry interval. +- If `restore_original` is set and the Activity is SCHEDULED or PAUSED: + - Update Options can recalculate the current retry interval when the interval was derived from the Retry Policy. +- If the Activity is SCHEDULED: + - Update Options reissues dispatch and Schedule-To-Start timeout tasks. + - If the Activity is in retry backoff: + - Reissued dispatch and Schedule-To-Start timeout tasks keep the existing retry deadline unless the retry interval + is recalculated by a Retry Policy update or cleared by another operation. +- If the Activity is PAUSED: + - Update Options stores the new options but doesn't dispatch the Activity. + - Updated Retry Policy values can affect when the Activity dispatches after Unpause. +- If the Activity is PAUSE_REQUESTED: + - Update Options reissues running-attempt timers so Start-To-Close and Heartbeat timeouts continue to be enforced. +- If the Activity is RESET_REQUESTED: + - Update Options reissues running-attempt timers so Start-To-Close and Heartbeat timeouts continue to be enforced. +- If cancellation is pending: + - Update Options is still allowed. + - Updated Start-To-Close and Heartbeat timers can still affect the running attempt. +- If Start Delay is updated: + - Start Delay can be updated only while the Activity is SCHEDULED and the first dispatch time is still in the future. + - Start Delay can't be updated after the first dispatch delay has elapsed. + - Start Delay can't be updated after a Worker has picked up the first attempt. + - A nonzero Start Delay update requires Start Delay to be enabled for the Namespace. +- Update Options doesn't reset attempts. +- Update Options doesn't clear heartbeat details. +- Updating a terminal Activity is rejected. + ## Observability {/* #observability */} Activity Operations have a limited audit trail because they are not recorded in a Workflow's Event History. However, you @@ -329,7 +587,7 @@ provided). The `TemporalPauseInfo` [Search Attribute](/search-attribute) is filterable within a Workflow. -There's no Namespace-wide query to find all Paused Activities across Workflows. You must know the Workflow Id. +There's no Namespace-wide query to find all Paused Activities across Workflows. You must know the Workflow ID. ### Audit trail {/* #audit-trail */} diff --git a/docs/encyclopedia/activities/standalone-activity.mdx b/docs/encyclopedia/activities/standalone-activity.mdx index e9b7aa3659..07e314ed3b 100644 --- a/docs/encyclopedia/activities/standalone-activity.mdx +++ b/docs/encyclopedia/activities/standalone-activity.mdx @@ -84,7 +84,8 @@ webhook, syncing data, or executing a single function reliably with built-in ret - Arbitrary length jobs with heartbeats for liveness and checkpointing progress - At-least-once execution by default with native retry policy and timeouts - At-most-once execution if retry max attempts is 1 -- Addressable - get an Activity ID / Run ID and get the result, cancel, and terminate +- Addressable - get an Activity ID / Run ID and get the result, cancel, pause, unpause, reset, update options, and + terminate - Deduplication - with conflict policy: (USE_EXISTING, ...), reuse policy: (REJECT_DUPLICATES, ...) - Separate ID space from Workflows - Standalone Activities are a different kind of top-level execution - Priority and fairness - multi-tenant fairness, weighted priority tiers, and safeguards against starvation of lower-weighted tasks @@ -110,7 +111,6 @@ failed, etc.) - not the number of queued tasks. The Public Preview of Standalone Activities has some known limitations: -- Pause, reset, and update options are not supported in Public Preview but scheduled for GA. - `TerminateExisting` conflict policy / `TerminateIfRunning` reuse policy is not supported yet. ## Temporal CLI support @@ -137,7 +137,11 @@ temporal version 1.7.0 (Server 1.31.0, UI 2.49.1) ``` The `temporal activity` subcommand supports Standalone Activities with commands including: `start`, -`execute`, `result`, `list`, `count`, `describe`, `cancel`, and `terminate`. +`execute`, `result`, `list`, `count`, `describe`, `cancel`, `pause`, `unpause`, `reset`, +`update-options`, and `terminate`. + +For details about Pause, Unpause, Reset, and Update Options, see +[Standalone Activity operator commands](/activity-operations#standalone-activity-operator-commands). The Temporal Dev Server has Standalone Activities enabled by default for local testing. From 35c009f2b7ef1c42c5caff0d0b5eff44f5514a50 Mon Sep 17 00:00:00 2001 From: Dan Davison Date: Tue, 7 Jul 2026 12:30:03 -0400 Subject: [PATCH 2/3] Edit existing docs --- .../activities/activity-operations.mdx | 93 ++++++++----------- 1 file changed, 40 insertions(+), 53 deletions(-) diff --git a/docs/encyclopedia/activities/activity-operations.mdx b/docs/encyclopedia/activities/activity-operations.mdx index c990d51523..be4df88a85 100644 --- a/docs/encyclopedia/activities/activity-operations.mdx +++ b/docs/encyclopedia/activities/activity-operations.mdx @@ -49,13 +49,13 @@ gRPC API - not for programmatic use in Workflow or Activity code. | Operation | What it does | CLI | | --------------------------------- | -------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------ | | [Pause](#pause) | Stops retries. In-flight execution continues unless the Activity uses Heartbeat. | [`temporal activity pause`](/cli/command-reference/activity#pause) | -| [Unpause](#unpause) | Resumes a Paused Activity. The next execution starts immediately. | [`temporal activity unpause`](/cli/command-reference/activity#unpause) | -| [Reset](#reset) | Clears retry state (attempts, backoff) and schedules a new execution. | [`temporal activity reset`](/cli/command-reference/activity#reset) | -| [Update Options](#update-options) | Changes timeouts, Retry Policy, or Task Queue without restarting the Activity. | [`temporal activity update-options`](/cli/command-reference/activity#update-options) | +| [Unpause](#unpause) | Resumes a Paused Activity. The next execution starts after any remaining retry backoff or start delay. | [`temporal activity unpause`](/cli/command-reference/activity#unpause) | +| [Reset](#reset) | Clears retry state (attempts, backoff) and schedules the Activity for execution again. | [`temporal activity reset`](/cli/command-reference/activity#reset) | +| [Update Options](#update-options) | Changes any Activity options (e.g. timeouts, Retry Policy, or Task Queue) without restarting the Activity. | [`temporal activity update-options`](/cli/command-reference/activity#update-options) | ## Pause {/* #pause */} -Pause stops the Temporal Service from scheduling new retries of an [Activity Execution](/activity-execution). +Pause stops the Temporal Service scheduling new retries of an [Activity Execution](/activity-execution). ### When to Pause @@ -66,20 +66,19 @@ Pause stops the Temporal Service from scheduling new retries of an [Activity Exe ### What happens when you Pause an Activity +- **No further retries are scheduled.** The Temporal Service stops scheduling retries. +- **[Heartbeating](/encyclopedia/detecting-activity-failures#activity-heartbeat) determines whether an interruption is sent to the in-flight + Activity Task Execution:** + - For **Activities with Heartbeat** the interruption is sent on their next Heartbeat. The SDK raises a Pause-specific error; + the execution can catch this to clean up resources before exiting. + - **Activities without Heartbeat** continue running to completion. If the Activity Task Execution succeeds, the result is delivered + to the Workflow normally. If it fails, no retry is scheduled. Pause takes effect after the in-flight execution ends. +- **Pause is idempotent.** Pausing an already-Paused Activity has no effect. Pausing a completed Activity returns an + error. - **Pausing an Activity doesn't affect the parent Workflow.** The Workflow continues Running, and [Signals, Queries, and Updates](/encyclopedia/workflow-message-passing) on the parent Workflow are unaffected. -- **No further retries are scheduled.** The Temporal Service stops scheduling retries. This is enforced server-side, not - by the SDK. - **Workflow code has no visibility into Activity Operations.** Pause doesn't produce an Event History event, so the Workflow can't detect or react to it. See [Observability](#observability). -- **[Heartbeating](/encyclopedia/detecting-activity-failures#activity-heartbeat) determines whether the in-flight - execution is interrupted:** - - **Activities with Heartbeat** are interrupted on their next Heartbeat. The SDK raises a Pause-specific error, and - the Activity can catch this to clean up resources before exiting. - - **Activities without Heartbeat** continue running to completion. If the execution succeeds, the result is delivered - to the Workflow normally. If it fails, no retry is scheduled. Pause takes effect after the in-flight execution ends. -- **Pause is idempotent.** Pausing an already-Paused Activity has no effect. Pausing a completed Activity returns an - error. ### CLI usage @@ -94,7 +93,7 @@ See the [CLI reference for `temporal activity pause`](/cli/command-reference/act ### Detect Pause in Activity code -Activities with Heartbeat can detect that an interruption was caused by Pause rather than a timeout or Workflow +Activities with a Heartbeat can detect that an interruption was caused by Pause rather than by a timeout or Workflow Cancellation. A Paused Activity resumes later. A Cancelled Activity doesn't. Your Activity code may need to handle these cases differently, for example releasing held resources on Pause while preserving them on Cancellation, or vice versa. @@ -111,7 +110,7 @@ cases differently, for example releasing held resources on Pause while preservin [Workflow Pause](/cli/command-reference/workflow#pause) and Activity Pause are independent. Both stop Activity retries, but they must be Unpaused separately. -- Workflow Pause blocks retries but doesn't interrupt in-flight executions via Heartbeat. Activity Pause does. +- Workflow Pause blocks retries but doesn't interrupt in-flight Activity Task Executions via Heartbeat. Activity Pause does. - If both are active, both must be Unpaused before the Activity resumes. ### Important considerations @@ -140,14 +139,11 @@ Unpause resumes a Paused Activity Execution. ### What happens when you Unpause an Activity -- **The Activity is rescheduled immediately.** Any remaining retry backoff is discarded. The next execution starts right - away. -- **Attempt count and Heartbeat data are preserved by default.** The Activity resumes from where it left off. Use - `--reset-attempts` or `--reset-heartbeats` on the CLI to clear these, or use [Reset](#reset) to restart from - attempt 1. +- **The Activity is rescheduled for its next attempt.** The next attempt wait for the end of any remaining retry backoff (or start delay) before starting. Unpause is idempotent. Unpausing an Activity that isn't Paused has no effect. Unpausing an Activity that has already -completed returns an error. +completed returns an error. Use [Reset](#reset) with `--keep-paused` to reset the attempt counter to 1 and optionally +clear the heartbeat data . ### CLI usage @@ -157,8 +153,7 @@ temporal activity unpause \ --activity-id my-activity ``` -See the [CLI reference for `temporal activity unpause`](/cli/command-reference/activity#unpause) for all options, -including `--reset-attempts` and `--reset-heartbeats` to clear state on resume. +See the [CLI reference for `temporal activity unpause`](/cli/command-reference/activity#unpause) for all options. ### Important considerations @@ -179,37 +174,34 @@ including `--reset-attempts` and `--reset-heartbeats` to clear state on resume. ## Reset {/* #reset */} -Reset clears an Activity's retry state and schedules a fresh execution. +Reset clears an Activity's retry state and schedules it to start again from attempt 1. ### When to Reset -- An Activity has exhausted most of its retries, and you want to give it a fresh set after fixing the underlying issue. +- An Activity has exhausted most of its retries, and you want to give it its full retries after fixing the underlying issue. - A Paused Activity needs to start clean after a configuration change or code deploy. -- You want to clear accumulated retry backoff and retry immediately instead of waiting for the next backoff interval. +- You want to retry immediately instead of waiting for the next backoff interval. - A batch of Activities failed due to a transient issue and you want to restart them all with staggered jitter. ### What happens when you Reset an Activity - **The attempt count resets to 1.** The Activity gets a full set of retry attempts regardless of how many it had used. - **Retry backoff is discarded.** If the Activity was in a backoff wait, it's rescheduled to run immediately. -- **If the Activity is Paused, Reset also Unpauses it.** Use `--keep-paused` to Reset the attempt count without resuming - execution. With `--keep-paused`, the attempt count and Heartbeat data (if `--reset-heartbeats`) are reset, but the - Activity stays Paused. No retry is scheduled until you [Unpause](#unpause) separately. +- **If the Activity is Paused, Reset also Unpauses it.** Use `--keep-paused` to Reset without Unpausing. + No attenpt will be scheduled until you [Unpause](#unpause) separately. - **Resetting an Activity doesn't affect the parent Workflow.** The Workflow continues Running, and Signals, Queries, and Updates on the parent Workflow are unaffected. - **Workflow code has no visibility into Activity Operations.** Reset doesn't produce an Event History event, so the Workflow can't detect or react to it. See [Observability](#observability). -- **[Heartbeating](/encyclopedia/detecting-activity-failures#activity-heartbeat) determines whether an in-flight - execution is interrupted:** - - **Activities with Heartbeat** are interrupted on their next Heartbeat. The SDK may raise a Reset-specific error so +- **[Heartbeating](/encyclopedia/detecting-activity-failures#activity-heartbeat) determines whether an interruption will + be sent to an in-flight execution:** + - For **Activities with Heartbeat** an interruption is sent on their next Heartbeat. The SDK raises a Reset-specific error so the Activity can clean up before exiting. The next execution starts at attempt 1. - **Activities without Heartbeat** continue running to completion. Reset doesn't cancel, interrupt, or schedule a - concurrent execution. If the Activity was already retrying, the Temporal Service rejects the current execution's - result because Reset changed the expected attempt number, and a fresh execution is scheduled after the - [Start-To-Close Timeout](/encyclopedia/detecting-activity-failures#start-to-close-timeout) expires. If it was on its - first execution, a successful result is still delivered to the Workflow normally. -- **Reset is idempotent.** Resetting an Activity that's already at attempt 1 with no backoff has no effect. Resetting a - completed Activity returns an error. + concurrent execution. If the Activity was running on the Worker, attempt 1 is scheduled to start after the attempt + finishes. If the attempt succeeds, the Reset request is ignored and the Activity completes successfully. +- **Reset is idempotent.** Resetting an Activity that's already at attempt 1 has no effect. If is is in its start delay period, + it still waits for this to end. Resetting a completed Activity returns an error. ### CLI usage @@ -230,8 +222,8 @@ See the [CLI reference for `temporal activity reset`](/cli/command-reference/act ### Detect Reset in Activity code -Activities with Heartbeat can detect that an interruption was caused by Reset rather than a timeout or Workflow -Cancellation. A Reset Activity is retried from attempt 1. A Cancelled Activity isn't. Your Activity code may need to +Activities with a Heartbeat can detect that an interruption was caused by Reset rather than by a timeout or Workflow +Cancellation. A Reset Activity will be retried from attempt 1. A Cancelled Activity won't be retried. Your Activity code may need to handle these cases differently, for example saving partial progress on Reset while discarding it on Cancellation. | SDK | How to detect Reset | @@ -252,11 +244,9 @@ handle these cases differently, for example saving partial progress on Reset whi want a clean restart, pass `--reset-heartbeats`. - **Reset won't interrupt an Activity that doesn't Heartbeat.** The current execution runs to completion, which could take up to the full [Start-To-Close Timeout](/encyclopedia/detecting-activity-failures#start-to-close-timeout). If the - Activity had already retried (attempt > 1), the Temporal Service rejects the current execution's result because Reset - changed the expected attempt number. The Activity waits for its Start-To-Close Timeout to expire before a new - execution is scheduled. -- **`--restore-original-options` restores the Activity's original configuration.** It reverts timeouts, Retry Policy, - and Task Queue to the values from when the Activity was first scheduled. + current attempt succeeds, the Activity completes successfully and the Reset request is ignored. +- **`--restore-original-options` restores the Activity's original configuration.** It reverts amy changes to timeouts, Retry Policy, + and Task Queue back to the values they had when the Activity was first scheduled. - **Bulk Reset can overwhelm downstream services.** When using `--query` to Reset Activities across many Workflows, use `--jitter` to stagger the restart times. @@ -280,12 +270,11 @@ You can change [timeouts](/encyclopedia/detecting-activity-failures) (Schedule-T Schedule-To-Start, Heartbeat), Retry Policy (initial interval, maximum interval, backoff coefficient, maximum attempts), and Task Queue. Only the fields you specify are changed. All other options remain unchanged. -- **If the Activity is waiting for retry (scheduled),** the new options take effect immediately. Any pending retry timer +- **If the Activity is waiting for its start delay or retry backoff (i.e. in scheduled state),** the new options take effect immediately. Any pending delay timer is regenerated with the updated configuration. -- **If the Activity is currently running,** the new options are stored but take effect on the next execution. The - in-flight execution isn't interrupted. -- **If the Activity is Paused,** the new options are stored immediately. They take effect when the Activity is Unpaused - and the next execution starts. +- **If the Activity is currently running,** the new options take effect on the next attempt. The + in-flight execution isn't interrupted or affected. +- **If the Activity is Paused,** the new options are stored immediately and will affect the next attempt on Unpause. - **Workflow code has no visibility into Activity Operations.** Update Options doesn't produce an Event History event, so the Workflow can't detect or react to it. See [Observability](#observability). @@ -306,8 +295,6 @@ options, including Retry Policy, Task Queue, and bulk mode via `--query`. ### Important considerations -- **Changes to a running Activity take effect on the next execution, not the current one.** If you need the change to - apply immediately, the Activity must finish or fail its current execution first. - **`--restore-original-options` is batch-only.** This flag only works with `--query`. It's silently ignored in single-workflow mode. It can't be combined with other option changes in the same command. From 4caf949813f0891144aacadf74e280eb4c1dc6b1 Mon Sep 17 00:00:00 2001 From: Sean Kane Date: Tue, 7 Jul 2026 11:45:16 -0600 Subject: [PATCH 3/3] additional documentation for reset heartbeats --- docs/cli/command-reference/activity.mdx | 10 ++----- .../activities/activity-operations.mdx | 28 ++++++++----------- 2 files changed, 15 insertions(+), 23 deletions(-) diff --git a/docs/cli/command-reference/activity.mdx b/docs/cli/command-reference/activity.mdx index 1512d372f5..ac40936404 100644 --- a/docs/cli/command-reference/activity.mdx +++ b/docs/cli/command-reference/activity.mdx @@ -236,8 +236,7 @@ Use the following options to change the behavior of this command. You can also u Reset an activity. Not supported for Standalone Activities. This restarts the activity as if it were first being scheduled. That is, it will reset both the number of attempts and the -activity timeout, as well as, optionally, the -[heartbeat details](#reset-heartbeats). +activity timeout, as well as the [heartbeat details](#reset-heartbeats). If the activity may be executing (i.e. it has not yet timed out), the reset will take effect the next time it fails, heartbeats, or times out. @@ -261,16 +260,15 @@ the next time they heartbeat after a reset. If, in your Activity, you need to do any cleanup when an Activity is reset, handle this error and then re-throw it when you've cleaned up. -If the `reset_heartbeats` flag is set, the heartbeat details will also be cleared. +Reset clears heartbeat details. Specify the Activity and Workflow IDs: ``` temporal activity reset \ --activity-id YourActivityId \ - --workflow-id YourWorkflowId + --workflow-id YourWorkflowId \ --keep-paused - --reset-heartbeats ``` Activities can be reset in bulk with a visibility query list filter: @@ -291,7 +289,6 @@ Use the following options to change the behavior of this command. You can also u | `--query`, `-q` | No | **string** Content for an SQL-like `QUERY` List Filter. You must set either --workflow-id or --query. Note: Using --query for batch activity operations is an experimental feature and may change in the future. | | `--reason` | No | **string** Reason for batch operation. Only use with --query. Defaults to user name. | | `--reset-attempts` | No | **bool** Reset the activity attempts. | -| `--reset-heartbeats` | No | **bool** Reset the Activity's heartbeats. | | `--restore-original-options` | No | **bool** Restore the original options of the activity. | | `--rps` | No | **float** Limit batch's requests per second. Only allowed if query is present. | | `--run-id`, `-r` | No | **string** Run ID. Only use with --workflow-id. Cannot use with --query. | @@ -525,4 +522,3 @@ The following options can be used with any command. | `--tls-key-data` | No | **string** Private certificate key data. Can't be used with --tls-key-path. | | | `--tls-key-path` | No | **string** Path to x509 private key. Can't be used with --tls-key-data. | | | `--tls-server-name` | No | **string** Override target TLS server name. | | - diff --git a/docs/encyclopedia/activities/activity-operations.mdx b/docs/encyclopedia/activities/activity-operations.mdx index be4df88a85..226c2971ce 100644 --- a/docs/encyclopedia/activities/activity-operations.mdx +++ b/docs/encyclopedia/activities/activity-operations.mdx @@ -188,7 +188,8 @@ Reset clears an Activity's retry state and schedules it to start again from atte - **The attempt count resets to 1.** The Activity gets a full set of retry attempts regardless of how many it had used. - **Retry backoff is discarded.** If the Activity was in a backoff wait, it's rescheduled to run immediately. - **If the Activity is Paused, Reset also Unpauses it.** Use `--keep-paused` to Reset without Unpausing. - No attenpt will be scheduled until you [Unpause](#unpause) separately. + With `--keep-paused`, the attempt count and Heartbeat data are reset, but the Activity stays Paused. No attempt is + scheduled until you [Unpause](#unpause) separately. - **Resetting an Activity doesn't affect the parent Workflow.** The Workflow continues Running, and Signals, Queries, and Updates on the parent Workflow are unaffected. - **Workflow code has no visibility into Activity Operations.** Reset doesn't produce an Event History event, so the @@ -200,8 +201,6 @@ Reset clears an Activity's retry state and schedules it to start again from atte - **Activities without Heartbeat** continue running to completion. Reset doesn't cancel, interrupt, or schedule a concurrent execution. If the Activity was running on the Worker, attempt 1 is scheduled to start after the attempt finishes. If the attempt succeeds, the Reset request is ignored and the Activity completes successfully. -- **Reset is idempotent.** Resetting an Activity that's already at attempt 1 has no effect. If is is in its start delay period, - it still waits for this to end. Resetting a completed Activity returns an error. ### CLI usage @@ -218,7 +217,7 @@ temporal activity reset \ ``` See the [CLI reference for `temporal activity reset`](/cli/command-reference/activity#reset) for all options, including -`--reset-heartbeats` and bulk mode via `--query`. +bulk mode via `--query`. ### Detect Reset in Activity code @@ -240,13 +239,13 @@ handle these cases differently, for example saving partial progress on Reset whi [Schedule-To-Close Timeout](/encyclopedia/detecting-activity-failures#schedule-to-close-timeout). The deadline is calculated from when the Activity was originally scheduled. Use [`update-options`](#update-options) to extend the timeout before or after Reset. -- **Heartbeat details are preserved by default.** If your Activity uses Heartbeat details for progress tracking and you - want a clean restart, pass `--reset-heartbeats`. +- **Heartbeat details are cleared.** If your Activity uses Heartbeat details for progress tracking, the next attempt + starts without those details after Reset. - **Reset won't interrupt an Activity that doesn't Heartbeat.** The current execution runs to completion, which could take up to the full [Start-To-Close Timeout](/encyclopedia/detecting-activity-failures#start-to-close-timeout). If the current attempt succeeds, the Activity completes successfully and the Reset request is ignored. -- **`--restore-original-options` restores the Activity's original configuration.** It reverts amy changes to timeouts, Retry Policy, - and Task Queue back to the values they had when the Activity was first scheduled. +- **`--restore-original-options` restores the Activity's original configuration.** It reverts any changes to timeouts, + Retry Policy, and Task Queue back to the values they had when the Activity was first scheduled. - **Bulk Reset can overwhelm downstream services.** When using `--query` to Reset Activities across many Workflows, use `--jitter` to stagger the restart times. @@ -457,14 +456,11 @@ Reset clears retry state and schedules, or requests, attempt 1 for a Standalone - After a reset dispatches a new attempt, the attempt's scheduled time reports the reset attempt's dispatch time, not the original first-attempt dispatch time. - Reset discards retry backoff, but it still respects any remaining Start Delay before the first Worker pickup. -- If `reset_heartbeat` is set: - - Reset clears stored heartbeat time and heartbeat details. - - If the Activity is non-running: - - Heartbeat state is cleared immediately. - - If the Activity is running: - - Heartbeat state is cleared when the reset is applied to the next attempt. -- If `reset_heartbeat` is not set: - - Heartbeat details are preserved. +- Reset clears stored heartbeat time and heartbeat details. +- If the Activity is non-running: + - Heartbeat state is cleared immediately. +- If the Activity is running: + - Heartbeat state is cleared when the reset is applied to the next attempt. - If `restore_original_options` is set: - Reset restores the options from when the Standalone Activity was created. - Reset can restore Task Queue, Schedule-To-Close Timeout, Schedule-To-Start Timeout, Start-To-Close Timeout,