Skip to content

[WORKFLOW SDK BUG] workflow connection to sidecar doesn't auto-heal #900

@olitomlinson

Description

@olitomlinson

1.17 RC3

Expected Behavior

When dapr goes away (think transient error, sidecar restart etc) the python workflow worker reconnects automatically when dapr becomes available again.

FYI Dotnet SDK has full auto-healing tolerating the loss of either app or sidecar.

Actual Behavior

The python workflow worker doesn't successfully connect back to the sidecar.

The only way to mitigate is to stop the python app, and then restart it.

Steps to Reproduce the Problem

While the python app is mid-way through processing one or more inflight Activity, kill the python sidecar. Observe the logs of the python app. The logs indicate that the connection is dropped, but there is no evidence from the logs that the connection is attempting to re-establish.

2026-02-05 23:08:11.887 durabletask-worker WARNING: Stream reader: RPC error (code=StatusCode.UNAVAILABLE): <_MultiThreadedRendezvous of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "Socket closed"
	debug_error_string = "UNKNOWN:Error received from peer ipv4:172.21.0.12:50001 {grpc_status:14, grpc_message:"Socket closed"}"

>

2026-02-05 23:08:11,887 - durabletask-worker - WARNING - Stream reader: RPC error (code=StatusCode.UNAVAILABLE): <_MultiThreadedRendezvous of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "Socket closed"
	debug_error_string = "UNKNOWN:Error received from peer ipv4:172.21.0.12:50001 {grpc_status:14, grpc_message:"Socket closed"}"

>

2026-02-05 23:08:11.887 durabletask-worker INFO: Work item stream ended normally
2026-02-05 23:08:11,887 - durabletask-worker - INFO - Work item stream ended normally
2026-02-05 23:08:11,888 - durabletask-worker - INFO - No longer listening for work items
2026-02-05 23:08:11.888 durabletask-worker INFO: No longer listening for work items

Release Note

RELEASE NOTE:

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions