You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Herman van Hövell (Jira)" <ji...@apache.org> on 2025/06/05 14:07:00 UTC
[jira] [Resolved] (SPARK-52397) Idempotent ExecutePlan: second ExecutePlan with same operationId and plan should reattach
[ https://issues.apache.org/jira/browse/SPARK-52397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Herman van Hövell resolved SPARK-52397.
---------------------------------------
Fix Version/s: 4.1.0
4.0.1
Resolution: Fixed
Issue resolved by pull request 51084
[https://github.com/apache/spark/pull/51084]
> Idempotent ExecutePlan: second ExecutePlan with same operationId and plan should reattach
> -----------------------------------------------------------------------------------------
>
> Key: SPARK-52397
> URL: https://issues.apache.org/jira/browse/SPARK-52397
> Project: Spark
> Issue Type: Improvement
> Components: Connect
> Affects Versions: 4.0.0, 4.1.0
> Reporter: Xi Lyu
> Assignee: Xi Lyu
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.1.0, 4.0.1
>
>
> In Spark Connect, queries can fail with the error INVALID_HANDLE.OPERATION_ALREADY_EXISTS, when a client retries an ExecutePlan RPC—often due to transient network issues—causing the server to receive the same request multiple times. Since each ExecutePlan request includes an operation_id, the server interprets the duplicate as an attempt to create an already existing operation, which results in the OPERATION_ALREADY_EXISTS exception. This behavior interrupts query execution and breaks the user experience under otherwise recoverable conditions.
> To resolve this, we should introduce idempotent handling of ExecutePlan on the server side. When a request with a previously seen operation_id and plan is received, instead of returning an error, the server now reattaches the response stream to the already running execution associated with that operation. This ensures that retries due to network flakiness no longer result in failed queries, thereby improving the resilience and robustness of query executions.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org