You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2026/09/07 13:26:00 UTC

[jira] [Updated] (SPARK-59135) Spark Connect command streams fail on valid non-dictionary ExecutePlan responses

     [ https://issues.apache.org/jira/browse/SPARK-59135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

ASF GitHub Bot updated SPARK-59135:
-----------------------------------
    Labels: pull-request-available  (was: )

> Spark Connect command streams fail on valid non-dictionary ExecutePlan responses
> --------------------------------------------------------------------------------
>
>                 Key: SPARK-59135
>                 URL: https://issues.apache.org/jira/browse/SPARK-59135
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect, PySpark
>    Affects Versions: 4.2.0
>            Reporter: Haoyan Geng
>            Priority: Major
>              Labels: pull-request-available
>
> ### Problem
> `SparkConnectClient.execute_command_as_iterator` assumes every decoded `ExecutePlanResponse` item is a dictionary. However, the response decoder legitimately produces other values, including `PlanMetrics`, `PlanObservedMetrics`, schemas, Arrow batches, and protobuf extensions.
> Python Data Source workers currently register profiler accumulators even when `spark.sql.pyspark.dataSource.profiler` is unset. Their empty accumulator updates can therefore be returned as observed metrics alongside command results. When this happens, `execute_command_as_iterator` raises `UNKNOWN_RESPONSE` and terminates the command stream.
> ### Impact
> This affects PySpark Spark Connect clients. Known affected consumers are the streaming-query listener bus and Declarative Pipelines event stream, so the problem is not limited to streaming queries. Accumulators are currently registered unconditionally.
> ### Expected behavior
> The low-level command iterator should preserve every valid decoded response. Command-specific consumers should ignore recognized execution metadata while continuing to reject malformed or unrelated responses. Python Data Source workers should only create profiler accumulators when performance or memory profiling is enabled. Ordinary accumulator updates must remain unaffected.
> ### Proposed change
> - Make `execute_command_as_iterator` yield every decoded response item.
> - Let streaming-listener and pipeline consumers ignore `PlanMetrics` and `ObservedMetrics`.
> - Continue raising errors for unexpected command-specific responses.
> - Register Python Data Source profiler accumulators only for `perf` and `memory` profiling modes.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org