[jira] [Assigned] (SPARK-59174) State-store instance metrics are shipped in every task result

["Jungtaek Lim (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-59174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jungtaek Lim reassigned SPARK-59174:
------------------------------------

    Assignee: Diveyam Mishra

> State-store instance metrics are shipped in every task result
> -------------------------------------------------------------
>
>                 Key: SPARK-59174
>                 URL: https://issues.apache.org/jira/browse/SPARK-59174
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL, Structured Streaming
>    Affects Versions: 4.1.0, 4.2.0, 4.3.0
>            Reporter: Diveyam Mishra
>            Assignee: Diveyam Mishra
>            Priority: Minor
>              Labels: pull-request-available
>
> Stateful streaming operators (StateStoreWriter) currently create a separate SQLMetric accumulator for every (state partition x instance metric x store name) combination upfront on the physical plan. On executors, every task deserializes and registers all of these accumulators with its TaskContext, and returns all of them in the task result payload even though only one partition was processed. For queries with high partition counts (e.g., 20,000 shuffle partitions), this causes inflated task result sizes (~3 MiB/task, ~60 GiB total across tasks), leading to spark.driver.maxResultSize failures in terminal ResultStages (such as RDD checkpointing and file writes) and severe driver memory pressure.
> For more details checkout : https://github.com/apache/spark/issues/58394 
> *Proposed Solution*
> 1. Replace the upfront O(numPartitions) SQLMetric accumulators with a single CollectionAccumulator on StateStoreWriter, allowing each executor task to append only its own active partition metrics.
> 2. In StateStoreWriter.getProgress(), the driver aggregates and reports the top-K instance metrics directly from this single collection accumulator.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org