You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Chao Sun (Jira)" <ji...@apache.org> on 2026/07/29 14:49:00 UTC
[jira] [Resolved] (SPARK-58412) Incorrect cache statistics after partial materialization or cache rebuild
[ https://issues.apache.org/jira/browse/SPARK-58412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chao Sun resolved SPARK-58412.
------------------------------
Fix Version/s: 4.3.0
5.0.0
Resolution: Fixed
> Incorrect cache statistics after partial materialization or cache rebuild
> -------------------------------------------------------------------------
>
> Key: SPARK-58412
> URL: https://issues.apache.org/jira/browse/SPARK-58412
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.2.0
> Reporter: Chao Sun
> Assignee: Chao Sun
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.3.0, 5.0.0
>
>
> CachedRDDBuilder can publish incorrect materialization statistics in two lifecycle cases.
> First, a task completion listener currently records per-partition row and byte counts whenever a task finishes successfully, even if the cache-building iterator was not exhausted. This can happen when a MEMORY_ONLY block cannot be stored and the partially unrolled iterator is returned to a downstream consumer that stops early. Because PartitionKeyedAccumulator is last-write-wins per partition, a partial recomputation can replace a complete value, such as (10 rows, N bytes), with (0 rows, 0 bytes) while leaving the partition key present. InMemoryRelation can therefore still appear fully materialized and AQE may treat a non-empty cache as empty.
> Second, clearCache resets the current accumulator in place. Tasks from the retired cache generation have already captured that same accumulator. Their late completions can write old partition keys and values after the reset, contaminating the rebuilt generation and potentially making it appear complete before its own partitions finish.
> The fix should:
> * publish partition statistics only after the wrapped iterator is fully consumed and the task completes successfully; and
> * install a new PartitionKeyedAccumulator for every cache generation instead of resetting the previous accumulator.
> This issue is related to SPARK-57547, which introduced partition-keyed materialization bookkeeping and is present in Spark 4.2.0. The fixes apply to general InMemoryRelation cache bookkeeping and are independent of runtime Bloom filters.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org