You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Dongjoon Hyun (Jira)" <ji...@apache.org> on 2026/10/06 19:28:00 UTC
[jira] [Resolved] (SPARK-59884) Avoid serializing cached columnar RDDs in whole-stage closures
[ https://issues.apache.org/jira/browse/SPARK-59884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dongjoon Hyun resolved SPARK-59884.
-----------------------------------
Fix Version/s: 4.4.0
Resolution: Fixed
Issue resolved by pull request 59146
[https://github.com/apache/spark/pull/59146]
> Avoid serializing cached columnar RDDs in whole-stage closures
> --------------------------------------------------------------
>
> Key: SPARK-59884
> URL: https://issues.apache.org/jira/browse/SPARK-59884
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 4.2.0
> Reporter: Chao Sun
> Assignee: Chao Sun
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.4.0
>
>
> SparkPlan.executeColumnarRDD retains the LazyTry containing a plan's columnar execution RDD, but unlike executeRDD it is not transient.
> Whole-stage sort-merge join code generation adds the join plan to CodegenContext.references for executor cleanup. If a descendant columnar plan has initialized executeColumnarRDD, serializing the WholeStageCodegenEvaluatorFactory also follows that driver-side cached RDD. This unnecessarily captures the cached RDD object graph and can fail with NotSerializableException when it contains driver-only state. Executable inputs are already supplied to the evaluator separately.
> A bounded reproducer builds a SortMergeJoinExec with a ColumnarToRowExec child whose cached RDD has a nonserializable sentinel, obtains actual whole-stage codegen references, initializes the columnar cache, and serializes the evaluator with Spark's Java closure serializer. Upstream fails through:
> WholeStageCodegenEvaluatorFactory.references -> SortMergeJoinExec -> ColumnarToRowExec -> SparkPlan.executeColumnarRDD -> LazyTry.tryT -> cached RDD -> nonserializable sentinel
> Marking executeColumnarRDD @transient fixes this capture path and matches executeRDD. Driver-side cache reuse is unchanged; serialization, deserialization, and cleanup succeed with the fix.
> The issue was reproduced using Apache Spark branch-4.2 source at fc01a9e3165f9c371074390f4b3e6f7aeabef04b against public Spark 4.2.0 dependencies. The same nontransient field exists on current master and branches 4.x, 4.2, 4.1, and 4.0. Branch 3.5 directly invokes doExecuteColumnar and has no such cache field.
> This report concerns the demonstrated cached-RDD capture path. It does not claim to eliminate every source of large task closures.
> Proposed fix and regression test: https://github.com/apache/spark/pull/59146
> Current-master validation at a026a0be4c88e7ace59d1da7c18ab73b0d81edde: the focused regression passes with the fix and fails with NotSerializableException when only the annotation is removed. The exact changed sources were compiled against public Spark 4.2.0 dependencies; this is not a full current-master build. Scoped Scalastyle passes with zero errors and warnings.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org