You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "azmatsiddique (via GitHub)" <gi...@apache.org> on 2026/03/22 13:12:58 UTC
[PR] [SPARK-54916][ML] Fix Parquet footer error in DecisionTree test suites by caching RDDs [spark]
azmatsiddique opened a new pull request, #54941:
URL: https://github.com/apache/spark/pull/54941
What changes were proposed in this pull request?
This PR fixes a CANNOT_READ_FILE_FOOTER error encountered in DecisionTreeClassifierSuite and DecisionTreeRegressorSuite when running with Scala 2.13 in Spark 4.2.0-preview3.
The fix involves explicitly calling .cache() on the RDDs initialized in beforeAll() (and in TreeTests.getTreeReadWriteData). This forces the materialization of the underlying ArraySeq (introduced via .toImmutableArraySeq) before the test cases attempt to fit models, which involves writing temporary Parquet files.
Why are the changes needed?
Following the migration to Scala 2.13 collection conversions (specifically toImmutableArraySeq), certain ML test suites began failing with corrupted Parquet files.
The root cause appears to be a race condition or serialization issue where the lazy evaluation of the ImmutableArraySeq coincided with Spark's background Parquet writing during DecisionTree.fit. By caching the RDDs immediately after creation in beforeAll, we ensure the data is fully materialized and stable before any Parquet write operations begin.
Does this PR introduce any user-facing change?
No. This is a fix for internal test suites only.
How was this patch tested?
Verified locally by running the affected test suite: build/sbt "mllib/testOnly org.apache.spark.ml.classification.DecisionTreeClassifierSuite" The tests now pass consistently without Parquet footer errors.
Was this patch authored or co-authored using generative AI tooling?
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54916][ML] Fix Parquet footer error in DecisionTree test suites by caching RDDs [spark]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #54941: [SPARK-54916][ML] Fix Parquet footer error in DecisionTree test suites by caching RDDs
URL: https://github.com/apache/spark/pull/54941
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54916][ML] Fix Parquet footer error in DecisionTree test suites by caching RDDs [spark]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #54941:
URL: https://github.com/apache/spark/pull/54941#issuecomment-4849311024
We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org