You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "linhongliu-db (via GitHub)" <gi...@apache.org> on 2026/03/17 02:43:32 UTC
[PR] [SPARK-XXXXX][SQL] Preserve SparkThrowable error classes in parmap/awaitResult [spark]
linhongliu-db opened a new pull request, #54846:
URL: https://github.com/apache/spark/pull/54846
### What changes were proposed in this pull request?
`SparkException` at `ParquetFileFormat$.readParquetFootersInParallel` already has the structured error class `FAILED_READ_FILE.CANNOT_READ_FILE_FOOTER` with SQL state `KD001`, but `SparkThreadUtils.awaitResult` wraps it in a generic `SparkException`, hiding the SQL state.
This PR:
1. **Adds `preserveSparkThrowable` parameter** to `SparkThreadUtils.awaitResult`, `ThreadUtils.awaitResult`, and `ThreadUtils.parmap` (default `false` — no behavior change for existing callers)
2. **Updates `ParquetFileFormat.readParquetFootersInParallel`** to pass `preserveSparkThrowable = true` to `parmap`, preserving the structured error class
### Files changed:
- `SparkThreadUtils.scala` — new 3-param `awaitResult` overload
- `ThreadUtils.scala` — new 3-param `awaitResult` overloads (Awaitable + JFuture) + `parmap` flag
- `ParquetFileFormat.scala` — caller passes `preserveSparkThrowable = true` to `parmap`
- `ThreadUtilsSuite.scala` — tests for `awaitResult` and `parmap` preservation
- `ParquetFileFormatSuite.scala` — updated test to verify error class is thrown directly
### Why are the changes needed?
When reading parquet footers in parallel, the `FAILED_READ_FILE.CANNOT_READ_FILE_FOOTER` error class is wrapped in a generic `SparkException` by `awaitResult`, losing the structured SQL state. This makes it harder for users and applications to programmatically handle specific error conditions.
### Does this PR introduce _any_ user-facing change?
Yes. `ParquetFileFormat.readParquetFootersInParallel` now throws the structured `SparkException` directly instead of wrapping it in a generic `SparkException`. The error class and SQL state are preserved.
### How was this patch tested?
1. Added unit tests in `ThreadUtilsSuite` for `awaitResult` (Awaitable and JFuture) and `parmap` with `preserveSparkThrowable` flag true/false
2. Updated `ParquetFileFormatSuite` to verify the error class is thrown directly
### Was this patch authored or co-authored using generative AI tooling?
Yes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56022][SQL] Preserve SparkThrowable error classes in parmap/awaitResult [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan closed pull request #54846: [SPARK-56022][SQL] Preserve SparkThrowable error classes in parmap/awaitResult
URL: https://github.com/apache/spark/pull/54846
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56022][SQL] Preserve SparkThrowable error classes in parmap/awaitResult [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on PR #54846:
URL: https://github.com/apache/spark/pull/54846#issuecomment-4380474245
thanks, merging to master/4.x/4.2!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org