You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "baibaichen (via GitHub)" <gi...@apache.org> on 2025/12/11 07:15:32 UTC
[PR] [GLUTEN-11088][VL] Fix GlutenParquetIOSuite compatibility issues for Spark 4.0 [incubator-gluten]
baibaichen opened a new pull request, #11281:
URL: https://github.com/apache/incubator-gluten/pull/11281
## What changes were proposed in this pull request?
This PR fixes compatibility issues in `GlutenParquetIOSuite` for Spark 4.0 by addressing the following Spark 4.0 shim layer changes:
1. **Respect `mapreduce.output.basename` configuration**: Updated `SparkWriteFilesCommitProtocol` to honor the `mapreduce.output.basename` configuration when generating output file names, aligning with [SPARK-49991](https://github.com/apache/spark/pull/48494).
2. **Proper error handling in write operations**: Replaced direct exception throwing with `GlutenFileFormatWriter.throwWriteError` to use Spark's standardized error handling mechanism (`QueryExecutionErrors.taskFailedWhileWritingRowsError`).
3. **Code quality improvements**:
- Added explicit type annotations to `sparkStageId`, `sparkPartitionId`, and `sparkAttemptNumber` for better type safety
- Changed `fileNames` initialization from `null` to underscore idiom (`_`) for cleaner Scala style
- Migrated from deprecated `scala.collection.JavaConverters` to `scala.jdk.CollectionConverters`
- Simplified `TextScan` instantiation by removing redundant `new` keyword (applies to Scala 3/case class patterns)
4. **Test coverage**: Re-enabled 3 previously excluded tests in `VeloxTestSettings`:
- `SPARK-49991: Respect 'mapreduce.output.basename' to generate file names`
- `SPARK-6330 regression test`
- `SPARK-7837 Do not close output writer twice when commitTask() fails`
## Why are the changes needed?
The Spark 4.0 upgrade introduced breaking changes in the shim layer:
- The file naming convention now supports custom basename configuration through `mapreduce.output.basename`
- Error handling APIs were refactored to use centralized error builders
- The previous direct exception throwing approach is incompatible with Spark 4.0's error handling framework
Without these changes, `GlutenParquetIOSuite` tests fail due to:
1. Incorrect file name generation (missing basename support)
2. Incompatible exception types when write operations fail
3. Deprecated Scala collection conversion APIs
## How was this patch tested?
- Re-enabled and verified all 3 previously excluded tests pass successfully
- Existing GlutenParquetIOSuite tests continue to pass
- Validated file naming with custom `mapreduce.output.basename` configurations
- Confirmed error handling produces correct exception types and messages
## Related Issue
Addresses #11088 (Track on Spark-4.0 failed unit tests)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-11088][VL] Fix GlutenParquetIOSuite compatibility issues for Spark 4.0 [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #11281:
URL: https://github.com/apache/incubator-gluten/pull/11281#issuecomment-3644526793
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-11088][VL] Fix GlutenParquetIOSuite compatibility issues for Spark 4.0 [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #11281:
URL: https://github.com/apache/incubator-gluten/pull/11281#issuecomment-3649032594
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-11088][VL] Fix GlutenParquetIOSuite compatibility issues for Spark 4.0 [incubator-gluten]
Posted by "baibaichen (via GitHub)" <gi...@apache.org>.
baibaichen commented on PR #11281:
URL: https://github.com/apache/incubator-gluten/pull/11281#issuecomment-3654336497
> @baibaichen it looks like we should disable below ANSI test as Gluten-velox does not support it yet
>
> ```
> 2025-12-13T07:06:56.4659042Z - Throw exceptions on inserting out-of-range int value with ANSI casting policy *** FAILED ***
> 2025-12-13T07:06:56.4660344Z Expected exception org.apache.spark.SparkArithmeticException to be thrown, but org.apache.spark.SparkException was thrown (InsertSuite.scala:775)
> 2025-12-13T07:06:56.4840500Z 07:06:56.483 WARN org.apache.spark.sql.execution.GlutenFallbackReporter: Validation failed for plan: Project[QueryId=47908], due to:
> 2025-12-13T07:06:56.4842112Z - Validation failed with exception from: ProjectExecTransformer, reason: CheckOverflowInTableInsert is used in ANSI mode, but Gluten does not support ANSI mode.
> 2025-12-13T07:06:56.4843981Z
> ```
Thanks @zhouyuan. This isn't related to ANSI mode; the issue was introduced by my fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-11088][VL] Fix GlutenParquetIOSuite compatibility issues for Spark 4.0 [incubator-gluten]
Posted by "zhouyuan (via GitHub)" <gi...@apache.org>.
zhouyuan commented on PR #11281:
URL: https://github.com/apache/incubator-gluten/pull/11281#issuecomment-3652483185
@baibaichen
it looks like we should disable below ANSI test as Gluten-velox does not support it yet
```
2025-12-13T07:06:56.4659042Z - Throw exceptions on inserting out-of-range int value with ANSI casting policy *** FAILED ***
2025-12-13T07:06:56.4660344Z Expected exception org.apache.spark.SparkArithmeticException to be thrown, but org.apache.spark.SparkException was thrown (InsertSuite.scala:775)
2025-12-13T07:06:56.4840500Z 07:06:56.483 WARN org.apache.spark.sql.execution.GlutenFallbackReporter: Validation failed for plan: Project[QueryId=47908], due to:
2025-12-13T07:06:56.4842112Z - Validation failed with exception from: ProjectExecTransformer, reason: CheckOverflowInTableInsert is used in ANSI mode, but Gluten does not support ANSI mode.
2025-12-13T07:06:56.4843981Z
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-11088][VL] Fix GlutenParquetIOSuite compatibility issues for Spark 4.0 [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #11281:
URL: https://github.com/apache/incubator-gluten/pull/11281#issuecomment-3649032913
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-11088][VL] Fix GlutenParquetIOSuite compatibility issues for Spark 4.0 [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #11281:
URL: https://github.com/apache/incubator-gluten/pull/11281#issuecomment-3654302569
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-11088][VL] Fix GlutenParquetIOSuite compatibility issues for Spark 4.0 [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #11281:
URL: https://github.com/apache/incubator-gluten/pull/11281#issuecomment-3640575698
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-11088][VL] Fix GlutenParquetIOSuite compatibility issues for Spark 4.0 [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #11281:
URL: https://github.com/apache/incubator-gluten/pull/11281#issuecomment-3658368241
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-11088][VL] Fix GlutenParquetIOSuite compatibility issues for Spark 4.0 [incubator-gluten]
Posted by "baibaichen (via GitHub)" <gi...@apache.org>.
baibaichen merged PR #11281:
URL: https://github.com/apache/incubator-gluten/pull/11281
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org