You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "sumitsingh-in (via GitHub)" <gi...@apache.org> on 2026/09/02 10:02:11 UTC
[PR] fix(spark): honor plan-time OPTION_RETURNING_BATCH in legacy parquet … [hudi]
sumitsingh-in opened a new pull request, #19822:
URL: https://github.com/apache/hudi/pull/19822
### Describe the issue this Pull Request addresses
Hudi's legacy (pre-file-group-reader) Spark Parquet file format
(`Spark3LegacyHoodieParquetFileFormat` and its per-Spark-version subclasses)
decides whether to hand back `ColumnarBatch` or row-based `InternalRow`
objects by recomputing `supportBatch(schema)` itself, ignoring
`FileFormat.OPTION_RETURNING_BATCH` — the option `FileSourceScanExec`
already threads down with the batching decision it made at plan time (and
which vanilla Spark's own `ParquetFileFormat` does honor).
The two answers disagree whenever `FileSourceScanExec.supportsColumnar`
depends on something `supportBatch` alone does not account for — most
commonly, `spark.sql.codegen.wholeStage` being disabled for the query (or
another operator's schema tripping `spark.sql.codegen.maxFields`). In that
case the planner marks the scan row-based ("Batched: false" in `explain`),
but the reader still returns `ColumnarBatch` objects. Any consumer that
reads the RDD as rows — most visibly `BroadcastExchangeExec`'s collection
path — then fails with:
java.lang.ClassCastException: class org.apache.spark.sql.vectorized.ColumnarBatch
cannot be cast to class org.apache.spark.sql.catalyst.InternalRow
Closes #19817
### Summary and Changelog
Reading a Hudi table through the legacy Parquet read path inside a query
with whole-stage codegen disabled (e.g. because it exceeds
`spark.sql.codegen.maxFields`, or has it turned off explicitly) could throw
a `ClassCastException` during execution, most commonly during a broadcast
join. This fixes it by making the reader honor
`FileFormat.OPTION_RETURNING_BATCH` when the plan supplies it, falling back
to the existing `supportBatch`-based computation only when the option is
absent.
- `Spark3LegacyHoodieParquetFileFormat.buildReaderWithPartitionValues`: read
`options.get(FileFormat.OPTION_RETURNING_BATCH)` first, matching the
precedent already set by Spark's own `ParquetFileFormat`.
- Added `TestLegacyParquetReadPath#testBroadcastJoinHonorsPlanTimeBatchingDecision`,
a regression test that writes a flat, all-atomic-type COW table (so
`supportBatch` alone would trivially say yes), disables
`spark.sql.codegen.wholeStage`, and broadcast-joins the legacy-format
DataFrame — reproducing the exact crash and asserting it no longer occurs.
No code was copied from elsewhere; the fix mirrors the existing
`OPTION_RETURNING_BATCH` handling already present in Spark's own
`ParquetFileFormat`.
### Impact
None to any public API, config, or storage format. Behavior-only fix to the
legacy Parquet read path, and only in the previously-broken case (whole-stage
codegen disabled or a wide-schema query alongside this scan). No change to
the common case where whole-stage codegen is enabled.
### Risk Level
low — the change is a single, narrowly-scoped computation gated behind a
Spark-supplied option key that is already part of Spark's public
`FileFormat` API, follows the exact precedent of vanilla Spark's own
`ParquetFileFormat`, and is covered by a new regression test reproducing the
originally-reported crash.
### Documentation Update
none — internal bug fix, no new config, feature, or user-facing behavior
change.
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): honor plan-time OPTION_RETURNING_BATCH in legacy parquet … [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19822:
URL: https://github.com/apache/hudi/pull/19822#issuecomment-5790617747
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0dbd1e9d2f8b0e4860a3633d2093a5cf9d07e79d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16770",
"triggerID" : "0dbd1e9d2f8b0e4860a3633d2093a5cf9d07e79d",
"triggerType" : "PUSH"
}, {
"hash" : "f8c81acdae41941880e2a3c266eed2093776ff5a",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16771",
"triggerID" : "f8c81acdae41941880e2a3c266eed2093776ff5a",
"triggerType" : "PUSH"
}, {
"hash" : "87b0d2b9f47b2d7c3400b49db324157fb1716e69",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17250",
"triggerID" : "87b0d2b9f47b2d7c3400b49db324157fb1716e69",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f8c81acdae41941880e2a3c266eed2093776ff5a Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16771)
* 87b0d2b9f47b2d7c3400b49db324157fb1716e69 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17250)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): honor plan-time OPTION_RETURNING_BATCH in legacy parquet … [hudi]
Posted by "sumitsingh-in (via GitHub)" <gi...@apache.org>.
sumitsingh-in commented on PR #19822:
URL: https://github.com/apache/hudi/pull/19822#issuecomment-5507861795
@codope PTAL
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): honor plan-time OPTION_RETURNING_BATCH in legacy parquet … [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19822:
URL: https://github.com/apache/hudi/pull/19822#issuecomment-5792965876
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0dbd1e9d2f8b0e4860a3633d2093a5cf9d07e79d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16770",
"triggerID" : "0dbd1e9d2f8b0e4860a3633d2093a5cf9d07e79d",
"triggerType" : "PUSH"
}, {
"hash" : "f8c81acdae41941880e2a3c266eed2093776ff5a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16771",
"triggerID" : "f8c81acdae41941880e2a3c266eed2093776ff5a",
"triggerType" : "PUSH"
}, {
"hash" : "87b0d2b9f47b2d7c3400b49db324157fb1716e69",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17250",
"triggerID" : "87b0d2b9f47b2d7c3400b49db324157fb1716e69",
"triggerType" : "PUSH"
}, {
"hash" : "f2b51c7b7bca72e087b0fbc3b8f375e0fe3b8dff",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17252",
"triggerID" : "f2b51c7b7bca72e087b0fbc3b8f375e0fe3b8dff",
"triggerType" : "PUSH"
}, {
"hash" : "3a2e0e8ce77375641286cf121687c7a1ab51853d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17255",
"triggerID" : "3a2e0e8ce77375641286cf121687c7a1ab51853d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f2b51c7b7bca72e087b0fbc3b8f375e0fe3b8dff Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17252)
* 3a2e0e8ce77375641286cf121687c7a1ab51853d Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17255)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): honor plan-time OPTION_RETURNING_BATCH in legacy parquet … [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19822:
URL: https://github.com/apache/hudi/pull/19822#issuecomment-5792885982
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0dbd1e9d2f8b0e4860a3633d2093a5cf9d07e79d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16770",
"triggerID" : "0dbd1e9d2f8b0e4860a3633d2093a5cf9d07e79d",
"triggerType" : "PUSH"
}, {
"hash" : "f8c81acdae41941880e2a3c266eed2093776ff5a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16771",
"triggerID" : "f8c81acdae41941880e2a3c266eed2093776ff5a",
"triggerType" : "PUSH"
}, {
"hash" : "87b0d2b9f47b2d7c3400b49db324157fb1716e69",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17250",
"triggerID" : "87b0d2b9f47b2d7c3400b49db324157fb1716e69",
"triggerType" : "PUSH"
}, {
"hash" : "f2b51c7b7bca72e087b0fbc3b8f375e0fe3b8dff",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17252",
"triggerID" : "f2b51c7b7bca72e087b0fbc3b8f375e0fe3b8dff",
"triggerType" : "PUSH"
}, {
"hash" : "3a2e0e8ce77375641286cf121687c7a1ab51853d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3a2e0e8ce77375641286cf121687c7a1ab51853d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f2b51c7b7bca72e087b0fbc3b8f375e0fe3b8dff Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17252)
* 3a2e0e8ce77375641286cf121687c7a1ab51853d UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): honor plan-time OPTION_RETURNING_BATCH in legacy parquet … [hudi]
Posted by "sumitsingh-in (via GitHub)" <gi...@apache.org>.
sumitsingh-in commented on code in PR #19822:
URL: https://github.com/apache/hudi/pull/19822#discussion_r4080079534
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestLegacyParquetReadPath.scala:
##########
@@ -301,6 +306,39 @@ class TestLegacyParquetReadPath extends HoodieSparkClientTestBase with ScalaAsse
}
}
+ @Test
+ def testBroadcastJoinHonorsPlanTimeBatchingDecision(): Unit = {
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): honor plan-time OPTION_RETURNING_BATCH in legacy parquet … [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19822:
URL: https://github.com/apache/hudi/pull/19822#issuecomment-5791199006
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0dbd1e9d2f8b0e4860a3633d2093a5cf9d07e79d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16770",
"triggerID" : "0dbd1e9d2f8b0e4860a3633d2093a5cf9d07e79d",
"triggerType" : "PUSH"
}, {
"hash" : "f8c81acdae41941880e2a3c266eed2093776ff5a",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16771",
"triggerID" : "f8c81acdae41941880e2a3c266eed2093776ff5a",
"triggerType" : "PUSH"
}, {
"hash" : "87b0d2b9f47b2d7c3400b49db324157fb1716e69",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17250",
"triggerID" : "87b0d2b9f47b2d7c3400b49db324157fb1716e69",
"triggerType" : "PUSH"
}, {
"hash" : "f2b51c7b7bca72e087b0fbc3b8f375e0fe3b8dff",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f2b51c7b7bca72e087b0fbc3b8f375e0fe3b8dff",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f8c81acdae41941880e2a3c266eed2093776ff5a Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16771)
* 87b0d2b9f47b2d7c3400b49db324157fb1716e69 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17250)
* f2b51c7b7bca72e087b0fbc3b8f375e0fe3b8dff UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): honor plan-time OPTION_RETURNING_BATCH in legacy parquet … [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19822:
URL: https://github.com/apache/hudi/pull/19822#issuecomment-5791739560
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0dbd1e9d2f8b0e4860a3633d2093a5cf9d07e79d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16770",
"triggerID" : "0dbd1e9d2f8b0e4860a3633d2093a5cf9d07e79d",
"triggerType" : "PUSH"
}, {
"hash" : "f8c81acdae41941880e2a3c266eed2093776ff5a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16771",
"triggerID" : "f8c81acdae41941880e2a3c266eed2093776ff5a",
"triggerType" : "PUSH"
}, {
"hash" : "87b0d2b9f47b2d7c3400b49db324157fb1716e69",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17250",
"triggerID" : "87b0d2b9f47b2d7c3400b49db324157fb1716e69",
"triggerType" : "PUSH"
}, {
"hash" : "f2b51c7b7bca72e087b0fbc3b8f375e0fe3b8dff",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17252",
"triggerID" : "f2b51c7b7bca72e087b0fbc3b8f375e0fe3b8dff",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 87b0d2b9f47b2d7c3400b49db324157fb1716e69 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17250)
* f2b51c7b7bca72e087b0fbc3b8f375e0fe3b8dff Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17252)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): honor plan-time OPTION_RETURNING_BATCH in legacy parquet … [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19822:
URL: https://github.com/apache/hudi/pull/19822#issuecomment-5795354854
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0dbd1e9d2f8b0e4860a3633d2093a5cf9d07e79d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16770",
"triggerID" : "0dbd1e9d2f8b0e4860a3633d2093a5cf9d07e79d",
"triggerType" : "PUSH"
}, {
"hash" : "f8c81acdae41941880e2a3c266eed2093776ff5a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16771",
"triggerID" : "f8c81acdae41941880e2a3c266eed2093776ff5a",
"triggerType" : "PUSH"
}, {
"hash" : "87b0d2b9f47b2d7c3400b49db324157fb1716e69",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17250",
"triggerID" : "87b0d2b9f47b2d7c3400b49db324157fb1716e69",
"triggerType" : "PUSH"
}, {
"hash" : "f2b51c7b7bca72e087b0fbc3b8f375e0fe3b8dff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17252",
"triggerID" : "f2b51c7b7bca72e087b0fbc3b8f375e0fe3b8dff",
"triggerType" : "PUSH"
}, {
"hash" : "3a2e0e8ce77375641286cf121687c7a1ab51853d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17255",
"triggerID" : "3a2e0e8ce77375641286cf121687c7a1ab51853d",
"triggerType" : "PUSH"
}, {
"hash" : "0733587bcfa350068eef96fc513efd07bed39382",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17259",
"triggerID" : "0733587bcfa350068eef96fc513efd07bed39382",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0733587bcfa350068eef96fc513efd07bed39382 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17259)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): honor plan-time OPTION_RETURNING_BATCH in legacy parquet … [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19822:
URL: https://github.com/apache/hudi/pull/19822#issuecomment-5790549195
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0dbd1e9d2f8b0e4860a3633d2093a5cf9d07e79d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16770",
"triggerID" : "0dbd1e9d2f8b0e4860a3633d2093a5cf9d07e79d",
"triggerType" : "PUSH"
}, {
"hash" : "f8c81acdae41941880e2a3c266eed2093776ff5a",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16771",
"triggerID" : "f8c81acdae41941880e2a3c266eed2093776ff5a",
"triggerType" : "PUSH"
}, {
"hash" : "87b0d2b9f47b2d7c3400b49db324157fb1716e69",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "87b0d2b9f47b2d7c3400b49db324157fb1716e69",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f8c81acdae41941880e2a3c266eed2093776ff5a Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16771)
* 87b0d2b9f47b2d7c3400b49db324157fb1716e69 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): honor plan-time OPTION_RETURNING_BATCH in legacy parquet … [hudi]
Posted by "sumitsingh-in (via GitHub)" <gi...@apache.org>.
sumitsingh-in commented on code in PR #19822:
URL: https://github.com/apache/hudi/pull/19822#discussion_r4080080929
##########
hudi-spark-datasource/hudi-spark3-common/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/Spark3LegacyHoodieParquetFileFormat.scala:
##########
@@ -171,7 +171,11 @@ abstract class Spark3LegacyHoodieParquetFileFormat(shouldAppendPartitionValues:
val int96RebaseModeInRead = parquetOptions.int96RebaseModeInRead
val timeZoneId = Option(sqlConf.sessionLocalTimeZone)
// Whole stage codegen (PhysicalRDD) is able to deal with batches directly.
- val returningBatch = getReturningBatch(sparkSession, resultSchema)
+ // Respect the plan-time OPTION_RETURNING_BATCH decision when present, instead of recomputing it here.
+ val returningBatch = enableVectorizedReader &&
Review Comment:
added fix in these module as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): honor plan-time OPTION_RETURNING_BATCH in legacy parquet … [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19822:
URL: https://github.com/apache/hudi/pull/19822#issuecomment-5791421401
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0dbd1e9d2f8b0e4860a3633d2093a5cf9d07e79d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16770",
"triggerID" : "0dbd1e9d2f8b0e4860a3633d2093a5cf9d07e79d",
"triggerType" : "PUSH"
}, {
"hash" : "f8c81acdae41941880e2a3c266eed2093776ff5a",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16771",
"triggerID" : "f8c81acdae41941880e2a3c266eed2093776ff5a",
"triggerType" : "PUSH"
}, {
"hash" : "87b0d2b9f47b2d7c3400b49db324157fb1716e69",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17250",
"triggerID" : "87b0d2b9f47b2d7c3400b49db324157fb1716e69",
"triggerType" : "PUSH"
}, {
"hash" : "f2b51c7b7bca72e087b0fbc3b8f375e0fe3b8dff",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17252",
"triggerID" : "f2b51c7b7bca72e087b0fbc3b8f375e0fe3b8dff",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f8c81acdae41941880e2a3c266eed2093776ff5a Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16771)
* 87b0d2b9f47b2d7c3400b49db324157fb1716e69 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17250)
* f2b51c7b7bca72e087b0fbc3b8f375e0fe3b8dff Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17252)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): honor plan-time OPTION_RETURNING_BATCH in legacy parquet … [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19822:
URL: https://github.com/apache/hudi/pull/19822#issuecomment-5508712223
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0dbd1e9d2f8b0e4860a3633d2093a5cf9d07e79d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16770",
"triggerID" : "0dbd1e9d2f8b0e4860a3633d2093a5cf9d07e79d",
"triggerType" : "PUSH"
}, {
"hash" : "f8c81acdae41941880e2a3c266eed2093776ff5a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16771",
"triggerID" : "f8c81acdae41941880e2a3c266eed2093776ff5a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0dbd1e9d2f8b0e4860a3633d2093a5cf9d07e79d Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16770)
* f8c81acdae41941880e2a3c266eed2093776ff5a Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16771)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): honor plan-time OPTION_RETURNING_BATCH in legacy parquet … [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19822:
URL: https://github.com/apache/hudi/pull/19822#issuecomment-5508302520
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0dbd1e9d2f8b0e4860a3633d2093a5cf9d07e79d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16770",
"triggerID" : "0dbd1e9d2f8b0e4860a3633d2093a5cf9d07e79d",
"triggerType" : "PUSH"
}, {
"hash" : "f8c81acdae41941880e2a3c266eed2093776ff5a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f8c81acdae41941880e2a3c266eed2093776ff5a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0dbd1e9d2f8b0e4860a3633d2093a5cf9d07e79d Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16770)
* f8c81acdae41941880e2a3c266eed2093776ff5a UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): honor plan-time OPTION_RETURNING_BATCH in legacy parquet … [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19822:
URL: https://github.com/apache/hudi/pull/19822#issuecomment-5509847957
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0dbd1e9d2f8b0e4860a3633d2093a5cf9d07e79d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16770",
"triggerID" : "0dbd1e9d2f8b0e4860a3633d2093a5cf9d07e79d",
"triggerType" : "PUSH"
}, {
"hash" : "f8c81acdae41941880e2a3c266eed2093776ff5a",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16771",
"triggerID" : "f8c81acdae41941880e2a3c266eed2093776ff5a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f8c81acdae41941880e2a3c266eed2093776ff5a Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16771)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): honor plan-time OPTION_RETURNING_BATCH in legacy parquet … [hudi]
Posted by "sumitsingh-in (via GitHub)" <gi...@apache.org>.
sumitsingh-in commented on code in PR #19822:
URL: https://github.com/apache/hudi/pull/19822#discussion_r4081680903
##########
hudi-spark-datasource/hudi-spark4.0.x/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/Spark40LegacyHoodieParquetFileFormat.scala:
##########
@@ -152,10 +152,11 @@ class Spark40LegacyHoodieParquetFileFormat(private val shouldAppendPartitionValu
val datetimeRebaseModeInRead = parquetOptions.datetimeRebaseModeInRead
val int96RebaseModeInRead = parquetOptions.int96RebaseModeInRead
val timeZoneId = Option(sqlConf.sessionLocalTimeZone)
- // Should always be set by FileSourceScanExec creating this.
- // Check conf before checking option, to allow working around an issue by changing conf.
+ // Respect the plan-time OPTION_RETURNING_BATCH decision when present, instead of recomputing it here.
val returningBatch = sparkSession.sessionState.conf.parquetVectorizedReaderEnabled &&
Review Comment:
we are not changing existing check.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): honor plan-time OPTION_RETURNING_BATCH in legacy parquet … [hudi]
Posted by "hudi-agent (via GitHub)" <gi...@apache.org>.
hudi-agent commented on code in PR #19822:
URL: https://github.com/apache/hudi/pull/19822#discussion_r4079915758
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestLegacyParquetReadPath.scala:
##########
@@ -301,6 +306,39 @@ class TestLegacyParquetReadPath extends HoodieSparkClientTestBase with ScalaAsse
}
}
+ @Test
+ def testBroadcastJoinHonorsPlanTimeBatchingDecision(): Unit = {
Review Comment:
🤖 Does this test actually fail without the fix on the default (Spark 3.5) profile? Spark35/34's `getReturningBatch` already routes through `supportsColumnar`, which checks `conf.wholeStageEnabled`, so the pre-fix code should also return false here — only Spark 3.3's override (`supportBatch`, which in 3.3 is just `isBatchReadSupportedForSchema`) has the gap. If so, the regression would only be guarded on the spark3.3 CI profile.
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>
##########
hudi-spark-datasource/hudi-spark3-common/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/Spark3LegacyHoodieParquetFileFormat.scala:
##########
@@ -171,7 +171,11 @@ abstract class Spark3LegacyHoodieParquetFileFormat(shouldAppendPartitionValues:
val int96RebaseModeInRead = parquetOptions.int96RebaseModeInRead
val timeZoneId = Option(sqlConf.sessionLocalTimeZone)
// Whole stage codegen (PhysicalRDD) is able to deal with batches directly.
- val returningBatch = getReturningBatch(sparkSession, resultSchema)
+ // Respect the plan-time OPTION_RETURNING_BATCH decision when present, instead of recomputing it here.
+ val returningBatch = enableVectorizedReader &&
Review Comment:
🤖 The Spark 4.0/4.1/4.2 legacy formats are standalone copies that still recompute `returningBatch` from `supportsColumnar(sparkSession, resultSchema)` and ignore the option. They're less exposed (their `supportsColumnar` includes the whole-stage check), but `isTooManyFields` there is evaluated over `partitionSchema ++ requiredSchema` rather than the plan's output schema, so they can still disagree with the plan. Worth applying the same fix there?
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): honor plan-time OPTION_RETURNING_BATCH in legacy parquet … [hudi]
Posted by "hudi-agent (via GitHub)" <gi...@apache.org>.
hudi-agent commented on code in PR #19822:
URL: https://github.com/apache/hudi/pull/19822#discussion_r3913108689
##########
hudi-spark-datasource/hudi-spark3-common/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/Spark3LegacyHoodieParquetFileFormat.scala:
##########
@@ -171,7 +171,11 @@ abstract class Spark3LegacyHoodieParquetFileFormat(shouldAppendPartitionValues:
val int96RebaseModeInRead = parquetOptions.int96RebaseModeInRead
val timeZoneId = Option(sqlConf.sessionLocalTimeZone)
// Whole stage codegen (PhysicalRDD) is able to deal with batches directly.
- val returningBatch = getReturningBatch(sparkSession, resultSchema)
+ // Respect the plan-time OPTION_RETURNING_BATCH decision when present, instead of recomputing it here.
+ val returningBatch = enableVectorizedReader &&
+ options.get(FileFormat.OPTION_RETURNING_BATCH)
+ .map(_.equals("true"))
Review Comment:
🤖 nit: `_.equals("true")` is Java-style — in Scala `_ == "true"` (or `_.toBoolean`) is the idiomatic equivalent here.
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): honor plan-time OPTION_RETURNING_BATCH in legacy parquet … [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19822:
URL: https://github.com/apache/hudi/pull/19822#issuecomment-5794470196
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0dbd1e9d2f8b0e4860a3633d2093a5cf9d07e79d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16770",
"triggerID" : "0dbd1e9d2f8b0e4860a3633d2093a5cf9d07e79d",
"triggerType" : "PUSH"
}, {
"hash" : "f8c81acdae41941880e2a3c266eed2093776ff5a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16771",
"triggerID" : "f8c81acdae41941880e2a3c266eed2093776ff5a",
"triggerType" : "PUSH"
}, {
"hash" : "87b0d2b9f47b2d7c3400b49db324157fb1716e69",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17250",
"triggerID" : "87b0d2b9f47b2d7c3400b49db324157fb1716e69",
"triggerType" : "PUSH"
}, {
"hash" : "f2b51c7b7bca72e087b0fbc3b8f375e0fe3b8dff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17252",
"triggerID" : "f2b51c7b7bca72e087b0fbc3b8f375e0fe3b8dff",
"triggerType" : "PUSH"
}, {
"hash" : "3a2e0e8ce77375641286cf121687c7a1ab51853d",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17255",
"triggerID" : "3a2e0e8ce77375641286cf121687c7a1ab51853d",
"triggerType" : "PUSH"
}, {
"hash" : "0733587bcfa350068eef96fc513efd07bed39382",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17259",
"triggerID" : "0733587bcfa350068eef96fc513efd07bed39382",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3a2e0e8ce77375641286cf121687c7a1ab51853d Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17255)
* 0733587bcfa350068eef96fc513efd07bed39382 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17259)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): honor plan-time OPTION_RETURNING_BATCH in legacy parquet … [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19822:
URL: https://github.com/apache/hudi/pull/19822#issuecomment-5793880102
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0dbd1e9d2f8b0e4860a3633d2093a5cf9d07e79d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16770",
"triggerID" : "0dbd1e9d2f8b0e4860a3633d2093a5cf9d07e79d",
"triggerType" : "PUSH"
}, {
"hash" : "f8c81acdae41941880e2a3c266eed2093776ff5a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16771",
"triggerID" : "f8c81acdae41941880e2a3c266eed2093776ff5a",
"triggerType" : "PUSH"
}, {
"hash" : "87b0d2b9f47b2d7c3400b49db324157fb1716e69",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17250",
"triggerID" : "87b0d2b9f47b2d7c3400b49db324157fb1716e69",
"triggerType" : "PUSH"
}, {
"hash" : "f2b51c7b7bca72e087b0fbc3b8f375e0fe3b8dff",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17252",
"triggerID" : "f2b51c7b7bca72e087b0fbc3b8f375e0fe3b8dff",
"triggerType" : "PUSH"
}, {
"hash" : "3a2e0e8ce77375641286cf121687c7a1ab51853d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17255",
"triggerID" : "3a2e0e8ce77375641286cf121687c7a1ab51853d",
"triggerType" : "PUSH"
}, {
"hash" : "0733587bcfa350068eef96fc513efd07bed39382",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17259",
"triggerID" : "0733587bcfa350068eef96fc513efd07bed39382",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f2b51c7b7bca72e087b0fbc3b8f375e0fe3b8dff Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17252)
* 3a2e0e8ce77375641286cf121687c7a1ab51853d Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17255)
* 0733587bcfa350068eef96fc513efd07bed39382 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17259)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): honor plan-time OPTION_RETURNING_BATCH in legacy parquet … [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19822:
URL: https://github.com/apache/hudi/pull/19822#issuecomment-5793056919
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0dbd1e9d2f8b0e4860a3633d2093a5cf9d07e79d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16770",
"triggerID" : "0dbd1e9d2f8b0e4860a3633d2093a5cf9d07e79d",
"triggerType" : "PUSH"
}, {
"hash" : "f8c81acdae41941880e2a3c266eed2093776ff5a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16771",
"triggerID" : "f8c81acdae41941880e2a3c266eed2093776ff5a",
"triggerType" : "PUSH"
}, {
"hash" : "87b0d2b9f47b2d7c3400b49db324157fb1716e69",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17250",
"triggerID" : "87b0d2b9f47b2d7c3400b49db324157fb1716e69",
"triggerType" : "PUSH"
}, {
"hash" : "f2b51c7b7bca72e087b0fbc3b8f375e0fe3b8dff",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17252",
"triggerID" : "f2b51c7b7bca72e087b0fbc3b8f375e0fe3b8dff",
"triggerType" : "PUSH"
}, {
"hash" : "3a2e0e8ce77375641286cf121687c7a1ab51853d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17255",
"triggerID" : "3a2e0e8ce77375641286cf121687c7a1ab51853d",
"triggerType" : "PUSH"
}, {
"hash" : "0733587bcfa350068eef96fc513efd07bed39382",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0733587bcfa350068eef96fc513efd07bed39382",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f2b51c7b7bca72e087b0fbc3b8f375e0fe3b8dff Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17252)
* 3a2e0e8ce77375641286cf121687c7a1ab51853d Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17255)
* 0733587bcfa350068eef96fc513efd07bed39382 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): honor plan-time OPTION_RETURNING_BATCH in legacy parquet … [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #19822:
URL: https://github.com/apache/hudi/pull/19822#issuecomment-5586395359
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/19822?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 78.30%. Comparing base ([`a2788ee`](https://app.codecov.io/gh/apache/hudi/commit/a2788eef922e2375432ead3fe50af7fe5f55faf7?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`f8c81ac`](https://app.codecov.io/gh/apache/hudi/commit/f8c81acdae41941880e2a3c266eed2093776ff5a?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 25 commits behind head on master.
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #19822 +/- ##
============================================
- Coverage 78.31% 78.30% -0.01%
- Complexity 33885 33887 +2
============================================
Files 2541 2541
Lines 141728 141859 +131
Branches 17182 17182
============================================
+ Hits 110988 111078 +90
- Misses 23032 23071 +39
- Partials 7708 7710 +2
```
| [Components](https://app.codecov.io/gh/apache/hudi/pull/19822/components?src=pr&el=components&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [hudi-common](https://app.codecov.io/gh/apache/hudi/pull/19822/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `83.58% <ø> (-0.07%)` | :arrow_down: |
| [hudi-client](https://app.codecov.io/gh/apache/hudi/pull/19822/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `83.21% <ø> (ø)` | |
| [hudi-flink](https://app.codecov.io/gh/apache/hudi/pull/19822/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `85.65% <ø> (ø)` | |
| [hudi-spark-datasource](https://app.codecov.io/gh/apache/hudi/pull/19822/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `73.22% <ø> (+<0.01%)` | :arrow_up: |
| [hudi-utilities](https://app.codecov.io/gh/apache/hudi/pull/19822/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `74.55% <ø> (ø)` | |
| [hudi-cli](https://app.codecov.io/gh/apache/hudi/pull/19822/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `15.06% <ø> (ø)` | |
| [hudi-hadoop](https://app.codecov.io/gh/apache/hudi/pull/19822/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `70.72% <ø> (ø)` | |
| [hudi-sync](https://app.codecov.io/gh/apache/hudi/pull/19822/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `75.54% <ø> (ø)` | |
| [hudi-io](https://app.codecov.io/gh/apache/hudi/pull/19822/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `80.02% <ø> (+0.04%)` | :arrow_up: |
| [hudi-timeline-service](https://app.codecov.io/gh/apache/hudi/pull/19822/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `83.44% <ø> (ø)` | |
| [hudi-cloud](https://app.codecov.io/gh/apache/hudi/pull/19822/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `65.81% <ø> (ø)` | |
| [hudi-kafka-connect](https://app.codecov.io/gh/apache/hudi/pull/19822/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `53.20% <ø> (ø)` | |
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/19822/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/19822/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.09% <ø> (+<0.01%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/19822?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [.../parquet/Spark3LegacyHoodieParquetFileFormat.scala](https://app.codecov.io/gh/apache/hudi/pull/19822?src=pr&el=tree&filepath=hudi-spark-datasource%2Fhudi-spark3-common%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Fspark%2Fsql%2Fexecution%2Fdatasources%2Fparquet%2FSpark3LegacyHoodieParquetFileFormat.scala&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zcGFyay1kYXRhc291cmNlL2h1ZGktc3BhcmszLWNvbW1vbi9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL3NwYXJrL3NxbC9leGVjdXRpb24vZGF0YXNvdXJjZXMvcGFycXVldC9TcGFyazNMZWdhY3lIb29kaWVQYXJxdWV0RmlsZUZvcm1hdC5zY2FsYQ==) | `0.00% <ø> (ø)` | |
... and [5 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/19822/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): honor plan-time OPTION_RETURNING_BATCH in legacy parquet … [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19822:
URL: https://github.com/apache/hudi/pull/19822#issuecomment-5507915340
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0dbd1e9d2f8b0e4860a3633d2093a5cf9d07e79d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0dbd1e9d2f8b0e4860a3633d2093a5cf9d07e79d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0dbd1e9d2f8b0e4860a3633d2093a5cf9d07e79d UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): honor plan-time OPTION_RETURNING_BATCH in legacy parquet … [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19822:
URL: https://github.com/apache/hudi/pull/19822#issuecomment-5507989018
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0dbd1e9d2f8b0e4860a3633d2093a5cf9d07e79d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16770",
"triggerID" : "0dbd1e9d2f8b0e4860a3633d2093a5cf9d07e79d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0dbd1e9d2f8b0e4860a3633d2093a5cf9d07e79d Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16770)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): honor plan-time OPTION_RETURNING_BATCH in legacy parquet … [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19822:
URL: https://github.com/apache/hudi/pull/19822#issuecomment-5508052116
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0dbd1e9d2f8b0e4860a3633d2093a5cf9d07e79d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16770",
"triggerID" : "0dbd1e9d2f8b0e4860a3633d2093a5cf9d07e79d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0dbd1e9d2f8b0e4860a3633d2093a5cf9d07e79d Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16770)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): honor plan-time OPTION_RETURNING_BATCH in legacy parquet … [hudi]
Posted by "hudi-agent (via GitHub)" <gi...@apache.org>.
hudi-agent commented on code in PR #19822:
URL: https://github.com/apache/hudi/pull/19822#discussion_r4080876428
##########
hudi-spark-datasource/hudi-spark4.0.x/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/Spark40LegacyHoodieParquetFileFormat.scala:
##########
@@ -152,10 +152,11 @@ class Spark40LegacyHoodieParquetFileFormat(private val shouldAppendPartitionValu
val datetimeRebaseModeInRead = parquetOptions.datetimeRebaseModeInRead
val int96RebaseModeInRead = parquetOptions.int96RebaseModeInRead
val timeZoneId = Option(sqlConf.sessionLocalTimeZone)
- // Should always be set by FileSourceScanExec creating this.
- // Check conf before checking option, to allow working around an issue by changing conf.
+ // Respect the plan-time OPTION_RETURNING_BATCH decision when present, instead of recomputing it here.
val returningBatch = sparkSession.sessionState.conf.parquetVectorizedReaderEnabled &&
Review Comment:
🤖 The spark3-common version gates this on `enableVectorizedReader &&`, but here it's only `parquetVectorizedReaderEnabled &&`. Previously the `supportsColumnar(resultSchema)` fallback implied `supportBatch(resultSchema)` (== `enableVectorizedReader`), so the guard was implicit; with the option path it isn't. If the plan says `true` but `enableVectorizedReader` is false (resultSchema = partitionSchema ++ requiredSchema is a superset of the plan's output schema), the reader takes the row branch while the plan expects columnar. Could you mirror the `enableVectorizedReader &&` guard here (and in 4.1/4.2)?
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): honor plan-time OPTION_RETURNING_BATCH in legacy parquet … [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19822:
URL: https://github.com/apache/hudi/pull/19822#issuecomment-5792797407
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0dbd1e9d2f8b0e4860a3633d2093a5cf9d07e79d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16770",
"triggerID" : "0dbd1e9d2f8b0e4860a3633d2093a5cf9d07e79d",
"triggerType" : "PUSH"
}, {
"hash" : "f8c81acdae41941880e2a3c266eed2093776ff5a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16771",
"triggerID" : "f8c81acdae41941880e2a3c266eed2093776ff5a",
"triggerType" : "PUSH"
}, {
"hash" : "87b0d2b9f47b2d7c3400b49db324157fb1716e69",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17250",
"triggerID" : "87b0d2b9f47b2d7c3400b49db324157fb1716e69",
"triggerType" : "PUSH"
}, {
"hash" : "f2b51c7b7bca72e087b0fbc3b8f375e0fe3b8dff",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17252",
"triggerID" : "f2b51c7b7bca72e087b0fbc3b8f375e0fe3b8dff",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f2b51c7b7bca72e087b0fbc3b8f375e0fe3b8dff Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17252)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): honor plan-time OPTION_RETURNING_BATCH in legacy parquet … [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope merged PR #19822:
URL: https://github.com/apache/hudi/pull/19822
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org