You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "linliu-code (via GitHub)" <gi...@apache.org> on 2025/11/08 23:21:31 UTC
[PR] fix: Disable positional merging for spark version < 3.5 [hudi]
linliu-code opened a new pull request, #14241:
URL: https://github.com/apache/hudi/pull/14241
### Describe the issue this Pull Request addresses
Some tests failed for spark3.4 and spark3.3, where the positional merging is enabled.
### Summary and Changelog
Since there is no meaningful row index information in parquet files for spark3.3 and spark3.4, we disable positional merging to avoid test failures.
### Impact
Less flaky tests.
### Risk Level
None.
### Documentation Update
<!-- Describe any necessary documentation update if there is any new feature, config, or user-facing change. If not, put "none".
- The config description must be updated if new configs are added or the default value of the configs are changed.
- Any new feature or user-facing change requires updating the Hudi website. Please follow the
[instruction](https://hudi.apache.org/contribute/developer-setup#website) to make changes to the website. -->
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Enough context is provided in the sections above
- [ ] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Disable positional merging for spark version < 3.5 [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua merged PR #14241:
URL: https://github.com/apache/hudi/pull/14241
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Disable positional merging for spark version < 3.5 [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14241:
URL: https://github.com/apache/hudi/pull/14241#issuecomment-3507145357
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "40ea322e68e45e92dc3713712dcab01cd9437980",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9548",
"triggerID" : "40ea322e68e45e92dc3713712dcab01cd9437980",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 40ea322e68e45e92dc3713712dcab01cd9437980 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9548)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Disable positional merging for spark version < 3.5 [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14241:
URL: https://github.com/apache/hudi/pull/14241#issuecomment-3507269803
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "40ea322e68e45e92dc3713712dcab01cd9437980",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9548",
"triggerID" : "40ea322e68e45e92dc3713712dcab01cd9437980",
"triggerType" : "PUSH"
}, {
"hash" : "7ab0b33b9a3bc1f8390d266a1ebc795d8a33db08",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9550",
"triggerID" : "7ab0b33b9a3bc1f8390d266a1ebc795d8a33db08",
"triggerType" : "PUSH"
}, {
"hash" : "d28b9a5692e9e45c3e8a022ccf89578086eb8056",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9552",
"triggerID" : "d28b9a5692e9e45c3e8a022ccf89578086eb8056",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 40ea322e68e45e92dc3713712dcab01cd9437980 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9548)
* 7ab0b33b9a3bc1f8390d266a1ebc795d8a33db08 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9550)
* d28b9a5692e9e45c3e8a022ccf89578086eb8056 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9552)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Disable positional merging for spark version < 3.5 [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14241:
URL: https://github.com/apache/hudi/pull/14241#issuecomment-3507264830
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "40ea322e68e45e92dc3713712dcab01cd9437980",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9548",
"triggerID" : "40ea322e68e45e92dc3713712dcab01cd9437980",
"triggerType" : "PUSH"
}, {
"hash" : "7ab0b33b9a3bc1f8390d266a1ebc795d8a33db08",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9550",
"triggerID" : "7ab0b33b9a3bc1f8390d266a1ebc795d8a33db08",
"triggerType" : "PUSH"
}, {
"hash" : "d28b9a5692e9e45c3e8a022ccf89578086eb8056",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d28b9a5692e9e45c3e8a022ccf89578086eb8056",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 40ea322e68e45e92dc3713712dcab01cd9437980 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9548)
* 7ab0b33b9a3bc1f8390d266a1ebc795d8a33db08 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9550)
* d28b9a5692e9e45c3e8a022ccf89578086eb8056 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Disable positional merging for spark version < 3.5 [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14241:
URL: https://github.com/apache/hudi/pull/14241#issuecomment-3507660095
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "40ea322e68e45e92dc3713712dcab01cd9437980",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9548",
"triggerID" : "40ea322e68e45e92dc3713712dcab01cd9437980",
"triggerType" : "PUSH"
}, {
"hash" : "7ab0b33b9a3bc1f8390d266a1ebc795d8a33db08",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9550",
"triggerID" : "7ab0b33b9a3bc1f8390d266a1ebc795d8a33db08",
"triggerType" : "PUSH"
}, {
"hash" : "d28b9a5692e9e45c3e8a022ccf89578086eb8056",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9552",
"triggerID" : "d28b9a5692e9e45c3e8a022ccf89578086eb8056",
"triggerType" : "PUSH"
}, {
"hash" : "c2bf1b4947a03ca7cf4df5c5a0eda298207e090f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c2bf1b4947a03ca7cf4df5c5a0eda298207e090f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d28b9a5692e9e45c3e8a022ccf89578086eb8056 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9552)
* c2bf1b4947a03ca7cf4df5c5a0eda298207e090f UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Disable positional merging for spark version < 3.5 [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14241:
URL: https://github.com/apache/hudi/pull/14241#issuecomment-3507472985
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "40ea322e68e45e92dc3713712dcab01cd9437980",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9548",
"triggerID" : "40ea322e68e45e92dc3713712dcab01cd9437980",
"triggerType" : "PUSH"
}, {
"hash" : "7ab0b33b9a3bc1f8390d266a1ebc795d8a33db08",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9550",
"triggerID" : "7ab0b33b9a3bc1f8390d266a1ebc795d8a33db08",
"triggerType" : "PUSH"
}, {
"hash" : "d28b9a5692e9e45c3e8a022ccf89578086eb8056",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9552",
"triggerID" : "d28b9a5692e9e45c3e8a022ccf89578086eb8056",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d28b9a5692e9e45c3e8a022ccf89578086eb8056 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9552)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Disable positional merging for spark version < 3.5 [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #14241:
URL: https://github.com/apache/hudi/pull/14241#discussion_r2507398065
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieHadoopFsRelationFactory.scala:
##########
@@ -205,7 +205,7 @@ abstract class HoodieBaseHadoopFsRelationFactory(val sqlContext: SQLContext,
shouldOmitPartitionColumns || shouldExtractPartitionValueFromPath || isBootstrap
}
- private lazy val shouldUseRecordPosition: Boolean = checkIfAConfigurationEnabled(HoodieReaderConfig.MERGE_USE_RECORD_POSITIONS)
+ private lazy val shouldUseRecordPosition: Boolean = checkIfPositionalMergingEnabled(HoodieReaderConfig.MERGE_USE_RECORD_POSITIONS)
Review Comment:
re-enabled.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Disable positional merging for spark version < 3.5 [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14241:
URL: https://github.com/apache/hudi/pull/14241#issuecomment-3507146865
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "40ea322e68e45e92dc3713712dcab01cd9437980",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9548",
"triggerID" : "40ea322e68e45e92dc3713712dcab01cd9437980",
"triggerType" : "PUSH"
}, {
"hash" : "7ab0b33b9a3bc1f8390d266a1ebc795d8a33db08",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7ab0b33b9a3bc1f8390d266a1ebc795d8a33db08",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 40ea322e68e45e92dc3713712dcab01cd9437980 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9548)
* 7ab0b33b9a3bc1f8390d266a1ebc795d8a33db08 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Disable positional merging for spark version < 3.5 [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #14241:
URL: https://github.com/apache/hudi/pull/14241#discussion_r2507377076
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieHadoopFsRelationFactory.scala:
##########
@@ -205,7 +205,7 @@ abstract class HoodieBaseHadoopFsRelationFactory(val sqlContext: SQLContext,
shouldOmitPartitionColumns || shouldExtractPartitionValueFromPath || isBootstrap
}
- private lazy val shouldUseRecordPosition: Boolean = checkIfAConfigurationEnabled(HoodieReaderConfig.MERGE_USE_RECORD_POSITIONS)
+ private lazy val shouldUseRecordPosition: Boolean = checkIfPositionalMergingEnabled(HoodieReaderConfig.MERGE_USE_RECORD_POSITIONS)
Review Comment:
Let's re-enable `TestFiltersInFileGroupReader#testFiltersInFileFormat` (currently disabled on master).
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieHadoopFsRelationFactory.scala:
##########
@@ -214,10 +214,14 @@ abstract class HoodieBaseHadoopFsRelationFactory(val sqlContext: SQLContext,
// NOTE: We're including compaction here since it's not considering a "commit" operation
metaClient.getCommitsAndCompactionTimeline.filterCompletedInstants
- private def checkIfAConfigurationEnabled(config: ConfigProperty[java.lang.Boolean],
- defaultValueOption: Option[String] = Option.empty): Boolean = {
- optParams.getOrElse(config.key(),
- sqlContext.getConf(config.key(), defaultValueOption.getOrElse(String.valueOf(config.defaultValue())))).toBoolean
+ private def checkIfPositionalMergingEnabled(config: ConfigProperty[java.lang.Boolean],
+ defaultValueOption: Option[String] = Option.empty): Boolean = {
+ if (!HoodieSparkUtils.gteqSpark3_5) {
Review Comment:
I see that `PositionBasedFileGroupRecordBuffer#hasNextBaseRecord` handles the case for Spark 3.4 and below if the record position is not returned from the parquet reader. Is there a bug inside this logic that the position is wrong in the relevant logic, causing the incorrect data from table read?
```
@Override
protected boolean hasNextBaseRecord(T baseRecord) throws IOException {
if (!readerContext.getShouldMergeUseRecordPosition()) {
return doHasNextFallbackBaseRecord(baseRecord);
}
nextRecordPosition = readerContext.getRecordContext().extractRecordPosition(baseRecord, readerSchema,
ROW_INDEX_TEMPORARY_COLUMN_NAME, nextRecordPosition);
BufferedRecord<T> logRecordInfo = records.remove(nextRecordPosition++);
return super.hasNextBaseRecord(baseRecord, logRecordInfo);
}
/**
* Extracts the record position value from the record itself.
*
* @return the record position in the base file.
*/
public long extractRecordPosition(T record, Schema schema, String fieldName, long providedPositionIfNeeded) {
if (supportsParquetRowIndex()) {
Object position = getValue(record, schema, fieldName);
if (position != null) {
return (long) position;
} else {
throw new IllegalStateException("Record position extraction failed");
}
}
return providedPositionIfNeeded;
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Disable positional merging for spark version < 3.5 [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14241:
URL: https://github.com/apache/hudi/pull/14241#issuecomment-3507260177
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "40ea322e68e45e92dc3713712dcab01cd9437980",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9548",
"triggerID" : "40ea322e68e45e92dc3713712dcab01cd9437980",
"triggerType" : "PUSH"
}, {
"hash" : "7ab0b33b9a3bc1f8390d266a1ebc795d8a33db08",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9550",
"triggerID" : "7ab0b33b9a3bc1f8390d266a1ebc795d8a33db08",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 40ea322e68e45e92dc3713712dcab01cd9437980 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9548)
* 7ab0b33b9a3bc1f8390d266a1ebc795d8a33db08 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9550)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Disable positional merging for spark version < 3.5 [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14241:
URL: https://github.com/apache/hudi/pull/14241#issuecomment-3507233589
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "40ea322e68e45e92dc3713712dcab01cd9437980",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9548",
"triggerID" : "40ea322e68e45e92dc3713712dcab01cd9437980",
"triggerType" : "PUSH"
}, {
"hash" : "7ab0b33b9a3bc1f8390d266a1ebc795d8a33db08",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9550",
"triggerID" : "7ab0b33b9a3bc1f8390d266a1ebc795d8a33db08",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 40ea322e68e45e92dc3713712dcab01cd9437980 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9548)
* 7ab0b33b9a3bc1f8390d266a1ebc795d8a33db08 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9550)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Disable positional merging for spark version < 3.5 [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14241:
URL: https://github.com/apache/hudi/pull/14241#issuecomment-3507661327
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "40ea322e68e45e92dc3713712dcab01cd9437980",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9548",
"triggerID" : "40ea322e68e45e92dc3713712dcab01cd9437980",
"triggerType" : "PUSH"
}, {
"hash" : "7ab0b33b9a3bc1f8390d266a1ebc795d8a33db08",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9550",
"triggerID" : "7ab0b33b9a3bc1f8390d266a1ebc795d8a33db08",
"triggerType" : "PUSH"
}, {
"hash" : "d28b9a5692e9e45c3e8a022ccf89578086eb8056",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9552",
"triggerID" : "d28b9a5692e9e45c3e8a022ccf89578086eb8056",
"triggerType" : "PUSH"
}, {
"hash" : "c2bf1b4947a03ca7cf4df5c5a0eda298207e090f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9555",
"triggerID" : "c2bf1b4947a03ca7cf4df5c5a0eda298207e090f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d28b9a5692e9e45c3e8a022ccf89578086eb8056 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9552)
* c2bf1b4947a03ca7cf4df5c5a0eda298207e090f Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9555)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Disable positional merging for spark version < 3.5 [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #14241:
URL: https://github.com/apache/hudi/pull/14241#discussion_r2507384903
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieHadoopFsRelationFactory.scala:
##########
@@ -214,10 +214,14 @@ abstract class HoodieBaseHadoopFsRelationFactory(val sqlContext: SQLContext,
// NOTE: We're including compaction here since it's not considering a "commit" operation
metaClient.getCommitsAndCompactionTimeline.filterCompletedInstants
- private def checkIfAConfigurationEnabled(config: ConfigProperty[java.lang.Boolean],
- defaultValueOption: Option[String] = Option.empty): Boolean = {
- optParams.getOrElse(config.key(),
- sqlContext.getConf(config.key(), defaultValueOption.getOrElse(String.valueOf(config.defaultValue())))).toBoolean
+ private def checkIfPositionalMergingEnabled(config: ConfigProperty[java.lang.Boolean],
+ defaultValueOption: Option[String] = Option.empty): Boolean = {
+ if (!HoodieSparkUtils.gteqSpark3_5) {
Review Comment:
Yeah, ideally we should root cause it since we have the fallback when position is not available. Then I can debug instead of disabling it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Disable positional merging for spark version < 3.5 [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14241:
URL: https://github.com/apache/hudi/pull/14241#issuecomment-3507144071
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "40ea322e68e45e92dc3713712dcab01cd9437980",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "40ea322e68e45e92dc3713712dcab01cd9437980",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 40ea322e68e45e92dc3713712dcab01cd9437980 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Disable positional merging for spark version < 3.5 [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #14241:
URL: https://github.com/apache/hudi/pull/14241#discussion_r2507542927
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieHadoopFsRelationFactory.scala:
##########
@@ -205,7 +205,7 @@ abstract class HoodieBaseHadoopFsRelationFactory(val sqlContext: SQLContext,
shouldOmitPartitionColumns || shouldExtractPartitionValueFromPath || isBootstrap
}
- private lazy val shouldUseRecordPosition: Boolean = checkIfAConfigurationEnabled(HoodieReaderConfig.MERGE_USE_RECORD_POSITIONS)
+ private lazy val shouldUseRecordPosition: Boolean = checkIfPositionalMergingEnabled(HoodieReaderConfig.MERGE_USE_RECORD_POSITIONS)
Review Comment:
Ok, one thing that I am not sure is: for mergehandle, and appendhandle, for fg reader based merge handle, they checked the positional merge config. Should we add the same fallback there?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Disable positional merging for spark version < 3.5 [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14241:
URL: https://github.com/apache/hudi/pull/14241#issuecomment-3507857307
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "40ea322e68e45e92dc3713712dcab01cd9437980",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9548",
"triggerID" : "40ea322e68e45e92dc3713712dcab01cd9437980",
"triggerType" : "PUSH"
}, {
"hash" : "7ab0b33b9a3bc1f8390d266a1ebc795d8a33db08",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9550",
"triggerID" : "7ab0b33b9a3bc1f8390d266a1ebc795d8a33db08",
"triggerType" : "PUSH"
}, {
"hash" : "d28b9a5692e9e45c3e8a022ccf89578086eb8056",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9552",
"triggerID" : "d28b9a5692e9e45c3e8a022ccf89578086eb8056",
"triggerType" : "PUSH"
}, {
"hash" : "c2bf1b4947a03ca7cf4df5c5a0eda298207e090f",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9555",
"triggerID" : "c2bf1b4947a03ca7cf4df5c5a0eda298207e090f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c2bf1b4947a03ca7cf4df5c5a0eda298207e090f Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9555)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Disable positional merging for spark version < 3.5 [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14241:
URL: https://github.com/apache/hudi/pull/14241#issuecomment-3507292758
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "40ea322e68e45e92dc3713712dcab01cd9437980",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9548",
"triggerID" : "40ea322e68e45e92dc3713712dcab01cd9437980",
"triggerType" : "PUSH"
}, {
"hash" : "7ab0b33b9a3bc1f8390d266a1ebc795d8a33db08",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9550",
"triggerID" : "7ab0b33b9a3bc1f8390d266a1ebc795d8a33db08",
"triggerType" : "PUSH"
}, {
"hash" : "d28b9a5692e9e45c3e8a022ccf89578086eb8056",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9552",
"triggerID" : "d28b9a5692e9e45c3e8a022ccf89578086eb8056",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7ab0b33b9a3bc1f8390d266a1ebc795d8a33db08 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9550)
* d28b9a5692e9e45c3e8a022ccf89578086eb8056 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9552)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Disable positional merging for spark version < 3.5 [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #14241:
URL: https://github.com/apache/hudi/pull/14241#discussion_r2507387860
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieHadoopFsRelationFactory.scala:
##########
@@ -205,7 +205,7 @@ abstract class HoodieBaseHadoopFsRelationFactory(val sqlContext: SQLContext,
shouldOmitPartitionColumns || shouldExtractPartitionValueFromPath || isBootstrap
}
- private lazy val shouldUseRecordPosition: Boolean = checkIfAConfigurationEnabled(HoodieReaderConfig.MERGE_USE_RECORD_POSITIONS)
+ private lazy val shouldUseRecordPosition: Boolean = checkIfPositionalMergingEnabled(HoodieReaderConfig.MERGE_USE_RECORD_POSITIONS)
Review Comment:
Could you also check other places using `HoodieReaderConfig.MERGE_USE_RECORD_POSITIONS` to determine if positional merge should be enabled? We may need the same fallback at the top level.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Disable positional merging for spark version < 3.5 [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #14241:
URL: https://github.com/apache/hudi/pull/14241#discussion_r2507570957
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieHadoopFsRelationFactory.scala:
##########
@@ -205,7 +205,7 @@ abstract class HoodieBaseHadoopFsRelationFactory(val sqlContext: SQLContext,
shouldOmitPartitionColumns || shouldExtractPartitionValueFromPath || isBootstrap
}
- private lazy val shouldUseRecordPosition: Boolean = checkIfAConfigurationEnabled(HoodieReaderConfig.MERGE_USE_RECORD_POSITIONS)
+ private lazy val shouldUseRecordPosition: Boolean = checkIfPositionalMergingEnabled(HoodieReaderConfig.MERGE_USE_RECORD_POSITIONS)
Review Comment:
If the current approach of assigning record positions incrementally in `PositionBasedFileGroupRecordBuffer` has bugs, it affects write handle using the file group reader with positional merging enabled as well. To be safe, such logic should also fall back to use key-based record buffer directly, to avoid correctness issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org