You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "jonvex (via GitHub)" <gi...@apache.org> on 2025/07/22 16:33:16 UTC
[PR] [HUDI-9621 ] Fix divergence between file index and incremental file index [hudi]
jonvex opened a new pull request, #13593:
URL: https://github.com/apache/hudi/pull/13593
### Change Logs
There was duplicate code and then some of it was changed in:
https://issues.apache.org/jira/browse/HUDI-9205
https://issues.apache.org/jira/browse/HUDI-9451
causing a divergence. This pr breaks out the common logic so that changes will be used by both branches now.
### Impact
benefits of HUDI-9205 and HUDI-9451 apply to incremental queries and prevent further divergence
### Risk level (write none, low medium or high below)
low
### Documentation Update
N/A
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9621] Fix divergence between file index and incremental file index [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13593:
URL: https://github.com/apache/hudi/pull/13593#discussion_r2224109944
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieFileIndex.scala:
##########
@@ -189,14 +196,11 @@ case class HoodieFileIndex(spark: SparkSession,
val files = logPathInfoStream.collect(Collectors.toList[StoragePathInfo]).asScala
baseFileStatusOpt.foreach(f => files.append(f))
val allCandidateFiles = files.map(fileInfo => new FileStatus(fileInfo.getLength, fileInfo.isDirectory, 0, fileInfo.getBlockSize,
- fileInfo.getModificationTime, new Path(fileInfo.getPath.toUri))).toSeq
- sparkAdapter.getSparkPartitionedFileUtils.newPartitionDirectory(
- InternalRow.fromSeq(partitionOpt.get.values), allCandidateFiles)
+ fileInfo.getModificationTime, new Path(fileInfo.getPath.toUri))).toSeq
+ sparkAdapter.getSparkPartitionedFileUtils.newPartitionDirectory(partitionValues, allCandidateFiles)
}
}
- hasPushedDownPartitionPredicates = true
Review Comment:
Was this a bug previously?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9621] Fix divergence between file index and incremental file index [hudi]
Posted by "TheR1sing3un (via GitHub)" <gi...@apache.org>.
TheR1sing3un commented on PR #13593:
URL: https://github.com/apache/hudi/pull/13593#issuecomment-3105490665
Overall, it seems reasonable to extract the common methods for better code maintainability. Sorry that the common logic of the `HoodieFileIndex` subclass was not taken into consideration when making the modification before :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9621] Fix divergence between file index and incremental file index [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on PR #13593:
URL: https://github.com/apache/hudi/pull/13593#issuecomment-3105353375
cc @TheR1sing3un can you review this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9621] Fix divergence between file index and incremental file index [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on PR #13593:
URL: https://github.com/apache/hudi/pull/13593#issuecomment-3111750846
> @danny0405 @KnightChess @TheR1sing3un @the-other-tim-brown This pr from last year changed the behavior: #11569.
>
> Here is the original pr adding this: #7423
>
> So should we remove that line from this method then?
Did you mean this line `hasPushedDownPartitionPredicates = true` ? I'm not very sure about this part, from the code it is still used within the `HoodiePruneFileSourcePartitions`. We can land this first and address it in another PR I think.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9621] Fix divergence between file index and incremental file index [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 merged PR #13593:
URL: https://github.com/apache/hudi/pull/13593
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9621] Fix divergence between file index and incremental file index [hudi]
Posted by "TheR1sing3un (via GitHub)" <gi...@apache.org>.
TheR1sing3un commented on code in PR #13593:
URL: https://github.com/apache/hudi/pull/13593#discussion_r2224199598
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieFileIndex.scala:
##########
@@ -172,15 +172,22 @@ case class HoodieFileIndex(spark: SparkSession,
* @return list of PartitionDirectory containing partition to base files mapping
*/
override def listFiles(partitionFilters: Seq[Expression], dataFilters: Seq[Expression]): Seq[PartitionDirectory] = {
- val prunedPartitionsAndFilteredFileSlices = filterFileSlices(dataFilters, partitionFilters).flatMap(
+ val slices = filterFileSlices(dataFilters, partitionFilters).flatMap(
{ case (partitionOpt, fileSlices) =>
- fileSlices.filter(!_.isEmpty).map(fs => (partitionOpt, fs))
+ fileSlices.filter(!_.isEmpty).map(fs => ( InternalRow.fromSeq(partitionOpt.get.values), fs))
}
- ).map {
- case (partitionOpt, fileSlice) =>
+ )
+ prepareFileSlices(slices)
+ }
+
+ protected def prepareFileSlices(slices: Seq[(InternalRow, FileSlice)]): Seq[PartitionDirectory] = {
+ hasPushedDownPartitionPredicates = true
Review Comment:
`hasPushedDownPartitionPredicates` should mainly be to indicate that a 'listFiles' operation has been performed. It seems more reasonable to place it in the' listFiles' method. 'prepareFileSlices' should just be a tool method
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9621 ] Fix divergence between file index and incremental file index [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13593:
URL: https://github.com/apache/hudi/pull/13593#issuecomment-3103747912
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7aa5701d632eb82492620697d36836e48f001c54",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6961",
"triggerID" : "7aa5701d632eb82492620697d36836e48f001c54",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7aa5701d632eb82492620697d36836e48f001c54 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6961)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9621 ] Fix divergence between file index and incremental file index [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13593:
URL: https://github.com/apache/hudi/pull/13593#issuecomment-3103736478
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7aa5701d632eb82492620697d36836e48f001c54",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7aa5701d632eb82492620697d36836e48f001c54",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7aa5701d632eb82492620697d36836e48f001c54 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9621] Fix divergence between file index and incremental file index [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13593:
URL: https://github.com/apache/hudi/pull/13593#issuecomment-3104465857
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7aa5701d632eb82492620697d36836e48f001c54",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6961",
"triggerID" : "7aa5701d632eb82492620697d36836e48f001c54",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7aa5701d632eb82492620697d36836e48f001c54 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6961)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9621] Fix divergence between file index and incremental file index [hudi]
Posted by "jonvex (via GitHub)" <gi...@apache.org>.
jonvex commented on PR #13593:
URL: https://github.com/apache/hudi/pull/13593#issuecomment-3108965988
@danny0405 @KnightChess @TheR1sing3un @the-other-tim-brown This pr from last year changed the behavior: https://github.com/apache/hudi/pull/11569.
Here is the original pr adding this: https://github.com/apache/hudi/pull/7423
So should we remove that line from this method then?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org