You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "TheR1sing3un (via GitHub)" <gi...@apache.org> on 2025/05/22 07:08:43 UTC
[PR] [HUDI-9482] Fix the issue where the file slice was wrongly filtered out [hudi]
TheR1sing3un opened a new pull request, #13342:
URL: https://github.com/apache/hudi/pull/13342
Please consider following case:
```
/**
* case: query with pending instant
*
* 1. insert-txn1 start, will create a log file
* 2. insert-txn2 start, will create a log file
* 3. insert-txn2 commit
* 4. query-txn3 execute
* 5. insert-txn1 commit
*
* |-------------------------------- txn1: insert --------------------------------------|
* |------ txn2: insert ------|
* |---txn3: query---|
*
* we expected txn3's query should see the data of txn2
*/
```
### Change Logs
1. Fix the issue where the file slice was wrongly filtered out when there were committed and pending logs
_Describe context and summary for this change. Highlight if any code was copied._
### Impact
_Describe any public API or user-facing feature change or any performance impact._
### Risk level (write none, low medium or high below)
_If medium or high, explain what verification was done to mitigate the risks._
### Documentation Update
_Describe any necessary documentation update if there is any new feature, config, or user-facing change. If not, put "none"._
- _The config description must be updated if new configs are added or the default value of the configs are changed_
- _Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make
changes to the website._
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9482] Fix the issue where the file slice was wrongly filtered out [hudi]
Posted by "TheR1sing3un (via GitHub)" <gi...@apache.org>.
TheR1sing3un commented on PR #13342:
URL: https://github.com/apache/hudi/pull/13342#issuecomment-2933196159
@danny0405 Hi, Danny, this is a bug in data quality that has already occurred in our production environment. Let's solve this problem together!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9482] Fix the issue where the file slice was wrongly filtered out [hudi]
Posted by "TheR1sing3un (via GitHub)" <gi...@apache.org>.
TheR1sing3un commented on code in PR #13342:
URL: https://github.com/apache/hudi/pull/13342#discussion_r2102348540
##########
hudi-common/src/main/java/org/apache/hudi/common/model/HoodieFileGroup.java:
##########
@@ -119,6 +121,10 @@ public void addLogFile(CompletionTimeQueryView completionTimeQueryView, HoodieLo
@VisibleForTesting
public String getBaseInstantTime(CompletionTimeQueryView completionTimeQueryView, HoodieLogFile logFile) {
if (fileSlices.isEmpty()) {
+ if (completionTimeQueryView.getTableVersion().greaterThanOrEquals(HoodieTableVersion.EIGHT)) {
+ // no base file in the file group, use INITIAL INSTANT TIME as base instant time
+ return HoodieActiveTimeline.INIT_INSTANT_TS;
Review Comment:
> Looks like a version compatibility issue?
Indeed, it is. Essentially, it is because the log file name of the old version contained the base instant time of this slice, while the log file name of the new version only contains the instant written to this file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9482] Fix the issue where the file slice was wrongly filtered out [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13342:
URL: https://github.com/apache/hudi/pull/13342#issuecomment-2912367399
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0376c2488eae2662f5521185afb2d8634d0f0660",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5636",
"triggerID" : "0376c2488eae2662f5521185afb2d8634d0f0660",
"triggerType" : "PUSH"
}, {
"hash" : "bfcc764ea299b40a91a7a2864f2759d4a485663f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5640",
"triggerID" : "bfcc764ea299b40a91a7a2864f2759d4a485663f",
"triggerType" : "PUSH"
}, {
"hash" : "e47bf80290369887dde115148528392967ac600d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5673",
"triggerID" : "e47bf80290369887dde115148528392967ac600d",
"triggerType" : "PUSH"
}, {
"hash" : "b35bcc913b34302e56c59e0c20ecbf09fd406990",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5740",
"triggerID" : "b35bcc913b34302e56c59e0c20ecbf09fd406990",
"triggerType" : "PUSH"
}, {
"hash" : "d31cfdc44cfdcd9c0168ec79b1e16d487c285209",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5742",
"triggerID" : "d31cfdc44cfdcd9c0168ec79b1e16d487c285209",
"triggerType" : "PUSH"
}, {
"hash" : "d31cfdc44cfdcd9c0168ec79b1e16d487c285209",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5744",
"triggerID" : "2909795186",
"triggerType" : "MANUAL"
}, {
"hash" : "d31cfdc44cfdcd9c0168ec79b1e16d487c285209",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5770",
"triggerID" : "2912360271",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* d31cfdc44cfdcd9c0168ec79b1e16d487c285209 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5742) Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5744) Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5770)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9482] Fix the issue where the file slice was wrongly filtered out [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13342:
URL: https://github.com/apache/hudi/pull/13342#issuecomment-2903802689
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0376c2488eae2662f5521185afb2d8634d0f0660",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5636",
"triggerID" : "0376c2488eae2662f5521185afb2d8634d0f0660",
"triggerType" : "PUSH"
}, {
"hash" : "bfcc764ea299b40a91a7a2864f2759d4a485663f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5640",
"triggerID" : "bfcc764ea299b40a91a7a2864f2759d4a485663f",
"triggerType" : "PUSH"
}, {
"hash" : "e47bf80290369887dde115148528392967ac600d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5673",
"triggerID" : "e47bf80290369887dde115148528392967ac600d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* bfcc764ea299b40a91a7a2864f2759d4a485663f Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5640)
* e47bf80290369887dde115148528392967ac600d Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5673)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9482] Fix the issue where the file slice was wrongly filtered out [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13342:
URL: https://github.com/apache/hudi/pull/13342#issuecomment-2909207775
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0376c2488eae2662f5521185afb2d8634d0f0660",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5636",
"triggerID" : "0376c2488eae2662f5521185afb2d8634d0f0660",
"triggerType" : "PUSH"
}, {
"hash" : "bfcc764ea299b40a91a7a2864f2759d4a485663f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5640",
"triggerID" : "bfcc764ea299b40a91a7a2864f2759d4a485663f",
"triggerType" : "PUSH"
}, {
"hash" : "e47bf80290369887dde115148528392967ac600d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5673",
"triggerID" : "e47bf80290369887dde115148528392967ac600d",
"triggerType" : "PUSH"
}, {
"hash" : "b35bcc913b34302e56c59e0c20ecbf09fd406990",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5740",
"triggerID" : "b35bcc913b34302e56c59e0c20ecbf09fd406990",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e47bf80290369887dde115148528392967ac600d Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5673)
* b35bcc913b34302e56c59e0c20ecbf09fd406990 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5740)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9482] Fix the issue where the file slice was wrongly filtered out [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13342:
URL: https://github.com/apache/hudi/pull/13342#issuecomment-2900156249
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0376c2488eae2662f5521185afb2d8634d0f0660",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0376c2488eae2662f5521185afb2d8634d0f0660",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0376c2488eae2662f5521185afb2d8634d0f0660 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9482] Fix the issue where the file slice was wrongly filtered out [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13342:
URL: https://github.com/apache/hudi/pull/13342#discussion_r2102216973
##########
hudi-common/src/main/java/org/apache/hudi/common/model/HoodieFileGroup.java:
##########
@@ -119,6 +121,10 @@ public void addLogFile(CompletionTimeQueryView completionTimeQueryView, HoodieLo
@VisibleForTesting
public String getBaseInstantTime(CompletionTimeQueryView completionTimeQueryView, HoodieLogFile logFile) {
if (fileSlices.isEmpty()) {
+ if (completionTimeQueryView.getTableVersion().greaterThanOrEquals(HoodieTableVersion.EIGHT)) {
+ // no base file in the file group, use INITIAL INSTANT TIME as base instant time
+ return HoodieActiveTimeline.INIT_INSTANT_TS;
Review Comment:
Looks like a version compatibility issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9482] Fix the issue where the file slice was wrongly filtered out [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13342:
URL: https://github.com/apache/hudi/pull/13342#issuecomment-2909480890
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0376c2488eae2662f5521185afb2d8634d0f0660",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5636",
"triggerID" : "0376c2488eae2662f5521185afb2d8634d0f0660",
"triggerType" : "PUSH"
}, {
"hash" : "bfcc764ea299b40a91a7a2864f2759d4a485663f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5640",
"triggerID" : "bfcc764ea299b40a91a7a2864f2759d4a485663f",
"triggerType" : "PUSH"
}, {
"hash" : "e47bf80290369887dde115148528392967ac600d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5673",
"triggerID" : "e47bf80290369887dde115148528392967ac600d",
"triggerType" : "PUSH"
}, {
"hash" : "b35bcc913b34302e56c59e0c20ecbf09fd406990",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5740",
"triggerID" : "b35bcc913b34302e56c59e0c20ecbf09fd406990",
"triggerType" : "PUSH"
}, {
"hash" : "d31cfdc44cfdcd9c0168ec79b1e16d487c285209",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d31cfdc44cfdcd9c0168ec79b1e16d487c285209",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b35bcc913b34302e56c59e0c20ecbf09fd406990 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5740)
* d31cfdc44cfdcd9c0168ec79b1e16d487c285209 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9482] Fix the issue where the file slice was wrongly filtered out [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13342:
URL: https://github.com/apache/hudi/pull/13342#issuecomment-2901215778
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0376c2488eae2662f5521185afb2d8634d0f0660",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5636",
"triggerID" : "0376c2488eae2662f5521185afb2d8634d0f0660",
"triggerType" : "PUSH"
}, {
"hash" : "bfcc764ea299b40a91a7a2864f2759d4a485663f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5640",
"triggerID" : "bfcc764ea299b40a91a7a2864f2759d4a485663f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* bfcc764ea299b40a91a7a2864f2759d4a485663f Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5640)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9482] Fix the issue where the file slice was wrongly filtered out [hudi]
Posted by "TheR1sing3un (via GitHub)" <gi...@apache.org>.
TheR1sing3un commented on code in PR #13342:
URL: https://github.com/apache/hudi/pull/13342#discussion_r2104173416
##########
hudi-common/src/main/java/org/apache/hudi/common/model/HoodieFileGroup.java:
##########
@@ -119,6 +121,10 @@ public void addLogFile(CompletionTimeQueryView completionTimeQueryView, HoodieLo
@VisibleForTesting
public String getBaseInstantTime(CompletionTimeQueryView completionTimeQueryView, HoodieLogFile logFile) {
if (fileSlices.isEmpty()) {
+ if (completionTimeQueryView.getTableVersion().greaterThanOrEquals(HoodieTableVersion.EIGHT)) {
+ // no base file in the file group, use INITIAL INSTANT TIME as base instant time
+ return HoodieActiveTimeline.INIT_INSTANT_TS;
Review Comment:
update:
When only the log with the smallest instant has not been completed yet, we adopt `INIT_INSTANT_TS` as the base instant because at this time we have no appropriate way to select a reasonable base instant.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9482] Fix the issue where the file slice was wrongly filtered out [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13342:
URL: https://github.com/apache/hudi/pull/13342#issuecomment-2903936439
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0376c2488eae2662f5521185afb2d8634d0f0660",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5636",
"triggerID" : "0376c2488eae2662f5521185afb2d8634d0f0660",
"triggerType" : "PUSH"
}, {
"hash" : "bfcc764ea299b40a91a7a2864f2759d4a485663f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5640",
"triggerID" : "bfcc764ea299b40a91a7a2864f2759d4a485663f",
"triggerType" : "PUSH"
}, {
"hash" : "e47bf80290369887dde115148528392967ac600d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5673",
"triggerID" : "e47bf80290369887dde115148528392967ac600d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e47bf80290369887dde115148528392967ac600d Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5673)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9482] Fix the issue where the file slice was wrongly filtered out [hudi]
Posted by "TheR1sing3un (via GitHub)" <gi...@apache.org>.
TheR1sing3un commented on code in PR #13342:
URL: https://github.com/apache/hudi/pull/13342#discussion_r2104177109
##########
hudi-common/src/main/java/org/apache/hudi/common/table/view/AbstractTableFileSystemView.java:
##########
@@ -899,7 +924,11 @@ public final Stream<FileSlice> getLatestFileSlicesStateless(String partitionStr)
.filter(Option::isPresent).map(Option::get)
.flatMap(slice -> tableVersion8AndAbove()
? this.filterUncommittedFiles(slice, true)
- : this.filterBaseFileAfterPendingCompaction(slice, true));
+ : this.filterBaseFileAfterPendingCompaction(slice, true))
+ .flatMap(slice -> tableVersion8AndAbove()
+ // for table version 8 and above, we need to filter out empty file slices
+ ? this.filterOutEmptyFileSlice(slice)
Review Comment:
same as above
##########
hudi-common/src/main/java/org/apache/hudi/common/table/view/AbstractTableFileSystemView.java:
##########
@@ -930,9 +959,13 @@ public final Option<FileSlice> getLatestFileSlice(String partitionStr, String fi
if (!fs.isPresent()) {
return Option.empty();
}
- Stream<FileSlice> fileSlices = tableVersion8AndAbove()
+ Stream<FileSlice> fileSlices = (tableVersion8AndAbove()
? this.filterUncommittedFiles(fs.get(), true)
- : this.filterBaseFileAfterPendingCompaction(fs.get(), true);
+ : this.filterBaseFileAfterPendingCompaction(fs.get(), true))
+ .flatMap(slice -> tableVersion8AndAbove()
+ // for table version 8 and above, we need to filter out empty file slices
+ ? this.filterOutEmptyFileSlice(slice)
+ : Stream.of(slice));
Review Comment:
same as above
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9482] Fix the issue where the file slice was wrongly filtered out [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13342:
URL: https://github.com/apache/hudi/pull/13342#issuecomment-2900159841
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0376c2488eae2662f5521185afb2d8634d0f0660",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5636",
"triggerID" : "0376c2488eae2662f5521185afb2d8634d0f0660",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0376c2488eae2662f5521185afb2d8634d0f0660 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5636)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9482] Fix the issue where the file slice was wrongly filtered out [hudi]
Posted by "TheR1sing3un (via GitHub)" <gi...@apache.org>.
TheR1sing3un commented on code in PR #13342:
URL: https://github.com/apache/hudi/pull/13342#discussion_r2102342300
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/action/commit/SparkInsertOverwriteCommitActionExecutor.java:
##########
@@ -96,8 +96,8 @@ protected Map<String, List<String>> getPartitionToReplacedFileIds(HoodieWriteMet
}
protected List<String> getAllExistingFileIds(String partitionPath) {
- // because new commit is not complete. it is safe to mark all existing file Ids as old files
- return table.getSliceView().getLatestFileSlices(partitionPath).map(FileSlice::getFileId).distinct().collect(Collectors.toList());
+ // we should only fetch the latest merged file slices with committed data
+ return table.getSliceView().getLatestMergedFileSlicesBeforeOrOn(partitionPath, instantTime).filter(slice -> !slice.isEmpty()).map(FileSlice::getFileId).distinct().collect(Collectors.toList());
Review Comment:
> In which case the file slice is empty?
When there is only one log file in this file group and this log file is still in the pending state.
<img width="987" alt="image" src="https://github.com/user-attachments/assets/2dd532a9-2c8f-4868-8f2b-22a405a61491" />
1. step1: fetch latest file slice which is filtered by `HoodieFileGroup#isFileSliceCommitted`
<img width="1020" alt="image" src="https://github.com/user-attachments/assets/75ae6e1e-0523-45a0-b020-616c2fbe70c4" />
`timeline.containsOrBeforeTimelineStarts(slice.getBaseInstantTime())` will always return true because of `slice.getBaseInstantTime` will return `HoodieActiveTimeline.INIT_INSTANT_TS`
So step-1 will get a flie-slice:`{base: empty, log: log-xxxx}`
2. step2: filter uncommitted log file by completion-time, so `log-xxxx` will be filtered out, so the step-2 will get a flie-slice: `{base: empty, log: empty}`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9482] Fix the issue where the file slice was wrongly filtered out [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13342:
URL: https://github.com/apache/hudi/pull/13342#issuecomment-2910013502
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0376c2488eae2662f5521185afb2d8634d0f0660",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5636",
"triggerID" : "0376c2488eae2662f5521185afb2d8634d0f0660",
"triggerType" : "PUSH"
}, {
"hash" : "bfcc764ea299b40a91a7a2864f2759d4a485663f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5640",
"triggerID" : "bfcc764ea299b40a91a7a2864f2759d4a485663f",
"triggerType" : "PUSH"
}, {
"hash" : "e47bf80290369887dde115148528392967ac600d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5673",
"triggerID" : "e47bf80290369887dde115148528392967ac600d",
"triggerType" : "PUSH"
}, {
"hash" : "b35bcc913b34302e56c59e0c20ecbf09fd406990",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5740",
"triggerID" : "b35bcc913b34302e56c59e0c20ecbf09fd406990",
"triggerType" : "PUSH"
}, {
"hash" : "d31cfdc44cfdcd9c0168ec79b1e16d487c285209",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5742",
"triggerID" : "d31cfdc44cfdcd9c0168ec79b1e16d487c285209",
"triggerType" : "PUSH"
}, {
"hash" : "d31cfdc44cfdcd9c0168ec79b1e16d487c285209",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5744",
"triggerID" : "2909795186",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* d31cfdc44cfdcd9c0168ec79b1e16d487c285209 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5742) Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5744)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9482] Fix the issue where the file slice was wrongly filtered out [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13342:
URL: https://github.com/apache/hudi/pull/13342#issuecomment-2909718568
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0376c2488eae2662f5521185afb2d8634d0f0660",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5636",
"triggerID" : "0376c2488eae2662f5521185afb2d8634d0f0660",
"triggerType" : "PUSH"
}, {
"hash" : "bfcc764ea299b40a91a7a2864f2759d4a485663f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5640",
"triggerID" : "bfcc764ea299b40a91a7a2864f2759d4a485663f",
"triggerType" : "PUSH"
}, {
"hash" : "e47bf80290369887dde115148528392967ac600d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5673",
"triggerID" : "e47bf80290369887dde115148528392967ac600d",
"triggerType" : "PUSH"
}, {
"hash" : "b35bcc913b34302e56c59e0c20ecbf09fd406990",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5740",
"triggerID" : "b35bcc913b34302e56c59e0c20ecbf09fd406990",
"triggerType" : "PUSH"
}, {
"hash" : "d31cfdc44cfdcd9c0168ec79b1e16d487c285209",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5742",
"triggerID" : "d31cfdc44cfdcd9c0168ec79b1e16d487c285209",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d31cfdc44cfdcd9c0168ec79b1e16d487c285209 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5742)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9482] Fix the issue where the file slice was wrongly filtered out [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13342:
URL: https://github.com/apache/hudi/pull/13342#issuecomment-2909485962
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0376c2488eae2662f5521185afb2d8634d0f0660",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5636",
"triggerID" : "0376c2488eae2662f5521185afb2d8634d0f0660",
"triggerType" : "PUSH"
}, {
"hash" : "bfcc764ea299b40a91a7a2864f2759d4a485663f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5640",
"triggerID" : "bfcc764ea299b40a91a7a2864f2759d4a485663f",
"triggerType" : "PUSH"
}, {
"hash" : "e47bf80290369887dde115148528392967ac600d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5673",
"triggerID" : "e47bf80290369887dde115148528392967ac600d",
"triggerType" : "PUSH"
}, {
"hash" : "b35bcc913b34302e56c59e0c20ecbf09fd406990",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5740",
"triggerID" : "b35bcc913b34302e56c59e0c20ecbf09fd406990",
"triggerType" : "PUSH"
}, {
"hash" : "d31cfdc44cfdcd9c0168ec79b1e16d487c285209",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5742",
"triggerID" : "d31cfdc44cfdcd9c0168ec79b1e16d487c285209",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b35bcc913b34302e56c59e0c20ecbf09fd406990 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5740)
* d31cfdc44cfdcd9c0168ec79b1e16d487c285209 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5742)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9482] Fix the issue where the file slice was wrongly filtered out [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13342:
URL: https://github.com/apache/hudi/pull/13342#issuecomment-2903797754
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0376c2488eae2662f5521185afb2d8634d0f0660",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5636",
"triggerID" : "0376c2488eae2662f5521185afb2d8634d0f0660",
"triggerType" : "PUSH"
}, {
"hash" : "bfcc764ea299b40a91a7a2864f2759d4a485663f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5640",
"triggerID" : "bfcc764ea299b40a91a7a2864f2759d4a485663f",
"triggerType" : "PUSH"
}, {
"hash" : "e47bf80290369887dde115148528392967ac600d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e47bf80290369887dde115148528392967ac600d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* bfcc764ea299b40a91a7a2864f2759d4a485663f Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5640)
* e47bf80290369887dde115148528392967ac600d UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9482] Fix the issue where the file slice was wrongly filtered out [hudi]
Posted by "TheR1sing3un (via GitHub)" <gi...@apache.org>.
TheR1sing3un commented on PR #13342:
URL: https://github.com/apache/hudi/pull/13342#issuecomment-2900915045
The essence of this problem is: How can we confirm for a file group that only has log files what the base instant of its slice is? The current logic is to directly take the instant with the smallest character order as the base instant. But in this way, the problems I mentioned above will arise. I think we can pretend that each file group initially has an `HoodieActiveTimeline.INIT_INSTANT_TS` version, but this version has no actual data written. In this way, we don't have to consider how to handle slices that are all logs.
My current code changes are based on the above logic.
I'm not sure if there are any other better methods. Looking forward to everyone's thoughts
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9482] Fix the issue where the file slice was wrongly filtered out [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13342:
URL: https://github.com/apache/hudi/pull/13342#issuecomment-2909201637
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0376c2488eae2662f5521185afb2d8634d0f0660",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5636",
"triggerID" : "0376c2488eae2662f5521185afb2d8634d0f0660",
"triggerType" : "PUSH"
}, {
"hash" : "bfcc764ea299b40a91a7a2864f2759d4a485663f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5640",
"triggerID" : "bfcc764ea299b40a91a7a2864f2759d4a485663f",
"triggerType" : "PUSH"
}, {
"hash" : "e47bf80290369887dde115148528392967ac600d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5673",
"triggerID" : "e47bf80290369887dde115148528392967ac600d",
"triggerType" : "PUSH"
}, {
"hash" : "b35bcc913b34302e56c59e0c20ecbf09fd406990",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b35bcc913b34302e56c59e0c20ecbf09fd406990",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e47bf80290369887dde115148528392967ac600d Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5673)
* b35bcc913b34302e56c59e0c20ecbf09fd406990 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9482] Fix the issue where the file slice was wrongly filtered out [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13342:
URL: https://github.com/apache/hudi/pull/13342#issuecomment-2901019184
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0376c2488eae2662f5521185afb2d8634d0f0660",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5636",
"triggerID" : "0376c2488eae2662f5521185afb2d8634d0f0660",
"triggerType" : "PUSH"
}, {
"hash" : "bfcc764ea299b40a91a7a2864f2759d4a485663f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bfcc764ea299b40a91a7a2864f2759d4a485663f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0376c2488eae2662f5521185afb2d8634d0f0660 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5636)
* bfcc764ea299b40a91a7a2864f2759d4a485663f UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9482] Fix the issue where the file slice was wrongly filtered out [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13342:
URL: https://github.com/apache/hudi/pull/13342#discussion_r2102216522
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/action/commit/SparkInsertOverwriteCommitActionExecutor.java:
##########
@@ -96,8 +96,8 @@ protected Map<String, List<String>> getPartitionToReplacedFileIds(HoodieWriteMet
}
protected List<String> getAllExistingFileIds(String partitionPath) {
- // because new commit is not complete. it is safe to mark all existing file Ids as old files
- return table.getSliceView().getLatestFileSlices(partitionPath).map(FileSlice::getFileId).distinct().collect(Collectors.toList());
+ // we should only fetch the latest merged file slices with committed data
+ return table.getSliceView().getLatestMergedFileSlicesBeforeOrOn(partitionPath, instantTime).filter(slice -> !slice.isEmpty()).map(FileSlice::getFileId).distinct().collect(Collectors.toList());
Review Comment:
In which case the file slice is empty?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9482] Fix the issue where the file slice was wrongly filtered out [hudi]
Posted by "TheR1sing3un (via GitHub)" <gi...@apache.org>.
TheR1sing3un commented on code in PR #13342:
URL: https://github.com/apache/hudi/pull/13342#discussion_r2104182907
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/action/commit/SparkInsertOverwriteCommitActionExecutor.java:
##########
@@ -96,8 +96,8 @@ protected Map<String, List<String>> getPartitionToReplacedFileIds(HoodieWriteMet
}
protected List<String> getAllExistingFileIds(String partitionPath) {
- // because new commit is not complete. it is safe to mark all existing file Ids as old files
- return table.getSliceView().getLatestFileSlices(partitionPath).map(FileSlice::getFileId).distinct().collect(Collectors.toList());
+ // we should only fetch the latest merged file slices with committed data
+ return table.getSliceView().getLatestMergedFileSlicesBeforeOrOn(partitionPath, instantTime).map(FileSlice::getFileId).distinct().collect(Collectors.toList());
Review Comment:
The changes here are intended to achieve the following two points:
1. only the view completed before this instant should be seen to ensure transactionality.
2. we should select the file id of the file slice where the valid commit data actually exists as the replaced file id.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9482] Fix the issue where the file slice was wrongly filtered out [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13342:
URL: https://github.com/apache/hudi/pull/13342#issuecomment-2909414812
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0376c2488eae2662f5521185afb2d8634d0f0660",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5636",
"triggerID" : "0376c2488eae2662f5521185afb2d8634d0f0660",
"triggerType" : "PUSH"
}, {
"hash" : "bfcc764ea299b40a91a7a2864f2759d4a485663f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5640",
"triggerID" : "bfcc764ea299b40a91a7a2864f2759d4a485663f",
"triggerType" : "PUSH"
}, {
"hash" : "e47bf80290369887dde115148528392967ac600d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5673",
"triggerID" : "e47bf80290369887dde115148528392967ac600d",
"triggerType" : "PUSH"
}, {
"hash" : "b35bcc913b34302e56c59e0c20ecbf09fd406990",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5740",
"triggerID" : "b35bcc913b34302e56c59e0c20ecbf09fd406990",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b35bcc913b34302e56c59e0c20ecbf09fd406990 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5740)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9482] Fix the issue where the file slice was wrongly filtered out [hudi]
Posted by "TheR1sing3un (via GitHub)" <gi...@apache.org>.
TheR1sing3un commented on code in PR #13342:
URL: https://github.com/apache/hudi/pull/13342#discussion_r2104176327
##########
hudi-common/src/main/java/org/apache/hudi/common/table/view/AbstractTableFileSystemView.java:
##########
@@ -867,6 +888,10 @@ public final Stream<FileSlice> getLatestFileSlices(String partitionStr) {
.flatMap(slice -> tableVersion8AndAbove()
? this.filterUncommittedFiles(slice, true)
: this.filterBaseFileAfterPendingCompaction(slice, true))
+ .flatMap(slice -> tableVersion8AndAbove()
+ // for table version 8 and above, we need to filter out empty file slices
+ ? this.filterOutEmptyFileSlice(slice)
+ : Stream.of(slice))
Review Comment:
should filter out the `isEmpty` slice, because this slice is not `isEmpty` due to previous pending compaction, but because there are no valid commits at all in this slice
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9482] Fix the issue where the file slice was wrongly filtered out [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13342:
URL: https://github.com/apache/hudi/pull/13342#issuecomment-2912762897
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0376c2488eae2662f5521185afb2d8634d0f0660",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5636",
"triggerID" : "0376c2488eae2662f5521185afb2d8634d0f0660",
"triggerType" : "PUSH"
}, {
"hash" : "bfcc764ea299b40a91a7a2864f2759d4a485663f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5640",
"triggerID" : "bfcc764ea299b40a91a7a2864f2759d4a485663f",
"triggerType" : "PUSH"
}, {
"hash" : "e47bf80290369887dde115148528392967ac600d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5673",
"triggerID" : "e47bf80290369887dde115148528392967ac600d",
"triggerType" : "PUSH"
}, {
"hash" : "b35bcc913b34302e56c59e0c20ecbf09fd406990",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5740",
"triggerID" : "b35bcc913b34302e56c59e0c20ecbf09fd406990",
"triggerType" : "PUSH"
}, {
"hash" : "d31cfdc44cfdcd9c0168ec79b1e16d487c285209",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5742",
"triggerID" : "d31cfdc44cfdcd9c0168ec79b1e16d487c285209",
"triggerType" : "PUSH"
}, {
"hash" : "d31cfdc44cfdcd9c0168ec79b1e16d487c285209",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5744",
"triggerID" : "2909795186",
"triggerType" : "MANUAL"
}, {
"hash" : "d31cfdc44cfdcd9c0168ec79b1e16d487c285209",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5770",
"triggerID" : "2912360271",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* d31cfdc44cfdcd9c0168ec79b1e16d487c285209 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5742) Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5744) Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5770)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9482] Fix the issue where the file slice was wrongly filtered out [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13342:
URL: https://github.com/apache/hudi/pull/13342#issuecomment-2900323217
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0376c2488eae2662f5521185afb2d8634d0f0660",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5636",
"triggerID" : "0376c2488eae2662f5521185afb2d8634d0f0660",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0376c2488eae2662f5521185afb2d8634d0f0660 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5636)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9482] Fix the issue where the file slice was wrongly filtered out [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13342:
URL: https://github.com/apache/hudi/pull/13342#issuecomment-2901024607
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0376c2488eae2662f5521185afb2d8634d0f0660",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5636",
"triggerID" : "0376c2488eae2662f5521185afb2d8634d0f0660",
"triggerType" : "PUSH"
}, {
"hash" : "bfcc764ea299b40a91a7a2864f2759d4a485663f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5640",
"triggerID" : "bfcc764ea299b40a91a7a2864f2759d4a485663f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0376c2488eae2662f5521185afb2d8634d0f0660 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5636)
* bfcc764ea299b40a91a7a2864f2759d4a485663f Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5640)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9482] Fix the issue where the file slice was wrongly filtered out [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13342:
URL: https://github.com/apache/hudi/pull/13342#issuecomment-2909799112
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0376c2488eae2662f5521185afb2d8634d0f0660",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5636",
"triggerID" : "0376c2488eae2662f5521185afb2d8634d0f0660",
"triggerType" : "PUSH"
}, {
"hash" : "bfcc764ea299b40a91a7a2864f2759d4a485663f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5640",
"triggerID" : "bfcc764ea299b40a91a7a2864f2759d4a485663f",
"triggerType" : "PUSH"
}, {
"hash" : "e47bf80290369887dde115148528392967ac600d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5673",
"triggerID" : "e47bf80290369887dde115148528392967ac600d",
"triggerType" : "PUSH"
}, {
"hash" : "b35bcc913b34302e56c59e0c20ecbf09fd406990",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5740",
"triggerID" : "b35bcc913b34302e56c59e0c20ecbf09fd406990",
"triggerType" : "PUSH"
}, {
"hash" : "d31cfdc44cfdcd9c0168ec79b1e16d487c285209",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5742",
"triggerID" : "d31cfdc44cfdcd9c0168ec79b1e16d487c285209",
"triggerType" : "PUSH"
}, {
"hash" : "d31cfdc44cfdcd9c0168ec79b1e16d487c285209",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5744",
"triggerID" : "2909795186",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* d31cfdc44cfdcd9c0168ec79b1e16d487c285209 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5742) Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5744)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9482] Fix the issue where the file slice was wrongly filtered out [hudi]
Posted by "TheR1sing3un (via GitHub)" <gi...@apache.org>.
TheR1sing3un commented on PR #13342:
URL: https://github.com/apache/hudi/pull/13342#issuecomment-2909795186
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9482] Fix the issue where the file slice was wrongly filtered out [hudi]
Posted by "TheR1sing3un (via GitHub)" <gi...@apache.org>.
TheR1sing3un commented on PR #13342:
URL: https://github.com/apache/hudi/pull/13342#issuecomment-2912360271
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org