You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "Raymond Xu (Jira)" <ji...@apache.org> on 2023/02/01 00:18:00 UTC
[jira] [Updated] (HUDI-5569) Files written by first commit/delta commit if it failed is detected as valid data files
[ https://issues.apache.org/jira/browse/HUDI-5569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raymond Xu updated HUDI-5569:
-----------------------------
Sprint: 0.13.0 Final Sprint 2, 0.13.0 Final Sprint 3, Sprint 2023-01-31 (was: 0.13.0 Final Sprint 2, 0.13.0 Final Sprint 3)
> Files written by first commit/delta commit if it failed is detected as valid data files
> ---------------------------------------------------------------------------------------
>
> Key: HUDI-5569
> URL: https://issues.apache.org/jira/browse/HUDI-5569
> Project: Apache Hudi
> Issue Type: Bug
> Components: writer-core
> Reporter: sivabalan narayanan
> Assignee: sivabalan narayanan
> Priority: Critical
> Labels: pull-request-available
> Fix For: 0.13.0
>
>
> We have an method in HoodieFileGroup which detects whether a file group is committed or not. If timeline is such that,
> c1.inflight
> c2.complete
> c3.complete
>
> when we check for c1, it will return true.
> HoodieFileGroup.java
> {code:java}
> /**
> * A FileSlice is considered committed, if one of the following is true - There is a committed data file - There are
> * some log files, that are based off a commit or delta commit.
> */
> private boolean isFileSliceCommitted(FileSlice slice) {
> if (!compareTimestamps(slice.getBaseInstantTime(), LESSER_THAN_OR_EQUALS, lastInstant.get().getTimestamp())) {
> return false;
> }
> return timeline.containsOrBeforeTimelineStarts(slice.getBaseInstantTime());
> } {code}
> HoodieDefaultTimeline :
> {code:java}
> @Override
> public boolean containsOrBeforeTimelineStarts(String instant) {
> return getInstantsAsStream().anyMatch(s -> s.getTimestamp().equals(instant)) || isBeforeTimelineStarts(instant);
> } {code}
>
> This needs to be fixed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)