[jira] [Updated] (HUDI-5569) Files written by first commit/delta commit if it failed is detected as valid data files

["Raymond Xu (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HUDI-5569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Raymond Xu updated HUDI-5569:
-----------------------------
    Sprint: 0.13.0 Final Sprint 2, 0.13.0 Final Sprint 3, Sprint 2023-01-31, Sprint 2023-02-14, Sprint 2023-02-28  (was: 0.13.0 Final Sprint 2, 0.13.0 Final Sprint 3, Sprint 2023-01-31, Sprint 2023-02-14)

> Files written by first commit/delta commit if it failed is detected as valid data files
> ---------------------------------------------------------------------------------------
>
>                 Key: HUDI-5569
>                 URL: https://issues.apache.org/jira/browse/HUDI-5569
>             Project: Apache Hudi
>          Issue Type: Bug
>          Components: writer-core
>            Reporter: sivabalan narayanan
>            Assignee: sivabalan narayanan
>            Priority: Critical
>              Labels: pull-request-available
>             Fix For: 0.14.0
>
>
> We have an method in HoodieFileGroup which detects whether a file group is committed or not. If timeline is such that, 
> c1.inflight
> c2.complete
> c3.complete
>  
> when we check for c1, it will return true. 
> HoodieFileGroup.java
> {code:java}
> /**
>  * A FileSlice is considered committed, if one of the following is true - There is a committed data file - There are
>  * some log files, that are based off a commit or delta commit.
>  */
> private boolean isFileSliceCommitted(FileSlice slice) {
>   if (!compareTimestamps(slice.getBaseInstantTime(), LESSER_THAN_OR_EQUALS, lastInstant.get().getTimestamp())) {
>     return false;
>   }
>   return timeline.containsOrBeforeTimelineStarts(slice.getBaseInstantTime());
> } {code}
> HoodieDefaultTimeline : 
> {code:java}
> @Override
> public boolean containsOrBeforeTimelineStarts(String instant) {
>   return getInstantsAsStream().anyMatch(s -> s.getTimestamp().equals(instant)) || isBeforeTimelineStarts(instant);
> } {code}
>  
> This needs to be fixed. 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)