Re: [PR] [HUDI-7532] Include only compaction instants for lastCompaction in getDeltaCommitsSinceLatestCompaction [hudi]

["codope (via GitHub)" <gi...@apache.org>]

codope commented on code in PR #10915:
URL: https://github.com/apache/hudi/pull/10915#discussion_r1549072910


##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieDefaultTimeline.java:
##########
@@ -147,6 +147,12 @@ public HoodieTimeline filterCompletedAndCompactionInstants() {
             || s.getAction().equals(HoodieTimeline.COMPACTION_ACTION)), details);
   }
 
+  @Override
+  public HoodieTimeline filterCompletedCommitInstants() {

Review Comment:
   Let's unit test this.



##########
hudi-common/src/main/java/org/apache/hudi/common/util/CompactionUtils.java:
##########
@@ -285,8 +285,7 @@ public static List<HoodieInstant> getPendingCompactionInstantTimes(HoodieTableMe
    */
   public static Option<Pair<HoodieTimeline, HoodieInstant>> getDeltaCommitsSinceLatestCompaction(
       HoodieActiveTimeline activeTimeline) {
-    Option<HoodieInstant> lastCompaction = activeTimeline.getCommitTimeline()
-        .filterCompletedInstants().lastInstant();
+    Option<HoodieInstant> lastCompaction = activeTimeline.filterCompletedCommitInstants().lastInstant();

Review Comment:
   This change is causing a test failure -  `TestCDCDataFrameSuite.testMORDataSourceWrite`. Can you please check that?
   Also, I think instead of adding yet another API, we can clean up the existing ones.
   For instance, have separate method for each action - `get<ACTION>Timeline`. So, `getCommitTimeline` only checks for  COMMIT action.



##########
hudi-common/src/test/java/org/apache/hudi/common/util/TestCompactionUtils.java:
##########
@@ -272,6 +272,59 @@ public void testGetDeltaCommitsSinceLatestCompaction(boolean hasCompletedCompact
     }
   }
 
+  @Test
+  public void testGetDeltaCommitsSinceLastCompactionWithCompletedReplaceCommits() {

Review Comment:
   +1 for the test



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org