[GitHub] [hudi] nsivabalan commented on a diff in pull request #9280: [HUDI-6587] Handle hollow commit for time travel query

["nsivabalan (via GitHub)" <gi...@apache.org>]

nsivabalan commented on code in PR #9280:
URL: https://github.com/apache/hudi/pull/9280#discussion_r1280073349


##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/TimelineUtils.java:
##########
@@ -343,7 +343,8 @@ public static HoodieTimeline handleHollowCommitIfNeeded(HoodieTimeline completed
     switch (handlingMode) {
       case FAIL:
         throw new HoodieException(String.format(
-            "Found hollow commit: '%s'. Adjust config `%s` accordingly if to avoid throwing this exception.",
+            "Found hollow commit: '%s'. If this happened for an incremental query, "

Review Comment:
   lets also add time travel query in addition to incremental query. 



##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/TimelineUtils.java:
##########
@@ -343,7 +343,8 @@ public static HoodieTimeline handleHollowCommitIfNeeded(HoodieTimeline completed
     switch (handlingMode) {
       case FAIL:
         throw new HoodieException(String.format(
-            "Found hollow commit: '%s'. Adjust config `%s` accordingly if to avoid throwing this exception.",
+            "Found hollow commit: '%s'. If this happened for an incremental query, "
+                + "adjust config `%s` accordingly if prefer to avoid throwing this exception.",

Review Comment:
   I am thinking if we can change the naming of this. 
   ConcurrentWritesReadStrategy
   or 
   OutofOrderCommitsReadStrategy. 
   



##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/MergeOnReadIncrementalRelation.scala:
##########
@@ -133,10 +134,7 @@ trait HoodieIncrementalRelationTrait extends HoodieBaseRelation {
   // Validate this Incremental implementation is properly configured
   validate()
 
-  protected val hollowCommitHandling: HollowCommitHandling =
-    optParams.get(DataSourceReadOptions.INCREMENTAL_READ_HANDLE_HOLLOW_COMMIT.key)
-      .map(HollowCommitHandling.valueOf)
-      .getOrElse(HollowCommitHandling.valueOf(DataSourceReadOptions.INCREMENTAL_READ_HANDLE_HOLLOW_COMMIT.defaultValue))
+  protected val hollowCommitHandling: HollowCommitHandling = getHollowCommitHandling(optParams)

Review Comment:
   can we add tests for diff HollowCommitHandling ? 



##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/TimelineUtils.java:
##########
@@ -343,7 +343,8 @@ public static HoodieTimeline handleHollowCommitIfNeeded(HoodieTimeline completed
     switch (handlingMode) {
       case FAIL:
         throw new HoodieException(String.format(
-            "Found hollow commit: '%s'. Adjust config `%s` accordingly if to avoid throwing this exception.",
+            "Found hollow commit: '%s'. If this happened for an incremental query, "
+                + "adjust config `%s` accordingly if prefer to avoid throwing this exception.",

Review Comment:
   Lets add documentation to enum values in L360. 



##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieBaseRelation.scala:
##########
@@ -420,7 +421,9 @@ abstract class HoodieBaseRelation(val sqlContext: SQLContext,
           inMemoryFileIndex.listFiles(partitionFilters, dataFilters)
         }
 
-        val fsView = new HoodieTableFileSystemView(metaClient, timeline, partitionDirs.flatMap(_.files).toArray)
+        // fail time travel query when covers hollow commit
+        val completedTimeline = handleHollowCommitIfNeeded(timeline, metaClient, HollowCommitHandling.FAIL)

Review Comment:
   can we add tests for diff HollowCommitHandling ? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org