[GitHub] [hudi] codope commented on a diff in pull request #6516: [HUDI-4729] Fix fq can not be queried in pending compaction when query ro table with spark

[GitBox <gi...@apache.org>]

codope commented on code in PR #6516:
URL: https://github.com/apache/hudi/pull/6516#discussion_r972973080


##########
hudi-common/src/main/java/org/apache/hudi/common/table/view/AbstractTableFileSystemView.java:
##########
@@ -665,13 +665,25 @@ public final Stream<FileSlice> getLatestFileSlicesBeforeOrOn(String partitionStr
       readLock.lock();
       String partitionPath = formatPartitionKey(partitionStr);
       ensurePartitionLoadedCorrectly(partitionPath);
-      Stream<FileSlice> fileSliceStream = fetchLatestFileSlicesBeforeOrOn(partitionPath, maxCommitTime)
-          .filter(slice -> !isFileGroupReplacedBeforeOrOn(slice.getFileGroupId(), maxCommitTime));
+      Stream<Stream<FileSlice>> allFileSliceStream = fetchAllStoredFileGroups(partitionPath)
+              .map(fg -> fg.getAllFileSlicesBeforeOn(maxCommitTime).filter(slice -> !isFileGroupReplacedBeforeOrOn(slice.getFileGroupId(), maxCommitTime)));
       if (includeFileSlicesInPendingCompaction) {
-        return fileSliceStream.map(this::filterBaseFileAfterPendingCompaction).map(this::addBootstrapBaseFileIfPresent);
+        return allFileSliceStream
+                .map(sliceStream ->

Review Comment:
   I still think it would be better to change in `filterBaseFileAfterPendingCompaction` otherwise we will have to apply the fix at other callers of this method as well. 
   
   You can change the method to accept a stream and return a stream, and then at the callers do something like:
   ```
   filterBaseFileAfterPendingCompaction(fileSliceStream).map(this::addBootstrapBaseFileIfPresent)
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org