You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@fluss.apache.org by "luoyuxia (via GitHub)" <gi...@apache.org> on 2025/12/15 03:51:01 UTC

[PR] Fix union read paimon issue [fluss]

luoyuxia opened a new pull request, #2170:
URL: https://github.com/apache/fluss/pull/2170

   <!--
   *Thank you very much for contributing to Fluss - we are happy that you want to help us improve Fluss. To help the community review your contribution in the best possible way, please go through the checklist below, which will get the contribution into a shape in which it can be best reviewed.*
   
   ## Contribution Checklist
   
     - Make sure that the pull request corresponds to a [GitHub issue](https://github.com/apache/fluss/issues). Exceptions are made for typos in JavaDoc or documentation files, which need no issue.
   
     - Name the pull request in the format "[component] Title of the pull request", where *[component]* should be replaced by the name of the component being changed. Typically, this corresponds to the component label assigned to the issue (e.g., [kv], [log], [client], [flink]). Skip *[component]* if you are unsure about which is the best component.
   
     - Fill out the template below to describe the changes contributed by the pull request. That will give reviewers the context they need to do the review.
   
     - Make sure that the change passes the automated tests, i.e., `mvn clean verify` passes.
   
     - Each pull request should address only one issue, not mix up code from multiple issues.
   
   
   **(The sections below can be removed for hotfixes or typos)**
   -->
   
   ### Purpose
   
   <!-- Linking this pull request to the issue -->
   Linked issue: close #xxx
   
   <!-- What is the purpose of the change -->
   
   ### Brief change log
   
   <!-- Please describe the changes made in this pull request and explain how they address the issue -->
   
   ### Tests
   
   <!-- List UT and IT cases to verify this change -->
   
   ### API and Format
   
   <!-- Does this change affect API or storage format -->
   
   ### Documentation
   
   <!-- Does this change introduce a new feature -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix union read paimon issue [fluss]

Posted by "luoyuxia (via GitHub)" <gi...@apache.org>.
luoyuxia commented on code in PR #2170:
URL: https://github.com/apache/fluss/pull/2170#discussion_r2618148866


##########
fluss-flink/fluss-flink-common/src/main/java/org/apache/fluss/flink/lake/reader/SortMergeReader.java:
##########
@@ -300,12 +307,27 @@ public void close() {
 
         @Override
         public boolean hasNext() {
-            return changeLogRecordIterator != null && changeLogRecordIterator.hasNext();

Review Comment:
   the next will be delete row.



##########
fluss-flink/fluss-flink-common/src/main/java/org/apache/fluss/flink/lake/reader/LakeSnapshotAndLogSplitScanner.java:
##########
@@ -192,23 +192,26 @@ public CloseableIterator<InternalRow> pollBatch(Duration timeout) throws IOExcep
             return currentSortMergeReader.readBatch();
         } else {
             if (lakeRecordIterators.isEmpty()) {
+                List<RecordReader> recordReaders = new ArrayList<>();
                 if (lakeSnapshotSplitAndFlussLogSplit.getLakeSplits() == null
                         || lakeSnapshotSplitAndFlussLogSplit.getLakeSplits().isEmpty()) {
-                    lakeRecordIterators = Collections.emptyList();
-                    logRows = new LinkedHashMap<>();

Review Comment:
   ProjectedRow doesn't support hash & equal. It'll throw exception if come to here.



##########
fluss-flink/fluss-flink-common/src/main/java/org/apache/fluss/flink/lake/LakeSplitReaderGenerator.java:
##########
@@ -55,8 +55,16 @@ public void addSplit(SourceSplitBase split, Queue<SourceSplitBase> boundedSplits
         if (split instanceof LakeSnapshotSplit) {
             boundedSplits.add(split);
         } else if (split instanceof LakeSnapshotAndFlussLogSplit) {
-            // lake split not finished, add to it
-            if (!((LakeSnapshotAndFlussLogSplit) split).isLakeSplitFinished()) {
+            LakeSnapshotAndFlussLogSplit lakeSnapshotAndFlussLogSplit =
+                    (LakeSnapshotAndFlussLogSplit) split;
+            boolean isStreaming = ((LakeSnapshotAndFlussLogSplit) split).isStreaming();
+            // if is streaming and lake split not finished, add to it
+            if (isStreaming) {
+                if (!lakeSnapshotAndFlussLogSplit.isLakeSplitFinished()) {
+                    boundedSplits.add(split);
+                }
+            } else {
+                // otherwise, in batch mode, always add it

Review Comment:
   in batch mode, we need to add it. otherwise, the split won't never be read & mark as finished.



##########
fluss-flink/fluss-flink-common/src/main/java/org/apache/fluss/flink/source/enumerator/FlinkSourceEnumerator.java:
##########
@@ -680,6 +688,7 @@ private void handleSplitsAdd(List<SourceSplitBase> splits, Throwable t) {
                         t);
             }
         }
+        doHandleSplitsAdd(splits);

Review Comment:
   handle split firstly before mark noMoreNewSplits to true



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix union read paimon issue [fluss]

Posted by "luoyuxia (via GitHub)" <gi...@apache.org>.
luoyuxia commented on PR #2170:
URL: https://github.com/apache/fluss/pull/2170#issuecomment-3652767585

   @beryllw Fix some issue during testing I found. I'll add test later


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix union read paimon issue [fluss]

Posted by "luoyuxia (via GitHub)" <gi...@apache.org>.
luoyuxia merged PR #2170:
URL: https://github.com/apache/fluss/pull/2170


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix union read paimon issue [fluss]

Posted by "beryllw (via GitHub)" <gi...@apache.org>.
beryllw commented on PR #2170:
URL: https://github.com/apache/fluss/pull/2170#issuecomment-3664383736

   LGTM


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix union read paimon issue [fluss]

Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #2170:
URL: https://github.com/apache/fluss/pull/2170#discussion_r2617861546


##########
fluss-flink/fluss-flink-common/src/main/java/org/apache/fluss/flink/lake/reader/LakeSnapshotAndLogSplitScanner.java:
##########
@@ -192,23 +192,26 @@ public CloseableIterator<InternalRow> pollBatch(Duration timeout) throws IOExcep
             return currentSortMergeReader.readBatch();
         } else {
             if (lakeRecordIterators.isEmpty()) {
+                List<RecordReader> recordReaders = new ArrayList<>();
                 if (lakeSnapshotSplitAndFlussLogSplit.getLakeSplits() == null
                         || lakeSnapshotSplitAndFlussLogSplit.getLakeSplits().isEmpty()) {
-                    lakeRecordIterators = Collections.emptyList();
-                    logRows = new LinkedHashMap<>();
+                    // pass null split to get rowComparator
+                    recordReaders.add(lakeSource.createRecordReader(() -> null));
                 } else {
                     for (LakeSplit lakeSplit : lakeSnapshotSplitAndFlussLogSplit.getLakeSplits()) {
-                        RecordReader reader = lakeSource.createRecordReader(() -> lakeSplit);
-                        if (reader instanceof SortedRecordReader) {
-                            rowComparator = ((SortedRecordReader) reader).order();
-                        } else {
-                            throw new UnsupportedOperationException(
-                                    "lake records must instance of sorted view.");
-                        }
-                        lakeRecordIterators.add(reader.read());
+                        recordReaders.add(lakeSource.createRecordReader(() -> lakeSplit));
+                    }
+                }
+                for (RecordReader reader : recordReaders) {
+                    if (reader instanceof SortedRecordReader) {
+                        rowComparator = ((SortedRecordReader) reader).order();
+                    } else {
+                        throw new UnsupportedOperationException(
+                                "lake records must instance of sorted view.");
                     }
-                    logRows = new TreeMap<>(rowComparator);
+                    lakeRecordIterators.add(reader.read());

Review Comment:
   When there are no lake splits, a RecordReader is created with a null split just to obtain the rowComparator. However, this reader is then added to lakeRecordIterators list, which means its iterator (containing no data) will be included in the sort merge process. This could be inefficient. Consider extracting the rowComparator without adding the empty iterator to lakeRecordIterators, or ensure the empty iterator is properly handled during merging.
   ```suggestion
                       // Only add iterator if the reader is not for a null split
                       // (i.e., only if there are actual splits)
                       if (!(lakeSnapshotSplitAndFlussLogSplit.getLakeSplits() == null
                             || lakeSnapshotSplitAndFlussLogSplit.getLakeSplits().isEmpty())) {
                           lakeRecordIterators.add(reader.read());
                       }
   ```



##########
fluss-flink/fluss-flink-common/src/main/java/org/apache/fluss/flink/lake/reader/LakeSnapshotAndLogSplitScanner.java:
##########
@@ -109,7 +107,9 @@ public LakeSnapshotAndLogSplitScanner(
                                                 "StoppingOffset is null for split: "
                                                         + lakeSnapshotAndFlussLogSplit));
 
-        this.logScanFinished = lakeSnapshotAndFlussLogSplit.getStartingOffset() >= stoppingOffset;
+        this.logScanFinished =
+                lakeSnapshotAndFlussLogSplit.getStartingOffset() >= stoppingOffset
+                        || stoppingOffset <= 0;

Review Comment:
   The condition `stoppingOffset <= 0` may not be the correct check. If stoppingOffset is exactly 0, this means we want to stop at offset 0, not that the scan should be finished immediately. Consider whether the intended logic is `stoppingOffset < 0` to indicate an invalid or uninitialized offset, or if this check should be removed entirely.
   ```suggestion
                           || stoppingOffset < 0;
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org