You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "JingsongLi (via GitHub)" <gi...@apache.org> on 2026/07/14 04:32:51 UTC

[PR] [core] Add LSM maintenance for sorted primary-key indexes [paimon]

JingsongLi opened a new pull request, #8613:
URL: https://github.com/apache/paimon/pull/8613

   ## What changed
   
   - add field-level `pk-index.compaction.level-fanout` and `pk-index.compaction.stale-ratio-threshold` options for primary-key indexes
   - share the LSM level planner between sorted and vector primary-key indexes, replacing the vector-only configuration
   - maintain sorted primary-key indexes incrementally with independent multi-source index compaction
   - merge stale or over-fanout sorted index sources without coupling the work to data compaction
   - fail fast when a selected index build plan fails instead of silently skipping it
   
   ## Why
   
   Sorted primary-key indexes previously produced per-write index files without an independent compaction policy. Over time, readers could accumulate too many sources and retain stale entries until unrelated data compaction happened.
   
   This change gives each primary-key index its own LSM maintenance policy so source count and stale data can be controlled independently from data compaction.
   
   ## Impact
   
   Users can tune primary-key index compaction per indexed field with:
   
   - `fields.<field>.pk-index.compaction.level-fanout`
   - `fields.<field>.pk-index.compaction.stale-ratio-threshold`
   
   The old vector-specific compaction options are replaced; backward compatibility is intentionally not retained.
   
   ## Validation
   
   - 100 focused primary-key index tests passed
   - `mvn -pl paimon-core -am -DskipTests compile`
   - `git diff --check`
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [core] Add LSM maintenance for sorted primary-key indexes [paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #8613:
URL: https://github.com/apache/paimon/pull/8613#issuecomment-4966027076

   Addressed the sorted-index.records-per-range feedback in 7c6adb293b: PK BTree/Bitmap options now inherit only algorithm-specific options, and the obsolete range-sizing setup was removed from the related tests.
   
   Regarding the proposed upper limit, PK sorted-index compaction is scoped to a single bucket. We do not expect a single bucket to become excessively large, so an additional limit is not needed for now. We can revisit this if real workloads show otherwise.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [core] Add LSM maintenance for sorted primary-key indexes [paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #8613:
URL: https://github.com/apache/paimon/pull/8613#discussion_r3577594934


##########
paimon-common/src/main/java/org/apache/paimon/globalindex/bitmap/BitmapIndexReader.java:
##########
@@ -346,26 +344,27 @@ private RoaringNavigableMap64 scanSerializedDictionary(
     private int firstPossibleDictionaryBlock(
             List<BitmapGlobalIndexFormat.DictionaryBlockMeta> blocks,
             BitmapGlobalIndexFormat.SerializedKey key) {
-        int index = findDictionaryBlockIndex(blocks, key);
+        int index = findSerializedDictionaryBlockIndex(blocks, key);
         return Math.max(index, 0);
     }
 
-    private BitmapGlobalIndexFormat.BlockInfo findBitmapBlock(
-            BitmapGlobalIndexFormat.SerializedKey key) {
+    private BitmapGlobalIndexFormat.BlockInfo findBitmapBlock(Object key) {
         List<BitmapGlobalIndexFormat.DictionaryBlockMeta> blocks = dictionaryBlocks.get();
         if (blocks.isEmpty()) {
             return null;
         }
 
-        int index = findDictionaryBlockIndex(blocks, key);
+        int index = findLogicalDictionaryBlockIndex(blocks, key);

Review Comment:
   Bitmap global index was introduced by #8276 and has not been included in any release yet, so there are no released v1 files whose ordering needs to remain readable. This PR is still adjusting the format before its first release; keeping VERSION = 1 while switching to logical key ordering is intentional, and we do not need a legacy serialized-order read path.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [core] Add LSM maintenance for sorted primary-key indexes [paimon]

Posted by "leaves12138 (via GitHub)" <gi...@apache.org>.
leaves12138 commented on code in PR #8613:
URL: https://github.com/apache/paimon/pull/8613#discussion_r3577729804


##########
paimon-common/src/main/java/org/apache/paimon/globalindex/bitmap/BitmapIndexReader.java:
##########
@@ -346,26 +344,27 @@ private RoaringNavigableMap64 scanSerializedDictionary(
     private int firstPossibleDictionaryBlock(
             List<BitmapGlobalIndexFormat.DictionaryBlockMeta> blocks,
             BitmapGlobalIndexFormat.SerializedKey key) {
-        int index = findDictionaryBlockIndex(blocks, key);
+        int index = findSerializedDictionaryBlockIndex(blocks, key);
         return Math.max(index, 0);
     }
 
-    private BitmapGlobalIndexFormat.BlockInfo findBitmapBlock(
-            BitmapGlobalIndexFormat.SerializedKey key) {
+    private BitmapGlobalIndexFormat.BlockInfo findBitmapBlock(Object key) {
         List<BitmapGlobalIndexFormat.DictionaryBlockMeta> blocks = dictionaryBlocks.get();
         if (blocks.isEmpty()) {
             return null;
         }
 
-        int index = findDictionaryBlockIndex(blocks, key);
+        int index = findLogicalDictionaryBlockIndex(blocks, key);

Review Comment:
   That makes sense. Since #8276 has not shipped in a release, keeping version 1 for this pre-release format change is fine. Thanks for clarifying.



##########
paimon-common/src/main/java/org/apache/paimon/globalindex/bitmap/BitmapGlobalIndexWriter.java:
##########
@@ -110,12 +121,30 @@ public List<ResultEntry> finish() {
         return Collections.singletonList(new ResultEntry(fileName, rowCount, meta));
     }
 
-    private void updateMinMax(Object key) {
-        if (firstKey == null || comparator.compare(key, firstKey) < 0) {
-            firstKey = key;
+    private void flushCurrentBitmap() {
+        if (currentBitmap.isEmpty()) {
+            return;
+        }
+        try {
+            streamingWriter()
+                    .write(
+                            BitmapGlobalIndexFormat.SerializedKey.fromObject(
+                                    keySerializer, lastKey),
+                            currentBitmap);
+            currentBitmap.clear();
+        } catch (IOException e) {
+            throw new RuntimeException("Error in writing bitmap index files.", e);
         }
-        if (lastKey == null || comparator.compare(key, lastKey) > 0) {
-            lastKey = key;
+    }
+
+    private BitmapGlobalIndexFormat.StreamingWriter streamingWriter() throws IOException {
+        if (streamingWriter == null) {
+            fileName = fileWriter.newFileName(BitmapGlobalIndexerFactory.IDENTIFIER);
+            outputStream = fileWriter.newOutputStream(fileName);

Review Comment:
   Addressed by `9cd6406`: `SortedSingleColumnIndexWriter` now closes its active writer through try-with-resources, and Flink's `WriteIndexOperator` closes the active writer from its lifecycle cleanup. I also reran the focused core and Flink tests successfully. Thanks.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [core] Add LSM maintenance for sorted primary-key indexes [paimon]

Posted by "leaves12138 (via GitHub)" <gi...@apache.org>.
leaves12138 commented on code in PR #8613:
URL: https://github.com/apache/paimon/pull/8613#discussion_r3576959629


##########
paimon-common/src/main/java/org/apache/paimon/globalindex/bitmap/BitmapGlobalIndexWriter.java:
##########
@@ -76,10 +81,21 @@ public void write(@Nullable Object key, long relativeRowId) {
         }
 
         nonNullRows.add(relativeRowId);
-        updateMinMax(key);
-        BitmapGlobalIndexFormat.SerializedKey serializedKey =
-                BitmapGlobalIndexFormat.SerializedKey.fromObject(keySerializer, key);
-        bitmaps.computeIfAbsent(serializedKey, k -> new RoaringNavigableMap64()).add(relativeRowId);
+        if (lastKey != null) {
+            int comparison = comparator.compare(key, lastKey);
+            if (comparison < 0) {

Review Comment:
   `BitmapGlobalIndexer.createWriter()` is also used by `GenericIndexTopoBuilder` and Spark's `DefaultGlobalIndexBuilder`; both feed rows in row-id/data-file order, not index-key order. This new monotonicity requirement therefore breaks existing bitmap global-index creation for ordinary unsorted columns (for example, the Spark bitmap test writes `name_9` before `name_10`, which is descending lexicographically). Sorting the unit-test inputs hides that production regression. Could we keep the unordered writer for generic callers and use a separate streaming writer only for the PK-sorted path, or add an external sort to every production builder?



##########
paimon-common/src/main/java/org/apache/paimon/globalindex/bitmap/BitmapGlobalIndexWriter.java:
##########
@@ -110,12 +121,30 @@ public List<ResultEntry> finish() {
         return Collections.singletonList(new ResultEntry(fileName, rowCount, meta));
     }
 
-    private void updateMinMax(Object key) {
-        if (firstKey == null || comparator.compare(key, firstKey) < 0) {
-            firstKey = key;
+    private void flushCurrentBitmap() {
+        if (currentBitmap.isEmpty()) {
+            return;
+        }
+        try {
+            streamingWriter()
+                    .write(
+                            BitmapGlobalIndexFormat.SerializedKey.fromObject(
+                                    keySerializer, lastKey),
+                            currentBitmap);
+            currentBitmap.clear();
+        } catch (IOException e) {
+            throw new RuntimeException("Error in writing bitmap index files.", e);
         }
-        if (lastKey == null || comparator.compare(key, lastKey) > 0) {
-            lastKey = key;
+    }
+
+    private BitmapGlobalIndexFormat.StreamingWriter streamingWriter() throws IOException {
+        if (streamingWriter == null) {
+            fileName = fileWriter.newFileName(BitmapGlobalIndexerFactory.IDENTIFIER);
+            outputStream = fileWriter.newOutputStream(fileName);

Review Comment:
   Once this stream is opened, it is only closed by `finish()`. If row iteration, validation, or a later write fails, the writer is not `Closeable`, so the existing Flink/Spark cleanup hooks cannot close it; `PkSortedIndexFile` also only deletes the path. This can leak the stream or an object-store multipart upload. Could the writer implement a failure-safe close/abort path, and could the PK-sorted builder close it before deleting created files?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [core] Add LSM maintenance for sorted primary-key indexes [paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #8613:
URL: https://github.com/apache/paimon/pull/8613


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [core] Add LSM maintenance for sorted primary-key indexes [paimon]

Posted by "leaves12138 (via GitHub)" <gi...@apache.org>.
leaves12138 commented on code in PR #8613:
URL: https://github.com/apache/paimon/pull/8613#discussion_r3576760875


##########
paimon-api/src/main/java/org/apache/paimon/CoreOptions.java:
##########
@@ -4339,7 +4333,12 @@ public Options primaryKeyBitmapIndexOptions(String column) {
 
     private Options primaryKeySortedIndexOptions(
             String column, String optionFamily, String algorithmPrefix) {
-        Options resolved = new Options(toConfiguration().toMap());
+        Options resolved = new Options();

Review Comment:
   Filtering the resolved options down to only the algorithm prefix also drops the generic sort settings consumed by `PkSortedIndexBuilder`, including `write-buffer-size`, `page-size`, `local-sort.max-num-file-handles`, spill compression, and `write-buffer-spill.max-disk-size`. User-provided resource limits will therefore silently fall back to defaults. Could we preserve the general build options and remove only the range-sizing option, or pass build and algorithm options separately?



##########
paimon-core/src/main/java/org/apache/paimon/index/pksorted/PkSortedIndexFile.java:
##########
@@ -40,94 +38,86 @@
 import javax.annotation.Nullable;
 
 import java.io.IOException;
-import java.util.ArrayList;
-import java.util.Collections;
 import java.util.Iterator;
 import java.util.LinkedHashMap;
 import java.util.List;
 import java.util.Map;
 
 import static org.apache.paimon.utils.Preconditions.checkArgument;
 
-/** Builds source-backed BTree or Bitmap payloads for one physical data file. */
+/** Builds source-backed BTree or Bitmap payloads for ordered physical data files. */
 public class PkSortedIndexFile extends IndexFile {
 
     public PkSortedIndexFile(FileIO fileIO, IndexPathFactory pathFactory) {
         super(fileIO, pathFactory);
     }
 
-    public List<IndexFileMeta> build(
-            PrimaryKeyIndexSourceFile sourceFile,
+    public IndexFileMeta build(
+            List<PrimaryKeyIndexSourceFile> sourceFiles,
             DataField indexField,
             String indexType,
             Options indexOptions,
             Iterator<Entry> sortedEntries)
             throws IOException {
+        long sourceRowCount = 0;
+        for (PrimaryKeyIndexSourceFile sourceFile : sourceFiles) {
+            sourceRowCount = Math.addExact(sourceRowCount, sourceFile.rowCount());
+        }
         checkArgument(
-                sourceFile.rowCount() > 0,
-                "A sorted index group must reference at least one source row.");
+                sourceRowCount > 0, "A sorted index group must reference at least one source row.");
 
         TrackingFileWriter fileWriter = new TrackingFileWriter();
         boolean success = false;
         try {
-            long recordsPerRange =
-                    indexOptions.get(SortedIndexOptions.SORTED_INDEX_RECORDS_PER_RANGE);
-            SortedSingleColumnIndexWriter writer =
-                    new SortedSingleColumnIndexWriter(
-                            recordsPerRange,
-                            () -> createWriter(indexType, indexField, indexOptions, fileWriter));
+            GlobalIndexSingleColumnWriter writer =
+                    createWriter(indexType, indexField, indexOptions, fileWriter);
 
             long writtenRows = 0;
             while (sortedEntries.hasNext()) {
                 Entry entry = sortedEntries.next();
                 checkArgument(
-                        entry.localRowId >= 0 && entry.localRowId < sourceFile.rowCount(),
-                        "Local row id %s is outside source file %s row range [0, %s).",
-                        entry.localRowId,
-                        sourceFile.fileName(),
-                        sourceFile.rowCount());
-                writer.write(entry.value, entry.localRowId);
+                        entry.rowId >= 0 && entry.rowId < sourceRowCount,
+                        "Row id %s is outside sorted index group row range [0, %s).",
+                        entry.rowId,
+                        sourceRowCount);
+                writer.write(entry.value, entry.rowId);
                 writtenRows++;
             }
             checkArgument(
-                    writtenRows == sourceFile.rowCount(),
-                    "Sorted index input row count %s does not match source file %s row count %s.",
+                    writtenRows == sourceRowCount,
+                    "Sorted index input row count %s does not match source row count %s.",
                     writtenRows,
-                    sourceFile.fileName(),
-                    sourceFile.rowCount());
-
-            List<List<ResultEntry>> resultGroups = writer.finish();
-            List<IndexFileMeta> payloads = new ArrayList<>();
-            long payloadRows = 0;
-            byte[] sourceMeta = new PrimaryKeyIndexSourceMeta(sourceFile).serialize();
-            for (List<ResultEntry> resultGroup : resultGroups) {
-                for (ResultEntry result : resultGroup) {
-                    payloadRows = Math.addExact(payloadRows, result.rowCount());
-                    Path payloadPath = fileWriter.path(result.fileName());
-                    payloads.add(
-                            new IndexFileMeta(
-                                    indexType,
-                                    result.fileName(),
-                                    fileIO.getFileSize(payloadPath),
-                                    result.rowCount(),
-                                    new GlobalIndexMeta(
-                                            0,
-                                            sourceFile.rowCount() - 1,
-                                            indexField.id(),
-                                            null,
-                                            result.meta(),
-                                            sourceMeta),
-                                    pathFactory.isExternalPath() ? payloadPath.toString() : null));
-                }
-            }
+                    sourceRowCount);
+
+            List<ResultEntry> results = writer.finish();
+            checkArgument(
+                    results.size() == 1,

Review Comment:
   Requiring exactly one physical payload removes the size bound for sorted primary-key index builds. In particular, `BitmapGlobalIndexWriter` materializes all key bitmaps for the whole merged LSM group in memory, while groups can continue growing across levels. This can cause OOMs or very large index files. Could one logical group still contain multiple physical payloads using group-global row IDs, or could we enforce an explicit row/file-size bound?



##########
paimon-core/src/main/java/org/apache/paimon/table/source/PrimaryKeySortedIndexScan.java:
##########
@@ -224,47 +242,459 @@ static EvaluatedPlan evaluate(
             }
         }
 
+        Map<PkSortedIndexGroup, SharedGlobalIndexReader> sharedReaders = new IdentityHashMap<>();
         List<EvaluatedFile> files = new ArrayList<>();
-        for (FilePlan file : plan.files()) {
-            GlobalIndexEvaluator evaluator =
-                    new GlobalIndexEvaluator(
-                            rowType,
-                            fieldId -> {
-                                PrimaryKeyIndexDefinition definition =
-                                        definitionsByField.get(fieldId);
-                                Optional<PkSortedIndexGroup> group = file.group(fieldId);
-                                if (definition == null || !group.isPresent()) {
-                                    return Collections.emptyList();
-                                }
-                                return Collections.singletonList(
-                                        readerFactory.create(
-                                                file, definition, group.get().payloads()));
-                            });
-            Optional<GlobalIndexResult> result;
-            try {
-                result = evaluator.evaluate(predicate);
-            } catch (RuntimeException e) {
-                rethrowIfInterrupted(e);
-                LOG.warn(
-                        "Failed to evaluate primary-key sorted index for data file {}; "
-                                + "falling back to a raw scan for this file.",
-                        file.dataFile().fileName(),
-                        e);
-                result = Optional.empty();
-            } finally {
-                evaluator.close();
+        try {
+            for (FilePlan file : plan.files()) {
+                GlobalIndexEvaluator evaluator =
+                        new GlobalIndexEvaluator(
+                                rowType,
+                                fieldId -> {
+                                    PrimaryKeyIndexDefinition definition =
+                                            definitionsByField.get(fieldId);
+                                    Optional<PkSortedIndexGroup> group = file.group(fieldId);
+                                    if (definition == null || !group.isPresent()) {
+                                        return Collections.emptyList();
+                                    }
+                                    SharedGlobalIndexReader reader = sharedReaders.get(group.get());
+                                    if (reader == null) {
+                                        reader =
+                                                new SharedGlobalIndexReader(
+                                                        () ->
+                                                                readerFactory.create(
+                                                                        file,
+                                                                        definition,
+                                                                        group.get().payloads()));
+                                        sharedReaders.put(group.get(), reader);
+                                    }
+                                    return Collections.singletonList(
+                                            fileLocalReader(file, group.get(), reader));
+                                });
+                Optional<GlobalIndexResult> result;
+                try {
+                    result = evaluator.evaluate(predicate);
+                } catch (RuntimeException e) {
+                    rethrowIfInterrupted(e);
+                    LOG.warn(
+                            "Failed to evaluate primary-key sorted index for data file {}; "
+                                    + "falling back to a raw scan for this file.",
+                            file.dataFile().fileName(),
+                            e);
+                    result = Optional.empty();
+                } finally {
+                    evaluator.close();
+                }
+                files.add(new EvaluatedFile(file, result));
             }
-            files.add(new EvaluatedFile(file, result));
+        } finally {
+            IOUtils.closeAllQuietly(sharedReaders.values());
         }
         return new EvaluatedPlan(plan.snapshotId(), files);
     }
 
+    private static GlobalIndexReader fileLocalReader(
+            FilePlan file, PkSortedIndexGroup group, GlobalIndexReader reader) {
+        List<PrimaryKeyIndexSourceFile> sourceFiles = group.sourceFiles();
+        PrimaryKeyIndexSourceFile target =
+                new PrimaryKeyIndexSourceFile(
+                        file.dataFile().fileName(), file.dataFile().rowCount());
+        long totalRowCount = 0;
+        long from = -1;
+        long to = -1;
+        for (PrimaryKeyIndexSourceFile sourceFile : sourceFiles) {
+            long next = Math.addExact(totalRowCount, sourceFile.rowCount());
+            if (sourceFile.equals(target)) {
+                from = totalRowCount;
+                to = next;
+            }
+            totalRowCount = next;
+        }
+        checkArgument(
+                from >= 0,
+                "Data file %s is not covered by its sorted-index source group.",
+                file.dataFile().fileName());
+        return new FileLocalGlobalIndexReader(reader, from, to, totalRowCount);
+    }
+
     private static void rethrowIfInterrupted(RuntimeException exception) {
         if (Thread.currentThread().isInterrupted()) {
             throw exception;
         }
     }
 
+    /** Shares one source-group reader and its group-global query results across source files. */
+    private static final class SharedGlobalIndexReader implements GlobalIndexReader {
+
+        private final Supplier<GlobalIndexReader> readerFactory;
+        private final Map<QueryKey, CompletableFuture<Optional<GlobalIndexResult>>> results;
+
+        private GlobalIndexReader reader;
+        private RuntimeException readerFailure;
+
+        private SharedGlobalIndexReader(Supplier<GlobalIndexReader> readerFactory) {
+            this.readerFactory = readerFactory;
+            this.results = new ConcurrentHashMap<>();
+        }
+
+        @Override
+        public CompletableFuture<Optional<GlobalIndexResult>> visitIsNotNull(FieldRef fieldRef) {
+            return query(
+                    QueryKey.of(QueryOperation.IS_NOT_NULL, fieldRef),
+                    () -> reader().visitIsNotNull(fieldRef));
+        }
+
+        @Override
+        public CompletableFuture<Optional<GlobalIndexResult>> visitIsNull(FieldRef fieldRef) {
+            return query(
+                    QueryKey.of(QueryOperation.IS_NULL, fieldRef),
+                    () -> reader().visitIsNull(fieldRef));
+        }
+
+        @Override
+        public CompletableFuture<Optional<GlobalIndexResult>> visitStartsWith(
+                FieldRef fieldRef, Object literal) {
+            return query(
+                    QueryKey.of(QueryOperation.STARTS_WITH, fieldRef, literal),
+                    () -> reader().visitStartsWith(fieldRef, literal));
+        }
+
+        @Override
+        public CompletableFuture<Optional<GlobalIndexResult>> visitEndsWith(
+                FieldRef fieldRef, Object literal) {
+            return query(
+                    QueryKey.of(QueryOperation.ENDS_WITH, fieldRef, literal),
+                    () -> reader().visitEndsWith(fieldRef, literal));
+        }
+
+        @Override
+        public CompletableFuture<Optional<GlobalIndexResult>> visitContains(
+                FieldRef fieldRef, Object literal) {
+            return query(
+                    QueryKey.of(QueryOperation.CONTAINS, fieldRef, literal),
+                    () -> reader().visitContains(fieldRef, literal));
+        }
+
+        @Override
+        public CompletableFuture<Optional<GlobalIndexResult>> visitLike(
+                FieldRef fieldRef, Object literal) {
+            return query(
+                    QueryKey.of(QueryOperation.LIKE, fieldRef, literal),
+                    () -> reader().visitLike(fieldRef, literal));
+        }
+
+        @Override
+        public CompletableFuture<Optional<GlobalIndexResult>> visitLessThan(
+                FieldRef fieldRef, Object literal) {
+            return query(
+                    QueryKey.of(QueryOperation.LESS_THAN, fieldRef, literal),
+                    () -> reader().visitLessThan(fieldRef, literal));
+        }
+
+        @Override
+        public CompletableFuture<Optional<GlobalIndexResult>> visitGreaterOrEqual(
+                FieldRef fieldRef, Object literal) {
+            return query(
+                    QueryKey.of(QueryOperation.GREATER_OR_EQUAL, fieldRef, literal),
+                    () -> reader().visitGreaterOrEqual(fieldRef, literal));
+        }
+
+        @Override
+        public CompletableFuture<Optional<GlobalIndexResult>> visitNotEqual(
+                FieldRef fieldRef, Object literal) {
+            return query(
+                    QueryKey.of(QueryOperation.NOT_EQUAL, fieldRef, literal),
+                    () -> reader().visitNotEqual(fieldRef, literal));
+        }
+
+        @Override
+        public CompletableFuture<Optional<GlobalIndexResult>> visitLessOrEqual(
+                FieldRef fieldRef, Object literal) {
+            return query(
+                    QueryKey.of(QueryOperation.LESS_OR_EQUAL, fieldRef, literal),
+                    () -> reader().visitLessOrEqual(fieldRef, literal));
+        }
+
+        @Override
+        public CompletableFuture<Optional<GlobalIndexResult>> visitEqual(
+                FieldRef fieldRef, Object literal) {
+            return query(
+                    QueryKey.of(QueryOperation.EQUAL, fieldRef, literal),
+                    () -> reader().visitEqual(fieldRef, literal));
+        }
+
+        @Override
+        public CompletableFuture<Optional<GlobalIndexResult>> visitGreaterThan(
+                FieldRef fieldRef, Object literal) {
+            return query(
+                    QueryKey.of(QueryOperation.GREATER_THAN, fieldRef, literal),
+                    () -> reader().visitGreaterThan(fieldRef, literal));
+        }
+
+        @Override
+        public CompletableFuture<Optional<GlobalIndexResult>> visitIn(
+                FieldRef fieldRef, List<Object> literals) {
+            return query(
+                    QueryKey.ofLiterals(QueryOperation.IN, fieldRef, literals),
+                    () -> reader().visitIn(fieldRef, literals));
+        }
+
+        @Override
+        public CompletableFuture<Optional<GlobalIndexResult>> visitNotIn(
+                FieldRef fieldRef, List<Object> literals) {
+            return query(
+                    QueryKey.ofLiterals(QueryOperation.NOT_IN, fieldRef, literals),
+                    () -> reader().visitNotIn(fieldRef, literals));
+        }
+
+        @Override
+        public CompletableFuture<Optional<GlobalIndexResult>> visitBetween(
+                FieldRef fieldRef, Object from, Object to) {
+            return query(
+                    QueryKey.of(QueryOperation.BETWEEN, fieldRef, from, to),
+                    () -> reader().visitBetween(fieldRef, from, to));
+        }
+
+        @Override
+        public CompletableFuture<Optional<GlobalIndexResult>> visitNotBetween(
+                FieldRef fieldRef, Object from, Object to) {
+            return query(
+                    QueryKey.of(QueryOperation.NOT_BETWEEN, fieldRef, from, to),
+                    () -> reader().visitNotBetween(fieldRef, from, to));
+        }
+
+        private CompletableFuture<Optional<GlobalIndexResult>> query(
+                QueryKey key,
+                Supplier<CompletableFuture<Optional<GlobalIndexResult>>> querySupplier) {
+            return results.computeIfAbsent(
+                    key,
+                    ignored -> {
+                        try {
+                            return checkNotNull(querySupplier.get());
+                        } catch (RuntimeException e) {
+                            CompletableFuture<Optional<GlobalIndexResult>> failed =
+                                    new CompletableFuture<>();
+                            failed.completeExceptionally(e);
+                            return failed;
+                        }
+                    });
+        }
+
+        private synchronized GlobalIndexReader reader() {
+            if (reader != null) {
+                return reader;
+            }
+            if (readerFailure != null) {
+                throw readerFailure;
+            }
+            try {
+                reader = checkNotNull(readerFactory.get());
+                return reader;
+            } catch (RuntimeException e) {
+                readerFailure = e;
+                throw e;
+            }
+        }
+
+        @Override
+        public synchronized void close() throws IOException {
+            if (reader != null) {
+                GlobalIndexReader readerToClose = reader;
+                reader = null;
+                readerToClose.close();
+            }
+        }
+    }
+
+    private enum QueryOperation {
+        IS_NOT_NULL,
+        IS_NULL,
+        STARTS_WITH,
+        ENDS_WITH,
+        CONTAINS,
+        LIKE,
+        LESS_THAN,
+        GREATER_OR_EQUAL,
+        NOT_EQUAL,
+        LESS_OR_EQUAL,
+        EQUAL,
+        GREATER_THAN,
+        IN,
+        NOT_IN,
+        BETWEEN,
+        NOT_BETWEEN
+    }
+
+    private static final class QueryKey {
+
+        private final QueryOperation operation;
+        private final FieldRef fieldRef;
+        private final List<Object> literals;
+
+        private QueryKey(QueryOperation operation, FieldRef fieldRef, List<Object> literals) {
+            this.operation = operation;
+            this.fieldRef = fieldRef;
+            this.literals = Collections.unmodifiableList(new ArrayList<>(literals));
+        }
+
+        private static QueryKey of(
+                QueryOperation operation, FieldRef fieldRef, Object... literals) {
+            return new QueryKey(operation, fieldRef, Arrays.asList(literals));
+        }
+
+        private static QueryKey ofLiterals(
+                QueryOperation operation, FieldRef fieldRef, List<Object> literals) {
+            return new QueryKey(operation, fieldRef, literals);
+        }
+
+        @Override
+        public boolean equals(Object o) {
+            if (this == o) {
+                return true;
+            }
+            if (!(o instanceof QueryKey)) {
+                return false;
+            }
+            QueryKey queryKey = (QueryKey) o;
+            return operation == queryKey.operation
+                    && Objects.equals(fieldRef, queryKey.fieldRef)
+                    && Objects.equals(literals, queryKey.literals);
+        }
+
+        @Override
+        public int hashCode() {
+            return Objects.hash(operation, fieldRef, literals);
+        }
+    }
+
+    /** Restricts merged source-group ordinals to one source file's local row positions. */
+    private static final class FileLocalGlobalIndexReader implements GlobalIndexReader {
+
+        private final GlobalIndexReader wrapped;
+        private final long from;
+        private final long to;
+        private final long totalRowCount;
+
+        private FileLocalGlobalIndexReader(
+                GlobalIndexReader wrapped, long from, long to, long totalRowCount) {
+            this.wrapped = wrapped;
+            this.from = from;
+            this.to = to;
+            this.totalRowCount = totalRowCount;
+        }
+
+        @Override
+        public CompletableFuture<Optional<GlobalIndexResult>> visitIsNotNull(FieldRef fieldRef) {
+            return wrapped.visitIsNotNull(fieldRef).thenApply(this::toFileLocal);
+        }
+
+        @Override
+        public CompletableFuture<Optional<GlobalIndexResult>> visitIsNull(FieldRef fieldRef) {
+            return wrapped.visitIsNull(fieldRef).thenApply(this::toFileLocal);
+        }
+
+        @Override
+        public CompletableFuture<Optional<GlobalIndexResult>> visitStartsWith(
+                FieldRef fieldRef, Object literal) {
+            return wrapped.visitStartsWith(fieldRef, literal).thenApply(this::toFileLocal);
+        }
+
+        @Override
+        public CompletableFuture<Optional<GlobalIndexResult>> visitEndsWith(
+                FieldRef fieldRef, Object literal) {
+            return wrapped.visitEndsWith(fieldRef, literal).thenApply(this::toFileLocal);
+        }
+
+        @Override
+        public CompletableFuture<Optional<GlobalIndexResult>> visitContains(
+                FieldRef fieldRef, Object literal) {
+            return wrapped.visitContains(fieldRef, literal).thenApply(this::toFileLocal);
+        }
+
+        @Override
+        public CompletableFuture<Optional<GlobalIndexResult>> visitLike(
+                FieldRef fieldRef, Object literal) {
+            return wrapped.visitLike(fieldRef, literal).thenApply(this::toFileLocal);
+        }
+
+        @Override
+        public CompletableFuture<Optional<GlobalIndexResult>> visitLessThan(
+                FieldRef fieldRef, Object literal) {
+            return wrapped.visitLessThan(fieldRef, literal).thenApply(this::toFileLocal);
+        }
+
+        @Override
+        public CompletableFuture<Optional<GlobalIndexResult>> visitGreaterOrEqual(
+                FieldRef fieldRef, Object literal) {
+            return wrapped.visitGreaterOrEqual(fieldRef, literal).thenApply(this::toFileLocal);
+        }
+
+        @Override
+        public CompletableFuture<Optional<GlobalIndexResult>> visitNotEqual(
+                FieldRef fieldRef, Object literal) {
+            return wrapped.visitNotEqual(fieldRef, literal).thenApply(this::toFileLocal);
+        }
+
+        @Override
+        public CompletableFuture<Optional<GlobalIndexResult>> visitLessOrEqual(
+                FieldRef fieldRef, Object literal) {
+            return wrapped.visitLessOrEqual(fieldRef, literal).thenApply(this::toFileLocal);
+        }
+
+        @Override
+        public CompletableFuture<Optional<GlobalIndexResult>> visitEqual(
+                FieldRef fieldRef, Object literal) {
+            return wrapped.visitEqual(fieldRef, literal).thenApply(this::toFileLocal);
+        }
+
+        @Override
+        public CompletableFuture<Optional<GlobalIndexResult>> visitGreaterThan(
+                FieldRef fieldRef, Object literal) {
+            return wrapped.visitGreaterThan(fieldRef, literal).thenApply(this::toFileLocal);
+        }
+
+        @Override
+        public CompletableFuture<Optional<GlobalIndexResult>> visitIn(
+                FieldRef fieldRef, List<Object> literals) {
+            return wrapped.visitIn(fieldRef, literals).thenApply(this::toFileLocal);
+        }
+
+        @Override
+        public CompletableFuture<Optional<GlobalIndexResult>> visitNotIn(
+                FieldRef fieldRef, List<Object> literals) {
+            return wrapped.visitNotIn(fieldRef, literals).thenApply(this::toFileLocal);
+        }
+
+        @Override
+        public CompletableFuture<Optional<GlobalIndexResult>> visitBetween(
+                FieldRef fieldRef, Object from, Object to) {
+            return wrapped.visitBetween(fieldRef, from, to).thenApply(this::toFileLocal);
+        }
+
+        @Override
+        public CompletableFuture<Optional<GlobalIndexResult>> visitNotBetween(
+                FieldRef fieldRef, Object from, Object to) {
+            return wrapped.visitNotBetween(fieldRef, from, to).thenApply(this::toFileLocal);
+        }
+
+        private Optional<GlobalIndexResult> toFileLocal(Optional<GlobalIndexResult> result) {
+            if (!result.isPresent() || (from == 0 && to == totalRowCount)) {
+                return result;
+            }
+            RoaringNavigableMap64 positions = new RoaringNavigableMap64();
+            for (long position : result.get().results()) {

Review Comment:
   Sharing the underlying query fixes the repeated index reads, but every source file still traverses the complete group-global result here. For a group with `N` source files and a broad predicate returning `M` positions, evaluation remains `O(N * M)` and allocates `N` file-local bitmaps. Could the cached result be partitioned across all source ranges once per query, or use bitmap range slicing to avoid repeatedly scanning the full result?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [core] Add LSM maintenance for sorted primary-key indexes [paimon]

Posted by "leaves12138 (via GitHub)" <gi...@apache.org>.
leaves12138 commented on code in PR #8613:
URL: https://github.com/apache/paimon/pull/8613#discussion_r3577468057


##########
paimon-common/src/main/java/org/apache/paimon/globalindex/bitmap/BitmapIndexReader.java:
##########
@@ -346,26 +344,27 @@ private RoaringNavigableMap64 scanSerializedDictionary(
     private int firstPossibleDictionaryBlock(
             List<BitmapGlobalIndexFormat.DictionaryBlockMeta> blocks,
             BitmapGlobalIndexFormat.SerializedKey key) {
-        int index = findDictionaryBlockIndex(blocks, key);
+        int index = findSerializedDictionaryBlockIndex(blocks, key);
         return Math.max(index, 0);
     }
 
-    private BitmapGlobalIndexFormat.BlockInfo findBitmapBlock(
-            BitmapGlobalIndexFormat.SerializedKey key) {
+    private BitmapGlobalIndexFormat.BlockInfo findBitmapBlock(Object key) {
         List<BitmapGlobalIndexFormat.DictionaryBlockMeta> blocks = dictionaryBlocks.get();
         if (blocks.isEmpty()) {
             return null;
         }
 
-        int index = findDictionaryBlockIndex(blocks, key);
+        int index = findLogicalDictionaryBlockIndex(blocks, key);

Review Comment:
   Existing version-1 bitmap files were written with dictionary blocks sorted by serialized key bytes, while this lookup now assumes logical key order. The new streaming writer also still writes `VERSION = 1`, so old and new files with different ordering semantics are indistinguishable. For numeric keys this can return false negatives; for example, an old v1 file with separate blocks for `0` and `-1` is ordered as `[0, -1]` by serialized bytes, but this binary search assumes `[-1, 0]`. Could we retain the version-1 serialized-order read path and write the new logical-order format with a new version, as in the earlier implementation?



##########
paimon-common/src/main/java/org/apache/paimon/globalindex/bitmap/BitmapGlobalIndexWriter.java:
##########
@@ -110,12 +121,30 @@ public List<ResultEntry> finish() {
         return Collections.singletonList(new ResultEntry(fileName, rowCount, meta));
     }
 
-    private void updateMinMax(Object key) {
-        if (firstKey == null || comparator.compare(key, firstKey) < 0) {
-            firstKey = key;
+    private void flushCurrentBitmap() {
+        if (currentBitmap.isEmpty()) {
+            return;
+        }
+        try {
+            streamingWriter()
+                    .write(
+                            BitmapGlobalIndexFormat.SerializedKey.fromObject(
+                                    keySerializer, lastKey),
+                            currentBitmap);
+            currentBitmap.clear();
+        } catch (IOException e) {
+            throw new RuntimeException("Error in writing bitmap index files.", e);
         }
-        if (lastKey == null || comparator.compare(key, lastKey) > 0) {
-            lastKey = key;
+    }
+
+    private BitmapGlobalIndexFormat.StreamingWriter streamingWriter() throws IOException {
+        if (streamingWriter == null) {
+            fileName = fileWriter.newFileName(BitmapGlobalIndexerFactory.IDENTIFIER);
+            outputStream = fileWriter.newOutputStream(fileName);

Review Comment:
   The PK-sorted path is covered now, but the sorted Flink/Spark paths still do not propagate this cleanup. Spark wraps the writer in `SortedSingleColumnIndexWriter`, which is not closeable and leaves `currentWriter` open if iteration, `write`, or `finish` fails. Flink's `WriteIndexOperator` similarly keeps `currentWriter` without closing it from an operator failure/close path. Could both paths close the active writer from a failure-safe `finally`/lifecycle hook?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org