You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "baiyangtx (via GitHub)" <gi...@apache.org> on 2026/05/15 08:11:48 UTC

[PR] feat(core): support sort manifest entries by partition [paimon]

baiyangtx opened a new pull request, #7866:
URL: https://github.com/apache/paimon/pull/7866

   ### Purpose
   
   Manifest entries are currently written in arrival order, which scatters entries belonging
   to the same partition across multiple manifest files. This leads to:
   
   - Inefficient partition-pruning during reads — the reader must scan many manifest files
     to find all entries for a single partition.
   - Suboptimal data locality when compaction reorganizes files.
   
   This PR introduces manifest entry sorting by partition, so that entries for the same
   partition are clustered together within manifest files.
   
   ### Changes
   
   **New configuration options:**
   
   | Option | Default | Description |
   |--------|---------|-------------|
   | `manifest.merge.sorted` | `true` | Sort entries by partition during manifest full compaction |
   | `manifest.merge.sort-on-commit` | `false` | Sort entries by partition during manifest full merge in commit |
   | `manifest.delta.sorted` | `true` | Sort entries by partition when writing manifest delta |
   
   **Core implementation:**
   
   - `ManifestFileMerger`: Introduces `mergeSortedByPartition()` and `mergeUnsorted()`.
     When `manifest.merge.sorted` is enabled, entries are collected into a
     `BinaryExternalSortBuffer` (with spill-to-disk support), then written in
     partition-major order: `(partition, bucket, level)`.
   - `ManifestFileMerger.createManifestEntryComparator()`: Comparator used for
     sorting delta manifests, falling back to pure-Java comparison when codegen
     is unavailable.
   - `FileStoreCommitImpl`: Wires sort parameters into all three paths —
     commit manifest merge, delta file writing, and manifest compaction.
   
   ### Tests
   
   ```
   # Manifest merge with sorting
   mvn -pl paimon-core -am -DfailIfNoTests=false \
       -Dtest=ManifestFileMetaTest test
   
   # No-partition edge case
   mvn -pl paimon-core -am -DfailIfNoTests=false \
       -Dtest=NoPartitionManifestFileMetaTest test
   
   # Spark compact procedure
   mvn -pl paimon-spark/paimon-spark-ut -am -DfailIfNoTests=false \
       -Dtest=CompactProcedureTestBase test
   ```
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(core): support sort manifest entries by partition [paimon]

Posted by "baiyangtx (via GitHub)" <gi...@apache.org>.
baiyangtx commented on PR #7866:
URL: https://github.com/apache/paimon/pull/7866#issuecomment-4515909075

   > Hi @baiyangtx @Aitozi , please take a look to #7842 , I didn't look closely, maybe the abilities of these two PR implementations are the same.
   
   Thanks @umi for working on this too. My PR #7866 takes a different approach — sorting inline during the existing full compaction path, rather than a separate post-commit sort pipeline. Key differences:
   
   - **No new compact type** — sorting happens inside `ManifestFileMerger.tryFullCompaction`, reusing the existing full compaction trigger and lifecycle. No additional manifest scans or rewrite passes.
   - **Memory-safe with external sort** — uses `BinaryExternalSortBuffer` (already battle-tested across Paimon). When entries exceed `manifest.merge.sort.buffer`, it automatically spills to disk. No risk of OOM regardless of manifest count.
   - **Finer-grained config** — separate toggles for merge compaction, commit-time merge, and delta writing, instead of a single switch.
   
   Happy to align with the community's direction, but I believe the inline approach is simpler and less invasive.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(core): support sort manifest entries by partition [paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #7866:
URL: https://github.com/apache/paimon/pull/7866#issuecomment-4514294549

   Hi @baiyangtx @Aitozi , please take a look to https://github.com/apache/paimon/pull/7842 , I didn't look closely, maybe the abilities of these two PR implementations are the same.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(core): support sort manifest entries by partition [paimon]

Posted by "baiyangtx (via GitHub)" <gi...@apache.org>.
baiyangtx commented on PR #7866:
URL: https://github.com/apache/paimon/pull/7866#issuecomment-4532036530

   Thanks @JingsongLi for the thorough review. I've addressed most of the feedback:
   
   1. **ADD filter regression** — Restored `FileEntry.addFilter()` in both `mergeUnsorted()` and `mergeSortedByPartition()`. DELETE entries are now filtered at the IO level as before.
   
   2. **IOManager hard-coded tmpdir** — Added `@Nullable IOManager` parameter to `tryFullCompaction` and `merge`. Falls back to `System.getProperty("java.io.tmpdir")` when null is passed, so existing callers are unaffected.
   
   3. **Inconsistent sort key** — Added `fileName` as the 5th field in the sort row, serving as a deterministic tiebreaker after (partition, bucket, level).
   
   4. **manifest.delta.sorted default** — Changed from `true` to `false`, consistent with `manifest.merge.sort-on-commit`.
   
   5. **Test assertion for internal order** — Reverted `containsExactlyInAnyOrderElementsOf` back to `containsExactlyElementsOf` for the `testLocalSortClusterUnpartitionedTable` internal-order assertion. With parallelism=1 and a single output file, the order is deterministic.
   
   6. **Minor** — Renamed `partitionRmpr` → `partitionCmp`.
   
   **For discussion — parallel manifest reads (#2)**: The original `sequentialBatchedExecute` was removed because the sorting path needs all entries collected before spilling to external sort. I'd appreciate guidance on whether to restore parallel reads for the non-sorted path only, or if the simplicity of unified sequential reads is acceptable here.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(core): support sort manifest entries by partition [paimon]

Posted by "discivigour (via GitHub)" <gi...@apache.org>.
discivigour commented on code in PR #7866:
URL: https://github.com/apache/paimon/pull/7866#discussion_r3301218562


##########
paimon-core/src/main/java/org/apache/paimon/operation/ManifestFileMerger.java:
##########
@@ -272,27 +350,276 @@ public static Optional<List<ManifestFileMeta>> tryFullCompaction(
         return Optional.of(result);
     }
 
-    private static FullCompactionReadResult readForFullCompaction(
-            ManifestFileMeta file,
-            ManifestFile manifestFile,
+    private static final NormalizedKeyComputer NO_NORMALIZED_KEY_COMPUTER =
+            new NormalizedKeyComputer() {
+                @Override
+                public void putKey(InternalRow record, MemorySegment target, int offset) {
+                    // no-op
+                }
+
+                @Override
+                public int compareKey(
+                        MemorySegment segI, int offsetI, MemorySegment segJ, int offsetJ) {
+                    return 0;
+                }
+
+                @Override
+                public void swapKey(
+                        MemorySegment segI, int offsetI, MemorySegment segJ, int offsetJ) {
+                    // no-op
+                }
+
+                @Override
+                public int getNumKeyBytes() {
+                    return 0;
+                }
+
+                @Override
+                public boolean isKeyFullyDetermines() {
+                    return false;
+                }
+
+                @Override
+                public boolean invertKey() {
+                    return false;
+                }
+            };
+
+    private static int mergeUnsorted(
+            List<ManifestFileMeta> toBeMerged,
             Filter<ManifestFileMeta> mustChange,
-            Set<FileEntry.Identifier> deleteEntries) {
-        List<ManifestEntry> entries = new ArrayList<>();
-        boolean requireChange = mustChange.test(file);
-        for (ManifestEntry entry :
-                manifestFile.read(
-                        file.fileName(),
-                        file.fileSize(),
-                        FileEntry.addFilter(),
-                        Filter.alwaysTrue())) {
-            if (deleteEntries.contains(entry.identifier())) {
-                requireChange = true;
+            Set<FileEntry.Identifier> deleteEntries,
+            ManifestFile manifestFile,
+            RollingFileWriter<ManifestEntry, ManifestFileMeta> writer,
+            List<ManifestFileMeta> result)
+            throws Exception {
+        int actualRewriteCount = 0;
+        for (ManifestFileMeta file : toBeMerged) {
+            List<ManifestEntry> entries = new ArrayList<>();
+            boolean requireChange = mustChange.test(file);
+            for (ManifestEntry entry : manifestFile.read(file.fileName(), file.fileSize())) {
+                if (entry.kind() == FileKind.DELETE) {
+                    continue;
+                }
+
+                if (deleteEntries.contains(entry.identifier())) {
+                    requireChange = true;
+                } else {
+                    entries.add(entry);
+                }
+            }
+
+            if (requireChange) {
+                writer.write(entries);
+                actualRewriteCount++;
             } else {
-                entries.add(entry);
+                result.add(file);
             }
         }
+        return actualRewriteCount;
+    }
+
+    private static int mergeSortedByPartition(
+            List<ManifestFileMeta> toBeMerged,

Review Comment:
   What does your 'inline compaction' mean? My method has done the following things:
   1. Incrementally sort the newly arrived data
   2. Sort the existing data



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(core): support sort manifest entries by partition [paimon]

Posted by "discivigour (via GitHub)" <gi...@apache.org>.
discivigour commented on code in PR #7866:
URL: https://github.com/apache/paimon/pull/7866#discussion_r3292018156


##########
paimon-core/src/main/java/org/apache/paimon/operation/ManifestFileMerger.java:
##########
@@ -272,27 +350,276 @@ public static Optional<List<ManifestFileMeta>> tryFullCompaction(
         return Optional.of(result);
     }
 
-    private static FullCompactionReadResult readForFullCompaction(
-            ManifestFileMeta file,
-            ManifestFile manifestFile,
+    private static final NormalizedKeyComputer NO_NORMALIZED_KEY_COMPUTER =
+            new NormalizedKeyComputer() {
+                @Override
+                public void putKey(InternalRow record, MemorySegment target, int offset) {
+                    // no-op
+                }
+
+                @Override
+                public int compareKey(
+                        MemorySegment segI, int offsetI, MemorySegment segJ, int offsetJ) {
+                    return 0;
+                }
+
+                @Override
+                public void swapKey(
+                        MemorySegment segI, int offsetI, MemorySegment segJ, int offsetJ) {
+                    // no-op
+                }
+
+                @Override
+                public int getNumKeyBytes() {
+                    return 0;
+                }
+
+                @Override
+                public boolean isKeyFullyDetermines() {
+                    return false;
+                }
+
+                @Override
+                public boolean invertKey() {
+                    return false;
+                }
+            };
+
+    private static int mergeUnsorted(
+            List<ManifestFileMeta> toBeMerged,
             Filter<ManifestFileMeta> mustChange,
-            Set<FileEntry.Identifier> deleteEntries) {
-        List<ManifestEntry> entries = new ArrayList<>();
-        boolean requireChange = mustChange.test(file);
-        for (ManifestEntry entry :
-                manifestFile.read(
-                        file.fileName(),
-                        file.fileSize(),
-                        FileEntry.addFilter(),
-                        Filter.alwaysTrue())) {
-            if (deleteEntries.contains(entry.identifier())) {
-                requireChange = true;
+            Set<FileEntry.Identifier> deleteEntries,
+            ManifestFile manifestFile,
+            RollingFileWriter<ManifestEntry, ManifestFileMeta> writer,
+            List<ManifestFileMeta> result)
+            throws Exception {
+        int actualRewriteCount = 0;
+        for (ManifestFileMeta file : toBeMerged) {
+            List<ManifestEntry> entries = new ArrayList<>();
+            boolean requireChange = mustChange.test(file);
+            for (ManifestEntry entry : manifestFile.read(file.fileName(), file.fileSize())) {
+                if (entry.kind() == FileKind.DELETE) {
+                    continue;
+                }
+
+                if (deleteEntries.contains(entry.identifier())) {
+                    requireChange = true;
+                } else {
+                    entries.add(entry);
+                }
+            }
+
+            if (requireChange) {
+                writer.write(entries);
+                actualRewriteCount++;
             } else {
-                entries.add(entry);
+                result.add(file);
             }
         }
+        return actualRewriteCount;
+    }
+
+    private static int mergeSortedByPartition(
+            List<ManifestFileMeta> toBeMerged,

Review Comment:
   Thanks @baiyangtx for working on this too. Can this code sort the out-of-order large file manifest that already exists in the table? If the user wants to enable this parameter halfway, it seems that the feature won't work.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(core): support sort manifest entries by partition [paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi closed pull request #7866: feat(core): support sort manifest entries by partition
URL: https://github.com/apache/paimon/pull/7866


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(core): support sort manifest entries by partition [paimon]

Posted by "discivigour (via GitHub)" <gi...@apache.org>.
discivigour commented on code in PR #7866:
URL: https://github.com/apache/paimon/pull/7866#discussion_r3292018156


##########
paimon-core/src/main/java/org/apache/paimon/operation/ManifestFileMerger.java:
##########
@@ -272,27 +350,276 @@ public static Optional<List<ManifestFileMeta>> tryFullCompaction(
         return Optional.of(result);
     }
 
-    private static FullCompactionReadResult readForFullCompaction(
-            ManifestFileMeta file,
-            ManifestFile manifestFile,
+    private static final NormalizedKeyComputer NO_NORMALIZED_KEY_COMPUTER =
+            new NormalizedKeyComputer() {
+                @Override
+                public void putKey(InternalRow record, MemorySegment target, int offset) {
+                    // no-op
+                }
+
+                @Override
+                public int compareKey(
+                        MemorySegment segI, int offsetI, MemorySegment segJ, int offsetJ) {
+                    return 0;
+                }
+
+                @Override
+                public void swapKey(
+                        MemorySegment segI, int offsetI, MemorySegment segJ, int offsetJ) {
+                    // no-op
+                }
+
+                @Override
+                public int getNumKeyBytes() {
+                    return 0;
+                }
+
+                @Override
+                public boolean isKeyFullyDetermines() {
+                    return false;
+                }
+
+                @Override
+                public boolean invertKey() {
+                    return false;
+                }
+            };
+
+    private static int mergeUnsorted(
+            List<ManifestFileMeta> toBeMerged,
             Filter<ManifestFileMeta> mustChange,
-            Set<FileEntry.Identifier> deleteEntries) {
-        List<ManifestEntry> entries = new ArrayList<>();
-        boolean requireChange = mustChange.test(file);
-        for (ManifestEntry entry :
-                manifestFile.read(
-                        file.fileName(),
-                        file.fileSize(),
-                        FileEntry.addFilter(),
-                        Filter.alwaysTrue())) {
-            if (deleteEntries.contains(entry.identifier())) {
-                requireChange = true;
+            Set<FileEntry.Identifier> deleteEntries,
+            ManifestFile manifestFile,
+            RollingFileWriter<ManifestEntry, ManifestFileMeta> writer,
+            List<ManifestFileMeta> result)
+            throws Exception {
+        int actualRewriteCount = 0;
+        for (ManifestFileMeta file : toBeMerged) {
+            List<ManifestEntry> entries = new ArrayList<>();
+            boolean requireChange = mustChange.test(file);
+            for (ManifestEntry entry : manifestFile.read(file.fileName(), file.fileSize())) {
+                if (entry.kind() == FileKind.DELETE) {
+                    continue;
+                }
+
+                if (deleteEntries.contains(entry.identifier())) {
+                    requireChange = true;
+                } else {
+                    entries.add(entry);
+                }
+            }
+
+            if (requireChange) {
+                writer.write(entries);
+                actualRewriteCount++;
             } else {
-                entries.add(entry);
+                result.add(file);
             }
         }
+        return actualRewriteCount;
+    }
+
+    private static int mergeSortedByPartition(
+            List<ManifestFileMeta> toBeMerged,

Review Comment:
   Can this code sort the out-of-order large file manifest that already exists in the table? If the user wants to enable this parameter halfway, it seems that the feature won't work.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(core): support sort manifest entries by partition [paimon]

Posted by "Aitozi (via GitHub)" <gi...@apache.org>.
Aitozi commented on PR #7866:
URL: https://github.com/apache/paimon/pull/7866#issuecomment-4509046385

   CC @JingsongLi for another look


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(core): support sort manifest entries by partition [paimon]

Posted by "baiyangtx (via GitHub)" <gi...@apache.org>.
baiyangtx commented on code in PR #7866:
URL: https://github.com/apache/paimon/pull/7866#discussion_r3295861538


##########
paimon-core/src/main/java/org/apache/paimon/operation/ManifestFileMerger.java:
##########
@@ -272,27 +350,276 @@ public static Optional<List<ManifestFileMeta>> tryFullCompaction(
         return Optional.of(result);
     }
 
-    private static FullCompactionReadResult readForFullCompaction(
-            ManifestFileMeta file,
-            ManifestFile manifestFile,
+    private static final NormalizedKeyComputer NO_NORMALIZED_KEY_COMPUTER =
+            new NormalizedKeyComputer() {
+                @Override
+                public void putKey(InternalRow record, MemorySegment target, int offset) {
+                    // no-op
+                }
+
+                @Override
+                public int compareKey(
+                        MemorySegment segI, int offsetI, MemorySegment segJ, int offsetJ) {
+                    return 0;
+                }
+
+                @Override
+                public void swapKey(
+                        MemorySegment segI, int offsetI, MemorySegment segJ, int offsetJ) {
+                    // no-op
+                }
+
+                @Override
+                public int getNumKeyBytes() {
+                    return 0;
+                }
+
+                @Override
+                public boolean isKeyFullyDetermines() {
+                    return false;
+                }
+
+                @Override
+                public boolean invertKey() {
+                    return false;
+                }
+            };
+
+    private static int mergeUnsorted(
+            List<ManifestFileMeta> toBeMerged,
             Filter<ManifestFileMeta> mustChange,
-            Set<FileEntry.Identifier> deleteEntries) {
-        List<ManifestEntry> entries = new ArrayList<>();
-        boolean requireChange = mustChange.test(file);
-        for (ManifestEntry entry :
-                manifestFile.read(
-                        file.fileName(),
-                        file.fileSize(),
-                        FileEntry.addFilter(),
-                        Filter.alwaysTrue())) {
-            if (deleteEntries.contains(entry.identifier())) {
-                requireChange = true;
+            Set<FileEntry.Identifier> deleteEntries,
+            ManifestFile manifestFile,
+            RollingFileWriter<ManifestEntry, ManifestFileMeta> writer,
+            List<ManifestFileMeta> result)
+            throws Exception {
+        int actualRewriteCount = 0;
+        for (ManifestFileMeta file : toBeMerged) {
+            List<ManifestEntry> entries = new ArrayList<>();
+            boolean requireChange = mustChange.test(file);
+            for (ManifestEntry entry : manifestFile.read(file.fileName(), file.fileSize())) {
+                if (entry.kind() == FileKind.DELETE) {
+                    continue;
+                }
+
+                if (deleteEntries.contains(entry.identifier())) {
+                    requireChange = true;
+                } else {
+                    entries.add(entry);
+                }
+            }
+
+            if (requireChange) {
+                writer.write(entries);
+                actualRewriteCount++;
             } else {
-                entries.add(entry);
+                result.add(file);
             }
         }
+        return actualRewriteCount;
+    }
+
+    private static int mergeSortedByPartition(
+            List<ManifestFileMeta> toBeMerged,

Review Comment:
   @discivigour Good point — my approach indeed doesn't handle existing out-of-order manifests. I think our two PRs are actually complementary: yours handles the bootstrap/rewrite path for historical manifests, mine maintains sorted order going forward via inline compaction. I'd like to collaborate and align the configuration surface first so users see one coherent "manifest sort" feature rather than two separate mechanisms. WDYT?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(core): support sort manifest entries by partition [paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #7866:
URL: https://github.com/apache/paimon/pull/7866#issuecomment-4619403304

   https://github.com/apache/paimon/pull/7842 has been merged, if you feel there is still room for improvement, please submit a separate PR. Thanks @baiyangtx and @Aitozi 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(core): support sort manifest entries by partition [paimon]

Posted by "discivigour (via GitHub)" <gi...@apache.org>.
discivigour commented on PR #7866:
URL: https://github.com/apache/paimon/pull/7866#issuecomment-4550992166

   Thank you for sharing this feature. I believe my feature can already support the following functions well: 
   1. The LSM Tree was introduced to achieve incremental sorting, uniformly sorting existing files and newly arrived files. 2. Integrate the three functions of merging small files, eliminating add and delete pairs, and manifest sorting into this feature. The original full compaction and minor compaction would undermine the sorting effect. 
   
   Moreover, I think the current PR has the following issues: 
   1. It does not reorder the large number of existing manifests. 
   2. The manifest sort function is limited by the trigger frequency of full compaction; if only minor compaction is performed, manifest sort cannot be triggered. 
   3. Manifest sort only sorts small files and delete-related files, so the sorting is not sufficient. 
   4. Minor compaction will destroy the sorting effect.
   
   To avoid conflicts and duplicated implementation, I’ll continue to improve my PR and port any useful parts if needed, with credit. Thanks for understanding! 
   
   However, thank you very much for your contribution! If you have any suggestions or find areas that need improvement in my PR #7842 , please feel free to continue providing review comments. Your feedback is very helpful for my work


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org