You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "wg1026688210 (via GitHub)" <gi...@apache.org> on 2023/05/26 12:41:42 UTC

[GitHub] [incubator-paimon] wg1026688210 commented on a diff in pull request #1199: [core] add FullCompaction for Manifest File

wg1026688210 commented on code in PR #1199:
URL: https://github.com/apache/incubator-paimon/pull/1199#discussion_r1206723808


##########
paimon-core/src/main/java/org/apache/paimon/manifest/ManifestFileMeta.java:
##########
@@ -185,4 +209,138 @@ private static void mergeCandidates(
             newMetas.addAll(merged);
         }
     }
+
+    public static List<ManifestFileMeta> tryFullCompaction(
+            List<ManifestFileMeta> inputs,
+            List<ManifestFileMeta> newMetas,
+            ManifestFile manifestFile,
+            RowType partitionType,
+            long suggestedMetaSize,
+            long sizeTrigger) {
+        List<ManifestFileMeta> base = new ArrayList<>();
+        int totalManifestSize = 0;
+        int i;
+        for (i = 0; i < inputs.size(); i++) {
+            ManifestFileMeta manifestFileMeta = inputs.get(i);
+            if (manifestFileMeta.numDeletedFiles == 0
+                    && manifestFileMeta.fileSize >= suggestedMetaSize) {
+                base.add(manifestFileMeta);
+                totalManifestSize += manifestFileMeta.fileSize;
+            } else {
+                break;
+            }
+        }
+
+        List<ManifestFileMeta> delta = new ArrayList<>();
+        for (; i < inputs.size(); i++) {
+            ManifestFileMeta manifestFileMeta = inputs.get(i);
+            delta.add(manifestFileMeta);
+            totalManifestSize += manifestFileMeta.fileSize;
+        }
+
+        long deltaDeleteFileNum = 0;
+        long totalDeltaFileSize = 0;
+        for (ManifestFileMeta manifestFileMeta : delta) {
+            deltaDeleteFileNum += manifestFileMeta.numDeletedFiles();
+            totalDeltaFileSize += manifestFileMeta.fileSize();
+            totalManifestSize += manifestFileMeta.fileSize();
+        }
+
+        if (totalDeltaFileSize < sizeTrigger) {
+            return inputs;
+        }
+
+        logger.info(
+                "start Manifest File Full Compaction,pick the number of delete file: {} ,total manifest file size: {}",
+                deltaDeleteFileNum,
+                totalManifestSize);
+        // do full-compaction
+        Map<ManifestEntry.Identifier, ManifestEntry> deltaMerged = new LinkedHashMap<>();
+        for (ManifestFileMeta manifest : delta) {
+            ManifestEntry.mergeEntries(manifestFile.read(manifest.fileName), deltaMerged);
+        }
+        Set<BinaryRow> deletePartitions = computeDeletePartitions(deltaMerged);
+
+        Predicate predicate = convertPartitionToPredicate(partitionType, deletePartitions);
+
+        List<ManifestFileMeta> result = new ArrayList<>();
+        FieldStatsArraySerializer fieldStatsArraySerializer =
+                new FieldStatsArraySerializer(partitionType);
+        int j;
+        for (j = 0; j < base.size(); j++) {
+            // todo  optimize this to binary search.
+            ManifestFileMeta file = base.get(j);
+            if (predicate.test(
+                    file.numAddedFiles + file.numDeletedFiles,
+                    file.partitionStats.fields(fieldStatsArraySerializer))) {
+                break;
+            } else {
+                result.add(file);
+            }
+        }
+
+        Map<ManifestEntry.Identifier, ManifestEntry> fullMerged = new LinkedHashMap<>();
+        for (; j < base.size(); j++) {
+            ManifestFileMeta manifestFileMeta = base.get(j);
+            ManifestEntry.mergeEntries(manifestFile.read(manifestFileMeta.fileName), fullMerged);
+        }
+
+        ManifestEntry.mergeEntries(deltaMerged.values(), fullMerged);
+        if (!fullMerged.isEmpty()) {
+            List<ManifestFileMeta> merged =
+                    manifestFile.write(new ArrayList<>(fullMerged.values()));

Review Comment:
   Maybe we can reuse this option `manifest.full-compaction-threshold-size`. Set the default value of the option to Long.MaxValue



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org