You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@amoro.apache.org by "klion26 (via GitHub)" <gi...@apache.org> on 2025/11/10 11:40:52 UTC

Re: [PR] [Bug]: Amoro optimization can result in the input files and the merge… [amoro]

klion26 commented on code in PR #3856:
URL: https://github.com/apache/amoro/pull/3856#discussion_r2510156492


##########
amoro-format-iceberg/src/main/java/org/apache/amoro/optimizing/IcebergRewriteExecutor.java:
##########
@@ -101,12 +101,25 @@ protected TaskWriter<Record> dataWriter() {
   @Override
   protected long targetSize() {
     long targetSize = super.targetSize();
-    long inputSize =
-        Arrays.stream(input.rewrittenDataFiles()).mapToLong(DataFile::fileSizeInBytes).sum();
-    // When the input files’ total size is below targetSize, remove the output file size limit to
+    DataFile[] rewrittenDataFiles = input.rewrittenDataFiles();
+    if (rewrittenDataFiles == null || rewrittenDataFiles.length == 0) {
+      return targetSize;
+    }
+    long inputSize = Arrays.stream(rewrittenDataFiles).mapToLong(DataFile::fileSizeInBytes).sum();
+    // When the input files' total size is below targetSize, remove the output file size limit to
     // avoid outputting multiple files.
     // For more details, please refer to: https://github.com/apache/amoro/issues/3645
-    return inputSize < targetSize ? Long.MAX_VALUE : targetSize;
+    if (inputSize < targetSize) {
+      return Long.MAX_VALUE;
+    }
+    // Even if total size >= targetSize, if average file size is small (less than targetSize),

Review Comment:
   Will this output a single file with too big file size?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org