[GitHub] [orc] hsnusonic commented on a change in pull request #996: ORC-1078: Row group end offset doesn't accommodate all the blocks

[GitBox <gi...@apache.org>]

hsnusonic commented on a change in pull request #996:
URL: https://github.com/apache/orc/pull/996#discussion_r781695405



##########
File path: java/core/src/java/org/apache/orc/impl/RecordReaderUtils.java
##########
@@ -182,10 +182,13 @@ public static long estimateRgEndOffset(boolean isCompressed,
                                          long streamLength) {
     // figure out the worst case last location
     // if adjacent groups have the same compressed block offset then stretch the slop
-    // by factor of 2 to safely accommodate the next compression block.
-    // One for the current compression block and another for the next compression block.
+    // by a factor to safely accommodate the next compression block.
+    // 512 is the MAX_SCOPE defined in RunLengthIntegerWriterV2.
+    // 8 is the maximum size of bytes for each value (see RunLengthIntegerWriterV2.zzBits100p).
+    // We need to calculate the maximum number of blocks by bufferSize accordingly.
+    int stretchFactor = bufferSize > 0 ? 2 + (512 * 8 - 1) / bufferSize : 2;
     long slop = isCompressed
-                    ? 2 * (OutStream.HEADER_SIZE + bufferSize)
+                    ? stretchFactor * (OutStream.HEADER_SIZE + bufferSize)
                     : WORST_UNCOMPRESSED_SLOP;
     return isLast ? streamLength : Math.min(streamLength, nextGroupOffset + slop);

Review comment:
       It should not cause performance issues here, since files should be read in stripe level (not row goup level) and we are fixing the boundary of each row group. @pgaref Please correct me if I misunderstood anything.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@orc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org