[GitHub] [orc] guiyanakuang commented on a diff in pull request #1118: ORC-1172: Add row count limit config in one stripe

[GitBox <gi...@apache.org>]

guiyanakuang commented on code in PR #1118:
URL: https://github.com/apache/orc/pull/1118#discussion_r876535187


##########
java/core/src/java/org/apache/orc/OrcConf.java:
##########
@@ -29,6 +29,8 @@ public enum OrcConf {
   STRIPE_SIZE("orc.stripe.size", "hive.exec.orc.default.stripe.size",
       64L * 1024 * 1024,
       "Define the default ORC stripe size, in bytes."),
+  STRIPE_ROW_COUNT("orc.stripe.row.count","orc.stripe.row.count",
+      Integer.MAX_VALUE,"This value limit the row count in one stripe."),

Review Comment:
   Keep a space after the comma.
   
   Agree with setting the default value to Integer.MAX_VALUE, so that the default config is compatible with future user upgrades and has no additional impact.



##########
java/core/src/java/org/apache/orc/impl/WriterImpl.java:
##########
@@ -300,7 +303,7 @@ private boolean checkMemory() throws IOException {
         LOG.debug("ORC writer " + physicalWriter + " size = " + size +
                       " limit = " + memoryLimit);
       }
-      if (size > memoryLimit) {
+      if (size > memoryLimit || rowsInStripe > stripeRowCount) {

Review Comment:
   Adding rowsInStripe and stripeRowCount to the debug print in the above lines would be better.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org