You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@auron.apache.org by "Sigma-Ma (via GitHub)" <gi...@apache.org> on 2026/09/13 10:58:29 UTC

[PR] [AURON #1743] Support ORC reader batch size [auron]

Sigma-Ma opened a new pull request, #2525:
URL: https://github.com/apache/auron/pull/2525

   # Which issue does this PR close?
   
   Closes #1743
   
   # Rationale for this change
   
   Native ORC reads use the global batch size. Reducing it to handle large String/Binary values also affects other native operators.
   
   # What changes are included in this PR?
   
   Add `spark.auron.orc.batchSize` and pass it to the ORC reader through the existing JNI configuration path. The value must be positive and defaults to `spark.auron.batchSize`.
   
   Add regression coverage for configuration validation, ORC batch boundaries, and unchanged Parquet batching. Document the new setting.
   
   # Are there any user-facing changes?
   
   Users can configure native ORC reader batches independently at application startup. Existing defaults are preserved; downstream operators may still coalesce batches.
   
   # How was this patch tested?
   
   - `env RUSTFLAGS='-C target-cpu=native' cargo test --profile pre --locked -p datafusion-ext-plans orc_exec::tests`.
   - `./build/mvn -Ppre -Pspark-3.5 -Pscala-2.12 -pl spark-extension-shims-spark -DskipBuildNative '-Dsuites=org.apache.auron.AuronQuerySuite ORC reader batch size' test`.
   - `./build/mvn -Ppre -Pspark-4.0 -Pscala-2.13 -pl spark-extension-shims-spark -DskipBuildNative '-Dsuites=org.apache.auron.AuronQuerySuite ORC reader batch size' test`.
   - `./dev/reformat --check`.
   
   # Was this patch authored or co-authored using generative AI tooling?
   
   - [x] Yes
   - [ ] No
   
   Generated-by: Codex (GPT-6)
   
   ASF guidance: https://www.apache.org/legal/generative-tooling.html


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@auron.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [AURON #1743] Support ORC reader batch size [auron]

Posted by "slfan1989 (via GitHub)" <gi...@apache.org>.
slfan1989 merged PR #2525:
URL: https://github.com/apache/auron/pull/2525


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@auron.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [AURON #1743] Support ORC reader batch size [auron]

Posted by "slfan1989 (via GitHub)" <gi...@apache.org>.
slfan1989 commented on PR #2525:
URL: https://github.com/apache/auron/pull/2525#issuecomment-5738457901

   @Sigma-Ma Thanks for the contribution! @cxzl25 Thanks for the review!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@auron.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org