You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "wecharyu (via GitHub)" <gi...@apache.org> on 2025/12/12 03:35:24 UTC

[I] [C++][Parquet] Add configure to limit the row group size [arrow]

wecharyu opened a new issue, #48467:
URL: https://github.com/apache/arrow/issues/48467

   ### Describe the enhancement requested
   
   Arrow has one config `max_row_group_length_` to limit the row number of a row group, but does not limit the size, which is even more important, see [parquet doc](https://parquet.apache.org/docs/file-format/configurations/).
   
   And other implementations like Java has such config:
   https://github.com/apache/parquet-java/blob/79977453b8cd65e6244f16316fac3a510aa87aa8/parquet-hadoop/src/main/java/parquet/hadoop/ParquetWriter.java#L35
   
   ### Component(s)
   
   C++


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [C++][Parquet] Add configure to limit the row group size [arrow]

Posted by "wecharyu (via GitHub)" <gi...@apache.org>.
wecharyu commented on issue #48467:
URL: https://github.com/apache/arrow/issues/48467#issuecomment-3645333414

   Hi @tusharbhatt7, I have created a PR, could you help review it?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [C++][Parquet] Add configure to limit the row group size [arrow]

Posted by "tusharbhatt7 (via GitHub)" <gi...@apache.org>.
tusharbhatt7 commented on issue #48467:
URL: https://github.com/apache/arrow/issues/48467#issuecomment-3645588491

   > Hi [@tusharbhatt7](https://github.com/tusharbhatt7), I have created a PR, could you help review it?
   
   Thanks for tagging me! I'm still new to the Arrow codebase, but I will go through the PR and try to understand the changes. I may not be able to give a full review yet, but I’ll share any observations I have.
   
   Thanks again for the update!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [C++][Parquet] Add configure to limit the row group size [arrow]

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on issue #48467:
URL: https://github.com/apache/arrow/issues/48467#issuecomment-4143063518

   Issue resolved by pull request 49527
   https://github.com/apache/arrow/pull/49527


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [C++][Parquet] Add configure to limit the row group size [arrow]

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac closed issue #48467: [C++][Parquet] Add configure to limit the row group size
URL: https://github.com/apache/arrow/issues/48467


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [C++][Parquet] Add configure to limit the row group size [arrow]

Posted by "tusharbhatt7 (via GitHub)" <gi...@apache.org>.
tusharbhatt7 commented on issue #48467:
URL: https://github.com/apache/arrow/issues/48467#issuecomment-3645269266

   Hi, I’d like to work on this issue.
   
   To confirm the expected behavior: the idea is to add a new ParquetWriter property (similar to parquet-mr’s `maxRowGroupSize`) that limits the *byte size* of a row group, complementing the existing `max_row_group_length` (which limits the number of rows).
   
   My plan:
   1. Add a new optional config field in `parquet::WriterProperties` for max row group size in bytes.
   2. Enforce this condition in `parquet::RowGroupWriter` so a new row group is started once the accumulated buffered size exceeds the limit.
   3. Mirror semantics from parquet-mr implementation.
   4. Add unit tests verifying correct splits for variable-size columns.
   5. Update documentation.
   
   Please let me know if this approach matches what you had in mind or if you’d prefer a specific naming/config structure.
   Thanks — Tushar


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org