You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "zhf999 (via GitHub)" <gi...@apache.org> on 2026/07/30 13:47:02 UTC
[PR] [GH-48476][C++][Parquet] Add max_row_group_size writer property to limit row groups by compressed byte size [arrow]
zhf999 opened a new pull request, #50743:
URL: https://github.com/apache/arrow/pull/50743
# Add max_row_group_size writer property to limit row groups by compressed byte size
GH-#48467 Add configure to limit the row group size
# PR Description
### Rationale for this change
Currently the Parquet writer only supports limiting row groups by row count
(`WriterProperties::max_row_group_length()`). For tables with wide rows or
highly variable row sizes, a row-count limit alone can produce row groups that
are much larger (or smaller) than desired. Many query engines and storage
systems tune around a target row group size in bytes (e.g. HDFS block size),
so it is useful to also support limiting row groups by their compressed byte
size.
### What changes are included in this PR?
- Add `WriterProperties::max_row_group_size()` and
`WriterProperties::Builder::max_row_group_size(int64_t)` to specify the max
row group size in compressed bytes. The default is unlimited
(`std::numeric_limits<int64_t>::max()`), so existing behavior is unchanged
unless users opt in.
- `parquet::arrow::FileWriter::WriteRecordBatch` now checks the size of the
current buffered row group before/after each write, counting:
- compressed pages already written to the sink
(`RowGroupWriter::total_compressed_bytes_written()`),
- compressed pages still buffered in column writers
(`RowGroupWriter::total_compressed_bytes()`),
- the buffered dictionary not yet written as dictionary pages
(`RowGroupWriter::estimated_buffered_stats().dict_bytes`, uncompressed).
When the accumulated size reaches the limit, a new buffered row group is
started. The check happens at write boundaries, so a row group may exceed
the limit by up to the size of a single written batch.
- The limit only applies to buffered row groups (`WriteRecordBatch`).
`WriteTable` and the manual `NewRowGroup()`/`WriteColumnChunk()` paths
control row group boundaries explicitly via `chunk_size`/API calls and are
intentionally not affected (row counts are fixed before writing in the
non-buffered mode, so the byte size limit cannot be enforced there). This is
documented on the builder method.
### Are these changes tested?
Yes.
- `WriterPropertiesTest.RoundTripThroughBuilder` (properties_test.cc) now
covers `max_row_group_size`, with a non-default value added to the
`override_defaults` test case.
- New test `TestArrowReadWrite.WriteRecordBatchRespectsMaxRowGroupSize`
(arrow_reader_writer_test.cc) verifies that record batches exceeding the
byte size limit cause row group rollover on subsequent writes.
- Existing `WriteRecordBatch`/`WriteTable` tests pass unchanged, confirming no
behavior change with the default (unlimited) setting.
### Are there any user-facing changes?
Yes, a new opt-in writer property `max_row_group_size` is added to the public
C++ API (`WriterProperties` / `WriterProperties::Builder`). The default is
unlimited, so there is no behavior change for existing users.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48476: [C++] [Parquet] Add max_row_group_size writer property to limit row groups by compressed byte size [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #50743:
URL: https://github.com/apache/arrow/pull/50743#issuecomment-5131822674
:warning: GitHub issue #48476 **has been automatically assigned in GitHub** to PR creator.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48476: [C++] [Parquet] Add max_row_group_size writer property to limit row groups by compressed byte size [arrow]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #50743:
URL: https://github.com/apache/arrow/pull/50743#discussion_r3688976953
##########
cpp/src/parquet/properties.h:
##########
@@ -492,6 +496,18 @@ class PARQUET_EXPORT WriterProperties {
return this;
}
+ /// Specify the max row group size in compressed bytes.
+ /// Default unlimited.
+ ///
+ /// The limit is checked against the compressed pages accumulated in the
+ /// current row group, so the actual row group size may slightly exceed it.
+ /// Only effective for buffered row groups (
+ /// parquet::arrow::FileWriter::WriteRecordBatch).
Review Comment:
IIRC, WriteTable also splits the table into several row groups if max_row_group_length is reached.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48476: [C++] [Parquet] Add max_row_group_size writer property to limit row groups by compressed byte size [arrow]
Posted by "zhf999 (via GitHub)" <gi...@apache.org>.
zhf999 commented on code in PR #50743:
URL: https://github.com/apache/arrow/pull/50743#discussion_r3687569621
##########
cpp/src/parquet/properties.h:
##########
@@ -492,6 +496,18 @@ class PARQUET_EXPORT WriterProperties {
return this;
}
+ /// Specify the max row group size in compressed bytes.
+ /// Default unlimited.
+ ///
+ /// The limit is checked against the compressed pages accumulated in the
+ /// current row group, so the actual row group size may slightly exceed it.
+ /// Only effective for buffered row groups (
+ /// parquet::arrow::FileWriter::WriteRecordBatch).
Review Comment:
`WriteRecordBatch` writes into a buffered row group, which accumulates across calls. That makes the byte size observable between writes, so we can simply stop adding to the current row group once the accumulated compressed size reaches the limit.
While `WriteTable` uses the non-buffered path, where each call maps to exactly one row group whose row count is fixed up front by the user-supplied chunk_size. Enforcing a byte limit there would require predicting the compressed size before writing, e.g. by estimating an average row size from the previous row group. That's inherently approximate, especially since chunk_size is an explicit contract from the caller.
I wonder if estimation is acceptable in `WriteTable`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [GH-48476][C++][Parquet] Add max_row_group_size writer property to limit row groups by compressed byte size [arrow]
Posted by "zhf999 (via GitHub)" <gi...@apache.org>.
zhf999 closed pull request #50743: [GH-48476][C++][Parquet] Add max_row_group_size writer property to limit row groups by compressed byte size
URL: https://github.com/apache/arrow/pull/50743
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [GH-48476][C++][Parquet] Add max_row_group_size writer property to limit row groups by compressed byte size [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #50743:
URL: https://github.com/apache/arrow/pull/50743#issuecomment-5131650419
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
Thanks for opening a pull request!
**This pull request has been automatically converted to a draft because its title doesn't match Arrow's required format.**
If this is not a [minor PR](https://github.com/apache/arrow/blob/main/CONTRIBUTING.md#Minor-Fixes). Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose
Opening GitHub issues ahead of time contributes to the [Openness](http://theapacheway.com/open/#:~:text=Openness%20allows%20new%20users%20the,must%20happen%20in%20the%20open.) of the Apache Arrow project.
Then could you also rename the pull request title in the following format?
GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}
or
MINOR: [${COMPONENT}] ${SUMMARY}
**After updating the title, you can mark the pull request as ready for review.**
See also:
* [Other pull requests](https://github.com/apache/arrow/pulls/)
* [Contribution Guidelines - Contributing Overview](https://arrow.apache.org/docs/developers/overview.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48476: [C++] [Parquet] Add max_row_group_size writer property to limit row groups by compressed byte size [arrow]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #50743:
URL: https://github.com/apache/arrow/pull/50743#discussion_r3684181821
##########
cpp/src/parquet/properties.h:
##########
@@ -492,6 +496,18 @@ class PARQUET_EXPORT WriterProperties {
return this;
}
+ /// Specify the max row group size in compressed bytes.
+ /// Default unlimited.
+ ///
+ /// The limit is checked against the compressed pages accumulated in the
+ /// current row group, so the actual row group size may slightly exceed it.
+ /// Only effective for buffered row groups (
+ /// parquet::arrow::FileWriter::WriteRecordBatch).
Review Comment:
Why only for `WriteRecordBatch`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48476: [C++] [Parquet] Add max_row_group_size writer property to limit row groups by compressed byte size [arrow]
Posted by "zhf999 (via GitHub)" <gi...@apache.org>.
zhf999 commented on code in PR #50743:
URL: https://github.com/apache/arrow/pull/50743#discussion_r3747144092
##########
cpp/src/parquet/properties.h:
##########
@@ -492,6 +496,18 @@ class PARQUET_EXPORT WriterProperties {
return this;
}
+ /// Specify the max row group size in compressed bytes.
+ /// Default unlimited.
+ ///
+ /// The limit is checked against the compressed pages accumulated in the
+ /// current row group, so the actual row group size may slightly exceed it.
+ /// Only effective for buffered row groups (
+ /// parquet::arrow::FileWriter::WriteRecordBatch).
Review Comment:
@wgtmac What about change the `WriteTable` to buffered path like `WriteRecordBatch`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48476: [C++] [Parquet] Add max_row_group_size writer property to limit row groups by compressed byte size [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #50743:
URL: https://github.com/apache/arrow/pull/50743#issuecomment-5237057294
:warning: GitHub issue #48476 **has been automatically assigned in GitHub** to PR creator.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48476: [C++] [Parquet] Add max_row_group_size writer property to limit row groups by compressed byte size [arrow]
Posted by "zhf999 (via GitHub)" <gi...@apache.org>.
zhf999 commented on PR #50743:
URL: https://github.com/apache/arrow/pull/50743#issuecomment-5131852219
Related PR: #48468
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48476: [C++] [Parquet] Add max_row_group_size writer property to limit row groups by compressed byte size [arrow]
Posted by "zhf999 (via GitHub)" <gi...@apache.org>.
zhf999 commented on code in PR #50743:
URL: https://github.com/apache/arrow/pull/50743#discussion_r3747144092
##########
cpp/src/parquet/properties.h:
##########
@@ -492,6 +496,18 @@ class PARQUET_EXPORT WriterProperties {
return this;
}
+ /// Specify the max row group size in compressed bytes.
+ /// Default unlimited.
+ ///
+ /// The limit is checked against the compressed pages accumulated in the
+ /// current row group, so the actual row group size may slightly exceed it.
+ /// Only effective for buffered row groups (
+ /// parquet::arrow::FileWriter::WriteRecordBatch).
Review Comment:
What about change the `WriteTable` to buffered path like `WriteRecordBatch`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48476: [C++] [Parquet] Add max_row_group_size writer property to limit row groups by compressed byte size [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #50743:
URL: https://github.com/apache/arrow/pull/50743#issuecomment-5138406369
:warning: GitHub issue #48476 **has been automatically assigned in GitHub** to PR creator.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48476: [C++] [Parquet] Add max_row_group_size writer property to limit row groups by compressed byte size [arrow]
Posted by "zhf999 (via GitHub)" <gi...@apache.org>.
zhf999 commented on PR #50743:
URL: https://github.com/apache/arrow/pull/50743#issuecomment-5131837220
#48468
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48476: [C++] [Parquet] Add max_row_group_size writer property to limit row groups by compressed byte size [arrow]
Posted by "Reranko05 (via GitHub)" <gi...@apache.org>.
Reranko05 commented on PR #50743:
URL: https://github.com/apache/arrow/pull/50743#issuecomment-5132048880
> GitHub Issue: https://github.com/apache/arrow/pull/48476
Thanks for contributing! Can you edit the PR desc and ref to the issue, it seems a PR is being referenced here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48476: [C++] [Parquet] Add max_row_group_size writer property to limit row groups by compressed byte size [arrow]
Posted by "zhf999 (via GitHub)" <gi...@apache.org>.
zhf999 commented on code in PR #50743:
URL: https://github.com/apache/arrow/pull/50743#discussion_r3689197098
##########
cpp/src/parquet/properties.h:
##########
@@ -492,6 +496,18 @@ class PARQUET_EXPORT WriterProperties {
return this;
}
+ /// Specify the max row group size in compressed bytes.
+ /// Default unlimited.
+ ///
+ /// The limit is checked against the compressed pages accumulated in the
+ /// current row group, so the actual row group size may slightly exceed it.
+ /// Only effective for buffered row groups (
+ /// parquet::arrow::FileWriter::WriteRecordBatch).
Review Comment:
`max_row_group_length` can be applied there because row counts are known before writing — it just clamps chunk_size. The byte size simply isn't knowable at that point, which is the asymmetry.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org