You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "adriangb (via GitHub)" <gi...@apache.org> on 2026/08/27 04:00:53 UTC
[PR] perf(parquet): resolve per-column writer properties once per column [arrow-rs]
adriangb opened a new pull request, #10880:
URL: https://github.com/apache/arrow-rs/pull/10880
# Which issue does this PR close?
- Contributes to #9722.
- Builds on #10878, which adds the `repeated_batches` benchmark cases used below.
# Rationale for this change
`WriterProperties` keeps per-column overrides in a
`HashMap<ColumnPath, ColumnProperties>`. Every accessor that takes a
`&ColumnPath` — `compression`, `encoding`, `dictionary_enabled`,
`statistics_enabled`, `write_page_header_statistics`,
`column_data_page_size_limit`, `column_dictionary_page_size_limit`,
`column_data_page_v2_compression_ratio_threshold` and
`bloom_filter_properties` — hashes that path and searches the map. `ColumnPath`
is a `Vec<String>`, so each search hashes every part of the path and then
compares strings on a hit.
The column writers call those accessors repeatedly for the same leaf column:
- 6 times while a column writer is constructed (`GenericColumnWriter::new` plus
`ColumnValueEncoder::try_new`);
- 2 more on every `write`, in `ByteBudgetChunker::new`;
- 1 per mini-batch in `should_add_data_page`, and another in
`should_dict_fallback` when a dictionary encoder is active;
- 1 per page in `add_data_page`, plus one more for Data Page v2.
`WriterProperties` is immutable once built, so all of that recomputes an answer
that was already available when the column writer was created.
Counting the searches directly (temporary instrumentation on the map access) over
the `writer_overhead` shapes:
| benchmark case | searches before | searches after |
| --- | --- | --- |
| `writer_overhead/1000_cols` | 10,000 | 1,000 |
| `writer_overhead/5000_cols` | 50,000 | 5,000 |
| `writer_overhead/10000_cols` | 100,000 | 10,000 |
| `writer_overhead/1000_cols/repeated_batches` | 103,000 | 1,000 |
| `writer_overhead/5000_cols/repeated_batches` | 515,000 | 5,000 |
After the change it is exactly one search per leaf column per row group.
# What changes are included in this PR?
- Adds `WriterProperties::resolve_column_properties`, which searches the
per-column map once and returns a `ResolvedColumnProperties` holding every
per-column setting already resolved against the file-wide defaults. The struct
is `pub(crate)`; no public API is added.
- `GenericColumnWriter::new` calls it once and stores the result. The per-batch
and per-page paths, `ByteBudgetChunker::new`, and both `ColumnValueEncoder`
implementations read from it instead of searching the map again.
`ColumnValueEncoder::try_new` and `create_bloom_filter` now take the resolved
settings; that trait is not nameable outside the crate.
- The existing per-column accessors keep their behaviour and are reimplemented on
top of the same resolution helpers, so the "column override, else file default,
else constant" rule has one definition rather than one per accessor.
# Are these changes tested?
Yes.
- The existing suite passes unchanged, with and without `encryption`:
`cargo test -p parquet --features "arrow async encryption test_common experimental"`.
- New `test_resolve_column_properties_matches_individual_accessors` asserts the
one-pass resolution agrees with every individual accessor, for a column with
overrides, a column that inherits the file defaults, and properties left
entirely at their defaults.
- To check the writer is byte-for-byte unchanged I wrote the same file from a
build before and after this change, with a workload covering the settings this
PR touches: per-column compression, encoding, dictionary on and off, page and
dictionary page size limits, page-header statistics, a bloom filter, dictionary
fallback on multi-KB values, and both writer versions. The two files are
identical.
Benchmark:
cargo bench -p parquet --bench writer_overhead
I did not have an idle machine, and criterion's wall-clock intervals were too
wide there to be usable. I measured CPU time (user + sys) per iteration instead,
which is far less sensitive to competing load: the minimum over 40 alternating
runs of the two builds, with a separately measured ~13.3 ms of fixed benchmark
setup subtracted.
| benchmark case | before | after | |
| --- | --- | --- | --- |
| `writer_overhead/1000_cols` | 2.7 ms | 2.1 ms | see note |
| `writer_overhead/5000_cols` | 20.2 ms | 18.5 ms | −8 % |
| `writer_overhead/10000_cols` | 38.8 ms | 34.2 ms | −12 % |
| `writer_overhead/1000_cols/repeated_batches` | 12.9 ms | 8.6 ms | −34 % |
| `writer_overhead/5000_cols/repeated_batches` | 88.0 ms | 70.1 ms | −20 % |
Note: at 1,000 columns with a single batch the remaining figure is close enough
to the spread of the setup subtraction that I would not read a percentage into
it. The other four imply about 40 ns per map search, consistently across all four
shapes, which matches the search counts above. A run on an idle machine would be
worth having, and the benchmark from #10878 makes that easy.
`ColumnWriterImpl<Int32Type>` grows from 1312 to 1376 bytes to hold the resolved
settings. Peak RSS on the 10,000 column benchmark was unchanged.
# Are there any user-facing changes?
No. No public API is added, removed or changed, and the bytes written are
unchanged.
# AI usage
This PR was written with Claude Code and reviewed by a human. The byte-for-byte
output comparison and the map-search counts above are the checks run to confirm
the behaviour is unchanged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(parquet): resolve per-column writer properties once per column [arrow-rs]
Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey commented on PR #10880:
URL: https://github.com/apache/arrow-rs/pull/10880#issuecomment-5474872406
thanks @adriangb, @alamb & @etseidl
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(parquet): resolve per-column writer properties once per column [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #10880:
URL: https://github.com/apache/arrow-rs/pull/10880#issuecomment-5443046139
(this PR has a merge conflict)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(parquet): resolve per-column writer properties once per column [arrow-rs]
Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey merged PR #10880:
URL: https://github.com/apache/arrow-rs/pull/10880
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(parquet): resolve per-column writer properties once per column [arrow-rs]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #10880:
URL: https://github.com/apache/arrow-rs/pull/10880#issuecomment-5443931231
Thanks Andrew. Addressed your comments and resolved the merge conflict. I opened https://github.com/apache/arrow-rs/issues/10896 to track hoisting this up further.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(parquet): resolve per-column writer properties once per column [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #10880:
URL: https://github.com/apache/arrow-rs/pull/10880#issuecomment-5443041691
It makes sense to me
Claude mentioned this still happens once per row group
So one potential follow on would be
> - Hoist across row groups. Column writers are recreated per row group, so a wide schema × many row groups still does one search per column per row group. Caching Vec<ResolvedColumnProperties> per leaf in SerializedFileWriter/ArrowWriter would make it once per file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org