You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "jonasdedden (via GitHub)" <gi...@apache.org> on 2026/07/30 13:31:09 UTC
[I] `DELTA_BYTE_ARRAY` stops deduplicating in parquet 59 for values larger than the page size limit [arrow-rs]
jonasdedden opened a new issue, #10489:
URL: https://github.com/apache/arrow-rs/issues/10489
### Describe the bug
## Summary
parquet 59 added byte-budget-aware mini-batch splitting to the column writer.
The new splitting has an unintended interaction with `DELTA_BYTE_ARRAY`. When an individual value is
larger than `data_page_size_limit`, the chunker is forced to emit one value per page. Prefix state is
reset at every page boundary, so every value gets `prefix_length = 0` and `DELTA_BYTE_ARRAY`
degenerates to exactly `PLAIN`. Columns of large values that share long prefixes — or are outright
identical — stop being deduplicated entirely.
The default `data_page_size_limit` is 1 MiB, so any `BYTE_ARRAY` column with values above 1 MiB is
affected out of the box.
**Affects:** `parquet` 59.0.0 and 59.1.0 (58.4.0 is fine)
**Impact:** 10× output size in our test corpus; write-syscall count rises by the same factor
### To Reproduce
Ten identical 8 MiB strings, `DELTA_BYTE_ARRAY` forced, dictionary and statistics off, no
compression. Requires only `arrow` and `parquet`.
```rust
use arrow::array::{ArrayRef, RecordBatch, StringArray};
use arrow::datatypes::{DataType, Field, Schema};
use parquet::arrow::ArrowWriter;
use parquet::basic::{Compression, Encoding};
use parquet::file::properties::{EnabledStatistics, WriterProperties};
use std::sync::Arc;
fn write(page_limit: Option<usize>) -> usize {
const NUM_ROWS: usize = 10;
const VALUE_LEN: usize = 8 * 1024 * 1024;
let value = "a".repeat(VALUE_LEN);
let schema = Arc::new(Schema::new(vec![Field::new("v", DataType::Utf8, false)]));
let array: ArrayRef = Arc::new(StringArray::from(vec![value.as_str(); NUM_ROWS]));
let batch = RecordBatch::try_new(schema.clone(), vec![array]).unwrap();
let mut props = WriterProperties::builder()
.set_compression(Compression::UNCOMPRESSED)
.set_statistics_enabled(EnabledStatistics::None)
.set_dictionary_enabled(false)
.set_encoding(Encoding::DELTA_BYTE_ARRAY);
if let Some(limit) = page_limit {
props = props.set_data_page_size_limit(limit);
}
let mut out = Vec::new();
let mut writer = ArrowWriter::try_new(&mut out, schema, Some(props.build())).unwrap();
writer.write(&batch).unwrap();
writer.close().unwrap();
out.len()
}
```
### Expected behavior
### Measured results
Raw input is 80.00 MiB. Ideal `DELTA_BYTE_ARRAY` output is ~8.00 MiB (one value stored, nine
zero-length suffixes).
| `data_page_size_limit` | parquet 58.4.0 | parquet 59.1.0 |
| --- | --- | --- |
| default (1 MiB) | **8.00 MiB** | **80.00 MiB** |
| 1 MiB | 8.00 MiB | 80.00 MiB |
| 4 MiB | 8.00 MiB | 80.00 MiB |
| 8 MiB | 8.00 MiB | 80.00 MiB |
| 9 MiB | 8.00 MiB | 8.00 MiB |
| 16 MiB | 8.00 MiB | 8.00 MiB |
| 128 MiB | 8.00 MiB | 8.00 MiB |
80.00 MiB is byte-for-byte what `PLAIN` produces: the encoding is doing no work at all.
The same result holds for values that merely *share a long prefix* rather than being identical, so
this is not specific to exact duplicates.
### Additional context
## Suggested fixes upstream
Roughly in increasing order of ambition.
1. **Do not split when the split cannot help.** If the value at the head of the chunk alone exceeds
the budget, a one-value page is unavoidable *for that value*, but forcing a page boundary
afterwards is what destroys the following values' prefix compression. Where the encoder reports
`Some(0)`, consider letting the mini-batch run to the encoder's own page-flush decision instead of
clamping to one value — i.e. treat "cannot fit" as "budget does not apply here" rather than "split
maximally". This restores 58's behaviour for exactly the case that regressed, while keeping the
memory fix for the common case where values are small and a 1024-row mini-batch is what overflows.
2. **Make the budget encoding-aware.** `count_within_budget_offsets` measures raw payload length. For
`DELTA_BYTE_ARRAY` the encoded contribution of a value that shares a prefix with its predecessor
is much smaller than its length, so the budget check systematically over-estimates and splits far
more eagerly than necessary. Consulting the encoder's incremental estimate — the same quantity
`estimated_data_page_size` already tracks — would fix both the eager splitting and the cliff.
3. **Warn, or raise the effective floor.** At minimum, `data_page_size_limit` silently changing a
column's effective encoding from `DELTA_BYTE_ARRAY` to `PLAIN`-equivalent deserves a documented
note on `set_data_page_size_limit` and on `DEFAULT_PAGE_SIZE`, since the default is what most
users hit.
Option 1 looks the most surgical: it is local to `byte_budget_sub_batch_size` and does not require
threading encoder state into the budget calculation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] `DELTA_BYTE_ARRAY` stops deduplicating in parquet 59 for values larger than the page size limit [arrow-rs]
Posted by "jonasdedden (via GitHub)" <gi...@apache.org>.
jonasdedden commented on issue #10489:
URL: https://github.com/apache/arrow-rs/issues/10489#issuecomment-5183640282
[No not really](https://github.com/jonasdedden/disco-parquet/blob/00b88e2394deddb85fc5123cdf958e5b28f0295e/disco-parquet-core/src/parquet/parquet_writer.rs#L76), basically the only thing exposed for keeping file sizes down is `compression`. For test cases I usually omit that though because I don't want to introduce unnecessary compute, and I somewhat trusted the default encoders to handle cases where one stores trivially encodable blobs (i.e. for example 8MiB of repeated `a`'s) 😅 But it's a niche use case, I have to admit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] `DELTA_BYTE_ARRAY` stops deduplicating in parquet 59 for values larger than the page size limit [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed issue #10489: `DELTA_BYTE_ARRAY` stops deduplicating in parquet 59 for values larger than the page size limit
URL: https://github.com/apache/arrow-rs/issues/10489
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] `DELTA_BYTE_ARRAY` stops deduplicating in parquet 59 for values larger than the page size limit [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #10489:
URL: https://github.com/apache/arrow-rs/issues/10489#issuecomment-5182744600
Thank you for the report @jonasdedden
> When an individual value is larger than data_page_size_limit, the chunker is forced to emit one value per page.
I think we could potentially argue this is an **improvement** in behavior in the sense that the output data pages are now closer to the specified limit.
I realize that for the usecase described in this ticket the compression ratio is likely much worse, but I would argue that may have been because prior versions weren't accurately applying the page size limit.
<img width="1123" height="584" alt="Image" src="https://github.com/user-attachments/assets/456bb9ea-9d8e-4c9b-b7da-4a4baa7ea396" />
It seems like the correct fix for the issue you are seeing would be to increase your page size limit to be larger than the values you are writing into the encoder, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] `DELTA_BYTE_ARRAY` stops deduplicating in parquet 59 for values larger than the page size limit [arrow-rs]
Posted by "jonasdedden (via GitHub)" <gi...@apache.org>.
jonasdedden commented on issue #10489:
URL: https://github.com/apache/arrow-rs/issues/10489#issuecomment-5183349102
Hey @alamb, I think in general, I'd agree to your sentiment - the problem is just that this behavioral change is very silent and there are basically no guards from the `parquet` crate for people who don't know that they specifically have to tune the data page size limit. That potentially also could be a pretty niche config setting for most people, I at least never had to tune it before.
For context, I stumbled over this issue when I was running a test system for a large internal library and just wanted to update some dependencies. That suddenly OOMed my machine (because test data was written to `/tmp` which happened to be a tmpfs volume). Note that the degenerated parquet file basically has size of `num_rows` * 8MiB, which can of course be troublesome for very high `num_rows`.
The change proposed in https://github.com/apache/arrow-rs/pull/10505 looks interesting, but I'm not really an expert on this topic to accurately judge what to do.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] `DELTA_BYTE_ARRAY` stops deduplicating in parquet 59 for values larger than the page size limit [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #10489:
URL: https://github.com/apache/arrow-rs/issues/10489#issuecomment-5183568318
> basically no guards from the parquet crate for people who don't know that they specifically have to tune the data page size limit.
This is a good point. Did your internal library set the DELTA_BYTE_ARRAY encoding? Maybe we can document there some of the known caveats
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org