You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "mapleFU (via GitHub)" <gi...@apache.org> on 2025/09/01 07:39:13 UTC
[PR] Parquet: change default type for parquet v2 from delta to delta-length-byte-array [arrow-rs]
mapleFU opened a new pull request, #8258:
URL: https://github.com/apache/arrow-rs/pull/8258
# Which issue does this PR close?
Previously, v2 uses "delta" to encoding values, however it's too expansive for non-delta. Can we shift it to delta length byte array?
# Rationale for this change
Change the parquet-v2 default encoding to delta-length-byte-array
# What changes are included in this PR?
Change the parquet-v2 default encoding to delta-length-byte-array
# Are these changes tested?
Covered by existing
# Are there any user-facing changes?
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Parquet: change default type for parquet v2 from delta to delta-length-byte-array [arrow-rs]
Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on PR #8258:
URL: https://github.com/apache/arrow-rs/pull/8258#issuecomment-3398657635
> I have no understanding about the implications of this change (e.g impact on output size, compatibility, etc)
>
> Could someone explain that?
When a dictionary for a given column chunk gets too large, the parquet encoders will fall back to some other encoding for subsequent pages. For V1, the fallback is the plain encoder. When the delta encodings were added, the entry for `DELTA_LENGTH_BYTE_ARRAY` stated that this encoding is always prefered over `PLAIN` for byte array columns ([link](https://github.com/apache/parquet-format/blob/master/Encodings.md#delta-length-byte-array-delta_length_byte_array--6)). Many writers then changed their fallback behavior when writing V2 pages, but rather than the recommended `DELTA_LENGTH_BYTE_ARRAY` they all seemed to favor `DELTA_BYTE_ARRAY`. The issue with `DELTA_BYTE_ARRAY` (which uses front compression) is that when you have unsorted data, you wind up with very short to non-existent prefixes, so it winds up being a slower `DELTA_LENGTH_BYTE_ARRAY`.
@mapleFU is suggesting we change this behavior and fall back to the faster encoder. Users can override encoders per-column if they have sorted data.
>
> A ticket would be an ideal location
I'll try to get an issue started on this. I'm going to mark this as draft in the meantime.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Parquet: change default type for parquet v2 from delta to delta-length-byte-array [arrow-rs]
Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on PR #8258:
URL: https://github.com/apache/arrow-rs/pull/8258#issuecomment-3254563120
Maybe we should discuss this one in an issue. I know the spec says delta_length_byte_array is the default fallback, but I think delta_byte_array is used by parquet-java (I'll admit to not having looked at this for several years). I'm mildly in favor of this change, but others might have different opinions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Parquet: change default type for parquet v2 from delta to delta-length-byte-array [arrow-rs]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #8258: Parquet: change default type for parquet v2 from delta to delta-length-byte-array
URL: https://github.com/apache/arrow-rs/pull/8258
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Parquet: change default type for parquet v2 from delta to delta-length-byte-array [arrow-rs]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #8258:
URL: https://github.com/apache/arrow-rs/pull/8258#issuecomment-4805865721
Thank you for your contribution. Unfortunately, this pull request is stale because it has been open 60 days with no activity. Please remove the stale label or comment or this will be closed in 7 days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Parquet: change default type for parquet v2 from delta to delta-length-byte-array [arrow-rs]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on PR #8258:
URL: https://github.com/apache/arrow-rs/pull/8258#issuecomment-3399662768
The problem of DELTA_BYTE_ARRAY is both:
1. In most case, the delta is not better. It works better in the case of sharing-prefix ( like url ). For mostly equal strings / binarys, zstd can works well
2. DELTA_BYTE_ARRAY is also more memory consuming in decoder, this is also good to understand, for `DELTA_LENGTH_BYTE_ARRAY`, no more copies for buffer is required, just zero-copying the data is ok. But Delta might need construct the buffer( and hard to vectorize ). So it only works well in merely cases like sorted url or some sharing prefix cases
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Parquet: change default type for parquet v2 from delta to delta-length-byte-array [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #8258:
URL: https://github.com/apache/arrow-rs/pull/8258#issuecomment-3335997381
I also have no understanding about the implications of this change (e.g impact on output size, compatibility, etc)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org