[D] Sorting and encoding of dictionary pages [arrow-rs]

["mobiusklein (via GitHub)" <gi...@apache.org>]

GitHub user mobiusklein created a discussion: Sorting and encoding of dictionary pages

I am working on a schema for storing multi-dimensional signal (float32/float64 type) from a mass spectrometer in Parquet, using Rust for the prototypal implementation. In some cases, dictionary encoding has helped as one dimension's values are repeated many times as another dimension varies.

Sometimes, that dictionary can be quite big, and it may not compress well on its own if stored with the plain encoding, based upon experiments where I collected the set of all unique values and just compressed them with Zstd. If I sorted them and byte shuffled them (as in the `BYTE_STREAM_SPLIT` encoding), the compression improves substantially. Reading https://parquet.apache.org/docs/file-format/metadata/#page-header, it looks like there is a flag on the `DictionaryPageHeader` that says if the dictionary is sorted, and that the dictionary page can be encoded. 

So far as I can tell, this implementation doesn't support writing out a sorted dictionary, or using any encoding other than `PLAIN` on that page, correct? Is this something that is technically supported but not implemented here, or not supported by the Parquet format definition?

If I understand what'd be involved, sorting the dictionary page wouldn't force new behavior on a reader, but using an encoding other than `PLAIN` might?

GitHub link: https://github.com/apache/arrow-rs/discussions/8778

----
This is an automatically sent email for github@arrow.apache.org.
To unsubscribe, please send an email to: github-unsubscribe@arrow.apache.org

Re: [D] Sorting and encoding of dictionary pages [arrow-rs]

["mobiusklein (via GitHub)" <gi...@apache.org>]

GitHub user mobiusklein added a comment to the discussion: Sorting and encoding of dictionary pages

Thank you. That answers my questions. I'll collect some data to help make the case, but without a transformation to make the data more compressible, the result won't be as good.

GitHub link: https://github.com/apache/arrow-rs/discussions/8778#discussioncomment-15190784

----
This is an automatically sent email for github@arrow.apache.org.
To unsubscribe, please send an email to: github-unsubscribe@arrow.apache.org

Re: [D] Sorting and encoding of dictionary pages [arrow-rs]

["etseidl (via GitHub)" <gi...@apache.org>]

GitHub user etseidl added a comment to the discussion: Sorting and encoding of dictionary pages

Dictionary pages must be encoded using `PLAIN` encoding ([ref](https://github.com/apache/parquet-format/blob/master/Encodings.md#dictionary-encoding-plain_dictionary--2-and-rle_dictionary--8)). The Parquet community is exploring new encoding options right now (https://lists.apache.org/thread/djnbbcnft0fqm9ldby2q96nbtrwqz477), you might want to ask on the parquet dev list if there's a desire to expand the possibilities for dictionary encoding.

Being able to sort the dictionary would be a nice addition, but I'm not sure what the level of effort would be. Please feel free to create an issue to request this feature.

GitHub link: https://github.com/apache/arrow-rs/discussions/8778#discussioncomment-14882050

----
This is an automatically sent email for github@arrow.apache.org.
To unsubscribe, please send an email to: github-unsubscribe@arrow.apache.org

Re: [D] Sorting and encoding of dictionary pages [arrow-rs]

["mobiusklein (via GitHub)" <gi...@apache.org>]

GitHub user mobiusklein closed a discussion: Sorting and encoding of dictionary pages

I am working on a schema for storing multi-dimensional signal (float32/float64 type) from a mass spectrometer in Parquet, using Rust for the prototypal implementation. In some cases, dictionary encoding has helped as one dimension's values are repeated many times as another dimension varies.

Sometimes, that dictionary can be quite big, and it may not compress well on its own if stored with the plain encoding, based upon experiments where I collected the set of all unique values and just compressed them with Zstd. If I sorted them and byte shuffled them (as in the `BYTE_STREAM_SPLIT` encoding), the compression improves substantially. Reading https://parquet.apache.org/docs/file-format/metadata/#page-header, it looks like there is a flag on the `DictionaryPageHeader` that says if the dictionary is sorted, and that the dictionary page can be encoded. 

So far as I can tell, this implementation doesn't support writing out a sorted dictionary, or using any encoding other than `PLAIN` on that page, correct? Is this something that is technically supported but not implemented here, or not supported by the Parquet format definition?

If I understand what'd be involved, sorting the dictionary page wouldn't force new behavior on a reader, but using an encoding other than `PLAIN` might?

GitHub link: https://github.com/apache/arrow-rs/discussions/8778

----
This is an automatically sent email for github@arrow.apache.org.
To unsubscribe, please send an email to: github-unsubscribe@arrow.apache.org