You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by GitBox <gi...@apache.org> on 2022/12/14 16:58:51 UTC
[GitHub] [arrow] pitrou opened a new issue, #14951: [C++][Parquet] Add benchmarks for DELTA_BINARY_PACKED
pitrou opened a new issue, #14951:
URL: https://github.com/apache/arrow/issues/14951
### Describe the enhancement requested
Now that we support the DELTA_BINARY_PACKED encoding for both reading and writing, we can add some benchmarks for it.
### Component(s)
C++, Parquet
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on issue #14951: [C++][Parquet] Add benchmarks for DELTA_BINARY_PACKED
Posted by GitBox <gi...@apache.org>.
pitrou commented on issue #14951:
URL: https://github.com/apache/arrow/issues/14951#issuecomment-1368217065
> By the way, should we generate the input using random? Seems that if we use a incremental number or fixed number, the data of `DELTA_BINARY_PACKED` should be well compressed and optimized, which is kind of unfair
Use random but with two different magnitudes (narrow or wide) such as to exercise both the easily compressible and poorly compressible cases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] mapleFU commented on issue #14951: [C++][Parquet] Add benchmarks for DELTA_BINARY_PACKED
Posted by GitBox <gi...@apache.org>.
mapleFU commented on issue #14951:
URL: https://github.com/apache/arrow/issues/14951#issuecomment-1368222472
By the way, I want to ask that what's the differences between `column_io_benchmark`, `column_reader_benchmark` and `encoding_benchmark`? And do we have some benchmark for `compress + encoding`? Because some encoding will not benifit a lot when encoding, but will greatly improve the performance or saving the space when compress and encoding, like byte_stream_split.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] mapleFU commented on issue #14951: [C++][Parquet] Add benchmarks for DELTA_BINARY_PACKED
Posted by GitBox <gi...@apache.org>.
mapleFU commented on issue #14951:
URL: https://github.com/apache/arrow/issues/14951#issuecomment-1369783356
> I'm not convinced it's useful to add compression here. We're not writing the compression routines ourselves.
Yes, you're right. But I think if compression is not tested, maybe we will only have benchmark for partial of reading parquet. For example, some benchmark would be "the decoder is how much slower than PlainDecoder" when CPU prefetch works well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on issue #14951: [C++][Parquet] Add benchmarks for DELTA_BINARY_PACKED
Posted by GitBox <gi...@apache.org>.
pitrou commented on issue #14951:
URL: https://github.com/apache/arrow/issues/14951#issuecomment-1369752966
> And do we have some benchmark for `compress + encoding`? Because some encoding will not benifit a lot when encoding, but will greatly improve the performance or saving the space when compress and encoding, like byte_stream_split.
I'm not convinced it's useful to add compression here. We're not writing the compression routines ourselves.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] mapleFU commented on issue #14951: [C++][Parquet] Add benchmarks for DELTA_BINARY_PACKED
Posted by GitBox <gi...@apache.org>.
mapleFU commented on issue #14951:
URL: https://github.com/apache/arrow/issues/14951#issuecomment-1366427834
Should it just follow `src/parquet/encoding_benchmark.cc`? I'd like to have a try.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] mapleFU commented on issue #14951: [C++][Parquet] Add benchmarks for DELTA_BINARY_PACKED
Posted by GitBox <gi...@apache.org>.
mapleFU commented on issue #14951:
URL: https://github.com/apache/arrow/issues/14951#issuecomment-1368216560
By the way, should we generate the input using random? Seems that if we use a incremental number or fixed number, the data of `DELTA_BINARY_PACKED` should be well compressed and optimized, which is kind of unfair
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou closed issue #14951: [C++][Parquet] Add benchmarks for DELTA_BINARY_PACKED
Posted by GitBox <gi...@apache.org>.
pitrou closed issue #14951: [C++][Parquet] Add benchmarks for DELTA_BINARY_PACKED
URL: https://github.com/apache/arrow/issues/14951
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] rok commented on issue #14951: [C++][Parquet] Add benchmarks for DELTA_BINARY_PACKED
Posted by GitBox <gi...@apache.org>.
rok commented on issue #14951:
URL: https://github.com/apache/arrow/issues/14951#issuecomment-1366834368
That seems like the right place @mapleFU. I think you can go ahead and open a PR! :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org