You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@parquet.apache.org by "gowa (via GitHub)" <gi...@apache.org> on 2025/01/12 22:54:29 UTC
[PR] [WIP] [PREVIEW] use ByteBuddy code generation to write proto to parquet faster [parquet-java]
gowa opened a new pull request, #3121:
URL: https://github.com/apache/parquet-java/pull/3121
Note: it is not a ready to merge pull request, but a request to check if the concept of using code generation for solving some performance issues, associated with the usage of protobuf reflection when writing or reading parquet files, is of potential interest of repository owners. I decided to verify the concept at a rather early stage due to a significant effort required to implement the change. Should the approach and a new optional dependency on ByteBuddy is found satisfactorily and potentially acceptable to be included into parquet-java, I will attempt to properly finish first the 'write' part and then the 'read' part (in terms of code quality and tests). Therefore, any feed is appreciated.
### Rationale for this change
We read and write a lot of parquet data, defined by protobuf schemas from Java. It is seen that this can be done faster than what is offered out of the box now.
The change introduced improves proto-to-parquet file writing performance by means of code generation (in my synthetic tests by around 50% with SNAPPY compression, especially, when structures have a lot of primitive type fields).
### What changes are included in this PR?
1. an extension point in MessageWriter that redirects writing to a generated on-the-fly class dealing with protobuf generated classes getters directly, not via Protobuf Java Reflection methods.
2. a separate class where all code generation logic is located.
### Are these changes tested?
current unit tests work fine.
### Are there any user-facing changes?
a configuration to disable code generation logic.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] [WIP] [PREVIEW] use ByteBuddy code generation to write proto to parquet faster [parquet-java]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #3121:
URL: https://github.com/apache/parquet-java/pull/3121#issuecomment-4323381634
This pull request has been automatically marked as stale because it has had no activity for at least 2 months. If you are still working on this change or plan to move it forward, please leave a comment or push a new commit so we know to keep it open. Otherwise, this PR will be closed automatically in about one month. Thank you for your contribution to Apache Parquet!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] [WIP] [PREVIEW] use ByteBuddy code generation to write proto to parquet faster [parquet-java]
Posted by "gowa (via GitHub)" <gi...@apache.org>.
gowa commented on PR #3121:
URL: https://github.com/apache/parquet-java/pull/3121#issuecomment-2608733828
Hi @gszadovszky , @wgtmac . Thank you for your feedback.
Yes, I see that it is a big feature and the implementation is far from being a simple fix. And, maybe, it should be a pluggable thing instead of being a first-class resident in the code. However, if you feel the changes can be incorporated into the main codebase, I could try to find someone to review ByteBuddy part and implement the reader part as well.
As for benchmarks. I've implemented some and committed. I attempted to replicate the original org.apache.parquet.benchmarks.WriteBenchmarks with some proto stuff in org.apache.parquet.benchmarks.ProtoWriteBenchmarks.
```
The result are as follows: the bigger number of fields (especially primitives), the bigger the gain.
E.g. for 100 int32 fields:
Benchmark (codegenMode) (protoClass) Mode Cnt Score Error Units
ProtoWriteBenchmarks.write1MRowsBS256MPS4MUncompressed OFF Test100Int32 ss 5 13.171 ± 1.206 s/op
ProtoWriteBenchmarks.write1MRowsBS256MPS4MUncompressed REQUIRED_ALL Test100Int32 ss 5 6.075 ± 1.258 s/op
ProtoWriteBenchmarks.write1MRowsBS256MPS8MUncompressed OFF Test100Int32 ss 5 13.304 ± 1.497 s/op
ProtoWriteBenchmarks.write1MRowsBS256MPS8MUncompressed REQUIRED_ALL Test100Int32 ss 5 6.235 ± 0.617 s/op
ProtoWriteBenchmarks.write1MRowsBS512MPS4MUncompressed OFF Test100Int32 ss 5 13.450 ± 3.429 s/op
ProtoWriteBenchmarks.write1MRowsBS512MPS4MUncompressed REQUIRED_ALL Test100Int32 ss 5 5.947 ± 0.430 s/op
ProtoWriteBenchmarks.write1MRowsBS512MPS8MUncompressed OFF Test100Int32 ss 5 13.433 ± 3.879 s/op
ProtoWriteBenchmarks.write1MRowsBS512MPS8MUncompressed REQUIRED_ALL Test100Int32 ss 5 6.523 ± 2.831 s/op
ProtoWriteBenchmarks.write1MRowsDefaultBlockAndPageSizeGZIP OFF Test100Int32 ss 5 13.288 ± 0.429 s/op
ProtoWriteBenchmarks.write1MRowsDefaultBlockAndPageSizeGZIP REQUIRED_ALL Test100Int32 ss 5 6.333 ± 0.444 s/op
ProtoWriteBenchmarks.write1MRowsDefaultBlockAndPageSizeSNAPPY OFF Test100Int32 ss 5 13.197 ± 1.396 s/op
ProtoWriteBenchmarks.write1MRowsDefaultBlockAndPageSizeSNAPPY REQUIRED_ALL Test100Int32 ss 5 6.855 ± 2.689 s/op
ProtoWriteBenchmarks.write1MRowsDefaultBlockAndPageSizeUncompressed OFF Test100Int32 ss 5 13.473 ± 1.930 s/op
ProtoWriteBenchmarks.write1MRowsDefaultBlockAndPageSizeUncompressed REQUIRED_ALL Test100Int32 ss 5 6.006 ± 0.285 s/op
```
For 30 int32 fields:
```
Benchmark (codegenMode) (protoClass) Mode Cnt Score Error Units
ProtoWriteBenchmarks.write1MRowsBS256MPS4MUncompressed OFF Test30Int32 ss 5 3.421 ± 1.303 s/op
ProtoWriteBenchmarks.write1MRowsBS256MPS4MUncompressed REQUIRED_ALL Test30Int32 ss 5 2.410 ± 0.357 s/op
ProtoWriteBenchmarks.write1MRowsBS256MPS8MUncompressed OFF Test30Int32 ss 5 3.396 ± 0.708 s/op
ProtoWriteBenchmarks.write1MRowsBS256MPS8MUncompressed REQUIRED_ALL Test30Int32 ss 5 2.362 ± 0.174 s/op
ProtoWriteBenchmarks.write1MRowsBS512MPS4MUncompressed OFF Test30Int32 ss 5 3.250 ± 0.721 s/op
ProtoWriteBenchmarks.write1MRowsBS512MPS4MUncompressed REQUIRED_ALL Test30Int32 ss 5 2.310 ± 0.168 s/op
ProtoWriteBenchmarks.write1MRowsBS512MPS8MUncompressed OFF Test30Int32 ss 5 3.447 ± 0.884 s/op
ProtoWriteBenchmarks.write1MRowsBS512MPS8MUncompressed REQUIRED_ALL Test30Int32 ss 5 2.416 ± 0.387 s/op
ProtoWriteBenchmarks.write1MRowsDefaultBlockAndPageSizeGZIP OFF Test30Int32 ss 5 3.156 ± 0.276 s/op
ProtoWriteBenchmarks.write1MRowsDefaultBlockAndPageSizeGZIP REQUIRED_ALL Test30Int32 ss 5 2.514 ± 0.687 s/op
ProtoWriteBenchmarks.write1MRowsDefaultBlockAndPageSizeSNAPPY OFF Test30Int32 ss 5 3.398 ± 0.853 s/op
ProtoWriteBenchmarks.write1MRowsDefaultBlockAndPageSizeSNAPPY REQUIRED_ALL Test30Int32 ss 5 2.501 ± 0.323 s/op
ProtoWriteBenchmarks.write1MRowsDefaultBlockAndPageSizeUncompressed OFF Test30Int32 ss 5 3.644 ± 3.423 s/op
ProtoWriteBenchmarks.write1MRowsDefaultBlockAndPageSizeUncompressed REQUIRED_ALL Test30Int32 ss 5 2.384 ± 0.203 s/op
```
For 30 strings ("fieldXX:XX"):
```
Benchmark (codegenMode) (protoClass) Mode Cnt Score Error Units
ProtoWriteBenchmarks.write1MRowsBS256MPS4MUncompressed OFF Test30String ss 5 9.426 ± 3.621 s/op
ProtoWriteBenchmarks.write1MRowsBS256MPS4MUncompressed REQUIRED_ALL Test30String ss 5 8.257 ± 1.113 s/op
ProtoWriteBenchmarks.write1MRowsBS256MPS8MUncompressed OFF Test30String ss 5 9.848 ± 1.141 s/op
ProtoWriteBenchmarks.write1MRowsBS256MPS8MUncompressed REQUIRED_ALL Test30String ss 5 8.302 ± 1.910 s/op
ProtoWriteBenchmarks.write1MRowsBS512MPS4MUncompressed OFF Test30String ss 5 10.216 ± 1.843 s/op
ProtoWriteBenchmarks.write1MRowsBS512MPS4MUncompressed REQUIRED_ALL Test30String ss 5 8.173 ± 1.419 s/op
ProtoWriteBenchmarks.write1MRowsBS512MPS8MUncompressed OFF Test30String ss 5 9.940 ± 1.680 s/op
ProtoWriteBenchmarks.write1MRowsBS512MPS8MUncompressed REQUIRED_ALL Test30String ss 5 8.242 ± 1.270 s/op
ProtoWriteBenchmarks.write1MRowsDefaultBlockAndPageSizeGZIP OFF Test30String ss 5 9.833 ± 1.010 s/op
ProtoWriteBenchmarks.write1MRowsDefaultBlockAndPageSizeGZIP REQUIRED_ALL Test30String ss 5 8.247 ± 1.284 s/op
ProtoWriteBenchmarks.write1MRowsDefaultBlockAndPageSizeSNAPPY OFF Test30String ss 5 9.638 ± 0.502 s/op
ProtoWriteBenchmarks.write1MRowsDefaultBlockAndPageSizeSNAPPY REQUIRED_ALL Test30String ss 5 7.935 ± 0.889 s/op
ProtoWriteBenchmarks.write1MRowsDefaultBlockAndPageSizeUncompressed OFF Test30String ss 5 9.968 ± 1.651 s/op
ProtoWriteBenchmarks.write1MRowsDefaultBlockAndPageSizeUncompressed REQUIRED_ALL Test30String ss 5 8.356 ± 1.319 s/op
```
For 5-7 fields the gain is negligeable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] [WIP] [PREVIEW] use ByteBuddy code generation to write proto to parquet faster [parquet-java]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #3121:
URL: https://github.com/apache/parquet-java/pull/3121#issuecomment-4550162190
Closing this pull request due to at least 3 months of inactivity. If you would like to continue the work, please feel free to reopen this pull request or open a new one. Thank you for your contribution to Apache Parquet!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] [WIP] [PREVIEW] use ByteBuddy code generation to write proto to parquet faster [parquet-java]
Posted by "gowa (via GitHub)" <gi...@apache.org>.
gowa commented on PR #3121:
URL: https://github.com/apache/parquet-java/pull/3121#issuecomment-2609457531
Hi @gszadovszky !
yes, with OFF is the old behavior. This whole codegeneration thing is currently implemented as an interceptor (Predicate). When OFF, then predicate is just x -> false, not bringing much overhead to the original impl and not even trying to generate classes.
I will check dependencies in details, but I've put bytebuddy as an optional one. If it is not in classpath, then for all modes codegen won't happen, safely falling back to the original implementation, except for REQUIRE_ALL mode, when we insist on codegeneration.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] [WIP] [PREVIEW] use ByteBuddy code generation to write proto to parquet faster [parquet-java]
Posted by "gszadovszky (via GitHub)" <gi...@apache.org>.
gszadovszky commented on PR #3121:
URL: https://github.com/apache/parquet-java/pull/3121#issuecomment-2609536099
Thanks, @gowa for the clarification.
It would be nice to add a description for the users in the [README](https://github.com/apache/parquet-java/blob/master/parquet-protobuf/README.md). If one would like to use ByteBuddy, what to do, what options do they have. How it will behave by default (if ByteBuddy is not on the classpath).
Overall, I'm good with this approach and your PR. If you would find somebody who can do a deeper review on the ByteBuddy related codes, I'll be more than happy to approve it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] [WIP] [PREVIEW] use ByteBuddy code generation to write proto to parquet faster [parquet-java]
Posted by "gszadovszky (via GitHub)" <gi...@apache.org>.
gszadovszky commented on PR #3121:
URL: https://github.com/apache/parquet-java/pull/3121#issuecomment-2609203251
Thank you for the results, @gowa. I'm convinced.
I am understand you code well that if `CodeGenMode.OFF` is used, then it is the original behavior without ByteBuddy? If that's the case, we might choose to have it as default for the upcoming release to give time for the users to test it. Then, we can change the default for the next release.
Another question is the dependency. What transitive dependencies ByteBuddy would get into `parquet-protobuf`? Does it have transitive dependencies that may conflict with `parquet-java` itself or the ecosystems who would use `parquet-protobuf`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] [WIP] [PREVIEW] use ByteBuddy code generation to write proto to parquet faster [parquet-java]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #3121: [WIP] [PREVIEW] use ByteBuddy code generation to write proto to parquet faster
URL: https://github.com/apache/parquet-java/pull/3121
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] [WIP] [PREVIEW] use ByteBuddy code generation to write proto to parquet faster [parquet-java]
Posted by "gowa (via GitHub)" <gi...@apache.org>.
gowa commented on PR #3121:
URL: https://github.com/apache/parquet-java/pull/3121#issuecomment-2609619760
@gszadovszky , all right. thanks. there is still a lot to do. like, implementing the read part. at least, I know that this all can make live. I will surely update the docs, etc. to make it a good quality commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] [WIP] [PREVIEW] use ByteBuddy code generation to write proto to parquet faster [parquet-java]
Posted by "gszadovszky (via GitHub)" <gi...@apache.org>.
gszadovszky commented on PR #3121:
URL: https://github.com/apache/parquet-java/pull/3121#issuecomment-2589251385
The noted performance gain is promising indeed. However, it would be nice to see actual numbers for different scenarios (flat columns, general nested columns, deeply nested columns) of read/write. You might even implement your performance tests in the module `parquet-benchmarks`.
Unfortunately, I'm not expert in `parquet-protobuf` either not even talking about `ByteBuddy`. For the final PR review it would be a great help to have someone who have some experience with `ByteBuddy` even if not being a Parquet committer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] [WIP] [PREVIEW] use ByteBuddy code generation to write proto to parquet faster [parquet-java]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on PR #3121:
URL: https://github.com/apache/parquet-java/pull/3121#issuecomment-2588624621
Thanks for your interest in contributing this! This seems to be a large feature and the performance gain is promising! However, I'm afraid that this PR may not get prompt review to do lack of active `parquet-protobuf` maintainers. I do not have any knowledge on ByteBuddy so it might take a long time to wrap it up. Is it possible to make it pluggable so the large portion of codegen logic do not have to exist in the parquet-java repo?
cc @gszadovszky @julienledem if you know someone can help review this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org