[GitHub] [incubator-pinot] FelixGV commented on pull request #6716: Upgrade to avro-1.92 and resolve transitive dependencies.

[GitBox <gi...@apache.org>]

FelixGV commented on pull request #6716:
URL: https://github.com/apache/incubator-pinot/pull/6716#issuecomment-808856352


   Hi,
   
   Have you considered leveraging the [avro-util](https://github.com/linkedin/avro-util) library? There are two parts to this project that could be relevant to Pinot:
   
   1. The [helper](https://github.com/linkedin/avro-util/tree/master/helper) module provides a version-agnostic shim for dealing with Avro. Using this shim would allow Pinot to work with any version of Avro between 1.4 and 1.10, even though the Apache Avro project introduced many breaking API changes throughout these versions. Whichever version is found on the classpath will be dealt transparently.
   
   2. The [avro-fastserde](https://github.com/linkedin/avro-util/tree/master/avro-fastserde) module is an alternative implementation of Avro serde which does runtime code generation to create an optimized serde for each schema reader/writer pair. It is more CPU-efficient and in some cases more GC-efficient as well. We've noticed 2-10x serde performance increase with it, depending on schemas. This module itself leverages the aforementioned helper module in order to achieve compatibility with Avro 1.4-1.10.
   
   Let me know if you have any questions.
   
   -F


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org