[GitHub] justinborromeo opened a new pull request #6917: #6714-Implement-KCL-Deaggregation

[GitBox <gi...@apache.org>]

justinborromeo opened a new pull request #6917: #6714-Implement-KCL-Deaggregation
URL: https://github.com/apache/incubator-druid/pull/6917
 
 
   These changes address https://github.com/apache/incubator-druid/issues/6714.  Rather than having users download the Kinesis Client Library jar, this change adds the capability to perform deaggregation on Kinesis Producer Library-aggregated messages without the KCL dependency.  These changes were all written without any knowledge of how KCL implements deaggregation.
   
   Changes:
   
   1) Messages are checked to see if they are aggregations based on whether they have the magic numbers and their checksum is valid (see https://github.com/awslabs/amazon-kinesis-producer/blob/master/aggregation-format.md for protobuf schema).  If so, they are deaggregated using the Google protobuf library.  If not, they are processed in the same manner as before.
   
   2) The deaggregation config value was removed.  The new deaggregation method can handle both aggregated and non-aggregated messages.
   
   3) A maven step was added to generate a Java file that matches the KPL protobuf schema.
   
   I know my doc changes were minimal so if anyone has any suggestions about what else to write about, feel free to lmk.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org