[GitHub] [pulsar] addisonj opened a new pull request #5180: [kinesis-source] Improvements to kinesis-source

[GitBox <gi...@apache.org>]

addisonj opened a new pull request #5180: [kinesis-source] Improvements to kinesis-source
URL: https://github.com/apache/pulsar/pull/5180
 
 
   ### Motivation
   
   This pull request improves the kinesis-source (and sink to some extent). Currently, the kinesis-source is actually broken, it never starts reading. This fixes that, but also does upgrades to make it more functionality. it also adds a few other features that clean up the kinesis integrations.
   
   Primarily:
   - Ports the kinesis-source to use v2 of the kinesis consumer library (kcl), kcl v2 adds supports for enhanced fan-out, which greatly enhances the ability of multiple consumers to not compete for resources. An option is provided to fall back to polling as well. 
   - Adds a few default plugins for the AWS credential plugin, these can be used for both the source and the sink. This allows for using either the default provider chain or to assume a role
   - Updates the kinesis-producer-library to 0.13. This release has no functionality changes, but updates the library to be published under the Apache license, which should reduce some concerns about linking against the library.
   
   ### Modifications
   
   - Adds kcl v2 library as well as other dependency changes, this brings in some v2 version of the AWS SDK
   - Reworks the record processor to the new v2 APIs of the KCL
   - Changes the credential provider plugin to support the v2 of the AWS SDK via a new method with a default implementation for any existing providers (i.e. no breaking API changes)
   - Adds two new credential provider plugins, one for the default provider change and one for the STS assume role
   
   One annoying bit here is that the jar gets a fair bit bigger as we have both v1 and v2 of the aws sdk here. Eventually, the kinesis producer library should move to v2 of the SDK and then we can have only the v2 of the aws sdk.
   
   ### Verifying this change
   
   - Some tests cover this functionality, however, an integration test is still needed.
   - this does some improvements to make that easier (allow you to use local dynamo for example)
   
   ### Does this pull request potentially affect one of the following parts:
   
   *If `yes` was chosen, please highlight the changes*
   
     - Dependencies (does it add or upgrade a dependency): yes
     - The public API: no
     - The schema: no
     - The default values of configurations:  yes (adds new config vals)
     - The wire protocol: no
     - The rest endpoints: no
     - The admin cli options: no
     - Anything that affects deployment: no
   
   ### Documentation
   
     - Does this pull request introduce a new feature? yes
     - If yes, how is the feature documented? not yet documented
     - If a feature is not applicable for documentation, explain why?
     - If a feature is not documented yet in this PR, please create a followup issue for adding the documentation
   
   The default provider plugins should be documented
   
   @david-streamlio if you want to take a look at this
   
   Fixes #5170 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services