[jira] [Updated] (CAMEL-12179) Implement Camel Aws-Kinesis Using Kinesis Client Library(KCL)

["Claus Ibsen (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/CAMEL-12179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Claus Ibsen updated CAMEL-12179:
--------------------------------
    Fix Version/s: Future

> Implement Camel Aws-Kinesis Using Kinesis Client Library(KCL)
> -------------------------------------------------------------
>
>                 Key: CAMEL-12179
>                 URL: https://issues.apache.org/jira/browse/CAMEL-12179
>             Project: Camel
>          Issue Type: New Feature
>          Components: camel-aws
>    Affects Versions: 2.17.0
>            Reporter: Hisao Furuichi
>            Assignee: Andrea Cosentino
>            Priority: Major
>             Fix For: Future
>
>
> Amazon provides 2 options for Kinesis Java Client:
> * *Kinesis Client Library(KCL)*
> * *Kinesis Streams API*
> "*Kinesis Streams API*" is a low level API set, and "*KCL*" is a rich library.
> In the official document, "*KCL*" is described like this:
> {quote}
> The KCL is compiled into your application to enable fault-tolerant consumption of data from the stream.
> The KCL ensures that for every shard there is a record processor running and processing that shard. 
> The library also simplifies reading data from the stream. 
> The KCL uses an Amazon DynamoDB table to store control data. It creates one table per application that is processing data.
> The KCL takes care of many of the complex tasks associated with distributed computing, such as load-balancing across multiple instances, responding to instance failures, checkpointing processed records, and reacting to resharding.
> The KCL enables you to focus on writing record processing logic.
> {quote}
> Since "*Kinesis Streams API*" is used to implement current aws-kinesis component, usability and scalability are not provided by default.
> * User needs to specify *shardID*(ENTESB-7517) which might be changed dynamically
> * If a Stream uses multiple shards, user needs to create multiple endpoints for it. (Increasing number of shards are tips to scale out. If there are 200 shards, user needs to maintain 200 endpoints)
> * If user wants to consume a same shard from multiple Camel runtimes, user needs to implement load-balancing
> By using "*KCL*" to implement *aws-kinesis*, above current difficulties/concern can be resolved.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)