You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@apex.apache.org by Siyuan Hua <si...@datatorrent.com> on 2015/12/15 19:42:27 UTC

New Kafka input operator

Hey guys,

I have just created a PR for the new Kafka input operator using new Kafka
consumer API

Some highlights of the operator

   1. Out-of-box One-to-one and one-to-many partition scheme support plus
   customizable partition schema
   2. Fault-tolerant
   3. At-least-once semantics for operator recovery (no matter which
   operator fails)
   4. At-least-once semantics for cold restart (no data loss even if you
   restart the application)
   5. Multi-cluster support, one operator can consume data from more than
   one kafka clusters
   6. Multi-topic support, one operator can subscribe multiple topics
   7. Throughput control support, you can throttle number of tuple for each
   streaming window

Things needs to be done,
More unit test for different partition scheme, application restart
More integration test
Metrics for operator
Bring back idempotency

Re: New Kafka input operator

Posted by Siyuan Hua <si...@datatorrent.com>.

Sorry, forgot the link
https://github.com/apache/incubator-apex-malhar/pull/134

On Tue, Dec 15, 2015 at 10:42 AM, Siyuan Hua <si...@datatorrent.com> wrote:

> Hey guys,
>
> I have just created a PR for the new Kafka input operator using new Kafka
> consumer API
>
> Some highlights of the operator
>
>    1. Out-of-box One-to-one and one-to-many partition scheme support plus
>    customizable partition schema
>    2. Fault-tolerant
>    3. At-least-once semantics for operator recovery (no matter which
>    operator fails)
>    4. At-least-once semantics for cold restart (no data loss even if you
>    restart the application)
>    5. Multi-cluster support, one operator can consume data from more than
>    one kafka clusters
>    6. Multi-topic support, one operator can subscribe multiple topics
>    7. Throughput control support, you can throttle number of tuple for
>    each streaming window
>
> Things needs to be done,
> More unit test for different partition scheme, application restart
> More integration test
> Metrics for operator
> Bring back idempotency
>
>