[jira] [Resolved] (NIFI-2732) ConsumeKafka 0.9 and 0.10 not handling partition reassignment case sufficiently

["Bryan Bende (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/NIFI-2732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Bryan Bende resolved NIFI-2732.
-------------------------------
    Resolution: Fixed

> ConsumeKafka 0.9 and 0.10 not handling partition reassignment case sufficiently
> -------------------------------------------------------------------------------
>
>                 Key: NIFI-2732
>                 URL: https://issues.apache.org/jira/browse/NIFI-2732
>             Project: Apache NiFi
>          Issue Type: Bug
>            Reporter: Joseph Witt
>            Assignee: Joseph Witt
>            Priority: Critical
>             Fix For: 1.1.0
>
>
> The new ConsumeKafka clients handle the threading model of the consumer api correctly.  However, they are not yet honoring partition reassignment cases sufficiently which means we could have avoidable cases of duplication.  By registering a partition reassignment listener we can handle it correctly.
> Further, the processor is loading subsequent polls of messages into memory rather than writing directly to the process session/disk.  We could write them to disk and achieve far better performance and efficiency.  Early testing shows easily achieving 100MB/s sustained per thread on a simple laptop setup with defaults which will scale very nicely on a legit installed content repository.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)