[jira] [Created] (NIFI-2732) ConsumeKafka 0.9 and 0.10 not handling partition reassignment case sufficiently

["Joseph Witt (JIRA)" <ji...@apache.org>]

Joseph Witt created NIFI-2732:
---------------------------------

             Summary: ConsumeKafka 0.9 and 0.10 not handling partition reassignment case sufficiently
                 Key: NIFI-2732
                 URL: https://issues.apache.org/jira/browse/NIFI-2732
             Project: Apache NiFi
          Issue Type: Bug
            Reporter: Joseph Witt
            Assignee: Joseph Witt
            Priority: Critical


The new ConsumeKafka clients handle the threading model of the consumer api correctly.  However, they are not yet honoring partition reassignment cases sufficiently which means we could have avoidable cases of duplication.  By registering a partition reassignment listener we can handle it correctly.

Further, the processor is loading subsequent polls of messages into memory rather than writing directly to the process session/disk.  We could write them to disk and achieve far better performance and efficiency.  Early testing shows easily achieving 100MB/s sustained per thread on a simple laptop setup with defaults which will scale very nicely on a legit installed content repository.





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)