You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by "Joseph Witt (JIRA)" <ji...@apache.org> on 2016/09/06 12:43:21 UTC
[jira] [Commented] (NIFI-2732) ConsumeKafka 0.9 and 0.10 not
handling partition reassignment case sufficiently
[ https://issues.apache.org/jira/browse/NIFI-2732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15467321#comment-15467321 ]
Joseph Witt commented on NIFI-2732:
-----------------------------------
had originally put the wrong JIRA on the commit. Fixed. here is the PR details
GitHub user joewitt opened a pull request:
https://github.com/apache/nifi/pull/987
NIFI-2372 ensure session and consumer aligned and has registered reba…
…lance listener. Make consumption far more memory and process efficient.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/joewitt/incubator-nifi NIFI-2732
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/nifi/pull/987.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #987
----
commit 6cca1eed679e8b9e3dbc020fa4de589e4578df92
Author: joewitt <jo...@apache.org>
Date: 2016-08-31T05:25:12Z
NIFI-2732 ensure session and consumer aligned and has registered rebalance listener. Make consumption far more memory and process efficient.
----
> ConsumeKafka 0.9 and 0.10 not handling partition reassignment case sufficiently
> -------------------------------------------------------------------------------
>
> Key: NIFI-2732
> URL: https://issues.apache.org/jira/browse/NIFI-2732
> Project: Apache NiFi
> Issue Type: Bug
> Reporter: Joseph Witt
> Assignee: Joseph Witt
> Priority: Critical
> Fix For: 1.1.0
>
>
> The new ConsumeKafka clients handle the threading model of the consumer api correctly. However, they are not yet honoring partition reassignment cases sufficiently which means we could have avoidable cases of duplication. By registering a partition reassignment listener we can handle it correctly.
> Further, the processor is loading subsequent polls of messages into memory rather than writing directly to the process session/disk. We could write them to disk and achieve far better performance and efficiency. Early testing shows easily achieving 100MB/s sustained per thread on a simple laptop setup with defaults which will scale very nicely on a legit installed content repository.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)