You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by "ASF subversion and git services (Jira)" <ji...@apache.org> on 2020/11/18 15:04:00 UTC
[jira] [Commented] (NIFI-8021) Allow ConsumeKafka processors to use
static partition mapping
[ https://issues.apache.org/jira/browse/NIFI-8021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17234697#comment-17234697 ]
ASF subversion and git services commented on NIFI-8021:
-------------------------------------------------------
Commit 7848ba59a250ba4ae98d7062869bcf01a9a58b71 in nifi's branch refs/heads/main from Mark Payne
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=7848ba5 ]
NIFI-8021: Provide the ability to pin partitions to particular hosts when using ConsumeKafka processors
Signed-off-by: Pierre Villard <pi...@gmail.com>
This closes #4672.
> Allow ConsumeKafka processors to use static partition mapping
> -------------------------------------------------------------
>
> Key: NIFI-8021
> URL: https://issues.apache.org/jira/browse/NIFI-8021
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
> Reporter: Mark Payne
> Assignee: Mark Payne
> Priority: Major
> Time Spent: 20m
> Remaining Estimate: 0h
>
> With the ConsumeKafka processors, we use the KafkaConsumer.subscribe() method, which is generally what is desirable. However, for a case where strict FIFO order per partition is critical, as may be the case with CDC use cases among others, we should allow users to pin a particular partition to a particular node. This way, even if the node is restarted or stops pulling data for a while, its partitions are not re-assigned, so there's no risk of messages being delivered out of order.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)