You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by "Joe Witt (Jira)" <ji...@apache.org> on 2020/01/09 20:03:00 UTC
[jira] [Comment Edited] (NIFI-6998) PutAzureEventHub to utilize
Batch, Properties & Partitioning key
[ https://issues.apache.org/jira/browse/NIFI-6998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17012189#comment-17012189 ]
Joe Witt edited comment on NIFI-6998 at 1/9/20 8:02 PM:
--------------------------------------------------------
Thanks for the contrib and it looks like a nice change. Once we get some review traction the fix version can be established. We're working to get a 1.11 RC out now.
was (Author: joewitt):
Once we get some review traction the fix version can be established. We're working to get a 1.11 RC out now.
> PutAzureEventHub to utilize Batch, Properties & Partitioning key
> ----------------------------------------------------------------
>
> Key: NIFI-6998
> URL: https://issues.apache.org/jira/browse/NIFI-6998
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
> Affects Versions: 1.8.0
> Reporter: Martin Šimek
> Priority: Major
> Attachments: EH_01_flow.png, EH_02_flowFile.png, EH_03_processor_settings.png, EH_04_EventHub.png, EH_05_receivedMessage.png
>
> Original Estimate: 72h
> Time Spent: 10m
> Remaining Estimate: 71h 50m
>
> Current implementation of _PutAzureEventHub_ is too simplistic. It sends one flow file at a time synchronously which poses a significant bottlenec. It doesn't lift attributes into the message's _User defined properties_ section. And most important problem is that you can't specify the _partitioning key_ which plays critical role in a delivery consistency across multiple partitions in case there are more partitions. Without partitioning key multiple use-cases can't choose NiFi as an option.
> Aim of this improvement is to consume up to specified amount of flow files, lift all _attributes_ of each flow file into _User defined properties_ section of each message and let processor to use one of those attributes as a partitioning key.
> Each flow file in a batch is sent asynchronously which is way more faster then processor wait for all confirmations (or errors) and based on the result messages are transfered to success or failure output.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)