You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by "Joe Witt (Jira)" <ji...@apache.org> on 2020/01/14 04:32:00 UTC
[jira] [Resolved] (NIFI-6998) PutAzureEventHub to utilize Batch,
Properties & Partitioning key
[ https://issues.apache.org/jira/browse/NIFI-6998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Joe Witt resolved NIFI-6998.
----------------------------
Resolution: Fixed
Thanks for the Contrib Martin.
Thanks for the reviews Pierre and Shayne.
+1 merged to master after local build verification and travis all clear on 8 and 11.
> PutAzureEventHub to utilize Batch, Properties & Partitioning key
> ----------------------------------------------------------------
>
> Key: NIFI-6998
> URL: https://issues.apache.org/jira/browse/NIFI-6998
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
> Affects Versions: 1.8.0
> Reporter: Martin Šimek
> Priority: Major
> Fix For: 1.11.0
>
> Attachments: EH_01_flow.png, EH_02_flowFile.png, EH_03_processor_settings.png, EH_04_EventHub.png, EH_05_receivedMessage.png
>
> Original Estimate: 72h
> Time Spent: 1h 40m
> Remaining Estimate: 70h 20m
>
> Current implementation of _PutAzureEventHub_ is too simplistic. It sends one flow file at a time synchronously which poses a significant bottlenec. It doesn't lift attributes into the message's _User defined properties_ section. And most important problem is that you can't specify the _partitioning key_ which plays critical role in a delivery consistency across multiple partitions in case there are more partitions. Without partitioning key multiple use-cases can't choose NiFi as an option.
> Aim of this improvement is to consume up to specified amount of flow files, lift all _attributes_ of each flow file into _User defined properties_ section of each message and let processor to use one of those attributes as a partitioning key.
> Each flow file in a batch is sent asynchronously which is way more faster then processor wait for all confirmations (or errors) and based on the result messages are transfered to success or failure output.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)