You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@nifi.apache.org by "Malthe Borch (Jira)" <ji...@apache.org> on 2022/06/01 15:29:00 UTC

[jira] [Commented] (NIFI-2835) GetAzureEventHub processor should leverage partition offset to better handle restarts

    [ https://issues.apache.org/jira/browse/NIFI-2835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17544975#comment-17544975 ] 

Malthe Borch commented on NIFI-2835:
------------------------------------

Why isn't GetAzureEventHub checkpointing using processor state? I understand that ConsumeAzureEventHub can use blob storage to do the same but this seems unnecessary when you've got processor state available.

> GetAzureEventHub processor should leverage partition offset to better handle restarts
> -------------------------------------------------------------------------------------
>
>                 Key: NIFI-2835
>                 URL: https://issues.apache.org/jira/browse/NIFI-2835
>             Project: Apache NiFi
>          Issue Type: Improvement
>            Reporter: Joe Percivall
>            Assignee: Eric Ulicny
>            Priority: Major
>
> The GetAzureEventHub processor utilizes the Azure client that consists of receivers for each partition. The processor stores them in a map[1] that gets cleared every time the processor is stopped[2]. These receivers have partition offsets which keep track of which message it's currently on and which it should receive next. So currently, when the processor is stopped/restarted, any tracking of which message is next to be received is lost.
> If instead of clearing the map each time, we hold onto the receivers, or kept track of the partitionId/Offsets when stopping, (barring any relevant configuration changes) the processor would restart exactly where it left off with no loss of data.
> This would work very well with NIFI-2826.
> [1]https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/eventhub/GetAzureEventHub.java#L122
> [2] https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/eventhub/GetAzureEventHub.java#L229



--
This message was sent by Atlassian Jira
(v8.20.7#820007)