[jira] [Created] (NIFI-2835) GetAzureEventHub processor should leverage partition offset to better handle restarts

["Joseph Percivall (JIRA)" <ji...@apache.org>]

Joseph Percivall created NIFI-2835:
--------------------------------------

             Summary: GetAzureEventHub processor should leverage partition offset to better handle restarts
                 Key: NIFI-2835
                 URL: https://issues.apache.org/jira/browse/NIFI-2835
             Project: Apache NiFi
          Issue Type: Improvement
            Reporter: Joseph Percivall


The GetAzureEventHub processor utilizes the Azure client that consists of receivers for each partition. The processor stores them in a map[1] that gets cleared every time the processor is stopped[2]. These receivers have partition offsets which keep track of which message it's currently on and which it should receive next. So currently, when the processor is stopped/restarted, any tracking of which message is next to be received is lost.

If instead of clearing the map each time, we hold onto the receivers, or kept track of the partitionId/Offsets when stopping, (barring any relevant configuration changes) the processor would restart exactly where it left off with no loss of data.

This would work very well with NIFI-2826.


[1]https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/eventhub/GetAzureEventHub.java#L122
[2] https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/eventhub/GetAzureEventHub.java#L229



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)