[jira] [Created] (NIFI-5861) Notify: Add support for age of duration

["Niklas (JIRA)" <ji...@apache.org>]

Niklas created NIFI-5861:
----------------------------

             Summary: Notify: Add support for age of duration
                 Key: NIFI-5861
                 URL: https://issues.apache.org/jira/browse/NIFI-5861
             Project: Apache NiFi
          Issue Type: Improvement
          Components: Extensions
            Reporter: Niklas


 

Since the Notify processor is writing to the cache it would make sense to have a "age of duration" property like the DetectDuplicate processor.

 

For example: I've 2 seperate flows which put Data from one DB to another DB. Flow 2 depends on Flow 1 due to Foreign Key. Both flows have GenerateTableFetch which listen to Updates in source DB.

At the end of Flow 1 it'll put all INSERTED IDs to a Notify Processor which put everything in the Cache.

 

Flow 2 runs simultaneously and it'll check if the Foreign Key Value exist in the DB that it'll stream the data. If the key doesn't exist it'll be routed to a DetectDuplicate and if it's a Duplicate (this ID already run in Flow 1 but isn't inserted yet) it'll be routed to a Wait Processor. The wait processor now waits for the Notify node on Flow 1.

 

This whole construct works fine. The wait processor removes the key from the cache if it's match but I'm writing all IDs to the cache, even IDs that doesn't hang in a wait processor.

 

I've tried to do a workaround with DetectDuplicate Processor that it'll only use Notify for things that hang in a wait processor but it didn't work that great due to bad timings.

 

It would be good to have a TTL setting similar to the DetectDuplicate processor for the Notify processor.

 

Additionally to this issue is that the Nifi Cache Server doesn't have a default TTL option to clear the cache automatically.

 

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)