RE: [EXT] Re: FlowFile Expiration - Lineage vs Queue Times

["Peter Wicks (pwicks)" <pw...@micron.com>]

Thanks for the input Mark, I can definitely see that being valuable.  That leads to some new ideas.

We could have an, "Expiration Strategy" drop down.  Options might be:

 - Lineage Expiration (default, keeps backwards compat.)
 - Queue Expiration
 - Back Pressure + Queue Expiration (which would follow the rules Mark described).

As for the order, I'm not confident I remember how it works right now.

Thanks,
  Peter


-----Original Message-----
From: Mark Bean <ma...@gmail.com> 
Sent: Wednesday, July 24, 2019 7:08 AM
To: dev@nifi.apache.org
Subject: [EXT] Re: FlowFile Expiration - Lineage vs Queue Times

On a similar note, we recently had a case where it would be desirable for the flowfile expiration to kick in only after the flowfile size/count back pressure limits have been reached. In other words, once a back pressure
(size) limit is reached, it would be desirable to then remove flowfiles - beginning with the oldest first - until the back pressure limit is no longer in violation.

Thanks,
Mark

On Tue, Jul 23, 2019 at 3:41 PM Peter Wicks (pwicks) <pw...@micron.com>
wrote:

> I was thinking it would be nice to expire FlowFile's based on their 
> time in queue, in addition to the current option of their total 
> lineage time (as in, have both options available).
> Any thoughts on pros/cons of having this available?
>
> Thanks,
>   Peter
>

Re: [EXT] Re: FlowFile Expiration - Lineage vs Queue Times

[Brian Ghigiarelli <br...@gmail.com>]

To add to the back pressure management options, we wrote a custom
"RouteOnBackPressure" processor [1] that would allow us to re-route the
incoming flow after outgoing queues hit their backpressure thresholds. The
processor itself doesn't handle anything with expiration (though output
queues could), and requires our third-party NAR bundle [2], but it works
well for handling surges.

[1]
https://github.com/Asymmetrik/nifi-nar-bundles/blob/develop/nifi-asymmetrik-standard-bundle/nifi-asymmetrik-standard-processors/src/main/java/com/asymmetrik/nifi/processors/RouteOnBackPressure.java
[2] https://github.com/Asymmetrik/nifi-nar-bundles

On Wed, Jul 24, 2019 at 9:18 AM Peter Wicks (pwicks) <pw...@micron.com>
wrote:

> Thanks for the input Mark, I can definitely see that being valuable.  That
> leads to some new ideas.
>
> We could have an, "Expiration Strategy" drop down.  Options might be:
>
>  - Lineage Expiration (default, keeps backwards compat.)
>  - Queue Expiration
>  - Back Pressure + Queue Expiration (which would follow the rules Mark
> described).
>
> As for the order, I'm not confident I remember how it works right now.
>
> Thanks,
>   Peter
>
>
> -----Original Message-----
> From: Mark Bean <ma...@gmail.com>
> Sent: Wednesday, July 24, 2019 7:08 AM
> To: dev@nifi.apache.org
> Subject: [EXT] Re: FlowFile Expiration - Lineage vs Queue Times
>
> On a similar note, we recently had a case where it would be desirable for
> the flowfile expiration to kick in only after the flowfile size/count back
> pressure limits have been reached. In other words, once a back pressure
> (size) limit is reached, it would be desirable to then remove flowfiles -
> beginning with the oldest first - until the back pressure limit is no
> longer in violation.
>
> Thanks,
> Mark
>
> On Tue, Jul 23, 2019 at 3:41 PM Peter Wicks (pwicks) <pw...@micron.com>
> wrote:
>
> > I was thinking it would be nice to expire FlowFile's based on their
> > time in queue, in addition to the current option of their total
> > lineage time (as in, have both options available).
> > Any thoughts on pros/cons of having this available?
> >
> > Thanks,
> >   Peter
> >
>