FlowFile Expiration - Lineage vs Queue Times

["Peter Wicks (pwicks)" <pw...@micron.com>]

I was thinking it would be nice to expire FlowFile's based on their time in queue, in addition to the current option of their total lineage time (as in, have both options available).
Any thoughts on pros/cons of having this available?

Thanks,
  Peter

Re: [EXT] Re: FlowFile Expiration - Lineage vs Queue Times

[Brian Ghigiarelli <br...@gmail.com>]

To add to the back pressure management options, we wrote a custom
"RouteOnBackPressure" processor [1] that would allow us to re-route the
incoming flow after outgoing queues hit their backpressure thresholds. The
processor itself doesn't handle anything with expiration (though output
queues could), and requires our third-party NAR bundle [2], but it works
well for handling surges.

[1]
https://github.com/Asymmetrik/nifi-nar-bundles/blob/develop/nifi-asymmetrik-standard-bundle/nifi-asymmetrik-standard-processors/src/main/java/com/asymmetrik/nifi/processors/RouteOnBackPressure.java
[2] https://github.com/Asymmetrik/nifi-nar-bundles

On Wed, Jul 24, 2019 at 9:18 AM Peter Wicks (pwicks) <pw...@micron.com>
wrote:

> Thanks for the input Mark, I can definitely see that being valuable.  That
> leads to some new ideas.
>
> We could have an, "Expiration Strategy" drop down.  Options might be:
>
>  - Lineage Expiration (default, keeps backwards compat.)
>  - Queue Expiration
>  - Back Pressure + Queue Expiration (which would follow the rules Mark
> described).
>
> As for the order, I'm not confident I remember how it works right now.
>
> Thanks,
>   Peter
>
>
> -----Original Message-----
> From: Mark Bean <ma...@gmail.com>
> Sent: Wednesday, July 24, 2019 7:08 AM
> To: dev@nifi.apache.org
> Subject: [EXT] Re: FlowFile Expiration - Lineage vs Queue Times
>
> On a similar note, we recently had a case where it would be desirable for
> the flowfile expiration to kick in only after the flowfile size/count back
> pressure limits have been reached. In other words, once a back pressure
> (size) limit is reached, it would be desirable to then remove flowfiles -
> beginning with the oldest first - until the back pressure limit is no
> longer in violation.
>
> Thanks,
> Mark
>
> On Tue, Jul 23, 2019 at 3:41 PM Peter Wicks (pwicks) <pw...@micron.com>
> wrote:
>
> > I was thinking it would be nice to expire FlowFile's based on their
> > time in queue, in addition to the current option of their total
> > lineage time (as in, have both options available).
> > Any thoughts on pros/cons of having this available?
> >
> > Thanks,
> >   Peter
> >
>

RE: [EXT] Re: FlowFile Expiration - Lineage vs Queue Times

["Peter Wicks (pwicks)" <pw...@micron.com>]

Thanks for the input Mark, I can definitely see that being valuable.  That leads to some new ideas.

We could have an, "Expiration Strategy" drop down.  Options might be:

 - Lineage Expiration (default, keeps backwards compat.)
 - Queue Expiration
 - Back Pressure + Queue Expiration (which would follow the rules Mark described).

As for the order, I'm not confident I remember how it works right now.

Thanks,
  Peter


-----Original Message-----
From: Mark Bean <ma...@gmail.com> 
Sent: Wednesday, July 24, 2019 7:08 AM
To: dev@nifi.apache.org
Subject: [EXT] Re: FlowFile Expiration - Lineage vs Queue Times

On a similar note, we recently had a case where it would be desirable for the flowfile expiration to kick in only after the flowfile size/count back pressure limits have been reached. In other words, once a back pressure
(size) limit is reached, it would be desirable to then remove flowfiles - beginning with the oldest first - until the back pressure limit is no longer in violation.

Thanks,
Mark

On Tue, Jul 23, 2019 at 3:41 PM Peter Wicks (pwicks) <pw...@micron.com>
wrote:

> I was thinking it would be nice to expire FlowFile's based on their 
> time in queue, in addition to the current option of their total 
> lineage time (as in, have both options available).
> Any thoughts on pros/cons of having this available?
>
> Thanks,
>   Peter
>

Re: FlowFile Expiration - Lineage vs Queue Times

[Mark Bean <ma...@gmail.com>]

On a similar note, we recently had a case where it would be desirable for
the flowfile expiration to kick in only after the flowfile size/count back
pressure limits have been reached. In other words, once a back pressure
(size) limit is reached, it would be desirable to then remove flowfiles -
beginning with the oldest first - until the back pressure limit is no
longer in violation.

Thanks,
Mark

On Tue, Jul 23, 2019 at 3:41 PM Peter Wicks (pwicks) <pw...@micron.com>
wrote:

> I was thinking it would be nice to expire FlowFile's based on their time
> in queue, in addition to the current option of their total lineage time (as
> in, have both options available).
> Any thoughts on pros/cons of having this available?
>
> Thanks,
>   Peter
>