You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Andre <an...@fucs.org> on 2017/03/03 04:54:19 UTC
Loss tolerance in NiFi
All,
A good number of NiFi related material makes the following statement about
NiFi:
Loss tolerant vs guaranteed delivery
I have not historically used NiFi in a loss tolerant environment but after
investigating the documentation it isn't clear to me how NiFi implements
loss tolerance?
Are we referring to Clustering? Flow expiration? To the ability to discard
flowfiles that get piped to the REL_FAILURE relationships? etc?
Would anyone mind shedding some light about what are we referring to when
presenting the "Loss tolerant vs guaranteed delivery"
I thank you in advance
Re: Loss tolerance in NiFi
Posted by Andre <an...@fucs.org>.
Andy, Bryan ,
Thanks, ok, so I wasn't off mark (I call it flow expiration in my original
email).
Now another question:
I was aware of expiration due to time, but not aware of discarding based on
queue levels.
Is there a way to do so and if not, should we allow users to discard
flowfiles that are stuck in a queue when it reaches a certain level?
Kind regards
On 4 Mar 2017 05:29, "Andy LoPresto" <al...@apache.org> wrote:
I agree with Bryan’s interpretation here. My understanding of that feature
reference is in relation to high volume streams of data coming in and the
DFM making proactive and/or realtime decisions about how to prioritize data
when they don’t have the capability to capture/operate on all of it.
Flowfile expiration and routing to failure (the provenance event is called
“drop” for a reason) are explicit design decisions that address the
scenario in legacy systems when "bad/less valuable/not currently able to be
operated on" data could prevent the "better/more valuable" data from
flowing.
Andy LoPresto
alopresto@apache.org
*alopresto.apache@gmail.com <al...@gmail.com>*
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
On Mar 3, 2017, at 6:15 AM, Bryan Bende <bb...@gmail.com> wrote:
Andre,
My understanding of that statement is that NiFi lets the user decide
how to build their dataflow, and allows them to build it in a way that
favors guaranteed delivery, or favors loss tolerance, depending on
their needs.
I would think loss tolerance is referencing the idea of expiring data
from queues after a certain threshold, and maybe terminating failure
relationships.
The overall point being that the user makes the decision and not the
framework, which is the same motto for a lot of other parts of NiFi.
-Bryan
On Thu, Mar 2, 2017 at 11:54 PM, Andre <an...@fucs.org> wrote:
All,
A good number of NiFi related material makes the following statement about
NiFi:
Loss tolerant vs guaranteed delivery
I have not historically used NiFi in a loss tolerant environment but after
investigating the documentation it isn't clear to me how NiFi implements
loss tolerance?
Are we referring to Clustering? Flow expiration? To the ability to discard
flowfiles that get piped to the REL_FAILURE relationships? etc?
Would anyone mind shedding some light about what are we referring to when
presenting the "Loss tolerant vs guaranteed delivery"
I thank you in advance
Re: Loss tolerance in NiFi
Posted by Andy LoPresto <al...@apache.org>.
I agree with Bryan’s interpretation here. My understanding of that feature reference is in relation to high volume streams of data coming in and the DFM making proactive and/or realtime decisions about how to prioritize data when they don’t have the capability to capture/operate on all of it. Flowfile expiration and routing to failure (the provenance event is called “drop” for a reason) are explicit design decisions that address the scenario in legacy systems when "bad/less valuable/not currently able to be operated on" data could prevent the "better/more valuable" data from flowing.
Andy LoPresto
alopresto@apache.org
alopresto.apache@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> On Mar 3, 2017, at 6:15 AM, Bryan Bende <bb...@gmail.com> wrote:
>
> Andre,
>
> My understanding of that statement is that NiFi lets the user decide
> how to build their dataflow, and allows them to build it in a way that
> favors guaranteed delivery, or favors loss tolerance, depending on
> their needs.
>
> I would think loss tolerance is referencing the idea of expiring data
> from queues after a certain threshold, and maybe terminating failure
> relationships.
>
> The overall point being that the user makes the decision and not the
> framework, which is the same motto for a lot of other parts of NiFi.
>
> -Bryan
>
>
> On Thu, Mar 2, 2017 at 11:54 PM, Andre <an...@fucs.org> wrote:
>> All,
>>
>> A good number of NiFi related material makes the following statement about
>> NiFi:
>>
>> Loss tolerant vs guaranteed delivery
>>
>> I have not historically used NiFi in a loss tolerant environment but after
>> investigating the documentation it isn't clear to me how NiFi implements
>> loss tolerance?
>>
>> Are we referring to Clustering? Flow expiration? To the ability to discard
>> flowfiles that get piped to the REL_FAILURE relationships? etc?
>>
>> Would anyone mind shedding some light about what are we referring to when
>> presenting the "Loss tolerant vs guaranteed delivery"
>>
>> I thank you in advance
Re: Loss tolerance in NiFi
Posted by Bryan Bende <bb...@gmail.com>.
Andre,
My understanding of that statement is that NiFi lets the user decide
how to build their dataflow, and allows them to build it in a way that
favors guaranteed delivery, or favors loss tolerance, depending on
their needs.
I would think loss tolerance is referencing the idea of expiring data
from queues after a certain threshold, and maybe terminating failure
relationships.
The overall point being that the user makes the decision and not the
framework, which is the same motto for a lot of other parts of NiFi.
-Bryan
On Thu, Mar 2, 2017 at 11:54 PM, Andre <an...@fucs.org> wrote:
> All,
>
> A good number of NiFi related material makes the following statement about
> NiFi:
>
> Loss tolerant vs guaranteed delivery
>
> I have not historically used NiFi in a loss tolerant environment but after
> investigating the documentation it isn't clear to me how NiFi implements
> loss tolerance?
>
> Are we referring to Clustering? Flow expiration? To the ability to discard
> flowfiles that get piped to the REL_FAILURE relationships? etc?
>
> Would anyone mind shedding some light about what are we referring to when
> presenting the "Loss tolerant vs guaranteed delivery"
>
> I thank you in advance