You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Manish Gupta 8 <mg...@sapient.com> on 2016/10/14 16:38:05 UTC
Penalize Flow File on Failure
Hello Everyone,
In some of the processors I have seen that flow files on failure are not being penalized. For example - Kite processors like ConvertJsonToAvro. Is there some specific reason why some processors have different behavior?
I think every processor should penalize every non-success relationship. This way, we can hold those failed files (using a self-referencing loop and some decent penalty duration) in the queue itself and debug and resolve later.
Regards,
Manish
Re: Penalize Flow File on Failure
Posted by Matt Burgess <ma...@apache.org>.
Manish,
You could use ExecuteScript with Groovy and the following:
def flowFile = session.get()
if(!flowFile) return
flowFile = session.penalize(flowFile)
session.transfer(flowFile, REL_SUCCESS)
Then you can route "success" back to the processor. You'd set the
Penalty Duration on the ExecuteScript processor configuration
(Settings tab).
Regards,
Matt
On Fri, Oct 14, 2016 at 1:01 PM, Manish Gupta 8 <mg...@sapient.com> wrote:
> Thanks Matt. As a workaround, is there a processor that does not modify the flow file (content or attribute) at all, and I can use it to delay the self-referencing flow files to hit the main processor again immediately?
>
> Regards,
> Manish
>
> -----Original Message-----
> From: Matt Burgess [mailto:mattyb149@apache.org]
> Sent: Friday, October 14, 2016 10:25 PM
> To: users@nifi.apache.org
> Subject: Re: Penalize Flow File on Failure
>
> Manish,
>
> The use of penalize(), yield(), etc. is not enforced by the framework,
> so processors can have different behavior, sometimes on purpose, and
> sometimes inadvertently. The Developer's Guide has guidance on when
> to use such methods [1], and reviewers often check the submissions to
> see if they exhibit such behavior, but it is possible for processors
> to handle these cases differently.
>
> Please feel free to log bugs/improvements for such processors, I think
> consistent behavior in this vein (when prudent) is a good idea.
>
> Regards,
> Matt
>
> [1] https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html#penalization-vs-yielding
>
> On Fri, Oct 14, 2016 at 12:38 PM, Manish Gupta 8 <mg...@sapient.com> wrote:
>> Hello Everyone,
>>
>>
>>
>> In some of the processors I have seen that flow files on failure are not
>> being penalized. For example – Kite processors like ConvertJsonToAvro. Is
>> there some specific reason why some processors have different behavior?
>>
>>
>>
>> I think every processor should penalize every non-success relationship. This
>> way, we can hold those failed files (using a self-referencing loop and some
>> decent penalty duration) in the queue itself and debug and resolve later.
>>
>>
>>
>> Regards,
>>
>> Manish
>>
>>
RE: Penalize Flow File on Failure
Posted by Manish Gupta 8 <mg...@sapient.com>.
Thanks Matt. As a workaround, is there a processor that does not modify the flow file (content or attribute) at all, and I can use it to delay the self-referencing flow files to hit the main processor again immediately?
Regards,
Manish
-----Original Message-----
From: Matt Burgess [mailto:mattyb149@apache.org]
Sent: Friday, October 14, 2016 10:25 PM
To: users@nifi.apache.org
Subject: Re: Penalize Flow File on Failure
Manish,
The use of penalize(), yield(), etc. is not enforced by the framework,
so processors can have different behavior, sometimes on purpose, and
sometimes inadvertently. The Developer's Guide has guidance on when
to use such methods [1], and reviewers often check the submissions to
see if they exhibit such behavior, but it is possible for processors
to handle these cases differently.
Please feel free to log bugs/improvements for such processors, I think
consistent behavior in this vein (when prudent) is a good idea.
Regards,
Matt
[1] https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html#penalization-vs-yielding
On Fri, Oct 14, 2016 at 12:38 PM, Manish Gupta 8 <mg...@sapient.com> wrote:
> Hello Everyone,
>
>
>
> In some of the processors I have seen that flow files on failure are not
> being penalized. For example – Kite processors like ConvertJsonToAvro. Is
> there some specific reason why some processors have different behavior?
>
>
>
> I think every processor should penalize every non-success relationship. This
> way, we can hold those failed files (using a self-referencing loop and some
> decent penalty duration) in the queue itself and debug and resolve later.
>
>
>
> Regards,
>
> Manish
>
>
Re: Penalize Flow File on Failure
Posted by Matt Burgess <ma...@apache.org>.
Manish,
The use of penalize(), yield(), etc. is not enforced by the framework,
so processors can have different behavior, sometimes on purpose, and
sometimes inadvertently. The Developer's Guide has guidance on when
to use such methods [1], and reviewers often check the submissions to
see if they exhibit such behavior, but it is possible for processors
to handle these cases differently.
Please feel free to log bugs/improvements for such processors, I think
consistent behavior in this vein (when prudent) is a good idea.
Regards,
Matt
[1] https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html#penalization-vs-yielding
On Fri, Oct 14, 2016 at 12:38 PM, Manish Gupta 8 <mg...@sapient.com> wrote:
> Hello Everyone,
>
>
>
> In some of the processors I have seen that flow files on failure are not
> being penalized. For example – Kite processors like ConvertJsonToAvro. Is
> there some specific reason why some processors have different behavior?
>
>
>
> I think every processor should penalize every non-success relationship. This
> way, we can hold those failed files (using a self-referencing loop and some
> decent penalty duration) in the queue itself and debug and resolve later.
>
>
>
> Regards,
>
> Manish
>
>