You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Colin Williams <co...@gmail.com> on 2018/11/08 23:23:50 UTC
Tasks / Time extremely high measure of tasks for LogAttribute
I have a LogAttribute processor connected to the failure of an
S3PutObject processor. Then I noticed when the Put fails the
LogAttribute Tasks/Time shot to 42,045,444 / 00:01:31.664 for In 14
(109KB) and Out 14 (109KB).
I'm not sure about HTML email or attachments then will attach
separately. Just curious what this high task value means.
Re: Tasks / Time extremely high measure of tasks for LogAttribute
Posted by Colin Williams <co...@gmail.com>.
That sounds reasonable. Then it sounds like I should look out for those
high task counts when logging or otherwise.
On Fri, Nov 9, 2018, 9:10 AM Matt Burgess <mattyb149@gmail.com wrote:
> Building off Pierre’s idea, you could set the Penalty Duration to 0 so the
> the flow files go immediately to LogMessage, then send along to
> ExecuteScript to “re-penalize” the file
>
> Regards,
> Matt
>
> On Nov 9, 2018, at 11:39 AM, Colin Williams <
> colin.williams.seattle@gmail.com> wrote:
>
> Hi Pierre,
>
> Thanks for explaining. Could this cause a performance penalty? I assume
> there's no way the next processor can be aware of the penalized flows
> state, or this would not be an issue.
>
> On Fri, Nov 9, 2018, 1:36 AM Pierre Villard <pierre.villard.fr@gmail.com
> wrote:
>
>> Hi Colin,
>>
>> That's because when the flow file is routed to failure by the PutS3, the
>> flow file is penalized and cannot be processed before a given penalty
>> duration by the LogMessage processor. However the LogMessage processor sees
>> there is a flow file to process, and then detect that it cannot be
>> processed because of the penalty duration. There is a JIRA for that [1]. An
>> option could be to set the penalty duration to 0s in the PutS3 processor
>> but I don't think that's a great idea as the processor would keep trying
>> sending the flow file and it could lead to unwanted pressure on S3.
>>
>> [1] https://issues.apache.org/jira/browse/NIFI-3229
>>
>> Pierre
>>
>>
>> Le ven. 9 nov. 2018 à 00:25, Colin Williams <
>> colin.williams.seattle@gmail.com> a écrit :
>>
>>>
>>>
>>> On Thu, Nov 8, 2018 at 3:24 PM Colin Williams <
>>> colin.williams.seattle@gmail.com> wrote:
>>>
>>>> [image: airflow_tasks_time.png]
>>>>
>>>>
>>>> On Thu, Nov 8, 2018 at 3:23 PM Colin Williams <
>>>> colin.williams.seattle@gmail.com> wrote:
>>>>
>>>>> I have a LogAttribute processor connected to the failure of an
>>>>> S3PutObject processor. Then I noticed when the Put fails the
>>>>> LogAttribute Tasks/Time shot to 42,045,444 / 00:01:31.664 for In 14
>>>>> (109KB) and Out 14 (109KB).
>>>>>
>>>>> I'm not sure about HTML email or attachments then will attach
>>>>> separately. Just curious what this high task value means.
>>>>>
>>>>
Re: Tasks / Time extremely high measure of tasks for LogAttribute
Posted by Matt Burgess <ma...@gmail.com>.
Building off Pierre’s idea, you could set the Penalty Duration to 0 so the the flow files go immediately to LogMessage, then send along to ExecuteScript to “re-penalize” the file
Regards,
Matt
> On Nov 9, 2018, at 11:39 AM, Colin Williams <co...@gmail.com> wrote:
>
> Hi Pierre,
>
> Thanks for explaining. Could this cause a performance penalty? I assume there's no way the next processor can be aware of the penalized flows state, or this would not be an issue.
>
>> On Fri, Nov 9, 2018, 1:36 AM Pierre Villard <pierre.villard.fr@gmail.com wrote:
>> Hi Colin,
>>
>> That's because when the flow file is routed to failure by the PutS3, the flow file is penalized and cannot be processed before a given penalty duration by the LogMessage processor. However the LogMessage processor sees there is a flow file to process, and then detect that it cannot be processed because of the penalty duration. There is a JIRA for that [1]. An option could be to set the penalty duration to 0s in the PutS3 processor but I don't think that's a great idea as the processor would keep trying sending the flow file and it could lead to unwanted pressure on S3.
>>
>> [1] https://issues.apache.org/jira/browse/NIFI-3229
>>
>> Pierre
>>
>>
>>> Le ven. 9 nov. 2018 à 00:25, Colin Williams <co...@gmail.com> a écrit :
>>>
>>>
>>>> On Thu, Nov 8, 2018 at 3:24 PM Colin Williams <co...@gmail.com> wrote:
>>>>
>>>>
>>>>
>>>>> On Thu, Nov 8, 2018 at 3:23 PM Colin Williams <co...@gmail.com> wrote:
>>>>> I have a LogAttribute processor connected to the failure of an
>>>>> S3PutObject processor. Then I noticed when the Put fails the
>>>>> LogAttribute Tasks/Time shot to 42,045,444 / 00:01:31.664 for In 14
>>>>> (109KB) and Out 14 (109KB).
>>>>>
>>>>> I'm not sure about HTML email or attachments then will attach
>>>>> separately. Just curious what this high task value means.
Re: Tasks / Time extremely high measure of tasks for LogAttribute
Posted by Colin Williams <co...@gmail.com>.
Hi Pierre,
Thanks for explaining. Could this cause a performance penalty? I assume
there's no way the next processor can be aware of the penalized flows
state, or this would not be an issue.
On Fri, Nov 9, 2018, 1:36 AM Pierre Villard <pierre.villard.fr@gmail.com
wrote:
> Hi Colin,
>
> That's because when the flow file is routed to failure by the PutS3, the
> flow file is penalized and cannot be processed before a given penalty
> duration by the LogMessage processor. However the LogMessage processor sees
> there is a flow file to process, and then detect that it cannot be
> processed because of the penalty duration. There is a JIRA for that [1]. An
> option could be to set the penalty duration to 0s in the PutS3 processor
> but I don't think that's a great idea as the processor would keep trying
> sending the flow file and it could lead to unwanted pressure on S3.
>
> [1] https://issues.apache.org/jira/browse/NIFI-3229
>
> Pierre
>
>
> Le ven. 9 nov. 2018 à 00:25, Colin Williams <
> colin.williams.seattle@gmail.com> a écrit :
>
>>
>>
>> On Thu, Nov 8, 2018 at 3:24 PM Colin Williams <
>> colin.williams.seattle@gmail.com> wrote:
>>
>>> [image: airflow_tasks_time.png]
>>>
>>>
>>> On Thu, Nov 8, 2018 at 3:23 PM Colin Williams <
>>> colin.williams.seattle@gmail.com> wrote:
>>>
>>>> I have a LogAttribute processor connected to the failure of an
>>>> S3PutObject processor. Then I noticed when the Put fails the
>>>> LogAttribute Tasks/Time shot to 42,045,444 / 00:01:31.664 for In 14
>>>> (109KB) and Out 14 (109KB).
>>>>
>>>> I'm not sure about HTML email or attachments then will attach
>>>> separately. Just curious what this high task value means.
>>>>
>>>
Re: Tasks / Time extremely high measure of tasks for LogAttribute
Posted by Pierre Villard <pi...@gmail.com>.
Hi Colin,
That's because when the flow file is routed to failure by the PutS3, the
flow file is penalized and cannot be processed before a given penalty
duration by the LogMessage processor. However the LogMessage processor sees
there is a flow file to process, and then detect that it cannot be
processed because of the penalty duration. There is a JIRA for that [1]. An
option could be to set the penalty duration to 0s in the PutS3 processor
but I don't think that's a great idea as the processor would keep trying
sending the flow file and it could lead to unwanted pressure on S3.
[1] https://issues.apache.org/jira/browse/NIFI-3229
Pierre
Le ven. 9 nov. 2018 à 00:25, Colin Williams <
colin.williams.seattle@gmail.com> a écrit :
>
>
> On Thu, Nov 8, 2018 at 3:24 PM Colin Williams <
> colin.williams.seattle@gmail.com> wrote:
>
>> [image: airflow_tasks_time.png]
>>
>>
>> On Thu, Nov 8, 2018 at 3:23 PM Colin Williams <
>> colin.williams.seattle@gmail.com> wrote:
>>
>>> I have a LogAttribute processor connected to the failure of an
>>> S3PutObject processor. Then I noticed when the Put fails the
>>> LogAttribute Tasks/Time shot to 42,045,444 / 00:01:31.664 for In 14
>>> (109KB) and Out 14 (109KB).
>>>
>>> I'm not sure about HTML email or attachments then will attach
>>> separately. Just curious what this high task value means.
>>>
>>
Re: Tasks / Time extremely high measure of tasks for LogAttribute
Posted by Colin Williams <co...@gmail.com>.
On Thu, Nov 8, 2018 at 3:24 PM Colin Williams <
colin.williams.seattle@gmail.com> wrote:
> [image: airflow_tasks_time.png]
>
>
> On Thu, Nov 8, 2018 at 3:23 PM Colin Williams <
> colin.williams.seattle@gmail.com> wrote:
>
>> I have a LogAttribute processor connected to the failure of an
>> S3PutObject processor. Then I noticed when the Put fails the
>> LogAttribute Tasks/Time shot to 42,045,444 / 00:01:31.664 for In 14
>> (109KB) and Out 14 (109KB).
>>
>> I'm not sure about HTML email or attachments then will attach
>> separately. Just curious what this high task value means.
>>
>
Re: Tasks / Time extremely high measure of tasks for LogAttribute
Posted by Colin Williams <co...@gmail.com>.
[image: airflow_tasks_time.png]
On Thu, Nov 8, 2018 at 3:23 PM Colin Williams <
colin.williams.seattle@gmail.com> wrote:
> I have a LogAttribute processor connected to the failure of an
> S3PutObject processor. Then I noticed when the Put fails the
> LogAttribute Tasks/Time shot to 42,045,444 / 00:01:31.664 for In 14
> (109KB) and Out 14 (109KB).
>
> I'm not sure about HTML email or attachments then will attach
> separately. Just curious what this high task value means.
>