Threading in a NiFi Processor

[Ryan Hendrickson <rh...@gmail.com>]

I have always stayed away from doing any threading inside a NiFi processor.

However, I recently came across a use-case where I'm calling a web service
from within a custom Nifi Processor and I don't want to overwhelm the web
service.

I'd like to instrument a sleep in the onTrigger() or process() method of
about 1 second, or a configurable amount.

Is there a pattern established for Nifi Processors to accomplish this
nicely?

Thanks,
Ryan

Re: Threading in a NiFi Processor

[Mike Drob <ma...@cloudera.com>]

Under processor configuration you can set Scheduling > Run Schedule > 1 sec
to control how often it is scheduled to run by the framework. Does that
address the need?

On Thu, Aug 13, 2015 at 1:19 PM, Ryan Hendrickson <
rhendrickson.work@gmail.com> wrote:

> I have always stayed away from doing any threading inside a NiFi processor.
>
> However, I recently came across a use-case where I'm calling a web service
> from within a custom Nifi Processor and I don't want to overwhelm the web
> service.
>
> I'd like to instrument a sleep in the onTrigger() or process() method of
> about 1 second, or a configurable amount.
>
> Is there a pattern established for Nifi Processors to accomplish this
> nicely?
>
> Thanks,
> Ryan
>

RE: Threading in a NiFi Processor

[Mark Payne <ma...@hotmail.com>]

Ryan,

When you call ProcessContext.yield(), all that does is tell the framework not to schedule you for a bit. 
So you can call it from anywhere. But the current thread will continue on. So the thread can return or can finish its job.

Thanks
-Mark

----------------------------------------
> Date: Thu, 13 Aug 2015 14:52:57 -0400
> Subject: Re: Threading in a NiFi Processor
> From: rhendrickson.work@gmail.com
> To: dev@nifi.apache.org
>
> That may work.. Can you call ProcessContext.yield() anywhere with in the
> processor, or only at the end of the onTrigger()?
>
> flowfile --> processor --> output
>
> I thought if you called the ProcessContext.yield() you were intending to
> end the the processor's responsibility portion and output..
>
>
> On Thu, Aug 13, 2015 at 2:38 PM, Mark Payne <ma...@hotmail.com> wrote:
>
>> Ryan,
>>
>> The down side to that approach is that you're tying up that thread from
>> the thread pool when you call sleep.
>>
>> Typically, in a situation like this, where you want to 'pause' processing
>> for a bit, you would call ProcessContext.yield() and then return.
>> This causes the framework not to trigger that processor for some period of
>> time (configurable in the Settings tab, default is 1 sec).
>>
>> Does this work for you?
>>
>> Thanks
>> -Mark
>>
>>
>> ----------------------------------------
>>> Date: Thu, 13 Aug 2015 14:19:36 -0400
>>> Subject: Threading in a NiFi Processor
>>> From: rhendrickson.work@gmail.com
>>> To: dev@nifi.apache.org
>>>
>>> I have always stayed away from doing any threading inside a NiFi
>> processor.
>>>
>>> However, I recently came across a use-case where I'm calling a web
>> service
>>> from within a custom Nifi Processor and I don't want to overwhelm the web
>>> service.
>>>
>>> I'd like to instrument a sleep in the onTrigger() or process() method of
>>> about 1 second, or a configurable amount.
>>>
>>> Is there a pattern established for Nifi Processors to accomplish this
>>> nicely?
>>>
>>> Thanks,
>>> Ryan
>>
>>
 		 	   		  

Re: Threading in a NiFi Processor

[Ryan Hendrickson <rh...@gmail.com>]

That may work.. Can you call ProcessContext.yield() anywhere with in the
processor, or only at the end of the onTrigger()?

flowfile --> processor --> output

I thought if you called the ProcessContext.yield() you were intending to
end the the processor's responsibility portion and output..


On Thu, Aug 13, 2015 at 2:38 PM, Mark Payne <ma...@hotmail.com> wrote:

> Ryan,
>
> The down side to that approach is that you're tying up that thread from
> the thread pool when you call sleep.
>
> Typically, in a situation like this, where you want to 'pause' processing
> for a bit, you would call ProcessContext.yield() and then return.
> This causes the framework not to trigger that processor for some period of
> time (configurable in the Settings tab, default is 1 sec).
>
> Does this work for you?
>
> Thanks
> -Mark
>
>
> ----------------------------------------
> > Date: Thu, 13 Aug 2015 14:19:36 -0400
> > Subject: Threading in a NiFi Processor
> > From: rhendrickson.work@gmail.com
> > To: dev@nifi.apache.org
> >
> > I have always stayed away from doing any threading inside a NiFi
> processor.
> >
> > However, I recently came across a use-case where I'm calling a web
> service
> > from within a custom Nifi Processor and I don't want to overwhelm the web
> > service.
> >
> > I'd like to instrument a sleep in the onTrigger() or process() method of
> > about 1 second, or a configurable amount.
> >
> > Is there a pattern established for Nifi Processors to accomplish this
> > nicely?
> >
> > Thanks,
> > Ryan
>
>

RE: Threading in a NiFi Processor

[Mark Payne <ma...@hotmail.com>]

Ryan,

The down side to that approach is that you're tying up that thread from the thread pool when you call sleep.

Typically, in a situation like this, where you want to 'pause' processing for a bit, you would call ProcessContext.yield() and then return. 
This causes the framework not to trigger that processor for some period of time (configurable in the Settings tab, default is 1 sec).

Does this work for you?

Thanks
-Mark


----------------------------------------
> Date: Thu, 13 Aug 2015 14:19:36 -0400
> Subject: Threading in a NiFi Processor
> From: rhendrickson.work@gmail.com
> To: dev@nifi.apache.org
>
> I have always stayed away from doing any threading inside a NiFi processor.
>
> However, I recently came across a use-case where I'm calling a web service
> from within a custom Nifi Processor and I don't want to overwhelm the web
> service.
>
> I'd like to instrument a sleep in the onTrigger() or process() method of
> about 1 second, or a configurable amount.
>
> Is there a pattern established for Nifi Processors to accomplish this
> nicely?
>
> Thanks,
> Ryan