Increase Concurrent Tasks While Running

["Peter Wicks (pwicks)" <pw...@micron.com>]

Apart from the feature not being there, is there any real technical hurdle from allowing users to increase the Concurrent Tasks count while a processor is already running?
I found myself waiting for a very large file to process and a second file came in on a different relationship. I couldn't stop the processor, and I couldn't move the relationship since the downstream processor was running. It would have been great to be able to increase the Concurrent Tasks count without stopping the processor.

Thoughts?

Re: Increase Concurrent Tasks While Running

[kedarchitale <ke...@yahoo.com>]

Peter, Typically the concurrent processing limit is proportional to the cores
a node has, and exceeding that would cause CPU Thrashing. Having used
concurrent processing frameworks like Quartz, Hadoop, Storm, Spark, Spring
Batch, it is best to configure the framework to use all cores on a node and
in case of additional capacity requirement, add a node on the fly. Let me
know if this works as a solution.
Regards, Kedar



--
View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/Increase-Concurrent-Tasks-While-Running-tp15510p15530.html
Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.

Re: Increase Concurrent Tasks While Running

[Mark Payne <ma...@hotmail.com>]

Peter,

We do not allow any configuration changes to be made to a processor while it is running. This is because some Processors
obtain configuration in order to create "Connection" objects, etc. during initialization. So in that case, changing properties on
a Processor wouldn't take effect until the Processor was stopped and restarted. In other cases, changes would take effect
immediately. In other cases, some property changes would take effect immediately while other property changes don't. 
It ends up leading to a lot of confusion, because it's not clear whether or not a Processor would have to be stopped
and started for changes to take effect. I say this from experience, as it used to work this way :)

So the framework was changed to ensure that properties, etc. were not changed while the processor was running. It could be
made to allow for some settings to change while a processor is running, but currently we don't try to inspect what is changing
when an update is made. Doing so may be possible but I'm not sure that it would be worth the potential confusion that it
may introduce.

-Mark


> On Apr 21, 2017, at 8:47 AM, Peter Wicks (pwicks) <pw...@micron.com> wrote:
> 
> Apart from the feature not being there, is there any real technical hurdle from allowing users to increase the Concurrent Tasks count while a processor is already running?
> I found myself waiting for a very large file to process and a second file came in on a different relationship. I couldn't stop the processor, and I couldn't move the relationship since the downstream processor was running. It would have been great to be able to increase the Concurrent Tasks count without stopping the processor.
> 
> Thoughts?