NiFi's Run schedule and its effects

["Woodcock, Michael W., M.B.A." <wo...@mayo.edu>]

What is the use case for implementing "Run Schedule" on a processor?  We have some JMS queue readers on a 5-node cluster that have been set to a Run schedule of 125 ms with 2 concurrent tasks.  Every so often our volume jumps from a 1,000-message-average per 5 minute period to 5,000.  Status history -indicates 1,182 / 1,517 / 1347.24 for Min/Max/Mean during normal operations and 20,553 / 20,713 / 20,633 when it get dumped on. When this surge occurs, we can't read off the JMS queue fast enough and it starts backing up.

Mike Woodcock                                                                        Mayo Clinic Department of Information Technology
Enterprise Core Platform Services- Big Data                 Valley High South, 2-205
                                                                                                        200 First St SW, Rochester, MN USA 55905

Re: NiFi's Run schedule and its effects

[Joe Witt <jo...@gmail.com>]

Mike

With that setting you're telling NiFi to run that processor no more than 8
times per second per thread.  If the processor involved is written to do so
in batches then this can be ok.  But often you just want to let it rip and
set it to 0 as the scheduling which means give it a thread to run as often
as possible. This shouldnt be overly burdensome to JMS and most processors
with such situations are coded to be nice to resources in either case.

Any reason not to just set scheduling period to 0?

We should make a ConsumeJMSRecord processor too.  For most common JMS cases
this would be super fast and more like ConsumeKafkaRecord is.

Thanks

On Tue, Apr 14, 2020 at 10:36 AM Woodcock, Michael W., M.B.A. <
woodcock.mike@mayo.edu> wrote:

> What is the use case for implementing “Run Schedule” on a processor?  We
> have some JMS queue readers on a 5-node cluster that have been set to a Run
> schedule of 125 ms with 2 concurrent tasks.  Every so often our volume
> jumps from a 1,000-message-average per 5 minute period to 5,000.  Status
> history -indicates 1,182 / 1,517 / 1347.24 for Min/Max/Mean during normal
> operations and 20,553 / 20,713 / 20,633 when it get dumped on. When this
> surge occurs, we can’t read off the JMS queue fast enough and it starts
> backing up.
>
>
>
> Mike Woodcock
>                                                               Mayo Clinic
> Department of Information Technology
> Enterprise Core Platform Services- Big Data                 Valley High
> South, 2-205
>
>
> 200 First St SW, Rochester, MN USA 55905
>
>
>