Question about performance tuning for Nifi

[BINLEI XUE <ko...@gmail.com>]

Hi,

Recently we received some errors "Internal queue at maximum capacity, could
not queue event" from our ListenSyslog processor

Currently settings


   - using UDP
   - Max Size of Socket Buffer 2MB
   - Max Batch Size 10000
   - Max Size of Message Queue 10000
   - Parse Message false

By checking the code of EventQueue.java, we found that once the offer
method timeout, we would lost the data.


   1. We could increase the message queue size but it would consume more
   memory and we not are sure about whether it would solve the problem or not.
   2. Can we increase the timeout, like 2 seconds?
   3. Can we replace offer method with put?


Best,

Re: Question about performance tuning for Nifi

[Bryan Bende <bb...@gmail.com>]

Have you increased the concurrent tasks on ListenSyslog? If not then going
from 1 to 2 would be the first thing to try.

Since UDP is connection-less, changing the offer to put or changing the
timeout would just move the problem to a different part of the stack. NiFis
code would be blocking waiting for space in the queue, meanwhile the client
would have no idea and keep sending data which would likely get lost
somewhere at the network level, but the client would have no way of knowing.

For TCP it would possibly be nice to block there since that would block all
the way back to the client connection.

On Thu, Nov 22, 2018 at 7:05 AM BINLEI XUE <ko...@gmail.com> wrote:

> Hi,
>
> Recently we received some errors "Internal queue at maximum capacity, could
> not queue event" from our ListenSyslog processor
>
> Currently settings
>
>
>    - using UDP
>    - Max Size of Socket Buffer 2MB
>    - Max Batch Size 10000
>    - Max Size of Message Queue 10000
>    - Parse Message false
>
> By checking the code of EventQueue.java, we found that once the offer
> method timeout, we would lost the data.
>
>
>    1. We could increase the message queue size but it would consume more
>    memory and we not are sure about whether it would solve the problem or
> not.
>    2. Can we increase the timeout, like 2 seconds?
>    3. Can we replace offer method with put?
>
>
> Best,
>
-- 
Sent from Gmail Mobile