You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Henrique Nascimento <he...@la.logicalis.com> on 2020/09/23 23:27:43 UTC

ListenUDP processor dropping packets

Hi all,

Thanks in advance for any help.

I have a ListenUdp processor receiving Syslog messages, but when i 
compare the results of the processor with a linux TCPDUMP i noticed that 
the processor drops a lot of the data, with no warning/error. (Well, its 
UDP...)

I already tried a lot of configurations, actually i'm trying these numbers:

Concurrent tasks = 6

Receive Buffer Size = 65507 B

Max Size of Message Queue = 10000000

Max Size of Socket Buffer = 512 MB

Max Batch Size = 200000

Running Nifi 1.11.4.

Docs that helped me:

https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.11.4/org.apache.nifi.processors.standard.ListenUDP/

https://bryanbende.com/development/2016/05/09/optimizing-performance-of-apache-nifis-network-listening-processors


Pls, any ideas of how can i better tune this processor?

-- 

*Henrique Nascimento*

Re: ListenUDP processor dropping packets

Posted by Jo...@swisscom.com.
Hi Henrique

Did you lift up the OS UDP Receive Buffer Size? Per default it’s way too small under linux.

sysctl -w net.core.rmem_max=16777216

With netstat -su (“Udp:” Section) you can check whether you have UDP buffer issues…

Cheers Josef

From: Henrique Nascimento <he...@la.logicalis.com>
Reply to: "users@nifi.apache.org" <us...@nifi.apache.org>
Date: Thursday, 24 September 2020 at 01:28
To: "users@nifi.apache.org" <us...@nifi.apache.org>
Subject: ListenUDP processor dropping packets


Hi all,

Thanks in advance for any help.

I have a ListenUdp processor receiving Syslog messages, but when i compare the results of the processor with a linux TCPDUMP i noticed that the processor drops a lot of the data, with no warning/error. (Well, its UDP...)

I already tried a lot of configurations, actually i'm trying these numbers:

Concurrent tasks = 6

Receive Buffer Size = 65507 B

Max Size of Message Queue = 10000000

Max Size of Socket Buffer = 512 MB

Max Batch Size = 200000

Running Nifi 1.11.4.

Docs that helped me:

https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.11.4/org.apache.nifi.processors.standard.ListenUDP/

https://bryanbende.com/development/2016/05/09/optimizing-performance-of-apache-nifis-network-listening-processors



Pls, any ideas of how can i better tune this processor?
--


Henrique Nascimento