You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Jean-Sebastien Vachon <js...@brizodata.com> on 2019/09/18 14:30:20 UTC
Too many open files
Hi all,
I've started to see "Too many open files" error messages in Nifi. I checked https://nifi.apache.org/quickstart.html to see the recommended values to fix this
and made the required changes to /etc/security/limits.conf, exited my shell and restarted Nifi. When I check the limits of the Java processes I can still see the limits to be at 4096
I've added the following to /etc/security/limits.conf
* hard nofile 50000
* soft nofile 50000
but the processes show this:
cat /proc/26861/limits
...
Max open files 4096 4096 files
...
Any idea where this 4096 comes from? I tried grepping in the init scripts, nifi configuration and nifi-env.sh but could not find this anywhere
thanks
Re: Too many open files
Posted by Jean-Sebastien Vachon <js...@brizodata.com>.
I managed to find the culprit.. it was the init script that I was using that was doing something weird.
I added MAX_FD=50000 to my nifi-env.sh file and everything seems to be fine now
thanks
________________________________
From: Abdou B <ab...@gmail.com>
Sent: Wednesday, September 18, 2019 10:42 AM
To: users@nifi.apache.org <us...@nifi.apache.org>
Subject: Re: Too many open files
Hello,
It seems to me that for some distribution, you should modify those values in the Cluster management tool.
For example in HDF, with Ambari, you should change the parameter : nifi_user_nofile_limit. for the change to take effect.
Best regards
Abdou
Le mer. 18 sept. 2019 à 16:34, Jean-Sebastien Vachon <js...@brizodata.com>> a écrit :
Does not seem to help... processes are still limited to 4096 fds
________________________________
From: Jean-Sebastien Vachon <js...@brizodata.com>>
Sent: Wednesday, September 18, 2019 10:31 AM
To: users@nifi.apache.org<ma...@nifi.apache.org> <us...@nifi.apache.org>>
Subject: Re: Too many open files
Oups.. just saw the following:
Your distribution may require an edit to /etc/security/limits.d/90-nproc.conf by adding:
* soft nproc 10000
I will try this
________________________________
From: Jean-Sebastien Vachon <js...@brizodata.com>>
Sent: Wednesday, September 18, 2019 10:30 AM
To: users@nifi.apache.org<ma...@nifi.apache.org> <us...@nifi.apache.org>>
Subject: Too many open files
Hi all,
I've started to see "Too many open files" error messages in Nifi. I checked https://nifi.apache.org/quickstart.html to see the recommended values to fix this
and made the required changes to /etc/security/limits.conf, exited my shell and restarted Nifi. When I check the limits of the Java processes I can still see the limits to be at 4096
I've added the following to /etc/security/limits.conf
* hard nofile 50000
* soft nofile 50000
but the processes show this:
cat /proc/26861/limits
...
Max open files 4096 4096 files
...
Any idea where this 4096 comes from? I tried grepping in the init scripts, nifi configuration and nifi-env.sh but could not find this anywhere
thanks
Re: Too many open files
Posted by Abdou B <ab...@gmail.com>.
Hello,
It seems to me that for some distribution, you should modify those values
in the Cluster management tool.
For example in HDF, with Ambari, you should change the parameter
: nifi_user_nofile_limit. for the change to take effect.
Best regards
Abdou
Le mer. 18 sept. 2019 à 16:34, Jean-Sebastien Vachon <js...@brizodata.com>
a écrit :
> Does not seem to help... processes are still limited to 4096 fds
> ------------------------------
> *From:* Jean-Sebastien Vachon <js...@brizodata.com>
> *Sent:* Wednesday, September 18, 2019 10:31 AM
> *To:* users@nifi.apache.org <us...@nifi.apache.org>
> *Subject:* Re: Too many open files
>
> Oups.. just saw the following:
>
> Your distribution may require an edit to
> */etc/security/limits.d/90-nproc.conf* by adding:
> * soft nproc 10000
>
> I will try this
> ------------------------------
> *From:* Jean-Sebastien Vachon <js...@brizodata.com>
> *Sent:* Wednesday, September 18, 2019 10:30 AM
> *To:* users@nifi.apache.org <us...@nifi.apache.org>
> *Subject:* Too many open files
>
> Hi all,
>
> I've started to see "Too many open files" error messages in Nifi. I
> checked https://nifi.apache.org/quickstart.html to see the recommended
> values to fix this
> and made the required changes to /etc/security/limits.conf, exited my
> shell and restarted Nifi. When I check the limits of the Java processes I
> can still see the limits to be at 4096
>
> I've added the following to /etc/security/limits.conf
> * hard nofile 50000
> * soft nofile 50000
>
> but the processes show this:
>
> cat /proc/26861/limits
> ...
> Max open files 4096 4096 files
> ...
>
> Any idea where this 4096 comes from? I tried grepping in the init scripts,
> nifi configuration and nifi-env.sh but could not find this anywhere
>
> thanks
>
Re: Too many open files
Posted by Jean-Sebastien Vachon <js...@brizodata.com>.
Does not seem to help... processes are still limited to 4096 fds
________________________________
From: Jean-Sebastien Vachon <js...@brizodata.com>
Sent: Wednesday, September 18, 2019 10:31 AM
To: users@nifi.apache.org <us...@nifi.apache.org>
Subject: Re: Too many open files
Oups.. just saw the following:
Your distribution may require an edit to /etc/security/limits.d/90-nproc.conf by adding:
* soft nproc 10000
I will try this
________________________________
From: Jean-Sebastien Vachon <js...@brizodata.com>
Sent: Wednesday, September 18, 2019 10:30 AM
To: users@nifi.apache.org <us...@nifi.apache.org>
Subject: Too many open files
Hi all,
I've started to see "Too many open files" error messages in Nifi. I checked https://nifi.apache.org/quickstart.html to see the recommended values to fix this
and made the required changes to /etc/security/limits.conf, exited my shell and restarted Nifi. When I check the limits of the Java processes I can still see the limits to be at 4096
I've added the following to /etc/security/limits.conf
* hard nofile 50000
* soft nofile 50000
but the processes show this:
cat /proc/26861/limits
...
Max open files 4096 4096 files
...
Any idea where this 4096 comes from? I tried grepping in the init scripts, nifi configuration and nifi-env.sh but could not find this anywhere
thanks
Re: Too many open files
Posted by Jean-Sebastien Vachon <js...@brizodata.com>.
Oups.. just saw the following:
Your distribution may require an edit to /etc/security/limits.d/90-nproc.conf by adding:
* soft nproc 10000
I will try this
________________________________
From: Jean-Sebastien Vachon <js...@brizodata.com>
Sent: Wednesday, September 18, 2019 10:30 AM
To: users@nifi.apache.org <us...@nifi.apache.org>
Subject: Too many open files
Hi all,
I've started to see "Too many open files" error messages in Nifi. I checked https://nifi.apache.org/quickstart.html to see the recommended values to fix this
and made the required changes to /etc/security/limits.conf, exited my shell and restarted Nifi. When I check the limits of the Java processes I can still see the limits to be at 4096
I've added the following to /etc/security/limits.conf
* hard nofile 50000
* soft nofile 50000
but the processes show this:
cat /proc/26861/limits
...
Max open files 4096 4096 files
...
Any idea where this 4096 comes from? I tried grepping in the init scripts, nifi configuration and nifi-env.sh but could not find this anywhere
thanks