You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Kessiler Rodrigues <ke...@callinize.com> on 2016/07/29 19:00:41 UTC
Too Many Open Files
Hi guys,
I have been experiencing some issues on kafka, where its throwing too many open files.
I have around of 6k topics and 5 partitions each.
My cluster was made with 6 brokers. All of them are running Ubuntu 16 and the file limits settings are:
`cat /proc/sys/fs/file-max`
2000000
`ulimit -n`
1000000
Anyone has experienced it before?
Re: Too Many Open Files
Posted by "Thakrar, Jayesh" <jt...@conversantmedia.com>.
What are the producers/consumers for the Kafka cluster?
Remember that its not just files but also sockets that add to the count.
I had seen issues when we had a network switch problem and had Storm consumers.
The switch would cause issues in connectivity between Kafka brokers, zookeepers and clients, causing a flood of connections from everyone to each other.
On 8/1/16, 7:14 AM, "Scott Thibault" <sc...@multiscalehn.com> wrote:
Did you verify that the process has the correct limit applied?
cat /proc/<your PID>/limits
--Scott Thibault
On Sun, Jul 31, 2016 at 4:14 PM, Kessiler Rodrigues <ke...@callinize.com>
wrote:
> I’m still experiencing this issue…
>
> Here are the kafka logs.
>
> [2016-07-31 20:10:35,658] ERROR Error while accepting connection
> (kafka.network.Acceptor)
> java.io.IOException: Too many open files
> at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> at
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
> at
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
> at kafka.network.Acceptor.accept(SocketServer.scala:323)
> at kafka.network.Acceptor.run(SocketServer.scala:268)
> at java.lang.Thread.run(Thread.java:745)
> [2016-07-31 20:10:35,658] ERROR Error while accepting connection
> (kafka.network.Acceptor)
> java.io.IOException: Too many open files
> at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> at
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
> at
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
> at kafka.network.Acceptor.accept(SocketServer.scala:323)
> at kafka.network.Acceptor.run(SocketServer.scala:268)
> at java.lang.Thread.run(Thread.java:745)
> [2016-07-31 20:10:35,658] ERROR Error while accepting connection
> (kafka.network.Acceptor)
> java.io.IOException: Too many open files
> at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> at
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
> at
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
> at kafka.network.Acceptor.accept(SocketServer.scala:323)
> at kafka.network.Acceptor.run(SocketServer.scala:268)
> at java.lang.Thread.run(Thread.java:745)
>
> My ulimit is 1 million, how is that possible?
>
> Can someone help with this?
>
>
> > On Jul 30, 2016, at 5:05 AM, Kessiler Rodrigues <ke...@callinize.com>
> wrote:
> >
> > I have changed it a bit.
> >
> > I have 10 brokers and 20k topics with 1 partition each.
> >
> > I looked at the kaka’s logs dir and I only have 3318 files.
> >
> > I’m doing some tests to see how many topics/partitions I can have, but
> it is throwing too many files once it hits 15k topics..
> >
> > Any thoughts?
> >
> >
> >
> >> On Jul 29, 2016, at 10:33 PM, Gwen Shapira <gw...@confluent.io> wrote:
> >>
> >> woah, it looks like you have 15,000 replicas per broker?
> >>
> >> You can go into the directory you configured for kafka's log.dir and
> >> see how many files you have there. Depending on your segment size and
> >> retention policy, you could have hundreds of files per partition
> >> there...
> >>
> >> Make sure you have at least that many file handles and then also add
> >> handles for the client connections.
> >>
> >> 1 million file handles sound like a lot, but you are running lots of
> >> partitions per broker...
> >>
> >> We normally don't see more than maybe 4000 per broker and most
> >> clusters have a lot fewer, so consider adding brokers and spreading
> >> partitions around a bit.
> >>
> >> Gwen
> >>
> >> On Fri, Jul 29, 2016 at 12:00 PM, Kessiler Rodrigues
> >> <ke...@callinize.com> wrote:
> >>> Hi guys,
> >>>
> >>> I have been experiencing some issues on kafka, where its throwing too
> many open files.
> >>>
> >>> I have around of 6k topics and 5 partitions each.
> >>>
> >>> My cluster was made with 6 brokers. All of them are running Ubuntu 16
> and the file limits settings are:
> >>>
> >>> `cat /proc/sys/fs/file-max`
> >>> 2000000
> >>>
> >>> `ulimit -n`
> >>> 1000000
> >>>
> >>> Anyone has experienced it before?
> >
>
>
--
*This e-mail is not encrypted. Due to the unsecured nature of unencrypted
e-mail, there may be some level of risk that the information in this e-mail
could be read by a third party. Accordingly, the recipient(s) named above
are hereby advised to not communicate protected health information using
this e-mail address. If you desire to send protected health information
electronically, please contact MultiScale Health Networks at (206)538-6090*
This email and any files included with it may contain privileged,
proprietary and/or confidential information that is for the sole use
of the intended recipient(s). Any disclosure, copying, distribution,
posting, or use of the information contained in or attached to this
email is prohibited unless permitted by the sender. If you have
received this email in error, please immediately notify the sender
via return email, telephone, or fax and destroy this original transmission
and its included files without reading or saving it in any manner.
Thank you.
Re: Too Many Open Files
Posted by Scott Thibault <sc...@multiscalehn.com>.
Did you verify that the process has the correct limit applied?
cat /proc/<your PID>/limits
--Scott Thibault
On Sun, Jul 31, 2016 at 4:14 PM, Kessiler Rodrigues <ke...@callinize.com>
wrote:
> I’m still experiencing this issue…
>
> Here are the kafka logs.
>
> [2016-07-31 20:10:35,658] ERROR Error while accepting connection
> (kafka.network.Acceptor)
> java.io.IOException: Too many open files
> at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> at
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
> at
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
> at kafka.network.Acceptor.accept(SocketServer.scala:323)
> at kafka.network.Acceptor.run(SocketServer.scala:268)
> at java.lang.Thread.run(Thread.java:745)
> [2016-07-31 20:10:35,658] ERROR Error while accepting connection
> (kafka.network.Acceptor)
> java.io.IOException: Too many open files
> at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> at
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
> at
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
> at kafka.network.Acceptor.accept(SocketServer.scala:323)
> at kafka.network.Acceptor.run(SocketServer.scala:268)
> at java.lang.Thread.run(Thread.java:745)
> [2016-07-31 20:10:35,658] ERROR Error while accepting connection
> (kafka.network.Acceptor)
> java.io.IOException: Too many open files
> at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> at
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
> at
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
> at kafka.network.Acceptor.accept(SocketServer.scala:323)
> at kafka.network.Acceptor.run(SocketServer.scala:268)
> at java.lang.Thread.run(Thread.java:745)
>
> My ulimit is 1 million, how is that possible?
>
> Can someone help with this?
>
>
> > On Jul 30, 2016, at 5:05 AM, Kessiler Rodrigues <ke...@callinize.com>
> wrote:
> >
> > I have changed it a bit.
> >
> > I have 10 brokers and 20k topics with 1 partition each.
> >
> > I looked at the kaka’s logs dir and I only have 3318 files.
> >
> > I’m doing some tests to see how many topics/partitions I can have, but
> it is throwing too many files once it hits 15k topics..
> >
> > Any thoughts?
> >
> >
> >
> >> On Jul 29, 2016, at 10:33 PM, Gwen Shapira <gw...@confluent.io> wrote:
> >>
> >> woah, it looks like you have 15,000 replicas per broker?
> >>
> >> You can go into the directory you configured for kafka's log.dir and
> >> see how many files you have there. Depending on your segment size and
> >> retention policy, you could have hundreds of files per partition
> >> there...
> >>
> >> Make sure you have at least that many file handles and then also add
> >> handles for the client connections.
> >>
> >> 1 million file handles sound like a lot, but you are running lots of
> >> partitions per broker...
> >>
> >> We normally don't see more than maybe 4000 per broker and most
> >> clusters have a lot fewer, so consider adding brokers and spreading
> >> partitions around a bit.
> >>
> >> Gwen
> >>
> >> On Fri, Jul 29, 2016 at 12:00 PM, Kessiler Rodrigues
> >> <ke...@callinize.com> wrote:
> >>> Hi guys,
> >>>
> >>> I have been experiencing some issues on kafka, where its throwing too
> many open files.
> >>>
> >>> I have around of 6k topics and 5 partitions each.
> >>>
> >>> My cluster was made with 6 brokers. All of them are running Ubuntu 16
> and the file limits settings are:
> >>>
> >>> `cat /proc/sys/fs/file-max`
> >>> 2000000
> >>>
> >>> `ulimit -n`
> >>> 1000000
> >>>
> >>> Anyone has experienced it before?
> >
>
>
--
*This e-mail is not encrypted. Due to the unsecured nature of unencrypted
e-mail, there may be some level of risk that the information in this e-mail
could be read by a third party. Accordingly, the recipient(s) named above
are hereby advised to not communicate protected health information using
this e-mail address. If you desire to send protected health information
electronically, please contact MultiScale Health Networks at (206)538-6090*
Re: Too Many Open Files
Posted by Kessiler Rodrigues <ke...@callinize.com>.
Hey guys
I got a solution for this. The kafka process wasn’t getting the limits config because I was running it under supervisor.
I changed it and right now I’m using systemd to put kafka up and running!
On systemd services you can setup your FD limit using a property called “LimitNOFile”.
Thanks for all your help!
> On Aug 1, 2016, at 5:04 AM, Anirudh P <pa...@gmail.com> wrote:
>
> I agree with Steve. We had a similar problem where we set the ulimit to a
> certain value but it was getting overridden.
> It only worked when we set the ulimit after logging in as root. You might
> want to give that a try if you have not done so already
>
> - Anirudh
>
> On Mon, Aug 1, 2016 at 1:19 PM, Steve Miller <st...@idrathernotsay.com>
> wrote:
>
>> Can you run lsof -p (pid) for whatever the pid is for your Kafka process?
>>
>> For the fd limits you've set, I don't think subtlety is required: if
>> there's a millionish lines in the output, the fd limit you set is where you
>> think it is, and if it's a lot lower than that, the limit isn't being
>> applied properly somehow (maybe you are running this under, say,
>> supervisord, and maybe its config is lowering the limit, or the limits for
>> root are as you say but the limits for the kafka user aren't being set
>> properly, that sort of thing).
>>
>> If you do have 1M lines in the output, at least this might give you a
>> place to start figuring out what's open and why.
>>
>> -Steve
>>
>>> On Jul 31, 2016, at 4:14 PM, Kessiler Rodrigues <ke...@callinize.com>
>> wrote:
>>>
>>> I’m still experiencing this issue…
>>>
>>> Here are the kafka logs.
>>>
>>> [2016-07-31 20:10:35,658] ERROR Error while accepting connection
>> (kafka.network.Acceptor)
>>> java.io.IOException: Too many open files
>>> at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
>>> at
>> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
>>> at
>> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
>>> at kafka.network.Acceptor.accept(SocketServer.scala:323)
>>> at kafka.network.Acceptor.run(SocketServer.scala:268)
>>> at java.lang.Thread.run(Thread.java:745)
>>> [2016-07-31 20:10:35,658] ERROR Error while accepting connection
>> (kafka.network.Acceptor)
>>> java.io.IOException: Too many open files
>>> at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
>>> at
>> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
>>> at
>> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
>>> at kafka.network.Acceptor.accept(SocketServer.scala:323)
>>> at kafka.network.Acceptor.run(SocketServer.scala:268)
>>> at java.lang.Thread.run(Thread.java:745)
>>> [2016-07-31 20:10:35,658] ERROR Error while accepting connection
>> (kafka.network.Acceptor)
>>> java.io.IOException: Too many open files
>>> at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
>>> at
>> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
>>> at
>> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
>>> at kafka.network.Acceptor.accept(SocketServer.scala:323)
>>> at kafka.network.Acceptor.run(SocketServer.scala:268)
>>> at java.lang.Thread.run(Thread.java:745)
>>>
>>> My ulimit is 1 million, how is that possible?
>>>
>>> Can someone help with this?
>>>
>>>
>>>> On Jul 30, 2016, at 5:05 AM, Kessiler Rodrigues <ke...@callinize.com>
>> wrote:
>>>>
>>>> I have changed it a bit.
>>>>
>>>> I have 10 brokers and 20k topics with 1 partition each.
>>>>
>>>> I looked at the kaka’s logs dir and I only have 3318 files.
>>>>
>>>> I’m doing some tests to see how many topics/partitions I can have, but
>> it is throwing too many files once it hits 15k topics..
>>>>
>>>> Any thoughts?
>>>>
>>>>
>>>>
>>>>> On Jul 29, 2016, at 10:33 PM, Gwen Shapira <gw...@confluent.io> wrote:
>>>>>
>>>>> woah, it looks like you have 15,000 replicas per broker?
>>>>>
>>>>> You can go into the directory you configured for kafka's log.dir and
>>>>> see how many files you have there. Depending on your segment size and
>>>>> retention policy, you could have hundreds of files per partition
>>>>> there...
>>>>>
>>>>> Make sure you have at least that many file handles and then also add
>>>>> handles for the client connections.
>>>>>
>>>>> 1 million file handles sound like a lot, but you are running lots of
>>>>> partitions per broker...
>>>>>
>>>>> We normally don't see more than maybe 4000 per broker and most
>>>>> clusters have a lot fewer, so consider adding brokers and spreading
>>>>> partitions around a bit.
>>>>>
>>>>> Gwen
>>>>>
>>>>> On Fri, Jul 29, 2016 at 12:00 PM, Kessiler Rodrigues
>>>>> <ke...@callinize.com> wrote:
>>>>>> Hi guys,
>>>>>>
>>>>>> I have been experiencing some issues on kafka, where its throwing too
>> many open files.
>>>>>>
>>>>>> I have around of 6k topics and 5 partitions each.
>>>>>>
>>>>>> My cluster was made with 6 brokers. All of them are running Ubuntu 16
>> and the file limits settings are:
>>>>>>
>>>>>> `cat /proc/sys/fs/file-max`
>>>>>> 2000000
>>>>>>
>>>>>> `ulimit -n`
>>>>>> 1000000
>>>>>>
>>>>>> Anyone has experienced it before?
>>>
>>
>>
Re: Too Many Open Files
Posted by Anirudh P <pa...@gmail.com>.
I agree with Steve. We had a similar problem where we set the ulimit to a
certain value but it was getting overridden.
It only worked when we set the ulimit after logging in as root. You might
want to give that a try if you have not done so already
- Anirudh
On Mon, Aug 1, 2016 at 1:19 PM, Steve Miller <st...@idrathernotsay.com>
wrote:
> Can you run lsof -p (pid) for whatever the pid is for your Kafka process?
>
> For the fd limits you've set, I don't think subtlety is required: if
> there's a millionish lines in the output, the fd limit you set is where you
> think it is, and if it's a lot lower than that, the limit isn't being
> applied properly somehow (maybe you are running this under, say,
> supervisord, and maybe its config is lowering the limit, or the limits for
> root are as you say but the limits for the kafka user aren't being set
> properly, that sort of thing).
>
> If you do have 1M lines in the output, at least this might give you a
> place to start figuring out what's open and why.
>
> -Steve
>
> > On Jul 31, 2016, at 4:14 PM, Kessiler Rodrigues <ke...@callinize.com>
> wrote:
> >
> > I’m still experiencing this issue…
> >
> > Here are the kafka logs.
> >
> > [2016-07-31 20:10:35,658] ERROR Error while accepting connection
> (kafka.network.Acceptor)
> > java.io.IOException: Too many open files
> > at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> > at
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
> > at
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
> > at kafka.network.Acceptor.accept(SocketServer.scala:323)
> > at kafka.network.Acceptor.run(SocketServer.scala:268)
> > at java.lang.Thread.run(Thread.java:745)
> > [2016-07-31 20:10:35,658] ERROR Error while accepting connection
> (kafka.network.Acceptor)
> > java.io.IOException: Too many open files
> > at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> > at
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
> > at
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
> > at kafka.network.Acceptor.accept(SocketServer.scala:323)
> > at kafka.network.Acceptor.run(SocketServer.scala:268)
> > at java.lang.Thread.run(Thread.java:745)
> > [2016-07-31 20:10:35,658] ERROR Error while accepting connection
> (kafka.network.Acceptor)
> > java.io.IOException: Too many open files
> > at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> > at
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
> > at
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
> > at kafka.network.Acceptor.accept(SocketServer.scala:323)
> > at kafka.network.Acceptor.run(SocketServer.scala:268)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > My ulimit is 1 million, how is that possible?
> >
> > Can someone help with this?
> >
> >
> >> On Jul 30, 2016, at 5:05 AM, Kessiler Rodrigues <ke...@callinize.com>
> wrote:
> >>
> >> I have changed it a bit.
> >>
> >> I have 10 brokers and 20k topics with 1 partition each.
> >>
> >> I looked at the kaka’s logs dir and I only have 3318 files.
> >>
> >> I’m doing some tests to see how many topics/partitions I can have, but
> it is throwing too many files once it hits 15k topics..
> >>
> >> Any thoughts?
> >>
> >>
> >>
> >>> On Jul 29, 2016, at 10:33 PM, Gwen Shapira <gw...@confluent.io> wrote:
> >>>
> >>> woah, it looks like you have 15,000 replicas per broker?
> >>>
> >>> You can go into the directory you configured for kafka's log.dir and
> >>> see how many files you have there. Depending on your segment size and
> >>> retention policy, you could have hundreds of files per partition
> >>> there...
> >>>
> >>> Make sure you have at least that many file handles and then also add
> >>> handles for the client connections.
> >>>
> >>> 1 million file handles sound like a lot, but you are running lots of
> >>> partitions per broker...
> >>>
> >>> We normally don't see more than maybe 4000 per broker and most
> >>> clusters have a lot fewer, so consider adding brokers and spreading
> >>> partitions around a bit.
> >>>
> >>> Gwen
> >>>
> >>> On Fri, Jul 29, 2016 at 12:00 PM, Kessiler Rodrigues
> >>> <ke...@callinize.com> wrote:
> >>>> Hi guys,
> >>>>
> >>>> I have been experiencing some issues on kafka, where its throwing too
> many open files.
> >>>>
> >>>> I have around of 6k topics and 5 partitions each.
> >>>>
> >>>> My cluster was made with 6 brokers. All of them are running Ubuntu 16
> and the file limits settings are:
> >>>>
> >>>> `cat /proc/sys/fs/file-max`
> >>>> 2000000
> >>>>
> >>>> `ulimit -n`
> >>>> 1000000
> >>>>
> >>>> Anyone has experienced it before?
> >
>
>
Re: Too Many Open Files
Posted by Steve Miller <st...@idrathernotsay.com>.
Can you run lsof -p (pid) for whatever the pid is for your Kafka process?
For the fd limits you've set, I don't think subtlety is required: if there's a millionish lines in the output, the fd limit you set is where you think it is, and if it's a lot lower than that, the limit isn't being applied properly somehow (maybe you are running this under, say, supervisord, and maybe its config is lowering the limit, or the limits for root are as you say but the limits for the kafka user aren't being set properly, that sort of thing).
If you do have 1M lines in the output, at least this might give you a place to start figuring out what's open and why.
-Steve
> On Jul 31, 2016, at 4:14 PM, Kessiler Rodrigues <ke...@callinize.com> wrote:
>
> I’m still experiencing this issue…
>
> Here are the kafka logs.
>
> [2016-07-31 20:10:35,658] ERROR Error while accepting connection (kafka.network.Acceptor)
> java.io.IOException: Too many open files
> at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
> at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
> at kafka.network.Acceptor.accept(SocketServer.scala:323)
> at kafka.network.Acceptor.run(SocketServer.scala:268)
> at java.lang.Thread.run(Thread.java:745)
> [2016-07-31 20:10:35,658] ERROR Error while accepting connection (kafka.network.Acceptor)
> java.io.IOException: Too many open files
> at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
> at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
> at kafka.network.Acceptor.accept(SocketServer.scala:323)
> at kafka.network.Acceptor.run(SocketServer.scala:268)
> at java.lang.Thread.run(Thread.java:745)
> [2016-07-31 20:10:35,658] ERROR Error while accepting connection (kafka.network.Acceptor)
> java.io.IOException: Too many open files
> at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
> at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
> at kafka.network.Acceptor.accept(SocketServer.scala:323)
> at kafka.network.Acceptor.run(SocketServer.scala:268)
> at java.lang.Thread.run(Thread.java:745)
>
> My ulimit is 1 million, how is that possible?
>
> Can someone help with this?
>
>
>> On Jul 30, 2016, at 5:05 AM, Kessiler Rodrigues <ke...@callinize.com> wrote:
>>
>> I have changed it a bit.
>>
>> I have 10 brokers and 20k topics with 1 partition each.
>>
>> I looked at the kaka’s logs dir and I only have 3318 files.
>>
>> I’m doing some tests to see how many topics/partitions I can have, but it is throwing too many files once it hits 15k topics..
>>
>> Any thoughts?
>>
>>
>>
>>> On Jul 29, 2016, at 10:33 PM, Gwen Shapira <gw...@confluent.io> wrote:
>>>
>>> woah, it looks like you have 15,000 replicas per broker?
>>>
>>> You can go into the directory you configured for kafka's log.dir and
>>> see how many files you have there. Depending on your segment size and
>>> retention policy, you could have hundreds of files per partition
>>> there...
>>>
>>> Make sure you have at least that many file handles and then also add
>>> handles for the client connections.
>>>
>>> 1 million file handles sound like a lot, but you are running lots of
>>> partitions per broker...
>>>
>>> We normally don't see more than maybe 4000 per broker and most
>>> clusters have a lot fewer, so consider adding brokers and spreading
>>> partitions around a bit.
>>>
>>> Gwen
>>>
>>> On Fri, Jul 29, 2016 at 12:00 PM, Kessiler Rodrigues
>>> <ke...@callinize.com> wrote:
>>>> Hi guys,
>>>>
>>>> I have been experiencing some issues on kafka, where its throwing too many open files.
>>>>
>>>> I have around of 6k topics and 5 partitions each.
>>>>
>>>> My cluster was made with 6 brokers. All of them are running Ubuntu 16 and the file limits settings are:
>>>>
>>>> `cat /proc/sys/fs/file-max`
>>>> 2000000
>>>>
>>>> `ulimit -n`
>>>> 1000000
>>>>
>>>> Anyone has experienced it before?
>
RE: Too Many Open Files
Posted by Krzysztof Nawara <kr...@cern.ch>.
Maybe you are exhausting your sockets, not file handles for some reason?
________________________________________
From: Kessiler Rodrigues [kessiler@callinize.com]
Sent: 31 July 2016 22:14
To: users@kafka.apache.org
Subject: Re: Too Many Open Files
I’m still experiencing this issue…
Here are the kafka logs.
[2016-07-31 20:10:35,658] ERROR Error while accepting connection (kafka.network.Acceptor)
java.io.IOException: Too many open files
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
at kafka.network.Acceptor.accept(SocketServer.scala:323)
at kafka.network.Acceptor.run(SocketServer.scala:268)
at java.lang.Thread.run(Thread.java:745)
[2016-07-31 20:10:35,658] ERROR Error while accepting connection (kafka.network.Acceptor)
java.io.IOException: Too many open files
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
at kafka.network.Acceptor.accept(SocketServer.scala:323)
at kafka.network.Acceptor.run(SocketServer.scala:268)
at java.lang.Thread.run(Thread.java:745)
[2016-07-31 20:10:35,658] ERROR Error while accepting connection (kafka.network.Acceptor)
java.io.IOException: Too many open files
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
at kafka.network.Acceptor.accept(SocketServer.scala:323)
at kafka.network.Acceptor.run(SocketServer.scala:268)
at java.lang.Thread.run(Thread.java:745)
My ulimit is 1 million, how is that possible?
Can someone help with this?
> On Jul 30, 2016, at 5:05 AM, Kessiler Rodrigues <ke...@callinize.com> wrote:
>
> I have changed it a bit.
>
> I have 10 brokers and 20k topics with 1 partition each.
>
> I looked at the kaka’s logs dir and I only have 3318 files.
>
> I’m doing some tests to see how many topics/partitions I can have, but it is throwing too many files once it hits 15k topics..
>
> Any thoughts?
>
>
>
>> On Jul 29, 2016, at 10:33 PM, Gwen Shapira <gw...@confluent.io> wrote:
>>
>> woah, it looks like you have 15,000 replicas per broker?
>>
>> You can go into the directory you configured for kafka's log.dir and
>> see how many files you have there. Depending on your segment size and
>> retention policy, you could have hundreds of files per partition
>> there...
>>
>> Make sure you have at least that many file handles and then also add
>> handles for the client connections.
>>
>> 1 million file handles sound like a lot, but you are running lots of
>> partitions per broker...
>>
>> We normally don't see more than maybe 4000 per broker and most
>> clusters have a lot fewer, so consider adding brokers and spreading
>> partitions around a bit.
>>
>> Gwen
>>
>> On Fri, Jul 29, 2016 at 12:00 PM, Kessiler Rodrigues
>> <ke...@callinize.com> wrote:
>>> Hi guys,
>>>
>>> I have been experiencing some issues on kafka, where its throwing too many open files.
>>>
>>> I have around of 6k topics and 5 partitions each.
>>>
>>> My cluster was made with 6 brokers. All of them are running Ubuntu 16 and the file limits settings are:
>>>
>>> `cat /proc/sys/fs/file-max`
>>> 2000000
>>>
>>> `ulimit -n`
>>> 1000000
>>>
>>> Anyone has experienced it before?
>
Re: Too Many Open Files
Posted by Kessiler Rodrigues <ke...@callinize.com>.
I’m still experiencing this issue…
Here are the kafka logs.
[2016-07-31 20:10:35,658] ERROR Error while accepting connection (kafka.network.Acceptor)
java.io.IOException: Too many open files
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
at kafka.network.Acceptor.accept(SocketServer.scala:323)
at kafka.network.Acceptor.run(SocketServer.scala:268)
at java.lang.Thread.run(Thread.java:745)
[2016-07-31 20:10:35,658] ERROR Error while accepting connection (kafka.network.Acceptor)
java.io.IOException: Too many open files
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
at kafka.network.Acceptor.accept(SocketServer.scala:323)
at kafka.network.Acceptor.run(SocketServer.scala:268)
at java.lang.Thread.run(Thread.java:745)
[2016-07-31 20:10:35,658] ERROR Error while accepting connection (kafka.network.Acceptor)
java.io.IOException: Too many open files
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
at kafka.network.Acceptor.accept(SocketServer.scala:323)
at kafka.network.Acceptor.run(SocketServer.scala:268)
at java.lang.Thread.run(Thread.java:745)
My ulimit is 1 million, how is that possible?
Can someone help with this?
> On Jul 30, 2016, at 5:05 AM, Kessiler Rodrigues <ke...@callinize.com> wrote:
>
> I have changed it a bit.
>
> I have 10 brokers and 20k topics with 1 partition each.
>
> I looked at the kaka’s logs dir and I only have 3318 files.
>
> I’m doing some tests to see how many topics/partitions I can have, but it is throwing too many files once it hits 15k topics..
>
> Any thoughts?
>
>
>
>> On Jul 29, 2016, at 10:33 PM, Gwen Shapira <gw...@confluent.io> wrote:
>>
>> woah, it looks like you have 15,000 replicas per broker?
>>
>> You can go into the directory you configured for kafka's log.dir and
>> see how many files you have there. Depending on your segment size and
>> retention policy, you could have hundreds of files per partition
>> there...
>>
>> Make sure you have at least that many file handles and then also add
>> handles for the client connections.
>>
>> 1 million file handles sound like a lot, but you are running lots of
>> partitions per broker...
>>
>> We normally don't see more than maybe 4000 per broker and most
>> clusters have a lot fewer, so consider adding brokers and spreading
>> partitions around a bit.
>>
>> Gwen
>>
>> On Fri, Jul 29, 2016 at 12:00 PM, Kessiler Rodrigues
>> <ke...@callinize.com> wrote:
>>> Hi guys,
>>>
>>> I have been experiencing some issues on kafka, where its throwing too many open files.
>>>
>>> I have around of 6k topics and 5 partitions each.
>>>
>>> My cluster was made with 6 brokers. All of them are running Ubuntu 16 and the file limits settings are:
>>>
>>> `cat /proc/sys/fs/file-max`
>>> 2000000
>>>
>>> `ulimit -n`
>>> 1000000
>>>
>>> Anyone has experienced it before?
>
Re: Too Many Open Files
Posted by Kessiler Rodrigues <ke...@callinize.com>.
I have changed it a bit.
I have 10 brokers and 20k topics with 1 partition each.
I looked at the kaka’s logs dir and I only have 3318 files.
I’m doing some tests to see how many topics/partitions I can have, but it is throwing too many files once it hits 15k topics..
Any thoughts?
> On Jul 29, 2016, at 10:33 PM, Gwen Shapira <gw...@confluent.io> wrote:
>
> woah, it looks like you have 15,000 replicas per broker?
>
> You can go into the directory you configured for kafka's log.dir and
> see how many files you have there. Depending on your segment size and
> retention policy, you could have hundreds of files per partition
> there...
>
> Make sure you have at least that many file handles and then also add
> handles for the client connections.
>
> 1 million file handles sound like a lot, but you are running lots of
> partitions per broker...
>
> We normally don't see more than maybe 4000 per broker and most
> clusters have a lot fewer, so consider adding brokers and spreading
> partitions around a bit.
>
> Gwen
>
> On Fri, Jul 29, 2016 at 12:00 PM, Kessiler Rodrigues
> <ke...@callinize.com> wrote:
>> Hi guys,
>>
>> I have been experiencing some issues on kafka, where its throwing too many open files.
>>
>> I have around of 6k topics and 5 partitions each.
>>
>> My cluster was made with 6 brokers. All of them are running Ubuntu 16 and the file limits settings are:
>>
>> `cat /proc/sys/fs/file-max`
>> 2000000
>>
>> `ulimit -n`
>> 1000000
>>
>> Anyone has experienced it before?
Re: Too Many Open Files
Posted by Chris Richardson <ch...@chrisrichardson.net>.
Gwen,
Is there any particular reason why "inactive" (no consumers or producers
for a topic) files need to be open?
Chris
--
Learn microservices - http://learnmicroservices.io
Microservices application platform http://eventuate.io
On Fri, Jul 29, 2016 at 6:33 PM, Gwen Shapira <gw...@confluent.io> wrote:
> woah, it looks like you have 15,000 replicas per broker?
>
> You can go into the directory you configured for kafka's log.dir and
> see how many files you have there. Depending on your segment size and
> retention policy, you could have hundreds of files per partition
> there...
>
> Make sure you have at least that many file handles and then also add
> handles for the client connections.
>
> 1 million file handles sound like a lot, but you are running lots of
> partitions per broker...
>
> We normally don't see more than maybe 4000 per broker and most
> clusters have a lot fewer, so consider adding brokers and spreading
> partitions around a bit.
>
> Gwen
>
> On Fri, Jul 29, 2016 at 12:00 PM, Kessiler Rodrigues
> <ke...@callinize.com> wrote:
> > Hi guys,
> >
> > I have been experiencing some issues on kafka, where its throwing too
> many open files.
> >
> > I have around of 6k topics and 5 partitions each.
> >
> > My cluster was made with 6 brokers. All of them are running Ubuntu 16
> and the file limits settings are:
> >
> > `cat /proc/sys/fs/file-max`
> > 2000000
> >
> > `ulimit -n`
> > 1000000
> >
> > Anyone has experienced it before?
>
Re: Too Many Open Files
Posted by Gwen Shapira <gw...@confluent.io>.
woah, it looks like you have 15,000 replicas per broker?
You can go into the directory you configured for kafka's log.dir and
see how many files you have there. Depending on your segment size and
retention policy, you could have hundreds of files per partition
there...
Make sure you have at least that many file handles and then also add
handles for the client connections.
1 million file handles sound like a lot, but you are running lots of
partitions per broker...
We normally don't see more than maybe 4000 per broker and most
clusters have a lot fewer, so consider adding brokers and spreading
partitions around a bit.
Gwen
On Fri, Jul 29, 2016 at 12:00 PM, Kessiler Rodrigues
<ke...@callinize.com> wrote:
> Hi guys,
>
> I have been experiencing some issues on kafka, where its throwing too many open files.
>
> I have around of 6k topics and 5 partitions each.
>
> My cluster was made with 6 brokers. All of them are running Ubuntu 16 and the file limits settings are:
>
> `cat /proc/sys/fs/file-max`
> 2000000
>
> `ulimit -n`
> 1000000
>
> Anyone has experienced it before?