You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@zookeeper.apache.org by Avinash Lakshman <av...@gmail.com> on 2010/10/05 17:10:32 UTC
Too many connections
I find this happening in my observers node in the logs. The observers are
running in a different data center from where the ZK non-observers are
running. The only way to fix this seems to be restarting. How can I start
addressing this? Here is the stack trace.
Too many connections from /10.30.84.207 - max is 10
WARN - Session 0x0 for server mybox.mydomain.com/10.30.84.207:5001,
unexpected error, closing socket connection and attempting reconnect
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcher.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:233)
at sun.nio.ch.IOUtil.read(IOUtil.java:200)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236)
at
org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:817)
at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1089)
Please advice.
Cheers
Avinash
Re: Too many connections
Posted by Patrick Hunt <ph...@apache.org>.
On Tue, Oct 5, 2010 at 10:23 AM, Avinash Lakshman <
avinash.lakshman@gmail.com> wrote:
> So shouldn't all servers in another DC just have one session? So even if I
> have 50 observers in another DC that should be 50 sessions established
> since
> the IP doesn't change correct? Am I missing something? In some ZK clients I
> see the following exception even though they are in the same DC.
>
>
This really depends on how you implemented your client. Each time you create
a ZooKeeper object a new session is established. If you have 50 clients each
creating a ZooKeeper object then you have 50 sessions.
Patrick
Re: Too many connections
Posted by Avinash Lakshman <av...@gmail.com>.
So shouldn't all servers in another DC just have one session? So even if I
have 50 observers in another DC that should be 50 sessions established since
the IP doesn't change correct? Am I missing something? In some ZK clients I
see the following exception even though they are in the same DC.
WARN - Session 0x0 for server msgzkapp013.abc.com/10.138.43.219:5001,
unexpected error, closing socket connection and attempting reconnect
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcher.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:233)
at sun.nio.ch.IOUtil.read(IOUtil.java:200)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236)
at
org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:817)
at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1089)
WARN - Session 0x0 for server msgzkapp012.abc.com/10.138.42.219:5001,
unexpected error, closing socket connection and attempting reconnect
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcher.write0(Native Method)
at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29)
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:104)
at sun.nio.ch.IOUtil.write(IOUtil.java:75)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334)
at
org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:851)
at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1089)
What might be happening in this case?
Cheers
Avinash
On Tue, Oct 5, 2010 at 9:47 AM, Patrick Hunt <ph...@apache.org> wrote:
> "A" (/10.30.84.207 a zookeeper client) is attempting to establish more
> then 10 sessions to the ZooKeeper server where you got the log. This can be
> caused by a bug in user code (we've seen bugs where incorrectly implemented
> ZK clients attempt to create an infinite number of sessions, which
> essentially DOS the service, so we added the maxClientCnxn default limit of
> 10).
>
> Often users see this problem when they are trying to simulate a real
> environment - they run a simulated set of clients sessions (>10) from a
> single host (ip) hitting the servers. However in your case I'm guessing that
> it has something to do with this
>
> "The observers are running in a different data center from where the ZK
> non-observers are running."
>
> Could you have a NAT or some other networking configuration that makes all
> the observers seem to be coming from the same IP address?
>
> Patrick
>
> On Tue, Oct 5, 2010 at 9:33 AM, Avinash Lakshman <
> avinash.lakshman@gmail.com> wrote:
>
>> Thanks Patrick. But what does this mean? I see the log on server A telling
>> me "Too many connections from A - default is 10". Too many connection from
>> A
>> to whom? I do not see who the other end of the connection is.
>>
>> Cheers
>> Avinash
>>
>> On Tue, Oct 5, 2010 at 9:27 AM, Patrick Hunt <ph...@apache.org> wrote:
>>
>> > See this configuration param in the docs "maxClientCnxns":
>> >
>> >
>> http://hadoop.apache.org/zookeeper/docs/current/zookeeperAdmin.html#sc_advancedConfiguration
>> >
>> > <
>> >
>> http://hadoop.apache.org/zookeeper/docs/current/zookeeperAdmin.html#sc_advancedConfiguration
>> > >
>> > Patrick
>> >
>> > On Tue, Oct 5, 2010 at 8:10 AM, Avinash Lakshman <
>> > avinash.lakshman@gmail.com
>> > > wrote:
>> >
>> > > I find this happening in my observers node in the logs. The observers
>> are
>> > > running in a different data center from where the ZK non-observers are
>> > > running. The only way to fix this seems to be restarting. How can I
>> start
>> > > addressing this? Here is the stack trace.
>> > >
>> > > Too many connections from /10.30.84.207 - max is 10
>> > > WARN - Session 0x0 for server mybox.mydomain.com/10.30.84.207:5001,
>> > > unexpected error, closing socket connection and attempting reconnect
>> > > java.io.IOException: Connection reset by peer
>> > > at sun.nio.ch.FileDispatcher.read0(Native Method)
>> > > at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
>> > > at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:233)
>> > > at sun.nio.ch.IOUtil.read(IOUtil.java:200)
>> > > at
>> sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236)
>> > > at
>> > > org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:817)
>> > > at
>> > > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1089)
>> > >
>> > > Please advice.
>> > >
>> > > Cheers
>> > > Avinash
>> > >
>> >
>>
>
>
Re: Too many connections
Posted by Patrick Hunt <ph...@apache.org>.
"A" (/10.30.84.207 a zookeeper client) is attempting to establish more then
10 sessions to the ZooKeeper server where you got the log. This can be
caused by a bug in user code (we've seen bugs where incorrectly implemented
ZK clients attempt to create an infinite number of sessions, which
essentially DOS the service, so we added the maxClientCnxn default limit of
10).
Often users see this problem when they are trying to simulate a real
environment - they run a simulated set of clients sessions (>10) from a
single host (ip) hitting the servers. However in your case I'm guessing that
it has something to do with this
"The observers are running in a different data center from where the ZK
non-observers are running."
Could you have a NAT or some other networking configuration that makes all
the observers seem to be coming from the same IP address?
Patrick
On Tue, Oct 5, 2010 at 9:33 AM, Avinash Lakshman <avinash.lakshman@gmail.com
> wrote:
> Thanks Patrick. But what does this mean? I see the log on server A telling
> me "Too many connections from A - default is 10". Too many connection from
> A
> to whom? I do not see who the other end of the connection is.
>
> Cheers
> Avinash
>
> On Tue, Oct 5, 2010 at 9:27 AM, Patrick Hunt <ph...@apache.org> wrote:
>
> > See this configuration param in the docs "maxClientCnxns":
> >
> >
> http://hadoop.apache.org/zookeeper/docs/current/zookeeperAdmin.html#sc_advancedConfiguration
> >
> > <
> >
> http://hadoop.apache.org/zookeeper/docs/current/zookeeperAdmin.html#sc_advancedConfiguration
> > >
> > Patrick
> >
> > On Tue, Oct 5, 2010 at 8:10 AM, Avinash Lakshman <
> > avinash.lakshman@gmail.com
> > > wrote:
> >
> > > I find this happening in my observers node in the logs. The observers
> are
> > > running in a different data center from where the ZK non-observers are
> > > running. The only way to fix this seems to be restarting. How can I
> start
> > > addressing this? Here is the stack trace.
> > >
> > > Too many connections from /10.30.84.207 - max is 10
> > > WARN - Session 0x0 for server mybox.mydomain.com/10.30.84.207:5001,
> > > unexpected error, closing socket connection and attempting reconnect
> > > java.io.IOException: Connection reset by peer
> > > at sun.nio.ch.FileDispatcher.read0(Native Method)
> > > at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
> > > at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:233)
> > > at sun.nio.ch.IOUtil.read(IOUtil.java:200)
> > > at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236)
> > > at
> > > org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:817)
> > > at
> > > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1089)
> > >
> > > Please advice.
> > >
> > > Cheers
> > > Avinash
> > >
> >
>
Re: Too many connections
Posted by Avinash Lakshman <av...@gmail.com>.
Thanks Patrick. But what does this mean? I see the log on server A telling
me "Too many connections from A - default is 10". Too many connection from A
to whom? I do not see who the other end of the connection is.
Cheers
Avinash
On Tue, Oct 5, 2010 at 9:27 AM, Patrick Hunt <ph...@apache.org> wrote:
> See this configuration param in the docs "maxClientCnxns":
>
> http://hadoop.apache.org/zookeeper/docs/current/zookeeperAdmin.html#sc_advancedConfiguration
>
> <
> http://hadoop.apache.org/zookeeper/docs/current/zookeeperAdmin.html#sc_advancedConfiguration
> >
> Patrick
>
> On Tue, Oct 5, 2010 at 8:10 AM, Avinash Lakshman <
> avinash.lakshman@gmail.com
> > wrote:
>
> > I find this happening in my observers node in the logs. The observers are
> > running in a different data center from where the ZK non-observers are
> > running. The only way to fix this seems to be restarting. How can I start
> > addressing this? Here is the stack trace.
> >
> > Too many connections from /10.30.84.207 - max is 10
> > WARN - Session 0x0 for server mybox.mydomain.com/10.30.84.207:5001,
> > unexpected error, closing socket connection and attempting reconnect
> > java.io.IOException: Connection reset by peer
> > at sun.nio.ch.FileDispatcher.read0(Native Method)
> > at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
> > at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:233)
> > at sun.nio.ch.IOUtil.read(IOUtil.java:200)
> > at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236)
> > at
> > org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:817)
> > at
> > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1089)
> >
> > Please advice.
> >
> > Cheers
> > Avinash
> >
>
Re: Too many connections
Posted by Patrick Hunt <ph...@apache.org>.
See this configuration param in the docs "maxClientCnxns":
http://hadoop.apache.org/zookeeper/docs/current/zookeeperAdmin.html#sc_advancedConfiguration
<http://hadoop.apache.org/zookeeper/docs/current/zookeeperAdmin.html#sc_advancedConfiguration>
Patrick
On Tue, Oct 5, 2010 at 8:10 AM, Avinash Lakshman <avinash.lakshman@gmail.com
> wrote:
> I find this happening in my observers node in the logs. The observers are
> running in a different data center from where the ZK non-observers are
> running. The only way to fix this seems to be restarting. How can I start
> addressing this? Here is the stack trace.
>
> Too many connections from /10.30.84.207 - max is 10
> WARN - Session 0x0 for server mybox.mydomain.com/10.30.84.207:5001,
> unexpected error, closing socket connection and attempting reconnect
> java.io.IOException: Connection reset by peer
> at sun.nio.ch.FileDispatcher.read0(Native Method)
> at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
> at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:233)
> at sun.nio.ch.IOUtil.read(IOUtil.java:200)
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236)
> at
> org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:817)
> at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1089)
>
> Please advice.
>
> Cheers
> Avinash
>