You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@zookeeper.apache.org by Narayanan A R <na...@gmail.com> on 2012/07/10 22:46:16 UTC
ZK not starting up due to socket timeouts
I have 3 servers in the cluster, all bare metal machines and installed ZK
3.4.3. Following is the config in all the servers.
tickTime=2000
dataDir=/opt/zookeeper-3.4.3/data
clientPort=2181
initLimit=60
syncLimit=60
server.1=10.7.78.77:2888:3888
server.2=10.7.66.54:2888:3888
server.3=10.7.56.61:2888:3888
When I startup the instances, I see socket timed out in all the instances.
I have attached logs of all the three machine.
Regards,
ARN
Re: ZK not starting up due to socket timeouts
Posted by David Nickerson <da...@gmail.com>.
For cross-reference, the leap second bug's effect on ZooKeeper has been
posted about before:
http://zookeeper-user.578899.n2.nabble.com/leap-second-excitement-td7577634.html
On Wed, Jul 11, 2012 at 6:31 PM, Narayanan A R <
narayanan.arunachalam@gmail.com> wrote:
> Fixed the problem. It had to do with the leap second bug. Ran the following
> command on all the three servers and is now working fine.
>
> date;/etc/init.d/ntp stop;/etc/init.d/ntpd stop; date `date
> +"%m%d%H%M%C%y.%S"`;/etc/init.d/ntp start;/etc/init.d/ntpd start
>
> You can find some overview of this problem here:
>
> http://www.wired.com/wiredenterprise/2012/07/leap-second-bug-wreaks-havoc-with-java-linux/
>
>
> On Wed, Jul 11, 2012 at 2:35 PM, Narayanan A R <
> narayanan.arunachalam@gmail.com> wrote:
>
> > Thanks David. Here they are:
> >
> > http://pastebin.com/STTnLf9s
> > http://pastebin.com/PiTgUWpA
> > http://pastebin.com/4V3AjT34
> >
> >
> > On Wed, Jul 11, 2012 at 4:32 AM, David Nickerson <
> > davidnickerson4mailinglists@gmail.com> wrote:
> >
> >> Narayanan, I don't think the attachments made it through. Can you link
> to
> >> the logs in Pastebin?
> >>
> >> On Tue, Jul 10, 2012 at 5:24 PM, Narayanan A R <
> >> narayanan.arunachalam@gmail.com> wrote:
> >>
> >> > Yeah I tried that. Right now I have that set to 2 mins.
> >> >
> >> > On Tue, Jul 10, 2012 at 1:47 PM, Jordan Zimmerman <
> >> > jordan@jordanzimmerman.com> wrote:
> >> >
> >> > > Another thing you might need to do is to increase initLimit and
> >> > syncLimit.
> >> > > It might be timing out when its syncing.
> >> > >
> >> > > -JZ
> >> > >
> >> > > On Jul 10, 2012, at 1:46 PM, Narayanan A R wrote:
> >> > >
> >> > > > I have 3 servers in the cluster, all bare metal machines and
> >> installed
> >> > > ZK 3.4.3. Following is the config in all the servers.
> >> > > >
> >> > > > tickTime=2000
> >> > > > dataDir=/opt/zookeeper-3.4.3/data
> >> > > > clientPort=2181
> >> > > > initLimit=60
> >> > > > syncLimit=60
> >> > > > server.1=10.7.78.77:2888:3888
> >> > > > server.2=10.7.66.54:2888:3888
> >> > > > server.3=10.7.56.61:2888:3888
> >> > > >
> >> > > > When I startup the instances, I see socket timed out in all the
> >> > > instances. I have attached logs of all the three machine.
> >> > > >
> >> > > > Regards,
> >> > > > ARN
> >> > >
> >> > >
> >> >
> >>
> >
> >
>
Re: ZK not starting up due to socket timeouts
Posted by Narayanan A R <na...@gmail.com>.
Fixed the problem. It had to do with the leap second bug. Ran the following
command on all the three servers and is now working fine.
date;/etc/init.d/ntp stop;/etc/init.d/ntpd stop; date `date
+"%m%d%H%M%C%y.%S"`;/etc/init.d/ntp start;/etc/init.d/ntpd start
You can find some overview of this problem here:
http://www.wired.com/wiredenterprise/2012/07/leap-second-bug-wreaks-havoc-with-java-linux/
On Wed, Jul 11, 2012 at 2:35 PM, Narayanan A R <
narayanan.arunachalam@gmail.com> wrote:
> Thanks David. Here they are:
>
> http://pastebin.com/STTnLf9s
> http://pastebin.com/PiTgUWpA
> http://pastebin.com/4V3AjT34
>
>
> On Wed, Jul 11, 2012 at 4:32 AM, David Nickerson <
> davidnickerson4mailinglists@gmail.com> wrote:
>
>> Narayanan, I don't think the attachments made it through. Can you link to
>> the logs in Pastebin?
>>
>> On Tue, Jul 10, 2012 at 5:24 PM, Narayanan A R <
>> narayanan.arunachalam@gmail.com> wrote:
>>
>> > Yeah I tried that. Right now I have that set to 2 mins.
>> >
>> > On Tue, Jul 10, 2012 at 1:47 PM, Jordan Zimmerman <
>> > jordan@jordanzimmerman.com> wrote:
>> >
>> > > Another thing you might need to do is to increase initLimit and
>> > syncLimit.
>> > > It might be timing out when its syncing.
>> > >
>> > > -JZ
>> > >
>> > > On Jul 10, 2012, at 1:46 PM, Narayanan A R wrote:
>> > >
>> > > > I have 3 servers in the cluster, all bare metal machines and
>> installed
>> > > ZK 3.4.3. Following is the config in all the servers.
>> > > >
>> > > > tickTime=2000
>> > > > dataDir=/opt/zookeeper-3.4.3/data
>> > > > clientPort=2181
>> > > > initLimit=60
>> > > > syncLimit=60
>> > > > server.1=10.7.78.77:2888:3888
>> > > > server.2=10.7.66.54:2888:3888
>> > > > server.3=10.7.56.61:2888:3888
>> > > >
>> > > > When I startup the instances, I see socket timed out in all the
>> > > instances. I have attached logs of all the three machine.
>> > > >
>> > > > Regards,
>> > > > ARN
>> > >
>> > >
>> >
>>
>
>
Re: ZK not starting up due to socket timeouts
Posted by Narayanan A R <na...@gmail.com>.
Fixed the problem. It had to do with the leap second bug. Ran the following
command on all the three servers and is now working fine.
date;/etc/init.d/ntp stop;/etc/init.d/ntpd stop; date `date
+"%m%d%H%M%C%y.%S"`;/etc/init.d/ntp start;/etc/init.d/ntpd start
You can find some overview of this problem here:
http://www.wired.com/wiredenterprise/2012/07/leap-second-bug-wreaks-havoc-with-java-linux/
On Wed, Jul 11, 2012 at 2:35 PM, Narayanan A R <
narayanan.arunachalam@gmail.com> wrote:
> Thanks David. Here they are:
>
> http://pastebin.com/STTnLf9s
> http://pastebin.com/PiTgUWpA
> http://pastebin.com/4V3AjT34
>
>
> On Wed, Jul 11, 2012 at 4:32 AM, David Nickerson <
> davidnickerson4mailinglists@gmail.com> wrote:
>
>> Narayanan, I don't think the attachments made it through. Can you link to
>> the logs in Pastebin?
>>
>> On Tue, Jul 10, 2012 at 5:24 PM, Narayanan A R <
>> narayanan.arunachalam@gmail.com> wrote:
>>
>> > Yeah I tried that. Right now I have that set to 2 mins.
>> >
>> > On Tue, Jul 10, 2012 at 1:47 PM, Jordan Zimmerman <
>> > jordan@jordanzimmerman.com> wrote:
>> >
>> > > Another thing you might need to do is to increase initLimit and
>> > syncLimit.
>> > > It might be timing out when its syncing.
>> > >
>> > > -JZ
>> > >
>> > > On Jul 10, 2012, at 1:46 PM, Narayanan A R wrote:
>> > >
>> > > > I have 3 servers in the cluster, all bare metal machines and
>> installed
>> > > ZK 3.4.3. Following is the config in all the servers.
>> > > >
>> > > > tickTime=2000
>> > > > dataDir=/opt/zookeeper-3.4.3/data
>> > > > clientPort=2181
>> > > > initLimit=60
>> > > > syncLimit=60
>> > > > server.1=10.7.78.77:2888:3888
>> > > > server.2=10.7.66.54:2888:3888
>> > > > server.3=10.7.56.61:2888:3888
>> > > >
>> > > > When I startup the instances, I see socket timed out in all the
>> > > instances. I have attached logs of all the three machine.
>> > > >
>> > > > Regards,
>> > > > ARN
>> > >
>> > >
>> >
>>
>
>
Re: ZK not starting up due to socket timeouts
Posted by Narayanan A R <na...@gmail.com>.
Thanks David. Here they are:
http://pastebin.com/STTnLf9s
http://pastebin.com/PiTgUWpA
http://pastebin.com/4V3AjT34
On Wed, Jul 11, 2012 at 4:32 AM, David Nickerson <
davidnickerson4mailinglists@gmail.com> wrote:
> Narayanan, I don't think the attachments made it through. Can you link to
> the logs in Pastebin?
>
> On Tue, Jul 10, 2012 at 5:24 PM, Narayanan A R <
> narayanan.arunachalam@gmail.com> wrote:
>
> > Yeah I tried that. Right now I have that set to 2 mins.
> >
> > On Tue, Jul 10, 2012 at 1:47 PM, Jordan Zimmerman <
> > jordan@jordanzimmerman.com> wrote:
> >
> > > Another thing you might need to do is to increase initLimit and
> > syncLimit.
> > > It might be timing out when its syncing.
> > >
> > > -JZ
> > >
> > > On Jul 10, 2012, at 1:46 PM, Narayanan A R wrote:
> > >
> > > > I have 3 servers in the cluster, all bare metal machines and
> installed
> > > ZK 3.4.3. Following is the config in all the servers.
> > > >
> > > > tickTime=2000
> > > > dataDir=/opt/zookeeper-3.4.3/data
> > > > clientPort=2181
> > > > initLimit=60
> > > > syncLimit=60
> > > > server.1=10.7.78.77:2888:3888
> > > > server.2=10.7.66.54:2888:3888
> > > > server.3=10.7.56.61:2888:3888
> > > >
> > > > When I startup the instances, I see socket timed out in all the
> > > instances. I have attached logs of all the three machine.
> > > >
> > > > Regards,
> > > > ARN
> > >
> > >
> >
>
Re: ZK not starting up due to socket timeouts
Posted by Narayanan A R <na...@gmail.com>.
Thanks David. Here they are:
http://pastebin.com/STTnLf9s
http://pastebin.com/PiTgUWpA
http://pastebin.com/4V3AjT34
On Wed, Jul 11, 2012 at 4:32 AM, David Nickerson <
davidnickerson4mailinglists@gmail.com> wrote:
> Narayanan, I don't think the attachments made it through. Can you link to
> the logs in Pastebin?
>
> On Tue, Jul 10, 2012 at 5:24 PM, Narayanan A R <
> narayanan.arunachalam@gmail.com> wrote:
>
> > Yeah I tried that. Right now I have that set to 2 mins.
> >
> > On Tue, Jul 10, 2012 at 1:47 PM, Jordan Zimmerman <
> > jordan@jordanzimmerman.com> wrote:
> >
> > > Another thing you might need to do is to increase initLimit and
> > syncLimit.
> > > It might be timing out when its syncing.
> > >
> > > -JZ
> > >
> > > On Jul 10, 2012, at 1:46 PM, Narayanan A R wrote:
> > >
> > > > I have 3 servers in the cluster, all bare metal machines and
> installed
> > > ZK 3.4.3. Following is the config in all the servers.
> > > >
> > > > tickTime=2000
> > > > dataDir=/opt/zookeeper-3.4.3/data
> > > > clientPort=2181
> > > > initLimit=60
> > > > syncLimit=60
> > > > server.1=10.7.78.77:2888:3888
> > > > server.2=10.7.66.54:2888:3888
> > > > server.3=10.7.56.61:2888:3888
> > > >
> > > > When I startup the instances, I see socket timed out in all the
> > > instances. I have attached logs of all the three machine.
> > > >
> > > > Regards,
> > > > ARN
> > >
> > >
> >
>
Re: ZK not starting up due to socket timeouts
Posted by David Nickerson <da...@gmail.com>.
Narayanan, I don't think the attachments made it through. Can you link to
the logs in Pastebin?
On Tue, Jul 10, 2012 at 5:24 PM, Narayanan A R <
narayanan.arunachalam@gmail.com> wrote:
> Yeah I tried that. Right now I have that set to 2 mins.
>
> On Tue, Jul 10, 2012 at 1:47 PM, Jordan Zimmerman <
> jordan@jordanzimmerman.com> wrote:
>
> > Another thing you might need to do is to increase initLimit and
> syncLimit.
> > It might be timing out when its syncing.
> >
> > -JZ
> >
> > On Jul 10, 2012, at 1:46 PM, Narayanan A R wrote:
> >
> > > I have 3 servers in the cluster, all bare metal machines and installed
> > ZK 3.4.3. Following is the config in all the servers.
> > >
> > > tickTime=2000
> > > dataDir=/opt/zookeeper-3.4.3/data
> > > clientPort=2181
> > > initLimit=60
> > > syncLimit=60
> > > server.1=10.7.78.77:2888:3888
> > > server.2=10.7.66.54:2888:3888
> > > server.3=10.7.56.61:2888:3888
> > >
> > > When I startup the instances, I see socket timed out in all the
> > instances. I have attached logs of all the three machine.
> > >
> > > Regards,
> > > ARN
> >
> >
>
Re: ZK not starting up due to socket timeouts
Posted by Narayanan A R <na...@gmail.com>.
Yeah I tried that. Right now I have that set to 2 mins.
On Tue, Jul 10, 2012 at 1:47 PM, Jordan Zimmerman <
jordan@jordanzimmerman.com> wrote:
> Another thing you might need to do is to increase initLimit and syncLimit.
> It might be timing out when its syncing.
>
> -JZ
>
> On Jul 10, 2012, at 1:46 PM, Narayanan A R wrote:
>
> > I have 3 servers in the cluster, all bare metal machines and installed
> ZK 3.4.3. Following is the config in all the servers.
> >
> > tickTime=2000
> > dataDir=/opt/zookeeper-3.4.3/data
> > clientPort=2181
> > initLimit=60
> > syncLimit=60
> > server.1=10.7.78.77:2888:3888
> > server.2=10.7.66.54:2888:3888
> > server.3=10.7.56.61:2888:3888
> >
> > When I startup the instances, I see socket timed out in all the
> instances. I have attached logs of all the three machine.
> >
> > Regards,
> > ARN
>
>
Re: ZK not starting up due to socket timeouts
Posted by Narayanan A R <na...@gmail.com>.
Yeah I tried that. Right now I have that set to 2 mins.
On Tue, Jul 10, 2012 at 1:47 PM, Jordan Zimmerman <
jordan@jordanzimmerman.com> wrote:
> Another thing you might need to do is to increase initLimit and syncLimit.
> It might be timing out when its syncing.
>
> -JZ
>
> On Jul 10, 2012, at 1:46 PM, Narayanan A R wrote:
>
> > I have 3 servers in the cluster, all bare metal machines and installed
> ZK 3.4.3. Following is the config in all the servers.
> >
> > tickTime=2000
> > dataDir=/opt/zookeeper-3.4.3/data
> > clientPort=2181
> > initLimit=60
> > syncLimit=60
> > server.1=10.7.78.77:2888:3888
> > server.2=10.7.66.54:2888:3888
> > server.3=10.7.56.61:2888:3888
> >
> > When I startup the instances, I see socket timed out in all the
> instances. I have attached logs of all the three machine.
> >
> > Regards,
> > ARN
>
>
Re: ZK not starting up due to socket timeouts
Posted by Jordan Zimmerman <jo...@jordanzimmerman.com>.
Another thing you might need to do is to increase initLimit and syncLimit. It might be timing out when its syncing.
-JZ
On Jul 10, 2012, at 1:46 PM, Narayanan A R wrote:
> I have 3 servers in the cluster, all bare metal machines and installed ZK 3.4.3. Following is the config in all the servers.
>
> tickTime=2000
> dataDir=/opt/zookeeper-3.4.3/data
> clientPort=2181
> initLimit=60
> syncLimit=60
> server.1=10.7.78.77:2888:3888
> server.2=10.7.66.54:2888:3888
> server.3=10.7.56.61:2888:3888
>
> When I startup the instances, I see socket timed out in all the instances. I have attached logs of all the three machine.
>
> Regards,
> ARN
Re: ZK not starting up due to socket timeouts
Posted by Jordan Zimmerman <jo...@jordanzimmerman.com>.
Another thing you might need to do is to increase initLimit and syncLimit. It might be timing out when its syncing.
-JZ
On Jul 10, 2012, at 1:46 PM, Narayanan A R wrote:
> I have 3 servers in the cluster, all bare metal machines and installed ZK 3.4.3. Following is the config in all the servers.
>
> tickTime=2000
> dataDir=/opt/zookeeper-3.4.3/data
> clientPort=2181
> initLimit=60
> syncLimit=60
> server.1=10.7.78.77:2888:3888
> server.2=10.7.66.54:2888:3888
> server.3=10.7.56.61:2888:3888
>
> When I startup the instances, I see socket timed out in all the instances. I have attached logs of all the three machine.
>
> Regards,
> ARN