You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@zookeeper.apache.org by Deepak Jagtap <de...@maxta.com> on 2012/05/17 01:39:00 UTC
Exceeded deadline by 16ms
Hi All,
I am using Zookeeper 3.4.3 sever and my setup is as follows:
I have a Quorum of 3 Zookeepers servers (installed on 3 virtual machines)
on my laptop.
Then I start 12 Zookeeper client processes on the same machine which start
updates on Zookeeper database.
After running test for a while, sometimes some of the client process got
disconnected from Zookeeper sever with following error:
*Exceeded deadline by 16ms* and then says session timeout.
Client connects with timeout value of 10000ms.
Zookeeper server configuration is as follows:
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/tmp/zookeeper
clientPort=2181
server.1=192.168.1.234:2888:3888
server.2=192.168.1.235:2888:3888
server.3=192.168.1.236:2888:3888
Does anybody facing similar problem? Please let me know if there anythin
wrong wit configuration.
Thanks & Regards,
Deepak
Re: Exceeded deadline by 16ms
Posted by 李赫元 <li...@gmail.com>.
As said by @phunt, the exceed deadline is client's warning.
It's different from SESSION_TIMEOUT, which require re-init handle and
reset all watcher.
The server may be a little overload.
2012/5/18 Deepak Jagtap <de...@maxta.com>:
> Hi,
>
> Thanks for quick reply!
> I am seeing sequence of "exceeded deadline" message then connection timed
> out error messages.
> My code has some usleep() calls, when a sequence of zoo_get and zoo_set
> returns ZBADVERSION error due to race condition,
> it invokes usleep() for random time(between 5ms to 100ms) and then retry
> the same operation. Does this may be cause any issues?
>
> Please find following log dump:
>
> 2012-05-17 11:10:12,910:6709(0x7f12d52d2700):ZOO_WARN@zookeeper_interest@1557:
> Exceeded deadline by 14ms
> 2012-05-17 11:10:13,200:6709(0x7f12d52d2700):ZOO_WARN@zookeeper_interest@1557:
> Exceeded deadline by 11ms
> .
> .
> 2012-05-17 11:10:13,270:6709(0x7f12d52d2700):ZOO_WARN@zookeeper_interest@1557:
> Exceeded deadline by 11ms
> 2012-05-17 11:10:13,745:6709(0x7f12d52d2700):ZOO_WARN@zookeeper_interest@1557:
> Exceeded deadline by 11ms
> 2012-05-17
> 11:10:14,164:6709(0x7f12d52d2700):ZOO_ERROR@handle_socket_error_msg@1644:
> Socket [192.168.1.236:2181] zk retcode=-7, errno=110(Connection timed out):
> connection timed out (exceeded timeout by 0ms)
> operation timeout
> operation timeout
> 2012-05-17 11:10:14,164:6709(0x7f12d52d2700):ZOO_INFO@check_events@1701:
> initiated connection to server [192.168.1.234:2181]
>
>
> Thanks & Regards,
> Deepak
>
> On Wed, May 16, 2012 at 6:30 PM, 李赫元 <li...@gmail.com> wrote:
>
>> Hello,
>>
>> Did you get the "ZOO_SESSION_EVENT" with "ZOO_EXPIRED_SESSION_STATE"
>> on watch callback event or just simply debug information print on
>> screen showing "Exceeded deadline by 16ms"?
>>
>> Thanks.
>>
>>
>> 2012/5/17 Deepak Jagtap <de...@maxta.com>:
>> > Hi All,
>> >
>> > I am using Zookeeper 3.4.3 sever and my setup is as follows:
>> >
>> > I have a Quorum of 3 Zookeepers servers (installed on 3 virtual machines)
>> > on my laptop.
>> > Then I start 12 Zookeeper client processes on the same machine which
>> start
>> > updates on Zookeeper database.
>> > After running test for a while, sometimes some of the client process got
>> > disconnected from Zookeeper sever with following error:
>> > *Exceeded deadline by 16ms* and then says session timeout.
>> > Client connects with timeout value of 10000ms.
>> >
>> > Zookeeper server configuration is as follows:
>> > tickTime=2000
>> > initLimit=10
>> > syncLimit=5
>> > dataDir=/tmp/zookeeper
>> > clientPort=2181
>> > server.1=192.168.1.234:2888:3888
>> > server.2=192.168.1.235:2888:3888
>> > server.3=192.168.1.236:2888:3888
>> >
>> > Does anybody facing similar problem? Please let me know if there anythin
>> > wrong wit configuration.
>> >
>> > Thanks & Regards,
>> > Deepak
>>
Re: Exceeded deadline by 16ms
Posted by Patrick Hunt <ph...@apache.org>.
NP, good luck.
Patrick
On Thu, May 17, 2012 at 4:01 PM, Deepak Jagtap <de...@maxta.com> wrote:
> Thanks Patrick, that helped!
>
> Best Regards,
> Deepak
>
> On Thu, May 17, 2012 at 3:23 PM, Patrick Hunt <ph...@apache.org> wrote:
>
>> The deadline exceeded messages are generated by select - the client
>> will select(timeout). If the requested timeout is exceeded by more
>> than 10ms then that message is printed. Often this will happen in a
>> virtualized client environment (not the case here from what you are
>> saying), or an overloaded OS, etc...
>>
>> You are running 12 client processes, where the client code expects a
>> 10second session timeout (so heartbeat responses w/in 3.3 seconds)
>> along three vms running three servers all on the same laptop, it's
>> likely that you are overloaded.
>>
>> Patrick
>>
>> On Thu, May 17, 2012 at 11:36 AM, Deepak Jagtap <de...@maxta.com>
>> wrote:
>> > Hi,
>> >
>> > Thanks for quick reply!
>> > I am seeing sequence of "exceeded deadline" message then connection
>> timed
>> > out error messages.
>> > My code has some usleep() calls, when a sequence of zoo_get and zoo_set
>> > returns ZBADVERSION error due to race condition,
>> > it invokes usleep() for random time(between 5ms to 100ms) and then retry
>> > the same operation. Does this may be cause any issues?
>> >
>> > Please find following log dump:
>> >
>> > 2012-05-17 11:10:12,910:6709(0x7f12d52d2700):ZOO_WARN@zookeeper_interest
>> @1557:
>> > Exceeded deadline by 14ms
>> > 2012-05-17 11:10:13,200:6709(0x7f12d52d2700):ZOO_WARN@zookeeper_interest
>> @1557:
>> > Exceeded deadline by 11ms
>> > .
>> > .
>> > 2012-05-17 11:10:13,270:6709(0x7f12d52d2700):ZOO_WARN@zookeeper_interest
>> @1557:
>> > Exceeded deadline by 11ms
>> > 2012-05-17 11:10:13,745:6709(0x7f12d52d2700):ZOO_WARN@zookeeper_interest
>> @1557:
>> > Exceeded deadline by 11ms
>> > 2012-05-17
>> > 11:10:14,164:6709(0x7f12d52d2700):ZOO_ERROR@handle_socket_error_msg
>> @1644:
>> > Socket [192.168.1.236:2181] zk retcode=-7, errno=110(Connection timed
>> out):
>> > connection timed out (exceeded timeout by 0ms)
>> > operation timeout
>> > operation timeout
>> > 2012-05-17 11:10:14,164:6709(0x7f12d52d2700):ZOO_INFO@check_events@1701:
>> > initiated connection to server [192.168.1.234:2181]
>> >
>> >
>> > Thanks & Regards,
>> > Deepak
>> >
>> > On Wed, May 16, 2012 at 6:30 PM, 李赫元 <li...@gmail.com> wrote:
>> >
>> >> Hello,
>> >>
>> >> Did you get the "ZOO_SESSION_EVENT" with "ZOO_EXPIRED_SESSION_STATE"
>> >> on watch callback event or just simply debug information print on
>> >> screen showing "Exceeded deadline by 16ms"?
>> >>
>> >> Thanks.
>> >>
>> >>
>> >> 2012/5/17 Deepak Jagtap <de...@maxta.com>:
>> >> > Hi All,
>> >> >
>> >> > I am using Zookeeper 3.4.3 sever and my setup is as follows:
>> >> >
>> >> > I have a Quorum of 3 Zookeepers servers (installed on 3 virtual
>> machines)
>> >> > on my laptop.
>> >> > Then I start 12 Zookeeper client processes on the same machine which
>> >> start
>> >> > updates on Zookeeper database.
>> >> > After running test for a while, sometimes some of the client process
>> got
>> >> > disconnected from Zookeeper sever with following error:
>> >> > *Exceeded deadline by 16ms* and then says session timeout.
>> >> > Client connects with timeout value of 10000ms.
>> >> >
>> >> > Zookeeper server configuration is as follows:
>> >> > tickTime=2000
>> >> > initLimit=10
>> >> > syncLimit=5
>> >> > dataDir=/tmp/zookeeper
>> >> > clientPort=2181
>> >> > server.1=192.168.1.234:2888:3888
>> >> > server.2=192.168.1.235:2888:3888
>> >> > server.3=192.168.1.236:2888:3888
>> >> >
>> >> > Does anybody facing similar problem? Please let me know if there
>> anythin
>> >> > wrong wit configuration.
>> >> >
>> >> > Thanks & Regards,
>> >> > Deepak
>> >>
>>
Re: Exceeded deadline by 16ms
Posted by Deepak Jagtap <de...@maxta.com>.
Thanks Patrick, that helped!
Best Regards,
Deepak
On Thu, May 17, 2012 at 3:23 PM, Patrick Hunt <ph...@apache.org> wrote:
> The deadline exceeded messages are generated by select - the client
> will select(timeout). If the requested timeout is exceeded by more
> than 10ms then that message is printed. Often this will happen in a
> virtualized client environment (not the case here from what you are
> saying), or an overloaded OS, etc...
>
> You are running 12 client processes, where the client code expects a
> 10second session timeout (so heartbeat responses w/in 3.3 seconds)
> along three vms running three servers all on the same laptop, it's
> likely that you are overloaded.
>
> Patrick
>
> On Thu, May 17, 2012 at 11:36 AM, Deepak Jagtap <de...@maxta.com>
> wrote:
> > Hi,
> >
> > Thanks for quick reply!
> > I am seeing sequence of "exceeded deadline" message then connection
> timed
> > out error messages.
> > My code has some usleep() calls, when a sequence of zoo_get and zoo_set
> > returns ZBADVERSION error due to race condition,
> > it invokes usleep() for random time(between 5ms to 100ms) and then retry
> > the same operation. Does this may be cause any issues?
> >
> > Please find following log dump:
> >
> > 2012-05-17 11:10:12,910:6709(0x7f12d52d2700):ZOO_WARN@zookeeper_interest
> @1557:
> > Exceeded deadline by 14ms
> > 2012-05-17 11:10:13,200:6709(0x7f12d52d2700):ZOO_WARN@zookeeper_interest
> @1557:
> > Exceeded deadline by 11ms
> > .
> > .
> > 2012-05-17 11:10:13,270:6709(0x7f12d52d2700):ZOO_WARN@zookeeper_interest
> @1557:
> > Exceeded deadline by 11ms
> > 2012-05-17 11:10:13,745:6709(0x7f12d52d2700):ZOO_WARN@zookeeper_interest
> @1557:
> > Exceeded deadline by 11ms
> > 2012-05-17
> > 11:10:14,164:6709(0x7f12d52d2700):ZOO_ERROR@handle_socket_error_msg
> @1644:
> > Socket [192.168.1.236:2181] zk retcode=-7, errno=110(Connection timed
> out):
> > connection timed out (exceeded timeout by 0ms)
> > operation timeout
> > operation timeout
> > 2012-05-17 11:10:14,164:6709(0x7f12d52d2700):ZOO_INFO@check_events@1701:
> > initiated connection to server [192.168.1.234:2181]
> >
> >
> > Thanks & Regards,
> > Deepak
> >
> > On Wed, May 16, 2012 at 6:30 PM, 李赫元 <li...@gmail.com> wrote:
> >
> >> Hello,
> >>
> >> Did you get the "ZOO_SESSION_EVENT" with "ZOO_EXPIRED_SESSION_STATE"
> >> on watch callback event or just simply debug information print on
> >> screen showing "Exceeded deadline by 16ms"?
> >>
> >> Thanks.
> >>
> >>
> >> 2012/5/17 Deepak Jagtap <de...@maxta.com>:
> >> > Hi All,
> >> >
> >> > I am using Zookeeper 3.4.3 sever and my setup is as follows:
> >> >
> >> > I have a Quorum of 3 Zookeepers servers (installed on 3 virtual
> machines)
> >> > on my laptop.
> >> > Then I start 12 Zookeeper client processes on the same machine which
> >> start
> >> > updates on Zookeeper database.
> >> > After running test for a while, sometimes some of the client process
> got
> >> > disconnected from Zookeeper sever with following error:
> >> > *Exceeded deadline by 16ms* and then says session timeout.
> >> > Client connects with timeout value of 10000ms.
> >> >
> >> > Zookeeper server configuration is as follows:
> >> > tickTime=2000
> >> > initLimit=10
> >> > syncLimit=5
> >> > dataDir=/tmp/zookeeper
> >> > clientPort=2181
> >> > server.1=192.168.1.234:2888:3888
> >> > server.2=192.168.1.235:2888:3888
> >> > server.3=192.168.1.236:2888:3888
> >> >
> >> > Does anybody facing similar problem? Please let me know if there
> anythin
> >> > wrong wit configuration.
> >> >
> >> > Thanks & Regards,
> >> > Deepak
> >>
>
Re: Exceeded deadline by 16ms
Posted by Patrick Hunt <ph...@apache.org>.
The deadline exceeded messages are generated by select - the client
will select(timeout). If the requested timeout is exceeded by more
than 10ms then that message is printed. Often this will happen in a
virtualized client environment (not the case here from what you are
saying), or an overloaded OS, etc...
You are running 12 client processes, where the client code expects a
10second session timeout (so heartbeat responses w/in 3.3 seconds)
along three vms running three servers all on the same laptop, it's
likely that you are overloaded.
Patrick
On Thu, May 17, 2012 at 11:36 AM, Deepak Jagtap <de...@maxta.com> wrote:
> Hi,
>
> Thanks for quick reply!
> I am seeing sequence of "exceeded deadline" message then connection timed
> out error messages.
> My code has some usleep() calls, when a sequence of zoo_get and zoo_set
> returns ZBADVERSION error due to race condition,
> it invokes usleep() for random time(between 5ms to 100ms) and then retry
> the same operation. Does this may be cause any issues?
>
> Please find following log dump:
>
> 2012-05-17 11:10:12,910:6709(0x7f12d52d2700):ZOO_WARN@zookeeper_interest@1557:
> Exceeded deadline by 14ms
> 2012-05-17 11:10:13,200:6709(0x7f12d52d2700):ZOO_WARN@zookeeper_interest@1557:
> Exceeded deadline by 11ms
> .
> .
> 2012-05-17 11:10:13,270:6709(0x7f12d52d2700):ZOO_WARN@zookeeper_interest@1557:
> Exceeded deadline by 11ms
> 2012-05-17 11:10:13,745:6709(0x7f12d52d2700):ZOO_WARN@zookeeper_interest@1557:
> Exceeded deadline by 11ms
> 2012-05-17
> 11:10:14,164:6709(0x7f12d52d2700):ZOO_ERROR@handle_socket_error_msg@1644:
> Socket [192.168.1.236:2181] zk retcode=-7, errno=110(Connection timed out):
> connection timed out (exceeded timeout by 0ms)
> operation timeout
> operation timeout
> 2012-05-17 11:10:14,164:6709(0x7f12d52d2700):ZOO_INFO@check_events@1701:
> initiated connection to server [192.168.1.234:2181]
>
>
> Thanks & Regards,
> Deepak
>
> On Wed, May 16, 2012 at 6:30 PM, 李赫元 <li...@gmail.com> wrote:
>
>> Hello,
>>
>> Did you get the "ZOO_SESSION_EVENT" with "ZOO_EXPIRED_SESSION_STATE"
>> on watch callback event or just simply debug information print on
>> screen showing "Exceeded deadline by 16ms"?
>>
>> Thanks.
>>
>>
>> 2012/5/17 Deepak Jagtap <de...@maxta.com>:
>> > Hi All,
>> >
>> > I am using Zookeeper 3.4.3 sever and my setup is as follows:
>> >
>> > I have a Quorum of 3 Zookeepers servers (installed on 3 virtual machines)
>> > on my laptop.
>> > Then I start 12 Zookeeper client processes on the same machine which
>> start
>> > updates on Zookeeper database.
>> > After running test for a while, sometimes some of the client process got
>> > disconnected from Zookeeper sever with following error:
>> > *Exceeded deadline by 16ms* and then says session timeout.
>> > Client connects with timeout value of 10000ms.
>> >
>> > Zookeeper server configuration is as follows:
>> > tickTime=2000
>> > initLimit=10
>> > syncLimit=5
>> > dataDir=/tmp/zookeeper
>> > clientPort=2181
>> > server.1=192.168.1.234:2888:3888
>> > server.2=192.168.1.235:2888:3888
>> > server.3=192.168.1.236:2888:3888
>> >
>> > Does anybody facing similar problem? Please let me know if there anythin
>> > wrong wit configuration.
>> >
>> > Thanks & Regards,
>> > Deepak
>>
Re: Exceeded deadline by 16ms
Posted by Deepak Jagtap <de...@maxta.com>.
Hi,
Thanks for quick reply!
I am seeing sequence of "exceeded deadline" message then connection timed
out error messages.
My code has some usleep() calls, when a sequence of zoo_get and zoo_set
returns ZBADVERSION error due to race condition,
it invokes usleep() for random time(between 5ms to 100ms) and then retry
the same operation. Does this may be cause any issues?
Please find following log dump:
2012-05-17 11:10:12,910:6709(0x7f12d52d2700):ZOO_WARN@zookeeper_interest@1557:
Exceeded deadline by 14ms
2012-05-17 11:10:13,200:6709(0x7f12d52d2700):ZOO_WARN@zookeeper_interest@1557:
Exceeded deadline by 11ms
.
.
2012-05-17 11:10:13,270:6709(0x7f12d52d2700):ZOO_WARN@zookeeper_interest@1557:
Exceeded deadline by 11ms
2012-05-17 11:10:13,745:6709(0x7f12d52d2700):ZOO_WARN@zookeeper_interest@1557:
Exceeded deadline by 11ms
2012-05-17
11:10:14,164:6709(0x7f12d52d2700):ZOO_ERROR@handle_socket_error_msg@1644:
Socket [192.168.1.236:2181] zk retcode=-7, errno=110(Connection timed out):
connection timed out (exceeded timeout by 0ms)
operation timeout
operation timeout
2012-05-17 11:10:14,164:6709(0x7f12d52d2700):ZOO_INFO@check_events@1701:
initiated connection to server [192.168.1.234:2181]
Thanks & Regards,
Deepak
On Wed, May 16, 2012 at 6:30 PM, 李赫元 <li...@gmail.com> wrote:
> Hello,
>
> Did you get the "ZOO_SESSION_EVENT" with "ZOO_EXPIRED_SESSION_STATE"
> on watch callback event or just simply debug information print on
> screen showing "Exceeded deadline by 16ms"?
>
> Thanks.
>
>
> 2012/5/17 Deepak Jagtap <de...@maxta.com>:
> > Hi All,
> >
> > I am using Zookeeper 3.4.3 sever and my setup is as follows:
> >
> > I have a Quorum of 3 Zookeepers servers (installed on 3 virtual machines)
> > on my laptop.
> > Then I start 12 Zookeeper client processes on the same machine which
> start
> > updates on Zookeeper database.
> > After running test for a while, sometimes some of the client process got
> > disconnected from Zookeeper sever with following error:
> > *Exceeded deadline by 16ms* and then says session timeout.
> > Client connects with timeout value of 10000ms.
> >
> > Zookeeper server configuration is as follows:
> > tickTime=2000
> > initLimit=10
> > syncLimit=5
> > dataDir=/tmp/zookeeper
> > clientPort=2181
> > server.1=192.168.1.234:2888:3888
> > server.2=192.168.1.235:2888:3888
> > server.3=192.168.1.236:2888:3888
> >
> > Does anybody facing similar problem? Please let me know if there anythin
> > wrong wit configuration.
> >
> > Thanks & Regards,
> > Deepak
>
Re: Exceeded deadline by 16ms
Posted by 李赫元 <li...@gmail.com>.
Hello,
Did you get the "ZOO_SESSION_EVENT" with "ZOO_EXPIRED_SESSION_STATE"
on watch callback event or just simply debug information print on
screen showing "Exceeded deadline by 16ms"?
Thanks.
2012/5/17 Deepak Jagtap <de...@maxta.com>:
> Hi All,
>
> I am using Zookeeper 3.4.3 sever and my setup is as follows:
>
> I have a Quorum of 3 Zookeepers servers (installed on 3 virtual machines)
> on my laptop.
> Then I start 12 Zookeeper client processes on the same machine which start
> updates on Zookeeper database.
> After running test for a while, sometimes some of the client process got
> disconnected from Zookeeper sever with following error:
> *Exceeded deadline by 16ms* and then says session timeout.
> Client connects with timeout value of 10000ms.
>
> Zookeeper server configuration is as follows:
> tickTime=2000
> initLimit=10
> syncLimit=5
> dataDir=/tmp/zookeeper
> clientPort=2181
> server.1=192.168.1.234:2888:3888
> server.2=192.168.1.235:2888:3888
> server.3=192.168.1.236:2888:3888
>
> Does anybody facing similar problem? Please let me know if there anythin
> wrong wit configuration.
>
> Thanks & Regards,
> Deepak