You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@zookeeper.apache.org by Jared Cantwell <ja...@gmail.com> on 2011/07/01 17:06:23 UTC
Re: Serious problem processing hearbeat on login stampede
As a note, I believe we just used this patch to solve a major issue we were
seeing. We were having problems when power to a node was pulled, and thus
hung tcp sessions on the servers. With many connections, each close
operation was taking 2 seconds and held up the server significantly enough
to start incorrectly closing other sessions. By disabling linger, these
hanging sessions were closed immediately and the problem went away.
Thanks Chang!
~Jared
On Tue, Apr 19, 2011 at 10:59 AM, Ted Dunning <te...@gmail.com> wrote:
> Where is this set?
>
> Why does this cause this problem?
>
> 2011/4/19 Chang Song <tr...@me.com>
>
> >
> > Problem solved.
> > it was socket linger option set to 2 sec timeout.
> >
> > We have verified that the original problem goes away when we turn off
> > linger option.
> > No longer a mystery ;)
> >
> >
> > https://issues.apache.org/jira/browse/ZOOKEEPER-1049
> >
> >
> > Chang
> >
> >
> > 2011. 4. 19., 오전 3:16, Mahadev Konar 작성:
> >
> > > Camille, Ted,
> > > Can we continue the discussion on
> > > https://issues.apache.org/jira/browse/ZOOKEEPER-1049?
> > >
> > > We should track all the suggestions/issues on the jira.
> > >
> > > thanks
> > > mahadev
> > >
> > > On Mon, Apr 18, 2011 at 9:03 AM, Ted Dunning <te...@gmail.com>
> > wrote:
> > >> Interesting. It does seem to suggestion the session expiration is
> > >> expensive.
> > >>
> > >> There is a concurrent table in guava that provides very good
> > multi-threaded
> > >> performance. I think that is achieved by using a number of locks and
> > then
> > >> distributing threads across the locks according to the hash slot being
> > used.
> > >> But I would have expected any in memory operation to complete very
> > quickly.
> > >>
> > >> Is it possible that the locks on the session table are held longer
> than
> > they
> > >> should be?
> > >>
> > >> 2011/4/18 Fournier, Camille F. [Tech] <Ca...@gs.com>
> > >>
> > >>> Is it possible this is related to this report back in February?
> > >>>
> > >>>
> >
> http://mail-archives.apache.org/mod_mbox/zookeeper-user/201102.mbox/%3C6642FC1CAF133548AA8FDF497C547F0A23C0C5265B@NYWEXMBX2126.msad.ms.com%3E
> > >>>
> > >>> I theorized that the issue might be due to synchronization on the
> > session
> > >>> table, but never got enough information to finish the investigation.
> > >>>
> > >>
> > >
> > >
> > >
> > > --
> > > thanks
> > > mahadev
> > > @mahadevkonar
> >
> >
>
Re: Serious problem processing hearbeat on login stampede
Posted by Chang Song <tr...@me.com>.
Actually, "netspider", Chisu Ryu, in my team fixed it.
Thanks, Chisu.
Chang
2011. 7. 6., 오전 3:04, Patrick Hunt 작성:
> Vishal brought up an issue at the ZK post-summit meetup that might
> also be (partially?) resolved by this patch.
>
> Thanks again Chang Song!
>
> Patrick
>
> 2011/7/1 Chang Song <tr...@me.com>:
>>
>> No problem.
>> Glad to contribute.
>>
>> Thanks a lot.
>>
>>
>> 2011. 7. 2., 오전 1:03, Ted Dunning 작성:
>>
>>> Thanks for the feedback Jared!
>>>
>>> (and thanks to Chang as well!)
>>>
>>> On Fri, Jul 1, 2011 at 8:06 AM, Jared Cantwell <ja...@gmail.com>wrote:
>>>
>>>> As a note, I believe we just used this patch to solve a major issue ...
>>>>
>>>> Thanks Chang!
>>>> ~Jared
>>>>
>>>> On Tue, Apr 19, 2011 at 10:59 AM, Ted Dunning <te...@gmail.com>
>>>> wrote:
>>>>
>>>>> Where is this set?
>>>>>
>>>>> Why does this cause this problem?
>>>>>
>>>>> 2011/4/19 Chang Song <tr...@me.com>
>>>>>
>>>>>>
>>>>>> Problem solved.
>>>>>> it was socket linger option set to 2 sec timeout.
>>>>>>
>>>>>> We have verified that the original problem goes away when we turn off
>>>>>> linger option.
>>>>>> No longer a mystery ;)
>>>>>>
>>>>>>
>>>>>> https://issues.apache.org/jira/browse/ZOOKEEPER-1049
>>>>>>
>>>>>>
>>>>>> Chang
>>>>>>
>>>>>>
>>>>>> 2011. 4. 19., 오전 3:16, Mahadev Konar 작성:
>>>>>>
>>>>>>> Camille, Ted,
>>>>>>> Can we continue the discussion on
>>>>>>> https://issues.apache.org/jira/browse/ZOOKEEPER-1049?
>>>>>>>
>>>>>>> We should track all the suggestions/issues on the jira.
>>>>>>>
>>>>>>> thanks
>>>>>>> mahadev
>>>>>>>
>>>>>>> On Mon, Apr 18, 2011 at 9:03 AM, Ted Dunning <te...@gmail.com>
>>>>>> wrote:
>>>>>>>> Interesting. It does seem to suggestion the session expiration is
>>>>>>>> expensive.
>>>>>>>>
>>>>>>>> There is a concurrent table in guava that provides very good
>>>>>> multi-threaded
>>>>>>>> performance. I think that is achieved by using a number of locks
>>>> and
>>>>>> then
>>>>>>>> distributing threads across the locks according to the hash slot
>>>> being
>>>>>> used.
>>>>>>>> But I would have expected any in memory operation to complete very
>>>>>> quickly.
>>>>>>>>
>>>>>>>> Is it possible that the locks on the session table are held longer
>>>>> than
>>>>>> they
>>>>>>>> should be?
>>>>>>>>
>>>>>>>> 2011/4/18 Fournier, Camille F. [Tech] <Ca...@gs.com>
>>>>>>>>
>>>>>>>>> Is it possible this is related to this report back in February?
>>>>>>>>>
>>>>>>>>>
>>>>>>
>>>>>
>>>> http://mail-archives.apache.org/mod_mbox/zookeeper-user/201102.mbox/%3C6642FC1CAF133548AA8FDF497C547F0A23C0C5265B@NYWEXMBX2126.msad.ms.com%3E
>>>>>>>>>
>>>>>>>>> I theorized that the issue might be due to synchronization on the
>>>>>> session
>>>>>>>>> table, but never got enough information to finish the
>>>> investigation.
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> thanks
>>>>>>> mahadev
>>>>>>> @mahadevkonar
>>>>>>
>>>>>>
>>>>>
>>>>
>>
>>
Re: Serious problem processing hearbeat on login stampede
Posted by Patrick Hunt <ph...@apache.org>.
Vishal brought up an issue at the ZK post-summit meetup that might
also be (partially?) resolved by this patch.
Thanks again Chang Song!
Patrick
2011/7/1 Chang Song <tr...@me.com>:
>
> No problem.
> Glad to contribute.
>
> Thanks a lot.
>
>
> 2011. 7. 2., 오전 1:03, Ted Dunning 작성:
>
>> Thanks for the feedback Jared!
>>
>> (and thanks to Chang as well!)
>>
>> On Fri, Jul 1, 2011 at 8:06 AM, Jared Cantwell <ja...@gmail.com>wrote:
>>
>>> As a note, I believe we just used this patch to solve a major issue ...
>>>
>>> Thanks Chang!
>>> ~Jared
>>>
>>> On Tue, Apr 19, 2011 at 10:59 AM, Ted Dunning <te...@gmail.com>
>>> wrote:
>>>
>>>> Where is this set?
>>>>
>>>> Why does this cause this problem?
>>>>
>>>> 2011/4/19 Chang Song <tr...@me.com>
>>>>
>>>>>
>>>>> Problem solved.
>>>>> it was socket linger option set to 2 sec timeout.
>>>>>
>>>>> We have verified that the original problem goes away when we turn off
>>>>> linger option.
>>>>> No longer a mystery ;)
>>>>>
>>>>>
>>>>> https://issues.apache.org/jira/browse/ZOOKEEPER-1049
>>>>>
>>>>>
>>>>> Chang
>>>>>
>>>>>
>>>>> 2011. 4. 19., 오전 3:16, Mahadev Konar 작성:
>>>>>
>>>>>> Camille, Ted,
>>>>>> Can we continue the discussion on
>>>>>> https://issues.apache.org/jira/browse/ZOOKEEPER-1049?
>>>>>>
>>>>>> We should track all the suggestions/issues on the jira.
>>>>>>
>>>>>> thanks
>>>>>> mahadev
>>>>>>
>>>>>> On Mon, Apr 18, 2011 at 9:03 AM, Ted Dunning <te...@gmail.com>
>>>>> wrote:
>>>>>>> Interesting. It does seem to suggestion the session expiration is
>>>>>>> expensive.
>>>>>>>
>>>>>>> There is a concurrent table in guava that provides very good
>>>>> multi-threaded
>>>>>>> performance. I think that is achieved by using a number of locks
>>> and
>>>>> then
>>>>>>> distributing threads across the locks according to the hash slot
>>> being
>>>>> used.
>>>>>>> But I would have expected any in memory operation to complete very
>>>>> quickly.
>>>>>>>
>>>>>>> Is it possible that the locks on the session table are held longer
>>>> than
>>>>> they
>>>>>>> should be?
>>>>>>>
>>>>>>> 2011/4/18 Fournier, Camille F. [Tech] <Ca...@gs.com>
>>>>>>>
>>>>>>>> Is it possible this is related to this report back in February?
>>>>>>>>
>>>>>>>>
>>>>>
>>>>
>>> http://mail-archives.apache.org/mod_mbox/zookeeper-user/201102.mbox/%3C6642FC1CAF133548AA8FDF497C547F0A23C0C5265B@NYWEXMBX2126.msad.ms.com%3E
>>>>>>>>
>>>>>>>> I theorized that the issue might be due to synchronization on the
>>>>> session
>>>>>>>> table, but never got enough information to finish the
>>> investigation.
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> thanks
>>>>>> mahadev
>>>>>> @mahadevkonar
>>>>>
>>>>>
>>>>
>>>
>
>
Re: Serious problem processing hearbeat on login stampede
Posted by Chang Song <tr...@me.com>.
No problem.
Glad to contribute.
Thanks a lot.
2011. 7. 2., 오전 1:03, Ted Dunning 작성:
> Thanks for the feedback Jared!
>
> (and thanks to Chang as well!)
>
> On Fri, Jul 1, 2011 at 8:06 AM, Jared Cantwell <ja...@gmail.com>wrote:
>
>> As a note, I believe we just used this patch to solve a major issue ...
>>
>> Thanks Chang!
>> ~Jared
>>
>> On Tue, Apr 19, 2011 at 10:59 AM, Ted Dunning <te...@gmail.com>
>> wrote:
>>
>>> Where is this set?
>>>
>>> Why does this cause this problem?
>>>
>>> 2011/4/19 Chang Song <tr...@me.com>
>>>
>>>>
>>>> Problem solved.
>>>> it was socket linger option set to 2 sec timeout.
>>>>
>>>> We have verified that the original problem goes away when we turn off
>>>> linger option.
>>>> No longer a mystery ;)
>>>>
>>>>
>>>> https://issues.apache.org/jira/browse/ZOOKEEPER-1049
>>>>
>>>>
>>>> Chang
>>>>
>>>>
>>>> 2011. 4. 19., 오전 3:16, Mahadev Konar 작성:
>>>>
>>>>> Camille, Ted,
>>>>> Can we continue the discussion on
>>>>> https://issues.apache.org/jira/browse/ZOOKEEPER-1049?
>>>>>
>>>>> We should track all the suggestions/issues on the jira.
>>>>>
>>>>> thanks
>>>>> mahadev
>>>>>
>>>>> On Mon, Apr 18, 2011 at 9:03 AM, Ted Dunning <te...@gmail.com>
>>>> wrote:
>>>>>> Interesting. It does seem to suggestion the session expiration is
>>>>>> expensive.
>>>>>>
>>>>>> There is a concurrent table in guava that provides very good
>>>> multi-threaded
>>>>>> performance. I think that is achieved by using a number of locks
>> and
>>>> then
>>>>>> distributing threads across the locks according to the hash slot
>> being
>>>> used.
>>>>>> But I would have expected any in memory operation to complete very
>>>> quickly.
>>>>>>
>>>>>> Is it possible that the locks on the session table are held longer
>>> than
>>>> they
>>>>>> should be?
>>>>>>
>>>>>> 2011/4/18 Fournier, Camille F. [Tech] <Ca...@gs.com>
>>>>>>
>>>>>>> Is it possible this is related to this report back in February?
>>>>>>>
>>>>>>>
>>>>
>>>
>> http://mail-archives.apache.org/mod_mbox/zookeeper-user/201102.mbox/%3C6642FC1CAF133548AA8FDF497C547F0A23C0C5265B@NYWEXMBX2126.msad.ms.com%3E
>>>>>>>
>>>>>>> I theorized that the issue might be due to synchronization on the
>>>> session
>>>>>>> table, but never got enough information to finish the
>> investigation.
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> thanks
>>>>> mahadev
>>>>> @mahadevkonar
>>>>
>>>>
>>>
>>
Re: Serious problem processing hearbeat on login stampede
Posted by Ted Dunning <te...@gmail.com>.
Thanks for the feedback Jared!
(and thanks to Chang as well!)
On Fri, Jul 1, 2011 at 8:06 AM, Jared Cantwell <ja...@gmail.com>wrote:
> As a note, I believe we just used this patch to solve a major issue ...
>
> Thanks Chang!
> ~Jared
>
> On Tue, Apr 19, 2011 at 10:59 AM, Ted Dunning <te...@gmail.com>
> wrote:
>
> > Where is this set?
> >
> > Why does this cause this problem?
> >
> > 2011/4/19 Chang Song <tr...@me.com>
> >
> > >
> > > Problem solved.
> > > it was socket linger option set to 2 sec timeout.
> > >
> > > We have verified that the original problem goes away when we turn off
> > > linger option.
> > > No longer a mystery ;)
> > >
> > >
> > > https://issues.apache.org/jira/browse/ZOOKEEPER-1049
> > >
> > >
> > > Chang
> > >
> > >
> > > 2011. 4. 19., 오전 3:16, Mahadev Konar 작성:
> > >
> > > > Camille, Ted,
> > > > Can we continue the discussion on
> > > > https://issues.apache.org/jira/browse/ZOOKEEPER-1049?
> > > >
> > > > We should track all the suggestions/issues on the jira.
> > > >
> > > > thanks
> > > > mahadev
> > > >
> > > > On Mon, Apr 18, 2011 at 9:03 AM, Ted Dunning <te...@gmail.com>
> > > wrote:
> > > >> Interesting. It does seem to suggestion the session expiration is
> > > >> expensive.
> > > >>
> > > >> There is a concurrent table in guava that provides very good
> > > multi-threaded
> > > >> performance. I think that is achieved by using a number of locks
> and
> > > then
> > > >> distributing threads across the locks according to the hash slot
> being
> > > used.
> > > >> But I would have expected any in memory operation to complete very
> > > quickly.
> > > >>
> > > >> Is it possible that the locks on the session table are held longer
> > than
> > > they
> > > >> should be?
> > > >>
> > > >> 2011/4/18 Fournier, Camille F. [Tech] <Ca...@gs.com>
> > > >>
> > > >>> Is it possible this is related to this report back in February?
> > > >>>
> > > >>>
> > >
> >
> http://mail-archives.apache.org/mod_mbox/zookeeper-user/201102.mbox/%3C6642FC1CAF133548AA8FDF497C547F0A23C0C5265B@NYWEXMBX2126.msad.ms.com%3E
> > > >>>
> > > >>> I theorized that the issue might be due to synchronization on the
> > > session
> > > >>> table, but never got enough information to finish the
> investigation.
> > > >>>
> > > >>
> > > >
> > > >
> > > >
> > > > --
> > > > thanks
> > > > mahadev
> > > > @mahadevkonar
> > >
> > >
> >
>