You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Michael Tsadikov <mi...@myheritage.com> on 2012/07/01 13:22:33 UTC
leap second bug
Our solr servers went into GC hell, and became non-responsive on date
change today.
Restarting tomcats did not help.
Rebooting the machine did.
http://www.wired.com/wiredenterprise/2012/07/leap-second-bug-wreaks-havoc-with-java-linux/
Re: leap second bug
Posted by Michael Tsadikov <mi...@myheritage.com>.
explanation of the cause:
https://lkml.org/lkml/2012/7/1/203
On Wed, Jul 4, 2012 at 1:48 AM, Óscar Marín Miró
<os...@gmail.com>wrote:
> So, this was the solution, sorry to post it so late, just in case it helps
> anyone:
>
> /etc/init.d/ntp stop; date; date `date +"%m%d%H%M%C%y.%S"`; date;
> /etc/init.d/ntp start
>
> And tomcat magically switched from 100% CPU to 0.5% :)
>
> From:
>
>
> https://groups.google.com/forum/?fromgroups#!topic/elasticsearch/_I1_OfaL7QY
>
> [from Michael McCandless help on this thread]
>
> On Sun, Jul 1, 2012 at 6:15 PM, Jack Krupansky <jack@basetechnology.com
> >wrote:
>
> > Interesting:
> >
> > "
> > The sequence of dates of the UTC second markers will be:
> >
> > 2012 June 30, 23h 59m 59s
> > 2012 June 30, 23h 59m 60s
> > 2012 July 1, 0h 0m 0s
> > "
> >
> > See:
> > http://wwp.greenwichmeantime.**com/info/leap-second.htm<
> http://wwp.greenwichmeantime.com/info/leap-second.htm>
> >
> > So, there were two consecutive second " markers" which were literally
> > distinct, but numerically identical.
> >
> > What "design pattern" for timing did Linux violate? In other words, what
> > lesson should we be learning to assure that we don't have a similar
> problem
> > at an application level on a future leap second?
> >
> > -- Jack Krupansky
> >
> > -----Original Message----- From: Óscar Marín Miró
> > Sent: Sunday, July 01, 2012 11:02 AM
> > To: solr-user@lucene.apache.org
> > Subject: Re: leap second bug
> >
> >
> > Thanks Michael, nice information :)
> >
> > On Sun, Jul 1, 2012 at 5:29 PM, Michael McCandless <
> > lucene@mikemccandless.com> wrote:
> >
> > Looks like this is a low-level Linux issue ... see Shay's email to the
> >> ElasticSearch list about it:
> >>
> >>
> >> https://groups.google.com/**forum/?fromgroups#!topic/**
> >> elasticsearch/_I1_OfaL7QY<
> https://groups.google.com/forum/?fromgroups#!topic/elasticsearch/_I1_OfaL7QY
> >
> >>
> >> Also see the comments here:
> >>
> >> http://news.ycombinator.com/**item?id=4182642<
> http://news.ycombinator.com/item?id=4182642>
> >>
> >> Mike McCandless
> >>
> >> http://blog.mikemccandless.com
> >>
> >> On Sun, Jul 1, 2012 at 8:08 AM, Óscar Marín Miró
> >> <os...@gmail.com> wrote:
> >> > Hello Michael, thanks for the note :)
> >> >
> >> > I'm having a similar problem since yesterday, tomcats are wild on CPU
> >> [near
> >> > 100%]. Did your solr servers did not reply to index/query requests?
> >> >
> >> > Thanks :)
> >> >
> >> > On Sun, Jul 1, 2012 at 1:22 PM, Michael Tsadikov <
> >> michael@myheritage.com
> >> >wrote:
> >> >
> >> >> Our solr servers went into GC hell, and became non-responsive on date
> >> >> change today.
> >> >>
> >> >> Restarting tomcats did not help.
> >> >>
> >> >> Rebooting the machine did.
> >> >>
> >> >>
> >> >>
> >> http://www.wired.com/**wiredenterprise/2012/07/leap-**
> >> second-bug-wreaks-havoc-with-**java-linux/<
> http://www.wired.com/wiredenterprise/2012/07/leap-second-bug-wreaks-havoc-with-java-linux/
> >
> >> >>
> >> >
> >> >
> >> >
> >> > --
> >> > Whether it's science, technology, personal experience, true love,
> >> > astrology, or gut feelings, each of us has confidence in something
> that
> >> we
> >> > will never fully comprehend.
> >> > --Roy H. William
> >>
> >>
> >
> >
> > --
> > Whether it's science, technology, personal experience, true love,
> > astrology, or gut feelings, each of us has confidence in something that
> we
> > will never fully comprehend.
> > --Roy H. William
> >
>
>
>
> --
> Whether it's science, technology, personal experience, true love,
> astrology, or gut feelings, each of us has confidence in something that we
> will never fully comprehend.
> --Roy H. William
>
Re: leap second bug
Posted by Óscar Marín Miró <os...@gmail.com>.
So, this was the solution, sorry to post it so late, just in case it helps
anyone:
/etc/init.d/ntp stop; date; date `date +"%m%d%H%M%C%y.%S"`; date;
/etc/init.d/ntp start
And tomcat magically switched from 100% CPU to 0.5% :)
From:
https://groups.google.com/forum/?fromgroups#!topic/elasticsearch/_I1_OfaL7QY
[from Michael McCandless help on this thread]
On Sun, Jul 1, 2012 at 6:15 PM, Jack Krupansky <ja...@basetechnology.com>wrote:
> Interesting:
>
> "
> The sequence of dates of the UTC second markers will be:
>
> 2012 June 30, 23h 59m 59s
> 2012 June 30, 23h 59m 60s
> 2012 July 1, 0h 0m 0s
> "
>
> See:
> http://wwp.greenwichmeantime.**com/info/leap-second.htm<http://wwp.greenwichmeantime.com/info/leap-second.htm>
>
> So, there were two consecutive second " markers" which were literally
> distinct, but numerically identical.
>
> What "design pattern" for timing did Linux violate? In other words, what
> lesson should we be learning to assure that we don't have a similar problem
> at an application level on a future leap second?
>
> -- Jack Krupansky
>
> -----Original Message----- From: Óscar Marín Miró
> Sent: Sunday, July 01, 2012 11:02 AM
> To: solr-user@lucene.apache.org
> Subject: Re: leap second bug
>
>
> Thanks Michael, nice information :)
>
> On Sun, Jul 1, 2012 at 5:29 PM, Michael McCandless <
> lucene@mikemccandless.com> wrote:
>
> Looks like this is a low-level Linux issue ... see Shay's email to the
>> ElasticSearch list about it:
>>
>>
>> https://groups.google.com/**forum/?fromgroups#!topic/**
>> elasticsearch/_I1_OfaL7QY<https://groups.google.com/forum/?fromgroups#!topic/elasticsearch/_I1_OfaL7QY>
>>
>> Also see the comments here:
>>
>> http://news.ycombinator.com/**item?id=4182642<http://news.ycombinator.com/item?id=4182642>
>>
>> Mike McCandless
>>
>> http://blog.mikemccandless.com
>>
>> On Sun, Jul 1, 2012 at 8:08 AM, Óscar Marín Miró
>> <os...@gmail.com> wrote:
>> > Hello Michael, thanks for the note :)
>> >
>> > I'm having a similar problem since yesterday, tomcats are wild on CPU
>> [near
>> > 100%]. Did your solr servers did not reply to index/query requests?
>> >
>> > Thanks :)
>> >
>> > On Sun, Jul 1, 2012 at 1:22 PM, Michael Tsadikov <
>> michael@myheritage.com
>> >wrote:
>> >
>> >> Our solr servers went into GC hell, and became non-responsive on date
>> >> change today.
>> >>
>> >> Restarting tomcats did not help.
>> >>
>> >> Rebooting the machine did.
>> >>
>> >>
>> >>
>> http://www.wired.com/**wiredenterprise/2012/07/leap-**
>> second-bug-wreaks-havoc-with-**java-linux/<http://www.wired.com/wiredenterprise/2012/07/leap-second-bug-wreaks-havoc-with-java-linux/>
>> >>
>> >
>> >
>> >
>> > --
>> > Whether it's science, technology, personal experience, true love,
>> > astrology, or gut feelings, each of us has confidence in something that
>> we
>> > will never fully comprehend.
>> > --Roy H. William
>>
>>
>
>
> --
> Whether it's science, technology, personal experience, true love,
> astrology, or gut feelings, each of us has confidence in something that we
> will never fully comprehend.
> --Roy H. William
>
--
Whether it's science, technology, personal experience, true love,
astrology, or gut feelings, each of us has confidence in something that we
will never fully comprehend.
--Roy H. William
Re: leap second bug
Posted by Jack Krupansky <ja...@basetechnology.com>.
Interesting:
"
The sequence of dates of the UTC second markers will be:
2012 June 30, 23h 59m 59s
2012 June 30, 23h 59m 60s
2012 July 1, 0h 0m 0s
"
See:
http://wwp.greenwichmeantime.com/info/leap-second.htm
So, there were two consecutive second " markers" which were literally
distinct, but numerically identical.
What "design pattern" for timing did Linux violate? In other words, what
lesson should we be learning to assure that we don't have a similar problem
at an application level on a future leap second?
-- Jack Krupansky
-----Original Message-----
From: Óscar Marín Miró
Sent: Sunday, July 01, 2012 11:02 AM
To: solr-user@lucene.apache.org
Subject: Re: leap second bug
Thanks Michael, nice information :)
On Sun, Jul 1, 2012 at 5:29 PM, Michael McCandless <
lucene@mikemccandless.com> wrote:
> Looks like this is a low-level Linux issue ... see Shay's email to the
> ElasticSearch list about it:
>
>
> https://groups.google.com/forum/?fromgroups#!topic/elasticsearch/_I1_OfaL7QY
>
> Also see the comments here:
>
> http://news.ycombinator.com/item?id=4182642
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
> On Sun, Jul 1, 2012 at 8:08 AM, Óscar Marín Miró
> <os...@gmail.com> wrote:
> > Hello Michael, thanks for the note :)
> >
> > I'm having a similar problem since yesterday, tomcats are wild on CPU
> [near
> > 100%]. Did your solr servers did not reply to index/query requests?
> >
> > Thanks :)
> >
> > On Sun, Jul 1, 2012 at 1:22 PM, Michael Tsadikov <michael@myheritage.com
> >wrote:
> >
> >> Our solr servers went into GC hell, and became non-responsive on date
> >> change today.
> >>
> >> Restarting tomcats did not help.
> >>
> >> Rebooting the machine did.
> >>
> >>
> >>
> http://www.wired.com/wiredenterprise/2012/07/leap-second-bug-wreaks-havoc-with-java-linux/
> >>
> >
> >
> >
> > --
> > Whether it's science, technology, personal experience, true love,
> > astrology, or gut feelings, each of us has confidence in something that
> we
> > will never fully comprehend.
> > --Roy H. William
>
--
Whether it's science, technology, personal experience, true love,
astrology, or gut feelings, each of us has confidence in something that we
will never fully comprehend.
--Roy H. William
Re: leap second bug
Posted by Óscar Marín Miró <os...@gmail.com>.
Thanks Michael, nice information :)
On Sun, Jul 1, 2012 at 5:29 PM, Michael McCandless <
lucene@mikemccandless.com> wrote:
> Looks like this is a low-level Linux issue ... see Shay's email to the
> ElasticSearch list about it:
>
>
> https://groups.google.com/forum/?fromgroups#!topic/elasticsearch/_I1_OfaL7QY
>
> Also see the comments here:
>
> http://news.ycombinator.com/item?id=4182642
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
> On Sun, Jul 1, 2012 at 8:08 AM, Óscar Marín Miró
> <os...@gmail.com> wrote:
> > Hello Michael, thanks for the note :)
> >
> > I'm having a similar problem since yesterday, tomcats are wild on CPU
> [near
> > 100%]. Did your solr servers did not reply to index/query requests?
> >
> > Thanks :)
> >
> > On Sun, Jul 1, 2012 at 1:22 PM, Michael Tsadikov <michael@myheritage.com
> >wrote:
> >
> >> Our solr servers went into GC hell, and became non-responsive on date
> >> change today.
> >>
> >> Restarting tomcats did not help.
> >>
> >> Rebooting the machine did.
> >>
> >>
> >>
> http://www.wired.com/wiredenterprise/2012/07/leap-second-bug-wreaks-havoc-with-java-linux/
> >>
> >
> >
> >
> > --
> > Whether it's science, technology, personal experience, true love,
> > astrology, or gut feelings, each of us has confidence in something that
> we
> > will never fully comprehend.
> > --Roy H. William
>
--
Whether it's science, technology, personal experience, true love,
astrology, or gut feelings, each of us has confidence in something that we
will never fully comprehend.
--Roy H. William
Re: leap second bug
Posted by Michael McCandless <lu...@mikemccandless.com>.
Looks like this is a low-level Linux issue ... see Shay's email to the
ElasticSearch list about it:
https://groups.google.com/forum/?fromgroups#!topic/elasticsearch/_I1_OfaL7QY
Also see the comments here:
http://news.ycombinator.com/item?id=4182642
Mike McCandless
http://blog.mikemccandless.com
On Sun, Jul 1, 2012 at 8:08 AM, Óscar Marín Miró
<os...@gmail.com> wrote:
> Hello Michael, thanks for the note :)
>
> I'm having a similar problem since yesterday, tomcats are wild on CPU [near
> 100%]. Did your solr servers did not reply to index/query requests?
>
> Thanks :)
>
> On Sun, Jul 1, 2012 at 1:22 PM, Michael Tsadikov <mi...@myheritage.com>wrote:
>
>> Our solr servers went into GC hell, and became non-responsive on date
>> change today.
>>
>> Restarting tomcats did not help.
>>
>> Rebooting the machine did.
>>
>>
>> http://www.wired.com/wiredenterprise/2012/07/leap-second-bug-wreaks-havoc-with-java-linux/
>>
>
>
>
> --
> Whether it's science, technology, personal experience, true love,
> astrology, or gut feelings, each of us has confidence in something that we
> will never fully comprehend.
> --Roy H. William
Re: leap second bug
Posted by Óscar Marín Miró <os...@gmail.com>.
Hello Michael, thanks for the note :)
I'm having a similar problem since yesterday, tomcats are wild on CPU [near
100%]. Did your solr servers did not reply to index/query requests?
Thanks :)
On Sun, Jul 1, 2012 at 1:22 PM, Michael Tsadikov <mi...@myheritage.com>wrote:
> Our solr servers went into GC hell, and became non-responsive on date
> change today.
>
> Restarting tomcats did not help.
>
> Rebooting the machine did.
>
>
> http://www.wired.com/wiredenterprise/2012/07/leap-second-bug-wreaks-havoc-with-java-linux/
>
--
Whether it's science, technology, personal experience, true love,
astrology, or gut feelings, each of us has confidence in something that we
will never fully comprehend.
--Roy H. William