You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Varun Sharma <va...@pinterest.com> on 2013/05/01 22:54:23 UTC
JVM seg fault in HBase region server
Hi,
I am seeing the following which is a JVM segfault:
hbase-regionser[28734]: segfault at 8 ip 00007f269bcc307e sp
00007fff50f7e638 error 4 in libc-2.15.so[7f269bc51000+1b5000]
Benoit Tsuna reported a similar issue a while back -
http://blog.tsunanet.net/2011/05/jvm-u24-segfault-in-clearerr-on-jaunty.html
I wonder if there was ever a resolution on the issue. We are running java
version "1.6.0_38"
, ubuntu precise 12.04 - kernel 3.2. The libc version from the above line
seems to be 2.15.
I wonder if people have seen this in the past and if yes, were they able to
resolve it and if yes, then how ? Was it a VM upgrade, a kernel upgrade or
a libc upgrade ?
Thanks
Varun
Re: JVM seg fault in HBase region server
Posted by Graeme Wallace <gr...@farecompare.com>.
Over the years, we've found that one of the causes of spontaneuous JVM
combustion is bad memory. Can you run a memory check on your machine ?
Its not always the case though, and sometimes we end up changing JDK
versions and that fixes it.
G.
On Thu, May 2, 2013 at 7:37 AM, Jean-Marc Spaggiari <jean-marc@spaggiari.org
> wrote:
> Hi Varun,
>
> Never faced this error ever. And I faced a lot ;) Even by running with
> OpenJDK 1.x. and I'm running eglibc on all my servers.
>
> ldd --version
> ldd (Debian EGLIBC 2.13-38) 2.13
> Copyright (C) 2011 Free Software Foundation, Inc.
> This is free software; see the source for copying conditions. There is NO
> warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
> Written by Roland McGrath and Ulrich Drepper.
>
>
> JM
>
> 2013/5/1 Varun Sharma <va...@pinterest.com>
>
> > From what I see from
> > ldd --version
> > ldd (Ubuntu EGLIBC 2.15-0ubuntu10.3) 2.15
> >
> > We are running eglibc which is somewhat different from glibc -
> > http://en.wikipedia.org/wiki/Embedded_GLIBC.
> >
> > It seems that this is a problem with Ubuntu, have folks seen this on non
> > ubuntu installs ?
> >
> > Thanks
> > Varun
> >
> >
> > On Wed, May 1, 2013 at 1:54 PM, Varun Sharma <va...@pinterest.com>
> wrote:
> >
> > > Hi,
> > >
> > > I am seeing the following which is a JVM segfault:
> > >
> > > hbase-regionser[28734]: segfault at 8 ip 00007f269bcc307e sp
> > > 00007fff50f7e638 error 4 in libc-2.15.so[7f269bc51000+1b5000]
> > >
> > > Benoit Tsuna reported a similar issue a while back -
> > >
> >
> http://blog.tsunanet.net/2011/05/jvm-u24-segfault-in-clearerr-on-jaunty.html
> > >
> > > I wonder if there was ever a resolution on the issue. We are running
> java
> > > version "1.6.0_38"
> > > , ubuntu precise 12.04 - kernel 3.2. The libc version from the above
> line
> > > seems to be 2.15.
> > >
> > > I wonder if people have seen this in the past and if yes, were they
> able
> > > to resolve it and if yes, then how ? Was it a VM upgrade, a kernel
> > upgrade
> > > or a libc upgrade ?
> > >
> > > Thanks
> > > Varun
> > >
> > >
> > >
> >
>
--
Graeme Wallace
CTO
FareCompare.com
O: 972 588 1414
M: 214 681 9018
Re: JVM seg fault in HBase region server
Posted by Jean-Marc Spaggiari <je...@spaggiari.org>.
Hi Varun,
Never faced this error ever. And I faced a lot ;) Even by running with
OpenJDK 1.x. and I'm running eglibc on all my servers.
ldd --version
ldd (Debian EGLIBC 2.13-38) 2.13
Copyright (C) 2011 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Written by Roland McGrath and Ulrich Drepper.
JM
2013/5/1 Varun Sharma <va...@pinterest.com>
> From what I see from
> ldd --version
> ldd (Ubuntu EGLIBC 2.15-0ubuntu10.3) 2.15
>
> We are running eglibc which is somewhat different from glibc -
> http://en.wikipedia.org/wiki/Embedded_GLIBC.
>
> It seems that this is a problem with Ubuntu, have folks seen this on non
> ubuntu installs ?
>
> Thanks
> Varun
>
>
> On Wed, May 1, 2013 at 1:54 PM, Varun Sharma <va...@pinterest.com> wrote:
>
> > Hi,
> >
> > I am seeing the following which is a JVM segfault:
> >
> > hbase-regionser[28734]: segfault at 8 ip 00007f269bcc307e sp
> > 00007fff50f7e638 error 4 in libc-2.15.so[7f269bc51000+1b5000]
> >
> > Benoit Tsuna reported a similar issue a while back -
> >
> http://blog.tsunanet.net/2011/05/jvm-u24-segfault-in-clearerr-on-jaunty.html
> >
> > I wonder if there was ever a resolution on the issue. We are running java
> > version "1.6.0_38"
> > , ubuntu precise 12.04 - kernel 3.2. The libc version from the above line
> > seems to be 2.15.
> >
> > I wonder if people have seen this in the past and if yes, were they able
> > to resolve it and if yes, then how ? Was it a VM upgrade, a kernel
> upgrade
> > or a libc upgrade ?
> >
> > Thanks
> > Varun
> >
> >
> >
>
Re: JVM seg fault in HBase region server
Posted by Varun Sharma <va...@pinterest.com>.
>From what I see from
ldd --version
ldd (Ubuntu EGLIBC 2.15-0ubuntu10.3) 2.15
We are running eglibc which is somewhat different from glibc -
http://en.wikipedia.org/wiki/Embedded_GLIBC.
It seems that this is a problem with Ubuntu, have folks seen this on non
ubuntu installs ?
Thanks
Varun
On Wed, May 1, 2013 at 1:54 PM, Varun Sharma <va...@pinterest.com> wrote:
> Hi,
>
> I am seeing the following which is a JVM segfault:
>
> hbase-regionser[28734]: segfault at 8 ip 00007f269bcc307e sp
> 00007fff50f7e638 error 4 in libc-2.15.so[7f269bc51000+1b5000]
>
> Benoit Tsuna reported a similar issue a while back -
> http://blog.tsunanet.net/2011/05/jvm-u24-segfault-in-clearerr-on-jaunty.html
>
> I wonder if there was ever a resolution on the issue. We are running java
> version "1.6.0_38"
> , ubuntu precise 12.04 - kernel 3.2. The libc version from the above line
> seems to be 2.15.
>
> I wonder if people have seen this in the past and if yes, were they able
> to resolve it and if yes, then how ? Was it a VM upgrade, a kernel upgrade
> or a libc upgrade ?
>
> Thanks
> Varun
>
>
>
Re: JVM seg fault in HBase region server
Posted by Varun Sharma <va...@pinterest.com>.
Benoit saw this issue with u24 looking from the blog.
Is anyone running older than 1.6.0_38 and not seeing this ?
Thanks
Varun
On Thu, May 2, 2013 at 1:49 PM, Andrew Purtell <ap...@apache.org> wrote:
> In that blog post Benoît does a fair amount of showing off to end up with
> "So this segfault remains a mystery so far" :-)
>
> My take is don't suspect the JVM, suspect your libc. I am not a fan of
> using Ubuntu on the server side. Perhaps you can try a distribution based
> on a different kernel+glibc? (RHEL comes to mind.) I realize that could
> be... inconvenient.
>
>
> On Thu, May 2, 2013 at 1:39 PM, Varun Sharma <va...@pinterest.com> wrote:
>
> > I don't have one unfortunately - We did not have the -XX:ErrorLog turned
> on
> > :(
> >
> > But I did some digging following what Benoit wrote in his Blog. Basically
> > the segfault happens in the same place inside a clearerr() function in
> > glibc which accepts a file descriptor.
> >
> > The JVM version is sun java 1.6.0u38. Is this possibly too old ?
> >
> > Thanks
> > Varun
> >
> >
> > On Thu, May 2, 2013 at 12:08 PM, Andrew Purtell <ap...@apache.org>
> > wrote:
> >
> > > Can you pastebin or post somewhere the entire hs_err* file?
> > >
> > >
> > > On Wed, May 1, 2013 at 1:54 PM, Varun Sharma <va...@pinterest.com>
> > wrote:
> > >
> > > > Hi,
> > > >
> > > > I am seeing the following which is a JVM segfault:
> > > >
> > > > hbase-regionser[28734]: segfault at 8 ip 00007f269bcc307e sp
> > > > 00007fff50f7e638 error 4 in libc-2.15.so[7f269bc51000+1b5000]
> > > >
> > > > Benoit Tsuna reported a similar issue a while back -
> > > >
> > > >
> > >
> >
> http://blog.tsunanet.net/2011/05/jvm-u24-segfault-in-clearerr-on-jaunty.html
> > > >
> > > > I wonder if there was ever a resolution on the issue. We are running
> > java
> > > > version "1.6.0_38"
> > > > , ubuntu precise 12.04 - kernel 3.2. The libc version from the above
> > line
> > > > seems to be 2.15.
> > > >
> > > > I wonder if people have seen this in the past and if yes, were they
> > able
> > > to
> > > > resolve it and if yes, then how ? Was it a VM upgrade, a kernel
> upgrade
> > > or
> > > > a libc upgrade ?
> > > >
> > > > Thanks
> > > > Varun
> > > >
> > >
> > >
> > >
> > > --
> > > Best regards,
> > >
> > > - Andy
> > >
> > > Problems worthy of attack prove their worth by hitting back. - Piet
> Hein
> > > (via Tom White)
> > >
> >
>
>
>
> --
> Best regards,
>
> - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
>
Re: JVM seg fault in HBase region server
Posted by Varun Sharma <va...@pinterest.com>.
Did you have the jvm error logging enabled -XX:ErrorLog or something and if
yes, did that spew anything out ?
Thanks
Varun
On Sun, May 5, 2013 at 10:18 PM, tsuna <ts...@gmail.com> wrote:
> On Thu, May 2, 2013 at 1:49 PM, Andrew Purtell <ap...@apache.org>
> wrote:
> > In that blog post Benoît does a fair amount of showing off to end up with
> > "So this segfault remains a mystery so far" :-)
>
> Yeah I never got to the bottom of it. :-/
>
> --
> Benoit "tsuna" Sigoure
>
Re: JVM seg fault in HBase region server
Posted by tsuna <ts...@gmail.com>.
On Thu, May 2, 2013 at 1:49 PM, Andrew Purtell <ap...@apache.org> wrote:
> In that blog post Benoît does a fair amount of showing off to end up with
> "So this segfault remains a mystery so far" :-)
Yeah I never got to the bottom of it. :-/
--
Benoit "tsuna" Sigoure
Re: JVM seg fault in HBase region server
Posted by Andrew Purtell <ap...@apache.org>.
In that blog post Benoît does a fair amount of showing off to end up with
"So this segfault remains a mystery so far" :-)
My take is don't suspect the JVM, suspect your libc. I am not a fan of
using Ubuntu on the server side. Perhaps you can try a distribution based
on a different kernel+glibc? (RHEL comes to mind.) I realize that could
be... inconvenient.
On Thu, May 2, 2013 at 1:39 PM, Varun Sharma <va...@pinterest.com> wrote:
> I don't have one unfortunately - We did not have the -XX:ErrorLog turned on
> :(
>
> But I did some digging following what Benoit wrote in his Blog. Basically
> the segfault happens in the same place inside a clearerr() function in
> glibc which accepts a file descriptor.
>
> The JVM version is sun java 1.6.0u38. Is this possibly too old ?
>
> Thanks
> Varun
>
>
> On Thu, May 2, 2013 at 12:08 PM, Andrew Purtell <ap...@apache.org>
> wrote:
>
> > Can you pastebin or post somewhere the entire hs_err* file?
> >
> >
> > On Wed, May 1, 2013 at 1:54 PM, Varun Sharma <va...@pinterest.com>
> wrote:
> >
> > > Hi,
> > >
> > > I am seeing the following which is a JVM segfault:
> > >
> > > hbase-regionser[28734]: segfault at 8 ip 00007f269bcc307e sp
> > > 00007fff50f7e638 error 4 in libc-2.15.so[7f269bc51000+1b5000]
> > >
> > > Benoit Tsuna reported a similar issue a while back -
> > >
> > >
> >
> http://blog.tsunanet.net/2011/05/jvm-u24-segfault-in-clearerr-on-jaunty.html
> > >
> > > I wonder if there was ever a resolution on the issue. We are running
> java
> > > version "1.6.0_38"
> > > , ubuntu precise 12.04 - kernel 3.2. The libc version from the above
> line
> > > seems to be 2.15.
> > >
> > > I wonder if people have seen this in the past and if yes, were they
> able
> > to
> > > resolve it and if yes, then how ? Was it a VM upgrade, a kernel upgrade
> > or
> > > a libc upgrade ?
> > >
> > > Thanks
> > > Varun
> > >
> >
> >
> >
> > --
> > Best regards,
> >
> > - Andy
> >
> > Problems worthy of attack prove their worth by hitting back. - Piet Hein
> > (via Tom White)
> >
>
--
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)
Re: JVM seg fault in HBase region server
Posted by Ted Yu <yu...@gmail.com>.
Maybe try java 1.6.0_43 ?
Cheers
On Thu, May 2, 2013 at 1:39 PM, Varun Sharma <va...@pinterest.com> wrote:
> I don't have one unfortunately - We did not have the -XX:ErrorLog turned on
> :(
>
> But I did some digging following what Benoit wrote in his Blog. Basically
> the segfault happens in the same place inside a clearerr() function in
> glibc which accepts a file descriptor.
>
> The JVM version is sun java 1.6.0u38. Is this possibly too old ?
>
> Thanks
> Varun
>
>
> On Thu, May 2, 2013 at 12:08 PM, Andrew Purtell <ap...@apache.org>
> wrote:
>
> > Can you pastebin or post somewhere the entire hs_err* file?
> >
> >
> > On Wed, May 1, 2013 at 1:54 PM, Varun Sharma <va...@pinterest.com>
> wrote:
> >
> > > Hi,
> > >
> > > I am seeing the following which is a JVM segfault:
> > >
> > > hbase-regionser[28734]: segfault at 8 ip 00007f269bcc307e sp
> > > 00007fff50f7e638 error 4 in libc-2.15.so[7f269bc51000+1b5000]
> > >
> > > Benoit Tsuna reported a similar issue a while back -
> > >
> > >
> >
> http://blog.tsunanet.net/2011/05/jvm-u24-segfault-in-clearerr-on-jaunty.html
> > >
> > > I wonder if there was ever a resolution on the issue. We are running
> java
> > > version "1.6.0_38"
> > > , ubuntu precise 12.04 - kernel 3.2. The libc version from the above
> line
> > > seems to be 2.15.
> > >
> > > I wonder if people have seen this in the past and if yes, were they
> able
> > to
> > > resolve it and if yes, then how ? Was it a VM upgrade, a kernel upgrade
> > or
> > > a libc upgrade ?
> > >
> > > Thanks
> > > Varun
> > >
> >
> >
> >
> > --
> > Best regards,
> >
> > - Andy
> >
> > Problems worthy of attack prove their worth by hitting back. - Piet Hein
> > (via Tom White)
> >
>
Re: JVM seg fault in HBase region server
Posted by Varun Sharma <va...@pinterest.com>.
I don't have one unfortunately - We did not have the -XX:ErrorLog turned on
:(
But I did some digging following what Benoit wrote in his Blog. Basically
the segfault happens in the same place inside a clearerr() function in
glibc which accepts a file descriptor.
The JVM version is sun java 1.6.0u38. Is this possibly too old ?
Thanks
Varun
On Thu, May 2, 2013 at 12:08 PM, Andrew Purtell <ap...@apache.org> wrote:
> Can you pastebin or post somewhere the entire hs_err* file?
>
>
> On Wed, May 1, 2013 at 1:54 PM, Varun Sharma <va...@pinterest.com> wrote:
>
> > Hi,
> >
> > I am seeing the following which is a JVM segfault:
> >
> > hbase-regionser[28734]: segfault at 8 ip 00007f269bcc307e sp
> > 00007fff50f7e638 error 4 in libc-2.15.so[7f269bc51000+1b5000]
> >
> > Benoit Tsuna reported a similar issue a while back -
> >
> >
> http://blog.tsunanet.net/2011/05/jvm-u24-segfault-in-clearerr-on-jaunty.html
> >
> > I wonder if there was ever a resolution on the issue. We are running java
> > version "1.6.0_38"
> > , ubuntu precise 12.04 - kernel 3.2. The libc version from the above line
> > seems to be 2.15.
> >
> > I wonder if people have seen this in the past and if yes, were they able
> to
> > resolve it and if yes, then how ? Was it a VM upgrade, a kernel upgrade
> or
> > a libc upgrade ?
> >
> > Thanks
> > Varun
> >
>
>
>
> --
> Best regards,
>
> - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
>
Re: JVM seg fault in HBase region server
Posted by Andrew Purtell <ap...@apache.org>.
Can you pastebin or post somewhere the entire hs_err* file?
On Wed, May 1, 2013 at 1:54 PM, Varun Sharma <va...@pinterest.com> wrote:
> Hi,
>
> I am seeing the following which is a JVM segfault:
>
> hbase-regionser[28734]: segfault at 8 ip 00007f269bcc307e sp
> 00007fff50f7e638 error 4 in libc-2.15.so[7f269bc51000+1b5000]
>
> Benoit Tsuna reported a similar issue a while back -
>
> http://blog.tsunanet.net/2011/05/jvm-u24-segfault-in-clearerr-on-jaunty.html
>
> I wonder if there was ever a resolution on the issue. We are running java
> version "1.6.0_38"
> , ubuntu precise 12.04 - kernel 3.2. The libc version from the above line
> seems to be 2.15.
>
> I wonder if people have seen this in the past and if yes, were they able to
> resolve it and if yes, then how ? Was it a VM upgrade, a kernel upgrade or
> a libc upgrade ?
>
> Thanks
> Varun
>
--
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)