You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Bryan Beaudreault <bb...@hubspot.com> on 2013/03/04 18:00:43 UTC
0.94.2 failing regionservers
We recently upgraded multiple clusters to CDH 4.2, which comes with hbase
0.94.2 Since then We've seen region servers die periodically in a way I
never saw before on CDH3.
Here are the exceptions:
First I see a slew of these: http://pastebin.com/WqSwMzuZ
Then the regionserver starts closing all its regions, after throwing this
exception: http://pastebin.com/396wX3iw
This has now happened on multiple servers across multiple clusters, all cdh
4.2.
Any thoughts? During the time our NameNodes seem to be doing fine and I
don't see any issue on our datanodes either.
Re: 0.94.2 failing regionservers
Posted by Bryan Beaudreault <bb...@hubspot.com>.
We upgraded from CDH3 which is HBase 0.90. Looking around we do have some
v2 format HFiles, just not entirely there yet. I'll try to get them all
converted.
On Mon, Mar 4, 2013 at 12:23 PM, Ted Yu <yu...@gmail.com> wrote:
> In hindsight, this should have helped:
>
> http://hbase.apache.org/book.html#upgrade0.94
>
> I would suggest upgrading existing HFile v1 files to v2 format.
>
> Cheers
>
> On Mon, Mar 4, 2013 at 9:20 AM, Bryan Beaudreault
> <bb...@hubspot.com>wrote:
>
> > Interesting.
> >
> > We upgraded by creating entirely new clusters with the CDH4.2 software
> > installed onto it. Then we distributed copied the /hbase directory to the
> > new clusters. We haven't run a major compaction since then, so that
> could
> > be the reason there are still v1 HFiles. This migration took place just
> a
> > couple days ago.
> >
> >
> > On Mon, Mar 4, 2013 at 12:04 PM, Ted Yu <yu...@gmail.com> wrote:
> >
> > > In the stack trace, I see HFileReaderV1.java
> > > I would expect your store files to be upgraded to HFile v2 after
> cluster
> > > upgrade.
> > >
> > > Can you tell us more about how you upgraded your clusters ?
> > >
> > > Thanks
> > >
> > > On Mon, Mar 4, 2013 at 9:00 AM, Bryan Beaudreault
> > > <bb...@hubspot.com>wrote:
> > >
> > > > We recently upgraded multiple clusters to CDH 4.2, which comes with
> > hbase
> > > > 0.94.2 Since then We've seen region servers die periodically in a
> way
> > I
> > > > never saw before on CDH3.
> > > >
> > > > Here are the exceptions:
> > > >
> > > > First I see a slew of these: http://pastebin.com/WqSwMzuZ
> > > >
> > > > Then the regionserver starts closing all its regions, after throwing
> > this
> > > > exception: http://pastebin.com/396wX3iw
> > > >
> > > > This has now happened on multiple servers across multiple clusters,
> all
> > > cdh
> > > > 4.2.
> > > >
> > > > Any thoughts? During the time our NameNodes seem to be doing fine
> and
> > I
> > > > don't see any issue on our datanodes either.
> > > >
> > >
> >
>
Re: 0.94.2 failing regionservers
Posted by Ted Yu <yu...@gmail.com>.
In hindsight, this should have helped:
http://hbase.apache.org/book.html#upgrade0.94
I would suggest upgrading existing HFile v1 files to v2 format.
Cheers
On Mon, Mar 4, 2013 at 9:20 AM, Bryan Beaudreault
<bb...@hubspot.com>wrote:
> Interesting.
>
> We upgraded by creating entirely new clusters with the CDH4.2 software
> installed onto it. Then we distributed copied the /hbase directory to the
> new clusters. We haven't run a major compaction since then, so that could
> be the reason there are still v1 HFiles. This migration took place just a
> couple days ago.
>
>
> On Mon, Mar 4, 2013 at 12:04 PM, Ted Yu <yu...@gmail.com> wrote:
>
> > In the stack trace, I see HFileReaderV1.java
> > I would expect your store files to be upgraded to HFile v2 after cluster
> > upgrade.
> >
> > Can you tell us more about how you upgraded your clusters ?
> >
> > Thanks
> >
> > On Mon, Mar 4, 2013 at 9:00 AM, Bryan Beaudreault
> > <bb...@hubspot.com>wrote:
> >
> > > We recently upgraded multiple clusters to CDH 4.2, which comes with
> hbase
> > > 0.94.2 Since then We've seen region servers die periodically in a way
> I
> > > never saw before on CDH3.
> > >
> > > Here are the exceptions:
> > >
> > > First I see a slew of these: http://pastebin.com/WqSwMzuZ
> > >
> > > Then the regionserver starts closing all its regions, after throwing
> this
> > > exception: http://pastebin.com/396wX3iw
> > >
> > > This has now happened on multiple servers across multiple clusters, all
> > cdh
> > > 4.2.
> > >
> > > Any thoughts? During the time our NameNodes seem to be doing fine and
> I
> > > don't see any issue on our datanodes either.
> > >
> >
>
Re: 0.94.2 failing regionservers
Posted by Bryan Beaudreault <bb...@hubspot.com>.
Interesting.
We upgraded by creating entirely new clusters with the CDH4.2 software
installed onto it. Then we distributed copied the /hbase directory to the
new clusters. We haven't run a major compaction since then, so that could
be the reason there are still v1 HFiles. This migration took place just a
couple days ago.
On Mon, Mar 4, 2013 at 12:04 PM, Ted Yu <yu...@gmail.com> wrote:
> In the stack trace, I see HFileReaderV1.java
> I would expect your store files to be upgraded to HFile v2 after cluster
> upgrade.
>
> Can you tell us more about how you upgraded your clusters ?
>
> Thanks
>
> On Mon, Mar 4, 2013 at 9:00 AM, Bryan Beaudreault
> <bb...@hubspot.com>wrote:
>
> > We recently upgraded multiple clusters to CDH 4.2, which comes with hbase
> > 0.94.2 Since then We've seen region servers die periodically in a way I
> > never saw before on CDH3.
> >
> > Here are the exceptions:
> >
> > First I see a slew of these: http://pastebin.com/WqSwMzuZ
> >
> > Then the regionserver starts closing all its regions, after throwing this
> > exception: http://pastebin.com/396wX3iw
> >
> > This has now happened on multiple servers across multiple clusters, all
> cdh
> > 4.2.
> >
> > Any thoughts? During the time our NameNodes seem to be doing fine and I
> > don't see any issue on our datanodes either.
> >
>
Re: 0.94.2 failing regionservers
Posted by Ted Yu <yu...@gmail.com>.
In the stack trace, I see HFileReaderV1.java
I would expect your store files to be upgraded to HFile v2 after cluster
upgrade.
Can you tell us more about how you upgraded your clusters ?
Thanks
On Mon, Mar 4, 2013 at 9:00 AM, Bryan Beaudreault
<bb...@hubspot.com>wrote:
> We recently upgraded multiple clusters to CDH 4.2, which comes with hbase
> 0.94.2 Since then We've seen region servers die periodically in a way I
> never saw before on CDH3.
>
> Here are the exceptions:
>
> First I see a slew of these: http://pastebin.com/WqSwMzuZ
>
> Then the regionserver starts closing all its regions, after throwing this
> exception: http://pastebin.com/396wX3iw
>
> This has now happened on multiple servers across multiple clusters, all cdh
> 4.2.
>
> Any thoughts? During the time our NameNodes seem to be doing fine and I
> don't see any issue on our datanodes either.
>
Re: 0.94.2 failing regionservers
Posted by Ted Yu <yu...@gmail.com>.
I logged HBASE-7991 for backport.
Cheers
On Mon, Mar 4, 2013 at 9:41 AM, Jean-Daniel Cryans <jd...@apache.org>wrote:
> This looks a lot like:
>
> https://issues.apache.org/jira/browse/HBASE-6479
>
> FWIW running with V1 files is ok (unless you hit this bug obviously).
>
> J-D
>
> On Mon, Mar 4, 2013 at 9:00 AM, Bryan Beaudreault
> <bb...@hubspot.com> wrote:
> > We recently upgraded multiple clusters to CDH 4.2, which comes with hbase
> > 0.94.2 Since then We've seen region servers die periodically in a way I
> > never saw before on CDH3.
> >
> > Here are the exceptions:
> >
> > First I see a slew of these: http://pastebin.com/WqSwMzuZ
> >
> > Then the regionserver starts closing all its regions, after throwing this
> > exception: http://pastebin.com/396wX3iw
> >
> > This has now happened on multiple servers across multiple clusters, all
> cdh
> > 4.2.
> >
> > Any thoughts? During the time our NameNodes seem to be doing fine and I
> > don't see any issue on our datanodes either.
>
Re: 0.94.2 failing regionservers
Posted by Bryan Beaudreault <bb...@hubspot.com>.
Interesting, thanks for pointing that out JD. I'm running major
compactions now on the clusters to try to get everything over to V2.
On Mon, Mar 4, 2013 at 12:41 PM, Jean-Daniel Cryans <jd...@apache.org>wrote:
> This looks a lot like:
>
> https://issues.apache.org/jira/browse/HBASE-6479
>
> FWIW running with V1 files is ok (unless you hit this bug obviously).
>
> J-D
>
> On Mon, Mar 4, 2013 at 9:00 AM, Bryan Beaudreault
> <bb...@hubspot.com> wrote:
> > We recently upgraded multiple clusters to CDH 4.2, which comes with hbase
> > 0.94.2 Since then We've seen region servers die periodically in a way I
> > never saw before on CDH3.
> >
> > Here are the exceptions:
> >
> > First I see a slew of these: http://pastebin.com/WqSwMzuZ
> >
> > Then the regionserver starts closing all its regions, after throwing this
> > exception: http://pastebin.com/396wX3iw
> >
> > This has now happened on multiple servers across multiple clusters, all
> cdh
> > 4.2.
> >
> > Any thoughts? During the time our NameNodes seem to be doing fine and I
> > don't see any issue on our datanodes either.
>
Re: 0.94.2 failing regionservers
Posted by Jean-Daniel Cryans <jd...@apache.org>.
This looks a lot like:
https://issues.apache.org/jira/browse/HBASE-6479
FWIW running with V1 files is ok (unless you hit this bug obviously).
J-D
On Mon, Mar 4, 2013 at 9:00 AM, Bryan Beaudreault
<bb...@hubspot.com> wrote:
> We recently upgraded multiple clusters to CDH 4.2, which comes with hbase
> 0.94.2 Since then We've seen region servers die periodically in a way I
> never saw before on CDH3.
>
> Here are the exceptions:
>
> First I see a slew of these: http://pastebin.com/WqSwMzuZ
>
> Then the regionserver starts closing all its regions, after throwing this
> exception: http://pastebin.com/396wX3iw
>
> This has now happened on multiple servers across multiple clusters, all cdh
> 4.2.
>
> Any thoughts? During the time our NameNodes seem to be doing fine and I
> don't see any issue on our datanodes either.