You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Jean-Marc Spaggiari <je...@spaggiari.org> on 2013/12/26 15:00:36 UTC
Master (should not?) abort startup on Unexpected PENDING_OPEN state
I think I stopped my master while it was doing a big balancing. At restart,
I'm getting the exception below and master exit. All RS are able to start
correctly, but not the master.
Since master is not starting I can not manually assign this region from the
shell. I guess I can simply delete the znode about the region, restart and
hbck, but my opinion is that we should not abord the startup when such
exception occurs.
JM
java.lang.IllegalStateException: Unexpected state :
page,www\x1Fhttp\x1F-1\x1F/vote/comment/27996/1/\x1Fnull,1379104524006.17bee313797fc1ce982c0e31fdb6620c.
state=PENDING_OPEN, ts=1388065670415, server=node6,60020,1388027343261 ..
Cannot transit it to OFFLINE.
at
org.apache.hadoop.hbase.master.AssignmentManager.setOfflineInZooKeeper(AssignmentManager.java:1890)
at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1690)
at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1426)
at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1398)
at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1393)
at
org.apache.hadoop.hbase.master.handler.ClosedRegionHandler.process(ClosedRegionHandler.java:105)
at
org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:175)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Re: Master (should not?) abort startup on Unexpected PENDING_OPEN state
Posted by Jean-Marc Spaggiari <je...@spaggiari.org>.
Faced this with 0.94. Vladimir point that a JIRA was already opened for
that under HBASE-8912.
2013/12/26 ramkrishna vasudevan <ra...@gmail.com>
> Is this in the 0.94 version or in the 0.96 version?
>
>
> On Fri, Dec 27, 2013 at 3:07 AM, lars hofhansl <la...@apache.org> wrote:
>
>> Yeah, sounds like a but. Mind filing a jira?
>>
>>
>>
>> ________________________________
>> From: Jean-Marc Spaggiari <je...@spaggiari.org>
>> To: user <us...@hbase.apache.org>; lars hofhansl <la...@apache.org>
>> Sent: Thursday, December 26, 2013 12:05 PM
>> Subject: Re: Master (should not?) abort startup on Unexpected
>> PENDING_OPEN state
>>
>>
>>
>> It was aborting each time I was trying. I tried at least 10 times. Failed
>> 10 times. I have deleted the znodes and restarted and it started correctly.
>>
>> I might be able to reproduce the situation.
>>
>>
>>
>>
>> 2013/12/26 lars hofhansl <la...@apache.org>
>>
>> When you start the master again, does it abort again?
>> >
>> >
>> >
>> >________________________________
>> > From: Jean-Marc Spaggiari <je...@spaggiari.org>
>> >To: user <us...@hbase.apache.org>
>> >Sent: Thursday, December 26, 2013 6:00 AM
>> >Subject: Master (should not?) abort startup on Unexpected PENDING_OPEN
>> state
>> >
>> >
>> >
>> >I think I stopped my master while it was doing a big balancing. At
>> restart,
>> >I'm getting the exception below and master exit. All RS are able to start
>> >correctly, but not the master.
>> >
>> >Since master is not starting I can not manually assign this region from
>> the
>> >shell. I guess I can simply delete the znode about the region, restart
>> and
>> >hbck, but my opinion is that we should not abord the startup when such
>> >exception occurs.
>> >
>> >JM
>> >
>> >
>> >
>> >java.lang.IllegalStateException: Unexpected state :
>>
>> >page,www\x1Fhttp\x1F-1\x1F/vote/comment/27996/1/\x1Fnull,1379104524006.17bee313797fc1ce982c0e31fdb6620c.
>> >state=PENDING_OPEN, ts=1388065670415, server=node6,60020,1388027343261 ..
>> >Cannot transit it to OFFLINE.
>> > at
>>
>> >org.apache.hadoop.hbase.master.AssignmentManager.setOfflineInZooKeeper(AssignmentManager.java:1890)
>> > at
>>
>> >org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1690)
>> > at
>>
>> >org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1426)
>> > at
>>
>> >org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1398)
>> > at
>>
>> >org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1393)
>> > at
>>
>> >org.apache.hadoop.hbase.master.handler.ClosedRegionHandler.process(ClosedRegionHandler.java:105)
>> > at
>> >org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:175)
>> > at
>>
>> >java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> > at
>>
>> >java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> > at java.lang.Thread.run(Thread.java:744)
>>
>
>
Re: Master (should not?) abort startup on Unexpected PENDING_OPEN state
Posted by ramkrishna vasudevan <ra...@gmail.com>.
Is this in the 0.94 version or in the 0.96 version?
On Fri, Dec 27, 2013 at 3:07 AM, lars hofhansl <la...@apache.org> wrote:
> Yeah, sounds like a but. Mind filing a jira?
>
>
>
> ________________________________
> From: Jean-Marc Spaggiari <je...@spaggiari.org>
> To: user <us...@hbase.apache.org>; lars hofhansl <la...@apache.org>
> Sent: Thursday, December 26, 2013 12:05 PM
> Subject: Re: Master (should not?) abort startup on Unexpected PENDING_OPEN
> state
>
>
>
> It was aborting each time I was trying. I tried at least 10 times. Failed
> 10 times. I have deleted the znodes and restarted and it started correctly.
>
> I might be able to reproduce the situation.
>
>
>
>
> 2013/12/26 lars hofhansl <la...@apache.org>
>
> When you start the master again, does it abort again?
> >
> >
> >
> >________________________________
> > From: Jean-Marc Spaggiari <je...@spaggiari.org>
> >To: user <us...@hbase.apache.org>
> >Sent: Thursday, December 26, 2013 6:00 AM
> >Subject: Master (should not?) abort startup on Unexpected PENDING_OPEN
> state
> >
> >
> >
> >I think I stopped my master while it was doing a big balancing. At
> restart,
> >I'm getting the exception below and master exit. All RS are able to start
> >correctly, but not the master.
> >
> >Since master is not starting I can not manually assign this region from
> the
> >shell. I guess I can simply delete the znode about the region, restart and
> >hbck, but my opinion is that we should not abord the startup when such
> >exception occurs.
> >
> >JM
> >
> >
> >
> >java.lang.IllegalStateException: Unexpected state :
>
> >page,www\x1Fhttp\x1F-1\x1F/vote/comment/27996/1/\x1Fnull,1379104524006.17bee313797fc1ce982c0e31fdb6620c.
> >state=PENDING_OPEN, ts=1388065670415, server=node6,60020,1388027343261 ..
> >Cannot transit it to OFFLINE.
> > at
>
> >org.apache.hadoop.hbase.master.AssignmentManager.setOfflineInZooKeeper(AssignmentManager.java:1890)
> > at
>
> >org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1690)
> > at
>
> >org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1426)
> > at
>
> >org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1398)
> > at
>
> >org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1393)
> > at
>
> >org.apache.hadoop.hbase.master.handler.ClosedRegionHandler.process(ClosedRegionHandler.java:105)
> > at
> >org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:175)
> > at
>
> >java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> > at
>
> >java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> > at java.lang.Thread.run(Thread.java:744)
>
Re: Master (should not?) abort startup on Unexpected PENDING_OPEN state
Posted by lars hofhansl <la...@apache.org>.
Yeah, sounds like a but. Mind filing a jira?
________________________________
From: Jean-Marc Spaggiari <je...@spaggiari.org>
To: user <us...@hbase.apache.org>; lars hofhansl <la...@apache.org>
Sent: Thursday, December 26, 2013 12:05 PM
Subject: Re: Master (should not?) abort startup on Unexpected PENDING_OPEN state
It was aborting each time I was trying. I tried at least 10 times. Failed 10 times. I have deleted the znodes and restarted and it started correctly.
I might be able to reproduce the situation.
2013/12/26 lars hofhansl <la...@apache.org>
When you start the master again, does it abort again?
>
>
>
>________________________________
> From: Jean-Marc Spaggiari <je...@spaggiari.org>
>To: user <us...@hbase.apache.org>
>Sent: Thursday, December 26, 2013 6:00 AM
>Subject: Master (should not?) abort startup on Unexpected PENDING_OPEN state
>
>
>
>I think I stopped my master while it was doing a big balancing. At restart,
>I'm getting the exception below and master exit. All RS are able to start
>correctly, but not the master.
>
>Since master is not starting I can not manually assign this region from the
>shell. I guess I can simply delete the znode about the region, restart and
>hbck, but my opinion is that we should not abord the startup when such
>exception occurs.
>
>JM
>
>
>
>java.lang.IllegalStateException: Unexpected state :
>page,www\x1Fhttp\x1F-1\x1F/vote/comment/27996/1/\x1Fnull,1379104524006.17bee313797fc1ce982c0e31fdb6620c.
>state=PENDING_OPEN, ts=1388065670415, server=node6,60020,1388027343261 ..
>Cannot transit it to OFFLINE.
> at
>org.apache.hadoop.hbase.master.AssignmentManager.setOfflineInZooKeeper(AssignmentManager.java:1890)
> at
>org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1690)
> at
>org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1426)
> at
>org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1398)
> at
>org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1393)
> at
>org.apache.hadoop.hbase.master.handler.ClosedRegionHandler.process(ClosedRegionHandler.java:105)
> at
>org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:175)
> at
>java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
>java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
Re: Master (should not?) abort startup on Unexpected PENDING_OPEN state
Posted by Jean-Marc Spaggiari <je...@spaggiari.org>.
It was aborting each time I was trying. I tried at least 10 times. Failed
10 times. I have deleted the znodes and restarted and it started correctly.
I might be able to reproduce the situation.
2013/12/26 lars hofhansl <la...@apache.org>
> When you start the master again, does it abort again?
>
>
>
> ________________________________
> From: Jean-Marc Spaggiari <je...@spaggiari.org>
> To: user <us...@hbase.apache.org>
> Sent: Thursday, December 26, 2013 6:00 AM
> Subject: Master (should not?) abort startup on Unexpected PENDING_OPEN
> state
>
>
> I think I stopped my master while it was doing a big balancing. At restart,
> I'm getting the exception below and master exit. All RS are able to start
> correctly, but not the master.
>
> Since master is not starting I can not manually assign this region from the
> shell. I guess I can simply delete the znode about the region, restart and
> hbck, but my opinion is that we should not abord the startup when such
> exception occurs.
>
> JM
>
>
>
> java.lang.IllegalStateException: Unexpected state :
>
> page,www\x1Fhttp\x1F-1\x1F/vote/comment/27996/1/\x1Fnull,1379104524006.17bee313797fc1ce982c0e31fdb6620c.
> state=PENDING_OPEN, ts=1388065670415, server=node6,60020,1388027343261 ..
> Cannot transit it to OFFLINE.
> at
>
> org.apache.hadoop.hbase.master.AssignmentManager.setOfflineInZooKeeper(AssignmentManager.java:1890)
> at
>
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1690)
> at
>
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1426)
> at
>
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1398)
> at
>
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1393)
> at
>
> org.apache.hadoop.hbase.master.handler.ClosedRegionHandler.process(ClosedRegionHandler.java:105)
> at
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:175)
> at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
>
Re: Master (should not?) abort startup on Unexpected PENDING_OPEN state
Posted by lars hofhansl <la...@apache.org>.
When you start the master again, does it abort again?
________________________________
From: Jean-Marc Spaggiari <je...@spaggiari.org>
To: user <us...@hbase.apache.org>
Sent: Thursday, December 26, 2013 6:00 AM
Subject: Master (should not?) abort startup on Unexpected PENDING_OPEN state
I think I stopped my master while it was doing a big balancing. At restart,
I'm getting the exception below and master exit. All RS are able to start
correctly, but not the master.
Since master is not starting I can not manually assign this region from the
shell. I guess I can simply delete the znode about the region, restart and
hbck, but my opinion is that we should not abord the startup when such
exception occurs.
JM
java.lang.IllegalStateException: Unexpected state :
page,www\x1Fhttp\x1F-1\x1F/vote/comment/27996/1/\x1Fnull,1379104524006.17bee313797fc1ce982c0e31fdb6620c.
state=PENDING_OPEN, ts=1388065670415, server=node6,60020,1388027343261 ..
Cannot transit it to OFFLINE.
at
org.apache.hadoop.hbase.master.AssignmentManager.setOfflineInZooKeeper(AssignmentManager.java:1890)
at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1690)
at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1426)
at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1398)
at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1393)
at
org.apache.hadoop.hbase.master.handler.ClosedRegionHandler.process(ClosedRegionHandler.java:105)
at
org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:175)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)