You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Jonathan Bender <jo...@gmail.com> on 2011/04/15 05:23:45 UTC
Re: HBase crashes on table creation
Just to provide a bit more context: here is the log for Zookeeper. I am
getting a "NoNode" exception after creating the table in the shell.
http://pastebin.com/0bUCYkXb
<http://pastebin.com/0bUCYkXb>Cheers,
Jon
On Thu, Apr 14, 2011 at 5:50 PM, Jonathan Bender
<jo...@gmail.com>wrote:
> Thanks Matt. I added the HBase libs and I could create tables.
>
> However...now I get an error when trying to create a table through the
> shell (either compressed or not). It can't seem to find a regionserver that
> will assign it, and then ZK loses connection to HBase
>
> hbase(main):011:0> create 'test_t', 'data'
>
> ERROR: java.io.IOException: java.io.IOException:
> java.lang.InterruptedException
>
> HBase Master log: http://pastebin.com/xbsVQvVn
>
>
> Anyone seen this before? I'm at a bit of a loss here, as I don't see how
> the LZO stuff I added relates to this.
>
> On Thu, Apr 14, 2011 at 3:33 PM, Matt Davies <ma...@tynt.com> wrote:
>
>> I'd suspect you will find a region server without LZO enabled. I actually
>> saw this today. It seems to create in the shell but never completes.
>>
>> I'd do (in hbase shell) status 'detailed' and see if there are any dead
>> servers. Then take a look at the hbase logs. I suspect you'll see a stack
>> trace about can't find LZO in the classpath.
>>
>> Do you have any other tables successfully enabled with LZO?
>>
>> BTW - we fixed this by putting lzo in the hbase/lib directory and
>> restarting. To make things easy we just dropped and recreated the table.
>>
>> HTH
>>
>> -Matt
>>
>> On Thu, Apr 14, 2011 at 4:28 PM, Jonathan Bender <
>> jonathan.bender@gmail.com> wrote:
>>
>>> Hello,
>>>
>>> I'm having an issue with a table I created earlier, testing out LZO
>>> compression. For some reason I can't disable/drop the table, since it is
>>> constantly in an OPENING state and never gets assigned to a region. Does
>>> anyone have experience with this, and a possible way to disable/manage
>>> this
>>> table?
>>>
>>> Thanks!
>>> Jon
>>>
>>> Log from the HBase master:
>>> http://pastebin.com/i00uXHJb
>>>
>>
>>
>
Re: HBase crashes on table creation
Posted by Stack <st...@duboce.net>.
Thanks for updating the list Jonathan.
St.Ack
On Fri, Apr 15, 2011 at 1:46 PM, Jonathan Bender
<jo...@gmail.com> wrote:
> Hey all,
> I figured out my issue finally. Here's a brief summary:
> ISSUE: NoNode exception when creating a table through the shell - HBase
> Master crashes.
> hbase(main):011:0> create 'test_t', 'data'
> ERROR: java.io.IOException: java.io.IOException:
> java.lang.InterruptedException
> From ZK Server Log:
> 2011-04-14 17:37:00,907 - INFO [ProcessThread:-1:PrepRequestProcessor@407]
> - Got user-level KeeperException when processing sessionid:0x12f569514a30000
> type:delete cxid:0xc6 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a
> Error Path:/hbase/table/test_t Error:KeeperErrorCode = NoNode for
> /hbase/table/test_t
> Master server starts assigning regions in round robin fashion, but fails to
> assign to certain nodes. From master server log:
> 2011-04-14 17:37:00,908 WARN org.apache.hadoop.hbase.zookeeper.ZKTable:
> Moving table test_t state to enabled but was already enabled
> 2011-04-14 17:37:01,404 FATAL org.apache.hadoop.hbase.master.HMaster: Failed
> assignment of regions to serverName=sv006.my.domain.com,60020,1302827775531
> CAUSE: In this case the master server had a fully specified hostname for
> each regionserver node. However, after adding a few new regionserver nodes,
> I left out the hostname definitions in the regionservers /etc/hosts file.
> In other words, the regionserver hosts file only had:
> 127.0.0.1 localhost.localdomain localhost
> I added fully specified hostnames in each region server's hosts file, which
> did the trick:
> 127.0.0.1 localhost.localdomain localhost
> <region.server.ip> my.region.server.domain region1
> <master.server.ip> my.master.server.domain master
> Cheers,
> Jon
> On Fri, Apr 15, 2011 at 11:22 AM, Stack <sa...@gmail.com> wrote:
>>
>> Good. Mind updating list so others benefit from your findings? I will add
>> an FAQ on proxy based off your list mail. Thanks.
>>
>>
>>
>> On Apr 15, 2011, at 11:14, Jonathan Bender <jo...@gmail.com>
>> wrote:
>>
>> > Just figured it out. I recently added some new Regionservers which had
>> > their Hostnames configured improperly. I just added every hostname / IP to
>> > the hosts file on each node to make it work again.
>> >
>> > Turns out it had nothing to do with LZO stuff, which was throwing me off
>> > big time.
>> >
>> > Many thanks for the help.
>
>
Re: HBase crashes on table creation
Posted by Jonathan Bender <jo...@gmail.com>.
Hey all,
I figured out my issue finally. Here's a brief summary:
ISSUE: NoNode exception when creating a table through the shell - HBase
Master crashes.
hbase(main):011:0> create 'test_t', 'data'
ERROR: java.io.IOException: java.io.IOException:
java.lang.InterruptedException
>From ZK Server Log:
2011-04-14 17:37:00,907 - INFO [ProcessThread:-1:PrepRequestProcessor@407]
- Got user-level KeeperException when processing sessionid:0x12f569514a30000
type:delete cxid:0xc6 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a
Error Path:/hbase/table/test_t Error:KeeperErrorCode = NoNode for
/hbase/table/test_t
Master server starts assigning regions in round robin fashion, but fails to
assign to certain nodes. From master server log:
2011-04-14 17:37:00,908 WARN org.apache.hadoop.hbase.zookeeper.ZKTable:
Moving table test_t state to enabled but was already enabled
2011-04-14 17:37:01,404 FATAL org.apache.hadoop.hbase.master.HMaster: Failed
assignment of regions to serverName=sv006.my.domain.com,60020,1302827775531
CAUSE: In this case the master server had a fully specified hostname for
each regionserver node. However, after adding a few new regionserver nodes,
I left out the hostname definitions in the regionservers /etc/hosts file.
In other words, the regionserver hosts file only had:
127.0.0.1 localhost.localdomain localhost
I added fully specified hostnames in each region server's hosts file, which
did the trick:
127.0.0.1 localhost.localdomain localhost
<region.server.ip> my.region.server.domain region1
<master.server.ip> my.master.server.domain master
Cheers,
Jon
On Fri, Apr 15, 2011 at 11:22 AM, Stack <sa...@gmail.com> wrote:
> Good. Mind updating list so others benefit from your findings? I will add
> an FAQ on proxy based off your list mail. Thanks.
>
>
>
> On Apr 15, 2011, at 11:14, Jonathan Bender <jo...@gmail.com>
> wrote:
>
> > Just figured it out. I recently added some new Regionservers which had
> their Hostnames configured improperly. I just added every hostname / IP to
> the hosts file on each node to make it work again.
> >
> > Turns out it had nothing to do with LZO stuff, which was throwing me off
> big time.
> >
> > Many thanks for the help.
>
Re: HBase crashes on table creation
Posted by Jonathan Bender <jo...@gmail.com>.
I was running the shell from the master node. Once I create the table it
just gives me that InterruptedException and that's the extent of it. The
master crashes and I have to start HBase up again.
As for the lzo CLASSPATH stuff--I only built the lzo libraries once so there
shouldn't be any conflicts. Did the compression test on each node and it
seemed fine.
Just to rehash all the logs I have posted:
Master: http://pastebin.com/xbsVQvVn
ZK: http://pastebin.com/eLqNgf1x
Regionserver (.236): http://pastebin.com/0bUCYkXb
Thanks for the assistance--if it comes down to it I can update to the new
CDH3 and try this whole mess again, but I'd like to figure this out all the
same.
<http://pastebin.com/xbsVQvVn>
On Thu, Apr 14, 2011 at 9:26 PM, Stack <st...@duboce.net> wrote:
> That all sounds reasonable.
>
> Whats in log at other end of this connection that is EOFE'ing? Can
> you see anything?
>
> St.Ack
>
> On Thu, Apr 14, 2011 at 9:25 PM, Jonathan Bender
> <jo...@gmail.com> wrote:
> > Currently running CDH3b4: hbase-0.90.1, zookeeper-3.3.2.
> > Only have one ZK server, it's a small cluster.
> >
> > On Thu, Apr 14, 2011 at 9:14 PM, Stack <st...@duboce.net> wrote:
> >>
> >> Anything in the zk logs for you zk ensemble?
> >>
> >> This is odd:
> >>
> >> 2011-04-14 18:04:35,279 - WARN
> >> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@634] -
> >> EndOfStreamException: Unable to read additional data from client
> >> sessionid 0x12f569514a30015, likely client has closed socket
> >>
> >> Your zk client is a compatible version?
> >>
> >> Then we do this:
> >>
> >> 2011-04-14 18:05:16,001 - INFO [SessionTracker:ZooKeeperServer@314] -
> >> Expiring session 0x12f569514a30015, timeout of 40000ms exceeded
> >>
> >> What versions are you using of hbase/zk?
> >>
> >> St.Ack
> >>
> >> On Thu, Apr 14, 2011 at 8:23 PM, Jonathan Bender
> >> <jo...@gmail.com> wrote:
> >> > Just to provide a bit more context: here is the log for Zookeeper. I
> am
> >> > getting a "NoNode" exception after creating the table in the shell.
> >> >
> >> > http://pastebin.com/0bUCYkXb
> >> >
> >> > <http://pastebin.com/0bUCYkXb>Cheers,
> >> > Jon
> >> >
> >> > On Thu, Apr 14, 2011 at 5:50 PM, Jonathan Bender
> >> > <jo...@gmail.com>wrote:
> >> >
> >> >> Thanks Matt. I added the HBase libs and I could create tables.
> >> >>
> >> >> However...now I get an error when trying to create a table through
> the
> >> >> shell (either compressed or not). It can't seem to find a
> regionserver
> >> >> that
> >> >> will assign it, and then ZK loses connection to HBase
> >> >>
> >> >> hbase(main):011:0> create 'test_t', 'data'
> >> >>
> >> >> ERROR: java.io.IOException: java.io.IOException:
> >> >> java.lang.InterruptedException
> >> >>
> >> >> HBase Master log: http://pastebin.com/xbsVQvVn
> >> >>
> >> >>
> >> >> Anyone seen this before? I'm at a bit of a loss here, as I don't see
> >> >> how
> >> >> the LZO stuff I added relates to this.
> >> >>
> >> >> On Thu, Apr 14, 2011 at 3:33 PM, Matt Davies <ma...@tynt.com>
> >> >> wrote:
> >> >>
> >> >>> I'd suspect you will find a region server without LZO enabled. I
> >> >>> actually
> >> >>> saw this today. It seems to create in the shell but never completes.
> >> >>>
> >> >>> I'd do (in hbase shell) status 'detailed' and see if there are any
> >> >>> dead
> >> >>> servers. Then take a look at the hbase logs. I suspect you'll see
> a
> >> >>> stack
> >> >>> trace about can't find LZO in the classpath.
> >> >>>
> >> >>> Do you have any other tables successfully enabled with LZO?
> >> >>>
> >> >>> BTW - we fixed this by putting lzo in the hbase/lib directory and
> >> >>> restarting. To make things easy we just dropped and recreated the
> >> >>> table.
> >> >>>
> >> >>> HTH
> >> >>>
> >> >>> -Matt
> >> >>>
> >> >>> On Thu, Apr 14, 2011 at 4:28 PM, Jonathan Bender <
> >> >>> jonathan.bender@gmail.com> wrote:
> >> >>>
> >> >>>> Hello,
> >> >>>>
> >> >>>> I'm having an issue with a table I created earlier, testing out LZO
> >> >>>> compression. For some reason I can't disable/drop the table, since
> >> >>>> it is
> >> >>>> constantly in an OPENING state and never gets assigned to a region.
> >> >>>> Does
> >> >>>> anyone have experience with this, and a possible way to
> >> >>>> disable/manage
> >> >>>> this
> >> >>>> table?
> >> >>>>
> >> >>>> Thanks!
> >> >>>> Jon
> >> >>>>
> >> >>>> Log from the HBase master:
> >> >>>> http://pastebin.com/i00uXHJb
> >> >>>>
> >> >>>
> >> >>>
> >> >>
> >> >
> >
> >
>
Re: HBase crashes on table creation
Posted by Stack <st...@duboce.net>.
That all sounds reasonable.
Whats in log at other end of this connection that is EOFE'ing? Can
you see anything?
St.Ack
On Thu, Apr 14, 2011 at 9:25 PM, Jonathan Bender
<jo...@gmail.com> wrote:
> Currently running CDH3b4: hbase-0.90.1, zookeeper-3.3.2.
> Only have one ZK server, it's a small cluster.
>
> On Thu, Apr 14, 2011 at 9:14 PM, Stack <st...@duboce.net> wrote:
>>
>> Anything in the zk logs for you zk ensemble?
>>
>> This is odd:
>>
>> 2011-04-14 18:04:35,279 - WARN
>> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@634] -
>> EndOfStreamException: Unable to read additional data from client
>> sessionid 0x12f569514a30015, likely client has closed socket
>>
>> Your zk client is a compatible version?
>>
>> Then we do this:
>>
>> 2011-04-14 18:05:16,001 - INFO [SessionTracker:ZooKeeperServer@314] -
>> Expiring session 0x12f569514a30015, timeout of 40000ms exceeded
>>
>> What versions are you using of hbase/zk?
>>
>> St.Ack
>>
>> On Thu, Apr 14, 2011 at 8:23 PM, Jonathan Bender
>> <jo...@gmail.com> wrote:
>> > Just to provide a bit more context: here is the log for Zookeeper. I am
>> > getting a "NoNode" exception after creating the table in the shell.
>> >
>> > http://pastebin.com/0bUCYkXb
>> >
>> > <http://pastebin.com/0bUCYkXb>Cheers,
>> > Jon
>> >
>> > On Thu, Apr 14, 2011 at 5:50 PM, Jonathan Bender
>> > <jo...@gmail.com>wrote:
>> >
>> >> Thanks Matt. I added the HBase libs and I could create tables.
>> >>
>> >> However...now I get an error when trying to create a table through the
>> >> shell (either compressed or not). It can't seem to find a regionserver
>> >> that
>> >> will assign it, and then ZK loses connection to HBase
>> >>
>> >> hbase(main):011:0> create 'test_t', 'data'
>> >>
>> >> ERROR: java.io.IOException: java.io.IOException:
>> >> java.lang.InterruptedException
>> >>
>> >> HBase Master log: http://pastebin.com/xbsVQvVn
>> >>
>> >>
>> >> Anyone seen this before? I'm at a bit of a loss here, as I don't see
>> >> how
>> >> the LZO stuff I added relates to this.
>> >>
>> >> On Thu, Apr 14, 2011 at 3:33 PM, Matt Davies <ma...@tynt.com>
>> >> wrote:
>> >>
>> >>> I'd suspect you will find a region server without LZO enabled. I
>> >>> actually
>> >>> saw this today. It seems to create in the shell but never completes.
>> >>>
>> >>> I'd do (in hbase shell) status 'detailed' and see if there are any
>> >>> dead
>> >>> servers. Then take a look at the hbase logs. I suspect you'll see a
>> >>> stack
>> >>> trace about can't find LZO in the classpath.
>> >>>
>> >>> Do you have any other tables successfully enabled with LZO?
>> >>>
>> >>> BTW - we fixed this by putting lzo in the hbase/lib directory and
>> >>> restarting. To make things easy we just dropped and recreated the
>> >>> table.
>> >>>
>> >>> HTH
>> >>>
>> >>> -Matt
>> >>>
>> >>> On Thu, Apr 14, 2011 at 4:28 PM, Jonathan Bender <
>> >>> jonathan.bender@gmail.com> wrote:
>> >>>
>> >>>> Hello,
>> >>>>
>> >>>> I'm having an issue with a table I created earlier, testing out LZO
>> >>>> compression. For some reason I can't disable/drop the table, since
>> >>>> it is
>> >>>> constantly in an OPENING state and never gets assigned to a region.
>> >>>> Does
>> >>>> anyone have experience with this, and a possible way to
>> >>>> disable/manage
>> >>>> this
>> >>>> table?
>> >>>>
>> >>>> Thanks!
>> >>>> Jon
>> >>>>
>> >>>> Log from the HBase master:
>> >>>> http://pastebin.com/i00uXHJb
>> >>>>
>> >>>
>> >>>
>> >>
>> >
>
>
Re: HBase crashes on table creation
Posted by Jonathan Bender <jo...@gmail.com>.
Currently running CDH3b4: hbase-0.90.1, zookeeper-3.3.2.
Only have one ZK server, it's a small cluster.
On Thu, Apr 14, 2011 at 9:14 PM, Stack <st...@duboce.net> wrote:
> Anything in the zk logs for you zk ensemble?
>
> This is odd:
>
> 2011-04-14 18:04:35,279 - WARN
> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@634] -
> EndOfStreamException: Unable to read additional data from client
> sessionid 0x12f569514a30015, likely client has closed socket
>
> Your zk client is a compatible version?
>
> Then we do this:
>
> 2011-04-14 18:05:16,001 - INFO [SessionTracker:ZooKeeperServer@314] -
> Expiring session 0x12f569514a30015, timeout of 40000ms exceeded
>
> What versions are you using of hbase/zk?
>
> St.Ack
>
> On Thu, Apr 14, 2011 at 8:23 PM, Jonathan Bender
> <jo...@gmail.com> wrote:
> > Just to provide a bit more context: here is the log for Zookeeper. I am
> > getting a "NoNode" exception after creating the table in the shell.
> >
> > http://pastebin.com/0bUCYkXb
> >
> > <http://pastebin.com/0bUCYkXb>Cheers,
> > Jon
> >
> > On Thu, Apr 14, 2011 at 5:50 PM, Jonathan Bender
> > <jo...@gmail.com>wrote:
> >
> >> Thanks Matt. I added the HBase libs and I could create tables.
> >>
> >> However...now I get an error when trying to create a table through the
> >> shell (either compressed or not). It can't seem to find a regionserver
> that
> >> will assign it, and then ZK loses connection to HBase
> >>
> >> hbase(main):011:0> create 'test_t', 'data'
> >>
> >> ERROR: java.io.IOException: java.io.IOException:
> >> java.lang.InterruptedException
> >>
> >> HBase Master log: http://pastebin.com/xbsVQvVn
> >>
> >>
> >> Anyone seen this before? I'm at a bit of a loss here, as I don't see
> how
> >> the LZO stuff I added relates to this.
> >>
> >> On Thu, Apr 14, 2011 at 3:33 PM, Matt Davies <ma...@tynt.com>
> wrote:
> >>
> >>> I'd suspect you will find a region server without LZO enabled. I
> actually
> >>> saw this today. It seems to create in the shell but never completes.
> >>>
> >>> I'd do (in hbase shell) status 'detailed' and see if there are any dead
> >>> servers. Then take a look at the hbase logs. I suspect you'll see a
> stack
> >>> trace about can't find LZO in the classpath.
> >>>
> >>> Do you have any other tables successfully enabled with LZO?
> >>>
> >>> BTW - we fixed this by putting lzo in the hbase/lib directory and
> >>> restarting. To make things easy we just dropped and recreated the
> table.
> >>>
> >>> HTH
> >>>
> >>> -Matt
> >>>
> >>> On Thu, Apr 14, 2011 at 4:28 PM, Jonathan Bender <
> >>> jonathan.bender@gmail.com> wrote:
> >>>
> >>>> Hello,
> >>>>
> >>>> I'm having an issue with a table I created earlier, testing out LZO
> >>>> compression. For some reason I can't disable/drop the table, since it
> is
> >>>> constantly in an OPENING state and never gets assigned to a region.
> Does
> >>>> anyone have experience with this, and a possible way to disable/manage
> >>>> this
> >>>> table?
> >>>>
> >>>> Thanks!
> >>>> Jon
> >>>>
> >>>> Log from the HBase master:
> >>>> http://pastebin.com/i00uXHJb
> >>>>
> >>>
> >>>
> >>
> >
>
Re: HBase crashes on table creation
Posted by Stack <st...@duboce.net>.
Anything in the zk logs for you zk ensemble?
This is odd:
2011-04-14 18:04:35,279 - WARN
[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@634] -
EndOfStreamException: Unable to read additional data from client
sessionid 0x12f569514a30015, likely client has closed socket
Your zk client is a compatible version?
Then we do this:
2011-04-14 18:05:16,001 - INFO [SessionTracker:ZooKeeperServer@314] -
Expiring session 0x12f569514a30015, timeout of 40000ms exceeded
What versions are you using of hbase/zk?
St.Ack
On Thu, Apr 14, 2011 at 8:23 PM, Jonathan Bender
<jo...@gmail.com> wrote:
> Just to provide a bit more context: here is the log for Zookeeper. I am
> getting a "NoNode" exception after creating the table in the shell.
>
> http://pastebin.com/0bUCYkXb
>
> <http://pastebin.com/0bUCYkXb>Cheers,
> Jon
>
> On Thu, Apr 14, 2011 at 5:50 PM, Jonathan Bender
> <jo...@gmail.com>wrote:
>
>> Thanks Matt. I added the HBase libs and I could create tables.
>>
>> However...now I get an error when trying to create a table through the
>> shell (either compressed or not). It can't seem to find a regionserver that
>> will assign it, and then ZK loses connection to HBase
>>
>> hbase(main):011:0> create 'test_t', 'data'
>>
>> ERROR: java.io.IOException: java.io.IOException:
>> java.lang.InterruptedException
>>
>> HBase Master log: http://pastebin.com/xbsVQvVn
>>
>>
>> Anyone seen this before? I'm at a bit of a loss here, as I don't see how
>> the LZO stuff I added relates to this.
>>
>> On Thu, Apr 14, 2011 at 3:33 PM, Matt Davies <ma...@tynt.com> wrote:
>>
>>> I'd suspect you will find a region server without LZO enabled. I actually
>>> saw this today. It seems to create in the shell but never completes.
>>>
>>> I'd do (in hbase shell) status 'detailed' and see if there are any dead
>>> servers. Then take a look at the hbase logs. I suspect you'll see a stack
>>> trace about can't find LZO in the classpath.
>>>
>>> Do you have any other tables successfully enabled with LZO?
>>>
>>> BTW - we fixed this by putting lzo in the hbase/lib directory and
>>> restarting. To make things easy we just dropped and recreated the table.
>>>
>>> HTH
>>>
>>> -Matt
>>>
>>> On Thu, Apr 14, 2011 at 4:28 PM, Jonathan Bender <
>>> jonathan.bender@gmail.com> wrote:
>>>
>>>> Hello,
>>>>
>>>> I'm having an issue with a table I created earlier, testing out LZO
>>>> compression. For some reason I can't disable/drop the table, since it is
>>>> constantly in an OPENING state and never gets assigned to a region. Does
>>>> anyone have experience with this, and a possible way to disable/manage
>>>> this
>>>> table?
>>>>
>>>> Thanks!
>>>> Jon
>>>>
>>>> Log from the HBase master:
>>>> http://pastebin.com/i00uXHJb
>>>>
>>>
>>>
>>
>