You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Norbert Burger <no...@gmail.com> on 2012/09/10 17:33:54 UTC
HBase UI missing region list for active/functioning table
Hi all -- we're currently on cdh3u3 (0.90.4 + patches). I have one
table in our cluster which seems to functioning fine (gets/puts/scans
are all working), but for which no regions are listed on the UI. The
table/regions exist in .META. Other tables in the same cluster show
their regions list fine. Seems like this might be a problem with
.META. or ZK, but would appreciate any pointers.
1) hbase hbck reports 2 "multiply assigned to region servers"
inconsistencies, but on a table different than the one I'm having
problems with.
2) The hbase master log shows this fragment when navigating to
table.jsp for the affected table:
2012-09-10 11:29:07,682 DEBUG org.apache.zookeeper.ClientCnxn: Reading
reply sessionid:0x1370e3604c49580, packet:: clientPath:null
serverPath:null finished:false header:: 10,4 replyHeader::
10,167713215,-101 request:: '/hbase/table/sessions,F response::
2012-09-10 11:29:07,682 DEBUG
org.apache.hadoop.hbase.zookeeper.ZKUtil:
hconnection-0x1370e3604c49580 Unable to get data of znode
/hbase/table/sessions because node does not exist (not an error)
2012-09-10 11:29:07,682 DEBUG
org.apache.hadoop.hbase.client.MetaScanner: Scanning .META. starting
at row=sessions,,00000000000000 for max=2147483647 rows
But since I see this "Unable to get data of znode" for all tables, my
assumption is that it's a red herring. Any thoughts as how to debug
further, or why only this table would not show a region list?
Norbert
Re: HBase UI missing region list for active/functioning table
Posted by Norbert Burger <no...@gmail.com>.
On Mon, Sep 10, 2012 at 3:29 PM, Stack <st...@duboce.net> wrote:
> On Mon, Sep 10, 2012 at 12:05 PM, Norbert Burger
> <no...@gmail.com> wrote:
>>
> Mind putting up full listing in pastebin?
Here's a link: http://pastebin.com/raw.php?i=4YhS8CpE. The table in
question is called 'sessions', I did delete other tables' info from
this dump, as the .META. was quite large otherwise.
> We could try a master restart too... so it refreshes its in-memory
> state. That might do it.
We've actually done this already, it hasn't seemed to resolve the situation.
Thanks,
Norbert
Re: HBase UI missing region list for active/functioning table
Posted by Stack <st...@duboce.net>.
On Mon, Sep 10, 2012 at 12:05 PM, Norbert Burger
<no...@gmail.com> wrote:
>
Mind putting up full listing in pastebin?
Let me have a look.
We could try a master restart too... so it refreshes its in-memory
state. That might do it.
St.Ack
Re: HBase UI missing region list for active/functioning table
Posted by Norbert Burger <no...@gmail.com>.
On Mon, Sep 10, 2012 at 2:17 PM, Stack <st...@duboce.net> wrote:
> Thanks. I was asking about the info:regioninfo column that prints out
> the HRegionInfo for each region. I was wondering if it included a
> status=offline attribute.
>
> You could try one region only and see if that makes a difference.
Hmmm... no status=offline anywhere in my dump of .META. Will dig into
the code and try disable/enable when I get I chance.
Attached below is a copy of .META. for one of the problematic regions.
As far as I can tell, it has all the required cols, and I don't see a
difference between this and a region which gets "displayed" correctly:
sessions,,1342211893146.8 column=info:regioninfo,
timestamp=1342211893177, value=REGION => {NAME =>
81ca12b4a6c9a7670bb7ef69b
'sessions,,1342211893146.881ca12b4a6c9a7670bb7ef69b3e5db4.', STARTKEY
=>
3e5db4. '', ENDKEY => '01', ENCODED =>
881ca12b4a6c9a7670bb7ef69b3e5db4, TABLE =
> {{NAME => 'sessions', FAMILIES => [{NAME
=> 'event', BLOOMFILTER => 'RO
W', REPLICATION_SCOPE => '0', VERSIONS =>
'1', COMPRESSION => 'LZO', TTL
=> '2147483647', BLOCKSIZE => '65536',
IN_MEMORY => 'false', BLOCKCACHE =
> 'true'}]}}
sessions,,1342211893146.8 column=info:server,
timestamp=1346979452245, value=aspen8hdp2.turner.com:
81ca12b4a6c9a7670bb7ef69b 60020
3e5db4.
sessions,,1342211893146.8 column=info:serverstartcode,
timestamp=1346979452245, value=1346977435555
81ca12b4a6c9a7670bb7ef69b
3e5db4.
Thanks again,
Norbert
Re: HBase UI missing region list for active/functioning table
Posted by Stack <st...@duboce.net>.
On Mon, Sep 10, 2012 at 10:50 AM, Norbert Burger
<no...@gmail.com> wrote:
> On Mon, Sep 10, 2012 at 1:37 PM, Stack <st...@duboce.net> wrote:
>> What version of hbase?
>
> We're on cdh3u3, 0.90.4 + patches.
>
>> Can you disable and reenable the table?
>
> I will try disabling/re-enabling at the next opportunity. Perhaps
> that'll resolve that the issue, but this is a PROD cluster, so
> unfortunately can't try right away.
>
>> When you scan the table in shell, do you see 'status=offline'.
Thanks. I was asking about the info:regioninfo column that prints out
the HRegionInfo for each region. I was wondering if it included a
status=offline attribute.
You could try one region only and see if that makes a difference.
My guess is that this a vestige of the rename script. You disabled
before using it (as it asks for at head of script).
St.Ack
Re: HBase UI missing region list for active/functioning table
Posted by Norbert Burger <no...@gmail.com>.
On Mon, Sep 10, 2012 at 1:37 PM, Stack <st...@duboce.net> wrote:
> What version of hbase?
We're on cdh3u3, 0.90.4 + patches.
> Can you disable and reenable the table?
I will try disabling/re-enabling at the next opportunity. Perhaps
that'll resolve that the issue, but this is a PROD cluster, so
unfortunately can't try right away.
> When you scan the table in shell, do you see 'status=offline'.
a) the master UI (table.jsp) shows the table, and reports enabled=true
b) is_enabled from the shell also reports true
Norbert
Re: HBase UI missing region list for active/functioning table
Posted by Stack <st...@duboce.net>.
On Mon, Sep 10, 2012 at 10:33 AM, Norbert Burger
<no...@gmail.com> wrote:
> On Mon, Sep 10, 2012 at 1:24 PM, Srinivas Mupparapu
> <j2...@gmail.com> wrote:
>> It scans .META. table just like any other table. I just tested it and it
>> produced the expected output.
>
> I'm pretty sure Srinivas scanned .META. in his own environment, not mine. ;-)
>
>> On Sep 10, 2012 12:19 PM, "Stack" <st...@duboce.net> wrote:
>>> What happens if you scan .META. in shell?
>>>
>>> hbase> scan ".META."
>>>
>>> Does it all show?
>
> Thanks, Stack. Strangely, all regions do show up in .META. The table
> in question has 256 regions, and all are listed as rowkeys in .META.
> Perhaps there's a column missing from region definitions which is
> preventing the UI from rendering the regions? I'll dig through the
> code, but are there specific columns known to be expected the UI?
>
> Fwiw, this particular table was the result of a rename_table.rb
> attempt that didn't go as smoothly as I would've liked. I had to dig
> through .META. and resolve inconsistencies.
>
What version of hbase? Can you disable and reenable the table? When
you scan the table in shell, do you see 'status=offline'.
St.Ack
Re: HBase UI missing region list for active/functioning table
Posted by Norbert Burger <no...@gmail.com>.
On Mon, Sep 10, 2012 at 1:24 PM, Srinivas Mupparapu
<j2...@gmail.com> wrote:
> It scans .META. table just like any other table. I just tested it and it
> produced the expected output.
I'm pretty sure Srinivas scanned .META. in his own environment, not mine. ;-)
> On Sep 10, 2012 12:19 PM, "Stack" <st...@duboce.net> wrote:
>> What happens if you scan .META. in shell?
>>
>> hbase> scan ".META."
>>
>> Does it all show?
Thanks, Stack. Strangely, all regions do show up in .META. The table
in question has 256 regions, and all are listed as rowkeys in .META.
Perhaps there's a column missing from region definitions which is
preventing the UI from rendering the regions? I'll dig through the
code, but are there specific columns known to be expected the UI?
Fwiw, this particular table was the result of a rename_table.rb
attempt that didn't go as smoothly as I would've liked. I had to dig
through .META. and resolve inconsistencies.
Norbert
Re: HBase UI missing region list for active/functioning table
Posted by Stack <st...@duboce.net>.
On Mon, Sep 10, 2012 at 10:24 AM, Srinivas Mupparapu
<j2...@gmail.com> wrote:
> It scans .META. table just like any other table. I just tested it and it
> produced the expected output.
>
When you refresh the master UI, it makes a few lines in the master
log. Are these the lines you posted? Mind checking again? What does
the Master UI page look like? Complete? Or is it cut off where its
should be listing regions (maybe look at html src?).
If shell can scan .META., odd that UI can't. Lets try and figure the
difference.
St.Ack
> Thanks,
> Srinivas M
> On Sep 10, 2012 12:19 PM, "Stack" <st...@duboce.net> wrote:
>
>> On Mon, Sep 10, 2012 at 8:33 AM, Norbert Burger
>> <no...@gmail.com> wrote:
>> > Hi all -- we're currently on cdh3u3 (0.90.4 + patches). I have one
>> > table in our cluster which seems to functioning fine (gets/puts/scans
>> > are all working), but for which no regions are listed on the UI. The
>> > table/regions exist in .META. Other tables in the same cluster show
>> > their regions list fine. Seems like this might be a problem with
>> > .META. or ZK, but would appreciate any pointers.
>> >
>> > 1) hbase hbck reports 2 "multiply assigned to region servers"
>> > inconsistencies, but on a table different than the one I'm having
>> > problems with.
>> > 2) The hbase master log shows this fragment when navigating to
>> > table.jsp for the affected table:
>> >
>> > 2012-09-10 11:29:07,682 DEBUG org.apache.zookeeper.ClientCnxn: Reading
>> > reply sessionid:0x1370e3604c49580, packet:: clientPath:null
>> > serverPath:null finished:false header:: 10,4 replyHeader::
>> > 10,167713215,-101 request:: '/hbase/table/sessions,F response::
>> > 2012-09-10 11:29:07,682 DEBUG
>> > org.apache.hadoop.hbase.zookeeper.ZKUtil:
>> > hconnection-0x1370e3604c49580 Unable to get data of znode
>> > /hbase/table/sessions because node does not exist (not an error)
>> > 2012-09-10 11:29:07,682 DEBUG
>> > org.apache.hadoop.hbase.client.MetaScanner: Scanning .META. starting
>> > at row=sessions,,00000000000000 for max=2147483647 rows
>> >
>> > But since I see this "Unable to get data of znode" for all tables, my
>> > assumption is that it's a red herring. Any thoughts as how to debug
>> > further, or why only this table would not show a region list?
>> >
>>
>> What happens if you scan .META. in shell?
>>
>> hbase> scan ".META."
>>
>> Does it all show?
>>
>> (You might want to echo into a file so you can poke around after scan is
>> done).
>>
>> St.Ack
>>
Re: HBase UI missing region list for active/functioning table
Posted by Srinivas Mupparapu <j2...@gmail.com>.
It scans .META. table just like any other table. I just tested it and it
produced the expected output.
Thanks,
Srinivas M
On Sep 10, 2012 12:19 PM, "Stack" <st...@duboce.net> wrote:
> On Mon, Sep 10, 2012 at 8:33 AM, Norbert Burger
> <no...@gmail.com> wrote:
> > Hi all -- we're currently on cdh3u3 (0.90.4 + patches). I have one
> > table in our cluster which seems to functioning fine (gets/puts/scans
> > are all working), but for which no regions are listed on the UI. The
> > table/regions exist in .META. Other tables in the same cluster show
> > their regions list fine. Seems like this might be a problem with
> > .META. or ZK, but would appreciate any pointers.
> >
> > 1) hbase hbck reports 2 "multiply assigned to region servers"
> > inconsistencies, but on a table different than the one I'm having
> > problems with.
> > 2) The hbase master log shows this fragment when navigating to
> > table.jsp for the affected table:
> >
> > 2012-09-10 11:29:07,682 DEBUG org.apache.zookeeper.ClientCnxn: Reading
> > reply sessionid:0x1370e3604c49580, packet:: clientPath:null
> > serverPath:null finished:false header:: 10,4 replyHeader::
> > 10,167713215,-101 request:: '/hbase/table/sessions,F response::
> > 2012-09-10 11:29:07,682 DEBUG
> > org.apache.hadoop.hbase.zookeeper.ZKUtil:
> > hconnection-0x1370e3604c49580 Unable to get data of znode
> > /hbase/table/sessions because node does not exist (not an error)
> > 2012-09-10 11:29:07,682 DEBUG
> > org.apache.hadoop.hbase.client.MetaScanner: Scanning .META. starting
> > at row=sessions,,00000000000000 for max=2147483647 rows
> >
> > But since I see this "Unable to get data of znode" for all tables, my
> > assumption is that it's a red herring. Any thoughts as how to debug
> > further, or why only this table would not show a region list?
> >
>
> What happens if you scan .META. in shell?
>
> hbase> scan ".META."
>
> Does it all show?
>
> (You might want to echo into a file so you can poke around after scan is
> done).
>
> St.Ack
>
Re: HBase UI missing region list for active/functioning table
Posted by Stack <st...@duboce.net>.
On Mon, Sep 10, 2012 at 8:33 AM, Norbert Burger
<no...@gmail.com> wrote:
> Hi all -- we're currently on cdh3u3 (0.90.4 + patches). I have one
> table in our cluster which seems to functioning fine (gets/puts/scans
> are all working), but for which no regions are listed on the UI. The
> table/regions exist in .META. Other tables in the same cluster show
> their regions list fine. Seems like this might be a problem with
> .META. or ZK, but would appreciate any pointers.
>
> 1) hbase hbck reports 2 "multiply assigned to region servers"
> inconsistencies, but on a table different than the one I'm having
> problems with.
> 2) The hbase master log shows this fragment when navigating to
> table.jsp for the affected table:
>
> 2012-09-10 11:29:07,682 DEBUG org.apache.zookeeper.ClientCnxn: Reading
> reply sessionid:0x1370e3604c49580, packet:: clientPath:null
> serverPath:null finished:false header:: 10,4 replyHeader::
> 10,167713215,-101 request:: '/hbase/table/sessions,F response::
> 2012-09-10 11:29:07,682 DEBUG
> org.apache.hadoop.hbase.zookeeper.ZKUtil:
> hconnection-0x1370e3604c49580 Unable to get data of znode
> /hbase/table/sessions because node does not exist (not an error)
> 2012-09-10 11:29:07,682 DEBUG
> org.apache.hadoop.hbase.client.MetaScanner: Scanning .META. starting
> at row=sessions,,00000000000000 for max=2147483647 rows
>
> But since I see this "Unable to get data of znode" for all tables, my
> assumption is that it's a red herring. Any thoughts as how to debug
> further, or why only this table would not show a region list?
>
What happens if you scan .META. in shell?
hbase> scan ".META."
Does it all show?
(You might want to echo into a file so you can poke around after scan is done).
St.Ack