You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by 张莉苹 <zl...@gmail.com> on 2013/10/23 07:46:10 UTC
HConnectionImplementation.listTables and list table exception
Dear HBase dev and users,
Did you meet this
"org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.listTables"
issue?
We setup a 600 nodes cluster, 9 zookeeper nodes to load data into hbase,
but it seemed hbase master was busy handling transition with zookeeper, and
hbase “list” could not get response. The hbase table was created but it
didn't do any insert.
Do you have any idea of the root cause and how to fix it? :)Highly
appreciate for your answers!
Here is the exception stack:
---------------------------------------------------
java.lang.reflect.UndeclaredThrowableException
at $Proxy7.getHTableDescriptors(Unknown Source)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.listTables(HConnectionManager.java:2237)
at
org.apache.hadoop.hbase.client.HBaseAdmin.listTables(HBaseAdmin.java:317)
hbase master log:
-----------------------------
2013-10-18 06:19:41,279 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
master:60000-0x341be88202300ab* Deleting existing unassigned node* for
0ec3308bd1e2bdd9576b2d60d2eee68e that is in expected state
RS_ZK_REGION_OPENED
2013-10-18 06:19:41,279 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager:* Handling
transition=RS_ZK_REGION_OPENING*, s*erver=node0878*.
ic.analyticsworkbench.com,60020,1381883086785,
region=15a4fb29aa1d905b13f33594e50bc8de, which is more than 15 seconds late
2013-10-18 06:19:41,280 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: *Handling
transition=RS_ZK_REGION_OPENING,
server=node0898*.ic.analyticsworkbench.com,60020,1381883200494,
region=1a4c929534e6828c85f22b062f949304, which is more than 15 seconds late
2013-10-18 06:19:41,289 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
master:60000-0x341be88202300ab Successfully *deleted unassigned node *for
region 0ec3308bd1e2bdd9576b2d60d2eee68e in expected state
RS_ZK_REGION_OPENED
2013-10-18 06:19:41,289 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING,
server=node0693.ic.analyticsworkbench.com,60020,1381881773670,
region=d47bfe1af0051c405de295a51c1c6e63, which is more than 15 seconds late
We also try to "list" in hbase shell,it also failed:
The hbase “list” got error as:
------------------------------------------
hbase(main):001:0> list
TABLE
ERROR: java.lang.reflect.UndeclaredThrowableException: Call to
node0997.ic.analyticsworkbench.com/10.1.50.17:60000 failed on socket
timeout exception: java.net.SocketTimeoutException: 120000 millis timeout
while waiting for channel to be ready for read. ch :
java.nio.channels.SocketChannel[connected local=/10.1.50.15:45726 remote=
node0997.ic.analyticsworkbench.com/10.1.50.17:60000]
Cheers,
-----
Big Data - Big Wisdom - Big Value
--------------
Michelle Zhang (Li Ping Zhang)
Re: HConnectionImplementation.listTables and list table exception
Posted by Ted Yu <yu...@gmail.com>.
What version of HBase and Hadoop are you using ?
Can you show us more of the master log ?
Thanks
On Oct 22, 2013, at 10:46 PM, 张莉苹 <zl...@gmail.com> wrote:
> Dear HBase dev and users,
>
> Did you meet this
> "org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.listTables"
> issue?
>
> We setup a 600 nodes cluster, 9 zookeeper nodes to load data into hbase,
> but it seemed hbase master was busy handling transition with zookeeper, and
> hbase “list” could not get response. The hbase table was created but it
> didn't do any insert.
>
> Do you have any idea of the root cause and how to fix it? :)Highly
> appreciate for your answers!
>
>
>
> Here is the exception stack:
> ---------------------------------------------------
> java.lang.reflect.UndeclaredThrowableException
> at $Proxy7.getHTableDescriptors(Unknown Source)
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.listTables(HConnectionManager.java:2237)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.listTables(HBaseAdmin.java:317)
>
>
>
>
> hbase master log:
>
> -----------------------------
>
> 2013-10-18 06:19:41,279 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
> master:60000-0x341be88202300ab* Deleting existing unassigned node* for
> 0ec3308bd1e2bdd9576b2d60d2eee68e that is in expected state
> RS_ZK_REGION_OPENED
>
> 2013-10-18 06:19:41,279 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager:* Handling
> transition=RS_ZK_REGION_OPENING*, s*erver=node0878*.
> ic.analyticsworkbench.com,60020,1381883086785,
> region=15a4fb29aa1d905b13f33594e50bc8de, which is more than 15 seconds late
>
> 2013-10-18 06:19:41,280 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: *Handling
> transition=RS_ZK_REGION_OPENING,
> server=node0898*.ic.analyticsworkbench.com,60020,1381883200494,
> region=1a4c929534e6828c85f22b062f949304, which is more than 15 seconds late
>
> 2013-10-18 06:19:41,289 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
> master:60000-0x341be88202300ab Successfully *deleted unassigned node *for
> region 0ec3308bd1e2bdd9576b2d60d2eee68e in expected state
> RS_ZK_REGION_OPENED
>
> 2013-10-18 06:19:41,289 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Handling
> transition=RS_ZK_REGION_OPENING,
> server=node0693.ic.analyticsworkbench.com,60020,1381881773670,
> region=d47bfe1af0051c405de295a51c1c6e63, which is more than 15 seconds late
>
>
>
> We also try to "list" in hbase shell,it also failed:
>
> The hbase “list” got error as:
>
> ------------------------------------------
>
>
>
> hbase(main):001:0> list
>
> TABLE
>
>
>
>
> ERROR: java.lang.reflect.UndeclaredThrowableException: Call to
> node0997.ic.analyticsworkbench.com/10.1.50.17:60000 failed on socket
> timeout exception: java.net.SocketTimeoutException: 120000 millis timeout
> while waiting for channel to be ready for read. ch :
> java.nio.channels.SocketChannel[connected local=/10.1.50.15:45726 remote=
> node0997.ic.analyticsworkbench.com/10.1.50.17:60000]
>
>
>
>
> Cheers,
> -----
> Big Data - Big Wisdom - Big Value
> --------------
> Michelle Zhang (Li Ping Zhang)
Re: HConnectionImplementation.listTables and list table exception
Posted by Michael Segel <mi...@hotmail.com>.
Why 9 zookeepers?
On Oct 23, 2013, at 12:46 AM, 张莉苹 <zl...@gmail.com> wrote:
> Dear HBase dev and users,
>
> Did you meet this
> "org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.listTables"
> issue?
>
> We setup a 600 nodes cluster, 9 zookeeper nodes to load data into hbase,
> but it seemed hbase master was busy handling transition with zookeeper, and
> hbase “list” could not get response. The hbase table was created but it
> didn't do any insert.
>
> Do you have any idea of the root cause and how to fix it? :)Highly
> appreciate for your answers!
>
>
>
> Here is the exception stack:
> ---------------------------------------------------
> java.lang.reflect.UndeclaredThrowableException
> at $Proxy7.getHTableDescriptors(Unknown Source)
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.listTables(HConnectionManager.java:2237)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.listTables(HBaseAdmin.java:317)
>
>
>
>
> hbase master log:
>
> -----------------------------
>
> 2013-10-18 06:19:41,279 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
> master:60000-0x341be88202300ab* Deleting existing unassigned node* for
> 0ec3308bd1e2bdd9576b2d60d2eee68e that is in expected state
> RS_ZK_REGION_OPENED
>
> 2013-10-18 06:19:41,279 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager:* Handling
> transition=RS_ZK_REGION_OPENING*, s*erver=node0878*.
> ic.analyticsworkbench.com,60020,1381883086785,
> region=15a4fb29aa1d905b13f33594e50bc8de, which is more than 15 seconds late
>
> 2013-10-18 06:19:41,280 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: *Handling
> transition=RS_ZK_REGION_OPENING,
> server=node0898*.ic.analyticsworkbench.com,60020,1381883200494,
> region=1a4c929534e6828c85f22b062f949304, which is more than 15 seconds late
>
> 2013-10-18 06:19:41,289 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
> master:60000-0x341be88202300ab Successfully *deleted unassigned node *for
> region 0ec3308bd1e2bdd9576b2d60d2eee68e in expected state
> RS_ZK_REGION_OPENED
>
> 2013-10-18 06:19:41,289 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Handling
> transition=RS_ZK_REGION_OPENING,
> server=node0693.ic.analyticsworkbench.com,60020,1381881773670,
> region=d47bfe1af0051c405de295a51c1c6e63, which is more than 15 seconds late
>
>
>
> We also try to "list" in hbase shell,it also failed:
>
> The hbase “list” got error as:
>
> ------------------------------------------
>
>
>
> hbase(main):001:0> list
>
> TABLE
>
>
>
>
> ERROR: java.lang.reflect.UndeclaredThrowableException: Call to
> node0997.ic.analyticsworkbench.com/10.1.50.17:60000 failed on socket
> timeout exception: java.net.SocketTimeoutException: 120000 millis timeout
> while waiting for channel to be ready for read. ch :
> java.nio.channels.SocketChannel[connected local=/10.1.50.15:45726 remote=
> node0997.ic.analyticsworkbench.com/10.1.50.17:60000]
>
>
>
>
> Cheers,
> -----
> Big Data - Big Wisdom - Big Value
> --------------
> Michelle Zhang (Li Ping Zhang)
The opinions expressed here are mine, while they may reflect a cognitive thought, that is purely accidental.
Use at your own risk.
Michael Segel
michael_segel (AT) hotmail.com
Re: HConnectionImplementation.listTables and list table exception
Posted by Ted Yu <yu...@gmail.com>.
What version of HBase and Hadoop are you using ?
Can you show us more of the master log ?
Thanks
On Oct 22, 2013, at 10:46 PM, 张莉苹 <zl...@gmail.com> wrote:
> Dear HBase dev and users,
>
> Did you meet this
> "org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.listTables"
> issue?
>
> We setup a 600 nodes cluster, 9 zookeeper nodes to load data into hbase,
> but it seemed hbase master was busy handling transition with zookeeper, and
> hbase “list” could not get response. The hbase table was created but it
> didn't do any insert.
>
> Do you have any idea of the root cause and how to fix it? :)Highly
> appreciate for your answers!
>
>
>
> Here is the exception stack:
> ---------------------------------------------------
> java.lang.reflect.UndeclaredThrowableException
> at $Proxy7.getHTableDescriptors(Unknown Source)
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.listTables(HConnectionManager.java:2237)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.listTables(HBaseAdmin.java:317)
>
>
>
>
> hbase master log:
>
> -----------------------------
>
> 2013-10-18 06:19:41,279 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
> master:60000-0x341be88202300ab* Deleting existing unassigned node* for
> 0ec3308bd1e2bdd9576b2d60d2eee68e that is in expected state
> RS_ZK_REGION_OPENED
>
> 2013-10-18 06:19:41,279 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager:* Handling
> transition=RS_ZK_REGION_OPENING*, s*erver=node0878*.
> ic.analyticsworkbench.com,60020,1381883086785,
> region=15a4fb29aa1d905b13f33594e50bc8de, which is more than 15 seconds late
>
> 2013-10-18 06:19:41,280 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: *Handling
> transition=RS_ZK_REGION_OPENING,
> server=node0898*.ic.analyticsworkbench.com,60020,1381883200494,
> region=1a4c929534e6828c85f22b062f949304, which is more than 15 seconds late
>
> 2013-10-18 06:19:41,289 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
> master:60000-0x341be88202300ab Successfully *deleted unassigned node *for
> region 0ec3308bd1e2bdd9576b2d60d2eee68e in expected state
> RS_ZK_REGION_OPENED
>
> 2013-10-18 06:19:41,289 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Handling
> transition=RS_ZK_REGION_OPENING,
> server=node0693.ic.analyticsworkbench.com,60020,1381881773670,
> region=d47bfe1af0051c405de295a51c1c6e63, which is more than 15 seconds late
>
>
>
> We also try to "list" in hbase shell,it also failed:
>
> The hbase “list” got error as:
>
> ------------------------------------------
>
>
>
> hbase(main):001:0> list
>
> TABLE
>
>
>
>
> ERROR: java.lang.reflect.UndeclaredThrowableException: Call to
> node0997.ic.analyticsworkbench.com/10.1.50.17:60000 failed on socket
> timeout exception: java.net.SocketTimeoutException: 120000 millis timeout
> while waiting for channel to be ready for read. ch :
> java.nio.channels.SocketChannel[connected local=/10.1.50.15:45726 remote=
> node0997.ic.analyticsworkbench.com/10.1.50.17:60000]
>
>
>
>
> Cheers,
> -----
> Big Data - Big Wisdom - Big Value
> --------------
> Michelle Zhang (Li Ping Zhang)
Re: HConnectionImplementation.listTables and list table exception
Posted by Ted Yu <yu...@gmail.com>.
The blog Julian mentioned referred to HBASE-7101
That JIRA was marked dup of HBASE-7551 which was resolved in 0.94.5
This is why upgrading HBase is desired.
On Wed, Oct 23, 2013 at 10:10 AM, Ted Yu <yu...@gmail.com> wrote:
> bq. our hbase version was 0.94.0 and hadoop version was 1.0.3
>
> If possible, consider upgrading HBase to 0.94.12 and Hadoop to 1.2.1
>
> Cheers
>
>
> On Wed, Oct 23, 2013 at 7:18 AM, 张莉苹 <zl...@gmail.com> wrote:
>
>> Hello Ted, and Julian,
>>
>> It seemed I didn't receive your mail from my gmail inbox. It was strange.
>> Anyhow I found your reply by google search. ：）
>>
>> I'll answer your questions here, and thanks very much for your reply.
>>
>>
>> From *Ted*:
>>
>> What version of HBase and Hadoop are you using ?
>>
>> >> our hbase version was 0.94.0 and hadoop version was 1.0.3
>>
>> Can you show us more of the master log ?
>>
>> >> I'm so sorry, the master log could not be accessed. The environment
>> was only used by me for a short while.
>>
>>
>> From * Michael <
>> http://www.mail-archive.com/search?l=user@hbase.apache.org&q=from:%22Michael+Segel%22
>> >:*
>>
>>
>> Why 9 zookeepers?
>>
>> >> the previous zookeeper number was 5, we also thought the number of
>> zookeeper was too small, so we increased it into 9, but it still failed.
>>
>>
>> From *Julian*:
>>
>>
>> Hello Michelle,
>> How many regions totally are there in your 600 nodes cluster? Looks
>> like many of them are pending for open and being assigned to region
>> servers.
>> Can you see many items under zookeeper dir /hbase/unassigned?
>>
>> >> we had 60K regions, the env could not be accessed.
>>
>> You would like to refer
>> http://blog.sina.com.cn/s/blog_4a1f59bf01018tu4.html?
>>
>> >> kind of help, thanks!
>>
>>
>>
>>
>>
>> Cheers,
>> -----
>> Big Data - Big Wisdom - Big Value
>> --------------
>> Michelle Zhang (Li Ping Zhang)
>>
>>
>> 2013/10/23 张莉苹 <zl...@gmail.com>
>>
>> > Dear HBase dev and users,
>> >
>> > Did you meet this
>> >
>> "org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.listTables"
>> > issue?
>> >
>> > We setup a 600 nodes cluster, 9 zookeeper nodes to load data into hbase,
>> > but it seemed hbase master was busy handling transition with zookeeper,
>> > and hbase “list” could not get response. The hbase table was created but
>> > it didn't do any insert.
>> >
>> > Do you have any idea of the root cause and how to fix it? :)Highly
>> > appreciate for your answers!
>> >
>> >
>> >
>> > Here is the exception stack:
>> > ---------------------------------------------------
>> > java.lang.reflect.UndeclaredThrowableException
>> > at $Proxy7.getHTableDescriptors(Unknown Source)
>> > at
>> >
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.listTables(HConnectionManager.java:2237)
>> > at
>> >
>> org.apache.hadoop.hbase.client.HBaseAdmin.listTables(HBaseAdmin.java:317)
>> >
>> >
>> >
>> >
>> > hbase master log:
>> >
>> > -----------------------------
>> >
>> > 2013-10-18 06:19:41,279 DEBUG
>> org.apache.hadoop.hbase.zookeeper.ZKAssign:
>> > master:60000-0x341be88202300ab* Deleting existing unassigned node* for
>> > 0ec3308bd1e2bdd9576b2d60d2eee68e that is in expected state
>> > RS_ZK_REGION_OPENED
>> >
>> > 2013-10-18 06:19:41,279 DEBUG
>> > org.apache.hadoop.hbase.master.AssignmentManager:* Handling
>> > transition=RS_ZK_REGION_OPENING*, s*erver=node0878*.
>> > ic.analyticsworkbench.com,60020,1381883086785,
>> > region=15a4fb29aa1d905b13f33594e50bc8de, which is more than 15 seconds
>> late
>> >
>> > 2013-10-18 06:19:41,280 DEBUG
>> > org.apache.hadoop.hbase.master.AssignmentManager: *Handling
>> > transition=RS_ZK_REGION_OPENING, server=node0898*.
>> > ic.analyticsworkbench.com,60020,1381883200494,
>> > region=1a4c929534e6828c85f22b062f949304, which is more than 15 seconds
>> late
>> >
>> > 2013-10-18 06:19:41,289 DEBUG
>> org.apache.hadoop.hbase.zookeeper.ZKAssign:
>> > master:60000-0x341be88202300ab Successfully *deleted unassigned node
>> *for
>> > region 0ec3308bd1e2bdd9576b2d60d2eee68e in expected state
>> > RS_ZK_REGION_OPENED
>> >
>> > 2013-10-18 06:19:41,289 DEBUG
>> > org.apache.hadoop.hbase.master.AssignmentManager: Handling
>> > transition=RS_ZK_REGION_OPENING, server=
>> node0693.ic.analyticsworkbench.com,60020,1381881773670,
>> > region=d47bfe1af0051c405de295a51c1c6e63, which is more than 15 seconds
>> late
>> >
>> >
>> >
>> > We also try to "list" in hbase shell,it also failed:
>> >
>> > The hbase “list” got error as:
>> >
>> > ------------------------------------------
>> >
>> >
>> >
>> > hbase(main):001:0> list
>> >
>> > TABLE
>> >
>> >
>> >
>> >
>> > ERROR: java.lang.reflect.UndeclaredThrowableException: Call to
>> > node0997.ic.analyticsworkbench.com/10.1.50.17:60000 failed on socket
>> > timeout exception: java.net.SocketTimeoutException: 120000 millis
>> timeout
>> > while waiting for channel to be ready for read. ch :
>> > java.nio.channels.SocketChannel[connected local=/10.1.50.15:45726remote=
>> > node0997.ic.analyticsworkbench.com/10.1.50.17:60000]
>> >
>> >
>> >
>> >
>> > Cheers,
>> > -----
>> > Big Data - Big Wisdom - Big Value
>> > --------------
>> > Michelle Zhang (Li Ping Zhang)
>> >
>>
>
>
Re: HConnectionImplementation.listTables and list table exception
Posted by Ted Yu <yu...@gmail.com>.
The blog Julian mentioned referred to HBASE-7101
That JIRA was marked dup of HBASE-7551 which was resolved in 0.94.5
This is why upgrading HBase is desired.
On Wed, Oct 23, 2013 at 10:10 AM, Ted Yu <yu...@gmail.com> wrote:
> bq. our hbase version was 0.94.0 and hadoop version was 1.0.3
>
> If possible, consider upgrading HBase to 0.94.12 and Hadoop to 1.2.1
>
> Cheers
>
>
> On Wed, Oct 23, 2013 at 7:18 AM, 张莉苹 <zl...@gmail.com> wrote:
>
>> Hello Ted, and Julian,
>>
>> It seemed I didn't receive your mail from my gmail inbox. It was strange.
>> Anyhow I found your reply by google search. ：）
>>
>> I'll answer your questions here, and thanks very much for your reply.
>>
>>
>> From *Ted*:
>>
>> What version of HBase and Hadoop are you using ?
>>
>> >> our hbase version was 0.94.0 and hadoop version was 1.0.3
>>
>> Can you show us more of the master log ?
>>
>> >> I'm so sorry, the master log could not be accessed. The environment
>> was only used by me for a short while.
>>
>>
>> From * Michael <
>> http://www.mail-archive.com/search?l=user@hbase.apache.org&q=from:%22Michael+Segel%22
>> >:*
>>
>>
>> Why 9 zookeepers?
>>
>> >> the previous zookeeper number was 5, we also thought the number of
>> zookeeper was too small, so we increased it into 9, but it still failed.
>>
>>
>> From *Julian*:
>>
>>
>> Hello Michelle,
>> How many regions totally are there in your 600 nodes cluster? Looks
>> like many of them are pending for open and being assigned to region
>> servers.
>> Can you see many items under zookeeper dir /hbase/unassigned?
>>
>> >> we had 60K regions, the env could not be accessed.
>>
>> You would like to refer
>> http://blog.sina.com.cn/s/blog_4a1f59bf01018tu4.html?
>>
>> >> kind of help, thanks!
>>
>>
>>
>>
>>
>> Cheers,
>> -----
>> Big Data - Big Wisdom - Big Value
>> --------------
>> Michelle Zhang (Li Ping Zhang)
>>
>>
>> 2013/10/23 张莉苹 <zl...@gmail.com>
>>
>> > Dear HBase dev and users,
>> >
>> > Did you meet this
>> >
>> "org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.listTables"
>> > issue?
>> >
>> > We setup a 600 nodes cluster, 9 zookeeper nodes to load data into hbase,
>> > but it seemed hbase master was busy handling transition with zookeeper,
>> > and hbase “list” could not get response. The hbase table was created but
>> > it didn't do any insert.
>> >
>> > Do you have any idea of the root cause and how to fix it? :)Highly
>> > appreciate for your answers!
>> >
>> >
>> >
>> > Here is the exception stack:
>> > ---------------------------------------------------
>> > java.lang.reflect.UndeclaredThrowableException
>> > at $Proxy7.getHTableDescriptors(Unknown Source)
>> > at
>> >
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.listTables(HConnectionManager.java:2237)
>> > at
>> >
>> org.apache.hadoop.hbase.client.HBaseAdmin.listTables(HBaseAdmin.java:317)
>> >
>> >
>> >
>> >
>> > hbase master log:
>> >
>> > -----------------------------
>> >
>> > 2013-10-18 06:19:41,279 DEBUG
>> org.apache.hadoop.hbase.zookeeper.ZKAssign:
>> > master:60000-0x341be88202300ab* Deleting existing unassigned node* for
>> > 0ec3308bd1e2bdd9576b2d60d2eee68e that is in expected state
>> > RS_ZK_REGION_OPENED
>> >
>> > 2013-10-18 06:19:41,279 DEBUG
>> > org.apache.hadoop.hbase.master.AssignmentManager:* Handling
>> > transition=RS_ZK_REGION_OPENING*, s*erver=node0878*.
>> > ic.analyticsworkbench.com,60020,1381883086785,
>> > region=15a4fb29aa1d905b13f33594e50bc8de, which is more than 15 seconds
>> late
>> >
>> > 2013-10-18 06:19:41,280 DEBUG
>> > org.apache.hadoop.hbase.master.AssignmentManager: *Handling
>> > transition=RS_ZK_REGION_OPENING, server=node0898*.
>> > ic.analyticsworkbench.com,60020,1381883200494,
>> > region=1a4c929534e6828c85f22b062f949304, which is more than 15 seconds
>> late
>> >
>> > 2013-10-18 06:19:41,289 DEBUG
>> org.apache.hadoop.hbase.zookeeper.ZKAssign:
>> > master:60000-0x341be88202300ab Successfully *deleted unassigned node
>> *for
>> > region 0ec3308bd1e2bdd9576b2d60d2eee68e in expected state
>> > RS_ZK_REGION_OPENED
>> >
>> > 2013-10-18 06:19:41,289 DEBUG
>> > org.apache.hadoop.hbase.master.AssignmentManager: Handling
>> > transition=RS_ZK_REGION_OPENING, server=
>> node0693.ic.analyticsworkbench.com,60020,1381881773670,
>> > region=d47bfe1af0051c405de295a51c1c6e63, which is more than 15 seconds
>> late
>> >
>> >
>> >
>> > We also try to "list" in hbase shell,it also failed:
>> >
>> > The hbase “list” got error as:
>> >
>> > ------------------------------------------
>> >
>> >
>> >
>> > hbase(main):001:0> list
>> >
>> > TABLE
>> >
>> >
>> >
>> >
>> > ERROR: java.lang.reflect.UndeclaredThrowableException: Call to
>> > node0997.ic.analyticsworkbench.com/10.1.50.17:60000 failed on socket
>> > timeout exception: java.net.SocketTimeoutException: 120000 millis
>> timeout
>> > while waiting for channel to be ready for read. ch :
>> > java.nio.channels.SocketChannel[connected local=/10.1.50.15:45726remote=
>> > node0997.ic.analyticsworkbench.com/10.1.50.17:60000]
>> >
>> >
>> >
>> >
>> > Cheers,
>> > -----
>> > Big Data - Big Wisdom - Big Value
>> > --------------
>> > Michelle Zhang (Li Ping Zhang)
>> >
>>
>
>
Re: HConnectionImplementation.listTables and list table exception
Posted by Ted Yu <yu...@gmail.com>.
bq. our hbase version was 0.94.0 and hadoop version was 1.0.3
If possible, consider upgrading HBase to 0.94.12 and Hadoop to 1.2.1
Cheers
On Wed, Oct 23, 2013 at 7:18 AM, 张莉苹 <zl...@gmail.com> wrote:
> Hello Ted, and Julian,
>
> It seemed I didn't receive your mail from my gmail inbox. It was strange.
> Anyhow I found your reply by google search. ：）
>
> I'll answer your questions here, and thanks very much for your reply.
>
>
> From *Ted*:
>
> What version of HBase and Hadoop are you using ?
>
> >> our hbase version was 0.94.0 and hadoop version was 1.0.3
>
> Can you show us more of the master log ?
>
> >> I'm so sorry, the master log could not be accessed. The environment was
> only used by me for a short while.
>
>
> From * Michael <
> http://www.mail-archive.com/search?l=user@hbase.apache.org&q=from:%22Michael+Segel%22
> >:*
>
>
> Why 9 zookeepers?
>
> >> the previous zookeeper number was 5, we also thought the number of
> zookeeper was too small, so we increased it into 9, but it still failed.
>
>
> From *Julian*:
>
>
> Hello Michelle,
> How many regions totally are there in your 600 nodes cluster? Looks
> like many of them are pending for open and being assigned to region
> servers.
> Can you see many items under zookeeper dir /hbase/unassigned?
>
> >> we had 60K regions, the env could not be accessed.
>
> You would like to refer
> http://blog.sina.com.cn/s/blog_4a1f59bf01018tu4.html?
>
> >> kind of help, thanks!
>
>
>
>
>
> Cheers,
> -----
> Big Data - Big Wisdom - Big Value
> --------------
> Michelle Zhang (Li Ping Zhang)
>
>
> 2013/10/23 张莉苹 <zl...@gmail.com>
>
> > Dear HBase dev and users,
> >
> > Did you meet this
> >
> "org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.listTables"
> > issue?
> >
> > We setup a 600 nodes cluster, 9 zookeeper nodes to load data into hbase,
> > but it seemed hbase master was busy handling transition with zookeeper,
> > and hbase “list” could not get response. The hbase table was created but
> > it didn't do any insert.
> >
> > Do you have any idea of the root cause and how to fix it? :)Highly
> > appreciate for your answers!
> >
> >
> >
> > Here is the exception stack:
> > ---------------------------------------------------
> > java.lang.reflect.UndeclaredThrowableException
> > at $Proxy7.getHTableDescriptors(Unknown Source)
> > at
> >
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.listTables(HConnectionManager.java:2237)
> > at
> > org.apache.hadoop.hbase.client.HBaseAdmin.listTables(HBaseAdmin.java:317)
> >
> >
> >
> >
> > hbase master log:
> >
> > -----------------------------
> >
> > 2013-10-18 06:19:41,279 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
> > master:60000-0x341be88202300ab* Deleting existing unassigned node* for
> > 0ec3308bd1e2bdd9576b2d60d2eee68e that is in expected state
> > RS_ZK_REGION_OPENED
> >
> > 2013-10-18 06:19:41,279 DEBUG
> > org.apache.hadoop.hbase.master.AssignmentManager:* Handling
> > transition=RS_ZK_REGION_OPENING*, s*erver=node0878*.
> > ic.analyticsworkbench.com,60020,1381883086785,
> > region=15a4fb29aa1d905b13f33594e50bc8de, which is more than 15 seconds
> late
> >
> > 2013-10-18 06:19:41,280 DEBUG
> > org.apache.hadoop.hbase.master.AssignmentManager: *Handling
> > transition=RS_ZK_REGION_OPENING, server=node0898*.
> > ic.analyticsworkbench.com,60020,1381883200494,
> > region=1a4c929534e6828c85f22b062f949304, which is more than 15 seconds
> late
> >
> > 2013-10-18 06:19:41,289 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
> > master:60000-0x341be88202300ab Successfully *deleted unassigned node *for
> > region 0ec3308bd1e2bdd9576b2d60d2eee68e in expected state
> > RS_ZK_REGION_OPENED
> >
> > 2013-10-18 06:19:41,289 DEBUG
> > org.apache.hadoop.hbase.master.AssignmentManager: Handling
> > transition=RS_ZK_REGION_OPENING, server=
> node0693.ic.analyticsworkbench.com,60020,1381881773670,
> > region=d47bfe1af0051c405de295a51c1c6e63, which is more than 15 seconds
> late
> >
> >
> >
> > We also try to "list" in hbase shell,it also failed:
> >
> > The hbase “list” got error as:
> >
> > ------------------------------------------
> >
> >
> >
> > hbase(main):001:0> list
> >
> > TABLE
> >
> >
> >
> >
> > ERROR: java.lang.reflect.UndeclaredThrowableException: Call to
> > node0997.ic.analyticsworkbench.com/10.1.50.17:60000 failed on socket
> > timeout exception: java.net.SocketTimeoutException: 120000 millis timeout
> > while waiting for channel to be ready for read. ch :
> > java.nio.channels.SocketChannel[connected local=/10.1.50.15:45726remote=
> > node0997.ic.analyticsworkbench.com/10.1.50.17:60000]
> >
> >
> >
> >
> > Cheers,
> > -----
> > Big Data - Big Wisdom - Big Value
> > --------------
> > Michelle Zhang (Li Ping Zhang)
> >
>
Re: HConnectionImplementation.listTables and list table exception
Posted by Ted Yu <yu...@gmail.com>.
bq. our hbase version was 0.94.0 and hadoop version was 1.0.3
If possible, consider upgrading HBase to 0.94.12 and Hadoop to 1.2.1
Cheers
On Wed, Oct 23, 2013 at 7:18 AM, 张莉苹 <zl...@gmail.com> wrote:
> Hello Ted, and Julian,
>
> It seemed I didn't receive your mail from my gmail inbox. It was strange.
> Anyhow I found your reply by google search. ：）
>
> I'll answer your questions here, and thanks very much for your reply.
>
>
> From *Ted*:
>
> What version of HBase and Hadoop are you using ?
>
> >> our hbase version was 0.94.0 and hadoop version was 1.0.3
>
> Can you show us more of the master log ?
>
> >> I'm so sorry, the master log could not be accessed. The environment was
> only used by me for a short while.
>
>
> From * Michael <
> http://www.mail-archive.com/search?l=user@hbase.apache.org&q=from:%22Michael+Segel%22
> >:*
>
>
> Why 9 zookeepers?
>
> >> the previous zookeeper number was 5, we also thought the number of
> zookeeper was too small, so we increased it into 9, but it still failed.
>
>
> From *Julian*:
>
>
> Hello Michelle,
> How many regions totally are there in your 600 nodes cluster? Looks
> like many of them are pending for open and being assigned to region
> servers.
> Can you see many items under zookeeper dir /hbase/unassigned?
>
> >> we had 60K regions, the env could not be accessed.
>
> You would like to refer
> http://blog.sina.com.cn/s/blog_4a1f59bf01018tu4.html?
>
> >> kind of help, thanks!
>
>
>
>
>
> Cheers,
> -----
> Big Data - Big Wisdom - Big Value
> --------------
> Michelle Zhang (Li Ping Zhang)
>
>
> 2013/10/23 张莉苹 <zl...@gmail.com>
>
> > Dear HBase dev and users,
> >
> > Did you meet this
> >
> "org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.listTables"
> > issue?
> >
> > We setup a 600 nodes cluster, 9 zookeeper nodes to load data into hbase,
> > but it seemed hbase master was busy handling transition with zookeeper,
> > and hbase “list” could not get response. The hbase table was created but
> > it didn't do any insert.
> >
> > Do you have any idea of the root cause and how to fix it? :)Highly
> > appreciate for your answers!
> >
> >
> >
> > Here is the exception stack:
> > ---------------------------------------------------
> > java.lang.reflect.UndeclaredThrowableException
> > at $Proxy7.getHTableDescriptors(Unknown Source)
> > at
> >
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.listTables(HConnectionManager.java:2237)
> > at
> > org.apache.hadoop.hbase.client.HBaseAdmin.listTables(HBaseAdmin.java:317)
> >
> >
> >
> >
> > hbase master log:
> >
> > -----------------------------
> >
> > 2013-10-18 06:19:41,279 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
> > master:60000-0x341be88202300ab* Deleting existing unassigned node* for
> > 0ec3308bd1e2bdd9576b2d60d2eee68e that is in expected state
> > RS_ZK_REGION_OPENED
> >
> > 2013-10-18 06:19:41,279 DEBUG
> > org.apache.hadoop.hbase.master.AssignmentManager:* Handling
> > transition=RS_ZK_REGION_OPENING*, s*erver=node0878*.
> > ic.analyticsworkbench.com,60020,1381883086785,
> > region=15a4fb29aa1d905b13f33594e50bc8de, which is more than 15 seconds
> late
> >
> > 2013-10-18 06:19:41,280 DEBUG
> > org.apache.hadoop.hbase.master.AssignmentManager: *Handling
> > transition=RS_ZK_REGION_OPENING, server=node0898*.
> > ic.analyticsworkbench.com,60020,1381883200494,
> > region=1a4c929534e6828c85f22b062f949304, which is more than 15 seconds
> late
> >
> > 2013-10-18 06:19:41,289 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
> > master:60000-0x341be88202300ab Successfully *deleted unassigned node *for
> > region 0ec3308bd1e2bdd9576b2d60d2eee68e in expected state
> > RS_ZK_REGION_OPENED
> >
> > 2013-10-18 06:19:41,289 DEBUG
> > org.apache.hadoop.hbase.master.AssignmentManager: Handling
> > transition=RS_ZK_REGION_OPENING, server=
> node0693.ic.analyticsworkbench.com,60020,1381881773670,
> > region=d47bfe1af0051c405de295a51c1c6e63, which is more than 15 seconds
> late
> >
> >
> >
> > We also try to "list" in hbase shell,it also failed:
> >
> > The hbase “list” got error as:
> >
> > ------------------------------------------
> >
> >
> >
> > hbase(main):001:0> list
> >
> > TABLE
> >
> >
> >
> >
> > ERROR: java.lang.reflect.UndeclaredThrowableException: Call to
> > node0997.ic.analyticsworkbench.com/10.1.50.17:60000 failed on socket
> > timeout exception: java.net.SocketTimeoutException: 120000 millis timeout
> > while waiting for channel to be ready for read. ch :
> > java.nio.channels.SocketChannel[connected local=/10.1.50.15:45726remote=
> > node0997.ic.analyticsworkbench.com/10.1.50.17:60000]
> >
> >
> >
> >
> > Cheers,
> > -----
> > Big Data - Big Wisdom - Big Value
> > --------------
> > Michelle Zhang (Li Ping Zhang)
> >
>
Re: HConnectionImplementation.listTables and list table exception
Posted by 张莉苹 <zl...@gmail.com>.
Hello Ted, and Julian,
It seemed I didn't receive your mail from my gmail inbox. It was strange.
Anyhow I found your reply by google search. ：）
I'll answer your questions here, and thanks very much for your reply.
>From *Ted*:
What version of HBase and Hadoop are you using ?
>> our hbase version was 0.94.0 and hadoop version was 1.0.3
Can you show us more of the master log ?
>> I'm so sorry, the master log could not be accessed. The environment was only used by me for a short while.
>From * Michael <http://www.mail-archive.com/search?l=user@hbase.apache.org&q=from:%22Michael+Segel%22>:*
Why 9 zookeepers?
>> the previous zookeeper number was 5, we also thought the number of zookeeper was too small, so we increased it into 9, but it still failed.
>From *Julian*:
Hello Michelle,
How many regions totally are there in your 600 nodes cluster? Looks
like many of them are pending for open and being assigned to region
servers.
Can you see many items under zookeeper dir /hbase/unassigned?
>> we had 60K regions, the env could not be accessed.
You would like to refer http://blog.sina.com.cn/s/blog_4a1f59bf01018tu4.html?
>> kind of help, thanks!
Cheers,
-----
Big Data - Big Wisdom - Big Value
--------------
Michelle Zhang (Li Ping Zhang)
2013/10/23 张莉苹 <zl...@gmail.com>
> Dear HBase dev and users,
>
> Did you meet this
> "org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.listTables"
> issue?
>
> We setup a 600 nodes cluster, 9 zookeeper nodes to load data into hbase,
> but it seemed hbase master was busy handling transition with zookeeper,
> and hbase “list” could not get response. The hbase table was created but
> it didn't do any insert.
>
> Do you have any idea of the root cause and how to fix it? :)Highly
> appreciate for your answers!
>
>
>
> Here is the exception stack:
> ---------------------------------------------------
> java.lang.reflect.UndeclaredThrowableException
> at $Proxy7.getHTableDescriptors(Unknown Source)
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.listTables(HConnectionManager.java:2237)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.listTables(HBaseAdmin.java:317)
>
>
>
>
> hbase master log:
>
> -----------------------------
>
> 2013-10-18 06:19:41,279 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
> master:60000-0x341be88202300ab* Deleting existing unassigned node* for
> 0ec3308bd1e2bdd9576b2d60d2eee68e that is in expected state
> RS_ZK_REGION_OPENED
>
> 2013-10-18 06:19:41,279 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager:* Handling
> transition=RS_ZK_REGION_OPENING*, s*erver=node0878*.
> ic.analyticsworkbench.com,60020,1381883086785,
> region=15a4fb29aa1d905b13f33594e50bc8de, which is more than 15 seconds late
>
> 2013-10-18 06:19:41,280 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: *Handling
> transition=RS_ZK_REGION_OPENING, server=node0898*.
> ic.analyticsworkbench.com,60020,1381883200494,
> region=1a4c929534e6828c85f22b062f949304, which is more than 15 seconds late
>
> 2013-10-18 06:19:41,289 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
> master:60000-0x341be88202300ab Successfully *deleted unassigned node *for
> region 0ec3308bd1e2bdd9576b2d60d2eee68e in expected state
> RS_ZK_REGION_OPENED
>
> 2013-10-18 06:19:41,289 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Handling
> transition=RS_ZK_REGION_OPENING, server=node0693.ic.analyticsworkbench.com,60020,1381881773670,
> region=d47bfe1af0051c405de295a51c1c6e63, which is more than 15 seconds late
>
>
>
> We also try to "list" in hbase shell,it also failed:
>
> The hbase “list” got error as:
>
> ------------------------------------------
>
>
>
> hbase(main):001:0> list
>
> TABLE
>
>
>
>
> ERROR: java.lang.reflect.UndeclaredThrowableException: Call to
> node0997.ic.analyticsworkbench.com/10.1.50.17:60000 failed on socket
> timeout exception: java.net.SocketTimeoutException: 120000 millis timeout
> while waiting for channel to be ready for read. ch :
> java.nio.channels.SocketChannel[connected local=/10.1.50.15:45726 remote=
> node0997.ic.analyticsworkbench.com/10.1.50.17:60000]
>
>
>
>
> Cheers,
> -----
> Big Data - Big Wisdom - Big Value
> --------------
> Michelle Zhang (Li Ping Zhang)
>
Re: HConnectionImplementation.listTables and list table exception
Posted by 张莉苹 <zl...@gmail.com>.
Hello Ted, and Julian,
It seemed I didn't receive your mail from my gmail inbox. It was strange.
Anyhow I found your reply by google search. ：）
I'll answer your questions here, and thanks very much for your reply.
>From *Ted*:
What version of HBase and Hadoop are you using ?
>> our hbase version was 0.94.0 and hadoop version was 1.0.3
Can you show us more of the master log ?
>> I'm so sorry, the master log could not be accessed. The environment was only used by me for a short while.
>From * Michael <http://www.mail-archive.com/search?l=user@hbase.apache.org&q=from:%22Michael+Segel%22>:*
Why 9 zookeepers?
>> the previous zookeeper number was 5, we also thought the number of zookeeper was too small, so we increased it into 9, but it still failed.
>From *Julian*:
Hello Michelle,
How many regions totally are there in your 600 nodes cluster? Looks
like many of them are pending for open and being assigned to region
servers.
Can you see many items under zookeeper dir /hbase/unassigned?
>> we had 60K regions, the env could not be accessed.
You would like to refer http://blog.sina.com.cn/s/blog_4a1f59bf01018tu4.html?
>> kind of help, thanks!
Cheers,
-----
Big Data - Big Wisdom - Big Value
--------------
Michelle Zhang (Li Ping Zhang)
2013/10/23 张莉苹 <zl...@gmail.com>
> Dear HBase dev and users,
>
> Did you meet this
> "org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.listTables"
> issue?
>
> We setup a 600 nodes cluster, 9 zookeeper nodes to load data into hbase,
> but it seemed hbase master was busy handling transition with zookeeper,
> and hbase “list” could not get response. The hbase table was created but
> it didn't do any insert.
>
> Do you have any idea of the root cause and how to fix it? :)Highly
> appreciate for your answers!
>
>
>
> Here is the exception stack:
> ---------------------------------------------------
> java.lang.reflect.UndeclaredThrowableException
> at $Proxy7.getHTableDescriptors(Unknown Source)
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.listTables(HConnectionManager.java:2237)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.listTables(HBaseAdmin.java:317)
>
>
>
>
> hbase master log:
>
> -----------------------------
>
> 2013-10-18 06:19:41,279 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
> master:60000-0x341be88202300ab* Deleting existing unassigned node* for
> 0ec3308bd1e2bdd9576b2d60d2eee68e that is in expected state
> RS_ZK_REGION_OPENED
>
> 2013-10-18 06:19:41,279 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager:* Handling
> transition=RS_ZK_REGION_OPENING*, s*erver=node0878*.
> ic.analyticsworkbench.com,60020,1381883086785,
> region=15a4fb29aa1d905b13f33594e50bc8de, which is more than 15 seconds late
>
> 2013-10-18 06:19:41,280 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: *Handling
> transition=RS_ZK_REGION_OPENING, server=node0898*.
> ic.analyticsworkbench.com,60020,1381883200494,
> region=1a4c929534e6828c85f22b062f949304, which is more than 15 seconds late
>
> 2013-10-18 06:19:41,289 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
> master:60000-0x341be88202300ab Successfully *deleted unassigned node *for
> region 0ec3308bd1e2bdd9576b2d60d2eee68e in expected state
> RS_ZK_REGION_OPENED
>
> 2013-10-18 06:19:41,289 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Handling
> transition=RS_ZK_REGION_OPENING, server=node0693.ic.analyticsworkbench.com,60020,1381881773670,
> region=d47bfe1af0051c405de295a51c1c6e63, which is more than 15 seconds late
>
>
>
> We also try to "list" in hbase shell,it also failed:
>
> The hbase “list” got error as:
>
> ------------------------------------------
>
>
>
> hbase(main):001:0> list
>
> TABLE
>
>
>
>
> ERROR: java.lang.reflect.UndeclaredThrowableException: Call to
> node0997.ic.analyticsworkbench.com/10.1.50.17:60000 failed on socket
> timeout exception: java.net.SocketTimeoutException: 120000 millis timeout
> while waiting for channel to be ready for read. ch :
> java.nio.channels.SocketChannel[connected local=/10.1.50.15:45726 remote=
> node0997.ic.analyticsworkbench.com/10.1.50.17:60000]
>
>
>
>
> Cheers,
> -----
> Big Data - Big Wisdom - Big Value
> --------------
> Michelle Zhang (Li Ping Zhang)
>