You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Prasan Ary <vo...@yahoo.com> on 2008/02/19 18:54:34 UTC
hbase regionserver fails
Hi all,
We are running tests with 2 regionservers running hbase on hadoop.
After a couple of tests involving creating a new table and inserting rows in it, we notice that one of the regionservers fails with following message:
2008-02-19 10:49:10,424 WARN org.apache.hadoop.fs.DFSClient: Problem renewing lease for DFSClient_-1702284126: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(Unknown Source)
at java.net.PlainSocketImpl.connectToAddress(Unknown Source)
at java.net.PlainSocketImpl.connect(Unknown Source)
at java.net.SocksSocketImpl.connect(Unknown Source)
at java.net.Socket.connect(Unknown Source)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:161)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:578)
at org.apache.hadoop.ipc.Client.call(Client.java:501)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:198)
at org.apache.hadoop.dfs.$Proxy1.renewLease(Unknown Source)
at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
at org.apache.hadoop.dfs.$Proxy1.renewLease(Unknown Source)
at org.apache.hadoop.dfs.DFSClient$LeaseChecker.run(DFSClient.java:588)
at java.lang.Thread.run(Unknown Source)
I have attached the full text of log with this mail.
While one of the regionserver fails, I notice that performance improves by multiple folds for the same test involving table creation and row insertion.
For eg:
When both regionservers were running: Insertion of 50000 rows in a newly created table took 106 seconds.
After one of the regionservers failed: Insertion of 50000 rows in a newly created table took 23887 seconds.
Any help/enlightment is appreciated.
V.
---------------------------------
Be a better friend, newshound, and know-it-all with Yahoo! Mobile. Try it now.
Re: hbase regionserver fails
Posted by stack <st...@duboce.net>.
Prasan: Thanks for sending logs (In future would suggest you run your
cluster w/ DEBUG and that you tar.gz all files together when putting
them up on yousendit).
Looking in 241's log, I see this:
2008-02-19 11:26:29,156 INFO org.apache.hadoop.hbase.HLog: new log
writer created at
hdfs://192.168.0.240:54310/hbase/log_192.168.0.241_1203440388143_60020/hlog.dat.018
2008-02-19 11:27:21,778 FATAL org.apache.hadoop.hbase.HRegionServer:
unable to report to master for 45266 milliseconds - aborting server
2008-02-19 11:27:21,996 INFO org.apache.hadoop.hbase.Leases:
regionserver/0:0:0:0:0:0:0:0:60020 closing leases
It doesn't say why its unable to reach the master. Anything in the
accompanying out file for the regionserver? Or was this machine hung
doing something else at the time? It could be hbase-25. Is it easy
for you to reproduce? If so, I'd be interested in DEBUG level logs (See
FAQ for how to do DEBUG).
Thanks,
St.Ack
Prasan Ary wrote:
> Yes the master is at 192.168.0.240:54310 .
>
> Two regionservers are at:
> 192.168.0.240:60020
> 192.168.0.241:60020
>
> As you can see, master machine also serves as one of the regionservers.
>
> It did find the master initially . Both the regionservers were seen listed on the web interface before one regionserver failed.
>
> We are running Hadoop 0.16.0 , Hbase 0.0.2
>
> Please download the regionserver logs and master server log from links below.
> http://download.yousendit.com/B57154F92047EB51
> http://download.yousendit.com/2FC1C030351001DA
> http://download.yousendit.com/3F839E72723DE7D2
>
>
>
> stack <st...@duboce.net> wrote:
> Thanks for the logs.
>
> Regionserver can't find the master. Is master at 192.168.0.240:54310?
> Did it ever find the master (The log is not complete. Next time, send
> all logs if you don't mind (master and regionservers).
>
> Are the two regionservers and master all up on a single machine?
>
> What version are you running?
>
> Unrelated, sort of, you are running as root. You should try and run as
> yourself instead.
>
> St.Ack
>
>
> Prasan Ary wrote:
>
>> Just in case mail from YouSendIt.com got filtered off, here is the link to download log file :
>> http://download.yousendit.com/83A7C5D4532F5B09
>>
>>
>> stack wrote:
>> Log did not come through Prasan. Can you put it somewhere where we can
>> get to it (or use something like yousendit.com.
>> Thanks,
>> St.Ack
>>
>> Prasan Ary wrote:
>>
>>
>>> Hi all,
>>> We are running tests with 2 regionservers running hbase on hadoop.
>>>
>>> After a couple of tests involving creating a new table and inserting
>>> rows in it, we notice that one of the regionservers fails with
>>> following message:
>>>
>>> 2008-02-19 10:49:10,424 WARN org.apache.hadoop.fs.DFSClient: Problem
>>> renewing lease for DFSClient_-1702284126: java.net.ConnectException:
>>> Connection refused
>>> at java.net.PlainSocketImpl.socketConnect(Native Method)
>>> at java.net.PlainSocketImpl.doConnect(Unknown Source)
>>> at java.net.PlainSocketImpl.connectToAddress(Unknown Source)
>>> at java.net.PlainSocketImpl.connect(Unknown Source)
>>> at java.net.SocksSocketImpl.connect(Unknown Source)
>>> at java.net.Socket.connect(Unknown Source)
>>> at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:161)
>>> at org.apache.hadoop.ipc.Client.getConnection(Client.java:578)
>>> at org.apache.hadoop.ipc.Client.call(Client.java:501)
>>> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:198)
>>> at org.apache.hadoop.dfs.$Proxy1.renewLease(Unknown Source)
>>> at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
>>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
>>> at java.lang.reflect.Method.invoke(Unknown Source)
>>> at
>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>>> at
>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>>> at org.apache.hadoop.dfs.$Proxy1.renewLease(Unknown Source)
>>> at org.apache.hadoop.dfs.DFSClient$LeaseChecker.run(DFSClient.java:588)
>>> at java.lang.Thread.run(Unknown Source)
>>>
>>>
>>> I have attached the full text of log with this mail.
>>>
>>> While one of the regionserver fails, I notice that performance
>>> improves by multiple folds for the same test involving table creation
>>> and row insertion.
>>> For eg:
>>> When both regionservers were running: Insertion of 50000 rows in a
>>> newly created table took 106 seconds.
>>> After one of the regionservers failed: Insertion of 50000 rows in a
>>> newly created table took 23887 seconds.
>>>
>>>
>>> Any help/enlightment is appreciated.
>>>
>>> V.
>>>
>>> ------------------------------------------------------------------------
>>> Be a better friend, newshound, and know-it-all with Yahoo! Mobile. Try
>>> it now.
>>>
>>>
>>>
>>
>>
>>
>> ---------------------------------
>> Looking for last minute shopping deals? Find them fast with Yahoo! Search.
>>
>>
>
>
>
>
> ---------------------------------
> Never miss a thing. Make Yahoo your homepage.
>
Re: hbase regionserver fails
Posted by Prasan Ary <vo...@yahoo.com>.
Yes the master is at 192.168.0.240:54310 .
Two regionservers are at:
192.168.0.240:60020
192.168.0.241:60020
As you can see, master machine also serves as one of the regionservers.
It did find the master initially . Both the regionservers were seen listed on the web interface before one regionserver failed.
We are running Hadoop 0.16.0 , Hbase 0.0.2
Please download the regionserver logs and master server log from links below.
http://download.yousendit.com/B57154F92047EB51
http://download.yousendit.com/2FC1C030351001DA
http://download.yousendit.com/3F839E72723DE7D2
stack <st...@duboce.net> wrote:
Thanks for the logs.
Regionserver can't find the master. Is master at 192.168.0.240:54310?
Did it ever find the master (The log is not complete. Next time, send
all logs if you don't mind (master and regionservers).
Are the two regionservers and master all up on a single machine?
What version are you running?
Unrelated, sort of, you are running as root. You should try and run as
yourself instead.
St.Ack
Prasan Ary wrote:
> Just in case mail from YouSendIt.com got filtered off, here is the link to download log file :
> http://download.yousendit.com/83A7C5D4532F5B09
>
>
> stack wrote:
> Log did not come through Prasan. Can you put it somewhere where we can
> get to it (or use something like yousendit.com.
> Thanks,
> St.Ack
>
> Prasan Ary wrote:
>
>> Hi all,
>> We are running tests with 2 regionservers running hbase on hadoop.
>>
>> After a couple of tests involving creating a new table and inserting
>> rows in it, we notice that one of the regionservers fails with
>> following message:
>>
>> 2008-02-19 10:49:10,424 WARN org.apache.hadoop.fs.DFSClient: Problem
>> renewing lease for DFSClient_-1702284126: java.net.ConnectException:
>> Connection refused
>> at java.net.PlainSocketImpl.socketConnect(Native Method)
>> at java.net.PlainSocketImpl.doConnect(Unknown Source)
>> at java.net.PlainSocketImpl.connectToAddress(Unknown Source)
>> at java.net.PlainSocketImpl.connect(Unknown Source)
>> at java.net.SocksSocketImpl.connect(Unknown Source)
>> at java.net.Socket.connect(Unknown Source)
>> at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:161)
>> at org.apache.hadoop.ipc.Client.getConnection(Client.java:578)
>> at org.apache.hadoop.ipc.Client.call(Client.java:501)
>> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:198)
>> at org.apache.hadoop.dfs.$Proxy1.renewLease(Unknown Source)
>> at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
>> at java.lang.reflect.Method.invoke(Unknown Source)
>> at
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>> at
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>> at org.apache.hadoop.dfs.$Proxy1.renewLease(Unknown Source)
>> at org.apache.hadoop.dfs.DFSClient$LeaseChecker.run(DFSClient.java:588)
>> at java.lang.Thread.run(Unknown Source)
>>
>>
>> I have attached the full text of log with this mail.
>>
>> While one of the regionserver fails, I notice that performance
>> improves by multiple folds for the same test involving table creation
>> and row insertion.
>> For eg:
>> When both regionservers were running: Insertion of 50000 rows in a
>> newly created table took 106 seconds.
>> After one of the regionservers failed: Insertion of 50000 rows in a
>> newly created table took 23887 seconds.
>>
>>
>> Any help/enlightment is appreciated.
>>
>> V.
>>
>> ------------------------------------------------------------------------
>> Be a better friend, newshound, and know-it-all with Yahoo! Mobile. Try
>> it now.
>>
>>
>
>
>
>
>
> ---------------------------------
> Looking for last minute shopping deals? Find them fast with Yahoo! Search.
>
---------------------------------
Never miss a thing. Make Yahoo your homepage.
Re: hbase regionserver fails
Posted by stack <st...@duboce.net>.
Thanks for the logs.
Regionserver can't find the master. Is master at 192.168.0.240:54310?
Did it ever find the master (The log is not complete. Next time, send
all logs if you don't mind (master and regionservers).
Are the two regionservers and master all up on a single machine?
What version are you running?
Unrelated, sort of, you are running as root. You should try and run as
yourself instead.
St.Ack
Prasan Ary wrote:
> Just in case mail from YouSendIt.com got filtered off, here is the link to download log file :
> http://download.yousendit.com/83A7C5D4532F5B09
>
>
> stack <st...@duboce.net> wrote:
> Log did not come through Prasan. Can you put it somewhere where we can
> get to it (or use something like yousendit.com.
> Thanks,
> St.Ack
>
> Prasan Ary wrote:
>
>> Hi all,
>> We are running tests with 2 regionservers running hbase on hadoop.
>>
>> After a couple of tests involving creating a new table and inserting
>> rows in it, we notice that one of the regionservers fails with
>> following message:
>>
>> 2008-02-19 10:49:10,424 WARN org.apache.hadoop.fs.DFSClient: Problem
>> renewing lease for DFSClient_-1702284126: java.net.ConnectException:
>> Connection refused
>> at java.net.PlainSocketImpl.socketConnect(Native Method)
>> at java.net.PlainSocketImpl.doConnect(Unknown Source)
>> at java.net.PlainSocketImpl.connectToAddress(Unknown Source)
>> at java.net.PlainSocketImpl.connect(Unknown Source)
>> at java.net.SocksSocketImpl.connect(Unknown Source)
>> at java.net.Socket.connect(Unknown Source)
>> at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:161)
>> at org.apache.hadoop.ipc.Client.getConnection(Client.java:578)
>> at org.apache.hadoop.ipc.Client.call(Client.java:501)
>> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:198)
>> at org.apache.hadoop.dfs.$Proxy1.renewLease(Unknown Source)
>> at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
>> at java.lang.reflect.Method.invoke(Unknown Source)
>> at
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>> at
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>> at org.apache.hadoop.dfs.$Proxy1.renewLease(Unknown Source)
>> at org.apache.hadoop.dfs.DFSClient$LeaseChecker.run(DFSClient.java:588)
>> at java.lang.Thread.run(Unknown Source)
>>
>>
>> I have attached the full text of log with this mail.
>>
>> While one of the regionserver fails, I notice that performance
>> improves by multiple folds for the same test involving table creation
>> and row insertion.
>> For eg:
>> When both regionservers were running: Insertion of 50000 rows in a
>> newly created table took 106 seconds.
>> After one of the regionservers failed: Insertion of 50000 rows in a
>> newly created table took 23887 seconds.
>>
>>
>> Any help/enlightment is appreciated.
>>
>> V.
>>
>> ------------------------------------------------------------------------
>> Be a better friend, newshound, and know-it-all with Yahoo! Mobile. Try
>> it now.
>>
>>
>
>
>
>
>
> ---------------------------------
> Looking for last minute shopping deals? Find them fast with Yahoo! Search.
>
Re: hbase regionserver fails
Posted by Prasan Ary <vo...@yahoo.com>.
Just in case mail from YouSendIt.com got filtered off, here is the link to download log file :
http://download.yousendit.com/83A7C5D4532F5B09
stack <st...@duboce.net> wrote:
Log did not come through Prasan. Can you put it somewhere where we can
get to it (or use something like yousendit.com.
Thanks,
St.Ack
Prasan Ary wrote:
> Hi all,
> We are running tests with 2 regionservers running hbase on hadoop.
>
> After a couple of tests involving creating a new table and inserting
> rows in it, we notice that one of the regionservers fails with
> following message:
>
> 2008-02-19 10:49:10,424 WARN org.apache.hadoop.fs.DFSClient: Problem
> renewing lease for DFSClient_-1702284126: java.net.ConnectException:
> Connection refused
> at java.net.PlainSocketImpl.socketConnect(Native Method)
> at java.net.PlainSocketImpl.doConnect(Unknown Source)
> at java.net.PlainSocketImpl.connectToAddress(Unknown Source)
> at java.net.PlainSocketImpl.connect(Unknown Source)
> at java.net.SocksSocketImpl.connect(Unknown Source)
> at java.net.Socket.connect(Unknown Source)
> at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:161)
> at org.apache.hadoop.ipc.Client.getConnection(Client.java:578)
> at org.apache.hadoop.ipc.Client.call(Client.java:501)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:198)
> at org.apache.hadoop.dfs.$Proxy1.renewLease(Unknown Source)
> at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
> at java.lang.reflect.Method.invoke(Unknown Source)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> at org.apache.hadoop.dfs.$Proxy1.renewLease(Unknown Source)
> at org.apache.hadoop.dfs.DFSClient$LeaseChecker.run(DFSClient.java:588)
> at java.lang.Thread.run(Unknown Source)
>
>
> I have attached the full text of log with this mail.
>
> While one of the regionserver fails, I notice that performance
> improves by multiple folds for the same test involving table creation
> and row insertion.
> For eg:
> When both regionservers were running: Insertion of 50000 rows in a
> newly created table took 106 seconds.
> After one of the regionservers failed: Insertion of 50000 rows in a
> newly created table took 23887 seconds.
>
>
> Any help/enlightment is appreciated.
>
> V.
>
> ------------------------------------------------------------------------
> Be a better friend, newshound, and know-it-all with Yahoo! Mobile. Try
> it now.
>
---------------------------------
Looking for last minute shopping deals? Find them fast with Yahoo! Search.
Re: hbase regionserver fails
Posted by stack <st...@duboce.net>.
Log did not come through Prasan. Can you put it somewhere where we can
get to it (or use something like yousendit.com.
Thanks,
St.Ack
Prasan Ary wrote:
> Hi all,
> We are running tests with 2 regionservers running hbase on hadoop.
>
> After a couple of tests involving creating a new table and inserting
> rows in it, we notice that one of the regionservers fails with
> following message:
>
> 2008-02-19 10:49:10,424 WARN org.apache.hadoop.fs.DFSClient: Problem
> renewing lease for DFSClient_-1702284126: java.net.ConnectException:
> Connection refused
> at java.net.PlainSocketImpl.socketConnect(Native Method)
> at java.net.PlainSocketImpl.doConnect(Unknown Source)
> at java.net.PlainSocketImpl.connectToAddress(Unknown Source)
> at java.net.PlainSocketImpl.connect(Unknown Source)
> at java.net.SocksSocketImpl.connect(Unknown Source)
> at java.net.Socket.connect(Unknown Source)
> at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:161)
> at org.apache.hadoop.ipc.Client.getConnection(Client.java:578)
> at org.apache.hadoop.ipc.Client.call(Client.java:501)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:198)
> at org.apache.hadoop.dfs.$Proxy1.renewLease(Unknown Source)
> at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
> at java.lang.reflect.Method.invoke(Unknown Source)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> at org.apache.hadoop.dfs.$Proxy1.renewLease(Unknown Source)
> at org.apache.hadoop.dfs.DFSClient$LeaseChecker.run(DFSClient.java:588)
> at java.lang.Thread.run(Unknown Source)
>
>
> I have attached the full text of log with this mail.
>
> While one of the regionserver fails, I notice that performance
> improves by multiple folds for the same test involving table creation
> and row insertion.
> For eg:
> When both regionservers were running: Insertion of 50000 rows in a
> newly created table took 106 seconds.
> After one of the regionservers failed: Insertion of 50000 rows in a
> newly created table took 23887 seconds.
>
>
> Any help/enlightment is appreciated.
>
> V.
>
> ------------------------------------------------------------------------
> Be a better friend, newshound, and know-it-all with Yahoo! Mobile. Try
> it now.
> <http://us.rd.yahoo.com/evt=51733/*http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ%20>
hbase regionserver fails ( correction on metric)
Posted by Prasan Ary <vo...@yahoo.com>.
Sorry, a typo on performance metric. It should read as :
For eg:
When both regionservers were running: Insertion of 50000 rows in a newly created table took 106 seconds.
After one of the regionservers failed: Insertion of 50000 rows in a newly created table took 23.8 seconds.
Thanks.
Prasan Ary <vo...@yahoo.com> wrote:
Hi all,
We are running tests with 2 regionservers running hbase on hadoop.
After a couple of tests involving creating a new table and inserting rows in it, we notice that one of the regionservers fails with following message:
2008-02-19 10:49:10,424 WARN org.apache.hadoop.fs.DFSClient: Problem renewing lease for DFSClient_-1702284126: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(Unknown Source)
at java.net.PlainSocketImpl.connectToAddress(Unknown Source)
at java.net.PlainSocketImpl.connect(Unknown Source)
at java.net.SocksSocketImpl.connect(Unknown Source)
at java.net.Socket.connect(Unknown Source)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:161)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:578)
at org.apache.hadoop.ipc.Client.call(Client.java:501)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:198)
at org.apache.hadoop.dfs.$Proxy1.renewLease(Unknown Source)
at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
at org.apache.hadoop.dfs.$Proxy1.renewLease(Unknown Source)
at org.apache.hadoop.dfs.DFSClient$LeaseChecker.run(DFSClient.java:588)
at java.lang.Thread.run(Unknown Source)
I have attached the full text of log with this mail.
While one of the regionserver fails, I notice that performance improves by multiple folds for the same test involving table creation and row insertion.
For eg:
When both regionservers were running: Insertion of 50000 rows in a newly created table took 106 seconds.
After one of the regionservers failed: Insertion of 50000 rows in a newly created table took 23887 seconds.
Any help/enlightment is appreciated.
V.
---------------------------------
Be a better friend, newshound, and know-it-all with Yahoo! Mobile. Try it now.
---------------------------------
Be a better friend, newshound, and know-it-all with Yahoo! Mobile. Try it now.