You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by NNever <nn...@gmail.com> on 2012/02/20 15:03:33 UTC

HBase0.92: In Coprocessor's prePut method, muilt put on same HTable cause HBase crash

Hi~

On HBase0.92.

I write a coprosessor on table 'File'.In the prePut method,  I put several
new rows on File HTable instance. As u know , this op will trigger prePut
again.
I use this logic to realize something like "copy a File and auto copy its
all level's subfiles" .
The code seems no wrong, but everytime i trigger this coprocessor, the
HBase stuck and couldn't shutdown.
(through the log I found the coprocssor's code fall into dead-loop, but the
prePut's code don't have any place may cause dead-loop)

I try lot of times and finally get the reason: I cache the File's HTable
instance and use it like this

*private HTableInterface htFile = null;*
*...*
*@Override*
*public void prePut(...){*

*...*
*if(null == htFile){*
*   htFile = e.getEnvironment().getTable(Bytes.toBytes("File"));*
*}*
*...*
*htFile.put(some list of new puts);*
*...*

*}*

if I change the code and get HTable's instance everytime, the problem fixed.
But I want to know why* using the same Instance of HTable to put puts in
same prePut method may cause coprocessor dead-loop*?
(it just trigger prePut with same put again and again util the HBase
stuck...)

Thanks~

Re: HBase0.92: In Coprocessor's prePut method, muilt put on same HTable cause HBase crash

Posted by Stack <st...@duboce.net>.
On Mon, Feb 20, 2012 at 6:03 AM, NNever <nn...@gmail.com> wrote:
> if I change the code and get HTable's instance everytime, the problem fixed.
> But I want to know why* using the same Instance of HTable to put puts in
> same prePut method may cause coprocessor dead-loop*?
> (it just trigger prePut with same put again and again util the HBase
> stuck...)
>

You must be deadlocking.  Thread dump it when its hung up to see why.
St.Ack

Re: "this file is already being created by NN_Recovery" error

Posted by Stack <st...@duboce.net>.
On Mon, Feb 20, 2012 at 6:52 AM, Agarwal, Saurabh
<sa...@citi.com> wrote:
> Hi,
>
> I am getting the following error and hbase shell is hanging and we are not able to connect hbase. Hadoop filesystem is fine and map reduce jobs are running ok.
>
> Here is the warning message we are getting in the master log file.
> for lease recovery on hdfs://<servername>:54310/user/root/hbase/.logs/<servername>,60020,1329596937708/<servername>%3A60020.1329607742925:org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException: failed to create file /user/root/hbase/.logs/<>,60020,1329596937708/<servername>%3A60020.1329607742925 for DFSClient_hb_m_:60000_1329688357813 on client <>, because this file is already being created by NN_Recovery on <>

What version of hadoop?  It looks like a version that does not support
the short-cut 'lease robbing'.
St.Ack

"this file is already being created by NN_Recovery" error

Posted by "Agarwal, Saurabh " <sa...@citi.com>.
Hi,

I am getting the following error and hbase shell is hanging and we are not able to connect hbase. Hadoop filesystem is fine and map reduce jobs are running ok.

Here is the warning message we are getting in the master log file.
for lease recovery on hdfs://<servername>:54310/user/root/hbase/.logs/<servername>,60020,1329596937708/<servername>%3A60020.1329607742925:org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException: failed to create file /user/root/hbase/.logs/<>,60020,1329596937708/<servername>%3A60020.1329607742925 for DFSClient_hb_m_:60000_1329688357813 on client <>, because this file is already being created by NN_Recovery on <>
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLeaseInternal(FSNamesystem.java:1202)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:1157)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.recoverLease(NameNode.java:404)
        at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:961)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:957)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:955)

Any help is appreciated,
Thanks,
Saurabh.
-----Original Message-----
From: NNever [mailto:nneverwei@gmail.com]
Sent: Monday, February 20, 2012 9:04 AM
To: user@hbase.apache.org
Subject: HBase0.92: In Coprocessor's prePut method, muilt put on same HTable cause HBase crash

Hi~

On HBase0.92.

I write a coprosessor on table 'File'.In the prePut method,  I put several
new rows on File HTable instance. As u know , this op will trigger prePut
again.
I use this logic to realize something like "copy a File and auto copy its
all level's subfiles" .
The code seems no wrong, but everytime i trigger this coprocessor, the
HBase stuck and couldn't shutdown.
(through the log I found the coprocssor's code fall into dead-loop, but the
prePut's code don't have any place may cause dead-loop)

I try lot of times and finally get the reason: I cache the File's HTable
instance and use it like this

*private HTableInterface htFile = null;*
*...*
*@Override*
*public void prePut(...){*

*...*
*if(null == htFile){*
*   htFile = e.getEnvironment().getTable(Bytes.toBytes("File"));*
*}*
*...*
*htFile.put(some list of new puts);*
*...*

*}*

if I change the code and get HTable's instance everytime, the problem fixed.
But I want to know why* using the same Instance of HTable to put puts in
same prePut method may cause coprocessor dead-loop*?
(it just trigger prePut with same put again and again util the HBase
stuck...)

Thanks~