You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Tianying Chang <ti...@ebaysf.com> on 2013/05/16 19:12:48 UTC
NullPointerException while loading large amount of new rows into
HBase, exception is thrown when trying to obtain lock for RowKey
Hi,
When our customers(using TSDB) loads large amount of data into HBase, we saw many NullPointerException in the RS logs as below. I checked the source code, it seems when trying to obtain the lock for a rowKey, if the entry for that row already exists and the "waitfoBlock" is false (therefore it won't retry, but just return a NULL value). I can see in doMiniBatchMutation(), the waitForBlock is set to be false. (most other places "waitForBlock" is always set to true).
This exception is thrown from function lockRow(), which has been deprecated. I am not sure why it is deprecated, and what is used to replace it. Is this normal? which implies the HBase should not throw this misleading error message to log. Or should the client call some other API?
Thanks
Tian-Ying
2013-05-14 12:45:30,911 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: Row lock -3430274391270203797 explicitly acquired by client
2013-05-14 12:45:30,911 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":29783,"call":"lockRow([B@339a6a5c, [B@5ebcd87b), rpc version=1, client version=29, methodsFingerPrint=0","client":"10.53.106.37:58892","starttimems":1368560701128,"queuetimems":847,"class":"HRegionServer","responsesize":0,"method":"lockRow"}
2013-05-14 12:46:00,911 ERROR org.apache.hadoop.hbase.regionserver.HRegionServer: Error obtaining row lock (fsOk: true)
java.lang.NullPointerException
at java.util.concurrent.ConcurrentHashMap.put(ConcurrentHashMap.java:881)
at org.apache.hadoop.hbase.regionserver.HRegionServer.addRowLock(HRegionServer.java:2346)
at org.apache.hadoop.hbase.regionserver.HRegionServer.lockRow(HRegionServer.java:2332)
at sun.reflect.GeneratedMethodAccessor156.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:384)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1336)
2013-05-14 12:46:02,514 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server Responder, call get([B@2166c821, {"timeRange":[0,9223372036854775807],"totalColumns":1,"cacheBlocks":true,"families":{"id":["tagv"]},"maxVersions":1,"row":"slcsn-s00314.slc.ebay.com"}), rpc version=1, client version=29, methodsFingerPrint=0 from 10.53.106.37:58892: output error
2013-05-14 12:46:02,514 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server handler 3 on 60020 caught: java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcher.write0(Native Method)
at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29)
RE: NullPointerException while loading large amount of new rows
into HBase, exception is thrown when trying to obtain lock for RowKey
Posted by Tianying Chang <ti...@ebaysf.com>.
Thanks Ram, will try that.
Thanks
Tian-Ying
________________________________________
From: ramkrishna vasudevan [ramkrishna.s.vasudevan@gmail.com]
Sent: Thursday, May 16, 2013 8:49 PM
To: user@hbase.apache.org
Subject: Re: NullPointerException while loading large amount of new rows into HBase, exception is thrown when trying to obtain lock for RowKey
The lockRow and unlockRow have been replaced by checkAndXXX and increment()
apis so that any operation on a particular row can be done automically.
Am not sure of the use case that you are addressing here but i recommend
taking a look at these apis if they solve the problem for you.
RowLocks are prone to more thread contentions and some deadlock situations
when there are lot of threads waiting for the same row lock.
Regards
Ram
On Fri, May 17, 2013 at 12:11 AM, Tianying Chang <ti...@ebaysf.com> wrote:
> FYI, below I quoted the customers' response after I explained the
> NULLException is caused by the row lock. So my question is if this is
> allowed situation for multiple threads/process to compete for the lock, the
> one who did not get should be considered normal and not throwing
> NullPointerException?
>
> Thanks
> Tian-Ying
>
> Quote from customer below:
> "Well yes, not only multiple threads, but multiple processes!
> That’s why we need a lock.
>
> Although I do see some practical problems, and am trying to NOT do the
> same lock from multiple threads within the same process, but it is needed
> to coordinate locks across processes"
>
> ________________________________________
> From: ramkrishna vasudevan [ramkrishna.s.vasudevan@gmail.com]
> Sent: Thursday, May 16, 2013 10:16 AM
> To: user@hbase.apache.org
> Subject: Re: NullPointerException while loading large amount of new rows
> into HBase, exception is thrown when trying to obtain lock for RowKey
>
> Which version of HBase?
>
> Regards
> Ram
>
>
> On Thu, May 16, 2013 at 10:42 PM, Tianying Chang <ti...@ebaysf.com>
> wrote:
>
> > Hi,
> >
> > When our customers(using TSDB) loads large amount of data into HBase, we
> > saw many NullPointerException in the RS logs as below. I checked the
> source
> > code, it seems when trying to obtain the lock for a rowKey, if the entry
> > for that row already exists and the "waitfoBlock" is false (therefore it
> > won't retry, but just return a NULL value). I can see in
> > doMiniBatchMutation(), the waitForBlock is set to be false. (most other
> > places "waitForBlock" is always set to true).
> >
> > This exception is thrown from function lockRow(), which has been
> > deprecated. I am not sure why it is deprecated, and what is used to
> replace
> > it. Is this normal? which implies the HBase should not throw this
> > misleading error message to log. Or should the client call some other
> API?
> >
> > Thanks
> > Tian-Ying
> >
> > 2013-05-14 12:45:30,911 DEBUG
> > org.apache.hadoop.hbase.regionserver.HRegionServer: Row lock
> > -3430274391270203797 explicitly acquired by client
> > 2013-05-14 12:45:30,911 WARN org.apache.hadoop.ipc.HBaseServer:
> > (responseTooSlow): {"processingtimems":29783,"call":"lockRow([B@339a6a5c
> ,
> > [B@5ebcd87b), rpc version=1, client version=29,
> > methodsFingerPrint=0","client":"10.53.106.37:58892
> >
> ","starttimems":1368560701128,"queuetimems":847,"class":"HRegionServer","responsesize":0,"method":"lockRow"}
> > 2013-05-14 12:46:00,911 ERROR
> > org.apache.hadoop.hbase.regionserver.HRegionServer: Error obtaining row
> > lock (fsOk: true)
> > java.lang.NullPointerException
> > at
> > java.util.concurrent.ConcurrentHashMap.put(ConcurrentHashMap.java:881)
> > at
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.addRowLock(HRegionServer.java:2346)
> > at
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.lockRow(HRegionServer.java:2332)
> > at sun.reflect.GeneratedMethodAccessor156.invoke(Unknown
> > Source)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at
> >
> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:384)
> > at
> >
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1336)
> > 2013-05-14 12:46:02,514 WARN org.apache.hadoop.ipc.HBaseServer: IPC
> Server
> > Responder, call get([B@2166c821,
> >
> {"timeRange":[0,9223372036854775807],"totalColumns":1,"cacheBlocks":true,"families":{"id":["tagv"]},"maxVersions":1,"row":"
> > slcsn-s00314.slc.ebay.com"}), rpc version=1, client version=29,
> > methodsFingerPrint=0 from 10.53.106.37:58892: output error
> > 2013-05-14 12:46:02,514 WARN org.apache.hadoop.ipc.HBaseServer: IPC
> Server
> > handler 3 on 60020 caught: java.io.IOException: Connection reset by peer
> > at sun.nio.ch.FileDispatcher.write0(Native Method)
> > at
> > sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29)
> >
>
Re: NullPointerException while loading large amount of new rows into
HBase, exception is thrown when trying to obtain lock for RowKey
Posted by ramkrishna vasudevan <ra...@gmail.com>.
The lockRow and unlockRow have been replaced by checkAndXXX and increment()
apis so that any operation on a particular row can be done automically.
Am not sure of the use case that you are addressing here but i recommend
taking a look at these apis if they solve the problem for you.
RowLocks are prone to more thread contentions and some deadlock situations
when there are lot of threads waiting for the same row lock.
Regards
Ram
On Fri, May 17, 2013 at 12:11 AM, Tianying Chang <ti...@ebaysf.com> wrote:
> FYI, below I quoted the customers' response after I explained the
> NULLException is caused by the row lock. So my question is if this is
> allowed situation for multiple threads/process to compete for the lock, the
> one who did not get should be considered normal and not throwing
> NullPointerException?
>
> Thanks
> Tian-Ying
>
> Quote from customer below:
> "Well yes, not only multiple threads, but multiple processes!
> That’s why we need a lock.
>
> Although I do see some practical problems, and am trying to NOT do the
> same lock from multiple threads within the same process, but it is needed
> to coordinate locks across processes"
>
> ________________________________________
> From: ramkrishna vasudevan [ramkrishna.s.vasudevan@gmail.com]
> Sent: Thursday, May 16, 2013 10:16 AM
> To: user@hbase.apache.org
> Subject: Re: NullPointerException while loading large amount of new rows
> into HBase, exception is thrown when trying to obtain lock for RowKey
>
> Which version of HBase?
>
> Regards
> Ram
>
>
> On Thu, May 16, 2013 at 10:42 PM, Tianying Chang <ti...@ebaysf.com>
> wrote:
>
> > Hi,
> >
> > When our customers(using TSDB) loads large amount of data into HBase, we
> > saw many NullPointerException in the RS logs as below. I checked the
> source
> > code, it seems when trying to obtain the lock for a rowKey, if the entry
> > for that row already exists and the "waitfoBlock" is false (therefore it
> > won't retry, but just return a NULL value). I can see in
> > doMiniBatchMutation(), the waitForBlock is set to be false. (most other
> > places "waitForBlock" is always set to true).
> >
> > This exception is thrown from function lockRow(), which has been
> > deprecated. I am not sure why it is deprecated, and what is used to
> replace
> > it. Is this normal? which implies the HBase should not throw this
> > misleading error message to log. Or should the client call some other
> API?
> >
> > Thanks
> > Tian-Ying
> >
> > 2013-05-14 12:45:30,911 DEBUG
> > org.apache.hadoop.hbase.regionserver.HRegionServer: Row lock
> > -3430274391270203797 explicitly acquired by client
> > 2013-05-14 12:45:30,911 WARN org.apache.hadoop.ipc.HBaseServer:
> > (responseTooSlow): {"processingtimems":29783,"call":"lockRow([B@339a6a5c
> ,
> > [B@5ebcd87b), rpc version=1, client version=29,
> > methodsFingerPrint=0","client":"10.53.106.37:58892
> >
> ","starttimems":1368560701128,"queuetimems":847,"class":"HRegionServer","responsesize":0,"method":"lockRow"}
> > 2013-05-14 12:46:00,911 ERROR
> > org.apache.hadoop.hbase.regionserver.HRegionServer: Error obtaining row
> > lock (fsOk: true)
> > java.lang.NullPointerException
> > at
> > java.util.concurrent.ConcurrentHashMap.put(ConcurrentHashMap.java:881)
> > at
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.addRowLock(HRegionServer.java:2346)
> > at
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.lockRow(HRegionServer.java:2332)
> > at sun.reflect.GeneratedMethodAccessor156.invoke(Unknown
> > Source)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at
> >
> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:384)
> > at
> >
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1336)
> > 2013-05-14 12:46:02,514 WARN org.apache.hadoop.ipc.HBaseServer: IPC
> Server
> > Responder, call get([B@2166c821,
> >
> {"timeRange":[0,9223372036854775807],"totalColumns":1,"cacheBlocks":true,"families":{"id":["tagv"]},"maxVersions":1,"row":"
> > slcsn-s00314.slc.ebay.com"}), rpc version=1, client version=29,
> > methodsFingerPrint=0 from 10.53.106.37:58892: output error
> > 2013-05-14 12:46:02,514 WARN org.apache.hadoop.ipc.HBaseServer: IPC
> Server
> > handler 3 on 60020 caught: java.io.IOException: Connection reset by peer
> > at sun.nio.ch.FileDispatcher.write0(Native Method)
> > at
> > sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29)
> >
>
RE: NullPointerException while loading large amount of new rows
into HBase, exception is thrown when trying to obtain lock for RowKey
Posted by Tianying Chang <ti...@ebaysf.com>.
FYI, below I quoted the customers' response after I explained the NULLException is caused by the row lock. So my question is if this is allowed situation for multiple threads/process to compete for the lock, the one who did not get should be considered normal and not throwing NullPointerException?
Thanks
Tian-Ying
Quote from customer below:
"Well yes, not only multiple threads, but multiple processes!
That’s why we need a lock.
Although I do see some practical problems, and am trying to NOT do the same lock from multiple threads within the same process, but it is needed to coordinate locks across processes"
________________________________________
From: ramkrishna vasudevan [ramkrishna.s.vasudevan@gmail.com]
Sent: Thursday, May 16, 2013 10:16 AM
To: user@hbase.apache.org
Subject: Re: NullPointerException while loading large amount of new rows into HBase, exception is thrown when trying to obtain lock for RowKey
Which version of HBase?
Regards
Ram
On Thu, May 16, 2013 at 10:42 PM, Tianying Chang <ti...@ebaysf.com> wrote:
> Hi,
>
> When our customers(using TSDB) loads large amount of data into HBase, we
> saw many NullPointerException in the RS logs as below. I checked the source
> code, it seems when trying to obtain the lock for a rowKey, if the entry
> for that row already exists and the "waitfoBlock" is false (therefore it
> won't retry, but just return a NULL value). I can see in
> doMiniBatchMutation(), the waitForBlock is set to be false. (most other
> places "waitForBlock" is always set to true).
>
> This exception is thrown from function lockRow(), which has been
> deprecated. I am not sure why it is deprecated, and what is used to replace
> it. Is this normal? which implies the HBase should not throw this
> misleading error message to log. Or should the client call some other API?
>
> Thanks
> Tian-Ying
>
> 2013-05-14 12:45:30,911 DEBUG
> org.apache.hadoop.hbase.regionserver.HRegionServer: Row lock
> -3430274391270203797 explicitly acquired by client
> 2013-05-14 12:45:30,911 WARN org.apache.hadoop.ipc.HBaseServer:
> (responseTooSlow): {"processingtimems":29783,"call":"lockRow([B@339a6a5c,
> [B@5ebcd87b), rpc version=1, client version=29,
> methodsFingerPrint=0","client":"10.53.106.37:58892
> ","starttimems":1368560701128,"queuetimems":847,"class":"HRegionServer","responsesize":0,"method":"lockRow"}
> 2013-05-14 12:46:00,911 ERROR
> org.apache.hadoop.hbase.regionserver.HRegionServer: Error obtaining row
> lock (fsOk: true)
> java.lang.NullPointerException
> at
> java.util.concurrent.ConcurrentHashMap.put(ConcurrentHashMap.java:881)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.addRowLock(HRegionServer.java:2346)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.lockRow(HRegionServer.java:2332)
> at sun.reflect.GeneratedMethodAccessor156.invoke(Unknown
> Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:384)
> at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1336)
> 2013-05-14 12:46:02,514 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server
> Responder, call get([B@2166c821,
> {"timeRange":[0,9223372036854775807],"totalColumns":1,"cacheBlocks":true,"families":{"id":["tagv"]},"maxVersions":1,"row":"
> slcsn-s00314.slc.ebay.com"}), rpc version=1, client version=29,
> methodsFingerPrint=0 from 10.53.106.37:58892: output error
> 2013-05-14 12:46:02,514 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 3 on 60020 caught: java.io.IOException: Connection reset by peer
> at sun.nio.ch.FileDispatcher.write0(Native Method)
> at
> sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29)
>
RE: NullPointerException while loading large amount of new rows
into HBase, exception is thrown when trying to obtain lock for RowKey
Posted by Tianying Chang <ti...@ebaysf.com>.
it is HBase 0.92. The customer is using TSDB and AsyncHBase. I am not sure what their client code is calling exactly. But from the calling stack, it feels it use HTable.lockRow. Is this not recommended? If so, what should they use instead?
Thanks
Tian-Ying
________________________________________
From: ramkrishna vasudevan [ramkrishna.s.vasudevan@gmail.com]
Sent: Thursday, May 16, 2013 10:41 AM
To: user@hbase.apache.org
Subject: Re: NullPointerException while loading large amount of new rows into HBase, exception is thrown when trying to obtain lock for RowKey
Are you trying to get the row lock explicitly ? Using HTable.lockRow?
Regards
Ram
On Thu, May 16, 2013 at 10:46 PM, ramkrishna vasudevan <
ramkrishna.s.vasudevan@gmail.com> wrote:
> Which version of HBase?
>
> Regards
> Ram
>
>
> On Thu, May 16, 2013 at 10:42 PM, Tianying Chang <ti...@ebaysf.com>wrote:
>
>> Hi,
>>
>> When our customers(using TSDB) loads large amount of data into HBase, we
>> saw many NullPointerException in the RS logs as below. I checked the source
>> code, it seems when trying to obtain the lock for a rowKey, if the entry
>> for that row already exists and the "waitfoBlock" is false (therefore it
>> won't retry, but just return a NULL value). I can see in
>> doMiniBatchMutation(), the waitForBlock is set to be false. (most other
>> places "waitForBlock" is always set to true).
>>
>> This exception is thrown from function lockRow(), which has been
>> deprecated. I am not sure why it is deprecated, and what is used to replace
>> it. Is this normal? which implies the HBase should not throw this
>> misleading error message to log. Or should the client call some other API?
>>
>> Thanks
>> Tian-Ying
>>
>> 2013-05-14 12:45:30,911 DEBUG
>> org.apache.hadoop.hbase.regionserver.HRegionServer: Row lock
>> -3430274391270203797 explicitly acquired by client
>> 2013-05-14 12:45:30,911 WARN org.apache.hadoop.ipc.HBaseServer:
>> (responseTooSlow): {"processingtimems":29783,"call":"lockRow([B@339a6a5c,
>> [B@5ebcd87b), rpc version=1, client version=29,
>> methodsFingerPrint=0","client":"10.53.106.37:58892
>> ","starttimems":1368560701128,"queuetimems":847,"class":"HRegionServer","responsesize":0,"method":"lockRow"}
>> 2013-05-14 12:46:00,911 ERROR
>> org.apache.hadoop.hbase.regionserver.HRegionServer: Error obtaining row
>> lock (fsOk: true)
>> java.lang.NullPointerException
>> at
>> java.util.concurrent.ConcurrentHashMap.put(ConcurrentHashMap.java:881)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegionServer.addRowLock(HRegionServer.java:2346)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegionServer.lockRow(HRegionServer.java:2332)
>> at sun.reflect.GeneratedMethodAccessor156.invoke(Unknown
>> Source)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at
>> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:384)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1336)
>> 2013-05-14 12:46:02,514 WARN org.apache.hadoop.ipc.HBaseServer: IPC
>> Server Responder, call get([B@2166c821,
>> {"timeRange":[0,9223372036854775807],"totalColumns":1,"cacheBlocks":true,"families":{"id":["tagv"]},"maxVersions":1,"row":"
>> slcsn-s00314.slc.ebay.com"}), rpc version=1, client version=29,
>> methodsFingerPrint=0 from 10.53.106.37:58892: output error
>> 2013-05-14 12:46:02,514 WARN org.apache.hadoop.ipc.HBaseServer: IPC
>> Server handler 3 on 60020 caught: java.io.IOException: Connection reset by
>> peer
>> at sun.nio.ch.FileDispatcher.write0(Native Method)
>> at
>> sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29)
>>
>
>
Re: NullPointerException while loading large amount of new rows into
HBase, exception is thrown when trying to obtain lock for RowKey
Posted by ramkrishna vasudevan <ra...@gmail.com>.
Are you trying to get the row lock explicitly ? Using HTable.lockRow?
Regards
Ram
On Thu, May 16, 2013 at 10:46 PM, ramkrishna vasudevan <
ramkrishna.s.vasudevan@gmail.com> wrote:
> Which version of HBase?
>
> Regards
> Ram
>
>
> On Thu, May 16, 2013 at 10:42 PM, Tianying Chang <ti...@ebaysf.com>wrote:
>
>> Hi,
>>
>> When our customers(using TSDB) loads large amount of data into HBase, we
>> saw many NullPointerException in the RS logs as below. I checked the source
>> code, it seems when trying to obtain the lock for a rowKey, if the entry
>> for that row already exists and the "waitfoBlock" is false (therefore it
>> won't retry, but just return a NULL value). I can see in
>> doMiniBatchMutation(), the waitForBlock is set to be false. (most other
>> places "waitForBlock" is always set to true).
>>
>> This exception is thrown from function lockRow(), which has been
>> deprecated. I am not sure why it is deprecated, and what is used to replace
>> it. Is this normal? which implies the HBase should not throw this
>> misleading error message to log. Or should the client call some other API?
>>
>> Thanks
>> Tian-Ying
>>
>> 2013-05-14 12:45:30,911 DEBUG
>> org.apache.hadoop.hbase.regionserver.HRegionServer: Row lock
>> -3430274391270203797 explicitly acquired by client
>> 2013-05-14 12:45:30,911 WARN org.apache.hadoop.ipc.HBaseServer:
>> (responseTooSlow): {"processingtimems":29783,"call":"lockRow([B@339a6a5c,
>> [B@5ebcd87b), rpc version=1, client version=29,
>> methodsFingerPrint=0","client":"10.53.106.37:58892
>> ","starttimems":1368560701128,"queuetimems":847,"class":"HRegionServer","responsesize":0,"method":"lockRow"}
>> 2013-05-14 12:46:00,911 ERROR
>> org.apache.hadoop.hbase.regionserver.HRegionServer: Error obtaining row
>> lock (fsOk: true)
>> java.lang.NullPointerException
>> at
>> java.util.concurrent.ConcurrentHashMap.put(ConcurrentHashMap.java:881)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegionServer.addRowLock(HRegionServer.java:2346)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegionServer.lockRow(HRegionServer.java:2332)
>> at sun.reflect.GeneratedMethodAccessor156.invoke(Unknown
>> Source)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at
>> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:384)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1336)
>> 2013-05-14 12:46:02,514 WARN org.apache.hadoop.ipc.HBaseServer: IPC
>> Server Responder, call get([B@2166c821,
>> {"timeRange":[0,9223372036854775807],"totalColumns":1,"cacheBlocks":true,"families":{"id":["tagv"]},"maxVersions":1,"row":"
>> slcsn-s00314.slc.ebay.com"}), rpc version=1, client version=29,
>> methodsFingerPrint=0 from 10.53.106.37:58892: output error
>> 2013-05-14 12:46:02,514 WARN org.apache.hadoop.ipc.HBaseServer: IPC
>> Server handler 3 on 60020 caught: java.io.IOException: Connection reset by
>> peer
>> at sun.nio.ch.FileDispatcher.write0(Native Method)
>> at
>> sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29)
>>
>
>
Re: NullPointerException while loading large amount of new rows into
HBase, exception is thrown when trying to obtain lock for RowKey
Posted by ramkrishna vasudevan <ra...@gmail.com>.
Which version of HBase?
Regards
Ram
On Thu, May 16, 2013 at 10:42 PM, Tianying Chang <ti...@ebaysf.com> wrote:
> Hi,
>
> When our customers(using TSDB) loads large amount of data into HBase, we
> saw many NullPointerException in the RS logs as below. I checked the source
> code, it seems when trying to obtain the lock for a rowKey, if the entry
> for that row already exists and the "waitfoBlock" is false (therefore it
> won't retry, but just return a NULL value). I can see in
> doMiniBatchMutation(), the waitForBlock is set to be false. (most other
> places "waitForBlock" is always set to true).
>
> This exception is thrown from function lockRow(), which has been
> deprecated. I am not sure why it is deprecated, and what is used to replace
> it. Is this normal? which implies the HBase should not throw this
> misleading error message to log. Or should the client call some other API?
>
> Thanks
> Tian-Ying
>
> 2013-05-14 12:45:30,911 DEBUG
> org.apache.hadoop.hbase.regionserver.HRegionServer: Row lock
> -3430274391270203797 explicitly acquired by client
> 2013-05-14 12:45:30,911 WARN org.apache.hadoop.ipc.HBaseServer:
> (responseTooSlow): {"processingtimems":29783,"call":"lockRow([B@339a6a5c,
> [B@5ebcd87b), rpc version=1, client version=29,
> methodsFingerPrint=0","client":"10.53.106.37:58892
> ","starttimems":1368560701128,"queuetimems":847,"class":"HRegionServer","responsesize":0,"method":"lockRow"}
> 2013-05-14 12:46:00,911 ERROR
> org.apache.hadoop.hbase.regionserver.HRegionServer: Error obtaining row
> lock (fsOk: true)
> java.lang.NullPointerException
> at
> java.util.concurrent.ConcurrentHashMap.put(ConcurrentHashMap.java:881)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.addRowLock(HRegionServer.java:2346)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.lockRow(HRegionServer.java:2332)
> at sun.reflect.GeneratedMethodAccessor156.invoke(Unknown
> Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:384)
> at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1336)
> 2013-05-14 12:46:02,514 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server
> Responder, call get([B@2166c821,
> {"timeRange":[0,9223372036854775807],"totalColumns":1,"cacheBlocks":true,"families":{"id":["tagv"]},"maxVersions":1,"row":"
> slcsn-s00314.slc.ebay.com"}), rpc version=1, client version=29,
> methodsFingerPrint=0 from 10.53.106.37:58892: output error
> 2013-05-14 12:46:02,514 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 3 on 60020 caught: java.io.IOException: Connection reset by peer
> at sun.nio.ch.FileDispatcher.write0(Native Method)
> at
> sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29)
>