You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Marco Gallotta <ma...@gallotta.co.za> on 2012/07/24 23:53:39 UTC
UnknownRowLockException
Hi there
I am getting an UnknownRowLockException when adding locks to the increment() function below. The full stack trace is at the end of this message.
There are a few other places I am acquiring locks, but I only ever acquire a single lock for one piece of code and the rest go through fine. It's only when I enable the locks in this function that I get the exception. It's fairly simple, and I can't find anything obviously wrong with it so my best guess is there's some quirk with hbase locks that I'm unaware of. Any ideas?
One thing to note is that I am calling increment() a couple times consecutively for the same row.
Marco
public static void increment(String tableName, String key, String family,
String qualifier, int value, boolean useShort,
Configuration config)
throws IOException{
HTable table = new HTable(config,
config.get("app", "geomon-test") + "." + tableName);
Get get = new Get(key.getBytes());
get.addColumn(family.getBytes(),
qualifier.getBytes());
int before = 0;
RowLock lock = table.lockRow(key.getBytes());
try {
Result result = table.get(get);
if (!result.isEmpty()) {
if (useShort) {
before = (int) Bytes.toShort(result.getValue(family.getBytes(),
qualifier.getBytes()));
} else {
before = Bytes.toInt(result.getValue(family.getBytes(),
qualifier.getBytes()));
}
}
if (useShort) {
value = Math.min(before + value, Short.MAX_VALUE);
} else {
value = (int) Math.min((long) before + value, Integer.MAX_VALUE);
}
Put put = new Put(key.getBytes());
put.add(family.getBytes(),
qualifier.getBytes(),
Bytes.toBytes(useShort ? (short) value : value));
table.put(put);
} catch (Exception e) {
System.err.println(e.getMessage());
} finally {
table.unlockRow(lock);
}
}
12/07/24 14:20:58 INFO mapred.JobClient: Task Id : attempt_201207241320_0002_r_000000_2, Status : FAILEDorg.apache.hadoop.hbase.UnknownRowLockException: org.apache.hadoop.hbase.UnknownRowLockException: -3110061788478328763
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:532)
at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:95)
at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:79)
at org.apache.hadoop.hbase.client.ServerCallable.translateException(ServerCallable.java:228)
at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:166)
at org.apache.hadoop.hbase.client.HTable.unlockRow(HTable.java:1031)
at Util.increment(Util.java:170)
at Aggregate$EventReducer.processUserEvent(Aggregate.java:161)
at Aggregate$EventReducer.processUser(Aggregate.java:119)
at Aggregate$EventReducer.reduce(Aggregate.java:189)
at Aggregate$EventReducer.reduce(Aggregate.java:78)
at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:176)
at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:649)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:417)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:416)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hbase.UnknownRowLockException: -3110061788478328763
at org.apache.hadoop.hbase.regionserver.HRegionServer.unlockRow(HRegionServer.java:2633)
at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1376)
at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:918)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:150)
at $Proxy3.unlockRow(Unknown Source)
at org.apache.hadoop.hbase.client.HTable$15.call(HTable.java:1033)
at org.apache.hadoop.hbase.client.HTable$15.call(HTable.java:1031)
at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:163)
... 14 more
--
Marco Gallotta | Mountain View, California
Software Engineer, Infrastructure | Loki Studios
fb.me/marco.gallotta | twitter.com/marcog
marco@gallotta.co.za | +1 (650) 417-3313
Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
Re: UnknownRowLockException
Posted by Jean-Daniel Cryans <jd...@apache.org>.
On Tue, Jul 24, 2012 at 3:55 PM, Marco Gallotta <ma...@gallotta.co.za> wrote:
>
> I have thousands of columns of per row, so I doubt that's still true?
No, since every value is stored along its key (row key + family +
qualifier + timestamp + length of each). That's why compression is
pretty much always recommended.
J-D
Re: UnknownRowLockException
Posted by Marco Gallotta <ma...@gallotta.co.za>.
On Tuesday 24 July 2012 at 3:51 PM, Jean-Daniel Cryans wrote:
> The space saving is a good question, are you really that short on hard
> drives? The key is still the bigger part of your whole row by an order
> of magnitude and this is usually where people try to shave a few bytes
> off.
>
>
I have thousands of columns of per row, so I doubt that's still true?
Marco
--
Marco Gallotta | Mountain View, California
Software Engineer, Infrastructure | Loki Studios
fb.me/marco.gallotta | twitter.com/marcog
marco@gallotta.co.za | +1 (650) 417-3313
Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
Re: UnknownRowLockException
Posted by Jean-Daniel Cryans <jd...@apache.org>.
On Tue, Jul 24, 2012 at 3:44 PM, Marco Gallotta <ma...@gallotta.co.za> wrote:
> Oh, oops, I misread increments documentation. I'm fine with that. The only problem now then is that it only operates on longs, but I'm storing ints and shorts to save space. Perhaps the space saving isn't worth the cost of not using this though.
That's what I thought, I don't know what a partially completed
increment on a single cell would look like... see 7 while someone is
incrementing 4 by 5? :)
The space saving is a good question, are you really that short on hard
drives? The key is still the bigger part of your whole row by an order
of magnitude and this is usually where people try to shave a few bytes
off.
J-D
Re: UnknownRowLockException
Posted by Marco Gallotta <ma...@gallotta.co.za>.
Oh, oops, I misread increments documentation. I'm fine with that. The only problem now then is that it only operates on longs, but I'm storing ints and shorts to save space. Perhaps the space saving isn't worth the cost of not using this though.
Marco
--
Marco Gallotta | Mountain View, California
Software Engineer, Infrastructure | Loki Studios
fb.me/marco.gallotta | twitter.com/marcog
marco@gallotta.co.za | +1 (650) 417-3313
Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
On Tuesday 24 July 2012 at 3:34 PM, Jean-Daniel Cryans wrote:
> On Tue, Jul 24, 2012 at 3:29 PM, Marco Gallotta <marco@gallotta.co.za (mailto:marco@gallotta.co.za)> wrote:
> > I acquire the lock before the get. I only setup the get before acquiring the lock.
>
>
> Yes, but let's say you want to increment "f:a" and "f:b" in the row
> "example". Your code requires 2 calls to increment() and would be seen
> as two different puts so a reader could see "f:a" incremented but not
> "f:b". It's the same for HBase's increment function, except it's going
> to be much faster and won't suffer row lock problems.
>
> J-D
Re: UnknownRowLockException
Posted by Jean-Daniel Cryans <jd...@apache.org>.
On Tue, Jul 24, 2012 at 3:29 PM, Marco Gallotta <ma...@gallotta.co.za> wrote:
> I acquire the lock before the get. I only setup the get before acquiring the lock.
Yes, but let's say you want to increment "f:a" and "f:b" in the row
"example". Your code requires 2 calls to increment() and would be seen
as two different puts so a reader could see "f:a" incremented but not
"f:b". It's the same for HBase's increment function, except it's going
to be much faster and won't suffer row lock problems.
J-D
Re: UnknownRowLockException
Posted by Marco Gallotta <ma...@gallotta.co.za>.
I acquire the lock before the get. I only setup the get before acquiring the lock.
I'm also looking into using checkAndPut.
Marco
--
Marco Gallotta | Mountain View, California
Software Engineer, Infrastructure | Loki Studios
fb.me/marco.gallotta | twitter.com/marcog
marco@gallotta.co.za | +1 (650) 417-3313
Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
On Tuesday 24 July 2012 at 3:13 PM, Jean-Daniel Cryans wrote:
> On Tue, Jul 24, 2012 at 3:08 PM, Marco Gallotta <marco@gallotta.co.za (mailto:marco@gallotta.co.za)> wrote:
> > HBase's increment method says "readers do not take row locks so get and scan operations can see this operation partially completed", which could cause problems.
>
>
> As far as I can tell your code suffers from the same issue eg if you
> need to increment 2 cells you do 2 puts under two separate locks.
>
> J-D
Re: UnknownRowLockException
Posted by Jean-Daniel Cryans <jd...@apache.org>.
On Tue, Jul 24, 2012 at 3:08 PM, Marco Gallotta <ma...@gallotta.co.za> wrote:
> HBase's increment method says "readers do not take row locks so get and scan operations can see this operation partially completed", which could cause problems.
As far as I can tell your code suffers from the same issue eg if you
need to increment 2 cells you do 2 puts under two separate locks.
J-D
Re: UnknownRowLockException
Posted by Marco Gallotta <ma...@gallotta.co.za>.
I was unaware of locks not moving with regions/splits. Hmm…I just came across http://jerryjcw.blogspot.com/2009/10/hbase-notes-casual-remark-about-row.html which I'm going to try.
HBase's increment method says "readers do not take row locks so get and scan operations can see this operation partially completed", which could cause problems.
Marco
--
Marco Gallotta | Mountain View, California
Software Engineer, Infrastructure | Loki Studios
fb.me/marco.gallotta | twitter.com/marcog
marco@gallotta.co.za | +1 (650) 417-3313
Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
On Tuesday 24 July 2012 at 2:58 PM, Jean-Daniel Cryans wrote:
> Row locks don't move with regions or splits... are such things
> happening frequently?
>
> Also, any reason to not use HBase's own increment method that's much
> more efficient?
>
> http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/HTable.html#incrementColumnValue(byte[],
> byte[], byte[], long)
>
> J-D
>
> On Tue, Jul 24, 2012 at 2:53 PM, Marco Gallotta <marco@gallotta.co.za (mailto:marco@gallotta.co.za)> wrote:
> > Hi there
> >
> > I am getting an UnknownRowLockException when adding locks to the increment() function below. The full stack trace is at the end of this message.
> >
> > There are a few other places I am acquiring locks, but I only ever acquire a single lock for one piece of code and the rest go through fine. It's only when I enable the locks in this function that I get the exception. It's fairly simple, and I can't find anything obviously wrong with it so my best guess is there's some quirk with hbase locks that I'm unaware of. Any ideas?
> >
> > One thing to note is that I am calling increment() a couple times consecutively for the same row.
> >
> > Marco
> >
> > public static void increment(String tableName, String key, String family,
> > String qualifier, int value, boolean useShort,
> > Configuration config)
> > throws IOException{
> > HTable table = new HTable(config,
> > config.get("app", "geomon-test") + "." + tableName);
> > Get get = new Get(key.getBytes());
> > get.addColumn(family.getBytes(),
> > qualifier.getBytes());
> > int before = 0;
> > RowLock lock = table.lockRow(key.getBytes());
> > try {
> > Result result = table.get(get);
> > if (!result.isEmpty()) {
> > if (useShort) {
> > before = (int) Bytes.toShort(result.getValue(family.getBytes(),
> > qualifier.getBytes()));
> > } else {
> > before = Bytes.toInt(result.getValue(family.getBytes(),
> > qualifier.getBytes()));
> > }
> > }
> > if (useShort) {
> > value = Math.min(before + value, Short.MAX_VALUE);
> > } else {
> > value = (int) Math.min((long) before + value, Integer.MAX_VALUE);
> > }
> >
> > Put put = new Put(key.getBytes());
> > put.add(family.getBytes(),
> > qualifier.getBytes(),
> > Bytes.toBytes(useShort ? (short) value : value));
> > table.put(put);
> > } catch (Exception e) {
> > System.err.println(e.getMessage());
> > } finally {
> > table.unlockRow(lock);
> > }
> > }
> >
> >
> > 12/07/24 14:20:58 INFO mapred.JobClient: Task Id : attempt_201207241320_0002_r_000000_2, Status : FAILEDorg.apache.hadoop.hbase.UnknownRowLockException: org.apache.hadoop.hbase.UnknownRowLockException: -3110061788478328763
> > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> > at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:532)
> > at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:95)
> > at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:79)
> > at org.apache.hadoop.hbase.client.ServerCallable.translateException(ServerCallable.java:228)
> > at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:166)
> > at org.apache.hadoop.hbase.client.HTable.unlockRow(HTable.java:1031)
> > at Util.increment(Util.java:170)
> > at Aggregate$EventReducer.processUserEvent(Aggregate.java:161)
> > at Aggregate$EventReducer.processUser(Aggregate.java:119)
> > at Aggregate$EventReducer.reduce(Aggregate.java:189)
> > at Aggregate$EventReducer.reduce(Aggregate.java:78)
> > at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:176)
> > at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:649)
> > at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:417)
> > at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at javax.security.auth.Subject.doAs(Subject.java:416)
> > at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
> > at org.apache.hadoop.mapred.Child.main(Child.java:249)
> > Caused by: org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hbase.UnknownRowLockException: -3110061788478328763
> > at org.apache.hadoop.hbase.regionserver.HRegionServer.unlockRow(HRegionServer.java:2633)
> > at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
> > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >
> > at java.lang.reflect.Method.invoke(Method.java:616)
> > at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
> > at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1376)
> >
> > at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:918)
> > at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:150)
> > at $Proxy3.unlockRow(Unknown Source)
> > at org.apache.hadoop.hbase.client.HTable$15.call(HTable.java:1033)
> > at org.apache.hadoop.hbase.client.HTable$15.call(HTable.java:1031)
> > at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:163)
> > ... 14 more
> >
> >
> >
> > --
> > Marco Gallotta | Mountain View, California
> > Software Engineer, Infrastructure | Loki Studios
> > fb.me/marco.gallotta (http://fb.me/marco.gallotta) | twitter.com/marcog (http://twitter.com/marcog)
> > marco@gallotta.co.za (mailto:marco@gallotta.co.za) | +1 (650) 417-3313
> >
> > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
Re: UnknownRowLockException
Posted by Jean-Daniel Cryans <jd...@apache.org>.
Row locks don't move with regions or splits... are such things
happening frequently?
Also, any reason to not use HBase's own increment method that's much
more efficient?
http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/HTable.html#incrementColumnValue(byte[],
byte[], byte[], long)
J-D
On Tue, Jul 24, 2012 at 2:53 PM, Marco Gallotta <ma...@gallotta.co.za> wrote:
> Hi there
>
> I am getting an UnknownRowLockException when adding locks to the increment() function below. The full stack trace is at the end of this message.
>
> There are a few other places I am acquiring locks, but I only ever acquire a single lock for one piece of code and the rest go through fine. It's only when I enable the locks in this function that I get the exception. It's fairly simple, and I can't find anything obviously wrong with it so my best guess is there's some quirk with hbase locks that I'm unaware of. Any ideas?
>
> One thing to note is that I am calling increment() a couple times consecutively for the same row.
>
> Marco
>
> public static void increment(String tableName, String key, String family,
> String qualifier, int value, boolean useShort,
> Configuration config)
> throws IOException{
> HTable table = new HTable(config,
> config.get("app", "geomon-test") + "." + tableName);
> Get get = new Get(key.getBytes());
> get.addColumn(family.getBytes(),
> qualifier.getBytes());
> int before = 0;
> RowLock lock = table.lockRow(key.getBytes());
> try {
> Result result = table.get(get);
> if (!result.isEmpty()) {
> if (useShort) {
> before = (int) Bytes.toShort(result.getValue(family.getBytes(),
> qualifier.getBytes()));
> } else {
> before = Bytes.toInt(result.getValue(family.getBytes(),
> qualifier.getBytes()));
> }
> }
> if (useShort) {
> value = Math.min(before + value, Short.MAX_VALUE);
> } else {
> value = (int) Math.min((long) before + value, Integer.MAX_VALUE);
> }
>
> Put put = new Put(key.getBytes());
> put.add(family.getBytes(),
> qualifier.getBytes(),
> Bytes.toBytes(useShort ? (short) value : value));
> table.put(put);
> } catch (Exception e) {
> System.err.println(e.getMessage());
> } finally {
> table.unlockRow(lock);
> }
> }
>
>
> 12/07/24 14:20:58 INFO mapred.JobClient: Task Id : attempt_201207241320_0002_r_000000_2, Status : FAILEDorg.apache.hadoop.hbase.UnknownRowLockException: org.apache.hadoop.hbase.UnknownRowLockException: -3110061788478328763
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:532)
> at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:95)
> at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:79)
> at org.apache.hadoop.hbase.client.ServerCallable.translateException(ServerCallable.java:228)
> at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:166)
> at org.apache.hadoop.hbase.client.HTable.unlockRow(HTable.java:1031)
> at Util.increment(Util.java:170)
> at Aggregate$EventReducer.processUserEvent(Aggregate.java:161)
> at Aggregate$EventReducer.processUser(Aggregate.java:119)
> at Aggregate$EventReducer.reduce(Aggregate.java:189)
> at Aggregate$EventReducer.reduce(Aggregate.java:78)
> at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:176)
> at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:649)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:417)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:416)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
> at org.apache.hadoop.mapred.Child.main(Child.java:249)
> Caused by: org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hbase.UnknownRowLockException: -3110061788478328763
> at org.apache.hadoop.hbase.regionserver.HRegionServer.unlockRow(HRegionServer.java:2633)
> at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>
> at java.lang.reflect.Method.invoke(Method.java:616)
> at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
> at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1376)
>
> at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:918)
> at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:150)
> at $Proxy3.unlockRow(Unknown Source)
> at org.apache.hadoop.hbase.client.HTable$15.call(HTable.java:1033)
> at org.apache.hadoop.hbase.client.HTable$15.call(HTable.java:1031)
> at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:163)
> ... 14 more
>
>
>
> --
> Marco Gallotta | Mountain View, California
> Software Engineer, Infrastructure | Loki Studios
> fb.me/marco.gallotta | twitter.com/marcog
> marco@gallotta.co.za | +1 (650) 417-3313
>
> Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
>