You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Clint Morgan (JIRA)" <ji...@apache.org> on 2010/05/04 20:43:59 UTC
[jira] Commented: (HBASE-2191) Commit fails occasionally with
NotServingRegionException in transactional hbase inserts
[ https://issues.apache.org/jira/browse/HBASE-2191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12863938#action_12863938 ]
Clint Morgan commented on HBASE-2191:
-------------------------------------
This is by design. We don't want pending transactions to hold up Region splitting, so if a region splits mid-transactions, the transaction will have to be retried.
If you are simply doing a bulk upload of data without any other reads/writes happening, then I wonder if you really need to use transactions.
In general your code should handle the CommitUnsucessfulException when trying to commit. The CME could happen when the region splits as above, but also when previously committed transaction had writes which interferer the trx being committed. In such a case the trx can not be committed and will throw CME. The solution is to run your transaction again from the start which would cause the next read to be valid...
If you really need individual transactions to work across splits, can you comment on your workload and requirements a bit more. Making it work would probably involve a deal of work.
> Commit fails occasionally with NotServingRegionException in transactional hbase inserts
> ---------------------------------------------------------------------------------------
>
> Key: HBASE-2191
> URL: https://issues.apache.org/jira/browse/HBASE-2191
> Project: Hadoop HBase
> Issue Type: Bug
> Components: contrib
> Affects Versions: 0.20.3
> Environment: OpenSolaris, java 1.6.0_10, pseudo-distributed setup; Hadoop 0.20.1 and HBase 0.20.3, single java client, Hadoop dfs.replication=1, HBase configured to use transactional server.
> Reporter: Jesse Hallio
>
> I'm testing a simple bulk upload to HBase. Single table, single family. The aim is to upload 200M+ lines into HBase. I've tried to commit the upload in various batches, from 1 line at time to 25000 lines at time. Each time, at seemingly random point, the upload fails in commit with a org.apache.hadoop.hbase.NotServingRegionException
> Failure stack trace from the client:
> ---
> 10/02/07 16:38:40 DEBUG transactional.TransactionManager: Begining transaction 5311384003652808027
> 10/02/07 16:38:40 DEBUG transactional.TransactionState: Adding new hregion [db,,1265553236583] to transaction [5311384003652808027]
> 10/02/07 16:38:46 DEBUG transactional.TransactionManager: atempting to commit trasaction: id: 5311384003652808027, particpants: 1
> 10/02/07 16:38:46 DEBUG transactional.TransactionManager: Region [db,,1265553236583] votes to commit transaction 5311384003652808027
> 10/02/07 16:38:46 DEBUG transactional.TransactionManager: Commiting [5311384003652808027]
> 10/02/07 16:38:46 DEBUG transactional.TransactionManager: Committed transaction [5311384003652808027] in [379]ms
> 10/02/07 16:38:46 DEBUG transactional.TransactionManager: Begining transaction -5274908504613584338
> 10/02/07 16:38:46 DEBUG transactional.TransactionState: Adding new hregion [db,,1265553236583] to transaction [-5274908504613584338]
> 10/02/07 16:38:47 DEBUG client.HConnectionManager$TableServers: Removed db,,1265553236583 for tableName=db from cache because of \x00\x00\x01\x24\xAC\xC3\x98\x1C
> 10/02/07 16:38:47 DEBUG client.HConnectionManager$TableServers: Cached location for db,,1265553236583 is 1.1.1.1:43917
> 10/02/07 16:38:48 DEBUG client.HConnectionManager$TableServers: Removed db,,1265553236583 for tableName=db from cache because of \x00\x00\x01\x24\xAC\xC3\x98\x1C
> 10/02/07 16:38:48 DEBUG client.HConnectionManager$TableServers: Cached location for db,,1265553236583 is 1.1.1.1:43917
> 10/02/07 16:38:49 DEBUG client.HConnectionManager$TableServers: Removed db,,1265553236583 for tableName=db from cache because of \x00\x00\x01\x24\xAC\xC3\x98\x1C
> 10/02/07 16:38:49 DEBUG client.HConnectionManager$TableServers: locateRegionInMeta attempt 0 of 10 failed; retrying after sleep of 1000 because: No server address listed in .META. for region db,\x00\x00\x01\x24\xAC\xA5\xEAB,1265553527794
> 10/02/07 16:38:49 DEBUG client.HConnectionManager$TableServers: Removed .META.,,1 for tableName=.META. from cache because of db,\x00\x00\x01\x24\xAC\xC3\x98\x1C,99999999999999
> 10/02/07 16:38:49 DEBUG client.HConnectionManager$TableServers: Cached location for .META.,,1 is 1.1.1.1:43917
> 10/02/07 16:38:50 DEBUG client.HConnectionManager$TableServers: Cached location for db,\x00\x00\x01\x24\xAC\xA5\xEAB,1265553527794 is 1.1.1.1:43917
> 10/02/07 16:38:50 DEBUG transactional.TransactionState: Adding new hregion [db,\x00\x00\x01\x24\xAC\xA5\xEAB,1265553527794] to transaction [-5274908504613584338]
> 10/02/07 16:38:51 DEBUG transactional.TransactionManager: atempting to commit trasaction: id: -5274908504613584338, particpants: 2
> 10/02/07 16:38:51 DEBUG transactional.TransactionManager: Commit of transaction [-5274908504613584338] was unsucsessful
> org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hbase.NotServingRegionException: db,,1265553236583
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2266)
> at org.apache.hadoop.hbase.regionserver.transactional.TransactionalRegionServer.getTransactionalRegion(TransactionalRegionServer.java:131)
> at org.apache.hadoop.hbase.regionserver.transactional.TransactionalRegionServer.commitRequest(TransactionalRegionServer.java:179)
> at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:657)
> at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
> at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:723)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:333)
> at $Proxy0.commitRequest(Unknown Source)
> at org.apache.hadoop.hbase.client.transactional.TransactionManager.prepareCommit(TransactionManager.java:94)
> at org.apache.hadoop.hbase.client.transactional.TransactionManager.tryCommit(TransactionManager.java:151)
> at dbloader.dbLoader.run2(dbLoader.java:181)
> at dbloader.dbLoader.main(dbLoader.java:24)
> Exception in thread "main" org.apache.hadoop.hbase.client.transactional.CommitUnsuccessfulException: org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hbase.NotServingRegionException: db,,1265553236583
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2266)
> at org.apache.hadoop.hbase.regionserver.transactional.TransactionalRegionServer.getTransactionalRegion(TransactionalRegionServer.java:131)
> at org.apache.hadoop.hbase.regionserver.transactional.TransactionalRegionServer.commitRequest(TransactionalRegionServer.java:179)
> at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:657)
> at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
> at org.apache.hadoop.hbase.client.transactional.TransactionManager.prepareCommit(TransactionManager.java:135)
> at org.apache.hadoop.hbase.client.transactional.TransactionManager.tryCommit(TransactionManager.java:151)
> at dbloader.dbLoader.run2(dbLoader.java:181)
> at dbloader.dbLoader.main(dbLoader.java:24)
> Caused by: org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hbase.NotServingRegionException: db,,1265553236583
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2266)
> at org.apache.hadoop.hbase.regionserver.transactional.TransactionalRegionServer.getTransactionalRegion(TransactionalRegionServer.java:131)
> at org.apache.hadoop.hbase.regionserver.transactional.TransactionalRegionServer.commitRequest(TransactionalRegionServer.java:179)
> at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:657)
> at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
> at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:723)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:333)
> at $Proxy0.commitRequest(Unknown Source)
> at org.apache.hadoop.hbase.client.transactional.TransactionManager.prepareCommit(TransactionManager.java:94)
> ... 3 more
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.