You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "stack (JIRA)" <ji...@apache.org> on 2007/12/19 19:22:43 UTC
[jira] Updated: (HADOOP-2203) [hbase] Hung MR job writing hbase
[ https://issues.apache.org/jira/browse/HADOOP-2203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HADOOP-2203:
--------------------------
Attachment: TriplesTest.java
Holger Stenzhorn gave his blessing. Here is the class. If you want the data file, let me know and I'll arrange it so you can pull it (too big to attach here).
> [hbase] Hung MR job writing hbase
> ---------------------------------
>
> Key: HADOOP-2203
> URL: https://issues.apache.org/jira/browse/HADOOP-2203
> Project: Hadoop
> Issue Type: Bug
> Components: contrib/hbase
> Reporter: stack
> Priority: Minor
> Attachments: TriplesTest.java
>
>
> Running a simple, fairly large job on the cluster, a few of the reduce steps failed; all that did are effectively hung. Looking at failures, I see this strange output:
> {code}
> 2007-11-13 02:09:09,606 DEBUG org.apache.hadoop.hbase.HTable: reloading table servers because: java.io.IOException: Region triples,http://purl.org/dc/terms/bibliographicCitation,1194919500610 closed
> at org.apache.hadoop.hbase.HRegion.startUpdate(HRegion.java:1142)
> at org.apache.hadoop.hbase.HRegionServer.startUpdate(HRegionServer.java:1239)
> at org.apache.hadoop.hbase.HRegionServer.batchUpdate(HRegionServer.java:1113)
> at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:379)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:596)
> 2007-11-13 02:09:19,778 DEBUG org.apache.hadoop.hbase.HTable: reloading table servers because: org.apache.hadoop.hbase.NotServingRegionException: triples,http://purl.org/dc/terms/bibliographicCitation,1194919500610
> at org.apache.hadoop.hbase.HRegionServer.getRegion(HRegionServer.java:1325)
> at org.apache.hadoop.hbase.HRegionServer.getRegion(HRegionServer.java:1297)
> at org.apache.hadoop.hbase.HRegionServer.startUpdate(HRegionServer.java:1238)
> at org.apache.hadoop.hbase.HRegionServer.batchUpdate(HRegionServer.java:1113)
> at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:379)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:596)
> 2007-11-13 02:09:30,039 DEBUG org.apache.hadoop.hbase.HTable: reloading table servers because: org.apache.hadoop.hbase.NotServingRegionException: triples,http://purl.org/dc/terms/bibliographicCitation,1194919500610
> at org.apache.hadoop.hbase.HRegionServer.getRegion(HRegionServer.java:1325)
> at org.apache.hadoop.hbase.HRegionServer.getRegion(HRegionServer.java:1297)
> at org.apache.hadoop.hbase.HRegionServer.startUpdate(HRegionServer.java:1238)
> at org.apache.hadoop.hbase.HRegionServer.batchUpdate(HRegionServer.java:1113)
> at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:379)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:596)
> 2007-11-13 02:09:40,275 DEBUG org.apache.hadoop.hbase.HTable: reloading table servers because: org.apache.hadoop.hbase.NotServingRegionException: triples,http://purl.org/dc/terms/bibliographicCitation,1194919500610
> at org.apache.hadoop.hbase.HRegionServer.getRegion(HRegionServer.java:1325)
> at org.apache.hadoop.hbase.HRegionServer.getRegion(HRegionServer.java:1297)
> at org.apache.hadoop.hbase.HRegionServer.startUpdate(HRegionServer.java:1238)
> at org.apache.hadoop.hbase.HRegionServer.batchUpdate(HRegionServer.java:1113)
> at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:379)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:596)
> 2007-11-13 02:09:55,379 WARN org.apache.hadoop.mapred.TaskTracker: Error running child
> org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: triples,http://purl.org/dc/terms/bibliographicCitation,1194919500610
> at org.apache.hadoop.hbase.HRegionServer.getRegion(HRegionServer.java:1325)
> at org.apache.hadoop.hbase.HRegionServer.getRegion(HRegionServer.java:1297)
> at org.apache.hadoop.hbase.HRegionServer.startUpdate(HRegionServer.java:1238)
> at org.apache.hadoop.hbase.HRegionServer.batchUpdate(HRegionServer.java:1113)
> at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:379)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:596)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> at org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:82)
> at org.apache.hadoop.hbase.HTable.commit(HTable.java:734)
> at org.apache.hadoop.hbase.HTable.commit(HTable.java:706)
> at org.apache.hadoop.hbase.mapred.TableOutputFormat$TableRecordWriter.write(TableOutputFormat.java:89)
> at org.apache.hadoop.hbase.mapred.TableOutputFormat$TableRecordWriter.write(TableOutputFormat.java:63)
> at org.apache.hadoop.mapred.ReduceTask$2.collect(ReduceTask.java:308)
> at TriplesTest$TestReducer.reduce(TriplesTest.java:66)
> at TriplesTest$TestReducer.reduce(TriplesTest.java:51)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:326)
> at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1935)
> {code}
> Why is the reducer client being pig-headed trying to go against a region that has been closed? Why not recalibrate?
> Here is master log from around same time on the problem region:
> {code}
> ...
> 2007-11-13 02:10:17,867 INFO org.apache.hadoop.hbase.HMaster: region triples,http://purl.org/dc/terms/bibliographicCitation,1194919500610 split. New regions are: triples,http://purl.org/dc/terms/bibliographicCitation,1194919707661, triples,http://purl.org/dc/terms/references,1194919707661
> ...
> 2007-11-13 02:11:47,313 DEBUG org.apache.hadoop.hbase.HMaster: HMaster.metaScanner scanner: -3075797136896920808 regioninfo: {regionname: triples,http://purl.org/dc/terms/bibliographicCitation,1194919500610, startKey: <http://purl.org/dc/terms/bibliographicCitation>, offline: true, split: true, tableDesc: {name: triples, families: {triples:={name: triples, max versions: 3, compression: NONE, in memory: false, max length: 2147483647, bloom filter: none}}}}, server: 208.76.44.140:60010, startCode: -647590197693256616
> ...
> 2007-11-13 02:11:47,317 INFO org.apache.hadoop.hbase.HMaster: Deleting region triples,http://purl.org/dc/terms/bibliographicCitation,1194919500610 because daughter splits no longer hold references
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.