You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "stack (JIRA)" <ji...@apache.org> on 2007/11/14 02:21:43 UTC

[jira] Created: (HADOOP-2203) [hbase] Hung MR job writing hbase

[hbase] Hung MR job writing hbase
---------------------------------

                 Key: HADOOP-2203
                 URL: https://issues.apache.org/jira/browse/HADOOP-2203
             Project: Hadoop
          Issue Type: Bug
          Components: contrib/hbase
            Reporter: stack
            Priority: Minor


Running a simple, fairly large job on the cluster, a few of the reduce steps failed; all that did are effectively hung.  Looking at failures, I see this strange output:

{code}
2007-11-13 02:09:09,606 DEBUG org.apache.hadoop.hbase.HTable: reloading table servers because: java.io.IOException: Region triples,http://purl.org/dc/terms/bibliographicCitation,1194919500610 closed
	at org.apache.hadoop.hbase.HRegion.startUpdate(HRegion.java:1142)
	at org.apache.hadoop.hbase.HRegionServer.startUpdate(HRegionServer.java:1239)
	at org.apache.hadoop.hbase.HRegionServer.batchUpdate(HRegionServer.java:1113)
	at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:379)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:596)

2007-11-13 02:09:19,778 DEBUG org.apache.hadoop.hbase.HTable: reloading table servers because: org.apache.hadoop.hbase.NotServingRegionException: triples,http://purl.org/dc/terms/bibliographicCitation,1194919500610
	at org.apache.hadoop.hbase.HRegionServer.getRegion(HRegionServer.java:1325)
	at org.apache.hadoop.hbase.HRegionServer.getRegion(HRegionServer.java:1297)
	at org.apache.hadoop.hbase.HRegionServer.startUpdate(HRegionServer.java:1238)
	at org.apache.hadoop.hbase.HRegionServer.batchUpdate(HRegionServer.java:1113)
	at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:379)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:596)

2007-11-13 02:09:30,039 DEBUG org.apache.hadoop.hbase.HTable: reloading table servers because: org.apache.hadoop.hbase.NotServingRegionException: triples,http://purl.org/dc/terms/bibliographicCitation,1194919500610
	at org.apache.hadoop.hbase.HRegionServer.getRegion(HRegionServer.java:1325)
	at org.apache.hadoop.hbase.HRegionServer.getRegion(HRegionServer.java:1297)
	at org.apache.hadoop.hbase.HRegionServer.startUpdate(HRegionServer.java:1238)
	at org.apache.hadoop.hbase.HRegionServer.batchUpdate(HRegionServer.java:1113)
	at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:379)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:596)

2007-11-13 02:09:40,275 DEBUG org.apache.hadoop.hbase.HTable: reloading table servers because: org.apache.hadoop.hbase.NotServingRegionException: triples,http://purl.org/dc/terms/bibliographicCitation,1194919500610
	at org.apache.hadoop.hbase.HRegionServer.getRegion(HRegionServer.java:1325)
	at org.apache.hadoop.hbase.HRegionServer.getRegion(HRegionServer.java:1297)
	at org.apache.hadoop.hbase.HRegionServer.startUpdate(HRegionServer.java:1238)
	at org.apache.hadoop.hbase.HRegionServer.batchUpdate(HRegionServer.java:1113)
	at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:379)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:596)

2007-11-13 02:09:55,379 WARN org.apache.hadoop.mapred.TaskTracker: Error running child
org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: triples,http://purl.org/dc/terms/bibliographicCitation,1194919500610
	at org.apache.hadoop.hbase.HRegionServer.getRegion(HRegionServer.java:1325)
	at org.apache.hadoop.hbase.HRegionServer.getRegion(HRegionServer.java:1297)
	at org.apache.hadoop.hbase.HRegionServer.startUpdate(HRegionServer.java:1238)
	at org.apache.hadoop.hbase.HRegionServer.batchUpdate(HRegionServer.java:1113)
	at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:379)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:596)

	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
	at org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:82)
	at org.apache.hadoop.hbase.HTable.commit(HTable.java:734)
	at org.apache.hadoop.hbase.HTable.commit(HTable.java:706)
	at org.apache.hadoop.hbase.mapred.TableOutputFormat$TableRecordWriter.write(TableOutputFormat.java:89)
	at org.apache.hadoop.hbase.mapred.TableOutputFormat$TableRecordWriter.write(TableOutputFormat.java:63)
	at org.apache.hadoop.mapred.ReduceTask$2.collect(ReduceTask.java:308)
	at TriplesTest$TestReducer.reduce(TriplesTest.java:66)
	at TriplesTest$TestReducer.reduce(TriplesTest.java:51)
	at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:326)
	at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1935)
{code}

Why is the reducer client being pig-headed trying to go against a region that has been closed?  Why not recalibrate?

Here is master log from around same time on the problem region:

{code}
...
2007-11-13 02:10:17,867 INFO org.apache.hadoop.hbase.HMaster: region triples,http://purl.org/dc/terms/bibliographicCitation,1194919500610 split. New regions are: triples,http://purl.org/dc/terms/bibliographicCitation,1194919707661, triples,http://purl.org/dc/terms/references,1194919707661
...
2007-11-13 02:11:47,313 DEBUG org.apache.hadoop.hbase.HMaster: HMaster.metaScanner scanner: -3075797136896920808 regioninfo: {regionname: triples,http://purl.org/dc/terms/bibliographicCitation,1194919500610, startKey: <http://purl.org/dc/terms/bibliographicCitation>, offline: true, split: true, tableDesc: {name: triples, families: {triples:={name: triples, max versions: 3, compression: NONE, in memory: false, max length: 2147483647, bloom filter: none}}}}, server: 208.76.44.140:60010, startCode: -647590197693256616
...
2007-11-13 02:11:47,317 INFO org.apache.hadoop.hbase.HMaster: Deleting region triples,http://purl.org/dc/terms/bibliographicCitation,1194919500610 because daughter splits no longer hold references
{code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HADOOP-2203) [hbase] Hung MR job writing hbase

Posted by "stack (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-2203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

stack updated HADOOP-2203:
--------------------------

    Attachment: TriplesTest.java

Holger Stenzhorn gave his blessing.  Here is the class.  If you want the data file, let me know and I'll arrange it so you can pull it (too big to attach here).

> [hbase] Hung MR job writing hbase
> ---------------------------------
>
>                 Key: HADOOP-2203
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2203
>             Project: Hadoop
>          Issue Type: Bug
>          Components: contrib/hbase
>            Reporter: stack
>            Priority: Minor
>         Attachments: TriplesTest.java
>
>
> Running a simple, fairly large job on the cluster, a few of the reduce steps failed; all that did are effectively hung.  Looking at failures, I see this strange output:
> {code}
> 2007-11-13 02:09:09,606 DEBUG org.apache.hadoop.hbase.HTable: reloading table servers because: java.io.IOException: Region triples,http://purl.org/dc/terms/bibliographicCitation,1194919500610 closed
> 	at org.apache.hadoop.hbase.HRegion.startUpdate(HRegion.java:1142)
> 	at org.apache.hadoop.hbase.HRegionServer.startUpdate(HRegionServer.java:1239)
> 	at org.apache.hadoop.hbase.HRegionServer.batchUpdate(HRegionServer.java:1113)
> 	at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> 	at java.lang.reflect.Method.invoke(Method.java:597)
> 	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:379)
> 	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:596)
> 2007-11-13 02:09:19,778 DEBUG org.apache.hadoop.hbase.HTable: reloading table servers because: org.apache.hadoop.hbase.NotServingRegionException: triples,http://purl.org/dc/terms/bibliographicCitation,1194919500610
> 	at org.apache.hadoop.hbase.HRegionServer.getRegion(HRegionServer.java:1325)
> 	at org.apache.hadoop.hbase.HRegionServer.getRegion(HRegionServer.java:1297)
> 	at org.apache.hadoop.hbase.HRegionServer.startUpdate(HRegionServer.java:1238)
> 	at org.apache.hadoop.hbase.HRegionServer.batchUpdate(HRegionServer.java:1113)
> 	at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> 	at java.lang.reflect.Method.invoke(Method.java:597)
> 	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:379)
> 	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:596)
> 2007-11-13 02:09:30,039 DEBUG org.apache.hadoop.hbase.HTable: reloading table servers because: org.apache.hadoop.hbase.NotServingRegionException: triples,http://purl.org/dc/terms/bibliographicCitation,1194919500610
> 	at org.apache.hadoop.hbase.HRegionServer.getRegion(HRegionServer.java:1325)
> 	at org.apache.hadoop.hbase.HRegionServer.getRegion(HRegionServer.java:1297)
> 	at org.apache.hadoop.hbase.HRegionServer.startUpdate(HRegionServer.java:1238)
> 	at org.apache.hadoop.hbase.HRegionServer.batchUpdate(HRegionServer.java:1113)
> 	at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> 	at java.lang.reflect.Method.invoke(Method.java:597)
> 	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:379)
> 	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:596)
> 2007-11-13 02:09:40,275 DEBUG org.apache.hadoop.hbase.HTable: reloading table servers because: org.apache.hadoop.hbase.NotServingRegionException: triples,http://purl.org/dc/terms/bibliographicCitation,1194919500610
> 	at org.apache.hadoop.hbase.HRegionServer.getRegion(HRegionServer.java:1325)
> 	at org.apache.hadoop.hbase.HRegionServer.getRegion(HRegionServer.java:1297)
> 	at org.apache.hadoop.hbase.HRegionServer.startUpdate(HRegionServer.java:1238)
> 	at org.apache.hadoop.hbase.HRegionServer.batchUpdate(HRegionServer.java:1113)
> 	at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> 	at java.lang.reflect.Method.invoke(Method.java:597)
> 	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:379)
> 	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:596)
> 2007-11-13 02:09:55,379 WARN org.apache.hadoop.mapred.TaskTracker: Error running child
> org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: triples,http://purl.org/dc/terms/bibliographicCitation,1194919500610
> 	at org.apache.hadoop.hbase.HRegionServer.getRegion(HRegionServer.java:1325)
> 	at org.apache.hadoop.hbase.HRegionServer.getRegion(HRegionServer.java:1297)
> 	at org.apache.hadoop.hbase.HRegionServer.startUpdate(HRegionServer.java:1238)
> 	at org.apache.hadoop.hbase.HRegionServer.batchUpdate(HRegionServer.java:1113)
> 	at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> 	at java.lang.reflect.Method.invoke(Method.java:597)
> 	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:379)
> 	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:596)
> 	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> 	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> 	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> 	at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> 	at org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:82)
> 	at org.apache.hadoop.hbase.HTable.commit(HTable.java:734)
> 	at org.apache.hadoop.hbase.HTable.commit(HTable.java:706)
> 	at org.apache.hadoop.hbase.mapred.TableOutputFormat$TableRecordWriter.write(TableOutputFormat.java:89)
> 	at org.apache.hadoop.hbase.mapred.TableOutputFormat$TableRecordWriter.write(TableOutputFormat.java:63)
> 	at org.apache.hadoop.mapred.ReduceTask$2.collect(ReduceTask.java:308)
> 	at TriplesTest$TestReducer.reduce(TriplesTest.java:66)
> 	at TriplesTest$TestReducer.reduce(TriplesTest.java:51)
> 	at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:326)
> 	at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1935)
> {code}
> Why is the reducer client being pig-headed trying to go against a region that has been closed?  Why not recalibrate?
> Here is master log from around same time on the problem region:
> {code}
> ...
> 2007-11-13 02:10:17,867 INFO org.apache.hadoop.hbase.HMaster: region triples,http://purl.org/dc/terms/bibliographicCitation,1194919500610 split. New regions are: triples,http://purl.org/dc/terms/bibliographicCitation,1194919707661, triples,http://purl.org/dc/terms/references,1194919707661
> ...
> 2007-11-13 02:11:47,313 DEBUG org.apache.hadoop.hbase.HMaster: HMaster.metaScanner scanner: -3075797136896920808 regioninfo: {regionname: triples,http://purl.org/dc/terms/bibliographicCitation,1194919500610, startKey: <http://purl.org/dc/terms/bibliographicCitation>, offline: true, split: true, tableDesc: {name: triples, families: {triples:={name: triples, max versions: 3, compression: NONE, in memory: false, max length: 2147483647, bloom filter: none}}}}, server: 208.76.44.140:60010, startCode: -647590197693256616
> ...
> 2007-11-13 02:11:47,317 INFO org.apache.hadoop.hbase.HMaster: Deleting region triples,http://purl.org/dc/terms/bibliographicCitation,1194919500610 because daughter splits no longer hold references
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-2203) [hbase] Hung MR job writing hbase

Posted by "Vuk Ercegovac (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-2203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12549195 ] 

Vuk Ercegovac commented on HADOOP-2203:
---------------------------------------

Thanks for the detailed log. Let me know if I'm interpreting this correctly:

1. The reduce HTable client issues a startUpdate and gets a valid xact id
from the RegionServer.

2. The region splits so the updating region is placed off-line.

3. The reduce HTable client issues a commit but finds the Region is offline
(and checks for this some number of times). An IOException is thrown
which kills the reduce task.

Do you expect the protocol to be in this case, for user code to catch IOExceptions,
abort, then retry the entire update? Are there certain exceptions that can be thrown 
that should be specialy handled, i.e., it would be futile to re-try?

> [hbase] Hung MR job writing hbase
> ---------------------------------
>
>                 Key: HADOOP-2203
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2203
>             Project: Hadoop
>          Issue Type: Bug
>          Components: contrib/hbase
>            Reporter: stack
>            Priority: Minor
>
> Running a simple, fairly large job on the cluster, a few of the reduce steps failed; all that did are effectively hung.  Looking at failures, I see this strange output:
> {code}
> 2007-11-13 02:09:09,606 DEBUG org.apache.hadoop.hbase.HTable: reloading table servers because: java.io.IOException: Region triples,http://purl.org/dc/terms/bibliographicCitation,1194919500610 closed
> 	at org.apache.hadoop.hbase.HRegion.startUpdate(HRegion.java:1142)
> 	at org.apache.hadoop.hbase.HRegionServer.startUpdate(HRegionServer.java:1239)
> 	at org.apache.hadoop.hbase.HRegionServer.batchUpdate(HRegionServer.java:1113)
> 	at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> 	at java.lang.reflect.Method.invoke(Method.java:597)
> 	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:379)
> 	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:596)
> 2007-11-13 02:09:19,778 DEBUG org.apache.hadoop.hbase.HTable: reloading table servers because: org.apache.hadoop.hbase.NotServingRegionException: triples,http://purl.org/dc/terms/bibliographicCitation,1194919500610
> 	at org.apache.hadoop.hbase.HRegionServer.getRegion(HRegionServer.java:1325)
> 	at org.apache.hadoop.hbase.HRegionServer.getRegion(HRegionServer.java:1297)
> 	at org.apache.hadoop.hbase.HRegionServer.startUpdate(HRegionServer.java:1238)
> 	at org.apache.hadoop.hbase.HRegionServer.batchUpdate(HRegionServer.java:1113)
> 	at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> 	at java.lang.reflect.Method.invoke(Method.java:597)
> 	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:379)
> 	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:596)
> 2007-11-13 02:09:30,039 DEBUG org.apache.hadoop.hbase.HTable: reloading table servers because: org.apache.hadoop.hbase.NotServingRegionException: triples,http://purl.org/dc/terms/bibliographicCitation,1194919500610
> 	at org.apache.hadoop.hbase.HRegionServer.getRegion(HRegionServer.java:1325)
> 	at org.apache.hadoop.hbase.HRegionServer.getRegion(HRegionServer.java:1297)
> 	at org.apache.hadoop.hbase.HRegionServer.startUpdate(HRegionServer.java:1238)
> 	at org.apache.hadoop.hbase.HRegionServer.batchUpdate(HRegionServer.java:1113)
> 	at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> 	at java.lang.reflect.Method.invoke(Method.java:597)
> 	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:379)
> 	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:596)
> 2007-11-13 02:09:40,275 DEBUG org.apache.hadoop.hbase.HTable: reloading table servers because: org.apache.hadoop.hbase.NotServingRegionException: triples,http://purl.org/dc/terms/bibliographicCitation,1194919500610
> 	at org.apache.hadoop.hbase.HRegionServer.getRegion(HRegionServer.java:1325)
> 	at org.apache.hadoop.hbase.HRegionServer.getRegion(HRegionServer.java:1297)
> 	at org.apache.hadoop.hbase.HRegionServer.startUpdate(HRegionServer.java:1238)
> 	at org.apache.hadoop.hbase.HRegionServer.batchUpdate(HRegionServer.java:1113)
> 	at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> 	at java.lang.reflect.Method.invoke(Method.java:597)
> 	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:379)
> 	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:596)
> 2007-11-13 02:09:55,379 WARN org.apache.hadoop.mapred.TaskTracker: Error running child
> org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: triples,http://purl.org/dc/terms/bibliographicCitation,1194919500610
> 	at org.apache.hadoop.hbase.HRegionServer.getRegion(HRegionServer.java:1325)
> 	at org.apache.hadoop.hbase.HRegionServer.getRegion(HRegionServer.java:1297)
> 	at org.apache.hadoop.hbase.HRegionServer.startUpdate(HRegionServer.java:1238)
> 	at org.apache.hadoop.hbase.HRegionServer.batchUpdate(HRegionServer.java:1113)
> 	at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> 	at java.lang.reflect.Method.invoke(Method.java:597)
> 	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:379)
> 	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:596)
> 	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> 	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> 	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> 	at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> 	at org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:82)
> 	at org.apache.hadoop.hbase.HTable.commit(HTable.java:734)
> 	at org.apache.hadoop.hbase.HTable.commit(HTable.java:706)
> 	at org.apache.hadoop.hbase.mapred.TableOutputFormat$TableRecordWriter.write(TableOutputFormat.java:89)
> 	at org.apache.hadoop.hbase.mapred.TableOutputFormat$TableRecordWriter.write(TableOutputFormat.java:63)
> 	at org.apache.hadoop.mapred.ReduceTask$2.collect(ReduceTask.java:308)
> 	at TriplesTest$TestReducer.reduce(TriplesTest.java:66)
> 	at TriplesTest$TestReducer.reduce(TriplesTest.java:51)
> 	at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:326)
> 	at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1935)
> {code}
> Why is the reducer client being pig-headed trying to go against a region that has been closed?  Why not recalibrate?
> Here is master log from around same time on the problem region:
> {code}
> ...
> 2007-11-13 02:10:17,867 INFO org.apache.hadoop.hbase.HMaster: region triples,http://purl.org/dc/terms/bibliographicCitation,1194919500610 split. New regions are: triples,http://purl.org/dc/terms/bibliographicCitation,1194919707661, triples,http://purl.org/dc/terms/references,1194919707661
> ...
> 2007-11-13 02:11:47,313 DEBUG org.apache.hadoop.hbase.HMaster: HMaster.metaScanner scanner: -3075797136896920808 regioninfo: {regionname: triples,http://purl.org/dc/terms/bibliographicCitation,1194919500610, startKey: <http://purl.org/dc/terms/bibliographicCitation>, offline: true, split: true, tableDesc: {name: triples, families: {triples:={name: triples, max versions: 3, compression: NONE, in memory: false, max length: 2147483647, bloom filter: none}}}}, server: 208.76.44.140:60010, startCode: -647590197693256616
> ...
> 2007-11-13 02:11:47,317 INFO org.apache.hadoop.hbase.HMaster: Deleting region triples,http://purl.org/dc/terms/bibliographicCitation,1194919500610 because daughter splits no longer hold references
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-2203) [hbase] Hung MR job writing hbase

Posted by "stack (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-2203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12553409 ] 

stack commented on HADOOP-2203:
-------------------------------

It was good to meetup last night Vuk. 

Your understanding above of whats going on is about right except the bit where you say the IOE causes the reduce to fail.  The reduce doesn't fail.  It just keeps on going but job never seems to complete.  It looks like the reduce keeps trickling in updates, just enough to get in the way of the TaskTracker killing the job.  

hbase has moved on since I pasted the above logs but last time I tried this job, a few weeks ago, so log output is different now but it still fails to complete; 2 of the 8 reduces finish up fairly promptly but the other 6 hang out for ever seemingly.

The problematic class is using TIF and TOF (Last night I was confused thinking it was doing its own updating and not using TIF/TOF at all).   The class came from a gentleman who was having other hbase issues.  Let me first get permission to post it and data.

> [hbase] Hung MR job writing hbase
> ---------------------------------
>
>                 Key: HADOOP-2203
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2203
>             Project: Hadoop
>          Issue Type: Bug
>          Components: contrib/hbase
>            Reporter: stack
>            Priority: Minor
>
> Running a simple, fairly large job on the cluster, a few of the reduce steps failed; all that did are effectively hung.  Looking at failures, I see this strange output:
> {code}
> 2007-11-13 02:09:09,606 DEBUG org.apache.hadoop.hbase.HTable: reloading table servers because: java.io.IOException: Region triples,http://purl.org/dc/terms/bibliographicCitation,1194919500610 closed
> 	at org.apache.hadoop.hbase.HRegion.startUpdate(HRegion.java:1142)
> 	at org.apache.hadoop.hbase.HRegionServer.startUpdate(HRegionServer.java:1239)
> 	at org.apache.hadoop.hbase.HRegionServer.batchUpdate(HRegionServer.java:1113)
> 	at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> 	at java.lang.reflect.Method.invoke(Method.java:597)
> 	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:379)
> 	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:596)
> 2007-11-13 02:09:19,778 DEBUG org.apache.hadoop.hbase.HTable: reloading table servers because: org.apache.hadoop.hbase.NotServingRegionException: triples,http://purl.org/dc/terms/bibliographicCitation,1194919500610
> 	at org.apache.hadoop.hbase.HRegionServer.getRegion(HRegionServer.java:1325)
> 	at org.apache.hadoop.hbase.HRegionServer.getRegion(HRegionServer.java:1297)
> 	at org.apache.hadoop.hbase.HRegionServer.startUpdate(HRegionServer.java:1238)
> 	at org.apache.hadoop.hbase.HRegionServer.batchUpdate(HRegionServer.java:1113)
> 	at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> 	at java.lang.reflect.Method.invoke(Method.java:597)
> 	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:379)
> 	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:596)
> 2007-11-13 02:09:30,039 DEBUG org.apache.hadoop.hbase.HTable: reloading table servers because: org.apache.hadoop.hbase.NotServingRegionException: triples,http://purl.org/dc/terms/bibliographicCitation,1194919500610
> 	at org.apache.hadoop.hbase.HRegionServer.getRegion(HRegionServer.java:1325)
> 	at org.apache.hadoop.hbase.HRegionServer.getRegion(HRegionServer.java:1297)
> 	at org.apache.hadoop.hbase.HRegionServer.startUpdate(HRegionServer.java:1238)
> 	at org.apache.hadoop.hbase.HRegionServer.batchUpdate(HRegionServer.java:1113)
> 	at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> 	at java.lang.reflect.Method.invoke(Method.java:597)
> 	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:379)
> 	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:596)
> 2007-11-13 02:09:40,275 DEBUG org.apache.hadoop.hbase.HTable: reloading table servers because: org.apache.hadoop.hbase.NotServingRegionException: triples,http://purl.org/dc/terms/bibliographicCitation,1194919500610
> 	at org.apache.hadoop.hbase.HRegionServer.getRegion(HRegionServer.java:1325)
> 	at org.apache.hadoop.hbase.HRegionServer.getRegion(HRegionServer.java:1297)
> 	at org.apache.hadoop.hbase.HRegionServer.startUpdate(HRegionServer.java:1238)
> 	at org.apache.hadoop.hbase.HRegionServer.batchUpdate(HRegionServer.java:1113)
> 	at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> 	at java.lang.reflect.Method.invoke(Method.java:597)
> 	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:379)
> 	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:596)
> 2007-11-13 02:09:55,379 WARN org.apache.hadoop.mapred.TaskTracker: Error running child
> org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: triples,http://purl.org/dc/terms/bibliographicCitation,1194919500610
> 	at org.apache.hadoop.hbase.HRegionServer.getRegion(HRegionServer.java:1325)
> 	at org.apache.hadoop.hbase.HRegionServer.getRegion(HRegionServer.java:1297)
> 	at org.apache.hadoop.hbase.HRegionServer.startUpdate(HRegionServer.java:1238)
> 	at org.apache.hadoop.hbase.HRegionServer.batchUpdate(HRegionServer.java:1113)
> 	at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> 	at java.lang.reflect.Method.invoke(Method.java:597)
> 	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:379)
> 	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:596)
> 	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> 	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> 	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> 	at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> 	at org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:82)
> 	at org.apache.hadoop.hbase.HTable.commit(HTable.java:734)
> 	at org.apache.hadoop.hbase.HTable.commit(HTable.java:706)
> 	at org.apache.hadoop.hbase.mapred.TableOutputFormat$TableRecordWriter.write(TableOutputFormat.java:89)
> 	at org.apache.hadoop.hbase.mapred.TableOutputFormat$TableRecordWriter.write(TableOutputFormat.java:63)
> 	at org.apache.hadoop.mapred.ReduceTask$2.collect(ReduceTask.java:308)
> 	at TriplesTest$TestReducer.reduce(TriplesTest.java:66)
> 	at TriplesTest$TestReducer.reduce(TriplesTest.java:51)
> 	at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:326)
> 	at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1935)
> {code}
> Why is the reducer client being pig-headed trying to go against a region that has been closed?  Why not recalibrate?
> Here is master log from around same time on the problem region:
> {code}
> ...
> 2007-11-13 02:10:17,867 INFO org.apache.hadoop.hbase.HMaster: region triples,http://purl.org/dc/terms/bibliographicCitation,1194919500610 split. New regions are: triples,http://purl.org/dc/terms/bibliographicCitation,1194919707661, triples,http://purl.org/dc/terms/references,1194919707661
> ...
> 2007-11-13 02:11:47,313 DEBUG org.apache.hadoop.hbase.HMaster: HMaster.metaScanner scanner: -3075797136896920808 regioninfo: {regionname: triples,http://purl.org/dc/terms/bibliographicCitation,1194919500610, startKey: <http://purl.org/dc/terms/bibliographicCitation>, offline: true, split: true, tableDesc: {name: triples, families: {triples:={name: triples, max versions: 3, compression: NONE, in memory: false, max length: 2147483647, bloom filter: none}}}}, server: 208.76.44.140:60010, startCode: -647590197693256616
> ...
> 2007-11-13 02:11:47,317 INFO org.apache.hadoop.hbase.HMaster: Deleting region triples,http://purl.org/dc/terms/bibliographicCitation,1194919500610 because daughter splits no longer hold references
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.