You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by YouPeng Yang <yy...@gmail.com> on 2013/06/07 15:00:01 UTC
protobuf.ServiceException: OutOfMemoryError
Hi all
I find that some if my DNs go to dead. and the datanode log shows as [1]:
I got the java.lang.OutOfMemoryError: Java heap space.
I wander how this could come out.
[1]:
1496 at file
/home/hadoop/datadir/current/BP-471453121-172.16.250.16-1369298226760/current/finalized/subdir56/subdir46/blk_8143366700047777895
2013-06-07 13:49:35,526 INFO
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService:
Deleted block BP-471453121-172.16.250.16-1369298226760
blk_7448905973553274685_691506 at file
/home/hadoop/datadir/current/BP-471453121-172.16.250.16-1369298226760/current/finalized/subdir56/subdir46/blk_7448905973553274685
2013-06-07 13:49:35,526 INFO
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService:
Deleted block BP-471453121-172.16.250.16-1369298226760
blk_-1754381991139616539_691508 at file
/home/hadoop/datadir/current/BP-471453121-172.16.250.16-1369298226760/current/finalized/subdir56/subdir46/blk_-1754381991139616539
2013-06-07 13:49:35,527 INFO
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService:
Deleted block BP-471453121-172.16.250.16-1369298226760
blk_-3379745807160766937_691492 at file
/home/hadoop/datadir/current/BP-471453121-172.16.250.16-1369298226760/current/finalized/subdir56/subdir46/blk_-3379745807160766937
2013-06-07 13:51:44,231 WARN
org.apache.hadoop.hdfs.server.datanode.DataNode: IOException in offerService
java.io.IOException: com.google.protobuf.ServiceException:
java.lang.OutOfMemoryError: Java heap space
at
org.apache.hadoop.ipc.ProtobufHelper.getRemoteException(ProtobufHelper.java:47)
at
org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolClientSideTranslatorPB.blockReport(DatanodeProtocolClientSideTranslatorPB.java:203)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.blockReport(BPServiceActor.java:399)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:551)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:674)
at java.lang.Thread.run(Thread.java:662)
Caused by: com.google.protobuf.ServiceException:
java.lang.OutOfMemoryError: Java heap space
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:212)
at $Proxy10.blockReport(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:164)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:83)
at $Proxy10.blockReport(Unknown Source)
at
org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolClientSideTranslatorPB.blockReport(DatanodeProtocolClientSideTranslatorPB.java:201)
... 4 more
Caused by: java.lang.OutOfMemoryError: Java heap space
2013-06-07 13:51:46,983 WARN
org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected exception in
block pool Block pool BP-471453121-172.16.250.16-1369298226760 (storage id
DS-1482330176-172.16.250.19-50010-1369298284371) service to wxossetl2/
172.16.250.16:8020
java.lang.OutOfMemoryError: Java heap space
2013-06-07 13:51:46,983 WARN
org.apache.hadoop.hdfs.server.datanode.DataNode: Ending block pool service
for: Block pool BP-471453121-172.16.250.16-1369298226760 (storage id
DS-1482330176-172.16.250.19-50010-1369298284371) service to wxossetl2/
172.16.250.16:8020
2013-06-07 13:51:47,090 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: Removed Block pool
BP-471453121-172.16.250.16-1369298226760 (storage id
DS-1482330176-172.16.250.19-50010-1369298284371)
2013-06-07 13:51:47,090 INFO
org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Removed
bpid=BP-471453121-172.16.250.16-1369298226760 from blockPoolScannerMap
2013-06-07 13:51:47,090 INFO
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl:
Removing block pool BP-471453121-172.16.250.16-1369298226760
2013-06-07 13:51:49,091 WARN
org.apache.hadoop.hdfs.server.datanode.DataNode: Exiting Datanode
2013-06-07 13:51:49,098 INFO org.apache.hadoop.util.ExitUtil: Exiting with
status 0
2013-06-07 13:51:49,101 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down DataNode at wxossetl5/172.16.250.19