You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Brahma Reddy Battula (JIRA)" <ji...@apache.org> on 2012/07/04 08:16:34 UTC
[jira] [Commented] (HDFS-3600) DataNode is not responding After
throwing java.lang.OutOfMemoryError: Direct buffer memory
[ https://issues.apache.org/jira/browse/HDFS-3600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13406291#comment-13406291 ]
Brahma Reddy Battula commented on HDFS-3600:
--------------------------------------------
Following trace is logged..
{noformat}
012-07-04 03:17:28,934 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-1438589468-HOST**.233-1341244293218:blk_-46591813407025079_624958, type=HAS_DOWNSTREAM_IN_PIPELINE terminating
2012-07-04 03:17:28,937 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: HOST-10-18-40-168:50010:DataXceiver error processing WRITE_BLOCK operation src: /10.18.40.168:13129 dest: /DN**.168:50010
java.lang.OutOfMemoryError: Direct buffer memory
at java.nio.Bits.reserveMemory(Bits.java:632)
at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:97)
at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:288)
at sun.nio.ch.Util.getTemporaryDirectBuffer(Util.java:155)
at sun.nio.ch.IOUtil.read(IOUtil.java:169)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:243)
at org.apache.hadoop.net.SocketInputStream$Reader.performIO(SocketInputStream.java:54)
at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:154)
at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:127)
at java.io.FilterInputStream.read(FilterInputStream.java:116)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
at java.io.DataInputStream.read(DataInputStream.java:132)
at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readToBuf(BlockReceiver.java:405)
at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readNextPacket(BlockReceiver.java:452)
at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:511)
at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:748)
at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:464)
at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:98)
at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:66)
at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:189)
at java.lang.Thread.run(Thread.java:662)
{noformat}
If we increase XX:MaxDirectMemorySize,i think OOME wn't come.currently I configured 1GB..But even then there is chance it will throw..May be native memory leak I am suspecting..
> DataNode is not responding After throwing java.lang.OutOfMemoryError: Direct buffer memory
> ------------------------------------------------------------------------------------------
>
> Key: HDFS-3600
> URL: https://issues.apache.org/jira/browse/HDFS-3600
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Affects Versions: 2.0.1-alpha
> Reporter: Brahma Reddy Battula
>
> Scenario:
> =========
> Started NN with four DN's
> written client program such that it will keep on write,append and read dta with 10 thread.
> After 4 hours ,got OOME.Then DN listed under Dead it's not sending any heartbeats but GC is happening.
> *GC OPTS configured for DN*
> -Xms3G -Xmx4G -XX:NewSize=256M -XX:MaxNewSize=512M -XX:PermSize=128M -XX:MaxPermSize=128M -XX:CMSFullGCsBeforeCompaction=1 -XX:MaxDirectMemorySize=1G -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:+UseCMSCompactAtFullCollection -XX:CMSInitiatingOccupancyFraction=65 -Xloggc:/home/install/hadoop/datanode/logs/datanode-root-gc.log -XX:+PrintGCDetails -XX:+DisableExplicitGC
>
> *CPU usage for DN*
> {noformat}
> Tasks: 1 total, 0 running, 1 sleeping, 0 stopped, 0 zombie
> Cpu(s): 0.1%us, 0.0%sy, 0.0%ni, 99.8%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
> Mem: 15955M total, 15806M used, 148M free, 436M buffers
> Swap: 12284M total, 9M used, 12274M free, 11422M cached
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 7431 root 20 0 6291m 2.6g 13m S 0 16.4 55:20.65 java
> {noformat}
> *JAVA Version*
> {noformat}
> sun.boot.library.path = /root/nodesetup/java/jdk1.6.0_31/jre/lib/amd64
> java version "1.6.0_31"
> Java(TM) SE Runtime Environment (build 1.6.0_31-b04)
> Java HotSpot(TM) 64-Bit Server VM (build 20.6-b01, mixed mode)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira