You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Yongjun Zhang (JIRA)" <ji...@apache.org> on 2014/09/16 17:19:34 UTC
[jira] [Updated] (HDFS-7070) TestWebHdfsFileSystemContract fails
occassionally
[ https://issues.apache.org/jira/browse/HDFS-7070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yongjun Zhang updated HDFS-7070:
--------------------------------
Description:
org.apache.hadoop.hdfs.web.TestWebHdfsFileSystemContract.testResponseCode
and org.apache.hadoop.hdfs.web.TestWebHdfsFileSystemContract.testRenameDirToSelf
failed recently.
Need to determine whether it's introduced by some latest code change due to file descriptor leak; or it's a similar issue as HDFS-6694 reported.
E.g. https://builds.apache.org/job/PreCommit-HDFS-Build/8026/testReport/org.apache.hadoop.hdfs.web/TestWebHdfsFileSystemContract/testResponseCode/.
{code}
2014-09-15 12:52:18,866 INFO datanode.DataNode (DataXceiver.java:writeBlock(749)) - opWriteBlock BP-23833599-67.195.81.147-1410785517350:blk_1073741827_1461 received exception java.io.IOException: Cannot run program "stat": java.io.IOException: error=24, Too many open files
2014-09-15 12:52:18,867 ERROR datanode.DataNode (DataXceiver.java:run(243)) - 127.0.0.1:47221:DataXceiver error processing WRITE_BLOCK operation src: /127.0.0.1:38112 dst: /127.0.0.1:47221
java.io.IOException: Cannot run program "stat": java.io.IOException: error=24, Too many open files
at java.lang.ProcessBuilder.start(ProcessBuilder.java:470)
at org.apache.hadoop.util.Shell.runCommand(Shell.java:485)
at org.apache.hadoop.util.Shell.run(Shell.java:455)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702)
at org.apache.hadoop.fs.HardLink.getLinkCount(HardLink.java:495)
at org.apache.hadoop.hdfs.server.datanode.ReplicaInfo.unlinkBlock(ReplicaInfo.java:288)
at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.append(FsDatasetImpl.java:702)
at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.append(FsDatasetImpl.java:680)
at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.append(FsDatasetImpl.java:101)
at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.<init>(BlockReceiver.java:193)
at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:604)
at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:126)
at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:72)
at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:225)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.IOException: java.io.IOException: error=24, Too many open files
at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
at java.lang.ProcessImpl.start(ProcessImpl.java:65)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:452)
... 14 more
2014-09-15 12:52:18,867 INFO hdfs.DFSClient (DFSOutputStream.java:createBlockOutputStream(1400)) - Exception in createBlockOutputStream
java.io.EOFException: Premature EOF: no length prefix available
at org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:2101)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1368)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1210)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:530)
2014-09-15 12:52:18,870 WARN hdfs.DFSClient (DFSOutputStream.java:run(883)) - DFSOutputStream ResponseProcessor exception for block BP-23833599-67.195.81.147-1410785517350:blk_1073741827_1461
java.lang.NullPointerException
at org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:2099)
at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:176)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:798)
2014-09-15 12:52:18,870 WARN hdfs.DFSClient (DFSOutputStream.java:run(627)) - DataStreamer Exception
java.lang.NullPointerException
at org.apache.hadoop.hdfs.DFSOutputStream$Packet.writeTo(DFSOutputStream.java:273)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:579)
{code}
was:
org.apache.hadoop.hdfs.web.TestWebHdfsFileSystemContract.testResponseCode
and org.apache.hadoop.hdfs.web.TestWebHdfsFileSystemContract.testRenameDirToSelf
failed recently, maybe introduced by some latest code change due to file descriptor leak (?)
E.g. https://builds.apache.org/job/PreCommit-HDFS-Build/8026/testReport/org.apache.hadoop.hdfs.web/TestWebHdfsFileSystemContract/testResponseCode/.
{code}
2014-09-15 12:52:18,866 INFO datanode.DataNode (DataXceiver.java:writeBlock(749)) - opWriteBlock BP-23833599-67.195.81.147-1410785517350:blk_1073741827_1461 received exception java.io.IOException: Cannot run program "stat": java.io.IOException: error=24, Too many open files
2014-09-15 12:52:18,867 ERROR datanode.DataNode (DataXceiver.java:run(243)) - 127.0.0.1:47221:DataXceiver error processing WRITE_BLOCK operation src: /127.0.0.1:38112 dst: /127.0.0.1:47221
java.io.IOException: Cannot run program "stat": java.io.IOException: error=24, Too many open files
at java.lang.ProcessBuilder.start(ProcessBuilder.java:470)
at org.apache.hadoop.util.Shell.runCommand(Shell.java:485)
at org.apache.hadoop.util.Shell.run(Shell.java:455)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702)
at org.apache.hadoop.fs.HardLink.getLinkCount(HardLink.java:495)
at org.apache.hadoop.hdfs.server.datanode.ReplicaInfo.unlinkBlock(ReplicaInfo.java:288)
at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.append(FsDatasetImpl.java:702)
at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.append(FsDatasetImpl.java:680)
at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.append(FsDatasetImpl.java:101)
at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.<init>(BlockReceiver.java:193)
at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:604)
at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:126)
at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:72)
at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:225)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.IOException: java.io.IOException: error=24, Too many open files
at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
at java.lang.ProcessImpl.start(ProcessImpl.java:65)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:452)
... 14 more
2014-09-15 12:52:18,867 INFO hdfs.DFSClient (DFSOutputStream.java:createBlockOutputStream(1400)) - Exception in createBlockOutputStream
java.io.EOFException: Premature EOF: no length prefix available
at org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:2101)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1368)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1210)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:530)
2014-09-15 12:52:18,870 WARN hdfs.DFSClient (DFSOutputStream.java:run(883)) - DFSOutputStream ResponseProcessor exception for block BP-23833599-67.195.81.147-1410785517350:blk_1073741827_1461
java.lang.NullPointerException
at org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:2099)
at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:176)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:798)
2014-09-15 12:52:18,870 WARN hdfs.DFSClient (DFSOutputStream.java:run(627)) - DataStreamer Exception
java.lang.NullPointerException
at org.apache.hadoop.hdfs.DFSOutputStream$Packet.writeTo(DFSOutputStream.java:273)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:579)
{code}
> TestWebHdfsFileSystemContract fails occassionally
> -------------------------------------------------
>
> Key: HDFS-7070
> URL: https://issues.apache.org/jira/browse/HDFS-7070
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
> Affects Versions: 2.6.0
> Reporter: Yongjun Zhang
> Assignee: Yongjun Zhang
>
> org.apache.hadoop.hdfs.web.TestWebHdfsFileSystemContract.testResponseCode
> and org.apache.hadoop.hdfs.web.TestWebHdfsFileSystemContract.testRenameDirToSelf
> failed recently.
> Need to determine whether it's introduced by some latest code change due to file descriptor leak; or it's a similar issue as HDFS-6694 reported.
> E.g. https://builds.apache.org/job/PreCommit-HDFS-Build/8026/testReport/org.apache.hadoop.hdfs.web/TestWebHdfsFileSystemContract/testResponseCode/.
> {code}
> 2014-09-15 12:52:18,866 INFO datanode.DataNode (DataXceiver.java:writeBlock(749)) - opWriteBlock BP-23833599-67.195.81.147-1410785517350:blk_1073741827_1461 received exception java.io.IOException: Cannot run program "stat": java.io.IOException: error=24, Too many open files
> 2014-09-15 12:52:18,867 ERROR datanode.DataNode (DataXceiver.java:run(243)) - 127.0.0.1:47221:DataXceiver error processing WRITE_BLOCK operation src: /127.0.0.1:38112 dst: /127.0.0.1:47221
> java.io.IOException: Cannot run program "stat": java.io.IOException: error=24, Too many open files
> at java.lang.ProcessBuilder.start(ProcessBuilder.java:470)
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:485)
> at org.apache.hadoop.util.Shell.run(Shell.java:455)
> at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702)
> at org.apache.hadoop.fs.HardLink.getLinkCount(HardLink.java:495)
> at org.apache.hadoop.hdfs.server.datanode.ReplicaInfo.unlinkBlock(ReplicaInfo.java:288)
> at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.append(FsDatasetImpl.java:702)
> at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.append(FsDatasetImpl.java:680)
> at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.append(FsDatasetImpl.java:101)
> at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.<init>(BlockReceiver.java:193)
> at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:604)
> at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:126)
> at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:72)
> at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:225)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: java.io.IOException: java.io.IOException: error=24, Too many open files
> at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
> at java.lang.ProcessImpl.start(ProcessImpl.java:65)
> at java.lang.ProcessBuilder.start(ProcessBuilder.java:452)
> ... 14 more
> 2014-09-15 12:52:18,867 INFO hdfs.DFSClient (DFSOutputStream.java:createBlockOutputStream(1400)) - Exception in createBlockOutputStream
> java.io.EOFException: Premature EOF: no length prefix available
> at org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:2101)
> at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1368)
> at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1210)
> at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:530)
> 2014-09-15 12:52:18,870 WARN hdfs.DFSClient (DFSOutputStream.java:run(883)) - DFSOutputStream ResponseProcessor exception for block BP-23833599-67.195.81.147-1410785517350:blk_1073741827_1461
> java.lang.NullPointerException
> at org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:2099)
> at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:176)
> at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:798)
> 2014-09-15 12:52:18,870 WARN hdfs.DFSClient (DFSOutputStream.java:run(627)) - DataStreamer Exception
> java.lang.NullPointerException
> at org.apache.hadoop.hdfs.DFSOutputStream$Packet.writeTo(DFSOutputStream.java:273)
> at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:579)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)