You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Haohui Mai (JIRA)" <ji...@apache.org> on 2013/10/12 06:14:42 UTC
[jira] [Commented] (HDFS-5335) DFSOutputStream#close() keeps
throwing exceptions when it is called multiple times
[ https://issues.apache.org/jira/browse/HDFS-5335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13793239#comment-13793239 ]
Haohui Mai commented on HDFS-5335:
----------------------------------
[~atm], thanks for the reminder. I've updated the summary / descriptions based on the analysis on our sides.
> DFSOutputStream#close() keeps throwing exceptions when it is called multiple times
> ----------------------------------------------------------------------------------
>
> Key: HDFS-5335
> URL: https://issues.apache.org/jira/browse/HDFS-5335
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 2.1.1-beta
> Reporter: Arpit Gupta
> Assignee: Haohui Mai
> Fix For: 2.2.1
>
> Attachments: HDFS-5335.000.patch, HDFS-5335.001.patch
>
>
> The semantic of DFSOutputStream#close() is incorrect.
> Close() will throw the last exception that it encounters during the flush operations when closing the stream. The exception that it remembers never get cleared therefore spurious exceptions arise when close() is called multiple times.
> Here is the stack trace from the client
> {code}
> java.nio.channels.ClosedChannelException
> at org.apache.hadoop.hdfs.DFSOutputStream.checkClosed(DFSOutputStream.java:1317)
> at org.apache.hadoop.hdfs.DFSOutputStream.waitForAckedSeqno(DFSOutputStream.java:1810)
> at org.apache.hadoop.hdfs.DFSOutputStream.flushInternal(DFSOutputStream.java:1789)
> at org.apache.hadoop.hdfs.DFSOutputStream.close(DFSOutputStream.java:1877)
> at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:71)
> at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:104)
> at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:54)
> at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:112)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:366)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:338)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:289)
> at org.apache.hadoop.mapreduce.JobSubmitter.copyRemoteFiles(JobSubmitter.java:139)
> at org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:212)
> at org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:300)
> at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:387)
> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1268)
> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1265)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1265)
> at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:562)
> at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:557)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
> at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:557)
> at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:548)
> at org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:425)
> at org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:136)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:151)
> at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:65)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1414)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1192)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1020)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:888)
> at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:259)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:413)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:348)
> at org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:446)
> at org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:456)
> at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:737)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:675)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:614)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
> Job Submission failed with exception 'java.nio.channels.ClosedChannelException(null)'
> FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
> {code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)