You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Raghu Angadi (JIRA)" <ji...@apache.org> on 2007/10/29 23:32:51 UTC
[jira] Created: (HADOOP-2121) Unexpected IOException in
DFSOutputStream.close()
Unexpected IOException in DFSOutputStream.close()
-------------------------------------------------
Key: HADOOP-2121
URL: https://issues.apache.org/jira/browse/HADOOP-2121
Project: Hadoop
Issue Type: Bug
Reporter: Raghu Angadi
Fix For: 0.14.3
While running a test with datanodes with disk space limitations, Hairong noticed many IOExceptions like this :
{noformat}
java.io.IOException: Mismatch in writeChunk() args
at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.writeChunk(DFSClient.java:1575)
at org.apache.hadoop.fs.FSOutputSummer.writeChecksumChunk(FSOutputSummer.java:140)
at org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:122)
at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.close(DFSClient.java:1715)
at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:49)
at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:64)
at org.apache.hadoop.io.SequenceFile$Writer.close(SequenceFile.java:918)
at org.apache.hadoop.mapred.SequenceFileOutputFormat$1.close(SequenceFileOutputFormat.java:72)
at org.apache.hadoop.mapred.MapTask$DirectMapOutputCollector.close(MapTask.java:232)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:197)
at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1935)
{noformat}
I will submit a patch. With the patch, we will still see an IOException, but an expected one.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2121) Unexpected IOException in
DFSOutputStream.close()
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12541495 ]
Hadoop QA commented on HADOOP-2121:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12368704/HADOOP-2121.patch
against trunk revision r593665.
@author +1. The patch does not contain any @author tags.
javadoc +1. The javadoc tool did not generate any warning messages.
javac +1. The applied patch does not generate any new compiler warnings.
findbugs -1. The patch appears to introduce 1 new Findbugs warnings.
core tests +1. The patch passed core unit tests.
contrib tests +1. The patch passed contrib unit tests.
Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1084/testReport/
Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1084/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1084/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1084/console
This message is automatically generated.
> Unexpected IOException in DFSOutputStream.close()
> -------------------------------------------------
>
> Key: HADOOP-2121
> URL: https://issues.apache.org/jira/browse/HADOOP-2121
> Project: Hadoop
> Issue Type: Bug
> Affects Versions: 0.14.3
> Reporter: Raghu Angadi
> Assignee: Raghu Angadi
> Fix For: 0.16.0
>
> Attachments: HADOOP-2121.patch, HADOOP-2121.patch
>
>
> While running a test with datanodes with disk space limitations, Hairong noticed many IOExceptions like this :
> {noformat}
> java.io.IOException: Mismatch in writeChunk() args
> at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.writeChunk(DFSClient.java:1575)
> at org.apache.hadoop.fs.FSOutputSummer.writeChecksumChunk(FSOutputSummer.java:140)
> at org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:122)
> at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.close(DFSClient.java:1715)
> at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:49)
> at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:64)
> at org.apache.hadoop.io.SequenceFile$Writer.close(SequenceFile.java:918)
> at org.apache.hadoop.mapred.SequenceFileOutputFormat$1.close(SequenceFileOutputFormat.java:72)
> at org.apache.hadoop.mapred.MapTask$DirectMapOutputCollector.close(MapTask.java:232)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:197)
> at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1935)
> {noformat}
> I will submit a patch. With the patch, we will still see an IOException, but an expected one.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2121) Unexpected IOException in
DFSOutputStream.close()
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raghu Angadi updated HADOOP-2121:
---------------------------------
Attachment: HADOOP-2121.patch
fixed a findbugs warning.
> Unexpected IOException in DFSOutputStream.close()
> -------------------------------------------------
>
> Key: HADOOP-2121
> URL: https://issues.apache.org/jira/browse/HADOOP-2121
> Project: Hadoop
> Issue Type: Bug
> Affects Versions: 0.14.3
> Reporter: Raghu Angadi
> Assignee: Raghu Angadi
> Fix For: 0.16.0
>
> Attachments: HADOOP-2121.patch, HADOOP-2121.patch, HADOOP-2121.patch
>
>
> While running a test with datanodes with disk space limitations, Hairong noticed many IOExceptions like this :
> {noformat}
> java.io.IOException: Mismatch in writeChunk() args
> at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.writeChunk(DFSClient.java:1575)
> at org.apache.hadoop.fs.FSOutputSummer.writeChecksumChunk(FSOutputSummer.java:140)
> at org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:122)
> at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.close(DFSClient.java:1715)
> at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:49)
> at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:64)
> at org.apache.hadoop.io.SequenceFile$Writer.close(SequenceFile.java:918)
> at org.apache.hadoop.mapred.SequenceFileOutputFormat$1.close(SequenceFileOutputFormat.java:72)
> at org.apache.hadoop.mapred.MapTask$DirectMapOutputCollector.close(MapTask.java:232)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:197)
> at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1935)
> {noformat}
> I will submit a patch. With the patch, we will still see an IOException, but an expected one.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2121) Unexpected IOException in
DFSOutputStream.close()
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raghu Angadi updated HADOOP-2121:
---------------------------------
Attachment: HADOOP-2121.patch
We still need to confirm, but I think what happened is that, {{out.write()}} failed because of lack of space on Datanodes and app did a {{out.close()}} after that. But inside {{clode()}}, FSOutputSummer ended up invoking lower level write() with stale data. The attached patch fixes that.
> Unexpected IOException in DFSOutputStream.close()
> -------------------------------------------------
>
> Key: HADOOP-2121
> URL: https://issues.apache.org/jira/browse/HADOOP-2121
> Project: Hadoop
> Issue Type: Bug
> Reporter: Raghu Angadi
> Assignee: Raghu Angadi
> Fix For: 0.14.3
>
> Attachments: HADOOP-2121.patch
>
>
> While running a test with datanodes with disk space limitations, Hairong noticed many IOExceptions like this :
> {noformat}
> java.io.IOException: Mismatch in writeChunk() args
> at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.writeChunk(DFSClient.java:1575)
> at org.apache.hadoop.fs.FSOutputSummer.writeChecksumChunk(FSOutputSummer.java:140)
> at org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:122)
> at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.close(DFSClient.java:1715)
> at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:49)
> at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:64)
> at org.apache.hadoop.io.SequenceFile$Writer.close(SequenceFile.java:918)
> at org.apache.hadoop.mapred.SequenceFileOutputFormat$1.close(SequenceFileOutputFormat.java:72)
> at org.apache.hadoop.mapred.MapTask$DirectMapOutputCollector.close(MapTask.java:232)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:197)
> at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1935)
> {noformat}
> I will submit a patch. With the patch, we will still see an IOException, but an expected one.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2121) Unexpected IOException in
DFSOutputStream.close()
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raghu Angadi updated HADOOP-2121:
---------------------------------
Status: Open (was: Patch Available)
> Unexpected IOException in DFSOutputStream.close()
> -------------------------------------------------
>
> Key: HADOOP-2121
> URL: https://issues.apache.org/jira/browse/HADOOP-2121
> Project: Hadoop
> Issue Type: Bug
> Affects Versions: 0.14.3
> Reporter: Raghu Angadi
> Assignee: Raghu Angadi
> Fix For: 0.16.0
>
> Attachments: HADOOP-2121.patch, HADOOP-2121.patch
>
>
> While running a test with datanodes with disk space limitations, Hairong noticed many IOExceptions like this :
> {noformat}
> java.io.IOException: Mismatch in writeChunk() args
> at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.writeChunk(DFSClient.java:1575)
> at org.apache.hadoop.fs.FSOutputSummer.writeChecksumChunk(FSOutputSummer.java:140)
> at org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:122)
> at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.close(DFSClient.java:1715)
> at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:49)
> at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:64)
> at org.apache.hadoop.io.SequenceFile$Writer.close(SequenceFile.java:918)
> at org.apache.hadoop.mapred.SequenceFileOutputFormat$1.close(SequenceFileOutputFormat.java:72)
> at org.apache.hadoop.mapred.MapTask$DirectMapOutputCollector.close(MapTask.java:232)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:197)
> at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1935)
> {noformat}
> I will submit a patch. With the patch, we will still see an IOException, but an expected one.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2121) Unexpected IOException in
DFSOutputStream.close()
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raghu Angadi updated HADOOP-2121:
---------------------------------
Status: Patch Available (was: Open)
Thanks Hairong.
> Unexpected IOException in DFSOutputStream.close()
> -------------------------------------------------
>
> Key: HADOOP-2121
> URL: https://issues.apache.org/jira/browse/HADOOP-2121
> Project: Hadoop
> Issue Type: Bug
> Affects Versions: 0.14.3
> Reporter: Raghu Angadi
> Assignee: Raghu Angadi
> Fix For: 0.16.0
>
> Attachments: HADOOP-2121.patch, HADOOP-2121.patch
>
>
> While running a test with datanodes with disk space limitations, Hairong noticed many IOExceptions like this :
> {noformat}
> java.io.IOException: Mismatch in writeChunk() args
> at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.writeChunk(DFSClient.java:1575)
> at org.apache.hadoop.fs.FSOutputSummer.writeChecksumChunk(FSOutputSummer.java:140)
> at org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:122)
> at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.close(DFSClient.java:1715)
> at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:49)
> at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:64)
> at org.apache.hadoop.io.SequenceFile$Writer.close(SequenceFile.java:918)
> at org.apache.hadoop.mapred.SequenceFileOutputFormat$1.close(SequenceFileOutputFormat.java:72)
> at org.apache.hadoop.mapred.MapTask$DirectMapOutputCollector.close(MapTask.java:232)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:197)
> at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1935)
> {noformat}
> I will submit a patch. With the patch, we will still see an IOException, but an expected one.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2121) Unexpected IOException in
DFSOutputStream.close()
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12541453 ]
Hairong Kuang commented on HADOOP-2121:
---------------------------------------
+1 The patch looks good.
> Unexpected IOException in DFSOutputStream.close()
> -------------------------------------------------
>
> Key: HADOOP-2121
> URL: https://issues.apache.org/jira/browse/HADOOP-2121
> Project: Hadoop
> Issue Type: Bug
> Affects Versions: 0.14.3
> Reporter: Raghu Angadi
> Assignee: Raghu Angadi
> Fix For: 0.16.0
>
> Attachments: HADOOP-2121.patch, HADOOP-2121.patch
>
>
> While running a test with datanodes with disk space limitations, Hairong noticed many IOExceptions like this :
> {noformat}
> java.io.IOException: Mismatch in writeChunk() args
> at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.writeChunk(DFSClient.java:1575)
> at org.apache.hadoop.fs.FSOutputSummer.writeChecksumChunk(FSOutputSummer.java:140)
> at org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:122)
> at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.close(DFSClient.java:1715)
> at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:49)
> at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:64)
> at org.apache.hadoop.io.SequenceFile$Writer.close(SequenceFile.java:918)
> at org.apache.hadoop.mapred.SequenceFileOutputFormat$1.close(SequenceFileOutputFormat.java:72)
> at org.apache.hadoop.mapred.MapTask$DirectMapOutputCollector.close(MapTask.java:232)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:197)
> at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1935)
> {noformat}
> I will submit a patch. With the patch, we will still see an IOException, but an expected one.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2121) Unexpected IOException in
DFSOutputStream.close()
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raghu Angadi updated HADOOP-2121:
---------------------------------
Status: Patch Available (was: Open)
> Unexpected IOException in DFSOutputStream.close()
> -------------------------------------------------
>
> Key: HADOOP-2121
> URL: https://issues.apache.org/jira/browse/HADOOP-2121
> Project: Hadoop
> Issue Type: Bug
> Affects Versions: 0.14.3
> Reporter: Raghu Angadi
> Assignee: Raghu Angadi
> Fix For: 0.16.0
>
> Attachments: HADOOP-2121.patch, HADOOP-2121.patch, HADOOP-2121.patch
>
>
> While running a test with datanodes with disk space limitations, Hairong noticed many IOExceptions like this :
> {noformat}
> java.io.IOException: Mismatch in writeChunk() args
> at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.writeChunk(DFSClient.java:1575)
> at org.apache.hadoop.fs.FSOutputSummer.writeChecksumChunk(FSOutputSummer.java:140)
> at org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:122)
> at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.close(DFSClient.java:1715)
> at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:49)
> at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:64)
> at org.apache.hadoop.io.SequenceFile$Writer.close(SequenceFile.java:918)
> at org.apache.hadoop.mapred.SequenceFileOutputFormat$1.close(SequenceFileOutputFormat.java:72)
> at org.apache.hadoop.mapred.MapTask$DirectMapOutputCollector.close(MapTask.java:232)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:197)
> at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1935)
> {noformat}
> I will submit a patch. With the patch, we will still see an IOException, but an expected one.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2121) Unexpected IOException in
DFSOutputStream.close()
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12542217 ]
Hudson commented on HADOOP-2121:
--------------------------------
Integrated in Hadoop-Nightly #302 (See [http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Nightly/302/])
> Unexpected IOException in DFSOutputStream.close()
> -------------------------------------------------
>
> Key: HADOOP-2121
> URL: https://issues.apache.org/jira/browse/HADOOP-2121
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.14.3
> Reporter: Raghu Angadi
> Assignee: Raghu Angadi
> Fix For: 0.16.0
>
> Attachments: HADOOP-2121.patch, HADOOP-2121.patch, HADOOP-2121.patch
>
>
> While running a test with datanodes with disk space limitations, Hairong noticed many IOExceptions like this :
> {noformat}
> java.io.IOException: Mismatch in writeChunk() args
> at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.writeChunk(DFSClient.java:1575)
> at org.apache.hadoop.fs.FSOutputSummer.writeChecksumChunk(FSOutputSummer.java:140)
> at org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:122)
> at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.close(DFSClient.java:1715)
> at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:49)
> at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:64)
> at org.apache.hadoop.io.SequenceFile$Writer.close(SequenceFile.java:918)
> at org.apache.hadoop.mapred.SequenceFileOutputFormat$1.close(SequenceFileOutputFormat.java:72)
> at org.apache.hadoop.mapred.MapTask$DirectMapOutputCollector.close(MapTask.java:232)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:197)
> at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1935)
> {noformat}
> I will submit a patch. With the patch, we will still see an IOException, but an expected one.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HADOOP-2121) Unexpected IOException in
DFSOutputStream.close()
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raghu Angadi reassigned HADOOP-2121:
------------------------------------
Assignee: Raghu Angadi
> Unexpected IOException in DFSOutputStream.close()
> -------------------------------------------------
>
> Key: HADOOP-2121
> URL: https://issues.apache.org/jira/browse/HADOOP-2121
> Project: Hadoop
> Issue Type: Bug
> Reporter: Raghu Angadi
> Assignee: Raghu Angadi
> Fix For: 0.14.3
>
>
> While running a test with datanodes with disk space limitations, Hairong noticed many IOExceptions like this :
> {noformat}
> java.io.IOException: Mismatch in writeChunk() args
> at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.writeChunk(DFSClient.java:1575)
> at org.apache.hadoop.fs.FSOutputSummer.writeChecksumChunk(FSOutputSummer.java:140)
> at org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:122)
> at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.close(DFSClient.java:1715)
> at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:49)
> at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:64)
> at org.apache.hadoop.io.SequenceFile$Writer.close(SequenceFile.java:918)
> at org.apache.hadoop.mapred.SequenceFileOutputFormat$1.close(SequenceFileOutputFormat.java:72)
> at org.apache.hadoop.mapred.MapTask$DirectMapOutputCollector.close(MapTask.java:232)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:197)
> at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1935)
> {noformat}
> I will submit a patch. With the patch, we will still see an IOException, but an expected one.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2121) Unexpected IOException in
DFSOutputStream.close()
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12541584 ]
Hadoop QA commented on HADOOP-2121:
-----------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12369309/HADOOP-2121.patch
against trunk revision r593748.
@author +1. The patch does not contain any @author tags.
javadoc +1. The javadoc tool did not generate any warning messages.
javac +1. The applied patch does not generate any new compiler warnings.
findbugs +1. The patch does not introduce any new Findbugs warnings.
core tests +1. The patch passed core unit tests.
contrib tests +1. The patch passed contrib unit tests.
Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1089/testReport/
Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1089/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1089/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1089/console
This message is automatically generated.
> Unexpected IOException in DFSOutputStream.close()
> -------------------------------------------------
>
> Key: HADOOP-2121
> URL: https://issues.apache.org/jira/browse/HADOOP-2121
> Project: Hadoop
> Issue Type: Bug
> Affects Versions: 0.14.3
> Reporter: Raghu Angadi
> Assignee: Raghu Angadi
> Fix For: 0.16.0
>
> Attachments: HADOOP-2121.patch, HADOOP-2121.patch, HADOOP-2121.patch
>
>
> While running a test with datanodes with disk space limitations, Hairong noticed many IOExceptions like this :
> {noformat}
> java.io.IOException: Mismatch in writeChunk() args
> at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.writeChunk(DFSClient.java:1575)
> at org.apache.hadoop.fs.FSOutputSummer.writeChecksumChunk(FSOutputSummer.java:140)
> at org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:122)
> at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.close(DFSClient.java:1715)
> at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:49)
> at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:64)
> at org.apache.hadoop.io.SequenceFile$Writer.close(SequenceFile.java:918)
> at org.apache.hadoop.mapred.SequenceFileOutputFormat$1.close(SequenceFileOutputFormat.java:72)
> at org.apache.hadoop.mapred.MapTask$DirectMapOutputCollector.close(MapTask.java:232)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:197)
> at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1935)
> {noformat}
> I will submit a patch. With the patch, we will still see an IOException, but an expected one.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2121) Unexpected IOException in
DFSOutputStream.close()
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HADOOP-2121:
-------------------------------------
Component/s: dfs
> Unexpected IOException in DFSOutputStream.close()
> -------------------------------------------------
>
> Key: HADOOP-2121
> URL: https://issues.apache.org/jira/browse/HADOOP-2121
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.14.3
> Reporter: Raghu Angadi
> Assignee: Raghu Angadi
> Fix For: 0.16.0
>
> Attachments: HADOOP-2121.patch, HADOOP-2121.patch, HADOOP-2121.patch
>
>
> While running a test with datanodes with disk space limitations, Hairong noticed many IOExceptions like this :
> {noformat}
> java.io.IOException: Mismatch in writeChunk() args
> at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.writeChunk(DFSClient.java:1575)
> at org.apache.hadoop.fs.FSOutputSummer.writeChecksumChunk(FSOutputSummer.java:140)
> at org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:122)
> at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.close(DFSClient.java:1715)
> at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:49)
> at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:64)
> at org.apache.hadoop.io.SequenceFile$Writer.close(SequenceFile.java:918)
> at org.apache.hadoop.mapred.SequenceFileOutputFormat$1.close(SequenceFileOutputFormat.java:72)
> at org.apache.hadoop.mapred.MapTask$DirectMapOutputCollector.close(MapTask.java:232)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:197)
> at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1935)
> {noformat}
> I will submit a patch. With the patch, we will still see an IOException, but an expected one.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2121) Unexpected IOException in
DFSOutputStream.close()
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raghu Angadi updated HADOOP-2121:
---------------------------------
Attachment: HADOOP-2121.patch
Updated patch makes sure clean up is done in {{DFSOutputStream.close()}} even when there are other errors.
> Unexpected IOException in DFSOutputStream.close()
> -------------------------------------------------
>
> Key: HADOOP-2121
> URL: https://issues.apache.org/jira/browse/HADOOP-2121
> Project: Hadoop
> Issue Type: Bug
> Affects Versions: 0.14.3
> Reporter: Raghu Angadi
> Assignee: Raghu Angadi
> Fix For: 0.16.0
>
> Attachments: HADOOP-2121.patch, HADOOP-2121.patch
>
>
> While running a test with datanodes with disk space limitations, Hairong noticed many IOExceptions like this :
> {noformat}
> java.io.IOException: Mismatch in writeChunk() args
> at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.writeChunk(DFSClient.java:1575)
> at org.apache.hadoop.fs.FSOutputSummer.writeChecksumChunk(FSOutputSummer.java:140)
> at org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:122)
> at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.close(DFSClient.java:1715)
> at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:49)
> at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:64)
> at org.apache.hadoop.io.SequenceFile$Writer.close(SequenceFile.java:918)
> at org.apache.hadoop.mapred.SequenceFileOutputFormat$1.close(SequenceFileOutputFormat.java:72)
> at org.apache.hadoop.mapred.MapTask$DirectMapOutputCollector.close(MapTask.java:232)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:197)
> at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1935)
> {noformat}
> I will submit a patch. With the patch, we will still see an IOException, but an expected one.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2121) Unexpected IOException in
DFSOutputStream.close()
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raghu Angadi updated HADOOP-2121:
---------------------------------
Fix Version/s: (was: 0.14.3)
0.16.0
Affects Version/s: 0.14.3
> Unexpected IOException in DFSOutputStream.close()
> -------------------------------------------------
>
> Key: HADOOP-2121
> URL: https://issues.apache.org/jira/browse/HADOOP-2121
> Project: Hadoop
> Issue Type: Bug
> Affects Versions: 0.14.3
> Reporter: Raghu Angadi
> Assignee: Raghu Angadi
> Fix For: 0.16.0
>
> Attachments: HADOOP-2121.patch
>
>
> While running a test with datanodes with disk space limitations, Hairong noticed many IOExceptions like this :
> {noformat}
> java.io.IOException: Mismatch in writeChunk() args
> at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.writeChunk(DFSClient.java:1575)
> at org.apache.hadoop.fs.FSOutputSummer.writeChecksumChunk(FSOutputSummer.java:140)
> at org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:122)
> at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.close(DFSClient.java:1715)
> at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:49)
> at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:64)
> at org.apache.hadoop.io.SequenceFile$Writer.close(SequenceFile.java:918)
> at org.apache.hadoop.mapred.SequenceFileOutputFormat$1.close(SequenceFileOutputFormat.java:72)
> at org.apache.hadoop.mapred.MapTask$DirectMapOutputCollector.close(MapTask.java:232)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:197)
> at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1935)
> {noformat}
> I will submit a patch. With the patch, we will still see an IOException, but an expected one.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2121) Unexpected IOException in
DFSOutputStream.close()
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HADOOP-2121:
-------------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
I just committed this. Thanks Raghu!
> Unexpected IOException in DFSOutputStream.close()
> -------------------------------------------------
>
> Key: HADOOP-2121
> URL: https://issues.apache.org/jira/browse/HADOOP-2121
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.14.3
> Reporter: Raghu Angadi
> Assignee: Raghu Angadi
> Fix For: 0.16.0
>
> Attachments: HADOOP-2121.patch, HADOOP-2121.patch, HADOOP-2121.patch
>
>
> While running a test with datanodes with disk space limitations, Hairong noticed many IOExceptions like this :
> {noformat}
> java.io.IOException: Mismatch in writeChunk() args
> at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.writeChunk(DFSClient.java:1575)
> at org.apache.hadoop.fs.FSOutputSummer.writeChecksumChunk(FSOutputSummer.java:140)
> at org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:122)
> at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.close(DFSClient.java:1715)
> at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:49)
> at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:64)
> at org.apache.hadoop.io.SequenceFile$Writer.close(SequenceFile.java:918)
> at org.apache.hadoop.mapred.SequenceFileOutputFormat$1.close(SequenceFileOutputFormat.java:72)
> at org.apache.hadoop.mapred.MapTask$DirectMapOutputCollector.close(MapTask.java:232)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:197)
> at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1935)
> {noformat}
> I will submit a patch. With the patch, we will still see an IOException, but an expected one.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.