You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Koji Noguchi (JIRA)" <ji...@apache.org> on 2010/04/07 00:51:33 UTC
[jira] Created: (HDFS-1085) hftp read failing silently
hftp read failing silently
---------------------------
Key: HDFS-1085
URL: https://issues.apache.org/jira/browse/HDFS-1085
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Koji Noguchi
When performing a massive distcp through hftp, we saw many tasks fail with
{quote}
2010-04-06 17:56:43,005 INFO org.apache.hadoop.tools.DistCp: FAIL 2010/0/part-00032 : java.io.IOException: File size not matched: copied 193855488 bytes (184.9m) to tmpfile (=hdfs://omehost.com:8020/somepath/part-00032)
but expected 1710327403 bytes (1.6g) from hftp://someotherhost/somepath/part-00032
at org.apache.hadoop.tools.DistCp$CopyFilesMapper.copy(DistCp.java:435)
at org.apache.hadoop.tools.DistCp$CopyFilesMapper.map(DistCp.java:543)
at org.apache.hadoop.tools.DistCp$CopyFilesMapper.map(DistCp.java:310)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
at org.apache.hadoop.mapred.Child.main(Child.java:159)
{quote}
This means that read itself didn't fail but the resulted file was somehow smaller.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1085) hftp read failing silently
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12888146#action_12888146 ]
Tsz Wo (Nicholas), SZE commented on HDFS-1085:
----------------------------------------------
BTW, thank you for commenting on this, Dmytro.
> hftp read failing silently
> ---------------------------
>
> Key: HDFS-1085
> URL: https://issues.apache.org/jira/browse/HDFS-1085
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Reporter: Koji Noguchi
> Assignee: Tsz Wo (Nicholas), SZE
> Attachments: h1085_20100713.patch
>
>
> When performing a massive distcp through hftp, we saw many tasks fail with
> {quote}
> 2010-04-06 17:56:43,005 INFO org.apache.hadoop.tools.DistCp: FAIL 2010/0/part-00032 : java.io.IOException: File size not matched: copied 193855488 bytes (184.9m) to tmpfile (=hdfs://omehost.com:8020/somepath/part-00032)
> but expected 1710327403 bytes (1.6g) from hftp://someotherhost/somepath/part-00032
> at org.apache.hadoop.tools.DistCp$CopyFilesMapper.copy(DistCp.java:435)
> at org.apache.hadoop.tools.DistCp$CopyFilesMapper.map(DistCp.java:543)
> at org.apache.hadoop.tools.DistCp$CopyFilesMapper.map(DistCp.java:310)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
> at org.apache.hadoop.mapred.Child.main(Child.java:159)
> {quote}
> This means that read itself didn't fail but the resulted file was somehow smaller.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1085) hftp read failing silently
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12889354#action_12889354 ]
Suresh Srinivas commented on HDFS-1085:
---------------------------------------
+1 for the trunk version of the patch.
> hftp read failing silently
> ---------------------------
>
> Key: HDFS-1085
> URL: https://issues.apache.org/jira/browse/HDFS-1085
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Reporter: Koji Noguchi
> Assignee: Tsz Wo (Nicholas), SZE
> Fix For: 0.22.0
>
> Attachments: h1085_20100713.patch, h1085_20100716b_y0.20.1xx.patch, h1085_20100716b_y0.20.1xx_with_test.patch, h1085_20100716c.patch, h1085_20100716c_y0.20.1xx.patch
>
>
> When performing a massive distcp through hftp, we saw many tasks fail with
> {quote}
> 2010-04-06 17:56:43,005 INFO org.apache.hadoop.tools.DistCp: FAIL 2010/0/part-00032 : java.io.IOException: File size not matched: copied 193855488 bytes (184.9m) to tmpfile (=hdfs://omehost.com:8020/somepath/part-00032)
> but expected 1710327403 bytes (1.6g) from hftp://someotherhost/somepath/part-00032
> at org.apache.hadoop.tools.DistCp$CopyFilesMapper.copy(DistCp.java:435)
> at org.apache.hadoop.tools.DistCp$CopyFilesMapper.map(DistCp.java:543)
> at org.apache.hadoop.tools.DistCp$CopyFilesMapper.map(DistCp.java:310)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
> at org.apache.hadoop.mapred.Child.main(Child.java:159)
> {quote}
> This means that read itself didn't fail but the resulted file was somehow smaller.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1085) hftp read failing silently
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12890130#action_12890130 ]
Hudson commented on HDFS-1085:
------------------------------
Integrated in Hadoop-Hdfs-trunk-Commit #346 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/346/])
> hftp read failing silently
> ---------------------------
>
> Key: HDFS-1085
> URL: https://issues.apache.org/jira/browse/HDFS-1085
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Reporter: Koji Noguchi
> Assignee: Tsz Wo (Nicholas), SZE
> Fix For: 0.22.0
>
> Attachments: h1085_20100713.patch, h1085_20100716b_y0.20.1xx.patch, h1085_20100716b_y0.20.1xx_with_test.patch, h1085_20100716c.patch, h1085_20100716c_y0.20.1xx.patch, h1085_20100716d.patch, h1085_20100716d_y0.20.1xx.patch, h1085_20100716d_y0.20.1xx_test.patch
>
>
> When performing a massive distcp through hftp, we saw many tasks fail with
> {quote}
> 2010-04-06 17:56:43,005 INFO org.apache.hadoop.tools.DistCp: FAIL 2010/0/part-00032 : java.io.IOException: File size not matched: copied 193855488 bytes (184.9m) to tmpfile (=hdfs://omehost.com:8020/somepath/part-00032)
> but expected 1710327403 bytes (1.6g) from hftp://someotherhost/somepath/part-00032
> at org.apache.hadoop.tools.DistCp$CopyFilesMapper.copy(DistCp.java:435)
> at org.apache.hadoop.tools.DistCp$CopyFilesMapper.map(DistCp.java:543)
> at org.apache.hadoop.tools.DistCp$CopyFilesMapper.map(DistCp.java:310)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
> at org.apache.hadoop.mapred.Child.main(Child.java:159)
> {quote}
> This means that read itself didn't fail but the resulted file was somehow smaller.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1085) hftp read failing silently
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HDFS-1085:
-----------------------------------------
Attachment: h1085_20100716d_y0.20.1xx_test.patch
h1085_20100716d_y0.20.1xx_test.patch: updated tests
> hftp read failing silently
> ---------------------------
>
> Key: HDFS-1085
> URL: https://issues.apache.org/jira/browse/HDFS-1085
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Reporter: Koji Noguchi
> Assignee: Tsz Wo (Nicholas), SZE
> Fix For: 0.22.0
>
> Attachments: h1085_20100713.patch, h1085_20100716b_y0.20.1xx.patch, h1085_20100716b_y0.20.1xx_with_test.patch, h1085_20100716c.patch, h1085_20100716c_y0.20.1xx.patch, h1085_20100716d_y0.20.1xx.patch, h1085_20100716d_y0.20.1xx_test.patch
>
>
> When performing a massive distcp through hftp, we saw many tasks fail with
> {quote}
> 2010-04-06 17:56:43,005 INFO org.apache.hadoop.tools.DistCp: FAIL 2010/0/part-00032 : java.io.IOException: File size not matched: copied 193855488 bytes (184.9m) to tmpfile (=hdfs://omehost.com:8020/somepath/part-00032)
> but expected 1710327403 bytes (1.6g) from hftp://someotherhost/somepath/part-00032
> at org.apache.hadoop.tools.DistCp$CopyFilesMapper.copy(DistCp.java:435)
> at org.apache.hadoop.tools.DistCp$CopyFilesMapper.map(DistCp.java:543)
> at org.apache.hadoop.tools.DistCp$CopyFilesMapper.map(DistCp.java:310)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
> at org.apache.hadoop.mapred.Child.main(Child.java:159)
> {quote}
> This means that read itself didn't fail but the resulted file was somehow smaller.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1085) hftp read failing silently
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12889351#action_12889351 ]
Suresh Srinivas commented on HDFS-1085:
---------------------------------------
+1 for y20 patch.
> hftp read failing silently
> ---------------------------
>
> Key: HDFS-1085
> URL: https://issues.apache.org/jira/browse/HDFS-1085
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Reporter: Koji Noguchi
> Assignee: Tsz Wo (Nicholas), SZE
> Fix For: 0.22.0
>
> Attachments: h1085_20100713.patch, h1085_20100716b_y0.20.1xx.patch, h1085_20100716b_y0.20.1xx_with_test.patch, h1085_20100716c.patch, h1085_20100716c_y0.20.1xx.patch
>
>
> When performing a massive distcp through hftp, we saw many tasks fail with
> {quote}
> 2010-04-06 17:56:43,005 INFO org.apache.hadoop.tools.DistCp: FAIL 2010/0/part-00032 : java.io.IOException: File size not matched: copied 193855488 bytes (184.9m) to tmpfile (=hdfs://omehost.com:8020/somepath/part-00032)
> but expected 1710327403 bytes (1.6g) from hftp://someotherhost/somepath/part-00032
> at org.apache.hadoop.tools.DistCp$CopyFilesMapper.copy(DistCp.java:435)
> at org.apache.hadoop.tools.DistCp$CopyFilesMapper.map(DistCp.java:543)
> at org.apache.hadoop.tools.DistCp$CopyFilesMapper.map(DistCp.java:310)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
> at org.apache.hadoop.mapred.Child.main(Child.java:159)
> {quote}
> This means that read itself didn't fail but the resulted file was somehow smaller.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1085) hftp read failing silently
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12890005#action_12890005 ]
Tsz Wo (Nicholas), SZE commented on HDFS-1085:
----------------------------------------------
All the failed tests are not related to this. The related JIRA are listed below.
- org.apache.hadoop.hdfs.TestFiHFlush.hFlushFi01_a; see HDFS-1206
- org.apache.hadoop.hdfs.TestFileAppend4; see HDFS-1306
- org.apache.hadoop.hdfs.security.token.block.TestBlockToken.testBlockTokenRpc; see HDFS-1284
- org.apache.hadoop.hdfs.server.common.TestJspHelper.testGetUgi; see HDFS-1285
> hftp read failing silently
> ---------------------------
>
> Key: HDFS-1085
> URL: https://issues.apache.org/jira/browse/HDFS-1085
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Reporter: Koji Noguchi
> Assignee: Tsz Wo (Nicholas), SZE
> Fix For: 0.22.0
>
> Attachments: h1085_20100713.patch, h1085_20100716b_y0.20.1xx.patch, h1085_20100716b_y0.20.1xx_with_test.patch, h1085_20100716c.patch, h1085_20100716c_y0.20.1xx.patch, h1085_20100716d.patch, h1085_20100716d_y0.20.1xx.patch, h1085_20100716d_y0.20.1xx_test.patch
>
>
> When performing a massive distcp through hftp, we saw many tasks fail with
> {quote}
> 2010-04-06 17:56:43,005 INFO org.apache.hadoop.tools.DistCp: FAIL 2010/0/part-00032 : java.io.IOException: File size not matched: copied 193855488 bytes (184.9m) to tmpfile (=hdfs://omehost.com:8020/somepath/part-00032)
> but expected 1710327403 bytes (1.6g) from hftp://someotherhost/somepath/part-00032
> at org.apache.hadoop.tools.DistCp$CopyFilesMapper.copy(DistCp.java:435)
> at org.apache.hadoop.tools.DistCp$CopyFilesMapper.map(DistCp.java:543)
> at org.apache.hadoop.tools.DistCp$CopyFilesMapper.map(DistCp.java:310)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
> at org.apache.hadoop.mapred.Child.main(Child.java:159)
> {quote}
> This means that read itself didn't fail but the resulted file was somehow smaller.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1085) hftp read failing silently
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12854263#action_12854263 ]
Tsz Wo (Nicholas), SZE commented on HDFS-1085:
----------------------------------------------
CancelledKeyException is a RuntimeException. The current codes may not handle it properly.
> hftp read failing silently
> ---------------------------
>
> Key: HDFS-1085
> URL: https://issues.apache.org/jira/browse/HDFS-1085
> Project: Hadoop HDFS
> Issue Type: Bug
> Reporter: Koji Noguchi
>
> When performing a massive distcp through hftp, we saw many tasks fail with
> {quote}
> 2010-04-06 17:56:43,005 INFO org.apache.hadoop.tools.DistCp: FAIL 2010/0/part-00032 : java.io.IOException: File size not matched: copied 193855488 bytes (184.9m) to tmpfile (=hdfs://omehost.com:8020/somepath/part-00032)
> but expected 1710327403 bytes (1.6g) from hftp://someotherhost/somepath/part-00032
> at org.apache.hadoop.tools.DistCp$CopyFilesMapper.copy(DistCp.java:435)
> at org.apache.hadoop.tools.DistCp$CopyFilesMapper.map(DistCp.java:543)
> at org.apache.hadoop.tools.DistCp$CopyFilesMapper.map(DistCp.java:310)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
> at org.apache.hadoop.mapred.Child.main(Child.java:159)
> {quote}
> This means that read itself didn't fail but the resulted file was somehow smaller.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1085) hftp read failing silently
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HDFS-1085:
-----------------------------------------
Attachment: h1085_20100713.patch
h1085_20100713.patch: when there is an exception, send error if the response is not committed, otherwise, leave output stream open and let the servlet container handle the exception.
Will try adding new tests.
> hftp read failing silently
> ---------------------------
>
> Key: HDFS-1085
> URL: https://issues.apache.org/jira/browse/HDFS-1085
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Reporter: Koji Noguchi
> Assignee: Tsz Wo (Nicholas), SZE
> Attachments: h1085_20100713.patch
>
>
> When performing a massive distcp through hftp, we saw many tasks fail with
> {quote}
> 2010-04-06 17:56:43,005 INFO org.apache.hadoop.tools.DistCp: FAIL 2010/0/part-00032 : java.io.IOException: File size not matched: copied 193855488 bytes (184.9m) to tmpfile (=hdfs://omehost.com:8020/somepath/part-00032)
> but expected 1710327403 bytes (1.6g) from hftp://someotherhost/somepath/part-00032
> at org.apache.hadoop.tools.DistCp$CopyFilesMapper.copy(DistCp.java:435)
> at org.apache.hadoop.tools.DistCp$CopyFilesMapper.map(DistCp.java:543)
> at org.apache.hadoop.tools.DistCp$CopyFilesMapper.map(DistCp.java:310)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
> at org.apache.hadoop.mapred.Child.main(Child.java:159)
> {quote}
> This means that read itself didn't fail but the resulted file was somehow smaller.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1085) hftp read failing silently
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HDFS-1085:
-----------------------------------------
Attachment: h1085_20100716d_y0.20.1xx.patch
h1085_20100716d_y0.20.1xx.patch: check null when getting Content-Length.
> hftp read failing silently
> ---------------------------
>
> Key: HDFS-1085
> URL: https://issues.apache.org/jira/browse/HDFS-1085
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Reporter: Koji Noguchi
> Assignee: Tsz Wo (Nicholas), SZE
> Fix For: 0.22.0
>
> Attachments: h1085_20100713.patch, h1085_20100716b_y0.20.1xx.patch, h1085_20100716b_y0.20.1xx_with_test.patch, h1085_20100716c.patch, h1085_20100716c_y0.20.1xx.patch, h1085_20100716d_y0.20.1xx.patch
>
>
> When performing a massive distcp through hftp, we saw many tasks fail with
> {quote}
> 2010-04-06 17:56:43,005 INFO org.apache.hadoop.tools.DistCp: FAIL 2010/0/part-00032 : java.io.IOException: File size not matched: copied 193855488 bytes (184.9m) to tmpfile (=hdfs://omehost.com:8020/somepath/part-00032)
> but expected 1710327403 bytes (1.6g) from hftp://someotherhost/somepath/part-00032
> at org.apache.hadoop.tools.DistCp$CopyFilesMapper.copy(DistCp.java:435)
> at org.apache.hadoop.tools.DistCp$CopyFilesMapper.map(DistCp.java:543)
> at org.apache.hadoop.tools.DistCp$CopyFilesMapper.map(DistCp.java:310)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
> at org.apache.hadoop.mapred.Child.main(Child.java:159)
> {quote}
> This means that read itself didn't fail but the resulted file was somehow smaller.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1085) hftp read failing silently
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HDFS-1085:
-----------------------------------------
Attachment: h1085_20100716c_y0.20.1xx.patch
Thanks Suresh for pointing out that we don't really need catch(..) in StreamFile.
h1085_20100716c_y0.20.1xx.patch:
- removed catch(..)
- slightly changed the code to make it more similar to trunk.
> hftp read failing silently
> ---------------------------
>
> Key: HDFS-1085
> URL: https://issues.apache.org/jira/browse/HDFS-1085
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Reporter: Koji Noguchi
> Assignee: Tsz Wo (Nicholas), SZE
> Attachments: h1085_20100713.patch, h1085_20100716b_y0.20.1xx.patch, h1085_20100716b_y0.20.1xx_with_test.patch, h1085_20100716c_y0.20.1xx.patch
>
>
> When performing a massive distcp through hftp, we saw many tasks fail with
> {quote}
> 2010-04-06 17:56:43,005 INFO org.apache.hadoop.tools.DistCp: FAIL 2010/0/part-00032 : java.io.IOException: File size not matched: copied 193855488 bytes (184.9m) to tmpfile (=hdfs://omehost.com:8020/somepath/part-00032)
> but expected 1710327403 bytes (1.6g) from hftp://someotherhost/somepath/part-00032
> at org.apache.hadoop.tools.DistCp$CopyFilesMapper.copy(DistCp.java:435)
> at org.apache.hadoop.tools.DistCp$CopyFilesMapper.map(DistCp.java:543)
> at org.apache.hadoop.tools.DistCp$CopyFilesMapper.map(DistCp.java:310)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
> at org.apache.hadoop.mapred.Child.main(Child.java:159)
> {quote}
> This means that read itself didn't fail but the resulted file was somehow smaller.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1085) hftp read failing silently
Posted by "Koji Noguchi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12854237#action_12854237 ]
Koji Noguchi commented on HDFS-1085:
------------------------------------
On the source datanode side, there was a log with
{quote}
2010-04-06 17:56:42,994 ERROR org.mortbay.log: /streamFile
java.nio.channels.CancelledKeyException
at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:55)
at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:59)
at org.mortbay.io.nio.SelectChannelEndPoint.updateKey(SelectChannelEndPoint.java:324)
at org.mortbay.io.nio.SelectChannelEndPoint.blockWritable(SelectChannelEndPoint.java:278)
at org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:542)
at org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:569)
at org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:946)
at org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:646)
at org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:577)
at org.apache.hadoop.hdfs.server.namenode.StreamFile.doGet(StreamFile.java:68)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1124)
at org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:663)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1115)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:361)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:324)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)
2010-04-06 17:56:43,007 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /99.99.99.99:50010, dest: /88.88.88.88:49774, bytes: 61292544, op: HDFS_READ, cliID: DFSClient_-563231950, offset: 0, srvID: DS-1411469763-98.136.245.125-50010-1268765357890, blockid: blk_-8819444732305373616_6553266, duration: 95558730000
{quote}
It was a distcp, so it should have read the whole 1 block(128MB) instead of bytes: 61292544.
{quote}
1. blk_-8819444732305373616_6553266 len=134217728
{quote}
Nicholas pointed out to me that this is very similar to error observed in HDFS-1024.
For distcp, thanks to the extra size checking from HADOOP-3294 we are ok, but does this mean I cannot trust the result of hadoop '-get' through hftp ?
{quote}
% hadoop dfs -get hftp://somehost/somepath/file1 .
{quote}
I have a feeling that this command can *silently* fail with incomplete output.
> hftp read failing silently
> ---------------------------
>
> Key: HDFS-1085
> URL: https://issues.apache.org/jira/browse/HDFS-1085
> Project: Hadoop HDFS
> Issue Type: Bug
> Reporter: Koji Noguchi
>
> When performing a massive distcp through hftp, we saw many tasks fail with
> {quote}
> 2010-04-06 17:56:43,005 INFO org.apache.hadoop.tools.DistCp: FAIL 2010/0/part-00032 : java.io.IOException: File size not matched: copied 193855488 bytes (184.9m) to tmpfile (=hdfs://omehost.com:8020/somepath/part-00032)
> but expected 1710327403 bytes (1.6g) from hftp://someotherhost/somepath/part-00032
> at org.apache.hadoop.tools.DistCp$CopyFilesMapper.copy(DistCp.java:435)
> at org.apache.hadoop.tools.DistCp$CopyFilesMapper.map(DistCp.java:543)
> at org.apache.hadoop.tools.DistCp$CopyFilesMapper.map(DistCp.java:310)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
> at org.apache.hadoop.mapred.Child.main(Child.java:159)
> {quote}
> This means that read itself didn't fail but the resulted file was somehow smaller.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1085) hftp read failing silently
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HDFS-1085:
-----------------------------------------
Attachment: h1085_20100716b_y0.20.1xx_with_test.patch
h1085_20100716b_y0.20.1xx_with_test.patch: same patch with new tests but it modified FileDataServlet in order to replace real hostname to localhost.
I ran the tests locally, the patch worked fine.
Will convert the tests with aop in HDFS-1304
> hftp read failing silently
> ---------------------------
>
> Key: HDFS-1085
> URL: https://issues.apache.org/jira/browse/HDFS-1085
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Reporter: Koji Noguchi
> Assignee: Tsz Wo (Nicholas), SZE
> Attachments: h1085_20100713.patch, h1085_20100716b_y0.20.1xx.patch, h1085_20100716b_y0.20.1xx_with_test.patch
>
>
> When performing a massive distcp through hftp, we saw many tasks fail with
> {quote}
> 2010-04-06 17:56:43,005 INFO org.apache.hadoop.tools.DistCp: FAIL 2010/0/part-00032 : java.io.IOException: File size not matched: copied 193855488 bytes (184.9m) to tmpfile (=hdfs://omehost.com:8020/somepath/part-00032)
> but expected 1710327403 bytes (1.6g) from hftp://someotherhost/somepath/part-00032
> at org.apache.hadoop.tools.DistCp$CopyFilesMapper.copy(DistCp.java:435)
> at org.apache.hadoop.tools.DistCp$CopyFilesMapper.map(DistCp.java:543)
> at org.apache.hadoop.tools.DistCp$CopyFilesMapper.map(DistCp.java:310)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
> at org.apache.hadoop.mapred.Child.main(Child.java:159)
> {quote}
> This means that read itself didn't fail but the resulted file was somehow smaller.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1085) hftp read failing silently
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HDFS-1085:
-----------------------------------------
Status: Patch Available (was: Open)
> hftp read failing silently
> ---------------------------
>
> Key: HDFS-1085
> URL: https://issues.apache.org/jira/browse/HDFS-1085
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Reporter: Koji Noguchi
> Assignee: Tsz Wo (Nicholas), SZE
> Fix For: 0.22.0
>
> Attachments: h1085_20100713.patch, h1085_20100716b_y0.20.1xx.patch, h1085_20100716b_y0.20.1xx_with_test.patch, h1085_20100716c.patch, h1085_20100716c_y0.20.1xx.patch, h1085_20100716d.patch, h1085_20100716d_y0.20.1xx.patch, h1085_20100716d_y0.20.1xx_test.patch
>
>
> When performing a massive distcp through hftp, we saw many tasks fail with
> {quote}
> 2010-04-06 17:56:43,005 INFO org.apache.hadoop.tools.DistCp: FAIL 2010/0/part-00032 : java.io.IOException: File size not matched: copied 193855488 bytes (184.9m) to tmpfile (=hdfs://omehost.com:8020/somepath/part-00032)
> but expected 1710327403 bytes (1.6g) from hftp://someotherhost/somepath/part-00032
> at org.apache.hadoop.tools.DistCp$CopyFilesMapper.copy(DistCp.java:435)
> at org.apache.hadoop.tools.DistCp$CopyFilesMapper.map(DistCp.java:543)
> at org.apache.hadoop.tools.DistCp$CopyFilesMapper.map(DistCp.java:310)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
> at org.apache.hadoop.mapred.Child.main(Child.java:159)
> {quote}
> This means that read itself didn't fail but the resulted file was somehow smaller.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HDFS-1085) hftp read failing silently
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE reassigned HDFS-1085:
--------------------------------------------
Assignee: Tsz Wo (Nicholas), SZE
> hftp read failing silently
> ---------------------------
>
> Key: HDFS-1085
> URL: https://issues.apache.org/jira/browse/HDFS-1085
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
> Reporter: Koji Noguchi
> Assignee: Tsz Wo (Nicholas), SZE
>
> When performing a massive distcp through hftp, we saw many tasks fail with
> {quote}
> 2010-04-06 17:56:43,005 INFO org.apache.hadoop.tools.DistCp: FAIL 2010/0/part-00032 : java.io.IOException: File size not matched: copied 193855488 bytes (184.9m) to tmpfile (=hdfs://omehost.com:8020/somepath/part-00032)
> but expected 1710327403 bytes (1.6g) from hftp://someotherhost/somepath/part-00032
> at org.apache.hadoop.tools.DistCp$CopyFilesMapper.copy(DistCp.java:435)
> at org.apache.hadoop.tools.DistCp$CopyFilesMapper.map(DistCp.java:543)
> at org.apache.hadoop.tools.DistCp$CopyFilesMapper.map(DistCp.java:310)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
> at org.apache.hadoop.mapred.Child.main(Child.java:159)
> {quote}
> This means that read itself didn't fail but the resulted file was somehow smaller.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1085) hftp read failing silently
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HDFS-1085:
-----------------------------------------
Status: Open (was: Patch Available)
> hftp read failing silently
> ---------------------------
>
> Key: HDFS-1085
> URL: https://issues.apache.org/jira/browse/HDFS-1085
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Reporter: Koji Noguchi
> Assignee: Tsz Wo (Nicholas), SZE
> Fix For: 0.22.0
>
> Attachments: h1085_20100713.patch, h1085_20100716b_y0.20.1xx.patch, h1085_20100716b_y0.20.1xx_with_test.patch, h1085_20100716c.patch, h1085_20100716c_y0.20.1xx.patch, h1085_20100716d.patch, h1085_20100716d_y0.20.1xx.patch, h1085_20100716d_y0.20.1xx_test.patch
>
>
> When performing a massive distcp through hftp, we saw many tasks fail with
> {quote}
> 2010-04-06 17:56:43,005 INFO org.apache.hadoop.tools.DistCp: FAIL 2010/0/part-00032 : java.io.IOException: File size not matched: copied 193855488 bytes (184.9m) to tmpfile (=hdfs://omehost.com:8020/somepath/part-00032)
> but expected 1710327403 bytes (1.6g) from hftp://someotherhost/somepath/part-00032
> at org.apache.hadoop.tools.DistCp$CopyFilesMapper.copy(DistCp.java:435)
> at org.apache.hadoop.tools.DistCp$CopyFilesMapper.map(DistCp.java:543)
> at org.apache.hadoop.tools.DistCp$CopyFilesMapper.map(DistCp.java:310)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
> at org.apache.hadoop.mapred.Child.main(Child.java:159)
> {quote}
> This means that read itself didn't fail but the resulted file was somehow smaller.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1085) hftp read failing silently
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HDFS-1085:
-----------------------------------------
Component/s: name-node
> hftp read failing silently
> ---------------------------
>
> Key: HDFS-1085
> URL: https://issues.apache.org/jira/browse/HDFS-1085
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
> Reporter: Koji Noguchi
>
> When performing a massive distcp through hftp, we saw many tasks fail with
> {quote}
> 2010-04-06 17:56:43,005 INFO org.apache.hadoop.tools.DistCp: FAIL 2010/0/part-00032 : java.io.IOException: File size not matched: copied 193855488 bytes (184.9m) to tmpfile (=hdfs://omehost.com:8020/somepath/part-00032)
> but expected 1710327403 bytes (1.6g) from hftp://someotherhost/somepath/part-00032
> at org.apache.hadoop.tools.DistCp$CopyFilesMapper.copy(DistCp.java:435)
> at org.apache.hadoop.tools.DistCp$CopyFilesMapper.map(DistCp.java:543)
> at org.apache.hadoop.tools.DistCp$CopyFilesMapper.map(DistCp.java:310)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
> at org.apache.hadoop.mapred.Child.main(Child.java:159)
> {quote}
> This means that read itself didn't fail but the resulted file was somehow smaller.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1085) hftp read failing silently
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12855012#action_12855012 ]
Tsz Wo (Nicholas), SZE commented on HDFS-1085:
----------------------------------------------
{code}
//org.apache.hadoop.hdfs.server.namenode.StreamFile.doGet(..)
try {
int bytesRead;
while ((bytesRead = in.read(buf)) != -1) {
os.write(buf, 0, bytesRead);
}
} finally {
in.close();
os.close();
dfs.close();
}
{code}
In the codes above, if in.read(..) or os.write(..) throw an exception (checked or unchecked), the streams will be closed as usual. Then, client side won't be able to know about the exception but considers that it is an EOF.
> hftp read failing silently
> ---------------------------
>
> Key: HDFS-1085
> URL: https://issues.apache.org/jira/browse/HDFS-1085
> Project: Hadoop HDFS
> Issue Type: Bug
> Reporter: Koji Noguchi
>
> When performing a massive distcp through hftp, we saw many tasks fail with
> {quote}
> 2010-04-06 17:56:43,005 INFO org.apache.hadoop.tools.DistCp: FAIL 2010/0/part-00032 : java.io.IOException: File size not matched: copied 193855488 bytes (184.9m) to tmpfile (=hdfs://omehost.com:8020/somepath/part-00032)
> but expected 1710327403 bytes (1.6g) from hftp://someotherhost/somepath/part-00032
> at org.apache.hadoop.tools.DistCp$CopyFilesMapper.copy(DistCp.java:435)
> at org.apache.hadoop.tools.DistCp$CopyFilesMapper.map(DistCp.java:543)
> at org.apache.hadoop.tools.DistCp$CopyFilesMapper.map(DistCp.java:310)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
> at org.apache.hadoop.mapred.Child.main(Child.java:159)
> {quote}
> This means that read itself didn't fail but the resulted file was somehow smaller.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1085) hftp read failing silently
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12889429#action_12889429 ]
Suresh Srinivas commented on HDFS-1085:
---------------------------------------
+1 the new y20 patch.
> hftp read failing silently
> ---------------------------
>
> Key: HDFS-1085
> URL: https://issues.apache.org/jira/browse/HDFS-1085
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Reporter: Koji Noguchi
> Assignee: Tsz Wo (Nicholas), SZE
> Fix For: 0.22.0
>
> Attachments: h1085_20100713.patch, h1085_20100716b_y0.20.1xx.patch, h1085_20100716b_y0.20.1xx_with_test.patch, h1085_20100716c.patch, h1085_20100716c_y0.20.1xx.patch, h1085_20100716d_y0.20.1xx.patch
>
>
> When performing a massive distcp through hftp, we saw many tasks fail with
> {quote}
> 2010-04-06 17:56:43,005 INFO org.apache.hadoop.tools.DistCp: FAIL 2010/0/part-00032 : java.io.IOException: File size not matched: copied 193855488 bytes (184.9m) to tmpfile (=hdfs://omehost.com:8020/somepath/part-00032)
> but expected 1710327403 bytes (1.6g) from hftp://someotherhost/somepath/part-00032
> at org.apache.hadoop.tools.DistCp$CopyFilesMapper.copy(DistCp.java:435)
> at org.apache.hadoop.tools.DistCp$CopyFilesMapper.map(DistCp.java:543)
> at org.apache.hadoop.tools.DistCp$CopyFilesMapper.map(DistCp.java:310)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
> at org.apache.hadoop.mapred.Child.main(Child.java:159)
> {quote}
> This means that read itself didn't fail but the resulted file was somehow smaller.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1085) hftp read failing silently
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HDFS-1085:
-----------------------------------------
Status: Patch Available (was: Open)
Fix Version/s: 0.22.0
> hftp read failing silently
> ---------------------------
>
> Key: HDFS-1085
> URL: https://issues.apache.org/jira/browse/HDFS-1085
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Reporter: Koji Noguchi
> Assignee: Tsz Wo (Nicholas), SZE
> Fix For: 0.22.0
>
> Attachments: h1085_20100713.patch, h1085_20100716b_y0.20.1xx.patch, h1085_20100716b_y0.20.1xx_with_test.patch, h1085_20100716c.patch, h1085_20100716c_y0.20.1xx.patch
>
>
> When performing a massive distcp through hftp, we saw many tasks fail with
> {quote}
> 2010-04-06 17:56:43,005 INFO org.apache.hadoop.tools.DistCp: FAIL 2010/0/part-00032 : java.io.IOException: File size not matched: copied 193855488 bytes (184.9m) to tmpfile (=hdfs://omehost.com:8020/somepath/part-00032)
> but expected 1710327403 bytes (1.6g) from hftp://someotherhost/somepath/part-00032
> at org.apache.hadoop.tools.DistCp$CopyFilesMapper.copy(DistCp.java:435)
> at org.apache.hadoop.tools.DistCp$CopyFilesMapper.map(DistCp.java:543)
> at org.apache.hadoop.tools.DistCp$CopyFilesMapper.map(DistCp.java:310)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
> at org.apache.hadoop.mapred.Child.main(Child.java:159)
> {quote}
> This means that read itself didn't fail but the resulted file was somehow smaller.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1085) hftp read failing silently
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12889445#action_12889445 ]
Hadoop QA commented on HDFS-1085:
---------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12449705/h1085_20100716c.patch
against trunk revision 964947.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/436/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/436/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/436/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/436/console
This message is automatically generated.
> hftp read failing silently
> ---------------------------
>
> Key: HDFS-1085
> URL: https://issues.apache.org/jira/browse/HDFS-1085
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Reporter: Koji Noguchi
> Assignee: Tsz Wo (Nicholas), SZE
> Fix For: 0.22.0
>
> Attachments: h1085_20100713.patch, h1085_20100716b_y0.20.1xx.patch, h1085_20100716b_y0.20.1xx_with_test.patch, h1085_20100716c.patch, h1085_20100716c_y0.20.1xx.patch, h1085_20100716d.patch, h1085_20100716d_y0.20.1xx.patch, h1085_20100716d_y0.20.1xx_test.patch
>
>
> When performing a massive distcp through hftp, we saw many tasks fail with
> {quote}
> 2010-04-06 17:56:43,005 INFO org.apache.hadoop.tools.DistCp: FAIL 2010/0/part-00032 : java.io.IOException: File size not matched: copied 193855488 bytes (184.9m) to tmpfile (=hdfs://omehost.com:8020/somepath/part-00032)
> but expected 1710327403 bytes (1.6g) from hftp://someotherhost/somepath/part-00032
> at org.apache.hadoop.tools.DistCp$CopyFilesMapper.copy(DistCp.java:435)
> at org.apache.hadoop.tools.DistCp$CopyFilesMapper.map(DistCp.java:543)
> at org.apache.hadoop.tools.DistCp$CopyFilesMapper.map(DistCp.java:310)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
> at org.apache.hadoop.mapred.Child.main(Child.java:159)
> {quote}
> This means that read itself didn't fail but the resulted file was somehow smaller.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1085) hftp read failing silently
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HDFS-1085:
-----------------------------------------
Attachment: h1085_20100716c.patch
h1085_20100716c.patch: for trunk. Also fixed some existing javac warnings.
> hftp read failing silently
> ---------------------------
>
> Key: HDFS-1085
> URL: https://issues.apache.org/jira/browse/HDFS-1085
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Reporter: Koji Noguchi
> Assignee: Tsz Wo (Nicholas), SZE
> Fix For: 0.22.0
>
> Attachments: h1085_20100713.patch, h1085_20100716b_y0.20.1xx.patch, h1085_20100716b_y0.20.1xx_with_test.patch, h1085_20100716c.patch, h1085_20100716c_y0.20.1xx.patch
>
>
> When performing a massive distcp through hftp, we saw many tasks fail with
> {quote}
> 2010-04-06 17:56:43,005 INFO org.apache.hadoop.tools.DistCp: FAIL 2010/0/part-00032 : java.io.IOException: File size not matched: copied 193855488 bytes (184.9m) to tmpfile (=hdfs://omehost.com:8020/somepath/part-00032)
> but expected 1710327403 bytes (1.6g) from hftp://someotherhost/somepath/part-00032
> at org.apache.hadoop.tools.DistCp$CopyFilesMapper.copy(DistCp.java:435)
> at org.apache.hadoop.tools.DistCp$CopyFilesMapper.map(DistCp.java:543)
> at org.apache.hadoop.tools.DistCp$CopyFilesMapper.map(DistCp.java:310)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
> at org.apache.hadoop.mapred.Child.main(Child.java:159)
> {quote}
> This means that read itself didn't fail but the resulted file was somehow smaller.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1085) hftp read failing silently
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12888111#action_12888111 ]
Tsz Wo (Nicholas), SZE commented on HDFS-1085:
----------------------------------------------
@Dmytro, this issue is not trying to fix the cause of the exception but we want to make sure that the client is able to see the exception, instead of failing silently.
> hftp read failing silently
> ---------------------------
>
> Key: HDFS-1085
> URL: https://issues.apache.org/jira/browse/HDFS-1085
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Reporter: Koji Noguchi
> Assignee: Tsz Wo (Nicholas), SZE
> Attachments: h1085_20100713.patch
>
>
> When performing a massive distcp through hftp, we saw many tasks fail with
> {quote}
> 2010-04-06 17:56:43,005 INFO org.apache.hadoop.tools.DistCp: FAIL 2010/0/part-00032 : java.io.IOException: File size not matched: copied 193855488 bytes (184.9m) to tmpfile (=hdfs://omehost.com:8020/somepath/part-00032)
> but expected 1710327403 bytes (1.6g) from hftp://someotherhost/somepath/part-00032
> at org.apache.hadoop.tools.DistCp$CopyFilesMapper.copy(DistCp.java:435)
> at org.apache.hadoop.tools.DistCp$CopyFilesMapper.map(DistCp.java:543)
> at org.apache.hadoop.tools.DistCp$CopyFilesMapper.map(DistCp.java:310)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
> at org.apache.hadoop.mapred.Child.main(Child.java:159)
> {quote}
> This means that read itself didn't fail but the resulted file was somehow smaller.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1085) hftp read failing silently
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HDFS-1085:
-----------------------------------------
Status: Resolved (was: Patch Available)
Hadoop Flags: [Reviewed]
Resolution: Fixed
Thanks Suresh for the reviewing.
I have committed this.
> hftp read failing silently
> ---------------------------
>
> Key: HDFS-1085
> URL: https://issues.apache.org/jira/browse/HDFS-1085
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Reporter: Koji Noguchi
> Assignee: Tsz Wo (Nicholas), SZE
> Fix For: 0.22.0
>
> Attachments: h1085_20100713.patch, h1085_20100716b_y0.20.1xx.patch, h1085_20100716b_y0.20.1xx_with_test.patch, h1085_20100716c.patch, h1085_20100716c_y0.20.1xx.patch, h1085_20100716d.patch, h1085_20100716d_y0.20.1xx.patch, h1085_20100716d_y0.20.1xx_test.patch
>
>
> When performing a massive distcp through hftp, we saw many tasks fail with
> {quote}
> 2010-04-06 17:56:43,005 INFO org.apache.hadoop.tools.DistCp: FAIL 2010/0/part-00032 : java.io.IOException: File size not matched: copied 193855488 bytes (184.9m) to tmpfile (=hdfs://omehost.com:8020/somepath/part-00032)
> but expected 1710327403 bytes (1.6g) from hftp://someotherhost/somepath/part-00032
> at org.apache.hadoop.tools.DistCp$CopyFilesMapper.copy(DistCp.java:435)
> at org.apache.hadoop.tools.DistCp$CopyFilesMapper.map(DistCp.java:543)
> at org.apache.hadoop.tools.DistCp$CopyFilesMapper.map(DistCp.java:310)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
> at org.apache.hadoop.mapred.Child.main(Child.java:159)
> {quote}
> This means that read itself didn't fail but the resulted file was somehow smaller.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1085) hftp read failing silently
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HDFS-1085:
-----------------------------------------
Attachment: h1085_20100716d.patch
h1085_20100716d.patch: for trunk.
> hftp read failing silently
> ---------------------------
>
> Key: HDFS-1085
> URL: https://issues.apache.org/jira/browse/HDFS-1085
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Reporter: Koji Noguchi
> Assignee: Tsz Wo (Nicholas), SZE
> Fix For: 0.22.0
>
> Attachments: h1085_20100713.patch, h1085_20100716b_y0.20.1xx.patch, h1085_20100716b_y0.20.1xx_with_test.patch, h1085_20100716c.patch, h1085_20100716c_y0.20.1xx.patch, h1085_20100716d.patch, h1085_20100716d_y0.20.1xx.patch, h1085_20100716d_y0.20.1xx_test.patch
>
>
> When performing a massive distcp through hftp, we saw many tasks fail with
> {quote}
> 2010-04-06 17:56:43,005 INFO org.apache.hadoop.tools.DistCp: FAIL 2010/0/part-00032 : java.io.IOException: File size not matched: copied 193855488 bytes (184.9m) to tmpfile (=hdfs://omehost.com:8020/somepath/part-00032)
> but expected 1710327403 bytes (1.6g) from hftp://someotherhost/somepath/part-00032
> at org.apache.hadoop.tools.DistCp$CopyFilesMapper.copy(DistCp.java:435)
> at org.apache.hadoop.tools.DistCp$CopyFilesMapper.map(DistCp.java:543)
> at org.apache.hadoop.tools.DistCp$CopyFilesMapper.map(DistCp.java:310)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
> at org.apache.hadoop.mapred.Child.main(Child.java:159)
> {quote}
> This means that read itself didn't fail but the resulted file was somehow smaller.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1085) hftp read failing silently
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HDFS-1085:
-----------------------------------------
Attachment: h1085_20100716b_y0.20.1xx.patch
h1085_20100716b_y0.20.1xx.patch:
- send Content-Length in StreamFile
- check byte read in HftpFileSystem
- if response is not committed, send error. Otherwise, client will throw an exception if it gets an EOF but byte read < Content-Length
> hftp read failing silently
> ---------------------------
>
> Key: HDFS-1085
> URL: https://issues.apache.org/jira/browse/HDFS-1085
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Reporter: Koji Noguchi
> Assignee: Tsz Wo (Nicholas), SZE
> Attachments: h1085_20100713.patch, h1085_20100716b_y0.20.1xx.patch
>
>
> When performing a massive distcp through hftp, we saw many tasks fail with
> {quote}
> 2010-04-06 17:56:43,005 INFO org.apache.hadoop.tools.DistCp: FAIL 2010/0/part-00032 : java.io.IOException: File size not matched: copied 193855488 bytes (184.9m) to tmpfile (=hdfs://omehost.com:8020/somepath/part-00032)
> but expected 1710327403 bytes (1.6g) from hftp://someotherhost/somepath/part-00032
> at org.apache.hadoop.tools.DistCp$CopyFilesMapper.copy(DistCp.java:435)
> at org.apache.hadoop.tools.DistCp$CopyFilesMapper.map(DistCp.java:543)
> at org.apache.hadoop.tools.DistCp$CopyFilesMapper.map(DistCp.java:310)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
> at org.apache.hadoop.mapred.Child.main(Child.java:159)
> {quote}
> This means that read itself didn't fail but the resulted file was somehow smaller.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1085) hftp read failing silently
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12889458#action_12889458 ]
Hadoop QA commented on HDFS-1085:
---------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12449743/h1085_20100716d.patch
against trunk revision 964947.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/437/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/437/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/437/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/437/console
This message is automatically generated.
> hftp read failing silently
> ---------------------------
>
> Key: HDFS-1085
> URL: https://issues.apache.org/jira/browse/HDFS-1085
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Reporter: Koji Noguchi
> Assignee: Tsz Wo (Nicholas), SZE
> Fix For: 0.22.0
>
> Attachments: h1085_20100713.patch, h1085_20100716b_y0.20.1xx.patch, h1085_20100716b_y0.20.1xx_with_test.patch, h1085_20100716c.patch, h1085_20100716c_y0.20.1xx.patch, h1085_20100716d.patch, h1085_20100716d_y0.20.1xx.patch, h1085_20100716d_y0.20.1xx_test.patch
>
>
> When performing a massive distcp through hftp, we saw many tasks fail with
> {quote}
> 2010-04-06 17:56:43,005 INFO org.apache.hadoop.tools.DistCp: FAIL 2010/0/part-00032 : java.io.IOException: File size not matched: copied 193855488 bytes (184.9m) to tmpfile (=hdfs://omehost.com:8020/somepath/part-00032)
> but expected 1710327403 bytes (1.6g) from hftp://someotherhost/somepath/part-00032
> at org.apache.hadoop.tools.DistCp$CopyFilesMapper.copy(DistCp.java:435)
> at org.apache.hadoop.tools.DistCp$CopyFilesMapper.map(DistCp.java:543)
> at org.apache.hadoop.tools.DistCp$CopyFilesMapper.map(DistCp.java:310)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
> at org.apache.hadoop.mapred.Child.main(Child.java:159)
> {quote}
> This means that read itself didn't fail but the resulted file was somehow smaller.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1085) hftp read failing silently
Posted by "Dmytro Molkov (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12888070#action_12888070 ]
Dmytro Molkov commented on HDFS-1085:
-------------------------------------
I think it has everything to do with the JVM bug and not the best way of jetty handling it. More details in https://issues.apache.org/jira/browse/HDFS-1194
We updated the jetty version in our production cluster and we do not see secondary namenode issues anymore (we are always able to fetch the full file). It should also help this case.
> hftp read failing silently
> ---------------------------
>
> Key: HDFS-1085
> URL: https://issues.apache.org/jira/browse/HDFS-1085
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Reporter: Koji Noguchi
> Assignee: Tsz Wo (Nicholas), SZE
> Attachments: h1085_20100713.patch
>
>
> When performing a massive distcp through hftp, we saw many tasks fail with
> {quote}
> 2010-04-06 17:56:43,005 INFO org.apache.hadoop.tools.DistCp: FAIL 2010/0/part-00032 : java.io.IOException: File size not matched: copied 193855488 bytes (184.9m) to tmpfile (=hdfs://omehost.com:8020/somepath/part-00032)
> but expected 1710327403 bytes (1.6g) from hftp://someotherhost/somepath/part-00032
> at org.apache.hadoop.tools.DistCp$CopyFilesMapper.copy(DistCp.java:435)
> at org.apache.hadoop.tools.DistCp$CopyFilesMapper.map(DistCp.java:543)
> at org.apache.hadoop.tools.DistCp$CopyFilesMapper.map(DistCp.java:310)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
> at org.apache.hadoop.mapred.Child.main(Child.java:159)
> {quote}
> This means that read itself didn't fail but the resulted file was somehow smaller.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1085) hftp read failing silently
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HDFS-1085:
-----------------------------------------
Component/s: data-node
(was: name-node)
StreamFile is a datanode servlet but it is misplaced in the namenode package. I will file an issue.
> hftp read failing silently
> ---------------------------
>
> Key: HDFS-1085
> URL: https://issues.apache.org/jira/browse/HDFS-1085
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Reporter: Koji Noguchi
> Assignee: Tsz Wo (Nicholas), SZE
>
> When performing a massive distcp through hftp, we saw many tasks fail with
> {quote}
> 2010-04-06 17:56:43,005 INFO org.apache.hadoop.tools.DistCp: FAIL 2010/0/part-00032 : java.io.IOException: File size not matched: copied 193855488 bytes (184.9m) to tmpfile (=hdfs://omehost.com:8020/somepath/part-00032)
> but expected 1710327403 bytes (1.6g) from hftp://someotherhost/somepath/part-00032
> at org.apache.hadoop.tools.DistCp$CopyFilesMapper.copy(DistCp.java:435)
> at org.apache.hadoop.tools.DistCp$CopyFilesMapper.map(DistCp.java:543)
> at org.apache.hadoop.tools.DistCp$CopyFilesMapper.map(DistCp.java:310)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
> at org.apache.hadoop.mapred.Child.main(Child.java:159)
> {quote}
> This means that read itself didn't fail but the resulted file was somehow smaller.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1085) hftp read failing silently
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12889443#action_12889443 ]
Tsz Wo (Nicholas), SZE commented on HDFS-1085:
----------------------------------------------
test-patch on h1085_20100716d.patch
{noformat}
[exec] +1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] +1 tests included. The patch appears to include 3 new or modified tests.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
[exec]
[exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
{noformat}
I will wait sometime to see if Hudson would response.
> hftp read failing silently
> ---------------------------
>
> Key: HDFS-1085
> URL: https://issues.apache.org/jira/browse/HDFS-1085
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Reporter: Koji Noguchi
> Assignee: Tsz Wo (Nicholas), SZE
> Fix For: 0.22.0
>
> Attachments: h1085_20100713.patch, h1085_20100716b_y0.20.1xx.patch, h1085_20100716b_y0.20.1xx_with_test.patch, h1085_20100716c.patch, h1085_20100716c_y0.20.1xx.patch, h1085_20100716d.patch, h1085_20100716d_y0.20.1xx.patch, h1085_20100716d_y0.20.1xx_test.patch
>
>
> When performing a massive distcp through hftp, we saw many tasks fail with
> {quote}
> 2010-04-06 17:56:43,005 INFO org.apache.hadoop.tools.DistCp: FAIL 2010/0/part-00032 : java.io.IOException: File size not matched: copied 193855488 bytes (184.9m) to tmpfile (=hdfs://omehost.com:8020/somepath/part-00032)
> but expected 1710327403 bytes (1.6g) from hftp://someotherhost/somepath/part-00032
> at org.apache.hadoop.tools.DistCp$CopyFilesMapper.copy(DistCp.java:435)
> at org.apache.hadoop.tools.DistCp$CopyFilesMapper.map(DistCp.java:543)
> at org.apache.hadoop.tools.DistCp$CopyFilesMapper.map(DistCp.java:310)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
> at org.apache.hadoop.mapred.Child.main(Child.java:159)
> {quote}
> This means that read itself didn't fail but the resulted file was somehow smaller.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.