You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Eli Collins (JIRA)" <ji...@apache.org> on 2012/07/16 19:54:34 UTC
[jira] [Created] (HDFS-3671) ByteRangeInputStream shouldn't require
the content length header be present
Eli Collins created HDFS-3671:
---------------------------------
Summary: ByteRangeInputStream shouldn't require the content length header be present
Key: HDFS-3671
URL: https://issues.apache.org/jira/browse/HDFS-3671
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.0.0-alpha
Reporter: Eli Collins
Priority: Blocker
Per HDFS-3318 the content length header check breaks distcp compatibility with previous releases (0.20.2 and earlier, and 0.21). Like branch-1 this check should be lenient.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3671) ByteRangeInputStream shouldn't
require the content length header be present
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13415499#comment-13415499 ]
Eli Collins commented on HDFS-3671:
-----------------------------------
Is lowering the client's idle timeout for reads a workaround?
> ByteRangeInputStream shouldn't require the content length header be present
> ---------------------------------------------------------------------------
>
> Key: HDFS-3671
> URL: https://issues.apache.org/jira/browse/HDFS-3671
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 2.0.0-alpha
> Reporter: Eli Collins
> Priority: Blocker
>
> Per HDFS-3318 the content length header check breaks distcp compatibility with previous releases (0.20.2 and earlier, and 0.21). Like branch-1 this check should be lenient.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3671) ByteRangeInputStream shouldn't require
the content length header be present
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HDFS-3671:
-----------------------------------------
Attachment: h3667_20120718.patch
h3667_20120718.patch: updated the test with trunk.
Tested read/write a 3GB file using WebHDFS with trunk and branch-1. It worked fine and the content-length check is unnecessary.
> ByteRangeInputStream shouldn't require the content length header be present
> ---------------------------------------------------------------------------
>
> Key: HDFS-3671
> URL: https://issues.apache.org/jira/browse/HDFS-3671
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 2.0.0-alpha
> Reporter: Eli Collins
> Priority: Blocker
> Attachments: h3667_20120718.patch, h3671_20120717.patch
>
>
> Per HDFS-3318 the content length header check breaks distcp compatibility with previous releases (0.20.2 and earlier, and 0.21). Like branch-1 this check should be lenient.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3671) ByteRangeInputStream shouldn't require
the content length header be present
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HDFS-3671:
-----------------------------------------
Attachment: (was: h3667_20120718.patch)
> ByteRangeInputStream shouldn't require the content length header be present
> ---------------------------------------------------------------------------
>
> Key: HDFS-3671
> URL: https://issues.apache.org/jira/browse/HDFS-3671
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 2.0.0-alpha
> Reporter: Eli Collins
> Priority: Blocker
> Attachments: h3671_20120717.patch, h3671_20120719.patch
>
>
> Per HDFS-3318 the content length header check breaks distcp compatibility with previous releases (0.20.2 and earlier, and 0.21). Like branch-1 this check should be lenient.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3671) ByteRangeInputStream shouldn't require
the content length header be present
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HDFS-3671:
-----------------------------------------
Attachment: h3671_20120812.patch
h3671_20120812.patch: updated with trunk.
> ByteRangeInputStream shouldn't require the content length header be present
> ---------------------------------------------------------------------------
>
> Key: HDFS-3671
> URL: https://issues.apache.org/jira/browse/HDFS-3671
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 2.0.0-alpha
> Reporter: Eli Collins
> Priority: Blocker
> Attachments: h3671_20120717.patch, h3671_20120719.patch, h3671_20120812.patch
>
>
> Per HDFS-3318 the content length header check breaks distcp compatibility with previous releases (0.20.2 and earlier, and 0.21). Like branch-1 this check should be lenient.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3671) ByteRangeInputStream shouldn't
require the content length header be present
Posted by "Daryn Sharp (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13419171#comment-13419171 ]
Daryn Sharp commented on HDFS-3671:
-----------------------------------
bq. Tested read/write a 3GB file using WebHDFS with trunk and branch-1. It worked fine and the content-length check is unnecessary.
I think trunk is chunking, and branch-1 sets the content-length. I think Eli's concern is about some pre-1.x releases which did neither.
> ByteRangeInputStream shouldn't require the content length header be present
> ---------------------------------------------------------------------------
>
> Key: HDFS-3671
> URL: https://issues.apache.org/jira/browse/HDFS-3671
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 2.0.0-alpha
> Reporter: Eli Collins
> Priority: Blocker
> Attachments: h3671_20120717.patch, h3671_20120719.patch
>
>
> Per HDFS-3318 the content length header check breaks distcp compatibility with previous releases (0.20.2 and earlier, and 0.21). Like branch-1 this check should be lenient.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3671) ByteRangeInputStream shouldn't
require the content length header be present
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13418961#comment-13418961 ]
Tsz Wo (Nicholas), SZE commented on HDFS-3671:
----------------------------------------------
> The dilemma is that if the content-length is not defined (and chunking is not used), ...
In our servers, is the case that either the content-length is defined or chunked transfer encoding is used?
> ByteRangeInputStream shouldn't require the content length header be present
> ---------------------------------------------------------------------------
>
> Key: HDFS-3671
> URL: https://issues.apache.org/jira/browse/HDFS-3671
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 2.0.0-alpha
> Reporter: Eli Collins
> Priority: Blocker
> Attachments: h3671_20120717.patch
>
>
> Per HDFS-3318 the content length header check breaks distcp compatibility with previous releases (0.20.2 and earlier, and 0.21). Like branch-1 this check should be lenient.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3671) ByteRangeInputStream shouldn't require
the content length header be present
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HDFS-3671:
-----------------------------------------
Attachment: (was: h3671_20120717.patch)
> ByteRangeInputStream shouldn't require the content length header be present
> ---------------------------------------------------------------------------
>
> Key: HDFS-3671
> URL: https://issues.apache.org/jira/browse/HDFS-3671
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 2.0.0-alpha
> Reporter: Eli Collins
> Priority: Blocker
>
> Per HDFS-3318 the content length header check breaks distcp compatibility with previous releases (0.20.2 and earlier, and 0.21). Like branch-1 this check should be lenient.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3671) ByteRangeInputStream shouldn't require
the content length header be present
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HDFS-3671:
-----------------------------------------
Status: Patch Available (was: Open)
Try running the test with Jenkins. The test probably won't be committed since it takes about 10 minutes.
> ByteRangeInputStream shouldn't require the content length header be present
> ---------------------------------------------------------------------------
>
> Key: HDFS-3671
> URL: https://issues.apache.org/jira/browse/HDFS-3671
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 2.0.0-alpha
> Reporter: Eli Collins
> Priority: Blocker
> Attachments: h3671_20120717.patch, h3671_20120719.patch
>
>
> Per HDFS-3318 the content length header check breaks distcp compatibility with previous releases (0.20.2 and earlier, and 0.21). Like branch-1 this check should be lenient.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3671) ByteRangeInputStream shouldn't
require the content length header be present
Posted by "Daryn Sharp (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13415527#comment-13415527 ]
Daryn Sharp commented on HDFS-3671:
-----------------------------------
Jetty has I think a 200s timeout, so increasing the client read timeout above that should be a quasi-reasonable workaround. Unfortunately the client read timeout isn't configurable, although it should be simple to make it configurable.
You may want to consider investigating how hard it would be to use the size from a file stat in lieu of a missing content-length.
> ByteRangeInputStream shouldn't require the content length header be present
> ---------------------------------------------------------------------------
>
> Key: HDFS-3671
> URL: https://issues.apache.org/jira/browse/HDFS-3671
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 2.0.0-alpha
> Reporter: Eli Collins
> Priority: Blocker
>
> Per HDFS-3318 the content length header check breaks distcp compatibility with previous releases (0.20.2 and earlier, and 0.21). Like branch-1 this check should be lenient.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3671) ByteRangeInputStream shouldn't
require the content length header be present
Posted by "Daryn Sharp (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13417080#comment-13417080 ]
Daryn Sharp commented on HDFS-3671:
-----------------------------------
Are you sure you tested a non-chunked download w/o a content-length in the response? If it worked, there should have been a 200s hang at the end of the transfer, and would mean the http read timeout has been broken...
> ByteRangeInputStream shouldn't require the content length header be present
> ---------------------------------------------------------------------------
>
> Key: HDFS-3671
> URL: https://issues.apache.org/jira/browse/HDFS-3671
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 2.0.0-alpha
> Reporter: Eli Collins
> Priority: Blocker
> Attachments: h3671_20120717.patch
>
>
> Per HDFS-3318 the content length header check breaks distcp compatibility with previous releases (0.20.2 and earlier, and 0.21). Like branch-1 this check should be lenient.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3671) ByteRangeInputStream shouldn't require
the content length header be present
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HDFS-3671:
-----------------------------------------
Attachment: h3671_20120717.patch
> ByteRangeInputStream shouldn't require the content length header be present
> ---------------------------------------------------------------------------
>
> Key: HDFS-3671
> URL: https://issues.apache.org/jira/browse/HDFS-3671
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 2.0.0-alpha
> Reporter: Eli Collins
> Priority: Blocker
> Attachments: h3671_20120717.patch
>
>
> Per HDFS-3318 the content length header check breaks distcp compatibility with previous releases (0.20.2 and earlier, and 0.21). Like branch-1 this check should be lenient.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3671) ByteRangeInputStream shouldn't
require the content length header be present
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13438026#comment-13438026 ]
Eli Collins commented on HDFS-3671:
-----------------------------------
Yes, you can't distcp from 20.x to v2 because in ByteRangeInputStream we still fail if the content length header is not present but 20.x releases don't set it, eg distcp using hftp from 0.20.2 to 2.0.0-alpha1 fails.
> ByteRangeInputStream shouldn't require the content length header be present
> ---------------------------------------------------------------------------
>
> Key: HDFS-3671
> URL: https://issues.apache.org/jira/browse/HDFS-3671
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 2.0.0-alpha
> Reporter: Eli Collins
> Priority: Blocker
> Attachments: h3671_20120717.patch, h3671_20120719.patch, h3671_20120812.patch
>
>
> Per HDFS-3318 the content length header check breaks distcp compatibility with previous releases (0.20.2 and earlier, and 0.21). Like branch-1 this check should be lenient.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3671) ByteRangeInputStream shouldn't
require the content length header be present
Posted by "Daryn Sharp (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13438034#comment-13438034 ]
Daryn Sharp commented on HDFS-3671:
-----------------------------------
Eli's right, there is one remaining change to for compatibility with earlier hadoop releases. The content-length will have to be obtained via a HEAD or file stat when no content-length is returned and the response isn't chunked.
> ByteRangeInputStream shouldn't require the content length header be present
> ---------------------------------------------------------------------------
>
> Key: HDFS-3671
> URL: https://issues.apache.org/jira/browse/HDFS-3671
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 2.0.0-alpha
> Reporter: Eli Collins
> Priority: Blocker
> Attachments: h3671_20120717.patch, h3671_20120719.patch, h3671_20120812.patch
>
>
> Per HDFS-3318 the content length header check breaks distcp compatibility with previous releases (0.20.2 and earlier, and 0.21). Like branch-1 this check should be lenient.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3671) ByteRangeInputStream shouldn't require
the content length header be present
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HDFS-3671:
-----------------------------------------
Attachment: h3671_20120717.patch
I believe we can safely remove Content-Length check in ByteRangeInputStream. Tested it with a 3GB file. It worked fine.
h3671_20120717.patch
> ByteRangeInputStream shouldn't require the content length header be present
> ---------------------------------------------------------------------------
>
> Key: HDFS-3671
> URL: https://issues.apache.org/jira/browse/HDFS-3671
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 2.0.0-alpha
> Reporter: Eli Collins
> Priority: Blocker
>
> Per HDFS-3318 the content length header check breaks distcp compatibility with previous releases (0.20.2 and earlier, and 0.21). Like branch-1 this check should be lenient.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3671) ByteRangeInputStream shouldn't
require the content length header be present
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13418523#comment-13418523 ]
Eli Collins commented on HDFS-3671:
-----------------------------------
Since HDFS-3577 is not removing the Content-Length header check we need to fix this here, otherwise we still can't distcp from the 20.x (ie pre-203) releases, which we don't want to break since that prevents users from migrating to new releases.
Daryn, I like your idea of doing the file stat if the length is not present, want to post a patch for that?
> ByteRangeInputStream shouldn't require the content length header be present
> ---------------------------------------------------------------------------
>
> Key: HDFS-3671
> URL: https://issues.apache.org/jira/browse/HDFS-3671
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 2.0.0-alpha
> Reporter: Eli Collins
> Priority: Blocker
> Attachments: h3671_20120717.patch
>
>
> Per HDFS-3318 the content length header check breaks distcp compatibility with previous releases (0.20.2 and earlier, and 0.21). Like branch-1 this check should be lenient.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3671) ByteRangeInputStream shouldn't
require the content length header be present
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13418981#comment-13418981 ]
Eli Collins commented on HDFS-3671:
-----------------------------------
Checkout my latest comment on HDFS-3577, a 3gb read/write may work but distcp still fails.
> ByteRangeInputStream shouldn't require the content length header be present
> ---------------------------------------------------------------------------
>
> Key: HDFS-3671
> URL: https://issues.apache.org/jira/browse/HDFS-3671
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 2.0.0-alpha
> Reporter: Eli Collins
> Priority: Blocker
> Attachments: h3671_20120717.patch, h3671_20120719.patch
>
>
> Per HDFS-3318 the content length header check breaks distcp compatibility with previous releases (0.20.2 and earlier, and 0.21). Like branch-1 this check should be lenient.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3671) ByteRangeInputStream shouldn't
require the content length header be present
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13436541#comment-13436541 ]
Tsz Wo (Nicholas), SZE commented on HDFS-3671:
----------------------------------------------
After HDFS-3788, do we still need this?
> ByteRangeInputStream shouldn't require the content length header be present
> ---------------------------------------------------------------------------
>
> Key: HDFS-3671
> URL: https://issues.apache.org/jira/browse/HDFS-3671
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 2.0.0-alpha
> Reporter: Eli Collins
> Priority: Blocker
> Attachments: h3671_20120717.patch, h3671_20120719.patch, h3671_20120812.patch
>
>
> Per HDFS-3318 the content length header check breaks distcp compatibility with previous releases (0.20.2 and earlier, and 0.21). Like branch-1 this check should be lenient.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3671) ByteRangeInputStream shouldn't
require the content length header be present
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13419374#comment-13419374 ]
Eli Collins commented on HDFS-3671:
-----------------------------------
The test for this is whether you can distcp a 2gb+ file from 0.20 (or 0.21) to 2.x. Currently that fails so users don't have a way to migrate data from these releases (and there are a lot of people on 0.20).
> ByteRangeInputStream shouldn't require the content length header be present
> ---------------------------------------------------------------------------
>
> Key: HDFS-3671
> URL: https://issues.apache.org/jira/browse/HDFS-3671
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 2.0.0-alpha
> Reporter: Eli Collins
> Priority: Blocker
> Attachments: h3671_20120717.patch, h3671_20120719.patch
>
>
> Per HDFS-3318 the content length header check breaks distcp compatibility with previous releases (0.20.2 and earlier, and 0.21). Like branch-1 this check should be lenient.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3671) ByteRangeInputStream shouldn't
require the content length header be present
Posted by "Daryn Sharp (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13418618#comment-13418618 ]
Daryn Sharp commented on HDFS-3671:
-----------------------------------
I've got to get a few token patches up for 2.x, so I might not be able to take a stab at it until early next week.
> ByteRangeInputStream shouldn't require the content length header be present
> ---------------------------------------------------------------------------
>
> Key: HDFS-3671
> URL: https://issues.apache.org/jira/browse/HDFS-3671
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 2.0.0-alpha
> Reporter: Eli Collins
> Priority: Blocker
> Attachments: h3671_20120717.patch
>
>
> Per HDFS-3318 the content length header check breaks distcp compatibility with previous releases (0.20.2 and earlier, and 0.21). Like branch-1 this check should be lenient.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3671) ByteRangeInputStream shouldn't require
the content length header be present
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HDFS-3671:
-----------------------------------------
Attachment: h3671_20120719.patch
> h3667_20120718.patch: updated the test with trunk.
The file should be h3671_20120719.patch.
> ByteRangeInputStream shouldn't require the content length header be present
> ---------------------------------------------------------------------------
>
> Key: HDFS-3671
> URL: https://issues.apache.org/jira/browse/HDFS-3671
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 2.0.0-alpha
> Reporter: Eli Collins
> Priority: Blocker
> Attachments: h3671_20120717.patch, h3671_20120719.patch
>
>
> Per HDFS-3318 the content length header check breaks distcp compatibility with previous releases (0.20.2 and earlier, and 0.21). Like branch-1 this check should be lenient.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3671) ByteRangeInputStream shouldn't
require the content length header be present
Posted by "Daryn Sharp (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13416233#comment-13416233 ]
Daryn Sharp commented on HDFS-3671:
-----------------------------------
Since any fix is a hacky workaround for very old hadoop clusters that violate the http rfc (see below), should we only increase the read timeout if a content-length is missing? Or as I suggested before, use a file stat (or simple HEAD request) if the content-length is missing? The latter will prevent a minimum 200s tail on the transfers.
(An http server is required to close the connection when a transfer lacks a content-length is complete in order to avoid this ambiguity on the client side. However it's strongly advised not to leave out the header since the client doesn't know if it only received a partial download, hence my suggestion to fallback to size from a file stat.)
> ByteRangeInputStream shouldn't require the content length header be present
> ---------------------------------------------------------------------------
>
> Key: HDFS-3671
> URL: https://issues.apache.org/jira/browse/HDFS-3671
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 2.0.0-alpha
> Reporter: Eli Collins
> Priority: Blocker
>
> Per HDFS-3318 the content length header check breaks distcp compatibility with previous releases (0.20.2 and earlier, and 0.21). Like branch-1 this check should be lenient.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3671) ByteRangeInputStream shouldn't
require the content length header be present
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13419008#comment-13419008 ]
Hadoop QA commented on HDFS-3671:
---------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12537309/h3671_20120719.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 1 new or modified test files.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2876//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2876//console
This message is automatically generated.
> ByteRangeInputStream shouldn't require the content length header be present
> ---------------------------------------------------------------------------
>
> Key: HDFS-3671
> URL: https://issues.apache.org/jira/browse/HDFS-3671
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 2.0.0-alpha
> Reporter: Eli Collins
> Priority: Blocker
> Attachments: h3671_20120717.patch, h3671_20120719.patch
>
>
> Per HDFS-3318 the content length header check breaks distcp compatibility with previous releases (0.20.2 and earlier, and 0.21). Like branch-1 this check should be lenient.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3671) ByteRangeInputStream shouldn't
require the content length header be present
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13432937#comment-13432937 ]
Hadoop QA commented on HDFS-3671:
---------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12540609/h3671_20120812.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 1 new or modified test files.
-1 javac. The patch appears to cause the build to fail.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2988//console
This message is automatically generated.
> ByteRangeInputStream shouldn't require the content length header be present
> ---------------------------------------------------------------------------
>
> Key: HDFS-3671
> URL: https://issues.apache.org/jira/browse/HDFS-3671
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 2.0.0-alpha
> Reporter: Eli Collins
> Priority: Blocker
> Attachments: h3671_20120717.patch, h3671_20120719.patch, h3671_20120812.patch
>
>
> Per HDFS-3318 the content length header check breaks distcp compatibility with previous releases (0.20.2 and earlier, and 0.21). Like branch-1 this check should be lenient.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3671) ByteRangeInputStream shouldn't
require the content length header be present
Posted by "Daryn Sharp (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13415469#comment-13415469 ]
Daryn Sharp commented on HDFS-3671:
-----------------------------------
The dilemma is that if the content-length is not defined (and chunking is not used), then the client doesn't know when it has received all data. The client hangs after all data is received until the remote server closes the connection. The server's connection idle timeout is greater than the client's idle timeout on reads, so the download will fail even if all bytes were received. The other problem with no content-length is the client won't know if it only received a partial download.
> ByteRangeInputStream shouldn't require the content length header be present
> ---------------------------------------------------------------------------
>
> Key: HDFS-3671
> URL: https://issues.apache.org/jira/browse/HDFS-3671
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 2.0.0-alpha
> Reporter: Eli Collins
> Priority: Blocker
>
> Per HDFS-3318 the content length header check breaks distcp compatibility with previous releases (0.20.2 and earlier, and 0.21). Like branch-1 this check should be lenient.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira