You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Daryn Sharp (Created) (JIRA)" <ji...@apache.org> on 2012/03/28 18:41:27 UTC
[jira] [Created] (HADOOP-8221) Hftp connections does not have a
timeout
Hftp connections does not have a timeout
----------------------------------------
Key: HADOOP-8221
URL: https://issues.apache.org/jira/browse/HADOOP-8221
Project: Hadoop Common
Issue Type: Bug
Components: fs
Reporter: Daryn Sharp
Assignee: Daryn Sharp
Priority: Critical
Hftp connections do not have read timeouts. This leads to indefinitely hung sockets when there is a network outage during which time the remote host closed the socket.
This may also affect WebHdfs, etc.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8221) Hftp connections do not have a
timeout
Posted by "Daryn Sharp (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daryn Sharp updated HADOOP-8221:
--------------------------------
Status: Patch Available (was: Open)
Resubmitting same patch to kick the build since it applies fine for me...
> Hftp connections do not have a timeout
> --------------------------------------
>
> Key: HADOOP-8221
> URL: https://issues.apache.org/jira/browse/HADOOP-8221
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
> Affects Versions: 0.23.0, 0.24.0
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
> Priority: Critical
> Attachments: HADOOP-8221.patch
>
>
> Hftp connections do not have read timeouts. This leads to indefinitely hung sockets when there is a network outage during which time the remote host closed the socket.
> This may also affect WebHdfs, etc.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8221) Hftp connections do not have a
timeout
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13241920#comment-13241920 ]
Hadoop QA commented on HADOOP-8221:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12520529/HADOOP-8221.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
-1 patch. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/799//console
This message is automatically generated.
> Hftp connections do not have a timeout
> --------------------------------------
>
> Key: HADOOP-8221
> URL: https://issues.apache.org/jira/browse/HADOOP-8221
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
> Affects Versions: 0.23.0, 0.24.0
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
> Priority: Critical
> Attachments: HADOOP-8221.patch
>
>
> Hftp connections do not have read timeouts. This leads to indefinitely hung sockets when there is a network outage during which time the remote host closed the socket.
> This may also affect WebHdfs, etc.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8221) Hftp connections do not have a
timeout
Posted by "Kihwal Lee (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13242366#comment-13242366 ]
Kihwal Lee commented on HADOOP-8221:
------------------------------------
It makes sense to get rid of throwIOExceptionFromConnection(). If connection attempt failed or sending request resulted in an exception, it shouldn't try to check the http response code.
+1 Looks good to me. Don't we need the same patch for branch-1?
> Hftp connections do not have a timeout
> --------------------------------------
>
> Key: HADOOP-8221
> URL: https://issues.apache.org/jira/browse/HADOOP-8221
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
> Affects Versions: 0.23.0, 0.24.0
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
> Priority: Critical
> Attachments: HADOOP-8221.patch
>
>
> Hftp connections do not have read timeouts. This leads to indefinitely hung sockets when there is a network outage during which time the remote host closed the socket.
> This may also affect WebHdfs, etc.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8221) Hftp connections do not have a
timeout
Posted by "Daryn Sharp (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daryn Sharp updated HADOOP-8221:
--------------------------------
Attachment: HADOOP-8221.patch
This is a minimalist fix to get jobs unblocked.
> Hftp connections do not have a timeout
> --------------------------------------
>
> Key: HADOOP-8221
> URL: https://issues.apache.org/jira/browse/HADOOP-8221
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
> Priority: Critical
> Attachments: HADOOP-8221.patch
>
>
> Hftp connections do not have read timeouts. This leads to indefinitely hung sockets when there is a network outage during which time the remote host closed the socket.
> This may also affect WebHdfs, etc.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8221) Hftp connections do not have a
timeout
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13242577#comment-13242577 ]
Hadoop QA commented on HADOOP-8221:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12520632/HADOOP-8221.branch-1.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
-1 patch. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/804//console
This message is automatically generated.
> Hftp connections do not have a timeout
> --------------------------------------
>
> Key: HADOOP-8221
> URL: https://issues.apache.org/jira/browse/HADOOP-8221
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
> Affects Versions: 0.23.0, 0.24.0
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
> Priority: Critical
> Attachments: HADOOP-8221.branch-1.patch, HADOOP-8221.patch
>
>
> Hftp connections do not have read timeouts. This leads to indefinitely hung sockets when there is a network outage during which time the remote host closed the socket.
> This may also affect WebHdfs, etc.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8221) Hftp connections do not have a
timeout
Posted by "Daryn Sharp (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13242395#comment-13242395 ]
Daryn Sharp commented on HADOOP-8221:
-------------------------------------
Yes, I should have commented that {{throwIOExceptionFromConnection}} is wrong. Http error codes do not generate an exception. If an exception occurs during the connect, something seriously went wrong. Ie. connect error, could send request, SSL negotiation failed, etc. Thus the client is left in a blocking read waiting for a response that will never come. The methods like {{getInputStream}} read the response code and headers.
On a side note, it's also odd that the DN's jetty doesn't have a timeout while waiting for a request. Maybe it does, but when the DN is getting jammed the timeout isn't kicking in. To clarify for others, we are addressing three problems:
# Socket is left dangling because the remote host closed the socket. Not sure why unless there's a linux kernel bug (unlikely?) or the tcp FIN packets were somehow lost.
# DN accepts the connection, but never sends a response.
# DN host becomes a "zombie". The host is inexplicably hung such that you can't even ssh or console into the box. Sockets connect into the listen backlog, but are never accepted and processed.
I'll work on a patch for 1.x.
> Hftp connections do not have a timeout
> --------------------------------------
>
> Key: HADOOP-8221
> URL: https://issues.apache.org/jira/browse/HADOOP-8221
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
> Affects Versions: 0.23.0, 0.24.0
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
> Priority: Critical
> Attachments: HADOOP-8221.patch
>
>
> Hftp connections do not have read timeouts. This leads to indefinitely hung sockets when there is a network outage during which time the remote host closed the socket.
> This may also affect WebHdfs, etc.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8221) Hftp connections do not have a
timeout
Posted by "Daryn Sharp (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daryn Sharp updated HADOOP-8221:
--------------------------------
Target Version/s: 0.23.2 (was: 0.24.0, 0.23.2)
Affects Version/s: 0.24.0
0.23.0
Status: Patch Available (was: Open)
> Hftp connections do not have a timeout
> --------------------------------------
>
> Key: HADOOP-8221
> URL: https://issues.apache.org/jira/browse/HADOOP-8221
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
> Affects Versions: 0.23.0, 0.24.0
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
> Priority: Critical
> Attachments: HADOOP-8221.patch
>
>
> Hftp connections do not have read timeouts. This leads to indefinitely hung sockets when there is a network outage during which time the remote host closed the socket.
> This may also affect WebHdfs, etc.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8221) Hftp connections do not have a
timeout
Posted by "Daryn Sharp (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daryn Sharp updated HADOOP-8221:
--------------------------------
Attachment: HADOOP-8221.branch-1.patch
> Hftp connections do not have a timeout
> --------------------------------------
>
> Key: HADOOP-8221
> URL: https://issues.apache.org/jira/browse/HADOOP-8221
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
> Affects Versions: 0.23.0, 0.24.0
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
> Priority: Critical
> Attachments: HADOOP-8221.branch-1.patch, HADOOP-8221.patch
>
>
> Hftp connections do not have read timeouts. This leads to indefinitely hung sockets when there is a network outage during which time the remote host closed the socket.
> This may also affect WebHdfs, etc.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8221) Hftp connections do not have a
timeout
Posted by "Daryn Sharp (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daryn Sharp updated HADOOP-8221:
--------------------------------
Status: Open (was: Patch Available)
> Hftp connections do not have a timeout
> --------------------------------------
>
> Key: HADOOP-8221
> URL: https://issues.apache.org/jira/browse/HADOOP-8221
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
> Affects Versions: 0.23.0, 0.24.0
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
> Priority: Critical
> Attachments: HADOOP-8221.patch
>
>
> Hftp connections do not have read timeouts. This leads to indefinitely hung sockets when there is a network outage during which time the remote host closed the socket.
> This may also affect WebHdfs, etc.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8221) Hftp connections do not have a
timeout
Posted by "Daryn Sharp (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daryn Sharp updated HADOOP-8221:
--------------------------------
Target Version/s: 0.23.2, 0.24.0 (was: 0.24.0, 0.23.2)
Summary: Hftp connections do not have a timeout (was: Hftp connections does not have a timeout)
> Hftp connections do not have a timeout
> --------------------------------------
>
> Key: HADOOP-8221
> URL: https://issues.apache.org/jira/browse/HADOOP-8221
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
> Priority: Critical
>
> Hftp connections do not have read timeouts. This leads to indefinitely hung sockets when there is a network outage during which time the remote host closed the socket.
> This may also affect WebHdfs, etc.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira