You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Attila Magyar (Jira)" <ji...@apache.org> on 2020/10/29 12:35:00 UTC
[jira] [Commented] (HADOOP-16881) PseudoAuthenticator does not
disconnect HttpURLConnection leading to CLOSE_WAIT cnxns
[ https://issues.apache.org/jira/browse/HADOOP-16881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17222866#comment-17222866 ]
Attila Magyar commented on HADOOP-16881:
----------------------------------------
Hey @[~stevel@apache.org], do you have any insights on this? There seems to be a socket leak in KerberosAuthentication/PseudoAuthenticator which causes lots of CLOSE_WAIT sockets to pile up and they're never cleared up. Adding explicit disconnect calls to the http client solves the issue. However it might prevent the connection pooling from reusing sockets.
{code:java}
diff --git a/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/client/KerberosAuthenticator.java b/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/client/KerberosAuthenticator.java
index 3bfa349880c..c035dd44ce0 100644
--- a/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/client/KerberosAuthenticator.java
+++ b/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/client/KerberosAuthenticator.java
@@ -183,8 +183,9 @@ public void authenticate(URL url, AuthenticatedURL.Token token)
if (!token.isSet()) {
this.url = url;
base64 = new Base64(0);
+ HttpURLConnection conn = null;
try {
- HttpURLConnection conn = token.openConnection(url, connConfigurator);
+ conn = token.openConnection(url, connConfigurator);
conn.setRequestMethod(AUTH_HTTP_METHOD);
conn.connect();
@@ -218,6 +219,10 @@ public void authenticate(URL url, AuthenticatedURL.Token token)
} catch (AuthenticationException ex){
throw wrapExceptionWithMessage(ex,
"Error while authenticating with endpoint: " + url);
+ } finally {
+ if (conn != null) {
+ conn.disconnect();
+ }
}
}
}
diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/web/DelegationTokenAuthenticator.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/web/DelegationTokenAuthenticator.java
index 4e2ee4fdbea..8546a76c1af 100644
--- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/web/DelegationTokenAuthenticator.java
+++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/web/DelegationTokenAuthenticator.java
@@ -312,8 +312,9 @@ private Map doDelegationTokenOperation(URL url,
dt = ((DelegationTokenAuthenticatedURL.Token) token).getDelegationToken();
((DelegationTokenAuthenticatedURL.Token) token).setDelegationToken(null);
}
+ HttpURLConnection conn = null;
try {
- HttpURLConnection conn = aUrl.openConnection(url, token);
+ conn = aUrl.openConnection(url, token);
conn.setRequestMethod(operation.getHttpMethod());
HttpExceptionUtils.validateResponse(conn, HttpURLConnection.HTTP_OK);
if (hasResponse) {
@@ -339,6 +340,9 @@ private Map doDelegationTokenOperation(URL url,
if (dt != null) {
((DelegationTokenAuthenticatedURL.Token) token).setDelegationToken(dt);
}
+ if (conn != null) {
+ conn.disconnect();
+ }
}
return ret;
}
{code}
> PseudoAuthenticator does not disconnect HttpURLConnection leading to CLOSE_WAIT cnxns
> -------------------------------------------------------------------------------------
>
> Key: HADOOP-16881
> URL: https://issues.apache.org/jira/browse/HADOOP-16881
> Project: Hadoop Common
> Issue Type: Bug
> Components: auth, security
> Affects Versions: 3.3.0
> Reporter: Prabhu Joseph
> Assignee: Prabhu Joseph
> Priority: Major
>
> PseudoAuthenticator and KerberosAuthentication does not disconnect HttpURLConnection leading to lot of CLOSE_WAIT connections. YARN-8414 issue is observed due to this.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org