You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@zookeeper.apache.org by GitBox <gi...@apache.org> on 2021/05/11 07:36:30 UTC

[GitHub] [zookeeper] rvaleti opened a new pull request #1692: ZOOKEEPER-4235 Java Client SendThread does not clean up created objects during constructor of SaslClient and Login

rvaleti opened a new pull request #1692:
URL: https://github.com/apache/zookeeper/pull/1692


   Change Description - Move initialization of ZookeeperSaslClient & Login to instance methods.
   
   - Moved ZookeeperSaslClient's initialization from constructor to  instance method createSaslClient()
   - Moved Login's initialization from constructor to an instance method loginAndStartThreadIfRequired()
   - Moved the anonymous Runnable implementation of login thread to LoginRunnable inner class.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [zookeeper] dbwong commented on pull request #1692: ZOOKEEPER-4235 Java Client SendThread does not clean up created objects during constructor of SaslClient and Login

Posted by GitBox <gi...@apache.org>.
dbwong commented on pull request #1692:
URL: https://github.com/apache/zookeeper/pull/1692#issuecomment-841857467


   Overall looks okay to me.  Do you think it is worth making this code more testable via introduction of a factory pattern (or alternatively something like power mock)?  And verifying the shutdown paths now work/didn't work?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [zookeeper] rvaleti commented on pull request #1692: ZOOKEEPER-4235 Java Client SendThread does not clean up created objects during constructor of SaslClient and Login

Posted by GitBox <gi...@apache.org>.
rvaleti commented on pull request #1692:
URL: https://github.com/apache/zookeeper/pull/1692#issuecomment-873831223


   @eolivelli If you are happy with the PR, can you please approve?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@zookeeper.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [zookeeper] eolivelli commented on a change in pull request #1692: ZOOKEEPER-4235 Java Client SendThread does not clean up created objects during constructor of SaslClient and Login

Posted by GitBox <gi...@apache.org>.
eolivelli commented on a change in pull request #1692:
URL: https://github.com/apache/zookeeper/pull/1692#discussion_r652156962



##########
File path: zookeeper-server/src/main/java/org/apache/zookeeper/ClientCnxn.java
##########
@@ -1148,7 +1148,8 @@ private void startConnect(InetSocketAddress addr) throws IOException {
                     if (zooKeeperSaslClient != null) {
                         zooKeeperSaslClient.shutdown();
                     }
-                    zooKeeperSaslClient = new ZooKeeperSaslClient(SaslServerPrincipal.getServerPrincipal(addr, clientConfig), clientConfig);
+                    zooKeeperSaslClient = new ZooKeeperSaslClient(clientConfig);
+                    zooKeeperSaslClient.createSaslClient(SaslServerPrincipal.getServerPrincipal(addr, clientConfig));

Review comment:
       The benefit is that you will never see the constructor fail and then you can have a handle to perform cleanup




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [zookeeper] rvaleti commented on a change in pull request #1692: ZOOKEEPER-4235 Java Client SendThread does not clean up created objects during constructor of SaslClient and Login

Posted by GitBox <gi...@apache.org>.
rvaleti commented on a change in pull request #1692:
URL: https://github.com/apache/zookeeper/pull/1692#discussion_r647618032



##########
File path: zookeeper-server/src/main/java/org/apache/zookeeper/Login.java
##########
@@ -95,10 +95,159 @@
      *             Thrown if authentication fails.
      */
     public Login(final String loginContextName, CallbackHandler callbackHandler, final ZKConfig zkConfig) throws LoginException {
-        this.zkConfig = zkConfig;
+        this.loginContextName = loginContextName;
         this.callbackHandler = callbackHandler;
+        this.zkConfig = zkConfig;
+    }
+
+    private class LoginRunnable implements Runnable {
+
+        @Override
+        public void run() {
+            LOG.info("TGT refresh thread started.");
+            while (true) {  // renewal thread's main loop. if it exits from here, thread will exit.
+                KerberosTicket tgt = getTGT();
+                long now = Time.currentWallTime();
+                long nextRefresh;
+                Date nextRefreshDate;
+                if (tgt == null) {
+                    nextRefresh = now + MIN_TIME_BEFORE_RELOGIN;
+                    nextRefreshDate = new Date(nextRefresh);
+                    LOG.warn("No TGT found: will try again at {}", nextRefreshDate);
+                } else {
+                    nextRefresh = getRefreshTime(tgt);
+                    long expiry = tgt.getEndTime().getTime();
+                    Date expiryDate = new Date(expiry);
+                    if ((isUsingTicketCache) && (tgt.getEndTime().equals(tgt.getRenewTill()))) {
+                        LOG.error(
+                            "The TGT cannot be renewed beyond the next expiry date: {}."
+                                + "This process will not be able to authenticate new SASL connections after that "
+                                + "time (for example, it will not be authenticate a new connection with a Zookeeper "
+                                + "Quorum member).  Ask your system administrator to either increase the "
+                                + "'renew until' time by doing : 'modprinc -maxrenewlife {}' within "
+                                + "kadmin, or instead, to generate a keytab for {}. Because the TGT's "
+                                + "expiry cannot be further extended by refreshing, exiting refresh thread now.",
+                            expiryDate,
+                            principal,
+                            principal);
+                        return;
+                    }
+                    // determine how long to sleep from looking at ticket's expiry.
+                    // We should not allow the ticket to expire, but we should take into consideration
+                    // MIN_TIME_BEFORE_RELOGIN. Will not sleep less than MIN_TIME_BEFORE_RELOGIN, unless doing so
+                    // would cause ticket expiration.
+                    if ((nextRefresh > expiry) || ((now + MIN_TIME_BEFORE_RELOGIN) > expiry)) {
+                        // expiry is before next scheduled refresh).
+                        nextRefresh = now;
+                    } else {
+                        if (nextRefresh < (now + MIN_TIME_BEFORE_RELOGIN)) {
+                            // next scheduled refresh is sooner than (now + MIN_TIME_BEFORE_LOGIN).
+                            Date until = new Date(nextRefresh);
+                            Date newuntil = new Date(now + MIN_TIME_BEFORE_RELOGIN);
+                            LOG.warn(
+                                "TGT refresh thread time adjusted from : {} to : {} since "
+                                    + "the former is sooner than the minimum refresh interval ("
+                                    + "{} seconds) from now.",
+                                until,
+                                newuntil,
+                                (MIN_TIME_BEFORE_RELOGIN / 1000));
+                        }
+                        nextRefresh = Math.max(nextRefresh, now + MIN_TIME_BEFORE_RELOGIN);
+                    }
+                    nextRefreshDate = new Date(nextRefresh);
+                    if (nextRefresh > expiry) {
+                        LOG.error(
+                            "next refresh: {} is later than expiry {}."
+                                + " This may indicate a clock skew problem."
+                                + " Check that this host and the KDC's "
+                                + "hosts' clocks are in sync. Exiting refresh thread.",
+                            nextRefreshDate,
+                            expiryDate);
+                        return;
+                    }
+                }
+                if (now == nextRefresh) {
+                    LOG.info("refreshing now because expiry is before next scheduled refresh time.");
+                } else if (now < nextRefresh) {
+                    Date until = new Date(nextRefresh);
+                    LOG.info("TGT refresh sleeping until: {}", until.toString());
+                    try {
+                        Thread.sleep(nextRefresh - now);
+                    } catch (InterruptedException ie) {
+                        LOG.warn("TGT renewal thread has been interrupted and will exit.");
+                        break;
+                    }
+                } else {
+                    LOG.error(
+                        "nextRefresh:{} is in the past: exiting refresh thread. Check"
+                            + " clock sync between this host and KDC - (KDC's clock is likely ahead of this host)."
+                            + " Manual intervention will be required for this client to successfully authenticate."
+                            + " Exiting refresh thread.",
+                        nextRefreshDate);
+                    break;
+                }
+                if (isUsingTicketCache) {
+                    String cmd = zkConfig.getProperty(ZKConfig.KINIT_COMMAND, KINIT_COMMAND_DEFAULT);
+                    String kinitArgs = "-R";
+                    int retry = 1;
+                    while (retry >= 0) {
+                        try {
+                            LOG.debug("running ticket cache refresh command: {} {}", cmd, kinitArgs);
+                            Shell.execCommand(cmd, kinitArgs);
+                            break;
+                        } catch (Exception e) {
+                            if (retry > 0) {
+                                --retry;
+                                // sleep for 10 seconds
+                                try {
+                                    Thread.sleep(10 * 1000);
+                                } catch (InterruptedException ie) {
+                                    LOG.error("Interrupted while renewing TGT, exiting Login thread");
+                                    return;
+                                }
+                            } else {
+                                LOG.warn(
+                                    "Could not renew TGT due to problem running shell command: '{} {}'."
+                                        + " Exiting refresh thread.",
+                                    cmd,
+                                    kinitArgs,
+                                    e);
+                                return;
+                            }
+                        }
+                    }
+                }
+                try {
+                    int retry = 1;
+                    while (retry >= 0) {
+                        try {
+                            reLogin();
+                            break;
+                        } catch (LoginException le) {
+                            if (retry > 0) {
+                                --retry;
+                                // sleep for 10 seconds.
+                                try {
+                                    Thread.sleep(10 * 1000);
+                                } catch (InterruptedException e) {
+                                    LOG.error("Interrupted during login retry after LoginException:", le);
+                                    throw le;
+                                }
+                            } else {
+                                LOG.error("Could not refresh TGT for principal: {}.", principal, le);
+                            }
+                        }
+                    }
+                } catch (LoginException le) {
+                    LOG.error("Failed to refresh TGT: refresh thread exiting now.", le);
+                    break;
+                }
+            }
+        }
+    }
+
+    public void loginAndStartThreadIfRequired() throws LoginException {
         login = login(loginContextName);
-        this.loginContextName = loginContextName;
         subject = login.getSubject();
         isKrbTicket = !subject.getPrivateCredentials(KerberosTicket.class).isEmpty();
         AppConfigurationEntry[] entries = Configuration.getConfiguration().getAppConfigurationEntry(loginContextName);

Review comment:
       @anmolnar 
   
   Yes, the refresh thread of Login object is not cleaned up leaving many login threads.
   The Jira's description is missing one key stack trace
   
   ```
   17971 Threads with the following stack trace
   
   3911 - TIMED_WAITING - Thread-822 | ORG: null RAC: None
   java.base@11.0.9/java.lang.Thread.sleep(Native Method)
   org.apache.zookeeper.Login$1.run(Login.java:197)
   java.base@11.0.9/java.lang.Thread.run(Thread.java:834)
   ```
   
   which when combined with another stack trace coming from ZookeeperSaslClient
   
   ```
   java.lang.Thread.State: RUNNABLE at java.net.PlainSocketImpl.socketConnect(java.base@11.0.4.0.101/Native Method) at java.net.AbstractPlainSocketImpl.doConnect(java.base@11.0.4.0.101/AbstractPlainSocketImpl.java:399) - locked <0x00000015004fde20> (a java.net.SocksSocketImpl) at java.net.AbstractPlainSocketImpl.connectToAddress(java.base@11.0.4.0.101/AbstractPlainSocketImpl.java:242) at java.net.AbstractPlainSocketImpl.connect(java.base@11.0.4.0.101/AbstractPlainSocketImpl.java:224) at java.net.SocksSocketImpl.connect(java.base@11.0.4.0.101/SocksSocketImpl.java:403) at java.net.Socket.connect(java.base@11.0.4.0.101/Socket.java:609) at sun.security.krb5.internal.TCPClient.<init>(java.security.jgss@11.0.4.0.101/NetClient.java:62) at sun.security.krb5.internal.NetClient.getInstance(java.security.jgss@11.0.4.0.101/NetClient.java:42) at sun.security.krb5.KdcComm$KdcCommunication.run(java.security.jgss@11.0.4.0.101/KdcComm.java:401) at sun.security.krb5.KdcComm$KdcCommunication.run(java.s
 ecurity.jgss@11.0.4.0.101/KdcComm.java:364) at java.security.AccessController.doPrivileged(java.base@11.0.4.0.101/Native Method) at sun.security.krb5.KdcComm.send(java.security.jgss@11.0.4.0.101/KdcComm.java:348) at sun.security.krb5.KdcComm.sendIfPossible(java.security.jgss@11.0.4.0.101/KdcComm.java:253) at sun.security.krb5.KdcComm.send(java.security.jgss@11.0.4.0.101/KdcComm.java:234) at sun.security.krb5.KdcComm.send(java.security.jgss@11.0.4.0.101/KdcComm.java:200) at sun.security.krb5.KrbAsReqBuilder.send(java.security.jgss@11.0.4.0.101/KrbAsReqBuilder.java:326) at sun.security.krb5.KrbAsReqBuilder.action(java.security.jgss@11.0.4.0.101/KrbAsReqBuilder.java:371) at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(jdk.security.auth@11.0.4.0.101/Krb5LoginModule.java:754) at com.sun.security.auth.module.Krb5LoginModule.login(jdk.security.auth@11.0.4.0.101/Krb5LoginModule.java:592) at javax.security.auth.login.LoginContext.invoke(java.base@11.0.4.0.101/LoginConte
 xt.java:726) at javax.security.auth.login.LoginContext$4.run(java.base@11.0.4.0.101/LoginContext.java:665) at javax.security.auth.login.LoginContext$4.run(java.base@11.0.4.0.101/LoginContext.java:663) at java.security.AccessController.doPrivileged(java.base@11.0.4.0.101/Native Method) at javax.security.auth.login.LoginContext.invokePriv(java.base@11.0.4.0.101/LoginContext.javaa:663) at javax.security.auth.login.LoginContext.login(java.base@11.0.4.0.101/LoginContext.java:574) at org.apache.zookeeper.Login.login(Login.java:304) - locked <0x000000151c477148> (a org.apache.zookeeper.Login) at org.apache.zookeeper.Login.<init>(Login.java:106) at org.apache.zookeeper.client.ZooKeeperSaslClient.createSaslClient(ZooKeeperSaslClient.java:249) - locked <0x000000151c476f68> (a org.apache.zookeeper.client.ZooKeeperSaslClient) at org.apache.zookeeper.client.ZooKeeperSaslClient.<init>(ZooKeeperSaslClient.java:141) at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:972) at 
 org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1031)
   ```
   
   Here is one possibility of Login threads may not cleared by shutdown()
   
   1. Zookeeper initialization creates & starts ClientCnxn -> spins up SendThread and starts it
   2. ClientCnxn.SendThread.run() calls ZookeeperSaslClient.<init> -> createSaslClient()
   3. Login.<init> -> Login.login() -> loginContext.login()
   4. Let's say thread1 executing a login (loginContext.login()) takes few seconds.
   5. Meantime, let's say some one called Zookeeper.close() -> ClientCnxn.close() -> ClientCnxn.disconnect()
   6. At this point, since ZooKeeperSaslClient's constructor is not yet finished, it's zooKeeperSaslClient.shutdown() is never called.
   7. Once 3 get successful, a login thread refresh is created which would become a zombie.
   
   So, the idea is 2 parts
   1. Move the actual "login" which can take time in to separate methods and make ZooKeeperSaslClient and Login constructors lightweight so that we have a reference to these at all times.
   2. Make sure either Login thread is interrupted (when shutdown called after the thread was started) OR not started (when shutdown called before the thread was started) - This was missed earlier. Updated files in PR to reflect this.
   
   Please let me know your thoughts.
   
   
   




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [zookeeper] rvaleti commented on pull request #1692: ZOOKEEPER-4235 Java Client SendThread does not clean up created objects during constructor of SaslClient and Login

Posted by GitBox <gi...@apache.org>.
rvaleti commented on pull request #1692:
URL: https://github.com/apache/zookeeper/pull/1692#issuecomment-920047026


   > @rvaleti - Is this done. I was wondering if we can pick it in our light fork?
   
   No pending work from the PR perspective. It's just that we need to get a green build (we had a green build once during the changes).
   
   @eolivelli @anmolnar could you please re-trigger the build and see if it passes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@zookeeper.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [zookeeper] anmolnar commented on pull request #1692: ZOOKEEPER-4235 Java Client SendThread does not clean up created objects during constructor of SaslClient and Login

Posted by GitBox <gi...@apache.org>.
anmolnar commented on pull request #1692:
URL: https://github.com/apache/zookeeper/pull/1692#issuecomment-868830808


   @rvaleti You got a green build now. Let's wait for @eolivelli 's approval and you're good to go.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@zookeeper.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [zookeeper] eolivelli commented on pull request #1692: ZOOKEEPER-4235 Java Client SendThread does not clean up created objects during constructor of SaslClient and Login

Posted by GitBox <gi...@apache.org>.
eolivelli commented on pull request #1692:
URL: https://github.com/apache/zookeeper/pull/1692#issuecomment-873964682


   I have forced a new run of the CI job


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@zookeeper.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [zookeeper] eolivelli commented on pull request #1692: ZOOKEEPER-4235 Java Client SendThread does not clean up created objects during constructor of SaslClient and Login

Posted by GitBox <gi...@apache.org>.
eolivelli commented on pull request #1692:
URL: https://github.com/apache/zookeeper/pull/1692#issuecomment-873964682


   I have forced a new run of the CI job


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@zookeeper.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [zookeeper] rvaleti commented on pull request #1692: ZOOKEEPER-4235 Java Client SendThread does not clean up created objects during constructor of SaslClient and Login

Posted by GitBox <gi...@apache.org>.
rvaleti commented on pull request #1692:
URL: https://github.com/apache/zookeeper/pull/1692#issuecomment-875977033


   @eolivelli , looks like previous build failure was due to server not coming up and not related to this change. And @anmolnar  forced a build earlier that succeeded. Do you want to re trigger the build or merge the PR?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@zookeeper.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [zookeeper] rvaleti commented on a change in pull request #1692: ZOOKEEPER-4235 Java Client SendThread does not clean up created objects during constructor of SaslClient and Login

Posted by GitBox <gi...@apache.org>.
rvaleti commented on a change in pull request #1692:
URL: https://github.com/apache/zookeeper/pull/1692#discussion_r647618032



##########
File path: zookeeper-server/src/main/java/org/apache/zookeeper/Login.java
##########
@@ -95,10 +95,159 @@
      *             Thrown if authentication fails.
      */
     public Login(final String loginContextName, CallbackHandler callbackHandler, final ZKConfig zkConfig) throws LoginException {
-        this.zkConfig = zkConfig;
+        this.loginContextName = loginContextName;
         this.callbackHandler = callbackHandler;
+        this.zkConfig = zkConfig;
+    }
+
+    private class LoginRunnable implements Runnable {
+
+        @Override
+        public void run() {
+            LOG.info("TGT refresh thread started.");
+            while (true) {  // renewal thread's main loop. if it exits from here, thread will exit.
+                KerberosTicket tgt = getTGT();
+                long now = Time.currentWallTime();
+                long nextRefresh;
+                Date nextRefreshDate;
+                if (tgt == null) {
+                    nextRefresh = now + MIN_TIME_BEFORE_RELOGIN;
+                    nextRefreshDate = new Date(nextRefresh);
+                    LOG.warn("No TGT found: will try again at {}", nextRefreshDate);
+                } else {
+                    nextRefresh = getRefreshTime(tgt);
+                    long expiry = tgt.getEndTime().getTime();
+                    Date expiryDate = new Date(expiry);
+                    if ((isUsingTicketCache) && (tgt.getEndTime().equals(tgt.getRenewTill()))) {
+                        LOG.error(
+                            "The TGT cannot be renewed beyond the next expiry date: {}."
+                                + "This process will not be able to authenticate new SASL connections after that "
+                                + "time (for example, it will not be authenticate a new connection with a Zookeeper "
+                                + "Quorum member).  Ask your system administrator to either increase the "
+                                + "'renew until' time by doing : 'modprinc -maxrenewlife {}' within "
+                                + "kadmin, or instead, to generate a keytab for {}. Because the TGT's "
+                                + "expiry cannot be further extended by refreshing, exiting refresh thread now.",
+                            expiryDate,
+                            principal,
+                            principal);
+                        return;
+                    }
+                    // determine how long to sleep from looking at ticket's expiry.
+                    // We should not allow the ticket to expire, but we should take into consideration
+                    // MIN_TIME_BEFORE_RELOGIN. Will not sleep less than MIN_TIME_BEFORE_RELOGIN, unless doing so
+                    // would cause ticket expiration.
+                    if ((nextRefresh > expiry) || ((now + MIN_TIME_BEFORE_RELOGIN) > expiry)) {
+                        // expiry is before next scheduled refresh).
+                        nextRefresh = now;
+                    } else {
+                        if (nextRefresh < (now + MIN_TIME_BEFORE_RELOGIN)) {
+                            // next scheduled refresh is sooner than (now + MIN_TIME_BEFORE_LOGIN).
+                            Date until = new Date(nextRefresh);
+                            Date newuntil = new Date(now + MIN_TIME_BEFORE_RELOGIN);
+                            LOG.warn(
+                                "TGT refresh thread time adjusted from : {} to : {} since "
+                                    + "the former is sooner than the minimum refresh interval ("
+                                    + "{} seconds) from now.",
+                                until,
+                                newuntil,
+                                (MIN_TIME_BEFORE_RELOGIN / 1000));
+                        }
+                        nextRefresh = Math.max(nextRefresh, now + MIN_TIME_BEFORE_RELOGIN);
+                    }
+                    nextRefreshDate = new Date(nextRefresh);
+                    if (nextRefresh > expiry) {
+                        LOG.error(
+                            "next refresh: {} is later than expiry {}."
+                                + " This may indicate a clock skew problem."
+                                + " Check that this host and the KDC's "
+                                + "hosts' clocks are in sync. Exiting refresh thread.",
+                            nextRefreshDate,
+                            expiryDate);
+                        return;
+                    }
+                }
+                if (now == nextRefresh) {
+                    LOG.info("refreshing now because expiry is before next scheduled refresh time.");
+                } else if (now < nextRefresh) {
+                    Date until = new Date(nextRefresh);
+                    LOG.info("TGT refresh sleeping until: {}", until.toString());
+                    try {
+                        Thread.sleep(nextRefresh - now);
+                    } catch (InterruptedException ie) {
+                        LOG.warn("TGT renewal thread has been interrupted and will exit.");
+                        break;
+                    }
+                } else {
+                    LOG.error(
+                        "nextRefresh:{} is in the past: exiting refresh thread. Check"
+                            + " clock sync between this host and KDC - (KDC's clock is likely ahead of this host)."
+                            + " Manual intervention will be required for this client to successfully authenticate."
+                            + " Exiting refresh thread.",
+                        nextRefreshDate);
+                    break;
+                }
+                if (isUsingTicketCache) {
+                    String cmd = zkConfig.getProperty(ZKConfig.KINIT_COMMAND, KINIT_COMMAND_DEFAULT);
+                    String kinitArgs = "-R";
+                    int retry = 1;
+                    while (retry >= 0) {
+                        try {
+                            LOG.debug("running ticket cache refresh command: {} {}", cmd, kinitArgs);
+                            Shell.execCommand(cmd, kinitArgs);
+                            break;
+                        } catch (Exception e) {
+                            if (retry > 0) {
+                                --retry;
+                                // sleep for 10 seconds
+                                try {
+                                    Thread.sleep(10 * 1000);
+                                } catch (InterruptedException ie) {
+                                    LOG.error("Interrupted while renewing TGT, exiting Login thread");
+                                    return;
+                                }
+                            } else {
+                                LOG.warn(
+                                    "Could not renew TGT due to problem running shell command: '{} {}'."
+                                        + " Exiting refresh thread.",
+                                    cmd,
+                                    kinitArgs,
+                                    e);
+                                return;
+                            }
+                        }
+                    }
+                }
+                try {
+                    int retry = 1;
+                    while (retry >= 0) {
+                        try {
+                            reLogin();
+                            break;
+                        } catch (LoginException le) {
+                            if (retry > 0) {
+                                --retry;
+                                // sleep for 10 seconds.
+                                try {
+                                    Thread.sleep(10 * 1000);
+                                } catch (InterruptedException e) {
+                                    LOG.error("Interrupted during login retry after LoginException:", le);
+                                    throw le;
+                                }
+                            } else {
+                                LOG.error("Could not refresh TGT for principal: {}.", principal, le);
+                            }
+                        }
+                    }
+                } catch (LoginException le) {
+                    LOG.error("Failed to refresh TGT: refresh thread exiting now.", le);
+                    break;
+                }
+            }
+        }
+    }
+
+    public void loginAndStartThreadIfRequired() throws LoginException {
         login = login(loginContextName);
-        this.loginContextName = loginContextName;
         subject = login.getSubject();
         isKrbTicket = !subject.getPrivateCredentials(KerberosTicket.class).isEmpty();
         AppConfigurationEntry[] entries = Configuration.getConfiguration().getAppConfigurationEntry(loginContextName);

Review comment:
       @anmolnar 
   
   Yes, the refresh thread of Login object is not cleaned up leaving many login threads.
   The Jira's description is missing one key stack trace
   
   ```
   17971 Threads with the following stack trace
   
   3911 - TIMED_WAITING - Thread-822 | ORG: null RAC: None
   java.base@11.0.9/java.lang.Thread.sleep(Native Method)
   org.apache.zookeeper.Login$1.run(Login.java:197)
   java.base@11.0.9/java.lang.Thread.run(Thread.java:834)
   ```
   
   which when combined with another stack trace coming from ZookeeperSaslClient
   
   ```
   java.lang.Thread.State: RUNNABLE at java.net.PlainSocketImpl.socketConnect(java.base@11.0.4.0.101/Native Method) at java.net.AbstractPlainSocketImpl.doConnect(java.base@11.0.4.0.101/AbstractPlainSocketImpl.java:399) - locked <0x00000015004fde20> (a java.net.SocksSocketImpl) at java.net.AbstractPlainSocketImpl.connectToAddress(java.base@11.0.4.0.101/AbstractPlainSocketImpl.java:242) at java.net.AbstractPlainSocketImpl.connect(java.base@11.0.4.0.101/AbstractPlainSocketImpl.java:224) at java.net.SocksSocketImpl.connect(java.base@11.0.4.0.101/SocksSocketImpl.java:403) at java.net.Socket.connect(java.base@11.0.4.0.101/Socket.java:609) at sun.security.krb5.internal.TCPClient.<init>(java.security.jgss@11.0.4.0.101/NetClient.java:62) at sun.security.krb5.internal.NetClient.getInstance(java.security.jgss@11.0.4.0.101/NetClient.java:42) at sun.security.krb5.KdcComm$KdcCommunication.run(java.security.jgss@11.0.4.0.101/KdcComm.java:401) at sun.security.krb5.KdcComm$KdcCommunication.run(java.s
 ecurity.jgss@11.0.4.0.101/KdcComm.java:364) at java.security.AccessController.doPrivileged(java.base@11.0.4.0.101/Native Method) at sun.security.krb5.KdcComm.send(java.security.jgss@11.0.4.0.101/KdcComm.java:348) at sun.security.krb5.KdcComm.sendIfPossible(java.security.jgss@11.0.4.0.101/KdcComm.java:253) at sun.security.krb5.KdcComm.send(java.security.jgss@11.0.4.0.101/KdcComm.java:234) at sun.security.krb5.KdcComm.send(java.security.jgss@11.0.4.0.101/KdcComm.java:200) at sun.security.krb5.KrbAsReqBuilder.send(java.security.jgss@11.0.4.0.101/KrbAsReqBuilder.java:326) at sun.security.krb5.KrbAsReqBuilder.action(java.security.jgss@11.0.4.0.101/KrbAsReqBuilder.java:371) at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(jdk.security.auth@11.0.4.0.101/Krb5LoginModule.java:754) at com.sun.security.auth.module.Krb5LoginModule.login(jdk.security.auth@11.0.4.0.101/Krb5LoginModule.java:592) at javax.security.auth.login.LoginContext.invoke(java.base@11.0.4.0.101/LoginConte
 xt.java:726) at javax.security.auth.login.LoginContext$4.run(java.base@11.0.4.0.101/LoginContext.java:665) at javax.security.auth.login.LoginContext$4.run(java.base@11.0.4.0.101/LoginContext.java:663) at java.security.AccessController.doPrivileged(java.base@11.0.4.0.101/Native Method) at javax.security.auth.login.LoginContext.invokePriv(java.base@11.0.4.0.101/LoginContext.javaa:663) at javax.security.auth.login.LoginContext.login(java.base@11.0.4.0.101/LoginContext.java:574) at org.apache.zookeeper.Login.login(Login.java:304) - locked <0x000000151c477148> (a org.apache.zookeeper.Login) at org.apache.zookeeper.Login.<init>(Login.java:106) at org.apache.zookeeper.client.ZooKeeperSaslClient.createSaslClient(ZooKeeperSaslClient.java:249) - locked <0x000000151c476f68> (a org.apache.zookeeper.client.ZooKeeperSaslClient) at org.apache.zookeeper.client.ZooKeeperSaslClient.<init>(ZooKeeperSaslClient.java:141) at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:972) at 
 org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1031)
   ```
   
   Here is one possibility of Login threads may not cleared by shutdown()
   
   1. Zookeeper initialization creates & starts ClientCnxn -> spins up SendThread and starts it
   2. ClientCnxn.SendThread.run() calls ZookeeperSaslClient.<init> -> createSaslClient()
   3. Login.<init> -> Login.login() -> loginContext.login()
   4. Let's say thread1 executing a login (loginContext.login()) takes few seconds.
   5. Meantime, let's say some one called Zookeeper.close() -> ClientCnxn.close() -> ClientCnxn.disconnect()
   6. At this point, since ZooKeeperSaslClient's constructor is not yet finished, it's zooKeeperSaslClient.shutdown() is never called.
   7. Once 3 get successful, a login thread refresh is created which would become a zombie.
   
   So, the idea is 2 parts
   1. Move the actual "login" which can take time in to separate methods and make ZooKeeperSaslClient and Login constructors lightweight so that we have a reference to these at all times.
   2. Make sure either Login thread is interrupted (when shutdown called after the thread was started) OR not started (when shutdown called before the thread was started) - This was missed earlier. Updated files in PR to reflect this.
   
   Let me know your thoughts.
   
   
   




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [zookeeper] anmolnar commented on a change in pull request #1692: ZOOKEEPER-4235 Java Client SendThread does not clean up created objects during constructor of SaslClient and Login

Posted by GitBox <gi...@apache.org>.
anmolnar commented on a change in pull request #1692:
URL: https://github.com/apache/zookeeper/pull/1692#discussion_r657948742



##########
File path: zookeeper-server/src/main/java/org/apache/zookeeper/ClientCnxn.java
##########
@@ -1148,7 +1148,8 @@ private void startConnect(InetSocketAddress addr) throws IOException {
                     if (zooKeeperSaslClient != null) {
                         zooKeeperSaslClient.shutdown();
                     }
-                    zooKeeperSaslClient = new ZooKeeperSaslClient(SaslServerPrincipal.getServerPrincipal(addr, clientConfig), clientConfig);
+                    zooKeeperSaslClient = new ZooKeeperSaslClient(clientConfig);
+                    zooKeeperSaslClient.createSaslClient(SaslServerPrincipal.getServerPrincipal(addr, clientConfig));

Review comment:
       Got it.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [zookeeper] anmolnar commented on a change in pull request #1692: ZOOKEEPER-4235 Java Client SendThread does not clean up created objects during constructor of SaslClient and Login

Posted by GitBox <gi...@apache.org>.
anmolnar commented on a change in pull request #1692:
URL: https://github.com/apache/zookeeper/pull/1692#discussion_r658698146



##########
File path: zookeeper-server/src/main/java/org/apache/zookeeper/Login.java
##########
@@ -120,172 +272,36 @@ public Login(final String loginContextName, CallbackHandler callbackHandler, fin
             // if no TGT, do not bother with ticket management.
             return;
         }
-
-        // Refresh the Ticket Granting Ticket (TGT) periodically. How often to refresh is determined by the
-        // TGT's existing expiry date and the configured MIN_TIME_BEFORE_RELOGIN. For testing and development,
-        // you can decrease the interval of expiration of tickets (for example, to 3 minutes) by running :
-        //  "modprinc -maxlife 3mins <principal>" in kadmin.
-        t = new Thread(new Runnable() {
-            public void run() {
-                LOG.info("TGT refresh thread started.");
-                while (true) {  // renewal thread's main loop. if it exits from here, thread will exit.
-                    KerberosTicket tgt = getTGT();
-                    long now = Time.currentWallTime();
-                    long nextRefresh;
-                    Date nextRefreshDate;
-                    if (tgt == null) {
-                        nextRefresh = now + MIN_TIME_BEFORE_RELOGIN;
-                        nextRefreshDate = new Date(nextRefresh);
-                        LOG.warn("No TGT found: will try again at {}", nextRefreshDate);
-                    } else {
-                        nextRefresh = getRefreshTime(tgt);
-                        long expiry = tgt.getEndTime().getTime();
-                        Date expiryDate = new Date(expiry);
-                        if ((isUsingTicketCache) && (tgt.getEndTime().equals(tgt.getRenewTill()))) {
-                            LOG.error(
-                                "The TGT cannot be renewed beyond the next expiry date: {}."
-                                    + "This process will not be able to authenticate new SASL connections after that "
-                                    + "time (for example, it will not be authenticate a new connection with a Zookeeper "
-                                    + "Quorum member).  Ask your system administrator to either increase the "
-                                    + "'renew until' time by doing : 'modprinc -maxrenewlife {}' within "
-                                    + "kadmin, or instead, to generate a keytab for {}. Because the TGT's "
-                                    + "expiry cannot be further extended by refreshing, exiting refresh thread now.",
-                                expiryDate,
-                                principal,
-                                principal);
-                            return;
-                        }
-                        // determine how long to sleep from looking at ticket's expiry.
-                        // We should not allow the ticket to expire, but we should take into consideration
-                        // MIN_TIME_BEFORE_RELOGIN. Will not sleep less than MIN_TIME_BEFORE_RELOGIN, unless doing so
-                        // would cause ticket expiration.
-                        if ((nextRefresh > expiry) || ((now + MIN_TIME_BEFORE_RELOGIN) > expiry)) {
-                            // expiry is before next scheduled refresh).
-                            nextRefresh = now;
-                        } else {
-                            if (nextRefresh < (now + MIN_TIME_BEFORE_RELOGIN)) {
-                                // next scheduled refresh is sooner than (now + MIN_TIME_BEFORE_LOGIN).
-                                Date until = new Date(nextRefresh);
-                                Date newuntil = new Date(now + MIN_TIME_BEFORE_RELOGIN);
-                                LOG.warn(
-                                    "TGT refresh thread time adjusted from : {} to : {} since "
-                                        + "the former is sooner than the minimum refresh interval ("
-                                        + "{} seconds) from now.",
-                                    until,
-                                    newuntil,
-                                    (MIN_TIME_BEFORE_RELOGIN / 1000));
-                            }
-                            nextRefresh = Math.max(nextRefresh, now + MIN_TIME_BEFORE_RELOGIN);
-                        }
-                        nextRefreshDate = new Date(nextRefresh);
-                        if (nextRefresh > expiry) {
-                            LOG.error(
-                                "next refresh: {} is later than expiry {}."
-                                    + " This may indicate a clock skew problem."
-                                    + " Check that this host and the KDC's "
-                                    + "hosts' clocks are in sync. Exiting refresh thread.",
-                                nextRefreshDate,
-                                expiryDate);
-                            return;
-                        }
-                    }
-                    if (now == nextRefresh) {
-                        LOG.info("refreshing now because expiry is before next scheduled refresh time.");
-                    } else if (now < nextRefresh) {
-                        Date until = new Date(nextRefresh);
-                        LOG.info("TGT refresh sleeping until: {}", until.toString());
-                        try {
-                            Thread.sleep(nextRefresh - now);
-                        } catch (InterruptedException ie) {
-                            LOG.warn("TGT renewal thread has been interrupted and will exit.");
-                            break;
-                        }
-                    } else {
-                        LOG.error(
-                            "nextRefresh:{} is in the past: exiting refresh thread. Check"
-                                + " clock sync between this host and KDC - (KDC's clock is likely ahead of this host)."
-                                + " Manual intervention will be required for this client to successfully authenticate."
-                                + " Exiting refresh thread.",
-                            nextRefreshDate);
-                        break;
-                    }
-                    if (isUsingTicketCache) {
-                        String cmd = zkConfig.getProperty(ZKConfig.KINIT_COMMAND, KINIT_COMMAND_DEFAULT);
-                        String kinitArgs = "-R";
-                        int retry = 1;
-                        while (retry >= 0) {
-                            try {
-                                LOG.debug("running ticket cache refresh command: {} {}", cmd, kinitArgs);
-                                Shell.execCommand(cmd, kinitArgs);
-                                break;
-                            } catch (Exception e) {
-                                if (retry > 0) {
-                                    --retry;
-                                    // sleep for 10 seconds
-                                    try {
-                                        Thread.sleep(10 * 1000);
-                                    } catch (InterruptedException ie) {
-                                        LOG.error("Interrupted while renewing TGT, exiting Login thread");
-                                        return;
-                                    }
-                                } else {
-                                    LOG.warn(
-                                        "Could not renew TGT due to problem running shell command: '{} {}'."
-                                            + " Exiting refresh thread.",
-                                        cmd,
-                                        kinitArgs,
-                                        e);
-                                    return;
-                                }
-                            }
-                        }
-                    }
-                    try {
-                        int retry = 1;
-                        while (retry >= 0) {
-                            try {
-                                reLogin();
-                                break;
-                            } catch (LoginException le) {
-                                if (retry > 0) {
-                                    --retry;
-                                    // sleep for 10 seconds.
-                                    try {
-                                        Thread.sleep(10 * 1000);
-                                    } catch (InterruptedException e) {
-                                        LOG.error("Interrupted during login retry after LoginException:", le);
-                                        throw le;
-                                    }
-                                } else {
-                                    LOG.error("Could not refresh TGT for principal: {}.", principal, le);
-                                }
-                            }
-                        }
-                    } catch (LoginException le) {
-                        LOG.error("Failed to refresh TGT: refresh thread exiting now.", le);
-                        break;
-                    }
-                }
-            }
-        });
-        t.setDaemon(true);
+        startLoginThread();
     }
 
-    public void startThreadIfNeeded() {
-        // thread object 't' will be null if a refresh thread is not needed.
-        if (t != null) {
+    private void startLoginThread() {
+        // Start the thread only if not canceled before reaching here.
+        synchronized (loginThreadLock) {

Review comment:
       Got you.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [zookeeper] anmolnar commented on pull request #1692: ZOOKEEPER-4235 Java Client SendThread does not clean up created objects during constructor of SaslClient and Login

Posted by GitBox <gi...@apache.org>.
anmolnar commented on pull request #1692:
URL: https://github.com/apache/zookeeper/pull/1692#issuecomment-868440431


   @rvaleti I restarted the Jenkins build for you.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [zookeeper] rvaleti commented on pull request #1692: ZOOKEEPER-4235 Java Client SendThread does not clean up created objects during constructor of SaslClient and Login

Posted by GitBox <gi...@apache.org>.
rvaleti commented on pull request #1692:
URL: https://github.com/apache/zookeeper/pull/1692#issuecomment-878908490


   @eolivelli @anmolnar , could you please take a look at this & merge. We would like to cherrypick this in to our light fork at the earliest.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@zookeeper.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [zookeeper] rvaleti commented on a change in pull request #1692: ZOOKEEPER-4235 Java Client SendThread does not clean up created objects during constructor of SaslClient and Login

Posted by GitBox <gi...@apache.org>.
rvaleti commented on a change in pull request #1692:
URL: https://github.com/apache/zookeeper/pull/1692#discussion_r647618032



##########
File path: zookeeper-server/src/main/java/org/apache/zookeeper/Login.java
##########
@@ -95,10 +95,159 @@
      *             Thrown if authentication fails.
      */
     public Login(final String loginContextName, CallbackHandler callbackHandler, final ZKConfig zkConfig) throws LoginException {
-        this.zkConfig = zkConfig;
+        this.loginContextName = loginContextName;
         this.callbackHandler = callbackHandler;
+        this.zkConfig = zkConfig;
+    }
+
+    private class LoginRunnable implements Runnable {
+
+        @Override
+        public void run() {
+            LOG.info("TGT refresh thread started.");
+            while (true) {  // renewal thread's main loop. if it exits from here, thread will exit.
+                KerberosTicket tgt = getTGT();
+                long now = Time.currentWallTime();
+                long nextRefresh;
+                Date nextRefreshDate;
+                if (tgt == null) {
+                    nextRefresh = now + MIN_TIME_BEFORE_RELOGIN;
+                    nextRefreshDate = new Date(nextRefresh);
+                    LOG.warn("No TGT found: will try again at {}", nextRefreshDate);
+                } else {
+                    nextRefresh = getRefreshTime(tgt);
+                    long expiry = tgt.getEndTime().getTime();
+                    Date expiryDate = new Date(expiry);
+                    if ((isUsingTicketCache) && (tgt.getEndTime().equals(tgt.getRenewTill()))) {
+                        LOG.error(
+                            "The TGT cannot be renewed beyond the next expiry date: {}."
+                                + "This process will not be able to authenticate new SASL connections after that "
+                                + "time (for example, it will not be authenticate a new connection with a Zookeeper "
+                                + "Quorum member).  Ask your system administrator to either increase the "
+                                + "'renew until' time by doing : 'modprinc -maxrenewlife {}' within "
+                                + "kadmin, or instead, to generate a keytab for {}. Because the TGT's "
+                                + "expiry cannot be further extended by refreshing, exiting refresh thread now.",
+                            expiryDate,
+                            principal,
+                            principal);
+                        return;
+                    }
+                    // determine how long to sleep from looking at ticket's expiry.
+                    // We should not allow the ticket to expire, but we should take into consideration
+                    // MIN_TIME_BEFORE_RELOGIN. Will not sleep less than MIN_TIME_BEFORE_RELOGIN, unless doing so
+                    // would cause ticket expiration.
+                    if ((nextRefresh > expiry) || ((now + MIN_TIME_BEFORE_RELOGIN) > expiry)) {
+                        // expiry is before next scheduled refresh).
+                        nextRefresh = now;
+                    } else {
+                        if (nextRefresh < (now + MIN_TIME_BEFORE_RELOGIN)) {
+                            // next scheduled refresh is sooner than (now + MIN_TIME_BEFORE_LOGIN).
+                            Date until = new Date(nextRefresh);
+                            Date newuntil = new Date(now + MIN_TIME_BEFORE_RELOGIN);
+                            LOG.warn(
+                                "TGT refresh thread time adjusted from : {} to : {} since "
+                                    + "the former is sooner than the minimum refresh interval ("
+                                    + "{} seconds) from now.",
+                                until,
+                                newuntil,
+                                (MIN_TIME_BEFORE_RELOGIN / 1000));
+                        }
+                        nextRefresh = Math.max(nextRefresh, now + MIN_TIME_BEFORE_RELOGIN);
+                    }
+                    nextRefreshDate = new Date(nextRefresh);
+                    if (nextRefresh > expiry) {
+                        LOG.error(
+                            "next refresh: {} is later than expiry {}."
+                                + " This may indicate a clock skew problem."
+                                + " Check that this host and the KDC's "
+                                + "hosts' clocks are in sync. Exiting refresh thread.",
+                            nextRefreshDate,
+                            expiryDate);
+                        return;
+                    }
+                }
+                if (now == nextRefresh) {
+                    LOG.info("refreshing now because expiry is before next scheduled refresh time.");
+                } else if (now < nextRefresh) {
+                    Date until = new Date(nextRefresh);
+                    LOG.info("TGT refresh sleeping until: {}", until.toString());
+                    try {
+                        Thread.sleep(nextRefresh - now);
+                    } catch (InterruptedException ie) {
+                        LOG.warn("TGT renewal thread has been interrupted and will exit.");
+                        break;
+                    }
+                } else {
+                    LOG.error(
+                        "nextRefresh:{} is in the past: exiting refresh thread. Check"
+                            + " clock sync between this host and KDC - (KDC's clock is likely ahead of this host)."
+                            + " Manual intervention will be required for this client to successfully authenticate."
+                            + " Exiting refresh thread.",
+                        nextRefreshDate);
+                    break;
+                }
+                if (isUsingTicketCache) {
+                    String cmd = zkConfig.getProperty(ZKConfig.KINIT_COMMAND, KINIT_COMMAND_DEFAULT);
+                    String kinitArgs = "-R";
+                    int retry = 1;
+                    while (retry >= 0) {
+                        try {
+                            LOG.debug("running ticket cache refresh command: {} {}", cmd, kinitArgs);
+                            Shell.execCommand(cmd, kinitArgs);
+                            break;
+                        } catch (Exception e) {
+                            if (retry > 0) {
+                                --retry;
+                                // sleep for 10 seconds
+                                try {
+                                    Thread.sleep(10 * 1000);
+                                } catch (InterruptedException ie) {
+                                    LOG.error("Interrupted while renewing TGT, exiting Login thread");
+                                    return;
+                                }
+                            } else {
+                                LOG.warn(
+                                    "Could not renew TGT due to problem running shell command: '{} {}'."
+                                        + " Exiting refresh thread.",
+                                    cmd,
+                                    kinitArgs,
+                                    e);
+                                return;
+                            }
+                        }
+                    }
+                }
+                try {
+                    int retry = 1;
+                    while (retry >= 0) {
+                        try {
+                            reLogin();
+                            break;
+                        } catch (LoginException le) {
+                            if (retry > 0) {
+                                --retry;
+                                // sleep for 10 seconds.
+                                try {
+                                    Thread.sleep(10 * 1000);
+                                } catch (InterruptedException e) {
+                                    LOG.error("Interrupted during login retry after LoginException:", le);
+                                    throw le;
+                                }
+                            } else {
+                                LOG.error("Could not refresh TGT for principal: {}.", principal, le);
+                            }
+                        }
+                    }
+                } catch (LoginException le) {
+                    LOG.error("Failed to refresh TGT: refresh thread exiting now.", le);
+                    break;
+                }
+            }
+        }
+    }
+
+    public void loginAndStartThreadIfRequired() throws LoginException {
         login = login(loginContextName);
-        this.loginContextName = loginContextName;
         subject = login.getSubject();
         isKrbTicket = !subject.getPrivateCredentials(KerberosTicket.class).isEmpty();
         AppConfigurationEntry[] entries = Configuration.getConfiguration().getAppConfigurationEntry(loginContextName);

Review comment:
       Yes, the refresh thread of Login object is not cleaned up leaving many login threads.
   The Jira's description is missing one key stack trace
   
   ```
   17971 Threads with the following stack trace
   
   3911 - TIMED_WAITING - Thread-822 | ORG: null RAC: None
   java.base@11.0.9/java.lang.Thread.sleep(Native Method)
   org.apache.zookeeper.Login$1.run(Login.java:197)
   java.base@11.0.9/java.lang.Thread.run(Thread.java:834)
   ```
   
   which when combined with another stack trace coming from ZookeeperSaslClient
   
   ```
   java.lang.Thread.State: RUNNABLE at java.net.PlainSocketImpl.socketConnect(java.base@11.0.4.0.101/Native Method) at java.net.AbstractPlainSocketImpl.doConnect(java.base@11.0.4.0.101/AbstractPlainSocketImpl.java:399) - locked <0x00000015004fde20> (a java.net.SocksSocketImpl) at java.net.AbstractPlainSocketImpl.connectToAddress(java.base@11.0.4.0.101/AbstractPlainSocketImpl.java:242) at java.net.AbstractPlainSocketImpl.connect(java.base@11.0.4.0.101/AbstractPlainSocketImpl.java:224) at java.net.SocksSocketImpl.connect(java.base@11.0.4.0.101/SocksSocketImpl.java:403) at java.net.Socket.connect(java.base@11.0.4.0.101/Socket.java:609) at sun.security.krb5.internal.TCPClient.<init>(java.security.jgss@11.0.4.0.101/NetClient.java:62) at sun.security.krb5.internal.NetClient.getInstance(java.security.jgss@11.0.4.0.101/NetClient.java:42) at sun.security.krb5.KdcComm$KdcCommunication.run(java.security.jgss@11.0.4.0.101/KdcComm.java:401) at sun.security.krb5.KdcComm$KdcCommunication.run(java.s
 ecurity.jgss@11.0.4.0.101/KdcComm.java:364) at java.security.AccessController.doPrivileged(java.base@11.0.4.0.101/Native Method) at sun.security.krb5.KdcComm.send(java.security.jgss@11.0.4.0.101/KdcComm.java:348) at sun.security.krb5.KdcComm.sendIfPossible(java.security.jgss@11.0.4.0.101/KdcComm.java:253) at sun.security.krb5.KdcComm.send(java.security.jgss@11.0.4.0.101/KdcComm.java:234) at sun.security.krb5.KdcComm.send(java.security.jgss@11.0.4.0.101/KdcComm.java:200) at sun.security.krb5.KrbAsReqBuilder.send(java.security.jgss@11.0.4.0.101/KrbAsReqBuilder.java:326) at sun.security.krb5.KrbAsReqBuilder.action(java.security.jgss@11.0.4.0.101/KrbAsReqBuilder.java:371) at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(jdk.security.auth@11.0.4.0.101/Krb5LoginModule.java:754) at com.sun.security.auth.module.Krb5LoginModule.login(jdk.security.auth@11.0.4.0.101/Krb5LoginModule.java:592) at javax.security.auth.login.LoginContext.invoke(java.base@11.0.4.0.101/LoginConte
 xt.java:726) at javax.security.auth.login.LoginContext$4.run(java.base@11.0.4.0.101/LoginContext.java:665) at javax.security.auth.login.LoginContext$4.run(java.base@11.0.4.0.101/LoginContext.java:663) at java.security.AccessController.doPrivileged(java.base@11.0.4.0.101/Native Method) at javax.security.auth.login.LoginContext.invokePriv(java.base@11.0.4.0.101/LoginContext.javaa:663) at javax.security.auth.login.LoginContext.login(java.base@11.0.4.0.101/LoginContext.java:574) at org.apache.zookeeper.Login.login(Login.java:304) - locked <0x000000151c477148> (a org.apache.zookeeper.Login) at org.apache.zookeeper.Login.<init>(Login.java:106) at org.apache.zookeeper.client.ZooKeeperSaslClient.createSaslClient(ZooKeeperSaslClient.java:249) - locked <0x000000151c476f68> (a org.apache.zookeeper.client.ZooKeeperSaslClient) at org.apache.zookeeper.client.ZooKeeperSaslClient.<init>(ZooKeeperSaslClient.java:141) at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:972) at 
 org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1031)
   ```
   
   Here is one possibility of Login threads may not cleared by shutdown()
   
   1. Zookeeper initialization creates & starts ClientCnxn -> spins up SendThread and starts it
   2. ClientCnxn.SendThread.run() calls ZookeeperSaslClient.<init> -> createSaslClient()
   3. Login.<init> -> Login.login() -> loginContext.login()
   4. Let's say thread1 executing a login (loginContext.login()) takes few seconds.
   5. Meantime, let's say some one called Zookeeper.close() -> ClientCnxn.close() -> ClientCnxn.disconnect()
   6. At this point, since ZooKeeperSaslClient's constructor is not yet finished, it's zooKeeperSaslClient.shutdown() is never called.
   7. Once 3 get successful, a login thread refresh is created which would become a zombie.
   
   So, the idea is 2 parts
   1. Move the actual "login" which can take time in to separate methods and make ZooKeeperSaslClient and Login constructors lightweight so that we have a reference to these at all times.
   2. Make sure either Login thread is interrupted (when shutdown called after the thread was started) OR not started (when shutdown called before the thread was started)
   
   
   




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [zookeeper] rvaleti commented on a change in pull request #1692: ZOOKEEPER-4235 Java Client SendThread does not clean up created objects during constructor of SaslClient and Login

Posted by GitBox <gi...@apache.org>.
rvaleti commented on a change in pull request #1692:
URL: https://github.com/apache/zookeeper/pull/1692#discussion_r650844253



##########
File path: zookeeper-server/src/main/java/org/apache/zookeeper/Login.java
##########
@@ -95,10 +95,159 @@
      *             Thrown if authentication fails.
      */
     public Login(final String loginContextName, CallbackHandler callbackHandler, final ZKConfig zkConfig) throws LoginException {
-        this.zkConfig = zkConfig;
+        this.loginContextName = loginContextName;
         this.callbackHandler = callbackHandler;
+        this.zkConfig = zkConfig;
+    }
+
+    private class LoginRunnable implements Runnable {
+
+        @Override
+        public void run() {
+            LOG.info("TGT refresh thread started.");
+            while (true) {  // renewal thread's main loop. if it exits from here, thread will exit.
+                KerberosTicket tgt = getTGT();
+                long now = Time.currentWallTime();
+                long nextRefresh;
+                Date nextRefreshDate;
+                if (tgt == null) {
+                    nextRefresh = now + MIN_TIME_BEFORE_RELOGIN;
+                    nextRefreshDate = new Date(nextRefresh);
+                    LOG.warn("No TGT found: will try again at {}", nextRefreshDate);
+                } else {
+                    nextRefresh = getRefreshTime(tgt);
+                    long expiry = tgt.getEndTime().getTime();
+                    Date expiryDate = new Date(expiry);
+                    if ((isUsingTicketCache) && (tgt.getEndTime().equals(tgt.getRenewTill()))) {
+                        LOG.error(
+                            "The TGT cannot be renewed beyond the next expiry date: {}."
+                                + "This process will not be able to authenticate new SASL connections after that "
+                                + "time (for example, it will not be authenticate a new connection with a Zookeeper "
+                                + "Quorum member).  Ask your system administrator to either increase the "
+                                + "'renew until' time by doing : 'modprinc -maxrenewlife {}' within "
+                                + "kadmin, or instead, to generate a keytab for {}. Because the TGT's "
+                                + "expiry cannot be further extended by refreshing, exiting refresh thread now.",
+                            expiryDate,
+                            principal,
+                            principal);
+                        return;
+                    }
+                    // determine how long to sleep from looking at ticket's expiry.
+                    // We should not allow the ticket to expire, but we should take into consideration
+                    // MIN_TIME_BEFORE_RELOGIN. Will not sleep less than MIN_TIME_BEFORE_RELOGIN, unless doing so
+                    // would cause ticket expiration.
+                    if ((nextRefresh > expiry) || ((now + MIN_TIME_BEFORE_RELOGIN) > expiry)) {
+                        // expiry is before next scheduled refresh).
+                        nextRefresh = now;
+                    } else {
+                        if (nextRefresh < (now + MIN_TIME_BEFORE_RELOGIN)) {
+                            // next scheduled refresh is sooner than (now + MIN_TIME_BEFORE_LOGIN).
+                            Date until = new Date(nextRefresh);
+                            Date newuntil = new Date(now + MIN_TIME_BEFORE_RELOGIN);
+                            LOG.warn(
+                                "TGT refresh thread time adjusted from : {} to : {} since "
+                                    + "the former is sooner than the minimum refresh interval ("
+                                    + "{} seconds) from now.",
+                                until,
+                                newuntil,
+                                (MIN_TIME_BEFORE_RELOGIN / 1000));
+                        }
+                        nextRefresh = Math.max(nextRefresh, now + MIN_TIME_BEFORE_RELOGIN);
+                    }
+                    nextRefreshDate = new Date(nextRefresh);
+                    if (nextRefresh > expiry) {
+                        LOG.error(
+                            "next refresh: {} is later than expiry {}."
+                                + " This may indicate a clock skew problem."
+                                + " Check that this host and the KDC's "
+                                + "hosts' clocks are in sync. Exiting refresh thread.",
+                            nextRefreshDate,
+                            expiryDate);
+                        return;
+                    }
+                }
+                if (now == nextRefresh) {
+                    LOG.info("refreshing now because expiry is before next scheduled refresh time.");
+                } else if (now < nextRefresh) {
+                    Date until = new Date(nextRefresh);
+                    LOG.info("TGT refresh sleeping until: {}", until.toString());
+                    try {
+                        Thread.sleep(nextRefresh - now);
+                    } catch (InterruptedException ie) {
+                        LOG.warn("TGT renewal thread has been interrupted and will exit.");
+                        break;
+                    }
+                } else {
+                    LOG.error(
+                        "nextRefresh:{} is in the past: exiting refresh thread. Check"
+                            + " clock sync between this host and KDC - (KDC's clock is likely ahead of this host)."
+                            + " Manual intervention will be required for this client to successfully authenticate."
+                            + " Exiting refresh thread.",
+                        nextRefreshDate);
+                    break;
+                }
+                if (isUsingTicketCache) {
+                    String cmd = zkConfig.getProperty(ZKConfig.KINIT_COMMAND, KINIT_COMMAND_DEFAULT);
+                    String kinitArgs = "-R";
+                    int retry = 1;
+                    while (retry >= 0) {
+                        try {
+                            LOG.debug("running ticket cache refresh command: {} {}", cmd, kinitArgs);
+                            Shell.execCommand(cmd, kinitArgs);
+                            break;
+                        } catch (Exception e) {
+                            if (retry > 0) {
+                                --retry;
+                                // sleep for 10 seconds
+                                try {
+                                    Thread.sleep(10 * 1000);
+                                } catch (InterruptedException ie) {
+                                    LOG.error("Interrupted while renewing TGT, exiting Login thread");
+                                    return;
+                                }
+                            } else {
+                                LOG.warn(
+                                    "Could not renew TGT due to problem running shell command: '{} {}'."
+                                        + " Exiting refresh thread.",
+                                    cmd,
+                                    kinitArgs,
+                                    e);
+                                return;
+                            }
+                        }
+                    }
+                }
+                try {
+                    int retry = 1;
+                    while (retry >= 0) {
+                        try {
+                            reLogin();
+                            break;
+                        } catch (LoginException le) {
+                            if (retry > 0) {
+                                --retry;
+                                // sleep for 10 seconds.
+                                try {
+                                    Thread.sleep(10 * 1000);
+                                } catch (InterruptedException e) {
+                                    LOG.error("Interrupted during login retry after LoginException:", le);
+                                    throw le;
+                                }
+                            } else {
+                                LOG.error("Could not refresh TGT for principal: {}.", principal, le);
+                            }
+                        }
+                    }
+                } catch (LoginException le) {
+                    LOG.error("Failed to refresh TGT: refresh thread exiting now.", le);
+                    break;
+                }
+            }
+        }
+    }
+
+    public void loginAndStartThreadIfRequired() throws LoginException {
         login = login(loginContextName);
-        this.loginContextName = loginContextName;
         subject = login.getSubject();
         isKrbTicket = !subject.getPrivateCredentials(KerberosTicket.class).isEmpty();
         AppConfigurationEntry[] entries = Configuration.getConfiguration().getAppConfigurationEntry(loginContextName);

Review comment:
       @anmolnar , could you please review the updated files with above context?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [zookeeper] anmolnar commented on a change in pull request #1692: ZOOKEEPER-4235 Java Client SendThread does not clean up created objects during constructor of SaslClient and Login

Posted by GitBox <gi...@apache.org>.
anmolnar commented on a change in pull request #1692:
URL: https://github.com/apache/zookeeper/pull/1692#discussion_r657955797



##########
File path: zookeeper-server/src/main/java/org/apache/zookeeper/Login.java
##########
@@ -120,172 +272,36 @@ public Login(final String loginContextName, CallbackHandler callbackHandler, fin
             // if no TGT, do not bother with ticket management.
             return;
         }
-
-        // Refresh the Ticket Granting Ticket (TGT) periodically. How often to refresh is determined by the
-        // TGT's existing expiry date and the configured MIN_TIME_BEFORE_RELOGIN. For testing and development,
-        // you can decrease the interval of expiration of tickets (for example, to 3 minutes) by running :
-        //  "modprinc -maxlife 3mins <principal>" in kadmin.
-        t = new Thread(new Runnable() {
-            public void run() {
-                LOG.info("TGT refresh thread started.");
-                while (true) {  // renewal thread's main loop. if it exits from here, thread will exit.
-                    KerberosTicket tgt = getTGT();
-                    long now = Time.currentWallTime();
-                    long nextRefresh;
-                    Date nextRefreshDate;
-                    if (tgt == null) {
-                        nextRefresh = now + MIN_TIME_BEFORE_RELOGIN;
-                        nextRefreshDate = new Date(nextRefresh);
-                        LOG.warn("No TGT found: will try again at {}", nextRefreshDate);
-                    } else {
-                        nextRefresh = getRefreshTime(tgt);
-                        long expiry = tgt.getEndTime().getTime();
-                        Date expiryDate = new Date(expiry);
-                        if ((isUsingTicketCache) && (tgt.getEndTime().equals(tgt.getRenewTill()))) {
-                            LOG.error(
-                                "The TGT cannot be renewed beyond the next expiry date: {}."
-                                    + "This process will not be able to authenticate new SASL connections after that "
-                                    + "time (for example, it will not be authenticate a new connection with a Zookeeper "
-                                    + "Quorum member).  Ask your system administrator to either increase the "
-                                    + "'renew until' time by doing : 'modprinc -maxrenewlife {}' within "
-                                    + "kadmin, or instead, to generate a keytab for {}. Because the TGT's "
-                                    + "expiry cannot be further extended by refreshing, exiting refresh thread now.",
-                                expiryDate,
-                                principal,
-                                principal);
-                            return;
-                        }
-                        // determine how long to sleep from looking at ticket's expiry.
-                        // We should not allow the ticket to expire, but we should take into consideration
-                        // MIN_TIME_BEFORE_RELOGIN. Will not sleep less than MIN_TIME_BEFORE_RELOGIN, unless doing so
-                        // would cause ticket expiration.
-                        if ((nextRefresh > expiry) || ((now + MIN_TIME_BEFORE_RELOGIN) > expiry)) {
-                            // expiry is before next scheduled refresh).
-                            nextRefresh = now;
-                        } else {
-                            if (nextRefresh < (now + MIN_TIME_BEFORE_RELOGIN)) {
-                                // next scheduled refresh is sooner than (now + MIN_TIME_BEFORE_LOGIN).
-                                Date until = new Date(nextRefresh);
-                                Date newuntil = new Date(now + MIN_TIME_BEFORE_RELOGIN);
-                                LOG.warn(
-                                    "TGT refresh thread time adjusted from : {} to : {} since "
-                                        + "the former is sooner than the minimum refresh interval ("
-                                        + "{} seconds) from now.",
-                                    until,
-                                    newuntil,
-                                    (MIN_TIME_BEFORE_RELOGIN / 1000));
-                            }
-                            nextRefresh = Math.max(nextRefresh, now + MIN_TIME_BEFORE_RELOGIN);
-                        }
-                        nextRefreshDate = new Date(nextRefresh);
-                        if (nextRefresh > expiry) {
-                            LOG.error(
-                                "next refresh: {} is later than expiry {}."
-                                    + " This may indicate a clock skew problem."
-                                    + " Check that this host and the KDC's "
-                                    + "hosts' clocks are in sync. Exiting refresh thread.",
-                                nextRefreshDate,
-                                expiryDate);
-                            return;
-                        }
-                    }
-                    if (now == nextRefresh) {
-                        LOG.info("refreshing now because expiry is before next scheduled refresh time.");
-                    } else if (now < nextRefresh) {
-                        Date until = new Date(nextRefresh);
-                        LOG.info("TGT refresh sleeping until: {}", until.toString());
-                        try {
-                            Thread.sleep(nextRefresh - now);
-                        } catch (InterruptedException ie) {
-                            LOG.warn("TGT renewal thread has been interrupted and will exit.");
-                            break;
-                        }
-                    } else {
-                        LOG.error(
-                            "nextRefresh:{} is in the past: exiting refresh thread. Check"
-                                + " clock sync between this host and KDC - (KDC's clock is likely ahead of this host)."
-                                + " Manual intervention will be required for this client to successfully authenticate."
-                                + " Exiting refresh thread.",
-                            nextRefreshDate);
-                        break;
-                    }
-                    if (isUsingTicketCache) {
-                        String cmd = zkConfig.getProperty(ZKConfig.KINIT_COMMAND, KINIT_COMMAND_DEFAULT);
-                        String kinitArgs = "-R";
-                        int retry = 1;
-                        while (retry >= 0) {
-                            try {
-                                LOG.debug("running ticket cache refresh command: {} {}", cmd, kinitArgs);
-                                Shell.execCommand(cmd, kinitArgs);
-                                break;
-                            } catch (Exception e) {
-                                if (retry > 0) {
-                                    --retry;
-                                    // sleep for 10 seconds
-                                    try {
-                                        Thread.sleep(10 * 1000);
-                                    } catch (InterruptedException ie) {
-                                        LOG.error("Interrupted while renewing TGT, exiting Login thread");
-                                        return;
-                                    }
-                                } else {
-                                    LOG.warn(
-                                        "Could not renew TGT due to problem running shell command: '{} {}'."
-                                            + " Exiting refresh thread.",
-                                        cmd,
-                                        kinitArgs,
-                                        e);
-                                    return;
-                                }
-                            }
-                        }
-                    }
-                    try {
-                        int retry = 1;
-                        while (retry >= 0) {
-                            try {
-                                reLogin();
-                                break;
-                            } catch (LoginException le) {
-                                if (retry > 0) {
-                                    --retry;
-                                    // sleep for 10 seconds.
-                                    try {
-                                        Thread.sleep(10 * 1000);
-                                    } catch (InterruptedException e) {
-                                        LOG.error("Interrupted during login retry after LoginException:", le);
-                                        throw le;
-                                    }
-                                } else {
-                                    LOG.error("Could not refresh TGT for principal: {}.", principal, le);
-                                }
-                            }
-                        }
-                    } catch (LoginException le) {
-                        LOG.error("Failed to refresh TGT: refresh thread exiting now.", le);
-                        break;
-                    }
-                }
-            }
-        });
-        t.setDaemon(true);
+        startLoginThread();
     }
 
-    public void startThreadIfNeeded() {
-        // thread object 't' will be null if a refresh thread is not needed.
-        if (t != null) {
+    private void startLoginThread() {
+        // Start the thread only if not canceled before reaching here.
+        synchronized (loginThreadLock) {

Review comment:
       I think in Java you can use synchronized methods if the block encapsulates the entire method. You don't need to introduce a separate lock object and there's no performance hit either. I believe the syntax sugar uses the same locking method, but on the object itself.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [zookeeper] eolivelli commented on a change in pull request #1692: ZOOKEEPER-4235 Java Client SendThread does not clean up created objects during constructor of SaslClient and Login

Posted by GitBox <gi...@apache.org>.
eolivelli commented on a change in pull request #1692:
URL: https://github.com/apache/zookeeper/pull/1692#discussion_r652469288



##########
File path: zookeeper-server/src/main/java/org/apache/zookeeper/Login.java
##########
@@ -125,150 +274,9 @@ public Login(final String loginContextName, CallbackHandler callbackHandler, fin
         // TGT's existing expiry date and the configured MIN_TIME_BEFORE_RELOGIN. For testing and development,
         // you can decrease the interval of expiration of tickets (for example, to 3 minutes) by running :
         //  "modprinc -maxlife 3mins <principal>" in kadmin.
-        t = new Thread(new Runnable() {
-            public void run() {
-                LOG.info("TGT refresh thread started.");
-                while (true) {  // renewal thread's main loop. if it exits from here, thread will exit.
-                    KerberosTicket tgt = getTGT();
-                    long now = Time.currentWallTime();
-                    long nextRefresh;
-                    Date nextRefreshDate;
-                    if (tgt == null) {
-                        nextRefresh = now + MIN_TIME_BEFORE_RELOGIN;
-                        nextRefreshDate = new Date(nextRefresh);
-                        LOG.warn("No TGT found: will try again at {}", nextRefreshDate);
-                    } else {
-                        nextRefresh = getRefreshTime(tgt);
-                        long expiry = tgt.getEndTime().getTime();
-                        Date expiryDate = new Date(expiry);
-                        if ((isUsingTicketCache) && (tgt.getEndTime().equals(tgt.getRenewTill()))) {
-                            LOG.error(
-                                "The TGT cannot be renewed beyond the next expiry date: {}."
-                                    + "This process will not be able to authenticate new SASL connections after that "
-                                    + "time (for example, it will not be authenticate a new connection with a Zookeeper "
-                                    + "Quorum member).  Ask your system administrator to either increase the "
-                                    + "'renew until' time by doing : 'modprinc -maxrenewlife {}' within "
-                                    + "kadmin, or instead, to generate a keytab for {}. Because the TGT's "
-                                    + "expiry cannot be further extended by refreshing, exiting refresh thread now.",
-                                expiryDate,
-                                principal,
-                                principal);
-                            return;
-                        }
-                        // determine how long to sleep from looking at ticket's expiry.
-                        // We should not allow the ticket to expire, but we should take into consideration
-                        // MIN_TIME_BEFORE_RELOGIN. Will not sleep less than MIN_TIME_BEFORE_RELOGIN, unless doing so
-                        // would cause ticket expiration.
-                        if ((nextRefresh > expiry) || ((now + MIN_TIME_BEFORE_RELOGIN) > expiry)) {
-                            // expiry is before next scheduled refresh).
-                            nextRefresh = now;
-                        } else {
-                            if (nextRefresh < (now + MIN_TIME_BEFORE_RELOGIN)) {
-                                // next scheduled refresh is sooner than (now + MIN_TIME_BEFORE_LOGIN).
-                                Date until = new Date(nextRefresh);
-                                Date newuntil = new Date(now + MIN_TIME_BEFORE_RELOGIN);
-                                LOG.warn(
-                                    "TGT refresh thread time adjusted from : {} to : {} since "
-                                        + "the former is sooner than the minimum refresh interval ("
-                                        + "{} seconds) from now.",
-                                    until,
-                                    newuntil,
-                                    (MIN_TIME_BEFORE_RELOGIN / 1000));
-                            }
-                            nextRefresh = Math.max(nextRefresh, now + MIN_TIME_BEFORE_RELOGIN);
-                        }
-                        nextRefreshDate = new Date(nextRefresh);
-                        if (nextRefresh > expiry) {
-                            LOG.error(
-                                "next refresh: {} is later than expiry {}."
-                                    + " This may indicate a clock skew problem."
-                                    + " Check that this host and the KDC's "
-                                    + "hosts' clocks are in sync. Exiting refresh thread.",
-                                nextRefreshDate,
-                                expiryDate);
-                            return;
-                        }
-                    }
-                    if (now == nextRefresh) {
-                        LOG.info("refreshing now because expiry is before next scheduled refresh time.");
-                    } else if (now < nextRefresh) {
-                        Date until = new Date(nextRefresh);
-                        LOG.info("TGT refresh sleeping until: {}", until.toString());
-                        try {
-                            Thread.sleep(nextRefresh - now);
-                        } catch (InterruptedException ie) {
-                            LOG.warn("TGT renewal thread has been interrupted and will exit.");
-                            break;
-                        }
-                    } else {
-                        LOG.error(
-                            "nextRefresh:{} is in the past: exiting refresh thread. Check"
-                                + " clock sync between this host and KDC - (KDC's clock is likely ahead of this host)."
-                                + " Manual intervention will be required for this client to successfully authenticate."
-                                + " Exiting refresh thread.",
-                            nextRefreshDate);
-                        break;
-                    }
-                    if (isUsingTicketCache) {
-                        String cmd = zkConfig.getProperty(ZKConfig.KINIT_COMMAND, KINIT_COMMAND_DEFAULT);
-                        String kinitArgs = "-R";
-                        int retry = 1;
-                        while (retry >= 0) {
-                            try {
-                                LOG.debug("running ticket cache refresh command: {} {}", cmd, kinitArgs);
-                                Shell.execCommand(cmd, kinitArgs);
-                                break;
-                            } catch (Exception e) {
-                                if (retry > 0) {
-                                    --retry;
-                                    // sleep for 10 seconds
-                                    try {
-                                        Thread.sleep(10 * 1000);
-                                    } catch (InterruptedException ie) {
-                                        LOG.error("Interrupted while renewing TGT, exiting Login thread");
-                                        return;
-                                    }
-                                } else {
-                                    LOG.warn(
-                                        "Could not renew TGT due to problem running shell command: '{} {}'."
-                                            + " Exiting refresh thread.",
-                                        cmd,
-                                        kinitArgs,
-                                        e);
-                                    return;
-                                }
-                            }
-                        }
-                    }
-                    try {
-                        int retry = 1;
-                        while (retry >= 0) {
-                            try {
-                                reLogin();
-                                break;
-                            } catch (LoginException le) {
-                                if (retry > 0) {
-                                    --retry;
-                                    // sleep for 10 seconds.
-                                    try {
-                                        Thread.sleep(10 * 1000);
-                                    } catch (InterruptedException e) {
-                                        LOG.error("Interrupted during login retry after LoginException:", le);
-                                        throw le;
-                                    }
-                                } else {
-                                    LOG.error("Could not refresh TGT for principal: {}.", principal, le);
-                                }
-                            }
-                        }
-                    } catch (LoginException le) {
-                        LOG.error("Failed to refresh TGT: refresh thread exiting now.", le);
-                        break;
-                    }
-                }
-            }
-        });
+        t = new Thread(new LoginRunnable());
         t.setDaemon(true);
+        t.start();
     }
 
     public void startThreadIfNeeded() {

Review comment:
       it looks like it is no more used




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [zookeeper] muse-dev[bot] commented on a change in pull request #1692: ZOOKEEPER-4235 Java Client SendThread does not clean up created objects during constructor of SaslClient and Login

Posted by GitBox <gi...@apache.org>.
muse-dev[bot] commented on a change in pull request #1692:
URL: https://github.com/apache/zookeeper/pull/1692#discussion_r647600798



##########
File path: zookeeper-server/src/main/java/org/apache/zookeeper/Login.java
##########
@@ -125,167 +276,37 @@ public Login(final String loginContextName, CallbackHandler callbackHandler, fin
         // TGT's existing expiry date and the configured MIN_TIME_BEFORE_RELOGIN. For testing and development,
         // you can decrease the interval of expiration of tickets (for example, to 3 minutes) by running :
         //  "modprinc -maxlife 3mins <principal>" in kadmin.
-        t = new Thread(new Runnable() {
-            public void run() {
-                LOG.info("TGT refresh thread started.");
-                while (true) {  // renewal thread's main loop. if it exits from here, thread will exit.
-                    KerberosTicket tgt = getTGT();
-                    long now = Time.currentWallTime();
-                    long nextRefresh;
-                    Date nextRefreshDate;
-                    if (tgt == null) {
-                        nextRefresh = now + MIN_TIME_BEFORE_RELOGIN;
-                        nextRefreshDate = new Date(nextRefresh);
-                        LOG.warn("No TGT found: will try again at {}", nextRefreshDate);
-                    } else {
-                        nextRefresh = getRefreshTime(tgt);
-                        long expiry = tgt.getEndTime().getTime();
-                        Date expiryDate = new Date(expiry);
-                        if ((isUsingTicketCache) && (tgt.getEndTime().equals(tgt.getRenewTill()))) {
-                            LOG.error(
-                                "The TGT cannot be renewed beyond the next expiry date: {}."
-                                    + "This process will not be able to authenticate new SASL connections after that "
-                                    + "time (for example, it will not be authenticate a new connection with a Zookeeper "
-                                    + "Quorum member).  Ask your system administrator to either increase the "
-                                    + "'renew until' time by doing : 'modprinc -maxrenewlife {}' within "
-                                    + "kadmin, or instead, to generate a keytab for {}. Because the TGT's "
-                                    + "expiry cannot be further extended by refreshing, exiting refresh thread now.",
-                                expiryDate,
-                                principal,
-                                principal);
-                            return;
-                        }
-                        // determine how long to sleep from looking at ticket's expiry.
-                        // We should not allow the ticket to expire, but we should take into consideration
-                        // MIN_TIME_BEFORE_RELOGIN. Will not sleep less than MIN_TIME_BEFORE_RELOGIN, unless doing so
-                        // would cause ticket expiration.
-                        if ((nextRefresh > expiry) || ((now + MIN_TIME_BEFORE_RELOGIN) > expiry)) {
-                            // expiry is before next scheduled refresh).
-                            nextRefresh = now;
-                        } else {
-                            if (nextRefresh < (now + MIN_TIME_BEFORE_RELOGIN)) {
-                                // next scheduled refresh is sooner than (now + MIN_TIME_BEFORE_LOGIN).
-                                Date until = new Date(nextRefresh);
-                                Date newuntil = new Date(now + MIN_TIME_BEFORE_RELOGIN);
-                                LOG.warn(
-                                    "TGT refresh thread time adjusted from : {} to : {} since "
-                                        + "the former is sooner than the minimum refresh interval ("
-                                        + "{} seconds) from now.",
-                                    until,
-                                    newuntil,
-                                    (MIN_TIME_BEFORE_RELOGIN / 1000));
-                            }
-                            nextRefresh = Math.max(nextRefresh, now + MIN_TIME_BEFORE_RELOGIN);
-                        }
-                        nextRefreshDate = new Date(nextRefresh);
-                        if (nextRefresh > expiry) {
-                            LOG.error(
-                                "next refresh: {} is later than expiry {}."
-                                    + " This may indicate a clock skew problem."
-                                    + " Check that this host and the KDC's "
-                                    + "hosts' clocks are in sync. Exiting refresh thread.",
-                                nextRefreshDate,
-                                expiryDate);
-                            return;
-                        }
-                    }
-                    if (now == nextRefresh) {
-                        LOG.info("refreshing now because expiry is before next scheduled refresh time.");
-                    } else if (now < nextRefresh) {
-                        Date until = new Date(nextRefresh);
-                        LOG.info("TGT refresh sleeping until: {}", until.toString());
-                        try {
-                            Thread.sleep(nextRefresh - now);
-                        } catch (InterruptedException ie) {
-                            LOG.warn("TGT renewal thread has been interrupted and will exit.");
-                            break;
-                        }
-                    } else {
-                        LOG.error(
-                            "nextRefresh:{} is in the past: exiting refresh thread. Check"
-                                + " clock sync between this host and KDC - (KDC's clock is likely ahead of this host)."
-                                + " Manual intervention will be required for this client to successfully authenticate."
-                                + " Exiting refresh thread.",
-                            nextRefreshDate);
-                        break;
-                    }
-                    if (isUsingTicketCache) {
-                        String cmd = zkConfig.getProperty(ZKConfig.KINIT_COMMAND, KINIT_COMMAND_DEFAULT);
-                        String kinitArgs = "-R";
-                        int retry = 1;
-                        while (retry >= 0) {
-                            try {
-                                LOG.debug("running ticket cache refresh command: {} {}", cmd, kinitArgs);
-                                Shell.execCommand(cmd, kinitArgs);
-                                break;
-                            } catch (Exception e) {
-                                if (retry > 0) {
-                                    --retry;
-                                    // sleep for 10 seconds
-                                    try {
-                                        Thread.sleep(10 * 1000);
-                                    } catch (InterruptedException ie) {
-                                        LOG.error("Interrupted while renewing TGT, exiting Login thread");
-                                        return;
-                                    }
-                                } else {
-                                    LOG.warn(
-                                        "Could not renew TGT due to problem running shell command: '{} {}'."
-                                            + " Exiting refresh thread.",
-                                        cmd,
-                                        kinitArgs,
-                                        e);
-                                    return;
-                                }
-                            }
-                        }
-                    }
-                    try {
-                        int retry = 1;
-                        while (retry >= 0) {
-                            try {
-                                reLogin();
-                                break;
-                            } catch (LoginException le) {
-                                if (retry > 0) {
-                                    --retry;
-                                    // sleep for 10 seconds.
-                                    try {
-                                        Thread.sleep(10 * 1000);
-                                    } catch (InterruptedException e) {
-                                        LOG.error("Interrupted during login retry after LoginException:", le);
-                                        throw le;
-                                    }
-                                } else {
-                                    LOG.error("Could not refresh TGT for principal: {}.", principal, le);
-                                }
-                            }
-                        }
-                    } catch (LoginException le) {
-                        LOG.error("Failed to refresh TGT: refresh thread exiting now.", le);
-                        break;
-                    }
-                }
-            }
-        });
+        t = new Thread(new LoginRunnable());
         t.setDaemon(true);
+        t.start();
     }
 
     public void startThreadIfNeeded() {
-        // thread object 't' will be null if a refresh thread is not needed.
-        if (t != null) {
-            t.start();
+        if (loginThreadCancelled) {
+            return;
+        }
+        synchronized (loginThreadCancelled) {
+            if (loginThreadCancelled) {
+                return;
+            }
+            // thread object 't' will be null if a refresh thread is not needed.
+            if (t != null) {
+                t.start();
+            }
         }
     }
 
     public void shutdown() {
-        if ((t != null) && (t.isAlive())) {
-            t.interrupt();
-            try {
-                t.join();
-            } catch (InterruptedException e) {
-                LOG.warn("error while waiting for Login thread to shutdown.", e);
+        synchronized (loginThreadCancelled) {

Review comment:
       *THREAD_SAFETY_VIOLATION:*  Read/Write race. Non-private method `Login.shutdown()` reads without synchronization from `this.loginThreadCancelled`. Potentially races with write in method `Login.shutdown()`.
    Reporting because this access may occur on a background thread.
   (at-me [in a reply](https://docs.muse.dev/docs/talk-to-muse/) with `help` or `ignore`)

##########
File path: zookeeper-server/src/main/java/org/apache/zookeeper/Login.java
##########
@@ -125,167 +276,37 @@ public Login(final String loginContextName, CallbackHandler callbackHandler, fin
         // TGT's existing expiry date and the configured MIN_TIME_BEFORE_RELOGIN. For testing and development,
         // you can decrease the interval of expiration of tickets (for example, to 3 minutes) by running :
         //  "modprinc -maxlife 3mins <principal>" in kadmin.
-        t = new Thread(new Runnable() {
-            public void run() {
-                LOG.info("TGT refresh thread started.");
-                while (true) {  // renewal thread's main loop. if it exits from here, thread will exit.
-                    KerberosTicket tgt = getTGT();
-                    long now = Time.currentWallTime();
-                    long nextRefresh;
-                    Date nextRefreshDate;
-                    if (tgt == null) {
-                        nextRefresh = now + MIN_TIME_BEFORE_RELOGIN;
-                        nextRefreshDate = new Date(nextRefresh);
-                        LOG.warn("No TGT found: will try again at {}", nextRefreshDate);
-                    } else {
-                        nextRefresh = getRefreshTime(tgt);
-                        long expiry = tgt.getEndTime().getTime();
-                        Date expiryDate = new Date(expiry);
-                        if ((isUsingTicketCache) && (tgt.getEndTime().equals(tgt.getRenewTill()))) {
-                            LOG.error(
-                                "The TGT cannot be renewed beyond the next expiry date: {}."
-                                    + "This process will not be able to authenticate new SASL connections after that "
-                                    + "time (for example, it will not be authenticate a new connection with a Zookeeper "
-                                    + "Quorum member).  Ask your system administrator to either increase the "
-                                    + "'renew until' time by doing : 'modprinc -maxrenewlife {}' within "
-                                    + "kadmin, or instead, to generate a keytab for {}. Because the TGT's "
-                                    + "expiry cannot be further extended by refreshing, exiting refresh thread now.",
-                                expiryDate,
-                                principal,
-                                principal);
-                            return;
-                        }
-                        // determine how long to sleep from looking at ticket's expiry.
-                        // We should not allow the ticket to expire, but we should take into consideration
-                        // MIN_TIME_BEFORE_RELOGIN. Will not sleep less than MIN_TIME_BEFORE_RELOGIN, unless doing so
-                        // would cause ticket expiration.
-                        if ((nextRefresh > expiry) || ((now + MIN_TIME_BEFORE_RELOGIN) > expiry)) {
-                            // expiry is before next scheduled refresh).
-                            nextRefresh = now;
-                        } else {
-                            if (nextRefresh < (now + MIN_TIME_BEFORE_RELOGIN)) {
-                                // next scheduled refresh is sooner than (now + MIN_TIME_BEFORE_LOGIN).
-                                Date until = new Date(nextRefresh);
-                                Date newuntil = new Date(now + MIN_TIME_BEFORE_RELOGIN);
-                                LOG.warn(
-                                    "TGT refresh thread time adjusted from : {} to : {} since "
-                                        + "the former is sooner than the minimum refresh interval ("
-                                        + "{} seconds) from now.",
-                                    until,
-                                    newuntil,
-                                    (MIN_TIME_BEFORE_RELOGIN / 1000));
-                            }
-                            nextRefresh = Math.max(nextRefresh, now + MIN_TIME_BEFORE_RELOGIN);
-                        }
-                        nextRefreshDate = new Date(nextRefresh);
-                        if (nextRefresh > expiry) {
-                            LOG.error(
-                                "next refresh: {} is later than expiry {}."
-                                    + " This may indicate a clock skew problem."
-                                    + " Check that this host and the KDC's "
-                                    + "hosts' clocks are in sync. Exiting refresh thread.",
-                                nextRefreshDate,
-                                expiryDate);
-                            return;
-                        }
-                    }
-                    if (now == nextRefresh) {
-                        LOG.info("refreshing now because expiry is before next scheduled refresh time.");
-                    } else if (now < nextRefresh) {
-                        Date until = new Date(nextRefresh);
-                        LOG.info("TGT refresh sleeping until: {}", until.toString());
-                        try {
-                            Thread.sleep(nextRefresh - now);
-                        } catch (InterruptedException ie) {
-                            LOG.warn("TGT renewal thread has been interrupted and will exit.");
-                            break;
-                        }
-                    } else {
-                        LOG.error(
-                            "nextRefresh:{} is in the past: exiting refresh thread. Check"
-                                + " clock sync between this host and KDC - (KDC's clock is likely ahead of this host)."
-                                + " Manual intervention will be required for this client to successfully authenticate."
-                                + " Exiting refresh thread.",
-                            nextRefreshDate);
-                        break;
-                    }
-                    if (isUsingTicketCache) {
-                        String cmd = zkConfig.getProperty(ZKConfig.KINIT_COMMAND, KINIT_COMMAND_DEFAULT);
-                        String kinitArgs = "-R";
-                        int retry = 1;
-                        while (retry >= 0) {
-                            try {
-                                LOG.debug("running ticket cache refresh command: {} {}", cmd, kinitArgs);
-                                Shell.execCommand(cmd, kinitArgs);
-                                break;
-                            } catch (Exception e) {
-                                if (retry > 0) {
-                                    --retry;
-                                    // sleep for 10 seconds
-                                    try {
-                                        Thread.sleep(10 * 1000);
-                                    } catch (InterruptedException ie) {
-                                        LOG.error("Interrupted while renewing TGT, exiting Login thread");
-                                        return;
-                                    }
-                                } else {
-                                    LOG.warn(
-                                        "Could not renew TGT due to problem running shell command: '{} {}'."
-                                            + " Exiting refresh thread.",
-                                        cmd,
-                                        kinitArgs,
-                                        e);
-                                    return;
-                                }
-                            }
-                        }
-                    }
-                    try {
-                        int retry = 1;
-                        while (retry >= 0) {
-                            try {
-                                reLogin();
-                                break;
-                            } catch (LoginException le) {
-                                if (retry > 0) {
-                                    --retry;
-                                    // sleep for 10 seconds.
-                                    try {
-                                        Thread.sleep(10 * 1000);
-                                    } catch (InterruptedException e) {
-                                        LOG.error("Interrupted during login retry after LoginException:", le);
-                                        throw le;
-                                    }
-                                } else {
-                                    LOG.error("Could not refresh TGT for principal: {}.", principal, le);
-                                }
-                            }
-                        }
-                    } catch (LoginException le) {
-                        LOG.error("Failed to refresh TGT: refresh thread exiting now.", le);
-                        break;
-                    }
-                }
-            }
-        });
+        t = new Thread(new LoginRunnable());
         t.setDaemon(true);
+        t.start();
     }
 
     public void startThreadIfNeeded() {
-        // thread object 't' will be null if a refresh thread is not needed.
-        if (t != null) {
-            t.start();
+        if (loginThreadCancelled) {

Review comment:
       *THREAD_SAFETY_VIOLATION:*  Read/Write race. Non-private method `Login.startThreadIfNeeded()` reads without synchronization from `this.loginThreadCancelled`. Potentially races with write in method `Login.shutdown()`.
    Reporting because this access may occur on a background thread.
   (at-me [in a reply](https://docs.muse.dev/docs/talk-to-muse/) with `help` or `ignore`)




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [zookeeper] rvaleti commented on a change in pull request #1692: ZOOKEEPER-4235 Java Client SendThread does not clean up created objects during constructor of SaslClient and Login

Posted by GitBox <gi...@apache.org>.
rvaleti commented on a change in pull request #1692:
URL: https://github.com/apache/zookeeper/pull/1692#discussion_r652414942



##########
File path: zookeeper-server/src/main/java/org/apache/zookeeper/client/ZooKeeperSaslClient.java
##########
@@ -112,14 +112,17 @@ public String getLoginContext() {
         return null;
     }
 
-    public ZooKeeperSaslClient(final String serverPrincipal, ZKClientConfig clientConfig) throws LoginException {
+    public ZooKeeperSaslClient(ZKClientConfig clientConfig) throws LoginException {
+        this.clientConfig = clientConfig;
+    }
+
+    public void createSaslClient(final String serverPrincipal) throws LoginException {

Review comment:
       @eolivelli , Makes sense. Update the files in the review.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [zookeeper] rvaleti commented on pull request #1692: ZOOKEEPER-4235 Java Client SendThread does not clean up created objects during constructor of SaslClient and Login

Posted by GitBox <gi...@apache.org>.
rvaleti commented on pull request #1692:
URL: https://github.com/apache/zookeeper/pull/1692#issuecomment-861302047


   > I have to take a look more closely, but an alternative way would be to catch the error in the constructor and perform clean up before failing the constructor.
   
   @eolivelli , Please have a look at my other comment on the pretext of the issue. If looks good, do you mind taking a look at the review?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [zookeeper] mnpoonia commented on pull request #1692: ZOOKEEPER-4235 Java Client SendThread does not clean up created objects during constructor of SaslClient and Login

Posted by GitBox <gi...@apache.org>.
mnpoonia commented on pull request #1692:
URL: https://github.com/apache/zookeeper/pull/1692#issuecomment-906110702


   @rvaleti - Is this done. I was wondering if we can pick it in our light fork?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@zookeeper.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [zookeeper] rvaleti commented on a change in pull request #1692: ZOOKEEPER-4235 Java Client SendThread does not clean up created objects during constructor of SaslClient and Login

Posted by GitBox <gi...@apache.org>.
rvaleti commented on a change in pull request #1692:
URL: https://github.com/apache/zookeeper/pull/1692#discussion_r657968696



##########
File path: zookeeper-server/src/main/java/org/apache/zookeeper/Login.java
##########
@@ -120,172 +272,36 @@ public Login(final String loginContextName, CallbackHandler callbackHandler, fin
             // if no TGT, do not bother with ticket management.
             return;
         }
-
-        // Refresh the Ticket Granting Ticket (TGT) periodically. How often to refresh is determined by the
-        // TGT's existing expiry date and the configured MIN_TIME_BEFORE_RELOGIN. For testing and development,
-        // you can decrease the interval of expiration of tickets (for example, to 3 minutes) by running :
-        //  "modprinc -maxlife 3mins <principal>" in kadmin.
-        t = new Thread(new Runnable() {
-            public void run() {
-                LOG.info("TGT refresh thread started.");
-                while (true) {  // renewal thread's main loop. if it exits from here, thread will exit.
-                    KerberosTicket tgt = getTGT();
-                    long now = Time.currentWallTime();
-                    long nextRefresh;
-                    Date nextRefreshDate;
-                    if (tgt == null) {
-                        nextRefresh = now + MIN_TIME_BEFORE_RELOGIN;
-                        nextRefreshDate = new Date(nextRefresh);
-                        LOG.warn("No TGT found: will try again at {}", nextRefreshDate);
-                    } else {
-                        nextRefresh = getRefreshTime(tgt);
-                        long expiry = tgt.getEndTime().getTime();
-                        Date expiryDate = new Date(expiry);
-                        if ((isUsingTicketCache) && (tgt.getEndTime().equals(tgt.getRenewTill()))) {
-                            LOG.error(
-                                "The TGT cannot be renewed beyond the next expiry date: {}."
-                                    + "This process will not be able to authenticate new SASL connections after that "
-                                    + "time (for example, it will not be authenticate a new connection with a Zookeeper "
-                                    + "Quorum member).  Ask your system administrator to either increase the "
-                                    + "'renew until' time by doing : 'modprinc -maxrenewlife {}' within "
-                                    + "kadmin, or instead, to generate a keytab for {}. Because the TGT's "
-                                    + "expiry cannot be further extended by refreshing, exiting refresh thread now.",
-                                expiryDate,
-                                principal,
-                                principal);
-                            return;
-                        }
-                        // determine how long to sleep from looking at ticket's expiry.
-                        // We should not allow the ticket to expire, but we should take into consideration
-                        // MIN_TIME_BEFORE_RELOGIN. Will not sleep less than MIN_TIME_BEFORE_RELOGIN, unless doing so
-                        // would cause ticket expiration.
-                        if ((nextRefresh > expiry) || ((now + MIN_TIME_BEFORE_RELOGIN) > expiry)) {
-                            // expiry is before next scheduled refresh).
-                            nextRefresh = now;
-                        } else {
-                            if (nextRefresh < (now + MIN_TIME_BEFORE_RELOGIN)) {
-                                // next scheduled refresh is sooner than (now + MIN_TIME_BEFORE_LOGIN).
-                                Date until = new Date(nextRefresh);
-                                Date newuntil = new Date(now + MIN_TIME_BEFORE_RELOGIN);
-                                LOG.warn(
-                                    "TGT refresh thread time adjusted from : {} to : {} since "
-                                        + "the former is sooner than the minimum refresh interval ("
-                                        + "{} seconds) from now.",
-                                    until,
-                                    newuntil,
-                                    (MIN_TIME_BEFORE_RELOGIN / 1000));
-                            }
-                            nextRefresh = Math.max(nextRefresh, now + MIN_TIME_BEFORE_RELOGIN);
-                        }
-                        nextRefreshDate = new Date(nextRefresh);
-                        if (nextRefresh > expiry) {
-                            LOG.error(
-                                "next refresh: {} is later than expiry {}."
-                                    + " This may indicate a clock skew problem."
-                                    + " Check that this host and the KDC's "
-                                    + "hosts' clocks are in sync. Exiting refresh thread.",
-                                nextRefreshDate,
-                                expiryDate);
-                            return;
-                        }
-                    }
-                    if (now == nextRefresh) {
-                        LOG.info("refreshing now because expiry is before next scheduled refresh time.");
-                    } else if (now < nextRefresh) {
-                        Date until = new Date(nextRefresh);
-                        LOG.info("TGT refresh sleeping until: {}", until.toString());
-                        try {
-                            Thread.sleep(nextRefresh - now);
-                        } catch (InterruptedException ie) {
-                            LOG.warn("TGT renewal thread has been interrupted and will exit.");
-                            break;
-                        }
-                    } else {
-                        LOG.error(
-                            "nextRefresh:{} is in the past: exiting refresh thread. Check"
-                                + " clock sync between this host and KDC - (KDC's clock is likely ahead of this host)."
-                                + " Manual intervention will be required for this client to successfully authenticate."
-                                + " Exiting refresh thread.",
-                            nextRefreshDate);
-                        break;
-                    }
-                    if (isUsingTicketCache) {
-                        String cmd = zkConfig.getProperty(ZKConfig.KINIT_COMMAND, KINIT_COMMAND_DEFAULT);
-                        String kinitArgs = "-R";
-                        int retry = 1;
-                        while (retry >= 0) {
-                            try {
-                                LOG.debug("running ticket cache refresh command: {} {}", cmd, kinitArgs);
-                                Shell.execCommand(cmd, kinitArgs);
-                                break;
-                            } catch (Exception e) {
-                                if (retry > 0) {
-                                    --retry;
-                                    // sleep for 10 seconds
-                                    try {
-                                        Thread.sleep(10 * 1000);
-                                    } catch (InterruptedException ie) {
-                                        LOG.error("Interrupted while renewing TGT, exiting Login thread");
-                                        return;
-                                    }
-                                } else {
-                                    LOG.warn(
-                                        "Could not renew TGT due to problem running shell command: '{} {}'."
-                                            + " Exiting refresh thread.",
-                                        cmd,
-                                        kinitArgs,
-                                        e);
-                                    return;
-                                }
-                            }
-                        }
-                    }
-                    try {
-                        int retry = 1;
-                        while (retry >= 0) {
-                            try {
-                                reLogin();
-                                break;
-                            } catch (LoginException le) {
-                                if (retry > 0) {
-                                    --retry;
-                                    // sleep for 10 seconds.
-                                    try {
-                                        Thread.sleep(10 * 1000);
-                                    } catch (InterruptedException e) {
-                                        LOG.error("Interrupted during login retry after LoginException:", le);
-                                        throw le;
-                                    }
-                                } else {
-                                    LOG.error("Could not refresh TGT for principal: {}.", principal, le);
-                                }
-                            }
-                        }
-                    } catch (LoginException le) {
-                        LOG.error("Failed to refresh TGT: refresh thread exiting now.", le);
-                        break;
-                    }
-                }
-            }
-        });
-        t.setDaemon(true);
+        startLoginThread();
     }
 
-    public void startThreadIfNeeded() {
-        // thread object 't' will be null if a refresh thread is not needed.
-        if (t != null) {
+    private void startLoginThread() {
+        // Start the thread only if not canceled before reaching here.
+        synchronized (loginThreadLock) {

Review comment:
       @anmolnar 
   Yes, I used synchronized block initially, but realized there are already a bunch of synchronized methods in Login (for ex reLogin() which does a krb relogin). We should be able to cancel/interrupt the thread even if it's in the process of relogin. So, that's why I turned to a synchronized block.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [zookeeper] eolivelli commented on pull request #1692: ZOOKEEPER-4235 Java Client SendThread does not clean up created objects during constructor of SaslClient and Login

Posted by GitBox <gi...@apache.org>.
eolivelli commented on pull request #1692:
URL: https://github.com/apache/zookeeper/pull/1692#issuecomment-879105138


   Sorry for late reply.
   I have restarted CI.
   Committing as soon as the test pass.
   
   thank you very much for this useful contribution !


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@zookeeper.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [zookeeper] rvaleti commented on a change in pull request #1692: ZOOKEEPER-4235 Java Client SendThread does not clean up created objects during constructor of SaslClient and Login

Posted by GitBox <gi...@apache.org>.
rvaleti commented on a change in pull request #1692:
URL: https://github.com/apache/zookeeper/pull/1692#discussion_r657968696



##########
File path: zookeeper-server/src/main/java/org/apache/zookeeper/Login.java
##########
@@ -120,172 +272,36 @@ public Login(final String loginContextName, CallbackHandler callbackHandler, fin
             // if no TGT, do not bother with ticket management.
             return;
         }
-
-        // Refresh the Ticket Granting Ticket (TGT) periodically. How often to refresh is determined by the
-        // TGT's existing expiry date and the configured MIN_TIME_BEFORE_RELOGIN. For testing and development,
-        // you can decrease the interval of expiration of tickets (for example, to 3 minutes) by running :
-        //  "modprinc -maxlife 3mins <principal>" in kadmin.
-        t = new Thread(new Runnable() {
-            public void run() {
-                LOG.info("TGT refresh thread started.");
-                while (true) {  // renewal thread's main loop. if it exits from here, thread will exit.
-                    KerberosTicket tgt = getTGT();
-                    long now = Time.currentWallTime();
-                    long nextRefresh;
-                    Date nextRefreshDate;
-                    if (tgt == null) {
-                        nextRefresh = now + MIN_TIME_BEFORE_RELOGIN;
-                        nextRefreshDate = new Date(nextRefresh);
-                        LOG.warn("No TGT found: will try again at {}", nextRefreshDate);
-                    } else {
-                        nextRefresh = getRefreshTime(tgt);
-                        long expiry = tgt.getEndTime().getTime();
-                        Date expiryDate = new Date(expiry);
-                        if ((isUsingTicketCache) && (tgt.getEndTime().equals(tgt.getRenewTill()))) {
-                            LOG.error(
-                                "The TGT cannot be renewed beyond the next expiry date: {}."
-                                    + "This process will not be able to authenticate new SASL connections after that "
-                                    + "time (for example, it will not be authenticate a new connection with a Zookeeper "
-                                    + "Quorum member).  Ask your system administrator to either increase the "
-                                    + "'renew until' time by doing : 'modprinc -maxrenewlife {}' within "
-                                    + "kadmin, or instead, to generate a keytab for {}. Because the TGT's "
-                                    + "expiry cannot be further extended by refreshing, exiting refresh thread now.",
-                                expiryDate,
-                                principal,
-                                principal);
-                            return;
-                        }
-                        // determine how long to sleep from looking at ticket's expiry.
-                        // We should not allow the ticket to expire, but we should take into consideration
-                        // MIN_TIME_BEFORE_RELOGIN. Will not sleep less than MIN_TIME_BEFORE_RELOGIN, unless doing so
-                        // would cause ticket expiration.
-                        if ((nextRefresh > expiry) || ((now + MIN_TIME_BEFORE_RELOGIN) > expiry)) {
-                            // expiry is before next scheduled refresh).
-                            nextRefresh = now;
-                        } else {
-                            if (nextRefresh < (now + MIN_TIME_BEFORE_RELOGIN)) {
-                                // next scheduled refresh is sooner than (now + MIN_TIME_BEFORE_LOGIN).
-                                Date until = new Date(nextRefresh);
-                                Date newuntil = new Date(now + MIN_TIME_BEFORE_RELOGIN);
-                                LOG.warn(
-                                    "TGT refresh thread time adjusted from : {} to : {} since "
-                                        + "the former is sooner than the minimum refresh interval ("
-                                        + "{} seconds) from now.",
-                                    until,
-                                    newuntil,
-                                    (MIN_TIME_BEFORE_RELOGIN / 1000));
-                            }
-                            nextRefresh = Math.max(nextRefresh, now + MIN_TIME_BEFORE_RELOGIN);
-                        }
-                        nextRefreshDate = new Date(nextRefresh);
-                        if (nextRefresh > expiry) {
-                            LOG.error(
-                                "next refresh: {} is later than expiry {}."
-                                    + " This may indicate a clock skew problem."
-                                    + " Check that this host and the KDC's "
-                                    + "hosts' clocks are in sync. Exiting refresh thread.",
-                                nextRefreshDate,
-                                expiryDate);
-                            return;
-                        }
-                    }
-                    if (now == nextRefresh) {
-                        LOG.info("refreshing now because expiry is before next scheduled refresh time.");
-                    } else if (now < nextRefresh) {
-                        Date until = new Date(nextRefresh);
-                        LOG.info("TGT refresh sleeping until: {}", until.toString());
-                        try {
-                            Thread.sleep(nextRefresh - now);
-                        } catch (InterruptedException ie) {
-                            LOG.warn("TGT renewal thread has been interrupted and will exit.");
-                            break;
-                        }
-                    } else {
-                        LOG.error(
-                            "nextRefresh:{} is in the past: exiting refresh thread. Check"
-                                + " clock sync between this host and KDC - (KDC's clock is likely ahead of this host)."
-                                + " Manual intervention will be required for this client to successfully authenticate."
-                                + " Exiting refresh thread.",
-                            nextRefreshDate);
-                        break;
-                    }
-                    if (isUsingTicketCache) {
-                        String cmd = zkConfig.getProperty(ZKConfig.KINIT_COMMAND, KINIT_COMMAND_DEFAULT);
-                        String kinitArgs = "-R";
-                        int retry = 1;
-                        while (retry >= 0) {
-                            try {
-                                LOG.debug("running ticket cache refresh command: {} {}", cmd, kinitArgs);
-                                Shell.execCommand(cmd, kinitArgs);
-                                break;
-                            } catch (Exception e) {
-                                if (retry > 0) {
-                                    --retry;
-                                    // sleep for 10 seconds
-                                    try {
-                                        Thread.sleep(10 * 1000);
-                                    } catch (InterruptedException ie) {
-                                        LOG.error("Interrupted while renewing TGT, exiting Login thread");
-                                        return;
-                                    }
-                                } else {
-                                    LOG.warn(
-                                        "Could not renew TGT due to problem running shell command: '{} {}'."
-                                            + " Exiting refresh thread.",
-                                        cmd,
-                                        kinitArgs,
-                                        e);
-                                    return;
-                                }
-                            }
-                        }
-                    }
-                    try {
-                        int retry = 1;
-                        while (retry >= 0) {
-                            try {
-                                reLogin();
-                                break;
-                            } catch (LoginException le) {
-                                if (retry > 0) {
-                                    --retry;
-                                    // sleep for 10 seconds.
-                                    try {
-                                        Thread.sleep(10 * 1000);
-                                    } catch (InterruptedException e) {
-                                        LOG.error("Interrupted during login retry after LoginException:", le);
-                                        throw le;
-                                    }
-                                } else {
-                                    LOG.error("Could not refresh TGT for principal: {}.", principal, le);
-                                }
-                            }
-                        }
-                    } catch (LoginException le) {
-                        LOG.error("Failed to refresh TGT: refresh thread exiting now.", le);
-                        break;
-                    }
-                }
-            }
-        });
-        t.setDaemon(true);
+        startLoginThread();
     }
 
-    public void startThreadIfNeeded() {
-        // thread object 't' will be null if a refresh thread is not needed.
-        if (t != null) {
+    private void startLoginThread() {
+        // Start the thread only if not canceled before reaching here.
+        synchronized (loginThreadLock) {

Review comment:
       @anmolnar 
   Yes, I used synchronized method only initially, but realized there are already a bunch of synchronized methods in Login (for ex reLogin() which does a krb relogin). We should be able to cancel/interrupt the thread even if it's in the process of relogin. So, that's why I turned to a synchronized block.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [zookeeper] mnpoonia commented on pull request #1692: ZOOKEEPER-4235 Java Client SendThread does not clean up created objects during constructor of SaslClient and Login

Posted by GitBox <gi...@apache.org>.
mnpoonia commented on pull request #1692:
URL: https://github.com/apache/zookeeper/pull/1692#issuecomment-911143913


   @eolivelli Can you please help here. Is there something more to be done to get this PR through?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@zookeeper.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [zookeeper] eolivelli commented on a change in pull request #1692: ZOOKEEPER-4235 Java Client SendThread does not clean up created objects during constructor of SaslClient and Login

Posted by GitBox <gi...@apache.org>.
eolivelli commented on a change in pull request #1692:
URL: https://github.com/apache/zookeeper/pull/1692#discussion_r652159142



##########
File path: zookeeper-server/src/main/java/org/apache/zookeeper/client/ZooKeeperSaslClient.java
##########
@@ -112,14 +112,17 @@ public String getLoginContext() {
         return null;
     }
 
-    public ZooKeeperSaslClient(final String serverPrincipal, ZKClientConfig clientConfig) throws LoginException {
+    public ZooKeeperSaslClient(ZKClientConfig clientConfig) throws LoginException {
+        this.clientConfig = clientConfig;
+    }
+
+    public void createSaslClient(final String serverPrincipal) throws LoginException {

Review comment:
       We could keep the serverPrincipal into a field, this way we preserve the original behaviour, that this value is set during the construction.
   Otherwise this method may be called with the wrong value.
   Does it make sense to you?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [zookeeper] anmolnar commented on a change in pull request #1692: ZOOKEEPER-4235 Java Client SendThread does not clean up created objects during constructor of SaslClient and Login

Posted by GitBox <gi...@apache.org>.
anmolnar commented on a change in pull request #1692:
URL: https://github.com/apache/zookeeper/pull/1692#discussion_r643861911



##########
File path: zookeeper-server/src/main/java/org/apache/zookeeper/Login.java
##########
@@ -95,10 +95,159 @@
      *             Thrown if authentication fails.
      */
     public Login(final String loginContextName, CallbackHandler callbackHandler, final ZKConfig zkConfig) throws LoginException {
-        this.zkConfig = zkConfig;
+        this.loginContextName = loginContextName;
         this.callbackHandler = callbackHandler;
+        this.zkConfig = zkConfig;
+    }
+
+    private class LoginRunnable implements Runnable {
+
+        @Override
+        public void run() {
+            LOG.info("TGT refresh thread started.");
+            while (true) {  // renewal thread's main loop. if it exits from here, thread will exit.
+                KerberosTicket tgt = getTGT();
+                long now = Time.currentWallTime();
+                long nextRefresh;
+                Date nextRefreshDate;
+                if (tgt == null) {
+                    nextRefresh = now + MIN_TIME_BEFORE_RELOGIN;
+                    nextRefreshDate = new Date(nextRefresh);
+                    LOG.warn("No TGT found: will try again at {}", nextRefreshDate);
+                } else {
+                    nextRefresh = getRefreshTime(tgt);
+                    long expiry = tgt.getEndTime().getTime();
+                    Date expiryDate = new Date(expiry);
+                    if ((isUsingTicketCache) && (tgt.getEndTime().equals(tgt.getRenewTill()))) {
+                        LOG.error(
+                            "The TGT cannot be renewed beyond the next expiry date: {}."
+                                + "This process will not be able to authenticate new SASL connections after that "
+                                + "time (for example, it will not be authenticate a new connection with a Zookeeper "
+                                + "Quorum member).  Ask your system administrator to either increase the "
+                                + "'renew until' time by doing : 'modprinc -maxrenewlife {}' within "
+                                + "kadmin, or instead, to generate a keytab for {}. Because the TGT's "
+                                + "expiry cannot be further extended by refreshing, exiting refresh thread now.",
+                            expiryDate,
+                            principal,
+                            principal);
+                        return;
+                    }
+                    // determine how long to sleep from looking at ticket's expiry.
+                    // We should not allow the ticket to expire, but we should take into consideration
+                    // MIN_TIME_BEFORE_RELOGIN. Will not sleep less than MIN_TIME_BEFORE_RELOGIN, unless doing so
+                    // would cause ticket expiration.
+                    if ((nextRefresh > expiry) || ((now + MIN_TIME_BEFORE_RELOGIN) > expiry)) {
+                        // expiry is before next scheduled refresh).
+                        nextRefresh = now;
+                    } else {
+                        if (nextRefresh < (now + MIN_TIME_BEFORE_RELOGIN)) {
+                            // next scheduled refresh is sooner than (now + MIN_TIME_BEFORE_LOGIN).
+                            Date until = new Date(nextRefresh);
+                            Date newuntil = new Date(now + MIN_TIME_BEFORE_RELOGIN);
+                            LOG.warn(
+                                "TGT refresh thread time adjusted from : {} to : {} since "
+                                    + "the former is sooner than the minimum refresh interval ("
+                                    + "{} seconds) from now.",
+                                until,
+                                newuntil,
+                                (MIN_TIME_BEFORE_RELOGIN / 1000));
+                        }
+                        nextRefresh = Math.max(nextRefresh, now + MIN_TIME_BEFORE_RELOGIN);
+                    }
+                    nextRefreshDate = new Date(nextRefresh);
+                    if (nextRefresh > expiry) {
+                        LOG.error(
+                            "next refresh: {} is later than expiry {}."
+                                + " This may indicate a clock skew problem."
+                                + " Check that this host and the KDC's "
+                                + "hosts' clocks are in sync. Exiting refresh thread.",
+                            nextRefreshDate,
+                            expiryDate);
+                        return;
+                    }
+                }
+                if (now == nextRefresh) {
+                    LOG.info("refreshing now because expiry is before next scheduled refresh time.");
+                } else if (now < nextRefresh) {
+                    Date until = new Date(nextRefresh);
+                    LOG.info("TGT refresh sleeping until: {}", until.toString());
+                    try {
+                        Thread.sleep(nextRefresh - now);
+                    } catch (InterruptedException ie) {
+                        LOG.warn("TGT renewal thread has been interrupted and will exit.");
+                        break;
+                    }
+                } else {
+                    LOG.error(
+                        "nextRefresh:{} is in the past: exiting refresh thread. Check"
+                            + " clock sync between this host and KDC - (KDC's clock is likely ahead of this host)."
+                            + " Manual intervention will be required for this client to successfully authenticate."
+                            + " Exiting refresh thread.",
+                        nextRefreshDate);
+                    break;
+                }
+                if (isUsingTicketCache) {
+                    String cmd = zkConfig.getProperty(ZKConfig.KINIT_COMMAND, KINIT_COMMAND_DEFAULT);
+                    String kinitArgs = "-R";
+                    int retry = 1;
+                    while (retry >= 0) {
+                        try {
+                            LOG.debug("running ticket cache refresh command: {} {}", cmd, kinitArgs);
+                            Shell.execCommand(cmd, kinitArgs);
+                            break;
+                        } catch (Exception e) {
+                            if (retry > 0) {
+                                --retry;
+                                // sleep for 10 seconds
+                                try {
+                                    Thread.sleep(10 * 1000);
+                                } catch (InterruptedException ie) {
+                                    LOG.error("Interrupted while renewing TGT, exiting Login thread");
+                                    return;
+                                }
+                            } else {
+                                LOG.warn(
+                                    "Could not renew TGT due to problem running shell command: '{} {}'."
+                                        + " Exiting refresh thread.",
+                                    cmd,
+                                    kinitArgs,
+                                    e);
+                                return;
+                            }
+                        }
+                    }
+                }
+                try {
+                    int retry = 1;
+                    while (retry >= 0) {
+                        try {
+                            reLogin();
+                            break;
+                        } catch (LoginException le) {
+                            if (retry > 0) {
+                                --retry;
+                                // sleep for 10 seconds.
+                                try {
+                                    Thread.sleep(10 * 1000);
+                                } catch (InterruptedException e) {
+                                    LOG.error("Interrupted during login retry after LoginException:", le);
+                                    throw le;
+                                }
+                            } else {
+                                LOG.error("Could not refresh TGT for principal: {}.", principal, le);
+                            }
+                        }
+                    }
+                } catch (LoginException le) {
+                    LOG.error("Failed to refresh TGT: refresh thread exiting now.", le);
+                    break;
+                }
+            }
+        }
+    }
+
+    public void loginAndStartThreadIfRequired() throws LoginException {
         login = login(loginContextName);
-        this.loginContextName = loginContextName;
         subject = login.getSubject();
         isKrbTicket = !subject.getPrivateCredentials(KerberosTicket.class).isEmpty();
         AppConfigurationEntry[] entries = Configuration.getConfiguration().getAppConfigurationEntry(loginContextName);

Review comment:
       You moved this logic from the constructor to the new method `loginAndStartThreadIfRequired()` which could be beneficial if the constuction of the Login object and the method call would be separated, but I don't see that would be the case.
   
   Login class creation always goes together with startIfNeeded(), so it doesn't really matter what happens in which. I'd like to see what exactly the resource which is not cleared properly. The thread object?

##########
File path: zookeeper-server/src/main/java/org/apache/zookeeper/ClientCnxn.java
##########
@@ -1148,7 +1148,8 @@ private void startConnect(InetSocketAddress addr) throws IOException {
                     if (zooKeeperSaslClient != null) {
                         zooKeeperSaslClient.shutdown();
                     }
-                    zooKeeperSaslClient = new ZooKeeperSaslClient(SaslServerPrincipal.getServerPrincipal(addr, clientConfig), clientConfig);
+                    zooKeeperSaslClient = new ZooKeeperSaslClient(clientConfig);
+                    zooKeeperSaslClient.createSaslClient(SaslServerPrincipal.getServerPrincipal(addr, clientConfig));

Review comment:
       What's the point of this change?
   You split the logic into 2 methods which you call one after the other. I don't see how this makes the code more resource efficient.

##########
File path: zookeeper-server/src/main/java/org/apache/zookeeper/Login.java
##########
@@ -125,150 +274,9 @@ public Login(final String loginContextName, CallbackHandler callbackHandler, fin
         // TGT's existing expiry date and the configured MIN_TIME_BEFORE_RELOGIN. For testing and development,
         // you can decrease the interval of expiration of tickets (for example, to 3 minutes) by running :
         //  "modprinc -maxlife 3mins <principal>" in kadmin.
-        t = new Thread(new Runnable() {
-            public void run() {
-                LOG.info("TGT refresh thread started.");
-                while (true) {  // renewal thread's main loop. if it exits from here, thread will exit.
-                    KerberosTicket tgt = getTGT();
-                    long now = Time.currentWallTime();
-                    long nextRefresh;
-                    Date nextRefreshDate;
-                    if (tgt == null) {
-                        nextRefresh = now + MIN_TIME_BEFORE_RELOGIN;
-                        nextRefreshDate = new Date(nextRefresh);
-                        LOG.warn("No TGT found: will try again at {}", nextRefreshDate);
-                    } else {
-                        nextRefresh = getRefreshTime(tgt);
-                        long expiry = tgt.getEndTime().getTime();
-                        Date expiryDate = new Date(expiry);
-                        if ((isUsingTicketCache) && (tgt.getEndTime().equals(tgt.getRenewTill()))) {
-                            LOG.error(
-                                "The TGT cannot be renewed beyond the next expiry date: {}."
-                                    + "This process will not be able to authenticate new SASL connections after that "
-                                    + "time (for example, it will not be authenticate a new connection with a Zookeeper "
-                                    + "Quorum member).  Ask your system administrator to either increase the "
-                                    + "'renew until' time by doing : 'modprinc -maxrenewlife {}' within "
-                                    + "kadmin, or instead, to generate a keytab for {}. Because the TGT's "
-                                    + "expiry cannot be further extended by refreshing, exiting refresh thread now.",
-                                expiryDate,
-                                principal,
-                                principal);
-                            return;
-                        }
-                        // determine how long to sleep from looking at ticket's expiry.
-                        // We should not allow the ticket to expire, but we should take into consideration
-                        // MIN_TIME_BEFORE_RELOGIN. Will not sleep less than MIN_TIME_BEFORE_RELOGIN, unless doing so
-                        // would cause ticket expiration.
-                        if ((nextRefresh > expiry) || ((now + MIN_TIME_BEFORE_RELOGIN) > expiry)) {
-                            // expiry is before next scheduled refresh).
-                            nextRefresh = now;
-                        } else {
-                            if (nextRefresh < (now + MIN_TIME_BEFORE_RELOGIN)) {
-                                // next scheduled refresh is sooner than (now + MIN_TIME_BEFORE_LOGIN).
-                                Date until = new Date(nextRefresh);
-                                Date newuntil = new Date(now + MIN_TIME_BEFORE_RELOGIN);
-                                LOG.warn(
-                                    "TGT refresh thread time adjusted from : {} to : {} since "
-                                        + "the former is sooner than the minimum refresh interval ("
-                                        + "{} seconds) from now.",
-                                    until,
-                                    newuntil,
-                                    (MIN_TIME_BEFORE_RELOGIN / 1000));
-                            }
-                            nextRefresh = Math.max(nextRefresh, now + MIN_TIME_BEFORE_RELOGIN);
-                        }
-                        nextRefreshDate = new Date(nextRefresh);
-                        if (nextRefresh > expiry) {
-                            LOG.error(
-                                "next refresh: {} is later than expiry {}."
-                                    + " This may indicate a clock skew problem."
-                                    + " Check that this host and the KDC's "
-                                    + "hosts' clocks are in sync. Exiting refresh thread.",
-                                nextRefreshDate,
-                                expiryDate);
-                            return;
-                        }
-                    }
-                    if (now == nextRefresh) {
-                        LOG.info("refreshing now because expiry is before next scheduled refresh time.");
-                    } else if (now < nextRefresh) {
-                        Date until = new Date(nextRefresh);
-                        LOG.info("TGT refresh sleeping until: {}", until.toString());
-                        try {
-                            Thread.sleep(nextRefresh - now);
-                        } catch (InterruptedException ie) {
-                            LOG.warn("TGT renewal thread has been interrupted and will exit.");
-                            break;
-                        }
-                    } else {
-                        LOG.error(
-                            "nextRefresh:{} is in the past: exiting refresh thread. Check"
-                                + " clock sync between this host and KDC - (KDC's clock is likely ahead of this host)."
-                                + " Manual intervention will be required for this client to successfully authenticate."
-                                + " Exiting refresh thread.",
-                            nextRefreshDate);
-                        break;
-                    }
-                    if (isUsingTicketCache) {
-                        String cmd = zkConfig.getProperty(ZKConfig.KINIT_COMMAND, KINIT_COMMAND_DEFAULT);
-                        String kinitArgs = "-R";
-                        int retry = 1;
-                        while (retry >= 0) {
-                            try {
-                                LOG.debug("running ticket cache refresh command: {} {}", cmd, kinitArgs);
-                                Shell.execCommand(cmd, kinitArgs);
-                                break;
-                            } catch (Exception e) {
-                                if (retry > 0) {
-                                    --retry;
-                                    // sleep for 10 seconds
-                                    try {
-                                        Thread.sleep(10 * 1000);
-                                    } catch (InterruptedException ie) {
-                                        LOG.error("Interrupted while renewing TGT, exiting Login thread");
-                                        return;
-                                    }
-                                } else {
-                                    LOG.warn(
-                                        "Could not renew TGT due to problem running shell command: '{} {}'."
-                                            + " Exiting refresh thread.",
-                                        cmd,
-                                        kinitArgs,
-                                        e);
-                                    return;
-                                }
-                            }
-                        }
-                    }
-                    try {
-                        int retry = 1;
-                        while (retry >= 0) {
-                            try {
-                                reLogin();
-                                break;
-                            } catch (LoginException le) {
-                                if (retry > 0) {
-                                    --retry;
-                                    // sleep for 10 seconds.
-                                    try {
-                                        Thread.sleep(10 * 1000);
-                                    } catch (InterruptedException e) {
-                                        LOG.error("Interrupted during login retry after LoginException:", le);
-                                        throw le;
-                                    }
-                                } else {
-                                    LOG.error("Could not refresh TGT for principal: {}.", principal, le);
-                                }
-                            }
-                        }
-                    } catch (LoginException le) {
-                        LOG.error("Failed to refresh TGT: refresh thread exiting now.", le);
-                        break;
-                    }
-                }
-            }
-        });
+        t = new Thread(new LoginRunnable());
         t.setDaemon(true);
+        t.start();
     }
 
     public void startThreadIfNeeded() {

Review comment:
       Do we still need this method?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [zookeeper] dbwong commented on pull request #1692: ZOOKEEPER-4235 Java Client SendThread does not clean up created objects during constructor of SaslClient and Login

Posted by GitBox <gi...@apache.org>.
dbwong commented on pull request #1692:
URL: https://github.com/apache/zookeeper/pull/1692#issuecomment-841857467


   Overall looks okay to me.  Do you think it is worth making this code more testable via introduction of a factory pattern (or alternatively something like power mock)?  And verifying the shutdown paths now work/didn't work?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [zookeeper] rvaleti edited a comment on pull request #1692: ZOOKEEPER-4235 Java Client SendThread does not clean up created objects during constructor of SaslClient and Login

Posted by GitBox <gi...@apache.org>.
rvaleti edited a comment on pull request #1692:
URL: https://github.com/apache/zookeeper/pull/1692#issuecomment-862321829


   > Now the patch looks good.
   > can you please answer to the question about removing `startThreadIfNeeded` ?
   
   
   It is required to take care of not starting the thread if already marked for cancellation. I missed calling it earlier. I updated the files to use the method.
   
   below are the changes
   - renamed the startThreadIfNeeded to startLoginThread
   - loginAndStartThreadIfRequired now calls startLoginThread to spin up the thread **only** if not marked for cancellation already.
   
   @eolivelli can you please take a look at this?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [zookeeper] anmolnar commented on a change in pull request #1692: ZOOKEEPER-4235 Java Client SendThread does not clean up created objects during constructor of SaslClient and Login

Posted by GitBox <gi...@apache.org>.
anmolnar commented on a change in pull request #1692:
URL: https://github.com/apache/zookeeper/pull/1692#discussion_r657951371



##########
File path: zookeeper-server/src/main/java/org/apache/zookeeper/Login.java
##########
@@ -95,10 +95,159 @@
      *             Thrown if authentication fails.
      */
     public Login(final String loginContextName, CallbackHandler callbackHandler, final ZKConfig zkConfig) throws LoginException {
-        this.zkConfig = zkConfig;
+        this.loginContextName = loginContextName;
         this.callbackHandler = callbackHandler;
+        this.zkConfig = zkConfig;
+    }
+
+    private class LoginRunnable implements Runnable {
+
+        @Override
+        public void run() {
+            LOG.info("TGT refresh thread started.");
+            while (true) {  // renewal thread's main loop. if it exits from here, thread will exit.
+                KerberosTicket tgt = getTGT();
+                long now = Time.currentWallTime();
+                long nextRefresh;
+                Date nextRefreshDate;
+                if (tgt == null) {
+                    nextRefresh = now + MIN_TIME_BEFORE_RELOGIN;
+                    nextRefreshDate = new Date(nextRefresh);
+                    LOG.warn("No TGT found: will try again at {}", nextRefreshDate);
+                } else {
+                    nextRefresh = getRefreshTime(tgt);
+                    long expiry = tgt.getEndTime().getTime();
+                    Date expiryDate = new Date(expiry);
+                    if ((isUsingTicketCache) && (tgt.getEndTime().equals(tgt.getRenewTill()))) {
+                        LOG.error(
+                            "The TGT cannot be renewed beyond the next expiry date: {}."
+                                + "This process will not be able to authenticate new SASL connections after that "
+                                + "time (for example, it will not be authenticate a new connection with a Zookeeper "
+                                + "Quorum member).  Ask your system administrator to either increase the "
+                                + "'renew until' time by doing : 'modprinc -maxrenewlife {}' within "
+                                + "kadmin, or instead, to generate a keytab for {}. Because the TGT's "
+                                + "expiry cannot be further extended by refreshing, exiting refresh thread now.",
+                            expiryDate,
+                            principal,
+                            principal);
+                        return;
+                    }
+                    // determine how long to sleep from looking at ticket's expiry.
+                    // We should not allow the ticket to expire, but we should take into consideration
+                    // MIN_TIME_BEFORE_RELOGIN. Will not sleep less than MIN_TIME_BEFORE_RELOGIN, unless doing so
+                    // would cause ticket expiration.
+                    if ((nextRefresh > expiry) || ((now + MIN_TIME_BEFORE_RELOGIN) > expiry)) {
+                        // expiry is before next scheduled refresh).
+                        nextRefresh = now;
+                    } else {
+                        if (nextRefresh < (now + MIN_TIME_BEFORE_RELOGIN)) {
+                            // next scheduled refresh is sooner than (now + MIN_TIME_BEFORE_LOGIN).
+                            Date until = new Date(nextRefresh);
+                            Date newuntil = new Date(now + MIN_TIME_BEFORE_RELOGIN);
+                            LOG.warn(
+                                "TGT refresh thread time adjusted from : {} to : {} since "
+                                    + "the former is sooner than the minimum refresh interval ("
+                                    + "{} seconds) from now.",
+                                until,
+                                newuntil,
+                                (MIN_TIME_BEFORE_RELOGIN / 1000));
+                        }
+                        nextRefresh = Math.max(nextRefresh, now + MIN_TIME_BEFORE_RELOGIN);
+                    }
+                    nextRefreshDate = new Date(nextRefresh);
+                    if (nextRefresh > expiry) {
+                        LOG.error(
+                            "next refresh: {} is later than expiry {}."
+                                + " This may indicate a clock skew problem."
+                                + " Check that this host and the KDC's "
+                                + "hosts' clocks are in sync. Exiting refresh thread.",
+                            nextRefreshDate,
+                            expiryDate);
+                        return;
+                    }
+                }
+                if (now == nextRefresh) {
+                    LOG.info("refreshing now because expiry is before next scheduled refresh time.");
+                } else if (now < nextRefresh) {
+                    Date until = new Date(nextRefresh);
+                    LOG.info("TGT refresh sleeping until: {}", until.toString());
+                    try {
+                        Thread.sleep(nextRefresh - now);
+                    } catch (InterruptedException ie) {
+                        LOG.warn("TGT renewal thread has been interrupted and will exit.");
+                        break;
+                    }
+                } else {
+                    LOG.error(
+                        "nextRefresh:{} is in the past: exiting refresh thread. Check"
+                            + " clock sync between this host and KDC - (KDC's clock is likely ahead of this host)."
+                            + " Manual intervention will be required for this client to successfully authenticate."
+                            + " Exiting refresh thread.",
+                        nextRefreshDate);
+                    break;
+                }
+                if (isUsingTicketCache) {
+                    String cmd = zkConfig.getProperty(ZKConfig.KINIT_COMMAND, KINIT_COMMAND_DEFAULT);
+                    String kinitArgs = "-R";
+                    int retry = 1;
+                    while (retry >= 0) {
+                        try {
+                            LOG.debug("running ticket cache refresh command: {} {}", cmd, kinitArgs);
+                            Shell.execCommand(cmd, kinitArgs);
+                            break;
+                        } catch (Exception e) {
+                            if (retry > 0) {
+                                --retry;
+                                // sleep for 10 seconds
+                                try {
+                                    Thread.sleep(10 * 1000);
+                                } catch (InterruptedException ie) {
+                                    LOG.error("Interrupted while renewing TGT, exiting Login thread");
+                                    return;
+                                }
+                            } else {
+                                LOG.warn(
+                                    "Could not renew TGT due to problem running shell command: '{} {}'."
+                                        + " Exiting refresh thread.",
+                                    cmd,
+                                    kinitArgs,
+                                    e);
+                                return;
+                            }
+                        }
+                    }
+                }
+                try {
+                    int retry = 1;
+                    while (retry >= 0) {
+                        try {
+                            reLogin();
+                            break;
+                        } catch (LoginException le) {
+                            if (retry > 0) {
+                                --retry;
+                                // sleep for 10 seconds.
+                                try {
+                                    Thread.sleep(10 * 1000);
+                                } catch (InterruptedException e) {
+                                    LOG.error("Interrupted during login retry after LoginException:", le);
+                                    throw le;
+                                }
+                            } else {
+                                LOG.error("Could not refresh TGT for principal: {}.", principal, le);
+                            }
+                        }
+                    }
+                } catch (LoginException le) {
+                    LOG.error("Failed to refresh TGT: refresh thread exiting now.", le);
+                    break;
+                }
+            }
+        }
+    }
+
+    public void loginAndStartThreadIfRequired() throws LoginException {
         login = login(loginContextName);
-        this.loginContextName = loginContextName;
         subject = login.getSubject();
         isKrbTicket = !subject.getPrivateCredentials(KerberosTicket.class).isEmpty();
         AppConfigurationEntry[] entries = Configuration.getConfiguration().getAppConfigurationEntry(loginContextName);

Review comment:
       Doing another round of review, sorry for the delay.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org