[jira] [Created] (SENTRY-1813) LeaderStatusMonitor could get into limbo state upon ZK connection loss

["Vamsee Yarlagadda (JIRA)" <ji...@apache.org>]

Vamsee Yarlagadda created SENTRY-1813:
-----------------------------------------

             Summary: LeaderStatusMonitor could get into limbo state upon ZK connection loss
                 Key: SENTRY-1813
                 URL: https://issues.apache.org/jira/browse/SENTRY-1813
             Project: Sentry
          Issue Type: Bug
    Affects Versions: sentry-ha-redesign
            Reporter: Vamsee Yarlagadda
            Priority: Critical
             Fix For: sentry-ha-redesign


I noticed that during failover testing, if there was a connection loss with ZK to the sentry servers, the one who is currently the leader gets into a limbo state as it interrupts the Curator-LeaderSelector thread which no longer gets revived in the running Sentry process (unless the process is restarted).

Relevant code under LeaderStatusMonitor
http://github.mtv.cloudera.com/CDH/sentry/blob/cdh5-1.5.1/sentry-provider/sentry-provider-db/src/main/java/org/apache/sentry/service/thrift/LeaderStatusMonitor.java#L243-L246
{code}
   try {
      isLeader = true;
      // Wait until we are interrupted or receive a signal
      cond.await();
    } catch (InterruptedException ignored) {
      Thread.currentThread().interrupt();
      LOG.info("LeaderStatusMonitor: interrupted");
    } finally {
      isLeader = false;
      lock.unlock();
      LOG.info("LeaderStatusMonitor: becoming standby");
    }
{code}

I realized even upon the loss of ZK connection, curator framework raises an Interrupted Exception in LeaderStausMonitor which attempts to call interrupt on Thread.currentThread which is essentially *Curator-LeaderSelector* thread.
<SCREENSHOT_ATTACHED>

So if the LeaderSelector thread is interrupted, this particular Sentry server loses the capability of participating in a leader election in the future. And if this happens to all the sentry servers in the cluster, any further loss could get into a limbo state.

And during this state, Sentry no longer reads events from HMS and thereby users can no longer be able to issue DDL statements like CREATE etc. However GRANT, REVOKE still work as they don't go through HMSFollower.
  



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)