[jira] [Comment Edited] (SENTRY-427) High availability for the SENTRY service

["Dapeng Sun (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SENTRY-427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14140245#comment-14140245 ] 

Dapeng Sun edited comment on SENTRY-427 at 9/19/14 9:37 AM:
------------------------------------------------------------

Hi, Prasad. Thanks very much for your comments!
{quote}sequence ID: Could you please add some details on how this global counter will be used ?{quote}
Okay. I will add a factory class for {{CommitContext}} , we can use {{getCommitContext(UUID serverUUID, long sequenceId)}} in the factory instead of {{new CommitContext(UUID serverUUID, long sequenceId)}}, the code will be like
{code:java}
  public CommitContext getCommitContext(UUID serverUUID, long sequenceId) {
    if (haEnabled) {
      try {
        return new CommitContext(serverUUID, counter.getGlobalSequenceId());
      } catch (Exception e) {
        LOGGER.error("Error in get global sequence Id, return the sequenceId in service!" ,e);
        return new CommitContext(serverUUID, sequenceId);
      }
    } else {
      return new CommitContext(serverUUID, sequenceId);
    }
  }
{code}
{{counter.getGlobalSequenceId()}} will return the sequence id. How do you think about it?
{quote}Regarding the transactions, I guess the example given could very well be applicable to a single Sentry server with multiple concurrent client (eg two different users from Hive). As you mentioned in the document, with many-to-many relationship, locking a Sentry object like role is equivalent to locking the mapping table rows for that object. I am not sure whether optimistic transactions in JDO can solve that.. The pessimistic (transaction isolation level set to serialized) would solve that but could be a huge performance penalty.
Another option could be to implement a rudimentary lock manager via ZooKeeper to lock logical sentry objects like roles and privileges. This is what Hive does to handle table locking to support concurrent DDL/DML and queries.{quote}
Yes, I’m agree with you that a single Sentry server with multiple concurrent client may also cause the problem, and the ZooKeeper is also a good choice, but I think optimistic transaction will minimal impact performance, I had tested the optimistic lock in local environment, relevant unit tests will be add to the patch.
{quote}Audit Log support
IMO it should be okay to have separate audit logs for each Sentry service. The downstream consumer of the audit log should be able to extract it from multiple sources. We are already introducing ZK as requirement, to add Flume/Kafka and HDFS dependency on top of that would make things too hard to setup and deploy.{quote}
Yes, I totally agree with you. Very appreciate your comments!


was (Author: dapengsun):
Hi, Prasad. Thanks very much for your comments!
{quote}sequence ID: Could you please add some details on how this global counter will be used ?{quote}
Okay. I will add a factory class for {{CommitContext}} , we can {{getCommitContext(UUID serverUUID, long sequenceId)}} in the factory instead of {{new CommitContext(UUID serverUUID, long sequenceId)}}, the code will be like
{code:java}
  public CommitContext getCommitContext(UUID serverUUID, long sequenceId) {
    if (haEnabled) {
      try {
        return new CommitContext(serverUUID, counter.getGlobalSequenceId());
      } catch (Exception e) {
        LOGGER.error("Error in get global sequence Id, return the sequenceId in service!" ,e);
        return new CommitContext(serverUUID, sequenceId);
      }
    } else {
      return new CommitContext(serverUUID, sequenceId);
    }
  }
{code}
{{counter.getGlobalSequenceId()}} will return the sequence id. How do you think about it?
{quote}Regarding the transactions, I guess the example given could very well be applicable to a single Sentry server with multiple concurrent client (eg two different users from Hive). As you mentioned in the document, with many-to-many relationship, locking a Sentry object like role is equivalent to locking the mapping table rows for that object. I am not sure whether optimistic transactions in JDO can solve that.. The pessimistic (transaction isolation level set to serialized) would solve that but could be a huge performance penalty.
Another option could be to implement a rudimentary lock manager via ZooKeeper to lock logical sentry objects like roles and privileges. This is what Hive does to handle table locking to support concurrent DDL/DML and queries.{quote}
Yes, I’m agree with you that a single Sentry server with multiple concurrent client may also cause the problem, and the ZooKeeper is also a good choice, but I think optimistic transaction will minimal impact performance, I had tested the optimistic lock in local environment, relevant unit tests will be add to the patch.
{quote}Audit Log support
IMO it should be okay to have separate audit logs for each Sentry service. The downstream consumer of the audit log should be able to extract it from multiple sources. We are already introducing ZK as requirement, to add Flume/Kafka and HDFS dependency on top of that would make things too hard to setup and deploy.{quote}
Yes, I totally agree with you. Very appreciate your comments!

> High availability for the SENTRY service
> ----------------------------------------
>
>                 Key: SENTRY-427
>                 URL: https://issues.apache.org/jira/browse/SENTRY-427
>             Project: Sentry
>          Issue Type: New Feature
>            Reporter: Dapeng Sun
>            Assignee: Dapeng Sun
>         Attachments: SENTRY_HA_Design.pdf
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)