[jira] [Commented] (SENTRY-2109) Fix the logic of identifying HMS out of Sync

["kalyan kumar kalvagadda (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SENTRY-2109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16301652#comment-16301652 ] 

kalyan kumar kalvagadda commented on SENTRY-2109:
-------------------------------------------------

With this below logic HMSFollower excepts that event-id for the first notification that is fetched should be greater than the last notification processed by 1. This may not be the case.
{noformat}
  public boolean areNotificationsOutOfSync(Collection<NotificationEvent> events,
      long latestProcessedId) {
    if (events.isEmpty()) {
      return false;
    }

    /*
     * If the sequence of notifications has a gap, then an out-of-sync might
     * have happened due to the following issue:
     *
     * - HDFS sync was disabled or Sentry was shutdown for a time period longer than
     * the HMS notification clean-up thread causing old notifications to be deleted.
     *
     * HMS notifications may contain both gaps in the sequence and duplicates
     * (the same ID repeated more then once for different events).
     *
     * To accept duplicates (see NotificationFetcher for more info), then a gap is found
     * if the 1st notification received is higher than the current ID processed + 1.
     * i.e.
     *   1st ID = 3, latest ID = 3 (duplicate found but no gap detected)
     *   1st ID = 4, latest ID = 3 (consecutive ID found but no gap detected)
     *   1st ID = 5, latest ID = 3 (a gap is detected)
     */

    List<NotificationEvent> eventList = (List<NotificationEvent>) events;
    long firstNotificationId = eventList.get(0).getEventId();

    if (firstNotificationId > (latestProcessedId + 1)) {
      LOGGER.info("Current HMS notifications are out-of-sync with latest Sentry processed"
          + "notifications. Need to request a full HMS snapshot.");
      return true;
    }

    return false;
  }
{noformat}

I have attached the snapshot for notification_log table for reference.

> Fix the logic of identifying HMS out of Sync
> --------------------------------------------
>
>                 Key: SENTRY-2109
>                 URL: https://issues.apache.org/jira/browse/SENTRY-2109
>             Project: Sentry
>          Issue Type: Bug
>          Components: Sentry
>    Affects Versions: 2.1.0
>            Reporter: kalyan kumar kalvagadda
>            Assignee: kalyan kumar kalvagadda
>             Fix For: 2.1.0
>
>         Attachments: Screenshot_HMS_NOTIFICATION_LOG.png
>
>
> Currently HMSFollower proactively checks if sentry is out of sync with HMS and initiates full snapshot, if needed.
> There will be false positives with the current logic if there are gaps in the event-id in the notification log sequence.
> This jira is aimed at making that logic robust.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)