You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@sentry.apache.org by "liujianhui (JIRA)" <ji...@apache.org> on 2017/12/19 15:19:02 UTC
[jira] [Updated] (SENTRY-2104) it take long time to request full
snapshot which lead to timeout
[ https://issues.apache.org/jira/browse/SENTRY-2104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
liujianhui updated SENTRY-2104:
-------------------------------
Description:
when latestsentry notification id is larger than the current notification id on HMS, it trigger hmsfollower to pull the full snapshot on hms, unfortunately, it takes long time. eventually lead notification request timeout
What conditions will lead to the latestSentryNotificationId larger than currentHmsNotificationId?
@Alexander Kolbasov
{code} private boolean isFullSnapshotRequired(long latestSentryNotificationId) throws Exception {
if (sentryStore.isHmsNotificationEmpty()) {
return true;
}
long currentHmsNotificationId = notificationFetcher.getCurrentNotificationId();
if (currentHmsNotificationId < latestSentryNotificationId) {
LOGGER.info("The latest notification ID on HMS is less than the latest notification ID "
+ "processed by Sentry. Need to request a full HMS snapshot.");
return true;
}
return false;
}{code}
the log of pull full snapshot
{code}
2017-12-19 07:14:14,947 INFO org.apache.sentry.service.thrift.HMSFollower: The latest notification ID on HMS is less than the latest notification ID processed by Sentry. Need to request a full HMS snapshot.
2017-12-19 07:14:15,107 INFO org.apache.sentry.service.thrift.SentryHMSClient: Request full HMS snapshot
2017-12-19 07:30:53,176 INFO org.apache.sentry.service.thrift.SentryHMSClient: Obtained full HMS snapshot
2017-12-19 07:30:53,197 INFO org.apache.sentry.service.thrift.SentryHMSClient: NotificationID, Before Snapshot: 301872, After Snapshot 303145
2017-12-19 07:30:53,197 INFO org.apache.sentry.service.thrift.SentryHMSClient: Reconciling full snapshot - applying 1273 changes
2017-12-19 07:30:53,482 INFO org.apache.sentry.service.thrift.SentryHMSClient: Successfully fetched hive full snapshot, Current NotificationID: 303145.
2017-12-19 07:30:53,486 INFO org.apache.sentry.service.thrift.HMSFollower: Sentry HMS support is ready
{code}
was:
when latestsentry notification id is larger than the current notification id on HMS, it trigger hmsfollower to pull the full snapshot on hms, unfortunately, it takes long time. eventually lead notification request timeout
What conditions will lead to the latestSentryNotificationId larger than currentHmsNotificationId?
{code} private boolean isFullSnapshotRequired(long latestSentryNotificationId) throws Exception {
if (sentryStore.isHmsNotificationEmpty()) {
return true;
}
long currentHmsNotificationId = notificationFetcher.getCurrentNotificationId();
if (currentHmsNotificationId < latestSentryNotificationId) {
LOGGER.info("The latest notification ID on HMS is less than the latest notification ID "
+ "processed by Sentry. Need to request a full HMS snapshot.");
return true;
}
return false;
}{code}
the log of pull full snapshot
{code}
2017-12-19 07:14:14,947 INFO org.apache.sentry.service.thrift.HMSFollower: The latest notification ID on HMS is less than the latest notification ID processed by Sentry. Need to request a full HMS snapshot.
2017-12-19 07:14:15,107 INFO org.apache.sentry.service.thrift.SentryHMSClient: Request full HMS snapshot
2017-12-19 07:30:53,176 INFO org.apache.sentry.service.thrift.SentryHMSClient: Obtained full HMS snapshot
2017-12-19 07:30:53,197 INFO org.apache.sentry.service.thrift.SentryHMSClient: NotificationID, Before Snapshot: 301872, After Snapshot 303145
2017-12-19 07:30:53,197 INFO org.apache.sentry.service.thrift.SentryHMSClient: Reconciling full snapshot - applying 1273 changes
2017-12-19 07:30:53,482 INFO org.apache.sentry.service.thrift.SentryHMSClient: Successfully fetched hive full snapshot, Current NotificationID: 303145.
2017-12-19 07:30:53,486 INFO org.apache.sentry.service.thrift.HMSFollower: Sentry HMS support is ready
{code}
> it take long time to request full snapshot which lead to timeout
> ----------------------------------------------------------------
>
> Key: SENTRY-2104
> URL: https://issues.apache.org/jira/browse/SENTRY-2104
> Project: Sentry
> Issue Type: Bug
> Affects Versions: 1.7.0
> Reporter: liujianhui
> Priority: Minor
>
> when latestsentry notification id is larger than the current notification id on HMS, it trigger hmsfollower to pull the full snapshot on hms, unfortunately, it takes long time. eventually lead notification request timeout
> What conditions will lead to the latestSentryNotificationId larger than currentHmsNotificationId?
> @Alexander Kolbasov
> {code} private boolean isFullSnapshotRequired(long latestSentryNotificationId) throws Exception {
> if (sentryStore.isHmsNotificationEmpty()) {
> return true;
> }
> long currentHmsNotificationId = notificationFetcher.getCurrentNotificationId();
> if (currentHmsNotificationId < latestSentryNotificationId) {
> LOGGER.info("The latest notification ID on HMS is less than the latest notification ID "
> + "processed by Sentry. Need to request a full HMS snapshot.");
> return true;
> }
> return false;
> }{code}
> the log of pull full snapshot
> {code}
> 2017-12-19 07:14:14,947 INFO org.apache.sentry.service.thrift.HMSFollower: The latest notification ID on HMS is less than the latest notification ID processed by Sentry. Need to request a full HMS snapshot.
> 2017-12-19 07:14:15,107 INFO org.apache.sentry.service.thrift.SentryHMSClient: Request full HMS snapshot
> 2017-12-19 07:30:53,176 INFO org.apache.sentry.service.thrift.SentryHMSClient: Obtained full HMS snapshot
> 2017-12-19 07:30:53,197 INFO org.apache.sentry.service.thrift.SentryHMSClient: NotificationID, Before Snapshot: 301872, After Snapshot 303145
> 2017-12-19 07:30:53,197 INFO org.apache.sentry.service.thrift.SentryHMSClient: Reconciling full snapshot - applying 1273 changes
> 2017-12-19 07:30:53,482 INFO org.apache.sentry.service.thrift.SentryHMSClient: Successfully fetched hive full snapshot, Current NotificationID: 303145.
> 2017-12-19 07:30:53,486 INFO org.apache.sentry.service.thrift.HMSFollower: Sentry HMS support is ready
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)