You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@sentry.apache.org by Vamsee Yarlagadda <va...@cloudera.com> on 2017/09/09 03:52:44 UTC
Review Request 62202: SENTRY-1939: Resetting the CounterWait during
full snapshot has to be handled across all sentry servers
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62202/
-----------------------------------------------------------
Review request for sentry, Alexander Kolbasov, Na Li, and Sergio Pena.
Repository: sentry
Description
-------
Currently as part of SENTRY-1929, we reset the CounterWait on the server that does the HMS full snapshot. Ideally this has to be done across all servers otherwise i see this error in the log of other servers.
```bash
2017-09-08 19:05:44,697 INFO org.apache.sentry.hdfs.DBUpdateForwarder: A newer full update is found with image number: 2
2017-09-08 19:05:44,894 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190
2017-09-08 19:05:45,115 INFO org.apache.sentry.hdfs.DBUpdateForwarder: A newer full update is found with image number: 2
2017-09-08 19:05:45,390 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190
2017-09-08 19:05:45,963 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190
2017-09-08 19:05:46,391 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190
2017-09-08 19:05:46,890 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190
2017-09-08 19:05:47,390 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190
2017-09-08 19:05:47,929 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190
2017-09-08 19:05:48,390 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190
2017-09-08 19:05:48,898 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190
2017-09-08 19:05:49,391 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190
2017-09-08 19:05:49,897 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190
2017-09-08 19:05:50,452 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190
2017-09-08 19:05:50,506 INFO org.apache.sentry.hdfs.HMSPathsDumper: Paths Dump created. 101104 total path strings, 579426 duplicate strings found, compacted to 1077 unique strings.
2017-09-08 19:05:50,535 INFO org.apache.sentry.hdfs.HMSPathsDumper: Paths Dump created. 101104 total path strings, 579426 duplicate strings found, compacted to 1077 unique strings.
2017-09-08 19:05:50,947 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190
```
Diffs
-----
sentry-provider/sentry-provider-db/src/main/java/org/apache/sentry/service/thrift/HMSFollower.java b600487dcf60b34f95d3eec7bf834b9a884db834
Diff: https://reviews.apache.org/r/62202/diff/1/
Testing
-------
In progress.
Thanks,
Vamsee Yarlagadda
Re: Review Request 62202: SENTRY-1939: Resetting the CounterWait
during full snapshot has to be handled across all sentry servers
Posted by Alexander Kolbasov <ak...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62202/#review185047
-----------------------------------------------------------
Fix it, then Ship it!
Ship It!
sentry-provider/sentry-provider-db/src/main/java/org/apache/sentry/service/thrift/HMSFollower.java
Lines 58 (patched)
<https://reviews.apache.org/r/62202/#comment261268>
There is no need to use atomic - HMSFollower is single-threaded.
sentry-provider/sentry-provider-db/src/main/java/org/apache/sentry/service/thrift/HMSFollower.java
Line 390 (original), 391 (patched)
<https://reviews.apache.org/r/62202/#comment261270>
Update the comment to reflect the changes.
sentry-provider/sentry-provider-db/src/main/java/org/apache/sentry/service/thrift/HMSFollower.java
Line 399 (original), 400 (patched)
<https://reviews.apache.org/r/62202/#comment261271>
it is easier to read if you just have
`if (counterWait != null) { return; }`
sentry-provider/sentry-provider-db/src/main/java/org/apache/sentry/service/thrift/HMSFollower.java
Lines 406 (patched)
<https://reviews.apache.org/r/62202/#comment261272>
Still makes sense to call counterWait.update(eventId)
- Alexander Kolbasov
On Sept. 9, 2017, 4:09 a.m., Vamsee Yarlagadda wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62202/
> -----------------------------------------------------------
>
> (Updated Sept. 9, 2017, 4:09 a.m.)
>
>
> Review request for sentry, Alexander Kolbasov, Na Li, and Sergio Pena.
>
>
> Repository: sentry
>
>
> Description
> -------
>
> Currently as part of SENTRY-1929, we reset the CounterWait on the server that does the HMS full snapshot. Ideally this has to be done across all servers otherwise i see this error in the log of other servers.
>
> ```bash
> 2017-09-08 19:05:44,697 INFO org.apache.sentry.hdfs.DBUpdateForwarder: A newer full update is found with image number: 2
> 2017-09-08 19:05:44,894 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190
> 2017-09-08 19:05:45,115 INFO org.apache.sentry.hdfs.DBUpdateForwarder: A newer full update is found with image number: 2
> 2017-09-08 19:05:45,390 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190
> 2017-09-08 19:05:45,963 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190
> 2017-09-08 19:05:46,391 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190
> 2017-09-08 19:05:46,890 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190
> 2017-09-08 19:05:47,390 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190
> 2017-09-08 19:05:47,929 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190
> 2017-09-08 19:05:48,390 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190
> 2017-09-08 19:05:48,898 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190
> 2017-09-08 19:05:49,391 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190
> 2017-09-08 19:05:49,897 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190
> 2017-09-08 19:05:50,452 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190
> 2017-09-08 19:05:50,506 INFO org.apache.sentry.hdfs.HMSPathsDumper: Paths Dump created. 101104 total path strings, 579426 duplicate strings found, compacted to 1077 unique strings.
> 2017-09-08 19:05:50,535 INFO org.apache.sentry.hdfs.HMSPathsDumper: Paths Dump created. 101104 total path strings, 579426 duplicate strings found, compacted to 1077 unique strings.
> 2017-09-08 19:05:50,947 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190
> ```
>
>
> Diffs
> -----
>
> sentry-provider/sentry-provider-db/src/main/java/org/apache/sentry/service/thrift/HMSFollower.java b600487dcf60b34f95d3eec7bf834b9a884db834
>
>
> Diff: https://reviews.apache.org/r/62202/diff/1/
>
>
> Testing
> -------
>
> Verifed it works seamlessely on real cluster.
>
> Sentry server (leader)
> ```bash
> 2017-09-08 21:04:43,112 INFO hive.metastore: Closed a connection to metastore, current connections: 8
> 2017-09-08 21:04:43,119 INFO hive.metastore: Closed a connection to metastore, current connections: 7
> 2017-09-08 21:04:43,126 INFO hive.metastore: Closed a connection to metastore, current connections: 6
> 2017-09-08 21:04:43,139 INFO hive.metastore: Closed a connection to metastore, current connections: 5
> 2017-09-08 21:04:43,143 INFO hive.metastore: Closed a connection to metastore, current connections: 4
> 2017-09-08 21:04:43,161 INFO hive.metastore: Closed a connection to metastore, current connections: 3
> 2017-09-08 21:04:43,184 INFO hive.metastore: Closed a connection to metastore, current connections: 2
> 2017-09-08 21:04:43,184 INFO org.apache.sentry.service.thrift.SentryHMSClient: Obtained full HMS snapshot
> 2017-09-08 21:04:43,187 INFO org.apache.sentry.service.thrift.SentryHMSClient: NotificationID, Before Snapshot: 0, After Snapshot 0
> 2017-09-08 21:04:43,187 INFO org.apache.sentry.service.thrift.SentryHMSClient: Successfully fetched hive full snapshot, Current NotificationID: CurrentNotificationEventId(eventId:0).
> 2017-09-08 21:04:43,188 INFO org.apache.sentry.provider.db.service.persistent.SentryStore: Purged 2 notification entries starting from 1
> 2017-09-08 21:05:08,209 WARN org.apache.sentry.service.thrift.CounterWait: resetting counter from 2 to smaller value 0
> 2017-09-08 21:05:08,209 INFO org.apache.sentry.service.thrift.HMSFollower: Sentry HMS support is ready
> ```
>
> Sentry server (other nodes)
> ```bash
> 2017-09-08 21:03:49,048 INFO org.apache.sentry.hdfs.DBUpdateForwarder: Newer delta updates are found up to sequence number: 2715
> 2017-09-08 21:03:54,698 INFO org.apache.sentry.hdfs.DBUpdateForwarder: Newer delta updates are found up to sequence number: 2715
> 2017-09-08 21:04:32,192 INFO org.apache.sentry.hdfs.DBUpdateForwarder: Newer delta updates are found up to sequence number: 2716
> 2017-09-08 21:05:08,113 WARN org.apache.sentry.service.thrift.CounterWait: resetting counter from 2 to smaller value 0
> 2017-09-08 21:05:08,310 INFO org.apache.sentry.hdfs.DBUpdateForwarder: A newer full update is found with image number: 4
> 2017-09-08 21:05:08,603 INFO org.apache.sentry.hdfs.DBUpdateForwarder: A newer full update is found with image number: 4
> 2017-09-08 21:05:14,161 INFO org.apache.sentry.hdfs.HMSPathsDumper: Paths Dump created. 101108 total path strings, 579426 duplicate strings found, compacted to 1077 unique strings.
> 2017-09-08 21:05:14,380 INFO org.apache.sentry.hdfs.HMSPathsDumper: Paths Dump created. 101108 total path strings, 579426 duplicate strings found, compacted to 1077 unique strings.
> 2017-09-08 21:05:29,480 INFO org.apache.sentry.hdfs.DBUpdateForwarder: Newer delta updates are found up to sequence number: 2717
> ```
>
>
> Thanks,
>
> Vamsee Yarlagadda
>
>
Re: Review Request 62202: SENTRY-1939: Resetting the CounterWait
during full snapshot has to be handled across all sentry servers
Posted by Vamsee Yarlagadda <va...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62202/
-----------------------------------------------------------
(Updated Sept. 9, 2017, 4:09 a.m.)
Review request for sentry, Alexander Kolbasov, Na Li, and Sergio Pena.
Changes
-------
Updated verified section.
Repository: sentry
Description
-------
Currently as part of SENTRY-1929, we reset the CounterWait on the server that does the HMS full snapshot. Ideally this has to be done across all servers otherwise i see this error in the log of other servers.
```bash
2017-09-08 19:05:44,697 INFO org.apache.sentry.hdfs.DBUpdateForwarder: A newer full update is found with image number: 2
2017-09-08 19:05:44,894 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190
2017-09-08 19:05:45,115 INFO org.apache.sentry.hdfs.DBUpdateForwarder: A newer full update is found with image number: 2
2017-09-08 19:05:45,390 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190
2017-09-08 19:05:45,963 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190
2017-09-08 19:05:46,391 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190
2017-09-08 19:05:46,890 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190
2017-09-08 19:05:47,390 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190
2017-09-08 19:05:47,929 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190
2017-09-08 19:05:48,390 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190
2017-09-08 19:05:48,898 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190
2017-09-08 19:05:49,391 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190
2017-09-08 19:05:49,897 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190
2017-09-08 19:05:50,452 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190
2017-09-08 19:05:50,506 INFO org.apache.sentry.hdfs.HMSPathsDumper: Paths Dump created. 101104 total path strings, 579426 duplicate strings found, compacted to 1077 unique strings.
2017-09-08 19:05:50,535 INFO org.apache.sentry.hdfs.HMSPathsDumper: Paths Dump created. 101104 total path strings, 579426 duplicate strings found, compacted to 1077 unique strings.
2017-09-08 19:05:50,947 ERROR org.apache.sentry.service.thrift.CounterWait: new counter value 0 is smaller then the previous one 2190
```
Diffs
-----
sentry-provider/sentry-provider-db/src/main/java/org/apache/sentry/service/thrift/HMSFollower.java b600487dcf60b34f95d3eec7bf834b9a884db834
Diff: https://reviews.apache.org/r/62202/diff/1/
Testing (updated)
-------
Verifed it works seamlessely on real cluster.
Sentry server (leader)
```bash
2017-09-08 21:04:43,112 INFO hive.metastore: Closed a connection to metastore, current connections: 8
2017-09-08 21:04:43,119 INFO hive.metastore: Closed a connection to metastore, current connections: 7
2017-09-08 21:04:43,126 INFO hive.metastore: Closed a connection to metastore, current connections: 6
2017-09-08 21:04:43,139 INFO hive.metastore: Closed a connection to metastore, current connections: 5
2017-09-08 21:04:43,143 INFO hive.metastore: Closed a connection to metastore, current connections: 4
2017-09-08 21:04:43,161 INFO hive.metastore: Closed a connection to metastore, current connections: 3
2017-09-08 21:04:43,184 INFO hive.metastore: Closed a connection to metastore, current connections: 2
2017-09-08 21:04:43,184 INFO org.apache.sentry.service.thrift.SentryHMSClient: Obtained full HMS snapshot
2017-09-08 21:04:43,187 INFO org.apache.sentry.service.thrift.SentryHMSClient: NotificationID, Before Snapshot: 0, After Snapshot 0
2017-09-08 21:04:43,187 INFO org.apache.sentry.service.thrift.SentryHMSClient: Successfully fetched hive full snapshot, Current NotificationID: CurrentNotificationEventId(eventId:0).
2017-09-08 21:04:43,188 INFO org.apache.sentry.provider.db.service.persistent.SentryStore: Purged 2 notification entries starting from 1
2017-09-08 21:05:08,209 WARN org.apache.sentry.service.thrift.CounterWait: resetting counter from 2 to smaller value 0
2017-09-08 21:05:08,209 INFO org.apache.sentry.service.thrift.HMSFollower: Sentry HMS support is ready
```
Sentry server (other nodes)
```bash
2017-09-08 21:03:49,048 INFO org.apache.sentry.hdfs.DBUpdateForwarder: Newer delta updates are found up to sequence number: 2715
2017-09-08 21:03:54,698 INFO org.apache.sentry.hdfs.DBUpdateForwarder: Newer delta updates are found up to sequence number: 2715
2017-09-08 21:04:32,192 INFO org.apache.sentry.hdfs.DBUpdateForwarder: Newer delta updates are found up to sequence number: 2716
2017-09-08 21:05:08,113 WARN org.apache.sentry.service.thrift.CounterWait: resetting counter from 2 to smaller value 0
2017-09-08 21:05:08,310 INFO org.apache.sentry.hdfs.DBUpdateForwarder: A newer full update is found with image number: 4
2017-09-08 21:05:08,603 INFO org.apache.sentry.hdfs.DBUpdateForwarder: A newer full update is found with image number: 4
2017-09-08 21:05:14,161 INFO org.apache.sentry.hdfs.HMSPathsDumper: Paths Dump created. 101108 total path strings, 579426 duplicate strings found, compacted to 1077 unique strings.
2017-09-08 21:05:14,380 INFO org.apache.sentry.hdfs.HMSPathsDumper: Paths Dump created. 101108 total path strings, 579426 duplicate strings found, compacted to 1077 unique strings.
2017-09-08 21:05:29,480 INFO org.apache.sentry.hdfs.DBUpdateForwarder: Newer delta updates are found up to sequence number: 2717
```
Thanks,
Vamsee Yarlagadda