You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by GitBox <gi...@apache.org> on 2020/08/28 23:36:54 UTC
[GitHub] [hadoop] xiaoyuyao opened a new pull request #2259: HADOOP-17208. LoadBalanceKMSClientProvider#deleteKey should invalidat…
xiaoyuyao opened a new pull request #2259:
URL: https://github.com/apache/hadoop/pull/2259
https://issues.apache.org/jira/browse/HADOOP-17208
Proposed Fix:
Send invalidCache to all KMS provider instances upon delete to ensure the server side key cache is drained upon deletion.
Also fix a NPE when invalidCache for a key that has been deleted.
How is it tested:
Tested with RangerKMS with 3 instances.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] xiaoyuyao commented on pull request #2259: HADOOP-17208. LoadBalanceKMSClientProvider#deleteKey should invalidat…
Posted by GitBox <gi...@apache.org>.
xiaoyuyao commented on pull request #2259:
URL: https://github.com/apache/hadoop/pull/2259#issuecomment-691368970
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] xiaoyuyao commented on a change in pull request #2259: HADOOP-17208. LoadBalanceKMSClientProvider#deleteKey should invalidat…
Posted by GitBox <gi...@apache.org>.
xiaoyuyao commented on a change in pull request #2259:
URL: https://github.com/apache/hadoop/pull/2259#discussion_r489599636
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms/ValueQueue.java
##########
@@ -299,19 +299,18 @@ public E getNext(String keyName)
* @param keyName the key to drain the Queue for
*/
public void drain(String keyName) {
+ Runnable e;
+ while ((e = queue.deleteByName(keyName)) != null) {
+ executor.remove(e);
+ }
+ writeLock(keyName);
try {
- Runnable e;
- while ((e = queue.deleteByName(keyName)) != null) {
- executor.remove(e);
- }
- writeLock(keyName);
- try {
- keyQueues.get(keyName).clear();
- } finally {
- writeUnlock(keyName);
+ LinkedBlockingQueue kq = keyQueues.getIfPresent(keyName);
+ if (kq != null) {
+ kq.clear();
}
- } catch (ExecutionException ex) {
Review comment:
We don't have compatibility issue here. LoadingCache#get() may throw ExecutionException. But this ticket has change to use LoadingCache#getIfPresent() which does not throw ExecutionException. This way, we don't need to worry about either NPE from LoadingCache.get().clear() or ExecutionException.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] xiaoyuyao commented on pull request #2259: HADOOP-17208. LoadBalanceKMSClientProvider#deleteKey should invalidat…
Posted by GitBox <gi...@apache.org>.
xiaoyuyao commented on pull request #2259:
URL: https://github.com/apache/hadoop/pull/2259#issuecomment-691368970
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] xiaoyuyao commented on pull request #2259: HADOOP-17208. LoadBalanceKMSClientProvider#deleteKey should invalidat…
Posted by GitBox <gi...@apache.org>.
xiaoyuyao commented on pull request #2259:
URL: https://github.com/apache/hadoop/pull/2259#issuecomment-691368970
Thanks @Hexiaoqiao and @jiwq for the review. The LBKMSCP is tricky to unit test within MiniCluster environment. The scenario here fits better for an end-to-end test that requires multiple KMS server instances to work with JCEKS, which does not have consistency guarantee like Ranger KMS. I would suggest open a separate ticket to add end-to-end test for KMS-HA, some of this can be implemented with docker based tests under junit.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] Hexiaoqiao commented on pull request #2259: HADOOP-17208. LoadBalanceKMSClientProvider#deleteKey should invalidat…
Posted by GitBox <gi...@apache.org>.
Hexiaoqiao commented on pull request #2259:
URL: https://github.com/apache/hadoop/pull/2259#issuecomment-691388253
Thanks @xiaoyuyao for your reply, it makes sense to me. asflicense warning seems not related, trigger ci manually and wait another building.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] Hexiaoqiao commented on pull request #2259: HADOOP-17208. LoadBalanceKMSClientProvider#deleteKey should invalidat…
Posted by GitBox <gi...@apache.org>.
Hexiaoqiao commented on pull request #2259:
URL: https://github.com/apache/hadoop/pull/2259#issuecomment-691388253
Thanks @xiaoyuyao for your reply, it makes sense to me. asflicense warning seems not related, trigger ci manually and wait another building.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2259: HADOOP-17208. LoadBalanceKMSClientProvider#deleteKey should invalidat…
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2259:
URL: https://github.com/apache/hadoop/pull/2259#issuecomment-683222587
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 30s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| -1 :x: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 30m 46s | trunk passed |
| +1 :green_heart: | compile | 21m 52s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | compile | 18m 17s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | checkstyle | 0m 46s | trunk passed |
| +1 :green_heart: | mvnsite | 1m 27s | trunk passed |
| +1 :green_heart: | shadedclient | 16m 4s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 30s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 1m 26s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +0 :ok: | spotbugs | 2m 31s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 2m 28s | trunk passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 54s | the patch passed |
| +1 :green_heart: | compile | 21m 18s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javac | 21m 18s | the patch passed |
| +1 :green_heart: | compile | 18m 19s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | javac | 18m 19s | the patch passed |
| +1 :green_heart: | checkstyle | 0m 43s | the patch passed |
| +1 :green_heart: | mvnsite | 1m 23s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 14m 0s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 27s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 1m 28s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | findbugs | 2m 39s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 9m 45s | hadoop-common in the patch passed. |
| -1 :x: | asflicense | 0m 44s | The patch generated 3 ASF License warnings. |
| | | 167m 46s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2259/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2259 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux fb302841850a 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 44542863f41 |
| Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2259/1/testReport/ |
| asflicense | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2259/1/artifact/out/patch-asflicense-problems.txt |
| Max. process+thread count | 1669 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2259/1/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] Hexiaoqiao commented on a change in pull request #2259: HADOOP-17208. LoadBalanceKMSClientProvider#deleteKey should invalidat…
Posted by GitBox <gi...@apache.org>.
Hexiaoqiao commented on a change in pull request #2259:
URL: https://github.com/apache/hadoop/pull/2259#discussion_r488362922
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms/ValueQueue.java
##########
@@ -299,19 +299,18 @@ public E getNext(String keyName)
* @param keyName the key to drain the Queue for
*/
public void drain(String keyName) {
+ Runnable e;
+ while ((e = queue.deleteByName(keyName)) != null) {
+ executor.remove(e);
+ }
+ writeLock(keyName);
try {
- Runnable e;
- while ((e = queue.deleteByName(keyName)) != null) {
- executor.remove(e);
- }
- writeLock(keyName);
- try {
- keyQueues.get(keyName).clear();
- } finally {
- writeUnlock(keyName);
+ LinkedBlockingQueue kq = keyQueues.getIfPresent(keyName);
+ if (kq != null) {
+ kq.clear();
}
- } catch (ExecutionException ex) {
Review comment:
Hi @xiaoyuyao , Not sure if it is compatible to throw out this exception. Other are fair enough to me. Thanks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2259: HADOOP-17208. LoadBalanceKMSClientProvider#deleteKey should invalidat…
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2259:
URL: https://github.com/apache/hadoop/pull/2259#issuecomment-691413716
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 31s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| -1 :x: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 28m 47s | trunk passed |
| +1 :green_heart: | compile | 19m 34s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | compile | 16m 44s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | checkstyle | 0m 53s | trunk passed |
| +1 :green_heart: | mvnsite | 1m 26s | trunk passed |
| +1 :green_heart: | shadedclient | 16m 44s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 39s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 1m 36s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +0 :ok: | spotbugs | 2m 19s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 2m 16s | trunk passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 50s | the patch passed |
| +1 :green_heart: | compile | 18m 46s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javac | 18m 46s | the patch passed |
| +1 :green_heart: | compile | 16m 51s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | javac | 16m 51s | the patch passed |
| +1 :green_heart: | checkstyle | 0m 51s | the patch passed |
| +1 :green_heart: | mvnsite | 1m 27s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 14m 24s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 37s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 1m 34s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | findbugs | 2m 21s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 9m 25s | hadoop-common in the patch passed. |
| +1 :green_heart: | asflicense | 0m 56s | The patch does not generate ASF License warnings. |
| | | 159m 31s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2259/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2259 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 68b90656a559 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 958cab804ef |
| Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2259/2/testReport/ |
| Max. process+thread count | 1668 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2259/2/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2259: HADOOP-17208. LoadBalanceKMSClientProvider#deleteKey should invalidat…
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2259:
URL: https://github.com/apache/hadoop/pull/2259#issuecomment-691413716
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2259: HADOOP-17208. LoadBalanceKMSClientProvider#deleteKey should invalidat…
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2259:
URL: https://github.com/apache/hadoop/pull/2259#issuecomment-691413716
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] Hexiaoqiao commented on pull request #2259: HADOOP-17208. LoadBalanceKMSClientProvider#deleteKey should invalidat…
Posted by GitBox <gi...@apache.org>.
Hexiaoqiao commented on pull request #2259:
URL: https://github.com/apache/hadoop/pull/2259#issuecomment-691388253
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] xiaoyuyao merged pull request #2259: HADOOP-17208. LoadBalanceKMSClientProvider#deleteKey should invalidat…
Posted by GitBox <gi...@apache.org>.
xiaoyuyao merged pull request #2259:
URL: https://github.com/apache/hadoop/pull/2259
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org