You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2022/03/23 19:42:00 UTC
[jira] [Work logged] (HDFS-16518) Cached KeyProvider in KeyProviderCache should be closed with ShutdownHookManager
[ https://issues.apache.org/jira/browse/HDFS-16518?focusedWorklogId=746830&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-746830 ]
ASF GitHub Bot logged work on HDFS-16518:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 23/Mar/22 19:41
Start Date: 23/Mar/22 19:41
Worklog Time Spent: 10m
Work Description: ibuenros commented on a change in pull request #4100:
URL: https://github.com/apache/hadoop/pull/4100#discussion_r833658315
##########
File path: hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/KeyProviderCache.java
##########
@@ -85,6 +90,26 @@ public KeyProvider call() throws Exception {
}
}
+ public static final int SHUTDOWN_HOOK_PRIORITY = FileSystem.SHUTDOWN_HOOK_PRIORITY - 1;
+
+ private class KeyProviderCacheFinalizer implements Runnable {
+ @Override
+ public synchronized void run() {
+ invalidateCache();
+ }
+ }
+
+ /**
+ * Invalidate cache and auto close KeyProviders in the cache
+ */
+ @VisibleForTesting
+ synchronized void invalidateCache() {
+ LOG.debug("Invalidating all cached KeyProviders in ShutdownHookManager.");
Review comment:
This log is technically not correct in that we don't know the call is coming form ShutdownHookManager. Maybe just remove the last two words from the log.
##########
File path: hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/KeyProviderCache.java
##########
@@ -85,6 +90,26 @@ public KeyProvider call() throws Exception {
}
}
+ public static final int SHUTDOWN_HOOK_PRIORITY = FileSystem.SHUTDOWN_HOOK_PRIORITY - 1;
+
+ private class KeyProviderCacheFinalizer implements Runnable {
+ @Override
+ public synchronized void run() {
+ invalidateCache();
+ }
+ }
+
+ /**
+ * Invalidate cache and auto close KeyProviders in the cache
+ */
+ @VisibleForTesting
+ synchronized void invalidateCache() {
+ LOG.debug("Invalidating all cached KeyProviders in ShutdownHookManager.");
+ if (cache != null) {
+ cache.invalidateAll();
Review comment:
Maybe add a comment that this will close the providers due to the cache hook?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 746830)
Time Spent: 0.5h (was: 20m)
> Cached KeyProvider in KeyProviderCache should be closed with ShutdownHookManager
> --------------------------------------------------------------------------------
>
> Key: HDFS-16518
> URL: https://issues.apache.org/jira/browse/HDFS-16518
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
> Affects Versions: 2.10.0
> Reporter: Lei Yang
> Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> We need to make sure the underlying KeyProvider used by multiple DFSClient instances is closed at one shot during jvm shutdown. Within the shutdownhook, we invalidate the cache and make sure they are all closed. The cache has a removeListener hook which is called when cache entry is invalidated.
> {code:java}
> Class KeyProviderCache
> ...
> public KeyProviderCache(long expiryMs) {
> cache = CacheBuilder.newBuilder()
> .expireAfterAccess(expiryMs, TimeUnit.MILLISECONDS)
> .removalListener(new RemovalListener<URI, KeyProvider>() {
> @Override
> public void onRemoval(
> @Nonnull RemovalNotification<URI, KeyProvider> notification) {
> try {
> assert notification.getValue() != null;
> notification.getValue().close();
> } catch (Throwable e) {
> LOG.error(
> "Error closing KeyProvider with uri ["
> + notification.getKey() + "]", e);
> }
> }
> })
> .build();
> }{code}
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org