You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Bozhen Liu (Jira)" <ji...@apache.org> on 2020/05/14 19:53:00 UTC
[jira] [Updated] (HBASE-24374) Potential Race in class
org.apache.hadoop.hbase.io.crypto.Encryption
[ https://issues.apache.org/jira/browse/HBASE-24374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bozhen Liu updated HBASE-24374:
-------------------------------
Description:
We run our static race detector on IntegrationTestRegionReplicaReplication, and discovered that multiple threads (type of WorkerThread) can invoke the method getKeyProvider() in class Encryption and read/write to keyProviderCache (type of ConcurrentHashMap) concurrently without proper lock protection. The related code is attached below:
{code:java}
public final class Encryption {
...
static final Map<Pair<String,String>,KeyProvider> keyProviderCache = new ConcurrentHashMap<>();
...
public static KeyProvider getKeyProvider(Configuration conf) {
String providerClassName = conf.get(HConstants.CRYPTO_KEYPROVIDER_CONF_KEY,
KeyStoreKeyProvider.class.getName());
String providerParameters = conf.get(HConstants.CRYPTO_KEYPROVIDER_PARAMETERS_KEY, "");
try {
Pair<String,String> providerCacheKey = new Pair<>(providerClassName,
providerParameters);
KeyProvider provider = keyProviderCache.get(providerCacheKey);
if (provider != null) {
return provider;
}
provider = (KeyProvider) ReflectionUtils.newInstance(
getClassLoaderForClass(KeyProvider.class).loadClass(providerClassName),
conf);
provider.init(providerParameters);
if (LOG.isDebugEnabled()) {
LOG.debug("Installed " + providerClassName + " into key provider cache");
}
keyProviderCache.put(providerCacheKey, provider);
return provider;
} catch (Exception e) {
throw new RuntimeException(e);
}
}
}
{code}
It is possible that multiple threads do the null check on provider, all of which return false, and each thread instantiated an instance of KeyProvider. However, only one provider will be added to the map and the other one is leaking.
The possible traces of the race are shown below, where 3863 and 3864 are threads created by a loop:
=> Race: org/apache/hadoop/hbase/io/crypto/Encryption.java/util/Map
(org/apache/hadoop/hbase/io/crypto/Encryption:553 (TID: 3863) , org/apache/hadoop/hbase/io/crypto/Encryption:542 (TID: 3864) )
Trace 1st node: (TID: 3863)
-> Call org/apache/hadoop/util/ToolRunner.run from org/apache/hadoop/hbase/IntegrationTestRegionReplicaReplication.main (line 224)
-> Call org/apache/hadoop/hbase/util/AbstractHBaseTool.run from org/apache/hadoop/util/ToolRunner.run (line 76)
-> Call org/apache/hadoop/hbase/util/LoadTestTool.doWork from org/apache/hadoop/hbase/util/AbstractHBaseTool.run (line 154)
-> Call org/apache/hadoop/hbase/util/LoadTestTool.parallelLoadTables from org/apache/hadoop/hbase/util/LoadTestTool.doWork (line 589)
*> Thread (3863) created by Application,Lorg/apache/hadoop/hbase/util/LoadTestTool.parallelLoadTables (line 880)
-> Call org/apache/hadoop/hbase/util/LoadTestTool$WorkerThread.run from java/lang/Thread.start (line -1)
-> Call org/apache/hadoop/util/ToolRunner.run from org/apache/hadoop/hbase/util/LoadTestTool$WorkerThread.run (line 931)
-> Call org/apache/hadoop/hbase/util/AbstractHBaseTool.run from org/apache/hadoop/util/ToolRunner.run (line 76)
-> Call org/apache/hadoop/hbase/util/LoadTestTool.doWork from org/apache/hadoop/hbase/util/AbstractHBaseTool.run (line 154)
-> Call org/apache/hadoop/hbase/util/LoadTestTool.loadTable from org/apache/hadoop/hbase/util/LoadTestTool.doWork (line 591)
-> Call org/apache/hadoop/hbase/util/LoadTestTool.initTestTable from org/apache/hadoop/hbase/util/LoadTestTool.loadTable (line 605)
-> Call org/apache/hadoop/hbase/util/LoadTestTool.applyColumnFamilyOptions from org/apache/hadoop/hbase/util/LoadTestTool.initTestTable (line 580)
-> Call org/apache/hadoop/hbase/security/EncryptionUtil.wrapKey from org/apache/hadoop/hbase/util/LoadTestTool.applyColumnFamilyOptions (line 300)
-> Call org/apache/hadoop/hbase/io/crypto/Encryption.encryptWithSubjectKey from org/apache/hadoop/hbase/security/EncryptionUtil.wrapKey (line 107)
-> Call org/apache/hadoop/hbase/io/crypto/Encryption.getSecretKeyForSubject from org/apache/hadoop/hbase/io/crypto/Encryption.encryptWithSubjectKey (line 448)
-> Call org/apache/hadoop/hbase/io/crypto/Encryption.getKeyProvider from org/apache/hadoop/hbase/io/crypto/Encryption.getSecretKeyForSubject (line 424)
=> Write to org/apache/hadoop/hbase/io/crypto/Encryption.java/util/Map(keyProviderCache) in org/apache/hadoop/hbase/io/crypto/Encryption.getKeyProvider (line 553)
Trace 2nd node: (TID: 3864)
-> Call org/apache/hadoop/util/ToolRunner.run from org/apache/hadoop/hbase/IntegrationTestRegionReplicaReplication.main (line 224)
-> Call org/apache/hadoop/hbase/util/AbstractHBaseTool.run from org/apache/hadoop/util/ToolRunner.run (line 76)
-> Call org/apache/hadoop/hbase/util/LoadTestTool.doWork from org/apache/hadoop/hbase/util/AbstractHBaseTool.run (line 154)
-> Call org/apache/hadoop/hbase/util/LoadTestTool.parallelLoadTables from org/apache/hadoop/hbase/util/LoadTestTool.doWork (line 589)
*> Thread (3864) created by Application,Lorg/apache/hadoop/hbase/util/LoadTestTool.parallelLoadTables (line 880)
-> Call org/apache/hadoop/hbase/util/LoadTestTool$WorkerThread.run from java/lang/Thread.start (line -1)
-> Call org/apache/hadoop/util/ToolRunner.run from org/apache/hadoop/hbase/util/LoadTestTool$WorkerThread.run (line 931)
-> Call org/apache/hadoop/hbase/util/AbstractHBaseTool.run from org/apache/hadoop/util/ToolRunner.run (line 76)
-> Call org/apache/hadoop/hbase/util/LoadTestTool.doWork from org/apache/hadoop/hbase/util/AbstractHBaseTool.run (line 154)
-> Call org/apache/hadoop/hbase/util/LoadTestTool.loadTable from org/apache/hadoop/hbase/util/LoadTestTool.doWork (line 591)
-> Call org/apache/hadoop/hbase/util/LoadTestTool.initTestTable from org/apache/hadoop/hbase/util/LoadTestTool.loadTable (line 605)
-> Call org/apache/hadoop/hbase/util/LoadTestTool.applyColumnFamilyOptions from org/apache/hadoop/hbase/util/LoadTestTool.initTestTable (line 580)
-> Call org/apache/hadoop/hbase/security/EncryptionUtil.wrapKey from org/apache/hadoop/hbase/util/LoadTestTool.applyColumnFamilyOptions (line 300)
-> Call org/apache/hadoop/hbase/io/crypto/Encryption.encryptWithSubjectKey from org/apache/hadoop/hbase/security/EncryptionUtil.wrapKey (line 107)
-> Call org/apache/hadoop/hbase/io/crypto/Encryption.getSecretKeyForSubject from org/apache/hadoop/hbase/io/crypto/Encryption.encryptWithSubjectKey (line 448)
-> Call org/apache/hadoop/hbase/io/crypto/Encryption.getKeyProvider from org/apache/hadoop/hbase/io/crypto/Encryption.getSecretKeyForSubject (line 424)
=> Write to org/apache/hadoop/hbase/io/crypto/Encryption.java/util/Map(keyProviderCache) in org/apache/hadoop/hbase/io/crypto/Encryption.getKeyProvider (line 542)
> Potential Race in class org.apache.hadoop.hbase.io.crypto.Encryption
> --------------------------------------------------------------------
>
> Key: HBASE-24374
> URL: https://issues.apache.org/jira/browse/HBASE-24374
> Project: HBase
> Issue Type: Bug
> Reporter: Bozhen Liu
> Priority: Minor
>
> We run our static race detector on IntegrationTestRegionReplicaReplication, and discovered that multiple threads (type of WorkerThread) can invoke the method getKeyProvider() in class Encryption and read/write to keyProviderCache (type of ConcurrentHashMap) concurrently without proper lock protection. The related code is attached below:
> {code:java}
> public final class Encryption {
> ...
> static final Map<Pair<String,String>,KeyProvider> keyProviderCache = new ConcurrentHashMap<>();
> ...
> public static KeyProvider getKeyProvider(Configuration conf) {
> String providerClassName = conf.get(HConstants.CRYPTO_KEYPROVIDER_CONF_KEY,
> KeyStoreKeyProvider.class.getName());
> String providerParameters = conf.get(HConstants.CRYPTO_KEYPROVIDER_PARAMETERS_KEY, "");
> try {
> Pair<String,String> providerCacheKey = new Pair<>(providerClassName,
> providerParameters);
> KeyProvider provider = keyProviderCache.get(providerCacheKey);
> if (provider != null) {
> return provider;
> }
> provider = (KeyProvider) ReflectionUtils.newInstance(
> getClassLoaderForClass(KeyProvider.class).loadClass(providerClassName),
> conf);
> provider.init(providerParameters);
> if (LOG.isDebugEnabled()) {
> LOG.debug("Installed " + providerClassName + " into key provider cache");
> }
> keyProviderCache.put(providerCacheKey, provider);
> return provider;
> } catch (Exception e) {
> throw new RuntimeException(e);
> }
> }
> }
> {code}
> It is possible that multiple threads do the null check on provider, all of which return false, and each thread instantiated an instance of KeyProvider. However, only one provider will be added to the map and the other one is leaking.
> The possible traces of the race are shown below, where 3863 and 3864 are threads created by a loop:
> => Race: org/apache/hadoop/hbase/io/crypto/Encryption.java/util/Map
> (org/apache/hadoop/hbase/io/crypto/Encryption:553 (TID: 3863) , org/apache/hadoop/hbase/io/crypto/Encryption:542 (TID: 3864) )
> Trace 1st node: (TID: 3863)
> -> Call org/apache/hadoop/util/ToolRunner.run from org/apache/hadoop/hbase/IntegrationTestRegionReplicaReplication.main (line 224)
> -> Call org/apache/hadoop/hbase/util/AbstractHBaseTool.run from org/apache/hadoop/util/ToolRunner.run (line 76)
> -> Call org/apache/hadoop/hbase/util/LoadTestTool.doWork from org/apache/hadoop/hbase/util/AbstractHBaseTool.run (line 154)
> -> Call org/apache/hadoop/hbase/util/LoadTestTool.parallelLoadTables from org/apache/hadoop/hbase/util/LoadTestTool.doWork (line 589)
> *> Thread (3863) created by Application,Lorg/apache/hadoop/hbase/util/LoadTestTool.parallelLoadTables (line 880)
> -> Call org/apache/hadoop/hbase/util/LoadTestTool$WorkerThread.run from java/lang/Thread.start (line -1)
> -> Call org/apache/hadoop/util/ToolRunner.run from org/apache/hadoop/hbase/util/LoadTestTool$WorkerThread.run (line 931)
> -> Call org/apache/hadoop/hbase/util/AbstractHBaseTool.run from org/apache/hadoop/util/ToolRunner.run (line 76)
> -> Call org/apache/hadoop/hbase/util/LoadTestTool.doWork from org/apache/hadoop/hbase/util/AbstractHBaseTool.run (line 154)
> -> Call org/apache/hadoop/hbase/util/LoadTestTool.loadTable from org/apache/hadoop/hbase/util/LoadTestTool.doWork (line 591)
> -> Call org/apache/hadoop/hbase/util/LoadTestTool.initTestTable from org/apache/hadoop/hbase/util/LoadTestTool.loadTable (line 605)
> -> Call org/apache/hadoop/hbase/util/LoadTestTool.applyColumnFamilyOptions from org/apache/hadoop/hbase/util/LoadTestTool.initTestTable (line 580)
> -> Call org/apache/hadoop/hbase/security/EncryptionUtil.wrapKey from org/apache/hadoop/hbase/util/LoadTestTool.applyColumnFamilyOptions (line 300)
> -> Call org/apache/hadoop/hbase/io/crypto/Encryption.encryptWithSubjectKey from org/apache/hadoop/hbase/security/EncryptionUtil.wrapKey (line 107)
> -> Call org/apache/hadoop/hbase/io/crypto/Encryption.getSecretKeyForSubject from org/apache/hadoop/hbase/io/crypto/Encryption.encryptWithSubjectKey (line 448)
> -> Call org/apache/hadoop/hbase/io/crypto/Encryption.getKeyProvider from org/apache/hadoop/hbase/io/crypto/Encryption.getSecretKeyForSubject (line 424)
> => Write to org/apache/hadoop/hbase/io/crypto/Encryption.java/util/Map(keyProviderCache) in org/apache/hadoop/hbase/io/crypto/Encryption.getKeyProvider (line 553)
> Trace 2nd node: (TID: 3864)
> -> Call org/apache/hadoop/util/ToolRunner.run from org/apache/hadoop/hbase/IntegrationTestRegionReplicaReplication.main (line 224)
> -> Call org/apache/hadoop/hbase/util/AbstractHBaseTool.run from org/apache/hadoop/util/ToolRunner.run (line 76)
> -> Call org/apache/hadoop/hbase/util/LoadTestTool.doWork from org/apache/hadoop/hbase/util/AbstractHBaseTool.run (line 154)
> -> Call org/apache/hadoop/hbase/util/LoadTestTool.parallelLoadTables from org/apache/hadoop/hbase/util/LoadTestTool.doWork (line 589)
> *> Thread (3864) created by Application,Lorg/apache/hadoop/hbase/util/LoadTestTool.parallelLoadTables (line 880)
> -> Call org/apache/hadoop/hbase/util/LoadTestTool$WorkerThread.run from java/lang/Thread.start (line -1)
> -> Call org/apache/hadoop/util/ToolRunner.run from org/apache/hadoop/hbase/util/LoadTestTool$WorkerThread.run (line 931)
> -> Call org/apache/hadoop/hbase/util/AbstractHBaseTool.run from org/apache/hadoop/util/ToolRunner.run (line 76)
> -> Call org/apache/hadoop/hbase/util/LoadTestTool.doWork from org/apache/hadoop/hbase/util/AbstractHBaseTool.run (line 154)
> -> Call org/apache/hadoop/hbase/util/LoadTestTool.loadTable from org/apache/hadoop/hbase/util/LoadTestTool.doWork (line 591)
> -> Call org/apache/hadoop/hbase/util/LoadTestTool.initTestTable from org/apache/hadoop/hbase/util/LoadTestTool.loadTable (line 605)
> -> Call org/apache/hadoop/hbase/util/LoadTestTool.applyColumnFamilyOptions from org/apache/hadoop/hbase/util/LoadTestTool.initTestTable (line 580)
> -> Call org/apache/hadoop/hbase/security/EncryptionUtil.wrapKey from org/apache/hadoop/hbase/util/LoadTestTool.applyColumnFamilyOptions (line 300)
> -> Call org/apache/hadoop/hbase/io/crypto/Encryption.encryptWithSubjectKey from org/apache/hadoop/hbase/security/EncryptionUtil.wrapKey (line 107)
> -> Call org/apache/hadoop/hbase/io/crypto/Encryption.getSecretKeyForSubject from org/apache/hadoop/hbase/io/crypto/Encryption.encryptWithSubjectKey (line 448)
> -> Call org/apache/hadoop/hbase/io/crypto/Encryption.getKeyProvider from org/apache/hadoop/hbase/io/crypto/Encryption.getSecretKeyForSubject (line 424)
> => Write to org/apache/hadoop/hbase/io/crypto/Encryption.java/util/Map(keyProviderCache) in org/apache/hadoop/hbase/io/crypto/Encryption.getKeyProvider (line 542)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)