[jira] [Commented] (HIVE-20740) Remove global lock in ObjectStore.setConf method

["Vihang Karajgaonkar (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HIVE-20740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16671953#comment-16671953 ] 

Vihang Karajgaonkar commented on HIVE-20740:
--------------------------------------------

Reattached the last patch since precommit queue was drained due to infrastructure issues.

> Remove global lock in ObjectStore.setConf method
> ------------------------------------------------
>
>                 Key: HIVE-20740
>                 URL: https://issues.apache.org/jira/browse/HIVE-20740
>             Project: Hive
>          Issue Type: Improvement
>            Reporter: Vihang Karajgaonkar
>            Assignee: Vihang Karajgaonkar
>            Priority: Major
>         Attachments: HIVE-20740.01.patch, HIVE-20740.02.patch, HIVE-20740.04.patch, HIVE-20740.05.patch, HIVE-20740.06.patch, HIVE-20740.08.patch
>
>
> The ObjectStore#setConf method has a global lock which can block other clients in concurrent workloads.
> {code}
> @Override
>   @SuppressWarnings("nls")
>   public void setConf(Configuration conf) {
>     // Although an instance of ObjectStore is accessed by one thread, there may
>     // be many threads with ObjectStore instances. So the static variables
>     // pmf and prop need to be protected with locks.
>     pmfPropLock.lock();
>     try {
>       isInitialized = false;
>       this.conf = conf;
>       this.areTxnStatsSupported = MetastoreConf.getBoolVar(conf, ConfVars.HIVE_TXN_STATS_ENABLED);
>       configureSSL(conf);
>       Properties propsFromConf = getDataSourceProps(conf);
>       boolean propsChanged = !propsFromConf.equals(prop);
>       if (propsChanged) {
>         if (pmf != null){
>           clearOutPmfClassLoaderCache(pmf);
>           if (!forTwoMetastoreTesting) {
>             // close the underlying connection pool to avoid leaks
>             pmf.close();
>           }
>         }
>         pmf = null;
>         prop = null;
>       }
>       assert(!isActiveTransaction());
>       shutdown();
>       // Always want to re-create pm as we don't know if it were created by the
>       // most recent instance of the pmf
>       pm = null;
>       directSql = null;
>       expressionProxy = null;
>       openTrasactionCalls = 0;
>       currentTransaction = null;
>       transactionStatus = TXN_STATUS.NO_STATE;
>       initialize(propsFromConf);
>       String partitionValidationRegex =
>           MetastoreConf.getVar(this.conf, ConfVars.PARTITION_NAME_WHITELIST_PATTERN);
>       if (partitionValidationRegex != null && !partitionValidationRegex.isEmpty()) {
>         partitionValidationPattern = Pattern.compile(partitionValidationRegex);
>       } else {
>         partitionValidationPattern = null;
>       }
>       // Note, if metrics have not been initialized this will return null, which means we aren't
>       // using metrics.  Thus we should always check whether this is non-null before using.
>       MetricRegistry registry = Metrics.getRegistry();
>       if (registry != null) {
>         directSqlErrors = Metrics.getOrCreateCounter(MetricsConstants.DIRECTSQL_ERRORS);
>       }
>       this.batchSize = MetastoreConf.getIntVar(conf, ConfVars.RAWSTORE_PARTITION_BATCH_SIZE);
>       if (!isInitialized) {
>         throw new RuntimeException(
>         "Unable to create persistence manager. Check dss.log for details");
>       } else {
>         LOG.debug("Initialized ObjectStore");
>       }
>     } finally {
>       pmfPropLock.unlock();
>     }
>   }
> {code}
> The {{pmfPropLock}} is a static object and it disallows any other new connection to HMS which is trying to instantiate ObjectStore. We should either remove the lock or reduce the scope of the lock so that it is held for a very small amount of time.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)