You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2022/09/01 12:57:00 UTC
[jira] [Work logged] (HIVE-26500) Improve TestHiveMetastore
[ https://issues.apache.org/jira/browse/HIVE-26500?focusedWorklogId=805513&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-805513 ]
ASF GitHub Bot logged work on HIVE-26500:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 01/Sep/22 12:56
Start Date: 01/Sep/22 12:56
Worklog Time Spent: 10m
Work Description: ayushtkn commented on code in PR #3556:
URL: https://github.com/apache/hive/pull/3556#discussion_r960610224
##########
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java:
##########
@@ -168,16 +169,16 @@ public void testNameMethods() {
try {
List<String> testVals = client.partitionNameToVals(partName);
- assertTrue("Values from name are incorrect", vals.equals(testVals));
+ assertTrue("Values from name are incorrect: " + testVals, vals.equals(testVals));
Map<String, String> testSpec = client.partitionNameToSpec(partName);
- assertTrue("Spec from name is incorrect", spec.equals(testSpec));
+ assertTrue("Spec from name is incorrect: " + testSpec, spec.equals(testSpec));
List<String> emptyVals = client.partitionNameToVals("");
- assertTrue("Values should be empty", emptyVals.size() == 0);
+ assertTrue("Values should be empty, instead its size is: " + emptyVals.size(), emptyVals.size() == 0);
Map<String, String> emptySpec = client.partitionNameToSpec("");
- assertTrue("Spec should be empty", emptySpec.size() == 0);
+ assertTrue("Spec should be empty, instead its size is: " + emptySpec.size(), emptySpec.size() == 0);
Review Comment:
Can be changed to ``assertEquals``
##########
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java:
##########
@@ -3306,22 +3314,38 @@ public void testJDOPersistanceManagerCleanup() throws Exception {
return;
}
- int numObjectsBeforeClose = getJDOPersistanceManagerCacheSize();
+ int numObjectsBeforeUse = getJDOPersistanceManagerCacheSize();
HiveMetaStoreClient closingClient = new HiveMetaStoreClient(conf);
closingClient.getAllDatabases();
closingClient.close();
- Thread.sleep(5 * 1000); // give HMS time to handle close request
- int numObjectsAfterClose = getJDOPersistanceManagerCacheSize();
- assertTrue(numObjectsBeforeClose == numObjectsAfterClose);
+ MetastoreTestUtils.waitForAssertion("Checking pm cachesize after client close", new Runnable() {
+ @Override
+ public void run() {
+ int numObjectsAfterClose = getJDOPersistanceManagerCacheSize();
+ assertTrue(String.format("numObjectsBeforeUse: %d != numObjectsAfterClose: %d", numObjectsBeforeUse,
+ numObjectsAfterClose), numObjectsBeforeUse == numObjectsAfterClose);
+ }
+ }, 500, 30000);
Review Comment:
Can use Lambda & assertTrue can be changed to ``assertEqual``?
Something like:
```
MetastoreTestUtils.waitForAssertion("Checking pm cachesize after client close", () -> {
int numObjectsAfterClose = getJDOPersistanceManagerCacheSize();
assertEquals(String
.format("numObjectsBeforeUse: %d != numObjectsAfterClose: %d", numObjectsBeforeUse, numObjectsAfterClose),
numObjectsBeforeUse, numObjectsAfterClose);
}, 500, 30000);
```
##########
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java:
##########
@@ -3306,22 +3314,38 @@ public void testJDOPersistanceManagerCleanup() throws Exception {
return;
}
- int numObjectsBeforeClose = getJDOPersistanceManagerCacheSize();
+ int numObjectsBeforeUse = getJDOPersistanceManagerCacheSize();
HiveMetaStoreClient closingClient = new HiveMetaStoreClient(conf);
closingClient.getAllDatabases();
closingClient.close();
- Thread.sleep(5 * 1000); // give HMS time to handle close request
- int numObjectsAfterClose = getJDOPersistanceManagerCacheSize();
- assertTrue(numObjectsBeforeClose == numObjectsAfterClose);
+ MetastoreTestUtils.waitForAssertion("Checking pm cachesize after client close", new Runnable() {
+ @Override
+ public void run() {
+ int numObjectsAfterClose = getJDOPersistanceManagerCacheSize();
+ assertTrue(String.format("numObjectsBeforeUse: %d != numObjectsAfterClose: %d", numObjectsBeforeUse,
+ numObjectsAfterClose), numObjectsBeforeUse == numObjectsAfterClose);
+ }
+ }, 500, 30000);
+
+ int numObjectsAfterClose = getJDOPersistanceManagerCacheSize();
HiveMetaStoreClient nonClosingClient = new HiveMetaStoreClient(conf);
nonClosingClient.getAllDatabases();
// Drop connection without calling close. HMS thread deleteContext
// will trigger cleanup
nonClosingClient.getTTransport().close();
- Thread.sleep(5 * 1000);
- int numObjectsAfterDroppedConnection = getJDOPersistanceManagerCacheSize();
- assertTrue(numObjectsAfterClose == numObjectsAfterDroppedConnection);
+
+ MetastoreTestUtils.waitForAssertion("Checking pm cachesize after transport close", new Runnable() {
+ @Override
+ public void run() {
+ int numObjectsAfterDroppedConnection = getJDOPersistanceManagerCacheSize();
+ assertTrue(String.format("numObjectsAfterClose: %d != numObjectsAfterDroppedConnection: %d",
+ numObjectsAfterClose, numObjectsAfterDroppedConnection),
+ numObjectsAfterClose == numObjectsAfterDroppedConnection);
+ }
+ }, 500, 30000);
Review Comment:
Same as above can use Lambda & change to ``assertEquals``
```
MetastoreTestUtils.waitForAssertion("Checking pm cachesize after transport close", () -> {
int numObjectsAfterDroppedConnection = getJDOPersistanceManagerCacheSize();
assertEquals(String
.format("numObjectsAfterClose: %d != numObjectsAfterDroppedConnection: %d", numObjectsAfterClose,
numObjectsAfterDroppedConnection), numObjectsAfterClose, numObjectsAfterDroppedConnection);
}, 500, 30000);
```
##########
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/utils/MetastoreTestUtils.java:
##########
@@ -0,0 +1,65 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore.utils;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Common utils for metastore tests.
+ */
+public class MetastoreTestUtils {
+
+ private static final Logger LOG = LoggerFactory.getLogger(MetastoreTestUtils.class);
+
+ /**
+ * This method can run an assertion wrapped in to a runnable, and keep retrying it for a certain amount of time.
+ * It can be useful when the assertion doesn't necessarily pass immediately, and it would be hard
+ * to mock into production code in order to wait for some conditions properly. Instead, it just
+ * waits, but not like a constant time before a single attempt (which is easy, but errorprone).
+ * @param assertionContext
+ * @param runnable
+ * @param msBetweenAssertionAttempts
+ * @param msOverallTimeout
+ * @throws Exception
+ */
+ public static void waitForAssertion(String assertionContext, Runnable assertionRunnable, int msBetweenAssertionAttempts,
Review Comment:
Hadoop common does provide similar util:
https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/test/GenericTestUtils.java#L440
It is used in hive-code at a couple of places, but I think there is no hadoop-common test dependency in this module? if it is we can use that? or just give a check if we can copy that if it has something better
Issue Time Tracking
-------------------
Worklog Id: (was: 805513)
Time Spent: 0.5h (was: 20m)
> Improve TestHiveMetastore
> -------------------------
>
> Key: HIVE-26500
> URL: https://issues.apache.org/jira/browse/HIVE-26500
> Project: Hive
> Issue Type: Improvement
> Reporter: László Bodor
> Assignee: László Bodor
> Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> {code}
> Thread.sleep(5 * 1000); // give HMS time to handle close request
> int numObjectsAfterClose = getJDOPersistanceManagerCacheSize();
> assertTrue(numObjectsBeforeClose == numObjectsAfterClose);
> {code}
> 1. easy fix: assertTrue(numObjectsBeforeClose == numObjectsAfterClose);
> this tells nothing in case of assertionerror:
> {code}
> java.lang.AssertionError
> at org.junit.Assert.fail(Assert.java:87)
> at org.junit.Assert.assertTrue(Assert.java:42)
> at org.junit.Assert.assertTrue(Assert.java:53)
> at org.apache.hadoop.hive.metastore.TestHiveMetaStore.testJDOPersistanceManagerCleanup(TestHiveMetaStore.java:3298)
> {code}
> should tell a more informative message with particular values
> 2. harder fix, maybe do it later
> {code}
> Thread.sleep(5 * 1000); // give HMS time to handle close request
> {code}
> sleep is always dangerous, cannot make sure it handled everything that we expected
--
This message was sent by Atlassian Jira
(v8.20.10#820010)