You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@impala.apache.org by st...@apache.org on 2025/12/18 08:22:33 UTC
(impala) branch master updated (d4644b038 -> 3725b4ea6)
This is an automated email from the ASF dual-hosted git repository.
stigahuang pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
from d4644b038 IMPALA-13607, IMPALA-14490: Deflake test_cache_valid_on_nontransactional_table_ddls
new e22c0d33d IMPALA-13917 (part 1): Remove Beeswax from protocol dimension
new 3725b4ea6 IMPALA-14617: Deflake TestMetadataReplicas.test_catalog_restart
The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.../java/org/apache/impala/catalog/ImpaladCatalog.java | 6 ++----
tests/common/test_dimensions.py | 16 +++++++---------
tests/custom_cluster/test_metadata_replicas.py | 7 +++++--
3 files changed, 14 insertions(+), 15 deletions(-)
(impala) 02/02: IMPALA-14617: Deflake TestMetadataReplicas.test_catalog_restart
Posted by st...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
stigahuang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
commit 3725b4ea63b32724b1a6904fdb4d11d4d6e9bdc6
Author: stiga-huang <hu...@gmail.com>
AuthorDate: Wed Dec 17 17:02:44 2025 +0800
IMPALA-14617: Deflake TestMetadataReplicas.test_catalog_restart
TestMetadataReplicas.test_catalog_restart creates a db and an underlying
table in Hive, then expects an INVALIDATE command can bring up the table
in impalads. The test runs in the legacy catalogd mode that has the bug
of IMPALA-12103. So if the INVALIDATE command runs in a state that
catalgod has the db in cache but the db doesn't show up in impalad's
cache yet, catalogd will just return the table and impalad will skip it
due to db not exists. Then the above assertion fails.
The db is added in catalogd cache by processing the CREATE_DATABASE HMS
event, which is asynchronous with executing the INVALIDATE command. If
the command is triggered before that, the test passes. If the command is
triggered after that, the test fails.
When the test was written, we don't have HMS event processing yet. It's
expected that the db is also added in catalogd by the INVALIDATE
command. To deflake the issue, this patch disables HMS event processing
in this test, so catalogd always has a consistent cache with impalad
when executing the INVALIDATE command.
This patch also changes the log level of a log in ImpaladCatalog to warn
if a table is not added due to its db is missing.
Tests:
- Ran the test locally 10 times.
Change-Id: I2d17404cc8093eacf9b51df3d22caf5cbb6a61a9
Reviewed-on: http://gerrit.cloudera.org:8080/23798
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
fe/src/main/java/org/apache/impala/catalog/ImpaladCatalog.java | 6 ++----
tests/custom_cluster/test_metadata_replicas.py | 7 +++++--
2 files changed, 7 insertions(+), 6 deletions(-)
diff --git a/fe/src/main/java/org/apache/impala/catalog/ImpaladCatalog.java b/fe/src/main/java/org/apache/impala/catalog/ImpaladCatalog.java
index af834430c..d5f8cd34b 100644
--- a/fe/src/main/java/org/apache/impala/catalog/ImpaladCatalog.java
+++ b/fe/src/main/java/org/apache/impala/catalog/ImpaladCatalog.java
@@ -473,10 +473,8 @@ public class ImpaladCatalog extends Catalog implements FeCatalog {
long catalogVersion, long lastLoadedTime) throws TableLoadingException {
Db db = getDb(thriftTable.db_name);
if (db == null) {
- if (LOG.isTraceEnabled()) {
- LOG.trace("Parent database of table does not exist: " +
- thriftTable.db_name + "." + thriftTable.tbl_name);
- }
+ LOG.warn("Not adding table since the database does not exist: " +
+ thriftTable.db_name + "." + thriftTable.tbl_name);
return;
}
diff --git a/tests/custom_cluster/test_metadata_replicas.py b/tests/custom_cluster/test_metadata_replicas.py
index 2a1dd3dfb..2480a6a44 100644
--- a/tests/custom_cluster/test_metadata_replicas.py
+++ b/tests/custom_cluster/test_metadata_replicas.py
@@ -26,11 +26,14 @@ from tests.util.hive_utils import HiveDbWrapper
@SkipIfFS.hive
@CustomClusterTestSuite.with_args(
impalad_args="--use_local_catalog=false",
- catalogd_args="--catalog_topic_mode=full",
+ catalogd_args="--catalog_topic_mode=full --hms_event_polling_interval_s=0",
statestored_args="--statestore_update_frequency_ms=1000")
class TestMetadataReplicas(CustomClusterTestSuite):
""" Validates metadata content across catalogd and impalad coordinators.
- This test is only valid in legacy catalog mode. """
+ This test is only valid in legacy catalog mode. Disables HMS event processing to avoid
+ catalogd cache being modified unintentionally by HMS events. The test expects the
+ INVALIDATE command to bring up the latest metadata from HMS. This avoids hitting the bug
+ of IMPALA-12103. """
@classmethod
def setup_class(cls):
(impala) 01/02: IMPALA-13917 (part 1): Remove Beeswax from protocol dimension
Posted by st...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
stigahuang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
commit e22c0d33d64aed93a9df1c7a0ca63308595252a5
Author: Riza Suminto <ri...@cloudera.com>
AuthorDate: Mon Jun 23 14:56:03 2025 -0700
IMPALA-13917 (part 1): Remove Beeswax from protocol dimension
Beeswax protocol has been due for deprecation in a long time. This patch
remove BEESWAX from create_client_protocol_dimension(). This will limit
protocol dimension to [HS2, HS2_HTTP] by default. It is still possible
to include BEESWAX again for testing if DEFAULT_TEST_PROTOCOL env var is
set to 'beeswax', such as:
DEFAULT_TEST_PROTOCOL=beeswax impala-py.test custom_cluster/test_ipv6.py
This patch does not disable beeswax server yet. Some tests that
specifically test against beeswax protocol, such as test_beeswax.py,
will continue to work. ImpalaTestSuite.beeswax_client also remain
unchanged.
Testing:
Run following command and confirm that beeswax protocol is skipped.
impala-py.test --collect-only --exploration=exhaustive \
custom_cluster/test_ipv6.py
Change-Id: I3cff79f59305b5d44944804ed1f1b92838575495
Reviewed-on: http://gerrit.cloudera.org:8080/23076
Reviewed-by: Jason Fehr <jf...@cloudera.com>
Tested-by: Riza Suminto <ri...@cloudera.com>
---
tests/common/test_dimensions.py | 16 +++++++---------
1 file changed, 7 insertions(+), 9 deletions(-)
diff --git a/tests/common/test_dimensions.py b/tests/common/test_dimensions.py
index de65d10b5..4137347f9 100644
--- a/tests/common/test_dimensions.py
+++ b/tests/common/test_dimensions.py
@@ -169,22 +169,20 @@ def default_client_protocol_dimension():
return ImpalaTestDimension(PROTOCOL, pytest.config.option.default_test_protocol)
-def beeswax_client_protocol_dimension():
- return ImpalaTestDimension(PROTOCOL, BEESWAX)
-
-
def hs2_client_protocol_dimension():
return ImpalaTestDimension(PROTOCOL, HS2)
def create_client_protocol_dimension():
+ protocols_to_test = [HS2]
+ if pytest.config.option.default_test_protocol == BEESWAX:
+ protocols_to_test.append(BEESWAX)
# IMPALA-8864: Older python versions do not support SSLContext object that the thrift
- # http client implementation depends on. Falls back to a dimension without http
- # transport.
+ # http client implementation depends on. Otherwise, include HS2_HTTP.
import ssl
- if not hasattr(ssl, "create_default_context"):
- return ImpalaTestDimension(PROTOCOL, BEESWAX, HS2)
- return ImpalaTestDimension(PROTOCOL, BEESWAX, HS2, HS2_HTTP)
+ if hasattr(ssl, "create_default_context"):
+ protocols_to_test.append(HS2_HTTP)
+ return ImpalaTestDimension(PROTOCOL, *protocols_to_test)
def create_client_protocol_http_transport():