You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@impala.apache.org by as...@apache.org on 2024/02/16 00:17:40 UTC
(impala) branch master updated (1dbde2c19 -> ac5ed0f84)
This is an automated email from the ASF dual-hosted git repository.
asherman pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
from 1dbde2c19 IMPALA-12790: Fix overestimation in ScanNode.getInputCardinality
new 66ab46f64 IMPALA-12808: test_iceberg_deletes_and_updates failed by timeout error
new ac5ed0f84 IMPALA-12816: Remove exclusion of Netty to allow SSL connections to HBase
The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
fe/pom.xml | 11 -----------
tests/stress/test_update_stress.py | 19 +++++++++----------
2 files changed, 9 insertions(+), 21 deletions(-)
(impala) 01/02: IMPALA-12808: test_iceberg_deletes_and_updates failed by timeout error
Posted by as...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
asherman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
commit 66ab46f646909fb6e8b944f454b203cace07a634
Author: Zoltan Borok-Nagy <bo...@cloudera.com>
AuthorDate: Tue Feb 13 14:21:29 2024 +0100
IMPALA-12808: test_iceberg_deletes_and_updates failed by timeout error
On some systems test_iceberg_deletes_and_updates fails because
the delete thread cannot succeed as it always gets validation
errors due to the concurrent UPDATEs.
This patch adds SYNC_DDL=true to the DELETE/UPDATE threads,
and also prints out the exceptions which can be useful to debug
test failures in the future.
Testing:
* validated the fix in an environment where the test failed before
Change-Id: Icb0b078df2d95f3a62b124b6148f2f32aa425be4
Reviewed-on: http://gerrit.cloudera.org:8080/21033
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
tests/stress/test_update_stress.py | 19 +++++++++----------
1 file changed, 9 insertions(+), 10 deletions(-)
diff --git a/tests/stress/test_update_stress.py b/tests/stress/test_update_stress.py
index dec84ddfc..9cc767746 100644
--- a/tests/stress/test_update_stress.py
+++ b/tests/stress/test_update_stress.py
@@ -168,17 +168,17 @@ class TestIcebergConcurrentDeletesAndUpdates(ImpalaTestSuite):
"""Deletes every row from the table one by one."""
target_impalad = random.randint(0, ImpalaTestSuite.get_impalad_cluster_size() - 1)
impalad_client = ImpalaTestSuite.create_client_for_nth_impalad(target_impalad)
+ impalad_client.set_configuration_option("SYNC_DDL", "true")
i = 0
while i < num_rows:
try:
impalad_client.execute(
"delete from {0} WHERE id = {1}".format(tbl_name, i))
i += 1
- # Sleep after a succesful operation.
- time.sleep(random.random())
- except Exception:
+ except Exception as e:
# Exceptions are expected due to concurrent operations.
- pass
+ print(str(e))
+ time.sleep(random.random())
flag.value = 1
impalad_client.close()
@@ -186,15 +186,15 @@ class TestIcebergConcurrentDeletesAndUpdates(ImpalaTestSuite):
"""Updates every row in the table in a loop."""
target_impalad = random.randint(0, ImpalaTestSuite.get_impalad_cluster_size() - 1)
impalad_client = ImpalaTestSuite.create_client_for_nth_impalad(target_impalad)
+ impalad_client.set_configuration_option("SYNC_DDL", "true")
while flag.value != 1:
try:
impalad_client.execute(
"update {0} set j = j + 1".format(tbl_name))
- # Sleep after a succesful operation.
- time.sleep(random.random())
- except Exception:
+ except Exception as e:
# Exceptions are expected due to concurrent operations.
- pass
+ print(str(e))
+ time.sleep(random.random())
impalad_client.close()
def _impala_role_concurrent_checker(self, tbl_name, flag, num_rows):
@@ -232,7 +232,7 @@ class TestIcebergConcurrentDeletesAndUpdates(ImpalaTestSuite):
stored as iceberg
tblproperties('format-version'='2')""".format(tbl_name,))
- num_rows = 20
+ num_rows = 10
values_str = ""
for i in range(num_rows):
values_str += "({}, 0)".format(i)
@@ -246,6 +246,5 @@ class TestIcebergConcurrentDeletesAndUpdates(ImpalaTestSuite):
checker = Task(self._impala_role_concurrent_checker, tbl_name, flag, num_rows)
run_tasks([deleter, updater, checker])
- self.client.execute("refresh {}".format(tbl_name))
result = self.client.execute("select count(*) from {}".format(tbl_name))
assert result.data == ['0']
(impala) 02/02: IMPALA-12816: Remove exclusion of Netty to allow SSL connections to HBase
Posted by as...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
asherman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
commit ac5ed0f84946203b165ae68c265fb1de59ad62a9
Author: Andrew Sherman <as...@cloudera.com>
AuthorDate: Tue Feb 13 13:46:50 2024 -0800
IMPALA-12816: Remove exclusion of Netty to allow SSL connections to HBase
It is possible to configure Impala to use SSL when connecting to HBase.
Under the covers, an HBase client connection also has a connection to
ZooKeeper. SSL connections to ZooKeeper require Netty. Netty has been
excluded from Impala’s classpath since IMPALA-9648 because of earlier
security problems. IMPALA-9648 says that we need to be at least at
version 4.1.44 of Netty and we now pull in 4.1.86. To allow use of SSL
with HBase, remove the exclusion of Netty.
TESTING:
Ran end-to-end tests.
Ran queries against HBase configured to use SSL in a Cloudera
deployment.
Change-Id: Ic3038369345ca74f01ce2d4248c5beb21b633b2b
Reviewed-on: http://gerrit.cloudera.org:8080/21034
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
fe/pom.xml | 11 -----------
1 file changed, 11 deletions(-)
diff --git a/fe/pom.xml b/fe/pom.xml
index 85ad37676..1e1f7d121 100644
--- a/fe/pom.xml
+++ b/fe/pom.xml
@@ -69,10 +69,6 @@ under the License.
<artifactId>hadoop-hdfs</artifactId>
<version>${hadoop.version}</version>
<exclusions>
- <exclusion>
- <groupId>io.netty</groupId>
- <artifactId>*</artifactId>
- </exclusion>
<exclusion>
<!-- IMPALA-9108: Avoid pulling in leveldbjni, which is unneeded. -->
<groupId>org.fusesource.leveldbjni</groupId>
@@ -488,11 +484,6 @@ under the License.
<groupId>org.fusesource.leveldbjni</groupId>
<artifactId>*</artifactId>
</exclusion>
- <exclusion>
- <!-- IMPALA-9468: Avoid pulling in netty for security reasons -->
- <groupId>io.netty</groupId>
- <artifactId>*</artifactId>
- </exclusion>
<exclusion>
<groupId>com.sun.jersey</groupId>
<artifactId>jersey-server</artifactId>
@@ -874,8 +865,6 @@ under the License.
<!-- IMPALA-9649 (Avoid pulling in shiro* due to CVE's) -->
<exclude>org.apache.shiro:shiro-core:*</exclude>
<exclude>org.apache.shiro:shiro-crypto-cipher:*</exclude>
- <!-- IMPALA-9648: Ensure that netty-all is not present. -->
- <exclude>io.netty:*</exclude>
<!-- Avoid runtime dependency on rocksdb. -->
<exclude>org.rocksdb:*</exclude>
<!-- Avoid runtime dependency on Jersey server components. -->