You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@impala.apache.org by st...@apache.org on 2023/08/16 06:15:12 UTC
[impala] branch master updated (75bf21341 -> ddcdfc220)
This is an automated email from the ASF dual-hosted git repository.
stigahuang pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
from 75bf21341 IMPALA-12288: Add BUILD_WITH_NO_TESTS option to remove test targets
new 7039e1ecc IMPALA-12257: Fix NPE in createInsertEvents when partitions only exist in HMS
new 240a9cddc [docs] Add note about REFRESH for tables in custom catalog
new ddcdfc220 IMPALA-11195: Disable SSL renegotiations for OpenSSL 1.0.2.
The 3 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
be/src/rpc/authentication.cc | 8 +
be/src/thirdparty/squeasel/squeasel.c | 28 +++-
bin/impala-config.sh | 2 +-
docs/topics/impala_iceberg.xml | 2 +
.../java/org/apache/impala/catalog/HdfsTable.java | 11 ++
.../apache/impala/service/CatalogOpExecutor.java | 164 ++++++++++++---------
tests/custom_cluster/test_insert_behaviour.py | 65 ++++++++
7 files changed, 199 insertions(+), 81 deletions(-)
[impala] 01/03: IMPALA-12257: Fix NPE in createInsertEvents when partitions only exist in HMS
Posted by st...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
stigahuang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
commit 7039e1ecc60bf75939803ca95414df113f68048f
Author: stiga-huang <hu...@gmail.com>
AuthorDate: Sat Jul 1 06:15:23 2023 +0800
IMPALA-12257: Fix NPE in createInsertEvents when partitions only exist in HMS
When files have been generated for INSERT statements, coordinator
invokes the updateCatalog RPC to let catalogd finalize the HMS changes.
Catalogd will fire INSERT events for external tables at the end of this
work. This helps other systems that consume HMS events (e.g. Hive
replication or other Impala clusters) be notified for the changes. It
should be best-effort that any errors in it should not fail the INSERT
statement.
We've seen a NullPointerException thrown in createInsertEvents() in the
case that an updated partition doesn't exist in the metadata cache of
catalogd but exist in HMS. Note that the partition list cached in
catalogd could differ from the actual list in HMS, especially when HMS
event processing is disabled. If an INSERT statement modifies a
partition that exists in HMS but not yet synced to catalogd,
createInsertEvents() will throw a NPE when finding such a partition in
catalogd. This fixes the NPE by skipping this finding step and using the
partition names directly.
This patch also refactors createInsertEvents to make it shorter.
Tests
- Add e2e tests
Change-Id: I7d77844e26283ecb16b3b3aeb9f634bb3113eacd
Reviewed-on: http://gerrit.cloudera.org:8080/20148
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
.../java/org/apache/impala/catalog/HdfsTable.java | 11 ++
.../apache/impala/service/CatalogOpExecutor.java | 164 ++++++++++++---------
tests/custom_cluster/test_insert_behaviour.py | 65 ++++++++
3 files changed, 167 insertions(+), 73 deletions(-)
diff --git a/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java b/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
index dba18957b..0d8b6ba80 100644
--- a/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
+++ b/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
@@ -1769,6 +1769,17 @@ public class HdfsTable extends Table implements FeFsTable {
return parts;
}
+ /**
+ * Tracks the in-flight INSERT event id in the partition.
+ * @return false if the partition doesn't exist. Otherwise returns true.
+ */
+ public boolean addInflightInsertEventToPartition(String partName, long eventId) {
+ HdfsPartition partition = nameToPartitionMap_.get(partName);
+ if (partition == null) return false;
+ partition.addToVersionsForInflightEvents(/*isInsertEvent*/true, eventId);
+ return true;
+ }
+
private void setUnpartitionedTableStats(HdfsPartition.Builder partBuilder) {
Preconditions.checkState(numClusteringCols_ == 0);
// For unpartitioned tables set the numRows in its single partition
diff --git a/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java b/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
index d115ea2d0..5d8391845 100644
--- a/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
+++ b/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
@@ -38,6 +38,7 @@ import java.util.Collections;
import java.util.HashMap;
import java.util.HashSet;
import java.util.Iterator;
+import java.util.LinkedHashSet;
import java.util.List;
import java.util.ListIterator;
import java.util.Map;
@@ -71,6 +72,7 @@ import org.apache.hadoop.hive.metastore.api.DataOperationType;
import org.apache.hadoop.hive.metastore.api.Database;
import org.apache.hadoop.hive.metastore.api.FieldSchema;
import org.apache.hadoop.hive.metastore.api.InsertEventRequestData;
+import org.apache.hadoop.hive.metastore.api.MetaException;
import org.apache.hadoop.hive.metastore.api.NoSuchObjectException;
import org.apache.hadoop.hive.metastore.api.NotificationEvent;
import org.apache.hadoop.hive.metastore.api.Partition;
@@ -6833,9 +6835,13 @@ public class CatalogOpExecutor {
}
// Before commit fire insert events if external event processing is
- // enabled.
- createInsertEvents((FeFsTable)table, update.getUpdated_partitions(),
- addedPartitionNames, update.is_overwrite, tblTxn);
+ // enabled. This is best-effort. Any errors in it should not fail the INSERT.
+ try {
+ createInsertEvents((FeFsTable) table, update.getUpdated_partitions(),
+ addedPartitionNames, update.is_overwrite, tblTxn);
+ } catch (Exception e) {
+ LOG.error("Failed to fire insert events for table {}", table.getFullName(), e);
+ }
// Commit transactional inserts on success. We don't abort the transaction
// here in case of failures, because the client, i.e. query coordinator, is
@@ -6886,10 +6892,7 @@ public class CatalogOpExecutor {
/**
* Populates insert event data and calls fireInsertEvents() if external event
- * processing is enabled. This is no-op if event processing is disabled.
- * TODO: I am not sure that it is the right thing to connect event polling and
- * event sending to the same config. This means that turning off automatic
- * refresh will also break replication.
+ * processing is enabled.
* This method is replicating what Hive does in case a table or partition is inserts
* into. There are 2 cases:
* 1. If the table is transactional, we should first generate ADD_PARTITION events
@@ -6911,81 +6914,57 @@ public class CatalogOpExecutor {
private void createInsertEvents(FeFsTable table,
Map<String, TUpdatedPartition> updatedPartitions,
Map<String, List<String>> addedPartitionNames,
- boolean isInsertOverwrite, TblTransaction tblTxn) throws CatalogException {
- if (!shouldGenerateInsertEvents(table)) {
- return;
- }
+ boolean isInsertOverwrite, TblTransaction tblTxn)
+ throws CatalogException, MetaException {
+ if (!shouldGenerateInsertEvents(table)) return;
long txnId = tblTxn == null ? -1 : tblTxn.txnId;
long writeId = tblTxn == null ? -1: tblTxn.writeId;
// If the table is transaction table we should generate a transactional
// insert event type. This would show up in HMS as an ACID_WRITE event.
boolean isTransactional = AcidUtils.isTransactionalTable(table.getMetaStoreTable()
.getParameters());
- Preconditions.checkState(!isTransactional || txnId > 0, String
- .format("Invalid transaction id %s for generating insert events on table %s",
- txnId, table.getFullName()));
- Preconditions.checkState(!isTransactional || writeId > 0,
- String.format("Invalid write id %s for generating insert events on table %s",
- writeId, table.getFullName()));
+ if (isTransactional) {
+ Preconditions.checkState(txnId > 0, "Invalid transaction id %s for table %s",
+ txnId, table.getFullName());
+ Preconditions.checkState(writeId > 0, "Invalid write id %s for table %s",
+ writeId, table.getFullName());
+ }
boolean isPartitioned = table.getNumClusteringCols() > 0;
// List of all insert events that we call HMS fireInsertEvent() on.
List<InsertEventRequestData> insertEventReqDatas = new ArrayList<>();
// The partition val list corresponding to insertEventReqDatas for Apache Hive-3
List<List<String>> insertEventPartVals = new ArrayList<>();
- // List of all existing partitions that we insert into.
- List<HdfsPartition> existingPartitions = new ArrayList<>();
+ // List of all existing partition names in HMS that we insert into. It's possible
+ // that the partition doesn't exist in catalogd's cache.
+ // Use LinkedHashSet to preserve the order.
+ LinkedHashSet<String> existingPartSetInHms = new LinkedHashSet<>();
if (isPartitioned) {
- Set<String> existingPartSet = new HashSet<String>(updatedPartitions.keySet());
- existingPartSet.removeAll(addedPartitionNames.keySet());
- // Only HdfsTable can have partitions, Iceberg tables are treated as unpartitioned.
- existingPartitions = ((HdfsTable) table).getPartitionsForNames(existingPartSet);
+ existingPartSetInHms.addAll(updatedPartitions.keySet());
+ existingPartSetInHms.removeAll(addedPartitionNames.keySet());
+ // Create events for existing partitions in partitioned tables.
+ for (String partName : existingPartSetInHms) {
+ List<String> partVals = MetaStoreUtil.getPartValsFromName(
+ table.getMetaStoreTable(), partName);
+ prepareInsertEventData(table, partName, partVals, updatedPartitions,
+ isInsertOverwrite, isPartitioned, insertEventReqDatas, insertEventPartVals);
+ }
} else {
Preconditions.checkState(updatedPartitions.size() == 1);
- // Unpartitioned tables have a single partition with empty name,
- // see HdfsTable.DEFAULT_PARTITION_NAME.
- List<String> newFiles = updatedPartitions.get("").getFiles();
- List<String> partVals = new ArrayList<>();
- LOG.info(String.format("%s new files detected for table %s", newFiles.size(),
- table.getFullName()));
- if (!newFiles.isEmpty() || isInsertOverwrite) {
- insertEventReqDatas.add(
- makeInsertEventData( table, partVals, newFiles, isInsertOverwrite));
- insertEventPartVals.add(partVals);
- }
- }
-
- // Create events for existing partitions in partitioned tables.
- for (HdfsPartition part : existingPartitions) {
- List<String> newFiles = updatedPartitions.get(part.getPartitionName()).getFiles();
- List<String> partVals = part.getPartitionValuesAsStrings(true);
- Preconditions.checkState(!partVals.isEmpty());
- if (!newFiles.isEmpty() || isInsertOverwrite) {
- LOG.info(String.format("%s new files detected for table %s partition %s",
- newFiles.size(), table.getFullName(), part.getPartitionName()));
- insertEventReqDatas.add(
- makeInsertEventData(table, partVals, newFiles, isInsertOverwrite));
- insertEventPartVals.add(partVals);
- }
+ prepareInsertEventData(table, HdfsTable.DEFAULT_PARTITION_NAME,
+ Collections.emptyList(), updatedPartitions, isInsertOverwrite, isPartitioned,
+ insertEventReqDatas, insertEventPartVals);
}
// Create events for new partitions only in ACID tables.
if (isTransactional) {
for (Map.Entry<String, List<String>> part : addedPartitionNames.entrySet()) {
- List<String> newFiles = updatedPartitions.get(part.getKey()).getFiles();
- List<String> partVals = part.getValue();
- Preconditions.checkState(!partVals.isEmpty());
- LOG.info(String.format("%s new files detected for table %s new partition %s",
- newFiles.size(), table.getFullName(), part.getKey()));
- insertEventReqDatas.add(
- makeInsertEventData(table, partVals, newFiles, isInsertOverwrite));
- insertEventPartVals.add(partVals);
+ prepareInsertEventData(table, part.getKey(), part.getValue(), updatedPartitions,
+ isInsertOverwrite, isPartitioned, insertEventReqDatas, insertEventPartVals);
}
}
- if (insertEventReqDatas.isEmpty()) {
- return;
- }
+ if (insertEventReqDatas.isEmpty()) return;
MetaStoreClient metaStoreClient = catalog_.getMetaStoreClient();
TableInsertEventInfo insertEventInfo = new TableInsertEventInfo(
@@ -6993,23 +6972,62 @@ public class CatalogOpExecutor {
List<Long> eventIds = MetastoreShim.fireInsertEvents(metaStoreClient,
insertEventInfo, table.getDb().getName(), table.getName());
if (isTransactional) {
- // ACID inserts do not generate INSERT events as it is enough to listen to the
- // COMMIT event fired by HMS. Impala ignores COMMIT events, so we don't
- // have to worry about reloading as a result of this "self" event.
- // Note that Hive inserts also lead to an ALTER event which is the actual event
- // that causes Impala to reload the table.
+ // ACID inserts do not generate INSERT events so there is nothing we need to track
+ // here for self-event detection. Note that It is enough to listen to the commit
+ // events fired by HMS, i.e. ALLOC_WRITE_ID_EVENT and COMMIT_TXN events.
+ // For commit events triggered by Impala itself, it skips reloads for them based
+ // on the writeIds. Commit events triggered by external systems (e.g. Hive) will
+ // bring new writeIds which causes Impala to reload the table.
+ // See more in addCommittedWriteIdsAndReloadPartitionsIfExist().
Preconditions.checkState(eventIds.isEmpty());
return;
}
- if (!eventIds.isEmpty()) {
- if (!isPartitioned) { // insert into table
- Preconditions.checkState(eventIds.size() == 1);
- catalog_.addVersionsForInflightEvents(true, (Table)table, eventIds.get(0));
- } else { // insert into partition
- Preconditions.checkState(existingPartitions.size() == eventIds.size());
- for (int par_idx = 0; par_idx < existingPartitions.size(); par_idx++) {
- existingPartitions.get(par_idx).addToVersionsForInflightEvents(
- true, eventIds.get(par_idx));
+ trackInsertEvents((HdfsTable)table, eventIds, existingPartSetInHms);
+ }
+
+ /**
+ * Helper method to prepare InsertEventRequestData. Also collect the list of
+ * partition values.
+ */
+ private void prepareInsertEventData(FeFsTable table,
+ String partName, List<String> partVals,
+ Map<String, TUpdatedPartition> updatedPartitions,
+ boolean isInsertOverwrite, boolean isPartitioned,
+ List<InsertEventRequestData> insertEventReqDatas,
+ List<List<String>> insertEventPartVals) throws CatalogException {
+ List<String> newFiles = updatedPartitions.get(partName).getFiles();
+ if (!newFiles.isEmpty() || isInsertOverwrite) {
+ LOG.info("{} new files detected for table {}{}",
+ newFiles.size(), table.getFullName(),
+ isPartitioned ? " partition " + partName : "");
+ insertEventReqDatas.add(
+ makeInsertEventData(table, partVals, newFiles, isInsertOverwrite));
+ insertEventPartVals.add(partVals);
+ }
+ }
+
+ /**
+ * Keeps track of the self-generated INSERT events for self-event detection.
+ * Each item in 'eventIds' corresponds to the partition in 'existingPartsInHms'
+ * based on the iteration order.
+ * Note that this is not called for transactional tables since INSERT on them
+ * don't trigger INSERT events.
+ */
+ private void trackInsertEvents(HdfsTable table, List<Long> eventIds,
+ LinkedHashSet<String> existingPartsInHms) {
+ if (eventIds == null || eventIds.isEmpty()) return;
+ if (table.getNumClusteringCols() == 0) { // insert into table
+ Preconditions.checkState(eventIds.size() == 1);
+ catalog_.addVersionsForInflightEvents(true, table, eventIds.get(0));
+ } else { // insert into partition
+ Preconditions.checkState(existingPartsInHms.size() == eventIds.size());
+ int par_idx = 0;
+ for (String partName : existingPartsInHms) {
+ long eventId = eventIds.get(par_idx++);
+ if (!table.addInflightInsertEventToPartition(partName, eventId)) {
+ LOG.warn("INSERT event {} on partition {} of table {} are not tracked since " +
+ "it doesn't exist in catalogd cache", eventId, partName,
+ table.getFullName());
}
}
}
diff --git a/tests/custom_cluster/test_insert_behaviour.py b/tests/custom_cluster/test_insert_behaviour.py
index 44076a2a6..f74ba80d2 100644
--- a/tests/custom_cluster/test_insert_behaviour.py
+++ b/tests/custom_cluster/test_insert_behaviour.py
@@ -20,6 +20,7 @@ import pytest
from tests.common.custom_cluster_test_suite import CustomClusterTestSuite
from tests.common.skip import SkipIfFS, SkipIfLocal
+from tests.util.event_processor_utils import EventProcessorUtils
from tests.util.filesystem_utils import IS_ISILON, WAREHOUSE
from tests.util.hdfs_util import (
HdfsConfig,
@@ -129,3 +130,67 @@ class TestInsertBehaviourCustomCluster(CustomClusterTestSuite):
self._check_partition_perms("p1=1/p2=3/p3=4/", default_perms)
finally:
client.close()
+
+
+@SkipIfFS.hive
+class TestInsertUnSyncedPartition(CustomClusterTestSuite):
+
+ @classmethod
+ def setup_class(cls):
+ super(TestInsertUnSyncedPartition, cls).setup_class()
+
+ @CustomClusterTestSuite.with_args(catalogd_args="--hms_event_polling_interval_s=0")
+ def test_insert_unsynced_partition(self, unique_database):
+ """Regression test for IMPALA-12257. Tests with event-processing disabled so
+ catalogd can easily have unsynced partition with HMS."""
+ self._test_insert_on_unsynced_partition(unique_database, "part1", False, False)
+ self._test_insert_on_unsynced_partition(unique_database, "part2", False, True)
+ self._test_insert_on_unsynced_partition(unique_database, "txn_part1", True, False)
+ self._test_insert_on_unsynced_partition(unique_database, "txn_part2", True, True)
+
+ def _test_insert_on_unsynced_partition(self, db, tbl, is_transactional, is_overwrite):
+ tbl_name = db + "." + tbl
+ create_stmt = """
+ create table {0} (i int) partitioned by (p int)
+ stored as textfile""".format(tbl_name)
+ if is_transactional:
+ create_stmt += """ tblproperties(
+ 'transactional'='true',
+ 'transactional_properties'='insert_only')"""
+ self.client.execute(create_stmt)
+ # Run any query on the table to make it loaded in catalogd.
+ self.client.execute("describe " + tbl_name)
+ # Add the partition in Hive so catalogd is not aware of it.
+ self.run_stmt_in_hive("""
+ insert into {0} partition (p=0) values (0)""".format(tbl_name))
+ # Track the last event id so we can fetch the generated events
+ last_event_id = EventProcessorUtils.get_current_notification_id(self.hive_client)
+ # Insert the new partition in Impala.
+ self.client.execute("""
+ insert {0} {1} partition(p=0) values (1)
+ """.format("overwrite" if is_overwrite else "into", tbl_name))
+ events = EventProcessorUtils.get_next_notification(self.hive_client, last_event_id)
+ if is_transactional:
+ assert len(events) > 2
+ assert events[0].eventType == "OPEN_TXN"
+ assert events[1].eventType == "ALLOC_WRITE_ID_EVENT"
+ assert events[1].dbName == db
+ assert events[1].tableName == tbl
+ # There is an empty ADD_PARTITION event due to Impala invokes the add_partitions
+ # HMS API but no new partitions are really added. This might change in future Hive
+ # versions. Here we just verify whether the last event is COMMIT_TXN.
+ assert events[len(events) - 1].eventType == "COMMIT_TXN"
+ else:
+ assert len(events) > 0
+ last_event = events[len(events) - 1]
+ assert last_event.dbName == db
+ assert last_event.tableName == tbl
+ assert last_event.eventType == "INSERT"
+
+ res = self.client.execute("select * from " + tbl_name)
+ if is_overwrite:
+ assert res.data == ["1\t0"]
+ else:
+ assert "0\t0" in res.data
+ assert "1\t0" in res.data
+ assert len(res.data) == 2
[impala] 03/03: IMPALA-11195: Disable SSL renegotiations for OpenSSL 1.0.2.
Posted by st...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
stigahuang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
commit ddcdfc2200c7d605b07d443ba911f89c48ecee39
Author: Zoltan Borok-Nagy <bo...@cloudera.com>
AuthorDate: Tue Aug 8 16:07:29 2023 +0200
IMPALA-11195: Disable SSL renegotiations for OpenSSL 1.0.2.
When OpenSSL 1.0.2 was being used Impala didn't disable SSL
renegotiations correctly. It wasn't enough setting the flag
SSL3_FLAGS_NO_RENEGOTIATE_CIPHERS right after SSL_new() as
due to an OpenSSL bug the SSL flags are getting reset in SSL_accept().
See https://github.com/openssl/openssl/discussions/21666
With this patch the followings happen depending on the OpenSSL versions:
* OpenSSL 1.0.2: we set the SSL3_FLAGS_NO_RENEGOTIATE_CIPHERS flag in
a callback function the same way as it is being set in various
open source projects
* OpenSSL 1.1.0h+: we set the SSL_OP_NO_RENEGOTIATION option in the
SSL_CTX object
* OpenSSL versions between the above two: we raise either compile-time
error (when compiled with such version) or runtime warning (when
Impala is running with such version)
This patch also upgrades the native toolchain version to have a Thrift
that is fixed the same way.
Testing:
* manually tested on CentOS 7.9 with
>openssl s_client -connect <host>:<port>
>R
Initially it outputs that "Secure Renegotiation IS supported" because
we can only set the SSL3_FLAGS_NO_RENEGOTIATE_CIPHERS flag after the
handshake, but then prompting "R" doesn't do the renegotiation.
Without these changes "R" renegotiates.
Change-Id: I6fd6a8dedcbca8f50a16dbe68ebd8303e3e5aed1
Reviewed-on: http://gerrit.cloudera.org:8080/20360
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
be/src/rpc/authentication.cc | 8 ++++++++
be/src/thirdparty/squeasel/squeasel.c | 28 +++++++++++++++++++++-------
bin/impala-config.sh | 2 +-
3 files changed, 30 insertions(+), 8 deletions(-)
diff --git a/be/src/rpc/authentication.cc b/be/src/rpc/authentication.cc
index 16bf5acf7..5dadd8e3a 100644
--- a/be/src/rpc/authentication.cc
+++ b/be/src/rpc/authentication.cc
@@ -1495,6 +1495,14 @@ Status AuthManager::Init() {
TSSLSocketFactory::setManualOpenSSLInitialization(true);
kudu::security::InitializeOpenSSL();
LOG(INFO) << "Initialized " << OPENSSL_VERSION_TEXT;
+ LOG(INFO) << "Runtime OpenSSL version " << SSLeay_version(SSLEAY_VERSION);
+ unsigned long openssl_version = SSLeay();
+ // Check if we are running against an OpenSSL version that is vulnerable to
+ // CVE-2009-3555
+ if (openssl_version >= 0x10100000L && openssl_version <= 0x1010007fL) {
+ LOG(WARNING) <<
+ "OpenSSL runtime version detected that is vulnerable to CVE-2009-3555";
+ }
// Could use any other requiered flag for SAML
bool use_saml = !FLAGS_saml2_sp_callback_url.empty();
diff --git a/be/src/thirdparty/squeasel/squeasel.c b/be/src/thirdparty/squeasel/squeasel.c
index 1ba9aa8bd..a3934fa22 100644
--- a/be/src/thirdparty/squeasel/squeasel.c
+++ b/be/src/thirdparty/squeasel/squeasel.c
@@ -3792,15 +3792,21 @@ static int set_uid_option(struct sq_context *ctx) {
static pthread_mutex_t *ssl_mutexes;
-static int sslize(struct sq_connection *conn, SSL_CTX *s, int (*func)(SSL *)) {
- return (conn->ssl = SSL_new(s)) != NULL &&
#if OPENSSL_VERSION_NUMBER < 0x10100000L
- // IMPALA-11195: disable TLS/SSL renegotiation. In version 1.0.2 and prior it's
- // possible to use the undocumented SSL3_FLAGS_NO_RENEGOTIATE_CIPHERS flag.
- // For more context, see a note on the SSL_OP_NO_RENEGOTIATION option in the
- // $OPENSSL_ROOT/CHANGES and https://github.com/openssl/openssl/issues/4739.
- (conn->ssl->s3->flags |= SSL3_FLAGS_NO_RENEGOTIATE_CIPHERS) &&
+// IMPALA-11195: disable TLS/SSL renegotiation. In version 1.0.2 and prior it's
+// possible to use the undocumented SSL3_FLAGS_NO_RENEGOTIATE_CIPHERS flag.
+static void ssl_disable_renegotiation_cb(const SSL *ssl, int where, int ret)
+{
+ (void)ret;
+ if ((where & SSL_CB_HANDSHAKE_DONE) != 0) {
+ // disable renegotiation (CVE-2009-3555)
+ ssl->s3->flags |= SSL3_FLAGS_NO_RENEGOTIATE_CIPHERS;
+ }
+}
#endif
+
+static int sslize(struct sq_connection *conn, SSL_CTX *s, int (*func)(SSL *)) {
+ return (conn->ssl = SSL_new(s)) != NULL &&
SSL_set_fd(conn->ssl, conn->client.sock) == 1 &&
func(conn->ssl) == 1;
}
@@ -3928,6 +3934,14 @@ static int set_ssl_option(struct sq_context *ctx) {
// will be accepted but nothing will happen, i.e. renegotiation will
// not be prevented.
options |= SSL_OP_NO_RENEGOTIATION;
+#elif OPENSSL_VERSION_NUMBER < 0x10100000L
+ // IMPALA-11195: disable TLS/SSL renegotiation. In version 1.0.2 and prior it's
+ // possible to use the undocumented SSL3_FLAGS_NO_RENEGOTIATE_CIPHERS flag.
+ // We need to set the flag in the callback 'ssl_disable_renegotiation_cb' after
+ // handshake is done, otherwise the flag would get reset in SSL_accept().
+ SSL_CTX_set_info_callback(ctx_, ssl_disable_renegotiation_cb);
+#else
+ static_error(false, "Found SSL version that is vulnerable to CVE-2009-3555.");
#endif
if ((SSL_CTX_set_options(ctx->ssl_ctx, options) & options) != options) {
diff --git a/bin/impala-config.sh b/bin/impala-config.sh
index b10541439..7afa739f4 100755
--- a/bin/impala-config.sh
+++ b/bin/impala-config.sh
@@ -85,7 +85,7 @@ export USE_AVRO_CPP=${USE_AVRO_CPP:=false}
# moving to a different build of the toolchain, e.g. when a version is bumped or a
# compile option is changed. The build id can be found in the output of the toolchain
# build jobs, it is constructed from the build number and toolchain git hash prefix.
-export IMPALA_TOOLCHAIN_BUILD_ID=324-bb64c6ed91
+export IMPALA_TOOLCHAIN_BUILD_ID=351-c240088ecb
# Versions of toolchain dependencies.
# -----------------------------------
if $USE_AVRO_CPP; then
[impala] 02/03: [docs] Add note about REFRESH for tables in custom catalog
Posted by st...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
stigahuang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
commit 240a9cddc2ae7cbcdbe3d1da569af7fb7e8c5859
Author: Zoltan Borok-Nagy <bo...@cloudera.com>
AuthorDate: Tue Aug 8 12:32:26 2023 +0200
[docs] Add note about REFRESH for tables in custom catalog
This patch adds a small note about Iceberg tables in custom catalog.
Change-Id: I909603dff20fc4fefde48539d301844d40167a57
Reviewed-on: http://gerrit.cloudera.org:8080/20328
Tested-by: Impala Public Jenkins <im...@cloudera.com>
Reviewed-by: Andrew Sherman <as...@cloudera.com>
---
docs/topics/impala_iceberg.xml | 2 ++
1 file changed, 2 insertions(+)
diff --git a/docs/topics/impala_iceberg.xml b/docs/topics/impala_iceberg.xml
index 871b1ba56..1504efcdb 100644
--- a/docs/topics/impala_iceberg.xml
+++ b/docs/topics/impala_iceberg.xml
@@ -140,6 +140,8 @@ TBLPROPERTIES('iceberg.catalog'='hadoop.catalog',
Alternatively, you can also use custom catalogs to use existing tables. It means you need to define
your catalog in hive-site.xml.
The advantage of this method is that other engines are more likely to be able to interact with this table.
+ Please note that the automatic metadata update will not work for these tables, you will have to manually
+ call REFRESH on the table when it changes outside Impala.
To globally register different catalogs, set the following Hadoop configurations:
<table rowsep="1" colsep="1" id="iceberg_custom_catalogs">
<tgroup cols="2">