You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@impala.apache.org by st...@apache.org on 2023/12/14 04:54:30 UTC
(impala) branch master updated: IMPALA-11501: Add flag to allow catalog-cache operations on masked tables
This is an automated email from the ASF dual-hosted git repository.
stigahuang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
The following commit(s) were added to refs/heads/master by this push:
new 4cdb6b225 IMPALA-11501: Add flag to allow catalog-cache operations on masked tables
4cdb6b225 is described below
commit 4cdb6b2250a2769c9954b0e23549640243a03002
Author: stiga-huang <hu...@gmail.com>
AuthorDate: Thu Nov 30 19:12:42 2023 +0800
IMPALA-11501: Add flag to allow catalog-cache operations on masked tables
REFRESH/INVALIDATE METADATA <table> are the table level catalog-cache
operations. In Hive-Ranger plugin, when a table is masked (either by
column-masking or row-filtering policy) for a user, the user is unable
to perform any modification (insert/delete/update) on the table, i.e.
it's considered a read-only user (RANGER-1087, RANGER-1100). However,
Hive doesn't have these catalog-cache operations. It's a grey area
whether they should be blocked.
Before this patch, these catalog-cache operations are considered as
modifications on the table so they are also blocked for masked users.
Table metadata is required to be loaded so we have the column names to
fetch Ranger column masking policies. This causes a performance
regression on INVALIDATE METADATA <table> commands since in older
versions (e.g. CDH), IM commands don't need to load the table metadata
and runs pretty fast.
This patch adds a flag, allow_catalog_cache_op_from_masked_users, for
coordinators to skip checking masking policies for such statements. When
this flag is on, coordinators don't need to load the table metadata thus
fix the performance regression as well.
Note that Ranger ownership based policies can't be applied correctly
when the table is unloaded (so the owner is unknown). REFRESH/INVALIDATE
METADATA <table> commands could be denied on owners even if there are
Ranger policies allowing the owner's operations. This is a known issue
since IMPALA-8228. To ensure a user can always perform these operations,
grant REFRESH privilege to them to get rid of the unloaded table issue.
This patch also fixes a bug in local catalog mode which only occurs
after adding the new flag. The bug is that LocalDb#getTableIfCached()
doesn't make good use of the cache. If the table meta is cahced but
LocalDb#getTable() hasn't been invoked on the table, getTableIfCached()
will always return a LocalIncompleteTable which is missing some table
info, e.g. ownership. This causes REFRESH/INVALIDATE statements not able
to pass the ownership context to RangerAccessResourceImpl so ownership
policies can't be correctly applied.
Ideally, LocalDb#getTableIfCached() should return a LocalTable instance
if the table is cached. However, in local catalog mode, we don't cache
everything that constructs a LocalTable instance. Constructing a
LocalTable instance might still trigger external RPCs which should be
avoided. As an alternative, this patch checks if the msTable is cached.
If it's cached, add it to the LocalIncompleteTable instance so most of
the table info can be retrieved, including the ownership string.
Tests:
- Add e2e tests on both the legacy and local catalog mode.
Change-Id: I45935654cbf05a55d740f1b04781022c271f7678
Reviewed-on: http://gerrit.cloudera.org:8080/20742
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
be/src/service/frontend.cc | 6 ++
be/src/util/backend-gflag-util.cc | 3 +
common/thrift/BackendGflags.thrift | 2 +
.../apache/impala/analysis/StmtMetadataLoader.java | 7 +-
.../authorization/BaseAuthorizationChecker.java | 11 ++-
.../org/apache/impala/authorization/Privilege.java | 7 +-
.../main/java/org/apache/impala/catalog/Table.java | 5 +-
.../impala/catalog/local/CatalogdMetaProvider.java | 14 ++++
.../impala/catalog/local/DirectMetaProvider.java | 10 +++
.../org/apache/impala/catalog/local/LocalDb.java | 17 ++++-
.../impala/catalog/local/LocalIncompleteTable.java | 9 +++
.../apache/impala/catalog/local/LocalTable.java | 19 +++--
.../apache/impala/catalog/local/MetaProvider.java | 2 +
.../org/apache/impala/service/BackendConfig.java | 4 ++
tests/authorization/test_ranger.py | 84 +++++++++++++++++++++-
15 files changed, 188 insertions(+), 12 deletions(-)
diff --git a/be/src/service/frontend.cc b/be/src/service/frontend.cc
index 40f23ce24..e8f53f3d7 100644
--- a/be/src/service/frontend.cc
+++ b/be/src/service/frontend.cc
@@ -90,6 +90,12 @@ DEFINE_string(jni_frontend_class, "org/apache/impala/service/JniFrontend", "By d
"overridden class needs to contain all the methods in the methods[] variable, so "
"most implementations should make their class a child of JniFrontend and "
"override only relevant methods.");
+DEFINE_bool(allow_catalog_cache_op_from_masked_users, false, "Whether to allow table "
+ "level catalog-cache operations, i.e. REFRESH/INVALIDATE METADATA <table>, from users"
+ " that have associate Ranger masking policies on the table. By default, such "
+ "operations are blocked since such users are considered read-only users. Note that "
+ "checking column masking policies requires loading column info of the table, which "
+ "could slow down simple commands like INVALIDATE METADATA <table>");
Frontend::Frontend() {
JniMethodDescriptor methods[] = {
diff --git a/be/src/util/backend-gflag-util.cc b/be/src/util/backend-gflag-util.cc
index 6ddbff9a8..adaa47ac5 100644
--- a/be/src/util/backend-gflag-util.cc
+++ b/be/src/util/backend-gflag-util.cc
@@ -113,6 +113,7 @@ DECLARE_bool(enable_skipping_older_events);
DECLARE_bool(enable_json_scanner);
DECLARE_int32(catalog_operation_log_size);
DECLARE_string(hostname);
+DECLARE_bool(allow_catalog_cache_op_from_masked_users);
// HS2 SAML2.0 configuration
// Defined here because TAG_FLAG caused issues in global-flags.cc
@@ -444,6 +445,8 @@ Status PopulateThriftBackendGflags(TBackendGflags& cfg) {
FLAGS_max_filter_error_rate_from_full_scan);
cfg.__set_catalog_operation_log_size(FLAGS_catalog_operation_log_size);
cfg.__set_hostname(FLAGS_hostname);
+ cfg.__set_allow_catalog_cache_op_from_masked_users(
+ FLAGS_allow_catalog_cache_op_from_masked_users);
return Status::OK();
}
diff --git a/common/thrift/BackendGflags.thrift b/common/thrift/BackendGflags.thrift
index 4b0ea6ce5..0f7818267 100644
--- a/common/thrift/BackendGflags.thrift
+++ b/common/thrift/BackendGflags.thrift
@@ -272,4 +272,6 @@ struct TBackendGflags {
120: required i32 catalog_operation_log_size
121: required string hostname
+
+ 122: required bool allow_catalog_cache_op_from_masked_users
}
diff --git a/fe/src/main/java/org/apache/impala/analysis/StmtMetadataLoader.java b/fe/src/main/java/org/apache/impala/analysis/StmtMetadataLoader.java
index 7bcdb81d2..e4dfa7752 100644
--- a/fe/src/main/java/org/apache/impala/analysis/StmtMetadataLoader.java
+++ b/fe/src/main/java/org/apache/impala/analysis/StmtMetadataLoader.java
@@ -39,6 +39,7 @@ import org.apache.impala.catalog.Table;
import org.apache.impala.common.AnalysisException;
import org.apache.impala.common.InternalException;
import org.apache.impala.compat.MetastoreShim;
+import org.apache.impala.service.BackendConfig;
import org.apache.impala.service.Frontend;
import org.apache.impala.thrift.TUniqueId;
import org.apache.impala.util.AcidUtils;
@@ -352,10 +353,12 @@ public class StmtMetadataLoader {
List<TableRef> tblRefs = new ArrayList<>();
// The information about whether table masking is supported is not available to
// ResetMetadataStmt so we collect the TableRef for ResetMetadataStmt whenever
- // applicable.
+ // applicable. Skip this if allow_catalog_cache_op_from_masked_users=true because
+ // we don't need column info for fetching column-masking policies.
if (stmt instanceof ResetMetadataStmt
&& fe_.getAuthzFactory().getAuthorizationConfig().isEnabled()
- && fe_.getAuthzFactory().supportsTableMasking()) {
+ && fe_.getAuthzFactory().supportsTableMasking()
+ && !BackendConfig.INSTANCE.allowCatalogCacheOpFromMaskedUsers()) {
TableName tableName = ((ResetMetadataStmt) stmt).getTableName();
if (tableName != null) tblRefs.add(new TableRef(tableName.toPath(), null));
} else {
diff --git a/fe/src/main/java/org/apache/impala/authorization/BaseAuthorizationChecker.java b/fe/src/main/java/org/apache/impala/authorization/BaseAuthorizationChecker.java
index 1ab05cf09..6badb909b 100644
--- a/fe/src/main/java/org/apache/impala/authorization/BaseAuthorizationChecker.java
+++ b/fe/src/main/java/org/apache/impala/authorization/BaseAuthorizationChecker.java
@@ -28,6 +28,7 @@ import org.apache.impala.catalog.FeIncompleteTable;
import org.apache.impala.catalog.FeTable;
import org.apache.impala.common.InternalException;
import org.apache.impala.common.Pair;
+import org.apache.impala.service.BackendConfig;
import org.apache.impala.util.EventSequence;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -215,8 +216,16 @@ public abstract class BaseAuthorizationChecker implements AuthorizationChecker {
return;
}
// Populate column names to check column masking policies in blocking updates.
+ // No need to do this for REFRESH if allow_catalog_cache_op_from_masked_users=true.
+ // Note that db.getTable() could be a heavy operation in local catalog mode since it
+ // triggers metadata loading on the table if it's unloaded in catalogd. Skipping this
+ // improves the performance of "INVALIDATE METADATA <table>" statements. For REFRESH
+ // statements, the performance doesn't differ a lot since there are other places that
+ // use db.getTable() (see IMPALA-12591).
if (config_.isEnabled() && request.getAuthorizable() != null
- && request.getAuthorizable().getType() == Type.TABLE) {
+ && request.getAuthorizable().getType() == Type.TABLE
+ && (request.getPrivilege() != Privilege.REFRESH
+ || !BackendConfig.INSTANCE.allowCatalogCacheOpFromMaskedUsers())) {
Preconditions.checkNotNull(dbName);
AuthorizableTable authorizableTable = (AuthorizableTable) request.getAuthorizable();
FeDb db = catalog.getDb(dbName);
diff --git a/fe/src/main/java/org/apache/impala/authorization/Privilege.java b/fe/src/main/java/org/apache/impala/authorization/Privilege.java
index f5177874a..d0d850a4f 100644
--- a/fe/src/main/java/org/apache/impala/authorization/Privilege.java
+++ b/fe/src/main/java/org/apache/impala/authorization/Privilege.java
@@ -18,6 +18,7 @@
package org.apache.impala.authorization;
import com.google.common.base.Preconditions;
+import org.apache.impala.service.BackendConfig;
import java.util.EnumSet;
@@ -92,7 +93,11 @@ public enum Privilege {
* Returns true if this implies modification on data or metadata.
*/
public boolean impliesUpdate() {
+ // When allow_catalog_cache_op_from_masked_users=false, REFRESH is considered as
+ // an update operation.
+ boolean considerCatalogCacheOp =
+ !BackendConfig.INSTANCE.allowCatalogCacheOpFromMaskedUsers();
return this == ALTER || this == DROP || this == CREATE || this == INSERT
- || this == REFRESH || this == ALL;
+ || (this == REFRESH && considerCatalogCacheOp) || this == ALL;
}
}
diff --git a/fe/src/main/java/org/apache/impala/catalog/Table.java b/fe/src/main/java/org/apache/impala/catalog/Table.java
index b42874beb..eb8a863bd 100644
--- a/fe/src/main/java/org/apache/impala/catalog/Table.java
+++ b/fe/src/main/java/org/apache/impala/catalog/Table.java
@@ -895,7 +895,10 @@ public abstract class Table extends CatalogObjectImpl implements FeTable {
@Override // FeTable
public String getOwnerUser() {
- if (msTable_ == null) return null;
+ if (msTable_ == null) {
+ LOG.warn("Owner of {} is unknown due to table is unloaded", getFullName());
+ return null;
+ }
return msTable_.getOwner();
}
diff --git a/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java b/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java
index 1e10169c0..cbfbfbf96 100644
--- a/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java
+++ b/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java
@@ -742,6 +742,20 @@ public class CatalogdMetaProvider implements MetaProvider {
return req;
}
+ @Override
+ public Pair<Table, TableMetaRef> getTableIfPresent(String dbName, String tblName) {
+ TableCacheKey cacheKey = new TableCacheKey(dbName.toLowerCase(),
+ tblName.toLowerCase());
+ try {
+ Object value = getIfPresent(cacheKey);
+ if (value == null) return null;
+ TableMetaRefImpl ref = (TableMetaRefImpl) value;
+ return Pair.create(ref.msTable_, ref);
+ } catch (TException e) {
+ return null;
+ }
+ }
+
@Override
public Pair<Table, TableMetaRef> loadTable(final String dbName, final String tableName)
throws NoSuchObjectException, MetaException, TException {
diff --git a/fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java b/fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java
index 6fa3f098e..58632a397 100644
--- a/fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java
+++ b/fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java
@@ -148,6 +148,16 @@ class DirectMetaProvider implements MetaProvider {
}
}
+ @Override
+ public Pair<Table, TableMetaRef> getTableIfPresent(String dbName, String tblName) {
+ try {
+ return loadTable(dbName, tblName);
+ } catch (TException e) {
+ LOG.error("Failed to load table", e);
+ return null;
+ }
+ }
+
@Override
public Pair<Table, TableMetaRef> loadTable(String dbName, String tableName)
throws MetaException, NoSuchObjectException, TException {
diff --git a/fe/src/main/java/org/apache/impala/catalog/local/LocalDb.java b/fe/src/main/java/org/apache/impala/catalog/local/LocalDb.java
index 7c50b414b..a12e4aa9e 100644
--- a/fe/src/main/java/org/apache/impala/catalog/local/LocalDb.java
+++ b/fe/src/main/java/org/apache/impala/catalog/local/LocalDb.java
@@ -39,6 +39,7 @@ import org.apache.impala.catalog.Function.CompareMode;
import org.apache.impala.catalog.TableLoadingException;
import org.apache.impala.common.ImpalaException;
import org.apache.impala.common.ImpalaRuntimeException;
+import org.apache.impala.common.Pair;
import org.apache.impala.thrift.TBriefTableMeta;
import org.apache.impala.thrift.TDatabase;
import org.apache.impala.thrift.TFunctionCategory;
@@ -51,6 +52,8 @@ import com.google.common.base.Preconditions;
import com.google.common.collect.ImmutableList;
import com.google.common.collect.Iterables;
import com.google.common.collect.Maps;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
/**
@@ -60,6 +63,7 @@ import com.google.common.collect.Maps;
* each catalog instance.
*/
public class LocalDb implements FeDb {
+ private final static Logger LOG = LoggerFactory.getLogger(LocalDb.class);
private final LocalCatalog catalog_;
/** The lower-case name of the database. */
private final String name_;
@@ -117,7 +121,18 @@ public class LocalDb implements FeDb {
// Table doesn't exist.
return null;
}
- return tables_.get(tblName);
+ FeTable tbl = tables_.get(tblName);
+ if (tbl instanceof LocalIncompleteTable && tbl.getMetaStoreTable() == null) {
+ // Add msTable if it's cached.
+ Pair<Table, MetaProvider.TableMetaRef> tblMeta =
+ catalog_.getMetaProvider().getTableIfPresent(name_, tblName);
+ if (tblMeta != null) {
+ tbl = new LocalIncompleteTable(this, tblMeta.first, tblMeta.second,
+ tbl.getTableType(), tbl.getTableComment());
+ tables_.put(tblName, tbl);
+ }
+ }
+ return tbl;
}
@Override
diff --git a/fe/src/main/java/org/apache/impala/catalog/local/LocalIncompleteTable.java b/fe/src/main/java/org/apache/impala/catalog/local/LocalIncompleteTable.java
index 3760b0421..e85bb3ac2 100644
--- a/fe/src/main/java/org/apache/impala/catalog/local/LocalIncompleteTable.java
+++ b/fe/src/main/java/org/apache/impala/catalog/local/LocalIncompleteTable.java
@@ -16,7 +16,9 @@
// under the License.
package org.apache.impala.catalog.local;
+import org.apache.hadoop.hive.metastore.api.Table;
import org.apache.impala.catalog.FeIncompleteTable;
+import org.apache.impala.catalog.local.MetaProvider.TableMetaRef;
import org.apache.impala.common.ImpalaException;
import org.apache.impala.thrift.TBriefTableMeta;
import org.apache.impala.thrift.TImpalaTableType;
@@ -42,6 +44,13 @@ public class LocalIncompleteTable extends LocalTable implements FeIncompleteTabl
tableComment_ = tableMeta.getComment();
}
+ public LocalIncompleteTable(LocalDb db, Table msTbl, TableMetaRef ref,
+ @Nullable TImpalaTableType tableType, @Nullable String tableComment) {
+ super(db, msTbl, ref);
+ this.tableType_ = tableType;
+ this.tableComment_ = tableComment;
+ }
+
@Override
public TImpalaTableType getTableType() { return tableType_; }
diff --git a/fe/src/main/java/org/apache/impala/catalog/local/LocalTable.java b/fe/src/main/java/org/apache/impala/catalog/local/LocalTable.java
index 62731ec4b..f2180b13a 100644
--- a/fe/src/main/java/org/apache/impala/catalog/local/LocalTable.java
+++ b/fe/src/main/java/org/apache/impala/catalog/local/LocalTable.java
@@ -54,13 +54,14 @@ import org.apache.impala.thrift.TCatalogObjectType;
import org.apache.impala.thrift.TImpalaTableType;
import org.apache.impala.thrift.TTableStats;
import org.apache.impala.util.AcidUtils;
-import org.apache.log4j.Logger;
import org.apache.thrift.TException;
import com.google.common.base.Preconditions;
import com.google.common.collect.ImmutableList;
import com.google.common.collect.ImmutableMap;
import com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
/**
* Table instance loaded from {@link LocalCatalog}.
@@ -69,7 +70,7 @@ import com.google.common.collect.Lists;
* each catalog instance.
*/
abstract class LocalTable implements FeTable {
- private static final Logger LOG = Logger.getLogger(LocalTable.class);
+ private static final Logger LOG = LoggerFactory.getLogger(LocalTable.class);
protected final LocalDb db_;
/** The lower-case name of the table. */
@@ -100,10 +101,15 @@ abstract class LocalTable implements FeTable {
protected final TableMetaRef ref_;
public static LocalTable load(LocalDb db, String tblName) throws TableLoadingException {
+ Pair<Table, TableMetaRef> tableMeta = loadTableMetadata(db, tblName);
+ return load(db, tableMeta);
+ }
+
+ public static LocalTable load(LocalDb db, Pair<Table, TableMetaRef> tableMeta)
+ throws TableLoadingException {
// In order to know which kind of table subclass to instantiate, we need
// to eagerly grab and parse the top-level Table object from the HMS.
LocalTable t = null;
- Pair<Table, TableMetaRef> tableMeta = loadTableMetadata(db, tblName);
Table msTbl = tableMeta.first;
TableMetaRef ref = tableMeta.second;
if (TableType.valueOf(msTbl.getTableType()) == TableType.VIRTUAL_VIEW) {
@@ -123,7 +129,7 @@ abstract class LocalTable implements FeTable {
if (t == null) {
throw new LocalCatalogException("Unknown table type for table " +
- db.getName() + "." + tblName);
+ db.getName() + "." + msTbl.getTableName());
}
// TODO(todd): it would be preferable to only load stats for those columns
@@ -201,7 +207,10 @@ abstract class LocalTable implements FeTable {
@Override
public String getOwnerUser() {
- if (msTable_ == null) return null;
+ if (msTable_ == null) {
+ LOG.warn("Owner of {} is unknown due to msTable is unloaded", getFullName());
+ return null;
+ }
return msTable_.getOwner();
}
diff --git a/fe/src/main/java/org/apache/impala/catalog/local/MetaProvider.java b/fe/src/main/java/org/apache/impala/catalog/local/MetaProvider.java
index a9d8c6671..b8770e6ac 100644
--- a/fe/src/main/java/org/apache/impala/catalog/local/MetaProvider.java
+++ b/fe/src/main/java/org/apache/impala/catalog/local/MetaProvider.java
@@ -78,6 +78,8 @@ public interface MetaProvider {
Pair<Table, TableMetaRef> loadTable(String dbName, String tableName)
throws NoSuchObjectException, MetaException, TException;
+ Pair<Table, TableMetaRef> getTableIfPresent(String dbName, String tableName);
+
String loadNullPartitionKeyValue()
throws MetaException, TException;
diff --git a/fe/src/main/java/org/apache/impala/service/BackendConfig.java b/fe/src/main/java/org/apache/impala/service/BackendConfig.java
index 8456bfd3d..fb776dcaf 100644
--- a/fe/src/main/java/org/apache/impala/service/BackendConfig.java
+++ b/fe/src/main/java/org/apache/impala/service/BackendConfig.java
@@ -445,4 +445,8 @@ public class BackendConfig {
public String getHostname() {
return backendCfg_.hostname;
}
+
+ public boolean allowCatalogCacheOpFromMaskedUsers() {
+ return backendCfg_.allow_catalog_cache_op_from_masked_users;
+ }
}
diff --git a/tests/authorization/test_ranger.py b/tests/authorization/test_ranger.py
index db117004f..d681e77aa 100644
--- a/tests/authorization/test_ranger.py
+++ b/tests/authorization/test_ranger.py
@@ -892,7 +892,9 @@ class TestRanger(CustomClusterTestSuite):
r = requests.post("{0}/service/public/v2/api/policy".format(RANGER_HOST),
auth=RANGER_AUTH, json=data, headers=REST_HEADERS)
assert 300 > r.status_code >= 200, r.content
- return json.loads(r.content)["id"]
+ policy_id = json.loads(r.content)["id"]
+ LOG.info("Added column masking policy " + str(policy_id))
+ return policy_id
@staticmethod
def _add_row_filtering_policy(policy_name, user, db, table, filter_expr):
@@ -1563,6 +1565,86 @@ class TestRanger(CustomClusterTestSuite):
TestRanger._remove_policy(unique_name)
admin_client.execute("revoke all on server from user {0}".format(user))
+ @pytest.mark.execute_serially
+ @CustomClusterTestSuite.with_args(
+ impalad_args="{0} {1}".format(IMPALAD_ARGS,
+ "--allow_catalog_cache_op_from_masked_users=true"),
+ catalogd_args=CATALOGD_ARGS)
+ def test_allow_metadata_update(self, unique_name):
+ self.__test_allow_catalog_cache_op_from_masked_users(unique_name)
+
+ @pytest.mark.execute_serially
+ @CustomClusterTestSuite.with_args(
+ impalad_args="{0} {1}".format(LOCAL_CATALOG_IMPALAD_ARGS,
+ "--allow_catalog_cache_op_from_masked_users=true"),
+ catalogd_args=LOCAL_CATALOG_CATALOGD_ARGS)
+ def test_allow_metadata_update_local_catalog(self, unique_name):
+ self.__test_allow_catalog_cache_op_from_masked_users(unique_name)
+
+ def __test_allow_catalog_cache_op_from_masked_users(self, unique_name):
+ """Verify that catalog cache operations are allowed for masked users
+ when allow_catalog_cache_op_from_masked_users=true."""
+ user = getuser()
+ admin_client = self.create_impala_client()
+ non_admin_client = self.create_impala_client()
+ try:
+ # Create a column masking policy on 'user' which is also the owner of the table
+ TestRanger._add_column_masking_policy(
+ unique_name, user, "functional", "alltypestiny", "id",
+ "CUSTOM", "id * 100")
+
+ # At a cold start, the table is unloaded so its owner is unknown.
+ # INVALIDATE METADATA <table> is denied since 'user' is not detected as the owner.
+ result = self.execute_query_expect_failure(
+ non_admin_client, "invalidate metadata functional.alltypestiny", user=user)
+ assert "User '{0}' does not have privileges to execute " \
+ "'INVALIDATE METADATA/REFRESH' on: functional.alltypestiny".format(user) \
+ in str(result)
+ # Verify catalogd never loads metadata of this table
+ table_loaded_log = "Loaded metadata for: functional.alltypestiny"
+ self.assert_catalogd_log_contains("INFO", table_loaded_log, expected_count=0)
+
+ # Run a query to trigger metadata loading on the table
+ self.execute_query_expect_success(
+ non_admin_client, "describe functional.alltypestiny", user=user)
+ # Verify catalogd loads metadata of this table
+ self.assert_catalogd_log_contains("INFO", table_loaded_log, expected_count=1)
+
+ # INVALIDATE METADATA <table> is allowed since 'user' is detected as the owner.
+ self.execute_query_expect_success(
+ non_admin_client, "invalidate metadata functional.alltypestiny", user=user)
+
+ # Run a query to trigger metadata loading on the table
+ self.execute_query_expect_success(
+ non_admin_client, "describe functional.alltypestiny", user=user)
+ # Verify catalogd loads metadata of this table
+ self.assert_catalogd_log_contains("INFO", table_loaded_log, expected_count=2)
+
+ # Verify REFRESH <table> is allowed since 'user' is detected as the owner.
+ self.execute_query_expect_success(
+ non_admin_client, "refresh functional.alltypestiny", user=user)
+ self.execute_query_expect_success(
+ non_admin_client,
+ "refresh functional.alltypestiny partition(year=2009, month=1)", user=user)
+
+ # Clear the catalog cache and grant 'user' enough privileges
+ self.execute_query_expect_success(
+ admin_client, "invalidate metadata functional.alltypestiny", user=ADMIN)
+ admin_client.execute("grant refresh on table functional.alltypestiny to user {0}"
+ .format(user), user=ADMIN)
+ try:
+ # Now 'user' should be able to run REFRESH/INVALIDATE even if the table is
+ # unloaded (not recognize it as the owner).
+ self.execute_query_expect_success(
+ non_admin_client, "invalidate metadata functional.alltypestiny", user=user)
+ self.execute_query_expect_success(
+ non_admin_client, "refresh functional.alltypestiny", user=user)
+ finally:
+ admin_client.execute(
+ "revoke refresh on table functional.alltypestiny from user {0}".format(user))
+ finally:
+ TestRanger._remove_policy(unique_name)
+
@pytest.mark.execute_serially
@CustomClusterTestSuite.with_args(
impalad_args=IMPALAD_ARGS, catalogd_args=CATALOGD_ARGS)