You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@impala.apache.org by st...@apache.org on 2021/03/22 14:25:30 UTC
[impala] branch master updated (98de1c5 -> c9d7bcb)
This is an automated email from the ASF dual-hosted git repository.
stigahuang pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git.
from 98de1c5 IMPALA-9234: Support Ranger row filtering policies
new 879986e IMPALA-10552: Support external frontends supplying timeline for profile
new c9d7bcb IMPALA-9661: Avoid introducing unused columns in table masking view
The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
be/src/service/client-request-state.cc | 11 +-
be/src/service/impala-hs2-server.cc | 10 +-
be/src/service/impala-server.cc | 16 ++-
common/thrift/Query.thrift | 5 +-
.../org/apache/impala/analysis/AlterViewStmt.java | 2 -
.../apache/impala/analysis/AnalysisContext.java | 62 +++++----
.../java/org/apache/impala/analysis/Analyzer.java | 90 ++++++++-----
.../impala/analysis/CreateTableAsSelectStmt.java | 5 +
.../org/apache/impala/analysis/CreateViewStmt.java | 2 -
.../main/java/org/apache/impala/analysis/Expr.java | 10 ++
.../org/apache/impala/analysis/FromClause.java | 96 +++++---------
.../org/apache/impala/analysis/InlineViewRef.java | 25 ++--
.../org/apache/impala/analysis/InsertStmt.java | 8 ++
.../java/org/apache/impala/analysis/QueryStmt.java | 15 ++-
.../org/apache/impala/analysis/SelectStmt.java | 27 +++-
.../apache/impala/analysis/SetOperationStmt.java | 6 +-
.../java/org/apache/impala/analysis/SlotRef.java | 5 +
.../java/org/apache/impala/analysis/StmtNode.java | 10 ++
.../java/org/apache/impala/analysis/Subquery.java | 12 ++
.../java/org/apache/impala/analysis/TableRef.java | 15 +++
.../org/apache/impala/analysis/WithClause.java | 9 ++
.../org/apache/impala/authorization/TableMask.java | 31 +++--
.../apache/impala/analysis/AnalyzeStmtsTest.java | 2 +-
.../authorization/ranger/RangerAuditLogTest.java | 147 ++++++++++++---------
.../ranger_column_masking_and_row_filtering.test | 52 +++++++-
.../queries/QueryTest/ranger_row_filtering.test | 118 ++++++++++++++++-
tests/authorization/test_ranger.py | 6 +-
27 files changed, 547 insertions(+), 250 deletions(-)
[impala] 01/02: IMPALA-10552: Support external frontends supplying
timeline for profile
Posted by st...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
stigahuang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
commit 879986ea6f478cfa35f78249d25441fc6d15cc3d
Author: John Sherman <jf...@cloudera.com>
AuthorDate: Mon Jul 20 20:27:02 2020 +0000
IMPALA-10552: Support external frontends supplying timeline for profile
- Add EXTERNAL_FRONTEND as a client session type
- Use EXTERNAL_FRONTEND session type for clients connected to
external frontend interface.
- Rename Query Timeline to Impala Backend Timeline for external
frontends
- the query timeline is no longer an end to end timeline when
executing a plan from an external frontend
- External frontends can provide timeline information through a
TExecRequest by filling in the timeline field with a valid
TEventSequence
- The frontend timeline and backend timeline are completely separate
entities, meaning there is no overall attempt to capture the timing
end to end
- This is due to the fact that the frontend and Impala may not share
the same time source (or even machine).
- It is safe to add together the backend + frontend timeline times
to get a rough idea how long the query took end to end to execute,
but keep in mind that this number does not capture the time it
took the frontend to send the plan to the backend (Impala) nor does
it capture how long it took the end user to read the results.
Example timeline with external frontend:
Frontend Timeline: 3s016ms
- Analysis finished: 1s130ms (1s130ms)
- Calcite plan generated: 2s170ms (1s040ms)
- Metadata load started: 2s245ms (74.486ms)
- Metadata load finished. loaded-tables=1: 2s654ms (409.847ms)
- Single node plan created: 2s726ms (71.659ms)
- Runtime filters computed: 2s756ms (30.000ms)
- Distributed plan created: 2s761ms (5.265ms)
- Execution request created: 2s890ms (128.387ms)
- Impala plan generated: 2s891ms (1.508ms)
- Planning finished: 2s893ms (1.894ms)
- Submitted query: 3s016ms (122.377ms)
Impala Backend Timeline: 79.998ms
- Query submitted: 0.000ns (0.000ns)
- Submit for admission: 0.000ns (0.000ns)
- Completed admission: 0.000ns (0.000ns)
- Ready to start on 1 backends: 3.999ms (3.999ms)
- All 1 execution backends (2 fragment instances) started: 7.999ms (3.999ms)
- Rows available: 55.999ms (47.999ms)
- Execution cancelled: 79.998ms (23.999ms)
- Released admission control resources: 79.998ms (0.000ns)
- Unregister query: 79.998ms (0.000ns)
Testing done:
- Manual inspection of profiles on the Impala web UI
- test_hs2.py
- test_tpch_queries.py
- test_tpcds_queries.py::TestTpcdsDecimalV2Query
Co-authored-by: Kurt Deschler <kd...@cloudera.com>
Change-Id: I2b3692b4118ea23c0f9f8ec4bcc27b0b68bb32ec
Reviewed-on: http://gerrit.cloudera.org:8080/17183
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
be/src/service/client-request-state.cc | 11 +++++++++--
be/src/service/impala-hs2-server.cc | 10 +++++++---
be/src/service/impala-server.cc | 16 ++++++++++------
common/thrift/Query.thrift | 5 ++++-
4 files changed, 30 insertions(+), 12 deletions(-)
diff --git a/be/src/service/client-request-state.cc b/be/src/service/client-request-state.cc
index 23b446f..e5a31cf 100644
--- a/be/src/service/client-request-state.cc
+++ b/be/src/service/client-request-state.cc
@@ -130,14 +130,21 @@ ClientRequestState::ClientRequestState(const TQueryCtx& query_ctx, Frontend* fro
num_rows_fetched_from_cache_counter_ =
ADD_COUNTER(server_profile_, "NumRowsFetchedFromCache", TUnit::UNIT);
client_wait_timer_ = ADD_TIMER(server_profile_, "ClientFetchWaitTimer");
- query_events_ = summary_profile_->AddEventSequence("Query Timeline");
+ bool is_external_fe = session_type() == TSessionType::EXTERNAL_FRONTEND;
+ // "Impala Backend Timeline" was specifically chosen to exploit the lexicographical
+ // ordering defined by the underlying std::map holding the timelines displayed in
+ // the web UI. This helps ensure that "Frontend Timeline" is displayed before
+ // "Impala Backend Timeline".
+ query_events_ = summary_profile_->AddEventSequence(
+ is_external_fe ? "Impala Backend Timeline" : "Query Timeline");
query_events_->Start();
profile_->AddChild(summary_profile_);
profile_->set_name("Query (id=" + PrintId(query_id()) + ")");
summary_profile_->AddInfoString("Session ID", PrintId(session_id()));
summary_profile_->AddInfoString("Session Type", PrintThriftEnum(session_type()));
- if (session_type() == TSessionType::HIVESERVER2) {
+ if (session_type() == TSessionType::HIVESERVER2 ||
+ session_type() == TSessionType::EXTERNAL_FRONTEND) {
summary_profile_->AddInfoString("HiveServer2 Protocol Version",
Substitute("V$0", 1 + session->hs2_version));
}
diff --git a/be/src/service/impala-hs2-server.cc b/be/src/service/impala-hs2-server.cc
index 2f6858b..15dd523 100644
--- a/be/src/service/impala-hs2-server.cc
+++ b/be/src/service/impala-hs2-server.cc
@@ -309,7 +309,13 @@ void ImpalaServer::OpenSession(TOpenSessionResp& return_val,
std::make_shared<SessionState>(this, session_id, secret);
state->closed = false;
state->start_time_ms = UnixMillis();
- state->session_type = TSessionType::HIVESERVER2;
+ const ThriftServer::ConnectionContext* connection_context =
+ ThriftServer::GetThreadConnectionContext();
+ if (connection_context->server_name == EXTERNAL_FRONTEND_SERVER_NAME) {
+ state->session_type = TSessionType::EXTERNAL_FRONTEND;
+ } else {
+ state->session_type = TSessionType::HIVESERVER2;
+ }
state->network_address = ThriftServer::GetThreadConnectionContext()->network_address;
state->last_accessed_ms = UnixMillis();
// request.client_protocol is not guaranteed to be a valid TProtocolVersion::type, so
@@ -323,8 +329,6 @@ void ImpalaServer::OpenSession(TOpenSessionResp& return_val,
state->kudu_latest_observed_ts = 0;
// If the username was set by a lower-level transport, use it.
- const ThriftServer::ConnectionContext* connection_context =
- ThriftServer::GetThreadConnectionContext();
if (!connection_context->username.empty()) {
state->connected_user = connection_context->username;
if (!connection_context->do_as_user.empty()) {
diff --git a/be/src/service/impala-server.cc b/be/src/service/impala-server.cc
index f9c50bd..b460d25 100644
--- a/be/src/service/impala-server.cc
+++ b/be/src/service/impala-server.cc
@@ -1188,14 +1188,14 @@ Status ImpalaServer::ExecuteInternal(const TQueryCtx& query_ctx,
TUniqueId query_id = (*query_handle)->query_id();
// Generate TExecRequest here if one was not passed in or we want one
// from the Impala planner to compare with
- if (external_exec_request == nullptr || !FLAGS_dump_exec_request_path.empty()) {
- // Takes the TQueryCtx and calls into the frontend to initialize the
- // TExecRequest for this query.
+ if (!is_external_req || !FLAGS_dump_exec_request_path.empty()) {
+ // Takes the TQueryCtx and calls into the frontend to initialize the TExecRequest
+ // for this query.
RETURN_IF_ERROR(query_handle->query_driver()->RunFrontendPlanner(query_ctx));
DumpTExecReq((*query_handle)->exec_request(), "internal", query_id);
}
- if (external_exec_request != nullptr) {
+ if (is_external_req) {
// Use passed in exec_request
RETURN_IF_ERROR(query_handle->query_driver()->SetExternalPlan(
query_ctx, *external_exec_request));
@@ -1209,7 +1209,10 @@ Status ImpalaServer::ExecuteInternal(const TQueryCtx& query_ctx,
}
const TExecRequest& result = (*query_handle)->exec_request();
- (*query_handle)->query_events()->MarkEvent("Planning finished");
+ // If this is an external request, planning was done by the external frontend
+ if (!is_external_req) {
+ (*query_handle)->query_events()->MarkEvent("Planning finished");
+ }
(*query_handle)->set_user_profile_access(result.user_has_profile_access);
(*query_handle)->summary_profile()->AddEventSequence(
result.timeline.name, result.timeline);
@@ -2513,7 +2516,8 @@ void ImpalaServer::UnregisterSessionTimeout(int32_t session_timeout) {
>= FLAGS_disconnected_session_timeout * 1000L) {
// This session has no active connections and is past the disconnected session
// timeout, so close it.
- DCHECK_ENUM_EQ(session_state->session_type, TSessionType::HIVESERVER2);
+ DCHECK(session_state->session_type == TSessionType::HIVESERVER2 ||
+ session_state->session_type == TSessionType::EXTERNAL_FRONTEND);
LOG(INFO) << "Closing session: " << PrintId(session_id)
<< ", user: " << session_state->connected_user
<< ", because it no longer has any open connections. The last "
diff --git a/common/thrift/Query.thrift b/common/thrift/Query.thrift
index 31c0e13..ce0d654 100644
--- a/common/thrift/Query.thrift
+++ b/common/thrift/Query.thrift
@@ -480,10 +480,13 @@ struct TQueryOptions {
PlanNodes.TMinmaxFilteringLevel.ROW_GROUP;
}
-// Impala currently has two types of sessions: Beeswax and HiveServer2
+// Impala currently has three types of sessions: Beeswax, HiveServer2 and external
+// frontend. External frontend is a variation of HiveServer2 to support external
+// planning.
enum TSessionType {
BEESWAX = 0
HIVESERVER2 = 1
+ EXTERNAL_FRONTEND = 2
}
// Client request including stmt to execute and query options.
[impala] 02/02: IMPALA-9661: Avoid introducing unused columns in
table masking view
Posted by st...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
stigahuang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
commit c9d7bcb4a1e77deaa431b152b0833e46cb267239
Author: stiga-huang <hu...@gmail.com>
AuthorDate: Thu Mar 18 21:20:58 2021 +0800
IMPALA-9661: Avoid introducing unused columns in table masking view
Previously, if a table has column masking policies, we replace its
unanalyzed TableRef with an analyzed InlineViewRef (table masking view)
in FromClause.analyze(). However, we can't detect which columns are
actually used in the original query at this point. In fact, analyze()
for SelectList, WhereClause, GroupByClause and other clauses containing
SlotRefs happen after FromClause.analyze(). After the whole query block
is analyzed, we can get the exact set of required columns.
This patch refactor the codes to do table masking after analyze() to
avoid introducing unused columns. Referenced columns of a TableRef are
registered in analyze(), which helps to figure out what columns are
actually needed.
With this, we don't need to revert table masking in FromClause.reset().
The doTableMasking flag in AST is also removed since now the table mask
is resolved once after analyze().
Tests:
- Add more e2e tests in test_ranger.py
- Run CORE tests
Change-Id: Ib015a8ab528065907b27fbdceb8e2818deb814e1
Reviewed-on: http://gerrit.cloudera.org:8080/17199
Reviewed-by: Aman Sinha <am...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
.../org/apache/impala/analysis/AlterViewStmt.java | 2 -
.../apache/impala/analysis/AnalysisContext.java | 62 +++++----
.../java/org/apache/impala/analysis/Analyzer.java | 90 ++++++++-----
.../impala/analysis/CreateTableAsSelectStmt.java | 5 +
.../org/apache/impala/analysis/CreateViewStmt.java | 2 -
.../main/java/org/apache/impala/analysis/Expr.java | 10 ++
.../org/apache/impala/analysis/FromClause.java | 96 +++++---------
.../org/apache/impala/analysis/InlineViewRef.java | 25 ++--
.../org/apache/impala/analysis/InsertStmt.java | 8 ++
.../java/org/apache/impala/analysis/QueryStmt.java | 15 ++-
.../org/apache/impala/analysis/SelectStmt.java | 27 +++-
.../apache/impala/analysis/SetOperationStmt.java | 6 +-
.../java/org/apache/impala/analysis/SlotRef.java | 5 +
.../java/org/apache/impala/analysis/StmtNode.java | 10 ++
.../java/org/apache/impala/analysis/Subquery.java | 12 ++
.../java/org/apache/impala/analysis/TableRef.java | 15 +++
.../org/apache/impala/analysis/WithClause.java | 9 ++
.../org/apache/impala/authorization/TableMask.java | 31 +++--
.../apache/impala/analysis/AnalyzeStmtsTest.java | 2 +-
.../authorization/ranger/RangerAuditLogTest.java | 147 ++++++++++++---------
.../ranger_column_masking_and_row_filtering.test | 52 +++++++-
.../queries/QueryTest/ranger_row_filtering.test | 118 ++++++++++++++++-
tests/authorization/test_ranger.py | 6 +-
23 files changed, 517 insertions(+), 238 deletions(-)
diff --git a/fe/src/main/java/org/apache/impala/analysis/AlterViewStmt.java b/fe/src/main/java/org/apache/impala/analysis/AlterViewStmt.java
index f241f1d..f89dba7 100644
--- a/fe/src/main/java/org/apache/impala/analysis/AlterViewStmt.java
+++ b/fe/src/main/java/org/apache/impala/analysis/AlterViewStmt.java
@@ -45,8 +45,6 @@ public class AlterViewStmt extends CreateOrAlterViewStmtBase {
public void analyze(Analyzer analyzer) throws AnalysisException {
// Enforce Hive column labels for view compatibility.
analyzer.setUseHiveColLabels(true);
- // Disable table masking since we don't actually read the data
- viewDefStmt_.setDoTableMasking(false);
viewDefStmt_.analyze(analyzer);
Preconditions.checkState(tableName_ != null && !tableName_.isEmpty());
diff --git a/fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java b/fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java
index 2f3678b..48bed79 100644
--- a/fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java
+++ b/fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java
@@ -487,10 +487,32 @@ public class AnalysisContext {
// expensive with large expression trees. For example, a SQL that takes a few seconds
// to analyze the first time may take 10 minutes for rewrites.
analysisResult_.analyzer_.checkStmtExprLimit();
- boolean isExplain = analysisResult_.isExplainStmt();
- // Apply expr, setop, and subquery rewrites.
+ // The rewrites should have no user-visible effect on query results, including types
+ // and labels. Remember the original result types and column labels to restore them
+ // after the rewritten stmt has been reset() and re-analyzed. For a CTAS statement,
+ // the types represent column types of the table that will be created, including the
+ // partition columns, if any.
+ List<Type> origResultTypes = new ArrayList<>();
+ for (Expr e : analysisResult_.stmt_.getResultExprs()) {
+ origResultTypes.add(e.getType());
+ }
+ List<String> origColLabels =
+ Lists.newArrayList(analysisResult_.stmt_.getColLabels());
+
+ // Apply column/row masking, expr, setop, and subquery rewrites.
boolean reAnalyze = false;
+ if (authzFactory_.getAuthorizationConfig().isEnabled()) {
+ reAnalyze = analysisResult_.stmt_.resolveTableMask(analysisResult_.analyzer_);
+ // If any catalog table/view is replaced by table masking views, we need to
+ // resolve them. Also re-analyze the SlotRefs to reference the output exprs of
+ // the table masking views.
+ if (reAnalyze) {
+ reAnalyzeWithoutPrivChecks(stmtTableCache, authzCtx, origResultTypes,
+ origColLabels);
+ }
+ reAnalyze = false;
+ }
ExprRewriter rewriter = analysisResult_.analyzer_.getExprRewriter();
if (analysisResult_.requiresExprRewrite()) {
rewriter.reset();
@@ -511,24 +533,6 @@ public class AnalysisContext {
}
if (!reAnalyze) return;
- // The rewrites should have no user-visible effect. Remember the original result
- // types and column labels to restore them after the rewritten stmt has been
- // reset() and re-analyzed. For a CTAS statement, the types represent column types
- // of the table that will be created, including the partition columns, if any.
- List<Type> origResultTypes = new ArrayList<>();
- for (Expr e : analysisResult_.stmt_.getResultExprs()) {
- origResultTypes.add(e.getType());
- }
- List<String> origColLabels =
- Lists.newArrayList(analysisResult_.stmt_.getColLabels());
-
- // Some expressions, such as function calls with constant arguments, can get
- // folded into literals. Since literals do not require privilege requests, we
- // must save the original privileges in order to not lose them during
- // re-analysis.
- ImmutableList<PrivilegeRequest> origPrivReqs =
- analysisResult_.analyzer_.getPrivilegeReqs();
-
// For SetOperationStmt we must replace the query statement with the rewritten version
// before re-analysis.
if (analysisResult_.requiresSetOperationRewrite()) {
@@ -540,12 +544,26 @@ public class AnalysisContext {
}
}
+ reAnalyzeWithoutPrivChecks(stmtTableCache, authzCtx, origResultTypes, origColLabels);
+ Preconditions.checkState(!analysisResult_.requiresSubqueryRewrite());
+ }
+
+ private void reAnalyzeWithoutPrivChecks(StmtTableCache stmtTableCache,
+ AuthorizationContext authzCtx, List<Type> origResultTypes,
+ List<String> origColLabels) throws AnalysisException {
+ boolean isExplain = analysisResult_.isExplainStmt();
+ // Some expressions, such as function calls with constant arguments, can get
+ // folded into literals. Since literals do not require privilege requests, we
+ // must save the original privileges in order to not lose them during
+ // re-analysis.
+ ImmutableList<PrivilegeRequest> origPrivReqs =
+ analysisResult_.analyzer_.getPrivilegeReqs();
+
// Re-analyze the stmt with a new analyzer.
analysisResult_.analyzer_ = createAnalyzer(stmtTableCache, authzCtx);
// We restore the privileges collected in the first pass below. So, no point in
// collecting them again.
analysisResult_.analyzer_.setEnablePrivChecks(false);
-
analysisResult_.stmt_.reset();
try {
analysisResult_.stmt_.analyze(analysisResult_.analyzer_);
@@ -556,7 +574,6 @@ public class AnalysisContext {
analysisResult_.stmt_.toSql(REWRITTEN)));
throw e;
}
-
// Restore the original result types and column labels.
analysisResult_.stmt_.castResultExprs(origResultTypes);
analysisResult_.stmt_.setColLabels(origColLabels);
@@ -569,7 +586,6 @@ public class AnalysisContext {
analysisResult_.analyzer_.registerPrivReq(req);
}
if (isExplain) analysisResult_.stmt_.setIsExplain();
- Preconditions.checkState(!analysisResult_.requiresSubqueryRewrite());
}
public Analyzer getAnalyzer() { return analysisResult_.getAnalyzer(); }
diff --git a/fe/src/main/java/org/apache/impala/analysis/Analyzer.java b/fe/src/main/java/org/apache/impala/analysis/Analyzer.java
index 0f67726..d36f9cd 100644
--- a/fe/src/main/java/org/apache/impala/analysis/Analyzer.java
+++ b/fe/src/main/java/org/apache/impala/analysis/Analyzer.java
@@ -730,10 +730,6 @@ public class Analyzer {
return result;
}
- public TableRef resolveTableRef(TableRef tableRef) throws AnalysisException {
- return resolveTableRef(tableRef, false);
- }
-
/**
* Resolves the given TableRef into a concrete BaseTableRef, ViewRef or
* CollectionTableRef. Returns the new resolved table ref or the given table
@@ -743,7 +739,7 @@ public class Analyzer {
* an AuthorizationException is preferred over an AnalysisException so as not to
* accidentally reveal the non-existence of tables/databases.
*/
- public TableRef resolveTableRef(TableRef tableRef, boolean doTableMasking)
+ public TableRef resolveTableRef(TableRef tableRef)
throws AnalysisException {
// Return the table if it is already resolved. This also avoids the table being
// masked again.
@@ -784,24 +780,15 @@ public class Analyzer {
Preconditions.checkNotNull(resolvedPath);
if (resolvedPath.destTable() != null) {
FeTable table = resolvedPath.destTable();
- TableRef resolvedTableRef;
- if (table instanceof FeView) {
- resolvedTableRef = new InlineViewRef((FeView) table, tableRef);
- } else {
- // The table must be a base table.
- Preconditions.checkState(table instanceof FeFsTable ||
- table instanceof FeKuduTable ||
- table instanceof FeHBaseTable ||
- table instanceof FeDataSourceTable);
- resolvedTableRef = new BaseTableRef(tableRef, resolvedPath);
- }
- if (!doTableMasking) return resolvedTableRef;
- return resolveTableMask(resolvedTableRef, table);
+ if (table instanceof FeView) return new InlineViewRef((FeView) table, tableRef);
+ // The table must be a base table.
+ Preconditions.checkState(table instanceof FeFsTable ||
+ table instanceof FeKuduTable ||
+ table instanceof FeHBaseTable ||
+ table instanceof FeDataSourceTable);
+ return new BaseTableRef(tableRef, resolvedPath);
} else {
- CollectionTableRef res = new CollectionTableRef(tableRef, resolvedPath);
- // Relative table refs don't need masking. Its base table will be masked.
- if (!doTableMasking || res.isRelative()) return res;
- return resolveTableMask(res, res.getTable());
+ return new CollectionTableRef(tableRef, resolvedPath);
}
}
@@ -838,10 +825,8 @@ public class Analyzer {
* so we know the target table/view/collection.
*
* @param resolvedTableRef A resolved TableRef for table masking
- * @param basedTable FeTable of the resolved table or the collection's based table
*/
- private TableRef resolveTableMask(TableRef resolvedTableRef, FeTable basedTable)
- throws AnalysisException {
+ public TableRef resolveTableMask(TableRef resolvedTableRef) throws AnalysisException {
Preconditions.checkState(resolvedTableRef.isResolved(), "Table should be resolved");
// Only do table masking when authorization is enabled and the authorization
// factory supports column-masking/row-filtering. If both of these are false,
@@ -853,27 +838,42 @@ public class Analyzer {
// Performing table masking.
AuthorizationChecker authChecker = getAuthzFactory().newAuthorizationChecker(
getCatalog().getAuthPolicy());
- TableMask tableMask = new TableMask(authChecker, basedTable, user_);
+ String dbName;
+ String tblName;
+ if (resolvedTableRef instanceof InlineViewRef) {
+ FeView view = ((InlineViewRef) resolvedTableRef).getView();
+ dbName = view.getDb().getName();
+ tblName = view.getName();
+ } else {
+ dbName = resolvedTableRef.getTable().getDb().getName();
+ tblName = resolvedTableRef.getTable().getName();
+ }
+ List<Column> columns = resolvedTableRef.getScalarColumns();
+ TableMask tableMask = new TableMask(authChecker, dbName, tblName, columns, user_);
try {
if (resolvedTableRef instanceof CollectionTableRef) {
- Preconditions.checkState(!resolvedTableRef.isRelative(),
- "Relative table refs don't need masking. Its base table will be masked.");
+ // Relative table refs don't need masking. Its base table will be masked.
+ if (resolvedTableRef.isRelative()) return resolvedTableRef;
if (tableMask.needsRowFiltering()) {
- // TODO: Support this in IMPALA-10484.
+ // The table ref is a non-relative CollectionTableRef, e.g. the table ref in
+ // "select item from functional_parquet.complextypestbl.int_array". We can't
+ // replace "complextypestbl" with a table masking view here.
+ // TODO: Support this in IMPALA-10484 by rewriting it to relative ref, e.g.
+ // select a.item from functional_parquet.complextypestbl t, t.int_array a;
throw new AnalysisException(String.format("Using non-relative collection " +
- "column %s of table %s is not supported since there are row-filtering " +
+ "column %s of table %s.%s is not supported since there are row-filtering " +
"policies on this table (IMPALA-10484). Rewrite query to use relative " +
"reference.",
String.join(".", resolvedTableRef.getResolvedPath().getRawPath()),
- basedTable.getFullName()));
+ dbName, tblName));
}
} else if (tableMask.needsMaskingOrFiltering()) {
- return InlineViewRef.createTableMaskView(basedTable, resolvedTableRef,
- tableMask, getAuthzCtx());
+ return InlineViewRef.createTableMaskView(resolvedTableRef, tableMask,
+ getAuthzCtx());
}
return resolvedTableRef;
} catch (InternalException e) {
- String msg = "Error performing table masking on " + basedTable.getFullName();
+ String msg = "Error resolving table mask on " + dbName + "." + tblName;
LOG.error(msg, e);
throw new AnalysisException(msg, e);
}
@@ -1327,6 +1327,28 @@ public class Analyzer {
}
/**
+ * Register scalar columns. Used in resolving column mask.
+ */
+ public void registerScalarColumnForMasking(SlotDescriptor slotDesc) {
+ Preconditions.checkNotNull(slotDesc.getPath());
+ Preconditions.checkState(!slotDesc.getType().isComplexType(),
+ "Don't register complex type columns");
+ TupleDescriptor tupleDesc = slotDesc.getParent();
+ Column column = new Column(slotDesc.getPath().getRawPath().get(0), slotDesc.getType(),
+ /*position*/-1);
+ Analyzer analyzer = this;
+ TableRef tblRef;
+ do {
+ tblRef = analyzer.tableRefMap_.get(tupleDesc.getId());
+ // Search in parent query block for correlative reference.
+ analyzer = analyzer.getParentAnalyzer();
+ } while (tblRef == null && analyzer != null);
+ Preconditions.checkNotNull(tblRef,
+ "Failed to find TableRef of tuple {}", tupleDesc);
+ tblRef.registerScalarColumn(column);
+ }
+
+ /**
* Creates a new slot descriptor and related state in globalState.
*/
public SlotDescriptor addSlotDescriptor(TupleDescriptor tupleDesc) {
diff --git a/fe/src/main/java/org/apache/impala/analysis/CreateTableAsSelectStmt.java b/fe/src/main/java/org/apache/impala/analysis/CreateTableAsSelectStmt.java
index cf952b6..b111d56 100644
--- a/fe/src/main/java/org/apache/impala/analysis/CreateTableAsSelectStmt.java
+++ b/fe/src/main/java/org/apache/impala/analysis/CreateTableAsSelectStmt.java
@@ -286,4 +286,9 @@ public class CreateTableAsSelectStmt extends StatementBase {
createStmt_.getPartitionColumnDefs().clear();
insertStmt_.reset();
}
+
+ @Override
+ public boolean resolveTableMask(Analyzer analyzer) throws AnalysisException {
+ return getQueryStmt().resolveTableMask(analyzer);
+ }
}
diff --git a/fe/src/main/java/org/apache/impala/analysis/CreateViewStmt.java b/fe/src/main/java/org/apache/impala/analysis/CreateViewStmt.java
index 3ff828a..a387022 100644
--- a/fe/src/main/java/org/apache/impala/analysis/CreateViewStmt.java
+++ b/fe/src/main/java/org/apache/impala/analysis/CreateViewStmt.java
@@ -46,8 +46,6 @@ public class CreateViewStmt extends CreateOrAlterViewStmtBase {
Analyzer viewAnalyzerr = new Analyzer(analyzer);
// Enforce Hive column labels for view compatibility.
viewAnalyzerr.setUseHiveColLabels(true);
- // Disable table masking since we don't actually read the data.
- viewDefStmt_.setDoTableMasking(false);
viewDefStmt_.analyze(viewAnalyzerr);
dbName_ = analyzer.getTargetDbName(tableName_);
diff --git a/fe/src/main/java/org/apache/impala/analysis/Expr.java b/fe/src/main/java/org/apache/impala/analysis/Expr.java
index 21fc846..2934598 100644
--- a/fe/src/main/java/org/apache/impala/analysis/Expr.java
+++ b/fe/src/main/java/org/apache/impala/analysis/Expr.java
@@ -1842,4 +1842,14 @@ abstract public class Expr extends TreeNode<Expr> implements ParseNode, Cloneabl
return new NullLiteral();
}
+ /**
+ * Subclass that contains query statements, e.g SubQuery, should override this.
+ */
+ public boolean resolveTableMask(Analyzer analyzer) throws AnalysisException {
+ boolean hasChanges = false;
+ for (Expr child : children_) {
+ hasChanges |= child.resolveTableMask(analyzer);
+ }
+ return hasChanges;
+ }
}
diff --git a/fe/src/main/java/org/apache/impala/analysis/FromClause.java b/fe/src/main/java/org/apache/impala/analysis/FromClause.java
index 9feb5fe..76c75b6 100644
--- a/fe/src/main/java/org/apache/impala/analysis/FromClause.java
+++ b/fe/src/main/java/org/apache/impala/analysis/FromClause.java
@@ -39,12 +39,6 @@ public class FromClause extends StmtNode implements Iterable<TableRef> {
private boolean analyzed_ = false;
- // Whether we should perform table masking. It will replace each table/view with a
- // masked subquery if there're column masking policies for the user on this table/view.
- // Turned off for CreateView and AlterView statements since they're not actually
- // reading data.
- private boolean doTableMasking_ = true;
-
public FromClause(List<TableRef> tableRefs) {
tableRefs_ = Lists.newArrayList(tableRefs);
// Set left table refs to ensure correct toSql() before analysis.
@@ -56,13 +50,24 @@ public class FromClause extends StmtNode implements Iterable<TableRef> {
public FromClause() { tableRefs_ = new ArrayList<>(); }
public List<TableRef> getTableRefs() { return tableRefs_; }
- public void setDoTableMasking(boolean doTableMasking) {
- doTableMasking_ = doTableMasking;
- for (TableRef tableRef : tableRefs_) {
- if (!(tableRef instanceof InlineViewRef)) continue;
- InlineViewRef viewRef = (InlineViewRef) tableRef;
- viewRef.getViewStmt().setDoTableMasking(doTableMasking);
+ @Override
+ public boolean resolveTableMask(Analyzer analyzer) throws AnalysisException {
+ boolean hasChanges = false;
+ for (int i = 0; i < size(); ++i) {
+ TableRef origRef = get(i);
+ if (origRef instanceof InlineViewRef) {
+ hasChanges |= ((InlineViewRef) origRef).getViewStmt().resolveTableMask(analyzer);
+ // Skip local views
+ if (!((InlineViewRef) origRef).isCatalogView()) continue;
+ }
+ TableRef newRef = analyzer.resolveTableMask(origRef);
+ if (newRef == origRef) continue;
+ set(i, newRef);
+ hasChanges = true;
+ Preconditions.checkState(newRef.isTableMaskingView(),
+ "resolved table mask should be a view");
}
+ return hasChanges;
}
@Override
@@ -72,7 +77,7 @@ public class FromClause extends StmtNode implements Iterable<TableRef> {
TableRef leftTblRef = null; // the one to the left of tblRef
for (int i = 0; i < tableRefs_.size(); ++i) {
TableRef tblRef = tableRefs_.get(i);
- tblRef = analyzer.resolveTableRef(tblRef, doTableMasking_);
+ tblRef = analyzer.resolveTableRef(tblRef);
tableRefs_.set(i, Preconditions.checkNotNull(tblRef));
tblRef.setLeftTblRef(leftTblRef);
tblRef.analyze(analyzer);
@@ -121,62 +126,27 @@ public class FromClause extends StmtNode implements Iterable<TableRef> {
return new FromClause(clone);
}
- /**
- * Unmask, un-resolve and reset all the tableRefs.
- * TableMasking views are created by analysis so we should unwrap them to restore the
- * unmasked TableRef.
- * 'un-resolve' here means replacing resolved tableRefs with unresolved ones. To make
- * sure we get the same results in later resolution, the unresolved tableRefs should
- * use fully qualified paths. Otherwise, non-fully qualified paths might incorrectly
- * match a local view.
- * However, we don't un-resolve views because local views don't have fully qualified
- * paths. Due to this we don't unmask a TableMasking view if the underlying TableRef is
- * a view. TODO(IMPALA-9286): These may make this FromClause still dirty after reset()
- * since it doesn't come back to the state before analyze(). We should introduce fully
- * qualified paths for local views to fix this.
- */
public void reset() {
for (int i = 0; i < size(); ++i) {
- unmaskAndUnresolveTableRef(i);
- get(i).reset(); // Reset() recursion happens here for views
+ TableRef origTblRef = get(i);
+ if (origTblRef.isResolved() && !(origTblRef instanceof InlineViewRef)) {
+ // Replace resolved table refs with unresolved ones.
+ TableRef newTblRef = new TableRef(origTblRef);
+ // Use the fully qualified raw path to preserve the original resolution.
+ // Otherwise, non-fully qualified paths might incorrectly match a local view.
+ // TODO(IMPALA-9286): This full qualification preserves analysis state which is
+ // contrary to the intended semantics of reset(). We could address this issue by
+ // changing the WITH-clause analysis to register local views that have
+ // fully-qualified table refs, and then remove the full qualification here.
+ newTblRef.rawPath_ = origTblRef.getResolvedPath().getFullyQualifiedRawPath();
+ set(i, newTblRef);
+ }
+ // recurse for views
+ get(i).reset();
}
this.analyzed_ = false;
}
- /**
- * Replace the i-th tableRef with an unmasked and unresolved one if the result is not a
- * view. See more in comments of reset().
- */
- private void unmaskAndUnresolveTableRef(int i) {
- TableRef origTblRef = get(i);
- if (!origTblRef.isResolved()
- || (origTblRef instanceof InlineViewRef && !origTblRef.isTableMaskingView())) {
- return;
- }
- // Unmasked the TableRef if it's an inline view for table masking.
- if (origTblRef.isTableMaskingView()) {
- Preconditions.checkState(origTblRef instanceof InlineViewRef);
- TableRef unMaskedTableRef = ((InlineViewRef) origTblRef).getUnMaskedTableRef();
- if (unMaskedTableRef instanceof InlineViewRef) return;
- // Migrate back the properties (e.g. join ops, hints) since we are going to
- // replace it with an unresolved one.
- origTblRef.migratePropertiesTo(unMaskedTableRef);
- origTblRef = unMaskedTableRef;
- }
- // Replace the resolved TableRef with an unresolved one if it's not a view.
- if (!(origTblRef instanceof InlineViewRef)) {
- TableRef newTblRef = new TableRef(origTblRef);
- // Use the fully qualified raw path to preserve the original resolution.
- // Otherwise, non-fully qualified paths might incorrectly match a local view.
- // TODO(IMPALA-9286): This full qualification preserves analysis state which is
- // contrary to the intended semantics of reset(). We could address this issue by
- // changing the WITH-clause analysis to register local views that have
- // fully-qualified table refs, and then remove the full qualification here.
- newTblRef.rawPath_ = origTblRef.getResolvedPath().getFullyQualifiedRawPath();
- set(i, newTblRef);
- }
- }
-
@Override
public final String toSql() {
return toSql(DEFAULT);
diff --git a/fe/src/main/java/org/apache/impala/analysis/InlineViewRef.java b/fe/src/main/java/org/apache/impala/analysis/InlineViewRef.java
index 770c7ec..da7ced2 100644
--- a/fe/src/main/java/org/apache/impala/analysis/InlineViewRef.java
+++ b/fe/src/main/java/org/apache/impala/analysis/InlineViewRef.java
@@ -143,29 +143,29 @@ public class InlineViewRef extends TableRef {
* Creates an inline-view doing table masking for column masking and row filtering
* policies. Callers should replace 'tableRef' with the returned view.
*
- * @param resolvedTable resolved FeTable for the original table/view
* @param tableRef original resolved table/view
* @param tableMask TableMask providing column masking and row filtering policies
* @param authzCtx AuthorizationContext containing RangerBufferAuditHandler
*/
- static InlineViewRef createTableMaskView(FeTable resolvedTable, TableRef tableRef,
- TableMask tableMask, AuthorizationContext authzCtx) throws AnalysisException,
- InternalException {
- Preconditions.checkNotNull(resolvedTable);
+ static InlineViewRef createTableMaskView(TableRef tableRef, TableMask tableMask,
+ AuthorizationContext authzCtx) throws AnalysisException, InternalException {
Preconditions.checkNotNull(tableRef);
Preconditions.checkNotNull(authzCtx);
Preconditions.checkState(tableRef instanceof InlineViewRef
|| tableRef instanceof BaseTableRef);
- List<Column> columns = resolvedTable.getColumnsInHiveOrder();
+ List<Column> columns = tableMask.getRequiredColumns();
List<SelectListItem> items = Lists.newArrayListWithCapacity(columns.size());
for (Column col: columns) {
- if (col.getType().isComplexType()) continue;
- // TODO: only add materialized columns to avoid introducing new privilege
- // requirements (IMPALA-9223)
+ Preconditions.checkState(!col.getType().isComplexType(),
+ "Complex type columns should not be registered to table mask");
items.add(new SelectListItem(
tableMask.createColumnMask(col.getName(), col.getType(), authzCtx),
/*alias*/ col.getName()));
}
+ if (columns.isEmpty()) {
+ // No columns so use "SELECT 1 FROM tbl" to make a valid statement.
+ items.add(new SelectListItem(NumericLiteral.create(1), /*alias*/null));
+ }
SelectList selectList = new SelectList(items);
FromClause fromClause = new FromClause(Lists.newArrayList(tableRef));
Expr wherePredicate = tableMask.createRowFilter(authzCtx);
@@ -212,8 +212,7 @@ public class InlineViewRef extends TableRef {
inlineViewAnalyzer_ = new Analyzer(analyzer);
// Catalog views refs require special analysis settings for authorization.
- boolean isCatalogView = (view_ != null && !view_.isLocalView());
- if (isCatalogView) {
+ if (isCatalogView()) {
analyzer.registerAuthAndAuditEvent(view_, priv_, requireGrantOption_);
if (inlineViewAnalyzer_.isExplain()) {
// If the user does not have privileges on the view's definition
@@ -231,7 +230,7 @@ public class InlineViewRef extends TableRef {
}
inlineViewAnalyzer_.setUseHiveColLabels(
- isCatalogView ? true : analyzer.useHiveColLabels());
+ isCatalogView() ? true : analyzer.useHiveColLabels());
queryStmt_.analyze(inlineViewAnalyzer_);
correlatedTupleIds_.addAll(queryStmt_.getCorrelatedTupleIds());
if (explicitColLabels_ != null) {
@@ -409,6 +408,8 @@ public class InlineViewRef extends TableRef {
public boolean isTableMaskingView() { return isTableMaskingView_; }
+ public boolean isCatalogView() { return view_ != null && !view_.isLocalView(); }
+
/**
* Return the unmasked TableRef if this is an inline view for table masking.
*/
diff --git a/fe/src/main/java/org/apache/impala/analysis/InsertStmt.java b/fe/src/main/java/org/apache/impala/analysis/InsertStmt.java
index 21cc3d2..a6a83d1 100644
--- a/fe/src/main/java/org/apache/impala/analysis/InsertStmt.java
+++ b/fe/src/main/java/org/apache/impala/analysis/InsertStmt.java
@@ -1179,4 +1179,12 @@ public class InsertStmt extends StatementBase {
tblRefs.add(new TableRef(targetTableName_.toPath(), null));
if (queryStmt_ != null) queryStmt_.collectTableRefs(tblRefs);
}
+
+ @Override
+ public boolean resolveTableMask(Analyzer analyzer) throws AnalysisException {
+ boolean hasChange = false;
+ if (withClause_ != null) hasChange = withClause_.resolveTableMask(analyzer);
+ if (queryStmt_ != null) hasChange |= queryStmt_.resolveTableMask(analyzer);
+ return hasChange;
+ }
}
diff --git a/fe/src/main/java/org/apache/impala/analysis/QueryStmt.java b/fe/src/main/java/org/apache/impala/analysis/QueryStmt.java
index a303794..e7f7f63 100644
--- a/fe/src/main/java/org/apache/impala/analysis/QueryStmt.java
+++ b/fe/src/main/java/org/apache/impala/analysis/QueryStmt.java
@@ -167,12 +167,15 @@ public abstract class QueryStmt extends StatementBase {
if (hasWithClause()) withClause_.analyze(analyzer);
}
- /**
- * Disable table masking when analyzing the FromClauses. Used in CreateView and
- * AlterView statements since they don't actually read the data.
- * @param doTableMasking
- */
- public void setDoTableMasking(boolean doTableMasking) {}
+ @Override
+ public boolean resolveTableMask(Analyzer analyzer) throws AnalysisException {
+ boolean hasChanges = false;
+ if (hasWithClause()) {
+ // Local views may be used in rewriting. Make sure they are masked as well.
+ hasChanges = withClause_.resolveTableMask(analyzer);
+ }
+ return hasChanges;
+ }
/**
* Returns a list containing all the materialized tuple ids that this stmt is
diff --git a/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java b/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java
index 49330e1..34232bc 100644
--- a/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java
+++ b/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java
@@ -265,8 +265,20 @@ public class SelectStmt extends QueryStmt {
}
@Override
- public void setDoTableMasking(boolean doTableMasking) {
- fromClause_.setDoTableMasking(doTableMasking);
+ public boolean resolveTableMask(Analyzer analyzer) throws AnalysisException {
+ boolean hasChanges = super.resolveTableMask(analyzer);
+ // Recurse in all places that could have subqueries. After resolveTableMask() is done
+ // on the whole AST, SlotRefs referencing the original table refs will be reset and
+ // re-analyzed to reference the table masking views. So we don't need to deal with
+ // SlotRefs here.
+ hasChanges |= fromClause_.resolveTableMask(analyzer);
+ for (SelectListItem item : selectList_.getItems()) {
+ if (item.isStar()) continue;
+ hasChanges |= item.getExpr().resolveTableMask(analyzer);
+ }
+ if (whereClause_ != null) hasChanges |= whereClause_.resolveTableMask(analyzer);
+ if (havingClause_ != null) hasChanges |= havingClause_.resolveTableMask(analyzer);
+ return hasChanges;
}
/**
@@ -642,7 +654,16 @@ public class SelectStmt extends QueryStmt {
if (p.destType().isComplexType()) return;
SlotDescriptor slotDesc = analyzer_.registerSlotRef(p);
SlotRef slotRef = new SlotRef(slotDesc);
- slotRef.analyze(analyzer_);
+ Preconditions.checkState(slotRef.isAnalyzed(),
+ "Analysis should be done in constructor");
+ // Empty matched types means this is expanded from star of a catalog table.
+ // For star of complex types, e.g. my_struct.*, my_array.*, my_map.*, the matched
+ // types will have the complex type so it's not empty.
+ if (resolvedPath.getMatchedTypes().isEmpty()) {
+ Preconditions.checkState(!slotDesc.getType().isComplexType(),
+ "Star expansion should only introduce scalar columns");
+ analyzer_.registerScalarColumnForMasking(slotDesc);
+ }
resultExprs_.add(slotRef);
colLabels_.add(relRawPath[relRawPath.length - 1]);
}
diff --git a/fe/src/main/java/org/apache/impala/analysis/SetOperationStmt.java b/fe/src/main/java/org/apache/impala/analysis/SetOperationStmt.java
index 864a1a3..5230f8f 100644
--- a/fe/src/main/java/org/apache/impala/analysis/SetOperationStmt.java
+++ b/fe/src/main/java/org/apache/impala/analysis/SetOperationStmt.java
@@ -299,10 +299,12 @@ public class SetOperationStmt extends QueryStmt {
}
@Override
- public void setDoTableMasking(boolean doTableMasking) {
+ public boolean resolveTableMask(Analyzer analyzer) throws AnalysisException {
+ boolean hasChanges = false;
for (SetOperand op : operands_) {
- op.getQueryStmt().setDoTableMasking(doTableMasking);
+ hasChanges |= op.getQueryStmt().resolveTableMask(analyzer);
}
+ return hasChanges;
}
@Override
diff --git a/fe/src/main/java/org/apache/impala/analysis/SlotRef.java b/fe/src/main/java/org/apache/impala/analysis/SlotRef.java
index 8c2beed..0ca1819 100644
--- a/fe/src/main/java/org/apache/impala/analysis/SlotRef.java
+++ b/fe/src/main/java/org/apache/impala/analysis/SlotRef.java
@@ -133,6 +133,11 @@ public class SlotRef extends Expr {
// HMS string.
throw new UnsupportedFeatureException("Unsupported type in '" + toSql() + "'.");
}
+ // Register scalar columns of a catalog table.
+ if (!resolvedPath.getMatchedTypes().isEmpty()
+ && !resolvedPath.getMatchedTypes().get(0).isComplexType()) {
+ analyzer.registerScalarColumnForMasking(desc_);
+ }
numDistinctValues_ = adjustNumDistinctValues();
FeTable rootTable = resolvedPath.getRootTable();
diff --git a/fe/src/main/java/org/apache/impala/analysis/StmtNode.java b/fe/src/main/java/org/apache/impala/analysis/StmtNode.java
index 1a0fcbc..b9d9110 100644
--- a/fe/src/main/java/org/apache/impala/analysis/StmtNode.java
+++ b/fe/src/main/java/org/apache/impala/analysis/StmtNode.java
@@ -29,4 +29,14 @@ public abstract class StmtNode implements ParseNode {
* @throws AnalysisException if any semantic errors were found
*/
public abstract void analyze(Analyzer analyzer) throws AnalysisException;
+
+ /**
+ * By default, table masking is not performed. When authorization is enabled and
+ * tbe base table/view has column-masking / row-filtering policies, the table ref
+ * will be masked by a table masking view. Called after analyze().
+ */
+ public boolean resolveTableMask(Analyzer analyzer) throws AnalysisException {
+ // Don't apply table mask by default.
+ return false;
+ }
}
diff --git a/fe/src/main/java/org/apache/impala/analysis/Subquery.java b/fe/src/main/java/org/apache/impala/analysis/Subquery.java
index 5ac8193..9c894c8 100644
--- a/fe/src/main/java/org/apache/impala/analysis/Subquery.java
+++ b/fe/src/main/java/org/apache/impala/analysis/Subquery.java
@@ -163,4 +163,16 @@ public class Subquery extends Expr {
@Override
protected void toThrift(TExprNode msg) {}
+
+ @Override
+ public boolean resolveTableMask(Analyzer analyzer) throws AnalysisException {
+ return stmt_.resolveTableMask(analyzer);
+ }
+
+ @Override
+ protected void resetAnalysisState() {
+ super.resetAnalysisState();
+ stmt_.reset();
+ analyzer_ = null;
+ }
}
diff --git a/fe/src/main/java/org/apache/impala/analysis/TableRef.java b/fe/src/main/java/org/apache/impala/analysis/TableRef.java
index 4f8b776..3a9b358 100644
--- a/fe/src/main/java/org/apache/impala/analysis/TableRef.java
+++ b/fe/src/main/java/org/apache/impala/analysis/TableRef.java
@@ -22,10 +22,13 @@ import static org.apache.impala.analysis.ToSqlOptions.DEFAULT;
import java.util.ArrayList;
import java.util.Collections;
import java.util.HashSet;
+import java.util.LinkedHashMap;
import java.util.List;
+import java.util.Map;
import java.util.Set;
import org.apache.impala.authorization.Privilege;
+import org.apache.impala.catalog.Column;
import org.apache.impala.catalog.FeFsTable;
import org.apache.impala.catalog.FeTable;
import org.apache.impala.common.AnalysisException;
@@ -141,6 +144,9 @@ public class TableRef extends StmtNode {
// columns via the view.
protected boolean exposeNestedColumnsByTableMaskView_ = false;
+ // Scalar columns referenced in the query. Used in resolving column mask.
+ protected Map<String, Column> scalarColumns_ = new LinkedHashMap<>();
+
// END: Members that need to be reset()
/////////////////////////////////////////
@@ -221,6 +227,7 @@ public class TableRef extends StmtNode {
correlatedTupleIds_ = Lists.newArrayList(other.correlatedTupleIds_);
desc_ = other.desc_;
exposeNestedColumnsByTableMaskView_ = other.exposeNestedColumnsByTableMaskView_;
+ scalarColumns_ = new LinkedHashMap<>(other.scalarColumns_);
}
@Override
@@ -706,6 +713,14 @@ public class TableRef extends StmtNode {
public boolean isTableMaskingView() { return false; }
+ public void registerScalarColumn(Column column) {
+ scalarColumns_.put(column.getName(), column);
+ }
+
+ public List<Column> getScalarColumns() {
+ return new ArrayList<>(scalarColumns_.values());
+ }
+
void migratePropertiesTo(TableRef other) {
other.aliases_ = aliases_;
other.onClause_ = onClause_;
diff --git a/fe/src/main/java/org/apache/impala/analysis/WithClause.java b/fe/src/main/java/org/apache/impala/analysis/WithClause.java
index 5721065..b229754 100644
--- a/fe/src/main/java/org/apache/impala/analysis/WithClause.java
+++ b/fe/src/main/java/org/apache/impala/analysis/WithClause.java
@@ -145,4 +145,13 @@ public class WithClause extends StmtNode {
}
public List<View> getViews() { return views_; }
+
+ @Override
+ public boolean resolveTableMask(Analyzer analyzer) throws AnalysisException {
+ boolean hasChanges = false;
+ for (View v : views_) {
+ hasChanges |= v.getQueryStmt().resolveTableMask(analyzer);
+ }
+ return hasChanges;
+ }
}
diff --git a/fe/src/main/java/org/apache/impala/authorization/TableMask.java b/fe/src/main/java/org/apache/impala/authorization/TableMask.java
index 81b0349..440ecf0 100644
--- a/fe/src/main/java/org/apache/impala/authorization/TableMask.java
+++ b/fe/src/main/java/org/apache/impala/authorization/TableMask.java
@@ -23,7 +23,7 @@ import org.apache.impala.analysis.Expr;
import org.apache.impala.analysis.Parser;
import org.apache.impala.analysis.SelectStmt;
import org.apache.impala.analysis.SlotRef;
-import org.apache.impala.catalog.FeTable;
+import org.apache.impala.catalog.Column;
import org.apache.impala.catalog.Type;
import org.apache.impala.common.AnalysisException;
import org.apache.impala.common.InternalException;
@@ -31,6 +31,7 @@ import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.util.List;
+import java.util.stream.Collectors;
/**
* A TableMask instance contains all the information for generating a subquery for table
@@ -39,28 +40,32 @@ import java.util.List;
public class TableMask {
private static final Logger LOG = LoggerFactory.getLogger(TableMask.class);
- private AuthorizationChecker authChecker_;
- private String dbName_;
- private String tableName_;
- private List<String> requiredColumns_;
- private User user_;
+ private final AuthorizationChecker authChecker_;
+ private final String dbName_;
+ private final String tableName_;
+ private final List<Column> requiredColumns_;
+ private final List<String> requiredColumnNames_;
+ private final User user_;
- public TableMask(AuthorizationChecker authzChecker, FeTable table, User user) {
+ public TableMask(AuthorizationChecker authzChecker, String dbName, String tableName,
+ List<Column> requiredColumns, User user) {
this.authChecker_ = authzChecker;
- this.dbName_ = table.getDb().getName();
- this.tableName_ = table.getName();
- // TODO: only require materialize columns to avoid unneccessary masking so we won't
- // hit IMPALA-9223
- this.requiredColumns_ = table.getColumnNames();
this.user_ = user;
+ this.dbName_ = dbName;
+ this.tableName_ = tableName;
+ this.requiredColumns_ = requiredColumns;
+ this.requiredColumnNames_ = requiredColumns.stream().map(Column::getName)
+ .collect(Collectors.toList());
}
+ public List<Column> getRequiredColumns() { return requiredColumns_; }
+
/**
* Returns whether the table/view has column masking or row filtering policies.
*/
public boolean needsMaskingOrFiltering() throws InternalException {
return authChecker_.needsMaskingOrFiltering(user_, dbName_, tableName_,
- requiredColumns_);
+ requiredColumnNames_);
}
/**
diff --git a/fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java b/fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java
index b91370b..402077a 100644
--- a/fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java
+++ b/fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java
@@ -4539,7 +4539,7 @@ public class AnalyzeStmtsTest extends AnalyzerTest {
testNumberOfMembers(ValuesStmt.class, 0);
// Also check TableRefs.
- testNumberOfMembers(TableRef.class, 24);
+ testNumberOfMembers(TableRef.class, 25);
testNumberOfMembers(BaseTableRef.class, 0);
testNumberOfMembers(InlineViewRef.class, 9);
}
diff --git a/fe/src/test/java/org/apache/impala/authorization/ranger/RangerAuditLogTest.java b/fe/src/test/java/org/apache/impala/authorization/ranger/RangerAuditLogTest.java
index a2e52d3..f0d0ce7 100644
--- a/fe/src/test/java/org/apache/impala/authorization/ranger/RangerAuditLogTest.java
+++ b/fe/src/test/java/org/apache/impala/authorization/ranger/RangerAuditLogTest.java
@@ -335,7 +335,24 @@ public class RangerAuditLogTest extends AuthorizationTestBase {
}
authzOk(events -> {
- assertEquals(15, events.size());
+ assertEquals(5, events.size());
+ assertEquals("select id, bool_col, string_col from functional.alltypestiny",
+ events.get(0).getRequestData());
+ assertEventEquals("@table", "select", "functional/alltypestiny", 1,
+ events.get(0));
+ assertEventEquals("@column", "select", "functional/alltypestiny/id", 1,
+ events.get(1));
+ assertEventEquals("@column", "select", "functional/alltypestiny/bool_col", 1,
+ events.get(2));
+ assertEventEquals("@column", "select", "functional/alltypestiny/string_col", 1,
+ events.get(3));
+ assertEventEquals("@column", "custom", "functional/alltypestiny/string_col", 1,
+ events.get(4));
+ }, "select id, bool_col, string_col from functional.alltypestiny",
+ onTable("functional", "alltypestiny", TPrivilegeLevel.SELECT));
+
+ authzOk(events -> {
+ assertEquals(16, events.size());
assertEquals("select * from functional.alltypestiny",
events.get(0).getRequestData());
assertEventEquals("@table", "select", "functional/alltypestiny", 1,
@@ -356,18 +373,20 @@ public class RangerAuditLogTest extends AuthorizationTestBase {
events.get(7));
assertEventEquals("@column", "select", "functional/alltypestiny/double_col", 1,
events.get(8));
+ assertEventEquals("@column", "select", "functional/alltypestiny/date_string_col",
+ 1, events.get(9));
assertEventEquals("@column", "select", "functional/alltypestiny/string_col", 1,
- events.get(9));
- assertEventEquals("@column", "select", "functional/alltypestiny/timestamp_col", 1,
events.get(10));
- assertEventEquals("@column", "select", "functional/alltypestiny/year", 1,
+ assertEventEquals("@column", "select", "functional/alltypestiny/timestamp_col", 1,
events.get(11));
- assertEventEquals("@column", "select", "functional/alltypestiny/month", 1,
+ assertEventEquals("@column", "select", "functional/alltypestiny/year", 1,
events.get(12));
+ assertEventEquals("@column", "select", "functional/alltypestiny/month", 1,
+ events.get(13));
assertEventEquals("@column", "mask_null",
- "functional/alltypestiny/date_string_col", 1, events.get(13));
+ "functional/alltypestiny/date_string_col", 1, events.get(14));
assertEventEquals("@column", "custom", "functional/alltypestiny/string_col", 1,
- events.get(14));
+ events.get(15));
}, "select * from functional.alltypestiny", onTable("functional", "alltypestiny",
TPrivilegeLevel.SELECT));
@@ -378,12 +397,7 @@ public class RangerAuditLogTest extends AuthorizationTestBase {
// this column. This test verifies that the duplicate event is indeed removed,
// i.e., only one entry for the column of 'string_col' having accessType as
// 'custom'.
- // TODO (IMPALA-9661): Note that the audits of unused columns, e.g., 'tinyint_col'
- // and 'date_string_col' are also logged, indicating that the privilege check for
- // the column of 'tinyint_col' and the evaluation of the column masking policy for
- // the column of 'date_string_col' are still performed. We should remove those
- // unnecessary checks.
- authzOk(events -> {assertEquals(15, events.size());
+ authzOk(events -> {assertEquals(5, events.size());
assertEquals("with iv as (select id, bool_col, string_col from " +
"functional.alltypestiny) select * from iv",
events.get(0).getRequestData());
@@ -393,41 +407,16 @@ public class RangerAuditLogTest extends AuthorizationTestBase {
events.get(1));
assertEventEquals("@column", "select", "functional/alltypestiny/bool_col", 1,
events.get(2));
- assertEventEquals("@column", "select", "functional/alltypestiny/tinyint_col", 1,
- events.get(3));
- assertEventEquals("@column", "select", "functional/alltypestiny/smallint_col", 1,
- events.get(4));
- assertEventEquals("@column", "select", "functional/alltypestiny/int_col", 1,
- events.get(5));
- assertEventEquals("@column", "select", "functional/alltypestiny/bigint_col", 1,
- events.get(6));
- assertEventEquals("@column", "select", "functional/alltypestiny/float_col", 1,
- events.get(7));
- assertEventEquals("@column", "select", "functional/alltypestiny/double_col", 1,
- events.get(8));
assertEventEquals("@column", "select", "functional/alltypestiny/string_col", 1,
- events.get(9));
- assertEventEquals("@column", "select", "functional/alltypestiny/timestamp_col", 1,
- events.get(10));
- assertEventEquals("@column", "select", "functional/alltypestiny/year", 1,
- events.get(11));
- assertEventEquals("@column", "select", "functional/alltypestiny/month", 1,
- events.get(12));
- assertEventEquals("@column", "mask_null",
- "functional/alltypestiny/date_string_col", 1, events.get(13));
+ events.get(3));
assertEventEquals("@column", "custom", "functional/alltypestiny/string_col", 1,
- events.get(14));
+ events.get(4));
}, "with iv as (select id, bool_col, string_col from functional.alltypestiny) " +
"select * from iv", onTable("functional", "alltypestiny",
TPrivilegeLevel.SELECT));
- // This query results in 6 calls to RangerImpalaPlugin#evalDataMaskPolicies() for
- // the column of 'string_col'. Two of them result from the first call to
- // SelectStmt#analyze() as described above. Two of them stem from the rewrite
- // operation of the subquery at StmtRewriter#SubqueryRewriter()#rewrite() and the
- // last two are due to the re-analysis of the query by SelectStmt#analyze().
- // TODO (IMPALA-9661): Remove checks for unused columns.
- authzOk(events -> {assertEquals(15, events.size());
+ // Test on masking subquery. No redundant audit events.
+ authzOk(events -> {assertEquals(4, events.size());
assertEquals("select id, string_col from functional.alltypestiny a " +
"where exists (select id from functional.alltypestiny where id = a.id) " +
"order by id;",
@@ -436,32 +425,10 @@ public class RangerAuditLogTest extends AuthorizationTestBase {
events.get(0));
assertEventEquals("@column", "select", "functional/alltypestiny/id", 1,
events.get(1));
- assertEventEquals("@column", "select", "functional/alltypestiny/bool_col", 1,
- events.get(2));
- assertEventEquals("@column", "select", "functional/alltypestiny/tinyint_col", 1,
- events.get(3));
- assertEventEquals("@column", "select", "functional/alltypestiny/smallint_col", 1,
- events.get(4));
- assertEventEquals("@column", "select", "functional/alltypestiny/int_col", 1,
- events.get(5));
- assertEventEquals("@column", "select", "functional/alltypestiny/bigint_col", 1,
- events.get(6));
- assertEventEquals("@column", "select", "functional/alltypestiny/float_col", 1,
- events.get(7));
- assertEventEquals("@column", "select", "functional/alltypestiny/double_col", 1,
- events.get(8));
assertEventEquals("@column", "select", "functional/alltypestiny/string_col", 1,
- events.get(9));
- assertEventEquals("@column", "select", "functional/alltypestiny/timestamp_col", 1,
- events.get(10));
- assertEventEquals("@column", "select", "functional/alltypestiny/year", 1,
- events.get(11));
- assertEventEquals("@column", "select", "functional/alltypestiny/month", 1,
- events.get(12));
- assertEventEquals("@column", "mask_null",
- "functional/alltypestiny/date_string_col", 1, events.get(13));
+ events.get(2));
assertEventEquals("@column", "custom", "functional/alltypestiny/string_col", 1,
- events.get(14));
+ events.get(3));
}, "select id, string_col from functional.alltypestiny a where exists " +
"(select id from functional.alltypestiny where id = a.id) order by id;",
onTable("functional", "alltypestiny", TPrivilegeLevel.SELECT));
@@ -541,6 +508,53 @@ public class RangerAuditLogTest extends AuthorizationTestBase {
createRangerPolicy(policyName, json);
}
+ // Verify row filter audits. Note that columns used in the row filter won't create
+ // column access audits.
+ authzOk(events -> {
+ assertEquals(3, events.size());
+ assertEquals("select bool_col from functional.alltypestiny",
+ events.get(0).getRequestData());
+ assertEventEquals("@table", "select", "functional/alltypestiny", 1,
+ events.get(0));
+ assertEventEquals("@column", "select", "functional/alltypestiny/bool_col", 1,
+ events.get(1));
+ assertEventEquals("@table", "row_filter", "functional/alltypestiny", 1,
+ events.get(2));
+ }, "select bool_col from functional.alltypestiny",
+ onTable("functional", "alltypestiny", TPrivilegeLevel.SELECT));
+
+ authzOk(events -> {
+ assertEquals(2, events.size());
+ assertEquals("select 1 from functional.alltypestiny",
+ events.get(0).getRequestData());
+ assertEventEquals("@table", "select", "functional/alltypestiny", 1,
+ events.get(0));
+ assertEventEquals("@table", "row_filter", "functional/alltypestiny", 1,
+ events.get(1));
+ }, "select 1 from functional.alltypestiny",
+ onTable("functional", "alltypestiny", TPrivilegeLevel.SELECT));
+
+ authzOk(events -> {
+ assertEquals(2, events.size());
+ assertEquals("select count(*) from functional.alltypestiny",
+ events.get(0).getRequestData());
+ assertEventEquals("@table", "select", "functional/alltypestiny", 1,
+ events.get(0));
+ assertEventEquals("@table", "row_filter", "functional/alltypestiny", 1,
+ events.get(1));
+ }, "select count(*) from functional.alltypestiny",
+ onTable("functional", "alltypestiny", TPrivilegeLevel.SELECT));
+
+ // No row filters applied to current user. So no audits of row_filter.
+ authzOk(events -> {
+ assertEquals(1, events.size());
+ assertEquals("select count(*) from functional.alltypes",
+ events.get(0).getRequestData());
+ assertEventEquals("@table", "select", "functional/alltypes", 1,
+ events.get(0));
+ }, "select count(*) from functional.alltypes",
+ onTable("functional", "alltypes", TPrivilegeLevel.SELECT));
+
authzOk(events -> {
assertEquals(15, events.size());
assertEquals("select * from functional.alltypestiny",
@@ -578,6 +592,7 @@ public class RangerAuditLogTest extends AuthorizationTestBase {
}, "select * from functional.alltypestiny", onTable("functional", "alltypestiny",
TPrivilegeLevel.SELECT));
+ // No row filters applied to current user. So no audits of row_filter.
authzOk(events -> {
assertEquals(14, events.size());
assertEquals("select * from functional.alltypes",
diff --git a/testdata/workloads/functional-query/queries/QueryTest/ranger_column_masking_and_row_filtering.test b/testdata/workloads/functional-query/queries/QueryTest/ranger_column_masking_and_row_filtering.test
index f431ee4..cc985ed 100644
--- a/testdata/workloads/functional-query/queries/QueryTest/ranger_column_masking_and_row_filtering.test
+++ b/testdata/workloads/functional-query/queries/QueryTest/ranger_column_masking_and_row_filtering.test
@@ -1,16 +1,17 @@
====
---- QUERY
-# Row-filtering policy keeps rows with "id % 3 = 0".
-# Column-masking policies mask "id" to "id + 100" and redact column "date_string_col".
+# Row-filtering policy keeps rows with "id % 3 = 0". Column-masking policies mask
+# "id" to "id + 100", "string_col" to NULL, and redact column "date_string_col".
# Note that row filtering policies take effects prior to any column masking policies,
# because column masking policies apply on result data.
-select id, bool_col, date_string_col, year, month from functional.alltypestiny
+select id, bool_col, int_col, string_col, date_string_col, year, month
+from functional.alltypestiny
---- RESULTS
-100,true,'nn/nn/nn',2009,1
-103,false,'nn/nn/nn',2009,2
-106,true,'nn/nn/nn',2009,4
+100,true,0,'NULL','nn/nn/nn',2009,1
+103,false,1,'NULL','nn/nn/nn',2009,2
+106,true,0,'NULL','nn/nn/nn',2009,4
---- TYPES
-INT,BOOLEAN,STRING,INT,INT
+INT,BOOLEAN,INT,STRING,STRING,INT,INT
====
---- QUERY
# Column-masking policies of functional.alltypes mask "id" to "-id" and redact column
@@ -32,3 +33,40 @@ select id, bool_col, date_string_col, year, month from functional.alltypes_view
---- TYPES
INT,BOOLEAN,STRING,INT,INT
====
+---- QUERY
+# Test with expr rewrite rules.
+select id, bool_col, string_col,
+ if (id <=> id, 0, 1),
+ case when bool_col <=> bool_col then 0 when true then 1 end,
+ coalesce(string_col, '0')
+from functional.alltypestiny
+---- RESULTS
+100,true,'NULL',0,0,'0'
+103,false,'NULL',0,0,'0'
+106,true,'NULL',0,0,'0'
+---- TYPES
+INT,BOOLEAN,STRING,TINYINT,TINYINT,STRING
+====
+---- QUERY
+# Test with statement rewrite rules on uncorrelated subquery.
+select a.id, a.int_col, a.string_col from functional_orc_def.alltypes a
+where a.id in (select id from functional.alltypestiny where bool_col = true)
+and a.id < 103
+---- RESULTS
+100,0,'0'
+---- TYPES
+INT,INT,STRING
+====
+---- QUERY
+# Test with statement rewrite rules on correlated subquery.
+select id, int_col from functional.alltypesagg a
+where int_col in
+ (select id from functional.alltypestiny t where a.id = t.id)
+---- RESULTS
+100,100
+100,100
+103,103
+106,106
+---- TYPES
+INT,INT
+====
diff --git a/testdata/workloads/functional-query/queries/QueryTest/ranger_row_filtering.test b/testdata/workloads/functional-query/queries/QueryTest/ranger_row_filtering.test
index 2f420a4..17be8d1 100644
--- a/testdata/workloads/functional-query/queries/QueryTest/ranger_row_filtering.test
+++ b/testdata/workloads/functional-query/queries/QueryTest/ranger_row_filtering.test
@@ -52,12 +52,108 @@ NULL,96,'1',true,2009,4
INT,INT,STRING,BOOLEAN,INT,INT
====
---- QUERY
+# Test set operation on the above two tables.
+select id, string_col, bool_col, year, month from functional.alltypestiny
+intersect
+select id, string_col, bool_col, year, month from functional.alltypessmall
+---- RESULTS
+0,'0',true,2009,1
+---- TYPES
+INT,STRING,BOOLEAN,INT,INT
+====
+---- QUERY
+# Row-filtering policy keeps rows with "year = 2009 and month = 1". Test on aggregate.
+select min(id), max(id), count(*) from functional.alltypes
+---- RESULTS
+0,309,310
+---- TYPES
+INT,INT,BIGINT
+====
+---- QUERY
+# Row-filtering policy keeps rows with "year = 2009 and month = 1". Test on aggregate.
+select count(distinct id), count(distinct string_col) from functional.alltypes
+---- RESULTS
+310,10
+---- TYPES
+BIGINT,BIGINT
+====
+---- QUERY
# Row-filtering policy keeps rows with "year = 2009 and month = 1". Test on aggregate.
-select count(*) from functional.alltypes
+select year, min(month), count(*) from functional.alltypes group by year
---- RESULTS
-310
+2009,1,310
---- TYPES
-BIGINT
+INT,INT,BIGINT
+====
+---- QUERY
+# Test row filter applied on subquery in HAVING clause.
+# functional_orc_def.alltypes has no row filters. functional.alltypes has a row filter
+# "year = 2009 and month = 1" which results in its row count to be 310.
+select year, month, max(id) from functional_orc_def.alltypes
+group by year, month
+having max(id) < 2 * (select count(*) from functional.alltypes)
+---- RESULTS
+2009,1,309
+2009,2,589
+---- TYPES
+INT,INT,INT
+====
+---- QUERY
+# Test row filter applied on subquery in SelectList.
+# functional.alltypestiny has row filter "id % 2 = 0", so its max(id) is 6.
+# functional.alltypes has row filter "year = 2009 and month = 1".
+SELECT id, int_col,
+ (select max(id) from functional.alltypestiny) as x,
+ case
+ when id < (select max(id) from functional.alltypestiny) then 0
+ else 1
+ end as y
+from functional.alltypes t2
+order by id
+limit 10
+---- RESULTS
+0,0,6,0
+1,1,6,0
+2,2,6,0
+3,3,6,0
+4,4,6,0
+5,5,6,0
+6,6,6,1
+7,7,6,1
+8,8,6,1
+9,9,6,1
+---- TYPES
+INT,INT,INT,TINYINT
+---- LABELS
+id,int_col,x,y
+====
+---- QUERY
+# Test row filter applied on subquery in SelectList.
+# functional.alltypestiny has row filter "id % 2 = 0", so its max(id) is 6.
+# functional.alltypes has row filter "year = 2009 and month = 1".
+SELECT id, count(*) over (partition by
+ case
+ when id > (select max(id) from functional.alltypestiny) then id
+ else id + 5
+ end) as x
+from functional.alltypes t2
+order by id
+limit 10
+---- RESULTS
+0,1
+1,1
+2,2
+3,2
+4,2
+5,2
+6,2
+7,2
+8,2
+9,2
+---- TYPES
+INT,BIGINT
+---- LABELS
+id,x
====
---- QUERY
# Test on local views. functional.alltypestiny has row filter "id % 2 = 0".
@@ -79,6 +175,10 @@ with alltypestiny as (select 1 as id)
select * from alltypestiny
---- RESULTS
1
+---- TYPES
+TINYINT
+---- LABELS
+id
====
---- QUERY
# Test on local views. Correctly ignore masking on local view names so row filter of
@@ -192,6 +292,18 @@ select * from $UNIQUE_DB.masked_tbl2;
INT,BOOLEAN,TINYINT,SMALLINT,INT,BIGINT,FLOAT,DOUBLE,STRING,STRING,TIMESTAMP,INT,INT
====
---- QUERY
+# Test on INSERT with local view
+create table $UNIQUE_DB.masked_tbl3 like functional.alltypestiny stored as textfile;
+with v as (select * from functional.alltypestiny where id < 4)
+insert into $UNIQUE_DB.masked_tbl3 partition(year, month) select * from v;
+select * from $UNIQUE_DB.masked_tbl3;
+---- RESULTS
+0,true,0,0,0,0,0,0,'01/01/09','0',2009-01-01 00:00:00,2009,1
+2,true,0,0,0,0,0,0,'02/01/09','0',2009-02-01 00:00:00,2009,2
+---- TYPES
+INT,BOOLEAN,TINYINT,SMALLINT,INT,BIGINT,FLOAT,DOUBLE,STRING,STRING,TIMESTAMP,INT,INT
+====
+---- QUERY
# Test on CreateView. Should not add row filters when used in sql generations.
create view $UNIQUE_DB.masked_view as select * from functional.alltypestiny;
show create view $UNIQUE_DB.masked_view;
diff --git a/tests/authorization/test_ranger.py b/tests/authorization/test_ranger.py
index 1411494..64927aa 100644
--- a/tests/authorization/test_ranger.py
+++ b/tests/authorization/test_ranger.py
@@ -1214,7 +1214,7 @@ class TestRanger(CustomClusterTestSuite):
admin_client = self.create_impala_client()
policy_cnt = 0
try:
- # 2 column masking policies and 1 row filtering policy on functional.alltypestiny.
+ # 3 column masking policies and 1 row filtering policy on functional.alltypestiny.
# The row filtering policy will take effect first, then the column masking policies
# mask the final results.
TestRanger._add_column_masking_policy(
@@ -1222,6 +1222,10 @@ class TestRanger(CustomClusterTestSuite):
"CUSTOM", "id + 100") # use column name 'id' directly
policy_cnt += 1
TestRanger._add_column_masking_policy(
+ unique_name + str(policy_cnt), user, "functional", "alltypestiny", "string_col",
+ "MASK_NULL")
+ policy_cnt += 1
+ TestRanger._add_column_masking_policy(
unique_name + str(policy_cnt), user, "functional", "alltypestiny",
"date_string_col", "MASK")
policy_cnt += 1