You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@impala.apache.org by st...@apache.org on 2022/08/18 01:52:56 UTC
[impala] branch master updated (d8c744aab -> 62e20d1ba)
This is an automated email from the ASF dual-hosted git repository.
stigahuang pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
from d8c744aab IMPALA-11281: Load table metadata for ResetMetadataStmt
new 46169e4ad IMPALA-11498: Change port range of TEZ's web UI server
new ff8466715 IMPALA-11257: (Addendum) Make KerberosPrograms optional
new 1f2e90a7f IMPALA-9442: (Addendum) Fix Ozone jar name for older Ozone
new 62e20d1ba IMPALA-11494: Don't always produce Ranger audit log for authorized query
The 4 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
CMakeLists.txt | 2 +-
bin/impala-config.sh | 2 +-
.../apache/impala/analysis/AnalysisContext.java | 3 +-
.../java/org/apache/impala/analysis/Analyzer.java | 18 ++++++------
.../impala/authorization/AuthorizationChecker.java | 2 +-
.../authorization/BaseAuthorizationChecker.java | 3 +-
.../ranger/RangerAuthorizationChecker.java | 18 ++++++++++--
.../authorization/AuthorizationTestBase.java | 33 ++++++++++++++++++++--
.../authorization/ranger/RangerAuditLogTest.java | 19 +++++++++++--
fe/src/test/resources/hive-site.xml.py | 1 +
10 files changed, 78 insertions(+), 23 deletions(-)
[impala] 03/04: IMPALA-9442: (Addendum) Fix Ozone jar name for older Ozone
Posted by st...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
stigahuang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
commit 1f2e90a7f92f3b5024f028212f5bdd79afc38ae7
Author: Michael Smith <mi...@cloudera.com>
AuthorDate: Mon Aug 15 13:46:39 2022 -0700
IMPALA-9442: (Addendum) Fix Ozone jar name for older Ozone
Ozone 1.2 changed the name of its hadoop adapter jar. Logic to link
against the jar in pom.xml was correctly updated, but the jar name used
in HADOOP_CLASSPATH assumed Ozone 1.2. Updates the jar in
HADOOP_CLASSPATH to use the correct jar for the detected Ozone version.
Change-Id: I7e401075268ebd81af8b8cac72504f0d1a4e59f1
Reviewed-on: http://gerrit.cloudera.org:8080/18852
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
bin/impala-config.sh | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/bin/impala-config.sh b/bin/impala-config.sh
index 42fb2924a..19868d806 100755
--- a/bin/impala-config.sh
+++ b/bin/impala-config.sh
@@ -717,7 +717,7 @@ export HADOOP_LIB_DIR=${HADOOP_LIB_DIR_OVERRIDE:-"${HADOOP_HOME}/lib"}
export HADOOP_CLASSPATH="${HADOOP_CLASSPATH-}"
# Add Ozone Hadoop filesystem implementation when using Ozone
if [ "${TARGET_FILESYSTEM}" = "ozone" ]; then
- OZONE_JAR="ozone-filesystem-hadoop3-${IMPALA_OZONE_VERSION}.jar"
+ OZONE_JAR="${IMPALA_OZONE_JAR_ARTIFACT_ID}-${IMPALA_OZONE_VERSION}.jar"
HADOOP_CLASSPATH="${HADOOP_CLASSPATH}:${OZONE_HOME}/share/ozone/lib/${OZONE_JAR}"
fi
# Add the path containing the hadoop-aws jar, which is required to access AWS from the
[impala] 01/04: IMPALA-11498: Change port range of TEZ's web UI server
Posted by st...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
stigahuang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
commit 46169e4ad29b5e5b4e50f88801f6a45da9764808
Author: Fang-Yu Rao <fa...@cloudera.com>
AuthorDate: Wed Aug 10 13:44:57 2022 -0700
IMPALA-11498: Change port range of TEZ's web UI server
After TEZ-4347, by default TEZ would attempt to start a web UI server
before opening a session. The default port range for the server
specified in TezConfiguration.java (in the TEZ repository) is
"50000-50050", which does not seem to be a good choice in Impala's
testing environment in that there are always some other client programs
holding those ports when TEZ attempts to start its web UI server. As a
result, TEZ could not bind a port in the port range to start its web UI
server, resulting in TEZ session not being created, which in turn failed
some end-to-end test relying on TEZ, e.g., test_json_file_unsupported.
This patch explicitly sets the port range to "32000-32100" to avoid port
collision based on the parameter of 'ip_local_port_range', which is
"32768-60999" on the machine where Impala's tests are run.
Testing:
- Verified in 2 Jenkins runs of exhaustive RELEASE build that we do not
see test_json_file_unsupported fail again after this patch.
Change-Id: I3d067953f943a762ad50b0561570e96f0dc39791
Reviewed-on: http://gerrit.cloudera.org:8080/18851
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
fe/src/test/resources/hive-site.xml.py | 1 +
1 file changed, 1 insertion(+)
diff --git a/fe/src/test/resources/hive-site.xml.py b/fe/src/test/resources/hive-site.xml.py
index db1ba97b2..3b3e5fb0b 100644
--- a/fe/src/test/resources/hive-site.xml.py
+++ b/fe/src/test/resources/hive-site.xml.py
@@ -118,6 +118,7 @@ if hive_major_version >= 3:
# We run YARN with Tez on the classpath directly
'tez.ignore.lib.uris': 'true',
'tez.use.cluster.hadoop-libs': 'true',
+ 'tez.am.tez-ui.webservice.port-range': '32000-32100',
# Some of the tests change the columns in a incompatible manner
# (eg. string to timestamp) this is disallowed by default in Hive-3 which causes
[impala] 02/04: IMPALA-11257: (Addendum) Make KerberosPrograms optional
Posted by st...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
stigahuang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
commit ff84667157c5c8e5c2ce02ba1db56bc2b757b872
Author: Michael Smith <mi...@cloudera.com>
AuthorDate: Tue Aug 16 11:24:58 2022 -0700
IMPALA-11257: (Addendum) Make KerberosPrograms optional
Fixing CMake warnings fixed detection logic for KerberosPrograms.
Security tests aren't run in all test suites, so make KerberosPrograms
optional to reduce basic dev/test requirements.
Change-Id: I76e5c3a36b806b746cc2b59bb5cdcd8251dc0f5d
Reviewed-on: http://gerrit.cloudera.org:8080/18859
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
CMakeLists.txt | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/CMakeLists.txt b/CMakeLists.txt
index 652cd2aec..d6c58972e 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -401,7 +401,7 @@ IMPALA_ADD_THIRDPARTY_LIB(krb5 ${KERBEROS_INCLUDE_DIR} "" ${KERBEROS_LIBRARY})
# We require certain binaries from the kerberos project for our automated kerberos
# testing.
-find_package(KerberosPrograms REQUIRED)
+find_package(KerberosPrograms)
# find curl headers and libs
find_package(Curl REQUIRED)
[impala] 04/04: IMPALA-11494: Don't always produce Ranger audit log for authorized query
Posted by st...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
stigahuang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
commit 62e20d1ba842a3f27395251c57dea9850f462fc9
Author: Fang-Yu Rao <fa...@cloudera.com>
AuthorDate: Thu Aug 11 16:37:23 2022 -0700
IMPALA-11494: Don't always produce Ranger audit log for authorized query
Before this patch, when Impala could not resolve a given table
'<db_name>.<tbl_name>' during the query analysis, Impala would still
attempt to register 2 privilege requests. One was for the table
'<tbl_name>' under the database '<db_name>' and the other was for the
table '<db_name>' under the database 'default'. The first one should be
registered since Impala had to determine whether such an access should
be allowed (even though in fact the table did not exist), whereas the
second one was incorrect in that 'default.<db_name>' definitely was not
'<db_name>.<tbl_name>' in general.
Furthermore, Impala always sent audit log entries to the Ranger server
for an authorized query against non-existing table(s).
The 2 facts described above resulted in Impala producing Ranger audit
log entries for the tables '<db_name>.<tbl_name>' and
'default.<db_name>' when a requesting user granted sufficient privileges
on the databases of '<db_name>' and 'default' submitted a query against
a non-existing table '<db_name>.<tbl_name>'. None of the audit log
entries should be generated because i) the privilege request for
'<db_name>.<tbl_name>' was allowed and '<db_name>.<tbl_name>' did not
exist, and ii) 'default.<db_name>' did not correspond to a table.
This patch fixes the 2 issues mentioned above so that Impala will not
generate any Ranger audit log entry for an authorized query against a
non-existing table.
Testing:
- Added a frontend test case to verify no Ranger audit log entry will
be produced for an authorized query against a non-existing table.
Change-Id: I701652e457d3118f43249e83be933713b17ce48f
Reviewed-on: http://gerrit.cloudera.org:8080/18850
Reviewed-by: Csaba Ringhofer <cs...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
.../apache/impala/analysis/AnalysisContext.java | 3 +-
.../java/org/apache/impala/analysis/Analyzer.java | 18 ++++++------
.../impala/authorization/AuthorizationChecker.java | 2 +-
.../authorization/BaseAuthorizationChecker.java | 3 +-
.../ranger/RangerAuthorizationChecker.java | 18 ++++++++++--
.../authorization/AuthorizationTestBase.java | 33 ++++++++++++++++++++--
.../authorization/ranger/RangerAuditLogTest.java | 19 +++++++++++--
7 files changed, 75 insertions(+), 21 deletions(-)
diff --git a/fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java b/fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java
index 9304ffbe5..bb21809a7 100644
--- a/fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java
+++ b/fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java
@@ -487,7 +487,8 @@ public class AnalysisContext {
} catch (AuthorizationException e) {
authException = e;
} finally {
- authzChecker.postAuthorize(authzCtx, authException == null);
+ authzChecker.postAuthorize(authzCtx, authException == null,
+ analysisException == null);
}
}
diff --git a/fe/src/main/java/org/apache/impala/analysis/Analyzer.java b/fe/src/main/java/org/apache/impala/analysis/Analyzer.java
index d0c4ac131..b2742e269 100644
--- a/fe/src/main/java/org/apache/impala/analysis/Analyzer.java
+++ b/fe/src/main/java/org/apache/impala/analysis/Analyzer.java
@@ -892,16 +892,16 @@ public class Analyzer {
}
return builder.build();
});
+ } else {
+ registerPrivReq(builder -> {
+ builder.onTableUnknownOwner(
+ getDefaultDb(), tableRawPath.get(0)).allOf(tableRef.getPrivilege());
+ if (tableRef.requireGrantOption()) {
+ builder.grantOption();
+ }
+ return builder.build();
+ });
}
-
- registerPrivReq(builder -> {
- builder.onTableUnknownOwner(
- getDefaultDb(), tableRawPath.get(0)).allOf(tableRef.getPrivilege());
- if (tableRef.requireGrantOption()) {
- builder.grantOption();
- }
- return builder.build();
- });
}
/**
diff --git a/fe/src/main/java/org/apache/impala/authorization/AuthorizationChecker.java b/fe/src/main/java/org/apache/impala/authorization/AuthorizationChecker.java
index 17749b561..10d698b2c 100644
--- a/fe/src/main/java/org/apache/impala/authorization/AuthorizationChecker.java
+++ b/fe/src/main/java/org/apache/impala/authorization/AuthorizationChecker.java
@@ -70,7 +70,7 @@ public interface AuthorizationChecker {
/**
* This method is to be executed after an authorization check has occurred.
*/
- void postAuthorize(AuthorizationContext authzCtx, boolean authzOk)
+ void postAuthorize(AuthorizationContext authzCtx, boolean authzOk, boolean analysisOk)
throws AuthorizationException, InternalException;
/**
diff --git a/fe/src/main/java/org/apache/impala/authorization/BaseAuthorizationChecker.java b/fe/src/main/java/org/apache/impala/authorization/BaseAuthorizationChecker.java
index 698332972..1ab05cf09 100644
--- a/fe/src/main/java/org/apache/impala/authorization/BaseAuthorizationChecker.java
+++ b/fe/src/main/java/org/apache/impala/authorization/BaseAuthorizationChecker.java
@@ -104,7 +104,8 @@ public abstract class BaseAuthorizationChecker implements AuthorizationChecker {
* Override this method to add custom post-authorization check.
*/
@Override
- public void postAuthorize(AuthorizationContext authzCtx, boolean authzOk) {
+ public void postAuthorize(AuthorizationContext authzCtx, boolean authzOk,
+ boolean analysisOk) {
if (authzCtx.getTimeline().isPresent()) {
EventSequence timeline = authzCtx.getTimeline().get();
long durationMs = timeline.markEvent(String.format("Authorization finished (%s)",
diff --git a/fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java b/fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java
index 03fe16ded..668d904a1 100644
--- a/fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java
+++ b/fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java
@@ -177,9 +177,10 @@ public class RangerAuthorizationChecker extends BaseAuthorizationChecker {
}
@Override
- public void postAuthorize(AuthorizationContext authzCtx, boolean authzOk) {
+ public void postAuthorize(AuthorizationContext authzCtx, boolean authzOk,
+ boolean analysisOk) {
Preconditions.checkArgument(authzCtx instanceof RangerAuthorizationContext);
- super.postAuthorize(authzCtx, authzOk);
+ super.postAuthorize(authzCtx, authzOk, analysisOk);
// Consolidate the audit log entries and apply the deduplicated column masking events
// to update the List of all AuthzAuditEvent's only if the authorization is
// successful.
@@ -194,7 +195,18 @@ public class RangerAuthorizationChecker extends BaseAuthorizationChecker {
}
RangerBufferAuditHandler auditHandler =
((RangerAuthorizationContext) authzCtx).getAuditHandler();
- auditHandler.flush();
+ if (authzOk && !analysisOk) {
+ // When the query was authorized, we do not send any audit log entry to the Ranger
+ // server if there was an AnalysisException during query analysis.
+ // We still have to call clear() to remove audit log entries in this case because
+ // the current test framework checks the contents in auditHandler.getAuthzEvents()
+ // to determine whether the correct audit events are collected.
+ auditHandler.getAuthzEvents().clear();
+ } else {
+ // We send audit log entries to the Ranger server only if authorization failed or
+ // analysis succeeded.
+ auditHandler.flush();
+ }
}
@Override
diff --git a/fe/src/test/java/org/apache/impala/authorization/AuthorizationTestBase.java b/fe/src/test/java/org/apache/impala/authorization/AuthorizationTestBase.java
index 082f35c46..bd65f9b8b 100644
--- a/fe/src/test/java/org/apache/impala/authorization/AuthorizationTestBase.java
+++ b/fe/src/test/java/org/apache/impala/authorization/AuthorizationTestBase.java
@@ -30,6 +30,7 @@ import org.apache.impala.authorization.ranger.RangerImpalaResourceBuilder;
import org.apache.impala.catalog.Role;
import org.apache.impala.catalog.ScalarFunction;
import org.apache.impala.catalog.Type;
+import org.apache.impala.common.AnalysisException;
import org.apache.impala.common.FrontendTestBase;
import org.apache.impala.common.ImpalaException;
import org.apache.impala.service.Frontend;
@@ -319,13 +320,22 @@ public abstract class AuthorizationTestBase extends FrontendTestBase {
*/
public AuthzTest ok(TPrivilege[]... privileges)
throws ImpalaException {
+ ok(/* expectAnalysisOk */ true, privileges);
+ return this;
+ }
+
+ /**
+ * This method runs with the specified privileges.
+ */
+ public AuthzTest ok(boolean expectAnalysisOk, TPrivilege[]... privileges)
+ throws ImpalaException {
for (WithPrincipal withPrincipal: buildWithPrincipals()) {
try {
withPrincipal.init(privileges);
if (context_ != null) {
- authzOk(context_, stmt_, withPrincipal);
+ authzOk(context_, stmt_, withPrincipal, expectAnalysisOk);
} else {
- authzOk(stmt_, withPrincipal);
+ authzOk(stmt_, withPrincipal, expectAnalysisOk);
}
} finally {
withPrincipal.cleanUp();
@@ -482,11 +492,21 @@ public abstract class AuthorizationTestBase extends FrontendTestBase {
}
private void authzOk(String stmt, WithPrincipal withPrincipal) throws ImpalaException {
- authzOk(authzCtx_, stmt, withPrincipal);
+ authzOk(authzCtx_, stmt, withPrincipal, /* expectAnalysisOk */ true);
+ }
+
+ private void authzOk(String stmt, WithPrincipal withPrincipal,
+ boolean expectAnalysisOk) throws ImpalaException {
+ authzOk(authzCtx_, stmt, withPrincipal, expectAnalysisOk);
}
private void authzOk(AnalysisContext context, String stmt, WithPrincipal withPrincipal)
throws ImpalaException {
+ authzOk(context, stmt, withPrincipal, /* expectAnalysisOk */ true);
+ }
+
+ private void authzOk(AnalysisContext context, String stmt, WithPrincipal withPrincipal,
+ boolean expectAnalysisOk) throws ImpalaException {
try {
LOG.info("Testing authzOk for {}", stmt);
parseAndAnalyze(stmt, context, authzFrontend_);
@@ -496,6 +516,13 @@ public abstract class AuthorizationTestBase extends FrontendTestBase {
throw new AuthorizationException(String.format(
"\nPrincipal: %s\nStatement: %s\nError: %s", withPrincipal.getName(),
stmt, e.getMessage(), e));
+ } catch (AnalysisException e) {
+ // We throw an AnalysisException only if we did not expect query analysis to fail.
+ if (expectAnalysisOk) {
+ throw new AnalysisException(String.format(
+ "\nPrincipal: %s\nStatement: %s\nError: %s", withPrincipal.getName(),
+ stmt, e.getMessage(), e));
+ }
}
}
diff --git a/fe/src/test/java/org/apache/impala/authorization/ranger/RangerAuditLogTest.java b/fe/src/test/java/org/apache/impala/authorization/ranger/RangerAuditLogTest.java
index 6f4e7b4ca..8c7e60cc5 100644
--- a/fe/src/test/java/org/apache/impala/authorization/ranger/RangerAuditLogTest.java
+++ b/fe/src/test/java/org/apache/impala/authorization/ranger/RangerAuditLogTest.java
@@ -53,8 +53,9 @@ public class RangerAuditLogTest extends AuthorizationTestBase {
}
@Override
- public void postAuthorize(AuthorizationContext authzCtx, boolean authzOk) {
- super.postAuthorize(authzCtx, authzOk);
+ public void postAuthorize(AuthorizationContext authzCtx, boolean authzOk,
+ boolean analysisOk) {
+ super.postAuthorize(authzCtx, authzOk, analysisOk);
authzCtx_ = authzCtx;
}
}
@@ -278,6 +279,13 @@ public class RangerAuditLogTest extends AuthorizationTestBase {
}, "select min(id) from functional.alltypes union all " +
"select max(id) from functional.alltypes",
onTable("functional", "alltypes", TPrivilegeLevel.SELECT));
+
+ // No audit log entry should be produced for an authorized query against a
+ // non-existing table.
+ authzOk(events -> {
+ assertEquals(0, events.size());
+ }, "select * from functional.non_existing_tbl",
+ /* expectAnalysisOk */ false, onDatabase("functional", TPrivilegeLevel.SELECT));
}
@Test
@@ -802,7 +810,12 @@ public class RangerAuditLogTest extends AuthorizationTestBase {
private void authzOk(Consumer<List<AuthzAuditEvent>> resultChecker, String stmt,
TPrivilege[]... privileges) throws ImpalaException {
- authorize(stmt).ok(privileges);
+ authzOk(resultChecker, stmt, /* expectAnalysisOk */ true, privileges);
+ }
+
+ private void authzOk(Consumer<List<AuthzAuditEvent>> resultChecker, String stmt,
+ boolean expectAnalysisOk, TPrivilege[]... privileges) throws ImpalaException {
+ authorize(stmt).ok(expectAnalysisOk, privileges);
RangerAuthorizationContext rangerCtx =
(RangerAuthorizationContext) authzChecker_.authzCtx_;
Preconditions.checkNotNull(rangerCtx);