You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@impala.apache.org by ta...@apache.org on 2019/04/22 15:28:07 UTC
[impala] branch master updated (8412c77 -> 67f77d4)
This is an automated email from the ASF dual-hosted git repository.
tarmstrong pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git.
from 8412c77 IMPALA-8270: fix MemTracker teardown in FeSupport
new 2fdc0a5 IMPALA-8293 (Part 1): Move SentryProxy out of CatalogServiceCatalog
new 67f77d4 IMPALA-8414: Skip header when parsing /proc/net/dev
The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
be/src/util/system-state-info.cc | 7 +-
.../impala/authorization/AuthorizationDelta.java | 63 ++++++++++++++
.../impala/authorization/AuthorizationFactory.java | 11 ++-
.../impala/authorization/AuthorizationManager.java | 11 +++
.../authorization/AuthorizationProvider.java | 2 +-
...nFactory.java => NoopAuthorizationFactory.java} | 13 ++-
.../ranger/RangerCatalogdAuthorizationManager.java | 18 +++-
.../sentry/SentryAuthorizationFactory.java | 7 +-
.../sentry/SentryCatalogdAuthorizationManager.java | 54 +++++++-----
.../sentry/SentryImpaladAuthorizationManager.java | 7 ++
.../impala/authorization/sentry/SentryProxy.java | 6 +-
.../impala/catalog/CatalogServiceCatalog.java | 99 ++++++++++------------
.../apache/impala/service/CatalogOpExecutor.java | 23 +++--
.../java/org/apache/impala/service/Frontend.java | 10 ++-
.../java/org/apache/impala/service/JniCatalog.java | 14 +--
.../org/apache/impala/service/JniFrontend.java | 4 +-
.../impala/analysis/AnalyzeAuthStmtsTest.java | 4 +-
.../org/apache/impala/analysis/AuditingTest.java | 10 +--
.../impala/analysis/AuthorizationStmtTest.java | 7 +-
.../apache/impala/analysis/AuthorizationTest.java | 21 +++--
.../impala/analysis/StmtMetadataLoaderTest.java | 6 +-
.../authorization/sentry/SentryProxyTest.java | 6 +-
.../apache/impala/catalog/AlterDatabaseTest.java | 9 +-
.../events/MetastoreEventsProcessorTest.java | 23 ++---
.../org/apache/impala/common/FrontendFixture.java | 22 +++--
.../org/apache/impala/common/FrontendTestBase.java | 19 ++++-
.../org/apache/impala/common/QueryFixture.java | 4 +-
.../impala/testutil/CatalogServiceTestCatalog.java | 22 ++---
.../apache/impala/testutil/ImpaladTestCatalog.java | 15 ++--
.../impala/testutil/PlannerTestCaseLoader.java | 8 +-
30 files changed, 334 insertions(+), 191 deletions(-)
create mode 100644 fe/src/main/java/org/apache/impala/authorization/AuthorizationDelta.java
rename fe/src/main/java/org/apache/impala/authorization/{NoneAuthorizationFactory.java => NoopAuthorizationFactory.java} (96%)
[impala] 01/02: IMPALA-8293 (Part 1): Move SentryProxy out of
CatalogServiceCatalog
Posted by ta...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
tarmstrong pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
commit 2fdc0a57d1b5ca85e47a4ac7dc955336fda693a6
Author: Fredy Wijaya <fw...@cloudera.com>
AuthorDate: Wed Apr 17 19:24:28 2019 -0500
IMPALA-8293 (Part 1): Move SentryProxy out of CatalogServiceCatalog
The patch refactors the SentryProxy, which does Sentry cache
invalidation from CatalogServiceCatalog into a Sentry-specific
implementation, i.e. SentryCatalogdAuthorizationManager. This patch
also adds a new method in the AuthorizationManager interface to allow
refreshing authorization for any authorization provider that caches the
authorization metadata.
This patch also contains minor clean-up to address comments in this
abandoned CR: https://gerrit.cloudera.org/c/12748
This patch has no functionality change.
Testing:
- Ran all FE tests
- Ran all E2E authorization tests
Change-Id: I3fa14abb09abfb4aaf6231d35114c0e121d6e568
Reviewed-on: http://gerrit.cloudera.org:8080/13065
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
.../impala/authorization/AuthorizationDelta.java | 63 ++++++++++++++
.../impala/authorization/AuthorizationFactory.java | 11 ++-
.../impala/authorization/AuthorizationManager.java | 11 +++
.../authorization/AuthorizationProvider.java | 2 +-
...nFactory.java => NoopAuthorizationFactory.java} | 13 ++-
.../ranger/RangerCatalogdAuthorizationManager.java | 18 +++-
.../sentry/SentryAuthorizationFactory.java | 7 +-
.../sentry/SentryCatalogdAuthorizationManager.java | 54 +++++++-----
.../sentry/SentryImpaladAuthorizationManager.java | 7 ++
.../impala/authorization/sentry/SentryProxy.java | 6 +-
.../impala/catalog/CatalogServiceCatalog.java | 99 ++++++++++------------
.../apache/impala/service/CatalogOpExecutor.java | 23 +++--
.../java/org/apache/impala/service/Frontend.java | 10 ++-
.../java/org/apache/impala/service/JniCatalog.java | 14 +--
.../org/apache/impala/service/JniFrontend.java | 4 +-
.../impala/analysis/AnalyzeAuthStmtsTest.java | 4 +-
.../org/apache/impala/analysis/AuditingTest.java | 10 +--
.../impala/analysis/AuthorizationStmtTest.java | 7 +-
.../apache/impala/analysis/AuthorizationTest.java | 21 +++--
.../impala/analysis/StmtMetadataLoaderTest.java | 6 +-
.../authorization/sentry/SentryProxyTest.java | 6 +-
.../apache/impala/catalog/AlterDatabaseTest.java | 9 +-
.../events/MetastoreEventsProcessorTest.java | 23 ++---
.../org/apache/impala/common/FrontendFixture.java | 22 +++--
.../org/apache/impala/common/FrontendTestBase.java | 19 ++++-
.../org/apache/impala/common/QueryFixture.java | 4 +-
.../impala/testutil/CatalogServiceTestCatalog.java | 22 ++---
.../apache/impala/testutil/ImpaladTestCatalog.java | 15 ++--
.../impala/testutil/PlannerTestCaseLoader.java | 8 +-
29 files changed, 329 insertions(+), 189 deletions(-)
diff --git a/fe/src/main/java/org/apache/impala/authorization/AuthorizationDelta.java b/fe/src/main/java/org/apache/impala/authorization/AuthorizationDelta.java
new file mode 100644
index 0000000..91e3e36
--- /dev/null
+++ b/fe/src/main/java/org/apache/impala/authorization/AuthorizationDelta.java
@@ -0,0 +1,63 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.impala.authorization;
+
+import com.google.common.base.Preconditions;
+import org.apache.impala.thrift.TCatalogObject;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+
+/**
+ * This class keeps track of authorization catalog objects added/removed, such as:
+ * - {@link org.apache.impala.catalog.Role}
+ * - {@link org.apache.impala.catalog.User}
+ * - {@link org.apache.impala.catalog.PrincipalPrivilege}
+ */
+public class AuthorizationDelta {
+ private final List<TCatalogObject> added_;
+ private final List<TCatalogObject> removed_;
+
+ public AuthorizationDelta() {
+ this(new ArrayList<>(), new ArrayList<>());
+ }
+
+ public AuthorizationDelta(List<TCatalogObject> added, List<TCatalogObject> removed) {
+ added_ = Preconditions.checkNotNull(added);
+ removed_ = Preconditions.checkNotNull(removed);
+ }
+
+ public AuthorizationDelta addCatalogObjectAdded(TCatalogObject catalogObject) {
+ added_.add(catalogObject);
+ return this;
+ }
+
+ public AuthorizationDelta addCatalogObjectRemoved(TCatalogObject catalogObject) {
+ removed_.add(catalogObject);
+ return this;
+ }
+
+ public List<TCatalogObject> getCatalogObjectsAdded() {
+ return Collections.unmodifiableList(added_);
+ }
+
+ public List<TCatalogObject> getCatalogObjectsRemoved() {
+ return Collections.unmodifiableList(removed_);
+ }
+}
\ No newline at end of file
diff --git a/fe/src/main/java/org/apache/impala/authorization/AuthorizationFactory.java b/fe/src/main/java/org/apache/impala/authorization/AuthorizationFactory.java
index 60e8701..bc85254 100644
--- a/fe/src/main/java/org/apache/impala/authorization/AuthorizationFactory.java
+++ b/fe/src/main/java/org/apache/impala/authorization/AuthorizationFactory.java
@@ -18,6 +18,7 @@
package org.apache.impala.authorization;
import org.apache.impala.catalog.CatalogServiceCatalog;
+import org.apache.impala.common.ImpalaException;
import org.apache.impala.service.BackendConfig;
import org.apache.impala.service.FeCatalogManager;
@@ -38,12 +39,13 @@ public interface AuthorizationFactory {
/**
* Creates a new instance of {@link AuthorizationChecker}.
*/
- AuthorizationChecker newAuthorizationChecker(AuthorizationPolicy authzPolicy);
+ AuthorizationChecker newAuthorizationChecker(AuthorizationPolicy authzPolicy)
+ throws ImpalaException;
/**
* Creates a new instance of {@link AuthorizationChecker}.
*/
- default AuthorizationChecker newAuthorizationChecker() {
+ default AuthorizationChecker newAuthorizationChecker() throws ImpalaException {
return newAuthorizationChecker(null);
}
@@ -61,10 +63,11 @@ public interface AuthorizationFactory {
* object may become stale.
*/
AuthorizationManager newAuthorizationManager(FeCatalogManager catalog,
- Supplier<? extends AuthorizationChecker> authzChecker);
+ Supplier<? extends AuthorizationChecker> authzChecker) throws ImpalaException;
/**
* Creates a new instance of {@link AuthorizationManager}.
*/
- AuthorizationManager newAuthorizationManager(CatalogServiceCatalog catalog);
+ AuthorizationManager newAuthorizationManager(CatalogServiceCatalog catalog)
+ throws ImpalaException;
}
diff --git a/fe/src/main/java/org/apache/impala/authorization/AuthorizationManager.java b/fe/src/main/java/org/apache/impala/authorization/AuthorizationManager.java
index e8ad5b9..529114a 100644
--- a/fe/src/main/java/org/apache/impala/authorization/AuthorizationManager.java
+++ b/fe/src/main/java/org/apache/impala/authorization/AuthorizationManager.java
@@ -18,6 +18,7 @@
package org.apache.impala.authorization;
import org.apache.hadoop.hive.metastore.api.PrincipalType;
+import org.apache.impala.catalog.CatalogException;
import org.apache.impala.common.ImpalaException;
import org.apache.impala.thrift.TCreateDropRoleParams;
import org.apache.impala.thrift.TDdlExecResponse;
@@ -122,4 +123,14 @@ public interface AuthorizationManager {
void updateTableOwnerPrivilege(String serverName, String databaseName, String tableName,
String oldOwner, PrincipalType oldOwnerType, String newOwner,
PrincipalType newOwnerType, TDdlExecResponse response) throws ImpalaException;
+
+ /**
+ * Performs a refresh authorization by updating the authorization catalog objects.
+ *
+ * @param resetVersions when resetVersions is true (used by INVALIDATE METADATA),
+ * catalog object versions will need to be incremented.
+ * @return {@link AuthorizationDelta} for the authorization catalog objects
+ * added/removed.
+ */
+ AuthorizationDelta refreshAuthorization(boolean resetVersions) throws ImpalaException;
}
diff --git a/fe/src/main/java/org/apache/impala/authorization/AuthorizationProvider.java b/fe/src/main/java/org/apache/impala/authorization/AuthorizationProvider.java
index 779f7b9..8e21689 100644
--- a/fe/src/main/java/org/apache/impala/authorization/AuthorizationProvider.java
+++ b/fe/src/main/java/org/apache/impala/authorization/AuthorizationProvider.java
@@ -24,5 +24,5 @@ package org.apache.impala.authorization;
public enum AuthorizationProvider {
SENTRY,
RANGER,
- NONE
+ NOOP
}
diff --git a/fe/src/main/java/org/apache/impala/authorization/NoneAuthorizationFactory.java b/fe/src/main/java/org/apache/impala/authorization/NoopAuthorizationFactory.java
similarity index 96%
rename from fe/src/main/java/org/apache/impala/authorization/NoneAuthorizationFactory.java
rename to fe/src/main/java/org/apache/impala/authorization/NoopAuthorizationFactory.java
index 01d5608..5a509f2 100644
--- a/fe/src/main/java/org/apache/impala/authorization/NoneAuthorizationFactory.java
+++ b/fe/src/main/java/org/apache/impala/authorization/NoopAuthorizationFactory.java
@@ -44,10 +44,10 @@ import java.util.function.Supplier;
* An implementation of {@link AuthorizationFactory} that does not do any
* authorization. This is the default implementation when authorization is disabled.
*/
-public class NoneAuthorizationFactory implements AuthorizationFactory {
+public class NoopAuthorizationFactory implements AuthorizationFactory {
private final AuthorizationConfig authzConfig_;
- public NoneAuthorizationFactory(BackendConfig backendConfig) {
+ public NoopAuthorizationFactory(BackendConfig backendConfig) {
Preconditions.checkNotNull(backendConfig);
authzConfig_ = disabledAuthorizationConfig();
}
@@ -56,7 +56,7 @@ public class NoneAuthorizationFactory implements AuthorizationFactory {
* This is for testing.
*/
@VisibleForTesting
- public NoneAuthorizationFactory() {
+ public NoopAuthorizationFactory() {
authzConfig_ = disabledAuthorizationConfig();
}
@@ -65,7 +65,7 @@ public class NoneAuthorizationFactory implements AuthorizationFactory {
@Override
public boolean isEnabled() { return false; }
@Override
- public AuthorizationProvider getProvider() { return AuthorizationProvider.NONE; }
+ public AuthorizationProvider getProvider() { return AuthorizationProvider.NOOP; }
@Override
public String getServerName() { return null; }
};
@@ -170,6 +170,11 @@ public class NoneAuthorizationFactory implements AuthorizationFactory {
throw new UnsupportedOperationException(String.format("%s is not supported",
ClassUtil.getMethodName()));
}
+
+ @Override
+ public AuthorizationDelta refreshAuthorization(boolean resetVersions) {
+ return new AuthorizationDelta();
+ }
}
@Override
diff --git a/fe/src/main/java/org/apache/impala/authorization/ranger/RangerCatalogdAuthorizationManager.java b/fe/src/main/java/org/apache/impala/authorization/ranger/RangerCatalogdAuthorizationManager.java
index ca768b8..28a9ba4 100644
--- a/fe/src/main/java/org/apache/impala/authorization/ranger/RangerCatalogdAuthorizationManager.java
+++ b/fe/src/main/java/org/apache/impala/authorization/ranger/RangerCatalogdAuthorizationManager.java
@@ -19,11 +19,21 @@ package org.apache.impala.authorization.ranger;
import com.google.common.annotations.VisibleForTesting;
import org.apache.hadoop.hive.metastore.api.PrincipalType;
+import org.apache.impala.authorization.AuthorizationDelta;
import org.apache.impala.authorization.AuthorizationManager;
import org.apache.impala.authorization.User;
import org.apache.impala.common.ImpalaException;
import org.apache.impala.common.InternalException;
-import org.apache.impala.thrift.*;
+import org.apache.impala.thrift.TCreateDropRoleParams;
+import org.apache.impala.thrift.TDdlExecResponse;
+import org.apache.impala.thrift.TGrantRevokePrivParams;
+import org.apache.impala.thrift.TGrantRevokeRoleParams;
+import org.apache.impala.thrift.TPrivilege;
+import org.apache.impala.thrift.TPrivilegeLevel;
+import org.apache.impala.thrift.TResultSet;
+import org.apache.impala.thrift.TShowGrantPrincipalParams;
+import org.apache.impala.thrift.TShowRolesParams;
+import org.apache.impala.thrift.TShowRolesResult;
import org.apache.impala.util.ClassUtil;
import org.apache.ranger.plugin.audit.RangerDefaultAuditHandler;
import org.apache.ranger.plugin.util.GrantRevokeRequest;
@@ -184,6 +194,12 @@ public class RangerCatalogdAuthorizationManager implements AuthorizationManager
PrincipalType newOwnerType, TDdlExecResponse response) throws ImpalaException {
}
+ @Override
+ public AuthorizationDelta refreshAuthorization(boolean resetVersions) {
+ // TODO: IMPALA-8293 (part 2)
+ return new AuthorizationDelta();
+ }
+
public static List<GrantRevokeRequest> createGrantRevokeRequests(String grantor,
String user, List<String> groups, String clusterName, List<TPrivilege> privileges) {
List<GrantRevokeRequest> requests = new ArrayList<>();
diff --git a/fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationFactory.java b/fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationFactory.java
index d2bb263..2ad1ec3 100644
--- a/fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationFactory.java
+++ b/fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationFactory.java
@@ -27,6 +27,7 @@ import org.apache.impala.authorization.AuthorizationManager;
import org.apache.impala.authorization.AuthorizationPolicy;
import org.apache.impala.authorization.AuthorizationFactory;
import org.apache.impala.catalog.CatalogServiceCatalog;
+import org.apache.impala.common.ImpalaException;
import org.apache.impala.service.BackendConfig;
import org.apache.impala.service.FeCatalogManager;
@@ -95,7 +96,9 @@ public class SentryAuthorizationFactory implements AuthorizationFactory {
}
@Override
- public AuthorizationManager newAuthorizationManager(CatalogServiceCatalog catalog) {
- return new SentryCatalogdAuthorizationManager(catalog);
+ public AuthorizationManager newAuthorizationManager(CatalogServiceCatalog catalog)
+ throws ImpalaException {
+ return new SentryCatalogdAuthorizationManager(
+ (SentryAuthorizationConfig) getAuthorizationConfig(), catalog);
}
}
diff --git a/fe/src/main/java/org/apache/impala/authorization/sentry/SentryCatalogdAuthorizationManager.java b/fe/src/main/java/org/apache/impala/authorization/sentry/SentryCatalogdAuthorizationManager.java
index ed9108c..0c3ca3c 100644
--- a/fe/src/main/java/org/apache/impala/authorization/sentry/SentryCatalogdAuthorizationManager.java
+++ b/fe/src/main/java/org/apache/impala/authorization/sentry/SentryCatalogdAuthorizationManager.java
@@ -20,6 +20,7 @@ package org.apache.impala.authorization.sentry;
import com.google.common.base.Preconditions;
import com.google.common.collect.Lists;
import org.apache.hadoop.hive.metastore.api.PrincipalType;
+import org.apache.impala.authorization.AuthorizationDelta;
import org.apache.impala.authorization.AuthorizationManager;
import org.apache.impala.authorization.User;
import org.apache.impala.catalog.CatalogException;
@@ -28,6 +29,7 @@ import org.apache.impala.catalog.Principal;
import org.apache.impala.catalog.PrincipalPrivilege;
import org.apache.impala.catalog.Role;
import org.apache.impala.common.ImpalaException;
+import org.apache.impala.common.ImpalaRuntimeException;
import org.apache.impala.common.InternalException;
import org.apache.impala.common.Reference;
import org.apache.impala.thrift.TCatalogObject;
@@ -47,6 +49,7 @@ import org.apache.impala.util.ClassUtil;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
+import java.util.ArrayList;
import java.util.List;
/**
@@ -65,17 +68,21 @@ public class SentryCatalogdAuthorizationManager implements AuthorizationManager
LoggerFactory.getLogger(SentryCatalogdAuthorizationManager.class);
private final CatalogServiceCatalog catalog_;
-
- public SentryCatalogdAuthorizationManager(CatalogServiceCatalog catalog) {
- Preconditions.checkNotNull(catalog);
- catalog_ = catalog;
+ // Proxy to access the Sentry Service and also periodically refreshes the
+ // policy metadata. Null if Sentry Service is not enabled.
+ private final SentryProxy sentryProxy_;
+
+ public SentryCatalogdAuthorizationManager(SentryAuthorizationConfig authzConfig,
+ CatalogServiceCatalog catalog) throws ImpalaException {
+ sentryProxy_ = new SentryProxy(Preconditions.checkNotNull(authzConfig), catalog);
+ catalog_ = Preconditions.checkNotNull(catalog);
}
/**
* Checks if the given user is a Sentry admin.
*/
public boolean isSentryAdmin(User user) throws ImpalaException {
- return catalog_.getSentryProxy().isSentryAdmin(user);
+ return sentryProxy_.isSentryAdmin(user);
}
@Override
@@ -83,8 +90,7 @@ public class SentryCatalogdAuthorizationManager implements AuthorizationManager
TDdlExecResponse response) throws ImpalaException {
verifySentryServiceEnabled();
- Role role = catalog_.getSentryProxy().createRole(requestingUser,
- params.getRole_name());
+ Role role = sentryProxy_.createRole(requestingUser, params.getRole_name());
Preconditions.checkNotNull(role);
TCatalogObject catalogObject = new TCatalogObject();
@@ -100,7 +106,7 @@ public class SentryCatalogdAuthorizationManager implements AuthorizationManager
TDdlExecResponse response) throws ImpalaException {
verifySentryServiceEnabled();
- Role role = catalog_.getSentryProxy().dropRole(requestingUser, params.getRole_name());
+ Role role = sentryProxy_.dropRole(requestingUser, params.getRole_name());
if (role == null) {
// Nothing was removed from the catalogd's cache.
response.result.setVersion(catalog_.getCatalogVersion());
@@ -131,8 +137,7 @@ public class SentryCatalogdAuthorizationManager implements AuthorizationManager
String roleName = params.getRole_names().get(0);
String groupName = params.getGroup_names().get(0);
- Role role = catalog_.getSentryProxy().grantRoleGroup(requestingUser, roleName,
- groupName);
+ Role role = sentryProxy_.grantRoleGroup(requestingUser, roleName, groupName);
Preconditions.checkNotNull(role);
TCatalogObject catalogObject = new TCatalogObject();
catalogObject.setType(role.getCatalogObjectType());
@@ -151,8 +156,7 @@ public class SentryCatalogdAuthorizationManager implements AuthorizationManager
String roleName = params.getRole_names().get(0);
String groupName = params.getGroup_names().get(0);
- Role role = catalog_.getSentryProxy().revokeRoleGroup(requestingUser, roleName,
- groupName);
+ Role role = sentryProxy_.revokeRoleGroup(requestingUser, roleName, groupName);
Preconditions.checkNotNull(role);
TCatalogObject catalogObject = new TCatalogObject();
catalogObject.setType(role.getCatalogObjectType());
@@ -178,8 +182,8 @@ public class SentryCatalogdAuthorizationManager implements AuthorizationManager
List<PrincipalPrivilege> removedGrantOptPrivileges =
Lists.newArrayListWithExpectedSize(privileges.size());
List<PrincipalPrivilege> addedRolePrivileges =
- catalog_.getSentryProxy().grantRolePrivileges(requestingUser, roleName,
- privileges, params.isHas_grant_opt(), removedGrantOptPrivileges);
+ sentryProxy_.grantRolePrivileges(requestingUser, roleName, privileges,
+ params.isHas_grant_opt(), removedGrantOptPrivileges);
Preconditions.checkNotNull(addedRolePrivileges);
List<TCatalogObject> updatedPrivs =
@@ -229,8 +233,8 @@ public class SentryCatalogdAuthorizationManager implements AuthorizationManager
List<PrincipalPrivilege> addedRolePrivileges =
Lists.newArrayListWithExpectedSize(privileges.size());
List<PrincipalPrivilege> removedGrantOptPrivileges =
- catalog_.getSentryProxy().revokeRolePrivileges(requestingUser, roleName,
- privileges, params.isHas_grant_opt(), addedRolePrivileges);
+ sentryProxy_.revokeRolePrivileges(requestingUser, roleName, privileges,
+ params.isHas_grant_opt(), addedRolePrivileges);
Preconditions.checkNotNull(addedRolePrivileges);
List<TCatalogObject> updatedPrivs =
@@ -301,7 +305,7 @@ public class SentryCatalogdAuthorizationManager implements AuthorizationManager
String oldOwner, PrincipalType oldOwnerType, String newOwner,
PrincipalType newOwnerType, TDdlExecResponse response) throws ImpalaException {
verifySentryServiceEnabled();
- if (!catalog_.getSentryProxy().isObjectOwnershipEnabled()) return;
+ if (!sentryProxy_.isObjectOwnershipEnabled()) return;
Preconditions.checkNotNull(serverName);
Preconditions.checkNotNull(databaseName);
@@ -335,7 +339,7 @@ public class SentryCatalogdAuthorizationManager implements AuthorizationManager
String tableName, String oldOwner, PrincipalType oldOwnerType, String newOwner,
PrincipalType newOwnerType, TDdlExecResponse response) throws ImpalaException {
verifySentryServiceEnabled();
- if (!catalog_.getSentryProxy().isObjectOwnershipEnabled()) return;
+ if (!sentryProxy_.isObjectOwnershipEnabled()) return;
Preconditions.checkNotNull(serverName);
Preconditions.checkNotNull(databaseName);
@@ -346,6 +350,15 @@ public class SentryCatalogdAuthorizationManager implements AuthorizationManager
filter);
}
+ @Override
+ public AuthorizationDelta refreshAuthorization(boolean resetVersions)
+ throws ImpalaException {
+ List<TCatalogObject> catalogObjectsAdded = new ArrayList<>();
+ List<TCatalogObject> catalogObjectsRemoved = new ArrayList<>();
+ sentryProxy_.refresh(resetVersions, catalogObjectsAdded, catalogObjectsRemoved);
+ return new AuthorizationDelta(catalogObjectsAdded, catalogObjectsRemoved);
+ }
+
private void updateOwnerPrivilege(String oldOwner, PrincipalType oldOwnerType,
String newOwner, PrincipalType newOwnerType, TDdlExecResponse response,
TPrivilege filter) {
@@ -361,7 +374,7 @@ public class SentryCatalogdAuthorizationManager implements AuthorizationManager
* Throws a CatalogException if the Sentry Service is not enabled.
*/
private void verifySentryServiceEnabled() throws CatalogException {
- if (catalog_.getSentryProxy() == null) {
+ if (sentryProxy_ == null) {
throw new CatalogException("Sentry Service is not enabled on the " +
"CatalogServer.");
}
@@ -371,8 +384,7 @@ public class SentryCatalogdAuthorizationManager implements AuthorizationManager
* Checks if with grant is enabled for object ownership in Sentry.
*/
private boolean isObjectOwnershipGrantEnabled() throws ImpalaException {
- return catalog_.getSentryProxy() == null ? false :
- catalog_.getSentryProxy().isObjectOwnershipGrantEnabled();
+ return sentryProxy_ == null ? false : sentryProxy_.isObjectOwnershipGrantEnabled();
}
/**
diff --git a/fe/src/main/java/org/apache/impala/authorization/sentry/SentryImpaladAuthorizationManager.java b/fe/src/main/java/org/apache/impala/authorization/sentry/SentryImpaladAuthorizationManager.java
index 932a267..92c13b3 100644
--- a/fe/src/main/java/org/apache/impala/authorization/sentry/SentryImpaladAuthorizationManager.java
+++ b/fe/src/main/java/org/apache/impala/authorization/sentry/SentryImpaladAuthorizationManager.java
@@ -23,6 +23,7 @@ import com.google.common.collect.Lists;
import com.google.common.collect.Sets;
import org.apache.hadoop.hive.metastore.api.PrincipalType;
import org.apache.impala.authorization.AuthorizationChecker;
+import org.apache.impala.authorization.AuthorizationDelta;
import org.apache.impala.authorization.AuthorizationException;
import org.apache.impala.authorization.AuthorizationManager;
import org.apache.impala.authorization.User;
@@ -221,6 +222,12 @@ public class SentryImpaladAuthorizationManager implements AuthorizationManager {
"%s is not supported in Impalad", ClassUtil.getMethodName()));
}
+ @Override
+ public AuthorizationDelta refreshAuthorization(boolean resetVersions) {
+ throw new UnsupportedOperationException(String.format(
+ "%s is not supported in Impalad", ClassUtil.getMethodName()));
+ }
+
/**
* Validates if the given user is a Sentry admin. The Sentry admin check will make an
* RPC call to the Catalog server. This check is necessary because some operations
diff --git a/fe/src/main/java/org/apache/impala/authorization/sentry/SentryProxy.java b/fe/src/main/java/org/apache/impala/authorization/sentry/SentryProxy.java
index 1dc6dbb..d2f7c9a 100644
--- a/fe/src/main/java/org/apache/impala/authorization/sentry/SentryProxy.java
+++ b/fe/src/main/java/org/apache/impala/authorization/sentry/SentryProxy.java
@@ -111,13 +111,14 @@ public class SentryProxy {
// The value for the object ownership config.
private final String objectOwnershipConfigValue_;
- public SentryProxy(SentryAuthorizationConfig authzConfig, CatalogServiceCatalog catalog,
- String kerberosPrincipal) throws ImpalaException {
+ public SentryProxy(SentryAuthorizationConfig authzConfig, CatalogServiceCatalog catalog)
+ throws ImpalaException {
Preconditions.checkNotNull(authzConfig);
SentryConfig sentryConfig = authzConfig.getSentryConfig();
Preconditions.checkNotNull(catalog);
Preconditions.checkNotNull(sentryConfig);
catalog_ = catalog;
+ String kerberosPrincipal = BackendConfig.INSTANCE.getBackendCfg().getPrincipal();
if (Strings.isNullOrEmpty(kerberosPrincipal)) {
processUser_ = new User(System.getProperty("user.name"));
} else {
@@ -647,5 +648,4 @@ public class SentryProxy {
}
return false;
}
-
}
diff --git a/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java b/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
index f3690ad..8bc6025 100644
--- a/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
+++ b/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
@@ -44,11 +44,9 @@ import org.apache.hadoop.hive.metastore.api.Database;
import org.apache.hadoop.hive.metastore.api.NoSuchObjectException;
import org.apache.hadoop.hive.metastore.api.UnknownDBException;
import org.apache.impala.analysis.TableName;
-import org.apache.impala.authorization.AuthorizationConfig;
+import org.apache.impala.authorization.AuthorizationDelta;
+import org.apache.impala.authorization.AuthorizationManager;
import org.apache.impala.authorization.AuthorizationPolicy;
-import org.apache.impala.authorization.AuthorizationProvider;
-import org.apache.impala.authorization.sentry.SentryAuthorizationConfig;
-import org.apache.impala.authorization.sentry.SentryProxy;
import org.apache.impala.catalog.MetaStoreClientPool.MetaStoreClient;
import org.apache.impala.catalog.events.ExternalEventsProcessor;
import org.apache.impala.catalog.events.MetastoreEventsProcessor;
@@ -186,6 +184,11 @@ public class CatalogServiceCatalog extends Catalog {
private static final int INITIAL_META_STORE_CLIENT_POOL_SIZE = 10;
private static final int MAX_NUM_SKIPPED_TOPIC_UPDATES = 2;
+ // Timeout for acquiring a table lock
+ // TODO: Make this configurable
+ private static final long TBL_LOCK_TIMEOUT_MS = 7200000;
+ // Time to sleep before retrying to acquire a table lock
+ private static final int TBL_LOCK_RETRY_MS = 10;
private final TUniqueId catalogServiceId_;
@@ -220,10 +223,6 @@ public class CatalogServiceCatalog extends Catalog {
private final ScheduledExecutorService cachePoolReader_ =
Executors.newScheduledThreadPool(1);
- // Proxy to access the Sentry Service and also periodically refreshes the
- // policy metadata. Null if Sentry Service is not enabled.
- private final SentryProxy sentryProxy_;
-
// Log of deleted catalog objects.
private final CatalogDeltaLog deleteLog_;
@@ -265,17 +264,20 @@ public class CatalogServiceCatalog extends Catalog {
private final Semaphore partialObjectFetchAccess_ =
new Semaphore(MAX_PARALLEL_PARTIAL_FETCH_RPC_COUNT, /*fair =*/ true);
- /**
- * Initialize the CatalogServiceCatalog using a given MetastoreClientPool impl.
- * @param loadInBackground If true, table metadata will be loaded in the background.
- * @param numLoadingThreads Number of threads used to load table metadata.
- * @param metaStoreClientPool A pool of HMS clients backing this Catalog.
- * @throws ImpalaException
- */
+ private AuthorizationManager authzManager_;
+
+ /**
+ * Initialize the CatalogServiceCatalog using a given MetastoreClientPool impl.
+ *
+ * @param loadInBackground If true, table metadata will be loaded in the background.
+ * @param numLoadingThreads Number of threads used to load table metadata.
+ * @param metaStoreClientPool A pool of HMS clients backing this Catalog.
+ * @throws ImpalaException
+ */
public CatalogServiceCatalog(boolean loadInBackground, int numLoadingThreads,
- AuthorizationConfig authConfig, TUniqueId catalogServiceId,
- String kerberosPrincipal, String localLibraryPath,
- MetaStoreClientPool metaStoreClientPool) throws ImpalaException {
+ TUniqueId catalogServiceId, String localLibraryPath,
+ MetaStoreClientPool metaStoreClientPool)
+ throws ImpalaException {
super(metaStoreClientPool);
catalogServiceId_ = catalogServiceId;
tableLoadingMgr_ = new TableLoadingMgr(this, numLoadingThreads);
@@ -290,13 +292,6 @@ public class CatalogServiceCatalog extends Catalog {
} catch (IOException e) {
LOG.error("Couldn't identify the default FS. Cache Pool reader will be disabled.");
}
- if (authConfig != null && authConfig.isEnabled() &&
- authConfig.getProvider() == AuthorizationProvider.SENTRY) {
- sentryProxy_ = new SentryProxy((SentryAuthorizationConfig) authConfig, this,
- kerberosPrincipal);
- } else {
- sentryProxy_ = null;
- }
localLibraryPath_ = localLibraryPath;
deleteLog_ = new CatalogDeltaLog();
topicMode_ = TopicMode.valueOf(
@@ -310,6 +305,23 @@ public class CatalogServiceCatalog extends Catalog {
}
/**
+ * Initializes the Catalog using the default MetastoreClientPool impl.
+ * @param initialHmsCnxnTimeoutSec Time (in seconds) CatalogServiceCatalog will wait
+ * to establish an initial connection to the HMS before giving up.
+ */
+ public CatalogServiceCatalog(boolean loadInBackground, int numLoadingThreads,
+ int initialHmsCnxnTimeoutSec, TUniqueId catalogServiceId, String localLibraryPath)
+ throws ImpalaException {
+ this(loadInBackground, numLoadingThreads, catalogServiceId, localLibraryPath,
+ new MetaStoreClientPool(INITIAL_META_STORE_CLIENT_POOL_SIZE,
+ initialHmsCnxnTimeoutSec));
+ }
+
+ public void setAuthzManager(AuthorizationManager authzManager) {
+ authzManager_ = Preconditions.checkNotNull(authzManager);
+ }
+
+ /**
* Returns a Metastore event processor object if
* <code>BackendConfig#getHMSPollingIntervalInSeconds</code> returns a non-zero
*.value of polling interval. Otherwise, returns a no-op events processor. It is
@@ -348,27 +360,6 @@ public class CatalogServiceCatalog extends Catalog {
}
/**
- * Initializes the Catalog using the default MetastoreClientPool impl.
- * @param initialHmsCnxnTimeoutSec Time (in seconds) CatalogServiceCatalog will wait
- * to establish an initial connection to the HMS before giving up.
- */
-
- public CatalogServiceCatalog(boolean loadInBackground, int numLoadingThreads,
- int initialHmsCnxnTimeoutSec, AuthorizationConfig authConfig,
- TUniqueId catalogServiceId, String kerberosPrincipal, String localLibraryPath)
- throws ImpalaException {
- this(loadInBackground, numLoadingThreads, authConfig, catalogServiceId,
- kerberosPrincipal, localLibraryPath, new MetaStoreClientPool(
- INITIAL_META_STORE_CLIENT_POOL_SIZE, initialHmsCnxnTimeoutSec));
- }
-
- // Timeout for acquiring a table lock
- // TODO: Make this configurable
- private static final long TBL_LOCK_TIMEOUT_MS = 7200000;
- // Time to sleep before retrying to acquire a table lock
- private static final int TBL_LOCK_RETRY_MS = 10;
-
- /**
* Tries to acquire versionLock_ and the lock of 'tbl' in that order. Returns true if it
* successfully acquires both within TBL_LOCK_TIMEOUT_MS millisecs; both locks are held
* when the function returns. Returns false otherwise and no lock is held in this case.
@@ -1367,16 +1358,14 @@ public class CatalogServiceCatalog extends Catalog {
}
/**
- * Refreshes Sentry authorization metadata. When authorization is not enabled, this
+ * Refreshes authorization metadata. When authorization is not enabled, this
* method is a no-op.
*/
- public void refreshAuthorization(boolean resetVersions, List<TCatalogObject> added,
- List<TCatalogObject> removed) throws CatalogException {
- // Do nothing if authorization is not enabled.
- if (sentryProxy_ == null) return;
+ public AuthorizationDelta refreshAuthorization(boolean resetVersions)
+ throws CatalogException {
+ Preconditions.checkState(authzManager_ != null);
try {
- // Update the authorization policy, waiting for the result to complete.
- sentryProxy_.refresh(resetVersions, added, removed);
+ return authzManager_.refreshAuthorization(resetVersions);
} catch (Exception e) {
throw new CatalogException("Error refreshing authorization policy: ", e);
}
@@ -1392,8 +1381,7 @@ public class CatalogServiceCatalog extends Catalog {
long currentCatalogVersion = getCatalogVersion();
LOG.info("Invalidating all metadata. Version: " + currentCatalogVersion);
// First update the policy metadata.
- refreshAuthorization(true, /*catalog objects added*/ new ArrayList<>(),
- /*catalog objects removed*/ new ArrayList<>());
+ refreshAuthorization(true);
// Even though we get the current notification event id before stopping the event
// processing here there is a small window of time where we could re-process some of
@@ -2349,7 +2337,6 @@ public class CatalogServiceCatalog extends Catalog {
}
public ReentrantReadWriteLock getLock() { return versionLock_; }
- public SentryProxy getSentryProxy() { return sentryProxy_; }
public AuthorizationPolicy getAuthPolicy() { return authPolicy_; }
/**
diff --git a/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java b/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
index 0f27510..b1ffc20 100644
--- a/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
+++ b/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
@@ -54,7 +54,7 @@ import org.apache.impala.analysis.AlterTableSortByStmt;
import org.apache.impala.analysis.FunctionName;
import org.apache.impala.analysis.TableName;
import org.apache.impala.authorization.AuthorizationConfig;
-import org.apache.impala.authorization.AuthorizationFactory;
+import org.apache.impala.authorization.AuthorizationDelta;
import org.apache.impala.authorization.AuthorizationManager;
import org.apache.impala.authorization.User;
import org.apache.impala.catalog.CatalogException;
@@ -267,13 +267,12 @@ public class CatalogOpExecutor {
// catalog_ and the corresponding RPC to apply the change in HMS are atomic.
private final Object metastoreDdlLock_ = new Object();
- public CatalogOpExecutor(CatalogServiceCatalog catalog,
- AuthorizationFactory authzFactory) {
- Preconditions.checkNotNull(catalog);
- Preconditions.checkNotNull(authzFactory);
- catalog_ = catalog;
- authzConfig_ = authzFactory.getAuthorizationConfig();
- authzManager_ = authzFactory.newAuthorizationManager(catalog);
+ public CatalogOpExecutor(CatalogServiceCatalog catalog, AuthorizationConfig authzConfig,
+ AuthorizationManager authzManager) throws ImpalaException {
+ Preconditions.checkNotNull(authzManager);
+ catalog_ = Preconditions.checkNotNull(catalog);
+ authzConfig_ = Preconditions.checkNotNull(authzConfig);
+ authzManager_ = Preconditions.checkNotNull(authzManager);
}
public CatalogServiceCatalog getCatalog() { return catalog_; }
@@ -3513,11 +3512,9 @@ public class CatalogOpExecutor {
}
resp.getResult().setVersion(updatedThriftTable.getCatalog_version());
} else if (req.isAuthorization()) {
- List<TCatalogObject> added = new ArrayList<>();
- List<TCatalogObject> removed = new ArrayList<>();
- catalog_.refreshAuthorization(false, added, removed);
- resp.result.setUpdated_catalog_objects(added);
- resp.result.setRemoved_catalog_objects(removed);
+ AuthorizationDelta authzDelta = catalog_.refreshAuthorization(false);
+ resp.result.setUpdated_catalog_objects(authzDelta.getCatalogObjectsAdded());
+ resp.result.setRemoved_catalog_objects(authzDelta.getCatalogObjectsRemoved());
resp.result.setVersion(catalog_.getCatalogVersion());
} else {
// Invalidate the entire catalog if no table name is provided.
diff --git a/fe/src/main/java/org/apache/impala/service/Frontend.java b/fe/src/main/java/org/apache/impala/service/Frontend.java
index f142245..16f1580 100644
--- a/fe/src/main/java/org/apache/impala/service/Frontend.java
+++ b/fe/src/main/java/org/apache/impala/service/Frontend.java
@@ -237,7 +237,7 @@ public class Frontend {
private final ImpaladTableUsageTracker impaladTableUsageTracker_;
- public Frontend(AuthorizationFactory authzFactory) {
+ public Frontend(AuthorizationFactory authzFactory) throws ImpalaException {
this(authzFactory, FeCatalogManager.createFromBackendConfig());
}
@@ -246,11 +246,13 @@ public class Frontend {
* updates and will be used for all requests.
*/
@VisibleForTesting
- public Frontend(AuthorizationFactory authzFactory, FeCatalog testCatalog) {
+ public Frontend(AuthorizationFactory authzFactory, FeCatalog testCatalog)
+ throws ImpalaException {
this(authzFactory, FeCatalogManager.createForTests(testCatalog));
}
- private Frontend(AuthorizationFactory authzFactory, FeCatalogManager catalogManager) {
+ private Frontend(AuthorizationFactory authzFactory, FeCatalogManager catalogManager)
+ throws ImpalaException {
catalogManager_ = catalogManager;
authzFactory_ = authzFactory;
@@ -278,7 +280,7 @@ public class Frontend {
}
public TUpdateCatalogCacheResponse updateCatalogCache(
- TUpdateCatalogCacheRequest req) throws CatalogException, TException {
+ TUpdateCatalogCacheRequest req) throws ImpalaException, TException {
TUpdateCatalogCacheResponse resp = catalogManager_.updateCatalogCache(req);
if (!req.is_delta) {
// In the case that it was a non-delta update, the catalog might have reloaded
diff --git a/fe/src/main/java/org/apache/impala/service/JniCatalog.java b/fe/src/main/java/org/apache/impala/service/JniCatalog.java
index eae7fd1..6d2700c 100644
--- a/fe/src/main/java/org/apache/impala/service/JniCatalog.java
+++ b/fe/src/main/java/org/apache/impala/service/JniCatalog.java
@@ -27,7 +27,8 @@ import java.util.UUID;
import org.apache.impala.authorization.AuthorizationConfig;
import org.apache.impala.authorization.AuthorizationFactory;
-import org.apache.impala.authorization.NoneAuthorizationFactory;
+import org.apache.impala.authorization.AuthorizationManager;
+import org.apache.impala.authorization.NoopAuthorizationFactory;
import org.apache.impala.authorization.User;
import org.apache.impala.authorization.sentry.SentryCatalogdAuthorizationManager;
import org.apache.impala.catalog.CatalogException;
@@ -84,6 +85,7 @@ public class JniCatalog {
new TBinaryProtocol.Factory();
private final CatalogServiceCatalog catalog_;
private final CatalogOpExecutor catalogOpExecutor_;
+ private final AuthorizationManager authzManager_;
// A unique identifier for this instance of the Catalog Service.
private static final TUniqueId catalogServiceId_ = generateId();
@@ -124,7 +126,7 @@ public class JniCatalog {
if (!authzConfig.isEnabled()) {
// For backward compatibility to keep the existing behavior, when authorization
// is not enabled, we need to use a dummy authorization config.
- authzFactory = new NoneAuthorizationFactory(BackendConfig.INSTANCE);
+ authzFactory = new NoopAuthorizationFactory(BackendConfig.INSTANCE);
authzConfig = authzFactory.getAuthorizationConfig();
LOG.info("Authorization is 'DISABLED'.");
} else {
@@ -134,14 +136,16 @@ public class JniCatalog {
LOG.info(JniUtil.getJavaVersion());
catalog_ = new CatalogServiceCatalog(cfg.load_catalog_in_background,
- cfg.num_metadata_loading_threads, cfg.initial_hms_cnxn_timeout_s,
- authzConfig, getServiceId(), cfg.principal, cfg.local_library_path);
+ cfg.num_metadata_loading_threads, cfg.initial_hms_cnxn_timeout_s, getServiceId(),
+ cfg.local_library_path);
+ authzManager_ = authzFactory.newAuthorizationManager(catalog_);
+ catalog_.setAuthzManager(authzManager_);
try {
catalog_.reset();
} catch (CatalogException e) {
LOG.error("Error initializing Catalog. Please run 'invalidate metadata'", e);
}
- catalogOpExecutor_ = new CatalogOpExecutor(catalog_, authzFactory);
+ catalogOpExecutor_ = new CatalogOpExecutor(catalog_, authzConfig, authzManager_);
}
public static TUniqueId getServiceId() { return catalogServiceId_; }
diff --git a/fe/src/main/java/org/apache/impala/service/JniFrontend.java b/fe/src/main/java/org/apache/impala/service/JniFrontend.java
index 3d0297d..ad12103 100644
--- a/fe/src/main/java/org/apache/impala/service/JniFrontend.java
+++ b/fe/src/main/java/org/apache/impala/service/JniFrontend.java
@@ -46,7 +46,7 @@ import org.apache.impala.analysis.ToSqlUtils;
import org.apache.impala.authorization.AuthorizationConfig;
import org.apache.impala.authorization.AuthorizationFactory;
import org.apache.impala.authorization.ImpalaInternalAdminUser;
-import org.apache.impala.authorization.NoneAuthorizationFactory;
+import org.apache.impala.authorization.NoopAuthorizationFactory;
import org.apache.impala.authorization.User;
import org.apache.impala.catalog.FeDataSource;
import org.apache.impala.catalog.FeDb;
@@ -148,7 +148,7 @@ public class JniFrontend {
if (!authzConfig.isEnabled()) {
// For backward compatibility to keep the existing behavior, when authorization
// is not enabled, we need to use a dummy authorization config.
- authzFactory = new NoneAuthorizationFactory(BackendConfig.INSTANCE);
+ authzFactory = new NoopAuthorizationFactory(BackendConfig.INSTANCE);
LOG.info("Authorization is 'DISABLED'.");
} else {
LOG.info(String.format("Authorization is 'ENABLED' using %s.",
diff --git a/fe/src/test/java/org/apache/impala/analysis/AnalyzeAuthStmtsTest.java b/fe/src/test/java/org/apache/impala/analysis/AnalyzeAuthStmtsTest.java
index 8c44de7..e4ec95b 100644
--- a/fe/src/test/java/org/apache/impala/analysis/AnalyzeAuthStmtsTest.java
+++ b/fe/src/test/java/org/apache/impala/analysis/AnalyzeAuthStmtsTest.java
@@ -19,7 +19,7 @@ package org.apache.impala.analysis;
import java.util.HashSet;
-import org.apache.impala.authorization.NoneAuthorizationFactory;
+import org.apache.impala.authorization.NoopAuthorizationFactory;
import org.apache.impala.authorization.sentry.SentryAuthorizationConfig;
import org.apache.impala.authorization.sentry.SentryAuthorizationFactory;
import org.apache.impala.catalog.Catalog;
@@ -80,7 +80,7 @@ public class AnalyzeAuthStmtsTest extends FrontendTestBase {
Catalog.DEFAULT_DB, System.getProperty("user.name"));
EventSequence timeline = new EventSequence("Authorization Test");
AnalysisContext analysisCtx = new AnalysisContext(queryCtx,
- new NoneAuthorizationFactory(), timeline);
+ new NoopAuthorizationFactory(), timeline);
return analysisCtx;
}
diff --git a/fe/src/test/java/org/apache/impala/analysis/AuditingTest.java b/fe/src/test/java/org/apache/impala/analysis/AuditingTest.java
index 14d87d7..1107389 100644
--- a/fe/src/test/java/org/apache/impala/analysis/AuditingTest.java
+++ b/fe/src/test/java/org/apache/impala/analysis/AuditingTest.java
@@ -19,9 +19,8 @@ package org.apache.impala.analysis;
import java.util.Set;
-import org.apache.impala.authorization.sentry.SentryAuthorizationConfig;
+import org.apache.impala.authorization.AuthorizationFactory;
import org.apache.impala.authorization.AuthorizationException;
-import org.apache.impala.authorization.sentry.SentryAuthorizationFactory;
import org.apache.impala.catalog.Catalog;
import org.apache.impala.catalog.ImpaladCatalog;
import org.apache.impala.common.AnalysisException;
@@ -368,11 +367,8 @@ public class AuditingTest extends FrontendTestBase {
public void TestAccessEventsOnAuthFailure() throws ImpalaException {
// The policy file doesn't exist so all operations will result in
// an AuthorizationError
- SentryAuthorizationConfig config =
- SentryAuthorizationConfig.createHadoopGroupAuthConfig("server1",
- System.getenv("IMPALA_HOME") + "/fe/src/test/resources/sentry-site.xml");
- try (ImpaladCatalog catalog = new ImpaladTestCatalog(config)) {
- SentryAuthorizationFactory authzFactory = new SentryAuthorizationFactory(config);
+ AuthorizationFactory authzFactory = createAuthorizationFactory(false);
+ try (ImpaladCatalog catalog = new ImpaladTestCatalog(authzFactory)) {
Frontend fe = new Frontend(authzFactory, catalog);
AnalysisContext analysisCtx = createAnalysisCtx(authzFactory);
// We should get an audit event even when an authorization failure occurs.
diff --git a/fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java b/fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java
index 809368f..0a9e64a 100644
--- a/fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java
+++ b/fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java
@@ -111,7 +111,8 @@ public class AuthorizationStmtTest extends FrontendTestBase {
private final RangerImpalaPlugin rangerImpalaPlugin_;
private final RangerRESTClient rangerRestClient_;
- public AuthorizationStmtTest(AuthorizationProvider authzProvider) {
+ public AuthorizationStmtTest(AuthorizationProvider authzProvider)
+ throws ImpalaException {
authzProvider_ = authzProvider;
switch (authzProvider) {
case SENTRY:
@@ -120,7 +121,7 @@ public class AuthorizationStmtTest extends FrontendTestBase {
System.getenv("IMPALA_HOME") + "/fe/src/test/resources/sentry-site.xml");
authzFactory_ = new SentryAuthorizationFactory(authzConfig_);
authzCtx_ = createAnalysisCtx(authzFactory_, USER.getName());
- authzCatalog_ = new ImpaladTestCatalog(authzConfig_);
+ authzCatalog_ = new ImpaladTestCatalog(authzFactory_);
authzFrontend_ = new Frontend(authzFactory_, authzCatalog_);
sentryService_ = new SentryPolicyService(
((SentryAuthorizationConfig) authzConfig_).getSentryConfig());
@@ -132,7 +133,7 @@ public class AuthorizationStmtTest extends FrontendTestBase {
SERVER_NAME);
authzFactory_ = new RangerAuthorizationFactory(authzConfig_);
authzCtx_ = createAnalysisCtx(authzFactory_, USER.getName());
- authzCatalog_ = new ImpaladTestCatalog(authzConfig_);
+ authzCatalog_ = new ImpaladTestCatalog(authzFactory_);
authzFrontend_ = new Frontend(authzFactory_, authzCatalog_);
rangerImpalaPlugin_ =
((RangerAuthorizationChecker) authzFrontend_.getAuthzChecker())
diff --git a/fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java b/fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java
index 23d99e8..e319bf3 100644
--- a/fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java
+++ b/fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java
@@ -83,11 +83,19 @@ public class AuthorizationTest extends FrontendTestBase {
SentryAuthorizationConfig.createHadoopGroupAuthConfig("server1",
System.getenv("IMPALA_HOME") + "/fe/src/test/resources/sentry-site.xml");
private static final ImpaladTestCatalog AUTHZ_CATALOG =
- new ImpaladTestCatalog(AUTHZ_CONFIG);
+ new ImpaladTestCatalog(new SentryAuthorizationFactory(AUTHZ_CONFIG));
private static final AuthorizationFactory AUTHZ_FACTORY =
new SentryAuthorizationFactory(AUTHZ_CONFIG);
- private static final Frontend AUTHZ_FE = new Frontend(AUTHZ_FACTORY, AUTHZ_CATALOG);
+ private static final Frontend AUTHZ_FE;
+
+ static {
+ try {
+ AUTHZ_FE = new Frontend(AUTHZ_FACTORY, AUTHZ_CATALOG);
+ } catch (ImpalaException e) {
+ throw new RuntimeException(e);
+ }
+ }
private final AnalysisContext authzCtx;
@@ -463,7 +471,9 @@ public class AuthorizationTest extends FrontendTestBase {
SentryAuthorizationConfig authzConfig = new SentryAuthorizationConfig("server1",
AuthorizationTest.AUTHZ_CONFIG.getSentryConfig().getConfigFile(),
CustomClusterResourceAuthorizationProvider.class.getName());
- try (ImpaladTestCatalog catalog = new ImpaladTestCatalog(authzConfig)) {
+ SentryAuthorizationFactory authzFactory = new SentryAuthorizationFactory(
+ authzConfig);
+ try (ImpaladTestCatalog catalog = new ImpaladTestCatalog(authzFactory)) {
setupImpalaCatalog(catalog);
// This test relies on the auth_to_local rule -
// "RULE:[2:$1@$0](authtest@REALM.COM)s/(.*)@REALM.COM/auth_to_local_user/"
@@ -475,8 +485,6 @@ public class AuthorizationTest extends FrontendTestBase {
User.setRulesForTesting(
new Configuration().get(HADOOP_SECURITY_AUTH_TO_LOCAL, "DEFAULT"));
User user = new User("authtest/hostname@REALM.COM");
- SentryAuthorizationFactory authzFactory = new SentryAuthorizationFactory(
- authzConfig);
AnalysisContext ctx = createAnalysisCtx(authzFactory, user.getName());
Frontend fe = new Frontend(authzFactory, catalog);
@@ -619,8 +627,7 @@ public class AuthorizationTest extends FrontendTestBase {
AuthorizationTest.AUTHZ_CONFIG.getSentryConfig().getConfigFile(),
CustomClusterResourceAuthorizationProvider.class.getName());
SentryAuthorizationFactory authzFactory = new SentryAuthorizationFactory(authzConfig);
-
- try (ImpaladTestCatalog catalog = new ImpaladTestCatalog(authzConfig)) {
+ try (ImpaladTestCatalog catalog = new ImpaladTestCatalog(authzFactory)) {
setupImpalaCatalog(catalog);
// Create an analysis context + FE with the test user
// (as defined in the policy file)
diff --git a/fe/src/test/java/org/apache/impala/analysis/StmtMetadataLoaderTest.java b/fe/src/test/java/org/apache/impala/analysis/StmtMetadataLoaderTest.java
index caba736..65dde6b 100644
--- a/fe/src/test/java/org/apache/impala/analysis/StmtMetadataLoaderTest.java
+++ b/fe/src/test/java/org/apache/impala/analysis/StmtMetadataLoaderTest.java
@@ -20,7 +20,7 @@ package org.apache.impala.analysis;
import java.util.Arrays;
import org.apache.impala.analysis.StmtMetadataLoader.StmtTableCache;
-import org.apache.impala.authorization.NoneAuthorizationFactory;
+import org.apache.impala.authorization.NoopAuthorizationFactory;
import org.apache.impala.catalog.Catalog;
import org.apache.impala.catalog.FeTable;
import org.apache.impala.common.ImpalaException;
@@ -38,7 +38,7 @@ public class StmtMetadataLoaderTest {
String[] expectedDbs, String[] expectedTables)
throws ImpalaException {
try (ImpaladTestCatalog catalog = new ImpaladTestCatalog()) {
- Frontend fe = new Frontend(new NoneAuthorizationFactory(), catalog);
+ Frontend fe = new Frontend(new NoopAuthorizationFactory(), catalog);
StatementBase stmt = Parser.parse(stmtStr);
// Catalog is fresh and no tables are cached.
validateUncached(stmt, fe, expectedNumLoadRequests, expectedNumCatalogUpdates,
@@ -50,7 +50,7 @@ public class StmtMetadataLoaderTest {
private void testNoLoad(String stmtStr) throws ImpalaException {
try (ImpaladTestCatalog catalog = new ImpaladTestCatalog()) {
- Frontend fe = new Frontend(new NoneAuthorizationFactory(), catalog);
+ Frontend fe = new Frontend(new NoopAuthorizationFactory(), catalog);
StatementBase stmt = Parser.parse(stmtStr);
validateCached(stmt, fe, new String[]{}, new String[]{});
}
diff --git a/fe/src/test/java/org/apache/impala/authorization/sentry/SentryProxyTest.java b/fe/src/test/java/org/apache/impala/authorization/sentry/SentryProxyTest.java
index eb8e425..e3f2516 100644
--- a/fe/src/test/java/org/apache/impala/authorization/sentry/SentryProxyTest.java
+++ b/fe/src/test/java/org/apache/impala/authorization/sentry/SentryProxyTest.java
@@ -272,7 +272,7 @@ public class SentryProxyTest {
lowerCaseRoleName.substring(1);
try (CatalogServiceCatalog catalog = CatalogServiceTestCatalog.createWithAuth(
- new SentryAuthorizationConfig(authzConfig_.getSentryConfig()))) {
+ new SentryAuthorizationFactory(authzConfig_))) {
addSentryRolePrivileges(sentryService_, lowerCaseRoleName, "functional");
CatalogState noReset = refreshSentryAuthorization(catalog, sentryService_, false);
@@ -317,7 +317,7 @@ public class SentryProxyTest {
lowerCaseUserName.substring(1);
try (CatalogServiceCatalog catalog = CatalogServiceTestCatalog.createWithAuth(
- new SentryAuthorizationConfig(authzConfig_.getSentryConfig()))) {
+ new SentryAuthorizationFactory(authzConfig_))) {
SentryPolicyServiceStub sentryService = createSentryPolicyServiceStub(
authzConfig_.getSentryConfig());
// We grant different privileges to different users to ensure each user is
@@ -599,7 +599,7 @@ public class SentryProxyTest {
private void withAllPrincipalTypes(Consumer<SentryProxyTestContext> consumer) {
for (TPrincipalType type: TPrincipalType.values()) {
try (CatalogServiceCatalog catalog = CatalogServiceTestCatalog.createWithAuth(
- new SentryAuthorizationConfig(authzConfig_.getSentryConfig()))) {
+ new SentryAuthorizationFactory(authzConfig_))) {
SentryPolicyService sentryService = sentryService_;
if (type == TPrincipalType.USER) {
sentryService = createSentryPolicyServiceStub(authzConfig_.getSentryConfig());
diff --git a/fe/src/test/java/org/apache/impala/catalog/AlterDatabaseTest.java b/fe/src/test/java/org/apache/impala/catalog/AlterDatabaseTest.java
index 5313d22..80f1991 100644
--- a/fe/src/test/java/org/apache/impala/catalog/AlterDatabaseTest.java
+++ b/fe/src/test/java/org/apache/impala/catalog/AlterDatabaseTest.java
@@ -33,7 +33,8 @@ import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicBoolean;
import org.apache.hadoop.hive.metastore.api.Database;
import org.apache.hadoop.hive.metastore.api.PrincipalType;
-import org.apache.impala.authorization.NoneAuthorizationFactory;
+import org.apache.impala.authorization.NoopAuthorizationFactory;
+import org.apache.impala.authorization.NoopAuthorizationFactory.NoopAuthorizationManager;
import org.apache.impala.common.ImpalaException;
import org.apache.impala.service.CatalogOpExecutor;
import org.apache.impala.testutil.CatalogServiceTestCatalog;
@@ -84,10 +85,12 @@ public class AlterDatabaseTest {
* @throws ImpalaException
*/
@BeforeClass
- public static void setUpTest() {
+ public static void setUpTest() throws ImpalaException {
catalog_ = new ImpaladTestCatalog(CatalogServiceTestCatalog.create());
catalogOpExecutor_ =
- new CatalogOpExecutor(catalog_.getSrcCatalog(), new NoneAuthorizationFactory());
+ new CatalogOpExecutor(catalog_.getSrcCatalog(),
+ new NoopAuthorizationFactory().getAuthorizationConfig(),
+ new NoopAuthorizationManager());
}
/**
diff --git a/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java b/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
index 00d8b78..20cb8dc 100644
--- a/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
+++ b/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
@@ -24,7 +24,6 @@ import static org.apache.impala.catalog.events.MetastoreEvents.MetastoreEventTyp
import static org.apache.impala.catalog.events.MetastoreEvents.MetastoreEventType.DROP_TABLE;
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertFalse;
-import static org.junit.Assert.assertNotEquals;
import static org.junit.Assert.assertNotNull;
import static org.junit.Assert.assertNull;
import static org.junit.Assert.assertTrue;
@@ -52,8 +51,8 @@ import org.apache.hadoop.hive.metastore.api.PrincipalType;
import org.apache.hadoop.hive.metastore.client.builder.DatabaseBuilder;
import org.apache.hadoop.hive.metastore.client.builder.PartitionBuilder;
import org.apache.hadoop.hive.metastore.client.builder.TableBuilder;
-import org.apache.impala.authorization.AuthorizationConfig;
-import org.apache.impala.authorization.NoneAuthorizationFactory;
+import org.apache.impala.authorization.NoopAuthorizationFactory;
+import org.apache.impala.authorization.NoopAuthorizationFactory.NoopAuthorizationManager;
import org.apache.impala.catalog.CatalogException;
import org.apache.impala.catalog.CatalogServiceCatalog;
import org.apache.impala.catalog.DatabaseNotFoundException;
@@ -140,9 +139,11 @@ public class MetastoreEventsProcessorTest {
LoggerFactory.getLogger(MetastoreEventsProcessorTest.class);
@BeforeClass
- public static void setUpTestEnvironment() throws TException {
+ public static void setUpTestEnvironment() throws TException, ImpalaException {
catalog_ = CatalogServiceTestCatalog.create();
- catalogOpExecutor_ = new CatalogOpExecutor(catalog_, new NoneAuthorizationFactory());
+ catalogOpExecutor_ = new CatalogOpExecutor(catalog_,
+ new NoopAuthorizationFactory().getAuthorizationConfig(),
+ new NoopAuthorizationManager());
try (MetaStoreClient metaStoreClient = catalog_.getMetaStoreClient()) {
CurrentNotificationEventId currentNotificationId =
metaStoreClient.getHiveClient().getCurrentNotificationEventId();
@@ -997,19 +998,19 @@ public class MetastoreEventsProcessorTest {
private String tblName_;
private FakeCatalogServiceCatalogForFlagTests(boolean loadInBackground,
- int numLoadingThreads, AuthorizationConfig authConfig, TUniqueId catalogServiceId,
- String kerberosPrincipal, String localLibraryPath,
+ int numLoadingThreads, TUniqueId catalogServiceId, String localLibraryPath,
MetaStoreClientPool metaStoreClientPool) throws ImpalaException {
- super(loadInBackground, numLoadingThreads, authConfig, catalogServiceId,
- kerberosPrincipal, localLibraryPath, metaStoreClientPool);
+ super(loadInBackground, numLoadingThreads, catalogServiceId, localLibraryPath,
+ metaStoreClientPool);
}
public static CatalogServiceCatalog create() {
FeSupport.loadLibrary();
CatalogServiceCatalog cs;
try {
- cs = new FakeCatalogServiceCatalogForFlagTests(false, 16, null, new TUniqueId(),
- null, System.getProperty("java.io.tmpdir"), new MetaStoreClientPool(0, 0));
+ cs = new FakeCatalogServiceCatalogForFlagTests(false, 16, new TUniqueId(),
+ System.getProperty("java.io.tmpdir"), new MetaStoreClientPool(0, 0));
+ cs.setAuthzManager(new NoopAuthorizationManager());
cs.reset();
} catch (ImpalaException e) {
throw new IllegalStateException(e.getMessage(), e);
diff --git a/fe/src/test/java/org/apache/impala/common/FrontendFixture.java b/fe/src/test/java/org/apache/impala/common/FrontendFixture.java
index a37717b..de1e9d8 100644
--- a/fe/src/test/java/org/apache/impala/common/FrontendFixture.java
+++ b/fe/src/test/java/org/apache/impala/common/FrontendFixture.java
@@ -37,7 +37,7 @@ import org.apache.impala.analysis.StatementBase;
import org.apache.impala.analysis.StmtMetadataLoader;
import org.apache.impala.analysis.StmtMetadataLoader.StmtTableCache;
import org.apache.impala.authorization.AuthorizationFactory;
-import org.apache.impala.authorization.NoneAuthorizationFactory;
+import org.apache.impala.authorization.NoopAuthorizationFactory;
import org.apache.impala.catalog.AggregateFunction;
import org.apache.impala.catalog.Catalog;
import org.apache.impala.catalog.CatalogException;
@@ -85,14 +85,21 @@ import com.google.common.collect.Lists;
public class FrontendFixture {
// Single instance used for all tests. Logically equivalent to a
// single Impala cluster used by many clients.
- protected static final FrontendFixture instance_ = new FrontendFixture();
+ protected static final FrontendFixture instance_;
+
+ static {
+ try {
+ instance_ = new FrontendFixture();
+ } catch (ImpalaException e) {
+ throw new RuntimeException(e);
+ }
+ }
// The test catalog that can hold test-only tables.
protected final ImpaladTestCatalog catalog_ = new ImpaladTestCatalog();
// The actual Impala frontend that backs this fixture.
- protected final Frontend frontend_ = new Frontend(new NoneAuthorizationFactory(),
- catalog_);
+ protected final Frontend frontend_;
// Test-local list of test databases and tables.
protected final List<Db> testDbs_ = new ArrayList<>();
@@ -108,8 +115,9 @@ public class FrontendFixture {
* Private constructor. Use {@link #instance()} to get access to
* the front-end fixture.
*/
- private FrontendFixture() {
+ private FrontendFixture() throws ImpalaException {
defaultSession_ = new AnalysisSessionFixture();
+ frontend_ = new Frontend(new NoopAuthorizationFactory(), catalog_);
}
/**
@@ -294,12 +302,12 @@ public class FrontendFixture {
defaultDb, System.getProperty("user.name"));
EventSequence timeline = new EventSequence("Frontend Test Timeline");
AnalysisContext analysisCtx = new AnalysisContext(queryCtx,
- new NoneAuthorizationFactory(), timeline);
+ new NoopAuthorizationFactory(), timeline);
return analysisCtx;
}
public AnalysisContext createAnalysisCtx(TQueryOptions queryOptions) {
- return createAnalysisCtx(queryOptions, new NoneAuthorizationFactory());
+ return createAnalysisCtx(queryOptions, new NoopAuthorizationFactory());
}
public AnalysisContext createAnalysisCtx(TQueryOptions queryOptions,
diff --git a/fe/src/test/java/org/apache/impala/common/FrontendTestBase.java b/fe/src/test/java/org/apache/impala/common/FrontendTestBase.java
index c415514..37efe95 100644
--- a/fe/src/test/java/org/apache/impala/common/FrontendTestBase.java
+++ b/fe/src/test/java/org/apache/impala/common/FrontendTestBase.java
@@ -41,6 +41,7 @@ import org.apache.impala.authorization.AuthorizationFactory;
import org.apache.impala.authorization.AuthorizationManager;
import org.apache.impala.authorization.AuthorizationPolicy;
import org.apache.impala.authorization.AuthorizationProvider;
+import org.apache.impala.authorization.NoopAuthorizationFactory.NoopAuthorizationManager;
import org.apache.impala.authorization.PrivilegeRequest;
import org.apache.impala.authorization.User;
import org.apache.impala.catalog.Catalog;
@@ -306,6 +307,16 @@ public class FrontendTestBase extends AbstractFrontendTest {
* not do the actual authorization.
*/
protected AuthorizationFactory createAuthorizationFactory() {
+ return createAuthorizationFactory(true);
+ }
+
+ /**
+ * Creates a dummy {@link AuthorizationFactory} with authorization enabled, but does
+ * not do the actual authorization.
+ *
+ * @param authorized the result of the authorization.
+ */
+ protected AuthorizationFactory createAuthorizationFactory(boolean authorized) {
return new AuthorizationFactory() {
@Override
public AuthorizationConfig getAuthorizationConfig() {
@@ -314,7 +325,7 @@ public class FrontendTestBase extends AbstractFrontendTest {
public boolean isEnabled() { return true; }
@Override
public AuthorizationProvider getProvider() {
- return AuthorizationProvider.NONE;
+ return AuthorizationProvider.NOOP;
}
@Override
public String getServerName() { return "server1"; }
@@ -329,7 +340,7 @@ public class FrontendTestBase extends AbstractFrontendTest {
@Override
protected boolean authorize(User user, PrivilegeRequest request)
throws InternalException {
- return true;
+ return authorized;
}
@Override
@@ -348,12 +359,12 @@ public class FrontendTestBase extends AbstractFrontendTest {
@Override
public AuthorizationManager newAuthorizationManager(FeCatalogManager catalog,
Supplier<? extends AuthorizationChecker> authzChecker) {
- return null;
+ return new NoopAuthorizationManager();
}
@Override
public AuthorizationManager newAuthorizationManager(CatalogServiceCatalog catalog) {
- return null;
+ return new NoopAuthorizationManager();
}
};
}
diff --git a/fe/src/test/java/org/apache/impala/common/QueryFixture.java b/fe/src/test/java/org/apache/impala/common/QueryFixture.java
index c72d9e7..cc550d3 100644
--- a/fe/src/test/java/org/apache/impala/common/QueryFixture.java
+++ b/fe/src/test/java/org/apache/impala/common/QueryFixture.java
@@ -33,7 +33,7 @@ import org.apache.impala.analysis.SqlScanner;
import org.apache.impala.analysis.StatementBase;
import org.apache.impala.analysis.StmtMetadataLoader;
import org.apache.impala.analysis.StmtMetadataLoader.StmtTableCache;
-import org.apache.impala.authorization.NoneAuthorizationFactory;
+import org.apache.impala.authorization.NoopAuthorizationFactory;
import org.apache.impala.testutil.TestUtils;
import org.apache.impala.thrift.TQueryCtx;
import org.apache.impala.thrift.TQueryOptions;
@@ -94,7 +94,7 @@ public class QueryFixture {
*/
protected AnalysisContext makeAnalysisContext() {
EventSequence timeline = new EventSequence("Frontend Test Timeline");
- return new AnalysisContext(queryCtx_, new NoneAuthorizationFactory(), timeline);
+ return new AnalysisContext(queryCtx_, new NoopAuthorizationFactory(), timeline);
}
/**
diff --git a/fe/src/test/java/org/apache/impala/testutil/CatalogServiceTestCatalog.java b/fe/src/test/java/org/apache/impala/testutil/CatalogServiceTestCatalog.java
index 7fc5ffc..124f6da 100644
--- a/fe/src/test/java/org/apache/impala/testutil/CatalogServiceTestCatalog.java
+++ b/fe/src/test/java/org/apache/impala/testutil/CatalogServiceTestCatalog.java
@@ -17,10 +17,10 @@
package org.apache.impala.testutil;
-import org.apache.impala.authorization.AuthorizationConfig;
-import org.apache.impala.authorization.sentry.SentryAuthorizationConfig;
-import org.apache.impala.authorization.sentry.SentryConfig;
+import org.apache.impala.authorization.AuthorizationFactory;
+import org.apache.impala.authorization.NoopAuthorizationFactory;
import org.apache.impala.authorization.AuthorizationPolicy;
+import org.apache.impala.authorization.NoopAuthorizationFactory.NoopAuthorizationManager;
import org.apache.impala.catalog.CatalogServiceCatalog;
import org.apache.impala.catalog.MetaStoreClientPool;
import org.apache.impala.common.ImpalaException;
@@ -37,9 +37,9 @@ import java.util.UUID;
*/
public class CatalogServiceTestCatalog extends CatalogServiceCatalog {
public CatalogServiceTestCatalog(boolean loadInBackground, int numLoadingThreads,
- AuthorizationConfig authConfig, TUniqueId catalogServiceId,
- MetaStoreClientPool metaStoreClientPool) throws ImpalaException {
- super(loadInBackground, numLoadingThreads, authConfig, catalogServiceId, null,
+ TUniqueId catalogServiceId, MetaStoreClientPool metaStoreClientPool)
+ throws ImpalaException {
+ super(loadInBackground, numLoadingThreads, catalogServiceId,
System.getProperty("java.io.tmpdir"), metaStoreClientPool);
// Cache pools are typically loaded asynchronously, but as there is no fixed execution
@@ -50,19 +50,20 @@ public class CatalogServiceTestCatalog extends CatalogServiceCatalog {
}
public static CatalogServiceCatalog create() {
- return createWithAuth(new SentryAuthorizationConfig(new SentryConfig(null)));
+ return createWithAuth(new NoopAuthorizationFactory());
}
/**
* Creates a catalog server that reads authorization policy metadata from the
* authorization config.
*/
- public static CatalogServiceCatalog createWithAuth(AuthorizationConfig config) {
+ public static CatalogServiceCatalog createWithAuth(AuthorizationFactory authzFactory) {
FeSupport.loadLibrary();
CatalogServiceCatalog cs;
try {
- cs = new CatalogServiceTestCatalog(false, 16, config, new TUniqueId(),
+ cs = new CatalogServiceTestCatalog(false, 16, new TUniqueId(),
new MetaStoreClientPool(0, 0));
+ cs.setAuthzManager(authzFactory.newAuthorizationManager(cs));
cs.reset();
} catch (ImpalaException e) {
throw new IllegalStateException(e.getMessage(), e);
@@ -80,8 +81,9 @@ public class CatalogServiceTestCatalog extends CatalogServiceCatalog {
FeSupport.loadLibrary();
Path derbyPath = Paths.get(System.getProperty("java.io.tmpdir"),
UUID.randomUUID().toString());
- CatalogServiceCatalog cs = new CatalogServiceTestCatalog(false, 16, null,
+ CatalogServiceCatalog cs = new CatalogServiceTestCatalog(false, 16,
new TUniqueId(), new EmbeddedMetastoreClientPool(0, derbyPath));
+ cs.setAuthzManager(new NoopAuthorizationManager());
cs.reset();
return cs;
}
diff --git a/fe/src/test/java/org/apache/impala/testutil/ImpaladTestCatalog.java b/fe/src/test/java/org/apache/impala/testutil/ImpaladTestCatalog.java
index 2cbe8c6..45cbed6 100644
--- a/fe/src/test/java/org/apache/impala/testutil/ImpaladTestCatalog.java
+++ b/fe/src/test/java/org/apache/impala/testutil/ImpaladTestCatalog.java
@@ -19,8 +19,8 @@ package org.apache.impala.testutil;
import com.google.common.base.Preconditions;
import org.apache.impala.analysis.TableName;
-import org.apache.impala.authorization.AuthorizationConfig;
-import org.apache.impala.authorization.NoneAuthorizationFactory;
+import org.apache.impala.authorization.AuthorizationFactory;
+import org.apache.impala.authorization.NoopAuthorizationFactory;
import org.apache.impala.catalog.BuiltinsDb;
import org.apache.impala.catalog.CatalogException;
import org.apache.impala.catalog.CatalogServiceCatalog;
@@ -48,17 +48,16 @@ public class ImpaladTestCatalog extends ImpaladCatalog {
private final CatalogServiceCatalog srcCatalog_;
public ImpaladTestCatalog() {
- this(new NoneAuthorizationFactory().getAuthorizationConfig());
+ this(new NoopAuthorizationFactory());
}
/**
- * Takes an AuthorizationConfig to bootstrap the backing CatalogServiceCatalog.
+ * Takes an {@link AuthorizationFactory} to bootstrap the backing CatalogServiceCatalog.
*/
- public ImpaladTestCatalog(AuthorizationConfig authzConfig) {
+ public ImpaladTestCatalog(AuthorizationFactory authzFactory) {
super("127.0.0.1");
- CatalogServiceCatalog catalogServerCatalog = authzConfig.isEnabled() ?
- CatalogServiceTestCatalog.createWithAuth(authzConfig) :
- CatalogServiceTestCatalog.create();
+ CatalogServiceCatalog catalogServerCatalog =
+ CatalogServiceTestCatalog.createWithAuth(authzFactory);
authPolicy_ = catalogServerCatalog.getAuthPolicy();
srcCatalog_ = catalogServerCatalog;
srcCatalog_.addDb(BuiltinsDb.getInstance());
diff --git a/fe/src/test/java/org/apache/impala/testutil/PlannerTestCaseLoader.java b/fe/src/test/java/org/apache/impala/testutil/PlannerTestCaseLoader.java
index ae9b92c..204b963 100644
--- a/fe/src/test/java/org/apache/impala/testutil/PlannerTestCaseLoader.java
+++ b/fe/src/test/java/org/apache/impala/testutil/PlannerTestCaseLoader.java
@@ -17,7 +17,8 @@
package org.apache.impala.testutil;
-import org.apache.impala.authorization.NoneAuthorizationFactory;
+import org.apache.impala.authorization.NoopAuthorizationFactory;
+import org.apache.impala.authorization.NoopAuthorizationFactory.NoopAuthorizationManager;
import org.apache.impala.catalog.Catalog;
import org.apache.impala.common.ImpalaException;
import org.apache.impala.service.CatalogOpExecutor;
@@ -56,9 +57,10 @@ public class PlannerTestCaseLoader implements AutoCloseable {
public PlannerTestCaseLoader() throws ImpalaException {
catalog_ = new ImpaladTestCatalog(
CatalogServiceTestCatalog.createTransientTestCatalog());
- frontend_ = new Frontend(new NoneAuthorizationFactory(), catalog_);
+ frontend_ = new Frontend(new NoopAuthorizationFactory(), catalog_);
catalogOpExecutor_ = new CatalogOpExecutor(catalog_.getSrcCatalog(),
- new NoneAuthorizationFactory());
+ new NoopAuthorizationFactory().getAuthorizationConfig(),
+ new NoopAuthorizationManager());
}
public Catalog getSrcCatalog() { return catalog_.getSrcCatalog(); }
[impala] 02/02: IMPALA-8414: Skip header when parsing /proc/net/dev
Posted by ta...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
tarmstrong pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
commit 67f77d41d40523074385b8dbccfa6ef6ef81dd57
Author: Lars Volker <lv...@cloudera.com>
AuthorDate: Sun Apr 14 18:23:50 2019 -0700
IMPALA-8414: Skip header when parsing /proc/net/dev
The fix for IMPALA-8395 does not skip the first to header lines of
/proc/net/dev. This change skips the header to prevent unnecessary
warnings.
Testing: Manually checked that impalad.WARNING did not contain any
warnings about failure to parse /proc/net/dev.
Change-Id: I7ff931671050d44926d0baa77ec374afed1f8225
Reviewed-on: http://gerrit.cloudera.org:8080/13016
Reviewed-by: Lars Volker <lv...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
be/src/util/system-state-info.cc | 7 +++++--
1 file changed, 5 insertions(+), 2 deletions(-)
diff --git a/be/src/util/system-state-info.cc b/be/src/util/system-state-info.cc
index e808724..b5495c5 100644
--- a/be/src/util/system-state-info.cc
+++ b/be/src/util/system-state-info.cc
@@ -167,8 +167,11 @@ void SystemStateInfo::ReadProcNetDevString(const string& dev_string) {
memset(&sum_values, 0, sizeof(sum_values));
StringPiece sp(dev_string);
- vector<StringPiece> lines = Split(sp, "\n");
- for (const StringPiece& line : lines) {
+ vector<StringPiece> lines = Split(sp, "\n", SkipWhitespace());
+ // The first two lines contain the header, skip them.
+ DCHECK_GT(lines.size(), 2);
+ for (int i = 2; i < lines.size(); ++i) {
+ const StringPiece& line = lines[i];
NetworkValues line_values;
ReadProcNetDevLine(line, &line_values);
AddNetworkValues(line_values, &sum_values);