You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hawq.apache.org by denalex <gi...@git.apache.org> on 2017/01/17 05:03:41 UTC
[GitHub] incubator-hawq pull request #1092: HAWQ-1203. Ranger Plugin Service Implemen...
GitHub user denalex opened a pull request:
https://github.com/apache/incubator-hawq/pull/1092
HAWQ-1203. Ranger Plugin Service Implementation. (with contributions \u2026
\u2026by Lav Jain and Leslie Chang)
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/denalex/incubator-hawq ranger-plugin-3
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-hawq/pull/1092.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1092
----
commit 1f829f7d1c77d140425eca7b02fcb22271ee3430
Author: Alexander Denissov <ad...@pivotal.io>
Date: 2017-01-16T19:46:36Z
HAWQ-1203. Ranger Plugin Service Implementation. (with contributions by Lav Jain and Leslie Chang)
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq issue #1092: HAWQ-1203. Ranger Plugin Service Implementation....
Posted by linwen <gi...@git.apache.org>.
Github user linwen commented on the issue:
https://github.com/apache/incubator-hawq/pull/1092
+1
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1092: HAWQ-1203. Ranger Plugin Service Implemen...
Posted by denalex <gi...@git.apache.org>.
Github user denalex commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1092#discussion_r96915483
--- Diff: ranger-plugin/scripts/register_hawq.sh ---
@@ -0,0 +1,217 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+function usage() {
+ echo "USAGE: register_hawq.sh -r ranger_host:ranger_port -u ranger_user -p ranger_password -h hawq_host:hawq_port -w hawq_user -q hawq_password"
--- End diff --
@ictmalili -- ok, makes sense, will rename the script as part of management scripts JIRA.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1092: HAWQ-1203. Ranger Plugin Service Implemen...
Posted by ictmalili <gi...@git.apache.org>.
Github user ictmalili commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1092#discussion_r96560193
--- Diff: ranger-plugin/admin-plugin/src/main/java/org/apache/hawq/ranger/service/HawqClient.java ---
@@ -0,0 +1,314 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.hawq.ranger.service;
+
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.hawq.ranger.model.HawqProtocols;
+
+import java.sql.Connection;
+import java.sql.DriverManager;
+import java.sql.PreparedStatement;
+import java.sql.SQLException;
+import java.sql.ResultSet;
+import java.util.*;
+
+import static org.apache.ranger.plugin.client.BaseClient.generateResponseDataMap;
+
+public class HawqClient {
+
+ private static final Log LOG = LogFactory.getLog(HawqClient.class);
+
+ public static final String CONNECTION_SUCCESSFUL_MESSAGE = "ConnectionTest Successful";
+ public static final String CONNECTION_FAILURE_MESSAGE = "Unable to retrieve any databases using given parameters.";
+
+ public static final String DATABASE_LIST_QUERY = "SELECT datname from pg_database WHERE " +
+ "datname NOT IN ('template0', 'template1', 'hcatalog') and datname LIKE ?";
+ public static final String TABLESPACE_LIST_QUERY = "SELECT spcname from pg_tablespace WHERE spcname LIKE ?";
+ public static final String PROTOCOL_LIST_QUERY = "SELECT ptcname from pg_catalog.pg_extprotocol WHERE ptcname LIKE ?";
+ public static final String SCHEMA_LIST_QUERY = "SELECT schema_name from information_schema.schemata WHERE " +
+ "schema_name NOT IN ('pg_catalog', 'information_schema', 'hawq_toolkit', 'pg_bitmapindex', 'pg_aoseg') AND schema_name NOT LIKE 'pg_toast%' AND schema_name LIKE ?";
+ public static final String LANGUAGE_LIST_QUERY = "SELECT lanname from pg_language WHERE lanname LIKE ?";
+ public static final String SEQUENCE_LIST_QUERY = "SELECT schemaname, relname from pg_statio_all_sequences WHERE relname LIKE ?";
+ public static final String FUNCTION_LIST_QUERY = "SELECT n.nspname, p.proname FROM pg_catalog.pg_proc p " +
+ "LEFT JOIN pg_catalog.pg_namespace n ON n.oid = p.pronamespace WHERE n.nspname NOT IN " +
+ "('pg_catalog', 'information_schema', 'hawq_toolkit', 'pg_bitmapindex', 'pg_aoseg') AND n.nspname NOT LIKE 'pg_toast%' AND p.proname LIKE ?";
+ public static final String TABLE_LIST_QUERY = "SELECT c.relname, n.nspname FROM pg_catalog.pg_class c " +
+ "LEFT JOIN pg_catalog.pg_namespace n ON n.oid = c.relnamespace WHERE c.relkind IN ('r','','v') AND n.nspname NOT IN " +
+ "('pg_catalog', 'information_schema', 'hawq_toolkit', 'pg_bitmapindex', 'pg_aoseg') AND n.nspname NOT LIKE 'pg_toast%' AND c.relname LIKE ?";
+
+ public static final String DATNAME = "datname";
+ public static final String SPCNAME = "spcname";
+ public static final String PTCNAME = "ptcname";
+ public static final String SCHEMA_NAME = "schema_name";
+ public static final String LANGUAGE_NAME = "lanname";
+ public static final String RELNAME = "relname";
+ public static final String SCHEMANAME = "schemaname";
+ public static final String PRONAME = "proname";
+ public static final String NSPNAME = "nspname";
+ public static final String WILDCARD = "*";
+
+ public static final List<String> INTERNAL_PROTOCOLS = HawqProtocols.getAllProtocols();
+ private static final String DEFAULT_DATABASE = "postgres";
+ private static final String JDBC_DRIVER_CLASS = "org.postgresql.Driver";
+
+ private Map<String, String> connectionProperties;
+
+ // we need to load class for the Postgres Driver directly to allow it to register with DriverManager
+ // since DriverManager's classloader will not be able to find it by itself due to plugin's special classloaders
+ static {
+ try {
+ Class.forName(JDBC_DRIVER_CLASS);
+ } catch (Throwable e) {
+ LOG.error("<== HawqClient.initializer : Unable to load JDBC driver " + JDBC_DRIVER_CLASS + " : " + e.getMessage());
+ throw new ExceptionInInitializerError(e);
+ }
+ }
+
+ public HawqClient(Map<String, String> connectionProperties) throws SQLException {
+ this.connectionProperties = connectionProperties;
+ }
+
+ /**
+ * Uses the connectionProperties and attempts to connect to Hawq.
+ * Returns a message depending on success or failure.
+ *
+ * @param connectionProperties Map which contains hostname, port, username, and password
+ * @return Map which contains connectivityStatus, message, description, objectId, and fieldName
+ */
+
+ public HashMap<String, Object> checkConnection(Map<String, String> connectionProperties) throws Exception {
+
+ boolean isConnected = false;
+ HashMap<String, Object> result = new HashMap<>();
+ Connection conn = null;
+
+ String description = CONNECTION_FAILURE_MESSAGE;
+
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("<== HawqClient.checkConnection Starting connection to hawq");
+ }
+
+ try {
+ conn = getConnection(connectionProperties);
+ if(conn.getCatalog() != null) {
+ isConnected = true;
+ description = CONNECTION_SUCCESSFUL_MESSAGE;
+ }
+ } catch (SQLException e) {
+ LOG.error("<== HawqClient.checkConnection Error: Failed to connect" + e);
+ description = e.getMessage();
+ } finally {
+ closeConnection(conn);
+ }
+
+ String message = isConnected ? CONNECTION_SUCCESSFUL_MESSAGE : CONNECTION_FAILURE_MESSAGE;
+
+ generateResponseDataMap(isConnected, message, description, null, null, result);
+
+ return result;
+ }
+
+ public List<String> getDatabaseList(String userInput) throws SQLException {
+ return queryHawq(userInput, DATNAME, DATABASE_LIST_QUERY, null);
+ }
+
+ public List<String> getTablespaceList(String userInput) throws SQLException {
+ return queryHawq(userInput, SPCNAME, TABLESPACE_LIST_QUERY, null);
+ }
+
+ public List<String> getProtocolList(String userInput) throws SQLException {
+ Set<String> allProtocols = new HashSet<>();
+ for (String protocol : INTERNAL_PROTOCOLS) {
+ if(protocol.startsWith(userInput) || userInput.equals(WILDCARD)) {
+ allProtocols.add(protocol);
+ }
+ }
+ allProtocols.addAll(queryHawq(userInput, PTCNAME, PROTOCOL_LIST_QUERY, null));
+ return new ArrayList<>(allProtocols);
+ }
+
+ public List<String> getSchemaList(String userInput, Map<String, List<String>> resources) throws SQLException {
+ return queryHawqPerDb(userInput, resources.get("database"), SCHEMA_NAME, SCHEMA_LIST_QUERY);
+ }
+
+ public List<String> getLanguageList(String userInput, Map<String, List<String>> resources) throws SQLException {
+ return queryHawqPerDb(userInput, resources.get("database"), LANGUAGE_NAME, LANGUAGE_LIST_QUERY);
+ }
+
+ public List<String> getTableList(String userInput, Map<String, List<String>> resources) throws SQLException {
+ return queryHawqPerDbAndSchema(userInput, resources, NSPNAME, RELNAME, TABLE_LIST_QUERY);
+ }
+
+ public List<String> getSequenceList(String userInput, Map<String, List<String>> resources) throws SQLException {
+ return queryHawqPerDbAndSchema(userInput, resources, SCHEMANAME, RELNAME, SEQUENCE_LIST_QUERY);
+ }
+
+ public List<String> getFunctionList(String userInput, Map<String, List<String>> resources) throws SQLException {
+ return queryHawqPerDbAndSchema(userInput, resources, NSPNAME, PRONAME, FUNCTION_LIST_QUERY);
+ }
+
+ private List<String> queryHawqPerDb(String userInput, List<String> databases, String columnName, String query) throws SQLException {
+ Set<String> uniqueResults = new HashSet<>();
+
+ //do for all databases
+ if (databases.contains(WILDCARD)) {
+ databases = getDatabaseList(WILDCARD);
+ }
+
+ for (String db : databases) {
+ uniqueResults.addAll(queryHawq(userInput, columnName, query, db));
+ }
+ return new ArrayList<>(uniqueResults);
+ }
+
+ private List<String> queryHawqPerDbAndSchema(String userInput, Map<String, List<String>> resources, String schemaColumnName, String resultColumnName, String query) throws SQLException {
+ Set<String> uniqueResults = new HashSet<>();
+ List<String> databases = resources.get("database");
+ List<String> schemas = resources.get("schema");
+
+ Connection conn = null;
+ PreparedStatement preparedStatement = null;
+ ResultSet resultSet = null;
+
+ if (databases.contains(WILDCARD)) {
+ databases = getDatabaseList(WILDCARD);
+ }
+
+ for (String db: databases) {
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("<== HawqClient.queryHawqPerDbAndSchema: Connecting to db: " + db);
+ }
+
+ try {
+ conn = getConnection(connectionProperties, db);
+ preparedStatement = handleWildcardPreparedStatement(userInput, query, conn);
+
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("<== HawqClient.queryHawqPerDbAndSchema Starting query: " + query);
+ }
+
+ resultSet = preparedStatement.executeQuery();
+
+ while(resultSet.next()) {
+ if(schemas.contains(resultSet.getString(schemaColumnName)) || schemas.contains(WILDCARD)) {
+ uniqueResults.add(resultSet.getString(resultColumnName));
+ }
+ }
+
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("<== HawqClient.queryHawqPerDbAndSchema Query result: " + uniqueResults.toString());
+ }
+
+ } catch (SQLException e) {
+ LOG.error("<== HawqClient.queryHawqPerDbAndSchema Error: Failed to get results for query: " + query + ", Error: " + e);
+ } finally {
+ closeResultSet(resultSet);
+ closeStatement(preparedStatement);
+ closeConnection(conn);
+ }
+ }
+ return new ArrayList<>(uniqueResults);
+ }
+
+ private List<String> queryHawq(String userInput, String columnName, String query, String database) {
+ List<String> result = new ArrayList<>();
+ Connection conn = null;
+ PreparedStatement preparedStatement = null;
+ ResultSet resultSet = null;
+
+ try {
+ conn = getConnection(connectionProperties, database);
+ preparedStatement = handleWildcardPreparedStatement(userInput, query, conn);
+
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("<== HawqClient.queryHawq Starting query: " + query);
+ }
+
+ resultSet = preparedStatement.executeQuery();
+
+ while(resultSet.next()) {
+ result.add(resultSet.getString(columnName));
+ }
+
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("<== HawqClient.queryHawq Query result: " + result.toString());
+ }
+
+ } catch (SQLException e) {
+ LOG.error("<== HawqClient.queryHawq Error: Failed to get result from query: " + query + ", Error: " + e);
+ } finally {
+ closeResultSet(resultSet);
+ closeStatement(preparedStatement);
+ closeConnection(conn);
+ }
+
+ return result;
+ }
+
+ private Connection getConnection(Map<String, String> connectionProperties) throws SQLException {
+ return getConnection(connectionProperties, null);
+ }
+
+ private Connection getConnection(Map<String, String> connectionProperties, String database) throws SQLException {
+
+ String db = database != null ? database : DEFAULT_DATABASE;
+ String url = String.format("jdbc:postgresql://%s:%s/%s", connectionProperties.get("hostname"), connectionProperties.get("port"), db);
+
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("<== HawqClient.checkConnection Connecting to: (" + url + ") with user: " + connectionProperties.get("username") );
+ }
+
+ Properties props = new Properties();
+ props.setProperty("user", connectionProperties.get("username"));
+ props.setProperty("password", connectionProperties.get("password"));
+ return DriverManager.getConnection(url, props);
+ }
+
+ private PreparedStatement handleWildcardPreparedStatement(String userInput, String query, Connection conn) throws SQLException {
+ PreparedStatement preparedStatement = conn.prepareStatement(query);
+ preparedStatement.setString(1, userInput.equals(WILDCARD) ? "%" : (userInput + "%"));
+ return preparedStatement;
+ }
+
+ private void closeResultSet(ResultSet rs) {
+ try {
+ if (rs != null) rs.close();
+ } catch (Exception e) {
+ // ignore
--- End diff --
OK. Make sense
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1092: HAWQ-1203. Ranger Plugin Service Implemen...
Posted by denalex <gi...@git.apache.org>.
Github user denalex commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1092#discussion_r96556804
--- Diff: ranger-plugin/admin-plugin/pom.xml ---
@@ -0,0 +1,90 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+ <groupId>org.apache.hawq</groupId>
+ <artifactId>ranger-plugin-admin</artifactId>
+ <packaging>jar</packaging>
+ <name>HAWQ Ranger Admin Plugin</name>
+ <description>HAWQ Ranger Admin Plugin</description>
+ <parent>
+ <groupId>org.apache.hawq</groupId>
+ <artifactId>ranger-plugin</artifactId>
+ <version>2.1.0.0</version>
--- End diff --
we want to version RPS (and it RPM) with the same version as HAWQ as RPS is not a standalone product. Maven requires version in pom.xml files, so we choose the one that HAWQ has and will need to update it after Apache releases.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1092: HAWQ-1203. Ranger Plugin Service Implemen...
Posted by denalex <gi...@git.apache.org>.
Github user denalex commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1092#discussion_r96579330
--- Diff: ranger-plugin/conf/ranger-servicedef-hawq.json ---
@@ -0,0 +1,287 @@
+{
+ "name": "hawq",
+ "implClass": "org.apache.hawq.ranger.service.RangerServiceHawq",
+ "label": "HAWQ",
+ "description": "HAWQ",
+ "guid": "1ebb27ee-549a-401d-85ab-818342ca54af",
+ "resources":
+ [
+ {
+ "itemId": 1,
+ "name": "database",
+ "type": "string",
+ "level": 10,
+ "parent": "",
+ "mandatory": true,
+ "lookupSupported": true,
+ "recursiveSupported": false,
--- End diff --
@ictmalili -- these are not well documented, we copied the definition from Hive. My understanding is that:
-- recursiveSupported would allow to define a policies as recursive which is used for HDFS access policies where a policy on a given folder, if declared as recursive, will be applied to all the files and subfolders inside a given folder. This is not applicable to us, so we do not allow creation of "recursive" policies for HAWQ.
-- excludesSupported will allow a resource specified in the policy to be "excluded". Ranger Admin UI will render the box to the right of the resource to be toggled between "included" and "excluded". I am not exactly sure what effect will that have on the policy.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1092: HAWQ-1203. Ranger Plugin Service Implemen...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-hawq/pull/1092
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1092: HAWQ-1203. Ranger Plugin Service Implemen...
Posted by ictmalili <gi...@git.apache.org>.
Github user ictmalili commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1092#discussion_r96557248
--- Diff: ranger-plugin/conf/ranger-servicedef-hawq.json ---
@@ -0,0 +1,287 @@
+{
+ "name": "hawq",
+ "implClass": "org.apache.hawq.ranger.service.RangerServiceHawq",
+ "label": "HAWQ",
+ "description": "HAWQ",
+ "guid": "1ebb27ee-549a-401d-85ab-818342ca54af",
+ "resources":
+ [
+ {
+ "itemId": 1,
+ "name": "database",
+ "type": "string",
+ "level": 10,
+ "parent": "",
+ "mandatory": true,
+ "lookupSupported": true,
+ "recursiveSupported": false,
--- End diff --
What are the meanings of recursiveSupported and excludesSupported? Why do we set the values to false and true respectively?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1092: HAWQ-1203. Ranger Plugin Service Implemen...
Posted by denalex <gi...@git.apache.org>.
Github user denalex commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1092#discussion_r96580025
--- Diff: ranger-plugin/service/src/main/java/org/apache/hawq/ranger/authorization/RangerHawqAuthorizer.java ---
@@ -0,0 +1,263 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hawq.ranger.authorization;
+
+import org.apache.commons.collections.CollectionUtils;
+import org.apache.commons.collections.MapUtils;
+import org.apache.commons.lang.StringUtils;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.hawq.ranger.authorization.model.AuthorizationRequest;
+import org.apache.hawq.ranger.authorization.model.AuthorizationResponse;
+import org.apache.hawq.ranger.authorization.model.HawqPrivilege;
+import org.apache.hawq.ranger.authorization.model.HawqResource;
+import org.apache.hawq.ranger.authorization.model.ResourceAccess;
+import org.apache.ranger.plugin.policyengine.RangerAccessRequest;
+import org.apache.ranger.plugin.policyengine.RangerAccessRequestImpl;
+import org.apache.ranger.plugin.policyengine.RangerAccessResource;
+import org.apache.ranger.plugin.policyengine.RangerAccessResourceImpl;
+import org.apache.ranger.plugin.policyengine.RangerAccessResult;
+import org.apache.ranger.plugin.service.RangerBasePlugin;
+
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+
+import static org.apache.hawq.ranger.authorization.Utils.HAWQ;
+
+/**
+ * Authorizer implementation that uses Ranger to make access decision. Implemented as a singleton.
+ */
+public class RangerHawqAuthorizer implements HawqAuthorizer {
+
+ private static final Log LOG = LogFactory.getLog(RangerHawqAuthorizer.class);
+
+ private static final RangerHawqAuthorizer INSTANCE = new RangerHawqAuthorizer();
+
+ private RangerBasePlugin rangerPlugin;
+
+ /**
+ * Returns the instance of the RangerHawqAuthorizer.
+ * @return the singleton instance
+ */
+ public static RangerHawqAuthorizer getInstance() {
+ return INSTANCE;
+ }
+
+ /**
+ * Constructor. Initializes Ranger Base Plugin to fetch policies from Ranger.
+ */
+ private RangerHawqAuthorizer() {
+
+ LOG.info("Initializing RangerHawqAuthorizer");
+
+ String appId = Utils.getAppId();
+
+ LOG.info(String.format("Initializing RangerBasePlugin for service %s:%s", HAWQ, appId));
+ rangerPlugin = new RangerBasePlugin(HAWQ, appId);
+ rangerPlugin.init();
+ LOG.info(String.format("Initialized RangerBasePlugin for service %s:%s", HAWQ, appId));
+ }
+
+ @Override
+ public AuthorizationResponse isAccessAllowed(AuthorizationRequest request) {
+
+ // validate request to make sure no data is missing
+ validateRequest(request);
+
+ // prepare response object
+ AuthorizationResponse response = new AuthorizationResponse();
+ response.setRequestId(request.getRequestId());
+ Set<ResourceAccess> access = new HashSet<>();
+ response.setAccess(access);
+
+ // iterate over resource requests, augment processed ones with the decision and add to the response
+ for (ResourceAccess resourceAccess : request.getAccess()) {
+ boolean accessAllowed = authorizeResource(resourceAccess, request.getUser());
+ resourceAccess.setAllowed(accessAllowed);
+ access.add(resourceAccess);
+ }
+
+ return response;
+ }
+
+ /**
+ * Authorizes access to a single resource for a given user.
+ *
+ * @param resourceAccess resource to authorize access to
+ * @param user user requesting authorization
+ * @return true if access is authorized, false otherwise
+ */
+ private boolean authorizeResource(ResourceAccess resourceAccess, String user) {
+
+ if (LOG.isDebugEnabled()) {
+ LOG.debug(String.format("Request: access for user=%s to resource=%s with privileges=%s",
+ user, resourceAccess.getResource(), resourceAccess.getPrivileges()));
+ }
+
+ RangerAccessResourceImpl rangerResource = new RangerAccessResourceImpl();
+ //resource.setOwnerUser();
+ for (Map.Entry<HawqResource, String> resourceEntry : resourceAccess.getResource().entrySet()) {
+ rangerResource.setValue(resourceEntry.getKey().name(), resourceEntry.getValue());
+ }
+
+ boolean accessAllowed = true;
+ // iterate over all privileges requested
+ for (HawqPrivilege privilege : resourceAccess.getPrivileges()) {
+ // TODO not clear how we will get user groups -- Kerberos case ?
+ Set<String> userGroups = Collections.emptySet();
+ boolean privilegeAuthorized = authorizeResourcePrivilege(rangerResource, privilege.name(), user, userGroups);
+ // ALL model of evaluation -- all privileges must be authorized for access to be allowed
+ if (!privilegeAuthorized) {
+ accessAllowed = false;
+ break; // terminate early if even a single privilege is not authorized
+ }
+ }
+
+ if (LOG.isDebugEnabled()) {
+ LOG.debug(String.format("Decision: accessAllowed=%s for user=%s to resource=%s with privileges=%s",
+ accessAllowed, user, resourceAccess.getResource(), resourceAccess.getPrivileges()));
+ }
+
+ return accessAllowed;
+ }
+
+ /**
+ * Authorizes access of a given type (privilege) to a single resource for a given user.
+ *
+ * @param rangerResource resource to authorize access to
+ * @param accessType privilege requested for a given resource
+ * @param user user requesting authorization
+ * @param userGroups groups a user belongs to
+ * @return true if access is authorized, false otherwise
+ */
+ private boolean authorizeResourcePrivilege(RangerAccessResource rangerResource, String accessType, String user, Set<String> userGroups) {
+
+ Map<String, String> resourceMap = rangerResource.getAsMap();
+ String database = resourceMap.get(HawqResource.database.name());
+ String schema = resourceMap.get(HawqResource.schema.name());
+ int resourceSize = resourceMap.size();
+
+ // special handling for non-leaf policies
+ if (accessType.equals(HawqPrivilege.create.name()) && database != null && schema == null && resourceSize == 1) {
--- End diff --
resourceSize reflects how many tuples of key:value were received from HAWQ in resource element of JSON request. For example: "resource" : {"database":"foo"} is only 1 entry while "resource" : {"database":"foo", "schema":"bar"} are 2 entries. We have to analyze this for our special handling of create-schema and usage-schema permissions.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1092: HAWQ-1203. Ranger Plugin Service Implemen...
Posted by denalex <gi...@git.apache.org>.
Github user denalex commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1092#discussion_r96557030
--- Diff: ranger-plugin/admin-plugin/src/main/java/org/apache/hawq/ranger/service/HawqClient.java ---
@@ -0,0 +1,314 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.hawq.ranger.service;
+
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.hawq.ranger.model.HawqProtocols;
+
+import java.sql.Connection;
+import java.sql.DriverManager;
+import java.sql.PreparedStatement;
+import java.sql.SQLException;
+import java.sql.ResultSet;
+import java.util.*;
+
+import static org.apache.ranger.plugin.client.BaseClient.generateResponseDataMap;
+
+public class HawqClient {
+
+ private static final Log LOG = LogFactory.getLog(HawqClient.class);
+
+ public static final String CONNECTION_SUCCESSFUL_MESSAGE = "ConnectionTest Successful";
+ public static final String CONNECTION_FAILURE_MESSAGE = "Unable to retrieve any databases using given parameters.";
+
+ public static final String DATABASE_LIST_QUERY = "SELECT datname from pg_database WHERE " +
+ "datname NOT IN ('template0', 'template1', 'hcatalog') and datname LIKE ?";
+ public static final String TABLESPACE_LIST_QUERY = "SELECT spcname from pg_tablespace WHERE spcname LIKE ?";
+ public static final String PROTOCOL_LIST_QUERY = "SELECT ptcname from pg_catalog.pg_extprotocol WHERE ptcname LIKE ?";
+ public static final String SCHEMA_LIST_QUERY = "SELECT schema_name from information_schema.schemata WHERE " +
+ "schema_name NOT IN ('pg_catalog', 'information_schema', 'hawq_toolkit', 'pg_bitmapindex', 'pg_aoseg') AND schema_name NOT LIKE 'pg_toast%' AND schema_name LIKE ?";
+ public static final String LANGUAGE_LIST_QUERY = "SELECT lanname from pg_language WHERE lanname LIKE ?";
+ public static final String SEQUENCE_LIST_QUERY = "SELECT schemaname, relname from pg_statio_all_sequences WHERE relname LIKE ?";
+ public static final String FUNCTION_LIST_QUERY = "SELECT n.nspname, p.proname FROM pg_catalog.pg_proc p " +
+ "LEFT JOIN pg_catalog.pg_namespace n ON n.oid = p.pronamespace WHERE n.nspname NOT IN " +
+ "('pg_catalog', 'information_schema', 'hawq_toolkit', 'pg_bitmapindex', 'pg_aoseg') AND n.nspname NOT LIKE 'pg_toast%' AND p.proname LIKE ?";
+ public static final String TABLE_LIST_QUERY = "SELECT c.relname, n.nspname FROM pg_catalog.pg_class c " +
+ "LEFT JOIN pg_catalog.pg_namespace n ON n.oid = c.relnamespace WHERE c.relkind IN ('r','','v') AND n.nspname NOT IN " +
+ "('pg_catalog', 'information_schema', 'hawq_toolkit', 'pg_bitmapindex', 'pg_aoseg') AND n.nspname NOT LIKE 'pg_toast%' AND c.relname LIKE ?";
+
+ public static final String DATNAME = "datname";
+ public static final String SPCNAME = "spcname";
+ public static final String PTCNAME = "ptcname";
+ public static final String SCHEMA_NAME = "schema_name";
+ public static final String LANGUAGE_NAME = "lanname";
+ public static final String RELNAME = "relname";
+ public static final String SCHEMANAME = "schemaname";
+ public static final String PRONAME = "proname";
+ public static final String NSPNAME = "nspname";
+ public static final String WILDCARD = "*";
+
+ public static final List<String> INTERNAL_PROTOCOLS = HawqProtocols.getAllProtocols();
+ private static final String DEFAULT_DATABASE = "postgres";
+ private static final String JDBC_DRIVER_CLASS = "org.postgresql.Driver";
+
+ private Map<String, String> connectionProperties;
+
+ // we need to load class for the Postgres Driver directly to allow it to register with DriverManager
+ // since DriverManager's classloader will not be able to find it by itself due to plugin's special classloaders
+ static {
+ try {
+ Class.forName(JDBC_DRIVER_CLASS);
+ } catch (Throwable e) {
+ LOG.error("<== HawqClient.initializer : Unable to load JDBC driver " + JDBC_DRIVER_CLASS + " : " + e.getMessage());
+ throw new ExceptionInInitializerError(e);
+ }
+ }
+
+ public HawqClient(Map<String, String> connectionProperties) throws SQLException {
+ this.connectionProperties = connectionProperties;
+ }
+
+ /**
+ * Uses the connectionProperties and attempts to connect to Hawq.
+ * Returns a message depending on success or failure.
+ *
+ * @param connectionProperties Map which contains hostname, port, username, and password
+ * @return Map which contains connectivityStatus, message, description, objectId, and fieldName
+ */
+
+ public HashMap<String, Object> checkConnection(Map<String, String> connectionProperties) throws Exception {
+
+ boolean isConnected = false;
+ HashMap<String, Object> result = new HashMap<>();
+ Connection conn = null;
+
+ String description = CONNECTION_FAILURE_MESSAGE;
+
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("<== HawqClient.checkConnection Starting connection to hawq");
+ }
+
+ try {
+ conn = getConnection(connectionProperties);
+ if(conn.getCatalog() != null) {
+ isConnected = true;
+ description = CONNECTION_SUCCESSFUL_MESSAGE;
+ }
+ } catch (SQLException e) {
+ LOG.error("<== HawqClient.checkConnection Error: Failed to connect" + e);
+ description = e.getMessage();
+ } finally {
+ closeConnection(conn);
+ }
+
+ String message = isConnected ? CONNECTION_SUCCESSFUL_MESSAGE : CONNECTION_FAILURE_MESSAGE;
+
+ generateResponseDataMap(isConnected, message, description, null, null, result);
+
+ return result;
+ }
+
+ public List<String> getDatabaseList(String userInput) throws SQLException {
+ return queryHawq(userInput, DATNAME, DATABASE_LIST_QUERY, null);
+ }
+
+ public List<String> getTablespaceList(String userInput) throws SQLException {
+ return queryHawq(userInput, SPCNAME, TABLESPACE_LIST_QUERY, null);
+ }
+
+ public List<String> getProtocolList(String userInput) throws SQLException {
+ Set<String> allProtocols = new HashSet<>();
+ for (String protocol : INTERNAL_PROTOCOLS) {
+ if(protocol.startsWith(userInput) || userInput.equals(WILDCARD)) {
+ allProtocols.add(protocol);
+ }
+ }
+ allProtocols.addAll(queryHawq(userInput, PTCNAME, PROTOCOL_LIST_QUERY, null));
+ return new ArrayList<>(allProtocols);
+ }
+
+ public List<String> getSchemaList(String userInput, Map<String, List<String>> resources) throws SQLException {
+ return queryHawqPerDb(userInput, resources.get("database"), SCHEMA_NAME, SCHEMA_LIST_QUERY);
+ }
+
+ public List<String> getLanguageList(String userInput, Map<String, List<String>> resources) throws SQLException {
+ return queryHawqPerDb(userInput, resources.get("database"), LANGUAGE_NAME, LANGUAGE_LIST_QUERY);
+ }
+
+ public List<String> getTableList(String userInput, Map<String, List<String>> resources) throws SQLException {
+ return queryHawqPerDbAndSchema(userInput, resources, NSPNAME, RELNAME, TABLE_LIST_QUERY);
+ }
+
+ public List<String> getSequenceList(String userInput, Map<String, List<String>> resources) throws SQLException {
+ return queryHawqPerDbAndSchema(userInput, resources, SCHEMANAME, RELNAME, SEQUENCE_LIST_QUERY);
+ }
+
+ public List<String> getFunctionList(String userInput, Map<String, List<String>> resources) throws SQLException {
+ return queryHawqPerDbAndSchema(userInput, resources, NSPNAME, PRONAME, FUNCTION_LIST_QUERY);
+ }
+
+ private List<String> queryHawqPerDb(String userInput, List<String> databases, String columnName, String query) throws SQLException {
+ Set<String> uniqueResults = new HashSet<>();
+
+ //do for all databases
+ if (databases.contains(WILDCARD)) {
+ databases = getDatabaseList(WILDCARD);
+ }
+
+ for (String db : databases) {
+ uniqueResults.addAll(queryHawq(userInput, columnName, query, db));
+ }
+ return new ArrayList<>(uniqueResults);
+ }
+
+ private List<String> queryHawqPerDbAndSchema(String userInput, Map<String, List<String>> resources, String schemaColumnName, String resultColumnName, String query) throws SQLException {
+ Set<String> uniqueResults = new HashSet<>();
+ List<String> databases = resources.get("database");
+ List<String> schemas = resources.get("schema");
+
+ Connection conn = null;
+ PreparedStatement preparedStatement = null;
+ ResultSet resultSet = null;
+
+ if (databases.contains(WILDCARD)) {
+ databases = getDatabaseList(WILDCARD);
+ }
+
+ for (String db: databases) {
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("<== HawqClient.queryHawqPerDbAndSchema: Connecting to db: " + db);
+ }
+
+ try {
+ conn = getConnection(connectionProperties, db);
+ preparedStatement = handleWildcardPreparedStatement(userInput, query, conn);
+
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("<== HawqClient.queryHawqPerDbAndSchema Starting query: " + query);
+ }
+
+ resultSet = preparedStatement.executeQuery();
+
+ while(resultSet.next()) {
+ if(schemas.contains(resultSet.getString(schemaColumnName)) || schemas.contains(WILDCARD)) {
+ uniqueResults.add(resultSet.getString(resultColumnName));
+ }
+ }
+
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("<== HawqClient.queryHawqPerDbAndSchema Query result: " + uniqueResults.toString());
+ }
+
+ } catch (SQLException e) {
+ LOG.error("<== HawqClient.queryHawqPerDbAndSchema Error: Failed to get results for query: " + query + ", Error: " + e);
+ } finally {
+ closeResultSet(resultSet);
+ closeStatement(preparedStatement);
+ closeConnection(conn);
+ }
+ }
+ return new ArrayList<>(uniqueResults);
+ }
+
+ private List<String> queryHawq(String userInput, String columnName, String query, String database) {
+ List<String> result = new ArrayList<>();
+ Connection conn = null;
+ PreparedStatement preparedStatement = null;
+ ResultSet resultSet = null;
+
+ try {
+ conn = getConnection(connectionProperties, database);
+ preparedStatement = handleWildcardPreparedStatement(userInput, query, conn);
+
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("<== HawqClient.queryHawq Starting query: " + query);
+ }
+
+ resultSet = preparedStatement.executeQuery();
+
+ while(resultSet.next()) {
+ result.add(resultSet.getString(columnName));
+ }
+
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("<== HawqClient.queryHawq Query result: " + result.toString());
+ }
+
+ } catch (SQLException e) {
+ LOG.error("<== HawqClient.queryHawq Error: Failed to get result from query: " + query + ", Error: " + e);
+ } finally {
+ closeResultSet(resultSet);
+ closeStatement(preparedStatement);
+ closeConnection(conn);
+ }
+
+ return result;
+ }
+
+ private Connection getConnection(Map<String, String> connectionProperties) throws SQLException {
+ return getConnection(connectionProperties, null);
+ }
+
+ private Connection getConnection(Map<String, String> connectionProperties, String database) throws SQLException {
+
+ String db = database != null ? database : DEFAULT_DATABASE;
+ String url = String.format("jdbc:postgresql://%s:%s/%s", connectionProperties.get("hostname"), connectionProperties.get("port"), db);
+
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("<== HawqClient.checkConnection Connecting to: (" + url + ") with user: " + connectionProperties.get("username") );
+ }
+
+ Properties props = new Properties();
+ props.setProperty("user", connectionProperties.get("username"));
+ props.setProperty("password", connectionProperties.get("password"));
+ return DriverManager.getConnection(url, props);
+ }
+
+ private PreparedStatement handleWildcardPreparedStatement(String userInput, String query, Connection conn) throws SQLException {
+ PreparedStatement preparedStatement = conn.prepareStatement(query);
+ preparedStatement.setString(1, userInput.equals(WILDCARD) ? "%" : (userInput + "%"));
+ return preparedStatement;
+ }
+
+ private void closeResultSet(ResultSet rs) {
+ try {
+ if (rs != null) rs.close();
+ } catch (Exception e) {
+ // ignore
--- End diff --
There is nothing we can do, this is a standard practice to ignore closing errors. We can log them, but that would not be of a great value to anybody.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1092: HAWQ-1203. Ranger Plugin Service Implemen...
Posted by ictmalili <gi...@git.apache.org>.
Github user ictmalili commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1092#discussion_r96557368
--- Diff: ranger-plugin/integration/admin/pom.xml ---
@@ -0,0 +1,70 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+ <groupId>org.apache.hawq</groupId>
+ <artifactId>ranger-plugin-integration-admin</artifactId>
+ <packaging>jar</packaging>
+ <name>HAWQ Ranger Plugin - Integration Tests for Admin Plugin</name>
+ <description>HAWQ Ranger Plugin - Integration Tests for Admin Plugin</description>
+
+ <parent>
+ <groupId>org.apache.hawq</groupId>
+ <artifactId>ranger-plugin-integration</artifactId>
+ <version>2.1.0.0</version>
--- End diff --
Again for the version number here, which version number we should keep it?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1092: HAWQ-1203. Ranger Plugin Service Implemen...
Posted by ictmalili <gi...@git.apache.org>.
Github user ictmalili commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1092#discussion_r96780739
--- Diff: ranger-plugin/scripts/register_hawq.sh ---
@@ -0,0 +1,217 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+function usage() {
+ echo "USAGE: register_hawq.sh -r ranger_host:ranger_port -u ranger_user -p ranger_password -h hawq_host:hawq_port -w hawq_user -q hawq_password"
--- End diff --
@denalex May we change it adding indication the meaning to register to Ranger, such as register_hawq_to_ranger.sh, or anything else? Because HAWQ now has a management tool named as hawq register
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1092: HAWQ-1203. Ranger Plugin Service Implemen...
Posted by denalex <gi...@git.apache.org>.
Github user denalex commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1092#discussion_r96556904
--- Diff: .gitignore ---
@@ -41,6 +41,12 @@ objfiles.txt
.pydevproject
.cproject
.settings
+.classpath
+
+# IDEA Project
+*.iml
+.idea
--- End diff --
this removes IDEA editor artifacts from the list of potential GIT changes as those artifacts should never be committed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1092: HAWQ-1203. Ranger Plugin Service Implemen...
Posted by ictmalili <gi...@git.apache.org>.
Github user ictmalili commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1092#discussion_r96556516
--- Diff: ranger-plugin/admin-plugin/pom.xml ---
@@ -0,0 +1,90 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+ <groupId>org.apache.hawq</groupId>
+ <artifactId>ranger-plugin-admin</artifactId>
+ <packaging>jar</packaging>
+ <name>HAWQ Ranger Admin Plugin</name>
+ <description>HAWQ Ranger Admin Plugin</description>
+ <parent>
+ <groupId>org.apache.hawq</groupId>
+ <artifactId>ranger-plugin</artifactId>
+ <version>2.1.0.0</version>
--- End diff --
+1 for this comment. I guess this is the version number for apache hawq? If it is, I think we don't need record this version?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1092: HAWQ-1203. Ranger Plugin Service Implemen...
Posted by ictmalili <gi...@git.apache.org>.
Github user ictmalili commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1092#discussion_r96556724
--- Diff: ranger-plugin/admin-plugin/src/main/java/org/apache/hawq/ranger/service/HawqClient.java ---
@@ -0,0 +1,314 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.hawq.ranger.service;
+
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.hawq.ranger.model.HawqProtocols;
+
+import java.sql.Connection;
+import java.sql.DriverManager;
+import java.sql.PreparedStatement;
+import java.sql.SQLException;
+import java.sql.ResultSet;
+import java.util.*;
+
+import static org.apache.ranger.plugin.client.BaseClient.generateResponseDataMap;
+
+public class HawqClient {
+
+ private static final Log LOG = LogFactory.getLog(HawqClient.class);
+
+ public static final String CONNECTION_SUCCESSFUL_MESSAGE = "ConnectionTest Successful";
+ public static final String CONNECTION_FAILURE_MESSAGE = "Unable to retrieve any databases using given parameters.";
+
+ public static final String DATABASE_LIST_QUERY = "SELECT datname from pg_database WHERE " +
+ "datname NOT IN ('template0', 'template1', 'hcatalog') and datname LIKE ?";
+ public static final String TABLESPACE_LIST_QUERY = "SELECT spcname from pg_tablespace WHERE spcname LIKE ?";
+ public static final String PROTOCOL_LIST_QUERY = "SELECT ptcname from pg_catalog.pg_extprotocol WHERE ptcname LIKE ?";
+ public static final String SCHEMA_LIST_QUERY = "SELECT schema_name from information_schema.schemata WHERE " +
+ "schema_name NOT IN ('pg_catalog', 'information_schema', 'hawq_toolkit', 'pg_bitmapindex', 'pg_aoseg') AND schema_name NOT LIKE 'pg_toast%' AND schema_name LIKE ?";
+ public static final String LANGUAGE_LIST_QUERY = "SELECT lanname from pg_language WHERE lanname LIKE ?";
+ public static final String SEQUENCE_LIST_QUERY = "SELECT schemaname, relname from pg_statio_all_sequences WHERE relname LIKE ?";
+ public static final String FUNCTION_LIST_QUERY = "SELECT n.nspname, p.proname FROM pg_catalog.pg_proc p " +
+ "LEFT JOIN pg_catalog.pg_namespace n ON n.oid = p.pronamespace WHERE n.nspname NOT IN " +
+ "('pg_catalog', 'information_schema', 'hawq_toolkit', 'pg_bitmapindex', 'pg_aoseg') AND n.nspname NOT LIKE 'pg_toast%' AND p.proname LIKE ?";
+ public static final String TABLE_LIST_QUERY = "SELECT c.relname, n.nspname FROM pg_catalog.pg_class c " +
+ "LEFT JOIN pg_catalog.pg_namespace n ON n.oid = c.relnamespace WHERE c.relkind IN ('r','','v') AND n.nspname NOT IN " +
+ "('pg_catalog', 'information_schema', 'hawq_toolkit', 'pg_bitmapindex', 'pg_aoseg') AND n.nspname NOT LIKE 'pg_toast%' AND c.relname LIKE ?";
+
+ public static final String DATNAME = "datname";
+ public static final String SPCNAME = "spcname";
+ public static final String PTCNAME = "ptcname";
+ public static final String SCHEMA_NAME = "schema_name";
+ public static final String LANGUAGE_NAME = "lanname";
+ public static final String RELNAME = "relname";
+ public static final String SCHEMANAME = "schemaname";
+ public static final String PRONAME = "proname";
+ public static final String NSPNAME = "nspname";
+ public static final String WILDCARD = "*";
+
+ public static final List<String> INTERNAL_PROTOCOLS = HawqProtocols.getAllProtocols();
+ private static final String DEFAULT_DATABASE = "postgres";
+ private static final String JDBC_DRIVER_CLASS = "org.postgresql.Driver";
+
+ private Map<String, String> connectionProperties;
+
+ // we need to load class for the Postgres Driver directly to allow it to register with DriverManager
+ // since DriverManager's classloader will not be able to find it by itself due to plugin's special classloaders
+ static {
+ try {
+ Class.forName(JDBC_DRIVER_CLASS);
+ } catch (Throwable e) {
+ LOG.error("<== HawqClient.initializer : Unable to load JDBC driver " + JDBC_DRIVER_CLASS + " : " + e.getMessage());
+ throw new ExceptionInInitializerError(e);
+ }
+ }
+
+ public HawqClient(Map<String, String> connectionProperties) throws SQLException {
+ this.connectionProperties = connectionProperties;
+ }
+
+ /**
+ * Uses the connectionProperties and attempts to connect to Hawq.
+ * Returns a message depending on success or failure.
+ *
+ * @param connectionProperties Map which contains hostname, port, username, and password
+ * @return Map which contains connectivityStatus, message, description, objectId, and fieldName
+ */
+
+ public HashMap<String, Object> checkConnection(Map<String, String> connectionProperties) throws Exception {
+
+ boolean isConnected = false;
+ HashMap<String, Object> result = new HashMap<>();
+ Connection conn = null;
+
+ String description = CONNECTION_FAILURE_MESSAGE;
+
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("<== HawqClient.checkConnection Starting connection to hawq");
+ }
+
+ try {
+ conn = getConnection(connectionProperties);
+ if(conn.getCatalog() != null) {
+ isConnected = true;
+ description = CONNECTION_SUCCESSFUL_MESSAGE;
+ }
+ } catch (SQLException e) {
+ LOG.error("<== HawqClient.checkConnection Error: Failed to connect" + e);
+ description = e.getMessage();
+ } finally {
+ closeConnection(conn);
+ }
+
+ String message = isConnected ? CONNECTION_SUCCESSFUL_MESSAGE : CONNECTION_FAILURE_MESSAGE;
+
+ generateResponseDataMap(isConnected, message, description, null, null, result);
+
+ return result;
+ }
+
+ public List<String> getDatabaseList(String userInput) throws SQLException {
+ return queryHawq(userInput, DATNAME, DATABASE_LIST_QUERY, null);
+ }
+
+ public List<String> getTablespaceList(String userInput) throws SQLException {
+ return queryHawq(userInput, SPCNAME, TABLESPACE_LIST_QUERY, null);
+ }
+
+ public List<String> getProtocolList(String userInput) throws SQLException {
+ Set<String> allProtocols = new HashSet<>();
+ for (String protocol : INTERNAL_PROTOCOLS) {
+ if(protocol.startsWith(userInput) || userInput.equals(WILDCARD)) {
+ allProtocols.add(protocol);
+ }
+ }
+ allProtocols.addAll(queryHawq(userInput, PTCNAME, PROTOCOL_LIST_QUERY, null));
+ return new ArrayList<>(allProtocols);
+ }
+
+ public List<String> getSchemaList(String userInput, Map<String, List<String>> resources) throws SQLException {
+ return queryHawqPerDb(userInput, resources.get("database"), SCHEMA_NAME, SCHEMA_LIST_QUERY);
+ }
+
+ public List<String> getLanguageList(String userInput, Map<String, List<String>> resources) throws SQLException {
+ return queryHawqPerDb(userInput, resources.get("database"), LANGUAGE_NAME, LANGUAGE_LIST_QUERY);
+ }
+
+ public List<String> getTableList(String userInput, Map<String, List<String>> resources) throws SQLException {
+ return queryHawqPerDbAndSchema(userInput, resources, NSPNAME, RELNAME, TABLE_LIST_QUERY);
+ }
+
+ public List<String> getSequenceList(String userInput, Map<String, List<String>> resources) throws SQLException {
+ return queryHawqPerDbAndSchema(userInput, resources, SCHEMANAME, RELNAME, SEQUENCE_LIST_QUERY);
+ }
+
+ public List<String> getFunctionList(String userInput, Map<String, List<String>> resources) throws SQLException {
+ return queryHawqPerDbAndSchema(userInput, resources, NSPNAME, PRONAME, FUNCTION_LIST_QUERY);
+ }
+
+ private List<String> queryHawqPerDb(String userInput, List<String> databases, String columnName, String query) throws SQLException {
+ Set<String> uniqueResults = new HashSet<>();
+
+ //do for all databases
+ if (databases.contains(WILDCARD)) {
+ databases = getDatabaseList(WILDCARD);
+ }
+
+ for (String db : databases) {
+ uniqueResults.addAll(queryHawq(userInput, columnName, query, db));
+ }
+ return new ArrayList<>(uniqueResults);
+ }
+
+ private List<String> queryHawqPerDbAndSchema(String userInput, Map<String, List<String>> resources, String schemaColumnName, String resultColumnName, String query) throws SQLException {
+ Set<String> uniqueResults = new HashSet<>();
+ List<String> databases = resources.get("database");
+ List<String> schemas = resources.get("schema");
+
+ Connection conn = null;
+ PreparedStatement preparedStatement = null;
+ ResultSet resultSet = null;
+
+ if (databases.contains(WILDCARD)) {
+ databases = getDatabaseList(WILDCARD);
+ }
+
+ for (String db: databases) {
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("<== HawqClient.queryHawqPerDbAndSchema: Connecting to db: " + db);
+ }
+
+ try {
+ conn = getConnection(connectionProperties, db);
+ preparedStatement = handleWildcardPreparedStatement(userInput, query, conn);
+
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("<== HawqClient.queryHawqPerDbAndSchema Starting query: " + query);
+ }
+
+ resultSet = preparedStatement.executeQuery();
+
+ while(resultSet.next()) {
+ if(schemas.contains(resultSet.getString(schemaColumnName)) || schemas.contains(WILDCARD)) {
+ uniqueResults.add(resultSet.getString(resultColumnName));
+ }
+ }
+
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("<== HawqClient.queryHawqPerDbAndSchema Query result: " + uniqueResults.toString());
+ }
+
+ } catch (SQLException e) {
+ LOG.error("<== HawqClient.queryHawqPerDbAndSchema Error: Failed to get results for query: " + query + ", Error: " + e);
+ } finally {
+ closeResultSet(resultSet);
+ closeStatement(preparedStatement);
+ closeConnection(conn);
+ }
+ }
+ return new ArrayList<>(uniqueResults);
+ }
+
+ private List<String> queryHawq(String userInput, String columnName, String query, String database) {
+ List<String> result = new ArrayList<>();
+ Connection conn = null;
+ PreparedStatement preparedStatement = null;
+ ResultSet resultSet = null;
+
+ try {
+ conn = getConnection(connectionProperties, database);
+ preparedStatement = handleWildcardPreparedStatement(userInput, query, conn);
+
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("<== HawqClient.queryHawq Starting query: " + query);
+ }
+
+ resultSet = preparedStatement.executeQuery();
+
+ while(resultSet.next()) {
+ result.add(resultSet.getString(columnName));
+ }
+
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("<== HawqClient.queryHawq Query result: " + result.toString());
+ }
+
+ } catch (SQLException e) {
+ LOG.error("<== HawqClient.queryHawq Error: Failed to get result from query: " + query + ", Error: " + e);
+ } finally {
+ closeResultSet(resultSet);
+ closeStatement(preparedStatement);
+ closeConnection(conn);
+ }
+
+ return result;
+ }
+
+ private Connection getConnection(Map<String, String> connectionProperties) throws SQLException {
+ return getConnection(connectionProperties, null);
+ }
+
+ private Connection getConnection(Map<String, String> connectionProperties, String database) throws SQLException {
+
+ String db = database != null ? database : DEFAULT_DATABASE;
+ String url = String.format("jdbc:postgresql://%s:%s/%s", connectionProperties.get("hostname"), connectionProperties.get("port"), db);
+
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("<== HawqClient.checkConnection Connecting to: (" + url + ") with user: " + connectionProperties.get("username") );
+ }
+
+ Properties props = new Properties();
+ props.setProperty("user", connectionProperties.get("username"));
+ props.setProperty("password", connectionProperties.get("password"));
+ return DriverManager.getConnection(url, props);
+ }
+
+ private PreparedStatement handleWildcardPreparedStatement(String userInput, String query, Connection conn) throws SQLException {
+ PreparedStatement preparedStatement = conn.prepareStatement(query);
+ preparedStatement.setString(1, userInput.equals(WILDCARD) ? "%" : (userInput + "%"));
+ return preparedStatement;
+ }
+
+ private void closeResultSet(ResultSet rs) {
+ try {
+ if (rs != null) rs.close();
+ } catch (Exception e) {
+ // ignore
--- End diff --
What shall we do for these exceptions? Do we need log it out?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1092: HAWQ-1203. Ranger Plugin Service Implemen...
Posted by ictmalili <gi...@git.apache.org>.
Github user ictmalili commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1092#discussion_r96558081
--- Diff: ranger-plugin/scripts/register_hawq.sh ---
@@ -0,0 +1,217 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+function usage() {
+ echo "USAGE: register_hawq.sh -r ranger_host:ranger_port -u ranger_user -p ranger_password -h hawq_host:hawq_port -w hawq_user -q hawq_password"
--- End diff --
What's the meaning for this script? Will it be exposed directly out to users or is it just an internal script? If it will be directly exposed out to users, can we change it to a more meaningful name? It looks a little confusing now.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq issue #1092: HAWQ-1203. Ranger Plugin Service Implementation....
Posted by ictmalili <gi...@git.apache.org>.
Github user ictmalili commented on the issue:
https://github.com/apache/incubator-hawq/pull/1092
+1 except above comments I have made.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq issue #1092: HAWQ-1203. Ranger Plugin Service Implementation....
Posted by denalex <gi...@git.apache.org>.
Github user denalex commented on the issue:
https://github.com/apache/incubator-hawq/pull/1092
Thank you everyone for reviewing the PR. I will commit it tomorrow morning (PST time) and will create separate tasks for:
- running shell scripts via the tool, once all shell scripts are finalized
- discussing version management (which should be in master, how to change them on the branch before release, how to change in master after branching)
Please create other JIRAs if needed. The work on RPS codebase will continue after this merge via further additional JIRAs.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1092: HAWQ-1203. Ranger Plugin Service Implemen...
Posted by edespino <gi...@git.apache.org>.
Github user edespino commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1092#discussion_r96547717
--- Diff: ranger-plugin/admin-plugin/pom.xml ---
@@ -0,0 +1,90 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+ <groupId>org.apache.hawq</groupId>
+ <artifactId>ranger-plugin-admin</artifactId>
+ <packaging>jar</packaging>
+ <name>HAWQ Ranger Admin Plugin</name>
+ <description>HAWQ Ranger Admin Plugin</description>
+ <parent>
+ <groupId>org.apache.hawq</groupId>
+ <artifactId>ranger-plugin</artifactId>
+ <version>2.1.0.0</version>
--- End diff --
Is it intentional to use "2.1.0.0" as the HAWQ Ranger Admin Plugin version? Shouldn't it be on it's own version track?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq issue #1092: HAWQ-1203. Ranger Plugin Service Implementation....
Posted by ictmalili <gi...@git.apache.org>.
Github user ictmalili commented on the issue:
https://github.com/apache/incubator-hawq/pull/1092
@linwen @zhangh43 @stanlyxiang @interma @wcl14 @xunzhang Please review this PR and provide some comments. Thanks :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1092: HAWQ-1203. Ranger Plugin Service Implemen...
Posted by ictmalili <gi...@git.apache.org>.
Github user ictmalili commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1092#discussion_r96558333
--- Diff: ranger-plugin/service/src/main/java/org/apache/hawq/ranger/authorization/RangerHawqAuthorizer.java ---
@@ -0,0 +1,263 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hawq.ranger.authorization;
+
+import org.apache.commons.collections.CollectionUtils;
+import org.apache.commons.collections.MapUtils;
+import org.apache.commons.lang.StringUtils;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.hawq.ranger.authorization.model.AuthorizationRequest;
+import org.apache.hawq.ranger.authorization.model.AuthorizationResponse;
+import org.apache.hawq.ranger.authorization.model.HawqPrivilege;
+import org.apache.hawq.ranger.authorization.model.HawqResource;
+import org.apache.hawq.ranger.authorization.model.ResourceAccess;
+import org.apache.ranger.plugin.policyengine.RangerAccessRequest;
+import org.apache.ranger.plugin.policyengine.RangerAccessRequestImpl;
+import org.apache.ranger.plugin.policyengine.RangerAccessResource;
+import org.apache.ranger.plugin.policyengine.RangerAccessResourceImpl;
+import org.apache.ranger.plugin.policyengine.RangerAccessResult;
+import org.apache.ranger.plugin.service.RangerBasePlugin;
+
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+
+import static org.apache.hawq.ranger.authorization.Utils.HAWQ;
+
+/**
+ * Authorizer implementation that uses Ranger to make access decision. Implemented as a singleton.
+ */
+public class RangerHawqAuthorizer implements HawqAuthorizer {
+
+ private static final Log LOG = LogFactory.getLog(RangerHawqAuthorizer.class);
+
+ private static final RangerHawqAuthorizer INSTANCE = new RangerHawqAuthorizer();
+
+ private RangerBasePlugin rangerPlugin;
+
+ /**
+ * Returns the instance of the RangerHawqAuthorizer.
+ * @return the singleton instance
+ */
+ public static RangerHawqAuthorizer getInstance() {
+ return INSTANCE;
+ }
+
+ /**
+ * Constructor. Initializes Ranger Base Plugin to fetch policies from Ranger.
+ */
+ private RangerHawqAuthorizer() {
+
+ LOG.info("Initializing RangerHawqAuthorizer");
+
+ String appId = Utils.getAppId();
+
+ LOG.info(String.format("Initializing RangerBasePlugin for service %s:%s", HAWQ, appId));
+ rangerPlugin = new RangerBasePlugin(HAWQ, appId);
+ rangerPlugin.init();
+ LOG.info(String.format("Initialized RangerBasePlugin for service %s:%s", HAWQ, appId));
+ }
+
+ @Override
+ public AuthorizationResponse isAccessAllowed(AuthorizationRequest request) {
+
+ // validate request to make sure no data is missing
+ validateRequest(request);
+
+ // prepare response object
+ AuthorizationResponse response = new AuthorizationResponse();
+ response.setRequestId(request.getRequestId());
+ Set<ResourceAccess> access = new HashSet<>();
+ response.setAccess(access);
+
+ // iterate over resource requests, augment processed ones with the decision and add to the response
+ for (ResourceAccess resourceAccess : request.getAccess()) {
+ boolean accessAllowed = authorizeResource(resourceAccess, request.getUser());
+ resourceAccess.setAllowed(accessAllowed);
+ access.add(resourceAccess);
+ }
+
+ return response;
+ }
+
+ /**
+ * Authorizes access to a single resource for a given user.
+ *
+ * @param resourceAccess resource to authorize access to
+ * @param user user requesting authorization
+ * @return true if access is authorized, false otherwise
+ */
+ private boolean authorizeResource(ResourceAccess resourceAccess, String user) {
+
+ if (LOG.isDebugEnabled()) {
+ LOG.debug(String.format("Request: access for user=%s to resource=%s with privileges=%s",
+ user, resourceAccess.getResource(), resourceAccess.getPrivileges()));
+ }
+
+ RangerAccessResourceImpl rangerResource = new RangerAccessResourceImpl();
+ //resource.setOwnerUser();
+ for (Map.Entry<HawqResource, String> resourceEntry : resourceAccess.getResource().entrySet()) {
+ rangerResource.setValue(resourceEntry.getKey().name(), resourceEntry.getValue());
+ }
+
+ boolean accessAllowed = true;
+ // iterate over all privileges requested
+ for (HawqPrivilege privilege : resourceAccess.getPrivileges()) {
+ // TODO not clear how we will get user groups -- Kerberos case ?
+ Set<String> userGroups = Collections.emptySet();
+ boolean privilegeAuthorized = authorizeResourcePrivilege(rangerResource, privilege.name(), user, userGroups);
+ // ALL model of evaluation -- all privileges must be authorized for access to be allowed
+ if (!privilegeAuthorized) {
+ accessAllowed = false;
+ break; // terminate early if even a single privilege is not authorized
+ }
+ }
+
+ if (LOG.isDebugEnabled()) {
+ LOG.debug(String.format("Decision: accessAllowed=%s for user=%s to resource=%s with privileges=%s",
+ accessAllowed, user, resourceAccess.getResource(), resourceAccess.getPrivileges()));
+ }
+
+ return accessAllowed;
+ }
+
+ /**
+ * Authorizes access of a given type (privilege) to a single resource for a given user.
+ *
+ * @param rangerResource resource to authorize access to
+ * @param accessType privilege requested for a given resource
+ * @param user user requesting authorization
+ * @param userGroups groups a user belongs to
+ * @return true if access is authorized, false otherwise
+ */
+ private boolean authorizeResourcePrivilege(RangerAccessResource rangerResource, String accessType, String user, Set<String> userGroups) {
+
+ Map<String, String> resourceMap = rangerResource.getAsMap();
+ String database = resourceMap.get(HawqResource.database.name());
+ String schema = resourceMap.get(HawqResource.schema.name());
+ int resourceSize = resourceMap.size();
+
+ // special handling for non-leaf policies
+ if (accessType.equals(HawqPrivilege.create.name()) && database != null && schema == null && resourceSize == 1) {
--- End diff --
What's the meaning of resourceSize valued as 1 and 2? Could you add some comments in the code here? Thanks
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq issue #1092: HAWQ-1203. Ranger Plugin Service Implementation....
Posted by zhangh43 <gi...@git.apache.org>.
Github user zhangh43 commented on the issue:
https://github.com/apache/incubator-hawq/pull/1092
+1
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1092: HAWQ-1203. Ranger Plugin Service Implemen...
Posted by denalex <gi...@git.apache.org>.
Github user denalex commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1092#discussion_r96579569
--- Diff: ranger-plugin/integration/admin/pom.xml ---
@@ -0,0 +1,70 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+ <groupId>org.apache.hawq</groupId>
+ <artifactId>ranger-plugin-integration-admin</artifactId>
+ <packaging>jar</packaging>
+ <name>HAWQ Ranger Plugin - Integration Tests for Admin Plugin</name>
+ <description>HAWQ Ranger Plugin - Integration Tests for Admin Plugin</description>
+
+ <parent>
+ <groupId>org.apache.hawq</groupId>
+ <artifactId>ranger-plugin-integration</artifactId>
+ <version>2.1.0.0</version>
--- End diff --
We will keep the version the same as for the rest of HAWQ code. Perhaps a separate discussion will be useful to decide how we should manage it between releases.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1092: HAWQ-1203. Ranger Plugin Service Implemen...
Posted by denalex <gi...@git.apache.org>.
Github user denalex commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1092#discussion_r96579770
--- Diff: ranger-plugin/scripts/register_hawq.sh ---
@@ -0,0 +1,217 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+function usage() {
+ echo "USAGE: register_hawq.sh -r ranger_host:ranger_port -u ranger_user -p ranger_password -h hawq_host:hawq_port -w hawq_user -q hawq_password"
--- End diff --
@ictmalili -- this script must be run to register HAWQ service definition with Ranger Admin and create default instance of HAWQ service within Ranger that points to HAWQ Active master for the lookup. For now it will have to be run manually by users once, later we can think of automating it with Ambari.
What name would you suggest ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1092: HAWQ-1203. Ranger Plugin Service Implemen...
Posted by ictmalili <gi...@git.apache.org>.
Github user ictmalili commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1092#discussion_r96556661
--- Diff: .gitignore ---
@@ -41,6 +41,12 @@ objfiles.txt
.pydevproject
.cproject
.settings
+.classpath
+
+# IDEA Project
+*.iml
+.idea
--- End diff --
Are these necessary for each one's .gitignore file?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---