You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@impala.apache.org by ta...@apache.org on 2019/04/07 19:45:21 UTC
[impala] branch master updated (80b82b5 -> da15310)
This is an automated email from the ASF dual-hosted git repository.
tarmstrong pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git.
from 80b82b5 IMPALA-8360: Fix race conditions in thread-pool-test
new 4057331 IMPALA-7640: RENAME on managed Kudu table should rename Kudu table
new bcd7b04 Clean up generation of XML configuration files
new daa1bf9 IMPALA-8101: Thrift 11 and ext-data-source compilation are always run
new da15310 IMPALA-8382 Add support for SLES 12 SP3
The 4 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
bin/bootstrap_toolchain.py | 1 +
bin/create-test-configuration.sh | 32 ++-
bin/generate_xml_config.py | 118 ++++++++++
bin/jenkins/critique-gerrit-review.py | 3 +-
common/thrift/CMakeLists.txt | 22 +-
.../apache/impala/service/CatalogOpExecutor.java | 27 +++
fe/src/test/resources/hive-site.xml.py | 116 ++++++++++
fe/src/test/resources/mysql-hive-site.xml.template | 173 ---------------
.../postgresql-hive-site.xml.cdp.template | 173 ---------------
.../resources/postgresql-hive-site.xml.template | 246 ---------------------
fe/src/test/resources/sentry-site.xml.py | 65 ++++++
fe/src/test/resources/sentry-site.xml.template | 81 -------
.../test/resources/sentry-site_no_oo.xml.template | 104 ---------
fe/src/test/resources/sentry-site_oo.xml.template | 104 ---------
.../resources/sentry-site_oo_nogrant.xml.template | 104 ---------
.../queries/QueryTest/kudu_alter.test | 26 ++-
16 files changed, 392 insertions(+), 1003 deletions(-)
create mode 100755 bin/generate_xml_config.py
create mode 100644 fe/src/test/resources/hive-site.xml.py
delete mode 100644 fe/src/test/resources/mysql-hive-site.xml.template
delete mode 100644 fe/src/test/resources/postgresql-hive-site.xml.cdp.template
delete mode 100644 fe/src/test/resources/postgresql-hive-site.xml.template
create mode 100644 fe/src/test/resources/sentry-site.xml.py
delete mode 100644 fe/src/test/resources/sentry-site.xml.template
delete mode 100644 fe/src/test/resources/sentry-site_no_oo.xml.template
delete mode 100644 fe/src/test/resources/sentry-site_oo.xml.template
delete mode 100644 fe/src/test/resources/sentry-site_oo_nogrant.xml.template
[impala] 02/04: Clean up generation of XML configuration files
Posted by ta...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
tarmstrong pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
commit bcd7b0424597698540fec4ae201786ca39ad0dd0
Author: Todd Lipcon <to...@apache.org>
AuthorDate: Wed Apr 3 17:03:51 2019 -0700
Clean up generation of XML configuration files
hive-site.xml and sentry-site.xml in particular had grown multiple
slightly-different variants, differing only in a few small pieces. This
was difficult to maintain: in fact, while attempting to clean them up I
found a number of places that the MySQL and Postgres versions of
hive-site had diverged for no apparent reason.
This moves away from using the sed-based templating for these
configuration files, and instead uses python as a poor man's template
system. That enables much simpler conditional logic.
I briefly considered XSLT for this, but decided that Python is probably
easier for the average developer to follow, modify, and debug.
Along the way, I removed a few flags which appear to be no longer used
by Hive 2 or later, and a few items which were already commented out in
the previous template:
- hive.stats.dbclass
- hive.stats.dbconnectionstring
- hive.stats.jdbcdriver
These are no longer relevant after HIVE-12164 ("Remove jdbc stats
collection mechanism") in Hive 2.0.
- hive.metastore.rawstore.impl
This has always defaulted to 'ObjectStore' in Hive, so there was no
reason to set it explicitly.
- test.log.dir
- test.src.dir
These were listed in the config in a commented-out section. These were
commented out ever since 2012 when the file was first introduced.
This also fixes the postgres URL to not include a misplaced ';create'
parameter (which applies to Derby but not postgres).
Change-Id: Ief4434d80baae0fd7be7ffe7b2e07bae1ac45e47
Reviewed-on: http://gerrit.cloudera.org:8080/12930
Reviewed-by: Fredy Wijaya <fw...@cloudera.com>
Reviewed-by: Tim Armstrong <ta...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
bin/create-test-configuration.sh | 32 ++-
bin/generate_xml_config.py | 118 ++++++++++
bin/jenkins/critique-gerrit-review.py | 3 +-
fe/src/test/resources/hive-site.xml.py | 116 ++++++++++
fe/src/test/resources/mysql-hive-site.xml.template | 173 ---------------
.../postgresql-hive-site.xml.cdp.template | 173 ---------------
.../resources/postgresql-hive-site.xml.template | 246 ---------------------
fe/src/test/resources/sentry-site.xml.py | 65 ++++++
fe/src/test/resources/sentry-site.xml.template | 81 -------
.../test/resources/sentry-site_no_oo.xml.template | 104 ---------
fe/src/test/resources/sentry-site_oo.xml.template | 104 ---------
.../resources/sentry-site_oo_nogrant.xml.template | 104 ---------
12 files changed, 321 insertions(+), 998 deletions(-)
diff --git a/bin/create-test-configuration.sh b/bin/create-test-configuration.sh
index 79d5744..6d8916d 100755
--- a/bin/create-test-configuration.sh
+++ b/bin/create-test-configuration.sh
@@ -29,6 +29,16 @@ setup_report_build_error
# Search $1 ($GCIN) for strings that look like "${FOO}". If FOO is defined in
# the environment then replace "${FOO}" with the environment value. Also
# remove or leave special kerberos settings as desired. Sanity check at end.
+#
+# NOTE: for Hadoop-style XML configuration files (foo-site.xml) prefer using
+# bin/generate_xml_config.py instead of this method. This method is useful for
+# ini-style or other configuration formats.
+#
+# TODO(todd): convert remaining 'foo-site.xml' files to use the preferred
+# mechanism.
+#
+# TODO(todd): consider a better Python-based templating system for the other
+# configuration files as well.
function generate_config {
GCIN="$1"
GCOUT="$2"
@@ -128,14 +138,9 @@ rm -f authz-provider.ini
# Generate hive configs first so that schemaTool can be used to init the metastore schema
# if needed
-if $USE_CDP_HIVE; then
- # Certain configurations (like SentrySyncHMSNotificationsPostListener) do not work
- # with HMS 3.1.0. Use a cdp specific configuration template
- generate_config postgresql-hive-site.xml.cdp.template hive-site.xml
-else
- # This is not a CDP Hive installation. Use the regular template
- generate_config postgresql-hive-site.xml.template hive-site.xml
-fi
+
+$IMPALA_HOME/bin/generate_xml_config.py hive-site.xml.py hive-site.xml
+
generate_config hive-log4j2.properties.template hive-log4j2.properties
if [ $CREATE_METASTORE -eq 1 ]; then
@@ -192,10 +197,13 @@ fi
generate_config log4j.properties.template log4j.properties
generate_config hbase-site.xml.template hbase-site.xml
generate_config authz-policy.ini.template authz-policy.ini
-generate_config sentry-site.xml.template sentry-site.xml
-generate_config sentry-site_oo.xml.template sentry-site_oo.xml
-generate_config sentry-site_oo_nogrant.xml.template sentry-site_oo_nogrant.xml
-generate_config sentry-site_no_oo.xml.template sentry-site_no_oo.xml
+
+$IMPALA_HOME/bin/generate_xml_config.py sentry-site.xml.py sentry-site.xml
+for SENTRY_VARIANT in oo oo_nogrant no_oo ; do
+ export SENTRY_VARIANT
+ $IMPALA_HOME/bin/generate_xml_config.py sentry-site.xml.py \
+ sentry-site_${SENTRY_VARIANT}.xml
+done
if [ ! -z "${IMPALA_KERBERIZE}" ]; then
generate_config hbase-jaas-server.conf.template hbase-jaas-server.conf
diff --git a/bin/generate_xml_config.py b/bin/generate_xml_config.py
new file mode 100755
index 0000000..a06da7e
--- /dev/null
+++ b/bin/generate_xml_config.py
@@ -0,0 +1,118 @@
+#!/usr/bin/env python
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Script which uses a Python "template" to generate a Hadoop-style XML
+configuration file.
+
+The "template" is a Python module which should export a global variable called
+'CONFIG'. This variable should be a dictionary of keys/values. The values may
+use the special syntax '${FOO}' to substitute an environment variable (as a
+convenience over manually implementing the same).
+
+If you have an existing XML configuration and want to see it in convenient
+python form, you can use a snippet like the following from within the Python
+REPL:
+
+ import xml.etree.ElementTree as ET
+ import pprint
+ def convert(path):
+ e = ET.parse(path)
+ c = dict([(property.findtext('name'), property.findtext('value'))
+ for property in e.getroot()])
+ pprint.pprint(c, stream=file(path + ".py", "w"))
+
+"""
+
+from __future__ import with_statement
+import imp
+import os
+import re
+import sys
+from textwrap import dedent
+from xml.sax.saxutils import escape as xmlescape
+
+ENV_VAR_RE = re.compile(r'\${(.+?)\}')
+
+
+def _substitute_env_vars(s):
+ """ Substitute ${FOO} with the $FOO environment variable in 's' """
+ def lookup_func(match):
+ return os.environ[match.group(1)]
+ return ENV_VAR_RE.sub(lookup_func, s)
+
+
+def dump_config(d, source_path, out):
+ """
+ Dump a Hadoop-style XML configuration file.
+
+ 'd': a dictionary of name/value pairs.
+ 'source_path': the path where 'd' was parsed from.
+ 'out': stream to write to
+ """
+ header = """\
+ <?xml version="1.0"?>
+ <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
+ <!--
+
+ NOTE: THIS FILE IS AUTO-GENERATED FROM:
+ {source_path}
+
+ EDITS BY HAND WILL BE LOST!
+
+ -->
+ <configuration>""".format(source_path=os.path.abspath(source_path))
+ print >>out, dedent(header)
+ for k, v in sorted(d.iteritems()):
+ try:
+ v = _substitute_env_vars(v)
+ except KeyError, e:
+ raise Exception("failed environment variable substitution for value {k}: {e}"
+ .format(k=k, e=e))
+ print >>out, """\
+ <property>
+ <name>{name}</name>
+ <value>{value}</value>
+ </property>""".format(name=xmlescape(k), value=xmlescape(v))
+ print >>out, "</configuration>"
+
+
+def main():
+ if len(sys.argv) != 3:
+ print >>sys.stderr, "usage: {prog} <template> <out>".format(prog=sys.argv[0])
+ sys.exit(1)
+
+ _, in_path, out_path = sys.argv
+ mod = imp.load_source('template', in_path)
+ conf = mod.__dict__.get('CONFIG')
+ if not isinstance(conf, dict):
+ raise Exception("module in '{path}' should define a dict named CONFIG"
+ .format(path=in_path))
+
+ tmp_path = out_path + ".tmp"
+ with file(tmp_path, "w") as out:
+ try:
+ dump_config(conf, in_path, out)
+ except: # noqa
+ os.unlink(tmp_path)
+ raise
+ os.rename(tmp_path, out_path)
+
+
+if __name__ == "__main__":
+ main()
diff --git a/bin/jenkins/critique-gerrit-review.py b/bin/jenkins/critique-gerrit-review.py
index 7e6f3fa..8a3ca27 100755
--- a/bin/jenkins/critique-gerrit-review.py
+++ b/bin/jenkins/critique-gerrit-review.py
@@ -68,7 +68,8 @@ EXCLUDE_FILE_PATTERNS = [
re.compile(r".*/function-registry/impala_functions.py"), # Many long strings.
re.compile(r".*/catalog/BuiltinsDb.java"), # Many long strings.
re.compile(r".*/codegen/gen_ir_descriptions.py"), # Many long strings.
- re.compile(r".*/shell/ext-py/*") # Third-party code.
+ re.compile(r".*/shell/ext-py/*"), # Third-party code.
+ re.compile(r".*/fe/src/test/resources/.*.py") # Long lines in config files.
]
diff --git a/fe/src/test/resources/hive-site.xml.py b/fe/src/test/resources/hive-site.xml.py
new file mode 100644
index 0000000..fb64374
--- /dev/null
+++ b/fe/src/test/resources/hive-site.xml.py
@@ -0,0 +1,116 @@
+#!/usr/bin/env python
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import os
+
+hive_major_version = int(os.environ['IMPALA_HIVE_VERSION'][0])
+kerberize = os.environ.get('IMPALA_KERBERIZE') == '1'
+
+CONFIG = {
+ 'dfs.replication': '3'
+}
+
+# General Hive configuration.
+CONFIG.update({
+ # IMPALA-781: Impala doesn't support the default LazyBinaryColumnarSerde for RCFile.
+ 'hive.default.rcfile.serde': 'org.apache.hadoop.hive.serde2.columnar.ColumnarSerDe',
+
+ 'hive.metastore.client.connect.retry.delay': '0',
+ 'hive.metastore.client.socket.timeout': '120',
+ 'hive.metastore.uris': 'thrift://${INTERNAL_LISTEN_HOST}:9083',
+ 'hive.metastore.warehouse.dir': '${WAREHOUSE_LOCATION_PREFIX}/test-warehouse',
+
+ # Location of Hive per-query log files of the form: hive_job_log_<hive_query_id>.txt
+ 'hive.querylog.location': '${IMPALA_CLUSTER_LOGS_DIR}/hive',
+ 'hive.sentry.conf.url': 'file:///${IMPALA_HOME}/fe/src/test/resources/sentry-site.xml',
+
+ # Change back to NOSASL when HIVE-4232 is fixed.
+ # With NONE, Hive uses the plain SASL transport.
+ 'hive.server2.authentication': '${HIVE_S2_AUTH}',
+ # Disable user impersonation for HiveServer2 to avoid launch failure
+ # if username contains dots (IMPALA-6789).
+ 'hive.server2.enable.doAs': 'false',
+
+ # TODO(todd): should we be enabling stats autogather?
+ 'hive.stats.autogather': 'false',
+ 'hive.support.concurrency': 'true',
+})
+
+# HBase-related configs.
+# Impala processes need to connect to zookeeper on INTERNAL_LISTEN_HOST for HBase.
+CONFIG.update({
+ 'hive.cluster.delegation.token.store.zookeeper.connectString': '${INTERNAL_LISTEN_HOST}:2181',
+})
+
+if kerberize:
+ CONFIG.update({
+ 'hive.server2.authentication.kerberos.keytab': '${KRB5_KTNAME}',
+ 'hive.server2.authentication.kerberos.principal': '${MINIKDC_PRINC_HIVE}',
+
+ # These are necessary to connect to Kerberized HBase from within Hive jobs.
+ 'hbase.coprocessor.region.classes': 'org.apache.hadoop.hbase.security.token.TokenProvider',
+ 'hbase.master.kerberos.principal': '${MINIKDC_PRINC_HBSE}',
+ 'hbase.regionserver.kerberos.principal': '${MINIKDC_PRINC_HBSE}',
+ 'hbase.security.authentication': 'kerberos',
+ 'hbase.zookeeper.quorum': '${INTERNAL_LISTEN_HOST}',
+ })
+ # TODO: we currently don't kerberize the metastore. If we want to, we need to
+ # set:
+ # hive.metastore.sasl.enabled
+ # hive.metastore.kerberos.keytab.file
+ # hive.metastore.kerberos.principal
+
+if hive_major_version < 3:
+ CONFIG.update({
+ # TODO(vihang) Disabled for HMS3.
+ 'hive.metastore.event.listeners': 'org.apache.sentry.binding.metastore.SentrySyncHMSNotificationsPostEventListener',
+ })
+
+# Notifications-related configuration.
+# These are for enabling notification between Hive and Sentry as well as
+# metastore event processing in Impala (see IMPALA-7954)
+CONFIG.update({
+ 'hive.metastore.alter.notifications.basic': 'false',
+ 'hive.metastore.notification.parameters.exclude.patterns': '',
+ 'hive.metastore.notifications.add.thrift.objects': 'true',
+ 'hive.metastore.transactional.event.listeners': 'org.apache.hive.hcatalog.listener.DbNotificationListener',
+ 'hcatalog.message.factory.impl.json': 'org.apache.sentry.binding.metastore.messaging.json.SentryJSONMessageFactory',
+})
+
+# Database and JDO-related configs:
+db_type = os.environ.get('HMS_DB_TYPE', 'postgres')
+CONFIG.update({
+ 'datanucleus.autoCreateSchema': 'false',
+ 'datanucleus.fixedDatastore': 'false',
+ 'datanucleus.metadata.validate': 'false',
+ 'javax.jdo.option.ConnectionUserName': 'hiveuser',
+ 'javax.jdo.option.ConnectionPassword': 'password',
+})
+if db_type == 'postgres':
+ CONFIG.update({
+ 'javax.jdo.option.ConnectionDriverName': 'org.postgresql.Driver',
+ 'javax.jdo.option.ConnectionURL': 'jdbc:postgresql://localhost:5432/${METASTORE_DB}',
+ })
+elif db_type == 'mysql':
+ CONFIG.update({
+ 'javax.jdo.option.ConnectionDriverName': 'com.mysql.jdbc.Driver',
+ 'javax.jdo.option.ConnectionURL': 'jdbc:mysql://localhost:3306/${METASTORE_DB}?createDatabaseIfNotExist=true'
+ })
+else:
+ raise Exception("Unknown db type: %s", db_type)
diff --git a/fe/src/test/resources/mysql-hive-site.xml.template b/fe/src/test/resources/mysql-hive-site.xml.template
deleted file mode 100644
index 0c3d97f..0000000
--- a/fe/src/test/resources/mysql-hive-site.xml.template
+++ /dev/null
@@ -1,173 +0,0 @@
-<?xml version="1.0"?>
-<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
-<!--
- Licensed to the Apache Software Foundation (ASF) under one or more
- contributor license agreements. See the NOTICE file distributed with
- this work for additional information regarding copyright ownership.
- The ASF licenses this file to You under the Apache License, Version 2.0
- (the "License"); you may not use this file except in compliance with
- the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License.
--->
-
-<configuration>
-
-<!-- Hive Configuration can either be stored in this file or in the hadoop configuration files -->
-<!-- that are implied by Hadoop setup variables. -->
-<!-- Aside from Hadoop setup variables - this file is provided as a convenience so that Hive -->
-<!-- users do not have to edit hadoop configuration files (that may be managed as a centralized -->
-<!-- resource). -->
-<property>
- <name>hive.metastore.uris</name>
- <value>thrift://${INTERNAL_LISTEN_HOST}:9083</value>
-</property>
-<property>
- <name>javax.jdo.option.ConnectionURL</name>
- <value>jdbc:mysql://localhost:3306/${METASTORE_DB}?createDatabaseIfNotExist=true</value>
-</property>
-<property>
- <name>javax.jdo.option.ConnectionDriverName</name>
- <value>com.mysql.jdbc.Driver</value>
-</property>
-<property>
- <name>javax.jdo.option.ConnectionUserName</name>
- <value>hiveuser</value>
-</property>
-<property>
- <name>javax.jdo.option.ConnectionPassword</name>
- <value>password</value>
-</property>
-<property>
- <name>datanucleus.autoCreateSchema</name>
- <value>true</value>
-</property>
-<property>
- <name>hive.support.concurrency</name>
- <value>true</value>
-</property>
-<property>
- <name>datanucleus.metadata.validate</name>
- <value>false</value>
-</property>
-<property>
- <name>hive.stats.autogather</name>
- <value>false</value>
-</property>
-<property>
- <name>hive.stats.dbclass</name>
- <value>jdbc:mysql</value>
-</property>
-<property>
- <name>hive.stats.dbconnectionstring</name>
- <value>jdbc:mysql://${INTERNAL_LISTEN_HOST}:3306/${METASTORE_DB}_Stats?createDatabaseIfNotExist=true&user=hiveuser&password=password</value>
-</property>
-<property>
- <name>hive.stats.jdbcdriver</name>
- <value>com.mysql.jdbc.Driver</value>
-</property>
-<property>
- <name>datanucleus.fixedDatastore</name>
- <value>false</value>
-</property>
-<property>
- <name>hive.metastore.client.connect.retry.delay</name>
- <value>0</value>
- <description>Number of seconds for the client to wait between consecutive connection attempts</description>
-</property>
-<property>
- <name>hive.metastore.client.socket.timeout</name>
- <value>120</value>
- <description>MetaStore Client socket timeout in seconds</description>
-</property>
-<property>
- <!-- this should eventually be deprecated since the metastore should supply this -->
- <name>hive.metastore.warehouse.dir</name>
- <value>/test-warehouse</value>
- <description></description>
-</property>
-
-<property>
- <!-- Location of Hive per-query log files of the form: hive_job_log_<hive_query_id>.txt -->
- <name>hive.querylog.location</name>
- <value>${IMPALA_CLUSTER_LOGS_DIR}/hive</value>
-</property>
-
-<!--property>
- <name>test.log.dir</name>
- <value>${build.dir}/test/logs</value>
- <description></description>
-</property-->
-
-<!--property>
- <name>test.src.dir</name>
- <value>file://${build.dir}/src/test</value>
- <description></description>
-</property-->
-
-<property>
- <name>hive.sentry.conf.url</name>
- <value>file:///${IMPALA_HOME}/fe/src/test/resources/sentry-site.xml</value>
-</property>
-
-<property>
- <name>hive.metastore.rawstore.impl</name>
- <value>org.apache.hadoop.hive.metastore.ObjectStore</value>
- <description>Name of the class that implements org.apache.hadoop.hive.metastore.rawstore interface. This class is used to store and retrieval of raw metadata objects such as table, database</description>
-</property>
-
-<property>
- <name>dfs.replication</name>
- <value>3</value>
-</property>
-
-<!-- Change back to NOSASL when HIVE-4232 is fixed.
- With NONE, Hive uses the plain SASL transport. -->
-<property>
- <name>hive.server2.authentication</name>
- <value>NONE</value>
-</property>
-
-<property>
- <name>hive.server2.enable.doAs</name>
- <description>
- Disable user impersonation for HiveServer2 to avoid launch failure
- if username contains dots (IMPALA-6789)
- </description>
- <value>false</value>
-</property>
-
-<property>
- <!-- TODO: Remove this once Hive has changed their default back to ColumnarSerDe -->
- <name>hive.default.rcfile.serde</name>
- <value>org.apache.hadoop.hive.serde2.columnar.ColumnarSerDe</value>
- <description>The default SerDe hive will use for the rcfile format</description>
-</property>
-
-<!-- These properties are for enabling the notifications between Hive and Sentry -->
-<property>
- <name>hive.metastore.transactional.event.listeners</name>
- <value>org.apache.hive.hcatalog.listener.DbNotificationListener</value>
-</property>
-<property>
- <name>hive.metastore.event.listeners</name>
- <value>org.apache.sentry.binding.metastore.SentrySyncHMSNotificationsPostEventListener</value>
-</property>
-<property>
- <name>hcatalog.message.factory.impl.json</name>
- <value>org.apache.sentry.binding.metastore.messaging.json.SentryJSONMessageFactory</value>
-</property>
-
-<!-- Impala processes need to connect to zookeeper on INTERNAL_LISTEN_HOST for HBase. -->
-<property>
- <name>hive.cluster.delegation.token.store.zookeeper.connectString</name>
- <value>${INTERNAL_LISTEN_HOST}:2181</value>
- <description>The ZooKeeper token store connect string.</description>
-</property>
-</configuration>
diff --git a/fe/src/test/resources/postgresql-hive-site.xml.cdp.template b/fe/src/test/resources/postgresql-hive-site.xml.cdp.template
deleted file mode 100644
index a1425c8..0000000
--- a/fe/src/test/resources/postgresql-hive-site.xml.cdp.template
+++ /dev/null
@@ -1,173 +0,0 @@
-<?xml version="1.0"?>
-<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
-<!--
- Licensed to the Apache Software Foundation (ASF) under one or more
- contributor license agreements. See the NOTICE file distributed with
- this work for additional information regarding copyright ownership.
- The ASF licenses this file to You under the Apache License, Version 2.0
- (the "License"); you may not use this file except in compliance with
- the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License.
--->
-<configuration>
- <!-- Hive Configuration can either be stored in this file or in the hadoop configuration files -->
- <!-- that are implied by Hadoop setup variables. -->
- <!-- Aside from Hadoop setup variables - this file is provided as a convenience so that Hive -->
- <!-- users do not have to edit hadoop configuration files (that may be managed as a centralized -->
- <!-- resource). -->
- <property>
- <name>hive.metastore.uris</name>
- <value>thrift://${INTERNAL_LISTEN_HOST}:9083</value>
- </property>
- <property>
- <name>javax.jdo.option.ConnectionURL</name>
- <value>jdbc:postgresql://localhost:5432/${METASTORE_DB}</value>
- </property>
- <property>
- <name>javax.jdo.option.ConnectionDriverName</name>
- <value>org.postgresql.Driver</value>
- </property>
- <property>
- <name>javax.jdo.option.ConnectionUserName</name>
- <value>hiveuser</value>
- </property>
- <property>
- <name>javax.jdo.option.ConnectionPassword</name>
- <value>password</value>
- </property>
- <property>
- <name>datanucleus.autoCreateSchema</name>
- <value>false</value>
- </property>
- <property>
- <name>hive.support.concurrency</name>
- <value>true</value>
- </property>
- <property>
- <name>datanucleus.metadata.validate</name>
- <value>false</value>
- </property>
- <property>
- <name>hive.stats.autogather</name>
- <value>false</value>
- </property>
- <property>
- <name>hive.stats.dbclass</name>
- <value>jdbc:postgresql</value>
- </property>
- <property>
- <name>hive.stats.jdbcdriver</name>
- <value>org.postgresql.Driver</value>
- </property>
- <property>
- <name>datanucleus.fixedDatastore</name>
- <value>false</value>
- </property>
- <property>
- <name>hive.metastore.client.connect.retry.delay</name>
- <value>0</value>
- <description>Number of seconds for the client to wait between consecutive connection attempts</description>
- </property>
- <property>
- <name>hive.metastore.client.socket.timeout</name>
- <value>120</value>
- <description>MetaStore Client socket timeout in seconds</description>
- </property>
- <property>
- <!-- this should eventually be deprecated since the metastore should supply this -->
- <name>hive.metastore.warehouse.dir</name>
- <value>${WAREHOUSE_LOCATION_PREFIX}/test-warehouse</value>
- <description/>
- </property>
- <property>
- <!-- Location of Hive per-query log files of the form: hive_job_log_<hive_query_id>.txt -->
- <name>hive.querylog.location</name>
- <value>${IMPALA_CLUSTER_LOGS_DIR}/hive</value>
- </property>
- <property>
- <name>hive.sentry.conf.url</name>
- <value>file:///${IMPALA_HOME}/fe/src/test/resources/sentry-site.xml</value>
- </property>
- <!-- Change back to NOSASL when HIVE-4232 is fixed.
- With NONE, Hive uses the plain SASL transport. -->
- <property>
- <name>hive.server2.authentication</name>
- <value>${HIVE_S2_AUTH}</value>
- <!-- NONE | KERBEROS -->
- </property>
- <!-- BEGIN Kerberos settings -->
- <property>
- <name>hive.server2.authentication.kerberos.principal</name>
- <value>${MINIKDC_PRINC_HIVE}</value>
- </property>
- <property>
- <name>hive.server2.authentication.kerberos.keytab</name>
- <value>${KRB5_KTNAME}</value>
- </property>
- <!-- Start IMPALA-4125 -->
- <property>
- <name>hbase.security.authentication</name>
- <value>kerberos</value>
- </property>
- <property>
- <name>hbase.master.kerberos.principal</name>
- <value>${MINIKDC_PRINC_HBSE}</value>
- </property>
- <property>
- <name>hbase.regionserver.kerberos.principal</name>
- <value>${MINIKDC_PRINC_HBSE}</value>
- </property>
- <property>
- <name>hbase.zookeeper.quorum</name>
- <value>${INTERNAL_LISTEN_HOST}</value>
- </property>
- <!-- END IMPALA-4125 -->
- <property>
- <name>hbase.coprocessor.region.classes</name>
- <value>org.apache.hadoop.hbase.security.token.TokenProvider</value>
- </property>
- <!-- END Kerberos settings -->
- <property>
- <name>hive.server2.enable.doAs</name>
- <description>
- Disable user impersonation for HiveServer2 to avoid launch failure
- if username contains dots (IMPALA-6789)
- </description>
- <value>false</value>
- </property>
- <property>
- <!-- TODO: Remove this once Hive has changed their default back to ColumnarSerDe -->
- <name>hive.default.rcfile.serde</name>
- <value>org.apache.hadoop.hive.serde2.columnar.ColumnarSerDe</value>
- <description>The default SerDe hive will use for the rcfile format</description>
- </property>
- <!-- These properties are for enabling the notifications between Hive and Sentry. This is
-also required for metastore event processing feature in Impala (See IMPALA-7954) -->
- <property>
- <name>hive.metastore.transactional.event.listeners</name>
- <value>org.apache.hive.hcatalog.listener.DbNotificationListener</value>
- </property>
- <!-- TODO Impala-HMS3 Integration: Commented this out since it throws an exception on HMS 3.1.0
-<property>
- <name>hive.metastore.event.listeners</name>
- <value>org.apache.sentry.binding.metastore.SentrySyncHMSNotificationsPostEventListener</value>
-</property>
--->
- <property>
- <name>hcatalog.message.factory.impl.json</name>
- <value>org.apache.sentry.binding.metastore.messaging.json.SentryJSONMessageFactory</value>
- </property>
- <!-- Impala processes need to connect to zookeeper on INTERNAL_LISTEN_HOST for HBase. -->
- <property>
- <name>hive.cluster.delegation.token.store.zookeeper.connectString</name>
- <value>${INTERNAL_LISTEN_HOST}:2181</value>
- <description>The ZooKeeper token store connect string.</description>
- </property>
-</configuration>
diff --git a/fe/src/test/resources/postgresql-hive-site.xml.template b/fe/src/test/resources/postgresql-hive-site.xml.template
deleted file mode 100644
index 8afb2ce..0000000
--- a/fe/src/test/resources/postgresql-hive-site.xml.template
+++ /dev/null
@@ -1,246 +0,0 @@
-<?xml version="1.0"?>
-<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
-<!--
- Licensed to the Apache Software Foundation (ASF) under one or more
- contributor license agreements. See the NOTICE file distributed with
- this work for additional information regarding copyright ownership.
- The ASF licenses this file to You under the Apache License, Version 2.0
- (the "License"); you may not use this file except in compliance with
- the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License.
--->
-
-<configuration>
-
-<!-- Hive Configuration can either be stored in this file or in the hadoop configuration files -->
-<!-- that are implied by Hadoop setup variables. -->
-<!-- Aside from Hadoop setup variables - this file is provided as a convenience so that Hive -->
-<!-- users do not have to edit hadoop configuration files (that may be managed as a centralized -->
-<!-- resource). -->
-<property>
- <name>hive.metastore.uris</name>
- <value>thrift://${INTERNAL_LISTEN_HOST}:9083</value>
-</property>
-<property>
- <name>javax.jdo.option.ConnectionURL</name>
- <value>jdbc:postgresql://localhost:5432/${METASTORE_DB}</value>
-</property>
-<property>
- <name>javax.jdo.option.ConnectionDriverName</name>
- <value>org.postgresql.Driver</value>
-</property>
-<property>
- <name>javax.jdo.option.ConnectionUserName</name>
- <value>hiveuser</value>
-</property>
-<property>
- <name>javax.jdo.option.ConnectionPassword</name>
- <value>password</value>
-</property>
-<property>
- <name>datanucleus.autoCreateSchema</name>
- <value>false</value>
-</property>
-<property>
- <name>hive.support.concurrency</name>
- <value>true</value>
-</property>
-<property>
- <name>datanucleus.metadata.validate</name>
- <value>false</value>
-</property>
-<property>
- <name>hive.stats.autogather</name>
- <value>false</value>
-</property>
-<property>
- <name>hive.stats.dbclass</name>
- <value>jdbc:postgresql</value>
-</property>
-<property>
- <name>hive.stats.jdbcdriver</name>
- <value>org.postgresql.Driver</value>
-</property>
-
-<property>
- <name>datanucleus.fixedDatastore</name>
- <value>false</value>
-</property>
-<property>
- <name>hive.metastore.client.connect.retry.delay</name>
- <value>0</value>
- <description>Number of seconds for the client to wait between consecutive connection attempts</description>
-</property>
-<property>
- <name>hive.metastore.client.socket.timeout</name>
- <value>120</value>
- <description>MetaStore Client socket timeout in seconds</description>
-</property>
-<property>
- <!-- this should eventually be deprecated since the metastore should supply this -->
- <name>hive.metastore.warehouse.dir</name>
- <value>${WAREHOUSE_LOCATION_PREFIX}/test-warehouse</value>
- <description></description>
-</property>
-
-<property>
- <!-- Location of Hive per-query log files of the form: hive_job_log_<hive_query_id>.txt -->
- <name>hive.querylog.location</name>
- <value>${IMPALA_CLUSTER_LOGS_DIR}/hive</value>
-</property>
-
-<!--property>
- <name>test.log.dir</name>
- <value>${build.dir}/test/logs</value>
- <description></description>
-</property-->
-
-<!--property>
- <name>test.src.dir</name>
- <value>file://${build.dir}/src/test</value>
- <description></description>
-</property-->
-
-<property>
- <name>hive.sentry.conf.url</name>
- <value>file:///${IMPALA_HOME}/fe/src/test/resources/sentry-site.xml</value>
-</property>
-
-<property>
- <name>hive.metastore.rawstore.impl</name>
- <value>org.apache.hadoop.hive.metastore.ObjectStore</value>
- <description>Name of the class that implements org.apache.hadoop.hive.metastore.rawstore interface. This class is used to store and retrieval of raw metadata objects such as table, database</description>
-</property>
-
-<property>
- <name>dfs.replication</name>
- <value>3</value>
-</property>
-
-<!-- Change back to NOSASL when HIVE-4232 is fixed.
- With NONE, Hive uses the plain SASL transport. -->
-<property>
- <name>hive.server2.authentication</name>
- <value>${HIVE_S2_AUTH}</value> <!-- NONE | KERBEROS -->
-</property>
-
-<!-- BEGIN Kerberos settings -->
-<property>
- <name>hive.server2.authentication.kerberos.principal</name>
- <value>${MINIKDC_PRINC_HIVE}</value>
-</property>
-
-<property>
- <name>hive.server2.authentication.kerberos.keytab</name>
- <value>${KRB5_KTNAME}</value>
-</property>
-
-<!-- Having problems getting the metastore up with Kerberos. Defer for now.
-<property>
- <name>hive.metastore.sasl.enabled</name>
- <value>true</value>
- <description>If true, the metastore thrift interface will be secured
- with SASL. Clients must authenticate with Kerberos.</description>
-</property>
-
-<property>
- <name>hive.metastore.kerberos.keytab.file</name>
- <value>${KRB5_KTNAME}</value>
-</property>
-
-<property>
- <name>hive.metastore.kerberos.principal</name>
- <value>${MINIKDC_PRINC_HIVE}</value>
-</property>
--->
-
-<!-- Start IMPALA-4125 -->
-<property>
- <name>hbase.security.authentication</name>
- <value>kerberos</value>
-</property>
-
-<property>
- <name>hbase.master.kerberos.principal</name>
- <value>${MINIKDC_PRINC_HBSE}</value>
-</property>
-
-<property>
- <name>hbase.regionserver.kerberos.principal</name>
- <value>${MINIKDC_PRINC_HBSE}</value>
-</property>
-
-<property>
- <name>hbase.zookeeper.quorum</name>
- <value>${INTERNAL_LISTEN_HOST}</value>
-</property>
-<!-- END IMPALA-4125 -->
-
-<property>
- <name>hbase.coprocessor.region.classes</name>
- <value>org.apache.hadoop.hbase.security.token.TokenProvider</value>
-</property>
-
-<!-- END Kerberos settings -->
-
-<property>
- <name>hive.server2.enable.doAs</name>
- <description>
- Disable user impersonation for HiveServer2 to avoid launch failure
- if username contains dots (IMPALA-6789)
- </description>
- <value>false</value>
-</property>
-
-<property>
- <!-- TODO: Remove this once Hive has changed their default back to ColumnarSerDe -->
- <name>hive.default.rcfile.serde</name>
- <value>org.apache.hadoop.hive.serde2.columnar.ColumnarSerDe</value>
- <description>The default SerDe hive will use for the rcfile format</description>
-</property>
-
-<!-- These properties are for enabling the notifications between Hive and Sentry -->
-<property>
- <name>hive.metastore.transactional.event.listeners</name>
- <value>org.apache.hive.hcatalog.listener.DbNotificationListener</value>
-</property>
-<property>
- <name>hive.metastore.event.listeners</name>
- <value>org.apache.sentry.binding.metastore.SentrySyncHMSNotificationsPostEventListener</value>
-</property>
-<property>
- <name>hcatalog.message.factory.impl.json</name>
- <value>org.apache.sentry.binding.metastore.messaging.json.SentryJSONMessageFactory</value>
-</property>
-
-<!-- Impala processes need to connect to zookeeper on INTERNAL_LISTEN_HOST for HBase. -->
-<property>
- <name>hive.cluster.delegation.token.store.zookeeper.connectString</name>
- <value>${INTERNAL_LISTEN_HOST}:2181</value>
- <description>The ZooKeeper token store connect string.</description>
-</property>
-
-<!-- This property is required to issue invalidates based on metastore events.
-See IMPALA-7954 for details -->
-<property>
- <name>hive.metastore.notifications.add.thrift.objects</name>
- <value>true</value>
-</property>
-<property>
- <name>hive.metastore.alter.notifications.basic</name>
- <value>false</value>
-</property>
-<!-- This property is required so that table parameters do not exclude keys which certain features
- like event processing depend on. See IMPALA-8273 for details -->
-<property>
- <name>hive.metastore.notification.parameters.exclude.patterns</name>
- <value></value>
-</property>
-</configuration>
diff --git a/fe/src/test/resources/sentry-site.xml.py b/fe/src/test/resources/sentry-site.xml.py
new file mode 100644
index 0000000..64a25fd
--- /dev/null
+++ b/fe/src/test/resources/sentry-site.xml.py
@@ -0,0 +1,65 @@
+#!/usr/bin/env python
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+import os
+
+_SETTINGS_BY_VARIANT = dict({
+ 'oo': 'all_with_grant',
+ 'oo_nogrant': 'all',
+ 'no_oo': 'none'})
+
+variant = os.environ.get('SENTRY_VARIANT')
+
+CONFIG = {
+ 'hive.sentry.server': 'server1',
+
+ # Use a space to mean allow all privileges. See SENTRY-2424
+ 'sentry.db.explicit.grants.permitted': ' ',
+
+ 'sentry.service.admin.group': '${USER}',
+ # TODO(IMPALA-5686): remove one of the following two properties when Sentry
+ # standardizes on one.
+ 'sentry.service.client.server.rpc-address': '${INTERNAL_LISTEN_HOST}',
+ 'sentry.service.client.server.rpc-addresses': '${INTERNAL_LISTEN_HOST}',
+ 'sentry.service.client.server.rpc-port': '30911',
+
+ # Enable HMS follower.
+ 'sentry.service.security.mode': 'none',
+ 'sentry.service.server.rpc-address': '${INTERNAL_LISTEN_HOST}',
+ 'sentry.service.server.rpc-port': '30911',
+
+ 'sentry.store.jdbc.driver': 'org.postgresql.Driver',
+ 'sentry.store.jdbc.password': 'password',
+ 'sentry.store.jdbc.url': 'jdbc:postgresql://localhost:5432/${SENTRY_POLICY_DB}',
+ 'sentry.store.jdbc.user': 'hiveuser',
+ 'sentry.verify.schema.version': 'false',
+}
+
+if variant is not None:
+ # TODO(todd): these settings seem generic, rather than related to the object ownership
+ # feature, but they're only set in the "variants". Is that right?
+ CONFIG.update({
+ 'sentry.hive.testing.mode': 'true',
+ 'sentry.policy.store.plugins': 'org.apache.sentry.hdfs.SentryPlugin',
+ # Custom group mapping for custom cluster tests .
+ 'sentry.store.group.mapping': 'org.apache.impala.service.CustomClusterGroupMapper',
+ 'sentry.service.processor.factories': '${SENTRY_PROCESSOR_FACTORIES}',
+ })
+
+ CONFIG['sentry.db.policy.store.owner.as.privilege'] = _SETTINGS_BY_VARIANT[variant]
diff --git a/fe/src/test/resources/sentry-site.xml.template b/fe/src/test/resources/sentry-site.xml.template
deleted file mode 100644
index e48b8ff..0000000
--- a/fe/src/test/resources/sentry-site.xml.template
+++ /dev/null
@@ -1,81 +0,0 @@
-<?xml version="1.0"?>
-<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
-<!--
- Licensed to the Apache Software Foundation (ASF) under one or more
- contributor license agreements. See the NOTICE file distributed with
- this work for additional information regarding copyright ownership.
- The ASF licenses this file to You under the Apache License, Version 2.0
- (the "License"); you may not use this file except in compliance with
- the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License.
--->
-
-<configuration>
-<property>
- <name>sentry.service.security.mode</name>
- <value>none</value>
-</property>
-<property>
- <name>hive.sentry.server</name>
- <value>server1</value>
-</property>
-<property>
- <name>sentry.service.admin.group</name>
- <value>${USER}</value>
-</property>
-<property>
- <name>sentry.service.server.rpc-address</name>
- <value>${INTERNAL_LISTEN_HOST}</value>
-</property>
-<property>
- <name>sentry.service.server.rpc-port</name>
- <value>30911</value>
-</property>
-<property>
- <name>sentry.service.client.server.rpc-address</name>
- <value>${INTERNAL_LISTEN_HOST}</value>
-</property>
-<property>
-<!--
- TODO(IMPALA-5686): Remove this or the previous property when Sentry standardizes on one.
--->
- <name>sentry.service.client.server.rpc-addresses</name>
- <value>${INTERNAL_LISTEN_HOST}</value>
-</property>
-<property>
- <name>sentry.service.client.server.rpc-port</name>
- <value>30911</value>
-</property>
-<property>
- <name>sentry.store.jdbc.url</name>
- <value>jdbc:postgresql://localhost:5432/${SENTRY_POLICY_DB}/;create=true</value>
-</property>
-<property>
- <name>sentry.store.jdbc.user</name>
- <value>hiveuser</value>
-</property>
-<property>
- <name>sentry.store.jdbc.password</name>
- <value>password</value>
-</property>
-<property>
- <name>sentry.verify.schema.version</name>
- <value>false</value>
-</property>
-<property>
- <name>sentry.store.jdbc.driver</name>
- <value>org.postgresql.Driver</value>
-</property>
-<!-- Use a space to mean allow all privileges. See SENTRY-2424 -->
-<property>
- <name>sentry.db.explicit.grants.permitted</name>
- <value> </value>
-</property>
-</configuration>
diff --git a/fe/src/test/resources/sentry-site_no_oo.xml.template b/fe/src/test/resources/sentry-site_no_oo.xml.template
deleted file mode 100644
index 8ad8be6..0000000
--- a/fe/src/test/resources/sentry-site_no_oo.xml.template
+++ /dev/null
@@ -1,104 +0,0 @@
-<?xml version="1.0"?>
-<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
-<!--
- Licensed to the Apache Software Foundation (ASF) under one or more
- contributor license agreements. See the NOTICE file distributed with
- this work for additional information regarding copyright ownership.
- The ASF licenses this file to You under the Apache License, Version 2.0
- (the "License"); you may not use this file except in compliance with
- the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License.
--->
-
-<configuration>
- <property>
- <name>sentry.service.security.mode</name>
- <value>none</value>
- </property>
- <property>
- <name>hive.sentry.server</name>
- <value>server1</value>
- </property>
- <property>
- <name>sentry.service.admin.group</name>
- <value>${USER}</value>
- </property>
- <property>
- <name>sentry.service.server.rpc-address</name>
- <value>${INTERNAL_LISTEN_HOST}</value>
- </property>
- <property>
- <name>sentry.service.server.rpc-port</name>
- <value>30911</value>
- </property>
- <property>
- <name>sentry.service.client.server.rpc-address</name>
- <value>${INTERNAL_LISTEN_HOST}</value>
- </property>
- <property>
- <!--
- TODO(IMPALA-5686): Remove this or the previous property when Sentry standardizes on one.
- -->
- <name>sentry.service.client.server.rpc-addresses</name>
- <value>${INTERNAL_LISTEN_HOST}</value>
- </property>
- <property>
- <name>sentry.service.client.server.rpc-port</name>
- <value>30911</value>
- </property>
- <property>
- <name>sentry.store.jdbc.url</name>
- <value>jdbc:postgresql://${INTERNAL_LISTEN_HOST}:5432/${SENTRY_POLICY_DB}/;create=true</value>
- </property>
- <property>
- <name>sentry.store.jdbc.user</name>
- <value>hiveuser</value>
- </property>
- <property>
- <name>sentry.store.jdbc.password</name>
- <value>password</value>
- </property>
- <property>
- <name>sentry.verify.schema.version</name>
- <value>false</value>
- </property>
- <property>
- <name>sentry.store.jdbc.driver</name>
- <value>org.postgresql.Driver</value>
- </property>
- <!-- This property disables owner privilege. -->
- <property>
- <name>sentry.db.policy.store.owner.as.privilege</name>
- <value>none</value>
- </property>
- <!-- These properties enable HMS follower. -->
- <property>
- <name>sentry.service.processor.factories</name>
- <value>${SENTRY_PROCESSOR_FACTORIES}</value>
- </property>
- <property>
- <name>sentry.policy.store.plugins</name>
- <value>org.apache.sentry.hdfs.SentryPlugin</value>
- </property>
- <property>
- <name>sentry.hive.testing.mode</name>
- <value>true</value>
- </property>
- <!-- Use a space to mean allow all privileges. See SENTRY-2424 -->
- <property>
- <name>sentry.db.explicit.grants.permitted</name>
- <value> </value>
- </property>
- <!-- Custom group mapping for custom cluster tests -->
- <property>
- <name>sentry.store.group.mapping</name>
- <value>org.apache.impala.service.CustomClusterGroupMapper</value>
- </property>
-</configuration>
diff --git a/fe/src/test/resources/sentry-site_oo.xml.template b/fe/src/test/resources/sentry-site_oo.xml.template
deleted file mode 100644
index 2a3fdce..0000000
--- a/fe/src/test/resources/sentry-site_oo.xml.template
+++ /dev/null
@@ -1,104 +0,0 @@
-<?xml version="1.0"?>
-<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
-<!--
- Licensed to the Apache Software Foundation (ASF) under one or more
- contributor license agreements. See the NOTICE file distributed with
- this work for additional information regarding copyright ownership.
- The ASF licenses this file to You under the Apache License, Version 2.0
- (the "License"); you may not use this file except in compliance with
- the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License.
--->
-
-<configuration>
- <property>
- <name>sentry.service.security.mode</name>
- <value>none</value>
- </property>
- <property>
- <name>hive.sentry.server</name>
- <value>server1</value>
- </property>
- <property>
- <name>sentry.service.admin.group</name>
- <value>${USER}</value>
- </property>
- <property>
- <name>sentry.service.server.rpc-address</name>
- <value>${INTERNAL_LISTEN_HOST}</value>
- </property>
- <property>
- <name>sentry.service.server.rpc-port</name>
- <value>30911</value>
- </property>
- <property>
- <name>sentry.service.client.server.rpc-address</name>
- <value>${INTERNAL_LISTEN_HOST}</value>
- </property>
- <property>
- <!--
- TODO(IMPALA-5686): Remove this or the previous property when Sentry standardizes on one.
- -->
- <name>sentry.service.client.server.rpc-addresses</name>
- <value>${INTERNAL_LISTEN_HOST}</value>
- </property>
- <property>
- <name>sentry.service.client.server.rpc-port</name>
- <value>30911</value>
- </property>
- <property>
- <name>sentry.store.jdbc.url</name>
- <value>jdbc:postgresql://${INTERNAL_LISTEN_HOST}:5432/${SENTRY_POLICY_DB}/;create=true</value>
- </property>
- <property>
- <name>sentry.store.jdbc.user</name>
- <value>hiveuser</value>
- </property>
- <property>
- <name>sentry.store.jdbc.password</name>
- <value>password</value>
- </property>
- <property>
- <name>sentry.verify.schema.version</name>
- <value>false</value>
- </property>
- <property>
- <name>sentry.store.jdbc.driver</name>
- <value>org.postgresql.Driver</value>
- </property>
- <!-- This property enables owner privilege with grant. -->
- <property>
- <name>sentry.db.policy.store.owner.as.privilege</name>
- <value>all_with_grant</value>
- </property>
- <!-- These properties enable HMS follower. -->
- <property>
- <name>sentry.service.processor.factories</name>
- <value>${SENTRY_PROCESSOR_FACTORIES}</value>
- </property>
- <property>
- <name>sentry.policy.store.plugins</name>
- <value>org.apache.sentry.hdfs.SentryPlugin</value>
- </property>
- <property>
- <name>sentry.hive.testing.mode</name>
- <value>true</value>
- </property>
- <!-- Use a space to mean allow all privileges. See SENTRY-2424 -->
- <property>
- <name>sentry.db.explicit.grants.permitted</name>
- <value> </value>
- </property>
- <!-- Custom group mapping for custom cluster tests -->
- <property>
- <name>sentry.store.group.mapping</name>
- <value>org.apache.impala.service.CustomClusterGroupMapper</value>
- </property>
-</configuration>
diff --git a/fe/src/test/resources/sentry-site_oo_nogrant.xml.template b/fe/src/test/resources/sentry-site_oo_nogrant.xml.template
deleted file mode 100644
index 444eb9f..0000000
--- a/fe/src/test/resources/sentry-site_oo_nogrant.xml.template
+++ /dev/null
@@ -1,104 +0,0 @@
-<?xml version="1.0"?>
-<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
-<!--
- Licensed to the Apache Software Foundation (ASF) under one or more
- contributor license agreements. See the NOTICE file distributed with
- this work for additional information regarding copyright ownership.
- The ASF licenses this file to You under the Apache License, Version 2.0
- (the "License"); you may not use this file except in compliance with
- the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License.
--->
-
-<configuration>
- <property>
- <name>sentry.service.security.mode</name>
- <value>none</value>
- </property>
- <property>
- <name>hive.sentry.server</name>
- <value>server1</value>
- </property>
- <property>
- <name>sentry.service.admin.group</name>
- <value>${USER}</value>
- </property>
- <property>
- <name>sentry.service.server.rpc-address</name>
- <value>${INTERNAL_LISTEN_HOST}</value>
- </property>
- <property>
- <name>sentry.service.server.rpc-port</name>
- <value>30911</value>
- </property>
- <property>
- <name>sentry.service.client.server.rpc-address</name>
- <value>${INTERNAL_LISTEN_HOST}</value>
- </property>
- <property>
- <!--
- TODO(IMPALA-5686): Remove this or the previous property when Sentry standardizes on one.
- -->
- <name>sentry.service.client.server.rpc-addresses</name>
- <value>${INTERNAL_LISTEN_HOST}</value>
- </property>
- <property>
- <name>sentry.service.client.server.rpc-port</name>
- <value>30911</value>
- </property>
- <property>
- <name>sentry.store.jdbc.url</name>
- <value>jdbc:postgresql://${INTERNAL_LISTEN_HOST}:5432/${SENTRY_POLICY_DB}/;create=true</value>
- </property>
- <property>
- <name>sentry.store.jdbc.user</name>
- <value>hiveuser</value>
- </property>
- <property>
- <name>sentry.store.jdbc.password</name>
- <value>password</value>
- </property>
- <property>
- <name>sentry.verify.schema.version</name>
- <value>false</value>
- </property>
- <property>
- <name>sentry.store.jdbc.driver</name>
- <value>org.postgresql.Driver</value>
- </property>
- <!-- This property enables owner privilege without grant. -->
- <property>
- <name>sentry.db.policy.store.owner.as.privilege</name>
- <value>all</value>
- </property>
- <!-- These properties enable HMS follower. -->
- <property>
- <name>sentry.service.processor.factories</name>
- <value>${SENTRY_PROCESSOR_FACTORIES}</value>
- </property>
- <property>
- <name>sentry.policy.store.plugins</name>
- <value>org.apache.sentry.hdfs.SentryPlugin</value>
- </property>
- <property>
- <name>sentry.hive.testing.mode</name>
- <value>true</value>
- </property>
- <!-- Use a space to mean allow all privileges. See SENTRY-2424 -->
- <property>
- <name>sentry.db.explicit.grants.permitted</name>
- <value> </value>
- </property>
- <!-- Custom group mapping for custom cluster tests -->
- <property>
- <name>sentry.store.group.mapping</name>
- <value>org.apache.impala.service.CustomClusterGroupMapper</value>
- </property>
-</configuration>
[impala] 01/04: IMPALA-7640: RENAME on managed Kudu table should
rename Kudu table
Posted by ta...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
tarmstrong pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
commit 4057331fb8cb976ea11364c374183bab01d795da
Author: stakiar <ta...@gmail.com>
AuthorDate: Fri Jan 4 13:22:47 2019 -0600
IMPALA-7640: RENAME on managed Kudu table should rename Kudu table
`ALTER TABLE managed_kudu_tbl RENAME TO new_managed_kudu_tbl` will now
rename the underlying Kudu table from `managed_kudu_tbl` to
`new_managed_kudu_tbl`. This is only triggered for managed tables, for
external tables renames do not modify the underlying Kudu table.
Testing:
* Ran core tests
* Updated kudu_alter.test
Change-Id: I77e7583ce93cba8f6e743c4bedd9900ae1fae081
Reviewed-on: http://gerrit.cloudera.org:8080/12179
Reviewed-by: Thomas Marshall <tm...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
.../apache/impala/service/CatalogOpExecutor.java | 27 ++++++++++++++++++++++
.../queries/QueryTest/kudu_alter.test | 26 ++++++++++++++++++++-
2 files changed, 52 insertions(+), 1 deletion(-)
diff --git a/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java b/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
index e0c5fe9..0f27510 100644
--- a/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
+++ b/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
@@ -165,6 +165,7 @@ import org.apache.impala.thrift.TUpdateCatalogResponse;
import org.apache.impala.util.CompressionUtil;
import org.apache.impala.util.FunctionUtils;
import org.apache.impala.util.HdfsCachingUtil;
+import org.apache.impala.util.KuduUtil;
import org.apache.impala.util.MetaStoreUtil;
import org.slf4j.Logger;
import org.apache.thrift.TException;
@@ -2526,6 +2527,13 @@ public class CatalogOpExecutor {
oldTbl.getMetaStoreTable().deepCopy();
msTbl.setDbName(newTableName.getDb());
msTbl.setTableName(newTableName.getTbl());
+
+ // If oldTbl is a managed Kudu table, rename the underlying Kudu table
+ if (oldTbl instanceof KuduTable && !Table.isExternalTable(msTbl)) {
+ Preconditions.checkState(KuduTable.isKuduTable(msTbl));
+ renameKuduTable((KuduTable) oldTbl, msTbl, newTableName);
+ }
+
try (MetaStoreClient msClient = catalog_.getMetaStoreClient()) {
msClient.getHiveClient().alter_table(tableName.getDb(), tableName.getTbl(), msTbl);
} catch (TException e) {
@@ -2554,6 +2562,25 @@ public class CatalogOpExecutor {
}
/**
+ * Renames the underlying Kudu table for the given Impala table. If the new Kudu
+ * table name is the same as the old Kudu table name, this method does nothing.
+ */
+ private void renameKuduTable(KuduTable oldTbl,
+ org.apache.hadoop.hive.metastore.api.Table oldMsTbl, TableName newTableName)
+ throws ImpalaRuntimeException {
+ String newKuduTableName = KuduUtil.getDefaultCreateKuduTableName(
+ newTableName.getDb(), newTableName.getTbl());
+
+ // If the name of the Kudu table has not changed, do nothing
+ if (oldTbl.getKuduTableName().equals(newKuduTableName)) return;
+
+ KuduCatalogOpExecutor.renameTable(oldTbl, newKuduTableName);
+
+ // Add the name of the new Kudu table to the HMS table parameters
+ oldMsTbl.getParameters().put(KuduTable.KEY_TABLE_NAME, newKuduTableName);
+ }
+
+ /**
* Changes the file format for the given table or partitions. This is a metadata only
* operation, existing table data will not be converted to the new format. Returns
* true if the file metadata to be reloaded.
diff --git a/testdata/workloads/functional-query/queries/QueryTest/kudu_alter.test b/testdata/workloads/functional-query/queries/QueryTest/kudu_alter.test
index 0bed123..801bad8 100644
--- a/testdata/workloads/functional-query/queries/QueryTest/kudu_alter.test
+++ b/testdata/workloads/functional-query/queries/QueryTest/kudu_alter.test
@@ -381,8 +381,10 @@ BIGINT
====
---- QUERY
# Create an external Kudu table pointing to an existing Kudu table
+# The Kudu table kudu_tbl_to_alter is used in order to validate that renaming
+# the managed Impala-Kudu table tbl_to_alter renames the underlying Kudu table
create external table external_tbl stored as kudu
- tblproperties('kudu.table_name'='impala::$DATABASE.tbl_to_alter');
+ tblproperties('kudu.table_name'='impala::$DATABASE.kudu_tbl_to_alter');
select count(*) from external_tbl
---- RESULTS
5
@@ -390,6 +392,14 @@ select count(*) from external_tbl
BIGINT
====
---- QUERY
+# Ensure that after renaming the Impala table that the old Kudu table no longer
+# exists
+create external table external_tbl_on_nonexistent_kudu_tbl stored as kudu
+ tblproperties('kudu.table_name'='impala::$DATABASE.tbl_to_alter');
+---- CATCH
+ImpalaRuntimeException: Table does not exist in Kudu: 'impala::$DATABASE.tbl_to_alter'
+====
+---- QUERY
# Insert an item into the table pointed by the external Kudu table
insert into kudu_tbl_to_alter (id, name, new_col1, new_col2)
values (7, 'test', 4, 400)
@@ -610,3 +620,17 @@ alter table multi_range_partition_cols add range partition ('b', 2) <= values <
---- CATCH
NonRecoverableException: New range partition conflicts with existing range partition: ("b", 2) <= VALUES < ("c", 1)
====
+---- QUERY
+# Check that renaming an external Impala-Kudu table does not rename the underlying Kudu
+# table; this test consists of this QUERY block as well as the next one
+alter table external_tbl rename to external_tbl_new;
+---- RESULTS
+'Renaming was successful.'
+====
+---- QUERY
+describe formatted external_tbl_new;
+---- RESULTS: VERIFY_IS_SUBSET
+'','kudu.table_name ','impala::$DATABASE.temp_kudu_table'
+---- TYPES
+STRING,STRING,STRING
+----
[impala] 04/04: IMPALA-8382 Add support for SLES 12 SP3
Posted by ta...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
tarmstrong pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
commit da153104f26ef94f1b4db323f5629d678c55f9ee
Author: Hector Acosta <he...@cloudera.com>
AuthorDate: Thu Mar 28 16:58:58 2019 -0700
IMPALA-8382 Add support for SLES 12 SP3
Testing: Ran a build, reployed a cluster on sles 12 sp3.
Change-Id: Ia3cb1311b15226f1130be7e1d79110d16e3287ef
Reviewed-on: http://gerrit.cloudera.org:8080/12922
Tested-by: Impala Public Jenkins <im...@cloudera.com>
Reviewed-by: Tim Armstrong <ta...@cloudera.com>
---
bin/bootstrap_toolchain.py | 1 +
1 file changed, 1 insertion(+)
diff --git a/bin/bootstrap_toolchain.py b/bin/bootstrap_toolchain.py
index dcc835d..05a0a81 100755
--- a/bin/bootstrap_toolchain.py
+++ b/bin/bootstrap_toolchain.py
@@ -71,6 +71,7 @@ OS_MAPPING = [
OsMapping("suselinux11", "ec2-package-sles-11", None),
OsMapping("suselinux12", "ec2-package-sles-12", "sles12"),
OsMapping("suse12.2", "ec2-package-sles-12", "sles12"),
+ OsMapping("suse12.3", "ec2-package-sles-12", "sles12"),
OsMapping("ubuntu12.04", "ec2-package-ubuntu-12-04", None),
OsMapping("ubuntu14.04", "ec2-package-ubuntu-14-04", None),
OsMapping("ubuntu15.04", "ec2-package-ubuntu-14-04", None),
[impala] 03/04: IMPALA-8101: Thrift 11 and ext-data-source
compilation are always run
Posted by ta...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
tarmstrong pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
commit daa1bf9883e65adb82b11576b5ada4273bc9dd7f
Author: stakiar <ta...@gmail.com>
AuthorDate: Mon Jan 28 16:13:07 2019 -0600
IMPALA-8101: Thrift 11 and ext-data-source compilation are always run
Compilation of Thrift 11 Python code (IMPALA-7924) and of ext-data-source
Thrift files (ErrorCodes.thrift, ExternalDataSource.thrift, Data.thrift,
Status.thrift, Types.thrift) is run during every build, regardless of
whether or not the .thrift files have changed. The issue is that the
CMake custom command for compilation of these files points to a
non-existent OUTPUT_FILE.
This patch fixes Thrift 11 compilation by pointing the OUTPUT_FILE of
each .thrift file to its corresponding __init__.py file. For
compilation of ext-data-source, things are a bit tricky as we only run
Java gen and it is difficult to map Java generated code to the
corresponding .thrift files purely based on file names. Instead, for
ext-data-source, this patch adds a dummy file under
ext-data-source/api/target/tmp/generated-sources/ to track if a .thrift
file has been compiled or not. A `mvn clean` of ext-data-source will
delete all of these files and trigger re-compilation of the
ext-data-source files.
Change-Id: I52520e4b099c7bac5d088b4ba5d8a335495f727d
Reviewed-on: http://gerrit.cloudera.org:8080/12290
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
common/thrift/CMakeLists.txt | 22 ++++++++++++++++++----
1 file changed, 18 insertions(+), 4 deletions(-)
diff --git a/common/thrift/CMakeLists.txt b/common/thrift/CMakeLists.txt
index 4d8646a..dd7e1f3 100644
--- a/common/thrift/CMakeLists.txt
+++ b/common/thrift/CMakeLists.txt
@@ -107,12 +107,19 @@ function(THRIFT_GEN_DS VAR)
get_filename_component(ABS_THRIFT_FILE ${THRIFT_FILE} ABSOLUTE)
get_filename_component(THRIFT_FILE_WE ${THRIFT_FILE} NAME_WE)
- # This isn't the exact output file that's created, just a unique file
- set(OUTPUT_FILE "${EXT_DS_OUTPUT_DIR}/${THRIFT_FILE_WE}.java")
+ # Create a dummy marker file to track if Thrift compilation of the current file has
+ # been run or not. This file is created only if Thrift compilation succeeds. Deletion
+ # of the file will re-trigger Thrift compilation for the current file.
+ set(OUTPUT_FILE_MESSAGE "CMake state file for Thrift compilation of"
+ "${ABS_THRIFT_FILE}, deletion of this file will cause"
+ "${THRIFT_FILE} to be recompiled during the next build")
+ set(OUTPUT_DIR "${EXT_DS_OUTPUT_DIR}/../target/tmp/generated-sources/")
+ set(OUTPUT_FILE "${OUTPUT_DIR}/${THRIFT_FILE_WE}.txt")
list(APPEND ${VAR} ${OUTPUT_FILE})
add_custom_command(
OUTPUT ${OUTPUT_FILE}
- COMMAND ${THRIFT_COMPILER} ${JAVA_EXT_DS_ARGS} ${THRIFT_FILE}
+ COMMAND ${THRIFT_COMPILER} ${JAVA_EXT_DS_ARGS} ${THRIFT_FILE} &&
+ mkdir -p ${OUTPUT_DIR} && echo ${OUTPUT_FILE_MESSAGE} > ${OUTPUT_FILE}
DEPENDS ${ABS_THRIFT_FILE}
COMMENT "Running thrift compiler for ext-data-source on ${THRIFT_FILE}"
VERBATIM
@@ -141,7 +148,14 @@ function(THRIFT11_GEN VAR)
get_filename_component(THRIFT_FILE_WE ${THRIFT_FILE} NAME_WE)
# This isn't the exact output file that's created, just a unique file
- set(OUTPUT_FILE "${THRIFT11_PYTHON_OUTPUT_DIR}/${THRIFT_FILE_WE}.py")
+ set(OUTPUT_FILE "${THRIFT11_PYTHON_OUTPUT_DIR}/gen-py/${THRIFT_FILE_WE}/__init__.py")
+
+ # We have to special case beeswax.thrift because its Python namespace is beeswaxd
+ # instead of beeswax
+ if (${THRIFT_FILE_WE} STREQUAL "beeswax")
+ set(OUTPUT_FILE "${THRIFT11_PYTHON_OUTPUT_DIR}/gen-py/beeswaxd/__init__.py")
+ endif(${THRIFT_FILE_WE} STREQUAL "beeswax")
+
list(APPEND ${VAR} ${OUTPUT_FILE})
add_custom_command(
OUTPUT ${OUTPUT_FILE}