You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@impala.apache.org by jo...@apache.org on 2019/05/01 15:46:36 UTC
[impala] 02/04: IMPALA-8369 : Impala should be able to interoperate
with Hive 3.1.0
This is an automated email from the ASF dual-hosted git repository.
joemcdonnell pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
commit a89762bc014e97095c5108fd5785f51bf54a7a5d
Author: Vihang Karajgaonkar <vi...@cloudera.com>
AuthorDate: Mon Apr 1 17:47:51 2019 -0700
IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
This change adds a compatibility shim in fe so that Impala can
interoperate with Hive 3.1.0. It moves the existing Metastoreshim class
to a compat-hive-2 directory and adds a new Metastoreshim class under
compat-hive-3 directory. These shim classes implement method which are
different in hive-2 v/s hive-3 and are used by front end code. At the
build time, based on the environment variable
IMPALA_HIVE_MAJOR_VERSION one of the two shims is added to as source
using the fe/pom.xml build plugin.
Additionally, in order to reduce the dependencies footprint of Hive in
the front end code, this patch also introduces a new module called
shaded-deps. This module using shade plugin to include only the source
files from hive-exec which are need by the fe code. For hive-2 build
path, no changes are done with respect to hive dependencies to minimize
the risk of destabilizing the master branch on the default build option
of using Hive-2.
The different set of dependencies are activated using maven profiles.
The activation of each profile is automatic based on the
IMPALA_HIVE_MAJOR_VERSION.
Testing:
1. Code compiles and runs against both HMS-3 and HMS-2
2. Ran full-suite of tests using the private jenkins job against HMS-2
3. Running full-tests against HMS-3 will need more work like supporting
Tez in the mini-cluster (for dataloading) and HMS transaction support
since HMS3 create transactional tables by default. THis will be on-going
effort and test failures on Hive-3 will be fixed in additional
sub-tasks.
Notes:
1. Patch uses a custom build of Hive to be deployed in mini-cluster. This
build has the fixes for HIVE-21596. This hack will be removed when the
patches are available in official CDP Hive builds.
2. Some of the existing tests rely on the fact the UDFs implement the
UDF interface in Hive (UDFLength, UDFHour, UDFYear). These built-in hive
functions have been moved to use GenericUDF interface in Hive 3. Impala
currently only supports UDFExecutor. In order to have a full
compatibility with all the functions in Hive 2.x we should support
GenericUDFs too. That would be taken up as a separate patch.
3. Sentry dependencies bring a lot of transitive hive dependencies. The
patch excludes such dependencies since they create problems while
building against Hive-3. Since these hive-2 dependencies are
already included when building against hive-2 this should not be a problem.
Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436
Reviewed-on: http://gerrit.cloudera.org:8080/13005
Reviewed-by: Joe McDonnell <jo...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
CMakeLists.txt | 1 +
README.md | 1 -
bin/bootstrap_toolchain.py | 4 +-
bin/impala-config.sh | 82 +--
common/thrift/.gitignore | 1 +
common/thrift/CMakeLists.txt | 9 +-
fe/CMakeLists.txt | 2 +-
fe/pom.xml | 436 +++++++++------
.../org/apache/impala/compat/MetastoreShim.java | 102 +++-
.../impala/compat/HiveMetadataFormatUtils.java | 612 +++++++++++++++++++++
.../org/apache/impala/compat/MetastoreShim.java | 367 ++++++++++++
.../org/apache/impala/analysis/StringLiteral.java | 4 +-
.../org/apache/impala/catalog/FeHBaseTable.java | 29 +-
.../org/apache/impala/catalog/TableLoader.java | 7 +-
.../impala/catalog/events/MetastoreEvents.java | 33 +-
.../catalog/events/MetastoreEventsProcessor.java | 18 +-
.../impala/catalog/local/DirectMetaProvider.java | 4 +-
.../apache/impala/hive/executor/UdfExecutor.java | 2 +
.../apache/impala/service/CatalogOpExecutor.java | 2 +-
.../impala/service/DescribeResultFactory.java | 8 +-
.../java/org/apache/impala/service/MetadataOp.java | 32 +-
.../events/MetastoreEventsProcessorTest.java | 105 ++--
.../impala/hive/executor/UdfExecutorTest.java | 3 +-
.../testutil/EmbeddedMetastoreClientPool.java | 24 +-
impala-parent/pom.xml | 9 +
shaded-deps/.gitignore | 1 +
shaded-deps/CMakeLists.txt | 20 +
shaded-deps/pom.xml | 108 ++++
testdata/bin/run-hive-server.sh | 5 +-
tests/custom_cluster/test_permanent_udfs.py | 6 +
30 files changed, 1694 insertions(+), 343 deletions(-)
diff --git a/CMakeLists.txt b/CMakeLists.txt
index 285e617..ebccb69 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -401,6 +401,7 @@ add_subdirectory(common/yarn-extras)
add_subdirectory(common/protobuf)
add_subdirectory(be)
add_subdirectory(docker)
+add_subdirectory(shaded-deps)
add_subdirectory(fe)
add_subdirectory(impala-parent)
add_subdirectory(ext-data-source)
diff --git a/README.md b/README.md
index 1ea3789..d2df5f1 100644
--- a/README.md
+++ b/README.md
@@ -100,7 +100,6 @@ can do so through the environment variables and scripts listed below.
| HADOOP_INCLUDE_DIR | "${HADOOP_HOME}/include" | For 'hdfs.h' |
| HADOOP_LIB_DIR | "${HADOOP_HOME}/lib" | For 'libhdfs.a' or 'libhdfs.so' |
| HIVE_HOME | "${CDH_COMPONENTS_HOME}/{hive-${IMPALA_HIVE_VERSION}/" | |
-| HIVE_SRC_DIR | "${HIVE_HOME}/src" | Used to find Hive thrift files. |
| HBASE_HOME | "${CDH_COMPONENTS_HOME}/hbase-${IMPALA_HBASE_VERSION}/" | |
| SENTRY_HOME | "${CDH_COMPONENTS_HOME}/sentry-${IMPALA_SENTRY_VERSION}/" | Used to setup test data |
| THRIFT_HOME | "${IMPALA_TOOLCHAIN}/thrift-${IMPALA_THRIFT_VERSION}" | |
diff --git a/bin/bootstrap_toolchain.py b/bin/bootstrap_toolchain.py
index d51b5cf..9bfd6c8 100755
--- a/bin/bootstrap_toolchain.py
+++ b/bin/bootstrap_toolchain.py
@@ -570,8 +570,10 @@ if __name__ == "__main__":
]
use_cdp_hive = os.getenv("USE_CDP_HIVE") == "true"
if use_cdp_hive:
+ cdp_components.append(CdpComponent("apache-hive-{0}-src"
+ .format(os.environ.get("IMPALA_HIVE_VERSION")))),
cdp_components.append(CdpComponent("apache-hive-{0}-bin"
- .format(os.environ.get("IMPALA_HIVE_VERSION"))))
+ .format(os.environ.get("IMPALA_HIVE_VERSION")))),
cdp_components.append(CdpComponent(
"tez-{0}-minimal".format(os.environ.get("IMPALA_TEZ_VERSION")),
makedir=True))
diff --git a/bin/impala-config.sh b/bin/impala-config.sh
index bcd1a83..ff811df 100755
--- a/bin/impala-config.sh
+++ b/bin/impala-config.sh
@@ -171,22 +171,11 @@ export IMPALA_AVRO_JAVA_VERSION=1.8.2-cdh6.x-SNAPSHOT
export IMPALA_LLAMA_MINIKDC_VERSION=1.0.0
export IMPALA_KITE_VERSION=1.0.0-cdh6.x-SNAPSHOT
export KUDU_JAVA_VERSION=1.10.0-cdh6.x-SNAPSHOT
-export USE_CDP_HIVE=${USE_CDP_HIVE-false}
-if $USE_CDP_HIVE; then
- export IMPALA_HIVE_VERSION=3.1.0.6.0.99.0-45
- # Temporary version of Tez, patched with the fix for TEZ-1348:
- # https://github.com/apache/tez/pull/40
- # We'll switch to a non-"todd" version of Tez once that fix is integrated.
- # For now, if you're bumping the CDP build number, you'll need to download
- # this tarball from an earlier build and re-upload it to the new directory
- # in the toolchain bucket.
- #
- # TODO(todd) switch to an official build.
- export IMPALA_TEZ_VERSION=0.10.0-todd-6fcc41e5798b.1
- export TEZ_HOME="$CDP_COMPONENTS_HOME/tez-${IMPALA_TEZ_VERSION}-minimal"
-else
- export IMPALA_HIVE_VERSION=2.1.1-cdh6.x-SNAPSHOT
-fi
+export CDH_HIVE_VERSION=2.1.1-cdh6.x-SNAPSHOT
+# This is a custom build of Hive which includes patches for HIVE-21586
+# HIVE-21077, HIVE-21526, HIVE-21561
+# TODO Use a official once these patches are merged
+export CDP_HIVE_VERSION=3.1.0.6.0.99.0-38-0e7f6337a50
# When IMPALA_(CDH_COMPONENT)_URL are overridden, they may contain '$(platform_label)'
# which will be substituted for the CDH platform label in bootstrap_toolchain.py
@@ -206,6 +195,33 @@ if [ -f "$IMPALA_HOME/bin/impala-config-local.sh" ]; then
. "$IMPALA_HOME/bin/impala-config-local.sh"
fi
+export CDH_COMPONENTS_HOME="$IMPALA_TOOLCHAIN/cdh_components-$CDH_BUILD_NUMBER"
+export CDP_COMPONENTS_HOME="$IMPALA_TOOLCHAIN/cdp_components-$CDP_BUILD_NUMBER"
+export USE_CDP_HIVE=${USE_CDP_HIVE-false}
+if $USE_CDP_HIVE; then
+ # When USE_CDP_HIVE is set we use the CDP hive version to build as well as deploy in
+ # the minicluster
+ export IMPALA_HIVE_VERSION=${CDP_HIVE_VERSION}
+ # Temporary version of Tez, patched with the fix for TEZ-1348:
+ # https://github.com/apache/tez/pull/40
+ # We'll switch to a non-"todd" version of Tez once that fix is integrated.
+ # For now, if you're bumping the CDP build number, you'll need to download
+ # this tarball from an earlier build and re-upload it to the new directory
+ # in the toolchain bucket.
+ #
+ # TODO(todd) switch to an official build.
+ export IMPALA_TEZ_VERSION=0.10.0-todd-6fcc41e5798b.1
+else
+ # CDH hive version is used to build and deploy in minicluster when USE_CDP_HIVE is
+ # false
+ export IMPALA_HIVE_VERSION=${CDH_HIVE_VERSION}
+fi
+# Extract the first component of the hive version.
+# Allow overriding of Hive source location in case we want to build Impala without
+# a complete Hive build. This is used by fe/pom.xml to activate compatibility shims
+# for Hive-2 or Hive-3
+export IMPALA_HIVE_MAJOR_VERSION=$(echo "$IMPALA_HIVE_VERSION" | cut -d . -f 1)
+
# It is important to have a coherent view of the JAVA_HOME and JAVA executable.
# The JAVA_HOME should be determined first, then the JAVA executable should be
# derived from JAVA_HOME. bin/bootstrap_development.sh adds code to
@@ -291,16 +307,29 @@ export EXTERNAL_LISTEN_HOST="${EXTERNAL_LISTEN_HOST-0.0.0.0}"
export DEFAULT_FS="${DEFAULT_FS-hdfs://${INTERNAL_LISTEN_HOST}:20500}"
export WAREHOUSE_LOCATION_PREFIX="${WAREHOUSE_LOCATION_PREFIX-}"
export LOCAL_FS="file:${WAREHOUSE_LOCATION_PREFIX}"
+
ESCAPED_IMPALA_HOME=$(sed "s/[^0-9a-zA-Z]/_/g" <<< "$IMPALA_HOME")
if $USE_CDP_HIVE; then
- # It is likely that devs will want to with both the versions of metastore
+ export HIVE_HOME="$CDP_COMPONENTS_HOME/apache-hive-${IMPALA_HIVE_VERSION}-bin"
+ export HIVE_SRC_DIR=${HIVE_SRC_DIR_OVERRIDE:-"${CDP_COMPONENTS_HOME}/apache-hive-\
+${IMPALA_HIVE_VERSION}-src"}
+ # Set the path to the hive_metastore.thrift which is used to build thrift code
+ export HIVE_METASTORE_THRIFT_DIR=$HIVE_SRC_DIR/standalone-metastore/src/main/thrift
+ # It is likely that devs will want to work with both the versions of metastore
# if cdp hive is being used change the metastore db name, so we don't have to
# format the metastore db everytime we switch between hive versions
export METASTORE_DB=${METASTORE_DB-"$(cut -c-59 <<< HMS$ESCAPED_IMPALA_HOME)_cdp"}
+ export TEZ_HOME="$CDP_COMPONENTS_HOME/tez-${IMPALA_TEZ_VERSION}-minimal"
else
+ export HIVE_HOME="$CDH_COMPONENTS_HOME/hive-${IMPALA_HIVE_VERSION}"
+ # Allow overriding of Hive source location in case we want to build Impala without
+# a complete Hive build.
+ export HIVE_SRC_DIR=${HIVE_SRC_DIR_OVERRIDE:-"${HIVE_HOME}/src"}
+ export HIVE_METASTORE_THRIFT_DIR=$HIVE_SRC_DIR/metastore/if
export METASTORE_DB=${METASTORE_DB-$(cut -c-63 <<< HMS$ESCAPED_IMPALA_HOME)}
fi
-
+# Set the Hive binaries in the path
+export PATH="$HIVE_HOME/bin:$PATH"
export SENTRY_POLICY_DB=${SENTRY_POLICY_DB-$(cut -c-63 <<< SP$ESCAPED_IMPALA_HOME)}
if [[ "${TARGET_FILESYSTEM}" == "s3" ]]; then
@@ -492,12 +521,6 @@ export IMPALA_COMMON_DIR="$IMPALA_HOME/common"
export PATH="$IMPALA_TOOLCHAIN/gdb-$IMPALA_GDB_VERSION/bin:$PATH"
export PATH="$IMPALA_HOME/bin:$IMPALA_TOOLCHAIN/cmake-$IMPALA_CMAKE_VERSION/bin/:$PATH"
-# The directory in which all the thirdparty CDH components live.
-export CDH_COMPONENTS_HOME="$IMPALA_TOOLCHAIN/cdh_components-$CDH_BUILD_NUMBER"
-
-# The directory in which all the thirdparty CDP components live.
-export CDP_COMPONENTS_HOME="$IMPALA_TOOLCHAIN/cdp_components-$CDP_BUILD_NUMBER"
-
# Typically we build against a snapshot build of Hadoop that includes everything we need
# for building Impala and running a minicluster.
export HADOOP_HOME="$CDH_COMPONENTS_HOME/hadoop-${IMPALA_HADOOP_VERSION}/"
@@ -531,17 +554,6 @@ export SENTRY_CONF_DIR="$IMPALA_HOME/fe/src/test/resources"
export RANGER_HOME="${CDP_COMPONENTS_HOME}/ranger-${IMPALA_RANGER_VERSION}-admin"
export RANGER_CONF_DIR="$IMPALA_HOME/fe/src/test/resources"
-# Extract the first component of the hive version.
-export IMPALA_HIVE_MAJOR_VERSION=$(echo "$IMPALA_HIVE_VERSION" | cut -d . -f 1)
-if $USE_CDP_HIVE; then
- export HIVE_HOME="$CDP_COMPONENTS_HOME/apache-hive-${IMPALA_HIVE_VERSION}-bin"
-else
- export HIVE_HOME="$CDH_COMPONENTS_HOME/hive-${IMPALA_HIVE_VERSION}/"
-fi
-export PATH="$HIVE_HOME/bin:$PATH"
-# Allow overriding of Hive source location in case we want to build Impala without
-# a complete Hive build.
-export HIVE_SRC_DIR=${HIVE_SRC_DIR_OVERRIDE:-"${HIVE_HOME}/src"}
# To configure Hive logging, there's a hive-log4j2.properties[.template]
# file in fe/src/test/resources. To get it into the classpath earlier
# than the hive-log4j2.properties file included in some Hive jars,
diff --git a/common/thrift/.gitignore b/common/thrift/.gitignore
index f0ee7f2..0f6c0c5 100644
--- a/common/thrift/.gitignore
+++ b/common/thrift/.gitignore
@@ -2,3 +2,4 @@ Opcodes.thrift
ErrorCodes.thrift
MetricDefs.thrift
hive-2-api/TCLIService.thrift
+hive-3-api/TCLIService.thrift
diff --git a/common/thrift/CMakeLists.txt b/common/thrift/CMakeLists.txt
index dd7e1f3..f297292 100644
--- a/common/thrift/CMakeLists.txt
+++ b/common/thrift/CMakeLists.txt
@@ -172,7 +172,7 @@ set(HIVE_THRIFT_SOURCE_DIR "hive-$ENV{IMPALA_HIVE_MAJOR_VERSION}-api")
set(TCLI_SERVICE_THRIFT "${HIVE_THRIFT_SOURCE_DIR}/TCLIService.thrift")
message("Using Thrift compiler: ${THRIFT_COMPILER}")
message("Using Thrift 11 compiler: ${THRIFT11_COMPILER}")
-set(THRIFT_INCLUDE_DIR_OPTION -I ${THRIFT_CONTRIB_DIR} -I $ENV{HIVE_SRC_DIR}/metastore/if
+set(THRIFT_INCLUDE_DIR_OPTION -I ${THRIFT_CONTRIB_DIR} -I $ENV{HIVE_METASTORE_THRIFT_DIR}
-I ${HIVE_THRIFT_SOURCE_DIR})
set(BE_OUTPUT_DIR ${CMAKE_SOURCE_DIR}/be/generated-sources)
set(FE_OUTPUT_DIR ${CMAKE_SOURCE_DIR}/fe/generated-sources)
@@ -258,6 +258,13 @@ add_custom_command(OUTPUT hive-2-api/TCLIService.thrift
DEPENDS hive-1-api/TCLIService.thrift
)
+add_custom_command(OUTPUT hive-3-api/TCLIService.thrift
+ COMMAND sed
+ 's/namespace java org.apache.hive.service.cli.thrift/namespace java org.apache.hive.service.rpc.thrift/'
+ hive-1-api/TCLIService.thrift > hive-3-api/TCLIService.thrift
+ DEPENDS hive-1-api/TCLIService.thrift
+)
+
# Create a build command for each of the thrift src files and generate
# a list of files they produce
THRIFT_GEN(THRIFT_ALL_FILES ${SRC_FILES})
diff --git a/fe/CMakeLists.txt b/fe/CMakeLists.txt
index c6af7db..5453387 100644
--- a/fe/CMakeLists.txt
+++ b/fe/CMakeLists.txt
@@ -16,7 +16,7 @@
# under the License.
add_custom_target(fe ALL DEPENDS
- thrift-deps fb-deps yarn-extras function-registry ext-data-source impala-parent
+ shaded-deps thrift-deps fb-deps yarn-extras function-registry ext-data-source impala-parent
COMMAND ${CMAKE_SOURCE_DIR}/bin/mvn-quiet.sh -B install -DskipTests
)
diff --git a/fe/pom.xml b/fe/pom.xml
index 920b7e9..2a20e58 100644
--- a/fe/pom.xml
+++ b/fe/pom.xml
@@ -176,6 +176,14 @@ under the License.
<groupId>net.minidev</groupId>
<artifactId>json-smart</artifactId>
</exclusion>
+ <exclusion>
+ <groupId>org.apache.hive.hcatalog</groupId>
+ <artifactId>*</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.apache.hive</groupId>
+ <artifactId>*</artifactId>
+ </exclusion>
</exclusions>
</dependency>
@@ -214,6 +222,10 @@ under the License.
<groupId>net.minidev</groupId>
<artifactId>json-smart</artifactId>
</exclusion>
+ <exclusion>
+ <groupId>org.apache.hive.hcatalog</groupId>
+ <artifactId>*</artifactId>
+ </exclusion>
</exclusions>
</dependency>
@@ -233,6 +245,10 @@ under the License.
<groupId>net.minidev</groupId>
<artifactId>json-smart</artifactId>
</exclusion>
+ <exclusion>
+ <groupId>org.apache.hive</groupId>
+ <artifactId>*</artifactId>
+ </exclusion>
</exclusions>
</dependency>
@@ -286,9 +302,6 @@ under the License.
<version>0.11-a-czt02-cdh</version>
</dependency>
- <!-- Moved above Hive, because Hive bundles its own Thrift version
- which supercedes this one if it comes first in the dependency
- tree -->
<dependency>
<groupId>org.apache.thrift</groupId>
<artifactId>libthrift</artifactId>
@@ -296,165 +309,6 @@ under the License.
</dependency>
<dependency>
- <groupId>org.apache.hive</groupId>
- <artifactId>hive-service</artifactId>
- <version>${hive.version}</version>
- <!--
- hive-service pulls in an incoherent (not ${hadoop.version}) version of
- hadoop-client via this dependency, which we don't need.
- -->
- <exclusions>
- <exclusion>
- <groupId>org.apache.hive</groupId>
- <artifactId>hive-llap-server</artifactId>
- </exclusion>
- <!-- https://issues.apache.org/jira/browse/HADOOP-14903 -->
- <exclusion>
- <groupId>net.minidev</groupId>
- <artifactId>json-smart</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.apache.calcite.avatica</groupId>
- <artifactId>avatica</artifactId>
- </exclusion>
- </exclusions>
- </dependency>
-
- <dependency>
- <groupId>org.apache.hive</groupId>
- <artifactId>hive-serde</artifactId>
- <version>${hive.version}</version>
- <exclusions>
- <!-- https://issues.apache.org/jira/browse/HADOOP-14903 -->
- <exclusion>
- <groupId>net.minidev</groupId>
- <artifactId>json-smart</artifactId>
- </exclusion>
- </exclusions>
- </dependency>
-
- <dependency>
- <groupId>org.apache.hive</groupId>
- <artifactId>hive-exec</artifactId>
- <version>${hive.version}</version>
- <exclusions>
- <!-- Impala uses log4j v1; avoid pulling in slf4j handling for log4j2 -->
- <exclusion>
- <groupId>org.apache.logging.log4j</groupId>
- <artifactId>log4j-slf4j-impl</artifactId>
- </exclusion>
- <!-- Similarly, avoid pulling in the "Log4j 1.2 Bridge" -->
- <exclusion>
- <groupId>org.apache.logging.log4j</groupId>
- <artifactId>log4j-1.2-api</artifactId>
- </exclusion>
- <!-- https://issues.apache.org/jira/browse/HADOOP-14903 -->
- <exclusion>
- <groupId>net.minidev</groupId>
- <artifactId>json-smart</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.apache.calcite.avatica</groupId>
- <artifactId>avatica</artifactId>
- </exclusion>
- </exclusions>
- </dependency>
-
- <dependency>
- <groupId>org.apache.hive</groupId>
- <artifactId>hive-common</artifactId>
- <version>${hive.version}</version>
- <exclusions>
- <!-- Impala uses log4j v1; avoid pulling in slf4j handling for log4j2 -->
- <exclusion>
- <groupId>org.apache.logging.log4j</groupId>
- <artifactId>log4j-slf4j-impl</artifactId>
- </exclusion>
- <!-- Similarly, avoid pulling in the "Log4j 1.2 Bridge" -->
- <exclusion>
- <groupId>org.apache.logging.log4j</groupId>
- <artifactId>log4j-1.2-api</artifactId>
- </exclusion>
- <!-- https://issues.apache.org/jira/browse/HADOOP-14903 -->
- <exclusion>
- <groupId>net.minidev</groupId>
- <artifactId>json-smart</artifactId>
- </exclusion>
- </exclusions>
- </dependency>
-
- <dependency>
- <groupId>org.apache.hive</groupId>
- <artifactId>hive-jdbc</artifactId>
- <version>${hive.version}</version>
- <scope>test</scope>
- <exclusions>
- <!-- Impala uses log4j v1; avoid pulling in slf4j handling for log4j2 -->
- <exclusion>
- <groupId>org.apache.logging.log4j</groupId>
- <artifactId>log4j-slf4j-impl</artifactId>
- </exclusion>
- <exclusion>
- <groupId>net.minidev</groupId>
- <artifactId>json-smart</artifactId>
- </exclusion>
- </exclusions>
- </dependency>
-
- <dependency>
- <groupId>org.apache.hive</groupId>
- <artifactId>hive-hbase-handler</artifactId>
- <version>${hive.version}</version>
- <exclusions>
- <!-- Impala uses log4j v1; avoid pulling in slf4j handling for log4j2 -->
- <exclusion>
- <groupId>org.apache.logging.log4j</groupId>
- <artifactId>log4j-slf4j-impl</artifactId>
- </exclusion>
- <!-- https://issues.apache.org/jira/browse/HADOOP-14903 -->
- <exclusion>
- <groupId>net.minidev</groupId>
- <artifactId>json-smart</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.apache.calcite.avatica</groupId>
- <artifactId>avatica</artifactId>
- </exclusion>
- </exclusions>
- </dependency>
-
- <dependency>
- <groupId>org.apache.hive</groupId>
- <artifactId>hive-metastore</artifactId>
- <version>${hive.version}</version>
- <exclusions>
- <!-- Impala uses log4j v1; avoid pulling in slf4j handling for log4j2 -->
- <exclusion>
- <groupId>org.apache.logging.log4j</groupId>
- <artifactId>log4j-slf4j-impl</artifactId>
- </exclusion>
- <!-- https://issues.apache.org/jira/browse/HADOOP-14903 -->
- <exclusion>
- <groupId>net.minidev</groupId>
- <artifactId>json-smart</artifactId>
- </exclusion>
- </exclusions>
- </dependency>
-
- <dependency>
- <groupId>org.apache.hive.shims</groupId>
- <artifactId>hive-shims-common</artifactId>
- <version>${hive.version}</version>
- <exclusions>
- <!-- Impala uses log4j v1; avoid pulling in slf4j handling for log4j2 -->
- <exclusion>
- <groupId>org.apache.logging.log4j</groupId>
- <artifactId>log4j-slf4j-impl</artifactId>
- </exclusion>
- </exclusions>
- </dependency>
-
- <dependency>
<groupId>org.apache.kudu</groupId>
<artifactId>kudu-client</artifactId>
<version>${kudu.version}</version>
@@ -550,7 +404,6 @@ under the License.
<version>2.23.4</version>
<scope>test</scope>
</dependency>
-
</dependencies>
<reporting>
@@ -694,7 +547,8 @@ under the License.
-->
<source>${project.basedir}/generated-sources/gen-java</source>
<source>${project.build.directory}/generated-sources/cup</source>
- </sources>
+ <source>${project.basedir}/src/compat-hive-${hive.major.version}/java</source>
+ </sources>
</configuration>
</execution>
</executions>
@@ -788,6 +642,7 @@ under the License.
<include>org.apache.hadoop:*:${hadoop.version}</include>
<include>org.apache.hbase:*:${hbase.version}</include>
<include>org.apache.hive:*:${hive.version}</include>
+ <include>org.apache.hive:hive-storage-api:${hive.storage.api.version}</include>
<include>org.apache.kudu:*:${kudu.version}</include>
<include>org.apache.sentry:*:${sentry.version}</include>
<include>org.apache.parquet:*:${parquet.version}</include>
@@ -799,8 +654,8 @@ under the License.
</execution>
</executions>
</plugin>
+ </plugins>
- </plugins>
<pluginManagement>
<plugins>
<!--This plugin's configuration is used to store Eclipse m2e settings only. It has no influence on the Maven build itself.-->
@@ -898,6 +753,257 @@ under the License.
</build>
<profiles>
+ <!-- Profile which includes all of the Hive 2.x dependencies and appropriate exclusions. -->
+ <profile>
+ <id>hive-2</id>
+ <activation>
+ <property>
+ <name>env.IMPALA_HIVE_MAJOR_VERSION</name>
+ <value>2</value>
+ </property>
+ </activation>
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.hive</groupId>
+ <artifactId>hive-service</artifactId>
+ <version>${hive.version}</version>
+ <!--
+ hive-service pulls in an incoherent (not ${hadoop.version}) version of
+ hadoop-client via this dependency, which we don't need.
+ -->
+ <exclusions>
+ <exclusion>
+ <groupId>org.apache.hive</groupId>
+ <artifactId>hive-llap-server</artifactId>
+ </exclusion>
+ <!-- https://issues.apache.org/jira/browse/HADOOP-14903 -->
+ <exclusion>
+ <groupId>net.minidev</groupId>
+ <artifactId>json-smart</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.apache.calcite.avatica</groupId>
+ <artifactId>avatica</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hive</groupId>
+ <artifactId>hive-serde</artifactId>
+ <version>${hive.version}</version>
+ <exclusions>
+ <!-- https://issues.apache.org/jira/browse/HADOOP-14903 -->
+ <exclusion>
+ <groupId>net.minidev</groupId>
+ <artifactId>json-smart</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hive</groupId>
+ <artifactId>hive-exec</artifactId>
+ <version>${hive.version}</version>
+ <exclusions>
+ <!-- Impala uses log4j v1; avoid pulling in slf4j handling for log4j2 -->
+ <exclusion>
+ <groupId>org.apache.logging.log4j</groupId>
+ <artifactId>log4j-slf4j-impl</artifactId>
+ </exclusion>
+ <!-- Similarly, avoid pulling in the "Log4j 1.2 Bridge" -->
+ <exclusion>
+ <groupId>org.apache.logging.log4j</groupId>
+ <artifactId>log4j-1.2-api</artifactId>
+ </exclusion>
+ <!-- https://issues.apache.org/jira/browse/HADOOP-14903 -->
+ <exclusion>
+ <groupId>net.minidev</groupId>
+ <artifactId>json-smart</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.apache.calcite.avatica</groupId>
+ <artifactId>avatica</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hive</groupId>
+ <artifactId>hive-common</artifactId>
+ <version>${hive.version}</version>
+ <exclusions>
+ <!-- Impala uses log4j v1; avoid pulling in slf4j handling for log4j2 -->
+ <exclusion>
+ <groupId>org.apache.logging.log4j</groupId>
+ <artifactId>log4j-slf4j-impl</artifactId>
+ </exclusion>
+ <!-- Similarly, avoid pulling in the "Log4j 1.2 Bridge" -->
+ <exclusion>
+ <groupId>org.apache.logging.log4j</groupId>
+ <artifactId>log4j-1.2-api</artifactId>
+ </exclusion>
+ <!-- https://issues.apache.org/jira/browse/HADOOP-14903 -->
+ <exclusion>
+ <groupId>net.minidev</groupId>
+ <artifactId>json-smart</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hive</groupId>
+ <artifactId>hive-jdbc</artifactId>
+ <version>${hive.version}</version>
+ <scope>test</scope>
+ <exclusions>
+ <!-- Impala uses log4j v1; avoid pulling in slf4j handling for log4j2 -->
+ <exclusion>
+ <groupId>org.apache.logging.log4j</groupId>
+ <artifactId>log4j-slf4j-impl</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>net.minidev</groupId>
+ <artifactId>json-smart</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hive</groupId>
+ <artifactId>hive-hbase-handler</artifactId>
+ <version>${hive.version}</version>
+ <exclusions>
+ <!-- Impala uses log4j v1; avoid pulling in slf4j handling for log4j2 -->
+ <exclusion>
+ <groupId>org.apache.logging.log4j</groupId>
+ <artifactId>log4j-slf4j-impl</artifactId>
+ </exclusion>
+ <!-- https://issues.apache.org/jira/browse/HADOOP-14903 -->
+ <exclusion>
+ <groupId>net.minidev</groupId>
+ <artifactId>json-smart</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.apache.calcite.avatica</groupId>
+ <artifactId>avatica</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hive</groupId>
+ <artifactId>hive-metastore</artifactId>
+ <version>${hive.version}</version>
+ <exclusions>
+ <!-- Impala uses log4j v1; avoid pulling in slf4j handling for log4j2 -->
+ <exclusion>
+ <groupId>org.apache.logging.log4j</groupId>
+ <artifactId>log4j-slf4j-impl</artifactId>
+ </exclusion>
+ <!-- https://issues.apache.org/jira/browse/HADOOP-14903 -->
+ <exclusion>
+ <groupId>net.minidev</groupId>
+ <artifactId>json-smart</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hive.shims</groupId>
+ <artifactId>hive-shims-common</artifactId>
+ <version>${hive.version}</version>
+ <exclusions>
+ <!-- Impala uses log4j v1; avoid pulling in slf4j handling for log4j2 -->
+ <exclusion>
+ <groupId>org.apache.logging.log4j</groupId>
+ <artifactId>log4j-slf4j-impl</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+ </dependencies>
+ </profile>
+
+ <!-- Profile which includes all of the Hive 3.x dependencies and appropriate exclusions. -->
+ <profile>
+ <id>hive-3</id>
+ <activation>
+ <property>
+ <name>env.IMPALA_HIVE_MAJOR_VERSION</name>
+ <value>3</value>
+ </property>
+ </activation>
+ <dependencies>
+ <!-- This reduced dependency is derived from hive-exec and only contains classes needed
+ by Impala. See shaded-deps/pom.xml for more details -->
+ <dependency>
+ <groupId>org.apache.impala</groupId>
+ <artifactId>impala-minimal-hive-exec</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+ <!-- Needed for tests like JdbcTest which instantiates HiveDriver -->
+ <dependency>
+ <groupId>org.apache.hive</groupId>
+ <artifactId>hive-jdbc</artifactId>
+ <version>${hive.version}</version>
+ <scope>test</scope>
+ <exclusions>
+ <exclusion>
+ <groupId>org.apache.hbase</groupId>
+ <artifactId>*</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.apache.logging.log4j</groupId>
+ <artifactId>log4j-slf4j-impl</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.apache.logging.log4j</groupId>
+ <artifactId>log4j-1.2-api</artifactId>
+ </exclusion>
+ <!-- We should exclude hive-serde since it brings along a
+ different version of flatbuffers causing problems for loading tables -->
+ <exclusion>
+ <groupId>org.apache.hive</groupId>
+ <artifactId>hive-serde</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.apache.hive</groupId>
+ <artifactId>hive-metastore</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.hive</groupId>
+ <artifactId>hive-standalone-metastore</artifactId>
+ <version>${hive.version}</version>
+ <exclusions>
+ <!-- Impala uses log4j v1; avoid pulling in slf4j handling for log4j2 -->
+ <exclusion>
+ <groupId>org.apache.logging.log4j</groupId>
+ <artifactId>log4j-slf4j-impl</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.apache.logging.log4j</groupId>
+ <artifactId>log4j-1.2-api</artifactId>
+ </exclusion>
+ <!-- https://issues.apache.org/jira/browse/HADOOP-14903 -->
+ <exclusion>
+ <groupId>net.minidev</groupId>
+ <artifactId>json-smart</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.apache.hive</groupId>
+ <artifactId>hive-serde</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.apache.hive</groupId>
+ <artifactId>hive-shims</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+ </dependencies>
+ </profile>
+
<profile>
<id>thrift-home-defined</id>
<activation>
diff --git a/fe/src/main/java/org/apache/impala/compat/MetastoreShim.java b/fe/src/compat-hive-2/java/org/apache/impala/compat/MetastoreShim.java
similarity index 55%
rename from fe/src/main/java/org/apache/impala/compat/MetastoreShim.java
rename to fe/src/compat-hive-2/java/org/apache/impala/compat/MetastoreShim.java
index 3d69545..8ceb127 100644
--- a/fe/src/main/java/org/apache/impala/compat/MetastoreShim.java
+++ b/fe/src/compat-hive-2/java/org/apache/impala/compat/MetastoreShim.java
@@ -17,16 +17,33 @@
package org.apache.impala.compat;
+import static org.apache.impala.service.MetadataOp.TABLE_TYPE_TABLE;
+import static org.apache.impala.service.MetadataOp.TABLE_TYPE_VIEW;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableMap;
+import java.util.EnumSet;
import java.util.List;
+import org.apache.hadoop.hive.common.FileUtils;
import org.apache.hadoop.hive.common.StatsSetupConst;
+import org.apache.hadoop.hive.ql.metadata.formatting.MetaDataFormatUtils;
import org.apache.hadoop.hive.conf.HiveConf;
import org.apache.hadoop.hive.metastore.IMetaStoreClient;
import org.apache.hadoop.hive.metastore.MetaStoreUtils;
+import org.apache.hadoop.hive.metastore.TableType;
import org.apache.hadoop.hive.metastore.Warehouse;
import org.apache.hadoop.hive.metastore.api.InvalidOperationException;
import org.apache.hadoop.hive.metastore.api.MetaException;
import org.apache.hadoop.hive.metastore.api.Partition;
+import org.apache.hadoop.hive.metastore.api.Table;
+import org.apache.hadoop.hive.metastore.api.FieldSchema;
+import org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer;
+import org.apache.hadoop.hive.metastore.messaging.AlterTableMessage;
+import org.apache.hadoop.hive.metastore.messaging.InsertMessage;
+import org.apache.hadoop.hive.metastore.messaging.MessageDeserializer;
+import org.apache.hadoop.hive.metastore.messaging.json.ExtendedJSONMessageFactory;
import org.apache.hive.service.rpc.thrift.TGetColumnsReq;
import org.apache.hive.service.rpc.thrift.TGetFunctionsReq;
import org.apache.hive.service.rpc.thrift.TGetSchemasReq;
@@ -76,8 +93,8 @@ public class MetastoreShim {
* Wrapper around MetaStoreUtils.updatePartitionStatsFast() to deal with added
* arguments.
*/
- public static void updatePartitionStatsFast(Partition partition, Warehouse warehouse)
- throws MetaException {
+ public static void updatePartitionStatsFast(Partition partition, Table tbl,
+ Warehouse warehouse) throws MetaException {
MetaStoreUtils.updatePartitionStatsFast(partition, warehouse, null);
}
@@ -124,4 +141,85 @@ public class MetastoreShim {
return MetadataOp.getSchemas(
frontend, req.getCatalogName(), req.getSchemaName(), user);
}
+
+ /**
+ * Supported HMS-2 types
+ */
+ public static final EnumSet<TableType> IMPALA_SUPPORTED_TABLE_TYPES = EnumSet
+ .of(TableType.EXTERNAL_TABLE, TableType.MANAGED_TABLE, TableType.VIRTUAL_VIEW);
+
+ /**
+ * mapping between the HMS-2 type the Impala types
+ */
+ public static final ImmutableMap<String, String> HMS_TO_IMPALA_TYPE =
+ new ImmutableMap.Builder<String, String>()
+ .put("EXTERNAL_TABLE", TABLE_TYPE_TABLE)
+ .put("MANAGED_TABLE", TABLE_TYPE_TABLE)
+ .put("INDEX_TABLE", TABLE_TYPE_TABLE)
+ .put("VIRTUAL_VIEW", TABLE_TYPE_VIEW).build();
+
+ /**
+ * Wrapper method which returns ExtendedJSONMessageFactory in case Impala is
+ * building against Hive-2 to keep compatibility with Sentry
+ */
+ public static MessageDeserializer getMessageDeserializer() {
+ return ExtendedJSONMessageFactory.getInstance().getDeserializer();
+ }
+
+ /**
+ * Wrapper around FileUtils.makePartName to deal with package relocation in Hive 3
+ * @param partitionColNames
+ * @param values
+ * @return
+ */
+ public static String makePartName(List<String> partitionColNames, List<String> values) {
+ return FileUtils.makePartName(partitionColNames, values);
+ }
+
+ /**
+ * Wrapper method around message factory's build alter table message due to added
+ * arguments in hive 3.
+ */
+ @VisibleForTesting
+ public static AlterTableMessage buildAlterTableMessage(Table before, Table after,
+ boolean isTruncateOp, long writeId) {
+ Preconditions.checkArgument(writeId < 0, "Write ids are not supported in Hive-2 "
+ + "compatible build");
+ Preconditions.checkArgument(!isTruncateOp, "Truncate operation is not supported in "
+ + "alter table messages in Hive-2 compatible build");
+ return ExtendedJSONMessageFactory.getInstance().buildAlterTableMessage(before, after);
+ }
+
+ @VisibleForTesting
+ public static InsertMessage buildInsertMessage(Table msTbl, Partition partition,
+ boolean isInsertOverwrite, List<String> newFiles) {
+ return ExtendedJSONMessageFactory.getInstance().buildInsertMessage(msTbl, partition,
+ isInsertOverwrite, newFiles);
+ }
+
+ public static String getAllColumnsInformation(List<FieldSchema> tabCols,
+ List<FieldSchema> partitionCols, boolean printHeader, boolean isOutputPadded,
+ boolean showPartColsSeparately) {
+ return MetaDataFormatUtils
+ .getAllColumnsInformation(tabCols, partitionCols, printHeader, isOutputPadded,
+ showPartColsSeparately);
+ }
+
+ /**
+ * Wrapper method around Hive's MetadataFormatUtils.getTableInformation which has
+ * changed significantly in Hive-3
+ * @return
+ */
+ public static String getTableInformation(
+ org.apache.hadoop.hive.ql.metadata.Table table) {
+ return MetaDataFormatUtils.getTableInformation(table);
+ }
+
+ /**
+ * Wrapper method around BaseSemanticAnalyzer's unespaceSQLString to be compatibility
+ * with Hive. Takes in a normalized value of the string surrounded by single quotes
+ */
+ public static String unescapeSQLString(String normalizedStringLiteral) {
+ return BaseSemanticAnalyzer.unescapeSQLString(normalizedStringLiteral);
+ }
}
diff --git a/fe/src/compat-hive-3/java/org/apache/impala/compat/HiveMetadataFormatUtils.java b/fe/src/compat-hive-3/java/org/apache/impala/compat/HiveMetadataFormatUtils.java
new file mode 100644
index 0000000..073031e
--- /dev/null
+++ b/fe/src/compat-hive-3/java/org/apache/impala/compat/HiveMetadataFormatUtils.java
@@ -0,0 +1,612 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package org.apache.impala.compat;
+
+import java.math.BigInteger;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.Date;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Set;
+import java.util.TreeMap;
+import org.apache.commons.lang.StringEscapeUtils;
+import org.apache.commons.lang3.text.translate.CharSequenceTranslator;
+import org.apache.commons.lang3.text.translate.EntityArrays;
+import org.apache.commons.lang3.text.translate.LookupTranslator;
+import org.apache.hadoop.hive.common.type.HiveDecimal;
+import org.apache.hadoop.hive.metastore.TableType;
+import org.apache.hadoop.hive.metastore.api.BinaryColumnStatsData;
+import org.apache.hadoop.hive.metastore.api.BooleanColumnStatsData;
+import org.apache.hadoop.hive.metastore.api.ColumnStatisticsData;
+import org.apache.hadoop.hive.metastore.api.ColumnStatisticsObj;
+import org.apache.hadoop.hive.metastore.api.DateColumnStatsData;
+import org.apache.hadoop.hive.metastore.api.Decimal;
+import org.apache.hadoop.hive.metastore.api.DecimalColumnStatsData;
+import org.apache.hadoop.hive.metastore.api.DoubleColumnStatsData;
+import org.apache.hadoop.hive.metastore.api.FieldSchema;
+import org.apache.hadoop.hive.metastore.api.LongColumnStatsData;
+import org.apache.hadoop.hive.metastore.api.StorageDescriptor;
+import org.apache.hadoop.hive.metastore.api.StringColumnStatsData;
+import org.apache.hadoop.hive.metastore.api.Table;
+import org.apache.hadoop.hive.serde2.io.DateWritable;
+
+/**
+ * Most of the code in this class is copied from Hive 2.1.1. This is used so that
+ * the describe table output from Impala matches to Hive as much as possible. Initially,
+ * Impala had a dependency with hive-exec which pulled in this class directly from
+ * hive-exec jar. But in Hive 3 this code has diverged a lot and getting it from hive-exec
+ * pulls in a lot of unnecessary dependencies. It could be argued that
+ * supporting describe table to be similar to Hive's describe table does not make much
+ * sense. Since the code has diverged anyways when compared to Hive 3, we should maintain
+ * our own code from now on and make changes as required.
+ */
+public class HiveMetadataFormatUtils {
+
+ public static final String FIELD_DELIM = "\t";
+ public static final String LINE_DELIM = "\n";
+
+ static final int DEFAULT_STRINGBUILDER_SIZE = 2048;
+ private static final int ALIGNMENT = 20;
+
+ /**
+ * Write formatted information about the given columns, including partition columns to a
+ * string
+ *
+ * @param cols - list of columns
+ * @param partCols - list of partition columns
+ * @param printHeader - if header should be included
+ * @param isOutputPadded - make it more human readable by setting indentation with
+ * spaces. Turned off for use by HiveServer2
+ * @return string with formatted column information
+ */
+ public static String getAllColumnsInformation(List<FieldSchema> cols,
+ List<FieldSchema> partCols, boolean printHeader, boolean isOutputPadded,
+ boolean showPartColsSep) {
+ StringBuilder columnInformation = new StringBuilder(DEFAULT_STRINGBUILDER_SIZE);
+ if (printHeader) {
+ formatColumnsHeader(columnInformation, null);
+ }
+ formatAllFields(columnInformation, cols, isOutputPadded, null);
+
+ if ((partCols != null) && !partCols.isEmpty() && showPartColsSep) {
+ columnInformation.append(LINE_DELIM).append("# Partition Information")
+ .append(LINE_DELIM);
+ formatColumnsHeader(columnInformation, null);
+ formatAllFields(columnInformation, partCols, isOutputPadded, null);
+ }
+
+ return columnInformation.toString();
+ }
+
+ private static void formatColumnsHeader(StringBuilder columnInformation,
+ List<ColumnStatisticsObj> colStats) {
+ columnInformation.append("# "); // Easy for shell scripts to ignore
+ formatOutput(getColumnsHeader(colStats), columnInformation, false);
+ columnInformation.append(LINE_DELIM);
+ }
+
+ /**
+ * Prints a row with the given fields into the builder The last field could be a
+ * multiline field, and the extra lines should be padded
+ *
+ * @param fields The fields to print
+ * @param tableInfo The target builder
+ * @param isLastLinePadded Is the last field could be printed in multiple lines, if
+ * contains newlines?
+ */
+ private static void formatOutput(String[] fields, StringBuilder tableInfo,
+ boolean isLastLinePadded) {
+ int[] paddings = new int[fields.length - 1];
+ if (fields.length > 1) {
+ for (int i = 0; i < fields.length - 1; i++) {
+ if (fields[i] == null) {
+ tableInfo.append(FIELD_DELIM);
+ continue;
+ }
+ tableInfo.append(String.format("%-" + ALIGNMENT + "s", fields[i]))
+ .append(FIELD_DELIM);
+ paddings[i] = ALIGNMENT > fields[i].length() ? ALIGNMENT : fields[i].length();
+ }
+ }
+ if (fields.length > 0) {
+ String value = fields[fields.length - 1];
+ String unescapedValue = (isLastLinePadded && value != null) ? value
+ .replaceAll("\\\\n|\\\\r|\\\\r\\\\n", "\n") : value;
+ indentMultilineValue(unescapedValue, tableInfo, paddings, false);
+ } else {
+ tableInfo.append(LINE_DELIM);
+ }
+ }
+
+ private static final String schema = "col_name,data_type,comment#string:string:string";
+ private static final String colStatsSchema = "col_name,data_type,min,max,num_nulls,"
+ + "distinct_count,avg_col_len,max_col_len,num_trues,num_falses,comment"
+ + "#string:string:string:string:string:string:string:string:string:string:string";
+
+ public static String[] getColumnsHeader(List<ColumnStatisticsObj> colStats) {
+ String colSchema = schema;
+ if (colStats != null) {
+ colSchema = colStatsSchema;
+ }
+ return colSchema.split("#")[0].split(",");
+ }
+
+ /**
+ * Write formatted column information into given StringBuilder
+ *
+ * @param tableInfo - StringBuilder to append column information into
+ * @param cols - list of columns
+ * @param isOutputPadded - make it more human readable by setting indentation with
+ * spaces. Turned off for use by HiveServer2
+ */
+ private static void formatAllFields(StringBuilder tableInfo, List<FieldSchema> cols,
+ boolean isOutputPadded, List<ColumnStatisticsObj> colStats) {
+ for (FieldSchema col : cols) {
+ if (isOutputPadded) {
+ formatWithIndentation(col.getName(), col.getType(), getComment(col), tableInfo,
+ colStats);
+ } else {
+ formatWithoutIndentation(col.getName(), col.getType(), col.getComment(),
+ tableInfo, colStats);
+ }
+ }
+ }
+
+ private static void formatWithoutIndentation(String name, String type, String comment,
+ StringBuilder colBuffer, List<ColumnStatisticsObj> colStats) {
+ colBuffer.append(name);
+ colBuffer.append(FIELD_DELIM);
+ colBuffer.append(type);
+ colBuffer.append(FIELD_DELIM);
+ if (colStats != null) {
+ ColumnStatisticsObj cso = getColumnStatisticsObject(name, type, colStats);
+ if (cso != null) {
+ ColumnStatisticsData csd = cso.getStatsData();
+ if (csd.isSetBinaryStats()) {
+ BinaryColumnStatsData bcsd = csd.getBinaryStats();
+ appendColumnStatsNoFormatting(colBuffer, "", "", bcsd.getNumNulls(), "",
+ bcsd.getAvgColLen(), bcsd.getMaxColLen(), "", "");
+ } else if (csd.isSetStringStats()) {
+ StringColumnStatsData scsd = csd.getStringStats();
+ appendColumnStatsNoFormatting(colBuffer, "", "", scsd.getNumNulls(),
+ scsd.getNumDVs(), scsd.getAvgColLen(), scsd.getMaxColLen(), "", "");
+ } else if (csd.isSetBooleanStats()) {
+ BooleanColumnStatsData bcsd = csd.getBooleanStats();
+ appendColumnStatsNoFormatting(colBuffer, "", "", bcsd.getNumNulls(), "", "", "",
+ bcsd.getNumTrues(), bcsd.getNumFalses());
+ } else if (csd.isSetDecimalStats()) {
+ DecimalColumnStatsData dcsd = csd.getDecimalStats();
+ appendColumnStatsNoFormatting(colBuffer, convertToString(dcsd.getLowValue()),
+ convertToString(dcsd.getHighValue()), dcsd.getNumNulls(), dcsd.getNumDVs(),
+ "", "", "", "");
+ } else if (csd.isSetDoubleStats()) {
+ DoubleColumnStatsData dcsd = csd.getDoubleStats();
+ appendColumnStatsNoFormatting(colBuffer, dcsd.getLowValue(),
+ dcsd.getHighValue(), dcsd.getNumNulls(), dcsd.getNumDVs(), "", "", "", "");
+ } else if (csd.isSetLongStats()) {
+ LongColumnStatsData lcsd = csd.getLongStats();
+ appendColumnStatsNoFormatting(colBuffer, lcsd.getLowValue(),
+ lcsd.getHighValue(), lcsd.getNumNulls(), lcsd.getNumDVs(), "", "", "", "");
+ } else if (csd.isSetDateStats()) {
+ DateColumnStatsData dcsd = csd.getDateStats();
+ appendColumnStatsNoFormatting(colBuffer, convertToString(dcsd.getLowValue()),
+ convertToString(dcsd.getHighValue()), dcsd.getNumNulls(), dcsd.getNumDVs(),
+ "", "", "", "");
+ }
+ } else {
+ appendColumnStatsNoFormatting(colBuffer, "", "", "", "", "", "", "", "");
+ }
+ }
+ colBuffer.append(comment == null ? "" : ESCAPE_JAVA.translate(comment));
+ colBuffer.append(LINE_DELIM);
+ }
+
+ private static final CharSequenceTranslator ESCAPE_JAVA =
+ new LookupTranslator(new String[][]{{"\"", "\\\""}, {"\\", "\\\\"},})
+ .with(new LookupTranslator(EntityArrays.JAVA_CTRL_CHARS_ESCAPE()));
+
+ private static void appendColumnStatsNoFormatting(StringBuilder sb, Object min,
+ Object max, Object numNulls, Object ndv, Object avgColLen, Object maxColLen,
+ Object numTrues, Object numFalses) {
+ sb.append(min).append(FIELD_DELIM);
+ sb.append(max).append(FIELD_DELIM);
+ sb.append(numNulls).append(FIELD_DELIM);
+ sb.append(ndv).append(FIELD_DELIM);
+ sb.append(avgColLen).append(FIELD_DELIM);
+ sb.append(maxColLen).append(FIELD_DELIM);
+ sb.append(numTrues).append(FIELD_DELIM);
+ sb.append(numFalses).append(FIELD_DELIM);
+ }
+
+ static String getComment(FieldSchema col) {
+ return col.getComment() != null ? col.getComment() : "";
+ }
+
+ private static void formatWithIndentation(String colName, String colType,
+ String colComment, StringBuilder tableInfo, List<ColumnStatisticsObj> colStats) {
+ tableInfo.append(String.format("%-" + ALIGNMENT + "s", colName)).append(FIELD_DELIM);
+ tableInfo.append(String.format("%-" + ALIGNMENT + "s", colType)).append(FIELD_DELIM);
+
+ if (colStats != null) {
+ ColumnStatisticsObj cso = getColumnStatisticsObject(colName, colType, colStats);
+ if (cso != null) {
+ ColumnStatisticsData csd = cso.getStatsData();
+ if (csd.isSetBinaryStats()) {
+ BinaryColumnStatsData bcsd = csd.getBinaryStats();
+ appendColumnStats(tableInfo, "", "", bcsd.getNumNulls(), "",
+ bcsd.getAvgColLen(), bcsd.getMaxColLen(), "", "");
+ } else if (csd.isSetStringStats()) {
+ StringColumnStatsData scsd = csd.getStringStats();
+ appendColumnStats(tableInfo, "", "", scsd.getNumNulls(), scsd.getNumDVs(),
+ scsd.getAvgColLen(), scsd.getMaxColLen(), "", "");
+ } else if (csd.isSetBooleanStats()) {
+ BooleanColumnStatsData bcsd = csd.getBooleanStats();
+ appendColumnStats(tableInfo, "", "", bcsd.getNumNulls(), "", "", "",
+ bcsd.getNumTrues(), bcsd.getNumFalses());
+ } else if (csd.isSetDecimalStats()) {
+ DecimalColumnStatsData dcsd = csd.getDecimalStats();
+ appendColumnStats(tableInfo, convertToString(dcsd.getLowValue()),
+ convertToString(dcsd.getHighValue()), dcsd.getNumNulls(), dcsd.getNumDVs(),
+ "", "", "", "");
+ } else if (csd.isSetDoubleStats()) {
+ DoubleColumnStatsData dcsd = csd.getDoubleStats();
+ appendColumnStats(tableInfo, dcsd.getLowValue(), dcsd.getHighValue(),
+ dcsd.getNumNulls(), dcsd.getNumDVs(), "", "", "", "");
+ } else if (csd.isSetLongStats()) {
+ LongColumnStatsData lcsd = csd.getLongStats();
+ appendColumnStats(tableInfo, lcsd.getLowValue(), lcsd.getHighValue(),
+ lcsd.getNumNulls(), lcsd.getNumDVs(), "", "", "", "");
+ } else if (csd.isSetDateStats()) {
+ DateColumnStatsData dcsd = csd.getDateStats();
+ appendColumnStats(tableInfo, convertToString(dcsd.getLowValue()),
+ convertToString(dcsd.getHighValue()), dcsd.getNumNulls(), dcsd.getNumDVs(),
+ "", "", "", "");
+ }
+ } else {
+ appendColumnStats(tableInfo, "", "", "", "", "", "", "", "");
+ }
+ }
+
+ int colNameLength = ALIGNMENT > colName.length() ? ALIGNMENT : colName.length();
+ int colTypeLength = ALIGNMENT > colType.length() ? ALIGNMENT : colType.length();
+ indentMultilineValue(colComment, tableInfo, new int[]{colNameLength, colTypeLength},
+ false);
+ }
+
+ /**
+ * comment indent processing for multi-line values values should be indented the same
+ * amount on each line if the first line comment starts indented by k, the following
+ * line comments should also be indented by k
+ *
+ * @param value the value to write
+ * @param tableInfo the buffer to write to
+ * @param columnWidths the widths of the previous columns
+ * @param printNull print null as a string, or do not print anything
+ */
+ private static void indentMultilineValue(String value, StringBuilder tableInfo,
+ int[] columnWidths, boolean printNull) {
+ if (value == null) {
+ if (printNull) {
+ tableInfo.append(String.format("%-" + ALIGNMENT + "s", value));
+ }
+ tableInfo.append(LINE_DELIM);
+ } else {
+ String[] valueSegments = value.split("\n|\r|\r\n");
+ tableInfo.append(String.format("%-" + ALIGNMENT + "s", valueSegments[0]))
+ .append(LINE_DELIM);
+ for (int i = 1; i < valueSegments.length; i++) {
+ printPadding(tableInfo, columnWidths);
+ tableInfo.append(String.format("%-" + ALIGNMENT + "s", valueSegments[i]))
+ .append(LINE_DELIM);
+ }
+ }
+ }
+
+ /**
+ * Print the rigth padding, with the given column widths
+ *
+ * @param tableInfo The buffer to write to
+ * @param columnWidths The column widths
+ */
+ private static void printPadding(StringBuilder tableInfo, int[] columnWidths) {
+ for (int columnWidth : columnWidths) {
+ if (columnWidth == 0) {
+ tableInfo.append(FIELD_DELIM);
+ } else {
+ tableInfo.append(String.format("%" + columnWidth + "s" + FIELD_DELIM, ""));
+ }
+ }
+ }
+
+ private static String convertToString(Decimal val) {
+ if (val == null) {
+ return "";
+ }
+
+ HiveDecimal result =
+ HiveDecimal.create(new BigInteger(val.getUnscaled()), val.getScale());
+ if (result != null) {
+ return result.toString();
+ } else {
+ return "";
+ }
+ }
+
+ private static String convertToString(org.apache.hadoop.hive.metastore.api.Date val) {
+ if (val == null) {
+ return "";
+ }
+
+ DateWritable writableValue = new DateWritable((int) val.getDaysSinceEpoch());
+ return writableValue.toString();
+ }
+
+ private static void appendColumnStats(StringBuilder sb, Object min, Object max,
+ Object numNulls, Object ndv, Object avgColLen, Object maxColLen, Object numTrues,
+ Object numFalses) {
+ sb.append(String.format("%-" + ALIGNMENT + "s", min)).append(FIELD_DELIM);
+ sb.append(String.format("%-" + ALIGNMENT + "s", max)).append(FIELD_DELIM);
+ sb.append(String.format("%-" + ALIGNMENT + "s", numNulls)).append(FIELD_DELIM);
+ sb.append(String.format("%-" + ALIGNMENT + "s", ndv)).append(FIELD_DELIM);
+ sb.append(String.format("%-" + ALIGNMENT + "s", avgColLen)).append(FIELD_DELIM);
+ sb.append(String.format("%-" + ALIGNMENT + "s", maxColLen)).append(FIELD_DELIM);
+ sb.append(String.format("%-" + ALIGNMENT + "s", numTrues)).append(FIELD_DELIM);
+ sb.append(String.format("%-" + ALIGNMENT + "s", numFalses)).append(FIELD_DELIM);
+ }
+
+ private static ColumnStatisticsObj getColumnStatisticsObject(String colName,
+ String colType, List<ColumnStatisticsObj> colStats) {
+ if (colStats != null && !colStats.isEmpty()) {
+ for (ColumnStatisticsObj cso : colStats) {
+ if (cso.getColName().equalsIgnoreCase(colName) && cso.getColType()
+ .equalsIgnoreCase(colType)) {
+ return cso;
+ }
+ }
+ }
+ return null;
+ }
+
+ public static String getTableInformation(Table table, boolean isOutputPadded) {
+ StringBuilder tableInfo = new StringBuilder(DEFAULT_STRINGBUILDER_SIZE);
+
+ // Table Metadata
+ tableInfo.append(LINE_DELIM).append("# Detailed Table Information")
+ .append(LINE_DELIM);
+ getTableMetaDataInformation(tableInfo, table, isOutputPadded);
+
+ // Storage information.
+ tableInfo.append(LINE_DELIM).append("# Storage Information").append(LINE_DELIM);
+ getStorageDescriptorInfo(tableInfo, table.getSd());
+
+ if (TableType.VIRTUAL_VIEW.equals(TableType.valueOf(table.getTableType()))) {
+ tableInfo.append(LINE_DELIM).append("# View Information").append(LINE_DELIM);
+ getViewInfo(tableInfo, table);
+ }
+
+ return tableInfo.toString();
+ }
+
+ private static void getViewInfo(StringBuilder tableInfo, Table tbl) {
+ formatOutput("View Original Text:", tbl.getViewOriginalText(), tableInfo);
+ formatOutput("View Expanded Text:", tbl.getViewExpandedText(), tableInfo);
+ }
+
+ private static void getTableMetaDataInformation(StringBuilder tableInfo, Table tbl,
+ boolean isOutputPadded) {
+ formatOutput("Database:", tbl.getDbName(), tableInfo);
+ formatOutput("OwnerType:",
+ (tbl.getOwnerType() != null) ? tbl.getOwnerType().name() : "null", tableInfo);
+ formatOutput("Owner:", tbl.getOwner(), tableInfo);
+ formatOutput("CreateTime:", formatDate(tbl.getCreateTime()), tableInfo);
+ formatOutput("LastAccessTime:", formatDate(tbl.getLastAccessTime()), tableInfo);
+ formatOutput("Retention:", Integer.toString(tbl.getRetention()), tableInfo);
+ if (!TableType.VIRTUAL_VIEW.toString().equals(tbl.getTableType())) {
+ String location = null;
+ if (tbl.getSd() != null) {
+ location = tbl.getSd().getLocation();
+ }
+ formatOutput("Location:", location, tableInfo);
+ }
+ formatOutput("Table Type:", tbl.getTableType(), tableInfo);
+
+ if (tbl.getParameters().size() > 0) {
+ tableInfo.append("Table Parameters:").append(LINE_DELIM);
+ displayAllParameters(tbl.getParameters(), tableInfo, false, isOutputPadded);
+ }
+ }
+
+ /**
+ * The name of the statistic for Number of Erasure Coded Files - to be published or
+ * gathered.
+ */
+ private static final String NUM_ERASURE_CODED_FILES = "numFilesErasureCoded";
+
+ /**
+ * Display key, value pairs of the parameters. The characters will be escaped including
+ * unicode if escapeUnicode is true; otherwise the characters other than unicode will be
+ * escaped.
+ */
+ private static void displayAllParameters(Map<String, String> params,
+ StringBuilder tableInfo, boolean escapeUnicode, boolean isOutputPadded) {
+ List<String> keys = new ArrayList<String>(params.keySet());
+ Collections.sort(keys);
+ for (String key : keys) {
+ String value = params.get(key);
+ //TODO(Vihang) HIVE-18118 should be ported to Hive-3.1
+ if (key.equals(NUM_ERASURE_CODED_FILES)) {
+ if ("0".equals(value)) {
+ continue;
+ }
+ }
+ tableInfo.append(FIELD_DELIM); // Ensures all params are indented.
+ formatOutput(key, escapeUnicode ? StringEscapeUtils.escapeJava(value)
+ : ESCAPE_JAVA.translate(value), tableInfo, isOutputPadded);
+ }
+ }
+
+ /**
+ * Prints the name value pair It the output is padded then unescape the value, so it
+ * could be printed in multiple lines. In this case it assumes the pair is already
+ * indented with a field delimiter
+ *
+ * @param name The field name to print
+ * @param value The value t print
+ * @param tableInfo The target builder
+ * @param isOutputPadded Should the value printed as a padded string?
+ */
+ protected static void formatOutput(String name, String value, StringBuilder tableInfo,
+ boolean isOutputPadded) {
+ String unescapedValue = (isOutputPadded && value != null) ? value
+ .replaceAll("\\\\n|\\\\r|\\\\r\\\\n", "\n") : value;
+ formatOutput(name, unescapedValue, tableInfo);
+ }
+
+ /**
+ * Prints the name value pair, and if the value contains newlines, it add one more empty
+ * field before the two values (Assumes, the name value pair is already indented with
+ * it)
+ *
+ * @param name The field name to print
+ * @param value The value to print - might contain newlines
+ * @param tableInfo The target builder
+ */
+ private static void formatOutput(String name, String value, StringBuilder tableInfo) {
+ tableInfo.append(String.format("%-" + ALIGNMENT + "s", name)).append(FIELD_DELIM);
+ int colNameLength = ALIGNMENT > name.length() ? ALIGNMENT : name.length();
+ indentMultilineValue(value, tableInfo, new int[]{0, colNameLength}, true);
+ }
+
+ private static String formatDate(long timeInSeconds) {
+ if (timeInSeconds != 0) {
+ Date date = new Date(timeInSeconds * 1000);
+ return date.toString();
+ }
+ return "UNKNOWN";
+ }
+
+ private static void getStorageDescriptorInfo(StringBuilder tableInfo,
+ StorageDescriptor storageDesc) {
+
+ formatOutput("SerDe Library:", storageDesc.getSerdeInfo().getSerializationLib(),
+ tableInfo);
+ formatOutput("InputFormat:", storageDesc.getInputFormat(), tableInfo);
+ formatOutput("OutputFormat:", storageDesc.getOutputFormat(), tableInfo);
+ formatOutput("Compressed:", storageDesc.isCompressed() ? "Yes" : "No", tableInfo);
+ formatOutput("Num Buckets:", String.valueOf(storageDesc.getNumBuckets()), tableInfo);
+ formatOutput("Bucket Columns:", storageDesc.getBucketCols().toString(), tableInfo);
+ formatOutput("Sort Columns:", storageDesc.getSortCols().toString(), tableInfo);
+ if (storageDesc.isStoredAsSubDirectories()) {// optional parameter
+ formatOutput("Stored As SubDirectories:", "Yes", tableInfo);
+ }
+
+ if (null != storageDesc.getSkewedInfo()) {
+ List<String> skewedColNames =
+ sortedList(storageDesc.getSkewedInfo().getSkewedColNames());
+ if ((skewedColNames != null) && (skewedColNames.size() > 0)) {
+ formatOutput("Skewed Columns:", skewedColNames.toString(), tableInfo);
+ }
+
+ List<List<String>> skewedColValues =
+ sortedList(storageDesc.getSkewedInfo().getSkewedColValues(),
+ new VectorComparator<String>());
+ if ((skewedColValues != null) && (skewedColValues.size() > 0)) {
+ formatOutput("Skewed Values:", skewedColValues.toString(), tableInfo);
+ }
+
+ Map<List<String>, String> skewedColMap =
+ new TreeMap<>(new VectorComparator<>());
+ skewedColMap.putAll(storageDesc.getSkewedInfo().getSkewedColValueLocationMaps());
+ if ((skewedColMap != null) && (skewedColMap.size() > 0)) {
+ formatOutput("Skewed Value to Path:", skewedColMap.toString(), tableInfo);
+ Map<List<String>, String> truncatedSkewedColMap =
+ new TreeMap<List<String>, String>(new VectorComparator<String>());
+ // walk through existing map to truncate path so that test won't mask it
+ // then we can verify location is right
+ Set<Entry<List<String>, String>> entries = skewedColMap.entrySet();
+ for (Entry<List<String>, String> entry : entries) {
+ truncatedSkewedColMap.put(entry.getKey(), entry.getValue());
+ }
+ formatOutput("Skewed Value to Truncated Path:", truncatedSkewedColMap.toString(),
+ tableInfo);
+ }
+ }
+
+ if (storageDesc.getSerdeInfo().getParametersSize() > 0) {
+ tableInfo.append("Storage Desc Params:").append(LINE_DELIM);
+ displayAllParameters(storageDesc.getSerdeInfo().getParameters(), tableInfo, true,
+ false);
+ }
+ }
+
+ /**
+ * Returns a sorted version of the given list, using the provided comparator
+ */
+ static <T> List<T> sortedList(List<T> list, Comparator<T> comp) {
+ if (list == null || list.size() <= 1) {
+ return list;
+ }
+ ArrayList<T> ret = new ArrayList<>();
+ ret.addAll(list);
+ Collections.sort(ret, comp);
+ return ret;
+ }
+
+ /**
+ * Returns a sorted version of the given list
+ */
+ static <T extends Comparable<T>> List<T> sortedList(List<T> list) {
+ if (list == null || list.size() <= 1) {
+ return list;
+ }
+ ArrayList<T> ret = new ArrayList<>();
+ ret.addAll(list);
+ Collections.sort(ret);
+ return ret;
+ }
+
+ /**
+ * Compares to lists of object T as vectors
+ *
+ * @param <T> the base object type. Must be {@link Comparable}
+ */
+ private static class VectorComparator<T extends Comparable<T>> implements
+ Comparator<List<T>> {
+
+ @Override
+ public int compare(List<T> listA, List<T> listB) {
+ for (int i = 0; i < listA.size() && i < listB.size(); i++) {
+ T valA = listA.get(i);
+ T valB = listB.get(i);
+ if (valA != null) {
+ int ret = valA.compareTo(valB);
+ if (ret != 0) {
+ return ret;
+ }
+ } else {
+ if (valB != null) {
+ return -1;
+ }
+ }
+ }
+ return Integer.compare(listA.size(), listB.size());
+ }
+ }
+}
diff --git a/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java b/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
new file mode 100644
index 0000000..ee22c2b
--- /dev/null
+++ b/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
@@ -0,0 +1,367 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.impala.compat;
+
+import static org.apache.impala.service.MetadataOp.TABLE_TYPE_TABLE;
+import static org.apache.impala.service.MetadataOp.TABLE_TYPE_VIEW;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.ImmutableMap;
+import java.util.EnumSet;
+import java.util.List;
+
+import org.apache.hadoop.hive.common.StatsSetupConst;
+import org.apache.hadoop.hive.metastore.IMetaStoreClient;
+import org.apache.hadoop.hive.metastore.TableType;
+import org.apache.hadoop.hive.metastore.Warehouse;
+import org.apache.hadoop.hive.metastore.api.FieldSchema;
+import org.apache.hadoop.hive.metastore.api.InvalidOperationException;
+import org.apache.hadoop.hive.metastore.api.MetaException;
+import org.apache.hadoop.hive.metastore.api.Partition;
+import org.apache.hadoop.hive.metastore.api.Table;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf;
+import org.apache.hadoop.hive.metastore.messaging.AlterTableMessage;
+import org.apache.hadoop.hive.metastore.messaging.InsertMessage;
+import org.apache.hadoop.hive.metastore.messaging.MessageBuilder;
+import org.apache.hadoop.hive.metastore.messaging.MessageDeserializer;
+import org.apache.hadoop.hive.metastore.messaging.MessageEncoder;
+import org.apache.hadoop.hive.metastore.messaging.MessageFactory;
+import org.apache.hadoop.hive.metastore.utils.FileUtils;
+import org.apache.hadoop.hive.metastore.utils.MetaStoreUtils;
+import org.apache.hive.service.rpc.thrift.TGetColumnsReq;
+import org.apache.hive.service.rpc.thrift.TGetFunctionsReq;
+import org.apache.hive.service.rpc.thrift.TGetSchemasReq;
+import org.apache.hive.service.rpc.thrift.TGetTablesReq;
+import org.apache.impala.authorization.User;
+import org.apache.impala.common.ImpalaException;
+import org.apache.impala.common.Pair;
+import org.apache.impala.compat.HiveMetadataFormatUtils;
+import org.apache.impala.service.Frontend;
+import org.apache.impala.service.MetadataOp;
+import org.apache.impala.thrift.TMetadataOpRequest;
+import org.apache.impala.thrift.TResultSet;
+import org.apache.thrift.TException;
+
+/**
+ * A wrapper around some of Hive's Metastore API's to abstract away differences
+ * between major versions of Hive. This implements the shimmed methods for Hive 2.
+ */
+public class MetastoreShim {
+ /**
+ * Wrapper around MetaStoreUtils.validateName() to deal with added arguments.
+ */
+ public static boolean validateName(String name) {
+ return MetaStoreUtils.validateName(name, null);
+ }
+
+ /**
+ * Wrapper around IMetaStoreClient.alter_partition() to deal with added
+ * arguments.
+ */
+ public static void alterPartition(IMetaStoreClient client, Partition partition)
+ throws InvalidOperationException, MetaException, TException {
+ client.alter_partition(
+ partition.getDbName(), partition.getTableName(), partition, null);
+ }
+
+ /**
+ * Wrapper around IMetaStoreClient.alter_partitions() to deal with added
+ * arguments.
+ */
+ public static void alterPartitions(IMetaStoreClient client, String dbName,
+ String tableName, List<Partition> partitions)
+ throws InvalidOperationException, MetaException, TException {
+ client.alter_partitions(dbName, tableName, partitions, null);
+ }
+
+ /**
+ * Wrapper around MetaStoreUtils.updatePartitionStatsFast() to deal with added
+ * arguments.
+ */
+ public static void updatePartitionStatsFast(Partition partition, Table tbl,
+ Warehouse warehouse) throws MetaException {
+ MetaStoreUtils.updatePartitionStatsFast(partition, tbl, warehouse, /*madeDir*/false,
+ /*forceRecompute*/false,
+ /*environmentContext*/null, /*isCreate*/false);
+ }
+
+ /**
+ * Return the maximum number of Metastore objects that should be retrieved in
+ * a batch.
+ */
+ public static String metastoreBatchRetrieveObjectsMaxConfigKey() {
+ return MetastoreConf.ConfVars.BATCH_RETRIEVE_OBJECTS_MAX.toString();
+ }
+
+ /**
+ * Return the key and value that should be set in the partition parameters to
+ * mark that the stats were generated automatically by a stats task.
+ */
+ public static Pair<String, String> statsGeneratedViaStatsTaskParam() {
+ return Pair.create(StatsSetupConst.STATS_GENERATED, StatsSetupConst.TASK);
+ }
+
+ public static TResultSet execGetFunctions(
+ Frontend frontend, TMetadataOpRequest request, User user) throws ImpalaException {
+ TGetFunctionsReq req = request.getGet_functions_req();
+ return MetadataOp.getFunctions(
+ frontend, req.getCatalogName(), req.getSchemaName(), req.getFunctionName(), user);
+ }
+
+ public static TResultSet execGetColumns(
+ Frontend frontend, TMetadataOpRequest request, User user) throws ImpalaException {
+ TGetColumnsReq req = request.getGet_columns_req();
+ return MetadataOp.getColumns(frontend, req.getCatalogName(), req.getSchemaName(),
+ req.getTableName(), req.getColumnName(), user);
+ }
+
+ public static TResultSet execGetTables(
+ Frontend frontend, TMetadataOpRequest request, User user) throws ImpalaException {
+ TGetTablesReq req = request.getGet_tables_req();
+ return MetadataOp.getTables(frontend, req.getCatalogName(), req.getSchemaName(),
+ req.getTableName(), req.getTableTypes(), user);
+ }
+
+ public static TResultSet execGetSchemas(
+ Frontend frontend, TMetadataOpRequest request, User user) throws ImpalaException {
+ TGetSchemasReq req = request.getGet_schemas_req();
+ return MetadataOp.getSchemas(
+ frontend, req.getCatalogName(), req.getSchemaName(), user);
+ }
+
+ /**
+ * Supported HMS-3 types
+ */
+ public static final EnumSet<TableType> IMPALA_SUPPORTED_TABLE_TYPES = EnumSet
+ .of(TableType.EXTERNAL_TABLE, TableType.MANAGED_TABLE, TableType.VIRTUAL_VIEW,
+ TableType.MATERIALIZED_VIEW);
+
+ /**
+ * mapping between the HMS-3 type the Impala types
+ */
+ public static final ImmutableMap<String, String> HMS_TO_IMPALA_TYPE =
+ new ImmutableMap.Builder<String, String>()
+ .put("EXTERNAL_TABLE", TABLE_TYPE_TABLE)
+ .put("MANAGED_TABLE", TABLE_TYPE_TABLE)
+ .put("INDEX_TABLE", TABLE_TYPE_TABLE)
+ .put("VIRTUAL_VIEW", TABLE_TYPE_VIEW)
+ .put("MATERIALIZED_VIEW", TABLE_TYPE_VIEW).build();
+ /**
+ * Method which maps Metastore's TableType to Impala's table type. In metastore 2
+ * Materialized view is not supported
+ */
+ public static String mapToInternalTableType(String typeStr) {
+ String defaultTableType = TABLE_TYPE_TABLE;
+ TableType tType;
+
+ if (typeStr == null) return defaultTableType;
+ try {
+ tType = TableType.valueOf(typeStr.toUpperCase());
+ } catch (Exception e) {
+ return defaultTableType;
+ }
+ switch (tType) {
+ case EXTERNAL_TABLE:
+ case MANAGED_TABLE:
+ //Deprecated and removed in Hive-3.. //TODO throw exception?
+ case INDEX_TABLE:
+ return TABLE_TYPE_TABLE;
+ case VIRTUAL_VIEW:
+ case MATERIALIZED_VIEW:
+ return TABLE_TYPE_VIEW;
+ default:
+ return defaultTableType;
+ }
+ }
+
+ //hive-3 has a different class to encode and decode event messages
+ private static final MessageEncoder eventMessageEncoder_ =
+ MessageFactory.getDefaultInstance(MetastoreConf.newMetastoreConf());
+
+ /**
+ * Wrapper method which returns HMS-3 Message factory in case Impala is
+ * building against Hive-3
+ */
+ public static MessageDeserializer getMessageDeserializer() {
+ return eventMessageEncoder_.getDeserializer();
+ }
+
+ /**
+ * Wrapper around FileUtils.makePartName to deal with package relocation in Hive 3.
+ * This method uses the metastore's FileUtils method instead of one from hive-exec
+ * @param partitionColNames
+ * @param values
+ * @return
+ */
+ public static String makePartName(List<String> partitionColNames, List<String> values) {
+ return FileUtils.makePartName(partitionColNames, values);
+ }
+
+ /**
+ * Wrapper method around message factory's build alter table message due to added
+ * arguments in hive 3.
+ */
+ @VisibleForTesting
+ public static AlterTableMessage buildAlterTableMessage(Table before, Table after,
+ boolean isTruncateOp, long writeId) {
+ return MessageBuilder.getInstance().buildAlterTableMessage(before, after,
+ isTruncateOp, writeId);
+ }
+
+ @VisibleForTesting
+ public static InsertMessage buildInsertMessage(Table msTbl, Partition partition,
+ boolean isInsertOverwrite, List<String> newFiles) {
+ return MessageBuilder.getInstance().buildInsertMessage(msTbl, partition,
+ isInsertOverwrite, newFiles.iterator());
+ }
+
+ /**
+ * Wrapper method to get the formatted string to represent the columns information of
+ * a metastore table. This method was changed in Hive-3 significantly when compared
+ * to Hive-2. In order to avoid adding unnecessary dependency to hive-exec this
+ * method copies the source code from hive-2's MetaDataFormatUtils class for this
+ * method.
+ * TODO : In order to avoid this copy, we move move this code from hive's ql module
+ * to a util method in MetastoreUtils in metastore module
+ * @return
+ */
+ public static String getAllColumnsInformation(List<FieldSchema> tabCols,
+ List<FieldSchema> partitionCols, boolean printHeader, boolean isOutputPadded,
+ boolean showPartColsSeparately) {
+ return HiveMetadataFormatUtils
+ .getAllColumnsInformation(tabCols, partitionCols, printHeader, isOutputPadded,
+ showPartColsSeparately);
+ }
+
+ /**
+ * Wrapper method around Hive's MetadataFormatUtils.getTableInformation which has
+ * changed significantly in Hive-3
+ * @return
+ */
+ public static String getTableInformation(
+ org.apache.hadoop.hive.ql.metadata.Table table) {
+ return HiveMetadataFormatUtils.getTableInformation(table.getTTable(), false);
+ }
+
+ /**
+ * This method has been copied from BaseSemanticAnalyzer class of Hive and is fairly
+ * stable now (last change was in mid 2016 as of April 2019). Copying is preferred over
+ * adding dependency to this class which pulls in a lot of other transitive
+ * dependencies from hive-exec
+ */
+ public static String unescapeSQLString(String stringLiteral) {
+ {
+ Character enclosure = null;
+
+ // Some of the strings can be passed in as unicode. For example, the
+ // delimiter can be passed in as \002 - So, we first check if the
+ // string is a unicode number, else go back to the old behavior
+ StringBuilder sb = new StringBuilder(stringLiteral.length());
+ for (int i = 0; i < stringLiteral.length(); i++) {
+
+ char currentChar = stringLiteral.charAt(i);
+ if (enclosure == null) {
+ if (currentChar == '\'' || stringLiteral.charAt(i) == '\"') {
+ enclosure = currentChar;
+ }
+ // ignore all other chars outside the enclosure
+ continue;
+ }
+
+ if (enclosure.equals(currentChar)) {
+ enclosure = null;
+ continue;
+ }
+
+ if (currentChar == '\\' && (i + 6 < stringLiteral.length()) && stringLiteral.charAt(i + 1) == 'u') {
+ int code = 0;
+ int base = i + 2;
+ for (int j = 0; j < 4; j++) {
+ int digit = Character.digit(stringLiteral.charAt(j + base), 16);
+ code = (code << 4) + digit;
+ }
+ sb.append((char)code);
+ i += 5;
+ continue;
+ }
+
+ if (currentChar == '\\' && (i + 4 < stringLiteral.length())) {
+ char i1 = stringLiteral.charAt(i + 1);
+ char i2 = stringLiteral.charAt(i + 2);
+ char i3 = stringLiteral.charAt(i + 3);
+ if ((i1 >= '0' && i1 <= '1') && (i2 >= '0' && i2 <= '7')
+ && (i3 >= '0' && i3 <= '7')) {
+ byte bVal = (byte) ((i3 - '0') + ((i2 - '0') * 8) + ((i1 - '0') * 8 * 8));
+ byte[] bValArr = new byte[1];
+ bValArr[0] = bVal;
+ String tmp = new String(bValArr);
+ sb.append(tmp);
+ i += 3;
+ continue;
+ }
+ }
+
+ if (currentChar == '\\' && (i + 2 < stringLiteral.length())) {
+ char n = stringLiteral.charAt(i + 1);
+ switch (n) {
+ case '0':
+ sb.append("\0");
+ break;
+ case '\'':
+ sb.append("'");
+ break;
+ case '"':
+ sb.append("\"");
+ break;
+ case 'b':
+ sb.append("\b");
+ break;
+ case 'n':
+ sb.append("\n");
+ break;
+ case 'r':
+ sb.append("\r");
+ break;
+ case 't':
+ sb.append("\t");
+ break;
+ case 'Z':
+ sb.append("\u001A");
+ break;
+ case '\\':
+ sb.append("\\");
+ break;
+ // The following 2 lines are exactly what MySQL does TODO: why do we do this?
+ case '%':
+ sb.append("\\%");
+ break;
+ case '_':
+ sb.append("\\_");
+ break;
+ default:
+ sb.append(n);
+ }
+ i++;
+ } else {
+ sb.append(currentChar);
+ }
+ }
+ return sb.toString();
+ }
+ }
+}
diff --git a/fe/src/main/java/org/apache/impala/analysis/StringLiteral.java b/fe/src/main/java/org/apache/impala/analysis/StringLiteral.java
index 797ad5e..09a542f 100644
--- a/fe/src/main/java/org/apache/impala/analysis/StringLiteral.java
+++ b/fe/src/main/java/org/apache/impala/analysis/StringLiteral.java
@@ -21,10 +21,10 @@ import java.io.IOException;
import java.io.StringReader;
import java.math.BigDecimal;
-import org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer;
import org.apache.impala.catalog.ScalarType;
import org.apache.impala.catalog.Type;
import org.apache.impala.common.AnalysisException;
+import org.apache.impala.compat.MetastoreShim;
import org.apache.impala.thrift.TExprNode;
import org.apache.impala.thrift.TExprNodeType;
import org.apache.impala.thrift.TStringLiteral;
@@ -91,7 +91,7 @@ public class StringLiteral extends LiteralExpr {
public String getUnescapedValue() {
// Unescape string exactly like Hive does. Hive's method assumes
// quotes so we add them here to reuse Hive's code.
- return BaseSemanticAnalyzer.unescapeSQLString("'" + getNormalizedValue()
+ return MetastoreShim.unescapeSQLString("'" + getNormalizedValue()
+ "'");
}
diff --git a/fe/src/main/java/org/apache/impala/catalog/FeHBaseTable.java b/fe/src/main/java/org/apache/impala/catalog/FeHBaseTable.java
index fbb6d5d..df4f12e 100644
--- a/fe/src/main/java/org/apache/impala/catalog/FeHBaseTable.java
+++ b/fe/src/main/java/org/apache/impala/catalog/FeHBaseTable.java
@@ -44,7 +44,6 @@ import org.apache.hadoop.hbase.client.ResultScanner;
import org.apache.hadoop.hbase.client.Scan;
import org.apache.hadoop.hbase.io.compress.Compression;
import org.apache.hadoop.hbase.util.Bytes;
-import org.apache.hadoop.hive.hbase.HBaseSerDe;
import org.apache.hadoop.hive.metastore.api.FieldSchema;
import org.apache.hadoop.hive.metastore.api.MetaException;
import org.apache.hadoop.hive.metastore.api.Table;
@@ -106,6 +105,16 @@ public interface FeHBaseTable extends FeTable {
// Minimum number of regions that are checked to estimate the row count
private static final int MIN_NUM_REGIONS_TO_CHECK = 5;
+ // constants from Hive's HBaseSerDe.java copied here to avoid dependending on
+ // hive-hbase-handler (and its transitive dependencies) These are user facing
+ // properties and pretty much guaranteed to not change without breaking backwards
+ // compatibility. Hence it is safe to just copy them here
+ private static final String HBASE_COLUMNS_MAPPING = "hbase.columns.mapping";
+ private static final String HBASE_TABLE_DEFAULT_STORAGE_TYPE =
+ "hbase.table.default.storage.type";
+ private static final String HBASE_KEY_COL = ":key";
+ private static final String HBASE_TABLE_NAME = "hbase.table.name";
+
/**
* Table client objects are thread-unsafe and cheap to create. The HBase docs
* recommend creating a new one for each task and then closing when done.
@@ -127,13 +136,13 @@ public interface FeHBaseTable extends FeTable {
org.apache.hadoop.hive.metastore.api.Table msTable)
throws MetaException, SerDeException {
Map<String, String> serdeParams = msTable.getSd().getSerdeInfo().getParameters();
- String hbaseColumnsMapping = serdeParams.get(HBaseSerDe.HBASE_COLUMNS_MAPPING);
+ String hbaseColumnsMapping = serdeParams.get(HBASE_COLUMNS_MAPPING);
if (hbaseColumnsMapping == null) {
throw new MetaException("No hbase.columns.mapping defined in Serde.");
}
String hbaseTableDefaultStorageType =
- msTable.getParameters().get(HBaseSerDe.HBASE_TABLE_DEFAULT_STORAGE_TYPE);
+ msTable.getParameters().get(HBASE_TABLE_DEFAULT_STORAGE_TYPE);
boolean tableDefaultStorageIsBinary = false;
if (hbaseTableDefaultStorageType != null &&
!hbaseTableDefaultStorageType.isEmpty()) {
@@ -141,7 +150,7 @@ public interface FeHBaseTable extends FeTable {
tableDefaultStorageIsBinary = true;
} else if (!hbaseTableDefaultStorageType.equalsIgnoreCase("string")) {
throw new SerDeException(
- "Error: " + HBaseSerDe.HBASE_TABLE_DEFAULT_STORAGE_TYPE +
+ "Error: " + HBASE_TABLE_DEFAULT_STORAGE_TYPE +
" parameter must be specified as" + " 'string' or 'binary'; '" +
hbaseTableDefaultStorageType +
"' is not a valid specification for this table/serde property.");
@@ -224,7 +233,7 @@ public interface FeHBaseTable extends FeTable {
}
if (columnsMappingSpec.equals("") ||
- columnsMappingSpec.equals(HBaseSerDe.HBASE_KEY_COL)) {
+ columnsMappingSpec.equals(HBASE_KEY_COL)) {
throw new SerDeException("Error: hbase.columns.mapping specifies only " +
"the HBase table row key. A valid Hive-HBase table must specify at " +
"least one additional column.");
@@ -258,7 +267,7 @@ public interface FeHBaseTable extends FeTable {
"badly formed column family, column qualifier specification.");
}
- if (colInfo.equals(HBaseSerDe.HBASE_KEY_COL)) {
+ if (colInfo.equals(HBASE_KEY_COL)) {
Preconditions.checkState(fsStartIdxOffset == 0);
rowKeyIndex = i;
columnFamilies.add(colInfo);
@@ -311,13 +320,13 @@ public interface FeHBaseTable extends FeTable {
} else {
// error in storage specification
throw new SerDeException(
- "Error: " + HBaseSerDe.HBASE_COLUMNS_MAPPING + " storage specification " +
+ "Error: " + HBASE_COLUMNS_MAPPING + " storage specification " +
mappingSpec + " is not valid for column: " + fieldSchema.getName());
}
}
if (rowKeyIndex == -1) {
- columnFamilies.add(0, HBaseSerDe.HBASE_KEY_COL);
+ columnFamilies.add(0, HBASE_KEY_COL);
columnQualifiers.add(0, null);
colIsBinaryEncoded.add(0, supportsBinaryEncoding(fieldSchemas.get(0), tblName) &&
tableDefaultStorageIsBinary);
@@ -488,10 +497,10 @@ public interface FeHBaseTable extends FeTable {
// Give preference to TBLPROPERTIES over SERDEPROPERTIES
// (really we should only use TBLPROPERTIES, so this is just
// for backwards compatibility with the original specs).
- String tableName = tbl.getParameters().get(HBaseSerDe.HBASE_TABLE_NAME);
+ String tableName = tbl.getParameters().get(HBASE_TABLE_NAME);
if (tableName == null) {
tableName =
- tbl.getSd().getSerdeInfo().getParameters().get(HBaseSerDe.HBASE_TABLE_NAME);
+ tbl.getSd().getSerdeInfo().getParameters().get(HBASE_TABLE_NAME);
}
if (tableName == null) {
tableName = tbl.getDbName() + "." + tbl.getTableName();
diff --git a/fe/src/main/java/org/apache/impala/catalog/TableLoader.java b/fe/src/main/java/org/apache/impala/catalog/TableLoader.java
index 40638a1..1c6c815 100644
--- a/fe/src/main/java/org/apache/impala/catalog/TableLoader.java
+++ b/fe/src/main/java/org/apache/impala/catalog/TableLoader.java
@@ -22,6 +22,7 @@ import java.util.concurrent.TimeUnit;
import org.apache.hadoop.hive.metastore.TableType;
import org.apache.hadoop.hive.metastore.api.NoSuchObjectException;
+import org.apache.impala.compat.MetastoreShim;
import org.apache.log4j.Logger;
import com.google.common.base.Stopwatch;
@@ -36,10 +37,6 @@ import org.apache.impala.util.ThreadNameAnnotator;
public class TableLoader {
private static final Logger LOG = Logger.getLogger(TableLoader.class);
- // Set of supported table types.
- private static EnumSet<TableType> SUPPORTED_TABLE_TYPES = EnumSet.of(
- TableType.EXTERNAL_TABLE, TableType.MANAGED_TABLE, TableType.VIRTUAL_VIEW);
-
private final CatalogServiceCatalog catalog_;
// Lock used to serialize calls to the Hive MetaStore to work around MetaStore
@@ -73,7 +70,7 @@ public class TableLoader {
}
// Check that the Hive TableType is supported
TableType tableType = TableType.valueOf(msTbl.getTableType());
- if (!SUPPORTED_TABLE_TYPES.contains(tableType)) {
+ if (!MetastoreShim.IMPALA_SUPPORTED_TABLE_TYPES.contains(tableType)) {
throw new TableLoadingException(String.format(
"Unsupported table type '%s' for: %s", tableType, fullTblName));
}
diff --git a/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java b/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
index 493b637..513390a 100644
--- a/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
+++ b/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
@@ -640,7 +640,7 @@ public class MetastoreEvents {
Preconditions
.checkNotNull(event.getMessage(), debugString("Event message is null"));
CreateTableMessage createTableMessage =
- MetastoreEventsProcessor.getMessageFactory().getDeserializer()
+ MetastoreEventsProcessor.getMessageDeserializer()
.getCreateTableMessage(event.getMessage());
try {
msTbl_ = createTableMessage.getTableObj();
@@ -735,8 +735,8 @@ public class MetastoreEvents {
super(catalog, metrics, event);
Preconditions.checkArgument(MetastoreEventType.INSERT.equals(eventType_));
InsertMessage insertMessage =
- MetastoreEventsProcessor.getMessageFactory()
- .getDeserializer().getInsertMessage(event.getMessage());
+ MetastoreEventsProcessor.getMessageDeserializer()
+ .getInsertMessage(event.getMessage());
try {
msTbl_ = Preconditions.checkNotNull(insertMessage.getTableObj());
insertPartition_ = insertMessage.getPtnObj();
@@ -848,8 +848,8 @@ public class MetastoreEvents {
super(catalog, metrics, event);
Preconditions.checkArgument(MetastoreEventType.ALTER_TABLE.equals(eventType_));
JSONAlterTableMessage alterTableMessage =
- (JSONAlterTableMessage) MetastoreEventsProcessor.getMessageFactory()
- .getDeserializer().getAlterTableMessage(event.getMessage());
+ (JSONAlterTableMessage) MetastoreEventsProcessor.getMessageDeserializer()
+ .getAlterTableMessage(event.getMessage());
try {
msTbl_ = Preconditions.checkNotNull(alterTableMessage.getTableObjBefore());
tableAfter_ = Preconditions.checkNotNull(alterTableMessage.getTableObjAfter());
@@ -1021,8 +1021,8 @@ public class MetastoreEvents {
super(catalog, metrics, event);
Preconditions.checkArgument(MetastoreEventType.DROP_TABLE.equals(eventType_));
JSONDropTableMessage dropTableMessage =
- (JSONDropTableMessage) MetastoreEventsProcessor.getMessageFactory()
- .getDeserializer().getDropTableMessage(event.getMessage());
+ (JSONDropTableMessage) MetastoreEventsProcessor.getMessageDeserializer()
+ .getDropTableMessage(event.getMessage());
try {
msTbl_ = Preconditions.checkNotNull(dropTableMessage.getTableObj());
} catch (Exception e) {
@@ -1083,8 +1083,8 @@ public class MetastoreEvents {
super(catalog, metrics, event);
Preconditions.checkArgument(MetastoreEventType.CREATE_DATABASE.equals(eventType_));
JSONCreateDatabaseMessage createDatabaseMessage =
- (JSONCreateDatabaseMessage) MetastoreEventsProcessor.getMessageFactory()
- .getDeserializer().getCreateDatabaseMessage(event.getMessage());
+ (JSONCreateDatabaseMessage) MetastoreEventsProcessor.getMessageDeserializer()
+ .getCreateDatabaseMessage(event.getMessage());
try {
createdDatabase_ =
Preconditions.checkNotNull(createDatabaseMessage.getDatabaseObject());
@@ -1147,8 +1147,7 @@ public class MetastoreEvents {
super(catalog, metrics, event);
Preconditions.checkArgument(MetastoreEventType.ALTER_DATABASE.equals(eventType_));
JSONAlterDatabaseMessage alterDatabaseMessage =
- (JSONAlterDatabaseMessage) MetastoreEventsProcessor.getMessageFactory()
- .getDeserializer()
+ (JSONAlterDatabaseMessage) MetastoreEventsProcessor.getMessageDeserializer()
.getAlterDatabaseMessage(event.getMessage());
try {
alteredDatabase_ =
@@ -1209,8 +1208,8 @@ public class MetastoreEvents {
super(catalog, metrics, event);
Preconditions.checkArgument(MetastoreEventType.DROP_DATABASE.equals(eventType_));
JSONDropDatabaseMessage dropDatabaseMessage =
- (JSONDropDatabaseMessage) MetastoreEventsProcessor.getMessageFactory()
- .getDeserializer().getDropDatabaseMessage(event.getMessage());
+ (JSONDropDatabaseMessage) MetastoreEventsProcessor.getMessageDeserializer()
+ .getDropDatabaseMessage(event.getMessage());
try {
droppedDatabase_ =
Preconditions.checkNotNull(dropDatabaseMessage.getDatabaseObject());
@@ -1318,8 +1317,7 @@ public class MetastoreEvents {
}
try {
AddPartitionMessage addPartitionMessage_ =
- MetastoreEventsProcessor.getMessageFactory()
- .getDeserializer()
+ MetastoreEventsProcessor.getMessageDeserializer()
.getAddPartitionMessage(event.getMessage());
addedPartitions_ =
Lists.newArrayList(addPartitionMessage_.getPartitionObjs());
@@ -1413,7 +1411,7 @@ public class MetastoreEvents {
Preconditions.checkState(eventType_.equals(MetastoreEventType.ALTER_PARTITION));
Preconditions.checkNotNull(event.getMessage());
AlterPartitionMessage alterPartitionMessage =
- MetastoreEventsProcessor.getMessageFactory().getDeserializer()
+ MetastoreEventsProcessor.getMessageDeserializer()
.getAlterPartitionMessage(event.getMessage());
try {
@@ -1496,8 +1494,7 @@ public class MetastoreEvents {
Preconditions.checkState(eventType_.equals(MetastoreEventType.DROP_PARTITION));
Preconditions.checkNotNull(event.getMessage());
DropPartitionMessage dropPartitionMessage =
- MetastoreEventsProcessor.getMessageFactory()
- .getDeserializer()
+ MetastoreEventsProcessor.getMessageDeserializer()
.getDropPartitionMessage(event.getMessage());
try {
msTbl_ = Preconditions.checkNotNull(dropPartitionMessage.getTableObj());
diff --git a/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java b/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
index 5723961..8a6841e 100644
--- a/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
+++ b/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
@@ -35,8 +35,7 @@ import org.apache.hadoop.hive.metastore.IMetaStoreClient;
import org.apache.hadoop.hive.metastore.api.CurrentNotificationEventId;
import org.apache.hadoop.hive.metastore.api.NotificationEvent;
import org.apache.hadoop.hive.metastore.api.NotificationEventResponse;
-import org.apache.hadoop.hive.metastore.messaging.MessageFactory;
-import org.apache.hadoop.hive.metastore.messaging.json.ExtendedJSONMessageFactory;
+import org.apache.hadoop.hive.metastore.messaging.MessageDeserializer;
import org.apache.impala.catalog.CatalogException;
import org.apache.impala.catalog.CatalogServiceCatalog;
import org.apache.impala.catalog.MetaStoreClientPool.MetaStoreClient;
@@ -44,6 +43,7 @@ import org.apache.impala.catalog.events.EventProcessorConfigValidator.Validation
import org.apache.impala.catalog.events.MetastoreEvents.MetastoreEvent;
import org.apache.impala.catalog.events.MetastoreEvents.MetastoreEventFactory;
import org.apache.impala.common.Metrics;
+import org.apache.impala.compat.MetastoreShim;
import org.apache.impala.thrift.TEventProcessorMetrics;
import org.apache.impala.thrift.TEventProcessorMetricsSummaryResponse;
import org.apache.impala.util.MetaStoreUtil;
@@ -175,13 +175,9 @@ public class MetastoreEventsProcessor implements ExternalEventsProcessor {
private static final Logger LOG =
LoggerFactory.getLogger(MetastoreEventsProcessor.class);
- // Use ExtendedJSONMessageFactory to deserialize the event messages.
- // ExtendedJSONMessageFactory adds additional information over JSONMessageFactory so
- // that events are compatible with Sentry
- // TODO this should be moved to JSONMessageFactory when Sentry switches to
- // JSONMessageFactory
- private static final MessageFactory messageFactory =
- ExtendedJSONMessageFactory.getInstance();
+
+ private static final MessageDeserializer MESSAGE_DESERIALIZER =
+ MetastoreShim.getMessageDeserializer();
private static MetastoreEventsProcessor instance;
@@ -632,7 +628,7 @@ public class MetastoreEventsProcessor implements ExternalEventsProcessor {
return metastoreEventFactory_;
}
- public static MessageFactory getMessageFactory() {
- return messageFactory;
+ public static MessageDeserializer getMessageDeserializer() {
+ return MESSAGE_DESERIALIZER;
}
}
diff --git a/fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java b/fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java
index b222ba6..ce75f1c 100644
--- a/fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java
+++ b/fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java
@@ -25,7 +25,6 @@ import java.util.Map;
import java.util.Set;
import org.apache.hadoop.fs.Path;
-import org.apache.hadoop.hive.common.FileUtils;
import org.apache.hadoop.hive.metastore.api.ColumnStatisticsObj;
import org.apache.hadoop.hive.metastore.api.Database;
import org.apache.hadoop.hive.metastore.api.MetaException;
@@ -40,6 +39,7 @@ import org.apache.impala.catalog.HdfsPartition.FileDescriptor;
import org.apache.impala.catalog.MetaStoreClientPool;
import org.apache.impala.catalog.MetaStoreClientPool.MetaStoreClient;
import org.apache.impala.common.Pair;
+import org.apache.impala.compat.MetastoreShim;
import org.apache.impala.service.BackendConfig;
import org.apache.impala.thrift.TBackendGflags;
import org.apache.impala.thrift.TNetworkAddress;
@@ -201,7 +201,7 @@ class DirectMetaProvider implements MetaProvider {
throw new MetaException("Unexpected number of partition values for " +
"partition " + vals + " (expected " + partitionColumnNames.size() + ")");
}
- String partName = FileUtils.makePartName(partitionColumnNames, p.getValues());
+ String partName = MetastoreShim.makePartName(partitionColumnNames, p.getValues());
if (!namesSet.contains(partName)) {
throw new MetaException("HMS returned unexpected partition " + partName +
" which was not requested. Requested: " + namesSet);
diff --git a/fe/src/main/java/org/apache/impala/hive/executor/UdfExecutor.java b/fe/src/main/java/org/apache/impala/hive/executor/UdfExecutor.java
index 50f907a..0c9560c 100644
--- a/fe/src/main/java/org/apache/impala/hive/executor/UdfExecutor.java
+++ b/fe/src/main/java/org/apache/impala/hive/executor/UdfExecutor.java
@@ -67,6 +67,8 @@ public class UdfExecutor {
private final static TBinaryProtocol.Factory PROTOCOL_FACTORY =
new TBinaryProtocol.Factory();
+ // TODO UDF is deprecated in Hive and newer implementation of built-in functions using
+ // GenericUDF interface, we should consider supporting GenericUDFs in the future
private UDF udf_;
// setup by init() and cleared by close()
private Method method_;
diff --git a/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java b/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
index a356749..3bf2434 100644
--- a/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
+++ b/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
@@ -3640,7 +3640,7 @@ public class CatalogOpExecutor {
partition.getSd().setSerdeInfo(msTbl.getSd().getSerdeInfo().deepCopy());
partition.getSd().setLocation(msTbl.getSd().getLocation() + "/" +
partName.substring(0, partName.length() - 1));
- MetastoreShim.updatePartitionStatsFast(partition, warehouse);
+ MetastoreShim.updatePartitionStatsFast(partition, msTbl, warehouse);
}
// First add_partitions and then alter_partitions the successful ones with
diff --git a/fe/src/main/java/org/apache/impala/service/DescribeResultFactory.java b/fe/src/main/java/org/apache/impala/service/DescribeResultFactory.java
index d1081c6..e530bf5 100644
--- a/fe/src/main/java/org/apache/impala/service/DescribeResultFactory.java
+++ b/fe/src/main/java/org/apache/impala/service/DescribeResultFactory.java
@@ -25,13 +25,13 @@ import java.util.Objects;
import org.apache.hadoop.hive.metastore.api.PrincipalPrivilegeSet;
import org.apache.hadoop.hive.metastore.api.PrincipalType;
import org.apache.hadoop.hive.metastore.api.PrivilegeGrantInfo;
-import org.apache.hadoop.hive.ql.metadata.formatting.MetaDataFormatUtils;
import org.apache.impala.catalog.Column;
import org.apache.impala.catalog.FeDb;
import org.apache.impala.catalog.FeTable;
import org.apache.impala.catalog.KuduColumn;
import org.apache.impala.catalog.StructField;
import org.apache.impala.catalog.StructType;
+import org.apache.impala.compat.MetastoreShim;
import org.apache.impala.thrift.TColumnValue;
import org.apache.impala.thrift.TDescribeOutputStyle;
import org.apache.impala.thrift.TDescribeResult;
@@ -216,15 +216,15 @@ public class DescribeResultFactory {
hiveTable.setTTable(msTable);
StringBuilder sb = new StringBuilder();
// First add all the columns (includes partition columns).
- sb.append(MetaDataFormatUtils.getAllColumnsInformation(msTable.getSd().getCols(),
+ sb.append(MetastoreShim.getAllColumnsInformation(msTable.getSd().getCols(),
msTable.getPartitionKeys(), true, false, true));
// Add the extended table metadata information.
- sb.append(MetaDataFormatUtils.getTableInformation(hiveTable));
+ sb.append(MetastoreShim.getTableInformation(hiveTable));
for (String line: sb.toString().split("\n")) {
// To match Hive's HiveServer2 output, split each line into multiple column
// values based on the field delimiter.
- String[] columns = line.split(MetaDataFormatUtils.FIELD_DELIM);
+ String[] columns = line.split("\t");
TResultRow resultRow = new TResultRow();
for (int i = 0; i < NUM_DESC_FORMATTED_RESULT_COLS; ++i) {
TColumnValue colVal = new TColumnValue();
diff --git a/fe/src/main/java/org/apache/impala/service/MetadataOp.java b/fe/src/main/java/org/apache/impala/service/MetadataOp.java
index 3d85aa9..0686ad99 100644
--- a/fe/src/main/java/org/apache/impala/service/MetadataOp.java
+++ b/fe/src/main/java/org/apache/impala/service/MetadataOp.java
@@ -40,6 +40,7 @@ import org.apache.impala.catalog.StructType;
import org.apache.impala.catalog.Type;
import org.apache.impala.catalog.local.InconsistentMetadataFetchException;
import org.apache.impala.common.ImpalaException;
+import org.apache.impala.compat.MetastoreShim;
import org.apache.impala.thrift.TColumn;
import org.apache.impala.thrift.TColumnValue;
import org.apache.impala.thrift.TResultRow;
@@ -64,8 +65,8 @@ public class MetadataOp {
// Static column values
private static final TColumnValue NULL_COL_VAL = new TColumnValue();
private static final TColumnValue EMPTY_COL_VAL = createTColumnValue("");
- private static final String TABLE_TYPE_TABLE = "TABLE";
- private static final String TABLE_TYPE_VIEW = "VIEW";
+ public static final String TABLE_TYPE_TABLE = "TABLE";
+ public static final String TABLE_TYPE_VIEW = "VIEW";
// Result set schema for each of the metadata operations.
private final static TResultSetMetadata GET_CATALOGS_MD = new TResultSetMetadata();
@@ -317,7 +318,10 @@ public class MetadataOp {
} else {
if (table.getMetaStoreTable() != null) {
comment = table.getMetaStoreTable().getParameters().get("comment");
- tableType = mapToInternalTableType(table.getMetaStoreTable().getTableType());
+ String tableTypeStr = table.getMetaStoreTable().getTableType() == null ?
+ null : table.getMetaStoreTable().getTableType().toUpperCase();
+ tableType = MetastoreShim.HMS_TO_IMPALA_TYPE
+ .getOrDefault(tableTypeStr, TABLE_TYPE_TABLE);
}
columns.addAll(fe.getColumns(table, columnPatternMatcher, user));
}
@@ -336,28 +340,6 @@ public class MetadataOp {
return result;
}
- private static String mapToInternalTableType(String typeStr) {
- String defaultTableType = TABLE_TYPE_TABLE;
- TableType tType;
-
- if (typeStr == null) return defaultTableType;
- try {
- tType = TableType.valueOf(typeStr.toUpperCase());
- } catch (Exception e) {
- return defaultTableType;
- }
- switch (tType) {
- case EXTERNAL_TABLE:
- case MANAGED_TABLE:
- case INDEX_TABLE:
- return TABLE_TYPE_TABLE;
- case VIRTUAL_VIEW:
- return TABLE_TYPE_VIEW;
- default:
- return defaultTableType;
- }
- }
-
/**
* Executes the GetCatalogs HiveServer2 operation and returns TResultSet.
* Hive does not have a catalog concept. It always returns an empty result set.
diff --git a/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java b/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
index bb2caed..c07d63c 100644
--- a/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
+++ b/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
@@ -56,19 +56,18 @@ import org.apache.hadoop.hive.metastore.IMetaStoreClient;
import org.apache.hadoop.hive.metastore.api.CurrentNotificationEventId;
import org.apache.hadoop.hive.metastore.api.Database;
import org.apache.hadoop.hive.metastore.api.FieldSchema;
-import org.apache.hadoop.hive.metastore.api.GetPartitionsRequest;
import org.apache.hadoop.hive.metastore.api.MetaException;
import org.apache.hadoop.hive.metastore.api.NotificationEvent;
import org.apache.hadoop.hive.metastore.api.Partition;
+import org.apache.hadoop.hive.metastore.api.SerDeInfo;
+import org.apache.hadoop.hive.metastore.api.StorageDescriptor;
import org.apache.hadoop.hive.metastore.api.PrincipalType;
-import org.apache.hadoop.hive.metastore.client.builder.DatabaseBuilder;
-import org.apache.hadoop.hive.metastore.client.builder.PartitionBuilder;
-import org.apache.hadoop.hive.metastore.client.builder.TableBuilder;
import org.apache.impala.authorization.NoopAuthorizationFactory;
import org.apache.impala.authorization.NoopAuthorizationFactory.NoopAuthorizationManager;
import org.apache.impala.catalog.CatalogException;
import org.apache.impala.catalog.CatalogServiceCatalog;
import org.apache.impala.catalog.DatabaseNotFoundException;
+import org.apache.impala.catalog.Db;
import org.apache.impala.catalog.FeCatalogUtils;
import org.apache.impala.catalog.FeFsPartition;
import org.apache.impala.catalog.HdfsFileFormat;
@@ -89,6 +88,7 @@ import org.apache.impala.catalog.events.MetastoreEventsProcessor.EventProcessorS
import org.apache.impala.common.FileSystemUtil;
import org.apache.impala.common.ImpalaException;
import org.apache.impala.common.Pair;
+import org.apache.impala.compat.MetastoreShim;
import org.apache.impala.service.CatalogOpExecutor;
import org.apache.impala.service.FeSupport;
import org.apache.impala.testutil.CatalogServiceTestCatalog;
@@ -508,11 +508,12 @@ public class MetastoreEventsProcessorTest {
String testDbParamKey = "testKey";
String testDbParamVal = "testVal";
eventsProcessor_.processEvents();
- assertFalse("Newly created test database has db should not have parameter with key "
+ Db db = catalog_.getDb(TEST_DB_NAME);
+ assertNotNull(db);
+ // db parameters should not have the test key
+ assertTrue("Newly created test database should not have parameter with key"
+ testDbParamKey,
- catalog_.getDb(TEST_DB_NAME)
- .getMetaStoreDb()
- .getParameters()
+ !db.getMetaStoreDb().isSetParameters() || !db.getMetaStoreDb().getParameters()
.containsKey(testDbParamKey));
// test change of parameters to the Database
addDatabaseParameters(testDbParamKey, testDbParamVal);
@@ -730,11 +731,9 @@ public class MetastoreEventsProcessorTest {
org.apache.hadoop.hive.metastore.api.Partition partition = null ;
if (isPartitionInsert) {
// Get the partition from metastore. This should now contain the new file.
- GetPartitionsRequest request = new GetPartitionsRequest();
- request.setDbName(dbName);
try (MetaStoreClient metaStoreClient = catalog_.getMetaStoreClient()) {
- partition = metaStoreClient.getHiveClient().getPartition(dbName,
- tblName, "p1=testPartVal");
+ partition = metaStoreClient.getHiveClient().getPartition(dbName, tblName, "p1"
+ + "=testPartVal");
}
}
// Simulate a load table
@@ -809,8 +808,8 @@ public class MetastoreEventsProcessorTest {
fakeEvent.setTableName(msTbl.getTableName());
fakeEvent.setDbName(msTbl.getDbName());
fakeEvent.setEventId(eventIdGenerator.incrementAndGet());
- fakeEvent.setMessage(MetastoreEventsProcessor.getMessageFactory()
- .buildInsertMessage(msTbl, partition, isInsertOverwrite, newFiles).toString());
+ fakeEvent.setMessage(MetastoreShim.buildInsertMessage(msTbl, partition,
+ isInsertOverwrite, newFiles).toString());
fakeEvent.setEventType("INSERT");
return fakeEvent;
@@ -1052,7 +1051,7 @@ public class MetastoreEventsProcessorTest {
// limitation : the DROP_TABLE event filtering expects that while processing events,
// the CREATION_TIME of two tables with same name won't have the same
// creation timestamp.
- sleep(2000);
+ Thread.sleep(2000);
dropTableFromImpala(TEST_DB_NAME, testTblName);
// now catalogD does not have the table entry, create the table again
createTableFromImpala(TEST_DB_NAME, testTblName, false);
@@ -1210,7 +1209,7 @@ public class MetastoreEventsProcessorTest {
// limitation : the DROP_DB event filtering expects that while processing events,
// the CREATION_TIME of two Databases with same name won't have the same
// creation timestamp.
- sleep(2000);
+ Thread.sleep(2000);
dropDatabaseCascadeFromImpala(TEST_DB_NAME);
assertNull(catalog_.getDb(TEST_DB_NAME));
createDatabaseFromImpala(TEST_DB_NAME, "second");
@@ -1475,8 +1474,9 @@ public class MetastoreEventsProcessorTest {
fakeEvent.setTableName(tblName);
fakeEvent.setDbName(dbName);
fakeEvent.setEventId(eventIdGenerator.incrementAndGet());
- fakeEvent.setMessage(MetastoreEventsProcessor.getMessageFactory()
- .buildAlterTableMessage(tableBefore, tableAfter).toString());
+ fakeEvent.setMessage(
+ MetastoreShim.buildAlterTableMessage(tableBefore, tableAfter, false, -1L)
+ .toString());
fakeEvent.setEventType("ALTER_TABLE");
return fakeEvent;
}
@@ -2110,17 +2110,16 @@ public class MetastoreEventsProcessorTest {
private void createDatabase(String dbName, Map<String, String> params)
throws TException {
- DatabaseBuilder databaseBuilder =
- new DatabaseBuilder()
- .setName(dbName)
- .setDescription("Notification test database")
- .setOwnerName("NotificationTestOwner")
- .setOwnerType(PrincipalType.USER);
+ Database database = new Database();
+ database.setName(dbName);
+ database.setDescription("Notification test database");
+ database.setOwnerName("NotificationOwner");
+ database.setOwnerType(PrincipalType.USER);
if (params != null && !params.isEmpty()) {
- databaseBuilder.setParams(params);
+ database.setParameters(params);
}
try (MetaStoreClient msClient = catalog_.getMetaStoreClient()) {
- msClient.getHiveClient().createDatabase(databaseBuilder.build());
+ msClient.getHiveClient().createDatabase(database);
}
}
@@ -2154,24 +2153,34 @@ public class MetastoreEventsProcessorTest {
private org.apache.hadoop.hive.metastore.api.Table getTestTable(String dbName,
String tblName, Map<String, String> params, boolean isPartitioned)
throws MetaException {
- TableBuilder tblBuilder =
- new TableBuilder()
- .setTableName(tblName)
- .setDbName(dbName)
- .addTableParam("tblParamKey", "tblParamValue")
- .addCol("c1", "string", "c1 description")
- .addCol("c2", "string", "c2 description")
- .setSerdeLib(HdfsFileFormat.PARQUET.serializationLib())
- .setInputFormat(HdfsFileFormat.PARQUET.inputFormat())
- .setOutputFormat(HdfsFileFormat.PARQUET.outputFormat());
- // if params are provided use them
+ org.apache.hadoop.hive.metastore.api.Table tbl =
+ new org.apache.hadoop.hive.metastore.api.Table();
+ tbl.setDbName(dbName);
+ tbl.setTableName(tblName);
+ tbl.putToParameters("tblParamKey", "tblParamValue");
+ List<FieldSchema> cols = Lists.newArrayList(
+ new FieldSchema("c1","string","c1 description"),
+ new FieldSchema("c2", "string","c2 description"));
+
+ StorageDescriptor sd = new StorageDescriptor();
+ sd.setCols(cols);
+ sd.setInputFormat(HdfsFileFormat.PARQUET.inputFormat());
+ sd.setOutputFormat(HdfsFileFormat.PARQUET.outputFormat());
+
+ SerDeInfo serDeInfo = new SerDeInfo();
+ serDeInfo.setSerializationLib(HdfsFileFormat.PARQUET.serializationLib());
+ sd.setSerdeInfo(serDeInfo);
+ tbl.setSd(sd);
+
if (params != null && !params.isEmpty()) {
- tblBuilder.setTableParams(params);
+ tbl.setParameters(params);
}
if (isPartitioned) {
- tblBuilder.addPartCol("p1", "string", "partition p1 description");
+ List<FieldSchema> pcols = Lists.newArrayList(
+ new FieldSchema("p1","string","partition p1 description"));
+ tbl.setPartitionKeys(pcols);
}
- return tblBuilder.build();
+ return tbl;
}
/**
@@ -2643,8 +2652,6 @@ public class MetastoreEventsProcessorTest {
private void alterPartitions(String tblName, List<List<String>> partValsList,
String location)
throws TException {
- GetPartitionsRequest request = new GetPartitionsRequest();
- request.setDbName(TEST_DB_NAME);
List<Partition> partitions = new ArrayList<>();
try (MetaStoreClient metaStoreClient = catalog_.getMetaStoreClient()) {
for (List<String> partVal : partValsList) {
@@ -2667,14 +2674,12 @@ public class MetastoreEventsProcessorTest {
org.apache.hadoop.hive.metastore.api.Table msTable =
msClient.getHiveClient().getTable(dbName, tblName);
for (List<String> partVals : partitionValues) {
- partitions.add(
- new PartitionBuilder()
- .fromTable(msTable)
- .setInputFormat(msTable.getSd().getInputFormat())
- .setSerdeLib(msTable.getSd().getSerdeInfo().getSerializationLib())
- .setOutputFormat(msTable.getSd().getOutputFormat())
- .setValues(partVals)
- .build());
+ Partition partition = new Partition();
+ partition.setDbName(msTable.getDbName());
+ partition.setTableName(msTable.getTableName());
+ partition.setSd(msTable.getSd().deepCopy());
+ partition.setValues(partVals);
+ partitions.add(partition);
}
}
try (MetaStoreClient metaStoreClient = catalog_.getMetaStoreClient()) {
diff --git a/fe/src/test/java/org/apache/impala/hive/executor/UdfExecutorTest.java b/fe/src/test/java/org/apache/impala/hive/executor/UdfExecutorTest.java
index 7ca7f4b..fe4a1a4 100644
--- a/fe/src/test/java/org/apache/impala/hive/executor/UdfExecutorTest.java
+++ b/fe/src/test/java/org/apache/impala/hive/executor/UdfExecutorTest.java
@@ -37,7 +37,6 @@ import org.apache.hadoop.hive.ql.udf.UDFE;
import org.apache.hadoop.hive.ql.udf.UDFExp;
import org.apache.hadoop.hive.ql.udf.UDFFindInSet;
import org.apache.hadoop.hive.ql.udf.UDFHex;
-import org.apache.hadoop.hive.ql.udf.UDFLength;
import org.apache.hadoop.hive.ql.udf.UDFLn;
import org.apache.hadoop.hive.ql.udf.UDFLog;
import org.apache.hadoop.hive.ql.udf.UDFLog10;
@@ -450,7 +449,7 @@ public class UdfExecutorTest {
throws ImpalaException, MalformedURLException, TException {
TestHiveUdf(UDFAscii.class, createInt('1'), "123");
TestHiveUdf(UDFFindInSet.class, createInt(2), "31", "12,31,23");
- TestHiveUdf(UDFLength.class, createInt(5), "Hello");
+ // UDFLength was moved to GenericUDFLength in Hive 2.3 (HIVE-15979)
TestHiveUdf(UDFRepeat.class, createText("abcabc"), "abc", createInt(2));
TestHiveUdf(UDFReverse.class, createText("cba"), "abc");
TestHiveUdf(UDFSpace.class, createText(" "), createInt(4));
diff --git a/fe/src/test/java/org/apache/impala/testutil/EmbeddedMetastoreClientPool.java b/fe/src/test/java/org/apache/impala/testutil/EmbeddedMetastoreClientPool.java
index 7e68d8e..a606d92 100644
--- a/fe/src/test/java/org/apache/impala/testutil/EmbeddedMetastoreClientPool.java
+++ b/fe/src/test/java/org/apache/impala/testutil/EmbeddedMetastoreClientPool.java
@@ -18,6 +18,7 @@
package org.apache.impala.testutil;
import org.apache.commons.io.FileUtils;
+import org.apache.hadoop.hive.conf.HiveConf.ConfVars;
import org.apache.impala.catalog.MetaStoreClientPool;
import org.apache.hadoop.hive.conf.HiveConf;
import org.apache.log4j.Logger;
@@ -46,6 +47,15 @@ public class EmbeddedMetastoreClientPool extends MetaStoreClientPool {
derbyDataStorePath_ = dbStorePath;
}
+ // Embedded HMS instantiates partition expression proxy which by default brings in a
+ // lot of runtime dependencies from hive-exec. Since we don't depend on this, we
+ // should use a DefaultPartitionExpressionProxy which is a no-op implementation of
+ // PartitionExpressionProxy interface. It throws UnsupportedOperationException on its
+ // APIs so we will find them in tests in case we start using these features when
+ // using embedded HMS
+ private static final String DEFAULT_PARTITION_EXPRESSION_PROXY_CLASS = "org.apache"
+ + ".hadoop.hive.metastore.DefaultPartitionExpressionProxy";
+
/**
* Generates the HiveConf required to connect to an embedded metastore backed by
* derby DB.
@@ -57,13 +67,17 @@ public class EmbeddedMetastoreClientPool extends MetaStoreClientPool {
// hive.metastore.uris - empty
// javax.jdo.option.ConnectionDriverName - org.apache.derby.jdbc.EmbeddedDriver
// javax.jdo.option.ConnectionURL - jdbc:derby:;databaseName=<path>;create=true"
- conf.set(HiveConf.ConfVars.METASTOREURIS.toString(), "");
- conf.set(HiveConf.ConfVars.METASTORE_CONNECTION_DRIVER.toString(),
+ conf.set(ConfVars.METASTOREURIS.varname, "");
+ conf.set(ConfVars.METASTORE_CONNECTION_DRIVER.varname,
"org.apache.derby.jdbc.EmbeddedDriver");
- conf.setBoolean(HiveConf.ConfVars.METASTORE_SCHEMA_VERIFICATION.toString(), false);
- conf.setBoolean(HiveConf.ConfVars.METASTORE_AUTO_CREATE_ALL.toString(), true);
- conf.set(HiveConf.ConfVars.METASTORECONNECTURLKEY.toString(),
+ conf.setBoolean(ConfVars.METASTORE_SCHEMA_VERIFICATION.varname, false);
+ conf.setBoolean(ConfVars.METASTORE_AUTO_CREATE_ALL.varname, true);
+ conf.set(ConfVars.METASTORECONNECTURLKEY.varname,
String.format(CONNECTION_URL_TEMPLATE, dbStorePath.toString()));
+ conf.set(ConfVars.METASTORE_EXPRESSION_PROXY_CLASS.varname,
+ DEFAULT_PARTITION_EXPRESSION_PROXY_CLASS);
+ // Disabling notification event listeners
+ conf.set(ConfVars.METASTORE_TRANSACTIONAL_EVENT_LISTENERS.varname, "");
return conf;
}
diff --git a/impala-parent/pom.xml b/impala-parent/pom.xml
index 1f8d878..a6bbbd9 100644
--- a/impala-parent/pom.xml
+++ b/impala-parent/pom.xml
@@ -37,6 +37,7 @@ under the License.
<testExecutionMode>reduced</testExecutionMode>
<hadoop.version>${env.IMPALA_HADOOP_VERSION}</hadoop.version>
<hive.version>${env.IMPALA_HIVE_VERSION}</hive.version>
+ <hive.storage.api.version>2.3.0.${env.IMPALA_HIVE_VERSION}</hive.storage.api.version>
<hive.major.version>${env.IMPALA_HIVE_MAJOR_VERSION}</hive.major.version>
<ranger.version>${env.IMPALA_RANGER_VERSION}</ranger.version>
<postgres.jdbc.version>${env.IMPALA_POSTGRES_JDBC_DRIVER_VERSION}</postgres.jdbc.version>
@@ -134,6 +135,14 @@ under the License.
<enabled>false</enabled>
</snapshots>
</repository>
+ <repository>
+ <id>hwx.public.repo</id>
+ <url>http://nexus-private.hortonworks.com/nexus/content/groups/public</url>
+ <name>Hortonworks public repository</name>
+ <snapshots>
+ <enabled>true</enabled>
+ </snapshots>
+ </repository>
</repositories>
<pluginRepositories>
diff --git a/shaded-deps/.gitignore b/shaded-deps/.gitignore
new file mode 100644
index 0000000..916e17c
--- /dev/null
+++ b/shaded-deps/.gitignore
@@ -0,0 +1 @@
+dependency-reduced-pom.xml
diff --git a/shaded-deps/CMakeLists.txt b/shaded-deps/CMakeLists.txt
new file mode 100644
index 0000000..73d353c
--- /dev/null
+++ b/shaded-deps/CMakeLists.txt
@@ -0,0 +1,20 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+add_custom_target(shaded-deps ALL DEPENDS impala-parent
+ COMMAND $ENV{IMPALA_HOME}/bin/mvn-quiet.sh -B install -DskipTests
+)
diff --git a/shaded-deps/pom.xml b/shaded-deps/pom.xml
new file mode 100644
index 0000000..6aad3c5
--- /dev/null
+++ b/shaded-deps/pom.xml
@@ -0,0 +1,108 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
+ http://maven.apache.org/xsd/maven-4.0.0.xsd">
+
+ <!-- This pom creates a jar which excludes most of the hive-exec classes to
+reduce the dependency footprint of hive in Impala. It uses maven-shade-plugin
+to include only those classes which are needed by fe to compile and run in a
+hive-3 environment. Additionally, it relocates some of the transitive dependencies
+coming from hive-exec so that it does not conflict with Impala's version of
+the same dependencies
+ -->
+ <parent>
+ <groupId>org.apache.impala</groupId>
+ <artifactId>impala-parent</artifactId>
+ <version>0.1-SNAPSHOT</version>
+ <relativePath>../impala-parent/pom.xml</relativePath>
+ </parent>
+ <modelVersion>4.0.0</modelVersion>
+ <groupId>org.apache.impala</groupId>
+ <artifactId>impala-minimal-hive-exec</artifactId>
+ <packaging>jar</packaging>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.hive</groupId>
+ <artifactId>hive-exec</artifactId>
+ <version>${hive.version}</version>
+ </dependency>
+ </dependencies>
+ <build>
+ <plugins>
+ <plugin>
+ <artifactId>maven-shade-plugin</artifactId>
+ <version>3.2.1</version>
+ <configuration>
+ <artifactSet>
+ <includes>
+ <include>org.apache.hive:hive-exec</include>
+ <include>org.apache.hadoop:hadoop-mapreduce-client</include>
+ </includes>
+ </artifactSet>
+ <relocations>
+ <relocation>
+ <pattern>com.google</pattern>
+ <shadedPattern>hiveexec.com.google</shadedPattern>
+ </relocation>
+ <relocation>
+ <pattern>org.joda.time</pattern>
+ <shadedPattern>hiveexec.org.joda.time</shadedPattern>
+ </relocation>
+ </relocations>
+ <filters>
+ <filter>
+ <artifact>org.apache.hive:hive-exec</artifact>
+ <includes>
+ <include>org/apache/hadoop/hive/conf/**/*</include>
+ <include>org/apache/hadoop/hive/common/FileUtils*</include>
+ <!-- Needed to support describe formatted command compat with Hive -->
+ <include>org/apache/hadoop/hive/ql/metadata/**/*</include>
+ <include>org/apache/hadoop/hive/ql/parse/SemanticException.class</include>
+ <!-- Needed to support Hive udfs -->
+ <include>org/apache/hadoop/hive/ql/exec/*UDF*</include>
+ <include>org/apache/hadoop/hive/ql/exec/FunctionUtils*</include>
+ <include>org/apache/hadoop/hive/ql/parse/HiveLexer*</include>
+ <include>org/apache/hadoop/hive/ql/udf/**/*</include>
+ <!-- Many of the UDFs are annotated with their vectorized counter-parts.
+ Including them makes sure that we don't break -->
+ <include>org/apache/hadoop/hive/ql/exec/vector/expressions/**/*</include>
+ <include>org/apache/hive/common/HiveVersionAnnotation.class</include>
+ <include>org/apache/hive/common/HiveCompat*</include>
+ <include>org/apache/hive/common/util/**</include>
+ <include>org/apache/hive/service/rpc/thrift/**</include>
+ <include>org/apache/hadoop/hive/serde/**</include>
+ <include>org/apache/hadoop/hive/serde2/**</include>
+ <include>org/apache/hive/service/rpc/thrift/**</include>
+ <include>org/apache/hive/common/HiveVersionAnnotation.class</include>
+ <include>com/google/**</include>
+ </includes>
+ </filter>
+ </filters>
+ </configuration>
+ <executions>
+ <execution>
+ <phase>package</phase>
+ <goals>
+ <goal>shade</goal>
+ </goals>
+ </execution>
+ </executions>
+ </plugin>
+ </plugins>
+ </build>
+</project>
diff --git a/testdata/bin/run-hive-server.sh b/testdata/bin/run-hive-server.sh
index 6bdaaee..473a654 100755
--- a/testdata/bin/run-hive-server.sh
+++ b/testdata/bin/run-hive-server.sh
@@ -63,7 +63,8 @@ done
# Kill for a clean start.
${CLUSTER_BIN}/kill-hive-server.sh &> /dev/null
-export HIVE_METASTORE_HADOOP_OPTS="-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=30010"
+export HIVE_METASTORE_HADOOP_OPTS="-Xdebug -Xrunjdwp:transport=dt_socket,server=y,\
+suspend=n,address=30010"
# If this is CDP Hive we need to manually add the sentry jars in the classpath since
# CDH Hive metastore scripts do not do so. This is currently to make sure that we can run
@@ -91,7 +92,7 @@ ${CLUSTER_BIN}/wait-for-metastore.py --transport=${METASTORE_TRANSPORT}
if [ ${ONLY_METASTORE} -eq 0 ]; then
# For Hive 3, we use Tez for execution. We have to add it to the HS2 classpath.
- if $USE_CDP_HIVE; then
+ if ${USE_CDP_HIVE} ; then
export HADOOP_CLASSPATH=${HADOOP_CLASSPATH}:${TEZ_HOME}/*
# This is a little hacky, but Tez bundles a bunch of junk into lib/, such
# as extra copies of the hadoop libraries, etc, and we want to avoid conflicts.
diff --git a/tests/custom_cluster/test_permanent_udfs.py b/tests/custom_cluster/test_permanent_udfs.py
index 5d28a2c..8e8819a 100644
--- a/tests/custom_cluster/test_permanent_udfs.py
+++ b/tests/custom_cluster/test_permanent_udfs.py
@@ -505,9 +505,15 @@ class TestUdfPersistence(CustomClusterTestSuite):
('udfbin', 'org.apache.hadoop.hive.ql.udf.UDFBin'),
('udfhex', 'org.apache.hadoop.hive.ql.udf.UDFHex'),
('udfconv', 'org.apache.hadoop.hive.ql.udf.UDFConv'),
+ # TODO UDFHour was moved from UDF to GenericUDF in Hive 3
+ # This test will fail when running against HMS-3 unless we add
+ # support for GenericUDFs to handle such cases
('udfhour', 'org.apache.hadoop.hive.ql.udf.UDFHour'),
('udflike', 'org.apache.hadoop.hive.ql.udf.UDFLike'),
('udfsign', 'org.apache.hadoop.hive.ql.udf.UDFSign'),
+ # TODO UDFYear moved to GenericUDF in Hive 3
+ # This test will fail when running against HMS-3 unless we add
+ # support for GenericUDFs
('udfyear', 'org.apache.hadoop.hive.ql.udf.UDFYear'),
('udfascii','org.apache.hadoop.hive.ql.udf.UDFAscii')
]