You are viewing a plain text version of this content. The canonical link for it is here.

Posted to reviews@impala.apache.org by "Vihang Karajgaonkar (Code Review)" <ge...@cloudera.org> on 2019/04/13 00:51:57 UTC

[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0

Vihang Karajgaonkar has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13005


Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
......................................................................

IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0

This change upgrades the hive dependencies of Impala to use Hive 3.1.0
based binaries. Most of the changes in this patch are based off patches
provided by Todd (links available in JIRA).

Upgrading the dependencies allows us to work with both Hive 3.1.0 and
Hive 2.1.0 in the same code line. In order to do this, the patch
trims down a lot of unnecessary hive dependencies of the front end code
by creating a shaded-deps module. The pom.xml of shaded-deps includes
only the files from Hive source which Impala depends for compilation.

Additionally, it also uses a custom build of Hive which is based of
Hive 3.1.0. This custom build includes patches for HIVE-21596 and
HIVE-21586 which are needed by Impala so that it can compile against
Hive-3 libraries and be able to talk to HMS-2.x metastore. Once these
patches are merged we can get rid of this custom build and rely on more
official sources of Hive builds.

The patch also changes impala-config.sh so that it always downloads the
Hive-3 libraries from the toolchain. The code is always built using
Hive-3 jars. However, based on the value of USE_CDP_HIVE, the
minicluster is deployed using Hive-3 or Hive-2 binaries. Since Impala
implements HiveServer2's TCLIService.thrift interface, it requires us to
use the existing mechanism of copying the hive-2/api TCLIService.thrift to
hive-3/api. It also adds a few environment variables which point to the
metastore's thrift file and the CDH Hive version.

Testing:
1. Code compiles and runs against both HMS-3 and HMS-2

Notes:
Testing is still a WIP. Will trigger a full-suite of tests for both when
USE_CDP_HIVE is true and USE_CDP_HIVE is false. Will update the patch as
and when the issues are found in the tests.

Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436
---
M CMakeLists.txt
M README.md
M bin/bootstrap_toolchain.py
M bin/impala-config.sh
M bin/set-classpath.sh
M common/thrift/.gitignore
M common/thrift/CMakeLists.txt
M fe/CMakeLists.txt
M fe/pom.xml
M fe/src/main/java/org/apache/impala/analysis/StringLiteral.java
M fe/src/main/java/org/apache/impala/catalog/FeHBaseTable.java
M fe/src/main/java/org/apache/impala/catalog/TableLoader.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java
M fe/src/main/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/main/java/org/apache/impala/hive/executor/UdfExecutor.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/DescribeResultFactory.java
M fe/src/main/java/org/apache/impala/service/MetadataOp.java
M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
M fe/src/test/java/org/apache/impala/hive/executor/UdfExecutorTest.java
M impala-parent/pom.xml
A shaded-deps/.gitignore
A shaded-deps/CMakeLists.txt
A shaded-deps/pom.xml
26 files changed, 442 insertions(+), 266 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/05/13005/2
-- 
To view, visit http://gerrit.cloudera.org:8080/13005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436
Gerrit-Change-Number: 13005
Gerrit-PatchSet: 2
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>

[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13005 )

Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
......................................................................


Patch Set 10:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/2938/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/13005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436
Gerrit-Change-Number: 13005
Gerrit-PatchSet: 10
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Sudhanshu Arora <su...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 26 Apr 2019 19:23:07 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0

Posted by "Todd Lipcon (Code Review)" <ge...@cloudera.org>.

Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/13005 )

Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
......................................................................


Patch Set 22:

(3 comments)

http://gerrit.cloudera.org:8080/#/c/13005/22//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/13005/22//COMMIT_MSG@21
PS22, Line 21: For hive-2 build
             : path, no changes are done with respect to hive dependencies to minimize
             : the risk of destabilizing the master branch on the default build option
             : of using Hive-2.
> IIUC, it means that we are keeping all the dependencies in Hive2 but reduci
I think the thinking is that we never wanted to pull all the transitive dependencies of Hive onto the Impala classpath. But, in existing releases we did this. UDF authors in particular might have written UDFs that implicitly depended on these things being on the classpath. So, removing them is somewhat of an incompatible change.

If we assume that upgrading to Hive 3 is a "clean slate" (ie we'd get a new set of transitive deps anyway) we should take the opportunity to minimize the classpath polution. Ideally, we'd run UDFs with a restricted classloader to only expose a small subset of UDF APIs but that's a complex project that we don't have appetite for.


http://gerrit.cloudera.org:8080/#/c/13005/22//COMMIT_MSG@34
PS22, Line 34: HMS transaction support
             : since HMS3 create transactional tables by default
> Does it create full transactional table or "insert_only" by default?
based on my initial testing it actually doesn't look like it's doing transactional by default. There are the following settings that change the default:

HiveConf.java:
    HIVE_CREATE_TABLES_AS_INSERT_ONLY("hive.create.as.insert.only", false,
        "Whether the eligible tables should be created as ACID insert-only by default. Does \n" +
        "not apply to external tables, the ones using storage handlers, etc."),

MetastoreConf.java:
    CREATE_TABLES_AS_ACID("metastore.create.as.acid", "hive.create.as.acid", false,
        "Whether the eligible tables should be created as full ACID by default. Does \n" +
            "not apply to external tables, the ones using storage handlers, etc."),

I think downstream distros of Hive (eg HDP3) may change these defaults, but in our minicluster build at the moment both are false.


http://gerrit.cloudera.org:8080/#/c/13005/22/fe/src/main/java/org/apache/impala/service/DescribeResultFactory.java
File fe/src/main/java/org/apache/impala/service/DescribeResultFactory.java:

http://gerrit.cloudera.org:8080/#/c/13005/22/fe/src/main/java/org/apache/impala/service/DescribeResultFactory.java@227
PS22, Line 227:       String[] columns = line.split("\t");
> Nit: Add this constant into MetastoreShim or if this is the only usage then
yea I'd suggested not shimming it. I don't have a strong opinion of whether to introduce a local constant. IMO seeing as tab is a pretty obvious delimiter to split on, I don't mind just having the direct value, but I know some people like adding constants for suc things



-- 
To view, visit http://gerrit.cloudera.org:8080/13005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436
Gerrit-Change-Number: 13005
Gerrit-PatchSet: 22
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Sudhanshu Arora <su...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 30 Apr 2019 21:06:13 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0

Posted by "Vihang Karajgaonkar (Code Review)" <ge...@cloudera.org>.

Vihang Karajgaonkar has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/13005 )

Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
......................................................................

IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0

This change upgrades the hive dependencies of Impala to use Hive 3.1.0
based binaries. Most of the changes in this patch are based off patches
provided by Todd (links available in JIRA).

Upgrading the dependencies allows us to work with both Hive 3.1.0 and
Hive 2.1.0 in the same code line. In order to do this, the patch
trims down a lot of unnecessary hive dependencies of the front end code
by creating a shaded-deps module. The pom.xml of shaded-deps includes
only the files from Hive source which Impala depends for compilation.

Additionally, it also uses a custom build of Hive which is based of
Hive 3.1.0. This custom build includes patches for HIVE-21596 and
HIVE-21586 which are needed by Impala so that it can compile against
Hive-3 libraries and be able to talk to HMS-2.x metastore. Once these
patches are merged we can get rid of this custom build and rely on more
official sources of Hive builds.

The patch also changes impala-config.sh so that it always downloads the
Hive-3 libraries from the toolchain. The code is always built using
Hive-3 jars. However, based on the value of USE_CDP_HIVE, the
minicluster is deployed using Hive-3 or Hive-2 binaries. Since Impala
implements HiveServer2's TCLIService.thrift interface, it requires us to
use the existing mechanism of copying the hive-2/api TCLIService.thrift
to
hive-3/api. It also adds a few environment variables which point to the
metastore's thrift file and the CDH Hive version.

Testing:
1. Code compiles and runs against both HMS-3 and HMS-2
2. Ran full-suite of tests using the private jenkins job against HMS-2
3. Running full-tests against HMS-3 will need more work like supporting
Tez in the mini-cluster (for dataloading) and HMS transaction support
since HMS3 create transactional tables by default.

Notes:
1. Patch uses a custom build of Hive to be deployed in mini-cluster. This
build has the fixes for HIVE-21596
2. The Hive 3.1 does not include some of the patches like
HIVE-21077. Some of the recent code in Impala depends on that patch to
exist in hive. We will need to port HIVE-21077 to Hive 3.1.0 in order to
compile against Hive 3.1.0 jars. I will continue working on getting
this merged along with HIVE-21586

Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436
---
M CMakeLists.txt
M README.md
M bin/bootstrap_toolchain.py
M bin/impala-config.sh
M bin/set-classpath.sh
M common/thrift/.gitignore
M common/thrift/CMakeLists.txt
M fe/CMakeLists.txt
M fe/pom.xml
M fe/src/main/java/org/apache/impala/analysis/StringLiteral.java
M fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java
M fe/src/main/java/org/apache/impala/catalog/FeHBaseTable.java
M fe/src/main/java/org/apache/impala/catalog/MetaStoreClientPool.java
M fe/src/main/java/org/apache/impala/catalog/TableLoader.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java
M fe/src/main/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/main/java/org/apache/impala/hive/executor/UdfExecutor.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/DescribeResultFactory.java
M fe/src/main/java/org/apache/impala/service/MetadataOp.java
A fe/src/main/java/org/apache/impala/util/MetadataFormatUtils.java
M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
M fe/src/test/java/org/apache/impala/hive/executor/UdfExecutorTest.java
M fe/src/test/java/org/apache/impala/testutil/EmbeddedMetastoreClientPool.java
M impala-parent/pom.xml
A shaded-deps/.gitignore
A shaded-deps/CMakeLists.txt
A shaded-deps/dependency-reduced-pom.xml
A shaded-deps/pom.xml
M testdata/bin/run-hive-server.sh
M tests/custom_cluster/test_permanent_udfs.py
33 files changed, 1,224 insertions(+), 258 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/05/13005/5
-- 
To view, visit http://gerrit.cloudera.org:8080/13005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436
Gerrit-Change-Number: 13005
Gerrit-PatchSet: 5
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Sudhanshu Arora <su...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>

[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0

Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.

Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/13005 )

Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
......................................................................


Patch Set 23: Code-Review+2

Fix looks correct to me, so I'm going ahead with +2


-- 
To view, visit http://gerrit.cloudera.org:8080/13005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436
Gerrit-Change-Number: 13005
Gerrit-PatchSet: 23
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Sudhanshu Arora <su...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 30 Apr 2019 22:19:22 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13005 )

Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
......................................................................


Patch Set 14:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/2979/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/13005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436
Gerrit-Change-Number: 13005
Gerrit-PatchSet: 14
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Sudhanshu Arora <su...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 29 Apr 2019 22:51:29 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13005 )

Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
......................................................................


Patch Set 3:

(4 comments)

http://gerrit.cloudera.org:8080/#/c/13005/3/bin/impala-config.sh
File bin/impala-config.sh:

http://gerrit.cloudera.org:8080/#/c/13005/3/bin/impala-config.sh@199
PS3, Line 199: # When USE_CDP_HIVE is set we use the latest hive version available to deply in minicluster
line too long (91 > 90)


http://gerrit.cloudera.org:8080/#/c/13005/3/bin/impala-config.sh@203
PS3, Line 203:   # TODO(Vihang) we should repackage the tarballs so that the src and binaries are extracted
line too long (92 > 90)


http://gerrit.cloudera.org:8080/#/c/13005/3/bin/impala-config.sh@212
PS3, Line 212:   export HIVE_HOME="$IMPALA_TOOLCHAIN/cdh_components-${CDH_BUILD_NUMBER}/hive-${MINICLUSTER_HIVE_VERSION}"
line too long (106 > 90)


http://gerrit.cloudera.org:8080/#/c/13005/3/bin/impala-config.sh@546
PS3, Line 546: export HIVE_METASTORE_THRIFT_DIR=$CDP_COMPONENTS_HOME/apache-hive-${IMPALA_HIVE_VERSION}-bin/src/standalone-metastore/src/main/thrift
line too long (133 > 90)



-- 
To view, visit http://gerrit.cloudera.org:8080/13005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436
Gerrit-Change-Number: 13005
Gerrit-PatchSet: 3
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Tue, 16 Apr 2019 17:54:21 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0

Posted by "Vihang Karajgaonkar (Code Review)" <ge...@cloudera.org>.

Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/13005 )

Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
......................................................................


Patch Set 8:

> (3 comments)
 > 
 > Before more work goes into this, this is changing the Hive major
 > version number and there should be a discussion on dev@ about this
 > approach and how it fits into Impala releases.

Just sent a note the dev list. Thanks for the suggestion.


-- 
To view, visit http://gerrit.cloudera.org:8080/13005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436
Gerrit-Change-Number: 13005
Gerrit-PatchSet: 8
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Sudhanshu Arora <su...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 25 Apr 2019 22:02:29 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0

Posted by "Vihang Karajgaonkar (Code Review)" <ge...@cloudera.org>.

Vihang Karajgaonkar has uploaded a new patch set (#7). ( http://gerrit.cloudera.org:8080/13005 )

Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
......................................................................

IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0

This change upgrades the hive dependencies of Impala to use Hive 3.1.0
based binaries. Most of the changes in this patch are based off patches
provided by Todd (links available in JIRA).

Upgrading the dependencies allows us to work with both Hive 3.1.0 and
Hive 2.1.0 in the same code line. In order to do this, the patch
trims down a lot of unnecessary hive dependencies of the front end code
by creating a shaded-deps module. The pom.xml of shaded-deps includes
only the files from Hive source which Impala depends for compilation.

Additionally, it also uses a custom build of Hive which is based of
Hive 3.1.0. This custom build includes patches for HIVE-21596 and
HIVE-21586 which are needed by Impala so that it can compile against
Hive-3 libraries and be able to talk to HMS-2.x metastore. Once these
patches are merged we can get rid of this custom build and rely on more
official sources of Hive builds.

The patch also changes impala-config.sh so that it always downloads the
Hive-3 libraries from the toolchain. The code is always built using
Hive-3 jars. However, based on the value of USE_CDP_HIVE, the
minicluster is deployed using Hive-3 or Hive-2 binaries. Since Impala
implements HiveServer2's TCLIService.thrift interface, it requires us to
use the existing mechanism of copying the hive-2/api TCLIService.thrift
to
hive-3/api. It also adds a few environment variables which point to the
metastore's thrift file and the CDH Hive version.

Testing:
1. Code compiles and runs against both HMS-3 and HMS-2
2. Ran full-suite of tests using the private jenkins job against HMS-2
3. Running full-tests against HMS-3 will need more work like supporting
Tez in the mini-cluster (for dataloading) and HMS transaction support
since HMS3 create transactional tables by default.

Notes:
1. Patch uses a custom build of Hive to be deployed in mini-cluster. This
build has the fixes for HIVE-21596
2. The Hive 3.1 does not include some of the patches like
HIVE-21077. Some of the recent code in Impala depends on that patch to
exist in hive. We will need to port HIVE-21077 to Hive 3.1.0 in order to
compile against Hive 3.1.0 jars. I will continue working on getting
this merged along with HIVE-21586

Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436
---
M CMakeLists.txt
M README.md
M bin/bootstrap_toolchain.py
M bin/impala-config.sh
M bin/set-classpath.sh
M common/thrift/.gitignore
M common/thrift/CMakeLists.txt
M fe/CMakeLists.txt
M fe/pom.xml
M fe/src/main/java/org/apache/impala/analysis/StringLiteral.java
M fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java
M fe/src/main/java/org/apache/impala/catalog/FeHBaseTable.java
M fe/src/main/java/org/apache/impala/catalog/MetaStoreClientPool.java
M fe/src/main/java/org/apache/impala/catalog/TableLoader.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java
M fe/src/main/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/main/java/org/apache/impala/hive/executor/UdfExecutor.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/DescribeResultFactory.java
M fe/src/main/java/org/apache/impala/service/MetadataOp.java
A fe/src/main/java/org/apache/impala/util/MetadataFormatUtils.java
M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
M fe/src/test/java/org/apache/impala/hive/executor/UdfExecutorTest.java
M fe/src/test/java/org/apache/impala/testutil/EmbeddedMetastoreClientPool.java
M impala-parent/pom.xml
A shaded-deps/.gitignore
A shaded-deps/CMakeLists.txt
A shaded-deps/pom.xml
M testdata/bin/run-hive-server.sh
M tests/authorization/test_owner_privileges.py
M tests/common/sentry_cache_test_suite.py
M tests/custom_cluster/test_permanent_udfs.py
34 files changed, 1,208 insertions(+), 294 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/05/13005/7
-- 
To view, visit http://gerrit.cloudera.org:8080/13005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436
Gerrit-Change-Number: 13005
Gerrit-PatchSet: 7
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Sudhanshu Arora <su...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>

[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13005 )

Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
......................................................................


Patch Set 5:

Build Failed 

https://jenkins.impala.io/job/gerrit-code-review-checks/2874/ : Initial code review checks failed. See linked job for details on the failure.


-- 
To view, visit http://gerrit.cloudera.org:8080/13005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436
Gerrit-Change-Number: 13005
Gerrit-PatchSet: 5
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Sudhanshu Arora <su...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 24 Apr 2019 02:17:47 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0

Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.

Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/13005 )

Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
......................................................................


Patch Set 3:

(6 comments)

http://gerrit.cloudera.org:8080/#/c/13005/3//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/13005/3//COMMIT_MSG@31
PS3, Line 31: t to
nit: wrap at 72 chars in the commit message


http://gerrit.cloudera.org:8080/#/c/13005/3/fe/src/main/java/org/apache/impala/analysis/StringLiteral.java
File fe/src/main/java/org/apache/impala/analysis/StringLiteral.java:

http://gerrit.cloudera.org:8080/#/c/13005/3/fe/src/main/java/org/apache/impala/analysis/StringLiteral.java@96
PS3, Line 96:   /**
            :    * Copied from Apache Hive's BaseSemanticAnalyzer. This method has not changed
            :    * since last several years so hoping that it is fairly stable by now. Sourcing it from
            :    * the Hive's code without copying brings along with it a lot of other unnecessary
            :    * dependencies
            :    * @param b
            :    * @return
            :    */
Can you move the parts copied from Hive to a separate file/directory? It will be easier to remove them later if they are at a single place and not spread around at call sites.


http://gerrit.cloudera.org:8080/#/c/13005/3/fe/src/main/java/org/apache/impala/catalog/FeHBaseTable.java
File fe/src/main/java/org/apache/impala/catalog/FeHBaseTable.java:

http://gerrit.cloudera.org:8080/#/c/13005/3/fe/src/main/java/org/apache/impala/catalog/FeHBaseTable.java@122
PS3, Line 122:     //TODO (Vihang) this pulls in hive-hbase-handler dependency which brings all the
             :     // other stuff we don't need. Perhaps we just need to copy the constants and mark
             :     // them public API in Hive source code
             :     private static final String HBASE_COLUMNS_MAPPING = "hbase.columns.mapping";
             :     private static final String HBASE_TABLE_DEFAULT_STORAGE_TYPE = "hbase.table.default"
             :         + ".storage.type";
             :     private static final String HBASE_KEY_COL = ":key";
             :     private static final String HBASE_TABLE_NAME = "hbase.table.name";
Constants copied from Hive 3 could be copied to a common place.


http://gerrit.cloudera.org:8080/#/c/13005/3/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java:

http://gerrit.cloudera.org:8080/#/c/13005/3/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@189
PS3, Line 189: `
nit: mistyping?


http://gerrit.cloudera.org:8080/#/c/13005/3/fe/src/main/java/org/apache/impala/service/DescribeResultFactory.java
File fe/src/main/java/org/apache/impala/service/DescribeResultFactory.java:

http://gerrit.cloudera.org:8080/#/c/13005/3/fe/src/main/java/org/apache/impala/service/DescribeResultFactory.java@29
PS3, Line 29:   
nit: wrong indentation


http://gerrit.cloudera.org:8080/#/c/13005/3/fe/src/main/java/org/apache/impala/service/MetadataOp.java
File fe/src/main/java/org/apache/impala/service/MetadataOp.java:

http://gerrit.cloudera.org:8080/#/c/13005/3/fe/src/main/java/org/apache/impala/service/MetadataOp.java@345
PS3, Line 345:       case VIRTUAL_VIEW:
             :       case MATERIALIZED_VIEW:
             :         return TABLE_TYPE_VIEW;
This returns TABLE_TYPE_VIEW even for EXTERNAL_TABLE and MANAGED_TABLE - is this intentional to avoid changing any table?



-- 
To view, visit http://gerrit.cloudera.org:8080/13005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436
Gerrit-Change-Number: 13005
Gerrit-PatchSet: 3
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Sudhanshu Arora <su...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Wed, 17 Apr 2019 19:50:32 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13005 )

Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
......................................................................


Patch Set 10: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4088/


-- 
To view, visit http://gerrit.cloudera.org:8080/13005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436
Gerrit-Change-Number: 13005
Gerrit-PatchSet: 10
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Sudhanshu Arora <su...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 26 Apr 2019 23:29:21 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13005 )

Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
......................................................................


Patch Set 11:

(26 comments)

http://gerrit.cloudera.org:8080/#/c/13005/11/bin/impala-config.sh
File bin/impala-config.sh:

http://gerrit.cloudera.org:8080/#/c/13005/11/bin/impala-config.sh@181
PS11, Line 181:   # When USE_CDP_HIVE is set we use the latest hive version available to deply in minicluster
line too long (93 > 90)


http://gerrit.cloudera.org:8080/#/c/13005/11/bin/impala-config.sh@185
PS11, Line 185:   export HIVE_METASTORE_THRIFT_DIR=$CDP_COMPONENTS_HOME/apache-hive-${IMPALA_HIVE_VERSION}-src/standalone-metastore/src/main/thrift
line too long (131 > 90)


http://gerrit.cloudera.org:8080/#/c/13005/11/bin/impala-config.sh@204
PS11, Line 204:   export HIVE_HOME="$IMPALA_TOOLCHAIN/cdh_components-${CDH_BUILD_NUMBER}/hive-${IMPALA_HIVE_VERSION}"
line too long (101 > 90)


http://gerrit.cloudera.org:8080/#/c/13005/11/bin/impala-config.sh@205
PS11, Line 205:   export HIVE_METASTORE_THRIFT_DIR=$CDH_COMPONENTS_HOME/hive-${IMPALA_HIVE_VERSION}/src/metastore/if
line too long (100 > 90)


http://gerrit.cloudera.org:8080/#/c/13005/11/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java:

http://gerrit.cloudera.org:8080/#/c/13005/11/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1073
PS11, Line 1073:                 
line has trailing whitespace


http://gerrit.cloudera.org:8080/#/c/13005/11/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1169
PS11, Line 1169:               
line has trailing whitespace


http://gerrit.cloudera.org:8080/#/c/13005/11/fe/src/main/java/org/apache/impala/service/MetadataOp.java
File fe/src/main/java/org/apache/impala/service/MetadataOp.java:

http://gerrit.cloudera.org:8080/#/c/13005/11/fe/src/main/java/org/apache/impala/service/MetadataOp.java@319
PS11, Line 319:                   MetastoreShim.mapToInternalTableType(table.getMetaStoreTable().getTableType());
line too long (97 > 90)


http://gerrit.cloudera.org:8080/#/c/13005/11/testdata/bin/run-hive-server.sh
File testdata/bin/run-hive-server.sh:

http://gerrit.cloudera.org:8080/#/c/13005/11/testdata/bin/run-hive-server.sh@66
PS11, Line 66: export HIVE_METASTORE_HADOOP_OPTS="-verbose:class -Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=30010"
line too long (121 > 90)


http://gerrit.cloudera.org:8080/#/c/13005/11/tests/authorization/test_owner_privileges.py
File tests/authorization/test_owner_privileges.py:

http://gerrit.cloudera.org:8080/#/c/13005/11/tests/authorization/test_owner_privileges.py@103
PS11, Line 103: )
flake8: E501 line too long (91 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/13005/11/tests/authorization/test_owner_privileges.py@179
PS11, Line 179: =
flake8: E501 line too long (93 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/13005/11/tests/authorization/test_owner_privileges.py@182
PS11, Line 182: =
flake8: E501 line too long (93 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/13005/11/tests/authorization/test_owner_privileges.py@186
PS11, Line 186: )
flake8: E501 line too long (91 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/13005/11/tests/authorization/test_owner_privileges.py@193
PS11, Line 193: ,
flake8: E501 line too long (91 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/13005/11/tests/authorization/test_owner_privileges.py@213
PS11, Line 213: )
flake8: E501 line too long (91 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/13005/11/tests/authorization/test_owner_privileges.py@226
PS11, Line 226: )
flake8: E501 line too long (91 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/13005/11/tests/authorization/test_owner_privileges.py@236
PS11, Line 236: )
flake8: E501 line too long (91 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/13005/11/tests/authorization/test_owner_privileges.py@282
PS11, Line 282: _
flake8: E501 line too long (100 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/13005/11/tests/authorization/test_owner_privileges.py@287
PS11, Line 287: t
flake8: E501 line too long (101 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/13005/11/tests/authorization/test_owner_privileges.py@293
PS11, Line 293: 2
flake8: E501 line too long (92 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/13005/11/tests/authorization/test_owner_privileges.py@355
PS11, Line 355: _
flake8: E501 line too long (100 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/13005/11/tests/authorization/test_owner_privileges.py@360
PS11, Line 360: t
flake8: E501 line too long (101 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/13005/11/tests/authorization/test_owner_privileges.py@364
PS11, Line 364: 2
flake8: E501 line too long (92 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/13005/11/tests/authorization/test_owner_privileges.py@368
PS11, Line 368: )
flake8: E501 line too long (91 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/13005/11/tests/authorization/test_owner_privileges.py@412
PS11, Line 412: t
flake8: E501 line too long (101 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/13005/11/tests/authorization/test_owner_privileges.py@421
PS11, Line 421: i
flake8: E501 line too long (102 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/13005/11/tests/common/sentry_cache_test_suite.py
File tests/common/sentry_cache_test_suite.py:

http://gerrit.cloudera.org:8080/#/c/13005/11/tests/common/sentry_cache_test_suite.py@83
PS11, Line 83: }
flake8: E501 line too long (92 > 90 characters)



-- 
To view, visit http://gerrit.cloudera.org:8080/13005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436
Gerrit-Change-Number: 13005
Gerrit-PatchSet: 11
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Sudhanshu Arora <su...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Sat, 27 Apr 2019 02:17:43 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13005 )

Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
......................................................................


Patch Set 6:

Build Failed 

https://jenkins.impala.io/job/gerrit-code-review-checks/2915/ : Initial code review checks failed. See linked job for details on the failure.


-- 
To view, visit http://gerrit.cloudera.org:8080/13005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436
Gerrit-Change-Number: 13005
Gerrit-PatchSet: 6
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Sudhanshu Arora <su...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 25 Apr 2019 18:14:26 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13005 )

Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
......................................................................


Patch Set 11:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4093/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/13005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436
Gerrit-Change-Number: 13005
Gerrit-PatchSet: 11
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Sudhanshu Arora <su...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Sat, 27 Apr 2019 02:16:54 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0

Posted by "Vihang Karajgaonkar (Code Review)" <ge...@cloudera.org>.

Vihang Karajgaonkar has uploaded a new patch set (#20). ( http://gerrit.cloudera.org:8080/13005 )

Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
......................................................................

IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0

This change adds a compatibility shim in fe so that Impala can
interoperate with Hive 3.1.0. It moves the existing Metastoreshim class
to a compat-hive-2 directory and adds a new Metastoreshim class under
compat-hive-3 directory. These shim classes implement method which are
different in hive-2 v/s hive-3 and are used by front end code. At the
build time, based on the environment variable
IMPALA_HIVE_MAJOR_VERSION one of the two shims is added to as source
using the fe/pom.xml build plugin.

Additionally, in order to reduce the dependencies footprint of Hive in
the front end code, this patch also introduces a new module called
shaded-deps. This module using shade plugin to include only the source
files from hive-exec which are need by the fe code. For hive-2 build
path, no changes are done with respect to hive dependencies to minimize
the risk of destabilizing the master branch on the default build option
of using Hive-2.

The different set of dependencies are activated using maven profiles.
The activation of each profile is automatic based on the
IMPALA_HIVE_MAJOR_VERSION.

Testing:
1. Code compiles and runs against both HMS-3 and HMS-2
2. Ran full-suite of tests using the private jenkins job against HMS-2
3. Running full-tests against HMS-3 will need more work like supporting
Tez in the mini-cluster (for dataloading) and HMS transaction support
since HMS3 create transactional tables by default. THis will be on-going
effort and test failures on Hive-3 will be fixed in additional
sub-tasks.

Notes:
1. Patch uses a custom build of Hive to be deployed in mini-cluster. This
build has the fixes for HIVE-21596. This hack will be removed when the
patches are available in official CDP Hive builds.
2. Some of the existing tests rely on the fact the UDFs implement the
UDF interface in Hive (UDFLength, UDFHour, UDFYear). These built-in hive
functions have been moved to use GenericUDF interface in Hive 3. Impala
currently only supports UDFExecutor. In order to have a full
compatibility with all the functions in Hive 2.x we should support
GenericUDFs too. That would be taken up as a separate patch.
3. Sentry dependencies bring a lot of transitive hive dependencies. The
patch excludes such dependencies since they create problems while
building against Hive-3. Since these hive-2 dependencies are
already included when building against hive-2 this should not be a problem.

Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436
---
M CMakeLists.txt
M README.md
M bin/bootstrap_toolchain.py
M bin/impala-config.sh
M common/thrift/.gitignore
M common/thrift/CMakeLists.txt
M fe/CMakeLists.txt
M fe/pom.xml
A fe/src/compat-hive-2/java/org/apache/impala/compat/MetastoreShim.java
A fe/src/compat-hive-3/java/org/apache/impala/compat/HiveMetadataFormatUtils.java
A fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/main/java/org/apache/impala/analysis/StringLiteral.java
M fe/src/main/java/org/apache/impala/catalog/FeHBaseTable.java
M fe/src/main/java/org/apache/impala/catalog/TableLoader.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java
D fe/src/main/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/main/java/org/apache/impala/hive/executor/UdfExecutor.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/DescribeResultFactory.java
M fe/src/main/java/org/apache/impala/service/MetadataOp.java
M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
M fe/src/test/java/org/apache/impala/hive/executor/UdfExecutorTest.java
M fe/src/test/java/org/apache/impala/testutil/EmbeddedMetastoreClientPool.java
M impala-parent/pom.xml
A shaded-deps/.gitignore
A shaded-deps/CMakeLists.txt
A shaded-deps/pom.xml
M testdata/bin/run-hive-server.sh
M tests/custom_cluster/test_permanent_udfs.py
31 files changed, 1,791 insertions(+), 456 deletions(-)

git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/05/13005/20
--
To view, visit http://gerrit.cloudera.org:8080/13005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436
Gerrit-Change-Number: 13005
Gerrit-PatchSet: 20
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Sudhanshu Arora <su...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>

[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0

Posted by "Vihang Karajgaonkar (Code Review)" <ge...@cloudera.org>.

Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/13005 )

Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
......................................................................


Patch Set 23:

> Latest patch fixes the test issue which causes the
 > MetastoreEventsProcessorTest to fail

The test was breaking on my patch because I had modified the createDatabase() method in the test from using the DatabaseBuilder to directly constructing the Database object. The difference is Databuilder has parameters as a empty map by default. So the event returns a null value for params when they are not set while database creation after the patch. This broke the test because of a NPE


-- 
To view, visit http://gerrit.cloudera.org:8080/13005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436
Gerrit-Change-Number: 13005
Gerrit-PatchSet: 23
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Sudhanshu Arora <su...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 30 Apr 2019 22:05:05 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13005 )

Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
......................................................................


Patch Set 11: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4093/


-- 
To view, visit http://gerrit.cloudera.org:8080/13005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436
Gerrit-Change-Number: 13005
Gerrit-PatchSet: 11
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Sudhanshu Arora <su...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Sat, 27 Apr 2019 07:21:45 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0

Posted by "Vihang Karajgaonkar (Code Review)" <ge...@cloudera.org>.

Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/13005 )

Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
......................................................................


Patch Set 12:

(2 comments)

Updated the patch with Joe's review comments.

http://gerrit.cloudera.org:8080/#/c/13005/12/bin/impala-config.sh
File bin/impala-config.sh:

http://gerrit.cloudera.org:8080/#/c/13005/12/bin/impala-config.sh@a1
PS12, Line 1: 
> Restore this line (RAT checks will fail)
Done


http://gerrit.cloudera.org:8080/#/c/13005/12/bin/impala-config.sh@513
PS12, Line 513: # The directory in which all the thirdparty CDP components live.
              : export CDP_COMPONENTS_HOME="$IMPALA_TOOLCHAIN/cdp_components-$CDP_BUILD_NUMBER"
> You move this up, so you can remove this.
Done



-- 
To view, visit http://gerrit.cloudera.org:8080/13005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436
Gerrit-Change-Number: 13005
Gerrit-PatchSet: 12
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Sudhanshu Arora <su...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 29 Apr 2019 20:30:35 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0

Posted by "Vihang Karajgaonkar (Code Review)" <ge...@cloudera.org>.

Vihang Karajgaonkar has uploaded a new patch set (#18). ( http://gerrit.cloudera.org:8080/13005 )

Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
......................................................................

IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0

The different set of dependencies are activated using maven profiles.
The activation of each profile is automatic based on the
IMPALA_HIVE_MAJOR_VERSION.

git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/05/13005/18
--
To view, visit http://gerrit.cloudera.org:8080/13005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436
Gerrit-Change-Number: 13005
Gerrit-PatchSet: 18
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Sudhanshu Arora <su...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>

[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13005 )

Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
......................................................................


Patch Set 2:

(4 comments)

http://gerrit.cloudera.org:8080/#/c/13005/2/bin/impala-config.sh
File bin/impala-config.sh:

http://gerrit.cloudera.org:8080/#/c/13005/2/bin/impala-config.sh@199
PS2, Line 199: # When USE_CDP_HIVE is set we use the latest hive version available to deply in minicluster
line too long (91 > 90)


http://gerrit.cloudera.org:8080/#/c/13005/2/bin/impala-config.sh@203
PS2, Line 203:   # TODO(Vihang) we should repackage the tarballs so that the src and binaries are extracted
line too long (92 > 90)


http://gerrit.cloudera.org:8080/#/c/13005/2/bin/impala-config.sh@212
PS2, Line 212:   export HIVE_HOME="$IMPALA_TOOLCHAIN/cdh_components-${CDH_BUILD_NUMBER}/hive-${MINICLUSTER_HIVE_VERSION}"
line too long (106 > 90)


http://gerrit.cloudera.org:8080/#/c/13005/2/bin/impala-config.sh@546
PS2, Line 546: export HIVE_METASTORE_THRIFT_DIR=$CDP_COMPONENTS_HOME/apache-hive-${IMPALA_HIVE_VERSION}-bin/src/standalone-metastore/src/main/thrift
line too long (133 > 90)



-- 
To view, visit http://gerrit.cloudera.org:8080/13005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436
Gerrit-Change-Number: 13005
Gerrit-PatchSet: 2
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Sat, 13 Apr 2019 00:52:44 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0

Posted by "Vihang Karajgaonkar (Code Review)" <ge...@cloudera.org>.

Vihang Karajgaonkar has uploaded a new patch set (#6). ( http://gerrit.cloudera.org:8080/13005 )

Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
......................................................................

IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0

This change upgrades the hive dependencies of Impala to use Hive 3.1.0
based binaries. Most of the changes in this patch are based off patches
provided by Todd (links available in JIRA).

Upgrading the dependencies allows us to work with both Hive 3.1.0 and
Hive 2.1.0 in the same code line. In order to do this, the patch
trims down a lot of unnecessary hive dependencies of the front end code
by creating a shaded-deps module. The pom.xml of shaded-deps includes
only the files from Hive source which Impala depends for compilation.

Additionally, it also uses a custom build of Hive which is based of
Hive 3.1.0. This custom build includes patches for HIVE-21596 and
HIVE-21586 which are needed by Impala so that it can compile against
Hive-3 libraries and be able to talk to HMS-2.x metastore. Once these
patches are merged we can get rid of this custom build and rely on more
official sources of Hive builds.

The patch also changes impala-config.sh so that it always downloads the
Hive-3 libraries from the toolchain. The code is always built using
Hive-3 jars. However, based on the value of USE_CDP_HIVE, the
minicluster is deployed using Hive-3 or Hive-2 binaries. Since Impala
implements HiveServer2's TCLIService.thrift interface, it requires us to
use the existing mechanism of copying the hive-2/api TCLIService.thrift
to
hive-3/api. It also adds a few environment variables which point to the
metastore's thrift file and the CDH Hive version.

Testing:
1. Code compiles and runs against both HMS-3 and HMS-2
2. Ran full-suite of tests using the private jenkins job against HMS-2
3. Running full-tests against HMS-3 will need more work like supporting
Tez in the mini-cluster (for dataloading) and HMS transaction support
since HMS3 create transactional tables by default.

Notes:
1. Patch uses a custom build of Hive to be deployed in mini-cluster. This
build has the fixes for HIVE-21596
2. The Hive 3.1 does not include some of the patches like
HIVE-21077. Some of the recent code in Impala depends on that patch to
exist in hive. We will need to port HIVE-21077 to Hive 3.1.0 in order to
compile against Hive 3.1.0 jars. I will continue working on getting
this merged along with HIVE-21586

Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436
---
M CMakeLists.txt
M README.md
M bin/bootstrap_toolchain.py
M bin/impala-config.sh
M bin/set-classpath.sh
M common/thrift/.gitignore
M common/thrift/CMakeLists.txt
M fe/CMakeLists.txt
M fe/pom.xml
M fe/src/main/java/org/apache/impala/analysis/StringLiteral.java
M fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java
M fe/src/main/java/org/apache/impala/catalog/FeHBaseTable.java
M fe/src/main/java/org/apache/impala/catalog/MetaStoreClientPool.java
M fe/src/main/java/org/apache/impala/catalog/TableLoader.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java
M fe/src/main/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/main/java/org/apache/impala/hive/executor/UdfExecutor.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/DescribeResultFactory.java
M fe/src/main/java/org/apache/impala/service/MetadataOp.java
A fe/src/main/java/org/apache/impala/util/MetadataFormatUtils.java
M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
M fe/src/test/java/org/apache/impala/hive/executor/UdfExecutorTest.java
M fe/src/test/java/org/apache/impala/testutil/EmbeddedMetastoreClientPool.java
M impala-parent/pom.xml
A shaded-deps/.gitignore
A shaded-deps/CMakeLists.txt
A shaded-deps/dependency-reduced-pom.xml
A shaded-deps/pom.xml
M testdata/bin/run-hive-server.sh
M tests/authorization/test_owner_privileges.py
M tests/common/sentry_cache_test_suite.py
M tests/custom_cluster/test_permanent_udfs.py
35 files changed, 1,275 insertions(+), 292 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/05/13005/6
-- 
To view, visit http://gerrit.cloudera.org:8080/13005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436
Gerrit-Change-Number: 13005
Gerrit-PatchSet: 6
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Sudhanshu Arora <su...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>

[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0

Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.

Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/13005 )

Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
......................................................................


Patch Set 19:

(3 comments)

I'm basically ready to approve this. One comment and a couple nitpicking things.

http://gerrit.cloudera.org:8080/#/c/13005/19//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/13005/19//COMMIT_MSG@40
PS19, Line 40: Patch uses a custom build of Hive
Curious: is this true?


http://gerrit.cloudera.org:8080/#/c/13005/19/bin/impala-config.sh
File bin/impala-config.sh:

http://gerrit.cloudera.org:8080/#/c/13005/19/bin/impala-config.sh@a264
PS19, Line 264: 
Nitpicking: restore this


http://gerrit.cloudera.org:8080/#/c/13005/19/fe/pom.xml
File fe/pom.xml:

http://gerrit.cloudera.org:8080/#/c/13005/19/fe/pom.xml@a316
PS19, Line 316: 
              : 
              : 
              : 
What happened to the exclusions for calcite? Here and for hive-exec and hive-hbase-handler. Do we need them for the Hive 2 profile?



-- 
To view, visit http://gerrit.cloudera.org:8080/13005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436
Gerrit-Change-Number: 13005
Gerrit-PatchSet: 19
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Sudhanshu Arora <su...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 30 Apr 2019 04:35:30 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13005 )

Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
......................................................................


Patch Set 10:

(25 comments)

http://gerrit.cloudera.org:8080/#/c/13005/10/bin/impala-config.sh
File bin/impala-config.sh:

http://gerrit.cloudera.org:8080/#/c/13005/10/bin/impala-config.sh@196
PS10, Line 196: # When USE_CDP_HIVE is set we use the latest hive version available to deply in minicluster
line too long (91 > 90)


http://gerrit.cloudera.org:8080/#/c/13005/10/bin/impala-config.sh@200
PS10, Line 200:   # TODO(Vihang) we should repackage the tarballs so that the src and binaries are extracted
line too long (92 > 90)


http://gerrit.cloudera.org:8080/#/c/13005/10/bin/impala-config.sh@209
PS10, Line 209:   export HIVE_HOME="$IMPALA_TOOLCHAIN/cdh_components-${CDH_BUILD_NUMBER}/hive-${MINICLUSTER_HIVE_VERSION}"
line too long (106 > 90)


http://gerrit.cloudera.org:8080/#/c/13005/10/bin/impala-config.sh@546
PS10, Line 546: export HIVE_METASTORE_THRIFT_DIR=$CDP_COMPONENTS_HOME/apache-hive-${IMPALA_HIVE_VERSION}-src/standalone-metastore/src/main/thrift
line too long (129 > 90)


http://gerrit.cloudera.org:8080/#/c/13005/10/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java:

http://gerrit.cloudera.org:8080/#/c/13005/10/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1073
PS10, Line 1073:                 
line has trailing whitespace


http://gerrit.cloudera.org:8080/#/c/13005/10/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1169
PS10, Line 1169:               
line has trailing whitespace


http://gerrit.cloudera.org:8080/#/c/13005/10/testdata/bin/run-hive-server.sh
File testdata/bin/run-hive-server.sh:

http://gerrit.cloudera.org:8080/#/c/13005/10/testdata/bin/run-hive-server.sh@66
PS10, Line 66: export HIVE_METASTORE_HADOOP_OPTS="-verbose:class -Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=30010"
line too long (121 > 90)


http://gerrit.cloudera.org:8080/#/c/13005/10/tests/authorization/test_owner_privileges.py
File tests/authorization/test_owner_privileges.py:

http://gerrit.cloudera.org:8080/#/c/13005/10/tests/authorization/test_owner_privileges.py@103
PS10, Line 103: )
flake8: E501 line too long (91 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/13005/10/tests/authorization/test_owner_privileges.py@179
PS10, Line 179: =
flake8: E501 line too long (93 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/13005/10/tests/authorization/test_owner_privileges.py@182
PS10, Line 182: =
flake8: E501 line too long (93 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/13005/10/tests/authorization/test_owner_privileges.py@186
PS10, Line 186: )
flake8: E501 line too long (91 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/13005/10/tests/authorization/test_owner_privileges.py@193
PS10, Line 193: ,
flake8: E501 line too long (91 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/13005/10/tests/authorization/test_owner_privileges.py@213
PS10, Line 213: )
flake8: E501 line too long (91 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/13005/10/tests/authorization/test_owner_privileges.py@226
PS10, Line 226: )
flake8: E501 line too long (91 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/13005/10/tests/authorization/test_owner_privileges.py@236
PS10, Line 236: )
flake8: E501 line too long (91 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/13005/10/tests/authorization/test_owner_privileges.py@282
PS10, Line 282: _
flake8: E501 line too long (100 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/13005/10/tests/authorization/test_owner_privileges.py@287
PS10, Line 287: t
flake8: E501 line too long (101 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/13005/10/tests/authorization/test_owner_privileges.py@293
PS10, Line 293: 2
flake8: E501 line too long (92 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/13005/10/tests/authorization/test_owner_privileges.py@355
PS10, Line 355: _
flake8: E501 line too long (100 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/13005/10/tests/authorization/test_owner_privileges.py@360
PS10, Line 360: t
flake8: E501 line too long (101 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/13005/10/tests/authorization/test_owner_privileges.py@364
PS10, Line 364: 2
flake8: E501 line too long (92 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/13005/10/tests/authorization/test_owner_privileges.py@368
PS10, Line 368: )
flake8: E501 line too long (91 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/13005/10/tests/authorization/test_owner_privileges.py@412
PS10, Line 412: t
flake8: E501 line too long (101 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/13005/10/tests/authorization/test_owner_privileges.py@421
PS10, Line 421: i
flake8: E501 line too long (102 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/13005/10/tests/common/sentry_cache_test_suite.py
File tests/common/sentry_cache_test_suite.py:

http://gerrit.cloudera.org:8080/#/c/13005/10/tests/common/sentry_cache_test_suite.py@83
PS10, Line 83: }
flake8: E501 line too long (92 > 90 characters)



-- 
To view, visit http://gerrit.cloudera.org:8080/13005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436
Gerrit-Change-Number: 13005
Gerrit-PatchSet: 10
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Sudhanshu Arora <su...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 26 Apr 2019 18:23:22 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13005 )

Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
......................................................................


Patch Set 16:

Build Failed 

https://jenkins.impala.io/job/gerrit-code-review-checks/2988/ : Initial code review checks failed. See linked job for details on the failure.


-- 
To view, visit http://gerrit.cloudera.org:8080/13005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436
Gerrit-Change-Number: 13005
Gerrit-PatchSet: 16
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Sudhanshu Arora <su...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 30 Apr 2019 01:14:08 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0

Posted by "Vihang Karajgaonkar (Code Review)" <ge...@cloudera.org>.

Vihang Karajgaonkar has uploaded a new patch set (#23). ( http://gerrit.cloudera.org:8080/13005 )

Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
......................................................................

IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0

The different set of dependencies are activated using maven profiles.
The activation of each profile is automatic based on the
IMPALA_HIVE_MAJOR_VERSION.

git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/05/13005/23
--
To view, visit http://gerrit.cloudera.org:8080/13005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436
Gerrit-Change-Number: 13005
Gerrit-PatchSet: 23
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Sudhanshu Arora <su...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>

[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0

Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.

Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/13005 )

Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
......................................................................


Patch Set 3:

Sorry, I didn't refresh before posting my own comments, so didn't notice what Sudhanshu wrote.


-- 
To view, visit http://gerrit.cloudera.org:8080/13005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436
Gerrit-Change-Number: 13005
Gerrit-PatchSet: 3
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Sudhanshu Arora <su...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Wed, 17 Apr 2019 19:51:41 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0

Posted by "Vihang Karajgaonkar (Code Review)" <ge...@cloudera.org>.

Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/13005 )

Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
......................................................................


Patch Set 12:

(3 comments)

http://gerrit.cloudera.org:8080/#/c/13005/11/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java:

http://gerrit.cloudera.org:8080/#/c/13005/11/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1073
PS11, Line 1073:                 .getAddPartitionMessage(event.getMessage());
> line has trailing whitespace
Done


http://gerrit.cloudera.org:8080/#/c/13005/11/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1169
PS11, Line 1169:       try {
> line has trailing whitespace
Done


http://gerrit.cloudera.org:8080/#/c/13005/11/testdata/bin/run-hive-server.sh
File testdata/bin/run-hive-server.sh:

http://gerrit.cloudera.org:8080/#/c/13005/11/testdata/bin/run-hive-server.sh@66
PS11, Line 66: export HIVE_METASTORE_HADOOP_OPTS="-Xdebug -Xrunjdwp:transport=dt_socket,server=y,\
> line too long (121 > 90)
Done



-- 
To view, visit http://gerrit.cloudera.org:8080/13005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436
Gerrit-Change-Number: 13005
Gerrit-PatchSet: 12
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Sudhanshu Arora <su...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 29 Apr 2019 18:03:35 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13005 )

Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
......................................................................


Patch Set 6:

(25 comments)

http://gerrit.cloudera.org:8080/#/c/13005/6/bin/impala-config.sh
File bin/impala-config.sh:

http://gerrit.cloudera.org:8080/#/c/13005/6/bin/impala-config.sh@196
PS6, Line 196: # When USE_CDP_HIVE is set we use the latest hive version available to deply in minicluster
line too long (91 > 90)


http://gerrit.cloudera.org:8080/#/c/13005/6/bin/impala-config.sh@200
PS6, Line 200:   # TODO(Vihang) we should repackage the tarballs so that the src and binaries are extracted
line too long (92 > 90)


http://gerrit.cloudera.org:8080/#/c/13005/6/bin/impala-config.sh@209
PS6, Line 209:   export HIVE_HOME="$IMPALA_TOOLCHAIN/cdh_components-${CDH_BUILD_NUMBER}/hive-${MINICLUSTER_HIVE_VERSION}"
line too long (106 > 90)


http://gerrit.cloudera.org:8080/#/c/13005/6/bin/impala-config.sh@546
PS6, Line 546: export HIVE_METASTORE_THRIFT_DIR=$CDP_COMPONENTS_HOME/apache-hive-${IMPALA_HIVE_VERSION}-src/standalone-metastore/src/main/thrift
line too long (129 > 90)


http://gerrit.cloudera.org:8080/#/c/13005/6/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java:

http://gerrit.cloudera.org:8080/#/c/13005/6/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1073
PS6, Line 1073:                 
line has trailing whitespace


http://gerrit.cloudera.org:8080/#/c/13005/6/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1169
PS6, Line 1169:               
line has trailing whitespace


http://gerrit.cloudera.org:8080/#/c/13005/6/testdata/bin/run-hive-server.sh
File testdata/bin/run-hive-server.sh:

http://gerrit.cloudera.org:8080/#/c/13005/6/testdata/bin/run-hive-server.sh@66
PS6, Line 66: export HIVE_METASTORE_HADOOP_OPTS="-verbose:class -Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=30010"
line too long (121 > 90)


http://gerrit.cloudera.org:8080/#/c/13005/6/tests/authorization/test_owner_privileges.py
File tests/authorization/test_owner_privileges.py:

http://gerrit.cloudera.org:8080/#/c/13005/6/tests/authorization/test_owner_privileges.py@103
PS6, Line 103: )
flake8: E501 line too long (91 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/13005/6/tests/authorization/test_owner_privileges.py@179
PS6, Line 179: =
flake8: E501 line too long (93 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/13005/6/tests/authorization/test_owner_privileges.py@182
PS6, Line 182: =
flake8: E501 line too long (93 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/13005/6/tests/authorization/test_owner_privileges.py@186
PS6, Line 186: )
flake8: E501 line too long (91 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/13005/6/tests/authorization/test_owner_privileges.py@193
PS6, Line 193: ,
flake8: E501 line too long (91 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/13005/6/tests/authorization/test_owner_privileges.py@213
PS6, Line 213: )
flake8: E501 line too long (91 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/13005/6/tests/authorization/test_owner_privileges.py@226
PS6, Line 226: )
flake8: E501 line too long (91 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/13005/6/tests/authorization/test_owner_privileges.py@236
PS6, Line 236: )
flake8: E501 line too long (91 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/13005/6/tests/authorization/test_owner_privileges.py@282
PS6, Line 282: _
flake8: E501 line too long (100 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/13005/6/tests/authorization/test_owner_privileges.py@287
PS6, Line 287: t
flake8: E501 line too long (101 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/13005/6/tests/authorization/test_owner_privileges.py@293
PS6, Line 293: 2
flake8: E501 line too long (92 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/13005/6/tests/authorization/test_owner_privileges.py@355
PS6, Line 355: _
flake8: E501 line too long (100 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/13005/6/tests/authorization/test_owner_privileges.py@360
PS6, Line 360: t
flake8: E501 line too long (101 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/13005/6/tests/authorization/test_owner_privileges.py@364
PS6, Line 364: 2
flake8: E501 line too long (92 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/13005/6/tests/authorization/test_owner_privileges.py@368
PS6, Line 368: )
flake8: E501 line too long (91 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/13005/6/tests/authorization/test_owner_privileges.py@412
PS6, Line 412: t
flake8: E501 line too long (101 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/13005/6/tests/authorization/test_owner_privileges.py@421
PS6, Line 421: i
flake8: E501 line too long (102 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/13005/6/tests/common/sentry_cache_test_suite.py
File tests/common/sentry_cache_test_suite.py:

http://gerrit.cloudera.org:8080/#/c/13005/6/tests/common/sentry_cache_test_suite.py@83
PS6, Line 83: }
flake8: E501 line too long (92 > 90 characters)



-- 
To view, visit http://gerrit.cloudera.org:8080/13005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436
Gerrit-Change-Number: 13005
Gerrit-PatchSet: 6
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Sudhanshu Arora <su...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 25 Apr 2019 17:29:02 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13005 )

Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
......................................................................


Patch Set 20:

Build Failed 

https://jenkins.impala.io/job/gerrit-code-review-checks/2994/ : Initial code review checks failed. See linked job for details on the failure.


-- 
To view, visit http://gerrit.cloudera.org:8080/13005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436
Gerrit-Change-Number: 13005
Gerrit-PatchSet: 20
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Sudhanshu Arora <su...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 30 Apr 2019 06:48:36 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13005 )

Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
......................................................................


Patch Set 8:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4071/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/13005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436
Gerrit-Change-Number: 13005
Gerrit-PatchSet: 8
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Sudhanshu Arora <su...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 25 Apr 2019 22:29:17 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0

Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.

Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/13005 )

Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
......................................................................


Patch Set 3:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/13005/3/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java:

http://gerrit.cloudera.org:8080/#/c/13005/3/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@172
PS3, Line 172: messageFactory
nit: maybe messageDeserializer would be a better name now



-- 
To view, visit http://gerrit.cloudera.org:8080/13005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436
Gerrit-Change-Number: 13005
Gerrit-PatchSet: 3
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Sudhanshu Arora <su...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 23 Apr 2019 16:46:42 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0

Posted by "Vihang Karajgaonkar (Code Review)" <ge...@cloudera.org>.

Vihang Karajgaonkar has uploaded a new patch set (#11). ( http://gerrit.cloudera.org:8080/13005 )

Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
......................................................................

IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0

This change upgrades the hive dependencies of Impala to use Hive 3.1.0
based binaries. Most of the changes in this patch are based off patches
provided by Todd (links available in JIRA).

Upgrading the dependencies allows us to work with both Hive 3.1.0 and
Hive 2.1.0 in the same code line. In order to do this, the patch
trims down a lot of unnecessary hive dependencies of the front end code
by creating a shaded-deps module. The pom.xml of shaded-deps includes
only the files from Hive source which Impala depends for compilation.

Additionally, it also uses a custom build of Hive which is based of
Hive 3.1.0. This custom build includes patches for HIVE-21596 and
HIVE-21586 which are needed by Impala so that it can compile against
Hive-3 libraries and be able to talk to HMS-2.x metastore. Once these
patches are merged we can get rid of this custom build and rely on more
official sources of Hive builds.

The patch also changes impala-config.sh so that it always downloads the
Hive-3 libraries from the toolchain. The code is always built using
Hive-3 jars. However, based on the value of USE_CDP_HIVE, the
minicluster is deployed using Hive-3 or Hive-2 binaries. Since Impala
implements HiveServer2's TCLIService.thrift interface, it requires us to
use the existing mechanism of copying the hive-2/api TCLIService.thrift
to
hive-3/api. It also adds a few environment variables which point to the
metastore's thrift file and the CDH Hive version.

Notes:
1. Patch uses a custom build of Hive to be deployed in mini-cluster. This
build has the fixes for HIVE-21596
2. The Hive 3.1 does not include some of the patches like
HIVE-21077. Some of the recent code in Impala depends on that patch to
exist in hive. We will need to port HIVE-21077 to Hive 3.1.0 in order to
compile against Hive 3.1.0 jars. I will continue working on getting
this merged along with HIVE-21586
3. Some of the existing tests rely on the fact the UDFs implement the
UDF interface in Hive (UDFLength, UDFHour, UDFYear). These built-in hive
functions have been moved to use GenericUDF interface in Hive 3. Impala
currently only supports UDFExecutor. In order to have a full
compatibility with all the functions in Hive 2.x we should support
GenericUDFs too. That would be taken up as a separate patch.
4. The Sentry object ownership tests are flaky. There are
race-conditions in that code which get exposed for some by this patch.
For example, when a database is created, the object privileges are
immediately updated in the catalog cache. If Sentry has not been updated
and there is a refresh authorization call in between it can clear the
privilege from the cache causing the subsequent statement using that
privilege to fail. The patch adds sleep statements in the test to
work-around these issues.

Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436
---
M CMakeLists.txt
M README.md
M bin/bootstrap_toolchain.py
M bin/impala-config.sh
M bin/set-classpath.sh
M common/thrift/.gitignore
M common/thrift/CMakeLists.txt
M fe/CMakeLists.txt
M fe/pom.xml
A fe/src/compat-hive-2/java/org/apache/impala/compat/HiveShims.java
R fe/src/compat-hive-2/java/org/apache/impala/compat/MetastoreShim.java
A fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/main/java/org/apache/impala/analysis/StringLiteral.java
M fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java
M fe/src/main/java/org/apache/impala/catalog/FeHBaseTable.java
M fe/src/main/java/org/apache/impala/catalog/TableLoader.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java
M fe/src/main/java/org/apache/impala/hive/executor/UdfExecutor.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/DescribeResultFactory.java
M fe/src/main/java/org/apache/impala/service/MetadataOp.java
A fe/src/main/java/org/apache/impala/util/MetadataFormatUtils.java
M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
M fe/src/test/java/org/apache/impala/hive/executor/UdfExecutorTest.java
M fe/src/test/java/org/apache/impala/testutil/EmbeddedMetastoreClientPool.java
M impala-parent/pom.xml
A shaded-deps/.gitignore
A shaded-deps/CMakeLists.txt
A shaded-deps/pom.xml
M testdata/bin/run-hive-server.sh
M tests/authorization/test_owner_privileges.py
M tests/common/sentry_cache_test_suite.py
M tests/custom_cluster/test_permanent_udfs.py
35 files changed, 1,633 insertions(+), 361 deletions(-)

git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/05/13005/11
--
To view, visit http://gerrit.cloudera.org:8080/13005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436
Gerrit-Change-Number: 13005
Gerrit-PatchSet: 11
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Sudhanshu Arora <su...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>

[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13005 )

Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
......................................................................


Patch Set 9:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4075/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/13005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436
Gerrit-Change-Number: 13005
Gerrit-PatchSet: 9
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Sudhanshu Arora <su...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 26 Apr 2019 04:30:25 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13005 )

Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
......................................................................


Patch Set 22:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4120/ DRY_RUN=false


-- 
To view, visit http://gerrit.cloudera.org:8080/13005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436
Gerrit-Change-Number: 13005
Gerrit-PatchSet: 22
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Sudhanshu Arora <su...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 30 Apr 2019 16:30:26 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0

Posted by "Vihang Karajgaonkar (Code Review)" <ge...@cloudera.org>.

Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/13005 )

Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
......................................................................


Patch Set 3:

(15 comments)

> (1 comment)
 > 
 > Nice work.
 > 
 > When I did the transitional work between Hive 1 and Hive 2, I
 > introduced a variable that switched at build time between the two
 > worlds. (See a203733fac3e1e37df8abeee39a88d187153a8c5 for the
 > revert and "git log --grep IMPALA_MINICLUSTER_PROFILE")
 > 
 > If I'm understanding right, the approach here is to produce a
 > single "binary" that works for both worlds? Or at run time do the
 > "original" Hive jars get run? I think both approaches are
 > plausible; just want to make sure we're clear about it.
 > 
 > (Is the shading slow? I've seen maven-shade-plugin be very slow...)

The original approach was to always build against hive-3, but we switched the approach to use a compatibility shim based approach as you suggested above.

Shading is not super slow. It task about 4 sec to build the jar if I remember correctly

http://gerrit.cloudera.org:8080/#/c/13005/2/bin/impala-config.sh
File bin/impala-config.sh:

http://gerrit.cloudera.org:8080/#/c/13005/2/bin/impala-config.sh@199
PS2, Line 199: # When USE_CDP_HIVE is set we use the latest hive version available to deply in minicluster
> line too long (91 > 90)
Done


http://gerrit.cloudera.org:8080/#/c/13005/2/bin/impala-config.sh@203
PS2, Line 203:   # TODO(Vihang) we should repackage the tarballs so that the src and binaries are extracted
> line too long (92 > 90)
This line was removed later in the patch


http://gerrit.cloudera.org:8080/#/c/13005/2/bin/impala-config.sh@212
PS2, Line 212:   export HIVE_HOME="$IMPALA_TOOLCHAIN/cdh_components-${CDH_BUILD_NUMBER}/hive-${MINICLUSTER_HIVE_VERSION}"
> line too long (106 > 90)
Done


http://gerrit.cloudera.org:8080/#/c/13005/2/bin/impala-config.sh@230
PS2, Line 230: . "$IMPALA_HOME/bin/impala-config-branch.sh"
             : if [ -f "$IMPALA_HOME/bin/impala-config-local.sh" ]; then
             :   . "$IMPALA_HOME/bin/impala-config-local.sh"
             : fi
> We need to be careful about which variables are assigned before this and wh
Good point. Moved all the new variable assignment logic post this block.


http://gerrit.cloudera.org:8080/#/c/13005/2/bin/impala-config.sh@546
PS2, Line 546: export HIVE_METASTORE_THRIFT_DIR=$CDP_COMPONENTS_HOME/apache-hive-${IMPALA_HIVE_VERSION}-bin/src/standalone-metastore/src/main/thrift
> line too long (133 > 90)
This line was moved later in the patch and the line length is under the 90 now.


http://gerrit.cloudera.org:8080/#/c/13005/2/fe/src/main/java/org/apache/impala/analysis/StringLiteral.java
File fe/src/main/java/org/apache/impala/analysis/StringLiteral.java:

http://gerrit.cloudera.org:8080/#/c/13005/2/fe/src/main/java/org/apache/impala/analysis/StringLiteral.java@105
PS2, Line 105:   public static String unescapeSQLString(String b) {
> Do we have any plans to create a public classes in Hive through which we ca
Yes, I think we may be able to do that add that. Will do it as a followup item.


http://gerrit.cloudera.org:8080/#/c/13005/3/fe/src/main/java/org/apache/impala/analysis/StringLiteral.java
File fe/src/main/java/org/apache/impala/analysis/StringLiteral.java:

http://gerrit.cloudera.org:8080/#/c/13005/3/fe/src/main/java/org/apache/impala/analysis/StringLiteral.java@96
PS3, Line 96:   /**
            :    * Copied from Apache Hive's BaseSemanticAnalyzer. This method has not changed
            :    * since last several years so hoping that it is fairly stable by now. Sourcing it from
            :    * the Hive's code without copying brings along with it a lot of other unnecessary
            :    * dependencies
            :    * @param b
            :    * @return
            :    */
> Can you move the parts copied from Hive to a separate file/directory? It wi
The copied code was moved into HiveShims class in the compat-3 directory


http://gerrit.cloudera.org:8080/#/c/13005/3/fe/src/main/java/org/apache/impala/catalog/FeHBaseTable.java
File fe/src/main/java/org/apache/impala/catalog/FeHBaseTable.java:

http://gerrit.cloudera.org:8080/#/c/13005/3/fe/src/main/java/org/apache/impala/catalog/FeHBaseTable.java@122
PS3, Line 122:     //TODO (Vihang) this pulls in hive-hbase-handler dependency which brings all the
             :     // other stuff we don't need. Perhaps we just need to copy the constants and mark
             :     // them public API in Hive source code
             :     private static final String HBASE_COLUMNS_MAPPING = "hbase.columns.mapping";
             :     private static final String HBASE_TABLE_DEFAULT_STORAGE_TYPE = "hbase.table.default"
             :         + ".storage.type";
             :     private static final String HBASE_KEY_COL = ":key";
             :     private static final String HBASE_TABLE_NAME = "hbase.table.name";
> Constants copied from Hive 3 could be copied to a common place.
Copied to HiveShims in compat-3. For Hive-2 build, we still use HBaseSerDe constants from the hbase-handler library


http://gerrit.cloudera.org:8080/#/c/13005/2/fe/src/main/java/org/apache/impala/catalog/TableLoader.java
File fe/src/main/java/org/apache/impala/catalog/TableLoader.java:

http://gerrit.cloudera.org:8080/#/c/13005/2/fe/src/main/java/org/apache/impala/catalog/TableLoader.java@42
PS2, Line 42:           TableType.MATERIALIZED_VIEW);
> Do we need to any tests to ensure that we support materialized views?
This line was moved to compatibility shim in the later patchsets. Right now, the test jobs are failing when CDP is turned on. Once we stabilize the test jobs on CDP, I think we should add new tests for materialized views as well. I will create a followup task for this.


http://gerrit.cloudera.org:8080/#/c/13005/3/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java:

http://gerrit.cloudera.org:8080/#/c/13005/3/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@189
PS3, Line 189: `
> nit: mistyping?
Done


http://gerrit.cloudera.org:8080/#/c/13005/3/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java:

http://gerrit.cloudera.org:8080/#/c/13005/3/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@172
PS3, Line 172: messageFactory
> nit: maybe messageDeserializer would be a better name now
Done


http://gerrit.cloudera.org:8080/#/c/13005/3/fe/src/main/java/org/apache/impala/service/DescribeResultFactory.java
File fe/src/main/java/org/apache/impala/service/DescribeResultFactory.java:

http://gerrit.cloudera.org:8080/#/c/13005/3/fe/src/main/java/org/apache/impala/service/DescribeResultFactory.java@29
PS3, Line 29:   
> nit: wrong indentation
Done


http://gerrit.cloudera.org:8080/#/c/13005/2/fe/src/main/java/org/apache/impala/service/MetadataOp.java
File fe/src/main/java/org/apache/impala/service/MetadataOp.java:

http://gerrit.cloudera.org:8080/#/c/13005/2/fe/src/main/java/org/apache/impala/service/MetadataOp.java@345
PS2, Line 345:       case VIRTUAL_VIEW:
> Is this intentional? If yes, can you please explain the reason?
INDEX_TABLE was added again in the later patchsets


http://gerrit.cloudera.org:8080/#/c/13005/3/fe/src/main/java/org/apache/impala/service/MetadataOp.java
File fe/src/main/java/org/apache/impala/service/MetadataOp.java:

http://gerrit.cloudera.org:8080/#/c/13005/3/fe/src/main/java/org/apache/impala/service/MetadataOp.java@345
PS3, Line 345:       case VIRTUAL_VIEW:
             :       case MATERIALIZED_VIEW:
             :         return TABLE_TYPE_VIEW;
> This returns TABLE_TYPE_VIEW even for EXTERNAL_TABLE and MANAGED_TABLE - is
The latest patch fixed this where we return TABLE_TYPE_TABLE for external and managed tables and TABLE_TYPE_VIEW for VIRTUAL_VIEW and MATERIALIZED_VIEW. In hive-2 build path there is no MATERIALIZED_VIEW type.


http://gerrit.cloudera.org:8080/#/c/13005/2/shaded-deps/pom.xml
File shaded-deps/pom.xml:

http://gerrit.cloudera.org:8080/#/c/13005/2/shaded-deps/pom.xml@24
PS2, Line 24: ../../impala-parent/pom.xml
> Think this should be ../impala-parent/pom.xml
Good catch. Fixed it.



-- 
To view, visit http://gerrit.cloudera.org:8080/13005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436
Gerrit-Change-Number: 13005
Gerrit-PatchSet: 3
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Sudhanshu Arora <su...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 29 Apr 2019 20:49:59 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13005 )

Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
......................................................................


Patch Set 21:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/13005/21/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
File fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java:

http://gerrit.cloudera.org:8080/#/c/13005/21/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@291
PS21, Line 291:         if (currentChar == '\\' && (i + 6 < stringLiteral.length()) && stringLiteral.charAt(i + 1) == 'u') {
line too long (108 > 90)



-- 
To view, visit http://gerrit.cloudera.org:8080/13005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436
Gerrit-Change-Number: 13005
Gerrit-PatchSet: 21
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Sudhanshu Arora <su...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 30 Apr 2019 06:48:22 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13005 )

Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
......................................................................


Patch Set 22: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4120/


-- 
To view, visit http://gerrit.cloudera.org:8080/13005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436
Gerrit-Change-Number: 13005
Gerrit-PatchSet: 22
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Sudhanshu Arora <su...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 30 Apr 2019 21:34:41 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13005 )

Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
......................................................................


Patch Set 23:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/13005/23/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
File fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java:

http://gerrit.cloudera.org:8080/#/c/13005/23/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@291
PS23, Line 291:         if (currentChar == '\\' && (i + 6 < stringLiteral.length()) && stringLiteral.charAt(i + 1) == 'u') {
line too long (108 > 90)



-- 
To view, visit http://gerrit.cloudera.org:8080/13005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436
Gerrit-Change-Number: 13005
Gerrit-PatchSet: 23
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Sudhanshu Arora <su...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 30 Apr 2019 21:32:24 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0

Posted by "Vihang Karajgaonkar (Code Review)" <ge...@cloudera.org>.

Vihang Karajgaonkar has uploaded a new patch set (#14). ( http://gerrit.cloudera.org:8080/13005 )

Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
......................................................................

IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0

The different set of dependencies are activated using maven profiles.
The activation of each profile is automatic based on the
IMPALA_HIVE_MAJOR_VERSION.

Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436
---
M CMakeLists.txt
M README.md
M bin/bootstrap_toolchain.py
M bin/impala-config.sh
M bin/set-classpath.sh
M common/thrift/.gitignore
M common/thrift/CMakeLists.txt
M fe/CMakeLists.txt
M fe/pom.xml
A fe/src/compat-hive-2/java/org/apache/impala/compat/HiveShim.java
A fe/src/compat-hive-2/java/org/apache/impala/compat/MetastoreShim.java
A fe/src/compat-hive-3/java/org/apache/impala/compat/HiveMetadataFormatUtils.java
A fe/src/compat-hive-3/java/org/apache/impala/compat/HiveShim.java
A fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/main/java/org/apache/impala/analysis/StringLiteral.java
M fe/src/main/java/org/apache/impala/catalog/FeHBaseTable.java
M fe/src/main/java/org/apache/impala/catalog/TableLoader.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java
D fe/src/main/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/main/java/org/apache/impala/hive/executor/UdfExecutor.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/DescribeResultFactory.java
M fe/src/main/java/org/apache/impala/service/MetadataOp.java
M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
M fe/src/test/java/org/apache/impala/hive/executor/UdfExecutorTest.java
M fe/src/test/java/org/apache/impala/testutil/EmbeddedMetastoreClientPool.java
M impala-parent/pom.xml
A shaded-deps/.gitignore
A shaded-deps/CMakeLists.txt
A shaded-deps/pom.xml
M testdata/bin/run-hive-server.sh
M tests/custom_cluster/test_permanent_udfs.py
34 files changed, 1,885 insertions(+), 465 deletions(-)

git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/05/13005/14
--
To view, visit http://gerrit.cloudera.org:8080/13005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436
Gerrit-Change-Number: 13005
Gerrit-PatchSet: 14
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Sudhanshu Arora <su...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>

[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0

Posted by "Vihang Karajgaonkar (Code Review)" <ge...@cloudera.org>.

Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/13005 )

Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
......................................................................

Patch Set 22:

(4 comments)

http://gerrit.cloudera.org:8080/#/c/13005/22//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/13005/22//COMMIT_MSG@21
PS22, Line 21: For hive-2 build
: path, no changes are done with respect to hive dependencies to minimize
: the risk of destabilizing the master branch on the default build option
: of using Hive-2.
> I think the thinking is that we never wanted to pull all the transitive dep
+1 to Todd's comment. We have to start somewhere and I think hive-3 integration is a good starting point. Hive dependencies are a hairball of transitive dependencies and we should be very careful of pulling them in. In a ideal world, Impala should probably only depend on standalone-metastore jar and nothing else from hive side. This is one step closer to that.

http://gerrit.cloudera.org:8080/#/c/13005/22//COMMIT_MSG@34
PS22, Line 34: HMS transaction support
: since HMS3 create transactional tables by default
> based on my initial testing it actually doesn't look like it's doing transa
Good to know this. I was under the impression that managed tables are transactional by default in Hive-3.

http://gerrit.cloudera.org:8080/#/c/13005/22/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java:

http://gerrit.cloudera.org:8080/#/c/13005/22/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@972
PS22, Line 972: JSONDropTableMessage dropTableMessage =
> Why not use DropTableMessage? i.e. Why typecast?
We use JSON*Messages because there are certain methods which are added to these classes which are not available in their base classes. This is kind of weird currently in CDH6.x hive because Sentry uses a non-standard messageFactory which adds methods on top of JSON*Messages.
Eg.
In in cdh6.x the CreateDatabaseMessage does not have getDatabaseObject() method. It is only available in JSONCreateDatabaseMessage. We could not add a abstract method in CreateDatabaseMessage without breaking Sentry (I am not 100% clear on the history behind this)

So we are stuck with using JSON*Messages to keep compatibility with Sentry on cdh6.x code lines until we deprecate support for Sentry

http://gerrit.cloudera.org:8080/#/c/13005/22/fe/src/main/java/org/apache/impala/service/DescribeResultFactory.java
File fe/src/main/java/org/apache/impala/service/DescribeResultFactory.java:

http://gerrit.cloudera.org:8080/#/c/13005/22/fe/src/main/java/org/apache/impala/service/DescribeResultFactory.java@227
PS22, Line 227: String[] columns = line.split("\t");
> yea I'd suggested not shimming it. I don't have a strong opinion of whether
I had a constant in shim. As Todd pointed out above I moved it to this class following his suggestion. I wanted to use a constant but then I saw the line above uses "\n" so kept it to using "\t" value for keeping it consistent or rather consistently bad in this method :( depending on which way you prefer.

--
To view, visit http://gerrit.cloudera.org:8080/13005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436
Gerrit-Change-Number: 13005
Gerrit-PatchSet: 22
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Sudhanshu Arora <su...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 01 May 2019 00:21:23 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13005 )

Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
......................................................................


Patch Set 14:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/13005/14/fe/src/compat-hive-3/java/org/apache/impala/compat/HiveShim.java
File fe/src/compat-hive-3/java/org/apache/impala/compat/HiveShim.java:

http://gerrit.cloudera.org:8080/#/c/13005/14/fe/src/compat-hive-3/java/org/apache/impala/compat/HiveShim.java@62
PS14, Line 62:         if (currentChar == '\\' && (i + 6 < stringLiteral.length()) && stringLiteral.charAt(i + 1) == 'u') {
line too long (108 > 90)


http://gerrit.cloudera.org:8080/#/c/13005/14/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java:

http://gerrit.cloudera.org:8080/#/c/13005/14/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1072
PS14, Line 1072:             MetastoreEventsProcessor.getMessageDeserializer()        
line has trailing whitespace



-- 
To view, visit http://gerrit.cloudera.org:8080/13005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436
Gerrit-Change-Number: 13005
Gerrit-PatchSet: 14
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Sudhanshu Arora <su...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 29 Apr 2019 21:58:58 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0

Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.

Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/13005 )

Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
......................................................................


Patch Set 21: Code-Review+2


-- 
To view, visit http://gerrit.cloudera.org:8080/13005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436
Gerrit-Change-Number: 13005
Gerrit-PatchSet: 21
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Sudhanshu Arora <su...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 30 Apr 2019 16:26:35 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13005 )

Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
......................................................................


Patch Set 3:

Build Failed 

https://jenkins.impala.io/job/gerrit-code-review-checks/2798/ : Initial code review checks failed. See linked job for details on the failure.


-- 
To view, visit http://gerrit.cloudera.org:8080/13005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436
Gerrit-Change-Number: 13005
Gerrit-PatchSet: 3
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Tue, 16 Apr 2019 18:15:11 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13005 )

Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
......................................................................


Patch Set 12: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/13005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436
Gerrit-Change-Number: 13005
Gerrit-PatchSet: 12
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Sudhanshu Arora <su...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 29 Apr 2019 23:24:51 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0

Posted by "Vihang Karajgaonkar (Code Review)" <ge...@cloudera.org>.

Vihang Karajgaonkar has uploaded a new patch set (#12). ( http://gerrit.cloudera.org:8080/13005 )

Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
......................................................................

IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0

The different set of dependencies are activated using maven profiles.
The activation of each profile is automatic based on the
IMPALA_HIVE_MAJOR_VERSION.

Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436
---
M CMakeLists.txt
M README.md
M bin/bootstrap_toolchain.py
M bin/impala-config.sh
M bin/set-classpath.sh
M common/thrift/.gitignore
M common/thrift/CMakeLists.txt
M fe/CMakeLists.txt
M fe/pom.xml
A fe/src/compat-hive-2/java/org/apache/impala/compat/HiveShim.java
A fe/src/compat-hive-2/java/org/apache/impala/compat/MetastoreShim.java
A fe/src/compat-hive-3/java/org/apache/impala/compat/HiveMetadataFormatUtils.java
A fe/src/compat-hive-3/java/org/apache/impala/compat/HiveShim.java
A fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/main/java/org/apache/impala/analysis/StringLiteral.java
M fe/src/main/java/org/apache/impala/catalog/FeHBaseTable.java
M fe/src/main/java/org/apache/impala/catalog/TableLoader.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java
D fe/src/main/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/main/java/org/apache/impala/hive/executor/UdfExecutor.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/DescribeResultFactory.java
M fe/src/main/java/org/apache/impala/service/MetadataOp.java
M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
M fe/src/test/java/org/apache/impala/hive/executor/UdfExecutorTest.java
M fe/src/test/java/org/apache/impala/testutil/EmbeddedMetastoreClientPool.java
M impala-parent/pom.xml
A shaded-deps/.gitignore
A shaded-deps/CMakeLists.txt
A shaded-deps/pom.xml
M testdata/bin/run-hive-server.sh
M tests/custom_cluster/test_permanent_udfs.py
34 files changed, 1,870 insertions(+), 449 deletions(-)

git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/05/13005/12
--
To view, visit http://gerrit.cloudera.org:8080/13005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436
Gerrit-Change-Number: 13005
Gerrit-PatchSet: 12
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Sudhanshu Arora <su...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>

[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0

Posted by "Sudhanshu Arora (Code Review)" <ge...@cloudera.org>.

Sudhanshu Arora has posted comments on this change. ( http://gerrit.cloudera.org:8080/13005 )

Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
......................................................................


Patch Set 22:

(6 comments)

http://gerrit.cloudera.org:8080/#/c/13005/22//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/13005/22//COMMIT_MSG@21
PS22, Line 21: For hive-2 build
             : path, no changes are done with respect to hive dependencies to minimize
             : the risk of destabilizing the master branch on the default build option
             : of using Hive-2.
IIUC, it means that we are keeping all the dependencies in Hive2 but reducing dependencies in Hive3. 
While I understand it avoid the situations where an untested path might require a dependency in master branch.
What I don't understand is we can face that situation in new release and how are we planning to de-risk that. 
In other words are we saying it is ok for new release to be less stable than previous release?


http://gerrit.cloudera.org:8080/#/c/13005/22//COMMIT_MSG@34
PS22, Line 34: HMS transaction support
             : since HMS3 create transactional tables by default
Does it create full transactional table or "insert_only" by default?
Are we planning on keeping that behavior or changing that?


http://gerrit.cloudera.org:8080/#/c/13005/22/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java:

http://gerrit.cloudera.org:8080/#/c/13005/22/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@972
PS22, Line 972:       JSONDropTableMessage dropTableMessage =
Why not use DropTableMessage? i.e. Why typecast?


http://gerrit.cloudera.org:8080/#/c/13005/22/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1035
PS22, Line 1035:           (JSONCreateDatabaseMessage) MetastoreEventsProcessor.getMessageDeserializer()
Isn't CreateDatabaseMessage sufficient?


http://gerrit.cloudera.org:8080/#/c/13005/22/fe/src/main/java/org/apache/impala/service/DescribeResultFactory.java
File fe/src/main/java/org/apache/impala/service/DescribeResultFactory.java:

http://gerrit.cloudera.org:8080/#/c/13005/22/fe/src/main/java/org/apache/impala/service/DescribeResultFactory.java@227
PS22, Line 227:       String[] columns = line.split("\t");
Nit: Add this constant into MetastoreShim or if this is the only usage then create a constant here?


http://gerrit.cloudera.org:8080/#/c/13005/22/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
File fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java:

http://gerrit.cloudera.org:8080/#/c/13005/22/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@575
PS22, Line 575:           partition = metaStoreClient.getHiveClient().getPartition(dbName, tblName, "p1"
Super Nit: Move p1 to next line and combine the string



-- 
To view, visit http://gerrit.cloudera.org:8080/13005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436
Gerrit-Change-Number: 13005
Gerrit-PatchSet: 22
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Sudhanshu Arora <su...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 30 Apr 2019 19:11:40 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13005 )

Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
......................................................................


Patch Set 12:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/13005/12/fe/src/compat-hive-3/java/org/apache/impala/compat/HiveShim.java
File fe/src/compat-hive-3/java/org/apache/impala/compat/HiveShim.java:

http://gerrit.cloudera.org:8080/#/c/13005/12/fe/src/compat-hive-3/java/org/apache/impala/compat/HiveShim.java@62
PS12, Line 62:         if (currentChar == '\\' && (i + 6 < stringLiteral.length()) && stringLiteral.charAt(i + 1) == 'u') {
line too long (108 > 90)


http://gerrit.cloudera.org:8080/#/c/13005/12/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java:

http://gerrit.cloudera.org:8080/#/c/13005/12/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1072
PS12, Line 1072:             MetastoreEventsProcessor.getMessageDeserializer()        
line has trailing whitespace



-- 
To view, visit http://gerrit.cloudera.org:8080/13005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436
Gerrit-Change-Number: 13005
Gerrit-PatchSet: 12
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Sudhanshu Arora <su...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 29 Apr 2019 18:03:28 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0

Posted by "Vihang Karajgaonkar (Code Review)" <ge...@cloudera.org>.

Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/13005 )

Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
......................................................................


Patch Set 23:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/13005/22/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
File fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java:

http://gerrit.cloudera.org:8080/#/c/13005/22/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@575
PS22, Line 575:     alterDbSetOwnerFromImpala(TEST_DB_NAME, newOwnerRole, TOwnerType.ROLE);
> Super Nit: Move p1 to next line and combine the string
sorry saw this late. The jenkins job to merge this patch is already triggered and I don't want to restart it again for this minor change. I will fix it in a subsequent patch which touches this file.



-- 
To view, visit http://gerrit.cloudera.org:8080/13005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436
Gerrit-Change-Number: 13005
Gerrit-PatchSet: 23
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Sudhanshu Arora <su...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 30 Apr 2019 23:19:17 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13005 )

Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
......................................................................


Patch Set 20:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/13005/20/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
File fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java:

http://gerrit.cloudera.org:8080/#/c/13005/20/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@283
PS20, Line 283:         if (currentChar == '\\' && (i + 6 < stringLiteral.length()) && stringLiteral.charAt(i + 1) == 'u') {
line too long (108 > 90)



-- 
To view, visit http://gerrit.cloudera.org:8080/13005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436
Gerrit-Change-Number: 13005
Gerrit-PatchSet: 20
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Sudhanshu Arora <su...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 30 Apr 2019 05:54:26 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0

Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.

Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/13005 )

Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
......................................................................


Patch Set 14:

(3 comments)

http://gerrit.cloudera.org:8080/#/c/13005/14/bin/impala-config.sh
File bin/impala-config.sh:

http://gerrit.cloudera.org:8080/#/c/13005/14/bin/impala-config.sh@a747
PS14, Line 747: 
I think we should retain this line for now. Some build systems pass in DOWNLOAD_CDH_COMPONENTS=false, and it is good to know about it.


http://gerrit.cloudera.org:8080/#/c/13005/14/bin/impala-config.sh@226
PS14, Line 226:   export HIVE_HOME="$IMPALA_TOOLCHAIN/cdh_components-${CDH_BUILD_NUMBER}/hive-\
              : ${IMPALA_HIVE_VERSION}"
              :   export HIVE_METASTORE_THRIFT_DIR=$CDH_COMPONENTS_HOME/hive-\
              : ${IMPALA_HIVE_VERSION}/src/metastore/if
Please retain a way to build with HIVE_SRC_DIR_OVERRIDE, as some build systems rely on it to pass in a local Hive directory. It looks like it would have different meanings for Hive 2 vs Hive 3, but it might be ok to assume Hive 2 has the version we expected before and Hive 3 is pointing at the top level now. i.e.
Hive 2:
HIVE_METASTORE_THRIFT_DIR = HIVE_SRC_DIR/metastore/if

Hive 3:
HIVE_METASTORE_THRIFT_DIR = HIVE_SRC_DIR/standalone-metastore/src/main/thrift

Alternatively, we could limit HIVE_SRC_DIR_OVERRIDE to Hive 2 or use separate variables for Hive 2 vs Hive 3, but both are going to be useful to build systems. We could do something like a HIVE_METASTORE_THRIFT_DIR_OVERRIDE (but we would still want to allow HIVE_SRC_DIR_OVERRIDE to allow an easy transition).


http://gerrit.cloudera.org:8080/#/c/13005/14/tests/custom_cluster/test_permanent_udfs.py
File tests/custom_cluster/test_permanent_udfs.py:

http://gerrit.cloudera.org:8080/#/c/13005/14/tests/custom_cluster/test_permanent_udfs.py@502
PS14, Line 502:   # Sample java udfs from hive-exec.jar. Function name to symbol class mapping
              :   SAMPLE_JAVA_UDFS = [
              :       ('udfpi', 'org.apache.hadoop.hive.ql.udf.UDFPI'),
              :       ('udfbin', 'org.apache.hadoop.hive.ql.udf.UDFBin'),
              :       ('udfhex', 'org.apache.hadoop.hive.ql.udf.UDFHex'),
              :       ('udfconv', 'org.apache.hadoop.hive.ql.udf.UDFConv'),
              :       # TODO UDFHour was moved from UDF to GenericUDF in Hive 3
              :       # Add support for GenericUDFs to handle such cases
              :       # ('udfhour', 'org.apache.hadoop.hive.ql.udf.UDFHour'),
              :       ('udflike', 'org.apache.hadoop.hive.ql.udf.UDFLike'),
              :       ('udfsign', 'org.apache.hadoop.hive.ql.udf.UDFSign'),
              :       # TODO UDFYear moved to GenericUDF in Hive 3
              :       # Add support for GenericUDFs
              :       # ('udfyear', 'org.apache.hadoop.hive.ql.udf.UDFYear'),
              :       ('udfascii','org.apache.hadoop.hive.ql.udf.UDFAscii')
              :   ]
If these UDF tests work on Hive 2, then I'd like to keep them enabled. Either leave this unchanged (knowing that Hive 3 will need some modification later) or look at the environment variable and tailor this to Hive 2 vs Hive 3.



-- 
To view, visit http://gerrit.cloudera.org:8080/13005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436
Gerrit-Change-Number: 13005
Gerrit-PatchSet: 14
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Sudhanshu Arora <su...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 29 Apr 2019 22:53:49 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0

Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.

Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/13005 )

Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
......................................................................


Patch Set 20: Code-Review+2

This looks good. Thanks for getting this done.


-- 
To view, visit http://gerrit.cloudera.org:8080/13005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436
Gerrit-Change-Number: 13005
Gerrit-PatchSet: 20
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Sudhanshu Arora <su...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 30 Apr 2019 06:02:49 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0

Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.

Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/13005 )

Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
......................................................................


Patch Set 12:

(2 comments)

Starting to review. Couple quick comments.

http://gerrit.cloudera.org:8080/#/c/13005/12/bin/impala-config.sh
File bin/impala-config.sh:

http://gerrit.cloudera.org:8080/#/c/13005/12/bin/impala-config.sh@a1
PS12, Line 1: 
Restore this line (RAT checks will fail)


http://gerrit.cloudera.org:8080/#/c/13005/12/bin/impala-config.sh@513
PS12, Line 513: # The directory in which all the thirdparty CDP components live.
              : export CDP_COMPONENTS_HOME="$IMPALA_TOOLCHAIN/cdp_components-$CDP_BUILD_NUMBER"
You move this up, so you can remove this.



-- 
To view, visit http://gerrit.cloudera.org:8080/13005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436
Gerrit-Change-Number: 13005
Gerrit-PatchSet: 12
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Sudhanshu Arora <su...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 29 Apr 2019 18:10:41 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13005 )

Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
......................................................................


Patch Set 12:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/2971/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/13005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436
Gerrit-Change-Number: 13005
Gerrit-PatchSet: 12
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Sudhanshu Arora <su...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 29 Apr 2019 18:58:38 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13005 )

Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
......................................................................


Patch Set 18:

Build Failed 

https://jenkins.impala.io/job/gerrit-code-review-checks/2989/ : Initial code review checks failed. See linked job for details on the failure.


-- 
To view, visit http://gerrit.cloudera.org:8080/13005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436
Gerrit-Change-Number: 13005
Gerrit-PatchSet: 18
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Sudhanshu Arora <su...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 30 Apr 2019 01:44:29 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13005 )

Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
......................................................................


Patch Set 9: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4075/


-- 
To view, visit http://gerrit.cloudera.org:8080/13005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436
Gerrit-Change-Number: 13005
Gerrit-PatchSet: 9
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Sudhanshu Arora <su...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 26 Apr 2019 09:12:43 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13005 )

Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
......................................................................


Patch Set 9:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/2928/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/13005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436
Gerrit-Change-Number: 13005
Gerrit-PatchSet: 9
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Sudhanshu Arora <su...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 26 Apr 2019 02:50:23 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13005 )

Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
......................................................................


Patch Set 22: Code-Review+2


-- 
To view, visit http://gerrit.cloudera.org:8080/13005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436
Gerrit-Change-Number: 13005
Gerrit-PatchSet: 22
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Sudhanshu Arora <su...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 30 Apr 2019 16:30:25 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0

Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.

Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/13005 )

Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
......................................................................


Patch Set 3:

(3 comments)

Before more work goes into this, this is changing the Hive major version number and there should be a discussion on dev@ about this approach and how it fits into Impala releases.

http://gerrit.cloudera.org:8080/#/c/13005/2/bin/impala-config.sh
File bin/impala-config.sh:

http://gerrit.cloudera.org:8080/#/c/13005/2/bin/impala-config.sh@182
PS2, Line 182: if [ -d "$IMPALA_HOME/thirdparty" ]; then
             :   NO_THIRDPARTY=false
             : else
             :   NO_THIRDPARTY=true
             : fi
We don't use thirdparty anymore, so let's kill it off. DOWNLOAD_CDH_COMPONENTS should default to true (unless set from the environment).


http://gerrit.cloudera.org:8080/#/c/13005/2/bin/impala-config.sh@230
PS2, Line 230: . "$IMPALA_HOME/bin/impala-config-branch.sh"
             : if [ -f "$IMPALA_HOME/bin/impala-config-local.sh" ]; then
             :   . "$IMPALA_HOME/bin/impala-config-local.sh"
             : fi
We need to be careful about which variables are assigned before this and which are assigned after this. A common thing is for developers to customize IMPALA_TOOLCHAIN (needed for distcc).

For example, my bin/impala-config-local.sh has:
export IMPALA_TOOLCHAIN=/opt/Impala-Toolchain


http://gerrit.cloudera.org:8080/#/c/13005/2/shaded-deps/pom.xml
File shaded-deps/pom.xml:

http://gerrit.cloudera.org:8080/#/c/13005/2/shaded-deps/pom.xml@24
PS2, Line 24: ../../impala-parent/pom.xml
Think this should be ../impala-parent/pom.xml



-- 
To view, visit http://gerrit.cloudera.org:8080/13005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436
Gerrit-Change-Number: 13005
Gerrit-PatchSet: 3
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Tue, 16 Apr 2019 17:57:28 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13005 )

Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
......................................................................

IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0

The different set of dependencies are activated using maven profiles.
The activation of each profile is automatic based on the
IMPALA_HIVE_MAJOR_VERSION.

Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436
Reviewed-on: http://gerrit.cloudera.org:8080/13005
Reviewed-by: Joe McDonnell <jo...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M CMakeLists.txt
M README.md
M bin/bootstrap_toolchain.py
M bin/impala-config.sh
M common/thrift/.gitignore
M common/thrift/CMakeLists.txt
M fe/CMakeLists.txt
M fe/pom.xml
A fe/src/compat-hive-2/java/org/apache/impala/compat/MetastoreShim.java
A fe/src/compat-hive-3/java/org/apache/impala/compat/HiveMetadataFormatUtils.java
A fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/main/java/org/apache/impala/analysis/StringLiteral.java
M fe/src/main/java/org/apache/impala/catalog/FeHBaseTable.java
M fe/src/main/java/org/apache/impala/catalog/TableLoader.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java
D fe/src/main/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/main/java/org/apache/impala/hive/executor/UdfExecutor.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/DescribeResultFactory.java
M fe/src/main/java/org/apache/impala/service/MetadataOp.java
M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
M fe/src/test/java/org/apache/impala/hive/executor/UdfExecutorTest.java
M fe/src/test/java/org/apache/impala/testutil/EmbeddedMetastoreClientPool.java
M impala-parent/pom.xml
A shaded-deps/.gitignore
A shaded-deps/CMakeLists.txt
A shaded-deps/pom.xml
M testdata/bin/run-hive-server.sh
M tests/custom_cluster/test_permanent_udfs.py
31 files changed, 1,819 insertions(+), 468 deletions(-)

Approvals:
Joe McDonnell: Looks good to me, approved
Impala Public Jenkins: Verified

--
To view, visit http://gerrit.cloudera.org:8080/13005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436
Gerrit-Change-Number: 13005
Gerrit-PatchSet: 24
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Sudhanshu Arora <su...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>

[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13005 )

Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
......................................................................


Patch Set 10:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4088/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/13005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436
Gerrit-Change-Number: 13005
Gerrit-PatchSet: 10
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Sudhanshu Arora <su...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 26 Apr 2019 18:29:09 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13005 )

Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
......................................................................


Patch Set 23:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/3018/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/13005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436
Gerrit-Change-Number: 13005
Gerrit-PatchSet: 23
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Sudhanshu Arora <su...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 30 Apr 2019 21:51:08 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0

Posted by "Vihang Karajgaonkar (Code Review)" <ge...@cloudera.org>.

Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/13005 )

Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
......................................................................


Patch Set 12:

(11 comments)

Addressed review comments

http://gerrit.cloudera.org:8080/#/c/13005/9//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/13005/9//COMMIT_MSG@58
PS9, Line 58: 
> Is this flakiness specific to the hive 3 config? Or the races were already 
I checked and confirmed that the notification listener is working is generating the events as expected but there may be more to it. I will dig more. Even if there is a some problem with the event data, these races are still present can show up when Sentry is slow for any reason to update its entries.


http://gerrit.cloudera.org:8080/#/c/13005/9//COMMIT_MSG@65
PS9, Line 65: 
> maybe we shoudl disable these tests when running with hive 3 since we don't
So far, I would expect these tests work without any modifications on hive-2 builds (results pending for the last job I triggered). When we turn on cdp for jobs we should re-investigate how to fix these tests.


http://gerrit.cloudera.org:8080/#/c/13005/3/bin/impala-config.sh
File bin/impala-config.sh:

http://gerrit.cloudera.org:8080/#/c/13005/3/bin/impala-config.sh@167
PS3, Line 167: export IMPALA_RANGER_VERSION=1.2.0.6.0.99.0-45
> This number looks like a CDH_BUILD_NUMBER, and is probably from the same na
I was not aware of this.. Most of the pending patches are merged into official builds so we may not need this anymore.


http://gerrit.cloudera.org:8080/#/c/13005/3/bin/impala-config.sh@199
PS3, Line 199:   # TODO(todd) switch to an official build.
> line too long (91 > 90)
Done


http://gerrit.cloudera.org:8080/#/c/13005/3/bin/impala-config.sh@203
PS3, Line 203:   # CDH hive version is used to build and deploy in minicluster when USE_CDP_HIVE is
> line too long (92 > 90)
Done


http://gerrit.cloudera.org:8080/#/c/13005/3/bin/impala-config.sh@212
PS3, Line 212: fi
> line too long (106 > 90)
Done


http://gerrit.cloudera.org:8080/#/c/13005/9/bin/set-classpath.sh
File bin/set-classpath.sh:

http://gerrit.cloudera.org:8080/#/c/13005/9/bin/set-classpath.sh@30
PS9, Line 30: #"$IMPALA_HOME"/shaded-deps/target/impala-shaded-deps-0.1-SNAPSHOT.jar:\
> why is this necessary? shouldn't the shaded-deps dependency also end up in 
You are right, this is not necessary. removed it.


http://gerrit.cloudera.org:8080/#/c/13005/9/fe/src/main/java/org/apache/impala/catalog/MetaStoreClientPool.java
File fe/src/main/java/org/apache/impala/catalog/MetaStoreClientPool.java:

http://gerrit.cloudera.org:8080/#/c/13005/9/fe/src/main/java/org/apache/impala/catalog/MetaStoreClientPool.java@68
PS9, Line 68:         throws MetaException {
            :       return null;
            :     }
            :   };
            : 
            :   /**
> Are these changes actually used right now? I think this stuff ended up bein
Yeah. reverted the changes to this file in the latest patch


http://gerrit.cloudera.org:8080/#/c/13005/9/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java:

http://gerrit.cloudera.org:8080/#/c/13005/9/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@173
PS9, Line 173:   private static MetastoreEventsProcessor instance;
> Can you explain what's going on with this part of the change? Did Sentry mo
Sentry did not move to the JSONMessageFactory (and I believe this may be the reason why the test_ownership.py fails). However, in the shims approach we use the ExtendedJsonFactory for hive-2 builds and in hive-3 build we don't expect Sentry to be there. I will keep investigating while I debug cdp jobs.


http://gerrit.cloudera.org:8080/#/c/13005/9/testdata/bin/run-hive-server.sh
File testdata/bin/run-hive-server.sh:

http://gerrit.cloudera.org:8080/#/c/13005/9/testdata/bin/run-hive-server.sh@66
PS9, Line 66: export HIVE_METASTORE_HADOOP_OPTS="-Xdebug -Xrunjdwp:transport=dt_socket,server=y,\
> probably remove this
Done


http://gerrit.cloudera.org:8080/#/c/13005/9/testdata/bin/run-hive-server.sh@93
PS9, Line 93: if [ ${ONLY_METASTORE} -eq 0 ]; then
> this is in another patch- guess we can rebase this on top of that one to pi
Rebased my patch.



-- 
To view, visit http://gerrit.cloudera.org:8080/13005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436
Gerrit-Change-Number: 13005
Gerrit-PatchSet: 12
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Sudhanshu Arora <su...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 29 Apr 2019 20:59:24 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13005 )

Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
......................................................................


Patch Set 23: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/13005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436
Gerrit-Change-Number: 13005
Gerrit-PatchSet: 23
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Sudhanshu Arora <su...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 01 May 2019 03:27:41 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13005 )

Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
......................................................................


Patch Set 23:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4126/ DRY_RUN=false


-- 
To view, visit http://gerrit.cloudera.org:8080/13005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436
Gerrit-Change-Number: 13005
Gerrit-PatchSet: 23
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Sudhanshu Arora <su...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 30 Apr 2019 22:21:25 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13005 )

Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
......................................................................


Patch Set 12:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4102/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/13005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436
Gerrit-Change-Number: 13005
Gerrit-PatchSet: 12
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Sudhanshu Arora <su...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 29 Apr 2019 18:04:30 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0

Posted by "Vihang Karajgaonkar (Code Review)" <ge...@cloudera.org>.

Vihang Karajgaonkar has uploaded a new patch set (#9). ( http://gerrit.cloudera.org:8080/13005 )

Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
......................................................................

IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0

This change upgrades the hive dependencies of Impala to use Hive 3.1.0
based binaries. Most of the changes in this patch are based off patches
provided by Todd (links available in JIRA).

Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436
---
M CMakeLists.txt
M README.md
M bin/bootstrap_toolchain.py
M bin/impala-config.sh
M bin/set-classpath.sh
M common/thrift/.gitignore
M common/thrift/CMakeLists.txt
M fe/CMakeLists.txt
M fe/pom.xml
M fe/src/main/java/org/apache/impala/analysis/StringLiteral.java
M fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java
M fe/src/main/java/org/apache/impala/catalog/FeHBaseTable.java
M fe/src/main/java/org/apache/impala/catalog/MetaStoreClientPool.java
M fe/src/main/java/org/apache/impala/catalog/TableLoader.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java
M fe/src/main/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/main/java/org/apache/impala/hive/executor/UdfExecutor.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/DescribeResultFactory.java
M fe/src/main/java/org/apache/impala/service/MetadataOp.java
A fe/src/main/java/org/apache/impala/util/MetadataFormatUtils.java
M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
M fe/src/test/java/org/apache/impala/hive/executor/UdfExecutorTest.java
M fe/src/test/java/org/apache/impala/testutil/EmbeddedMetastoreClientPool.java
M impala-parent/pom.xml
A shaded-deps/.gitignore
A shaded-deps/CMakeLists.txt
A shaded-deps/pom.xml
M testdata/bin/run-hive-server.sh
M tests/authorization/test_owner_privileges.py
M tests/common/sentry_cache_test_suite.py
M tests/custom_cluster/test_permanent_udfs.py
34 files changed, 1,210 insertions(+), 294 deletions(-)

git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/05/13005/9
--
To view, visit http://gerrit.cloudera.org:8080/13005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436
Gerrit-Change-Number: 13005
Gerrit-PatchSet: 9
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Sudhanshu Arora <su...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>

[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0

Posted by "Vihang Karajgaonkar (Code Review)" <ge...@cloudera.org>.

Vihang Karajgaonkar has uploaded a new patch set (#21). ( http://gerrit.cloudera.org:8080/13005 )

Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
......................................................................

IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0

The different set of dependencies are activated using maven profiles.
The activation of each profile is automatic based on the
IMPALA_HIVE_MAJOR_VERSION.

git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/05/13005/21
--
To view, visit http://gerrit.cloudera.org:8080/13005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436
Gerrit-Change-Number: 13005
Gerrit-PatchSet: 21
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Sudhanshu Arora <su...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>

[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13005 )

Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
......................................................................


Patch Set 19:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/2990/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/13005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436
Gerrit-Change-Number: 13005
Gerrit-PatchSet: 19
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Sudhanshu Arora <su...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 30 Apr 2019 02:48:04 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13005 )

Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
......................................................................


Patch Set 5:

(7 comments)

http://gerrit.cloudera.org:8080/#/c/13005/5/bin/impala-config.sh
File bin/impala-config.sh:

http://gerrit.cloudera.org:8080/#/c/13005/5/bin/impala-config.sh@196
PS5, Line 196: # When USE_CDP_HIVE is set we use the latest hive version available to deply in minicluster
line too long (91 > 90)


http://gerrit.cloudera.org:8080/#/c/13005/5/bin/impala-config.sh@200
PS5, Line 200:   # TODO(Vihang) we should repackage the tarballs so that the src and binaries are extracted
line too long (92 > 90)


http://gerrit.cloudera.org:8080/#/c/13005/5/bin/impala-config.sh@209
PS5, Line 209:   export HIVE_HOME="$IMPALA_TOOLCHAIN/cdh_components-${CDH_BUILD_NUMBER}/hive-${MINICLUSTER_HIVE_VERSION}"
line too long (106 > 90)


http://gerrit.cloudera.org:8080/#/c/13005/5/bin/impala-config.sh@546
PS5, Line 546: export HIVE_METASTORE_THRIFT_DIR=$CDP_COMPONENTS_HOME/apache-hive-${IMPALA_HIVE_VERSION}-src/standalone-metastore/src/main/thrift
line too long (129 > 90)


http://gerrit.cloudera.org:8080/#/c/13005/5/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java:

http://gerrit.cloudera.org:8080/#/c/13005/5/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1073
PS5, Line 1073:                 
line has trailing whitespace


http://gerrit.cloudera.org:8080/#/c/13005/5/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1169
PS5, Line 1169:               
line has trailing whitespace


http://gerrit.cloudera.org:8080/#/c/13005/5/testdata/bin/run-hive-server.sh
File testdata/bin/run-hive-server.sh:

http://gerrit.cloudera.org:8080/#/c/13005/5/testdata/bin/run-hive-server.sh@66
PS5, Line 66: export HIVE_METASTORE_HADOOP_OPTS="-verbose:class -Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=30010"
line too long (121 > 90)



-- 
To view, visit http://gerrit.cloudera.org:8080/13005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436
Gerrit-Change-Number: 13005
Gerrit-PatchSet: 5
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Sudhanshu Arora <su...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 24 Apr 2019 02:07:16 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0

Posted by "Vihang Karajgaonkar (Code Review)" <ge...@cloudera.org>.

Vihang Karajgaonkar has uploaded a new patch set (#8). ( http://gerrit.cloudera.org:8080/13005 )

Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
......................................................................

IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0

This change upgrades the hive dependencies of Impala to use Hive 3.1.0
based binaries. Most of the changes in this patch are based off patches
provided by Todd (links available in JIRA).

Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436
---
M CMakeLists.txt
M README.md
M bin/bootstrap_toolchain.py
M bin/impala-config.sh
M bin/set-classpath.sh
M common/thrift/.gitignore
M common/thrift/CMakeLists.txt
M fe/CMakeLists.txt
M fe/pom.xml
M fe/src/main/java/org/apache/impala/analysis/StringLiteral.java
M fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java
M fe/src/main/java/org/apache/impala/catalog/FeHBaseTable.java
M fe/src/main/java/org/apache/impala/catalog/MetaStoreClientPool.java
M fe/src/main/java/org/apache/impala/catalog/TableLoader.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java
M fe/src/main/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/main/java/org/apache/impala/hive/executor/UdfExecutor.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/DescribeResultFactory.java
M fe/src/main/java/org/apache/impala/service/MetadataOp.java
A fe/src/main/java/org/apache/impala/util/MetadataFormatUtils.java
M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
M fe/src/test/java/org/apache/impala/hive/executor/UdfExecutorTest.java
M fe/src/test/java/org/apache/impala/testutil/EmbeddedMetastoreClientPool.java
M impala-parent/pom.xml
A shaded-deps/.gitignore
A shaded-deps/CMakeLists.txt
A shaded-deps/pom.xml
M testdata/bin/run-hive-server.sh
M tests/authorization/test_owner_privileges.py
M tests/common/sentry_cache_test_suite.py
M tests/custom_cluster/test_permanent_udfs.py
34 files changed, 1,208 insertions(+), 294 deletions(-)

git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/05/13005/8
--
To view, visit http://gerrit.cloudera.org:8080/13005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436
Gerrit-Change-Number: 13005
Gerrit-PatchSet: 8
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Sudhanshu Arora <su...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>

[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13005 )

Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
......................................................................


Patch Set 8:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/2921/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/13005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436
Gerrit-Change-Number: 13005
Gerrit-PatchSet: 8
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Sudhanshu Arora <su...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 25 Apr 2019 21:27:17 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0

Posted by "Vihang Karajgaonkar (Code Review)" <ge...@cloudera.org>.

Vihang Karajgaonkar has uploaded a new patch set (#16). ( http://gerrit.cloudera.org:8080/13005 )

Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
......................................................................

IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0

The different set of dependencies are activated using maven profiles.
The activation of each profile is automatic based on the
IMPALA_HIVE_MAJOR_VERSION.

git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/05/13005/16
--
To view, visit http://gerrit.cloudera.org:8080/13005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436
Gerrit-Change-Number: 13005
Gerrit-PatchSet: 16
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Sudhanshu Arora <su...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>

[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13005 )

Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
......................................................................


Patch Set 2:

Build Failed 

https://jenkins.impala.io/job/gerrit-code-review-checks/2767/ : Initial code review checks failed. See linked job for details on the failure.


-- 
To view, visit http://gerrit.cloudera.org:8080/13005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436
Gerrit-Change-Number: 13005
Gerrit-PatchSet: 2
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Sat, 13 Apr 2019 01:09:52 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13005 )

Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
......................................................................


Patch Set 8:

(25 comments)

http://gerrit.cloudera.org:8080/#/c/13005/8/bin/impala-config.sh
File bin/impala-config.sh:

http://gerrit.cloudera.org:8080/#/c/13005/8/bin/impala-config.sh@196
PS8, Line 196: # When USE_CDP_HIVE is set we use the latest hive version available to deply in minicluster
line too long (91 > 90)


http://gerrit.cloudera.org:8080/#/c/13005/8/bin/impala-config.sh@200
PS8, Line 200:   # TODO(Vihang) we should repackage the tarballs so that the src and binaries are extracted
line too long (92 > 90)


http://gerrit.cloudera.org:8080/#/c/13005/8/bin/impala-config.sh@209
PS8, Line 209:   export HIVE_HOME="$IMPALA_TOOLCHAIN/cdh_components-${CDH_BUILD_NUMBER}/hive-${MINICLUSTER_HIVE_VERSION}"
line too long (106 > 90)


http://gerrit.cloudera.org:8080/#/c/13005/8/bin/impala-config.sh@546
PS8, Line 546: export HIVE_METASTORE_THRIFT_DIR=$CDP_COMPONENTS_HOME/apache-hive-${IMPALA_HIVE_VERSION}-src/standalone-metastore/src/main/thrift
line too long (129 > 90)


http://gerrit.cloudera.org:8080/#/c/13005/8/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java:

http://gerrit.cloudera.org:8080/#/c/13005/8/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1073
PS8, Line 1073:                 
line has trailing whitespace


http://gerrit.cloudera.org:8080/#/c/13005/8/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1169
PS8, Line 1169:               
line has trailing whitespace


http://gerrit.cloudera.org:8080/#/c/13005/8/testdata/bin/run-hive-server.sh
File testdata/bin/run-hive-server.sh:

http://gerrit.cloudera.org:8080/#/c/13005/8/testdata/bin/run-hive-server.sh@66
PS8, Line 66: export HIVE_METASTORE_HADOOP_OPTS="-verbose:class -Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=30010"
line too long (121 > 90)


http://gerrit.cloudera.org:8080/#/c/13005/8/tests/authorization/test_owner_privileges.py
File tests/authorization/test_owner_privileges.py:

http://gerrit.cloudera.org:8080/#/c/13005/8/tests/authorization/test_owner_privileges.py@103
PS8, Line 103: )
flake8: E501 line too long (91 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/13005/8/tests/authorization/test_owner_privileges.py@179
PS8, Line 179: =
flake8: E501 line too long (93 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/13005/8/tests/authorization/test_owner_privileges.py@182
PS8, Line 182: =
flake8: E501 line too long (93 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/13005/8/tests/authorization/test_owner_privileges.py@186
PS8, Line 186: )
flake8: E501 line too long (91 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/13005/8/tests/authorization/test_owner_privileges.py@193
PS8, Line 193: ,
flake8: E501 line too long (91 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/13005/8/tests/authorization/test_owner_privileges.py@213
PS8, Line 213: )
flake8: E501 line too long (91 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/13005/8/tests/authorization/test_owner_privileges.py@226
PS8, Line 226: )
flake8: E501 line too long (91 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/13005/8/tests/authorization/test_owner_privileges.py@236
PS8, Line 236: )
flake8: E501 line too long (91 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/13005/8/tests/authorization/test_owner_privileges.py@282
PS8, Line 282: _
flake8: E501 line too long (100 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/13005/8/tests/authorization/test_owner_privileges.py@287
PS8, Line 287: t
flake8: E501 line too long (101 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/13005/8/tests/authorization/test_owner_privileges.py@293
PS8, Line 293: 2
flake8: E501 line too long (92 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/13005/8/tests/authorization/test_owner_privileges.py@355
PS8, Line 355: _
flake8: E501 line too long (100 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/13005/8/tests/authorization/test_owner_privileges.py@360
PS8, Line 360: t
flake8: E501 line too long (101 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/13005/8/tests/authorization/test_owner_privileges.py@364
PS8, Line 364: 2
flake8: E501 line too long (92 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/13005/8/tests/authorization/test_owner_privileges.py@368
PS8, Line 368: )
flake8: E501 line too long (91 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/13005/8/tests/authorization/test_owner_privileges.py@412
PS8, Line 412: t
flake8: E501 line too long (101 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/13005/8/tests/authorization/test_owner_privileges.py@421
PS8, Line 421: i
flake8: E501 line too long (102 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/13005/8/tests/common/sentry_cache_test_suite.py
File tests/common/sentry_cache_test_suite.py:

http://gerrit.cloudera.org:8080/#/c/13005/8/tests/common/sentry_cache_test_suite.py@83
PS8, Line 83: }
flake8: E501 line too long (92 > 90 characters)



-- 
To view, visit http://gerrit.cloudera.org:8080/13005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436
Gerrit-Change-Number: 13005
Gerrit-PatchSet: 8
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Sudhanshu Arora <su...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 25 Apr 2019 20:32:53 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0

Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.

Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/13005 )

Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
......................................................................


Patch Set 19:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/13005/19/bin/impala-config.sh
File bin/impala-config.sh:

http://gerrit.cloudera.org:8080/#/c/13005/19/bin/impala-config.sh@175
PS19, Line 175: export CDP_HIVE_VERSION=3.1.0.6.0.99.0-38-0e7f6337a50
Nitpicking: add a comment here that this is a custom Hive build



-- 
To view, visit http://gerrit.cloudera.org:8080/13005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436
Gerrit-Change-Number: 13005
Gerrit-PatchSet: 19
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Sudhanshu Arora <su...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 30 Apr 2019 05:12:40 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0

Posted by "Vihang Karajgaonkar (Code Review)" <ge...@cloudera.org>.

Vihang Karajgaonkar has uploaded a new patch set (#10). ( http://gerrit.cloudera.org:8080/13005 )

Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
......................................................................

IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0

This change upgrades the hive dependencies of Impala to use Hive 3.1.0
based binaries. Most of the changes in this patch are based off patches
provided by Todd (links available in JIRA).

Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436
---
M CMakeLists.txt
M README.md
M bin/bootstrap_toolchain.py
M bin/impala-config.sh
M bin/set-classpath.sh
M common/thrift/.gitignore
M common/thrift/CMakeLists.txt
M fe/CMakeLists.txt
M fe/pom.xml
M fe/src/main/java/org/apache/impala/analysis/StringLiteral.java
M fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java
M fe/src/main/java/org/apache/impala/catalog/FeHBaseTable.java
M fe/src/main/java/org/apache/impala/catalog/MetaStoreClientPool.java
M fe/src/main/java/org/apache/impala/catalog/TableLoader.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java
M fe/src/main/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/main/java/org/apache/impala/hive/executor/UdfExecutor.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/DescribeResultFactory.java
M fe/src/main/java/org/apache/impala/service/MetadataOp.java
A fe/src/main/java/org/apache/impala/util/MetadataFormatUtils.java
M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
M fe/src/test/java/org/apache/impala/hive/executor/UdfExecutorTest.java
M fe/src/test/java/org/apache/impala/testutil/EmbeddedMetastoreClientPool.java
M impala-parent/pom.xml
A shaded-deps/.gitignore
A shaded-deps/CMakeLists.txt
A shaded-deps/pom.xml
M testdata/bin/run-hive-server.sh
M tests/authorization/test_owner_privileges.py
M tests/common/sentry_cache_test_suite.py
M tests/custom_cluster/test_permanent_udfs.py
34 files changed, 1,220 insertions(+), 294 deletions(-)

git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/05/13005/10
--
To view, visit http://gerrit.cloudera.org:8080/13005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436
Gerrit-Change-Number: 13005
Gerrit-PatchSet: 10
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Sudhanshu Arora <su...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>

[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0

Posted by "Vihang Karajgaonkar (Code Review)" <ge...@cloudera.org>.

Vihang Karajgaonkar has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/13005 )

Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
......................................................................

IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0

This change upgrades the hive dependencies of Impala to use Hive 3.1.0
based binaries. Most of the changes in this patch are based off patches
provided by Todd (links available in JIRA).

Upgrading the dependencies allows us to work with both Hive 3.1.0 and
Hive 2.1.0 in the same code line. In order to do this, the patch
trims down a lot of unnecessary hive dependencies of the front end code
by creating a shaded-deps module. The pom.xml of shaded-deps includes
only the files from Hive source which Impala depends for compilation.

Additionally, it also uses a custom build of Hive which is based of
Hive 3.1.0. This custom build includes patches for HIVE-21596 and
HIVE-21586 which are needed by Impala so that it can compile against
Hive-3 libraries and be able to talk to HMS-2.x metastore. Once these
patches are merged we can get rid of this custom build and rely on more
official sources of Hive builds.

The patch also changes impala-config.sh so that it always downloads the
Hive-3 libraries from the toolchain. The code is always built using
Hive-3 jars. However, based on the value of USE_CDP_HIVE, the
minicluster is deployed using Hive-3 or Hive-2 binaries. Since Impala
implements HiveServer2's TCLIService.thrift interface, it requires us to
use the existing mechanism of copying the hive-2/api TCLIService.thrift to
hive-3/api. It also adds a few environment variables which point to the
metastore's thrift file and the CDH Hive version.

Testing:
1. Code compiles and runs against both HMS-3 and HMS-2

Notes:
Testing is still a WIP. Will trigger a full-suite of tests for both when
USE_CDP_HIVE is true and USE_CDP_HIVE is false. Will update the patch as
and when the issues are found in the tests.

Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436
---
M CMakeLists.txt
M README.md
M bin/bootstrap_toolchain.py
M bin/impala-config.sh
M bin/set-classpath.sh
M common/thrift/.gitignore
M common/thrift/CMakeLists.txt
M fe/CMakeLists.txt
M fe/pom.xml
M fe/src/main/java/org/apache/impala/analysis/StringLiteral.java
M fe/src/main/java/org/apache/impala/catalog/FeHBaseTable.java
M fe/src/main/java/org/apache/impala/catalog/TableLoader.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java
M fe/src/main/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/main/java/org/apache/impala/hive/executor/UdfExecutor.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/DescribeResultFactory.java
M fe/src/main/java/org/apache/impala/service/MetadataOp.java
A fe/src/main/java/org/apache/impala/util/MetadataFormatUtils.java
M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
M fe/src/test/java/org/apache/impala/hive/executor/UdfExecutorTest.java
M impala-parent/pom.xml
A shaded-deps/.gitignore
A shaded-deps/CMakeLists.txt
A shaded-deps/pom.xml
27 files changed, 1,050 insertions(+), 268 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/05/13005/3
-- 
To view, visit http://gerrit.cloudera.org:8080/13005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436
Gerrit-Change-Number: 13005
Gerrit-PatchSet: 3
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>

[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13005 )

Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
......................................................................


Patch Set 7:

(25 comments)

http://gerrit.cloudera.org:8080/#/c/13005/7/bin/impala-config.sh
File bin/impala-config.sh:

http://gerrit.cloudera.org:8080/#/c/13005/7/bin/impala-config.sh@196
PS7, Line 196: # When USE_CDP_HIVE is set we use the latest hive version available to deply in minicluster
line too long (91 > 90)


http://gerrit.cloudera.org:8080/#/c/13005/7/bin/impala-config.sh@200
PS7, Line 200:   # TODO(Vihang) we should repackage the tarballs so that the src and binaries are extracted
line too long (92 > 90)


http://gerrit.cloudera.org:8080/#/c/13005/7/bin/impala-config.sh@209
PS7, Line 209:   export HIVE_HOME="$IMPALA_TOOLCHAIN/cdh_components-${CDH_BUILD_NUMBER}/hive-${MINICLUSTER_HIVE_VERSION}"
line too long (106 > 90)


http://gerrit.cloudera.org:8080/#/c/13005/7/bin/impala-config.sh@546
PS7, Line 546: export HIVE_METASTORE_THRIFT_DIR=$CDP_COMPONENTS_HOME/apache-hive-${IMPALA_HIVE_VERSION}-src/standalone-metastore/src/main/thrift
line too long (129 > 90)


http://gerrit.cloudera.org:8080/#/c/13005/7/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java:

http://gerrit.cloudera.org:8080/#/c/13005/7/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1073
PS7, Line 1073:                 
line has trailing whitespace


http://gerrit.cloudera.org:8080/#/c/13005/7/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1169
PS7, Line 1169:               
line has trailing whitespace


http://gerrit.cloudera.org:8080/#/c/13005/7/testdata/bin/run-hive-server.sh
File testdata/bin/run-hive-server.sh:

http://gerrit.cloudera.org:8080/#/c/13005/7/testdata/bin/run-hive-server.sh@66
PS7, Line 66: export HIVE_METASTORE_HADOOP_OPTS="-verbose:class -Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=30010"
line too long (121 > 90)


http://gerrit.cloudera.org:8080/#/c/13005/7/tests/authorization/test_owner_privileges.py
File tests/authorization/test_owner_privileges.py:

http://gerrit.cloudera.org:8080/#/c/13005/7/tests/authorization/test_owner_privileges.py@103
PS7, Line 103: )
flake8: E501 line too long (91 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/13005/7/tests/authorization/test_owner_privileges.py@179
PS7, Line 179: =
flake8: E501 line too long (93 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/13005/7/tests/authorization/test_owner_privileges.py@182
PS7, Line 182: =
flake8: E501 line too long (93 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/13005/7/tests/authorization/test_owner_privileges.py@186
PS7, Line 186: )
flake8: E501 line too long (91 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/13005/7/tests/authorization/test_owner_privileges.py@193
PS7, Line 193: ,
flake8: E501 line too long (91 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/13005/7/tests/authorization/test_owner_privileges.py@213
PS7, Line 213: )
flake8: E501 line too long (91 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/13005/7/tests/authorization/test_owner_privileges.py@226
PS7, Line 226: )
flake8: E501 line too long (91 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/13005/7/tests/authorization/test_owner_privileges.py@236
PS7, Line 236: )
flake8: E501 line too long (91 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/13005/7/tests/authorization/test_owner_privileges.py@282
PS7, Line 282: _
flake8: E501 line too long (100 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/13005/7/tests/authorization/test_owner_privileges.py@287
PS7, Line 287: t
flake8: E501 line too long (101 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/13005/7/tests/authorization/test_owner_privileges.py@293
PS7, Line 293: 2
flake8: E501 line too long (92 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/13005/7/tests/authorization/test_owner_privileges.py@355
PS7, Line 355: _
flake8: E501 line too long (100 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/13005/7/tests/authorization/test_owner_privileges.py@360
PS7, Line 360: t
flake8: E501 line too long (101 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/13005/7/tests/authorization/test_owner_privileges.py@364
PS7, Line 364: 2
flake8: E501 line too long (92 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/13005/7/tests/authorization/test_owner_privileges.py@368
PS7, Line 368: )
flake8: E501 line too long (91 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/13005/7/tests/authorization/test_owner_privileges.py@412
PS7, Line 412: t
flake8: E501 line too long (101 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/13005/7/tests/authorization/test_owner_privileges.py@421
PS7, Line 421: i
flake8: E501 line too long (102 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/13005/7/tests/common/sentry_cache_test_suite.py
File tests/common/sentry_cache_test_suite.py:

http://gerrit.cloudera.org:8080/#/c/13005/7/tests/common/sentry_cache_test_suite.py@83
PS7, Line 83: }
flake8: E501 line too long (92 > 90 characters)



-- 
To view, visit http://gerrit.cloudera.org:8080/13005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436
Gerrit-Change-Number: 13005
Gerrit-PatchSet: 7
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Sudhanshu Arora <su...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 25 Apr 2019 20:25:56 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13005 )

Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
......................................................................


Patch Set 7:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/2919/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/13005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436
Gerrit-Change-Number: 13005
Gerrit-PatchSet: 7
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Sudhanshu Arora <su...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 25 Apr 2019 21:19:03 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13005 )

Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
......................................................................


Patch Set 9:

(25 comments)

http://gerrit.cloudera.org:8080/#/c/13005/9/bin/impala-config.sh
File bin/impala-config.sh:

http://gerrit.cloudera.org:8080/#/c/13005/9/bin/impala-config.sh@196
PS9, Line 196: # When USE_CDP_HIVE is set we use the latest hive version available to deply in minicluster
line too long (91 > 90)


http://gerrit.cloudera.org:8080/#/c/13005/9/bin/impala-config.sh@200
PS9, Line 200:   # TODO(Vihang) we should repackage the tarballs so that the src and binaries are extracted
line too long (92 > 90)


http://gerrit.cloudera.org:8080/#/c/13005/9/bin/impala-config.sh@209
PS9, Line 209:   export HIVE_HOME="$IMPALA_TOOLCHAIN/cdh_components-${CDH_BUILD_NUMBER}/hive-${MINICLUSTER_HIVE_VERSION}"
line too long (106 > 90)


http://gerrit.cloudera.org:8080/#/c/13005/9/bin/impala-config.sh@546
PS9, Line 546: export HIVE_METASTORE_THRIFT_DIR=$CDP_COMPONENTS_HOME/apache-hive-${IMPALA_HIVE_VERSION}-src/standalone-metastore/src/main/thrift
line too long (129 > 90)


http://gerrit.cloudera.org:8080/#/c/13005/9/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java:

http://gerrit.cloudera.org:8080/#/c/13005/9/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1073
PS9, Line 1073:                 
line has trailing whitespace


http://gerrit.cloudera.org:8080/#/c/13005/9/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1169
PS9, Line 1169:               
line has trailing whitespace


http://gerrit.cloudera.org:8080/#/c/13005/9/testdata/bin/run-hive-server.sh
File testdata/bin/run-hive-server.sh:

http://gerrit.cloudera.org:8080/#/c/13005/9/testdata/bin/run-hive-server.sh@66
PS9, Line 66: export HIVE_METASTORE_HADOOP_OPTS="-verbose:class -Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=30010"
line too long (121 > 90)


http://gerrit.cloudera.org:8080/#/c/13005/9/tests/authorization/test_owner_privileges.py
File tests/authorization/test_owner_privileges.py:

http://gerrit.cloudera.org:8080/#/c/13005/9/tests/authorization/test_owner_privileges.py@103
PS9, Line 103: )
flake8: E501 line too long (91 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/13005/9/tests/authorization/test_owner_privileges.py@179
PS9, Line 179: =
flake8: E501 line too long (93 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/13005/9/tests/authorization/test_owner_privileges.py@182
PS9, Line 182: =
flake8: E501 line too long (93 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/13005/9/tests/authorization/test_owner_privileges.py@186
PS9, Line 186: )
flake8: E501 line too long (91 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/13005/9/tests/authorization/test_owner_privileges.py@193
PS9, Line 193: ,
flake8: E501 line too long (91 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/13005/9/tests/authorization/test_owner_privileges.py@213
PS9, Line 213: )
flake8: E501 line too long (91 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/13005/9/tests/authorization/test_owner_privileges.py@226
PS9, Line 226: )
flake8: E501 line too long (91 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/13005/9/tests/authorization/test_owner_privileges.py@236
PS9, Line 236: )
flake8: E501 line too long (91 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/13005/9/tests/authorization/test_owner_privileges.py@282
PS9, Line 282: _
flake8: E501 line too long (100 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/13005/9/tests/authorization/test_owner_privileges.py@287
PS9, Line 287: t
flake8: E501 line too long (101 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/13005/9/tests/authorization/test_owner_privileges.py@293
PS9, Line 293: 2
flake8: E501 line too long (92 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/13005/9/tests/authorization/test_owner_privileges.py@355
PS9, Line 355: _
flake8: E501 line too long (100 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/13005/9/tests/authorization/test_owner_privileges.py@360
PS9, Line 360: t
flake8: E501 line too long (101 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/13005/9/tests/authorization/test_owner_privileges.py@364
PS9, Line 364: 2
flake8: E501 line too long (92 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/13005/9/tests/authorization/test_owner_privileges.py@368
PS9, Line 368: )
flake8: E501 line too long (91 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/13005/9/tests/authorization/test_owner_privileges.py@412
PS9, Line 412: t
flake8: E501 line too long (101 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/13005/9/tests/authorization/test_owner_privileges.py@421
PS9, Line 421: i
flake8: E501 line too long (102 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/13005/9/tests/common/sentry_cache_test_suite.py
File tests/common/sentry_cache_test_suite.py:

http://gerrit.cloudera.org:8080/#/c/13005/9/tests/common/sentry_cache_test_suite.py@83
PS9, Line 83: }
flake8: E501 line too long (92 > 90 characters)



-- 
To view, visit http://gerrit.cloudera.org:8080/13005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436
Gerrit-Change-Number: 13005
Gerrit-PatchSet: 9
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Sudhanshu Arora <su...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 26 Apr 2019 02:17:58 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0

Posted by "Todd Lipcon (Code Review)" <ge...@cloudera.org>.

Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/13005 )

Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
......................................................................


Patch Set 19: Code-Review+1


-- 
To view, visit http://gerrit.cloudera.org:8080/13005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436
Gerrit-Change-Number: 13005
Gerrit-PatchSet: 19
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Sudhanshu Arora <su...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 30 Apr 2019 04:31:54 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13005 )

Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
......................................................................


Patch Set 18:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/13005/18/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
File fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java:

http://gerrit.cloudera.org:8080/#/c/13005/18/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@283
PS18, Line 283:         if (currentChar == '\\' && (i + 6 < stringLiteral.length()) && stringLiteral.charAt(i + 1) == 'u') {
line too long (108 > 90)



-- 
To view, visit http://gerrit.cloudera.org:8080/13005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436
Gerrit-Change-Number: 13005
Gerrit-PatchSet: 18
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Sudhanshu Arora <su...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 30 Apr 2019 01:29:27 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0

Posted by "Todd Lipcon (Code Review)" <ge...@cloudera.org>.

Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/13005 )

Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
......................................................................


Patch Set 9:

(7 comments)

Took a look through this. I'll see if I can get some time tomorrow to try out a "shims" approach as well for comparison.

It looks like the majority of the changes here, though, are pretty innocuous (eg copying some stuff into our code where we were relying on hive-internal stuff) and could be made either way.

http://gerrit.cloudera.org:8080/#/c/13005/9//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/13005/9//COMMIT_MSG@58
PS9, Line 58: 4. The Sentry object ownership tests are flaky. There are
Is this flakiness specific to the hive 3 config? Or the races were already there? Is the problem here that we don't have the notification listener working properly so we're relying on some periodic sync instead of notifications?


http://gerrit.cloudera.org:8080/#/c/13005/9//COMMIT_MSG@65
PS9, Line 65: work-around these issues.
maybe we shoudl disable these tests when running with hive 3 since we don't anticipate using Sentry with Hive 3 anyway


http://gerrit.cloudera.org:8080/#/c/13005/9/bin/set-classpath.sh
File bin/set-classpath.sh:

http://gerrit.cloudera.org:8080/#/c/13005/9/bin/set-classpath.sh@30
PS9, Line 30: "$IMPALA_HOME"/shaded-deps/target/impala-shaded-deps-0.1-SNAPSHOT.jar:\
why is this necessary? shouldn't the shaded-deps dependency also end up in target/dependency/ like the others handled on line 35?


http://gerrit.cloudera.org:8080/#/c/13005/9/fe/src/main/java/org/apache/impala/catalog/MetaStoreClientPool.java
File fe/src/main/java/org/apache/impala/catalog/MetaStoreClientPool.java:

http://gerrit.cloudera.org:8080/#/c/13005/9/fe/src/main/java/org/apache/impala/catalog/MetaStoreClientPool.java@68
PS9, Line 68:   private List<Integer> serverVersion = null;
            : 
            :   private static final Ordering<Iterable<Integer>> VERSION_ORDERING =
            :       Ordering.<Integer>natural().lexicographical().nullsLast();
            :   private static final ImmutableList<Integer> CATALOGS_VERSION =
            :       ImmutableList.of(3, 0, 0);
Are these changes actually used right now? I think this stuff ended up being in the Hive impl rather than being doable in this wrapper


http://gerrit.cloudera.org:8080/#/c/13005/9/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java:

http://gerrit.cloudera.org:8080/#/c/13005/9/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@173
PS9, Line 173:       MessageFactory.getDefaultInstance(new HiveConf()).getDeserializer();
Can you explain what's going on with this part of the change? Did Sentry move to the normal JSONMessageFactory?


http://gerrit.cloudera.org:8080/#/c/13005/9/testdata/bin/run-hive-server.sh
File testdata/bin/run-hive-server.sh:

http://gerrit.cloudera.org:8080/#/c/13005/9/testdata/bin/run-hive-server.sh@66
PS9, Line 66: export HIVE_METASTORE_HADOOP_OPTS="-verbose:class -Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=30010"
probably remove this


http://gerrit.cloudera.org:8080/#/c/13005/9/testdata/bin/run-hive-server.sh@93
PS9, Line 93:   # For Hive 3, we use Tez for execution. We have to add it to the HS2 classpath.
this is in another patch- guess we can rebase this on top of that one to pick it up



-- 
To view, visit http://gerrit.cloudera.org:8080/13005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436
Gerrit-Change-Number: 13005
Gerrit-PatchSet: 9
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Sudhanshu Arora <su...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 26 Apr 2019 06:51:52 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13005 )

Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
......................................................................


Patch Set 8: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4071/


-- 
To view, visit http://gerrit.cloudera.org:8080/13005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436
Gerrit-Change-Number: 13005
Gerrit-PatchSet: 8
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Sudhanshu Arora <su...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 26 Apr 2019 01:23:15 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13005 )

Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
......................................................................


Patch Set 19:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/13005/19/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
File fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java:

http://gerrit.cloudera.org:8080/#/c/13005/19/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@283
PS19, Line 283:         if (currentChar == '\\' && (i + 6 < stringLiteral.length()) && stringLiteral.charAt(i + 1) == 'u') {
line too long (108 > 90)



-- 
To view, visit http://gerrit.cloudera.org:8080/13005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436
Gerrit-Change-Number: 13005
Gerrit-PatchSet: 19
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Sudhanshu Arora <su...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 30 Apr 2019 02:02:37 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13005 )

Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
......................................................................


Patch Set 16:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/13005/16/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
File fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java:

http://gerrit.cloudera.org:8080/#/c/13005/16/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@283
PS16, Line 283:         if (currentChar == '\\' && (i + 6 < stringLiteral.length()) && stringLiteral.charAt(i + 1) == 'u') {
line too long (108 > 90)



-- 
To view, visit http://gerrit.cloudera.org:8080/13005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436
Gerrit-Change-Number: 13005
Gerrit-PatchSet: 16
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Sudhanshu Arora <su...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 30 Apr 2019 01:04:53 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13005 )

Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
......................................................................


Patch Set 21:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/2997/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/13005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436
Gerrit-Change-Number: 13005
Gerrit-PatchSet: 21
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Sudhanshu Arora <su...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 30 Apr 2019 07:13:04 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13005 )

Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
......................................................................


Patch Set 11:

Build Failed 

https://jenkins.impala.io/job/gerrit-code-review-checks/2947/ : Initial code review checks failed. See linked job for details on the failure.


-- 
To view, visit http://gerrit.cloudera.org:8080/13005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436
Gerrit-Change-Number: 13005
Gerrit-PatchSet: 11
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Sudhanshu Arora <su...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Sat, 27 Apr 2019 02:59:13 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0

Posted by "Philip Zeyliger (Code Review)" <ge...@cloudera.org>.

Philip Zeyliger has posted comments on this change. ( http://gerrit.cloudera.org:8080/13005 )

Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
......................................................................


Patch Set 3:

(1 comment)

Nice work.

When I did the transitional work between Hive 1 and Hive 2, I introduced a variable that switched at build time between the two worlds. (See a203733fac3e1e37df8abeee39a88d187153a8c5 for the revert and "git log --grep IMPALA_MINICLUSTER_PROFILE")

If I'm understanding right, the approach here is to produce a single "binary" that works for both worlds? Or at run time do the "original" Hive jars get run? I think both approaches are plausible; just want to make sure we're clear about it.

(Is the shading slow? I've seen maven-shade-plugin be very slow...)

http://gerrit.cloudera.org:8080/#/c/13005/3/bin/impala-config.sh
File bin/impala-config.sh:

http://gerrit.cloudera.org:8080/#/c/13005/3/bin/impala-config.sh@167
PS3, Line 167: export CDP_BUILD_NUMBER=9999994
This number looks like a CDH_BUILD_NUMBER, and is probably from the same name space, but looks like something you hand-wrote. You can ask the same oracle that CDH_BUILD_NUMBER uses for your own number which is guaranteed not to conflict in the future.



-- 
To view, visit http://gerrit.cloudera.org:8080/13005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436
Gerrit-Change-Number: 13005
Gerrit-PatchSet: 3
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Sudhanshu Arora <su...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 23 Apr 2019 17:10:25 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0

Posted by "Vihang Karajgaonkar (Code Review)" <ge...@cloudera.org>.

Vihang Karajgaonkar has uploaded a new patch set (#19). ( http://gerrit.cloudera.org:8080/13005 )

Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
......................................................................

IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0

The different set of dependencies are activated using maven profiles.
The activation of each profile is automatic based on the
IMPALA_HIVE_MAJOR_VERSION.

git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/05/13005/19
--
To view, visit http://gerrit.cloudera.org:8080/13005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436
Gerrit-Change-Number: 13005
Gerrit-PatchSet: 19
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Sudhanshu Arora <su...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>