You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Fucun Chu (Code Review)" <ge...@cloudera.org> on 2021/08/21 00:51:44 UTC

[Impala-ASF-CR] IMPALA-10870: Add Apache Hive 3.1.2 to the minicluster 

Fucun Chu has uploaded this change for review. ( http://gerrit.cloudera.org:8080/17793


Change subject: IMPALA-10870: Add Apache Hive 3.1.2 to the minicluster
......................................................................

IMPALA-10870: Add Apache Hive 3.1.2 to the minicluster

This patch modifies the minicluster script to optionally use Apache
Hive 3.1.2 instead of CDP Hive 3.1.3.

In order to make sure that existing setups don't break this is
enabled via a environment variable override to bin/impala-config.sh.
When the environment variable USE_APACHE_HIVE is set to true the
bootstrap_toolchain script downloads Apache Hive 3.1.2 tarballs and
extracts it in the toolchain directory. These binaries are used to
start the Hive services (Hiveserver2 and metastore). The default is
CDP Hive 3.1.3

Since CDP Hive 3 uses some features of Apache Hive 4, this patch uses
a different database name so that it is easy to switch from working
from one environment which uses CDP Hive 3.1.3 metastore to another
which usese Apache Hive 3.1.2 metastore.

In order to start a minicluster which uses Apache Hive 3.1.2 users
should follow the steps below:

1. Make sure that minicluster, if running, is stopped before you run
the following commands.
2. Open a new terminal and run following commands.
> export USE_APACHE_HIVE=true
> source bin/impala-config.sh
> bin/bootstrap_toolchain.py
  The above command downloads the Apache Hive 3.1.2 tarballs and
extracts them in toolchain/apache_components directory.

> rm $HIVE_HOME/lib/guava-*jar
> cp $HADOOP_HOME/share/hadoop/hdfs/lib/guava-*.jar $HIVE_HOME/lib/
  The above command is to fix HIVE-22915

> bin/create-test-configuration.sh -create_metastore
  The above step should provide "-create-metastore" only the first time
so that a new metastore db is created and the Apache Hive 3.1.2 schema is
initialized.

> testdata/bin/run-all.sh

Follow-up:
 - Add MetastoreShim to support Apache Hive 3.x in IMPALA-10871

Tests:
 - Made sure that the cluster comes up with Apache Hive 3.1.2 when the
   steps above are performed.
 - Made sure that existing scripts work as they do currently when
   argument is not provided.

Change-Id: I1978909589ecacb15d32d874e97f050a85adf1f6
---
M bin/bootstrap_toolchain.py
M bin/impala-config.sh
M buildall.sh
3 files changed, 81 insertions(+), 23 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/93/17793/3
-- 
To view, visit http://gerrit.cloudera.org:8080/17793
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I1978909589ecacb15d32d874e97f050a85adf1f6
Gerrit-Change-Number: 17793
Gerrit-PatchSet: 3
Gerrit-Owner: Fucun Chu <ch...@hotmail.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>

[Impala-ASF-CR] IMPALA-10870: Add Apache Hive 3.1.2 to the minicluster

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17793 )

Change subject: IMPALA-10870: Add Apache Hive 3.1.2 to the minicluster
......................................................................


Patch Set 9: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/17793
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1978909589ecacb15d32d874e97f050a85adf1f6
Gerrit-Change-Number: 17793
Gerrit-PatchSet: 9
Gerrit-Owner: Fucun Chu <ch...@hotmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Fucun Chu <ch...@hotmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Mon, 27 Sep 2021 12:07:38 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10870: Add Apache Hive 3.1.2 to the minicluster

Posted by "Fucun Chu (Code Review)" <ge...@cloudera.org>.
Fucun Chu has posted comments on this change. ( http://gerrit.cloudera.org:8080/17793 )

Change subject: IMPALA-10870: Add Apache Hive 3.1.2 to the minicluster
......................................................................


Patch Set 3:

(3 comments)

http://gerrit.cloudera.org:8080/#/c/17793/3/bin/bootstrap_toolchain.py
File bin/bootstrap_toolchain.py:

http://gerrit.cloudera.org:8080/#/c/17793/3/bin/bootstrap_toolchain.py@324
PS3, Line 324: class ApacheComponent(EnvVersionedPackage):
> flake8: E302 expected 2 blank lines, found 1
Done


http://gerrit.cloudera.org:8080/#/c/17793/3/bin/bootstrap_toolchain.py@326
PS3, Line 326:  
> flake8: E251 unexpected spaces around keyword / parameter equals
Done


http://gerrit.cloudera.org:8080/#/c/17793/3/bin/impala-config.sh
File bin/impala-config.sh:

http://gerrit.cloudera.org:8080/#/c/17793/3/bin/impala-config.sh@261
PS3, Line 261:   # When USE_APACHE_HIVE is set we use the apache hive version to build as well as deploy in
> line too long (92 > 90)
Done



-- 
To view, visit http://gerrit.cloudera.org:8080/17793
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1978909589ecacb15d32d874e97f050a85adf1f6
Gerrit-Change-Number: 17793
Gerrit-PatchSet: 3
Gerrit-Owner: Fucun Chu <ch...@hotmail.com>
Gerrit-Reviewer: Fucun Chu <ch...@hotmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Sat, 21 Aug 2021 04:22:13 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-10870: Add Apache Hive 3.1.2 to the minicluster

Posted by "abeltian (Code Review)" <ge...@cloudera.org>.
abeltian has posted comments on this change. ( http://gerrit.cloudera.org:8080/17793 )

Change subject: IMPALA-10870: Add Apache Hive 3.1.2 to the minicluster
......................................................................


Patch Set 10: Code-Review+1


-- 
To view, visit http://gerrit.cloudera.org:8080/17793
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1978909589ecacb15d32d874e97f050a85adf1f6
Gerrit-Change-Number: 17793
Gerrit-PatchSet: 10
Gerrit-Owner: Fucun Chu <ch...@hotmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Fucun Chu <ch...@hotmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: abeltian <ab...@tencent.com>
Gerrit-Comment-Date: Sat, 02 Oct 2021 03:29:49 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10870: Add Apache Hive 3.1.2 to the minicluster

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17793 )

Change subject: IMPALA-10870: Add Apache Hive 3.1.2 to the minicluster
......................................................................


Patch Set 4:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/9343/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/17793
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1978909589ecacb15d32d874e97f050a85adf1f6
Gerrit-Change-Number: 17793
Gerrit-PatchSet: 4
Gerrit-Owner: Fucun Chu <ch...@hotmail.com>
Gerrit-Reviewer: Fucun Chu <ch...@hotmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Sat, 21 Aug 2021 04:46:27 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10870: Add Apache Hive 3.1.2 to the minicluster

Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/17793 )

Change subject: IMPALA-10870: Add Apache Hive 3.1.2 to the minicluster
......................................................................


Patch Set 6:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/17793/6/buildall.sh
File buildall.sh:

http://gerrit.cloudera.org:8080/#/c/17793/6/buildall.sh@403
PS6, Line 403:       # HIVE-22915
             :       if $USE_APACHE_HIVE; then
             :         rm $HIVE_HOME/lib/guava-*jar
             :         cp $HADOOP_HOME/share/hadoop/hdfs/lib/guava-*.jar $HIVE_HOME/lib/
             :       fi
Do we really need these? This branch downloads cdh components and their guava versions should be compatible.
Or should we move them to line 425 below?



-- 
To view, visit http://gerrit.cloudera.org:8080/17793
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1978909589ecacb15d32d874e97f050a85adf1f6
Gerrit-Change-Number: 17793
Gerrit-PatchSet: 6
Gerrit-Owner: Fucun Chu <ch...@hotmail.com>
Gerrit-Reviewer: Fucun Chu <ch...@hotmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Mon, 30 Aug 2021 07:25:32 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-10870: Add Apache Hive 3.1.2 to the minicluster

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17793 )

Change subject: IMPALA-10870: Add Apache Hive 3.1.2 to the minicluster
......................................................................


Patch Set 9:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7490/ DRY_RUN=false


-- 
To view, visit http://gerrit.cloudera.org:8080/17793
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1978909589ecacb15d32d874e97f050a85adf1f6
Gerrit-Change-Number: 17793
Gerrit-PatchSet: 9
Gerrit-Owner: Fucun Chu <ch...@hotmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Fucun Chu <ch...@hotmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Mon, 27 Sep 2021 05:53:16 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10870: Add Apache Hive 3.1.2 to the minicluster

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17793 )

Change subject: IMPALA-10870: Add Apache Hive 3.1.2 to the minicluster
......................................................................


Patch Set 7:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/9401/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/17793
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1978909589ecacb15d32d874e97f050a85adf1f6
Gerrit-Change-Number: 17793
Gerrit-PatchSet: 7
Gerrit-Owner: Fucun Chu <ch...@hotmail.com>
Gerrit-Reviewer: Fucun Chu <ch...@hotmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Mon, 30 Aug 2021 15:42:52 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10870: Add Apache Hive 3.1.2 to the minicluster

Posted by "Fucun Chu (Code Review)" <ge...@cloudera.org>.
Fucun Chu has uploaded a new patch set (#6). ( http://gerrit.cloudera.org:8080/17793 )

Change subject: IMPALA-10870: Add Apache Hive 3.1.2 to the minicluster
......................................................................

IMPALA-10870: Add Apache Hive 3.1.2 to the minicluster

This patch modifies the minicluster script to optionally use Apache
Hive 3.1.2 instead of CDP Hive 3.1.3.

In order to make sure that existing setups don't break this is
enabled via a environment variable override to bin/impala-config.sh.
When the environment variable USE_APACHE_HIVE is set to true the
bootstrap_toolchain script downloads Apache Hive 3.1.2 tarballs and
extracts it in the toolchain directory. These binaries are used to
start the Hive services (Hiveserver2 and metastore). The default is
CDP Hive 3.1.3

Since CDP Hive 3 uses some features of Apache Hive 4, this patch uses
a different database name so that it is easy to switch from working
from one environment which uses CDP Hive 3.1.3 metastore to another
which usese Apache Hive 3.1.2 metastore.

In order to start a minicluster which uses Apache Hive 3.1.2 users
should follow the steps below:

1. Make sure that minicluster, if running, is stopped before you run
the following commands.
2. Open a new terminal and run following commands.
> export USE_APACHE_HIVE=true
> source bin/impala-config.sh
> bin/bootstrap_toolchain.py
  The above command downloads the Apache Hive 3.1.2 tarballs and
extracts them in toolchain/apache_components directory.

> rm $HIVE_HOME/lib/guava-*jar
> cp $HADOOP_HOME/share/hadoop/hdfs/lib/guava-*.jar $HIVE_HOME/lib/
  The above command is to fix HIVE-22915

> bin/create-test-configuration.sh -create_metastore
  The above step should provide "-create-metastore" only the first time
so that a new metastore db is created and the Apache Hive 3.1.2 schema is
initialized.

> testdata/bin/run-all.sh

Follow-up:
 - Add MetastoreShim to support Apache Hive 3.x in IMPALA-10871

Tests:
 - Made sure that the cluster comes up with Apache Hive 3.1.2 when the
   steps above are performed.
 - Made sure that existing scripts work as they do currently when
   argument is not provided.

Change-Id: I1978909589ecacb15d32d874e97f050a85adf1f6
---
M bin/bootstrap_toolchain.py
M bin/impala-config.sh
M buildall.sh
3 files changed, 89 insertions(+), 12 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/93/17793/6
-- 
To view, visit http://gerrit.cloudera.org:8080/17793
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I1978909589ecacb15d32d874e97f050a85adf1f6
Gerrit-Change-Number: 17793
Gerrit-PatchSet: 6
Gerrit-Owner: Fucun Chu <ch...@hotmail.com>
Gerrit-Reviewer: Fucun Chu <ch...@hotmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>

[Impala-ASF-CR] IMPALA-10870: Add Apache Hive 3.1.2 to the minicluster

Posted by "Fucun Chu (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Laszlo Gaal, Csaba Ringhofer, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/17793

to look at the new patch set (#8).

Change subject: IMPALA-10870: Add Apache Hive 3.1.2 to the minicluster
......................................................................

IMPALA-10870: Add Apache Hive 3.1.2 to the minicluster

This patch modifies the minicluster script to optionally use Apache
Hive 3.1.2 instead of CDP Hive 3.1.3.

In order to make sure that existing setups don't break this is
enabled via a environment variable override to bin/impala-config.sh.
When the environment variable USE_APACHE_HIVE is set to true the
bootstrap_toolchain script downloads Apache Hive 3.1.2 tarballs and
extracts it in the toolchain directory. These binaries are used to
start the Hive services (Hiveserver2 and metastore). The default is
CDP Hive 3.1.3

Since CDP Hive 3 uses some features of Apache Hive 4, this patch uses
a different database name so that it is easy to switch from working
from one environment which uses CDP Hive 3.1.3 metastore to another
which usese Apache Hive 3.1.2 metastore.

In order to start a minicluster which uses Apache Hive 3.1.2 users
should follow the steps below:

1. Make sure that minicluster, if running, is stopped before you run
the following commands.
2. Open a new terminal and run following commands.
> export USE_APACHE_HIVE=true
> source bin/impala-config.sh
> bin/bootstrap_toolchain.py
  The above command downloads the Apache Hive 3.1.2 tarballs and
extracts them in toolchain/apache_components directory.

> rm $HIVE_HOME/lib/guava-*jar
> cp $HADOOP_HOME/share/hadoop/hdfs/lib/guava-*.jar $HIVE_HOME/lib/
  The above command is to fix HIVE-22915

> bin/create-test-configuration.sh -create_metastore
  The above step should provide "-create-metastore" only the first time
so that a new metastore db is created and the Apache Hive 3.1.2 schema
is initialized.

> testdata/bin/run-all.sh

Follow-up:
 - Add MetastoreShim to support Apache Hive 3.x in IMPALA-10871

Tests:
 - Made sure that the cluster comes up with Apache Hive 3.1.2 when the
   steps above are performed.
 - Made sure that existing scripts work as they do currently when
   argument is not provided.

Change-Id: I1978909589ecacb15d32d874e97f050a85adf1f6
---
M bin/bootstrap_toolchain.py
M bin/impala-config.sh
M buildall.sh
3 files changed, 89 insertions(+), 12 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/93/17793/8
-- 
To view, visit http://gerrit.cloudera.org:8080/17793
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I1978909589ecacb15d32d874e97f050a85adf1f6
Gerrit-Change-Number: 17793
Gerrit-PatchSet: 8
Gerrit-Owner: Fucun Chu <ch...@hotmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Fucun Chu <ch...@hotmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>

[Impala-ASF-CR] IMPALA-10870: Add Apache Hive 3.1.2 to the minicluster

Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/17793 )

Change subject: IMPALA-10870: Add Apache Hive 3.1.2 to the minicluster
......................................................................


Patch Set 7: Code-Review+1

LGTM. Thanks for working on this! I'll see if other guys want to have a look.


-- 
To view, visit http://gerrit.cloudera.org:8080/17793
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1978909589ecacb15d32d874e97f050a85adf1f6
Gerrit-Change-Number: 17793
Gerrit-PatchSet: 7
Gerrit-Owner: Fucun Chu <ch...@hotmail.com>
Gerrit-Reviewer: Fucun Chu <ch...@hotmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Tue, 31 Aug 2021 00:28:09 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10870: Add Apache Hive 3.1.2 to the minicluster

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/17793 )

Change subject: IMPALA-10870: Add Apache Hive 3.1.2 to the minicluster
......................................................................

IMPALA-10870: Add Apache Hive 3.1.2 to the minicluster

This patch modifies the minicluster script to optionally use Apache
Hive 3.1.2 instead of CDP Hive 3.1.3.

In order to make sure that existing setups don't break this is
enabled via a environment variable override to bin/impala-config.sh.
When the environment variable USE_APACHE_HIVE is set to true the
bootstrap_toolchain script downloads Apache Hive 3.1.2 tarballs and
extracts it in the toolchain directory. These binaries are used to
start the Hive services (Hiveserver2 and metastore). The default is
CDP Hive 3.1.3

Since CDP Hive 3 uses some features of Apache Hive 4, this patch uses
a different database name so that it is easy to switch from working
from one environment which uses CDP Hive 3.1.3 metastore to another
which usese Apache Hive 3.1.2 metastore.

In order to start a minicluster which uses Apache Hive 3.1.2 users
should follow the steps below:

1. Make sure that minicluster, if running, is stopped before you run
the following commands.
2. Open a new terminal and run following commands.
> export USE_APACHE_HIVE=true
> source bin/impala-config.sh
> bin/bootstrap_toolchain.py
  The above command downloads the Apache Hive 3.1.2 tarballs and
extracts them in toolchain/apache_components directory.

> rm $HIVE_HOME/lib/guava-*jar
> cp $HADOOP_HOME/share/hadoop/hdfs/lib/guava-*.jar $HIVE_HOME/lib/
  The above command is to fix HIVE-22915

> bin/create-test-configuration.sh -create_metastore
  The above step should provide "-create-metastore" only the first time
so that a new metastore db is created and the Apache Hive 3.1.2 schema
is initialized.

> testdata/bin/run-all.sh

Follow-up:
 - Add MetastoreShim to support Apache Hive 3.x in IMPALA-10871

Tests:
 - Made sure that the cluster comes up with Apache Hive 3.1.2 when the
   steps above are performed.
 - Made sure that existing scripts work as they do currently when
   argument is not provided.

Change-Id: I1978909589ecacb15d32d874e97f050a85adf1f6
Reviewed-on: http://gerrit.cloudera.org:8080/17793
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M bin/bootstrap_toolchain.py
M bin/impala-config.sh
M buildall.sh
3 files changed, 89 insertions(+), 12 deletions(-)

Approvals:
  Impala Public Jenkins: Looks good to me, approved; Verified

-- 
To view, visit http://gerrit.cloudera.org:8080/17793
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I1978909589ecacb15d32d874e97f050a85adf1f6
Gerrit-Change-Number: 17793
Gerrit-PatchSet: 10
Gerrit-Owner: Fucun Chu <ch...@hotmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Fucun Chu <ch...@hotmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>

[Impala-ASF-CR] IMPALA-10870: Add Apache Hive 3.1.2 to the minicluster

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17793 )

Change subject: IMPALA-10870: Add Apache Hive 3.1.2 to the minicluster
......................................................................


Patch Set 9: Code-Review+2


-- 
To view, visit http://gerrit.cloudera.org:8080/17793
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1978909589ecacb15d32d874e97f050a85adf1f6
Gerrit-Change-Number: 17793
Gerrit-PatchSet: 9
Gerrit-Owner: Fucun Chu <ch...@hotmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Fucun Chu <ch...@hotmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Mon, 27 Sep 2021 05:53:16 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10870: Add Apache Hive 3.1.2 to the minicluster

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17793 )

Change subject: IMPALA-10870: Add Apache Hive 3.1.2 to the minicluster
......................................................................


Patch Set 6:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/9369/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/17793
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1978909589ecacb15d32d874e97f050a85adf1f6
Gerrit-Change-Number: 17793
Gerrit-PatchSet: 6
Gerrit-Owner: Fucun Chu <ch...@hotmail.com>
Gerrit-Reviewer: Fucun Chu <ch...@hotmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Wed, 25 Aug 2021 13:39:03 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10870: Add Apache Hive 3.1.2 to the minicluster

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17793 )

Change subject: IMPALA-10870: Add Apache Hive 3.1.2 to the minicluster
......................................................................


Patch Set 5:

(3 comments)

http://gerrit.cloudera.org:8080/#/c/17793/5/bin/impala-config.sh
File bin/impala-config.sh:

http://gerrit.cloudera.org:8080/#/c/17793/5/bin/impala-config.sh@276
PS5, Line 276:   export IMPALA_HIVE_STORAGE_API_VERSION=${HIVE_STORAGE_API_VERSION_OVERRIDE:-"2.3.0.$IMPALA_HIVE_VERSION"}
line too long (107 > 90)


http://gerrit.cloudera.org:8080/#/c/17793/5/bin/impala-config.sh@400
PS5, Line 400:   export HIVE_HOME=${HIVE_HOME_OVERRIDE:-"$CDP_COMPONENTS_HOME/apache-hive-${IMPALA_HIVE_VERSION}-bin"}
line too long (103 > 90)


http://gerrit.cloudera.org:8080/#/c/17793/5/bin/impala-config.sh@401
PS5, Line 401:   export HIVE_SRC_DIR=${HIVE_SRC_DIR_OVERRIDE:-"${CDP_COMPONENTS_HOME}/hive-${IMPALA_HIVE_VERSION}"}
line too long (100 > 90)



-- 
To view, visit http://gerrit.cloudera.org:8080/17793
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1978909589ecacb15d32d874e97f050a85adf1f6
Gerrit-Change-Number: 17793
Gerrit-PatchSet: 5
Gerrit-Owner: Fucun Chu <ch...@hotmail.com>
Gerrit-Reviewer: Fucun Chu <ch...@hotmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Wed, 25 Aug 2021 12:02:40 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-10870: Add Apache Hive 3.1.2 to the minicluster

Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/17793 )

Change subject: IMPALA-10870: Add Apache Hive 3.1.2 to the minicluster
......................................................................


Patch Set 8: Code-Review+2

Carry on Csaba's +1 and mine to +2. Thanks Fucun for this work!


-- 
To view, visit http://gerrit.cloudera.org:8080/17793
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1978909589ecacb15d32d874e97f050a85adf1f6
Gerrit-Change-Number: 17793
Gerrit-PatchSet: 8
Gerrit-Owner: Fucun Chu <ch...@hotmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Fucun Chu <ch...@hotmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Mon, 27 Sep 2021 05:52:48 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10870: Add Apache Hive 3.1.2 to the minicluster

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17793 )

Change subject: IMPALA-10870: Add Apache Hive 3.1.2 to the minicluster
......................................................................


Patch Set 5:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/9367/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/17793
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1978909589ecacb15d32d874e97f050a85adf1f6
Gerrit-Change-Number: 17793
Gerrit-PatchSet: 5
Gerrit-Owner: Fucun Chu <ch...@hotmail.com>
Gerrit-Reviewer: Fucun Chu <ch...@hotmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Wed, 25 Aug 2021 12:25:11 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10870: Add Apache Hive 3.1.2 to the minicluster

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17793 )

Change subject: IMPALA-10870: Add Apache Hive 3.1.2 to the minicluster
......................................................................


Patch Set 3:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/9342/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/17793
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1978909589ecacb15d32d874e97f050a85adf1f6
Gerrit-Change-Number: 17793
Gerrit-PatchSet: 3
Gerrit-Owner: Fucun Chu <ch...@hotmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Sat, 21 Aug 2021 01:15:15 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10870: Add Apache Hive 3.1.2 to the minicluster

Posted by "Fucun Chu (Code Review)" <ge...@cloudera.org>.
Fucun Chu has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/17793 )

Change subject: IMPALA-10870: Add Apache Hive 3.1.2 to the minicluster
......................................................................

IMPALA-10870: Add Apache Hive 3.1.2 to the minicluster

This patch modifies the minicluster script to optionally use Apache
Hive 3.1.2 instead of CDP Hive 3.1.3.

In order to make sure that existing setups don't break this is
enabled via a environment variable override to bin/impala-config.sh.
When the environment variable USE_APACHE_HIVE is set to true the
bootstrap_toolchain script downloads Apache Hive 3.1.2 tarballs and
extracts it in the toolchain directory. These binaries are used to
start the Hive services (Hiveserver2 and metastore). The default is
CDP Hive 3.1.3

Since CDP Hive 3 uses some features of Apache Hive 4, this patch uses
a different database name so that it is easy to switch from working
from one environment which uses CDP Hive 3.1.3 metastore to another
which usese Apache Hive 3.1.2 metastore.

In order to start a minicluster which uses Apache Hive 3.1.2 users
should follow the steps below:

1. Make sure that minicluster, if running, is stopped before you run
the following commands.
2. Open a new terminal and run following commands.
> export USE_APACHE_HIVE=true
> source bin/impala-config.sh
> bin/bootstrap_toolchain.py
  The above command downloads the Apache Hive 3.1.2 tarballs and
extracts them in toolchain/apache_components directory.

> rm $HIVE_HOME/lib/guava-*jar
> cp $HADOOP_HOME/share/hadoop/hdfs/lib/guava-*.jar $HIVE_HOME/lib/
  The above command is to fix HIVE-22915

> bin/create-test-configuration.sh -create_metastore
  The above step should provide "-create-metastore" only the first time
so that a new metastore db is created and the Apache Hive 3.1.2 schema is
initialized.

> testdata/bin/run-all.sh

Follow-up:
 - Add MetastoreShim to support Apache Hive 3.x in IMPALA-10871

Tests:
 - Made sure that the cluster comes up with Apache Hive 3.1.2 when the
   steps above are performed.
 - Made sure that existing scripts work as they do currently when
   argument is not provided.

Change-Id: I1978909589ecacb15d32d874e97f050a85adf1f6
---
M bin/bootstrap_toolchain.py
M bin/impala-config.sh
M buildall.sh
3 files changed, 84 insertions(+), 23 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/93/17793/4
-- 
To view, visit http://gerrit.cloudera.org:8080/17793
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I1978909589ecacb15d32d874e97f050a85adf1f6
Gerrit-Change-Number: 17793
Gerrit-PatchSet: 4
Gerrit-Owner: Fucun Chu <ch...@hotmail.com>
Gerrit-Reviewer: Fucun Chu <ch...@hotmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>

[Impala-ASF-CR] IMPALA-10870: Add Apache Hive 3.1.2 to the minicluster

Posted by "Fucun Chu (Code Review)" <ge...@cloudera.org>.
Fucun Chu has uploaded a new patch set (#7). ( http://gerrit.cloudera.org:8080/17793 )

Change subject: IMPALA-10870: Add Apache Hive 3.1.2 to the minicluster
......................................................................

IMPALA-10870: Add Apache Hive 3.1.2 to the minicluster

This patch modifies the minicluster script to optionally use Apache
Hive 3.1.2 instead of CDP Hive 3.1.3.

In order to make sure that existing setups don't break this is
enabled via a environment variable override to bin/impala-config.sh.
When the environment variable USE_APACHE_HIVE is set to true the
bootstrap_toolchain script downloads Apache Hive 3.1.2 tarballs and
extracts it in the toolchain directory. These binaries are used to
start the Hive services (Hiveserver2 and metastore). The default is
CDP Hive 3.1.3

Since CDP Hive 3 uses some features of Apache Hive 4, this patch uses
a different database name so that it is easy to switch from working
from one environment which uses CDP Hive 3.1.3 metastore to another
which usese Apache Hive 3.1.2 metastore.

In order to start a minicluster which uses Apache Hive 3.1.2 users
should follow the steps below:

1. Make sure that minicluster, if running, is stopped before you run
the following commands.
2. Open a new terminal and run following commands.
> export USE_APACHE_HIVE=true
> source bin/impala-config.sh
> bin/bootstrap_toolchain.py
  The above command downloads the Apache Hive 3.1.2 tarballs and
extracts them in toolchain/apache_components directory.

> rm $HIVE_HOME/lib/guava-*jar
> cp $HADOOP_HOME/share/hadoop/hdfs/lib/guava-*.jar $HIVE_HOME/lib/
  The above command is to fix HIVE-22915

> bin/create-test-configuration.sh -create_metastore
  The above step should provide "-create-metastore" only the first time
so that a new metastore db is created and the Apache Hive 3.1.2 schema is
initialized.

> testdata/bin/run-all.sh

Follow-up:
 - Add MetastoreShim to support Apache Hive 3.x in IMPALA-10871

Tests:
 - Made sure that the cluster comes up with Apache Hive 3.1.2 when the
   steps above are performed.
 - Made sure that existing scripts work as they do currently when
   argument is not provided.

Change-Id: I1978909589ecacb15d32d874e97f050a85adf1f6
---
M bin/bootstrap_toolchain.py
M bin/impala-config.sh
M buildall.sh
3 files changed, 89 insertions(+), 12 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/93/17793/7
-- 
To view, visit http://gerrit.cloudera.org:8080/17793
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I1978909589ecacb15d32d874e97f050a85adf1f6
Gerrit-Change-Number: 17793
Gerrit-PatchSet: 7
Gerrit-Owner: Fucun Chu <ch...@hotmail.com>
Gerrit-Reviewer: Fucun Chu <ch...@hotmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>

[Impala-ASF-CR] IMPALA-10870: Add Apache Hive 3.1.2 to the minicluster

Posted by "Fucun Chu (Code Review)" <ge...@cloudera.org>.
Fucun Chu has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/17793 )

Change subject: IMPALA-10870: Add Apache Hive 3.1.2 to the minicluster
......................................................................

IMPALA-10870: Add Apache Hive 3.1.2 to the minicluster

This patch modifies the minicluster script to optionally use Apache
Hive 3.1.2 instead of CDP Hive 3.1.3.

In order to make sure that existing setups don't break this is
enabled via a environment variable override to bin/impala-config.sh.
When the environment variable USE_APACHE_HIVE is set to true the
bootstrap_toolchain script downloads Apache Hive 3.1.2 tarballs and
extracts it in the toolchain directory. These binaries are used to
start the Hive services (Hiveserver2 and metastore). The default is
CDP Hive 3.1.3

Since CDP Hive 3 uses some features of Apache Hive 4, this patch uses
a different database name so that it is easy to switch from working
from one environment which uses CDP Hive 3.1.3 metastore to another
which usese Apache Hive 3.1.2 metastore.

In order to start a minicluster which uses Apache Hive 3.1.2 users
should follow the steps below:

1. Make sure that minicluster, if running, is stopped before you run
the following commands.
2. Open a new terminal and run following commands.
> export USE_APACHE_HIVE=true
> source bin/impala-config.sh
> bin/bootstrap_toolchain.py
  The above command downloads the Apache Hive 3.1.2 tarballs and
extracts them in toolchain/apache_components directory.

> rm $HIVE_HOME/lib/guava-*jar
> cp $HADOOP_HOME/share/hadoop/hdfs/lib/guava-*.jar $HIVE_HOME/lib/
  The above command is to fix HIVE-22915

> bin/create-test-configuration.sh -create_metastore
  The above step should provide "-create-metastore" only the first time
so that a new metastore db is created and the Apache Hive 3.1.2 schema is
initialized.

> testdata/bin/run-all.sh

Follow-up:
 - Add MetastoreShim to support Apache Hive 3.x in IMPALA-10871

Tests:
 - Made sure that the cluster comes up with Apache Hive 3.1.2 when the
   steps above are performed.
 - Made sure that existing scripts work as they do currently when
   argument is not provided.

Change-Id: I1978909589ecacb15d32d874e97f050a85adf1f6
---
M bin/bootstrap_toolchain.py
M bin/impala-config.sh
M buildall.sh
3 files changed, 86 insertions(+), 12 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/93/17793/5
-- 
To view, visit http://gerrit.cloudera.org:8080/17793
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I1978909589ecacb15d32d874e97f050a85adf1f6
Gerrit-Change-Number: 17793
Gerrit-PatchSet: 5
Gerrit-Owner: Fucun Chu <ch...@hotmail.com>
Gerrit-Reviewer: Fucun Chu <ch...@hotmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>

[Impala-ASF-CR] IMPALA-10870: Add Apache Hive 3.1.2 to the minicluster

Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/17793 )

Change subject: IMPALA-10870: Add Apache Hive 3.1.2 to the minicluster
......................................................................


Patch Set 7: Code-Review+1

(1 comment)

Just bumped into this - looks great!

http://gerrit.cloudera.org:8080/#/c/17793/7//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/17793/7//COMMIT_MSG@43
PS7, Line 43: so that a new metastore db is created and the Apache Hive 3.1.2 schema is
nit: wrap at 72



-- 
To view, visit http://gerrit.cloudera.org:8080/17793
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1978909589ecacb15d32d874e97f050a85adf1f6
Gerrit-Change-Number: 17793
Gerrit-PatchSet: 7
Gerrit-Owner: Fucun Chu <ch...@hotmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Fucun Chu <ch...@hotmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Fri, 10 Sep 2021 16:14:57 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-10870: Add Apache Hive 3.1.2 to the minicluster

Posted by "Fucun Chu (Code Review)" <ge...@cloudera.org>.
Fucun Chu has posted comments on this change. ( http://gerrit.cloudera.org:8080/17793 )

Change subject: IMPALA-10870: Add Apache Hive 3.1.2 to the minicluster
......................................................................


Patch Set 8:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/17793/7//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/17793/7//COMMIT_MSG@43
PS7, Line 43: so that a new metastore db is created and the Apache Hive 3.1.2 schema
> nit: wrap at 72
Done



-- 
To view, visit http://gerrit.cloudera.org:8080/17793
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1978909589ecacb15d32d874e97f050a85adf1f6
Gerrit-Change-Number: 17793
Gerrit-PatchSet: 8
Gerrit-Owner: Fucun Chu <ch...@hotmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Fucun Chu <ch...@hotmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Mon, 13 Sep 2021 02:29:49 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-10870: Add Apache Hive 3.1.2 to the minicluster

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17793 )

Change subject: IMPALA-10870: Add Apache Hive 3.1.2 to the minicluster
......................................................................


Patch Set 3:

(3 comments)

http://gerrit.cloudera.org:8080/#/c/17793/3/bin/bootstrap_toolchain.py
File bin/bootstrap_toolchain.py:

http://gerrit.cloudera.org:8080/#/c/17793/3/bin/bootstrap_toolchain.py@324
PS3, Line 324: class ApacheComponent(EnvVersionedPackage):
flake8: E302 expected 2 blank lines, found 1


http://gerrit.cloudera.org:8080/#/c/17793/3/bin/bootstrap_toolchain.py@326
PS3, Line 326:  
flake8: E251 unexpected spaces around keyword / parameter equals


http://gerrit.cloudera.org:8080/#/c/17793/3/bin/impala-config.sh
File bin/impala-config.sh:

http://gerrit.cloudera.org:8080/#/c/17793/3/bin/impala-config.sh@261
PS3, Line 261:   # When USE_APACHE_HIVE is set we use the apache hive version to build as well as deploy in
line too long (92 > 90)



-- 
To view, visit http://gerrit.cloudera.org:8080/17793
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1978909589ecacb15d32d874e97f050a85adf1f6
Gerrit-Change-Number: 17793
Gerrit-PatchSet: 3
Gerrit-Owner: Fucun Chu <ch...@hotmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Sat, 21 Aug 2021 00:52:41 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-10870: Add Apache Hive 3.1.2 to the minicluster

Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/17793 )

Change subject: IMPALA-10870: Add Apache Hive 3.1.2 to the minicluster
......................................................................


Patch Set 4:

(6 comments)

Thanks for contributing this!

I see some codes that dealing with *_OVERRIDE env vars are removed. Please keep them since we have internal jobs still depending on them.

http://gerrit.cloudera.org:8080/#/c/17793/4/bin/bootstrap_toolchain.py
File bin/bootstrap_toolchain.py:

http://gerrit.cloudera.org:8080/#/c/17793/4/bin/bootstrap_toolchain.py@a468
PS4, Line 468: 
             : 
             : 
             : 
             : 
             : 
             : 
Please don't remove these codes. We have internal jobs still using them.


http://gerrit.cloudera.org:8080/#/c/17793/4/bin/bootstrap_toolchain.py@329
PS4, Line 329: APACHE_ARCHIVE
Can we use APACHE_COMPONENTS_HOME instead? The implementation of this class has lots of hive stuffs. In the future, we may add other components like Hadoop, which are unrelated to Hive. It'd be good to be more abstractive.


http://gerrit.cloudera.org:8080/#/c/17793/4/bin/bootstrap_toolchain.py@336
PS4, Line 336:     url_prefix_tmpl = "${apache_archive}/${name}/%s/" % (baseversion_tmpl)
I think different components have different base urls. We need a real prefix url as the input parameter.


http://gerrit.cloudera.org:8080/#/c/17793/4/bin/impala-config.sh
File bin/impala-config.sh:

http://gerrit.cloudera.org:8080/#/c/17793/4/bin/impala-config.sh@190
PS4, Line 190: : ${APACHE_ARCHIVE:=https://archive.apache.org/dist}
It'd be better to download from apache mirrors. An example URL is http://www.apache.org/dyn/closer.cgi/hive/hive-3.1.2/apache-hive-3.1.2-src.tar.gz?action=download

Ref: https://infra.apache.org/release-download-pages.html#closer


http://gerrit.cloudera.org:8080/#/c/17793/4/bin/impala-config.sh@272
PS4, Line 272: ${CDP_HIVE_VERSION}
Let's keep the original code, i.e. ${HIVE_VERSION_OVERRIDE:-"$CDP_HIVE_VERSION"}, so bin/impala-config-branch.sh or bin/impala-config-local.sh can still override this.


http://gerrit.cloudera.org:8080/#/c/17793/4/bin/impala-config.sh@275
PS4, Line 275: "2.3.0.$IMPALA_HIVE_VERSION"
Same as above. Let's don't remove HIVE_STORAGE_API_VERSION_OVERRIDE.



-- 
To view, visit http://gerrit.cloudera.org:8080/17793
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1978909589ecacb15d32d874e97f050a85adf1f6
Gerrit-Change-Number: 17793
Gerrit-PatchSet: 4
Gerrit-Owner: Fucun Chu <ch...@hotmail.com>
Gerrit-Reviewer: Fucun Chu <ch...@hotmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Tue, 24 Aug 2021 14:06:58 +0000
Gerrit-HasComments: Yes