You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Joe McDonnell (Code Review)" <ge...@cloudera.org> on 2022/08/04 22:46:29 UTC

[Impala-ASF-CR] PROTOTYPE: IMPALA-11207: Use hadoop-cloud-storage for Cloud dependencies

Joe McDonnell has uploaded this change for review. ( http://gerrit.cloudera.org:8080/18817


Change subject: PROTOTYPE: IMPALA-11207: Use hadoop-cloud-storage for Cloud dependencies
......................................................................

PROTOTYPE: IMPALA-11207: Use hadoop-cloud-storage for Cloud dependencies

Hadoop provides hadoop-cloud-storage, which includes most of
the dependencies that Impala currently uses like hadoop-aws,
hadoop-azure, Knox's gateway-cloud-bindings, etc. Hadoop has
put in a lot of work to make sure that this package includes
the right version of dependencies (including shading some
dependencies for GCS). It seems like this is a more reliable
way to consume these dependencies.

This switches the Java build to use hadoop-cloud-storage
and removes the dependencies that it replaces. This eliminates
the need to control the version of oauth and GCS, as those
are determined by hadoop-cloud-storage.

Change-Id: I3a1631289f990513823c2b17eb9241cc1b5a7ffd
---
M bin/impala-config.sh
M java/executor-deps/pom.xml
M java/pom.xml
3 files changed, 11 insertions(+), 56 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/17/18817/1
-- 
To view, visit http://gerrit.cloudera.org:8080/18817
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I3a1631289f990513823c2b17eb9241cc1b5a7ffd
Gerrit-Change-Number: 18817
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>

[Impala-ASF-CR] PROTOTYPE: IMPALA-11207: Use hadoop-cloud-storage for Cloud dependencies

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18817 )

Change subject: PROTOTYPE: IMPALA-11207: Use hadoop-cloud-storage for Cloud dependencies
......................................................................


Patch Set 1: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/8404/


-- 
To view, visit http://gerrit.cloudera.org:8080/18817
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I3a1631289f990513823c2b17eb9241cc1b5a7ffd
Gerrit-Change-Number: 18817
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Fri, 05 Aug 2022 03:20:11 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] PROTOTYPE: IMPALA-11207: Use hadoop-cloud-storage for Cloud dependencies

Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/18817 )

Change subject: PROTOTYPE: IMPALA-11207: Use hadoop-cloud-storage for Cloud dependencies
......................................................................


Patch Set 1: Code-Review+1

This seems reasonable. Some of this code is only used at runtime and I'm not sure how good the test coverage is for it. However the fe/target/build-classpath.txt diff between this and just reverting the google-oauth-client change (since this version of hadoop-cloud-storage still uses 0.31) is

34d33
< /home/michael/.m2/repository/com/google/cloud/bigdataoss/gcs-connector/2.1.2.7.2.16.0-77/gcs-connector-2.1.2.7.2.16.0-77.jar
59d57
< /home/michael/.m2/repository/com/jayway/jsonpath/json-path/2.4.0/json-path-2.4.0.jar
144,145d141
< /home/michael/.m2/repository/org/apache/directory/server/apacheds-i18n/2.0.0.AM25/apacheds-i18n-2.0.0.AM25.jar
< /home/michael/.m2/repository/org/apache/directory/server/apacheds-jdbm/2.0.0-M5/apacheds-jdbm-2.0.0-M5.jar
150a147
> /home/michael/.m2/repository/org/apache/hadoop/hadoop-cloud-storage/3.1.1.7.2.16.0-77/hadoop-cloud-storage-3.1.1.7.2.16.0-77.jar
226a224,227
> /home/michael/.m2/repository/org/apache/ranger/ranger-raz-hook-abfs/2.1.0.7.2.16.0-77/ranger-raz-hook-abfs-2.1.0.7.2.16.0-77.jar
> /home/michael/.m2/repository/org/apache/ranger/ranger-raz-hook-s3/2.1.0.7.2.16.0-77/ranger-raz-hook-s3-2.1.0.7.2.16.0-77.jar
> /home/michael/.m2/repository/org/apache/ranger/ranger-raz-intg/2.1.0.7.2.16.0-77/ranger-raz-intg-2.1.0.7.2.16.0-77.jar
> /home/michael/.m2/repository/org/apache/ranger/ranger-raz-s3-lib/2.1.0.7.2.16.0-77/ranger-raz-s3-lib-2.1.0.7.2.16.0-77.jar
238,244d238
< /home/michael/.m2/repository/org/codehaus/groovy/groovy/3.0.7/groovy-3.0.7.jar
< /home/michael/.m2/repository/org/codehaus/groovy/groovy-console/3.0.7/groovy-console-3.0.7.jar
< /home/michael/.m2/repository/org/codehaus/groovy/groovy-groovysh/3.0.7/groovy-groovysh-3.0.7.jar
< /home/michael/.m2/repository/org/codehaus/groovy/groovy-json/3.0.7/groovy-json-3.0.7.jar
< /home/michael/.m2/repository/org/codehaus/groovy/groovy-swing/3.0.7/groovy-swing-3.0.7.jar
< /home/michael/.m2/repository/org/codehaus/groovy/groovy-templates/3.0.7/groovy-templates-3.0.7.jar
< /home/michael/.m2/repository/org/codehaus/groovy/groovy-xml/3.0.7/groovy-xml-3.0.7.jar
276d269
< /home/michael/.m2/repository/org/fusesource/jansi/jansi/1.18/jansi-1.18.jar

which doesn't seem too concerning.


-- 
To view, visit http://gerrit.cloudera.org:8080/18817
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I3a1631289f990513823c2b17eb9241cc1b5a7ffd
Gerrit-Change-Number: 18817
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Fri, 05 Aug 2022 22:18:37 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11207: Use hadoop-cloud-storage for Cloud dependencies

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18817 )

Change subject: IMPALA-11207: Use hadoop-cloud-storage for Cloud dependencies
......................................................................


Patch Set 3: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/18817
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I3a1631289f990513823c2b17eb9241cc1b5a7ffd
Gerrit-Change-Number: 18817
Gerrit-PatchSet: 3
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Mon, 15 Aug 2022 21:11:41 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11207: Use hadoop-cloud-storage for Cloud dependencies

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18817 )

Change subject: IMPALA-11207: Use hadoop-cloud-storage for Cloud dependencies
......................................................................


Patch Set 3:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8444/ DRY_RUN=false


-- 
To view, visit http://gerrit.cloudera.org:8080/18817
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I3a1631289f990513823c2b17eb9241cc1b5a7ffd
Gerrit-Change-Number: 18817
Gerrit-PatchSet: 3
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Mon, 15 Aug 2022 16:19:34 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11207: Use hadoop-cloud-storage for Cloud dependencies

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18817 )

Change subject: IMPALA-11207: Use hadoop-cloud-storage for Cloud dependencies
......................................................................


Patch Set 3: Code-Review+2


-- 
To view, visit http://gerrit.cloudera.org:8080/18817
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I3a1631289f990513823c2b17eb9241cc1b5a7ffd
Gerrit-Change-Number: 18817
Gerrit-PatchSet: 3
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Mon, 15 Aug 2022 16:19:33 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11207: Use hadoop-cloud-storage for Cloud dependencies

Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/18817 )

Change subject: IMPALA-11207: Use hadoop-cloud-storage for Cloud dependencies
......................................................................


Patch Set 2:

I don't think hadoop-cos is available in the versions of hadoop-cloud-storage we pull in yet, so I plan to leave it for now. This seems ready to merge.


-- 
To view, visit http://gerrit.cloudera.org:8080/18817
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I3a1631289f990513823c2b17eb9241cc1b5a7ffd
Gerrit-Change-Number: 18817
Gerrit-PatchSet: 2
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Mon, 15 Aug 2022 16:06:12 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11207: Use hadoop-cloud-storage for Cloud dependencies

Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has uploaded a new patch set (#2) to the change originally created by Joe McDonnell. ( http://gerrit.cloudera.org:8080/18817 )

Change subject: IMPALA-11207: Use hadoop-cloud-storage for Cloud dependencies
......................................................................

IMPALA-11207: Use hadoop-cloud-storage for Cloud dependencies

Hadoop provides hadoop-cloud-storage, which includes most of
the dependencies that Impala currently uses like hadoop-aws,
hadoop-azure, Knox's gateway-cloud-bindings, etc. Hadoop has
put in a lot of work to make sure that this package includes
the right version of dependencies (including shading some
dependencies for GCS). It seems like this is a more reliable
way to consume these dependencies.

This switches the Java build to use hadoop-cloud-storage
and removes the dependencies that it replaces. This eliminates
the need to control the version of oauth and GCS, as those
are determined by hadoop-cloud-storage.

Change-Id: I3a1631289f990513823c2b17eb9241cc1b5a7ffd
---
M bin/impala-config.sh
M java/executor-deps/pom.xml
M java/pom.xml
3 files changed, 11 insertions(+), 56 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/17/18817/2
-- 
To view, visit http://gerrit.cloudera.org:8080/18817
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I3a1631289f990513823c2b17eb9241cc1b5a7ffd
Gerrit-Change-Number: 18817
Gerrit-PatchSet: 2
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>

[Impala-ASF-CR] IMPALA-11207: Use hadoop-cloud-storage for Cloud dependencies

Posted by "Laszlo Gaal (Code Review)" <ge...@cloudera.org>.
Laszlo Gaal has posted comments on this change. ( http://gerrit.cloudera.org:8080/18817 )

Change subject: IMPALA-11207: Use hadoop-cloud-storage for Cloud dependencies
......................................................................


Patch Set 2: Code-Review+1

Thanks a lot for switching to the cloud-storage package!


-- 
To view, visit http://gerrit.cloudera.org:8080/18817
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I3a1631289f990513823c2b17eb9241cc1b5a7ffd
Gerrit-Change-Number: 18817
Gerrit-PatchSet: 2
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Thu, 11 Aug 2022 19:13:17 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] PROTOTYPE: IMPALA-11207: Use hadoop-cloud-storage for Cloud dependencies

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18817 )

Change subject: PROTOTYPE: IMPALA-11207: Use hadoop-cloud-storage for Cloud dependencies
......................................................................


Patch Set 1: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/18817
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I3a1631289f990513823c2b17eb9241cc1b5a7ffd
Gerrit-Change-Number: 18817
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Fri, 05 Aug 2022 20:59:14 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11207: Use hadoop-cloud-storage for Cloud dependencies

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18817 )

Change subject: IMPALA-11207: Use hadoop-cloud-storage for Cloud dependencies
......................................................................


Patch Set 2:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/11148/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/18817
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I3a1631289f990513823c2b17eb9241cc1b5a7ffd
Gerrit-Change-Number: 18817
Gerrit-PatchSet: 2
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Thu, 11 Aug 2022 16:55:16 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] PROTOTYPE: IMPALA-11207: Use hadoop-cloud-storage for Cloud dependencies

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18817 )

Change subject: PROTOTYPE: IMPALA-11207: Use hadoop-cloud-storage for Cloud dependencies
......................................................................


Patch Set 1:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8404/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/18817
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I3a1631289f990513823c2b17eb9241cc1b5a7ffd
Gerrit-Change-Number: 18817
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Thu, 04 Aug 2022 23:02:11 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11207: Use hadoop-cloud-storage for Cloud dependencies

Posted by "Wenzhe Zhou (Code Review)" <ge...@cloudera.org>.
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/18817 )

Change subject: IMPALA-11207: Use hadoop-cloud-storage for Cloud dependencies
......................................................................


Patch Set 2: Code-Review+2


-- 
To view, visit http://gerrit.cloudera.org:8080/18817
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I3a1631289f990513823c2b17eb9241cc1b5a7ffd
Gerrit-Change-Number: 18817
Gerrit-PatchSet: 2
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Thu, 11 Aug 2022 18:50:55 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11207: Use hadoop-cloud-storage for Cloud dependencies

Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/18817 )

Change subject: IMPALA-11207: Use hadoop-cloud-storage for Cloud dependencies
......................................................................


Patch Set 2:

I've been told we might be able to remove the dependency on hadoop-cos, looking into it.


-- 
To view, visit http://gerrit.cloudera.org:8080/18817
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I3a1631289f990513823c2b17eb9241cc1b5a7ffd
Gerrit-Change-Number: 18817
Gerrit-PatchSet: 2
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Thu, 11 Aug 2022 20:27:49 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] PROTOTYPE: IMPALA-11207: Use hadoop-cloud-storage for Cloud dependencies

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18817 )

Change subject: PROTOTYPE: IMPALA-11207: Use hadoop-cloud-storage for Cloud dependencies
......................................................................


Patch Set 1:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/11101/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/18817
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I3a1631289f990513823c2b17eb9241cc1b5a7ffd
Gerrit-Change-Number: 18817
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Thu, 04 Aug 2022 23:10:50 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] PROTOTYPE: IMPALA-11207: Use hadoop-cloud-storage for Cloud dependencies

Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/18817 )

Change subject: PROTOTYPE: IMPALA-11207: Use hadoop-cloud-storage for Cloud dependencies
......................................................................


Patch Set 1:

Ran an S3 test suite. Hit a single failure talking to hiveserver2 that seems unrelated: https://master-03.jenkins.cloudera.com/job/impala-private-s3-parameterized/33/testReport/junit/metadata.test_ddl/TestAsyncDDL/test_get_operation_status_for_async_ddl_protocol__hs2_http___exec_option____sync_ddl___0___test_replan___1___batch_size___0___num_nodes___0___disable_codegen_rows_threshold___0___disable_codegen___False___abort_on_error___1___exec_single_node_rows_threshold___0____table_format__text_none_/


-- 
To view, visit http://gerrit.cloudera.org:8080/18817
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I3a1631289f990513823c2b17eb9241cc1b5a7ffd
Gerrit-Change-Number: 18817
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Tue, 09 Aug 2022 17:29:51 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] PROTOTYPE: IMPALA-11207: Use hadoop-cloud-storage for Cloud dependencies

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18817 )

Change subject: PROTOTYPE: IMPALA-11207: Use hadoop-cloud-storage for Cloud dependencies
......................................................................


Patch Set 1:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8405/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/18817
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I3a1631289f990513823c2b17eb9241cc1b5a7ffd
Gerrit-Change-Number: 18817
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Fri, 05 Aug 2022 16:15:40 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11207: Use hadoop-cloud-storage for Cloud dependencies

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/18817 )

Change subject: IMPALA-11207: Use hadoop-cloud-storage for Cloud dependencies
......................................................................

IMPALA-11207: Use hadoop-cloud-storage for Cloud dependencies

Hadoop provides hadoop-cloud-storage, which includes most of
the dependencies that Impala currently uses like hadoop-aws,
hadoop-azure, Knox's gateway-cloud-bindings, etc. Hadoop has
put in a lot of work to make sure that this package includes
the right version of dependencies (including shading some
dependencies for GCS). It seems like this is a more reliable
way to consume these dependencies.

This switches the Java build to use hadoop-cloud-storage
and removes the dependencies that it replaces. This eliminates
the need to control the version of oauth and GCS, as those
are determined by hadoop-cloud-storage.

Change-Id: I3a1631289f990513823c2b17eb9241cc1b5a7ffd
Reviewed-on: http://gerrit.cloudera.org:8080/18817
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M bin/impala-config.sh
M java/executor-deps/pom.xml
M java/pom.xml
3 files changed, 11 insertions(+), 56 deletions(-)

Approvals:
  Impala Public Jenkins: Looks good to me, approved; Verified

-- 
To view, visit http://gerrit.cloudera.org:8080/18817
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I3a1631289f990513823c2b17eb9241cc1b5a7ffd
Gerrit-Change-Number: 18817
Gerrit-PatchSet: 4
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>