You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Joe McDonnell (Code Review)" <ge...@cloudera.org> on 2019/07/16 23:37:01 UTC

[Impala-ASF-CR] IMPALA-8766: Change cloud dependencies to use hadoop-cloud-storage 

Joe McDonnell has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13872


Change subject: IMPALA-8766: Change cloud dependencies to use hadoop-cloud-storage
......................................................................

IMPALA-8766: Change cloud dependencies to use hadoop-cloud-storage

Currently, fe/pom.xml includes direct dependencies on hadoop-aws,
hadoop-azure, and hadoop-azure-datalake. Hadoop has a meta-package
hadoop-cloud-storage that provides these dependencies with appropriate
exclusions. This would not change Impala's build significantly, but
it allows different versions or distributions of Hadoop to add
necessary runtime jars to the classpath. For example, distributions
of Hadoop that use Apache Knox for identity management can add
Knox jars to this meta-package.

This switches Impala to use the hadoop-cloud-storage package. This
does not change the default configuration, but the USE_CDP_HIVE=true
configuration gains Apache Knox jars.

Due to the structure of the CDP maven repository, it has dependencies
on versions of jars that are not in the CDP maven repository. So,
USE_CDP_HIVE=true requires the Hortonworks public maven repository.
This is added, but it is only enabled with USE_CDP_HIVE=true and only
for releases (no SNAPSHOTs).

Testing:
 - Ran core tests
 - Manually verified classpath changes

Change-Id: I08f1c36ecf54ac277d99e2d2843163eada732e50
---
M fe/pom.xml
M impala-parent/pom.xml
2 files changed, 18 insertions(+), 14 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/72/13872/1
-- 
To view, visit http://gerrit.cloudera.org:8080/13872
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I08f1c36ecf54ac277d99e2d2843163eada732e50
Gerrit-Change-Number: 13872
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>

[Impala-ASF-CR] IMPALA-8766: Change cloud dependencies to use hadoop-cloud-storage

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13872 )

Change subject: IMPALA-8766: Change cloud dependencies to use hadoop-cloud-storage
......................................................................


Patch Set 1: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4607/


-- 
To view, visit http://gerrit.cloudera.org:8080/13872
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I08f1c36ecf54ac277d99e2d2843163eada732e50
Gerrit-Change-Number: 13872
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Wed, 17 Jul 2019 05:56:25 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8766: Change cloud dependencies to use hadoop-cloud-storage

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13872 )

Change subject: IMPALA-8766: Change cloud dependencies to use hadoop-cloud-storage
......................................................................


Patch Set 4:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/3953/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/13872
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I08f1c36ecf54ac277d99e2d2843163eada732e50
Gerrit-Change-Number: 13872
Gerrit-PatchSet: 4
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Tue, 23 Jul 2019 00:19:36 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8766: Change cloud dependencies to use hadoop-cloud-storage

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13872 )

Change subject: IMPALA-8766: Change cloud dependencies to use hadoop-cloud-storage
......................................................................


Patch Set 2: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4612/


-- 
To view, visit http://gerrit.cloudera.org:8080/13872
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I08f1c36ecf54ac277d99e2d2843163eada732e50
Gerrit-Change-Number: 13872
Gerrit-PatchSet: 2
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Comment-Date: Thu, 18 Jul 2019 22:40:16 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8766: Change cloud dependencies to use hadoop-cloud-storage

Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/13872 )

Change subject: IMPALA-8766: Change cloud dependencies to use hadoop-cloud-storage
......................................................................


Patch Set 3:

The alternative to this is to add org.apache.knox gateway-cloud-storage directly rather than through hadoop-cloud-storage.


-- 
To view, visit http://gerrit.cloudera.org:8080/13872
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I08f1c36ecf54ac277d99e2d2843163eada732e50
Gerrit-Change-Number: 13872
Gerrit-PatchSet: 3
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Fri, 19 Jul 2019 22:35:28 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8766: Change cloud dependencies to use hadoop-cloud-storage

Posted by "Vihang Karajgaonkar (Code Review)" <ge...@cloudera.org>.
Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/13872 )

Change subject: IMPALA-8766: Change cloud dependencies to use hadoop-cloud-storage
......................................................................


Patch Set 4: Code-Review+2

Thanks for adding the comments as suggested.


-- 
To view, visit http://gerrit.cloudera.org:8080/13872
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I08f1c36ecf54ac277d99e2d2843163eada732e50
Gerrit-Change-Number: 13872
Gerrit-PatchSet: 4
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Wed, 24 Jul 2019 16:49:09 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8766: Change cloud dependencies to use hadoop-cloud-storage

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13872 )

Change subject: IMPALA-8766: Change cloud dependencies to use hadoop-cloud-storage
......................................................................


Patch Set 2:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4612/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/13872
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I08f1c36ecf54ac277d99e2d2843163eada732e50
Gerrit-Change-Number: 13872
Gerrit-PatchSet: 2
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Comment-Date: Thu, 18 Jul 2019 15:50:32 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8766: Change cloud dependencies to use hadoop-cloud-storage

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13872 )

Change subject: IMPALA-8766: Change cloud dependencies to use hadoop-cloud-storage
......................................................................


Patch Set 1:

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4608/


-- 
To view, visit http://gerrit.cloudera.org:8080/13872
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I08f1c36ecf54ac277d99e2d2843163eada732e50
Gerrit-Change-Number: 13872
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Comment-Date: Thu, 18 Jul 2019 01:31:25 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8766: Change cloud dependencies to use hadoop-cloud-storage

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13872 )

Change subject: IMPALA-8766: Change cloud dependencies to use hadoop-cloud-storage
......................................................................


Patch Set 1:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4608/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/13872
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I08f1c36ecf54ac277d99e2d2843163eada732e50
Gerrit-Change-Number: 13872
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Wed, 17 Jul 2019 19:09:48 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8766: Change cloud dependencies to use hadoop-cloud-storage

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13872 )

Change subject: IMPALA-8766: Change cloud dependencies to use hadoop-cloud-storage
......................................................................


Patch Set 4: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/13872
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I08f1c36ecf54ac277d99e2d2843163eada732e50
Gerrit-Change-Number: 13872
Gerrit-PatchSet: 4
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Tue, 23 Jul 2019 23:47:59 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8766: Change cloud dependencies to use hadoop-cloud-storage

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13872 )

Change subject: IMPALA-8766: Change cloud dependencies to use hadoop-cloud-storage
......................................................................


Patch Set 4:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4640/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/13872
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I08f1c36ecf54ac277d99e2d2843163eada732e50
Gerrit-Change-Number: 13872
Gerrit-PatchSet: 4
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Tue, 23 Jul 2019 17:14:05 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8766: Change cloud dependencies to use hadoop-cloud-storage

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13872 )

Change subject: IMPALA-8766: Change cloud dependencies to use hadoop-cloud-storage
......................................................................


Patch Set 1:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/3890/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/13872
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I08f1c36ecf54ac277d99e2d2843163eada732e50
Gerrit-Change-Number: 13872
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Wed, 17 Jul 2019 00:16:55 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8766: Change cloud dependencies to use hadoop-cloud-storage

Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/13872

to look at the new patch set (#2).

Change subject: IMPALA-8766: Change cloud dependencies to use hadoop-cloud-storage
......................................................................

IMPALA-8766: Change cloud dependencies to use hadoop-cloud-storage

Currently, fe/pom.xml includes direct dependencies on hadoop-aws,
hadoop-azure, and hadoop-azure-datalake. Hadoop has a meta-package
hadoop-cloud-storage that provides these dependencies with appropriate
exclusions. This would not change Impala's build significantly, but
it allows different versions or distributions of Hadoop to add
necessary runtime jars to the classpath. For example, distributions
of Hadoop that use Apache Knox for identity management can add
Knox jars to this meta-package.

This switches Impala to use the hadoop-cloud-storage package. This
does not change the default configuration, but the USE_CDP_HIVE=true
configuration gains Apache Knox jars.

Due to the structure of the CDP maven repository, it has dependencies
on versions of jars that are not in the CDP maven repository. So,
USE_CDP_HIVE=true requires the Hortonworks public maven repository.
This is added, but it is only enabled with USE_CDP_HIVE=true and only
for releases (no SNAPSHOTs).

Testing:
 - Ran core tests
 - Manually verified classpath changes

Change-Id: I08f1c36ecf54ac277d99e2d2843163eada732e50
---
M fe/pom.xml
M impala-parent/pom.xml
2 files changed, 18 insertions(+), 14 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/72/13872/2
-- 
To view, visit http://gerrit.cloudera.org:8080/13872
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I08f1c36ecf54ac277d99e2d2843163eada732e50
Gerrit-Change-Number: 13872
Gerrit-PatchSet: 2
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>

[Impala-ASF-CR] IMPALA-8766: Change cloud dependencies to use hadoop-cloud-storage

Posted by "Vihang Karajgaonkar (Code Review)" <ge...@cloudera.org>.
Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/13872 )

Change subject: IMPALA-8766: Change cloud dependencies to use hadoop-cloud-storage
......................................................................


Patch Set 3:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/13872/3/impala-parent/pom.xml
File impala-parent/pom.xml:

http://gerrit.cloudera.org:8080/#/c/13872/3/impala-parent/pom.xml@186
PS3, Line 186: repository
I think it would great, if we could add a comment here describing why we need this repository and reference to the JIRA number of this patch.



-- 
To view, visit http://gerrit.cloudera.org:8080/13872
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I08f1c36ecf54ac277d99e2d2843163eada732e50
Gerrit-Change-Number: 13872
Gerrit-PatchSet: 3
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Mon, 22 Jul 2019 19:16:17 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-8766: Change cloud dependencies to use hadoop-cloud-storage

Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Hello Vihang Karajgaonkar, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/13872

to look at the new patch set (#4).

Change subject: IMPALA-8766: Change cloud dependencies to use hadoop-cloud-storage
......................................................................

IMPALA-8766: Change cloud dependencies to use hadoop-cloud-storage

Currently, fe/pom.xml includes direct dependencies on hadoop-aws,
hadoop-azure, and hadoop-azure-datalake. Hadoop has a meta-package
hadoop-cloud-storage that provides these dependencies with appropriate
exclusions. This would not change Impala's build significantly, but
it allows different versions or distributions of Hadoop to add
necessary runtime jars to the classpath. For example, distributions
of Hadoop that use Apache Knox for identity management can add
Knox jars to this meta-package.

This switches Impala to use the hadoop-cloud-storage package. This
does not change the default configuration, but the USE_CDP_HIVE=true
configuration gains Apache Knox jars.

Due to the structure of the CDP maven repository, it has dependencies
on versions of jars that are not in the CDP maven repository. So,
USE_CDP_HIVE=true requires the Hortonworks public maven repository.
This is added, but it is only enabled with USE_CDP_HIVE=true and only
for releases (no SNAPSHOTs).

Testing:
 - Ran core tests
 - Manually verified classpath changes

Change-Id: I08f1c36ecf54ac277d99e2d2843163eada732e50
---
M fe/pom.xml
M impala-parent/pom.xml
2 files changed, 29 insertions(+), 14 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/72/13872/4
-- 
To view, visit http://gerrit.cloudera.org:8080/13872
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I08f1c36ecf54ac277d99e2d2843163eada732e50
Gerrit-Change-Number: 13872
Gerrit-PatchSet: 4
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>

[Impala-ASF-CR] IMPALA-8766: Change cloud dependencies to use hadoop-cloud-storage

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13872 )

Change subject: IMPALA-8766: Change cloud dependencies to use hadoop-cloud-storage
......................................................................


Patch Set 3:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4621/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/13872
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I08f1c36ecf54ac277d99e2d2843163eada732e50
Gerrit-Change-Number: 13872
Gerrit-PatchSet: 3
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Comment-Date: Fri, 19 Jul 2019 22:34:41 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8766: Change cloud dependencies to use hadoop-cloud-storage

Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/13872 )

Change subject: IMPALA-8766: Change cloud dependencies to use hadoop-cloud-storage
......................................................................


Patch Set 2:

Nexus needs https.


-- 
To view, visit http://gerrit.cloudera.org:8080/13872
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I08f1c36ecf54ac277d99e2d2843163eada732e50
Gerrit-Change-Number: 13872
Gerrit-PatchSet: 2
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Comment-Date: Thu, 18 Jul 2019 15:50:09 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8766: Change cloud dependencies to use hadoop-cloud-storage

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13872 )

Change subject: IMPALA-8766: Change cloud dependencies to use hadoop-cloud-storage
......................................................................


Patch Set 1:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4607/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/13872
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I08f1c36ecf54ac277d99e2d2843163eada732e50
Gerrit-Change-Number: 13872
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Tue, 16 Jul 2019 23:37:28 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8766: Change cloud dependencies to use hadoop-cloud-storage

Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/13872 )

Change subject: IMPALA-8766: Change cloud dependencies to use hadoop-cloud-storage
......................................................................


Patch Set 1:

Investigating whether this works on upstream infrastructure.


-- 
To view, visit http://gerrit.cloudera.org:8080/13872
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I08f1c36ecf54ac277d99e2d2843163eada732e50
Gerrit-Change-Number: 13872
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Comment-Date: Wed, 17 Jul 2019 19:11:38 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8766: Change cloud dependencies to use hadoop-cloud-storage

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13872 )

Change subject: IMPALA-8766: Change cloud dependencies to use hadoop-cloud-storage
......................................................................


Patch Set 2:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/3909/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/13872
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I08f1c36ecf54ac277d99e2d2843163eada732e50
Gerrit-Change-Number: 13872
Gerrit-PatchSet: 2
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Comment-Date: Thu, 18 Jul 2019 16:30:05 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8766: Change cloud dependencies to use hadoop-cloud-storage

Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Joe McDonnell has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13872 )

Change subject: IMPALA-8766: Change cloud dependencies to use hadoop-cloud-storage
......................................................................

IMPALA-8766: Change cloud dependencies to use hadoop-cloud-storage

Currently, fe/pom.xml includes direct dependencies on hadoop-aws,
hadoop-azure, and hadoop-azure-datalake. Hadoop has a meta-package
hadoop-cloud-storage that provides these dependencies with appropriate
exclusions. This would not change Impala's build significantly, but
it allows different versions or distributions of Hadoop to add
necessary runtime jars to the classpath. For example, distributions
of Hadoop that use Apache Knox for identity management can add
Knox jars to this meta-package.

This switches Impala to use the hadoop-cloud-storage package. This
does not change the default configuration, but the USE_CDP_HIVE=true
configuration gains Apache Knox jars.

Due to the structure of the CDP maven repository, it has dependencies
on versions of jars that are not in the CDP maven repository. So,
USE_CDP_HIVE=true requires the Hortonworks public maven repository.
This is added, but it is only enabled with USE_CDP_HIVE=true and only
for releases (no SNAPSHOTs).

Testing:
 - Ran core tests
 - Manually verified classpath changes

Change-Id: I08f1c36ecf54ac277d99e2d2843163eada732e50
Reviewed-on: http://gerrit.cloudera.org:8080/13872
Tested-by: Impala Public Jenkins <im...@cloudera.com>
Reviewed-by: Vihang Karajgaonkar <vi...@cloudera.com>
---
M fe/pom.xml
M impala-parent/pom.xml
2 files changed, 29 insertions(+), 14 deletions(-)

Approvals:
  Impala Public Jenkins: Verified
  Vihang Karajgaonkar: Looks good to me, approved

-- 
To view, visit http://gerrit.cloudera.org:8080/13872
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I08f1c36ecf54ac277d99e2d2843163eada732e50
Gerrit-Change-Number: 13872
Gerrit-PatchSet: 5
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>

[Impala-ASF-CR] IMPALA-8766: Change cloud dependencies to use hadoop-cloud-storage

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13872 )

Change subject: IMPALA-8766: Change cloud dependencies to use hadoop-cloud-storage
......................................................................


Patch Set 3: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/13872
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I08f1c36ecf54ac277d99e2d2843163eada732e50
Gerrit-Change-Number: 13872
Gerrit-PatchSet: 3
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Sat, 20 Jul 2019 04:50:36 +0000
Gerrit-HasComments: No