You are viewing a plain text version of this content. The canonical link for it is here.

Posted to reviews@impala.apache.org by "Joe McDonnell (Code Review)" <ge...@cloudera.org> on 2019/08/06 23:13:22 UTC

[Impala-ASF-CR] IMPALA-8766: Exorcise hadoop-cloud-storage + HWX Nexus

Joe McDonnell has uploaded this change for review. ( http://gerrit.cloudera.org:8080/14024

Change subject: IMPALA-8766: Exorcise hadoop-cloud-storage + HWX Nexus
......................................................................

IMPALA-8766: Exorcise hadoop-cloud-storage + HWX Nexus

Previous commits for IMPALA-8766 attempted to use hadoop-cloud-storage
to satisfy Impala's cloud dependencies (e.g. hadoop-aws, hadoop-azure,
etc). On builds with USE_CDP_HIVE=true, this adds Knox
gateway-cloud-bindings. However, the entry for hadoop-cloud-storage
artifact in the impala.cdp.repo maven repository introduces
dependencies that are external to that repository. This requires the
HWX Nexus repository to resolve those dangling dependencies.
Unfortunately, HWX Nexus ages out old jars, including the ones we
need.

This stops using hadoop-cloud-storage, and instead adds a direct
dependency to Knox for USE_CDP_HIVE=true. It disables the HWX Nexus
repository and leaves a tombstone explaining why.

Testing:
- Deleted my .m2 directory and rebuilt Impala with USE_CDP_HIVE=true
- Verified the CLASSPATH still contains the right jars on USE_CDP_HIVE=true

Change-Id: I79a0c2575fc50bbc3b393c150c0bce22258ea1bd
---
M bin/impala-config.sh
M fe/pom.xml
M impala-parent/pom.xml
3 files changed, 49 insertions(+), 49 deletions(-)

git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/24/14024/1
--
To view, visit http://gerrit.cloudera.org:8080/14024
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I79a0c2575fc50bbc3b393c150c0bce22258ea1bd
Gerrit-Change-Number: 14024
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>

[Impala-ASF-CR] IMPALA-8766: Undo hadoop-cloud-storage + HWX Nexus

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14024 )

Change subject: IMPALA-8766: Undo hadoop-cloud-storage + HWX Nexus
......................................................................


Patch Set 2:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4760/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/14024
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I79a0c2575fc50bbc3b393c150c0bce22258ea1bd
Gerrit-Change-Number: 14024
Gerrit-PatchSet: 2
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Thu, 08 Aug 2019 17:06:20 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8766: Undo hadoop-cloud-storage + HWX Nexus

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14024 )

Change subject: IMPALA-8766: Undo hadoop-cloud-storage + HWX Nexus
......................................................................


Patch Set 3:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/4217/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/14024
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I79a0c2575fc50bbc3b393c150c0bce22258ea1bd
Gerrit-Change-Number: 14024
Gerrit-PatchSet: 3
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Sun, 11 Aug 2019 04:27:57 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8766: Undo hadoop-cloud-storage + HWX Nexus

Posted by "Vihang Karajgaonkar (Code Review)" <ge...@cloudera.org>.

Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/14024 )

Change subject: IMPALA-8766: Undo hadoop-cloud-storage + HWX Nexus
......................................................................


Patch Set 2:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/14024/2/bin/impala-config.sh
File bin/impala-config.sh:

http://gerrit.cloudera.org:8080/#/c/14024/2/bin/impala-config.sh@219
PS2, Line 219: UNUSED
Why do we need this env variable in case of hive-2? May be add a comment since its not obvious to me.



-- 
To view, visit http://gerrit.cloudera.org:8080/14024
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I79a0c2575fc50bbc3b393c150c0bce22258ea1bd
Gerrit-Change-Number: 14024
Gerrit-PatchSet: 2
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Fri, 09 Aug 2019 21:20:31 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-8766: Undo hadoop-cloud-storage + HWX Nexus

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14024 )

Change subject: IMPALA-8766: Undo hadoop-cloud-storage + HWX Nexus
......................................................................


Patch Set 2: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/14024
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I79a0c2575fc50bbc3b393c150c0bce22258ea1bd
Gerrit-Change-Number: 14024
Gerrit-PatchSet: 2
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Fri, 09 Aug 2019 01:52:14 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8766: Undo hadoop-cloud-storage + HWX Nexus

Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.

Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/14024 )

Change subject: IMPALA-8766: Undo hadoop-cloud-storage + HWX Nexus
......................................................................


Patch Set 2:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/14024/2/bin/impala-config.sh
File bin/impala-config.sh:

http://gerrit.cloudera.org:8080/#/c/14024/2/bin/impala-config.sh@219
PS2, Line 219: UNUSED
> Why do we need this env variable in case of hive-2? May be add a comment si
Good point, removed this (and USE_CDP_HIVE=false still compiles fine).



-- 
To view, visit http://gerrit.cloudera.org:8080/14024
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I79a0c2575fc50bbc3b393c150c0bce22258ea1bd
Gerrit-Change-Number: 14024
Gerrit-PatchSet: 2
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Sun, 11 Aug 2019 03:47:27 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-8766: Undo hadoop-cloud-storage + HWX Nexus

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14024 )

Change subject: IMPALA-8766: Undo hadoop-cloud-storage + HWX Nexus
......................................................................


Patch Set 2: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4760/


-- 
To view, visit http://gerrit.cloudera.org:8080/14024
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I79a0c2575fc50bbc3b393c150c0bce22258ea1bd
Gerrit-Change-Number: 14024
Gerrit-PatchSet: 2
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Thu, 08 Aug 2019 21:17:34 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8766: Undo hadoop-cloud-storage + HWX Nexus

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14024 )

Change subject: IMPALA-8766: Undo hadoop-cloud-storage + HWX Nexus
......................................................................


Patch Set 3: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/14024
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I79a0c2575fc50bbc3b393c150c0bce22258ea1bd
Gerrit-Change-Number: 14024
Gerrit-PatchSet: 3
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Sun, 11 Aug 2019 07:55:30 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8766: Undo hadoop-cloud-storage + HWX Nexus

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14024 )

Change subject: IMPALA-8766: Undo hadoop-cloud-storage + HWX Nexus
......................................................................


Patch Set 3:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4769/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/14024
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I79a0c2575fc50bbc3b393c150c0bce22258ea1bd
Gerrit-Change-Number: 14024
Gerrit-PatchSet: 3
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Sun, 11 Aug 2019 03:47:57 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8766: Undo hadoop-cloud-storage + HWX Nexus

Posted by "Vihang Karajgaonkar (Code Review)" <ge...@cloudera.org>.

Vihang Karajgaonkar has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/14024 )

Change subject: IMPALA-8766: Undo hadoop-cloud-storage + HWX Nexus
......................................................................

IMPALA-8766: Undo hadoop-cloud-storage + HWX Nexus

Previous commits for IMPALA-8766 attempted to use hadoop-cloud-storage
to satisfy Impala's cloud dependencies (e.g. hadoop-aws, hadoop-azure,
etc). On builds with USE_CDP_HIVE=true, this adds Knox
gateway-cloud-bindings. However, the entry for hadoop-cloud-storage
artifact in the impala.cdp.repo maven repository introduces
dependencies that are external to that repository. This requires the
HWX Nexus repository to resolve those dangling dependencies.
Unfortunately, HWX Nexus ages out old jars, including the ones we
need.

This stops using hadoop-cloud-storage, and instead adds a direct
dependency to Knox for USE_CDP_HIVE=true. It disables the HWX Nexus
repository and leaves a tombstone explaining why.

Testing:
 - Deleted my .m2 directory and rebuilt Impala with USE_CDP_HIVE=true
 - Verified the CLASSPATH still contains the right jars on USE_CDP_HIVE=true

Change-Id: I79a0c2575fc50bbc3b393c150c0bce22258ea1bd
Reviewed-on: http://gerrit.cloudera.org:8080/14024
Tested-by: Impala Public Jenkins <im...@cloudera.com>
Reviewed-by: Vihang Karajgaonkar <vi...@cloudera.com>
---
M bin/impala-config.sh
M fe/pom.xml
M impala-parent/pom.xml
3 files changed, 48 insertions(+), 49 deletions(-)

Approvals:
  Impala Public Jenkins: Verified
  Vihang Karajgaonkar: Looks good to me, approved

-- 
To view, visit http://gerrit.cloudera.org:8080/14024
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I79a0c2575fc50bbc3b393c150c0bce22258ea1bd
Gerrit-Change-Number: 14024
Gerrit-PatchSet: 4
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>

[Impala-ASF-CR] IMPALA-8766: Undo hadoop-cloud-storage + HWX Nexus

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14024 )

Change subject: IMPALA-8766: Undo hadoop-cloud-storage + HWX Nexus
......................................................................


Patch Set 2:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/4183/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/14024
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I79a0c2575fc50bbc3b393c150c0bce22258ea1bd
Gerrit-Change-Number: 14024
Gerrit-PatchSet: 2
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Thu, 08 Aug 2019 17:00:22 +0000
Gerrit-HasComments: No