You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "ASF subversion and git services (JIRA)" <ji...@apache.org> on 2019/08/13 03:13:00 UTC
[jira] [Commented] (IMPALA-8766) Change hadoop cloud dependencies
to use hadoop-cloud-storage
[ https://issues.apache.org/jira/browse/IMPALA-8766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16905765#comment-16905765 ]
ASF subversion and git services commented on IMPALA-8766:
---------------------------------------------------------
Commit 8094811b5d975e18e20071552f86c2e3f8c0fc8f in impala's branch refs/heads/master from Joe McDonnell
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=8094811 ]
IMPALA-8766: Undo hadoop-cloud-storage + HWX Nexus
Previous commits for IMPALA-8766 attempted to use hadoop-cloud-storage
to satisfy Impala's cloud dependencies (e.g. hadoop-aws, hadoop-azure,
etc). On builds with USE_CDP_HIVE=true, this adds Knox
gateway-cloud-bindings. However, the entry for hadoop-cloud-storage
artifact in the impala.cdp.repo maven repository introduces
dependencies that are external to that repository. This requires the
HWX Nexus repository to resolve those dangling dependencies.
Unfortunately, HWX Nexus ages out old jars, including the ones we
need.
This stops using hadoop-cloud-storage, and instead adds a direct
dependency to Knox for USE_CDP_HIVE=true. It disables the HWX Nexus
repository and leaves a tombstone explaining why.
Testing:
- Deleted my .m2 directory and rebuilt Impala with USE_CDP_HIVE=true
- Verified the CLASSPATH still contains the right jars on USE_CDP_HIVE=true
Change-Id: I79a0c2575fc50bbc3b393c150c0bce22258ea1bd
Reviewed-on: http://gerrit.cloudera.org:8080/14024
Tested-by: Impala Public Jenkins <im...@cloudera.com>
Reviewed-by: Vihang Karajgaonkar <vi...@cloudera.com>
> Change hadoop cloud dependencies to use hadoop-cloud-storage
> ------------------------------------------------------------
>
> Key: IMPALA-8766
> URL: https://issues.apache.org/jira/browse/IMPALA-8766
> Project: IMPALA
> Issue Type: Improvement
> Components: Infrastructure
> Affects Versions: Impala 3.3.0
> Reporter: Joe McDonnell
> Assignee: Joe McDonnell
> Priority: Major
> Fix For: Impala 3.3.0
>
>
> Currently, fe/pom.xml specifically includes hadoop-aws, hadoop-azure, and hadoop-azure-datalake directly. There is a meta-package in hadoop called hadoop-cloud-storage that includes these dependencies and others as customized by the hadoop provider, with appropriate exclusions applied to each package.
> Migrating Impala to use this meta-package would make it easier for different providers of hadoop to customize hadoop-cloud-storage and the resulting CLASSPATH without needing to change Impala. For example, a hadoop provider may want to include Apache Knox for cloud identity management.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org