You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2021/05/18 08:40:00 UTC
[jira] [Work logged] (HIVE-25101) Remove HBase libraries from Hive
distribution
[ https://issues.apache.org/jira/browse/HIVE-25101?focusedWorklogId=598485&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-598485 ]
ASF GitHub Bot logged work on HIVE-25101:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 18/May/21 08:39
Start Date: 18/May/21 08:39
Worklog Time Spent: 10m
Work Description: stoty commented on pull request #2259:
URL: https://github.com/apache/hive/pull/2259#issuecomment-842975995
Here's what gets changed/removed from the distribution with the patch.
Apart from the obvious HBase libraries, there are some HBase dependencies.
The big one is Jersey 2.25. AFAICT, where Hive uses jersey, it includes the older com.sun.jersey artifacts from Hadoop, so this shouldn't be a problem.
I have added some missing explicit dependencies where the Hive source used undeclared dependencies for validation-api and commons-math3 .
diff <(cd hbase;find . |sort) <(cd nohbase;find . |sort)
515d514
< ./apache-hive-4.0.0-SNAPSHOT-bin/lib/aopalliance-repackaged-2.5.0-b32.jar
574c573
< ./apache-hive-4.0.0-SNAPSHOT-bin/lib/disruptor-3.3.6.jar
---
> ./apache-hive-4.0.0-SNAPSHOT-bin/lib/disruptor-3.3.7.jar
578d576
< ./apache-hive-4.0.0-SNAPSHOT-bin/lib/findbugs-annotations-1.3.9-1.jar
584,602d581
< ./apache-hive-4.0.0-SNAPSHOT-bin/lib/hbase-client-2.0.0-alpha4.jar
< ./apache-hive-4.0.0-SNAPSHOT-bin/lib/hbase-common-2.0.0-alpha4-tests.jar
< ./apache-hive-4.0.0-SNAPSHOT-bin/lib/hbase-common-2.0.0-alpha4.jar
< ./apache-hive-4.0.0-SNAPSHOT-bin/lib/hbase-hadoop-compat-2.0.0-alpha4.jar
< ./apache-hive-4.0.0-SNAPSHOT-bin/lib/hbase-hadoop2-compat-2.0.0-alpha4-tests.jar
< ./apache-hive-4.0.0-SNAPSHOT-bin/lib/hbase-hadoop2-compat-2.0.0-alpha4.jar
< ./apache-hive-4.0.0-SNAPSHOT-bin/lib/hbase-http-2.0.0-alpha4.jar
< ./apache-hive-4.0.0-SNAPSHOT-bin/lib/hbase-mapreduce-2.0.0-alpha4.jar
< ./apache-hive-4.0.0-SNAPSHOT-bin/lib/hbase-metrics-2.0.0-alpha4.jar
< ./apache-hive-4.0.0-SNAPSHOT-bin/lib/hbase-metrics-api-2.0.0-alpha4.jar
< ./apache-hive-4.0.0-SNAPSHOT-bin/lib/hbase-prefix-tree-2.0.0-alpha4.jar
< ./apache-hive-4.0.0-SNAPSHOT-bin/lib/hbase-procedure-2.0.0-alpha4.jar
< ./apache-hive-4.0.0-SNAPSHOT-bin/lib/hbase-protocol-2.0.0-alpha4.jar
< ./apache-hive-4.0.0-SNAPSHOT-bin/lib/hbase-protocol-shaded-2.0.0-alpha4.jar
< ./apache-hive-4.0.0-SNAPSHOT-bin/lib/hbase-replication-2.0.0-alpha4.jar
< ./apache-hive-4.0.0-SNAPSHOT-bin/lib/hbase-server-2.0.0-alpha4.jar
< ./apache-hive-4.0.0-SNAPSHOT-bin/lib/hbase-shaded-miscellaneous-1.0.1.jar
< ./apache-hive-4.0.0-SNAPSHOT-bin/lib/hbase-shaded-netty-1.0.1.jar
< ./apache-hive-4.0.0-SNAPSHOT-bin/lib/hbase-shaded-protobuf-1.0.1.jar
641,643d619
< ./apache-hive-4.0.0-SNAPSHOT-bin/lib/hk2-api-2.5.0-b32.jar
< ./apache-hive-4.0.0-SNAPSHOT-bin/lib/hk2-locator-2.5.0-b32.jar
< ./apache-hive-4.0.0-SNAPSHOT-bin/lib/hk2-utils-2.5.0-b32.jar
645c621
< ./apache-hive-4.0.0-SNAPSHOT-bin/lib/htrace-core-3.2.0-incubating.jar
---
> ./apache-hive-4.0.0-SNAPSHOT-bin/lib/htrace-core-3.1.0-incubating.jar
671d646
< ./apache-hive-4.0.0-SNAPSHOT-bin/lib/javax.inject-2.5.0-b32.jar
680d654
< ./apache-hive-4.0.0-SNAPSHOT-bin/lib/jcodings-1.0.18.jar
683,687d656
< ./apache-hive-4.0.0-SNAPSHOT-bin/lib/jersey-client-2.25.1.jar
< ./apache-hive-4.0.0-SNAPSHOT-bin/lib/jersey-common-2.25.1.jar
< ./apache-hive-4.0.0-SNAPSHOT-bin/lib/jersey-container-servlet-core-2.25.1.jar
< ./apache-hive-4.0.0-SNAPSHOT-bin/lib/jersey-guava-2.25.1.jar
< ./apache-hive-4.0.0-SNAPSHOT-bin/lib/jersey-media-jaxb-2.25.1.jar
689d657
< ./apache-hive-4.0.0-SNAPSHOT-bin/lib/jersey-server-2.25.1.jar
720d687
< ./apache-hive-4.0.0-SNAPSHOT-bin/lib/joni-2.1.11.jar
777d743
< ./apache-hive-4.0.0-SNAPSHOT-bin/lib/osgi-resource-locator-1.0.1.jar
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 598485)
Time Spent: 20m (was: 10m)
> Remove HBase libraries from Hive distribution
> ---------------------------------------------
>
> Key: HIVE-25101
> URL: https://issues.apache.org/jira/browse/HIVE-25101
> Project: Hive
> Issue Type: Improvement
> Components: HBase Handler, Hive
> Affects Versions: 4.0.0
> Reporter: Istvan Toth
> Assignee: Istvan Toth
> Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Hive currently packages HBase libraries into its lib directory.
> It also adds the HBase libraries separately to its classpath in the hive startup script.
> Having both mechanisms is redundant, and it also causes errors, as the standard HBase libraries packaged into Hive are unshaded, while the libraries added by _hbase mapredcp_
> are shaded, and the two are NOT compatible when custom coprocessors are used, and in some cases the classpaths during local execution and for MR/TEZ jobs are mutually incompatible.
> I propose removing all HBase libraries from the distribution, and pulling them via the hbase mapredcp mechanism.
> This also solves the old problem of including ancient HBase alpha versions Hive.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)