You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "Stamatis Zampetakis (Jira)" <ji...@apache.org> on 2023/06/27 10:58:00 UTC

[jira] [Updated] (HIVE-27465) Binary distribution contains multiple identical classes in different jars

     [ https://issues.apache.org/jira/browse/HIVE-27465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Stamatis Zampetakis updated HIVE-27465:
---------------------------------------
    Attachment: all_lib_dup_classes_sorted.txt

> Binary distribution contains multiple identical classes in different jars
> -------------------------------------------------------------------------
>
>                 Key: HIVE-27465
>                 URL: https://issues.apache.org/jira/browse/HIVE-27465
>             Project: Hive
>          Issue Type: Bug
>            Reporter: Stamatis Zampetakis
>            Priority: Major
>         Attachments: all_lib_dup_classes_sorted.txt
>
>
> The problem exists in current master but is also present in previous Hive releases.
> Consider for example the 4.0.0-alpha-2 release.
> Download and untar the respective archive:
> https://dlcdn.apache.org/hive/hive-4.0.0-alpha-2/apache-hive-4.0.0-alpha-2-bin.tar.gz
> Inspect the lib directory and observe that some classes can be found both inside the hive-exec-4.0.0-alpha-2.jar and also in the original jar of the dependency.
> For instance check the AvaticaUtils.class:
> {noformat}
> jar tf  hive-exec-4.0.0-alpha-2.jar | grep org/apache/calcite/avatica/AvaticaUtils.class
> jar tf  avatica-1.12.0.jar | grep org/apache/calcite/avatica/AvaticaUtils.class
> {noformat}
> This is not specific to avatica but appears also for other dependencies. It comes from the fact that hive-exec module shades a lot of dependencies but at the same time the maven-assembly-plugin that is used to build the binary distribution copies all these dependencies under the lib directory.
> As long as we use the same version of the class in both places this shouldn't be a big problem. However, there are still some inconveniences in having duplicate classes:
> * Increases the size of the binary distro
> * Consumes more file descriptors
> * Increases likelihood of classpath problems
> In current master the problem can be see by running:
> {noformat}
> mvn clean package -Pdist -Piceberg -Pitests -DskipTests
> {noformat}
> And then inspecting the jars in the generated bin directory:
> {noformat}
> find packaging/target/apache-hive-4.0.0-beta-1-SNAPSHOT-bin/apache-hive-4.0.0-beta-1-SNAPSHOT-bin -name "*.jar" -exec jar tf {} \; | grep ".class" | sort | uniq -c | grep -v "[[:space:]]\+1 " | sort -n -k1 -r > all_lib_dup_classes_sorted.txt 
> {noformat}
> There are roughly 30K classes that appear more than once in various jars.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)