[jira] [Commented] (PHOENIX-7118) Fix Shading Regressions in Connectors

["Istvan Toth (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/PHOENIX-7118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17788706#comment-17788706 ] 

Istvan Toth commented on PHOENIX-7118:
--------------------------------------

revisiting the shaded beanutils and configuration2 conflicts issues:
 * commons-configuration2
 ** hbase (2.1+): transitive only from Hadoop
 ** phoenix (5.1.0 + ) : 2.2.1
 ** omid: NOT USED
 ** hadoop (3.1): 2.1.1
 * commons-beanutils
 ** hbase :  transitive only from hadoop
 ** phoenix (5.1.0 + ) : only transitive 
 ** omid (1.0.2): 1.9.4 
 ** hadoop: 3.0: 1.9.3 | 3.1: 1.9.4

This does not mesh with that I wrote in PHOENIX-6053 .
configuration2 seems to use compatible versions.

Beanutils is also at least 1.9.3 ever since HBase 2.1.
It transitively gets 1.8.0 when built with Hadoop2, but that is not supported anyway.

So it looks like the version differences are not significant enough to cause problems.

However, the shading still can.


*Hadoop metrics2* leaks *configuration2* in the MetricsSink API, so we need to make sure that
if we do not include Hadoop, then we shade according to the Hadoop on the classpath.

hbase-shaded-client uses: *org/apache/hadoop/hbase/shaded/org/apache/commons/configuration2/*
hbase-shaded-client-byo-hadoop uses: {*}not present{*}, taken from Hadoop on classpath

hadoop-client-runtime uses: *org/apache/hadoop/shaded/org/apache/commons/configuration2*

Apparently Spark 3.2+ uses hbase-client-api, while older Spark versions use unshaded Hadoop.
Hive seems to use the standard (unshaded) Hadoop libraries even in HEAD.

BeanUtils shading does not differ, both the shaded and the byo packages include *org/apache/hadoop/hbase/shaded/org/apache/commons/beanutils/*
However, neither HBase 2, nor Hadoop3 actually use commons-beanutils, Hadoop3 just depends on it for no reason.
So it looks like commons-beanutils does not actually need any special handling, apart from making sure that the included version is new enough for Omid.

> Fix Shading Regressions in Connectors
> -------------------------------------
>
>                 Key: PHOENIX-7118
>                 URL: https://issues.apache.org/jira/browse/PHOENIX-7118
>             Project: Phoenix
>          Issue Type: Bug
>          Components: connectors, hive-connector, spark-connector
>    Affects Versions: connectors-6.0.0
>            Reporter: Istvan Toth
>            Assignee: Istvan Toth
>            Priority: Major
>
> It seems like sometime during the refactors, we have broken the shading, particularly the exclusions / provided settings.
> Only the Spark3 connector shading looks correct, both the Hive and Spark2 connector shadings seem to have regressed.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)