[jira] [Comment Edited] (HBASE-26909) hbase-shaded-mapreduce and hbase-shaded-client expose some of the same classes

["Sean Busbey (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HBASE-26909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17514915#comment-17514915 ] 

Sean Busbey edited comment on HBASE-26909 at 3/30/22, 7:46 PM:
---------------------------------------------------------------

that's an important distinction. if we want a refactoring to pull out the duplicate classes, hbase-shaded-mapreduce will need to show a runtime dependency on the hbase-shaded-client-byo-hadoop artifact and not the one that include hadoop classes. the intended use of hbase-shaded-mapreduce is always with an existing hadoop; the hadoop classes should come from that existing install.


was (Author: busbey):
that's an important distinction. if we want a refactoring to pull out the duplicate classes, hbase-shaded-mapreduce will need to show a runtime dependency on the hbase-shaded-client-byo-hadoop artifact and not the one that include hadoop classes. the intended use of abase-shaded-mapreduce is always with an existing hadoop; the hadoop classes should come from that existing install.

> hbase-shaded-mapreduce and hbase-shaded-client expose some of the same classes
> ------------------------------------------------------------------------------
>
>                 Key: HBASE-26909
>                 URL: https://issues.apache.org/jira/browse/HBASE-26909
>             Project: HBase
>          Issue Type: Improvement
>            Reporter: Bryan Beaudreault
>            Priority: Major
>
> We supply 2 primary artifacts for end-users to consume:
>  * hbase-shaded-client, which is for general use
>  * hbase-shaded-mapreduce, which is for use when you need to connect to hbase via mapreduce. For example, TableInputFormat
> The problem is that these artifacts expose tons of duplicate classes. One example (among many) is org.apache.hadoop.hbase.Cell, which appears in both jars.
> This may not be a problem if your projects are always very isolated – either doing mapreduce, or not. In that case you just depend in the one you need. Many users might exist in much more complicated environments where dependencies tend to bleed along more between projects. Here's an illustration:
>  * Imagine a project FooService, which includes two modules FooServiceRestWeb (for the rest http resources) and FooServiceData (which includes DAOs for accessing data). FooServiceRestWeb depends on FooServiceData to access hbase.  In this case, FooServiceData should depend on hbase-shaded-client.
>  * Now imagine another project FooPipeline, which has modules FooPipelineHadoop (with M/R jobs for processing data) and FooPipelineData (which has some DAOs for accessing data). In this case, FooPipelineData might depend on hbase-shaded-mapreduce since the context is intended for M/R.
>  * The problem arises when suddenly we want to include some data from FooService into our pipeline. The most straightforward way to achieve this is by depending on FooServiceData,  which has all of he DAOs for that data but also depends on hbase-shaded-client. At this point you have a problem, because FooPipelineHadoop now depends on both hbase-shaded-mapreduce and hbase-shaded-client.
> (Note, this obviously skirts around potential microservice solutions like only accessing FooService's data through the API... it's just for illustration, and it does come up.)
> From a plain java perspective, having these 2 jars on the classpath is somewhat wasteful but not a huge issue since the implementations are all the same.
> From a maven perspective, it's problematic because the maven dependency plugin will complain about the conflicting classes.
> One potential fix is to add exclusions to the FooServiceData dependency, to avoid pulling in hbase-shaded-client. This works on a one-off basis but is much more painful in a large and complicated environment where this may come up hundreds of times.
> A better fix in my opinion is to make hbase-shaded-mapreduce depend on hbase-shaded-client and then only expose the classes that aren't already exposed by the shaded client.
> [~busbey] also mentioned a BOM being a potential solution, but I don't have experience with that.
>  



--
This message was sent by Atlassian Jira
(v8.20.1#820001)