You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Josh Elser (JIRA)" <ji...@apache.org> on 2018/06/05 16:51:00 UTC
[jira] [Commented] (HBASE-20332) shaded mapreduce module shouldn't
include hadoop
[ https://issues.apache.org/jira/browse/HBASE-20332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16502096#comment-16502096 ]
Josh Elser commented on HBASE-20332:
------------------------------------
{quote}everyone fine with me punting the check for old hbase apis to a follow-on that's blocked on the checkstyle update needed for the "illegal classes" rule?
{quote}
+1
> shaded mapreduce module shouldn't include hadoop
> ------------------------------------------------
>
> Key: HBASE-20332
> URL: https://issues.apache.org/jira/browse/HBASE-20332
> Project: HBase
> Issue Type: Sub-task
> Components: mapreduce, shading
> Affects Versions: 2.0.0
> Reporter: Sean Busbey
> Assignee: Sean Busbey
> Priority: Critical
> Fix For: 3.0.0, 2.1.0
>
> Attachments: HBASE-20332.0.patch, HBASE-20332.1.WIP.patch, HBASE-20332.2.WIP.patch, HBASE-20332.3.patch
>
>
> AFAICT, we should just entirely skip including hadoop in our shaded mapreduce module
> 1) Folks expect to run yarn / mr apps via {{hadoop jar}} / {{yarn jar}}
> 2) those commands include all the needed Hadoop jars in your classpath by default (both client side and in the containers)
> 3) If you try to use "user classpath first" for your job as a workaround (e.g. for some library your application needs that hadoop provides) then our inclusion of *some but not all* hadoop classes then causes everything to fall over because of mixing rewritten and non-rewritten hadoop classes
> 4) if you don't use "user classpath first" then all of our non-relocated-but-still-shaded hadoop classes are ignored anyways so we're just wasting space
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)