[jira] [Commented] (HBASE-20332) shaded mapreduce module shouldn't include hadoop

["Sean Busbey (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HBASE-20332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16424152#comment-16424152 ] 

Sean Busbey commented on HBASE-20332:
-------------------------------------

re #1, you currently can't use our shaded mapreduce jar to submit MR applications directly without the use of the {{hadoop jar}} / {{yarn jar}} commands, because we need some Hadoop common bits for dealing with the local filesystem. so if we want a standalone jar for e.g. submitting MR jobs from a node that has no Hadoop installation, that's a different pile of work (and I'd argue a less common case that we should handle after making sure we have a simple "easy" path)

> shaded mapreduce module shouldn't include hadoop
> ------------------------------------------------
>
>                 Key: HBASE-20332
>                 URL: https://issues.apache.org/jira/browse/HBASE-20332
>             Project: HBase
>          Issue Type: Sub-task
>          Components: mapreduce, shading
>    Affects Versions: 2.0.0
>            Reporter: Sean Busbey
>            Priority: Critical
>             Fix For: 2.0.0
>
>
> AFAICT, we should just entirely skip including hadoop in our shaded mapreduce module
> 1) Folks expect to run yarn / mr apps via {{hadoop jar}} / {{yarn jar}}
> 2) those commands include all the needed Hadoop jars in your classpath by default (both client side and in the containers)
> 3) If you try to use "user classpath first" for your job as a workaround (e.g. for some library your application needs that hadoop provides) then our inclusion of *some but not all* hadoop classes then causes everything to fall over because of mixing rewritten and non-rewritten hadoop classes
> 4) if you don't use "user classpath first" then all of our non-relocated-but-still-shaded hadoop classes are ignored anyways so we're just wasting space



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)