[jira] Updated: (HADOOP-3570) Including user specified jar files in the client side classpath path in Hadoop 0.17 streaming

["Sharad Agarwal (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HADOOP-3570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sharad Agarwal updated HADOOP-3570:
-----------------------------------

    Attachment: 3570_v1.patch

I think this is applicable for all kinds of jobs, not limited to streaming only.
One way to handle this is in GenericOptionsParser itself. Adding the parsed jars to the conf classpath. It is demonstrated in attached patch. It is not tested yet.
Other thing which needs to be handled is that if the libjars exist on the HDFS then we would require to do copyToLocal.
Thoughts?

> Including user specified jar files in the client side classpath path in Hadoop 0.17 streaming
> ---------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-3570
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3570
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: contrib/streaming
>    Affects Versions: 0.17.0
>            Reporter: Suhas Gogate
>            Assignee: Mahadev konar
>             Fix For: 0.19.0
>
>         Attachments: 3570_v1.patch
>
>
> Hadoop 0.17 streaming allows specifying multiple user specific jar files on the command line along with the hadoop-streaming.jar. These user specific jar files are not automatically  added on the client side class path when hadoop runs the streaming job
> For example, in the following streaming job <user-app.jar> is not automatically included in client side path before executing the streaming job
>  
> hadoop  jar -libjars <user-app.jar> $HADOOP_HOME/hadoop-streaming.jar \
>         -input </input-data-path> \
>         -output </output data path> \
>         -inputformat  <Input-format> \
>         -outputformat <OutputFormat> \
>         -mapper /bin/cat -reducer /bin/cat \
>         -additionalconfspec  <xxxx> \
>         -jobconf mapred.reduce.tasks=3\
> Workaround:
> Current work around is to set these user specific jar files in
> HADOOP_CLASSPATH environment variable on the client side before running above example.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.