You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Remy Dubois <rd...@talend.com> on 2014/08/06 19:17:19 UTC
WebHCat and MapReduce JAR
Hi,
I'm trying to submit a MapReduce job through webhcat using the templeton/v1/mapreduce/jar path.
My job depends on third party libraries. I have tried a lot of things such as using the libjars FormParam (the jar is located on HDFS), but it doesn't work, I got ClassNotFoundException.
I've workaround my issue merging the job jar and the dependencies jars, but doing that is not really possible with my use case.
Can you please let me know if what I'm trying to do is possible ? I am able to make the same job work, using the ToolRunner API (with the -libjars parameter) without any problem.
Thanks a lot in advance,
Remy Dubois
Component team manager and Big Data principal
Tel +33 1 46 25 06 00 | Mobile +33 6 11 73 56 72 | Email rdubois@talend.com<ma...@talend.com> | Skype remy.dubois.talend
Talend SA | 9, rue Pagès - 92150 Suresnes - France | +33 1 46 25 06 00 | www.talend.com<http://www.talend.com>
Global Leader in Open Source Integration
[http://www.talend.com/sites/default/files/logo-talend.jpg]<http://www.talend.com/>
RE: WebHCat and MapReduce JAR
Posted by Remy Dubois <rd...@talend.com>.
My problem is solved. I didn’t use the ToolRunner in that M/R job. So it was unable to read the libjars correctly.
Thanks for your time.
Remy Dubois
Component team manager and Big Data principal
Tel +33 1 46 25 06 00 | Mobile +33 6 11 73 56 72 | Email rdubois@talend.com<ma...@talend.com> | Skype remy.dubois.talend
Talend SA | 9, rue Pagès - 92150 Suresnes - France | +33 1 46 25 06 00 | www.talend.com<http://www.talend.com>
Global Leader in Open Source Integration
[http://www.talend.com/sites/default/files/logo-talend.jpg]<http://www.talend.com/>
De : Eugene Koifman [mailto:ekoifman@hortonworks.com]
Envoyé : mercredi 6 août 2014 22:04
À : user@hive.apache.org
Objet : Re: WebHCat and MapReduce JAR
can you try specify the path with "hdfs://...." otherwise it maybe looking at local FS
On Wed, Aug 6, 2014 at 11:24 AM, Remy Dubois <rd...@talend.com>> wrote:
Hi again,
And the issue I have is :
2014-08-06 22:06:37,083 FATAL [IPC Server handler 0 on 47378] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Task: attempt_1407355085865_0004_m_000000_0 - exited : java.lang.ClassNotFoundException: routines.StringHandling
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at mapreduce.MyMapper.map(MyMapper.java:14)
at mapreduce.MyMapper.map(MyMapper.java:1)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
at org.apache.hadoop.mapreduce.lib.chain.Chain.runMapper(Chain.java:389)
at org.apache.hadoop.mapreduce.lib.chain.ChainMapper.run(ChainMapper.java:149)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1557)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
Remy Dubois
Component team manager and Big Data principal
Tel +33 1 46 25 06 00<tel:%2B33%201%2046%2025%2006%2000> | Mobile +33 6 11 73 56 72<tel:%2B33%206%2011%2073%2056%2072> | Email rdubois@talend.com<ma...@talend.com> | Skype remy.dubois.talend
Talend SA | 9, rue Pagès - 92150 Suresnes - France | +33 1 46 25 06 00<tel:%2B33%201%2046%2025%2006%2000> | www.talend.com<http://www.talend.com>
Global Leader in Open Source Integration
[http://www.talend.com/sites/default/files/logo-talend.jpg]<http://www.talend.com/>
De : Remy Dubois [mailto:rdubois@talend.com<ma...@talend.com>]
Envoyé : mercredi 6 août 2014 20:19
À : user@hive.apache.org<ma...@hive.apache.org>
Objet : RE: WebHCat and MapReduce JAR
Hi,
Thanks a lot for your answer. Following some information about what I do.
WebHCat version: Hive 0.13 embedded in HDP 2.1 GA.
The webhcat logs don’t show anything interesting.
REST call:
[cid:image004.jpg@01CFB263.ACDBB4E0]
Jar location on HDFS:
[hdfs@hdp21 root]$ hadoop fs -ls /tmp/systemRoutines.jar
-rw-r--r-- 3 rdubois rdubois 264008 2014-08-06 19:16 /tmp/systemRoutines.jar
[hdfs@hdp21 root]$ hadoop fs -get /tmp/systemRoutines.jar /home/hdfs
[hdfs@hdp21 root]$ unzip -l /home/hdfs/systemRoutines.jar
Archive: /home/hdfs/systemRoutines.jar
Length Date Time Name
-------- ---- ---- ----
102 08-06-14 14:39 META-INF/MANIFEST.MF
3609 08-06-14 14:39 routines/StringHandling.class
Thanks,
Remy Dubois
Component team manager and Big Data principal
Tel +33 1 46 25 06 00<tel:%2B33%201%2046%2025%2006%2000> | Mobile +33 6 11 73 56 72<tel:%2B33%206%2011%2073%2056%2072> | Email rdubois@talend.com<ma...@talend.com> | Skype remy.dubois.talend
Talend SA | 9, rue Pagès - 92150 Suresnes - France | +33 1 46 25 06 00<tel:%2B33%201%2046%2025%2006%2000> | www.talend.com<http://www.talend.com>
Global Leader in Open Source Integration
[http://www.talend.com/sites/default/files/logo-talend.jpg]<http://www.talend.com/>
De : Eugene Koifman [mailto:ekoifman@hortonworks.com]
Envoyé : mercredi 6 août 2014 19:43
À : user@hive.apache.org<ma...@hive.apache.org>
Objet : Re: WebHCat and MapReduce JAR
libjars as form param should work. Is there anything relevant in webhcat.log? What version of webhcat are you using? What is the exact REST call you are making?
On Wed, Aug 6, 2014 at 10:17 AM, Remy Dubois <rd...@talend.com>> wrote:
Hi,
I’m trying to submit a MapReduce job through webhcat using the templeton/v1/mapreduce/jar path.
My job depends on third party libraries. I have tried a lot of things such as using the libjars FormParam (the jar is located on HDFS), but it doesn’t work, I got ClassNotFoundException.
I’ve workaround my issue merging the job jar and the dependencies jars, but doing that is not really possible with my use case.
Can you please let me know if what I’m trying to do is possible ? I am able to make the same job work, using the ToolRunner API (with the –libjars parameter) without any problem.
Thanks a lot in advance,
Remy Dubois
Component team manager and Big Data principal
Tel +33 1 46 25 06 00<tel:%2B33%201%2046%2025%2006%2000> | Mobile +33 6 11 73 56 72<tel:%2B33%206%2011%2073%2056%2072> | Email rdubois@talend.com<ma...@talend.com> | Skype remy.dubois.talend
Talend SA | 9, rue Pagès - 92150 Suresnes - France | +33 1 46 25 06 00<tel:%2B33%201%2046%2025%2006%2000> | www.talend.com<http://www.talend.com>
Global Leader in Open Source Integration
[http://www.talend.com/sites/default/files/logo-talend.jpg]<http://www.talend.com/>
--
Thanks,
Eugene
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
--
Thanks,
Eugene
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: WebHCat and MapReduce JAR
Posted by Eugene Koifman <ek...@hortonworks.com>.
can you try specify the path with "hdfs://...." otherwise it maybe looking
at local FS
On Wed, Aug 6, 2014 at 11:24 AM, Remy Dubois <rd...@talend.com> wrote:
> Hi again,
>
>
>
> And the issue I have is :
>
>
>
> 2014-08-06 22:06:37,083 FATAL [IPC Server handler 0 on 47378]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Task:
> attempt_1407355085865_0004_m_000000_0 - exited :
> java.lang.ClassNotFoundException: routines.StringHandling
>
> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
>
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>
> at mapreduce.MyMapper.map(MyMapper.java:14)
>
> at mapreduce.MyMapper.map(MyMapper.java:1)
>
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
>
> at
> org.apache.hadoop.mapreduce.lib.chain.Chain.runMapper(Chain.java:389)
>
> at
> org.apache.hadoop.mapreduce.lib.chain.ChainMapper.run(ChainMapper.java:149)
>
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
>
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
>
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at javax.security.auth.Subject.doAs(Subject.java:415)
>
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1557)
>
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
>
>
>
>
>
> *Remy Dubois*
> *Component team manager and Big Data principal *
> *Tel *+33 1 46 25 06 00 | *Mobile *+33 6 11 73 56 72 | *Email *
> rdubois@talend.com | *Skype *remy.dubois.talend
> *Talend* *SA *| 9, rue Pagès - 92150 Suresnes - France | +33 1 46 25 06 00
> | www.talend.com
> *Global Leader in Open Source Integration*
>
> [image: http://www.talend.com/sites/default/files/logo-talend.jpg]
> <http://www.talend.com/>
>
> *De :* Remy Dubois [mailto:rdubois@talend.com]
> *Envoyé :* mercredi 6 août 2014 20:19
> *À :* user@hive.apache.org
> *Objet :* RE: WebHCat and MapReduce JAR
>
>
>
> Hi,
>
>
>
> Thanks a lot for your answer. Following some information about what I do.
>
>
>
> *WebHCat version*: Hive 0.13 embedded in HDP 2.1 GA.
>
>
>
> The webhcat logs don’t show anything interesting.
>
>
>
> *REST call:*
>
>
>
>
>
> *Jar location on HDFS:*
>
> *[hdfs@hdp21 root]$ hadoop fs -ls /tmp/systemRoutines.jar*
>
> -rw-r--r-- 3 rdubois rdubois 264008 2014-08-06 19:16
> /tmp/systemRoutines.jar
>
> *[hdfs@hdp21 root]$ hadoop fs -get /tmp/systemRoutines.jar /home/hdfs*
>
> *[hdfs@hdp21 root]$ unzip -l /home/hdfs/systemRoutines.jar*
>
> Archive: /home/hdfs/systemRoutines.jar
>
> Length Date Time Name
>
> -------- ---- ---- ----
>
> 102 08-06-14 14:39 META-INF/MANIFEST.MF
>
> 3609 08-06-14 14:39 routines/StringHandling.class
>
>
>
> Thanks,
>
>
>
> *Remy Dubois*
> *Component team manager and Big Data principal *
> *Tel *+33 1 46 25 06 00 | *Mobile *+33 6 11 73 56 72 | *Email *
> rdubois@talend.com | *Skype *remy.dubois.talend
> *Talend* *SA *| 9, rue Pagès - 92150 Suresnes - France | +33 1 46 25 06 00
> | www.talend.com
> *Global Leader in Open Source Integration*
>
> [image: http://www.talend.com/sites/default/files/logo-talend.jpg]
> <http://www.talend.com/>
>
> *De :* Eugene Koifman [mailto:ekoifman@hortonworks.com
> <ek...@hortonworks.com>]
> *Envoyé :* mercredi 6 août 2014 19:43
> *À :* user@hive.apache.org
> *Objet :* Re: WebHCat and MapReduce JAR
>
>
>
> libjars as form param should work. Is there anything relevant in
> webhcat.log? What version of webhcat are you using? What is the exact
> REST call you are making?
>
>
>
> On Wed, Aug 6, 2014 at 10:17 AM, Remy Dubois <rd...@talend.com> wrote:
>
> Hi,
>
>
>
> I’m trying to submit a MapReduce job through webhcat using the
> templeton/v1/mapreduce/jar path.
>
> My job depends on third party libraries. I have tried a lot of things such
> as using the libjars FormParam (the jar is located on HDFS), but it doesn’t
> work, I got ClassNotFoundException.
>
>
>
> I’ve workaround my issue merging the job jar and the dependencies jars,
> but doing that is not really possible with my use case.
>
>
>
> Can you please let me know if what I’m trying to do is possible ? I am
> able to make the same job work, using the ToolRunner API (with the –libjars
> parameter) without any problem.
>
>
>
> Thanks a lot in advance,
>
>
>
> *Remy Dubois*
> *Component team manager and Big Data principal *
> *Tel *+33 1 46 25 06 00 | *Mobile *+33 6 11 73 56 72 | *Email *
> rdubois@talend.com | *Skype *remy.dubois.talend
> *Talend* *SA *| 9, rue Pagès - 92150 Suresnes - France | +33 1 46 25 06 00
> | www.talend.com
> *Global Leader in Open Source Integration*
>
> [image: http://www.talend.com/sites/default/files/logo-talend.jpg]
> <http://www.talend.com/>
>
>
>
>
>
> --
>
>
>
> Thanks,
>
> Eugene
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
--
Thanks,
Eugene
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
RE: WebHCat and MapReduce JAR
Posted by Remy Dubois <rd...@talend.com>.
Hi again,
And the issue I have is :
2014-08-06 22:06:37,083 FATAL [IPC Server handler 0 on 47378] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Task: attempt_1407355085865_0004_m_000000_0 - exited : java.lang.ClassNotFoundException: routines.StringHandling
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at mapreduce.MyMapper.map(MyMapper.java:14)
at mapreduce.MyMapper.map(MyMapper.java:1)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
at org.apache.hadoop.mapreduce.lib.chain.Chain.runMapper(Chain.java:389)
at org.apache.hadoop.mapreduce.lib.chain.ChainMapper.run(ChainMapper.java:149)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1557)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
Remy Dubois
Component team manager and Big Data principal
Tel +33 1 46 25 06 00 | Mobile +33 6 11 73 56 72 | Email rdubois@talend.com<ma...@talend.com> | Skype remy.dubois.talend
Talend SA | 9, rue Pagès - 92150 Suresnes - France | +33 1 46 25 06 00 | www.talend.com<http://www.talend.com>
Global Leader in Open Source Integration
[http://www.talend.com/sites/default/files/logo-talend.jpg]<http://www.talend.com/>
De : Remy Dubois [mailto:rdubois@talend.com]
Envoyé : mercredi 6 août 2014 20:19
À : user@hive.apache.org
Objet : RE: WebHCat and MapReduce JAR
Hi,
Thanks a lot for your answer. Following some information about what I do.
WebHCat version: Hive 0.13 embedded in HDP 2.1 GA.
The webhcat logs don’t show anything interesting.
REST call:
[cid:image002.jpg@01CFB1B4.8C0E22C0]
Jar location on HDFS:
[hdfs@hdp21 root]$ hadoop fs -ls /tmp/systemRoutines.jar
-rw-r--r-- 3 rdubois rdubois 264008 2014-08-06 19:16 /tmp/systemRoutines.jar
[hdfs@hdp21 root]$ hadoop fs -get /tmp/systemRoutines.jar /home/hdfs
[hdfs@hdp21 root]$ unzip -l /home/hdfs/systemRoutines.jar
Archive: /home/hdfs/systemRoutines.jar
Length Date Time Name
-------- ---- ---- ----
102 08-06-14 14:39 META-INF/MANIFEST.MF
3609 08-06-14 14:39 routines/StringHandling.class
Thanks,
Remy Dubois
Component team manager and Big Data principal
Tel +33 1 46 25 06 00 | Mobile +33 6 11 73 56 72 | Email rdubois@talend.com<ma...@talend.com> | Skype remy.dubois.talend
Talend SA | 9, rue Pagès - 92150 Suresnes - France | +33 1 46 25 06 00 | www.talend.com<http://www.talend.com>
Global Leader in Open Source Integration
[http://www.talend.com/sites/default/files/logo-talend.jpg]<http://www.talend.com/>
De : Eugene Koifman [mailto:ekoifman@hortonworks.com]
Envoyé : mercredi 6 août 2014 19:43
À : user@hive.apache.org<ma...@hive.apache.org>
Objet : Re: WebHCat and MapReduce JAR
libjars as form param should work. Is there anything relevant in webhcat.log? What version of webhcat are you using? What is the exact REST call you are making?
On Wed, Aug 6, 2014 at 10:17 AM, Remy Dubois <rd...@talend.com>> wrote:
Hi,
I’m trying to submit a MapReduce job through webhcat using the templeton/v1/mapreduce/jar path.
My job depends on third party libraries. I have tried a lot of things such as using the libjars FormParam (the jar is located on HDFS), but it doesn’t work, I got ClassNotFoundException.
I’ve workaround my issue merging the job jar and the dependencies jars, but doing that is not really possible with my use case.
Can you please let me know if what I’m trying to do is possible ? I am able to make the same job work, using the ToolRunner API (with the –libjars parameter) without any problem.
Thanks a lot in advance,
Remy Dubois
Component team manager and Big Data principal
Tel +33 1 46 25 06 00 | Mobile +33 6 11 73 56 72 | Email rdubois@talend.com<ma...@talend.com> | Skype remy.dubois.talend
Talend SA | 9, rue Pagès - 92150 Suresnes - France | +33 1 46 25 06 00<tel:%2B33%201%2046%2025%2006%2000> | www.talend.com<http://www.talend.com>
Global Leader in Open Source Integration
[http://www.talend.com/sites/default/files/logo-talend.jpg]<http://www.talend.com/>
--
Thanks,
Eugene
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
RE: WebHCat and MapReduce JAR
Posted by Remy Dubois <rd...@talend.com>.
Hi,
Thanks a lot for your answer. Following some information about what I do.
WebHCat version: Hive 0.13 embedded in HDP 2.1 GA.
The webhcat logs don’t show anything interesting.
REST call:
[cid:image005.jpg@01CFB1B3.A5A3C6F0]
Jar location on HDFS:
[hdfs@hdp21 root]$ hadoop fs -ls /tmp/systemRoutines.jar
-rw-r--r-- 3 rdubois rdubois 264008 2014-08-06 19:16 /tmp/systemRoutines.jar
[hdfs@hdp21 root]$ hadoop fs -get /tmp/systemRoutines.jar /home/hdfs
[hdfs@hdp21 root]$ unzip -l /home/hdfs/systemRoutines.jar
Archive: /home/hdfs/systemRoutines.jar
Length Date Time Name
-------- ---- ---- ----
102 08-06-14 14:39 META-INF/MANIFEST.MF
3609 08-06-14 14:39 routines/StringHandling.class
Thanks,
Remy Dubois
Component team manager and Big Data principal
Tel +33 1 46 25 06 00 | Mobile +33 6 11 73 56 72 | Email rdubois@talend.com<ma...@talend.com> | Skype remy.dubois.talend
Talend SA | 9, rue Pagès - 92150 Suresnes - France | +33 1 46 25 06 00 | www.talend.com<http://www.talend.com>
Global Leader in Open Source Integration
[http://www.talend.com/sites/default/files/logo-talend.jpg]<http://www.talend.com/>
De : Eugene Koifman [mailto:ekoifman@hortonworks.com]
Envoyé : mercredi 6 août 2014 19:43
À : user@hive.apache.org
Objet : Re: WebHCat and MapReduce JAR
libjars as form param should work. Is there anything relevant in webhcat.log? What version of webhcat are you using? What is the exact REST call you are making?
On Wed, Aug 6, 2014 at 10:17 AM, Remy Dubois <rd...@talend.com>> wrote:
Hi,
I’m trying to submit a MapReduce job through webhcat using the templeton/v1/mapreduce/jar path.
My job depends on third party libraries. I have tried a lot of things such as using the libjars FormParam (the jar is located on HDFS), but it doesn’t work, I got ClassNotFoundException.
I’ve workaround my issue merging the job jar and the dependencies jars, but doing that is not really possible with my use case.
Can you please let me know if what I’m trying to do is possible ? I am able to make the same job work, using the ToolRunner API (with the –libjars parameter) without any problem.
Thanks a lot in advance,
Remy Dubois
Component team manager and Big Data principal
Tel +33 1 46 25 06 00 | Mobile +33 6 11 73 56 72 | Email rdubois@talend.com<ma...@talend.com> | Skype remy.dubois.talend
Talend SA | 9, rue Pagès - 92150 Suresnes - France | +33 1 46 25 06 00<tel:%2B33%201%2046%2025%2006%2000> | www.talend.com<http://www.talend.com>
Global Leader in Open Source Integration
[http://www.talend.com/sites/default/files/logo-talend.jpg]<http://www.talend.com/>
--
Thanks,
Eugene
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: WebHCat and MapReduce JAR
Posted by Eugene Koifman <ek...@hortonworks.com>.
libjars as form param should work. Is there anything relevant in
webhcat.log? What version of webhcat are you using? What is the exact
REST call you are making?
On Wed, Aug 6, 2014 at 10:17 AM, Remy Dubois <rd...@talend.com> wrote:
> Hi,
>
>
>
> I’m trying to submit a MapReduce job through webhcat using the
> templeton/v1/mapreduce/jar path.
>
> My job depends on third party libraries. I have tried a lot of things such
> as using the libjars FormParam (the jar is located on HDFS), but it doesn’t
> work, I got ClassNotFoundException.
>
>
>
> I’ve workaround my issue merging the job jar and the dependencies jars,
> but doing that is not really possible with my use case.
>
>
>
> Can you please let me know if what I’m trying to do is possible ? I am
> able to make the same job work, using the ToolRunner API (with the –libjars
> parameter) without any problem.
>
>
>
> Thanks a lot in advance,
>
>
>
> *Remy Dubois*
> *Component team manager and Big Data principal *
> *Tel *+33 1 46 25 06 00 | *Mobile *+33 6 11 73 56 72 | *Email *
> rdubois@talend.com | *Skype *remy.dubois.talend
> *Talend* *SA *| 9, rue Pagès - 92150 Suresnes - France | +33 1 46 25 06 00
> | www.talend.com
> *Global Leader in Open Source Integration*
>
> [image: http://www.talend.com/sites/default/files/logo-talend.jpg]
> <http://www.talend.com/>
>
--
Thanks,
Eugene
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.