You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Zhengguo 'Mike' SUN <zh...@yahoo.com> on 2009/11/24 05:28:58 UTC
Job.setJarByClass in Hadoop 0.20.1
Hi,
After porting my code from Hadoop 0.17 to 0.20, I am starting to have problems setting my jar file. I used to be able to set jar file by using JobConf.setJar(). But now I am using Job.setJarByClass(). It looks to me that this method is not working. I kept getting ClassNotFoundException when submitting my job. I also added Job.getJar() after Job.setJarByClass, but it did return null somehow. By the way, I am using CloudEra's distribution of hadoop-0.20.1+133.
Anyone has the same problem?
Re: Job.setJarByClass in Hadoop 0.20.1
Posted by Todd Lipcon <to...@cloudera.com>.
Hi Mike,
It sounds like you're doing something weird when you create your jar.
What platform are you submitting from, and how are you making the jar?
-Todd
On Mon, Nov 23, 2009 at 9:09 PM, Zhengguo 'Mike' SUN
<zh...@yahoo.com>wrote:
> Hi Todd,
>
> The hadoop-0.20.1+133-examples.jar worked fine. I just tried an Apache
> release. I copied the example source file of WordCount.java to my jar and
> submitted the job. It worked with Apache's release, but failed with
> CloudEra's release. But my code failed for both of the versions.
>
>
>
>
> ________________________________
> From: Todd Lipcon <to...@cloudera.com>
> To: common-user@hadoop.apache.org
> Sent: Mon, November 23, 2009 11:34:22 PM
> Subject: Re: Job.setJarByClass in Hadoop 0.20.1
>
> Hi Mike,
>
> I haven't seen that problem. There is one patch in the Cloudera
> distribution
> that does modify the behavior of that method, though. Would you mind trying
> this on the stock Apache 0.20.1 release? I see no reason to believe this is
> the issue, since hundreds of other people are using our distro without
> issues, but it's worth checking out.
>
> Is this happening with every jar? What platform are you running on?
>
> Thanks,
> -Todd
>
> On Mon, Nov 23, 2009 at 8:28 PM, Zhengguo 'Mike' SUN
> <zh...@yahoo.com>wrote:
>
> > Hi,
> >
> > After porting my code from Hadoop 0.17 to 0.20, I am starting to have
> > problems setting my jar file. I used to be able to set jar file by using
> > JobConf.setJar(). But now I am using Job.setJarByClass(). It looks to me
> > that this method is not working. I kept getting ClassNotFoundException
> when
> > submitting my job. I also added Job.getJar() after Job.setJarByClass, but
> it
> > did return null somehow. By the way, I am using CloudEra's distribution
> of
> > hadoop-0.20.1+133.
> >
> > Anyone has the same problem?
> >
> >
> >
> >
>
>
>
>
>
Re: Job.setJarByClass in Hadoop 0.20.1
Posted by Zhengguo 'Mike' SUN <zh...@yahoo.com>.
Hi Todd,
The hadoop-0.20.1+133-examples.jar worked fine. I just tried an Apache release. I copied the example source file of WordCount.java to my jar and submitted the job. It worked with Apache's release, but failed with CloudEra's release. But my code failed for both of the versions.
________________________________
From: Todd Lipcon <to...@cloudera.com>
To: common-user@hadoop.apache.org
Sent: Mon, November 23, 2009 11:34:22 PM
Subject: Re: Job.setJarByClass in Hadoop 0.20.1
Hi Mike,
I haven't seen that problem. There is one patch in the Cloudera distribution
that does modify the behavior of that method, though. Would you mind trying
this on the stock Apache 0.20.1 release? I see no reason to believe this is
the issue, since hundreds of other people are using our distro without
issues, but it's worth checking out.
Is this happening with every jar? What platform are you running on?
Thanks,
-Todd
On Mon, Nov 23, 2009 at 8:28 PM, Zhengguo 'Mike' SUN
<zh...@yahoo.com>wrote:
> Hi,
>
> After porting my code from Hadoop 0.17 to 0.20, I am starting to have
> problems setting my jar file. I used to be able to set jar file by using
> JobConf.setJar(). But now I am using Job.setJarByClass(). It looks to me
> that this method is not working. I kept getting ClassNotFoundException when
> submitting my job. I also added Job.getJar() after Job.setJarByClass, but it
> did return null somehow. By the way, I am using CloudEra's distribution of
> hadoop-0.20.1+133.
>
> Anyone has the same problem?
>
>
>
>
Re: Job.setJarByClass in Hadoop 0.20.1
Posted by Amandeep Khurana <am...@gmail.com>.
What are you putting as the argument in job.setJarByClass( ?? )
On Mon, Nov 23, 2009 at 8:34 PM, Todd Lipcon <to...@cloudera.com> wrote:
> Hi Mike,
>
> I haven't seen that problem. There is one patch in the Cloudera
> distribution
> that does modify the behavior of that method, though. Would you mind trying
> this on the stock Apache 0.20.1 release? I see no reason to believe this is
> the issue, since hundreds of other people are using our distro without
> issues, but it's worth checking out.
>
> Is this happening with every jar? What platform are you running on?
>
> Thanks,
> -Todd
>
> On Mon, Nov 23, 2009 at 8:28 PM, Zhengguo 'Mike' SUN
> <zh...@yahoo.com>wrote:
>
> > Hi,
> >
> > After porting my code from Hadoop 0.17 to 0.20, I am starting to have
> > problems setting my jar file. I used to be able to set jar file by using
> > JobConf.setJar(). But now I am using Job.setJarByClass(). It looks to me
> > that this method is not working. I kept getting ClassNotFoundException
> when
> > submitting my job. I also added Job.getJar() after Job.setJarByClass, but
> it
> > did return null somehow. By the way, I am using CloudEra's distribution
> of
> > hadoop-0.20.1+133.
> >
> > Anyone has the same problem?
> >
> >
> >
> >
>
Re: Job.setJarByClass in Hadoop 0.20.1
Posted by Todd Lipcon <to...@cloudera.com>.
Hi Mike,
I haven't seen that problem. There is one patch in the Cloudera distribution
that does modify the behavior of that method, though. Would you mind trying
this on the stock Apache 0.20.1 release? I see no reason to believe this is
the issue, since hundreds of other people are using our distro without
issues, but it's worth checking out.
Is this happening with every jar? What platform are you running on?
Thanks,
-Todd
On Mon, Nov 23, 2009 at 8:28 PM, Zhengguo 'Mike' SUN
<zh...@yahoo.com>wrote:
> Hi,
>
> After porting my code from Hadoop 0.17 to 0.20, I am starting to have
> problems setting my jar file. I used to be able to set jar file by using
> JobConf.setJar(). But now I am using Job.setJarByClass(). It looks to me
> that this method is not working. I kept getting ClassNotFoundException when
> submitting my job. I also added Job.getJar() after Job.setJarByClass, but it
> did return null somehow. By the way, I am using CloudEra's distribution of
> hadoop-0.20.1+133.
>
> Anyone has the same problem?
>
>
>
>
Re: Job.setJarByClass in Hadoop 0.20.1
Posted by Todd Lipcon <to...@cloudera.com>.
My guess is that you have set your CLASSPATH variable such that when you ran
your jar, it was actually finding the classes from the original .class
files, and not from the jar. When you moved it out, it was just the jar on
the classpath as it should be, and this solved the issue.
-Todd
On Mon, Nov 23, 2009 at 9:17 PM, Zhengguo 'Mike' SUN
<zh...@yahoo.com>wrote:
> Hi All,
>
> I figured out. It is related to my directory structure. I have my jar under
> $HADOOP_HOME and my package and class files are under $HADOOP_HOME too. When
> I moved my package and class files somewhere else, it worked. It seemed that
> this structure made JobConf.findContainingJar break.
>
>
>
>
> ________________________________
> From: Rekha Joshi <re...@yahoo-inc.com>
> To: "common-user@hadoop.apache.org" <co...@hadoop.apache.org>; "
> core-user@hadoop.apache.org" <co...@hadoop.apache.org>
> Sent: Mon, November 23, 2009 11:51:12 PM
> Subject: Re: Job.setJarByClass in Hadoop 0.20.1
>
> If your paths/files are alright, think
> job.setJarByClass(yourClassName.class); should work.Thanks!
>
>
> On 11/24/09 9:58 AM, "Zhengguo 'Mike' SUN" <zh...@yahoo.com> wrote:
>
> Job.setJarByClass()
>
>
>
>
>
Re: Job.setJarByClass in Hadoop 0.20.1
Posted by Zhengguo 'Mike' SUN <zh...@yahoo.com>.
Hi All,
I figured out. It is related to my directory structure. I have my jar under $HADOOP_HOME and my package and class files are under $HADOOP_HOME too. When I moved my package and class files somewhere else, it worked. It seemed that this structure made JobConf.findContainingJar break.
________________________________
From: Rekha Joshi <re...@yahoo-inc.com>
To: "common-user@hadoop.apache.org" <co...@hadoop.apache.org>; "core-user@hadoop.apache.org" <co...@hadoop.apache.org>
Sent: Mon, November 23, 2009 11:51:12 PM
Subject: Re: Job.setJarByClass in Hadoop 0.20.1
If your paths/files are alright, think job.setJarByClass(yourClassName.class); should work.Thanks!
On 11/24/09 9:58 AM, "Zhengguo 'Mike' SUN" <zh...@yahoo.com> wrote:
Job.setJarByClass()
Re: Job.setJarByClass in Hadoop 0.20.1
Posted by Rekha Joshi <re...@yahoo-inc.com>.
If your paths/files are alright, think job.setJarByClass(yourClassName.class); should work.Thanks!
On 11/24/09 9:58 AM, "Zhengguo 'Mike' SUN" <zh...@yahoo.com> wrote:
Job.setJarByClass()