You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Farhan Husain <ru...@gmail.com> on 2009/04/01 19:57:08 UTC
Running MapReduce without setJar
Hello,
Can anyone tell me if there is any way running a map-reduce job from a java
program without specifying the jar file by JobConf.setJar() method?
Thanks,
--
Mohammad Farhan Husain
Research Assistant
Department of Computer Science
Erik Jonsson School of Engineering and Computer Science
University of Texas at Dallas
Re: Running MapReduce without setJar
Posted by Farhan Husain <ru...@gmail.com>.
Thanks Aaron for the explanation.
On Tue, Apr 7, 2009 at 1:51 PM, Aaron Kimball <aa...@cloudera.com> wrote:
> All the nodes in your Hadoop cluster need access to the class files for
> your
> MapReduce job. The current mechanism that Hadoop has to distribute classes
> and attach them to the classpath assumes they're in a JAR together. Thus,
> merely specifying the names of mapper/reducer classes with
> setMapperClass(),
> etc, isn't enough -- you need to actually deliver a jar containing those
> classes to all your nodes. Since the mapper and reducer classes are
> separate
> classes, you'd need to bundle those .class files together somehow. JAR is
> the standard way to do this, so that's what Hadoop supports.
>
> If you're running in fully-local mode (e.g., with
> jobConf.set("mapred.job.tracker", "local")), then no jar is needed since
> it's all running inside the original process space.
>
> - Aaron
>
>
> On Thu, Apr 2, 2009 at 1:13 PM, Farhan Husain <ru...@gmail.com> wrote:
>
> > I did all of them i.e. I used setMapClass, setReduceClass and new
> > JobConf(MapReduceWork.class) but still it cannot run the job without a
> jar
> > file. I understand the reason that it looks for those classes inside a
> jar
> > but I think there should be some better way to find those classes without
> > using a jar. But I am not sure whether it is possible at all.
> >
> > On Thu, Apr 2, 2009 at 2:56 PM, Rasit OZDAS <ra...@gmail.com>
> wrote:
> >
> > > You can point to them by using
> > > conf.setMapClass(..) and conf.setReduceClass(..) - or something
> > > similar, I don't have the source nearby.
> > >
> > > But something weird has happened to my code. It runs locally when I
> > > start it as java process (tries to find input path locally). I'm now
> > > using trunk, maybe something has changed with new version. With
> > > version 0.19 it was fine.
> > > Can somebody point out a clue?
> > >
> > > Rasit
> > >
> >
> >
> >
> > --
> > Mohammad Farhan Husain
> > Research Assistant
> > Department of Computer Science
> > Erik Jonsson School of Engineering and Computer Science
> > University of Texas at Dallas
> >
>
--
Mohammad Farhan Husain
Research Assistant
Department of Computer Science
Erik Jonsson School of Engineering and Computer Science
University of Texas at Dallas
Re: Running MapReduce without setJar
Posted by Aaron Kimball <aa...@cloudera.com>.
All the nodes in your Hadoop cluster need access to the class files for your
MapReduce job. The current mechanism that Hadoop has to distribute classes
and attach them to the classpath assumes they're in a JAR together. Thus,
merely specifying the names of mapper/reducer classes with setMapperClass(),
etc, isn't enough -- you need to actually deliver a jar containing those
classes to all your nodes. Since the mapper and reducer classes are separate
classes, you'd need to bundle those .class files together somehow. JAR is
the standard way to do this, so that's what Hadoop supports.
If you're running in fully-local mode (e.g., with
jobConf.set("mapred.job.tracker", "local")), then no jar is needed since
it's all running inside the original process space.
- Aaron
On Thu, Apr 2, 2009 at 1:13 PM, Farhan Husain <ru...@gmail.com> wrote:
> I did all of them i.e. I used setMapClass, setReduceClass and new
> JobConf(MapReduceWork.class) but still it cannot run the job without a jar
> file. I understand the reason that it looks for those classes inside a jar
> but I think there should be some better way to find those classes without
> using a jar. But I am not sure whether it is possible at all.
>
> On Thu, Apr 2, 2009 at 2:56 PM, Rasit OZDAS <ra...@gmail.com> wrote:
>
> > You can point to them by using
> > conf.setMapClass(..) and conf.setReduceClass(..) - or something
> > similar, I don't have the source nearby.
> >
> > But something weird has happened to my code. It runs locally when I
> > start it as java process (tries to find input path locally). I'm now
> > using trunk, maybe something has changed with new version. With
> > version 0.19 it was fine.
> > Can somebody point out a clue?
> >
> > Rasit
> >
>
>
>
> --
> Mohammad Farhan Husain
> Research Assistant
> Department of Computer Science
> Erik Jonsson School of Engineering and Computer Science
> University of Texas at Dallas
>
Re: Running MapReduce without setJar
Posted by Farhan Husain <ru...@gmail.com>.
I did all of them i.e. I used setMapClass, setReduceClass and new
JobConf(MapReduceWork.class) but still it cannot run the job without a jar
file. I understand the reason that it looks for those classes inside a jar
but I think there should be some better way to find those classes without
using a jar. But I am not sure whether it is possible at all.
On Thu, Apr 2, 2009 at 2:56 PM, Rasit OZDAS <ra...@gmail.com> wrote:
> You can point to them by using
> conf.setMapClass(..) and conf.setReduceClass(..) - or something
> similar, I don't have the source nearby.
>
> But something weird has happened to my code. It runs locally when I
> start it as java process (tries to find input path locally). I'm now
> using trunk, maybe something has changed with new version. With
> version 0.19 it was fine.
> Can somebody point out a clue?
>
> Rasit
>
--
Mohammad Farhan Husain
Research Assistant
Department of Computer Science
Erik Jonsson School of Engineering and Computer Science
University of Texas at Dallas
Re: Running MapReduce without setJar
Posted by Rasit OZDAS <ra...@gmail.com>.
You can point to them by using
conf.setMapClass(..) and conf.setReduceClass(..) - or something
similar, I don't have the source nearby.
But something weird has happened to my code. It runs locally when I
start it as java process (tries to find input path locally). I'm now
using trunk, maybe something has changed with new version. With
version 0.19 it was fine.
Can somebody point out a clue?
Rasit
Re: Running MapReduce without setJar
Posted by Farhan Husain <ru...@gmail.com>.
Does this class need to have the mapper and reducer classes too?
On Wed, Apr 1, 2009 at 1:52 PM, javateck javateck <ja...@gmail.com>wrote:
> you can run from java program:
>
> JobConf conf = new JobConf(MapReduceWork.class);
>
> // setting your params
>
> JobClient.runJob(conf);
>
>
> On Wed, Apr 1, 2009 at 11:42 AM, Farhan Husain <ru...@gmail.com> wrote:
>
> > Can I get rid of the whole jar thing? Is there any way to run map reduce
> > programs without using a jar? I do not want to use "hadoop jar ..."
> either.
> >
> > On Wed, Apr 1, 2009 at 1:10 PM, javateck javateck <javateck@gmail.com
> > >wrote:
> >
> > > I think you need to set a property (mapred.jar) inside hadoop-site.xml,
> > > then
> > > you don't need to hardcode in your java code, and it will be fine.
> > > But I don't know if there is any way that we can set multiple jars,
> since
> > a
> > > lot of times our own mapreduce class needs to reference other jars.
> > >
> > > On Wed, Apr 1, 2009 at 10:57 AM, Farhan Husain <ru...@gmail.com>
> > wrote:
> > >
> > > > Hello,
> > > >
> > > > Can anyone tell me if there is any way running a map-reduce job from
> a
> > > java
> > > > program without specifying the jar file by JobConf.setJar() method?
> > > >
> > > > Thanks,
> > > >
> > > > --
> > > > Mohammad Farhan Husain
> > > > Research Assistant
> > > > Department of Computer Science
> > > > Erik Jonsson School of Engineering and Computer Science
> > > > University of Texas at Dallas
> > > >
> > >
> >
> >
> >
> > --
> > Mohammad Farhan Husain
> > Research Assistant
> > Department of Computer Science
> > Erik Jonsson School of Engineering and Computer Science
> > University of Texas at Dallas
> >
>
--
Mohammad Farhan Husain
Research Assistant
Department of Computer Science
Erik Jonsson School of Engineering and Computer Science
University of Texas at Dallas
Re: Running MapReduce without setJar
Posted by Rasit OZDAS <ra...@gmail.com>.
Yes, as an additional info,
you can use this code just to start the job, not wait until it's finished:
JobClient client = new JobClient(conf);
client.runJob(conf);
2009/4/1 javateck javateck <ja...@gmail.com>
> you can run from java program:
>
> JobConf conf = new JobConf(MapReduceWork.class);
>
> // setting your params
>
> JobClient.runJob(conf);
>
>
Re: Running MapReduce without setJar
Posted by javateck javateck <ja...@gmail.com>.
you can run from java program:
JobConf conf = new JobConf(MapReduceWork.class);
// setting your params
JobClient.runJob(conf);
On Wed, Apr 1, 2009 at 11:42 AM, Farhan Husain <ru...@gmail.com> wrote:
> Can I get rid of the whole jar thing? Is there any way to run map reduce
> programs without using a jar? I do not want to use "hadoop jar ..." either.
>
> On Wed, Apr 1, 2009 at 1:10 PM, javateck javateck <javateck@gmail.com
> >wrote:
>
> > I think you need to set a property (mapred.jar) inside hadoop-site.xml,
> > then
> > you don't need to hardcode in your java code, and it will be fine.
> > But I don't know if there is any way that we can set multiple jars, since
> a
> > lot of times our own mapreduce class needs to reference other jars.
> >
> > On Wed, Apr 1, 2009 at 10:57 AM, Farhan Husain <ru...@gmail.com>
> wrote:
> >
> > > Hello,
> > >
> > > Can anyone tell me if there is any way running a map-reduce job from a
> > java
> > > program without specifying the jar file by JobConf.setJar() method?
> > >
> > > Thanks,
> > >
> > > --
> > > Mohammad Farhan Husain
> > > Research Assistant
> > > Department of Computer Science
> > > Erik Jonsson School of Engineering and Computer Science
> > > University of Texas at Dallas
> > >
> >
>
>
>
> --
> Mohammad Farhan Husain
> Research Assistant
> Department of Computer Science
> Erik Jonsson School of Engineering and Computer Science
> University of Texas at Dallas
>
Re: Running MapReduce without setJar
Posted by Farhan Husain <ru...@gmail.com>.
Can I get rid of the whole jar thing? Is there any way to run map reduce
programs without using a jar? I do not want to use "hadoop jar ..." either.
On Wed, Apr 1, 2009 at 1:10 PM, javateck javateck <ja...@gmail.com>wrote:
> I think you need to set a property (mapred.jar) inside hadoop-site.xml,
> then
> you don't need to hardcode in your java code, and it will be fine.
> But I don't know if there is any way that we can set multiple jars, since a
> lot of times our own mapreduce class needs to reference other jars.
>
> On Wed, Apr 1, 2009 at 10:57 AM, Farhan Husain <ru...@gmail.com> wrote:
>
> > Hello,
> >
> > Can anyone tell me if there is any way running a map-reduce job from a
> java
> > program without specifying the jar file by JobConf.setJar() method?
> >
> > Thanks,
> >
> > --
> > Mohammad Farhan Husain
> > Research Assistant
> > Department of Computer Science
> > Erik Jonsson School of Engineering and Computer Science
> > University of Texas at Dallas
> >
>
--
Mohammad Farhan Husain
Research Assistant
Department of Computer Science
Erik Jonsson School of Engineering and Computer Science
University of Texas at Dallas
Re: Running MapReduce without setJar
Posted by javateck javateck <ja...@gmail.com>.
I think you need to set a property (mapred.jar) inside hadoop-site.xml, then
you don't need to hardcode in your java code, and it will be fine.
But I don't know if there is any way that we can set multiple jars, since a
lot of times our own mapreduce class needs to reference other jars.
On Wed, Apr 1, 2009 at 10:57 AM, Farhan Husain <ru...@gmail.com> wrote:
> Hello,
>
> Can anyone tell me if there is any way running a map-reduce job from a java
> program without specifying the jar file by JobConf.setJar() method?
>
> Thanks,
>
> --
> Mohammad Farhan Husain
> Research Assistant
> Department of Computer Science
> Erik Jonsson School of Engineering and Computer Science
> University of Texas at Dallas
>