You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Radhe Radhe <ra...@live.com> on 2014/03/06 20:36:09 UTC
Running a Job in a Local Job Runner:Windows 7 64-bit
Hi All,
I'm trying to get some hands-on on the Map Reduce programming.
I downloaded the code examples from Hadoop-The definitive guide, 3rd edition and build it using Maven:
mvn package -DskipTests -Dhadoop.distro=apache-2
Next I imported the maven projects into Eclipse. Using Eclipse now I can develop my own Map Reduce jobs.
Bu how do I test\run the job locally using the "Local Job Runner"?
The book excerpt says:
Now we can run this application against some local files. Hadoop comes with a local
job runner, a cut-down version of the MapReduce execution engine for running Map-
Reduce jobs in a single JVM. Itβs designed for testing, and is very convenient for use in
an IDE, since you can run it in a debugger to step through the code in your mapper and
reducer.
Do I also need to install Hadoop locally on Windows for that?
Thanks,
-RR
Re: Running a Job in a Local Job Runner:Windows 7 64-bit
Posted by Rakesh Davanum <ra...@gmail.com>.
Hi RR,
You don't need to have the actual Hadoop daemons running on windows
machince. Just install Cygwin and ensure that you have all the required
Hadoop jars in the class path of your program. You can test/debug directly
from the IDE itself just by saying "Run As" -> "Java Application" on the
driver class. This will run the program in the Local Job Runner mode. You
can use this to verify the basic logic of your MR code. We you start using
advanced features of MR, the behavior/output on Local Job Runner may be
different from when running the program on a real distributed cluster.
Regards,
Rakesh
On Thu, Mar 6, 2014 at 2:36 PM, Radhe Radhe <ra...@live.com>wrote:
> Hi All,
>
> I'm trying to get some hands-on on the Map Reduce programming.
>
> I downloaded the code examples from Hadoop-The definitive guide, 3rd
> edition and build it using Maven:
> *mvn package -DskipTests -Dhadoop.distro=apache-2*
>
> Next I imported the maven projects into Eclipse. Using Eclipse now I can
> develop my own Map Reduce jobs.
> Bu how do I test\run the job locally using the "*Local Job Runner*"?
>
> The book excerpt says:
>
>
>
>
>
> *Now we can run this application against some local files. Hadoop comes
> with a localjob runner, a cut-down version of the MapReduce execution
> engine for running Map-Reduce jobs in a single JVM. It's designed for
> testing, and is very convenient for use inan IDE, since you can run it in a
> debugger to step through the code in your mapper andreducer.*
> Do I also need to install Hadoop locally on Windows for that?
>
> Thanks,
> -RR
>
Re: Running a Job in a Local Job Runner:Windows 7 64-bit
Posted by Rakesh Davanum <ra...@gmail.com>.
Hi RR,
You don't need to have the actual Hadoop daemons running on windows
machince. Just install Cygwin and ensure that you have all the required
Hadoop jars in the class path of your program. You can test/debug directly
from the IDE itself just by saying "Run As" -> "Java Application" on the
driver class. This will run the program in the Local Job Runner mode. You
can use this to verify the basic logic of your MR code. We you start using
advanced features of MR, the behavior/output on Local Job Runner may be
different from when running the program on a real distributed cluster.
Regards,
Rakesh
On Thu, Mar 6, 2014 at 2:36 PM, Radhe Radhe <ra...@live.com>wrote:
> Hi All,
>
> I'm trying to get some hands-on on the Map Reduce programming.
>
> I downloaded the code examples from Hadoop-The definitive guide, 3rd
> edition and build it using Maven:
> *mvn package -DskipTests -Dhadoop.distro=apache-2*
>
> Next I imported the maven projects into Eclipse. Using Eclipse now I can
> develop my own Map Reduce jobs.
> Bu how do I test\run the job locally using the "*Local Job Runner*"?
>
> The book excerpt says:
>
>
>
>
>
> *Now we can run this application against some local files. Hadoop comes
> with a localjob runner, a cut-down version of the MapReduce execution
> engine for running Map-Reduce jobs in a single JVM. It's designed for
> testing, and is very convenient for use inan IDE, since you can run it in a
> debugger to step through the code in your mapper andreducer.*
> Do I also need to install Hadoop locally on Windows for that?
>
> Thanks,
> -RR
>
RE: Running a Job in a Local Job Runner:Windows 7 64-bit
Posted by Radhe Radhe <ra...@live.com>.
Thanks a lot Harsh,
I just ran the Driver class from Eclipse and it worked fine.π
Can I use local job runner to test Sequence File and Map File as well ?π
-RR
> From: harsh@cloudera.com
> Date: Thu, 6 Mar 2014 19:37:18 -0800
> Subject: Re: Running a Job in a Local Job Runner:Windows 7 64-bit
> To: user@hadoop.apache.org
>
> Running your Driver class from Eclipse should automatically run it in
> the local runner mode (as thats the default mode).
>
> You shouldn't need a local Hadoop install for this.
>
> On Thu, Mar 6, 2014 at 11:36 AM, Radhe Radhe
> <ra...@live.com> wrote:
> > Hi All,
> >
> > I'm trying to get some hands-on on the Map Reduce programming.
> >
> > I downloaded the code examples from Hadoop-The definitive guide, 3rd edition
> > and build it using Maven:
> > mvn package -DskipTests -Dhadoop.distro=apache-2
> >
> > Next I imported the maven projects into Eclipse. Using Eclipse now I can
> > develop my own Map Reduce jobs.
> > Bu how do I test\run the job locally using the "Local Job Runner"?
> >
> > The book excerpt says:
> > Now we can run this application against some local files. Hadoop comes with
> > a local
> > job runner, a cut-down version of the MapReduce execution engine for running
> > Map-
> > Reduce jobs in a single JVM. It's designed for testing, and is very
> > convenient for use in
> > an IDE, since you can run it in a debugger to step through the code in your
> > mapper and
> > reducer.
> >
> > Do I also need to install Hadoop locally on Windows for that?
> >
> > Thanks,
> > -RR
>
>
>
> --
> Harsh J
RE: Running a Job in a Local Job Runner:Windows 7 64-bit
Posted by Radhe Radhe <ra...@live.com>.
Thanks a lot Harsh,
I just ran the Driver class from Eclipse and it worked fine.π
Can I use local job runner to test Sequence File and Map File as well ?π
-RR
> From: harsh@cloudera.com
> Date: Thu, 6 Mar 2014 19:37:18 -0800
> Subject: Re: Running a Job in a Local Job Runner:Windows 7 64-bit
> To: user@hadoop.apache.org
>
> Running your Driver class from Eclipse should automatically run it in
> the local runner mode (as thats the default mode).
>
> You shouldn't need a local Hadoop install for this.
>
> On Thu, Mar 6, 2014 at 11:36 AM, Radhe Radhe
> <ra...@live.com> wrote:
> > Hi All,
> >
> > I'm trying to get some hands-on on the Map Reduce programming.
> >
> > I downloaded the code examples from Hadoop-The definitive guide, 3rd edition
> > and build it using Maven:
> > mvn package -DskipTests -Dhadoop.distro=apache-2
> >
> > Next I imported the maven projects into Eclipse. Using Eclipse now I can
> > develop my own Map Reduce jobs.
> > Bu how do I test\run the job locally using the "Local Job Runner"?
> >
> > The book excerpt says:
> > Now we can run this application against some local files. Hadoop comes with
> > a local
> > job runner, a cut-down version of the MapReduce execution engine for running
> > Map-
> > Reduce jobs in a single JVM. It's designed for testing, and is very
> > convenient for use in
> > an IDE, since you can run it in a debugger to step through the code in your
> > mapper and
> > reducer.
> >
> > Do I also need to install Hadoop locally on Windows for that?
> >
> > Thanks,
> > -RR
>
>
>
> --
> Harsh J
RE: Running a Job in a Local Job Runner:Windows 7 64-bit
Posted by Radhe Radhe <ra...@live.com>.
Thanks a lot Harsh,
I just ran the Driver class from Eclipse and it worked fine.π
Can I use local job runner to test Sequence File and Map File as well ?π
-RR
> From: harsh@cloudera.com
> Date: Thu, 6 Mar 2014 19:37:18 -0800
> Subject: Re: Running a Job in a Local Job Runner:Windows 7 64-bit
> To: user@hadoop.apache.org
>
> Running your Driver class from Eclipse should automatically run it in
> the local runner mode (as thats the default mode).
>
> You shouldn't need a local Hadoop install for this.
>
> On Thu, Mar 6, 2014 at 11:36 AM, Radhe Radhe
> <ra...@live.com> wrote:
> > Hi All,
> >
> > I'm trying to get some hands-on on the Map Reduce programming.
> >
> > I downloaded the code examples from Hadoop-The definitive guide, 3rd edition
> > and build it using Maven:
> > mvn package -DskipTests -Dhadoop.distro=apache-2
> >
> > Next I imported the maven projects into Eclipse. Using Eclipse now I can
> > develop my own Map Reduce jobs.
> > Bu how do I test\run the job locally using the "Local Job Runner"?
> >
> > The book excerpt says:
> > Now we can run this application against some local files. Hadoop comes with
> > a local
> > job runner, a cut-down version of the MapReduce execution engine for running
> > Map-
> > Reduce jobs in a single JVM. It's designed for testing, and is very
> > convenient for use in
> > an IDE, since you can run it in a debugger to step through the code in your
> > mapper and
> > reducer.
> >
> > Do I also need to install Hadoop locally on Windows for that?
> >
> > Thanks,
> > -RR
>
>
>
> --
> Harsh J
RE: Running a Job in a Local Job Runner:Windows 7 64-bit
Posted by Radhe Radhe <ra...@live.com>.
Thanks a lot Harsh,
I just ran the Driver class from Eclipse and it worked fine.π
Can I use local job runner to test Sequence File and Map File as well ?π
-RR
> From: harsh@cloudera.com
> Date: Thu, 6 Mar 2014 19:37:18 -0800
> Subject: Re: Running a Job in a Local Job Runner:Windows 7 64-bit
> To: user@hadoop.apache.org
>
> Running your Driver class from Eclipse should automatically run it in
> the local runner mode (as thats the default mode).
>
> You shouldn't need a local Hadoop install for this.
>
> On Thu, Mar 6, 2014 at 11:36 AM, Radhe Radhe
> <ra...@live.com> wrote:
> > Hi All,
> >
> > I'm trying to get some hands-on on the Map Reduce programming.
> >
> > I downloaded the code examples from Hadoop-The definitive guide, 3rd edition
> > and build it using Maven:
> > mvn package -DskipTests -Dhadoop.distro=apache-2
> >
> > Next I imported the maven projects into Eclipse. Using Eclipse now I can
> > develop my own Map Reduce jobs.
> > Bu how do I test\run the job locally using the "Local Job Runner"?
> >
> > The book excerpt says:
> > Now we can run this application against some local files. Hadoop comes with
> > a local
> > job runner, a cut-down version of the MapReduce execution engine for running
> > Map-
> > Reduce jobs in a single JVM. It's designed for testing, and is very
> > convenient for use in
> > an IDE, since you can run it in a debugger to step through the code in your
> > mapper and
> > reducer.
> >
> > Do I also need to install Hadoop locally on Windows for that?
> >
> > Thanks,
> > -RR
>
>
>
> --
> Harsh J
Re: Running a Job in a Local Job Runner:Windows 7 64-bit
Posted by Harsh J <ha...@cloudera.com>.
Running your Driver class from Eclipse should automatically run it in
the local runner mode (as thats the default mode).
You shouldn't need a local Hadoop install for this.
On Thu, Mar 6, 2014 at 11:36 AM, Radhe Radhe
<ra...@live.com> wrote:
> Hi All,
>
> I'm trying to get some hands-on on the Map Reduce programming.
>
> I downloaded the code examples from Hadoop-The definitive guide, 3rd edition
> and build it using Maven:
> mvn package -DskipTests -Dhadoop.distro=apache-2
>
> Next I imported the maven projects into Eclipse. Using Eclipse now I can
> develop my own Map Reduce jobs.
> Bu how do I test\run the job locally using the "Local Job Runner"?
>
> The book excerpt says:
> Now we can run this application against some local files. Hadoop comes with
> a local
> job runner, a cut-down version of the MapReduce execution engine for running
> Map-
> Reduce jobs in a single JVM. It's designed for testing, and is very
> convenient for use in
> an IDE, since you can run it in a debugger to step through the code in your
> mapper and
> reducer.
>
> Do I also need to install Hadoop locally on Windows for that?
>
> Thanks,
> -RR
--
Harsh J
Re: Running a Job in a Local Job Runner:Windows 7 64-bit
Posted by Rakesh Davanum <ra...@gmail.com>.
Hi RR,
You don't need to have the actual Hadoop daemons running on windows
machince. Just install Cygwin and ensure that you have all the required
Hadoop jars in the class path of your program. You can test/debug directly
from the IDE itself just by saying "Run As" -> "Java Application" on the
driver class. This will run the program in the Local Job Runner mode. You
can use this to verify the basic logic of your MR code. We you start using
advanced features of MR, the behavior/output on Local Job Runner may be
different from when running the program on a real distributed cluster.
Regards,
Rakesh
On Thu, Mar 6, 2014 at 2:36 PM, Radhe Radhe <ra...@live.com>wrote:
> Hi All,
>
> I'm trying to get some hands-on on the Map Reduce programming.
>
> I downloaded the code examples from Hadoop-The definitive guide, 3rd
> edition and build it using Maven:
> *mvn package -DskipTests -Dhadoop.distro=apache-2*
>
> Next I imported the maven projects into Eclipse. Using Eclipse now I can
> develop my own Map Reduce jobs.
> Bu how do I test\run the job locally using the "*Local Job Runner*"?
>
> The book excerpt says:
>
>
>
>
>
> *Now we can run this application against some local files. Hadoop comes
> with a localjob runner, a cut-down version of the MapReduce execution
> engine for running Map-Reduce jobs in a single JVM. It's designed for
> testing, and is very convenient for use inan IDE, since you can run it in a
> debugger to step through the code in your mapper andreducer.*
> Do I also need to install Hadoop locally on Windows for that?
>
> Thanks,
> -RR
>
Re: Running a Job in a Local Job Runner:Windows 7 64-bit
Posted by Harsh J <ha...@cloudera.com>.
Running your Driver class from Eclipse should automatically run it in
the local runner mode (as thats the default mode).
You shouldn't need a local Hadoop install for this.
On Thu, Mar 6, 2014 at 11:36 AM, Radhe Radhe
<ra...@live.com> wrote:
> Hi All,
>
> I'm trying to get some hands-on on the Map Reduce programming.
>
> I downloaded the code examples from Hadoop-The definitive guide, 3rd edition
> and build it using Maven:
> mvn package -DskipTests -Dhadoop.distro=apache-2
>
> Next I imported the maven projects into Eclipse. Using Eclipse now I can
> develop my own Map Reduce jobs.
> Bu how do I test\run the job locally using the "Local Job Runner"?
>
> The book excerpt says:
> Now we can run this application against some local files. Hadoop comes with
> a local
> job runner, a cut-down version of the MapReduce execution engine for running
> Map-
> Reduce jobs in a single JVM. It's designed for testing, and is very
> convenient for use in
> an IDE, since you can run it in a debugger to step through the code in your
> mapper and
> reducer.
>
> Do I also need to install Hadoop locally on Windows for that?
>
> Thanks,
> -RR
--
Harsh J
Re: Running a Job in a Local Job Runner:Windows 7 64-bit
Posted by Rakesh Davanum <ra...@gmail.com>.
Hi RR,
You don't need to have the actual Hadoop daemons running on windows
machince. Just install Cygwin and ensure that you have all the required
Hadoop jars in the class path of your program. You can test/debug directly
from the IDE itself just by saying "Run As" -> "Java Application" on the
driver class. This will run the program in the Local Job Runner mode. You
can use this to verify the basic logic of your MR code. We you start using
advanced features of MR, the behavior/output on Local Job Runner may be
different from when running the program on a real distributed cluster.
Regards,
Rakesh
On Thu, Mar 6, 2014 at 2:36 PM, Radhe Radhe <ra...@live.com>wrote:
> Hi All,
>
> I'm trying to get some hands-on on the Map Reduce programming.
>
> I downloaded the code examples from Hadoop-The definitive guide, 3rd
> edition and build it using Maven:
> *mvn package -DskipTests -Dhadoop.distro=apache-2*
>
> Next I imported the maven projects into Eclipse. Using Eclipse now I can
> develop my own Map Reduce jobs.
> Bu how do I test\run the job locally using the "*Local Job Runner*"?
>
> The book excerpt says:
>
>
>
>
>
> *Now we can run this application against some local files. Hadoop comes
> with a localjob runner, a cut-down version of the MapReduce execution
> engine for running Map-Reduce jobs in a single JVM. It's designed for
> testing, and is very convenient for use inan IDE, since you can run it in a
> debugger to step through the code in your mapper andreducer.*
> Do I also need to install Hadoop locally on Windows for that?
>
> Thanks,
> -RR
>
Re: Running a Job in a Local Job Runner:Windows 7 64-bit
Posted by Harsh J <ha...@cloudera.com>.
Running your Driver class from Eclipse should automatically run it in
the local runner mode (as thats the default mode).
You shouldn't need a local Hadoop install for this.
On Thu, Mar 6, 2014 at 11:36 AM, Radhe Radhe
<ra...@live.com> wrote:
> Hi All,
>
> I'm trying to get some hands-on on the Map Reduce programming.
>
> I downloaded the code examples from Hadoop-The definitive guide, 3rd edition
> and build it using Maven:
> mvn package -DskipTests -Dhadoop.distro=apache-2
>
> Next I imported the maven projects into Eclipse. Using Eclipse now I can
> develop my own Map Reduce jobs.
> Bu how do I test\run the job locally using the "Local Job Runner"?
>
> The book excerpt says:
> Now we can run this application against some local files. Hadoop comes with
> a local
> job runner, a cut-down version of the MapReduce execution engine for running
> Map-
> Reduce jobs in a single JVM. It's designed for testing, and is very
> convenient for use in
> an IDE, since you can run it in a debugger to step through the code in your
> mapper and
> reducer.
>
> Do I also need to install Hadoop locally on Windows for that?
>
> Thanks,
> -RR
--
Harsh J
Re: Running a Job in a Local Job Runner:Windows 7 64-bit
Posted by Harsh J <ha...@cloudera.com>.
Running your Driver class from Eclipse should automatically run it in
the local runner mode (as thats the default mode).
You shouldn't need a local Hadoop install for this.
On Thu, Mar 6, 2014 at 11:36 AM, Radhe Radhe
<ra...@live.com> wrote:
> Hi All,
>
> I'm trying to get some hands-on on the Map Reduce programming.
>
> I downloaded the code examples from Hadoop-The definitive guide, 3rd edition
> and build it using Maven:
> mvn package -DskipTests -Dhadoop.distro=apache-2
>
> Next I imported the maven projects into Eclipse. Using Eclipse now I can
> develop my own Map Reduce jobs.
> Bu how do I test\run the job locally using the "Local Job Runner"?
>
> The book excerpt says:
> Now we can run this application against some local files. Hadoop comes with
> a local
> job runner, a cut-down version of the MapReduce execution engine for running
> Map-
> Reduce jobs in a single JVM. It's designed for testing, and is very
> convenient for use in
> an IDE, since you can run it in a debugger to step through the code in your
> mapper and
> reducer.
>
> Do I also need to install Hadoop locally on Windows for that?
>
> Thanks,
> -RR
--
Harsh J