You are viewing a plain text version of this content. The canonical link for it is here.

Posted to common-user@hadoop.apache.org by praveenesh kumar <pr...@gmail.com> on 2011/04/20 07:30:54 UTC

Map - Reduce Programs in ECLIPSE JAVA ??

Hello people,

I am beginner..
I already set up hadoop cluster of around 4 nodes.
Now I am looking forward to make map-reduce programs on them.
I am using eclipse plugin to connect to hadoop master node..but I am not
able to get any useful tutorials on net to develop map reduce programs on
hadoop clusters..I am working on windows to connect to hadoop cluster..
Can anyone suggest me any good tutorials so that I can start making map
reduce programs on my hadoop cluster.. using eclipse..

Thanks,
Praveenesh

Re: Map - Reduce Programs in ECLIPSE JAVA ??

Posted by Matt Pouttu-Clarke <Ma...@icrossing.com>.

Hi Praveenesh,

Actually, we have had great success using VMWare images from Cloudera:
https://downloads.cloudera.com/cloudera-demo-0.3.7.vmwarevm.tar.bz2

This is way easier than Cygwin and the image also includes Eclipse so you
can run and debug everything locally...

Just be aware that Cloudera bundles extra jars in the Hadoop class path so
if you are planning on running on an Apache based cloud computing platform
in production (such as Elastic Map Reduce) you may need to eventually build
your own VMWare image for your target platform.

Cheers,
Matt 


On 4/21/11 10:01 AM, "Harsh J" <ha...@cloudera.com> wrote:

> What happens when you try to submit? Does it not pack a jar and
> submit? You should use the plugin provided run configuration to launch
> jobs.
> 
> Note that you need Cygwin on Windows for using Hadoop, whether it
> hosts the cluster or not (there are ugly workarounds to this that you
> wouldn't want to do all the time...).
> 
> On Wed, Apr 20, 2011 at 11:31 AM, praveenesh kumar <pr...@gmail.com>
> wrote:
>> Hi,
>> Thanks for the tutorial.
>> But the problem is I am trying to connect to hadoop clusters from a windows
>> machine that is not a part of hadoop cluster..
>> Bascially we installed hadoop on linux server machines.. so no GUI is
>> available. So for that reason, what we are trying to do is to run eclipse
>> with hadoop eclipse plugin on windows machine.. and trying to runn code from
>> windows machine.
>> 
>> I am able to connect to hadoop master node using this plugin.. I can see the
>> hdfs files.. but I am not able to figure out how I can run map reduce
>> programs from this windows machine to the hadoop master node..
>> 
>> Am I doing something wrong .. ?? or I can run the M-R programs only on the
>> hadoop master node ??
>> Thanks,
>> Praveenesh
>> On Wed, Apr 20, 2011 at 11:06 AM, Harsh J <ha...@cloudera.com> wrote:
>> 
>>> Hello Praveenesh,
>>> 
>>> On Wed, Apr 20, 2011 at 11:00 AM, praveenesh kumar <pr...@gmail.com>
>>> wrote:
>>>> Hello people,
>>>> 
>>>> I am beginner..
>>>> I already set up hadoop cluster of around 4 nodes.
>>>> Now I am looking forward to make map-reduce programs on them.
>>>> I am using eclipse plugin to connect to hadoop master node..but I am not
>>>> able to get any useful tutorials on net to develop map reduce programs on
>>>> hadoop clusters..I am working on windows to connect to hadoop cluster..
>>>> Can anyone suggest me any good tutorials so that I can start making map
>>>> reduce programs on my hadoop cluster.. using eclipse..
>>> 
>>> With the Eclipse plugin configured, you should be able to create a
>>> Map/Reduce project in your IDE (File -> New -> Map/Reduce). This
>>> should be sufficient to get started. Here's an excellent resource for
>>> Windows' Eclipse:
>>> http://v-lad.org/Tutorials/Hadoop/23%20-%20create%20the%20project.html
>>> 
>>> Do note, however, that Linux is the preferred dev platform for many of
>>> Hadoop users.
>>> 
>>> --
>>> Harsh J
>>> 
>> 
> 
> 


iCrossing Privileged and Confidential Information
This email message is for the sole use of the intended recipient(s) and may contain confidential and privileged information of iCrossing. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message.

Re: Map - Reduce Programs in ECLIPSE JAVA ??

Posted by Harsh J <ha...@cloudera.com>.

What happens when you try to submit? Does it not pack a jar and
submit? You should use the plugin provided run configuration to launch
jobs.

Note that you need Cygwin on Windows for using Hadoop, whether it
hosts the cluster or not (there are ugly workarounds to this that you
wouldn't want to do all the time...).

On Wed, Apr 20, 2011 at 11:31 AM, praveenesh kumar <pr...@gmail.com> wrote:
> Hi,
> Thanks for the tutorial.
> But the problem is I am trying to connect to hadoop clusters from a windows
> machine that is not a part of hadoop cluster..
> Bascially we installed hadoop on linux server machines.. so no GUI is
> available. So for that reason, what we are trying to do is to run eclipse
> with hadoop eclipse plugin on windows machine.. and trying to runn code from
> windows machine.
>
> I am able to connect to hadoop master node using this plugin.. I can see the
> hdfs files.. but I am not able to figure out how I can run map reduce
> programs from this windows machine to the hadoop master node..
>
> Am I doing something wrong .. ?? or I can run the M-R programs only on the
> hadoop master node ??
> Thanks,
> Praveenesh
> On Wed, Apr 20, 2011 at 11:06 AM, Harsh J <ha...@cloudera.com> wrote:
>
>> Hello Praveenesh,
>>
>> On Wed, Apr 20, 2011 at 11:00 AM, praveenesh kumar <pr...@gmail.com>
>> wrote:
>> > Hello people,
>> >
>> > I am beginner..
>> > I already set up hadoop cluster of around 4 nodes.
>> > Now I am looking forward to make map-reduce programs on them.
>> > I am using eclipse plugin to connect to hadoop master node..but I am not
>> > able to get any useful tutorials on net to develop map reduce programs on
>> > hadoop clusters..I am working on windows to connect to hadoop cluster..
>> > Can anyone suggest me any good tutorials so that I can start making map
>> > reduce programs on my hadoop cluster.. using eclipse..
>>
>> With the Eclipse plugin configured, you should be able to create a
>> Map/Reduce project in your IDE (File -> New -> Map/Reduce). This
>> should be sufficient to get started. Here's an excellent resource for
>> Windows' Eclipse:
>> http://v-lad.org/Tutorials/Hadoop/23%20-%20create%20the%20project.html
>>
>> Do note, however, that Linux is the preferred dev platform for many of
>> Hadoop users.
>>
>> --
>> Harsh J
>>
>



-- 
Harsh J

Re: Map - Reduce Programs in ECLIPSE JAVA ??

Posted by abhay ratnaparkhi <ab...@gmail.com>.

I do have same question.
I have hadoop cluster on set of linux machines.
I have eclipse installed on windows machine. Can I run MR tasks from my
eclipse on hadoop cluster?

Abhay

On Wed, Apr 20, 2011 at 11:31 AM, praveenesh kumar <pr...@gmail.com>wrote:

> Hi,
> Thanks for the tutorial.
> But the problem is I am trying to connect to hadoop clusters from a windows
> machine that is not a part of hadoop cluster..
> Bascially we installed hadoop on linux server machines.. so no GUI is
> available. So for that reason, what we are trying to do is to run eclipse
> with hadoop eclipse plugin on windows machine.. and trying to runn code
> from
> windows machine.
>
> I am able to connect to hadoop master node using this plugin.. I can see
> the
> hdfs files.. but I am not able to figure out how I can run map reduce
> programs from this windows machine to the hadoop master node..
>
> Am I doing something wrong .. ?? or I can run the M-R programs only on the
> hadoop master node ??
> Thanks,
> Praveenesh
> On Wed, Apr 20, 2011 at 11:06 AM, Harsh J <ha...@cloudera.com> wrote:
>
> > Hello Praveenesh,
> >
> > On Wed, Apr 20, 2011 at 11:00 AM, praveenesh kumar <praveenesh@gmail.com
> >
> > wrote:
> > > Hello people,
> > >
> > > I am beginner..
> > > I already set up hadoop cluster of around 4 nodes.
> > > Now I am looking forward to make map-reduce programs on them.
> > > I am using eclipse plugin to connect to hadoop master node..but I am
> not
> > > able to get any useful tutorials on net to develop map reduce programs
> on
> > > hadoop clusters..I am working on windows to connect to hadoop cluster..
> > > Can anyone suggest me any good tutorials so that I can start making map
> > > reduce programs on my hadoop cluster.. using eclipse..
> >
> > With the Eclipse plugin configured, you should be able to create a
> > Map/Reduce project in your IDE (File -> New -> Map/Reduce). This
> > should be sufficient to get started. Here's an excellent resource for
> > Windows' Eclipse:
> > http://v-lad.org/Tutorials/Hadoop/23%20-%20create%20the%20project.html
> >
> > Do note, however, that Linux is the preferred dev platform for many of
> > Hadoop users.
> >
> > --
> > Harsh J
> >
>

Re: Map - Reduce Programs in ECLIPSE JAVA ??

Posted by praveenesh kumar <pr...@gmail.com>.

Hi,
Thanks for the tutorial.
But the problem is I am trying to connect to hadoop clusters from a windows
machine that is not a part of hadoop cluster..
Bascially we installed hadoop on linux server machines.. so no GUI is
available. So for that reason, what we are trying to do is to run eclipse
with hadoop eclipse plugin on windows machine.. and trying to runn code from
windows machine.

I am able to connect to hadoop master node using this plugin.. I can see the
hdfs files.. but I am not able to figure out how I can run map reduce
programs from this windows machine to the hadoop master node..

Am I doing something wrong .. ?? or I can run the M-R programs only on the
hadoop master node ??
Thanks,
Praveenesh
On Wed, Apr 20, 2011 at 11:06 AM, Harsh J <ha...@cloudera.com> wrote:

> Hello Praveenesh,
>
> On Wed, Apr 20, 2011 at 11:00 AM, praveenesh kumar <pr...@gmail.com>
> wrote:
> > Hello people,
> >
> > I am beginner..
> > I already set up hadoop cluster of around 4 nodes.
> > Now I am looking forward to make map-reduce programs on them.
> > I am using eclipse plugin to connect to hadoop master node..but I am not
> > able to get any useful tutorials on net to develop map reduce programs on
> > hadoop clusters..I am working on windows to connect to hadoop cluster..
> > Can anyone suggest me any good tutorials so that I can start making map
> > reduce programs on my hadoop cluster.. using eclipse..
>
> With the Eclipse plugin configured, you should be able to create a
> Map/Reduce project in your IDE (File -> New -> Map/Reduce). This
> should be sufficient to get started. Here's an excellent resource for
> Windows' Eclipse:
> http://v-lad.org/Tutorials/Hadoop/23%20-%20create%20the%20project.html
>
> Do note, however, that Linux is the preferred dev platform for many of
> Hadoop users.
>
> --
> Harsh J
>

Re: Map - Reduce Programs in ECLIPSE JAVA ??

Posted by Harsh J <ha...@cloudera.com>.

Hello Praveenesh,

On Wed, Apr 20, 2011 at 11:00 AM, praveenesh kumar <pr...@gmail.com> wrote:
> Hello people,
>
> I am beginner..
> I already set up hadoop cluster of around 4 nodes.
> Now I am looking forward to make map-reduce programs on them.
> I am using eclipse plugin to connect to hadoop master node..but I am not
> able to get any useful tutorials on net to develop map reduce programs on
> hadoop clusters..I am working on windows to connect to hadoop cluster..
> Can anyone suggest me any good tutorials so that I can start making map
> reduce programs on my hadoop cluster.. using eclipse..

With the Eclipse plugin configured, you should be able to create a
Map/Reduce project in your IDE (File -> New -> Map/Reduce). This
should be sufficient to get started. Here's an excellent resource for
Windows' Eclipse:
http://v-lad.org/Tutorials/Hadoop/23%20-%20create%20the%20project.html

Do note, however, that Linux is the preferred dev platform for many of
Hadoop users.

-- 
Harsh J