MapReduce jobs from remote

[Alberto Cordioli <co...@gmail.com>]

Hi all,

Now I'd like to deploy a simple MapReduce job, written in Java, to a
remote cluster within Eclipse.
For the moment I've found this solution:

1) Put the hadoop conf file in the classpath
1) Put the jar containing the job in the classpath.
2) Run

If I don't put the jar in the classpath when I run the job it returns
a ClassNotFoundException:
java.lang.RuntimeException: java.lang.ClassNotFoundException:  <MyMapperClass>

I was wondering if there are other methods to do so in simpler way.


Thank you very much,
Alberto

Re: MapReduce jobs from remote

[Bertrand Dechoux <de...@gmail.com>]

I haven't tried it but there a Hadoop plugin for eclipse :
http://wiki.apache.org/hadoop/EclipsePlugIn

Regards

Bertrand

On Thu, Sep 20, 2012 at 4:14 PM, Alberto Cordioli <
cordioli.alberto@gmail.com> wrote:

> Hi all,
>
> Now I'd like to deploy a simple MapReduce job, written in Java, to a
> remote cluster within Eclipse.
> For the moment I've found this solution:
>
> 1) Put the hadoop conf file in the classpath
> 1) Put the jar containing the job in the classpath.
> 2) Run
>
> If I don't put the jar in the classpath when I run the job it returns
> a ClassNotFoundException:
> java.lang.RuntimeException: java.lang.ClassNotFoundException:
>  <MyMapperClass>
>
> I was wondering if there are other methods to do so in simpler way.
>
>
> Thank you very much,
> Alberto
>



-- 
Bertrand Dechoux

Re: MapReduce jobs from remote

[Bertrand Dechoux <de...@gmail.com>]

I haven't tried it but there a Hadoop plugin for eclipse :
http://wiki.apache.org/hadoop/EclipsePlugIn

Regards

Bertrand

On Thu, Sep 20, 2012 at 4:14 PM, Alberto Cordioli <
cordioli.alberto@gmail.com> wrote:

> Hi all,
>
> Now I'd like to deploy a simple MapReduce job, written in Java, to a
> remote cluster within Eclipse.
> For the moment I've found this solution:
>
> 1) Put the hadoop conf file in the classpath
> 1) Put the jar containing the job in the classpath.
> 2) Run
>
> If I don't put the jar in the classpath when I run the job it returns
> a ClassNotFoundException:
> java.lang.RuntimeException: java.lang.ClassNotFoundException:
>  <MyMapperClass>
>
> I was wondering if there are other methods to do so in simpler way.
>
>
> Thank you very much,
> Alberto
>



-- 
Bertrand Dechoux

Re: MapReduce jobs from remote

[Alberto Cordioli <co...@gmail.com>]

Ah, ok! I've thought so.

Anyway, thanks for the help,
Alberto

Il giorno 21/set/2012 17.01, "Harsh J" <ha...@cloudera.com> ha scritto:
Alberto,

Oh alright, understood. Yes I just looked through the code and it does
expect to find a jar with the set class in it.

I guess there's no way around this then, aside of using the Eclipse
plugin which automates the jar build process anyway (You can build one
for your own hadoop version).

On Fri, Sep 21, 2012 at 8:21 PM, Alberto Cordioli

<co...@gmail.com> wrote:
> No, I'm speaking about the context within Eclipse.
> If I do: ...
--
Harsh J

Re: MapReduce jobs from remote

[Alberto Cordioli <co...@gmail.com>]

Ah, ok! I've thought so.

Anyway, thanks for the help,
Alberto

Il giorno 21/set/2012 17.01, "Harsh J" <ha...@cloudera.com> ha scritto:
Alberto,

Oh alright, understood. Yes I just looked through the code and it does
expect to find a jar with the set class in it.

I guess there's no way around this then, aside of using the Eclipse
plugin which automates the jar build process anyway (You can build one
for your own hadoop version).

On Fri, Sep 21, 2012 at 8:21 PM, Alberto Cordioli

<co...@gmail.com> wrote:
> No, I'm speaking about the context within Eclipse.
> If I do: ...
--
Harsh J

Re: MapReduce jobs from remote

[Alberto Cordioli <co...@gmail.com>]

Ah, ok! I've thought so.

Anyway, thanks for the help,
Alberto

Il giorno 21/set/2012 17.01, "Harsh J" <ha...@cloudera.com> ha scritto:
Alberto,

Oh alright, understood. Yes I just looked through the code and it does
expect to find a jar with the set class in it.

I guess there's no way around this then, aside of using the Eclipse
plugin which automates the jar build process anyway (You can build one
for your own hadoop version).

On Fri, Sep 21, 2012 at 8:21 PM, Alberto Cordioli

<co...@gmail.com> wrote:
> No, I'm speaking about the context within Eclipse.
> If I do: ...
--
Harsh J

Re: MapReduce jobs from remote

[Alberto Cordioli <co...@gmail.com>]

Ah, ok! I've thought so.

Anyway, thanks for the help,
Alberto

Il giorno 21/set/2012 17.01, "Harsh J" <ha...@cloudera.com> ha scritto:
Alberto,

Oh alright, understood. Yes I just looked through the code and it does
expect to find a jar with the set class in it.

I guess there's no way around this then, aside of using the Eclipse
plugin which automates the jar build process anyway (You can build one
for your own hadoop version).

On Fri, Sep 21, 2012 at 8:21 PM, Alberto Cordioli

<co...@gmail.com> wrote:
> No, I'm speaking about the context within Eclipse.
> If I do: ...
--
Harsh J

Re: MapReduce jobs from remote

[Harsh J <ha...@cloudera.com>]

Alberto,

Oh alright, understood. Yes I just looked through the code and it does
expect to find a jar with the set class in it.

I guess there's no way around this then, aside of using the Eclipse
plugin which automates the jar build process anyway (You can build one
for your own hadoop version).

On Fri, Sep 21, 2012 at 8:21 PM, Alberto Cordioli
<co...@gmail.com> wrote:
> No, I'm speaking about the context within Eclipse.
> If I do: "Run" in Eclipse, only the classes are in the classpath, not
> the jar file.
>
> Recap: if I want to run my MapReduce job within eclipse I need to:
>
>  - generate first the jar file (I use Maven to do this)
>  - Put the jar in classpath
>  - RightClick -> Run
>
> an it works.
> I was just wondering if there is a method to skip step 1, generating
> automatically the jar to put in classpath.
>
>
> On 21 September 2012 16:13, Harsh J <ha...@cloudera.com> wrote:
>> Alberto,
>>
>> Do you speak of this outside Eclipse? Of course the jar has to exist
>> for the classes to be found. If you do not want that, then you need to
>> either have the jar in the Hadoop's lib directories itself, or on HDFS
>> and submissions via Oozie.
>>
>> On Fri, Sep 21, 2012 at 7:10 PM, Alberto Cordioli
>> <co...@gmail.com> wrote:
>>> But, the call setJarByClass(Class) does not build the jar itself, right?
>>> The compiled jar should be in any way in the classpath.
>>>
>>> Actually, I've tried adding this command, but I still get
>>> ClassNotFoundException if I don't put the jar in the run classpath.
>>>
>>>
>>> Alberto
>>>
>>>
>>> On 21 September 2012 12:39, Harsh J <ha...@cloudera.com> wrote:
>>>> Hi Alberto,
>>>>
>>>> When doing a "hadoop jar <jar>" you'll of course need a compiled jar, yes.
>>>>
>>>> When running from Eclipse the job class files are on your classpath
>>>> already. If you use the job.setJarByClass(Driver.class); call in your
>>>> submitter, it will ensure it can pack up all job-required jars
>>>> properly and ship it as the job jar when you call submit()/runJob().
>>>> Is this something you're missing to do?
>>>>
>>>> On Fri, Sep 21, 2012 at 1:00 PM, Alberto Cordioli
>>>> <co...@gmail.com> wrote:
>>>>> Bertrand, I read about the plugin, but it seems to work only with
>>>>> Hadoop 0.20.2 and I am currently working with Hadoop 1.0.3
>>>>>
>>>>> Harsh, it's not really clear to me why you're saying that it's enough
>>>>> to have the config file in the classpath.
>>>>> Also the jar is required, right? Otherwise I get
>>>>> ClassNotFoundException. Am I missing something?
>>>>>
>>>>> Thank you very much,
>>>>> Alberto
>>>>>
>>>>>
>>>>> On 20 September 2012 17:18, Harsh J <ha...@cloudera.com> wrote:
>>>>>> You have already found the simplest way of doing this, I think. The
>>>>>> other way may be to use Oozie if your job jars don't change much and
>>>>>> can be staged directly into HDFS for ready submit-when-required.
>>>>>>
>>>>>> However, if you wanna run from Eclipse, you just need the config file
>>>>>> resources on the classpath of the run configuration of your job
>>>>>> project. That should work well enough.
>>>>>>
>>>>>> On Thu, Sep 20, 2012 at 7:44 PM, Alberto Cordioli
>>>>>> <co...@gmail.com> wrote:
>>>>>>> Hi all,
>>>>>>>
>>>>>>> Now I'd like to deploy a simple MapReduce job, written in Java, to a
>>>>>>> remote cluster within Eclipse.
>>>>>>> For the moment I've found this solution:
>>>>>>>
>>>>>>> 1) Put the hadoop conf file in the classpath
>>>>>>> 1) Put the jar containing the job in the classpath.
>>>>>>> 2) Run
>>>>>>>
>>>>>>> If I don't put the jar in the classpath when I run the job it returns
>>>>>>> a ClassNotFoundException:
>>>>>>> java.lang.RuntimeException: java.lang.ClassNotFoundException:  <MyMapperClass>
>>>>>>>
>>>>>>> I was wondering if there are other methods to do so in simpler way.
>>>>>>>
>>>>>>>
>>>>>>> Thank you very much,
>>>>>>> Alberto
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Harsh J
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Alberto Cordioli
>>>>
>>>>
>>>>
>>>> --
>>>> Harsh J
>>>
>>>
>>>
>>> --
>>> Alberto Cordioli
>>
>>
>>
>> --
>> Harsh J
>
>
>
> --
> Alberto Cordioli



-- 
Harsh J

Re: MapReduce jobs from remote

[Harsh J <ha...@cloudera.com>]

Alberto,

Oh alright, understood. Yes I just looked through the code and it does
expect to find a jar with the set class in it.

I guess there's no way around this then, aside of using the Eclipse
plugin which automates the jar build process anyway (You can build one
for your own hadoop version).

On Fri, Sep 21, 2012 at 8:21 PM, Alberto Cordioli
<co...@gmail.com> wrote:
> No, I'm speaking about the context within Eclipse.
> If I do: "Run" in Eclipse, only the classes are in the classpath, not
> the jar file.
>
> Recap: if I want to run my MapReduce job within eclipse I need to:
>
>  - generate first the jar file (I use Maven to do this)
>  - Put the jar in classpath
>  - RightClick -> Run
>
> an it works.
> I was just wondering if there is a method to skip step 1, generating
> automatically the jar to put in classpath.
>
>
> On 21 September 2012 16:13, Harsh J <ha...@cloudera.com> wrote:
>> Alberto,
>>
>> Do you speak of this outside Eclipse? Of course the jar has to exist
>> for the classes to be found. If you do not want that, then you need to
>> either have the jar in the Hadoop's lib directories itself, or on HDFS
>> and submissions via Oozie.
>>
>> On Fri, Sep 21, 2012 at 7:10 PM, Alberto Cordioli
>> <co...@gmail.com> wrote:
>>> But, the call setJarByClass(Class) does not build the jar itself, right?
>>> The compiled jar should be in any way in the classpath.
>>>
>>> Actually, I've tried adding this command, but I still get
>>> ClassNotFoundException if I don't put the jar in the run classpath.
>>>
>>>
>>> Alberto
>>>
>>>
>>> On 21 September 2012 12:39, Harsh J <ha...@cloudera.com> wrote:
>>>> Hi Alberto,
>>>>
>>>> When doing a "hadoop jar <jar>" you'll of course need a compiled jar, yes.
>>>>
>>>> When running from Eclipse the job class files are on your classpath
>>>> already. If you use the job.setJarByClass(Driver.class); call in your
>>>> submitter, it will ensure it can pack up all job-required jars
>>>> properly and ship it as the job jar when you call submit()/runJob().
>>>> Is this something you're missing to do?
>>>>
>>>> On Fri, Sep 21, 2012 at 1:00 PM, Alberto Cordioli
>>>> <co...@gmail.com> wrote:
>>>>> Bertrand, I read about the plugin, but it seems to work only with
>>>>> Hadoop 0.20.2 and I am currently working with Hadoop 1.0.3
>>>>>
>>>>> Harsh, it's not really clear to me why you're saying that it's enough
>>>>> to have the config file in the classpath.
>>>>> Also the jar is required, right? Otherwise I get
>>>>> ClassNotFoundException. Am I missing something?
>>>>>
>>>>> Thank you very much,
>>>>> Alberto
>>>>>
>>>>>
>>>>> On 20 September 2012 17:18, Harsh J <ha...@cloudera.com> wrote:
>>>>>> You have already found the simplest way of doing this, I think. The
>>>>>> other way may be to use Oozie if your job jars don't change much and
>>>>>> can be staged directly into HDFS for ready submit-when-required.
>>>>>>
>>>>>> However, if you wanna run from Eclipse, you just need the config file
>>>>>> resources on the classpath of the run configuration of your job
>>>>>> project. That should work well enough.
>>>>>>
>>>>>> On Thu, Sep 20, 2012 at 7:44 PM, Alberto Cordioli
>>>>>> <co...@gmail.com> wrote:
>>>>>>> Hi all,
>>>>>>>
>>>>>>> Now I'd like to deploy a simple MapReduce job, written in Java, to a
>>>>>>> remote cluster within Eclipse.
>>>>>>> For the moment I've found this solution:
>>>>>>>
>>>>>>> 1) Put the hadoop conf file in the classpath
>>>>>>> 1) Put the jar containing the job in the classpath.
>>>>>>> 2) Run
>>>>>>>
>>>>>>> If I don't put the jar in the classpath when I run the job it returns
>>>>>>> a ClassNotFoundException:
>>>>>>> java.lang.RuntimeException: java.lang.ClassNotFoundException:  <MyMapperClass>
>>>>>>>
>>>>>>> I was wondering if there are other methods to do so in simpler way.
>>>>>>>
>>>>>>>
>>>>>>> Thank you very much,
>>>>>>> Alberto
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Harsh J
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Alberto Cordioli
>>>>
>>>>
>>>>
>>>> --
>>>> Harsh J
>>>
>>>
>>>
>>> --
>>> Alberto Cordioli
>>
>>
>>
>> --
>> Harsh J
>
>
>
> --
> Alberto Cordioli



-- 
Harsh J

Re: MapReduce jobs from remote

[Harsh J <ha...@cloudera.com>]

Alberto,

Oh alright, understood. Yes I just looked through the code and it does
expect to find a jar with the set class in it.

I guess there's no way around this then, aside of using the Eclipse
plugin which automates the jar build process anyway (You can build one
for your own hadoop version).

On Fri, Sep 21, 2012 at 8:21 PM, Alberto Cordioli
<co...@gmail.com> wrote:
> No, I'm speaking about the context within Eclipse.
> If I do: "Run" in Eclipse, only the classes are in the classpath, not
> the jar file.
>
> Recap: if I want to run my MapReduce job within eclipse I need to:
>
>  - generate first the jar file (I use Maven to do this)
>  - Put the jar in classpath
>  - RightClick -> Run
>
> an it works.
> I was just wondering if there is a method to skip step 1, generating
> automatically the jar to put in classpath.
>
>
> On 21 September 2012 16:13, Harsh J <ha...@cloudera.com> wrote:
>> Alberto,
>>
>> Do you speak of this outside Eclipse? Of course the jar has to exist
>> for the classes to be found. If you do not want that, then you need to
>> either have the jar in the Hadoop's lib directories itself, or on HDFS
>> and submissions via Oozie.
>>
>> On Fri, Sep 21, 2012 at 7:10 PM, Alberto Cordioli
>> <co...@gmail.com> wrote:
>>> But, the call setJarByClass(Class) does not build the jar itself, right?
>>> The compiled jar should be in any way in the classpath.
>>>
>>> Actually, I've tried adding this command, but I still get
>>> ClassNotFoundException if I don't put the jar in the run classpath.
>>>
>>>
>>> Alberto
>>>
>>>
>>> On 21 September 2012 12:39, Harsh J <ha...@cloudera.com> wrote:
>>>> Hi Alberto,
>>>>
>>>> When doing a "hadoop jar <jar>" you'll of course need a compiled jar, yes.
>>>>
>>>> When running from Eclipse the job class files are on your classpath
>>>> already. If you use the job.setJarByClass(Driver.class); call in your
>>>> submitter, it will ensure it can pack up all job-required jars
>>>> properly and ship it as the job jar when you call submit()/runJob().
>>>> Is this something you're missing to do?
>>>>
>>>> On Fri, Sep 21, 2012 at 1:00 PM, Alberto Cordioli
>>>> <co...@gmail.com> wrote:
>>>>> Bertrand, I read about the plugin, but it seems to work only with
>>>>> Hadoop 0.20.2 and I am currently working with Hadoop 1.0.3
>>>>>
>>>>> Harsh, it's not really clear to me why you're saying that it's enough
>>>>> to have the config file in the classpath.
>>>>> Also the jar is required, right? Otherwise I get
>>>>> ClassNotFoundException. Am I missing something?
>>>>>
>>>>> Thank you very much,
>>>>> Alberto
>>>>>
>>>>>
>>>>> On 20 September 2012 17:18, Harsh J <ha...@cloudera.com> wrote:
>>>>>> You have already found the simplest way of doing this, I think. The
>>>>>> other way may be to use Oozie if your job jars don't change much and
>>>>>> can be staged directly into HDFS for ready submit-when-required.
>>>>>>
>>>>>> However, if you wanna run from Eclipse, you just need the config file
>>>>>> resources on the classpath of the run configuration of your job
>>>>>> project. That should work well enough.
>>>>>>
>>>>>> On Thu, Sep 20, 2012 at 7:44 PM, Alberto Cordioli
>>>>>> <co...@gmail.com> wrote:
>>>>>>> Hi all,
>>>>>>>
>>>>>>> Now I'd like to deploy a simple MapReduce job, written in Java, to a
>>>>>>> remote cluster within Eclipse.
>>>>>>> For the moment I've found this solution:
>>>>>>>
>>>>>>> 1) Put the hadoop conf file in the classpath
>>>>>>> 1) Put the jar containing the job in the classpath.
>>>>>>> 2) Run
>>>>>>>
>>>>>>> If I don't put the jar in the classpath when I run the job it returns
>>>>>>> a ClassNotFoundException:
>>>>>>> java.lang.RuntimeException: java.lang.ClassNotFoundException:  <MyMapperClass>
>>>>>>>
>>>>>>> I was wondering if there are other methods to do so in simpler way.
>>>>>>>
>>>>>>>
>>>>>>> Thank you very much,
>>>>>>> Alberto
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Harsh J
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Alberto Cordioli
>>>>
>>>>
>>>>
>>>> --
>>>> Harsh J
>>>
>>>
>>>
>>> --
>>> Alberto Cordioli
>>
>>
>>
>> --
>> Harsh J
>
>
>
> --
> Alberto Cordioli



-- 
Harsh J

Re: MapReduce jobs from remote

[Harsh J <ha...@cloudera.com>]

Alberto,

Oh alright, understood. Yes I just looked through the code and it does
expect to find a jar with the set class in it.

I guess there's no way around this then, aside of using the Eclipse
plugin which automates the jar build process anyway (You can build one
for your own hadoop version).

On Fri, Sep 21, 2012 at 8:21 PM, Alberto Cordioli
<co...@gmail.com> wrote:
> No, I'm speaking about the context within Eclipse.
> If I do: "Run" in Eclipse, only the classes are in the classpath, not
> the jar file.
>
> Recap: if I want to run my MapReduce job within eclipse I need to:
>
>  - generate first the jar file (I use Maven to do this)
>  - Put the jar in classpath
>  - RightClick -> Run
>
> an it works.
> I was just wondering if there is a method to skip step 1, generating
> automatically the jar to put in classpath.
>
>
> On 21 September 2012 16:13, Harsh J <ha...@cloudera.com> wrote:
>> Alberto,
>>
>> Do you speak of this outside Eclipse? Of course the jar has to exist
>> for the classes to be found. If you do not want that, then you need to
>> either have the jar in the Hadoop's lib directories itself, or on HDFS
>> and submissions via Oozie.
>>
>> On Fri, Sep 21, 2012 at 7:10 PM, Alberto Cordioli
>> <co...@gmail.com> wrote:
>>> But, the call setJarByClass(Class) does not build the jar itself, right?
>>> The compiled jar should be in any way in the classpath.
>>>
>>> Actually, I've tried adding this command, but I still get
>>> ClassNotFoundException if I don't put the jar in the run classpath.
>>>
>>>
>>> Alberto
>>>
>>>
>>> On 21 September 2012 12:39, Harsh J <ha...@cloudera.com> wrote:
>>>> Hi Alberto,
>>>>
>>>> When doing a "hadoop jar <jar>" you'll of course need a compiled jar, yes.
>>>>
>>>> When running from Eclipse the job class files are on your classpath
>>>> already. If you use the job.setJarByClass(Driver.class); call in your
>>>> submitter, it will ensure it can pack up all job-required jars
>>>> properly and ship it as the job jar when you call submit()/runJob().
>>>> Is this something you're missing to do?
>>>>
>>>> On Fri, Sep 21, 2012 at 1:00 PM, Alberto Cordioli
>>>> <co...@gmail.com> wrote:
>>>>> Bertrand, I read about the plugin, but it seems to work only with
>>>>> Hadoop 0.20.2 and I am currently working with Hadoop 1.0.3
>>>>>
>>>>> Harsh, it's not really clear to me why you're saying that it's enough
>>>>> to have the config file in the classpath.
>>>>> Also the jar is required, right? Otherwise I get
>>>>> ClassNotFoundException. Am I missing something?
>>>>>
>>>>> Thank you very much,
>>>>> Alberto
>>>>>
>>>>>
>>>>> On 20 September 2012 17:18, Harsh J <ha...@cloudera.com> wrote:
>>>>>> You have already found the simplest way of doing this, I think. The
>>>>>> other way may be to use Oozie if your job jars don't change much and
>>>>>> can be staged directly into HDFS for ready submit-when-required.
>>>>>>
>>>>>> However, if you wanna run from Eclipse, you just need the config file
>>>>>> resources on the classpath of the run configuration of your job
>>>>>> project. That should work well enough.
>>>>>>
>>>>>> On Thu, Sep 20, 2012 at 7:44 PM, Alberto Cordioli
>>>>>> <co...@gmail.com> wrote:
>>>>>>> Hi all,
>>>>>>>
>>>>>>> Now I'd like to deploy a simple MapReduce job, written in Java, to a
>>>>>>> remote cluster within Eclipse.
>>>>>>> For the moment I've found this solution:
>>>>>>>
>>>>>>> 1) Put the hadoop conf file in the classpath
>>>>>>> 1) Put the jar containing the job in the classpath.
>>>>>>> 2) Run
>>>>>>>
>>>>>>> If I don't put the jar in the classpath when I run the job it returns
>>>>>>> a ClassNotFoundException:
>>>>>>> java.lang.RuntimeException: java.lang.ClassNotFoundException:  <MyMapperClass>
>>>>>>>
>>>>>>> I was wondering if there are other methods to do so in simpler way.
>>>>>>>
>>>>>>>
>>>>>>> Thank you very much,
>>>>>>> Alberto
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Harsh J
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Alberto Cordioli
>>>>
>>>>
>>>>
>>>> --
>>>> Harsh J
>>>
>>>
>>>
>>> --
>>> Alberto Cordioli
>>
>>
>>
>> --
>> Harsh J
>
>
>
> --
> Alberto Cordioli



-- 
Harsh J

Re: MapReduce jobs from remote

[Alberto Cordioli <co...@gmail.com>]

No, I'm speaking about the context within Eclipse.
If I do: "Run" in Eclipse, only the classes are in the classpath, not
the jar file.

Recap: if I want to run my MapReduce job within eclipse I need to:

 - generate first the jar file (I use Maven to do this)
 - Put the jar in classpath
 - RightClick -> Run

an it works.
I was just wondering if there is a method to skip step 1, generating
automatically the jar to put in classpath.


On 21 September 2012 16:13, Harsh J <ha...@cloudera.com> wrote:
> Alberto,
>
> Do you speak of this outside Eclipse? Of course the jar has to exist
> for the classes to be found. If you do not want that, then you need to
> either have the jar in the Hadoop's lib directories itself, or on HDFS
> and submissions via Oozie.
>
> On Fri, Sep 21, 2012 at 7:10 PM, Alberto Cordioli
> <co...@gmail.com> wrote:
>> But, the call setJarByClass(Class) does not build the jar itself, right?
>> The compiled jar should be in any way in the classpath.
>>
>> Actually, I've tried adding this command, but I still get
>> ClassNotFoundException if I don't put the jar in the run classpath.
>>
>>
>> Alberto
>>
>>
>> On 21 September 2012 12:39, Harsh J <ha...@cloudera.com> wrote:
>>> Hi Alberto,
>>>
>>> When doing a "hadoop jar <jar>" you'll of course need a compiled jar, yes.
>>>
>>> When running from Eclipse the job class files are on your classpath
>>> already. If you use the job.setJarByClass(Driver.class); call in your
>>> submitter, it will ensure it can pack up all job-required jars
>>> properly and ship it as the job jar when you call submit()/runJob().
>>> Is this something you're missing to do?
>>>
>>> On Fri, Sep 21, 2012 at 1:00 PM, Alberto Cordioli
>>> <co...@gmail.com> wrote:
>>>> Bertrand, I read about the plugin, but it seems to work only with
>>>> Hadoop 0.20.2 and I am currently working with Hadoop 1.0.3
>>>>
>>>> Harsh, it's not really clear to me why you're saying that it's enough
>>>> to have the config file in the classpath.
>>>> Also the jar is required, right? Otherwise I get
>>>> ClassNotFoundException. Am I missing something?
>>>>
>>>> Thank you very much,
>>>> Alberto
>>>>
>>>>
>>>> On 20 September 2012 17:18, Harsh J <ha...@cloudera.com> wrote:
>>>>> You have already found the simplest way of doing this, I think. The
>>>>> other way may be to use Oozie if your job jars don't change much and
>>>>> can be staged directly into HDFS for ready submit-when-required.
>>>>>
>>>>> However, if you wanna run from Eclipse, you just need the config file
>>>>> resources on the classpath of the run configuration of your job
>>>>> project. That should work well enough.
>>>>>
>>>>> On Thu, Sep 20, 2012 at 7:44 PM, Alberto Cordioli
>>>>> <co...@gmail.com> wrote:
>>>>>> Hi all,
>>>>>>
>>>>>> Now I'd like to deploy a simple MapReduce job, written in Java, to a
>>>>>> remote cluster within Eclipse.
>>>>>> For the moment I've found this solution:
>>>>>>
>>>>>> 1) Put the hadoop conf file in the classpath
>>>>>> 1) Put the jar containing the job in the classpath.
>>>>>> 2) Run
>>>>>>
>>>>>> If I don't put the jar in the classpath when I run the job it returns
>>>>>> a ClassNotFoundException:
>>>>>> java.lang.RuntimeException: java.lang.ClassNotFoundException:  <MyMapperClass>
>>>>>>
>>>>>> I was wondering if there are other methods to do so in simpler way.
>>>>>>
>>>>>>
>>>>>> Thank you very much,
>>>>>> Alberto
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Harsh J
>>>>
>>>>
>>>>
>>>> --
>>>> Alberto Cordioli
>>>
>>>
>>>
>>> --
>>> Harsh J
>>
>>
>>
>> --
>> Alberto Cordioli
>
>
>
> --
> Harsh J



-- 
Alberto Cordioli

Re: MapReduce jobs from remote

[Alberto Cordioli <co...@gmail.com>]

No, I'm speaking about the context within Eclipse.
If I do: "Run" in Eclipse, only the classes are in the classpath, not
the jar file.

Recap: if I want to run my MapReduce job within eclipse I need to:

 - generate first the jar file (I use Maven to do this)
 - Put the jar in classpath
 - RightClick -> Run

an it works.
I was just wondering if there is a method to skip step 1, generating
automatically the jar to put in classpath.


On 21 September 2012 16:13, Harsh J <ha...@cloudera.com> wrote:
> Alberto,
>
> Do you speak of this outside Eclipse? Of course the jar has to exist
> for the classes to be found. If you do not want that, then you need to
> either have the jar in the Hadoop's lib directories itself, or on HDFS
> and submissions via Oozie.
>
> On Fri, Sep 21, 2012 at 7:10 PM, Alberto Cordioli
> <co...@gmail.com> wrote:
>> But, the call setJarByClass(Class) does not build the jar itself, right?
>> The compiled jar should be in any way in the classpath.
>>
>> Actually, I've tried adding this command, but I still get
>> ClassNotFoundException if I don't put the jar in the run classpath.
>>
>>
>> Alberto
>>
>>
>> On 21 September 2012 12:39, Harsh J <ha...@cloudera.com> wrote:
>>> Hi Alberto,
>>>
>>> When doing a "hadoop jar <jar>" you'll of course need a compiled jar, yes.
>>>
>>> When running from Eclipse the job class files are on your classpath
>>> already. If you use the job.setJarByClass(Driver.class); call in your
>>> submitter, it will ensure it can pack up all job-required jars
>>> properly and ship it as the job jar when you call submit()/runJob().
>>> Is this something you're missing to do?
>>>
>>> On Fri, Sep 21, 2012 at 1:00 PM, Alberto Cordioli
>>> <co...@gmail.com> wrote:
>>>> Bertrand, I read about the plugin, but it seems to work only with
>>>> Hadoop 0.20.2 and I am currently working with Hadoop 1.0.3
>>>>
>>>> Harsh, it's not really clear to me why you're saying that it's enough
>>>> to have the config file in the classpath.
>>>> Also the jar is required, right? Otherwise I get
>>>> ClassNotFoundException. Am I missing something?
>>>>
>>>> Thank you very much,
>>>> Alberto
>>>>
>>>>
>>>> On 20 September 2012 17:18, Harsh J <ha...@cloudera.com> wrote:
>>>>> You have already found the simplest way of doing this, I think. The
>>>>> other way may be to use Oozie if your job jars don't change much and
>>>>> can be staged directly into HDFS for ready submit-when-required.
>>>>>
>>>>> However, if you wanna run from Eclipse, you just need the config file
>>>>> resources on the classpath of the run configuration of your job
>>>>> project. That should work well enough.
>>>>>
>>>>> On Thu, Sep 20, 2012 at 7:44 PM, Alberto Cordioli
>>>>> <co...@gmail.com> wrote:
>>>>>> Hi all,
>>>>>>
>>>>>> Now I'd like to deploy a simple MapReduce job, written in Java, to a
>>>>>> remote cluster within Eclipse.
>>>>>> For the moment I've found this solution:
>>>>>>
>>>>>> 1) Put the hadoop conf file in the classpath
>>>>>> 1) Put the jar containing the job in the classpath.
>>>>>> 2) Run
>>>>>>
>>>>>> If I don't put the jar in the classpath when I run the job it returns
>>>>>> a ClassNotFoundException:
>>>>>> java.lang.RuntimeException: java.lang.ClassNotFoundException:  <MyMapperClass>
>>>>>>
>>>>>> I was wondering if there are other methods to do so in simpler way.
>>>>>>
>>>>>>
>>>>>> Thank you very much,
>>>>>> Alberto
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Harsh J
>>>>
>>>>
>>>>
>>>> --
>>>> Alberto Cordioli
>>>
>>>
>>>
>>> --
>>> Harsh J
>>
>>
>>
>> --
>> Alberto Cordioli
>
>
>
> --
> Harsh J



-- 
Alberto Cordioli

Re: MapReduce jobs from remote

[Alberto Cordioli <co...@gmail.com>]

No, I'm speaking about the context within Eclipse.
If I do: "Run" in Eclipse, only the classes are in the classpath, not
the jar file.

Recap: if I want to run my MapReduce job within eclipse I need to:

 - generate first the jar file (I use Maven to do this)
 - Put the jar in classpath
 - RightClick -> Run

an it works.
I was just wondering if there is a method to skip step 1, generating
automatically the jar to put in classpath.


On 21 September 2012 16:13, Harsh J <ha...@cloudera.com> wrote:
> Alberto,
>
> Do you speak of this outside Eclipse? Of course the jar has to exist
> for the classes to be found. If you do not want that, then you need to
> either have the jar in the Hadoop's lib directories itself, or on HDFS
> and submissions via Oozie.
>
> On Fri, Sep 21, 2012 at 7:10 PM, Alberto Cordioli
> <co...@gmail.com> wrote:
>> But, the call setJarByClass(Class) does not build the jar itself, right?
>> The compiled jar should be in any way in the classpath.
>>
>> Actually, I've tried adding this command, but I still get
>> ClassNotFoundException if I don't put the jar in the run classpath.
>>
>>
>> Alberto
>>
>>
>> On 21 September 2012 12:39, Harsh J <ha...@cloudera.com> wrote:
>>> Hi Alberto,
>>>
>>> When doing a "hadoop jar <jar>" you'll of course need a compiled jar, yes.
>>>
>>> When running from Eclipse the job class files are on your classpath
>>> already. If you use the job.setJarByClass(Driver.class); call in your
>>> submitter, it will ensure it can pack up all job-required jars
>>> properly and ship it as the job jar when you call submit()/runJob().
>>> Is this something you're missing to do?
>>>
>>> On Fri, Sep 21, 2012 at 1:00 PM, Alberto Cordioli
>>> <co...@gmail.com> wrote:
>>>> Bertrand, I read about the plugin, but it seems to work only with
>>>> Hadoop 0.20.2 and I am currently working with Hadoop 1.0.3
>>>>
>>>> Harsh, it's not really clear to me why you're saying that it's enough
>>>> to have the config file in the classpath.
>>>> Also the jar is required, right? Otherwise I get
>>>> ClassNotFoundException. Am I missing something?
>>>>
>>>> Thank you very much,
>>>> Alberto
>>>>
>>>>
>>>> On 20 September 2012 17:18, Harsh J <ha...@cloudera.com> wrote:
>>>>> You have already found the simplest way of doing this, I think. The
>>>>> other way may be to use Oozie if your job jars don't change much and
>>>>> can be staged directly into HDFS for ready submit-when-required.
>>>>>
>>>>> However, if you wanna run from Eclipse, you just need the config file
>>>>> resources on the classpath of the run configuration of your job
>>>>> project. That should work well enough.
>>>>>
>>>>> On Thu, Sep 20, 2012 at 7:44 PM, Alberto Cordioli
>>>>> <co...@gmail.com> wrote:
>>>>>> Hi all,
>>>>>>
>>>>>> Now I'd like to deploy a simple MapReduce job, written in Java, to a
>>>>>> remote cluster within Eclipse.
>>>>>> For the moment I've found this solution:
>>>>>>
>>>>>> 1) Put the hadoop conf file in the classpath
>>>>>> 1) Put the jar containing the job in the classpath.
>>>>>> 2) Run
>>>>>>
>>>>>> If I don't put the jar in the classpath when I run the job it returns
>>>>>> a ClassNotFoundException:
>>>>>> java.lang.RuntimeException: java.lang.ClassNotFoundException:  <MyMapperClass>
>>>>>>
>>>>>> I was wondering if there are other methods to do so in simpler way.
>>>>>>
>>>>>>
>>>>>> Thank you very much,
>>>>>> Alberto
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Harsh J
>>>>
>>>>
>>>>
>>>> --
>>>> Alberto Cordioli
>>>
>>>
>>>
>>> --
>>> Harsh J
>>
>>
>>
>> --
>> Alberto Cordioli
>
>
>
> --
> Harsh J



-- 
Alberto Cordioli

Re: MapReduce jobs from remote

[Alberto Cordioli <co...@gmail.com>]

No, I'm speaking about the context within Eclipse.
If I do: "Run" in Eclipse, only the classes are in the classpath, not
the jar file.

Recap: if I want to run my MapReduce job within eclipse I need to:

 - generate first the jar file (I use Maven to do this)
 - Put the jar in classpath
 - RightClick -> Run

an it works.
I was just wondering if there is a method to skip step 1, generating
automatically the jar to put in classpath.


On 21 September 2012 16:13, Harsh J <ha...@cloudera.com> wrote:
> Alberto,
>
> Do you speak of this outside Eclipse? Of course the jar has to exist
> for the classes to be found. If you do not want that, then you need to
> either have the jar in the Hadoop's lib directories itself, or on HDFS
> and submissions via Oozie.
>
> On Fri, Sep 21, 2012 at 7:10 PM, Alberto Cordioli
> <co...@gmail.com> wrote:
>> But, the call setJarByClass(Class) does not build the jar itself, right?
>> The compiled jar should be in any way in the classpath.
>>
>> Actually, I've tried adding this command, but I still get
>> ClassNotFoundException if I don't put the jar in the run classpath.
>>
>>
>> Alberto
>>
>>
>> On 21 September 2012 12:39, Harsh J <ha...@cloudera.com> wrote:
>>> Hi Alberto,
>>>
>>> When doing a "hadoop jar <jar>" you'll of course need a compiled jar, yes.
>>>
>>> When running from Eclipse the job class files are on your classpath
>>> already. If you use the job.setJarByClass(Driver.class); call in your
>>> submitter, it will ensure it can pack up all job-required jars
>>> properly and ship it as the job jar when you call submit()/runJob().
>>> Is this something you're missing to do?
>>>
>>> On Fri, Sep 21, 2012 at 1:00 PM, Alberto Cordioli
>>> <co...@gmail.com> wrote:
>>>> Bertrand, I read about the plugin, but it seems to work only with
>>>> Hadoop 0.20.2 and I am currently working with Hadoop 1.0.3
>>>>
>>>> Harsh, it's not really clear to me why you're saying that it's enough
>>>> to have the config file in the classpath.
>>>> Also the jar is required, right? Otherwise I get
>>>> ClassNotFoundException. Am I missing something?
>>>>
>>>> Thank you very much,
>>>> Alberto
>>>>
>>>>
>>>> On 20 September 2012 17:18, Harsh J <ha...@cloudera.com> wrote:
>>>>> You have already found the simplest way of doing this, I think. The
>>>>> other way may be to use Oozie if your job jars don't change much and
>>>>> can be staged directly into HDFS for ready submit-when-required.
>>>>>
>>>>> However, if you wanna run from Eclipse, you just need the config file
>>>>> resources on the classpath of the run configuration of your job
>>>>> project. That should work well enough.
>>>>>
>>>>> On Thu, Sep 20, 2012 at 7:44 PM, Alberto Cordioli
>>>>> <co...@gmail.com> wrote:
>>>>>> Hi all,
>>>>>>
>>>>>> Now I'd like to deploy a simple MapReduce job, written in Java, to a
>>>>>> remote cluster within Eclipse.
>>>>>> For the moment I've found this solution:
>>>>>>
>>>>>> 1) Put the hadoop conf file in the classpath
>>>>>> 1) Put the jar containing the job in the classpath.
>>>>>> 2) Run
>>>>>>
>>>>>> If I don't put the jar in the classpath when I run the job it returns
>>>>>> a ClassNotFoundException:
>>>>>> java.lang.RuntimeException: java.lang.ClassNotFoundException:  <MyMapperClass>
>>>>>>
>>>>>> I was wondering if there are other methods to do so in simpler way.
>>>>>>
>>>>>>
>>>>>> Thank you very much,
>>>>>> Alberto
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Harsh J
>>>>
>>>>
>>>>
>>>> --
>>>> Alberto Cordioli
>>>
>>>
>>>
>>> --
>>> Harsh J
>>
>>
>>
>> --
>> Alberto Cordioli
>
>
>
> --
> Harsh J



-- 
Alberto Cordioli

Re: MapReduce jobs from remote

[Harsh J <ha...@cloudera.com>]

Alberto,

Do you speak of this outside Eclipse? Of course the jar has to exist
for the classes to be found. If you do not want that, then you need to
either have the jar in the Hadoop's lib directories itself, or on HDFS
and submissions via Oozie.

On Fri, Sep 21, 2012 at 7:10 PM, Alberto Cordioli
<co...@gmail.com> wrote:
> But, the call setJarByClass(Class) does not build the jar itself, right?
> The compiled jar should be in any way in the classpath.
>
> Actually, I've tried adding this command, but I still get
> ClassNotFoundException if I don't put the jar in the run classpath.
>
>
> Alberto
>
>
> On 21 September 2012 12:39, Harsh J <ha...@cloudera.com> wrote:
>> Hi Alberto,
>>
>> When doing a "hadoop jar <jar>" you'll of course need a compiled jar, yes.
>>
>> When running from Eclipse the job class files are on your classpath
>> already. If you use the job.setJarByClass(Driver.class); call in your
>> submitter, it will ensure it can pack up all job-required jars
>> properly and ship it as the job jar when you call submit()/runJob().
>> Is this something you're missing to do?
>>
>> On Fri, Sep 21, 2012 at 1:00 PM, Alberto Cordioli
>> <co...@gmail.com> wrote:
>>> Bertrand, I read about the plugin, but it seems to work only with
>>> Hadoop 0.20.2 and I am currently working with Hadoop 1.0.3
>>>
>>> Harsh, it's not really clear to me why you're saying that it's enough
>>> to have the config file in the classpath.
>>> Also the jar is required, right? Otherwise I get
>>> ClassNotFoundException. Am I missing something?
>>>
>>> Thank you very much,
>>> Alberto
>>>
>>>
>>> On 20 September 2012 17:18, Harsh J <ha...@cloudera.com> wrote:
>>>> You have already found the simplest way of doing this, I think. The
>>>> other way may be to use Oozie if your job jars don't change much and
>>>> can be staged directly into HDFS for ready submit-when-required.
>>>>
>>>> However, if you wanna run from Eclipse, you just need the config file
>>>> resources on the classpath of the run configuration of your job
>>>> project. That should work well enough.
>>>>
>>>> On Thu, Sep 20, 2012 at 7:44 PM, Alberto Cordioli
>>>> <co...@gmail.com> wrote:
>>>>> Hi all,
>>>>>
>>>>> Now I'd like to deploy a simple MapReduce job, written in Java, to a
>>>>> remote cluster within Eclipse.
>>>>> For the moment I've found this solution:
>>>>>
>>>>> 1) Put the hadoop conf file in the classpath
>>>>> 1) Put the jar containing the job in the classpath.
>>>>> 2) Run
>>>>>
>>>>> If I don't put the jar in the classpath when I run the job it returns
>>>>> a ClassNotFoundException:
>>>>> java.lang.RuntimeException: java.lang.ClassNotFoundException:  <MyMapperClass>
>>>>>
>>>>> I was wondering if there are other methods to do so in simpler way.
>>>>>
>>>>>
>>>>> Thank you very much,
>>>>> Alberto
>>>>
>>>>
>>>>
>>>> --
>>>> Harsh J
>>>
>>>
>>>
>>> --
>>> Alberto Cordioli
>>
>>
>>
>> --
>> Harsh J
>
>
>
> --
> Alberto Cordioli



-- 
Harsh J

Re: MapReduce jobs from remote

[Harsh J <ha...@cloudera.com>]

Alberto,

Do you speak of this outside Eclipse? Of course the jar has to exist
for the classes to be found. If you do not want that, then you need to
either have the jar in the Hadoop's lib directories itself, or on HDFS
and submissions via Oozie.

On Fri, Sep 21, 2012 at 7:10 PM, Alberto Cordioli
<co...@gmail.com> wrote:
> But, the call setJarByClass(Class) does not build the jar itself, right?
> The compiled jar should be in any way in the classpath.
>
> Actually, I've tried adding this command, but I still get
> ClassNotFoundException if I don't put the jar in the run classpath.
>
>
> Alberto
>
>
> On 21 September 2012 12:39, Harsh J <ha...@cloudera.com> wrote:
>> Hi Alberto,
>>
>> When doing a "hadoop jar <jar>" you'll of course need a compiled jar, yes.
>>
>> When running from Eclipse the job class files are on your classpath
>> already. If you use the job.setJarByClass(Driver.class); call in your
>> submitter, it will ensure it can pack up all job-required jars
>> properly and ship it as the job jar when you call submit()/runJob().
>> Is this something you're missing to do?
>>
>> On Fri, Sep 21, 2012 at 1:00 PM, Alberto Cordioli
>> <co...@gmail.com> wrote:
>>> Bertrand, I read about the plugin, but it seems to work only with
>>> Hadoop 0.20.2 and I am currently working with Hadoop 1.0.3
>>>
>>> Harsh, it's not really clear to me why you're saying that it's enough
>>> to have the config file in the classpath.
>>> Also the jar is required, right? Otherwise I get
>>> ClassNotFoundException. Am I missing something?
>>>
>>> Thank you very much,
>>> Alberto
>>>
>>>
>>> On 20 September 2012 17:18, Harsh J <ha...@cloudera.com> wrote:
>>>> You have already found the simplest way of doing this, I think. The
>>>> other way may be to use Oozie if your job jars don't change much and
>>>> can be staged directly into HDFS for ready submit-when-required.
>>>>
>>>> However, if you wanna run from Eclipse, you just need the config file
>>>> resources on the classpath of the run configuration of your job
>>>> project. That should work well enough.
>>>>
>>>> On Thu, Sep 20, 2012 at 7:44 PM, Alberto Cordioli
>>>> <co...@gmail.com> wrote:
>>>>> Hi all,
>>>>>
>>>>> Now I'd like to deploy a simple MapReduce job, written in Java, to a
>>>>> remote cluster within Eclipse.
>>>>> For the moment I've found this solution:
>>>>>
>>>>> 1) Put the hadoop conf file in the classpath
>>>>> 1) Put the jar containing the job in the classpath.
>>>>> 2) Run
>>>>>
>>>>> If I don't put the jar in the classpath when I run the job it returns
>>>>> a ClassNotFoundException:
>>>>> java.lang.RuntimeException: java.lang.ClassNotFoundException:  <MyMapperClass>
>>>>>
>>>>> I was wondering if there are other methods to do so in simpler way.
>>>>>
>>>>>
>>>>> Thank you very much,
>>>>> Alberto
>>>>
>>>>
>>>>
>>>> --
>>>> Harsh J
>>>
>>>
>>>
>>> --
>>> Alberto Cordioli
>>
>>
>>
>> --
>> Harsh J
>
>
>
> --
> Alberto Cordioli



-- 
Harsh J

Re: MapReduce jobs from remote

[Harsh J <ha...@cloudera.com>]

Alberto,

Do you speak of this outside Eclipse? Of course the jar has to exist
for the classes to be found. If you do not want that, then you need to
either have the jar in the Hadoop's lib directories itself, or on HDFS
and submissions via Oozie.

On Fri, Sep 21, 2012 at 7:10 PM, Alberto Cordioli
<co...@gmail.com> wrote:
> But, the call setJarByClass(Class) does not build the jar itself, right?
> The compiled jar should be in any way in the classpath.
>
> Actually, I've tried adding this command, but I still get
> ClassNotFoundException if I don't put the jar in the run classpath.
>
>
> Alberto
>
>
> On 21 September 2012 12:39, Harsh J <ha...@cloudera.com> wrote:
>> Hi Alberto,
>>
>> When doing a "hadoop jar <jar>" you'll of course need a compiled jar, yes.
>>
>> When running from Eclipse the job class files are on your classpath
>> already. If you use the job.setJarByClass(Driver.class); call in your
>> submitter, it will ensure it can pack up all job-required jars
>> properly and ship it as the job jar when you call submit()/runJob().
>> Is this something you're missing to do?
>>
>> On Fri, Sep 21, 2012 at 1:00 PM, Alberto Cordioli
>> <co...@gmail.com> wrote:
>>> Bertrand, I read about the plugin, but it seems to work only with
>>> Hadoop 0.20.2 and I am currently working with Hadoop 1.0.3
>>>
>>> Harsh, it's not really clear to me why you're saying that it's enough
>>> to have the config file in the classpath.
>>> Also the jar is required, right? Otherwise I get
>>> ClassNotFoundException. Am I missing something?
>>>
>>> Thank you very much,
>>> Alberto
>>>
>>>
>>> On 20 September 2012 17:18, Harsh J <ha...@cloudera.com> wrote:
>>>> You have already found the simplest way of doing this, I think. The
>>>> other way may be to use Oozie if your job jars don't change much and
>>>> can be staged directly into HDFS for ready submit-when-required.
>>>>
>>>> However, if you wanna run from Eclipse, you just need the config file
>>>> resources on the classpath of the run configuration of your job
>>>> project. That should work well enough.
>>>>
>>>> On Thu, Sep 20, 2012 at 7:44 PM, Alberto Cordioli
>>>> <co...@gmail.com> wrote:
>>>>> Hi all,
>>>>>
>>>>> Now I'd like to deploy a simple MapReduce job, written in Java, to a
>>>>> remote cluster within Eclipse.
>>>>> For the moment I've found this solution:
>>>>>
>>>>> 1) Put the hadoop conf file in the classpath
>>>>> 1) Put the jar containing the job in the classpath.
>>>>> 2) Run
>>>>>
>>>>> If I don't put the jar in the classpath when I run the job it returns
>>>>> a ClassNotFoundException:
>>>>> java.lang.RuntimeException: java.lang.ClassNotFoundException:  <MyMapperClass>
>>>>>
>>>>> I was wondering if there are other methods to do so in simpler way.
>>>>>
>>>>>
>>>>> Thank you very much,
>>>>> Alberto
>>>>
>>>>
>>>>
>>>> --
>>>> Harsh J
>>>
>>>
>>>
>>> --
>>> Alberto Cordioli
>>
>>
>>
>> --
>> Harsh J
>
>
>
> --
> Alberto Cordioli



-- 
Harsh J

Re: MapReduce jobs from remote

[Harsh J <ha...@cloudera.com>]

Alberto,

Do you speak of this outside Eclipse? Of course the jar has to exist
for the classes to be found. If you do not want that, then you need to
either have the jar in the Hadoop's lib directories itself, or on HDFS
and submissions via Oozie.

On Fri, Sep 21, 2012 at 7:10 PM, Alberto Cordioli
<co...@gmail.com> wrote:
> But, the call setJarByClass(Class) does not build the jar itself, right?
> The compiled jar should be in any way in the classpath.
>
> Actually, I've tried adding this command, but I still get
> ClassNotFoundException if I don't put the jar in the run classpath.
>
>
> Alberto
>
>
> On 21 September 2012 12:39, Harsh J <ha...@cloudera.com> wrote:
>> Hi Alberto,
>>
>> When doing a "hadoop jar <jar>" you'll of course need a compiled jar, yes.
>>
>> When running from Eclipse the job class files are on your classpath
>> already. If you use the job.setJarByClass(Driver.class); call in your
>> submitter, it will ensure it can pack up all job-required jars
>> properly and ship it as the job jar when you call submit()/runJob().
>> Is this something you're missing to do?
>>
>> On Fri, Sep 21, 2012 at 1:00 PM, Alberto Cordioli
>> <co...@gmail.com> wrote:
>>> Bertrand, I read about the plugin, but it seems to work only with
>>> Hadoop 0.20.2 and I am currently working with Hadoop 1.0.3
>>>
>>> Harsh, it's not really clear to me why you're saying that it's enough
>>> to have the config file in the classpath.
>>> Also the jar is required, right? Otherwise I get
>>> ClassNotFoundException. Am I missing something?
>>>
>>> Thank you very much,
>>> Alberto
>>>
>>>
>>> On 20 September 2012 17:18, Harsh J <ha...@cloudera.com> wrote:
>>>> You have already found the simplest way of doing this, I think. The
>>>> other way may be to use Oozie if your job jars don't change much and
>>>> can be staged directly into HDFS for ready submit-when-required.
>>>>
>>>> However, if you wanna run from Eclipse, you just need the config file
>>>> resources on the classpath of the run configuration of your job
>>>> project. That should work well enough.
>>>>
>>>> On Thu, Sep 20, 2012 at 7:44 PM, Alberto Cordioli
>>>> <co...@gmail.com> wrote:
>>>>> Hi all,
>>>>>
>>>>> Now I'd like to deploy a simple MapReduce job, written in Java, to a
>>>>> remote cluster within Eclipse.
>>>>> For the moment I've found this solution:
>>>>>
>>>>> 1) Put the hadoop conf file in the classpath
>>>>> 1) Put the jar containing the job in the classpath.
>>>>> 2) Run
>>>>>
>>>>> If I don't put the jar in the classpath when I run the job it returns
>>>>> a ClassNotFoundException:
>>>>> java.lang.RuntimeException: java.lang.ClassNotFoundException:  <MyMapperClass>
>>>>>
>>>>> I was wondering if there are other methods to do so in simpler way.
>>>>>
>>>>>
>>>>> Thank you very much,
>>>>> Alberto
>>>>
>>>>
>>>>
>>>> --
>>>> Harsh J
>>>
>>>
>>>
>>> --
>>> Alberto Cordioli
>>
>>
>>
>> --
>> Harsh J
>
>
>
> --
> Alberto Cordioli



-- 
Harsh J

Re: MapReduce jobs from remote

[Alberto Cordioli <co...@gmail.com>]

But, the call setJarByClass(Class) does not build the jar itself, right?
The compiled jar should be in any way in the classpath.

Actually, I've tried adding this command, but I still get
ClassNotFoundException if I don't put the jar in the run classpath.


Alberto


On 21 September 2012 12:39, Harsh J <ha...@cloudera.com> wrote:
> Hi Alberto,
>
> When doing a "hadoop jar <jar>" you'll of course need a compiled jar, yes.
>
> When running from Eclipse the job class files are on your classpath
> already. If you use the job.setJarByClass(Driver.class); call in your
> submitter, it will ensure it can pack up all job-required jars
> properly and ship it as the job jar when you call submit()/runJob().
> Is this something you're missing to do?
>
> On Fri, Sep 21, 2012 at 1:00 PM, Alberto Cordioli
> <co...@gmail.com> wrote:
>> Bertrand, I read about the plugin, but it seems to work only with
>> Hadoop 0.20.2 and I am currently working with Hadoop 1.0.3
>>
>> Harsh, it's not really clear to me why you're saying that it's enough
>> to have the config file in the classpath.
>> Also the jar is required, right? Otherwise I get
>> ClassNotFoundException. Am I missing something?
>>
>> Thank you very much,
>> Alberto
>>
>>
>> On 20 September 2012 17:18, Harsh J <ha...@cloudera.com> wrote:
>>> You have already found the simplest way of doing this, I think. The
>>> other way may be to use Oozie if your job jars don't change much and
>>> can be staged directly into HDFS for ready submit-when-required.
>>>
>>> However, if you wanna run from Eclipse, you just need the config file
>>> resources on the classpath of the run configuration of your job
>>> project. That should work well enough.
>>>
>>> On Thu, Sep 20, 2012 at 7:44 PM, Alberto Cordioli
>>> <co...@gmail.com> wrote:
>>>> Hi all,
>>>>
>>>> Now I'd like to deploy a simple MapReduce job, written in Java, to a
>>>> remote cluster within Eclipse.
>>>> For the moment I've found this solution:
>>>>
>>>> 1) Put the hadoop conf file in the classpath
>>>> 1) Put the jar containing the job in the classpath.
>>>> 2) Run
>>>>
>>>> If I don't put the jar in the classpath when I run the job it returns
>>>> a ClassNotFoundException:
>>>> java.lang.RuntimeException: java.lang.ClassNotFoundException:  <MyMapperClass>
>>>>
>>>> I was wondering if there are other methods to do so in simpler way.
>>>>
>>>>
>>>> Thank you very much,
>>>> Alberto
>>>
>>>
>>>
>>> --
>>> Harsh J
>>
>>
>>
>> --
>> Alberto Cordioli
>
>
>
> --
> Harsh J



-- 
Alberto Cordioli

Re: MapReduce jobs from remote

[Alberto Cordioli <co...@gmail.com>]

But, the call setJarByClass(Class) does not build the jar itself, right?
The compiled jar should be in any way in the classpath.

Actually, I've tried adding this command, but I still get
ClassNotFoundException if I don't put the jar in the run classpath.


Alberto


On 21 September 2012 12:39, Harsh J <ha...@cloudera.com> wrote:
> Hi Alberto,
>
> When doing a "hadoop jar <jar>" you'll of course need a compiled jar, yes.
>
> When running from Eclipse the job class files are on your classpath
> already. If you use the job.setJarByClass(Driver.class); call in your
> submitter, it will ensure it can pack up all job-required jars
> properly and ship it as the job jar when you call submit()/runJob().
> Is this something you're missing to do?
>
> On Fri, Sep 21, 2012 at 1:00 PM, Alberto Cordioli
> <co...@gmail.com> wrote:
>> Bertrand, I read about the plugin, but it seems to work only with
>> Hadoop 0.20.2 and I am currently working with Hadoop 1.0.3
>>
>> Harsh, it's not really clear to me why you're saying that it's enough
>> to have the config file in the classpath.
>> Also the jar is required, right? Otherwise I get
>> ClassNotFoundException. Am I missing something?
>>
>> Thank you very much,
>> Alberto
>>
>>
>> On 20 September 2012 17:18, Harsh J <ha...@cloudera.com> wrote:
>>> You have already found the simplest way of doing this, I think. The
>>> other way may be to use Oozie if your job jars don't change much and
>>> can be staged directly into HDFS for ready submit-when-required.
>>>
>>> However, if you wanna run from Eclipse, you just need the config file
>>> resources on the classpath of the run configuration of your job
>>> project. That should work well enough.
>>>
>>> On Thu, Sep 20, 2012 at 7:44 PM, Alberto Cordioli
>>> <co...@gmail.com> wrote:
>>>> Hi all,
>>>>
>>>> Now I'd like to deploy a simple MapReduce job, written in Java, to a
>>>> remote cluster within Eclipse.
>>>> For the moment I've found this solution:
>>>>
>>>> 1) Put the hadoop conf file in the classpath
>>>> 1) Put the jar containing the job in the classpath.
>>>> 2) Run
>>>>
>>>> If I don't put the jar in the classpath when I run the job it returns
>>>> a ClassNotFoundException:
>>>> java.lang.RuntimeException: java.lang.ClassNotFoundException:  <MyMapperClass>
>>>>
>>>> I was wondering if there are other methods to do so in simpler way.
>>>>
>>>>
>>>> Thank you very much,
>>>> Alberto
>>>
>>>
>>>
>>> --
>>> Harsh J
>>
>>
>>
>> --
>> Alberto Cordioli
>
>
>
> --
> Harsh J



-- 
Alberto Cordioli

Re: MapReduce jobs from remote

[Alberto Cordioli <co...@gmail.com>]

But, the call setJarByClass(Class) does not build the jar itself, right?
The compiled jar should be in any way in the classpath.

Actually, I've tried adding this command, but I still get
ClassNotFoundException if I don't put the jar in the run classpath.


Alberto


On 21 September 2012 12:39, Harsh J <ha...@cloudera.com> wrote:
> Hi Alberto,
>
> When doing a "hadoop jar <jar>" you'll of course need a compiled jar, yes.
>
> When running from Eclipse the job class files are on your classpath
> already. If you use the job.setJarByClass(Driver.class); call in your
> submitter, it will ensure it can pack up all job-required jars
> properly and ship it as the job jar when you call submit()/runJob().
> Is this something you're missing to do?
>
> On Fri, Sep 21, 2012 at 1:00 PM, Alberto Cordioli
> <co...@gmail.com> wrote:
>> Bertrand, I read about the plugin, but it seems to work only with
>> Hadoop 0.20.2 and I am currently working with Hadoop 1.0.3
>>
>> Harsh, it's not really clear to me why you're saying that it's enough
>> to have the config file in the classpath.
>> Also the jar is required, right? Otherwise I get
>> ClassNotFoundException. Am I missing something?
>>
>> Thank you very much,
>> Alberto
>>
>>
>> On 20 September 2012 17:18, Harsh J <ha...@cloudera.com> wrote:
>>> You have already found the simplest way of doing this, I think. The
>>> other way may be to use Oozie if your job jars don't change much and
>>> can be staged directly into HDFS for ready submit-when-required.
>>>
>>> However, if you wanna run from Eclipse, you just need the config file
>>> resources on the classpath of the run configuration of your job
>>> project. That should work well enough.
>>>
>>> On Thu, Sep 20, 2012 at 7:44 PM, Alberto Cordioli
>>> <co...@gmail.com> wrote:
>>>> Hi all,
>>>>
>>>> Now I'd like to deploy a simple MapReduce job, written in Java, to a
>>>> remote cluster within Eclipse.
>>>> For the moment I've found this solution:
>>>>
>>>> 1) Put the hadoop conf file in the classpath
>>>> 1) Put the jar containing the job in the classpath.
>>>> 2) Run
>>>>
>>>> If I don't put the jar in the classpath when I run the job it returns
>>>> a ClassNotFoundException:
>>>> java.lang.RuntimeException: java.lang.ClassNotFoundException:  <MyMapperClass>
>>>>
>>>> I was wondering if there are other methods to do so in simpler way.
>>>>
>>>>
>>>> Thank you very much,
>>>> Alberto
>>>
>>>
>>>
>>> --
>>> Harsh J
>>
>>
>>
>> --
>> Alberto Cordioli
>
>
>
> --
> Harsh J



-- 
Alberto Cordioli

Re: MapReduce jobs from remote

[Alberto Cordioli <co...@gmail.com>]

But, the call setJarByClass(Class) does not build the jar itself, right?
The compiled jar should be in any way in the classpath.

Actually, I've tried adding this command, but I still get
ClassNotFoundException if I don't put the jar in the run classpath.


Alberto


On 21 September 2012 12:39, Harsh J <ha...@cloudera.com> wrote:
> Hi Alberto,
>
> When doing a "hadoop jar <jar>" you'll of course need a compiled jar, yes.
>
> When running from Eclipse the job class files are on your classpath
> already. If you use the job.setJarByClass(Driver.class); call in your
> submitter, it will ensure it can pack up all job-required jars
> properly and ship it as the job jar when you call submit()/runJob().
> Is this something you're missing to do?
>
> On Fri, Sep 21, 2012 at 1:00 PM, Alberto Cordioli
> <co...@gmail.com> wrote:
>> Bertrand, I read about the plugin, but it seems to work only with
>> Hadoop 0.20.2 and I am currently working with Hadoop 1.0.3
>>
>> Harsh, it's not really clear to me why you're saying that it's enough
>> to have the config file in the classpath.
>> Also the jar is required, right? Otherwise I get
>> ClassNotFoundException. Am I missing something?
>>
>> Thank you very much,
>> Alberto
>>
>>
>> On 20 September 2012 17:18, Harsh J <ha...@cloudera.com> wrote:
>>> You have already found the simplest way of doing this, I think. The
>>> other way may be to use Oozie if your job jars don't change much and
>>> can be staged directly into HDFS for ready submit-when-required.
>>>
>>> However, if you wanna run from Eclipse, you just need the config file
>>> resources on the classpath of the run configuration of your job
>>> project. That should work well enough.
>>>
>>> On Thu, Sep 20, 2012 at 7:44 PM, Alberto Cordioli
>>> <co...@gmail.com> wrote:
>>>> Hi all,
>>>>
>>>> Now I'd like to deploy a simple MapReduce job, written in Java, to a
>>>> remote cluster within Eclipse.
>>>> For the moment I've found this solution:
>>>>
>>>> 1) Put the hadoop conf file in the classpath
>>>> 1) Put the jar containing the job in the classpath.
>>>> 2) Run
>>>>
>>>> If I don't put the jar in the classpath when I run the job it returns
>>>> a ClassNotFoundException:
>>>> java.lang.RuntimeException: java.lang.ClassNotFoundException:  <MyMapperClass>
>>>>
>>>> I was wondering if there are other methods to do so in simpler way.
>>>>
>>>>
>>>> Thank you very much,
>>>> Alberto
>>>
>>>
>>>
>>> --
>>> Harsh J
>>
>>
>>
>> --
>> Alberto Cordioli
>
>
>
> --
> Harsh J



-- 
Alberto Cordioli

Re: MapReduce jobs from remote

[Harsh J <ha...@cloudera.com>]

Hi Alberto,

When doing a "hadoop jar <jar>" you'll of course need a compiled jar, yes.

When running from Eclipse the job class files are on your classpath
already. If you use the job.setJarByClass(Driver.class); call in your
submitter, it will ensure it can pack up all job-required jars
properly and ship it as the job jar when you call submit()/runJob().
Is this something you're missing to do?

On Fri, Sep 21, 2012 at 1:00 PM, Alberto Cordioli
<co...@gmail.com> wrote:
> Bertrand, I read about the plugin, but it seems to work only with
> Hadoop 0.20.2 and I am currently working with Hadoop 1.0.3
>
> Harsh, it's not really clear to me why you're saying that it's enough
> to have the config file in the classpath.
> Also the jar is required, right? Otherwise I get
> ClassNotFoundException. Am I missing something?
>
> Thank you very much,
> Alberto
>
>
> On 20 September 2012 17:18, Harsh J <ha...@cloudera.com> wrote:
>> You have already found the simplest way of doing this, I think. The
>> other way may be to use Oozie if your job jars don't change much and
>> can be staged directly into HDFS for ready submit-when-required.
>>
>> However, if you wanna run from Eclipse, you just need the config file
>> resources on the classpath of the run configuration of your job
>> project. That should work well enough.
>>
>> On Thu, Sep 20, 2012 at 7:44 PM, Alberto Cordioli
>> <co...@gmail.com> wrote:
>>> Hi all,
>>>
>>> Now I'd like to deploy a simple MapReduce job, written in Java, to a
>>> remote cluster within Eclipse.
>>> For the moment I've found this solution:
>>>
>>> 1) Put the hadoop conf file in the classpath
>>> 1) Put the jar containing the job in the classpath.
>>> 2) Run
>>>
>>> If I don't put the jar in the classpath when I run the job it returns
>>> a ClassNotFoundException:
>>> java.lang.RuntimeException: java.lang.ClassNotFoundException:  <MyMapperClass>
>>>
>>> I was wondering if there are other methods to do so in simpler way.
>>>
>>>
>>> Thank you very much,
>>> Alberto
>>
>>
>>
>> --
>> Harsh J
>
>
>
> --
> Alberto Cordioli



-- 
Harsh J

Re: MapReduce jobs from remote

[Dino Kečo <di...@gmail.com>]

Hi Alberto,

I was doing same thing on older version of Hadoop (0.20) and i had same
issue.

The solution was to add *mapred*.*jar **property in configuration with
location of JAR which will be distributed to cluster. I am not sure does
this work in new versions.*

Regards,
Dino Kečo
msn: xdinno@hotmail.com
mail: dino.keco@gmail.com
skype: dino.keco
phone: +387 61 507 851


On Fri, Sep 21, 2012 at 9:30 AM, Alberto Cordioli <
cordioli.alberto@gmail.com> wrote:

> Bertrand, I read about the plugin, but it seems to work only with
> Hadoop 0.20.2 and I am currently working with Hadoop 1.0.3
>
> Harsh, it's not really clear to me why you're saying that it's enough
> to have the config file in the classpath.
> Also the jar is required, right? Otherwise I get
> ClassNotFoundException. Am I missing something?
>
> Thank you very much,
> Alberto
>
>
> On 20 September 2012 17:18, Harsh J <ha...@cloudera.com> wrote:
> > You have already found the simplest way of doing this, I think. The
> > other way may be to use Oozie if your job jars don't change much and
> > can be staged directly into HDFS for ready submit-when-required.
> >
> > However, if you wanna run from Eclipse, you just need the config file
> > resources on the classpath of the run configuration of your job
> > project. That should work well enough.
> >
> > On Thu, Sep 20, 2012 at 7:44 PM, Alberto Cordioli
> > <co...@gmail.com> wrote:
> >> Hi all,
> >>
> >> Now I'd like to deploy a simple MapReduce job, written in Java, to a
> >> remote cluster within Eclipse.
> >> For the moment I've found this solution:
> >>
> >> 1) Put the hadoop conf file in the classpath
> >> 1) Put the jar containing the job in the classpath.
> >> 2) Run
> >>
> >> If I don't put the jar in the classpath when I run the job it returns
> >> a ClassNotFoundException:
> >> java.lang.RuntimeException: java.lang.ClassNotFoundException:
>  <MyMapperClass>
> >>
> >> I was wondering if there are other methods to do so in simpler way.
> >>
> >>
> >> Thank you very much,
> >> Alberto
> >
> >
> >
> > --
> > Harsh J
>
>
>
> --
> Alberto Cordioli
>

Re: MapReduce jobs from remote

[Dino Kečo <di...@gmail.com>]

Hi Alberto,

I was doing same thing on older version of Hadoop (0.20) and i had same
issue.

The solution was to add *mapred*.*jar **property in configuration with
location of JAR which will be distributed to cluster. I am not sure does
this work in new versions.*

Regards,
Dino Kečo
msn: xdinno@hotmail.com
mail: dino.keco@gmail.com
skype: dino.keco
phone: +387 61 507 851


On Fri, Sep 21, 2012 at 9:30 AM, Alberto Cordioli <
cordioli.alberto@gmail.com> wrote:

> Bertrand, I read about the plugin, but it seems to work only with
> Hadoop 0.20.2 and I am currently working with Hadoop 1.0.3
>
> Harsh, it's not really clear to me why you're saying that it's enough
> to have the config file in the classpath.
> Also the jar is required, right? Otherwise I get
> ClassNotFoundException. Am I missing something?
>
> Thank you very much,
> Alberto
>
>
> On 20 September 2012 17:18, Harsh J <ha...@cloudera.com> wrote:
> > You have already found the simplest way of doing this, I think. The
> > other way may be to use Oozie if your job jars don't change much and
> > can be staged directly into HDFS for ready submit-when-required.
> >
> > However, if you wanna run from Eclipse, you just need the config file
> > resources on the classpath of the run configuration of your job
> > project. That should work well enough.
> >
> > On Thu, Sep 20, 2012 at 7:44 PM, Alberto Cordioli
> > <co...@gmail.com> wrote:
> >> Hi all,
> >>
> >> Now I'd like to deploy a simple MapReduce job, written in Java, to a
> >> remote cluster within Eclipse.
> >> For the moment I've found this solution:
> >>
> >> 1) Put the hadoop conf file in the classpath
> >> 1) Put the jar containing the job in the classpath.
> >> 2) Run
> >>
> >> If I don't put the jar in the classpath when I run the job it returns
> >> a ClassNotFoundException:
> >> java.lang.RuntimeException: java.lang.ClassNotFoundException:
>  <MyMapperClass>
> >>
> >> I was wondering if there are other methods to do so in simpler way.
> >>
> >>
> >> Thank you very much,
> >> Alberto
> >
> >
> >
> > --
> > Harsh J
>
>
>
> --
> Alberto Cordioli
>

Re: MapReduce jobs from remote

[Harsh J <ha...@cloudera.com>]

Hi Alberto,

When doing a "hadoop jar <jar>" you'll of course need a compiled jar, yes.

When running from Eclipse the job class files are on your classpath
already. If you use the job.setJarByClass(Driver.class); call in your
submitter, it will ensure it can pack up all job-required jars
properly and ship it as the job jar when you call submit()/runJob().
Is this something you're missing to do?

On Fri, Sep 21, 2012 at 1:00 PM, Alberto Cordioli
<co...@gmail.com> wrote:
> Bertrand, I read about the plugin, but it seems to work only with
> Hadoop 0.20.2 and I am currently working with Hadoop 1.0.3
>
> Harsh, it's not really clear to me why you're saying that it's enough
> to have the config file in the classpath.
> Also the jar is required, right? Otherwise I get
> ClassNotFoundException. Am I missing something?
>
> Thank you very much,
> Alberto
>
>
> On 20 September 2012 17:18, Harsh J <ha...@cloudera.com> wrote:
>> You have already found the simplest way of doing this, I think. The
>> other way may be to use Oozie if your job jars don't change much and
>> can be staged directly into HDFS for ready submit-when-required.
>>
>> However, if you wanna run from Eclipse, you just need the config file
>> resources on the classpath of the run configuration of your job
>> project. That should work well enough.
>>
>> On Thu, Sep 20, 2012 at 7:44 PM, Alberto Cordioli
>> <co...@gmail.com> wrote:
>>> Hi all,
>>>
>>> Now I'd like to deploy a simple MapReduce job, written in Java, to a
>>> remote cluster within Eclipse.
>>> For the moment I've found this solution:
>>>
>>> 1) Put the hadoop conf file in the classpath
>>> 1) Put the jar containing the job in the classpath.
>>> 2) Run
>>>
>>> If I don't put the jar in the classpath when I run the job it returns
>>> a ClassNotFoundException:
>>> java.lang.RuntimeException: java.lang.ClassNotFoundException:  <MyMapperClass>
>>>
>>> I was wondering if there are other methods to do so in simpler way.
>>>
>>>
>>> Thank you very much,
>>> Alberto
>>
>>
>>
>> --
>> Harsh J
>
>
>
> --
> Alberto Cordioli



-- 
Harsh J

Re: MapReduce jobs from remote

[Dino Kečo <di...@gmail.com>]

Hi Alberto,

I was doing same thing on older version of Hadoop (0.20) and i had same
issue.

The solution was to add *mapred*.*jar **property in configuration with
location of JAR which will be distributed to cluster. I am not sure does
this work in new versions.*

Regards,
Dino Kečo
msn: xdinno@hotmail.com
mail: dino.keco@gmail.com
skype: dino.keco
phone: +387 61 507 851


On Fri, Sep 21, 2012 at 9:30 AM, Alberto Cordioli <
cordioli.alberto@gmail.com> wrote:

> Bertrand, I read about the plugin, but it seems to work only with
> Hadoop 0.20.2 and I am currently working with Hadoop 1.0.3
>
> Harsh, it's not really clear to me why you're saying that it's enough
> to have the config file in the classpath.
> Also the jar is required, right? Otherwise I get
> ClassNotFoundException. Am I missing something?
>
> Thank you very much,
> Alberto
>
>
> On 20 September 2012 17:18, Harsh J <ha...@cloudera.com> wrote:
> > You have already found the simplest way of doing this, I think. The
> > other way may be to use Oozie if your job jars don't change much and
> > can be staged directly into HDFS for ready submit-when-required.
> >
> > However, if you wanna run from Eclipse, you just need the config file
> > resources on the classpath of the run configuration of your job
> > project. That should work well enough.
> >
> > On Thu, Sep 20, 2012 at 7:44 PM, Alberto Cordioli
> > <co...@gmail.com> wrote:
> >> Hi all,
> >>
> >> Now I'd like to deploy a simple MapReduce job, written in Java, to a
> >> remote cluster within Eclipse.
> >> For the moment I've found this solution:
> >>
> >> 1) Put the hadoop conf file in the classpath
> >> 1) Put the jar containing the job in the classpath.
> >> 2) Run
> >>
> >> If I don't put the jar in the classpath when I run the job it returns
> >> a ClassNotFoundException:
> >> java.lang.RuntimeException: java.lang.ClassNotFoundException:
>  <MyMapperClass>
> >>
> >> I was wondering if there are other methods to do so in simpler way.
> >>
> >>
> >> Thank you very much,
> >> Alberto
> >
> >
> >
> > --
> > Harsh J
>
>
>
> --
> Alberto Cordioli
>

Re: MapReduce jobs from remote

[Dino Kečo <di...@gmail.com>]

Hi Alberto,

I was doing same thing on older version of Hadoop (0.20) and i had same
issue.

The solution was to add *mapred*.*jar **property in configuration with
location of JAR which will be distributed to cluster. I am not sure does
this work in new versions.*

Regards,
Dino Kečo
msn: xdinno@hotmail.com
mail: dino.keco@gmail.com
skype: dino.keco
phone: +387 61 507 851


On Fri, Sep 21, 2012 at 9:30 AM, Alberto Cordioli <
cordioli.alberto@gmail.com> wrote:

> Bertrand, I read about the plugin, but it seems to work only with
> Hadoop 0.20.2 and I am currently working with Hadoop 1.0.3
>
> Harsh, it's not really clear to me why you're saying that it's enough
> to have the config file in the classpath.
> Also the jar is required, right? Otherwise I get
> ClassNotFoundException. Am I missing something?
>
> Thank you very much,
> Alberto
>
>
> On 20 September 2012 17:18, Harsh J <ha...@cloudera.com> wrote:
> > You have already found the simplest way of doing this, I think. The
> > other way may be to use Oozie if your job jars don't change much and
> > can be staged directly into HDFS for ready submit-when-required.
> >
> > However, if you wanna run from Eclipse, you just need the config file
> > resources on the classpath of the run configuration of your job
> > project. That should work well enough.
> >
> > On Thu, Sep 20, 2012 at 7:44 PM, Alberto Cordioli
> > <co...@gmail.com> wrote:
> >> Hi all,
> >>
> >> Now I'd like to deploy a simple MapReduce job, written in Java, to a
> >> remote cluster within Eclipse.
> >> For the moment I've found this solution:
> >>
> >> 1) Put the hadoop conf file in the classpath
> >> 1) Put the jar containing the job in the classpath.
> >> 2) Run
> >>
> >> If I don't put the jar in the classpath when I run the job it returns
> >> a ClassNotFoundException:
> >> java.lang.RuntimeException: java.lang.ClassNotFoundException:
>  <MyMapperClass>
> >>
> >> I was wondering if there are other methods to do so in simpler way.
> >>
> >>
> >> Thank you very much,
> >> Alberto
> >
> >
> >
> > --
> > Harsh J
>
>
>
> --
> Alberto Cordioli
>

Re: MapReduce jobs from remote

[Harsh J <ha...@cloudera.com>]

Hi Alberto,

When doing a "hadoop jar <jar>" you'll of course need a compiled jar, yes.

When running from Eclipse the job class files are on your classpath
already. If you use the job.setJarByClass(Driver.class); call in your
submitter, it will ensure it can pack up all job-required jars
properly and ship it as the job jar when you call submit()/runJob().
Is this something you're missing to do?

On Fri, Sep 21, 2012 at 1:00 PM, Alberto Cordioli
<co...@gmail.com> wrote:
> Bertrand, I read about the plugin, but it seems to work only with
> Hadoop 0.20.2 and I am currently working with Hadoop 1.0.3
>
> Harsh, it's not really clear to me why you're saying that it's enough
> to have the config file in the classpath.
> Also the jar is required, right? Otherwise I get
> ClassNotFoundException. Am I missing something?
>
> Thank you very much,
> Alberto
>
>
> On 20 September 2012 17:18, Harsh J <ha...@cloudera.com> wrote:
>> You have already found the simplest way of doing this, I think. The
>> other way may be to use Oozie if your job jars don't change much and
>> can be staged directly into HDFS for ready submit-when-required.
>>
>> However, if you wanna run from Eclipse, you just need the config file
>> resources on the classpath of the run configuration of your job
>> project. That should work well enough.
>>
>> On Thu, Sep 20, 2012 at 7:44 PM, Alberto Cordioli
>> <co...@gmail.com> wrote:
>>> Hi all,
>>>
>>> Now I'd like to deploy a simple MapReduce job, written in Java, to a
>>> remote cluster within Eclipse.
>>> For the moment I've found this solution:
>>>
>>> 1) Put the hadoop conf file in the classpath
>>> 1) Put the jar containing the job in the classpath.
>>> 2) Run
>>>
>>> If I don't put the jar in the classpath when I run the job it returns
>>> a ClassNotFoundException:
>>> java.lang.RuntimeException: java.lang.ClassNotFoundException:  <MyMapperClass>
>>>
>>> I was wondering if there are other methods to do so in simpler way.
>>>
>>>
>>> Thank you very much,
>>> Alberto
>>
>>
>>
>> --
>> Harsh J
>
>
>
> --
> Alberto Cordioli



-- 
Harsh J

Re: MapReduce jobs from remote

[Harsh J <ha...@cloudera.com>]

Hi Alberto,

When doing a "hadoop jar <jar>" you'll of course need a compiled jar, yes.

When running from Eclipse the job class files are on your classpath
already. If you use the job.setJarByClass(Driver.class); call in your
submitter, it will ensure it can pack up all job-required jars
properly and ship it as the job jar when you call submit()/runJob().
Is this something you're missing to do?

On Fri, Sep 21, 2012 at 1:00 PM, Alberto Cordioli
<co...@gmail.com> wrote:
> Bertrand, I read about the plugin, but it seems to work only with
> Hadoop 0.20.2 and I am currently working with Hadoop 1.0.3
>
> Harsh, it's not really clear to me why you're saying that it's enough
> to have the config file in the classpath.
> Also the jar is required, right? Otherwise I get
> ClassNotFoundException. Am I missing something?
>
> Thank you very much,
> Alberto
>
>
> On 20 September 2012 17:18, Harsh J <ha...@cloudera.com> wrote:
>> You have already found the simplest way of doing this, I think. The
>> other way may be to use Oozie if your job jars don't change much and
>> can be staged directly into HDFS for ready submit-when-required.
>>
>> However, if you wanna run from Eclipse, you just need the config file
>> resources on the classpath of the run configuration of your job
>> project. That should work well enough.
>>
>> On Thu, Sep 20, 2012 at 7:44 PM, Alberto Cordioli
>> <co...@gmail.com> wrote:
>>> Hi all,
>>>
>>> Now I'd like to deploy a simple MapReduce job, written in Java, to a
>>> remote cluster within Eclipse.
>>> For the moment I've found this solution:
>>>
>>> 1) Put the hadoop conf file in the classpath
>>> 1) Put the jar containing the job in the classpath.
>>> 2) Run
>>>
>>> If I don't put the jar in the classpath when I run the job it returns
>>> a ClassNotFoundException:
>>> java.lang.RuntimeException: java.lang.ClassNotFoundException:  <MyMapperClass>
>>>
>>> I was wondering if there are other methods to do so in simpler way.
>>>
>>>
>>> Thank you very much,
>>> Alberto
>>
>>
>>
>> --
>> Harsh J
>
>
>
> --
> Alberto Cordioli



-- 
Harsh J

Re: MapReduce jobs from remote

[Alberto Cordioli <co...@gmail.com>]

Bertrand, I read about the plugin, but it seems to work only with
Hadoop 0.20.2 and I am currently working with Hadoop 1.0.3

Harsh, it's not really clear to me why you're saying that it's enough
to have the config file in the classpath.
Also the jar is required, right? Otherwise I get
ClassNotFoundException. Am I missing something?

Thank you very much,
Alberto


On 20 September 2012 17:18, Harsh J <ha...@cloudera.com> wrote:
> You have already found the simplest way of doing this, I think. The
> other way may be to use Oozie if your job jars don't change much and
> can be staged directly into HDFS for ready submit-when-required.
>
> However, if you wanna run from Eclipse, you just need the config file
> resources on the classpath of the run configuration of your job
> project. That should work well enough.
>
> On Thu, Sep 20, 2012 at 7:44 PM, Alberto Cordioli
> <co...@gmail.com> wrote:
>> Hi all,
>>
>> Now I'd like to deploy a simple MapReduce job, written in Java, to a
>> remote cluster within Eclipse.
>> For the moment I've found this solution:
>>
>> 1) Put the hadoop conf file in the classpath
>> 1) Put the jar containing the job in the classpath.
>> 2) Run
>>
>> If I don't put the jar in the classpath when I run the job it returns
>> a ClassNotFoundException:
>> java.lang.RuntimeException: java.lang.ClassNotFoundException:  <MyMapperClass>
>>
>> I was wondering if there are other methods to do so in simpler way.
>>
>>
>> Thank you very much,
>> Alberto
>
>
>
> --
> Harsh J



-- 
Alberto Cordioli

Re: MapReduce jobs from remote

[Alberto Cordioli <co...@gmail.com>]

Bertrand, I read about the plugin, but it seems to work only with
Hadoop 0.20.2 and I am currently working with Hadoop 1.0.3

Harsh, it's not really clear to me why you're saying that it's enough
to have the config file in the classpath.
Also the jar is required, right? Otherwise I get
ClassNotFoundException. Am I missing something?

Thank you very much,
Alberto


On 20 September 2012 17:18, Harsh J <ha...@cloudera.com> wrote:
> You have already found the simplest way of doing this, I think. The
> other way may be to use Oozie if your job jars don't change much and
> can be staged directly into HDFS for ready submit-when-required.
>
> However, if you wanna run from Eclipse, you just need the config file
> resources on the classpath of the run configuration of your job
> project. That should work well enough.
>
> On Thu, Sep 20, 2012 at 7:44 PM, Alberto Cordioli
> <co...@gmail.com> wrote:
>> Hi all,
>>
>> Now I'd like to deploy a simple MapReduce job, written in Java, to a
>> remote cluster within Eclipse.
>> For the moment I've found this solution:
>>
>> 1) Put the hadoop conf file in the classpath
>> 1) Put the jar containing the job in the classpath.
>> 2) Run
>>
>> If I don't put the jar in the classpath when I run the job it returns
>> a ClassNotFoundException:
>> java.lang.RuntimeException: java.lang.ClassNotFoundException:  <MyMapperClass>
>>
>> I was wondering if there are other methods to do so in simpler way.
>>
>>
>> Thank you very much,
>> Alberto
>
>
>
> --
> Harsh J



-- 
Alberto Cordioli

Re: MapReduce jobs from remote

[Alberto Cordioli <co...@gmail.com>]

Bertrand, I read about the plugin, but it seems to work only with
Hadoop 0.20.2 and I am currently working with Hadoop 1.0.3

Harsh, it's not really clear to me why you're saying that it's enough
to have the config file in the classpath.
Also the jar is required, right? Otherwise I get
ClassNotFoundException. Am I missing something?

Thank you very much,
Alberto


On 20 September 2012 17:18, Harsh J <ha...@cloudera.com> wrote:
> You have already found the simplest way of doing this, I think. The
> other way may be to use Oozie if your job jars don't change much and
> can be staged directly into HDFS for ready submit-when-required.
>
> However, if you wanna run from Eclipse, you just need the config file
> resources on the classpath of the run configuration of your job
> project. That should work well enough.
>
> On Thu, Sep 20, 2012 at 7:44 PM, Alberto Cordioli
> <co...@gmail.com> wrote:
>> Hi all,
>>
>> Now I'd like to deploy a simple MapReduce job, written in Java, to a
>> remote cluster within Eclipse.
>> For the moment I've found this solution:
>>
>> 1) Put the hadoop conf file in the classpath
>> 1) Put the jar containing the job in the classpath.
>> 2) Run
>>
>> If I don't put the jar in the classpath when I run the job it returns
>> a ClassNotFoundException:
>> java.lang.RuntimeException: java.lang.ClassNotFoundException:  <MyMapperClass>
>>
>> I was wondering if there are other methods to do so in simpler way.
>>
>>
>> Thank you very much,
>> Alberto
>
>
>
> --
> Harsh J



-- 
Alberto Cordioli

Re: MapReduce jobs from remote

[Alberto Cordioli <co...@gmail.com>]

Bertrand, I read about the plugin, but it seems to work only with
Hadoop 0.20.2 and I am currently working with Hadoop 1.0.3

Harsh, it's not really clear to me why you're saying that it's enough
to have the config file in the classpath.
Also the jar is required, right? Otherwise I get
ClassNotFoundException. Am I missing something?

Thank you very much,
Alberto


On 20 September 2012 17:18, Harsh J <ha...@cloudera.com> wrote:
> You have already found the simplest way of doing this, I think. The
> other way may be to use Oozie if your job jars don't change much and
> can be staged directly into HDFS for ready submit-when-required.
>
> However, if you wanna run from Eclipse, you just need the config file
> resources on the classpath of the run configuration of your job
> project. That should work well enough.
>
> On Thu, Sep 20, 2012 at 7:44 PM, Alberto Cordioli
> <co...@gmail.com> wrote:
>> Hi all,
>>
>> Now I'd like to deploy a simple MapReduce job, written in Java, to a
>> remote cluster within Eclipse.
>> For the moment I've found this solution:
>>
>> 1) Put the hadoop conf file in the classpath
>> 1) Put the jar containing the job in the classpath.
>> 2) Run
>>
>> If I don't put the jar in the classpath when I run the job it returns
>> a ClassNotFoundException:
>> java.lang.RuntimeException: java.lang.ClassNotFoundException:  <MyMapperClass>
>>
>> I was wondering if there are other methods to do so in simpler way.
>>
>>
>> Thank you very much,
>> Alberto
>
>
>
> --
> Harsh J



-- 
Alberto Cordioli

Re: MapReduce jobs from remote

[Harsh J <ha...@cloudera.com>]

You have already found the simplest way of doing this, I think. The
other way may be to use Oozie if your job jars don't change much and
can be staged directly into HDFS for ready submit-when-required.

However, if you wanna run from Eclipse, you just need the config file
resources on the classpath of the run configuration of your job
project. That should work well enough.

On Thu, Sep 20, 2012 at 7:44 PM, Alberto Cordioli
<co...@gmail.com> wrote:
> Hi all,
>
> Now I'd like to deploy a simple MapReduce job, written in Java, to a
> remote cluster within Eclipse.
> For the moment I've found this solution:
>
> 1) Put the hadoop conf file in the classpath
> 1) Put the jar containing the job in the classpath.
> 2) Run
>
> If I don't put the jar in the classpath when I run the job it returns
> a ClassNotFoundException:
> java.lang.RuntimeException: java.lang.ClassNotFoundException:  <MyMapperClass>
>
> I was wondering if there are other methods to do so in simpler way.
>
>
> Thank you very much,
> Alberto



-- 
Harsh J

Re: MapReduce jobs from remote

[Harsh J <ha...@cloudera.com>]

You have already found the simplest way of doing this, I think. The
other way may be to use Oozie if your job jars don't change much and
can be staged directly into HDFS for ready submit-when-required.

However, if you wanna run from Eclipse, you just need the config file
resources on the classpath of the run configuration of your job
project. That should work well enough.

On Thu, Sep 20, 2012 at 7:44 PM, Alberto Cordioli
<co...@gmail.com> wrote:
> Hi all,
>
> Now I'd like to deploy a simple MapReduce job, written in Java, to a
> remote cluster within Eclipse.
> For the moment I've found this solution:
>
> 1) Put the hadoop conf file in the classpath
> 1) Put the jar containing the job in the classpath.
> 2) Run
>
> If I don't put the jar in the classpath when I run the job it returns
> a ClassNotFoundException:
> java.lang.RuntimeException: java.lang.ClassNotFoundException:  <MyMapperClass>
>
> I was wondering if there are other methods to do so in simpler way.
>
>
> Thank you very much,
> Alberto



-- 
Harsh J

Re: MapReduce jobs from remote

[Harsh J <ha...@cloudera.com>]

You have already found the simplest way of doing this, I think. The
other way may be to use Oozie if your job jars don't change much and
can be staged directly into HDFS for ready submit-when-required.

However, if you wanna run from Eclipse, you just need the config file
resources on the classpath of the run configuration of your job
project. That should work well enough.

On Thu, Sep 20, 2012 at 7:44 PM, Alberto Cordioli
<co...@gmail.com> wrote:
> Hi all,
>
> Now I'd like to deploy a simple MapReduce job, written in Java, to a
> remote cluster within Eclipse.
> For the moment I've found this solution:
>
> 1) Put the hadoop conf file in the classpath
> 1) Put the jar containing the job in the classpath.
> 2) Run
>
> If I don't put the jar in the classpath when I run the job it returns
> a ClassNotFoundException:
> java.lang.RuntimeException: java.lang.ClassNotFoundException:  <MyMapperClass>
>
> I was wondering if there are other methods to do so in simpler way.
>
>
> Thank you very much,
> Alberto



-- 
Harsh J

Re: MapReduce jobs from remote

[Harsh J <ha...@cloudera.com>]

You have already found the simplest way of doing this, I think. The
other way may be to use Oozie if your job jars don't change much and
can be staged directly into HDFS for ready submit-when-required.

However, if you wanna run from Eclipse, you just need the config file
resources on the classpath of the run configuration of your job
project. That should work well enough.

On Thu, Sep 20, 2012 at 7:44 PM, Alberto Cordioli
<co...@gmail.com> wrote:
> Hi all,
>
> Now I'd like to deploy a simple MapReduce job, written in Java, to a
> remote cluster within Eclipse.
> For the moment I've found this solution:
>
> 1) Put the hadoop conf file in the classpath
> 1) Put the jar containing the job in the classpath.
> 2) Run
>
> If I don't put the jar in the classpath when I run the job it returns
> a ClassNotFoundException:
> java.lang.RuntimeException: java.lang.ClassNotFoundException:  <MyMapperClass>
>
> I was wondering if there are other methods to do so in simpler way.
>
>
> Thank you very much,
> Alberto



-- 
Harsh J

Re: MapReduce jobs from remote

[Bertrand Dechoux <de...@gmail.com>]

I haven't tried it but there a Hadoop plugin for eclipse :
http://wiki.apache.org/hadoop/EclipsePlugIn

Regards

Bertrand

On Thu, Sep 20, 2012 at 4:14 PM, Alberto Cordioli <
cordioli.alberto@gmail.com> wrote:

> Hi all,
>
> Now I'd like to deploy a simple MapReduce job, written in Java, to a
> remote cluster within Eclipse.
> For the moment I've found this solution:
>
> 1) Put the hadoop conf file in the classpath
> 1) Put the jar containing the job in the classpath.
> 2) Run
>
> If I don't put the jar in the classpath when I run the job it returns
> a ClassNotFoundException:
> java.lang.RuntimeException: java.lang.ClassNotFoundException:
>  <MyMapperClass>
>
> I was wondering if there are other methods to do so in simpler way.
>
>
> Thank you very much,
> Alberto
>



-- 
Bertrand Dechoux

Re: MapReduce jobs from remote

[Bertrand Dechoux <de...@gmail.com>]

I haven't tried it but there a Hadoop plugin for eclipse :
http://wiki.apache.org/hadoop/EclipsePlugIn

Regards

Bertrand

On Thu, Sep 20, 2012 at 4:14 PM, Alberto Cordioli <
cordioli.alberto@gmail.com> wrote:

> Hi all,
>
> Now I'd like to deploy a simple MapReduce job, written in Java, to a
> remote cluster within Eclipse.
> For the moment I've found this solution:
>
> 1) Put the hadoop conf file in the classpath
> 1) Put the jar containing the job in the classpath.
> 2) Run
>
> If I don't put the jar in the classpath when I run the job it returns
> a ClassNotFoundException:
> java.lang.RuntimeException: java.lang.ClassNotFoundException:
>  <MyMapperClass>
>
> I was wondering if there are other methods to do so in simpler way.
>
>
> Thank you very much,
> Alberto
>



-- 
Bertrand Dechoux