You are viewing a plain text version of this content. The canonical link for it is here.
Posted to general@hadoop.apache.org by Deepika Khera <De...@avg.com> on 2010/08/29 02:37:36 UTC
Deploying my job jar on hadoop cluster
Hi,
I want to deploy my map reduce job jar on the Hadoop cluster. I've always done that by doing the following -
1. Copying the job jar to all datanodes
2. Having the job jar on the hadoop classpath on all machines.
Isn't hadoop capable of copying over the job jar to all machines in the cluster? This is what I read (that job tracker copies the job jar, etc) , but if I don't do the above the task trackers cannot find the job. I know I am missing something.
Could someone please let me know how I can run my job without having to copy it over all machines in the cluster?
Thanks,
Deepika
Re: Deploying my job jar on hadoop cluster
Posted by Chandraprakash Bhagtani <cp...@gmail.com>.
check whether the user from which you are running the job has permissions on
the jar or the jar is properly generated i.e. the class is present in the
jar or not
On Sun, Aug 29, 2010 at 11:45 AM, Deepika Khera <De...@avg.com>wrote:
> Thanks. I had tried running the command as you suggested but got a
> ClassNotFoundException exception(for the mapper class) in the task trackers.
> I did a
>
> bin/hadoop jar /root/myjob.jar
>
> (Already had the main class defined in manifest)
>
> There is probably something else that I am doing wrong.
>
> Appreciate your help.
>
> Thanks,
> Deepika
>
> -----Original Message-----
> From: Chandraprakash Bhagtani [mailto:cpbhagtani@gmail.com]
> Sent: Saturday, August 28, 2010 10:40 PM
> To: general@hadoop.apache.org
> Subject: Re: Deploying my job jar on hadoop cluster
>
> Deepika,
>
> You just have to run the following command on any of the cluster node
>
> HADOOP_HOME/bin/hadoop jar <job_jar_name> <class_name> <other_args>
>
> this command will automatically copy the jar on all the tasktrackers.
>
> On Sun, Aug 29, 2010 at 6:07 AM, Deepika Khera <Deepika.Khera@avg.com
> >wrote:
>
> > Hi,
> >
> > I want to deploy my map reduce job jar on the Hadoop cluster. I've always
> > done that by doing the following -
> >
> > 1. Copying the job jar to all datanodes
> > 2. Having the job jar on the hadoop classpath on all machines.
> >
> > Isn't hadoop capable of copying over the job jar to all machines in the
> > cluster? This is what I read (that job tracker copies the job jar, etc) ,
> > but if I don't do the above the task trackers cannot find the job. I know
> I
> > am missing something.
> >
> > Could someone please let me know how I can run my job without having to
> > copy it over all machines in the cluster?
> >
> > Thanks,
> > Deepika
> >
>
>
>
> --
> Thanks & Regards,
> Chandra Prakash Bhagtani,
> Nokia India Pvt. Ltd.
>
--
Thanks & Regards,
Chandra Prakash Bhagtani,
Nokia India Pvt. Ltd.
RE: Deploying my job jar on hadoop cluster
Posted by Deepika Khera <De...@avg.com>.
Thanks! Added the line and it worked :)
-----Original Message-----
From: Angus He [mailto:angushe@gmail.com]
Sent: Monday, August 30, 2010 1:41 AM
To: general@hadoop.apache.org
Subject: Re: Deploying my job jar on hadoop cluster
Did the Job.setJarByClass get called? If not, just add it and take
another try.
On Sun, Aug 29, 2010 at 2:15 PM, Deepika Khera <De...@avg.com> wrote:
> Thanks. I had tried running the command as you suggested but got a ClassNotFoundException exception(for the mapper class) in the task trackers. I did a
>
> bin/hadoop jar /root/myjob.jar
>
> (Already had the main class defined in manifest)
>
> There is probably something else that I am doing wrong.
>
> Appreciate your help.
>
> Thanks,
> Deepika
>
> -----Original Message-----
> From: Chandraprakash Bhagtani [mailto:cpbhagtani@gmail.com]
> Sent: Saturday, August 28, 2010 10:40 PM
> To: general@hadoop.apache.org
> Subject: Re: Deploying my job jar on hadoop cluster
>
> Deepika,
>
> You just have to run the following command on any of the cluster node
>
> HADOOP_HOME/bin/hadoop jar <job_jar_name> <class_name> <other_args>
>
> this command will automatically copy the jar on all the tasktrackers.
>
> On Sun, Aug 29, 2010 at 6:07 AM, Deepika Khera <De...@avg.com>wrote:
>
>> Hi,
>>
>> I want to deploy my map reduce job jar on the Hadoop cluster. I've always
>> done that by doing the following -
>>
>> 1. Copying the job jar to all datanodes
>> 2. Having the job jar on the hadoop classpath on all machines.
>>
>> Isn't hadoop capable of copying over the job jar to all machines in the
>> cluster? This is what I read (that job tracker copies the job jar, etc) ,
>> but if I don't do the above the task trackers cannot find the job. I know I
>> am missing something.
>>
>> Could someone please let me know how I can run my job without having to
>> copy it over all machines in the cluster?
>>
>> Thanks,
>> Deepika
>>
>
>
>
> --
> Thanks & Regards,
> Chandra Prakash Bhagtani,
> Nokia India Pvt. Ltd.
>
--
Regards
Angus
Re: Deploying my job jar on hadoop cluster
Posted by Angus He <an...@gmail.com>.
Did the Job.setJarByClass get called? If not, just add it and take
another try.
On Sun, Aug 29, 2010 at 2:15 PM, Deepika Khera <De...@avg.com> wrote:
> Thanks. I had tried running the command as you suggested but got a ClassNotFoundException exception(for the mapper class) in the task trackers. I did a
>
> bin/hadoop jar /root/myjob.jar
>
> (Already had the main class defined in manifest)
>
> There is probably something else that I am doing wrong.
>
> Appreciate your help.
>
> Thanks,
> Deepika
>
> -----Original Message-----
> From: Chandraprakash Bhagtani [mailto:cpbhagtani@gmail.com]
> Sent: Saturday, August 28, 2010 10:40 PM
> To: general@hadoop.apache.org
> Subject: Re: Deploying my job jar on hadoop cluster
>
> Deepika,
>
> You just have to run the following command on any of the cluster node
>
> HADOOP_HOME/bin/hadoop jar <job_jar_name> <class_name> <other_args>
>
> this command will automatically copy the jar on all the tasktrackers.
>
> On Sun, Aug 29, 2010 at 6:07 AM, Deepika Khera <De...@avg.com>wrote:
>
>> Hi,
>>
>> I want to deploy my map reduce job jar on the Hadoop cluster. I've always
>> done that by doing the following -
>>
>> 1. Copying the job jar to all datanodes
>> 2. Having the job jar on the hadoop classpath on all machines.
>>
>> Isn't hadoop capable of copying over the job jar to all machines in the
>> cluster? This is what I read (that job tracker copies the job jar, etc) ,
>> but if I don't do the above the task trackers cannot find the job. I know I
>> am missing something.
>>
>> Could someone please let me know how I can run my job without having to
>> copy it over all machines in the cluster?
>>
>> Thanks,
>> Deepika
>>
>
>
>
> --
> Thanks & Regards,
> Chandra Prakash Bhagtani,
> Nokia India Pvt. Ltd.
>
--
Regards
Angus
RE: Deploying my job jar on hadoop cluster
Posted by Deepika Khera <De...@avg.com>.
Thanks. I had tried running the command as you suggested but got a ClassNotFoundException exception(for the mapper class) in the task trackers. I did a
bin/hadoop jar /root/myjob.jar
(Already had the main class defined in manifest)
There is probably something else that I am doing wrong.
Appreciate your help.
Thanks,
Deepika
-----Original Message-----
From: Chandraprakash Bhagtani [mailto:cpbhagtani@gmail.com]
Sent: Saturday, August 28, 2010 10:40 PM
To: general@hadoop.apache.org
Subject: Re: Deploying my job jar on hadoop cluster
Deepika,
You just have to run the following command on any of the cluster node
HADOOP_HOME/bin/hadoop jar <job_jar_name> <class_name> <other_args>
this command will automatically copy the jar on all the tasktrackers.
On Sun, Aug 29, 2010 at 6:07 AM, Deepika Khera <De...@avg.com>wrote:
> Hi,
>
> I want to deploy my map reduce job jar on the Hadoop cluster. I've always
> done that by doing the following -
>
> 1. Copying the job jar to all datanodes
> 2. Having the job jar on the hadoop classpath on all machines.
>
> Isn't hadoop capable of copying over the job jar to all machines in the
> cluster? This is what I read (that job tracker copies the job jar, etc) ,
> but if I don't do the above the task trackers cannot find the job. I know I
> am missing something.
>
> Could someone please let me know how I can run my job without having to
> copy it over all machines in the cluster?
>
> Thanks,
> Deepika
>
--
Thanks & Regards,
Chandra Prakash Bhagtani,
Nokia India Pvt. Ltd.
Re: Deploying my job jar on hadoop cluster
Posted by Chandraprakash Bhagtani <cp...@gmail.com>.
Deepika,
You just have to run the following command on any of the cluster node
HADOOP_HOME/bin/hadoop jar <job_jar_name> <class_name> <other_args>
this command will automatically copy the jar on all the tasktrackers.
On Sun, Aug 29, 2010 at 6:07 AM, Deepika Khera <De...@avg.com>wrote:
> Hi,
>
> I want to deploy my map reduce job jar on the Hadoop cluster. I've always
> done that by doing the following -
>
> 1. Copying the job jar to all datanodes
> 2. Having the job jar on the hadoop classpath on all machines.
>
> Isn't hadoop capable of copying over the job jar to all machines in the
> cluster? This is what I read (that job tracker copies the job jar, etc) ,
> but if I don't do the above the task trackers cannot find the job. I know I
> am missing something.
>
> Could someone please let me know how I can run my job without having to
> copy it over all machines in the cluster?
>
> Thanks,
> Deepika
>
--
Thanks & Regards,
Chandra Prakash Bhagtani,
Nokia India Pvt. Ltd.