You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by Shashidhar Rao <ra...@gmail.com> on 2013/05/11 20:24:22 UTC
Submitting a hadoop job in large clusters.
Hello users,
I am running hadoop in my windows laptop using linux virtual machine.Since
every Hadoop daemon is running in one machine I have no problem running
hadoop jar command and then submittng the job and everything works fine.
But, I think when hadoop is running in multiple nodes submitting a job
would be different. My understanding is that I have to remote connect to
one of the node ex. any datanode not necessary JobTracker and then copy the
job jar to a location, cd to hadoop bin and then fire up the console and
then run the hadoop jar command.
Is this how to submit a hadoop job in a cluster or is there any other way
to submit a job .
Basically, I have never used a cluster so am having this doubt
Advanced thanks
Re: Submitting a hadoop job in large clusters.
Posted by Bertrand Dechoux <de...@gmail.com>.
Which doesn't imply that you should log yourself to the physical machine
where the JobTracker is hosted. It only implies that the hadoop client must
be able to reach the JobTracker. It could be from any physical machines
hosting the slaves (DataNode, Tasktracker) but it is rarely the case.
Often, job are submitted from a machine which doesn't belong to the cluster
but can reach every machine of it.
Regards
Bertrand
On Sun, May 12, 2013 at 7:59 PM, shashwat shriparv <
dwivedishashwat@gmail.com> wrote:
> As nitin said , its responsibility of Jobtracker to distribute the job to
> task to the tasktrackers so you need to submitt the job to the job tracker
>
> *Thanks & Regards *
>
> ∞
> Shashwat Shriparv
>
>
>
> On Sun, May 12, 2013 at 11:26 PM, Nitin Pawar <ni...@gmail.com>wrote:
>
>> nope
>> in MRv1 only jobtracker can accept jobs. You can not trigger job on any
>> other process in hadoop other than jobtracker.
>>
>>
>> On Sun, May 12, 2013 at 11:25 PM, Shashidhar Rao <
>> raoshashidhar123@gmail.com> wrote:
>>
>>> @shashwat shriparv
>>>
>>> Can the a hadoop job be submitted to any datanode in the cluster and not
>>> to jobTracker.
>>>
>>> Correct me if it I am wrong , I was told that a hadoop job can be
>>> submitted to datanode also apart from JobTracker. Is it correct?
>>>
>>> Advanced thanks
>>>
>>>
>>> On Sun, May 12, 2013 at 11:02 PM, shashwat shriparv <
>>> dwivedishashwat@gmail.com> wrote:
>>>
>>>>
>>>> On Sun, May 12, 2013 at 12:19 AM, Nitin Pawar <ni...@gmail.com>wrote:
>>>>
>>>>>
>>>>> normally if you want to copy the jar then hadoop admins setu
>>>>>
>>>>
>>>> Submit you job to Job tracker it will distribute throughout the
>>>> tasktrackers.
>>>>
>>>> *Thanks & Regards *
>>>>
>>>> ∞
>>>> Shashwat Shriparv
>>>>
>>>>
>>>
>>
>>
>> --
>> Nitin Pawar
>>
>
>
Re: Submitting a hadoop job in large clusters.
Posted by Bertrand Dechoux <de...@gmail.com>.
Which doesn't imply that you should log yourself to the physical machine
where the JobTracker is hosted. It only implies that the hadoop client must
be able to reach the JobTracker. It could be from any physical machines
hosting the slaves (DataNode, Tasktracker) but it is rarely the case.
Often, job are submitted from a machine which doesn't belong to the cluster
but can reach every machine of it.
Regards
Bertrand
On Sun, May 12, 2013 at 7:59 PM, shashwat shriparv <
dwivedishashwat@gmail.com> wrote:
> As nitin said , its responsibility of Jobtracker to distribute the job to
> task to the tasktrackers so you need to submitt the job to the job tracker
>
> *Thanks & Regards *
>
> ∞
> Shashwat Shriparv
>
>
>
> On Sun, May 12, 2013 at 11:26 PM, Nitin Pawar <ni...@gmail.com>wrote:
>
>> nope
>> in MRv1 only jobtracker can accept jobs. You can not trigger job on any
>> other process in hadoop other than jobtracker.
>>
>>
>> On Sun, May 12, 2013 at 11:25 PM, Shashidhar Rao <
>> raoshashidhar123@gmail.com> wrote:
>>
>>> @shashwat shriparv
>>>
>>> Can the a hadoop job be submitted to any datanode in the cluster and not
>>> to jobTracker.
>>>
>>> Correct me if it I am wrong , I was told that a hadoop job can be
>>> submitted to datanode also apart from JobTracker. Is it correct?
>>>
>>> Advanced thanks
>>>
>>>
>>> On Sun, May 12, 2013 at 11:02 PM, shashwat shriparv <
>>> dwivedishashwat@gmail.com> wrote:
>>>
>>>>
>>>> On Sun, May 12, 2013 at 12:19 AM, Nitin Pawar <ni...@gmail.com>wrote:
>>>>
>>>>>
>>>>> normally if you want to copy the jar then hadoop admins setu
>>>>>
>>>>
>>>> Submit you job to Job tracker it will distribute throughout the
>>>> tasktrackers.
>>>>
>>>> *Thanks & Regards *
>>>>
>>>> ∞
>>>> Shashwat Shriparv
>>>>
>>>>
>>>
>>
>>
>> --
>> Nitin Pawar
>>
>
>
Re: Submitting a hadoop job in large clusters.
Posted by Bertrand Dechoux <de...@gmail.com>.
Which doesn't imply that you should log yourself to the physical machine
where the JobTracker is hosted. It only implies that the hadoop client must
be able to reach the JobTracker. It could be from any physical machines
hosting the slaves (DataNode, Tasktracker) but it is rarely the case.
Often, job are submitted from a machine which doesn't belong to the cluster
but can reach every machine of it.
Regards
Bertrand
On Sun, May 12, 2013 at 7:59 PM, shashwat shriparv <
dwivedishashwat@gmail.com> wrote:
> As nitin said , its responsibility of Jobtracker to distribute the job to
> task to the tasktrackers so you need to submitt the job to the job tracker
>
> *Thanks & Regards *
>
> ∞
> Shashwat Shriparv
>
>
>
> On Sun, May 12, 2013 at 11:26 PM, Nitin Pawar <ni...@gmail.com>wrote:
>
>> nope
>> in MRv1 only jobtracker can accept jobs. You can not trigger job on any
>> other process in hadoop other than jobtracker.
>>
>>
>> On Sun, May 12, 2013 at 11:25 PM, Shashidhar Rao <
>> raoshashidhar123@gmail.com> wrote:
>>
>>> @shashwat shriparv
>>>
>>> Can the a hadoop job be submitted to any datanode in the cluster and not
>>> to jobTracker.
>>>
>>> Correct me if it I am wrong , I was told that a hadoop job can be
>>> submitted to datanode also apart from JobTracker. Is it correct?
>>>
>>> Advanced thanks
>>>
>>>
>>> On Sun, May 12, 2013 at 11:02 PM, shashwat shriparv <
>>> dwivedishashwat@gmail.com> wrote:
>>>
>>>>
>>>> On Sun, May 12, 2013 at 12:19 AM, Nitin Pawar <ni...@gmail.com>wrote:
>>>>
>>>>>
>>>>> normally if you want to copy the jar then hadoop admins setu
>>>>>
>>>>
>>>> Submit you job to Job tracker it will distribute throughout the
>>>> tasktrackers.
>>>>
>>>> *Thanks & Regards *
>>>>
>>>> ∞
>>>> Shashwat Shriparv
>>>>
>>>>
>>>
>>
>>
>> --
>> Nitin Pawar
>>
>
>
Re: Submitting a hadoop job in large clusters.
Posted by Bertrand Dechoux <de...@gmail.com>.
Which doesn't imply that you should log yourself to the physical machine
where the JobTracker is hosted. It only implies that the hadoop client must
be able to reach the JobTracker. It could be from any physical machines
hosting the slaves (DataNode, Tasktracker) but it is rarely the case.
Often, job are submitted from a machine which doesn't belong to the cluster
but can reach every machine of it.
Regards
Bertrand
On Sun, May 12, 2013 at 7:59 PM, shashwat shriparv <
dwivedishashwat@gmail.com> wrote:
> As nitin said , its responsibility of Jobtracker to distribute the job to
> task to the tasktrackers so you need to submitt the job to the job tracker
>
> *Thanks & Regards *
>
> ∞
> Shashwat Shriparv
>
>
>
> On Sun, May 12, 2013 at 11:26 PM, Nitin Pawar <ni...@gmail.com>wrote:
>
>> nope
>> in MRv1 only jobtracker can accept jobs. You can not trigger job on any
>> other process in hadoop other than jobtracker.
>>
>>
>> On Sun, May 12, 2013 at 11:25 PM, Shashidhar Rao <
>> raoshashidhar123@gmail.com> wrote:
>>
>>> @shashwat shriparv
>>>
>>> Can the a hadoop job be submitted to any datanode in the cluster and not
>>> to jobTracker.
>>>
>>> Correct me if it I am wrong , I was told that a hadoop job can be
>>> submitted to datanode also apart from JobTracker. Is it correct?
>>>
>>> Advanced thanks
>>>
>>>
>>> On Sun, May 12, 2013 at 11:02 PM, shashwat shriparv <
>>> dwivedishashwat@gmail.com> wrote:
>>>
>>>>
>>>> On Sun, May 12, 2013 at 12:19 AM, Nitin Pawar <ni...@gmail.com>wrote:
>>>>
>>>>>
>>>>> normally if you want to copy the jar then hadoop admins setu
>>>>>
>>>>
>>>> Submit you job to Job tracker it will distribute throughout the
>>>> tasktrackers.
>>>>
>>>> *Thanks & Regards *
>>>>
>>>> ∞
>>>> Shashwat Shriparv
>>>>
>>>>
>>>
>>
>>
>> --
>> Nitin Pawar
>>
>
>
Re: Submitting a hadoop job in large clusters.
Posted by shashwat shriparv <dw...@gmail.com>.
As nitin said , its responsibility of Jobtracker to distribute the job to
task to the tasktrackers so you need to submitt the job to the job tracker
*Thanks & Regards *
∞
Shashwat Shriparv
On Sun, May 12, 2013 at 11:26 PM, Nitin Pawar <ni...@gmail.com>wrote:
> nope
> in MRv1 only jobtracker can accept jobs. You can not trigger job on any
> other process in hadoop other than jobtracker.
>
>
> On Sun, May 12, 2013 at 11:25 PM, Shashidhar Rao <
> raoshashidhar123@gmail.com> wrote:
>
>> @shashwat shriparv
>>
>> Can the a hadoop job be submitted to any datanode in the cluster and not
>> to jobTracker.
>>
>> Correct me if it I am wrong , I was told that a hadoop job can be
>> submitted to datanode also apart from JobTracker. Is it correct?
>>
>> Advanced thanks
>>
>>
>> On Sun, May 12, 2013 at 11:02 PM, shashwat shriparv <
>> dwivedishashwat@gmail.com> wrote:
>>
>>>
>>> On Sun, May 12, 2013 at 12:19 AM, Nitin Pawar <ni...@gmail.com>wrote:
>>>
>>>>
>>>> normally if you want to copy the jar then hadoop admins setu
>>>>
>>>
>>> Submit you job to Job tracker it will distribute throughout the
>>> tasktrackers.
>>>
>>> *Thanks & Regards *
>>>
>>> ∞
>>> Shashwat Shriparv
>>>
>>>
>>
>
>
> --
> Nitin Pawar
>
Re: Submitting a hadoop job in large clusters.
Posted by shashwat shriparv <dw...@gmail.com>.
As nitin said , its responsibility of Jobtracker to distribute the job to
task to the tasktrackers so you need to submitt the job to the job tracker
*Thanks & Regards *
∞
Shashwat Shriparv
On Sun, May 12, 2013 at 11:26 PM, Nitin Pawar <ni...@gmail.com>wrote:
> nope
> in MRv1 only jobtracker can accept jobs. You can not trigger job on any
> other process in hadoop other than jobtracker.
>
>
> On Sun, May 12, 2013 at 11:25 PM, Shashidhar Rao <
> raoshashidhar123@gmail.com> wrote:
>
>> @shashwat shriparv
>>
>> Can the a hadoop job be submitted to any datanode in the cluster and not
>> to jobTracker.
>>
>> Correct me if it I am wrong , I was told that a hadoop job can be
>> submitted to datanode also apart from JobTracker. Is it correct?
>>
>> Advanced thanks
>>
>>
>> On Sun, May 12, 2013 at 11:02 PM, shashwat shriparv <
>> dwivedishashwat@gmail.com> wrote:
>>
>>>
>>> On Sun, May 12, 2013 at 12:19 AM, Nitin Pawar <ni...@gmail.com>wrote:
>>>
>>>>
>>>> normally if you want to copy the jar then hadoop admins setu
>>>>
>>>
>>> Submit you job to Job tracker it will distribute throughout the
>>> tasktrackers.
>>>
>>> *Thanks & Regards *
>>>
>>> ∞
>>> Shashwat Shriparv
>>>
>>>
>>
>
>
> --
> Nitin Pawar
>
Re: Submitting a hadoop job in large clusters.
Posted by shashwat shriparv <dw...@gmail.com>.
As nitin said , its responsibility of Jobtracker to distribute the job to
task to the tasktrackers so you need to submitt the job to the job tracker
*Thanks & Regards *
∞
Shashwat Shriparv
On Sun, May 12, 2013 at 11:26 PM, Nitin Pawar <ni...@gmail.com>wrote:
> nope
> in MRv1 only jobtracker can accept jobs. You can not trigger job on any
> other process in hadoop other than jobtracker.
>
>
> On Sun, May 12, 2013 at 11:25 PM, Shashidhar Rao <
> raoshashidhar123@gmail.com> wrote:
>
>> @shashwat shriparv
>>
>> Can the a hadoop job be submitted to any datanode in the cluster and not
>> to jobTracker.
>>
>> Correct me if it I am wrong , I was told that a hadoop job can be
>> submitted to datanode also apart from JobTracker. Is it correct?
>>
>> Advanced thanks
>>
>>
>> On Sun, May 12, 2013 at 11:02 PM, shashwat shriparv <
>> dwivedishashwat@gmail.com> wrote:
>>
>>>
>>> On Sun, May 12, 2013 at 12:19 AM, Nitin Pawar <ni...@gmail.com>wrote:
>>>
>>>>
>>>> normally if you want to copy the jar then hadoop admins setu
>>>>
>>>
>>> Submit you job to Job tracker it will distribute throughout the
>>> tasktrackers.
>>>
>>> *Thanks & Regards *
>>>
>>> ∞
>>> Shashwat Shriparv
>>>
>>>
>>
>
>
> --
> Nitin Pawar
>
Re: Submitting a hadoop job in large clusters.
Posted by shashwat shriparv <dw...@gmail.com>.
As nitin said , its responsibility of Jobtracker to distribute the job to
task to the tasktrackers so you need to submitt the job to the job tracker
*Thanks & Regards *
∞
Shashwat Shriparv
On Sun, May 12, 2013 at 11:26 PM, Nitin Pawar <ni...@gmail.com>wrote:
> nope
> in MRv1 only jobtracker can accept jobs. You can not trigger job on any
> other process in hadoop other than jobtracker.
>
>
> On Sun, May 12, 2013 at 11:25 PM, Shashidhar Rao <
> raoshashidhar123@gmail.com> wrote:
>
>> @shashwat shriparv
>>
>> Can the a hadoop job be submitted to any datanode in the cluster and not
>> to jobTracker.
>>
>> Correct me if it I am wrong , I was told that a hadoop job can be
>> submitted to datanode also apart from JobTracker. Is it correct?
>>
>> Advanced thanks
>>
>>
>> On Sun, May 12, 2013 at 11:02 PM, shashwat shriparv <
>> dwivedishashwat@gmail.com> wrote:
>>
>>>
>>> On Sun, May 12, 2013 at 12:19 AM, Nitin Pawar <ni...@gmail.com>wrote:
>>>
>>>>
>>>> normally if you want to copy the jar then hadoop admins setu
>>>>
>>>
>>> Submit you job to Job tracker it will distribute throughout the
>>> tasktrackers.
>>>
>>> *Thanks & Regards *
>>>
>>> ∞
>>> Shashwat Shriparv
>>>
>>>
>>
>
>
> --
> Nitin Pawar
>
Re: Submitting a hadoop job in large clusters.
Posted by Nitin Pawar <ni...@gmail.com>.
nope
in MRv1 only jobtracker can accept jobs. You can not trigger job on any
other process in hadoop other than jobtracker.
On Sun, May 12, 2013 at 11:25 PM, Shashidhar Rao <raoshashidhar123@gmail.com
> wrote:
> @shashwat shriparv
>
> Can the a hadoop job be submitted to any datanode in the cluster and not
> to jobTracker.
>
> Correct me if it I am wrong , I was told that a hadoop job can be
> submitted to datanode also apart from JobTracker. Is it correct?
>
> Advanced thanks
>
>
> On Sun, May 12, 2013 at 11:02 PM, shashwat shriparv <
> dwivedishashwat@gmail.com> wrote:
>
>>
>> On Sun, May 12, 2013 at 12:19 AM, Nitin Pawar <ni...@gmail.com>wrote:
>>
>>>
>>> normally if you want to copy the jar then hadoop admins setu
>>>
>>
>> Submit you job to Job tracker it will distribute throughout the
>> tasktrackers.
>>
>> *Thanks & Regards *
>>
>> ∞
>> Shashwat Shriparv
>>
>>
>
--
Nitin Pawar
Re: Submitting a hadoop job in large clusters.
Posted by Nitin Pawar <ni...@gmail.com>.
nope
in MRv1 only jobtracker can accept jobs. You can not trigger job on any
other process in hadoop other than jobtracker.
On Sun, May 12, 2013 at 11:25 PM, Shashidhar Rao <raoshashidhar123@gmail.com
> wrote:
> @shashwat shriparv
>
> Can the a hadoop job be submitted to any datanode in the cluster and not
> to jobTracker.
>
> Correct me if it I am wrong , I was told that a hadoop job can be
> submitted to datanode also apart from JobTracker. Is it correct?
>
> Advanced thanks
>
>
> On Sun, May 12, 2013 at 11:02 PM, shashwat shriparv <
> dwivedishashwat@gmail.com> wrote:
>
>>
>> On Sun, May 12, 2013 at 12:19 AM, Nitin Pawar <ni...@gmail.com>wrote:
>>
>>>
>>> normally if you want to copy the jar then hadoop admins setu
>>>
>>
>> Submit you job to Job tracker it will distribute throughout the
>> tasktrackers.
>>
>> *Thanks & Regards *
>>
>> ∞
>> Shashwat Shriparv
>>
>>
>
--
Nitin Pawar
Re: Submitting a hadoop job in large clusters.
Posted by Nitin Pawar <ni...@gmail.com>.
nope
in MRv1 only jobtracker can accept jobs. You can not trigger job on any
other process in hadoop other than jobtracker.
On Sun, May 12, 2013 at 11:25 PM, Shashidhar Rao <raoshashidhar123@gmail.com
> wrote:
> @shashwat shriparv
>
> Can the a hadoop job be submitted to any datanode in the cluster and not
> to jobTracker.
>
> Correct me if it I am wrong , I was told that a hadoop job can be
> submitted to datanode also apart from JobTracker. Is it correct?
>
> Advanced thanks
>
>
> On Sun, May 12, 2013 at 11:02 PM, shashwat shriparv <
> dwivedishashwat@gmail.com> wrote:
>
>>
>> On Sun, May 12, 2013 at 12:19 AM, Nitin Pawar <ni...@gmail.com>wrote:
>>
>>>
>>> normally if you want to copy the jar then hadoop admins setu
>>>
>>
>> Submit you job to Job tracker it will distribute throughout the
>> tasktrackers.
>>
>> *Thanks & Regards *
>>
>> ∞
>> Shashwat Shriparv
>>
>>
>
--
Nitin Pawar
Re: Submitting a hadoop job in large clusters.
Posted by Nitin Pawar <ni...@gmail.com>.
nope
in MRv1 only jobtracker can accept jobs. You can not trigger job on any
other process in hadoop other than jobtracker.
On Sun, May 12, 2013 at 11:25 PM, Shashidhar Rao <raoshashidhar123@gmail.com
> wrote:
> @shashwat shriparv
>
> Can the a hadoop job be submitted to any datanode in the cluster and not
> to jobTracker.
>
> Correct me if it I am wrong , I was told that a hadoop job can be
> submitted to datanode also apart from JobTracker. Is it correct?
>
> Advanced thanks
>
>
> On Sun, May 12, 2013 at 11:02 PM, shashwat shriparv <
> dwivedishashwat@gmail.com> wrote:
>
>>
>> On Sun, May 12, 2013 at 12:19 AM, Nitin Pawar <ni...@gmail.com>wrote:
>>
>>>
>>> normally if you want to copy the jar then hadoop admins setu
>>>
>>
>> Submit you job to Job tracker it will distribute throughout the
>> tasktrackers.
>>
>> *Thanks & Regards *
>>
>> ∞
>> Shashwat Shriparv
>>
>>
>
--
Nitin Pawar
Re: Submitting a hadoop job in large clusters.
Posted by Shashidhar Rao <ra...@gmail.com>.
@shashwat shriparv
Can the a hadoop job be submitted to any datanode in the cluster and not to
jobTracker.
Correct me if it I am wrong , I was told that a hadoop job can be submitted
to datanode also apart from JobTracker. Is it correct?
Advanced thanks
On Sun, May 12, 2013 at 11:02 PM, shashwat shriparv <
dwivedishashwat@gmail.com> wrote:
>
> On Sun, May 12, 2013 at 12:19 AM, Nitin Pawar <ni...@gmail.com>wrote:
>
>>
>> normally if you want to copy the jar then hadoop admins setu
>>
>
> Submit you job to Job tracker it will distribute throughout the
> tasktrackers.
>
> *Thanks & Regards *
>
> ∞
> Shashwat Shriparv
>
>
Re: Submitting a hadoop job in large clusters.
Posted by Shashidhar Rao <ra...@gmail.com>.
@shashwat shriparv
Can the a hadoop job be submitted to any datanode in the cluster and not to
jobTracker.
Correct me if it I am wrong , I was told that a hadoop job can be submitted
to datanode also apart from JobTracker. Is it correct?
Advanced thanks
On Sun, May 12, 2013 at 11:02 PM, shashwat shriparv <
dwivedishashwat@gmail.com> wrote:
>
> On Sun, May 12, 2013 at 12:19 AM, Nitin Pawar <ni...@gmail.com>wrote:
>
>>
>> normally if you want to copy the jar then hadoop admins setu
>>
>
> Submit you job to Job tracker it will distribute throughout the
> tasktrackers.
>
> *Thanks & Regards *
>
> ∞
> Shashwat Shriparv
>
>
Re: Submitting a hadoop job in large clusters.
Posted by Shashidhar Rao <ra...@gmail.com>.
@shashwat shriparv
Can the a hadoop job be submitted to any datanode in the cluster and not to
jobTracker.
Correct me if it I am wrong , I was told that a hadoop job can be submitted
to datanode also apart from JobTracker. Is it correct?
Advanced thanks
On Sun, May 12, 2013 at 11:02 PM, shashwat shriparv <
dwivedishashwat@gmail.com> wrote:
>
> On Sun, May 12, 2013 at 12:19 AM, Nitin Pawar <ni...@gmail.com>wrote:
>
>>
>> normally if you want to copy the jar then hadoop admins setu
>>
>
> Submit you job to Job tracker it will distribute throughout the
> tasktrackers.
>
> *Thanks & Regards *
>
> ∞
> Shashwat Shriparv
>
>
Re: Submitting a hadoop job in large clusters.
Posted by Shashidhar Rao <ra...@gmail.com>.
@shashwat shriparv
Can the a hadoop job be submitted to any datanode in the cluster and not to
jobTracker.
Correct me if it I am wrong , I was told that a hadoop job can be submitted
to datanode also apart from JobTracker. Is it correct?
Advanced thanks
On Sun, May 12, 2013 at 11:02 PM, shashwat shriparv <
dwivedishashwat@gmail.com> wrote:
>
> On Sun, May 12, 2013 at 12:19 AM, Nitin Pawar <ni...@gmail.com>wrote:
>
>>
>> normally if you want to copy the jar then hadoop admins setu
>>
>
> Submit you job to Job tracker it will distribute throughout the
> tasktrackers.
>
> *Thanks & Regards *
>
> ∞
> Shashwat Shriparv
>
>
Re: Submitting a hadoop job in large clusters.
Posted by shashwat shriparv <dw...@gmail.com>.
On Sun, May 12, 2013 at 12:19 AM, Nitin Pawar <ni...@gmail.com>wrote:
>
> normally if you want to copy the jar then hadoop admins setu
>
Submit you job to Job tracker it will distribute throughout the
tasktrackers.
*Thanks & Regards *
∞
Shashwat Shriparv
Re: Submitting a hadoop job in large clusters.
Posted by shashwat shriparv <dw...@gmail.com>.
On Sun, May 12, 2013 at 12:19 AM, Nitin Pawar <ni...@gmail.com>wrote:
>
> normally if you want to copy the jar then hadoop admins setu
>
Submit you job to Job tracker it will distribute throughout the
tasktrackers.
*Thanks & Regards *
∞
Shashwat Shriparv
Re: Submitting a hadoop job in large clusters.
Posted by shashwat shriparv <dw...@gmail.com>.
On Sun, May 12, 2013 at 12:19 AM, Nitin Pawar <ni...@gmail.com>wrote:
>
> normally if you want to copy the jar then hadoop admins setu
>
Submit you job to Job tracker it will distribute throughout the
tasktrackers.
*Thanks & Regards *
∞
Shashwat Shriparv
Re: Submitting a hadoop job in large clusters.
Posted by shashwat shriparv <dw...@gmail.com>.
On Sun, May 12, 2013 at 12:19 AM, Nitin Pawar <ni...@gmail.com>wrote:
>
> normally if you want to copy the jar then hadoop admins setu
>
Submit you job to Job tracker it will distribute throughout the
tasktrackers.
*Thanks & Regards *
∞
Shashwat Shriparv
Re: Submitting a hadoop job in large clusters.
Posted by Nitin Pawar <ni...@gmail.com>.
you do not need to get inside the cluster to submit the job
normally if you want to copy the jar then hadoop admins setup a gateway or
UI to copy your jars to hdfs
if you just want to run your jar using the hadoop command,
then just point your hadoop conf to the cluster job tracker and then it
should be able to submit job to the cluster.
On Sat, May 11, 2013 at 11:54 PM, Shashidhar Rao <raoshashidhar123@gmail.com
> wrote:
> Hello users,
>
> I am running hadoop in my windows laptop using linux virtual machine.Since
> every Hadoop daemon is running in one machine I have no problem running
> hadoop jar command and then submittng the job and everything works fine.
>
> But, I think when hadoop is running in multiple nodes submitting a job
> would be different. My understanding is that I have to remote connect to
> one of the node ex. any datanode not necessary JobTracker and then copy the
> job jar to a location, cd to hadoop bin and then fire up the console and
> then run the hadoop jar command.
>
> Is this how to submit a hadoop job in a cluster or is there any other way
> to submit a job .
> Basically, I have never used a cluster so am having this doubt
>
> Advanced thanks
>
>
>
--
Nitin Pawar
Re: Submitting a hadoop job in large clusters.
Posted by Nitin Pawar <ni...@gmail.com>.
you do not need to get inside the cluster to submit the job
normally if you want to copy the jar then hadoop admins setup a gateway or
UI to copy your jars to hdfs
if you just want to run your jar using the hadoop command,
then just point your hadoop conf to the cluster job tracker and then it
should be able to submit job to the cluster.
On Sat, May 11, 2013 at 11:54 PM, Shashidhar Rao <raoshashidhar123@gmail.com
> wrote:
> Hello users,
>
> I am running hadoop in my windows laptop using linux virtual machine.Since
> every Hadoop daemon is running in one machine I have no problem running
> hadoop jar command and then submittng the job and everything works fine.
>
> But, I think when hadoop is running in multiple nodes submitting a job
> would be different. My understanding is that I have to remote connect to
> one of the node ex. any datanode not necessary JobTracker and then copy the
> job jar to a location, cd to hadoop bin and then fire up the console and
> then run the hadoop jar command.
>
> Is this how to submit a hadoop job in a cluster or is there any other way
> to submit a job .
> Basically, I have never used a cluster so am having this doubt
>
> Advanced thanks
>
>
>
--
Nitin Pawar
Re: Submitting a hadoop job in large clusters.
Posted by Nitin Pawar <ni...@gmail.com>.
you do not need to get inside the cluster to submit the job
normally if you want to copy the jar then hadoop admins setup a gateway or
UI to copy your jars to hdfs
if you just want to run your jar using the hadoop command,
then just point your hadoop conf to the cluster job tracker and then it
should be able to submit job to the cluster.
On Sat, May 11, 2013 at 11:54 PM, Shashidhar Rao <raoshashidhar123@gmail.com
> wrote:
> Hello users,
>
> I am running hadoop in my windows laptop using linux virtual machine.Since
> every Hadoop daemon is running in one machine I have no problem running
> hadoop jar command and then submittng the job and everything works fine.
>
> But, I think when hadoop is running in multiple nodes submitting a job
> would be different. My understanding is that I have to remote connect to
> one of the node ex. any datanode not necessary JobTracker and then copy the
> job jar to a location, cd to hadoop bin and then fire up the console and
> then run the hadoop jar command.
>
> Is this how to submit a hadoop job in a cluster or is there any other way
> to submit a job .
> Basically, I have never used a cluster so am having this doubt
>
> Advanced thanks
>
>
>
--
Nitin Pawar
Re: Submitting a hadoop job in large clusters.
Posted by Nitin Pawar <ni...@gmail.com>.
you do not need to get inside the cluster to submit the job
normally if you want to copy the jar then hadoop admins setup a gateway or
UI to copy your jars to hdfs
if you just want to run your jar using the hadoop command,
then just point your hadoop conf to the cluster job tracker and then it
should be able to submit job to the cluster.
On Sat, May 11, 2013 at 11:54 PM, Shashidhar Rao <raoshashidhar123@gmail.com
> wrote:
> Hello users,
>
> I am running hadoop in my windows laptop using linux virtual machine.Since
> every Hadoop daemon is running in one machine I have no problem running
> hadoop jar command and then submittng the job and everything works fine.
>
> But, I think when hadoop is running in multiple nodes submitting a job
> would be different. My understanding is that I have to remote connect to
> one of the node ex. any datanode not necessary JobTracker and then copy the
> job jar to a location, cd to hadoop bin and then fire up the console and
> then run the hadoop jar command.
>
> Is this how to submit a hadoop job in a cluster or is there any other way
> to submit a job .
> Basically, I have never used a cluster so am having this doubt
>
> Advanced thanks
>
>
>
--
Nitin Pawar