You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by KayVajj <va...@gmail.com> on 2013/03/13 17:01:42 UTC
Question regarding hadoop jar command usage
I have a question regarding the hadoop jar command. In a cluster of say
nodes n1,n2...n100
the node n1 has jar Myjar on its local file system.
If I run the command
hadoop jar local/path/to/Myjar Myclass other-args
Is the MR job executed just on n1 or any arbitrary node n1..n100?
If it is any arbitrary node, since Myjar exists only on n1. Is the jar
copied across all nodes on the cluster to run the MR code.
Re: Question regarding hadoop jar command usage
Posted by be...@gmail.com.
Hi
Any node would submit the job to JobTracker which distributes the jar to TaskTrackers and individual tasks are executed on nodes across the cluster.
MR tasks are executed across the cluster.
Regards
Bejoy KS
Sent from remote device, Please excuse typos
-----Original Message-----
From: KayVajj <va...@gmail.com>
Date: Wed, 13 Mar 2013 09:01:42
To: <us...@hadoop.apache.org>
Reply-To: user@hadoop.apache.org
Subject: Question regarding hadoop jar command usage
I have a question regarding the hadoop jar command. In a cluster of say
nodes n1,n2...n100
the node n1 has jar Myjar on its local file system.
If I run the command
hadoop jar local/path/to/Myjar Myclass other-args
Is the MR job executed just on n1 or any arbitrary node n1..n100?
If it is any arbitrary node, since Myjar exists only on n1. Is the jar
copied across all nodes on the cluster to run the MR code.
Re: Question regarding hadoop jar command usage
Posted by Sandy Ryza <sa...@cloudera.com>.
Hi Kay,
Yeah, that line does set your jar as the job jar. "hadoop jar" expects
java code to configure and submit your job. "mapred job" takes in a
job.xml configuration file and runs the job based on that.
-Sandy
On Wed, Mar 13, 2013 at 11:07 AM, KayVajj <va...@gmail.com> wrote:
> Hi Sandy,
>
> I was going through the RunJar source code and the jar executes locally.
> When the jar fires a mapreduce job,
>
> the way I create JobConf is
>
> JobConf conf = new JobConf(MyJob.class);
>>
>
> Does this set MyJar as the job jar?
>
> Can you explain what is the difference between running an MR job using the
> jar command vs mapred job command (looks like the hadoop job command is
> deprecated).
>
> Thanks
> Kay
>
>
> On Wed, Mar 13, 2013 at 10:14 AM, Sandy Ryza <sa...@cloudera.com>wrote:
>
>> Hi Kay,
>>
>> The jar is just executed locally. If the jar fires up a mapreduce job
>> and sets itself as the job jar, then mapreduce will handle copying it to
>> the nodes that will use it.
>>
>> -Sandy
>>
>>
>> On Wed, Mar 13, 2013 at 9:01 AM, KayVajj <va...@gmail.com> wrote:
>>
>>> I have a question regarding the hadoop jar command. In a cluster of say
>>> nodes n1,n2...n100
>>>
>>> the node n1 has jar Myjar on its local file system.
>>>
>>> If I run the command
>>> hadoop jar local/path/to/Myjar Myclass other-args
>>>
>>> Is the MR job executed just on n1 or any arbitrary node n1..n100?
>>>
>>> If it is any arbitrary node, since Myjar exists only on n1. Is the jar
>>> copied across all nodes on the cluster to run the MR code.
>>>
>>>
>>>
>>>
>>>
>>
>
Re: Question regarding hadoop jar command usage
Posted by Sandy Ryza <sa...@cloudera.com>.
Hi Kay,
Yeah, that line does set your jar as the job jar. "hadoop jar" expects
java code to configure and submit your job. "mapred job" takes in a
job.xml configuration file and runs the job based on that.
-Sandy
On Wed, Mar 13, 2013 at 11:07 AM, KayVajj <va...@gmail.com> wrote:
> Hi Sandy,
>
> I was going through the RunJar source code and the jar executes locally.
> When the jar fires a mapreduce job,
>
> the way I create JobConf is
>
> JobConf conf = new JobConf(MyJob.class);
>>
>
> Does this set MyJar as the job jar?
>
> Can you explain what is the difference between running an MR job using the
> jar command vs mapred job command (looks like the hadoop job command is
> deprecated).
>
> Thanks
> Kay
>
>
> On Wed, Mar 13, 2013 at 10:14 AM, Sandy Ryza <sa...@cloudera.com>wrote:
>
>> Hi Kay,
>>
>> The jar is just executed locally. If the jar fires up a mapreduce job
>> and sets itself as the job jar, then mapreduce will handle copying it to
>> the nodes that will use it.
>>
>> -Sandy
>>
>>
>> On Wed, Mar 13, 2013 at 9:01 AM, KayVajj <va...@gmail.com> wrote:
>>
>>> I have a question regarding the hadoop jar command. In a cluster of say
>>> nodes n1,n2...n100
>>>
>>> the node n1 has jar Myjar on its local file system.
>>>
>>> If I run the command
>>> hadoop jar local/path/to/Myjar Myclass other-args
>>>
>>> Is the MR job executed just on n1 or any arbitrary node n1..n100?
>>>
>>> If it is any arbitrary node, since Myjar exists only on n1. Is the jar
>>> copied across all nodes on the cluster to run the MR code.
>>>
>>>
>>>
>>>
>>>
>>
>
Re: Question regarding hadoop jar command usage
Posted by Sandy Ryza <sa...@cloudera.com>.
Hi Kay,
Yeah, that line does set your jar as the job jar. "hadoop jar" expects
java code to configure and submit your job. "mapred job" takes in a
job.xml configuration file and runs the job based on that.
-Sandy
On Wed, Mar 13, 2013 at 11:07 AM, KayVajj <va...@gmail.com> wrote:
> Hi Sandy,
>
> I was going through the RunJar source code and the jar executes locally.
> When the jar fires a mapreduce job,
>
> the way I create JobConf is
>
> JobConf conf = new JobConf(MyJob.class);
>>
>
> Does this set MyJar as the job jar?
>
> Can you explain what is the difference between running an MR job using the
> jar command vs mapred job command (looks like the hadoop job command is
> deprecated).
>
> Thanks
> Kay
>
>
> On Wed, Mar 13, 2013 at 10:14 AM, Sandy Ryza <sa...@cloudera.com>wrote:
>
>> Hi Kay,
>>
>> The jar is just executed locally. If the jar fires up a mapreduce job
>> and sets itself as the job jar, then mapreduce will handle copying it to
>> the nodes that will use it.
>>
>> -Sandy
>>
>>
>> On Wed, Mar 13, 2013 at 9:01 AM, KayVajj <va...@gmail.com> wrote:
>>
>>> I have a question regarding the hadoop jar command. In a cluster of say
>>> nodes n1,n2...n100
>>>
>>> the node n1 has jar Myjar on its local file system.
>>>
>>> If I run the command
>>> hadoop jar local/path/to/Myjar Myclass other-args
>>>
>>> Is the MR job executed just on n1 or any arbitrary node n1..n100?
>>>
>>> If it is any arbitrary node, since Myjar exists only on n1. Is the jar
>>> copied across all nodes on the cluster to run the MR code.
>>>
>>>
>>>
>>>
>>>
>>
>
Re: Question regarding hadoop jar command usage
Posted by Sandy Ryza <sa...@cloudera.com>.
Hi Kay,
Yeah, that line does set your jar as the job jar. "hadoop jar" expects
java code to configure and submit your job. "mapred job" takes in a
job.xml configuration file and runs the job based on that.
-Sandy
On Wed, Mar 13, 2013 at 11:07 AM, KayVajj <va...@gmail.com> wrote:
> Hi Sandy,
>
> I was going through the RunJar source code and the jar executes locally.
> When the jar fires a mapreduce job,
>
> the way I create JobConf is
>
> JobConf conf = new JobConf(MyJob.class);
>>
>
> Does this set MyJar as the job jar?
>
> Can you explain what is the difference between running an MR job using the
> jar command vs mapred job command (looks like the hadoop job command is
> deprecated).
>
> Thanks
> Kay
>
>
> On Wed, Mar 13, 2013 at 10:14 AM, Sandy Ryza <sa...@cloudera.com>wrote:
>
>> Hi Kay,
>>
>> The jar is just executed locally. If the jar fires up a mapreduce job
>> and sets itself as the job jar, then mapreduce will handle copying it to
>> the nodes that will use it.
>>
>> -Sandy
>>
>>
>> On Wed, Mar 13, 2013 at 9:01 AM, KayVajj <va...@gmail.com> wrote:
>>
>>> I have a question regarding the hadoop jar command. In a cluster of say
>>> nodes n1,n2...n100
>>>
>>> the node n1 has jar Myjar on its local file system.
>>>
>>> If I run the command
>>> hadoop jar local/path/to/Myjar Myclass other-args
>>>
>>> Is the MR job executed just on n1 or any arbitrary node n1..n100?
>>>
>>> If it is any arbitrary node, since Myjar exists only on n1. Is the jar
>>> copied across all nodes on the cluster to run the MR code.
>>>
>>>
>>>
>>>
>>>
>>
>
Re: Question regarding hadoop jar command usage
Posted by KayVajj <va...@gmail.com>.
Hi Sandy,
I was going through the RunJar source code and the jar executes locally.
When the jar fires a mapreduce job,
the way I create JobConf is
JobConf conf = new JobConf(MyJob.class);
>
Does this set MyJar as the job jar?
Can you explain what is the difference between running an MR job using the
jar command vs mapred job command (looks like the hadoop job command is
deprecated).
Thanks
Kay
On Wed, Mar 13, 2013 at 10:14 AM, Sandy Ryza <sa...@cloudera.com>wrote:
> Hi Kay,
>
> The jar is just executed locally. If the jar fires up a mapreduce job and
> sets itself as the job jar, then mapreduce will handle copying it to the
> nodes that will use it.
>
> -Sandy
>
>
> On Wed, Mar 13, 2013 at 9:01 AM, KayVajj <va...@gmail.com> wrote:
>
>> I have a question regarding the hadoop jar command. In a cluster of say
>> nodes n1,n2...n100
>>
>> the node n1 has jar Myjar on its local file system.
>>
>> If I run the command
>> hadoop jar local/path/to/Myjar Myclass other-args
>>
>> Is the MR job executed just on n1 or any arbitrary node n1..n100?
>>
>> If it is any arbitrary node, since Myjar exists only on n1. Is the jar
>> copied across all nodes on the cluster to run the MR code.
>>
>>
>>
>>
>>
>
Re: Question regarding hadoop jar command usage
Posted by KayVajj <va...@gmail.com>.
Hi Sandy,
I was going through the RunJar source code and the jar executes locally.
When the jar fires a mapreduce job,
the way I create JobConf is
JobConf conf = new JobConf(MyJob.class);
>
Does this set MyJar as the job jar?
Can you explain what is the difference between running an MR job using the
jar command vs mapred job command (looks like the hadoop job command is
deprecated).
Thanks
Kay
On Wed, Mar 13, 2013 at 10:14 AM, Sandy Ryza <sa...@cloudera.com>wrote:
> Hi Kay,
>
> The jar is just executed locally. If the jar fires up a mapreduce job and
> sets itself as the job jar, then mapreduce will handle copying it to the
> nodes that will use it.
>
> -Sandy
>
>
> On Wed, Mar 13, 2013 at 9:01 AM, KayVajj <va...@gmail.com> wrote:
>
>> I have a question regarding the hadoop jar command. In a cluster of say
>> nodes n1,n2...n100
>>
>> the node n1 has jar Myjar on its local file system.
>>
>> If I run the command
>> hadoop jar local/path/to/Myjar Myclass other-args
>>
>> Is the MR job executed just on n1 or any arbitrary node n1..n100?
>>
>> If it is any arbitrary node, since Myjar exists only on n1. Is the jar
>> copied across all nodes on the cluster to run the MR code.
>>
>>
>>
>>
>>
>
Re: Question regarding hadoop jar command usage
Posted by KayVajj <va...@gmail.com>.
Hi Sandy,
I was going through the RunJar source code and the jar executes locally.
When the jar fires a mapreduce job,
the way I create JobConf is
JobConf conf = new JobConf(MyJob.class);
>
Does this set MyJar as the job jar?
Can you explain what is the difference between running an MR job using the
jar command vs mapred job command (looks like the hadoop job command is
deprecated).
Thanks
Kay
On Wed, Mar 13, 2013 at 10:14 AM, Sandy Ryza <sa...@cloudera.com>wrote:
> Hi Kay,
>
> The jar is just executed locally. If the jar fires up a mapreduce job and
> sets itself as the job jar, then mapreduce will handle copying it to the
> nodes that will use it.
>
> -Sandy
>
>
> On Wed, Mar 13, 2013 at 9:01 AM, KayVajj <va...@gmail.com> wrote:
>
>> I have a question regarding the hadoop jar command. In a cluster of say
>> nodes n1,n2...n100
>>
>> the node n1 has jar Myjar on its local file system.
>>
>> If I run the command
>> hadoop jar local/path/to/Myjar Myclass other-args
>>
>> Is the MR job executed just on n1 or any arbitrary node n1..n100?
>>
>> If it is any arbitrary node, since Myjar exists only on n1. Is the jar
>> copied across all nodes on the cluster to run the MR code.
>>
>>
>>
>>
>>
>
Re: Question regarding hadoop jar command usage
Posted by KayVajj <va...@gmail.com>.
Hi Sandy,
I was going through the RunJar source code and the jar executes locally.
When the jar fires a mapreduce job,
the way I create JobConf is
JobConf conf = new JobConf(MyJob.class);
>
Does this set MyJar as the job jar?
Can you explain what is the difference between running an MR job using the
jar command vs mapred job command (looks like the hadoop job command is
deprecated).
Thanks
Kay
On Wed, Mar 13, 2013 at 10:14 AM, Sandy Ryza <sa...@cloudera.com>wrote:
> Hi Kay,
>
> The jar is just executed locally. If the jar fires up a mapreduce job and
> sets itself as the job jar, then mapreduce will handle copying it to the
> nodes that will use it.
>
> -Sandy
>
>
> On Wed, Mar 13, 2013 at 9:01 AM, KayVajj <va...@gmail.com> wrote:
>
>> I have a question regarding the hadoop jar command. In a cluster of say
>> nodes n1,n2...n100
>>
>> the node n1 has jar Myjar on its local file system.
>>
>> If I run the command
>> hadoop jar local/path/to/Myjar Myclass other-args
>>
>> Is the MR job executed just on n1 or any arbitrary node n1..n100?
>>
>> If it is any arbitrary node, since Myjar exists only on n1. Is the jar
>> copied across all nodes on the cluster to run the MR code.
>>
>>
>>
>>
>>
>
Re: Question regarding hadoop jar command usage
Posted by Sandy Ryza <sa...@cloudera.com>.
Hi Kay,
The jar is just executed locally. If the jar fires up a mapreduce job and
sets itself as the job jar, then mapreduce will handle copying it to the
nodes that will use it.
-Sandy
On Wed, Mar 13, 2013 at 9:01 AM, KayVajj <va...@gmail.com> wrote:
> I have a question regarding the hadoop jar command. In a cluster of say
> nodes n1,n2...n100
>
> the node n1 has jar Myjar on its local file system.
>
> If I run the command
> hadoop jar local/path/to/Myjar Myclass other-args
>
> Is the MR job executed just on n1 or any arbitrary node n1..n100?
>
> If it is any arbitrary node, since Myjar exists only on n1. Is the jar
> copied across all nodes on the cluster to run the MR code.
>
>
>
>
>
Re: Question regarding hadoop jar command usage
Posted by Sandy Ryza <sa...@cloudera.com>.
Hi Kay,
The jar is just executed locally. If the jar fires up a mapreduce job and
sets itself as the job jar, then mapreduce will handle copying it to the
nodes that will use it.
-Sandy
On Wed, Mar 13, 2013 at 9:01 AM, KayVajj <va...@gmail.com> wrote:
> I have a question regarding the hadoop jar command. In a cluster of say
> nodes n1,n2...n100
>
> the node n1 has jar Myjar on its local file system.
>
> If I run the command
> hadoop jar local/path/to/Myjar Myclass other-args
>
> Is the MR job executed just on n1 or any arbitrary node n1..n100?
>
> If it is any arbitrary node, since Myjar exists only on n1. Is the jar
> copied across all nodes on the cluster to run the MR code.
>
>
>
>
>
Re: Question regarding hadoop jar command usage
Posted by be...@gmail.com.
Hi
Any node would submit the job to JobTracker which distributes the jar to TaskTrackers and individual tasks are executed on nodes across the cluster.
MR tasks are executed across the cluster.
Regards
Bejoy KS
Sent from remote device, Please excuse typos
-----Original Message-----
From: KayVajj <va...@gmail.com>
Date: Wed, 13 Mar 2013 09:01:42
To: <us...@hadoop.apache.org>
Reply-To: user@hadoop.apache.org
Subject: Question regarding hadoop jar command usage
I have a question regarding the hadoop jar command. In a cluster of say
nodes n1,n2...n100
the node n1 has jar Myjar on its local file system.
If I run the command
hadoop jar local/path/to/Myjar Myclass other-args
Is the MR job executed just on n1 or any arbitrary node n1..n100?
If it is any arbitrary node, since Myjar exists only on n1. Is the jar
copied across all nodes on the cluster to run the MR code.
Re: Question regarding hadoop jar command usage
Posted by be...@gmail.com.
Hi
Any node would submit the job to JobTracker which distributes the jar to TaskTrackers and individual tasks are executed on nodes across the cluster.
MR tasks are executed across the cluster.
Regards
Bejoy KS
Sent from remote device, Please excuse typos
-----Original Message-----
From: KayVajj <va...@gmail.com>
Date: Wed, 13 Mar 2013 09:01:42
To: <us...@hadoop.apache.org>
Reply-To: user@hadoop.apache.org
Subject: Question regarding hadoop jar command usage
I have a question regarding the hadoop jar command. In a cluster of say
nodes n1,n2...n100
the node n1 has jar Myjar on its local file system.
If I run the command
hadoop jar local/path/to/Myjar Myclass other-args
Is the MR job executed just on n1 or any arbitrary node n1..n100?
If it is any arbitrary node, since Myjar exists only on n1. Is the jar
copied across all nodes on the cluster to run the MR code.
Re: Question regarding hadoop jar command usage
Posted by be...@gmail.com.
Hi
Any node would submit the job to JobTracker which distributes the jar to TaskTrackers and individual tasks are executed on nodes across the cluster.
MR tasks are executed across the cluster.
Regards
Bejoy KS
Sent from remote device, Please excuse typos
-----Original Message-----
From: KayVajj <va...@gmail.com>
Date: Wed, 13 Mar 2013 09:01:42
To: <us...@hadoop.apache.org>
Reply-To: user@hadoop.apache.org
Subject: Question regarding hadoop jar command usage
I have a question regarding the hadoop jar command. In a cluster of say
nodes n1,n2...n100
the node n1 has jar Myjar on its local file system.
If I run the command
hadoop jar local/path/to/Myjar Myclass other-args
Is the MR job executed just on n1 or any arbitrary node n1..n100?
If it is any arbitrary node, since Myjar exists only on n1. Is the jar
copied across all nodes on the cluster to run the MR code.
Re: Question regarding hadoop jar command usage
Posted by Sandy Ryza <sa...@cloudera.com>.
Hi Kay,
The jar is just executed locally. If the jar fires up a mapreduce job and
sets itself as the job jar, then mapreduce will handle copying it to the
nodes that will use it.
-Sandy
On Wed, Mar 13, 2013 at 9:01 AM, KayVajj <va...@gmail.com> wrote:
> I have a question regarding the hadoop jar command. In a cluster of say
> nodes n1,n2...n100
>
> the node n1 has jar Myjar on its local file system.
>
> If I run the command
> hadoop jar local/path/to/Myjar Myclass other-args
>
> Is the MR job executed just on n1 or any arbitrary node n1..n100?
>
> If it is any arbitrary node, since Myjar exists only on n1. Is the jar
> copied across all nodes on the cluster to run the MR code.
>
>
>
>
>
Re: Question regarding hadoop jar command usage
Posted by Sandy Ryza <sa...@cloudera.com>.
Hi Kay,
The jar is just executed locally. If the jar fires up a mapreduce job and
sets itself as the job jar, then mapreduce will handle copying it to the
nodes that will use it.
-Sandy
On Wed, Mar 13, 2013 at 9:01 AM, KayVajj <va...@gmail.com> wrote:
> I have a question regarding the hadoop jar command. In a cluster of say
> nodes n1,n2...n100
>
> the node n1 has jar Myjar on its local file system.
>
> If I run the command
> hadoop jar local/path/to/Myjar Myclass other-args
>
> Is the MR job executed just on n1 or any arbitrary node n1..n100?
>
> If it is any arbitrary node, since Myjar exists only on n1. Is the jar
> copied across all nodes on the cluster to run the MR code.
>
>
>
>
>