You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by Kim Chew <kc...@gmail.com> on 2014/04/16 23:27:43 UTC
Submit a Hadoop 1.1.1 job remotely to a Hadoop 2 cluster
I have a cluster running Hadoop 2 but it is not running YARN, i.e. "
mapreduce.framework.name" is set to "classic" therefore the ResourceManager
is not running.
On the Client side, I want to submit a job compiled with Hadoop-1.1.1 to
the above cluster. Here how my Hadoop-1.1.1 mapred-site.xml looks like,
<property>
<!-- Pointed to the remote JobTracker -->
<name>mapred.job.tracker</name>
<value>172.31.3.150:8021</value>
</property>
Not surprisingly I got a version mismatched when I submit my job using the
Hadoop-1.1.1 jars,
org.apache.hadoop.ipc.RemoteException: Server IPC version 9 cannot
communicate with client version 4
at org.apache.hadoop.ipc.Client.call(Client.java:1107)
So I recompiled my job with Hadoop 2 and submitted it using the Hadoop 2
jars. Here is how my Hadoop 2 mapred-site.xml looks like,
<property>
<!-- Pointed to the remote JobTracker -->
<name>mapreduce.job.tracker.address</name>
<value>172.31.3.150:8021</value>
</property>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
Note that I have to set "mapreduce.framework.name" to "yarn" otherwise the
job will be run locally instead of on the targeted cluster. But my targeted
cluster is not running YARN as stated above,
14/04/16 13:35:47 INFO client.RMProxy: Connecting to ResourceManager at /
172.31.3.150:8032
14/04/16 13:35:49 INFO ipc.Client: Retrying connect to server:
hadoop-host1.eng.narus.com/172.31.3.150:8032. Already tried 0 time(s);
retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1 SECONDS)
(Yes I have set my "yarn.resourcemanager.hostname" to "172.31.3.150" in
yarn-site.xml on my client.)
Therefore it seems to me that it does not matter I have to recompile my job
with Hadoop 2 or not. The question is what should I do to enable submitting
my job remotely to the Hadoop 2 cluster ? What are the configurations I
need to set on the client side?
The only solution I can think of is to enable YARN on the Hadoop 2 cluster
but is it necessary?
I am running out of pointers and stuck 8-(
TIA
Kim
Re: Submit a Hadoop 1.1.1 job remotely to a Hadoop 2 cluster
Posted by Mike <mi...@unitedrmr.com>.
> On Apr 16, 2014, at 9:16 PM, Kim Chew <kc...@gmail.com> wrote:
>
> Vinod, I am confused here.
>
> So could you please explain what actually happened under the hood if "mapreduce.framework.name" is set to "classic" on the cluster side? Or it is supposed to be set to "yarn" in the first place?
>
> Thanks.
>
> Kim
>
>
>> On Wed, Apr 16, 2014 at 7:06 PM, Vinod Kumar Vavilapalli <vi...@apache.org> wrote:
>> You cannot run JobTracker/TaskTracker in Hadoop 2. It's neither supported nor even possible.
>>
>> +Vinod
>>
>>> On Apr 16, 2014, at 2:27 PM, Kim Chew <kc...@gmail.com> wrote:
>>>
>>> I have a cluster running Hadoop 2 but it is not running YARN, i.e. "mapreduce.framework.name" is set to "classic" therefore the ResourceManager is not running.
>>>
>>> On the Client side, I want to submit a job compiled with Hadoop-1.1.1 to the above cluster. Here how my Hadoop-1.1.1 mapred-site.xml looks like,
>>>
>>> <property>
>>> <!-- Pointed to the remote JobTracker -->
>>> <name>mapred.job.tracker</name>
>>> <value>172.31.3.150:8021</value>
>>> </property>
>>>
>>> Not surprisingly I got a version mismatched when I submit my job using the Hadoop-1.1.1 jars,
>>>
>>> org.apache.hadoop.ipc.RemoteException: Server IPC version 9 cannot communicate with client version 4
>>> at org.apache.hadoop.ipc.Client.call(Client.java:1107)
>>>
>>> So I recompiled my job with Hadoop 2 and submitted it using the Hadoop 2 jars. Here is how my Hadoop 2 mapred-site.xml looks like,
>>>
>>> <property>
>>> <!-- Pointed to the remote JobTracker -->
>>> <name>mapreduce.job.tracker.address</name>
>>> <value>172.31.3.150:8021</value>
>>> </property>
>>> <property>
>>> <name>mapreduce.framework.name</name>
>>> <value>yarn</value>
>>> </property>
>>>
>>> Note that I have to set "mapreduce.framework.name" to "yarn" otherwise the job will be run locally instead of on the targeted cluster. But my targeted cluster is not running YARN as stated above,
>>>
>>> 14/04/16 13:35:47 INFO client.RMProxy: Connecting to ResourceManager at /172.31.3.150:8032
>>> 14/04/16 13:35:49 INFO ipc.Client: Retrying connect to server: hadoop-host1.eng.narus.com/172.31.3.150:8032. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
>>>
>>> (Yes I have set my "yarn.resourcemanager.hostname" to "172.31.3.150" in yarn-site.xml on my client.)
>>>
>>> Therefore it seems to me that it does not matter I have to recompile my job with Hadoop 2 or not. The question is what should I do to enable submitting my job remotely to the Hadoop 2 cluster ? What are the configurations I need to set on the client side?
>>>
>>> The only solution I can think of is to enable YARN on the Hadoop 2 cluster but is it necessary?
>>>
>>> I am running out of pointers and stuck 8-(
>>>
>>> TIA
>>>
>>> Kim
>>
>>
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
>
Re: Submit a Hadoop 1.1.1 job remotely to a Hadoop 2 cluster
Posted by Mike <mi...@unitedrmr.com>.
> On Apr 16, 2014, at 9:16 PM, Kim Chew <kc...@gmail.com> wrote:
>
> Vinod, I am confused here.
>
> So could you please explain what actually happened under the hood if "mapreduce.framework.name" is set to "classic" on the cluster side? Or it is supposed to be set to "yarn" in the first place?
>
> Thanks.
>
> Kim
>
>
>> On Wed, Apr 16, 2014 at 7:06 PM, Vinod Kumar Vavilapalli <vi...@apache.org> wrote:
>> You cannot run JobTracker/TaskTracker in Hadoop 2. It's neither supported nor even possible.
>>
>> +Vinod
>>
>>> On Apr 16, 2014, at 2:27 PM, Kim Chew <kc...@gmail.com> wrote:
>>>
>>> I have a cluster running Hadoop 2 but it is not running YARN, i.e. "mapreduce.framework.name" is set to "classic" therefore the ResourceManager is not running.
>>>
>>> On the Client side, I want to submit a job compiled with Hadoop-1.1.1 to the above cluster. Here how my Hadoop-1.1.1 mapred-site.xml looks like,
>>>
>>> <property>
>>> <!-- Pointed to the remote JobTracker -->
>>> <name>mapred.job.tracker</name>
>>> <value>172.31.3.150:8021</value>
>>> </property>
>>>
>>> Not surprisingly I got a version mismatched when I submit my job using the Hadoop-1.1.1 jars,
>>>
>>> org.apache.hadoop.ipc.RemoteException: Server IPC version 9 cannot communicate with client version 4
>>> at org.apache.hadoop.ipc.Client.call(Client.java:1107)
>>>
>>> So I recompiled my job with Hadoop 2 and submitted it using the Hadoop 2 jars. Here is how my Hadoop 2 mapred-site.xml looks like,
>>>
>>> <property>
>>> <!-- Pointed to the remote JobTracker -->
>>> <name>mapreduce.job.tracker.address</name>
>>> <value>172.31.3.150:8021</value>
>>> </property>
>>> <property>
>>> <name>mapreduce.framework.name</name>
>>> <value>yarn</value>
>>> </property>
>>>
>>> Note that I have to set "mapreduce.framework.name" to "yarn" otherwise the job will be run locally instead of on the targeted cluster. But my targeted cluster is not running YARN as stated above,
>>>
>>> 14/04/16 13:35:47 INFO client.RMProxy: Connecting to ResourceManager at /172.31.3.150:8032
>>> 14/04/16 13:35:49 INFO ipc.Client: Retrying connect to server: hadoop-host1.eng.narus.com/172.31.3.150:8032. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
>>>
>>> (Yes I have set my "yarn.resourcemanager.hostname" to "172.31.3.150" in yarn-site.xml on my client.)
>>>
>>> Therefore it seems to me that it does not matter I have to recompile my job with Hadoop 2 or not. The question is what should I do to enable submitting my job remotely to the Hadoop 2 cluster ? What are the configurations I need to set on the client side?
>>>
>>> The only solution I can think of is to enable YARN on the Hadoop 2 cluster but is it necessary?
>>>
>>> I am running out of pointers and stuck 8-(
>>>
>>> TIA
>>>
>>> Kim
>>
>>
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
>
Re: Submit a Hadoop 1.1.1 job remotely to a Hadoop 2 cluster
Posted by Mike <mi...@unitedrmr.com>.
> On Apr 16, 2014, at 9:16 PM, Kim Chew <kc...@gmail.com> wrote:
>
> Vinod, I am confused here.
>
> So could you please explain what actually happened under the hood if "mapreduce.framework.name" is set to "classic" on the cluster side? Or it is supposed to be set to "yarn" in the first place?
>
> Thanks.
>
> Kim
>
>
>> On Wed, Apr 16, 2014 at 7:06 PM, Vinod Kumar Vavilapalli <vi...@apache.org> wrote:
>> You cannot run JobTracker/TaskTracker in Hadoop 2. It's neither supported nor even possible.
>>
>> +Vinod
>>
>>> On Apr 16, 2014, at 2:27 PM, Kim Chew <kc...@gmail.com> wrote:
>>>
>>> I have a cluster running Hadoop 2 but it is not running YARN, i.e. "mapreduce.framework.name" is set to "classic" therefore the ResourceManager is not running.
>>>
>>> On the Client side, I want to submit a job compiled with Hadoop-1.1.1 to the above cluster. Here how my Hadoop-1.1.1 mapred-site.xml looks like,
>>>
>>> <property>
>>> <!-- Pointed to the remote JobTracker -->
>>> <name>mapred.job.tracker</name>
>>> <value>172.31.3.150:8021</value>
>>> </property>
>>>
>>> Not surprisingly I got a version mismatched when I submit my job using the Hadoop-1.1.1 jars,
>>>
>>> org.apache.hadoop.ipc.RemoteException: Server IPC version 9 cannot communicate with client version 4
>>> at org.apache.hadoop.ipc.Client.call(Client.java:1107)
>>>
>>> So I recompiled my job with Hadoop 2 and submitted it using the Hadoop 2 jars. Here is how my Hadoop 2 mapred-site.xml looks like,
>>>
>>> <property>
>>> <!-- Pointed to the remote JobTracker -->
>>> <name>mapreduce.job.tracker.address</name>
>>> <value>172.31.3.150:8021</value>
>>> </property>
>>> <property>
>>> <name>mapreduce.framework.name</name>
>>> <value>yarn</value>
>>> </property>
>>>
>>> Note that I have to set "mapreduce.framework.name" to "yarn" otherwise the job will be run locally instead of on the targeted cluster. But my targeted cluster is not running YARN as stated above,
>>>
>>> 14/04/16 13:35:47 INFO client.RMProxy: Connecting to ResourceManager at /172.31.3.150:8032
>>> 14/04/16 13:35:49 INFO ipc.Client: Retrying connect to server: hadoop-host1.eng.narus.com/172.31.3.150:8032. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
>>>
>>> (Yes I have set my "yarn.resourcemanager.hostname" to "172.31.3.150" in yarn-site.xml on my client.)
>>>
>>> Therefore it seems to me that it does not matter I have to recompile my job with Hadoop 2 or not. The question is what should I do to enable submitting my job remotely to the Hadoop 2 cluster ? What are the configurations I need to set on the client side?
>>>
>>> The only solution I can think of is to enable YARN on the Hadoop 2 cluster but is it necessary?
>>>
>>> I am running out of pointers and stuck 8-(
>>>
>>> TIA
>>>
>>> Kim
>>
>>
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
>
Re: Submit a Hadoop 1.1.1 job remotely to a Hadoop 2 cluster
Posted by Mike <mi...@unitedrmr.com>.
> On Apr 16, 2014, at 9:16 PM, Kim Chew <kc...@gmail.com> wrote:
>
> Vinod, I am confused here.
>
> So could you please explain what actually happened under the hood if "mapreduce.framework.name" is set to "classic" on the cluster side? Or it is supposed to be set to "yarn" in the first place?
>
> Thanks.
>
> Kim
>
>
>> On Wed, Apr 16, 2014 at 7:06 PM, Vinod Kumar Vavilapalli <vi...@apache.org> wrote:
>> You cannot run JobTracker/TaskTracker in Hadoop 2. It's neither supported nor even possible.
>>
>> +Vinod
>>
>>> On Apr 16, 2014, at 2:27 PM, Kim Chew <kc...@gmail.com> wrote:
>>>
>>> I have a cluster running Hadoop 2 but it is not running YARN, i.e. "mapreduce.framework.name" is set to "classic" therefore the ResourceManager is not running.
>>>
>>> On the Client side, I want to submit a job compiled with Hadoop-1.1.1 to the above cluster. Here how my Hadoop-1.1.1 mapred-site.xml looks like,
>>>
>>> <property>
>>> <!-- Pointed to the remote JobTracker -->
>>> <name>mapred.job.tracker</name>
>>> <value>172.31.3.150:8021</value>
>>> </property>
>>>
>>> Not surprisingly I got a version mismatched when I submit my job using the Hadoop-1.1.1 jars,
>>>
>>> org.apache.hadoop.ipc.RemoteException: Server IPC version 9 cannot communicate with client version 4
>>> at org.apache.hadoop.ipc.Client.call(Client.java:1107)
>>>
>>> So I recompiled my job with Hadoop 2 and submitted it using the Hadoop 2 jars. Here is how my Hadoop 2 mapred-site.xml looks like,
>>>
>>> <property>
>>> <!-- Pointed to the remote JobTracker -->
>>> <name>mapreduce.job.tracker.address</name>
>>> <value>172.31.3.150:8021</value>
>>> </property>
>>> <property>
>>> <name>mapreduce.framework.name</name>
>>> <value>yarn</value>
>>> </property>
>>>
>>> Note that I have to set "mapreduce.framework.name" to "yarn" otherwise the job will be run locally instead of on the targeted cluster. But my targeted cluster is not running YARN as stated above,
>>>
>>> 14/04/16 13:35:47 INFO client.RMProxy: Connecting to ResourceManager at /172.31.3.150:8032
>>> 14/04/16 13:35:49 INFO ipc.Client: Retrying connect to server: hadoop-host1.eng.narus.com/172.31.3.150:8032. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
>>>
>>> (Yes I have set my "yarn.resourcemanager.hostname" to "172.31.3.150" in yarn-site.xml on my client.)
>>>
>>> Therefore it seems to me that it does not matter I have to recompile my job with Hadoop 2 or not. The question is what should I do to enable submitting my job remotely to the Hadoop 2 cluster ? What are the configurations I need to set on the client side?
>>>
>>> The only solution I can think of is to enable YARN on the Hadoop 2 cluster but is it necessary?
>>>
>>> I am running out of pointers and stuck 8-(
>>>
>>> TIA
>>>
>>> Kim
>>
>>
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
>
Re: Submit a Hadoop 1.1.1 job remotely to a Hadoop 2 cluster
Posted by Kim Chew <kc...@gmail.com>.
Vinod, I am confused here.
So could you please explain what actually happened under the hood if "
mapreduce.framework.name" is set to "classic" on the cluster side? Or it is
supposed to be set to "yarn" in the first place?
Thanks.
Kim
On Wed, Apr 16, 2014 at 7:06 PM, Vinod Kumar Vavilapalli <vinodkv@apache.org
> wrote:
> You cannot run JobTracker/TaskTracker in Hadoop 2. It's neither supported
> nor even possible.
>
> +Vinod
>
> On Apr 16, 2014, at 2:27 PM, Kim Chew <kc...@gmail.com> wrote:
>
> I have a cluster running Hadoop 2 but it is not running YARN, i.e. "
> mapreduce.framework.name" is set to "classic" therefore the
> ResourceManager is not running.
>
> On the Client side, I want to submit a job compiled with Hadoop-1.1.1 to
> the above cluster. Here how my Hadoop-1.1.1 mapred-site.xml looks like,
>
> <property>
> <!-- Pointed to the remote JobTracker -->
> <name>mapred.job.tracker</name>
> <value>172.31.3.150:8021</value>
> </property>
>
> Not surprisingly I got a version mismatched when I submit my job using the
> Hadoop-1.1.1 jars,
>
> org.apache.hadoop.ipc.RemoteException: Server IPC version 9 cannot
> communicate with client version 4
> at org.apache.hadoop.ipc.Client.call(Client.java:1107)
>
> So I recompiled my job with Hadoop 2 and submitted it using the Hadoop 2
> jars. Here is how my Hadoop 2 mapred-site.xml looks like,
>
> <property>
> <!-- Pointed to the remote JobTracker -->
> <name>mapreduce.job.tracker.address</name>
> <value>172.31.3.150:8021</value>
> </property>
> <property>
> <name>mapreduce.framework.name</name>
> <value>yarn</value>
> </property>
>
> Note that I have to set "mapreduce.framework.name" to "yarn" otherwise
> the job will be run locally instead of on the targeted cluster. But my
> targeted cluster is not running YARN as stated above,
>
> 14/04/16 13:35:47 INFO client.RMProxy: Connecting to ResourceManager at /
> 172.31.3.150:8032
> 14/04/16 13:35:49 INFO ipc.Client: Retrying connect to server:
> hadoop-host1.eng.narus.com/172.31.3.150:8032. Already tried 0 time(s);
> retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
> sleepTime=1 SECONDS)
>
> (Yes I have set my "yarn.resourcemanager.hostname" to "172.31.3.150" in
> yarn-site.xml on my client.)
>
> Therefore it seems to me that it does not matter I have to recompile my
> job with Hadoop 2 or not. The question is what should I do to enable
> submitting my job remotely to the Hadoop 2 cluster ? What are the
> configurations I need to set on the client side?
>
> The only solution I can think of is to enable YARN on the Hadoop 2 cluster
> but is it necessary?
>
> I am running out of pointers and stuck 8-(
>
> TIA
>
> Kim
>
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
Re: Submit a Hadoop 1.1.1 job remotely to a Hadoop 2 cluster
Posted by Kim Chew <kc...@gmail.com>.
Vinod, I am confused here.
So could you please explain what actually happened under the hood if "
mapreduce.framework.name" is set to "classic" on the cluster side? Or it is
supposed to be set to "yarn" in the first place?
Thanks.
Kim
On Wed, Apr 16, 2014 at 7:06 PM, Vinod Kumar Vavilapalli <vinodkv@apache.org
> wrote:
> You cannot run JobTracker/TaskTracker in Hadoop 2. It's neither supported
> nor even possible.
>
> +Vinod
>
> On Apr 16, 2014, at 2:27 PM, Kim Chew <kc...@gmail.com> wrote:
>
> I have a cluster running Hadoop 2 but it is not running YARN, i.e. "
> mapreduce.framework.name" is set to "classic" therefore the
> ResourceManager is not running.
>
> On the Client side, I want to submit a job compiled with Hadoop-1.1.1 to
> the above cluster. Here how my Hadoop-1.1.1 mapred-site.xml looks like,
>
> <property>
> <!-- Pointed to the remote JobTracker -->
> <name>mapred.job.tracker</name>
> <value>172.31.3.150:8021</value>
> </property>
>
> Not surprisingly I got a version mismatched when I submit my job using the
> Hadoop-1.1.1 jars,
>
> org.apache.hadoop.ipc.RemoteException: Server IPC version 9 cannot
> communicate with client version 4
> at org.apache.hadoop.ipc.Client.call(Client.java:1107)
>
> So I recompiled my job with Hadoop 2 and submitted it using the Hadoop 2
> jars. Here is how my Hadoop 2 mapred-site.xml looks like,
>
> <property>
> <!-- Pointed to the remote JobTracker -->
> <name>mapreduce.job.tracker.address</name>
> <value>172.31.3.150:8021</value>
> </property>
> <property>
> <name>mapreduce.framework.name</name>
> <value>yarn</value>
> </property>
>
> Note that I have to set "mapreduce.framework.name" to "yarn" otherwise
> the job will be run locally instead of on the targeted cluster. But my
> targeted cluster is not running YARN as stated above,
>
> 14/04/16 13:35:47 INFO client.RMProxy: Connecting to ResourceManager at /
> 172.31.3.150:8032
> 14/04/16 13:35:49 INFO ipc.Client: Retrying connect to server:
> hadoop-host1.eng.narus.com/172.31.3.150:8032. Already tried 0 time(s);
> retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
> sleepTime=1 SECONDS)
>
> (Yes I have set my "yarn.resourcemanager.hostname" to "172.31.3.150" in
> yarn-site.xml on my client.)
>
> Therefore it seems to me that it does not matter I have to recompile my
> job with Hadoop 2 or not. The question is what should I do to enable
> submitting my job remotely to the Hadoop 2 cluster ? What are the
> configurations I need to set on the client side?
>
> The only solution I can think of is to enable YARN on the Hadoop 2 cluster
> but is it necessary?
>
> I am running out of pointers and stuck 8-(
>
> TIA
>
> Kim
>
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
Re: Submit a Hadoop 1.1.1 job remotely to a Hadoop 2 cluster
Posted by Kim Chew <kc...@gmail.com>.
Vinod, I am confused here.
So could you please explain what actually happened under the hood if "
mapreduce.framework.name" is set to "classic" on the cluster side? Or it is
supposed to be set to "yarn" in the first place?
Thanks.
Kim
On Wed, Apr 16, 2014 at 7:06 PM, Vinod Kumar Vavilapalli <vinodkv@apache.org
> wrote:
> You cannot run JobTracker/TaskTracker in Hadoop 2. It's neither supported
> nor even possible.
>
> +Vinod
>
> On Apr 16, 2014, at 2:27 PM, Kim Chew <kc...@gmail.com> wrote:
>
> I have a cluster running Hadoop 2 but it is not running YARN, i.e. "
> mapreduce.framework.name" is set to "classic" therefore the
> ResourceManager is not running.
>
> On the Client side, I want to submit a job compiled with Hadoop-1.1.1 to
> the above cluster. Here how my Hadoop-1.1.1 mapred-site.xml looks like,
>
> <property>
> <!-- Pointed to the remote JobTracker -->
> <name>mapred.job.tracker</name>
> <value>172.31.3.150:8021</value>
> </property>
>
> Not surprisingly I got a version mismatched when I submit my job using the
> Hadoop-1.1.1 jars,
>
> org.apache.hadoop.ipc.RemoteException: Server IPC version 9 cannot
> communicate with client version 4
> at org.apache.hadoop.ipc.Client.call(Client.java:1107)
>
> So I recompiled my job with Hadoop 2 and submitted it using the Hadoop 2
> jars. Here is how my Hadoop 2 mapred-site.xml looks like,
>
> <property>
> <!-- Pointed to the remote JobTracker -->
> <name>mapreduce.job.tracker.address</name>
> <value>172.31.3.150:8021</value>
> </property>
> <property>
> <name>mapreduce.framework.name</name>
> <value>yarn</value>
> </property>
>
> Note that I have to set "mapreduce.framework.name" to "yarn" otherwise
> the job will be run locally instead of on the targeted cluster. But my
> targeted cluster is not running YARN as stated above,
>
> 14/04/16 13:35:47 INFO client.RMProxy: Connecting to ResourceManager at /
> 172.31.3.150:8032
> 14/04/16 13:35:49 INFO ipc.Client: Retrying connect to server:
> hadoop-host1.eng.narus.com/172.31.3.150:8032. Already tried 0 time(s);
> retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
> sleepTime=1 SECONDS)
>
> (Yes I have set my "yarn.resourcemanager.hostname" to "172.31.3.150" in
> yarn-site.xml on my client.)
>
> Therefore it seems to me that it does not matter I have to recompile my
> job with Hadoop 2 or not. The question is what should I do to enable
> submitting my job remotely to the Hadoop 2 cluster ? What are the
> configurations I need to set on the client side?
>
> The only solution I can think of is to enable YARN on the Hadoop 2 cluster
> but is it necessary?
>
> I am running out of pointers and stuck 8-(
>
> TIA
>
> Kim
>
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
Re: Submit a Hadoop 1.1.1 job remotely to a Hadoop 2 cluster
Posted by Kim Chew <kc...@gmail.com>.
Vinod, I am confused here.
So could you please explain what actually happened under the hood if "
mapreduce.framework.name" is set to "classic" on the cluster side? Or it is
supposed to be set to "yarn" in the first place?
Thanks.
Kim
On Wed, Apr 16, 2014 at 7:06 PM, Vinod Kumar Vavilapalli <vinodkv@apache.org
> wrote:
> You cannot run JobTracker/TaskTracker in Hadoop 2. It's neither supported
> nor even possible.
>
> +Vinod
>
> On Apr 16, 2014, at 2:27 PM, Kim Chew <kc...@gmail.com> wrote:
>
> I have a cluster running Hadoop 2 but it is not running YARN, i.e. "
> mapreduce.framework.name" is set to "classic" therefore the
> ResourceManager is not running.
>
> On the Client side, I want to submit a job compiled with Hadoop-1.1.1 to
> the above cluster. Here how my Hadoop-1.1.1 mapred-site.xml looks like,
>
> <property>
> <!-- Pointed to the remote JobTracker -->
> <name>mapred.job.tracker</name>
> <value>172.31.3.150:8021</value>
> </property>
>
> Not surprisingly I got a version mismatched when I submit my job using the
> Hadoop-1.1.1 jars,
>
> org.apache.hadoop.ipc.RemoteException: Server IPC version 9 cannot
> communicate with client version 4
> at org.apache.hadoop.ipc.Client.call(Client.java:1107)
>
> So I recompiled my job with Hadoop 2 and submitted it using the Hadoop 2
> jars. Here is how my Hadoop 2 mapred-site.xml looks like,
>
> <property>
> <!-- Pointed to the remote JobTracker -->
> <name>mapreduce.job.tracker.address</name>
> <value>172.31.3.150:8021</value>
> </property>
> <property>
> <name>mapreduce.framework.name</name>
> <value>yarn</value>
> </property>
>
> Note that I have to set "mapreduce.framework.name" to "yarn" otherwise
> the job will be run locally instead of on the targeted cluster. But my
> targeted cluster is not running YARN as stated above,
>
> 14/04/16 13:35:47 INFO client.RMProxy: Connecting to ResourceManager at /
> 172.31.3.150:8032
> 14/04/16 13:35:49 INFO ipc.Client: Retrying connect to server:
> hadoop-host1.eng.narus.com/172.31.3.150:8032. Already tried 0 time(s);
> retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
> sleepTime=1 SECONDS)
>
> (Yes I have set my "yarn.resourcemanager.hostname" to "172.31.3.150" in
> yarn-site.xml on my client.)
>
> Therefore it seems to me that it does not matter I have to recompile my
> job with Hadoop 2 or not. The question is what should I do to enable
> submitting my job remotely to the Hadoop 2 cluster ? What are the
> configurations I need to set on the client side?
>
> The only solution I can think of is to enable YARN on the Hadoop 2 cluster
> but is it necessary?
>
> I am running out of pointers and stuck 8-(
>
> TIA
>
> Kim
>
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
Re: Submit a Hadoop 1.1.1 job remotely to a Hadoop 2 cluster
Posted by Vinod Kumar Vavilapalli <vi...@apache.org>.
You cannot run JobTracker/TaskTracker in Hadoop 2. It's neither supported nor even possible.
+Vinod
On Apr 16, 2014, at 2:27 PM, Kim Chew <kc...@gmail.com> wrote:
> I have a cluster running Hadoop 2 but it is not running YARN, i.e. "mapreduce.framework.name" is set to "classic" therefore the ResourceManager is not running.
>
> On the Client side, I want to submit a job compiled with Hadoop-1.1.1 to the above cluster. Here how my Hadoop-1.1.1 mapred-site.xml looks like,
>
> <property>
> <!-- Pointed to the remote JobTracker -->
> <name>mapred.job.tracker</name>
> <value>172.31.3.150:8021</value>
> </property>
>
> Not surprisingly I got a version mismatched when I submit my job using the Hadoop-1.1.1 jars,
>
> org.apache.hadoop.ipc.RemoteException: Server IPC version 9 cannot communicate with client version 4
> at org.apache.hadoop.ipc.Client.call(Client.java:1107)
>
> So I recompiled my job with Hadoop 2 and submitted it using the Hadoop 2 jars. Here is how my Hadoop 2 mapred-site.xml looks like,
>
> <property>
> <!-- Pointed to the remote JobTracker -->
> <name>mapreduce.job.tracker.address</name>
> <value>172.31.3.150:8021</value>
> </property>
> <property>
> <name>mapreduce.framework.name</name>
> <value>yarn</value>
> </property>
>
> Note that I have to set "mapreduce.framework.name" to "yarn" otherwise the job will be run locally instead of on the targeted cluster. But my targeted cluster is not running YARN as stated above,
>
> 14/04/16 13:35:47 INFO client.RMProxy: Connecting to ResourceManager at /172.31.3.150:8032
> 14/04/16 13:35:49 INFO ipc.Client: Retrying connect to server: hadoop-host1.eng.narus.com/172.31.3.150:8032. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
>
> (Yes I have set my "yarn.resourcemanager.hostname" to "172.31.3.150" in yarn-site.xml on my client.)
>
> Therefore it seems to me that it does not matter I have to recompile my job with Hadoop 2 or not. The question is what should I do to enable submitting my job remotely to the Hadoop 2 cluster ? What are the configurations I need to set on the client side?
>
> The only solution I can think of is to enable YARN on the Hadoop 2 cluster but is it necessary?
>
> I am running out of pointers and stuck 8-(
>
> TIA
>
> Kim
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Submit a Hadoop 1.1.1 job remotely to a Hadoop 2 cluster
Posted by Vinod Kumar Vavilapalli <vi...@apache.org>.
You cannot run JobTracker/TaskTracker in Hadoop 2. It's neither supported nor even possible.
+Vinod
On Apr 16, 2014, at 2:27 PM, Kim Chew <kc...@gmail.com> wrote:
> I have a cluster running Hadoop 2 but it is not running YARN, i.e. "mapreduce.framework.name" is set to "classic" therefore the ResourceManager is not running.
>
> On the Client side, I want to submit a job compiled with Hadoop-1.1.1 to the above cluster. Here how my Hadoop-1.1.1 mapred-site.xml looks like,
>
> <property>
> <!-- Pointed to the remote JobTracker -->
> <name>mapred.job.tracker</name>
> <value>172.31.3.150:8021</value>
> </property>
>
> Not surprisingly I got a version mismatched when I submit my job using the Hadoop-1.1.1 jars,
>
> org.apache.hadoop.ipc.RemoteException: Server IPC version 9 cannot communicate with client version 4
> at org.apache.hadoop.ipc.Client.call(Client.java:1107)
>
> So I recompiled my job with Hadoop 2 and submitted it using the Hadoop 2 jars. Here is how my Hadoop 2 mapred-site.xml looks like,
>
> <property>
> <!-- Pointed to the remote JobTracker -->
> <name>mapreduce.job.tracker.address</name>
> <value>172.31.3.150:8021</value>
> </property>
> <property>
> <name>mapreduce.framework.name</name>
> <value>yarn</value>
> </property>
>
> Note that I have to set "mapreduce.framework.name" to "yarn" otherwise the job will be run locally instead of on the targeted cluster. But my targeted cluster is not running YARN as stated above,
>
> 14/04/16 13:35:47 INFO client.RMProxy: Connecting to ResourceManager at /172.31.3.150:8032
> 14/04/16 13:35:49 INFO ipc.Client: Retrying connect to server: hadoop-host1.eng.narus.com/172.31.3.150:8032. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
>
> (Yes I have set my "yarn.resourcemanager.hostname" to "172.31.3.150" in yarn-site.xml on my client.)
>
> Therefore it seems to me that it does not matter I have to recompile my job with Hadoop 2 or not. The question is what should I do to enable submitting my job remotely to the Hadoop 2 cluster ? What are the configurations I need to set on the client side?
>
> The only solution I can think of is to enable YARN on the Hadoop 2 cluster but is it necessary?
>
> I am running out of pointers and stuck 8-(
>
> TIA
>
> Kim
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Submit a Hadoop 1.1.1 job remotely to a Hadoop 2 cluster
Posted by Vinod Kumar Vavilapalli <vi...@apache.org>.
You cannot run JobTracker/TaskTracker in Hadoop 2. It's neither supported nor even possible.
+Vinod
On Apr 16, 2014, at 2:27 PM, Kim Chew <kc...@gmail.com> wrote:
> I have a cluster running Hadoop 2 but it is not running YARN, i.e. "mapreduce.framework.name" is set to "classic" therefore the ResourceManager is not running.
>
> On the Client side, I want to submit a job compiled with Hadoop-1.1.1 to the above cluster. Here how my Hadoop-1.1.1 mapred-site.xml looks like,
>
> <property>
> <!-- Pointed to the remote JobTracker -->
> <name>mapred.job.tracker</name>
> <value>172.31.3.150:8021</value>
> </property>
>
> Not surprisingly I got a version mismatched when I submit my job using the Hadoop-1.1.1 jars,
>
> org.apache.hadoop.ipc.RemoteException: Server IPC version 9 cannot communicate with client version 4
> at org.apache.hadoop.ipc.Client.call(Client.java:1107)
>
> So I recompiled my job with Hadoop 2 and submitted it using the Hadoop 2 jars. Here is how my Hadoop 2 mapred-site.xml looks like,
>
> <property>
> <!-- Pointed to the remote JobTracker -->
> <name>mapreduce.job.tracker.address</name>
> <value>172.31.3.150:8021</value>
> </property>
> <property>
> <name>mapreduce.framework.name</name>
> <value>yarn</value>
> </property>
>
> Note that I have to set "mapreduce.framework.name" to "yarn" otherwise the job will be run locally instead of on the targeted cluster. But my targeted cluster is not running YARN as stated above,
>
> 14/04/16 13:35:47 INFO client.RMProxy: Connecting to ResourceManager at /172.31.3.150:8032
> 14/04/16 13:35:49 INFO ipc.Client: Retrying connect to server: hadoop-host1.eng.narus.com/172.31.3.150:8032. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
>
> (Yes I have set my "yarn.resourcemanager.hostname" to "172.31.3.150" in yarn-site.xml on my client.)
>
> Therefore it seems to me that it does not matter I have to recompile my job with Hadoop 2 or not. The question is what should I do to enable submitting my job remotely to the Hadoop 2 cluster ? What are the configurations I need to set on the client side?
>
> The only solution I can think of is to enable YARN on the Hadoop 2 cluster but is it necessary?
>
> I am running out of pointers and stuck 8-(
>
> TIA
>
> Kim
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Submit a Hadoop 1.1.1 job remotely to a Hadoop 2 cluster
Posted by Vinod Kumar Vavilapalli <vi...@apache.org>.
You cannot run JobTracker/TaskTracker in Hadoop 2. It's neither supported nor even possible.
+Vinod
On Apr 16, 2014, at 2:27 PM, Kim Chew <kc...@gmail.com> wrote:
> I have a cluster running Hadoop 2 but it is not running YARN, i.e. "mapreduce.framework.name" is set to "classic" therefore the ResourceManager is not running.
>
> On the Client side, I want to submit a job compiled with Hadoop-1.1.1 to the above cluster. Here how my Hadoop-1.1.1 mapred-site.xml looks like,
>
> <property>
> <!-- Pointed to the remote JobTracker -->
> <name>mapred.job.tracker</name>
> <value>172.31.3.150:8021</value>
> </property>
>
> Not surprisingly I got a version mismatched when I submit my job using the Hadoop-1.1.1 jars,
>
> org.apache.hadoop.ipc.RemoteException: Server IPC version 9 cannot communicate with client version 4
> at org.apache.hadoop.ipc.Client.call(Client.java:1107)
>
> So I recompiled my job with Hadoop 2 and submitted it using the Hadoop 2 jars. Here is how my Hadoop 2 mapred-site.xml looks like,
>
> <property>
> <!-- Pointed to the remote JobTracker -->
> <name>mapreduce.job.tracker.address</name>
> <value>172.31.3.150:8021</value>
> </property>
> <property>
> <name>mapreduce.framework.name</name>
> <value>yarn</value>
> </property>
>
> Note that I have to set "mapreduce.framework.name" to "yarn" otherwise the job will be run locally instead of on the targeted cluster. But my targeted cluster is not running YARN as stated above,
>
> 14/04/16 13:35:47 INFO client.RMProxy: Connecting to ResourceManager at /172.31.3.150:8032
> 14/04/16 13:35:49 INFO ipc.Client: Retrying connect to server: hadoop-host1.eng.narus.com/172.31.3.150:8032. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
>
> (Yes I have set my "yarn.resourcemanager.hostname" to "172.31.3.150" in yarn-site.xml on my client.)
>
> Therefore it seems to me that it does not matter I have to recompile my job with Hadoop 2 or not. The question is what should I do to enable submitting my job remotely to the Hadoop 2 cluster ? What are the configurations I need to set on the client side?
>
> The only solution I can think of is to enable YARN on the Hadoop 2 cluster but is it necessary?
>
> I am running out of pointers and stuck 8-(
>
> TIA
>
> Kim
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.