You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by darekg11 <da...@tlen.pl> on 2015/11/17 16:54:49 UTC
Does MapReduceApplicationMaster prevents data node from spawning YarnChild?
Hello again dear users.
Today I ran into following problem:
My mini cluster consist of:
1 NameNode and 2 SlaveNodes.
When I ran my MapReduce program written in java with number of reducers equals to two.
As the result on first SlaveNode I got MRAppMaster task and only the second slave launched Yarn Child which actually was producing output results.
I understand that MRAppMaster is essential process repsonsible for managing life of given task.
And because of that single slave node can't launch MrAppMaster and Yarn Child at the same time or am I misunderstanding something?
RE: Does MapReduceApplicationMaster prevents data node from spawning YarnChild?
Posted by darekg11 <da...@tlen.pl>.
Thank You very much if it is the case then ApplicationMaster alone takes alomst everything that my datanode can offer - that would explain why task itself can't start. Thank You very much again.
Dnia 17 listopada 2015 20:40 Bikas Saha <bikas@apache.org> napisał(a):
You can check for “yarn.app.mapreduce.am.resource.mb” in your configs. Its default is 1.5GB and other MR task defaults are 1GB. From: darekg11 [mailto:darekg11@tlen.pl]
Sent: Tuesday, November 17, 2015 11:34 AM
To: user@hadoop.apache.org
Subject: RE: Does MapReduceApplicationMaster prevents data node from spawning YarnChild? Thank You very much, will check but the problem is that machines aren't really powerful - only 2GB of RAM and 2xCore each at 3.3GHz.
Do You maybe know approx value of required resources for AppMaster? And can we check current resource consumption of AppMaster?
Dnia 17 listopada 2015 18:40 Bikas Saha <bikas@apache.org> napisał(a):No. In general App Masters and their containers can be launched on any machine and both can be launched on the same machine. If your case happens repeatedly then you could check the RM UI, while the job is running, to see the maximum resource on a node manager and the resource currently assigned. Perhaps your node managers don’t have enough resources to run multiple containers? From: darekg11 [mailto:darekg11@tlen.pl]
Sent: Tuesday, November 17, 2015 7:55 AM
To: user@hadoop.apache.org
Subject: Does MapReduceApplicationMaster prevents data node from spawning YarnChild? Hello again dear users.
Today I ran into following problem:
My mini cluster consist of:
1 NameNode and 2 SlaveNodes.
When I ran my MapReduce program written in java with number of reducers equals to two.
As the result on first SlaveNode I got MRAppMaster task and only the second slave launched Yarn Child which actually was producing output results.
I understand that MRAppMaster is essential process repsonsible for managing life of given task.
And because of that single slave node can't launch MrAppMaster and Yarn Child at the same time or am I misunderstanding something?
RE: Does MapReduceApplicationMaster prevents data node from spawning YarnChild?
Posted by darekg11 <da...@tlen.pl>.
Thank You very much if it is the case then ApplicationMaster alone takes alomst everything that my datanode can offer - that would explain why task itself can't start. Thank You very much again.
Dnia 17 listopada 2015 20:40 Bikas Saha <bikas@apache.org> napisał(a):
You can check for “yarn.app.mapreduce.am.resource.mb” in your configs. Its default is 1.5GB and other MR task defaults are 1GB. From: darekg11 [mailto:darekg11@tlen.pl]
Sent: Tuesday, November 17, 2015 11:34 AM
To: user@hadoop.apache.org
Subject: RE: Does MapReduceApplicationMaster prevents data node from spawning YarnChild? Thank You very much, will check but the problem is that machines aren't really powerful - only 2GB of RAM and 2xCore each at 3.3GHz.
Do You maybe know approx value of required resources for AppMaster? And can we check current resource consumption of AppMaster?
Dnia 17 listopada 2015 18:40 Bikas Saha <bikas@apache.org> napisał(a):No. In general App Masters and their containers can be launched on any machine and both can be launched on the same machine. If your case happens repeatedly then you could check the RM UI, while the job is running, to see the maximum resource on a node manager and the resource currently assigned. Perhaps your node managers don’t have enough resources to run multiple containers? From: darekg11 [mailto:darekg11@tlen.pl]
Sent: Tuesday, November 17, 2015 7:55 AM
To: user@hadoop.apache.org
Subject: Does MapReduceApplicationMaster prevents data node from spawning YarnChild? Hello again dear users.
Today I ran into following problem:
My mini cluster consist of:
1 NameNode and 2 SlaveNodes.
When I ran my MapReduce program written in java with number of reducers equals to two.
As the result on first SlaveNode I got MRAppMaster task and only the second slave launched Yarn Child which actually was producing output results.
I understand that MRAppMaster is essential process repsonsible for managing life of given task.
And because of that single slave node can't launch MrAppMaster and Yarn Child at the same time or am I misunderstanding something?
RE: Does MapReduceApplicationMaster prevents data node from spawning YarnChild?
Posted by darekg11 <da...@tlen.pl>.
Thank You very much if it is the case then ApplicationMaster alone takes alomst everything that my datanode can offer - that would explain why task itself can't start. Thank You very much again.
Dnia 17 listopada 2015 20:40 Bikas Saha <bikas@apache.org> napisał(a):
You can check for “yarn.app.mapreduce.am.resource.mb” in your configs. Its default is 1.5GB and other MR task defaults are 1GB. From: darekg11 [mailto:darekg11@tlen.pl]
Sent: Tuesday, November 17, 2015 11:34 AM
To: user@hadoop.apache.org
Subject: RE: Does MapReduceApplicationMaster prevents data node from spawning YarnChild? Thank You very much, will check but the problem is that machines aren't really powerful - only 2GB of RAM and 2xCore each at 3.3GHz.
Do You maybe know approx value of required resources for AppMaster? And can we check current resource consumption of AppMaster?
Dnia 17 listopada 2015 18:40 Bikas Saha <bikas@apache.org> napisał(a):No. In general App Masters and their containers can be launched on any machine and both can be launched on the same machine. If your case happens repeatedly then you could check the RM UI, while the job is running, to see the maximum resource on a node manager and the resource currently assigned. Perhaps your node managers don’t have enough resources to run multiple containers? From: darekg11 [mailto:darekg11@tlen.pl]
Sent: Tuesday, November 17, 2015 7:55 AM
To: user@hadoop.apache.org
Subject: Does MapReduceApplicationMaster prevents data node from spawning YarnChild? Hello again dear users.
Today I ran into following problem:
My mini cluster consist of:
1 NameNode and 2 SlaveNodes.
When I ran my MapReduce program written in java with number of reducers equals to two.
As the result on first SlaveNode I got MRAppMaster task and only the second slave launched Yarn Child which actually was producing output results.
I understand that MRAppMaster is essential process repsonsible for managing life of given task.
And because of that single slave node can't launch MrAppMaster and Yarn Child at the same time or am I misunderstanding something?
RE: Does MapReduceApplicationMaster prevents data node from spawning YarnChild?
Posted by darekg11 <da...@tlen.pl>.
Thank You very much if it is the case then ApplicationMaster alone takes alomst everything that my datanode can offer - that would explain why task itself can't start. Thank You very much again.
Dnia 17 listopada 2015 20:40 Bikas Saha <bikas@apache.org> napisał(a):
You can check for “yarn.app.mapreduce.am.resource.mb” in your configs. Its default is 1.5GB and other MR task defaults are 1GB. From: darekg11 [mailto:darekg11@tlen.pl]
Sent: Tuesday, November 17, 2015 11:34 AM
To: user@hadoop.apache.org
Subject: RE: Does MapReduceApplicationMaster prevents data node from spawning YarnChild? Thank You very much, will check but the problem is that machines aren't really powerful - only 2GB of RAM and 2xCore each at 3.3GHz.
Do You maybe know approx value of required resources for AppMaster? And can we check current resource consumption of AppMaster?
Dnia 17 listopada 2015 18:40 Bikas Saha <bikas@apache.org> napisał(a):No. In general App Masters and their containers can be launched on any machine and both can be launched on the same machine. If your case happens repeatedly then you could check the RM UI, while the job is running, to see the maximum resource on a node manager and the resource currently assigned. Perhaps your node managers don’t have enough resources to run multiple containers? From: darekg11 [mailto:darekg11@tlen.pl]
Sent: Tuesday, November 17, 2015 7:55 AM
To: user@hadoop.apache.org
Subject: Does MapReduceApplicationMaster prevents data node from spawning YarnChild? Hello again dear users.
Today I ran into following problem:
My mini cluster consist of:
1 NameNode and 2 SlaveNodes.
When I ran my MapReduce program written in java with number of reducers equals to two.
As the result on first SlaveNode I got MRAppMaster task and only the second slave launched Yarn Child which actually was producing output results.
I understand that MRAppMaster is essential process repsonsible for managing life of given task.
And because of that single slave node can't launch MrAppMaster and Yarn Child at the same time or am I misunderstanding something?
RE: Does MapReduceApplicationMaster prevents data node from spawning YarnChild?
Posted by Bikas Saha <bi...@apache.org>.
You can check for “yarn.app.mapreduce.am.resource.mb” in your configs. Its default is 1.5GB and other MR task defaults are 1GB.
From: darekg11 [mailto:darekg11@tlen.pl]
Sent: Tuesday, November 17, 2015 11:34 AM
To: user@hadoop.apache.org
Subject: RE: Does MapReduceApplicationMaster prevents data node from spawning YarnChild?
Thank You very much, will check but the problem is that machines aren't really powerful - only 2GB of RAM and 2xCore each at 3.3GHz.
Do You maybe know approx value of required resources for AppMaster? And can we check current resource consumption of AppMaster?
Dnia 17 listopada 2015 18:40 Bikas Saha <bikas@apache.org <ma...@apache.org> > napisał(a):
No. In general App Masters and their containers can be launched on any machine and both can be launched on the same machine.
If your case happens repeatedly then you could check the RM UI, while the job is running, to see the maximum resource on a node manager and the resource currently assigned. Perhaps your node managers don’t have enough resources to run multiple containers?
From: darekg11 [mailto:darekg11@tlen.pl]
Sent: Tuesday, November 17, 2015 7:55 AM
To: user@hadoop.apache.org <ma...@hadoop.apache.org>
Subject: Does MapReduceApplicationMaster prevents data node from spawning YarnChild?
Hello again dear users.
Today I ran into following problem:
My mini cluster consist of:
1 NameNode and 2 SlaveNodes.
When I ran my MapReduce program written in java with number of reducers equals to two.
As the result on first SlaveNode I got MRAppMaster task and only the second slave launched Yarn Child which actually was producing output results.
I understand that MRAppMaster is essential process repsonsible for managing life of given task.
And because of that single slave node can't launch MrAppMaster and Yarn Child at the same time or am I misunderstanding something?
RE: Does MapReduceApplicationMaster prevents data node from spawning YarnChild?
Posted by Bikas Saha <bi...@apache.org>.
You can check for “yarn.app.mapreduce.am.resource.mb” in your configs. Its default is 1.5GB and other MR task defaults are 1GB.
From: darekg11 [mailto:darekg11@tlen.pl]
Sent: Tuesday, November 17, 2015 11:34 AM
To: user@hadoop.apache.org
Subject: RE: Does MapReduceApplicationMaster prevents data node from spawning YarnChild?
Thank You very much, will check but the problem is that machines aren't really powerful - only 2GB of RAM and 2xCore each at 3.3GHz.
Do You maybe know approx value of required resources for AppMaster? And can we check current resource consumption of AppMaster?
Dnia 17 listopada 2015 18:40 Bikas Saha <bikas@apache.org <ma...@apache.org> > napisał(a):
No. In general App Masters and their containers can be launched on any machine and both can be launched on the same machine.
If your case happens repeatedly then you could check the RM UI, while the job is running, to see the maximum resource on a node manager and the resource currently assigned. Perhaps your node managers don’t have enough resources to run multiple containers?
From: darekg11 [mailto:darekg11@tlen.pl]
Sent: Tuesday, November 17, 2015 7:55 AM
To: user@hadoop.apache.org <ma...@hadoop.apache.org>
Subject: Does MapReduceApplicationMaster prevents data node from spawning YarnChild?
Hello again dear users.
Today I ran into following problem:
My mini cluster consist of:
1 NameNode and 2 SlaveNodes.
When I ran my MapReduce program written in java with number of reducers equals to two.
As the result on first SlaveNode I got MRAppMaster task and only the second slave launched Yarn Child which actually was producing output results.
I understand that MRAppMaster is essential process repsonsible for managing life of given task.
And because of that single slave node can't launch MrAppMaster and Yarn Child at the same time or am I misunderstanding something?
RE: Does MapReduceApplicationMaster prevents data node from spawning YarnChild?
Posted by Bikas Saha <bi...@apache.org>.
You can check for “yarn.app.mapreduce.am.resource.mb” in your configs. Its default is 1.5GB and other MR task defaults are 1GB.
From: darekg11 [mailto:darekg11@tlen.pl]
Sent: Tuesday, November 17, 2015 11:34 AM
To: user@hadoop.apache.org
Subject: RE: Does MapReduceApplicationMaster prevents data node from spawning YarnChild?
Thank You very much, will check but the problem is that machines aren't really powerful - only 2GB of RAM and 2xCore each at 3.3GHz.
Do You maybe know approx value of required resources for AppMaster? And can we check current resource consumption of AppMaster?
Dnia 17 listopada 2015 18:40 Bikas Saha <bikas@apache.org <ma...@apache.org> > napisał(a):
No. In general App Masters and their containers can be launched on any machine and both can be launched on the same machine.
If your case happens repeatedly then you could check the RM UI, while the job is running, to see the maximum resource on a node manager and the resource currently assigned. Perhaps your node managers don’t have enough resources to run multiple containers?
From: darekg11 [mailto:darekg11@tlen.pl]
Sent: Tuesday, November 17, 2015 7:55 AM
To: user@hadoop.apache.org <ma...@hadoop.apache.org>
Subject: Does MapReduceApplicationMaster prevents data node from spawning YarnChild?
Hello again dear users.
Today I ran into following problem:
My mini cluster consist of:
1 NameNode and 2 SlaveNodes.
When I ran my MapReduce program written in java with number of reducers equals to two.
As the result on first SlaveNode I got MRAppMaster task and only the second slave launched Yarn Child which actually was producing output results.
I understand that MRAppMaster is essential process repsonsible for managing life of given task.
And because of that single slave node can't launch MrAppMaster and Yarn Child at the same time or am I misunderstanding something?
RE: Does MapReduceApplicationMaster prevents data node from spawning YarnChild?
Posted by Bikas Saha <bi...@apache.org>.
You can check for “yarn.app.mapreduce.am.resource.mb” in your configs. Its default is 1.5GB and other MR task defaults are 1GB.
From: darekg11 [mailto:darekg11@tlen.pl]
Sent: Tuesday, November 17, 2015 11:34 AM
To: user@hadoop.apache.org
Subject: RE: Does MapReduceApplicationMaster prevents data node from spawning YarnChild?
Thank You very much, will check but the problem is that machines aren't really powerful - only 2GB of RAM and 2xCore each at 3.3GHz.
Do You maybe know approx value of required resources for AppMaster? And can we check current resource consumption of AppMaster?
Dnia 17 listopada 2015 18:40 Bikas Saha <bikas@apache.org <ma...@apache.org> > napisał(a):
No. In general App Masters and their containers can be launched on any machine and both can be launched on the same machine.
If your case happens repeatedly then you could check the RM UI, while the job is running, to see the maximum resource on a node manager and the resource currently assigned. Perhaps your node managers don’t have enough resources to run multiple containers?
From: darekg11 [mailto:darekg11@tlen.pl]
Sent: Tuesday, November 17, 2015 7:55 AM
To: user@hadoop.apache.org <ma...@hadoop.apache.org>
Subject: Does MapReduceApplicationMaster prevents data node from spawning YarnChild?
Hello again dear users.
Today I ran into following problem:
My mini cluster consist of:
1 NameNode and 2 SlaveNodes.
When I ran my MapReduce program written in java with number of reducers equals to two.
As the result on first SlaveNode I got MRAppMaster task and only the second slave launched Yarn Child which actually was producing output results.
I understand that MRAppMaster is essential process repsonsible for managing life of given task.
And because of that single slave node can't launch MrAppMaster and Yarn Child at the same time or am I misunderstanding something?
RE: Does MapReduceApplicationMaster prevents data node from spawning YarnChild?
Posted by darekg11 <da...@tlen.pl>.
Thank You very much, will check but the problem is that machines aren't really powerful - only 2GB of RAM and 2xCore each at 3.3GHz.
Do You maybe know approx value of required resources for AppMaster? And can we check current resource consumption of AppMaster?
Dnia 17 listopada 2015 18:40 Bikas Saha <bikas@apache.org> napisał(a):
No. In general App Masters and their containers can be launched on any machine and both can be launched on the same machine. If your case happens repeatedly then you could check the RM UI, while the job is running, to see the maximum resource on a node manager and the resource currently assigned. Perhaps your node managers don’t have enough resources to run multiple containers? From: darekg11 [mailto:darekg11@tlen.pl]
Sent: Tuesday, November 17, 2015 7:55 AM
To: user@hadoop.apache.org
Subject: Does MapReduceApplicationMaster prevents data node from spawning YarnChild? Hello again dear users.
Today I ran into following problem:
My mini cluster consist of:
1 NameNode and 2 SlaveNodes.
When I ran my MapReduce program written in java with number of reducers equals to two.
As the result on first SlaveNode I got MRAppMaster task and only the second slave launched Yarn Child which actually was producing output results.
I understand that MRAppMaster is essential process repsonsible for managing life of given task.
And because of that single slave node can't launch MrAppMaster and Yarn Child at the same time or am I misunderstanding something?
RE: Does MapReduceApplicationMaster prevents data node from spawning YarnChild?
Posted by darekg11 <da...@tlen.pl>.
Thank You very much, will check but the problem is that machines aren't really powerful - only 2GB of RAM and 2xCore each at 3.3GHz.
Do You maybe know approx value of required resources for AppMaster? And can we check current resource consumption of AppMaster?
Dnia 17 listopada 2015 18:40 Bikas Saha <bikas@apache.org> napisał(a):
No. In general App Masters and their containers can be launched on any machine and both can be launched on the same machine. If your case happens repeatedly then you could check the RM UI, while the job is running, to see the maximum resource on a node manager and the resource currently assigned. Perhaps your node managers don’t have enough resources to run multiple containers? From: darekg11 [mailto:darekg11@tlen.pl]
Sent: Tuesday, November 17, 2015 7:55 AM
To: user@hadoop.apache.org
Subject: Does MapReduceApplicationMaster prevents data node from spawning YarnChild? Hello again dear users.
Today I ran into following problem:
My mini cluster consist of:
1 NameNode and 2 SlaveNodes.
When I ran my MapReduce program written in java with number of reducers equals to two.
As the result on first SlaveNode I got MRAppMaster task and only the second slave launched Yarn Child which actually was producing output results.
I understand that MRAppMaster is essential process repsonsible for managing life of given task.
And because of that single slave node can't launch MrAppMaster and Yarn Child at the same time or am I misunderstanding something?
RE: Does MapReduceApplicationMaster prevents data node from spawning YarnChild?
Posted by darekg11 <da...@tlen.pl>.
Thank You very much, will check but the problem is that machines aren't really powerful - only 2GB of RAM and 2xCore each at 3.3GHz.
Do You maybe know approx value of required resources for AppMaster? And can we check current resource consumption of AppMaster?
Dnia 17 listopada 2015 18:40 Bikas Saha <bikas@apache.org> napisał(a):
No. In general App Masters and their containers can be launched on any machine and both can be launched on the same machine. If your case happens repeatedly then you could check the RM UI, while the job is running, to see the maximum resource on a node manager and the resource currently assigned. Perhaps your node managers don’t have enough resources to run multiple containers? From: darekg11 [mailto:darekg11@tlen.pl]
Sent: Tuesday, November 17, 2015 7:55 AM
To: user@hadoop.apache.org
Subject: Does MapReduceApplicationMaster prevents data node from spawning YarnChild? Hello again dear users.
Today I ran into following problem:
My mini cluster consist of:
1 NameNode and 2 SlaveNodes.
When I ran my MapReduce program written in java with number of reducers equals to two.
As the result on first SlaveNode I got MRAppMaster task and only the second slave launched Yarn Child which actually was producing output results.
I understand that MRAppMaster is essential process repsonsible for managing life of given task.
And because of that single slave node can't launch MrAppMaster and Yarn Child at the same time or am I misunderstanding something?
RE: Does MapReduceApplicationMaster prevents data node from spawning YarnChild?
Posted by darekg11 <da...@tlen.pl>.
Thank You very much, will check but the problem is that machines aren't really powerful - only 2GB of RAM and 2xCore each at 3.3GHz.
Do You maybe know approx value of required resources for AppMaster? And can we check current resource consumption of AppMaster?
Dnia 17 listopada 2015 18:40 Bikas Saha <bikas@apache.org> napisał(a):
No. In general App Masters and their containers can be launched on any machine and both can be launched on the same machine. If your case happens repeatedly then you could check the RM UI, while the job is running, to see the maximum resource on a node manager and the resource currently assigned. Perhaps your node managers don’t have enough resources to run multiple containers? From: darekg11 [mailto:darekg11@tlen.pl]
Sent: Tuesday, November 17, 2015 7:55 AM
To: user@hadoop.apache.org
Subject: Does MapReduceApplicationMaster prevents data node from spawning YarnChild? Hello again dear users.
Today I ran into following problem:
My mini cluster consist of:
1 NameNode and 2 SlaveNodes.
When I ran my MapReduce program written in java with number of reducers equals to two.
As the result on first SlaveNode I got MRAppMaster task and only the second slave launched Yarn Child which actually was producing output results.
I understand that MRAppMaster is essential process repsonsible for managing life of given task.
And because of that single slave node can't launch MrAppMaster and Yarn Child at the same time or am I misunderstanding something?
RE: Does MapReduceApplicationMaster prevents data node from spawning YarnChild?
Posted by Bikas Saha <bi...@apache.org>.
No. In general App Masters and their containers can be launched on any machine and both can be launched on the same machine.
If your case happens repeatedly then you could check the RM UI, while the job is running, to see the maximum resource on a node manager and the resource currently assigned. Perhaps your node managers don’t have enough resources to run multiple containers?
From: darekg11 [mailto:darekg11@tlen.pl]
Sent: Tuesday, November 17, 2015 7:55 AM
To: user@hadoop.apache.org
Subject: Does MapReduceApplicationMaster prevents data node from spawning YarnChild?
Hello again dear users.
Today I ran into following problem:
My mini cluster consist of:
1 NameNode and 2 SlaveNodes.
When I ran my MapReduce program written in java with number of reducers equals to two.
As the result on first SlaveNode I got MRAppMaster task and only the second slave launched Yarn Child which actually was producing output results.
I understand that MRAppMaster is essential process repsonsible for managing life of given task.
And because of that single slave node can't launch MrAppMaster and Yarn Child at the same time or am I misunderstanding something?
RE: Does MapReduceApplicationMaster prevents data node from spawning YarnChild?
Posted by Bikas Saha <bi...@apache.org>.
No. In general App Masters and their containers can be launched on any machine and both can be launched on the same machine.
If your case happens repeatedly then you could check the RM UI, while the job is running, to see the maximum resource on a node manager and the resource currently assigned. Perhaps your node managers don’t have enough resources to run multiple containers?
From: darekg11 [mailto:darekg11@tlen.pl]
Sent: Tuesday, November 17, 2015 7:55 AM
To: user@hadoop.apache.org
Subject: Does MapReduceApplicationMaster prevents data node from spawning YarnChild?
Hello again dear users.
Today I ran into following problem:
My mini cluster consist of:
1 NameNode and 2 SlaveNodes.
When I ran my MapReduce program written in java with number of reducers equals to two.
As the result on first SlaveNode I got MRAppMaster task and only the second slave launched Yarn Child which actually was producing output results.
I understand that MRAppMaster is essential process repsonsible for managing life of given task.
And because of that single slave node can't launch MrAppMaster and Yarn Child at the same time or am I misunderstanding something?
RE: Does MapReduceApplicationMaster prevents data node from spawning YarnChild?
Posted by Bikas Saha <bi...@apache.org>.
No. In general App Masters and their containers can be launched on any machine and both can be launched on the same machine.
If your case happens repeatedly then you could check the RM UI, while the job is running, to see the maximum resource on a node manager and the resource currently assigned. Perhaps your node managers don’t have enough resources to run multiple containers?
From: darekg11 [mailto:darekg11@tlen.pl]
Sent: Tuesday, November 17, 2015 7:55 AM
To: user@hadoop.apache.org
Subject: Does MapReduceApplicationMaster prevents data node from spawning YarnChild?
Hello again dear users.
Today I ran into following problem:
My mini cluster consist of:
1 NameNode and 2 SlaveNodes.
When I ran my MapReduce program written in java with number of reducers equals to two.
As the result on first SlaveNode I got MRAppMaster task and only the second slave launched Yarn Child which actually was producing output results.
I understand that MRAppMaster is essential process repsonsible for managing life of given task.
And because of that single slave node can't launch MrAppMaster and Yarn Child at the same time or am I misunderstanding something?
RE: Does MapReduceApplicationMaster prevents data node from spawning YarnChild?
Posted by Bikas Saha <bi...@apache.org>.
No. In general App Masters and their containers can be launched on any machine and both can be launched on the same machine.
If your case happens repeatedly then you could check the RM UI, while the job is running, to see the maximum resource on a node manager and the resource currently assigned. Perhaps your node managers don’t have enough resources to run multiple containers?
From: darekg11 [mailto:darekg11@tlen.pl]
Sent: Tuesday, November 17, 2015 7:55 AM
To: user@hadoop.apache.org
Subject: Does MapReduceApplicationMaster prevents data node from spawning YarnChild?
Hello again dear users.
Today I ran into following problem:
My mini cluster consist of:
1 NameNode and 2 SlaveNodes.
When I ran my MapReduce program written in java with number of reducers equals to two.
As the result on first SlaveNode I got MRAppMaster task and only the second slave launched Yarn Child which actually was producing output results.
I understand that MRAppMaster is essential process repsonsible for managing life of given task.
And because of that single slave node can't launch MrAppMaster and Yarn Child at the same time or am I misunderstanding something?