You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Radim Kolar <hs...@filez.com> on 2012/04/26 19:56:14 UTC
max 1 mapper per node
Its possible to configure in YARN scheduling requirement to not run more
than 1 mapper per node?
Its not because of memory requirements. Who decides this kind of
scheduling? ResourceManager or ApplicationManager?
Re: Terasort
Posted by Robert Evans <ev...@yahoo-inc.com>.
Jeff,
Inside the MR AM there are a number of priorities given to the various tasks request when handing them to the RM for scheduling. Map tasks have a higher priority than reduce tasks do. That means that the scheduler should never return a container for a reduce task until all outstanding map task requests have been satisfied. That is the real problem that you are seeing. I cannot say completely the reasoning behind this, but I assume that it is in part because of the dependency ordering between the two tasks. Apparently in your case having the map tasks finish faster because they have more free slots does not make up for the extra time it takes for the reducers to launch. Which is a little odd because I don't believe that we have seen the same slowdown in our terrasort benchmarks, but I will have to verify that.
Perhaps what you want to look into then is to vary the priority of the requests. Be careful though, because the request priority is slightly abused to also be used as a cookie indicating the type of request. So just setting the map and reduce task priorities to be the same is also going to require some changes in the code that matches up the assigned containers to the outstanding tasks.
Alternatively you could change the block size/splits so that the total number of map tasks + the total number of reduce tasks = the total number of slots in your cluster. That way once all of the map tasks are satisfied the reduce tasks can be launched. Each map task would process more data then it did before, so it might not be as fast as previous tasks were, but hopefully that will not be too bad.
--Bobby Evans
On 5/10/12 6:16 PM, "Jeffrey Buell" <jb...@vmware.com> wrote:
I have slowstart set to 0.05 (I think that is the default). In MR1 all of the reducers start when 5% of the maps finish (expected and desired behavior). This allows the shuffle phase to keep up with the maps, and it completes soon after all maps are finished. In MR2 (YARN), one reducer starts at that point, and the others start slowly over time. When the maps are all finished, not much of the shuffle work is done, hence the increased elapsed time.
Will slowstart=0 will behave much differently than 0.05? But let's say slowstart=0 and you have 4000 map tasks, 100 reduce tasks, and 200 slots. How many tasks of each kind would you expect to occupy those slots initially? Equal weighting to map and reduce would mean 100 of each. Equal weighting to all tasks ready to run means 195 maps and 5 reduces. That's a big difference in behavior.
Jeff
From: Arun C Murthy [mailto:acm@hortonworks.com]
Sent: Thursday, May 10, 2012 3:50 PM
To: mapreduce-user@hadoop.apache.org
Subject: Terasort
Changing subject...
On May 10, 2012, at 3:40 PM, Jeffrey Buell wrote:
I have the right #slots to fill up memory across the cluster, and all those slots are filled with tasks. The problem I ran into was that the maps grabbed all the slots initially and the reduces had a hard time getting started. As maps finished, more maps were started and only rarely was a reduce started. I assume this behavior occurred because I had ~4000 map tasks in the queue, but only ~100 reduce tasks. If the scheduler lumps maps and reduces together, then whenever a slot opens up it will almost surely be taken by a map task. To get good performance I need all reduce tasks started early on, and have only map tasks compete for open slots. Other apps may need different priorities between maps and reduces. In any case, I don't understand how treating maps and reduces the same is workable.
Are you playing with YARN or MR1?
IAC, you are getting hit by 'slowstart' for reduces where-in reduces aren't scheduled till sufficient % of maps are completed.
Set mapred.reduce.slowstart.completed.maps to 0. (That should work for either MR1 or MR2).
Arun
Jeff
From: Arun C Murthy [mailto:acm@hortonworks.com]
Sent: Thursday, May 10, 2012 1:27 PM
To: mapreduce-user@hadoop.apache.org
Subject: Re: max 1 mapper per node
For terasort you want to fill up your entire cluster with maps/reduces as fast as you can to get the best performance.
Just play with #slots.
Arun
On May 9, 2012, at 12:36 PM, Jeffrey Buell wrote:
Not to speak for Radim, but what I'm trying to achieve is performance at least as good as 0.20 for all cases. That is, no regressions. For something as simple as terasort, I don't think that is possible without being able to specify the max number of mappers/reducers per node. As it is, I see slowdowns as much as 2X. Hopefully I'm wrong and somebody will straighten me out. But if I'm not, adding such a feature won't lead to bad behavior of any kind since the default could be set to unlimited and thus have no effect whatsoever.
I should emphasize that I support the goal of greater automation since Hadoop has way too many parameters and is so hard to tune. Just not at the expense of performance regressions.
Jeff
We've been against these 'features' since it leads to very bad behaviour across the cluster with multiple apps/users etc.
What is your use-case i.e. what are you trying to achieve with this?
thanks,
Arun
On May 3, 2012, at 5:59 AM, Radim Kolar wrote:
if plugin system for AM is overkill, something simpler can be made like:
maximum number of mappers per node
maximum number of reducers per node
maximum percentage of non data local tasks
maximum percentage of rack local tasks
and set this in job properties.
--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/
--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/
RE: Terasort
Posted by Jeffrey Buell <jb...@vmware.com>.
I have slowstart set to 0.05 (I think that is the default). In MR1 all of the reducers start when 5% of the maps finish (expected and desired behavior). This allows the shuffle phase to keep up with the maps, and it completes soon after all maps are finished. In MR2 (YARN), one reducer starts at that point, and the others start slowly over time. When the maps are all finished, not much of the shuffle work is done, hence the increased elapsed time.
Will slowstart=0 will behave much differently than 0.05? But let's say slowstart=0 and you have 4000 map tasks, 100 reduce tasks, and 200 slots. How many tasks of each kind would you expect to occupy those slots initially? Equal weighting to map and reduce would mean 100 of each. Equal weighting to all tasks ready to run means 195 maps and 5 reduces. That's a big difference in behavior.
Jeff
From: Arun C Murthy [mailto:acm@hortonworks.com]
Sent: Thursday, May 10, 2012 3:50 PM
To: mapreduce-user@hadoop.apache.org
Subject: Terasort
Changing subject...
On May 10, 2012, at 3:40 PM, Jeffrey Buell wrote:
I have the right #slots to fill up memory across the cluster, and all those slots are filled with tasks. The problem I ran into was that the maps grabbed all the slots initially and the reduces had a hard time getting started. As maps finished, more maps were started and only rarely was a reduce started. I assume this behavior occurred because I had ~4000 map tasks in the queue, but only ~100 reduce tasks. If the scheduler lumps maps and reduces together, then whenever a slot opens up it will almost surely be taken by a map task. To get good performance I need all reduce tasks started early on, and have only map tasks compete for open slots. Other apps may need different priorities between maps and reduces. In any case, I don't understand how treating maps and reduces the same is workable.
Are you playing with YARN or MR1?
IAC, you are getting hit by 'slowstart' for reduces where-in reduces aren't scheduled till sufficient % of maps are completed.
Set mapred.reduce.slowstart.completed.maps to 0. (That should work for either MR1 or MR2).
Arun
Jeff
From: Arun C Murthy [mailto:acm@hortonworks.com]
Sent: Thursday, May 10, 2012 1:27 PM
To: mapreduce-user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Re: max 1 mapper per node
For terasort you want to fill up your entire cluster with maps/reduces as fast as you can to get the best performance.
Just play with #slots.
Arun
On May 9, 2012, at 12:36 PM, Jeffrey Buell wrote:
Not to speak for Radim, but what I'm trying to achieve is performance at least as good as 0.20 for all cases. That is, no regressions. For something as simple as terasort, I don't think that is possible without being able to specify the max number of mappers/reducers per node. As it is, I see slowdowns as much as 2X. Hopefully I'm wrong and somebody will straighten me out. But if I'm not, adding such a feature won't lead to bad behavior of any kind since the default could be set to unlimited and thus have no effect whatsoever.
I should emphasize that I support the goal of greater automation since Hadoop has way too many parameters and is so hard to tune. Just not at the expense of performance regressions.
Jeff
We've been against these 'features' since it leads to very bad behaviour across the cluster with multiple apps/users etc.
What is your use-case i.e. what are you trying to achieve with this?
thanks,
Arun
On May 3, 2012, at 5:59 AM, Radim Kolar wrote:
if plugin system for AM is overkill, something simpler can be made like:
maximum number of mappers per node
maximum number of reducers per node
maximum percentage of non data local tasks
maximum percentage of rack local tasks
and set this in job properties.
--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/
--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/
Terasort
Posted by Arun C Murthy <ac...@hortonworks.com>.
Changing subject...
On May 10, 2012, at 3:40 PM, Jeffrey Buell wrote:
> I have the right #slots to fill up memory across the cluster, and all those slots are filled with tasks. The problem I ran into was that the maps grabbed all the slots initially and the reduces had a hard time getting started. As maps finished, more maps were started and only rarely was a reduce started. I assume this behavior occurred because I had ~4000 map tasks in the queue, but only ~100 reduce tasks. If the scheduler lumps maps and reduces together, then whenever a slot opens up it will almost surely be taken by a map task. To get good performance I need all reduce tasks started early on, and have only map tasks compete for open slots. Other apps may need different priorities between maps and reduces. In any case, I don’t understand how treating maps and reduces the same is workable.
>
Are you playing with YARN or MR1?
IAC, you are getting hit by 'slowstart' for reduces where-in reduces aren't scheduled till sufficient % of maps are completed.
Set mapred.reduce.slowstart.completed.maps to 0. (That should work for either MR1 or MR2).
Arun
> Jeff
>
> From: Arun C Murthy [mailto:acm@hortonworks.com]
> Sent: Thursday, May 10, 2012 1:27 PM
> To: mapreduce-user@hadoop.apache.org
> Subject: Re: max 1 mapper per node
>
> For terasort you want to fill up your entire cluster with maps/reduces as fast as you can to get the best performance.
>
> Just play with #slots.
>
> Arun
>
> On May 9, 2012, at 12:36 PM, Jeffrey Buell wrote:
>
>
> Not to speak for Radim, but what I’m trying to achieve is performance at least as good as 0.20 for all cases. That is, no regressions. For something as simple as terasort, I don’t think that is possible without being able to specify the max number of mappers/reducers per node. As it is, I see slowdowns as much as 2X. Hopefully I’m wrong and somebody will straighten me out. But if I’m not, adding such a feature won’t lead to bad behavior of any kind since the default could be set to unlimited and thus have no effect whatsoever.
>
> I should emphasize that I support the goal of greater automation since Hadoop has way too many parameters and is so hard to tune. Just not at the expense of performance regressions.
>
> Jeff
>
>
> We've been against these 'features' since it leads to very bad behaviour across the cluster with multiple apps/users etc.
>
> What is your use-case i.e. what are you trying to achieve with this?
>
> thanks,
> Arun
>
> On May 3, 2012, at 5:59 AM, Radim Kolar wrote:
>
>
>
> if plugin system for AM is overkill, something simpler can be made like:
>
> maximum number of mappers per node
> maximum number of reducers per node
>
> maximum percentage of non data local tasks
> maximum percentage of rack local tasks
>
> and set this in job properties.
>
>
>
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/
>
>
--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/
RE: max 1 mapper per node
Posted by Jeffrey Buell <jb...@vmware.com>.
I have the right #slots to fill up memory across the cluster, and all those slots are filled with tasks. The problem I ran into was that the maps grabbed all the slots initially and the reduces had a hard time getting started. As maps finished, more maps were started and only rarely was a reduce started. I assume this behavior occurred because I had ~4000 map tasks in the queue, but only ~100 reduce tasks. If the scheduler lumps maps and reduces together, then whenever a slot opens up it will almost surely be taken by a map task. To get good performance I need all reduce tasks started early on, and have only map tasks compete for open slots. Other apps may need different priorities between maps and reduces. In any case, I don't understand how treating maps and reduces the same is workable.
Jeff
From: Arun C Murthy [mailto:acm@hortonworks.com]
Sent: Thursday, May 10, 2012 1:27 PM
To: mapreduce-user@hadoop.apache.org
Subject: Re: max 1 mapper per node
For terasort you want to fill up your entire cluster with maps/reduces as fast as you can to get the best performance.
Just play with #slots.
Arun
On May 9, 2012, at 12:36 PM, Jeffrey Buell wrote:
Not to speak for Radim, but what I'm trying to achieve is performance at least as good as 0.20 for all cases. That is, no regressions. For something as simple as terasort, I don't think that is possible without being able to specify the max number of mappers/reducers per node. As it is, I see slowdowns as much as 2X. Hopefully I'm wrong and somebody will straighten me out. But if I'm not, adding such a feature won't lead to bad behavior of any kind since the default could be set to unlimited and thus have no effect whatsoever.
I should emphasize that I support the goal of greater automation since Hadoop has way too many parameters and is so hard to tune. Just not at the expense of performance regressions.
Jeff
We've been against these 'features' since it leads to very bad behaviour across the cluster with multiple apps/users etc.
What is your use-case i.e. what are you trying to achieve with this?
thanks,
Arun
On May 3, 2012, at 5:59 AM, Radim Kolar wrote:
if plugin system for AM is overkill, something simpler can be made like:
maximum number of mappers per node
maximum number of reducers per node
maximum percentage of non data local tasks
maximum percentage of rack local tasks
and set this in job properties.
--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/
Re: max 1 mapper per node
Posted by Arun C Murthy <ac...@hortonworks.com>.
For terasort you want to fill up your entire cluster with maps/reduces as fast as you can to get the best performance.
Just play with #slots.
Arun
On May 9, 2012, at 12:36 PM, Jeffrey Buell wrote:
> Not to speak for Radim, but what I’m trying to achieve is performance at least as good as 0.20 for all cases. That is, no regressions. For something as simple as terasort, I don’t think that is possible without being able to specify the max number of mappers/reducers per node. As it is, I see slowdowns as much as 2X. Hopefully I’m wrong and somebody will straighten me out. But if I’m not, adding such a feature won’t lead to bad behavior of any kind since the default could be set to unlimited and thus have no effect whatsoever.
>
> I should emphasize that I support the goal of greater automation since Hadoop has way too many parameters and is so hard to tune. Just not at the expense of performance regressions.
>
> Jeff
>
>
> We've been against these 'features' since it leads to very bad behaviour across the cluster with multiple apps/users etc.
>
> What is your use-case i.e. what are you trying to achieve with this?
>
> thanks,
> Arun
>
> On May 3, 2012, at 5:59 AM, Radim Kolar wrote:
>
>
> if plugin system for AM is overkill, something simpler can be made like:
>
> maximum number of mappers per node
> maximum number of reducers per node
>
> maximum percentage of non data local tasks
> maximum percentage of rack local tasks
>
> and set this in job properties.
>
>
--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/
RE: max 1 mapper per node
Posted by Jeffrey Buell <jb...@vmware.com>.
Not to speak for Radim, but what I'm trying to achieve is performance at least as good as 0.20 for all cases. That is, no regressions. For something as simple as terasort, I don't think that is possible without being able to specify the max number of mappers/reducers per node. As it is, I see slowdowns as much as 2X. Hopefully I'm wrong and somebody will straighten me out. But if I'm not, adding such a feature won't lead to bad behavior of any kind since the default could be set to unlimited and thus have no effect whatsoever.
I should emphasize that I support the goal of greater automation since Hadoop has way too many parameters and is so hard to tune. Just not at the expense of performance regressions.
Jeff
We've been against these 'features' since it leads to very bad behaviour across the cluster with multiple apps/users etc.
What is your use-case i.e. what are you trying to achieve with this?
thanks,
Arun
On May 3, 2012, at 5:59 AM, Radim Kolar wrote:
if plugin system for AM is overkill, something simpler can be made like:
maximum number of mappers per node
maximum number of reducers per node
maximum percentage of non data local tasks
maximum percentage of rack local tasks
and set this in job properties.
Re: max 1 mapper per node
Posted by GUOJUN Zhu <gu...@freddiemac.com>.
One nice feature is to attach different configuration for different job.
You can do it either by command line "-conf" or programatically. We are
using that extensively while experimenting different setting.
Zhu, Guojun
Modeling Sr Graduate
571-3824370
guojun_zhu@freddiemac.com
Financial Engineering
Freddie Mac
Radim Kolar <hs...@filez.com>
05/10/2012 07:56 AM
Please respond to
mapreduce-user@hadoop.apache.org
To
mapreduce-user@hadoop.apache.org
cc
Subject
Re: max 1 mapper per node
> We've been against these 'features' since it leads to very bad
> behaviour across the cluster with multiple apps/users etc.
Its not new feature, its extension of existing resource scheduling which
works good enough only for RAM. There are 2 other resources - CPU cores
and network IO which needs to be considered.
We have job which is doing lot of network IO in mapper and its desirable
to run mappers on different nodes even if reading blocks from HDFS will
not be local.
Our second job is burning all CPU cores on machine while doing
computations, its important for mappers not to land on same node.
Re: max 1 mapper per node
Posted by Radim Kolar <hs...@filez.com>.
Dne 10.5.2012 15:29, Robert Evans napsal(a):
> Yes adding in more resources in the scheduling request would be the
> ideal solution to the problem. But sadly that is not a trivial change.
Best will be to have custom resources.
Example: in node config define that this node has:
1 x GPU
1 x Gbit Network
etc.
then add property to job configuration that this job needs 1 GPU. RM
will pickup node with free GPU.
Re: max 1 mapper per node
Posted by Robert Evans <ev...@yahoo-inc.com>.
Yes adding in more resources in the scheduling request would be the ideal solution to the problem. But sadly that is not a trivial change. The initial solution I suggested is an ugly hack, and will not work for the cases you have suggested. If you feel that this is important work please feel free to file a JIRA for this. We can continue discussion on that JIRA about the details of how to add in this type of functionality. I am very interested in the scheduler and would be happy to help out, but sadly my time right now is very limited.
--Bobby Evans
On 5/10/12 6:56 AM, "Radim Kolar" <hs...@filez.com> wrote:
> We've been against these 'features' since it leads to very bad
> behaviour across the cluster with multiple apps/users etc.
Its not new feature, its extension of existing resource scheduling which
works good enough only for RAM. There are 2 other resources - CPU cores
and network IO which needs to be considered.
We have job which is doing lot of network IO in mapper and its desirable
to run mappers on different nodes even if reading blocks from HDFS will
not be local.
Our second job is burning all CPU cores on machine while doing
computations, its important for mappers not to land on same node.
Re: max 1 mapper per node
Posted by Radim Kolar <hs...@filez.com>.
> We've been against these 'features' since it leads to very bad
> behaviour across the cluster with multiple apps/users etc.
Its not new feature, its extension of existing resource scheduling which
works good enough only for RAM. There are 2 other resources - CPU cores
and network IO which needs to be considered.
We have job which is doing lot of network IO in mapper and its desirable
to run mappers on different nodes even if reading blocks from HDFS will
not be local.
Our second job is burning all CPU cores on machine while doing
computations, its important for mappers not to land on same node.
Re: max 1 mapper per node
Posted by Arun C Murthy <ac...@hortonworks.com>.
We've been against these 'features' since it leads to very bad behaviour across the cluster with multiple apps/users etc.
What is your use-case i.e. what are you trying to achieve with this?
thanks,
Arun
On May 3, 2012, at 5:59 AM, Radim Kolar wrote:
> if plugin system for AM is overkill, something simpler can be made like:
>
> maximum number of mappers per node
> maximum number of reducers per node
>
> maximum percentage of non data local tasks
> maximum percentage of rack local tasks
>
> and set this in job properties.
--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/
Re: max 1 mapper per node
Posted by Robert Evans <ev...@yahoo-inc.com>.
Either plugins or configuration options would be possible to do. The real question is what is the use case for this, and is that use case big enough to warrant it being part of core Hadoop? I have seen a few situations where this perhaps makes since, but most of those are because the resource scheduling is currently very basic (meaning it only knows abut RAM and machine/rack locality which is really a surrogate for requesting a HDFS block as a resource). For example GPGPU computation, where sharing the GPU between processes can be complicated, or more commonly where the network is the bottleneck and multiple processes running on the same box is not an optimal solution because it will saturate the network.
In both of those cases I would prefer to see resource scheduling updated rather then trying to work around it by having the AM throw away containers. But that is just me and is a major undertaking. Throwing away the containers feels like a hack to me but it is something that can work right now. It just depends on what the use case is and if it is compelling enough to make that hack a fully supported part of the Hadoop map/reduce API.
--Bobby Evans
On 5/3/12 7:59 AM, "Radim Kolar" <hs...@filez.com> wrote:
if plugin system for AM is overkill, something simpler can be made like:
maximum number of mappers per node
maximum number of reducers per node
maximum percentage of non data local tasks
maximum percentage of rack local tasks
and set this in job properties.
Re: max 1 mapper per node
Posted by Radim Kolar <hs...@filez.com>.
if plugin system for AM is overkill, something simpler can be made like:
maximum number of mappers per node
maximum number of reducers per node
maximum percentage of non data local tasks
maximum percentage of rack local tasks
and set this in job properties.
Re: max 1 mapper per node
Posted by Radim Kolar <hs...@filez.com>.
Dne 27.4.2012 17:30, Robert Evans napsal(a):
> Radim,
>
> You would want to modify the application master for this, and it is
> likely to be a bit of a hack because the RM scheduler itself is not
> really designed for this.
What about to do something like this:
In job JAR there will be loadable plugin which will be loaded by AM
and used for allocating containers. For example mapreduce job should
request to launch only data-local tasks or do some other customization.
Re: max 1 mapper per node
Posted by Robert Evans <ev...@yahoo-inc.com>.
Radim,
You would want to modify the application master for this, and it is likely to be a bit of a hack because the RM scheduler itself is not really designed for this. You can have the AM return containers to the RM that it does not want (which it does already in some cases), because they are on the same node as an already running map task. This could slow down your application a lot, as it makes it more difficult for the RM to find free containers that the AM wants. You could also modify the AM so that it will not request any more then just one container per node, which would help with this, but not solve it completely.
--Bobby Evans
On 4/27/12 12:38 AM, "Radim Kolar" <hs...@filez.com> wrote:
Do you want it so that the node can run only 1 mapper at a time but the node can also run multiple reducers at the same time?
I currently want it per-application. Node can run more mappers/reducers from other jobs at same time.
Do you want it so that this particular application will only use that node for a single mapper, but other applications may also run mappers on the node?
yes. After this will be coded, i will rerun tests if it is sufficient for other types of tasks too.
For start, i would need per job settings - max mappers/reducers per node, so modifying AM would be enough.
Re: max 1 mapper per node
Posted by Radim Kolar <hs...@filez.com>.
> Do you want it so that the node can run only 1 mapper at a time but
> the node can also run multiple reducers at the same time?
I currently want it per-application. Node can run more mappers/reducers
from other jobs at same time.
> Do you want it so that this particular application will only use that
> node for a single mapper, but other applications may also run mappers
> on the node?
yes. After this will be coded, i will rerun tests if it is sufficient
for other types of tasks too.
For start, i would need per job settings - max mappers/reducers per
node, so modifying AM would be enough.
Re: max 1 mapper per node
Posted by Robert Evans <ev...@yahoo-inc.com>.
I do not believe that there is currently a way to do this.
Do you want it so that the node can run only 1 mapper at a time but the node can also run multiple reducers at the same time? Do you want it so that this particular application will only use that node for a single mapper, but other applications may also run mappers on the node?
The RM has no knowledge about mapper vs. reducer, but the AM has no knowledge about other applications so it may require modifications in multiple places.
--Bobby Evans
On 4/26/12 12:56 PM, "Radim Kolar" <hs...@filez.com> wrote:
Its possible to configure in YARN scheduling requirement to not run more
than 1 mapper per node?
Its not because of memory requirements. Who decides this kind of
scheduling? ResourceManager or ApplicationManager?