You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by twinkle sachdeva <tw...@gmail.com> on 2015/02/26 09:35:03 UTC
Node manager contributing to one queue's resources
Hi,
I have to run two kind of applications, one requiring less cores but more
memory ( Application_High_Mem) and another application which requires more
cores but less memory ( Application_High_Core).
I can use specific queues to submit them to, but that can lead to one node
contributing to only one one such application and having some part of
resources as idle.
Is there a way, let's say extending concept of queues at node manager level
to do this or some other way, in which i can achieve it in YARN?
Thanks,
Twinkle
Re: Node manager contributing to one queue's resources
Posted by twinkle sachdeva <tw...@gmail.com>.
Thanks for the sharing another view point, will look around it.
But seems like, we might need something more specific.
On Thu, Feb 26, 2015 at 2:43 PM, Rohith Sharma K S <
rohithsharmaks@huawei.com> wrote:
> Hi
>
>
>
> If you are using CapacityScheduler, can you try using
> DominantResourceCalculator i.e configuring below property value in
> capacity-scheduler.xml file.
>
>
>
> <property>
>
> <name>yarn.scheduler.capacity.resource-calculator</name>
>
> <value>org.apache.hadoop.yarn.util.resource.
> DominantResourceCalculator </value>
>
> </property>
>
>
>
> The basic Idea it works is as follows ‘if user A runs CPU-heavy tasks
> and user B runs memory-heavy tasks, it attempts to equalize CPU share of
> user A with Memory-share of user B’
>
>
>
> See Java Doc
>
>
> https://apache.googlesource.com/hadoop-common/+/60e3b885ba8344d9f448202f5f2c290b5606ff8f/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/resource/DominantResourceCalculator.java
>
>
>
> I think this may help you!!!
>
>
>
> Thanks & Regards
>
> Rohith Sharma K S
>
>
>
> *From:* twinkle sachdeva [mailto:twinkle.sachdeva@gmail.com]
> *Sent:* 26 February 2015 14:05
> *To:* USers Hadoop
> *Subject:* Node manager contributing to one queue's resources
>
>
>
> Hi,
>
>
>
> I have to run two kind of applications, one requiring less cores but more
> memory ( Application_High_Mem) and another application which requires more
> cores but less memory ( Application_High_Core).
>
>
>
> I can use specific queues to submit them to, but that can lead to one node
> contributing to only one one such application and having some part of
> resources as idle.
>
>
>
> Is there a way, let's say extending concept of queues at node manager
> level to do this or some other way, in which i can achieve it in YARN?
>
>
>
> Thanks,
>
> Twinkle
>
Re: Node manager contributing to one queue's resources
Posted by twinkle sachdeva <tw...@gmail.com>.
Thanks for the sharing another view point, will look around it.
But seems like, we might need something more specific.
On Thu, Feb 26, 2015 at 2:43 PM, Rohith Sharma K S <
rohithsharmaks@huawei.com> wrote:
> Hi
>
>
>
> If you are using CapacityScheduler, can you try using
> DominantResourceCalculator i.e configuring below property value in
> capacity-scheduler.xml file.
>
>
>
> <property>
>
> <name>yarn.scheduler.capacity.resource-calculator</name>
>
> <value>org.apache.hadoop.yarn.util.resource.
> DominantResourceCalculator </value>
>
> </property>
>
>
>
> The basic Idea it works is as follows ‘if user A runs CPU-heavy tasks
> and user B runs memory-heavy tasks, it attempts to equalize CPU share of
> user A with Memory-share of user B’
>
>
>
> See Java Doc
>
>
> https://apache.googlesource.com/hadoop-common/+/60e3b885ba8344d9f448202f5f2c290b5606ff8f/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/resource/DominantResourceCalculator.java
>
>
>
> I think this may help you!!!
>
>
>
> Thanks & Regards
>
> Rohith Sharma K S
>
>
>
> *From:* twinkle sachdeva [mailto:twinkle.sachdeva@gmail.com]
> *Sent:* 26 February 2015 14:05
> *To:* USers Hadoop
> *Subject:* Node manager contributing to one queue's resources
>
>
>
> Hi,
>
>
>
> I have to run two kind of applications, one requiring less cores but more
> memory ( Application_High_Mem) and another application which requires more
> cores but less memory ( Application_High_Core).
>
>
>
> I can use specific queues to submit them to, but that can lead to one node
> contributing to only one one such application and having some part of
> resources as idle.
>
>
>
> Is there a way, let's say extending concept of queues at node manager
> level to do this or some other way, in which i can achieve it in YARN?
>
>
>
> Thanks,
>
> Twinkle
>
Re: Node manager contributing to one queue's resources
Posted by twinkle sachdeva <tw...@gmail.com>.
Thanks for the sharing another view point, will look around it.
But seems like, we might need something more specific.
On Thu, Feb 26, 2015 at 2:43 PM, Rohith Sharma K S <
rohithsharmaks@huawei.com> wrote:
> Hi
>
>
>
> If you are using CapacityScheduler, can you try using
> DominantResourceCalculator i.e configuring below property value in
> capacity-scheduler.xml file.
>
>
>
> <property>
>
> <name>yarn.scheduler.capacity.resource-calculator</name>
>
> <value>org.apache.hadoop.yarn.util.resource.
> DominantResourceCalculator </value>
>
> </property>
>
>
>
> The basic Idea it works is as follows ‘if user A runs CPU-heavy tasks
> and user B runs memory-heavy tasks, it attempts to equalize CPU share of
> user A with Memory-share of user B’
>
>
>
> See Java Doc
>
>
> https://apache.googlesource.com/hadoop-common/+/60e3b885ba8344d9f448202f5f2c290b5606ff8f/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/resource/DominantResourceCalculator.java
>
>
>
> I think this may help you!!!
>
>
>
> Thanks & Regards
>
> Rohith Sharma K S
>
>
>
> *From:* twinkle sachdeva [mailto:twinkle.sachdeva@gmail.com]
> *Sent:* 26 February 2015 14:05
> *To:* USers Hadoop
> *Subject:* Node manager contributing to one queue's resources
>
>
>
> Hi,
>
>
>
> I have to run two kind of applications, one requiring less cores but more
> memory ( Application_High_Mem) and another application which requires more
> cores but less memory ( Application_High_Core).
>
>
>
> I can use specific queues to submit them to, but that can lead to one node
> contributing to only one one such application and having some part of
> resources as idle.
>
>
>
> Is there a way, let's say extending concept of queues at node manager
> level to do this or some other way, in which i can achieve it in YARN?
>
>
>
> Thanks,
>
> Twinkle
>
Re: Node manager contributing to one queue's resources
Posted by twinkle sachdeva <tw...@gmail.com>.
Thanks for the sharing another view point, will look around it.
But seems like, we might need something more specific.
On Thu, Feb 26, 2015 at 2:43 PM, Rohith Sharma K S <
rohithsharmaks@huawei.com> wrote:
> Hi
>
>
>
> If you are using CapacityScheduler, can you try using
> DominantResourceCalculator i.e configuring below property value in
> capacity-scheduler.xml file.
>
>
>
> <property>
>
> <name>yarn.scheduler.capacity.resource-calculator</name>
>
> <value>org.apache.hadoop.yarn.util.resource.
> DominantResourceCalculator </value>
>
> </property>
>
>
>
> The basic Idea it works is as follows ‘if user A runs CPU-heavy tasks
> and user B runs memory-heavy tasks, it attempts to equalize CPU share of
> user A with Memory-share of user B’
>
>
>
> See Java Doc
>
>
> https://apache.googlesource.com/hadoop-common/+/60e3b885ba8344d9f448202f5f2c290b5606ff8f/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/resource/DominantResourceCalculator.java
>
>
>
> I think this may help you!!!
>
>
>
> Thanks & Regards
>
> Rohith Sharma K S
>
>
>
> *From:* twinkle sachdeva [mailto:twinkle.sachdeva@gmail.com]
> *Sent:* 26 February 2015 14:05
> *To:* USers Hadoop
> *Subject:* Node manager contributing to one queue's resources
>
>
>
> Hi,
>
>
>
> I have to run two kind of applications, one requiring less cores but more
> memory ( Application_High_Mem) and another application which requires more
> cores but less memory ( Application_High_Core).
>
>
>
> I can use specific queues to submit them to, but that can lead to one node
> contributing to only one one such application and having some part of
> resources as idle.
>
>
>
> Is there a way, let's say extending concept of queues at node manager
> level to do this or some other way, in which i can achieve it in YARN?
>
>
>
> Thanks,
>
> Twinkle
>
RE: Node manager contributing to one queue's resources
Posted by Rohith Sharma K S <ro...@huawei.com>.
Hi
If you are using CapacityScheduler, can you try using DominantResourceCalculator i.e configuring below property value in capacity-scheduler.xml file.
<property>
<name>yarn.scheduler.capacity.resource-calculator</name>
<value>org.apache.hadoop.yarn.util.resource. DominantResourceCalculator </value>
</property>
The basic Idea it works is as follows ‘if user A runs CPU-heavy tasks and user B runs memory-heavy tasks, it attempts to equalize CPU share of user A with Memory-share of user B’
See Java Doc
https://apache.googlesource.com/hadoop-common/+/60e3b885ba8344d9f448202f5f2c290b5606ff8f/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/resource/DominantResourceCalculator.java
I think this may help you!!!
Thanks & Regards
Rohith Sharma K S
From: twinkle sachdeva [mailto:twinkle.sachdeva@gmail.com]
Sent: 26 February 2015 14:05
To: USers Hadoop
Subject: Node manager contributing to one queue's resources
Hi,
I have to run two kind of applications, one requiring less cores but more memory ( Application_High_Mem) and another application which requires more cores but less memory ( Application_High_Core).
I can use specific queues to submit them to, but that can lead to one node contributing to only one one such application and having some part of resources as idle.
Is there a way, let's say extending concept of queues at node manager level to do this or some other way, in which i can achieve it in YARN?
Thanks,
Twinkle
RE: Node manager contributing to one queue's resources
Posted by Rohith Sharma K S <ro...@huawei.com>.
Hi
If you are using CapacityScheduler, can you try using DominantResourceCalculator i.e configuring below property value in capacity-scheduler.xml file.
<property>
<name>yarn.scheduler.capacity.resource-calculator</name>
<value>org.apache.hadoop.yarn.util.resource. DominantResourceCalculator </value>
</property>
The basic Idea it works is as follows ‘if user A runs CPU-heavy tasks and user B runs memory-heavy tasks, it attempts to equalize CPU share of user A with Memory-share of user B’
See Java Doc
https://apache.googlesource.com/hadoop-common/+/60e3b885ba8344d9f448202f5f2c290b5606ff8f/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/resource/DominantResourceCalculator.java
I think this may help you!!!
Thanks & Regards
Rohith Sharma K S
From: twinkle sachdeva [mailto:twinkle.sachdeva@gmail.com]
Sent: 26 February 2015 14:05
To: USers Hadoop
Subject: Node manager contributing to one queue's resources
Hi,
I have to run two kind of applications, one requiring less cores but more memory ( Application_High_Mem) and another application which requires more cores but less memory ( Application_High_Core).
I can use specific queues to submit them to, but that can lead to one node contributing to only one one such application and having some part of resources as idle.
Is there a way, let's say extending concept of queues at node manager level to do this or some other way, in which i can achieve it in YARN?
Thanks,
Twinkle
RE: Node manager contributing to one queue's resources
Posted by Rohith Sharma K S <ro...@huawei.com>.
Hi
If you are using CapacityScheduler, can you try using DominantResourceCalculator i.e configuring below property value in capacity-scheduler.xml file.
<property>
<name>yarn.scheduler.capacity.resource-calculator</name>
<value>org.apache.hadoop.yarn.util.resource. DominantResourceCalculator </value>
</property>
The basic Idea it works is as follows ‘if user A runs CPU-heavy tasks and user B runs memory-heavy tasks, it attempts to equalize CPU share of user A with Memory-share of user B’
See Java Doc
https://apache.googlesource.com/hadoop-common/+/60e3b885ba8344d9f448202f5f2c290b5606ff8f/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/resource/DominantResourceCalculator.java
I think this may help you!!!
Thanks & Regards
Rohith Sharma K S
From: twinkle sachdeva [mailto:twinkle.sachdeva@gmail.com]
Sent: 26 February 2015 14:05
To: USers Hadoop
Subject: Node manager contributing to one queue's resources
Hi,
I have to run two kind of applications, one requiring less cores but more memory ( Application_High_Mem) and another application which requires more cores but less memory ( Application_High_Core).
I can use specific queues to submit them to, but that can lead to one node contributing to only one one such application and having some part of resources as idle.
Is there a way, let's say extending concept of queues at node manager level to do this or some other way, in which i can achieve it in YARN?
Thanks,
Twinkle
RE: Node manager contributing to one queue's resources
Posted by Rohith Sharma K S <ro...@huawei.com>.
Hi
If you are using CapacityScheduler, can you try using DominantResourceCalculator i.e configuring below property value in capacity-scheduler.xml file.
<property>
<name>yarn.scheduler.capacity.resource-calculator</name>
<value>org.apache.hadoop.yarn.util.resource. DominantResourceCalculator </value>
</property>
The basic Idea it works is as follows ‘if user A runs CPU-heavy tasks and user B runs memory-heavy tasks, it attempts to equalize CPU share of user A with Memory-share of user B’
See Java Doc
https://apache.googlesource.com/hadoop-common/+/60e3b885ba8344d9f448202f5f2c290b5606ff8f/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/resource/DominantResourceCalculator.java
I think this may help you!!!
Thanks & Regards
Rohith Sharma K S
From: twinkle sachdeva [mailto:twinkle.sachdeva@gmail.com]
Sent: 26 February 2015 14:05
To: USers Hadoop
Subject: Node manager contributing to one queue's resources
Hi,
I have to run two kind of applications, one requiring less cores but more memory ( Application_High_Mem) and another application which requires more cores but less memory ( Application_High_Core).
I can use specific queues to submit them to, but that can lead to one node contributing to only one one such application and having some part of resources as idle.
Is there a way, let's say extending concept of queues at node manager level to do this or some other way, in which i can achieve it in YARN?
Thanks,
Twinkle