You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by Henry Hung <YT...@winbond.com> on 2014/06/03 03:10:10 UTC
RE: change yarn application priority
@Rohith Sharma
Thank you for the confirmation.
I already google some information regarding scheduler in Hadoop 2.2.0, there are 2 ways to do it: FairScheduler or CapacityScheduler.
Apparently from google result, there are more article mentions that FairScheduler is better than CapacityScheduler.
So, I intend to use FairScheduler first, if you have any more suggestion, please let me know, thank you again.
Best regards,
Henry
From: Rohith Sharma K S [mailto:rohithsharmaks@huawei.com]
Sent: Friday, May 30, 2014 5:49 PM
To: user@hadoop.apache.org
Subject: RE: change yarn application priority
Hi
Currently there is no provision for changing application priority within the same queue. Follow the Jira https://issues.apache.org/jira/i#browse/YARN-1963 for this new feature.
One way you can achieve by using enabling scheduler monitors for CapacitySchedulers.
Steps to be follow is
1. Configure 2 queues, follow http://hadoop.apache.org/docs/r2.3.0/hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html
2. Enable scheduler monitor
yarn.resourcemanager.scheduler.monitor.enable = true
One job you submit to queue 1 which run 2hours. Another job you submit queue 2.
Hope this will help you.
Thanks & Regards
Rohith Sharma K S
This e-mail and its attachments contain confidential information from HUAWEI, which
is intended only for the person or entity whose address is listed above. Any use of the
information contained herein in any way (including, but not limited to, total or partial
disclosure, reproduction, or dissemination) by persons other than the intended
recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender by
phone or email immediately and delete it!
From: Henry Hung [mailto:YTHung1@winbond.com]
Sent: 30 May 2014 11:53
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: change yarn application priority
HI All,
I have an application that consumes all of nodemanager capacity (30 Map and 1 Reducer) and will need 4 hours to finish.
Let's say I need to run another application that will be quicker to finish (30 minutes) and only need 1 Map and 1 Reducer.
If I just execute the new application, it will be in queue waiting for the 1st application to finish.
Is there a way to change the 2nd application priority to higher than the 1st and let resourcemanager immediately execute the 2nd application?
I'm using Hadoop-2.2.0.
Best regards,
Henry
________________________________
The privileged confidential information contained in this email is intended for use only by the addressees as indicated by the original sender of this email. If you are not the addressee indicated in this email or are not responsible for delivery of the email to such a person, please kindly reply to the sender indicating this fact and delete all copies of it from your computer and network server immediately. Your cooperation is highly appreciated. It is advised that any unauthorized use of confidential information of Winbond is strictly prohibited; and any information in this email irrelevant to the official business of Winbond shall be deemed as neither given nor endorsed by Winbond.
________________________________
The privileged confidential information contained in this email is intended for use only by the addressees as indicated by the original sender of this email. If you are not the addressee indicated in this email or are not responsible for delivery of the email to such a person, please kindly reply to the sender indicating this fact and delete all copies of it from your computer and network server immediately. Your cooperation is highly appreciated. It is advised that any unauthorized use of confidential information of Winbond is strictly prohibited; and any information in this email irrelevant to the official business of Winbond shall be deemed as neither given nor endorsed by Winbond.
RE: change yarn application priority
Posted by Henry Hung <YT...@winbond.com>.
I manage to implement the fair scheduler and very happy with the result.
The way I use it is to create a queue that constraint the number of resources (maxResources) to be around 70% of total resources, fortunately the fair-scheduler.xml can be modified on-fly without restarting the yarn cluster.
Thank to every one that provide me with all the information.
From: Michael Segel [mailto:msegel_hadoop@hotmail.com]
Sent: Tuesday, June 03, 2014 8:13 PM
To: user@hadoop.apache.org
Subject: Re: change yarn application priority
WRT capacity scheduler, its not so much changing the priority of a job, but allowing for pre-emption. Note that I guess you could raise the one job's priority, and then the other job's priority so that when a task finishes the other job gets the next slot. However, you're still stuck waiting and that can take time.)
Here's the rub though... pre-empting a task could have a negative impact on the long running job.
If you have large jobs and small jobs, you could create multiple queues and leave some capacity to the smaller job queue.
So you need to specify the queue up front. (Its been a while, but can you move jobs across queues? )
I would suggest that you look at fair scheduler, but also consider multiple queues under capacity scheduler. You may have admin jobs you want to run in the background while other tasks are running.
On Jun 2, 2014, at 8:10 PM, Henry Hung <YT...@winbond.com>> wrote:
@Rohith Sharma
Thank you for the confirmation.
I already google some information regarding scheduler in Hadoop 2.2.0, there are 2 ways to do it: FairScheduler or CapacityScheduler.
Apparently from google result, there are more article mentions that FairScheduler is better than CapacityScheduler.
So, I intend to use FairScheduler first, if you have any more suggestion, please let me know, thank you again.
Best regards,
Henry
From: Rohith Sharma K S [mailto:rohithsharmaks@huawei.com]
Sent: Friday, May 30, 2014 5:49 PM
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: RE: change yarn application priority
Hi
Currently there is no provision for changing application priority within the same queue. Follow the Jirahttps://issues.apache.org/jira/i#browse/YARN-1963 for this new feature.
One way you can achieve by using enabling scheduler monitors for CapacitySchedulers.
Steps to be follow is
1. Configure 2 queues, follow http://hadoop.apache.org/docs/r2.3.0/hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html
2. Enable scheduler monitor
yarn.resourcemanager.scheduler.monitor.enable = true
One job you submit to queue 1 which run 2hours. Another job you submit queue 2.
Hope this will help you.
Thanks & Regards
Rohith Sharma K S
This e-mail and its attachments contain confidential information from HUAWEI, which
is intended only for the person or entity whose address is listed above. Any use of the
information contained herein in any way (including, but not limited to, total or partial
disclosure, reproduction, or dissemination) by persons other than the intended
recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender by
phone or email immediately and delete it!
From: Henry Hung [mailto:YTHung1@winbond.com]
Sent: 30 May 2014 11:53
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: change yarn application priority
HI All,
I have an application that consumes all of nodemanager capacity (30 Map and 1 Reducer) and will need 4 hours to finish.
Let's say I need to run another application that will be quicker to finish (30 minutes) and only need 1 Map and 1 Reducer.
If I just execute the new application, it will be in queue waiting for the 1st application to finish.
Is there a way to change the 2nd application priority to higher than the 1st and let resourcemanager immediately execute the 2ndapplication?
I'm using Hadoop-2.2.0.
Best regards,
Henry
________________________________
The privileged confidential information contained in this email is intended for use only by the addressees as indicated by the original sender of this email. If you are not the addressee indicated in this email or are not responsible for delivery of the email to such a person, please kindly reply to the sender indicating this fact and delete all copies of it from your computer and network server immediately. Your cooperation is highly appreciated. It is advised that any unauthorized use of confidential information of Winbond is strictly prohibited; and any information in this email irrelevant to the official business of Winbond shall be deemed as neither given nor endorsed by Winbond.
________________________________
The privileged confidential information contained in this email is intended for use only by the addressees as indicated by the original sender of this email. If you are not the addressee indicated in this email or are not responsible for delivery of the email to such a person, please kindly reply to the sender indicating this fact and delete all copies of it from your computer and network server immediately. Your cooperation is highly appreciated. It is advised that any unauthorized use of confidential information of Winbond is strictly prohibited; and any information in this email irrelevant to the official business of Winbond shall be deemed as neither given nor endorsed by Winbond.
________________________________
The privileged confidential information contained in this email is intended for use only by the addressees as indicated by the original sender of this email. If you are not the addressee indicated in this email or are not responsible for delivery of the email to such a person, please kindly reply to the sender indicating this fact and delete all copies of it from your computer and network server immediately. Your cooperation is highly appreciated. It is advised that any unauthorized use of confidential information of Winbond is strictly prohibited; and any information in this email irrelevant to the official business of Winbond shall be deemed as neither given nor endorsed by Winbond.
RE: change yarn application priority
Posted by Henry Hung <YT...@winbond.com>.
I manage to implement the fair scheduler and very happy with the result.
The way I use it is to create a queue that constraint the number of resources (maxResources) to be around 70% of total resources, fortunately the fair-scheduler.xml can be modified on-fly without restarting the yarn cluster.
Thank to every one that provide me with all the information.
From: Michael Segel [mailto:msegel_hadoop@hotmail.com]
Sent: Tuesday, June 03, 2014 8:13 PM
To: user@hadoop.apache.org
Subject: Re: change yarn application priority
WRT capacity scheduler, its not so much changing the priority of a job, but allowing for pre-emption. Note that I guess you could raise the one job's priority, and then the other job's priority so that when a task finishes the other job gets the next slot. However, you're still stuck waiting and that can take time.)
Here's the rub though... pre-empting a task could have a negative impact on the long running job.
If you have large jobs and small jobs, you could create multiple queues and leave some capacity to the smaller job queue.
So you need to specify the queue up front. (Its been a while, but can you move jobs across queues? )
I would suggest that you look at fair scheduler, but also consider multiple queues under capacity scheduler. You may have admin jobs you want to run in the background while other tasks are running.
On Jun 2, 2014, at 8:10 PM, Henry Hung <YT...@winbond.com>> wrote:
@Rohith Sharma
Thank you for the confirmation.
I already google some information regarding scheduler in Hadoop 2.2.0, there are 2 ways to do it: FairScheduler or CapacityScheduler.
Apparently from google result, there are more article mentions that FairScheduler is better than CapacityScheduler.
So, I intend to use FairScheduler first, if you have any more suggestion, please let me know, thank you again.
Best regards,
Henry
From: Rohith Sharma K S [mailto:rohithsharmaks@huawei.com]
Sent: Friday, May 30, 2014 5:49 PM
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: RE: change yarn application priority
Hi
Currently there is no provision for changing application priority within the same queue. Follow the Jirahttps://issues.apache.org/jira/i#browse/YARN-1963 for this new feature.
One way you can achieve by using enabling scheduler monitors for CapacitySchedulers.
Steps to be follow is
1. Configure 2 queues, follow http://hadoop.apache.org/docs/r2.3.0/hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html
2. Enable scheduler monitor
yarn.resourcemanager.scheduler.monitor.enable = true
One job you submit to queue 1 which run 2hours. Another job you submit queue 2.
Hope this will help you.
Thanks & Regards
Rohith Sharma K S
This e-mail and its attachments contain confidential information from HUAWEI, which
is intended only for the person or entity whose address is listed above. Any use of the
information contained herein in any way (including, but not limited to, total or partial
disclosure, reproduction, or dissemination) by persons other than the intended
recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender by
phone or email immediately and delete it!
From: Henry Hung [mailto:YTHung1@winbond.com]
Sent: 30 May 2014 11:53
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: change yarn application priority
HI All,
I have an application that consumes all of nodemanager capacity (30 Map and 1 Reducer) and will need 4 hours to finish.
Let's say I need to run another application that will be quicker to finish (30 minutes) and only need 1 Map and 1 Reducer.
If I just execute the new application, it will be in queue waiting for the 1st application to finish.
Is there a way to change the 2nd application priority to higher than the 1st and let resourcemanager immediately execute the 2ndapplication?
I'm using Hadoop-2.2.0.
Best regards,
Henry
________________________________
The privileged confidential information contained in this email is intended for use only by the addressees as indicated by the original sender of this email. If you are not the addressee indicated in this email or are not responsible for delivery of the email to such a person, please kindly reply to the sender indicating this fact and delete all copies of it from your computer and network server immediately. Your cooperation is highly appreciated. It is advised that any unauthorized use of confidential information of Winbond is strictly prohibited; and any information in this email irrelevant to the official business of Winbond shall be deemed as neither given nor endorsed by Winbond.
________________________________
The privileged confidential information contained in this email is intended for use only by the addressees as indicated by the original sender of this email. If you are not the addressee indicated in this email or are not responsible for delivery of the email to such a person, please kindly reply to the sender indicating this fact and delete all copies of it from your computer and network server immediately. Your cooperation is highly appreciated. It is advised that any unauthorized use of confidential information of Winbond is strictly prohibited; and any information in this email irrelevant to the official business of Winbond shall be deemed as neither given nor endorsed by Winbond.
________________________________
The privileged confidential information contained in this email is intended for use only by the addressees as indicated by the original sender of this email. If you are not the addressee indicated in this email or are not responsible for delivery of the email to such a person, please kindly reply to the sender indicating this fact and delete all copies of it from your computer and network server immediately. Your cooperation is highly appreciated. It is advised that any unauthorized use of confidential information of Winbond is strictly prohibited; and any information in this email irrelevant to the official business of Winbond shall be deemed as neither given nor endorsed by Winbond.
RE: change yarn application priority
Posted by Henry Hung <YT...@winbond.com>.
I manage to implement the fair scheduler and very happy with the result.
The way I use it is to create a queue that constraint the number of resources (maxResources) to be around 70% of total resources, fortunately the fair-scheduler.xml can be modified on-fly without restarting the yarn cluster.
Thank to every one that provide me with all the information.
From: Michael Segel [mailto:msegel_hadoop@hotmail.com]
Sent: Tuesday, June 03, 2014 8:13 PM
To: user@hadoop.apache.org
Subject: Re: change yarn application priority
WRT capacity scheduler, its not so much changing the priority of a job, but allowing for pre-emption. Note that I guess you could raise the one job's priority, and then the other job's priority so that when a task finishes the other job gets the next slot. However, you're still stuck waiting and that can take time.)
Here's the rub though... pre-empting a task could have a negative impact on the long running job.
If you have large jobs and small jobs, you could create multiple queues and leave some capacity to the smaller job queue.
So you need to specify the queue up front. (Its been a while, but can you move jobs across queues? )
I would suggest that you look at fair scheduler, but also consider multiple queues under capacity scheduler. You may have admin jobs you want to run in the background while other tasks are running.
On Jun 2, 2014, at 8:10 PM, Henry Hung <YT...@winbond.com>> wrote:
@Rohith Sharma
Thank you for the confirmation.
I already google some information regarding scheduler in Hadoop 2.2.0, there are 2 ways to do it: FairScheduler or CapacityScheduler.
Apparently from google result, there are more article mentions that FairScheduler is better than CapacityScheduler.
So, I intend to use FairScheduler first, if you have any more suggestion, please let me know, thank you again.
Best regards,
Henry
From: Rohith Sharma K S [mailto:rohithsharmaks@huawei.com]
Sent: Friday, May 30, 2014 5:49 PM
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: RE: change yarn application priority
Hi
Currently there is no provision for changing application priority within the same queue. Follow the Jirahttps://issues.apache.org/jira/i#browse/YARN-1963 for this new feature.
One way you can achieve by using enabling scheduler monitors for CapacitySchedulers.
Steps to be follow is
1. Configure 2 queues, follow http://hadoop.apache.org/docs/r2.3.0/hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html
2. Enable scheduler monitor
yarn.resourcemanager.scheduler.monitor.enable = true
One job you submit to queue 1 which run 2hours. Another job you submit queue 2.
Hope this will help you.
Thanks & Regards
Rohith Sharma K S
This e-mail and its attachments contain confidential information from HUAWEI, which
is intended only for the person or entity whose address is listed above. Any use of the
information contained herein in any way (including, but not limited to, total or partial
disclosure, reproduction, or dissemination) by persons other than the intended
recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender by
phone or email immediately and delete it!
From: Henry Hung [mailto:YTHung1@winbond.com]
Sent: 30 May 2014 11:53
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: change yarn application priority
HI All,
I have an application that consumes all of nodemanager capacity (30 Map and 1 Reducer) and will need 4 hours to finish.
Let's say I need to run another application that will be quicker to finish (30 minutes) and only need 1 Map and 1 Reducer.
If I just execute the new application, it will be in queue waiting for the 1st application to finish.
Is there a way to change the 2nd application priority to higher than the 1st and let resourcemanager immediately execute the 2ndapplication?
I'm using Hadoop-2.2.0.
Best regards,
Henry
________________________________
The privileged confidential information contained in this email is intended for use only by the addressees as indicated by the original sender of this email. If you are not the addressee indicated in this email or are not responsible for delivery of the email to such a person, please kindly reply to the sender indicating this fact and delete all copies of it from your computer and network server immediately. Your cooperation is highly appreciated. It is advised that any unauthorized use of confidential information of Winbond is strictly prohibited; and any information in this email irrelevant to the official business of Winbond shall be deemed as neither given nor endorsed by Winbond.
________________________________
The privileged confidential information contained in this email is intended for use only by the addressees as indicated by the original sender of this email. If you are not the addressee indicated in this email or are not responsible for delivery of the email to such a person, please kindly reply to the sender indicating this fact and delete all copies of it from your computer and network server immediately. Your cooperation is highly appreciated. It is advised that any unauthorized use of confidential information of Winbond is strictly prohibited; and any information in this email irrelevant to the official business of Winbond shall be deemed as neither given nor endorsed by Winbond.
________________________________
The privileged confidential information contained in this email is intended for use only by the addressees as indicated by the original sender of this email. If you are not the addressee indicated in this email or are not responsible for delivery of the email to such a person, please kindly reply to the sender indicating this fact and delete all copies of it from your computer and network server immediately. Your cooperation is highly appreciated. It is advised that any unauthorized use of confidential information of Winbond is strictly prohibited; and any information in this email irrelevant to the official business of Winbond shall be deemed as neither given nor endorsed by Winbond.
RE: change yarn application priority
Posted by Henry Hung <YT...@winbond.com>.
I manage to implement the fair scheduler and very happy with the result.
The way I use it is to create a queue that constraint the number of resources (maxResources) to be around 70% of total resources, fortunately the fair-scheduler.xml can be modified on-fly without restarting the yarn cluster.
Thank to every one that provide me with all the information.
From: Michael Segel [mailto:msegel_hadoop@hotmail.com]
Sent: Tuesday, June 03, 2014 8:13 PM
To: user@hadoop.apache.org
Subject: Re: change yarn application priority
WRT capacity scheduler, its not so much changing the priority of a job, but allowing for pre-emption. Note that I guess you could raise the one job's priority, and then the other job's priority so that when a task finishes the other job gets the next slot. However, you're still stuck waiting and that can take time.)
Here's the rub though... pre-empting a task could have a negative impact on the long running job.
If you have large jobs and small jobs, you could create multiple queues and leave some capacity to the smaller job queue.
So you need to specify the queue up front. (Its been a while, but can you move jobs across queues? )
I would suggest that you look at fair scheduler, but also consider multiple queues under capacity scheduler. You may have admin jobs you want to run in the background while other tasks are running.
On Jun 2, 2014, at 8:10 PM, Henry Hung <YT...@winbond.com>> wrote:
@Rohith Sharma
Thank you for the confirmation.
I already google some information regarding scheduler in Hadoop 2.2.0, there are 2 ways to do it: FairScheduler or CapacityScheduler.
Apparently from google result, there are more article mentions that FairScheduler is better than CapacityScheduler.
So, I intend to use FairScheduler first, if you have any more suggestion, please let me know, thank you again.
Best regards,
Henry
From: Rohith Sharma K S [mailto:rohithsharmaks@huawei.com]
Sent: Friday, May 30, 2014 5:49 PM
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: RE: change yarn application priority
Hi
Currently there is no provision for changing application priority within the same queue. Follow the Jirahttps://issues.apache.org/jira/i#browse/YARN-1963 for this new feature.
One way you can achieve by using enabling scheduler monitors for CapacitySchedulers.
Steps to be follow is
1. Configure 2 queues, follow http://hadoop.apache.org/docs/r2.3.0/hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html
2. Enable scheduler monitor
yarn.resourcemanager.scheduler.monitor.enable = true
One job you submit to queue 1 which run 2hours. Another job you submit queue 2.
Hope this will help you.
Thanks & Regards
Rohith Sharma K S
This e-mail and its attachments contain confidential information from HUAWEI, which
is intended only for the person or entity whose address is listed above. Any use of the
information contained herein in any way (including, but not limited to, total or partial
disclosure, reproduction, or dissemination) by persons other than the intended
recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender by
phone or email immediately and delete it!
From: Henry Hung [mailto:YTHung1@winbond.com]
Sent: 30 May 2014 11:53
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: change yarn application priority
HI All,
I have an application that consumes all of nodemanager capacity (30 Map and 1 Reducer) and will need 4 hours to finish.
Let's say I need to run another application that will be quicker to finish (30 minutes) and only need 1 Map and 1 Reducer.
If I just execute the new application, it will be in queue waiting for the 1st application to finish.
Is there a way to change the 2nd application priority to higher than the 1st and let resourcemanager immediately execute the 2ndapplication?
I'm using Hadoop-2.2.0.
Best regards,
Henry
________________________________
The privileged confidential information contained in this email is intended for use only by the addressees as indicated by the original sender of this email. If you are not the addressee indicated in this email or are not responsible for delivery of the email to such a person, please kindly reply to the sender indicating this fact and delete all copies of it from your computer and network server immediately. Your cooperation is highly appreciated. It is advised that any unauthorized use of confidential information of Winbond is strictly prohibited; and any information in this email irrelevant to the official business of Winbond shall be deemed as neither given nor endorsed by Winbond.
________________________________
The privileged confidential information contained in this email is intended for use only by the addressees as indicated by the original sender of this email. If you are not the addressee indicated in this email or are not responsible for delivery of the email to such a person, please kindly reply to the sender indicating this fact and delete all copies of it from your computer and network server immediately. Your cooperation is highly appreciated. It is advised that any unauthorized use of confidential information of Winbond is strictly prohibited; and any information in this email irrelevant to the official business of Winbond shall be deemed as neither given nor endorsed by Winbond.
________________________________
The privileged confidential information contained in this email is intended for use only by the addressees as indicated by the original sender of this email. If you are not the addressee indicated in this email or are not responsible for delivery of the email to such a person, please kindly reply to the sender indicating this fact and delete all copies of it from your computer and network server immediately. Your cooperation is highly appreciated. It is advised that any unauthorized use of confidential information of Winbond is strictly prohibited; and any information in this email irrelevant to the official business of Winbond shall be deemed as neither given nor endorsed by Winbond.
Re: change yarn application priority
Posted by Michael Segel <ms...@hotmail.com>.
WRT capacity scheduler, its not so much changing the priority of a job, but allowing for pre-emption. Note that I guess you could raise the one job's priority, and then the other job's priority so that when a task finishes the other job gets the next slot. However, you're still stuck waiting and that can take time.)
Here's the rub though... pre-empting a task could have a negative impact on the long running job.
If you have large jobs and small jobs, you could create multiple queues and leave some capacity to the smaller job queue.
So you need to specify the queue up front. (Its been a while, but can you move jobs across queues? )
I would suggest that you look at fair scheduler, but also consider multiple queues under capacity scheduler. You may have admin jobs you want to run in the background while other tasks are running.
On Jun 2, 2014, at 8:10 PM, Henry Hung <YT...@winbond.com> wrote:
> @Rohith Sharma
>
> Thank you for the confirmation.
> I already google some information regarding scheduler in Hadoop 2.2.0, there are 2 ways to do it: FairScheduler or CapacityScheduler.
> Apparently from google result, there are more article mentions that FairScheduler is better than CapacityScheduler.
> So, I intend to use FairScheduler first, if you have any more suggestion, please let me know, thank you again.
>
> Best regards,
> Henry
>
> From: Rohith Sharma K S [mailto:rohithsharmaks@huawei.com]
> Sent: Friday, May 30, 2014 5:49 PM
> To: user@hadoop.apache.org
> Subject: RE: change yarn application priority
>
> Hi
>
> Currently there is no provision for changing application priority within the same queue. Follow the Jirahttps://issues.apache.org/jira/i#browse/YARN-1963 for this new feature.
>
> One way you can achieve by using enabling scheduler monitors for CapacitySchedulers.
> Steps to be follow is
> 1. Configure 2 queues, follow http://hadoop.apache.org/docs/r2.3.0/hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html
> 2. Enable scheduler monitor
> yarn.resourcemanager.scheduler.monitor.enable = true
>
> One job you submit to queue 1 which run 2hours. Another job you submit queue 2.
>
> Hope this will help you.
>
> Thanks & Regards
> Rohith Sharma K S
>
>
> This e-mail and its attachments contain confidential information from HUAWEI, which
> is intended only for the person or entity whose address is listed above. Any use of the
> information contained herein in any way (including, but not limited to, total or partial
> disclosure, reproduction, or dissemination) by persons other than the intended
> recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender by
> phone or email immediately and delete it!
>
> From: Henry Hung [mailto:YTHung1@winbond.com]
> Sent: 30 May 2014 11:53
> To: user@hadoop.apache.org
> Subject: change yarn application priority
>
> HI All,
>
> I have an application that consumes all of nodemanager capacity (30 Map and 1 Reducer) and will need 4 hours to finish.
> Let’s say I need to run another application that will be quicker to finish (30 minutes) and only need 1 Map and 1 Reducer.
> If I just execute the new application, it will be in queue waiting for the 1st application to finish.
> Is there a way to change the 2nd application priority to higher than the 1st and let resourcemanager immediately execute the 2ndapplication?
>
> I’m using Hadoop-2.2.0.
>
> Best regards,
> Henry
>
> The privileged confidential information contained in this email is intended for use only by the addressees as indicated by the original sender of this email. If you are not the addressee indicated in this email or are not responsible for delivery of the email to such a person, please kindly reply to the sender indicating this fact and delete all copies of it from your computer and network server immediately. Your cooperation is highly appreciated. It is advised that any unauthorized use of confidential information of Winbond is strictly prohibited; and any information in this email irrelevant to the official business of Winbond shall be deemed as neither given nor endorsed by Winbond.
>
> The privileged confidential information contained in this email is intended for use only by the addressees as indicated by the original sender of this email. If you are not the addressee indicated in this email or are not responsible for delivery of the email to such a person, please kindly reply to the sender indicating this fact and delete all copies of it from your computer and network server immediately. Your cooperation is highly appreciated. It is advised that any unauthorized use of confidential information of Winbond is strictly prohibited; and any information in this email irrelevant to the official business of Winbond shall be deemed as neither given nor endorsed by Winbond.
Re: change yarn application priority
Posted by Michael Segel <ms...@hotmail.com>.
WRT capacity scheduler, its not so much changing the priority of a job, but allowing for pre-emption. Note that I guess you could raise the one job's priority, and then the other job's priority so that when a task finishes the other job gets the next slot. However, you're still stuck waiting and that can take time.)
Here's the rub though... pre-empting a task could have a negative impact on the long running job.
If you have large jobs and small jobs, you could create multiple queues and leave some capacity to the smaller job queue.
So you need to specify the queue up front. (Its been a while, but can you move jobs across queues? )
I would suggest that you look at fair scheduler, but also consider multiple queues under capacity scheduler. You may have admin jobs you want to run in the background while other tasks are running.
On Jun 2, 2014, at 8:10 PM, Henry Hung <YT...@winbond.com> wrote:
> @Rohith Sharma
>
> Thank you for the confirmation.
> I already google some information regarding scheduler in Hadoop 2.2.0, there are 2 ways to do it: FairScheduler or CapacityScheduler.
> Apparently from google result, there are more article mentions that FairScheduler is better than CapacityScheduler.
> So, I intend to use FairScheduler first, if you have any more suggestion, please let me know, thank you again.
>
> Best regards,
> Henry
>
> From: Rohith Sharma K S [mailto:rohithsharmaks@huawei.com]
> Sent: Friday, May 30, 2014 5:49 PM
> To: user@hadoop.apache.org
> Subject: RE: change yarn application priority
>
> Hi
>
> Currently there is no provision for changing application priority within the same queue. Follow the Jirahttps://issues.apache.org/jira/i#browse/YARN-1963 for this new feature.
>
> One way you can achieve by using enabling scheduler monitors for CapacitySchedulers.
> Steps to be follow is
> 1. Configure 2 queues, follow http://hadoop.apache.org/docs/r2.3.0/hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html
> 2. Enable scheduler monitor
> yarn.resourcemanager.scheduler.monitor.enable = true
>
> One job you submit to queue 1 which run 2hours. Another job you submit queue 2.
>
> Hope this will help you.
>
> Thanks & Regards
> Rohith Sharma K S
>
>
> This e-mail and its attachments contain confidential information from HUAWEI, which
> is intended only for the person or entity whose address is listed above. Any use of the
> information contained herein in any way (including, but not limited to, total or partial
> disclosure, reproduction, or dissemination) by persons other than the intended
> recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender by
> phone or email immediately and delete it!
>
> From: Henry Hung [mailto:YTHung1@winbond.com]
> Sent: 30 May 2014 11:53
> To: user@hadoop.apache.org
> Subject: change yarn application priority
>
> HI All,
>
> I have an application that consumes all of nodemanager capacity (30 Map and 1 Reducer) and will need 4 hours to finish.
> Let’s say I need to run another application that will be quicker to finish (30 minutes) and only need 1 Map and 1 Reducer.
> If I just execute the new application, it will be in queue waiting for the 1st application to finish.
> Is there a way to change the 2nd application priority to higher than the 1st and let resourcemanager immediately execute the 2ndapplication?
>
> I’m using Hadoop-2.2.0.
>
> Best regards,
> Henry
>
> The privileged confidential information contained in this email is intended for use only by the addressees as indicated by the original sender of this email. If you are not the addressee indicated in this email or are not responsible for delivery of the email to such a person, please kindly reply to the sender indicating this fact and delete all copies of it from your computer and network server immediately. Your cooperation is highly appreciated. It is advised that any unauthorized use of confidential information of Winbond is strictly prohibited; and any information in this email irrelevant to the official business of Winbond shall be deemed as neither given nor endorsed by Winbond.
>
> The privileged confidential information contained in this email is intended for use only by the addressees as indicated by the original sender of this email. If you are not the addressee indicated in this email or are not responsible for delivery of the email to such a person, please kindly reply to the sender indicating this fact and delete all copies of it from your computer and network server immediately. Your cooperation is highly appreciated. It is advised that any unauthorized use of confidential information of Winbond is strictly prohibited; and any information in this email irrelevant to the official business of Winbond shall be deemed as neither given nor endorsed by Winbond.
Re: change yarn application priority
Posted by Michael Segel <ms...@hotmail.com>.
WRT capacity scheduler, its not so much changing the priority of a job, but allowing for pre-emption. Note that I guess you could raise the one job's priority, and then the other job's priority so that when a task finishes the other job gets the next slot. However, you're still stuck waiting and that can take time.)
Here's the rub though... pre-empting a task could have a negative impact on the long running job.
If you have large jobs and small jobs, you could create multiple queues and leave some capacity to the smaller job queue.
So you need to specify the queue up front. (Its been a while, but can you move jobs across queues? )
I would suggest that you look at fair scheduler, but also consider multiple queues under capacity scheduler. You may have admin jobs you want to run in the background while other tasks are running.
On Jun 2, 2014, at 8:10 PM, Henry Hung <YT...@winbond.com> wrote:
> @Rohith Sharma
>
> Thank you for the confirmation.
> I already google some information regarding scheduler in Hadoop 2.2.0, there are 2 ways to do it: FairScheduler or CapacityScheduler.
> Apparently from google result, there are more article mentions that FairScheduler is better than CapacityScheduler.
> So, I intend to use FairScheduler first, if you have any more suggestion, please let me know, thank you again.
>
> Best regards,
> Henry
>
> From: Rohith Sharma K S [mailto:rohithsharmaks@huawei.com]
> Sent: Friday, May 30, 2014 5:49 PM
> To: user@hadoop.apache.org
> Subject: RE: change yarn application priority
>
> Hi
>
> Currently there is no provision for changing application priority within the same queue. Follow the Jirahttps://issues.apache.org/jira/i#browse/YARN-1963 for this new feature.
>
> One way you can achieve by using enabling scheduler monitors for CapacitySchedulers.
> Steps to be follow is
> 1. Configure 2 queues, follow http://hadoop.apache.org/docs/r2.3.0/hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html
> 2. Enable scheduler monitor
> yarn.resourcemanager.scheduler.monitor.enable = true
>
> One job you submit to queue 1 which run 2hours. Another job you submit queue 2.
>
> Hope this will help you.
>
> Thanks & Regards
> Rohith Sharma K S
>
>
> This e-mail and its attachments contain confidential information from HUAWEI, which
> is intended only for the person or entity whose address is listed above. Any use of the
> information contained herein in any way (including, but not limited to, total or partial
> disclosure, reproduction, or dissemination) by persons other than the intended
> recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender by
> phone or email immediately and delete it!
>
> From: Henry Hung [mailto:YTHung1@winbond.com]
> Sent: 30 May 2014 11:53
> To: user@hadoop.apache.org
> Subject: change yarn application priority
>
> HI All,
>
> I have an application that consumes all of nodemanager capacity (30 Map and 1 Reducer) and will need 4 hours to finish.
> Let’s say I need to run another application that will be quicker to finish (30 minutes) and only need 1 Map and 1 Reducer.
> If I just execute the new application, it will be in queue waiting for the 1st application to finish.
> Is there a way to change the 2nd application priority to higher than the 1st and let resourcemanager immediately execute the 2ndapplication?
>
> I’m using Hadoop-2.2.0.
>
> Best regards,
> Henry
>
> The privileged confidential information contained in this email is intended for use only by the addressees as indicated by the original sender of this email. If you are not the addressee indicated in this email or are not responsible for delivery of the email to such a person, please kindly reply to the sender indicating this fact and delete all copies of it from your computer and network server immediately. Your cooperation is highly appreciated. It is advised that any unauthorized use of confidential information of Winbond is strictly prohibited; and any information in this email irrelevant to the official business of Winbond shall be deemed as neither given nor endorsed by Winbond.
>
> The privileged confidential information contained in this email is intended for use only by the addressees as indicated by the original sender of this email. If you are not the addressee indicated in this email or are not responsible for delivery of the email to such a person, please kindly reply to the sender indicating this fact and delete all copies of it from your computer and network server immediately. Your cooperation is highly appreciated. It is advised that any unauthorized use of confidential information of Winbond is strictly prohibited; and any information in this email irrelevant to the official business of Winbond shall be deemed as neither given nor endorsed by Winbond.
Re: change yarn application priority
Posted by Michael Segel <ms...@hotmail.com>.
WRT capacity scheduler, its not so much changing the priority of a job, but allowing for pre-emption. Note that I guess you could raise the one job's priority, and then the other job's priority so that when a task finishes the other job gets the next slot. However, you're still stuck waiting and that can take time.)
Here's the rub though... pre-empting a task could have a negative impact on the long running job.
If you have large jobs and small jobs, you could create multiple queues and leave some capacity to the smaller job queue.
So you need to specify the queue up front. (Its been a while, but can you move jobs across queues? )
I would suggest that you look at fair scheduler, but also consider multiple queues under capacity scheduler. You may have admin jobs you want to run in the background while other tasks are running.
On Jun 2, 2014, at 8:10 PM, Henry Hung <YT...@winbond.com> wrote:
> @Rohith Sharma
>
> Thank you for the confirmation.
> I already google some information regarding scheduler in Hadoop 2.2.0, there are 2 ways to do it: FairScheduler or CapacityScheduler.
> Apparently from google result, there are more article mentions that FairScheduler is better than CapacityScheduler.
> So, I intend to use FairScheduler first, if you have any more suggestion, please let me know, thank you again.
>
> Best regards,
> Henry
>
> From: Rohith Sharma K S [mailto:rohithsharmaks@huawei.com]
> Sent: Friday, May 30, 2014 5:49 PM
> To: user@hadoop.apache.org
> Subject: RE: change yarn application priority
>
> Hi
>
> Currently there is no provision for changing application priority within the same queue. Follow the Jirahttps://issues.apache.org/jira/i#browse/YARN-1963 for this new feature.
>
> One way you can achieve by using enabling scheduler monitors for CapacitySchedulers.
> Steps to be follow is
> 1. Configure 2 queues, follow http://hadoop.apache.org/docs/r2.3.0/hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html
> 2. Enable scheduler monitor
> yarn.resourcemanager.scheduler.monitor.enable = true
>
> One job you submit to queue 1 which run 2hours. Another job you submit queue 2.
>
> Hope this will help you.
>
> Thanks & Regards
> Rohith Sharma K S
>
>
> This e-mail and its attachments contain confidential information from HUAWEI, which
> is intended only for the person or entity whose address is listed above. Any use of the
> information contained herein in any way (including, but not limited to, total or partial
> disclosure, reproduction, or dissemination) by persons other than the intended
> recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender by
> phone or email immediately and delete it!
>
> From: Henry Hung [mailto:YTHung1@winbond.com]
> Sent: 30 May 2014 11:53
> To: user@hadoop.apache.org
> Subject: change yarn application priority
>
> HI All,
>
> I have an application that consumes all of nodemanager capacity (30 Map and 1 Reducer) and will need 4 hours to finish.
> Let’s say I need to run another application that will be quicker to finish (30 minutes) and only need 1 Map and 1 Reducer.
> If I just execute the new application, it will be in queue waiting for the 1st application to finish.
> Is there a way to change the 2nd application priority to higher than the 1st and let resourcemanager immediately execute the 2ndapplication?
>
> I’m using Hadoop-2.2.0.
>
> Best regards,
> Henry
>
> The privileged confidential information contained in this email is intended for use only by the addressees as indicated by the original sender of this email. If you are not the addressee indicated in this email or are not responsible for delivery of the email to such a person, please kindly reply to the sender indicating this fact and delete all copies of it from your computer and network server immediately. Your cooperation is highly appreciated. It is advised that any unauthorized use of confidential information of Winbond is strictly prohibited; and any information in this email irrelevant to the official business of Winbond shall be deemed as neither given nor endorsed by Winbond.
>
> The privileged confidential information contained in this email is intended for use only by the addressees as indicated by the original sender of this email. If you are not the addressee indicated in this email or are not responsible for delivery of the email to such a person, please kindly reply to the sender indicating this fact and delete all copies of it from your computer and network server immediately. Your cooperation is highly appreciated. It is advised that any unauthorized use of confidential information of Winbond is strictly prohibited; and any information in this email irrelevant to the official business of Winbond shall be deemed as neither given nor endorsed by Winbond.