You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Viswanathan J <ja...@gmail.com> on 2013/11/21 19:24:54 UTC
Jobtracker fair scheduler
Hi,
I'm running hadoop with 1.2.1, all my jobs are running in single queue
(Queue 1) only all the time. But I have configured default, queue 1&2.
Why jobs are not scheduled to all the queues.
Please help. Running like this will be any issue?
Thanks,
Re: Jobtracker fair scheduler
Posted by Viswanathan J <ja...@gmail.com>.
Thanks Anurag. I will check and keep you posted in case of any issue.
On Nov 22, 2013 12:54 AM, "anurag tangri" <an...@yahoo.com> wrote:
> Here's step-by-step on how to do this:
> ================
>
> Here's step by step on how to do this:
>
> 1. Add following in mapred-site.xml:
>
>
> <property>
> <name>mapred.fairscheduler.preemption</name>
> <final>true</final>
> <value>true</value>
> </property>
> <property>
> <name>mapred.jobtracker.taskScheduler</name>
> <final>true</final>
> <value>org.apache.hadoop.mapred.FairScheduler</value>
> </property>
> <property>
> <name>mapred.acls.enabled</name>
> <final>true</final>
> <value>true</value>
> </property>
> <property>
> <name>mapred.fairscheduler.allow.undeclared.pools</name>
> <final>true</final>
> <value>false</value>
> </property>
> <property>
> <name>mapred.fairscheduler.poolnameproperty</name>
> <final>true</final>
> <value>mapred.job.queue.name</value>
> </property>
> <property>
> <name>mapred.fairscheduler.allocation.file</name>
> <final>true</final>
> <value>/etc/hadoop/conf/allocations.xml</value>
> </property>
> <property>
> <name>mapred.queue.names</name>
> <final>true</final>
> <value>sqoop,default</value>
> </property>
> <property>
> <name>mapreduce.job.acl-view-job</name>
> <final>true</final>
> <value>*</value>
> </property>
>
>
> 2. Create mapred-queue-acls.xml in same dir as mapred-site.xml where you
> define all the queues:
>
> <configuration>
> <property>
> <name>mapred.queue.sqoop.acl-submit-job</name>
> <value>usera, userb</value>
> </property>
> <property>
> <name>mapred.queue.sqoop.acl-administer-jobs</name>
> <value>usera, userc</value>
> </property>
> <! above two properties for every queue defined in mapred-site.xml -->
> </configuration>
>
>
> 3. Define fair scheduler allocations file (location specified in
> maped-site.xml) to use the queues defined and assign resources to them:
>
> <?xml version="1.0"?>
> <allocations>
> <defaultMinSharePreemptionTimeout>300</defaultMinSharePreemptionTimeout>
> <pool name="sqoop">
> <minMaps>700</minMaps>
> <minReduces>175</minReduces>
> <maxRunningJobs>25</maxRunningJobs>
> </pool>
> <pool name="default">
> <minMaps>120</minMaps>
> <minReduces>40</minReduces>
> <maxRunningJobs>40</maxRunningJobs>
> </pool>
> <fairSharePreemptionTimeout>600</fairSharePreemptionTimeout>
> </allocations>
>
>
> 4. After this, restart job tracker.
>
>
> And your queues should be in place. You can then submit job to right queue
> using
> -Dmapred.job.queue.name=sqoop or default
>
> And check the job going to actual queue at the link:
> http://<job tracker hostname>:50030/scheduler
>
>
> Look at the property mapreduce.job.acl-view-job in mapred-site. Means
> anyone can view the jobs.
>
>
> Try above and let us know any issues you still face.
>
>
> ================================
>
>
> Anurag Tangri
>
>
> Never wear your best trousers when you go out to fight for freedom and
> truth.- Henrik Ibsen
>
>
> On Thursday, November 21, 2013 11:21 AM, anurag tangri <
> anurag_tangri@yahoo.com> wrote:
> Hi Viswanathan,
> What steps have you followed to set fair scheduler ?
>
> Thanks,
> Anurag Tangri
>
>
> Never wear your best trousers when you go out to fight for freedom and
> truth.- Henrik Ibsen
>
>
> On Thursday, November 21, 2013 10:25 AM, Viswanathan J <
> jayamviswanathan@gmail.com> wrote:
> Hi,
> I'm running hadoop with 1.2.1, all my jobs are running in single queue
> (Queue 1) only all the time. But I have configured default, queue 1&2.
> Why jobs are not scheduled to all the queues.
> Please help. Running like this will be any issue?
> Thanks,
>
>
>
>
>
Re: Jobtracker fair scheduler
Posted by Viswanathan J <ja...@gmail.com>.
Thanks Anurag. I will check and keep you posted in case of any issue.
On Nov 22, 2013 12:54 AM, "anurag tangri" <an...@yahoo.com> wrote:
> Here's step-by-step on how to do this:
> ================
>
> Here's step by step on how to do this:
>
> 1. Add following in mapred-site.xml:
>
>
> <property>
> <name>mapred.fairscheduler.preemption</name>
> <final>true</final>
> <value>true</value>
> </property>
> <property>
> <name>mapred.jobtracker.taskScheduler</name>
> <final>true</final>
> <value>org.apache.hadoop.mapred.FairScheduler</value>
> </property>
> <property>
> <name>mapred.acls.enabled</name>
> <final>true</final>
> <value>true</value>
> </property>
> <property>
> <name>mapred.fairscheduler.allow.undeclared.pools</name>
> <final>true</final>
> <value>false</value>
> </property>
> <property>
> <name>mapred.fairscheduler.poolnameproperty</name>
> <final>true</final>
> <value>mapred.job.queue.name</value>
> </property>
> <property>
> <name>mapred.fairscheduler.allocation.file</name>
> <final>true</final>
> <value>/etc/hadoop/conf/allocations.xml</value>
> </property>
> <property>
> <name>mapred.queue.names</name>
> <final>true</final>
> <value>sqoop,default</value>
> </property>
> <property>
> <name>mapreduce.job.acl-view-job</name>
> <final>true</final>
> <value>*</value>
> </property>
>
>
> 2. Create mapred-queue-acls.xml in same dir as mapred-site.xml where you
> define all the queues:
>
> <configuration>
> <property>
> <name>mapred.queue.sqoop.acl-submit-job</name>
> <value>usera, userb</value>
> </property>
> <property>
> <name>mapred.queue.sqoop.acl-administer-jobs</name>
> <value>usera, userc</value>
> </property>
> <! above two properties for every queue defined in mapred-site.xml -->
> </configuration>
>
>
> 3. Define fair scheduler allocations file (location specified in
> maped-site.xml) to use the queues defined and assign resources to them:
>
> <?xml version="1.0"?>
> <allocations>
> <defaultMinSharePreemptionTimeout>300</defaultMinSharePreemptionTimeout>
> <pool name="sqoop">
> <minMaps>700</minMaps>
> <minReduces>175</minReduces>
> <maxRunningJobs>25</maxRunningJobs>
> </pool>
> <pool name="default">
> <minMaps>120</minMaps>
> <minReduces>40</minReduces>
> <maxRunningJobs>40</maxRunningJobs>
> </pool>
> <fairSharePreemptionTimeout>600</fairSharePreemptionTimeout>
> </allocations>
>
>
> 4. After this, restart job tracker.
>
>
> And your queues should be in place. You can then submit job to right queue
> using
> -Dmapred.job.queue.name=sqoop or default
>
> And check the job going to actual queue at the link:
> http://<job tracker hostname>:50030/scheduler
>
>
> Look at the property mapreduce.job.acl-view-job in mapred-site. Means
> anyone can view the jobs.
>
>
> Try above and let us know any issues you still face.
>
>
> ================================
>
>
> Anurag Tangri
>
>
> Never wear your best trousers when you go out to fight for freedom and
> truth.- Henrik Ibsen
>
>
> On Thursday, November 21, 2013 11:21 AM, anurag tangri <
> anurag_tangri@yahoo.com> wrote:
> Hi Viswanathan,
> What steps have you followed to set fair scheduler ?
>
> Thanks,
> Anurag Tangri
>
>
> Never wear your best trousers when you go out to fight for freedom and
> truth.- Henrik Ibsen
>
>
> On Thursday, November 21, 2013 10:25 AM, Viswanathan J <
> jayamviswanathan@gmail.com> wrote:
> Hi,
> I'm running hadoop with 1.2.1, all my jobs are running in single queue
> (Queue 1) only all the time. But I have configured default, queue 1&2.
> Why jobs are not scheduled to all the queues.
> Please help. Running like this will be any issue?
> Thanks,
>
>
>
>
>
Re: Jobtracker fair scheduler
Posted by Viswanathan J <ja...@gmail.com>.
Thanks Anurag. I will check and keep you posted in case of any issue.
On Nov 22, 2013 12:54 AM, "anurag tangri" <an...@yahoo.com> wrote:
> Here's step-by-step on how to do this:
> ================
>
> Here's step by step on how to do this:
>
> 1. Add following in mapred-site.xml:
>
>
> <property>
> <name>mapred.fairscheduler.preemption</name>
> <final>true</final>
> <value>true</value>
> </property>
> <property>
> <name>mapred.jobtracker.taskScheduler</name>
> <final>true</final>
> <value>org.apache.hadoop.mapred.FairScheduler</value>
> </property>
> <property>
> <name>mapred.acls.enabled</name>
> <final>true</final>
> <value>true</value>
> </property>
> <property>
> <name>mapred.fairscheduler.allow.undeclared.pools</name>
> <final>true</final>
> <value>false</value>
> </property>
> <property>
> <name>mapred.fairscheduler.poolnameproperty</name>
> <final>true</final>
> <value>mapred.job.queue.name</value>
> </property>
> <property>
> <name>mapred.fairscheduler.allocation.file</name>
> <final>true</final>
> <value>/etc/hadoop/conf/allocations.xml</value>
> </property>
> <property>
> <name>mapred.queue.names</name>
> <final>true</final>
> <value>sqoop,default</value>
> </property>
> <property>
> <name>mapreduce.job.acl-view-job</name>
> <final>true</final>
> <value>*</value>
> </property>
>
>
> 2. Create mapred-queue-acls.xml in same dir as mapred-site.xml where you
> define all the queues:
>
> <configuration>
> <property>
> <name>mapred.queue.sqoop.acl-submit-job</name>
> <value>usera, userb</value>
> </property>
> <property>
> <name>mapred.queue.sqoop.acl-administer-jobs</name>
> <value>usera, userc</value>
> </property>
> <! above two properties for every queue defined in mapred-site.xml -->
> </configuration>
>
>
> 3. Define fair scheduler allocations file (location specified in
> maped-site.xml) to use the queues defined and assign resources to them:
>
> <?xml version="1.0"?>
> <allocations>
> <defaultMinSharePreemptionTimeout>300</defaultMinSharePreemptionTimeout>
> <pool name="sqoop">
> <minMaps>700</minMaps>
> <minReduces>175</minReduces>
> <maxRunningJobs>25</maxRunningJobs>
> </pool>
> <pool name="default">
> <minMaps>120</minMaps>
> <minReduces>40</minReduces>
> <maxRunningJobs>40</maxRunningJobs>
> </pool>
> <fairSharePreemptionTimeout>600</fairSharePreemptionTimeout>
> </allocations>
>
>
> 4. After this, restart job tracker.
>
>
> And your queues should be in place. You can then submit job to right queue
> using
> -Dmapred.job.queue.name=sqoop or default
>
> And check the job going to actual queue at the link:
> http://<job tracker hostname>:50030/scheduler
>
>
> Look at the property mapreduce.job.acl-view-job in mapred-site. Means
> anyone can view the jobs.
>
>
> Try above and let us know any issues you still face.
>
>
> ================================
>
>
> Anurag Tangri
>
>
> Never wear your best trousers when you go out to fight for freedom and
> truth.- Henrik Ibsen
>
>
> On Thursday, November 21, 2013 11:21 AM, anurag tangri <
> anurag_tangri@yahoo.com> wrote:
> Hi Viswanathan,
> What steps have you followed to set fair scheduler ?
>
> Thanks,
> Anurag Tangri
>
>
> Never wear your best trousers when you go out to fight for freedom and
> truth.- Henrik Ibsen
>
>
> On Thursday, November 21, 2013 10:25 AM, Viswanathan J <
> jayamviswanathan@gmail.com> wrote:
> Hi,
> I'm running hadoop with 1.2.1, all my jobs are running in single queue
> (Queue 1) only all the time. But I have configured default, queue 1&2.
> Why jobs are not scheduled to all the queues.
> Please help. Running like this will be any issue?
> Thanks,
>
>
>
>
>
Re: Jobtracker fair scheduler
Posted by Viswanathan J <ja...@gmail.com>.
Thanks Anurag. I will check and keep you posted in case of any issue.
On Nov 22, 2013 12:54 AM, "anurag tangri" <an...@yahoo.com> wrote:
> Here's step-by-step on how to do this:
> ================
>
> Here's step by step on how to do this:
>
> 1. Add following in mapred-site.xml:
>
>
> <property>
> <name>mapred.fairscheduler.preemption</name>
> <final>true</final>
> <value>true</value>
> </property>
> <property>
> <name>mapred.jobtracker.taskScheduler</name>
> <final>true</final>
> <value>org.apache.hadoop.mapred.FairScheduler</value>
> </property>
> <property>
> <name>mapred.acls.enabled</name>
> <final>true</final>
> <value>true</value>
> </property>
> <property>
> <name>mapred.fairscheduler.allow.undeclared.pools</name>
> <final>true</final>
> <value>false</value>
> </property>
> <property>
> <name>mapred.fairscheduler.poolnameproperty</name>
> <final>true</final>
> <value>mapred.job.queue.name</value>
> </property>
> <property>
> <name>mapred.fairscheduler.allocation.file</name>
> <final>true</final>
> <value>/etc/hadoop/conf/allocations.xml</value>
> </property>
> <property>
> <name>mapred.queue.names</name>
> <final>true</final>
> <value>sqoop,default</value>
> </property>
> <property>
> <name>mapreduce.job.acl-view-job</name>
> <final>true</final>
> <value>*</value>
> </property>
>
>
> 2. Create mapred-queue-acls.xml in same dir as mapred-site.xml where you
> define all the queues:
>
> <configuration>
> <property>
> <name>mapred.queue.sqoop.acl-submit-job</name>
> <value>usera, userb</value>
> </property>
> <property>
> <name>mapred.queue.sqoop.acl-administer-jobs</name>
> <value>usera, userc</value>
> </property>
> <! above two properties for every queue defined in mapred-site.xml -->
> </configuration>
>
>
> 3. Define fair scheduler allocations file (location specified in
> maped-site.xml) to use the queues defined and assign resources to them:
>
> <?xml version="1.0"?>
> <allocations>
> <defaultMinSharePreemptionTimeout>300</defaultMinSharePreemptionTimeout>
> <pool name="sqoop">
> <minMaps>700</minMaps>
> <minReduces>175</minReduces>
> <maxRunningJobs>25</maxRunningJobs>
> </pool>
> <pool name="default">
> <minMaps>120</minMaps>
> <minReduces>40</minReduces>
> <maxRunningJobs>40</maxRunningJobs>
> </pool>
> <fairSharePreemptionTimeout>600</fairSharePreemptionTimeout>
> </allocations>
>
>
> 4. After this, restart job tracker.
>
>
> And your queues should be in place. You can then submit job to right queue
> using
> -Dmapred.job.queue.name=sqoop or default
>
> And check the job going to actual queue at the link:
> http://<job tracker hostname>:50030/scheduler
>
>
> Look at the property mapreduce.job.acl-view-job in mapred-site. Means
> anyone can view the jobs.
>
>
> Try above and let us know any issues you still face.
>
>
> ================================
>
>
> Anurag Tangri
>
>
> Never wear your best trousers when you go out to fight for freedom and
> truth.- Henrik Ibsen
>
>
> On Thursday, November 21, 2013 11:21 AM, anurag tangri <
> anurag_tangri@yahoo.com> wrote:
> Hi Viswanathan,
> What steps have you followed to set fair scheduler ?
>
> Thanks,
> Anurag Tangri
>
>
> Never wear your best trousers when you go out to fight for freedom and
> truth.- Henrik Ibsen
>
>
> On Thursday, November 21, 2013 10:25 AM, Viswanathan J <
> jayamviswanathan@gmail.com> wrote:
> Hi,
> I'm running hadoop with 1.2.1, all my jobs are running in single queue
> (Queue 1) only all the time. But I have configured default, queue 1&2.
> Why jobs are not scheduled to all the queues.
> Please help. Running like this will be any issue?
> Thanks,
>
>
>
>
>
Re: Jobtracker fair scheduler
Posted by anurag tangri <an...@yahoo.com>.
Here's step-by-step on how to do this:
================
Here's step by step on how to do this:
1. Add following in mapred-site.xml:
<property>
<name>mapred.fairscheduler.preemption</name>
<final>true</final>
<value>true</value>
</property>
<property>
<name>mapred.jobtracker.taskScheduler</name>
<final>true</final>
<value>org.apache.hadoop.mapred.FairScheduler</value>
</property>
<property>
<name>mapred.acls.enabled</name>
<final>true</final>
<value>true</value>
</property>
<property>
<name>mapred.fairscheduler.allow.undeclared.pools</name>
<final>true</final>
<value>false</value>
</property>
<property>
<name>mapred.fairscheduler.poolnameproperty</name>
<final>true</final>
<value>mapred.job.queue.name</value>
</property>
<property>
<name>mapred.fairscheduler.allocation.file</name>
<final>true</final>
<value>/etc/hadoop/conf/allocations.xml</value>
</property>
<property>
<name>mapred.queue.names</name>
<final>true</final>
<value>sqoop,default</value>
</property>
<property>
<name>mapreduce.job.acl-view-job</name>
<final>true</final>
<value>*</value>
</property>
2. Create mapred-queue-acls.xml in same dir as mapred-site.xml where you define all the queues:
<configuration>
<property>
<name>mapred.queue.sqoop.acl-submit-job</name>
<value>usera, userb</value>
</property>
<property>
<name>mapred.queue.sqoop.acl-administer-jobs</name>
<value>usera, userc</value>
</property>
<! above two properties for every queue defined in mapred-site.xml -->
</configuration>
3. Define fair scheduler allocations file (location specified in maped-site.xml) to use the queues defined and assign resources to them:
<?xml version="1.0"?>
<allocations>
<defaultMinSharePreemptionTimeout>300</defaultMinSharePreemptionTimeout>
<pool name="sqoop">
<minMaps>700</minMaps>
<minReduces>175</minReduces>
<maxRunningJobs>25</maxRunningJobs>
</pool>
<pool name="default">
<minMaps>120</minMaps>
<minReduces>40</minReduces>
<maxRunningJobs>40</maxRunningJobs>
</pool>
<fairSharePreemptionTimeout>600</fairSharePreemptionTimeout>
</allocations>
4. After this, restart job tracker.
And your queues should be in place. You can then submit job to right queue using
-Dmapred.job.queue.name=sqoop or default
And check the job going to actual queue at the link:
http://<job tracker hostname>:50030/scheduler
Look at the property mapreduce.job.acl-view-job in mapred-site. Means anyone can view the jobs.
Try above and let us know any issues you still face.
================================
Anurag Tangri
Never wear your best trousers when you go out to fight for freedom and
truth.- Henrik Ibsen
On Thursday, November 21, 2013 11:21 AM, anurag tangri <an...@yahoo.com> wrote:
Hi Viswanathan,
What steps have you followed to set fair scheduler ?
Thanks,
Anurag Tangri
Never wear your best trousers when you go out to fight for freedom and
truth.- Henrik Ibsen
On Thursday, November 21, 2013 10:25 AM, Viswanathan J <ja...@gmail.com> wrote:
Hi,
I'm running hadoop with 1.2.1, all my jobs are running in single queue (Queue 1) only all the time. But I have configured default, queue 1&2.
Why jobs are not scheduled to all the queues.
Please help. Running like this will be any issue?
Thanks,
Re: Jobtracker fair scheduler
Posted by anurag tangri <an...@yahoo.com>.
Here's step-by-step on how to do this:
================
Here's step by step on how to do this:
1. Add following in mapred-site.xml:
<property>
<name>mapred.fairscheduler.preemption</name>
<final>true</final>
<value>true</value>
</property>
<property>
<name>mapred.jobtracker.taskScheduler</name>
<final>true</final>
<value>org.apache.hadoop.mapred.FairScheduler</value>
</property>
<property>
<name>mapred.acls.enabled</name>
<final>true</final>
<value>true</value>
</property>
<property>
<name>mapred.fairscheduler.allow.undeclared.pools</name>
<final>true</final>
<value>false</value>
</property>
<property>
<name>mapred.fairscheduler.poolnameproperty</name>
<final>true</final>
<value>mapred.job.queue.name</value>
</property>
<property>
<name>mapred.fairscheduler.allocation.file</name>
<final>true</final>
<value>/etc/hadoop/conf/allocations.xml</value>
</property>
<property>
<name>mapred.queue.names</name>
<final>true</final>
<value>sqoop,default</value>
</property>
<property>
<name>mapreduce.job.acl-view-job</name>
<final>true</final>
<value>*</value>
</property>
2. Create mapred-queue-acls.xml in same dir as mapred-site.xml where you define all the queues:
<configuration>
<property>
<name>mapred.queue.sqoop.acl-submit-job</name>
<value>usera, userb</value>
</property>
<property>
<name>mapred.queue.sqoop.acl-administer-jobs</name>
<value>usera, userc</value>
</property>
<! above two properties for every queue defined in mapred-site.xml -->
</configuration>
3. Define fair scheduler allocations file (location specified in maped-site.xml) to use the queues defined and assign resources to them:
<?xml version="1.0"?>
<allocations>
<defaultMinSharePreemptionTimeout>300</defaultMinSharePreemptionTimeout>
<pool name="sqoop">
<minMaps>700</minMaps>
<minReduces>175</minReduces>
<maxRunningJobs>25</maxRunningJobs>
</pool>
<pool name="default">
<minMaps>120</minMaps>
<minReduces>40</minReduces>
<maxRunningJobs>40</maxRunningJobs>
</pool>
<fairSharePreemptionTimeout>600</fairSharePreemptionTimeout>
</allocations>
4. After this, restart job tracker.
And your queues should be in place. You can then submit job to right queue using
-Dmapred.job.queue.name=sqoop or default
And check the job going to actual queue at the link:
http://<job tracker hostname>:50030/scheduler
Look at the property mapreduce.job.acl-view-job in mapred-site. Means anyone can view the jobs.
Try above and let us know any issues you still face.
================================
Anurag Tangri
Never wear your best trousers when you go out to fight for freedom and
truth.- Henrik Ibsen
On Thursday, November 21, 2013 11:21 AM, anurag tangri <an...@yahoo.com> wrote:
Hi Viswanathan,
What steps have you followed to set fair scheduler ?
Thanks,
Anurag Tangri
Never wear your best trousers when you go out to fight for freedom and
truth.- Henrik Ibsen
On Thursday, November 21, 2013 10:25 AM, Viswanathan J <ja...@gmail.com> wrote:
Hi,
I'm running hadoop with 1.2.1, all my jobs are running in single queue (Queue 1) only all the time. But I have configured default, queue 1&2.
Why jobs are not scheduled to all the queues.
Please help. Running like this will be any issue?
Thanks,
Re: Jobtracker fair scheduler
Posted by anurag tangri <an...@yahoo.com>.
Here's step-by-step on how to do this:
================
Here's step by step on how to do this:
1. Add following in mapred-site.xml:
<property>
<name>mapred.fairscheduler.preemption</name>
<final>true</final>
<value>true</value>
</property>
<property>
<name>mapred.jobtracker.taskScheduler</name>
<final>true</final>
<value>org.apache.hadoop.mapred.FairScheduler</value>
</property>
<property>
<name>mapred.acls.enabled</name>
<final>true</final>
<value>true</value>
</property>
<property>
<name>mapred.fairscheduler.allow.undeclared.pools</name>
<final>true</final>
<value>false</value>
</property>
<property>
<name>mapred.fairscheduler.poolnameproperty</name>
<final>true</final>
<value>mapred.job.queue.name</value>
</property>
<property>
<name>mapred.fairscheduler.allocation.file</name>
<final>true</final>
<value>/etc/hadoop/conf/allocations.xml</value>
</property>
<property>
<name>mapred.queue.names</name>
<final>true</final>
<value>sqoop,default</value>
</property>
<property>
<name>mapreduce.job.acl-view-job</name>
<final>true</final>
<value>*</value>
</property>
2. Create mapred-queue-acls.xml in same dir as mapred-site.xml where you define all the queues:
<configuration>
<property>
<name>mapred.queue.sqoop.acl-submit-job</name>
<value>usera, userb</value>
</property>
<property>
<name>mapred.queue.sqoop.acl-administer-jobs</name>
<value>usera, userc</value>
</property>
<! above two properties for every queue defined in mapred-site.xml -->
</configuration>
3. Define fair scheduler allocations file (location specified in maped-site.xml) to use the queues defined and assign resources to them:
<?xml version="1.0"?>
<allocations>
<defaultMinSharePreemptionTimeout>300</defaultMinSharePreemptionTimeout>
<pool name="sqoop">
<minMaps>700</minMaps>
<minReduces>175</minReduces>
<maxRunningJobs>25</maxRunningJobs>
</pool>
<pool name="default">
<minMaps>120</minMaps>
<minReduces>40</minReduces>
<maxRunningJobs>40</maxRunningJobs>
</pool>
<fairSharePreemptionTimeout>600</fairSharePreemptionTimeout>
</allocations>
4. After this, restart job tracker.
And your queues should be in place. You can then submit job to right queue using
-Dmapred.job.queue.name=sqoop or default
And check the job going to actual queue at the link:
http://<job tracker hostname>:50030/scheduler
Look at the property mapreduce.job.acl-view-job in mapred-site. Means anyone can view the jobs.
Try above and let us know any issues you still face.
================================
Anurag Tangri
Never wear your best trousers when you go out to fight for freedom and
truth.- Henrik Ibsen
On Thursday, November 21, 2013 11:21 AM, anurag tangri <an...@yahoo.com> wrote:
Hi Viswanathan,
What steps have you followed to set fair scheduler ?
Thanks,
Anurag Tangri
Never wear your best trousers when you go out to fight for freedom and
truth.- Henrik Ibsen
On Thursday, November 21, 2013 10:25 AM, Viswanathan J <ja...@gmail.com> wrote:
Hi,
I'm running hadoop with 1.2.1, all my jobs are running in single queue (Queue 1) only all the time. But I have configured default, queue 1&2.
Why jobs are not scheduled to all the queues.
Please help. Running like this will be any issue?
Thanks,
Re: Jobtracker fair scheduler
Posted by anurag tangri <an...@yahoo.com>.
Here's step-by-step on how to do this:
================
Here's step by step on how to do this:
1. Add following in mapred-site.xml:
<property>
<name>mapred.fairscheduler.preemption</name>
<final>true</final>
<value>true</value>
</property>
<property>
<name>mapred.jobtracker.taskScheduler</name>
<final>true</final>
<value>org.apache.hadoop.mapred.FairScheduler</value>
</property>
<property>
<name>mapred.acls.enabled</name>
<final>true</final>
<value>true</value>
</property>
<property>
<name>mapred.fairscheduler.allow.undeclared.pools</name>
<final>true</final>
<value>false</value>
</property>
<property>
<name>mapred.fairscheduler.poolnameproperty</name>
<final>true</final>
<value>mapred.job.queue.name</value>
</property>
<property>
<name>mapred.fairscheduler.allocation.file</name>
<final>true</final>
<value>/etc/hadoop/conf/allocations.xml</value>
</property>
<property>
<name>mapred.queue.names</name>
<final>true</final>
<value>sqoop,default</value>
</property>
<property>
<name>mapreduce.job.acl-view-job</name>
<final>true</final>
<value>*</value>
</property>
2. Create mapred-queue-acls.xml in same dir as mapred-site.xml where you define all the queues:
<configuration>
<property>
<name>mapred.queue.sqoop.acl-submit-job</name>
<value>usera, userb</value>
</property>
<property>
<name>mapred.queue.sqoop.acl-administer-jobs</name>
<value>usera, userc</value>
</property>
<! above two properties for every queue defined in mapred-site.xml -->
</configuration>
3. Define fair scheduler allocations file (location specified in maped-site.xml) to use the queues defined and assign resources to them:
<?xml version="1.0"?>
<allocations>
<defaultMinSharePreemptionTimeout>300</defaultMinSharePreemptionTimeout>
<pool name="sqoop">
<minMaps>700</minMaps>
<minReduces>175</minReduces>
<maxRunningJobs>25</maxRunningJobs>
</pool>
<pool name="default">
<minMaps>120</minMaps>
<minReduces>40</minReduces>
<maxRunningJobs>40</maxRunningJobs>
</pool>
<fairSharePreemptionTimeout>600</fairSharePreemptionTimeout>
</allocations>
4. After this, restart job tracker.
And your queues should be in place. You can then submit job to right queue using
-Dmapred.job.queue.name=sqoop or default
And check the job going to actual queue at the link:
http://<job tracker hostname>:50030/scheduler
Look at the property mapreduce.job.acl-view-job in mapred-site. Means anyone can view the jobs.
Try above and let us know any issues you still face.
================================
Anurag Tangri
Never wear your best trousers when you go out to fight for freedom and
truth.- Henrik Ibsen
On Thursday, November 21, 2013 11:21 AM, anurag tangri <an...@yahoo.com> wrote:
Hi Viswanathan,
What steps have you followed to set fair scheduler ?
Thanks,
Anurag Tangri
Never wear your best trousers when you go out to fight for freedom and
truth.- Henrik Ibsen
On Thursday, November 21, 2013 10:25 AM, Viswanathan J <ja...@gmail.com> wrote:
Hi,
I'm running hadoop with 1.2.1, all my jobs are running in single queue (Queue 1) only all the time. But I have configured default, queue 1&2.
Why jobs are not scheduled to all the queues.
Please help. Running like this will be any issue?
Thanks,
Re: Jobtracker fair scheduler
Posted by anurag tangri <an...@yahoo.com>.
Hi Viswanathan,
What steps have you followed to set fair scheduler ?
Thanks,
Anurag Tangri
Never wear your best trousers when you go out to fight for freedom and
truth.- Henrik Ibsen
On Thursday, November 21, 2013 10:25 AM, Viswanathan J <ja...@gmail.com> wrote:
Hi,
I'm running hadoop with 1.2.1, all my jobs are running in single queue (Queue 1) only all the time. But I have configured default, queue 1&2.
Why jobs are not scheduled to all the queues.
Please help. Running like this will be any issue?
Thanks,
Re: Jobtracker fair scheduler
Posted by anurag tangri <an...@yahoo.com>.
Hi Viswanathan,
What steps have you followed to set fair scheduler ?
Thanks,
Anurag Tangri
Never wear your best trousers when you go out to fight for freedom and
truth.- Henrik Ibsen
On Thursday, November 21, 2013 10:25 AM, Viswanathan J <ja...@gmail.com> wrote:
Hi,
I'm running hadoop with 1.2.1, all my jobs are running in single queue (Queue 1) only all the time. But I have configured default, queue 1&2.
Why jobs are not scheduled to all the queues.
Please help. Running like this will be any issue?
Thanks,
Re: Jobtracker fair scheduler
Posted by anurag tangri <an...@yahoo.com>.
Hi Viswanathan,
What steps have you followed to set fair scheduler ?
Thanks,
Anurag Tangri
Never wear your best trousers when you go out to fight for freedom and
truth.- Henrik Ibsen
On Thursday, November 21, 2013 10:25 AM, Viswanathan J <ja...@gmail.com> wrote:
Hi,
I'm running hadoop with 1.2.1, all my jobs are running in single queue (Queue 1) only all the time. But I have configured default, queue 1&2.
Why jobs are not scheduled to all the queues.
Please help. Running like this will be any issue?
Thanks,
Re: Jobtracker fair scheduler
Posted by anurag tangri <an...@yahoo.com>.
Hi Viswanathan,
What steps have you followed to set fair scheduler ?
Thanks,
Anurag Tangri
Never wear your best trousers when you go out to fight for freedom and
truth.- Henrik Ibsen
On Thursday, November 21, 2013 10:25 AM, Viswanathan J <ja...@gmail.com> wrote:
Hi,
I'm running hadoop with 1.2.1, all my jobs are running in single queue (Queue 1) only all the time. But I have configured default, queue 1&2.
Why jobs are not scheduled to all the queues.
Please help. Running like this will be any issue?
Thanks,