You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by Julien Naour <ju...@gmail.com> on 2014/08/01 11:41:05 UTC
Fair Scheduler issue
Hello,
I'm currently using HDP 2.0 so it's Hadoop 2.2.0.
My cluster consist in 4 nodes, 16 coeurs 16 GB RAM 4*3To each.
Recently we passed from 2 users to 8. We need now a more appropriate
Scheduler.
We begin with Capacity Scheduler. There was some issues with the different
queues particularly when using some spark shell that used some resources
for a long time.
So we decide to try Fair Scheduler which seems to be a good solution.
The problem is that FairScheduler doesn't allow all available resources.
It's capped at 73% of the available memory for one jobs 63% for 2 jobs and
45% for 3 jobs. The problem could come from shells that take resources for
a long time.
We tried some configuration like
yarn.scheduler.fair.user-as-default-queue=false
or play with the minimum ressources allocated minResources in
fair-scheduler.xml but it doesn't seems to resolve the issue.
Any advices or good practices to held a good Fair Scheduler?
Regards,
Julien
Re: Fair Scheduler issue
Posted by Julien Naour <ju...@gmail.com>.
We are at 11GB for yarn nodemanager.resource.memory-mb
It seems that the problem is due to the number of CPUs.
Each Spark executor needed too many CPUs in comparaison to available CPUs.
In consequence the Fair Scheduler didn't allow all the available memory
because all CPUs where all-ready used.
Problem solved (or so it seems) by allowing less CPUs by Spark Executors
Thanks,
Julien
2014-08-02 21:13 GMT+02:00 Yehia Elshater <y....@gmail.com>:
> Hi Julien,
>
> Did you try to change yarn.nodemanager.resource.memory-mb to 13 GB for
> example (the other 3 for OS) ?
>
> Thanks
>
>
>
>
> On 1 August 2014 05:41, Julien Naour <ju...@gmail.com> wrote:
>
>> Hello,
>>
>> I'm currently using HDP 2.0 so it's Hadoop 2.2.0.
>> My cluster consist in 4 nodes, 16 coeurs 16 GB RAM 4*3To each.
>>
>> Recently we passed from 2 users to 8. We need now a more appropriate
>> Scheduler.
>> We begin with Capacity Scheduler. There was some issues with the
>> different queues particularly when using some spark shell that used some
>> resources for a long time.
>> So we decide to try Fair Scheduler which seems to be a good solution.
>> The problem is that FairScheduler doesn't allow all available resources.
>> It's capped at 73% of the available memory for one jobs 63% for 2 jobs and
>> 45% for 3 jobs. The problem could come from shells that take resources for
>> a long time.
>>
>> We tried some configuration like
>> yarn.scheduler.fair.user-as-default-queue=false
>> or play with the minimum ressources allocated minResources in
>> fair-scheduler.xml but it doesn't seems to resolve the issue.
>>
>> Any advices or good practices to held a good Fair Scheduler?
>>
>> Regards,
>>
>> Julien
>>
>
>
Re: Fair Scheduler issue
Posted by Julien Naour <ju...@gmail.com>.
We are at 11GB for yarn nodemanager.resource.memory-mb
It seems that the problem is due to the number of CPUs.
Each Spark executor needed too many CPUs in comparaison to available CPUs.
In consequence the Fair Scheduler didn't allow all the available memory
because all CPUs where all-ready used.
Problem solved (or so it seems) by allowing less CPUs by Spark Executors
Thanks,
Julien
2014-08-02 21:13 GMT+02:00 Yehia Elshater <y....@gmail.com>:
> Hi Julien,
>
> Did you try to change yarn.nodemanager.resource.memory-mb to 13 GB for
> example (the other 3 for OS) ?
>
> Thanks
>
>
>
>
> On 1 August 2014 05:41, Julien Naour <ju...@gmail.com> wrote:
>
>> Hello,
>>
>> I'm currently using HDP 2.0 so it's Hadoop 2.2.0.
>> My cluster consist in 4 nodes, 16 coeurs 16 GB RAM 4*3To each.
>>
>> Recently we passed from 2 users to 8. We need now a more appropriate
>> Scheduler.
>> We begin with Capacity Scheduler. There was some issues with the
>> different queues particularly when using some spark shell that used some
>> resources for a long time.
>> So we decide to try Fair Scheduler which seems to be a good solution.
>> The problem is that FairScheduler doesn't allow all available resources.
>> It's capped at 73% of the available memory for one jobs 63% for 2 jobs and
>> 45% for 3 jobs. The problem could come from shells that take resources for
>> a long time.
>>
>> We tried some configuration like
>> yarn.scheduler.fair.user-as-default-queue=false
>> or play with the minimum ressources allocated minResources in
>> fair-scheduler.xml but it doesn't seems to resolve the issue.
>>
>> Any advices or good practices to held a good Fair Scheduler?
>>
>> Regards,
>>
>> Julien
>>
>
>
Re: Fair Scheduler issue
Posted by Julien Naour <ju...@gmail.com>.
We are at 11GB for yarn nodemanager.resource.memory-mb
It seems that the problem is due to the number of CPUs.
Each Spark executor needed too many CPUs in comparaison to available CPUs.
In consequence the Fair Scheduler didn't allow all the available memory
because all CPUs where all-ready used.
Problem solved (or so it seems) by allowing less CPUs by Spark Executors
Thanks,
Julien
2014-08-02 21:13 GMT+02:00 Yehia Elshater <y....@gmail.com>:
> Hi Julien,
>
> Did you try to change yarn.nodemanager.resource.memory-mb to 13 GB for
> example (the other 3 for OS) ?
>
> Thanks
>
>
>
>
> On 1 August 2014 05:41, Julien Naour <ju...@gmail.com> wrote:
>
>> Hello,
>>
>> I'm currently using HDP 2.0 so it's Hadoop 2.2.0.
>> My cluster consist in 4 nodes, 16 coeurs 16 GB RAM 4*3To each.
>>
>> Recently we passed from 2 users to 8. We need now a more appropriate
>> Scheduler.
>> We begin with Capacity Scheduler. There was some issues with the
>> different queues particularly when using some spark shell that used some
>> resources for a long time.
>> So we decide to try Fair Scheduler which seems to be a good solution.
>> The problem is that FairScheduler doesn't allow all available resources.
>> It's capped at 73% of the available memory for one jobs 63% for 2 jobs and
>> 45% for 3 jobs. The problem could come from shells that take resources for
>> a long time.
>>
>> We tried some configuration like
>> yarn.scheduler.fair.user-as-default-queue=false
>> or play with the minimum ressources allocated minResources in
>> fair-scheduler.xml but it doesn't seems to resolve the issue.
>>
>> Any advices or good practices to held a good Fair Scheduler?
>>
>> Regards,
>>
>> Julien
>>
>
>
Re: Fair Scheduler issue
Posted by Julien Naour <ju...@gmail.com>.
We are at 11GB for yarn nodemanager.resource.memory-mb
It seems that the problem is due to the number of CPUs.
Each Spark executor needed too many CPUs in comparaison to available CPUs.
In consequence the Fair Scheduler didn't allow all the available memory
because all CPUs where all-ready used.
Problem solved (or so it seems) by allowing less CPUs by Spark Executors
Thanks,
Julien
2014-08-02 21:13 GMT+02:00 Yehia Elshater <y....@gmail.com>:
> Hi Julien,
>
> Did you try to change yarn.nodemanager.resource.memory-mb to 13 GB for
> example (the other 3 for OS) ?
>
> Thanks
>
>
>
>
> On 1 August 2014 05:41, Julien Naour <ju...@gmail.com> wrote:
>
>> Hello,
>>
>> I'm currently using HDP 2.0 so it's Hadoop 2.2.0.
>> My cluster consist in 4 nodes, 16 coeurs 16 GB RAM 4*3To each.
>>
>> Recently we passed from 2 users to 8. We need now a more appropriate
>> Scheduler.
>> We begin with Capacity Scheduler. There was some issues with the
>> different queues particularly when using some spark shell that used some
>> resources for a long time.
>> So we decide to try Fair Scheduler which seems to be a good solution.
>> The problem is that FairScheduler doesn't allow all available resources.
>> It's capped at 73% of the available memory for one jobs 63% for 2 jobs and
>> 45% for 3 jobs. The problem could come from shells that take resources for
>> a long time.
>>
>> We tried some configuration like
>> yarn.scheduler.fair.user-as-default-queue=false
>> or play with the minimum ressources allocated minResources in
>> fair-scheduler.xml but it doesn't seems to resolve the issue.
>>
>> Any advices or good practices to held a good Fair Scheduler?
>>
>> Regards,
>>
>> Julien
>>
>
>
Re: Fair Scheduler issue
Posted by Yehia Elshater <y....@gmail.com>.
Hi Julien,
Did you try to change yarn.nodemanager.resource.memory-mb to 13 GB for
example (the other 3 for OS) ?
Thanks
On 1 August 2014 05:41, Julien Naour <ju...@gmail.com> wrote:
> Hello,
>
> I'm currently using HDP 2.0 so it's Hadoop 2.2.0.
> My cluster consist in 4 nodes, 16 coeurs 16 GB RAM 4*3To each.
>
> Recently we passed from 2 users to 8. We need now a more appropriate
> Scheduler.
> We begin with Capacity Scheduler. There was some issues with the different
> queues particularly when using some spark shell that used some resources
> for a long time.
> So we decide to try Fair Scheduler which seems to be a good solution.
> The problem is that FairScheduler doesn't allow all available resources.
> It's capped at 73% of the available memory for one jobs 63% for 2 jobs and
> 45% for 3 jobs. The problem could come from shells that take resources for
> a long time.
>
> We tried some configuration like
> yarn.scheduler.fair.user-as-default-queue=false
> or play with the minimum ressources allocated minResources in
> fair-scheduler.xml but it doesn't seems to resolve the issue.
>
> Any advices or good practices to held a good Fair Scheduler?
>
> Regards,
>
> Julien
>
Re: Fair Scheduler issue
Posted by Yehia Elshater <y....@gmail.com>.
Hi Julien,
Did you try to change yarn.nodemanager.resource.memory-mb to 13 GB for
example (the other 3 for OS) ?
Thanks
On 1 August 2014 05:41, Julien Naour <ju...@gmail.com> wrote:
> Hello,
>
> I'm currently using HDP 2.0 so it's Hadoop 2.2.0.
> My cluster consist in 4 nodes, 16 coeurs 16 GB RAM 4*3To each.
>
> Recently we passed from 2 users to 8. We need now a more appropriate
> Scheduler.
> We begin with Capacity Scheduler. There was some issues with the different
> queues particularly when using some spark shell that used some resources
> for a long time.
> So we decide to try Fair Scheduler which seems to be a good solution.
> The problem is that FairScheduler doesn't allow all available resources.
> It's capped at 73% of the available memory for one jobs 63% for 2 jobs and
> 45% for 3 jobs. The problem could come from shells that take resources for
> a long time.
>
> We tried some configuration like
> yarn.scheduler.fair.user-as-default-queue=false
> or play with the minimum ressources allocated minResources in
> fair-scheduler.xml but it doesn't seems to resolve the issue.
>
> Any advices or good practices to held a good Fair Scheduler?
>
> Regards,
>
> Julien
>
Re: Fair Scheduler issue
Posted by Yehia Elshater <y....@gmail.com>.
Hi Julien,
Did you try to change yarn.nodemanager.resource.memory-mb to 13 GB for
example (the other 3 for OS) ?
Thanks
On 1 August 2014 05:41, Julien Naour <ju...@gmail.com> wrote:
> Hello,
>
> I'm currently using HDP 2.0 so it's Hadoop 2.2.0.
> My cluster consist in 4 nodes, 16 coeurs 16 GB RAM 4*3To each.
>
> Recently we passed from 2 users to 8. We need now a more appropriate
> Scheduler.
> We begin with Capacity Scheduler. There was some issues with the different
> queues particularly when using some spark shell that used some resources
> for a long time.
> So we decide to try Fair Scheduler which seems to be a good solution.
> The problem is that FairScheduler doesn't allow all available resources.
> It's capped at 73% of the available memory for one jobs 63% for 2 jobs and
> 45% for 3 jobs. The problem could come from shells that take resources for
> a long time.
>
> We tried some configuration like
> yarn.scheduler.fair.user-as-default-queue=false
> or play with the minimum ressources allocated minResources in
> fair-scheduler.xml but it doesn't seems to resolve the issue.
>
> Any advices or good practices to held a good Fair Scheduler?
>
> Regards,
>
> Julien
>
Re: Fair Scheduler issue
Posted by Yehia Elshater <y....@gmail.com>.
Hi Julien,
Did you try to change yarn.nodemanager.resource.memory-mb to 13 GB for
example (the other 3 for OS) ?
Thanks
On 1 August 2014 05:41, Julien Naour <ju...@gmail.com> wrote:
> Hello,
>
> I'm currently using HDP 2.0 so it's Hadoop 2.2.0.
> My cluster consist in 4 nodes, 16 coeurs 16 GB RAM 4*3To each.
>
> Recently we passed from 2 users to 8. We need now a more appropriate
> Scheduler.
> We begin with Capacity Scheduler. There was some issues with the different
> queues particularly when using some spark shell that used some resources
> for a long time.
> So we decide to try Fair Scheduler which seems to be a good solution.
> The problem is that FairScheduler doesn't allow all available resources.
> It's capped at 73% of the available memory for one jobs 63% for 2 jobs and
> 45% for 3 jobs. The problem could come from shells that take resources for
> a long time.
>
> We tried some configuration like
> yarn.scheduler.fair.user-as-default-queue=false
> or play with the minimum ressources allocated minResources in
> fair-scheduler.xml but it doesn't seems to resolve the issue.
>
> Any advices or good practices to held a good Fair Scheduler?
>
> Regards,
>
> Julien
>