You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by William Slacum <ws...@gmail.com> on 2015/08/17 22:48:12 UTC
FairScheduler + single resource policy
Hi all,
I'm using Hadoop 2.6.0 after upgrading from Hadoop 2.2.0. Previously, I
didn't mess with any vcore settings as we weren't doing anything special
with containers and it seemed happy enough to only consider memory as a
contended resource. This is consistent with the behavior described in the
documentation for both releases:
```
minResources: minimum resources the queue is entitled to, in the form "X
mb, Y vcores". For the single-resource fairness policy, the vcores value is
ignored.
```
This is from
http://hadoop.apache.org/docs/r2.6.0/hadoop-yarn/hadoop-yarn-site/FairScheduler.html
Unfortunately, it seems after moving to 2.6.0 the default behavior actually
does consider vcores as a contended resource and I can't seem to figure out
how to tell it not to.
I was looking in FairScheduler.java, saw some refactorings had happened,
and one thing that sticks out is:
```
@Override
public EnumSet<SchedulerResourceTypes> getSchedulingResourceTypes() {
return EnumSet
.of(SchedulerResourceTypes.MEMORY, SchedulerResourceTypes.CPU);
}
```
So, I'm wondering the following:
1- How can I enable the single-resource fairness policy?
2- If this is a bug and/or feature drift, is it possible to trick the fair
scheduler into behaving as if memory is the only contented resource by
setting the vcores insanely high (say to Integer.MAX_VALUE) and configuring
the memory as I had previously done?
Thanks,
Bill
Re: spark
Posted by Jay Vyas <ja...@gmail.com>.
For a start compare sparks word count with mapreduce word count.
Then compare sparksql with hive.
If you get that far for the final exersize, Find out for yourself by running bigpetstore-mapreduce and bigpetstore-spark side by side :). They are two similar applications which generate data sets and process them for etl and product recommendations which we are curating in Apache bigtop.
> On Aug 17, 2015, at 6:33 PM, Publius <t_...@yahoo.com> wrote:
>
> Hello
>
> what is the difference between Hadoop and Spark?
>
> How is Spark better?
>
Re: spark
Posted by Jay Vyas <ja...@gmail.com>.
For a start compare sparks word count with mapreduce word count.
Then compare sparksql with hive.
If you get that far for the final exersize, Find out for yourself by running bigpetstore-mapreduce and bigpetstore-spark side by side :). They are two similar applications which generate data sets and process them for etl and product recommendations which we are curating in Apache bigtop.
> On Aug 17, 2015, at 6:33 PM, Publius <t_...@yahoo.com> wrote:
>
> Hello
>
> what is the difference between Hadoop and Spark?
>
> How is Spark better?
>
Re: spark
Posted by Jay Vyas <ja...@gmail.com>.
For a start compare sparks word count with mapreduce word count.
Then compare sparksql with hive.
If you get that far for the final exersize, Find out for yourself by running bigpetstore-mapreduce and bigpetstore-spark side by side :). They are two similar applications which generate data sets and process them for etl and product recommendations which we are curating in Apache bigtop.
> On Aug 17, 2015, at 6:33 PM, Publius <t_...@yahoo.com> wrote:
>
> Hello
>
> what is the difference between Hadoop and Spark?
>
> How is Spark better?
>
Re: spark
Posted by Jay Vyas <ja...@gmail.com>.
For a start compare sparks word count with mapreduce word count.
Then compare sparksql with hive.
If you get that far for the final exersize, Find out for yourself by running bigpetstore-mapreduce and bigpetstore-spark side by side :). They are two similar applications which generate data sets and process them for etl and product recommendations which we are curating in Apache bigtop.
> On Aug 17, 2015, at 6:33 PM, Publius <t_...@yahoo.com> wrote:
>
> Hello
>
> what is the difference between Hadoop and Spark?
>
> How is Spark better?
>
spark
Posted by Publius <t_...@yahoo.com>.
Hello
what is the difference between Hadoop and Spark?
How is Spark better?
spark
Posted by Publius <t_...@yahoo.com>.
Hello
what is the difference between Hadoop and Spark?
How is Spark better?
spark
Posted by Publius <t_...@yahoo.com>.
Hello
what is the difference between Hadoop and Spark?
How is Spark better?
spark
Posted by Publius <t_...@yahoo.com>.
Hello
what is the difference between Hadoop and Spark?
How is Spark better?