You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by rcharaya <ri...@gmail.com> on 2015/04/13 23:09:11 UTC
Rack locality
I want to use Rack locality feature of Apache Spark in my application.
Is YARN the only resource manager which supports it as of now?
After going through the source code, I found that default implementation of
getRackForHost() method returns NONE in TaskSchedulerImpl which (I suppose)
would be used by standalone mode.
On the other hand, it is overriden in YarnScheduler.scala to fetch the rack
information by invoking RackResolver api of hadoop which would be used when
its run on YARN.
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Rack-locality-tp22483.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Rack locality
Posted by Sandy Ryza <sa...@cloudera.com>.
Hi Riya,
As far as I know, that is correct, unless Mesos fine-grained mode handles
this in some mysterious way.
-Sandy
On Mon, Apr 13, 2015 at 2:09 PM, rcharaya <ri...@gmail.com> wrote:
> I want to use Rack locality feature of Apache Spark in my application.
>
> Is YARN the only resource manager which supports it as of now?
>
> After going through the source code, I found that default implementation of
> getRackForHost() method returns NONE in TaskSchedulerImpl which (I suppose)
> would be used by standalone mode.
>
> On the other hand, it is overriden in YarnScheduler.scala to fetch the rack
> information by invoking RackResolver api of hadoop which would be used when
> its run on YARN.
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Rack-locality-tp22483.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>