Spark scheduling, data locality

[Zoltán Zvara <zo...@gmail.com>]

I'm trying to understand the task scheduling mechanism of Spark, and I'm
curious about where does locality preferences get evaluated? I'm trying to
determine if locality preferences are fetchable before the task get
serialized. A HintSet would be most appreciated!

Have nice day!

Zvara Zoltán



mail, hangout, skype: zoltan.zvara@gmail.com

mobile, viber: +36203129543

bank: 10918001-00000021-50480008

address: Hungary, 2475 Kápolnásnyék, Kossuth 6/a

elte: HSKSJZ (ZVZOAAI.ELTE)

Re: Spark scheduling, data locality

[Hui WANG <he...@gmail.com>]

Hello Zoltan,

I'm a spark beginner but i think that the locality preferences should be
prepared before the sending of
tasks.
One important element of a RDD is the metadata on the scheme and location
of its partitions. Tasks created in the driver program should be based on
this information.

I'm also interested by a detailed answer of this question. Could someone
please provide a few more hints on it ?

Regards,
Hui

On Thu, Mar 19, 2015 at 11:20 AM, Zoltán Zvara <zo...@gmail.com>
wrote:

> I'm trying to understand the task scheduling mechanism of Spark, and I'm
> curious about where does locality preferences get evaluated? I'm trying to
> determine if locality preferences are fetchable before the task get
> serialized. A HintSet would be most appreciated!
>
> Have nice day!
>
> Zvara Zoltán
>
>
>
> mail, hangout, skype: zoltan.zvara@gmail.com
>
> mobile, viber: +36203129543
>
> bank: 10918001-00000021-50480008
>
> address: Hungary, 2475 Kápolnásnyék, Kossuth 6/a
>
> elte: HSKSJZ (ZVZOAAI.ELTE)
>



-- 
Hui WANG
Tel : +33 (0) 6 71 33 45 39
Blog : http://www.hui-wang.info