回复：Re: Problems with spark.locality.wait

[MaChong <ma...@sina.com>]

Hi Mridul, 
I have tried your method, it works fine for this case. I set locality.process to 30 minutes, locality.node and locality.rack to 3 seconds. I got loading stage's RACK level tasks submitted after 3 seconds wait, and only PROCESS tasks after loading stage.

Hi Kay, 
Our case is exactly same as what you say. I do agree with you that this is an issue need to be fixed. In my opinion, locality.wait is a simple way to handle locality problem, but not good enough. In fact I always disable the timeout in my application to get better performance. 
Every time I set the timeout to lower value, It will run into a loop that: some tasks slow -> locality wait timeout -> run tasks in poor locality -> network and memory overload-> more tasks become slow.
So I think may be a better locality strategy is needed, Such as determined by cache size or actual useable locality. 


Thanks for your guy's help.

--Ma Chong


--------------------------------


----- 原始邮件 -----
发件人：Mridul Muralidharan <mr...@gmail.com>
收件人：Kay Ousterhout <ke...@eecs.berkeley.edu>
抄送人：MaChong <ma...@sina.com>, dev <de...@spark.apache.org>
主题：Re: Problems with spark.locality.wait
日期：2014年11月14日 04点53分