You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Grandl Robert <rg...@yahoo.com.INVALID> on 2015/04/27 17:30:03 UTC

data locality in spark

Hi guys,
I am running some SQL queries, but all my tasks are reported as either NODE_LOCAL or PROCESS_LOCAL. 
In case of Hadoop world, the reduce tasks are RACK or NON_RACK LOCAL because they have to aggregate data from multiple hosts. However, in Spark even the aggregation stages are reported as NODE/PROCESS LOCAL.
Do I miss something, or why the reduce-like tasks are still NODE/PROCESS LOCAL ?
Thanks,Robert