You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Larry Xiao <xi...@sjtu.edu.cn> on 2014/08/05 08:57:09 UTC
GraphX partitioning and threading details
Hi all,
about GraphX partitioning details and possible optimization.
* Can you tell how are partitions distributed to nodes? And inside
worker, how does partitions get allocated to threads?
o Is it possible to make manual configuration, like partition A =>
node 1, thread 1
* How is memory organized among threads?
o Can we exploit the shared memory to combine mirror cache on same
node into one?
o (our experiment shows that more partitions requires much more
memory)
Larry