You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Larry Xiao <xi...@sjtu.edu.cn> on 2014/08/05 08:57:09 UTC

GraphX partitioning and threading details

Hi all,

about GraphX partitioning details and possible optimization.

  * Can you tell how are partitions distributed to nodes? And inside
    worker, how does partitions get allocated to threads?
      o Is it possible to make manual configuration, like partition A =>
        node 1, thread 1
  * How is memory organized among threads?
      o Can we exploit the shared memory to combine mirror cache on same
        node into one?
      o (our experiment shows that more partitions requires much more
        memory)

Larry