How does GraphX stores the routing table?

[mas <ma...@gmail.com>]

Hi,

How does GraphX stores the routing table? Is it stored on the master node or
chunks of the routing table is send to each partition that maintains the
record of vertices and edges at that node? 
If only customized edge partitioning is performed will the corresponding
vertices be sent to same partition or not ?

Thanks,



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/How-does-GraphX-stores-the-routing-table-tp22595.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org

Re: How does GraphX stores the routing table?

[MUHAMMAD AAMIR <ma...@gmail.com>]

Hi Ankur,
Thanks for the answer. However i still have following queries.

On Wed, Apr 22, 2015 at 8:39 AM, Ankur Dave <an...@gmail.com> wrote:

> On Tue, Apr 21, 2015 at 10:39 AM, mas <ma...@gmail.com> wrote:
>
>> How does GraphX stores the routing table? Is it stored on the master node
>> or
>> chunks of the routing table is send to each partition that maintains the
>> record of vertices and edges at that node?
>>
>
> The latter: the routing table is stored alongside the vertices, and for
> each vertex it stores the set of edge partitions that reference that
> vertex.
>
>
*Then how the master node tracks that where(in which partition) a
particular vertex and edge is?*


*Further, does it mean that to fetch a particular edge we first have to
find its source or destination vertex  ?*



> If only customized edge partitioning is performed will the corresponding
>> vertices be sent to same partition or not ?
>
>
> If I understand correctly, you're asking whether it's possible to colocate
> the vertices with the edges so they don't have to move during replication.
> It's possible to do this in some cases by partitioning each edge based on a
> hash partitioner of its source or destination vertex. GraphX will still do
> replication using a shuffle, but most of the shuffle files should be local
> in this case.
>
> I tried this a while ago but didn't find a very big improvement for
> PageRank. Ultimately a more general solution would be to unify the vertex
> and edge RDDs by designating one replica for each vertex as the master.
> This would also reduce the storage cost by a factor of (average degree -
> 1)/(average degree).
>
>
*What do you exactly mean here by "desingating one replica for each vertex
as the master" ? How can we perform this ?*


> Ankur <http://www.ankurdave.com/>
>



-- 
Regards,
Muhammad Aamir


*CONFIDENTIALITY:This email is intended solely for the person(s) named and
may be confidential and/or privileged.If you are not the intended
recipient,please delete it,notify me and do not copy,use,or disclose its
content.*

Re: How does GraphX stores the routing table?

[Ankur Dave <an...@gmail.com>]

On Tue, Apr 21, 2015 at 10:39 AM, mas <ma...@gmail.com> wrote:

> How does GraphX stores the routing table? Is it stored on the master node
> or
> chunks of the routing table is send to each partition that maintains the
> record of vertices and edges at that node?
>

The latter: the routing table is stored alongside the vertices, and for
each vertex it stores the set of edge partitions that reference that
vertex.

If only customized edge partitioning is performed will the corresponding
> vertices be sent to same partition or not ?


If I understand correctly, you're asking whether it's possible to colocate
the vertices with the edges so they don't have to move during replication.
It's possible to do this in some cases by partitioning each edge based on a
hash partitioner of its source or destination vertex. GraphX will still do
replication using a shuffle, but most of the shuffle files should be local
in this case.

I tried this a while ago but didn't find a very big improvement for
PageRank. Ultimately a more general solution would be to unify the vertex
and edge RDDs by designating one replica for each vertex as the master.
This would also reduce the storage cost by a factor of (average degree -
1)/(average degree).

Ankur <http://www.ankurdave.com/>