VertexPartition and ShippableVertexPartition

[shijiaxin <sh...@gmail.com>]

There is a VertexPartition in the EdgePartition,which is created by
EdgePartitionBuilder.toEdgePartition.
and There is also a ShippableVertexPartition in the VertexRDD.
These two Partitions have a lot of common things like index, data and
Bitset, why is this necessary?



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/VertexPartition-and-ShippableVertexPartition-tp10763.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

Re: VertexPartition and ShippableVertexPartition

[Ankur Dave <an...@gmail.com>]

On Mon, Jul 28, 2014 at 4:29 AM, Larry Xiao <xi...@sjtu.edu.cn> wrote:

> On 7/28/14, 3:41 PM, shijiaxin wrote:
>
>> There is a VertexPartition in the EdgePartition,which is created by
>>
>> EdgePartitionBuilder.toEdgePartition.
>>
>> and There is also a ShippableVertexPartition in the VertexRDD.
>>
>> These two Partitions have a lot of common things like index, data and
>>
>> Bitset, why is this necessary?
>>
>>

There is a VertexPartition in the EdgePartition,which is created by
>
Is the VertexPartition in the EdgePartition, the Mirror Cache part?


Yes, exactly. The primary copy of each vertex is stored in the VertexRDD
using the index, values, and mask data structures, which together form a
hash map. In addition, each partition of the VertexRDD stores the
corresponding partition of the routing table to facilitate joining with the
edges. The ShippableVertexPartition class encapsulates the vertex hash map
along with a RoutingTablePartition.

After joining the vertices with the edges, the edge partitions cache their
adjacent vertices in the mirror cache. They use the VertexPartition for
this, which provides only the hash map functionality and not the routing
table.

Ankur <http://www.ankurdave.com/>

Re: VertexPartition and ShippableVertexPartition

[Ankur Dave <an...@gmail.com>]

On Mon, Jul 28, 2014 at 4:29 AM, Larry Xiao <xi...@sjtu.edu.cn> wrote:

> On 7/28/14, 3:41 PM, shijiaxin wrote:
>
>> There is a VertexPartition in the EdgePartition,which is created by
>>
>> EdgePartitionBuilder.toEdgePartition.
>>
>> and There is also a ShippableVertexPartition in the VertexRDD.
>>
>> These two Partitions have a lot of common things like index, data and
>>
>> Bitset, why is this necessary?
>>
>>

There is a VertexPartition in the EdgePartition,which is created by
>
Is the VertexPartition in the EdgePartition, the Mirror Cache part?


Yes, exactly. The primary copy of each vertex is stored in the VertexRDD
using the index, values, and mask data structures, which together form a
hash map. In addition, each partition of the VertexRDD stores the
corresponding partition of the routing table to facilitate joining with the
edges. The ShippableVertexPartition class encapsulates the vertex hash map
along with a RoutingTablePartition.

After joining the vertices with the edges, the edge partitions cache their
adjacent vertices in the mirror cache. They use the VertexPartition for
this, which provides only the hash map functionality and not the routing
table.

Ankur <http://www.ankurdave.com/>

Re: VertexPartition and ShippableVertexPartition

[Larry Xiao <xi...@sjtu.edu.cn>]

I discussed with shijiaxin and find this graph helpful for understanding.
In a Graph class, the vertices (VertexRDD) corresponds to the left RDD, 
and edges (EdgeRDD) to the right one.

Is the VertexPartition in the EdgePartition, the Mirror Cache part?


On 7/28/14, 3:41 PM, shijiaxin wrote:
> There is a VertexPartition in the EdgePartition,which is created by
> EdgePartitionBuilder.toEdgePartition.
> and There is also a ShippableVertexPartition in the VertexRDD.
> These two Partitions have a lot of common things like index, data and
> Bitset, why is this necessary?
>
>
>
> --
> View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/VertexPartition-and-ShippableVertexPartition-tp10763.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>