You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Arun Kumar <to...@gmail.com> on 2014/07/17 12:54:40 UTC
GraphX Pragel implementation
Hi
I am trying to implement belief propagation algorithm in GraphX using the
pragel API.
*def* pregel[A]
(initialMsg*:* A,
maxIter*:* Int = *Int*.*MaxValue*,
activeDir*:* EdgeDirection = *EdgeDirection*.*Out*)
(vprog*:* (VertexId, VD, A) *=>* *VD*,
sendMsg*:* EdgeTriplet[VD, ED] *=>* *Iterator*[(VertexId, A)],
mergeMsg*:* (A, A) *=>* A)
In this can we create messages in vprog function(From in coming messages)
and send them using sendMsg ?
Regards
Arun
Re: GraphX Pragel implementation
Posted by Arun Kumar <to...@gmail.com>.
Thanks
On Fri, Jul 18, 2014 at 12:22 AM, Ankur Dave <an...@gmail.com> wrote:
> If your sendMsg function needs to know the incoming messages as well as
> the vertex value, you could define VD to be a tuple of the vertex value and
> the last received message. The vprog function would then store the incoming
> messages into the tuple, allowing sendMsg to access them.
>
> For example, if the vertex value was a String and the message type was an
> Int, you could call Pregel as follows:
>
> val graph: Graph[String, _] = ...
>
> graph.mapVertices((id, attr) => (attr, 0)).pregel(0)(
> (id, attr: (String, Int), msg: Int) => (attr._1, msg),
> edge => Iterator(...), // can use edge.srcAttr._2 and edge.dstAttr._2 to access the messages (a: Int, b: Int) => a + b)
>
>
> Ankur <http://www.ankurdave.com/>
>
Re: GraphX Pragel implementation
Posted by Ankur Dave <an...@gmail.com>.
If your sendMsg function needs to know the incoming messages as well as the
vertex value, you could define VD to be a tuple of the vertex value and the
last received message. The vprog function would then store the incoming
messages into the tuple, allowing sendMsg to access them.
For example, if the vertex value was a String and the message type was an
Int, you could call Pregel as follows:
val graph: Graph[String, _] = ...
graph.mapVertices((id, attr) => (attr, 0)).pregel(0)(
(id, attr: (String, Int), msg: Int) => (attr._1, msg),
edge => Iterator(...), // can use edge.srcAttr._2 and
edge.dstAttr._2 to access the messages (a: Int, b: Int) => a + b)
Ankur <http://www.ankurdave.com/>
Re: GraphX Pragel implementation
Posted by Ankur Dave <an...@gmail.com>.
On Wed, Jul 30, 2014 at 04:55 PM, Arun Kumar <to...@gmail.com> wrote:
> For my implementation to work the vprog function which is responsible for
> handling in coming messages and the sendMsg function should be aware of
> which super step they are in.
> Is it possible to pass super step information in this methods?
Sorry about the delay on this. Here's a patch to pass the iteration number to vprog and sendMsg:
https://github.com/ankurdave/spark/commit/pregel-iteration-number
You can check this out (`git clone https://github.com/ankurdave/spark -b pregel-iteration-number`) and build it. The iteration number is passed to vprog and sendMsg as the first argument (the Int).
Ankur
Re: GraphX Pragel implementation
Posted by Arun Kumar <to...@gmail.com>.
Hello Ankur,
For my implementation to work the vprog function which is responsible for
handling in coming messages and the sendMsg function should be aware of
which super step they are in.
Is it possible to pass super step information in this methods?
Can u through some light on how to approach this.
Regards
Arun
On Fri, Jul 25, 2014 at 3:04 PM, Arun Kumar <to...@gmail.com> wrote:
> Hi
>
> Thanks for the quick response.I am new to scala and some help will be
> required
>
> Regards
> -Arun
>
>
> On Fri, Jul 25, 2014 at 10:37 AM, Ankur Dave <an...@gmail.com> wrote:
>
>> On Thu, Jul 24, 2014 at 9:52 AM, Arun Kumar <to...@gmail.com> wrote:
>>
>>> While using pregel API for Iterations how to figure out which super
>>> step the iteration currently in.
>>
>>
>> The Pregel API doesn't currently expose this, but it's very
>> straightforward to modify Pregel.scala
>> <https://github.com/apache/spark/blob/master/graphx/src/main/scala/org/apache/spark/graphx/Pregel.scala#L112>
>> to do so. Let me know if you'd like help doing this.
>>
>> Ankur <http://www.ankurdave.com/>
>>
>
>
Re: GraphX Pragel implementation
Posted by Arun Kumar <to...@gmail.com>.
Hi
Thanks for the quick response.I am new to scala and some help will be
required
Regards
-Arun
On Fri, Jul 25, 2014 at 10:37 AM, Ankur Dave <an...@gmail.com> wrote:
> On Thu, Jul 24, 2014 at 9:52 AM, Arun Kumar <to...@gmail.com> wrote:
>
>> While using pregel API for Iterations how to figure out which super step
>> the iteration currently in.
>
>
> The Pregel API doesn't currently expose this, but it's very
> straightforward to modify Pregel.scala
> <https://github.com/apache/spark/blob/master/graphx/src/main/scala/org/apache/spark/graphx/Pregel.scala#L112>
> to do so. Let me know if you'd like help doing this.
>
> Ankur <http://www.ankurdave.com/>
>
Re: GraphX Pragel implementation
Posted by Ankur Dave <an...@gmail.com>.
On Thu, Jul 24, 2014 at 9:52 AM, Arun Kumar <to...@gmail.com> wrote:
> While using pregel API for Iterations how to figure out which super step
> the iteration currently in.
The Pregel API doesn't currently expose this, but it's very straightforward
to modify Pregel.scala
<https://github.com/apache/spark/blob/master/graphx/src/main/scala/org/apache/spark/graphx/Pregel.scala#L112>
to do so. Let me know if you'd like help doing this.
Ankur <http://www.ankurdave.com/>
Re: GraphX Pragel implementation
Posted by Arun Kumar <to...@gmail.com>.
Hi
While using pregel API for Iterations how to figure out which super step
the iteration currently in.
Regards
Arun
On Thu, Jul 17, 2014 at 4:24 PM, Arun Kumar <to...@gmail.com> wrote:
> Hi
>
>
>
> I am trying to implement belief propagation algorithm in GraphX using the
> pragel API.
>
> *def* pregel[A]
>
> (initialMsg*:* A,
>
> maxIter*:* Int = *Int*.*MaxValue*,
>
> activeDir*:* EdgeDirection = *EdgeDirection*.*Out*)
>
> (vprog*:* (VertexId, VD, A) *=>* *VD*,
>
> sendMsg*:* EdgeTriplet[VD, ED] *=>* *Iterator*[(VertexId, A)],
>
> mergeMsg*:* (A, A) *=>* A)
>
> In this can we create messages in vprog function(From in coming messages)
> and send them using sendMsg ?
>
>
>
> Regards
> Arun
>