Message Passing among workers

[laxmanvemula <la...@gmail.com>]

Hi,

I would like to implement an asynchronous distributed optimization algorithm
where workers communicate among one another. It is similar to belief
propagation where each worker is a vertex in the graph. Can some one let me
know if this is possible using spark?

Thanks,
Laxman



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Message-Passing-among-workers-tp13355.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org

Re: Message Passing among workers

["Evan R. Sparks" <ev...@gmail.com>]

Asynchrony is not supported directly - spark's programming model is
naturally BSP. I have seen cases where people have instantiated actors with
akka on worker nodes to enable message passing, or even used spark's own
ActorSystem to do this. But, I do not recommend this, since you lose a
bunch of benefits of spark - e.g. fault tolerance.

Instead, I would think about whether your algorithm can be cast as a BSP
one, or think about how frequently you really need to synchronize state
among your workers. It may be that having the occasional synchronization
barrier is OK.


On Wed, Sep 3, 2014 at 7:28 AM, laxmanvemula <la...@gmail.com> wrote:

> Hi,
>
> I would like to implement an asynchronous distributed optimization
> algorithm
> where workers communicate among one another. It is similar to belief
> propagation where each worker is a vertex in the graph. Can some one let me
> know if this is possible using spark?
>
> Thanks,
> Laxman
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Message-Passing-among-workers-tp13355.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>