You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Disha Shrivastava <di...@gmail.com> on 2015/12/26 19:08:33 UTC
Akka with Spark
Hi,
I wanted to know how to use Akka framework with Spark starting from basics.
I saw online that Spark uses Akka framework but I am not really sure if I
can define Actors and use it in Spark.
Also, how to integrate Akka with Spark as in how will I know how many Akka
actors are running on each of my worker machines? Can I control that?
Please help. The only useful resource which I could find online was Akka
with Spark Streaming which was also not very clear.
Thanks,
Disha
Re: Akka with Spark
Posted by Ted Yu <yu...@gmail.com>.
Disha:
Please consider these resources:
https://groups.google.com/forum/#!forum/akka-user
https://groups.google.com/forum/#!forum/akka-dev
On Sun, Dec 27, 2015 at 5:15 AM, Dean Wampler <de...@gmail.com> wrote:
> As Reynold said, you can still use Akka with Spark, but now it's more like
> using any third-party library that isn't already a Spark dependency (at
> least once the current Akka dependency is fully removed).
>
> Dean Wampler, Ph.D.
> Author: Programming Scala, 2nd Edition
> <http://shop.oreilly.com/product/0636920033073.do> (O'Reilly)
> Typesafe <http://typesafe.com>
> @deanwampler <http://twitter.com/deanwampler>
> http://polyglotprogramming.com
>
> On Sun, Dec 27, 2015 at 4:06 AM, Disha Shrivastava <di...@gmail.com>
> wrote:
>
>> Hi All,
>>
>> I need an Akka like framework to implement model parallelism in neural
>> networks, an architecture similar to that given in the link
>> http://alexminnaar.com/implementing-the-distbelief-deep-neural-network-training-framework-with-akka.html.
>> I need to divide a big neural network ( which can't fit into the memory of
>> one machine) layer by layer and do message passing across actors which are
>> distributed across different worker machines. I found Akka to be most
>> suitable for the job.
>>
>> Please suggest if it can be done by any other suitable frameworks.
>>
>> Regards,
>> Disha
>>
>> On Sun, Dec 27, 2015 at 1:04 PM, Reynold Xin <rx...@databricks.com> wrote:
>>
>>> We are just removing Spark's dependency on Akka. It has nothing to do
>>> with whether user applications can use Akka or not. As a matter of fact, by
>>> removing the Akka dependency from Spark, it becomes easier for user
>>> applications to use Akka, because there is no more dependency conflict.
>>>
>>> For more information, see
>>> https://issues.apache.org/jira/browse/SPARK-5293
>>>
>>> On Sat, Dec 26, 2015 at 9:31 PM, Soumya Simanta <
>>> soumya.simanta@gmail.com> wrote:
>>>
>>>>
>>>>
>>>> Any rationale for removing Akka from Spark ? Also, what is the
>>>> replacement ?
>>>>
>>>> Thanks
>>>>
>>>> On Dec 27, 2015, at 8:31 AM, Dean Wampler <de...@gmail.com>
>>>> wrote:
>>>>
>>>> Note that Akka is being removed from Spark. Even if it weren't, I would
>>>> consider keeping Akka processes separate from Spark processes, so you can
>>>> monitor, debug, and scale them independently. So consider streaming data
>>>> from Akka to Spark Streaming or go the other way, from Spark to Akka
>>>> Streams.
>>>>
>>>> dean
>>>>
>>>> Dean Wampler, Ph.D.
>>>> Author: Programming Scala, 2nd Edition
>>>> <http://shop.oreilly.com/product/0636920033073.do> (O'Reilly)
>>>> Typesafe <http://typesafe.com>
>>>> @deanwampler <http://twitter.com/deanwampler>
>>>> http://polyglotprogramming.com
>>>>
>>>> On Sat, Dec 26, 2015 at 12:54 PM, Ted Yu <yu...@gmail.com> wrote:
>>>>
>>>>> Do you mind sharing your use case ?
>>>>>
>>>>> It may be possible to use a different approach than Akka.
>>>>>
>>>>> Cheers
>>>>>
>>>>> On Sat, Dec 26, 2015 at 10:08 AM, Disha Shrivastava <
>>>>> dishu.905@gmail.com> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I wanted to know how to use Akka framework with Spark starting from
>>>>>> basics. I saw online that Spark uses Akka framework but I am not really
>>>>>> sure if I can define Actors and use it in Spark.
>>>>>>
>>>>>> Also, how to integrate Akka with Spark as in how will I know how many
>>>>>> Akka actors are running on each of my worker machines? Can I control that?
>>>>>>
>>>>>> Please help. The only useful resource which I could find online was
>>>>>> Akka with Spark Streaming which was also not very clear.
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Disha
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Akka with Spark
Posted by Dean Wampler <de...@gmail.com>.
As Reynold said, you can still use Akka with Spark, but now it's more like
using any third-party library that isn't already a Spark dependency (at
least once the current Akka dependency is fully removed).
Dean Wampler, Ph.D.
Author: Programming Scala, 2nd Edition
<http://shop.oreilly.com/product/0636920033073.do> (O'Reilly)
Typesafe <http://typesafe.com>
@deanwampler <http://twitter.com/deanwampler>
http://polyglotprogramming.com
On Sun, Dec 27, 2015 at 4:06 AM, Disha Shrivastava <di...@gmail.com>
wrote:
> Hi All,
>
> I need an Akka like framework to implement model parallelism in neural
> networks, an architecture similar to that given in the link
> http://alexminnaar.com/implementing-the-distbelief-deep-neural-network-training-framework-with-akka.html.
> I need to divide a big neural network ( which can't fit into the memory of
> one machine) layer by layer and do message passing across actors which are
> distributed across different worker machines. I found Akka to be most
> suitable for the job.
>
> Please suggest if it can be done by any other suitable frameworks.
>
> Regards,
> Disha
>
> On Sun, Dec 27, 2015 at 1:04 PM, Reynold Xin <rx...@databricks.com> wrote:
>
>> We are just removing Spark's dependency on Akka. It has nothing to do
>> with whether user applications can use Akka or not. As a matter of fact, by
>> removing the Akka dependency from Spark, it becomes easier for user
>> applications to use Akka, because there is no more dependency conflict.
>>
>> For more information, see
>> https://issues.apache.org/jira/browse/SPARK-5293
>>
>> On Sat, Dec 26, 2015 at 9:31 PM, Soumya Simanta <soumya.simanta@gmail.com
>> > wrote:
>>
>>>
>>>
>>> Any rationale for removing Akka from Spark ? Also, what is the
>>> replacement ?
>>>
>>> Thanks
>>>
>>> On Dec 27, 2015, at 8:31 AM, Dean Wampler <de...@gmail.com> wrote:
>>>
>>> Note that Akka is being removed from Spark. Even if it weren't, I would
>>> consider keeping Akka processes separate from Spark processes, so you can
>>> monitor, debug, and scale them independently. So consider streaming data
>>> from Akka to Spark Streaming or go the other way, from Spark to Akka
>>> Streams.
>>>
>>> dean
>>>
>>> Dean Wampler, Ph.D.
>>> Author: Programming Scala, 2nd Edition
>>> <http://shop.oreilly.com/product/0636920033073.do> (O'Reilly)
>>> Typesafe <http://typesafe.com>
>>> @deanwampler <http://twitter.com/deanwampler>
>>> http://polyglotprogramming.com
>>>
>>> On Sat, Dec 26, 2015 at 12:54 PM, Ted Yu <yu...@gmail.com> wrote:
>>>
>>>> Do you mind sharing your use case ?
>>>>
>>>> It may be possible to use a different approach than Akka.
>>>>
>>>> Cheers
>>>>
>>>> On Sat, Dec 26, 2015 at 10:08 AM, Disha Shrivastava <
>>>> dishu.905@gmail.com> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I wanted to know how to use Akka framework with Spark starting from
>>>>> basics. I saw online that Spark uses Akka framework but I am not really
>>>>> sure if I can define Actors and use it in Spark.
>>>>>
>>>>> Also, how to integrate Akka with Spark as in how will I know how many
>>>>> Akka actors are running on each of my worker machines? Can I control that?
>>>>>
>>>>> Please help. The only useful resource which I could find online was
>>>>> Akka with Spark Streaming which was also not very clear.
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Disha
>>>>>
>>>>
>>>>
>>>
>>
>
Re: Akka with Spark
Posted by Disha Shrivastava <di...@gmail.com>.
Hi All,
I need an Akka like framework to implement model parallelism in neural
networks, an architecture similar to that given in the link
http://alexminnaar.com/implementing-the-distbelief-deep-neural-network-training-framework-with-akka.html.
I need to divide a big neural network ( which can't fit into the memory of
one machine) layer by layer and do message passing across actors which are
distributed across different worker machines. I found Akka to be most
suitable for the job.
Please suggest if it can be done by any other suitable frameworks.
Regards,
Disha
On Sun, Dec 27, 2015 at 1:04 PM, Reynold Xin <rx...@databricks.com> wrote:
> We are just removing Spark's dependency on Akka. It has nothing to do with
> whether user applications can use Akka or not. As a matter of fact, by
> removing the Akka dependency from Spark, it becomes easier for user
> applications to use Akka, because there is no more dependency conflict.
>
> For more information, see https://issues.apache.org/jira/browse/SPARK-5293
>
> On Sat, Dec 26, 2015 at 9:31 PM, Soumya Simanta <so...@gmail.com>
> wrote:
>
>>
>>
>> Any rationale for removing Akka from Spark ? Also, what is the
>> replacement ?
>>
>> Thanks
>>
>> On Dec 27, 2015, at 8:31 AM, Dean Wampler <de...@gmail.com> wrote:
>>
>> Note that Akka is being removed from Spark. Even if it weren't, I would
>> consider keeping Akka processes separate from Spark processes, so you can
>> monitor, debug, and scale them independently. So consider streaming data
>> from Akka to Spark Streaming or go the other way, from Spark to Akka
>> Streams.
>>
>> dean
>>
>> Dean Wampler, Ph.D.
>> Author: Programming Scala, 2nd Edition
>> <http://shop.oreilly.com/product/0636920033073.do> (O'Reilly)
>> Typesafe <http://typesafe.com>
>> @deanwampler <http://twitter.com/deanwampler>
>> http://polyglotprogramming.com
>>
>> On Sat, Dec 26, 2015 at 12:54 PM, Ted Yu <yu...@gmail.com> wrote:
>>
>>> Do you mind sharing your use case ?
>>>
>>> It may be possible to use a different approach than Akka.
>>>
>>> Cheers
>>>
>>> On Sat, Dec 26, 2015 at 10:08 AM, Disha Shrivastava <dishu.905@gmail.com
>>> > wrote:
>>>
>>>> Hi,
>>>>
>>>> I wanted to know how to use Akka framework with Spark starting from
>>>> basics. I saw online that Spark uses Akka framework but I am not really
>>>> sure if I can define Actors and use it in Spark.
>>>>
>>>> Also, how to integrate Akka with Spark as in how will I know how many
>>>> Akka actors are running on each of my worker machines? Can I control that?
>>>>
>>>> Please help. The only useful resource which I could find online was
>>>> Akka with Spark Streaming which was also not very clear.
>>>>
>>>> Thanks,
>>>>
>>>> Disha
>>>>
>>>
>>>
>>
>
Re: Akka with Spark
Posted by Reynold Xin <rx...@databricks.com>.
We are just removing Spark's dependency on Akka. It has nothing to do with
whether user applications can use Akka or not. As a matter of fact, by
removing the Akka dependency from Spark, it becomes easier for user
applications to use Akka, because there is no more dependency conflict.
For more information, see https://issues.apache.org/jira/browse/SPARK-5293
On Sat, Dec 26, 2015 at 9:31 PM, Soumya Simanta <so...@gmail.com>
wrote:
>
>
> Any rationale for removing Akka from Spark ? Also, what is the replacement
> ?
>
> Thanks
>
> On Dec 27, 2015, at 8:31 AM, Dean Wampler <de...@gmail.com> wrote:
>
> Note that Akka is being removed from Spark. Even if it weren't, I would
> consider keeping Akka processes separate from Spark processes, so you can
> monitor, debug, and scale them independently. So consider streaming data
> from Akka to Spark Streaming or go the other way, from Spark to Akka
> Streams.
>
> dean
>
> Dean Wampler, Ph.D.
> Author: Programming Scala, 2nd Edition
> <http://shop.oreilly.com/product/0636920033073.do> (O'Reilly)
> Typesafe <http://typesafe.com>
> @deanwampler <http://twitter.com/deanwampler>
> http://polyglotprogramming.com
>
> On Sat, Dec 26, 2015 at 12:54 PM, Ted Yu <yu...@gmail.com> wrote:
>
>> Do you mind sharing your use case ?
>>
>> It may be possible to use a different approach than Akka.
>>
>> Cheers
>>
>> On Sat, Dec 26, 2015 at 10:08 AM, Disha Shrivastava <di...@gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> I wanted to know how to use Akka framework with Spark starting from
>>> basics. I saw online that Spark uses Akka framework but I am not really
>>> sure if I can define Actors and use it in Spark.
>>>
>>> Also, how to integrate Akka with Spark as in how will I know how many
>>> Akka actors are running on each of my worker machines? Can I control that?
>>>
>>> Please help. The only useful resource which I could find online was Akka
>>> with Spark Streaming which was also not very clear.
>>>
>>> Thanks,
>>>
>>> Disha
>>>
>>
>>
>
Re: Akka with Spark
Posted by Soumya Simanta <so...@gmail.com>.
Any rationale for removing Akka from Spark ? Also, what is the replacement ?
Thanks
> On Dec 27, 2015, at 8:31 AM, Dean Wampler <de...@gmail.com> wrote:
>
> Note that Akka is being removed from Spark. Even if it weren't, I would consider keeping Akka processes separate from Spark processes, so you can monitor, debug, and scale them independently. So consider streaming data from Akka to Spark Streaming or go the other way, from Spark to Akka Streams.
>
> dean
>
> Dean Wampler, Ph.D.
> Author: Programming Scala, 2nd Edition (O'Reilly)
> Typesafe
> @deanwampler
> http://polyglotprogramming.com
>
>> On Sat, Dec 26, 2015 at 12:54 PM, Ted Yu <yu...@gmail.com> wrote:
>> Do you mind sharing your use case ?
>>
>> It may be possible to use a different approach than Akka.
>>
>> Cheers
>>
>>> On Sat, Dec 26, 2015 at 10:08 AM, Disha Shrivastava <di...@gmail.com> wrote:
>>> Hi,
>>>
>>> I wanted to know how to use Akka framework with Spark starting from basics. I saw online that Spark uses Akka framework but I am not really sure if I can define Actors and use it in Spark.
>>>
>>> Also, how to integrate Akka with Spark as in how will I know how many Akka actors are running on each of my worker machines? Can I control that?
>>>
>>> Please help. The only useful resource which I could find online was Akka with Spark Streaming which was also not very clear.
>>>
>>> Thanks,
>>>
>>> Disha
>
Re: Akka with Spark
Posted by Dean Wampler <de...@gmail.com>.
Note that Akka is being removed from Spark. Even if it weren't, I would
consider keeping Akka processes separate from Spark processes, so you can
monitor, debug, and scale them independently. So consider streaming data
from Akka to Spark Streaming or go the other way, from Spark to Akka
Streams.
dean
Dean Wampler, Ph.D.
Author: Programming Scala, 2nd Edition
<http://shop.oreilly.com/product/0636920033073.do> (O'Reilly)
Typesafe <http://typesafe.com>
@deanwampler <http://twitter.com/deanwampler>
http://polyglotprogramming.com
On Sat, Dec 26, 2015 at 12:54 PM, Ted Yu <yu...@gmail.com> wrote:
> Do you mind sharing your use case ?
>
> It may be possible to use a different approach than Akka.
>
> Cheers
>
> On Sat, Dec 26, 2015 at 10:08 AM, Disha Shrivastava <di...@gmail.com>
> wrote:
>
>> Hi,
>>
>> I wanted to know how to use Akka framework with Spark starting from
>> basics. I saw online that Spark uses Akka framework but I am not really
>> sure if I can define Actors and use it in Spark.
>>
>> Also, how to integrate Akka with Spark as in how will I know how many
>> Akka actors are running on each of my worker machines? Can I control that?
>>
>> Please help. The only useful resource which I could find online was Akka
>> with Spark Streaming which was also not very clear.
>>
>> Thanks,
>>
>> Disha
>>
>
>
Re: Akka with Spark
Posted by Ted Yu <yu...@gmail.com>.
Do you mind sharing your use case ?
It may be possible to use a different approach than Akka.
Cheers
On Sat, Dec 26, 2015 at 10:08 AM, Disha Shrivastava <di...@gmail.com>
wrote:
> Hi,
>
> I wanted to know how to use Akka framework with Spark starting from
> basics. I saw online that Spark uses Akka framework but I am not really
> sure if I can define Actors and use it in Spark.
>
> Also, how to integrate Akka with Spark as in how will I know how many Akka
> actors are running on each of my worker machines? Can I control that?
>
> Please help. The only useful resource which I could find online was Akka
> with Spark Streaming which was also not very clear.
>
> Thanks,
>
> Disha
>