You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Soumitra Johri <so...@gmail.com> on 2016/04/18 21:04:54 UTC
inter spark application communication
Hi,
I have two applications : App1 and App2.
On a single cluster I have to spawn 5 instances os App1 and 1 instance of
App2.
What would be the best way to send data from the 5 App1 instances to the
single App2 instance ?
Right now I am using Kafka to send data from one spark application to the
spark application but the setup doesn't seem right and I hope there is a
better way to do this.
Warm Regards
Soumitra
Re: inter spark application communication
Posted by Michael Segel <ms...@hotmail.com>.
Hi,
Putting this back in User email list…
Ok,
So its not inter-job communication, but taking the output of Job 1 and using it as input to job 2.
(Chaining of jobs.)
So you don’t need anything fancy…
> On Apr 18, 2016, at 1:47 PM, Soumitra Johri <so...@gmail.com> wrote:
>
> Yes , I want to chain spark applications.
> On Mon, Apr 18, 2016 at 4:46 PM Michael Segel <msegel_hadoop@hotmail.com <ma...@hotmail.com>> wrote:
> Yes, but I’m confused. Are you chaining your spark jobs? So you run job one and its output is the input to job 2?
>
>> On Apr 18, 2016, at 1:21 PM, Soumitra Johri <soumitra.siddharth@gmail.com <ma...@gmail.com>> wrote:
>>
>> By Akka you mean the Akka Actors ?
>>
>> I am sending basically sending counts which are further aggregated on the second application.
>>
>>
>> Warm Regards
>> Soumitra
>>
>> On Mon, Apr 18, 2016 at 3:54 PM, Michael Segel <msegel_hadoop@hotmail.com <ma...@hotmail.com>> wrote:
>> have you thought about Akka?
>>
>> What are you trying to send? Why do you want them to talk to one another?
>>
>> > On Apr 18, 2016, at 12:04 PM, Soumitra Johri <soumitra.siddharth@gmail.com <ma...@gmail.com>> wrote:
>> >
>> > Hi,
>> >
>> > I have two applications : App1 and App2.
>> > On a single cluster I have to spawn 5 instances os App1 and 1 instance of App2.
>> >
>> > What would be the best way to send data from the 5 App1 instances to the single App2 instance ?
>> >
>> > Right now I am using Kafka to send data from one spark application to the spark application but the setup doesn't seem right and I hope there is a better way to do this.
>> >
>> > Warm Regards
>> > Soumitra
>>
>>
>
Re: inter spark application communication
Posted by Michael Segel <ms...@hotmail.com>.
have you thought about Akka?
What are you trying to send? Why do you want them to talk to one another?
> On Apr 18, 2016, at 12:04 PM, Soumitra Johri <so...@gmail.com> wrote:
>
> Hi,
>
> I have two applications : App1 and App2.
> On a single cluster I have to spawn 5 instances os App1 and 1 instance of App2.
>
> What would be the best way to send data from the 5 App1 instances to the single App2 instance ?
>
> Right now I am using Kafka to send data from one spark application to the spark application but the setup doesn't seem right and I hope there is a better way to do this.
>
> Warm Regards
> Soumitra
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: inter spark application communication
Posted by Michael Segel <ms...@hotmail.com>.
have you thought about Akka?
What are you trying to send? Why do you want them to talk to one another?
> On Apr 18, 2016, at 12:04 PM, Soumitra Johri <so...@gmail.com> wrote:
>
> Hi,
>
> I have two applications : App1 and App2.
> On a single cluster I have to spawn 5 instances os App1 and 1 instance of App2.
>
> What would be the best way to send data from the 5 App1 instances to the single App2 instance ?
>
> Right now I am using Kafka to send data from one spark application to the spark application but the setup doesn't seem right and I hope there is a better way to do this.
>
> Warm Regards
> Soumitra
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org