You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by shyla deshpande <de...@gmail.com> on 2016/12/15 00:42:10 UTC
How many Spark streaming applications can be run at a time on a Spark cluster?
How many Spark streaming applications can be run at a time on a Spark
cluster?
Is it better to have 1 spark streaming application to consume all the Kafka
topics or have multiple streaming applications when possible to keep it
simple?
Thanks
Re: How many Spark streaming applications can be run at a time on a
Spark cluster?
Posted by Dirceu Semighini Filho <di...@gmail.com>.
Hi,
You can start multiple spark apps per cluster. You will have one stream
context per app.
Em 24 de dez de 2016 18:22, "shyla deshpande" <de...@gmail.com>
escreveu:
> Hi All,
>
> Thank you for the response.
>
> As per
>
> https://docs.cloud.databricks.com/docs/latest/databricks_
> guide/index.html#07%20Spark%20Streaming/15%20Streaming%20FAQs.html
>
> There can be only one streaming context in a cluster which implies only
> one streaming job.
>
> So, I am still confused. Anyone having more than 1 spark streaming app in
> a cluster running at the same time, please share your experience.
>
> Thanks
>
> On Wed, Dec 14, 2016 at 6:54 PM, Akhilesh Pathodia <
> pathodia.akhilesh@gmail.com> wrote:
>
>> If you have enough cores/resources, run them separately depending on your
>> use case.
>>
>>
>> On Thursday 15 December 2016, Divya Gehlot <di...@gmail.com>
>> wrote:
>>
>>> It depends on the use case ...
>>> Spark always depends on the resource availability .
>>> As long as you have resource to acoomodate ,can run as many spark/spark
>>> streaming application.
>>>
>>>
>>> Thanks,
>>> Divya
>>>
>>> On 15 December 2016 at 08:42, shyla deshpande <de...@gmail.com>
>>> wrote:
>>>
>>>> How many Spark streaming applications can be run at a time on a Spark
>>>> cluster?
>>>>
>>>> Is it better to have 1 spark streaming application to consume all the
>>>> Kafka topics or have multiple streaming applications when possible to keep
>>>> it simple?
>>>>
>>>> Thanks
>>>>
>>>>
>>>
>
Re: How many Spark streaming applications can be run at a time on a
Spark cluster?
Posted by shyla deshpande <de...@gmail.com>.
Hi All,
Thank you for the response.
As per
https://docs.cloud.databricks.com/docs/latest/databricks_guide/index.html#07%20Spark%20Streaming/15%20Streaming%20FAQs.html
There can be only one streaming context in a cluster which implies only one
streaming job.
So, I am still confused. Anyone having more than 1 spark streaming app in a
cluster running at the same time, please share your experience.
Thanks
On Wed, Dec 14, 2016 at 6:54 PM, Akhilesh Pathodia <
pathodia.akhilesh@gmail.com> wrote:
> If you have enough cores/resources, run them separately depending on your
> use case.
>
>
> On Thursday 15 December 2016, Divya Gehlot <di...@gmail.com>
> wrote:
>
>> It depends on the use case ...
>> Spark always depends on the resource availability .
>> As long as you have resource to acoomodate ,can run as many spark/spark
>> streaming application.
>>
>>
>> Thanks,
>> Divya
>>
>> On 15 December 2016 at 08:42, shyla deshpande <de...@gmail.com>
>> wrote:
>>
>>> How many Spark streaming applications can be run at a time on a Spark
>>> cluster?
>>>
>>> Is it better to have 1 spark streaming application to consume all the
>>> Kafka topics or have multiple streaming applications when possible to keep
>>> it simple?
>>>
>>> Thanks
>>>
>>>
>>
Re: How many Spark streaming applications can be run at a time on a
Spark cluster?
Posted by Akhilesh Pathodia <pa...@gmail.com>.
If you have enough cores/resources, run them separately depending on your
use case.
On Thursday 15 December 2016, Divya Gehlot <di...@gmail.com> wrote:
> It depends on the use case ...
> Spark always depends on the resource availability .
> As long as you have resource to acoomodate ,can run as many spark/spark
> streaming application.
>
>
> Thanks,
> Divya
>
> On 15 December 2016 at 08:42, shyla deshpande <deshpandeshyla@gmail.com
> <javascript:_e(%7B%7D,'cvml','deshpandeshyla@gmail.com');>> wrote:
>
>> How many Spark streaming applications can be run at a time on a Spark
>> cluster?
>>
>> Is it better to have 1 spark streaming application to consume all the
>> Kafka topics or have multiple streaming applications when possible to keep
>> it simple?
>>
>> Thanks
>>
>>
>
Re: How many Spark streaming applications can be run at a time on a
Spark cluster?
Posted by Divya Gehlot <di...@gmail.com>.
It depends on the use case ...
Spark always depends on the resource availability .
As long as you have resource to acoomodate ,can run as many spark/spark
streaming application.
Thanks,
Divya
On 15 December 2016 at 08:42, shyla deshpande <de...@gmail.com>
wrote:
> How many Spark streaming applications can be run at a time on a Spark
> cluster?
>
> Is it better to have 1 spark streaming application to consume all the
> Kafka topics or have multiple streaming applications when possible to keep
> it simple?
>
> Thanks
>
>