You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by prateek arora <pr...@gmail.com> on 2015/12/03 19:19:00 UTC
how to spark streaming application start working on next batch
before completing on previous batch .
Hi
I am using spark streaming with Kafka. spark version is 1.5.0 and batch
interval is 1 sec.
In my scenario , algorithm take 7-10 sec to process 1 batch period data. so
after completing previous batch
, spark streaming start processing on next batch.
i want that my spark streaming application start working on next batch
before completing on previous batch . means batches will execute in
parallel.
please help me to solve this problem.
Regards
Prateek
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/how-to-spark-streaming-application-start-working-on-next-batch-before-completing-on-previous-batch-tp25559.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: how to spark streaming application start working on next batch
before completing on previous batch .
Posted by Mukesh Jha <me...@gmail.com>.
Are the issues related to wal based KafkaReliableReceivers or with any
receiver in general.
Any insights will be helpful.
On 16 Dec 2015 05:44, "Tathagata Das" <td...@databricks.com> wrote:
> Just to be clear. spark.treaming.concurrentJobs is NOT officially
> supported. There are issues with fault-tolerance and data loss if that is
> set to more than 1.
>
>
>
> On Tue, Dec 15, 2015 at 9:19 AM, Mukesh Jha <me...@gmail.com>
> wrote:
>
>> Try setting *spark*.streaming.*concurrent*. *jobs* to number of
>> concurrent jobs you want to run.
>> On 15 Dec 2015 17:35, "ikmal" <ik...@gmail.com> wrote:
>>
>>> The best practice is to set batch interval lesser than processing time.
>>> I'm
>>> sure your application is suffering from constantly increasing of
>>> scheduling
>>> delay.
>>>
>>>
>>>
>>> --
>>> View this message in context:
>>> http://apache-spark-user-list.1001560.n3.nabble.com/how-to-spark-streaming-application-start-working-on-next-batch-before-completing-on-previous-batch-tp25559p25707.html
>>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>>> For additional commands, e-mail: user-help@spark.apache.org
>>>
>>>
>
Re: how to spark streaming application start working on next batch
before completing on previous batch .
Posted by Tathagata Das <td...@databricks.com>.
Just to be clear. spark.treaming.concurrentJobs is NOT officially
supported. There are issues with fault-tolerance and data loss if that is
set to more than 1.
On Tue, Dec 15, 2015 at 9:19 AM, Mukesh Jha <me...@gmail.com> wrote:
> Try setting *spark*.streaming.*concurrent*. *jobs* to number of
> concurrent jobs you want to run.
> On 15 Dec 2015 17:35, "ikmal" <ik...@gmail.com> wrote:
>
>> The best practice is to set batch interval lesser than processing time.
>> I'm
>> sure your application is suffering from constantly increasing of
>> scheduling
>> delay.
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-spark-user-list.1001560.n3.nabble.com/how-to-spark-streaming-application-start-working-on-next-batch-before-completing-on-previous-batch-tp25559p25707.html
>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> For additional commands, e-mail: user-help@spark.apache.org
>>
>>
Re: how to spark streaming application start working on next batch
before completing on previous batch .
Posted by Mukesh Jha <me...@gmail.com>.
Try setting *spark*.streaming.*concurrent*. *jobs* to number of concurrent
jobs you want to run.
On 15 Dec 2015 17:35, "ikmal" <ik...@gmail.com> wrote:
> The best practice is to set batch interval lesser than processing time. I'm
> sure your application is suffering from constantly increasing of scheduling
> delay.
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/how-to-spark-streaming-application-start-working-on-next-batch-before-completing-on-previous-batch-tp25559p25707.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>
Re: how to spark streaming application start working on next batch
before completing on previous batch .
Posted by ikmal <ik...@gmail.com>.
The best practice is to set batch interval lesser than processing time. I'm
sure your application is suffering from constantly increasing of scheduling
delay.
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/how-to-spark-streaming-application-start-working-on-next-batch-before-completing-on-previous-batch-tp25559p25707.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: how to spark streaming application start working on next batch
before completing on previous batch .
Posted by manasdebashiskar <po...@gmail.com>.
I don't think it is possible that way.
Spark streaming is a minibatch processing system.
If processing contents of 2 batch is your objective what you can do is
1) keep a cache(or two) that represent the previous batch(s).
2) every new batch replaces the old cache by one time slot back.
3) you process each batch as rdd in parallel.
...Manas
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/how-to-spark-streaming-application-start-working-on-next-batch-before-completing-on-previous-batch-tp25559p25581.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org