You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Abhishek Anand <ab...@gmail.com> on 2016/02/17 07:57:27 UTC
Spark Streaming with Kafka Use Case
I have a spark streaming application running in production. I am trying to
find a solution for a particular use case when my application has a
downtime of say 5 hours and is restarted. Now, when I start my streaming
application after 5 hours there would be considerable amount of data then
in the Kafka and my cluster would be unable to repartition and process that.
Is there any workaround so that when my streaming application starts it
starts taking data for 1-2 hours, process it , then take the data for next
1 hour process it. Now when its done processing of previous 5 hours data
which missed, normal streaming should start with the given slide interval.
Please suggest any ideas and feasibility of this.
Thanks !!
Abhi
Re: Spark Streaming with Kafka Use Case
Posted by Cody Koeninger <co...@koeninger.org>.
If by smaller block interval you mean the value in seconds passed to the
streaming context constructor, no. You'll still get everything from the
starting offset until now in the first batch.
On Thu, Feb 18, 2016 at 10:02 AM, praveen S <my...@gmail.com> wrote:
> Sorry.. Rephrasing :
> Can this issue be resolved by having a smaller block interval?
>
> Regards,
> Praveen
> On 18 Feb 2016 21:30, "praveen S" <my...@gmail.com> wrote:
>
>> Can having a smaller block interval only resolve this?
>>
>> Regards,
>> Praveen
>> On 18 Feb 2016 21:13, "Cody Koeninger" <co...@koeninger.org> wrote:
>>
>>> Backpressure won't help you with the first batch, you'd need spark.streaming.kafka.maxRatePerPartition
>>> for that
>>>
>>> On Thu, Feb 18, 2016 at 9:40 AM, praveen S <my...@gmail.com> wrote:
>>>
>>>> Have a look at
>>>>
>>>> spark.streaming.backpressure.enabled
>>>> Property
>>>>
>>>> Regards,
>>>> Praveen
>>>> On 18 Feb 2016 00:13, "Abhishek Anand" <ab...@gmail.com> wrote:
>>>>
>>>>> I have a spark streaming application running in production. I am
>>>>> trying to find a solution for a particular use case when my application has
>>>>> a downtime of say 5 hours and is restarted. Now, when I start my streaming
>>>>> application after 5 hours there would be considerable amount of data then
>>>>> in the Kafka and my cluster would be unable to repartition and process that.
>>>>>
>>>>> Is there any workaround so that when my streaming application starts
>>>>> it starts taking data for 1-2 hours, process it , then take the data for
>>>>> next 1 hour process it. Now when its done processing of previous 5 hours
>>>>> data which missed, normal streaming should start with the given slide
>>>>> interval.
>>>>>
>>>>> Please suggest any ideas and feasibility of this.
>>>>>
>>>>>
>>>>> Thanks !!
>>>>> Abhi
>>>>>
>>>>
>>>
Re: Spark Streaming with Kafka Use Case
Posted by praveen S <my...@gmail.com>.
Sorry.. Rephrasing :
Can this issue be resolved by having a smaller block interval?
Regards,
Praveen
On 18 Feb 2016 21:30, "praveen S" <my...@gmail.com> wrote:
> Can having a smaller block interval only resolve this?
>
> Regards,
> Praveen
> On 18 Feb 2016 21:13, "Cody Koeninger" <co...@koeninger.org> wrote:
>
>> Backpressure won't help you with the first batch, you'd need spark.streaming.kafka.maxRatePerPartition
>> for that
>>
>> On Thu, Feb 18, 2016 at 9:40 AM, praveen S <my...@gmail.com> wrote:
>>
>>> Have a look at
>>>
>>> spark.streaming.backpressure.enabled
>>> Property
>>>
>>> Regards,
>>> Praveen
>>> On 18 Feb 2016 00:13, "Abhishek Anand" <ab...@gmail.com> wrote:
>>>
>>>> I have a spark streaming application running in production. I am trying
>>>> to find a solution for a particular use case when my application has a
>>>> downtime of say 5 hours and is restarted. Now, when I start my streaming
>>>> application after 5 hours there would be considerable amount of data then
>>>> in the Kafka and my cluster would be unable to repartition and process that.
>>>>
>>>> Is there any workaround so that when my streaming application starts it
>>>> starts taking data for 1-2 hours, process it , then take the data for next
>>>> 1 hour process it. Now when its done processing of previous 5 hours data
>>>> which missed, normal streaming should start with the given slide interval.
>>>>
>>>> Please suggest any ideas and feasibility of this.
>>>>
>>>>
>>>> Thanks !!
>>>> Abhi
>>>>
>>>
>>
Re: Spark Streaming with Kafka Use Case
Posted by praveen S <my...@gmail.com>.
Can having a smaller block interval only resolve this?
Regards,
Praveen
On 18 Feb 2016 21:13, "Cody Koeninger" <co...@koeninger.org> wrote:
> Backpressure won't help you with the first batch, you'd need spark.streaming.kafka.maxRatePerPartition
> for that
>
> On Thu, Feb 18, 2016 at 9:40 AM, praveen S <my...@gmail.com> wrote:
>
>> Have a look at
>>
>> spark.streaming.backpressure.enabled
>> Property
>>
>> Regards,
>> Praveen
>> On 18 Feb 2016 00:13, "Abhishek Anand" <ab...@gmail.com> wrote:
>>
>>> I have a spark streaming application running in production. I am trying
>>> to find a solution for a particular use case when my application has a
>>> downtime of say 5 hours and is restarted. Now, when I start my streaming
>>> application after 5 hours there would be considerable amount of data then
>>> in the Kafka and my cluster would be unable to repartition and process that.
>>>
>>> Is there any workaround so that when my streaming application starts it
>>> starts taking data for 1-2 hours, process it , then take the data for next
>>> 1 hour process it. Now when its done processing of previous 5 hours data
>>> which missed, normal streaming should start with the given slide interval.
>>>
>>> Please suggest any ideas and feasibility of this.
>>>
>>>
>>> Thanks !!
>>> Abhi
>>>
>>
>
Re: Spark Streaming with Kafka Use Case
Posted by Cody Koeninger <co...@koeninger.org>.
Backpressure won't help you with the first batch, you'd need
spark.streaming.kafka.maxRatePerPartition
for that
On Thu, Feb 18, 2016 at 9:40 AM, praveen S <my...@gmail.com> wrote:
> Have a look at
>
> spark.streaming.backpressure.enabled
> Property
>
> Regards,
> Praveen
> On 18 Feb 2016 00:13, "Abhishek Anand" <ab...@gmail.com> wrote:
>
>> I have a spark streaming application running in production. I am trying
>> to find a solution for a particular use case when my application has a
>> downtime of say 5 hours and is restarted. Now, when I start my streaming
>> application after 5 hours there would be considerable amount of data then
>> in the Kafka and my cluster would be unable to repartition and process that.
>>
>> Is there any workaround so that when my streaming application starts it
>> starts taking data for 1-2 hours, process it , then take the data for next
>> 1 hour process it. Now when its done processing of previous 5 hours data
>> which missed, normal streaming should start with the given slide interval.
>>
>> Please suggest any ideas and feasibility of this.
>>
>>
>> Thanks !!
>> Abhi
>>
>
Re: Spark Streaming with Kafka Use Case
Posted by praveen S <my...@gmail.com>.
Have a look at
spark.streaming.backpressure.enabled
Property
Regards,
Praveen
On 18 Feb 2016 00:13, "Abhishek Anand" <ab...@gmail.com> wrote:
> I have a spark streaming application running in production. I am trying to
> find a solution for a particular use case when my application has a
> downtime of say 5 hours and is restarted. Now, when I start my streaming
> application after 5 hours there would be considerable amount of data then
> in the Kafka and my cluster would be unable to repartition and process that.
>
> Is there any workaround so that when my streaming application starts it
> starts taking data for 1-2 hours, process it , then take the data for next
> 1 hour process it. Now when its done processing of previous 5 hours data
> which missed, normal streaming should start with the given slide interval.
>
> Please suggest any ideas and feasibility of this.
>
>
> Thanks !!
> Abhi
>
Re: Spark Streaming with Kafka Use Case
Posted by Cody Koeninger <co...@koeninger.org>.
Just use a kafka rdd in a batch job or two, then start your streaming job.
On Wed, Feb 17, 2016 at 12:57 AM, Abhishek Anand <ab...@gmail.com>
wrote:
> I have a spark streaming application running in production. I am trying to
> find a solution for a particular use case when my application has a
> downtime of say 5 hours and is restarted. Now, when I start my streaming
> application after 5 hours there would be considerable amount of data then
> in the Kafka and my cluster would be unable to repartition and process that.
>
> Is there any workaround so that when my streaming application starts it
> starts taking data for 1-2 hours, process it , then take the data for next
> 1 hour process it. Now when its done processing of previous 5 hours data
> which missed, normal streaming should start with the given slide interval.
>
> Please suggest any ideas and feasibility of this.
>
>
> Thanks !!
> Abhi
>