You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Sandeep Giri <sa...@knowbigdata.com> on 2015/10/29 23:08:42 UTC
Maintaining overall cumulative data in Spark Streaming
Dear All,
If a continuous stream of text is coming in and you have to keep publishing
the overall word count so far since 0:00 today, what would you do?
Publishing the results for a window is easy but if we have to keep
aggregating the results, how to go about it?
I have tried to keep an StreamRDD with aggregated count and keep doing a
fullouterjoin but didn't work. Seems like the StreamRDD gets reset.
Kindly help.
Regards,
Sandeep Giri
Re: Maintaining overall cumulative data in Spark Streaming
Posted by Julio Antonio Soto de Vicente <ju...@esbet.es>.
-dev +user
Hi Sandeep,
Perhaps (flat)mapping values and using an accumulator?
> El 29/10/2015, a las 23:08, Sandeep Giri <sa...@knowbigdata.com> escribió:
>
> Dear All,
>
> If a continuous stream of text is coming in and you have to keep publishing the overall word count so far since 0:00 today, what would you do?
>
> Publishing the results for a window is easy but if we have to keep aggregating the results, how to go about it?
>
> I have tried to keep an StreamRDD with aggregated count and keep doing a fullouterjoin but didn't work. Seems like the StreamRDD gets reset.
>
> Kindly help.
>
> Regards,
> Sandeep Giri
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: Maintaining overall cumulative data in Spark Streaming
Posted by Silvio Fiorito <si...@granturing.com>.
In the update function you can return None for a key and it will remove it. If you’re restarting your app you can delete your checkpoint directory to start from scratch, rather than continuing from the previous state.
From: Sandeep Giri <sa...@knowbigdata.com>>
Date: Friday, October 30, 2015 at 9:29 AM
To: skaarthik oss <sk...@gmail.com>>
Cc: dev <de...@spark.apache.org>>, user <us...@spark.apache.org>>
Subject: Re: Maintaining overall cumulative data in Spark Streaming
How to we reset the aggregated statistics to null?
Regards,
Sandeep Giri,
+1 347 781 4573 (US)
+91-953-899-8962 (IN)
www.KnowBigData.com.<http://KnowBigData.com.>
Phone: +1-253-397-1945 (Office)
[linkedin icon]<https://linkedin.com/company/knowbigdata>[other site icon]<http://knowbigdata.com> [facebook icon] <https://facebook.com/knowbigdata> [twitter icon] <https://twitter.com/IKnowBigData> <https://twitter.com/IKnowBigData>
On Fri, Oct 30, 2015 at 9:49 AM, Sandeep Giri <sa...@knowbigdata.com>> wrote:
Yes, update state by key worked.
Though there are some more complications.
On Oct 30, 2015 8:27 AM, "skaarthik oss" <sk...@gmail.com>> wrote:
Did you consider UpdateStateByKey operation?
From: Sandeep Giri [mailto:sandeep@knowbigdata.com<ma...@knowbigdata.com>]
Sent: Thursday, October 29, 2015 3:09 PM
To: user <us...@spark.apache.org>>; dev <de...@spark.apache.org>>
Subject: Maintaining overall cumulative data in Spark Streaming
Dear All,
If a continuous stream of text is coming in and you have to keep publishing the overall word count so far since 0:00 today, what would you do?
Publishing the results for a window is easy but if we have to keep aggregating the results, how to go about it?
I have tried to keep an StreamRDD with aggregated count and keep doing a fullouterjoin but didn't work. Seems like the StreamRDD gets reset.
Kindly help.
Regards,
Sandeep Giri
Re: Maintaining overall cumulative data in Spark Streaming
Posted by Sandeep Giri <sa...@knowbigdata.com>.
How to we reset the aggregated statistics to null?
Regards,
Sandeep Giri,
+1 347 781 4573 (US)
+91-953-899-8962 (IN)
www.KnowBigData.com. <http://KnowBigData.com.>
Phone: +1-253-397-1945 (Office)
[image: linkedin icon] <https://linkedin.com/company/knowbigdata> [image:
other site icon] <http://knowbigdata.com> [image: facebook icon]
<https://facebook.com/knowbigdata> [image: twitter icon]
<https://twitter.com/IKnowBigData> <https://twitter.com/IKnowBigData>
On Fri, Oct 30, 2015 at 9:49 AM, Sandeep Giri <sa...@knowbigdata.com>
wrote:
> Yes, update state by key worked.
>
> Though there are some more complications.
> On Oct 30, 2015 8:27 AM, "skaarthik oss" <sk...@gmail.com> wrote:
>
>> Did you consider UpdateStateByKey operation?
>>
>>
>>
>> *From:* Sandeep Giri [mailto:sandeep@knowbigdata.com]
>> *Sent:* Thursday, October 29, 2015 3:09 PM
>> *To:* user <us...@spark.apache.org>; dev <de...@spark.apache.org>
>> *Subject:* Maintaining overall cumulative data in Spark Streaming
>>
>>
>>
>> Dear All,
>>
>>
>>
>> If a continuous stream of text is coming in and you have to keep
>> publishing the overall word count so far since 0:00 today, what would you
>> do?
>>
>>
>>
>> Publishing the results for a window is easy but if we have to keep
>> aggregating the results, how to go about it?
>>
>>
>>
>> I have tried to keep an StreamRDD with aggregated count and keep doing a
>> fullouterjoin but didn't work. Seems like the StreamRDD gets reset.
>>
>>
>>
>> Kindly help.
>>
>>
>>
>> Regards,
>>
>> Sandeep Giri
>>
>>
>>
>
Re: Maintaining overall cumulative data in Spark Streaming
Posted by Sandeep Giri <sa...@knowbigdata.com>.
How to we reset the aggregated statistics to null?
Regards,
Sandeep Giri,
+1 347 781 4573 (US)
+91-953-899-8962 (IN)
www.KnowBigData.com. <http://KnowBigData.com.>
Phone: +1-253-397-1945 (Office)
[image: linkedin icon] <https://linkedin.com/company/knowbigdata> [image:
other site icon] <http://knowbigdata.com> [image: facebook icon]
<https://facebook.com/knowbigdata> [image: twitter icon]
<https://twitter.com/IKnowBigData> <https://twitter.com/IKnowBigData>
On Fri, Oct 30, 2015 at 9:49 AM, Sandeep Giri <sa...@knowbigdata.com>
wrote:
> Yes, update state by key worked.
>
> Though there are some more complications.
> On Oct 30, 2015 8:27 AM, "skaarthik oss" <sk...@gmail.com> wrote:
>
>> Did you consider UpdateStateByKey operation?
>>
>>
>>
>> *From:* Sandeep Giri [mailto:sandeep@knowbigdata.com]
>> *Sent:* Thursday, October 29, 2015 3:09 PM
>> *To:* user <us...@spark.apache.org>; dev <de...@spark.apache.org>
>> *Subject:* Maintaining overall cumulative data in Spark Streaming
>>
>>
>>
>> Dear All,
>>
>>
>>
>> If a continuous stream of text is coming in and you have to keep
>> publishing the overall word count so far since 0:00 today, what would you
>> do?
>>
>>
>>
>> Publishing the results for a window is easy but if we have to keep
>> aggregating the results, how to go about it?
>>
>>
>>
>> I have tried to keep an StreamRDD with aggregated count and keep doing a
>> fullouterjoin but didn't work. Seems like the StreamRDD gets reset.
>>
>>
>>
>> Kindly help.
>>
>>
>>
>> Regards,
>>
>> Sandeep Giri
>>
>>
>>
>
RE: Maintaining overall cumulative data in Spark Streaming
Posted by Sandeep Giri <sa...@knowbigdata.com>.
Yes, update state by key worked.
Though there are some more complications.
On Oct 30, 2015 8:27 AM, "skaarthik oss" <sk...@gmail.com> wrote:
> Did you consider UpdateStateByKey operation?
>
>
>
> *From:* Sandeep Giri [mailto:sandeep@knowbigdata.com]
> *Sent:* Thursday, October 29, 2015 3:09 PM
> *To:* user <us...@spark.apache.org>; dev <de...@spark.apache.org>
> *Subject:* Maintaining overall cumulative data in Spark Streaming
>
>
>
> Dear All,
>
>
>
> If a continuous stream of text is coming in and you have to keep
> publishing the overall word count so far since 0:00 today, what would you
> do?
>
>
>
> Publishing the results for a window is easy but if we have to keep
> aggregating the results, how to go about it?
>
>
>
> I have tried to keep an StreamRDD with aggregated count and keep doing a
> fullouterjoin but didn't work. Seems like the StreamRDD gets reset.
>
>
>
> Kindly help.
>
>
>
> Regards,
>
> Sandeep Giri
>
>
>
RE: Maintaining overall cumulative data in Spark Streaming
Posted by Sandeep Giri <sa...@knowbigdata.com>.
Yes, update state by key worked.
Though there are some more complications.
On Oct 30, 2015 8:27 AM, "skaarthik oss" <sk...@gmail.com> wrote:
> Did you consider UpdateStateByKey operation?
>
>
>
> *From:* Sandeep Giri [mailto:sandeep@knowbigdata.com]
> *Sent:* Thursday, October 29, 2015 3:09 PM
> *To:* user <us...@spark.apache.org>; dev <de...@spark.apache.org>
> *Subject:* Maintaining overall cumulative data in Spark Streaming
>
>
>
> Dear All,
>
>
>
> If a continuous stream of text is coming in and you have to keep
> publishing the overall word count so far since 0:00 today, what would you
> do?
>
>
>
> Publishing the results for a window is easy but if we have to keep
> aggregating the results, how to go about it?
>
>
>
> I have tried to keep an StreamRDD with aggregated count and keep doing a
> fullouterjoin but didn't work. Seems like the StreamRDD gets reset.
>
>
>
> Kindly help.
>
>
>
> Regards,
>
> Sandeep Giri
>
>
>
RE: Maintaining overall cumulative data in Spark Streaming
Posted by skaarthik oss <sk...@gmail.com>.
Did you consider UpdateStateByKey operation?
From: Sandeep Giri [mailto:sandeep@knowbigdata.com]
Sent: Thursday, October 29, 2015 3:09 PM
To: user <us...@spark.apache.org>; dev <de...@spark.apache.org>
Subject: Maintaining overall cumulative data in Spark Streaming
Dear All,
If a continuous stream of text is coming in and you have to keep publishing the overall word count so far since 0:00 today, what would you do?
Publishing the results for a window is easy but if we have to keep aggregating the results, how to go about it?
I have tried to keep an StreamRDD with aggregated count and keep doing a fullouterjoin but didn't work. Seems like the StreamRDD gets reset.
Kindly help.
Regards,
Sandeep Giri
RE: Maintaining overall cumulative data in Spark Streaming
Posted by Silvio Fiorito <si...@granturing.com>.
You could use updateStateByKey. There's a stateful word count example on Github.
https://github.com/apache/spark/blob/master/examples/src/main/scala/org/apache/spark/examples/streaming/StatefulNetworkWordCount.scala
________________________________
From: Sandeep Giri<ma...@knowbigdata.com>
Sent: 10/29/2015 6:08 PM
To: user<ma...@spark.apache.org>; dev<ma...@spark.apache.org>
Subject: Maintaining overall cumulative data in Spark Streaming
Dear All,
If a continuous stream of text is coming in and you have to keep publishing the overall word count so far since 0:00 today, what would you do?
Publishing the results for a window is easy but if we have to keep aggregating the results, how to go about it?
I have tried to keep an StreamRDD with aggregated count and keep doing a fullouterjoin but didn't work. Seems like the StreamRDD gets reset.
Kindly help.
Regards,
Sandeep Giri