You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Amir Rahnama <am...@gmail.com> on 2015/11/11 22:53:31 UTC
How can you sort wordcounts by counts in stateful_network_wordcount.py
example
Hey,
Anybody knows how can one sort the result in the stateful example?
Python would be prefered.
https://github.com/apache/spark/blob/859dff56eb0f8c63c86e7e900a12340c199e6247/examples/src/main/python/streaming/stateful_network_wordcount.py
--
Thanks and Regards,
Amir Hossein Rahnama
*Tel: +46 (0) 761 681 102*
Website: www.ambodi.com
Twitter: @_ambodi <https://twitter.com/_ambodi>
Re: How can you sort wordcounts by counts in stateful_network_wordcount.py
example
Posted by Amir Rahnama <am...@gmail.com>.
Thanks Matthew.
According to documentation, foreachRDD is used for writing right?
I did with like this and it worked:
address_by_destination =
address.updateStateByKey(updateFunction).transform(lambda rdd:
rdd.sortBy(lambda x: x[1], False))
Thanks anyways :)
On Thu, Nov 12, 2015 at 4:53 PM, Young, Matthew T <matthew.t.young@intel.com
> wrote:
> You can use foreachRDD
> <http://spark.apache.org/docs/latest/streaming-programming-guide.html#output-operations-on-dstreams>
> to get access to the batch API in streaming jobs.
>
>
>
>
>
> *From:* Amir Rahnama [mailto:amirrahnama@gmail.com]
> *Sent:* Thursday, November 12, 2015 12:11 AM
> *To:* ayan guha <gu...@gmail.com>
> *Cc:* user <us...@spark.apache.org>
> *Subject:* Re: How can you sort wordcounts by counts in
> stateful_network_wordcount.py example
>
>
>
> Hi Ayan,
>
>
>
> Thanks for help,
>
>
>
> Your example is not the streaming example. There we don't have sortByKey.
>
>
>
>
>
>
>
> On Wed, Nov 11, 2015 at 11:35 PM, ayan guha <gu...@gmail.com> wrote:
>
> how about this?
>
>
>
> sorted = running_counts.map(lambda t: t[1],t[0]).sortByKey()
>
>
>
> Basically swap key and value of the RDD and then sort?
>
>
>
> On Thu, Nov 12, 2015 at 8:53 AM, Amir Rahnama <am...@gmail.com>
> wrote:
>
> Hey,
>
>
>
> Anybody knows how can one sort the result in the stateful example?
>
>
>
> Python would be prefered.
>
>
>
>
> https://github.com/apache/spark/blob/859dff56eb0f8c63c86e7e900a12340c199e6247/examples/src/main/python/streaming/stateful_network_wordcount.py
>
> --
>
> Thanks and Regards,
>
>
>
> Amir Hossein Rahnama
>
>
>
> *Tel: +46 (0) 761 681 102*
>
> Website: www.ambodi.com
>
> Twitter: @_ambodi <https://twitter.com/_ambodi>
>
>
>
>
>
> --
>
> Best Regards,
> Ayan Guha
>
>
>
>
>
> --
>
> Thanks and Regards,
>
>
>
> Amir Hossein Rahnama
>
>
>
> *Tel: +46 (0) 761 681 102*
>
> Website: www.ambodi.com
>
> Twitter: @_ambodi <https://twitter.com/_ambodi>
>
--
Thanks and Regards,
Amir Hossein Rahnama
*Tel: +46 (0) 761 681 102*
Website: www.ambodi.com
Twitter: @_ambodi <https://twitter.com/_ambodi>
RE: How can you sort wordcounts by counts in
stateful_network_wordcount.py example
Posted by "Young, Matthew T" <ma...@intel.com>.
You can use foreachRDD<http://spark.apache.org/docs/latest/streaming-programming-guide.html#output-operations-on-dstreams> to get access to the batch API in streaming jobs.
From: Amir Rahnama [mailto:amirrahnama@gmail.com]
Sent: Thursday, November 12, 2015 12:11 AM
To: ayan guha <gu...@gmail.com>
Cc: user <us...@spark.apache.org>
Subject: Re: How can you sort wordcounts by counts in stateful_network_wordcount.py example
Hi Ayan,
Thanks for help,
Your example is not the streaming example. There we don't have sortByKey.
On Wed, Nov 11, 2015 at 11:35 PM, ayan guha <gu...@gmail.com>> wrote:
how about this?
sorted = running_counts.map(lambda t: t[1],t[0]).sortByKey()
Basically swap key and value of the RDD and then sort?
On Thu, Nov 12, 2015 at 8:53 AM, Amir Rahnama <am...@gmail.com>> wrote:
Hey,
Anybody knows how can one sort the result in the stateful example?
Python would be prefered.
https://github.com/apache/spark/blob/859dff56eb0f8c63c86e7e900a12340c199e6247/examples/src/main/python/streaming/stateful_network_wordcount.py
--
Thanks and Regards,
Amir Hossein Rahnama
Tel: +46 (0) 761 681 102
Website: www.ambodi.com<http://www.ambodi.com>
Twitter: @_ambodi<https://twitter.com/_ambodi>
--
Best Regards,
Ayan Guha
--
Thanks and Regards,
Amir Hossein Rahnama
Tel: +46 (0) 761 681 102
Website: www.ambodi.com<http://www.ambodi.com>
Twitter: @_ambodi<https://twitter.com/_ambodi>
Re: How can you sort wordcounts by counts in stateful_network_wordcount.py
example
Posted by Amir Rahnama <am...@gmail.com>.
Hi Ayan,
Thanks for help,
Your example is not the streaming example. There we don't have sortByKey.
On Wed, Nov 11, 2015 at 11:35 PM, ayan guha <gu...@gmail.com> wrote:
> how about this?
>
> sorted = running_counts.map(lambda t: t[1],t[0]).sortByKey()
>
> Basically swap key and value of the RDD and then sort?
>
> On Thu, Nov 12, 2015 at 8:53 AM, Amir Rahnama <am...@gmail.com>
> wrote:
>
>> Hey,
>>
>> Anybody knows how can one sort the result in the stateful example?
>>
>> Python would be prefered.
>>
>>
>> https://github.com/apache/spark/blob/859dff56eb0f8c63c86e7e900a12340c199e6247/examples/src/main/python/streaming/stateful_network_wordcount.py
>> --
>> Thanks and Regards,
>>
>> Amir Hossein Rahnama
>>
>> *Tel: +46 (0) 761 681 102*
>> Website: www.ambodi.com
>> Twitter: @_ambodi <https://twitter.com/_ambodi>
>>
>
>
>
> --
> Best Regards,
> Ayan Guha
>
--
Thanks and Regards,
Amir Hossein Rahnama
*Tel: +46 (0) 761 681 102*
Website: www.ambodi.com
Twitter: @_ambodi <https://twitter.com/_ambodi>
Re: How can you sort wordcounts by counts in stateful_network_wordcount.py
example
Posted by ayan guha <gu...@gmail.com>.
how about this?
sorted = running_counts.map(lambda t: t[1],t[0]).sortByKey()
Basically swap key and value of the RDD and then sort?
On Thu, Nov 12, 2015 at 8:53 AM, Amir Rahnama <am...@gmail.com> wrote:
> Hey,
>
> Anybody knows how can one sort the result in the stateful example?
>
> Python would be prefered.
>
>
> https://github.com/apache/spark/blob/859dff56eb0f8c63c86e7e900a12340c199e6247/examples/src/main/python/streaming/stateful_network_wordcount.py
> --
> Thanks and Regards,
>
> Amir Hossein Rahnama
>
> *Tel: +46 (0) 761 681 102*
> Website: www.ambodi.com
> Twitter: @_ambodi <https://twitter.com/_ambodi>
>
--
Best Regards,
Ayan Guha