You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Arpan Ghosh <ar...@automatic.com> on 2014/08/22 22:32:34 UTC
[PySpark] order of values in GroupByKey()
Is there any way to control the ordering of values for each key during a
groupByKey() operation? Is there some sort of implicit ordering in place
already?
Thanks
Arpan
Re: [PySpark] order of values in GroupByKey()
Posted by Matthew Farrellee <ma...@redhat.com>.
you can kv.mapValues(sorted), but that's definitely less efficient than
sorting during the groupBy
you could try using combineByKey directly w/ heapq...
from heapq import heapify, heappush, merge
def createCombiner(x):
return [x]
def mergeValues(xs, x):
heappush(xs, x)
return xs
def mergeCombiners(a, b):
return merge(a, b)
rdd.combineByKey(createCombiner, mergeValues, mergeCombiners)
best,
matt
On 08/22/2014 10:41 PM, Arpan Ghosh wrote:
> I was grouping time series data by a key. I want the values to be sorted
> by timestamp after the grouping.
>
>
> On Fri, Aug 22, 2014 at 7:26 PM, Matthew Farrellee <matt@redhat.com
> <ma...@redhat.com>> wrote:
>
> On 08/22/2014 04:32 PM, Arpan Ghosh wrote:
>
> Is there any way to control the ordering of values for each key
> during a
> groupByKey() operation? Is there some sort of implicit ordering
> in place
> already?
>
> Thanks
>
> Arpan
>
>
> there's no implicit ordering in place. the same holds for the order
> of keys, unless you use sortByKey.
>
> what are you trying to achieve?
>
> best,
>
>
> matt
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: [PySpark] order of values in GroupByKey()
Posted by Arpan Ghosh <ar...@automatic.com>.
I was grouping time series data by a key. I want the values to be sorted by
timestamp after the grouping.
On Fri, Aug 22, 2014 at 7:26 PM, Matthew Farrellee <ma...@redhat.com> wrote:
> On 08/22/2014 04:32 PM, Arpan Ghosh wrote:
>
>> Is there any way to control the ordering of values for each key during a
>> groupByKey() operation? Is there some sort of implicit ordering in place
>> already?
>>
>> Thanks
>>
>> Arpan
>>
>
> there's no implicit ordering in place. the same holds for the order of
> keys, unless you use sortByKey.
>
> what are you trying to achieve?
>
> best,
>
>
> matt
>
Re: [PySpark] order of values in GroupByKey()
Posted by Matthew Farrellee <ma...@redhat.com>.
On 08/22/2014 04:32 PM, Arpan Ghosh wrote:
> Is there any way to control the ordering of values for each key during a
> groupByKey() operation? Is there some sort of implicit ordering in place
> already?
>
> Thanks
>
> Arpan
there's no implicit ordering in place. the same holds for the order of
keys, unless you use sortByKey.
what are you trying to achieve?
best,
matt
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org