You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "ashish shenoy (JIRA)" <ji...@apache.org> on 2015/09/02 00:38:45 UTC
[jira] [Comment Edited] (SPARK-10405) Support takeOrdered and topK
values per key
[ https://issues.apache.org/jira/browse/SPARK-10405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14726331#comment-14726331 ]
ashish shenoy edited comment on SPARK-10405 at 9/1/15 10:38 PM:
----------------------------------------------------------------
[~srowen] yes, technically its a good to have not a must have. I could think of many instances where such an API would be very convenient and useful for users to have.
I was using the aggregateByKey() with a custom written bounded priority queue. As per the spark documentation, the func param to foldByKey() should be an associative merge function. So I can think of how this can be used to get the max or min value per key, but not the top or bottom values. Since I am a spark-noob, can you pls give an example of how one could use a priorityQueue with foldByKey() ?
Also, the default PriorityQueue implementation in java.util is unbounded; could this cause OOM exceptions if the cardinality of the keyset is very large ?
was (Author: ashishenoyp@gmail.com):
[~srowen] yes, technically its a good to have not a must have. I could think of many instances where such an API would be very convenient and useful for users to have.
Thanks for that foldByKey() tip; I was using the aggregateByKey() with a custom written bounded priority queue. Since I am a spark-noob, can you pls give an example of how one could use a priorityQueue with foldByKey() ? Also, the default PriorityQueue implementation in java.util is unbounded; could this cause OOM exceptions if the cardinality of the keyset is very large ?
> Support takeOrdered and topK values per key
> -------------------------------------------
>
> Key: SPARK-10405
> URL: https://issues.apache.org/jira/browse/SPARK-10405
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Reporter: ashish shenoy
> Labels: features, newbie
>
> Spark provides the top() and takeOrdered() APIs that return "top" or "bottom" items from a given RDD.
> It'd be good to have an API that returned the "top" values per key for a keyed RDD i.e. RDDpair. Such an API would be very useful for cases where the task is to only display an ordered subset of the input data.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org