You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Justin Lent <ju...@gmail.com> on 2014/01/30 02:45:49 UTC
SparkR: filter() function?
any idea when the Scala filter() function equivalent will be availalbe
in the SparkR implementation? Or is there a simple way to implement it
in R with the existing functions?
Thanks!
-Justin
Re: SparkR: filter() function?
Posted by Shivaram Venkataraman <sh...@eecs.berkeley.edu>.
I replied on the SO post - The bug you ran into is a slightly different one
and is with the `show` method in RDDs. I've opened a PR to fix this at
https://github.com/apache/spark/pull/6035
Thanks
Shivaram
On Wed, May 6, 2015 at 1:55 AM, himaeda <hi...@deloitte.co.uk> wrote:
> Has this issue re-appeared?
>
> I posted this on SO before I knew about this list...
>
>
> http://stackoverflow.com/questions/30057702/sparkr-filterrdd-and-flatmap-not-working
> <
> http://stackoverflow.com/questions/30057702/sparkr-filterrdd-and-flatmap-not-working
> >
>
> Also I don't have access to the issues on github so wasn't sure if it was a
> known issue or not....
>
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/SparkR-filter-function-tp1033p22782.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>
Re: SparkR: filter() function?
Posted by himaeda <hi...@deloitte.co.uk>.
Has this issue re-appeared?
I posted this on SO before I knew about this list...
http://stackoverflow.com/questions/30057702/sparkr-filterrdd-and-flatmap-not-working
<http://stackoverflow.com/questions/30057702/sparkr-filterrdd-and-flatmap-not-working>
Also I don't have access to the issues on github so wasn't sure if it was a
known issue or not....
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/SparkR-filter-function-tp1033p22782.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: SparkR: filter() function?
Posted by ConcreteVitamin <co...@gmail.com>.
I opened an issue
(https://github.com/amplab-extras/SparkR-pkg/issues/14) to track this.
Zongheng
On Wed, Jan 29, 2014 at 6:11 PM, Shivaram Venkataraman
<sh...@eecs.berkeley.edu> wrote:
> You could do a mapPartitions and Filter within each partition -- Something like
>
> numbers <- parallelize(sc, 1:20, 4L)
> evenNumbers <- lapplyPartition(numbers, function(part) { Filter(
> function(x) { x%%2 == 0} , part) })
> collect(evenNumbers)
>
> It should be simple to add this to the SparkR API as well -- Let me
> know if you want to send a PR !
>
> Thanks
> Shivaram
>
> On Wed, Jan 29, 2014 at 5:45 PM, Justin Lent <ju...@gmail.com> wrote:
>> any idea when the Scala filter() function equivalent will be availalbe
>> in the SparkR implementation? Or is there a simple way to implement it
>> in R with the existing functions?
>>
>> Thanks!
>> -Justin
Re: SparkR: filter() function?
Posted by Shivaram Venkataraman <sh...@eecs.berkeley.edu>.
You could do a mapPartitions and Filter within each partition -- Something like
numbers <- parallelize(sc, 1:20, 4L)
evenNumbers <- lapplyPartition(numbers, function(part) { Filter(
function(x) { x%%2 == 0} , part) })
collect(evenNumbers)
It should be simple to add this to the SparkR API as well -- Let me
know if you want to send a PR !
Thanks
Shivaram
On Wed, Jan 29, 2014 at 5:45 PM, Justin Lent <ju...@gmail.com> wrote:
> any idea when the Scala filter() function equivalent will be availalbe
> in the SparkR implementation? Or is there a simple way to implement it
> in R with the existing functions?
>
> Thanks!
> -Justin