You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by ajay garg <aj...@mobileum.com> on 2014/06/27 08:42:48 UTC

Map with filter on JavaRdd

Hi All,
         Is it possible to map and filter a javardd in a single operation?

Thanks



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Map-with-filter-on-JavaRdd-tp8401.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

Re: Map with filter on JavaRdd

Posted by Daniel Siegmann <da...@velos.io>.

If for some reason it would be easier to do your mapping and filtering in a
single function, you can also use RDD.flatMap (returning an empty sequence
is equivalent to a filter). But unless you have good reason you should have
a separate map and filter transform, as Mayur said.

On Fri, Jun 27, 2014 at 7:43 AM, ajay garg <aj...@mobileum.com> wrote:

> Thanks Mayur for clarification..
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Map-with-filter-on-JavaRdd-tp8401p8410.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>

-- 
Daniel Siegmann, Software Developer
Velos
Accelerating Machine Learning

440 NINTH AVENUE, 11TH FLOOR, NEW YORK, NY 10001
E: daniel.siegmann@velos.io W: www.velos.io

Re: Map with filter on JavaRdd

Posted by ajay garg <aj...@mobileum.com>.

Thanks Mayur for clarification..



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Map-with-filter-on-JavaRdd-tp8401p8410.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

Re: Map with filter on JavaRdd

Posted by Mayur Rustagi <ma...@gmail.com>.

It happens in a single operation itself. You may write it separately but
the stages are performed together if its possible. You will see only one
task in the output of your application.

Mayur Rustagi
Ph: +1 (760) 203 3257
http://www.sigmoidanalytics.com
@mayur_rustagi <https://twitter.com/mayur_rustagi>

On Fri, Jun 27, 2014 at 12:12 PM, ajay garg <aj...@mobileum.com> wrote:

> Hi All,
>          Is it possible to map and filter a javardd in a single operation?
>
> Thanks
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Map-with-filter-on-JavaRdd-tp8401.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>