You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Shepherd <Ch...@huawei.com> on 2015/10/20 01:27:05 UTC
Filter RDD
Hi all, <br>
I have a very simple question.<br>
I have a RDD, saying r1, which contains 5 columns, with both string and
Int.<br>
How can I get a sub RDD, based on a rule, that the second column equals to a
string (s)?<br>
Thanks a lot.
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Filter-RDD-tp25133.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Filter RDD
Posted by Ted Yu <yu...@gmail.com>.
See the filter() method:
https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/rdd/RDD.scala#L334
Cheers
On Mon, Oct 19, 2015 at 4:27 PM, Shepherd <Ch...@huawei.com> wrote:
> Hi all, <br>
> I have a very simple question.<br>
> I have a RDD, saying r1, which contains 5 columns, with both string and
> Int.<br>
> How can I get a sub RDD, based on a rule, that the second column equals to
> a
> string (s)?<br>
>
> Thanks a lot.
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Filter-RDD-tp25133.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>
Re: Filter RDD
Posted by "ravi.gawai" <ra...@gmail.com>.
you can use mapfunction..
This is java example..
final JavaRDD<Product> rdd1 = sc.textFile("filepath").map((line) -> {
//logic for line to product converstion});
Product class might have 5 attributes like you said class Product{
String str1;
int i1;
String str2;
int i2;
String str3;
// with getter setters
}
Now you can convert this Product RDD to another CustomRDD lets say Class
SubProduct{ String str1;int i1; //getter setters }
final JavaRDD<SubProduct> rdd2 = rdd1.map(product -> {
final SubProduct subProduct = new SubProduct();
// map product attributes to subProduct attributes;
return subProduct;
});
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Filter-RDD-tp25133p25148.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org