pyspark - extract 1 field from string

[Chop <th...@att.net>]

I'm stumped with how to take 1 RDD that has lines like:

 4,01012009,00:00,1289,4
 5,01012009,00:00,1326,4
 6,01012009,00:00,1497,7

and produce a new RDD with just the 4th field from each line (1289, 1326,
1497)

I don't want to apply a conditional, I just want to grab that one field from
each line in the existing RDD

TIA






--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/pyspark-extract-1-field-from-string-tp16456.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org

Re: pyspark - extract 1 field from string

[Davies Liu <da...@databricks.com>]

rdd.map(lambda line: int(line.split(',')[3]))

On Tue, Oct 14, 2014 at 6:58 PM, Chop <th...@att.net> wrote:
> I'm stumped with how to take 1 RDD that has lines like:
>
>  4,01012009,00:00,1289,4
>  5,01012009,00:00,1326,4
>  6,01012009,00:00,1497,7
>
> and produce a new RDD with just the 4th field from each line (1289, 1326,
> 1497)
>
> I don't want to apply a conditional, I just want to grab that one field from
> each line in the existing RDD
>
> TIA
>
>
>
>
>
>
> --
> View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/pyspark-extract-1-field-from-string-tp16456.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org