You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Akhil Das <ak...@sigmoidanalytics.com> on 2015/09/09 12:13:27 UTC
Re: I am very new to Spark. I have a very basic question. I have an
array of values: listofECtokens: Array[String] = Array(EC-17A5206955089011B,
EC-17A5206955089011A) I want to filter an RDD for all of these token values.
I tried the following way: val ECtokens = for (token <- listofECtokens)
rddAll.filter(line => line.contains(token)) Output: ECtokens: Unit = () I got
an empty Unit even when there are records with these tokens. What am I doing wrong?
Try this:
val tocks = Array("EC-17A5206955089011B","EC-17A5206955089011A")
val rddAll = sc.parallelize(List("This contains EC-17A5206955089011B","This
doesnt"))
rddAll.filter(line => {
var found = false
for(item <- tocks){
if(line.contains(item)) found = true
}
found
}).collect()
Output:
res8: Array[String] = Array(This contains EC-17A5206955089011B)
Thanks
Best Regards
On Wed, Sep 9, 2015 at 3:25 PM, prachicsa <pr...@gmail.com> wrote:
>
>
> I am very new to Spark.
>
> I have a very basic question. I have an array of values:
>
> listofECtokens: Array[String] = Array(EC-17A5206955089011B,
> EC-17A5206955089011A)
>
> I want to filter an RDD for all of these token values. I tried the
> following
> way:
>
> val ECtokens = for (token <- listofECtokens) rddAll.filter(line =>
> line.contains(token))
>
> Output:
>
> ECtokens: Unit = ()
>
> I got an empty Unit even when there are records with these tokens. What am
> I
> doing wrong?
>
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/I-am-very-new-to-Spark-I-have-a-very-basic-question-I-have-an-array-of-values-listofECtokens-Array-S-tp24617.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>
Re: I am very new to Spark. I have a very basic question. I have an array of values: listofECtokens: Array[String] = Array(EC-17A5206955089011B, EC-17A5206955089011A) I want to filter an RDD for all of these token values. I tried the following way: val ECtokens = for (token <- listofECtokens) rddAll.filter(line => line.contains(token)) Output: ECtokens: Unit = () I got an empty Unit even when there are records with these tokens. What am I doing wrong?
Posted by Ted Yu <yu...@gmail.com>.
Prachicsa:
If the number of EC tokens is high, please consider using a set instead of array for better lookup performance.
BTW use short, descriptive subject for future emails.
> On Sep 9, 2015, at 3:13 AM, Akhil Das <ak...@sigmoidanalytics.com> wrote:
>
> Try this:
>
> val tocks = Array("EC-17A5206955089011B","EC-17A5206955089011A")
>
> val rddAll = sc.parallelize(List("This contains EC-17A5206955089011B","This doesnt"))
>
> rddAll.filter(line => {
> var found = false
> for(item <- tocks){
> if(line.contains(item)) found = true
> }
> found
> }).collect()
>
>
> Output:
> res8: Array[String] = Array(This contains EC-17A5206955089011B)
>
> Thanks
> Best Regards
>
>> On Wed, Sep 9, 2015 at 3:25 PM, prachicsa <pr...@gmail.com> wrote:
>>
>>
>> I am very new to Spark.
>>
>> I have a very basic question. I have an array of values:
>>
>> listofECtokens: Array[String] = Array(EC-17A5206955089011B,
>> EC-17A5206955089011A)
>>
>> I want to filter an RDD for all of these token values. I tried the following
>> way:
>>
>> val ECtokens = for (token <- listofECtokens) rddAll.filter(line =>
>> line.contains(token))
>>
>> Output:
>>
>> ECtokens: Unit = ()
>>
>> I got an empty Unit even when there are records with these tokens. What am I
>> doing wrong?
>>
>>
>>
>>
>> --
>> View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/I-am-very-new-to-Spark-I-have-a-very-basic-question-I-have-an-array-of-values-listofECtokens-Array-S-tp24617.html
>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> For additional commands, e-mail: user-help@spark.apache.org
>