You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Amit Behera <am...@gmail.com> on 2014/12/27 21:24:42 UTC
unable to check whether an item is present in RDD
Hi All,
I want to check an item is present or not in a RDD of Iterable[Int] using
scala
something like in java we do :
*list.contains(item)*
and the statement returns true if the item is present otherwise false.
Please help me to find the solution.
Thanks
Amit
Re: unable to check whether an item is present in RDD
Posted by Nicholas Chammas <ni...@gmail.com>.
Is the item you're looking up an Int? So you want to find which of the
Iterable[Int] elements in your RDD contains the Int you're looking for?
On Sat Dec 27 2014 at 3:26:41 PM Amit Behera <am...@gmail.com> wrote:
> Hi All,
>
> I want to check an item is present or not in a RDD of Iterable[Int] using
> scala
>
> something like in java we do :
>
> *list.contains(item)*
>
> and the statement returns true if the item is present otherwise false.
>
> Please help me to find the solution.
>
> Thanks
> Amit
>
>
Re: unable to check whether an item is present in RDD
Posted by Amit Behera <am...@gmail.com>.
Hi Sean,
I have a RDD like
*theItems: org.apache.spark.rdd.RDD[Iterable[Int]]*
I did like
*val items = theItems.collect *//to get it as an array
items: Array[Iterable[Int]]
*val check = items.contains(item)*
Thanks
Amit
On Sun, Dec 28, 2014 at 1:58 PM, Amit Behera <am...@gmail.com> wrote:
> Hi Nicholas,
>
> The RDD contains only one Iterable[Int].
>
> Pankaj,
> I used *collect* and I am getting as *items: Array[Iterable[Int]].*
>
> Then I did like :
>
> *val check = items.take(1).contains(item)*
>
> I am getting *check: Boolean = false, *but the item is present.
>
>
> Thanks
> Amit
>
>
>
> On Sun, Dec 28, 2014 at 8:56 AM, Pankaj <pa...@gmail.com> wrote:
>
>> Amit, I think you can use collect method to change RDD to Array so you
>> can use array methods for the same.
>>
>> I think transformation functions would not allow to do that as they are
>> lazily called you need to use Actions LIke collect or foreach method
>>
>> On Sun, Dec 28, 2014 at 1:54 AM, Amit Behera <am...@gmail.com>
>> wrote:
>>
>>> Hi All,
>>>
>>> I want to check an item is present or not in a RDD of Iterable[Int]
>>> using scala
>>>
>>> something like in java we do :
>>>
>>> *list.contains(item)*
>>>
>>> and the statement returns true if the item is present otherwise false.
>>>
>>> Please help me to find the solution.
>>>
>>> Thanks
>>> Amit
>>>
>>>
>>
>
Re: unable to check whether an item is present in RDD
Posted by Amit Behera <am...@gmail.com>.
Hi Sean and Nicholas
Thank you very much, *exists* method works here :)
On Sun, Dec 28, 2014 at 2:27 PM, Sean Owen <so...@cloudera.com> wrote:
> Try instead i.exists(_ == target)
> On Dec 28, 2014 8:46 AM, "Amit Behera" <am...@gmail.com> wrote:
>
>> Hi Nicholas,
>>
>> I am getting
>> error: value contains is not a member of Iterable[Int]
>>
>> On Sun, Dec 28, 2014 at 2:06 PM, Nicholas Chammas <
>> nicholas.chammas@gmail.com> wrote:
>>
>>> take(1) will just give you a single item from the RDD. RDDs are not
>>> ideal for point lookups like you are doing, but you can find the element
>>> you want by doing something like:
>>>
>>> rdd.filter(i => i.contains(target)).collect()
>>>
>>> Where target is the Int you are looking for.
>>>
>>> Nick
>>>
>>>
>>> On Sun Dec 28 2014 at 3:28:45 AM Amit Behera <am...@gmail.com>
>>> wrote:
>>>
>>>> Hi Nicholas,
>>>>
>>>> The RDD contains only one Iterable[Int].
>>>>
>>>> Pankaj,
>>>> I used *collect* and I am getting as *items: Array[Iterable[Int]].*
>>>>
>>>> Then I did like :
>>>>
>>>> *val check = items.take(1).contains(item)*
>>>>
>>>> I am getting *check: Boolean = false, *but the item is present.
>>>>
>>>>
>>>> Thanks
>>>> Amit
>>>>
>>>>
>>>>
>>>> On Sun, Dec 28, 2014 at 8:56 AM, Pankaj <pa...@gmail.com>
>>>> wrote:
>>>>
>>>>> Amit, I think you can use collect method to change RDD to Array so you
>>>>> can use array methods for the same.
>>>>>
>>>>> I think transformation functions would not allow to do that as they
>>>>> are lazily called you need to use Actions LIke collect or foreach method
>>>>>
>>>>
>>>>> On Sun, Dec 28, 2014 at 1:54 AM, Amit Behera <am...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi All,
>>>>>>
>>>>>> I want to check an item is present or not in a RDD of Iterable[Int]
>>>>>> using scala
>>>>>>
>>>>>> something like in java we do :
>>>>>>
>>>>>> *list.contains(item)*
>>>>>>
>>>>>> and the statement returns true if the item is present otherwise false.
>>>>>>
>>>>>> Please help me to find the solution.
>>>>>>
>>>>>> Thanks
>>>>>> Amit
>>>>>>
>>>>>>
>>>>>
>>
Re: unable to check whether an item is present in RDD
Posted by Sean Owen <so...@cloudera.com>.
Try instead i.exists(_ == target)
On Dec 28, 2014 8:46 AM, "Amit Behera" <am...@gmail.com> wrote:
> Hi Nicholas,
>
> I am getting
> error: value contains is not a member of Iterable[Int]
>
> On Sun, Dec 28, 2014 at 2:06 PM, Nicholas Chammas <
> nicholas.chammas@gmail.com> wrote:
>
>> take(1) will just give you a single item from the RDD. RDDs are not ideal
>> for point lookups like you are doing, but you can find the element you want
>> by doing something like:
>>
>> rdd.filter(i => i.contains(target)).collect()
>>
>> Where target is the Int you are looking for.
>>
>> Nick
>>
>>
>> On Sun Dec 28 2014 at 3:28:45 AM Amit Behera <am...@gmail.com>
>> wrote:
>>
>>> Hi Nicholas,
>>>
>>> The RDD contains only one Iterable[Int].
>>>
>>> Pankaj,
>>> I used *collect* and I am getting as *items: Array[Iterable[Int]].*
>>>
>>> Then I did like :
>>>
>>> *val check = items.take(1).contains(item)*
>>>
>>> I am getting *check: Boolean = false, *but the item is present.
>>>
>>>
>>> Thanks
>>> Amit
>>>
>>>
>>>
>>> On Sun, Dec 28, 2014 at 8:56 AM, Pankaj <pa...@gmail.com>
>>> wrote:
>>>
>>>> Amit, I think you can use collect method to change RDD to Array so you
>>>> can use array methods for the same.
>>>>
>>>> I think transformation functions would not allow to do that as they are
>>>> lazily called you need to use Actions LIke collect or foreach method
>>>>
>>>
>>>> On Sun, Dec 28, 2014 at 1:54 AM, Amit Behera <am...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi All,
>>>>>
>>>>> I want to check an item is present or not in a RDD of Iterable[Int]
>>>>> using scala
>>>>>
>>>>> something like in java we do :
>>>>>
>>>>> *list.contains(item)*
>>>>>
>>>>> and the statement returns true if the item is present otherwise false.
>>>>>
>>>>> Please help me to find the solution.
>>>>>
>>>>> Thanks
>>>>> Amit
>>>>>
>>>>>
>>>>
>
Re: unable to check whether an item is present in RDD
Posted by Amit Behera <am...@gmail.com>.
Hi Nicholas,
I am getting
error: value contains is not a member of Iterable[Int]
On Sun, Dec 28, 2014 at 2:06 PM, Nicholas Chammas <
nicholas.chammas@gmail.com> wrote:
> take(1) will just give you a single item from the RDD. RDDs are not ideal
> for point lookups like you are doing, but you can find the element you want
> by doing something like:
>
> rdd.filter(i => i.contains(target)).collect()
>
> Where target is the Int you are looking for.
>
> Nick
>
>
> On Sun Dec 28 2014 at 3:28:45 AM Amit Behera <am...@gmail.com> wrote:
>
>> Hi Nicholas,
>>
>> The RDD contains only one Iterable[Int].
>>
>> Pankaj,
>> I used *collect* and I am getting as *items: Array[Iterable[Int]].*
>>
>> Then I did like :
>>
>> *val check = items.take(1).contains(item)*
>>
>> I am getting *check: Boolean = false, *but the item is present.
>>
>>
>> Thanks
>> Amit
>>
>>
>>
>> On Sun, Dec 28, 2014 at 8:56 AM, Pankaj <pa...@gmail.com> wrote:
>>
>>> Amit, I think you can use collect method to change RDD to Array so you
>>> can use array methods for the same.
>>>
>>> I think transformation functions would not allow to do that as they are
>>> lazily called you need to use Actions LIke collect or foreach method
>>>
>>
>>> On Sun, Dec 28, 2014 at 1:54 AM, Amit Behera <am...@gmail.com>
>>> wrote:
>>>
>>>> Hi All,
>>>>
>>>> I want to check an item is present or not in a RDD of Iterable[Int]
>>>> using scala
>>>>
>>>> something like in java we do :
>>>>
>>>> *list.contains(item)*
>>>>
>>>> and the statement returns true if the item is present otherwise false.
>>>>
>>>> Please help me to find the solution.
>>>>
>>>> Thanks
>>>> Amit
>>>>
>>>>
>>>
Re: unable to check whether an item is present in RDD
Posted by Nicholas Chammas <ni...@gmail.com>.
take(1) will just give you a single item from the RDD. RDDs are not ideal
for point lookups like you are doing, but you can find the element you want
by doing something like:
rdd.filter(i => i.contains(target)).collect()
Where target is the Int you are looking for.
Nick
On Sun Dec 28 2014 at 3:28:45 AM Amit Behera <am...@gmail.com> wrote:
> Hi Nicholas,
>
> The RDD contains only one Iterable[Int].
>
> Pankaj,
> I used *collect* and I am getting as *items: Array[Iterable[Int]].*
>
> Then I did like :
>
> *val check = items.take(1).contains(item)*
>
> I am getting *check: Boolean = false, *but the item is present.
>
>
> Thanks
> Amit
>
>
>
> On Sun, Dec 28, 2014 at 8:56 AM, Pankaj <pa...@gmail.com> wrote:
>
>> Amit, I think you can use collect method to change RDD to Array so you
>> can use array methods for the same.
>>
>> I think transformation functions would not allow to do that as they are
>> lazily called you need to use Actions LIke collect or foreach method
>>
>
>> On Sun, Dec 28, 2014 at 1:54 AM, Amit Behera <am...@gmail.com>
>> wrote:
>>
>>> Hi All,
>>>
>>> I want to check an item is present or not in a RDD of Iterable[Int]
>>> using scala
>>>
>>> something like in java we do :
>>>
>>> *list.contains(item)*
>>>
>>> and the statement returns true if the item is present otherwise false.
>>>
>>> Please help me to find the solution.
>>>
>>> Thanks
>>> Amit
>>>
>>>
>>
Re: unable to check whether an item is present in RDD
Posted by Amit Behera <am...@gmail.com>.
Hi Nicholas,
The RDD contains only one Iterable[Int].
Pankaj,
I used *collect* and I am getting as *items: Array[Iterable[Int]].*
Then I did like :
*val check = items.take(1).contains(item)*
I am getting *check: Boolean = false, *but the item is present.
Thanks
Amit
On Sun, Dec 28, 2014 at 8:56 AM, Pankaj <pa...@gmail.com> wrote:
> Amit, I think you can use collect method to change RDD to Array so you can
> use array methods for the same.
>
> I think transformation functions would not allow to do that as they are
> lazily called you need to use Actions LIke collect or foreach method
>
> On Sun, Dec 28, 2014 at 1:54 AM, Amit Behera <am...@gmail.com> wrote:
>
>> Hi All,
>>
>> I want to check an item is present or not in a RDD of Iterable[Int] using
>> scala
>>
>> something like in java we do :
>>
>> *list.contains(item)*
>>
>> and the statement returns true if the item is present otherwise false.
>>
>> Please help me to find the solution.
>>
>> Thanks
>> Amit
>>
>>
>