How to access a RDD (that has been broadcasted) inside the filter method of another RDD?

[Abhishek Shivkumar <ab...@bigdatapartnership.com>]

Hi,

 I have RDD1 that is broadcasted.

I have a user defined method for the filter functionality of RDD2, written
as follows:

RDD2.filter(my_func)


I want to access the values of RDD1 inside my_func. Is that possible?
Should I pass RDD1 as a parameter into my_func?

Thanks
Abhishek S

-- 
 

*NOTICE AND DISCLAIMER*

This email (including attachments) is confidential. If you are not the 
intended recipient, notify the sender immediately, delete this email from 
your system and do not disclose or use for any purpose.

Business Address: Eagle House, 163 City Road, London, EC1V 1NR. United 
Kingdom
Registered Office: Finsgate, 5-7 Cranwood Street, London, EC1V 9EE. United 
Kingdom
Big Data Partnership Limited is a company registered in England & Wales 
with Company No 7904824

Re: How to access a RDD (that has been broadcasted) inside the filter method of another RDD?

[Sean Owen <so...@cloudera.com>]

You can't broadcast an RDD to begin with, and can't use RDDs inside
RDDs. They are really driver-side concepts.

Yes that's how you'd use a broadcast of anything else though, though
you need to reference ".value" on the broadcast. The 'if' is redundant
in that example, and if it's a map- or collection-like structure, you
don't even need the arg.

RDD2.filter(broadcasted.value.contains)

On Mon, Dec 7, 2015 at 2:43 PM, Akhil Das <ak...@sigmoidanalytics.com> wrote:
> Something like this?
>
> val broadcasted = sc.broadcast(...)
>
> RDD2.filter(value => {
>
> //simply use broadcasted
> if(broadcasted.contains(value)) true
>
> })
>
>
>
> Thanks
> Best Regards
>
> On Fri, Dec 4, 2015 at 10:43 PM, Abhishek Shivkumar
> <ab...@bigdatapartnership.com> wrote:
>>
>> Hi,
>>
>>  I have RDD1 that is broadcasted.
>>
>> I have a user defined method for the filter functionality of RDD2, written
>> as follows:
>>
>> RDD2.filter(my_func)
>>
>>
>> I want to access the values of RDD1 inside my_func. Is that possible?
>> Should I pass RDD1 as a parameter into my_func?
>>
>> Thanks
>> Abhishek S
>>
>> NOTICE AND DISCLAIMER
>>
>> This email (including attachments) is confidential. If you are not the
>> intended recipient, notify the sender immediately, delete this email from
>> your system and do not disclose or use for any purpose.
>>
>> Business Address: Eagle House, 163 City Road, London, EC1V 1NR. United
>> Kingdom
>> Registered Office: Finsgate, 5-7 Cranwood Street, London, EC1V 9EE. United
>> Kingdom
>> Big Data Partnership Limited is a company registered in England & Wales
>> with Company No 7904824
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org

Re: How to access a RDD (that has been broadcasted) inside the filter method of another RDD?

[Akhil Das <ak...@sigmoidanalytics.com>]

Something like this?

val broadcasted = sc.broadcast(...)

RDD2.filter(value => {

//simply use *broadcasted*
if(broadcasted.contains(value)) true

})



Thanks
Best Regards

On Fri, Dec 4, 2015 at 10:43 PM, Abhishek Shivkumar <
abhishek.shivkumar@bigdatapartnership.com> wrote:

> Hi,
>
>  I have RDD1 that is broadcasted.
>
> I have a user defined method for the filter functionality of RDD2, written
> as follows:
>
> RDD2.filter(my_func)
>
>
> I want to access the values of RDD1 inside my_func. Is that possible?
> Should I pass RDD1 as a parameter into my_func?
>
> Thanks
> Abhishek S
>
> *NOTICE AND DISCLAIMER*
>
> This email (including attachments) is confidential. If you are not the
> intended recipient, notify the sender immediately, delete this email from
> your system and do not disclose or use for any purpose.
>
> Business Address: Eagle House, 163 City Road, London, EC1V 1NR. United
> Kingdom
> Registered Office: Finsgate, 5-7 Cranwood Street, London, EC1V 9EE. United
> Kingdom
> Big Data Partnership Limited is a company registered in England & Wales
> with Company No 7904824
>