You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Abhishek Shivkumar <ab...@bigdatapartnership.com> on 2015/12/04 13:19:43 UTC
Is it possible to pass additional parameters to a python function
when used inside RDD.filter method?
Hi,
I am using spark with python and I have a filter constraint as follows:
|my_rdd.filter(my_func)|
where my_func is a method I wrote to filter the rdd items based on my
own logic. I have defined the my_func as follows:
|def my_func(my_item):
{
...
}|
Now, I want to pass another separate parameter to my_func, besides the
item that goes into it. How can I do that? I know my_item will refer to
one item that comes from my_rdd and how can I pass my own parameter
(let's say my_param) as an additional parameter to my_func?
Thanks
Abhishek S
--
*NOTICE AND DISCLAIMER*
This email (including attachments) is confidential. If you are not the
intended recipient, notify the sender immediately, delete this email from
your system and do not disclose or use for any purpose.
Business Address: Eagle House, 163 City Road, London, EC1V 1NR. United
Kingdom
Registered Office: Finsgate, 5-7 Cranwood Street, London, EC1V 9EE. United
Kingdom
Big Data Partnership Limited is a company registered in England & Wales
with Company No 7904824
Re: Is it possible to pass additional parameters to a python function
when used inside RDD.filter method?
Posted by Abhishek Shivkumar <ab...@bigdatapartnership.com>.
Excellent. that did work - thanks.
On 4 December 2015 at 12:35, Praveen Chundi <ma...@gmail.com> wrote:
> Passing a lambda function should work.
>
> my_rrd.filter(lambda x: myfunc(x,newparam))
>
> Best regards,
> Praveen Chundi
>
>
> On 04.12.2015 13:19, Abhishek Shivkumar wrote:
>
> Hi,
>
> I am using spark with python and I have a filter constraint as follows:
>
> my_rdd.filter(my_func)
>
> where my_func is a method I wrote to filter the rdd items based on my own
> logic. I have defined the my_func as follows:
>
> def my_func(my_item):
> {...}
>
> Now, I want to pass another separate parameter to my_func, besides the
> item that goes into it. How can I do that? I know my_item will refer to one
> item that comes from my_rdd and how can I pass my own parameter (let's say
> my_param) as an additional parameter to my_func?
>
> Thanks
> Abhishek S
>
> *NOTICE AND DISCLAIMER*
>
> This email (including attachments) is confidential. If you are not the
> intended recipient, notify the sender immediately, delete this email from
> your system and do not disclose or use for any purpose.
>
> Business Address: Eagle House, 163 City Road, London, EC1V 1NR. United
> Kingdom
> Registered Office: Finsgate, 5-7 Cranwood Street, London, EC1V 9EE. United
> Kingdom
> Big Data Partnership Limited is a company registered in England & Wales
> with Company No 7904824
>
>
>
--
*NOTICE AND DISCLAIMER*
This email (including attachments) is confidential. If you are not the
intended recipient, notify the sender immediately, delete this email from
your system and do not disclose or use for any purpose.
Business Address: Eagle House, 163 City Road, London, EC1V 1NR. United
Kingdom
Registered Office: Finsgate, 5-7 Cranwood Street, London, EC1V 9EE. United
Kingdom
Big Data Partnership Limited is a company registered in England & Wales
with Company No 7904824
Re: Is it possible to pass additional parameters to a python function
when used inside RDD.filter method?
Posted by Praveen Chundi <ma...@gmail.com>.
Passing a lambda function should work.
my_rrd.filter(lambda x: myfunc(x,newparam))
Best regards,
Praveen Chundi
On 04.12.2015 13:19, Abhishek Shivkumar wrote:
>
> Hi,
>
> I am using spark with python and I have a filter constraint as follows:
>
> |my_rdd.filter(my_func)|
>
> where my_func is a method I wrote to filter the rdd items based on my
> own logic. I have defined the my_func as follows:
>
> |defmy_func(my_item):{...}|
>
> Now, I want to pass another separate parameter to my_func, besides the
> item that goes into it. How can I do that? I know my_item will refer
> to one item that comes from my_rdd and how can I pass my own parameter
> (let's say my_param) as an additional parameter to my_func?
>
> Thanks
> Abhishek S
>
>
> *NOTICE AND DISCLAIMER*
>
> This email (including attachments) is confidential. If you are not the
> intended recipient, notify the sender immediately, delete this email
> from your system and do not disclose or use for any purpose.
>
> Business Address: Eagle House, 163 City Road, London, EC1V 1NR. United
> Kingdom
> Registered Office: Finsgate, 5-7 Cranwood Street, London, EC1V 9EE.
> United Kingdom
> Big Data Partnership Limited is a company registered in England &
> Wales with Company No 7904824
>