You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Karlson <ks...@siberie.de> on 2015/03/25 13:42:32 UTC
functools.partial as UserDefinedFunction
Hi all,
passing a functools.partial-function as a UserDefinedFunction to
DataFrame.select raises an AttributeException, because functools.partial
does not have the attribute __name__. Is there any alternative to
relying on __name__ in pyspark/sql/functions.py:126 ?
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: functools.partial as UserDefinedFunction
Posted by Karlson <ks...@siberie.de>.
Hi,
I've filed a JIRA (https://issues.apache.org/jira/browse/SPARK-6553) and
suggested a fix (https://github.com/apache/spark/pull/5206).
On 2015-03-25 19:49, Davies Liu wrote:
> It’s good to support functools.partial, could you file a JIRA for it?
>
>
> On Wednesday, March 25, 2015 at 5:42 AM, Karlson wrote:
>
>>
>> Hi all,
>>
>> passing a functools.partial-function as a UserDefinedFunction to
>> DataFrame.select raises an AttributeException, because
>> functools.partial
>> does not have the attribute __name__. Is there any alternative to
>> relying on __name__ in pyspark/sql/functions.py:126 ?
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
>> (mailto:dev-unsubscribe@spark.apache.org)
>> For additional commands, e-mail: dev-help@spark.apache.org
>> (mailto:dev-help@spark.apache.org)
>>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: functools.partial as UserDefinedFunction
Posted by Davies Liu <da...@databricks.com>.
It’s good to support functools.partial, could you file a JIRA for it?
On Wednesday, March 25, 2015 at 5:42 AM, Karlson wrote:
>
> Hi all,
>
> passing a functools.partial-function as a UserDefinedFunction to
> DataFrame.select raises an AttributeException, because functools.partial
> does not have the attribute __name__. Is there any alternative to
> relying on __name__ in pyspark/sql/functions.py:126 ?
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org (mailto:dev-unsubscribe@spark.apache.org)
> For additional commands, e-mail: dev-help@spark.apache.org (mailto:dev-help@spark.apache.org)
>
>