You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Ahmed Kamal` (JIRA)" <ji...@apache.org> on 2018/12/18 11:10:00 UTC
[jira] [Updated] (SPARK-26393) Different behaviors of date_add when
calling it inside expr
[ https://issues.apache.org/jira/browse/SPARK-26393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ahmed Kamal` updated SPARK-26393:
---------------------------------
Description:
When Calling date_add from pyspark.sql.functions directly without using expr, like this :
{code:java}
df.withColumn("added", F.date_add(F.to_date(F.lit('1998-9-26')), F.col('days'))).toPandas(){code}
It will raise Error : `TypeError: Column is not iterable`
because it only taking a number not a column
but when i try to use it inside an expr, like this :
{code:java}
df.withColumn("added", F.expr("date_add(to_date('1998-9-26'), days)")).toPandas(){code}
It will work fine.
Shouldn't it behave the same way ?
and i think its logical to accept a column here as well.
A python Notebook to demonstrate :
[https://gist.github.com/AhmedKamal20/fec10337e815baa44f115d307e3b07eb]
was:
When Calling date_add from pyspark.sql.functions directly without using expr, like this :
{code:java}
df.withColumn("added", F.date_add(F.to_date(F.lit('1998-9-26')), F.col('days'))).toPandas(){code}
It will raise Error : `TypeError: Column is not iterable`
because it only taking a number not a column
but when i try to use it inside an expr, like this :
{code:java}
df.withColumn("added", F.expr("date_add(to_date('1998-9-26'), days)")).toPandas()
{code}
it will work fine.
Shouldn't it behave the same way ?
and i thin its logical to accept a column here as well.
A python Notebook to demonstrate :
https://gist.github.com/AhmedKamal20/fec10337e815baa44f115d307e3b07eb
> Different behaviors of date_add when calling it inside expr
> -----------------------------------------------------------
>
> Key: SPARK-26393
> URL: https://issues.apache.org/jira/browse/SPARK-26393
> Project: Spark
> Issue Type: Bug
> Components: PySpark
> Affects Versions: 2.3.2
> Reporter: Ahmed Kamal`
> Priority: Minor
>
> When Calling date_add from pyspark.sql.functions directly without using expr, like this :
> {code:java}
> df.withColumn("added", F.date_add(F.to_date(F.lit('1998-9-26')), F.col('days'))).toPandas(){code}
> It will raise Error : `TypeError: Column is not iterable`
> because it only taking a number not a column
> but when i try to use it inside an expr, like this :
> {code:java}
> df.withColumn("added", F.expr("date_add(to_date('1998-9-26'), days)")).toPandas(){code}
> It will work fine.
> Shouldn't it behave the same way ?
> and i think its logical to accept a column here as well.
> A python Notebook to demonstrate :
> [https://gist.github.com/AhmedKamal20/fec10337e815baa44f115d307e3b07eb]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org