You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Ron Serruya (Jira)" <ji...@apache.org> on 2024/06/06 14:17:00 UTC
[jira] [Commented] (SPARK-48555) Support Column type for several SQL functions in scala and python
[ https://issues.apache.org/jira/browse/SPARK-48555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17852806#comment-17852806 ]
Ron Serruya commented on SPARK-48555:
-------------------------------------
I think I don't have permission to assign this issue to myself, but I did already fix it, so I'll open a PR soon
> Support Column type for several SQL functions in scala and python
> -----------------------------------------------------------------
>
> Key: SPARK-48555
> URL: https://issues.apache.org/jira/browse/SPARK-48555
> Project: Spark
> Issue Type: New Feature
> Components: Connect, PySpark, Spark Core
> Affects Versions: 3.5.1
> Reporter: Ron Serruya
> Priority: Minor
>
> Currently, several SQL functions accept both native types and Columns, but only accept native types in their scala/python APIs:
> * array_remove (works in SQL, scala, not in python)
> * array_position(works in SQL, scala, not in python)
> * map_contains_key (works in SQL, scala, not in python)
> * substring (works only in SQL)
> For example, this is possible in SQL:
> {code:python}
> spark.sql("select array_remove(col1, col2) from values(array(1,2,3), 2)")
> {code}
> But not in python:
> {code:python}
> df.select(F.array_remove(F.col("col1"), F.col("col2"))
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org