[jira] [Created] (SPARK-31915) Remove projection that adds grouping keys in grouped and cogrouped pandas UDFs

["Hyukjin Kwon (Jira)" <ji...@apache.org>]

Hyukjin Kwon created SPARK-31915:
------------------------------------

             Summary: Remove projection that adds grouping keys in grouped and cogrouped pandas UDFs
                 Key: SPARK-31915
                 URL: https://issues.apache.org/jira/browse/SPARK-31915
             Project: Spark
          Issue Type: Bug
          Components: PySpark, SQL
    Affects Versions: 3.0.0
            Reporter: Hyukjin Kwon


Currently, grouped and cogrouped pandas UDFs in Spark unnecessarily projects the grouping keys. This results in case-sensitivity resolution failure when the project contains columns such as "Column" and "column" as they are considered different but ambiguous columns. 

It results as below:

{code}
from pyspark.sql.functions import *

df = spark.createDataFrame([[1, 1]], ["column", "Score"])

@pandas_udf("column integer, Score float", PandasUDFType.GROUPED_MAP)
def my_pandas_udf(pdf):
    return pdf.assign(Score=0.5)

df.groupby('COLUMN').apply(my_pandas_udf).show()
{code}

{code}
pyspark.sql.utils.AnalysisException: Reference 'COLUMN' is ambiguous, could be: COLUMN, COLUMN.;
{code}

{code}
pyspark.sql.utils.AnalysisException: cannot resolve '`COLUMN`' given input columns: [COLUMN, COLUMN, value, value];;
'FlatMapCoGroupsInPandas ['COLUMN], ['COLUMN], <lambda>(column#9L, value#10L, column#13L, value#14L), [column#22L, value#23L]
:- Project [COLUMN#9L, column#9L, value#10L]
:  +- LogicalRDD [column#9L, value#10L], false
+- Project [COLUMN#13L, column#13L, value#14L]
   +- LogicalRDD [column#13L, value#14L], false
{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org