[jira] [Commented] (SPARK-37449) Side effects between PySpark Pandas UDF and Numpy indexing

["Hyukjin Kwon (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-37449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17448321#comment-17448321 ] 

Hyukjin Kwon commented on SPARK-37449:
--------------------------------------

{{applyInPandas}} does not maintain its index. Each pandas DataFrame in {{udf_example}} is a chunk of PySpark DataFrame that has a (default) index starting from one.

So, in PySpark, you would have an output of 0 ~ 3 for each chunk.

> Side effects between PySpark Pandas UDF and Numpy indexing
> ----------------------------------------------------------
>
>                 Key: SPARK-37449
>                 URL: https://issues.apache.org/jira/browse/SPARK-37449
>             Project: Spark
>          Issue Type: Bug
>          Components: PySpark
>    Affects Versions: 3.1.2
>            Reporter: Carlos Gameiro
>            Priority: Major
>              Labels: NumPy, Pandas, Pygeos, UDF, applyInPandas
>
> Let's create a simple Pandas Dataframe with a single column named 'id' that contains a sequential range.
> {code:java}
> df = pd.DataFrame(np.arange(0,1000), columns=['id']){code}
> Consider this function that selects the first 4 indexes of the 'id' column of an array.
> {code:java}
> def udf_example(df):
>   
>   some_index = np.array([0, 1, 2, 3])
>   values = df['id'].values[some_index]
>   
>   df = pd.DataFrame(values, columns=['id'])
>   return df{code}
> If I apply this function in Pyspark I get this result:
> {code:java}
> schema = t.StructType([t.StructField('id', t.LongType(), True)])
> df_spark = spark.createDataFrame(df).groupBy().applyInPandas(udf_example, schema)
> display(df_spark)
> # id
> # 125
> # 126
> # 127
> # 128
> {code}
> If I apply it in Python I get the correct and expected result:
> {code:java}
> udf_example(df)
> # id
> # 0
> # 1
> # 2
> # 3
> {code}
> Using NumPy indexing operations inside a Pandas UDF in Spark causes side effects and unexpected results.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org