[jira] [Created] (SPARK-25147) GroupedData.apply pandas_udf crashing

["Mike Sukmanowsky (JIRA)" <ji...@apache.org>]

Mike Sukmanowsky created SPARK-25147:
----------------------------------------

             Summary: GroupedData.apply pandas_udf crashing
                 Key: SPARK-25147
                 URL: https://issues.apache.org/jira/browse/SPARK-25147
             Project: Spark
          Issue Type: Bug
          Components: PySpark
    Affects Versions: 2.3.1
         Environment: OS: Mac OS 10.13.6

Python: 2.7.15, 3.6.6

PyArrow: 0.10.0

Pandas: 0.23.4

Numpy: 1.15.0
            Reporter: Mike Sukmanowsky


Running the following example taken straight from the docs results in {{org.apache.spark.SparkException: Python worker exited unexpectedly (crashed)}} for reasons that aren't clear from any logs I can see:

{code:java}
from pyspark.sql import SparkSession
from pyspark.sql import functions as F


spark = (
    SparkSession
    .builder
    .appName("pandas_udf")
    .getOrCreate()
)

df = spark.createDataFrame(
    [(1, 1.0), (1, 2.0), (2, 3.0), (2, 5.0), (2, 10.0)],
    ("id", "v")
)

@F.pandas_udf("id long, v double", F.PandasUDFType.GROUPED_MAP)
def normalize(pdf):
    v = pdf.v
    return pdf.assign(v=(v - v.mean()) / v.std())

(
    df
    .groupby("id")
    .apply(normalize)
    .show()
)
{code}
 See output.log for [stacktrace|https://gist.github.com/msukmanowsky/b9cb6700e8ccaf93f265962000403f28]. 

 

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org