[GitHub] [spark] li36909 commented on pull request #30248: [SPARK-33339][PYTHON] Pyspark application maybe hangup because of wor…

[GitBox <gi...@apache.org>]

li36909 commented on pull request #30248:
URL: https://github.com/apache/spark/pull/30248#issuecomment-722168240


   > Just curious. Why would you run `system.exit`? You should better just throw a proper exception. Calling `system.exit` inside the UDFs wouldn't terminate user applications as intended.
   
   Below is the case I encountered, and it's cause by the code had a method named 'testUdf' and a udf obj also named 'testUdf', then worker exit. And I think the application should not hangup without any information anyway
   ```
   import pyspark.sql.functions as F
   from pyspark.sql.types import *
   from pyspark.sql import SparkSession
   
   def testUdf(x):
       return x+1
   
   spark = SparkSession.builder.appName("test").getOrCreate()
   testUdf=F.udf(lambda x: testUdf(x),DoubleType())
   
   df = spark.sql("select float(2.145025959889993) as a")
   df.withColumn("test", testUdf(df.a)).show()
   ```


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org