[jira] [Created] (SPARK-33041) Better error messages when PySpark Java Gateway Fails to Start or Crashes

["Russell Spitzer (Jira)" <ji...@apache.org>]

Russell Spitzer created SPARK-33041:
---------------------------------------

             Summary: Better error messages when PySpark Java Gateway Fails to Start or Crashes
                 Key: SPARK-33041
                 URL: https://issues.apache.org/jira/browse/SPARK-33041
             Project: Spark
          Issue Type: Improvement
          Components: PySpark
    Affects Versions: 2.4.7
            Reporter: Russell Spitzer


Currently the startup works by opening the Gateway process and waiting until the the process has written the conn_info_file. Once the conn_file is written it proceeds to attempt to connect to the port.

This connection can succeed and the process can start normally, but if the gateway process dies or is killed the error that the user ends up getting is a confusing "connection_failed" style error like

{code}
Traceback (most recent call last):
  File "/usr/lib/spark-packages/spark2.4.4/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 929, in _get_connection
    connection = self.deque.pop()
IndexError: pop from an empty deque
{code}

Since we have a handle on the py4j process, we should probably check whether it has terminated before surfacing any exceptions like this. 

CC [~holden]




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org