[jira] [Created] (SPARK-27427) cannot change list of packages from PYSPARK_SUBMIT_ARGS after restarting context

["Natalino Busa (JIRA)" <ji...@apache.org>]

Natalino Busa created SPARK-27427:
-------------------------------------

             Summary: cannot change list of packages from PYSPARK_SUBMIT_ARGS after restarting context
                 Key: SPARK-27427
                 URL: https://issues.apache.org/jira/browse/SPARK-27427
             Project: Spark
          Issue Type: Bug
          Components: PySpark
    Affects Versions: 2.4.0
            Reporter: Natalino Busa


Upon creating the gateway the env variable PYSPARK_SUBMIT_ARGS , if packages are provided the jars will be downloaded and added to the list of files to distribute to the cluster.

However this mechanism only works once because the gateway is kept, when the SparkContext is stopped and created again. Possible root cause the gateway is only created once.

This more aggressive SparkSession stop forces the gateway to be recreated
```
    def stop(spark_session=None):
        try:
            sc = None
            if spark_session:
                sc = spark_session.sparkContext
                spark_session.stop()
                
            cls = pyspark.SparkContext
            sc = sc or cls._active_spark_context
            
            if sc:
                sc.stop()
                sc._gateway.shutdown()
                
            cls._active_spark_context = None
            cls._gateway = None
            cls._jvm = None
        except Exception as e:
            print(e)
            logging.warning('Could not fully stop the engine context')
```



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org