[jira] [Created] (SPARK-30002) Reuse SparkSession in pyspark via Gateway

["Kaushal Prajapati (Jira)" <ji...@apache.org>]

Kaushal Prajapati created SPARK-30002:
-----------------------------------------

             Summary: Reuse SparkSession in pyspark via Gateway
                 Key: SPARK-30002
                 URL: https://issues.apache.org/jira/browse/SPARK-30002
             Project: Spark
          Issue Type: Improvement
          Components: PySpark
    Affects Versions: 2.4.4, 2.4.0
            Reporter: Kaushal Prajapati


In PySpark, we create SparkContext via user spark configurations or the default ones, and it gets launched through py4j gateway internally.

Let's say if I have launched py4j gateway from another application then to communicate with the same py4j gateway I have to set below configuration:-

 
{code:java}
export PYSPARK_GATEWAY_PORT=12345
export PYSPARK_GATEWAY_SECRET=***********************
{code}
 

So when PySpark tries to create its own SparkContext after the communication has been set up, it doesn't check whether there is any available SparkContext in the same JVM.

Current code snippet:-

 
{code:java}
def _initialize_context(self, jconf):
    """
    Initialize SparkContext in function to allow subclass specific initialization
    """
    return self._jvm.JavaSparkContext(jconf){code}
 

 

If we change it to the following, it works fine for me.
{code:java}
def _initialize_context(self, jconf):
    """
    Initialize SparkContext in function to allow subclass specific initialization
    """
    return self._jvm.JavaSparkContext(self._jvm.org.apache.spark.SparkContext.getOrCreate(jconf)){code}
 

It looks like a good use case for improvement.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org