[jira] [Created] (SPARK-46873) PySpark spark.streams should not recreate new StreamingQueryManager

["Wei Liu (Jira)" <ji...@apache.org>]

Wei Liu created SPARK-46873:
-------------------------------

             Summary: PySpark spark.streams should not recreate new StreamingQueryManager
                 Key: SPARK-46873
                 URL: https://issues.apache.org/jira/browse/SPARK-46873
             Project: Spark
          Issue Type: Task
          Components: Connect, PySpark, SS
    Affects Versions: 4.0.0
            Reporter: Wei Liu


In Scala, there is only one streaming query manager for one spark session:

```

scala> spark.streams

val *res0*: *org.apache.spark.sql.streaming.StreamingQueryManager* = org.apache.spark.sql.streaming.StreamingQueryManager@46bb8cba

 

scala> spark.streams

val *res1*: *org.apache.spark.sql.streaming.StreamingQueryManager* = org.apache.spark.sql.streaming.StreamingQueryManager@46bb8cba

 

scala> spark.streams

val *res2*: *org.apache.spark.sql.streaming.StreamingQueryManager* = org.apache.spark.sql.streaming.StreamingQueryManager@46bb8cba

 

scala> spark.streams

val *res3*: *org.apache.spark.sql.streaming.StreamingQueryManager* = org.apache.spark.sql.streaming.StreamingQueryManager@46bb8cba

```

 

In Python, this is currently false:

```

>>> spark.streams

<pyspark.sql.connect.streaming.query.StreamingQueryManager object at 0x1011f7c10>

>>> spark.streams

<pyspark.sql.connect.streaming.query.StreamingQueryManager object at 0x1011f71f0>

>>> spark.streams

<pyspark.sql.connect.streaming.query.StreamingQueryManager object at 0x1011f7be0>

>>> spark.streams

<pyspark.sql.connect.streaming.query.StreamingQueryManager object at 0x1011f7c40>

```

 

Python should align scala behavior. 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org