You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Juliusz Sompolski (Jira)" <ji...@apache.org> on 2023/08/08 23:51:00 UTC

[jira] [Created] (SPARK-44730) Spark Connect: Cleaner thread not stopped when SparkSession stops

Juliusz Sompolski created SPARK-44730:
-----------------------------------------

             Summary: Spark Connect: Cleaner thread not stopped when SparkSession stops
                 Key: SPARK-44730
                 URL: https://issues.apache.org/jira/browse/SPARK-44730
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: 3.5.0, 4.0.0
            Reporter: Juliusz Sompolski


Spark Connect scala client SparkSession has a cleaner, which starts a daemon thread to clean up Closeable objects after GC. This daemon thread is never stopped, and every SparkSession creates a new one.

Cleaner implements a stop() function, but no-one ever calls it. Possibly because even after SparkSession.stop(), the cleaner may still be needed when remaining references are GCed... For this reason it seems that the Cleaner should rather be a global singleton than within a session.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org