[GitHub] [spark] HyukjinKwon commented on a change in pull request #32989: [SPARK-35834][PYTHON] Use the same cleanup logic as Py4J in inheritable thread API

[GitBox <gi...@apache.org>]

HyukjinKwon commented on a change in pull request #32989:
URL: https://github.com/apache/spark/pull/32989#discussion_r655020331



##########
File path: python/pyspark/util.py
##########
@@ -378,20 +378,18 @@ def _clean_py4j_conn_for_current_thread():
         from pyspark import SparkContext
 
         jvm = SparkContext._jvm
-        thread_connection = jvm._gateway_client.thread_connection.connection()
+        thread_connection = jvm._gateway_client.get_thread_connection()
         if thread_connection is not None:
-            connections = jvm._gateway_client.deque
-            # Reuse the lock for Py4J in PySpark
-            with SparkContext._lock:
-                for i in range(len(connections)):
-                    if connections[i] is thread_connection:
-                        connections[i].close()
-                        del connections[i]
-                        break
-                else:
-                    # Just in case the connection was not closed but removed from the
-                    # queue.
-                    thread_connection.close()
+            try:
+                # Dequeue is shared across other threads but it's thread-safe.
+                # If this function has to be invoked one more time in the same thead
+                # Py4J will create a new connection automatically.
+                jvm._gateway_client.deque.remove(thread_connection)
+            except ValueError:
+                # Should never reach this point

Review comment:
       Because the removal on the deque on the specific thread local will only happen in the same thread.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org