You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Li Jin (JIRA)" <ji...@apache.org> on 2018/05/21 20:24:00 UTC

[jira] [Created] (SPARK-24334) Race condition in ArrowPythonRunner causes unclean shutdown of Arrow memory allocator

Li Jin created SPARK-24334:
------------------------------

             Summary: Race condition in ArrowPythonRunner causes unclean shutdown of Arrow memory allocator
                 Key: SPARK-24334
                 URL: https://issues.apache.org/jira/browse/SPARK-24334
             Project: Spark
          Issue Type: Sub-task
          Components: PySpark
    Affects Versions: 2.3.0
            Reporter: Li Jin


Currently, ArrowPythonRunner has two thread that frees the Arrow vector schema root and allocator - The main writer thread and task completion listener thread. 

Having both thread doing the clean up leads to weird case (e.g., negative ref cnt, NPE, and memory leak exception) when an exceptions are thrown from the user function.

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org