[jira] [Commented] (SPARK-19348) pyspark.ml.Pipeline gets corrupted under multi threaded use

["Bryan Cutler (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-19348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15850705#comment-15850705 ] 

Bryan Cutler commented on SPARK-19348:
--------------------------------------

The problem here is with the @keyword_only decorator that is used in the Pipeline constructor (and every other ML class also).  It relies on saving the params, stages in this case, to a static class variable.  When multiple threads call the wrapped constructor, it becomes a race condition to read from that static variable before it is over-written by another thread.  I can put up a potential fix, but it affects all of PySpark ML so it would need to be checked out carefully.

As a workaround, you could just protect the construction of {{Pipeline}} with a shared lock.  Other calls to {{fit}} etc, should be ok since they don't use that keyword_only decorator.

> pyspark.ml.Pipeline gets corrupted under multi threaded use
> -----------------------------------------------------------
>
>                 Key: SPARK-19348
>                 URL: https://issues.apache.org/jira/browse/SPARK-19348
>             Project: Spark
>          Issue Type: Bug
>          Components: ML, PySpark
>    Affects Versions: 1.6.0, 2.0.0, 2.1.0
>            Reporter: Vinayak Joshi
>         Attachments: pyspark_pipeline_threads.py
>
>
> When pyspark.ml.Pipeline objects are constructed concurrently in separate python threads, it is observed that the stages used to construct a pipeline object get corrupted i.e the stages supplied to a Pipeline object in one thread appear inside a different Pipeline object constructed in a different thread. 
> Things work fine if construction of pyspark.ml.Pipeline objects is serialized, so this looks like a thread safety problem with pyspark.ml.Pipeline object construction. 
> Confirmed that the problem exists with Spark 1.6.x as well as 2.x.
> While the corruption of the Pipeline stages is easily caught, we need to know if performing other pipeline operations, such as pyspark.ml.pipeline.fit( ) are also affected by the underlying cause of this problem. That is, whether other pipeline operations like pyspark.ml.pipeline.fit( )  may be performed in separate threads (on distinct pipeline objects) concurrently without any cross contamination between them.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org