[jira] [Commented] (SPARK-25286) Remove dangerous parmap

["Apache Spark (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-25286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16597921#comment-16597921 ] 

Apache Spark commented on SPARK-25286:
--------------------------------------

User 'MaxGekk' has created a pull request for this issue:
https://github.com/apache/spark/pull/22292

> Remove dangerous parmap
> -----------------------
>
>                 Key: SPARK-25286
>                 URL: https://issues.apache.org/jira/browse/SPARK-25286
>             Project: Spark
>          Issue Type: Improvement
>          Components: Spark Core
>    Affects Versions: 2.4.0
>            Reporter: Maxim Gekk
>            Priority: Major
>
> One of parmap methods accepts an execution context created outside of parmap. If the parmap method is called recursively on a thread pool limited by size, it could lead to deadlocks. See the JIRA tickets: SPARK-25240 and SPARK-25283 . To eliminate the problems in the future, need to remove parmap() with the signature:
> {code:scala}
> def parmap[I, O, Col[X] <: TraversableLike[X, Col[X]]]
>       (in: Col[I])
>       (f: I => O)
>       (implicit
>         cbf: CanBuildFrom[Col[I], Future[O], Col[Future[O]]], // For in.map
>         cbf2: CanBuildFrom[Col[Future[O]], O, Col[O]], // for Future.sequence
>         ec: ExecutionContext
>       ): Col[O]
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org