[jira] [Commented] (SPARK-4514) ComplexFutureAction does not preserve job group IDs

["Josh Rosen (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-4514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14220359#comment-14220359 ] 

Josh Rosen commented on SPARK-4514:
-----------------------------------

I plan to investigate / fix this soon, but I'm a bit swamped with other work; [~eje], any chance that you have spare cycles to hack on this?

> ComplexFutureAction does not preserve job group IDs
> ---------------------------------------------------
>
>                 Key: SPARK-4514
>                 URL: https://issues.apache.org/jira/browse/SPARK-4514
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Core
>    Affects Versions: 1.1.0, 1.1.1, 1.2.0
>            Reporter: Erik Erlandson
>            Assignee: Josh Rosen
>            Priority: Critical
>
> The current job group id of a Spark context is stored in the {{localProperties}} member value.   This data structure is designed to be thread local, and its settings are not preserved when {{ComplexFutureAction}} instantiates a new {{Future}}.  
> One consequence of this is that {{takeAsync()}} does not behave in the same way as other async actions, e.g. {{countAsync()}}.  For example, this test (if copied into StatusTrackerSuite.scala), will fail, because {{"my-job-group2"}} is not propagated to the Future which actually instantiates the job:
> {code:java}
>   test("getJobIdsForGroup() with takeAsync()") {
>     sc = new SparkContext("local", "test", new SparkConf(false))
>     sc.setJobGroup("my-job-group2", "description")
>     sc.statusTracker.getJobIdsForGroup("my-job-group2") should be (Seq.empty)
>     val firstJobFuture = sc.parallelize(1 to 1000, 1).takeAsync(1)
>     val firstJobId = eventually(timeout(10 seconds)) {
>       firstJobFuture.jobIds.head
>     }
>     eventually(timeout(10 seconds)) {
>       sc.statusTracker.getJobIdsForGroup("my-job-group2") should be (Seq(firstJobId))
>     }
>   }
> {code}
> It also impacts current PR for SPARK-1021, which involves additional uses of {{ComplexFutureAction}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org