[jira] [Created] (SPARK-21009) SparkListenerTaskEnd.taskInfo.accumulables might not be accurate

["Bogdan Raducanu (JIRA)" <ji...@apache.org>]

Bogdan Raducanu created SPARK-21009:
---------------------------------------

             Summary: SparkListenerTaskEnd.taskInfo.accumulables might not be accurate
                 Key: SPARK-21009
                 URL: https://issues.apache.org/jira/browse/SPARK-21009
             Project: Spark
          Issue Type: Bug
          Components: Spark Core
    Affects Versions: 2.2.0
            Reporter: Bogdan Raducanu


The following code reproduces it:
{code}
  test("test") {
    val foundMetrics = mutable.Set.empty[String]
    spark.sparkContext.addSparkListener(new SparkListener {
      override def onTaskEnd(taskEnd: SparkListenerTaskEnd): Unit = {
        taskEnd.taskInfo.accumulables.foreach { a =>
          if (a.name.isDefined) {
            foundMetrics.add(a.name.get)
          }
        }
      }
    })
    for (iter <- 0 until 100) {
      foundMetrics.clear()
      println(s"iter = $iter")
      spark.range(10).groupBy().agg("id" -> "sum").collect
      spark.sparkContext.listenerBus.waitUntilEmpty(3000)
      assert(foundMetrics.size > 0)
    }
  }
{code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org