[jira] [Created] (SPARK-22371) dag-scheduler-event-loop thread stopped with error Attempted to access garbage collected accumulator 5605982

["Mayank Agarwal (JIRA)" <ji...@apache.org>]

Mayank Agarwal created SPARK-22371:
--------------------------------------

             Summary: dag-scheduler-event-loop thread stopped with error  Attempted to access garbage collected accumulator 5605982
                 Key: SPARK-22371
                 URL: https://issues.apache.org/jira/browse/SPARK-22371
             Project: Spark
          Issue Type: Bug
          Components: Spark Core
    Affects Versions: 2.1.0
            Reporter: Mayank Agarwal


Our Spark Jobs are getting stuck on DagScheduler.runJob as dagscheduler thread is stopped because of *Attempted to access garbage collected accumulator 5605982*.

from our investigation it look like accumulator is cleaned by GC first and same accumulator is used for merging the results from executor on task completion event.


As the error java.lang.IllegalAccessError is LinkageError which is treated as FatalError so dag-scheduler loop is finished with below exception.

---ERROR stack trace --
Exception in thread "dag-scheduler-event-loop" java.lang.IllegalAccessError: Attempted to access garbage collected accumulator 5605982
	at org.apache.spark.util.AccumulatorContext$$anonfun$get$1.apply(AccumulatorV2.scala:253)
	at org.apache.spark.util.AccumulatorContext$$anonfun$get$1.apply(AccumulatorV2.scala:249)
	at scala.Option.map(Option.scala:146)
	at org.apache.spark.util.AccumulatorContext$.get(AccumulatorV2.scala:249)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$updateAccumulators$1.apply(DAGScheduler.scala:1083)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$updateAccumulators$1.apply(DAGScheduler.scala:1080)
	at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
	at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
	at org.apache.spark.scheduler.DAGScheduler.updateAccumulators(DAGScheduler.scala:1080)
	at org.apache.spark.scheduler.DAGScheduler.handleTaskCompletion(DAGScheduler.scala:1183)
	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1647)
	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1605)
	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1594)
at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)

I am attaching the thread dump of driver as well 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org