Spark 2.0.0 Error Caused by: java.lang.IllegalArgumentException: requirement failed: Block broadcast_21_piece0 is already present in the MemoryStore

[sandesh deshmane <sa...@gmail.com>]

I am getting this error some times when I run pyspark with spark 2.0.0

App > at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
App > at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
App > at java.lang.reflect.Method.invoke(Method.java:497)
App > at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:237)
App > at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
App > at py4j.Gateway.invoke(Gateway.java:280)
App > at
py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:128)
App > at py4j.commands.CallCommand.execute(CallCommand.java:79)
App > at py4j.GatewayConnection.run(GatewayConnection.java:211)
App > at java.lang.Thread.run(Thread.java:745)
App > Caused by: java.io.IOException: java.lang.IllegalArgumentException:
requirement failed: Block broadcast_21_piece0 is already present in the
MemoryStore
App > at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1284)
App > at
org.apache.spark.broadcast.TorrentBroadcast.readBroadcastBlock(TorrentBroadcast.scala:174)
App > at
org.apache.spark.broadcast.TorrentBroadcast._value$lzycompute(TorrentBroadcast.scala:65)
App > at
org.apache.spark.broadcast.TorrentBroadcast._value(TorrentBroadcast.scala:65)
App > at
org.apache.spark.broadcast.TorrentBroadcast.getValue(TorrentBroadcast.scala:89)
App > at org.apache.spark.broadcast.Broadcast.value(Broadcast.scala:70)
App > at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:67)
App > at org.apache.spark.scheduler.Task.run(Task.scala:85)
App > at
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)
App > at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
App > at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
App > ... 1 more
App > *Caused by: java.lang.IllegalArgumentException: requirement failed:
Block broadcast_21_piece0 is already present in the MemoryStore*
App > at scala.Predef$.require(Predef.scala:224)
App > at
org.apache.spark.storage.memory.MemoryStore.putBytes(MemoryStore.scala:144)
App > at
org.apache.spark.storage.BlockManager$$anonfun$doPutBytes$1.apply(BlockManager.scala:792)
App > at
org.apache.spark.storage.BlockManager$$anonfun$doPutBytes$1.apply(BlockManager.scala:761)
App > at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:866)
App > at
org.apache.spark.storage.BlockManager.doPutBytes(BlockManager.scala:761)
App > at
org.apache.spark.storage.BlockManager.putBytes(BlockManager.scala:740)
App > at
org.apache.spark.broadcast.TorrentBroadcast$$anonfun$org$apache$spark$broadcast$TorrentBroadcast$$readBlocks$1.apply$mcVI$sp(TorrentBroadcast.scala:140)
App > at
org.apache.spark.broadcast.TorrentBroadcast$$anonfun$org$apache$spark$broadcast$TorrentBroadcast$$readBlocks$1.apply(TorrentBroadcast.scala:125)
App > at
org.apache.spark.broadcast.TorrentBroadcast$$anonfun$org$apache$spark$broadcast$TorrentBroadcast$$readBlocks$1.apply(TorrentBroadcast.scala:125)
App > at scala.collection.immutable.List.foreach(List.scala:381)
App > at org.apache.spark.broadcast.TorrentBroadcast.org
$apache$spark$broadcast$TorrentBroadcast$$readBlocks(TorrentBroadcast.scala:125)
App > at
org.apache.spark.broadcast.TorrentBroadcast$$anonfun$readBroadcastBlock$1.apply(TorrentBroadcast.scala:186)
App > at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1277)
App > ... 11 more
App >
App > 16/10/10 20:01:59 main INFO SparkContext: sc.stop called from
[SparkSubmit.exceptionExitHook[failure]]
App > 16/10/10 20:01:59 main INFO Autoscaler: Shutting down Autoscaler!!
App > 16/10/10 20:01:59 main INFO ServerConnector: Stopped
ServerConnector@1797a20
{HTTP/1.1} {0.0.0.0:4040}

Thanks
Sandesh