You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@zeppelin.apache.org by Karan Sewani <Ka...@guenstiger.de> on 2018/02/22 16:48:03 UTC

Zeppelin Spark Context finishes after one paragraph run.

Hello


I have posted this on SO as well.

https://stackoverflow.com/questions/48920607/zeppelin-spark-context-closed-after-one-paragraph


I have a notebook in Zeppelin containing multiple paragraphs which were running fine earlier; suddenly, after a cluster restart, it has started behaving weirdly.

The first paragraph runs fine while anything that runs afterwards says Connection Refused

On checking the logs in $ZEPPELIN_HOME/logs folder zeppelin-interpreter-spark-root-mn.log (where mn is machine name).


INFO [2018-02-21 21:42:43,301] ({dispatcher-event-loop-15} Logging.scala[logInfo]:54) - Removed broadcast_12_piece0 on mn5:45284 in memory (size: 88.2 KB, free: 2004.5 MB)
 INFO [2018-02-21 21:42:43,401] ({Thread-3} Logging.scala[logInfo]:54) - Invoking stop() from shutdown hook
 INFO [2018-02-21 21:42:43,412] ({Thread-3} AbstractConnector.java[doStop]:310) - Stopped Spark@7de3e842{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
 INFO [2018-02-21 21:42:43,416] ({Thread-3} Logging.scala[logInfo]:54) - Stopped Spark web UI at http://10.28.37.82:4040
 INFO [2018-02-21 21:42:43,440] ({Yarn application state monitor} Logging.scala[logInfo]:54) - Interrupting monitor thread
 INFO [2018-02-21 21:42:43,442] ({Thread-3} Logging.scala[logInfo]:54) - Shutting down all executors
 INFO [2018-02-21 21:42:43,443] ({dispatcher-event-loop-4} Logging.scala[logInfo]:54) - Asking each executor to shut down
 INFO [2018-02-21 21:42:43,447] ({Thread-3} Logging.scala[logInfo]:54) - Stopping SchedulerExtensionServices
(serviceOption=None,
 services=List(),
 started=false)
 INFO [2018-02-21 21:42:43,450] ({Thread-3} Logging.scala[logInfo]:54) - Stopped
 INFO [2018-02-21 21:42:43,454] ({dispatcher-event-loop-9} Logging.scala[logInfo]:54) - MapOutputTrackerMasterEndpoint stopped!
 INFO [2018-02-21 21:42:43,466] ({Thread-3} Logging.scala[logInfo]:54) - MemoryStore cleared
 INFO [2018-02-21 21:42:43,466] ({Thread-3} Logging.scala[logInfo]:54) - BlockManager stopped
 INFO [2018-02-21 21:42:43,467] ({Thread-3} Logging.scala[logInfo]:54) - BlockManagerMaster stopped
 INFO [2018-02-21 21:42:43,471] ({dispatcher-event-loop-0} Logging.scala[logInfo]:54) - OutputCommitCoordinator stopped!
 INFO [2018-02-21 21:42:43,472] ({Thread-3} Logging.scala[logInfo]:54) - Successfully stopped SparkContext
 INFO [2018-02-21 21:42:43,473] ({Thread-3} Logging.scala[logInfo]:54) - Shutdown hook called

So the shut down hook is getting called. I have tried to check other posts on SO (like this<https://stackoverflow.com/questions/40993104/spark-workers-stopped-after-driver-commanded-a-shutdown> and this<https://stackoverflow.com/questions/28967500/why-spark-application-fail-with-executor-coarsegrainedexecutorbackend-driver-d>) but it didn't help. Logs are not much helpful either.

Do I need to tweak code to add additional logging to fix this problem? Has someone has already faced and resolved the same?

Best

Karan