You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@zeppelin.apache.org by Dave Boyd <db...@incadencecorp.com> on 2019/03/14 22:31:28 UTC
Spark issue moving from local to yarn-client
All:
I have some code that worked fine in Zeppelin 0.7.4 but I am having issues in 0.8.2 when going from spark master of local to yarn-client. Yarn client worked in 0.7.4.
When my master is set to local[*] it runs just fine. However, as soon as I switch to yarn-client I get the Cannot call methods on a stopped SparkContext error. In looking at my yarn logs everything creates fine and the job finishes without an error. The executors start just fine
from what I get out of yarn logs.
Any suggestions on where to look? This happens with any note that trys to run spark.
If I try this very simple code:
// Spark Version
spark.version
I get this error:
java.lang.IllegalStateException: Spark context stopped while waiting for backend at org.apache.spark.scheduler.TaskSchedulerImpl.waitBackendReady(TaskSchedulerImpl.scala:614) at org.apache.spark.scheduler.TaskSchedulerImpl.postStartHook(TaskSchedulerImpl.scala:169) at org.apache.spark.SparkContext.<init>(SparkContext.scala:567) at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2313) at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:868) at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:860) at scala.Option.getOrElse(Option.scala:121) at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:860) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.zeppelin.spark.BaseSparkScalaInterpreter.spark2CreateContext(BaseSparkScalaInterpreter.scala:259) at org.apache.zeppelin.spark.BaseSparkScalaInterpreter.createSparkContext(BaseSparkScalaInterpreter.scala:178) at org.apache.zeppelin.spark.SparkScala211Interpreter.open(SparkScala211Interpreter.scala:89) at org.apache.zeppelin.spark.NewSparkInterpreter.open(NewSparkInterpreter.java:102) at org.apache.zeppelin.spark.SparkInterpreter.open(SparkInterpreter.java:62) at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:69) at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:616) at org.apache.zeppelin.scheduler.Job.run(Job.java:188) at org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:140) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)
What am I missing?
--
========= mailto:dboyd@incadencecorp.com ============
David W. Boyd
VP, Data Solutions
10432 Balls Ford, Suite 240
Manassas, VA 20109
office: +1-703-552-2862
cell: +1-703-402-7908
============== http://www.incadencecorp.com/ ============
ISO/IEC JTC1 WG9, editor ISO/IEC 20547 Big Data Reference Architecture
Chair ANSI/INCITS TC Big Data
Co-chair NIST Big Data Public Working Group Reference Architecture
First Robotic Mentor - FRC, FTC - www.iliterobotics.org<http://www.iliterobotics.org>
Board Member- USSTEM Foundation - www.usstem.org<http://www.usstem.org>
The information contained in this message may be privileged
and/or confidential and protected from disclosure.
If the reader of this message is not the intended recipient
or an employee or agent responsible for delivering this message
to the intended recipient, you are hereby notified that any
dissemination, distribution or copying of this communication
is strictly prohibited. If you have received this communication
in error, please notify the sender immediately by replying to
this message and deleting the material from any computer.
Re: zeppelin url append '/#/'
Posted by Weilan Wu <we...@icloud.com>.
Hey Anthony,
Thanks for the pointer! appreciate it. however my request is the other way, so, instead of add ‘#/‘ for each zeppelin url, can we make it so that it always appends ‘/‘ for each url does NOT have trailing slash?
e.g. https://host.zeppelin.com/instance => https://host.zeppelin.com/instance/
for host only url, there is no problem, all browsers will append ‘/‘ for url with pattern: 'https://host.zeppelin.com’ (by HTTP protocol, the URI cannot be null, so ‘/‘ is automatically added).
Thanks in advance!
Cheers,
Weilan
> On Mar 22, 2019, at 3:41 AM, Anthony Corbacho <an...@apache.org> wrote:
>
> Hi weilan,
> This behavior comes from angular (we are using angular 1.5.7)
> You can find imformation here:
> https://scotch.io/tutorials/pretty-urls-in-angularjs-removing-the-hashtag <https://scotch.io/tutorials/pretty-urls-in-angularjs-removing-the-hashtag>
>
> On Fri, 22 Mar 2019 at 4:59 PM Jongyoul Lee <jongyoul@gmail.com <ma...@gmail.com>> wrote:
> It means Front End.
>
> On Fri, Mar 22, 2019 at 3:42 PM Weilan Wu <weilan.wu@icloud.com <ma...@icloud.com>> wrote:
> Hi Jongyoul,
> Could you explain what “FE” means? It’s an important question for us to solve when running Zeppelin it in Kubernetes...
>
> Thanks,
> Weilan
>
> Sent from my iPhone 6s
>
> On Mar 21, 2019, at 7:18 PM, Jongyoul Lee <jongyoul@gmail.com <ma...@gmail.com>> wrote:
>
>> It's not 100% sure but mainly because of the behaviours of FE.
>>
>> On Fri, Mar 22, 2019 at 7:25 AM Weilan Wu <weilan.wu@icloud.com <ma...@icloud.com>> wrote:
>> Hi, I am observing that the zeppelin web server always append ‘/#/‘ to end of the url, what’s the purpose of this? (and which code/module is responsible for it)
>>
>> e.g. if i type in browser: http://host.myzeppelin.com/ <http://host.myzeppelin.com/> the browser ends up showing (in address bar): http://host.myzeppelin.com/#/ <http://host.myzeppelin.com/#/>
>>
>> Thanks in advance,
>>
>> Weilan
>>
>>
>>
>> --
>> 이종열, Jongyoul Lee, 李宗烈
>> http://madeng.net <http://madeng.net/>
>
>
> --
> 이종열, Jongyoul Lee, 李宗烈
> http://madeng.net <http://madeng.net/>
Re: zeppelin url append '/#/'
Posted by Anthony Corbacho <an...@apache.org>.
Hi weilan,
This behavior comes from angular (we are using angular 1.5.7)
You can find imformation here:
https://scotch.io/tutorials/pretty-urls-in-angularjs-removing-the-hashtag
On Fri, 22 Mar 2019 at 4:59 PM Jongyoul Lee <jo...@gmail.com> wrote:
> It means Front End.
>
> On Fri, Mar 22, 2019 at 3:42 PM Weilan Wu <we...@icloud.com> wrote:
>
>> Hi Jongyoul,
>> Could you explain what “FE” means? It’s an important question for us to
>> solve when running Zeppelin it in Kubernetes...
>>
>> Thanks,
>> Weilan
>>
>> Sent from my iPhone 6s
>>
>> On Mar 21, 2019, at 7:18 PM, Jongyoul Lee <jo...@gmail.com> wrote:
>>
>> It's not 100% sure but mainly because of the behaviours of FE.
>>
>> On Fri, Mar 22, 2019 at 7:25 AM Weilan Wu <we...@icloud.com> wrote:
>>
>>> Hi, I am observing that the zeppelin web server always append ‘/#/‘ to
>>> end of the url, what’s the purpose of this? (and which code/module is
>>> responsible for it)
>>>
>>> e.g. if i type in browser: http://host.myzeppelin.com/ the browser ends
>>> up showing (in address bar): http://host.myzeppelin.com/#/
>>>
>>> Thanks in advance,
>>>
>>> Weilan
>>>
>>>
>>
>> --
>> 이종열, Jongyoul Lee, 李宗烈
>> http://madeng.net
>>
>>
>
> --
> 이종열, Jongyoul Lee, 李宗烈
> http://madeng.net
>
Re: zeppelin url append '/#/'
Posted by Jongyoul Lee <jo...@gmail.com>.
It means Front End.
On Fri, Mar 22, 2019 at 3:42 PM Weilan Wu <we...@icloud.com> wrote:
> Hi Jongyoul,
> Could you explain what “FE” means? It’s an important question for us to
> solve when running Zeppelin it in Kubernetes...
>
> Thanks,
> Weilan
>
> Sent from my iPhone 6s
>
> On Mar 21, 2019, at 7:18 PM, Jongyoul Lee <jo...@gmail.com> wrote:
>
> It's not 100% sure but mainly because of the behaviours of FE.
>
> On Fri, Mar 22, 2019 at 7:25 AM Weilan Wu <we...@icloud.com> wrote:
>
>> Hi, I am observing that the zeppelin web server always append ‘/#/‘ to
>> end of the url, what’s the purpose of this? (and which code/module is
>> responsible for it)
>>
>> e.g. if i type in browser: http://host.myzeppelin.com/ the browser ends
>> up showing (in address bar): http://host.myzeppelin.com/#/
>>
>> Thanks in advance,
>>
>> Weilan
>>
>>
>
> --
> 이종열, Jongyoul Lee, 李宗烈
> http://madeng.net
>
>
--
이종열, Jongyoul Lee, 李宗烈
http://madeng.net
Re: zeppelin url append '/#/'
Posted by Weilan Wu <we...@icloud.com>.
Hi Jongyoul,
Could you explain what “FE” means? It’s an important question for us to solve when running Zeppelin it in Kubernetes...
Thanks,
Weilan
Sent from my iPhone 6s
> On Mar 21, 2019, at 7:18 PM, Jongyoul Lee <jo...@gmail.com> wrote:
>
> It's not 100% sure but mainly because of the behaviours of FE.
>
>> On Fri, Mar 22, 2019 at 7:25 AM Weilan Wu <we...@icloud.com> wrote:
>> Hi, I am observing that the zeppelin web server always append ‘/#/‘ to end of the url, what’s the purpose of this? (and which code/module is responsible for it)
>>
>> e.g. if i type in browser: http://host.myzeppelin.com/ the browser ends up showing (in address bar): http://host.myzeppelin.com/#/
>>
>> Thanks in advance,
>>
>> Weilan
>>
>
>
> --
> 이종열, Jongyoul Lee, 李宗烈
> http://madeng.net
Re: zeppelin url append '/#/'
Posted by Jongyoul Lee <jo...@gmail.com>.
It's not 100% sure but mainly because of the behaviours of FE.
On Fri, Mar 22, 2019 at 7:25 AM Weilan Wu <we...@icloud.com> wrote:
> Hi, I am observing that the zeppelin web server always append ‘/#/‘ to end
> of the url, what’s the purpose of this? (and which code/module is
> responsible for it)
>
> e.g. if i type in browser: http://host.myzeppelin.com/ the browser ends
> up showing (in address bar): http://host.myzeppelin.com/#/
>
> Thanks in advance,
>
> Weilan
>
>
--
이종열, Jongyoul Lee, 李宗烈
http://madeng.net
zeppelin url append '/#/'
Posted by Weilan Wu <we...@icloud.com>.
Hi, I am observing that the zeppelin web server always append ‘/#/‘ to end of the url, what’s the purpose of this? (and which code/module is responsible for it)
e.g. if i type in browser: http://host.myzeppelin.com/ <http://host.myzeppelin.com/> the browser ends up showing (in address bar): http://host.myzeppelin.com/#/ <http://host.myzeppelin.com/#/>
Thanks in advance,
Weilan
Re: Spark issue moving from local to yarn-client
Posted by Dave Boyd <db...@incadencecorp.com>.
Ok, this had more information:
> INFO [2019-03-15 02:00:46,364] ({pool-2-thread-3}
> Logging.scala[logInfo]:54) - Logging events to
> hdfs:///var/log/spark/applicationHistory/application_1551287663522_0145
> ERROR [2019-03-15 02:00:46,366] ({SparkListenerBus}
> Logging.scala[logError]:91) - uncaught error in thread
> SparkListenerBus, stopping SparkContext
> java.lang.NoSuchMethodError:
> org.json4s.Formats.emptyValueStrategy()Lorg/json4s/prefs/EmptyValueStrategy;
> at org.json4s.jackson.JsonMethods$class.render(JsonMethods.scala:32)
> at org.json4s.jackson.JsonMethods$.render(JsonMethods.scala:50)
> at
> org.apache.spark.scheduler.EventLoggingListener$$anonfun$logEvent$1.apply(EventLoggingListener.scala:136)
> at
> org.apache.spark.scheduler.EventLoggingListener$$anonfun$logEvent$1.apply(EventLoggingListener.scala:136)
> at scala.Option.foreach(Option.scala:257)
> at
> org.apache.spark.scheduler.EventLoggingListener.logEvent(EventLoggingListener.scala:136)
> at
> org.apache.spark.scheduler.EventLoggingListener.onBlockManagerAdded(EventLoggingListener.scala:168)
> at
> org.apache.spark.scheduler.SparkListenerBus$class.doPostEvent(SparkListenerBus.scala:49)
> at
> org.apache.spark.scheduler.LiveListenerBus.doPostEvent(LiveListenerBus.scala:36)
> at
> org.apache.spark.scheduler.LiveListenerBus.doPostEvent(LiveListenerBus.scala:36)
> at
> org.apache.spark.util.ListenerBus$class.postToAll(ListenerBus.scala:63)
> at
> org.apache.spark.scheduler.LiveListenerBus.postToAll(LiveListenerBus.scala:36)
> at
> org.apache.spark.scheduler.LiveListenerBus$$anon$1$$anonfun$run$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(LiveListenerBus.scala:94)
> at
> org.apache.spark.scheduler.LiveListenerBus$$anon$1$$anonfun$run$1$$anonfun$apply$mcV$sp$1.apply(LiveListenerBus.scala:79)
> at
> org.apache.spark.scheduler.LiveListenerBus$$anon$1$$anonfun$run$1$$anonfun$apply$mcV$sp$1.apply(LiveListenerBus.scala:79)
> at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58)
> at
> org.apache.spark.scheduler.LiveListenerBus$$anon$1$$anonfun$run$1.apply$mcV$sp(LiveListenerBus.scala:78)
> at
> org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1245)
> at
> org.apache.spark.scheduler.LiveListenerBus$$anon$1.run(LiveListenerBus.scala:77)
> ERROR [2019-03-15 02:00:46,367] ({SparkListenerBus}
> Logging.scala[logError]:91) - throw uncaught fatal error in thread
> SparkListenerBus
> java.lang.NoSuchMethodError:
> org.json4s.Formats.emptyValueStrategy()Lorg/json4s/prefs/EmptyValueStrategy;
> at org.json4s.jackson.JsonMethods$class.render(JsonMethods.scala:32)
> at org.json4s.jackson.JsonMethods$.render(JsonMethods.scala:50)
> at
> org.apache.spark.scheduler.EventLoggingListener$$anonfun$logEvent$1.apply(EventLoggingListener.scala:136)
> at
> org.apache.spark.scheduler.EventLoggingListener$$anonfun$logEvent$1.apply(EventLoggingListener.scala:136)
> at scala.Option.foreach(Option.scala:257)
> at
> org.apache.spark.scheduler.EventLoggingListener.logEvent(EventLoggingListener.scala:136)
> at
> org.apache.spark.scheduler.EventLoggingListener.onBlockManagerAdded(EventLoggingListener.scala:168)
> at
> org.apache.spark.scheduler.SparkListenerBus$class.doPostEvent(SparkListenerBus.scala:49)
> at
> org.apache.spark.scheduler.LiveListenerBus.doPostEvent(LiveListenerBus.scala:36)
> at
> org.apache.spark.scheduler.LiveListenerBus.doPostEvent(LiveListenerBus.scala:36)
> at
> org.apache.spark.util.ListenerBus$class.postToAll(ListenerBus.scala:63)
> at
> org.apache.spark.scheduler.LiveListenerBus.postToAll(LiveListenerBus.scala:36)
> at
> org.apache.spark.scheduler.LiveListenerBus$$anon$1$$anonfun$run$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(LiveListenerBus.scala:94)
> at
> org.apache.spark.scheduler.LiveListenerBus$$anon$1$$anonfun$run$1$$anonfun$apply$mcV$sp$1.apply(LiveListenerBus.scala:79)
> at
> org.apache.spark.scheduler.LiveListenerBus$$anon$1$$anonfun$run$1$$anonfun$apply$mcV$sp$1.apply(LiveListenerBus.scala:79)
> at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58)
> at
> org.apache.spark.scheduler.LiveListenerBus$$anon$1$$anonfun$run$1.apply$mcV$sp(LiveListenerBus.scala:78)
> at
> org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1245)
> at
> org.apache.spark.scheduler.LiveListenerBus$$anon$1.run(LiveListenerBus.scala:77)
> INFO [2019-03-15 02:00:46,368] ({pool-2-thread-3}
> Logging.scala[logInfo]:54) - SchedulerBackend is ready for scheduling
> beginning after waiting maxRegisteredResourcesWaitingTime: 30000(ms)
> INFO [2019-03-15 02:00:46,375] ({stop-spark-context}
> AbstractConnector.java[doStop]:306) - Stopped
> ServerConnector@718326a2{HTTP/1.1}{0.0.0.0:55600}
> INFO [2019-03-15 02:00:46,376] ({stop-spark-context}
> ContextHandler.java[doStop]:865) - Stopped
> o.s.j.s.ServletContextHandler@69eb6481{/stages/stage/kill,null,UNAVAILABLE}
> INFO [2019-03-15 02:00:46,376] ({stop-spark-context}
> ContextHandler.java[doStop]:865) - Stopped
> o.s.j.s.ServletContextHandler@1b8dfd51{/jobs/job/kill,null,UNAVAILABLE}
> INFO [2019-03-15 02:00:46,377] ({stop-spark-context}
> ContextHandler.java[doStop]:865) - Stopped
> o.s.j.s.ServletContextHandler@6c8c0f44{/api,null,UNAVAILABLE}
> INFO [2019-03-15 02:00:46,378] ({stop-spark-context}
> ContextHandler.java[doStop]:865) - Stopped
> o.s.j.s.ServletContextHandler@37481c3{/,null,UNAVAILABLE}
> INFO [2019-03-15 02:00:46,378] ({stop-spark-context}
> ContextHandler.java[doStop]:865) - Stopped
> o.s.j.s.ServletContextHandler@5da5f62f{/static,null,UNAVAILABLE}
> INFO [2019-03-15 02:00:46,379] ({stop-spark-context}
> ContextHandler.java[doStop]:865) - Stopped
> o.s.j.s.ServletContextHandler@7ee10d{/executors/threadDump/json,null,UNAVAILABLE}
> INFO [2019-03-15 02:00:46,379] ({stop-spark-context}
> ContextHandler.java[doStop]:865) - Stopped
> o.s.j.s.ServletContextHandler@2b5899f0{/executors/threadDump,null,UNAVAILABLE}
> INFO [2019-03-15 02:00:46,379] ({stop-spark-context}
> ContextHandler.java[doStop]:865) - Stopped
> o.s.j.s.ServletContextHandler@4bd2abab{/executors/json,null,UNAVAILABLE}
> INFO [2019-03-15 02:00:46,380] ({stop-spark-context}
> ContextHandler.java[doStop]:865) - Stopped
> o.s.j.s.ServletContextHandler@260f612c{/executors,null,UNAVAILABLE}
> INFO [2019-03-15 02:00:46,380] ({stop-spark-context}
> ContextHandler.java[doStop]:865) - Stopped
> o.s.j.s.ServletContextHandler@7d8ad0f6{/environment/json,null,UNAVAILABLE}
> INFO [2019-03-15 02:00:46,380] ({stop-spark-context}
> ContextHandler.java[doStop]:865) - Stopped
> o.s.j.s.ServletContextHandler@4ee5c1e0{/environment,null,UNAVAILABLE}
> INFO [2019-03-15 02:00:46,381] ({stop-spark-context}
> ContextHandler.java[doStop]:865) - Stopped
> o.s.j.s.ServletContextHandler@64c12e28{/storage/rdd/json,null,UNAVAILABLE}
> INFO [2019-03-15 02:00:46,381] ({stop-spark-context}
> ContextHandler.java[doStop]:865) - Stopped
> o.s.j.s.ServletContextHandler@e9ab88b{/storage/rdd,null,UNAVAILABLE}
> INFO [2019-03-15 02:00:46,381] ({stop-spark-context}
> ContextHandler.java[doStop]:865) - Stopped
> o.s.j.s.ServletContextHandler@72f8348e{/storage/json,null,UNAVAILABLE}
> INFO [2019-03-15 02:00:46,382] ({stop-spark-context}
> ContextHandler.java[doStop]:865) - Stopped
> o.s.j.s.ServletContextHandler@322305bd{/storage,null,UNAVAILABLE}
> INFO [2019-03-15 02:00:46,382] ({stop-spark-context}
> ContextHandler.java[doStop]:865) - Stopped
> o.s.j.s.ServletContextHandler@528716d6{/stages/pool/json,null,UNAVAILABLE}
> INFO [2019-03-15 02:00:46,383] ({stop-spark-context}
> ContextHandler.java[doStop]:865) - Stopped
> o.s.j.s.ServletContextHandler@197cbc56{/stages/pool,null,UNAVAILABLE}
> INFO [2019-03-15 02:00:46,383] ({stop-spark-context}
> ContextHandler.java[doStop]:865) - Stopped
> o.s.j.s.ServletContextHandler@fb9f45b{/stages/stage/json,null,UNAVAILABLE}
> INFO [2019-03-15 02:00:46,383] ({stop-spark-context}
> ContextHandler.java[doStop]:865) - Stopped
> o.s.j.s.ServletContextHandler@3fabede6{/stages/stage,null,UNAVAILABLE}
> INFO [2019-03-15 02:00:46,383] ({stop-spark-context}
> ContextHandler.java[doStop]:865) - Stopped
> o.s.j.s.ServletContextHandler@26fda9b6{/stages/json,null,UNAVAILABLE}
> ERROR [2019-03-15 02:00:46,383] ({pool-2-thread-3}
> NewSparkInterpreter.java[open]:127) - Fail to open SparkInterpreter
> ERROR [2019-03-15 02:00:46,384] ({pool-2-thread-3} Job.java[run]:190)
> - Job failed
> org.apache.zeppelin.interpreter.InterpreterException: Fail to open
> SparkInterpreter
> at
> org.apache.zeppelin.spark.NewSparkInterpreter.open(NewSparkInterpreter.java:128)
> at
> org.apache.zeppelin.spark.SparkInterpreter.open(SparkInterpreter.java:62)
> at
> org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:69)
> at
> org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:616)
> at org.apache.zeppelin.scheduler.Job.run(Job.java:188)
> at
> org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:140)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.zeppelin.spark.BaseSparkScalaInterpreter.spark2CreateContext(BaseSparkScalaInterpreter.scala:259)
> at
> org.apache.zeppelin.spark.BaseSparkScalaInterpreter.createSparkContext(BaseSparkScalaInterpreter.scala:178)
> at
> org.apache.zeppelin.spark.SparkScala211Interpreter.open(SparkScala211Interpreter.scala:89)
> at
> org.apache.zeppelin.spark.NewSparkInterpreter.open(NewSparkInterpreter.java:102)
> ... 12 more
> Caused by: java.lang.IllegalStateException: Cannot call methods on a
> stopped SparkContext.
> This stopped SparkContext was created at:
>
> org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:860)
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> java.lang.reflect.Method.invoke(Method.java:498)
> org.apache.zeppelin.spark.BaseSparkScalaInterpreter.spark2CreateContext(BaseSparkScalaInterpreter.scala:259)
> org.apache.zeppelin.spark.BaseSparkScalaInterpreter.createSparkContext(BaseSparkScalaInterpreter.scala:178)
> org.apache.zeppelin.spark.SparkScala211Interpreter.open(SparkScala211Interpreter.scala:89)
> org.apache.zeppelin.spark.NewSparkInterpreter.open(NewSparkInterpreter.java:102)
> org.apache.zeppelin.spark.SparkInterpreter.open(SparkInterpreter.java:62)
> org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:69)
> org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:616)
> org.apache.zeppelin.scheduler.Job.run(Job.java:188)
> org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:140)
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> java.util.concurrent.FutureTask.run(FutureTask.java:266)
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>
> The currently active SparkContext was created at:
>
> org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:860)
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> java.lang.reflect.Method.invoke(Method.java:498)
> org.apache.zeppelin.spark.BaseSparkScalaInterpreter.spark2CreateContext(BaseSparkScalaInterpreter.scala:259)
> org.apache.zeppelin.spark.BaseSparkScalaInterpreter.createSparkContext(BaseSparkScalaInterpreter.scala:178)
> org.apache.zeppelin.spark.SparkScala211Interpreter.open(SparkScala211Interpreter.scala:89)
> org.apache.zeppelin.spark.NewSparkInterpreter.open(NewSparkInterpreter.java:102)
> org.apache.zeppelin.spark.SparkInterpreter.open(SparkInterpreter.java:62)
> org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:69)
> org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:616)
> org.apache.zeppelin.scheduler.Job.run(Job.java:188)
> org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:140)
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> java.util.concurrent.FutureTask.run(FutureTask.java:266)
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>
> at
> org.apache.spark.SparkContext.assertNotStopped(SparkContext.scala:100)
> at org.apache.spark.sql.SparkSession.<init>(SparkSession.scala:82)
> at org.apache.spark.sql.SparkSession.<init>(SparkSession.scala:79)
> at
> org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:877)
> ... 20 more
> INFO [2019-03-15 02:00:46,384] ({stop-spark-context}
> ContextHandler.java[doStop]:865) - Stopped
> o.s.j.s.ServletContextHandler@12009a60{/stages,null,UNAVAILABLE}
> INFO [2019-03-15 02:00:46,384] ({stop-spark-context}
> ContextHandler.java[doStop]:865) - Stopped
> o.s.j.s.ServletContextHandler@3cc19c74{/jobs/job/json,null,UNAVAILABLE}
> INFO [2019-03-15 02:00:46,385] ({stop-spark-context}
> ContextHandler.java[doStop]:865) - Stopped
> o.s.j.s.ServletContextHandler@21371f10{/jobs/job,null,UNAVAILABLE}
> INFO [2019-03-15 02:00:46,385] ({stop-spark-context}
> ContextHandler.java[doStop]:865) - Stopped
> o.s.j.s.ServletContextHandler@440e08ee{/jobs/json,null,UNAVAILABLE}
> INFO [2019-03-15 02:00:46,385] ({stop-spark-context}
> ContextHandler.java[doStop]:865) - Stopped
> o.s.j.s.ServletContextHandler@491cfe62{/jobs,null,UNAVAILABLE}
> INFO [2019-03-15 02:00:46,387] ({stop-spark-context}
> Logging.scala[logInfo]:54) - Stopped Spark web UI at
> http://10.1.50.128:55600
> INFO [2019-03-15 02:00:46,387] ({pool-2-thread-3}
> SchedulerFactory.java[jobFinished]:120) - Job
> 20190222-204451_856915056 finished by scheduler interpreter_587572627
> INFO [2019-03-15 02:00:46,502] ({Yarn application state monitor}
> Logging.scala[logInfo]:54) - Interrupting monitor thread
> INFO [2019-03-15 02:00:46,549] ({stop-spark-context}
> Logging.scala[logInfo]:54) - Shutting down all executors
> INFO [2019-03-15 02:00:46,549] ({dispatcher-event-loop-4}
> Logging.scala[logInfo]:54) - Asking each executor to shut down
> INFO [2019-03-15 02:00:46,550] ({stop-spark-context}
> Logging.scala[logInfo]:54) - Stopping SchedulerExtensionServices
> (serviceOption=None,
> services=List(),
> started=false)
> INFO [2019-03-15 02:00:46,552] ({stop-spark-context}
> Logging.scala[logInfo]:54) - Stopped
> INFO [2019-03-15 02:00:46,554] ({dispatcher-event-loop-1}
> Logging.scala[logInfo]:54) - MapOutputTrackerMasterEndpoint stopped!
> INFO [2019-03-15 02:00:46,560] ({stop-spark-context}
> Logging.scala[logInfo]:54) - MemoryStore cleared
> INFO [2019-03-15 02:00:46,560] ({stop-spark-context}
> Logging.scala[logInfo]:54) - BlockManager stopped
> INFO [2019-03-15 02:00:46,560] ({stop-spark-context}
> Logging.scala[logInfo]:54) - BlockManagerMaster stopped
> INFO [2019-03-15 02:00:46,561] ({dispatcher-event-loop-3}
> Logging.scala[logInfo]:54) - OutputCommitCoordinator stopped!
> INFO [2019-03-15 02:00:46,564] ({stop-spark-context}
> Logging.scala[logInfo]:54) - Successfully stopped SparkContext
On 3/14/19 9:42 PM, Jeff Zhang wrote:
> This log is zeppelin server log, the root log should be in the spark
> interpreter log. The file name is something like this :
> zeppelin-interpreter-spark*.log
> --
========= mailto:dboyd@incadencecorp.com ============
David W. Boyd
VP, Data Solutions
10432 Balls Ford, Suite 240
Manassas, VA 20109
office: +1-703-552-2862
cell: +1-703-402-7908
============== http://www.incadencecorp.com/ ============
ISO/IEC JTC1 WG9, editor ISO/IEC 20547 Big Data Reference Architecture
Chair ANSI/INCITS TC Big Data
Co-chair NIST Big Data Public Working Group Reference Architecture
First Robotic Mentor - FRC, FTC - www.iliterobotics.org
Board Member- USSTEM Foundation - www.usstem.org
The information contained in this message may be privileged
and/or confidential and protected from disclosure.
If the reader of this message is not the intended recipient
or an employee or agent responsible for delivering this message
to the intended recipient, you are hereby notified that any
dissemination, distribution or copying of this communication
is strictly prohibited. If you have received this communication
in error, please notify the sender immediately by replying to
this message and deleting the material from any computer.
Re: Spark issue moving from local to yarn-client
Posted by Jeff Zhang <zj...@gmail.com>.
This log is zeppelin server log, the root log should be in the spark
interpreter log. The file name is something like this :
zeppelin-interpreter-spark*.log
Dave Boyd <db...@incadencecorp.com> 于2019年3月15日周五 上午9:31写道：
> Jeff:
>
> Running a simple spark.version paragraph I sometimes get this:
>
> INFO [2019-03-15 01:12:18,720] ({pool-2-thread-49}
> RemoteInterpreter.java[call]:142) - Open RemoteInterpreter
> org.apache.zeppelin.spark.SparkInterpreter
> INFO [2019-03-15 01:12:18,721] ({pool-2-thread-49}
> RemoteInterpreter.java[pushAngularObjectRegistryToRemote]:436) - Push local
> angular object registry from ZeppelinServer to remote interpreter group
> spark:shared_process
> WARN [2019-03-15 01:13:30,593] ({pool-2-thread-49}
> NotebookServer.java[afterStatusChange]:2316) - Job
> 20190207-030535_192412278 is finished, status: ERROR, exception: null,
> result: %text java.lang.IllegalStateException: Spark context stopped while
> waiting for backend
> at
> org.apache.spark.scheduler.TaskSchedulerImpl.waitBackendReady(TaskSchedulerImpl.scala:614)
> at
> org.apache.spark.scheduler.TaskSchedulerImpl.postStartHook(TaskSchedulerImpl.scala:169)
> at org.apache.spark.SparkContext.<init>(SparkContext.scala:567)
> at org.apache.spark.SparkContext.<init>(SparkContext.scala:117)
> at
> org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2336)
> at org.apache.spark.SparkContext.getOrCreate(SparkContext.scala)
> at
> org.apache.zeppelin.spark.Spark2Shims.setupSparkListener(Spark2Shims.java:38)
> at
> org.apache.zeppelin.spark.NewSparkInterpreter.open(NewSparkInterpreter.java:120)
> at
> org.apache.zeppelin.spark.SparkInterpreter.open(SparkInterpreter.java:62)
> at
> org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:69)
> at
> org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:616)
> at org.apache.zeppelin.scheduler.Job.run(Job.java:188)
> at
> org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:140)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
>
> INFO [2019-03-15 01:13:30,598] ({pool-2-thread-49}
> VFSNotebookRepo.java[save]:196) - Saving note:2E4D6HQ3F
> INFO [2019-03-15 01:13:30,600] ({pool-2-thread-49}
> SchedulerFactory.java[jobFinished]:120) - Job 20190207-030535_192412278
> finished by scheduler
> org.apache.zeppelin.interpreter.remote.RemoteInterpreter-spark:shared_process-shared_session
>
> When I run this spark sql paragraph:
>
> // DataStore params to a hypothetical GeoMesa Accumulo table
> val dsParams = Map(
> "instanceId" -> "oedl",
> "zookeepers" -> "oedevnode00,oedevnode01,oedevnode02",
> "user" -> "oe_user",
> "password" -> "XXXXXXX",
> "tableName" -> "CoalesceSearch")
>
> // Create DataFrame using the "geomesa" format
> val docdataFrame =
> spark.read.format("geomesa").options(dsParams).option("geomesa.feature",
> "oedocumentrecordset").load()
> docdataFrame.createOrReplaceTempView("documentview")
>
> Here is the complete stack trace:
>
> INFO [2019-03-15 01:07:21,569] ({pool-2-thread-43}
> Paragraph.java[jobRun]:380) - Run paragraph [paragraph_id:
> 20190222-204451_856915056, interpreter: , note_id: 2E6X2CDWW, user:
> anonymous]
> WARN [2019-03-15 01:07:27,098] ({pool-2-thread-43}
> NotebookServer.java[afterStatusChange]:2316) - Job
> 20190222-204451_856915056 is finished, status: ERROR, exception: null,
> result: %text java.lang.IllegalStateException: Cannot call methods on a
> stopped SparkContext.
> This stopped SparkContext was created at:
>
>
> org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:860)
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> java.lang.reflect.Method.invoke(Method.java:498)
>
> org.apache.zeppelin.spark.BaseSparkScalaInterpreter.spark2CreateContext(BaseSparkScalaInterpreter.scala:259)
>
> org.apache.zeppelin.spark.BaseSparkScalaInterpreter.createSparkContext(BaseSparkScalaInterpreter.scala:178)
>
> org.apache.zeppelin.spark.SparkScala211Interpreter.open(SparkScala211Interpreter.scala:89)
>
> org.apache.zeppelin.spark.NewSparkInterpreter.open(NewSparkInterpreter.java:102)
> org.apache.zeppelin.spark.SparkInterpreter.open(SparkInterpreter.java:62)
>
> org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:69)
>
> org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:616)
> org.apache.zeppelin.scheduler.Job.run(Job.java:188)
> org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:140)
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> java.util.concurrent.FutureTask.run(FutureTask.java:266)
>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>
> The currently active SparkContext was created at:
>
> (No active SparkContext.)
>
> at org.apache.spark.SparkContext.assertNotStopped(SparkContext.scala:100)
> at
> org.apache.spark.SparkContext$$anonfun$parallelize$1.apply(SparkContext.scala:716)
> at
> org.apache.spark.SparkContext$$anonfun$parallelize$1.apply(SparkContext.scala:715)
> at
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
> at
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
> at org.apache.spark.SparkContext.withScope(SparkContext.scala:701)
> at org.apache.spark.SparkContext.parallelize(SparkContext.scala:715)
> at
> org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:74)
> at
> org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:114)
> at
> org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:114)
> at
> org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:135)
> at
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
> at
> org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:132)
> at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:113)
> at
> org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:87)
> at org.apache.spark.sql.Dataset.<init>(Dataset.scala:185)
> at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:64)
> at org.apache.spark.sql.Dataset.org
> $apache$spark$sql$Dataset$$withPlan(Dataset.scala:2822)
> at
> org.apache.spark.sql.Dataset.createOrReplaceTempView(Dataset.scala:2605)
> ... 47 elided
>
> INFO [2019-03-15 01:07:27,118] ({pool-2-thread-43}
> VFSNotebookRepo.java[save]:196) - Saving note:2E6X2CDWW
> INFO [2019-03-15 01:07:27,124] ({pool-2-thread-43}
> SchedulerFactory.java[jobFinished]:120) - Job 20190222-204451_856915056
> finished by scheduler
> org.apache.zeppelin.interpreter.remote.RemoteInterpreter-spark:shared_process-shared_session
> On 3/14/19 9:02 PM, Jeff Zhang wrote:
>
> Hi Dave,
>
> Could you paste the full stacktrace ? You can find it in the spark
> interpreter log file which is located in ZEPPELIN_HOME/logs
>
> Xun Liu <ne...@163.com> 于2019年3月15日周五 上午8:21写道：
>
>> Hi
>>
>> You can first execute a simple statement in spark, through sparksql, to
>> see if it can run normally in YARN.
>> If sparksql is running without problems, check the zeppelin and spark on
>> yarn issues.
>>
>> Also, what do you use for zeppelin-0.7.4? zeppelin-0.8.2? Is it a branch
>> that you maintain yourself?
>>
>> 在 2019年3月15日，上午6:31，Dave Boyd <db...@incadencecorp.com> 写道：
>>
>> All:
>>
>> I have some code that worked fine in Zeppelin 0.7.4 but I am having
>> issues in 0.8.2 when going from spark master of local to yarn-client. Yarn
>> client worked in 0.7.4.
>>
>> When my master is set to local[*] it runs just fine. However, as soon as
>> I switch to yarn-client I get the Cannot call methods on a stopped
>> SparkContext error. In looking at my yarn logs everything creates fine
>> and the job finishes without an error. The executors start just fine
>> from what I get out of yarn logs.
>>
>> Any suggestions on where to look? This happens with any note that trys
>> to run spark.
>>
>> If I try this very simple code:
>>
>> // Spark Version
>> spark.version
>>
>> I get this error:
>>
>> java.lang.IllegalStateException: Spark context stopped while waiting for
>> backend at
>> org.apache.spark.scheduler.TaskSchedulerImpl.waitBackendReady(TaskSchedulerImpl.scala:614)
>> at
>> org.apache.spark.scheduler.TaskSchedulerImpl.postStartHook(TaskSchedulerImpl.scala:169)
>> at org.apache.spark.SparkContext.<init>(SparkContext.scala:567) at
>> org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2313) at
>> org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:868)
>> at
>> org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:860)
>> at scala.Option.getOrElse(Option.scala:121) at
>> org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:860)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:498) at
>> org.apache.zeppelin.spark.BaseSparkScalaInterpreter.spark2CreateContext(BaseSparkScalaInterpreter.scala:259)
>> at
>> org.apache.zeppelin.spark.BaseSparkScalaInterpreter.createSparkContext(BaseSparkScalaInterpreter.scala:178)
>> at
>> org.apache.zeppelin.spark.SparkScala211Interpreter.open(SparkScala211Interpreter.scala:89)
>> at
>> org.apache.zeppelin.spark.NewSparkInterpreter.open(NewSparkInterpreter.java:102)
>> at
>> org.apache.zeppelin.spark.SparkInterpreter.open(SparkInterpreter.java:62)
>> at
>> org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:69)
>> at
>> org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:616)
>> at org.apache.zeppelin.scheduler.Job.run(Job.java:188) at
>> org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:140)
>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>> at java.lang.Thread.run(Thread.java:748)
>>
>> What am I missing?
>>
>> --
>> ========= mailto:dboyd@incadencecorp.com <db...@incadencecorp.com> ============
>> David W. Boyd
>> VP, Data Solutions
>> 10432 Balls Ford, Suite 240
>> Manassas, VA 20109
>> office: +1-703-552-2862
>> cell: +1-703-402-7908
>> ============== http://www.incadencecorp.com/ ============
>> ISO/IEC JTC1 WG9, editor ISO/IEC 20547 Big Data Reference Architecture
>> Chair ANSI/INCITS TC Big Data
>> Co-chair NIST Big Data Public Working Group Reference Architecture
>> First Robotic Mentor - FRC, FTC - www.iliterobotics.org
>> Board Member- USSTEM Foundation - www.usstem.org
>>
>> The information contained in this message may be privileged
>> and/or confidential and protected from disclosure.
>> If the reader of this message is not the intended recipient
>> or an employee or agent responsible for delivering this message
>> to the intended recipient, you are hereby notified that any
>> dissemination, distribution or copying of this communication
>> is strictly prohibited. If you have received this communication
>> in error, please notify the sender immediately by replying to
>> this message and deleting the material from any computer.
>>
>>
>>
>>
>>
>
> --
> Best Regards
>
> Jeff Zhang
>
> --
> ========= mailto:dboyd@incadencecorp.com <db...@incadencecorp.com> ============
> David W. Boyd
> VP, Data Solutions
> 10432 Balls Ford, Suite 240
> Manassas, VA 20109
> office: +1-703-552-2862
> cell: +1-703-402-7908
> ============== http://www.incadencecorp.com/ ============
> ISO/IEC JTC1 WG9, editor ISO/IEC 20547 Big Data Reference Architecture
> Chair ANSI/INCITS TC Big Data
> Co-chair NIST Big Data Public Working Group Reference Architecture
> First Robotic Mentor - FRC, FTC - www.iliterobotics.org
> Board Member- USSTEM Foundation - www.usstem.org
>
> The information contained in this message may be privileged
> and/or confidential and protected from disclosure.
> If the reader of this message is not the intended recipient
> or an employee or agent responsible for delivering this message
> to the intended recipient, you are hereby notified that any
> dissemination, distribution or copying of this communication
> is strictly prohibited. If you have received this communication
> in error, please notify the sender immediately by replying to
> this message and deleting the material from any computer.
>
>
>
>
--
Best Regards
Jeff Zhang
Re: Spark issue moving from local to yarn-client
Posted by Dave Boyd <db...@incadencecorp.com>.
Jeff:
Running a simple spark.version paragraph I sometimes get this:
INFO [2019-03-15 01:12:18,720] ({pool-2-thread-49} RemoteInterpreter.java[call]:142) - Open RemoteInterpreter org.apache.zeppelin.spark.SparkInterpreter
INFO [2019-03-15 01:12:18,721] ({pool-2-thread-49} RemoteInterpreter.java[pushAngularObjectRegistryToRemote]:436) - Push local angular object registry from ZeppelinServer to remote interpreter group spark:shared_process
WARN [2019-03-15 01:13:30,593] ({pool-2-thread-49} NotebookServer.java[afterStatusChange]:2316) - Job 20190207-030535_192412278 is finished, status: ERROR, exception: null, result: %text java.lang.IllegalStateException: Spark context stopped while waiting for backend
at org.apache.spark.scheduler.TaskSchedulerImpl.waitBackendReady(TaskSchedulerImpl.scala:614)
at org.apache.spark.scheduler.TaskSchedulerImpl.postStartHook(TaskSchedulerImpl.scala:169)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:567)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:117)
at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2336)
at org.apache.spark.SparkContext.getOrCreate(SparkContext.scala)
at org.apache.zeppelin.spark.Spark2Shims.setupSparkListener(Spark2Shims.java:38)
at org.apache.zeppelin.spark.NewSparkInterpreter.open(NewSparkInterpreter.java:120)
at org.apache.zeppelin.spark.SparkInterpreter.open(SparkInterpreter.java:62)
at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:69)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:616)
at org.apache.zeppelin.scheduler.Job.run(Job.java:188)
at org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:140)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
INFO [2019-03-15 01:13:30,598] ({pool-2-thread-49} VFSNotebookRepo.java[save]:196) - Saving note:2E4D6HQ3F
INFO [2019-03-15 01:13:30,600] ({pool-2-thread-49} SchedulerFactory.java[jobFinished]:120) - Job 20190207-030535_192412278 finished by scheduler org.apache.zeppelin.interpreter.remote.RemoteInterpreter-spark:shared_process-shared_session
When I run this spark sql paragraph:
// DataStore params to a hypothetical GeoMesa Accumulo table
val dsParams = Map(
"instanceId" -> "oedl",
"zookeepers" -> "oedevnode00,oedevnode01,oedevnode02",
"user" -> "oe_user",
"password" -> "XXXXXXX",
"tableName" -> "CoalesceSearch")
// Create DataFrame using the "geomesa" format
val docdataFrame = spark.read.format("geomesa").options(dsParams).option("geomesa.feature", "oedocumentrecordset").load()
docdataFrame.createOrReplaceTempView("documentview")
Here is the complete stack trace:
INFO [2019-03-15 01:07:21,569] ({pool-2-thread-43} Paragraph.java[jobRun]:380) - Run paragraph [paragraph_id: 20190222-204451_856915056, interpreter: , note_id: 2E6X2CDWW, user: anonymous]
WARN [2019-03-15 01:07:27,098] ({pool-2-thread-43} NotebookServer.java[afterStatusChange]:2316) - Job 20190222-204451_856915056 is finished, status: ERROR, exception: null, result: %text java.lang.IllegalStateException: Cannot call methods on a stopped SparkContext.
This stopped SparkContext was created at:
org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:860)
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:498)
org.apache.zeppelin.spark.BaseSparkScalaInterpreter.spark2CreateContext(BaseSparkScalaInterpreter.scala:259)
org.apache.zeppelin.spark.BaseSparkScalaInterpreter.createSparkContext(BaseSparkScalaInterpreter.scala:178)
org.apache.zeppelin.spark.SparkScala211Interpreter.open(SparkScala211Interpreter.scala:89)
org.apache.zeppelin.spark.NewSparkInterpreter.open(NewSparkInterpreter.java:102)
org.apache.zeppelin.spark.SparkInterpreter.open(SparkInterpreter.java:62)
org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:69)
org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:616)
org.apache.zeppelin.scheduler.Job.run(Job.java:188)
org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:140)
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
java.util.concurrent.FutureTask.run(FutureTask.java:266)
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
The currently active SparkContext was created at:
(No active SparkContext.)
at org.apache.spark.SparkContext.assertNotStopped(SparkContext.scala:100)
at org.apache.spark.SparkContext$$anonfun$parallelize$1.apply(SparkContext.scala:716)
at org.apache.spark.SparkContext$$anonfun$parallelize$1.apply(SparkContext.scala:715)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
at org.apache.spark.SparkContext.withScope(SparkContext.scala:701)
at org.apache.spark.SparkContext.parallelize(SparkContext.scala:715)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:74)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:114)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:114)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:135)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:132)
at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:113)
at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:87)
at org.apache.spark.sql.Dataset.<init>(Dataset.scala:185)
at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:64)
at org.apache.spark.sql.Dataset.org$apache$spark$sql$Dataset$$withPlan(Dataset.scala:2822)
at org.apache.spark.sql.Dataset.createOrReplaceTempView(Dataset.scala:2605)
... 47 elided
INFO [2019-03-15 01:07:27,118] ({pool-2-thread-43} VFSNotebookRepo.java[save]:196) - Saving note:2E6X2CDWW
INFO [2019-03-15 01:07:27,124] ({pool-2-thread-43} SchedulerFactory.java[jobFinished]:120) - Job 20190222-204451_856915056 finished by scheduler org.apache.zeppelin.interpreter.remote.RemoteInterpreter-spark:shared_process-shared_session
On 3/14/19 9:02 PM, Jeff Zhang wrote:
Hi Dave,
Could you paste the full stacktrace ? You can find it in the spark interpreter log file which is located in ZEPPELIN_HOME/logs
Xun Liu <ne...@163.com>> 于2019年3月15日周五 上午8:21写道：
Hi
You can first execute a simple statement in spark, through sparksql, to see if it can run normally in YARN.
If sparksql is running without problems, check the zeppelin and spark on yarn issues.
Also, what do you use for zeppelin-0.7.4? zeppelin-0.8.2? Is it a branch that you maintain yourself?
在 2019年3月15日，上午6:31，Dave Boyd <db...@incadencecorp.com>> 写道：
All:
I have some code that worked fine in Zeppelin 0.7.4 but I am having issues in 0.8.2 when going from spark master of local to yarn-client. Yarn client worked in 0.7.4.
When my master is set to local[*] it runs just fine. However, as soon as I switch to yarn-client I get the Cannot call methods on a stopped SparkContext error. In looking at my yarn logs everything creates fine and the job finishes without an error. The executors start just fine
from what I get out of yarn logs.
Any suggestions on where to look? This happens with any note that trys to run spark.
If I try this very simple code:
// Spark Version
spark.version
I get this error:
java.lang.IllegalStateException: Spark context stopped while waiting for backend at org.apache.spark.scheduler.TaskSchedulerImpl.waitBackendReady(TaskSchedulerImpl.scala:614) at org.apache.spark.scheduler.TaskSchedulerImpl.postStartHook(TaskSchedulerImpl.scala:169) at org.apache.spark.SparkContext.<init>(SparkContext.scala:567) at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2313) at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:868) at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:860) at scala.Option.getOrElse(Option.scala:121) at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:860) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.zeppelin.spark.BaseSparkScalaInterpreter.spark2CreateContext(BaseSparkScalaInterpreter.scala:259) at org.apache.zeppelin.spark.BaseSparkScalaInterpreter.createSparkContext(BaseSparkScalaInterpreter.scala:178) at org.apache.zeppelin.spark.SparkScala211Interpreter.open(SparkScala211Interpreter.scala:89) at org.apache.zeppelin.spark.NewSparkInterpreter.open(NewSparkInterpreter.java:102) at org.apache.zeppelin.spark.SparkInterpreter.open(SparkInterpreter.java:62) at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:69) at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:616) at org.apache.zeppelin.scheduler.Job.run(Job.java:188) at org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:140) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)
What am I missing?
--
========= mailto:dboyd@incadencecorp.com ============
David W. Boyd
VP, Data Solutions
10432 Balls Ford, Suite 240
Manassas, VA 20109
office: +1-703-552-2862
cell: +1-703-402-7908
============== http://www.incadencecorp.com/ ============
ISO/IEC JTC1 WG9, editor ISO/IEC 20547 Big Data Reference Architecture
Chair ANSI/INCITS TC Big Data
Co-chair NIST Big Data Public Working Group Reference Architecture
First Robotic Mentor - FRC, FTC - www.iliterobotics.org<http://www.iliterobotics.org/>
Board Member- USSTEM Foundation - www.usstem.org<http://www.usstem.org/>
The information contained in this message may be privileged
and/or confidential and protected from disclosure.
If the reader of this message is not the intended recipient
or an employee or agent responsible for delivering this message
to the intended recipient, you are hereby notified that any
dissemination, distribution or copying of this communication
is strictly prohibited. If you have received this communication
in error, please notify the sender immediately by replying to
this message and deleting the material from any computer.
--
Best Regards
Jeff Zhang
--
========= mailto:dboyd@incadencecorp.com ============
David W. Boyd
VP, Data Solutions
10432 Balls Ford, Suite 240
Manassas, VA 20109
office: +1-703-552-2862
cell: +1-703-402-7908
============== http://www.incadencecorp.com/ ============
ISO/IEC JTC1 WG9, editor ISO/IEC 20547 Big Data Reference Architecture
Chair ANSI/INCITS TC Big Data
Co-chair NIST Big Data Public Working Group Reference Architecture
First Robotic Mentor - FRC, FTC - www.iliterobotics.org<http://www.iliterobotics.org>
Board Member- USSTEM Foundation - www.usstem.org<http://www.usstem.org>
The information contained in this message may be privileged
and/or confidential and protected from disclosure.
If the reader of this message is not the intended recipient
or an employee or agent responsible for delivering this message
to the intended recipient, you are hereby notified that any
dissemination, distribution or copying of this communication
is strictly prohibited. If you have received this communication
in error, please notify the sender immediately by replying to
this message and deleting the material from any computer.
Re: Spark issue moving from local to yarn-client
Posted by Jeff Zhang <zj...@gmail.com>.
Hi Dave,
Could you paste the full stacktrace ? You can find it in the spark
interpreter log file which is located in ZEPPELIN_HOME/logs
Xun Liu <ne...@163.com> 于2019年3月15日周五 上午8:21写道：
> Hi
>
> You can first execute a simple statement in spark, through sparksql, to
> see if it can run normally in YARN.
> If sparksql is running without problems, check the zeppelin and spark on
> yarn issues.
>
> Also, what do you use for zeppelin-0.7.4? zeppelin-0.8.2? Is it a branch
> that you maintain yourself?
>
> 在 2019年3月15日，上午6:31，Dave Boyd <db...@incadencecorp.com> 写道：
>
> All:
>
> I have some code that worked fine in Zeppelin 0.7.4 but I am having
> issues in 0.8.2 when going from spark master of local to yarn-client. Yarn
> client worked in 0.7.4.
>
> When my master is set to local[*] it runs just fine. However, as soon as
> I switch to yarn-client I get the Cannot call methods on a stopped
> SparkContext error. In looking at my yarn logs everything creates fine
> and the job finishes without an error. The executors start just fine
> from what I get out of yarn logs.
>
> Any suggestions on where to look? This happens with any note that trys
> to run spark.
>
> If I try this very simple code:
>
> // Spark Version
> spark.version
>
> I get this error:
>
> java.lang.IllegalStateException: Spark context stopped while waiting for
> backend at
> org.apache.spark.scheduler.TaskSchedulerImpl.waitBackendReady(TaskSchedulerImpl.scala:614)
> at
> org.apache.spark.scheduler.TaskSchedulerImpl.postStartHook(TaskSchedulerImpl.scala:169)
> at org.apache.spark.SparkContext.<init>(SparkContext.scala:567) at
> org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2313) at
> org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:868)
> at
> org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:860)
> at scala.Option.getOrElse(Option.scala:121) at
> org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:860)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> org.apache.zeppelin.spark.BaseSparkScalaInterpreter.spark2CreateContext(BaseSparkScalaInterpreter.scala:259)
> at
> org.apache.zeppelin.spark.BaseSparkScalaInterpreter.createSparkContext(BaseSparkScalaInterpreter.scala:178)
> at
> org.apache.zeppelin.spark.SparkScala211Interpreter.open(SparkScala211Interpreter.scala:89)
> at
> org.apache.zeppelin.spark.NewSparkInterpreter.open(NewSparkInterpreter.java:102)
> at
> org.apache.zeppelin.spark.SparkInterpreter.open(SparkInterpreter.java:62)
> at
> org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:69)
> at
> org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:616)
> at org.apache.zeppelin.scheduler.Job.run(Job.java:188) at
> org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:140)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
>
> What am I missing?
>
> --
> ========= mailto:dboyd@incadencecorp.com <db...@incadencecorp.com> ============
> David W. Boyd
> VP, Data Solutions
> 10432 Balls Ford, Suite 240
> Manassas, VA 20109
> office: +1-703-552-2862
> cell: +1-703-402-7908
> ============== http://www.incadencecorp.com/ ============
> ISO/IEC JTC1 WG9, editor ISO/IEC 20547 Big Data Reference Architecture
> Chair ANSI/INCITS TC Big Data
> Co-chair NIST Big Data Public Working Group Reference Architecture
> First Robotic Mentor - FRC, FTC - www.iliterobotics.org
> Board Member- USSTEM Foundation - www.usstem.org
>
> The information contained in this message may be privileged
> and/or confidential and protected from disclosure.
> If the reader of this message is not the intended recipient
> or an employee or agent responsible for delivering this message
> to the intended recipient, you are hereby notified that any
> dissemination, distribution or copying of this communication
> is strictly prohibited. If you have received this communication
> in error, please notify the sender immediately by replying to
> this message and deleting the material from any computer.
>
>
>
>
>
--
Best Regards
Jeff Zhang
Re: Spark issue moving from local to yarn-client
Posted by Xun Liu <ne...@163.com>.
Hi
You can first execute a simple statement in spark, through sparksql, to see if it can run normally in YARN.
If sparksql is running without problems, check the zeppelin and spark on yarn issues.
Also, what do you use for zeppelin-0.7.4? zeppelin-0.8.2? Is it a branch that you maintain yourself?
> 在 2019年3月15日，上午6:31，Dave Boyd <db...@incadencecorp.com> 写道：
>
> All:
>
> I have some code that worked fine in Zeppelin 0.7.4 but I am having issues in 0.8.2 when going from spark master of local to yarn-client. Yarn client worked in 0.7.4.
> When my master is set to local[*] it runs just fine. However, as soon as I switch to yarn-client I get the Cannot call methods on a stopped SparkContext error. In looking at my yarn logs everything creates fine and the job finishes without an error. The executors start just fine
> from what I get out of yarn logs.
> Any suggestions on where to look? This happens with any note that trys to run spark.
>
> If I try this very simple code:
>
> // Spark Version
> spark.version
> I get this error:
>
>
>>
>> java.lang.IllegalStateException: Spark context stopped while waiting for backend at org.apache.spark.scheduler.TaskSchedulerImpl.waitBackendReady(TaskSchedulerImpl.scala:614) at org.apache.spark.scheduler.TaskSchedulerImpl.postStartHook(TaskSchedulerImpl.scala:169)
>> at org.apache.spark.SparkContext.<init>(SparkContext.scala:567) at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2313) at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:868) at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:860)
>> at scala.Option.getOrElse(Option.scala:121) at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:860) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.zeppelin.spark.BaseSparkScalaInterpreter.spark2CreateContext(BaseSparkScalaInterpreter.scala:259) at
>> org.apache.zeppelin.spark.BaseSparkScalaInterpreter.createSparkContext(BaseSparkScalaInterpreter.scala:178) at org.apache.zeppelin.spark.SparkScala211Interpreter.open(SparkScala211Interpreter.scala:89) at org.apache.zeppelin.spark.NewSparkInterpreter.open(NewSparkInterpreter.java:102)
>> at org.apache.zeppelin.spark.SparkInterpreter.open(SparkInterpreter.java:62) at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:69) at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:616)
>> at org.apache.zeppelin.scheduler.Job.run(Job.java:188) at org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:140) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)
>>
> What am I missing?
> --
> ========= mailto:dboyd@incadencecorp.com <ma...@incadencecorp.com> ============
> David W. Boyd
> VP, Data Solutions
> 10432 Balls Ford, Suite 240
> Manassas, VA 20109
> office: +1-703-552-2862
> cell: +1-703-402-7908
> ============== http://www.incadencecorp.com/ <http://www.incadencecorp.com/> ============
> ISO/IEC JTC1 WG9, editor ISO/IEC 20547 Big Data Reference Architecture
> Chair ANSI/INCITS TC Big Data
> Co-chair NIST Big Data Public Working Group Reference Architecture
> First Robotic Mentor - FRC, FTC - www.iliterobotics.org <http://www.iliterobotics.org/>
> Board Member- USSTEM Foundation - www.usstem.org <http://www.usstem.org/>
>
> The information contained in this message may be privileged
> and/or confidential and protected from disclosure.
> If the reader of this message is not the intended recipient
> or an employee or agent responsible for delivering this message
> to the intended recipient, you are hereby notified that any
> dissemination, distribution or copying of this communication
> is strictly prohibited. If you have received this communication
> in error, please notify the sender immediately by replying to
> this message and deleting the material from any computer.
>
>