You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@zeppelin.apache.org by David Boyd <db...@incadencecorp.com> on 2018/05/08 22:46:03 UTC
Artifact dependency for geomesa causes NPE
All:
I am following the instructions here:
http://www.geomesa.org/documentation/current/user/spark/zeppelin.html
To use geomesa spark with zeppelin.
Whenever I add the artifact dependency I get the following error on any
code I try to run (this includes the basic features -spark
tutorial).
> java.lang.NullPointerException
> at org.apache.zeppelin.spark.Utils.invokeMethod(Utils.java:38)
> at org.apache.zeppelin.spark.Utils.invokeMethod(Utils.java:33)
> at
> org.apache.zeppelin.spark.SparkInterpreter.createSparkContext_2(SparkInterpreter.java:398)
> at
> org.apache.zeppelin.spark.SparkInterpreter.createSparkContext(SparkInterpreter.java:387)
> at
> org.apache.zeppelin.spark.SparkInterpreter.getSparkContext(SparkInterpreter.java:146)
> at
> org.apache.zeppelin.spark.SparkInterpreter.open(SparkInterpreter.java:843)
> at
> org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:70)
> at
> org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:491)
> at org.apache.zeppelin.scheduler.Job.run(Job.java:175)
> at
> org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:139)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
I have tried specifying the jar as a maven artifact and as an absolute
path. I have tried multiple versions (1.3.4, 1.3.5, and 2.0.0) of the
artifact. As soon as I remove the dependency the code works again.
Is there another external dependency I can try to see if that is the
problem.
I have tried this with both the 0.7.3 Zeppelin binary distribution, and
also with a 0.7.4 distribution I built specifically.
I am running spark 2.1 on my cluster. Like I said without this
dependency the example code works just fine.
Here is the build command I used for the distribution I am running:
mvn clean package -DskipTests -Pspark-2.1 -Phadoop-2.7 -Pyarn -Ppyspark
-Psparkr -Pr -Pscala-2.11 -Pexamples -Pbuild-distr
From looking at the code around the trace it appears either a class is
not found or something is getting dorked with SparkContext.
Any help would be appreciated.
--
========= mailto:dboyd@incadencecorp.com ============
David W. Boyd
VP, Data Solutions
10432 Balls Ford, Suite 240
Manassas, VA 20109
office: +1-703-552-2862
cell: +1-703-402-7908
============== http://www.incadencecorp.com/ ============
ISO/IEC JTC1 WG9, editor ISO/IEC 20547 Big Data Reference Architecture
Chair ANSI/INCITS TC Big Data
Co-chair NIST Big Data Public Working Group Reference Architecture
First Robotic Mentor - FRC, FTC - www.iliterobotics.org
Board Member- USSTEM Foundation - www.usstem.org
The information contained in this message may be privileged
and/or confidential and protected from disclosure.
If the reader of this message is not the intended recipient
or an employee or agent responsible for delivering this message
to the intended recipient, you are hereby notified that any
dissemination, distribution or copying of this communication
is strictly prohibited. If you have received this communication
in error, please notify the sender immediately by replying to
this message and deleting the material from any computer.
Re: Artifact dependency for geomesa causes NPE
Posted by Jongyoul Lee <jo...@gmail.com>.
Hi,
I'm not familiar with geomesa but you don't have to build Zeppelin with a
specific Spark version because Zeppelin support all of external Spark
without rebuilding it. I suggest you download binary version and extract it
and set `SPARK_HOME` inside conf/zeppelin-env. As you mentioned, it might
look like a dependency problem.
Hope this help,
JL
On Wed, May 9, 2018 at 7:46 AM, David Boyd <db...@incadencecorp.com> wrote:
> All:
>
> I am following the instructions here: http://www.geomesa.org/
> documentation/current/user/spark/zeppelin.html
> To use geomesa spark with zeppelin.
> Whenever I add the artifact dependency I get the following error on any
> code I try to run (this includes the basic features -spark
> tutorial).
>
> java.lang.NullPointerException
> at org.apache.zeppelin.spark.Utils.invokeMethod(Utils.java:38)
> at org.apache.zeppelin.spark.Utils.invokeMethod(Utils.java:33)
> at org.apache.zeppelin.spark.SparkInterpreter.createSparkContext_2(
> SparkInterpreter.java:398)
> at org.apache.zeppelin.spark.SparkInterpreter.createSparkContext(
> SparkInterpreter.java:387)
> at org.apache.zeppelin.spark.SparkInterpreter.getSparkContext(
> SparkInterpreter.java:146)
> at org.apache.zeppelin.spark.SparkInterpreter.open(
> SparkInterpreter.java:843)
> at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(
> LazyOpenInterpreter.java:70)
> at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$
> InterpretJob.jobRun(RemoteInterpreterServer.java:491)
> at org.apache.zeppelin.scheduler.Job.run(Job.java:175)
> at org.apache.zeppelin.scheduler.FIFOScheduler$1.run(
> FIFOScheduler.java:139)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ScheduledThreadPoolExecutor$
> ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> at java.util.concurrent.ScheduledThreadPoolExecutor$
> ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
>
> I have tried specifying the jar as a maven artifact and as an absolute
> path. I have tried multiple versions (1.3.4, 1.3.5, and 2.0.0) of the
> artifact. As soon as I remove the dependency the code works again.
>
> Is there another external dependency I can try to see if that is the
> problem.
> I have tried this with both the 0.7.3 Zeppelin binary distribution, and
> also with a 0.7.4 distribution I built specifically.
> I am running spark 2.1 on my cluster. Like I said without this dependency
> the example code works just fine.
> Here is the build command I used for the distribution I am running:
>
> mvn clean package -DskipTests -Pspark-2.1 -Phadoop-2.7 -Pyarn -Ppyspark
> -Psparkr -Pr -Pscala-2.11 -Pexamples -Pbuild-distr
>
> From looking at the code around the trace it appears either a class is not
> found or something is getting dorked with SparkContext.
>
> Any help would be appreciated.
>
>
> --
> ========= mailto:dboyd@incadencecorp.com <db...@incadencecorp.com> ============
> David W. Boyd
> VP, Data Solutions
> 10432 Balls Ford, Suite 240
> Manassas, VA 20109
> office: +1-703-552-2862
> cell: +1-703-402-7908
> ============== http://www.incadencecorp.com/ ============
> ISO/IEC JTC1 WG9, editor ISO/IEC 20547 Big Data Reference Architecture
> Chair ANSI/INCITS TC Big Data
> Co-chair NIST Big Data Public Working Group Reference Architecture
> First Robotic Mentor - FRC, FTC - www.iliterobotics.org
> Board Member- USSTEM Foundation - www.usstem.org
>
> The information contained in this message may be privileged
> and/or confidential and protected from disclosure.
> If the reader of this message is not the intended recipient
> or an employee or agent responsible for delivering this message
> to the intended recipient, you are hereby notified that any
> dissemination, distribution or copying of this communication
> is strictly prohibited. If you have received this communication
> in error, please notify the sender immediately by replying to
> this message and deleting the material from any computer.
>
>
>
>
--
이종열, Jongyoul Lee, 李宗烈
http://madeng.net
Re: Artifact dependency for geomesa causes NPE
Posted by David Boyd <db...@incadencecorp.com>.
All: Ok more details after sifting through logs. This appears to be a
dependency issue. In the interpreter log I
am seeing:
> ERROR [2018-05-08 23:27:27,331] ({SparkListenerBus}
> Logging.scala[logError]:91) - uncaught error in thread
> SparkListenerBus, stopping SparkContext
> java.lang.NoSuchMethodError:
> org.json4s.Formats.emptyValueStrategy()Lorg/json4s/prefs/EmptyValueStrategy;
> at org.json4s.jackson.JsonMethods$class.render(JsonMethods.scala:32)
> at org.json4s.jackson.JsonMethods$.render(JsonMethods.scala:50)
> at
> org.apache.spark.scheduler.EventLoggingListener$$anonfun$logEvent$1.apply(EventLoggingListener.scala:136)
> at
> org.apache.spark.scheduler.EventLoggingListener$$anonfun$logEvent$1.apply(EventLoggingListener.scala:136)
> at scala.Option.foreach(Option.scala:257)
> at
> org.apache.spark.scheduler.EventLoggingListener.logEvent(EventLoggingListener.scala:136)
> at
> org.apache.spark.scheduler.EventLoggingListener.onExecutorAdded(EventLoggingListener.scala:187)
> at
> org.apache.spark.scheduler.SparkListenerBus$class.doPostEvent(SparkListenerBus.scala:61)
> at
> org.apache.spark.scheduler.LiveListenerBus.doPostEvent(LiveListenerBus.scala:36)
> at
> org.apache.spark.scheduler.LiveListenerBus.doPostEvent(LiveListenerBus.scala:36)
> at
> org.apache.spark.util.ListenerBus$class.postToAll(ListenerBus.scala:63)
> at
> org.apache.spark.scheduler.LiveListenerBus.postToAll(LiveListenerBus.scala:36)
> at
> org.apache.spark.scheduler.LiveListenerBus$$anon$1$$anonfun$run$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(LiveListenerBus.scala:94)
> at
> org.apache.spark.scheduler.LiveListenerBus$$anon$1$$anonfun$run$1$$anonfun$apply$mcV$sp$1.apply(LiveListenerBus.scala:79)
> at
> org.apache.spark.scheduler.LiveListenerBus$$anon$1$$anonfun$run$1$$anonfun$apply$mcV$sp$1.apply(LiveListenerBus.scala:79)
> at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58)
> at
> org.apache.spark.scheduler.LiveListenerBus$$anon$1$$anonfun$run$1.apply$mcV$sp(LiveListenerBus.scala:78)
> at
> org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1245)
> at
> org.apache.spark.scheduler.LiveListenerBus$$anon$1.run(LiveListenerBus.scala:77)
Based on that and after chasing down things in Maven I added the
following excludes to my dependency:
org.json4s:json4s-core_2.11, org.json4s:json4s-jackson_2.11,
org.json4s:json4s-native_2.11
That did not make a difference.
Googling it appears to be an incompatibility between 3.2.10 and 3.2.11
of json4s.
On 5/8/18 6:46 PM, David Boyd wrote:
>
> All:
>
> I am following the instructions here:
> http://www.geomesa.org/documentation/current/user/spark/zeppelin.html
> To use geomesa spark with zeppelin.
> Whenever I add the artifact dependency I get the following error on
> any code I try to run (this includes the basic features -spark
> tutorial).
>
>> java.lang.NullPointerException
>> at org.apache.zeppelin.spark.Utils.invokeMethod(Utils.java:38)
>> at org.apache.zeppelin.spark.Utils.invokeMethod(Utils.java:33)
>> at
>> org.apache.zeppelin.spark.SparkInterpreter.createSparkContext_2(SparkInterpreter.java:398)
>> at
>> org.apache.zeppelin.spark.SparkInterpreter.createSparkContext(SparkInterpreter.java:387)
>> at
>> org.apache.zeppelin.spark.SparkInterpreter.getSparkContext(SparkInterpreter.java:146)
>> at
>> org.apache.zeppelin.spark.SparkInterpreter.open(SparkInterpreter.java:843)
>> at
>> org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:70)
>> at
>> org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:491)
>> at org.apache.zeppelin.scheduler.Job.run(Job.java:175)
>> at
>> org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:139)
>> at
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>> at java.lang.Thread.run(Thread.java:748)
> I have tried specifying the jar as a maven artifact and as an absolute
> path. I have tried multiple versions (1.3.4, 1.3.5, and 2.0.0) of
> the artifact. As soon as I remove the dependency the code works again.
>
> Is there another external dependency I can try to see if that is the
> problem.
> I have tried this with both the 0.7.3 Zeppelin binary distribution,
> and also with a 0.7.4 distribution I built specifically.
> I am running spark 2.1 on my cluster. Like I said without this
> dependency the example code works just fine.
> Here is the build command I used for the distribution I am running:
>
> mvn clean package -DskipTests -Pspark-2.1 -Phadoop-2.7 -Pyarn
> -Ppyspark -Psparkr -Pr -Pscala-2.11 -Pexamples -Pbuild-distr
>
> From looking at the code around the trace it appears either a class is
> not found or something is getting dorked with SparkContext.
>
> Any help would be appreciated.
>
>
> --
> =========mailto:dboyd@incadencecorp.com ============
> David W. Boyd
> VP, Data Solutions
> 10432 Balls Ford, Suite 240
> Manassas, VA 20109
> office: +1-703-552-2862
> cell: +1-703-402-7908
> ==============http://www.incadencecorp.com/ ============
> ISO/IEC JTC1 WG9, editor ISO/IEC 20547 Big Data Reference Architecture
> Chair ANSI/INCITS TC Big Data
> Co-chair NIST Big Data Public Working Group Reference Architecture
> First Robotic Mentor - FRC, FTC -www.iliterobotics.org
> Board Member- USSTEM Foundation -www.usstem.org
>
> The information contained in this message may be privileged
> and/or confidential and protected from disclosure.
> If the reader of this message is not the intended recipient
> or an employee or agent responsible for delivering this message
> to the intended recipient, you are hereby notified that any
> dissemination, distribution or copying of this communication
> is strictly prohibited. If you have received this communication
> in error, please notify the sender immediately by replying to
> this message and deleting the material from any computer.
>
>
--
========= mailto:dboyd@incadencecorp.com ============
David W. Boyd
VP, Data Solutions
10432 Balls Ford, Suite 240
Manassas, VA 20109
office: +1-703-552-2862
cell: +1-703-402-7908
============== http://www.incadencecorp.com/ ============
ISO/IEC JTC1 WG9, editor ISO/IEC 20547 Big Data Reference Architecture
Chair ANSI/INCITS TC Big Data
Co-chair NIST Big Data Public Working Group Reference Architecture
First Robotic Mentor - FRC, FTC - www.iliterobotics.org
Board Member- USSTEM Foundation - www.usstem.org
The information contained in this message may be privileged
and/or confidential and protected from disclosure.
If the reader of this message is not the intended recipient
or an employee or agent responsible for delivering this message
to the intended recipient, you are hereby notified that any
dissemination, distribution or copying of this communication
is strictly prohibited. If you have received this communication
in error, please notify the sender immediately by replying to
this message and deleting the material from any computer.