spark.jars.packages not working in spark interpreter tutorial

[David Boyd <db...@incadencecorp.com>]

All:

    Trying to run the Spark Interpreter tutorial note.

The spark.conf paragraph which specifies spark.jars.packages runs clean.
But the next paragraph which tries to use the avro jar fails with a 
class not found for

org.apache.spark.sql.avro.AvroFileFormat.DefaultSource

Spark is set to run Per Note in scoped process.

There are no errors in the 
zeppelin-interpreter-spark-shared_process-zeppelin-dspcnode11.dspc.incadencecorp.com.log

Any thoughts would be appreciated.

Note Spark Basic Features tutorial works fine.


The base zeppelin log has this error:

>  INFO [2020-07-02 15:57:58,339] ({qtp923219673-142} 
> VFSNotebookRepo.java[save]:145) - Saving note 2F8KN6TKK to Spark 
> Tutorial/1. Spark Interpreter Introduction_2F8KN6TKK.zpln
>  INFO [2020-07-02 15:57:58,343] ({SchedulerFactory3} 
> AbstractScheduler.java[runJob]:125) - Job 20180530-222838_1995256600 
> started by scheduler RemoteInterpreter-spark-shared_process-2F8KN6TKK
>  INFO [2020-07-02 15:57:58,343] ({SchedulerFactory3} 
> Paragraph.java[jobRun]:388) - Run paragraph [paragraph_id: 
> 20180530-222838_1995256600, interpreter: 
> org.apache.zeppelin.spark.SparkInterpreter, note_id: 2F8KN6TKK, user: 
> dspc_demo]
>  INFO [2020-07-02 15:57:58,444] 
> ({JobStatusPoller-20180530-222838_1995256600} 
> NotebookServer.java[onStatusChange]:1927) - Job 
> 20180530-222838_1995256600 starts to RUNNING
>  INFO [2020-07-02 15:57:58,445] 
> ({JobStatusPoller-20180530-222838_1995256600} 
> VFSNotebookRepo.java[save]:145) - Saving note 2F8KN6TKK to Spark 
> Tutorial/1. Spark Interpreter Introduction_2F8KN6TKK.zpln
>  WARN [2020-07-02 15:57:58,734] ({SchedulerFactory3} 
> NotebookServer.java[onStatusChange]:1924) - Job 
> 20180530-222838_1995256600 is finished, status: ERROR, exception: 
> null, result: %text java.lang.ClassNotFoundException: Failed to find 
> data source: org.apache.spark.sql.avro.AvroFileFormat. Please find 
> packages at http://spark.apache.org/third-party-projects.html
>   at 
> org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSource(DataSource.scala:657)
>   at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:194)
>   at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:178)
>   ... 45 elided
> Caused by: java.lang.ClassNotFoundException: 
> org.apache.spark.sql.avro.AvroFileFormat.DefaultSource
>   at 
> scala.reflect.internal.util.AbstractFileClassLoader.findClass(AbstractFileClassLoader.scala:62)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
>   at 
> org.apache.spark.sql.execution.datasources.DataSource$$anonfun$20$$anonfun$apply$12.apply(DataSource.scala:634)
>   at 
> org.apache.spark.sql.execution.datasources.DataSource$$anonfun$20$$anonfun$apply$12.apply(DataSource.scala:634)
>   at scala.util.Try$.apply(Try.scala:192)
>   at 
> org.apache.spark.sql.execution.datasources.DataSource$$anonfun$20.apply(DataSource.scala:634)
>   at 
> org.apache.spark.sql.execution.datasources.DataSource$$anonfun$20.apply(DataSource.scala:634)
>   at scala.util.Try.orElse(Try.scala:84)
>   at 
> org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSource(DataSource.scala:634)
>   ... 47 more
>
>  INFO [2020-07-02 15:57:58,735] ({SchedulerFactory3} 
> VFSNotebookRepo.java[save]:145) - Saving note 2F8KN6TKK to Spark 
> Tutorial/1. Spark Interpreter Introduction_2F8KN6TKK.zpln


-- 
========= mailto:dboyd@incadencecorp.com ============
David W. Boyd
VP,  Data Solutions
10432 Balls Ford, Suite 240
Manassas, VA 20109
office:   +1-703-552-2862
cell:     +1-703-402-7908
============== http://www.incadencecorp.com/ ============
ISO/IEC JTC1 SC42/WG2, editor ISO/IEC 20546, ISO/IEC 20547-1
Chair INCITS TG Big Data
Co-chair NIST Big Data Public Working Group Reference Architecture
First Robotic Mentor - FRC, FTC - www.iliterobotics.org
Board Member- USSTEM Foundation - www.usstem.org

The information contained in this message may be privileged
and/or confidential and protected from disclosure.
If the reader of this message is not the intended recipient
or an employee or agent responsible for delivering this message
to the intended recipient, you are hereby notified that any
dissemination, distribution or copying of this communication
is strictly prohibited.  If you have received this communication
in error, please notify the sender immediately by replying to
this message and deleting the material from any computer.