Question: Adding Dependencies in with the Spark Interpreter with Kubernetes

[Hetul Patel <he...@outlier.ai>]

Hi all,

I've been trying the 0.9.0-preview1 build on minikube with the spark
interpreter. It's working, but I'm unable to work with any dependencies
that I've added to the spark interpreter.

(Note: I had to add `SPARK_SUBMIT_OPTIONS=--conf spark.jars.ivy=/tmp/.ivy`
and `SPARK_USER=root` to the default interpreter options.)

I'm trying to connect spark to Cassandra, and I've added the following
dependency to the spark interpreter: `
com.datastax.cassandra:cassandra-driver-core:3.9.0`.

In the `zeppelin-server` pod logs, I see this:

```
 INFO [2020-05-13 02:52:15,840] ({Thread-18}
InterpreterSetting.java[run]:953) - Start to download dependencies for
interpreter: spark
 INFO [2020-05-13 02:52:21,565] ({Thread-18}
InterpreterSetting.java[run]:966) - Finish downloading dependencies for
interpreter: spark
 INFO [2020-05-13 02:52:21,565] ({Thread-18}
InterpreterSetting.java[setStatus]:740) - Set interpreter spark status to
READY
```

However, when a run a cell, I don't see any note of dependencies being
downloaded to the actual spark pod, and I get this error:

```
<console>:23: error: object datastax is not a member of package com import
com.datastax.driver.core.Cluster
```

Are dependency downloads supported with zeppelin and spark over kubernetes?
Or am I required to add the dependency jars directly to my spark docker
image and add them to the classpath?

Thanks,
Hetu

Re: Question: Adding Dependencies in with the Spark Interpreter with Kubernetes

[Sebastian Albrecht <al...@googlemail.com>]

Am Mi., 13. Mai 2020 um 21:59 Uhr schrieb Hetul Patel <he...@outlier.ai>:

>
> Are dependency downloads supported with zeppelin and spark over
> kubernetes? Or am I required to add the dependency jars directly to my
> spark docker image and add them to the classpath?
>
>
Hi Hetu,
i don't use docker but to connect to my cassandra from the spark cluster i
have to set SPARK_SUBMIT_OPTIONS='--packages
com.datastax.spark:spark-cassandra-connector_2.11:2.4.3'

HTH,
Sebastian.


> Thanks,
> Hetu
>