[jira] [Created] (SPARK-45255) Spark connect client failing with java.lang.NoClassDefFoundError

["Faiz Halde (Jira)" <ji...@apache.org>]

Faiz Halde created SPARK-45255:
----------------------------------

             Summary: Spark connect client failing with java.lang.NoClassDefFoundError
                 Key: SPARK-45255
                 URL: https://issues.apache.org/jira/browse/SPARK-45255
             Project: Spark
          Issue Type: New Feature
          Components: Connect
    Affects Versions: 3.5.0
            Reporter: Faiz Halde


I have a very simple repo with the following dependency in `build.sbt`

```

{{libraryDependencies ++= Seq("org.apache.spark" %% "spark-connect-client-jvm" % "3.5.0")}}

```

A simple application

```

{{object Main extends App {}}
{{   val s = SparkSession.builder().remote("sc://localhost").getOrCreate()}}
{{   s.read.json("/tmp/input.json").repartition(10).show(false)}}
{{}}}

```

But when I run it, I get the following error

 

```

Exception in thread "main" java.lang.NoClassDefFoundError: org/sparkproject/connect/client/com/google/common/cache/CacheLoader
    at Main$.delayedEndpoint$Main$1(Main.scala:4)
    at Main$delayedInit$body.apply(Main.scala:3)
    at scala.Function0.apply$mcV$sp(Function0.scala:39)
    at scala.Function0.apply$mcV$sp$(Function0.scala:39)
    at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:17)
    at scala.App.$anonfun$main$1$adapted(App.scala:80)
    at scala.collection.immutable.List.foreach(List.scala:431)
    at scala.App.main(App.scala:80)
    at scala.App.main$(App.scala:78)
    at Main$.main(Main.scala:3)
    at Main.main(Main.scala)
Caused by: java.lang.ClassNotFoundException: org.sparkproject.connect.client.com.google.common.cache.CacheLoader
    at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
    ... 11 more

```

I know the connect does a bunch of shading during assembly so it could be related to that. This application is not started via spark-submit or anything. Neither under `SPARK_HOME` ( I guess that's the whole point of connect client )

 

I followed the doc exactly as described. Can somebody help?

BTW it did work if I copied the exact shading rules in my project but I wonder if that's the right thing to do?



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org