[jira] [Created] (HUDI-1568) Issues w/ spark_bundle_2.12 : ClassNotFoundError for org/apache/spark/sql/execution/datasources/v2/FileDataSourceV2

["sivabalan narayanan (Jira)" <ji...@apache.org>]

sivabalan narayanan created HUDI-1568:
-----------------------------------------

             Summary: Issues w/ spark_bundle_2.12 : ClassNotFoundError for org/apache/spark/sql/execution/datasources/v2/FileDataSourceV2
                 Key: HUDI-1568
                 URL: https://issues.apache.org/jira/browse/HUDI-1568
             Project: Apache Hudi
          Issue Type: Bug
          Components: Spark Integration
            Reporter: sivabalan narayanan


I tried Quick Start with spark_bundle_2.12 and it fails w/ ClassNotFoundError for

org/apache/spark/sql/execution/datasources/v2/FileDataSourceV2. spark_bundle_2.11 works fine. 

 

```

scala> df.write.format("hudi").

     |   options(getQuickstartWriteConfigs).

     |   option(PRECOMBINE_FIELD_OPT_KEY, "ts").

     |   option(RECORDKEY_FIELD_OPT_KEY, "uuid").

     |   option(PARTITIONPATH_FIELD_OPT_KEY, "partitionpath").

     |   option(TABLE_NAME, tableName).

     |   mode(Overwrite).

     |   save(basePath)

java.lang.NoClassDefFoundError: org/apache/spark/sql/execution/datasources/v2/FileDataSourceV2

  at java.lang.ClassLoader.defineClass1(Native Method)

  at java.lang.ClassLoader.defineClass(ClassLoader.java:763)

  at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)

  at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)

  at java.net.URLClassLoader.access$100(URLClassLoader.java:74)

  at java.net.URLClassLoader$1.run(URLClassLoader.java:369)

  at java.net.URLClassLoader$1.run(URLClassLoader.java:363)

  at java.security.AccessController.doPrivileged(Native Method)

  at java.net.URLClassLoader.findClass(URLClassLoader.java:362)

  at java.lang.ClassLoader.loadClass(ClassLoader.java:424)

  at java.lang.ClassLoader.loadClass(ClassLoader.java:411)

  at java.lang.ClassLoader.loadClass(ClassLoader.java:357)

  at java.lang.Class.forName0(Native Method)

  at java.lang.Class.forName(Class.java:348)

  at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:370)

  at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)

  at java.util.ServiceLoader$1.next(ServiceLoader.java:480)

  at scala.collection.convert.Wrappers$JIteratorWrapper.next(Wrappers.scala:43)

  at scala.collection.Iterator$class.foreach(Iterator.scala:891)

  at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)

  at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)

  at scala.collection.AbstractIterable.foreach(Iterable.scala:54)

  at scala.collection.TraversableLike$class.filterImpl(TraversableLike.scala:247)

  at scala.collection.TraversableLike$class.filter(TraversableLike.scala:259)

  at scala.collection.AbstractTraversable.filter(Traversable.scala:104)

  at org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSource(DataSource.scala:630)

  at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:245)

  at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:229)

  ... 68 elided

Caused by: java.lang.ClassNotFoundException: org.apache.spark.sql.execution.datasources.v2.FileDataSourceV2

  at java.net.URLClassLoader.findClass(URLClassLoader.java:382)

  at java.lang.ClassLoader.loadClass(ClassLoader.java:424)

  at java.lang.ClassLoader.loadClass(ClassLoader.java:357)

  ... 96 more

 

```



--
This message was sent by Atlassian Jira
(v8.3.4#803005)