You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/04/08 00:15:29 UTC
[GitHub] [hudi] ChenShuai1981 opened a new issue, #5258: [SUPPORT]
ChenShuai1981 opened a new issue, #5258:
URL: https://github.com/apache/hudi/issues/5258
**_Tips before filing an issue_**
- Have you gone through our [FAQs](https://hudi.apache.org/learn/faq/)?
- Join the mailing list to engage in conversations and get faster support at dev-subscribe@hudi.apache.org.
- If you have triaged this as a bug, then file an [issue](https://issues.apache.org/jira/projects/HUDI/issues) directly.
**Describe the problem you faced**
Following Spark Quick Started Guide with Spark v2.4.4 but met java.lang.NoSuchMethodError
**To Reproduce**
Steps to reproduce the behavior:
1. Download spark-2.4.4-bin-hadoop2.7.tgz and unzip it then config spark-env.sh with JAVA_HOME, SCALA_HOME, SPARK_HOME environment variables
2. Download hudi-spark-bundle_2.12-0.10.1.jar and spark-avro_2.12-2.4.4.jar from maven central repository and placed them in $SPARK_HOME/jars/ folder
3. cd to $SPARK_HOME and execute command `bin/spark-shell --jars ./jars/hudi-spark-bundle_2.12-0.10.1.jar,./jars/spark-avro_2.12-2.4.4.jar --conf 'spark.serializer=org.apache.spark.serializer.KryoSerializer' `
4. Follow the quick started guide https://hudi.apache.org/docs/quick-start-guide
5. After executed script of `df.write.format("hudi")...`, the terminal reported the following exception:
**Expected behavior**
It should be executed without exception and the sample data should be written well into hudi table.
**Environment Description**
* Hudi version : 0.10.1
* Spark version : 2.4.4
* Hive version : N/A
* Hadoop version : N/A
* Storage (HDFS/S3/GCS..) : LOCAL FILE
* Running on Docker? (yes/no) : no
**Additional context**
Add any other context about the problem here.
**Stacktrace**
```java.util.ServiceConfigurationError: org.apache.spark.sql.sources.DataSourceRegister: Provider org.apache.spark.sql.avro.AvroFileFormat could not be instantiated
at java.util.ServiceLoader.fail(ServiceLoader.java:232)
at java.util.ServiceLoader.access$100(ServiceLoader.java:185)
at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384)
at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
at scala.collection.convert.Wrappers$JIteratorWrapper.next(Wrappers.scala:43)
at scala.collection.Iterator$class.foreach(Iterator.scala:891)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at scala.collection.TraversableLike$class.filterImpl(TraversableLike.scala:247)
at scala.collection.TraversableLike$class.filter(TraversableLike.scala:259)
at scala.collection.AbstractTraversable.filter(Traversable.scala:104)
at org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSource(DataSource.scala:630)
at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:245)
at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:229)
... 68 elided
Caused by: java.lang.NoSuchMethodError: org.apache.spark.sql.execution.datasources.FileFormat.$init$(Lorg/apache/spark/sql/execution/datasources/FileFormat;)V
at org.apache.spark.sql.avro.AvroFileFormat.<init>(AvroFileFormat.scala:44)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at java.lang.Class.newInstance(Class.java:442)
at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380)
... 81 more```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan closed issue #5258: [SUPPORT] Write hudi data throws NoSuchMethodError with spark v2.4.4 and hudi v0.10.1
Posted by GitBox <gi...@apache.org>.
nsivabalan closed issue #5258: [SUPPORT] Write hudi data throws NoSuchMethodError with spark v2.4.4 and hudi v0.10.1
URL: https://github.com/apache/hudi/issues/5258
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #5258: [SUPPORT] Write hudi data throws NoSuchMethodError with spark v2.4.4 and hudi v0.10.1
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #5258:
URL: https://github.com/apache/hudi/issues/5258#issuecomment-1125375333
please re-open if you are having issues still. thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #5258: [SUPPORT] Write hudi data throws NoSuchMethodError with spark v2.4.4 and hudi v0.10.1
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #5258:
URL: https://github.com/apache/hudi/issues/5258#issuecomment-1100571599
can you try w/ scala 11 bundle and let us know if it succeeds.
hudi-spark-bundle_2.11-0.10.1.jar
and for spark-avro, can you try setting it via `--packages org.apache.spark:spark-avro_2.11:2.4.4`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #5258: [SUPPORT] Write hudi data throws NoSuchMethodError with spark v2.4.4 and hudi v0.10.1
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #5258:
URL: https://github.com/apache/hudi/issues/5258#issuecomment-1110470954
@ChenShuai1981 : do you have any updates on this regard?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org