You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2022/01/07 15:40:00 UTC
[jira] [Updated] (HUDI-3195) optimize spark3 pom and modify build command
[ https://issues.apache.org/jira/browse/HUDI-3195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ASF GitHub Bot updated HUDI-3195:
---------------------------------
Labels: pull-request-available (was: )
> optimize spark3 pom and modify build command
> --------------------------------------------
>
> Key: HUDI-3195
> URL: https://issues.apache.org/jira/browse/HUDI-3195
> Project: Apache Hudi
> Issue Type: Bug
> Components: Spark Integration
> Reporter: Yann Byron
> Assignee: Yann Byron
> Priority: Major
> Labels: pull-request-available
>
> when use `mvn clean install -P "scala-2.12,spark3,spark3.1.x"` to build spark and write data by spark, error is as follows:
>
>
> {code:java}
> ERROR Executor: Exception in task 0.0 in stage 26.0 (TID 2005)
> java.lang.RuntimeException: org.apache.hudi.exception.HoodieException: org.apache.hudi.exception.HoodieException: java.util.concurrent.ExecutionException: java.lang.NoClassDefFoundError: org/apache/parquet/schema/LogicalTypeAnnotation$LogicalTypeAnnotationVisitor
> at org.apache.hudi.client.utils.LazyIterableIterator.next(LazyIterableIterator.java:121)
> at scala.collection.convert.Wrappers$JIteratorWrapper.next(Wrappers.scala:44)
> at scala.collection.Iterator$$anon$11.nextCur(Iterator.scala:484)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:490)
> at org.apache.spark.storage.memory.MemoryStore.putIterator(MemoryStore.scala:221) {code}
>
>
> That's due to the use of 1.12.1 parquet.
>
> Also, we use this build command in CI, which will mislead users.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)