[jira] [Created] (HUDI-3195) optimize spark3 pom and modify build command

["Yann Byron (Jira)" <ji...@apache.org>]

Yann Byron created HUDI-3195:
--------------------------------

             Summary: optimize spark3 pom and modify build command
                 Key: HUDI-3195
                 URL: https://issues.apache.org/jira/browse/HUDI-3195
             Project: Apache Hudi
          Issue Type: Bug
          Components: Spark Integration
            Reporter: Yann Byron


when use `mvn clean install -P "scala-2.12,spark3,spark3.1.x"` to build spark and write data by spark, error is as follows: 

 

 
{code:java}
ERROR Executor: Exception in task 0.0 in stage 26.0 (TID 2005)
java.lang.RuntimeException: org.apache.hudi.exception.HoodieException: org.apache.hudi.exception.HoodieException: java.util.concurrent.ExecutionException: java.lang.NoClassDefFoundError: org/apache/parquet/schema/LogicalTypeAnnotation$LogicalTypeAnnotationVisitor
    at org.apache.hudi.client.utils.LazyIterableIterator.next(LazyIterableIterator.java:121)
    at scala.collection.convert.Wrappers$JIteratorWrapper.next(Wrappers.scala:44)
    at scala.collection.Iterator$$anon$11.nextCur(Iterator.scala:484)
    at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:490)
    at org.apache.spark.storage.memory.MemoryStore.putIterator(MemoryStore.scala:221) {code}
 

 

That's due to the use of 1.12.1 parquet.

 

Also, we use this build command in CI, which will mislead users.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)