[jira] [Comment Edited] (HUDI-1870) Move spark avro serialization class into hudi repo

["Raymond Xu (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HUDI-1870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17445536#comment-17445536 ] 

Raymond Xu edited comment on HUDI-1870 at 11/17/21, 11:02 PM:
--------------------------------------------------------------

when build with and change spark3.version to 3.0.[0-3]
{code:java}
➜ mvn clean install -DskipTests -Drat.skip -Djacoco.skip -Dcheckstyle.skip -P "scala-2.12,spark3,spark3.0.x"{code}
i ran into 
{code:java}
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.8.0:testCompile (default-testCompile) on project hudi-spark3_2.12: Compilation failure
[ERROR] /Users/rxu/Code/xushiyan/hudi/hudi-spark-datasource/hudi-spark3/src/test/java/org/apache/hudi/spark3/internal/TestReflectUtil.java:[55,40] cannot find symbol
[ERROR]   symbol:   method userSpecifiedCols()
[ERROR]   location: variable newStatment of type org.apache.spark.sql.catalyst.plans.logical.InsertIntoStatement{code}
the original issue resolved by [~biyan900116@gmail.com] in the 3.1 support PR. still some different issue remaining with 3.0.[0-3]

 


was (Author: xushiyan):
when build with and change spark3.version to 3.0.\{1,2,3}

 
{code:java}
➜ mvn clean install -DskipTests -Drat.skip -Djacoco.skip -Dcheckstyle.skip -Dscala-2.12 -Dspark3 -Dspark3.0.x{code}
i ran into 
{code:java}
Symbol 'type org.slf4j.Logger' is missing from the classpath.
[ERROR] This symbol is required by 'method org.apache.spark.internal.Logging.log'.
[ERROR] Make sure that type Logger is in your classpath and check for conflicting dependencies with `-Ylog-classpath`.
[ERROR] A full rebuild may help if 'Logging.class' was compiled against an incompatible version of org.slf4j.
[ERROR]     if (log.isDebugEnabled) {
[ERROR]         ^{code}
the original issue resolved by [~biyan900116@gmail.com] in the 3.1 support PR. still some different issue remaining with 3.0.\{1,2,3}

 

> Move spark avro serialization class into hudi repo
> --------------------------------------------------
>
>                 Key: HUDI-1870
>                 URL: https://issues.apache.org/jira/browse/HUDI-1870
>             Project: Apache Hudi
>          Issue Type: Improvement
>          Components: Spark Integration
>            Reporter: Gary Li
>            Assignee: XiaoyuGeng
>            Priority: Blocker
>              Labels: sev:critical
>             Fix For: 0.10.0
>
>
> in Spark 3.1.1, avro serialization-related class become private. We need to mvoe those classes into Hudi's repo.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)