[jira] [Commented] (HUDI-1870) Move spark avro serialization class into hudi repo

["Robert Dunton (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HUDI-1870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17412198#comment-17412198 ] 

Robert Dunton commented on HUDI-1870:
-------------------------------------

This issue seems to affect building Huid with any Spark 3.0.X version aside from 3.0.0, not just any 3.1 version.  I was working on compiling Huid with the latest dot releases for Spark, Hive, and Hadoop (based on Hudi parent pom in 0.9) on arm64 w/ Scala 2.12 and getting {{}}
{code:java}
error: class IncompatibleSchemaException in package avro cannot be accessed in package org.apache.spark.sql.avro{code}
for anything *NOT* exactly Spark 3.0.0, eg -

 
 * Spark 3.0.3, Hive 2.3.7, Hadoop 2.7.7 in Scala 2.12 - Fails ->[https://gitlab.com/therackio/big-data/binaries/apache-hudi-bin-arm64/-/pipelines/367542690]
 * Spark 3.0.2, Hive 2.3.7, Hadoop 2.7.7 in Scala 2.12 - Fails ->[https://gitlab.com/therackio/big-data/binaries/apache-hudi-bin-arm64/-/pipelines/367547796]
 * Spark 3.0.1, Hive 2.3.7, Hadoop 2.7.7 in Scala 2.12 - Fails ->https://gitlab.com/therackio/big-data/binaries/apache-hudi-bin-arm64/-/pipelines/367548060
 * Spark 3.0.0, Hive 2.3.7, Hadoop 2.7.7 in Scala 2.12 - WORKS ->https://gitlab.com/therackio/big-data/binaries/apache-hudi-bin-arm64/-/pipelines/367548118

> Move spark avro serialization class into hudi repo
> --------------------------------------------------
>
>                 Key: HUDI-1870
>                 URL: https://issues.apache.org/jira/browse/HUDI-1870
>             Project: Apache Hudi
>          Issue Type: Improvement
>            Reporter: Gary Li
>            Priority: Major
>
> in Spark 3.1.1, avro serialization-related class become private. We need to mvoe those classes into Hudi's repo.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)