[jira] [Assigned] (HUDI-1512) Fix hudi-spark2 unit tests failure with Spark 3.0.0

["Wenning Ding (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HUDI-1512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Wenning Ding reassigned HUDI-1512:
----------------------------------

    Assignee: Wenning Ding

> Fix hudi-spark2 unit tests failure with Spark 3.0.0 
> ----------------------------------------------------
>
>                 Key: HUDI-1512
>                 URL: https://issues.apache.org/jira/browse/HUDI-1512
>             Project: Apache Hudi
>          Issue Type: Bug
>            Reporter: Wenning Ding
>            Assignee: Wenning Ding
>            Priority: Major
>
> hudi-spark2 unit tests failed when running with Spark 3.0.0:
> {code:java}
> mvn clean install -Dspark3 -pl hudi-spark-datasource/hudi-spark2{code}
> Threw a class not found error.
> {code:java}
> java.lang.NoClassDefFoundError: org/spark_project/guava/cache/CacheBuilder
> 	at org.apache.hudi.internal.TestHoodieBulkInsertDataInternalWriter.testGlobalFailure(TestHoodieBulkInsertDataInternalWriter.java:97)
> {code}
> When enabling spark3 profile, in hudi-spark2 module, some of the test code is compiled with Spark 3 and run with Spark 2. The above error is also because of this incompatibility.
> To solve this compatibility issue is kind of hard, but we can skip hudi-spark2 unit testing when enable spark3 profile since hudi-spark2 code is actually not used in the Spark 3.0.0 environment.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)