[GitHub] [hudi] Gatsby-Lee commented on issue #1751: [SUPPORT] Hudi not working with Spark 3.0.0

[GitBox <gi...@apache.org>]

Gatsby-Lee commented on issue #1751:
URL: https://github.com/apache/hudi/issues/1751#issuecomment-968173564


   For someone who gets here due to the issue with AWS Glue 3.0 + Hudi Connector.
   
   """# 2021-11-13 status"""
   AWS Glue 3.0 fails to load Image from ECR for Hudi Connector dependencies.
   If you don't need Hudi Schema Evolution, then go with AWS Glue 2.0.
   If you need Hudi Schema Evolution, then you have to use AWS Glue 3.0 otherwise you will see the issue related to Glue Catalog caused by out-dated EMRFS.
   
   You can still use AWS Glue 3.0 + Hudi by adding Hudi JARs dependencies by yourself instead Glue Connector does it for you.
   
   you need four dependencies.
   - https://mvnrepository.com/artifact/org.apache.spark/spark-avro_2.12/3.1.1
   - https://mvnrepository.com/artifact/org.apache.hudi/hudi-spark3-bundle_2.12/0.9.0
   - https://mvnrepository.com/artifact/org.apache.calcite/calcite-core/1.16.0
   - https://repo1.maven.org/maven2/org/apache/thrift/libfb303/0.9.3/
   
   I hope this can help someone. I was stuck in this for a day.
   
   
    


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org