[GitHub] [hudi] rchamarthi commented on issue #1977: Error running hudi on aws glue

[GitBox <gi...@apache.org>]

rchamarthi commented on issue #1977:
URL: https://github.com/apache/hudi/issues/1977#issuecomment-706445482


   @kimberlyamandalu I have a version of this working, but with GlueDevEndpoints, which has pretty similar behavior. Here are the settings I have.
   
   * Extra jars should have the *full path* of the jar file
   
   ```
   --extra-jars "s3://mybucket/jars//hudi-spark-bundle_2.11-0.5.2-incubating.jar"
   ```
   
   Job configuration should include the following config paramters
   
   ```
   --conf "spark.serializer=org.apache.spark.serializer.KryoSerializer" --conf "spark.sql.hive.convertMetastoreParquet=false"
   ```
   
   Here's the sample pyspark command that I ran and the format I used is just "hudi"
   
   ```
   (inputDF1.write
    .format("hudi")
    .options(**hudi_options)
    .mode("overwrite")
    .save(args['target_s3_path']))
   ```
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org