[GitHub] [hudi] kimberlyamandalu edited a comment on issue #1977: Error running hudi on aws glue

[GitBox <gi...@apache.org>]

kimberlyamandalu edited a comment on issue #1977:
URL: https://github.com/apache/hudi/issues/1977#issuecomment-706402272


   To give some more info, in my glue job, i add the following parameters:
   
   --extra-jars": s3://my_s3_bucket/hudi_jars/
   --conf: "spark.serializer=org.apache.spark.serializer.KryoSerializer --conf spark.sql.hive.convertMetastoreParquet=false"
   
   I have also tested with --user-jars-first: true but that did not help
   
   below is how i build my test data. 
   
   test_json = [
    {
      "key": "test1"
    },
    {
      "key": "test2"
    }
   ]
   
   rdd = sc.parallelize(test_json)
   df = spark.read.json(rdd)
   
   Below are the jars in my s3 location
     spark-avro_2.11-2.4.4.jar
     hudi-spark-bundle_2.11-0.6.1-SNAPSHOT.jar
     httpclient-4.5.13.jar


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org