[GitHub] [hudi] bhasudha commented on issue #1823: [SUPPORT] MOR trigger compaction from Hudi CLI

[GitBox <gi...@apache.org>]

bhasudha commented on issue #1823:
URL: https://github.com/apache/hudi/issues/1823#issuecomment-657297717


   @RajasekarSribalan  For your first question, unfortunately currently in Spark Streaming writes only support inline compaction is supported. So you have to enable that config. Good news is, this [PR](https://github.com/apache/hudi/pull/1752) is working on enabling the async compaction for Spark Streaming  and is in priority. 
   
   For second question, couple clarifications.
   - Hudi moved to Spark 2.4. I see that you are using spark 2.2.0. Could you try on spark 2.4* ? 
   - Also, in your spark submit command are you passing in these jars and cones - https://hudi.apache.org/docs/quick-start-guide.html#setup
   
   1. The conf `'spark.serializer=org.apache.spark.serializer.KryoSerializer'`
   2. in addition to hudi-spark-bundle,  you need to pass `,org.apache.spark:spark-avro_2.11:2.4.4` Note the spark-avro must match your spark version which is 2.4.4.  This applies if you are using spark-shell as it does not ship with spark-avro explicitly.
   
   
   
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org