[GitHub] [hudi] xushiyan commented on issue #3831: Deltastreamer through Pyspark/livy

[GitBox <gi...@apache.org>]

xushiyan commented on issue #3831:
URL: https://github.com/apache/hudi/issues/3831#issuecomment-974689954


   @Kavin88 so basically it is a feature request where you wish to pass these deltastreamer-specific arguments via options. It is indeed possible to create a set of configs say `hoodie.deltastreamer.xxx` to achieve this. But first i want to understand why it is not possible for you to submit args via Livy API. from what i read, https://community.cloudera.com/t5/Community-Articles/How-to-Submit-Spark-Application-through-Livy-REST-API/ta-p/247502, livy does allow you to specify args to the spark app.
   
   > My thought is to implement this through python code with sparksession and do the livy submit calling the pyfile.
   
   Not quite sure what this pyfile is supposed to contain. Can you elaborate?
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org