You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@airflow.apache.org by Anton Mushin <An...@epam.com> on 2018/05/14 10:46:04 UTC

Spark submit operator doesn't support properties-file

Hi all,
I see that spark submit operator [0] doesn't support properties-file[1] parameter. 
I need resolve properties from file for my job and now I'm reading properties line-by-line from file and build conf property to spark submit. It is not user friendly for me.
Do community have objections to  implement this property in spark submit hook? 
I would like create issue on Jira and Implement it, if I can.

[0] https://github.com/apache/incubator-airflow/blob/master/airflow/contrib/hooks/spark_submit_hook.py#L202) 
[1] https://jaceklaskowski.gitbooks.io/mastering-apache-spark/content/spark-submit.html#properties-file   

Best Regards,
Anton