You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/06/04 23:26:12 UTC

[GitHub] [airflow] jaketf opened a new issue #9146: DataprocSubmitPySparkJobOperator main parameter should be templated

jaketf opened a new issue #9146:
URL: https://github.com/apache/airflow/issues/9146


   
   **Description**
   [`DataprocSubmitPySparkJobOperator.template_fields`](
   https://github.com/apache/airflow/blob/master/airflow/providers/google/cloud/operators/dataproc.py#L1259) does not contain main.
   
   **Use case / motivation**
   
   Allow user to render the path to main pyspark file based on an airflow variable at runtime rather than dag parse time. This is helpful in keeping user code the same and bumping a versioned path as the airflow variable w/o touching the dag source. 
   
   Right now the alternative is using `Variable.get` in the dag file which gets evaluated at parse time not runtime (if I'm not mistaken)
   This may cause unexpected behavior if the variable is changed between parse time and runtime. 
   
   **Related Issues**
   N/A


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] kaxil closed issue #9146: DataprocSubmitPySparkJobOperator main parameter should be templated

Posted by GitBox <gi...@apache.org>.
kaxil closed issue #9146:
URL: https://github.com/apache/airflow/issues/9146


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] jaketf commented on issue #9146: DataprocSubmitPySparkJobOperator main parameter should be templated

Posted by GitBox <gi...@apache.org>.
jaketf commented on issue #9146:
URL: https://github.com/apache/airflow/issues/9146#issuecomment-639168938


   CC: @mik-laj 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org