[jira] [Commented] (AIRFLOW-2964) Lazy generation of the job description

["Andrew Chen (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/AIRFLOW-2964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16593997#comment-16593997 ] 

Andrew Chen commented on AIRFLOW-2964:
--------------------------------------

This certainly makes a lot of sense to me, especially if we can maintain the current behavior (use the callable only if it is defined). That being said, maybe it's a good idea to get some input from the community whether or not this is the right way to do things. A quick spot check of other operators seems to suggest they don't support this sort of lazy generation of parameters.

> Lazy generation of the job description
> --------------------------------------
>
>                 Key: AIRFLOW-2964
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-2964
>             Project: Apache Airflow
>          Issue Type: Improvement
>          Components: contrib, operators
>    Affects Versions: Airflow 1.9.0, 1.10
>            Reporter: Victor Jimenez
>            Priority: Major
>
> When instantiating a {{DatabricksSubmitRunOperator}} users need to pass the description of the job that will later be executed on Databricks.
> The job description is only needed at execution time (when the hook is called). However, the {{json}} parameter must already have the full job description when constructing the operator. This may present a problem if computing the job description needs to execute expensive operations (e.g., querying a database). The expensive operation will be invoked every single time the DAG is reprocessed (which may happen quite frequently).
> It would be good to have an equivalent mechanism to the {{python_callable}} parameter in the {{PythonOperator}}. In this way, users could pass a function that would generate the job description only when the operator is actually executed.
> [~andrewmchen] Is there any other way to do this? If not, does this sound reasonable? I can create a PR implementing this proposal.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)