[jira] [Updated] (AIRFLOW-3986) PythonOperator - allow lazy loading of python_callable

["Ricardo Garcia Silva (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/AIRFLOW-3986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ricardo Garcia Silva updated AIRFLOW-3986:
------------------------------------------
    Description: 
It would be nice if the `PythonOperator` (and its derived operators too) would accept a string as the `python_callable` argument. This string would then be properly resolved and the respective callable be imported only when the operator is going to actually execute its work. For extra nicety, this string would be templated too. This would allow a usage like this:

{code:python} 
# somewhere in the DAG definition file
my_task = PythonOperator(
    task_id="do_stuff",
    python_callable=(
       "{{ conf.AIRFLOW_HOME }}/processing/mymodule.py:do_stuff_caller"
    ),
    # other usual parameters
)
{code}

Implementation seems very non-disruptive. However, I believe some code in the UI also needs tweaking in order to prevent (or force, according to customization) showing the actual code).

{code:python} 
class PythonOperator(BaseOperator):

    def execute(self, context):
         # other code (not relevant now) here
         python_callable = _import_callable(self.python_callable, context)
         return_value = python_callable(*self.op_args, **self.op_kwargs)

    def _import_callable(python_callable_path: str, context: Dict):
        # - render the `python_callable` string with the input `context`
        # - use python's importing mechanism to import the rendered string
        raise NotImplementedError
{code}

  was:
It would be nice if the `PythonOperator` (and its derived operators too) would accept a string as the `python_callable` argument. This string would then be properly resolved and the respective callable be imported only when the operator is going to actually execute its work. For extra nicety, this string would be templated too. This would allow a usage like this:

```
# somewhere in the DAG definition file
my_task = PythonOperator(
    task_id="do_stuff",
    python_callable="{{ conf.AIRFLOW_HOME }}/processing/mymodule.py:do_stuff_caller",
    # other usual parameters

)
```

Implementation seems very non-disruptive. However, I believe some code in the UI also needs tweaking in order to prevent (or force, according to customization) showing the actual code).

```

class PythonOperator(BaseOperator):


    def execute(self, context):
        # other code (not relevant now) here
        python_callable = _import_callable(self.python_callable, context)
        return_value = python_callable(*self.op_args, **self.op_kwargs)

def _import_callable(python_callable_path: str, context: Dict):
    # - render the `python_callable` string with the input `context`
    # - use python's importing mechanism to import the rendered string
    """
    raise NotImplementedError
```


> PythonOperator - allow lazy loading of python_callable
> ------------------------------------------------------
>
>                 Key: AIRFLOW-3986
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-3986
>             Project: Apache Airflow
>          Issue Type: Improvement
>          Components: operators
>    Affects Versions: 1.10.2
>            Reporter: Ricardo Garcia Silva
>            Priority: Trivial
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> It would be nice if the `PythonOperator` (and its derived operators too) would accept a string as the `python_callable` argument. This string would then be properly resolved and the respective callable be imported only when the operator is going to actually execute its work. For extra nicety, this string would be templated too. This would allow a usage like this:
> {code:python} 
> # somewhere in the DAG definition file
> my_task = PythonOperator(
>     task_id="do_stuff",
>     python_callable=(
>        "{{ conf.AIRFLOW_HOME }}/processing/mymodule.py:do_stuff_caller"
>     ),
>     # other usual parameters
> )
> {code}
> Implementation seems very non-disruptive. However, I believe some code in the UI also needs tweaking in order to prevent (or force, according to customization) showing the actual code).
> {code:python} 
> class PythonOperator(BaseOperator):
>     def execute(self, context):
>          # other code (not relevant now) here
>          python_callable = _import_callable(self.python_callable, context)
>          return_value = python_callable(*self.op_args, **self.op_kwargs)
>     def _import_callable(python_callable_path: str, context: Dict):
>         # - render the `python_callable` string with the input `context`
>         # - use python's importing mechanism to import the rendered string
>         raise NotImplementedError
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)