[jira] [Created] (AIRFLOW-6602) Make "executor_config" templated field to support dynamic parameters for kubernetes executor

["Jun Xie (Jira)" <ji...@apache.org>]

Jun Xie created AIRFLOW-6602:
--------------------------------

             Summary: Make "executor_config" templated field to support dynamic parameters for kubernetes executor
                 Key: AIRFLOW-6602
                 URL: https://issues.apache.org/jira/browse/AIRFLOW-6602
             Project: Apache Airflow
          Issue Type: New Feature
          Components: executor-kubernetes
    Affects Versions: 1.10.7
            Reporter: Jun Xie
            Assignee: Daniel Imberman


When running airflow with Kubernetes Executor, one specifies the configurations through 

"executor_config". At the moment, this field is not templated, meaning that we won't be able to have dynamic parameters. So I did an experiment that I created MyPythonOperator which inherits PythonOperator but with with "executor_config" added to template_fields. However, the result shows that this change itself isn't enough, because airflow first creates a Pod based on executor_config without rendering it, and then run the task inside the pod (the running will trigger the Jinja template rendering)

See an example config below showing a use case where one can mount dynamic "subPath" to the image

```

executor_config = {
 "KubernetesExecutor": {
 "image": "...",
 "request_memory": "2Gi",
 'request_cpu': '1',
 "volumes": [
 {
 "name": "data",
 "persistentVolumeClaim": \{"claimName": "some_claim_name"},
 },
 ],
 "volume_mounts": [
 {
 "mountPath": "/code",
 "name": "data",
 "subPath": "\{{ ds }}"
 },
 ]
 }
}

```

 

I have then did a further experiment that in 

trigger_tasks() from airflow/executors/base_executor.py, right before execute_async() is called, I called simple_ti.render_templates() which will trigger the rendering, so the kubernetes_executor.execute_async() will pick up the resolved parameters

 

I think this is a very useful feature to include into Airflow



--
This message was sent by Atlassian Jira
(v8.3.4#803005)