[jira] [Updated] (AIRFLOW-6953) KubernetesExecutor: ba able to mount volumes in all worker pods

["Nicolas (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/AIRFLOW-6953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Nicolas updated AIRFLOW-6953:
-----------------------------
    Summary: KubernetesExecutor: ba able to mount volumes in all worker pods   (was: Kubernetes: Mount Volumes in all worker pods )

> KubernetesExecutor: ba able to mount volumes in all worker pods 
> ----------------------------------------------------------------
>
>                 Key: AIRFLOW-6953
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-6953
>             Project: Apache Airflow
>          Issue Type: Improvement
>          Components: configuration
>    Affects Versions: 1.10.0, 1.10.1, 1.10.2, 1.10.3, 1.10.4, 1.10.5, 1.10.6, 1.10.7, 1.10.8, 1.10.9
>         Environment: Linux (Debian), on K8S
>            Reporter: Nicolas
>            Priority: Major
>
> As far as I can tell, there is no easy way to mount one or more volumes into all worker pods when using the {{KubernetesExecutor}}. Though most of the code infrastructure is there (add to address AIRFLOW-3022) one has to add this kind of snippet of code in all his/her DAG to be able to mount a volume systematically.
> {code:python}
> volume_config = {
>     'nfs': {'path': '/',
>             'server': 'fs-23d987e8.efs.eu-west-1.amazonaws.com',
>             },
> }
> default_args = {
>     # Other args ...
>     'executor_config': {
>         'KubernetesExecutor': {
>             'volumes': [{'name': 'nfs-synchronised-dags', **volume_config}],
>             'volume_mounts': [{'name':'nfs-synchronised-dags',
>                                'mountPath': '/usr/local/airflow/dags',
>                                'readOnly': True,
>                                'subPath': None,
>                                },
>                               ],
>         },
>     },
> }
> {code}
> Not very DRY.
> This makes it particularly difficult to use methods other than git-sync or a PVC to mount a DAG volume. On the long run allowing this could also great simplify the config: instead of having many config stanzas for specific case, you just declare volumes, volumes
>  mount and on happens to be a DAG volume, it is only a matter to setting the right path into dags_folder. I really don't understand why Airflow cares so much about PVC and such, that's K8S's deal.
> It would be a matter of adding these sections to Airflow's configuration:
> {code:java}
> [kubernetes_volumes]
> # e.g. for an NFS volume
> <volume name1> = {"nfs": {"path": "/", "server": "my-server.provider.could"}}
> <volume name2> = ...
> [kubernetes_volume_mounts]
> <volume name1> = {"mountPath": "/usr/local/airflow/dags", "readOnly": true, "subPath": null}
> <volume name2> = ..{code}
> The key is the name of the volume and the value is the volume spec, as a JSON document.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)