You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/12/02 16:56:43 UTC
[GitHub] [airflow] scauglog opened a new issue #12760: KubernetesPodOperator can't mount sercret as volume
scauglog opened a new issue #12760:
URL: https://github.com/apache/airflow/issues/12760
**Apache Airflow version**: 1.10.12 and 2.0.0b3
**Kubernetes version (if you are using kubernetes)** (use `kubectl version`): 1.19.2
**Environment**:
- **Cloud provider or hardware configuration**: minikube
- **OS** (e.g. from /etc/os-release): Ubuntu 20.04
- **Kernel** (e.g. `uname -a`): 5.4.0-53
**What happened**:
With the kubernetePodOperator and KubernetesExecutor when I try to mount secret as volume the pod (the one in kubernetesPodOperator) didn't launch and the task return as failed. The error occur with either V1Volume object or airflow Secret object.
Persistent volume claim work perfectly.
**What you expected to happen**:
airflow launch a worker pod on kubernetes and the worker pod launch a pod on kubernetes
**How to reproduce it**:
```python
import airflow
from airflow import DAG
from kubernetes.client import models as k8s
from airflow.providers.cncf.kubernetes.operators.kubernetes_pod import KubernetesPodOperator
from airflow.operators.dummy_operator import DummyOperator
from airflow.kubernetes.secret import Secret
default_args = {
'owner': 'debug',
'depends_on_past': False,
'start_date': airflow.utils.dates.days_ago(1),
}
pvc_volume = k8s.V1Volume(
name='ml-data',
persistent_volume_claim=k8s.V1PersistentVolumeClaimVolumeSource(claim_name='ml-data')
)
secret_volume = k8s.V1Volume(
name='deploy-key',
secret=k8s.V1SecretVolumeSource(default_mode=600, secret_name="dvc-deploy-key")
)
pvc_volume_mount = k8s.V1VolumeMount(
name='ml-data', mount_path='/data/', sub_path=None, read_only=False
)
secret_volume_mount = k8s.V1VolumeMount(
name='deploy-key', mount_path='/root/.ssh', sub_path=None, read_only=True
)
secret_file = Secret(deploy_type='volume',
deploy_target='/root/.ssh/',
secret='dvc-deploy-key')
dag = DAG(
"testing",
default_args=default_args,
description='indexation train pipeline',
schedule_interval=None
)
start = DummyOperator(
task_id="start",
dag=dag
)
t1 = KubernetesPodOperator(
task_id=f't1',
name=f't1',
namespace='airflow',
image="busybox",
cmds=["echo", "main"],
volume_mounts=[pvc_volume_mount],
volumes=[pvc_volume],
is_delete_operator_pod=False,
get_logs=True,
dag=dag,
)
t10 = KubernetesPodOperator(
task_id=f't10',
name=f't10',
namespace='airflow',
image="busybox",
cmds=["echo", "main"],
secrets=secret_file,
volume_mounts=[pvc_volume_mount],
volumes=[pvc_volume],
is_delete_operator_pod=False,
get_logs=True,
dag=dag,
)
t2 = KubernetesPodOperator(
task_id=f't2',
name=f't2',
namespace='airflow',
image="busybox",
cmds=["echo", "main"],
volume_mounts=[secret_volume_mount],
volumes=[secret_volume],
is_delete_operator_pod=False,
get_logs=True,
dag=dag,
)
t3 = KubernetesPodOperator(
task_id=f't3',
name=f't3',
namespace='airflow',
image="busybox",
cmds=["echo", "main"],
volume_mounts=[pvc_volume_mount, secret_volume_mount],
volumes=[pvc_volume, secret_volume],
is_delete_operator_pod=False,
get_logs=True,
dag=dag,
)
start >> [t1, t10, t2, t3]
```
**Anything else we need to know**:
I'm not sure if it's an airflow issue or a kubernetes-client issue
How often does this problem occur? Once? Every time etc?
This problem occur every time
Any relevant logs to include? Put them here in side a detail tag:
<details><summary>worker.log</summary> airflow@testingpulldata:/opt/airflow$ airflow tasks run testing pull_data "2020-12-02T15:12:29.757041+00:00" --local --pool default_pool --subdir /opt/airflow/dags/test.py
[2020-12-02 15:30:19,007] {dagbag.py:440} INFO - Filling up the DagBag from /opt/airflow/dags/test.py
/home/airflow/.local/lib/python3.6/site-packages/airflow/providers/cncf/kubernetes/backcompat/backwards_compat_converters.py:26 DeprecationWarning: This module is deprecated. Please use `kub
ernetes.client.models.V1Volume`.
/home/airflow/.local/lib/python3.6/site-packages/airflow/providers/cncf/kubernetes/backcompat/backwards_compat_converters.py:27 DeprecationWarning: This module is deprecated. Please use `kub
ernetes.client.models.V1VolumeMount`.
Running <TaskInstance: testing.pull_data 2020-12-02T15:12:29.757041+00:00 [success]> on host testingpulldata
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jedcunningham commented on issue #12760: KubernetesPodOperator can't mount sercret as volume
Posted by GitBox <gi...@apache.org>.
jedcunningham commented on issue #12760:
URL: https://github.com/apache/airflow/issues/12760#issuecomment-796219133
@sstevens303 are you still having issues mounting secrets or can this be closed?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] sstevens303 commented on issue #12760: KubernetesPodOperator can't mount sercret as volume
Posted by GitBox <gi...@apache.org>.
sstevens303 commented on issue #12760:
URL: https://github.com/apache/airflow/issues/12760#issuecomment-796752380
This can be closed
________________________________
From: Jed Cunningham ***@***.***>
Sent: Wednesday, March 10, 2021 5:18 PM
To: apache/airflow ***@***.***>
Cc: Sharon Stevens ***@***.***>; Mention ***@***.***>
Subject: [External Origin] Re: [apache/airflow] KubernetesPodOperator can't mount sercret as volume (#12760)
@sstevens303<https://github.com/sstevens303> are you still having issues mounting secrets or can this be closed?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub<https://github.com/apache/airflow/issues/12760#issuecomment-796219133>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ARY5W5E27S3XZD7R4RTFTNDTC7O27ANCNFSM4UKYLRIA>.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] VBhojawala commented on issue #12760: KubernetesPodOperator can't mount sercret as volume
Posted by GitBox <gi...@apache.org>.
VBhojawala commented on issue #12760:
URL: https://github.com/apache/airflow/issues/12760#issuecomment-739995119
Hello @scauglog ,
I Just wrote KubernetesPodOperator Guide Here.https://github.com/VBhojawala/airflow/blob/k8s-docs/docs/apache-airflow-providers-cncf-kubernetes/operators.rst#mounting-secrets-as-volume
Example Dag in Guide runs fine and mounts Secret volume.
The worker.log given mentions Only DeprecationWarning which does not stop Pod from being launch IMHO.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] sstevens303 commented on issue #12760: KubernetesPodOperator can't mount sercret as volume
Posted by GitBox <gi...@apache.org>.
sstevens303 commented on issue #12760:
URL: https://github.com/apache/airflow/issues/12760#issuecomment-750944297
@VBhojawala I read your document and I was wondering how do map a volume. I want to do the equivalent of
docker run -v /host/directory:/container/directory using airflow and kubernetespodoperator. So I can take an existing directory path available to airflow and map it to a directory path inside the container. I cannot find any examples of this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil closed issue #12760: KubernetesPodOperator can't mount sercret as volume
Posted by GitBox <gi...@apache.org>.
kaxil closed issue #12760:
URL: https://github.com/apache/airflow/issues/12760
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] VBhojawala commented on issue #12760: KubernetesPodOperator can't mount sercret as volume
Posted by GitBox <gi...@apache.org>.
VBhojawala commented on issue #12760:
URL: https://github.com/apache/airflow/issues/12760#issuecomment-753276984
There is an example of mounting host volume to pod :
Here is the link https://github.com/VBhojawala/airflow/blob/k8s-docs/docs/apache-airflow-providers-cncf-kubernetes/operators.rst#mounting-persistent--volume-to-pod
In this example local machine's ``/tmp/myapp`` is mounted to pod
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org