You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/09/08 07:39:38 UTC
[GitHub] [airflow] gardnerdev opened a new issue #10793: Mounting DAGS from an externally populated PVC doesn't work in K8 Executor
gardnerdev opened a new issue #10793:
URL: https://github.com/apache/airflow/issues/10793
**Apache Airflow version**: 1.10.12
**Kubernetes version (if you are using kubernetes)** (use `kubectl version`): 1.18
**What happened**: Mounting DAGS from an externally populated PVC doesn't work:
```
--set dags.persistence.enabled=true \
--set dags.persistence.existingClaim=my-volume-claim
--set dags.gitSync.enabled=false
```
Envionment variables from K8 Executor worker
> │ Environment:
> ││ AIRFLOW_HOME: /opt/airflow ││ AIRFLOW__CORE__DAGS_FOLDER: /opt/airflow/dags/repo/
> ││ AIRFLOW__CORE__DAG_CONCURRENCY: 5 │
> │ AIRFLOW__CORE__EXECUTOR: LocalExecutor │
> │ AIRFLOW__CORE__FERNET_KEY: <set to the key 'fernet-key' in secret 'airflow-fernet-key'> Optional: false │
> │ AIRFLOW__CORE__PARALLELISM: 5 │
> │ AIRFLOW__CORE__SQL_ALCHEMY_CONN: <set to the key 'connection' in secret 'airflow-airflow-metadata'> Optional: false │
> │ AIRFLOW__KUBERNETES__DAGS_VOLUME_SUBPATH: repo/
<!-- (please include exact error messages if you can) -->
**What you expected to happen**: Dags mounted in workers from PVC
<!-- What do you think went wrong? -->
**How to reproduce it**: Use chart from master and set variables as above
<!---
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] gardnerdev commented on issue #10793: Mounting DAGS from an externally populated PVC doesn't work in K8 Executor
Posted by GitBox <gi...@apache.org>.
gardnerdev commented on issue #10793:
URL: https://github.com/apache/airflow/issues/10793#issuecomment-689699882
@dimberman @kaxil where can I found worker definition of K8 executor in charts?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dimberman commented on issue #10793: Mounting DAGS from an externally populated PVC doesn't work in K8 Executor
Posted by GitBox <gi...@apache.org>.
dimberman commented on issue #10793:
URL: https://github.com/apache/airflow/issues/10793#issuecomment-692875673
@gardnerdev can you please post the entire yaml for the worker? If it was mounted via PVC I imagine that would be in the volume mounts?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #10793: Mounting DAGS from an externally populated PVC doesn't work in K8 Executor
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #10793:
URL: https://github.com/apache/airflow/issues/10793
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on issue #10793: Mounting DAGS from an externally populated PVC doesn't work in K8 Executor
Posted by GitBox <gi...@apache.org>.
kaxil commented on issue #10793:
URL: https://github.com/apache/airflow/issues/10793#issuecomment-689195627
@gardnerdev Can you confirm if this worked before Airflow 1.10.12?
cc @dimberman
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #10793: Mounting DAGS from an externally populated PVC doesn't work in K8 Executor
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #10793:
URL: https://github.com/apache/airflow/issues/10793#issuecomment-689509156
Is that correct?:
```
AIRFLOW__CORE__EXECUTOR: LocalExecutor
```
Should not that be KubernetesExecutor?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] cheunhong commented on issue #10793: Mounting DAGS from an externally populated PVC doesn't work in K8 Executor
Posted by GitBox <gi...@apache.org>.
cheunhong commented on issue #10793:
URL: https://github.com/apache/airflow/issues/10793#issuecomment-735591985
@dimberman I have faced this issue today too, solved it by modifying this line https://github.com/apache/airflow/blob/bb00f164da6c84f078e66582217832aeedb2a966/chart/files/pod-template-file.kubernetes-helm-yaml#L68
Should have been ```{{- if .Values.dags.gitSync.enabled }} ```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #10793: Mounting DAGS from an externally populated PVC doesn't work in K8 Executor
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #10793:
URL: https://github.com/apache/airflow/issues/10793#issuecomment-689515035
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] gardnerdev commented on issue #10793: Mounting DAGS from an externally populated PVC doesn't work in K8 Executor
Posted by GitBox <gi...@apache.org>.
gardnerdev commented on issue #10793:
URL: https://github.com/apache/airflow/issues/10793#issuecomment-689686708
@kaxil yes, airflow 1.10.10 behaves in the same way
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on issue #10793: Mounting DAGS from an externally populated PVC doesn't work in K8 Executor
Posted by GitBox <gi...@apache.org>.
kaxil commented on issue #10793:
URL: https://github.com/apache/airflow/issues/10793#issuecomment-689512665
No that is correct, the Kubernetes worker runs the tasks locally with LocalExecutor.
Scheduler uses Kubernetes Executor to launch a pod, but ince launched that pod uses local exexutor
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org