You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/05/27 21:00:39 UTC
[GitHub] [airflow] danielcrisap opened a new issue #16131: Missing dags mount in WebServer
danielcrisap opened a new issue #16131:
URL: https://github.com/apache/airflow/issues/16131
**Apache Airflow version**: 2.1.0
**Kubernetes version**: `Server Version: version.Info{Major:"1", Minor:"19+", GitVersion:"v1.19.8-eks-96780e", GitCommit:"96780e1b30acbf0a52c38b6030d7853e575bcdf3", GitTreeState:"clean", BuildDate:"2021-03-10T21:32:29Z", GoVersion:"go1.15.8", Compiler:"gc", Platform:"linux/amd64"}`
**Environment**:
- **Cloud provider or hardware configuration**: AWS
- **OS** (e.g. from /etc/os-release):
- **Kernel** (e.g. `uname -a`):
- **Install tools**: Helm
- **Others**:
- **Helm Chart Version:** 1.0.0
- **Kubernetes:** EKS 1.19
**What happened**:
Hello 👋 Everyone,
I'm new using Airflow, but I notice that my webservers are not mounting my dags with gitSync.
**What you expected to happen**:
It’s expect to have a volume mounting with my dags from a git repo.
**How to reproduce it**:
`helm install airflow/airflow --values values.yaml -n airflow`
`values.yaml` content.
```yaml
fullnameOverride: ${app_name}
airflowVersion: "2.1.0"
images:
airflow:
repository: ${private_repo}
tag: "2.1.0"
pullPolicy: Always
env:
- name: "PYTHONPATH"
value: "/opt/airflow/dags/repo/dags:/opt/airflow/plugins"
extraEnvFrom: |
- secretRef:
name: 'airflow-creds'
data:
metadataConnection:
protocol: postgresql
port: 5432
db: ${database}
user: ${username}
host: ${rendpoint}
sslmode: disable
ingress:
enabled: true
web:
annotations:
kubernetes.io/ingress.class: "alb"
alb.ingress.kubernetes.io/scheme: "internet-facing"
alb.ingress.kubernetes.io/group.name: "external"
alb.ingress.kubernetes.io/certificate-arn: ${acm_arn}
alb.ingress.kubernetes.io/target-type: "ip"
alb.ingress.kubernetes.io/healthcheck-path: /health
alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS":443}]'
alb.ingress.kubernetes.io/actions.ssl-redirect: '{"Type": "redirect", "RedirectConfig": { "Protocol": "HTTPS", "Port": "443", "StatusCode": "HTTP_301"}}'
path: "" # or /*
host: ${url}
precedingPaths:
- path: /*
serviceName: ssl-redirect
servicePort: use-annotation
succeedingPaths: []
statsd:
enabled: false
postgresql:
enabled: false
flower:
enabled: false
redis:
persistence:
enabled: false
# Airflow Worker Config
workers:
serviceAccount:
annotations:
"eks.amazonaws.com/role-arn": ${iam_role_arn}
persistence:
enabled: false
# Airflow WebServer Config
webserver:
serviceAccount:
annotations:
"eks.amazonaws.com/role-arn": ${iam_role_arn}
defaultUser:
enabled: false
# Airflow scheduler settings
scheduler:
serviceAccount:
annotations:
"eks.amazonaws.com/role-arn": ${iam_role_arn}
dags:
gitSync:
enabled: true
repo: https://github.com/MY_ORG/airflow.git
branch: master
rev: HEAD
subPath: "dags"
credentialsSecret: git-credentials
# Airflow database migration job settings
migrateDatabaseJob:
serviceAccount:
annotations:
"eks.amazonaws.com/role-arn": ${iam_role_arn}
config:
core:
plugins_folder: /opt/airflow/plugins
```
**Anything else we need to know**:
I think that is because a wrong condition on webserver deployment.
FROM: ⬇️
```yaml
semverCompare "<2.0.0" .Values.airflowVersion
```
https://github.com/apache/airflow/blob/9d06ee8019ecbc07d041ccede15d0e322aa797a3/chart/templates/webserver/webserver-deployment.yaml#L137
https://github.com/apache/airflow/blob/9d06ee8019ecbc07d041ccede15d0e322aa797a3/chart/templates/webserver/webserver-deployment.yaml#L181
https://github.com/apache/airflow/blob/9d06ee8019ecbc07d041ccede15d0e322aa797a3/chart/templates/webserver/webserver-deployment.yaml#L200
TO: ⬇️
```yaml
semverCompare ">=2.0.0" .Values.airflowVersion
```
https://github.com/apache/airflow/blob/9d06ee8019ecbc07d041ccede15d0e322aa797a3/chart/templates/webserver/webserver-deployment.yaml#L43
I checked the Worker deployment and they are not using this condition clause
https://github.com/apache/airflow/blob/master/chart/templates/workers/worker-deployment.yaml#L168
https://github.com/apache/airflow/blob/master/chart/templates/workers/worker-deployment.yaml#L182
https://github.com/apache/airflow/blob/master/chart/templates/workers/worker-deployment.yaml#L258
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] danielcrisap commented on issue #16131: Missing dags mount in WebServer
Posted by GitBox <gi...@apache.org>.
danielcrisap commented on issue #16131:
URL: https://github.com/apache/airflow/issues/16131#issuecomment-857668963
Thanks for the explanation on how this works.
I could understand and correct in my environment, took advantage to have a PVC with my DAGs and now it's working normally.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jedcunningham commented on issue #16131: Missing dags mount in WebServer
Posted by GitBox <gi...@apache.org>.
jedcunningham commented on issue #16131:
URL: https://github.com/apache/airflow/issues/16131#issuecomment-850717530
Let me expand a little more: The scheduler parses the DAGs and serializes them into the database, removing the need for the webserver to also parse them constantly. The workers, however, do still need the DAGs to actually run the tasks.
More info here: https://airflow.apache.org/docs/apache-airflow/stable/dag-serialization.html
DAG serialization cannot be turned off in 2.0+, and thus the DAG files aren't need in the webserver when running 2.0+.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] danielcrisap closed issue #16131: Missing dags mount in WebServer
Posted by GitBox <gi...@apache.org>.
danielcrisap closed issue #16131:
URL: https://github.com/apache/airflow/issues/16131
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jedcunningham commented on issue #16131: Missing dags mount in WebServer
Posted by GitBox <gi...@apache.org>.
jedcunningham commented on issue #16131:
URL: https://github.com/apache/airflow/issues/16131#issuecomment-850716168
@danielcrisap, the webserver doesn't need the DAGs in 2.0+. Why do you feel like they should be there?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org