You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "jaruji (via GitHub)" <gi...@apache.org> on 2024/02/15 18:08:56 UTC
[I] ArgoCD reverting the app to Missing, OutOfSync even when successfully deployed [airflow]
jaruji opened a new issue, #37460:
URL: https://github.com/apache/airflow/issues/37460
### Official Helm Chart version
1.12.0 (latest released)
### Apache Airflow version
2.8.1
### Kubernetes Version
1.26.6
### Helm Chart configuration
```
config:
webserver:
expose_config: 'True'
logging:
remote_logging: 'True'
remote_base_log_folder: wasb-airflow/logs
remote_wasb_log_container: airflow
remote_log_conn_id: wasb_default
images:
airflow:
# define custom airflow image here (with PyPi packages installed)
repository: org.azurecr.io/internal-airflow
# CHANGE THIS when updating
tag: "0.2.0"
executor: KubernetesExecutor
fernetKeySecretName: airflow-fernet-secret
webserverSecretKeySecretName: airflow-webserver-secret
createUserJob:
useHelmHooks: false
applyCustomEnv: false
migrateDatabaseJob:
enabled: true
useHelmHooks: false
applyCustomEnv: false
jobAnnotations:
"argocd.argoproj.io/hook": Sync
useStandardNaming: true
dags:
gitSync:
enabled: true
repo: git@github.com:ORG/custom-airflow.git
branch: master
subPath: "dags"
sshKeySecret: airflow-ssh-secret
ingress:
web:
enabled: true
annotations:
cert-manager.io/cluster-issuer: "letsencrypt"
# The path for the web Ingress
path: "/"
# The pathType for the above path (used only with Kubernetes v1.19 and above)
pathType: "ImplementationSpecific"
# The hostnames or hosts configuration for the web Ingress
# Set in argoCD application yaml
hosts: []
# # The hostname for the web Ingress (can be templated)
# - name: ""
# # configs for web Ingress TLS
# tls:
# # Enable TLS termination for the web Ingress
# enabled: false
# # the name of a pre-created Secret containing a TLS private key and certificate
# secretName: ""
# The Ingress Class for the web Ingress (used only with Kubernetes v1.19 and above)
ingressClassName: "nginx"
```
### Docker Image customizations
```
#### Use the specified Apache Airflow image as a base
FROM apache/airflow:2.8.1
##### Install dependencies required for building pymssql
USER root
RUN apt-get update && apt-get install -y \
freetds-dev \
build-essential \
&& rm -rf /var/lib/apt/lists/*
RUN apt-get update -y \
&& apt-get install -y \
libglib2.0-0 \
libnss3 \
libnspr4 \
libdbus-1-3 \
libatk1.0-0 \
libatk-bridge2.0-0 \
libcups2 \
libdrm2 \
libxkbcommon0 \
libatspi2.0-0 \
libxcomposite1 \
libxdamage1 \
libxext6 \
libxfixes3 \
libxrandr2 \
libgbm1 \
libpango-1.0-0 \
libcairo2 \
libasound2 \
&& rm -rf /var/lib/apt/lists/*
#### Copy the requirements file into the container
COPY requirements.txt /
COPY .env /
#### Switch back to the airflow user
USER airflow
#### Install the requirements, including Apache Airflow
RUN pip install --no-cache-dir "apache-airflow==${AIRFLOW_VERSION}" -r /requirements.txt
RUN pip install python-dotenv
#install azure provider for airflow, needed for remote logging to azure blob
RUN pip install apache-airflow-providers-microsoft-azure
RUN playwright install
```
### What happened
When deploying the chart using ArgoCD, everything seems to more or less work correctly. But after a while, the argoCD app reverts to OutOfSync and Missing statuses. This persists until the chart is re-synced. When the sync if successfully finished, everything displays correctly - synced, etc... but it always reverts back to OutOfSync and the health status goes to Missing, which can be confusing. I have not found out what the cause of this is yet.
### What you think should happen instead
After the successful helm chart deploy through ArgoCD, correctly display that the app is Healthy and Synced.
### How to reproduce
I assume using the provided configurations to deploy the chart to a kubernetes cluster should be enough to reproduce - you also need to add the argoCD app config for airflow into your argoCD apps.
The argocd airflow app config:
```
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: airflow
namespace: {{ .Values.argocdNamespace }}
finalizers:
- resources-finalizer.argocd.argoproj.io
spec:
project: {{ .Values.projects.infra }}
source:
repoURL: {{ .Values.repoUrl }}
targetRevision: {{ .Values.targetRevision }}
path: infrastructure/airflow
helm:
valuesObject:
ingress:
web:
hosts:
- name: "airflow.{{ .Values.ingress.host }}"
tls:
enabled: true
secretName: "airflow-ingress-secret"
destination:
server: https://kubernetes.default.svc
namespace: "airflow"
syncPolicy:
syncOptions:
- CreateNamespace=true
```
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] ArgoCD reverting the app to Missing, OutOfSync even when successfully deployed [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #37460:
URL: https://github.com/apache/airflow/issues/37460#issuecomment-1947873445
Also - what I could recommend maybe - try to install the Chart from main. The two issues mentioned above are already fixed in main and we will soon release 1.12.1 - but maybe it's one of the two that causes the behaviour you observe - this is by far the fastest way to asses is - both on our and your side
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] ArgoCD reverting the app to Missing, OutOfSync even when successfully deployed [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #37460:
URL: https://github.com/apache/airflow/issues/37460#issuecomment-1947846025
Yep I think discussion is better.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] ArgoCD reverting the app to Missing, OutOfSync even when successfully deployed [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #37460:
URL: https://github.com/apache/airflow/issues/37460#issuecomment-1947407531
Have you opened the issue to ArgoCD? What do they say ? It could be caused by some issues that we already know and plan to fix https://github.com/apache/airflow/issues/37399 or https://github.com/apache/airflow/pull/37356 - can you please check with Argo CD support?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] ArgoCD reverting the app to Missing, OutOfSync even when successfully deployed [airflow]
Posted by "jaruji (via GitHub)" <gi...@apache.org>.
jaruji commented on issue #37460:
URL: https://github.com/apache/airflow/issues/37460#issuecomment-1947441852
I can open an issue there tomorrow. I did some more research, and enabled automated prune and selfHeal in the argoCD app like so:
```
syncPolicy:
automated:
prune: true
selfHeal: true
```
The argo airflow app is now re-syncing every couple of minutes to keep the Healthy and Synced status. Specifically, the run-airflow-migrations and create-user jobs are executing every couple of minutes in this case. If the automated selfHeal is turned off, the app just shifts to Missing and OutOfSync after a couple of minutes - but it keeps working just fine. Just wanted to give a better description of what is happening!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] ArgoCD reverting the app to Missing, OutOfSync even when successfully deployed [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk closed issue #37460: ArgoCD reverting the app to Missing, OutOfSync even when successfully deployed
URL: https://github.com/apache/airflow/issues/37460
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] ArgoCD reverting the app to Missing, OutOfSync even when successfully deployed [airflow]
Posted by "jaruji (via GitHub)" <gi...@apache.org>.
jaruji commented on issue #37460:
URL: https://github.com/apache/airflow/issues/37460#issuecomment-1947700115
I'm mainly opening the issue to see whether somebody else encountered something similar and managed to fix it. I do not have any expectations regarding you doing the necessary research and figuring it out for me. Guess this can be converted into a discussion if you feel like an issue is not appropriate in this case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] ArgoCD reverting the app to Missing, OutOfSync even when successfully deployed [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #37460:
URL: https://github.com/apache/airflow/issues/37460#issuecomment-1947527524
> Specifically, the run-airflow-migrations and create-user jobs are executing every couple of minutes in this case. If the automated selfHeal is turned off, the app just shifts to Missing and OutOfSync after a couple of minute but it keeps working just fine. Just wanted to give a better description of what is happening!
Just bear with us that for most people here who do not use ArgoCD - none of this - syncing, out-of-syncing has any significant meaning, and for people who voluntarily maintain Ariflow, they won't spend awfully lot of time digging in the details of Argo.
For us we deliver Helm Chart is a helm chart and we generally deploy and test it in plain K8S. If there is anything that Argo Does which is specific to Argo, it's pretty much up to you to see what's happening and translate it into "what really happens".
By looking at logs etc. and trying to figure out thibgs - while Argo is popular, it's still pretty much 3rd-party technology that we have little to none expert knowedge there and we have to rely on our users - who are part of the community and actually have the skills and incentive to investigate things like that. If you can diagnose it and translate it to what happens in the "non-argo" terms but - generally what happens with resources, memory, networking and other things - then we can try to see how we can help with that.
Maybe other Argo users might chime in and provide some input - but from maintainers point of view it's like that .
Just to set some expectations here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org