You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/08/18 08:17:46 UTC
[GitHub] [airflow] yevsh opened a new issue #17679: Airflow not running as root when deployed to k8s via helm
yevsh opened a new issue #17679:
URL: https://github.com/apache/airflow/issues/17679
**Apache Airflow version**: v2.1.2
**Deployment**: Helm Chart + k8s
**What happened**:
helm install with values:
uid=0
gid=0
Airflow pods must run as root.
error:
from container's bash:
root@airflow-xxx-scheduler-7f49549459-w9s67:/opt/airflow# airflow
```
Traceback (most recent call last):
File "/home/airflow/.local/bin/airflow", line 5, in <module>
from airflow.__main__ import main
ModuleNotFoundError: No module named 'airflow'
```
**What you expected to happen**:
should run as root
using airflow's helm only
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #17679: Airflow not running as root when deployed to k8s via helm
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #17679:
URL: https://github.com/apache/airflow/issues/17679#issuecomment-901850402
Are you sure you get the latest merged `main` code?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #17679: Airflow not running as root when deployed to k8s via helm
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #17679:
URL: https://github.com/apache/airflow/issues/17679
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] yevsh commented on issue #17679: Airflow not running as root when deployed to k8s via helm
Posted by GitBox <gi...@apache.org>.
yevsh commented on issue #17679:
URL: https://github.com/apache/airflow/issues/17679#issuecomment-901871648
I just added
- /entrypoint
to scheduler-deployment.yaml
and pod is not restarting .
/entrypoint bash when you exec into the container. also works
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jedcunningham commented on issue #17679: Airflow not running as root when deployed to k8s via helm
Posted by GitBox <gi...@apache.org>.
jedcunningham commented on issue #17679:
URL: https://github.com/apache/airflow/issues/17679#issuecomment-901280807
This reproduces it:
```
helm upgrade --install --set uid=0 test airflow/airflow
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] yevsh edited a comment on issue #17679: Airflow not running as root when deployed to k8s via helm
Posted by GitBox <gi...@apache.org>.
yevsh edited a comment on issue #17679:
URL: https://github.com/apache/airflow/issues/17679#issuecomment-901871648
I just added
- /entrypoint
to scheduler-deployment.yaml
and pod is not restarting .
/entrypoint bash when you exec into the container, airflow command also works
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] yevsh edited a comment on issue #17679: Airflow not running as root when deployed to k8s via helm
Posted by GitBox <gi...@apache.org>.
yevsh edited a comment on issue #17679:
URL: https://github.com/apache/airflow/issues/17679#issuecomment-900965856
@potiuk
from readme
Full documentation for Helm Chart (latest **stable** release) lives [on the website](https://airflow.apache.org/docs/helm-chart/).
So please reopen?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #17679: Airflow not running as root when deployed to k8s via helm
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #17679:
URL: https://github.com/apache/airflow/issues/17679#issuecomment-900994751
Before I reopen it, I need to know what error do you have when you configure user as root and what kind of modifications you've made.
I think you could have jumped to conclusion that it does not work. If you enter the container via "/bin/bash" you will indeed get this error. But you should only enter the bash image via Airflow's image /entrypoint not via `/bin/bash`. This is the only way that can make it openshift-compatible and allow to run airflow as arbitrary user (including root). You can see more what the entrypoint is doing in https://airflow.apache.org/docs/docker-stack/entrypoint.html
`docker run -it <airflow_image> bash` as parameter will allow you to enter bash and utilising the entrypoint in the way that root user works (and all containers in the Helm Chart should use it).
For example `docker run -it apache/airflow:2.1.2-python3.6 bash -c "airflow version"` should work regardless which user you are running it with. All the containers including extra containers you define should use this approach rather than directly trying to execute `/bin/bash` within the container (which again - I am guessing) did not work for you.
But I need to understand what you've done because It's a lot of guesses for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #17679: Airflow not running as root when deployed to k8s via helm
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #17679:
URL: https://github.com/apache/airflow/issues/17679#issuecomment-901850070
> If you enter the container via "/bin/bash" you will indeed get this error. But you should only enter the bash image via Airflow's image /entrypoint not via /bin/bash. This is the only way that can make it openshift-compatible and allow to run airflow as arbitrary user (including root). You can see more what the entrypoint is doing in https://airflow.apache.org/docs/docker-stack/entrypoint.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #17679: Airflow not running as root when deployed to k8s via helm
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #17679:
URL: https://github.com/apache/airflow/issues/17679#issuecomment-901873463
Side comment. I really, really love the non-empathetic "it does not work" description of the bug when you want to get some help. It's never helpful and a bit frustrating.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #17679: Airflow not running as root when deployed to k8s via helm
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #17679:
URL: https://github.com/apache/airflow/issues/17679
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] yevsh edited a comment on issue #17679: Airflow not running as root when deployed to k8s via helm
Posted by GitBox <gi...@apache.org>.
yevsh edited a comment on issue #17679:
URL: https://github.com/apache/airflow/issues/17679#issuecomment-901875613
no erros = pod not restarting = good :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] yevsh edited a comment on issue #17679: Airflow not running as root when deployed to k8s via helm
Posted by GitBox <gi...@apache.org>.
yevsh edited a comment on issue #17679:
URL: https://github.com/apache/airflow/issues/17679#issuecomment-900965856
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] yevsh commented on issue #17679: Airflow not running as root when deployed to k8s via helm
Posted by GitBox <gi...@apache.org>.
yevsh commented on issue #17679:
URL: https://github.com/apache/airflow/issues/17679#issuecomment-901016712
oh.. do you want to see values yaml?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #17679: Airflow not running as root when deployed to k8s via helm
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #17679:
URL: https://github.com/apache/airflow/issues/17679#issuecomment-901014361
I think I know what it is - looked at the liveness probe and it indeed seems to run exec directly - this looks like a bug in the chart.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #17679: Airflow not running as root when deployed to k8s via helm
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #17679:
URL: https://github.com/apache/airflow/issues/17679#issuecomment-901092909
Yes I think only that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] yevsh commented on issue #17679: Airflow not running as root when deployed to k8s via helm
Posted by GitBox <gi...@apache.org>.
yevsh commented on issue #17679:
URL: https://github.com/apache/airflow/issues/17679#issuecomment-901124205
the only pvc are for dags and logs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #17679: Airflow not running as root when deployed to k8s via helm
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #17679:
URL: https://github.com/apache/airflow/issues/17679#issuecomment-901850873
BTW. You can test if the root user works by running `/entrypoint bash` when you `exec` into the container.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #17679: Airflow not running as root when deployed to k8s via helm
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #17679:
URL: https://github.com/apache/airflow/issues/17679#issuecomment-901905550
Thanks @jedcunningham !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] yevsh edited a comment on issue #17679: Airflow not running as root when deployed to k8s via helm
Posted by GitBox <gi...@apache.org>.
yevsh edited a comment on issue #17679:
URL: https://github.com/apache/airflow/issues/17679#issuecomment-901615020
thanks, so I can take the latest helms?
I see it's not officialy released yet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #17679: Airflow not running as root when deployed to k8s via helm
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #17679:
URL: https://github.com/apache/airflow/issues/17679#issuecomment-901904926
Glad it helped !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] yevsh commented on issue #17679: Airflow not running as root when deployed to k8s via helm
Posted by GitBox <gi...@apache.org>.
yevsh commented on issue #17679:
URL: https://github.com/apache/airflow/issues/17679#issuecomment-901811918
now pod is not restarting.
from within the container (bash), running command 'airflow' as root user still yields
No module named 'airflow'
is that normal?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #17679: Airflow not running as root when deployed to k8s via helm
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #17679:
URL: https://github.com/apache/airflow/issues/17679#issuecomment-901872500
What errors do you see? "Pod not restarting" is really vague description that is difficult to reason about :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] yevsh commented on issue #17679: Airflow not running as root when deployed to k8s via helm
Posted by GitBox <gi...@apache.org>.
yevsh commented on issue #17679:
URL: https://github.com/apache/airflow/issues/17679#issuecomment-901008433
Fair enough,
First, when running with airflow user, I am able to run airflow command within hte contaner's bash.
Now, when I use root user
I am not able to do so and :
airflow-xxx-scheduler pod is in a restart loop
kubectl describe pod airflow-xxx-scheduler-6fc986d95c-44246
under events:
```
Warning Unhealthy 4m3s (x110 over 58m) kubelet Liveness probe failed: Traceback (most recent call last):
File "<string>", line 5, in <module>
ModuleNotFoundError: No module named 'airflow'
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] yevsh commented on issue #17679: Airflow not running as root when deployed to k8s via helm
Posted by GitBox <gi...@apache.org>.
yevsh commented on issue #17679:
URL: https://github.com/apache/airflow/issues/17679#issuecomment-900965856
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #17679: Airflow not running as root when deployed to k8s via helm
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #17679:
URL: https://github.com/apache/airflow/issues/17679#issuecomment-901904491
Cool ! Sorry, misunderstood it ❤️
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] yevsh edited a comment on issue #17679: Airflow not running as root when deployed to k8s via helm
Posted by GitBox <gi...@apache.org>.
yevsh edited a comment on issue #17679:
URL: https://github.com/apache/airflow/issues/17679#issuecomment-901111783
```
helm install airflow-xxx airflow \
--set webserver.service.type=NodePort \
--set executor="LocalExecutor" \
--set workers.persistence.size=1Gi \
--set logs.persistence.size=1Gi \
--set workers.persistence.enabled=false \
--set postgresql.persistence.enabled=false \
--set dags.persistence.enabled=true \
--set logs.persistence.enabled=true \
--set dags.persistence.accessMode=ReadWriteMany \
--set dags.persistence.storageClassName=airflow-dags-xxx \
--set logs.persistence.storageClassName=airflow-logs-xxx \
--set images.airflow.repository=${EXTERNAL_IMAGES}/airflow \
--set postgresql.image.registry=${EXTERNAL_IMAGES} \
--set redis.enabled=false \
--set uid=0 \
--set gid=0 \
--namespace xxx
```
from values.yaml,
#User and group of airflow user
uid: 0
gid: 0
if you need anything specific from values please tell.
result of helm install:
NAME: airflow-xxx
LAST DEPLOYED: Wed Aug 18 16:21:19 2021
NAMESPACE: xxx
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
Thank you for installing Apache Airflow 2.1.2!
Your release is named airflow-xxx.
You can now access your dashboard(s) by executing the following command(s) and visiting the corresponding port at localhost in your browser:
Airflow Webserver: kubectl port-forward svc/airflow-xxx-webserver 8080:8080 --namespace xxx
Default Webserver (Airflow UI) Login credentials:
username: admin
password: admin
Default Postgres connection credentials:
username: postgres
password: postgres
port: 5432
You can get Fernet Key value by running the following:
echo Fernet Key: $(kubectl get secret --namespace xxx airflow-xxx-fernet-key -o jsonpath="{.data.fernet-key}" | base64 --decode)
status=0
Traceback (most recent call last):
File "/home/airflow/.local/bin/airflow", line 5, in <module>
from airflow.__main__ import main
ModuleNotFoundError: No module named 'airflow'
command terminated with exit code 1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #17679: Airflow not running as root when deployed to k8s via helm
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #17679:
URL: https://github.com/apache/airflow/issues/17679#issuecomment-900994751
Before I reopen it, I need to know what error do you have when you configure user as root and what kind of modifications you've made.
I think you could have jumped to conclusion that it does not work. If you enter the container via "/bin/bash" you will indeed get this error. But you should only enter the bash image via Airflow's image /entrypoint not via `/bin/bash`. This is the only way that can make it openshift-compatible and allow to run airflow as arbitrary user (including root). You can see more what the entrypoint is doing in https://airflow.apache.org/docs/docker-stack/entrypoint.html
`docker run -it <airflow_image> bash` as parameter will allow you to enter bash and utilising the entrypoint in the way that root user works (and all containers in the Helm Chart should use it).
For example `docker run -it apache/airflow:2.1.2-python3.6 bash -c "airflow version"` should work regardless which user you are running it with (and all the containers including extra containers you define) should use this approach rather than directly trying to execute `/bin/bash` within the container (which again - I am guessing) did not work for you.
But I need to understand what you've done because It's a lot of guesses for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #17679: Airflow not running as root when deployed to k8s via helm
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #17679:
URL: https://github.com/apache/airflow/issues/17679#issuecomment-901872500
What error do you see? "Pod not restarting" is really vague description that is difficult to reason about :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #17679: Airflow not running as root when deployed to k8s via helm
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #17679:
URL: https://github.com/apache/airflow/issues/17679#issuecomment-901011931
Can you please install airflow from the scratch (delete the previous deployment you have), with the the configuration you want and provide the exact steps you do and exact modifications you've made in the values.yaml to get there? I am afraid that if you just change the user after initially running it, it might not work (but also it might be that there is an error in the helm chart, so I want to make sure that we can reproduce what you exactly did to see if it is a real issue or not).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] yevsh commented on issue #17679: Airflow not running as root when deployed to k8s via helm
Posted by GitBox <gi...@apache.org>.
yevsh commented on issue #17679:
URL: https://github.com/apache/airflow/issues/17679#issuecomment-901111783
```
helm install airflow-xxx airflow \
--set webserver.service.type=NodePort \
--set executor="LocalExecutor" \
--set workers.persistence.size=1Gi \
--set logs.persistence.size=1Gi \
--set workers.persistence.enabled=false \
--set postgresql.persistence.enabled=false \
--set dags.persistence.enabled=true \
--set logs.persistence.enabled=true \
--set dags.persistence.accessMode=ReadWriteMany \
--set dags.persistence.storageClassName=airflow-dags-xxx \
--set logs.persistence.storageClassName=airflow-logs-xxx \
--set images.airflow.repository=${EXTERNAL_IMAGES}/airflow \
--set postgresql.image.registry=${EXTERNAL_IMAGES} \
--set redis.enabled=false \
--set uid=0 \
--set gid=0 \
--namespace xxx
```
from values.yaml,
#User and group of airflow user
uid: 0
gid: 0
if you need anything specific from values please tell.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] yevsh commented on issue #17679: Airflow not running as root when deployed to k8s via helm
Posted by GitBox <gi...@apache.org>.
yevsh commented on issue #17679:
URL: https://github.com/apache/airflow/issues/17679#issuecomment-901875613
no erros
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #17679: Airflow not running as root when deployed to k8s via helm
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #17679:
URL: https://github.com/apache/airflow/issues/17679#issuecomment-900994751
Before I reopen it, I need to know what error do you have when you configure user as root and what kind of modifications you've made.
I think you could have jumped to conclusion that it does not work. If you enter the container via "/bin/bash" you will indeed get this error. But you should only enter the bash image via Airflow's image /entrypoint not via `/bin/bash`. This is the only way that can make it openshift-compatible and allow to run airflow as arbitrary user. You can see more what the entrypoint is doing in https://airflow.apache.org/docs/docker-stack/entrypoint.html
`docker run -it <airflow_image> bash` as parameter will allow you to enter bash and utilising the entrypoint in the way that root user works (and all containers in the Helm Chart should use it).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #17679: Airflow not running as root when deployed to k8s via helm
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #17679:
URL: https://github.com/apache/airflow/issues/17679#issuecomment-901794731
You could try the main branch of airflow. We are following a release process where we vote and put version etc. But you can install the 'main' version too (with some potential instabilities but i think it is pretty safe) just checkout main repo and install it from 'chart'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #17679: Airflow not running as root when deployed to k8s via helm
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #17679:
URL: https://github.com/apache/airflow/issues/17679#issuecomment-900999850
This is an example I just tried:
```
[jarek@penguini:~] $ docker run --user "0:0" -it apache/airflow:2.1.2-python3.6 bash -c "airflow version"
The container is run as root user. For security, consider using a regular user account.
2.1.2
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #17679: Airflow not running as root when deployed to k8s via helm
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #17679:
URL: https://github.com/apache/airflow/issues/17679#issuecomment-900962840
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #17679: Airflow not running as root when deployed to k8s via helm
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #17679:
URL: https://github.com/apache/airflow/issues/17679#issuecomment-901794731
You could try Fri. The main branch of airflow. We are following a release process where we vote and put version etc. But you can install the 'main' version too (with some potential instabilities but i think it is pretty safe) just checkout main repo and install it from 'chart'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #17679: Airflow not running as root when deployed to k8s via helm
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #17679:
URL: https://github.com/apache/airflow/issues/17679#issuecomment-900994751
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #17679: Airflow not running as root when deployed to k8s via helm
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #17679:
URL: https://github.com/apache/airflow/issues/17679#issuecomment-900994751
Before I reopen it, I need to know what error do you have when you configure user as root and what kind of modifications you've made.
I think you could have jumped to conclusion that it does not work. If you enter the container via "/bin/bash" you will indeed get this error. But you should only enter the bash image via Airflow's image /entrypoint not via `/bin/bash`. This is the only way that can make it openshift-compatible and allow to run airflow as arbitrary user (including root). You can see more what the entrypoint is doing in https://airflow.apache.org/docs/docker-stack/entrypoint.html
`docker run -it <airflow_image> bash` as parameter will allow you to enter bash and utilising the entrypoint in the way that root user works (and all containers in the Helm Chart should use it).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] yevsh edited a comment on issue #17679: Airflow not running as root when deployed to k8s via helm
Posted by GitBox <gi...@apache.org>.
yevsh edited a comment on issue #17679:
URL: https://github.com/apache/airflow/issues/17679#issuecomment-901111783
```
helm install airflow-xxx airflow \
--set webserver.service.type=NodePort \
--set executor="LocalExecutor" \
--set workers.persistence.size=1Gi \
--set logs.persistence.size=1Gi \
--set workers.persistence.enabled=false \
--set postgresql.persistence.enabled=false \
--set dags.persistence.enabled=true \
--set logs.persistence.enabled=true \
--set dags.persistence.accessMode=ReadWriteMany \
--set dags.persistence.storageClassName=airflow-dags-xxx \
--set logs.persistence.storageClassName=airflow-logs-xxx \
--set images.airflow.repository=${EXTERNAL_IMAGES}/airflow \
--set postgresql.image.registry=${EXTERNAL_IMAGES} \
--set redis.enabled=false \
--set uid=0 \
--set gid=0 \
--namespace xxx
```
from values.yaml,
#User and group of airflow user
uid: 0
gid: 0
if you need anything specific from values please tell.
NAME: airflow-xxx
LAST DEPLOYED: Wed Aug 18 16:21:19 2021
NAMESPACE: xxx
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
Thank you for installing Apache Airflow 2.1.2!
Your release is named airflow-xxx.
You can now access your dashboard(s) by executing the following command(s) and visiting the corresponding port at localhost in your browser:
Airflow Webserver: kubectl port-forward svc/airflow-xxx-webserver 8080:8080 --namespace xxx
Default Webserver (Airflow UI) Login credentials:
username: admin
password: admin
Default Postgres connection credentials:
username: postgres
password: postgres
port: 5432
You can get Fernet Key value by running the following:
echo Fernet Key: $(kubectl get secret --namespace xxx airflow-xxx-fernet-key -o jsonpath="{.data.fernet-key}" | base64 --decode)
status=0
Traceback (most recent call last):
File "/home/airflow/.local/bin/airflow", line 5, in <module>
from airflow.__main__ import main
ModuleNotFoundError: No module named 'airflow'
command terminated with exit code 1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] yevsh edited a comment on issue #17679: Airflow not running as root when deployed to k8s via helm
Posted by GitBox <gi...@apache.org>.
yevsh edited a comment on issue #17679:
URL: https://github.com/apache/airflow/issues/17679#issuecomment-901016712
oh.. do you want to see values yaml?
you think it's only related to livness?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #17679: Airflow not running as root when deployed to k8s via helm
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #17679:
URL: https://github.com/apache/airflow/issues/17679#issuecomment-901097633
Hmm. Actually not really. It should work fine IMHO. Yes. Please provide the command and try to install from the scratch with it. I think the liveness probe should work IMHO if correctly configured (unless you do some funny things with extra mounting volumes as HOME directory of the user - in which case it might be the problem you describe.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] yevsh commented on issue #17679: Airflow not running as root when deployed to k8s via helm
Posted by GitBox <gi...@apache.org>.
yevsh commented on issue #17679:
URL: https://github.com/apache/airflow/issues/17679#issuecomment-901615020
thanks, so I can take the latest helms?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #17679: Airflow not running as root when deployed to k8s via helm
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #17679:
URL: https://github.com/apache/airflow/issues/17679
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #17679: Airflow not running as root when deployed to k8s via helm
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #17679:
URL: https://github.com/apache/airflow/issues/17679#issuecomment-900962840
You have not written which Helm Chart you used. The official Helm Chart of Apache Airflow Community https://airflow.apache.org/docs/helm-chart/stable/index.html should work with root user without a problem (though it's not recommended in general). So I assume you use User-community chart though https://github.com/airflow-helm/charts (which I believe has that problem) and if you need help you should open your issue there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] yevsh edited a comment on issue #17679: Airflow not running as root when deployed to k8s via helm
Posted by GitBox <gi...@apache.org>.
yevsh edited a comment on issue #17679:
URL: https://github.com/apache/airflow/issues/17679#issuecomment-900965856
@potiuk
from readme
Full documentation for Helm Chart (latest **stable** release) lives [on the website](https://airflow.apache.org/docs/helm-chart/).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #17679: Airflow not running as root when deployed to k8s via helm
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #17679:
URL: https://github.com/apache/airflow/issues/17679#issuecomment-901904149
aaaa :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] yevsh commented on issue #17679: Airflow not running as root when deployed to k8s via helm
Posted by GitBox <gi...@apache.org>.
yevsh commented on issue #17679:
URL: https://github.com/apache/airflow/issues/17679#issuecomment-900965856
from readme
Full documentation for Helm Chart (latest **stable** release) lives [on the website](https://airflow.apache.org/docs/helm-chart/).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #17679: Airflow not running as root when deployed to k8s via helm
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #17679:
URL: https://github.com/apache/airflow/issues/17679#issuecomment-901873463
Side comment. I really, really `love` the non-empathetic "it does not work" description of the bug when you want to get some help. It's never helpful and a bit frustrating.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org