You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/07/02 14:45:23 UTC
[GitHub] [airflow] dennisbakhuis opened a new issue #16777: Official Helm chart - unable to get persistent logs using KubernetesExecutor and PV
dennisbakhuis opened a new issue #16777:
URL: https://github.com/apache/airflow/issues/16777
I feel a bit like an idiot but I cannot seem to get the logging working on persistent volumes when using the KubernetesExecutor and the freshly released official Helm chart (version 1.0.0).
After creating a simple [PV](https://github.com/marclamberti/webinar-airflow-chart/blob/master/pv.yaml) and [PVC](https://github.com/marclamberti/webinar-airflow-chart/blob/master/pvc.yaml) manually, I changed the following on the bottom of `values.yaml` file:
```
logs:
persistence:
# Enable persistent volume for storing logs
enabled: true
# Volume size for logs
size: 100Gi
# If using a custom storageClass, pass name here
storageClassName:
## the name of an existing PVC to use
existingClaim: airflow-logs
```
This process is partly described in the [official Helm documentation](https://airflow.apache.org/docs/helm-chart/stable/manage-logs.html). Still, the airflow-scheduler pod crashes due to permission errors as i cannot write in the mounted logs folder: [logs in this pastebin](https://pastebin.com/ek3iH4EZ).
When the persistent logging is turned off, all is working, except for task logging as these are deleted when the worker-pod is deleted. A solution might be to have an init container to set the folder permissions but I guess that is not yet implemented in the current chart (or I do not know how to do it).
Any help towards a solution would be greatly appreciated!
**Apache Airflow version**: 2.1.0
** Helm chart version**: 1.0.0
**Kubernetes version (if you are using kubernetes)** (use `kubectl version`): v1.20.7
**Environment**: Microsoft AKS
- **Cloud provider or hardware configuration**: Microsoft AKS
- **OS** (e.g. from /etc/os-release):
- **Kernel** (e.g. `uname -a`):
- **Install tools**:
- **Others**:
**What happened**:
I added a persistent volume (PV) and a persistent volume claim (PVC) manually to my AKS cluster. Next, I set the persistent logging settings on the bottom, i.e. logs.persistence.enabled=true; logs.persistence.existingClaim=airflow-logs
The exact [PV](https://github.com/marclamberti/webinar-airflow-chart/blob/master/pv.yaml) and [PVC](https://github.com/marclamberti/webinar-airflow-chart/blob/master/pvc.yaml) settings very basic so I would not expect anything from that.
After updating the deployment using `helm upgrade` the scheduler pod crashes. The full error log is [here](https://pastebin.com/ek3iH4EZ).
**What you expected to happen**:
I expect Airflow and the worker pods to write all logs to the persistent volume, mounted at `/opt/airflow/logs`
Inspecting the logs it looks like the mkdir is failing due to permission issues which makes the whole pod crash.
**How to reproduce it**:
create the PV and PVC, turn on logs.persistence.enable and set logs.persistence.existingClaim=<PVC-name>
**Anything else we need to know**:
I have seen a similar problem with the unofficial chart. The solution was to include a extraInitContainer that 'fixes' the permissions of the logs folder. I have tried that but helm complained that such a setting does not exist.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jedcunningham commented on issue #16777: Official Helm chart - unable to get persistent logs using KubernetesExecutor and PV
Posted by GitBox <gi...@apache.org>.
jedcunningham commented on issue #16777:
URL: https://github.com/apache/airflow/issues/16777#issuecomment-873167223
Using a hostpath pv almost certainly isn't what you want. You'll want a "real" ReadyWriteMany pv that'll work across nodes too. What you use will depend on your environment, in your case, try Azure Files. I think you can get the volume with the right permissions automatically via StorageClass, but I haven't actually tried it and it sounds like the default is 777 anyways?
https://docs.microsoft.com/en-us/azure/aks/azure-files-dynamic-pv
If you still have permissions issues (e.g. writable by root only by default) with your new storage provider, then you do have some options. My favorite is to deploy a temporary pod that mounts the pvc to adjust the permissions. This has the benefit of being a one time setup step instead of an init container needing to run with every new pod.
Lastly, `1.0.0` doesn't have support for `extraInitContainers` but it is coming in the next chart release (which should be happening in the next couple weeks).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #16777: Official Helm chart - unable to get persistent logs using KubernetesExecutor and PV
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #16777:
URL: https://github.com/apache/airflow/issues/16777#issuecomment-873379184
Just to add on top of what @jedcunningham wrote.
At least by reading K8S documentation, I believe, the fsGroup setting which we use for securityContext should take care of the permissions/ownerships without the need of having custom init (unless you changed also some of the parameters of the securityContext or in case the volume you are mounting cannot have permissions changed for some reason by Kubernetes).
Kubernetes should change the group ownership and permissions of the files/folders recursively while mounting the volume:
https://kubernetes.io/docs/tasks/configure-pod-container/security-context/#set-the-security-context-for-a-pod
Also see here:
https://kubernetes.io/docs/tasks/configure-pod-container/security-context/#configure-volume-permission-and-ownership-change-policy-for-pods
I am not sure why it does not work for you. What you do need to make sure of is that the volume you are mounting is group-writeable - I believe, also the "change permission" possibility for the mounted volume might not be possible because the volume might not have this capability or your K8S installation is limited to be able to do that.
BTW. This goes very much in line with official image being redshift-compatible. The best way You have to make sure that runAsGroup is set to '0' (which is the default in our helm chart https://github.com/apache/airflow/blob/4c5376b1b73c3816be4cb6f62c776844f3563ecd/chart/values.yaml#L30) , and that fsGroup is set to `0` in securityContext (in our chart it is set to the same value as runAsGroup: https://github.com/apache/airflow/blob/4c5376b1b73c3816be4cb6f62c776844f3563ecd/chart/templates/workers/worker-deployment.yaml#L95)
I'd love to hear why this setting did not work in your case @dennisbakhuis
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #16777: Official Helm chart - unable to get persistent logs using KubernetesExecutor and PV
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #16777:
URL: https://github.com/apache/airflow/issues/16777#issuecomment-873379184
Just to add on top of what @jedcunningham wrote.
At least by reading K8S documentation, I believe, the fsGroup setting which we use for securityContext should take care of the permissions/ownerships without the need of having custom init (unless you changed also some of the parameters of the securityContext or in case the volume you are mounting cannot have permissions changed for some reason by Kubernetes).
Kubernetes should change the group ownership and permissions of the files/folders recursively while mounting the volume:
https://kubernetes.io/docs/tasks/configure-pod-container/security-context/#set-the-security-context-for-a-pod
Also see here:
https://kubernetes.io/docs/tasks/configure-pod-container/security-context/#configure-volume-permission-and-ownership-change-policy-for-pods
I am not sure why it does not work for you. What you do need to make sure of is that the volume you are mounting is group-writeable - I believe, also the "change permission" for the mounted volume might not be possible because the volume might not have this capability or your K8S installation is limited to be able to do that.
BTW. This goes very much in line with official image being redshift-compatible. The best way You have to make sure that runAsGroup is set to '0' (which is the default in our helm chart https://github.com/apache/airflow/blob/4c5376b1b73c3816be4cb6f62c776844f3563ecd/chart/values.yaml#L30) , and that fsGroup is set to `0` in securityContext (in our chart it is set to the same value as runAsGroup: https://github.com/apache/airflow/blob/4c5376b1b73c3816be4cb6f62c776844f3563ecd/chart/templates/workers/worker-deployment.yaml#L95)
I'd love to hear why this setting did not work in your case @dennisbakhuis
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #16777: Official Helm chart - unable to get persistent logs using KubernetesExecutor and PV
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #16777:
URL: https://github.com/apache/airflow/issues/16777#issuecomment-873052640
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dennisbakhuis commented on issue #16777: Official Helm chart - unable to get persistent logs using KubernetesExecutor and PV
Posted by GitBox <gi...@apache.org>.
dennisbakhuis commented on issue #16777:
URL: https://github.com/apache/airflow/issues/16777#issuecomment-873383510
Hi @jedcunningham, @potiuk,
Thanks for the quick reply.
I assumed that using standard persistent volume approach was the easiest (I am still a k8s novice) I did not expect that setting up one using azure-file-storage-class (SC) was this easy. These mounts can be set up using 777 rights from the SC yaml file, not sure if this is the sole cure as I also set the uid/gid in the SC mount options. Anyhow, all seems to be working perfectly.
As a reference for otheres here my azure-file-sc.yaml:
```
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: airflow-logs
provisioner: kubernetes.io/azure-file
mountOptions:
- dir_mode=0777
- file_mode=0777
- uid=50000
- gid=0
- mfsymlinks
- cache=strict
- actimeo=30
parameters:
skuName: Standard_LRS
```
My azure-file-pvc.yaml:
```
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: airflow-logs
namespace: airflow
labels:
app: airflow-logs
spec:
accessModes:
- ReadWriteMany
storageClassName: airflow-logs
resources:
requests:
storage: 20Gi
```
The values.yaml is unchanged.
@potiuk: I have not changed any particular settings regarding uid/gid. Here the PVC mount did not have any permission to create new files, so there is definitely something wrong in the mount options. Are there any similar mountOptions such as with the storage class?
I am closing the bug as it is not really a bug I guess. It probably is some misconfiguration on my side using PV.
Thanks again for the great help, read suggestions, and pointing out the final solution!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dennisbakhuis closed issue #16777: Official Helm chart - unable to get persistent logs using KubernetesExecutor and PV
Posted by GitBox <gi...@apache.org>.
dennisbakhuis closed issue #16777:
URL: https://github.com/apache/airflow/issues/16777
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org