You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/12/19 21:57:09 UTC
[GitHub] [airflow] grepthat opened a new issue #13189: Handle long task IDs in k8s
grepthat opened a new issue #13189:
URL: https://github.com/apache/airflow/issues/13189
**Apache Airflow version**: master
**Kubernetes version**: v1.19.3
**Environment**: Ubuntu 18.04 Server
Using task IDs **longer than 253 characters** seems to cause problems in multiple areas. Considering we can now use **nested task groups** (which are basically prefixes for the task ID afaik) the final task ID can be pretty long.
- `pod_generator.py` does not use `make_safe_label_value` and writes the task ID directly as label. For k8s labels the limit is **only 63 characters**. This causes k8s not being able to schedule the task pod. https://github.com/apache/airflow/blob/4ad1accd3f6600ff398500e03c35eda8e3a6ec4a/airflow/kubernetes/pod_generator.py#L386
- *kubernetes_executor.py* uses `create_pod_id` to create a preliminary *pod_id* (not yet launched to k8s) and later uses `make_unique_pod_id` to trim the *pod_id* to 253 characters - though in my test I hit the max. filename length limit of Linux (*or rather ext4 filesystem*), which is 255 characters. So it seems some code is appending additional characters?
https://github.com/apache/airflow/blob/4ad1accd3f6600ff398500e03c35eda8e3a6ec4a/airflow/executors/kubernetes_executor.py#L294
```bash
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 3m3s default-scheduler Successfully assigned airflow/processairflowtesttgsurvey00000tgincrementaladjustmentsurvey00000f608c63d9btgmsac10survey000001c1a34cf10tgbundleadjuster786931747dtgcolor0521cd0b3f7tbundleadjusterfinalize5b57782bb2-aad4395c77ba400e9ee24c04d2cdbb72 to xxx.lan
Warning FailedCreatePodSandBox 7s (x14 over 3m) kubelet Failed to create pod sandbox: mkdir /var/log/pods/airflow_processairflowtesttgsurvey00000tgincrementaladjustmentsurvey00000f608c63d9btgmsac10survey000001c1a34cf10tgbundleadjuster786931747dtgcolor0521cd0b3f7tbundleadjusterfinalize5b57782bb2-aad4395c77ba400e9ee24c04d2cdbb72_8138b684-e7ce-41bc-8b31-4bbf9774e56a: file name too long
```
- As far as I understood *kubernetes_executor.py* `adopt_launched_task` uses the pod labels to "reconstruct" the *pod_id* - considering the pod was created by additionally applying `make_unique_pod_id` and the k8s label limit of 63 characters we will never reconstruct the original pod_id?
https://github.com/apache/airflow/blob/4ad1accd3f6600ff398500e03c35eda8e3a6ec4a/airflow/executors/kubernetes_executor.py#L628
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dimberman commented on issue #13189: Handle long task IDs in k8s
Posted by GitBox <gi...@apache.org>.
dimberman commented on issue #13189:
URL: https://github.com/apache/airflow/issues/13189#issuecomment-779419350
Closing as this appears to be fixed in 2.0.1
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] VBhojawala removed a comment on issue #13189: Handle long task IDs in k8s
Posted by GitBox <gi...@apache.org>.
VBhojawala removed a comment on issue #13189:
URL: https://github.com/apache/airflow/issues/13189#issuecomment-753292165
https://github.com/apache/airflow/blob/abf34b8aba33db4e751db44194429aeb2e8791c0/airflow/models/base.py#L33
https://github.com/apache/airflow/blob/abf34b8aba33db4e751db44194429aeb2e8791c0/airflow/models/taskinstance.py#L213
Currently ID_LEN is set to MAX 250, i think that is the reason why it is giving error.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dimberman closed issue #13189: Handle long task IDs in k8s
Posted by GitBox <gi...@apache.org>.
dimberman closed issue #13189:
URL: https://github.com/apache/airflow/issues/13189
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] VBhojawala commented on issue #13189: Handle long task IDs in k8s
Posted by GitBox <gi...@apache.org>.
VBhojawala commented on issue #13189:
URL: https://github.com/apache/airflow/issues/13189#issuecomment-753292165
https://github.com/apache/airflow/blob/abf34b8aba33db4e751db44194429aeb2e8791c0/airflow/models/base.py#L33
https://github.com/apache/airflow/blob/abf34b8aba33db4e751db44194429aeb2e8791c0/airflow/models/taskinstance.py#L213
Currently ID_LEN is set to MAX 250, i think that is the reason why it is giving error.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org