You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/12/10 18:28:56 UTC
[GitHub] [airflow] bparhy opened a new issue #20203: Airflow scheduler with Kubernetes executor trying to adopt pod from other deployment
bparhy opened a new issue #20203:
URL: https://github.com/apache/airflow/issues/20203
### Apache Airflow version
2.1.3
### What happened
We are using Airflow version 2.1.3
Using kubernetes executor
We have other deployment in this k8s namespace which are running using airflow 1.10.10 and 1.10.12 etc.
For the airflow version 2.1.3 we are seeing an error in the scheduler logs where it looks like the scheduler is trying to adopt pods which are scheduled by other airflow deployments. Below is the error.
Failed to adopt pod <pod_id> Reason: (422)
Reason: Unprocessable Entity
HTTP response headers:
This mainly happens for pods/task which are completed already.
### What you expected to happen
Airflow scheduler should not attempt to adopt pods from other deployments.
### How to reproduce
Create two airflow deployments one with version 1.10.10 and other with 2.1.3
Set up a dag in 1.10.10 with a frequency of 5 minutes and then check the logs for airflow version 2.1.3.
You should see 2.1.3 scheduler trying to adopt pods with the below error
Failed to adopt pod <pod_id> Reason: (422)
Reason: Unprocessable Entity
HTTP response headers:
### Operating System
linux
### Versions of Apache Airflow Providers
Not dependent on provider version
### Deployment
Other 3rd-party Helm chart
### Deployment details
We are deploying on k8s instance with 2 scheduler and 1 web server with airflow 2.1.3
We are using helm to deploy and metadata DB is postgres with AWS Aurora.
### Anything else
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jedcunningham commented on issue #20203: Airflow scheduler with Kubernetes executor trying to adopt pod from other deployment
Posted by GitBox <gi...@apache.org>.
jedcunningham commented on issue #20203:
URL: https://github.com/apache/airflow/issues/20203#issuecomment-991355746
Right now, the most reliable way is to run in separate namespaces.
Both running multiple instances in a single namespace and using the `multi_namespace_mode` option have various edge cases that just aren't handled currently (though, it is on my radar!). This hasn't changed in recent versions and it's likely you just got lucky or didn't notice issues previously.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #20203: Airflow scheduler with Kubernetes executor trying to adopt pod from other deployment
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #20203:
URL: https://github.com/apache/airflow/issues/20203#issuecomment-991689722
Yeah. My though exactly
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] bparhy commented on issue #20203: Airflow scheduler with Kubernetes executor trying to adopt pod from other deployment
Posted by GitBox <gi...@apache.org>.
bparhy commented on issue #20203:
URL: https://github.com/apache/airflow/issues/20203#issuecomment-991245765
Is this a new requirement as all our previous deployment prior to 2.1.3 is running without this issue. Also I am seeing this fix https://github.com/apache/airflow/pull/14795 is that related ?
Thanks for responding so quick.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #20203: Airflow scheduler with Kubernetes executor trying to adopt pod from other deployment
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #20203:
URL: https://github.com/apache/airflow/issues/20203#issuecomment-991242303
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #20203: Airflow scheduler with Kubernetes executor trying to adopt pod from other deployment
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #20203:
URL: https://github.com/apache/airflow/issues/20203#issuecomment-991690415
Moving that into a discussion then. I think, if we want to make multiple airflows in one namespace that shoudl be a separate feature. So maybe @bparhy - if you really think this is needed and you cannot rearrange your airflows to multipe instances - opening a feature request might be a good thing,
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #20203: Airflow scheduler with Kubernetes executor trying to adopt pod from other deployment
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #20203:
URL: https://github.com/apache/airflow/issues/20203#issuecomment-991242303
You should make sure each airflow deployment is in a different namespace if I am not mistaken @jedcunningham @kaxil ? Can you please confirm it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #20203: Airflow scheduler with Kubernetes executor trying to adopt pod from other deployment
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #20203:
URL: https://github.com/apache/airflow/issues/20203#issuecomment-991242303
You should make sure each airflow deployment is in a different namespece if I am not mistaken @jedcunningham @kaxil ? Can you please confirm it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jedcunningham commented on issue #20203: Airflow scheduler with Kubernetes executor trying to adopt pod from other deployment
Posted by GitBox <gi...@apache.org>.
jedcunningham commented on issue #20203:
URL: https://github.com/apache/airflow/issues/20203#issuecomment-991355746
Right now, the most reliable way is to run in separate namespaces.
Both running multiple instances in a single namespace and using the `multi_namespace_mode` option have various edge cases that just aren't handled currently (though, it is on my radar!). This hasn't changed in recent versions and it's likely you just got lucky or didn't notice issues previously.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #20203: Airflow scheduler with Kubernetes executor trying to adopt pod from other deployment
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #20203:
URL: https://github.com/apache/airflow/issues/20203#issuecomment-991248930
> Is this a new requirement as all our previous deployment prior to 2.1.3 is running without this issue. Also I am seeing this fix #14795 is that related ? Thanks for responding so quick.
I am not sure - it's just seems logical to me to keep each airlfow deployment in a separate namespace (that 's what I'd do in general at least). But yeah. I might be wrong and Airlfow should work this way (but it also could be accidental that it worked)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #20203: Airflow scheduler with Kubernetes executor trying to adopt pod from other deployment
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #20203:
URL: https://github.com/apache/airflow/issues/20203#issuecomment-991249280
That's why I am asking those who **likely** know better :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #20203: Airflow scheduler with Kubernetes executor trying to adopt pod from other deployment
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #20203:
URL: https://github.com/apache/airflow/issues/20203#issuecomment-991242303
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] bparhy commented on issue #20203: Airflow scheduler with Kubernetes executor trying to adopt pod from other deployment
Posted by GitBox <gi...@apache.org>.
bparhy commented on issue #20203:
URL: https://github.com/apache/airflow/issues/20203#issuecomment-991245765
Is this a new requirement as all our previous deployment prior to 2.1.3 is running without this issue. Also I am seeing this fix https://github.com/apache/airflow/pull/14795 is that related ?
Thanks for responding so quick.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #20203: Airflow scheduler with Kubernetes executor trying to adopt pod from other deployment
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #20203:
URL: https://github.com/apache/airflow/issues/20203#issuecomment-991248930
> Is this a new requirement as all our previous deployment prior to 2.1.3 is running without this issue. Also I am seeing this fix #14795 is that related ? Thanks for responding so quick.
I am not sure - it's just seems logical to me to keep each airlfow deployment in a separate namespace (that 's what I'd do in general at least). But yeah. I might be wrong and Airlfow should work this way (but it also could be acceidental that it worked)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org