You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/11/19 04:23:37 UTC
[GitHub] [airflow] nayanen opened a new issue #19698: Graceful handle for airflow application pods during K8s API brief interuption
nayanen opened a new issue #19698:
URL: https://github.com/apache/airflow/issues/19698
### Description
### Description
I have come across this issue where k8s customer using airflow application. The pod get terminated when there is brief API server interruption. API server interruption can happen due to lot of scenarios so ideally the application should handle it gracefully. Another key thing here is this happens only when you set the _"is_delete_pod_operator"_ set to _True_.
Now if i read the [code](https://github.com/apache/airflow/blob/main/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py#L523-L533) i understand that the launcher.start_pod or launcher.monitor_pod have some logic which check heartbeat to the AI server and once its find some interruption it goes to the AirflowException part. So if we have some kind of nested exception or some retry logic to handle the brief interruption of API sever unavailability that should help to avoid the pod termination, because it just takes very less time for API server to re-establish the connection.
```
'airflow_version': airflow_version.replace('+', '-'),
'kubernetes_pod_operator': 'True',
}
)
self.log.debug("Starting pod:\n%s", yaml.safe_dump(self.pod.to_dict()))
final_state = None
try:
launcher.start_pod(self.pod, startup_timeout=self.startup_timeout_seconds)
final_state, remote_pod, result = launcher.monitor_pod(pod=self.pod, get_logs=self.get_logs)
except AirflowException:
```
The kubernetesPodOperator is provided as open-source Airflow and [more details](https://airflow.apache.org/docs/apache-airflow/1.10.12/_api/airflow/contrib/operators/kubernetes_pod_operator/index.html) about the behavior can be found below.
```
is_delete_operator_pod (bool) – What to do when the pod reaches its final state, or the execution is interrupted. If False (default): do nothing, If True: delete the pod
```
### Use case/motivation
_No response_
### Related issues
_No response_
### Are you willing to submit a PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] raphaelauv commented on issue #19698: Graceful handle for airflow application pods during K8s API brief interuption
Posted by GitBox <gi...@apache.org>.
raphaelauv commented on issue #19698:
URL: https://github.com/apache/airflow/issues/19698#issuecomment-974794742
WIP -> https://github.com/apache/airflow/pull/19572
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] raphaelauv commented on issue #19698: Graceful handle for airflow application pods during K8s API brief interuption
Posted by GitBox <gi...@apache.org>.
raphaelauv commented on issue #19698:
URL: https://github.com/apache/airflow/issues/19698#issuecomment-974794742
WIP -> https://github.com/apache/airflow/pull/19572
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] raphaelauv commented on issue #19698: Graceful handle for airflow application pods during K8s API brief interuption
Posted by GitBox <gi...@apache.org>.
raphaelauv commented on issue #19698:
URL: https://github.com/apache/airflow/issues/19698#issuecomment-1045997145
I think we can close this issue , now that the new kubernetesPodOperator is retrying.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #19698: Graceful handle for airflow application pods during K8s API brief interuption
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #19698:
URL: https://github.com/apache/airflow/issues/19698
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #19698: Graceful handle for airflow application pods during K8s API brief interuption
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #19698:
URL: https://github.com/apache/airflow/issues/19698#issuecomment-973723520
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on issue #19698: Graceful handle for airflow application pods during K8s API brief interuption
Posted by GitBox <gi...@apache.org>.
uranusjr commented on issue #19698:
URL: https://github.com/apache/airflow/issues/19698#issuecomment-975005848
Assigning to avoid acceidental overlap.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org