You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/11/05 16:04:51 UTC
[GitHub] [airflow] pceric opened a new issue #12111: Kubernetes operator retry regression
pceric opened a new issue #12111:
URL: https://github.com/apache/airflow/issues/12111
**Apache Airflow version**: 1.10.12
**Kubernetes version (if you are using kubernetes)** (use `kubectl version`): 1.15.9
**Environment**:
- **Cloud provider or hardware configuration**: AWS
- **OS** (e.g. from /etc/os-release): Debian 9 (Stretch)
**What happened**: As of Airflow 1.10.12, and going back to sometime around 1.10.10 or 1.10.11, the behavior of the retry mechanism in the kubernetes pod operator regressed. Previously when a pod failed due to an error, Airflow would spin up a new pod in kubernetes on retry. As of 1.10.12 Airflow now tries to re-use the same broken pod over and over:
`INFO - found a running pod with labels {'dag_id': 'my_dad', 'task_id': 'my_task', 'execution_date': '2020-11-04T1300000000-e807cde8a', 'try_number': '6'} but a different try_number. Will attach to this pod and monitor instead of starting new one`
This is bad because most failures we encounter are due to the underlying "physical" hardware failing and retrying on the same pod is pointless, it will never succeed.
**What you expected to happen**: I would expect the k8s Airflow operator to start a new pod that would allow it to be scheduled on a new k8s node that does not have an underlying "physical" hardware problem, just like it was on earlier versions of Airflow.
**How to reproduce it**: Run a kubernetes pod operator task with a retry count set and error the node in a way that it can never succeed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] pceric commented on issue #12111: Kubernetes operator retry regression
Posted by GitBox <gi...@apache.org>.
pceric commented on issue #12111:
URL: https://github.com/apache/airflow/issues/12111#issuecomment-744723729
I installed 1.10.14 today and while the behavior is a bit odd, it does work. If I have retries set to 5, Airflow will run 10 retries with every odd retry being a "dummy retry", gathering the output from the previous failure. But since everything works as expected I'm calling this fixed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #12111: Kubernetes operator retry regression
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #12111:
URL: https://github.com/apache/airflow/issues/12111#issuecomment-722471964
Thanks for opening your first issue here! Be sure to follow the issue template!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] philipherrmann commented on issue #12111: Kubernetes operator retry regression
Posted by GitBox <gi...@apache.org>.
philipherrmann commented on issue #12111:
URL: https://github.com/apache/airflow/issues/12111#issuecomment-734829587
I think this is because of the
`reattach_on_restart`idea. The parameter is documented as "if the scheduler dies while the pod is running, reattach and monitor", but, I think, "while the pod is running" seems to checked in an incorrect manner. It seems that the Operator searches the pod's metadata for an "already_checked" label with value True. This label seems to be set only in the "call stack"
```
handle_pod_overlap > monitor_launched_pod > final_state != State.SUCCESS
```
This might be intended, in terms of idempotency - unclear to me. At least it unexpected to me.
Possible workarounds are setting `is_delete_operator_pod = True` or `reattach_on_restart = False` when construction the Operator. While both settings allow retries, they have different side effects.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dimberman commented on issue #12111: Kubernetes operator retry regression
Posted by GitBox <gi...@apache.org>.
dimberman commented on issue #12111:
URL: https://github.com/apache/airflow/issues/12111#issuecomment-741067520
Hi @philipherrmann @pceric please let me know if this bug is fixed. I would also recommend using the cncf.kubernetes backport provider instead of the operator in airflow itself as those are being deprecated in 2.0 (you'll also get fixes much faster through providers)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] philipherrmann commented on issue #12111: Kubernetes operator retry regression
Posted by GitBox <gi...@apache.org>.
philipherrmann commented on issue #12111:
URL: https://github.com/apache/airflow/issues/12111#issuecomment-738805521
I think this might be fixed with version 1.10.13 due to #11368, has anyone "already checked"?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] pceric commented on issue #12111: Kubernetes operator retry regression
Posted by GitBox <gi...@apache.org>.
pceric commented on issue #12111:
URL: https://github.com/apache/airflow/issues/12111#issuecomment-738873164
That does look promising, although it introduced critical bug #12659. Waiting for 1.10.14 before upgrading.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] pceric commented on issue #12111: Kubernetes operator retry regression
Posted by GitBox <gi...@apache.org>.
pceric commented on issue #12111:
URL: https://github.com/apache/airflow/issues/12111#issuecomment-735016331
Yes, I ended up enabling `is_delete_operator_pod` as a workaround, at the cost of being unable to jump into a finished pod to debug easier.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] pceric closed issue #12111: Kubernetes operator retry regression
Posted by GitBox <gi...@apache.org>.
pceric closed issue #12111:
URL: https://github.com/apache/airflow/issues/12111
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org