You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/12/07 08:24:19 UTC
[GitHub] [airflow] hterik opened a new issue, #28186: Pod name incorrect in RenderedTaskInstanceFields
hterik opened a new issue, #28186:
URL: https://github.com/apache/airflow/issues/28186
### Apache Airflow version
2.5.0
### What happened
1. Start a task running with KubernetesExecutor
2. Open the **_/rendered-k8s?dag_id=...&task_id=..._** endpoint in the web UI
3. Copy the metadata.name
4. `kubectl logs pod/$name`
5. `Error from server (NotFound): pods "example-sla-dag-sleep-20-1bc7f8d6892849eaac4642c4177e2eab" not found`
By inspecting the
### What you think should happen instead
Pod name presented in **Rendered K8s Pod Spec** should match the pod name used for running task. Otherwise pod name should not be presented at all.
This is not only a problem for users of the UI, also in the code it can be useful to have the real pod-id.
I'm in the process of troubleshooting strange errors with task adoption, where there are frequently orphaned tasks found. Currently there appears to be no way to go from TI to pod_id, the adoption-process instead goes the other way, using pod labels to find TIs. So for any TIs that were not found in the adoption, one can not find if they at any point have had matching pods or not.
### How to reproduce
_No response_
### Operating System
Ubuntu 22.04
### Versions of Apache Airflow Providers
_No response_
### Deployment
Other Docker-based deployment
### Deployment details
_No response_
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on issue #28186: Pod name incorrect in RenderedTaskInstanceFields
Posted by GitBox <gi...@apache.org>.
eladkal commented on issue #28186:
URL: https://github.com/apache/airflow/issues/28186#issuecomment-1342548847
So it seems that this effects KubernetesExecutor but not using KPO as stand alone
cc @jedcunningham @dstandish
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] csp33 commented on issue #28186: Pod name incorrect in RenderedTaskInstanceFields
Posted by GitBox <gi...@apache.org>.
csp33 commented on issue #28186:
URL: https://github.com/apache/airflow/issues/28186#issuecomment-1365739980
@eladkal sure! However, I still do not understand what is the pod name listed unther the "K8s Pod Spec" tab.
I think its value should match the task hostname.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on issue #28186: Pod name incorrect in RenderedTaskInstanceFields
Posted by GitBox <gi...@apache.org>.
eladkal commented on issue #28186:
URL: https://github.com/apache/airflow/issues/28186#issuecomment-1364534066
@csp33 would you like to raise PR ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on issue #28186: Pod name incorrect in RenderedTaskInstanceFields
Posted by GitBox <gi...@apache.org>.
eladkal commented on issue #28186:
URL: https://github.com/apache/airflow/issues/28186#issuecomment-1364533938
ah right. so this is not a bug.. maybe we just need to clarify what pod is shown in render tab for users of Kubernetes executor
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jedcunningham commented on issue #28186: Pod name incorrect in RenderedTaskInstanceFields
Posted by GitBox <gi...@apache.org>.
jedcunningham commented on issue #28186:
URL: https://github.com/apache/airflow/issues/28186#issuecomment-1373999859
That rendered pod spec is essentially the result of regenerating the pod spec on the fly. You'll notice if you refresh you get a new pod name every time.
Similarly, if you look at the pod spec for an old, historical task, you see what the pod would look like if it was ran now, not the spec it used when it was ran. For example, say you updated Airflow versions/images since it was ran, you'd see the new image not the old image in the spec.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] csp33 commented on issue #28186: Pod name incorrect in RenderedTaskInstanceFields
Posted by GitBox <gi...@apache.org>.
csp33 commented on issue #28186:
URL: https://github.com/apache/airflow/issues/28186#issuecomment-1363819293
I found out that the correct pod name is shown under Task Instance Details -> Hostname
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish commented on issue #28186: Pod name incorrect in RenderedTaskInstanceFields
Posted by GitBox <gi...@apache.org>.
dstandish commented on issue #28186:
URL: https://github.com/apache/airflow/issues/28186#issuecomment-1374090763
i guess the issue is that, RTIF gets stamped within _run_raw_task. and by the time it gets here, we're already _within_ the pod. so we no longer have the actual pod details.
probably this pod spec does not belong in RTIF. (it's not a templated field, after all) probably it was just a convenient / easy way to implement this feature, nowithstanding the limitations pointed out here.
i suppose one thing we could do is, create the RTIF record earlier for KE (just with pod spec), then in run raw update it with the RTIFs.
another option would be, put more KE info (or if you want to make an abstraction, executor info) on the TI record. we could include spec here. adding namespace and pod name to the TI record would also help with certain logging scenarios.
another option would be add separate table like TIExecutorInfo which is 1-1 with TI and has arbitrary executor info such as this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on issue #28186: Pod name incorrect in RenderedTaskInstanceFields
Posted by GitBox <gi...@apache.org>.
eladkal commented on issue #28186:
URL: https://github.com/apache/airflow/issues/28186#issuecomment-1340816686
Could it be that you can't find the pod because it was deleted when reached final state? Did you set `is_delete_operator_pod=False`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] hterik commented on issue #28186: Pod name incorrect in RenderedTaskInstanceFields
Posted by GitBox <gi...@apache.org>.
hterik commented on issue #28186:
URL: https://github.com/apache/airflow/issues/28186#issuecomment-1340830773
The pod executing the task is not deleted, it is visible in kubernetes but with a different name.
Pod creation happens in the scheduler, `kubernetes_executor.run_pod_async`. Here the real name is generated that is sent to kubernetes.
However, if i understand the flow correctly, the RenderedTaskInstanceFields is later created inside the pod itself, via the local executor when it calls `taskinstance._execute_task_with_callbacks`
rtif = RenderedTaskInstanceFields(ti=self, render_templates=False)
RenderedTaskInstanceFields.write(rtif)
RenderedTaskInstanceFields.delete_old_records(self.task_id, self.dag_id)
RenderedTaskInstanceFields calls ti.render_k8s_pod_yaml() which calls `construct_pod`, where the pod name is regenerated using `make_unique_pod_id`, which as the name suggests is random and a new unique value :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org