You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "arley-wilches (via GitHub)" <gi...@apache.org> on 2023/03/07 19:48:38 UTC
[GitHub] [airflow] arley-wilches opened a new issue, #29963: Task logs show 404 once the pods are gone
arley-wilches opened a new issue, #29963:
URL: https://github.com/apache/airflow/issues/29963
### Apache Airflow version
Other Airflow 2 version (please specify below)
### What happened
Hi, once ran DAG's and POD's gone, I am not able to see the logs, I got
`*** Trying to get logs (last 100 lines) from worker pod awsta-0ddbcfdcda294a08a370507891a1ea09 ***
*** Unable to fetch logs from worker pod awsta-0ddbcfdcda294a08a370507891a1ea09 ***
(404)
Reason: Not Found
HTTP response headers: HTTPHeaderDict({'Audit-Id': 'd1606aa1-d236-411b-9ac8-927ac7c0f311', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Date': 'Tue, 07 Mar 2023 19:17:43 GMT', 'Content-Length': '294'})
HTTP response body: b'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods \\"awsta-0ddbcfdcda294a08a370507891a1ea09\\" not found","reason":"NotFound","details":{"name":"awsta-0ddbcfdcda294a08a370507891a1ea09","kind":"pods"},"code":404}\n'
`
I am able to access the logs going directly to the filesystem, how can I see the logs from the UI if the PODs are gone?
### What you think should happen instead
should be a way to access logs even if the PODs are gone.
### How to reproduce
the DAGs ran without problems no matter the output, but once the pods finish, I am not able to access the logs from the UI.
apache-airflow: 2.4.2
### Operating System
Kubernetes EKS
### Versions of Apache Airflow Providers
`apache-airflow-providers-amazon==6.0.0
apache-airflow-providers-celery==3.0.0
apache-airflow-providers-cncf-kubernetes==4.4.0
apache-airflow-providers-common-sql==1.2.0
apache-airflow-providers-docker==3.2.0
apache-airflow-providers-elasticsearch==4.2.1
apache-airflow-providers-ftp==3.1.0
apache-airflow-providers-google==8.4.0
apache-airflow-providers-grpc==3.0.0
apache-airflow-providers-hashicorp==3.1.0
apache-airflow-providers-http==4.0.0
apache-airflow-providers-imap==3.0.0
apache-airflow-providers-microsoft-azure==4.3.0
apache-airflow-providers-mysql==3.2.1
apache-airflow-providers-odbc==3.1.2
apache-airflow-providers-postgres==5.2.2
apache-airflow-providers-redis==3.0.0
apache-airflow-providers-sendgrid==3.0.0
apache-airflow-providers-sftp==4.1.0
apache-airflow-providers-slack==6.0.0
apache-airflow-providers-sqlite==3.2.1
apache-airflow-providers-ssh==3.2.0`
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
_No response_
### Anything else
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #29963: Task logs show 404 once the pods are gone
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #29963:
URL: https://github.com/apache/airflow/issues/29963#issuecomment-1458740261
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] arley-wilches commented on issue #29963: Task logs show 404 once the pods are gone
Posted by "arley-wilches (via GitHub)" <gi...@apache.org>.
arley-wilches commented on issue #29963:
URL: https://github.com/apache/airflow/issues/29963#issuecomment-1469138743
hi @potiuk
I am deleting the pods once the task is done, I already have remote logging using NFS as a filesystem to store the logs.
but once the DAGs have been completely finished and the PODs were deleted, If I want to get the log for previous tasks, am not able to access them, I got 404, but the logs are in NFS.
that is the issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #29963: Task logs show 404 once the pods are gone
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #29963:
URL: https://github.com/apache/airflow/issues/29963#issuecomment-1468917547
You can set `delete_worker_pods` to False or configure remote logging if you want to delete the pods.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #29963: Task logs show 404 once the pods are gone
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk closed issue #29963: Task logs show 404 once the pods are gone
URL: https://github.com/apache/airflow/issues/29963
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] arley-wilches commented on issue #29963: Task logs show 404 once the pods are gone
Posted by "arley-wilches (via GitHub)" <gi...@apache.org>.
arley-wilches commented on issue #29963:
URL: https://github.com/apache/airflow/issues/29963#issuecomment-1469145085
@potiuk the 404 is because airflow is trying to get the POD from k8s, but they are deleted
*** Unable to fetch logs from worker pod awsta-0ddbcfdcda294a08a370507891a1ea09 ***
(404)
Reason: Not Found
HTTP response headers: HTTPHeaderDict({'Audit-Id': 'd1606aa1-d236-411b-9ac8-927ac7c0f311', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Date': 'Tue, 07 Mar 2023 19:17:43 GMT', 'Content-Length': '294'})
HTTP response body: b'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods \"awsta-0ddbcfdcda294a08a370507891a1ea09\" not found","reason":"NotFound","details":{"name":"awsta-0ddbcfdcda294a08a370507891a1ea09","kind":"pods"},"code":404}\n'
I am aware that the pod no longer exists, but the logs are in the file system.
What I am trying to say is that airflow could recover the logs even if the pod has been deleted.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #29963: Task logs show 404 once the pods are gone
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #29963:
URL: https://github.com/apache/airflow/issues/29963#issuecomment-1470947763
> I am aware that the pod no longer exists, but the logs are in the file system.
> What I am trying to say is that airflow could recover the logs even if the pod has been deleted.
No. It's wrong if you consider airflow's architecture. Instead, Airflow approach is to use remote logs. The problem is that Airflow Webserver might not have permissions nor access and can be completely shielded from Kubernetes to read the logs. Also Airflow Webserver works in a way that is independent which executor is used - and has to work in all circumstances, therefore remote loggging solution is the right approach.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org