You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "raphaelauv (via GitHub)" <gi...@apache.org> on 2024/05/17 12:06:43 UTC
[I] KPO - get_logs - logs missing [airflow]
raphaelauv opened a new issue, #39682:
URL: https://github.com/apache/airflow/issues/39682
### Apache Airflow version
2.9.1
### If "Other Airflow 2 version" selected, which one?
_No response_
### What happened?
I tried with apache-airflow-providers-cncf-kubernetes 8.2.0 or 8.1.1
the KPO is missing some logs ( in defer or not )
example put this script in file src/job.py
```python
import time
if __name__ == '__main__':
for i in range(10):
time.sleep(2)
print("RUNNING")
```
```dockerfile
FROM python:3.12-slim
RUN mkdir -p /opt/program
WORKDIR /opt/program
COPY src/job.py /opt/program/job.py
CMD ["python3","job.py"]
```
```shelll
docker build -t job_example:0.1 .
```
```python
from airflow.providers.cncf.kubernetes.utils.pod_manager import OnFinishAction
from pendulum import today
from airflow import DAG
from airflow.providers.cncf.kubernetes.operators.pod import KubernetesPodOperator
dag = DAG(
dag_id="kpo_async_get_logs",
schedule_interval="0 0 * * *",
start_date=today("UTC").add(days=-1)
)
with dag:
KubernetesPodOperator(
task_id="task-two",
namespace="default",
kubernetes_conn_id="kubernetes_default",
name="airflow-toto-pod",
image="job_example:0.1",
# deferrable=True,
poll_interval=2,
on_finish_action=OnFinishAction.DELETE_POD,
get_logs=True
)
```
the logs in airflow give
```log
{pod.py:536} INFO - `try_number` of task_instance: 1
{pod.py:537} INFO - `try_number` of pod: 1
{pod_manager.py:374} WARNING - Pod not yet started: airflow-toto-pod-l3mfagyt
{pod_manager.py:468} INFO - [base] RUNNING
{pod_manager.py:468} INFO - [base] RUNNING
{pod_manager.py:468} INFO - [base] RUNNING
{pod_manager.py:468} INFO - [base] RUNNING
{pod_manager.py:468} INFO - [base] RUNNING
{pod_manager.py:486} INFO - [base] RUNNING
{pod_manager.py:619} INFO - Pod airflow-toto-pod-l3mfagyt has phase Running
{pod.py:931} INFO - Deleting pod: airflow-toto-pod-l3mfagyt
{taskinstance.py:441} ▼ Post task execution logs
{taskinstance.py:1206} INFO - Marking task as SUCCESS. dag_id=kpo_async_get_logs, task_id=task-two, run_id=manual__2024-05-17T12:00:38.699092+00:00, execution_date=20240517T120038, start_date=20240517T120039, end_date=20240517T120103
{local_task_job_runner.py:240} INFO - Task exited with return code 0
{taskinstance.py:3498} INFO - 0 downstream tasks scheduled from follow-on schedule check
```
if I check the logs in the pod directly it give
### What you think should happen instead?
_No response_
### How to reproduce
---
### Operating System
---
### Versions of Apache Airflow Providers
_No response_
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
_No response_
### Anything else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] KPO - get_logs - logs missing [airflow]
Posted by "raphaelauv (via GitHub)" <gi...@apache.org>.
raphaelauv commented on issue #39682:
URL: https://github.com/apache/airflow/issues/39682#issuecomment-2146645698
No stale
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] KPO - get_logs - logs missing [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #39682: KPO - get_logs - logs missing
URL: https://github.com/apache/airflow/issues/39682
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] KPO - get_logs - logs missing [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #39682:
URL: https://github.com/apache/airflow/issues/39682#issuecomment-2146333403
This issue has been automatically marked as stale because it has been open for 14 days with no response from the author. It will be closed in next 7 days if no further activity occurs from the issue author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] KPO - get_logs - logs missing [airflow]
Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis commented on issue #39682:
URL: https://github.com/apache/airflow/issues/39682#issuecomment-2117764089
What happen if you change your script to?
```python
import time
if __name__ == '__main__':
for i in range(10):
time.sleep(2)
print(f"RUNNING {i}")
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] KPO - get_logs - logs missing [airflow]
Posted by "Lee2532 (via GitHub)" <gi...@apache.org>.
Lee2532 commented on issue #39682:
URL: https://github.com/apache/airflow/issues/39682#issuecomment-2120627003
What happen if you change your script to? print to logging
import time
import logging
if __name__ == '__main__':
for i in range(10):
time.sleep(2)
logging.info("RUNNING")
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] KPO - get_logs - logs missing [airflow]
Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis commented on issue #39682:
URL: https://github.com/apache/airflow/issues/39682#issuecomment-2120013361
I guess it similar to https://github.com/apache/airflow/issues/39686
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] KPO - get_logs - logs missing [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #39682:
URL: https://github.com/apache/airflow/issues/39682#issuecomment-3068078763
This issue has been automatically marked as stale because it has been open for 365 days without any activity. There has been several Airflow releases since last activity on this issue. Kindly asking to recheck the report against latest Airflow version and let us know if the issue is reproducible. The issue will be closed in next 30 days if no further activity occurs from the issue author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] KPO - get_logs - logs missing [airflow]
Posted by "raphaelauv (via GitHub)" <gi...@apache.org>.
raphaelauv commented on issue #39682:
URL: https://github.com/apache/airflow/issues/39682#issuecomment-2118836786
@Taragolis
in that case there is no missing logs
```log
{pod_manager.py:374} WARNING - Pod not yet started: airflow-toto-pod-s0p6ivyu
{pod_manager.py:468} INFO - [base] RUNNING_0
{pod_manager.py:468} INFO - [base] RUNNING_1
{pod_manager.py:468} INFO - [base] RUNNING_2
{pod_manager.py:468} INFO - [base] RUNNING_3
{pod_manager.py:468} INFO - [base] RUNNING_4
{pod_manager.py:468} INFO - [base] RUNNING_5
{pod_manager.py:468} INFO - [base] RUNNING_6
{pod_manager.py:468} INFO - [base] RUNNING_7
{pod_manager.py:468} INFO - [base] RUNNING_8
{pod_manager.py:486} INFO - [base] RUNNING_9
{pod_manager.py:619} INFO - Pod airflow-toto-pod-s0p6ivyu has phase Running
{pod.py:931} INFO - Deleting pod: airflow-toto-pod-s0p6ivyu
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] KPO - get_logs - logs missing [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #39682:
URL: https://github.com/apache/airflow/issues/39682#issuecomment-3190801098
This issue has been closed because it has not received response from the issue author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org