You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "chyzouakia (via GitHub)" <gi...@apache.org> on 2025/09/09 01:17:17 UTC
[D] S3TaskHandler doesn't seem to run the close method and doesn't push the logs to S3 [airflow]
GitHub user chyzouakia created a discussion: S3TaskHandler doesn't seem to run the close method and doesn't push the logs to S3
### Apache Airflow Provider(s)
amazon
### Versions of Apache Airflow Providers
apache-airflow-providers-amazon==8.29.0
### Apache Airflow version
2.10.5
### Operating System
Debian GNU/Linux 12 (bookworm)
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
Deployment through the Helm chart `https://airflow-helm.github.io/charts` version `8.9.0`.
Airflow image : `slim-2.10.5-python3.10`
My deployment creates a Service Account to allow to connect to our AWS S3 to push the logs there.
Configuration for logging :
AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER: "s3://{{ bucket }}/airflow/logs"
AIRFLOW__LOGGING__REMOTE_LOGGING: "True"
AIRFLOW__LOGGING__REMOTE_LOG_CONN_ID: "s3_remote_logging"
### What happened
It seems the `close()` method in `S3TaskHandler` is not called. I don't have any log stored in S3.
I am able to read a file from S3 :
<img width="909" height="325" alt="Image" src="https://github.com/user-attachments/assets/b442d462-8f7c-4cb7-9205-ff10b7d85f2d" />
I am able to push a log from the worker manually by running this code :
```
from airflow.providers.amazon.aws.hooks.s3 import S3Hook
s3_hook=S3Hook(aws_conn_id="s3_remote_logging")
import os
import pathlib
log = pathlib.Path([LOCAL_PATH]).read_text()
s3_hook.load_string(log, key=[S3_KEY], replace=True, encrypt=False)
```
### What you think should happen instead
When closing the task, The Task Handler should have pushed the logs to my S3 bucket.
### How to reproduce
* Run the Airflow image `slim-2.10.5-python3.10` with the amazon provider package
* Configure Airflow to push the logs to S3 (the connection should be created) :
* AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER: "s3://{{ bucket }}/airflow/logs"
* AIRFLOW__LOGGING__REMOTE_LOGGING: "True"
* AIRFLOW__LOGGING__REMOTE_LOG_CONN_ID: "s3_remote_logging"
* Run a task and see if the code pushes the logs to S3.
### Anything else
This problem is occurring on all my tasks now. Please let me know if I misconfigured something. Thanks !
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
GitHub link: https://github.com/apache/airflow/discussions/55402
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] S3TaskHandler doesn't seem to run the close method and doesn't push the logs to S3 [airflow]
Posted by "chyzouakia (via GitHub)" <gi...@apache.org>.
GitHub user chyzouakia edited a comment on the discussion: S3TaskHandler doesn't seem to run the close method and doesn't push the logs to S3
I tried in local with Docker and was not able to push the logs to S3 either 🙂 The variables are OK since I am able to read a log file I pushed manually in the S3 bucket (Screenshot shared above).
What local logs are you referring to ? I couldn't find any logs, I tried all the exception messages in the logs and couldn't find one
GitHub link: https://github.com/apache/airflow/discussions/55402#discussioncomment-14349483
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] S3TaskHandler doesn't seem to run the close method and doesn't push the logs to S3 [airflow]
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
GitHub user boring-cyborg[bot] added a comment to the discussion: S3TaskHandler doesn't seem to run the close method and doesn't push the logs to S3
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
GitHub link: https://github.com/apache/airflow/discussions/55402#discussioncomment-14345761
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] S3TaskHandler doesn't seem to run the close method and doesn't push the logs to S3 [airflow]
Posted by "chyzouakia (via GitHub)" <gi...@apache.org>.
GitHub user chyzouakia edited a comment on the discussion: S3TaskHandler doesn't seem to run the close method and doesn't push the logs to S3
I was able to push the logs from the worker and from local manually. Everything is explained in the description above, I could run the code that pushes the logs manually. I was not able to push the logs to S3 using the S3TaskHandler (with s3 logging configured). I tried checking the logs and couldn't find any error.
It feels like you didn't check the `what happened `section above and just dismissed the issue 🙂
GitHub link: https://github.com/apache/airflow/discussions/55402#discussioncomment-14359099
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] S3TaskHandler doesn't seem to run the close method and doesn't push the logs to S3 [airflow]
Posted by "chyzouakia (via GitHub)" <gi...@apache.org>.
GitHub user chyzouakia edited a comment on the discussion: S3TaskHandler doesn't seem to run the close method and doesn't push the logs to S3
I was able to push the logs from the worker and from local manually. Everything is explained in the description above, I could run the code that pushes the logs manually. I tried checking the logs and couldn't find any error.
It feels like you didn't check the `what happened `section above and just dismissed the issue 🙂
GitHub link: https://github.com/apache/airflow/discussions/55402#discussioncomment-14359099
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] S3TaskHandler doesn't seem to run the close method and doesn't push the logs to S3 [airflow]
Posted by "chyzouakia (via GitHub)" <gi...@apache.org>.
GitHub user chyzouakia deleted a comment on the discussion: S3TaskHandler doesn't seem to run the close method and doesn't push the logs to S3
I tried in local with Docker and was not able to push the logs to S3 either 🙂
GitHub link: https://github.com/apache/airflow/discussions/55402#discussioncomment-14348935
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] S3TaskHandler doesn't seem to run the close method and doesn't push the logs to S3 [airflow]
Posted by "chyzouakia (via GitHub)" <gi...@apache.org>.
GitHub user chyzouakia added a comment to the discussion: S3TaskHandler doesn't seem to run the close method and doesn't push the logs to S3
I tried in local with Docker and was not able to push the logs to S3 either 🙂
GitHub link: https://github.com/apache/airflow/discussions/55402#discussioncomment-14349483
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] S3TaskHandler doesn't seem to run the close method and doesn't push the logs to S3 [airflow]
Posted by "chyzouakia (via GitHub)" <gi...@apache.org>.
GitHub user chyzouakia added a comment to the discussion: S3TaskHandler doesn't seem to run the close method and doesn't push the logs to S3
I tried in local with Docker and was not able to push the logs to S3 either 🙂
GitHub link: https://github.com/apache/airflow/discussions/55402#discussioncomment-14348935
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] S3TaskHandler doesn't seem to run the close method and doesn't push the logs to S3 [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
GitHub user potiuk added a comment to the discussion: S3TaskHandler doesn't seem to run the close method and doesn't push the logs to S3
My guess is something wrong with the chart, maybe the variables are not passed as you think they are - but you should check there - this chart is not managed by us. This works if you configure it locally in general (a lot of people would complain if not) - so something must be problematic in the configuration of the chart, or you have some typo. Generally speaking you should find information in the local logs (writtent in the place where the chart configures them to be written) and you should look there for clues.
GitHub link: https://github.com/apache/airflow/discussions/55402#discussioncomment-14345762
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] S3TaskHandler doesn't seem to run the close method and doesn't push the logs to S3 [airflow]
Posted by "chyzouakia (via GitHub)" <gi...@apache.org>.
GitHub user chyzouakia edited a comment on the discussion: S3TaskHandler doesn't seem to run the close method and doesn't push the logs to S3
I tried in local with Docker and was not able to push the logs to S3 either 🙂 The variables are OK since I am able to read a log file I pushed manually in the Repository (Screenshot shared above).
What local logs are you referring to ? I couldn't find any logs, I tried all the exception messages in the logs and couldn't find one
GitHub link: https://github.com/apache/airflow/discussions/55402#discussioncomment-14349483
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] S3TaskHandler doesn't seem to run the close method and doesn't push the logs to S3 [airflow]
Posted by "chyzouakia (via GitHub)" <gi...@apache.org>.
GitHub user chyzouakia edited a comment on the discussion: S3TaskHandler doesn't seem to run the close method and doesn't push the logs to S3
I tried in local with Docker and was not able to push the logs to S3 either 🙂 What local logs are you referring to ? I couldn't find any logs, I tried all the exception messages in the logs and couldn't find one
GitHub link: https://github.com/apache/airflow/discussions/55402#discussioncomment-14349483
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] S3TaskHandler doesn't seem to run the close method and doesn't push the logs to S3 [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
GitHub user potiuk added a comment to the discussion: S3TaskHandler doesn't seem to run the close method and doesn't push the logs to S3
Converting to a discussion.
GitHub link: https://github.com/apache/airflow/discussions/55402#discussioncomment-14345764
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] S3TaskHandler doesn't seem to run the close method and doesn't push the logs to S3 [airflow]
Posted by "chyzouakia (via GitHub)" <gi...@apache.org>.
GitHub user chyzouakia added a comment to the discussion: S3TaskHandler doesn't seem to run the close method and doesn't push the logs to S3
I was able to push the logs from the worker and from local manually. Everything is explained in the description above, I could run the code that pushes the logs manually.
It feels like you didn't check the `what happened `section above and just dismissed the issue 🙂
GitHub link: https://github.com/apache/airflow/discussions/55402#discussioncomment-14359099
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] S3TaskHandler doesn't seem to run the close method and doesn't push the logs to S3 [airflow]
Posted by "chyzouakia (via GitHub)" <gi...@apache.org>.
GitHub user chyzouakia edited a comment on the discussion: S3TaskHandler doesn't seem to run the close method and doesn't push the logs to S3
I was able to push the logs from the worker and from local manually. Everything is explained in the description above, I could run the code that pushes the logs manually. I was not able to push the logs to S3 using the S3TaskHandler (with s3 logging configured). I tried checking the logs and couldn't find any error.
It feels like you didn't check the `what happened `section above and just dismissed the issue 🙂
All my tasks when trying to read in S3 the logs return `No logs found on s3` :
<img width="690" height="95" alt="image" src="https://github.com/user-attachments/assets/bd9023bb-7ac3-49ec-85c2-fa75a35a9500" />
I don't have any logs pushed into S3 automatically which lead me to believe there is an issue with the `close()` method in S3TaskHandler because :
* When I connect to any worker in my cluster, I am able to push logs to the S3 manually using the service account and the S3 remote logging connection in Airflow (The code is in the `what happened` section)
* When I push the logs manually to S3 the task is able to read the logs in S3 :
<img width="909" height="325" alt="image" src="https://github.com/user-attachments/assets/edc5dfcd-34bd-45b9-8d70-85ac471c1298" />
GitHub link: https://github.com/apache/airflow/discussions/55402#discussioncomment-14359099
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] S3TaskHandler doesn't seem to run the close method and doesn't push the logs to S3 [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
GitHub user potiuk added a comment to the discussion: S3TaskHandler doesn't seem to run the close method and doesn't push the logs to S3
> was not able to push the logs to S3 either 🙂
I think it would be good to look why -> if there is a reason you could not push it manually, it should be also the same reason logs are not sent by Airflow. You need to see what the problem is.
> What local logs are you referring to ? I couldn't find any logs, I tried all the exception messages in the logs and couldn't find one
Any logs produced in the logs folder of Airflow - see "logging" docs in Airflow documentation.
GitHub link: https://github.com/apache/airflow/discussions/55402#discussioncomment-14355394
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org