You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/02/23 18:52:46 UTC
[GitHub] [airflow] ecerulm opened a new pull request #14399: Avoid using threads in S3 remote logging
ecerulm opened a new pull request #14399:
URL: https://github.com/apache/airflow/pull/14399
This prevents `RuntimeError: cannot schedule new futures after
interpreter shutdown` as described in #14089.
Closes: #14089
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on pull request #14399: Avoid using threads in S3 remote logging
Posted by GitBox <gi...@apache.org>.
ashb commented on pull request #14399:
URL: https://github.com/apache/airflow/pull/14399#issuecomment-785038720
Let's go with https://github.com/apache/airflow/pull/1441 over this one.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ecerulm commented on pull request #14399: Avoid using threads in S3 remote logging
Posted by GitBox <gi...@apache.org>.
ecerulm commented on pull request #14399:
URL: https://github.com/apache/airflow/pull/14399#issuecomment-784474383
I've tested this by creating a new docker image with the changes:
```
FROM apache/airflow:2.0.1
COPY airflow /home/airflow/.local/lib/python3.6/site-packages/airflow
```
where the `airflow` directory contains the changed files:
```
find airflow -type f
airflow/providers/amazon/aws/hooks/s3.py
airflow/providers/amazon/aws/log/s3_task_handler.py
```
after building `docker build . -t xxxx.dkr.ecr.eu-north-1.amazonaws.com/airflow:2.0.1` and pushing to ECR with `docker push xxxx.dkr.ecr.eu-north-1.amazonaws.com/airflow:2.0.1` I've used the the current helm chart with
```
defaultAirflowTag: 2.0.1
defaultAirflowRepository: xxxx.dkr.ecr.eu-north-1.amazonaws.com/airflow
```
I can see that remote logging with S3 is now working and the task pods do not get stuck trying to upload the log to S3.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ecerulm closed pull request #14399: Avoid using threads in S3 remote logging
Posted by GitBox <gi...@apache.org>.
ecerulm closed pull request #14399:
URL: https://github.com/apache/airflow/pull/14399
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb edited a comment on pull request #14399: Avoid using threads in S3 remote logging
Posted by GitBox <gi...@apache.org>.
ashb edited a comment on pull request #14399:
URL: https://github.com/apache/airflow/pull/14399#issuecomment-785038720
Let's go with https://github.com/apache/airflow/pull/14414 over this one.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ecerulm commented on pull request #14399: Avoid using threads in S3 remote logging
Posted by GitBox <gi...@apache.org>.
ecerulm commented on pull request #14399:
URL: https://github.com/apache/airflow/pull/14399#issuecomment-785038719
Closed in favor of #14414
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb closed pull request #14399: Avoid using threads in S3 remote logging
Posted by GitBox <gi...@apache.org>.
ashb closed pull request #14399:
URL: https://github.com/apache/airflow/pull/14399
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org