You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/11/19 11:21:10 UTC
[GitHub] [airflow] auvipy opened a new pull request #19703: try celery 5.2.1
auvipy opened a new pull request #19703:
URL: https://github.com/apache/airflow/pull/19703
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/main/UPDATING.md).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] auvipy commented on pull request #19703: upgrade celery 5.2.3
Posted by GitBox <gi...@apache.org>.
auvipy commented on pull request #19703:
URL: https://github.com/apache/airflow/pull/19703#issuecomment-1019013399
really happy to know this!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #19703: upgrde celery 5.2.3
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #19703:
URL: https://github.com/apache/airflow/pull/19703#issuecomment-1003763949
That should be enough @auvipy !
Looks good all tests passed. Anyone knows any potential problem involved with it ?
When we merge it, we will have to refresh the constraints as this is conflicting with old ones, but I am happy to merge it it any other committer confirms it is fine to merge.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] auvipy commented on pull request #19703: try celery 5.2.1
Posted by GitBox <gi...@apache.org>.
auvipy commented on pull request #19703:
URL: https://github.com/apache/airflow/pull/19703#issuecomment-1003704625
let me know what else needs to be updated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #19703: upgrade celery 5.2.3
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #19703:
URL: https://github.com/apache/airflow/pull/19703#issuecomment-1004041117
I will upgrade constratints and merge this one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on a change in pull request #19703: upgrade celery 5.2.3
Posted by GitBox <gi...@apache.org>.
uranusjr commented on a change in pull request #19703:
URL: https://github.com/apache/airflow/pull/19703#discussion_r777274833
##########
File path: setup.py
##########
@@ -227,7 +227,7 @@ def write_version(filename: str = os.path.join(*[my_dir, "airflow", "git_version
'cassandra-driver>=3.13.0,<4',
]
celery = [
- 'celery~=5.1,>=5.1.2',
+ 'celery~=5.2,>=5.2.3',
Review comment:
Can this not continue to allow Celery 5.1.x by declaring `celery>=5.1` instead?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #19703: upgrade celery 5.2.3
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #19703:
URL: https://github.com/apache/airflow/pull/19703#discussion_r777370408
##########
File path: setup.py
##########
@@ -227,7 +227,7 @@ def write_version(filename: str = os.path.join(*[my_dir, "airflow", "git_version
'cassandra-driver>=3.13.0,<4',
]
celery = [
- 'celery~=5.1,>=5.1.2',
+ 'celery~=5.2,>=5.2.3',
Review comment:
IF this one contains some security issue, then I am ok with >=5.2.3.
But I thought a bit and we should really remove the upper limit - this is our default behaviour and our constraint mechanism will keep it in check but we will avoid any need for future changes like that for 5.3 and above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #19703: upgrade celery 5.2.3
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #19703:
URL: https://github.com/apache/airflow/pull/19703#issuecomment-1004041468
The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] auvipy commented on pull request #19703: upgrade celery 5.2.3
Posted by GitBox <gi...@apache.org>.
auvipy commented on pull request #19703:
URL: https://github.com/apache/airflow/pull/19703#issuecomment-1019016519
> I'm not sure regarding your release cadence, but this includes some pretty big fixes.
can you please try https://github.com/celery/py-amqp/pull/368 and linked PR? I will be very grateful
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] auvipy edited a comment on pull request #19703: upgrade celery 5.2.3
Posted by GitBox <gi...@apache.org>.
auvipy edited a comment on pull request #19703:
URL: https://github.com/apache/airflow/pull/19703#issuecomment-1003809598
actually it contains several memory leaks/usage issues fixes in amqp & redis & 1 moderate but very old security issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #19703: upgrade celery 5.2.3
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #19703:
URL: https://github.com/apache/airflow/pull/19703#issuecomment-1004041015
The error is something I will have to fix - nothing to worry about (aftermath of #20624 - which I was not able to quickly test).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #19703: upgrade celery 5.2.3
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #19703:
URL: https://github.com/apache/airflow/pull/19703#discussion_r777370408
##########
File path: setup.py
##########
@@ -227,7 +227,7 @@ def write_version(filename: str = os.path.join(*[my_dir, "airflow", "git_version
'cassandra-driver>=3.13.0,<4',
]
celery = [
- 'celery~=5.1,>=5.1.2',
+ 'celery~=5.2,>=5.2.3',
Review comment:
IF this one contains some security issue and memory leaks, then I am ok with >=5.2.3.
But I thought a bit and we should really remove the upper limit - this is our default behaviour and our constraint mechanism will keep it in check but we will avoid any need for future changes like that for 5.3 and above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #19703: upgrade celery 5.2.3
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #19703:
URL: https://github.com/apache/airflow/pull/19703#issuecomment-1004635923
The Issues is JIRA are not used any more - look at the https://github.com/apache/airflow/contribute instead.
The instrructions for contribution are thre too (top right CONTRIBUTING.rst) link :).
Also you might take a look at a contribution workshop we held last year; https://youtu.be/kvccZizzfTk
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on pull request #19703: upgrade celery 5.2.3
Posted by GitBox <gi...@apache.org>.
eladkal commented on pull request #19703:
URL: https://github.com/apache/airflow/pull/19703#issuecomment-1004896670
> I wish to contribute to good first issues
You can find good first issues with this [link](https://github.com/apache/airflow/issues?q=is%3Aopen+is%3Aissue+label%3A%22good+first+issue%22+sort%3Aupdated-desc)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] MrManicotti edited a comment on pull request #19703: upgrade celery 5.2.3
Posted by GitBox <gi...@apache.org>.
MrManicotti edited a comment on pull request #19703:
URL: https://github.com/apache/airflow/pull/19703#issuecomment-1018778222
FYI this version increase includes some much needed fixes regarding memory leaks in Celery w/ Redis brokers. We were experiencing our airflow workers failing from memory leaks due to a problem with health checks - after some further investigation it seems due to https://github.com/celery/celery/issues/4843.
Pretty good description of the issue in https://github.com/celery/celery/issues/4843#issuecomment-991394349
For our side, a fix was made in **Kombo 5.2.3** and included in **Celery 5.2.3** which resolve the memory leak issue:
https://github.com/celery/kombu/pull/1476 to resolve the leaking `Transport` objects that stayed in mem from closed connections
A change was also included to shrink the overall memory usage of `Transport` and `Connection` objects in case of further memory leaks that were missed. https://github.com/celery/kombu/pull/1470
For an overall review of what was changed to resolve the memory leaks, please see https://github.com/celery/celery/issues/4843#issuecomment-1000657871
I'm not sure regarding your release cadence, but this includes some pretty big fixes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #19703: upgrade celery 5.2.3
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #19703:
URL: https://github.com/apache/airflow/pull/19703#issuecomment-1004131266
Actually we were already at 5.2.3 so it was not needed to upgrade constraints before merge
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk merged pull request #19703: upgrade celery 5.2.3
Posted by GitBox <gi...@apache.org>.
potiuk merged pull request #19703:
URL: https://github.com/apache/airflow/pull/19703
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] auvipy commented on pull request #19703: upgrade celery 5.2.3
Posted by GitBox <gi...@apache.org>.
auvipy commented on pull request #19703:
URL: https://github.com/apache/airflow/pull/19703#issuecomment-1003809598
actually it contains several memory leaks/usage issues in amqp & redis & 1 moderate but very old security issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] auvipy commented on pull request #19703: upgrade celery 5.2.3
Posted by GitBox <gi...@apache.org>.
auvipy commented on pull request #19703:
URL: https://github.com/apache/airflow/pull/19703#issuecomment-1004826163
ok though signed up! will check the video
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #19703: upgrade celery 5.2.3
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #19703:
URL: https://github.com/apache/airflow/pull/19703#issuecomment-1018799634
> For our side, a fix was made in Kombo 5.2.3 and included in Celery 5.2.3 which resolve the memory leak issue:
Consider it done!
As mentioned before - our automated uggrade process already picked it up and I am also just about to cherry-pick and refresh constraints and docker images for 2.2 branch for the upcoming 2.2.4 release (weeks rather than months).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] auvipy commented on a change in pull request #19703: upgrade celery 5.2.3
Posted by GitBox <gi...@apache.org>.
auvipy commented on a change in pull request #19703:
URL: https://github.com/apache/airflow/pull/19703#discussion_r777275872
##########
File path: setup.py
##########
@@ -227,7 +227,7 @@ def write_version(filename: str = os.path.join(*[my_dir, "airflow", "git_version
'cassandra-driver>=3.13.0,<4',
]
celery = [
- 'celery~=5.1,>=5.1.2',
+ 'celery~=5.2,>=5.2.3',
Review comment:
we usually only support the latest stable version in celery & https://github.com/celery/celery/releases/tag/v5.2.2 is not back ported.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #19703: upgrade celery 5.2.3
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #19703:
URL: https://github.com/apache/airflow/pull/19703#discussion_r777369491
##########
File path: setup.py
##########
@@ -227,7 +227,7 @@ def write_version(filename: str = os.path.join(*[my_dir, "airflow", "git_version
'cassandra-driver>=3.13.0,<4',
]
celery = [
- 'celery~=5.1,>=5.1.2',
+ 'celery~=5.2,>=5.2.3',
Review comment:
```suggestion
'celery>=5.2.3',
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] auvipy commented on a change in pull request #19703: upgrade celery 5.2.3
Posted by GitBox <gi...@apache.org>.
auvipy commented on a change in pull request #19703:
URL: https://github.com/apache/airflow/pull/19703#discussion_r777398418
##########
File path: setup.py
##########
@@ -227,7 +227,7 @@ def write_version(filename: str = os.path.join(*[my_dir, "airflow", "git_version
'cassandra-driver>=3.13.0,<4',
]
celery = [
- 'celery~=5.1,>=5.1.2',
+ 'celery~=5.2,>=5.2.3',
Review comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] auvipy commented on pull request #19703: upgrade celery 5.2.3
Posted by GitBox <gi...@apache.org>.
auvipy commented on pull request #19703:
URL: https://github.com/apache/airflow/pull/19703#issuecomment-1004487249
thanks. btw I have triage access to github, can I use https://issues.apache.org/jira/projects/AIRFLOW/issues/AIRFLOW-4470?filter=allopenissues as well? any instructions? I wish to contribute to good first issues
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] MrManicotti commented on pull request #19703: upgrade celery 5.2.3
Posted by GitBox <gi...@apache.org>.
MrManicotti commented on pull request #19703:
URL: https://github.com/apache/airflow/pull/19703#issuecomment-1018778222
FYI this version increase includes some much needed fixes regarding memory leaks in Celery w/ Redis brokers. We were experiencing our airflow workers failing from memory leaks due to health checks - after some further investigation it seems due to https://github.com/celery/celery/issues/4843.
Pretty good description of the issue in https://github.com/celery/celery/issues/4843#issuecomment-991394349
For our side, a fix was made in **Kombo 5.2.3** and included in **Celery 5.2.3** which resolve the memory leak issue:
https://github.com/celery/kombu/pull/1476 to resolve the leaking `Transport` objects that stayed in mem from closed connections
A change was also included to shrink the overall memory usage of `Transport` and `Connection` objects in case of further memory leaks that were missed. https://github.com/celery/kombu/pull/1470
For an overall review of what was changed to resolve the memory leaks, please see https://github.com/celery/celery/issues/4843#issuecomment-1000657871
I'm not sure regarding your release cadence, but this includes some pretty big fixes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org