You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/08/03 17:33:22 UTC
[GitHub] [airflow] jedcunningham opened a new pull request #17397: Update to Celery 5
jedcunningham opened a new pull request #17397:
URL: https://github.com/apache/airflow/pull/17397
Celery 4 is no longer supported as of 2021-08-01:
https://docs.celeryproject.org/en/stable/history/whatsnew-5.0.html#long-term-support-policy
Closes: #11301
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jedcunningham closed pull request #17397: Update to Celery 5
Posted by GitBox <gi...@apache.org>.
jedcunningham closed pull request #17397:
URL: https://github.com/apache/airflow/pull/17397
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil merged pull request #17397: Update to Celery 5
Posted by GitBox <gi...@apache.org>.
kaxil merged pull request #17397:
URL: https://github.com/apache/airflow/pull/17397
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jedcunningham commented on a change in pull request #17397: Update to Celery 5
Posted by GitBox <gi...@apache.org>.
jedcunningham commented on a change in pull request #17397:
URL: https://github.com/apache/airflow/pull/17397#discussion_r695298146
##########
File path: Dockerfile.ci
##########
@@ -278,7 +278,7 @@ ENV AIRFLOW_REPO=${AIRFLOW_REPO}\
# * lazy-object-proxy<1.5.0: required by astroid
# * pyjwt<2.0.0: flask-jwt-extended requires it
# * dill<0.3.3 required by apache-beam
-ARG EAGER_UPGRADE_ADDITIONAL_REQUIREMENTS="lazy-object-proxy<1.5.0 pyjwt<2.0.0 dill<0.3.3 certifi<2021.0.0"
+ARG EAGER_UPGRADE_ADDITIONAL_REQUIREMENTS="lazy-object-proxy<1.5.0 pyjwt<2.0.0 dill<0.3.3 certifi<2021.0.0 botocore>=1.20.107,<1.21.0"
Review comment:
Props to @uranusjr for figuring out what the conflict was (`amqp`).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #17397: Update to Celery 5
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #17397:
URL: https://github.com/apache/airflow/pull/17397#issuecomment-896506628
The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #17397: Update to Celery 5
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #17397:
URL: https://github.com/apache/airflow/pull/17397#issuecomment-901347152
Shall I take a look at the deps ? tomorrow I might have some time :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on a change in pull request #17397: Update to Celery 5
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #17397:
URL: https://github.com/apache/airflow/pull/17397#discussion_r691116597
##########
File path: setup.py
##########
@@ -218,9 +218,8 @@ def write_version(filename: str = os.path.join(*[my_dir, "airflow", "git_version
'cassandra-driver>=3.13.0,<4',
]
celery = [
- 'celery~=4.4.2',
- 'flower>=0.7.3, <1.0',
- 'vine~=1.3', # https://stackoverflow.com/questions/32757259/celery-no-module-named-five
+ 'celery~=5.1.2',
Review comment:
This is equivalent to `>=5.1.2,<5.2` which probably isn't what we intend
```suggestion
'celery~=5.1,>=5.1.2',
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jedcunningham commented on pull request #17397: Update to Celery 5
Posted by GitBox <gi...@apache.org>.
jedcunningham commented on pull request #17397:
URL: https://github.com/apache/airflow/pull/17397#issuecomment-906687758
Shout out to @uranusjr for identifying the conflicting package! Ready for more eyes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr closed pull request #17397: Update to Celery 5
Posted by GitBox <gi...@apache.org>.
uranusjr closed pull request #17397:
URL: https://github.com/apache/airflow/pull/17397
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jedcunningham commented on a change in pull request #17397: Update to Celery 5
Posted by GitBox <gi...@apache.org>.
jedcunningham commented on a change in pull request #17397:
URL: https://github.com/apache/airflow/pull/17397#discussion_r685515621
##########
File path: Dockerfile.ci
##########
@@ -278,7 +278,7 @@ ENV AIRFLOW_REPO=${AIRFLOW_REPO}\
# * lazy-object-proxy<1.5.0: required by astroid
# * pyjwt<2.0.0: flask-jwt-extended requires it
# * dill<0.3.3 required by apache-beam
-ARG EAGER_UPGRADE_ADDITIONAL_REQUIREMENTS="lazy-object-proxy<1.5.0 pyjwt<2.0.0 dill<0.3.3 certifi<2021.0.0"
+ARG EAGER_UPGRADE_ADDITIONAL_REQUIREMENTS="lazy-object-proxy<1.5.0 pyjwt<2.0.0 dill<0.3.3 certifi<2021.0.0 botocore>=1.20.107,<1.21.0"
Review comment:
Hmm, okay I'll keep playing with it. This still wasn't enough to get it working and removing the lower limit made it even worse.
Btw, I think something is broken for these eager update PRs - it only tries to do the eager update when there is a single commit. If I add a second fixup commit, it installs with constraints and pip complains about the conflict. That's why I've been squashing my commits in this PR, but I haven't really dug into it yet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #17397: Update to Celery 5
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #17397:
URL: https://github.com/apache/airflow/pull/17397#discussion_r684608167
##########
File path: Dockerfile.ci
##########
@@ -278,7 +278,7 @@ ENV AIRFLOW_REPO=${AIRFLOW_REPO}\
# * lazy-object-proxy<1.5.0: required by astroid
# * pyjwt<2.0.0: flask-jwt-extended requires it
# * dill<0.3.3 required by apache-beam
-ARG EAGER_UPGRADE_ADDITIONAL_REQUIREMENTS="lazy-object-proxy<1.5.0 pyjwt<2.0.0 dill<0.3.3 certifi<2021.0.0"
+ARG EAGER_UPGRADE_ADDITIONAL_REQUIREMENTS="lazy-object-proxy<1.5.0 pyjwt<2.0.0 dill<0.3.3 certifi<2021.0.0 botocore>=1.20.107,<1.21.0"
Review comment:
Generally speaking here i think only upper bounds should be stated and only when this requirement is one of the transitive dependencies of 'airflow' package - so if you want to upper-bounds limit botocore in one of the extras, AND setup.cfg install-requires already add botocore as transitive dependency, then you need to add the upper-bound limit here for botocore. You should never need to have lower-bound limit here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr closed pull request #17397: Update to Celery 5
Posted by GitBox <gi...@apache.org>.
uranusjr closed pull request #17397:
URL: https://github.com/apache/airflow/pull/17397
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jedcunningham commented on pull request #17397: Update to Celery 5
Posted by GitBox <gi...@apache.org>.
jedcunningham commented on pull request #17397:
URL: https://github.com/apache/airflow/pull/17397#issuecomment-901573360
@potiuk, have at it. I've not made any progress over the last few days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #17397: Update to Celery 5
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #17397:
URL: https://github.com/apache/airflow/pull/17397#issuecomment-907004522
Oh yeah! Good stuff @uranusjr with the conflict :). This is yet another example where I am waiting for new PIP releases to show the conflicts better :). You need to automate your investigative skills there!.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr closed pull request #17397: Update to Celery 5
Posted by GitBox <gi...@apache.org>.
uranusjr closed pull request #17397:
URL: https://github.com/apache/airflow/pull/17397
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on a change in pull request #17397: Update to Celery 5
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #17397:
URL: https://github.com/apache/airflow/pull/17397#discussion_r691116597
##########
File path: setup.py
##########
@@ -218,9 +218,8 @@ def write_version(filename: str = os.path.join(*[my_dir, "airflow", "git_version
'cassandra-driver>=3.13.0,<4',
]
celery = [
- 'celery~=4.4.2',
- 'flower>=0.7.3, <1.0',
- 'vine~=1.3', # https://stackoverflow.com/questions/32757259/celery-no-module-named-five
+ 'celery~=5.1.2',
Review comment:
This is equivalent to `>=5.1.2,<5.2` which probably isn't what we intend
```suggestion
'celery~=5.1,>=5.1.2',
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #17397: Update to Celery 5
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #17397:
URL: https://github.com/apache/airflow/pull/17397#issuecomment-894628805
Ah sorry by accident marked it as ready for review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #17397: Update to Celery 5
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #17397:
URL: https://github.com/apache/airflow/pull/17397#discussion_r686251850
##########
File path: Dockerfile.ci
##########
@@ -278,7 +278,7 @@ ENV AIRFLOW_REPO=${AIRFLOW_REPO}\
# * lazy-object-proxy<1.5.0: required by astroid
# * pyjwt<2.0.0: flask-jwt-extended requires it
# * dill<0.3.3 required by apache-beam
-ARG EAGER_UPGRADE_ADDITIONAL_REQUIREMENTS="lazy-object-proxy<1.5.0 pyjwt<2.0.0 dill<0.3.3 certifi<2021.0.0"
+ARG EAGER_UPGRADE_ADDITIONAL_REQUIREMENTS="lazy-object-proxy<1.5.0 pyjwt<2.0.0 dill<0.3.3 certifi<2021.0.0 botocore>=1.20.107,<1.21.0"
Review comment:
ah yeah might be ... worth aking a look after I come back from vacations
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org