You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/03/04 22:22:30 UTC
[GitHub] [airflow] kazanzhy opened a new pull request #22005: Update `moto` library to version 3.0
kazanzhy opened a new pull request #22005:
URL: https://github.com/apache/airflow/pull/22005
Some of the RDS functionality was added to `moto` recently.
Therefore, there are needed to upgrade version of the library
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #22005: Upgrade `moto` library to version 3.0
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #22005:
URL: https://github.com/apache/airflow/pull/22005#issuecomment-1059801597
> I had many problems with the "latest version" in my projects
>
> If we are not fixing the version of `moto` then we should keep in mind that we can have failed tests for all PRs if let's say in version 3.1 something will be changed.
This is what our constraint mechanism provides. All "regular" PRs if they do not change setup.py/setup.cfg use the constraints to run the PR - i.e. latest "good" version of dependencies. Only in `main` builds (after merge or on schedule) or in case someone modifies the setup.py/.cfg we run "eager upgrade" of all dependencies (and if the tests succeed, those become new constraints - they are automatically pushed to our repo).
This way:
1) we have stable set of dependencies for regular constraints
2) we automatically upgrade the stable set with the newly released versions (with every merge and on nightly schedule) - and only when ALL tests succeed with them (this is great for security!). When we have no frequent flakes, we have 2-3 updates a day https://github.com/apache/airflow/commits/constraints-main (because we have 580 dependencies in total)
3) when there is a breaking release we detect it (as our main and "dependecy changing" PRs start failing) - and it gives us a chance to fix them without impacting regular PRs
So - getting a failure in `main` when 3.1 is released with breaking change is GOOD and we are prepared to handle it - this will give us a chance to see it and fix it before most of the contributors even realise it happened.
If want to see more - my talk about it from November: https://youtu.be/_SjMdQLP30s?t=2519
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on pull request #22005: Upgrade `moto` library to version 3.0
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on pull request #22005:
URL: https://github.com/apache/airflow/pull/22005#issuecomment-1059804266
Ah yeah. With 580 deps we have ~ 20 new released dep versions every week :). Hard to keep up without the automation :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #22005: Upgrade `moto` library to version 3.0
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #22005:
URL: https://github.com/apache/airflow/pull/22005#issuecomment-1059801806
And here is our (agreed in the community after recent discussion) approach on how we treat dependencies: https://github.com/apache/airflow#approach-to-dependencies-of-airflow
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #22005: Upgrade `moto` library to version 3.0
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #22005:
URL: https://github.com/apache/airflow/pull/22005#issuecomment-1059731609
The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #22005: Upgrade `moto` library to version 3.0
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #22005:
URL: https://github.com/apache/airflow/pull/22005#issuecomment-1059793194
> But as I understand for Airflow 2.2 and less we will have moto==2.x and for Airflow 2.3+ there will be moto==3.0.
I think it does not really matter, moto is really a devel-only dependency - only used in tests, so it is just used for tests only. Once it works and constraints are updated - it will work. But I agree with @uranusjr, it might be a bit easier for us to **just** remove the limit (and it will get upgraded automatically). According to our newly agreed rules it should not be upper-bound (and this is even more so as this is test dependency only)
I will slightly modify it and see if it will pass (and use moto > 3) - it should.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk merged pull request #22005: Upgrade `moto` library to version 3.0
Posted by GitBox <gi...@apache.org>.
potiuk merged pull request #22005:
URL: https://github.com/apache/airflow/pull/22005
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #22005: Upgrade `moto` library to version 3.0
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #22005:
URL: https://github.com/apache/airflow/pull/22005#issuecomment-1059804266
Ah yeah. With 580 deps we have ~ 20 new release dep versions every week :). Hard to keep up without the automation :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kazanzhy commented on pull request #22005: Upgrade `moto` library to version 3.0
Posted by GitBox <gi...@apache.org>.
kazanzhy commented on pull request #22005:
URL: https://github.com/apache/airflow/pull/22005#issuecomment-1059803510
Thanks for the information.
That's a little bit more complex that I imagined
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #22005: Upgrade `moto` library to version 3.0
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #22005:
URL: https://github.com/apache/airflow/pull/22005#issuecomment-1059731504
This is fine - but it requires constraints to get regenerated. The failure there indicates that. I will need to manually update constraints and merge it at about the same time to accomodate !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #22005: Upgrade `moto` library to version 3.0
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #22005:
URL: https://github.com/apache/airflow/pull/22005#issuecomment-1059793560
UPDATE. Ah I see that we shoudl anyway have > 3.0.3 so I will just remove the ~ (and update the constraints as planned)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kazanzhy commented on pull request #22005: Upgrade `moto` library to version 3.0
Posted by GitBox <gi...@apache.org>.
kazanzhy commented on pull request #22005:
URL: https://github.com/apache/airflow/pull/22005#issuecomment-1059780314
Good question @uranusjr. I think it's not possible.
But as I understand for Airflow 2.2 and less we will have `moto==2.x` and for Airflow 2.3+ there will be `moto==3.0`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on pull request #22005: Upgrade `moto` library to version 3.0
Posted by GitBox <gi...@apache.org>.
uranusjr commented on pull request #22005:
URL: https://github.com/apache/airflow/pull/22005#issuecomment-1059754295
Is it impossible to support both 2.x and 3.0+?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #22005: Upgrade `moto` library to version 3.0
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #22005:
URL: https://github.com/apache/airflow/pull/22005#discussion_r820123867
##########
File path: setup.py
##########
@@ -619,9 +619,7 @@ def write_version(filename: str = os.path.join(*[my_dir, "airflow", "git_version
'jira',
'jsondiff',
'mongomock',
- # Moto3 is limited for unknown reason
- # TODO: attempt to remove the limitation
- 'moto~=2.2,>=2.2.12',
+ 'moto~=3.0,>=3.0.3',
Review comment:
```suggestion
'moto>=3.0.3',
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #22005: Upgrade `moto` library to version 3.0
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #22005:
URL: https://github.com/apache/airflow/pull/22005#issuecomment-1059803159
I rebuilt the constraints and merged :). All Good!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kazanzhy commented on pull request #22005: Upgrade `moto` library to version 3.0
Posted by GitBox <gi...@apache.org>.
kazanzhy commented on pull request #22005:
URL: https://github.com/apache/airflow/pull/22005#issuecomment-1059798598
I had many problems with the "latest version" in my projects
If we are not fixing the version of `moto` then we should keep in mind that we can have failed tests for all PRs if let's say in version 3.1 something will be changed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org