You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "potiuk (via GitHub)" <gi...@apache.org> on 2025/11/17 13:15:27 UTC

[PR] Add "update-providers-next-version" command to manage common provider… [airflow]

potiuk opened a new pull request, #58393:
URL: https://github.com/apache/airflow/pull/58393

   …s bump
   
   When contributors want to upgrade to the next version of a dependent provider (for example when new feature is added to a common providers and it is already used in another provider) - instead of bumping the minimum version of the dependent provider, contributors should mark it with `# use next version` comment. This comment will be picked by the new command `upgrade-providers-next-version` and the bump will happen after provider documentation is prepared for release.
   
   <!--
    Licensed to the Apache Software Foundation (ASF) under one
    or more contributor license agreements.  See the NOTICE file
    distributed with this work for additional information
    regarding copyright ownership.  The ASF licenses this file
    to you under the Apache License, Version 2.0 (the
    "License"); you may not use this file except in compliance
    with the License.  You may obtain a copy of the License at
   
      http://www.apache.org/licenses/LICENSE-2.0
   
    Unless required by applicable law or agreed to in writing,
    software distributed under the License is distributed on an
    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    KIND, either express or implied.  See the License for the
    specific language governing permissions and limitations
    under the License.
    -->
   
   <!--
   Thank you for contributing! Please make sure that your code changes
   are covered with tests. And in case of new features or big changes
   remember to adjust the documentation.
   
   Feel free to ping committers for the review!
   
   In case of an existing issue, reference it using one of the following:
   
   closes: #ISSUE
   related: #ISSUE
   
   How to write a good git commit message:
   http://chris.beams.io/posts/git-commit/
   -->
   
   
   
   <!-- Please keep an empty line above the dashes. -->
   ---
   **^ Add meaningful description above**
   Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information.
   In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add "update-providers-next-version" command to manage common provider… [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #58393:
URL: https://github.com/apache/airflow/pull/58393#issuecomment-3547403180

   > I think we can probably generalize to all providers and not only common ? 
   
   Yes.
   
   > Not sure what to in the case when someone wants to bump f.e. sql provider from 1.1 to 1.2 when the latest is already 1.5 - we'd probably have to check the current latest and allow this?
   
   Yes. But this can be done by just assigning the same label that release manager will assign to their PR.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add "update-providers-next-version" command to manage common provider… [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #58393:
URL: https://github.com/apache/airflow/pull/58393#issuecomment-3542144197

   Generally with that one ONLY RM should be able to bump provider versions. Zero exceptions


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add "update-providers-next-version" command to manage common provider… [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #58393:
URL: https://github.com/apache/airflow/pull/58393#issuecomment-3543123861

   > Selective checks is a good idea on PRs. Maybe we can think about a prek hook for it 🤔
   
   Prek is not best - because it's best to see "what's coming in PR" - prek hook are not really good for it  - we tried. 
   
   Prek naturally works on all files by default or on currently staged changes - and it consider those files as "whole" - and you need to do some extra checks to compare current vs. previous version and it might be heavily skewed anyway - becuase you might not be rebased, and have multiple commits and such - so naturally the scope of `prek hook` is `current commit`.
   
   What we are looking for here is "change  vs previous commit" - because we only want to flag situation when someone wants to change the dependent version in their PR (i.e. we have to compare vs. target branch - not what happened in a single commit).. Note that it should not flag if "other" parts of pyproject.toml changed -- we should only flag it if "version changed in the pr"
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add "update-providers-next-version" command to manage common provider… [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #58393:
URL: https://github.com/apache/airflow/pull/58393#issuecomment-3542134754

   > The more I think about it, the more questions/thoughts I have, but maybe they are all related to each other (I'll use common.compat as example):
   > 1. When this `use next version` comment is encountered, we use the value from provider.yml of specific provider, make a replacement and we remove this comment. Is there any mechanism that forces the release manager to first bump the versions in provider.yml
   
   Yes. Release process sequence 
   
   > 2. What is the incentive for contributors to use this method over explicitly bumping this version in the provider.yml of the dependent provider (e.g. common.compat)? I for sure would like to be certain, that this change is made and not merely left as a comment that may or may not be implemented to avoid potential problem with releasing provider with broken features (see above).
   
   2. They currently can't do it
   
   
   > 3. I guess it's related to the above, but does this mean that contributors have two ways of bumping this? 
   
   3. no. currently Contributors CAN'T and SHOULD NoT BUMP versions at all. Only RM can  looks t the comment at the top of project.toml


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add "update-providers-next-version" command to manage common provider… [airflow]

Posted by "kacpermuda (via GitHub)" <gi...@apache.org>.
kacpermuda commented on PR #58393:
URL: https://github.com/apache/airflow/pull/58393#issuecomment-3542251278

   >I am thinking about selective check failing and adding a label If someone tries to modify a cross dependency in project toml. So that RM will have to set that label on their PR.
   Thus however still does not fully solve the problem of how to communicate contributors what they should do (i.e. add the right comment) any ideas ?
   
   If this CI check's fail message would point to the doc paragraph about adding `use next version` comment, would that solve the problem?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add "update-providers-next-version" command to manage common provider… [airflow]

Posted by "kacpermuda (via GitHub)" <gi...@apache.org>.
kacpermuda commented on PR #58393:
URL: https://github.com/apache/airflow/pull/58393#issuecomment-3546073202

   > selective checks / prek hook to ensure nobody bumps the common provider version in pyproject.toml for other providers is a good idea
   
   I think we can probably generalize to all providers and not only common ? Not sure what to in the case when someone wants to bump f.e. sql provider from 1.1 to 1.2 when the latest is already 1.5 - we'd probably have to check the current latest and allow this?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add "update-providers-next-version" command to manage common provider… [airflow]

Posted by "bugraoz93 (via GitHub)" <gi...@apache.org>.
bugraoz93 commented on PR #58393:
URL: https://github.com/apache/airflow/pull/58393#issuecomment-3542763577

   Selective checks is a good idea on PRs. Maybe we can think about a prek hook for it :thinking:


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add "update-providers-next-version" command to manage common provider… [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #58393:
URL: https://github.com/apache/airflow/pull/58393#issuecomment-3544185318

   cc: @eladkal  -> I think that one will nicely solve the problem of bumping versions of dependent providers - only RM will be able to update them in "prepare-documentation" step.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add "update-providers-next-version" command to manage common provider… [airflow]

Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on PR #58393:
URL: https://github.com/apache/airflow/pull/58393#issuecomment-3545325108

   @potiuk selective checks / prek hook to ensure nobody bumps the common provider version in pyproject.toml for other providers is a good idea


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add "update-providers-next-version" command to manage common provider… [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #58393:
URL: https://github.com/apache/airflow/pull/58393#issuecomment-3544182718

   cc: @kacpermuda @bugraoz93 @jscheffl -> https://github.com/apache/airflow/pull/58412 this is how upgrade will look like in doc-preparation PR. I think it is fairly straightforward. Now we will have to have guardrails to prevent contributors to bump those cross-dependencies.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add "update-providers-next-version" command to manage common provider… [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #58393:
URL: https://github.com/apache/airflow/pull/58393#issuecomment-3541785443

   This is what RM will see:
   
   <img width="1095" height="811" alt="Screenshot 2025-11-17 at 14 14 37" src="https://github.com/user-attachments/assets/1d3ac013-05d2-422a-8ce9-72414648041c" />
   
   Now the only remaining part is how we make sure the comments are added.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add "update-providers-next-version" command to manage common provider… [airflow]

Posted by "bugraoz93 (via GitHub)" <gi...@apache.org>.
bugraoz93 commented on PR #58393:
URL: https://github.com/apache/airflow/pull/58393#issuecomment-3544270501

   Looks really centralized for providers. I like the approach. Making RM life easier. Decoupling and more automation on structured approach good for our release processes


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add "update-providers-next-version" command to manage common provider… [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #58393:
URL: https://github.com/apache/airflow/pull/58393#issuecomment-3543438020

   I will add those precautions later. In the meantime I would love to merge this one and #58390 so that I can try it on rc2 :)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add "update-providers-next-version" command to manage common provider… [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #58393:
URL: https://github.com/apache/airflow/pull/58393#issuecomment-3542121302

   The comment WILL be replaced.. it is now (with this PR) part of the regular release management process. So you can be sure 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add "update-providers-next-version" command to manage common provider… [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #58393:
URL: https://github.com/apache/airflow/pull/58393#issuecomment-3543101885

   > If this CI check's fail message would point to the doc paragraph about adding use next version comment, would that solve the problem?
   
   Yes. But again that's part of the problem only - because what happens if the contributes won't even try :) ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add "update-providers-next-version" command to manage common provider… [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #58393:
URL: https://github.com/apache/airflow/pull/58393#issuecomment-3542159757

   I am thinking about selective check failing and adding a label If someone tries to modify a cross dependency in project toml.  So that RM will have to set that label on their PR.
   
   
   Thus however still does not fully solve the problem of how to communicate contributors what they should do.  any ideas ?  


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add "update-providers-next-version" command to manage common provider… [airflow]

Posted by "kacpermuda (via GitHub)" <gi...@apache.org>.
kacpermuda commented on PR #58393:
URL: https://github.com/apache/airflow/pull/58393#issuecomment-3542059485

   Ad. 2 - I mean that I'd like to be sure that the dependency is bumped in parent provider (as it's critical for new features to work), so I'd go and bump the common.compat and then use the updated version as dependency in parent provider. Not sure if I'd trust that this comment will be replaced before releasing unless there is some check blocking it.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add "update-providers-next-version" command to manage common provider… [airflow]

Posted by "kacpermuda (via GitHub)" <gi...@apache.org>.
kacpermuda commented on PR #58393:
URL: https://github.com/apache/airflow/pull/58393#issuecomment-3542040747

   The more I think about it, the more questions/thoughts I have, but maybe they are all related to each other (I'll use common.compat as example):
   1. When this `use next version` comment is encountered, we use the value from provider.yml of specific provider, make a replacement and we remove this comment. Is there any mechanism that forces the release manager to first bump the versions in provider.yml of all providers and then run this automation? Otherwise, we may end up silently removing this comment BEFORE versions are bumped, therefore not actually bumping the dependency - possibly releasing provider with broken features (it will try to use features from new version of some provider without requiring it in dependencies).
   2. What is the incentive for contributors to use this method over explicitly bumping this version in the provider.yml of the dependent provider (e.g. common.compat)? I for sure would like to be certain, that this change is made and not merely left as a comment that may or may not be implemented to avoid potential problem with releasing provider with broken features (see above).
   3. I guess it's related to the above, but does this mean that contributors have two ways of bumping this? One can bump the version of common.compat in its provider.yml and use new version in parent provider, or just leave the comment in parent provider and count that it will be seen? May be a bit confusing if there are two ways of achieving the same, and possibly, what if one contributor chooses first and the other chooses the second - we'll probably count on release manager to spot that it has already been bumped and he does not have to do it manually and then run the automation to update other provider.yml files.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add "update-providers-next-version" command to manage common provider… [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #58393:
URL: https://github.com/apache/airflow/pull/58393#discussion_r2535686278


##########
dev/breeze/src/airflow_breeze/utils/packages.py:
##########


Review Comment:
   IT IS AN AI CODE. But I can ask AI to refactor :)
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add "update-providers-next-version" command to manage common provider… [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #58393:
URL: https://github.com/apache/airflow/pull/58393#issuecomment-3547911725

   PR. Here https://github.com/apache/airflow/pull/58430


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add "update-providers-next-version" command to manage common provider… [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #58393:
URL: https://github.com/apache/airflow/pull/58393#discussion_r2535702841


##########
dev/breeze/src/airflow_breeze/utils/packages.py:
##########


Review Comment:
   Done :) . It nicely split it and even added docs.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add "update-providers-next-version" command to manage common provider… [airflow]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #58393:
URL: https://github.com/apache/airflow/pull/58393#issuecomment-3544138899

   ### Backport failed to create: v3-1-test. View the failure log <a href='https://github.com/apache/airflow/actions/runs/19446820202'> Run details </a>
   
   <table>
       <tr>
           <th>Status</th>
           <th>Branch</th>
           <th>Result</th>
       </tr>
       <tr>
           <td>❌</td>
           <td>v3-1-test</td>
           <td><a href="https://github.com/apache/airflow/commit/61a3c55f1a4d4502bfac6b6eb70886dd8c5cf7f0"><img src='https://img.shields.io/badge/Commit-61a3c55-red' alt='Commit Link'></a></td>
       </tr>
   </table>
   
   You can attempt to backport this manually by running:
   
   ```bash
   cherry_picker 61a3c55 v3-1-test
   ```
   
   This should apply the commit to the v3-1-test branch and leave the commit in conflict state marking
   the files that need manual conflict resolution.
   
   After you have resolved the conflicts, you can continue the backport process by running:
   
   ```bash
   cherry_picker --continue
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add "update-providers-next-version" command to manage common provider… [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #58393:
URL: https://github.com/apache/airflow/pull/58393#discussion_r2535705189


##########
dev/breeze/src/airflow_breeze/utils/packages.py:
##########


Review Comment:
   That was quick :)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add "update-providers-next-version" command to manage common provider… [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk merged PR #58393:
URL: https://github.com/apache/airflow/pull/58393


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add "update-providers-next-version" command to manage common provider… [airflow]

Posted by "jscheffl (via GitHub)" <gi...@apache.org>.
jscheffl commented on code in PR #58393:
URL: https://github.com/apache/airflow/pull/58393#discussion_r2535658824


##########
dev/breeze/src/airflow_breeze/utils/packages.py:
##########


Review Comment:
   Looks like AI code. At least readable and I'd have expected prek or ruff compliants about level of nesting too deep.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org