[GitHub] [airflow] potiuk commented on pull request #27067: Remove deprecated dag_file_processor_timeouts metric

[GitBox <gi...@apache.org>]

potiuk commented on PR #27067:
URL: https://github.com/apache/airflow/pull/27067#issuecomment-1288237635

   > The cruelty of SemVer I suppose!
   
   I think this is  just the price to pay to make our users confident they can upgrade without worrying about incompatibilities. This is the main reason we adopted SemVer, so that we can convinvce our users, that they should migrate early and often without a fear of doing so. I think (based on our survey - https://airflow.apache.org/blog/airflow-survey-2022/ ) we mostly succeeded as the distribution of Airlfow versions is getting very much to what I expected to be (very similar to Python version distribution - with the very same reason behind I think), where users are migrating to newer versions way faster than they did in Airlfow 1.10. I believe adopting SemVer and spreading the word about 'safer" migrations play important part in it (on top of adding really cool features of course). Of course with Providers introducing their own SemVer it makes it more difficult to migrate sometimes, but this is one more reason to migrate often and early - becasue you can address provider incompati
 blities one-by-one. This is the "story" we keep on telling our users and using SemVer is part of that "story".
   
   I think as long as "cruelty" does not make a havy toll on maintianers to keep the backwards compatibility, this is not cruel really. Also I think we should figure ways where maintinag compatibility is done in the way that it does not have any impact on maintainability ( I think #26179 is a very good example of that - where the way we provide the compatibilty hsa prety much zero overhead on maintenance).  Whenever possible I think we should find a way to keep compatibility in the way that has similar zero-overhead and does not make further development problematic. I see that as much better approach than removing deprecations lightly. And quite often it is a one-time effort to figure out how to do it without leaving a lot of maintainance effort to keep it (for example we can move deprecated code to separate packages/folders where it can be all-but-forgotten). 
   
   Only when we have a very good reason to break compatibility and huge maintenance effort, we should do it IMHO.  This should be a very deliberate decision and alternatives should be considered before we knowingly break compatibility.
   
   > do we use any mechanism to track all the changes that need to be released before the next major version?
   
   There was an attempt to do it for K8S https://github.com/apache/airflow/pull/26848/files#diff-04b8bc56eb54d028990da2c4cab3101eb2a4aec4f6c8aad34d5230272b5f9874R335 l by @dstandish and add tests that will fail when deprecations are still there. 
   
   This would be a nice contribution to track those cases we have already and add tests for it in similar way (possibly even figure out how to do it with minimum effort as a parameterized set of tests. 
   
   However I think it makes little sense until we actually agree that there is an Airflow 3 **at all**. So far we have not discussed neither whether at allo, and when we will even attempt to release Airflow 3. 
   
   I personally think, unless we have very good reason we should not release Airlfow 3 unless we want to make some marketing around it and it has to be very well thought - why we are doing it and what are the consequences of it. This is pretty much the same approach as Python maintainers have - no-one even attemps to discuss Python 4. And after the Python 2.7 -> Python 3 drama, there must be very good reasons to even start discussing it.
   
   Also one more point -- getting to the subject of this PR. 
   
   I think the way it is currently done is a **very good one**. I personally think, there is no reason to document deprecated parameters, classes etc.  This is precisely what we've done in #26179 - while the deprecated classes continue to work via dynamic attrs, you do not see them any more. This is one of the best ways you can handle deprecations - if someone used it in the past, they will continue to work, but lack of documentation on them and lack of "easy way' of using it will make new users ... not use them. While we keep compatibility. This is precisely the behaviour we want to induce:
   
   * make 'old users" continue without doing changes if they want (but continue to be annoyed by warnings)
   * make "new users" not even aware that there is a deprecated parameter they can use
   
   I think it is good as it is now.
    


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org