You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Jarek Potiuk <ja...@potiuk.com> on 2021/12/30 09:57:33 UTC
[DISCUSSION] Make one of the next provider waves (January or February) 2.3+ only for all providers
Hello Everyone,
We are close to the 2.3.0 release and we are starting to accumulate
changes that make the maintenance of Airflow 2.1 compatibility in
Providers more and more difficult. We implemented a lot of changes in
the Airflow core that we could take advantage of that we are not able
to now (common utils and functions) and we have a number of
deprecations we could remove if we get rid of 2.1 and 2.2
compatibility.
There is also a big change coming in Kubernetes Pod Operator (and
cncf.kubernetes) which will vastly improve and simplify the way it is
use and keeping 2.1 compatibility seems to be unnecessary and quite a
heavy burden (more about it here:
https://apache-airflow.slack.com/archives/CCPRP7943/p1640840944067500).
Also, we have a number of typing improvements in our "Bring back MyPY"
project https://github.com/apache/airflow/issues/19891 (which is
progressing well) and a number of benefits from applying those changes
across the board (one of the most important - Context being typing and
IDE friendly). Those will only show its true potential in Airflow
2.3+. Currently we have some workarounds related to that across pretty
much all providers (`if TYPE_CHECKING`) but they have some side
effects, and we could get rid of them and simplify provider's code if
we make our providers 2.3+ only.
I have a feeling that - similarly to what we 've done for Airflow 2.1
and @apply_defaults - it's about the time to make our Providers
"Airflow 2.3+ only" in one of the upcoming waves of providers.
This has also the nice benefit of incentivising our users to migrate
to 2.3+ if they need some features or bug-fixes only available in a
2.3+ provider. And this is a good thing IMHO.
What do you all think? Is it a good idea? Any strong opinions and
reasons why we should not do it?
J.
Re: [DISCUSSION] Make one of the next provider waves (January or February) 2.3+ only for all providers
Posted by Jarek Potiuk <ja...@potiuk.com>.
I proposed PR for that one: https://github.com/apache/airflow/pull/21696 -
if there are any comments, feel free to comment there, Otherwise I
will call for a lazy consensus on that (unless there will be some doubts
and concerns).
On Fri, Feb 11, 2022 at 1:01 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> Just to conclude - I think we should consider this change when needed. And
> one year seems to be fine. I think this also applies when we release
> Airflow 3 (so we should support the latest 2.x line for 12 months after the
> 2.x version has been released.
>
> One thing that we need to clarify is whether we take the FIRST or the LAST
> release from the X.Y line.
>
> We could support providers for 12 months after the "LAST" patchlevel
> released in this version:
>
> """
> Providers released by the community have a minimum supported version of
> Airflow. The minimum version of Airflow is airflow's minor version (2.1.0,
> 2.2.0 etc.) indicating that the providers might use features that appeared
> in this release. By default (there could be justified exceptions) we drop
> the minimum Airflow version, when 12 months passed since the last
> PATCHELEVEL release for the MINOR version.
> For example this means that by default we can upgrade the minimum version
> of Airflow supported by providers to 2.2.0 in the first Provider's release
> after 18th of September 2022 (18th of September 2021 is the date when the
> last PATCHLEVEL of 2.1 (2.1.4) has been released).
> """
>
> Alternatively we could support the Airflow version 12 months after the
> "FIRST" release of the line has been released (x.y.0). Then the statement
> would be:
>
> """
> Providers released by the community have a minimum supported version of
> Airflow. The minimum version of Airflow is airflow's minor version (2.1.0,
> 2.2.0 etc.) indicating that the providers might use features that appeared
> in this release. By default (there could be justified exceptions) we drop
> the minimum Airflow version, when 12 months passed since the first release
> for the MINOR version.
> For example this means that by default we can upgrade the minimum version
> of Airflow supported by providers to 2.2.0 in the first Provider's release
> after 21st of May 2022 (21st of May 2021 is the date when the first version
> of 2.1 (2.1.0) has been released).
> """
>
> I am leaning for the "FIRST". This is much more "foreseable" rule -
> especially that there might be cases (critical security bugs) that we will
> issue a very small "bugfix" release of (for example 2.2.5 version) even
> months after the last "regular" release in this line. And I think it makes
> perfect sense - the patchlevels we do, are basically already doing this -
> supporting fixes to something released in the .0 version for some time. It
> would make sense to do the same for providers.
>
> WDYT?
>
> On Mon, Jan 24, 2022 at 8:59 PM Kaxil Naik <ka...@gmail.com> wrote:
>
>> Completely agree with Elad. Targetting one year seems reasonable, however
>> we should be clear that we will have exceptions when there is no other
>> options.
>>
>> On Sun, Jan 23, 2022 at 6:29 AM Jarek Potiuk <ja...@potiuk.com> wrote:
>>
>>> Any comments from others :) ? This is not something urgent, I think the
>>> most important part here is to "catch" the moment where maintaining
>>> providers to keep backwards compatibility becomes a burden. So far I
>>> Thiink there are no "huge" problems. Some common utils and some name
>>> changes. but I do not see anything "serious" enough to break the limits
>>>
>>> But maybe there are some strong points.
>>>
>>> J,
>>>
>>> On Sat, Jan 8, 2022 at 12:10 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>>>
>>>> Thanks Elad,
>>>>
>>>> Important point indeed.
>>>>
>>>> Agree this might be frustrating, but on the other hand we have to make
>>>> sure Airflow is maintainable in the long term - this means for example
>>>> that in a year or two we will have Airflow 2.0, Airflow 2.1, Airflow
>>>> 2.2, Airflow 2.3, Airflow 2.4. Airflow 2.5 lines. Does it mean that we
>>>> will be releasing fixes to all the 'minor' lines ? We are already in
>>>> the situation that we have not released a single fix to 2.0 once 2.1
>>>> is released, we have not released a single fix to 2.1 once 2.2 was
>>>> released and we've never released a single fix to 2.2 once 2.3 was
>>>> released. Does it mean that 2.0, 2.1 and 2.2 are bug-free? Nope. A lot
>>>> of bugs that were also present in 2.0 were released in 2.3.
>>>>
>>>> As I see our current "de-facto" policy is - whenever you need
>>>> something new OR you see a bug in 2.0, 2.1, 2.2 - unless there is a
>>>> workaround - migrate to the latest 2.3. So we already encourage our
>>>> users to go with the latest version (which is good).
>>>>
>>>> So for example releasing providers that are 2.3+ once we have 2.4 or
>>>> 2.5 seems to me like a natural extension of that approach. And yeah -
>>>> one of the important points of providers is that you can upgrade them
>>>> independently. And it should stay like that. I would never propose a
>>>> "fast" increase of min version. I think however that at some point
>>>> getting such a "bump" in min-version makes sense.
>>>>
>>>> How often ? When? I am not sure, but I am pretty sure it should happen
>>>> from time to time.
>>>>
>>>> However I do agree it might surprise some users that some providers
>>>> will not get more fixes for the version of airflow they have, and
>>>> surprise is never a good thing.
>>>>
>>>> Just a though maybe - what could help is just agreeing on the rules
>>>> now and communicating them would be a good thing to do?
>>>>
>>>> Same as we agreed on Python version/K8S version policies. If we do
>>>> some kind of agreement (for example - we support providers for - say 1
>>>> year after the first minor release of Airflow all providers we
>>>> guarantee that all providers released will be compatible with that
>>>> line?
>>>>
>>>> If we agree that 1 year is enough that would meaan:
>>>>
>>>> * 31 May 2022 - we bump min version to 2.2 (1 year after 2.1.0 was
>>>> released)
>>>> * 11 October 2022 - we bump min version to 2.3 (1 year after 2.2.0 was
>>>> released)
>>>>
>>>> We can change it to 1.5 year or even 2 years if we think 1 year is
>>>> enough, but I think 1 year should be pretty good.
>>>>
>>>> That would give a really nice cadence, it would be very
>>>> straightforward rule, easy to understand and reason about, it will
>>>> also give as an opportunity to periodically modernize our providers -
>>>> on the other hand it would give the users a significant period where
>>>> they can expect regular provider's release and enough time to prepare
>>>> for bigger migration.
>>>>
>>>> Also this is somewhat a community-imposed upgrade cadence. Which I
>>>> think is fair, reasonable and the only way we can actually (as a
>>>> community) give our users certainty that they will get a good level of
>>>> support from the community. It's really a trade - " You get the
>>>> software from us, we agree that we will maintain it with fixes, but in
>>>> return, you should follow this upgrade cadence if you want to get the
>>>> full support". This is just "fair". On the other hand I think it's
>>>> quite "unfair" from the users to expect from the community an infinite
>>>> time of support. Making clear rules on what to expect and rules that
>>>> will allow the community to provide a good level of support to the
>>>> users seems like a good idea.
>>>>
>>>> WDYT?
>>>>
>>>> J.
>>>>
>>>> On Wed, Jan 5, 2022 at 5:39 PM Elad Kalif <el...@apache.org> wrote:
>>>> >
>>>> > We need to be also mindful about the fact that not everyone is able
>>>> to migrate to the new Airflow version as soon as it's being released.
>>>> > One of the concepts of providers is that it's not tightly bound to
>>>> Airflow core - If we can give users more time before setting a minimum
>>>> version of airflow 2.3 that would be ideal.
>>>> > It's frustrating when a small bug fix that you really need is beyond
>>>> your reach due to some other issue that forced bumping version. It may be
>>>> more hussle for us but if we can release provider bug fixes/features first
>>>> and a few days later release the breaking changes in a separate release it
>>>> may be valuable for some users.
>>>> >
>>>> > On Thu, Dec 30, 2021 at 8:11 PM Kaxil Naik <ka...@gmail.com>
>>>> wrote:
>>>> >>
>>>> >> One other solution for just using "utils" from core is we can
>>>> release "airflow.backports" or "airflow.futures" package same as Python
>>>> Backports and Python Future where we can handle the compat shim without
>>>> restricting Airflow versions.
>>>> >>
>>>> >> Regards,
>>>> >> Kaxil
>>>> >>
>>>> >> On Thu, Dec 30, 2021 at 2:56 PM Jarek Potiuk <ja...@potiuk.com>
>>>> wrote:
>>>> >>>
>>>> >>> > At this point, I think 2.3.0 will be around early Feb to get
>>>> AIP-42 in. We can reassess how the progress is moving ahead with the AIP
>>>> mid-Jan and based on that we can either release 2.3.0 end of Jan without
>>>> AIP-42 changes and mark it for 2.4.0.
>>>> >>>
>>>> >>> Yeah Agree here. I do not think we need to make a decision now - we
>>>> >>> can do it later. But I'd love to hear various opinions - from our
>>>> >>> users, various stakeholders (Amazon/Google/Astronomer who run it as
>>>> a
>>>> >>> service for one).
>>>> >>>
>>>> >>> Just to be clear - I am not fully convinced myself if this is the
>>>> >>> right time to do it already (but there are some good indicators that
>>>> >>> we should at least start discussing it).
>>>> >>>
>>>> >>> This is more opening the discussion and gathering feedback/feelings
>>>> of
>>>> >>> people. I think such decisions should be discussed (and their
>>>> >>> consequences realized) well in-advance so that at the time of
>>>> decision
>>>> >>> we have all arguments and opinions and know the consequences.
>>>> >>>
>>>> >>> And it is not "strictly" necessary. It's just (like with all
>>>> similar
>>>> >>> cases) an increased burden for maintenance and extra/boilerplate
>>>> code
>>>> >>> that might be removed. In those cases there is rarely a "0/1"
>>>> decision
>>>> >>> that can be made "here we know for sure that we should increase
>>>> >>> min-version" - it's more of the collective thinking: "Are the
>>>> >>> burden/overhead heavy enough we don't want to carry it any more as a
>>>> >>> community".
>>>> >>>
>>>> >>> J.
>>>>
>>>
Re: [DISCUSSION] Make one of the next provider waves (January or February) 2.3+ only for all providers
Posted by Jarek Potiuk <ja...@potiuk.com>.
Just to conclude - I think we should consider this change when needed. And
one year seems to be fine. I think this also applies when we release
Airflow 3 (so we should support the latest 2.x line for 12 months after the
2.x version has been released.
One thing that we need to clarify is whether we take the FIRST or the LAST
release from the X.Y line.
We could support providers for 12 months after the "LAST" patchlevel
released in this version:
"""
Providers released by the community have a minimum supported version of
Airflow. The minimum version of Airflow is airflow's minor version (2.1.0,
2.2.0 etc.) indicating that the providers might use features that appeared
in this release. By default (there could be justified exceptions) we drop
the minimum Airflow version, when 12 months passed since the last
PATCHELEVEL release for the MINOR version.
For example this means that by default we can upgrade the minimum version
of Airflow supported by providers to 2.2.0 in the first Provider's release
after 18th of September 2022 (18th of September 2021 is the date when the
last PATCHLEVEL of 2.1 (2.1.4) has been released).
"""
Alternatively we could support the Airflow version 12 months after the
"FIRST" release of the line has been released (x.y.0). Then the statement
would be:
"""
Providers released by the community have a minimum supported version of
Airflow. The minimum version of Airflow is airflow's minor version (2.1.0,
2.2.0 etc.) indicating that the providers might use features that appeared
in this release. By default (there could be justified exceptions) we drop
the minimum Airflow version, when 12 months passed since the first release
for the MINOR version.
For example this means that by default we can upgrade the minimum version
of Airflow supported by providers to 2.2.0 in the first Provider's release
after 21st of May 2022 (21st of May 2021 is the date when the first version
of 2.1 (2.1.0) has been released).
"""
I am leaning for the "FIRST". This is much more "foreseable" rule -
especially that there might be cases (critical security bugs) that we will
issue a very small "bugfix" release of (for example 2.2.5 version) even
months after the last "regular" release in this line. And I think it makes
perfect sense - the patchlevels we do, are basically already doing this -
supporting fixes to something released in the .0 version for some time. It
would make sense to do the same for providers.
WDYT?
On Mon, Jan 24, 2022 at 8:59 PM Kaxil Naik <ka...@gmail.com> wrote:
> Completely agree with Elad. Targetting one year seems reasonable, however
> we should be clear that we will have exceptions when there is no other
> options.
>
> On Sun, Jan 23, 2022 at 6:29 AM Jarek Potiuk <ja...@potiuk.com> wrote:
>
>> Any comments from others :) ? This is not something urgent, I think the
>> most important part here is to "catch" the moment where maintaining
>> providers to keep backwards compatibility becomes a burden. So far I
>> Thiink there are no "huge" problems. Some common utils and some name
>> changes. but I do not see anything "serious" enough to break the limits
>>
>> But maybe there are some strong points.
>>
>> J,
>>
>> On Sat, Jan 8, 2022 at 12:10 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>>
>>> Thanks Elad,
>>>
>>> Important point indeed.
>>>
>>> Agree this might be frustrating, but on the other hand we have to make
>>> sure Airflow is maintainable in the long term - this means for example
>>> that in a year or two we will have Airflow 2.0, Airflow 2.1, Airflow
>>> 2.2, Airflow 2.3, Airflow 2.4. Airflow 2.5 lines. Does it mean that we
>>> will be releasing fixes to all the 'minor' lines ? We are already in
>>> the situation that we have not released a single fix to 2.0 once 2.1
>>> is released, we have not released a single fix to 2.1 once 2.2 was
>>> released and we've never released a single fix to 2.2 once 2.3 was
>>> released. Does it mean that 2.0, 2.1 and 2.2 are bug-free? Nope. A lot
>>> of bugs that were also present in 2.0 were released in 2.3.
>>>
>>> As I see our current "de-facto" policy is - whenever you need
>>> something new OR you see a bug in 2.0, 2.1, 2.2 - unless there is a
>>> workaround - migrate to the latest 2.3. So we already encourage our
>>> users to go with the latest version (which is good).
>>>
>>> So for example releasing providers that are 2.3+ once we have 2.4 or
>>> 2.5 seems to me like a natural extension of that approach. And yeah -
>>> one of the important points of providers is that you can upgrade them
>>> independently. And it should stay like that. I would never propose a
>>> "fast" increase of min version. I think however that at some point
>>> getting such a "bump" in min-version makes sense.
>>>
>>> How often ? When? I am not sure, but I am pretty sure it should happen
>>> from time to time.
>>>
>>> However I do agree it might surprise some users that some providers
>>> will not get more fixes for the version of airflow they have, and
>>> surprise is never a good thing.
>>>
>>> Just a though maybe - what could help is just agreeing on the rules
>>> now and communicating them would be a good thing to do?
>>>
>>> Same as we agreed on Python version/K8S version policies. If we do
>>> some kind of agreement (for example - we support providers for - say 1
>>> year after the first minor release of Airflow all providers we
>>> guarantee that all providers released will be compatible with that
>>> line?
>>>
>>> If we agree that 1 year is enough that would meaan:
>>>
>>> * 31 May 2022 - we bump min version to 2.2 (1 year after 2.1.0 was
>>> released)
>>> * 11 October 2022 - we bump min version to 2.3 (1 year after 2.2.0 was
>>> released)
>>>
>>> We can change it to 1.5 year or even 2 years if we think 1 year is
>>> enough, but I think 1 year should be pretty good.
>>>
>>> That would give a really nice cadence, it would be very
>>> straightforward rule, easy to understand and reason about, it will
>>> also give as an opportunity to periodically modernize our providers -
>>> on the other hand it would give the users a significant period where
>>> they can expect regular provider's release and enough time to prepare
>>> for bigger migration.
>>>
>>> Also this is somewhat a community-imposed upgrade cadence. Which I
>>> think is fair, reasonable and the only way we can actually (as a
>>> community) give our users certainty that they will get a good level of
>>> support from the community. It's really a trade - " You get the
>>> software from us, we agree that we will maintain it with fixes, but in
>>> return, you should follow this upgrade cadence if you want to get the
>>> full support". This is just "fair". On the other hand I think it's
>>> quite "unfair" from the users to expect from the community an infinite
>>> time of support. Making clear rules on what to expect and rules that
>>> will allow the community to provide a good level of support to the
>>> users seems like a good idea.
>>>
>>> WDYT?
>>>
>>> J.
>>>
>>> On Wed, Jan 5, 2022 at 5:39 PM Elad Kalif <el...@apache.org> wrote:
>>> >
>>> > We need to be also mindful about the fact that not everyone is able to
>>> migrate to the new Airflow version as soon as it's being released.
>>> > One of the concepts of providers is that it's not tightly bound to
>>> Airflow core - If we can give users more time before setting a minimum
>>> version of airflow 2.3 that would be ideal.
>>> > It's frustrating when a small bug fix that you really need is beyond
>>> your reach due to some other issue that forced bumping version. It may be
>>> more hussle for us but if we can release provider bug fixes/features first
>>> and a few days later release the breaking changes in a separate release it
>>> may be valuable for some users.
>>> >
>>> > On Thu, Dec 30, 2021 at 8:11 PM Kaxil Naik <ka...@gmail.com>
>>> wrote:
>>> >>
>>> >> One other solution for just using "utils" from core is we can release
>>> "airflow.backports" or "airflow.futures" package same as Python Backports
>>> and Python Future where we can handle the compat shim without restricting
>>> Airflow versions.
>>> >>
>>> >> Regards,
>>> >> Kaxil
>>> >>
>>> >> On Thu, Dec 30, 2021 at 2:56 PM Jarek Potiuk <ja...@potiuk.com>
>>> wrote:
>>> >>>
>>> >>> > At this point, I think 2.3.0 will be around early Feb to get
>>> AIP-42 in. We can reassess how the progress is moving ahead with the AIP
>>> mid-Jan and based on that we can either release 2.3.0 end of Jan without
>>> AIP-42 changes and mark it for 2.4.0.
>>> >>>
>>> >>> Yeah Agree here. I do not think we need to make a decision now - we
>>> >>> can do it later. But I'd love to hear various opinions - from our
>>> >>> users, various stakeholders (Amazon/Google/Astronomer who run it as a
>>> >>> service for one).
>>> >>>
>>> >>> Just to be clear - I am not fully convinced myself if this is the
>>> >>> right time to do it already (but there are some good indicators that
>>> >>> we should at least start discussing it).
>>> >>>
>>> >>> This is more opening the discussion and gathering feedback/feelings
>>> of
>>> >>> people. I think such decisions should be discussed (and their
>>> >>> consequences realized) well in-advance so that at the time of
>>> decision
>>> >>> we have all arguments and opinions and know the consequences.
>>> >>>
>>> >>> And it is not "strictly" necessary. It's just (like with all similar
>>> >>> cases) an increased burden for maintenance and extra/boilerplate code
>>> >>> that might be removed. In those cases there is rarely a "0/1"
>>> decision
>>> >>> that can be made "here we know for sure that we should increase
>>> >>> min-version" - it's more of the collective thinking: "Are the
>>> >>> burden/overhead heavy enough we don't want to carry it any more as a
>>> >>> community".
>>> >>>
>>> >>> J.
>>>
>>
Re: [DISCUSSION] Make one of the next provider waves (January or February) 2.3+ only for all providers
Posted by Kaxil Naik <ka...@gmail.com>.
Completely agree with Elad. Targetting one year seems reasonable, however
we should be clear that we will have exceptions when there is no other
options.
On Sun, Jan 23, 2022 at 6:29 AM Jarek Potiuk <ja...@potiuk.com> wrote:
> Any comments from others :) ? This is not something urgent, I think the
> most important part here is to "catch" the moment where maintaining
> providers to keep backwards compatibility becomes a burden. So far I
> Thiink there are no "huge" problems. Some common utils and some name
> changes. but I do not see anything "serious" enough to break the limits
>
> But maybe there are some strong points.
>
> J,
>
> On Sat, Jan 8, 2022 at 12:10 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>
>> Thanks Elad,
>>
>> Important point indeed.
>>
>> Agree this might be frustrating, but on the other hand we have to make
>> sure Airflow is maintainable in the long term - this means for example
>> that in a year or two we will have Airflow 2.0, Airflow 2.1, Airflow
>> 2.2, Airflow 2.3, Airflow 2.4. Airflow 2.5 lines. Does it mean that we
>> will be releasing fixes to all the 'minor' lines ? We are already in
>> the situation that we have not released a single fix to 2.0 once 2.1
>> is released, we have not released a single fix to 2.1 once 2.2 was
>> released and we've never released a single fix to 2.2 once 2.3 was
>> released. Does it mean that 2.0, 2.1 and 2.2 are bug-free? Nope. A lot
>> of bugs that were also present in 2.0 were released in 2.3.
>>
>> As I see our current "de-facto" policy is - whenever you need
>> something new OR you see a bug in 2.0, 2.1, 2.2 - unless there is a
>> workaround - migrate to the latest 2.3. So we already encourage our
>> users to go with the latest version (which is good).
>>
>> So for example releasing providers that are 2.3+ once we have 2.4 or
>> 2.5 seems to me like a natural extension of that approach. And yeah -
>> one of the important points of providers is that you can upgrade them
>> independently. And it should stay like that. I would never propose a
>> "fast" increase of min version. I think however that at some point
>> getting such a "bump" in min-version makes sense.
>>
>> How often ? When? I am not sure, but I am pretty sure it should happen
>> from time to time.
>>
>> However I do agree it might surprise some users that some providers
>> will not get more fixes for the version of airflow they have, and
>> surprise is never a good thing.
>>
>> Just a though maybe - what could help is just agreeing on the rules
>> now and communicating them would be a good thing to do?
>>
>> Same as we agreed on Python version/K8S version policies. If we do
>> some kind of agreement (for example - we support providers for - say 1
>> year after the first minor release of Airflow all providers we
>> guarantee that all providers released will be compatible with that
>> line?
>>
>> If we agree that 1 year is enough that would meaan:
>>
>> * 31 May 2022 - we bump min version to 2.2 (1 year after 2.1.0 was
>> released)
>> * 11 October 2022 - we bump min version to 2.3 (1 year after 2.2.0 was
>> released)
>>
>> We can change it to 1.5 year or even 2 years if we think 1 year is
>> enough, but I think 1 year should be pretty good.
>>
>> That would give a really nice cadence, it would be very
>> straightforward rule, easy to understand and reason about, it will
>> also give as an opportunity to periodically modernize our providers -
>> on the other hand it would give the users a significant period where
>> they can expect regular provider's release and enough time to prepare
>> for bigger migration.
>>
>> Also this is somewhat a community-imposed upgrade cadence. Which I
>> think is fair, reasonable and the only way we can actually (as a
>> community) give our users certainty that they will get a good level of
>> support from the community. It's really a trade - " You get the
>> software from us, we agree that we will maintain it with fixes, but in
>> return, you should follow this upgrade cadence if you want to get the
>> full support". This is just "fair". On the other hand I think it's
>> quite "unfair" from the users to expect from the community an infinite
>> time of support. Making clear rules on what to expect and rules that
>> will allow the community to provide a good level of support to the
>> users seems like a good idea.
>>
>> WDYT?
>>
>> J.
>>
>> On Wed, Jan 5, 2022 at 5:39 PM Elad Kalif <el...@apache.org> wrote:
>> >
>> > We need to be also mindful about the fact that not everyone is able to
>> migrate to the new Airflow version as soon as it's being released.
>> > One of the concepts of providers is that it's not tightly bound to
>> Airflow core - If we can give users more time before setting a minimum
>> version of airflow 2.3 that would be ideal.
>> > It's frustrating when a small bug fix that you really need is beyond
>> your reach due to some other issue that forced bumping version. It may be
>> more hussle for us but if we can release provider bug fixes/features first
>> and a few days later release the breaking changes in a separate release it
>> may be valuable for some users.
>> >
>> > On Thu, Dec 30, 2021 at 8:11 PM Kaxil Naik <ka...@gmail.com> wrote:
>> >>
>> >> One other solution for just using "utils" from core is we can release
>> "airflow.backports" or "airflow.futures" package same as Python Backports
>> and Python Future where we can handle the compat shim without restricting
>> Airflow versions.
>> >>
>> >> Regards,
>> >> Kaxil
>> >>
>> >> On Thu, Dec 30, 2021 at 2:56 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>> >>>
>> >>> > At this point, I think 2.3.0 will be around early Feb to get AIP-42
>> in. We can reassess how the progress is moving ahead with the AIP mid-Jan
>> and based on that we can either release 2.3.0 end of Jan without AIP-42
>> changes and mark it for 2.4.0.
>> >>>
>> >>> Yeah Agree here. I do not think we need to make a decision now - we
>> >>> can do it later. But I'd love to hear various opinions - from our
>> >>> users, various stakeholders (Amazon/Google/Astronomer who run it as a
>> >>> service for one).
>> >>>
>> >>> Just to be clear - I am not fully convinced myself if this is the
>> >>> right time to do it already (but there are some good indicators that
>> >>> we should at least start discussing it).
>> >>>
>> >>> This is more opening the discussion and gathering feedback/feelings of
>> >>> people. I think such decisions should be discussed (and their
>> >>> consequences realized) well in-advance so that at the time of decision
>> >>> we have all arguments and opinions and know the consequences.
>> >>>
>> >>> And it is not "strictly" necessary. It's just (like with all similar
>> >>> cases) an increased burden for maintenance and extra/boilerplate code
>> >>> that might be removed. In those cases there is rarely a "0/1" decision
>> >>> that can be made "here we know for sure that we should increase
>> >>> min-version" - it's more of the collective thinking: "Are the
>> >>> burden/overhead heavy enough we don't want to carry it any more as a
>> >>> community".
>> >>>
>> >>> J.
>>
>
Re: [DISCUSSION] Make one of the next provider waves (January or February) 2.3+ only for all providers
Posted by Jarek Potiuk <ja...@potiuk.com>.
Any comments from others :) ? This is not something urgent, I think the
most important part here is to "catch" the moment where maintaining
providers to keep backwards compatibility becomes a burden. So far I
Thiink there are no "huge" problems. Some common utils and some name
changes. but I do not see anything "serious" enough to break the limits
But maybe there are some strong points.
J,
On Sat, Jan 8, 2022 at 12:10 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> Thanks Elad,
>
> Important point indeed.
>
> Agree this might be frustrating, but on the other hand we have to make
> sure Airflow is maintainable in the long term - this means for example
> that in a year or two we will have Airflow 2.0, Airflow 2.1, Airflow
> 2.2, Airflow 2.3, Airflow 2.4. Airflow 2.5 lines. Does it mean that we
> will be releasing fixes to all the 'minor' lines ? We are already in
> the situation that we have not released a single fix to 2.0 once 2.1
> is released, we have not released a single fix to 2.1 once 2.2 was
> released and we've never released a single fix to 2.2 once 2.3 was
> released. Does it mean that 2.0, 2.1 and 2.2 are bug-free? Nope. A lot
> of bugs that were also present in 2.0 were released in 2.3.
>
> As I see our current "de-facto" policy is - whenever you need
> something new OR you see a bug in 2.0, 2.1, 2.2 - unless there is a
> workaround - migrate to the latest 2.3. So we already encourage our
> users to go with the latest version (which is good).
>
> So for example releasing providers that are 2.3+ once we have 2.4 or
> 2.5 seems to me like a natural extension of that approach. And yeah -
> one of the important points of providers is that you can upgrade them
> independently. And it should stay like that. I would never propose a
> "fast" increase of min version. I think however that at some point
> getting such a "bump" in min-version makes sense.
>
> How often ? When? I am not sure, but I am pretty sure it should happen
> from time to time.
>
> However I do agree it might surprise some users that some providers
> will not get more fixes for the version of airflow they have, and
> surprise is never a good thing.
>
> Just a though maybe - what could help is just agreeing on the rules
> now and communicating them would be a good thing to do?
>
> Same as we agreed on Python version/K8S version policies. If we do
> some kind of agreement (for example - we support providers for - say 1
> year after the first minor release of Airflow all providers we
> guarantee that all providers released will be compatible with that
> line?
>
> If we agree that 1 year is enough that would meaan:
>
> * 31 May 2022 - we bump min version to 2.2 (1 year after 2.1.0 was
> released)
> * 11 October 2022 - we bump min version to 2.3 (1 year after 2.2.0 was
> released)
>
> We can change it to 1.5 year or even 2 years if we think 1 year is
> enough, but I think 1 year should be pretty good.
>
> That would give a really nice cadence, it would be very
> straightforward rule, easy to understand and reason about, it will
> also give as an opportunity to periodically modernize our providers -
> on the other hand it would give the users a significant period where
> they can expect regular provider's release and enough time to prepare
> for bigger migration.
>
> Also this is somewhat a community-imposed upgrade cadence. Which I
> think is fair, reasonable and the only way we can actually (as a
> community) give our users certainty that they will get a good level of
> support from the community. It's really a trade - " You get the
> software from us, we agree that we will maintain it with fixes, but in
> return, you should follow this upgrade cadence if you want to get the
> full support". This is just "fair". On the other hand I think it's
> quite "unfair" from the users to expect from the community an infinite
> time of support. Making clear rules on what to expect and rules that
> will allow the community to provide a good level of support to the
> users seems like a good idea.
>
> WDYT?
>
> J.
>
> On Wed, Jan 5, 2022 at 5:39 PM Elad Kalif <el...@apache.org> wrote:
> >
> > We need to be also mindful about the fact that not everyone is able to
> migrate to the new Airflow version as soon as it's being released.
> > One of the concepts of providers is that it's not tightly bound to
> Airflow core - If we can give users more time before setting a minimum
> version of airflow 2.3 that would be ideal.
> > It's frustrating when a small bug fix that you really need is beyond
> your reach due to some other issue that forced bumping version. It may be
> more hussle for us but if we can release provider bug fixes/features first
> and a few days later release the breaking changes in a separate release it
> may be valuable for some users.
> >
> > On Thu, Dec 30, 2021 at 8:11 PM Kaxil Naik <ka...@gmail.com> wrote:
> >>
> >> One other solution for just using "utils" from core is we can release
> "airflow.backports" or "airflow.futures" package same as Python Backports
> and Python Future where we can handle the compat shim without restricting
> Airflow versions.
> >>
> >> Regards,
> >> Kaxil
> >>
> >> On Thu, Dec 30, 2021 at 2:56 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> >>>
> >>> > At this point, I think 2.3.0 will be around early Feb to get AIP-42
> in. We can reassess how the progress is moving ahead with the AIP mid-Jan
> and based on that we can either release 2.3.0 end of Jan without AIP-42
> changes and mark it for 2.4.0.
> >>>
> >>> Yeah Agree here. I do not think we need to make a decision now - we
> >>> can do it later. But I'd love to hear various opinions - from our
> >>> users, various stakeholders (Amazon/Google/Astronomer who run it as a
> >>> service for one).
> >>>
> >>> Just to be clear - I am not fully convinced myself if this is the
> >>> right time to do it already (but there are some good indicators that
> >>> we should at least start discussing it).
> >>>
> >>> This is more opening the discussion and gathering feedback/feelings of
> >>> people. I think such decisions should be discussed (and their
> >>> consequences realized) well in-advance so that at the time of decision
> >>> we have all arguments and opinions and know the consequences.
> >>>
> >>> And it is not "strictly" necessary. It's just (like with all similar
> >>> cases) an increased burden for maintenance and extra/boilerplate code
> >>> that might be removed. In those cases there is rarely a "0/1" decision
> >>> that can be made "here we know for sure that we should increase
> >>> min-version" - it's more of the collective thinking: "Are the
> >>> burden/overhead heavy enough we don't want to carry it any more as a
> >>> community".
> >>>
> >>> J.
>
Re: [DISCUSSION] Make one of the next provider waves (January or February) 2.3+ only for all providers
Posted by Jarek Potiuk <ja...@potiuk.com>.
Thanks Elad,
Important point indeed.
Agree this might be frustrating, but on the other hand we have to make
sure Airflow is maintainable in the long term - this means for example
that in a year or two we will have Airflow 2.0, Airflow 2.1, Airflow
2.2, Airflow 2.3, Airflow 2.4. Airflow 2.5 lines. Does it mean that we
will be releasing fixes to all the 'minor' lines ? We are already in
the situation that we have not released a single fix to 2.0 once 2.1
is released, we have not released a single fix to 2.1 once 2.2 was
released and we've never released a single fix to 2.2 once 2.3 was
released. Does it mean that 2.0, 2.1 and 2.2 are bug-free? Nope. A lot
of bugs that were also present in 2.0 were released in 2.3.
As I see our current "de-facto" policy is - whenever you need
something new OR you see a bug in 2.0, 2.1, 2.2 - unless there is a
workaround - migrate to the latest 2.3. So we already encourage our
users to go with the latest version (which is good).
So for example releasing providers that are 2.3+ once we have 2.4 or
2.5 seems to me like a natural extension of that approach. And yeah -
one of the important points of providers is that you can upgrade them
independently. And it should stay like that. I would never propose a
"fast" increase of min version. I think however that at some point
getting such a "bump" in min-version makes sense.
How often ? When? I am not sure, but I am pretty sure it should happen
from time to time.
However I do agree it might surprise some users that some providers
will not get more fixes for the version of airflow they have, and
surprise is never a good thing.
Just a though maybe - what could help is just agreeing on the rules
now and communicating them would be a good thing to do?
Same as we agreed on Python version/K8S version policies. If we do
some kind of agreement (for example - we support providers for - say 1
year after the first minor release of Airflow all providers we
guarantee that all providers released will be compatible with that
line?
If we agree that 1 year is enough that would meaan:
* 31 May 2022 - we bump min version to 2.2 (1 year after 2.1.0 was released)
* 11 October 2022 - we bump min version to 2.3 (1 year after 2.2.0 was released)
We can change it to 1.5 year or even 2 years if we think 1 year is
enough, but I think 1 year should be pretty good.
That would give a really nice cadence, it would be very
straightforward rule, easy to understand and reason about, it will
also give as an opportunity to periodically modernize our providers -
on the other hand it would give the users a significant period where
they can expect regular provider's release and enough time to prepare
for bigger migration.
Also this is somewhat a community-imposed upgrade cadence. Which I
think is fair, reasonable and the only way we can actually (as a
community) give our users certainty that they will get a good level of
support from the community. It's really a trade - " You get the
software from us, we agree that we will maintain it with fixes, but in
return, you should follow this upgrade cadence if you want to get the
full support". This is just "fair". On the other hand I think it's
quite "unfair" from the users to expect from the community an infinite
time of support. Making clear rules on what to expect and rules that
will allow the community to provide a good level of support to the
users seems like a good idea.
WDYT?
J.
On Wed, Jan 5, 2022 at 5:39 PM Elad Kalif <el...@apache.org> wrote:
>
> We need to be also mindful about the fact that not everyone is able to migrate to the new Airflow version as soon as it's being released.
> One of the concepts of providers is that it's not tightly bound to Airflow core - If we can give users more time before setting a minimum version of airflow 2.3 that would be ideal.
> It's frustrating when a small bug fix that you really need is beyond your reach due to some other issue that forced bumping version. It may be more hussle for us but if we can release provider bug fixes/features first and a few days later release the breaking changes in a separate release it may be valuable for some users.
>
> On Thu, Dec 30, 2021 at 8:11 PM Kaxil Naik <ka...@gmail.com> wrote:
>>
>> One other solution for just using "utils" from core is we can release "airflow.backports" or "airflow.futures" package same as Python Backports and Python Future where we can handle the compat shim without restricting Airflow versions.
>>
>> Regards,
>> Kaxil
>>
>> On Thu, Dec 30, 2021 at 2:56 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>>>
>>> > At this point, I think 2.3.0 will be around early Feb to get AIP-42 in. We can reassess how the progress is moving ahead with the AIP mid-Jan and based on that we can either release 2.3.0 end of Jan without AIP-42 changes and mark it for 2.4.0.
>>>
>>> Yeah Agree here. I do not think we need to make a decision now - we
>>> can do it later. But I'd love to hear various opinions - from our
>>> users, various stakeholders (Amazon/Google/Astronomer who run it as a
>>> service for one).
>>>
>>> Just to be clear - I am not fully convinced myself if this is the
>>> right time to do it already (but there are some good indicators that
>>> we should at least start discussing it).
>>>
>>> This is more opening the discussion and gathering feedback/feelings of
>>> people. I think such decisions should be discussed (and their
>>> consequences realized) well in-advance so that at the time of decision
>>> we have all arguments and opinions and know the consequences.
>>>
>>> And it is not "strictly" necessary. It's just (like with all similar
>>> cases) an increased burden for maintenance and extra/boilerplate code
>>> that might be removed. In those cases there is rarely a "0/1" decision
>>> that can be made "here we know for sure that we should increase
>>> min-version" - it's more of the collective thinking: "Are the
>>> burden/overhead heavy enough we don't want to carry it any more as a
>>> community".
>>>
>>> J.
Re: [DISCUSSION] Make one of the next provider waves (January or February) 2.3+ only for all providers
Posted by Elad Kalif <el...@apache.org>.
We need to be also mindful about the fact that not everyone is able to
migrate to the new Airflow version as soon as it's being released.
One of the concepts of providers is that it's not tightly bound to Airflow
core - If we can give users more time before setting a minimum version of
airflow 2.3 that would be ideal.
It's frustrating when a small bug fix that you really need is beyond your
reach due to some other issue that forced bumping version. It may be more
hussle for us but if we can release provider bug fixes/features first and a
few days later release the breaking changes in a separate release it may be
valuable for some users.
On Thu, Dec 30, 2021 at 8:11 PM Kaxil Naik <ka...@gmail.com> wrote:
> One other solution for just using "utils" from core is we can release
> "airflow.backports" or "airflow.futures" package same as Python Backports
> <https://pypi.org/project/backports/> and Python Future
> <https://pypi.org/project/future/> where we can handle the compat shim
> without restricting Airflow versions.
>
> Regards,
> Kaxil
>
> On Thu, Dec 30, 2021 at 2:56 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>
>> > At this point, I think 2.3.0 will be around early Feb to get AIP-42 in.
>> We can reassess how the progress is moving ahead with the AIP mid-Jan and
>> based on that we can either release 2.3.0 end of Jan without AIP-42 changes
>> and mark it for 2.4.0.
>>
>> Yeah Agree here. I do not think we need to make a decision now - we
>> can do it later. But I'd love to hear various opinions - from our
>> users, various stakeholders (Amazon/Google/Astronomer who run it as a
>> service for one).
>>
>> Just to be clear - I am not fully convinced myself if this is the
>> right time to do it already (but there are some good indicators that
>> we should at least start discussing it).
>>
>> This is more opening the discussion and gathering feedback/feelings of
>> people. I think such decisions should be discussed (and their
>> consequences realized) well in-advance so that at the time of decision
>> we have all arguments and opinions and know the consequences.
>>
>> And it is not "strictly" necessary. It's just (like with all similar
>> cases) an increased burden for maintenance and extra/boilerplate code
>> that might be removed. In those cases there is rarely a "0/1" decision
>> that can be made "here we know for sure that we should increase
>> min-version" - it's more of the collective thinking: "Are the
>> burden/overhead heavy enough we don't want to carry it any more as a
>> community".
>>
>> J.
>>
>
Re: [DISCUSSION] Make one of the next provider waves (January or February) 2.3+ only for all providers
Posted by Kaxil Naik <ka...@gmail.com>.
One other solution for just using "utils" from core is we can release
"airflow.backports" or "airflow.futures" package same as Python Backports
<https://pypi.org/project/backports/> and Python Future
<https://pypi.org/project/future/> where we can handle the compat shim
without restricting Airflow versions.
Regards,
Kaxil
On Thu, Dec 30, 2021 at 2:56 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> > At this point, I think 2.3.0 will be around early Feb to get AIP-42 in.
> We can reassess how the progress is moving ahead with the AIP mid-Jan and
> based on that we can either release 2.3.0 end of Jan without AIP-42 changes
> and mark it for 2.4.0.
>
> Yeah Agree here. I do not think we need to make a decision now - we
> can do it later. But I'd love to hear various opinions - from our
> users, various stakeholders (Amazon/Google/Astronomer who run it as a
> service for one).
>
> Just to be clear - I am not fully convinced myself if this is the
> right time to do it already (but there are some good indicators that
> we should at least start discussing it).
>
> This is more opening the discussion and gathering feedback/feelings of
> people. I think such decisions should be discussed (and their
> consequences realized) well in-advance so that at the time of decision
> we have all arguments and opinions and know the consequences.
>
> And it is not "strictly" necessary. It's just (like with all similar
> cases) an increased burden for maintenance and extra/boilerplate code
> that might be removed. In those cases there is rarely a "0/1" decision
> that can be made "here we know for sure that we should increase
> min-version" - it's more of the collective thinking: "Are the
> burden/overhead heavy enough we don't want to carry it any more as a
> community".
>
> J.
>
Re: [DISCUSSION] Make one of the next provider waves (January or February) 2.3+ only for all providers
Posted by Jarek Potiuk <ja...@potiuk.com>.
> At this point, I think 2.3.0 will be around early Feb to get AIP-42 in. We can reassess how the progress is moving ahead with the AIP mid-Jan and based on that we can either release 2.3.0 end of Jan without AIP-42 changes and mark it for 2.4.0.
Yeah Agree here. I do not think we need to make a decision now - we
can do it later. But I'd love to hear various opinions - from our
users, various stakeholders (Amazon/Google/Astronomer who run it as a
service for one).
Just to be clear - I am not fully convinced myself if this is the
right time to do it already (but there are some good indicators that
we should at least start discussing it).
This is more opening the discussion and gathering feedback/feelings of
people. I think such decisions should be discussed (and their
consequences realized) well in-advance so that at the time of decision
we have all arguments and opinions and know the consequences.
And it is not "strictly" necessary. It's just (like with all similar
cases) an increased burden for maintenance and extra/boilerplate code
that might be removed. In those cases there is rarely a "0/1" decision
that can be made "here we know for sure that we should increase
min-version" - it's more of the collective thinking: "Are the
burden/overhead heavy enough we don't want to carry it any more as a
community".
J.
Re: [DISCUSSION] Make one of the next provider waves (January or February) 2.3+ only for all providers
Posted by Jarek Potiuk <ja...@potiuk.com>.
> Regarding the KPO -- I think we should be able to release a version compatible with Airflow 2.2.x .
Is it 2.2.0+ or 2.1.0+ (this is the current min-version) ? Or maybe 2.2.4+ ?
J.
Re: [DISCUSSION] Make one of the next provider waves (January or February) 2.3+ only for all providers
Posted by Kaxil Naik <ka...@gmail.com>.
Regarding the KPO -- I think we should be able to release a version
compatible with Airflow 2.2.x .
At this point, I think 2.3.0 will be around early Feb to get AIP-42 in. We
can reassess how the progress is moving ahead with the AIP *mid-Jan* and
based on that we can either release 2.3.0 end of Jan without AIP-42 changes
and mark it for 2.4.0.
Regarding the Typing changes, all those can be optional so I don't think we
should make our provider 2.3+ only as yet.
Regards,
Kaxil
On Thu, Dec 30, 2021 at 9:58 AM Jarek Potiuk <ja...@potiuk.com> wrote:
> Hello Everyone,
>
> We are close to the 2.3.0 release and we are starting to accumulate
> changes that make the maintenance of Airflow 2.1 compatibility in
> Providers more and more difficult. We implemented a lot of changes in
> the Airflow core that we could take advantage of that we are not able
> to now (common utils and functions) and we have a number of
> deprecations we could remove if we get rid of 2.1 and 2.2
> compatibility.
>
> There is also a big change coming in Kubernetes Pod Operator (and
> cncf.kubernetes) which will vastly improve and simplify the way it is
> use and keeping 2.1 compatibility seems to be unnecessary and quite a
> heavy burden (more about it here:
> https://apache-airflow.slack.com/archives/CCPRP7943/p1640840944067500).
>
> Also, we have a number of typing improvements in our "Bring back MyPY"
> project https://github.com/apache/airflow/issues/19891 (which is
> progressing well) and a number of benefits from applying those changes
> across the board (one of the most important - Context being typing and
> IDE friendly). Those will only show its true potential in Airflow
> 2.3+. Currently we have some workarounds related to that across pretty
> much all providers (`if TYPE_CHECKING`) but they have some side
> effects, and we could get rid of them and simplify provider's code if
> we make our providers 2.3+ only.
>
> I have a feeling that - similarly to what we 've done for Airflow 2.1
> and @apply_defaults - it's about the time to make our Providers
> "Airflow 2.3+ only" in one of the upcoming waves of providers.
>
> This has also the nice benefit of incentivising our users to migrate
> to 2.3+ if they need some features or bug-fixes only available in a
> 2.3+ provider. And this is a good thing IMHO.
>
> What do you all think? Is it a good idea? Any strong opinions and
> reasons why we should not do it?
>
>
> J.
>