[DISCUSS] Deprecation policy for apache-airflow-providers-google package

[Eugen Kosteev <eu...@kosteev.com>]

Hi.

I would like to discuss/propose a deprecation policy for
apache-airflow-providers-google package, mostly for deprecating Airflow
operators (but not limited to it).

*Some background:*
Airflow google provider package (as most of other provider packages in
Airflow) has a tendency to constantly evolve.
There are cases when its features or operators/hooks have to be deprecated.
The typical example of such a case is “refactoring” which leads to the
renaming of operators, parameters or public methods. Such refactorings
maybe consequence of:
- using a new dependent library (that causes some methods to be deprecated)
- using a new version of API (that causes some parameters to be deprecated)
- restructuring/migration of the operators (that causes some operators to
be deprecated)
Given the need of a “deprecation” procedure, it looks essential to
establish policy around it to have users of Airflow google provider package
clearly understand when and how to adapt their DAGs.

*Preambula (versioning of the package):*
As mentioned in “Airflow’s release process and version policy” (
https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#airflow-s-release-process-and-version-policy)
google provider package (and others) should follow SemVer, meaning that any
breaking changes should be released together with bumping major version of
the package.
The change is considered to be a breaking (in the context of google
providers package) if a DAG that was working before, stops to work after
the change.

*My proposal (for discussion):*
The entire procedure of deprecating (either method, parameter or operator)
consists of two steps:
- Emission of the deprecation warning message
Example of the message:
“””
The “given” parameter is deprecated and will be removed after dd.mm.yyyy.
Please use “this” parameter instead.
“””
- Once the date of the deprecated method/parameter/operator is passed, it
can be removed (together with bumping major version of the package)

The format of the warning message may vary, but it has to contain:
- What should be used instead
- The date after which the method/parameter/operator will be removed

*Additional considerations:*
- Bumping a major version of the Airflow google provider package shouldn’t
happen very frequently (presumably not more frequently than 6 months),
therefore when it happens it may accumulate removals of something that was
communicated to be removed even a couple of months ago. (related discussion
https://github.com/apache/airflow/blob/main/PROVIDERS.rst)
Example: if today is 2025-12-20 and we are releasing new major version of
the package, we can/should remove all deprecated
methods/parameters/operators which have date prior to 2025-12-20 - it can
be November 2025, October 2025 or even earlier, since previous bump of
major version happened e.g. in summer of 2025.
- By default all deprecations should allow a 6 months time period until
they will be removed and not available. This period will give Airflow users
enough time and flexibility to update their DAGs before actual removal
happens.
On a case by case basis this period can be adjusted given specific
circumstances (e.g. in case deprecation is because of underlying API sunset
which can happen earlier than in 6 months).

Apart from deprecation warning messages that will be printed as logs, this
information will be put as well in:
- Airflow documentation
- Release notes of Airflow google provider package

Please, let me know what you think about this.

-- 
Eugene

Re: [DISCUSS] Deprecation policy for apache-airflow-providers-google package

[Jarek Potiuk <ja...@potiuk.com>]

Just one comment. It might be nit-picking and a question of wording. I
don't think we can say that  `we guarantee` something. The whole ASF
licence is very explicit that we don't guarantee basically anything
and users are solely responsible for assessing their risks and we take
no responsibility whatsoever for anything. See section 7. of
https://www.apache.org/licenses/LICENSE-2.0

We can at most (as a PMC) express our intentions to do something.
There are legitimate cases where our intentions cannot be kept and we
will have to make tough decisions, bite the bullet and release
breaking changes. I don't think we can make `absolute` promises. But
we can do some safeguarding here, for example we could add a need to
cast a vote or run `lazy consensus`. That would be pretty acceptable
and the safeguard will be "serious" enough to make sure that yes,
indeed we are going to keep to our intentions.

But again - this is nit-picking for the wording possibly.

Just ane side note - in case the reason about breaking compatibility
would be about critical security issue causing it, the vote will have
to be kept on private@a.o with PMC's only, but this is also quite
acceptable and security is one of the very few exceptions were votes
(including voting on release) can be held in private@a.a.o and results
announced after the release.


J.

On Fri, Mar 8, 2024 at 11:58 PM Michał Modras
<mi...@google.com.invalid> wrote:
>
> To Jarek's phrasing - I think it is mostly right. The way I think about it:
> when deprecating we guarantee that for 6 months we will not do breaking
> changes, but after that period, on an occasion of releasing a major
> provider package version (which is probably breaking anyway) we can do
> cleanup and remove the old deprecated code.
>
> Elad - we are planning to do the cleanups, hence we wanted to put the
> policy in place for the Google provider, so we constraint ourselves and our
> users know what to expect, as opposed to doing things ad hoc and at will.
> In this sense the policy is redundant - we could not surface it, and
> internally follow these rules to the same effect, but we prefer to be
> transparent and inform the users of how the deprecation process will look.
> The goal of the policy is to inform as opposed to enforce anything stricter
> that we would like to do anyway.
>
> On Wed, Mar 6, 2024 at 5:14 AM Jarek Potiuk <ja...@potiuk.com> wrote:
>
> > Hey Elad. I think (and let me repeat that) - nobody suggests that we
> > ALWAYS remove some deprecations after 6 months. The idea  - as I see
> > it - is completely reversed. As I understand current idea is that we
> > say "By default we do not introduce any breaking changes (i.e. we do
> > not remove deprecations) for 6 months from the moment we introduced
> > last breaking changes. This means that we will have have (again by
> > default) at most two breaking changes in each provider per year). Of
> > course this is just "default" and we can do deprecations more - or
> > less - often if we see we want to make exception, but we need to have
> > a very good reason for it. But I do not think anyone is putting an
> > expiry date and will "force" removal of deprecations at specific date.
> > Introducing breaking change in provider means that there are SOME
> > deprecations removed - maybe all, maybe just a few. I think the clue
> > of this change is to introduce more-or-less stable cadence "generally
> > speaking we make a breaking change in provider not more often than
> > every 6 months".
> >
> > It's very similar to our rule that we relase providers more-or-less
> > every 2 weeks. Sometimes we do it less often, sometimes we don't
> > release provider even if we could (for example we did that with FAB)
> > and sometimes we do ad-hoc release because we see a need for it.
> >
> > Is this correct rephrasing of the proposal - Eugen and Michał? Do I
> > understand it correctly ?
> >
> > J.
> >
> > On Tue, Mar 5, 2024 at 2:23 PM Elad Kalif <el...@apache.org> wrote:
> > >
> > > Eugen, about your two items:
> > > - What should be used instead
> > > - The date after which the method/parameter/operator will be removed
> > >
> > >
> > > The first is already included today in all deprecation warnings.
> > > The second is a big -1 from me.  We normally say that it will be removed
> > in
> > > the next major release. Sometimes we enforce it and sometimes we don't.
> > > That is our choice.
> > > I said it before and I will say it again. Time based deprecation is not
> > the
> > > way to do it. Sometimes the author of a PR and the reviewers are not
> > fully
> > > aware of the blast radius of a specific deprecation.
> > > In Airflow any committer can merge any PR - I do not think it's right to
> > > put on the shoulders of the committer the decision of when something is
> > to
> > > be removed before we even accept the deprecation.
> > >
> > > Michał, The policy we have now is a shared governance model which means
> > > that Google can decide and act on deprecating code and also on when to
> > > remove code.
> > > We as maintainers, and specifically I as release manager for provider
> > will
> > > try to accomodate Google decision on this as Google knows best what is
> > > right for their customers.
> > > In the meantime, Google provider has many deprecation warnings some of
> > them
> > > are very old - I didn't see Google start a cleanup deprecation process.
> > > If you think this is the right time then feel free to raise PRs for this.
> > > This is why I say that policy may be redundant - you can do it at will.
> > >
> > > On Tue, Mar 5, 2024 at 12:49 PM Eugen Kosteev <eu...@kosteev.com> wrote:
> > >
> > > > Hi.
> > > >
> > > > Thanks for feedback and discussion.
> > > > Let me summarise what actually I want to propose (because there were a
> > lot
> > > > in my initial email).
> > > >
> > > > My proposal is, in case of deprecation, to emit messages in the
> > specific
> > > > format, example of the message:
> > > > “””
> > > > The “given” parameter is deprecated and will be removed after
> > dd.mm.yyyy.
> > > > Please use “this” parameter instead.
> > > > “””
> > > > The format of the warning message may vary, but it has to contain:
> > > > - What should be used instead
> > > > - The date after which the method/parameter/operator will be removed
> > > >
> > > > Everything else, such as decommission/removal only once bumping major
> > > > version, cadence of releases, etc. is not something that I propose to
> > > > change.
> > > >
> > > > The idea in general is to make Airflow users a convenient way to adjust
> > > > their DAGs to deprecations, thus:
> > > > - what to use instead
> > > > - when they need to update
> > > > should help them.
> > > >
> > > > If we have a date in the deprecation message, it means that the removal
> > > > (actual decommission) will happen in one of the major releases after
> > this
> > > > date.
> > > >
> > > > What do you think about this?
> > > >
> > > >
> > > > On Mon, Mar 4, 2024 at 7:03 PM Michał Modras
> > > > <mi...@google.com.invalid> wrote:
> > > >
> > > > > First, I'd like to say that I support Eugen's proposal and I agree
> > with
> > > > the
> > > > > enhancements suggested by Jarek.
> > > > >
> > > > > I'm a bit confused about Elad's point 3 - are you suggesting having a
> > > > > global policy for all providers, or that we should not codify our
> > > > approach
> > > > > at all since different providers have different requirements,
> > instead of
> > > > > having a policy per provider?
> > > > >
> > > > > I would be happy for this proposal (with some enhancements as
> > needed) to
> > > > be
> > > > > adopted by all providers, or we can keep it locally to Google
> > provider.
> > > > >
> > > > > On Thu, Feb 29, 2024 at 12:47 PM Jarek Potiuk <ja...@potiuk.com>
> > wrote:
> > > > >
> > > > > > One more thing. This one is essentially impossible (Hyrum's law
> > > > explains
> > > > > > that very well https://www.hyrumslaw.com/:
> > > > > >
> > > > > > > The change is considered to be a breaking (in the context of
> > google
> > > > > > providers package) if a DAG that was working before, stops to work
> > > > after
> > > > > > the change.
> > > > > >
> > > > > > I propose to change it to:
> > > > > >
> > > > > > > The change is considered to be a breaking if a DAG using the
> > > > providers
> > > > > > (in the
> > > > > > way they were intended to be used), that was working before, stops
> > to
> > > > > work
> > > > > > after
> > > > > > the change.
> > > > > >
> > > > > > It is absolutely not possible to make sure that every single DAG
> > > > written
> > > > > by
> > > > > > someone will continue working after any change. Literally.
> > > > > >
> > > > > > For example if someone extends BigQueryOperator and calls
> > > > > > (self._this_very_much_internal_method()` in his operator's new
> > > > execute(),
> > > > > > and we rename the method, their DAG will stop working. Is it
> > breaking ?
> > > > > No.
> > > > > > Is the user using it with the intention how it should be used?
> > > > Absolutely
> > > > > > not.
> > > > > >
> > > > > > SemVer - is not a promise things won't break, it's a promise that
> > our
> > > > > > intention is that things won't break. But whether the way our
> > users use
> > > > > it
> > > > > > will make it break or not, that's another story.
> > > > > >
> > > > > > J
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Thu, Feb 29, 2024 at 12:38 PM Jarek Potiuk <ja...@potiuk.com>
> > > > wrote:
> > > > > >
> > > > > > > BTW. Another example of an exception would be a security fix. We
> > > > > **migh**
> > > > > > > find a security issue that will require a breaking change. But
> > then
> > > > > > again -
> > > > > > > the breaking change might **only** cover that security fix - we
> > do
> > > > not
> > > > > > have
> > > > > > > to remove all the deprecations in this case (but we could if we
> > > > decide
> > > > > > so -
> > > > > > > very much depending on nature of those deprecations)
> > > > > > >
> > > > > > > On Thu, Feb 29, 2024 at 12:36 PM Jarek Potiuk <ja...@potiuk.com>
> > > > > wrote:
> > > > > > >
> > > > > > >> I think it's a good idea to codify the rules, indeed what Elad
> > > > > > mentioned,
> > > > > > >> we **mostly** follow a very similar approach, but  it's not
> > written
> > > > > > >> anywhere.
> > > > > > >>
> > > > > > >>
> > > > > > >> Re: points of Elad:
> > > > > > >>
> > > > > > >> 1. I am ok with 6 months, but I agree here we should not have
> > > > "strict"
> > > > > > >> limits. It's fine to have some default (i.e. accumulate
> > deprecating
> > > > > > changes
> > > > > > >> for some time - max. 6 months) - but I think the policy should
> > have
> > > > an
> > > > > > >> option to apply exceptions - for example when we have a major
> > change
> > > > > in
> > > > > > >> many libraries and we need to make some breaking changes (ads
> > case)
> > > > we
> > > > > > >> shall do it (but in this case we don't have to immediately
> > remove
> > > > all
> > > > > > >> deprecations - they can wait for the regular (say 6 months)
> > breaking
> > > > > > change
> > > > > > >> that will be coming. The users can - if needed - even in new
> > > > versions
> > > > > of
> > > > > > >> Airflow - downgrade to previous provider versions in case they
> > have
> > > > > > problem
> > > > > > >> with it.
> > > > > > >>
> > > > > > >> I like the idea of having some time (6 months) where we can very
> > > > > safely
> > > > > > >> say "OK, enough is enough, lets remove deprecations"  - and this
> > > > > > proposal
> > > > > > >> addresses it nicely. Currently it's purely subjective judgment.
> > One
> > > > > > small
> > > > > > >> thing here to clarify for "accumulating" deprecations - I think
> > we
> > > > > > should
> > > > > > >> make sure the deprecation is present in already released
> > > > > > minor/patchlevel
> > > > > > >> version to be able to remove it (so even if it is deprecated 2
> > > > months
> > > > > > ago
> > > > > > >> but we had release with the deprecation, it's ok to remove it
> > in the
> > > > > > next
> > > > > > >> breaking change.
> > > > > > >>
> > > > > > >> 2. Agree (and again here "strict" is not good I think). I'd
> > treat
> > > > > that 6
> > > > > > >> months as a baseline. Before that - we should have very good
> > reason
> > > > to
> > > > > > make
> > > > > > >> a breaking change (and we might decide to do it partially only)
> > -
> > > > > after
> > > > > > -
> > > > > > >> we are free to remove things that have been already announced as
> > > > > > deprecated
> > > > > > >> and we should consider making breaking change whenever we
> > release a
> > > > > new
> > > > > > >> version for whatever reason (but again - not mandatory).  I's
> > just
> > > > an
> > > > > > >> indication that tells us "Last breaking release was 6 months
> > ago,
> > > > it's
> > > > > > now
> > > > > > >> OK to remove all the deprecations without having an extremely
> > good
> > > > > > reason".
> > > > > > >>
> > > > > > >> Very much agree with 3. -> this should be the same as for other
> > > > > > >> providers. We should adopt the same rule for all of them. And
> > if we
> > > > > > apply
> > > > > > >> the adjustments above - we should - I think - be ok to apply it
> > to
> > > > any
> > > > > > >> provider I think.
> > > > > > >>
> > > > > > >> J
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >> On Thu, Feb 29, 2024 at 12:20 PM Elad Kalif <eladkal@apache.org
> > >
> > > > > wrote:
> > > > > > >>
> > > > > > >>> This is not too much different than what we already do.
> > > > > > >>> We deprecate then we remove. We almost never create breaking
> > change
> > > > > > >>> without
> > > > > > >>> deprecating first.
> > > > > > >>>
> > > > > > >>> I'd like to raise a few notes:
> > > > > > >>>
> > > > > > >>> 1. Major releases can happen frequently. I see no problem with
> > it.
> > > > > This
> > > > > > >>> is
> > > > > > >>> more a question of what are the changes. If google has 3 major
> > > > > releases
> > > > > > >>> one
> > > > > > >>> with breaking change to ads, another to leveldb and 3rd to
> > cloud
> > > > > these
> > > > > > >>> are
> > > > > > >>> not related one to another and unlikely that there is 1 user
> > that
> > > > > uses
> > > > > > >>> all
> > > > > > >>> of these together. We create major releases when needed and we
> > > > keep a
> > > > > > >>> close
> > > > > > >>> eye and challenge why a change must be a breaking change. If
> > > > possible
> > > > > > we
> > > > > > >>> will always prefer to be backward compatible.
> > > > > > >>> 2. Time based deprecations are not a good approach. Some
> > changes
> > > > > > require
> > > > > > >>> much more time to be acted upon and others may be trivial.
> > > > > > >>> As a rule we almost never cut a breaking change release just to
> > > > > remove
> > > > > > >>> deprecations.
> > > > > > >>> 3. Setting a policy per provider is not the right way to go. We
> > > > have
> > > > > a
> > > > > > >>> shared governance model for provider handling and I as release
> > > > > manager
> > > > > > >>> will
> > > > > > >>> do what I can to accommodate requests from companies who
> > > > participate
> > > > > in
> > > > > > >>> managing the provider. Which means that if Google wants to
> > remove
> > > > > > >>> deprecations and ask for a breaking change release most likely
> > we
> > > > > will
> > > > > > >>> accommodate their request. My point of view is that the company
> > > > knows
> > > > > > >>> what
> > > > > > >>> is best for their own users. I don't think that what is right
> > for
> > > > > > Google
> > > > > > >>> must bind AWS or Microsoft. 1 policy goes against the shared
> > > > > > >>> governance idea.
> > > > > > >>>
> > > > > > >>>
> > > > > > >>>
> > > > > > >>> On Thu, Feb 29, 2024 at 11:04 AM Eugen Kosteev <
> > eugen@kosteev.com>
> > > > > > >>> wrote:
> > > > > > >>>
> > > > > > >>> > Hi.
> > > > > > >>> >
> > > > > > >>> > I would like to discuss/propose a deprecation policy for
> > > > > > >>> > apache-airflow-providers-google package, mostly for
> > deprecating
> > > > > > Airflow
> > > > > > >>> > operators (but not limited to it).
> > > > > > >>> >
> > > > > > >>> > *Some background:*
> > > > > > >>> > Airflow google provider package (as most of other provider
> > > > packages
> > > > > > in
> > > > > > >>> > Airflow) has a tendency to constantly evolve.
> > > > > > >>> > There are cases when its features or operators/hooks have to
> > be
> > > > > > >>> deprecated.
> > > > > > >>> > The typical example of such a case is “refactoring” which
> > leads
> > > > to
> > > > > > the
> > > > > > >>> > renaming of operators, parameters or public methods. Such
> > > > > > refactorings
> > > > > > >>> > maybe consequence of:
> > > > > > >>> > - using a new dependent library (that causes some methods to
> > be
> > > > > > >>> deprecated)
> > > > > > >>> > - using a new version of API (that causes some parameters to
> > be
> > > > > > >>> deprecated)
> > > > > > >>> > - restructuring/migration of the operators (that causes some
> > > > > > operators
> > > > > > >>> to
> > > > > > >>> > be deprecated)
> > > > > > >>> > Given the need of a “deprecation” procedure, it looks
> > essential
> > > > to
> > > > > > >>> > establish policy around it to have users of Airflow google
> > > > provider
> > > > > > >>> package
> > > > > > >>> > clearly understand when and how to adapt their DAGs.
> > > > > > >>> >
> > > > > > >>> > *Preambula (versioning of the package):*
> > > > > > >>> > As mentioned in “Airflow’s release process and version
> > policy” (
> > > > > > >>> >
> > > > > > >>> >
> > > > > > >>>
> > > > > >
> > > > >
> > > >
> > https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#airflow-s-release-process-and-version-policy
> > > > > > >>> > )
> > > > > > >>> > google provider package (and others) should follow SemVer,
> > > > meaning
> > > > > > >>> that any
> > > > > > >>> > breaking changes should be released together with bumping
> > major
> > > > > > >>> version of
> > > > > > >>> > the package.
> > > > > > >>> > The change is considered to be a breaking (in the context of
> > > > google
> > > > > > >>> > providers package) if a DAG that was working before, stops to
> > > > work
> > > > > > >>> after
> > > > > > >>> > the change.
> > > > > > >>> >
> > > > > > >>> > *My proposal (for discussion):*
> > > > > > >>> > The entire procedure of deprecating (either method,
> > parameter or
> > > > > > >>> operator)
> > > > > > >>> > consists of two steps:
> > > > > > >>> > - Emission of the deprecation warning message
> > > > > > >>> > Example of the message:
> > > > > > >>> > “””
> > > > > > >>> > The “given” parameter is deprecated and will be removed after
> > > > > > >>> dd.mm.yyyy.
> > > > > > >>> > Please use “this” parameter instead.
> > > > > > >>> > “””
> > > > > > >>> > - Once the date of the deprecated method/parameter/operator
> > is
> > > > > > passed,
> > > > > > >>> it
> > > > > > >>> > can be removed (together with bumping major version of the
> > > > package)
> > > > > > >>> >
> > > > > > >>> > The format of the warning message may vary, but it has to
> > > > contain:
> > > > > > >>> > - What should be used instead
> > > > > > >>> > - The date after which the method/parameter/operator will be
> > > > > removed
> > > > > > >>> >
> > > > > > >>> > *Additional considerations:*
> > > > > > >>> > - Bumping a major version of the Airflow google provider
> > package
> > > > > > >>> shouldn’t
> > > > > > >>> > happen very frequently (presumably not more frequently than 6
> > > > > > months),
> > > > > > >>> > therefore when it happens it may accumulate removals of
> > something
> > > > > > that
> > > > > > >>> was
> > > > > > >>> > communicated to be removed even a couple of months ago.
> > (related
> > > > > > >>> discussion
> > > > > > >>> > https://github.com/apache/airflow/blob/main/PROVIDERS.rst)
> > > > > > >>> > Example: if today is 2025-12-20 and we are releasing new
> > major
> > > > > > version
> > > > > > >>> of
> > > > > > >>> > the package, we can/should remove all deprecated
> > > > > > >>> > methods/parameters/operators which have date prior to
> > 2025-12-20
> > > > -
> > > > > it
> > > > > > >>> can
> > > > > > >>> > be November 2025, October 2025 or even earlier, since
> > previous
> > > > bump
> > > > > > of
> > > > > > >>> > major version happened e.g. in summer of 2025.
> > > > > > >>> > - By default all deprecations should allow a 6 months time
> > period
> > > > > > until
> > > > > > >>> > they will be removed and not available. This period will give
> > > > > Airflow
> > > > > > >>> users
> > > > > > >>> > enough time and flexibility to update their DAGs before
> > actual
> > > > > > removal
> > > > > > >>> > happens.
> > > > > > >>> > On a case by case basis this period can be adjusted given
> > > > specific
> > > > > > >>> > circumstances (e.g. in case deprecation is because of
> > underlying
> > > > > API
> > > > > > >>> sunset
> > > > > > >>> > which can happen earlier than in 6 months).
> > > > > > >>> >
> > > > > > >>> > Apart from deprecation warning messages that will be printed
> > as
> > > > > logs,
> > > > > > >>> this
> > > > > > >>> > information will be put as well in:
> > > > > > >>> > - Airflow documentation
> > > > > > >>> > - Release notes of Airflow google provider package
> > > > > > >>> >
> > > > > > >>> > Please, let me know what you think about this.
> > > > > > >>> >
> > > > > > >>> > --
> > > > > > >>> > Eugene
> > > > > > >>> >
> > > > > > >>>
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > Eugene
> > > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > For additional commands, e-mail: dev-help@airflow.apache.org
> >
> >

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
For additional commands, e-mail: dev-help@airflow.apache.org

Re: [DISCUSS] Deprecation policy for apache-airflow-providers-google package

[Michał Modras <mi...@google.com.INVALID>]

To Jarek's phrasing - I think it is mostly right. The way I think about it:
when deprecating we guarantee that for 6 months we will not do breaking
changes, but after that period, on an occasion of releasing a major
provider package version (which is probably breaking anyway) we can do
cleanup and remove the old deprecated code.

Elad - we are planning to do the cleanups, hence we wanted to put the
policy in place for the Google provider, so we constraint ourselves and our
users know what to expect, as opposed to doing things ad hoc and at will.
In this sense the policy is redundant - we could not surface it, and
internally follow these rules to the same effect, but we prefer to be
transparent and inform the users of how the deprecation process will look.
The goal of the policy is to inform as opposed to enforce anything stricter
that we would like to do anyway.

On Wed, Mar 6, 2024 at 5:14 AM Jarek Potiuk <ja...@potiuk.com> wrote:

> Hey Elad. I think (and let me repeat that) - nobody suggests that we
> ALWAYS remove some deprecations after 6 months. The idea  - as I see
> it - is completely reversed. As I understand current idea is that we
> say "By default we do not introduce any breaking changes (i.e. we do
> not remove deprecations) for 6 months from the moment we introduced
> last breaking changes. This means that we will have have (again by
> default) at most two breaking changes in each provider per year). Of
> course this is just "default" and we can do deprecations more - or
> less - often if we see we want to make exception, but we need to have
> a very good reason for it. But I do not think anyone is putting an
> expiry date and will "force" removal of deprecations at specific date.
> Introducing breaking change in provider means that there are SOME
> deprecations removed - maybe all, maybe just a few. I think the clue
> of this change is to introduce more-or-less stable cadence "generally
> speaking we make a breaking change in provider not more often than
> every 6 months".
>
> It's very similar to our rule that we relase providers more-or-less
> every 2 weeks. Sometimes we do it less often, sometimes we don't
> release provider even if we could (for example we did that with FAB)
> and sometimes we do ad-hoc release because we see a need for it.
>
> Is this correct rephrasing of the proposal - Eugen and Michał? Do I
> understand it correctly ?
>
> J.
>
> On Tue, Mar 5, 2024 at 2:23 PM Elad Kalif <el...@apache.org> wrote:
> >
> > Eugen, about your two items:
> > - What should be used instead
> > - The date after which the method/parameter/operator will be removed
> >
> >
> > The first is already included today in all deprecation warnings.
> > The second is a big -1 from me.  We normally say that it will be removed
> in
> > the next major release. Sometimes we enforce it and sometimes we don't.
> > That is our choice.
> > I said it before and I will say it again. Time based deprecation is not
> the
> > way to do it. Sometimes the author of a PR and the reviewers are not
> fully
> > aware of the blast radius of a specific deprecation.
> > In Airflow any committer can merge any PR - I do not think it's right to
> > put on the shoulders of the committer the decision of when something is
> to
> > be removed before we even accept the deprecation.
> >
> > Michał, The policy we have now is a shared governance model which means
> > that Google can decide and act on deprecating code and also on when to
> > remove code.
> > We as maintainers, and specifically I as release manager for provider
> will
> > try to accomodate Google decision on this as Google knows best what is
> > right for their customers.
> > In the meantime, Google provider has many deprecation warnings some of
> them
> > are very old - I didn't see Google start a cleanup deprecation process.
> > If you think this is the right time then feel free to raise PRs for this.
> > This is why I say that policy may be redundant - you can do it at will.
> >
> > On Tue, Mar 5, 2024 at 12:49 PM Eugen Kosteev <eu...@kosteev.com> wrote:
> >
> > > Hi.
> > >
> > > Thanks for feedback and discussion.
> > > Let me summarise what actually I want to propose (because there were a
> lot
> > > in my initial email).
> > >
> > > My proposal is, in case of deprecation, to emit messages in the
> specific
> > > format, example of the message:
> > > “””
> > > The “given” parameter is deprecated and will be removed after
> dd.mm.yyyy.
> > > Please use “this” parameter instead.
> > > “””
> > > The format of the warning message may vary, but it has to contain:
> > > - What should be used instead
> > > - The date after which the method/parameter/operator will be removed
> > >
> > > Everything else, such as decommission/removal only once bumping major
> > > version, cadence of releases, etc. is not something that I propose to
> > > change.
> > >
> > > The idea in general is to make Airflow users a convenient way to adjust
> > > their DAGs to deprecations, thus:
> > > - what to use instead
> > > - when they need to update
> > > should help them.
> > >
> > > If we have a date in the deprecation message, it means that the removal
> > > (actual decommission) will happen in one of the major releases after
> this
> > > date.
> > >
> > > What do you think about this?
> > >
> > >
> > > On Mon, Mar 4, 2024 at 7:03 PM Michał Modras
> > > <mi...@google.com.invalid> wrote:
> > >
> > > > First, I'd like to say that I support Eugen's proposal and I agree
> with
> > > the
> > > > enhancements suggested by Jarek.
> > > >
> > > > I'm a bit confused about Elad's point 3 - are you suggesting having a
> > > > global policy for all providers, or that we should not codify our
> > > approach
> > > > at all since different providers have different requirements,
> instead of
> > > > having a policy per provider?
> > > >
> > > > I would be happy for this proposal (with some enhancements as
> needed) to
> > > be
> > > > adopted by all providers, or we can keep it locally to Google
> provider.
> > > >
> > > > On Thu, Feb 29, 2024 at 12:47 PM Jarek Potiuk <ja...@potiuk.com>
> wrote:
> > > >
> > > > > One more thing. This one is essentially impossible (Hyrum's law
> > > explains
> > > > > that very well https://www.hyrumslaw.com/:
> > > > >
> > > > > > The change is considered to be a breaking (in the context of
> google
> > > > > providers package) if a DAG that was working before, stops to work
> > > after
> > > > > the change.
> > > > >
> > > > > I propose to change it to:
> > > > >
> > > > > > The change is considered to be a breaking if a DAG using the
> > > providers
> > > > > (in the
> > > > > way they were intended to be used), that was working before, stops
> to
> > > > work
> > > > > after
> > > > > the change.
> > > > >
> > > > > It is absolutely not possible to make sure that every single DAG
> > > written
> > > > by
> > > > > someone will continue working after any change. Literally.
> > > > >
> > > > > For example if someone extends BigQueryOperator and calls
> > > > > (self._this_very_much_internal_method()` in his operator's new
> > > execute(),
> > > > > and we rename the method, their DAG will stop working. Is it
> breaking ?
> > > > No.
> > > > > Is the user using it with the intention how it should be used?
> > > Absolutely
> > > > > not.
> > > > >
> > > > > SemVer - is not a promise things won't break, it's a promise that
> our
> > > > > intention is that things won't break. But whether the way our
> users use
> > > > it
> > > > > will make it break or not, that's another story.
> > > > >
> > > > > J
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Thu, Feb 29, 2024 at 12:38 PM Jarek Potiuk <ja...@potiuk.com>
> > > wrote:
> > > > >
> > > > > > BTW. Another example of an exception would be a security fix. We
> > > > **migh**
> > > > > > find a security issue that will require a breaking change. But
> then
> > > > > again -
> > > > > > the breaking change might **only** cover that security fix - we
> do
> > > not
> > > > > have
> > > > > > to remove all the deprecations in this case (but we could if we
> > > decide
> > > > > so -
> > > > > > very much depending on nature of those deprecations)
> > > > > >
> > > > > > On Thu, Feb 29, 2024 at 12:36 PM Jarek Potiuk <ja...@potiuk.com>
> > > > wrote:
> > > > > >
> > > > > >> I think it's a good idea to codify the rules, indeed what Elad
> > > > > mentioned,
> > > > > >> we **mostly** follow a very similar approach, but  it's not
> written
> > > > > >> anywhere.
> > > > > >>
> > > > > >>
> > > > > >> Re: points of Elad:
> > > > > >>
> > > > > >> 1. I am ok with 6 months, but I agree here we should not have
> > > "strict"
> > > > > >> limits. It's fine to have some default (i.e. accumulate
> deprecating
> > > > > changes
> > > > > >> for some time - max. 6 months) - but I think the policy should
> have
> > > an
> > > > > >> option to apply exceptions - for example when we have a major
> change
> > > > in
> > > > > >> many libraries and we need to make some breaking changes (ads
> case)
> > > we
> > > > > >> shall do it (but in this case we don't have to immediately
> remove
> > > all
> > > > > >> deprecations - they can wait for the regular (say 6 months)
> breaking
> > > > > change
> > > > > >> that will be coming. The users can - if needed - even in new
> > > versions
> > > > of
> > > > > >> Airflow - downgrade to previous provider versions in case they
> have
> > > > > problem
> > > > > >> with it.
> > > > > >>
> > > > > >> I like the idea of having some time (6 months) where we can very
> > > > safely
> > > > > >> say "OK, enough is enough, lets remove deprecations"  - and this
> > > > > proposal
> > > > > >> addresses it nicely. Currently it's purely subjective judgment.
> One
> > > > > small
> > > > > >> thing here to clarify for "accumulating" deprecations - I think
> we
> > > > > should
> > > > > >> make sure the deprecation is present in already released
> > > > > minor/patchlevel
> > > > > >> version to be able to remove it (so even if it is deprecated 2
> > > months
> > > > > ago
> > > > > >> but we had release with the deprecation, it's ok to remove it
> in the
> > > > > next
> > > > > >> breaking change.
> > > > > >>
> > > > > >> 2. Agree (and again here "strict" is not good I think). I'd
> treat
> > > > that 6
> > > > > >> months as a baseline. Before that - we should have very good
> reason
> > > to
> > > > > make
> > > > > >> a breaking change (and we might decide to do it partially only)
> -
> > > > after
> > > > > -
> > > > > >> we are free to remove things that have been already announced as
> > > > > deprecated
> > > > > >> and we should consider making breaking change whenever we
> release a
> > > > new
> > > > > >> version for whatever reason (but again - not mandatory).  I's
> just
> > > an
> > > > > >> indication that tells us "Last breaking release was 6 months
> ago,
> > > it's
> > > > > now
> > > > > >> OK to remove all the deprecations without having an extremely
> good
> > > > > reason".
> > > > > >>
> > > > > >> Very much agree with 3. -> this should be the same as for other
> > > > > >> providers. We should adopt the same rule for all of them. And
> if we
> > > > > apply
> > > > > >> the adjustments above - we should - I think - be ok to apply it
> to
> > > any
> > > > > >> provider I think.
> > > > > >>
> > > > > >> J
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> On Thu, Feb 29, 2024 at 12:20 PM Elad Kalif <eladkal@apache.org
> >
> > > > wrote:
> > > > > >>
> > > > > >>> This is not too much different than what we already do.
> > > > > >>> We deprecate then we remove. We almost never create breaking
> change
> > > > > >>> without
> > > > > >>> deprecating first.
> > > > > >>>
> > > > > >>> I'd like to raise a few notes:
> > > > > >>>
> > > > > >>> 1. Major releases can happen frequently. I see no problem with
> it.
> > > > This
> > > > > >>> is
> > > > > >>> more a question of what are the changes. If google has 3 major
> > > > releases
> > > > > >>> one
> > > > > >>> with breaking change to ads, another to leveldb and 3rd to
> cloud
> > > > these
> > > > > >>> are
> > > > > >>> not related one to another and unlikely that there is 1 user
> that
> > > > uses
> > > > > >>> all
> > > > > >>> of these together. We create major releases when needed and we
> > > keep a
> > > > > >>> close
> > > > > >>> eye and challenge why a change must be a breaking change. If
> > > possible
> > > > > we
> > > > > >>> will always prefer to be backward compatible.
> > > > > >>> 2. Time based deprecations are not a good approach. Some
> changes
> > > > > require
> > > > > >>> much more time to be acted upon and others may be trivial.
> > > > > >>> As a rule we almost never cut a breaking change release just to
> > > > remove
> > > > > >>> deprecations.
> > > > > >>> 3. Setting a policy per provider is not the right way to go. We
> > > have
> > > > a
> > > > > >>> shared governance model for provider handling and I as release
> > > > manager
> > > > > >>> will
> > > > > >>> do what I can to accommodate requests from companies who
> > > participate
> > > > in
> > > > > >>> managing the provider. Which means that if Google wants to
> remove
> > > > > >>> deprecations and ask for a breaking change release most likely
> we
> > > > will
> > > > > >>> accommodate their request. My point of view is that the company
> > > knows
> > > > > >>> what
> > > > > >>> is best for their own users. I don't think that what is right
> for
> > > > > Google
> > > > > >>> must bind AWS or Microsoft. 1 policy goes against the shared
> > > > > >>> governance idea.
> > > > > >>>
> > > > > >>>
> > > > > >>>
> > > > > >>> On Thu, Feb 29, 2024 at 11:04 AM Eugen Kosteev <
> eugen@kosteev.com>
> > > > > >>> wrote:
> > > > > >>>
> > > > > >>> > Hi.
> > > > > >>> >
> > > > > >>> > I would like to discuss/propose a deprecation policy for
> > > > > >>> > apache-airflow-providers-google package, mostly for
> deprecating
> > > > > Airflow
> > > > > >>> > operators (but not limited to it).
> > > > > >>> >
> > > > > >>> > *Some background:*
> > > > > >>> > Airflow google provider package (as most of other provider
> > > packages
> > > > > in
> > > > > >>> > Airflow) has a tendency to constantly evolve.
> > > > > >>> > There are cases when its features or operators/hooks have to
> be
> > > > > >>> deprecated.
> > > > > >>> > The typical example of such a case is “refactoring” which
> leads
> > > to
> > > > > the
> > > > > >>> > renaming of operators, parameters or public methods. Such
> > > > > refactorings
> > > > > >>> > maybe consequence of:
> > > > > >>> > - using a new dependent library (that causes some methods to
> be
> > > > > >>> deprecated)
> > > > > >>> > - using a new version of API (that causes some parameters to
> be
> > > > > >>> deprecated)
> > > > > >>> > - restructuring/migration of the operators (that causes some
> > > > > operators
> > > > > >>> to
> > > > > >>> > be deprecated)
> > > > > >>> > Given the need of a “deprecation” procedure, it looks
> essential
> > > to
> > > > > >>> > establish policy around it to have users of Airflow google
> > > provider
> > > > > >>> package
> > > > > >>> > clearly understand when and how to adapt their DAGs.
> > > > > >>> >
> > > > > >>> > *Preambula (versioning of the package):*
> > > > > >>> > As mentioned in “Airflow’s release process and version
> policy” (
> > > > > >>> >
> > > > > >>> >
> > > > > >>>
> > > > >
> > > >
> > >
> https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#airflow-s-release-process-and-version-policy
> > > > > >>> > )
> > > > > >>> > google provider package (and others) should follow SemVer,
> > > meaning
> > > > > >>> that any
> > > > > >>> > breaking changes should be released together with bumping
> major
> > > > > >>> version of
> > > > > >>> > the package.
> > > > > >>> > The change is considered to be a breaking (in the context of
> > > google
> > > > > >>> > providers package) if a DAG that was working before, stops to
> > > work
> > > > > >>> after
> > > > > >>> > the change.
> > > > > >>> >
> > > > > >>> > *My proposal (for discussion):*
> > > > > >>> > The entire procedure of deprecating (either method,
> parameter or
> > > > > >>> operator)
> > > > > >>> > consists of two steps:
> > > > > >>> > - Emission of the deprecation warning message
> > > > > >>> > Example of the message:
> > > > > >>> > “””
> > > > > >>> > The “given” parameter is deprecated and will be removed after
> > > > > >>> dd.mm.yyyy.
> > > > > >>> > Please use “this” parameter instead.
> > > > > >>> > “””
> > > > > >>> > - Once the date of the deprecated method/parameter/operator
> is
> > > > > passed,
> > > > > >>> it
> > > > > >>> > can be removed (together with bumping major version of the
> > > package)
> > > > > >>> >
> > > > > >>> > The format of the warning message may vary, but it has to
> > > contain:
> > > > > >>> > - What should be used instead
> > > > > >>> > - The date after which the method/parameter/operator will be
> > > > removed
> > > > > >>> >
> > > > > >>> > *Additional considerations:*
> > > > > >>> > - Bumping a major version of the Airflow google provider
> package
> > > > > >>> shouldn’t
> > > > > >>> > happen very frequently (presumably not more frequently than 6
> > > > > months),
> > > > > >>> > therefore when it happens it may accumulate removals of
> something
> > > > > that
> > > > > >>> was
> > > > > >>> > communicated to be removed even a couple of months ago.
> (related
> > > > > >>> discussion
> > > > > >>> > https://github.com/apache/airflow/blob/main/PROVIDERS.rst)
> > > > > >>> > Example: if today is 2025-12-20 and we are releasing new
> major
> > > > > version
> > > > > >>> of
> > > > > >>> > the package, we can/should remove all deprecated
> > > > > >>> > methods/parameters/operators which have date prior to
> 2025-12-20
> > > -
> > > > it
> > > > > >>> can
> > > > > >>> > be November 2025, October 2025 or even earlier, since
> previous
> > > bump
> > > > > of
> > > > > >>> > major version happened e.g. in summer of 2025.
> > > > > >>> > - By default all deprecations should allow a 6 months time
> period
> > > > > until
> > > > > >>> > they will be removed and not available. This period will give
> > > > Airflow
> > > > > >>> users
> > > > > >>> > enough time and flexibility to update their DAGs before
> actual
> > > > > removal
> > > > > >>> > happens.
> > > > > >>> > On a case by case basis this period can be adjusted given
> > > specific
> > > > > >>> > circumstances (e.g. in case deprecation is because of
> underlying
> > > > API
> > > > > >>> sunset
> > > > > >>> > which can happen earlier than in 6 months).
> > > > > >>> >
> > > > > >>> > Apart from deprecation warning messages that will be printed
> as
> > > > logs,
> > > > > >>> this
> > > > > >>> > information will be put as well in:
> > > > > >>> > - Airflow documentation
> > > > > >>> > - Release notes of Airflow google provider package
> > > > > >>> >
> > > > > >>> > Please, let me know what you think about this.
> > > > > >>> >
> > > > > >>> > --
> > > > > >>> > Eugene
> > > > > >>> >
> > > > > >>>
> > > > > >>
> > > > >
> > > >
> > >
> > >
> > > --
> > > Eugene
> > >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> For additional commands, e-mail: dev-help@airflow.apache.org
>
>

Re: [DISCUSS] Deprecation policy for apache-airflow-providers-google package

[Jarek Potiuk <ja...@potiuk.com>]

Hey Elad. I think (and let me repeat that) - nobody suggests that we
ALWAYS remove some deprecations after 6 months. The idea  - as I see
it - is completely reversed. As I understand current idea is that we
say "By default we do not introduce any breaking changes (i.e. we do
not remove deprecations) for 6 months from the moment we introduced
last breaking changes. This means that we will have have (again by
default) at most two breaking changes in each provider per year). Of
course this is just "default" and we can do deprecations more - or
less - often if we see we want to make exception, but we need to have
a very good reason for it. But I do not think anyone is putting an
expiry date and will "force" removal of deprecations at specific date.
Introducing breaking change in provider means that there are SOME
deprecations removed - maybe all, maybe just a few. I think the clue
of this change is to introduce more-or-less stable cadence "generally
speaking we make a breaking change in provider not more often than
every 6 months".

It's very similar to our rule that we relase providers more-or-less
every 2 weeks. Sometimes we do it less often, sometimes we don't
release provider even if we could (for example we did that with FAB)
and sometimes we do ad-hoc release because we see a need for it.

Is this correct rephrasing of the proposal - Eugen and Michał? Do I
understand it correctly ?

J.

On Tue, Mar 5, 2024 at 2:23 PM Elad Kalif <el...@apache.org> wrote:
>
> Eugen, about your two items:
> - What should be used instead
> - The date after which the method/parameter/operator will be removed
>
>
> The first is already included today in all deprecation warnings.
> The second is a big -1 from me.  We normally say that it will be removed in
> the next major release. Sometimes we enforce it and sometimes we don't.
> That is our choice.
> I said it before and I will say it again. Time based deprecation is not the
> way to do it. Sometimes the author of a PR and the reviewers are not fully
> aware of the blast radius of a specific deprecation.
> In Airflow any committer can merge any PR - I do not think it's right to
> put on the shoulders of the committer the decision of when something is to
> be removed before we even accept the deprecation.
>
> Michał, The policy we have now is a shared governance model which means
> that Google can decide and act on deprecating code and also on when to
> remove code.
> We as maintainers, and specifically I as release manager for provider will
> try to accomodate Google decision on this as Google knows best what is
> right for their customers.
> In the meantime, Google provider has many deprecation warnings some of them
> are very old - I didn't see Google start a cleanup deprecation process.
> If you think this is the right time then feel free to raise PRs for this.
> This is why I say that policy may be redundant - you can do it at will.
>
> On Tue, Mar 5, 2024 at 12:49 PM Eugen Kosteev <eu...@kosteev.com> wrote:
>
> > Hi.
> >
> > Thanks for feedback and discussion.
> > Let me summarise what actually I want to propose (because there were a lot
> > in my initial email).
> >
> > My proposal is, in case of deprecation, to emit messages in the specific
> > format, example of the message:
> > “””
> > The “given” parameter is deprecated and will be removed after dd.mm.yyyy.
> > Please use “this” parameter instead.
> > “””
> > The format of the warning message may vary, but it has to contain:
> > - What should be used instead
> > - The date after which the method/parameter/operator will be removed
> >
> > Everything else, such as decommission/removal only once bumping major
> > version, cadence of releases, etc. is not something that I propose to
> > change.
> >
> > The idea in general is to make Airflow users a convenient way to adjust
> > their DAGs to deprecations, thus:
> > - what to use instead
> > - when they need to update
> > should help them.
> >
> > If we have a date in the deprecation message, it means that the removal
> > (actual decommission) will happen in one of the major releases after this
> > date.
> >
> > What do you think about this?
> >
> >
> > On Mon, Mar 4, 2024 at 7:03 PM Michał Modras
> > <mi...@google.com.invalid> wrote:
> >
> > > First, I'd like to say that I support Eugen's proposal and I agree with
> > the
> > > enhancements suggested by Jarek.
> > >
> > > I'm a bit confused about Elad's point 3 - are you suggesting having a
> > > global policy for all providers, or that we should not codify our
> > approach
> > > at all since different providers have different requirements, instead of
> > > having a policy per provider?
> > >
> > > I would be happy for this proposal (with some enhancements as needed) to
> > be
> > > adopted by all providers, or we can keep it locally to Google provider.
> > >
> > > On Thu, Feb 29, 2024 at 12:47 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> > >
> > > > One more thing. This one is essentially impossible (Hyrum's law
> > explains
> > > > that very well https://www.hyrumslaw.com/:
> > > >
> > > > > The change is considered to be a breaking (in the context of google
> > > > providers package) if a DAG that was working before, stops to work
> > after
> > > > the change.
> > > >
> > > > I propose to change it to:
> > > >
> > > > > The change is considered to be a breaking if a DAG using the
> > providers
> > > > (in the
> > > > way they were intended to be used), that was working before, stops to
> > > work
> > > > after
> > > > the change.
> > > >
> > > > It is absolutely not possible to make sure that every single DAG
> > written
> > > by
> > > > someone will continue working after any change. Literally.
> > > >
> > > > For example if someone extends BigQueryOperator and calls
> > > > (self._this_very_much_internal_method()` in his operator's new
> > execute(),
> > > > and we rename the method, their DAG will stop working. Is it breaking ?
> > > No.
> > > > Is the user using it with the intention how it should be used?
> > Absolutely
> > > > not.
> > > >
> > > > SemVer - is not a promise things won't break, it's a promise that our
> > > > intention is that things won't break. But whether the way our users use
> > > it
> > > > will make it break or not, that's another story.
> > > >
> > > > J
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On Thu, Feb 29, 2024 at 12:38 PM Jarek Potiuk <ja...@potiuk.com>
> > wrote:
> > > >
> > > > > BTW. Another example of an exception would be a security fix. We
> > > **migh**
> > > > > find a security issue that will require a breaking change. But then
> > > > again -
> > > > > the breaking change might **only** cover that security fix - we do
> > not
> > > > have
> > > > > to remove all the deprecations in this case (but we could if we
> > decide
> > > > so -
> > > > > very much depending on nature of those deprecations)
> > > > >
> > > > > On Thu, Feb 29, 2024 at 12:36 PM Jarek Potiuk <ja...@potiuk.com>
> > > wrote:
> > > > >
> > > > >> I think it's a good idea to codify the rules, indeed what Elad
> > > > mentioned,
> > > > >> we **mostly** follow a very similar approach, but  it's not written
> > > > >> anywhere.
> > > > >>
> > > > >>
> > > > >> Re: points of Elad:
> > > > >>
> > > > >> 1. I am ok with 6 months, but I agree here we should not have
> > "strict"
> > > > >> limits. It's fine to have some default (i.e. accumulate deprecating
> > > > changes
> > > > >> for some time - max. 6 months) - but I think the policy should have
> > an
> > > > >> option to apply exceptions - for example when we have a major change
> > > in
> > > > >> many libraries and we need to make some breaking changes (ads case)
> > we
> > > > >> shall do it (but in this case we don't have to immediately remove
> > all
> > > > >> deprecations - they can wait for the regular (say 6 months) breaking
> > > > change
> > > > >> that will be coming. The users can - if needed - even in new
> > versions
> > > of
> > > > >> Airflow - downgrade to previous provider versions in case they have
> > > > problem
> > > > >> with it.
> > > > >>
> > > > >> I like the idea of having some time (6 months) where we can very
> > > safely
> > > > >> say "OK, enough is enough, lets remove deprecations"  - and this
> > > > proposal
> > > > >> addresses it nicely. Currently it's purely subjective judgment. One
> > > > small
> > > > >> thing here to clarify for "accumulating" deprecations - I think we
> > > > should
> > > > >> make sure the deprecation is present in already released
> > > > minor/patchlevel
> > > > >> version to be able to remove it (so even if it is deprecated 2
> > months
> > > > ago
> > > > >> but we had release with the deprecation, it's ok to remove it in the
> > > > next
> > > > >> breaking change.
> > > > >>
> > > > >> 2. Agree (and again here "strict" is not good I think). I'd treat
> > > that 6
> > > > >> months as a baseline. Before that - we should have very good reason
> > to
> > > > make
> > > > >> a breaking change (and we might decide to do it partially only) -
> > > after
> > > > -
> > > > >> we are free to remove things that have been already announced as
> > > > deprecated
> > > > >> and we should consider making breaking change whenever we release a
> > > new
> > > > >> version for whatever reason (but again - not mandatory).  I's just
> > an
> > > > >> indication that tells us "Last breaking release was 6 months ago,
> > it's
> > > > now
> > > > >> OK to remove all the deprecations without having an extremely good
> > > > reason".
> > > > >>
> > > > >> Very much agree with 3. -> this should be the same as for other
> > > > >> providers. We should adopt the same rule for all of them. And if we
> > > > apply
> > > > >> the adjustments above - we should - I think - be ok to apply it to
> > any
> > > > >> provider I think.
> > > > >>
> > > > >> J
> > > > >>
> > > > >>
> > > > >>
> > > > >> On Thu, Feb 29, 2024 at 12:20 PM Elad Kalif <el...@apache.org>
> > > wrote:
> > > > >>
> > > > >>> This is not too much different than what we already do.
> > > > >>> We deprecate then we remove. We almost never create breaking change
> > > > >>> without
> > > > >>> deprecating first.
> > > > >>>
> > > > >>> I'd like to raise a few notes:
> > > > >>>
> > > > >>> 1. Major releases can happen frequently. I see no problem with it.
> > > This
> > > > >>> is
> > > > >>> more a question of what are the changes. If google has 3 major
> > > releases
> > > > >>> one
> > > > >>> with breaking change to ads, another to leveldb and 3rd to cloud
> > > these
> > > > >>> are
> > > > >>> not related one to another and unlikely that there is 1 user that
> > > uses
> > > > >>> all
> > > > >>> of these together. We create major releases when needed and we
> > keep a
> > > > >>> close
> > > > >>> eye and challenge why a change must be a breaking change. If
> > possible
> > > > we
> > > > >>> will always prefer to be backward compatible.
> > > > >>> 2. Time based deprecations are not a good approach. Some changes
> > > > require
> > > > >>> much more time to be acted upon and others may be trivial.
> > > > >>> As a rule we almost never cut a breaking change release just to
> > > remove
> > > > >>> deprecations.
> > > > >>> 3. Setting a policy per provider is not the right way to go. We
> > have
> > > a
> > > > >>> shared governance model for provider handling and I as release
> > > manager
> > > > >>> will
> > > > >>> do what I can to accommodate requests from companies who
> > participate
> > > in
> > > > >>> managing the provider. Which means that if Google wants to remove
> > > > >>> deprecations and ask for a breaking change release most likely we
> > > will
> > > > >>> accommodate their request. My point of view is that the company
> > knows
> > > > >>> what
> > > > >>> is best for their own users. I don't think that what is right for
> > > > Google
> > > > >>> must bind AWS or Microsoft. 1 policy goes against the shared
> > > > >>> governance idea.
> > > > >>>
> > > > >>>
> > > > >>>
> > > > >>> On Thu, Feb 29, 2024 at 11:04 AM Eugen Kosteev <eu...@kosteev.com>
> > > > >>> wrote:
> > > > >>>
> > > > >>> > Hi.
> > > > >>> >
> > > > >>> > I would like to discuss/propose a deprecation policy for
> > > > >>> > apache-airflow-providers-google package, mostly for deprecating
> > > > Airflow
> > > > >>> > operators (but not limited to it).
> > > > >>> >
> > > > >>> > *Some background:*
> > > > >>> > Airflow google provider package (as most of other provider
> > packages
> > > > in
> > > > >>> > Airflow) has a tendency to constantly evolve.
> > > > >>> > There are cases when its features or operators/hooks have to be
> > > > >>> deprecated.
> > > > >>> > The typical example of such a case is “refactoring” which leads
> > to
> > > > the
> > > > >>> > renaming of operators, parameters or public methods. Such
> > > > refactorings
> > > > >>> > maybe consequence of:
> > > > >>> > - using a new dependent library (that causes some methods to be
> > > > >>> deprecated)
> > > > >>> > - using a new version of API (that causes some parameters to be
> > > > >>> deprecated)
> > > > >>> > - restructuring/migration of the operators (that causes some
> > > > operators
> > > > >>> to
> > > > >>> > be deprecated)
> > > > >>> > Given the need of a “deprecation” procedure, it looks essential
> > to
> > > > >>> > establish policy around it to have users of Airflow google
> > provider
> > > > >>> package
> > > > >>> > clearly understand when and how to adapt their DAGs.
> > > > >>> >
> > > > >>> > *Preambula (versioning of the package):*
> > > > >>> > As mentioned in “Airflow’s release process and version policy” (
> > > > >>> >
> > > > >>> >
> > > > >>>
> > > >
> > >
> > https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#airflow-s-release-process-and-version-policy
> > > > >>> > )
> > > > >>> > google provider package (and others) should follow SemVer,
> > meaning
> > > > >>> that any
> > > > >>> > breaking changes should be released together with bumping major
> > > > >>> version of
> > > > >>> > the package.
> > > > >>> > The change is considered to be a breaking (in the context of
> > google
> > > > >>> > providers package) if a DAG that was working before, stops to
> > work
> > > > >>> after
> > > > >>> > the change.
> > > > >>> >
> > > > >>> > *My proposal (for discussion):*
> > > > >>> > The entire procedure of deprecating (either method, parameter or
> > > > >>> operator)
> > > > >>> > consists of two steps:
> > > > >>> > - Emission of the deprecation warning message
> > > > >>> > Example of the message:
> > > > >>> > “””
> > > > >>> > The “given” parameter is deprecated and will be removed after
> > > > >>> dd.mm.yyyy.
> > > > >>> > Please use “this” parameter instead.
> > > > >>> > “””
> > > > >>> > - Once the date of the deprecated method/parameter/operator is
> > > > passed,
> > > > >>> it
> > > > >>> > can be removed (together with bumping major version of the
> > package)
> > > > >>> >
> > > > >>> > The format of the warning message may vary, but it has to
> > contain:
> > > > >>> > - What should be used instead
> > > > >>> > - The date after which the method/parameter/operator will be
> > > removed
> > > > >>> >
> > > > >>> > *Additional considerations:*
> > > > >>> > - Bumping a major version of the Airflow google provider package
> > > > >>> shouldn’t
> > > > >>> > happen very frequently (presumably not more frequently than 6
> > > > months),
> > > > >>> > therefore when it happens it may accumulate removals of something
> > > > that
> > > > >>> was
> > > > >>> > communicated to be removed even a couple of months ago. (related
> > > > >>> discussion
> > > > >>> > https://github.com/apache/airflow/blob/main/PROVIDERS.rst)
> > > > >>> > Example: if today is 2025-12-20 and we are releasing new major
> > > > version
> > > > >>> of
> > > > >>> > the package, we can/should remove all deprecated
> > > > >>> > methods/parameters/operators which have date prior to 2025-12-20
> > -
> > > it
> > > > >>> can
> > > > >>> > be November 2025, October 2025 or even earlier, since previous
> > bump
> > > > of
> > > > >>> > major version happened e.g. in summer of 2025.
> > > > >>> > - By default all deprecations should allow a 6 months time period
> > > > until
> > > > >>> > they will be removed and not available. This period will give
> > > Airflow
> > > > >>> users
> > > > >>> > enough time and flexibility to update their DAGs before actual
> > > > removal
> > > > >>> > happens.
> > > > >>> > On a case by case basis this period can be adjusted given
> > specific
> > > > >>> > circumstances (e.g. in case deprecation is because of underlying
> > > API
> > > > >>> sunset
> > > > >>> > which can happen earlier than in 6 months).
> > > > >>> >
> > > > >>> > Apart from deprecation warning messages that will be printed as
> > > logs,
> > > > >>> this
> > > > >>> > information will be put as well in:
> > > > >>> > - Airflow documentation
> > > > >>> > - Release notes of Airflow google provider package
> > > > >>> >
> > > > >>> > Please, let me know what you think about this.
> > > > >>> >
> > > > >>> > --
> > > > >>> > Eugene
> > > > >>> >
> > > > >>>
> > > > >>
> > > >
> > >
> >
> >
> > --
> > Eugene
> >

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
For additional commands, e-mail: dev-help@airflow.apache.org

Re: [DISCUSS] Deprecation policy for apache-airflow-providers-google package

[Elad Kalif <el...@apache.org>]

Eugen, about your two items:
- What should be used instead
- The date after which the method/parameter/operator will be removed


The first is already included today in all deprecation warnings.
The second is a big -1 from me.  We normally say that it will be removed in
the next major release. Sometimes we enforce it and sometimes we don't.
That is our choice.
I said it before and I will say it again. Time based deprecation is not the
way to do it. Sometimes the author of a PR and the reviewers are not fully
aware of the blast radius of a specific deprecation.
In Airflow any committer can merge any PR - I do not think it's right to
put on the shoulders of the committer the decision of when something is to
be removed before we even accept the deprecation.

Michał, The policy we have now is a shared governance model which means
that Google can decide and act on deprecating code and also on when to
remove code.
We as maintainers, and specifically I as release manager for provider will
try to accomodate Google decision on this as Google knows best what is
right for their customers.
In the meantime, Google provider has many deprecation warnings some of them
are very old - I didn't see Google start a cleanup deprecation process.
If you think this is the right time then feel free to raise PRs for this.
This is why I say that policy may be redundant - you can do it at will.

On Tue, Mar 5, 2024 at 12:49 PM Eugen Kosteev <eu...@kosteev.com> wrote:

> Hi.
>
> Thanks for feedback and discussion.
> Let me summarise what actually I want to propose (because there were a lot
> in my initial email).
>
> My proposal is, in case of deprecation, to emit messages in the specific
> format, example of the message:
> “””
> The “given” parameter is deprecated and will be removed after dd.mm.yyyy.
> Please use “this” parameter instead.
> “””
> The format of the warning message may vary, but it has to contain:
> - What should be used instead
> - The date after which the method/parameter/operator will be removed
>
> Everything else, such as decommission/removal only once bumping major
> version, cadence of releases, etc. is not something that I propose to
> change.
>
> The idea in general is to make Airflow users a convenient way to adjust
> their DAGs to deprecations, thus:
> - what to use instead
> - when they need to update
> should help them.
>
> If we have a date in the deprecation message, it means that the removal
> (actual decommission) will happen in one of the major releases after this
> date.
>
> What do you think about this?
>
>
> On Mon, Mar 4, 2024 at 7:03 PM Michał Modras
> <mi...@google.com.invalid> wrote:
>
> > First, I'd like to say that I support Eugen's proposal and I agree with
> the
> > enhancements suggested by Jarek.
> >
> > I'm a bit confused about Elad's point 3 - are you suggesting having a
> > global policy for all providers, or that we should not codify our
> approach
> > at all since different providers have different requirements, instead of
> > having a policy per provider?
> >
> > I would be happy for this proposal (with some enhancements as needed) to
> be
> > adopted by all providers, or we can keep it locally to Google provider.
> >
> > On Thu, Feb 29, 2024 at 12:47 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> >
> > > One more thing. This one is essentially impossible (Hyrum's law
> explains
> > > that very well https://www.hyrumslaw.com/:
> > >
> > > > The change is considered to be a breaking (in the context of google
> > > providers package) if a DAG that was working before, stops to work
> after
> > > the change.
> > >
> > > I propose to change it to:
> > >
> > > > The change is considered to be a breaking if a DAG using the
> providers
> > > (in the
> > > way they were intended to be used), that was working before, stops to
> > work
> > > after
> > > the change.
> > >
> > > It is absolutely not possible to make sure that every single DAG
> written
> > by
> > > someone will continue working after any change. Literally.
> > >
> > > For example if someone extends BigQueryOperator and calls
> > > (self._this_very_much_internal_method()` in his operator's new
> execute(),
> > > and we rename the method, their DAG will stop working. Is it breaking ?
> > No.
> > > Is the user using it with the intention how it should be used?
> Absolutely
> > > not.
> > >
> > > SemVer - is not a promise things won't break, it's a promise that our
> > > intention is that things won't break. But whether the way our users use
> > it
> > > will make it break or not, that's another story.
> > >
> > > J
> > >
> > >
> > >
> > >
> > >
> > >
> > > On Thu, Feb 29, 2024 at 12:38 PM Jarek Potiuk <ja...@potiuk.com>
> wrote:
> > >
> > > > BTW. Another example of an exception would be a security fix. We
> > **migh**
> > > > find a security issue that will require a breaking change. But then
> > > again -
> > > > the breaking change might **only** cover that security fix - we do
> not
> > > have
> > > > to remove all the deprecations in this case (but we could if we
> decide
> > > so -
> > > > very much depending on nature of those deprecations)
> > > >
> > > > On Thu, Feb 29, 2024 at 12:36 PM Jarek Potiuk <ja...@potiuk.com>
> > wrote:
> > > >
> > > >> I think it's a good idea to codify the rules, indeed what Elad
> > > mentioned,
> > > >> we **mostly** follow a very similar approach, but  it's not written
> > > >> anywhere.
> > > >>
> > > >>
> > > >> Re: points of Elad:
> > > >>
> > > >> 1. I am ok with 6 months, but I agree here we should not have
> "strict"
> > > >> limits. It's fine to have some default (i.e. accumulate deprecating
> > > changes
> > > >> for some time - max. 6 months) - but I think the policy should have
> an
> > > >> option to apply exceptions - for example when we have a major change
> > in
> > > >> many libraries and we need to make some breaking changes (ads case)
> we
> > > >> shall do it (but in this case we don't have to immediately remove
> all
> > > >> deprecations - they can wait for the regular (say 6 months) breaking
> > > change
> > > >> that will be coming. The users can - if needed - even in new
> versions
> > of
> > > >> Airflow - downgrade to previous provider versions in case they have
> > > problem
> > > >> with it.
> > > >>
> > > >> I like the idea of having some time (6 months) where we can very
> > safely
> > > >> say "OK, enough is enough, lets remove deprecations"  - and this
> > > proposal
> > > >> addresses it nicely. Currently it's purely subjective judgment. One
> > > small
> > > >> thing here to clarify for "accumulating" deprecations - I think we
> > > should
> > > >> make sure the deprecation is present in already released
> > > minor/patchlevel
> > > >> version to be able to remove it (so even if it is deprecated 2
> months
> > > ago
> > > >> but we had release with the deprecation, it's ok to remove it in the
> > > next
> > > >> breaking change.
> > > >>
> > > >> 2. Agree (and again here "strict" is not good I think). I'd treat
> > that 6
> > > >> months as a baseline. Before that - we should have very good reason
> to
> > > make
> > > >> a breaking change (and we might decide to do it partially only) -
> > after
> > > -
> > > >> we are free to remove things that have been already announced as
> > > deprecated
> > > >> and we should consider making breaking change whenever we release a
> > new
> > > >> version for whatever reason (but again - not mandatory).  I's just
> an
> > > >> indication that tells us "Last breaking release was 6 months ago,
> it's
> > > now
> > > >> OK to remove all the deprecations without having an extremely good
> > > reason".
> > > >>
> > > >> Very much agree with 3. -> this should be the same as for other
> > > >> providers. We should adopt the same rule for all of them. And if we
> > > apply
> > > >> the adjustments above - we should - I think - be ok to apply it to
> any
> > > >> provider I think.
> > > >>
> > > >> J
> > > >>
> > > >>
> > > >>
> > > >> On Thu, Feb 29, 2024 at 12:20 PM Elad Kalif <el...@apache.org>
> > wrote:
> > > >>
> > > >>> This is not too much different than what we already do.
> > > >>> We deprecate then we remove. We almost never create breaking change
> > > >>> without
> > > >>> deprecating first.
> > > >>>
> > > >>> I'd like to raise a few notes:
> > > >>>
> > > >>> 1. Major releases can happen frequently. I see no problem with it.
> > This
> > > >>> is
> > > >>> more a question of what are the changes. If google has 3 major
> > releases
> > > >>> one
> > > >>> with breaking change to ads, another to leveldb and 3rd to cloud
> > these
> > > >>> are
> > > >>> not related one to another and unlikely that there is 1 user that
> > uses
> > > >>> all
> > > >>> of these together. We create major releases when needed and we
> keep a
> > > >>> close
> > > >>> eye and challenge why a change must be a breaking change. If
> possible
> > > we
> > > >>> will always prefer to be backward compatible.
> > > >>> 2. Time based deprecations are not a good approach. Some changes
> > > require
> > > >>> much more time to be acted upon and others may be trivial.
> > > >>> As a rule we almost never cut a breaking change release just to
> > remove
> > > >>> deprecations.
> > > >>> 3. Setting a policy per provider is not the right way to go. We
> have
> > a
> > > >>> shared governance model for provider handling and I as release
> > manager
> > > >>> will
> > > >>> do what I can to accommodate requests from companies who
> participate
> > in
> > > >>> managing the provider. Which means that if Google wants to remove
> > > >>> deprecations and ask for a breaking change release most likely we
> > will
> > > >>> accommodate their request. My point of view is that the company
> knows
> > > >>> what
> > > >>> is best for their own users. I don't think that what is right for
> > > Google
> > > >>> must bind AWS or Microsoft. 1 policy goes against the shared
> > > >>> governance idea.
> > > >>>
> > > >>>
> > > >>>
> > > >>> On Thu, Feb 29, 2024 at 11:04 AM Eugen Kosteev <eu...@kosteev.com>
> > > >>> wrote:
> > > >>>
> > > >>> > Hi.
> > > >>> >
> > > >>> > I would like to discuss/propose a deprecation policy for
> > > >>> > apache-airflow-providers-google package, mostly for deprecating
> > > Airflow
> > > >>> > operators (but not limited to it).
> > > >>> >
> > > >>> > *Some background:*
> > > >>> > Airflow google provider package (as most of other provider
> packages
> > > in
> > > >>> > Airflow) has a tendency to constantly evolve.
> > > >>> > There are cases when its features or operators/hooks have to be
> > > >>> deprecated.
> > > >>> > The typical example of such a case is “refactoring” which leads
> to
> > > the
> > > >>> > renaming of operators, parameters or public methods. Such
> > > refactorings
> > > >>> > maybe consequence of:
> > > >>> > - using a new dependent library (that causes some methods to be
> > > >>> deprecated)
> > > >>> > - using a new version of API (that causes some parameters to be
> > > >>> deprecated)
> > > >>> > - restructuring/migration of the operators (that causes some
> > > operators
> > > >>> to
> > > >>> > be deprecated)
> > > >>> > Given the need of a “deprecation” procedure, it looks essential
> to
> > > >>> > establish policy around it to have users of Airflow google
> provider
> > > >>> package
> > > >>> > clearly understand when and how to adapt their DAGs.
> > > >>> >
> > > >>> > *Preambula (versioning of the package):*
> > > >>> > As mentioned in “Airflow’s release process and version policy” (
> > > >>> >
> > > >>> >
> > > >>>
> > >
> >
> https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#airflow-s-release-process-and-version-policy
> > > >>> > )
> > > >>> > google provider package (and others) should follow SemVer,
> meaning
> > > >>> that any
> > > >>> > breaking changes should be released together with bumping major
> > > >>> version of
> > > >>> > the package.
> > > >>> > The change is considered to be a breaking (in the context of
> google
> > > >>> > providers package) if a DAG that was working before, stops to
> work
> > > >>> after
> > > >>> > the change.
> > > >>> >
> > > >>> > *My proposal (for discussion):*
> > > >>> > The entire procedure of deprecating (either method, parameter or
> > > >>> operator)
> > > >>> > consists of two steps:
> > > >>> > - Emission of the deprecation warning message
> > > >>> > Example of the message:
> > > >>> > “””
> > > >>> > The “given” parameter is deprecated and will be removed after
> > > >>> dd.mm.yyyy.
> > > >>> > Please use “this” parameter instead.
> > > >>> > “””
> > > >>> > - Once the date of the deprecated method/parameter/operator is
> > > passed,
> > > >>> it
> > > >>> > can be removed (together with bumping major version of the
> package)
> > > >>> >
> > > >>> > The format of the warning message may vary, but it has to
> contain:
> > > >>> > - What should be used instead
> > > >>> > - The date after which the method/parameter/operator will be
> > removed
> > > >>> >
> > > >>> > *Additional considerations:*
> > > >>> > - Bumping a major version of the Airflow google provider package
> > > >>> shouldn’t
> > > >>> > happen very frequently (presumably not more frequently than 6
> > > months),
> > > >>> > therefore when it happens it may accumulate removals of something
> > > that
> > > >>> was
> > > >>> > communicated to be removed even a couple of months ago. (related
> > > >>> discussion
> > > >>> > https://github.com/apache/airflow/blob/main/PROVIDERS.rst)
> > > >>> > Example: if today is 2025-12-20 and we are releasing new major
> > > version
> > > >>> of
> > > >>> > the package, we can/should remove all deprecated
> > > >>> > methods/parameters/operators which have date prior to 2025-12-20
> -
> > it
> > > >>> can
> > > >>> > be November 2025, October 2025 or even earlier, since previous
> bump
> > > of
> > > >>> > major version happened e.g. in summer of 2025.
> > > >>> > - By default all deprecations should allow a 6 months time period
> > > until
> > > >>> > they will be removed and not available. This period will give
> > Airflow
> > > >>> users
> > > >>> > enough time and flexibility to update their DAGs before actual
> > > removal
> > > >>> > happens.
> > > >>> > On a case by case basis this period can be adjusted given
> specific
> > > >>> > circumstances (e.g. in case deprecation is because of underlying
> > API
> > > >>> sunset
> > > >>> > which can happen earlier than in 6 months).
> > > >>> >
> > > >>> > Apart from deprecation warning messages that will be printed as
> > logs,
> > > >>> this
> > > >>> > information will be put as well in:
> > > >>> > - Airflow documentation
> > > >>> > - Release notes of Airflow google provider package
> > > >>> >
> > > >>> > Please, let me know what you think about this.
> > > >>> >
> > > >>> > --
> > > >>> > Eugene
> > > >>> >
> > > >>>
> > > >>
> > >
> >
>
>
> --
> Eugene
>

Re: [DISCUSS] Deprecation policy for apache-airflow-providers-google package

[Eugen Kosteev <eu...@kosteev.com>]

Hi.

Thanks for feedback and discussion.
Let me summarise what actually I want to propose (because there were a lot
in my initial email).

My proposal is, in case of deprecation, to emit messages in the specific
format, example of the message:
“””
The “given” parameter is deprecated and will be removed after dd.mm.yyyy.
Please use “this” parameter instead.
“””
The format of the warning message may vary, but it has to contain:
- What should be used instead
- The date after which the method/parameter/operator will be removed

Everything else, such as decommission/removal only once bumping major
version, cadence of releases, etc. is not something that I propose to
change.

The idea in general is to make Airflow users a convenient way to adjust
their DAGs to deprecations, thus:
- what to use instead
- when they need to update
should help them.

If we have a date in the deprecation message, it means that the removal
(actual decommission) will happen in one of the major releases after this
date.

What do you think about this?


On Mon, Mar 4, 2024 at 7:03 PM Michał Modras
<mi...@google.com.invalid> wrote:

> First, I'd like to say that I support Eugen's proposal and I agree with the
> enhancements suggested by Jarek.
>
> I'm a bit confused about Elad's point 3 - are you suggesting having a
> global policy for all providers, or that we should not codify our approach
> at all since different providers have different requirements, instead of
> having a policy per provider?
>
> I would be happy for this proposal (with some enhancements as needed) to be
> adopted by all providers, or we can keep it locally to Google provider.
>
> On Thu, Feb 29, 2024 at 12:47 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>
> > One more thing. This one is essentially impossible (Hyrum's law explains
> > that very well https://www.hyrumslaw.com/:
> >
> > > The change is considered to be a breaking (in the context of google
> > providers package) if a DAG that was working before, stops to work after
> > the change.
> >
> > I propose to change it to:
> >
> > > The change is considered to be a breaking if a DAG using the providers
> > (in the
> > way they were intended to be used), that was working before, stops to
> work
> > after
> > the change.
> >
> > It is absolutely not possible to make sure that every single DAG written
> by
> > someone will continue working after any change. Literally.
> >
> > For example if someone extends BigQueryOperator and calls
> > (self._this_very_much_internal_method()` in his operator's new execute(),
> > and we rename the method, their DAG will stop working. Is it breaking ?
> No.
> > Is the user using it with the intention how it should be used? Absolutely
> > not.
> >
> > SemVer - is not a promise things won't break, it's a promise that our
> > intention is that things won't break. But whether the way our users use
> it
> > will make it break or not, that's another story.
> >
> > J
> >
> >
> >
> >
> >
> >
> > On Thu, Feb 29, 2024 at 12:38 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> >
> > > BTW. Another example of an exception would be a security fix. We
> **migh**
> > > find a security issue that will require a breaking change. But then
> > again -
> > > the breaking change might **only** cover that security fix - we do not
> > have
> > > to remove all the deprecations in this case (but we could if we decide
> > so -
> > > very much depending on nature of those deprecations)
> > >
> > > On Thu, Feb 29, 2024 at 12:36 PM Jarek Potiuk <ja...@potiuk.com>
> wrote:
> > >
> > >> I think it's a good idea to codify the rules, indeed what Elad
> > mentioned,
> > >> we **mostly** follow a very similar approach, but  it's not written
> > >> anywhere.
> > >>
> > >>
> > >> Re: points of Elad:
> > >>
> > >> 1. I am ok with 6 months, but I agree here we should not have "strict"
> > >> limits. It's fine to have some default (i.e. accumulate deprecating
> > changes
> > >> for some time - max. 6 months) - but I think the policy should have an
> > >> option to apply exceptions - for example when we have a major change
> in
> > >> many libraries and we need to make some breaking changes (ads case) we
> > >> shall do it (but in this case we don't have to immediately remove all
> > >> deprecations - they can wait for the regular (say 6 months) breaking
> > change
> > >> that will be coming. The users can - if needed - even in new versions
> of
> > >> Airflow - downgrade to previous provider versions in case they have
> > problem
> > >> with it.
> > >>
> > >> I like the idea of having some time (6 months) where we can very
> safely
> > >> say "OK, enough is enough, lets remove deprecations"  - and this
> > proposal
> > >> addresses it nicely. Currently it's purely subjective judgment. One
> > small
> > >> thing here to clarify for "accumulating" deprecations - I think we
> > should
> > >> make sure the deprecation is present in already released
> > minor/patchlevel
> > >> version to be able to remove it (so even if it is deprecated 2 months
> > ago
> > >> but we had release with the deprecation, it's ok to remove it in the
> > next
> > >> breaking change.
> > >>
> > >> 2. Agree (and again here "strict" is not good I think). I'd treat
> that 6
> > >> months as a baseline. Before that - we should have very good reason to
> > make
> > >> a breaking change (and we might decide to do it partially only) -
> after
> > -
> > >> we are free to remove things that have been already announced as
> > deprecated
> > >> and we should consider making breaking change whenever we release a
> new
> > >> version for whatever reason (but again - not mandatory).  I's just an
> > >> indication that tells us "Last breaking release was 6 months ago, it's
> > now
> > >> OK to remove all the deprecations without having an extremely good
> > reason".
> > >>
> > >> Very much agree with 3. -> this should be the same as for other
> > >> providers. We should adopt the same rule for all of them. And if we
> > apply
> > >> the adjustments above - we should - I think - be ok to apply it to any
> > >> provider I think.
> > >>
> > >> J
> > >>
> > >>
> > >>
> > >> On Thu, Feb 29, 2024 at 12:20 PM Elad Kalif <el...@apache.org>
> wrote:
> > >>
> > >>> This is not too much different than what we already do.
> > >>> We deprecate then we remove. We almost never create breaking change
> > >>> without
> > >>> deprecating first.
> > >>>
> > >>> I'd like to raise a few notes:
> > >>>
> > >>> 1. Major releases can happen frequently. I see no problem with it.
> This
> > >>> is
> > >>> more a question of what are the changes. If google has 3 major
> releases
> > >>> one
> > >>> with breaking change to ads, another to leveldb and 3rd to cloud
> these
> > >>> are
> > >>> not related one to another and unlikely that there is 1 user that
> uses
> > >>> all
> > >>> of these together. We create major releases when needed and we keep a
> > >>> close
> > >>> eye and challenge why a change must be a breaking change. If possible
> > we
> > >>> will always prefer to be backward compatible.
> > >>> 2. Time based deprecations are not a good approach. Some changes
> > require
> > >>> much more time to be acted upon and others may be trivial.
> > >>> As a rule we almost never cut a breaking change release just to
> remove
> > >>> deprecations.
> > >>> 3. Setting a policy per provider is not the right way to go. We have
> a
> > >>> shared governance model for provider handling and I as release
> manager
> > >>> will
> > >>> do what I can to accommodate requests from companies who participate
> in
> > >>> managing the provider. Which means that if Google wants to remove
> > >>> deprecations and ask for a breaking change release most likely we
> will
> > >>> accommodate their request. My point of view is that the company knows
> > >>> what
> > >>> is best for their own users. I don't think that what is right for
> > Google
> > >>> must bind AWS or Microsoft. 1 policy goes against the shared
> > >>> governance idea.
> > >>>
> > >>>
> > >>>
> > >>> On Thu, Feb 29, 2024 at 11:04 AM Eugen Kosteev <eu...@kosteev.com>
> > >>> wrote:
> > >>>
> > >>> > Hi.
> > >>> >
> > >>> > I would like to discuss/propose a deprecation policy for
> > >>> > apache-airflow-providers-google package, mostly for deprecating
> > Airflow
> > >>> > operators (but not limited to it).
> > >>> >
> > >>> > *Some background:*
> > >>> > Airflow google provider package (as most of other provider packages
> > in
> > >>> > Airflow) has a tendency to constantly evolve.
> > >>> > There are cases when its features or operators/hooks have to be
> > >>> deprecated.
> > >>> > The typical example of such a case is “refactoring” which leads to
> > the
> > >>> > renaming of operators, parameters or public methods. Such
> > refactorings
> > >>> > maybe consequence of:
> > >>> > - using a new dependent library (that causes some methods to be
> > >>> deprecated)
> > >>> > - using a new version of API (that causes some parameters to be
> > >>> deprecated)
> > >>> > - restructuring/migration of the operators (that causes some
> > operators
> > >>> to
> > >>> > be deprecated)
> > >>> > Given the need of a “deprecation” procedure, it looks essential to
> > >>> > establish policy around it to have users of Airflow google provider
> > >>> package
> > >>> > clearly understand when and how to adapt their DAGs.
> > >>> >
> > >>> > *Preambula (versioning of the package):*
> > >>> > As mentioned in “Airflow’s release process and version policy” (
> > >>> >
> > >>> >
> > >>>
> >
> https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#airflow-s-release-process-and-version-policy
> > >>> > )
> > >>> > google provider package (and others) should follow SemVer, meaning
> > >>> that any
> > >>> > breaking changes should be released together with bumping major
> > >>> version of
> > >>> > the package.
> > >>> > The change is considered to be a breaking (in the context of google
> > >>> > providers package) if a DAG that was working before, stops to work
> > >>> after
> > >>> > the change.
> > >>> >
> > >>> > *My proposal (for discussion):*
> > >>> > The entire procedure of deprecating (either method, parameter or
> > >>> operator)
> > >>> > consists of two steps:
> > >>> > - Emission of the deprecation warning message
> > >>> > Example of the message:
> > >>> > “””
> > >>> > The “given” parameter is deprecated and will be removed after
> > >>> dd.mm.yyyy.
> > >>> > Please use “this” parameter instead.
> > >>> > “””
> > >>> > - Once the date of the deprecated method/parameter/operator is
> > passed,
> > >>> it
> > >>> > can be removed (together with bumping major version of the package)
> > >>> >
> > >>> > The format of the warning message may vary, but it has to contain:
> > >>> > - What should be used instead
> > >>> > - The date after which the method/parameter/operator will be
> removed
> > >>> >
> > >>> > *Additional considerations:*
> > >>> > - Bumping a major version of the Airflow google provider package
> > >>> shouldn’t
> > >>> > happen very frequently (presumably not more frequently than 6
> > months),
> > >>> > therefore when it happens it may accumulate removals of something
> > that
> > >>> was
> > >>> > communicated to be removed even a couple of months ago. (related
> > >>> discussion
> > >>> > https://github.com/apache/airflow/blob/main/PROVIDERS.rst)
> > >>> > Example: if today is 2025-12-20 and we are releasing new major
> > version
> > >>> of
> > >>> > the package, we can/should remove all deprecated
> > >>> > methods/parameters/operators which have date prior to 2025-12-20 -
> it
> > >>> can
> > >>> > be November 2025, October 2025 or even earlier, since previous bump
> > of
> > >>> > major version happened e.g. in summer of 2025.
> > >>> > - By default all deprecations should allow a 6 months time period
> > until
> > >>> > they will be removed and not available. This period will give
> Airflow
> > >>> users
> > >>> > enough time and flexibility to update their DAGs before actual
> > removal
> > >>> > happens.
> > >>> > On a case by case basis this period can be adjusted given specific
> > >>> > circumstances (e.g. in case deprecation is because of underlying
> API
> > >>> sunset
> > >>> > which can happen earlier than in 6 months).
> > >>> >
> > >>> > Apart from deprecation warning messages that will be printed as
> logs,
> > >>> this
> > >>> > information will be put as well in:
> > >>> > - Airflow documentation
> > >>> > - Release notes of Airflow google provider package
> > >>> >
> > >>> > Please, let me know what you think about this.
> > >>> >
> > >>> > --
> > >>> > Eugene
> > >>> >
> > >>>
> > >>
> >
>


-- 
Eugene

Re: [DISCUSS] Deprecation policy for apache-airflow-providers-google package

[Michał Modras <mi...@google.com.INVALID>]

First, I'd like to say that I support Eugen's proposal and I agree with the
enhancements suggested by Jarek.

I'm a bit confused about Elad's point 3 - are you suggesting having a
global policy for all providers, or that we should not codify our approach
at all since different providers have different requirements, instead of
having a policy per provider?

I would be happy for this proposal (with some enhancements as needed) to be
adopted by all providers, or we can keep it locally to Google provider.

On Thu, Feb 29, 2024 at 12:47 PM Jarek Potiuk <ja...@potiuk.com> wrote:

> One more thing. This one is essentially impossible (Hyrum's law explains
> that very well https://www.hyrumslaw.com/:
>
> > The change is considered to be a breaking (in the context of google
> providers package) if a DAG that was working before, stops to work after
> the change.
>
> I propose to change it to:
>
> > The change is considered to be a breaking if a DAG using the providers
> (in the
> way they were intended to be used), that was working before, stops to work
> after
> the change.
>
> It is absolutely not possible to make sure that every single DAG written by
> someone will continue working after any change. Literally.
>
> For example if someone extends BigQueryOperator and calls
> (self._this_very_much_internal_method()` in his operator's new execute(),
> and we rename the method, their DAG will stop working. Is it breaking ? No.
> Is the user using it with the intention how it should be used? Absolutely
> not.
>
> SemVer - is not a promise things won't break, it's a promise that our
> intention is that things won't break. But whether the way our users use it
> will make it break or not, that's another story.
>
> J
>
>
>
>
>
>
> On Thu, Feb 29, 2024 at 12:38 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>
> > BTW. Another example of an exception would be a security fix. We **migh**
> > find a security issue that will require a breaking change. But then
> again -
> > the breaking change might **only** cover that security fix - we do not
> have
> > to remove all the deprecations in this case (but we could if we decide
> so -
> > very much depending on nature of those deprecations)
> >
> > On Thu, Feb 29, 2024 at 12:36 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> >
> >> I think it's a good idea to codify the rules, indeed what Elad
> mentioned,
> >> we **mostly** follow a very similar approach, but  it's not written
> >> anywhere.
> >>
> >>
> >> Re: points of Elad:
> >>
> >> 1. I am ok with 6 months, but I agree here we should not have "strict"
> >> limits. It's fine to have some default (i.e. accumulate deprecating
> changes
> >> for some time - max. 6 months) - but I think the policy should have an
> >> option to apply exceptions - for example when we have a major change in
> >> many libraries and we need to make some breaking changes (ads case) we
> >> shall do it (but in this case we don't have to immediately remove all
> >> deprecations - they can wait for the regular (say 6 months) breaking
> change
> >> that will be coming. The users can - if needed - even in new versions of
> >> Airflow - downgrade to previous provider versions in case they have
> problem
> >> with it.
> >>
> >> I like the idea of having some time (6 months) where we can very safely
> >> say "OK, enough is enough, lets remove deprecations"  - and this
> proposal
> >> addresses it nicely. Currently it's purely subjective judgment. One
> small
> >> thing here to clarify for "accumulating" deprecations - I think we
> should
> >> make sure the deprecation is present in already released
> minor/patchlevel
> >> version to be able to remove it (so even if it is deprecated 2 months
> ago
> >> but we had release with the deprecation, it's ok to remove it in the
> next
> >> breaking change.
> >>
> >> 2. Agree (and again here "strict" is not good I think). I'd treat that 6
> >> months as a baseline. Before that - we should have very good reason to
> make
> >> a breaking change (and we might decide to do it partially only) - after
> -
> >> we are free to remove things that have been already announced as
> deprecated
> >> and we should consider making breaking change whenever we release a new
> >> version for whatever reason (but again - not mandatory).  I's just an
> >> indication that tells us "Last breaking release was 6 months ago, it's
> now
> >> OK to remove all the deprecations without having an extremely good
> reason".
> >>
> >> Very much agree with 3. -> this should be the same as for other
> >> providers. We should adopt the same rule for all of them. And if we
> apply
> >> the adjustments above - we should - I think - be ok to apply it to any
> >> provider I think.
> >>
> >> J
> >>
> >>
> >>
> >> On Thu, Feb 29, 2024 at 12:20 PM Elad Kalif <el...@apache.org> wrote:
> >>
> >>> This is not too much different than what we already do.
> >>> We deprecate then we remove. We almost never create breaking change
> >>> without
> >>> deprecating first.
> >>>
> >>> I'd like to raise a few notes:
> >>>
> >>> 1. Major releases can happen frequently. I see no problem with it. This
> >>> is
> >>> more a question of what are the changes. If google has 3 major releases
> >>> one
> >>> with breaking change to ads, another to leveldb and 3rd to cloud these
> >>> are
> >>> not related one to another and unlikely that there is 1 user that uses
> >>> all
> >>> of these together. We create major releases when needed and we keep a
> >>> close
> >>> eye and challenge why a change must be a breaking change. If possible
> we
> >>> will always prefer to be backward compatible.
> >>> 2. Time based deprecations are not a good approach. Some changes
> require
> >>> much more time to be acted upon and others may be trivial.
> >>> As a rule we almost never cut a breaking change release just to remove
> >>> deprecations.
> >>> 3. Setting a policy per provider is not the right way to go. We have a
> >>> shared governance model for provider handling and I as release manager
> >>> will
> >>> do what I can to accommodate requests from companies who participate in
> >>> managing the provider. Which means that if Google wants to remove
> >>> deprecations and ask for a breaking change release most likely we will
> >>> accommodate their request. My point of view is that the company knows
> >>> what
> >>> is best for their own users. I don't think that what is right for
> Google
> >>> must bind AWS or Microsoft. 1 policy goes against the shared
> >>> governance idea.
> >>>
> >>>
> >>>
> >>> On Thu, Feb 29, 2024 at 11:04 AM Eugen Kosteev <eu...@kosteev.com>
> >>> wrote:
> >>>
> >>> > Hi.
> >>> >
> >>> > I would like to discuss/propose a deprecation policy for
> >>> > apache-airflow-providers-google package, mostly for deprecating
> Airflow
> >>> > operators (but not limited to it).
> >>> >
> >>> > *Some background:*
> >>> > Airflow google provider package (as most of other provider packages
> in
> >>> > Airflow) has a tendency to constantly evolve.
> >>> > There are cases when its features or operators/hooks have to be
> >>> deprecated.
> >>> > The typical example of such a case is “refactoring” which leads to
> the
> >>> > renaming of operators, parameters or public methods. Such
> refactorings
> >>> > maybe consequence of:
> >>> > - using a new dependent library (that causes some methods to be
> >>> deprecated)
> >>> > - using a new version of API (that causes some parameters to be
> >>> deprecated)
> >>> > - restructuring/migration of the operators (that causes some
> operators
> >>> to
> >>> > be deprecated)
> >>> > Given the need of a “deprecation” procedure, it looks essential to
> >>> > establish policy around it to have users of Airflow google provider
> >>> package
> >>> > clearly understand when and how to adapt their DAGs.
> >>> >
> >>> > *Preambula (versioning of the package):*
> >>> > As mentioned in “Airflow’s release process and version policy” (
> >>> >
> >>> >
> >>>
> https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#airflow-s-release-process-and-version-policy
> >>> > )
> >>> > google provider package (and others) should follow SemVer, meaning
> >>> that any
> >>> > breaking changes should be released together with bumping major
> >>> version of
> >>> > the package.
> >>> > The change is considered to be a breaking (in the context of google
> >>> > providers package) if a DAG that was working before, stops to work
> >>> after
> >>> > the change.
> >>> >
> >>> > *My proposal (for discussion):*
> >>> > The entire procedure of deprecating (either method, parameter or
> >>> operator)
> >>> > consists of two steps:
> >>> > - Emission of the deprecation warning message
> >>> > Example of the message:
> >>> > “””
> >>> > The “given” parameter is deprecated and will be removed after
> >>> dd.mm.yyyy.
> >>> > Please use “this” parameter instead.
> >>> > “””
> >>> > - Once the date of the deprecated method/parameter/operator is
> passed,
> >>> it
> >>> > can be removed (together with bumping major version of the package)
> >>> >
> >>> > The format of the warning message may vary, but it has to contain:
> >>> > - What should be used instead
> >>> > - The date after which the method/parameter/operator will be removed
> >>> >
> >>> > *Additional considerations:*
> >>> > - Bumping a major version of the Airflow google provider package
> >>> shouldn’t
> >>> > happen very frequently (presumably not more frequently than 6
> months),
> >>> > therefore when it happens it may accumulate removals of something
> that
> >>> was
> >>> > communicated to be removed even a couple of months ago. (related
> >>> discussion
> >>> > https://github.com/apache/airflow/blob/main/PROVIDERS.rst)
> >>> > Example: if today is 2025-12-20 and we are releasing new major
> version
> >>> of
> >>> > the package, we can/should remove all deprecated
> >>> > methods/parameters/operators which have date prior to 2025-12-20 - it
> >>> can
> >>> > be November 2025, October 2025 or even earlier, since previous bump
> of
> >>> > major version happened e.g. in summer of 2025.
> >>> > - By default all deprecations should allow a 6 months time period
> until
> >>> > they will be removed and not available. This period will give Airflow
> >>> users
> >>> > enough time and flexibility to update their DAGs before actual
> removal
> >>> > happens.
> >>> > On a case by case basis this period can be adjusted given specific
> >>> > circumstances (e.g. in case deprecation is because of underlying API
> >>> sunset
> >>> > which can happen earlier than in 6 months).
> >>> >
> >>> > Apart from deprecation warning messages that will be printed as logs,
> >>> this
> >>> > information will be put as well in:
> >>> > - Airflow documentation
> >>> > - Release notes of Airflow google provider package
> >>> >
> >>> > Please, let me know what you think about this.
> >>> >
> >>> > --
> >>> > Eugene
> >>> >
> >>>
> >>
>

Re: [DISCUSS] Deprecation policy for apache-airflow-providers-google package

[Jarek Potiuk <ja...@potiuk.com>]

One more thing. This one is essentially impossible (Hyrum's law explains
that very well https://www.hyrumslaw.com/:

> The change is considered to be a breaking (in the context of google
providers package) if a DAG that was working before, stops to work after
the change.

I propose to change it to:

> The change is considered to be a breaking if a DAG using the providers
(in the
way they were intended to be used), that was working before, stops to work
after
the change.

It is absolutely not possible to make sure that every single DAG written by
someone will continue working after any change. Literally.

For example if someone extends BigQueryOperator and calls
(self._this_very_much_internal_method()` in his operator's new execute(),
and we rename the method, their DAG will stop working. Is it breaking ? No.
Is the user using it with the intention how it should be used? Absolutely
not.

SemVer - is not a promise things won't break, it's a promise that our
intention is that things won't break. But whether the way our users use it
will make it break or not, that's another story.

J






On Thu, Feb 29, 2024 at 12:38 PM Jarek Potiuk <ja...@potiuk.com> wrote:

> BTW. Another example of an exception would be a security fix. We **migh**
> find a security issue that will require a breaking change. But then again -
> the breaking change might **only** cover that security fix - we do not have
> to remove all the deprecations in this case (but we could if we decide so -
> very much depending on nature of those deprecations)
>
> On Thu, Feb 29, 2024 at 12:36 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>
>> I think it's a good idea to codify the rules, indeed what Elad mentioned,
>> we **mostly** follow a very similar approach, but  it's not written
>> anywhere.
>>
>>
>> Re: points of Elad:
>>
>> 1. I am ok with 6 months, but I agree here we should not have "strict"
>> limits. It's fine to have some default (i.e. accumulate deprecating changes
>> for some time - max. 6 months) - but I think the policy should have an
>> option to apply exceptions - for example when we have a major change in
>> many libraries and we need to make some breaking changes (ads case) we
>> shall do it (but in this case we don't have to immediately remove all
>> deprecations - they can wait for the regular (say 6 months) breaking change
>> that will be coming. The users can - if needed - even in new versions of
>> Airflow - downgrade to previous provider versions in case they have problem
>> with it.
>>
>> I like the idea of having some time (6 months) where we can very safely
>> say "OK, enough is enough, lets remove deprecations"  - and this proposal
>> addresses it nicely. Currently it's purely subjective judgment. One small
>> thing here to clarify for "accumulating" deprecations - I think we should
>> make sure the deprecation is present in already released minor/patchlevel
>> version to be able to remove it (so even if it is deprecated 2 months ago
>> but we had release with the deprecation, it's ok to remove it in the next
>> breaking change.
>>
>> 2. Agree (and again here "strict" is not good I think). I'd treat that 6
>> months as a baseline. Before that - we should have very good reason to make
>> a breaking change (and we might decide to do it partially only) - after -
>> we are free to remove things that have been already announced as deprecated
>> and we should consider making breaking change whenever we release a new
>> version for whatever reason (but again - not mandatory).  I's just an
>> indication that tells us "Last breaking release was 6 months ago, it's now
>> OK to remove all the deprecations without having an extremely good reason".
>>
>> Very much agree with 3. -> this should be the same as for other
>> providers. We should adopt the same rule for all of them. And if we apply
>> the adjustments above - we should - I think - be ok to apply it to any
>> provider I think.
>>
>> J
>>
>>
>>
>> On Thu, Feb 29, 2024 at 12:20 PM Elad Kalif <el...@apache.org> wrote:
>>
>>> This is not too much different than what we already do.
>>> We deprecate then we remove. We almost never create breaking change
>>> without
>>> deprecating first.
>>>
>>> I'd like to raise a few notes:
>>>
>>> 1. Major releases can happen frequently. I see no problem with it. This
>>> is
>>> more a question of what are the changes. If google has 3 major releases
>>> one
>>> with breaking change to ads, another to leveldb and 3rd to cloud these
>>> are
>>> not related one to another and unlikely that there is 1 user that uses
>>> all
>>> of these together. We create major releases when needed and we keep a
>>> close
>>> eye and challenge why a change must be a breaking change. If possible we
>>> will always prefer to be backward compatible.
>>> 2. Time based deprecations are not a good approach. Some changes require
>>> much more time to be acted upon and others may be trivial.
>>> As a rule we almost never cut a breaking change release just to remove
>>> deprecations.
>>> 3. Setting a policy per provider is not the right way to go. We have a
>>> shared governance model for provider handling and I as release manager
>>> will
>>> do what I can to accommodate requests from companies who participate in
>>> managing the provider. Which means that if Google wants to remove
>>> deprecations and ask for a breaking change release most likely we will
>>> accommodate their request. My point of view is that the company knows
>>> what
>>> is best for their own users. I don't think that what is right for Google
>>> must bind AWS or Microsoft. 1 policy goes against the shared
>>> governance idea.
>>>
>>>
>>>
>>> On Thu, Feb 29, 2024 at 11:04 AM Eugen Kosteev <eu...@kosteev.com>
>>> wrote:
>>>
>>> > Hi.
>>> >
>>> > I would like to discuss/propose a deprecation policy for
>>> > apache-airflow-providers-google package, mostly for deprecating Airflow
>>> > operators (but not limited to it).
>>> >
>>> > *Some background:*
>>> > Airflow google provider package (as most of other provider packages in
>>> > Airflow) has a tendency to constantly evolve.
>>> > There are cases when its features or operators/hooks have to be
>>> deprecated.
>>> > The typical example of such a case is “refactoring” which leads to the
>>> > renaming of operators, parameters or public methods. Such refactorings
>>> > maybe consequence of:
>>> > - using a new dependent library (that causes some methods to be
>>> deprecated)
>>> > - using a new version of API (that causes some parameters to be
>>> deprecated)
>>> > - restructuring/migration of the operators (that causes some operators
>>> to
>>> > be deprecated)
>>> > Given the need of a “deprecation” procedure, it looks essential to
>>> > establish policy around it to have users of Airflow google provider
>>> package
>>> > clearly understand when and how to adapt their DAGs.
>>> >
>>> > *Preambula (versioning of the package):*
>>> > As mentioned in “Airflow’s release process and version policy” (
>>> >
>>> >
>>> https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#airflow-s-release-process-and-version-policy
>>> > )
>>> > google provider package (and others) should follow SemVer, meaning
>>> that any
>>> > breaking changes should be released together with bumping major
>>> version of
>>> > the package.
>>> > The change is considered to be a breaking (in the context of google
>>> > providers package) if a DAG that was working before, stops to work
>>> after
>>> > the change.
>>> >
>>> > *My proposal (for discussion):*
>>> > The entire procedure of deprecating (either method, parameter or
>>> operator)
>>> > consists of two steps:
>>> > - Emission of the deprecation warning message
>>> > Example of the message:
>>> > “””
>>> > The “given” parameter is deprecated and will be removed after
>>> dd.mm.yyyy.
>>> > Please use “this” parameter instead.
>>> > “””
>>> > - Once the date of the deprecated method/parameter/operator is passed,
>>> it
>>> > can be removed (together with bumping major version of the package)
>>> >
>>> > The format of the warning message may vary, but it has to contain:
>>> > - What should be used instead
>>> > - The date after which the method/parameter/operator will be removed
>>> >
>>> > *Additional considerations:*
>>> > - Bumping a major version of the Airflow google provider package
>>> shouldn’t
>>> > happen very frequently (presumably not more frequently than 6 months),
>>> > therefore when it happens it may accumulate removals of something that
>>> was
>>> > communicated to be removed even a couple of months ago. (related
>>> discussion
>>> > https://github.com/apache/airflow/blob/main/PROVIDERS.rst)
>>> > Example: if today is 2025-12-20 and we are releasing new major version
>>> of
>>> > the package, we can/should remove all deprecated
>>> > methods/parameters/operators which have date prior to 2025-12-20 - it
>>> can
>>> > be November 2025, October 2025 or even earlier, since previous bump of
>>> > major version happened e.g. in summer of 2025.
>>> > - By default all deprecations should allow a 6 months time period until
>>> > they will be removed and not available. This period will give Airflow
>>> users
>>> > enough time and flexibility to update their DAGs before actual removal
>>> > happens.
>>> > On a case by case basis this period can be adjusted given specific
>>> > circumstances (e.g. in case deprecation is because of underlying API
>>> sunset
>>> > which can happen earlier than in 6 months).
>>> >
>>> > Apart from deprecation warning messages that will be printed as logs,
>>> this
>>> > information will be put as well in:
>>> > - Airflow documentation
>>> > - Release notes of Airflow google provider package
>>> >
>>> > Please, let me know what you think about this.
>>> >
>>> > --
>>> > Eugene
>>> >
>>>
>>

Re: [DISCUSS] Deprecation policy for apache-airflow-providers-google package

[Jarek Potiuk <ja...@potiuk.com>]

BTW. Another example of an exception would be a security fix. We **migh**
find a security issue that will require a breaking change. But then again -
the breaking change might **only** cover that security fix - we do not have
to remove all the deprecations in this case (but we could if we decide so -
very much depending on nature of those deprecations)

On Thu, Feb 29, 2024 at 12:36 PM Jarek Potiuk <ja...@potiuk.com> wrote:

> I think it's a good idea to codify the rules, indeed what Elad mentioned,
> we **mostly** follow a very similar approach, but  it's not written
> anywhere.
>
>
> Re: points of Elad:
>
> 1. I am ok with 6 months, but I agree here we should not have "strict"
> limits. It's fine to have some default (i.e. accumulate deprecating changes
> for some time - max. 6 months) - but I think the policy should have an
> option to apply exceptions - for example when we have a major change in
> many libraries and we need to make some breaking changes (ads case) we
> shall do it (but in this case we don't have to immediately remove all
> deprecations - they can wait for the regular (say 6 months) breaking change
> that will be coming. The users can - if needed - even in new versions of
> Airflow - downgrade to previous provider versions in case they have problem
> with it.
>
> I like the idea of having some time (6 months) where we can very safely
> say "OK, enough is enough, lets remove deprecations"  - and this proposal
> addresses it nicely. Currently it's purely subjective judgment. One small
> thing here to clarify for "accumulating" deprecations - I think we should
> make sure the deprecation is present in already released minor/patchlevel
> version to be able to remove it (so even if it is deprecated 2 months ago
> but we had release with the deprecation, it's ok to remove it in the next
> breaking change.
>
> 2. Agree (and again here "strict" is not good I think). I'd treat that 6
> months as a baseline. Before that - we should have very good reason to make
> a breaking change (and we might decide to do it partially only) - after -
> we are free to remove things that have been already announced as deprecated
> and we should consider making breaking change whenever we release a new
> version for whatever reason (but again - not mandatory).  I's just an
> indication that tells us "Last breaking release was 6 months ago, it's now
> OK to remove all the deprecations without having an extremely good reason".
>
> Very much agree with 3. -> this should be the same as for other providers.
> We should adopt the same rule for all of them. And if we apply
> the adjustments above - we should - I think - be ok to apply it to any
> provider I think.
>
> J
>
>
>
> On Thu, Feb 29, 2024 at 12:20 PM Elad Kalif <el...@apache.org> wrote:
>
>> This is not too much different than what we already do.
>> We deprecate then we remove. We almost never create breaking change
>> without
>> deprecating first.
>>
>> I'd like to raise a few notes:
>>
>> 1. Major releases can happen frequently. I see no problem with it. This is
>> more a question of what are the changes. If google has 3 major releases
>> one
>> with breaking change to ads, another to leveldb and 3rd to cloud these are
>> not related one to another and unlikely that there is 1 user that uses all
>> of these together. We create major releases when needed and we keep a
>> close
>> eye and challenge why a change must be a breaking change. If possible we
>> will always prefer to be backward compatible.
>> 2. Time based deprecations are not a good approach. Some changes require
>> much more time to be acted upon and others may be trivial.
>> As a rule we almost never cut a breaking change release just to remove
>> deprecations.
>> 3. Setting a policy per provider is not the right way to go. We have a
>> shared governance model for provider handling and I as release manager
>> will
>> do what I can to accommodate requests from companies who participate in
>> managing the provider. Which means that if Google wants to remove
>> deprecations and ask for a breaking change release most likely we will
>> accommodate their request. My point of view is that the company knows what
>> is best for their own users. I don't think that what is right for Google
>> must bind AWS or Microsoft. 1 policy goes against the shared
>> governance idea.
>>
>>
>>
>> On Thu, Feb 29, 2024 at 11:04 AM Eugen Kosteev <eu...@kosteev.com> wrote:
>>
>> > Hi.
>> >
>> > I would like to discuss/propose a deprecation policy for
>> > apache-airflow-providers-google package, mostly for deprecating Airflow
>> > operators (but not limited to it).
>> >
>> > *Some background:*
>> > Airflow google provider package (as most of other provider packages in
>> > Airflow) has a tendency to constantly evolve.
>> > There are cases when its features or operators/hooks have to be
>> deprecated.
>> > The typical example of such a case is “refactoring” which leads to the
>> > renaming of operators, parameters or public methods. Such refactorings
>> > maybe consequence of:
>> > - using a new dependent library (that causes some methods to be
>> deprecated)
>> > - using a new version of API (that causes some parameters to be
>> deprecated)
>> > - restructuring/migration of the operators (that causes some operators
>> to
>> > be deprecated)
>> > Given the need of a “deprecation” procedure, it looks essential to
>> > establish policy around it to have users of Airflow google provider
>> package
>> > clearly understand when and how to adapt their DAGs.
>> >
>> > *Preambula (versioning of the package):*
>> > As mentioned in “Airflow’s release process and version policy” (
>> >
>> >
>> https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#airflow-s-release-process-and-version-policy
>> > )
>> > google provider package (and others) should follow SemVer, meaning that
>> any
>> > breaking changes should be released together with bumping major version
>> of
>> > the package.
>> > The change is considered to be a breaking (in the context of google
>> > providers package) if a DAG that was working before, stops to work after
>> > the change.
>> >
>> > *My proposal (for discussion):*
>> > The entire procedure of deprecating (either method, parameter or
>> operator)
>> > consists of two steps:
>> > - Emission of the deprecation warning message
>> > Example of the message:
>> > “””
>> > The “given” parameter is deprecated and will be removed after
>> dd.mm.yyyy.
>> > Please use “this” parameter instead.
>> > “””
>> > - Once the date of the deprecated method/parameter/operator is passed,
>> it
>> > can be removed (together with bumping major version of the package)
>> >
>> > The format of the warning message may vary, but it has to contain:
>> > - What should be used instead
>> > - The date after which the method/parameter/operator will be removed
>> >
>> > *Additional considerations:*
>> > - Bumping a major version of the Airflow google provider package
>> shouldn’t
>> > happen very frequently (presumably not more frequently than 6 months),
>> > therefore when it happens it may accumulate removals of something that
>> was
>> > communicated to be removed even a couple of months ago. (related
>> discussion
>> > https://github.com/apache/airflow/blob/main/PROVIDERS.rst)
>> > Example: if today is 2025-12-20 and we are releasing new major version
>> of
>> > the package, we can/should remove all deprecated
>> > methods/parameters/operators which have date prior to 2025-12-20 - it
>> can
>> > be November 2025, October 2025 or even earlier, since previous bump of
>> > major version happened e.g. in summer of 2025.
>> > - By default all deprecations should allow a 6 months time period until
>> > they will be removed and not available. This period will give Airflow
>> users
>> > enough time and flexibility to update their DAGs before actual removal
>> > happens.
>> > On a case by case basis this period can be adjusted given specific
>> > circumstances (e.g. in case deprecation is because of underlying API
>> sunset
>> > which can happen earlier than in 6 months).
>> >
>> > Apart from deprecation warning messages that will be printed as logs,
>> this
>> > information will be put as well in:
>> > - Airflow documentation
>> > - Release notes of Airflow google provider package
>> >
>> > Please, let me know what you think about this.
>> >
>> > --
>> > Eugene
>> >
>>
>

Re: [DISCUSS] Deprecation policy for apache-airflow-providers-google package

[Jarek Potiuk <ja...@potiuk.com>]

I think it's a good idea to codify the rules, indeed what Elad mentioned,
we **mostly** follow a very similar approach, but  it's not written
anywhere.


Re: points of Elad:

1. I am ok with 6 months, but I agree here we should not have "strict"
limits. It's fine to have some default (i.e. accumulate deprecating changes
for some time - max. 6 months) - but I think the policy should have an
option to apply exceptions - for example when we have a major change in
many libraries and we need to make some breaking changes (ads case) we
shall do it (but in this case we don't have to immediately remove all
deprecations - they can wait for the regular (say 6 months) breaking change
that will be coming. The users can - if needed - even in new versions of
Airflow - downgrade to previous provider versions in case they have problem
with it.

I like the idea of having some time (6 months) where we can very safely say
"OK, enough is enough, lets remove deprecations"  - and this proposal
addresses it nicely. Currently it's purely subjective judgment. One small
thing here to clarify for "accumulating" deprecations - I think we should
make sure the deprecation is present in already released minor/patchlevel
version to be able to remove it (so even if it is deprecated 2 months ago
but we had release with the deprecation, it's ok to remove it in the next
breaking change.

2. Agree (and again here "strict" is not good I think). I'd treat that 6
months as a baseline. Before that - we should have very good reason to make
a breaking change (and we might decide to do it partially only) - after -
we are free to remove things that have been already announced as deprecated
and we should consider making breaking change whenever we release a new
version for whatever reason (but again - not mandatory).  I's just an
indication that tells us "Last breaking release was 6 months ago, it's now
OK to remove all the deprecations without having an extremely good reason".

Very much agree with 3. -> this should be the same as for other providers.
We should adopt the same rule for all of them. And if we apply
the adjustments above - we should - I think - be ok to apply it to any
provider I think.

J



On Thu, Feb 29, 2024 at 12:20 PM Elad Kalif <el...@apache.org> wrote:

> This is not too much different than what we already do.
> We deprecate then we remove. We almost never create breaking change without
> deprecating first.
>
> I'd like to raise a few notes:
>
> 1. Major releases can happen frequently. I see no problem with it. This is
> more a question of what are the changes. If google has 3 major releases one
> with breaking change to ads, another to leveldb and 3rd to cloud these are
> not related one to another and unlikely that there is 1 user that uses all
> of these together. We create major releases when needed and we keep a close
> eye and challenge why a change must be a breaking change. If possible we
> will always prefer to be backward compatible.
> 2. Time based deprecations are not a good approach. Some changes require
> much more time to be acted upon and others may be trivial.
> As a rule we almost never cut a breaking change release just to remove
> deprecations.
> 3. Setting a policy per provider is not the right way to go. We have a
> shared governance model for provider handling and I as release manager will
> do what I can to accommodate requests from companies who participate in
> managing the provider. Which means that if Google wants to remove
> deprecations and ask for a breaking change release most likely we will
> accommodate their request. My point of view is that the company knows what
> is best for their own users. I don't think that what is right for Google
> must bind AWS or Microsoft. 1 policy goes against the shared
> governance idea.
>
>
>
> On Thu, Feb 29, 2024 at 11:04 AM Eugen Kosteev <eu...@kosteev.com> wrote:
>
> > Hi.
> >
> > I would like to discuss/propose a deprecation policy for
> > apache-airflow-providers-google package, mostly for deprecating Airflow
> > operators (but not limited to it).
> >
> > *Some background:*
> > Airflow google provider package (as most of other provider packages in
> > Airflow) has a tendency to constantly evolve.
> > There are cases when its features or operators/hooks have to be
> deprecated.
> > The typical example of such a case is “refactoring” which leads to the
> > renaming of operators, parameters or public methods. Such refactorings
> > maybe consequence of:
> > - using a new dependent library (that causes some methods to be
> deprecated)
> > - using a new version of API (that causes some parameters to be
> deprecated)
> > - restructuring/migration of the operators (that causes some operators to
> > be deprecated)
> > Given the need of a “deprecation” procedure, it looks essential to
> > establish policy around it to have users of Airflow google provider
> package
> > clearly understand when and how to adapt their DAGs.
> >
> > *Preambula (versioning of the package):*
> > As mentioned in “Airflow’s release process and version policy” (
> >
> >
> https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#airflow-s-release-process-and-version-policy
> > )
> > google provider package (and others) should follow SemVer, meaning that
> any
> > breaking changes should be released together with bumping major version
> of
> > the package.
> > The change is considered to be a breaking (in the context of google
> > providers package) if a DAG that was working before, stops to work after
> > the change.
> >
> > *My proposal (for discussion):*
> > The entire procedure of deprecating (either method, parameter or
> operator)
> > consists of two steps:
> > - Emission of the deprecation warning message
> > Example of the message:
> > “””
> > The “given” parameter is deprecated and will be removed after dd.mm.yyyy.
> > Please use “this” parameter instead.
> > “””
> > - Once the date of the deprecated method/parameter/operator is passed, it
> > can be removed (together with bumping major version of the package)
> >
> > The format of the warning message may vary, but it has to contain:
> > - What should be used instead
> > - The date after which the method/parameter/operator will be removed
> >
> > *Additional considerations:*
> > - Bumping a major version of the Airflow google provider package
> shouldn’t
> > happen very frequently (presumably not more frequently than 6 months),
> > therefore when it happens it may accumulate removals of something that
> was
> > communicated to be removed even a couple of months ago. (related
> discussion
> > https://github.com/apache/airflow/blob/main/PROVIDERS.rst)
> > Example: if today is 2025-12-20 and we are releasing new major version of
> > the package, we can/should remove all deprecated
> > methods/parameters/operators which have date prior to 2025-12-20 - it can
> > be November 2025, October 2025 or even earlier, since previous bump of
> > major version happened e.g. in summer of 2025.
> > - By default all deprecations should allow a 6 months time period until
> > they will be removed and not available. This period will give Airflow
> users
> > enough time and flexibility to update their DAGs before actual removal
> > happens.
> > On a case by case basis this period can be adjusted given specific
> > circumstances (e.g. in case deprecation is because of underlying API
> sunset
> > which can happen earlier than in 6 months).
> >
> > Apart from deprecation warning messages that will be printed as logs,
> this
> > information will be put as well in:
> > - Airflow documentation
> > - Release notes of Airflow google provider package
> >
> > Please, let me know what you think about this.
> >
> > --
> > Eugene
> >
>

Re: [DISCUSS] Deprecation policy for apache-airflow-providers-google package

[Elad Kalif <el...@apache.org>]

This is not too much different than what we already do.
We deprecate then we remove. We almost never create breaking change without
deprecating first.

I'd like to raise a few notes:

1. Major releases can happen frequently. I see no problem with it. This is
more a question of what are the changes. If google has 3 major releases one
with breaking change to ads, another to leveldb and 3rd to cloud these are
not related one to another and unlikely that there is 1 user that uses all
of these together. We create major releases when needed and we keep a close
eye and challenge why a change must be a breaking change. If possible we
will always prefer to be backward compatible.
2. Time based deprecations are not a good approach. Some changes require
much more time to be acted upon and others may be trivial.
As a rule we almost never cut a breaking change release just to remove
deprecations.
3. Setting a policy per provider is not the right way to go. We have a
shared governance model for provider handling and I as release manager will
do what I can to accommodate requests from companies who participate in
managing the provider. Which means that if Google wants to remove
deprecations and ask for a breaking change release most likely we will
accommodate their request. My point of view is that the company knows what
is best for their own users. I don't think that what is right for Google
must bind AWS or Microsoft. 1 policy goes against the shared
governance idea.



On Thu, Feb 29, 2024 at 11:04 AM Eugen Kosteev <eu...@kosteev.com> wrote:

> Hi.
>
> I would like to discuss/propose a deprecation policy for
> apache-airflow-providers-google package, mostly for deprecating Airflow
> operators (but not limited to it).
>
> *Some background:*
> Airflow google provider package (as most of other provider packages in
> Airflow) has a tendency to constantly evolve.
> There are cases when its features or operators/hooks have to be deprecated.
> The typical example of such a case is “refactoring” which leads to the
> renaming of operators, parameters or public methods. Such refactorings
> maybe consequence of:
> - using a new dependent library (that causes some methods to be deprecated)
> - using a new version of API (that causes some parameters to be deprecated)
> - restructuring/migration of the operators (that causes some operators to
> be deprecated)
> Given the need of a “deprecation” procedure, it looks essential to
> establish policy around it to have users of Airflow google provider package
> clearly understand when and how to adapt their DAGs.
>
> *Preambula (versioning of the package):*
> As mentioned in “Airflow’s release process and version policy” (
>
> https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#airflow-s-release-process-and-version-policy
> )
> google provider package (and others) should follow SemVer, meaning that any
> breaking changes should be released together with bumping major version of
> the package.
> The change is considered to be a breaking (in the context of google
> providers package) if a DAG that was working before, stops to work after
> the change.
>
> *My proposal (for discussion):*
> The entire procedure of deprecating (either method, parameter or operator)
> consists of two steps:
> - Emission of the deprecation warning message
> Example of the message:
> “””
> The “given” parameter is deprecated and will be removed after dd.mm.yyyy.
> Please use “this” parameter instead.
> “””
> - Once the date of the deprecated method/parameter/operator is passed, it
> can be removed (together with bumping major version of the package)
>
> The format of the warning message may vary, but it has to contain:
> - What should be used instead
> - The date after which the method/parameter/operator will be removed
>
> *Additional considerations:*
> - Bumping a major version of the Airflow google provider package shouldn’t
> happen very frequently (presumably not more frequently than 6 months),
> therefore when it happens it may accumulate removals of something that was
> communicated to be removed even a couple of months ago. (related discussion
> https://github.com/apache/airflow/blob/main/PROVIDERS.rst)
> Example: if today is 2025-12-20 and we are releasing new major version of
> the package, we can/should remove all deprecated
> methods/parameters/operators which have date prior to 2025-12-20 - it can
> be November 2025, October 2025 or even earlier, since previous bump of
> major version happened e.g. in summer of 2025.
> - By default all deprecations should allow a 6 months time period until
> they will be removed and not available. This period will give Airflow users
> enough time and flexibility to update their DAGs before actual removal
> happens.
> On a case by case basis this period can be adjusted given specific
> circumstances (e.g. in case deprecation is because of underlying API sunset
> which can happen earlier than in 6 months).
>
> Apart from deprecation warning messages that will be printed as logs, this
> information will be put as well in:
> - Airflow documentation
> - Release notes of Airflow google provider package
>
> Please, let me know what you think about this.
>
> --
> Eugene
>