You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Kaxil Naik <ka...@apache.org> on 2021/03/08 22:38:30 UTC
[DISCUSS] Guidelines for Releasing Providers packages
Hi all,
I would like to open the release process of providers up for discussion.
Testing and Voting needs more discussion, the other points are mostly
straight-forward and had an agreement on the last dev call.
(Backport Providers won't be released after the end of this month - link
<http://apache-airflow-docs.s3-website.eu-central-1.amazonaws.com/docs/apache-airflow/latest/upgrading-to-2.html#support-for-airflow-1-10-x-releases>
so
let's ignore them in conversation)
*Batch vs Ad-hoc*:
- Release Manager would default to releasing Providers in Batch
- ad-hoc releases are OK (i.e. if there is a critical bug that needs
fixing in a single provider)
*Frequency:*
- For Batch release, we will release *every month *(starting of the
month - 1 to 7 most likely)
- Just a note that it generally takes around a week for the vote to pass
even though we have 72 hours minimum period
*Doc-only changes*
- When we have doc-only changes for Providers (during batch-release), we
should still release a new version. The majority on the Dev call had agreed
that releasing docs asap is good instead of waiting for the next release
with a code-change.
*Testing*
- License and Signature Checks are mandatory (following the ASF rules)
- For Providers, not all changes require strict testing -- you make a
judgement based on the changes for a particular provider
- Wherever possible community can help test those but not
strictly necessary. *This is where more discussion is needed. *
*Voting*
- Automate and create separate voting threads to avoid confusions vs a
single vote where we exclude the providers if someone finds a bug, let's
keep the discussion for this on
https://lists.apache.org/thread.html/r9dcc03840f478669e9bd0dc61f4088b725097da2b48ea274b7f0593e%40%3Cdev.airflow.apache.org%3E
Regards,
Kaxil
Re: [DISCUSS] Guidelines for Releasing Providers packages
Posted by Kaxil Naik <ka...@apache.org>.
I have also created a PR for adding documentation for all project-related
guidelines: https://github.com/apache/airflow/pull/14674/ -- happy to hear
everyone's thoughts.
Regards,
Kaxil
On Mon, Mar 8, 2021 at 10:43 PM Kaxil Naik <ka...@apache.org> wrote:
> Personally, I agree with *Batch vs Ad-hoc, **Frequency *and *Doc-only
> changes*.
>
> For testing, I agree that not all changes need strict testing,
> "best-effort" and judgement is fine enough for providers because of the
> low-risk nature of them.
>
> For voting, I think separating the VOTE emails can avoid confusions and
> the community and individuals can pick and choose what they vote on.
>
> Regards,
> Kaxil
>
> On Mon, Mar 8, 2021 at 10:38 PM Kaxil Naik <ka...@apache.org> wrote:
>
>> Hi all,
>>
>> I would like to open the release process of providers up for discussion.
>> Testing and Voting needs more discussion, the other points are mostly
>> straight-forward and had an agreement on the last dev call.
>>
>> (Backport Providers won't be released after the end of this month - link
>> <http://apache-airflow-docs.s3-website.eu-central-1.amazonaws.com/docs/apache-airflow/latest/upgrading-to-2.html#support-for-airflow-1-10-x-releases> so
>> let's ignore them in conversation)
>>
>> *Batch vs Ad-hoc*:
>>
>> - Release Manager would default to releasing Providers in Batch
>> - ad-hoc releases are OK (i.e. if there is a critical bug that needs
>> fixing in a single provider)
>>
>> *Frequency:*
>>
>> - For Batch release, we will release *every month *(starting of the
>> month - 1 to 7 most likely)
>> - Just a note that it generally takes around a week for the vote to
>> pass even though we have 72 hours minimum period
>>
>> *Doc-only changes*
>>
>> - When we have doc-only changes for Providers (during batch-release),
>> we should still release a new version. The majority on the Dev call had
>> agreed that releasing docs asap is good instead of waiting for the next
>> release with a code-change.
>>
>> *Testing*
>>
>> - License and Signature Checks are mandatory (following the ASF rules)
>> - For Providers, not all changes require strict testing -- you make a
>> judgement based on the changes for a particular provider
>> - Wherever possible community can help test those but not
>> strictly necessary. *This is where more discussion is needed. *
>>
>> *Voting*
>>
>> - Automate and create separate voting threads to avoid confusions vs
>> a single vote where we exclude the providers if someone finds a bug, let's
>> keep the discussion for this on
>> https://lists.apache.org/thread.html/r9dcc03840f478669e9bd0dc61f4088b725097da2b48ea274b7f0593e%40%3Cdev.airflow.apache.org%3E
>>
>> Regards,
>> Kaxil
>>
>>
Re: [DISCUSS] Guidelines for Releasing Providers packages
Posted by Kaxil Naik <ka...@apache.org>.
Personally, I agree with *Batch vs Ad-hoc, **Frequency *and *Doc-only
changes*.
For testing, I agree that not all changes need strict testing,
"best-effort" and judgement is fine enough for providers because of the
low-risk nature of them.
For voting, I think separating the VOTE emails can avoid confusions and the
community and individuals can pick and choose what they vote on.
Regards,
Kaxil
On Mon, Mar 8, 2021 at 10:38 PM Kaxil Naik <ka...@apache.org> wrote:
> Hi all,
>
> I would like to open the release process of providers up for discussion.
> Testing and Voting needs more discussion, the other points are mostly
> straight-forward and had an agreement on the last dev call.
>
> (Backport Providers won't be released after the end of this month - link
> <http://apache-airflow-docs.s3-website.eu-central-1.amazonaws.com/docs/apache-airflow/latest/upgrading-to-2.html#support-for-airflow-1-10-x-releases> so
> let's ignore them in conversation)
>
> *Batch vs Ad-hoc*:
>
> - Release Manager would default to releasing Providers in Batch
> - ad-hoc releases are OK (i.e. if there is a critical bug that needs
> fixing in a single provider)
>
> *Frequency:*
>
> - For Batch release, we will release *every month *(starting of the
> month - 1 to 7 most likely)
> - Just a note that it generally takes around a week for the vote to
> pass even though we have 72 hours minimum period
>
> *Doc-only changes*
>
> - When we have doc-only changes for Providers (during batch-release),
> we should still release a new version. The majority on the Dev call had
> agreed that releasing docs asap is good instead of waiting for the next
> release with a code-change.
>
> *Testing*
>
> - License and Signature Checks are mandatory (following the ASF rules)
> - For Providers, not all changes require strict testing -- you make a
> judgement based on the changes for a particular provider
> - Wherever possible community can help test those but not
> strictly necessary. *This is where more discussion is needed. *
>
> *Voting*
>
> - Automate and create separate voting threads to avoid confusions vs a
> single vote where we exclude the providers if someone finds a bug, let's
> keep the discussion for this on
> https://lists.apache.org/thread.html/r9dcc03840f478669e9bd0dc61f4088b725097da2b48ea274b7f0593e%40%3Cdev.airflow.apache.org%3E
>
> Regards,
> Kaxil
>
>
Re: [DISCUSS] Guidelines for Releasing Providers packages
Posted by Jarek Potiuk <ja...@potiuk.com>.
+1 on bumping min version for some providers that need it once we release
airflow 2.1.
On Fri, Apr 16, 2021 at 12:57 AM Kaxil Naik <ka...@gmail.com> wrote:
> +1 on bumping minimum required Airflow version for certain providers. That
> should be the main benefit of decoupling Providers from Core Airflow.
>
> On Thu, Apr 15, 2021 at 10:42 PM Ash Berlin-Taylor <as...@apache.org> wrote:
>
>> +1 for bumping the major version of the few (3 or 4 or so) providers that
>> need this, and setting `apache-airflow>=2.1` in their requirements then!
>>
>> On Thu, 15 Apr, 2021 at 23:30, .... <dz...@gmail.com> wrote:
>>
>> Yes. We want to use a feature that will only be available in Airflow 2.1.
>> Do we allow such situations or do we want to try to maintain backward
>> compatibility with Airflow 2.0 at all cost? If we copy some code to the
>> providers we can still be backward compatible with Airflow 2.0. Earlier,
>> Jarek suggested that we can and should keep these compatible, but then we
>> came to an agreement that if we manage the package and core versions
>> properly, this may not be needed.
>>
>> . Have I understood correctly, and more importantly, providers of v1 will
>> continue to work with Airflow 2.1?
>>
>> Yes. Since providers don't use a private API, we can move
>> _create_connection method to another package and make it public. This is a
>> safe operation. czw., 15 kwi 2021 o 13:02 Ash Berlin-Taylor <
>> ash@apache.org> napisał(a):
>>
>> Bumping the major version of the providers is only half the story -- what
>> is the behaviour if someone has pins apache-airflow-providers-foo==1.0.0
>> and installs apache-airflow==2.1 -- pip check wouldn't complain but the
>> code might not work? That's far from ideal. Ah, looking at the PR linked up
>> thread , I would call this is more an issue off forward compat, rather than
>> back compat, so the issue is that we want to release providers that need a
>> feature that only exists in 2.1, and for any such provider, we will bump
>> the major version. Have I understood correctly, and more importantly,
>> providers of v1 will continue to work with Airflow 2.1? -ash On Wed, 14
>> Apr, 2021 at 22:31, Daniel Standish <dp...@gmail.com> wrote: The
>> proposal to bump major for all providers with every core minor version
>> seems like a reasonable solution to me, and it sounds like there may be
>> consensus on that? Though eventially the version numbers may get pretty
>> large :) This discussion came to my attiontion from engagement with the
>> vault secrets enhancement. Looking at this as a test case for this
>> question, to implement this change while maintaining compatibility with
>> 2.0.0 would require an unacceptable amount of code duplication and mess.
>> And it's a relatively simple change (so we can imagine it being much worse
>> in other instances).
>>
>>
--
+48 660 796 129
Re: [DISCUSS] Guidelines for Releasing Providers packages
Posted by Kaxil Naik <ka...@gmail.com>.
+1 on bumping minimum required Airflow version for certain providers. That
should be the main benefit of decoupling Providers from Core Airflow.
On Thu, Apr 15, 2021 at 10:42 PM Ash Berlin-Taylor <as...@apache.org> wrote:
> +1 for bumping the major version of the few (3 or 4 or so) providers that
> need this, and setting `apache-airflow>=2.1` in their requirements then!
>
> On Thu, 15 Apr, 2021 at 23:30, .... <dz...@gmail.com> wrote:
>
> Yes. We want to use a feature that will only be available in Airflow 2.1.
> Do we allow such situations or do we want to try to maintain backward
> compatibility with Airflow 2.0 at all cost? If we copy some code to the
> providers we can still be backward compatible with Airflow 2.0. Earlier,
> Jarek suggested that we can and should keep these compatible, but then we
> came to an agreement that if we manage the package and core versions
> properly, this may not be needed.
>
> . Have I understood correctly, and more importantly, providers of v1 will
> continue to work with Airflow 2.1?
>
> Yes. Since providers don't use a private API, we can move
> _create_connection method to another package and make it public. This is a
> safe operation. czw., 15 kwi 2021 o 13:02 Ash Berlin-Taylor <
> ash@apache.org> napisał(a):
>
> Bumping the major version of the providers is only half the story -- what
> is the behaviour if someone has pins apache-airflow-providers-foo==1.0.0
> and installs apache-airflow==2.1 -- pip check wouldn't complain but the
> code might not work? That's far from ideal. Ah, looking at the PR linked up
> thread , I would call this is more an issue off forward compat, rather than
> back compat, so the issue is that we want to release providers that need a
> feature that only exists in 2.1, and for any such provider, we will bump
> the major version. Have I understood correctly, and more importantly,
> providers of v1 will continue to work with Airflow 2.1? -ash On Wed, 14
> Apr, 2021 at 22:31, Daniel Standish <dp...@gmail.com> wrote: The
> proposal to bump major for all providers with every core minor version
> seems like a reasonable solution to me, and it sounds like there may be
> consensus on that? Though eventially the version numbers may get pretty
> large :) This discussion came to my attiontion from engagement with the
> vault secrets enhancement. Looking at this as a test case for this
> question, to implement this change while maintaining compatibility with
> 2.0.0 would require an unacceptable amount of code duplication and mess.
> And it's a relatively simple change (so we can imagine it being much worse
> in other instances).
>
>
Re: [DISCUSS] Guidelines for Releasing Providers packages
Posted by Ash Berlin-Taylor <as...@apache.org>.
+1 for bumping the major version of the few (3 or 4 or so) providers
that need this, and setting `apache-airflow>=2.1` in their requirements
then!
On Thu, 15 Apr, 2021 at 23:30, .... <dz...@gmail.com> wrote:
> Yes. We want to use a feature that will only be available in Airflow
> 2.1. Do we allow such situations or do we want to try to maintain
> backward compatibility with Airflow 2.0 at all cost? If we copy some
> code to the providers we can still be backward compatible with Airflow
> 2.0.
>
> Earlier, Jarek suggested that we can and should keep these compatible,
> but then we came to an agreement that if we manage the package and
> core versions properly, this may not be needed.
>
>> . Have I understood correctly, and more importantly, providers of v1
>> will continue to work with Airflow 2.1?
>
> Yes. Since providers don't use a private API, we can move
> _create_connection method to another package and make it public. This
> is a safe operation.
>
> czw., 15 kwi 2021 o 13:02 Ash Berlin-Taylor <ash@apache.org
> <ma...@apache.org>> napisał(a):
>>
>> Bumping the major version of the providers is only half the story
>> -- what is the behaviour if someone has pins
>> apache-airflow-providers-foo==1.0.0 and installs apache-airflow==2.1
>> -- pip check wouldn't complain but the code might not work?
>>
>> That's far from ideal.
>>
>> Ah, looking at the PR linked up thread , I would call this is more
>> an issue off forward compat, rather than back compat, so the issue
>> is that we want to release providers that need a feature that only
>> exists in 2.1, and for any such provider, we will bump the major
>> version.
>>
>> Have I understood correctly, and more importantly, providers of v1
>> will continue to work with Airflow 2.1?
>>
>> -ash
>>
>>
>>
>>
>> On Wed, 14 Apr, 2021 at 22:31, Daniel Standish
>> <dpstandish@gmail.com <ma...@gmail.com>> wrote:
>>
>> The proposal to bump major for all providers with every core minor
>> version seems like a reasonable solution to me, and it sounds like
>> there may be consensus on that? Though eventially the version
>> numbers may get pretty large :)
>>
>> This discussion came to my attiontion from engagement with the
>> vault secrets enhancement. Looking at this as a test case for this
>> question, to implement this change while maintaining compatibility
>> with 2.0.0 would require an unacceptable amount of code duplication
>> and mess. And it's a relatively simple change (so we can imagine it
>> being much worse in other instances).
Re: [DISCUSS] Guidelines for Releasing Providers packages
Posted by "...." <dz...@gmail.com>.
Yes. We want to use a feature that will only be available in Airflow
2.1. Do we allow such situations or do we want to try to maintain
backward compatibility with Airflow 2.0 at all cost? If we copy some
code to the providers we can still be backward compatible with Airflow
2.0.
Earlier, Jarek suggested that we can and should keep these compatible,
but then we came to an agreement that if we manage the package and
core versions properly, this may not be needed.
>. Have I understood correctly, and more importantly, providers of v1 will continue to work with Airflow 2.1?
Yes. Since providers don't use a private API, we can move
_create_connection method to another package and make it public. This
is a safe operation.
czw., 15 kwi 2021 o 13:02 Ash Berlin-Taylor <as...@apache.org> napisał(a):
>
> Bumping the major version of the providers is only half the story -- what is the behaviour if someone has pins apache-airflow-providers-foo==1.0.0 and installs apache-airflow==2.1 -- pip check wouldn't complain but the code might not work?
>
> That's far from ideal.
>
> Ah, looking at the PR linked up thread , I would call this is more an issue off forward compat, rather than back compat, so the issue is that we want to release providers that need a feature that only exists in 2.1, and for any such provider, we will bump the major version.
>
> Have I understood correctly, and more importantly, providers of v1 will continue to work with Airflow 2.1?
>
> -ash
>
>
>
>
> On Wed, 14 Apr, 2021 at 22:31, Daniel Standish <dp...@gmail.com> wrote:
>
> The proposal to bump major for all providers with every core minor version seems like a reasonable solution to me, and it sounds like there may be consensus on that? Though eventially the version numbers may get pretty large :)
>
> This discussion came to my attiontion from engagement with the vault secrets enhancement. Looking at this as a test case for this question, to implement this change while maintaining compatibility with 2.0.0 would require an unacceptable amount of code duplication and mess. And it's a relatively simple change (so we can imagine it being much worse in other instances).
Re: [DISCUSS] Guidelines for Releasing Providers packages
Posted by Ash Berlin-Taylor <as...@apache.org>.
Bumping the major version of the providers is only half the story --
what is the behaviour if someone has pins
apache-airflow-providers-foo==1.0.0 and installs apache-airflow==2.1 --
pip check wouldn't complain but the code might not work?
That's far from ideal.
Ah, looking at the PR linked up thread , I would call this is more an
issue off forward compat, rather than back compat, so the issue is that
we want to release providers that need a feature that only exists in
2.1, and for any such provider, we will bump the major version.
Have I understood correctly, and more importantly, providers of v1 will
continue to work with Airflow 2.1?
-ash
On Wed, 14 Apr, 2021 at 22:31, Daniel Standish <dp...@gmail.com>
wrote:
> The proposal to bump major for all providers with every core minor
> version seems like a reasonable solution to me, and it sounds like
> there may be consensus on that? Though eventially the version
> numbers may get pretty large :)
>
> This discussion came to my attiontion from engagement with the vault
> secrets enhancement
> <https://github.com/apache/airflow/pull/15013/files>. Looking at
> this as a test case for this question, to implement this change while
> maintaining compatibility with 2.0.0 would require an unacceptable
> amount of code duplication and mess. And it's a relatively simple
> change (so we can imagine it being much worse in other instances).
Re: [DISCUSS] Guidelines for Releasing Providers packages
Posted by Daniel Standish <dp...@gmail.com>.
The proposal to bump major for all providers with every core minor version
seems like a reasonable solution to me, and it sounds like there may be
consensus on that? Though eventially the version numbers may get pretty
large :)
This discussion came to my attiontion from engagement with the vault
secrets enhancement <https://github.com/apache/airflow/pull/15013/files>.
Looking at this as a test case for this question, to implement this change
while maintaining compatibility with 2.0.0 would require an unacceptable
amount of code duplication and mess. And it's a relatively simple change
(so we can imagine it being much worse in other instances).
Re: [DISCUSS] Guidelines for Releasing Providers packages
Posted by Jarek Potiuk <ja...@potiuk.com>.
>
>
> Alternatively, we can release a new Airflow 2.1 release - bump minor,
> because we add functionality in a backward-compatible manner and bump
> major version for all provider packages, because we are introducing
> version restrictions.
>
That sounds better, We could do it without breaking any SemVer properties.
We could bump all provider's versions and make them >= 2.1.
That could indeed drive 2.1 adoption this way. Not sure if "apply_defaults"
and "common backend" is a good-enough justification for it for our users,
but for me that sounds good and I would be all for it. Maybe we can find
few more
things that we could already target for 2.1 and this could - when
accumulated -
give a nice "2.1"-worthy set of changes.
I wonder what others think :)
J,
Re: [DISCUSS] Guidelines for Releasing Providers packages
Posted by "...." <dz...@gmail.com>.
> Can we be smarter here, and skip warnings when we import from providers?
I would prefer to avoid creating divisions between community providers
and core providers. Lots of people read Airflow code to learn how to
write operators. If we don't come up with other solutions, I can do
it.
Alternatively, we can release a new Airflow 2.1 release - bump minor,
because we add functionality in a backward-compatible manner and bump
major version for all provider packages, because we are introducing
version restrictions.
> I think that can (and should) still be made in a backward-compatible way. Probably it
is 'nicer' to break the compatibility, but we are breaking the basic
premise we have with
SemVer, which is not nice. I think by adopting SemVer, we agreed to
bear the cost of
maintaining backwards compatibility.
We do not break backward compatibility, but we are releasing a new
version with new version restrictions to be able to take advantage of
the functionality that will be available in Airflow 2.1. This may of
course require a bump major version for providers packages, but we
agreed for semantic versioning just to allow for constraints between
packages.
It is possible to implement this change in a backward-compatible
manner, but it will require duplicating the common part for each
secret backend.
sob., 3 kwi 2021 o 13:51 Jarek Potiuk <ja...@potiuk.com> napisał(a):
>
> Few comments:
>
> To Kamil's post:
>
>> When it comes to testing, I think it's worth thinking about which
>> versions of Airflow we want to test on and how to be compatible with
>> Airflow. For now, we are trying to make all the changes compatible
>> with Airflow 2.0.0, but I see a few changes that may require a
>> compatibility break between Airflow and providers packages. If we do
>> not clearly specify when a given package does not need to be
>> compatible, we may soon have a serious problem with maintaining these
>> packages, as the effort needed to test and maintain compatibility
>> between multiple packages and multiple Airflow versions will only
>> increase.
>
>
> I think we should not drop 2.0.0 as the "base compatibility" level unless we "yank"
> that specific release. I would rather defer any such changes to 3.0 and speed up
> releasing 3.0 if we neeed that, rather than breaking compatibility.
> Semver has very nice properties for users (see the discussion about API
> versioning) and I think trading "some evolution speed/development easiness" for
> "user backwards compatibility piece of mind by following SemVer strictly"
> is a wise choice.
>
>> This will also limit the development of Airflow as it is now not
>> possible to add any new API / modules / functions to Apache Airflow.
>> We for providers packages can only rely on the API available on
>> Airflow 2.0.0 or on other providers' packages.
>
>
> I think we are limited only on Airflow 2.0 API, not on other providers. The providers
> have their own SemVer versioning and we've already released a couple of
> backwards-incompatible releases of providers. We can even easily add some
> minimal cross-provider dependencies (such as Google 3.0 might [optionally] depend
> on Beam 2.0+ or something like that). This is easily manageable (we have no such
> feature yet, but we can easily add such cross-package dependency with
> any future release).
>
>> For now, I found two PRs that may be problematic, but I'm sure this
>> problem will only get worse over time.
>
>
> I am not sure if it will get worse - I think any feature we see as an incompatibility,
> we should mark as 3.0 and when we accumulate enough of those, we will simply
> start working on 3.0 release. I do not think we should compromise here. This
> is what strict SemVer is all about and it has more pros than cons, I think (from
> the user perspective).
>
>>
>> My change - [AIRFLOW-6829] Auto-apply apply_default (
>> https://github.com/apache/airflow/pull/15044 ) makes all package
>> providers not backward compatible. In order not to display a warning
>> about the decommissioning of the apply defaults decorator, we need to
>> clear references to that decorator in providers packages. This
>> decorator is required in older versions.
>
>
> Can we be smarter here, and skip warnings when we import from providers?
> I think it should be possible to check what's the call-stack and not to display
> warning in case the import comes from any community-managed provider.
> And in 3.0 we can drop it completely. Is there any problem with this approach?
> Would it make it possible to be fully backwards-compatible for 2.0 and
> released providers?
>
>>
>> @davilum change - Enable Connection creation from Vault parameters
>> (https://github.com/apache/airflow/pull/15013/files) this may require
>> extracting the common part between two backends into a new module i.e.
>> create a new API.
>
>
> I think that can (and should) still be made in a backwards compatible way. Probably it
> is 'nicer' to break the compatibility, but we are breaking the basic premise we have with
> SemVer, which is not nice. I think by adopting SemVer, we agreed to bear the cost of
> maintaining backwards compatibility.
>
> To Tomek's post:
>
>> >
>> > I believe we should ask contributors to confirm that they tested the
>> > provider E2E (or at least some parts of it). But it should not be
>> > required (resources are expensive, OSS is free). As long as there's no
>> > E2E (aka system) test for every operator, running from time to time we
>> > won't be able to assure that all works as expected. But that's ok.
>> >
>
>
> Wholeheartedly agree. We should ask the users to help, but it should not be
> a hard requirement. The nice thing about providers vs. Airflow, that we have a MUCH
> easier way to mitigate any problems - the user can simply use the previous version
> of the provider until the ad-hoc fix is ready. Downgrading Airflow is not easy,
> but downgrading a single provider is super-easy. Any problem in providers
> has much smaller "overall effect" of problems introduced - it only impacts small
> amount of users and with the "each provider can be independently downgraded"
> mitigation, it should be acceptable risk that sometimes, some provider will have
> some feature broken temporarily, It happens anyway with any software and any
> amount of testing you might think of - but in this case we pretty much always have
> an easy mitigation as "given".
>
> There is an exception however here - the "big providers" like
> Google or Amazon that have bigger impact. But there,
> we have System Tests in many of those already (for Google we have
> System Tests for > 90% of the provider),
> I think we should work with the Composer an MWAA teams to
> automatically run those on their infrastructure - AKA AIP-4
> https://github.com/apache/airflow/issues/10835.
> I think this might be the best contribution from Composer/MWAA teams they
> can make - provide us with the credits to run the tests there and automate
> those. As soon as we get some of the stabilization/CI work, I am happy to
> drive it together with Tobiasz would like to take on that task -
> it is rather close to finish technically, we just need credits from both Amazon
> and Google and some time to set it up.
>
Re: [DISCUSS] Guidelines for Releasing Providers packages
Posted by Jarek Potiuk <ja...@potiuk.com>.
Few comments:
To Kamil's post:
When it comes to testing, I think it's worth thinking about which
> versions of Airflow we want to test on and how to be compatible with
> Airflow. For now, we are trying to make all the changes compatible
> with Airflow 2.0.0, but I see a few changes that may require a
> compatibility break between Airflow and providers packages. If we do
> not clearly specify when a given package does not need to be
> compatible, we may soon have a serious problem with maintaining these
> packages, as the effort needed to test and maintain compatibility
> between multiple packages and multiple Airflow versions will only
> increase.
>
I think we should not drop 2.0.0 as the "base compatibility" level unless
we "yank"
that specific release. I would rather defer any such changes to 3.0 and
speed up
releasing 3.0 if we neeed that, rather than breaking compatibility.
Semver has very nice properties for users (see the discussion about API
versioning) and I think trading "some evolution speed/development easiness"
for
"user backwards compatibility piece of mind by following SemVer strictly"
is a wise choice.
This will also limit the development of Airflow as it is now not
> possible to add any new API / modules / functions to Apache Airflow.
> We for providers packages can only rely on the API available on
> Airflow 2.0.0 or on other providers' packages.
>
I think we are limited only on Airflow 2.0 API, not on other providers. The
providers
have their own SemVer versioning and we've already released a couple of
backwards-incompatible releases of providers. We can even easily add some
minimal cross-provider dependencies (such as Google 3.0 might [optionally]
depend
on Beam 2.0+ or something like that). This is easily manageable (we have no
such
feature yet, but we can easily add such cross-package dependency with
any future release).
For now, I found two PRs that may be problematic, but I'm sure this
> problem will only get worse over time.
>
I am not sure if it will get worse - I think any feature we see as an
incompatibility,
we should mark as 3.0 and when we accumulate enough of those, we will simply
start working on 3.0 release. I do not think we should compromise here. This
is what strict SemVer is all about and it has more pros than cons, I think
(from
the user perspective).
> My change - [AIRFLOW-6829] Auto-apply apply_default (
> https://github.com/apache/airflow/pull/15044 ) makes all package
> providers not backward compatible. In order not to display a warning
> about the decommissioning of the apply defaults decorator, we need to
> clear references to that decorator in providers packages. This
> decorator is required in older versions.
>
Can we be smarter here, and skip warnings when we import from providers?
I think it should be possible to check what's the call-stack and not to
display
warning in case the import comes from any community-managed provider.
And in 3.0 we can drop it completely. Is there any problem with this
approach?
Would it make it possible to be fully backwards-compatible for 2.0 and
released providers?
> @davilum change - Enable Connection creation from Vault parameters
> (https://github.com/apache/airflow/pull/15013/files) this may require
> extracting the common part between two backends into a new module i.e.
> create a new API.
>
I think that can (and should) still be made in a backwards compatible way.
Probably it
is 'nicer' to break the compatibility, but we are breaking the basic
premise we have with
SemVer, which is not nice. I think by adopting SemVer, we agreed to bear
the cost of
maintaining backwards compatibility.
To Tomek's post:
>
> > I believe we should ask contributors to confirm that they tested the
> > provider E2E (or at least some parts of it). But it should not be
> > required (resources are expensive, OSS is free). As long as there's no
> > E2E (aka system) test for every operator, running from time to time we
> > won't be able to assure that all works as expected. But that's ok.
> >
>
Wholeheartedly agree. We should ask the users to help, but it should not be
a hard requirement. The nice thing about providers vs. Airflow, that we
have a MUCH
easier way to mitigate any problems - the user can simply use the previous
version
of the provider until the ad-hoc fix is ready. Downgrading Airflow is not
easy,
but downgrading a single provider is super-easy. Any problem in providers
has much smaller "overall effect" of problems introduced - it only impacts
small
amount of users and with the "each provider can be independently downgraded"
mitigation, it should be acceptable risk that sometimes, some provider will
have
some feature broken temporarily, It happens anyway with any software and any
amount of testing you might think of - but in this case we pretty much
always have
an easy mitigation as "given".
There is an exception however here - the "big providers" like
Google or Amazon that have bigger impact. But there,
we have System Tests in many of those already (for Google we have
System Tests for > 90% of the provider),
I think we should work with the Composer an MWAA teams to
automatically run those on their infrastructure - AKA AIP-4
https://github.com/apache/airflow/issues/10835.
I think this might be the best contribution from Composer/MWAA teams they
can make - provide us with the credits to run the tests there and automate
those. As soon as we get some of the stabilization/CI work, I am happy to
drive it together with Tobiasz would like to take on that task -
it is rather close to finish technically, we just need credits from both
Amazon
and Google and some time to set it up.
Re: [DISCUSS] Guidelines for Releasing Providers packages
Posted by Kamil Breguła <dz...@gmail.com>.
When it comes to testing, I think it's worth thinking about which
versions of Airflow we want to test on and how to be compatible with
Airflow. For now, we are trying to make all the changes compatible
with Airflow 2.0.0, but I see a few changes that may require a
compatibility break between Airflow and providers packages. If we do
not clearly specify when a given package does not need to be
compatible, we may soon have a serious problem with maintaining these
packages, as the effort needed to test and maintain compatibility
between multiple packages and multiple Airflow versions will only
increase.
This will also limit the development of Airflow as it is now not
possible to add any new API / modules / functions to Apache Airflow.
We for providers packages can only rely on the API available on
Airflow 2.0.0 or on other providers' packages.
For now, I found two PRs that may be problematic, but I'm sure this
problem will only get worse over time.
My change - [AIRFLOW-6829] Auto-apply apply_default (
https://github.com/apache/airflow/pull/15044 ) makes all package
providers not backward compatible. In order not to display a warning
about the decommissioning of the apply defaults decorator, we need to
clear references to that decorator in providers packages. This
decorator is required in older versions.
@davilum change - Enable Connection creation from Vault parameters
(https://github.com/apache/airflow/pull/15013/files) this may require
extracting the common part between two backends into a new module i.e.
create a new API.
sob., 3 kwi 2021 o 09:34 Tomasz Urbaszek <tu...@apache.org> napisał(a):
>
> > Do we want strict testing when we release a provider for the first time?
>
> I believe we should ask contributors to confirm that they tested the
> provider E2E (or at least some parts of it). But it should not be
> required (resources are expensive, OSS is free). As long as there's no
> E2E (aka system) test for every operator, running from time to time we
> won't be able to assure that all works as expected. But that's ok.
>
> If something was broken, that's good because we know users use the
> code. We can fix it and the ad-hoc approach for releases seems to
> cover this case.
>
> My personal belief is that we won't be able to test the
> operators/hooks as well as our users will do. So in my opinion the
> best thing to do is to encourage users to help with testing.
>
> Cheers,
> Tomek
>
> On Sat, 3 Apr 2021 at 02:07, Elad Kalif <el...@apache.org> wrote:
> >
> > I agree with Kaxil about testing.
> > I'm wondering if we should have a special case for testing a new provider.
> > Do we want strict testing when we release a provider for the first time?
> >
> >
> >
> > On Sat, Apr 3, 2021 at 2:24 AM Kaxil Naik <ka...@gmail.com> wrote:
> >>
> >> Thanks all, looks like we have an agreement on 3 points, I have created a PR to add those in Project Guidelines: https://github.com/apache/airflow/pull/15168
> >>
> >> For testing, I have not received enough feedback yet. Only Tomek and I have said something about it yet. I will wait for a week to hear any other thoughts on it before a create another PR to say "Not all changes need strict testing, "best-effort" and judgement is fine enough for providers because of the low-risk nature of them."
> >>
> >> Regards,
> >> Kaxil
> >>
> >> On Thu, Mar 11, 2021 at 10:37 AM Ash Berlin-Taylor <as...@apache.org> wrote:
> >>>
> >>> Not releasing doc only changes sounds like a better solution, and fair point about SemVer, I was just thinking about what is possible with python versions.
> >>>
> >>> On Tue, 9 Mar, 2021 at 22:42, Vikram Koka <vi...@astronomer.io.INVALID> wrote:
> >>>
> >>> I agree with Batch vs Ad-hoc and Frequency.
> >>>
> >>> For doc-only changes, I would prefer NOT to change the version. Primarily because of the end user perspective, as was articulated earlier in the thread.
> >>>
> >>> Best regards,
> >>> Vikram
> >>>
> >>>
> >>> On Tue, Mar 9, 2021 at 6:05 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> >>>>
> >>>> Fully agree batch and ad-hoc approach as well as with the frequency.
> >>>>
> >>>> For docs-only changes I think -post release in PyPI is not a good idea. It does not follow semver (https://semver.org/). The only way you can extend the number in SEMVER is for pre-releases:
> >>>>
> >>>> > A pre-release version MAY be denoted by appending a hyphen and a series of dot separated identifiers immediately following the patch version. Identifiers MUST comprise only ASCII alphanumerics and hyphens [0-9A-Za-z-]. Identifiers MUST NOT be empty. Numeric identifiers MUST NOT include leading zeroes. Pre-release versions have a lower precedence than the associated normal version. A pre-release version indicates that the version is unstable and might not satisfy the intended compatibility requirements as denoted by its associated normal version. Examples: 1.0.0-alpha, 1.0.0-alpha.1, 1.0.0-0.3.7, 1.0.0-x.7.z.92, 1.0.0-x-y-z.–.
> >>>>
> >>>> Not only violating the semver but this might break a number of tools.
> >>>>
> >>>> However, I think we can do a different thing in this case. I do not think we strictly need to release the packages when we see doc-only change. What we can do - we can simply tag it with *-doc1, *-doc2 tags in the repo and add some logic to "skip" such doc-only commits next time when we prepare packages.
> >>>> Then we can simply skip (automatically) building/releasing/voting doc-only packages at all, However we will continue with documentation and only release the documentation (using the existing version).
> >>>>
> >>>> Unlike packages, our documentation is mutable and we can override it.
> >>>>
> >>>> J.
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> On Tue, Mar 9, 2021 at 7:52 PM Tomasz Urbaszek <tu...@apache.org> wrote:
> >>>>>
> >>>>> I'm ok with batch and ad-hoc approach as well as with the frequency.
> >>>>>
> >>>>> In case of docs-only changes I second what Ash proposed - let's not alter the version. 100% of functionality is the same so users are not affected and there's no need to update.
> >>>>>
> >>>>> As per testing I agree that E2E testing of all providers should not be necessary to cast +1 vote. The point about ad-hoc releases allows us to release a fix to a provider if users find something that is broken beyond acceptance.
> >>>>>
> >>>>> Tomek
> >>>>>
> >>>>>
> >>>>> On Tue, 9 Mar 2021 at 11:15, Ash Berlin-Taylor <as...@apache.org> wrote:
> >>>>>>
> >>>>>> For doc-only changes there is one extra thing to decide:
> >>>>>>
> >>>>>> Should we do it as a patch version (1.0.2 etc) or as a "post" release? Python packages have this concept: https://www.python.org/dev/peps/pep-0440/#post-releases
> >>>>>>
> >>>>>> > Some projects use post-releases to address minor errors in a final release that do not affect the distributed software (for example, correcting an error in the release notes).
> >>>>>>
> >>>>>> So we could update our tooling to support this, or we could just bump the patch version and release re-release it.
> >>>>>>
> >>>>>> I have an ever so slight preference for doc only changes to be done this way, as I think is clearer to users that they don't have to update.
> >>>>>>
> >>>>>> What does everyone else think?
> >>>>>>
> >>>>>> -ash
> >>>>>>
> >>>>>> On Mon, 8 Mar, 2021 at 22:38, Kaxil Naik <ka...@apache.org> wrote:
> >>>>>>
> >>>>>> Hi all,
> >>>>>>
> >>>>>> I would like to open the release process of providers up for discussion. Testing and Voting needs more discussion, the other points are mostly straight-forward and had an agreement on the last dev call.
> >>>>>>
> >>>>>> (Backport Providers won't be released after the end of this month - link so let's ignore them in conversation)
> >>>>>>
> >>>>>> Batch vs Ad-hoc:
> >>>>>>
> >>>>>> Release Manager would default to releasing Providers in Batch
> >>>>>> ad-hoc releases are OK (i.e. if there is a critical bug that needs fixing in a single provider)
> >>>>>>
> >>>>>> Frequency:
> >>>>>>
> >>>>>> For Batch release, we will release every month (starting of the month - 1 to 7 most likely)
> >>>>>> Just a note that it generally takes around a week for the vote to pass even though we have 72 hours minimum period
> >>>>>>
> >>>>>> Doc-only changes
> >>>>>>
> >>>>>> When we have doc-only changes for Providers (during batch-release), we should still release a new version. The majority on the Dev call had agreed that releasing docs asap is good instead of waiting for the next release with a code-change.
> >>>>>>
> >>>>>> Testing
> >>>>>>
> >>>>>> License and Signature Checks are mandatory (following the ASF rules)
> >>>>>> For Providers, not all changes require strict testing -- you make a judgement based on the changes for a particular provider
> >>>>>> Wherever possible community can help test those but not strictly necessary. This is where more discussion is needed.
> >>>>>>
> >>>>>> Voting
> >>>>>>
> >>>>>> Automate and create separate voting threads to avoid confusions vs a single vote where we exclude the providers if someone finds a bug, let's keep the discussion for this on https://lists.apache.org/thread.html/r9dcc03840f478669e9bd0dc61f4088b725097da2b48ea274b7f0593e%40%3Cdev.airflow.apache.org%3E
> >>>>>>
> >>>>>> Regards,
> >>>>>> Kaxil
> >>>>>>
> >>>>
> >>>>
> >>>> --
> >>>> +48 660 796 129
Re: [DISCUSS] Guidelines for Releasing Providers packages
Posted by Tomasz Urbaszek <tu...@apache.org>.
> Do we want strict testing when we release a provider for the first time?
I believe we should ask contributors to confirm that they tested the
provider E2E (or at least some parts of it). But it should not be
required (resources are expensive, OSS is free). As long as there's no
E2E (aka system) test for every operator, running from time to time we
won't be able to assure that all works as expected. But that's ok.
If something was broken, that's good because we know users use the
code. We can fix it and the ad-hoc approach for releases seems to
cover this case.
My personal belief is that we won't be able to test the
operators/hooks as well as our users will do. So in my opinion the
best thing to do is to encourage users to help with testing.
Cheers,
Tomek
On Sat, 3 Apr 2021 at 02:07, Elad Kalif <el...@apache.org> wrote:
>
> I agree with Kaxil about testing.
> I'm wondering if we should have a special case for testing a new provider.
> Do we want strict testing when we release a provider for the first time?
>
>
>
> On Sat, Apr 3, 2021 at 2:24 AM Kaxil Naik <ka...@gmail.com> wrote:
>>
>> Thanks all, looks like we have an agreement on 3 points, I have created a PR to add those in Project Guidelines: https://github.com/apache/airflow/pull/15168
>>
>> For testing, I have not received enough feedback yet. Only Tomek and I have said something about it yet. I will wait for a week to hear any other thoughts on it before a create another PR to say "Not all changes need strict testing, "best-effort" and judgement is fine enough for providers because of the low-risk nature of them."
>>
>> Regards,
>> Kaxil
>>
>> On Thu, Mar 11, 2021 at 10:37 AM Ash Berlin-Taylor <as...@apache.org> wrote:
>>>
>>> Not releasing doc only changes sounds like a better solution, and fair point about SemVer, I was just thinking about what is possible with python versions.
>>>
>>> On Tue, 9 Mar, 2021 at 22:42, Vikram Koka <vi...@astronomer.io.INVALID> wrote:
>>>
>>> I agree with Batch vs Ad-hoc and Frequency.
>>>
>>> For doc-only changes, I would prefer NOT to change the version. Primarily because of the end user perspective, as was articulated earlier in the thread.
>>>
>>> Best regards,
>>> Vikram
>>>
>>>
>>> On Tue, Mar 9, 2021 at 6:05 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>>>>
>>>> Fully agree batch and ad-hoc approach as well as with the frequency.
>>>>
>>>> For docs-only changes I think -post release in PyPI is not a good idea. It does not follow semver (https://semver.org/). The only way you can extend the number in SEMVER is for pre-releases:
>>>>
>>>> > A pre-release version MAY be denoted by appending a hyphen and a series of dot separated identifiers immediately following the patch version. Identifiers MUST comprise only ASCII alphanumerics and hyphens [0-9A-Za-z-]. Identifiers MUST NOT be empty. Numeric identifiers MUST NOT include leading zeroes. Pre-release versions have a lower precedence than the associated normal version. A pre-release version indicates that the version is unstable and might not satisfy the intended compatibility requirements as denoted by its associated normal version. Examples: 1.0.0-alpha, 1.0.0-alpha.1, 1.0.0-0.3.7, 1.0.0-x.7.z.92, 1.0.0-x-y-z.–.
>>>>
>>>> Not only violating the semver but this might break a number of tools.
>>>>
>>>> However, I think we can do a different thing in this case. I do not think we strictly need to release the packages when we see doc-only change. What we can do - we can simply tag it with *-doc1, *-doc2 tags in the repo and add some logic to "skip" such doc-only commits next time when we prepare packages.
>>>> Then we can simply skip (automatically) building/releasing/voting doc-only packages at all, However we will continue with documentation and only release the documentation (using the existing version).
>>>>
>>>> Unlike packages, our documentation is mutable and we can override it.
>>>>
>>>> J.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Tue, Mar 9, 2021 at 7:52 PM Tomasz Urbaszek <tu...@apache.org> wrote:
>>>>>
>>>>> I'm ok with batch and ad-hoc approach as well as with the frequency.
>>>>>
>>>>> In case of docs-only changes I second what Ash proposed - let's not alter the version. 100% of functionality is the same so users are not affected and there's no need to update.
>>>>>
>>>>> As per testing I agree that E2E testing of all providers should not be necessary to cast +1 vote. The point about ad-hoc releases allows us to release a fix to a provider if users find something that is broken beyond acceptance.
>>>>>
>>>>> Tomek
>>>>>
>>>>>
>>>>> On Tue, 9 Mar 2021 at 11:15, Ash Berlin-Taylor <as...@apache.org> wrote:
>>>>>>
>>>>>> For doc-only changes there is one extra thing to decide:
>>>>>>
>>>>>> Should we do it as a patch version (1.0.2 etc) or as a "post" release? Python packages have this concept: https://www.python.org/dev/peps/pep-0440/#post-releases
>>>>>>
>>>>>> > Some projects use post-releases to address minor errors in a final release that do not affect the distributed software (for example, correcting an error in the release notes).
>>>>>>
>>>>>> So we could update our tooling to support this, or we could just bump the patch version and release re-release it.
>>>>>>
>>>>>> I have an ever so slight preference for doc only changes to be done this way, as I think is clearer to users that they don't have to update.
>>>>>>
>>>>>> What does everyone else think?
>>>>>>
>>>>>> -ash
>>>>>>
>>>>>> On Mon, 8 Mar, 2021 at 22:38, Kaxil Naik <ka...@apache.org> wrote:
>>>>>>
>>>>>> Hi all,
>>>>>>
>>>>>> I would like to open the release process of providers up for discussion. Testing and Voting needs more discussion, the other points are mostly straight-forward and had an agreement on the last dev call.
>>>>>>
>>>>>> (Backport Providers won't be released after the end of this month - link so let's ignore them in conversation)
>>>>>>
>>>>>> Batch vs Ad-hoc:
>>>>>>
>>>>>> Release Manager would default to releasing Providers in Batch
>>>>>> ad-hoc releases are OK (i.e. if there is a critical bug that needs fixing in a single provider)
>>>>>>
>>>>>> Frequency:
>>>>>>
>>>>>> For Batch release, we will release every month (starting of the month - 1 to 7 most likely)
>>>>>> Just a note that it generally takes around a week for the vote to pass even though we have 72 hours minimum period
>>>>>>
>>>>>> Doc-only changes
>>>>>>
>>>>>> When we have doc-only changes for Providers (during batch-release), we should still release a new version. The majority on the Dev call had agreed that releasing docs asap is good instead of waiting for the next release with a code-change.
>>>>>>
>>>>>> Testing
>>>>>>
>>>>>> License and Signature Checks are mandatory (following the ASF rules)
>>>>>> For Providers, not all changes require strict testing -- you make a judgement based on the changes for a particular provider
>>>>>> Wherever possible community can help test those but not strictly necessary. This is where more discussion is needed.
>>>>>>
>>>>>> Voting
>>>>>>
>>>>>> Automate and create separate voting threads to avoid confusions vs a single vote where we exclude the providers if someone finds a bug, let's keep the discussion for this on https://lists.apache.org/thread.html/r9dcc03840f478669e9bd0dc61f4088b725097da2b48ea274b7f0593e%40%3Cdev.airflow.apache.org%3E
>>>>>>
>>>>>> Regards,
>>>>>> Kaxil
>>>>>>
>>>>
>>>>
>>>> --
>>>> +48 660 796 129
Re: [DISCUSS] Guidelines for Releasing Providers packages
Posted by Elad Kalif <el...@apache.org>.
I agree with Kaxil about testing.
I'm wondering if we should have a special case for testing a new provider.
Do we want strict testing when we release a provider for the first time?
On Sat, Apr 3, 2021 at 2:24 AM Kaxil Naik <ka...@gmail.com> wrote:
> Thanks all, looks like we have an agreement on 3 points, I have created a
> PR to add those in Project Guidelines:
> https://github.com/apache/airflow/pull/15168
>
> For *testing*, I have not received enough feedback yet. Only Tomek and I
> have said something about it yet. I will wait for a week to hear any other
> thoughts on it before a create another PR to say "Not all changes need
> strict testing, "best-effort" and judgement is fine enough for providers
> because of the low-risk nature of them."
>
> Regards,
> Kaxil
>
> On Thu, Mar 11, 2021 at 10:37 AM Ash Berlin-Taylor <as...@apache.org> wrote:
>
>> Not releasing doc only changes sounds like a better solution, and fair
>> point about SemVer, I was just thinking about what is possible with python
>> versions.
>>
>> On Tue, 9 Mar, 2021 at 22:42, Vikram Koka <vi...@astronomer.io.INVALID>
>> wrote:
>>
>> I agree with *Batch vs Ad-hoc *and *Frequency.*
>>
>> For doc-only changes, I would prefer NOT to change the version. Primarily
>> because of the end user perspective, as was articulated earlier in the
>> thread.
>>
>> Best regards,
>> Vikram
>>
>>
>> On Tue, Mar 9, 2021 at 6:05 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>>
>>> Fully agree batch and ad-hoc approach as well as with the frequency.
>>>
>>> For docs-only changes I think -post release in PyPI is not a good
>>> idea. It does not follow semver (https://semver.org/). The only way you
>>> can extend the number in SEMVER is for pre-releases:
>>>
>>> > A pre-release version MAY be denoted by appending a hyphen and a
>>> series of dot separated identifiers immediately following the patch
>>> version. Identifiers MUST comprise only ASCII alphanumerics and hyphens
>>> [0-9A-Za-z-]. Identifiers MUST NOT be empty. Numeric identifiers MUST NOT
>>> include leading zeroes. Pre-release versions have a lower precedence than
>>> the associated normal version. A pre-release version indicates that the
>>> version is unstable and might not satisfy the intended compatibility
>>> requirements as denoted by its associated normal version. Examples:
>>> 1.0.0-alpha, 1.0.0-alpha.1, 1.0.0-0.3.7, 1.0.0-x.7.z.92, 1.0.0-x-y-z.–.
>>>
>>> Not only violating the semver but this might break a number of tools.
>>>
>>> However, I think we can do a different thing in this case. I do not
>>> think we strictly need to release the packages when we see doc-only change.
>>> What we can do - we can simply tag it with *-doc1, *-doc2 tags in the repo
>>> and add some logic to "skip" such doc-only commits next time when we
>>> prepare packages.
>>> Then we can simply skip (automatically) building/releasing/voting
>>> doc-only packages at all, However we will continue with documentation and
>>> only release the documentation (using the existing version).
>>>
>>> Unlike packages, our documentation is mutable and we can override it.
>>>
>>> J.
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Tue, Mar 9, 2021 at 7:52 PM Tomasz Urbaszek <tu...@apache.org>
>>> wrote:
>>>
>>>> I'm ok with batch and ad-hoc approach as well as with the frequency.
>>>>
>>>> In case of docs-only changes I second what Ash proposed - let's not
>>>> alter the version. 100% of functionality is the same so users are not
>>>> affected and there's no need to update.
>>>>
>>>> As per testing I agree that E2E testing of all providers should not be
>>>> necessary to cast +1 vote. The point about ad-hoc releases allows us to
>>>> release a fix to a provider if users find something that is broken
>>>> beyond acceptance.
>>>>
>>>> Tomek
>>>>
>>>>
>>>> On Tue, 9 Mar 2021 at 11:15, Ash Berlin-Taylor <as...@apache.org> wrote:
>>>>
>>>>> For doc-only changes there is one extra thing to decide:
>>>>>
>>>>> Should we do it as a patch version (1.0.2 etc) or as a "post" release?
>>>>> Python packages have this concept:
>>>>> https://www.python.org/dev/peps/pep-0440/#post-releases
>>>>>
>>>>> > Some projects use post-releases to address minor errors in a final
>>>>> release that do not affect the distributed software (for example,
>>>>> correcting an error in the release notes).
>>>>>
>>>>> So we could update our tooling to support this, or we could just bump
>>>>> the patch version and release re-release it.
>>>>>
>>>>> I have an ever so slight preference for doc only changes to be done
>>>>> this way, as I think is clearer to users that they don't have to update.
>>>>>
>>>>> What does everyone else think?
>>>>>
>>>>> -ash
>>>>>
>>>>> On Mon, 8 Mar, 2021 at 22:38, Kaxil Naik <ka...@apache.org> wrote:
>>>>>
>>>>> Hi all,
>>>>>
>>>>> I would like to open the release process of providers up for
>>>>> discussion. Testing and Voting needs more discussion, the other points are
>>>>> mostly straight-forward and had an agreement on the last dev call.
>>>>>
>>>>> (Backport Providers won't be released after the end of this month -
>>>>> link
>>>>> <http://apache-airflow-docs.s3-website.eu-central-1.amazonaws.com/docs/apache-airflow/latest/upgrading-to-2.html#support-for-airflow-1-10-x-releases> so
>>>>> let's ignore them in conversation)
>>>>>
>>>>> *Batch vs Ad-hoc*:
>>>>>
>>>>> - Release Manager would default to releasing Providers in Batch
>>>>> - ad-hoc releases are OK (i.e. if there is a critical bug that
>>>>> needs fixing in a single provider)
>>>>>
>>>>> *Frequency:*
>>>>>
>>>>> - For Batch release, we will release *every month *(starting of
>>>>> the month - 1 to 7 most likely)
>>>>> - Just a note that it generally takes around a week for the vote
>>>>> to pass even though we have 72 hours minimum period
>>>>>
>>>>> *Doc-only changes*
>>>>>
>>>>> - When we have doc-only changes for Providers (during
>>>>> batch-release), we should still release a new version. The majority on the
>>>>> Dev call had agreed that releasing docs asap is good instead of waiting for
>>>>> the next release with a code-change.
>>>>>
>>>>> *Testing*
>>>>>
>>>>> - License and Signature Checks are mandatory (following the ASF
>>>>> rules)
>>>>> - For Providers, not all changes require strict testing -- you
>>>>> make a judgement based on the changes for a particular provider
>>>>> - Wherever possible community can help test those but not
>>>>> strictly necessary. *This is where more discussion is needed. *
>>>>>
>>>>> *Voting*
>>>>>
>>>>> - Automate and create separate voting threads to avoid confusions
>>>>> vs a single vote where we exclude the providers if someone finds a bug,
>>>>> let's keep the discussion for this on
>>>>> https://lists.apache.org/thread.html/r9dcc03840f478669e9bd0dc61f4088b725097da2b48ea274b7f0593e%40%3Cdev.airflow.apache.org%3E
>>>>>
>>>>> Regards,
>>>>> Kaxil
>>>>>
>>>>>
>>>
>>> --
>>> +48 660 796 129
>>>
>>
Re: [DISCUSS] Guidelines for Releasing Providers packages
Posted by Kaxil Naik <ka...@gmail.com>.
Thanks all, looks like we have an agreement on 3 points, I have created a
PR to add those in Project Guidelines:
https://github.com/apache/airflow/pull/15168
For *testing*, I have not received enough feedback yet. Only Tomek and I
have said something about it yet. I will wait for a week to hear any other
thoughts on it before a create another PR to say "Not all changes need
strict testing, "best-effort" and judgement is fine enough for providers
because of the low-risk nature of them."
Regards,
Kaxil
On Thu, Mar 11, 2021 at 10:37 AM Ash Berlin-Taylor <as...@apache.org> wrote:
> Not releasing doc only changes sounds like a better solution, and fair
> point about SemVer, I was just thinking about what is possible with python
> versions.
>
> On Tue, 9 Mar, 2021 at 22:42, Vikram Koka <vi...@astronomer.io.INVALID>
> wrote:
>
> I agree with *Batch vs Ad-hoc *and *Frequency.*
>
> For doc-only changes, I would prefer NOT to change the version. Primarily
> because of the end user perspective, as was articulated earlier in the
> thread.
>
> Best regards,
> Vikram
>
>
> On Tue, Mar 9, 2021 at 6:05 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>
>> Fully agree batch and ad-hoc approach as well as with the frequency.
>>
>> For docs-only changes I think -post release in PyPI is not a good
>> idea. It does not follow semver (https://semver.org/). The only way you
>> can extend the number in SEMVER is for pre-releases:
>>
>> > A pre-release version MAY be denoted by appending a hyphen and a series
>> of dot separated identifiers immediately following the patch version.
>> Identifiers MUST comprise only ASCII alphanumerics and hyphens
>> [0-9A-Za-z-]. Identifiers MUST NOT be empty. Numeric identifiers MUST NOT
>> include leading zeroes. Pre-release versions have a lower precedence than
>> the associated normal version. A pre-release version indicates that the
>> version is unstable and might not satisfy the intended compatibility
>> requirements as denoted by its associated normal version. Examples:
>> 1.0.0-alpha, 1.0.0-alpha.1, 1.0.0-0.3.7, 1.0.0-x.7.z.92, 1.0.0-x-y-z.–.
>>
>> Not only violating the semver but this might break a number of tools.
>>
>> However, I think we can do a different thing in this case. I do not think
>> we strictly need to release the packages when we see doc-only change. What
>> we can do - we can simply tag it with *-doc1, *-doc2 tags in the repo and
>> add some logic to "skip" such doc-only commits next time when we prepare
>> packages.
>> Then we can simply skip (automatically) building/releasing/voting
>> doc-only packages at all, However we will continue with documentation and
>> only release the documentation (using the existing version).
>>
>> Unlike packages, our documentation is mutable and we can override it.
>>
>> J.
>>
>>
>>
>>
>>
>>
>>
>>
>> On Tue, Mar 9, 2021 at 7:52 PM Tomasz Urbaszek <tu...@apache.org>
>> wrote:
>>
>>> I'm ok with batch and ad-hoc approach as well as with the frequency.
>>>
>>> In case of docs-only changes I second what Ash proposed - let's not
>>> alter the version. 100% of functionality is the same so users are not
>>> affected and there's no need to update.
>>>
>>> As per testing I agree that E2E testing of all providers should not be
>>> necessary to cast +1 vote. The point about ad-hoc releases allows us to
>>> release a fix to a provider if users find something that is broken
>>> beyond acceptance.
>>>
>>> Tomek
>>>
>>>
>>> On Tue, 9 Mar 2021 at 11:15, Ash Berlin-Taylor <as...@apache.org> wrote:
>>>
>>>> For doc-only changes there is one extra thing to decide:
>>>>
>>>> Should we do it as a patch version (1.0.2 etc) or as a "post" release?
>>>> Python packages have this concept:
>>>> https://www.python.org/dev/peps/pep-0440/#post-releases
>>>>
>>>> > Some projects use post-releases to address minor errors in a final
>>>> release that do not affect the distributed software (for example,
>>>> correcting an error in the release notes).
>>>>
>>>> So we could update our tooling to support this, or we could just bump
>>>> the patch version and release re-release it.
>>>>
>>>> I have an ever so slight preference for doc only changes to be done
>>>> this way, as I think is clearer to users that they don't have to update.
>>>>
>>>> What does everyone else think?
>>>>
>>>> -ash
>>>>
>>>> On Mon, 8 Mar, 2021 at 22:38, Kaxil Naik <ka...@apache.org> wrote:
>>>>
>>>> Hi all,
>>>>
>>>> I would like to open the release process of providers up for
>>>> discussion. Testing and Voting needs more discussion, the other points are
>>>> mostly straight-forward and had an agreement on the last dev call.
>>>>
>>>> (Backport Providers won't be released after the end of this month -
>>>> link
>>>> <http://apache-airflow-docs.s3-website.eu-central-1.amazonaws.com/docs/apache-airflow/latest/upgrading-to-2.html#support-for-airflow-1-10-x-releases> so
>>>> let's ignore them in conversation)
>>>>
>>>> *Batch vs Ad-hoc*:
>>>>
>>>> - Release Manager would default to releasing Providers in Batch
>>>> - ad-hoc releases are OK (i.e. if there is a critical bug that
>>>> needs fixing in a single provider)
>>>>
>>>> *Frequency:*
>>>>
>>>> - For Batch release, we will release *every month *(starting of the
>>>> month - 1 to 7 most likely)
>>>> - Just a note that it generally takes around a week for the vote to
>>>> pass even though we have 72 hours minimum period
>>>>
>>>> *Doc-only changes*
>>>>
>>>> - When we have doc-only changes for Providers (during
>>>> batch-release), we should still release a new version. The majority on the
>>>> Dev call had agreed that releasing docs asap is good instead of waiting for
>>>> the next release with a code-change.
>>>>
>>>> *Testing*
>>>>
>>>> - License and Signature Checks are mandatory (following the ASF
>>>> rules)
>>>> - For Providers, not all changes require strict testing -- you make
>>>> a judgement based on the changes for a particular provider
>>>> - Wherever possible community can help test those but not
>>>> strictly necessary. *This is where more discussion is needed. *
>>>>
>>>> *Voting*
>>>>
>>>> - Automate and create separate voting threads to avoid confusions
>>>> vs a single vote where we exclude the providers if someone finds a bug,
>>>> let's keep the discussion for this on
>>>> https://lists.apache.org/thread.html/r9dcc03840f478669e9bd0dc61f4088b725097da2b48ea274b7f0593e%40%3Cdev.airflow.apache.org%3E
>>>>
>>>> Regards,
>>>> Kaxil
>>>>
>>>>
>>
>> --
>> +48 660 796 129
>>
>
Re: [DISCUSS] Guidelines for Releasing Providers packages
Posted by Ash Berlin-Taylor <as...@apache.org>.
Not releasing doc only changes sounds like a better solution, and fair
point about SemVer, I was just thinking about what is possible with
python versions.
On Tue, 9 Mar, 2021 at 22:42, Vikram Koka
<vi...@astronomer.io.INVALID> wrote:
> I agree with *Batch vs Ad-hoc *and *Frequency.*
>
> For doc-only changes, I would prefer NOT to change the version.
> Primarily because of the end user perspective, as was articulated
> earlier in the thread.
>
> Best regards,
> Vikram
>
>
> On Tue, Mar 9, 2021 at 6:05 PM Jarek Potiuk <jarek@potiuk.com
> <ma...@potiuk.com>> wrote:
>> Fully agree batch and ad-hoc approach as well as with the frequency.
>>
>> For docs-only changes I think -post release in PyPI is not a good
>> idea. It does not follow semver (<https://semver.org/>). The only
>> way you can extend the number in SEMVER is for pre-releases:
>>
>> > A pre-release version MAY be denoted by appending a hyphen and a
>> series of dot separated identifiers immediately following the patch
>> version. Identifiers MUST comprise only ASCII alphanumerics and
>> hyphens [0-9A-Za-z-]. Identifiers MUST NOT be empty. Numeric
>> identifiers MUST NOT include leading zeroes. Pre-release versions
>> have a lower precedence than the associated normal version. A
>> pre-release version indicates that the version is unstable and might
>> not satisfy the intended compatibility requirements as denoted by
>> its associated normal version. Examples: 1.0.0-alpha, 1.0.0-alpha.1,
>> 1.0.0-0.3.7, 1.0.0-x.7.z.92, 1.0.0-x-y-z.–.
>>
>> Not only violating the semver but this might break a number of tools.
>>
>> However, I think we can do a different thing in this case. I do not
>> think we strictly need to release the packages when we see doc-only
>> change. What we can do - we can simply tag it with *-doc1, *-doc2
>> tags in the repo and add some logic to "skip" such doc-only commits
>> next time when we prepare packages.
>> Then we can simply skip (automatically) building/releasing/voting
>> doc-only packages at all, However we will continue with
>> documentation and only release the documentation (using the existing
>> version).
>>
>> Unlike packages, our documentation is mutable and we can override it.
>>
>> J.
>>
>>
>>
>>
>>
>>
>>
>>
>> On Tue, Mar 9, 2021 at 7:52 PM Tomasz Urbaszek <turbaszek@apache.org
>> <ma...@apache.org>> wrote:
>>> I'm ok with batch and ad-hoc approach as well as with the frequency.
>>>
>>> In case of docs-only changes I second what Ash proposed - let's not
>>> alter the version. 100% of functionality is the same so users are
>>> not affected and there's no need to update.
>>>
>>> As per testing I agree that E2E testing of all providers should not
>>> be necessary to cast +1 vote. The point about ad-hoc releases
>>> allows us to release a fix to a provider if users find something
>>> that is broken beyond acceptance.
>>>
>>> Tomek
>>>
>>>
>>> On Tue, 9 Mar 2021 at 11:15, Ash Berlin-Taylor <ash@apache.org
>>> <ma...@apache.org>> wrote:
>>>> For doc-only changes there is one extra thing to decide:
>>>>
>>>> Should we do it as a patch version (1.0.2 etc) or as a "post"
>>>> release? Python packages have this concept:
>>>> <https://www.python.org/dev/peps/pep-0440/#post-releases>
>>>>
>>>> > Some projects use post-releases to address minor errors in a
>>>> final release that do not affect the distributed software (for
>>>> example, correcting an error in the release notes).
>>>>
>>>> So we could update our tooling to support this, or we could just
>>>> bump the patch version and release re-release it.
>>>>
>>>> I have an ever so slight preference for doc only changes to be
>>>> done this way, as I think is clearer to users that they don't have
>>>> to update.
>>>>
>>>> What does everyone else think?
>>>>
>>>> -ash
>>>>
>>>> On Mon, 8 Mar, 2021 at 22:38, Kaxil Naik <kaxilnaik@apache.org
>>>> <ma...@apache.org>> wrote:
>>>>> Hi all,
>>>>>
>>>>> I would like to open the release process of providers up for
>>>>> discussion. Testing and Voting needs more discussion, the other
>>>>> points are mostly straight-forward and had an agreement on the
>>>>> last dev call.
>>>>>
>>>>> (Backport Providers won't be released after the end of this month
>>>>> - link
>>>>> <http://apache-airflow-docs.s3-website.eu-central-1.amazonaws.com/docs/apache-airflow/latest/upgrading-to-2.html#support-for-airflow-1-10-x-releases>
>>>>> so let's ignore them in conversation)
>>>>>
>>>>> *Batch vs Ad-hoc*:
>>>>> Release Manager would default to releasing Providers in Batch
>>>>> ad-hoc releases are OK (i.e. if there is a critical bug that
>>>>> needs fixing in a single provider)
>>>>> *Frequency:*
>>>>> For Batch release, we will release *every month *(starting of the
>>>>> month - 1 to 7 most likely)
>>>>> Just a note that it generally takes around a week for the vote to
>>>>> pass even though we have 72 hours minimum period
>>>>> *Doc-only changes*
>>>>> When we have doc-only changes for Providers (during
>>>>> batch-release), we should still release a new version. The
>>>>> majority on the Dev call had agreed that releasing docs asap is
>>>>> good instead of waiting for the next release with a code-change.
>>>>> *Testing*
>>>>> License and Signature Checks are mandatory (following the ASF
>>>>> rules)
>>>>> For Providers, not all changes require strict testing -- you make
>>>>> a judgement based on the changes for a particular provider
>>>>> Wherever possible community can help test those but not strictly
>>>>> necessary. *This is where more discussion is needed. *
>>>>> *Voting*
>>>>> Automate and create separate voting threads to avoid confusions
>>>>> vs a single vote where we exclude the providers if someone finds
>>>>> a bug, let's keep the discussion for this on
>>>>> <https://lists.apache.org/thread.html/r9dcc03840f478669e9bd0dc61f4088b725097da2b48ea274b7f0593e%40%3Cdev.airflow.apache.org%3E>
>>>>> Regards,
>>>>> Kaxil
>>>>> *
>>>>> *
>>
>>
>> --
>> +48 660 796 129
Re: [DISCUSS] Guidelines for Releasing Providers packages
Posted by Vikram Koka <vi...@astronomer.io.INVALID>.
I agree with *Batch vs Ad-hoc *and *Frequency.*
For doc-only changes, I would prefer NOT to change the version. Primarily
because of the end user perspective, as was articulated earlier in the
thread.
Best regards,
Vikram
On Tue, Mar 9, 2021 at 6:05 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> Fully agree batch and ad-hoc approach as well as with the frequency.
>
> For docs-only changes I think -post release in PyPI is not a good idea. It
> does not follow semver (https://semver.org/). The only way you can extend
> the number in SEMVER is for pre-releases:
>
> > A pre-release version MAY be denoted by appending a hyphen and a series
> of dot separated identifiers immediately following the patch version.
> Identifiers MUST comprise only ASCII alphanumerics and hyphens
> [0-9A-Za-z-]. Identifiers MUST NOT be empty. Numeric identifiers MUST NOT
> include leading zeroes. Pre-release versions have a lower precedence than
> the associated normal version. A pre-release version indicates that the
> version is unstable and might not satisfy the intended compatibility
> requirements as denoted by its associated normal version. Examples:
> 1.0.0-alpha, 1.0.0-alpha.1, 1.0.0-0.3.7, 1.0.0-x.7.z.92, 1.0.0-x-y-z.–.
>
> Not only violating the semver but this might break a number of tools.
>
> However, I think we can do a different thing in this case. I do not think
> we strictly need to release the packages when we see doc-only change. What
> we can do - we can simply tag it with *-doc1, *-doc2 tags in the repo and
> add some logic to "skip" such doc-only commits next time when we prepare
> packages.
> Then we can simply skip (automatically) building/releasing/voting doc-only
> packages at all, However we will continue with documentation and only
> release the documentation (using the existing version).
>
> Unlike packages, our documentation is mutable and we can override it.
>
> J.
>
>
>
>
>
>
>
>
> On Tue, Mar 9, 2021 at 7:52 PM Tomasz Urbaszek <tu...@apache.org>
> wrote:
>
>> I'm ok with batch and ad-hoc approach as well as with the frequency.
>>
>> In case of docs-only changes I second what Ash proposed - let's not alter
>> the version. 100% of functionality is the same so users are not affected
>> and there's no need to update.
>>
>> As per testing I agree that E2E testing of all providers should not be
>> necessary to cast +1 vote. The point about ad-hoc releases allows us to
>> release a fix to a provider if users find something that is broken
>> beyond acceptance.
>>
>> Tomek
>>
>>
>> On Tue, 9 Mar 2021 at 11:15, Ash Berlin-Taylor <as...@apache.org> wrote:
>>
>>> For doc-only changes there is one extra thing to decide:
>>>
>>> Should we do it as a patch version (1.0.2 etc) or as a "post" release?
>>> Python packages have this concept:
>>> https://www.python.org/dev/peps/pep-0440/#post-releases
>>>
>>> > Some projects use post-releases to address minor errors in a final
>>> release that do not affect the distributed software (for example,
>>> correcting an error in the release notes).
>>>
>>> So we could update our tooling to support this, or we could just bump
>>> the patch version and release re-release it.
>>>
>>> I have an ever so slight preference for doc only changes to be done this
>>> way, as I think is clearer to users that they don't have to update.
>>>
>>> What does everyone else think?
>>>
>>> -ash
>>>
>>> On Mon, 8 Mar, 2021 at 22:38, Kaxil Naik <ka...@apache.org> wrote:
>>>
>>> Hi all,
>>>
>>> I would like to open the release process of providers up for discussion.
>>> Testing and Voting needs more discussion, the other points are mostly
>>> straight-forward and had an agreement on the last dev call.
>>>
>>> (Backport Providers won't be released after the end of this month - link
>>> <http://apache-airflow-docs.s3-website.eu-central-1.amazonaws.com/docs/apache-airflow/latest/upgrading-to-2.html#support-for-airflow-1-10-x-releases> so
>>> let's ignore them in conversation)
>>>
>>> *Batch vs Ad-hoc*:
>>>
>>> - Release Manager would default to releasing Providers in Batch
>>> - ad-hoc releases are OK (i.e. if there is a critical bug that needs
>>> fixing in a single provider)
>>>
>>> *Frequency:*
>>>
>>> - For Batch release, we will release *every month *(starting of the
>>> month - 1 to 7 most likely)
>>> - Just a note that it generally takes around a week for the vote to
>>> pass even though we have 72 hours minimum period
>>>
>>> *Doc-only changes*
>>>
>>> - When we have doc-only changes for Providers (during
>>> batch-release), we should still release a new version. The majority on the
>>> Dev call had agreed that releasing docs asap is good instead of waiting for
>>> the next release with a code-change.
>>>
>>> *Testing*
>>>
>>> - License and Signature Checks are mandatory (following the ASF
>>> rules)
>>> - For Providers, not all changes require strict testing -- you make
>>> a judgement based on the changes for a particular provider
>>> - Wherever possible community can help test those but not
>>> strictly necessary. *This is where more discussion is needed. *
>>>
>>> *Voting*
>>>
>>> - Automate and create separate voting threads to avoid confusions vs
>>> a single vote where we exclude the providers if someone finds a bug, let's
>>> keep the discussion for this on
>>> https://lists.apache.org/thread.html/r9dcc03840f478669e9bd0dc61f4088b725097da2b48ea274b7f0593e%40%3Cdev.airflow.apache.org%3E
>>>
>>> Regards,
>>> Kaxil
>>>
>>>
>
> --
> +48 660 796 129
>
Re: [DISCUSS] Guidelines for Releasing Providers packages
Posted by Jarek Potiuk <ja...@potiuk.com>.
Fully agree batch and ad-hoc approach as well as with the frequency.
For docs-only changes I think -post release in PyPI is not a good idea. It
does not follow semver (https://semver.org/). The only way you can extend
the number in SEMVER is for pre-releases:
> A pre-release version MAY be denoted by appending a hyphen and a series
of dot separated identifiers immediately following the patch version.
Identifiers MUST comprise only ASCII alphanumerics and hyphens
[0-9A-Za-z-]. Identifiers MUST NOT be empty. Numeric identifiers MUST NOT
include leading zeroes. Pre-release versions have a lower precedence than
the associated normal version. A pre-release version indicates that the
version is unstable and might not satisfy the intended compatibility
requirements as denoted by its associated normal version. Examples:
1.0.0-alpha, 1.0.0-alpha.1, 1.0.0-0.3.7, 1.0.0-x.7.z.92, 1.0.0-x-y-z.–.
Not only violating the semver but this might break a number of tools.
However, I think we can do a different thing in this case. I do not think
we strictly need to release the packages when we see doc-only change. What
we can do - we can simply tag it with *-doc1, *-doc2 tags in the repo and
add some logic to "skip" such doc-only commits next time when we prepare
packages.
Then we can simply skip (automatically) building/releasing/voting doc-only
packages at all, However we will continue with documentation and only
release the documentation (using the existing version).
Unlike packages, our documentation is mutable and we can override it.
J.
On Tue, Mar 9, 2021 at 7:52 PM Tomasz Urbaszek <tu...@apache.org> wrote:
> I'm ok with batch and ad-hoc approach as well as with the frequency.
>
> In case of docs-only changes I second what Ash proposed - let's not alter
> the version. 100% of functionality is the same so users are not affected
> and there's no need to update.
>
> As per testing I agree that E2E testing of all providers should not be
> necessary to cast +1 vote. The point about ad-hoc releases allows us to
> release a fix to a provider if users find something that is broken
> beyond acceptance.
>
> Tomek
>
>
> On Tue, 9 Mar 2021 at 11:15, Ash Berlin-Taylor <as...@apache.org> wrote:
>
>> For doc-only changes there is one extra thing to decide:
>>
>> Should we do it as a patch version (1.0.2 etc) or as a "post" release?
>> Python packages have this concept:
>> https://www.python.org/dev/peps/pep-0440/#post-releases
>>
>> > Some projects use post-releases to address minor errors in a final
>> release that do not affect the distributed software (for example,
>> correcting an error in the release notes).
>>
>> So we could update our tooling to support this, or we could just bump the
>> patch version and release re-release it.
>>
>> I have an ever so slight preference for doc only changes to be done this
>> way, as I think is clearer to users that they don't have to update.
>>
>> What does everyone else think?
>>
>> -ash
>>
>> On Mon, 8 Mar, 2021 at 22:38, Kaxil Naik <ka...@apache.org> wrote:
>>
>> Hi all,
>>
>> I would like to open the release process of providers up for discussion.
>> Testing and Voting needs more discussion, the other points are mostly
>> straight-forward and had an agreement on the last dev call.
>>
>> (Backport Providers won't be released after the end of this month - link
>> <http://apache-airflow-docs.s3-website.eu-central-1.amazonaws.com/docs/apache-airflow/latest/upgrading-to-2.html#support-for-airflow-1-10-x-releases> so
>> let's ignore them in conversation)
>>
>> *Batch vs Ad-hoc*:
>>
>> - Release Manager would default to releasing Providers in Batch
>> - ad-hoc releases are OK (i.e. if there is a critical bug that needs
>> fixing in a single provider)
>>
>> *Frequency:*
>>
>> - For Batch release, we will release *every month *(starting of the
>> month - 1 to 7 most likely)
>> - Just a note that it generally takes around a week for the vote to
>> pass even though we have 72 hours minimum period
>>
>> *Doc-only changes*
>>
>> - When we have doc-only changes for Providers (during batch-release),
>> we should still release a new version. The majority on the Dev call had
>> agreed that releasing docs asap is good instead of waiting for the next
>> release with a code-change.
>>
>> *Testing*
>>
>> - License and Signature Checks are mandatory (following the ASF rules)
>> - For Providers, not all changes require strict testing -- you make a
>> judgement based on the changes for a particular provider
>> - Wherever possible community can help test those but not
>> strictly necessary. *This is where more discussion is needed. *
>>
>> *Voting*
>>
>> - Automate and create separate voting threads to avoid confusions vs
>> a single vote where we exclude the providers if someone finds a bug, let's
>> keep the discussion for this on
>> https://lists.apache.org/thread.html/r9dcc03840f478669e9bd0dc61f4088b725097da2b48ea274b7f0593e%40%3Cdev.airflow.apache.org%3E
>>
>> Regards,
>> Kaxil
>>
>>
--
+48 660 796 129
Re: [DISCUSS] Guidelines for Releasing Providers packages
Posted by Tomasz Urbaszek <tu...@apache.org>.
I'm ok with batch and ad-hoc approach as well as with the frequency.
In case of docs-only changes I second what Ash proposed - let's not alter
the version. 100% of functionality is the same so users are not affected
and there's no need to update.
As per testing I agree that E2E testing of all providers should not be
necessary to cast +1 vote. The point about ad-hoc releases allows us to
release a fix to a provider if users find something that is broken
beyond acceptance.
Tomek
On Tue, 9 Mar 2021 at 11:15, Ash Berlin-Taylor <as...@apache.org> wrote:
> For doc-only changes there is one extra thing to decide:
>
> Should we do it as a patch version (1.0.2 etc) or as a "post" release?
> Python packages have this concept:
> https://www.python.org/dev/peps/pep-0440/#post-releases
>
> > Some projects use post-releases to address minor errors in a final
> release that do not affect the distributed software (for example,
> correcting an error in the release notes).
>
> So we could update our tooling to support this, or we could just bump the
> patch version and release re-release it.
>
> I have an ever so slight preference for doc only changes to be done this
> way, as I think is clearer to users that they don't have to update.
>
> What does everyone else think?
>
> -ash
>
> On Mon, 8 Mar, 2021 at 22:38, Kaxil Naik <ka...@apache.org> wrote:
>
> Hi all,
>
> I would like to open the release process of providers up for discussion.
> Testing and Voting needs more discussion, the other points are mostly
> straight-forward and had an agreement on the last dev call.
>
> (Backport Providers won't be released after the end of this month - link
> <http://apache-airflow-docs.s3-website.eu-central-1.amazonaws.com/docs/apache-airflow/latest/upgrading-to-2.html#support-for-airflow-1-10-x-releases> so
> let's ignore them in conversation)
>
> *Batch vs Ad-hoc*:
>
> - Release Manager would default to releasing Providers in Batch
> - ad-hoc releases are OK (i.e. if there is a critical bug that needs
> fixing in a single provider)
>
> *Frequency:*
>
> - For Batch release, we will release *every month *(starting of the
> month - 1 to 7 most likely)
> - Just a note that it generally takes around a week for the vote to
> pass even though we have 72 hours minimum period
>
> *Doc-only changes*
>
> - When we have doc-only changes for Providers (during batch-release),
> we should still release a new version. The majority on the Dev call had
> agreed that releasing docs asap is good instead of waiting for the next
> release with a code-change.
>
> *Testing*
>
> - License and Signature Checks are mandatory (following the ASF rules)
> - For Providers, not all changes require strict testing -- you make a
> judgement based on the changes for a particular provider
> - Wherever possible community can help test those but not
> strictly necessary. *This is where more discussion is needed. *
>
> *Voting*
>
> - Automate and create separate voting threads to avoid confusions vs a
> single vote where we exclude the providers if someone finds a bug, let's
> keep the discussion for this on
> https://lists.apache.org/thread.html/r9dcc03840f478669e9bd0dc61f4088b725097da2b48ea274b7f0593e%40%3Cdev.airflow.apache.org%3E
>
> Regards,
> Kaxil
>
>
Re: [DISCUSS] Guidelines for Releasing Providers packages
Posted by Ash Berlin-Taylor <as...@apache.org>.
For doc-only changes there is one extra thing to decide:
Should we do it as a patch version (1.0.2 etc) or as a "post" release?
Python packages have this concept:
<https://www.python.org/dev/peps/pep-0440/#post-releases>
> Some projects use post-releases to address minor errors in a final
release that do not affect the distributed software (for example,
correcting an error in the release notes).
So we could update our tooling to support this, or we could just bump
the patch version and release re-release it.
I have an ever so slight preference for doc only changes to be done
this way, as I think is clearer to users that they don't have to update.
What does everyone else think?
-ash
On Mon, 8 Mar, 2021 at 22:38, Kaxil Naik <ka...@apache.org> wrote:
> Hi all,
>
> I would like to open the release process of providers up for
> discussion. Testing and Voting needs more discussion, the other
> points are mostly straight-forward and had an agreement on the last
> dev call.
>
> (Backport Providers won't be released after the end of this month -
> link
> <http://apache-airflow-docs.s3-website.eu-central-1.amazonaws.com/docs/apache-airflow/latest/upgrading-to-2.html#support-for-airflow-1-10-x-releases>
> so let's ignore them in conversation)
>
> *Batch vs Ad-hoc*:
> Release Manager would default to releasing Providers in Batch
> ad-hoc releases are OK (i.e. if there is a critical bug that needs
> fixing in a single provider)
> *Frequency:*
> For Batch release, we will release *every month *(starting of the
> month - 1 to 7 most likely)
> Just a note that it generally takes around a week for the vote to
> pass even though we have 72 hours minimum period
> *Doc-only changes*
> When we have doc-only changes for Providers (during batch-release),
> we should still release a new version. The majority on the Dev call
> had agreed that releasing docs asap is good instead of waiting for
> the next release with a code-change.
> *Testing*
> License and Signature Checks are mandatory (following the ASF rules)
> For Providers, not all changes require strict testing -- you make a
> judgement based on the changes for a particular provider
> Wherever possible community can help test those but not strictly
> necessary. *This is where more discussion is needed. *
> *Voting*
> Automate and create separate voting threads to avoid confusions vs a
> single vote where we exclude the providers if someone finds a bug,
> let's keep the discussion for this on
> <https://lists.apache.org/thread.html/r9dcc03840f478669e9bd0dc61f4088b725097da2b48ea274b7f0593e%40%3Cdev.airflow.apache.org%3E>
> Regards,
> Kaxil
> *
> *