You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Jarek Potiuk <ja...@potiuk.com> on 2021/03/02 19:44:50 UTC
Re: [VOTE] Airflow Providers - release candidates from 2021-02-27
Hey Ryan,
There is no **must** in re-testing it. Providing that you tested it before
with real GSuite account is for me enough of a confirmation ;).
J.
On Sun, Feb 28, 2021 at 10:00 PM Abdur-Rahmaan Janhangeer <
arj.python@gmail.com> wrote:
> Salutes for having a GSuite account just for the functionality 👍👍👍
>
> On Mon, 1 Mar 2021, 00:05 Ryan Hatter, <ry...@gmail.com> wrote:
>
>> I canceled my GSuite account when my PR for the gdrive to gcs operator
>> was approved & merged. Could anyone maybe help me ensure correct
>> functionality?
>>
>>
>> On Feb 27, 2021, at 08:48, Jarek Potiuk <ja...@potiuk.com> wrote:
>>
>>
>> I created issue, where we will track the status of tests for the
>> providers (again - it is experiment - but I'd really love to get feedback
>> on the new providers from those who contributed):
>> https://github.com/apache/airflow/issues/14511
>>
>> On Sat, Feb 27, 2021 at 4:28 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>>
>>>
>>> Hey all,
>>>
>>> I have just cut the new wave Airflow Providers packages. This email is
>>> calling a vote on the release,
>>> which will last for 72 hours + day for the weekend - which means that
>>> it will end on Wed 3 Mar 15:59:34 CET 2021.
>>>
>>> Consider this my (binding) +1.
>>>
>>> *KIND REQUEST*
>>>
>>> There was a recent discussion about test quality of the providers and I
>>> would like to try to address it, still keeping the batch release process
>>> every 3 weeks.
>>>
>>> We need a bit of help from the community. I have a kind request to the
>>> authors of fixes and new features. I group the providers into those that
>>> likely need more testing, and those that do not. I also added names of
>>> those who submitted the changes and are most likely to be able to verify if
>>> the RC packages are solving the problems/adding features.
>>>
>>> This is a bit of experiment (apologies for calling out) - but if we
>>> find that it works, we can automate that. I will create a separate Issue in
>>> Github where you will be able to "tick" the boxes for those providers which
>>> they are added to. It would not be a blocker if not tested, but it will be
>>> a great help if you could test the new RC provider and see if it works as
>>> expected according to your changes.
>>>
>>> Providers with new features and fixes - likely needs some testing.:
>>>
>>> * *amazon* : Cristòfol Torrens, Ruben Laguna, Arati Nagmal, Ivica
>>> Kolenkaš, JavierLopezT
>>> * *apache.druid*: Xinbin Huang
>>> * *apache.spark*: Igor Khrol
>>> * *cncf.kubernetes*: jpyen, Ash Berlin-Taylor, Daniel Imberman
>>> * *google*: Vivek Bhojawala, Xinbin Huang, Pak Andrey, uma66, Ryan
>>> Yuan, morrme, Sam Wheating, YingyingPeng22, Ryan Hatter,Tobiasz Kędzierski
>>> * *jenkins*: Maxim Lisovsky
>>> * *microsift.azure*: flvndh, yyu
>>> * *mysql*: Constantino Schillebeeckx
>>> * *qubole*: Xinbin Huang
>>> * *salesforce*: Jyoti Dhiman
>>> * *slack*: Igor Khrol
>>> * *tableau*: Jyoti Dhiman
>>> * *telegram*: Shekhar Sing, Adil Khashtamov
>>>
>>> Providers with doc only changes (no need to test):
>>>
>>> * apache-beam
>>> * apache-hive
>>> * dingding
>>> * docker
>>> * elasticsearch
>>> * exasol
>>> * http
>>> * neo4j
>>> * openfaas
>>> * papermill
>>> * presto
>>> * sendgrid
>>> * sftp
>>> * snowflake
>>> * sqlite
>>> * ssh
>>> *
>>>
>>>
>>> Airflow Providers are available at:
>>> https://dist.apache.org/repos/dist/dev/airflow/providers/
>>>
>>> *apache-airflow-providers-<PROVIDER>-*-bin.tar.gz* are the binary
>>> Python "sdist" release - they are also official "sources" for the
>>> provider packages.
>>>
>>> *apache_airflow_providers_<PROVIDER>-*.whl are the binary
>>> Python "wheel" release.
>>>
>>> The test procedure for PMC members who would like to test the RC
>>> candidates are described in
>>>
>>> https://github.com/apache/airflow/blob/master/dev/README_RELEASE_PROVIDER_PACKAGES.md#verify-the-release-by-pmc-members
>>>
>>> and for Contributors:
>>>
>>>
>>> https://github.com/apache/airflow/blob/master/dev/README_RELEASE_PROVIDER_PACKAGES.md#verify-by-contributors
>>>
>>>
>>> Public keys are available at:
>>> https://dist.apache.org/repos/dist/release/airflow/KEYS
>>>
>>> Please vote accordingly:
>>>
>>> [ ] +1 approve
>>> [ ] +0 no opinion
>>> [ ] -1 disapprove with the reason
>>>
>>>
>>> Only votes from PMC members are binding, but members of the community are
>>> encouraged to test the release and vote with "(non-binding)".
>>>
>>> Please note that the version number excludes the 'rcX' string.
>>> This will allow us to rename the artifact without modifying
>>> the artifact checksums when we actually release.
>>>
>>>
>>> Each of the packages contains a link to the detailed changelog. The
>>> changelogs are moved to the official airflow documentation:
>>> https://github.com/apache/airflow-site/<TODO COPY LINK TO BRANCH>
>>>
>>> <PASTE ANY HIGH-LEVEL DESCRIPTION OF THE CHANGES HERE!>
>>>
>>>
>>> Note the links to documentation from PyPI packages are not working until
>>> we merge
>>> the changes to airflow site after releasing the packages officially.
>>>
>>> https://pypi.org/project/apache-airflow-providers-amazon/1.2.0rc1/
>>> https://pypi.org/project/apache-airflow-providers-apache-beam/1.0.1rc1/
>>> https://pypi.org/project/apache-airflow-providers-apache-druid/1.1.0rc1/
>>> https://pypi.org/project/apache-airflow-providers-apache-hive/1.0.2rc1/
>>> https://pypi.org/project/apache-airflow-providers-apache-spark/1.0.2rc1/
>>>
>>> https://pypi.org/project/apache-airflow-providers-cncf-kubernetes/1.0.2rc1/
>>> https://pypi.org/project/apache-airflow-providers-dingding/1.0.2rc1/
>>> https://pypi.org/project/apache-airflow-providers-docker/1.0.2rc1/
>>> https://pypi.org/project/apache-airflow-providers-elasticsearch/1.0.2rc1/
>>> https://pypi.org/project/apache-airflow-providers-exasol/1.1.1rc1/
>>> https://pypi.org/project/apache-airflow-providers-google/2.1.0rc1/
>>> https://pypi.org/project/apache-airflow-providers-http/1.1.1rc1/
>>> https://pypi.org/project/apache-airflow-providers-jenkins/1.1.0rc1/
>>>
>>> https://pypi.org/project/apache-airflow-providers-microsoft-azure/1.2.0rc1/
>>> https://pypi.org/project/apache-airflow-providers-mysql/1.0.2rc1/
>>> https://pypi.org/project/apache-airflow-providers-neo4j/1.0.1rc1/
>>> https://pypi.org/project/apache-airflow-providers-openfaas/1.1.1rc1/
>>> https://pypi.org/project/apache-airflow-providers-papermill/1.0.2rc1/
>>> https://pypi.org/project/apache-airflow-providers-presto/1.0.2rc1/
>>> https://pypi.org/project/apache-airflow-providers-qubole/1.0.2rc1/
>>> https://pypi.org/project/apache-airflow-providers-salesforce/2.0.0rc1/
>>> https://pypi.org/project/apache-airflow-providers-sendgrid/1.0.2rc1/
>>> https://pypi.org/project/apache-airflow-providers-sftp/1.1.1rc1/
>>> https://pypi.org/project/apache-airflow-providers-slack/3.0.0rc1/
>>> https://pypi.org/project/apache-airflow-providers-snowflake/1.1.1rc1/
>>> https://pypi.org/project/apache-airflow-providers-sqlite/1.0.2rc1/
>>> https://pypi.org/project/apache-airflow-providers-ssh/1.2.0rc1/
>>> https://pypi.org/project/apache-airflow-providers-tableau/1.0.0rc1/
>>> https://pypi.org/project/apache-airflow-providers-telegram/1.0.2rc1/
>>>
>>> Cheers,
>>> J.
>>>
>>>
>>>
>>> --
>>> +48 660 796 129
>>>
>>
>>
>> --
>> +48 660 796 129
>>
>>
--
+48 660 796 129
Re: Voting and release process for providers
Posted by Jarek Potiuk <ja...@potiuk.com>.
Based on the experiment I've done I'd update it to make it similar to the
follow up email I sent here:
https://lists.apache.org/thread.html/reac8871d8ad55d5c0c33819e8a10b0c38cba09297bde84a89adb102c%40%3Cdev.airflow.apache.org%3E
My goal is to streamline it - so that neither release manager, nor PMCs,
nor users spend too much unnecessary overhead time for the process -
automation of information and grouping is key to that as this is
something we will continue doing for many months ahead - every month
repeating the mantra.
* Providers list with information if this is a bugfix or feature ( I assume
we could get away with doc updates with my proposal of updating
documentation only, not packages)
* List of people and links to #PRs of the changes in each provider
* if applicable, assessment of the release manager.
Something along the lines of (wording can be probably improved):
-------------------------
Please respond with one of:
* +1/-1/0 [all] : if you vote on all packages
* +1/-1/0 [ package1 package2 ]: if you vote on selected packages
* +1/-1/0 [ all but package1 package2 ]: if you want to exclude certain
packages from the vote
Here, the list of packages we are voting on. You can cast your vote
individually for each package, "bulk" vote on all the providers, or
specifically exclude providers from your vote:
* [ ] amazon 1.2.0
* [ ] google 3.0.0
* [ ] dingding 1.1.3
* [ ] datadog 2.0.4
.....
The assessment of the release manager is that amazon and google needs
extensive testing, dingding, datadog contain trivial fixes that are likely
to be OK without extensive testing.
The current status of the provider tests is kept in
https://github.com/apache/airflow/issues/14670
Details of the changes:
* amazon 1.2.0
Features:
* #1234 - this by @ashb
* #1223 - that by @kaxil
Bugfixes:
* #1234 - this by @potiuk
* #1244 that
* google ....
Breaking changes:
* ,,,,,
---------------------------
J.
On Wed, Mar 10, 2021 at 10:16 AM Ash Berlin-Taylor <as...@apache.org> wrote:
> Our approach of having 60+ subpackages is I think fairly unique in ASF
> terms, where most projects just have one package that they release, so the
> ASF guidance doesn't consider this situation -- we are on our own to come
> up with something that works for us.
>
> The only other project I know that has as many sub-packages is Maven, and
> they do one email = one package = one vote
> https://lists.apache.org/list.html?dev@maven.apache.org:gte=1d:VOTE (they
> don't batch either)
>
> For example these are started on the same day
>
>
> https://lists.apache.org/thread.html/r92026b74d0c32b2d3dc06794ec071375d6a135057b41b13359263003%40%3Cdev.maven.apache.org%3E
>
> https://lists.apache.org/thread.html/r6cd5e53070bc033fae07d3e236c7c0cbdf956da358759a46919821ac%40%3Cdev.maven.apache.org%3E
>
> For me what we are silently (and wrongly) assuming 1 mail = 1 VOTE. But I
> do not believe anywhere in all ASF documentation there is such a
> requirement.
>
>
> My counter argument to this point: the ASF guidance talks about "Votes on
> whether *a* package is ready to be released" (emphasis mine). But
> ultimately I think the guidance is just that -- guidance and we should just
> come up with and write down how we want to operate here.
>
> Could you give an snippet/example of what 20 votes in one email would look
> like? Are you thinking of the same email we send right now or something
> different?
>
>
> On Wed, 10 Mar, 2021 at 04:02, Jarek Potiuk <ja...@potiuk.com> wrote:
>
> TL;DR; My very, very strong preference goes for one email which is
> effectively N votes for N packages we release. I actually think we are not
> able to efficiently follow up and manage 1-vote for email for ~20 or so
> packages we are going to release every month.
>
> I think we should just refer back to what ASF wrote about it.
> https://www.apache.org/foundation/voting.html
>
> > VOTES ON PACKAGE RELEASES
> > Votes on whether a package is ready to be released use majority approval
> -- i.e. at least three PMC members must vote affirmatively for release, and
> there must be more positive than negative votes. Releases may not be
> vetoed. Generally the community will cancel the release vote if anyone
> identifies serious problems, but in most cases the ultimate decision, lies
> with the individual serving as release manager. The specifics of the
> process may vary from project to project, but the 'minimum
> > quorum of three +1 votes' rule is universal.
>
> "Generally the community will cancel the release vote if anyone identifies
> serious problems, but in most cases the ultimate decision, lies with the
> individual serving as release manager." - but as I read it - it's NOT the
> PMC, but the community decides when to cancel release and the ultimate
> decision lies in the hands of the release manager.
> This has happened in the past with Airflow many times, that the release
> manager usually Kaxil decided on canceling the release vote.
>
> And yes - there is no mentioning of partial "canceling". But, this does
> not mean that there MUST be a single mail per vote either. I think we can
> have a single email and a N votes that cover all packages in a single
> email. And you can vote in bulk on all packages. And some users might vote
> only on some of the packages. For me this is purely for the efficiency of
> process and PMCs responding to the votes.
>
> Imagine we have 20 packages to send regularly (guesstimate), following up
> 20 messages to vote will be quite overwhelming - borderline ridiculous in
> fact. I am talking about following up and counting, not sending. Sending is
> easy.
> We often have difficulty chasing PMCs to respond to one email, and the job
> of tracking how many responses were sent to each of the 20 emails and
> following up on each of them will make the release manager's life
> miserable.
> First two weeks after sending votes, the release manager will have to
> chase 20 emails, follow up, and count the votes for every single release. I
> would certainly reject such a task as a release manager. It is not easy to
> automate at all and is a lot of very stupid and tim-consuming overhead on
> top of the work done already by release manager.
>
> I do not really think the analogy of ice cream and dogs is appropriate (at
> all). It's misleading actually. We are not voting on different things. It's
> all about cancelling some votes out of many that we agreed to in bulk at
> most. But not about changing the votes. This can already be done if the
> community raises an issue - with the ultimate decision of the release
> manager - so nothing changes here vs. the current Apache voting process.
> For me it's only the question about granularity of the votes in one email
> and whether we can cast a "bulk" N votes by single +1. I think we can.
>
> For me what we are silently (and wrongly) assuming 1 mail = 1 VOTE. But I
> do not believe anywhere in all ASF documentation there is such a
> requirement. We have example "templates" for emails but I do not know if
> anywhere this 1-1 relationship (email = Vote) is mentioned. I'd love to see
> it if it is. For me as long as it is clear what we are voting on and what
> eventually gets cancelled and we summaarise it in the "RESULT" email - we
> should be good with having N votes in a single email.
>
> So for me "bulk" voting looks as follows:
>
> 1) we send 20 packages and start 20 votes in one email.
> 2) Anyone - user, PMC could respond +1 (for all) or +1 (some) or +1 (-
> this packages I tested which do not work).
> 3) Community (ultimately release manager) decides which of those votes to
> cancel - if any (providing that all of the votes have +3 votes from PMC
> members).
> 4) Releasing packages can still be done in waves - if we see after 3 days
> that we can to release 10 out of the 20 package (but 10 has just +2 PMC
> votes) - the release manager can still release the 10 and continue voting
> on the remaining 10 (and even eventually cancel them).
>
> Also a few words about testing those 20 packages.
>
> Regarding testing - yes the packages MUST be tested by PMC members. This
> is a must. But testing means in this case - bulk installing the packages on
> top of airflow and see if the providers have been installed and shown in
> `airlow providers list` with the correct version. I believe this is more
> than enough to fulfill the "test" criteria when paired with user testing of
> new/big functionalities.
> This is (or should be - I followed it for sure) already part of the
> process (alongside signing/checksum verification). And with svn checkout
> where you can pull the whole folder to your machine, this can be done
> rather quickly.
> As Kaxil proposed in the separate thread - also a judgement of release
> manager can be used to see whether some additional testing is necessary.
> And with single email and Github Issue we can very efficiently organise
> the users to test their changes (even automate it).
> I have shown during the last two weeks that this is not only doable, but
> that our users and contributors generally respond very well to such
> requests.
> Those contributors await their changes to be released and they also want
> to make sure it is tested before they get to the hands of the users.
>
> J.
>
>
>
>
> On Tue, Mar 9, 2021 at 12:40 PM Kaxil Naik <ka...@gmail.com> wrote:
>
>> +1 -- I think users can pick and choose which providers to test and vote
>> on. And since each vote would be in a separate email, a single provider bug
>> in a particular provider won't derail conversation.
>>
>> Regards,
>> Kaxil
>>
>> On Tue, Mar 9, 2021, 11:38 Kaxil Naik <ka...@gmail.com> wrote:
>>
>>> +1
>>>
>>> On Sat, Mar 6, 2021, 21:33 Ash Berlin-Taylor <as...@apache.org> wrote:
>>>
>>>> Just for absolutely clarity here: my -1 has nothing to do with the vote
>>>> on the release, merely against adopting the proposed policy after this
>>>> release/current set of votes.
>>>>
>>>> -a
>>>>
>>>> On 4 March 2021 11:04:13 GMT, Ash Berlin-Taylor <as...@apache.org> wrote:
>>>>>
>>>>> (Split out proposal from voting thread here
>>>>> https://lists.apache.org/thread.html/r47daa0e594eb01c8196a96b62c40d5282b25c41ca1520781934e9236%40%3Cdev.airflow.apache.org%3E
>>>>> .)
>>>>>
>>>>> I don't like the idea of changing what a vote means mid way through.
>>>>> To take an extreme case:
>>>>>
>>>>> "+1 if you like ice cream"
>>>>>
>>>>> *people vote*
>>>>>
>>>>> "Vote now changed to +1 if you kick dogs".
>>>>>
>>>>> Now clearly that is not what you are proposing, but changing what the
>>>>> vote is on at all opens a slippery slope and I'd rather we didn't
>>>>> continue/codify this precedent.
>>>>>
>>>>> *This gets a -1 from me on adopting this part of the proposal.*
>>>>>
>>>>> Not changing the vote is also why I favour many smaller
>>>>> releases/votes: a problem with one doesn't block the others, and additional
>>>>> I feel it is easier for non-core contributors to get involved and test just
>>>>> the provider they are interested in, without feeling daunted that they
>>>>> would have to test all the rest to cast their vote.
>>>>>
>>>>> Much of the release process is already automated, up-to-and including
>>>>> sending emails (dev/send_email.py -- may need tweaking for provider release
>>>>> votes.) so this is can be achieved with little extra work to the release
>>>>> manager.
>>>>>
>>>>> To be clear, I do not have a problem with batch releases and batch
>>>>> votes (even if it is not my first choice, and I would rather N parallel
>>>>> votes as default.)
>>>>>
>>>>> -ash
>>>>>
>>>>>
>>>>> On Wed, 3 Mar, 2021 at 22:44, Jarek Potiuk <ja...@potiuk.com> wrote:
>>>>>
>>>>> I just wanted to use the opportunity to describe the current process
>>>>> for deciding providers, because apparently not everyone is aware that we
>>>>> already have an established and documented process for provider's releases
>>>>> that was discussed on the devlist and it is documented in our release
>>>>> process description.
>>>>>
>>>>> Possibly we will extract it into a separate policy and maybe discuss
>>>>> some aspects of it (the discussion was raised today at the dev call of ours
>>>>> but I wanted to make sure that we are all referring to the same "starting
>>>>> point" and the "process" I based my recent actions on.
>>>>>
>>>>> *1) Batching the providers as the default*
>>>>>
>>>>> The decision on when to release and particularly preference for
>>>>> releasing providers in batches is described in the
>>>>> https://github.com/apache/airflow/blob/master/dev/README_RELEASE_PROVIDER_PACKAGES.md#decide-when-to-release
>>>>>
>>>>>
>>>>> > Decide when to release
>>>>> > You can release provider packages separately from the main Airflow
>>>>> on an ad-hoc basis, whenever we find that
>>>>> a given provider needs to be released - due to new features or due to
>>>>> bug fixes
>>>>> You can release each provider package separately, but due to voting
>>>>> and release overhead we try to group
>>>>> releases of provider packages together.
>>>>>
>>>>> *2) Possibility of excluding certain packages from the release.*
>>>>>
>>>>> The possibility of excluding certain packages which (for whatever
>>>>> reason) we decide to remove at the discretion of release manager is
>>>>> described here:
>>>>> https://github.com/apache/airflow/blob/master/dev/README_RELEASE_PROVIDER_PACKAGES.md#prepare-voting-email-for-providers-release-candidate
>>>>> <https://github.com/apache/airflow/blob/master/dev/README_RELEASE_PROVIDER_PACKAGESmd#prepare-voting-email-for-providers-release-candidate>
>>>>>
>>>>> Prepare voting email for Providers release candidate
>>>>> ...
>>>>>
>>>>> > Due to the nature of packages, not all packages have to be released
>>>>> as convenience packages in the final release. During the voting process the
>>>>> voting PMCs might decide to exclude certain packages from the release if
>>>>> some critical problems have been found in some packages.
>>>>>
>>>>> And the process of removing it is part of the described release
>>>>> process:
>>>>>
>>>>> > In case you decided to remove some of the packages. remove them from
>>>>> dist folder now:
>>>>>
>>>>> > ls dist/*<provider>*
>>>>> > rm dist/*<provider>*
>>>>>
>>>>> The issue of excluding certain packages have been discussed in this
>>>>> thread in the mailing list :
>>>>> https://lists.apache.org/thread.html/rc620a8a503cc7b14850c0a2a1fca1f6051d08a7e3e6d2cbdeb691dda%40%3Cdev.airflow.apache.org%3E
>>>>> - where we had a -1 veto from a PMC member on the whole batch of providers
>>>>> where we found a cncf.kubrenetes and google providers had critical
>>>>> problems.
>>>>>
>>>>> We discussed it then and the two PMC members proposed a solution that
>>>>> was not objected to by anyone in the VOTE thread - to remove the packages
>>>>> from the batch.
>>>>>
>>>>> I continued this in the continuation of the voting thread
>>>>> https://lists.apache.org/thread.html/r752c5d5171de4ff626663d30e1d50c4b0d2994f66bf8918d816dabd8%40%3Cdev.airflow.apache.org%3E
>>>>> with the following message which specifically pointed out to my proposal
>>>>> where I specifically linked to the message above and asked for comments.
>>>>>
>>>>> > As discussed before: -1 on a single provider does not invalidate the
>>>>> whole
>>>>> vote (from
>>>>>
>>>>> https://github.com/apache/airflow/tree/master/dev#vote-and-verify-the-backport-providers-release-candidate
>>>>> ):
>>>>>
>>>>> > "Due to the nature of backport packages, not all packages have to be
>>>>> released as convenience packages in the final release. During the
>>>>> voting
>>>>> process the voting PMCs might decide to exclude certain packages from
>>>>> the
>>>>> release if some critical problems have been found in some packages."
>>>>>
>>>>> > We will merge the fix and most likely release a new google package
>>>>> right
>>>>> after this one. Looking at the super-localized problem here my current
>>>>> decision will be to release 2020.10.29 'google package" - together with
>>>>> other packages and release 2020.11.01 (or smth) - but only the google
>>>>> one -
>>>>> right after we merge the fix.
>>>>>
>>>>> > Any comments to that?
>>>>>
>>>>>
>>>>>
>>>>>
>
> --
> +48 660 796 129
>
>
--
+48 660 796 129
Re: Voting and release process for providers
Posted by Ash Berlin-Taylor <as...@apache.org>.
Our approach of having 60+ subpackages is I think fairly unique in ASF
terms, where most projects just have one package that they release, so
the ASF guidance doesn't consider this situation -- we are on our own
to come up with something that works for us.
The only other project I know that has as many sub-packages is Maven,
and they do one email = one package = one vote
<https://lists.apache.org/list.html?dev@maven.apache.org:gte=1d:VOTE>
(they don't batch either)
For example these are started on the same day
<https://lists.apache.org/thread.html/r92026b74d0c32b2d3dc06794ec071375d6a135057b41b13359263003%40%3Cdev.maven.apache.org%3E>
<https://lists.apache.org/thread.html/r6cd5e53070bc033fae07d3e236c7c0cbdf956da358759a46919821ac%40%3Cdev.maven.apache.org%3E>
> For me what we are silently (and wrongly) assuming 1 mail = 1 VOTE.
> But I do not believe anywhere in all ASF documentation there is such
> a requirement.
My counter argument to this point: the ASF guidance talks about "Votes
on whether *a* package is ready to be released" (emphasis mine). But
ultimately I think the guidance is just that -- guidance and we should
just come up with and write down how we want to operate here.
Could you give an snippet/example of what 20 votes in one email would
look like? Are you thinking of the same email we send right now or
something different?
On Wed, 10 Mar, 2021 at 04:02, Jarek Potiuk <ja...@potiuk.com> wrote:
> TL;DR; My very, very strong preference goes for one email which is
> effectively N votes for N packages we release. I actually think we
> are not able to efficiently follow up and manage 1-vote for email for
> ~20 or so packages we are going to release every month.
>
> I think we should just refer back to what ASF wrote about it.
> <https://www.apache.org/foundation/voting.html>
>
> > VOTES ON PACKAGE RELEASES
> > Votes on whether a package is ready to be released use majority
> approval -- i.e. at least three PMC members must vote affirmatively
> for release, and there must be more positive than negative votes.
> Releases may not be vetoed. Generally the community will cancel the
> release vote if anyone identifies serious problems, but in most cases
> the ultimate decision, lies with the individual serving as release
> manager. The specifics of the process may vary from project to
> project, but the 'minimum
> > quorum of three +1 votes' rule is universal.
>
> "Generally the community will cancel the release vote if anyone
> identifies serious problems, but in most cases the ultimate decision,
> lies with the individual serving as release manager." - but as I read
> it - it's NOT the PMC, but the community decides when to cancel
> release and the ultimate decision lies in the hands of the release
> manager.
> This has happened in the past with Airflow many times, that the
> release manager usually Kaxil decided on canceling the release vote.
>
> And yes - there is no mentioning of partial "canceling". But, this
> does not mean that there MUST be a single mail per vote either. I
> think we can have a single email and a N votes that cover all
> packages in a single email. And you can vote in bulk on all packages.
> And some users might vote only on some of the packages. For me this
> is purely for the efficiency of process and PMCs responding to the
> votes.
>
> Imagine we have 20 packages to send regularly (guesstimate),
> following up 20 messages to vote will be quite overwhelming -
> borderline ridiculous in fact. I am talking about following up and
> counting, not sending. Sending is easy.
> We often have difficulty chasing PMCs to respond to one email, and
> the job of tracking how many responses were sent to each of the 20
> emails and following up on each of them will make the release
> manager's life miserable.
> First two weeks after sending votes, the release manager will have to
> chase 20 emails, follow up, and count the votes for every single
> release. I would certainly reject such a task as a release manager.
> It is not easy to automate at all and is a lot of very stupid and
> tim-consuming overhead on top of the work done already by release
> manager.
>
> I do not really think the analogy of ice cream and dogs is
> appropriate (at all). It's misleading actually. We are not voting on
> different things. It's all about cancelling some votes out of many
> that we agreed to in bulk at most. But not about changing the votes.
> This can already be done if the community raises an issue - with the
> ultimate decision of the release manager - so nothing changes here
> vs. the current Apache voting process. For me it's only the question
> about granularity of the votes in one email and whether we can cast a
> "bulk" N votes by single +1. I think we can.
>
> For me what we are silently (and wrongly) assuming 1 mail = 1 VOTE.
> But I do not believe anywhere in all ASF documentation there is such
> a requirement. We have example "templates" for emails but I do not
> know if anywhere this 1-1 relationship (email = Vote) is mentioned.
> I'd love to see it if it is. For me as long as it is clear what we
> are voting on and what eventually gets cancelled and we summaarise it
> in the "RESULT" email - we should be good with having N votes in a
> single email.
>
> So for me "bulk" voting looks as follows:
>
> 1) we send 20 packages and start 20 votes in one email.
> 2) Anyone - user, PMC could respond +1 (for all) or +1 (some) or +1
> (- this packages I tested which do not work).
> 3) Community (ultimately release manager) decides which of those
> votes to cancel - if any (providing that all of the votes have +3
> votes from PMC members).
> 4) Releasing packages can still be done in waves - if we see after 3
> days that we can to release 10 out of the 20 package (but 10 has just
> +2 PMC votes) - the release manager can still release the 10 and
> continue voting on the remaining 10 (and even eventually cancel them).
>
> Also a few words about testing those 20 packages.
>
> Regarding testing - yes the packages MUST be tested by PMC members.
> This is a must. But testing means in this case - bulk installing the
> packages on top of airflow and see if the providers have been
> installed and shown in `airlow providers list` with the correct
> version. I believe this is more than enough to fulfill the "test"
> criteria when paired with user testing of new/big functionalities.
> This is (or should be - I followed it for sure) already part of the
> process (alongside signing/checksum verification). And with svn
> checkout where you can pull the whole folder to your machine, this
> can be done rather quickly.
> As Kaxil proposed in the separate thread - also a judgement of
> release manager can be used to see whether some additional testing is
> necessary.
> And with single email and Github Issue we can very efficiently
> organise the users to test their changes (even automate it).
> I have shown during the last two weeks that this is not only doable,
> but that our users and contributors generally respond very well to
> such requests.
> Those contributors await their changes to be released and they also
> want to make sure it is tested before they get to the hands of the
> users.
>
> J.
>
>
>
>
> On Tue, Mar 9, 2021 at 12:40 PM Kaxil Naik <kaxilnaik@gmail.com
> <ma...@gmail.com>> wrote:
>> +1 -- I think users can pick and choose which providers to test and
>> vote on. And since each vote would be in a separate email, a single
>> provider bug in a particular provider won't derail conversation.
>>
>> Regards,
>> Kaxil
>>
>> On Tue, Mar 9, 2021, 11:38 Kaxil Naik <kaxilnaik@gmail.com
>> <ma...@gmail.com>> wrote:
>>> +1
>>>
>>> On Sat, Mar 6, 2021, 21:33 Ash Berlin-Taylor <ash@apache.org
>>> <ma...@apache.org>> wrote:
>>>> Just for absolutely clarity here: my -1 has nothing to do with the
>>>> vote on the release, merely against adopting the proposed policy
>>>> after this release/current set of votes.
>>>>
>>>> -a
>>>>
>>>> On 4 March 2021 11:04:13 GMT, Ash Berlin-Taylor <ash@apache.org
>>>> <ma...@apache.org>> wrote:
>>>>> (Split out proposal from voting thread here
>>>>> <https://lists.apache.org/thread.html/r47daa0e594eb01c8196a96b62c40d5282b25c41ca1520781934e9236%40%3Cdev.airflow.apache.org%3E>.)
>>>>>
>>>>> I don't like the idea of changing what a vote means mid way
>>>>> through. To take an extreme case:
>>>>>
>>>>> "+1 if you like ice cream"
>>>>>
>>>>> *people vote*
>>>>>
>>>>> "Vote now changed to +1 if you kick dogs".
>>>>>
>>>>> Now clearly that is not what you are proposing, but changing what
>>>>> the vote is on at all opens a slippery slope and I'd rather we
>>>>> didn't continue/codify this precedent.
>>>>>
>>>>> *This gets a -1 from me on adopting this part of the proposal.*
>>>>>
>>>>> Not changing the vote is also why I favour many smaller
>>>>> releases/votes: a problem with one doesn't block the others, and
>>>>> additional I feel it is easier for non-core contributors to get
>>>>> involved and test just the provider they are interested in,
>>>>> without feeling daunted that they would have to test all the rest
>>>>> to cast their vote.
>>>>>
>>>>> Much of the release process is already automated, up-to-and
>>>>> including sending emails (dev/send_email.py -- may need tweaking
>>>>> for provider release votes.) so this is can be achieved with
>>>>> little extra work to the release manager.
>>>>>
>>>>> To be clear, I do not have a problem with batch releases and
>>>>> batch votes (even if it is not my first choice, and I would
>>>>> rather N parallel votes as default.)
>>>>>
>>>>> -ash
>>>>>
>>>>>
>>>>> On Wed, 3 Mar, 2021 at 22:44, Jarek Potiuk <jarek@potiuk.com
>>>>> <ma...@potiuk.com>> wrote:
>>>>>> I just wanted to use the opportunity to describe the current
>>>>>> process for deciding providers, because apparently not everyone
>>>>>> is aware that we already have an established and documented
>>>>>> process for provider's releases that was discussed on the
>>>>>> devlist and it is documented in our release process description.
>>>>>>
>>>>>> Possibly we will extract it into a separate policy and maybe
>>>>>> discuss some aspects of it (the discussion was raised today at
>>>>>> the dev call of ours but I wanted to make sure that we are all
>>>>>> referring to the same "starting point" and the "process" I based
>>>>>> my recent actions on.
>>>>>>
>>>>>> *1) Batching the providers as the default*
>>>>>>
>>>>>> The decision on when to release and particularly preference for
>>>>>> releasing providers in batches is described in the
>>>>>> <https://github.com/apache/airflow/blob/master/dev/README_RELEASE_PROVIDER_PACKAGES.md#decide-when-to-release>
>>>>>>
>>>>>> > Decide when to release
>>>>>> > You can release provider packages separately from the main
>>>>>> Airflow on an ad-hoc basis, whenever we find that
>>>>>> a given provider needs to be released - due to new features or
>>>>>> due to bug fixes
>>>>>> You can release each provider package separately, but due to
>>>>>> voting and release overhead we try to group
>>>>>> releases of provider packages together.
>>>>>>
>>>>>> *2) Possibility of excluding certain packages from the release.*
>>>>>>
>>>>>> The possibility of excluding certain packages which (for
>>>>>> whatever reason) we decide to remove at the discretion of
>>>>>> release manager is described here:
>>>>>> https://github.com/apache/airflow/blob/master/dev/README_RELEASE_PROVIDER_PACKAGES.md#prepare-voting-email-for-providers-release-candidate
>>>>>> <https://github.com/apache/airflow/blob/master/dev/README_RELEASE_PROVIDER_PACKAGESmd#prepare-voting-email-for-providers-release-candidate>
>>>>>>
>>>>>> Prepare voting email for Providers release candidate
>>>>>> ...
>>>>>>
>>>>>> > Due to the nature of packages, not all packages have to be
>>>>>> released as convenience packages in the final release. During
>>>>>> the voting process the voting PMCs might decide to exclude
>>>>>> certain packages from the release if some critical problems have
>>>>>> been found in some packages.
>>>>>>
>>>>>> And the process of removing it is part of the described release
>>>>>> process:
>>>>>>
>>>>>> > In case you decided to remove some of the packages. remove
>>>>>> them from dist folder now:
>>>>>>
>>>>>> > ls dist/*<provider>*
>>>>>> > rm dist/*<provider>*
>>>>>>
>>>>>> The issue of excluding certain packages have been discussed in
>>>>>> this thread in the mailing list :
>>>>>> <https://lists.apache.org/thread.html/rc620a8a503cc7b14850c0a2a1fca1f6051d08a7e3e6d2cbdeb691dda%40%3Cdev.airflow.apache.org%3E>
>>>>>> - where we had a -1 veto from a PMC member on the whole batch
>>>>>> of providers where we found a cncf.kubrenetes and google
>>>>>> providers had critical problems.
>>>>>>
>>>>>> We discussed it then and the two PMC members proposed a solution
>>>>>> that was not objected to by anyone in the VOTE thread - to
>>>>>> remove the packages from the batch.
>>>>>>
>>>>>> I continued this in the continuation of the voting thread
>>>>>> <https://lists.apache.org/thread.html/r752c5d5171de4ff626663d30e1d50c4b0d2994f66bf8918d816dabd8%40%3Cdev.airflow.apache.org%3E>
>>>>>> with the following message which specifically pointed out to my
>>>>>> proposal where I specifically linked to the message above and
>>>>>> asked for comments.
>>>>>>
>>>>>> > As discussed before: -1 on a single provider does not
>>>>>> invalidate the whole
>>>>>> vote (from
>>>>>> <https://github.com/apache/airflow/tree/master/dev#vote-and-verify-the-backport-providers-release-candidate>
>>>>>> ):
>>>>>>
>>>>>> > "Due to the nature of backport packages, not all packages have
>>>>>> to be
>>>>>> released as convenience packages in the final release. During
>>>>>> the voting
>>>>>> process the voting PMCs might decide to exclude certain packages
>>>>>> from the
>>>>>> release if some critical problems have been found in some
>>>>>> packages."
>>>>>>
>>>>>> > We will merge the fix and most likely release a new google
>>>>>> package right
>>>>>> after this one. Looking at the super-localized problem here my
>>>>>> current
>>>>>> decision will be to release 2020.10.29 'google package" -
>>>>>> together with
>>>>>> other packages and release 2020.11.01 (or smth) - but only the
>>>>>> google one -
>>>>>> right after we merge the fix.
>>>>>>
>>>>>> > Any comments to that?
>>>>>
>>>>>
>
>
> --
> +48 660 796 129
Re: Voting and release process for providers
Posted by Jarek Potiuk <ja...@potiuk.com>.
TL;DR; My very, very strong preference goes for one email which is
effectively N votes for N packages we release. I actually think we are not
able to efficiently follow up and manage 1-vote for email for ~20 or so
packages we are going to release every month.
I think we should just refer back to what ASF wrote about it.
https://www.apache.org/foundation/voting.html
> VOTES ON PACKAGE RELEASES
> Votes on whether a package is ready to be released use majority approval
-- i.e. at least three PMC members must vote affirmatively for release, and
there must be more positive than negative votes. Releases may not be
vetoed. Generally the community will cancel the release vote if anyone
identifies serious problems, but in most cases the ultimate decision, lies
with the individual serving as release manager. The specifics of the
process may vary from project to project, but the 'minimum
> quorum of three +1 votes' rule is universal.
"Generally the community will cancel the release vote if anyone identifies
serious problems, but in most cases the ultimate decision, lies with the
individual serving as release manager." - but as I read it - it's NOT the
PMC, but the community decides when to cancel release and the ultimate
decision lies in the hands of the release manager.
This has happened in the past with Airflow many times, that the release
manager usually Kaxil decided on canceling the release vote.
And yes - there is no mentioning of partial "canceling". But, this does not
mean that there MUST be a single mail per vote either. I think we can have
a single email and a N votes that cover all packages in a single email. And
you can vote in bulk on all packages. And some users might vote only on
some of the packages. For me this is purely for the efficiency of process
and PMCs responding to the votes.
Imagine we have 20 packages to send regularly (guesstimate), following up
20 messages to vote will be quite overwhelming - borderline ridiculous in
fact. I am talking about following up and counting, not sending. Sending is
easy.
We often have difficulty chasing PMCs to respond to one email, and the job
of tracking how many responses were sent to each of the 20 emails and
following up on each of them will make the release manager's life
miserable.
First two weeks after sending votes, the release manager will have to chase
20 emails, follow up, and count the votes for every single release. I would
certainly reject such a task as a release manager. It is not easy to
automate at all and is a lot of very stupid and tim-consuming overhead on
top of the work done already by release manager.
I do not really think the analogy of ice cream and dogs is appropriate (at
all). It's misleading actually. We are not voting on different things. It's
all about cancelling some votes out of many that we agreed to in bulk at
most. But not about changing the votes. This can already be done if the
community raises an issue - with the ultimate decision of the release
manager - so nothing changes here vs. the current Apache voting process.
For me it's only the question about granularity of the votes in one email
and whether we can cast a "bulk" N votes by single +1. I think we can.
For me what we are silently (and wrongly) assuming 1 mail = 1 VOTE. But I
do not believe anywhere in all ASF documentation there is such a
requirement. We have example "templates" for emails but I do not know if
anywhere this 1-1 relationship (email = Vote) is mentioned. I'd love to see
it if it is. For me as long as it is clear what we are voting on and what
eventually gets cancelled and we summaarise it in the "RESULT" email - we
should be good with having N votes in a single email.
So for me "bulk" voting looks as follows:
1) we send 20 packages and start 20 votes in one email.
2) Anyone - user, PMC could respond +1 (for all) or +1 (some) or +1 (- this
packages I tested which do not work).
3) Community (ultimately release manager) decides which of those votes to
cancel - if any (providing that all of the votes have +3 votes from PMC
members).
4) Releasing packages can still be done in waves - if we see after 3 days
that we can to release 10 out of the 20 package (but 10 has just +2 PMC
votes) - the release manager can still release the 10 and continue voting
on the remaining 10 (and even eventually cancel them).
Also a few words about testing those 20 packages.
Regarding testing - yes the packages MUST be tested by PMC members. This is
a must. But testing means in this case - bulk installing the packages on
top of airflow and see if the providers have been installed and shown in
`airlow providers list` with the correct version. I believe this is more
than enough to fulfill the "test" criteria when paired with user testing of
new/big functionalities.
This is (or should be - I followed it for sure) already part of the process
(alongside signing/checksum verification). And with svn checkout where you
can pull the whole folder to your machine, this can be done rather quickly.
As Kaxil proposed in the separate thread - also a judgement of release
manager can be used to see whether some additional testing is necessary.
And with single email and Github Issue we can very efficiently organise the
users to test their changes (even automate it).
I have shown during the last two weeks that this is not only doable, but
that our users and contributors generally respond very well to such
requests.
Those contributors await their changes to be released and they also want to
make sure it is tested before they get to the hands of the users.
J.
On Tue, Mar 9, 2021 at 12:40 PM Kaxil Naik <ka...@gmail.com> wrote:
> +1 -- I think users can pick and choose which providers to test and vote
> on. And since each vote would be in a separate email, a single provider bug
> in a particular provider won't derail conversation.
>
> Regards,
> Kaxil
>
> On Tue, Mar 9, 2021, 11:38 Kaxil Naik <ka...@gmail.com> wrote:
>
>> +1
>>
>> On Sat, Mar 6, 2021, 21:33 Ash Berlin-Taylor <as...@apache.org> wrote:
>>
>>> Just for absolutely clarity here: my -1 has nothing to do with the vote
>>> on the release, merely against adopting the proposed policy after this
>>> release/current set of votes.
>>>
>>> -a
>>>
>>> On 4 March 2021 11:04:13 GMT, Ash Berlin-Taylor <as...@apache.org> wrote:
>>>>
>>>> (Split out proposal from voting thread here
>>>> https://lists.apache.org/thread.html/r47daa0e594eb01c8196a96b62c40d5282b25c41ca1520781934e9236%40%3Cdev.airflow.apache.org%3E
>>>> .)
>>>>
>>>> I don't like the idea of changing what a vote means mid way through. To
>>>> take an extreme case:
>>>>
>>>> "+1 if you like ice cream"
>>>>
>>>> *people vote*
>>>>
>>>> "Vote now changed to +1 if you kick dogs".
>>>>
>>>> Now clearly that is not what you are proposing, but changing what the
>>>> vote is on at all opens a slippery slope and I'd rather we didn't
>>>> continue/codify this precedent.
>>>>
>>>> *This gets a -1 from me on adopting this part of the proposal.*
>>>>
>>>> Not changing the vote is also why I favour many smaller releases/votes:
>>>> a problem with one doesn't block the others, and additional I feel it is
>>>> easier for non-core contributors to get involved and test just the provider
>>>> they are interested in, without feeling daunted that they would have to
>>>> test all the rest to cast their vote.
>>>>
>>>> Much of the release process is already automated, up-to-and including
>>>> sending emails (dev/send_email.py -- may need tweaking for provider release
>>>> votes.) so this is can be achieved with little extra work to the release
>>>> manager.
>>>>
>>>> To be clear, I do not have a problem with batch releases and batch
>>>> votes (even if it is not my first choice, and I would rather N parallel
>>>> votes as default.)
>>>>
>>>> -ash
>>>>
>>>>
>>>> On Wed, 3 Mar, 2021 at 22:44, Jarek Potiuk <ja...@potiuk.com> wrote:
>>>>
>>>> I just wanted to use the opportunity to describe the current process
>>>> for deciding providers, because apparently not everyone is aware that we
>>>> already have an established and documented process for provider's releases
>>>> that was discussed on the devlist and it is documented in our release
>>>> process description.
>>>>
>>>> Possibly we will extract it into a separate policy and maybe discuss
>>>> some aspects of it (the discussion was raised today at the dev call of ours
>>>> but I wanted to make sure that we are all referring to the same "starting
>>>> point" and the "process" I based my recent actions on.
>>>>
>>>> *1) Batching the providers as the default*
>>>>
>>>> The decision on when to release and particularly preference for
>>>> releasing providers in batches is described in the
>>>> https://github.com/apache/airflow/blob/master/dev/README_RELEASE_PROVIDER_PACKAGES.md#decide-when-to-release
>>>>
>>>>
>>>> > Decide when to release
>>>> > You can release provider packages separately from the main Airflow on
>>>> an ad-hoc basis, whenever we find that
>>>> a given provider needs to be released - due to new features or due to
>>>> bug fixes
>>>> You can release each provider package separately, but due to voting and
>>>> release overhead we try to group
>>>> releases of provider packages together.
>>>>
>>>> *2) Possibility of excluding certain packages from the release.*
>>>>
>>>> The possibility of excluding certain packages which (for whatever
>>>> reason) we decide to remove at the discretion of release manager is
>>>> described here:
>>>> https://github.com/apache/airflow/blob/master/dev/README_RELEASE_PROVIDER_PACKAGES.md#prepare-voting-email-for-providers-release-candidate
>>>> <https://github.com/apache/airflow/blob/master/dev/README_RELEASE_PROVIDER_PACKAGESmd#prepare-voting-email-for-providers-release-candidate>
>>>>
>>>> Prepare voting email for Providers release candidate
>>>> ...
>>>>
>>>> > Due to the nature of packages, not all packages have to be released
>>>> as convenience packages in the final release. During the voting process the
>>>> voting PMCs might decide to exclude certain packages from the release if
>>>> some critical problems have been found in some packages.
>>>>
>>>> And the process of removing it is part of the described release process:
>>>>
>>>> > In case you decided to remove some of the packages. remove them from
>>>> dist folder now:
>>>>
>>>> > ls dist/*<provider>*
>>>> > rm dist/*<provider>*
>>>>
>>>> The issue of excluding certain packages have been discussed in this
>>>> thread in the mailing list :
>>>> https://lists.apache.org/thread.html/rc620a8a503cc7b14850c0a2a1fca1f6051d08a7e3e6d2cbdeb691dda%40%3Cdev.airflow.apache.org%3E
>>>> - where we had a -1 veto from a PMC member on the whole batch of providers
>>>> where we found a cncf.kubrenetes and google providers had critical
>>>> problems.
>>>>
>>>> We discussed it then and the two PMC members proposed a solution that
>>>> was not objected to by anyone in the VOTE thread - to remove the packages
>>>> from the batch.
>>>>
>>>> I continued this in the continuation of the voting thread
>>>> https://lists.apache.org/thread.html/r752c5d5171de4ff626663d30e1d50c4b0d2994f66bf8918d816dabd8%40%3Cdev.airflow.apache.org%3E
>>>> with the following message which specifically pointed out to my proposal
>>>> where I specifically linked to the message above and asked for comments.
>>>>
>>>> > As discussed before: -1 on a single provider does not invalidate the
>>>> whole
>>>> vote (from
>>>>
>>>> https://github.com/apache/airflow/tree/master/dev#vote-and-verify-the-backport-providers-release-candidate
>>>> ):
>>>>
>>>> > "Due to the nature of backport packages, not all packages have to be
>>>> released as convenience packages in the final release. During the voting
>>>> process the voting PMCs might decide to exclude certain packages from
>>>> the
>>>> release if some critical problems have been found in some packages."
>>>>
>>>> > We will merge the fix and most likely release a new google package
>>>> right
>>>> after this one. Looking at the super-localized problem here my current
>>>> decision will be to release 2020.10.29 'google package" - together with
>>>> other packages and release 2020.11.01 (or smth) - but only the google
>>>> one -
>>>> right after we merge the fix.
>>>>
>>>> > Any comments to that?
>>>>
>>>>
>>>>
>>>>
--
+48 660 796 129
Re: Voting and release process for providers
Posted by Kaxil Naik <ka...@gmail.com>.
+1 -- I think users can pick and choose which providers to test and vote
on. And since each vote would be in a separate email, a single provider bug
in a particular provider won't derail conversation.
Regards,
Kaxil
On Tue, Mar 9, 2021, 11:38 Kaxil Naik <ka...@gmail.com> wrote:
> +1
>
> On Sat, Mar 6, 2021, 21:33 Ash Berlin-Taylor <as...@apache.org> wrote:
>
>> Just for absolutely clarity here: my -1 has nothing to do with the vote
>> on the release, merely against adopting the proposed policy after this
>> release/current set of votes.
>>
>> -a
>>
>> On 4 March 2021 11:04:13 GMT, Ash Berlin-Taylor <as...@apache.org> wrote:
>>>
>>> (Split out proposal from voting thread here
>>> https://lists.apache.org/thread.html/r47daa0e594eb01c8196a96b62c40d5282b25c41ca1520781934e9236%40%3Cdev.airflow.apache.org%3E
>>> .)
>>>
>>> I don't like the idea of changing what a vote means mid way through. To
>>> take an extreme case:
>>>
>>> "+1 if you like ice cream"
>>>
>>> *people vote*
>>>
>>> "Vote now changed to +1 if you kick dogs".
>>>
>>> Now clearly that is not what you are proposing, but changing what the
>>> vote is on at all opens a slippery slope and I'd rather we didn't
>>> continue/codify this precedent.
>>>
>>> *This gets a -1 from me on adopting this part of the proposal.*
>>>
>>> Not changing the vote is also why I favour many smaller releases/votes:
>>> a problem with one doesn't block the others, and additional I feel it is
>>> easier for non-core contributors to get involved and test just the provider
>>> they are interested in, without feeling daunted that they would have to
>>> test all the rest to cast their vote.
>>>
>>> Much of the release process is already automated, up-to-and including
>>> sending emails (dev/send_email.py -- may need tweaking for provider release
>>> votes.) so this is can be achieved with little extra work to the release
>>> manager.
>>>
>>> To be clear, I do not have a problem with batch releases and batch votes
>>> (even if it is not my first choice, and I would rather N parallel votes as
>>> default.)
>>>
>>> -ash
>>>
>>>
>>> On Wed, 3 Mar, 2021 at 22:44, Jarek Potiuk <ja...@potiuk.com> wrote:
>>>
>>> I just wanted to use the opportunity to describe the current process for
>>> deciding providers, because apparently not everyone is aware that we
>>> already have an established and documented process for provider's releases
>>> that was discussed on the devlist and it is documented in our release
>>> process description.
>>>
>>> Possibly we will extract it into a separate policy and maybe discuss
>>> some aspects of it (the discussion was raised today at the dev call of ours
>>> but I wanted to make sure that we are all referring to the same "starting
>>> point" and the "process" I based my recent actions on.
>>>
>>> *1) Batching the providers as the default*
>>>
>>> The decision on when to release and particularly preference for
>>> releasing providers in batches is described in the
>>> https://github.com/apache/airflow/blob/master/dev/README_RELEASE_PROVIDER_PACKAGES.md#decide-when-to-release
>>>
>>>
>>> > Decide when to release
>>> > You can release provider packages separately from the main Airflow on
>>> an ad-hoc basis, whenever we find that
>>> a given provider needs to be released - due to new features or due to
>>> bug fixes
>>> You can release each provider package separately, but due to voting and
>>> release overhead we try to group
>>> releases of provider packages together.
>>>
>>> *2) Possibility of excluding certain packages from the release.*
>>>
>>> The possibility of excluding certain packages which (for whatever
>>> reason) we decide to remove at the discretion of release manager is
>>> described here:
>>> https://github.com/apache/airflow/blob/master/dev/README_RELEASE_PROVIDER_PACKAGES.md#prepare-voting-email-for-providers-release-candidate
>>> <https://github.com/apache/airflow/blob/master/dev/README_RELEASE_PROVIDER_PACKAGESmd#prepare-voting-email-for-providers-release-candidate>
>>>
>>> Prepare voting email for Providers release candidate
>>> ...
>>>
>>> > Due to the nature of packages, not all packages have to be released as
>>> convenience packages in the final release. During the voting process the
>>> voting PMCs might decide to exclude certain packages from the release if
>>> some critical problems have been found in some packages.
>>>
>>> And the process of removing it is part of the described release process:
>>>
>>> > In case you decided to remove some of the packages. remove them from
>>> dist folder now:
>>>
>>> > ls dist/*<provider>*
>>> > rm dist/*<provider>*
>>>
>>> The issue of excluding certain packages have been discussed in this
>>> thread in the mailing list :
>>> https://lists.apache.org/thread.html/rc620a8a503cc7b14850c0a2a1fca1f6051d08a7e3e6d2cbdeb691dda%40%3Cdev.airflow.apache.org%3E
>>> - where we had a -1 veto from a PMC member on the whole batch of providers
>>> where we found a cncf.kubrenetes and google providers had critical
>>> problems.
>>>
>>> We discussed it then and the two PMC members proposed a solution that
>>> was not objected to by anyone in the VOTE thread - to remove the packages
>>> from the batch.
>>>
>>> I continued this in the continuation of the voting thread
>>> https://lists.apache.org/thread.html/r752c5d5171de4ff626663d30e1d50c4b0d2994f66bf8918d816dabd8%40%3Cdev.airflow.apache.org%3E
>>> with the following message which specifically pointed out to my proposal
>>> where I specifically linked to the message above and asked for comments.
>>>
>>> > As discussed before: -1 on a single provider does not invalidate the
>>> whole
>>> vote (from
>>>
>>> https://github.com/apache/airflow/tree/master/dev#vote-and-verify-the-backport-providers-release-candidate
>>> ):
>>>
>>> > "Due to the nature of backport packages, not all packages have to be
>>> released as convenience packages in the final release. During the voting
>>> process the voting PMCs might decide to exclude certain packages from the
>>> release if some critical problems have been found in some packages."
>>>
>>> > We will merge the fix and most likely release a new google package
>>> right
>>> after this one. Looking at the super-localized problem here my current
>>> decision will be to release 2020.10.29 'google package" - together with
>>> other packages and release 2020.11.01 (or smth) - but only the google
>>> one -
>>> right after we merge the fix.
>>>
>>> > Any comments to that?
>>>
>>>
>>>
>>>
Re: Voting and release process for providers
Posted by Kaxil Naik <ka...@gmail.com>.
+1
On Sat, Mar 6, 2021, 21:33 Ash Berlin-Taylor <as...@apache.org> wrote:
> Just for absolutely clarity here: my -1 has nothing to do with the vote on
> the release, merely against adopting the proposed policy after this
> release/current set of votes.
>
> -a
>
> On 4 March 2021 11:04:13 GMT, Ash Berlin-Taylor <as...@apache.org> wrote:
>>
>> (Split out proposal from voting thread here
>> https://lists.apache.org/thread.html/r47daa0e594eb01c8196a96b62c40d5282b25c41ca1520781934e9236%40%3Cdev.airflow.apache.org%3E
>> .)
>>
>> I don't like the idea of changing what a vote means mid way through. To
>> take an extreme case:
>>
>> "+1 if you like ice cream"
>>
>> *people vote*
>>
>> "Vote now changed to +1 if you kick dogs".
>>
>> Now clearly that is not what you are proposing, but changing what the
>> vote is on at all opens a slippery slope and I'd rather we didn't
>> continue/codify this precedent.
>>
>> *This gets a -1 from me on adopting this part of the proposal.*
>>
>> Not changing the vote is also why I favour many smaller releases/votes: a
>> problem with one doesn't block the others, and additional I feel it is
>> easier for non-core contributors to get involved and test just the provider
>> they are interested in, without feeling daunted that they would have to
>> test all the rest to cast their vote.
>>
>> Much of the release process is already automated, up-to-and including
>> sending emails (dev/send_email.py -- may need tweaking for provider release
>> votes.) so this is can be achieved with little extra work to the release
>> manager.
>>
>> To be clear, I do not have a problem with batch releases and batch votes
>> (even if it is not my first choice, and I would rather N parallel votes as
>> default.)
>>
>> -ash
>>
>>
>> On Wed, 3 Mar, 2021 at 22:44, Jarek Potiuk <ja...@potiuk.com> wrote:
>>
>> I just wanted to use the opportunity to describe the current process for
>> deciding providers, because apparently not everyone is aware that we
>> already have an established and documented process for provider's releases
>> that was discussed on the devlist and it is documented in our release
>> process description.
>>
>> Possibly we will extract it into a separate policy and maybe discuss some
>> aspects of it (the discussion was raised today at the dev call of ours but
>> I wanted to make sure that we are all referring to the same "starting
>> point" and the "process" I based my recent actions on.
>>
>> *1) Batching the providers as the default*
>>
>> The decision on when to release and particularly preference for releasing
>> providers in batches is described in the
>> https://github.com/apache/airflow/blob/master/dev/README_RELEASE_PROVIDER_PACKAGES.md#decide-when-to-release
>>
>>
>> > Decide when to release
>> > You can release provider packages separately from the main Airflow on
>> an ad-hoc basis, whenever we find that
>> a given provider needs to be released - due to new features or due to bug
>> fixes
>> You can release each provider package separately, but due to voting and
>> release overhead we try to group
>> releases of provider packages together.
>>
>> *2) Possibility of excluding certain packages from the release.*
>>
>> The possibility of excluding certain packages which (for whatever reason)
>> we decide to remove at the discretion of release manager is described here:
>> https://github.com/apache/airflow/blob/master/dev/README_RELEASE_PROVIDER_PACKAGES.md#prepare-voting-email-for-providers-release-candidate
>> <https://github.com/apache/airflow/blob/master/dev/README_RELEASE_PROVIDER_PACKAGESmd#prepare-voting-email-for-providers-release-candidate>
>>
>> Prepare voting email for Providers release candidate
>> ...
>>
>> > Due to the nature of packages, not all packages have to be released as
>> convenience packages in the final release. During the voting process the
>> voting PMCs might decide to exclude certain packages from the release if
>> some critical problems have been found in some packages.
>>
>> And the process of removing it is part of the described release process:
>>
>> > In case you decided to remove some of the packages. remove them from
>> dist folder now:
>>
>> > ls dist/*<provider>*
>> > rm dist/*<provider>*
>>
>> The issue of excluding certain packages have been discussed in this
>> thread in the mailing list :
>> https://lists.apache.org/thread.html/rc620a8a503cc7b14850c0a2a1fca1f6051d08a7e3e6d2cbdeb691dda%40%3Cdev.airflow.apache.org%3E
>> - where we had a -1 veto from a PMC member on the whole batch of providers
>> where we found a cncf.kubrenetes and google providers had critical
>> problems.
>>
>> We discussed it then and the two PMC members proposed a solution that was
>> not objected to by anyone in the VOTE thread - to remove the packages from
>> the batch.
>>
>> I continued this in the continuation of the voting thread
>> https://lists.apache.org/thread.html/r752c5d5171de4ff626663d30e1d50c4b0d2994f66bf8918d816dabd8%40%3Cdev.airflow.apache.org%3E
>> with the following message which specifically pointed out to my proposal
>> where I specifically linked to the message above and asked for comments.
>>
>> > As discussed before: -1 on a single provider does not invalidate the
>> whole
>> vote (from
>>
>> https://github.com/apache/airflow/tree/master/dev#vote-and-verify-the-backport-providers-release-candidate
>> ):
>>
>> > "Due to the nature of backport packages, not all packages have to be
>> released as convenience packages in the final release. During the voting
>> process the voting PMCs might decide to exclude certain packages from the
>> release if some critical problems have been found in some packages."
>>
>> > We will merge the fix and most likely release a new google package right
>> after this one. Looking at the super-localized problem here my current
>> decision will be to release 2020.10.29 'google package" - together with
>> other packages and release 2020.11.01 (or smth) - but only the google one
>> -
>> right after we merge the fix.
>>
>> > Any comments to that?
>>
>>
>>
>>
Re: Voting and release process for providers
Posted by Ash Berlin-Taylor <as...@apache.org>.
Just for absolutely clarity here: my -1 has nothing to do with the vote on the release, merely against adopting the proposed policy after this release/current set of votes.
-a
On 4 March 2021 11:04:13 GMT, Ash Berlin-Taylor <as...@apache.org> wrote:
>(Split out proposal from voting thread here
><https://lists.apache.org/thread.html/r47daa0e594eb01c8196a96b62c40d5282b25c41ca1520781934e9236%40%3Cdev.airflow.apache.org%3E>.)
>
>I don't like the idea of changing what a vote means mid way through. To
>take an extreme case:
>
>"+1 if you like ice cream"
>
>*people vote*
>
>"Vote now changed to +1 if you kick dogs".
>
>Now clearly that is not what you are proposing, but changing what the
>vote is on at all opens a slippery slope and I'd rather we didn't
>continue/codify this precedent.
>
>*This gets a -1 from me on adopting this part of the proposal.*
>
>Not changing the vote is also why I favour many smaller releases/votes:
>a problem with one doesn't block the others, and additional I feel it
>is easier for non-core contributors to get involved and test just the
>provider they are interested in, without feeling daunted that they
>would have to test all the rest to cast their vote.
>
> Much of the release process is already automated, up-to-and including
>sending emails (dev/send_email.py -- may need tweaking for provider
>release votes.) so this is can be achieved with little extra work to
>the release manager.
>
>To be clear, I do not have a problem with batch releases and batch
>votes (even if it is not my first choice, and I would rather N parallel
>votes as default.)
>
>-ash
>
>
>On Wed, 3 Mar, 2021 at 22:44, Jarek Potiuk <ja...@potiuk.com> wrote:
>> I just wanted to use the opportunity to describe the current process
>> for deciding providers, because apparently not everyone is aware that
>> we already have an established and documented process for provider's
>> releases that was discussed on the devlist and it is documented in
>> our release process description.
>>
>> Possibly we will extract it into a separate policy and maybe discuss
>> some aspects of it (the discussion was raised today at the dev call
>> of ours but I wanted to make sure that we are all referring to the
>> same "starting point" and the "process" I based my recent actions on.
>>
>> *1) Batching the providers as the default*
>>
>> The decision on when to release and particularly preference for
>> releasing providers in batches is described in the
>> <https://github.com/apache/airflow/blob/master/dev/README_RELEASE_PROVIDER_PACKAGES.md#decide-when-to-release>
>>
>> > Decide when to release
>> > You can release provider packages separately from the main Airflow
>> on an ad-hoc basis, whenever we find that
>> a given provider needs to be released - due to new features or due to
>> bug fixes.
>> You can release each provider package separately, but due to voting
>> and release overhead we try to group
>> releases of provider packages together.
>>
>> *2) Possibility of excluding certain packages from the release.*
>>
>> The possibility of excluding certain packages which (for whatever
>> reason) we decide to remove at the discretion of release manager is
>> described here:
>> <https://github.com/apache/airflow/blob/master/dev/README_RELEASE_PROVIDER_PACKAGES.md#prepare-voting-email-for-providers-release-candidate>
>>
>> Prepare voting email for Providers release candidate
>> ....
>>
>> > Due to the nature of packages, not all packages have to be released
>> as convenience packages in the final release. During the voting
>> process the voting PMCs might decide to exclude certain packages from
>> the release if some critical problems have been found in some
>> packages.
>>
>> And the process of removing it is part of the described release
>> process:
>>
>> > In case you decided to remove some of the packages. remove them
>> from dist folder now:
>>
>> > ls dist/*<provider>*
>> > rm dist/*<provider>*
>>
>> The issue of excluding certain packages have been discussed in this
>> thread in the mailing list :
>> <https://lists.apache.org/thread.html/rc620a8a503cc7b14850c0a2a1fca1f6051d08a7e3e6d2cbdeb691dda%40%3Cdev.airflow.apache.org%3E>
>> - where we had a -1 veto from a PMC member on the whole batch of
>> providers where we found a cncf.kubrenetes and google providers had
>> critical problems.
>>
>> We discussed it then and the two PMC members proposed a solution that
>> was not objected to by anyone in the VOTE thread - to remove the
>> packages from the batch.
>>
>> I continued this in the continuation of the voting thread
>> <https://lists.apache.org/thread.html/r752c5d5171de4ff626663d30e1d50c4b0d2994f66bf8918d816dabd8%40%3Cdev.airflow.apache.org%3E>
>> with the following message which specifically pointed out to my
>> proposal where I specifically linked to the message above and asked
>> for comments.
>>
>> > As discussed before: -1 on a single provider does not invalidate
>> the whole
>> vote (from
>> <https://github.com/apache/airflow/tree/master/dev#vote-and-verify-the-backport-providers-release-candidate>
>> ):
>>
>> > "Due to the nature of backport packages, not all packages have to be
>> released as convenience packages in the final release. During the
>> voting
>> process the voting PMCs might decide to exclude certain packages from
>> the
>> release if some critical problems have been found in some packages."
>>
>> > We will merge the fix and most likely release a new google package
>> right
>> after this one. Looking at the super-localized problem here my current
>> decision will be to release 2020.10.29 'google package" - together
>> with
>> other packages and release 2020.11.01 (or smth) - but only the google
>> one -
>> right after we merge the fix.
>>
>> > Any comments to that?
>
>
>
Voting and release process for providers
Posted by Ash Berlin-Taylor <as...@apache.org>.
(Split out proposal from voting thread here
<https://lists.apache.org/thread.html/r47daa0e594eb01c8196a96b62c40d5282b25c41ca1520781934e9236%40%3Cdev.airflow.apache.org%3E>.)
I don't like the idea of changing what a vote means mid way through. To
take an extreme case:
"+1 if you like ice cream"
*people vote*
"Vote now changed to +1 if you kick dogs".
Now clearly that is not what you are proposing, but changing what the
vote is on at all opens a slippery slope and I'd rather we didn't
continue/codify this precedent.
*This gets a -1 from me on adopting this part of the proposal.*
Not changing the vote is also why I favour many smaller releases/votes:
a problem with one doesn't block the others, and additional I feel it
is easier for non-core contributors to get involved and test just the
provider they are interested in, without feeling daunted that they
would have to test all the rest to cast their vote.
Much of the release process is already automated, up-to-and including
sending emails (dev/send_email.py -- may need tweaking for provider
release votes.) so this is can be achieved with little extra work to
the release manager.
To be clear, I do not have a problem with batch releases and batch
votes (even if it is not my first choice, and I would rather N parallel
votes as default.)
-ash
On Wed, 3 Mar, 2021 at 22:44, Jarek Potiuk <ja...@potiuk.com> wrote:
> I just wanted to use the opportunity to describe the current process
> for deciding providers, because apparently not everyone is aware that
> we already have an established and documented process for provider's
> releases that was discussed on the devlist and it is documented in
> our release process description.
>
> Possibly we will extract it into a separate policy and maybe discuss
> some aspects of it (the discussion was raised today at the dev call
> of ours but I wanted to make sure that we are all referring to the
> same "starting point" and the "process" I based my recent actions on.
>
> *1) Batching the providers as the default*
>
> The decision on when to release and particularly preference for
> releasing providers in batches is described in the
> <https://github.com/apache/airflow/blob/master/dev/README_RELEASE_PROVIDER_PACKAGES.md#decide-when-to-release>
>
> > Decide when to release
> > You can release provider packages separately from the main Airflow
> on an ad-hoc basis, whenever we find that
> a given provider needs to be released - due to new features or due to
> bug fixes.
> You can release each provider package separately, but due to voting
> and release overhead we try to group
> releases of provider packages together.
>
> *2) Possibility of excluding certain packages from the release.*
>
> The possibility of excluding certain packages which (for whatever
> reason) we decide to remove at the discretion of release manager is
> described here:
> <https://github.com/apache/airflow/blob/master/dev/README_RELEASE_PROVIDER_PACKAGES.md#prepare-voting-email-for-providers-release-candidate>
>
> Prepare voting email for Providers release candidate
> ....
>
> > Due to the nature of packages, not all packages have to be released
> as convenience packages in the final release. During the voting
> process the voting PMCs might decide to exclude certain packages from
> the release if some critical problems have been found in some
> packages.
>
> And the process of removing it is part of the described release
> process:
>
> > In case you decided to remove some of the packages. remove them
> from dist folder now:
>
> > ls dist/*<provider>*
> > rm dist/*<provider>*
>
> The issue of excluding certain packages have been discussed in this
> thread in the mailing list :
> <https://lists.apache.org/thread.html/rc620a8a503cc7b14850c0a2a1fca1f6051d08a7e3e6d2cbdeb691dda%40%3Cdev.airflow.apache.org%3E>
> - where we had a -1 veto from a PMC member on the whole batch of
> providers where we found a cncf.kubrenetes and google providers had
> critical problems.
>
> We discussed it then and the two PMC members proposed a solution that
> was not objected to by anyone in the VOTE thread - to remove the
> packages from the batch.
>
> I continued this in the continuation of the voting thread
> <https://lists.apache.org/thread.html/r752c5d5171de4ff626663d30e1d50c4b0d2994f66bf8918d816dabd8%40%3Cdev.airflow.apache.org%3E>
> with the following message which specifically pointed out to my
> proposal where I specifically linked to the message above and asked
> for comments.
>
> > As discussed before: -1 on a single provider does not invalidate
> the whole
> vote (from
> <https://github.com/apache/airflow/tree/master/dev#vote-and-verify-the-backport-providers-release-candidate>
> ):
>
> > "Due to the nature of backport packages, not all packages have to be
> released as convenience packages in the final release. During the
> voting
> process the voting PMCs might decide to exclude certain packages from
> the
> release if some critical problems have been found in some packages."
>
> > We will merge the fix and most likely release a new google package
> right
> after this one. Looking at the super-localized problem here my current
> decision will be to release 2020.10.29 'google package" - together
> with
> other packages and release 2020.11.01 (or smth) - but only the google
> one -
> right after we merge the fix.
>
> > Any comments to that?
Re: [VOTE] Airflow Providers - release candidates from 2021-02-27
Posted by Deng Xiaodong <xd...@gmail.com>.
Thanks everyone for the efforts to make this release happen!
+1 (binding) from me.
XD
On Sun, Mar 7, 2021 at 11:05 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> I'd really appreciate one more +1 here from a PMC. No matter what the
> future holds for the provider's release. The vast majority of the providers
> have been tested by the users :
> https://github.com/apache/airflow/issues/14511. And we are one PMC vote
> away from simply pushing the button and releasing the new providers.We can
> proceed right after with proceeding the next ad-hoc wave.
>
> J.
>
> On Thu, Mar 4, 2021 at 12:27 PM Kaxil Naik <ka...@gmail.com> wrote:
>
>> Daniel,
>>
>> The problem is whenever we do batch — every month from now on — let’s say
>> all providers work but just one of them failed in rc’s — if we cancel the
>> entire vote again and start from scratch — it means +3 days. And since
>> getting to the result of providers already takes a good amount of time +3
>> days just delays it further.
>>
>> And delaying all other providers just because one of them (let’s say
>> telegram like provider fails) — might not be what we want.
>>
>> So how I look at it is yes it is a Single Vote (which we can argue and
>> change to a VOTE email per provider to avoid all confusion) — but we are
>> voting on each individual provider too (at least me until now).
>>
>> I will stick with my +1 vote.
>>
>> Regards,
>> Kaxil
>>
>> On Thu, Mar 4, 2021 at 8:47 AM Jarek Potiuk <ja...@potiuk.com> wrote:
>>
>>> Hey Daniel,
>>>
>>> The proposal is not new. We followed the very same proposal several
>>> times already for a number of batches of providers (I think 5 or 6 times
>>> already), I honestly do not feel there is anything "new" about it.
>>>
>>> I just tried to be helpful and explain what was there already because I
>>> think some of the people involved apparently did not realize we had this in
>>> place. I hope it is helpful to catch-up for those who missed it.
>>>
>>> Even in the email that I sent there is the link to the process for PMC
>>> members and contributors explaining what the responsibilities of PMC and
>>> contributors is :) - and those are the very same documents I mentioned in
>>> the explanation.
>>>
>>> Rather than restarting the vote I would prefer to continue with the
>>> release process - I know our users are waiting on it, and if we restart the
>>> vote now this means another at least 3 days of delay. Rather than that I
>>> would love to continue the voting process (we've also done that in the past
>>> - voting process lasts until 72H pass and 3 +1 votes are cast.
>>>
>>> Kaxil is the only one who made a vote so far (besides me) - so I will
>>> leave to you Kaxil, if you would like to withdraw the vote. (in case the
>>> process was not clear and now you changed your mind). But as soon as we
>>> have three PMC member votes the voting ends.
>>>
>>> J.
>>>
>>>
>>> On Wed, Mar 3, 2021 at 11:58 PM Daniel Imberman <
>>> daniel.imberman@gmail.com> wrote:
>>>
>>>> Hi Jarek,
>>>>
>>>> I think all of this sounds fine, but I think that we should start a new
>>>> vote with this understanding. I wouldn't feel comfortable assuming that any
>>>> of the previous +1's are still applicable as we have changed what people
>>>> are +1’ing.
>>>>
>>>> At the minimum I think we could have people re-affirm their votes on
>>>> this thread based on the new proposal
>>>>
>>>> Once we figure that out then +1 from me :)
>>>>
>>>> On Wed, Mar 3, 2021 at 1:44 PM, Jarek Potiuk <ja...@potiuk.com> wrote:
>>>>
>>>> Hello Everyone,
>>>>
>>>> We need one more PMC member vote in order to be able to release the
>>>> packages.
>>>>
>>>> Just to describe what the current status of the batch is:
>>>>
>>>> Based on discussion here:
>>>> https://github.com/apache/airflow/issues/14511 - I am planning to
>>>> follow the process we had previously documented in our release procedures
>>>> so far - that release manager tries to batch a number of providers in
>>>> single "voting process" and when there are problems discovered with certain
>>>> providers they might be excluded.
>>>>
>>>> I plan to skip the following providers from release now and release
>>>> them together on a ad-hoc basis whenever all the relevant issues are merged:
>>>>
>>>> * apache.druid, microsoft.azure, apache.beam
>>>>
>>>> I also noticed that snowflake python connector has been released this
>>>> morning as promised
>>>> https://pypi.org/project/snowflake-connector-python/2.4.0/ - it fixes
>>>> the last problem with the dependencies that were plaguing us, so I also
>>>> plan to remove the snowflake provider from this batch.
>>>>
>>>>
>>>> ---------
>>>>
>>>> I just wanted to use the opportunity to describe the current process
>>>> for deciding providers, because apparently not everyone is aware that we
>>>> already have an established and documented process for provider's releases
>>>> that was discussed on the devlist and it is documented in our release
>>>> process description.
>>>>
>>>> Possibly we will extract it into a separate policy and maybe discuss
>>>> some aspects of it (the discussion was raised today at the dev call of ours
>>>> but I wanted to make sure that we are all referring to the same "starting
>>>> point" and the "process" I based my recent actions on.
>>>>
>>>> *1) Batching the providers as the default*
>>>>
>>>> The decision on when to release and particularly preference for
>>>> releasing providers in batches is described in the
>>>> https://github.com/apache/airflow/blob/master/dev/README_RELEASE_PROVIDER_PACKAGES.md#decide-when-to-release
>>>>
>>>> > Decide when to release
>>>> > You can release provider packages separately from the main Airflow on
>>>> an ad-hoc basis, whenever we find that
>>>> a given provider needs to be released - due to new features or due to
>>>> bug fixes.
>>>> You can release each provider package separately, but due to voting and
>>>> release overhead we try to group
>>>> releases of provider packages together.
>>>>
>>>> *2) Possibility of excluding certain packages from the release.*
>>>>
>>>> The possibility of excluding certain packages which (for whatever
>>>> reason) we decide to remove at the discretion of release manager is
>>>> described here:
>>>> https://github.com/apache/airflow/blob/master/dev/README_RELEASE_PROVIDER_PACKAGES.md#prepare-voting-email-for-providers-release-candidate
>>>>
>>>> Prepare voting email for Providers release candidate
>>>> ....
>>>>
>>>> > Due to the nature of packages, not all packages have to be released
>>>> as convenience packages in the final release. During the voting process the
>>>> voting PMCs might decide to exclude certain packages from the release if
>>>> some critical problems have been found in some packages.
>>>>
>>>> And the process of removing it is part of the described release process:
>>>>
>>>> > In case you decided to remove some of the packages. remove them from
>>>> dist folder now:
>>>>
>>>> > ls dist/*<provider>*
>>>> > rm dist/*<provider>*
>>>>
>>>> The issue of excluding certain packages have been discussed in this
>>>> thread in the mailing list :
>>>> https://lists.apache.org/thread.html/rc620a8a503cc7b14850c0a2a1fca1f6051d08a7e3e6d2cbdeb691dda%40%3Cdev.airflow.apache.org%3E
>>>> - where we had a -1 veto from a PMC member on the whole batch of providers
>>>> where we found a cncf.kubrenetes and google providers had critical
>>>> problems.
>>>>
>>>> We discussed it then and the two PMC members proposed a solution that
>>>> was not objected to by anyone in the VOTE thread - to remove the packages
>>>> from the batch.
>>>>
>>>> I continued this in the continuation of the voting thread
>>>> https://lists.apache.org/thread.html/r752c5d5171de4ff626663d30e1d50c4b0d2994f66bf8918d816dabd8%40%3Cdev.airflow.apache.org%3E
>>>> with the following message which specifically pointed out to my proposal
>>>> where I specifically linked to the message above and asked for comments.
>>>>
>>>> > As discussed before: -1 on a single provider does not invalidate the
>>>> whole
>>>> vote (from
>>>>
>>>> https://github.com/apache/airflow/tree/master/dev#vote-and-verify-the-backport-providers-release-candidate
>>>> ):
>>>>
>>>> > "Due to the nature of backport packages, not all packages have to be
>>>> released as convenience packages in the final release. During the voting
>>>> process the voting PMCs might decide to exclude certain packages from
>>>> the
>>>> release if some critical problems have been found in some packages."
>>>>
>>>> > We will merge the fix and most likely release a new google package
>>>> right
>>>> after this one. Looking at the super-localized problem here my current
>>>> decision will be to release 2020.10.29 'google package" - together with
>>>> other packages and release 2020.11.01 (or smth) - but only the google
>>>> one -
>>>> right after we merge the fix.
>>>>
>>>> > Any comments to that?
>>>>
>>>>
>>>> J.
>>>>
>>>>
>>>>
>>>> On Wed, Mar 3, 2021 at 1:23 AM Kaxil Naik <ka...@gmail.com> wrote:
>>>>
>>>>> +1 (binding).
>>>>>
>>>>> Verified Signature and SHA12.
>>>>>
>>>>> Based on the changes (and Changelog) I can verify that the following
>>>>> providers should work fine:
>>>>>
>>>>>
>>>>> - spark
>>>>> - kubernetes
>>>>> - jenkins
>>>>> - microsoft.azure
>>>>> - mysql
>>>>> - telegram
>>>>> - and all the ones that just have doc changes
>>>>>
>>>>>
>>>>> Regards,
>>>>> Kaxil
>>>>>
>>>>> On Tue, Mar 2, 2021 at 9:01 PM Ryan Hatter <ry...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> There were some changes to the operator after my PR was merged:
>>>>>> https://github.com/apache/airflow/blob/master/airflow/providers/google/cloud/transfers/gdrive_to_gcs.py
>>>>>>
>>>>>> Pak Andrey (Scuall1992 on GitHub) might be able to confirm the
>>>>>> operator is functional.
>>>>>>
>>>>>> On Mar 2, 2021, at 13:16, Jarek Potiuk <ja...@potiuk.com> wrote:
>>>>>>
>>>>>>
>>>>>> Hello everyone - just a reminder that we have voting (hopefully)
>>>>>> finishing tomorrow.
>>>>>>
>>>>>> I'd love to get some votes for that.
>>>>>>
>>>>>> Just to clarify what the PMC votes mean, because I believe there were
>>>>>> some question raised about the release process which we are going to
>>>>>> discuss it tomorrow at the dev call but let me just express my
>>>>>> interpretation of https://infra.apache.org/release-publishing.html
>>>>>>
>>>>>> PMC member vote (as I understand it) does not mean that this PMC
>>>>>> member tested the release functionality (neither Release Manager).
>>>>>> This merely means that the PMC member agrees that the software was
>>>>>> released according to the requirements and process described in
>>>>>> https://infra.apache.org/release-publishing.html and that the
>>>>>> signatures, hash-sums and software packages are as expected by the process.
>>>>>> This is how I interpret this part of the release process "Release
>>>>>> managers do the mechanical work; but the PMC in general, and the PMC chair
>>>>>> in particular (as an officer of the Foundation), are responsible for
>>>>>> compliance with ASF requirements."
>>>>>>
>>>>>> My understanding is that it is not feasible (neither for Airflow nor
>>>>>> Providers) that the PMC members (nor release manager) tests the software
>>>>>> and all features/bugfixes. We've never done that and I believe we will
>>>>>> never do. We are reaching out to the community to test and we make a best
>>>>>> effort to test whatever we release automatically (unit tests, integration
>>>>>> tests, testing if providers are installable/importable with Airflow 2.0 and
>>>>>> latest source code of Airflow). And we hardly can do more than that.
>>>>>>
>>>>>> Happy to discuss it tomorrow, but in the meantime If some of the PMCs
>>>>>> could do the review of the process and check the compliance, to be ready to
>>>>>> cast your votes - I'd love that.
>>>>>>
>>>>>> J.
>>>>>>
>>>>>> On Tue, Mar 2, 2021 at 8:44 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>>>>>>
>>>>>>> Hey Ryan,
>>>>>>>
>>>>>>> There is no **must** in re-testing it. Providing that you tested it
>>>>>>> before with real GSuite account is for me enough of a confirmation ;).
>>>>>>>
>>>>>>> J.
>>>>>>>
>>>>>>> On Sun, Feb 28, 2021 at 10:00 PM Abdur-Rahmaan Janhangeer <
>>>>>>> arj.python@gmail.com> wrote:
>>>>>>>
>>>>>>>> Salutes for having a GSuite account just for the functionality
>>>>>>>> 👍👍👍
>>>>>>>>
>>>>>>>> On Mon, 1 Mar 2021, 00:05 Ryan Hatter, <ry...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> I canceled my GSuite account when my PR for the gdrive to gcs
>>>>>>>>> operator was approved & merged. Could anyone maybe help me ensure correct
>>>>>>>>> functionality?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Feb 27, 2021, at 08:48, Jarek Potiuk <ja...@potiuk.com> wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I created issue, where we will track the status of tests for the
>>>>>>>>> providers (again - it is experiment - but I'd really love to get feedback
>>>>>>>>> on the new providers from those who contributed):
>>>>>>>>> https://github.com/apache/airflow/issues/14511
>>>>>>>>>
>>>>>>>>> On Sat, Feb 27, 2021 at 4:28 PM Jarek Potiuk <ja...@potiuk.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Hey all,
>>>>>>>>>>
>>>>>>>>>> I have just cut the new wave Airflow Providers packages. This
>>>>>>>>>> email is calling a vote on the release,
>>>>>>>>>> which will last for 72 hours + day for the weekend - which means
>>>>>>>>>> that it will end on Wed 3 Mar 15:59:34 CET 2021.
>>>>>>>>>>
>>>>>>>>>> Consider this my (binding) +1.
>>>>>>>>>>
>>>>>>>>>> *KIND REQUEST*
>>>>>>>>>>
>>>>>>>>>> There was a recent discussion about test quality of the providers
>>>>>>>>>> and I would like to try to address it, still keeping the batch release
>>>>>>>>>> process every 3 weeks.
>>>>>>>>>>
>>>>>>>>>> We need a bit of help from the community. I have a kind request
>>>>>>>>>> to the authors of fixes and new features. I group the providers into those
>>>>>>>>>> that likely need more testing, and those that do not. I also added names of
>>>>>>>>>> those who submitted the changes and are most likely to be able to verify if
>>>>>>>>>> the RC packages are solving the problems/adding features.
>>>>>>>>>>
>>>>>>>>>> This is a bit of experiment (apologies for calling out) - but if
>>>>>>>>>> we find that it works, we can automate that. I will create a separate Issue
>>>>>>>>>> in Github where you will be able to "tick" the boxes for those providers
>>>>>>>>>> which they are added to. It would not be a blocker if not tested, but it
>>>>>>>>>> will be a great help if you could test the new RC provider and see if it
>>>>>>>>>> works as expected according to your changes.
>>>>>>>>>>
>>>>>>>>>> Providers with new features and fixes - likely needs some
>>>>>>>>>> testing.:
>>>>>>>>>>
>>>>>>>>>> * *amazon* : Cristòfol Torrens, Ruben Laguna, Arati Nagmal,
>>>>>>>>>> Ivica Kolenkaš, JavierLopezT
>>>>>>>>>> * *apache.druid*: Xinbin Huang
>>>>>>>>>> * *apache.spark*: Igor Khrol
>>>>>>>>>> * *cncf.kubernetes*: jpyen, Ash Berlin-Taylor, Daniel Imberman
>>>>>>>>>> * *google*: Vivek Bhojawala, Xinbin Huang, Pak Andrey, uma66,
>>>>>>>>>> Ryan Yuan, morrme, Sam Wheating, YingyingPeng22, Ryan Hatter,Tobiasz
>>>>>>>>>> Kędzierski
>>>>>>>>>> * *jenkins*: Maxim Lisovsky
>>>>>>>>>> * *microsift.azure <http://microsift.azure>*: flvndh, yyu
>>>>>>>>>> * *mysql*: Constantino Schillebeeckx
>>>>>>>>>> * *qubole*: Xinbin Huang
>>>>>>>>>> * *salesforce*: Jyoti Dhiman
>>>>>>>>>> * *slack*: Igor Khrol
>>>>>>>>>> * *tableau*: Jyoti Dhiman
>>>>>>>>>> * *telegram*: Shekhar Sing, Adil Khashtamov
>>>>>>>>>>
>>>>>>>>>> Providers with doc only changes (no need to test):
>>>>>>>>>>
>>>>>>>>>> * apache-beam
>>>>>>>>>> * apache-hive
>>>>>>>>>> * dingding
>>>>>>>>>> * docker
>>>>>>>>>> * elasticsearch
>>>>>>>>>> * exasol
>>>>>>>>>> * http
>>>>>>>>>> * neo4j
>>>>>>>>>> * openfaas
>>>>>>>>>> * papermill
>>>>>>>>>> * presto
>>>>>>>>>> * sendgrid
>>>>>>>>>> * sftp
>>>>>>>>>> * snowflake
>>>>>>>>>> * sqlite
>>>>>>>>>> * ssh
>>>>>>>>>> *
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Airflow Providers are available at:
>>>>>>>>>> https://dist.apache.org/repos/dist/dev/airflow/providers/
>>>>>>>>>>
>>>>>>>>>> *apache-airflow-providers-<PROVIDER>-*-bin.tar.gz* are the binary
>>>>>>>>>> Python "sdist" release - they are also official "sources" for the
>>>>>>>>>> provider packages.
>>>>>>>>>>
>>>>>>>>>> *apache_airflow_providers_<PROVIDER>-*.whl are the binary
>>>>>>>>>> Python "wheel" release.
>>>>>>>>>>
>>>>>>>>>> The test procedure for PMC members who would like to test the RC
>>>>>>>>>> candidates are described in
>>>>>>>>>>
>>>>>>>>>> https://github.com/apache/airflow/blob/master/dev/README_RELEASE_PROVIDER_PACKAGES.md#verify-the-release-by-pmc-members
>>>>>>>>>>
>>>>>>>>>> and for Contributors:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> https://github.com/apache/airflow/blob/master/dev/README_RELEASE_PROVIDER_PACKAGES.md#verify-by-contributors
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Public keys are available at:
>>>>>>>>>> https://dist.apache.org/repos/dist/release/airflow/KEYS
>>>>>>>>>>
>>>>>>>>>> Please vote accordingly:
>>>>>>>>>>
>>>>>>>>>> [ ] +1 approve
>>>>>>>>>> [ ] +0 no opinion
>>>>>>>>>> [ ] -1 disapprove with the reason
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Only votes from PMC members are binding, but members of the
>>>>>>>>>> community are
>>>>>>>>>> encouraged to test the release and vote with "(non-binding)".
>>>>>>>>>>
>>>>>>>>>> Please note that the version number excludes the 'rcX' string.
>>>>>>>>>> This will allow us to rename the artifact without modifying
>>>>>>>>>> the artifact checksums when we actually release.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Each of the packages contains a link to the detailed changelog.
>>>>>>>>>> The changelogs are moved to the official airflow documentation:
>>>>>>>>>> https://github.com/apache/airflow-site/<TODO COPY LINK TO BRANCH>
>>>>>>>>>>
>>>>>>>>>> <PASTE ANY HIGH-LEVEL DESCRIPTION OF THE CHANGES HERE!>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Note the links to documentation from PyPI packages are not
>>>>>>>>>> working until we merge
>>>>>>>>>> the changes to airflow site after releasing the packages
>>>>>>>>>> officially.
>>>>>>>>>>
>>>>>>>>>> https://pypi.org/project/apache-airflow-providers-amazon/1.2.0rc1/
>>>>>>>>>>
>>>>>>>>>> https://pypi.org/project/apache-airflow-providers-apache-beam/1.0.1rc1/
>>>>>>>>>>
>>>>>>>>>> https://pypi.org/project/apache-airflow-providers-apache-druid/1.1.0rc1/
>>>>>>>>>>
>>>>>>>>>> https://pypi.org/project/apache-airflow-providers-apache-hive/1.0.2rc1/
>>>>>>>>>>
>>>>>>>>>> https://pypi.org/project/apache-airflow-providers-apache-spark/1.0.2rc1/
>>>>>>>>>>
>>>>>>>>>> https://pypi.org/project/apache-airflow-providers-cncf-kubernetes/1.0.2rc1/
>>>>>>>>>>
>>>>>>>>>> https://pypi.org/project/apache-airflow-providers-dingding/1.0.2rc1/
>>>>>>>>>> https://pypi.org/project/apache-airflow-providers-docker/1.0.2rc1/
>>>>>>>>>>
>>>>>>>>>> https://pypi.org/project/apache-airflow-providers-elasticsearch/1.0.2rc1/
>>>>>>>>>> https://pypi.org/project/apache-airflow-providers-exasol/1.1.1rc1/
>>>>>>>>>> https://pypi.org/project/apache-airflow-providers-google/2.1.0rc1/
>>>>>>>>>> https://pypi.org/project/apache-airflow-providers-http/1.1.1rc1/
>>>>>>>>>>
>>>>>>>>>> https://pypi.org/project/apache-airflow-providers-jenkins/1.1.0rc1/
>>>>>>>>>>
>>>>>>>>>> https://pypi.org/project/apache-airflow-providers-microsoft-azure/1.2.0rc1/
>>>>>>>>>> https://pypi.org/project/apache-airflow-providers-mysql/1.0.2rc1/
>>>>>>>>>> https://pypi.org/project/apache-airflow-providers-neo4j/1.0.1rc1/
>>>>>>>>>>
>>>>>>>>>> https://pypi.org/project/apache-airflow-providers-openfaas/1.1.1rc1/
>>>>>>>>>>
>>>>>>>>>> https://pypi.org/project/apache-airflow-providers-papermill/1.0.2rc1/
>>>>>>>>>> https://pypi.org/project/apache-airflow-providers-presto/1.0.2rc1/
>>>>>>>>>> https://pypi.org/project/apache-airflow-providers-qubole/1.0.2rc1/
>>>>>>>>>>
>>>>>>>>>> https://pypi.org/project/apache-airflow-providers-salesforce/2.0.0rc1/
>>>>>>>>>>
>>>>>>>>>> https://pypi.org/project/apache-airflow-providers-sendgrid/1.0.2rc1/
>>>>>>>>>> https://pypi.org/project/apache-airflow-providers-sftp/1.1.1rc1/
>>>>>>>>>> https://pypi.org/project/apache-airflow-providers-slack/3.0.0rc1/
>>>>>>>>>>
>>>>>>>>>> https://pypi.org/project/apache-airflow-providers-snowflake/1.1.1rc1/
>>>>>>>>>> https://pypi.org/project/apache-airflow-providers-sqlite/1.0.2rc1/
>>>>>>>>>> https://pypi.org/project/apache-airflow-providers-ssh/1.2.0rc1/
>>>>>>>>>>
>>>>>>>>>> https://pypi.org/project/apache-airflow-providers-tableau/1.0.0rc1/
>>>>>>>>>>
>>>>>>>>>> https://pypi.org/project/apache-airflow-providers-telegram/1.0.2rc1/
>>>>>>>>>>
>>>>>>>>>> Cheers,
>>>>>>>>>> J.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> +48 660 796 129 <+48660796129>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> +48 660 796 129 <+48660796129>
>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> +48 660 796 129 <+48660796129>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> +48 660 796 129 <+48660796129>
>>>>>>
>>>>>>
>>>>
>>>> --
>>>> +48 660 796 129 <+48660796129>
>>>>
>>>>
>>>
>>> --
>>> +48 660 796 129
>>>
>>
>
> --
> +48 660 796 129
>
Re: [VOTE] Airflow Providers - release candidates from 2021-02-27
Posted by Jarek Potiuk <ja...@potiuk.com>.
I'd really appreciate one more +1 here from a PMC. No matter what the
future holds for the provider's release. The vast majority of the providers
have been tested by the users :
https://github.com/apache/airflow/issues/14511. And we are one PMC vote
away from simply pushing the button and releasing the new providers.We can
proceed right after with proceeding the next ad-hoc wave.
J.
On Thu, Mar 4, 2021 at 12:27 PM Kaxil Naik <ka...@gmail.com> wrote:
> Daniel,
>
> The problem is whenever we do batch — every month from now on — let’s say
> all providers work but just one of them failed in rc’s — if we cancel the
> entire vote again and start from scratch — it means +3 days. And since
> getting to the result of providers already takes a good amount of time +3
> days just delays it further.
>
> And delaying all other providers just because one of them (let’s say
> telegram like provider fails) — might not be what we want.
>
> So how I look at it is yes it is a Single Vote (which we can argue and
> change to a VOTE email per provider to avoid all confusion) — but we are
> voting on each individual provider too (at least me until now).
>
> I will stick with my +1 vote.
>
> Regards,
> Kaxil
>
> On Thu, Mar 4, 2021 at 8:47 AM Jarek Potiuk <ja...@potiuk.com> wrote:
>
>> Hey Daniel,
>>
>> The proposal is not new. We followed the very same proposal several
>> times already for a number of batches of providers (I think 5 or 6 times
>> already), I honestly do not feel there is anything "new" about it.
>>
>> I just tried to be helpful and explain what was there already because I
>> think some of the people involved apparently did not realize we had this in
>> place. I hope it is helpful to catch-up for those who missed it.
>>
>> Even in the email that I sent there is the link to the process for PMC
>> members and contributors explaining what the responsibilities of PMC and
>> contributors is :) - and those are the very same documents I mentioned in
>> the explanation.
>>
>> Rather than restarting the vote I would prefer to continue with the
>> release process - I know our users are waiting on it, and if we restart the
>> vote now this means another at least 3 days of delay. Rather than that I
>> would love to continue the voting process (we've also done that in the past
>> - voting process lasts until 72H pass and 3 +1 votes are cast.
>>
>> Kaxil is the only one who made a vote so far (besides me) - so I will
>> leave to you Kaxil, if you would like to withdraw the vote. (in case the
>> process was not clear and now you changed your mind). But as soon as we
>> have three PMC member votes the voting ends.
>>
>> J.
>>
>>
>> On Wed, Mar 3, 2021 at 11:58 PM Daniel Imberman <
>> daniel.imberman@gmail.com> wrote:
>>
>>> Hi Jarek,
>>>
>>> I think all of this sounds fine, but I think that we should start a new
>>> vote with this understanding. I wouldn't feel comfortable assuming that any
>>> of the previous +1's are still applicable as we have changed what people
>>> are +1’ing.
>>>
>>> At the minimum I think we could have people re-affirm their votes on
>>> this thread based on the new proposal
>>>
>>> Once we figure that out then +1 from me :)
>>>
>>> On Wed, Mar 3, 2021 at 1:44 PM, Jarek Potiuk <ja...@potiuk.com> wrote:
>>>
>>> Hello Everyone,
>>>
>>> We need one more PMC member vote in order to be able to release the
>>> packages.
>>>
>>> Just to describe what the current status of the batch is:
>>>
>>> Based on discussion here: https://github.com/apache/airflow/issues/14511
>>> - I am planning to follow the process we had previously documented in our
>>> release procedures so far - that release manager tries to batch a number of
>>> providers in single "voting process" and when there are problems discovered
>>> with certain providers they might be excluded.
>>>
>>> I plan to skip the following providers from release now and release them
>>> together on a ad-hoc basis whenever all the relevant issues are merged:
>>>
>>> * apache.druid, microsoft.azure, apache.beam
>>>
>>> I also noticed that snowflake python connector has been released this
>>> morning as promised
>>> https://pypi.org/project/snowflake-connector-python/2.4.0/ - it fixes
>>> the last problem with the dependencies that were plaguing us, so I also
>>> plan to remove the snowflake provider from this batch.
>>>
>>>
>>> ---------
>>>
>>> I just wanted to use the opportunity to describe the current process for
>>> deciding providers, because apparently not everyone is aware that we
>>> already have an established and documented process for provider's releases
>>> that was discussed on the devlist and it is documented in our release
>>> process description.
>>>
>>> Possibly we will extract it into a separate policy and maybe discuss
>>> some aspects of it (the discussion was raised today at the dev call of ours
>>> but I wanted to make sure that we are all referring to the same "starting
>>> point" and the "process" I based my recent actions on.
>>>
>>> *1) Batching the providers as the default*
>>>
>>> The decision on when to release and particularly preference for
>>> releasing providers in batches is described in the
>>> https://github.com/apache/airflow/blob/master/dev/README_RELEASE_PROVIDER_PACKAGES.md#decide-when-to-release
>>>
>>> > Decide when to release
>>> > You can release provider packages separately from the main Airflow on
>>> an ad-hoc basis, whenever we find that
>>> a given provider needs to be released - due to new features or due to
>>> bug fixes.
>>> You can release each provider package separately, but due to voting and
>>> release overhead we try to group
>>> releases of provider packages together.
>>>
>>> *2) Possibility of excluding certain packages from the release.*
>>>
>>> The possibility of excluding certain packages which (for whatever
>>> reason) we decide to remove at the discretion of release manager is
>>> described here:
>>> https://github.com/apache/airflow/blob/master/dev/README_RELEASE_PROVIDER_PACKAGES.md#prepare-voting-email-for-providers-release-candidate
>>>
>>> Prepare voting email for Providers release candidate
>>> ....
>>>
>>> > Due to the nature of packages, not all packages have to be released as
>>> convenience packages in the final release. During the voting process the
>>> voting PMCs might decide to exclude certain packages from the release if
>>> some critical problems have been found in some packages.
>>>
>>> And the process of removing it is part of the described release process:
>>>
>>> > In case you decided to remove some of the packages. remove them from
>>> dist folder now:
>>>
>>> > ls dist/*<provider>*
>>> > rm dist/*<provider>*
>>>
>>> The issue of excluding certain packages have been discussed in this
>>> thread in the mailing list :
>>> https://lists.apache.org/thread.html/rc620a8a503cc7b14850c0a2a1fca1f6051d08a7e3e6d2cbdeb691dda%40%3Cdev.airflow.apache.org%3E
>>> - where we had a -1 veto from a PMC member on the whole batch of providers
>>> where we found a cncf.kubrenetes and google providers had critical
>>> problems.
>>>
>>> We discussed it then and the two PMC members proposed a solution that
>>> was not objected to by anyone in the VOTE thread - to remove the packages
>>> from the batch.
>>>
>>> I continued this in the continuation of the voting thread
>>> https://lists.apache.org/thread.html/r752c5d5171de4ff626663d30e1d50c4b0d2994f66bf8918d816dabd8%40%3Cdev.airflow.apache.org%3E
>>> with the following message which specifically pointed out to my proposal
>>> where I specifically linked to the message above and asked for comments.
>>>
>>> > As discussed before: -1 on a single provider does not invalidate the
>>> whole
>>> vote (from
>>>
>>> https://github.com/apache/airflow/tree/master/dev#vote-and-verify-the-backport-providers-release-candidate
>>> ):
>>>
>>> > "Due to the nature of backport packages, not all packages have to be
>>> released as convenience packages in the final release. During the voting
>>> process the voting PMCs might decide to exclude certain packages from the
>>> release if some critical problems have been found in some packages."
>>>
>>> > We will merge the fix and most likely release a new google package
>>> right
>>> after this one. Looking at the super-localized problem here my current
>>> decision will be to release 2020.10.29 'google package" - together with
>>> other packages and release 2020.11.01 (or smth) - but only the google
>>> one -
>>> right after we merge the fix.
>>>
>>> > Any comments to that?
>>>
>>>
>>> J.
>>>
>>>
>>>
>>> On Wed, Mar 3, 2021 at 1:23 AM Kaxil Naik <ka...@gmail.com> wrote:
>>>
>>>> +1 (binding).
>>>>
>>>> Verified Signature and SHA12.
>>>>
>>>> Based on the changes (and Changelog) I can verify that the following
>>>> providers should work fine:
>>>>
>>>>
>>>> - spark
>>>> - kubernetes
>>>> - jenkins
>>>> - microsoft.azure
>>>> - mysql
>>>> - telegram
>>>> - and all the ones that just have doc changes
>>>>
>>>>
>>>> Regards,
>>>> Kaxil
>>>>
>>>> On Tue, Mar 2, 2021 at 9:01 PM Ryan Hatter <ry...@gmail.com>
>>>> wrote:
>>>>
>>>>> There were some changes to the operator after my PR was merged:
>>>>> https://github.com/apache/airflow/blob/master/airflow/providers/google/cloud/transfers/gdrive_to_gcs.py
>>>>>
>>>>> Pak Andrey (Scuall1992 on GitHub) might be able to confirm the
>>>>> operator is functional.
>>>>>
>>>>> On Mar 2, 2021, at 13:16, Jarek Potiuk <ja...@potiuk.com> wrote:
>>>>>
>>>>>
>>>>> Hello everyone - just a reminder that we have voting (hopefully)
>>>>> finishing tomorrow.
>>>>>
>>>>> I'd love to get some votes for that.
>>>>>
>>>>> Just to clarify what the PMC votes mean, because I believe there were
>>>>> some question raised about the release process which we are going to
>>>>> discuss it tomorrow at the dev call but let me just express my
>>>>> interpretation of https://infra.apache.org/release-publishing.html
>>>>>
>>>>> PMC member vote (as I understand it) does not mean that this PMC
>>>>> member tested the release functionality (neither Release Manager).
>>>>> This merely means that the PMC member agrees that the software was
>>>>> released according to the requirements and process described in
>>>>> https://infra.apache.org/release-publishing.html and that the
>>>>> signatures, hash-sums and software packages are as expected by the process.
>>>>> This is how I interpret this part of the release process "Release
>>>>> managers do the mechanical work; but the PMC in general, and the PMC chair
>>>>> in particular (as an officer of the Foundation), are responsible for
>>>>> compliance with ASF requirements."
>>>>>
>>>>> My understanding is that it is not feasible (neither for Airflow nor
>>>>> Providers) that the PMC members (nor release manager) tests the software
>>>>> and all features/bugfixes. We've never done that and I believe we will
>>>>> never do. We are reaching out to the community to test and we make a best
>>>>> effort to test whatever we release automatically (unit tests, integration
>>>>> tests, testing if providers are installable/importable with Airflow 2.0 and
>>>>> latest source code of Airflow). And we hardly can do more than that.
>>>>>
>>>>> Happy to discuss it tomorrow, but in the meantime If some of the PMCs
>>>>> could do the review of the process and check the compliance, to be ready to
>>>>> cast your votes - I'd love that.
>>>>>
>>>>> J.
>>>>>
>>>>> On Tue, Mar 2, 2021 at 8:44 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>>>>>
>>>>>> Hey Ryan,
>>>>>>
>>>>>> There is no **must** in re-testing it. Providing that you tested it
>>>>>> before with real GSuite account is for me enough of a confirmation ;).
>>>>>>
>>>>>> J.
>>>>>>
>>>>>> On Sun, Feb 28, 2021 at 10:00 PM Abdur-Rahmaan Janhangeer <
>>>>>> arj.python@gmail.com> wrote:
>>>>>>
>>>>>>> Salutes for having a GSuite account just for the functionality 👍👍👍
>>>>>>>
>>>>>>> On Mon, 1 Mar 2021, 00:05 Ryan Hatter, <ry...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> I canceled my GSuite account when my PR for the gdrive to gcs
>>>>>>>> operator was approved & merged. Could anyone maybe help me ensure correct
>>>>>>>> functionality?
>>>>>>>>
>>>>>>>>
>>>>>>>> On Feb 27, 2021, at 08:48, Jarek Potiuk <ja...@potiuk.com> wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>> I created issue, where we will track the status of tests for the
>>>>>>>> providers (again - it is experiment - but I'd really love to get feedback
>>>>>>>> on the new providers from those who contributed):
>>>>>>>> https://github.com/apache/airflow/issues/14511
>>>>>>>>
>>>>>>>> On Sat, Feb 27, 2021 at 4:28 PM Jarek Potiuk <ja...@potiuk.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>>
>>>>>>>>> Hey all,
>>>>>>>>>
>>>>>>>>> I have just cut the new wave Airflow Providers packages. This
>>>>>>>>> email is calling a vote on the release,
>>>>>>>>> which will last for 72 hours + day for the weekend - which means
>>>>>>>>> that it will end on Wed 3 Mar 15:59:34 CET 2021.
>>>>>>>>>
>>>>>>>>> Consider this my (binding) +1.
>>>>>>>>>
>>>>>>>>> *KIND REQUEST*
>>>>>>>>>
>>>>>>>>> There was a recent discussion about test quality of the providers
>>>>>>>>> and I would like to try to address it, still keeping the batch release
>>>>>>>>> process every 3 weeks.
>>>>>>>>>
>>>>>>>>> We need a bit of help from the community. I have a kind request to
>>>>>>>>> the authors of fixes and new features. I group the providers into those
>>>>>>>>> that likely need more testing, and those that do not. I also added names of
>>>>>>>>> those who submitted the changes and are most likely to be able to verify if
>>>>>>>>> the RC packages are solving the problems/adding features.
>>>>>>>>>
>>>>>>>>> This is a bit of experiment (apologies for calling out) - but if
>>>>>>>>> we find that it works, we can automate that. I will create a separate Issue
>>>>>>>>> in Github where you will be able to "tick" the boxes for those providers
>>>>>>>>> which they are added to. It would not be a blocker if not tested, but it
>>>>>>>>> will be a great help if you could test the new RC provider and see if it
>>>>>>>>> works as expected according to your changes.
>>>>>>>>>
>>>>>>>>> Providers with new features and fixes - likely needs some testing.:
>>>>>>>>>
>>>>>>>>> * *amazon* : Cristòfol Torrens, Ruben Laguna, Arati Nagmal, Ivica
>>>>>>>>> Kolenkaš, JavierLopezT
>>>>>>>>> * *apache.druid*: Xinbin Huang
>>>>>>>>> * *apache.spark*: Igor Khrol
>>>>>>>>> * *cncf.kubernetes*: jpyen, Ash Berlin-Taylor, Daniel Imberman
>>>>>>>>> * *google*: Vivek Bhojawala, Xinbin Huang, Pak Andrey, uma66,
>>>>>>>>> Ryan Yuan, morrme, Sam Wheating, YingyingPeng22, Ryan Hatter,Tobiasz
>>>>>>>>> Kędzierski
>>>>>>>>> * *jenkins*: Maxim Lisovsky
>>>>>>>>> * *microsift.azure <http://microsift.azure>*: flvndh, yyu
>>>>>>>>> * *mysql*: Constantino Schillebeeckx
>>>>>>>>> * *qubole*: Xinbin Huang
>>>>>>>>> * *salesforce*: Jyoti Dhiman
>>>>>>>>> * *slack*: Igor Khrol
>>>>>>>>> * *tableau*: Jyoti Dhiman
>>>>>>>>> * *telegram*: Shekhar Sing, Adil Khashtamov
>>>>>>>>>
>>>>>>>>> Providers with doc only changes (no need to test):
>>>>>>>>>
>>>>>>>>> * apache-beam
>>>>>>>>> * apache-hive
>>>>>>>>> * dingding
>>>>>>>>> * docker
>>>>>>>>> * elasticsearch
>>>>>>>>> * exasol
>>>>>>>>> * http
>>>>>>>>> * neo4j
>>>>>>>>> * openfaas
>>>>>>>>> * papermill
>>>>>>>>> * presto
>>>>>>>>> * sendgrid
>>>>>>>>> * sftp
>>>>>>>>> * snowflake
>>>>>>>>> * sqlite
>>>>>>>>> * ssh
>>>>>>>>> *
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Airflow Providers are available at:
>>>>>>>>> https://dist.apache.org/repos/dist/dev/airflow/providers/
>>>>>>>>>
>>>>>>>>> *apache-airflow-providers-<PROVIDER>-*-bin.tar.gz* are the binary
>>>>>>>>> Python "sdist" release - they are also official "sources" for the
>>>>>>>>> provider packages.
>>>>>>>>>
>>>>>>>>> *apache_airflow_providers_<PROVIDER>-*.whl are the binary
>>>>>>>>> Python "wheel" release.
>>>>>>>>>
>>>>>>>>> The test procedure for PMC members who would like to test the RC
>>>>>>>>> candidates are described in
>>>>>>>>>
>>>>>>>>> https://github.com/apache/airflow/blob/master/dev/README_RELEASE_PROVIDER_PACKAGES.md#verify-the-release-by-pmc-members
>>>>>>>>>
>>>>>>>>> and for Contributors:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> https://github.com/apache/airflow/blob/master/dev/README_RELEASE_PROVIDER_PACKAGES.md#verify-by-contributors
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Public keys are available at:
>>>>>>>>> https://dist.apache.org/repos/dist/release/airflow/KEYS
>>>>>>>>>
>>>>>>>>> Please vote accordingly:
>>>>>>>>>
>>>>>>>>> [ ] +1 approve
>>>>>>>>> [ ] +0 no opinion
>>>>>>>>> [ ] -1 disapprove with the reason
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Only votes from PMC members are binding, but members of the
>>>>>>>>> community are
>>>>>>>>> encouraged to test the release and vote with "(non-binding)".
>>>>>>>>>
>>>>>>>>> Please note that the version number excludes the 'rcX' string.
>>>>>>>>> This will allow us to rename the artifact without modifying
>>>>>>>>> the artifact checksums when we actually release.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Each of the packages contains a link to the detailed changelog.
>>>>>>>>> The changelogs are moved to the official airflow documentation:
>>>>>>>>> https://github.com/apache/airflow-site/<TODO COPY LINK TO BRANCH>
>>>>>>>>>
>>>>>>>>> <PASTE ANY HIGH-LEVEL DESCRIPTION OF THE CHANGES HERE!>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Note the links to documentation from PyPI packages are not working
>>>>>>>>> until we merge
>>>>>>>>> the changes to airflow site after releasing the packages
>>>>>>>>> officially.
>>>>>>>>>
>>>>>>>>> https://pypi.org/project/apache-airflow-providers-amazon/1.2.0rc1/
>>>>>>>>>
>>>>>>>>> https://pypi.org/project/apache-airflow-providers-apache-beam/1.0.1rc1/
>>>>>>>>>
>>>>>>>>> https://pypi.org/project/apache-airflow-providers-apache-druid/1.1.0rc1/
>>>>>>>>>
>>>>>>>>> https://pypi.org/project/apache-airflow-providers-apache-hive/1.0.2rc1/
>>>>>>>>>
>>>>>>>>> https://pypi.org/project/apache-airflow-providers-apache-spark/1.0.2rc1/
>>>>>>>>>
>>>>>>>>> https://pypi.org/project/apache-airflow-providers-cncf-kubernetes/1.0.2rc1/
>>>>>>>>>
>>>>>>>>> https://pypi.org/project/apache-airflow-providers-dingding/1.0.2rc1/
>>>>>>>>> https://pypi.org/project/apache-airflow-providers-docker/1.0.2rc1/
>>>>>>>>>
>>>>>>>>> https://pypi.org/project/apache-airflow-providers-elasticsearch/1.0.2rc1/
>>>>>>>>> https://pypi.org/project/apache-airflow-providers-exasol/1.1.1rc1/
>>>>>>>>> https://pypi.org/project/apache-airflow-providers-google/2.1.0rc1/
>>>>>>>>> https://pypi.org/project/apache-airflow-providers-http/1.1.1rc1/
>>>>>>>>> https://pypi.org/project/apache-airflow-providers-jenkins/1.1.0rc1/
>>>>>>>>>
>>>>>>>>> https://pypi.org/project/apache-airflow-providers-microsoft-azure/1.2.0rc1/
>>>>>>>>> https://pypi.org/project/apache-airflow-providers-mysql/1.0.2rc1/
>>>>>>>>> https://pypi.org/project/apache-airflow-providers-neo4j/1.0.1rc1/
>>>>>>>>>
>>>>>>>>> https://pypi.org/project/apache-airflow-providers-openfaas/1.1.1rc1/
>>>>>>>>>
>>>>>>>>> https://pypi.org/project/apache-airflow-providers-papermill/1.0.2rc1/
>>>>>>>>> https://pypi.org/project/apache-airflow-providers-presto/1.0.2rc1/
>>>>>>>>> https://pypi.org/project/apache-airflow-providers-qubole/1.0.2rc1/
>>>>>>>>>
>>>>>>>>> https://pypi.org/project/apache-airflow-providers-salesforce/2.0.0rc1/
>>>>>>>>>
>>>>>>>>> https://pypi.org/project/apache-airflow-providers-sendgrid/1.0.2rc1/
>>>>>>>>> https://pypi.org/project/apache-airflow-providers-sftp/1.1.1rc1/
>>>>>>>>> https://pypi.org/project/apache-airflow-providers-slack/3.0.0rc1/
>>>>>>>>>
>>>>>>>>> https://pypi.org/project/apache-airflow-providers-snowflake/1.1.1rc1/
>>>>>>>>> https://pypi.org/project/apache-airflow-providers-sqlite/1.0.2rc1/
>>>>>>>>> https://pypi.org/project/apache-airflow-providers-ssh/1.2.0rc1/
>>>>>>>>> https://pypi.org/project/apache-airflow-providers-tableau/1.0.0rc1/
>>>>>>>>>
>>>>>>>>> https://pypi.org/project/apache-airflow-providers-telegram/1.0.2rc1/
>>>>>>>>>
>>>>>>>>> Cheers,
>>>>>>>>> J.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> +48 660 796 129 <+48660796129>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> +48 660 796 129 <+48660796129>
>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>> --
>>>>>> +48 660 796 129 <+48660796129>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> +48 660 796 129 <+48660796129>
>>>>>
>>>>>
>>>
>>> --
>>> +48 660 796 129 <+48660796129>
>>>
>>>
>>
>> --
>> +48 660 796 129
>>
>
--
+48 660 796 129
Re: [VOTE] Airflow Providers - release candidates from 2021-02-27
Posted by Kaxil Naik <ka...@gmail.com>.
Daniel,
The problem is whenever we do batch — every month from now on — let’s say
all providers work but just one of them failed in rc’s — if we cancel the
entire vote again and start from scratch — it means +3 days. And since
getting to the result of providers already takes a good amount of time +3
days just delays it further.
And delaying all other providers just because one of them (let’s say
telegram like provider fails) — might not be what we want.
So how I look at it is yes it is a Single Vote (which we can argue and
change to a VOTE email per provider to avoid all confusion) — but we are
voting on each individual provider too (at least me until now).
I will stick with my +1 vote.
Regards,
Kaxil
On Thu, Mar 4, 2021 at 8:47 AM Jarek Potiuk <ja...@potiuk.com> wrote:
> Hey Daniel,
>
> The proposal is not new. We followed the very same proposal several times
> already for a number of batches of providers (I think 5 or 6 times
> already), I honestly do not feel there is anything "new" about it.
>
> I just tried to be helpful and explain what was there already because I
> think some of the people involved apparently did not realize we had this in
> place. I hope it is helpful to catch-up for those who missed it.
>
> Even in the email that I sent there is the link to the process for PMC
> members and contributors explaining what the responsibilities of PMC and
> contributors is :) - and those are the very same documents I mentioned in
> the explanation.
>
> Rather than restarting the vote I would prefer to continue with the
> release process - I know our users are waiting on it, and if we restart the
> vote now this means another at least 3 days of delay. Rather than that I
> would love to continue the voting process (we've also done that in the past
> - voting process lasts until 72H pass and 3 +1 votes are cast.
>
> Kaxil is the only one who made a vote so far (besides me) - so I will
> leave to you Kaxil, if you would like to withdraw the vote. (in case the
> process was not clear and now you changed your mind). But as soon as we
> have three PMC member votes the voting ends.
>
> J.
>
>
> On Wed, Mar 3, 2021 at 11:58 PM Daniel Imberman <da...@gmail.com>
> wrote:
>
>> Hi Jarek,
>>
>> I think all of this sounds fine, but I think that we should start a new
>> vote with this understanding. I wouldn't feel comfortable assuming that any
>> of the previous +1's are still applicable as we have changed what people
>> are +1’ing.
>>
>> At the minimum I think we could have people re-affirm their votes on this
>> thread based on the new proposal
>>
>> Once we figure that out then +1 from me :)
>>
>> On Wed, Mar 3, 2021 at 1:44 PM, Jarek Potiuk <ja...@potiuk.com> wrote:
>>
>> Hello Everyone,
>>
>> We need one more PMC member vote in order to be able to release the
>> packages.
>>
>> Just to describe what the current status of the batch is:
>>
>> Based on discussion here: https://github.com/apache/airflow/issues/14511
>> - I am planning to follow the process we had previously documented in our
>> release procedures so far - that release manager tries to batch a number of
>> providers in single "voting process" and when there are problems discovered
>> with certain providers they might be excluded.
>>
>> I plan to skip the following providers from release now and release them
>> together on a ad-hoc basis whenever all the relevant issues are merged:
>>
>> * apache.druid, microsoft.azure, apache.beam
>>
>> I also noticed that snowflake python connector has been released this
>> morning as promised
>> https://pypi.org/project/snowflake-connector-python/2.4.0/ - it fixes
>> the last problem with the dependencies that were plaguing us, so I also
>> plan to remove the snowflake provider from this batch.
>>
>>
>> ---------
>>
>> I just wanted to use the opportunity to describe the current process for
>> deciding providers, because apparently not everyone is aware that we
>> already have an established and documented process for provider's releases
>> that was discussed on the devlist and it is documented in our release
>> process description.
>>
>> Possibly we will extract it into a separate policy and maybe discuss some
>> aspects of it (the discussion was raised today at the dev call of ours but
>> I wanted to make sure that we are all referring to the same "starting
>> point" and the "process" I based my recent actions on.
>>
>> *1) Batching the providers as the default*
>>
>> The decision on when to release and particularly preference for releasing
>> providers in batches is described in the
>> https://github.com/apache/airflow/blob/master/dev/README_RELEASE_PROVIDER_PACKAGES.md#decide-when-to-release
>>
>> > Decide when to release
>> > You can release provider packages separately from the main Airflow on
>> an ad-hoc basis, whenever we find that
>> a given provider needs to be released - due to new features or due to bug
>> fixes.
>> You can release each provider package separately, but due to voting and
>> release overhead we try to group
>> releases of provider packages together.
>>
>> *2) Possibility of excluding certain packages from the release.*
>>
>> The possibility of excluding certain packages which (for whatever reason)
>> we decide to remove at the discretion of release manager is described here:
>> https://github.com/apache/airflow/blob/master/dev/README_RELEASE_PROVIDER_PACKAGES.md#prepare-voting-email-for-providers-release-candidate
>>
>> Prepare voting email for Providers release candidate
>> ....
>>
>> > Due to the nature of packages, not all packages have to be released as
>> convenience packages in the final release. During the voting process the
>> voting PMCs might decide to exclude certain packages from the release if
>> some critical problems have been found in some packages.
>>
>> And the process of removing it is part of the described release process:
>>
>> > In case you decided to remove some of the packages. remove them from
>> dist folder now:
>>
>> > ls dist/*<provider>*
>> > rm dist/*<provider>*
>>
>> The issue of excluding certain packages have been discussed in this
>> thread in the mailing list :
>> https://lists.apache.org/thread.html/rc620a8a503cc7b14850c0a2a1fca1f6051d08a7e3e6d2cbdeb691dda%40%3Cdev.airflow.apache.org%3E
>> - where we had a -1 veto from a PMC member on the whole batch of providers
>> where we found a cncf.kubrenetes and google providers had critical
>> problems.
>>
>> We discussed it then and the two PMC members proposed a solution that was
>> not objected to by anyone in the VOTE thread - to remove the packages from
>> the batch.
>>
>> I continued this in the continuation of the voting thread
>> https://lists.apache.org/thread.html/r752c5d5171de4ff626663d30e1d50c4b0d2994f66bf8918d816dabd8%40%3Cdev.airflow.apache.org%3E
>> with the following message which specifically pointed out to my proposal
>> where I specifically linked to the message above and asked for comments.
>>
>> > As discussed before: -1 on a single provider does not invalidate the
>> whole
>> vote (from
>>
>> https://github.com/apache/airflow/tree/master/dev#vote-and-verify-the-backport-providers-release-candidate
>> ):
>>
>> > "Due to the nature of backport packages, not all packages have to be
>> released as convenience packages in the final release. During the voting
>> process the voting PMCs might decide to exclude certain packages from the
>> release if some critical problems have been found in some packages."
>>
>> > We will merge the fix and most likely release a new google package right
>> after this one. Looking at the super-localized problem here my current
>> decision will be to release 2020.10.29 'google package" - together with
>> other packages and release 2020.11.01 (or smth) - but only the google one
>> -
>> right after we merge the fix.
>>
>> > Any comments to that?
>>
>>
>> J.
>>
>>
>>
>> On Wed, Mar 3, 2021 at 1:23 AM Kaxil Naik <ka...@gmail.com> wrote:
>>
>>> +1 (binding).
>>>
>>> Verified Signature and SHA12.
>>>
>>> Based on the changes (and Changelog) I can verify that the following
>>> providers should work fine:
>>>
>>>
>>> - spark
>>> - kubernetes
>>> - jenkins
>>> - microsoft.azure
>>> - mysql
>>> - telegram
>>> - and all the ones that just have doc changes
>>>
>>>
>>> Regards,
>>> Kaxil
>>>
>>> On Tue, Mar 2, 2021 at 9:01 PM Ryan Hatter <ry...@gmail.com>
>>> wrote:
>>>
>>>> There were some changes to the operator after my PR was merged:
>>>> https://github.com/apache/airflow/blob/master/airflow/providers/google/cloud/transfers/gdrive_to_gcs.py
>>>>
>>>> Pak Andrey (Scuall1992 on GitHub) might be able to confirm the operator
>>>> is functional.
>>>>
>>>> On Mar 2, 2021, at 13:16, Jarek Potiuk <ja...@potiuk.com> wrote:
>>>>
>>>>
>>>> Hello everyone - just a reminder that we have voting (hopefully)
>>>> finishing tomorrow.
>>>>
>>>> I'd love to get some votes for that.
>>>>
>>>> Just to clarify what the PMC votes mean, because I believe there were
>>>> some question raised about the release process which we are going to
>>>> discuss it tomorrow at the dev call but let me just express my
>>>> interpretation of https://infra.apache.org/release-publishing.html
>>>>
>>>> PMC member vote (as I understand it) does not mean that this PMC member
>>>> tested the release functionality (neither Release Manager).
>>>> This merely means that the PMC member agrees that the software was
>>>> released according to the requirements and process described in
>>>> https://infra.apache.org/release-publishing.html and that the
>>>> signatures, hash-sums and software packages are as expected by the process.
>>>> This is how I interpret this part of the release process "Release
>>>> managers do the mechanical work; but the PMC in general, and the PMC chair
>>>> in particular (as an officer of the Foundation), are responsible for
>>>> compliance with ASF requirements."
>>>>
>>>> My understanding is that it is not feasible (neither for Airflow nor
>>>> Providers) that the PMC members (nor release manager) tests the software
>>>> and all features/bugfixes. We've never done that and I believe we will
>>>> never do. We are reaching out to the community to test and we make a best
>>>> effort to test whatever we release automatically (unit tests, integration
>>>> tests, testing if providers are installable/importable with Airflow 2.0 and
>>>> latest source code of Airflow). And we hardly can do more than that.
>>>>
>>>> Happy to discuss it tomorrow, but in the meantime If some of the PMCs
>>>> could do the review of the process and check the compliance, to be ready to
>>>> cast your votes - I'd love that.
>>>>
>>>> J.
>>>>
>>>> On Tue, Mar 2, 2021 at 8:44 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>>>>
>>>>> Hey Ryan,
>>>>>
>>>>> There is no **must** in re-testing it. Providing that you tested it
>>>>> before with real GSuite account is for me enough of a confirmation ;).
>>>>>
>>>>> J.
>>>>>
>>>>> On Sun, Feb 28, 2021 at 10:00 PM Abdur-Rahmaan Janhangeer <
>>>>> arj.python@gmail.com> wrote:
>>>>>
>>>>>> Salutes for having a GSuite account just for the functionality 👍👍👍
>>>>>>
>>>>>> On Mon, 1 Mar 2021, 00:05 Ryan Hatter, <ry...@gmail.com> wrote:
>>>>>>
>>>>>>> I canceled my GSuite account when my PR for the gdrive to gcs
>>>>>>> operator was approved & merged. Could anyone maybe help me ensure correct
>>>>>>> functionality?
>>>>>>>
>>>>>>>
>>>>>>> On Feb 27, 2021, at 08:48, Jarek Potiuk <ja...@potiuk.com> wrote:
>>>>>>>
>>>>>>>
>>>>>>> I created issue, where we will track the status of tests for the
>>>>>>> providers (again - it is experiment - but I'd really love to get feedback
>>>>>>> on the new providers from those who contributed):
>>>>>>> https://github.com/apache/airflow/issues/14511
>>>>>>>
>>>>>>> On Sat, Feb 27, 2021 at 4:28 PM Jarek Potiuk <ja...@potiuk.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>>
>>>>>>>> Hey all,
>>>>>>>>
>>>>>>>> I have just cut the new wave Airflow Providers packages. This email
>>>>>>>> is calling a vote on the release,
>>>>>>>> which will last for 72 hours + day for the weekend - which means
>>>>>>>> that it will end on Wed 3 Mar 15:59:34 CET 2021.
>>>>>>>>
>>>>>>>> Consider this my (binding) +1.
>>>>>>>>
>>>>>>>> *KIND REQUEST*
>>>>>>>>
>>>>>>>> There was a recent discussion about test quality of the providers
>>>>>>>> and I would like to try to address it, still keeping the batch release
>>>>>>>> process every 3 weeks.
>>>>>>>>
>>>>>>>> We need a bit of help from the community. I have a kind request to
>>>>>>>> the authors of fixes and new features. I group the providers into those
>>>>>>>> that likely need more testing, and those that do not. I also added names of
>>>>>>>> those who submitted the changes and are most likely to be able to verify if
>>>>>>>> the RC packages are solving the problems/adding features.
>>>>>>>>
>>>>>>>> This is a bit of experiment (apologies for calling out) - but if we
>>>>>>>> find that it works, we can automate that. I will create a separate Issue in
>>>>>>>> Github where you will be able to "tick" the boxes for those providers which
>>>>>>>> they are added to. It would not be a blocker if not tested, but it will be
>>>>>>>> a great help if you could test the new RC provider and see if it works as
>>>>>>>> expected according to your changes.
>>>>>>>>
>>>>>>>> Providers with new features and fixes - likely needs some testing.:
>>>>>>>>
>>>>>>>> * *amazon* : Cristòfol Torrens, Ruben Laguna, Arati Nagmal, Ivica
>>>>>>>> Kolenkaš, JavierLopezT
>>>>>>>> * *apache.druid*: Xinbin Huang
>>>>>>>> * *apache.spark*: Igor Khrol
>>>>>>>> * *cncf.kubernetes*: jpyen, Ash Berlin-Taylor, Daniel Imberman
>>>>>>>> * *google*: Vivek Bhojawala, Xinbin Huang, Pak Andrey, uma66, Ryan
>>>>>>>> Yuan, morrme, Sam Wheating, YingyingPeng22, Ryan Hatter,Tobiasz Kędzierski
>>>>>>>> * *jenkins*: Maxim Lisovsky
>>>>>>>> * *microsift.azure <http://microsift.azure>*: flvndh, yyu
>>>>>>>> * *mysql*: Constantino Schillebeeckx
>>>>>>>> * *qubole*: Xinbin Huang
>>>>>>>> * *salesforce*: Jyoti Dhiman
>>>>>>>> * *slack*: Igor Khrol
>>>>>>>> * *tableau*: Jyoti Dhiman
>>>>>>>> * *telegram*: Shekhar Sing, Adil Khashtamov
>>>>>>>>
>>>>>>>> Providers with doc only changes (no need to test):
>>>>>>>>
>>>>>>>> * apache-beam
>>>>>>>> * apache-hive
>>>>>>>> * dingding
>>>>>>>> * docker
>>>>>>>> * elasticsearch
>>>>>>>> * exasol
>>>>>>>> * http
>>>>>>>> * neo4j
>>>>>>>> * openfaas
>>>>>>>> * papermill
>>>>>>>> * presto
>>>>>>>> * sendgrid
>>>>>>>> * sftp
>>>>>>>> * snowflake
>>>>>>>> * sqlite
>>>>>>>> * ssh
>>>>>>>> *
>>>>>>>>
>>>>>>>>
>>>>>>>> Airflow Providers are available at:
>>>>>>>> https://dist.apache.org/repos/dist/dev/airflow/providers/
>>>>>>>>
>>>>>>>> *apache-airflow-providers-<PROVIDER>-*-bin.tar.gz* are the binary
>>>>>>>> Python "sdist" release - they are also official "sources" for the
>>>>>>>> provider packages.
>>>>>>>>
>>>>>>>> *apache_airflow_providers_<PROVIDER>-*.whl are the binary
>>>>>>>> Python "wheel" release.
>>>>>>>>
>>>>>>>> The test procedure for PMC members who would like to test the RC
>>>>>>>> candidates are described in
>>>>>>>>
>>>>>>>> https://github.com/apache/airflow/blob/master/dev/README_RELEASE_PROVIDER_PACKAGES.md#verify-the-release-by-pmc-members
>>>>>>>>
>>>>>>>> and for Contributors:
>>>>>>>>
>>>>>>>>
>>>>>>>> https://github.com/apache/airflow/blob/master/dev/README_RELEASE_PROVIDER_PACKAGES.md#verify-by-contributors
>>>>>>>>
>>>>>>>>
>>>>>>>> Public keys are available at:
>>>>>>>> https://dist.apache.org/repos/dist/release/airflow/KEYS
>>>>>>>>
>>>>>>>> Please vote accordingly:
>>>>>>>>
>>>>>>>> [ ] +1 approve
>>>>>>>> [ ] +0 no opinion
>>>>>>>> [ ] -1 disapprove with the reason
>>>>>>>>
>>>>>>>>
>>>>>>>> Only votes from PMC members are binding, but members of the
>>>>>>>> community are
>>>>>>>> encouraged to test the release and vote with "(non-binding)".
>>>>>>>>
>>>>>>>> Please note that the version number excludes the 'rcX' string.
>>>>>>>> This will allow us to rename the artifact without modifying
>>>>>>>> the artifact checksums when we actually release.
>>>>>>>>
>>>>>>>>
>>>>>>>> Each of the packages contains a link to the detailed changelog. The
>>>>>>>> changelogs are moved to the official airflow documentation:
>>>>>>>> https://github.com/apache/airflow-site/<TODO COPY LINK TO BRANCH>
>>>>>>>>
>>>>>>>> <PASTE ANY HIGH-LEVEL DESCRIPTION OF THE CHANGES HERE!>
>>>>>>>>
>>>>>>>>
>>>>>>>> Note the links to documentation from PyPI packages are not working
>>>>>>>> until we merge
>>>>>>>> the changes to airflow site after releasing the packages officially.
>>>>>>>>
>>>>>>>> https://pypi.org/project/apache-airflow-providers-amazon/1.2.0rc1/
>>>>>>>>
>>>>>>>> https://pypi.org/project/apache-airflow-providers-apache-beam/1.0.1rc1/
>>>>>>>>
>>>>>>>> https://pypi.org/project/apache-airflow-providers-apache-druid/1.1.0rc1/
>>>>>>>>
>>>>>>>> https://pypi.org/project/apache-airflow-providers-apache-hive/1.0.2rc1/
>>>>>>>>
>>>>>>>> https://pypi.org/project/apache-airflow-providers-apache-spark/1.0.2rc1/
>>>>>>>>
>>>>>>>> https://pypi.org/project/apache-airflow-providers-cncf-kubernetes/1.0.2rc1/
>>>>>>>> https://pypi.org/project/apache-airflow-providers-dingding/1.0.2rc1/
>>>>>>>> https://pypi.org/project/apache-airflow-providers-docker/1.0.2rc1/
>>>>>>>>
>>>>>>>> https://pypi.org/project/apache-airflow-providers-elasticsearch/1.0.2rc1/
>>>>>>>> https://pypi.org/project/apache-airflow-providers-exasol/1.1.1rc1/
>>>>>>>> https://pypi.org/project/apache-airflow-providers-google/2.1.0rc1/
>>>>>>>> https://pypi.org/project/apache-airflow-providers-http/1.1.1rc1/
>>>>>>>> https://pypi.org/project/apache-airflow-providers-jenkins/1.1.0rc1/
>>>>>>>>
>>>>>>>> https://pypi.org/project/apache-airflow-providers-microsoft-azure/1.2.0rc1/
>>>>>>>> https://pypi.org/project/apache-airflow-providers-mysql/1.0.2rc1/
>>>>>>>> https://pypi.org/project/apache-airflow-providers-neo4j/1.0.1rc1/
>>>>>>>> https://pypi.org/project/apache-airflow-providers-openfaas/1.1.1rc1/
>>>>>>>>
>>>>>>>> https://pypi.org/project/apache-airflow-providers-papermill/1.0.2rc1/
>>>>>>>> https://pypi.org/project/apache-airflow-providers-presto/1.0.2rc1/
>>>>>>>> https://pypi.org/project/apache-airflow-providers-qubole/1.0.2rc1/
>>>>>>>>
>>>>>>>> https://pypi.org/project/apache-airflow-providers-salesforce/2.0.0rc1/
>>>>>>>> https://pypi.org/project/apache-airflow-providers-sendgrid/1.0.2rc1/
>>>>>>>> https://pypi.org/project/apache-airflow-providers-sftp/1.1.1rc1/
>>>>>>>> https://pypi.org/project/apache-airflow-providers-slack/3.0.0rc1/
>>>>>>>>
>>>>>>>> https://pypi.org/project/apache-airflow-providers-snowflake/1.1.1rc1/
>>>>>>>> https://pypi.org/project/apache-airflow-providers-sqlite/1.0.2rc1/
>>>>>>>> https://pypi.org/project/apache-airflow-providers-ssh/1.2.0rc1/
>>>>>>>> https://pypi.org/project/apache-airflow-providers-tableau/1.0.0rc1/
>>>>>>>> https://pypi.org/project/apache-airflow-providers-telegram/1.0.2rc1/
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>> J.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> +48 660 796 129 <+48660796129>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> +48 660 796 129 <+48660796129>
>>>>>>>
>>>>>>>
>>>>>
>>>>> --
>>>>> +48 660 796 129 <+48660796129>
>>>>>
>>>>
>>>>
>>>> --
>>>> +48 660 796 129 <+48660796129>
>>>>
>>>>
>>
>> --
>> +48 660 796 129 <+48660796129>
>>
>>
>
> --
> +48 660 796 129
>
Re: [VOTE] Airflow Providers - release candidates from 2021-02-27
Posted by Jarek Potiuk <ja...@potiuk.com>.
Hey Daniel,
The proposal is not new. We followed the very same proposal several times
already for a number of batches of providers (I think 5 or 6 times
already), I honestly do not feel there is anything "new" about it.
I just tried to be helpful and explain what was there already because I
think some of the people involved apparently did not realize we had this in
place. I hope it is helpful to catch-up for those who missed it.
Even in the email that I sent there is the link to the process for PMC
members and contributors explaining what the responsibilities of PMC and
contributors is :) - and those are the very same documents I mentioned in
the explanation.
Rather than restarting the vote I would prefer to continue with the release
process - I know our users are waiting on it, and if we restart the vote
now this means another at least 3 days of delay. Rather than that I would
love to continue the voting process (we've also done that in the past -
voting process lasts until 72H pass and 3 +1 votes are cast.
Kaxil is the only one who made a vote so far (besides me) - so I will leave
to you Kaxil, if you would like to withdraw the vote. (in case the process
was not clear and now you changed your mind). But as soon as we have three
PMC member votes the voting ends.
J.
On Wed, Mar 3, 2021 at 11:58 PM Daniel Imberman <da...@gmail.com>
wrote:
> Hi Jarek,
>
> I think all of this sounds fine, but I think that we should start a new
> vote with this understanding. I wouldn't feel comfortable assuming that any
> of the previous +1's are still applicable as we have changed what people
> are +1’ing.
>
> At the minimum I think we could have people re-affirm their votes on this
> thread based on the new proposal
>
> Once we figure that out then +1 from me :)
>
> On Wed, Mar 3, 2021 at 1:44 PM, Jarek Potiuk <ja...@potiuk.com> wrote:
>
> Hello Everyone,
>
> We need one more PMC member vote in order to be able to release the
> packages.
>
> Just to describe what the current status of the batch is:
>
> Based on discussion here: https://github.com/apache/airflow/issues/14511
> - I am planning to follow the process we had previously documented in our
> release procedures so far - that release manager tries to batch a number of
> providers in single "voting process" and when there are problems discovered
> with certain providers they might be excluded.
>
> I plan to skip the following providers from release now and release them
> together on a ad-hoc basis whenever all the relevant issues are merged:
>
> * apache.druid, microsoft.azure, apache.beam
>
> I also noticed that snowflake python connector has been released this
> morning as promised
> https://pypi.org/project/snowflake-connector-python/2.4.0/ - it fixes the
> last problem with the dependencies that were plaguing us, so I also plan to
> remove the snowflake provider from this batch.
>
>
> ---------
>
> I just wanted to use the opportunity to describe the current process for
> deciding providers, because apparently not everyone is aware that we
> already have an established and documented process for provider's releases
> that was discussed on the devlist and it is documented in our release
> process description.
>
> Possibly we will extract it into a separate policy and maybe discuss some
> aspects of it (the discussion was raised today at the dev call of ours but
> I wanted to make sure that we are all referring to the same "starting
> point" and the "process" I based my recent actions on.
>
> *1) Batching the providers as the default*
>
> The decision on when to release and particularly preference for releasing
> providers in batches is described in the
> https://github.com/apache/airflow/blob/master/dev/README_RELEASE_PROVIDER_PACKAGES.md#decide-when-to-release
>
> > Decide when to release
> > You can release provider packages separately from the main Airflow on an
> ad-hoc basis, whenever we find that
> a given provider needs to be released - due to new features or due to bug
> fixes.
> You can release each provider package separately, but due to voting and
> release overhead we try to group
> releases of provider packages together.
>
> *2) Possibility of excluding certain packages from the release.*
>
> The possibility of excluding certain packages which (for whatever reason)
> we decide to remove at the discretion of release manager is described here:
> https://github.com/apache/airflow/blob/master/dev/README_RELEASE_PROVIDER_PACKAGES.md#prepare-voting-email-for-providers-release-candidate
>
> Prepare voting email for Providers release candidate
> ....
>
> > Due to the nature of packages, not all packages have to be released as
> convenience packages in the final release. During the voting process the
> voting PMCs might decide to exclude certain packages from the release if
> some critical problems have been found in some packages.
>
> And the process of removing it is part of the described release process:
>
> > In case you decided to remove some of the packages. remove them from
> dist folder now:
>
> > ls dist/*<provider>*
> > rm dist/*<provider>*
>
> The issue of excluding certain packages have been discussed in this thread
> in the mailing list :
> https://lists.apache.org/thread.html/rc620a8a503cc7b14850c0a2a1fca1f6051d08a7e3e6d2cbdeb691dda%40%3Cdev.airflow.apache.org%3E
> - where we had a -1 veto from a PMC member on the whole batch of providers
> where we found a cncf.kubrenetes and google providers had critical
> problems.
>
> We discussed it then and the two PMC members proposed a solution that was
> not objected to by anyone in the VOTE thread - to remove the packages from
> the batch.
>
> I continued this in the continuation of the voting thread
> https://lists.apache.org/thread.html/r752c5d5171de4ff626663d30e1d50c4b0d2994f66bf8918d816dabd8%40%3Cdev.airflow.apache.org%3E
> with the following message which specifically pointed out to my proposal
> where I specifically linked to the message above and asked for comments.
>
> > As discussed before: -1 on a single provider does not invalidate the
> whole
> vote (from
>
> https://github.com/apache/airflow/tree/master/dev#vote-and-verify-the-backport-providers-release-candidate
> ):
>
> > "Due to the nature of backport packages, not all packages have to be
> released as convenience packages in the final release. During the voting
> process the voting PMCs might decide to exclude certain packages from the
> release if some critical problems have been found in some packages."
>
> > We will merge the fix and most likely release a new google package right
> after this one. Looking at the super-localized problem here my current
> decision will be to release 2020.10.29 'google package" - together with
> other packages and release 2020.11.01 (or smth) - but only the google one -
> right after we merge the fix.
>
> > Any comments to that?
>
>
> J.
>
>
>
> On Wed, Mar 3, 2021 at 1:23 AM Kaxil Naik <ka...@gmail.com> wrote:
>
>> +1 (binding).
>>
>> Verified Signature and SHA12.
>>
>> Based on the changes (and Changelog) I can verify that the following
>> providers should work fine:
>>
>>
>> - spark
>> - kubernetes
>> - jenkins
>> - microsoft.azure
>> - mysql
>> - telegram
>> - and all the ones that just have doc changes
>>
>>
>> Regards,
>> Kaxil
>>
>> On Tue, Mar 2, 2021 at 9:01 PM Ryan Hatter <ry...@gmail.com> wrote:
>>
>>> There were some changes to the operator after my PR was merged:
>>> https://github.com/apache/airflow/blob/master/airflow/providers/google/cloud/transfers/gdrive_to_gcs.py
>>>
>>> Pak Andrey (Scuall1992 on GitHub) might be able to confirm the operator
>>> is functional.
>>>
>>> On Mar 2, 2021, at 13:16, Jarek Potiuk <ja...@potiuk.com> wrote:
>>>
>>>
>>> Hello everyone - just a reminder that we have voting (hopefully)
>>> finishing tomorrow.
>>>
>>> I'd love to get some votes for that.
>>>
>>> Just to clarify what the PMC votes mean, because I believe there were
>>> some question raised about the release process which we are going to
>>> discuss it tomorrow at the dev call but let me just express my
>>> interpretation of https://infra.apache.org/release-publishing.html
>>>
>>> PMC member vote (as I understand it) does not mean that this PMC member
>>> tested the release functionality (neither Release Manager).
>>> This merely means that the PMC member agrees that the software was
>>> released according to the requirements and process described in
>>> https://infra.apache.org/release-publishing.html and that the
>>> signatures, hash-sums and software packages are as expected by the process.
>>> This is how I interpret this part of the release process "Release
>>> managers do the mechanical work; but the PMC in general, and the PMC chair
>>> in particular (as an officer of the Foundation), are responsible for
>>> compliance with ASF requirements."
>>>
>>> My understanding is that it is not feasible (neither for Airflow nor
>>> Providers) that the PMC members (nor release manager) tests the software
>>> and all features/bugfixes. We've never done that and I believe we will
>>> never do. We are reaching out to the community to test and we make a best
>>> effort to test whatever we release automatically (unit tests, integration
>>> tests, testing if providers are installable/importable with Airflow 2.0 and
>>> latest source code of Airflow). And we hardly can do more than that.
>>>
>>> Happy to discuss it tomorrow, but in the meantime If some of the PMCs
>>> could do the review of the process and check the compliance, to be ready to
>>> cast your votes - I'd love that.
>>>
>>> J.
>>>
>>> On Tue, Mar 2, 2021 at 8:44 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>>>
>>>> Hey Ryan,
>>>>
>>>> There is no **must** in re-testing it. Providing that you tested it
>>>> before with real GSuite account is for me enough of a confirmation ;).
>>>>
>>>> J.
>>>>
>>>> On Sun, Feb 28, 2021 at 10:00 PM Abdur-Rahmaan Janhangeer <
>>>> arj.python@gmail.com> wrote:
>>>>
>>>>> Salutes for having a GSuite account just for the functionality 👍👍👍
>>>>>
>>>>> On Mon, 1 Mar 2021, 00:05 Ryan Hatter, <ry...@gmail.com> wrote:
>>>>>
>>>>>> I canceled my GSuite account when my PR for the gdrive to gcs
>>>>>> operator was approved & merged. Could anyone maybe help me ensure correct
>>>>>> functionality?
>>>>>>
>>>>>>
>>>>>> On Feb 27, 2021, at 08:48, Jarek Potiuk <ja...@potiuk.com> wrote:
>>>>>>
>>>>>>
>>>>>> I created issue, where we will track the status of tests for the
>>>>>> providers (again - it is experiment - but I'd really love to get feedback
>>>>>> on the new providers from those who contributed):
>>>>>> https://github.com/apache/airflow/issues/14511
>>>>>>
>>>>>> On Sat, Feb 27, 2021 at 4:28 PM Jarek Potiuk <ja...@potiuk.com>
>>>>>> wrote:
>>>>>>
>>>>>>>
>>>>>>> Hey all,
>>>>>>>
>>>>>>> I have just cut the new wave Airflow Providers packages. This email
>>>>>>> is calling a vote on the release,
>>>>>>> which will last for 72 hours + day for the weekend - which means
>>>>>>> that it will end on Wed 3 Mar 15:59:34 CET 2021.
>>>>>>>
>>>>>>> Consider this my (binding) +1.
>>>>>>>
>>>>>>> *KIND REQUEST*
>>>>>>>
>>>>>>> There was a recent discussion about test quality of the providers
>>>>>>> and I would like to try to address it, still keeping the batch release
>>>>>>> process every 3 weeks.
>>>>>>>
>>>>>>> We need a bit of help from the community. I have a kind request to
>>>>>>> the authors of fixes and new features. I group the providers into those
>>>>>>> that likely need more testing, and those that do not. I also added names of
>>>>>>> those who submitted the changes and are most likely to be able to verify if
>>>>>>> the RC packages are solving the problems/adding features.
>>>>>>>
>>>>>>> This is a bit of experiment (apologies for calling out) - but if we
>>>>>>> find that it works, we can automate that. I will create a separate Issue in
>>>>>>> Github where you will be able to "tick" the boxes for those providers which
>>>>>>> they are added to. It would not be a blocker if not tested, but it will be
>>>>>>> a great help if you could test the new RC provider and see if it works as
>>>>>>> expected according to your changes.
>>>>>>>
>>>>>>> Providers with new features and fixes - likely needs some testing.:
>>>>>>>
>>>>>>> * *amazon* : Cristòfol Torrens, Ruben Laguna, Arati Nagmal, Ivica
>>>>>>> Kolenkaš, JavierLopezT
>>>>>>> * *apache.druid*: Xinbin Huang
>>>>>>> * *apache.spark*: Igor Khrol
>>>>>>> * *cncf.kubernetes*: jpyen, Ash Berlin-Taylor, Daniel Imberman
>>>>>>> * *google*: Vivek Bhojawala, Xinbin Huang, Pak Andrey, uma66, Ryan
>>>>>>> Yuan, morrme, Sam Wheating, YingyingPeng22, Ryan Hatter,Tobiasz Kędzierski
>>>>>>> * *jenkins*: Maxim Lisovsky
>>>>>>> * *microsift.azure <http://microsift.azure>*: flvndh, yyu
>>>>>>> * *mysql*: Constantino Schillebeeckx
>>>>>>> * *qubole*: Xinbin Huang
>>>>>>> * *salesforce*: Jyoti Dhiman
>>>>>>> * *slack*: Igor Khrol
>>>>>>> * *tableau*: Jyoti Dhiman
>>>>>>> * *telegram*: Shekhar Sing, Adil Khashtamov
>>>>>>>
>>>>>>> Providers with doc only changes (no need to test):
>>>>>>>
>>>>>>> * apache-beam
>>>>>>> * apache-hive
>>>>>>> * dingding
>>>>>>> * docker
>>>>>>> * elasticsearch
>>>>>>> * exasol
>>>>>>> * http
>>>>>>> * neo4j
>>>>>>> * openfaas
>>>>>>> * papermill
>>>>>>> * presto
>>>>>>> * sendgrid
>>>>>>> * sftp
>>>>>>> * snowflake
>>>>>>> * sqlite
>>>>>>> * ssh
>>>>>>> *
>>>>>>>
>>>>>>>
>>>>>>> Airflow Providers are available at:
>>>>>>> https://dist.apache.org/repos/dist/dev/airflow/providers/
>>>>>>>
>>>>>>> *apache-airflow-providers-<PROVIDER>-*-bin.tar.gz* are the binary
>>>>>>> Python "sdist" release - they are also official "sources" for the
>>>>>>> provider packages.
>>>>>>>
>>>>>>> *apache_airflow_providers_<PROVIDER>-*.whl are the binary
>>>>>>> Python "wheel" release.
>>>>>>>
>>>>>>> The test procedure for PMC members who would like to test the RC
>>>>>>> candidates are described in
>>>>>>>
>>>>>>> https://github.com/apache/airflow/blob/master/dev/README_RELEASE_PROVIDER_PACKAGES.md#verify-the-release-by-pmc-members
>>>>>>>
>>>>>>> and for Contributors:
>>>>>>>
>>>>>>>
>>>>>>> https://github.com/apache/airflow/blob/master/dev/README_RELEASE_PROVIDER_PACKAGES.md#verify-by-contributors
>>>>>>>
>>>>>>>
>>>>>>> Public keys are available at:
>>>>>>> https://dist.apache.org/repos/dist/release/airflow/KEYS
>>>>>>>
>>>>>>> Please vote accordingly:
>>>>>>>
>>>>>>> [ ] +1 approve
>>>>>>> [ ] +0 no opinion
>>>>>>> [ ] -1 disapprove with the reason
>>>>>>>
>>>>>>>
>>>>>>> Only votes from PMC members are binding, but members of the
>>>>>>> community are
>>>>>>> encouraged to test the release and vote with "(non-binding)".
>>>>>>>
>>>>>>> Please note that the version number excludes the 'rcX' string.
>>>>>>> This will allow us to rename the artifact without modifying
>>>>>>> the artifact checksums when we actually release.
>>>>>>>
>>>>>>>
>>>>>>> Each of the packages contains a link to the detailed changelog. The
>>>>>>> changelogs are moved to the official airflow documentation:
>>>>>>> https://github.com/apache/airflow-site/<TODO COPY LINK TO BRANCH>
>>>>>>>
>>>>>>> <PASTE ANY HIGH-LEVEL DESCRIPTION OF THE CHANGES HERE!>
>>>>>>>
>>>>>>>
>>>>>>> Note the links to documentation from PyPI packages are not working
>>>>>>> until we merge
>>>>>>> the changes to airflow site after releasing the packages officially.
>>>>>>>
>>>>>>> https://pypi.org/project/apache-airflow-providers-amazon/1.2.0rc1/
>>>>>>>
>>>>>>> https://pypi.org/project/apache-airflow-providers-apache-beam/1.0.1rc1/
>>>>>>>
>>>>>>> https://pypi.org/project/apache-airflow-providers-apache-druid/1.1.0rc1/
>>>>>>>
>>>>>>> https://pypi.org/project/apache-airflow-providers-apache-hive/1.0.2rc1/
>>>>>>>
>>>>>>> https://pypi.org/project/apache-airflow-providers-apache-spark/1.0.2rc1/
>>>>>>>
>>>>>>> https://pypi.org/project/apache-airflow-providers-cncf-kubernetes/1.0.2rc1/
>>>>>>> https://pypi.org/project/apache-airflow-providers-dingding/1.0.2rc1/
>>>>>>> https://pypi.org/project/apache-airflow-providers-docker/1.0.2rc1/
>>>>>>>
>>>>>>> https://pypi.org/project/apache-airflow-providers-elasticsearch/1.0.2rc1/
>>>>>>> https://pypi.org/project/apache-airflow-providers-exasol/1.1.1rc1/
>>>>>>> https://pypi.org/project/apache-airflow-providers-google/2.1.0rc1/
>>>>>>> https://pypi.org/project/apache-airflow-providers-http/1.1.1rc1/
>>>>>>> https://pypi.org/project/apache-airflow-providers-jenkins/1.1.0rc1/
>>>>>>>
>>>>>>> https://pypi.org/project/apache-airflow-providers-microsoft-azure/1.2.0rc1/
>>>>>>> https://pypi.org/project/apache-airflow-providers-mysql/1.0.2rc1/
>>>>>>> https://pypi.org/project/apache-airflow-providers-neo4j/1.0.1rc1/
>>>>>>> https://pypi.org/project/apache-airflow-providers-openfaas/1.1.1rc1/
>>>>>>> https://pypi.org/project/apache-airflow-providers-papermill/1.0.2rc1/
>>>>>>> https://pypi.org/project/apache-airflow-providers-presto/1.0.2rc1/
>>>>>>> https://pypi.org/project/apache-airflow-providers-qubole/1.0.2rc1/
>>>>>>>
>>>>>>> https://pypi.org/project/apache-airflow-providers-salesforce/2.0.0rc1/
>>>>>>> https://pypi.org/project/apache-airflow-providers-sendgrid/1.0.2rc1/
>>>>>>> https://pypi.org/project/apache-airflow-providers-sftp/1.1.1rc1/
>>>>>>> https://pypi.org/project/apache-airflow-providers-slack/3.0.0rc1/
>>>>>>> https://pypi.org/project/apache-airflow-providers-snowflake/1.1.1rc1/
>>>>>>> https://pypi.org/project/apache-airflow-providers-sqlite/1.0.2rc1/
>>>>>>> https://pypi.org/project/apache-airflow-providers-ssh/1.2.0rc1/
>>>>>>> https://pypi.org/project/apache-airflow-providers-tableau/1.0.0rc1/
>>>>>>> https://pypi.org/project/apache-airflow-providers-telegram/1.0.2rc1/
>>>>>>>
>>>>>>> Cheers,
>>>>>>> J.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> +48 660 796 129 <+48660796129>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> +48 660 796 129 <+48660796129>
>>>>>>
>>>>>>
>>>>
>>>> --
>>>> +48 660 796 129 <+48660796129>
>>>>
>>>
>>>
>>> --
>>> +48 660 796 129 <+48660796129>
>>>
>>>
>
> --
> +48 660 796 129 <+48660796129>
>
>
--
+48 660 796 129
Re: [VOTE] Airflow Providers - release candidates from 2021-02-27
Posted by Daniel Imberman <da...@gmail.com>.
Hi Jarek,
I think all of this sounds fine, but I think that we should start a new vote with this understanding. I wouldn't feel comfortable assuming that any of the previous +1's are still applicable as we have changed what people are +1’ing.
At the minimum I think we could have people re-affirm their votes on this thread based on the new proposal
Once we figure that out then +1 from me :)
On Wed, Mar 3, 2021 at 1:44 PM, Jarek Potiuk <ja...@potiuk.com> wrote:
Hello Everyone,
We need one more PMC member vote in order to be able to release the packages.
Just to describe what the current status of the batch is:
Based on discussion here: https://github.com/apache/airflow/issues/14511 [https://github.com/apache/airflow/issues/14511] - I am planning to follow the process we had previously documented in our release procedures so far - that release manager tries to batch a number of providers in single "voting process" and when there are problems discovered with certain providers they might be excluded.
I plan to skip the following providers from release now and release them together on a ad-hoc basis whenever all the relevant issues are merged:
* apache.druid, microsoft.azure, apache.beam
I also noticed that snowflake python connector has been released this morning as promised https://pypi.org/project/snowflake-connector-python/2.4.0/ [https://pypi.org/project/snowflake-connector-python/2.4.0/] - it fixes the last problem with the dependencies that were plaguing us, so I also plan to remove the snowflake provider from this batch.
---------
I just wanted to use the opportunity to describe the current process for deciding providers, because apparently not everyone is aware that we already have an established and documented process for provider's releases that was discussed on the devlist and it is documented in our release process description.
Possibly we will extract it into a separate policy and maybe discuss some aspects of it (the discussion was raised today at the dev call of ours but I wanted to make sure that we are all referring to the same "starting point" and the "process" I based my recent actions on.
1) Batching the providers as the default
The decision on when to release and particularly preference for releasing providers in batches is described in the https://github.com/apache/airflow/blob/master/dev/README_RELEASE_PROVIDER_PACKAGES.md#decide-when-to-release [https://github.com/apache/airflow/blob/master/dev/README_RELEASE_PROVIDER_PACKAGES.md#decide-when-to-release]
> Decide when to release
> You can release provider packages separately from the main Airflow on an ad-hoc basis, whenever we find that
a given provider needs to be released - due to new features or due to bug fixes.
You can release each provider package separately, but due to voting and release overhead we try to group
releases of provider packages together.
2) Possibility of excluding certain packages from the release.
The possibility of excluding certain packages which (for whatever reason) we decide to remove at the discretion of release manager is described here: https://github.com/apache/airflow/blob/master/dev/README_RELEASE_PROVIDER_PACKAGES.md#prepare-voting-email-for-providers-release-candidate [https://github.com/apache/airflow/blob/master/dev/README_RELEASE_PROVIDER_PACKAGES.md#prepare-voting-email-for-providers-release-candidate]
Prepare voting email for Providers release candidate
....
> Due to the nature of packages, not all packages have to be released as convenience packages in the final release. During the voting process the voting PMCs might decide to exclude certain packages from the release if some critical problems have been found in some packages.
And the process of removing it is part of the described release process:
> In case you decided to remove some of the packages. remove them from dist folder now:
> ls dist/*<provider>*
> rm dist/*<provider>*
The issue of excluding certain packages have been discussed in this thread in the mailing list : https://lists.apache.org/thread.html/rc620a8a503cc7b14850c0a2a1fca1f6051d08a7e3e6d2cbdeb691dda%40%3Cdev.airflow.apache.org%3E [https://lists.apache.org/thread.html/rc620a8a503cc7b14850c0a2a1fca1f6051d08a7e3e6d2cbdeb691dda%40%3Cdev.airflow.apache.org%3E] - where we had a -1 veto from a PMC member on the whole batch of providers where we found a cncf.kubrenetes and google providers had critical problems.
We discussed it then and the two PMC members proposed a solution that was not objected to by anyone in the VOTE thread - to remove the packages from the batch.
I continued this in the continuation of the voting thread https://lists.apache.org/thread.html/r752c5d5171de4ff626663d30e1d50c4b0d2994f66bf8918d816dabd8%40%3Cdev.airflow.apache.org%3E [https://lists.apache.org/thread.html/r752c5d5171de4ff626663d30e1d50c4b0d2994f66bf8918d816dabd8%40%3Cdev.airflow.apache.org%3E] with the following message which specifically pointed out to my proposal where I specifically linked to the message above and asked for comments.
> As discussed before: -1 on a single provider does not invalidate the whole
vote (from
https://github.com/apache/airflow/tree/master/dev#vote-and-verify-the-backport-providers-release-candidate [https://github.com/apache/airflow/tree/master/dev#vote-and-verify-the-backport-providers-release-candidate]
):
> "Due to the nature of backport packages, not all packages have to be
released as convenience packages in the final release. During the voting
process the voting PMCs might decide to exclude certain packages from the
release if some critical problems have been found in some packages."
> We will merge the fix and most likely release a new google package right
after this one. Looking at the super-localized problem here my current
decision will be to release 2020.10.29 'google package" - together with
other packages and release 2020.11.01 (or smth) - but only the google one -
right after we merge the fix.
> Any comments to that?
J.
On Wed, Mar 3, 2021 at 1:23 AM Kaxil Naik < kaxilnaik@gmail.com [kaxilnaik@gmail.com] > wrote:
+1 (binding).
Verified Signature and SHA12.
Based on the changes (and Changelog) I can verify that the following providers should work fine:
* spark
* kubernetes
* jenkins
* microsoft.azure
* mysql
* telegram
* and all the ones that just have doc changes
Regards, Kaxil
On Tue, Mar 2, 2021 at 9:01 PM Ryan Hatter < ryannhatter@gmail.com [ryannhatter@gmail.com] > wrote:
There were some changes to the operator after my PR was merged: https://github.com/apache/airflow/blob/master/airflow/providers/google/cloud/transfers/gdrive_to_gcs.py [https://github.com/apache/airflow/blob/master/airflow/providers/google/cloud/transfers/gdrive_to_gcs.py]
Pak Andrey (Scuall1992 on GitHub) might be able to confirm the operator is functional.
On Mar 2, 2021, at 13:16, Jarek Potiuk < jarek@potiuk.com [jarek@potiuk.com] > wrote:
Hello everyone - just a reminder that we have voting (hopefully) finishing tomorrow.
I'd love to get some votes for that.
Just to clarify what the PMC votes mean, because I believe there were some question raised about the release process which we are going to discuss it tomorrow at the dev call but let me just express my interpretation of https://infra.apache.org/release-publishing.html [https://infra.apache.org/release-publishing.html]
PMC member vote (as I understand it) does not mean that this PMC member tested the release functionality (neither Release Manager). This merely means that the PMC member agrees that the software was released according to the requirements and process described in https://infra.apache.org/release-publishing.html [https://infra.apache.org/release-publishing.html] and that the signatures, hash-sums and software packages are as expected by the process. This is how I interpret this part of the release process "Release managers do the mechanical work; but the PMC in general, and the PMC chair in particular (as an officer of the Foundation), are responsible for compliance with ASF requirements."
My understanding is that it is not feasible (neither for Airflow nor Providers) that the PMC members (nor release manager) tests the software and all features/bugfixes. We've never done that and I believe we will never do. We are reaching out to the community to test and we make a best effort to test whatever we release automatically (unit tests, integration tests, testing if providers are installable/importable with Airflow 2.0 and latest source code of Airflow). And we hardly can do more than that.
Happy to discuss it tomorrow, but in the meantime If some of the PMCs could do the review of the process and check the compliance, to be ready to cast your votes - I'd love that.
J.
On Tue, Mar 2, 2021 at 8:44 PM Jarek Potiuk < jarek@potiuk.com [jarek@potiuk.com] > wrote:
Hey Ryan,
There is no **must** in re-testing it. Providing that you tested it before with real GSuite account is for me enough of a confirmation ;).
J.
On Sun, Feb 28, 2021 at 10:00 PM Abdur-Rahmaan Janhangeer < arj.python@gmail.com [arj.python@gmail.com] > wrote:
Salutes for having a GSuite account just for the functionality 👍👍👍
On Mon, 1 Mar 2021, 00:05 Ryan Hatter, < ryannhatter@gmail.com [ryannhatter@gmail.com] > wrote:
I canceled my GSuite account when my PR for the gdrive to gcs operator was approved & merged. Could anyone maybe help me ensure correct functionality?
On Feb 27, 2021, at 08:48, Jarek Potiuk < jarek@potiuk.com [jarek@potiuk.com] > wrote:
I created issue, where we will track the status of tests for the providers (again - it is experiment - but I'd really love to get feedback on the new providers from those who contributed): https://github.com/apache/airflow/issues/14511 [https://github.com/apache/airflow/issues/14511]
On Sat, Feb 27, 2021 at 4:28 PM Jarek Potiuk < jarek@potiuk.com [jarek@potiuk.com] > wrote:
Hey all,
I have just cut the new wave Airflow Providers packages. This email is calling a vote on the release,
which will last for 72 hours + day for the weekend - which means that it will end on Wed 3 Mar 15:59:34 CET 2021.
Consider this my (binding) +1.
KIND REQUEST
There was a recent discussion about test quality of the providers and I would like to try to address it, still keeping the batch release process every 3 weeks.
We need a bit of help from the community. I have a kind request to the authors of fixes and new features. I group the providers into those that likely need more testing, and those that do not. I also added names of those who submitted the changes and are most likely to be able to verify if the RC packages are solving the problems/adding features.
This is a bit of experiment (apologies for calling out) - but if we find that it works, we can automate that. I will create a separate Issue in Github where you will be able to "tick" the boxes for those providers which they are added to. It would not be a blocker if not tested, but it will be a great help if you could test the new RC provider and see if it works as expected according to your changes.
Providers with new features and fixes - likely needs some testing.:
* amazon : Cristòfol Torrens, Ruben Laguna, Arati Nagmal, Ivica Kolenkaš, JavierLopezT * apache.druid : Xinbin Huang * apache.spark : Igor Khrol * cncf.kubernetes : jpyen, Ash Berlin-Taylor, Daniel Imberman * google : Vivek Bhojawala, Xinbin Huang, Pak Andrey, uma66, Ryan Yuan, morrme, Sam Wheating, YingyingPeng22, Ryan Hatter,Tobiasz Kędzierski * jenkins : Maxim Lisovsky * microsift.azure : flvndh, yyu * mysql : Constantino Schillebeeckx * qubole : Xinbin Huang * salesforce : Jyoti Dhiman * slack : Igor Khrol * tableau : Jyoti Dhiman * telegram : Shekhar Sing, Adil Khashtamov
Providers with doc only changes (no need to test):
* apache-beam * apache-hive
* dingding * docker * elasticsearch * exasol * http * neo4j * openfaas * papermill * presto * sendgrid * sftp * snowflake * sqlite * ssh *
Airflow Providers are available at:
https://dist.apache.org/repos/dist/dev/airflow/providers/ [https://dist.apache.org/repos/dist/dev/airflow/providers/]
*apache-airflow-providers-<PROVIDER>-*-bin.tar.gz* are the binary
Python "sdist" release - they are also official "sources" for the provider packages.
*apache_airflow_providers_<PROVIDER>-*.whl are the binary
Python "wheel" release.
The test procedure for PMC members who would like to test the RC candidates are described in
https://github.com/apache/airflow/blob/master/dev/README_RELEASE_PROVIDER_PACKAGES.md#verify-the-release-by-pmc-members [https://github.com/apache/airflow/blob/master/dev/README_RELEASE_PROVIDER_PACKAGES.md#verify-the-release-by-pmc-members]
and for Contributors:
https://github.com/apache/airflow/blob/master/dev/README_RELEASE_PROVIDER_PACKAGES.md#verify-by-contributors [https://github.com/apache/airflow/blob/master/dev/README_RELEASE_PROVIDER_PACKAGES.md#verify-by-contributors]
Public keys are available at:
https://dist.apache.org/repos/dist/release/airflow/KEYS [https://dist.apache.org/repos/dist/release/airflow/KEYS]
Please vote accordingly:
[ ] +1 approve
[ ] +0 no opinion
[ ] -1 disapprove with the reason
Only votes from PMC members are binding, but members of the community are
encouraged to test the release and vote with "(non-binding)".
Please note that the version number excludes the 'rcX' string.
This will allow us to rename the artifact without modifying
the artifact checksums when we actually release.
Each of the packages contains a link to the detailed changelog. The changelogs are moved to the official airflow documentation:
https://github.com/apache/airflow-site/ [https://github.com/apache/airflow-site/] <TODO COPY LINK TO BRANCH>
<PASTE ANY HIGH-LEVEL DESCRIPTION OF THE CHANGES HERE!>
Note the links to documentation from PyPI packages are not working until we merge
the changes to airflow site after releasing the packages officially.
https://pypi.org/project/apache-airflow-providers-amazon/1.2.0rc1/ [https://pypi.org/project/apache-airflow-providers-amazon/1.2.0rc1/]
https://pypi.org/project/apache-airflow-providers-apache-beam/1.0.1rc1/ [https://pypi.org/project/apache-airflow-providers-apache-beam/1.0.1rc1/]
https://pypi.org/project/apache-airflow-providers-apache-druid/1.1.0rc1/ [https://pypi.org/project/apache-airflow-providers-apache-druid/1.1.0rc1/]
https://pypi.org/project/apache-airflow-providers-apache-hive/1.0.2rc1/ [https://pypi.org/project/apache-airflow-providers-apache-hive/1.0.2rc1/]
https://pypi.org/project/apache-airflow-providers-apache-spark/1.0.2rc1/ [https://pypi.org/project/apache-airflow-providers-apache-spark/1.0.2rc1/]
https://pypi.org/project/apache-airflow-providers-cncf-kubernetes/1.0.2rc1/ [https://pypi.org/project/apache-airflow-providers-cncf-kubernetes/1.0.2rc1/]
https://pypi.org/project/apache-airflow-providers-dingding/1.0.2rc1/ [https://pypi.org/project/apache-airflow-providers-dingding/1.0.2rc1/]
https://pypi.org/project/apache-airflow-providers-docker/1.0.2rc1/ [https://pypi.org/project/apache-airflow-providers-docker/1.0.2rc1/]
https://pypi.org/project/apache-airflow-providers-elasticsearch/1.0.2rc1/ [https://pypi.org/project/apache-airflow-providers-elasticsearch/1.0.2rc1/]
https://pypi.org/project/apache-airflow-providers-exasol/1.1.1rc1/ [https://pypi.org/project/apache-airflow-providers-exasol/1.1.1rc1/]
https://pypi.org/project/apache-airflow-providers-google/2.1.0rc1/ [https://pypi.org/project/apache-airflow-providers-google/2.1.0rc1/]
https://pypi.org/project/apache-airflow-providers-http/1.1.1rc1/ [https://pypi.org/project/apache-airflow-providers-http/1.1.1rc1/]
https://pypi.org/project/apache-airflow-providers-jenkins/1.1.0rc1/ [https://pypi.org/project/apache-airflow-providers-jenkins/1.1.0rc1/]
https://pypi.org/project/apache-airflow-providers-microsoft-azure/1.2.0rc1/ [https://pypi.org/project/apache-airflow-providers-microsoft-azure/1.2.0rc1/]
https://pypi.org/project/apache-airflow-providers-mysql/1.0.2rc1/ [https://pypi.org/project/apache-airflow-providers-mysql/1.0.2rc1/]
https://pypi.org/project/apache-airflow-providers-neo4j/1.0.1rc1/ [https://pypi.org/project/apache-airflow-providers-neo4j/1.0.1rc1/]
https://pypi.org/project/apache-airflow-providers-openfaas/1.1.1rc1/ [https://pypi.org/project/apache-airflow-providers-openfaas/1.1.1rc1/]
https://pypi.org/project/apache-airflow-providers-papermill/1.0.2rc1/ [https://pypi.org/project/apache-airflow-providers-papermill/1.0.2rc1/]
https://pypi.org/project/apache-airflow-providers-presto/1.0.2rc1/ [https://pypi.org/project/apache-airflow-providers-presto/1.0.2rc1/]
https://pypi.org/project/apache-airflow-providers-qubole/1.0.2rc1/ [https://pypi.org/project/apache-airflow-providers-qubole/1.0.2rc1/]
https://pypi.org/project/apache-airflow-providers-salesforce/2.0.0rc1/ [https://pypi.org/project/apache-airflow-providers-salesforce/2.0.0rc1/]
https://pypi.org/project/apache-airflow-providers-sendgrid/1.0.2rc1/ [https://pypi.org/project/apache-airflow-providers-sendgrid/1.0.2rc1/]
https://pypi.org/project/apache-airflow-providers-sftp/1.1.1rc1/ [https://pypi.org/project/apache-airflow-providers-sftp/1.1.1rc1/]
https://pypi.org/project/apache-airflow-providers-slack/3.0.0rc1/ [https://pypi.org/project/apache-airflow-providers-slack/3.0.0rc1/]
https://pypi.org/project/apache-airflow-providers-snowflake/1.1.1rc1/ [https://pypi.org/project/apache-airflow-providers-snowflake/1.1.1rc1/]
https://pypi.org/project/apache-airflow-providers-sqlite/1.0.2rc1/ [https://pypi.org/project/apache-airflow-providers-sqlite/1.0.2rc1/]
https://pypi.org/project/apache-airflow-providers-ssh/1.2.0rc1/ [https://pypi.org/project/apache-airflow-providers-ssh/1.2.0rc1/]
https://pypi.org/project/apache-airflow-providers-tableau/1.0.0rc1/ [https://pypi.org/project/apache-airflow-providers-tableau/1.0.0rc1/]
https://pypi.org/project/apache-airflow-providers-telegram/1.0.2rc1/ [https://pypi.org/project/apache-airflow-providers-telegram/1.0.2rc1/]
Cheers,
J.
--
+48 660 796 129
--
+48 660 796 129
--
+48 660 796 129
--
+48 660 796 129
--
+48 660 796 129
Re: [VOTE] Airflow Providers - release candidates from 2021-02-27
Posted by Jarek Potiuk <ja...@potiuk.com>.
Hello Everyone,
We need one more PMC member vote in order to be able to release the
packages.
Just to describe what the current status of the batch is:
Based on discussion here: https://github.com/apache/airflow/issues/14511 -
I am planning to follow the process we had previously documented in our
release procedures so far - that release manager tries to batch a number of
providers in single "voting process" and when there are problems discovered
with certain providers they might be excluded.
I plan to skip the following providers from release now and release them
together on a ad-hoc basis whenever all the relevant issues are merged:
* apache.druid, microsoft.azure, apache.beam
I also noticed that snowflake python connector has been released this
morning as promised
https://pypi.org/project/snowflake-connector-python/2.4.0/ - it fixes
the last problem with the dependencies that were plaguing us, so I also
plan to remove the snowflake provider from this batch.
---------
I just wanted to use the opportunity to describe the current process for
deciding providers, because apparently not everyone is aware that we
already have an established and documented process for provider's releases
that was discussed on the devlist and it is documented in our release
process description.
Possibly we will extract it into a separate policy and maybe discuss some
aspects of it (the discussion was raised today at the dev call of ours but
I wanted to make sure that we are all referring to the same "starting
point" and the "process" I based my recent actions on.
*1) Batching the providers as the default*
The decision on when to release and particularly preference for releasing
providers in batches is described in the
https://github.com/apache/airflow/blob/master/dev/README_RELEASE_PROVIDER_PACKAGES.md#decide-when-to-release
> Decide when to release
> You can release provider packages separately from the main Airflow on an
ad-hoc basis, whenever we find that
a given provider needs to be released - due to new features or due to bug
fixes.
You can release each provider package separately, but due to voting and
release overhead we try to group
releases of provider packages together.
*2) Possibility of excluding certain packages from the release.*
The possibility of excluding certain packages which (for whatever reason)
we decide to remove at the discretion of release manager is described here:
https://github.com/apache/airflow/blob/master/dev/README_RELEASE_PROVIDER_PACKAGES.md#prepare-voting-email-for-providers-release-candidate
Prepare voting email for Providers release candidate
....
> Due to the nature of packages, not all packages have to be released as
convenience packages in the final release. During the voting process the
voting PMCs might decide to exclude certain packages from the release if
some critical problems have been found in some packages.
And the process of removing it is part of the described release process:
> In case you decided to remove some of the packages. remove them from dist
folder now:
> ls dist/*<provider>*
> rm dist/*<provider>*
The issue of excluding certain packages have been discussed in this thread
in the mailing list :
https://lists.apache.org/thread.html/rc620a8a503cc7b14850c0a2a1fca1f6051d08a7e3e6d2cbdeb691dda%40%3Cdev.airflow.apache.org%3E
- where we had a -1 veto from a PMC member on the whole batch of providers
where we found a cncf.kubrenetes and google providers had critical
problems.
We discussed it then and the two PMC members proposed a solution that was
not objected to by anyone in the VOTE thread - to remove the packages from
the batch.
I continued this in the continuation of the voting thread
https://lists.apache.org/thread.html/r752c5d5171de4ff626663d30e1d50c4b0d2994f66bf8918d816dabd8%40%3Cdev.airflow.apache.org%3E
with the following message which specifically pointed out to my proposal
where I specifically linked to the message above and asked for comments.
> As discussed before: -1 on a single provider does not invalidate the whole
vote (from
https://github.com/apache/airflow/tree/master/dev#vote-and-verify-the-backport-providers-release-candidate
):
> "Due to the nature of backport packages, not all packages have to be
released as convenience packages in the final release. During the voting
process the voting PMCs might decide to exclude certain packages from the
release if some critical problems have been found in some packages."
> We will merge the fix and most likely release a new google package right
after this one. Looking at the super-localized problem here my current
decision will be to release 2020.10.29 'google package" - together with
other packages and release 2020.11.01 (or smth) - but only the google one -
right after we merge the fix.
> Any comments to that?
J.
On Wed, Mar 3, 2021 at 1:23 AM Kaxil Naik <ka...@gmail.com> wrote:
> +1 (binding).
>
> Verified Signature and SHA12.
>
> Based on the changes (and Changelog) I can verify that the following
> providers should work fine:
>
>
> - spark
> - kubernetes
> - jenkins
> - microsoft.azure
> - mysql
> - telegram
> - and all the ones that just have doc changes
>
>
> Regards,
> Kaxil
>
> On Tue, Mar 2, 2021 at 9:01 PM Ryan Hatter <ry...@gmail.com> wrote:
>
>> There were some changes to the operator after my PR was merged:
>> https://github.com/apache/airflow/blob/master/airflow/providers/google/cloud/transfers/gdrive_to_gcs.py
>>
>> Pak Andrey (Scuall1992 on GitHub) might be able to confirm the operator
>> is functional.
>>
>> On Mar 2, 2021, at 13:16, Jarek Potiuk <ja...@potiuk.com> wrote:
>>
>>
>> Hello everyone - just a reminder that we have voting (hopefully)
>> finishing tomorrow.
>>
>> I'd love to get some votes for that.
>>
>> Just to clarify what the PMC votes mean, because I believe there were
>> some question raised about the release process which we are going to
>> discuss it tomorrow at the dev call but let me just express my
>> interpretation of https://infra.apache.org/release-publishing.html
>>
>> PMC member vote (as I understand it) does not mean that this PMC member
>> tested the release functionality (neither Release Manager).
>> This merely means that the PMC member agrees that the software was
>> released according to the requirements and process described in
>> https://infra.apache.org/release-publishing.html and that the
>> signatures, hash-sums and software packages are as expected by the process.
>> This is how I interpret this part of the release process "Release
>> managers do the mechanical work; but the PMC in general, and the PMC chair
>> in particular (as an officer of the Foundation), are responsible for
>> compliance with ASF requirements."
>>
>> My understanding is that it is not feasible (neither for Airflow nor
>> Providers) that the PMC members (nor release manager) tests the software
>> and all features/bugfixes. We've never done that and I believe we will
>> never do. We are reaching out to the community to test and we make a best
>> effort to test whatever we release automatically (unit tests, integration
>> tests, testing if providers are installable/importable with Airflow 2.0 and
>> latest source code of Airflow). And we hardly can do more than that.
>>
>> Happy to discuss it tomorrow, but in the meantime If some of the PMCs
>> could do the review of the process and check the compliance, to be ready to
>> cast your votes - I'd love that.
>>
>> J.
>>
>> On Tue, Mar 2, 2021 at 8:44 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>>
>>> Hey Ryan,
>>>
>>> There is no **must** in re-testing it. Providing that you tested it
>>> before with real GSuite account is for me enough of a confirmation ;).
>>>
>>> J.
>>>
>>> On Sun, Feb 28, 2021 at 10:00 PM Abdur-Rahmaan Janhangeer <
>>> arj.python@gmail.com> wrote:
>>>
>>>> Salutes for having a GSuite account just for the functionality 👍👍👍
>>>>
>>>> On Mon, 1 Mar 2021, 00:05 Ryan Hatter, <ry...@gmail.com> wrote:
>>>>
>>>>> I canceled my GSuite account when my PR for the gdrive to gcs operator
>>>>> was approved & merged. Could anyone maybe help me ensure correct
>>>>> functionality?
>>>>>
>>>>>
>>>>> On Feb 27, 2021, at 08:48, Jarek Potiuk <ja...@potiuk.com> wrote:
>>>>>
>>>>>
>>>>> I created issue, where we will track the status of tests for the
>>>>> providers (again - it is experiment - but I'd really love to get feedback
>>>>> on the new providers from those who contributed):
>>>>> https://github.com/apache/airflow/issues/14511
>>>>>
>>>>> On Sat, Feb 27, 2021 at 4:28 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>>>>>
>>>>>>
>>>>>> Hey all,
>>>>>>
>>>>>> I have just cut the new wave Airflow Providers packages. This email
>>>>>> is calling a vote on the release,
>>>>>> which will last for 72 hours + day for the weekend - which means
>>>>>> that it will end on Wed 3 Mar 15:59:34 CET 2021.
>>>>>>
>>>>>> Consider this my (binding) +1.
>>>>>>
>>>>>> *KIND REQUEST*
>>>>>>
>>>>>> There was a recent discussion about test quality of the providers and
>>>>>> I would like to try to address it, still keeping the batch release process
>>>>>> every 3 weeks.
>>>>>>
>>>>>> We need a bit of help from the community. I have a kind request to
>>>>>> the authors of fixes and new features. I group the providers into those
>>>>>> that likely need more testing, and those that do not. I also added names of
>>>>>> those who submitted the changes and are most likely to be able to verify if
>>>>>> the RC packages are solving the problems/adding features.
>>>>>>
>>>>>> This is a bit of experiment (apologies for calling out) - but if we
>>>>>> find that it works, we can automate that. I will create a separate Issue in
>>>>>> Github where you will be able to "tick" the boxes for those providers which
>>>>>> they are added to. It would not be a blocker if not tested, but it will be
>>>>>> a great help if you could test the new RC provider and see if it works as
>>>>>> expected according to your changes.
>>>>>>
>>>>>> Providers with new features and fixes - likely needs some testing.:
>>>>>>
>>>>>> * *amazon* : Cristòfol Torrens, Ruben Laguna, Arati Nagmal, Ivica
>>>>>> Kolenkaš, JavierLopezT
>>>>>> * *apache.druid*: Xinbin Huang
>>>>>> * *apache.spark*: Igor Khrol
>>>>>> * *cncf.kubernetes*: jpyen, Ash Berlin-Taylor, Daniel Imberman
>>>>>> * *google*: Vivek Bhojawala, Xinbin Huang, Pak Andrey, uma66, Ryan
>>>>>> Yuan, morrme, Sam Wheating, YingyingPeng22, Ryan Hatter,Tobiasz Kędzierski
>>>>>> * *jenkins*: Maxim Lisovsky
>>>>>> * *microsift.azure*: flvndh, yyu
>>>>>> * *mysql*: Constantino Schillebeeckx
>>>>>> * *qubole*: Xinbin Huang
>>>>>> * *salesforce*: Jyoti Dhiman
>>>>>> * *slack*: Igor Khrol
>>>>>> * *tableau*: Jyoti Dhiman
>>>>>> * *telegram*: Shekhar Sing, Adil Khashtamov
>>>>>>
>>>>>> Providers with doc only changes (no need to test):
>>>>>>
>>>>>> * apache-beam
>>>>>> * apache-hive
>>>>>> * dingding
>>>>>> * docker
>>>>>> * elasticsearch
>>>>>> * exasol
>>>>>> * http
>>>>>> * neo4j
>>>>>> * openfaas
>>>>>> * papermill
>>>>>> * presto
>>>>>> * sendgrid
>>>>>> * sftp
>>>>>> * snowflake
>>>>>> * sqlite
>>>>>> * ssh
>>>>>> *
>>>>>>
>>>>>>
>>>>>> Airflow Providers are available at:
>>>>>> https://dist.apache.org/repos/dist/dev/airflow/providers/
>>>>>>
>>>>>> *apache-airflow-providers-<PROVIDER>-*-bin.tar.gz* are the binary
>>>>>> Python "sdist" release - they are also official "sources" for the
>>>>>> provider packages.
>>>>>>
>>>>>> *apache_airflow_providers_<PROVIDER>-*.whl are the binary
>>>>>> Python "wheel" release.
>>>>>>
>>>>>> The test procedure for PMC members who would like to test the RC
>>>>>> candidates are described in
>>>>>>
>>>>>> https://github.com/apache/airflow/blob/master/dev/README_RELEASE_PROVIDER_PACKAGES.md#verify-the-release-by-pmc-members
>>>>>>
>>>>>> and for Contributors:
>>>>>>
>>>>>>
>>>>>> https://github.com/apache/airflow/blob/master/dev/README_RELEASE_PROVIDER_PACKAGES.md#verify-by-contributors
>>>>>>
>>>>>>
>>>>>> Public keys are available at:
>>>>>> https://dist.apache.org/repos/dist/release/airflow/KEYS
>>>>>>
>>>>>> Please vote accordingly:
>>>>>>
>>>>>> [ ] +1 approve
>>>>>> [ ] +0 no opinion
>>>>>> [ ] -1 disapprove with the reason
>>>>>>
>>>>>>
>>>>>> Only votes from PMC members are binding, but members of the community
>>>>>> are
>>>>>> encouraged to test the release and vote with "(non-binding)".
>>>>>>
>>>>>> Please note that the version number excludes the 'rcX' string.
>>>>>> This will allow us to rename the artifact without modifying
>>>>>> the artifact checksums when we actually release.
>>>>>>
>>>>>>
>>>>>> Each of the packages contains a link to the detailed changelog. The
>>>>>> changelogs are moved to the official airflow documentation:
>>>>>> https://github.com/apache/airflow-site/<TODO COPY LINK TO BRANCH>
>>>>>>
>>>>>> <PASTE ANY HIGH-LEVEL DESCRIPTION OF THE CHANGES HERE!>
>>>>>>
>>>>>>
>>>>>> Note the links to documentation from PyPI packages are not working
>>>>>> until we merge
>>>>>> the changes to airflow site after releasing the packages officially.
>>>>>>
>>>>>> https://pypi.org/project/apache-airflow-providers-amazon/1.2.0rc1/
>>>>>>
>>>>>> https://pypi.org/project/apache-airflow-providers-apache-beam/1.0.1rc1/
>>>>>>
>>>>>> https://pypi.org/project/apache-airflow-providers-apache-druid/1.1.0rc1/
>>>>>>
>>>>>> https://pypi.org/project/apache-airflow-providers-apache-hive/1.0.2rc1/
>>>>>>
>>>>>> https://pypi.org/project/apache-airflow-providers-apache-spark/1.0.2rc1/
>>>>>>
>>>>>> https://pypi.org/project/apache-airflow-providers-cncf-kubernetes/1.0.2rc1/
>>>>>> https://pypi.org/project/apache-airflow-providers-dingding/1.0.2rc1/
>>>>>> https://pypi.org/project/apache-airflow-providers-docker/1.0.2rc1/
>>>>>>
>>>>>> https://pypi.org/project/apache-airflow-providers-elasticsearch/1.0.2rc1/
>>>>>> https://pypi.org/project/apache-airflow-providers-exasol/1.1.1rc1/
>>>>>> https://pypi.org/project/apache-airflow-providers-google/2.1.0rc1/
>>>>>> https://pypi.org/project/apache-airflow-providers-http/1.1.1rc1/
>>>>>> https://pypi.org/project/apache-airflow-providers-jenkins/1.1.0rc1/
>>>>>>
>>>>>> https://pypi.org/project/apache-airflow-providers-microsoft-azure/1.2.0rc1/
>>>>>> https://pypi.org/project/apache-airflow-providers-mysql/1.0.2rc1/
>>>>>> https://pypi.org/project/apache-airflow-providers-neo4j/1.0.1rc1/
>>>>>> https://pypi.org/project/apache-airflow-providers-openfaas/1.1.1rc1/
>>>>>> https://pypi.org/project/apache-airflow-providers-papermill/1.0.2rc1/
>>>>>> https://pypi.org/project/apache-airflow-providers-presto/1.0.2rc1/
>>>>>> https://pypi.org/project/apache-airflow-providers-qubole/1.0.2rc1/
>>>>>> https://pypi.org/project/apache-airflow-providers-salesforce/2.0.0rc1/
>>>>>> https://pypi.org/project/apache-airflow-providers-sendgrid/1.0.2rc1/
>>>>>> https://pypi.org/project/apache-airflow-providers-sftp/1.1.1rc1/
>>>>>> https://pypi.org/project/apache-airflow-providers-slack/3.0.0rc1/
>>>>>> https://pypi.org/project/apache-airflow-providers-snowflake/1.1.1rc1/
>>>>>> https://pypi.org/project/apache-airflow-providers-sqlite/1.0.2rc1/
>>>>>> https://pypi.org/project/apache-airflow-providers-ssh/1.2.0rc1/
>>>>>> https://pypi.org/project/apache-airflow-providers-tableau/1.0.0rc1/
>>>>>> https://pypi.org/project/apache-airflow-providers-telegram/1.0.2rc1/
>>>>>>
>>>>>> Cheers,
>>>>>> J.
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> +48 660 796 129
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> +48 660 796 129
>>>>>
>>>>>
>>>
>>> --
>>> +48 660 796 129
>>>
>>
>>
>> --
>> +48 660 796 129
>>
>>
--
+48 660 796 129
Re: [VOTE] Airflow Providers - release candidates from 2021-02-27
Posted by Kaxil Naik <ka...@gmail.com>.
+1 (binding).
Verified Signature and SHA12.
Based on the changes (and Changelog) I can verify that the following
providers should work fine:
- spark
- kubernetes
- jenkins
- microsoft.azure
- mysql
- telegram
- and all the ones that just have doc changes
Regards,
Kaxil
On Tue, Mar 2, 2021 at 9:01 PM Ryan Hatter <ry...@gmail.com> wrote:
> There were some changes to the operator after my PR was merged:
> https://github.com/apache/airflow/blob/master/airflow/providers/google/cloud/transfers/gdrive_to_gcs.py
>
> Pak Andrey (Scuall1992 on GitHub) might be able to confirm the operator is
> functional.
>
> On Mar 2, 2021, at 13:16, Jarek Potiuk <ja...@potiuk.com> wrote:
>
>
> Hello everyone - just a reminder that we have voting (hopefully) finishing
> tomorrow.
>
> I'd love to get some votes for that.
>
> Just to clarify what the PMC votes mean, because I believe there were some
> question raised about the release process which we are going to discuss it
> tomorrow at the dev call but let me just express my interpretation of
> https://infra.apache.org/release-publishing.html
>
> PMC member vote (as I understand it) does not mean that this PMC member
> tested the release functionality (neither Release Manager).
> This merely means that the PMC member agrees that the software was
> released according to the requirements and process described in
> https://infra.apache.org/release-publishing.html and that the signatures,
> hash-sums and software packages are as expected by the process.
> This is how I interpret this part of the release process "Release managers
> do the mechanical work; but the PMC in general, and the PMC chair in
> particular (as an officer of the Foundation), are responsible for
> compliance with ASF requirements."
>
> My understanding is that it is not feasible (neither for Airflow nor
> Providers) that the PMC members (nor release manager) tests the software
> and all features/bugfixes. We've never done that and I believe we will
> never do. We are reaching out to the community to test and we make a best
> effort to test whatever we release automatically (unit tests, integration
> tests, testing if providers are installable/importable with Airflow 2.0 and
> latest source code of Airflow). And we hardly can do more than that.
>
> Happy to discuss it tomorrow, but in the meantime If some of the PMCs
> could do the review of the process and check the compliance, to be ready to
> cast your votes - I'd love that.
>
> J.
>
> On Tue, Mar 2, 2021 at 8:44 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>
>> Hey Ryan,
>>
>> There is no **must** in re-testing it. Providing that you tested it
>> before with real GSuite account is for me enough of a confirmation ;).
>>
>> J.
>>
>> On Sun, Feb 28, 2021 at 10:00 PM Abdur-Rahmaan Janhangeer <
>> arj.python@gmail.com> wrote:
>>
>>> Salutes for having a GSuite account just for the functionality 👍👍👍
>>>
>>> On Mon, 1 Mar 2021, 00:05 Ryan Hatter, <ry...@gmail.com> wrote:
>>>
>>>> I canceled my GSuite account when my PR for the gdrive to gcs operator
>>>> was approved & merged. Could anyone maybe help me ensure correct
>>>> functionality?
>>>>
>>>>
>>>> On Feb 27, 2021, at 08:48, Jarek Potiuk <ja...@potiuk.com> wrote:
>>>>
>>>>
>>>> I created issue, where we will track the status of tests for the
>>>> providers (again - it is experiment - but I'd really love to get feedback
>>>> on the new providers from those who contributed):
>>>> https://github.com/apache/airflow/issues/14511
>>>>
>>>> On Sat, Feb 27, 2021 at 4:28 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>>>>
>>>>>
>>>>> Hey all,
>>>>>
>>>>> I have just cut the new wave Airflow Providers packages. This email is
>>>>> calling a vote on the release,
>>>>> which will last for 72 hours + day for the weekend - which means that
>>>>> it will end on Wed 3 Mar 15:59:34 CET 2021.
>>>>>
>>>>> Consider this my (binding) +1.
>>>>>
>>>>> *KIND REQUEST*
>>>>>
>>>>> There was a recent discussion about test quality of the providers and
>>>>> I would like to try to address it, still keeping the batch release process
>>>>> every 3 weeks.
>>>>>
>>>>> We need a bit of help from the community. I have a kind request to the
>>>>> authors of fixes and new features. I group the providers into those that
>>>>> likely need more testing, and those that do not. I also added names of
>>>>> those who submitted the changes and are most likely to be able to verify if
>>>>> the RC packages are solving the problems/adding features.
>>>>>
>>>>> This is a bit of experiment (apologies for calling out) - but if we
>>>>> find that it works, we can automate that. I will create a separate Issue in
>>>>> Github where you will be able to "tick" the boxes for those providers which
>>>>> they are added to. It would not be a blocker if not tested, but it will be
>>>>> a great help if you could test the new RC provider and see if it works as
>>>>> expected according to your changes.
>>>>>
>>>>> Providers with new features and fixes - likely needs some testing.:
>>>>>
>>>>> * *amazon* : Cristòfol Torrens, Ruben Laguna, Arati Nagmal, Ivica
>>>>> Kolenkaš, JavierLopezT
>>>>> * *apache.druid*: Xinbin Huang
>>>>> * *apache.spark*: Igor Khrol
>>>>> * *cncf.kubernetes*: jpyen, Ash Berlin-Taylor, Daniel Imberman
>>>>> * *google*: Vivek Bhojawala, Xinbin Huang, Pak Andrey, uma66, Ryan
>>>>> Yuan, morrme, Sam Wheating, YingyingPeng22, Ryan Hatter,Tobiasz Kędzierski
>>>>> * *jenkins*: Maxim Lisovsky
>>>>> * *microsift.azure*: flvndh, yyu
>>>>> * *mysql*: Constantino Schillebeeckx
>>>>> * *qubole*: Xinbin Huang
>>>>> * *salesforce*: Jyoti Dhiman
>>>>> * *slack*: Igor Khrol
>>>>> * *tableau*: Jyoti Dhiman
>>>>> * *telegram*: Shekhar Sing, Adil Khashtamov
>>>>>
>>>>> Providers with doc only changes (no need to test):
>>>>>
>>>>> * apache-beam
>>>>> * apache-hive
>>>>> * dingding
>>>>> * docker
>>>>> * elasticsearch
>>>>> * exasol
>>>>> * http
>>>>> * neo4j
>>>>> * openfaas
>>>>> * papermill
>>>>> * presto
>>>>> * sendgrid
>>>>> * sftp
>>>>> * snowflake
>>>>> * sqlite
>>>>> * ssh
>>>>> *
>>>>>
>>>>>
>>>>> Airflow Providers are available at:
>>>>> https://dist.apache.org/repos/dist/dev/airflow/providers/
>>>>>
>>>>> *apache-airflow-providers-<PROVIDER>-*-bin.tar.gz* are the binary
>>>>> Python "sdist" release - they are also official "sources" for the
>>>>> provider packages.
>>>>>
>>>>> *apache_airflow_providers_<PROVIDER>-*.whl are the binary
>>>>> Python "wheel" release.
>>>>>
>>>>> The test procedure for PMC members who would like to test the RC
>>>>> candidates are described in
>>>>>
>>>>> https://github.com/apache/airflow/blob/master/dev/README_RELEASE_PROVIDER_PACKAGES.md#verify-the-release-by-pmc-members
>>>>>
>>>>> and for Contributors:
>>>>>
>>>>>
>>>>> https://github.com/apache/airflow/blob/master/dev/README_RELEASE_PROVIDER_PACKAGES.md#verify-by-contributors
>>>>>
>>>>>
>>>>> Public keys are available at:
>>>>> https://dist.apache.org/repos/dist/release/airflow/KEYS
>>>>>
>>>>> Please vote accordingly:
>>>>>
>>>>> [ ] +1 approve
>>>>> [ ] +0 no opinion
>>>>> [ ] -1 disapprove with the reason
>>>>>
>>>>>
>>>>> Only votes from PMC members are binding, but members of the community
>>>>> are
>>>>> encouraged to test the release and vote with "(non-binding)".
>>>>>
>>>>> Please note that the version number excludes the 'rcX' string.
>>>>> This will allow us to rename the artifact without modifying
>>>>> the artifact checksums when we actually release.
>>>>>
>>>>>
>>>>> Each of the packages contains a link to the detailed changelog. The
>>>>> changelogs are moved to the official airflow documentation:
>>>>> https://github.com/apache/airflow-site/<TODO COPY LINK TO BRANCH>
>>>>>
>>>>> <PASTE ANY HIGH-LEVEL DESCRIPTION OF THE CHANGES HERE!>
>>>>>
>>>>>
>>>>> Note the links to documentation from PyPI packages are not working
>>>>> until we merge
>>>>> the changes to airflow site after releasing the packages officially.
>>>>>
>>>>> https://pypi.org/project/apache-airflow-providers-amazon/1.2.0rc1/
>>>>> https://pypi.org/project/apache-airflow-providers-apache-beam/1.0.1rc1/
>>>>>
>>>>> https://pypi.org/project/apache-airflow-providers-apache-druid/1.1.0rc1/
>>>>> https://pypi.org/project/apache-airflow-providers-apache-hive/1.0.2rc1/
>>>>>
>>>>> https://pypi.org/project/apache-airflow-providers-apache-spark/1.0.2rc1/
>>>>>
>>>>> https://pypi.org/project/apache-airflow-providers-cncf-kubernetes/1.0.2rc1/
>>>>> https://pypi.org/project/apache-airflow-providers-dingding/1.0.2rc1/
>>>>> https://pypi.org/project/apache-airflow-providers-docker/1.0.2rc1/
>>>>>
>>>>> https://pypi.org/project/apache-airflow-providers-elasticsearch/1.0.2rc1/
>>>>> https://pypi.org/project/apache-airflow-providers-exasol/1.1.1rc1/
>>>>> https://pypi.org/project/apache-airflow-providers-google/2.1.0rc1/
>>>>> https://pypi.org/project/apache-airflow-providers-http/1.1.1rc1/
>>>>> https://pypi.org/project/apache-airflow-providers-jenkins/1.1.0rc1/
>>>>>
>>>>> https://pypi.org/project/apache-airflow-providers-microsoft-azure/1.2.0rc1/
>>>>> https://pypi.org/project/apache-airflow-providers-mysql/1.0.2rc1/
>>>>> https://pypi.org/project/apache-airflow-providers-neo4j/1.0.1rc1/
>>>>> https://pypi.org/project/apache-airflow-providers-openfaas/1.1.1rc1/
>>>>> https://pypi.org/project/apache-airflow-providers-papermill/1.0.2rc1/
>>>>> https://pypi.org/project/apache-airflow-providers-presto/1.0.2rc1/
>>>>> https://pypi.org/project/apache-airflow-providers-qubole/1.0.2rc1/
>>>>> https://pypi.org/project/apache-airflow-providers-salesforce/2.0.0rc1/
>>>>> https://pypi.org/project/apache-airflow-providers-sendgrid/1.0.2rc1/
>>>>> https://pypi.org/project/apache-airflow-providers-sftp/1.1.1rc1/
>>>>> https://pypi.org/project/apache-airflow-providers-slack/3.0.0rc1/
>>>>> https://pypi.org/project/apache-airflow-providers-snowflake/1.1.1rc1/
>>>>> https://pypi.org/project/apache-airflow-providers-sqlite/1.0.2rc1/
>>>>> https://pypi.org/project/apache-airflow-providers-ssh/1.2.0rc1/
>>>>> https://pypi.org/project/apache-airflow-providers-tableau/1.0.0rc1/
>>>>> https://pypi.org/project/apache-airflow-providers-telegram/1.0.2rc1/
>>>>>
>>>>> Cheers,
>>>>> J.
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> +48 660 796 129
>>>>>
>>>>
>>>>
>>>> --
>>>> +48 660 796 129
>>>>
>>>>
>>
>> --
>> +48 660 796 129
>>
>
>
> --
> +48 660 796 129
>
>
Re: [VOTE] Airflow Providers - release candidates from 2021-02-27
Posted by Ryan Hatter <ry...@gmail.com>.
There were some changes to the operator after my PR was merged: https://github.com/apache/airflow/blob/master/airflow/providers/google/cloud/transfers/gdrive_to_gcs.py
Pak Andrey (Scuall1992 on GitHub) might be able to confirm the operator is functional.
> On Mar 2, 2021, at 13:16, Jarek Potiuk <ja...@potiuk.com> wrote:
>
> Hello everyone - just a reminder that we have voting (hopefully) finishing tomorrow.
>
> I'd love to get some votes for that.
>
> Just to clarify what the PMC votes mean, because I believe there were some question raised about the release process which we are going to discuss it tomorrow at the dev call but let me just express my interpretation of https://infra.apache.org/release-publishing.html
>
> PMC member vote (as I understand it) does not mean that this PMC member tested the release functionality (neither Release Manager).
> This merely means that the PMC member agrees that the software was released according to the requirements and process described in https://infra.apache.org/release-publishing.html and that the signatures, hash-sums and software packages are as expected by the process.
> This is how I interpret this part of the release process "Release managers do the mechanical work; but the PMC in general, and the PMC chair in particular (as an officer of the Foundation), are responsible for compliance with ASF requirements."
>
> My understanding is that it is not feasible (neither for Airflow nor Providers) that the PMC members (nor release manager) tests the software and all features/bugfixes. We've never done that and I believe we will never do. We are reaching out to the community to test and we make a best effort to test whatever we release automatically (unit tests, integration tests, testing if providers are installable/importable with Airflow 2.0 and latest source code of Airflow). And we hardly can do more than that.
>
> Happy to discuss it tomorrow, but in the meantime If some of the PMCs could do the review of the process and check the compliance, to be ready to cast your votes - I'd love that.
>
> J.
>
> On Tue, Mar 2, 2021 at 8:44 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>> Hey Ryan,
>>
>> There is no **must** in re-testing it. Providing that you tested it before with real GSuite account is for me enough of a confirmation ;).
>>
>> J.
>>
>> On Sun, Feb 28, 2021 at 10:00 PM Abdur-Rahmaan Janhangeer <ar...@gmail.com> wrote:
>>> Salutes for having a GSuite account just for the functionality 👍👍👍
>>>
>>> On Mon, 1 Mar 2021, 00:05 Ryan Hatter, <ry...@gmail.com> wrote:
>>>> I canceled my GSuite account when my PR for the gdrive to gcs operator was approved & merged. Could anyone maybe help me ensure correct functionality?
>>>>
>>>>
>>>>> On Feb 27, 2021, at 08:48, Jarek Potiuk <ja...@potiuk.com> wrote:
>>>>>
>>>>> I created issue, where we will track the status of tests for the providers (again - it is experiment - but I'd really love to get feedback on the new providers from those who contributed): https://github.com/apache/airflow/issues/14511
>>>>>
>>>>> On Sat, Feb 27, 2021 at 4:28 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>>>>>>
>>>>>> Hey all,
>>>>>>
>>>>>> I have just cut the new wave Airflow Providers packages. This email is calling a vote on the release,
>>>>>> which will last for 72 hours + day for the weekend - which means that it will end on Wed 3 Mar 15:59:34 CET 2021.
>>>>>>
>>>>>> Consider this my (binding) +1.
>>>>>>
>>>>>> KIND REQUEST
>>>>>>
>>>>>> There was a recent discussion about test quality of the providers and I would like to try to address it, still keeping the batch release process every 3 weeks.
>>>>>>
>>>>>> We need a bit of help from the community. I have a kind request to the authors of fixes and new features. I group the providers into those that likely need more testing, and those that do not. I also added names of those who submitted the changes and are most likely to be able to verify if the RC packages are solving the problems/adding features.
>>>>>>
>>>>>> This is a bit of experiment (apologies for calling out) - but if we find that it works, we can automate that. I will create a separate Issue in Github where you will be able to "tick" the boxes for those providers which they are added to. It would not be a blocker if not tested, but it will be a great help if you could test the new RC provider and see if it works as expected according to your changes.
>>>>>>
>>>>>> Providers with new features and fixes - likely needs some testing.:
>>>>>>
>>>>>> * amazon : Cristòfol Torrens, Ruben Laguna, Arati Nagmal, Ivica Kolenkaš, JavierLopezT
>>>>>> * apache.druid: Xinbin Huang
>>>>>> * apache.spark: Igor Khrol
>>>>>> * cncf.kubernetes: jpyen, Ash Berlin-Taylor, Daniel Imberman
>>>>>> * google: Vivek Bhojawala, Xinbin Huang, Pak Andrey, uma66, Ryan Yuan, morrme, Sam Wheating, YingyingPeng22, Ryan Hatter,Tobiasz Kędzierski
>>>>>> * jenkins: Maxim Lisovsky
>>>>>> * microsift.azure: flvndh, yyu
>>>>>> * mysql: Constantino Schillebeeckx
>>>>>> * qubole: Xinbin Huang
>>>>>> * salesforce: Jyoti Dhiman
>>>>>> * slack: Igor Khrol
>>>>>> * tableau: Jyoti Dhiman
>>>>>> * telegram: Shekhar Sing, Adil Khashtamov
>>>>>>
>>>>>> Providers with doc only changes (no need to test):
>>>>>>
>>>>>> * apache-beam
>>>>>> * apache-hive
>>>>>> * dingding
>>>>>> * docker
>>>>>> * elasticsearch
>>>>>> * exasol
>>>>>> * http
>>>>>> * neo4j
>>>>>> * openfaas
>>>>>> * papermill
>>>>>> * presto
>>>>>> * sendgrid
>>>>>> * sftp
>>>>>> * snowflake
>>>>>> * sqlite
>>>>>> * ssh
>>>>>> *
>>>>>>
>>>>>>
>>>>>> Airflow Providers are available at:
>>>>>> https://dist.apache.org/repos/dist/dev/airflow/providers/
>>>>>>
>>>>>> *apache-airflow-providers-<PROVIDER>-*-bin.tar.gz* are the binary
>>>>>> Python "sdist" release - they are also official "sources" for the provider packages.
>>>>>>
>>>>>> *apache_airflow_providers_<PROVIDER>-*.whl are the binary
>>>>>> Python "wheel" release.
>>>>>>
>>>>>> The test procedure for PMC members who would like to test the RC candidates are described in
>>>>>> https://github.com/apache/airflow/blob/master/dev/README_RELEASE_PROVIDER_PACKAGES.md#verify-the-release-by-pmc-members
>>>>>>
>>>>>> and for Contributors:
>>>>>>
>>>>>> https://github.com/apache/airflow/blob/master/dev/README_RELEASE_PROVIDER_PACKAGES.md#verify-by-contributors
>>>>>>
>>>>>>
>>>>>> Public keys are available at:
>>>>>> https://dist.apache.org/repos/dist/release/airflow/KEYS
>>>>>>
>>>>>> Please vote accordingly:
>>>>>>
>>>>>> [ ] +1 approve
>>>>>> [ ] +0 no opinion
>>>>>> [ ] -1 disapprove with the reason
>>>>>>
>>>>>>
>>>>>> Only votes from PMC members are binding, but members of the community are
>>>>>> encouraged to test the release and vote with "(non-binding)".
>>>>>>
>>>>>> Please note that the version number excludes the 'rcX' string.
>>>>>> This will allow us to rename the artifact without modifying
>>>>>> the artifact checksums when we actually release.
>>>>>>
>>>>>>
>>>>>> Each of the packages contains a link to the detailed changelog. The changelogs are moved to the official airflow documentation:
>>>>>> https://github.com/apache/airflow-site/<TODO COPY LINK TO BRANCH>
>>>>>>
>>>>>> <PASTE ANY HIGH-LEVEL DESCRIPTION OF THE CHANGES HERE!>
>>>>>>
>>>>>>
>>>>>> Note the links to documentation from PyPI packages are not working until we merge
>>>>>> the changes to airflow site after releasing the packages officially.
>>>>>>
>>>>>> https://pypi.org/project/apache-airflow-providers-amazon/1.2.0rc1/
>>>>>> https://pypi.org/project/apache-airflow-providers-apache-beam/1.0.1rc1/
>>>>>> https://pypi.org/project/apache-airflow-providers-apache-druid/1.1.0rc1/
>>>>>> https://pypi.org/project/apache-airflow-providers-apache-hive/1.0.2rc1/
>>>>>> https://pypi.org/project/apache-airflow-providers-apache-spark/1.0.2rc1/
>>>>>> https://pypi.org/project/apache-airflow-providers-cncf-kubernetes/1.0.2rc1/
>>>>>> https://pypi.org/project/apache-airflow-providers-dingding/1.0.2rc1/
>>>>>> https://pypi.org/project/apache-airflow-providers-docker/1.0.2rc1/
>>>>>> https://pypi.org/project/apache-airflow-providers-elasticsearch/1.0.2rc1/
>>>>>> https://pypi.org/project/apache-airflow-providers-exasol/1.1.1rc1/
>>>>>> https://pypi.org/project/apache-airflow-providers-google/2.1.0rc1/
>>>>>> https://pypi.org/project/apache-airflow-providers-http/1.1.1rc1/
>>>>>> https://pypi.org/project/apache-airflow-providers-jenkins/1.1.0rc1/
>>>>>> https://pypi.org/project/apache-airflow-providers-microsoft-azure/1.2.0rc1/
>>>>>> https://pypi.org/project/apache-airflow-providers-mysql/1.0.2rc1/
>>>>>> https://pypi.org/project/apache-airflow-providers-neo4j/1.0.1rc1/
>>>>>> https://pypi.org/project/apache-airflow-providers-openfaas/1.1.1rc1/
>>>>>> https://pypi.org/project/apache-airflow-providers-papermill/1.0.2rc1/
>>>>>> https://pypi.org/project/apache-airflow-providers-presto/1.0.2rc1/
>>>>>> https://pypi.org/project/apache-airflow-providers-qubole/1.0.2rc1/
>>>>>> https://pypi.org/project/apache-airflow-providers-salesforce/2.0.0rc1/
>>>>>> https://pypi.org/project/apache-airflow-providers-sendgrid/1.0.2rc1/
>>>>>> https://pypi.org/project/apache-airflow-providers-sftp/1.1.1rc1/
>>>>>> https://pypi.org/project/apache-airflow-providers-slack/3.0.0rc1/
>>>>>> https://pypi.org/project/apache-airflow-providers-snowflake/1.1.1rc1/
>>>>>> https://pypi.org/project/apache-airflow-providers-sqlite/1.0.2rc1/
>>>>>> https://pypi.org/project/apache-airflow-providers-ssh/1.2.0rc1/
>>>>>> https://pypi.org/project/apache-airflow-providers-tableau/1.0.0rc1/
>>>>>> https://pypi.org/project/apache-airflow-providers-telegram/1.0.2rc1/
>>>>>>
>>>>>> Cheers,
>>>>>> J.
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> +48 660 796 129
>>>>>
>>>>>
>>>>> --
>>>>> +48 660 796 129
>>
>>
>> --
>> +48 660 796 129
>
>
> --
> +48 660 796 129
Re: [VOTE] Airflow Providers - release candidates from 2021-02-27
Posted by Jarek Potiuk <ja...@potiuk.com>.
Hello everyone - just a reminder that we have voting (hopefully) finishing
tomorrow.
I'd love to get some votes for that.
Just to clarify what the PMC votes mean, because I believe there were some
question raised about the release process which we are going to discuss it
tomorrow at the dev call but let me just express my interpretation of
https://infra.apache.org/release-publishing.html
PMC member vote (as I understand it) does not mean that this PMC member
tested the release functionality (neither Release Manager).
This merely means that the PMC member agrees that the software was released
according to the requirements and process described in
https://infra.apache.org/release-publishing.html and that the signatures,
hash-sums and software packages are as expected by the process.
This is how I interpret this part of the release process "Release managers
do the mechanical work; but the PMC in general, and the PMC chair in
particular (as an officer of the Foundation), are responsible for
compliance with ASF requirements."
My understanding is that it is not feasible (neither for Airflow nor
Providers) that the PMC members (nor release manager) tests the software
and all features/bugfixes. We've never done that and I believe we will
never do. We are reaching out to the community to test and we make a best
effort to test whatever we release automatically (unit tests, integration
tests, testing if providers are installable/importable with Airflow 2.0 and
latest source code of Airflow). And we hardly can do more than that.
Happy to discuss it tomorrow, but in the meantime If some of the PMCs could
do the review of the process and check the compliance, to be ready to cast
your votes - I'd love that.
J.
On Tue, Mar 2, 2021 at 8:44 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> Hey Ryan,
>
> There is no **must** in re-testing it. Providing that you tested it before
> with real GSuite account is for me enough of a confirmation ;).
>
> J.
>
> On Sun, Feb 28, 2021 at 10:00 PM Abdur-Rahmaan Janhangeer <
> arj.python@gmail.com> wrote:
>
>> Salutes for having a GSuite account just for the functionality 👍👍👍
>>
>> On Mon, 1 Mar 2021, 00:05 Ryan Hatter, <ry...@gmail.com> wrote:
>>
>>> I canceled my GSuite account when my PR for the gdrive to gcs operator
>>> was approved & merged. Could anyone maybe help me ensure correct
>>> functionality?
>>>
>>>
>>> On Feb 27, 2021, at 08:48, Jarek Potiuk <ja...@potiuk.com> wrote:
>>>
>>>
>>> I created issue, where we will track the status of tests for the
>>> providers (again - it is experiment - but I'd really love to get feedback
>>> on the new providers from those who contributed):
>>> https://github.com/apache/airflow/issues/14511
>>>
>>> On Sat, Feb 27, 2021 at 4:28 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>>>
>>>>
>>>> Hey all,
>>>>
>>>> I have just cut the new wave Airflow Providers packages. This email is
>>>> calling a vote on the release,
>>>> which will last for 72 hours + day for the weekend - which means that
>>>> it will end on Wed 3 Mar 15:59:34 CET 2021.
>>>>
>>>> Consider this my (binding) +1.
>>>>
>>>> *KIND REQUEST*
>>>>
>>>> There was a recent discussion about test quality of the providers and I
>>>> would like to try to address it, still keeping the batch release process
>>>> every 3 weeks.
>>>>
>>>> We need a bit of help from the community. I have a kind request to the
>>>> authors of fixes and new features. I group the providers into those that
>>>> likely need more testing, and those that do not. I also added names of
>>>> those who submitted the changes and are most likely to be able to verify if
>>>> the RC packages are solving the problems/adding features.
>>>>
>>>> This is a bit of experiment (apologies for calling out) - but if we
>>>> find that it works, we can automate that. I will create a separate Issue in
>>>> Github where you will be able to "tick" the boxes for those providers which
>>>> they are added to. It would not be a blocker if not tested, but it will be
>>>> a great help if you could test the new RC provider and see if it works as
>>>> expected according to your changes.
>>>>
>>>> Providers with new features and fixes - likely needs some testing.:
>>>>
>>>> * *amazon* : Cristòfol Torrens, Ruben Laguna, Arati Nagmal, Ivica
>>>> Kolenkaš, JavierLopezT
>>>> * *apache.druid*: Xinbin Huang
>>>> * *apache.spark*: Igor Khrol
>>>> * *cncf.kubernetes*: jpyen, Ash Berlin-Taylor, Daniel Imberman
>>>> * *google*: Vivek Bhojawala, Xinbin Huang, Pak Andrey, uma66, Ryan
>>>> Yuan, morrme, Sam Wheating, YingyingPeng22, Ryan Hatter,Tobiasz Kędzierski
>>>> * *jenkins*: Maxim Lisovsky
>>>> * *microsift.azure*: flvndh, yyu
>>>> * *mysql*: Constantino Schillebeeckx
>>>> * *qubole*: Xinbin Huang
>>>> * *salesforce*: Jyoti Dhiman
>>>> * *slack*: Igor Khrol
>>>> * *tableau*: Jyoti Dhiman
>>>> * *telegram*: Shekhar Sing, Adil Khashtamov
>>>>
>>>> Providers with doc only changes (no need to test):
>>>>
>>>> * apache-beam
>>>> * apache-hive
>>>> * dingding
>>>> * docker
>>>> * elasticsearch
>>>> * exasol
>>>> * http
>>>> * neo4j
>>>> * openfaas
>>>> * papermill
>>>> * presto
>>>> * sendgrid
>>>> * sftp
>>>> * snowflake
>>>> * sqlite
>>>> * ssh
>>>> *
>>>>
>>>>
>>>> Airflow Providers are available at:
>>>> https://dist.apache.org/repos/dist/dev/airflow/providers/
>>>>
>>>> *apache-airflow-providers-<PROVIDER>-*-bin.tar.gz* are the binary
>>>> Python "sdist" release - they are also official "sources" for the
>>>> provider packages.
>>>>
>>>> *apache_airflow_providers_<PROVIDER>-*.whl are the binary
>>>> Python "wheel" release.
>>>>
>>>> The test procedure for PMC members who would like to test the RC
>>>> candidates are described in
>>>>
>>>> https://github.com/apache/airflow/blob/master/dev/README_RELEASE_PROVIDER_PACKAGES.md#verify-the-release-by-pmc-members
>>>>
>>>> and for Contributors:
>>>>
>>>>
>>>> https://github.com/apache/airflow/blob/master/dev/README_RELEASE_PROVIDER_PACKAGES.md#verify-by-contributors
>>>>
>>>>
>>>> Public keys are available at:
>>>> https://dist.apache.org/repos/dist/release/airflow/KEYS
>>>>
>>>> Please vote accordingly:
>>>>
>>>> [ ] +1 approve
>>>> [ ] +0 no opinion
>>>> [ ] -1 disapprove with the reason
>>>>
>>>>
>>>> Only votes from PMC members are binding, but members of the community
>>>> are
>>>> encouraged to test the release and vote with "(non-binding)".
>>>>
>>>> Please note that the version number excludes the 'rcX' string.
>>>> This will allow us to rename the artifact without modifying
>>>> the artifact checksums when we actually release.
>>>>
>>>>
>>>> Each of the packages contains a link to the detailed changelog. The
>>>> changelogs are moved to the official airflow documentation:
>>>> https://github.com/apache/airflow-site/<TODO COPY LINK TO BRANCH>
>>>>
>>>> <PASTE ANY HIGH-LEVEL DESCRIPTION OF THE CHANGES HERE!>
>>>>
>>>>
>>>> Note the links to documentation from PyPI packages are not working
>>>> until we merge
>>>> the changes to airflow site after releasing the packages officially.
>>>>
>>>> https://pypi.org/project/apache-airflow-providers-amazon/1.2.0rc1/
>>>> https://pypi.org/project/apache-airflow-providers-apache-beam/1.0.1rc1/
>>>> https://pypi.org/project/apache-airflow-providers-apache-druid/1.1.0rc1/
>>>> https://pypi.org/project/apache-airflow-providers-apache-hive/1.0.2rc1/
>>>> https://pypi.org/project/apache-airflow-providers-apache-spark/1.0.2rc1/
>>>>
>>>> https://pypi.org/project/apache-airflow-providers-cncf-kubernetes/1.0.2rc1/
>>>> https://pypi.org/project/apache-airflow-providers-dingding/1.0.2rc1/
>>>> https://pypi.org/project/apache-airflow-providers-docker/1.0.2rc1/
>>>>
>>>> https://pypi.org/project/apache-airflow-providers-elasticsearch/1.0.2rc1/
>>>> https://pypi.org/project/apache-airflow-providers-exasol/1.1.1rc1/
>>>> https://pypi.org/project/apache-airflow-providers-google/2.1.0rc1/
>>>> https://pypi.org/project/apache-airflow-providers-http/1.1.1rc1/
>>>> https://pypi.org/project/apache-airflow-providers-jenkins/1.1.0rc1/
>>>>
>>>> https://pypi.org/project/apache-airflow-providers-microsoft-azure/1.2.0rc1/
>>>> https://pypi.org/project/apache-airflow-providers-mysql/1.0.2rc1/
>>>> https://pypi.org/project/apache-airflow-providers-neo4j/1.0.1rc1/
>>>> https://pypi.org/project/apache-airflow-providers-openfaas/1.1.1rc1/
>>>> https://pypi.org/project/apache-airflow-providers-papermill/1.0.2rc1/
>>>> https://pypi.org/project/apache-airflow-providers-presto/1.0.2rc1/
>>>> https://pypi.org/project/apache-airflow-providers-qubole/1.0.2rc1/
>>>> https://pypi.org/project/apache-airflow-providers-salesforce/2.0.0rc1/
>>>> https://pypi.org/project/apache-airflow-providers-sendgrid/1.0.2rc1/
>>>> https://pypi.org/project/apache-airflow-providers-sftp/1.1.1rc1/
>>>> https://pypi.org/project/apache-airflow-providers-slack/3.0.0rc1/
>>>> https://pypi.org/project/apache-airflow-providers-snowflake/1.1.1rc1/
>>>> https://pypi.org/project/apache-airflow-providers-sqlite/1.0.2rc1/
>>>> https://pypi.org/project/apache-airflow-providers-ssh/1.2.0rc1/
>>>> https://pypi.org/project/apache-airflow-providers-tableau/1.0.0rc1/
>>>> https://pypi.org/project/apache-airflow-providers-telegram/1.0.2rc1/
>>>>
>>>> Cheers,
>>>> J.
>>>>
>>>>
>>>>
>>>> --
>>>> +48 660 796 129
>>>>
>>>
>>>
>>> --
>>> +48 660 796 129
>>>
>>>
>
> --
> +48 660 796 129
>
--
+48 660 796 129