Re: [PROPOSAL] Approach for releasing the backported "providers" packages

[Jarek Potiuk <Ja...@polidea.com>]

Going back to the topic i rethought it and had some discussions with Tomek
and I think the best and least invasive way to implement the dependencies
between provider packages will be to use extras - similar to airflow. We
can generate the extras automatically for each providers package:

Then if for example you want to install providers-google package and you
would like to use Aws/GCp transfer operator you would just have to 'pip
install airflow-providers-google[aws]'

That would reflect very well what we have now in Airflow and it would be
much less overhead for whoever creates package. We would not have to add
try/except to imports. And It lays good foundation for AIP-8 (as
dependencies between provider packages will be built-in).

Anyone wants to comment/has some opinions? Does it look like we can vote on
this proposal to split providers and release them separately when the tests
are working?

J.


sob., 22 lut 2020, 18:22 użytkownik Jarek Potiuk <Ja...@polidea.com>
napisał:

> TL;DR; I have an update to the approach for "backported" packages - I have
> working solution that should enable to release each "providers" backport
> package separately from each other and it will help us to release only
> those packages that are tested rather than everything in one go.
>
> This PR is an enabler to release each provider package totally
> independently - even if they import each other:
> https://github.com/apache/airflow/pull/7506 . I think once I complete it,
> we should be fully ready to "release small" - i.e release only what is
> needed and tested  (@Ash - I think that was what you also thought about). I
> just wanted to share it  with you before I solve all the remaining 63
> imports to go to see what you think about it.
>
> I've implemented a mechanism to find out what are the cross-dependencies
> between the different "providers" packages  from the back-porting approach.
> I not only check what are the dependencies but I also check if they are
> properly handled with try/except (so that the modules can still be imported
> even if the other provider's packages are not available). And this
> behaviour is checked and enforced via pre-commit.
>
> This is to handle the famous "transfer" operators dependencies - and I
> think this is a great step towards AIP-8 (and when we make the final
> decision about releasing the backporting packages - it is the only way to
> make the backporting packages usable and installable on their own. I will
> also add automated tests in Travis to see if we can install every package
> separately on airflow 1.10 automatically. We have all the building blocks
> for it.
>
> We even have automatically generated and updated json file with
> dependencies between the provider packages. And we could even make those
> "real" dependencies between packages - i.e have those provider packages
> depend on each other - though I think it's not a good idea as those
> dependencies are "soft" rather than "hard".
>
> I think with this change I am fully confident we can propose a reasonable
> approach for separately (when ready) releasable providers packages with
> CalVer approach for the packages (for 1.10) and ready to start discussing
> what versioning/approach we would use for 2.0.
>
> WDYT? Shall we start a vote? Anyone still has some comments on that?
>
> J.
>
>
>
> On Thu, Feb 20, 2020 at 4:09 PM Jarek Potiuk <Ja...@polidea.com>
> wrote:
>
>> Any more comments :) ?
>>
>>
>>
>> On Wed, Feb 19, 2020 at 2:13 PM Jarek Potiuk <Ja...@polidea.com>
>> wrote:
>>
>>> And just to clarify - I do not think we should make "massive" releases
>>> of all providers. Ash - you are completely right we should only release
>>> what's changed AND when it is tested. And it can all be automated so that
>>> the overhead will be rather small.
>>>
>>> And in case it is your concern - together with my team we are
>>> really happy to coordinate the whole process once we agree to it and make
>>> sure it is manageable and support any provider release as long as
>>> someone can contribute system tests for that provider.
>>>
>>> Just to be clear and transparent - Google Cloud is (obviously)
>>> supporting a lot of this work. We are in close contact with the Composer
>>> team and they want to make sure they can release internally the "2.0"
>>> versions of GCP operators in Composer. In fact they already can do it and
>>> they know how to prepare their own packages as discussed previously and
>>> there is nothing to prevent them doing so, but my personal goal is to make
>>> it available to all providers and to all users (regardless if they are
>>> using Composer or not and which cloud services they use).
>>>
>>> Releasing backport packages to pypi officially and managing release
>>> process for everyone seems like a good idea for the community I think :).
>>>
>>> J.
>>>
>>>
>>> On Wed, Feb 19, 2020 at 2:04 PM Jarek Potiuk <Ja...@polidea.com>
>>> wrote:
>>>
>>>>
>>>> To address point 1: I would favour individual, non-cal ver releases of
>>>>> the providers, and that way we don't have to release the providers that
>>>>> don't change.
>>>>>
>>>>
>>>> Agree with releasing only what's needed. I still think CalVer is good
>>>> as we can release (for example) GCP packages several times if we add new
>>>> operators (we plan to add at least a few in the coming months). Do you have
>>>> an alternative proposal to that Ash if we want to release
>>>> providers-google-cloud several times? What versioning scheme you propose
>>>> for that ?
>>>>
>>>>
>>>>> We _could_ split out the providers in to separate repos if we wanted,
>>>>> much like https://github.com/terraform-providers repos (though we
>>>>> wouldn't do it as a separate github org). Again, all optional. But
>>>>> splitting things in to separate repos should make tests for providers much
>>>>> much quicker, as they'd only have to run a tiny number of tests in
>>>>> comparison to now.
>>>>> That said it's not without it's organisational challenges.
>>>>
>>>>
>>>> I agree - AIP-8 and future releases of Airflow 2.0 are a whole
>>>> different subject. That will include not only splitting but also how we
>>>> approach installation for AIrflow 2.0, whether we use semantic versioning
>>>> and whether we do bugfix releases. This is far more complex discussion and
>>>> it will take a lot of time to decide. And I do not think we need to decide
>>>> now nor we are blocked by lack of decisions there. I see backporting
>>>> packages as related and necessary prerequisite to AIP-8 but I treat it as a
>>>> tactical 1.10 backporting solution now (it does provide some prerequisites
>>>> for AIP-8 but it does not block any approach for the 2.0).
>>>>
>>>> And this is why I would like us to decide how we want to handle AIP-8 :)
>>>>>
>>>>
>>>> My goal is  to improve the lives of current users ASAP and I think the
>>>> plan to do it is quite viable. The future plan needs to be decided but I
>>>> think waiting for it is totally not needed. I don't think we need to decide
>>>> now - we can proceed with backporting first and start another discussion on
>>>> 2.0 versioning and packaging later. I would really love to turn it into
>>>> AIP-8 eventually. I just think we can give big value to current 1.10 users
>>>> and we are not at all blocked or locked by this decision with AIP-8
>>>> discussion. Those two are linked but not depend on each other (similarly as
>>>> AIP-21 - it's linked via system tests but automation is not fully needed to
>>>> be hashed out before we release backporting). I really want to treat
>>>> "airflow-1.10" and "airflow-2.0" separately.
>>>>
>>>> I really don't think there is any real need to wait with that nor it
>>>> puts any big issue on us. People will upgrade to 2.0 regardless if they
>>>> will have providers backport packages installed or not and no matter what
>>>> we decide for 2.0 it won't be impacted by how we treat backport
>>>> installations. I proposed (from the very beginning) that the packages we
>>>> produce are strictly limited to be "1.10-backport" only (and that's why I
>>>> put it in the name of the packages).
>>>>
>>>>
>>>>> -ash
>>>>> On Feb 19 2020, at 12:19 pm, Driesprong, Fokko <fo...@driesprong.frl>
>>>>> wrote:
>>>>> > I'm sorry, I've been a bit busy lately. Keeping track of these
>>>>> discussions
>>>>> > in the evening doesn't always for out, as it seems.
>>>>> >
>>>>> > 1. So instead of releasing one package, as stated in the wiki, we're
>>>>> going
>>>>> > to release over 50 provider packages
>>>>> > (apache-airflow-providers-*-*-1.10-YYYY.MM.DD)? If I understand
>>>>> > it correctly. That sounds to me like a lot of work to release.
>>>>> > 2. CalVer is fine with me.
>>>>> > 3. Why is this compatibility matrix needed? Providers package still
>>>>> has a
>>>>> > version in them, right? If we keep backward compatibility between
>>>>> major
>>>>> > versions, this should go fine.
>>>>> > 4. Personally I like the idea, but I'm not sure if this is going to
>>>>> happen.
>>>>> >
>>>>> > What's the future plan here? Are we going to remove the
>>>>> operators/sensors
>>>>> > completely from the Airflow codebase, as suggested in
>>>>> >
>>>>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=100827303
>>>>> >
>>>>> > Cheers, Fokko
>>>>> > Op wo 19 feb. 2020 om 13:00 schreef Jarek Potiuk <
>>>>> Jarek.Potiuk@polidea.com>:
>>>>> > > +Fokko Driesprong <fo...@driesprong.frl> -> in case you missed it
>>>>> this is
>>>>> > > the thread where I proposed and we had some discussion about
>>>>> backporting
>>>>> > > release process (you asked for it in the cancelled [VOTE] thread).
>>>>> The
>>>>> > > links to PR are earlier in the discussion but for the sake of
>>>>> restarting
>>>>> > > the discussion I will summarise it here again. Please let me know
>>>>> what you
>>>>> > > think,
>>>>> > >
>>>>> > > There were doubts raised by Ash in
>>>>> > >
>>>>> https://lists.apache.org/thread.html/re9f71fa838671e484ef0e7a1a4011b7c7946838008fce8eb05ad2fa0%40%3Cdev.airflow.apache.org%3E
>>>>> > > so I cancelled the vote and I want to return to the proposal.
>>>>> > >
>>>>> > > I have a modified proposal that is - I think addressing those
>>>>> concerns.
>>>>> > > It also relates very much to parallel discussion we have about
>>>>> AIP-4 -
>>>>> > > automated system tests, which in my view are the best way to
>>>>> "stamp" the
>>>>> > > released backport packages as tested. The AIP-4 discussion is here:
>>>>> > >
>>>>> https://lists.apache.org/thread.html/r8c37e2289e8c8b438acc1415e9f2a1b74cd6feac831e02a17b01d88d%40%3Cdev.airflow.apache.org%3E
>>>>> and
>>>>> > > while it is related - it's not a prerequisite for backport
>>>>> packages.
>>>>> > >
>>>>> > > *Summary of the discussion:*
>>>>> > > Just to summarize the discussion above. We have two PRs now which
>>>>> are
>>>>> > > related:
>>>>> > >
>>>>> > > * [AIRFLOW-6663] Prepare backporting packages -
>>>>> > > https://github.com/apache/airflow/pull/7391
>>>>> > > * [AIRFLOW-6763] Make systems tests ready for backport tests -
>>>>> > > https://github.com/apache/airflow/pull/7389
>>>>> > >
>>>>> > > The AIRFLOW-6663 introduce automated scripts that can prepare
>>>>> backport
>>>>> > > packages, the AIRFLOW-6763 introduces changes that allow to use
>>>>> breeze to
>>>>> > > install older releases of Airflow and run system tests
>>>>> automatically using
>>>>> > > backport packages produced in AIRFLOW-6663.
>>>>> > >
>>>>> > > The AIRFLOW-6663 for now produces .whl packages (so ready to be
>>>>> uploaded
>>>>> > > to pypi). We have both one-big "providers" package as well as
>>>>> smaller
>>>>> > > "per-provider" packages ("google.cloud", "apache.cassandra" etc.).
>>>>> So this
>>>>> > > discussion is mainly about how are we going to release them
>>>>> "officially" -
>>>>> > > for anyone to be able to install it via PyPI.
>>>>> > >
>>>>> > > The AIRFLOW-6763 modifies current system tests execution approach
>>>>> that we
>>>>> > > have in GCP to make it more generic and reusable across all
>>>>> operators.
>>>>> > > The current state is that we have nearly 100% coverage of "system
>>>>> tests"
>>>>> > > in GCP. We were adding system tests in GCP for 1.5 year now and
>>>>> they allow
>>>>> > > to test automatically master/backported packages against the old
>>>>> airflow in
>>>>> > > fully automated way (providing that we have external systems we
>>>>> can talk to
>>>>> > > and can authorise in). AIP-4 is a related discussion - on how we
>>>>> can bring
>>>>> > > those system tests to other services and automate execution of
>>>>> those tests
>>>>> > > in regular way in CI.
>>>>> > >
>>>>> > > It's not necessary to implement AIP-4 fully to test backport
>>>>> packages - we
>>>>> > > do not have to fully automate it. We can trigger the tests
>>>>> manually -
>>>>> > > separately for each provider (as soon as those tests will be
>>>>> implemented by
>>>>> > > particular provider). Once we merge AIRFLOW-6763 other providers -
>>>>> not only
>>>>> > > GCP will be able to use similar approach as GCP system tests (it
>>>>> boils down
>>>>> > > to turning example_dags into actually runnable DAGs that will run
>>>>> > > backported operators in a DAG run on Airflow 1.10.x). With
>>>>> AIRFLOW-6763
>>>>> > > it's super simple and it opens up the possibility for anyone to
>>>>> write and
>>>>> > > test "backport" providers against any 1.10. version of airflow
>>>>> very easily.
>>>>> > >
>>>>> > > *Modified proposal:*
>>>>> > > 1) We should release separate packages per provider. After a bit of
>>>>> > > thought I agree with Ash one big blob is not a good idea for
>>>>> multiple
>>>>> > > reasons (stability, dependencies, etc). I already have a change in
>>>>> my mind
>>>>> > > which will make those packages fully independent to install even
>>>>> if they
>>>>> > > are using cross-provider transfer operators. With that change,
>>>>> transfer
>>>>> > > operators will not work if the "other" package is not installed
>>>>> but the
>>>>> > > "provider" package will be importable at least. +Ash Berlin-Taylor
>>>>> > > <as...@apache.org> - I think it addresses your "blob" concern.
>>>>> Until each
>>>>> > > provider is tested, this provider's package should be "devel"
>>>>> packages that
>>>>> > > are only available via our CI or manual preparation.
>>>>> > >
>>>>> > > 2) I think still CalVer is best idea for versioning of the provider
>>>>> > > package apache-airflow-providers-google-cloud-1.10-YYYY.MM.DD is an
>>>>> > > example package name/version. Those packages have automatically
>>>>> proper
>>>>> > > dependencies set and we can automatically check if they are clearly
>>>>> > > installing on top of any airflow version. This can be very easily
>>>>> automated
>>>>> > > in CI.
>>>>> > >
>>>>> > > 3) I can introduce and lead a compatibility/test matrix (I already
>>>>> planned
>>>>> > > to keep it in the
>>>>> > >
>>>>> https://cwiki.apache.org/confluence/display/AIRFLOW/Backported+providers+packages+for+Airflow+1.10.*+series
>>>>> document)
>>>>> > > where we will mark version of released packages and whether it was
>>>>> tested
>>>>> > > automatically with system tests (and by whom !). I am super-happy
>>>>> to assume
>>>>> > > the role of release manager for the backport packages and make the
>>>>> process
>>>>> > > happen, update the release guidelines in wiki etc. and finally
>>>>> physically
>>>>> > > make the releases when voting completes on each package and
>>>>> testing is done.
>>>>> > >
>>>>> > > 4) Initially this will be only GCP-related packages (as they
>>>>> already have
>>>>> > > full system test coverage as well as we are dedicated to test
>>>>> them) but
>>>>> > > this should be a learning experience. We will provide clear
>>>>> instructions
>>>>> > > and examples for others how to add and run system tests and maybe
>>>>> later
>>>>> > > with AIP-4 we fully automate it. This way we will be able to mark
>>>>> those
>>>>> > > tested packages as "tested" and release them officially via usual
>>>>> PyPi
>>>>> > > channel. Note that the providers operators are already unit-tested
>>>>> in
>>>>> > > master, so running system tests with old airflow IMHO is the only
>>>>> thing we
>>>>> > > need to see if there are no backporting problems. I will also start
>>>>> > > campaign and encourage our community members to add system tests
>>>>> and run
>>>>> > > the tests for the providers they are interested in. The idea will
>>>>> be to add
>>>>> > > the system tests gradually so that with every future release of
>>>>> backport
>>>>> > > packages we can release more and more provider packages because
>>>>> system
>>>>> > > tests will be automated. I think that should address your ("_we_
>>>>> should
>>>>> > > test it" concern Ash).
>>>>> > >
>>>>> > > Let me know what you think - Fokko, Ash (does it address your
>>>>> concerns?)
>>>>> > > and others who are interested.
>>>>> > >
>>>>> > > J.
>>>>> > >
>>>>> > >
>>>>> > >
>>>>> > >
>>>>> > > On Mon, Feb 17, 2020 at 9:54 AM Kamil Breguła <
>>>>> kamil.bregula@polidea.com>
>>>>> > > wrote:
>>>>> > >
>>>>> > > > +1 on voting.
>>>>> > > > I read the discussions very carefully and I agree that it is
>>>>> worth
>>>>> > > > creating a interm solution. It is a very good idea if it will
>>>>> not be a
>>>>> > > > temporary and experimental solution indefinitely. We have a clear
>>>>> > > > deadline, so we can go further with this.
>>>>> > > >
>>>>> > > > On Mon, Feb 17, 2020 at 8:35 AM Jarek Potiuk <
>>>>> Jarek.Potiuk@polidea.com>
>>>>> > > > wrote:
>>>>> > > > >
>>>>> > > > > Others? WDYT? Shall we start voting on it ? Any more comments?
>>>>> > > > > I think I would like to propose an interim solution where all
>>>>> the
>>>>> > > > > backported packages for 1.10 will be released as a single big
>>>>> package
>>>>> > > >
>>>>> > > > with
>>>>> > > > > Calver Versioning and with some compatibility matrix where we
>>>>> will mark
>>>>> > > > > which of the providers were tested (semi-automatically ?)
>>>>> possibly over
>>>>> > > > > time automatically using system tests (following the AIP-4
>>>>> proposal).
>>>>> > > > >
>>>>> > > > > Eventually - maybe even for 2.0 - we will be able to split the
>>>>> packages
>>>>> > > > on
>>>>> > > > > per-provider basis and release them independently - but that is
>>>>> > > >
>>>>> > > > something
>>>>> > > > > that we can test and agree later - when we will be discussing
>>>>> overall
>>>>> > > > > release approach (including possibly semantic or calendar
>>>>> versioning for
>>>>> > > > > 2.* releases).
>>>>> > > > >
>>>>> > > > > Let me know if you have any objections, if not, I will call a
>>>>> vote on
>>>>> > > > that
>>>>> > > > > in a day or so.
>>>>> > > > >
>>>>> > > > > J.
>>>>> > > > >
>>>>> > > > > On Fri, Feb 14, 2020 at 9:46 PM Jarek Potiuk <
>>>>> Jarek.Potiuk@polidea.com>
>>>>> > > > > wrote:
>>>>> > > > >
>>>>> > > > > > How about going both routes ?
>>>>> > > > > > 1) Provide one big "backport" package for 1.10
>>>>> > > > > > 2) Once we release 2.0 split providers to micro-packages
>>>>> > > > > >
>>>>> > > > > > J.
>>>>> > > > > > On Fri, Feb 14, 2020 at 9:30 PM Ash Berlin-Taylor <
>>>>> ash@apache.org>
>>>>> > > > wrote:
>>>>> > > > > >
>>>>> > > > > > > I think before we take this discussion any further we
>>>>> should work out
>>>>> > > > > > > what our plan is for AIP-8
>>>>> > > > > > >
>>>>> > > >
>>>>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=100827303&src=contextnavpagetreemode
>>>>> > > > > > > ( though likely needs updating as it still talks about
>>>>> contrib which
>>>>> > > > > >
>>>>> > > > >
>>>>> > > >
>>>>> > > > isn't
>>>>> > > > > > > relevant anymore)
>>>>> > > > > > >
>>>>> > > > > > > AIP-8 talks about "One hook or operator per package,
>>>>> following the
>>>>> > > > "micro
>>>>> > > > > > > package" philosophy." as it's long term goal, and I think
>>>>> I broadly
>>>>> > > > > >
>>>>> > > > >
>>>>> > > >
>>>>> > > > agree
>>>>> > > > > > > with that.
>>>>> > > > > > > Given we have almost all the things in place to have this,
>>>>> I would
>>>>> > > > > >
>>>>> > > > >
>>>>> > > >
>>>>> > > > rather
>>>>> > > > > > > we didn't release a single large "backport" package, only
>>>>> to have to
>>>>> > > > > >
>>>>> > > > >
>>>>> > > >
>>>>> > > > have
>>>>> > > > > > > users to then switch over to using new packages.
>>>>> > > > > > > > We can follow the same process/keys etc as for releasing
>>>>> the main
>>>>> > > > > > >
>>>>> > > > > > > airflow
>>>>> > > > > > > > package, but I think it can be a bit more relaxed in
>>>>> terms of
>>>>> > > > > > >
>>>>> > > > > >
>>>>> > > > >
>>>>> > > >
>>>>> > > > testing -
>>>>> > > > > > > and
>>>>> > > > > > > > we can release it more often (as long as there will be
>>>>> new changes
>>>>> > > > > > >
>>>>> > > > > >
>>>>> > > > >
>>>>> > > >
>>>>> > > > in
>>>>> > > > > > > > providers). Those packages might be released on "as-is"
>>>>> basis -
>>>>> > > > > > >
>>>>> > > > > >
>>>>> > > > >
>>>>> > > >
>>>>> > > > without
>>>>> > > > > > > > guarantee that they work for all operators/hooks/sensors
>>>>> - and
>>>>> > > > > > >
>>>>> > > > > >
>>>>> > > > >
>>>>> > > >
>>>>> > > > without
>>>>> > > > > > > > guarantee that they will work for all 1.10.* versions.
>>>>> > > > > > >
>>>>> > > > > > >
>>>>> > > > > > > I'm in favour of this as a general idea.
>>>>> > > > > > > My preferred way is to have each "provider" be it's own
>>>>> package.
>>>>> > > > > >
>>>>> > > > >
>>>>> > > >
>>>>> > > > This is
>>>>> > > > > > > a slightly fuzzy concept, as for instance
>>>>> airflow.providers.goolgle
>>>>> > > > > > > probably makes sense as a single package rather than a
>>>>> .google.cloud
>>>>> > > > > >
>>>>> > > > >
>>>>> > > >
>>>>> > > > and
>>>>> > > > > > > .google.marketing etc packages (as per Kamil's comment on
>>>>> Github),
>>>>> > > > > >
>>>>> > > > >
>>>>> > > >
>>>>> > > > but
>>>>> > > > > > > apache.airflow.providers.apache should _not_ be one
>>>>> package. So
>>>>> > > > > >
>>>>> > > > >
>>>>> > > >
>>>>> > > > there's no
>>>>> > > > > > > easily expressible rule here, but (to me) there is an
>>>>> obvious way
>>>>> > > > > >
>>>>> > > > >
>>>>> > > >
>>>>> > > > for each
>>>>> > > > > > > case.
>>>>> > > > > > > Anyway, to provide smalle releases of providers as per
>>>>> terraform, or
>>>>> > > > > >
>>>>> > > > >
>>>>> > > >
>>>>> > > > to
>>>>> > > > > > > backport to make 2.0 adoption easier?
>>>>> > > > > > > -a
>>>>> > > > > > > On Feb 11 2020, at 3:43 pm, Jarek Potiuk <
>>>>> Jarek.Potiuk@polidea.com>
>>>>> > > > > > > wrote:
>>>>> > > > > > > > Any more opinions?
>>>>> > > > > > > >
>>>>> > > > > > > > I gave some thoughts to that and I think we should :
>>>>> > > > > > > > 1) release one big providers* package with Calver
>>>>> versioning -
>>>>> > > > > > > > apache-airflow-providers-backport-2020.02.11 if were to
>>>>> release it
>>>>> > > > > > >
>>>>> > > > > >
>>>>> > > > >
>>>>> > > >
>>>>> > > > today
>>>>> > > > > > > > (we can always break them into smaller packages when we
>>>>> decide in
>>>>> > > > > > >
>>>>> > > > > >
>>>>> > > > >
>>>>> > > >
>>>>> > > > 2.0).
>>>>> > > > > > > And
>>>>> > > > > > > > then we could change the package names.
>>>>> > > > > > > > 2) scheduled or regular releases. We should release them
>>>>> as needed
>>>>> > > > > > >
>>>>> > > > > >
>>>>> > > > >
>>>>> > > >
>>>>> > > > -
>>>>> > > > > > > i.e.
>>>>> > > > > > > > if we have large change at one or few of the providers
>>>>> or serious
>>>>> > > > > > >
>>>>> > > > > > > bugfix,
>>>>> > > > > > > > we can release it again.
>>>>> > > > > > > > 3) it should be manual effort involving voting and PMC
>>>>> approvals.
>>>>> > > > > > > >
>>>>> > > > > > > > What do you think?
>>>>> > > > > > > > J.
>>>>> > > > > > > >
>>>>> > > > > > > > On Mon, Feb 10, 2020 at 2:43 PM Tomasz Urbaszek <
>>>>> > > > > > > tomasz.urbaszek@polidea.com>
>>>>> > > > > > > > wrote:
>>>>> > > > > > > >
>>>>> > > > > > > > > I am ok with users building their own packages.
>>>>> > > > > > > > > T.
>>>>> > > > > > > > > On Mon, Feb 10, 2020 at 1:47 PM Jarek Potiuk <
>>>>> > > > > > > >
>>>>> > > > > > >
>>>>> > > > > > > Jarek.Potiuk@polidea.com>
>>>>> > > > > > > > > wrote:
>>>>> > > > > > > > >
>>>>> > > > > > > > > > I think it should be a deliberate effort for
>>>>> releasing - with
>>>>> > > > > > > voting. We
>>>>> > > > > > > > > > are releasing the source code and IMHO it should
>>>>> follow the
>>>>> > > > > > > > >
>>>>> > > > > > > >
>>>>> > > > > > >
>>>>> > > > > >
>>>>> > > > >
>>>>> > > >
>>>>> > > > same
>>>>> > > > > > > rules as
>>>>> > > > > > > > > > releasing airflow itself.
>>>>> > > > > > > > > > With this change - anyone will be able to build and
>>>>> prepare
>>>>> > > > > > > > >
>>>>> > > > > > > >
>>>>> > > > > > >
>>>>> > > > > >
>>>>> > > > >
>>>>> > > >
>>>>> > > > their
>>>>> > > > > > > own
>>>>> > > > > > > > >
>>>>> > > > > > > > > .whl
>>>>> > > > > > > > > > packages and install them locally, so I do not think
>>>>> there is a
>>>>> > > > > > > > >
>>>>> > > > > > > >
>>>>> > > > > > >
>>>>> > > > > > > need to
>>>>> > > > > > > > > > automatically release those packages?
>>>>> > > > > > > > > >
>>>>> > > > > > > > > > However releasing them in PyPi should be quite an
>>>>> important
>>>>> > > > event
>>>>> > > > > > > as pypi
>>>>> > > > > > > > > > releases are supposed to be used by users not
>>>>> developers.
>>>>> > > > > > > > > >
>>>>> > > > > > > > > > J.
>>>>> > > > > > > > > > On Mon, Feb 10, 2020 at 11:16 AM Tomasz Urbaszek <
>>>>> > > > > > > > > > tomasz.urbaszek@polidea.com> wrote:
>>>>> > > > > > > > > >
>>>>> > > > > > > > > > > I think as long as we follow:
>>>>> > > > > > > > > > > > The only people who are supposed to know about
>>>>> such
>>>>> > > > > > > > > > >
>>>>> > > > > > > > > >
>>>>> > > > > > > > >
>>>>> > > > > > > >
>>>>> > > > > > >
>>>>> > > > > >
>>>>> > > > >
>>>>> > > >
>>>>> > > > developer
>>>>> > > > > > > > > > >
>>>>> > > > > > > > > >
>>>>> > > > > > > > >
>>>>> > > > > > > > > resources
>>>>> > > > > > > > > > > are individuals actively participating in
>>>>> development or
>>>>> > > > > > > > > >
>>>>> > > > > > > > >
>>>>> > > > > > > >
>>>>> > > > > > >
>>>>> > > > > > > following the
>>>>> > > > > > > > > >
>>>>> > > > > > > > > > dev
>>>>> > > > > > > > > > > list and thus aware of the conditions placed on
>>>>> unreleased
>>>>> > > > > > > > > >
>>>>> > > > > > > > >
>>>>> > > > > > > >
>>>>> > > > > > >
>>>>> > > > > > > materials.
>>>>> > > > > > > > > > >
>>>>> > > > > > > > > > > we should be ok. My impression is that people are
>>>>> usually
>>>>> > > > aware of
>>>>> > > > > > > > > > > what "nightly build" means and what are the risks.
>>>>> But it's
>>>>> > > > > > > > > >
>>>>> > > > > > > > >
>>>>> > > > > > > >
>>>>> > > > > > >
>>>>> > > > > >
>>>>> > > > >
>>>>> > > >
>>>>> > > > just a
>>>>> > > > > > > > > > > suggestion that I made thinking about all those
>>>>> people who
>>>>> > > > > > > > > >
>>>>> > > > > > > > >
>>>>> > > > > > > >
>>>>> > > > > > >
>>>>> > > > > > > contribute
>>>>> > > > > > > > > > > integration and can't use it "officialy" for let
>>>>> say the
>>>>> > > > > > > > > >
>>>>> > > > > > > > >
>>>>> > > > > > > >
>>>>> > > > > > >
>>>>> > > > > > > following 2
>>>>> > > > > > > > > > > months. I was also thinking about this result
>>>>> > > > > > > > > > >
>>>>> > > > > > > > > > >
>>>>> > > > > > > > > >
>>>>> > > > > > > > >
>>>>> > > > > > >
>>>>> > > >
>>>>> https://www.digitalocean.com/currents/december-2019/#generational-expectations-for-open-source-maintenance
>>>>> > > > > > > > > > > :)
>>>>> > > > > > > > > > >
>>>>> > > > > > > > > > > T.
>>>>> > > > > > > > > > > On Mon, Feb 10, 2020 at 10:52 AM Ash Berlin-Taylor
>>>>> <
>>>>> > > > > > > > > >
>>>>> > > > > > > > >
>>>>> > > > > > > >
>>>>> > > > > > >
>>>>> > > > > > > ash@apache.org>
>>>>> > > > > > > > > > wrote:
>>>>> > > > > > > > > > > >
>>>>> > > > > > > > > > > > That might be a grey area according to my
>>>>> reading of the
>>>>> > > > Apache
>>>>> > > > > > > > > release
>>>>> > > > > > > > > > > policies:
>>>>> > > > > > > > > > > >
>>>>> > > > > > > > > > > >
>>>>> https://apache.org/legal/release-policy.html#publication
>>>>> > > > > > > > > > > > > During the process of developing software and
>>>>> preparing a
>>>>> > > > > > > > > > > >
>>>>> > > > > > > > > > >
>>>>> > > > > > > > > >
>>>>> > > > > > > > >
>>>>> > > > > > > >
>>>>> > > > > > >
>>>>> > > > > > > release,
>>>>> > > > > > > > > > > >
>>>>> > > > > > > > > > >
>>>>> > > > > > > > > > > various packages are made available to the
>>>>> development
>>>>> > > > community
>>>>> > > > > > > for
>>>>> > > > > > > > > > > testing purposes. Projects MUST direct outsiders
>>>>> towards
>>>>> > > > > > > > > >
>>>>> > > > > > > > >
>>>>> > > > > > > >
>>>>> > > > > > >
>>>>> > > > > >
>>>>> > > > >
>>>>> > > >
>>>>> > > > official
>>>>> > > > > > > > > >
>>>>> > > > > > > > > > releases
>>>>> > > > > > > > > > > rather than raw source repositories, nightly
>>>>> builds,
>>>>> > > > > > > > > >
>>>>> > > > > > > > >
>>>>> > > > > > > >
>>>>> > > > > > >
>>>>> > > > > >
>>>>> > > > >
>>>>> > > >
>>>>> > > > snapshots,
>>>>> > > > > > > release
>>>>> > > > > > > > > > > candidates, or any other similar packages. The
>>>>> only people
>>>>> > > > > > > > > >
>>>>> > > > > > > > >
>>>>> > > > > > > >
>>>>> > > > > > >
>>>>> > > > > >
>>>>> > > > >
>>>>> > > >
>>>>> > > > who are
>>>>> > > > > > > > > >
>>>>> > > > > > > > > > supposed
>>>>> > > > > > > > > > > to know about such developer resources are
>>>>> individuals
>>>>> > > > > > > > > >
>>>>> > > > > > > > >
>>>>> > > > > > > >
>>>>> > > > > > >
>>>>> > > > > >
>>>>> > > > >
>>>>> > > >
>>>>> > > > actively
>>>>> > > > > > > > > > > participating in development or following the dev
>>>>> list and
>>>>> > > > > > > > > >
>>>>> > > > > > > > >
>>>>> > > > > > > >
>>>>> > > > > > >
>>>>> > > > > >
>>>>> > > > >
>>>>> > > >
>>>>> > > > thus
>>>>> > > > > > > aware
>>>>> > > > > > > > > >
>>>>> > > > > > > > >
>>>>> > > > > > > > > of
>>>>> > > > > > > > > > > the conditions placed on unreleased materials.
>>>>> > > > > > > > > > > > On Feb 10 2020, at 9:49 am, Tomasz Urbaszek <
>>>>> > > > > > > > > > >
>>>>> > > > > > > > > > >
>>>>> > > > > > > > > >
>>>>> > > > > > > > > > tomasz.urbaszek@polidea.com>
>>>>> > > > > > > > > > > wrote:
>>>>> > > > > > > > > > > > > As per the frequency of releases maybe we can
>>>>> consider
>>>>> > > > > > > > > > > >
>>>>> > > > > > > > > > >
>>>>> > > > > > > > > >
>>>>> > > > > > > > >
>>>>> > > > > > > >
>>>>> > > > > > >
>>>>> > > > > > > "nightly
>>>>> > > > > > > > > > > > > builds" for providers? In this way any
>>>>> contributed
>>>>> > > > > > > > > > > >
>>>>> > > > > > > > > > >
>>>>> > > > > > > > > >
>>>>> > > > > > > > >
>>>>> > > > > > > >
>>>>> > > > > > >
>>>>> > > > > > > hook/operator
>>>>> > > > > > > > > > > >
>>>>> > > > > > > > > > >
>>>>> > > > > > > > > >
>>>>> > > > > > > > >
>>>>> > > > > > > > > will
>>>>> > > > > > > > > > > > > be pip-installable in 24h, so users can start
>>>>> to use it =
>>>>> > > > > > > > > > > >
>>>>> > > > > > > > > > >
>>>>> > > > > > > > > >
>>>>> > > > > > > > >
>>>>> > > > > > > >
>>>>> > > > > > >
>>>>> > > > > > > test it.
>>>>> > > > > > > > > > > > > This can help us reduce the number of releases
>>>>> with
>>>>> > > > > > > > > > > >
>>>>> > > > > > > > > > >
>>>>> > > > > > > > > >
>>>>> > > > > > > > >
>>>>> > > > > > > >
>>>>> > > > > > >
>>>>> > > > > >
>>>>> > > > >
>>>>> > > >
>>>>> > > > unworking
>>>>> > > > > > > > > > > > > integrations.
>>>>> > > > > > > > > > > > >
>>>>> > > > > > > > > > > > > Tomek
>>>>> > > > > > > > > > > > > On Mon, Feb 10, 2020 at 12:11 AM Jarek Potiuk <
>>>>> > > > > > > > > > > >
>>>>> > > > > > > > > > >
>>>>> > > > > > > > > > > Jarek.Potiuk@polidea.com> wrote:
>>>>> > > > > > > > > > > > > >
>>>>> > > > > > > > > > > > > > TL;DR; I wanted to discuss the approach we
>>>>> are going to
>>>>> > > > > > > take for
>>>>> > > > > > > > > > > backported
>>>>> > > > > > > > > > > > > > providers packages. This is important for
>>>>> PMCs to
>>>>> > > > > > > > > > > > >
>>>>> > > > > > > > > > > >
>>>>> > > > > > > > > > >
>>>>> > > > > > > > > >
>>>>> > > > > > > > >
>>>>> > > > > > > >
>>>>> > > > > > >
>>>>> > > > > >
>>>>> > > > >
>>>>> > > >
>>>>> > > > decide
>>>>> > > > > > > about
>>>>> > > > > > > > > > > > >
>>>>> > > > > > > > > > > >
>>>>> > > > > > > > > > >
>>>>> > > > > > > > > >
>>>>> > > > > > > > >
>>>>> > > > > > > > > how
>>>>> > > > > > > > > > > we are
>>>>> > > > > > > > > > > > > > going to make release process for it, but I
>>>>> wanted to
>>>>> > > > > > > > > > > > >
>>>>> > > > > > > > > > > >
>>>>> > > > > > > > > > >
>>>>> > > > > > > > > >
>>>>> > > > > > > > >
>>>>> > > > > > > >
>>>>> > > > > > >
>>>>> > > > > >
>>>>> > > > >
>>>>> > > >
>>>>> > > > make
>>>>> > > > > > > it
>>>>> > > > > > > > > > > > >
>>>>> > > > > > > > > > > >
>>>>> > > > > > > > > > >
>>>>> > > > > > > > > >
>>>>> > > > > > > > > > public
>>>>> > > > > > > > > > > > > > discussion so that anyone else can chime-in
>>>>> and we can
>>>>> > > > > > > > > > > > >
>>>>> > > > > > > > > > > >
>>>>> > > > > > > > > > >
>>>>> > > > > > > > > >
>>>>> > > > > > > > >
>>>>> > > > > > > >
>>>>> > > > > > >
>>>>> > > > > > > discuss it
>>>>> > > > > > > > > > > > >
>>>>> > > > > > > > > > > >
>>>>> > > > > > > > > > >
>>>>> > > > > > > > > >
>>>>> > > > > > > > > > as
>>>>> > > > > > > > > > > a
>>>>> > > > > > > > > > > > > > community.
>>>>> > > > > > > > > > > > > >
>>>>> > > > > > > > > > > > > > *Context*
>>>>> > > > > > > > > > > > > > As explained in the other thread - we are
>>>>> close to have
>>>>> > > > > > > > > > > > >
>>>>> > > > > > > > > > > > >
>>>>> > > > > > > > > > > >
>>>>> > > > > > > > > > >
>>>>> > > > > > > > > > > releasable/tested
>>>>> > > > > > > > > > > > > > backport packages for Airflow 1.10.* series
>>>>> for
>>>>> > > > > > > > > > > > >
>>>>> > > > > > > > > > > >
>>>>> > > > > > > > > > >
>>>>> > > > > > > > > >
>>>>> > > > > > > > >
>>>>> > > > > > > >
>>>>> > > > > > >
>>>>> > > > > >
>>>>> > > > >
>>>>> > > >
>>>>> > > > "providers"
>>>>> > > > > > > > > > > > > > operators/hooks/packages. The main purpose
>>>>> of those
>>>>> > > > > > > > > > > > >
>>>>> > > > > > > > > > > >
>>>>> > > > > > > > > > >
>>>>> > > > > > > > > >
>>>>> > > > > > > > >
>>>>> > > > > > > >
>>>>> > > > > > >
>>>>> > > > > >
>>>>> > > > >
>>>>> > > >
>>>>> > > > backport
>>>>> > > > > > > > > > > > >
>>>>> > > > > > > > > > > >
>>>>> > > > > > > > > > >
>>>>> > > > > > > > > > > packages is to
>>>>> > > > > > > > > > > > > > let users migrate to the new operators
>>>>> before they
>>>>> > > > > > > > > > > > >
>>>>> > > > > > > > > > > >
>>>>> > > > > > > > > > >
>>>>> > > > > > > > > >
>>>>> > > > > > > > >
>>>>> > > > > > > >
>>>>> > > > > > >
>>>>> > > > > >
>>>>> > > > >
>>>>> > > >
>>>>> > > > migrate
>>>>> > > > > > > to
>>>>> > > > > > > > > > > > >
>>>>> > > > > > > > > > > >
>>>>> > > > > > > > > > >
>>>>> > > > > > > > > >
>>>>> > > > > > > > >
>>>>> > > > > > > > > 2.0.*
>>>>> > > > > > > > > > > version
>>>>> > > > > > > > > > > > > > of Airflow.
>>>>> > > > > > > > > > > > > >
>>>>> > > > > > > > > > > > > > The 2.0 version is still some time in the
>>>>> future, and
>>>>> > > > we
>>>>> > > > > > > have a
>>>>> > > > > > > > > > > number of
>>>>> > > > > > > > > > > > > > operators/hooks/sensors implemented that are
>>>>> not
>>>>> > > > > > > > > > > > >
>>>>> > > > > > > > > > > >
>>>>> > > > > > > > > > >
>>>>> > > > > > > > > >
>>>>> > > > > > > > >
>>>>> > > > > > > >
>>>>> > > > > > >
>>>>> > > > > >
>>>>> > > > >
>>>>> > > >
>>>>> > > > actively
>>>>> > > > > > > > > > > > >
>>>>> > > > > > > > > > > >
>>>>> > > > > > > > > > >
>>>>> > > > > > > > > >
>>>>> > > > > > > > > > used/tests
>>>>> > > > > > > > > > > > > > because they are in master version. There
>>>>> are a number
>>>>> > > > > > > > > > > > >
>>>>> > > > > > > > > > > >
>>>>> > > > > > > > > > >
>>>>> > > > > > > > > >
>>>>> > > > > > > > >
>>>>> > > > > > > >
>>>>> > > > > > >
>>>>> > > > > >
>>>>> > > > >
>>>>> > > >
>>>>> > > > of
>>>>> > > > > > > changes
>>>>> > > > > > > > > > > > >
>>>>> > > > > > > > > > > >
>>>>> > > > > > > > > > >
>>>>> > > > > > > > > > > and fixes
>>>>> > > > > > > > > > > > > > only implemented in master/2.0 so it would
>>>>> be great to
>>>>> > > > > > > > > > > > >
>>>>> > > > > > > > > > > >
>>>>> > > > > > > > > > >
>>>>> > > > > > > > > >
>>>>> > > > > > > > >
>>>>> > > > > > > >
>>>>> > > > > > >
>>>>> > > > > >
>>>>> > > > >
>>>>> > > >
>>>>> > > > use
>>>>> > > > > > > them
>>>>> > > > > > > > > > > > >
>>>>> > > > > > > > > > > >
>>>>> > > > > > > > > > >
>>>>> > > > > > > > > >
>>>>> > > > > > > > >
>>>>> > > > > > > > > in
>>>>> > > > > > > > > > > 1.10 -
>>>>> > > > > > > > > > > > > > to use the new features but also to test the
>>>>> master
>>>>> > > > > > > > > > > > >
>>>>> > > > > > > > > > > >
>>>>> > > > > > > > > > >
>>>>> > > > > > > > > >
>>>>> > > > > > > > >
>>>>> > > > > > > >
>>>>> > > > > > >
>>>>> > > > > > > versions as
>>>>> > > > > > > > > > > > >
>>>>> > > > > > > > > > > >
>>>>> > > > > > > > > > >
>>>>> > > > > > > > > > > early as
>>>>> > > > > > > > > > > > > > possible.
>>>>> > > > > > > > > > > > > >
>>>>> > > > > > > > > > > > > > Another great property of the backport
>>>>> packages is that
>>>>> > > > > > > they can
>>>>> > > > > > > > > be
>>>>> > > > > > > > > > > used to
>>>>> > > > > > > > > > > > > > ease migration process - users can install
>>>>> the
>>>>> > > > > > > > > > > > >
>>>>> > > > > > > > > > > > >
>>>>> > > > > > > > > > > >
>>>>> > > > > > > > > > >
>>>>> > > > > > > > > > > "apache-airflow-providers"
>>>>> > > > > > > > > > > > > > package and start using the new operators
>>>>> without
>>>>> > > > > > > > > > > > >
>>>>> > > > > > > > > > > >
>>>>> > > > > > > > > > >
>>>>> > > > > > > > > >
>>>>> > > > > > > > >
>>>>> > > > > > > >
>>>>> > > > > > >
>>>>> > > > > >
>>>>> > > > >
>>>>> > > >
>>>>> > > > migrating
>>>>> > > > > > > to a
>>>>> > > > > > > > > > > > >
>>>>> > > > > > > > > > > >
>>>>> > > > > > > > > > >
>>>>> > > > > > > > > >
>>>>> > > > > > > > > > new
>>>>> > > > > > > > > > > > > > Airflow. They can incrementally move all
>>>>> their DAGs to
>>>>> > > > > > > > > > > > >
>>>>> > > > > > > > > > > >
>>>>> > > > > > > > > > >
>>>>> > > > > > > > > >
>>>>> > > > > > > > >
>>>>> > > > > > > >
>>>>> > > > > > >
>>>>> > > > > >
>>>>> > > > >
>>>>> > > >
>>>>> > > > use
>>>>> > > > > > > the
>>>>> > > > > > > > > > > > >
>>>>> > > > > > > > > > > >
>>>>> > > > > > > > > > >
>>>>> > > > > > > > > >
>>>>> > > > > > > > >
>>>>> > > > > > > > > new
>>>>> > > > > > > > > > > > > > "providers" package and only after all that
>>>>> is migrated
>>>>> > > > > > > > > > > > >
>>>>> > > > > > > > > > > >
>>>>> > > > > > > > > > >
>>>>> > > > > > > > > >
>>>>> > > > > > > > >
>>>>> > > > > > > >
>>>>> > > > > > >
>>>>> > > > > > > they can
>>>>> > > > > > > > > > > > >
>>>>> > > > > > > > > > > >
>>>>> > > > > > > > > > >
>>>>> > > > > > > > > > > migrate
>>>>> > > > > > > > > > > > > > Airflow to 2.0 when they are ready. That
>>>>> allows to
>>>>> > > > > > > > > > > > >
>>>>> > > > > > > > > > > >
>>>>> > > > > > > > > > >
>>>>> > > > > > > > > >
>>>>> > > > > > > > >
>>>>> > > > > > > >
>>>>> > > > > > >
>>>>> > > > > >
>>>>> > > > >
>>>>> > > >
>>>>> > > > have a
>>>>> > > > > > > smooth
>>>>> > > > > > > > > > > > >
>>>>> > > > > > > > > > > >
>>>>> > > > > > > > > > >
>>>>> > > > > > > > > > > migration
>>>>> > > > > > > > > > > > > > path for those users.
>>>>> > > > > > > > > > > > > >
>>>>> > > > > > > > > > > > > > *Testing*
>>>>> > > > > > > > > > > > > > The issue we have with those packages is
>>>>> that we are
>>>>> > > > > > > > > > > > >
>>>>> > > > > > > > > > > >
>>>>> > > > > > > > > > >
>>>>> > > > > > > > > >
>>>>> > > > > > > > >
>>>>> > > > > > > >
>>>>> > > > > > >
>>>>> > > > > >
>>>>> > > > >
>>>>> > > >
>>>>> > > > not
>>>>> > > > > > > 100%
>>>>> > > > > > > > > > > > >
>>>>> > > > > > > > > > > >
>>>>> > > > > > > > > > >
>>>>> > > > > > > > > >
>>>>> > > > > > > > >
>>>>> > > > > > > > > sure
>>>>> > > > > > > > > > > if the
>>>>> > > > > > > > > > > > > > "providers" operators will work with any
>>>>> 1.10.* airflow
>>>>> > > > > > > > > > > > >
>>>>> > > > > > > > > > > >
>>>>> > > > > > > > > > >
>>>>> > > > > > > > > >
>>>>> > > > > > > > >
>>>>> > > > > > > >
>>>>> > > > > > >
>>>>> > > > > > > version.
>>>>> > > > > > > > > > > > >
>>>>> > > > > > > > > > > >
>>>>> > > > > > > > > > >
>>>>> > > > > > > > > > > There were
>>>>> > > > > > > > > > > > > > no fundamental changes and they SHOULD work
>>>>> - but we
>>>>> > > > > > > > > > > > >
>>>>> > > > > > > > > > > >
>>>>> > > > > > > > > > >
>>>>> > > > > > > > > >
>>>>> > > > > > > > >
>>>>> > > > > > > >
>>>>> > > > > > >
>>>>> > > > > >
>>>>> > > > >
>>>>> > > >
>>>>> > > > never
>>>>> > > > > > > know
>>>>> > > > > > > > > > > > >
>>>>> > > > > > > > > > > >
>>>>> > > > > > > > > > >
>>>>> > > > > > > > > > > until we
>>>>> > > > > > > > > > > > > > test.
>>>>> > > > > > > > > > > > > >
>>>>> > > > > > > > > > > > > > Some preliminary tests with subset of GCP
>>>>> operators
>>>>> > > > show
>>>>> > > > > > > that the
>>>>> > > > > > > > > > > operators
>>>>> > > > > > > > > > > > > > work out-of-the box. We have a big set of
>>>>> "system"
>>>>> > > > > > > > > > > > >
>>>>> > > > > > > > > > > >
>>>>> > > > > > > > > > >
>>>>> > > > > > > > > >
>>>>> > > > > > > > >
>>>>> > > > > > > >
>>>>> > > > > > >
>>>>> > > > > >
>>>>> > > > >
>>>>> > > >
>>>>> > > > tests for
>>>>> > > > > > > > > > > > >
>>>>> > > > > > > > > > > >
>>>>> > > > > > > > > > >
>>>>> > > > > > > > > >
>>>>> > > > > > > > >
>>>>> > > > > > > > > "GCP"
>>>>> > > > > > > > > > > > > > operators that we will run
>>>>> semi-automatically and make
>>>>> > > > > > > > > > > > >
>>>>> > > > > > > > > > > >
>>>>> > > > > > > > > > >
>>>>> > > > > > > > > >
>>>>> > > > > > > > >
>>>>> > > > > > > >
>>>>> > > > > > >
>>>>> > > > > >
>>>>> > > > >
>>>>> > > >
>>>>> > > > sure
>>>>> > > > > > > that
>>>>> > > > > > > > > > > > >
>>>>> > > > > > > > > > > >
>>>>> > > > > > > > > > >
>>>>> > > > > > > > > >
>>>>> > > > > > > > > > all
>>>>> > > > > > > > > > > GCP
>>>>> > > > > > > > > > > > > > operators are working fine. This is already
>>>>> a great
>>>>> > > > > > > > > > > > >
>>>>> > > > > > > > > > > >
>>>>> > > > > > > > > > >
>>>>> > > > > > > > > >
>>>>> > > > > > > > >
>>>>> > > > > > > >
>>>>> > > > > > >
>>>>> > > > > > > compatibility
>>>>> > > > > > > > > > > > >
>>>>> > > > > > > > > > > >
>>>>> > > > > > > > > > >
>>>>> > > > > > > > > > > test (GCP
>>>>> > > > > > > > > > > > > > operators are about 1/3 of all operators for
>>>>> Airflow).
>>>>> > > > > > > > > > > > >
>>>>> > > > > > > > > > > >
>>>>> > > > > > > > > > >
>>>>> > > > > > > > > >
>>>>> > > > > > > > >
>>>>> > > > > > > >
>>>>> > > > > > >
>>>>> > > > > >
>>>>> > > > >
>>>>> > > >
>>>>> > > > But
>>>>> > > > > > > also
>>>>> > > > > > > > > > > > >
>>>>> > > > > > > > > > > >
>>>>> > > > > > > > > > >
>>>>> > > > > > > > > >
>>>>> > > > > > > > >
>>>>> > > > > > > > > the
>>>>> > > > > > > > > > > > > > approach used in GCP system tests can be
>>>>> applied to
>>>>> > > > > > > > > > > > >
>>>>> > > > > > > > > > > >
>>>>> > > > > > > > > > >
>>>>> > > > > > > > > >
>>>>> > > > > > > > >
>>>>> > > > > > > >
>>>>> > > > > > >
>>>>> > > > > >
>>>>> > > > >
>>>>> > > >
>>>>> > > > other
>>>>> > > > > > > > > > > > >
>>>>> > > > > > > > > > > >
>>>>> > > > > > > > > > >
>>>>> > > > > > > > > >
>>>>> > > > > > > > > > operators.
>>>>> > > > > > > > > > > > > >
>>>>> > > > > > > > > > > > > > I plan to have a matrix of "compatibilities"
>>>>> in
>>>>> > > > > > > > > >
>>>>> > > > > > > > >
>>>>> > > > > > >
>>>>> > > >
>>>>> https://cwiki.apache.org/confluence/display/AIRFLOW/Backported+providers+packages+for+Airflow+1.10.*+series
>>>>> > > > > > > > > > > > > > and
>>>>> > > > > > > > > > > > > > ask community to add/run tests with other
>>>>> packages as
>>>>> > > > > > > > > > > > >
>>>>> > > > > > > > > > > >
>>>>> > > > > > > > > > >
>>>>> > > > > > > > > >
>>>>> > > > > > > > >
>>>>> > > > > > > >
>>>>> > > > > > >
>>>>> > > > > >
>>>>> > > > >
>>>>> > > >
>>>>> > > > well.
>>>>> > > > > > > It
>>>>> > > > > > > > > > > > >
>>>>> > > > > > > > > > > >
>>>>> > > > > > > > > > >
>>>>> > > > > > > > > > > should be
>>>>> > > > > > > > > > > > > > rather easy to add system tests for other
>>>>> systems -
>>>>> > > > > > > > > > > > >
>>>>> > > > > > > > > > > >
>>>>> > > > > > > > > > >
>>>>> > > > > > > > > >
>>>>> > > > > > > > >
>>>>> > > > > > > >
>>>>> > > > > > >
>>>>> > > > > > > following the
>>>>> > > > > > > > > > > > >
>>>>> > > > > > > > > > > >
>>>>> > > > > > > > > > >
>>>>> > > > > > > > > > > way it is
>>>>> > > > > > > > > > > > > > implemented for GCP.
>>>>> > > > > > > > > > > > > >
>>>>> > > > > > > > > > > > > > *Releases*
>>>>> > > > > > > > > > > > > > I think the most important decision is how
>>>>> we are
>>>>> > > > > > > > > > > > >
>>>>> > > > > > > > > > > >
>>>>> > > > > > > > > > >
>>>>> > > > > > > > > >
>>>>> > > > > > > > >
>>>>> > > > > > > >
>>>>> > > > > > >
>>>>> > > > > >
>>>>> > > > >
>>>>> > > >
>>>>> > > > going to
>>>>> > > > > > > > > > > > >
>>>>> > > > > > > > > > > >
>>>>> > > > > > > > > > >
>>>>> > > > > > > > > >
>>>>> > > > > > > > >
>>>>> > > > > > > > > release
>>>>> > > > > > > > > > > the
>>>>> > > > > > > > > > > > > > packages. This is where PMCs have to decide
>>>>> I think as
>>>>> > > > > > > > > > > > >
>>>>> > > > > > > > > > > >
>>>>> > > > > > > > > > >
>>>>> > > > > > > > > >
>>>>> > > > > > > > >
>>>>> > > > > > > >
>>>>> > > > > > >
>>>>> > > > > >
>>>>> > > > >
>>>>> > > >
>>>>> > > > we
>>>>> > > > > > > have
>>>>> > > > > > > > > > > > >
>>>>> > > > > > > > > > > >
>>>>> > > > > > > > > > >
>>>>> > > > > > > > > >
>>>>> > > > > > > > > > legal
>>>>> > > > > > > > > > > > > > responsibility for releasing Apache Airflow
>>>>> official
>>>>> > > > > > > > > > > > >
>>>>> > > > > > > > > > > >
>>>>> > > > > > > > > > >
>>>>> > > > > > > > > >
>>>>> > > > > > > > >
>>>>> > > > > > > >
>>>>> > > > > > >
>>>>> > > > > > > software.
>>>>> > > > > > > > > > > > > >
>>>>> > > > > > > > > > > > > > What we have now (after the PRs get merged)
>>>>> - wheel and
>>>>> > > > > > > source
>>>>> > > > > > > > > > > packages
>>>>> > > > > > > > > > > > > > build automatically in Travis CI and
>>>>> uploaded to
>>>>> > > > > > > > > > > > >
>>>>> > > > > > > > > > > >
>>>>> > > > > > > > > > >
>>>>> > > > > > > > > >
>>>>> > > > > > > > >
>>>>> > > > > > > >
>>>>> > > > > > >
>>>>> > > > > >
>>>>> > > > >
>>>>> > > >
>>>>> > > > file.io
>>>>> > > > > > > > > > > > >
>>>>> > > > > > > > > > > >
>>>>> > > > > > > > > > >
>>>>> > > > > > > > > >
>>>>> > > > > > > > >
>>>>> > > > > > > > > ephemeral
>>>>> > > > > > > > > > > storage.
>>>>> > > > > > > > > > > > > > The builds upload all the packages there -
>>>>> one big
>>>>> > > > > > > > > > > > >
>>>>> > > > > > > > > > > >
>>>>> > > > > > > > > > >
>>>>> > > > > > > > > >
>>>>> > > > > > > > >
>>>>> > > > > > > >
>>>>> > > > > > >
>>>>> > > > > > > "providers"
>>>>> > > > > > > > > > > > >
>>>>> > > > > > > > > > > >
>>>>> > > > > > > > > > >
>>>>> > > > > > > > > > > package and
>>>>> > > > > > > > > > > > > > separate packages for each "provider".
>>>>> > > > > > > > > > > > > >
>>>>> > > > > > > > > > > > > > It would be great if we can officially
>>>>> publish
>>>>> > > > packages for
>>>>> > > > > > > > > > > backporting in
>>>>> > > > > > > > > > > > > > pypi however and here where we have to agree
>>>>> on the
>>>>> > > > > > > > > > > > > > process/versioning/cadence.
>>>>> > > > > > > > > > > > > >
>>>>> > > > > > > > > > > > > > We can follow the same process/keys etc as
>>>>> for
>>>>> > > > releasing
>>>>> > > > > > > the main
>>>>> > > > > > > > > > > airflow
>>>>> > > > > > > > > > > > > > package, but I think it can be a bit more
>>>>> relaxed in
>>>>> > > > > > > > > > > > >
>>>>> > > > > > > > > > > >
>>>>> > > > > > > > > > >
>>>>> > > > > > > > > >
>>>>> > > > > > > > >
>>>>> > > > > > > >
>>>>> > > > > > >
>>>>> > > > > >
>>>>> > > > >
>>>>> > > >
>>>>> > > > terms
>>>>> > > > > > > of
>>>>> > > > > > > > > > > > >
>>>>> > > > > > > > > > > >
>>>>> > > > > > > > > > >
>>>>> > > > > > > > > > > testing - and
>>>>> > > > > > > > > > > > > > we can release it more often (as long as
>>>>> there will be
>>>>> > > > > > > > > > > > >
>>>>> > > > > > > > > > > >
>>>>> > > > > > > > > > >
>>>>> > > > > > > > > >
>>>>> > > > > > > > >
>>>>> > > > > > > >
>>>>> > > > > > >
>>>>> > > > > >
>>>>> > > > >
>>>>> > > >
>>>>> > > > new
>>>>> > > > > > > > > > > > >
>>>>> > > > > > > > > > > >
>>>>> > > > > > > > > > >
>>>>> > > > > > > > > >
>>>>> > > > > > > > >
>>>>> > > > > > > > > changes
>>>>> > > > > > > > > > in
>>>>> > > > > > > > > > > > > > providers). Those packages might be released
>>>>> on "as-is"
>>>>> > > > > > > > > > > > >
>>>>> > > > > > > > > > > >
>>>>> > > > > > > > > > >
>>>>> > > > > > > > > >
>>>>> > > > > > > > >
>>>>> > > > > > > >
>>>>> > > > > > >
>>>>> > > > > > > basis -
>>>>> > > > > > > > > > > > >
>>>>> > > > > > > > > > > >
>>>>> > > > > > > > > > >
>>>>> > > > > > > > > > > without
>>>>> > > > > > > > > > > > > > guarantee that they work for all
>>>>> > > > > > > > > > > > >
>>>>> > > > > > > > > > > >
>>>>> > > > > > > > > > >
>>>>> > > > > > > > > >
>>>>> > > > > > > > >
>>>>> > > > > > > >
>>>>> > > > > > >
>>>>> > > > > >
>>>>> > > > >
>>>>> > > >
>>>>> > > > operators/hooks/sensors -
>>>>> > > > > > > and
>>>>> > > > > > > > > > > > >
>>>>> > > > > > > > > > > >
>>>>> > > > > > > > > > >
>>>>> > > > > > > > > > > without
>>>>> > > > > > > > > > > > > > guarantee that they will work for all 1.10.*
>>>>> versions.
>>>>> > > > > > > > > > > > >
>>>>> > > > > > > > > > > >
>>>>> > > > > > > > > > >
>>>>> > > > > > > > > >
>>>>> > > > > > > > >
>>>>> > > > > > > >
>>>>> > > > > > >
>>>>> > > > > >
>>>>> > > > >
>>>>> > > >
>>>>> > > > We
>>>>> > > > > > > can
>>>>> > > > > > > > > > > > >
>>>>> > > > > > > > > > > >
>>>>> > > > > > > > > > >
>>>>> > > > > > > > > >
>>>>> > > > > > > > >
>>>>> > > > > > > > > have
>>>>> > > > > > > > > > > the
>>>>> > > > > > > > > > > > > > "compatibility" statement/matrix in our wiki
>>>>> where
>>>>> > > > > > > > > > > > >
>>>>> > > > > > > > > > > >
>>>>> > > > > > > > > > >
>>>>> > > > > > > > > >
>>>>> > > > > > > > >
>>>>> > > > > > > >
>>>>> > > > > > >
>>>>> > > > > >
>>>>> > > > >
>>>>> > > >
>>>>> > > > people
>>>>> > > > > > > who
>>>>> > > > > > > > > > > > >
>>>>> > > > > > > > > > > >
>>>>> > > > > > > > > > >
>>>>> > > > > > > > > >
>>>>> > > > > > > > > > tested
>>>>> > > > > > > > > > > some
>>>>> > > > > > > > > > > > > > package might simply state that it works for
>>>>> them. At
>>>>> > > > > > > > > > > > >
>>>>> > > > > > > > > > > >
>>>>> > > > > > > > > > >
>>>>> > > > > > > > > >
>>>>> > > > > > > > >
>>>>> > > > > > > >
>>>>> > > > > > >
>>>>> > > > > > > Polidea we
>>>>> > > > > > > > > > > > >
>>>>> > > > > > > > > > > >
>>>>> > > > > > > > > > >
>>>>> > > > > > > > > >
>>>>> > > > > > > > > > can
>>>>> > > > > > > > > > > assume
>>>>> > > > > > > > > > > > > > stewardship on the GCP packages and test
>>>>> them using our
>>>>> > > > > > > > > > > > >
>>>>> > > > > > > > > > > >
>>>>> > > > > > > > > > >
>>>>> > > > > > > > > >
>>>>> > > > > > > > >
>>>>> > > > > > > >
>>>>> > > > > > >
>>>>> > > > > > > automated
>>>>> > > > > > > > > > > > >
>>>>> > > > > > > > > > > >
>>>>> > > > > > > > > > >
>>>>> > > > > > > > > > > system
>>>>> > > > > > > > > > > > > > tests for every release for example - maybe
>>>>> others can
>>>>> > > > > > > > > > > > >
>>>>> > > > > > > > > > > >
>>>>> > > > > > > > > > >
>>>>> > > > > > > > > >
>>>>> > > > > > > > >
>>>>> > > > > > > >
>>>>> > > > > > >
>>>>> > > > > > > assume
>>>>> > > > > > > > > > > > > > stewardship for other providers.
>>>>> > > > > > > > > > > > > >
>>>>> > > > > > > > > > > > > > For that - we will need some
>>>>> versioning/release
>>>>> > > > policy. I
>>>>> > > > > > > would
>>>>> > > > > > > > > say
>>>>> > > > > > > > > > > a CalVer
>>>>> > > > > > > > > > > > > > <https://calver.org/> approach might work
>>>>> best
>>>>> > > > > > > > > > > > >
>>>>> > > > > > > > > > > >
>>>>> > > > > > > > > > >
>>>>> > > > > > > > > >
>>>>> > > > > > > > >
>>>>> > > > > > > >
>>>>> > > > > > >
>>>>> > > > > > > (YYYY.MM.DD). And
>>>>> > > > > > > > > > > > >
>>>>> > > > > > > > > > > >
>>>>> > > > > > > > > > >
>>>>> > > > > > > > > >
>>>>> > > > > > > > > > to
>>>>> > > > > > > > > > > make it
>>>>> > > > > > > > > > > > > > simple we should release one "big" providers
>>>>> package
>>>>> > > > > > > > > > > > >
>>>>> > > > > > > > > > > >
>>>>> > > > > > > > > > >
>>>>> > > > > > > > > >
>>>>> > > > > > > > >
>>>>> > > > > > > >
>>>>> > > > > > >
>>>>> > > > > >
>>>>> > > > >
>>>>> > > >
>>>>> > > > with
>>>>> > > > > > > all
>>>>> > > > > > > > > > > > >
>>>>> > > > > > > > > > > >
>>>>> > > > > > > > > > >
>>>>> > > > > > > > > > > providers in.
>>>>> > > > > > > > > > > > > > We can have roughly monthly cadence for it.
>>>>> > > > > > > > > > > > > >
>>>>> > > > > > > > > > > > > > But I am also open to any suggestions here.
>>>>> > > > > > > > > > > > > > Please let me know what you think.
>>>>> > > > > > > > > > > > > > J.
>>>>> > > > > > > > > > > > > >
>>>>> > > > > > > > > > > > > >
>>>>> > > > > > > > > > > > > >
>>>>> > > > > > > > > > > > > >
>>>>> > > > > > > > > > > > > >
>>>>> > > > > > > > > > > > > >
>>>>> > > > > > > > > > > > > > --
>>>>> > > > > > > > > > > > > > Jarek Potiuk
>>>>> > > > > > > > > > > > > > Polidea <https://www.polidea.com/> |
>>>>> Principal
>>>>> > > > > > > > > > > > >
>>>>> > > > > > > > > > > >
>>>>> > > > > > > > > > >
>>>>> > > > > > > > > >
>>>>> > > > > > > > >
>>>>> > > > > > > >
>>>>> > > > > > >
>>>>> > > > > >
>>>>> > > > >
>>>>> > > >
>>>>> > > > Software
>>>>> > > > > > > Engineer
>>>>> > > > > > > > > > > > > >
>>>>> > > > > > > > > > > > > > M: +48 660 796 129 <+48660796129>
>>>>> > > > > > > > > > > > > > [image: Polidea] <https://www.polidea.com/>
>>>>> > > > > > > > > > > > >
>>>>> > > > > > > > > > > > >
>>>>> > > > > > > > > > > > >
>>>>> > > > > > > > > > > > >
>>>>> > > > > > > > > > > > >
>>>>> > > > > > > > > > > > >
>>>>> > > > > > > > > > > > > --
>>>>> > > > > > > > > > > > > Tomasz Urbaszek
>>>>> > > > > > > > > > > > > Polidea | Software Engineer
>>>>> > > > > > > > > > > > >
>>>>> > > > > > > > > > > > > M: +48 505 628 493
>>>>> > > > > > > > > > > > > E: tomasz.urbaszek@polidea.com
>>>>> > > > > > > > > > > > >
>>>>> > > > > > > > > > > > > Unique Tech
>>>>> > > > > > > > > > > > > Check out our projects!
>>>>> > > > > > > > > > > > >
>>>>> > > > > > > > > > > >
>>>>> > > > > > > > > > >
>>>>> > > > > > > > > > >
>>>>> > > > > > > > > > > --
>>>>> > > > > > > > > > > Tomasz Urbaszek
>>>>> > > > > > > > > > > Polidea | Software Engineer
>>>>> > > > > > > > > > >
>>>>> > > > > > > > > > > M: +48 505 628 493
>>>>> > > > > > > > > > > E: tomasz.urbaszek@polidea.com
>>>>> > > > > > > > > > >
>>>>> > > > > > > > > > > Unique Tech
>>>>> > > > > > > > > > > Check out our projects!
>>>>> > > > > > > > > > >
>>>>> > > > > > > > > >
>>>>> > > > > > > > > >
>>>>> > > > > > > > > > --
>>>>> > > > > > > > > > Jarek Potiuk
>>>>> > > > > > > > > > Polidea <https://www.polidea.com/> | Principal
>>>>> Software
>>>>> > > > > > > > >
>>>>> > > > > > > >
>>>>> > > > > > >
>>>>> > > > > >
>>>>> > > > >
>>>>> > > >
>>>>> > > > Engineer
>>>>> > > > > > > > > >
>>>>> > > > > > > > > > M: +48 660 796 129 <+48660796129>
>>>>> > > > > > > > > > [image: Polidea] <https://www.polidea.com/>
>>>>> > > > > > > > > >
>>>>> > > > > > > > >
>>>>> > > > > > > > >
>>>>> > > > > > > > > --
>>>>> > > > > > > > > Tomasz Urbaszek
>>>>> > > > > > > > > Polidea <https://www.polidea.com/> | Software Engineer
>>>>> > > > > > > > >
>>>>> > > > > > > > > M: +48 505 628 493 <+48505628493>
>>>>> > > > > > > > > E: tomasz.urbaszek@polidea.com <
>>>>> tomasz.urbaszeki@polidea.com>
>>>>> > > > > > > > >
>>>>> > > > > > > > > Unique Tech
>>>>> > > > > > > > > Check out our projects! <
>>>>> https://www.polidea.com/our-work>
>>>>> > > > > > > > >
>>>>> > > > > > > >
>>>>> > > > > > > >
>>>>> > > > > > > > --
>>>>> > > > > > > > Jarek Potiuk
>>>>> > > > > > > > Polidea <https://www.polidea.com/> | Principal Software
>>>>> Engineer
>>>>> > > > > > > >
>>>>> > > > > > > > M: +48 660 796 129 <+48660796129>
>>>>> > > > > > > > [image: Polidea] <https://www.polidea.com/>
>>>>> > > > > > > >
>>>>> > > > > > >
>>>>> > > > > > >
>>>>> > > > > >
>>>>> > > > > > --
>>>>> > > > > > Jarek Potiuk
>>>>> > > > > > Polidea <https://www.polidea.com/> | Principal Software
>>>>> Engineer
>>>>> > > > > >
>>>>> > > > > > M: +48 660 796 129 <+48660796129>
>>>>> > > > > > [image: Polidea] <https://www.polidea.com/>
>>>>> > > > > >
>>>>> > > > > >
>>>>> > > > >
>>>>> > > > > --
>>>>> > > > > Jarek Potiuk
>>>>> > > > > Polidea <https://www.polidea.com/> | Principal Software
>>>>> Engineer
>>>>> > > > >
>>>>> > > > > M: +48 660 796 129 <+48660796129>
>>>>> > > > > [image: Polidea] <https://www.polidea.com/>
>>>>> > > >
>>>>> > > >
>>>>> > >
>>>>> > >
>>>>> > > --
>>>>> > > Jarek Potiuk
>>>>> > > Polidea <https://www.polidea.com/> | Principal Software Engineer
>>>>> > >
>>>>> > > M: +48 660 796 129 <+48660796129>
>>>>> > > [image: Polidea] <https://www.polidea.com/>
>>>>> >
>>>>> >
>>>>>
>>>>>
>>>>
>>>> --
>>>>
>>>> Jarek Potiuk
>>>> Polidea <https://www.polidea.com/> | Principal Software Engineer
>>>>
>>>> M: +48 660 796 129 <+48660796129>
>>>> [image: Polidea] <https://www.polidea.com/>
>>>>
>>>>
>>>
>>> --
>>>
>>> Jarek Potiuk
>>> Polidea <https://www.polidea.com/> | Principal Software Engineer
>>>
>>> M: +48 660 796 129 <+48660796129>
>>> [image: Polidea] <https://www.polidea.com/>
>>>
>>>
>>
>> --
>>
>> Jarek Potiuk
>> Polidea <https://www.polidea.com/> | Principal Software Engineer
>>
>> M: +48 660 796 129 <+48660796129>
>> [image: Polidea] <https://www.polidea.com/>
>>
>>
>
> --
>
> Jarek Potiuk
> Polidea <https://www.polidea.com/> | Principal Software Engineer
>
> M: +48 660 796 129 <+48660796129>
> [image: Polidea] <https://www.polidea.com/>
>
>