You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by David Sanchez Plaza <da...@huawei.com.INVALID> on 2023/03/20 07:16:31 UTC
【Airflow】New provider - Huawei Cloud
Dear community of Airflow,
I am happy to announce that our development team has finished the developments of the providers to connect to Huawei Cloud Services. Therefore, we will open a PR very soon, and as a gesture of respect, we want to request for authorization from community.
Our team is reviewing the guidelines to comply with the requirements.
We will continue to update and add more providers in the near future. We hope this integration can grow customer base and Airflow project.
Best regards
David Sanchez Plaza
Huawei Cloud Business Dept, International Cloud & AI
David Sanchez Plaza - 大卫
Huawei HCIE Cloud Service Solutions Architect (Link<https://www.youracclaim.com/badges/332b8d61-055a-45a1-a301-0c4094a77202/public_url>)
Mobile: +86 17722639223
D District, Huawei Bantian Base, Huawei, Shenzhen, China.
[cid:image002.png@01D4D50A.2F372780] [cid:image004.png@01D4D50A.2F372780] <http://intl.huaweicloud.com/> [cid:image005.png@01D4D50A.2F372780] <https://www.linkedin.com/showcase/huawei-cloud/> [cid:image006.png@01D4D50A.2F372780] <https://www.facebook.com/Huawei-Cloud-1995859487299836/> [cid:image007.png@01D4D50A.2F372780] <https://twitter.com/Huawei_Cloud> [494f24f47d32af5819074491a59e2458] <https://mp.weixin.qq.com/s?__biz=MzI1Mzc1MzMyOQ==&mid=100007026&idx=1&sn=91c07c2a030ab5fee15f24d601da530c&chksm=69cefeec5eb977faae9c29647b7512ec7e0f9bb6301e863c892dde95106d2ec732ff43662666#rd> [5edacfeb19fb3f9d99604eeaee3be363] <https://weibo.com/p/1006061930559805/home?from=page_100606&mod=TAB&is_all=1#place>
Re: 【Airflow】New provider - Huawei Cloud
Posted by Daniel Imberman <da...@gmail.com>.
Hi David,
Ultimately what it comes down to is this: when people add new providers to
Airflow it becomes the PMCs responsibility to maintain them in perpetuity.
If the Huawei cloud provider is broken and we can’t reach you, then it
reflects poorly on the project that we have a broken provider.
If you release the provider on your own GitHub repo we’d be glad to help
you promote it and even mention it in any relevant documentation 🙂.
Best,
Daniel
On Wed, Apr 19, 2023 at 8:03 AM Jarek Potiuk <ja...@potiuk.com> wrote:
> Hello David,
>
> In case you were not following: The important part of the discussion and
> reasoning (and strong preference of external providers) have now been
> captured in our repo
>
> https://github.com/apache/airflow/blob/main/PROVIDERS.rst#accepting-new-community-providers
> .
> You can refer to it when you further think about submitting your provider,
> the whole document is intended so that we do not have to discuss it again,
> next time when someone will ask about it.
>
> Nothing changed, but things got explained and clarified also with reasoning
> why this is the case.
>
> J.
>
>
>
> On Tue, Mar 21, 2023 at 1:48 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>
> > Just to be brutally honest your resources are better allocated to
> > release and maintain the provider on your own.
> >
> > It's not only requirements but you would have to justify why
> > maintenance burden of Huawei Cloud provider is supposed to be done by
> > the community and not Huawei Cloud team (where it would normally
> > belong).
> > I can't see the reason why we would want it in the community to
> > maintain and until there is a good justification, my vote will be -1
> > personally.
> >
> > It's my personal opinion of course and ultimately not the requirements
> > matter but formally a vote would be needed following
> > https://www.apache.org/foundation/voting.html#votes-on-code-modification
> .
> > You might want to convince me and others otherwise, but I seriously
> > doubt you will get it passed the vote if you call for it and there is
> > no good justification for it. I think spending time on releasing it as
> > your provider is a much better spent time for you.
> >
> > J.
> >
> > On Tue, Mar 21, 2023 at 11:07 AM David Sanchez Plaza
> > <da...@huawei.com> wrote:
> > >
> > > Hi Jarek and Airflow community,
> > >
> > >
> > >
> > > First of all, thanks a lot for your detailed explanation.
> > >
> > >
> > >
> > > We will take it internally with our team, and feedback the conclusions.
> > >
> > >
> > >
> > > Our next step is to provide in this email chain with the list of
> > requirements required by Airflow (after investigating all sources and
> links
> > provided). We do please request the community to help us confirm, and add
> > any missing point, to better allocate resources.
> > >
> > >
> > >
> > >
> > >
> > > Thanks a lot!
> > >
> > >
> > >
> > > Best regards
> > >
> > > David Sanchez Plaza
> > >
> > >
> > >
> > > Huawei Cloud Business Dept, International Cloud & AI
> > >
> > > David Sanchez Plaza - 大卫
> > >
> > > Huawei HCIE Cloud Service Solutions Architect (Link)
> > >
> > > Mobile: +86 17722639223
> > >
> > > D District, Huawei Bantian Base, Huawei, Shenzhen, China.
> > >
> > >
> > >
> > >
> > >
> > > From: Jarek Potiuk [mailto:jarek@potiuk.com]
> > > Sent: Monday, March 20, 2023 5:13 PM
> > > To: David Sanchez Plaza <da...@huawei.com>
> > > Cc: dev@airflow.apache.org; Emincan Ozcan (A) <
> > emincan.ozcan1@huawei-partners.com>
> > > Subject: Re: 【Airflow】New provider - Huawei Cloud
> > >
> > >
> > >
> > > Hello David.
> > >
> > >
> > >
> > > I think you should first review the discussions we kept on having for
> at
> > least the last half a year about accepting donations of "service" bound
> > providers from the teams of those services. General consensus is that we
> > are extremely reluctant to get new providers added as part of the
> community
> > providers and our strong preference is that service providers simply
> > release their own providers and have the providers continue to be
> managed,
> > tested and released independently by the respective teams that manage the
> > services.
> > >
> > >
> > >
> > > You (and Huawei Cloud Provider) actually have been part of that
> > discussion already
> > https://lists.apache.org/thread/1gtw5vyypxh0p72wh4dss7cllcvhgh01
> > >
> > >
> > >
> > > There are already a number of such providers released outside of the
> > airflow's community repository and we are happy to add such providers to
> > our "ecosystem page"
> >
> https://airflow.apache.org/ecosystem/#third-party-airflow-plugins-and-providers
> > .
> > >
> > >
> > >
> > > There are multiple reasons why this is better for both - the community
> > and service teams, all of them documented in numerous discussions - so I
> > will not repeat them here.
> > >
> > >
> > >
> > > * https://lists.apache.org/thread/0srsjbdo470vvmk07cx7vpfr3krnyljy -
> > Weaviate
> > >
> > > * https://lists.apache.org/thread/hvl2sg7mc6gwxs1h5kzhrcdtt8cc36dd. -
> > SAS
> > > * https://lists.apache.org/thread/1gtw5vyypxh0p72wh4dss7cllcvhgh01 -
> > the discussion about your provider
> > >
> > > * https://lists.apache.org/thread/qk2co6trd7gm57744shprw2fhgmjr637 -
> > Pandera
> > > * https://lists.apache.org/thread/8b1jvld3npgzz2z0o3gv14lvtornbdrm -
> > Cloudera
> > >
> > >
> > >
> > > In short - adding a new provider to the community for a service that
> has
> > capability of managing, hosting and releasing it on their own, puts
> > unnecessary burden and responsibility of the maintenance and fixing bugs
> on
> > the community where we often have no means, ways of testing those changes
> > and making sure that the provider still works while the service evolves.
> > The Huawei Cloud team is in a much better position to maintain such a
> > provider and make sure it works than anyone here in the community. And we
> > treat such a code as a liability, not an asset, so there must be a really
> > good reason why we would accept it. There is absolutely nothing that
> > Airflow provider released by Huawei cannot do compared to a
> > community-managed provider so there is absolutely no reason why you
> should
> > shy away from releasing it and hosting on your own (Similarly as Grafana,
> > SAS, Cloudera. Pandera, Great Expectations and many others did). There
> are
> > even registries of Airflow Provider (most complete is the
> > >
> > >
> > >
> > > There are a few exceptions historically where we maintain other clouds,
> > but we are working very closely with the cloud provider teams to get on
> the
> > mixed-governance process (as described in
> > https://github.com/apache/airflow#release-process-for-providers) where
> > all the end-to-end testing efforts remain on the side of the service
> > provider and it's responsibility of the service provider to add a
> complete
> > set of "system tests" to the provider and to run them in their cloud and
> > provide the dashboard that can drive our release process:
> > >
> > >
> > >
> > > Example AWS dashboard here:
> > https://aws-mwaa.github.io/open-source/system-tests/dashboard.html
> > >
> > >
> > >
> > > IMHO - having such a dashboard and dedicated team from the cloud
> > provider to manage the system tests in the provider and make sure the
> tests
> > are running daily and take care of their health is an absolute
> > non-negotiable prerequisite (and it should be already running and proven
> > before we accept such provider to the community). And there should be a
> > very good reason why we decide to accept it even if there is such a
> > dashboard.
> > >
> > >
> > >
> > > To be perfectly honest, it's very unlikely IMHO that we will choose the
> > path of accepting your provider rather than simply letting you manage,
> host
> > and release the provider on your own. So I would encourage you not to
> > spend too much time on trying to convince us to accept it. If you have no
> > system tests and similar dashboard running for quite some time, the
> chances
> > for it are really "None".
> > >
> > >
> > >
> > > J.
> > >
> > >
> > >
> > >
> > >
> > > On Mon, Mar 20, 2023 at 8:16 AM David Sanchez Plaza
> > <da...@huawei.com.invalid> wrote:
> > >
> > > Dear community of Airflow,
> > >
> > >
> > >
> > > I am happy to announce that our development team has finished the
> > developments of the providers to connect to Huawei Cloud Services.
> > Therefore, we will open a PR very soon, and as a gesture of respect, we
> > want to request for authorization from community.
> > >
> > >
> > >
> > > Our team is reviewing the guidelines to comply with the requirements.
> > >
> > >
> > >
> > > We will continue to update and add more providers in the near future.
> We
> > hope this integration can grow customer base and Airflow project.
> > >
> > >
> > >
> > > Best regards
> > >
> > > David Sanchez Plaza
> > >
> > >
> > >
> > > Huawei Cloud Business Dept, International Cloud & AI
> > >
> > > David Sanchez Plaza - 大卫
> > >
> > > Huawei HCIE Cloud Service Solutions Architect (Link)
> > >
> > > Mobile: +86 17722639223
> > >
> > > D District, Huawei Bantian Base, Huawei, Shenzhen, China.
> > >
> > >
> > >
> > >
> > >
> > >
> >
>
Re: 【Airflow】New provider - Huawei Cloud
Posted by Jarek Potiuk <ja...@potiuk.com>.
Hello David,
In case you were not following: The important part of the discussion and
reasoning (and strong preference of external providers) have now been
captured in our repo
https://github.com/apache/airflow/blob/main/PROVIDERS.rst#accepting-new-community-providers.
You can refer to it when you further think about submitting your provider,
the whole document is intended so that we do not have to discuss it again,
next time when someone will ask about it.
Nothing changed, but things got explained and clarified also with reasoning
why this is the case.
J.
On Tue, Mar 21, 2023 at 1:48 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> Just to be brutally honest your resources are better allocated to
> release and maintain the provider on your own.
>
> It's not only requirements but you would have to justify why
> maintenance burden of Huawei Cloud provider is supposed to be done by
> the community and not Huawei Cloud team (where it would normally
> belong).
> I can't see the reason why we would want it in the community to
> maintain and until there is a good justification, my vote will be -1
> personally.
>
> It's my personal opinion of course and ultimately not the requirements
> matter but formally a vote would be needed following
> https://www.apache.org/foundation/voting.html#votes-on-code-modification.
> You might want to convince me and others otherwise, but I seriously
> doubt you will get it passed the vote if you call for it and there is
> no good justification for it. I think spending time on releasing it as
> your provider is a much better spent time for you.
>
> J.
>
> On Tue, Mar 21, 2023 at 11:07 AM David Sanchez Plaza
> <da...@huawei.com> wrote:
> >
> > Hi Jarek and Airflow community,
> >
> >
> >
> > First of all, thanks a lot for your detailed explanation.
> >
> >
> >
> > We will take it internally with our team, and feedback the conclusions.
> >
> >
> >
> > Our next step is to provide in this email chain with the list of
> requirements required by Airflow (after investigating all sources and links
> provided). We do please request the community to help us confirm, and add
> any missing point, to better allocate resources.
> >
> >
> >
> >
> >
> > Thanks a lot!
> >
> >
> >
> > Best regards
> >
> > David Sanchez Plaza
> >
> >
> >
> > Huawei Cloud Business Dept, International Cloud & AI
> >
> > David Sanchez Plaza - 大卫
> >
> > Huawei HCIE Cloud Service Solutions Architect (Link)
> >
> > Mobile: +86 17722639223
> >
> > D District, Huawei Bantian Base, Huawei, Shenzhen, China.
> >
> >
> >
> >
> >
> > From: Jarek Potiuk [mailto:jarek@potiuk.com]
> > Sent: Monday, March 20, 2023 5:13 PM
> > To: David Sanchez Plaza <da...@huawei.com>
> > Cc: dev@airflow.apache.org; Emincan Ozcan (A) <
> emincan.ozcan1@huawei-partners.com>
> > Subject: Re: 【Airflow】New provider - Huawei Cloud
> >
> >
> >
> > Hello David.
> >
> >
> >
> > I think you should first review the discussions we kept on having for at
> least the last half a year about accepting donations of "service" bound
> providers from the teams of those services. General consensus is that we
> are extremely reluctant to get new providers added as part of the community
> providers and our strong preference is that service providers simply
> release their own providers and have the providers continue to be managed,
> tested and released independently by the respective teams that manage the
> services.
> >
> >
> >
> > You (and Huawei Cloud Provider) actually have been part of that
> discussion already
> https://lists.apache.org/thread/1gtw5vyypxh0p72wh4dss7cllcvhgh01
> >
> >
> >
> > There are already a number of such providers released outside of the
> airflow's community repository and we are happy to add such providers to
> our "ecosystem page"
> https://airflow.apache.org/ecosystem/#third-party-airflow-plugins-and-providers
> .
> >
> >
> >
> > There are multiple reasons why this is better for both - the community
> and service teams, all of them documented in numerous discussions - so I
> will not repeat them here.
> >
> >
> >
> > * https://lists.apache.org/thread/0srsjbdo470vvmk07cx7vpfr3krnyljy -
> Weaviate
> >
> > * https://lists.apache.org/thread/hvl2sg7mc6gwxs1h5kzhrcdtt8cc36dd. -
> SAS
> > * https://lists.apache.org/thread/1gtw5vyypxh0p72wh4dss7cllcvhgh01 -
> the discussion about your provider
> >
> > * https://lists.apache.org/thread/qk2co6trd7gm57744shprw2fhgmjr637 -
> Pandera
> > * https://lists.apache.org/thread/8b1jvld3npgzz2z0o3gv14lvtornbdrm -
> Cloudera
> >
> >
> >
> > In short - adding a new provider to the community for a service that has
> capability of managing, hosting and releasing it on their own, puts
> unnecessary burden and responsibility of the maintenance and fixing bugs on
> the community where we often have no means, ways of testing those changes
> and making sure that the provider still works while the service evolves.
> The Huawei Cloud team is in a much better position to maintain such a
> provider and make sure it works than anyone here in the community. And we
> treat such a code as a liability, not an asset, so there must be a really
> good reason why we would accept it. There is absolutely nothing that
> Airflow provider released by Huawei cannot do compared to a
> community-managed provider so there is absolutely no reason why you should
> shy away from releasing it and hosting on your own (Similarly as Grafana,
> SAS, Cloudera. Pandera, Great Expectations and many others did). There are
> even registries of Airflow Provider (most complete is the
> >
> >
> >
> > There are a few exceptions historically where we maintain other clouds,
> but we are working very closely with the cloud provider teams to get on the
> mixed-governance process (as described in
> https://github.com/apache/airflow#release-process-for-providers) where
> all the end-to-end testing efforts remain on the side of the service
> provider and it's responsibility of the service provider to add a complete
> set of "system tests" to the provider and to run them in their cloud and
> provide the dashboard that can drive our release process:
> >
> >
> >
> > Example AWS dashboard here:
> https://aws-mwaa.github.io/open-source/system-tests/dashboard.html
> >
> >
> >
> > IMHO - having such a dashboard and dedicated team from the cloud
> provider to manage the system tests in the provider and make sure the tests
> are running daily and take care of their health is an absolute
> non-negotiable prerequisite (and it should be already running and proven
> before we accept such provider to the community). And there should be a
> very good reason why we decide to accept it even if there is such a
> dashboard.
> >
> >
> >
> > To be perfectly honest, it's very unlikely IMHO that we will choose the
> path of accepting your provider rather than simply letting you manage, host
> and release the provider on your own. So I would encourage you not to
> spend too much time on trying to convince us to accept it. If you have no
> system tests and similar dashboard running for quite some time, the chances
> for it are really "None".
> >
> >
> >
> > J.
> >
> >
> >
> >
> >
> > On Mon, Mar 20, 2023 at 8:16 AM David Sanchez Plaza
> <da...@huawei.com.invalid> wrote:
> >
> > Dear community of Airflow,
> >
> >
> >
> > I am happy to announce that our development team has finished the
> developments of the providers to connect to Huawei Cloud Services.
> Therefore, we will open a PR very soon, and as a gesture of respect, we
> want to request for authorization from community.
> >
> >
> >
> > Our team is reviewing the guidelines to comply with the requirements.
> >
> >
> >
> > We will continue to update and add more providers in the near future. We
> hope this integration can grow customer base and Airflow project.
> >
> >
> >
> > Best regards
> >
> > David Sanchez Plaza
> >
> >
> >
> > Huawei Cloud Business Dept, International Cloud & AI
> >
> > David Sanchez Plaza - 大卫
> >
> > Huawei HCIE Cloud Service Solutions Architect (Link)
> >
> > Mobile: +86 17722639223
> >
> > D District, Huawei Bantian Base, Huawei, Shenzhen, China.
> >
> >
> >
> >
> >
> >
>
Re: 【Airflow】New provider - Huawei Cloud
Posted by Jarek Potiuk <ja...@potiuk.com>.
Just to be brutally honest your resources are better allocated to
release and maintain the provider on your own.
It's not only requirements but you would have to justify why
maintenance burden of Huawei Cloud provider is supposed to be done by
the community and not Huawei Cloud team (where it would normally
belong).
I can't see the reason why we would want it in the community to
maintain and until there is a good justification, my vote will be -1
personally.
It's my personal opinion of course and ultimately not the requirements
matter but formally a vote would be needed following
https://www.apache.org/foundation/voting.html#votes-on-code-modification.
You might want to convince me and others otherwise, but I seriously
doubt you will get it passed the vote if you call for it and there is
no good justification for it. I think spending time on releasing it as
your provider is a much better spent time for you.
J.
On Tue, Mar 21, 2023 at 11:07 AM David Sanchez Plaza
<da...@huawei.com> wrote:
>
> Hi Jarek and Airflow community,
>
>
>
> First of all, thanks a lot for your detailed explanation.
>
>
>
> We will take it internally with our team, and feedback the conclusions.
>
>
>
> Our next step is to provide in this email chain with the list of requirements required by Airflow (after investigating all sources and links provided). We do please request the community to help us confirm, and add any missing point, to better allocate resources.
>
>
>
>
>
> Thanks a lot!
>
>
>
> Best regards
>
> David Sanchez Plaza
>
>
>
> Huawei Cloud Business Dept, International Cloud & AI
>
> David Sanchez Plaza - 大卫
>
> Huawei HCIE Cloud Service Solutions Architect (Link)
>
> Mobile: +86 17722639223
>
> D District, Huawei Bantian Base, Huawei, Shenzhen, China.
>
>
>
>
>
> From: Jarek Potiuk [mailto:jarek@potiuk.com]
> Sent: Monday, March 20, 2023 5:13 PM
> To: David Sanchez Plaza <da...@huawei.com>
> Cc: dev@airflow.apache.org; Emincan Ozcan (A) <em...@huawei-partners.com>
> Subject: Re: 【Airflow】New provider - Huawei Cloud
>
>
>
> Hello David.
>
>
>
> I think you should first review the discussions we kept on having for at least the last half a year about accepting donations of "service" bound providers from the teams of those services. General consensus is that we are extremely reluctant to get new providers added as part of the community providers and our strong preference is that service providers simply release their own providers and have the providers continue to be managed, tested and released independently by the respective teams that manage the services.
>
>
>
> You (and Huawei Cloud Provider) actually have been part of that discussion already https://lists.apache.org/thread/1gtw5vyypxh0p72wh4dss7cllcvhgh01
>
>
>
> There are already a number of such providers released outside of the airflow's community repository and we are happy to add such providers to our "ecosystem page" https://airflow.apache.org/ecosystem/#third-party-airflow-plugins-and-providers.
>
>
>
> There are multiple reasons why this is better for both - the community and service teams, all of them documented in numerous discussions - so I will not repeat them here.
>
>
>
> * https://lists.apache.org/thread/0srsjbdo470vvmk07cx7vpfr3krnyljy - Weaviate
>
> * https://lists.apache.org/thread/hvl2sg7mc6gwxs1h5kzhrcdtt8cc36dd. - SAS
> * https://lists.apache.org/thread/1gtw5vyypxh0p72wh4dss7cllcvhgh01 - the discussion about your provider
>
> * https://lists.apache.org/thread/qk2co6trd7gm57744shprw2fhgmjr637 - Pandera
> * https://lists.apache.org/thread/8b1jvld3npgzz2z0o3gv14lvtornbdrm - Cloudera
>
>
>
> In short - adding a new provider to the community for a service that has capability of managing, hosting and releasing it on their own, puts unnecessary burden and responsibility of the maintenance and fixing bugs on the community where we often have no means, ways of testing those changes and making sure that the provider still works while the service evolves. The Huawei Cloud team is in a much better position to maintain such a provider and make sure it works than anyone here in the community. And we treat such a code as a liability, not an asset, so there must be a really good reason why we would accept it. There is absolutely nothing that Airflow provider released by Huawei cannot do compared to a community-managed provider so there is absolutely no reason why you should shy away from releasing it and hosting on your own (Similarly as Grafana, SAS, Cloudera. Pandera, Great Expectations and many others did). There are even registries of Airflow Provider (most complete is the
>
>
>
> There are a few exceptions historically where we maintain other clouds, but we are working very closely with the cloud provider teams to get on the mixed-governance process (as described in https://github.com/apache/airflow#release-process-for-providers) where all the end-to-end testing efforts remain on the side of the service provider and it's responsibility of the service provider to add a complete set of "system tests" to the provider and to run them in their cloud and provide the dashboard that can drive our release process:
>
>
>
> Example AWS dashboard here: https://aws-mwaa.github.io/open-source/system-tests/dashboard.html
>
>
>
> IMHO - having such a dashboard and dedicated team from the cloud provider to manage the system tests in the provider and make sure the tests are running daily and take care of their health is an absolute non-negotiable prerequisite (and it should be already running and proven before we accept such provider to the community). And there should be a very good reason why we decide to accept it even if there is such a dashboard.
>
>
>
> To be perfectly honest, it's very unlikely IMHO that we will choose the path of accepting your provider rather than simply letting you manage, host and release the provider on your own. So I would encourage you not to spend too much time on trying to convince us to accept it. If you have no system tests and similar dashboard running for quite some time, the chances for it are really "None".
>
>
>
> J.
>
>
>
>
>
> On Mon, Mar 20, 2023 at 8:16 AM David Sanchez Plaza <da...@huawei.com.invalid> wrote:
>
> Dear community of Airflow,
>
>
>
> I am happy to announce that our development team has finished the developments of the providers to connect to Huawei Cloud Services. Therefore, we will open a PR very soon, and as a gesture of respect, we want to request for authorization from community.
>
>
>
> Our team is reviewing the guidelines to comply with the requirements.
>
>
>
> We will continue to update and add more providers in the near future. We hope this integration can grow customer base and Airflow project.
>
>
>
> Best regards
>
> David Sanchez Plaza
>
>
>
> Huawei Cloud Business Dept, International Cloud & AI
>
> David Sanchez Plaza - 大卫
>
> Huawei HCIE Cloud Service Solutions Architect (Link)
>
> Mobile: +86 17722639223
>
> D District, Huawei Bantian Base, Huawei, Shenzhen, China.
>
>
>
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
For additional commands, e-mail: dev-help@airflow.apache.org
RE: 【Airflow】New provider - Huawei Cloud
Posted by David Sanchez Plaza <da...@huawei.com.INVALID>.
Hi Jarek and Airflow community,
First of all, thanks a lot for your detailed explanation.
We will take it internally with our team, and feedback the conclusions.
Our next step is to provide in this email chain with the list of requirements required by Airflow (after investigating all sources and links provided). We do please request the community to help us confirm, and add any missing point, to better allocate resources.
Thanks a lot!
Best regards
David Sanchez Plaza
Huawei Cloud Business Dept, International Cloud & AI
David Sanchez Plaza - 大卫
Huawei HCIE Cloud Service Solutions Architect (Link<https://www.youracclaim.com/badges/332b8d61-055a-45a1-a301-0c4094a77202/public_url>)
Mobile: +86 17722639223
D District, Huawei Bantian Base, Huawei, Shenzhen, China.
[cid:image002.png@01D4D50A.2F372780] [cid:image004.png@01D4D50A.2F372780] <http://intl.huaweicloud.com/> [cid:image005.png@01D4D50A.2F372780] <https://www.linkedin.com/showcase/huawei-cloud/> [cid:image006.png@01D4D50A.2F372780] <https://www.facebook.com/Huawei-Cloud-1995859487299836/> [cid:image007.png@01D4D50A.2F372780] <https://twitter.com/Huawei_Cloud> [494f24f47d32af5819074491a59e2458] <https://mp.weixin.qq.com/s?__biz=MzI1Mzc1MzMyOQ==&mid=100007026&idx=1&sn=91c07c2a030ab5fee15f24d601da530c&chksm=69cefeec5eb977faae9c29647b7512ec7e0f9bb6301e863c892dde95106d2ec732ff43662666#rd> [5edacfeb19fb3f9d99604eeaee3be363] <https://weibo.com/p/1006061930559805/home?from=page_100606&mod=TAB&is_all=1#place>
From: Jarek Potiuk [mailto:jarek@potiuk.com]
Sent: Monday, March 20, 2023 5:13 PM
To: David Sanchez Plaza <da...@huawei.com>
Cc: dev@airflow.apache.org; Emincan Ozcan (A) <em...@huawei-partners.com>
Subject: Re: 【Airflow】New provider - Huawei Cloud
Hello David.
I think you should first review the discussions we kept on having for at least the last half a year about accepting donations of "service" bound providers from the teams of those services. General consensus is that we are extremely reluctant to get new providers added as part of the community providers and our strong preference is that service providers simply release their own providers and have the providers continue to be managed, tested and released independently by the respective teams that manage the services.
You (and Huawei Cloud Provider) actually have been part of that discussion already https://lists.apache.org/thread/1gtw5vyypxh0p72wh4dss7cllcvhgh01
There are already a number of such providers released outside of the airflow's community repository and we are happy to add such providers to our "ecosystem page" https://airflow.apache.org/ecosystem/#third-party-airflow-plugins-and-providers.
There are multiple reasons why this is better for both - the community and service teams, all of them documented in numerous discussions - so I will not repeat them here.
* https://lists.apache.org/thread/0srsjbdo470vvmk07cx7vpfr3krnyljy - Weaviate
* https://lists.apache.org/thread/hvl2sg7mc6gwxs1h5kzhrcdtt8cc36dd. - SAS
* https://lists.apache.org/thread/1gtw5vyypxh0p72wh4dss7cllcvhgh01 - the discussion about your provider
* https://lists.apache.org/thread/qk2co6trd7gm57744shprw2fhgmjr637 - Pandera
* https://lists.apache.org/thread/8b1jvld3npgzz2z0o3gv14lvtornbdrm - Cloudera
In short - adding a new provider to the community for a service that has capability of managing, hosting and releasing it on their own, puts unnecessary burden and responsibility of the maintenance and fixing bugs on the community where we often have no means, ways of testing those changes and making sure that the provider still works while the service evolves. The Huawei Cloud team is in a much better position to maintain such a provider and make sure it works than anyone here in the community. And we treat such a code as a liability, not an asset, so there must be a really good reason why we would accept it. There is absolutely nothing that Airflow provider released by Huawei cannot do compared to a community-managed provider so there is absolutely no reason why you should shy away from releasing it and hosting on your own (Similarly as Grafana, SAS, Cloudera. Pandera, Great Expectations and many others did). There are even registries of Airflow Provider (most complete is the
There are a few exceptions historically where we maintain other clouds, but we are working very closely with the cloud provider teams to get on the mixed-governance process (as described in https://github.com/apache/airflow#release-process-for-providers) where all the end-to-end testing efforts remain on the side of the service provider and it's responsibility of the service provider to add a complete set of "system tests" to the provider and to run them in their cloud and provide the dashboard that can drive our release process:
Example AWS dashboard here: https://aws-mwaa.github.io/open-source/system-tests/dashboard.html
IMHO - having such a dashboard and dedicated team from the cloud provider to manage the system tests in the provider and make sure the tests are running daily and take care of their health is an absolute non-negotiable prerequisite (and it should be already running and proven before we accept such provider to the community). And there should be a very good reason why we decide to accept it even if there is such a dashboard.
To be perfectly honest, it's very unlikely IMHO that we will choose the path of accepting your provider rather than simply letting you manage, host and release the provider on your own. So I would encourage you not to spend too much time on trying to convince us to accept it. If you have no system tests and similar dashboard running for quite some time, the chances for it are really "None".
J.
On Mon, Mar 20, 2023 at 8:16 AM David Sanchez Plaza <da...@huawei.com.invalid>> wrote:
Dear community of Airflow,
I am happy to announce that our development team has finished the developments of the providers to connect to Huawei Cloud Services. Therefore, we will open a PR very soon, and as a gesture of respect, we want to request for authorization from community.
Our team is reviewing the guidelines to comply with the requirements.
We will continue to update and add more providers in the near future. We hope this integration can grow customer base and Airflow project.
Best regards
David Sanchez Plaza
Huawei Cloud Business Dept, International Cloud & AI
David Sanchez Plaza - 大卫
Huawei HCIE Cloud Service Solutions Architect (Link<https://www.youracclaim.com/badges/332b8d61-055a-45a1-a301-0c4094a77202/public_url>)
Mobile: +86 17722639223
D District, Huawei Bantian Base, Huawei, Shenzhen, China.
<http://intl.huaweicloud.com/>
Re: 【Airflow】New provider - Huawei Cloud
Posted by Jarek Potiuk <ja...@potiuk.com>.
Hello David.
I think you should first review the discussions we kept on having for at
least the last half a year about accepting donations of "service" bound
providers from the teams of those services. General consensus is that we
are extremely reluctant to get new providers added as part of the community
providers and our strong preference is that service providers simply
release their own providers and have the providers continue to be managed,
tested and released independently by the respective teams that manage the
services.
You (and Huawei Cloud Provider) actually have been part of that
discussion already
https://lists.apache.org/thread/1gtw5vyypxh0p72wh4dss7cllcvhgh01
There are already a number of such providers released outside of the
airflow's community repository and we are happy to add such providers to
our "ecosystem page"
https://airflow.apache.org/ecosystem/#third-party-airflow-plugins-and-providers
.
There are multiple reasons why this is better for both - the community and
service teams, all of them documented in numerous discussions - so I will
not repeat them here.
* https://lists.apache.org/thread/0srsjbdo470vvmk07cx7vpfr3krnyljy -
Weaviate
* https://lists.apache.org/thread/hvl2sg7mc6gwxs1h5kzhrcdtt8cc36dd. - SAS
* https://lists.apache.org/thread/1gtw5vyypxh0p72wh4dss7cllcvhgh01 - the
discussion about your provider
* https://lists.apache.org/thread/qk2co6trd7gm57744shprw2fhgmjr637 - Pandera
* https://lists.apache.org/thread/8b1jvld3npgzz2z0o3gv14lvtornbdrm -
Cloudera
In short - adding a new provider to the community for a service that has
capability of managing, hosting and releasing it on their own, puts
unnecessary burden and responsibility of the maintenance and fixing bugs on
the community where we often have no means, ways of testing those changes
and making sure that the provider still works while the service evolves.
The Huawei Cloud team is in a much better position to maintain such a
provider and make sure it works than anyone here in the community. And we
treat such a code as a liability, not an asset, so there must be a really
good reason why we would accept it. There is absolutely nothing that
Airflow provider released by Huawei cannot do compared to a
community-managed provider so there is absolutely no reason why you should
shy away from releasing it and hosting on your own (Similarly as Grafana,
SAS, Cloudera. Pandera, Great Expectations and many others did). There are
even registries of Airflow Provider (most complete is the
There are a few exceptions historically where we maintain other clouds, but
we are working very closely with the cloud provider teams to get on the
mixed-governance process (as described in
https://github.com/apache/airflow#release-process-for-providers) where all
the end-to-end testing efforts remain on the side of the service provider
and it's responsibility of the service provider to add a complete set of
"system tests" to the provider and to run them in their cloud and provide
the dashboard that can drive our release process:
Example AWS dashboard here:
https://aws-mwaa.github.io/open-source/system-tests/dashboard.html
IMHO - having such a dashboard and dedicated team from the cloud provider
to manage the system tests in the provider and make sure the tests are
running daily and take care of their health is an absolute non-negotiable
prerequisite (and it should be already running and proven before we accept
such provider to the community). And there should be a very good reason why
we decide to accept it even if there is such a dashboard.
To be perfectly honest, it's very unlikely IMHO that we will choose the
path of accepting your provider rather than simply letting you manage, host
and release the provider on your own. So I would encourage you not to
spend too much time on trying to convince us to accept it. If you have no
system tests and similar dashboard running for quite some time, the chances
for it are really "None".
J.
On Mon, Mar 20, 2023 at 8:16 AM David Sanchez Plaza
<da...@huawei.com.invalid> wrote:
> Dear community of Airflow,
>
>
>
> I am happy to announce that our development team has finished the
> developments of the providers to connect to Huawei Cloud Services.
> Therefore, we will open a PR very soon, and as a gesture of respect, we
> want to request for authorization from community.
>
>
>
> Our team is reviewing the guidelines to comply with the requirements.
>
>
>
> We will continue to update and add more providers in the near future. We
> hope this integration can grow customer base and Airflow project.
>
>
>
> Best regards
>
> David Sanchez Plaza
>
>
>
> *Huawei Cloud Business Dept, International Cloud & AI*
>
> *David Sanchez Plaza - **大卫*
>
> *Huawei HCIE Cloud Service Solutions Architect (Link
> <https://www.youracclaim.com/badges/332b8d61-055a-45a1-a301-0c4094a77202/public_url>)*
>
> Mobile: +86 17722639223
>
> D District, Huawei Bantian Base, Huawei, Shenzhen, China.
>
> [image: cid:image002.png@01D4D50A.2F372780] [image:
> cid:image004.png@01D4D50A.2F372780] <http://intl.huaweicloud.com/> [image:
> cid:image005.png@01D4D50A.2F372780]
> <https://www.linkedin.com/showcase/huawei-cloud/> [image:
> cid:image006.png@01D4D50A.2F372780]
> <https://www.facebook.com/Huawei-Cloud-1995859487299836/> [image:
> cid:image007.png@01D4D50A.2F372780] <https://twitter.com/Huawei_Cloud> [image:
> 494f24f47d32af5819074491a59e2458]
> <https://mp.weixin.qq.com/s?__biz=MzI1Mzc1MzMyOQ==&mid=100007026&idx=1&sn=91c07c2a030ab5fee15f24d601da530c&chksm=69cefeec5eb977faae9c29647b7512ec7e0f9bb6301e863c892dde95106d2ec732ff43662666#rd>
> [image: 5edacfeb19fb3f9d99604eeaee3be363]
> <https://weibo.com/p/1006061930559805/home?from=page_100606&mod=TAB&is_all=1#place>
>
>
>