You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Blain David <da...@infrabel.be> on 2024/03/01 07:56:43 UTC
RE: [PROPOSAL] Adding MSGraphSDK Async Operator to Airflow
Hello Jarek and Elad,
Indeed maintenance could be a concern, but I think that's already the same case regarding all other Microsoft related providers in Airflow.
Also know that I also already contributed to the Microsoft Graph Python SDK project on the Microsoft repo: https://github.com/microsoftgraph/msgraph-sdk-python
I also already contributed to the Airflow providers fixing bugs and doing enhancements, so there is a commitment on my part also 😉
The hook, operator and trigger are well tested, I always try to deliver code that is tested as much as possible, we have a score of 90% on our sonarqube and only a technical debt of 26 mins (which will probably decrease).
In the meantime I've been finetuning and testing the operator in our environment, we now have multiple DAG's using it.
Bellow some advantages of using the MSGraphSDKOperator:
- handles and refreshes bearer tokens automatically thanks to the azure provider classes by Microsoft, which you don't need to maintain you just use it.
- the operator is fully async using triggerers so requests and responses are handled by separate workers, which is non-blocking resources for nothing, which is good as the Microsoft implementation only allows async calls anyway.
- you don't need to worry about paging, this is also handled automatically asynchronously for you as Microsoft is following the OData spec, the operator can handle this in a generic way (https://learn.microsoft.com/en-us/odata/client/pagination).
- uses the newer httpx library (https://github.com/encode/httpx), which in our case as we are behind a corporate proxy, solves a lot of proxy related connection issues which we encounter with the requests library but don't when using httpx.
- only depends on the msgraph-core and the kiota_abstractions library from Microsoft (https://github.com/microsoft/kiota-abstractions-python), which are the foundation libs on which their full msgraph_sdk dependency is build, which we don't need as this is only useful when you want to use their generated Python client, which doesn't make sense in Airflow.
- as it's Microsoft, it's also compatible calling the PowerBI API, we already had multiple DA'sG using this operator to call dataset refreshes on the PowerBI REST API.
- probably the Intune API will also work with this operator, I'm going to test this soon once we are migrating all our intune related custom jobs to Airflow DAG's.
- another advantage is that all parameters can be defined in a Http Airflow connection, independently if you want to interact with MS Graph or PowerBI.
- as mentioned before, the hook, operator and trigger are well tested, we have a score of 90% on our sonar and only a technical debt of 26 mins.
The operator could be beneficial for everyone, as it avoids needing custom code to achieve the same using the regular HttpOperator, which I think could also need refactoring and maybe should be migrated using the httpx library.
I'm willing to contribute there also, maybe we can come to a point that the MSGraphSDKOperator could be based on the refactored HttpOperator, in the meantime it would be nice if the operator would be already available in Airflow
What do you guys think?
Kind regards,
David
-----Original Message-----
From: Jarek Potiuk <ja...@potiuk.com>
Sent: Friday, 26 January 2024 17:25
To: dev@airflow.apache.org
Subject: Re: [PROPOSAL] Adding MSGraphSDK Async Operator to Airflow
[You don't often get email from jarek@potiuk.com. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]
EXTERNAL MAIL: Indien je de afzender van deze e-mail niet kent en deze niet vertrouwt, klik niet op een link of open geen bijlages. Bij twijfel, stuur deze e-mail als bijlage naar abuse@infrabel.be<ma...@infrabel.be>.
Hey David - any comments on that ?
On Mon, Jan 15, 2024 at 1:07 PM Elad Kalif <el...@apache.org> wrote:
> Hi David,
>
> Thanks for raising this discussion.
> following the protocol established about accepting new providers -
>
> https://gith/
> ub.com%2Fapache%2Fairflow%2Fblob%2Fmain%2FPROVIDERS.rst%23accepting-ne
> w-community-providers&data=05%7C02%7Cdavid.blain%40infrabel.be%7C2da54
> 87c207f4c39debd08dc1e8b75ac%7Cb82bc314ab8e4d6fb18946f02e1f27f2%7C0%7C0
> %7C638418831528623158%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQ
> IjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=mrnCt
> dP4HLfDYw1HeMfylfZkEv0p%2BMO4X3Sn6QJu8hU%3D&reserved=0
> My main concern here is how will provide ongoing mantaince for this
> provider?
> This provider is to handle a service by Microsoft yet Microsoft is not
> in the picture here (as far as I can see)
>
>
> On Sat, Jan 13, 2024 at 12:39 PM Blain David <da...@infrabel.be>
> wrote:
>
> > Hello everyone,
> >
> > I've already started a discussion about this on the Airflow discussions:
> > https://gi/
> > thub.com%2Fapache%2Fairflow%2Fdiscussions%2F36315&data=05%7C02%7Cdav
> > id.blain%40infrabel.be%7C2da5487c207f4c39debd08dc1e8b75ac%7Cb82bc314
> > ab8e4d6fb18946f02e1f27f2%7C0%7C0%7C638418831528632454%7CUnknown%7CTW
> > FpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVC
> > I6Mn0%3D%7C3000%7C%7C%7C&sdata=WLeKg%2FSCcqkEGVGpK9OvuUaQIc02GAjjglM
> > tPkGeWWQ%3D&reserved=0
> >
> > As we have multiple DAG's interacting with MS Graph API endpoints,
> > and as we want to avoid custom code as much as possible as we have
> > to handle
> lot's
> > of data due to paging.
> > We thought of implementing an operator for it using the official
> > python client from Microsoft<
> https://gith/
> ub.com%2Fmicrosoftgraph%2Fmsgraph-sdk-python&data=05%7C02%7Cdavid.blai
> n%40infrabel.be%7C2da5487c207f4c39debd08dc1e8b75ac%7Cb82bc314ab8e4d6fb
> 18946f02e1f27f2%7C0%7C0%7C638418831528638199%7CUnknown%7CTWFpbGZsb3d8e
> yJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C30
> 00%7C%7C%7C&sdata=qdH4r0OJxGjYXJ8jIQn41BS%2BIH64Nj8%2BKWSsYL2iobo%3D&r
> eserved=0>,
> > that way we can simplify our DAGs and remove custom code as much as
> > possible.
> > That's why we implemented an MSGraph SDK Provider which is on
> > GitHub<
> > https://gi/
> > thub.com%2Finfrabel%2Fapache-airflow-providers-msgraph&data=05%7C02%7Cdavid.blain%40infrabel.be%7C2da5487c207f4c39debd08dc1e8b75ac%7Cb82bc314ab8e4d6fb18946f02e1f27f2%7C0%7C0%7C638418831528642865%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Gs27ZiTInN1qI3%2FVojdkFFP%2BV4OLSeGTQ6sbFBItdsA%3D&reserved=0> and also published it as an artifact on PyPi< https://pypi.org/project/apache-airflow-providers-msgraph/>.
> >
> > As Jarek pointed out in the pull request for Third Party providers<
> > https://gi/
> > thub.com%2Fapache%2Fairflow-site%2Fpull%2F933&data=05%7C02%7Cdavid.b
> > lain%40infrabel.be%7C2da5487c207f4c39debd08dc1e8b75ac%7Cb82bc314ab8e
> > 4d6fb18946f02e1f27f2%7C0%7C0%7C638418831528651442%7CUnknown%7CTWFpbG
> > Zsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn
> > 0%3D%7C3000%7C%7C%7C&sdata=UbwEODVhHelR%2FAKDtW5pMiZ169qILlL4Vqe4ZD8
> > Mkak%3D&reserved=0>, it's not a good idea
> to
> > use the apache-airflow prefix for the library as it is not
> > maintained by the Apache community (yet).
> > So there are 2 options, or we change the name or we donate the
> > project to the Apache Airflow community, which I think the later one
> > is the best option if there is interest to do it, hence why I also
> > started a
> discussion<
> > https://gi/
> > thub.com%2Fapache%2Fairflow%2Fdiscussions%2F36315&data=05%7C02%7Cdav
> > id.blain%40infrabel.be%7C2da5487c207f4c39debd08dc1e8b75ac%7Cb82bc314
> > ab8e4d6fb18946f02e1f27f2%7C0%7C0%7C638418831528655433%7CUnknown%7CTW
> > FpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVC
> > I6Mn0%3D%7C3000%7C%7C%7C&sdata=IwiIleDSIkBC1IpYRYxOBtFHQAUBceLBvqZWs
> > cKsG6s%3D&reserved=0> about it a few
> weeks
> > ago on GitHub<https://github.com/apache/airflow/discussions/36315>.
> >
> > Kind regards,
> > David
> >
> >
>
RE: [PROPOSAL] Adding MSGraphSDK Async Operator to Airflow
Posted by Blain David <da...@infrabel.be>.
Alright that's good news thanks Jarek :)
-----Original Message-----
From: Jarek Potiuk <ja...@potiuk.com>
Sent: Monday, 11 March 2024 18:31
To: dev@airflow.apache.org
Cc: Bienvenu Joffrey <jo...@infrabel.be>
Subject: Re: [PROPOSAL] Adding MSGraphSDK Async Operator to Airflow
EXTERNAL MAIL: Indien je de afzender van deze e-mail niet kent en deze niet vertrouwt, klik niet op een link of open geen bijlages. Bij twijfel, stuur deze e-mail als bijlage naar abuse@infrabel.be<ma...@infrabel.be>.
Then - just add it as another operator in Azure :).
It's not more than yet-another-operator for Azure service. It would be a different story if it was a different "service" - with completely different stakeholders and maintenance burden involved. Being a "just another Azure operator" it adds very little overhead and generally we do not need to discuss it - just getting good quality PR with tests, docs etc. is enough.
On Mon, Mar 11, 2024 at 6:27 PM Blain David <da...@infrabel.be> wrote:
>
> Hello Jarek,
>
> There is no particular need to add this into a separate provider, I just did it as I wanted to deploy it myself, it could perfectly reside within the Microsoft Azure provider one.
>
> There is nothing special about it, except it depends on the msgraph-core dependency (which is light and doesn't have the generated client code the msgraph-sdk has which we don't need in Airflow) which facilitates the REST calls.
> In my first POC it used the full msgraph-sdk dependency, which did not make sense unless you would use the full blow client through the Hook in Airflow.
> But as the main focus of this provider was to avoid as much as possible custom python code in our DAG's, it made no sense.
> The azure-identity dependency is also needed but that one is already present in Airflow as the Microsoft Azure provider already uses it for the Fabric hooks.
> So indeed, it's just another hook, triggerer and operator, with the particularity it only works in async (e.g. deferred) mode.
>
> What do you think?
>
> Kind regards,
> David
>
> -----Original Message-----
> From: Jarek Potiuk <ja...@potiuk.com>
> Sent: Sunday, 10 March 2024 18:19
> To: dev@airflow.apache.org
> Cc: Bienvenu Joffrey <jo...@infrabel.be>
> Subject: Re: [PROPOSAL] Adding MSGraphSDK Async Operator to Airflow
>
> EXTERNAL MAIL: Indien je de afzender van deze e-mail niet kent en deze niet vertrouwt, klik niet op een link of open geen bijlages. Bij twijfel, stuur deze e-mail als bijlage naar abuse@infrabel.be<ma...@infrabel.be>.
>
> Question - is there any reason you want to add it as a separate provider and not just another operator in the azure provider ? I looked at the code and it's not a lot, and I see no particular reason why it should not be simply yet another operator/Hook there. Just as many other operators - I was under the impression there is something special about the SDK you use or "proprietaredness" (for lack of a better word) - but that seems like yet-another operator, hook, triggerer in `microsoft.azure`.
>
> Or am I missing something?
>
> J.
>
> On Fri, Mar 1, 2024 at 8:57 AM Blain David <da...@infrabel.be> wrote:
>
> > Hello Jarek and Elad,
> >
> > Indeed maintenance could be a concern, but I think that's already
> > the same case regarding all other Microsoft related providers in Airflow.
> > Also know that I also already contributed to the Microsoft Graph
> > Python SDK project on the Microsoft repo:
> > https://gith/
> > ub.com%2Fmicrosoftgraph%2Fmsgraph-sdk-python&data=05%7C02%7Cdavid.bl
> > ai
> > n%40infrabel.be%7C049ea8fb06b54fbd934f08dc412649ed%7Cb82bc314ab8e4d6
> > fb
> > 18946f02e1f27f2%7C0%7C0%7C638456879909688297%7CUnknown%7CTWFpbGZsb3d
> > 8e
> > yJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C
> > 0%
> > 7C%7C%7C&sdata=l7lEGyQ3IGHA7wOA4MFmDaQDLtF5PE3dSmxFWZEXgdc%3D&reserv
> > ed
> > =0 I also already contributed to the Airflow providers fixing bugs
> > and doing enhancements, so there is a commitment on my part also 😉
> > The hook, operator and trigger are well tested, I always try to
> > deliver code that is tested as much as possible, we have a score of
> > 90% on our sonarqube and only a technical debt of 26 mins (which
> > will probably decrease).
> >
> > In the meantime I've been finetuning and testing the operator in our
> > environment, we now have multiple DAG's using it.
> >
> > Bellow some advantages of using the MSGraphSDKOperator:
> > - handles and refreshes bearer tokens automatically thanks to the
> > azure provider classes by Microsoft, which you don't need to
> > maintain you just use it.
> > - the operator is fully async using triggerers so requests and
> > responses are handled by separate workers, which is non-blocking
> > resources for nothing, which is good as the Microsoft implementation
> > only allows async calls anyway.
> > - you don't need to worry about paging, this is also handled
> > automatically asynchronously for you as Microsoft is following the
> > OData spec, the operator can handle this in a generic way (
> > https://learn.microsoft.com/en-us/odata/client/pagination).
> > - uses the newer httpx library
> > (https://git/
> > hub.com%2Fencode%2Fhttpx&data=05%7C02%7Cdavid.blain%40infrabel.be%7C
> > 04
> > 9ea8fb06b54fbd934f08dc412649ed%7Cb82bc314ab8e4d6fb18946f02e1f27f2%7C0%7C0%7C638456879909701865%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=Q24YPUOKEYPx%2FJ9ux1eyGraZ6YeWkGRuD0Z%2B40nFrak%3D&reserved=0), which in our case as we are behind a corporate proxy, solves a lot of proxy related connection issues which we encounter with the requests library but don't when using httpx.
> > - only depends on the msgraph-core and the kiota_abstractions
> > library from Microsoft (https://git/
> > hub.com%2Fmicrosoft%2Fkiota-abstractions-python&data=05%7C02%7Cdavid
> > .b
> > lain%40infrabel.be%7C049ea8fb06b54fbd934f08dc412649ed%7Cb82bc314ab8e4d6fb18946f02e1f27f2%7C0%7C0%7C638456879909705929%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=qCdOyIeVSoN%2F%2BmTFuEyGmrM1TZERyA0Axgtbv6PtnOo%3D&reserved=0), which are the foundation libs on which their full msgraph_sdk dependency is build, which we don't need as this is only useful when you want to use their generated Python client, which doesn't make sense in Airflow.
> > - as it's Microsoft, it's also compatible calling the PowerBI API,
> > we already had multiple DA'sG using this operator to call dataset
> > refreshes on the PowerBI REST API.
> > - probably the Intune API will also work with this operator, I'm
> > going to test this soon once we are migrating all our intune related
> > custom jobs to Airflow DAG's.
> > - another advantage is that all parameters can be defined in a Http
> > Airflow connection, independently if you want to interact with MS
> > Graph or PowerBI.
> > - as mentioned before, the hook, operator and trigger are well
> > tested, we have a score of 90% on our sonar and only a technical debt of 26 mins.
> >
> > The operator could be beneficial for everyone, as it avoids needing
> > custom code to achieve the same using the regular HttpOperator,
> > which I think could also need refactoring and maybe should be
> > migrated using the httpx library.
> > I'm willing to contribute there also, maybe we can come to a point
> > that the MSGraphSDKOperator could be based on the refactored
> > HttpOperator, in the meantime it would be nice if the operator would
> > be already available in Airflow
> >
> > What do you guys think?
> >
> > Kind regards,
> > David
> >
> >
> > -----Original Message-----
> > From: Jarek Potiuk <ja...@potiuk.com>
> > Sent: Friday, 26 January 2024 17:25
> > To: dev@airflow.apache.org
> > Subject: Re: [PROPOSAL] Adding MSGraphSDK Async Operator to Airflow
> >
> > [You don't often get email from jarek@potiuk.com. Learn why this is
> > important at https://aka.ms/LearnAboutSenderIdentification ]
> >
> > EXTERNAL MAIL: Indien je de afzender van deze e-mail niet kent en
> > deze niet vertrouwt, klik niet op een link of open geen bijlages.
> > Bij twijfel, stuur deze e-mail als bijlage naar abuse@infrabel.be<mailto:
> > abuse@infrabel.be>.
> >
> > Hey David - any comments on that ?
> >
> > On Mon, Jan 15, 2024 at 1:07 PM Elad Kalif <el...@apache.org> wrote:
> >
> > > Hi David,
> > >
> > > Thanks for raising this discussion.
> > > following the protocol established about accepting new providers -
> > >
> > > https://gith/
> > > ub.com%2Fapache%2Fairflow%2Fblob%2Fmain%2FPROVIDERS.rst%23acceptin
> > > g-
> > > ne
> > > w-community-providers&data=05%7C02%7Cdavid.blain%40infrabel.be%7C2
> > > da
> > > 54
> > > 87c207f4c39debd08dc1e8b75ac%7Cb82bc314ab8e4d6fb18946f02e1f27f2%7C0
> > > %7
> > > C0
> > > %7C638418831528623158%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAi
> > > LC
> > > JQ
> > > IjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=m
> > > rn
> > > Ct
> > > dP4HLfDYw1HeMfylfZkEv0p%2BMO4X3Sn6QJu8hU%3D&reserved=0
> > > My main concern here is how will provide ongoing mantaince for
> > > this provider?
> > > This provider is to handle a service by Microsoft yet Microsoft is
> > > not in the picture here (as far as I can see)
> > >
> > >
> > > On Sat, Jan 13, 2024 at 12:39 PM Blain David
> > > <da...@infrabel.be>
> > > wrote:
> > >
> > > > Hello everyone,
> > > >
> > > > I've already started a discussion about this on the Airflow
> > discussions:
> > > > https://gi/
> > > > thub.com%2Fapache%2Fairflow%2Fdiscussions%2F36315&data=05%7C02%7
> > > > Cd
> > > > av
> > > > id.blain%40infrabel.be%7C2da5487c207f4c39debd08dc1e8b75ac%7Cb82b
> > > > c3
> > > > 14
> > > > ab8e4d6fb18946f02e1f27f2%7C0%7C0%7C638418831528632454%7CUnknown%
> > > > 7C
> > > > TW
> > > > FpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLC
> > > > JX
> > > > VC
> > > > I6Mn0%3D%7C3000%7C%7C%7C&sdata=WLeKg%2FSCcqkEGVGpK9OvuUaQIc02GAj
> > > > jg
> > > > lM
> > > > tPkGeWWQ%3D&reserved=0
> > > >
> > > > As we have multiple DAG's interacting with MS Graph API
> > > > endpoints, and as we want to avoid custom code as much as
> > > > possible as we have to handle
> > > lot's
> > > > of data due to paging.
> > > > We thought of implementing an operator for it using the official
> > > > python client from Microsoft<
> > > https://gith/
> > > ub.com%2Fmicrosoftgraph%2Fmsgraph-sdk-python&data=05%7C02%7Cdavid.
> > > bl
> > > ai
> > > n%40infrabel.be%7C2da5487c207f4c39debd08dc1e8b75ac%7Cb82bc314ab8e4
> > > d6
> > > fb
> > > 18946f02e1f27f2%7C0%7C0%7C638418831528638199%7CUnknown%7CTWFpbGZsb
> > > 3d
> > > 8e
> > > yJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%
> > > 7C
> > > 30
> > > 00%7C%7C%7C&sdata=qdH4r0OJxGjYXJ8jIQn41BS%2BIH64Nj8%2BKWSsYL2iobo%
> > > 3D
> > > &r
> > > eserved=0>,
> > > > that way we can simplify our DAGs and remove custom code as much
> > > > as possible.
> > > > That's why we implemented an MSGraph SDK Provider which is on
> > > > GitHub< https://gi/ thub.com
> > %2Finfrabel%2Fapache-airflow-providers-msgraph&data=05%7C02%7Cdavid.
> > bl
> > ain%
> > 40infrabel.be%7C2da5487c207f4c39debd08dc1e8b75ac%7Cb82bc314ab8e4d6fb
> > 18
> > 946f02e1f27f2%7C0%7C0%7C638418831528642865%7CUnknown%7CTWFpbGZsb3d8e
> > yJ
> > WIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C30
> > 00
> > %7C%7C%7C&sdata=Gs27ZiTInN1qI3%2FVojdkFFP%2BV4OLSeGTQ6sbFBItdsA%3D&r
> > es erved=0> and also published it as an artifact on PyPi<
> > https://pypi.org/project/apache-airflow-providers-msgraph/>.
> > > >
> > > > As Jarek pointed out in the pull request for Third Party
> > > > providers< https://gi/
> > > > thub.com%2Fapache%2Fairflow-site%2Fpull%2F933&data=05%7C02%7Cdav
> > > > id
> > > > .b
> > > > lain%40infrabel.be%7C2da5487c207f4c39debd08dc1e8b75ac%7Cb82bc314
> > > > ab
> > > > 8e
> > > > 4d6fb18946f02e1f27f2%7C0%7C0%7C638418831528651442%7CUnknown%7CTW
> > > > Fp
> > > > bG
> > > > Zsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVC
> > > > I6
> > > > Mn
> > > > 0%3D%7C3000%7C%7C%7C&sdata=UbwEODVhHelR%2FAKDtW5pMiZ169qILlL4Vqe
> > > > 4Z
> > > > D8 Mkak%3D&reserved=0>, it's not a good idea
> > > to
> > > > use the apache-airflow prefix for the library as it is not
> > > > maintained by the Apache community (yet).
> > > > So there are 2 options, or we change the name or we donate the
> > > > project to the Apache Airflow community, which I think the later
> > > > one is the best option if there is interest to do it, hence why
> > > > I also started a
> > > discussion<
> > > > https://gi/
> > > > thub.com%2Fapache%2Fairflow%2Fdiscussions%2F36315&data=05%7C02%7
> > > > Cd
> > > > av
> > > > id.blain%40infrabel.be%7C2da5487c207f4c39debd08dc1e8b75ac%7Cb82b
> > > > c3
> > > > 14
> > > > ab8e4d6fb18946f02e1f27f2%7C0%7C0%7C638418831528655433%7CUnknown%
> > > > 7C
> > > > TW
> > > > FpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLC
> > > > JX
> > > > VC
> > > > I6Mn0%3D%7C3000%7C%7C%7C&sdata=IwiIleDSIkBC1IpYRYxOBtFHQAUBceLBv
> > > > qZ Ws cKsG6s%3D&reserved=0> about it a few
> > > weeks
> > > > ago on GitHub<https://github.com/apache/airflow/discussions/36315>.
> > > >
> > > > Kind regards,
> > > > David
> > > >
> > > >
> > >
> >
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
For additional commands, e-mail: dev-help@airflow.apache.org
Re: [PROPOSAL] Adding MSGraphSDK Async Operator to Airflow
Posted by Jarek Potiuk <ja...@potiuk.com>.
Then - just add it as another operator in Azure :).
It's not more than yet-another-operator for Azure service. It would be
a different story if it was a different "service" - with completely
different stakeholders and maintenance burden involved. Being a "just
another Azure operator" it adds very little overhead and generally we
do not need to discuss it - just getting good quality PR with tests,
docs etc. is enough.
On Mon, Mar 11, 2024 at 6:27 PM Blain David <da...@infrabel.be> wrote:
>
> Hello Jarek,
>
> There is no particular need to add this into a separate provider, I just did it as I wanted to deploy it myself, it could perfectly reside within the Microsoft Azure provider one.
>
> There is nothing special about it, except it depends on the msgraph-core dependency (which is light and doesn't have the generated client code the msgraph-sdk has which we don't need in Airflow) which facilitates the REST calls.
> In my first POC it used the full msgraph-sdk dependency, which did not make sense unless you would use the full blow client through the Hook in Airflow.
> But as the main focus of this provider was to avoid as much as possible custom python code in our DAG's, it made no sense.
> The azure-identity dependency is also needed but that one is already present in Airflow as the Microsoft Azure provider already uses it for the Fabric hooks.
> So indeed, it's just another hook, triggerer and operator, with the particularity it only works in async (e.g. deferred) mode.
>
> What do you think?
>
> Kind regards,
> David
>
> -----Original Message-----
> From: Jarek Potiuk <ja...@potiuk.com>
> Sent: Sunday, 10 March 2024 18:19
> To: dev@airflow.apache.org
> Cc: Bienvenu Joffrey <jo...@infrabel.be>
> Subject: Re: [PROPOSAL] Adding MSGraphSDK Async Operator to Airflow
>
> EXTERNAL MAIL: Indien je de afzender van deze e-mail niet kent en deze niet vertrouwt, klik niet op een link of open geen bijlages. Bij twijfel, stuur deze e-mail als bijlage naar abuse@infrabel.be<ma...@infrabel.be>.
>
> Question - is there any reason you want to add it as a separate provider and not just another operator in the azure provider ? I looked at the code and it's not a lot, and I see no particular reason why it should not be simply yet another operator/Hook there. Just as many other operators - I was under the impression there is something special about the SDK you use or "proprietaredness" (for lack of a better word) - but that seems like yet-another operator, hook, triggerer in `microsoft.azure`.
>
> Or am I missing something?
>
> J.
>
> On Fri, Mar 1, 2024 at 8:57 AM Blain David <da...@infrabel.be> wrote:
>
> > Hello Jarek and Elad,
> >
> > Indeed maintenance could be a concern, but I think that's already the
> > same case regarding all other Microsoft related providers in Airflow.
> > Also know that I also already contributed to the Microsoft Graph
> > Python SDK project on the Microsoft repo:
> > https://gith/
> > ub.com%2Fmicrosoftgraph%2Fmsgraph-sdk-python&data=05%7C02%7Cdavid.blai
> > n%40infrabel.be%7C049ea8fb06b54fbd934f08dc412649ed%7Cb82bc314ab8e4d6fb
> > 18946f02e1f27f2%7C0%7C0%7C638456879909688297%7CUnknown%7CTWFpbGZsb3d8e
> > yJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%
> > 7C%7C%7C&sdata=l7lEGyQ3IGHA7wOA4MFmDaQDLtF5PE3dSmxFWZEXgdc%3D&reserved
> > =0 I also already contributed to the Airflow providers fixing bugs and
> > doing enhancements, so there is a commitment on my part also 😉
> > The hook, operator and trigger are well tested, I always try to
> > deliver code that is tested as much as possible, we have a score of
> > 90% on our sonarqube and only a technical debt of 26 mins (which will
> > probably decrease).
> >
> > In the meantime I've been finetuning and testing the operator in our
> > environment, we now have multiple DAG's using it.
> >
> > Bellow some advantages of using the MSGraphSDKOperator:
> > - handles and refreshes bearer tokens automatically thanks to the
> > azure provider classes by Microsoft, which you don't need to maintain
> > you just use it.
> > - the operator is fully async using triggerers so requests and
> > responses are handled by separate workers, which is non-blocking
> > resources for nothing, which is good as the Microsoft implementation
> > only allows async calls anyway.
> > - you don't need to worry about paging, this is also handled
> > automatically asynchronously for you as Microsoft is following the
> > OData spec, the operator can handle this in a generic way (
> > https://learn.microsoft.com/en-us/odata/client/pagination).
> > - uses the newer httpx library
> > (https://git/
> > hub.com%2Fencode%2Fhttpx&data=05%7C02%7Cdavid.blain%40infrabel.be%7C04
> > 9ea8fb06b54fbd934f08dc412649ed%7Cb82bc314ab8e4d6fb18946f02e1f27f2%7C0%7C0%7C638456879909701865%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=Q24YPUOKEYPx%2FJ9ux1eyGraZ6YeWkGRuD0Z%2B40nFrak%3D&reserved=0), which in our case as we are behind a corporate proxy, solves a lot of proxy related connection issues which we encounter with the requests library but don't when using httpx.
> > - only depends on the msgraph-core and the kiota_abstractions library
> > from Microsoft
> > (https://git/
> > hub.com%2Fmicrosoft%2Fkiota-abstractions-python&data=05%7C02%7Cdavid.b
> > lain%40infrabel.be%7C049ea8fb06b54fbd934f08dc412649ed%7Cb82bc314ab8e4d6fb18946f02e1f27f2%7C0%7C0%7C638456879909705929%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=qCdOyIeVSoN%2F%2BmTFuEyGmrM1TZERyA0Axgtbv6PtnOo%3D&reserved=0), which are the foundation libs on which their full msgraph_sdk dependency is build, which we don't need as this is only useful when you want to use their generated Python client, which doesn't make sense in Airflow.
> > - as it's Microsoft, it's also compatible calling the PowerBI API, we
> > already had multiple DA'sG using this operator to call dataset
> > refreshes on the PowerBI REST API.
> > - probably the Intune API will also work with this operator, I'm going
> > to test this soon once we are migrating all our intune related custom
> > jobs to Airflow DAG's.
> > - another advantage is that all parameters can be defined in a Http
> > Airflow connection, independently if you want to interact with MS
> > Graph or PowerBI.
> > - as mentioned before, the hook, operator and trigger are well tested,
> > we have a score of 90% on our sonar and only a technical debt of 26 mins.
> >
> > The operator could be beneficial for everyone, as it avoids needing
> > custom code to achieve the same using the regular HttpOperator, which
> > I think could also need refactoring and maybe should be migrated using
> > the httpx library.
> > I'm willing to contribute there also, maybe we can come to a point
> > that the MSGraphSDKOperator could be based on the refactored
> > HttpOperator, in the meantime it would be nice if the operator would
> > be already available in Airflow
> >
> > What do you guys think?
> >
> > Kind regards,
> > David
> >
> >
> > -----Original Message-----
> > From: Jarek Potiuk <ja...@potiuk.com>
> > Sent: Friday, 26 January 2024 17:25
> > To: dev@airflow.apache.org
> > Subject: Re: [PROPOSAL] Adding MSGraphSDK Async Operator to Airflow
> >
> > [You don't often get email from jarek@potiuk.com. Learn why this is
> > important at https://aka.ms/LearnAboutSenderIdentification ]
> >
> > EXTERNAL MAIL: Indien je de afzender van deze e-mail niet kent en deze
> > niet vertrouwt, klik niet op een link of open geen bijlages. Bij
> > twijfel, stuur deze e-mail als bijlage naar abuse@infrabel.be<mailto:
> > abuse@infrabel.be>.
> >
> > Hey David - any comments on that ?
> >
> > On Mon, Jan 15, 2024 at 1:07 PM Elad Kalif <el...@apache.org> wrote:
> >
> > > Hi David,
> > >
> > > Thanks for raising this discussion.
> > > following the protocol established about accepting new providers -
> > >
> > > https://gith/
> > > ub.com%2Fapache%2Fairflow%2Fblob%2Fmain%2FPROVIDERS.rst%23accepting-
> > > ne
> > > w-community-providers&data=05%7C02%7Cdavid.blain%40infrabel.be%7C2da
> > > 54
> > > 87c207f4c39debd08dc1e8b75ac%7Cb82bc314ab8e4d6fb18946f02e1f27f2%7C0%7
> > > C0
> > > %7C638418831528623158%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLC
> > > JQ
> > > IjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=mrn
> > > Ct
> > > dP4HLfDYw1HeMfylfZkEv0p%2BMO4X3Sn6QJu8hU%3D&reserved=0
> > > My main concern here is how will provide ongoing mantaince for this
> > > provider?
> > > This provider is to handle a service by Microsoft yet Microsoft is
> > > not in the picture here (as far as I can see)
> > >
> > >
> > > On Sat, Jan 13, 2024 at 12:39 PM Blain David
> > > <da...@infrabel.be>
> > > wrote:
> > >
> > > > Hello everyone,
> > > >
> > > > I've already started a discussion about this on the Airflow
> > discussions:
> > > > https://gi/
> > > > thub.com%2Fapache%2Fairflow%2Fdiscussions%2F36315&data=05%7C02%7Cd
> > > > av
> > > > id.blain%40infrabel.be%7C2da5487c207f4c39debd08dc1e8b75ac%7Cb82bc3
> > > > 14
> > > > ab8e4d6fb18946f02e1f27f2%7C0%7C0%7C638418831528632454%7CUnknown%7C
> > > > TW
> > > > FpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJX
> > > > VC
> > > > I6Mn0%3D%7C3000%7C%7C%7C&sdata=WLeKg%2FSCcqkEGVGpK9OvuUaQIc02GAjjg
> > > > lM
> > > > tPkGeWWQ%3D&reserved=0
> > > >
> > > > As we have multiple DAG's interacting with MS Graph API endpoints,
> > > > and as we want to avoid custom code as much as possible as we have
> > > > to handle
> > > lot's
> > > > of data due to paging.
> > > > We thought of implementing an operator for it using the official
> > > > python client from Microsoft<
> > > https://gith/
> > > ub.com%2Fmicrosoftgraph%2Fmsgraph-sdk-python&data=05%7C02%7Cdavid.bl
> > > ai
> > > n%40infrabel.be%7C2da5487c207f4c39debd08dc1e8b75ac%7Cb82bc314ab8e4d6
> > > fb
> > > 18946f02e1f27f2%7C0%7C0%7C638418831528638199%7CUnknown%7CTWFpbGZsb3d
> > > 8e
> > > yJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C
> > > 30
> > > 00%7C%7C%7C&sdata=qdH4r0OJxGjYXJ8jIQn41BS%2BIH64Nj8%2BKWSsYL2iobo%3D
> > > &r
> > > eserved=0>,
> > > > that way we can simplify our DAGs and remove custom code as much
> > > > as possible.
> > > > That's why we implemented an MSGraph SDK Provider which is on
> > > > GitHub< https://gi/ thub.com
> > %2Finfrabel%2Fapache-airflow-providers-msgraph&data=05%7C02%7Cdavid.bl
> > ain%
> > 40infrabel.be%7C2da5487c207f4c39debd08dc1e8b75ac%7Cb82bc314ab8e4d6fb18
> > 946f02e1f27f2%7C0%7C0%7C638418831528642865%7CUnknown%7CTWFpbGZsb3d8eyJ
> > WIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000
> > %7C%7C%7C&sdata=Gs27ZiTInN1qI3%2FVojdkFFP%2BV4OLSeGTQ6sbFBItdsA%3D&res
> > erved=0> and also published it as an artifact on PyPi<
> > https://pypi.org/project/apache-airflow-providers-msgraph/>.
> > > >
> > > > As Jarek pointed out in the pull request for Third Party
> > > > providers< https://gi/
> > > > thub.com%2Fapache%2Fairflow-site%2Fpull%2F933&data=05%7C02%7Cdavid
> > > > .b
> > > > lain%40infrabel.be%7C2da5487c207f4c39debd08dc1e8b75ac%7Cb82bc314ab
> > > > 8e
> > > > 4d6fb18946f02e1f27f2%7C0%7C0%7C638418831528651442%7CUnknown%7CTWFp
> > > > bG
> > > > Zsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6
> > > > Mn
> > > > 0%3D%7C3000%7C%7C%7C&sdata=UbwEODVhHelR%2FAKDtW5pMiZ169qILlL4Vqe4Z
> > > > D8 Mkak%3D&reserved=0>, it's not a good idea
> > > to
> > > > use the apache-airflow prefix for the library as it is not
> > > > maintained by the Apache community (yet).
> > > > So there are 2 options, or we change the name or we donate the
> > > > project to the Apache Airflow community, which I think the later
> > > > one is the best option if there is interest to do it, hence why I
> > > > also started a
> > > discussion<
> > > > https://gi/
> > > > thub.com%2Fapache%2Fairflow%2Fdiscussions%2F36315&data=05%7C02%7Cd
> > > > av
> > > > id.blain%40infrabel.be%7C2da5487c207f4c39debd08dc1e8b75ac%7Cb82bc3
> > > > 14
> > > > ab8e4d6fb18946f02e1f27f2%7C0%7C0%7C638418831528655433%7CUnknown%7C
> > > > TW
> > > > FpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJX
> > > > VC
> > > > I6Mn0%3D%7C3000%7C%7C%7C&sdata=IwiIleDSIkBC1IpYRYxOBtFHQAUBceLBvqZ
> > > > Ws cKsG6s%3D&reserved=0> about it a few
> > > weeks
> > > > ago on GitHub<https://github.com/apache/airflow/discussions/36315>.
> > > >
> > > > Kind regards,
> > > > David
> > > >
> > > >
> > >
> >
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
For additional commands, e-mail: dev-help@airflow.apache.org
RE: [PROPOSAL] Adding MSGraphSDK Async Operator to Airflow
Posted by Blain David <da...@infrabel.be>.
Hello Jarek,
There is no particular need to add this into a separate provider, I just did it as I wanted to deploy it myself, it could perfectly reside within the Microsoft Azure provider one.
There is nothing special about it, except it depends on the msgraph-core dependency (which is light and doesn't have the generated client code the msgraph-sdk has which we don't need in Airflow) which facilitates the REST calls.
In my first POC it used the full msgraph-sdk dependency, which did not make sense unless you would use the full blow client through the Hook in Airflow.
But as the main focus of this provider was to avoid as much as possible custom python code in our DAG's, it made no sense.
The azure-identity dependency is also needed but that one is already present in Airflow as the Microsoft Azure provider already uses it for the Fabric hooks.
So indeed, it's just another hook, triggerer and operator, with the particularity it only works in async (e.g. deferred) mode.
What do you think?
Kind regards,
David
-----Original Message-----
From: Jarek Potiuk <ja...@potiuk.com>
Sent: Sunday, 10 March 2024 18:19
To: dev@airflow.apache.org
Cc: Bienvenu Joffrey <jo...@infrabel.be>
Subject: Re: [PROPOSAL] Adding MSGraphSDK Async Operator to Airflow
EXTERNAL MAIL: Indien je de afzender van deze e-mail niet kent en deze niet vertrouwt, klik niet op een link of open geen bijlages. Bij twijfel, stuur deze e-mail als bijlage naar abuse@infrabel.be<ma...@infrabel.be>.
Question - is there any reason you want to add it as a separate provider and not just another operator in the azure provider ? I looked at the code and it's not a lot, and I see no particular reason why it should not be simply yet another operator/Hook there. Just as many other operators - I was under the impression there is something special about the SDK you use or "proprietaredness" (for lack of a better word) - but that seems like yet-another operator, hook, triggerer in `microsoft.azure`.
Or am I missing something?
J.
On Fri, Mar 1, 2024 at 8:57 AM Blain David <da...@infrabel.be> wrote:
> Hello Jarek and Elad,
>
> Indeed maintenance could be a concern, but I think that's already the
> same case regarding all other Microsoft related providers in Airflow.
> Also know that I also already contributed to the Microsoft Graph
> Python SDK project on the Microsoft repo:
> https://gith/
> ub.com%2Fmicrosoftgraph%2Fmsgraph-sdk-python&data=05%7C02%7Cdavid.blai
> n%40infrabel.be%7C049ea8fb06b54fbd934f08dc412649ed%7Cb82bc314ab8e4d6fb
> 18946f02e1f27f2%7C0%7C0%7C638456879909688297%7CUnknown%7CTWFpbGZsb3d8e
> yJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%
> 7C%7C%7C&sdata=l7lEGyQ3IGHA7wOA4MFmDaQDLtF5PE3dSmxFWZEXgdc%3D&reserved
> =0 I also already contributed to the Airflow providers fixing bugs and
> doing enhancements, so there is a commitment on my part also 😉
> The hook, operator and trigger are well tested, I always try to
> deliver code that is tested as much as possible, we have a score of
> 90% on our sonarqube and only a technical debt of 26 mins (which will
> probably decrease).
>
> In the meantime I've been finetuning and testing the operator in our
> environment, we now have multiple DAG's using it.
>
> Bellow some advantages of using the MSGraphSDKOperator:
> - handles and refreshes bearer tokens automatically thanks to the
> azure provider classes by Microsoft, which you don't need to maintain
> you just use it.
> - the operator is fully async using triggerers so requests and
> responses are handled by separate workers, which is non-blocking
> resources for nothing, which is good as the Microsoft implementation
> only allows async calls anyway.
> - you don't need to worry about paging, this is also handled
> automatically asynchronously for you as Microsoft is following the
> OData spec, the operator can handle this in a generic way (
> https://learn.microsoft.com/en-us/odata/client/pagination).
> - uses the newer httpx library
> (https://git/
> hub.com%2Fencode%2Fhttpx&data=05%7C02%7Cdavid.blain%40infrabel.be%7C04
> 9ea8fb06b54fbd934f08dc412649ed%7Cb82bc314ab8e4d6fb18946f02e1f27f2%7C0%7C0%7C638456879909701865%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=Q24YPUOKEYPx%2FJ9ux1eyGraZ6YeWkGRuD0Z%2B40nFrak%3D&reserved=0), which in our case as we are behind a corporate proxy, solves a lot of proxy related connection issues which we encounter with the requests library but don't when using httpx.
> - only depends on the msgraph-core and the kiota_abstractions library
> from Microsoft
> (https://git/
> hub.com%2Fmicrosoft%2Fkiota-abstractions-python&data=05%7C02%7Cdavid.b
> lain%40infrabel.be%7C049ea8fb06b54fbd934f08dc412649ed%7Cb82bc314ab8e4d6fb18946f02e1f27f2%7C0%7C0%7C638456879909705929%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=qCdOyIeVSoN%2F%2BmTFuEyGmrM1TZERyA0Axgtbv6PtnOo%3D&reserved=0), which are the foundation libs on which their full msgraph_sdk dependency is build, which we don't need as this is only useful when you want to use their generated Python client, which doesn't make sense in Airflow.
> - as it's Microsoft, it's also compatible calling the PowerBI API, we
> already had multiple DA'sG using this operator to call dataset
> refreshes on the PowerBI REST API.
> - probably the Intune API will also work with this operator, I'm going
> to test this soon once we are migrating all our intune related custom
> jobs to Airflow DAG's.
> - another advantage is that all parameters can be defined in a Http
> Airflow connection, independently if you want to interact with MS
> Graph or PowerBI.
> - as mentioned before, the hook, operator and trigger are well tested,
> we have a score of 90% on our sonar and only a technical debt of 26 mins.
>
> The operator could be beneficial for everyone, as it avoids needing
> custom code to achieve the same using the regular HttpOperator, which
> I think could also need refactoring and maybe should be migrated using
> the httpx library.
> I'm willing to contribute there also, maybe we can come to a point
> that the MSGraphSDKOperator could be based on the refactored
> HttpOperator, in the meantime it would be nice if the operator would
> be already available in Airflow
>
> What do you guys think?
>
> Kind regards,
> David
>
>
> -----Original Message-----
> From: Jarek Potiuk <ja...@potiuk.com>
> Sent: Friday, 26 January 2024 17:25
> To: dev@airflow.apache.org
> Subject: Re: [PROPOSAL] Adding MSGraphSDK Async Operator to Airflow
>
> [You don't often get email from jarek@potiuk.com. Learn why this is
> important at https://aka.ms/LearnAboutSenderIdentification ]
>
> EXTERNAL MAIL: Indien je de afzender van deze e-mail niet kent en deze
> niet vertrouwt, klik niet op een link of open geen bijlages. Bij
> twijfel, stuur deze e-mail als bijlage naar abuse@infrabel.be<mailto:
> abuse@infrabel.be>.
>
> Hey David - any comments on that ?
>
> On Mon, Jan 15, 2024 at 1:07 PM Elad Kalif <el...@apache.org> wrote:
>
> > Hi David,
> >
> > Thanks for raising this discussion.
> > following the protocol established about accepting new providers -
> >
> > https://gith/
> > ub.com%2Fapache%2Fairflow%2Fblob%2Fmain%2FPROVIDERS.rst%23accepting-
> > ne
> > w-community-providers&data=05%7C02%7Cdavid.blain%40infrabel.be%7C2da
> > 54
> > 87c207f4c39debd08dc1e8b75ac%7Cb82bc314ab8e4d6fb18946f02e1f27f2%7C0%7
> > C0
> > %7C638418831528623158%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLC
> > JQ
> > IjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=mrn
> > Ct
> > dP4HLfDYw1HeMfylfZkEv0p%2BMO4X3Sn6QJu8hU%3D&reserved=0
> > My main concern here is how will provide ongoing mantaince for this
> > provider?
> > This provider is to handle a service by Microsoft yet Microsoft is
> > not in the picture here (as far as I can see)
> >
> >
> > On Sat, Jan 13, 2024 at 12:39 PM Blain David
> > <da...@infrabel.be>
> > wrote:
> >
> > > Hello everyone,
> > >
> > > I've already started a discussion about this on the Airflow
> discussions:
> > > https://gi/
> > > thub.com%2Fapache%2Fairflow%2Fdiscussions%2F36315&data=05%7C02%7Cd
> > > av
> > > id.blain%40infrabel.be%7C2da5487c207f4c39debd08dc1e8b75ac%7Cb82bc3
> > > 14
> > > ab8e4d6fb18946f02e1f27f2%7C0%7C0%7C638418831528632454%7CUnknown%7C
> > > TW
> > > FpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJX
> > > VC
> > > I6Mn0%3D%7C3000%7C%7C%7C&sdata=WLeKg%2FSCcqkEGVGpK9OvuUaQIc02GAjjg
> > > lM
> > > tPkGeWWQ%3D&reserved=0
> > >
> > > As we have multiple DAG's interacting with MS Graph API endpoints,
> > > and as we want to avoid custom code as much as possible as we have
> > > to handle
> > lot's
> > > of data due to paging.
> > > We thought of implementing an operator for it using the official
> > > python client from Microsoft<
> > https://gith/
> > ub.com%2Fmicrosoftgraph%2Fmsgraph-sdk-python&data=05%7C02%7Cdavid.bl
> > ai
> > n%40infrabel.be%7C2da5487c207f4c39debd08dc1e8b75ac%7Cb82bc314ab8e4d6
> > fb
> > 18946f02e1f27f2%7C0%7C0%7C638418831528638199%7CUnknown%7CTWFpbGZsb3d
> > 8e
> > yJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C
> > 30
> > 00%7C%7C%7C&sdata=qdH4r0OJxGjYXJ8jIQn41BS%2BIH64Nj8%2BKWSsYL2iobo%3D
> > &r
> > eserved=0>,
> > > that way we can simplify our DAGs and remove custom code as much
> > > as possible.
> > > That's why we implemented an MSGraph SDK Provider which is on
> > > GitHub< https://gi/ thub.com
> %2Finfrabel%2Fapache-airflow-providers-msgraph&data=05%7C02%7Cdavid.bl
> ain%
> 40infrabel.be%7C2da5487c207f4c39debd08dc1e8b75ac%7Cb82bc314ab8e4d6fb18
> 946f02e1f27f2%7C0%7C0%7C638418831528642865%7CUnknown%7CTWFpbGZsb3d8eyJ
> WIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000
> %7C%7C%7C&sdata=Gs27ZiTInN1qI3%2FVojdkFFP%2BV4OLSeGTQ6sbFBItdsA%3D&res
> erved=0> and also published it as an artifact on PyPi<
> https://pypi.org/project/apache-airflow-providers-msgraph/>.
> > >
> > > As Jarek pointed out in the pull request for Third Party
> > > providers< https://gi/
> > > thub.com%2Fapache%2Fairflow-site%2Fpull%2F933&data=05%7C02%7Cdavid
> > > .b
> > > lain%40infrabel.be%7C2da5487c207f4c39debd08dc1e8b75ac%7Cb82bc314ab
> > > 8e
> > > 4d6fb18946f02e1f27f2%7C0%7C0%7C638418831528651442%7CUnknown%7CTWFp
> > > bG
> > > Zsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6
> > > Mn
> > > 0%3D%7C3000%7C%7C%7C&sdata=UbwEODVhHelR%2FAKDtW5pMiZ169qILlL4Vqe4Z
> > > D8 Mkak%3D&reserved=0>, it's not a good idea
> > to
> > > use the apache-airflow prefix for the library as it is not
> > > maintained by the Apache community (yet).
> > > So there are 2 options, or we change the name or we donate the
> > > project to the Apache Airflow community, which I think the later
> > > one is the best option if there is interest to do it, hence why I
> > > also started a
> > discussion<
> > > https://gi/
> > > thub.com%2Fapache%2Fairflow%2Fdiscussions%2F36315&data=05%7C02%7Cd
> > > av
> > > id.blain%40infrabel.be%7C2da5487c207f4c39debd08dc1e8b75ac%7Cb82bc3
> > > 14
> > > ab8e4d6fb18946f02e1f27f2%7C0%7C0%7C638418831528655433%7CUnknown%7C
> > > TW
> > > FpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJX
> > > VC
> > > I6Mn0%3D%7C3000%7C%7C%7C&sdata=IwiIleDSIkBC1IpYRYxOBtFHQAUBceLBvqZ
> > > Ws cKsG6s%3D&reserved=0> about it a few
> > weeks
> > > ago on GitHub<https://github.com/apache/airflow/discussions/36315>.
> > >
> > > Kind regards,
> > > David
> > >
> > >
> >
>
Re: [PROPOSAL] Adding MSGraphSDK Async Operator to Airflow
Posted by Jarek Potiuk <ja...@potiuk.com>.
Question - is there any reason you want to add it as a separate provider
and not just another operator in the azure provider ? I looked at the code
and it's not a lot, and I see no particular reason why it should not be
simply yet another operator/Hook there. Just as many other operators - I
was under the impression there is something special about the SDK you use
or "proprietaredness" (for lack of a better word) - but that seems like
yet-another operator, hook, triggerer in `microsoft.azure`.
Or am I missing something?
J.
On Fri, Mar 1, 2024 at 8:57 AM Blain David <da...@infrabel.be> wrote:
> Hello Jarek and Elad,
>
> Indeed maintenance could be a concern, but I think that's already the same
> case regarding all other Microsoft related providers in Airflow.
> Also know that I also already contributed to the Microsoft Graph Python
> SDK project on the Microsoft repo:
> https://github.com/microsoftgraph/msgraph-sdk-python
> I also already contributed to the Airflow providers fixing bugs and doing
> enhancements, so there is a commitment on my part also 😉
> The hook, operator and trigger are well tested, I always try to deliver
> code that is tested as much as possible, we have a score of 90% on our
> sonarqube and only a technical debt of 26 mins (which will probably
> decrease).
>
> In the meantime I've been finetuning and testing the operator in our
> environment, we now have multiple DAG's using it.
>
> Bellow some advantages of using the MSGraphSDKOperator:
> - handles and refreshes bearer tokens automatically thanks to the azure
> provider classes by Microsoft, which you don't need to maintain you just
> use it.
> - the operator is fully async using triggerers so requests and responses
> are handled by separate workers, which is non-blocking resources for
> nothing, which is good as the Microsoft implementation only allows async
> calls anyway.
> - you don't need to worry about paging, this is also handled automatically
> asynchronously for you as Microsoft is following the OData spec, the
> operator can handle this in a generic way (
> https://learn.microsoft.com/en-us/odata/client/pagination).
> - uses the newer httpx library (https://github.com/encode/httpx), which
> in our case as we are behind a corporate proxy, solves a lot of proxy
> related connection issues which we encounter with the requests library but
> don't when using httpx.
> - only depends on the msgraph-core and the kiota_abstractions library from
> Microsoft (https://github.com/microsoft/kiota-abstractions-python), which
> are the foundation libs on which their full msgraph_sdk dependency is
> build, which we don't need as this is only useful when you want to use
> their generated Python client, which doesn't make sense in Airflow.
> - as it's Microsoft, it's also compatible calling the PowerBI API, we
> already had multiple DA'sG using this operator to call dataset refreshes on
> the PowerBI REST API.
> - probably the Intune API will also work with this operator, I'm going to
> test this soon once we are migrating all our intune related custom jobs to
> Airflow DAG's.
> - another advantage is that all parameters can be defined in a Http
> Airflow connection, independently if you want to interact with MS Graph or
> PowerBI.
> - as mentioned before, the hook, operator and trigger are well tested, we
> have a score of 90% on our sonar and only a technical debt of 26 mins.
>
> The operator could be beneficial for everyone, as it avoids needing custom
> code to achieve the same using the regular HttpOperator, which I think
> could also need refactoring and maybe should be migrated using the httpx
> library.
> I'm willing to contribute there also, maybe we can come to a point that
> the MSGraphSDKOperator could be based on the refactored HttpOperator, in
> the meantime it would be nice if the operator would be already available in
> Airflow
>
> What do you guys think?
>
> Kind regards,
> David
>
>
> -----Original Message-----
> From: Jarek Potiuk <ja...@potiuk.com>
> Sent: Friday, 26 January 2024 17:25
> To: dev@airflow.apache.org
> Subject: Re: [PROPOSAL] Adding MSGraphSDK Async Operator to Airflow
>
> [You don't often get email from jarek@potiuk.com. Learn why this is
> important at https://aka.ms/LearnAboutSenderIdentification ]
>
> EXTERNAL MAIL: Indien je de afzender van deze e-mail niet kent en deze
> niet vertrouwt, klik niet op een link of open geen bijlages. Bij twijfel,
> stuur deze e-mail als bijlage naar abuse@infrabel.be<mailto:
> abuse@infrabel.be>.
>
> Hey David - any comments on that ?
>
> On Mon, Jan 15, 2024 at 1:07 PM Elad Kalif <el...@apache.org> wrote:
>
> > Hi David,
> >
> > Thanks for raising this discussion.
> > following the protocol established about accepting new providers -
> >
> > https://gith/
> > ub.com%2Fapache%2Fairflow%2Fblob%2Fmain%2FPROVIDERS.rst%23accepting-ne
> > w-community-providers&data=05%7C02%7Cdavid.blain%40infrabel.be%7C2da54
> > 87c207f4c39debd08dc1e8b75ac%7Cb82bc314ab8e4d6fb18946f02e1f27f2%7C0%7C0
> > %7C638418831528623158%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQ
> > IjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=mrnCt
> > dP4HLfDYw1HeMfylfZkEv0p%2BMO4X3Sn6QJu8hU%3D&reserved=0
> > My main concern here is how will provide ongoing mantaince for this
> > provider?
> > This provider is to handle a service by Microsoft yet Microsoft is not
> > in the picture here (as far as I can see)
> >
> >
> > On Sat, Jan 13, 2024 at 12:39 PM Blain David <da...@infrabel.be>
> > wrote:
> >
> > > Hello everyone,
> > >
> > > I've already started a discussion about this on the Airflow
> discussions:
> > > https://gi/
> > > thub.com%2Fapache%2Fairflow%2Fdiscussions%2F36315&data=05%7C02%7Cdav
> > > id.blain%40infrabel.be%7C2da5487c207f4c39debd08dc1e8b75ac%7Cb82bc314
> > > ab8e4d6fb18946f02e1f27f2%7C0%7C0%7C638418831528632454%7CUnknown%7CTW
> > > FpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVC
> > > I6Mn0%3D%7C3000%7C%7C%7C&sdata=WLeKg%2FSCcqkEGVGpK9OvuUaQIc02GAjjglM
> > > tPkGeWWQ%3D&reserved=0
> > >
> > > As we have multiple DAG's interacting with MS Graph API endpoints,
> > > and as we want to avoid custom code as much as possible as we have
> > > to handle
> > lot's
> > > of data due to paging.
> > > We thought of implementing an operator for it using the official
> > > python client from Microsoft<
> > https://gith/
> > ub.com%2Fmicrosoftgraph%2Fmsgraph-sdk-python&data=05%7C02%7Cdavid.blai
> > n%40infrabel.be%7C2da5487c207f4c39debd08dc1e8b75ac%7Cb82bc314ab8e4d6fb
> > 18946f02e1f27f2%7C0%7C0%7C638418831528638199%7CUnknown%7CTWFpbGZsb3d8e
> > yJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C30
> > 00%7C%7C%7C&sdata=qdH4r0OJxGjYXJ8jIQn41BS%2BIH64Nj8%2BKWSsYL2iobo%3D&r
> > eserved=0>,
> > > that way we can simplify our DAGs and remove custom code as much as
> > > possible.
> > > That's why we implemented an MSGraph SDK Provider which is on
> > > GitHub<
> > > https://gi/
> > > thub.com
> %2Finfrabel%2Fapache-airflow-providers-msgraph&data=05%7C02%7Cdavid.blain%
> 40infrabel.be%7C2da5487c207f4c39debd08dc1e8b75ac%7Cb82bc314ab8e4d6fb18946f02e1f27f2%7C0%7C0%7C638418831528642865%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Gs27ZiTInN1qI3%2FVojdkFFP%2BV4OLSeGTQ6sbFBItdsA%3D&reserved=0>
> and also published it as an artifact on PyPi<
> https://pypi.org/project/apache-airflow-providers-msgraph/>.
> > >
> > > As Jarek pointed out in the pull request for Third Party providers<
> > > https://gi/
> > > thub.com%2Fapache%2Fairflow-site%2Fpull%2F933&data=05%7C02%7Cdavid.b
> > > lain%40infrabel.be%7C2da5487c207f4c39debd08dc1e8b75ac%7Cb82bc314ab8e
> > > 4d6fb18946f02e1f27f2%7C0%7C0%7C638418831528651442%7CUnknown%7CTWFpbG
> > > Zsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn
> > > 0%3D%7C3000%7C%7C%7C&sdata=UbwEODVhHelR%2FAKDtW5pMiZ169qILlL4Vqe4ZD8
> > > Mkak%3D&reserved=0>, it's not a good idea
> > to
> > > use the apache-airflow prefix for the library as it is not
> > > maintained by the Apache community (yet).
> > > So there are 2 options, or we change the name or we donate the
> > > project to the Apache Airflow community, which I think the later one
> > > is the best option if there is interest to do it, hence why I also
> > > started a
> > discussion<
> > > https://gi/
> > > thub.com%2Fapache%2Fairflow%2Fdiscussions%2F36315&data=05%7C02%7Cdav
> > > id.blain%40infrabel.be%7C2da5487c207f4c39debd08dc1e8b75ac%7Cb82bc314
> > > ab8e4d6fb18946f02e1f27f2%7C0%7C0%7C638418831528655433%7CUnknown%7CTW
> > > FpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVC
> > > I6Mn0%3D%7C3000%7C%7C%7C&sdata=IwiIleDSIkBC1IpYRYxOBtFHQAUBceLBvqZWs
> > > cKsG6s%3D&reserved=0> about it a few
> > weeks
> > > ago on GitHub<https://github.com/apache/airflow/discussions/36315>.
> > >
> > > Kind regards,
> > > David
> > >
> > >
> >
>