Testing structure of your provider code

[Bartłomiej Hirsz <ba...@gmail.com>]

Hi,
When I was working on AIP-47 (new design of system tests) I noticed that
Google has tests for coverage of operators with examples.
I revived it a bit to also work with new system tests, but I wasn't fully
satisfied. I have decided to rewrite it so it can be easily reused by other
providers. As a bonus it now has pluggable architecture. Here is PR:
https://github.com/apache/airflow/pull/23351

Any provider that wishes to test for the coverage of their examples/system
tests or test anything else in the structure of their operators can reuse
this code (create their own classes in the file and inherit from base
classes). In the case of Google I have created two kinds of tests: example
coverage and also extra assets coverage. I also did it for elasticsearch
and docker providers to show how it's possible to extend it for other
providers.

Check out the PR and let me know what you think and if there is room for
improvements :)

Regards,
Bartłomiej Hirsz

Re: Testing structure of your provider code

["Oliveira, Niko" <on...@amazon.com.INVALID>]

> I hope we can - as part of system tests improvements - add it for other providers too :)


I plan to add the tests for AWS soon :)

________________________________
From: Jarek Potiuk <ja...@potiuk.com>
Sent: Sunday, May 8, 2022 12:34 PM
To: dev@airflow.apache.org
Subject: RE: [EXTERNAL]Testing structure of your provider code


CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.


I love it. It makes it more usable for other providers too. Merged it now after some reviews.

I hope we can - as part of system tests improvements - add it for other providers too :)

On Fri, Apr 29, 2022 at 9:26 AM Bartlomiej Hirsz <ba...@gmail.com>> wrote:
Hi,
When I was working on AIP-47 (new design of system tests) I noticed that Google has tests for coverage of operators with examples.
I revived it a bit to also work with new system tests, but I wasn't fully satisfied. I have decided to rewrite it so it can be easily reused by other providers. As a bonus it now has pluggable architecture. Here is PR:
https://github.com/apache/airflow/pull/23351

Any provider that wishes to test for the coverage of their examples/system tests or test anything else in the structure of their operators can reuse this code (create their own classes in the file and inherit from base classes). In the case of Google I have created two kinds of tests: example coverage and also extra assets coverage. I also did it for elasticsearch and docker providers to show how it's possible to extend it for other providers.

Check out the PR and let me know what you think and if there is room for improvements :)

Regards,
Bartlomiej Hirsz

Re: Testing structure of your provider code

[Jarek Potiuk <ja...@potiuk.com>]

I love it. It makes it more usable for other providers too. Merged it now
after some reviews.

I hope we can - as part of system tests improvements - add it for other
providers too :)

On Fri, Apr 29, 2022 at 9:26 AM Bartłomiej Hirsz <ba...@gmail.com>
wrote:

> Hi,
> When I was working on AIP-47 (new design of system tests) I noticed that
> Google has tests for coverage of operators with examples.
> I revived it a bit to also work with new system tests, but I wasn't fully
> satisfied. I have decided to rewrite it so it can be easily reused by other
> providers. As a bonus it now has pluggable architecture. Here is PR:
> https://github.com/apache/airflow/pull/23351
>
> Any provider that wishes to test for the coverage of their examples/system
> tests or test anything else in the structure of their operators can reuse
> this code (create their own classes in the file and inherit from base
> classes). In the case of Google I have created two kinds of tests: example
> coverage and also extra assets coverage. I also did it for elasticsearch
> and docker providers to show how it's possible to extend it for other
> providers.
>
> Check out the PR and let me know what you think and if there is room for
> improvements :)
>
> Regards,
> Bartłomiej Hirsz
>