You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Rahul Vats <ra...@gmail.com> on 2025/09/23 11:03:40 UTC
[Discuss] Improving Airflow UI Testing Coverage
Hi all,
With Airflow 3.0, the UI has undergone a complete revamp, and new changes
are naturally more prone to regressions. We've already seen more UI
regressions that are hard to catch with our current limited test coverage.
Manual testing is also challenging given the wide variety of workflows and
screens in the UI.
Currently, the Airflow UI has a few unit tests using Vitest and React
Testing Library, but coverage is quite limited — only about 4 test files
covering major pages like DagsList and TaskInstance. Many critical
pages (Connections,
Variables, Pools, Dashboard, etc.) have no test coverage at all.
I’m exploring a two-pronged approach to improve UI testing:
1.
Expand unit test coverage for individual components and pages
2.
Add end-to-end (E2E) tests using Playwright for complete user workflows
This would give us both component-level safety and full user-journey
testing across browsers, helping catch regressions early.
A few questions:
-
Should we prioritize expanding unit test coverage first, or work on unit
and E2E tests in parallel?
-
For E2E testing, is Playwright a good choice, or are there other
preferred tools we should explore?
-
If Playwright, should we use TypeScript or Python as language for tests?
-
Any other suggestions to strengthen our UI testing strategy?
I’d love to hear your thoughts on whether this would be valuable for the
project before diving into implementation details.
Regards,
Rahul Vats
Re: [Discuss] Improving Airflow UI Testing Coverage
Posted by Vincent Beck <vi...@apache.org>.
I agree, developers who will write e2e tests will most likely be developers working on the front-end .. who tend to prefer typescript.
On 2025/09/24 15:43:39 Constance Martineau via dev wrote:
> Echo'ing Pierre. I don't really have an opinion on the tech stack itself,
> but feels it comes down to who we expect to own most of the tests. If it's
> mostly front-end folks, then TypeScript probably makes more sense. If it's
> mostly core Airflow engineers writing end-to-end flow tests, then Python
> would be more natural. Sounds like we're leaning toward the former?
>
> On Wed, Sep 24, 2025 at 11:31 AM Pierre Jeambrun <pi...@gmail.com>
> wrote:
>
> > I also think that having a good unit test coverage + e2e tests would really
> > help maintaining quality of releases and prevent regressions early, making
> > all that much more manageable.
> >
> > My two cents regarding the questions above:
> >
> > -
> > Should we prioritize expanding unit test coverage first, or work on unit
> > and E2E tests in parallel?
> >
> > Both in parallel sounds good to me. They do not depend on each other (At
> > least in my mind)
> > -
> > For E2E testing, is Playwright a good choice, or are there other
> > preferred tools we should explore?
> >
> > I like playwright and I feel this is the modern solution to go to.
> > -
> > If Playwright, should we use TypeScript or Python as language for tests?
> >
> > Good arguments mentioned above for both sides. I personally think (maybe
> > wrongly), that ideally people maintaining / developing e2e tests will be
> > people with
> > front-end knowledge. (events, identifiers, selectors, css styling,
> > browser local states, etc..), so I tend to think that those people will
> > most likely come from the front-end community and
> > having that in typescript would probably help them get started.
> >
> >
> > On Wed, Sep 24, 2025 at 10:01 AM Amogh Desai <am...@apache.org>
> > wrote:
> >
> > > Thanks for starting this discussion, Rahul.
> > >
> > > I am aligned with your thoughts, we have seen many many UI issues lately
> > > which I believe we should prioritize fixing or at least reducing going
> > > forward.
> > >
> > > To answer your question(s) --
> > >
> > > * I think the unit tests and e2e / user flow tests as orthogonal, so they
> > > can be
> > > developed in parallel.
> > >
> > > * While I do not have real experience with Playwright but from what I
> > hear
> > > / read on reddit,
> > > github, and related forums, it looks like Playwright can do similar job
> > as
> > > alternatives such as
> > > Selenium, Cypress, while being more beginner friendly, which could play
> > > better in our favour.
> > >
> > > * Again, not very strong on this but I feel that having Typescript as the
> > > language of choice would limit
> > > the number of contributors we get to help on this. Python would be my
> > > weapon of choice, given
> > > the "testing" is not limited by it and it potentially could encourage
> > > broader participation.
> > >
> > > (Folks active on CI/CD(atleast me), are not very comfortable with
> > > Typescript tests compared to
> > > Python)
> > >
> > > One additional point: It would be valuable if we extract the e2e / user
> > > flow testing into a GH job that
> > > a release manager could run as "sort" of last validation before releasing
> > > to avoid any last
> > > minute issues.
> > >
> > > Thanks & Regards,
> > > Amogh Desai
> > >
> > >
> > > On Tue, Sep 23, 2025 at 9:15 PM Brent Bovenzi via dev <
> > > dev@airflow.apache.org> wrote:
> > >
> > > > I agree that we need to improve UI tests. Happy to make that a priority
> > > > during 3.1.x and 3.2 development.
> > > >
> > > > Personally, I will start with expanding unit tests first. But that
> > > doesn't
> > > > mean others can't get started on E2E tests.
> > > >
> > > > I lean towards using typescript for playwright. With E2E we are testing
> > > the
> > > > UI more than we are testing the API so I think the code should sit
> > closer
> > > > to the UI code.
> > > >
> > > > On Tue, Sep 23, 2025 at 9:38 AM Vincent Beck <vi...@apache.org>
> > > wrote:
> > > >
> > > > > Not answering any of these questions (sorry) but strong +1 on adding
> > > > > end-to-end (E2E) tests. This will significantly make our UI more
> > robust
> > > > but
> > > > > also will speed up some of our work. An example is front-end
> > dependency
> > > > > upgrades by dependabot. Today we have no way of knowing whether these
> > > PRs
> > > > > break the UI, in order to merge these PRs we need to compile the
> > > > front-end
> > > > > assets and test manually whether the UI work. Having end-to-end tests
> > > on
> > > > > the front-end and making them run in the CI would speed up this very
> > > > > example (on top of increasing code robustness, increase our
> > confidence
> > > > etc).
> > > > >
> > > > > On 2025/09/23 11:03:40 Rahul Vats wrote:
> > > > > > Hi all,
> > > > > >
> > > > > > With Airflow 3.0, the UI has undergone a complete revamp, and new
> > > > changes
> > > > > > are naturally more prone to regressions. We've already seen more UI
> > > > > > regressions that are hard to catch with our current limited test
> > > > > coverage.
> > > > > > Manual testing is also challenging given the wide variety of
> > > workflows
> > > > > and
> > > > > > screens in the UI.
> > > > > >
> > > > > > Currently, the Airflow UI has a few unit tests using Vitest and
> > React
> > > > > > Testing Library, but coverage is quite limited — only about 4 test
> > > > files
> > > > > > covering major pages like DagsList and TaskInstance. Many critical
> > > > > > pages (Connections,
> > > > > > Variables, Pools, Dashboard, etc.) have no test coverage at all.
> > > > > >
> > > > > > I’m exploring a two-pronged approach to improve UI testing:
> > > > > >
> > > > > > 1.
> > > > > >
> > > > > > Expand unit test coverage for individual components and pages
> > > > > >
> > > > > > 2.
> > > > > >
> > > > > > Add end-to-end (E2E) tests using Playwright for complete user
> > > > > workflows
> > > > > >
> > > > > >
> > > > > > This would give us both component-level safety and full
> > user-journey
> > > > > > testing across browsers, helping catch regressions early.
> > > > > >
> > > > > > A few questions:
> > > > > >
> > > > > > -
> > > > > >
> > > > > > Should we prioritize expanding unit test coverage first, or work
> > > on
> > > > > unit
> > > > > > and E2E tests in parallel?
> > > > > >
> > > > > > -
> > > > > >
> > > > > > For E2E testing, is Playwright a good choice, or are there other
> > > > > > preferred tools we should explore?
> > > > > >
> > > > > > -
> > > > > >
> > > > > > If Playwright, should we use TypeScript or Python as language
> > for
> > > > > tests?
> > > > > >
> > > > > > -
> > > > > >
> > > > > > Any other suggestions to strengthen our UI testing strategy?
> > > > > >
> > > > > >
> > > > > > I’d love to hear your thoughts on whether this would be valuable
> > for
> > > > the
> > > > > > project before diving into implementation details.
> > > > > >
> > > > > > Regards,
> > > > > > Rahul Vats
> > > > > >
> > > > >
> > > > > ---------------------------------------------------------------------
> > > > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > > > For additional commands, e-mail: dev-help@airflow.apache.org
> > > > >
> > > > >
> > > >
> > >
> >
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
For additional commands, e-mail: dev-help@airflow.apache.org
Re: [Discuss] Improving Airflow UI Testing Coverage
Posted by Constance Martineau via dev <de...@airflow.apache.org>.
Echo'ing Pierre. I don't really have an opinion on the tech stack itself,
but feels it comes down to who we expect to own most of the tests. If it's
mostly front-end folks, then TypeScript probably makes more sense. If it's
mostly core Airflow engineers writing end-to-end flow tests, then Python
would be more natural. Sounds like we're leaning toward the former?
On Wed, Sep 24, 2025 at 11:31 AM Pierre Jeambrun <pi...@gmail.com>
wrote:
> I also think that having a good unit test coverage + e2e tests would really
> help maintaining quality of releases and prevent regressions early, making
> all that much more manageable.
>
> My two cents regarding the questions above:
>
> -
> Should we prioritize expanding unit test coverage first, or work on unit
> and E2E tests in parallel?
>
> Both in parallel sounds good to me. They do not depend on each other (At
> least in my mind)
> -
> For E2E testing, is Playwright a good choice, or are there other
> preferred tools we should explore?
>
> I like playwright and I feel this is the modern solution to go to.
> -
> If Playwright, should we use TypeScript or Python as language for tests?
>
> Good arguments mentioned above for both sides. I personally think (maybe
> wrongly), that ideally people maintaining / developing e2e tests will be
> people with
> front-end knowledge. (events, identifiers, selectors, css styling,
> browser local states, etc..), so I tend to think that those people will
> most likely come from the front-end community and
> having that in typescript would probably help them get started.
>
>
> On Wed, Sep 24, 2025 at 10:01 AM Amogh Desai <am...@apache.org>
> wrote:
>
> > Thanks for starting this discussion, Rahul.
> >
> > I am aligned with your thoughts, we have seen many many UI issues lately
> > which I believe we should prioritize fixing or at least reducing going
> > forward.
> >
> > To answer your question(s) --
> >
> > * I think the unit tests and e2e / user flow tests as orthogonal, so they
> > can be
> > developed in parallel.
> >
> > * While I do not have real experience with Playwright but from what I
> hear
> > / read on reddit,
> > github, and related forums, it looks like Playwright can do similar job
> as
> > alternatives such as
> > Selenium, Cypress, while being more beginner friendly, which could play
> > better in our favour.
> >
> > * Again, not very strong on this but I feel that having Typescript as the
> > language of choice would limit
> > the number of contributors we get to help on this. Python would be my
> > weapon of choice, given
> > the "testing" is not limited by it and it potentially could encourage
> > broader participation.
> >
> > (Folks active on CI/CD(atleast me), are not very comfortable with
> > Typescript tests compared to
> > Python)
> >
> > One additional point: It would be valuable if we extract the e2e / user
> > flow testing into a GH job that
> > a release manager could run as "sort" of last validation before releasing
> > to avoid any last
> > minute issues.
> >
> > Thanks & Regards,
> > Amogh Desai
> >
> >
> > On Tue, Sep 23, 2025 at 9:15 PM Brent Bovenzi via dev <
> > dev@airflow.apache.org> wrote:
> >
> > > I agree that we need to improve UI tests. Happy to make that a priority
> > > during 3.1.x and 3.2 development.
> > >
> > > Personally, I will start with expanding unit tests first. But that
> > doesn't
> > > mean others can't get started on E2E tests.
> > >
> > > I lean towards using typescript for playwright. With E2E we are testing
> > the
> > > UI more than we are testing the API so I think the code should sit
> closer
> > > to the UI code.
> > >
> > > On Tue, Sep 23, 2025 at 9:38 AM Vincent Beck <vi...@apache.org>
> > wrote:
> > >
> > > > Not answering any of these questions (sorry) but strong +1 on adding
> > > > end-to-end (E2E) tests. This will significantly make our UI more
> robust
> > > but
> > > > also will speed up some of our work. An example is front-end
> dependency
> > > > upgrades by dependabot. Today we have no way of knowing whether these
> > PRs
> > > > break the UI, in order to merge these PRs we need to compile the
> > > front-end
> > > > assets and test manually whether the UI work. Having end-to-end tests
> > on
> > > > the front-end and making them run in the CI would speed up this very
> > > > example (on top of increasing code robustness, increase our
> confidence
> > > etc).
> > > >
> > > > On 2025/09/23 11:03:40 Rahul Vats wrote:
> > > > > Hi all,
> > > > >
> > > > > With Airflow 3.0, the UI has undergone a complete revamp, and new
> > > changes
> > > > > are naturally more prone to regressions. We've already seen more UI
> > > > > regressions that are hard to catch with our current limited test
> > > > coverage.
> > > > > Manual testing is also challenging given the wide variety of
> > workflows
> > > > and
> > > > > screens in the UI.
> > > > >
> > > > > Currently, the Airflow UI has a few unit tests using Vitest and
> React
> > > > > Testing Library, but coverage is quite limited — only about 4 test
> > > files
> > > > > covering major pages like DagsList and TaskInstance. Many critical
> > > > > pages (Connections,
> > > > > Variables, Pools, Dashboard, etc.) have no test coverage at all.
> > > > >
> > > > > I’m exploring a two-pronged approach to improve UI testing:
> > > > >
> > > > > 1.
> > > > >
> > > > > Expand unit test coverage for individual components and pages
> > > > >
> > > > > 2.
> > > > >
> > > > > Add end-to-end (E2E) tests using Playwright for complete user
> > > > workflows
> > > > >
> > > > >
> > > > > This would give us both component-level safety and full
> user-journey
> > > > > testing across browsers, helping catch regressions early.
> > > > >
> > > > > A few questions:
> > > > >
> > > > > -
> > > > >
> > > > > Should we prioritize expanding unit test coverage first, or work
> > on
> > > > unit
> > > > > and E2E tests in parallel?
> > > > >
> > > > > -
> > > > >
> > > > > For E2E testing, is Playwright a good choice, or are there other
> > > > > preferred tools we should explore?
> > > > >
> > > > > -
> > > > >
> > > > > If Playwright, should we use TypeScript or Python as language
> for
> > > > tests?
> > > > >
> > > > > -
> > > > >
> > > > > Any other suggestions to strengthen our UI testing strategy?
> > > > >
> > > > >
> > > > > I’d love to hear your thoughts on whether this would be valuable
> for
> > > the
> > > > > project before diving into implementation details.
> > > > >
> > > > > Regards,
> > > > > Rahul Vats
> > > > >
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > > For additional commands, e-mail: dev-help@airflow.apache.org
> > > >
> > > >
> > >
> >
>
Re: [Discuss] Improving Airflow UI Testing Coverage
Posted by GUAN-MING CHIU <gu...@gmail.com>.
Hi,
Thanks for bringing this up, Rahul.
I think strengthening UI testing is definitely valuable and I'm also happy
to help with implementation!
On the e2e side, I support using Playwright. I’ve worked with both
Playwright, Cypress and others testing libraries before,
and Playwright feels like a better fit here, especially since it offers
*codegen* (docs <https://playwright.dev/docs/codegen>), which makes writing
and maintaining tests much faster.
For the language, I’d lean towards TypeScript, since it’s closer to the UI
code and also keep the tests familiar to contributors who already work on
the UI.
Regarding priorities, I believe unit tests and e2e tests can run in
parallel, since they serve different purposes and won’t block each other.
Unit tests can continue to expand for broader coverage, while we could
start with e2e for P0 workflows first to quickly gain confidence in
critical user journeys.
I think this approach should help reduce regressions while improving
development velocity.
Best,
Guan-Ming
Shubham Raj <sh...@apache.org> 於 2025年9月24日 週三 下午11:51寫道：
> This addition will be really nice.
> Also, at this point of 3.x, I think UI has been relatively stable and
> writing e2e would be a good starting point. Unit tests can go parallel.
> It's right that people maintaining CI/CD would be first persons to see any
> breakage, but I feel writing in ts will be easier for folks working on
> frontend, and also for the new contributors who would like to contribute in
> this UI e2e tests coming from cypress or similar experience, since majority
> of other choices do come with ts/js support.
> I would be happy to help with the e2e tests. We should probably go in
> phases with P0/1/2.
>
> Thanks.
>
>
> On Wed, Sep 24, 2025 at 9:01 PM Pierre Jeambrun <pi...@gmail.com>
> wrote:
>
> > I also think that having a good unit test coverage + e2e tests would
> really
> > help maintaining quality of releases and prevent regressions early,
> making
> > all that much more manageable.
> >
> > My two cents regarding the questions above:
> >
> > -
> > Should we prioritize expanding unit test coverage first, or work on
> unit
> > and E2E tests in parallel?
> >
> > Both in parallel sounds good to me. They do not depend on each other
> (At
> > least in my mind)
> > -
> > For E2E testing, is Playwright a good choice, or are there other
> > preferred tools we should explore?
> >
> > I like playwright and I feel this is the modern solution to go to.
> > -
> > If Playwright, should we use TypeScript or Python as language for
> tests?
> >
> > Good arguments mentioned above for both sides. I personally think
> (maybe
> > wrongly), that ideally people maintaining / developing e2e tests will be
> > people with
> > front-end knowledge. (events, identifiers, selectors, css styling,
> > browser local states, etc..), so I tend to think that those people will
> > most likely come from the front-end community and
> > having that in typescript would probably help them get started.
> >
> >
> > On Wed, Sep 24, 2025 at 10:01 AM Amogh Desai <am...@apache.org>
> > wrote:
> >
> > > Thanks for starting this discussion, Rahul.
> > >
> > > I am aligned with your thoughts, we have seen many many UI issues
> lately
> > > which I believe we should prioritize fixing or at least reducing going
> > > forward.
> > >
> > > To answer your question(s) --
> > >
> > > * I think the unit tests and e2e / user flow tests as orthogonal, so
> they
> > > can be
> > > developed in parallel.
> > >
> > > * While I do not have real experience with Playwright but from what I
> > hear
> > > / read on reddit,
> > > github, and related forums, it looks like Playwright can do similar job
> > as
> > > alternatives such as
> > > Selenium, Cypress, while being more beginner friendly, which could play
> > > better in our favour.
> > >
> > > * Again, not very strong on this but I feel that having Typescript as
> the
> > > language of choice would limit
> > > the number of contributors we get to help on this. Python would be my
> > > weapon of choice, given
> > > the "testing" is not limited by it and it potentially could encourage
> > > broader participation.
> > >
> > > (Folks active on CI/CD(atleast me), are not very comfortable with
> > > Typescript tests compared to
> > > Python)
> > >
> > > One additional point: It would be valuable if we extract the e2e / user
> > > flow testing into a GH job that
> > > a release manager could run as "sort" of last validation before
> releasing
> > > to avoid any last
> > > minute issues.
> > >
> > > Thanks & Regards,
> > > Amogh Desai
> > >
> > >
> > > On Tue, Sep 23, 2025 at 9:15 PM Brent Bovenzi via dev <
> > > dev@airflow.apache.org> wrote:
> > >
> > > > I agree that we need to improve UI tests. Happy to make that a
> priority
> > > > during 3.1.x and 3.2 development.
> > > >
> > > > Personally, I will start with expanding unit tests first. But that
> > > doesn't
> > > > mean others can't get started on E2E tests.
> > > >
> > > > I lean towards using typescript for playwright. With E2E we are
> testing
> > > the
> > > > UI more than we are testing the API so I think the code should sit
> > closer
> > > > to the UI code.
> > > >
> > > > On Tue, Sep 23, 2025 at 9:38 AM Vincent Beck <vi...@apache.org>
> > > wrote:
> > > >
> > > > > Not answering any of these questions (sorry) but strong +1 on
> adding
> > > > > end-to-end (E2E) tests. This will significantly make our UI more
> > robust
> > > > but
> > > > > also will speed up some of our work. An example is front-end
> > dependency
> > > > > upgrades by dependabot. Today we have no way of knowing whether
> these
> > > PRs
> > > > > break the UI, in order to merge these PRs we need to compile the
> > > > front-end
> > > > > assets and test manually whether the UI work. Having end-to-end
> tests
> > > on
> > > > > the front-end and making them run in the CI would speed up this
> very
> > > > > example (on top of increasing code robustness, increase our
> > confidence
> > > > etc).
> > > > >
> > > > > On 2025/09/23 11:03:40 Rahul Vats wrote:
> > > > > > Hi all,
> > > > > >
> > > > > > With Airflow 3.0, the UI has undergone a complete revamp, and new
> > > > changes
> > > > > > are naturally more prone to regressions. We've already seen more
> UI
> > > > > > regressions that are hard to catch with our current limited test
> > > > > coverage.
> > > > > > Manual testing is also challenging given the wide variety of
> > > workflows
> > > > > and
> > > > > > screens in the UI.
> > > > > >
> > > > > > Currently, the Airflow UI has a few unit tests using Vitest and
> > React
> > > > > > Testing Library, but coverage is quite limited — only about 4
> test
> > > > files
> > > > > > covering major pages like DagsList and TaskInstance. Many
> critical
> > > > > > pages (Connections,
> > > > > > Variables, Pools, Dashboard, etc.) have no test coverage at all.
> > > > > >
> > > > > > I’m exploring a two-pronged approach to improve UI testing:
> > > > > >
> > > > > > 1.
> > > > > >
> > > > > > Expand unit test coverage for individual components and pages
> > > > > >
> > > > > > 2.
> > > > > >
> > > > > > Add end-to-end (E2E) tests using Playwright for complete user
> > > > > workflows
> > > > > >
> > > > > >
> > > > > > This would give us both component-level safety and full
> > user-journey
> > > > > > testing across browsers, helping catch regressions early.
> > > > > >
> > > > > > A few questions:
> > > > > >
> > > > > > -
> > > > > >
> > > > > > Should we prioritize expanding unit test coverage first, or
> work
> > > on
> > > > > unit
> > > > > > and E2E tests in parallel?
> > > > > >
> > > > > > -
> > > > > >
> > > > > > For E2E testing, is Playwright a good choice, or are there
> other
> > > > > > preferred tools we should explore?
> > > > > >
> > > > > > -
> > > > > >
> > > > > > If Playwright, should we use TypeScript or Python as language
> > for
> > > > > tests?
> > > > > >
> > > > > > -
> > > > > >
> > > > > > Any other suggestions to strengthen our UI testing strategy?
> > > > > >
> > > > > >
> > > > > > I’d love to hear your thoughts on whether this would be valuable
> > for
> > > > the
> > > > > > project before diving into implementation details.
> > > > > >
> > > > > > Regards,
> > > > > > Rahul Vats
> > > > > >
> > > > >
> > > > >
> ---------------------------------------------------------------------
> > > > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > > > For additional commands, e-mail: dev-help@airflow.apache.org
> > > > >
> > > > >
> > > >
> > >
> >
>
Re: [Discuss] Improving Airflow UI Testing Coverage
Posted by Rahul Vats <ra...@gmail.com>.
Hi Everyone!
Thanks to all contributors who picked up sub-issues - we've made great
progress on UI E2E testing!
We have a few more sub-issues ready to be picked up:
• https://github.com/apache/airflow/issues/59929
• https://github.com/apache/airflow/issues/59930
• https://github.com/apache/airflow/issues/59931
• https://github.com/apache/airflow/issues/59934
• https://github.com/apache/airflow/issues/59357
Contribution Guide:
https://github.com/apache/airflow/blob/main/airflow-core/src/airflow/ui/tests/e2e/README.md#contributing-new-tests
Join #sig-airflow-ui-e2e-tests
<https://apache-airflow.slack.com/archives/C0A2QT7UUGP> for questions and
discussions.
Regards,
Rahul Vats
On Wed, 17 Dec 2025 at 22:09, Buğra Öztürk <oz...@gmail.com> wrote:
> Cool, it has been working for a while in scheduled CI and it is pretty
> stable. Amazing work!
>
> Bugra Ozturk
>
> On Wed, 17 Dec 2025, 06:26 Amogh Desai, <am...@apache.org> wrote:
>
> > Nice!
> >
> > Thanks & Regards,
> > Amogh Desai
> >
> >
> > On Fri, Dec 12, 2025 at 9:58 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> >
> > > Cool!
> > >
> > > On Fri, Dec 12, 2025 at 4:34 PM Vincent Beck <vi...@apache.org>
> > wrote:
> > >
> > > > Nice! Thanks for driving it! Really excited about it!
> > > >
> > > > On 2025/12/12 10:02:18 Rahul Vats wrote:
> > > > > Hi Everyone,
> > > > >
> > > > > Following up on our earlier discussions about UI E2E testing -- the
> > > > initial
> > > > > framework is now in place and ready for community contributions!
> > > > >
> > > > > *What's Been Merged:*
> > > > > - Playwright framework with TypeScript (#58548
> > > > > <https://github.com/apache/airflow/pull/58548> )
> > > > > - CI workflow integrated (#58901
> > > > > <https://github.com/apache/airflow/pull/58901>)
> > > > > - Initial test covering login and DAG triggering
> > > > >
> > > > > *Meta Issue for Tracking:*
> > > > > #59028 <https://github.com/apache/airflow/issues/59028>
> > > > >
> > > > > *Sub-Issues Ready to Pick Up*:
> > > > >
> > > > >
> > > > > - https://github.com/apache/airflow/issues/59306
> > > > > - https://github.com/apache/airflow/issues/59307
> > > > > - https://github.com/apache/airflow/issues/59308
> > > > > - https://github.com/apache/airflow/issues/59309
> > > > > - https://github.com/apache/airflow/issues/59357
> > > > > - https://github.com/apache/airflow/issues/59358
> > > > >
> > > > >
> > > > > More sub-issues will be created and linked to the meta issue #59028
> > > > > <https://github.com/apache/airflow/issues/59028>
> > > > >
> > > > > *Contribution Guide*
> > > > >
> > > >
> > >
> >
> https://github.com/apache/airflow/blob/main/airflow-core/src/airflow/ui/tests/e2e/README.md#contributing-new-tests
> > > > >
> > > > > *Slack Channel:*
> > > > > Join #sig-airflow-ui-e2e-tests
> > > > > <https://apache-airflow.slack.com/archives/C0A2QT7UUGP> for
> > questions
> > > > and
> > > > > discussions.
> > > > >
> > > > > Looking forward to contributions!
> > > > >
> > > > > Regards,
> > > > > Rahul Vats
> > > > >
> > > > > On Wed, 15 Oct 2025 at 10:45, Amogh Desai <am...@apache.org>
> > > wrote:
> > > > >
> > > > > > Thanks for the summary, Rahul.
> > > > > >
> > > > > > Sounds like a good plan, you could coordinate the work better
> > either
> > > by
> > > > > > using
> > > > > > a Github Board or Slack channel or both too.
> > > > > >
> > > > > > Thanks & Regards,
> > > > > > Amogh Desai
> > > > > >
> > > > > >
> > > > > > On Fri, Oct 10, 2025 at 9:11 PM Rahul Vats <
> rah.sharma11@gmail.com
> > >
> > > > wrote:
> > > > > >
> > > > > > > Hi Everyone,
> > > > > > >
> > > > > > > Thank you all for the fantastic feedback and strong support!
> The
> > > > > > community
> > > > > > > consensus is clear and encouraging:
> > > > > > >
> > > > > > > - High priority for improving UI testing
> > > > > > > - Playwright + TypeScript for e2e testing
> > > > > > > - Parallel development of unit and e2e tests
> > > > > > > - Test pyramid approach with focused e2e scope
> > > > > > > - Multiple volunteers ready to contribute!
> > > > > > >
> > > > > > >
> > > > > > > *Addressing Key Concerns*
> > > > > > > Important points on e2e complexity are well taken. I
> completely
> > > > agree
> > > > > > that
> > > > > > > we should follow the test pyramid.
> > > > > > >
> > > > > > > - E2E tests should cover only critical workflows (~10-15
> > scenarios)
> > > > > > > - Unit tests are the foundation and should be the primary focus
> > and
> > > > we
> > > > > > > should have an enforcement so that every UI change comes with a
> > > unit
> > > > > > test.
> > > > > > > - CI integration needs careful planning to avoid slowing
> > > development.
> > > > > > >
> > > > > > > *Next Steps*
> > > > > > >
> > > > > > > - I plan to work on a POC for E2E tests and will raise a PR for
> > > > review
> > > > > > > soon. After that, I'll share a detailed implementation plan and
> > > > > > coordinate
> > > > > > > with volunteers who offered to help.
> > > > > > > - In the meantime, I suggest we should continue expanding unit
> > test
> > > > > > > coverage as planned.
> > > > > > >
> > > > > > >
> > > > > > > It was great meeting you all at the Airflow Summit. Momentum
> here
> > > is
> > > > > > > fantastic, and I'm excited to work with everyone who offered to
> > > help
> > > > make
> > > > > > > Airflow 3 UI much more reliable!
> > > > > > >
> > > > > > > Regards,
> > > > > > > Rahul Vats
> > > > > > >
> > > > > > > On Thu, 25 Sept 2025 at 07:49, Vikram Koka via dev <
> > > > > > dev@airflow.apache.org
> > > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Rahul,
> > > > > > > > Thank you for starting this thread.
> > > > > > > >
> > > > > > > > Strong +1 from me as well.
> > > > > > > > I not only agree with this, but I believe this needs to be a
> > high
> > > > > > > priority
> > > > > > > > going forward and before 3.2.
> > > > > > > >
> > > > > > > > I also agree with the comments on the "test pyramid"
> approach,
> > of
> > > > there
> > > > > > > > needing to be FAR more unit tests as compared to end to end
> > > tests.
> > > > > > > > I also believe that a certain level of rigor around this i.e.
> > > some
> > > > form
> > > > > > > of
> > > > > > > > enforcement to ensure that all new UI changes come with a
> unit
> > > > test.
> > > > > > > >
> > > > > > > > I would also add the Core API changes (for UI) to the mix
> here,
> > > > since a
> > > > > > > > fair number of UI changes requires underlying API changes.
> > > > > > > >
> > > > > > > > Vikram
> > > > > > > >
> > > > > > > >
> > > > > > > > On Thu, Sep 25, 2025 at 12:04 AM Amogh Desai <
> > > > amoghdesai@apache.org>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Valid points.
> > > > > > > > >
> > > > > > > > > I was just expressing concern from a CI/CD failure
> > perspective
> > > > but in
> > > > > > > no
> > > > > > > > > way
> > > > > > > > > was I objecting to usage of typescript.
> > > > > > > > >
> > > > > > > > > Anything that works for the team that owns most of the
> tests
> > > > works by
> > > > > > > me!
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Thanks & Regards,
> > > > > > > > > Amogh Desai
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Thu, Sep 25, 2025 at 1:39 AM Jens Scheffler <
> > > > jscheffl@apache.org>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Echoing a full agreement to all statement before.
> > > > > > > > > >
> > > > > > > > > > I also would emphasize the test pyramid. As we really
> have
> > a
> > > > low
> > > > > > > > > > coverage we should maybe also have a metric. I would also
> > > > propose
> > > > > > > that
> > > > > > > > > > we focus more on testing coverage == hopefully also
> > improved
> > > > > > quality
> > > > > > > > > > preventing regression in 3.2. One option could a bit of
> > force
> > > > to
> > > > > > > force
> > > > > > > > > > all new changes applied to UI to request a unit test
> being
> > > > added
> > > > > > (all
> > > > > > > > > > changes TS/x files need to have a corresponding
> .test.ts/x
> > > > file)
> > > > > > > except
> > > > > > > > > > for cases where it does not make sense we allow exclusion
> > of
> > > > the
> > > > > > rule
> > > > > > > > by
> > > > > > > > > > adding a label by maintainers. (just want to drop the
> idea,
> > > > might
> > > > > > be
> > > > > > > > > > many will hate it)
> > > > > > > > > >
> > > > > > > > > > For language also have no strong preference but even
> > though I
> > > > am a
> > > > > > TS
> > > > > > > > > > noob I see that testing UI is closer to UI development,
> to
> > > > using
> > > > > > > Python
> > > > > > > > > > for TS UI testing might make a knowledge gap also for
> > > potential
> > > > > > > utility
> > > > > > > > > > code. So as most voices would propose TS (maybe then
> > finally
> > > I
> > > > get
> > > > > > to
> > > > > > > > > > understand it)
> > > > > > > > > >
> > > > > > > > > > Jens
> > > > > > > > > >
> > > > > > > > > > On 24.09.25 18:33, Jarek Potiuk wrote:
> > > > > > > > > > > Small poking at important things from holidays while
> > > having a
> > > > > > > coffee
> > > > > > > > > at a
> > > > > > > > > > > place with internet :)
> > > > > > > > > > >
> > > > > > > > > > > +1 on having both. I have no strong opinion on the
> > language
> > > > and
> > > > > > > > > > Playwright.
> > > > > > > > > > >
> > > > > > > > > > > But there are few things for consideration:
> > > > > > > > > > >
> > > > > > > > > > > We should have many, many more unit tests than e2e
> tests.
> > > > This is
> > > > > > > the
> > > > > > > > > > > classic test pyramid case: The Practical Test Pyramid
> > > > > > > > > > > https://share.google/YHVIt8EBptT1yYFur
> > > > > > > > > > >
> > > > > > > > > > > The e2e tests are slow, brittle and difficult to
> maintain
> > > > and we
> > > > > > > > should
> > > > > > > > > > > have rather small number of those - mostly green path
> on
> > > all
> > > > > > major
> > > > > > > > > > > functionalities. Possibly with some cross browser
> checks
> > -
> > > > but
> > > > > > > those
> > > > > > > > > are
> > > > > > > > > > > very difficult to get right.
> > > > > > > > > > >
> > > > > > > > > > > It will be a pain to maintain those. Like our k8s e2e
> > > tests -
> > > > > > when
> > > > > > > > they
> > > > > > > > > > > break (rarely) people will have hard time to create and
> > run
> > > > the
> > > > > > > tests
> > > > > > > > > so
> > > > > > > > > > > the tooling around that has to drive people who will be
> > > > > > > reproducing
> > > > > > > > > the
> > > > > > > > > > > failures by hand - how to set it up and how to
> reproduce
> > it
> > > > and
> > > > > > how
> > > > > > > > to
> > > > > > > > > > > iterate as fast as possible.
> > > > > > > > > > >
> > > > > > > > > > > Keeping the tests stable and side-effect free will be a
> > > > > > challenge -
> > > > > > > > so
> > > > > > > > > > > likely a good 'reset the environment' before each test
> > is a
> > > > must.
> > > > > > > > Also
> > > > > > > > > > they
> > > > > > > > > > > need to be run with all UI changes in CI which will
> make
> > UI
> > > > PR
> > > > > > run
> > > > > > > > > > longer.
> > > > > > > > > > > It's an imperative to get them run not only in canary
> > > builds
> > > > but
> > > > > > in
> > > > > > > > all
> > > > > > > > > > PRs
> > > > > > > > > > > for ui and API and likely whenever any core model
> change.
> > > It
> > > > will
> > > > > > > > slow
> > > > > > > > > > down
> > > > > > > > > > > UI development and create frustration for contributors
> > > > unfamiliar
> > > > > > > > with
> > > > > > > > > > the
> > > > > > > > > > > test framework so we need to have at least a few people
> > > > > > proficient
> > > > > > > in
> > > > > > > > > > > intricacies of the test framework dedicated and willing
> > to
> > > > help
> > > > > > > those
> > > > > > > > > who
> > > > > > > > > > > will struggle with issues.
> > > > > > > > > > >
> > > > > > > > > > > Language is secondary - e2e tests will require all
> stack
> > -
> > > > Python
> > > > > > > > > > (backend)
> > > > > > > > > > > and Typescript and react experiences anyway. Most
> likely
> > > > people
> > > > > > who
> > > > > > > > > write
> > > > > > > > > > > UI should be involved so Typescript is fine, and also
> > even
> > > > those
> > > > > > > who
> > > > > > > > do
> > > > > > > > > > not
> > > > > > > > > > > know TypeScript will be able to fix some obvious cases.
> > > > > > > > > > >
> > > > > > > > > > > J.
> > > > > > > > > > >
> > > > > > > > > > > śr., 24 wrz 2025, 09:50 użytkownik Shubham Raj <
> > > > > > > > shubhamraj@apache.org>
> > > > > > > > > > > napisał:
> > > > > > > > > > >
> > > > > > > > > > >> This addition will be really nice.
> > > > > > > > > > >> Also, at this point of 3.x, I think UI has been
> > relatively
> > > > > > stable
> > > > > > > > and
> > > > > > > > > > >> writing e2e would be a good starting point. Unit tests
> > can
> > > > go
> > > > > > > > > parallel.
> > > > > > > > > > >> It's right that people maintaining CI/CD would be
> first
> > > > persons
> > > > > > to
> > > > > > > > see
> > > > > > > > > > any
> > > > > > > > > > >> breakage, but I feel writing in ts will be easier for
> > > folks
> > > > > > > working
> > > > > > > > on
> > > > > > > > > > >> frontend, and also for the new contributors who would
> > like
> > > > to
> > > > > > > > > > contribute in
> > > > > > > > > > >> this UI e2e tests coming from cypress or similar
> > > experience,
> > > > > > since
> > > > > > > > > > majority
> > > > > > > > > > >> of other choices do come with ts/js support.
> > > > > > > > > > >> I would be happy to help with the e2e tests. We should
> > > > probably
> > > > > > go
> > > > > > > > in
> > > > > > > > > > >> phases with P0/1/2.
> > > > > > > > > > >>
> > > > > > > > > > >> Thanks.
> > > > > > > > > > >>
> > > > > > > > > > >>
> > > > > > > > > > >> On Wed, Sep 24, 2025 at 9:01 PM Pierre Jeambrun <
> > > > > > > > > pierrejbrun@gmail.com>
> > > > > > > > > > >> wrote:
> > > > > > > > > > >>
> > > > > > > > > > >>> I also think that having a good unit test coverage +
> > e2e
> > > > tests
> > > > > > > > would
> > > > > > > > > > >> really
> > > > > > > > > > >>> help maintaining quality of releases and prevent
> > > > regressions
> > > > > > > early,
> > > > > > > > > > >> making
> > > > > > > > > > >>> all that much more manageable.
> > > > > > > > > > >>>
> > > > > > > > > > >>> My two cents regarding the questions above:
> > > > > > > > > > >>>
> > > > > > > > > > >>> -
> > > > > > > > > > >>> Should we prioritize expanding unit test coverage
> > > > first, or
> > > > > > > > work
> > > > > > > > > on
> > > > > > > > > > >> unit
> > > > > > > > > > >>> and E2E tests in parallel?
> > > > > > > > > > >>>
> > > > > > > > > > >>> Both in parallel sounds good to me. They do not
> > > depend
> > > > on
> > > > > > > each
> > > > > > > > > > other
> > > > > > > > > > >> (At
> > > > > > > > > > >>> least in my mind)
> > > > > > > > > > >>> -
> > > > > > > > > > >>> For E2E testing, is Playwright a good choice, or
> > are
> > > > there
> > > > > > > > other
> > > > > > > > > > >>> preferred tools we should explore?
> > > > > > > > > > >>>
> > > > > > > > > > >>> I like playwright and I feel this is the modern
> > > > solution to
> > > > > > > go
> > > > > > > > > to.
> > > > > > > > > > >>> -
> > > > > > > > > > >>> If Playwright, should we use TypeScript or Python
> > as
> > > > > > language
> > > > > > > > for
> > > > > > > > > > >> tests?
> > > > > > > > > > >>> Good arguments mentioned above for both sides. I
> > > > personally
> > > > > > > > think
> > > > > > > > > > >> (maybe
> > > > > > > > > > >>> wrongly), that ideally people maintaining /
> developing
> > > e2e
> > > > > > tests
> > > > > > > > will
> > > > > > > > > > be
> > > > > > > > > > >>> people with
> > > > > > > > > > >>> front-end knowledge. (events, identifiers,
> > selectors,
> > > > css
> > > > > > > > > styling,
> > > > > > > > > > >>> browser local states, etc..), so I tend to think
> that
> > > > those
> > > > > > > people
> > > > > > > > > > will
> > > > > > > > > > >>> most likely come from the front-end community and
> > > > > > > > > > >>> having that in typescript would probably help
> them
> > > get
> > > > > > > started.
> > > > > > > > > > >>>
> > > > > > > > > > >>>
> > > > > > > > > > >>> On Wed, Sep 24, 2025 at 10:01 AM Amogh Desai <
> > > > > > > > amoghdesai@apache.org>
> > > > > > > > > > >>> wrote:
> > > > > > > > > > >>>
> > > > > > > > > > >>>> Thanks for starting this discussion, Rahul.
> > > > > > > > > > >>>>
> > > > > > > > > > >>>> I am aligned with your thoughts, we have seen many
> > many
> > > UI
> > > > > > > issues
> > > > > > > > > > >> lately
> > > > > > > > > > >>>> which I believe we should prioritize fixing or at
> > least
> > > > > > reducing
> > > > > > > > > going
> > > > > > > > > > >>>> forward.
> > > > > > > > > > >>>>
> > > > > > > > > > >>>> To answer your question(s) --
> > > > > > > > > > >>>>
> > > > > > > > > > >>>> * I think the unit tests and e2e / user flow tests
> as
> > > > > > > orthogonal,
> > > > > > > > so
> > > > > > > > > > >> they
> > > > > > > > > > >>>> can be
> > > > > > > > > > >>>> developed in parallel.
> > > > > > > > > > >>>>
> > > > > > > > > > >>>> * While I do not have real experience with
> Playwright
> > > but
> > > > from
> > > > > > > > what
> > > > > > > > > I
> > > > > > > > > > >>> hear
> > > > > > > > > > >>>> / read on reddit,
> > > > > > > > > > >>>> github, and related forums, it looks like Playwright
> > can
> > > > do
> > > > > > > > similar
> > > > > > > > > > job
> > > > > > > > > > >>> as
> > > > > > > > > > >>>> alternatives such as
> > > > > > > > > > >>>> Selenium, Cypress, while being more beginner
> friendly,
> > > > which
> > > > > > > could
> > > > > > > > > > play
> > > > > > > > > > >>>> better in our favour.
> > > > > > > > > > >>>>
> > > > > > > > > > >>>> * Again, not very strong on this but I feel that
> > having
> > > > > > > Typescript
> > > > > > > > > as
> > > > > > > > > > >> the
> > > > > > > > > > >>>> language of choice would limit
> > > > > > > > > > >>>> the number of contributors we get to help on this.
> > > Python
> > > > > > would
> > > > > > > be
> > > > > > > > > my
> > > > > > > > > > >>>> weapon of choice, given
> > > > > > > > > > >>>> the "testing" is not limited by it and it
> potentially
> > > > could
> > > > > > > > > encourage
> > > > > > > > > > >>>> broader participation.
> > > > > > > > > > >>>>
> > > > > > > > > > >>>> (Folks active on CI/CD(atleast me), are not very
> > > > comfortable
> > > > > > > with
> > > > > > > > > > >>>> Typescript tests compared to
> > > > > > > > > > >>>> Python)
> > > > > > > > > > >>>>
> > > > > > > > > > >>>> One additional point: It would be valuable if we
> > extract
> > > > the
> > > > > > > e2e /
> > > > > > > > > > user
> > > > > > > > > > >>>> flow testing into a GH job that
> > > > > > > > > > >>>> a release manager could run as "sort" of last
> > validation
> > > > > > before
> > > > > > > > > > >> releasing
> > > > > > > > > > >>>> to avoid any last
> > > > > > > > > > >>>> minute issues.
> > > > > > > > > > >>>>
> > > > > > > > > > >>>> Thanks & Regards,
> > > > > > > > > > >>>> Amogh Desai
> > > > > > > > > > >>>>
> > > > > > > > > > >>>>
> > > > > > > > > > >>>> On Tue, Sep 23, 2025 at 9:15 PM Brent Bovenzi via
> dev
> > <
> > > > > > > > > > >>>> dev@airflow.apache.org> wrote:
> > > > > > > > > > >>>>
> > > > > > > > > > >>>>> I agree that we need to improve UI tests. Happy to
> > make
> > > > that
> > > > > > a
> > > > > > > > > > >> priority
> > > > > > > > > > >>>>> during 3.1.x and 3.2 development.
> > > > > > > > > > >>>>>
> > > > > > > > > > >>>>> Personally, I will start with expanding unit tests
> > > > first. But
> > > > > > > > that
> > > > > > > > > > >>>> doesn't
> > > > > > > > > > >>>>> mean others can't get started on E2E tests.
> > > > > > > > > > >>>>>
> > > > > > > > > > >>>>> I lean towards using typescript for playwright.
> With
> > > E2E
> > > > we
> > > > > > are
> > > > > > > > > > >> testing
> > > > > > > > > > >>>> the
> > > > > > > > > > >>>>> UI more than we are testing the API so I think the
> > code
> > > > > > should
> > > > > > > > sit
> > > > > > > > > > >>> closer
> > > > > > > > > > >>>>> to the UI code.
> > > > > > > > > > >>>>>
> > > > > > > > > > >>>>> On Tue, Sep 23, 2025 at 9:38 AM Vincent Beck <
> > > > > > > > vincbeck@apache.org>
> > > > > > > > > > >>>> wrote:
> > > > > > > > > > >>>>>> Not answering any of these questions (sorry) but
> > > strong
> > > > +1
> > > > > > on
> > > > > > > > > > >> adding
> > > > > > > > > > >>>>>> end-to-end (E2E) tests. This will significantly
> make
> > > > our UI
> > > > > > > more
> > > > > > > > > > >>> robust
> > > > > > > > > > >>>>> but
> > > > > > > > > > >>>>>> also will speed up some of our work. An example is
> > > > front-end
> > > > > > > > > > >>> dependency
> > > > > > > > > > >>>>>> upgrades by dependabot. Today we have no way of
> > > knowing
> > > > > > > whether
> > > > > > > > > > >> these
> > > > > > > > > > >>>> PRs
> > > > > > > > > > >>>>>> break the UI, in order to merge these PRs we need
> to
> > > > compile
> > > > > > > the
> > > > > > > > > > >>>>> front-end
> > > > > > > > > > >>>>>> assets and test manually whether the UI work.
> Having
> > > > > > > end-to-end
> > > > > > > > > > >> tests
> > > > > > > > > > >>>> on
> > > > > > > > > > >>>>>> the front-end and making them run in the CI would
> > > speed
> > > > up
> > > > > > > this
> > > > > > > > > > >> very
> > > > > > > > > > >>>>>> example (on top of increasing code robustness,
> > > increase
> > > > our
> > > > > > > > > > >>> confidence
> > > > > > > > > > >>>>> etc).
> > > > > > > > > > >>>>>> On 2025/09/23 11:03:40 Rahul Vats wrote:
> > > > > > > > > > >>>>>>> Hi all,
> > > > > > > > > > >>>>>>>
> > > > > > > > > > >>>>>>> With Airflow 3.0, the UI has undergone a complete
> > > > revamp,
> > > > > > and
> > > > > > > > new
> > > > > > > > > > >>>>> changes
> > > > > > > > > > >>>>>>> are naturally more prone to regressions. We've
> > > already
> > > > seen
> > > > > > > > more
> > > > > > > > > > >> UI
> > > > > > > > > > >>>>>>> regressions that are hard to catch with our
> current
> > > > limited
> > > > > > > > test
> > > > > > > > > > >>>>>> coverage.
> > > > > > > > > > >>>>>>> Manual testing is also challenging given the wide
> > > > variety
> > > > > > of
> > > > > > > > > > >>>> workflows
> > > > > > > > > > >>>>>> and
> > > > > > > > > > >>>>>>> screens in the UI.
> > > > > > > > > > >>>>>>>
> > > > > > > > > > >>>>>>> Currently, the Airflow UI has a few unit tests
> > using
> > > > Vitest
> > > > > > > and
> > > > > > > > > > >>> React
> > > > > > > > > > >>>>>>> Testing Library, but coverage is quite limited —
> > only
> > > > > > about 4
> > > > > > > > > > >> test
> > > > > > > > > > >>>>> files
> > > > > > > > > > >>>>>>> covering major pages like DagsList and
> > TaskInstance.
> > > > Many
> > > > > > > > > > >> critical
> > > > > > > > > > >>>>>>> pages (Connections,
> > > > > > > > > > >>>>>>> Variables, Pools, Dashboard, etc.) have no test
> > > > coverage at
> > > > > > > > all.
> > > > > > > > > > >>>>>>>
> > > > > > > > > > >>>>>>> I’m exploring a two-pronged approach to improve
> UI
> > > > testing:
> > > > > > > > > > >>>>>>>
> > > > > > > > > > >>>>>>> 1.
> > > > > > > > > > >>>>>>>
> > > > > > > > > > >>>>>>> Expand unit test coverage for individual
> > > > components and
> > > > > > > > pages
> > > > > > > > > > >>>>>>>
> > > > > > > > > > >>>>>>> 2.
> > > > > > > > > > >>>>>>>
> > > > > > > > > > >>>>>>> Add end-to-end (E2E) tests using Playwright
> for
> > > > > > complete
> > > > > > > > user
> > > > > > > > > > >>>>>> workflows
> > > > > > > > > > >>>>>>>
> > > > > > > > > > >>>>>>> This would give us both component-level safety
> and
> > > full
> > > > > > > > > > >>> user-journey
> > > > > > > > > > >>>>>>> testing across browsers, helping catch
> regressions
> > > > early.
> > > > > > > > > > >>>>>>>
> > > > > > > > > > >>>>>>> A few questions:
> > > > > > > > > > >>>>>>>
> > > > > > > > > > >>>>>>> -
> > > > > > > > > > >>>>>>>
> > > > > > > > > > >>>>>>> Should we prioritize expanding unit test
> > coverage
> > > > > > first,
> > > > > > > or
> > > > > > > > > > >> work
> > > > > > > > > > >>>> on
> > > > > > > > > > >>>>>> unit
> > > > > > > > > > >>>>>>> and E2E tests in parallel?
> > > > > > > > > > >>>>>>>
> > > > > > > > > > >>>>>>> -
> > > > > > > > > > >>>>>>>
> > > > > > > > > > >>>>>>> For E2E testing, is Playwright a good choice,
> > or
> > > > are
> > > > > > > there
> > > > > > > > > > >> other
> > > > > > > > > > >>>>>>> preferred tools we should explore?
> > > > > > > > > > >>>>>>>
> > > > > > > > > > >>>>>>> -
> > > > > > > > > > >>>>>>>
> > > > > > > > > > >>>>>>> If Playwright, should we use TypeScript or
> > Python
> > > > as
> > > > > > > > language
> > > > > > > > > > >>> for
> > > > > > > > > > >>>>>> tests?
> > > > > > > > > > >>>>>>> -
> > > > > > > > > > >>>>>>>
> > > > > > > > > > >>>>>>> Any other suggestions to strengthen our UI
> > > testing
> > > > > > > > strategy?
> > > > > > > > > > >>>>>>>
> > > > > > > > > > >>>>>>>
> > > > > > > > > > >>>>>>> I’d love to hear your thoughts on whether this
> > would
> > > be
> > > > > > > > valuable
> > > > > > > > > > >>> for
> > > > > > > > > > >>>>> the
> > > > > > > > > > >>>>>>> project before diving into implementation
> details.
> > > > > > > > > > >>>>>>>
> > > > > > > > > > >>>>>>> Regards,
> > > > > > > > > > >>>>>>> Rahul Vats
> > > > > > > > > > >>>>>>>
> > > > > > > > > > >>>>>>
> > > > > > > > > > >>
> > > > > > > >
> > > > ---------------------------------------------------------------------
> > > > > > > > > > >>>>>> To unsubscribe, e-mail:
> > > > dev-unsubscribe@airflow.apache.org
> > > > > > > > > > >>>>>> For additional commands, e-mail:
> > > > > > dev-help@airflow.apache.org
> > > > > > > > > > >>>>>>
> > > > > > > > > > >>>>>>
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > > >
> > > > > >
> > ---------------------------------------------------------------------
> > > > > > > > > > To unsubscribe, e-mail:
> dev-unsubscribe@airflow.apache.org
> > > > > > > > > > For additional commands, e-mail:
> > dev-help@airflow.apache.org
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > > For additional commands, e-mail: dev-help@airflow.apache.org
> > > >
> > > >
> > >
> >
>
Re: [Discuss] Improving Airflow UI Testing Coverage
Posted by Buğra Öztürk <oz...@gmail.com>.
Cool, it has been working for a while in scheduled CI and it is pretty
stable. Amazing work!
Bugra Ozturk
On Wed, 17 Dec 2025, 06:26 Amogh Desai, <am...@apache.org> wrote:
> Nice!
>
> Thanks & Regards,
> Amogh Desai
>
>
> On Fri, Dec 12, 2025 at 9:58 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>
> > Cool!
> >
> > On Fri, Dec 12, 2025 at 4:34 PM Vincent Beck <vi...@apache.org>
> wrote:
> >
> > > Nice! Thanks for driving it! Really excited about it!
> > >
> > > On 2025/12/12 10:02:18 Rahul Vats wrote:
> > > > Hi Everyone,
> > > >
> > > > Following up on our earlier discussions about UI E2E testing -- the
> > > initial
> > > > framework is now in place and ready for community contributions!
> > > >
> > > > *What's Been Merged:*
> > > > - Playwright framework with TypeScript (#58548
> > > > <https://github.com/apache/airflow/pull/58548> )
> > > > - CI workflow integrated (#58901
> > > > <https://github.com/apache/airflow/pull/58901>)
> > > > - Initial test covering login and DAG triggering
> > > >
> > > > *Meta Issue for Tracking:*
> > > > #59028 <https://github.com/apache/airflow/issues/59028>
> > > >
> > > > *Sub-Issues Ready to Pick Up*:
> > > >
> > > >
> > > > - https://github.com/apache/airflow/issues/59306
> > > > - https://github.com/apache/airflow/issues/59307
> > > > - https://github.com/apache/airflow/issues/59308
> > > > - https://github.com/apache/airflow/issues/59309
> > > > - https://github.com/apache/airflow/issues/59357
> > > > - https://github.com/apache/airflow/issues/59358
> > > >
> > > >
> > > > More sub-issues will be created and linked to the meta issue #59028
> > > > <https://github.com/apache/airflow/issues/59028>
> > > >
> > > > *Contribution Guide*
> > > >
> > >
> >
> https://github.com/apache/airflow/blob/main/airflow-core/src/airflow/ui/tests/e2e/README.md#contributing-new-tests
> > > >
> > > > *Slack Channel:*
> > > > Join #sig-airflow-ui-e2e-tests
> > > > <https://apache-airflow.slack.com/archives/C0A2QT7UUGP> for
> questions
> > > and
> > > > discussions.
> > > >
> > > > Looking forward to contributions!
> > > >
> > > > Regards,
> > > > Rahul Vats
> > > >
> > > > On Wed, 15 Oct 2025 at 10:45, Amogh Desai <am...@apache.org>
> > wrote:
> > > >
> > > > > Thanks for the summary, Rahul.
> > > > >
> > > > > Sounds like a good plan, you could coordinate the work better
> either
> > by
> > > > > using
> > > > > a Github Board or Slack channel or both too.
> > > > >
> > > > > Thanks & Regards,
> > > > > Amogh Desai
> > > > >
> > > > >
> > > > > On Fri, Oct 10, 2025 at 9:11 PM Rahul Vats <rah.sharma11@gmail.com
> >
> > > wrote:
> > > > >
> > > > > > Hi Everyone,
> > > > > >
> > > > > > Thank you all for the fantastic feedback and strong support! The
> > > > > community
> > > > > > consensus is clear and encouraging:
> > > > > >
> > > > > > - High priority for improving UI testing
> > > > > > - Playwright + TypeScript for e2e testing
> > > > > > - Parallel development of unit and e2e tests
> > > > > > - Test pyramid approach with focused e2e scope
> > > > > > - Multiple volunteers ready to contribute!
> > > > > >
> > > > > >
> > > > > > *Addressing Key Concerns*
> > > > > > Important points on e2e complexity are well taken. I completely
> > > agree
> > > > > that
> > > > > > we should follow the test pyramid.
> > > > > >
> > > > > > - E2E tests should cover only critical workflows (~10-15
> scenarios)
> > > > > > - Unit tests are the foundation and should be the primary focus
> and
> > > we
> > > > > > should have an enforcement so that every UI change comes with a
> > unit
> > > > > test.
> > > > > > - CI integration needs careful planning to avoid slowing
> > development.
> > > > > >
> > > > > > *Next Steps*
> > > > > >
> > > > > > - I plan to work on a POC for E2E tests and will raise a PR for
> > > review
> > > > > > soon. After that, I'll share a detailed implementation plan and
> > > > > coordinate
> > > > > > with volunteers who offered to help.
> > > > > > - In the meantime, I suggest we should continue expanding unit
> test
> > > > > > coverage as planned.
> > > > > >
> > > > > >
> > > > > > It was great meeting you all at the Airflow Summit. Momentum here
> > is
> > > > > > fantastic, and I'm excited to work with everyone who offered to
> > help
> > > make
> > > > > > Airflow 3 UI much more reliable!
> > > > > >
> > > > > > Regards,
> > > > > > Rahul Vats
> > > > > >
> > > > > > On Thu, 25 Sept 2025 at 07:49, Vikram Koka via dev <
> > > > > dev@airflow.apache.org
> > > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Rahul,
> > > > > > > Thank you for starting this thread.
> > > > > > >
> > > > > > > Strong +1 from me as well.
> > > > > > > I not only agree with this, but I believe this needs to be a
> high
> > > > > > priority
> > > > > > > going forward and before 3.2.
> > > > > > >
> > > > > > > I also agree with the comments on the "test pyramid" approach,
> of
> > > there
> > > > > > > needing to be FAR more unit tests as compared to end to end
> > tests.
> > > > > > > I also believe that a certain level of rigor around this i.e.
> > some
> > > form
> > > > > > of
> > > > > > > enforcement to ensure that all new UI changes come with a unit
> > > test.
> > > > > > >
> > > > > > > I would also add the Core API changes (for UI) to the mix here,
> > > since a
> > > > > > > fair number of UI changes requires underlying API changes.
> > > > > > >
> > > > > > > Vikram
> > > > > > >
> > > > > > >
> > > > > > > On Thu, Sep 25, 2025 at 12:04 AM Amogh Desai <
> > > amoghdesai@apache.org>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Valid points.
> > > > > > > >
> > > > > > > > I was just expressing concern from a CI/CD failure
> perspective
> > > but in
> > > > > > no
> > > > > > > > way
> > > > > > > > was I objecting to usage of typescript.
> > > > > > > >
> > > > > > > > Anything that works for the team that owns most of the tests
> > > works by
> > > > > > me!
> > > > > > > >
> > > > > > > >
> > > > > > > > Thanks & Regards,
> > > > > > > > Amogh Desai
> > > > > > > >
> > > > > > > >
> > > > > > > > On Thu, Sep 25, 2025 at 1:39 AM Jens Scheffler <
> > > jscheffl@apache.org>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Echoing a full agreement to all statement before.
> > > > > > > > >
> > > > > > > > > I also would emphasize the test pyramid. As we really have
> a
> > > low
> > > > > > > > > coverage we should maybe also have a metric. I would also
> > > propose
> > > > > > that
> > > > > > > > > we focus more on testing coverage == hopefully also
> improved
> > > > > quality
> > > > > > > > > preventing regression in 3.2. One option could a bit of
> force
> > > to
> > > > > > force
> > > > > > > > > all new changes applied to UI to request a unit test being
> > > added
> > > > > (all
> > > > > > > > > changes TS/x files need to have a corresponding .test.ts/x
> > > file)
> > > > > > except
> > > > > > > > > for cases where it does not make sense we allow exclusion
> of
> > > the
> > > > > rule
> > > > > > > by
> > > > > > > > > adding a label by maintainers. (just want to drop the idea,
> > > might
> > > > > be
> > > > > > > > > many will hate it)
> > > > > > > > >
> > > > > > > > > For language also have no strong preference but even
> though I
> > > am a
> > > > > TS
> > > > > > > > > noob I see that testing UI is closer to UI development, to
> > > using
> > > > > > Python
> > > > > > > > > for TS UI testing might make a knowledge gap also for
> > potential
> > > > > > utility
> > > > > > > > > code. So as most voices would propose TS (maybe then
> finally
> > I
> > > get
> > > > > to
> > > > > > > > > understand it)
> > > > > > > > >
> > > > > > > > > Jens
> > > > > > > > >
> > > > > > > > > On 24.09.25 18:33, Jarek Potiuk wrote:
> > > > > > > > > > Small poking at important things from holidays while
> > having a
> > > > > > coffee
> > > > > > > > at a
> > > > > > > > > > place with internet :)
> > > > > > > > > >
> > > > > > > > > > +1 on having both. I have no strong opinion on the
> language
> > > and
> > > > > > > > > Playwright.
> > > > > > > > > >
> > > > > > > > > > But there are few things for consideration:
> > > > > > > > > >
> > > > > > > > > > We should have many, many more unit tests than e2e tests.
> > > This is
> > > > > > the
> > > > > > > > > > classic test pyramid case: The Practical Test Pyramid
> > > > > > > > > > https://share.google/YHVIt8EBptT1yYFur
> > > > > > > > > >
> > > > > > > > > > The e2e tests are slow, brittle and difficult to maintain
> > > and we
> > > > > > > should
> > > > > > > > > > have rather small number of those - mostly green path on
> > all
> > > > > major
> > > > > > > > > > functionalities. Possibly with some cross browser checks
> -
> > > but
> > > > > > those
> > > > > > > > are
> > > > > > > > > > very difficult to get right.
> > > > > > > > > >
> > > > > > > > > > It will be a pain to maintain those. Like our k8s e2e
> > tests -
> > > > > when
> > > > > > > they
> > > > > > > > > > break (rarely) people will have hard time to create and
> run
> > > the
> > > > > > tests
> > > > > > > > so
> > > > > > > > > > the tooling around that has to drive people who will be
> > > > > > reproducing
> > > > > > > > the
> > > > > > > > > > failures by hand - how to set it up and how to reproduce
> it
> > > and
> > > > > how
> > > > > > > to
> > > > > > > > > > iterate as fast as possible.
> > > > > > > > > >
> > > > > > > > > > Keeping the tests stable and side-effect free will be a
> > > > > challenge -
> > > > > > > so
> > > > > > > > > > likely a good 'reset the environment' before each test
> is a
> > > must.
> > > > > > > Also
> > > > > > > > > they
> > > > > > > > > > need to be run with all UI changes in CI which will make
> UI
> > > PR
> > > > > run
> > > > > > > > > longer.
> > > > > > > > > > It's an imperative to get them run not only in canary
> > builds
> > > but
> > > > > in
> > > > > > > all
> > > > > > > > > PRs
> > > > > > > > > > for ui and API and likely whenever any core model change.
> > It
> > > will
> > > > > > > slow
> > > > > > > > > down
> > > > > > > > > > UI development and create frustration for contributors
> > > unfamiliar
> > > > > > > with
> > > > > > > > > the
> > > > > > > > > > test framework so we need to have at least a few people
> > > > > proficient
> > > > > > in
> > > > > > > > > > intricacies of the test framework dedicated and willing
> to
> > > help
> > > > > > those
> > > > > > > > who
> > > > > > > > > > will struggle with issues.
> > > > > > > > > >
> > > > > > > > > > Language is secondary - e2e tests will require all stack
> -
> > > Python
> > > > > > > > > (backend)
> > > > > > > > > > and Typescript and react experiences anyway. Most likely
> > > people
> > > > > who
> > > > > > > > write
> > > > > > > > > > UI should be involved so Typescript is fine, and also
> even
> > > those
> > > > > > who
> > > > > > > do
> > > > > > > > > not
> > > > > > > > > > know TypeScript will be able to fix some obvious cases.
> > > > > > > > > >
> > > > > > > > > > J.
> > > > > > > > > >
> > > > > > > > > > śr., 24 wrz 2025, 09:50 użytkownik Shubham Raj <
> > > > > > > shubhamraj@apache.org>
> > > > > > > > > > napisał:
> > > > > > > > > >
> > > > > > > > > >> This addition will be really nice.
> > > > > > > > > >> Also, at this point of 3.x, I think UI has been
> relatively
> > > > > stable
> > > > > > > and
> > > > > > > > > >> writing e2e would be a good starting point. Unit tests
> can
> > > go
> > > > > > > > parallel.
> > > > > > > > > >> It's right that people maintaining CI/CD would be first
> > > persons
> > > > > to
> > > > > > > see
> > > > > > > > > any
> > > > > > > > > >> breakage, but I feel writing in ts will be easier for
> > folks
> > > > > > working
> > > > > > > on
> > > > > > > > > >> frontend, and also for the new contributors who would
> like
> > > to
> > > > > > > > > contribute in
> > > > > > > > > >> this UI e2e tests coming from cypress or similar
> > experience,
> > > > > since
> > > > > > > > > majority
> > > > > > > > > >> of other choices do come with ts/js support.
> > > > > > > > > >> I would be happy to help with the e2e tests. We should
> > > probably
> > > > > go
> > > > > > > in
> > > > > > > > > >> phases with P0/1/2.
> > > > > > > > > >>
> > > > > > > > > >> Thanks.
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > > >> On Wed, Sep 24, 2025 at 9:01 PM Pierre Jeambrun <
> > > > > > > > pierrejbrun@gmail.com>
> > > > > > > > > >> wrote:
> > > > > > > > > >>
> > > > > > > > > >>> I also think that having a good unit test coverage +
> e2e
> > > tests
> > > > > > > would
> > > > > > > > > >> really
> > > > > > > > > >>> help maintaining quality of releases and prevent
> > > regressions
> > > > > > early,
> > > > > > > > > >> making
> > > > > > > > > >>> all that much more manageable.
> > > > > > > > > >>>
> > > > > > > > > >>> My two cents regarding the questions above:
> > > > > > > > > >>>
> > > > > > > > > >>> -
> > > > > > > > > >>> Should we prioritize expanding unit test coverage
> > > first, or
> > > > > > > work
> > > > > > > > on
> > > > > > > > > >> unit
> > > > > > > > > >>> and E2E tests in parallel?
> > > > > > > > > >>>
> > > > > > > > > >>> Both in parallel sounds good to me. They do not
> > depend
> > > on
> > > > > > each
> > > > > > > > > other
> > > > > > > > > >> (At
> > > > > > > > > >>> least in my mind)
> > > > > > > > > >>> -
> > > > > > > > > >>> For E2E testing, is Playwright a good choice, or
> are
> > > there
> > > > > > > other
> > > > > > > > > >>> preferred tools we should explore?
> > > > > > > > > >>>
> > > > > > > > > >>> I like playwright and I feel this is the modern
> > > solution to
> > > > > > go
> > > > > > > > to.
> > > > > > > > > >>> -
> > > > > > > > > >>> If Playwright, should we use TypeScript or Python
> as
> > > > > language
> > > > > > > for
> > > > > > > > > >> tests?
> > > > > > > > > >>> Good arguments mentioned above for both sides. I
> > > personally
> > > > > > > think
> > > > > > > > > >> (maybe
> > > > > > > > > >>> wrongly), that ideally people maintaining / developing
> > e2e
> > > > > tests
> > > > > > > will
> > > > > > > > > be
> > > > > > > > > >>> people with
> > > > > > > > > >>> front-end knowledge. (events, identifiers,
> selectors,
> > > css
> > > > > > > > styling,
> > > > > > > > > >>> browser local states, etc..), so I tend to think that
> > > those
> > > > > > people
> > > > > > > > > will
> > > > > > > > > >>> most likely come from the front-end community and
> > > > > > > > > >>> having that in typescript would probably help them
> > get
> > > > > > started.
> > > > > > > > > >>>
> > > > > > > > > >>>
> > > > > > > > > >>> On Wed, Sep 24, 2025 at 10:01 AM Amogh Desai <
> > > > > > > amoghdesai@apache.org>
> > > > > > > > > >>> wrote:
> > > > > > > > > >>>
> > > > > > > > > >>>> Thanks for starting this discussion, Rahul.
> > > > > > > > > >>>>
> > > > > > > > > >>>> I am aligned with your thoughts, we have seen many
> many
> > UI
> > > > > > issues
> > > > > > > > > >> lately
> > > > > > > > > >>>> which I believe we should prioritize fixing or at
> least
> > > > > reducing
> > > > > > > > going
> > > > > > > > > >>>> forward.
> > > > > > > > > >>>>
> > > > > > > > > >>>> To answer your question(s) --
> > > > > > > > > >>>>
> > > > > > > > > >>>> * I think the unit tests and e2e / user flow tests as
> > > > > > orthogonal,
> > > > > > > so
> > > > > > > > > >> they
> > > > > > > > > >>>> can be
> > > > > > > > > >>>> developed in parallel.
> > > > > > > > > >>>>
> > > > > > > > > >>>> * While I do not have real experience with Playwright
> > but
> > > from
> > > > > > > what
> > > > > > > > I
> > > > > > > > > >>> hear
> > > > > > > > > >>>> / read on reddit,
> > > > > > > > > >>>> github, and related forums, it looks like Playwright
> can
> > > do
> > > > > > > similar
> > > > > > > > > job
> > > > > > > > > >>> as
> > > > > > > > > >>>> alternatives such as
> > > > > > > > > >>>> Selenium, Cypress, while being more beginner friendly,
> > > which
> > > > > > could
> > > > > > > > > play
> > > > > > > > > >>>> better in our favour.
> > > > > > > > > >>>>
> > > > > > > > > >>>> * Again, not very strong on this but I feel that
> having
> > > > > > Typescript
> > > > > > > > as
> > > > > > > > > >> the
> > > > > > > > > >>>> language of choice would limit
> > > > > > > > > >>>> the number of contributors we get to help on this.
> > Python
> > > > > would
> > > > > > be
> > > > > > > > my
> > > > > > > > > >>>> weapon of choice, given
> > > > > > > > > >>>> the "testing" is not limited by it and it potentially
> > > could
> > > > > > > > encourage
> > > > > > > > > >>>> broader participation.
> > > > > > > > > >>>>
> > > > > > > > > >>>> (Folks active on CI/CD(atleast me), are not very
> > > comfortable
> > > > > > with
> > > > > > > > > >>>> Typescript tests compared to
> > > > > > > > > >>>> Python)
> > > > > > > > > >>>>
> > > > > > > > > >>>> One additional point: It would be valuable if we
> extract
> > > the
> > > > > > e2e /
> > > > > > > > > user
> > > > > > > > > >>>> flow testing into a GH job that
> > > > > > > > > >>>> a release manager could run as "sort" of last
> validation
> > > > > before
> > > > > > > > > >> releasing
> > > > > > > > > >>>> to avoid any last
> > > > > > > > > >>>> minute issues.
> > > > > > > > > >>>>
> > > > > > > > > >>>> Thanks & Regards,
> > > > > > > > > >>>> Amogh Desai
> > > > > > > > > >>>>
> > > > > > > > > >>>>
> > > > > > > > > >>>> On Tue, Sep 23, 2025 at 9:15 PM Brent Bovenzi via dev
> <
> > > > > > > > > >>>> dev@airflow.apache.org> wrote:
> > > > > > > > > >>>>
> > > > > > > > > >>>>> I agree that we need to improve UI tests. Happy to
> make
> > > that
> > > > > a
> > > > > > > > > >> priority
> > > > > > > > > >>>>> during 3.1.x and 3.2 development.
> > > > > > > > > >>>>>
> > > > > > > > > >>>>> Personally, I will start with expanding unit tests
> > > first. But
> > > > > > > that
> > > > > > > > > >>>> doesn't
> > > > > > > > > >>>>> mean others can't get started on E2E tests.
> > > > > > > > > >>>>>
> > > > > > > > > >>>>> I lean towards using typescript for playwright. With
> > E2E
> > > we
> > > > > are
> > > > > > > > > >> testing
> > > > > > > > > >>>> the
> > > > > > > > > >>>>> UI more than we are testing the API so I think the
> code
> > > > > should
> > > > > > > sit
> > > > > > > > > >>> closer
> > > > > > > > > >>>>> to the UI code.
> > > > > > > > > >>>>>
> > > > > > > > > >>>>> On Tue, Sep 23, 2025 at 9:38 AM Vincent Beck <
> > > > > > > vincbeck@apache.org>
> > > > > > > > > >>>> wrote:
> > > > > > > > > >>>>>> Not answering any of these questions (sorry) but
> > strong
> > > +1
> > > > > on
> > > > > > > > > >> adding
> > > > > > > > > >>>>>> end-to-end (E2E) tests. This will significantly make
> > > our UI
> > > > > > more
> > > > > > > > > >>> robust
> > > > > > > > > >>>>> but
> > > > > > > > > >>>>>> also will speed up some of our work. An example is
> > > front-end
> > > > > > > > > >>> dependency
> > > > > > > > > >>>>>> upgrades by dependabot. Today we have no way of
> > knowing
> > > > > > whether
> > > > > > > > > >> these
> > > > > > > > > >>>> PRs
> > > > > > > > > >>>>>> break the UI, in order to merge these PRs we need to
> > > compile
> > > > > > the
> > > > > > > > > >>>>> front-end
> > > > > > > > > >>>>>> assets and test manually whether the UI work. Having
> > > > > > end-to-end
> > > > > > > > > >> tests
> > > > > > > > > >>>> on
> > > > > > > > > >>>>>> the front-end and making them run in the CI would
> > speed
> > > up
> > > > > > this
> > > > > > > > > >> very
> > > > > > > > > >>>>>> example (on top of increasing code robustness,
> > increase
> > > our
> > > > > > > > > >>> confidence
> > > > > > > > > >>>>> etc).
> > > > > > > > > >>>>>> On 2025/09/23 11:03:40 Rahul Vats wrote:
> > > > > > > > > >>>>>>> Hi all,
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> With Airflow 3.0, the UI has undergone a complete
> > > revamp,
> > > > > and
> > > > > > > new
> > > > > > > > > >>>>> changes
> > > > > > > > > >>>>>>> are naturally more prone to regressions. We've
> > already
> > > seen
> > > > > > > more
> > > > > > > > > >> UI
> > > > > > > > > >>>>>>> regressions that are hard to catch with our current
> > > limited
> > > > > > > test
> > > > > > > > > >>>>>> coverage.
> > > > > > > > > >>>>>>> Manual testing is also challenging given the wide
> > > variety
> > > > > of
> > > > > > > > > >>>> workflows
> > > > > > > > > >>>>>> and
> > > > > > > > > >>>>>>> screens in the UI.
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> Currently, the Airflow UI has a few unit tests
> using
> > > Vitest
> > > > > > and
> > > > > > > > > >>> React
> > > > > > > > > >>>>>>> Testing Library, but coverage is quite limited —
> only
> > > > > about 4
> > > > > > > > > >> test
> > > > > > > > > >>>>> files
> > > > > > > > > >>>>>>> covering major pages like DagsList and
> TaskInstance.
> > > Many
> > > > > > > > > >> critical
> > > > > > > > > >>>>>>> pages (Connections,
> > > > > > > > > >>>>>>> Variables, Pools, Dashboard, etc.) have no test
> > > coverage at
> > > > > > > all.
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> I’m exploring a two-pronged approach to improve UI
> > > testing:
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> 1.
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> Expand unit test coverage for individual
> > > components and
> > > > > > > pages
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> 2.
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> Add end-to-end (E2E) tests using Playwright for
> > > > > complete
> > > > > > > user
> > > > > > > > > >>>>>> workflows
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> This would give us both component-level safety and
> > full
> > > > > > > > > >>> user-journey
> > > > > > > > > >>>>>>> testing across browsers, helping catch regressions
> > > early.
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> A few questions:
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> -
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> Should we prioritize expanding unit test
> coverage
> > > > > first,
> > > > > > or
> > > > > > > > > >> work
> > > > > > > > > >>>> on
> > > > > > > > > >>>>>> unit
> > > > > > > > > >>>>>>> and E2E tests in parallel?
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> -
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> For E2E testing, is Playwright a good choice,
> or
> > > are
> > > > > > there
> > > > > > > > > >> other
> > > > > > > > > >>>>>>> preferred tools we should explore?
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> -
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> If Playwright, should we use TypeScript or
> Python
> > > as
> > > > > > > language
> > > > > > > > > >>> for
> > > > > > > > > >>>>>> tests?
> > > > > > > > > >>>>>>> -
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> Any other suggestions to strengthen our UI
> > testing
> > > > > > > strategy?
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> I’d love to hear your thoughts on whether this
> would
> > be
> > > > > > > valuable
> > > > > > > > > >>> for
> > > > > > > > > >>>>> the
> > > > > > > > > >>>>>>> project before diving into implementation details.
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> Regards,
> > > > > > > > > >>>>>>> Rahul Vats
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>
> > > > > > > > > >>
> > > > > > >
> > > ---------------------------------------------------------------------
> > > > > > > > > >>>>>> To unsubscribe, e-mail:
> > > dev-unsubscribe@airflow.apache.org
> > > > > > > > > >>>>>> For additional commands, e-mail:
> > > > > dev-help@airflow.apache.org
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>>
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > > >
> > > > >
> ---------------------------------------------------------------------
> > > > > > > > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > > > > > > > For additional commands, e-mail:
> dev-help@airflow.apache.org
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > For additional commands, e-mail: dev-help@airflow.apache.org
> > >
> > >
> >
>
Re: [Discuss] Improving Airflow UI Testing Coverage
Posted by Amogh Desai <am...@apache.org>.
Nice!
Thanks & Regards,
Amogh Desai
On Fri, Dec 12, 2025 at 9:58 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> Cool!
>
> On Fri, Dec 12, 2025 at 4:34 PM Vincent Beck <vi...@apache.org> wrote:
>
> > Nice! Thanks for driving it! Really excited about it!
> >
> > On 2025/12/12 10:02:18 Rahul Vats wrote:
> > > Hi Everyone,
> > >
> > > Following up on our earlier discussions about UI E2E testing -- the
> > initial
> > > framework is now in place and ready for community contributions!
> > >
> > > *What's Been Merged:*
> > > - Playwright framework with TypeScript (#58548
> > > <https://github.com/apache/airflow/pull/58548> )
> > > - CI workflow integrated (#58901
> > > <https://github.com/apache/airflow/pull/58901>)
> > > - Initial test covering login and DAG triggering
> > >
> > > *Meta Issue for Tracking:*
> > > #59028 <https://github.com/apache/airflow/issues/59028>
> > >
> > > *Sub-Issues Ready to Pick Up*:
> > >
> > >
> > > - https://github.com/apache/airflow/issues/59306
> > > - https://github.com/apache/airflow/issues/59307
> > > - https://github.com/apache/airflow/issues/59308
> > > - https://github.com/apache/airflow/issues/59309
> > > - https://github.com/apache/airflow/issues/59357
> > > - https://github.com/apache/airflow/issues/59358
> > >
> > >
> > > More sub-issues will be created and linked to the meta issue #59028
> > > <https://github.com/apache/airflow/issues/59028>
> > >
> > > *Contribution Guide*
> > >
> >
> https://github.com/apache/airflow/blob/main/airflow-core/src/airflow/ui/tests/e2e/README.md#contributing-new-tests
> > >
> > > *Slack Channel:*
> > > Join #sig-airflow-ui-e2e-tests
> > > <https://apache-airflow.slack.com/archives/C0A2QT7UUGP> for questions
> > and
> > > discussions.
> > >
> > > Looking forward to contributions!
> > >
> > > Regards,
> > > Rahul Vats
> > >
> > > On Wed, 15 Oct 2025 at 10:45, Amogh Desai <am...@apache.org>
> wrote:
> > >
> > > > Thanks for the summary, Rahul.
> > > >
> > > > Sounds like a good plan, you could coordinate the work better either
> by
> > > > using
> > > > a Github Board or Slack channel or both too.
> > > >
> > > > Thanks & Regards,
> > > > Amogh Desai
> > > >
> > > >
> > > > On Fri, Oct 10, 2025 at 9:11 PM Rahul Vats <ra...@gmail.com>
> > wrote:
> > > >
> > > > > Hi Everyone,
> > > > >
> > > > > Thank you all for the fantastic feedback and strong support! The
> > > > community
> > > > > consensus is clear and encouraging:
> > > > >
> > > > > - High priority for improving UI testing
> > > > > - Playwright + TypeScript for e2e testing
> > > > > - Parallel development of unit and e2e tests
> > > > > - Test pyramid approach with focused e2e scope
> > > > > - Multiple volunteers ready to contribute!
> > > > >
> > > > >
> > > > > *Addressing Key Concerns*
> > > > > Important points on e2e complexity are well taken. I completely
> > agree
> > > > that
> > > > > we should follow the test pyramid.
> > > > >
> > > > > - E2E tests should cover only critical workflows (~10-15 scenarios)
> > > > > - Unit tests are the foundation and should be the primary focus and
> > we
> > > > > should have an enforcement so that every UI change comes with a
> unit
> > > > test.
> > > > > - CI integration needs careful planning to avoid slowing
> development.
> > > > >
> > > > > *Next Steps*
> > > > >
> > > > > - I plan to work on a POC for E2E tests and will raise a PR for
> > review
> > > > > soon. After that, I'll share a detailed implementation plan and
> > > > coordinate
> > > > > with volunteers who offered to help.
> > > > > - In the meantime, I suggest we should continue expanding unit test
> > > > > coverage as planned.
> > > > >
> > > > >
> > > > > It was great meeting you all at the Airflow Summit. Momentum here
> is
> > > > > fantastic, and I'm excited to work with everyone who offered to
> help
> > make
> > > > > Airflow 3 UI much more reliable!
> > > > >
> > > > > Regards,
> > > > > Rahul Vats
> > > > >
> > > > > On Thu, 25 Sept 2025 at 07:49, Vikram Koka via dev <
> > > > dev@airflow.apache.org
> > > > > >
> > > > > wrote:
> > > > >
> > > > > > Rahul,
> > > > > > Thank you for starting this thread.
> > > > > >
> > > > > > Strong +1 from me as well.
> > > > > > I not only agree with this, but I believe this needs to be a high
> > > > > priority
> > > > > > going forward and before 3.2.
> > > > > >
> > > > > > I also agree with the comments on the "test pyramid" approach, of
> > there
> > > > > > needing to be FAR more unit tests as compared to end to end
> tests.
> > > > > > I also believe that a certain level of rigor around this i.e.
> some
> > form
> > > > > of
> > > > > > enforcement to ensure that all new UI changes come with a unit
> > test.
> > > > > >
> > > > > > I would also add the Core API changes (for UI) to the mix here,
> > since a
> > > > > > fair number of UI changes requires underlying API changes.
> > > > > >
> > > > > > Vikram
> > > > > >
> > > > > >
> > > > > > On Thu, Sep 25, 2025 at 12:04 AM Amogh Desai <
> > amoghdesai@apache.org>
> > > > > > wrote:
> > > > > >
> > > > > > > Valid points.
> > > > > > >
> > > > > > > I was just expressing concern from a CI/CD failure perspective
> > but in
> > > > > no
> > > > > > > way
> > > > > > > was I objecting to usage of typescript.
> > > > > > >
> > > > > > > Anything that works for the team that owns most of the tests
> > works by
> > > > > me!
> > > > > > >
> > > > > > >
> > > > > > > Thanks & Regards,
> > > > > > > Amogh Desai
> > > > > > >
> > > > > > >
> > > > > > > On Thu, Sep 25, 2025 at 1:39 AM Jens Scheffler <
> > jscheffl@apache.org>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Echoing a full agreement to all statement before.
> > > > > > > >
> > > > > > > > I also would emphasize the test pyramid. As we really have a
> > low
> > > > > > > > coverage we should maybe also have a metric. I would also
> > propose
> > > > > that
> > > > > > > > we focus more on testing coverage == hopefully also improved
> > > > quality
> > > > > > > > preventing regression in 3.2. One option could a bit of force
> > to
> > > > > force
> > > > > > > > all new changes applied to UI to request a unit test being
> > added
> > > > (all
> > > > > > > > changes TS/x files need to have a corresponding .test.ts/x
> > file)
> > > > > except
> > > > > > > > for cases where it does not make sense we allow exclusion of
> > the
> > > > rule
> > > > > > by
> > > > > > > > adding a label by maintainers. (just want to drop the idea,
> > might
> > > > be
> > > > > > > > many will hate it)
> > > > > > > >
> > > > > > > > For language also have no strong preference but even though I
> > am a
> > > > TS
> > > > > > > > noob I see that testing UI is closer to UI development, to
> > using
> > > > > Python
> > > > > > > > for TS UI testing might make a knowledge gap also for
> potential
> > > > > utility
> > > > > > > > code. So as most voices would propose TS (maybe then finally
> I
> > get
> > > > to
> > > > > > > > understand it)
> > > > > > > >
> > > > > > > > Jens
> > > > > > > >
> > > > > > > > On 24.09.25 18:33, Jarek Potiuk wrote:
> > > > > > > > > Small poking at important things from holidays while
> having a
> > > > > coffee
> > > > > > > at a
> > > > > > > > > place with internet :)
> > > > > > > > >
> > > > > > > > > +1 on having both. I have no strong opinion on the language
> > and
> > > > > > > > Playwright.
> > > > > > > > >
> > > > > > > > > But there are few things for consideration:
> > > > > > > > >
> > > > > > > > > We should have many, many more unit tests than e2e tests.
> > This is
> > > > > the
> > > > > > > > > classic test pyramid case: The Practical Test Pyramid
> > > > > > > > > https://share.google/YHVIt8EBptT1yYFur
> > > > > > > > >
> > > > > > > > > The e2e tests are slow, brittle and difficult to maintain
> > and we
> > > > > > should
> > > > > > > > > have rather small number of those - mostly green path on
> all
> > > > major
> > > > > > > > > functionalities. Possibly with some cross browser checks -
> > but
> > > > > those
> > > > > > > are
> > > > > > > > > very difficult to get right.
> > > > > > > > >
> > > > > > > > > It will be a pain to maintain those. Like our k8s e2e
> tests -
> > > > when
> > > > > > they
> > > > > > > > > break (rarely) people will have hard time to create and run
> > the
> > > > > tests
> > > > > > > so
> > > > > > > > > the tooling around that has to drive people who will be
> > > > > reproducing
> > > > > > > the
> > > > > > > > > failures by hand - how to set it up and how to reproduce it
> > and
> > > > how
> > > > > > to
> > > > > > > > > iterate as fast as possible.
> > > > > > > > >
> > > > > > > > > Keeping the tests stable and side-effect free will be a
> > > > challenge -
> > > > > > so
> > > > > > > > > likely a good 'reset the environment' before each test is a
> > must.
> > > > > > Also
> > > > > > > > they
> > > > > > > > > need to be run with all UI changes in CI which will make UI
> > PR
> > > > run
> > > > > > > > longer.
> > > > > > > > > It's an imperative to get them run not only in canary
> builds
> > but
> > > > in
> > > > > > all
> > > > > > > > PRs
> > > > > > > > > for ui and API and likely whenever any core model change.
> It
> > will
> > > > > > slow
> > > > > > > > down
> > > > > > > > > UI development and create frustration for contributors
> > unfamiliar
> > > > > > with
> > > > > > > > the
> > > > > > > > > test framework so we need to have at least a few people
> > > > proficient
> > > > > in
> > > > > > > > > intricacies of the test framework dedicated and willing to
> > help
> > > > > those
> > > > > > > who
> > > > > > > > > will struggle with issues.
> > > > > > > > >
> > > > > > > > > Language is secondary - e2e tests will require all stack -
> > Python
> > > > > > > > (backend)
> > > > > > > > > and Typescript and react experiences anyway. Most likely
> > people
> > > > who
> > > > > > > write
> > > > > > > > > UI should be involved so Typescript is fine, and also even
> > those
> > > > > who
> > > > > > do
> > > > > > > > not
> > > > > > > > > know TypeScript will be able to fix some obvious cases.
> > > > > > > > >
> > > > > > > > > J.
> > > > > > > > >
> > > > > > > > > śr., 24 wrz 2025, 09:50 użytkownik Shubham Raj <
> > > > > > shubhamraj@apache.org>
> > > > > > > > > napisał:
> > > > > > > > >
> > > > > > > > >> This addition will be really nice.
> > > > > > > > >> Also, at this point of 3.x, I think UI has been relatively
> > > > stable
> > > > > > and
> > > > > > > > >> writing e2e would be a good starting point. Unit tests can
> > go
> > > > > > > parallel.
> > > > > > > > >> It's right that people maintaining CI/CD would be first
> > persons
> > > > to
> > > > > > see
> > > > > > > > any
> > > > > > > > >> breakage, but I feel writing in ts will be easier for
> folks
> > > > > working
> > > > > > on
> > > > > > > > >> frontend, and also for the new contributors who would like
> > to
> > > > > > > > contribute in
> > > > > > > > >> this UI e2e tests coming from cypress or similar
> experience,
> > > > since
> > > > > > > > majority
> > > > > > > > >> of other choices do come with ts/js support.
> > > > > > > > >> I would be happy to help with the e2e tests. We should
> > probably
> > > > go
> > > > > > in
> > > > > > > > >> phases with P0/1/2.
> > > > > > > > >>
> > > > > > > > >> Thanks.
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >> On Wed, Sep 24, 2025 at 9:01 PM Pierre Jeambrun <
> > > > > > > pierrejbrun@gmail.com>
> > > > > > > > >> wrote:
> > > > > > > > >>
> > > > > > > > >>> I also think that having a good unit test coverage + e2e
> > tests
> > > > > > would
> > > > > > > > >> really
> > > > > > > > >>> help maintaining quality of releases and prevent
> > regressions
> > > > > early,
> > > > > > > > >> making
> > > > > > > > >>> all that much more manageable.
> > > > > > > > >>>
> > > > > > > > >>> My two cents regarding the questions above:
> > > > > > > > >>>
> > > > > > > > >>> -
> > > > > > > > >>> Should we prioritize expanding unit test coverage
> > first, or
> > > > > > work
> > > > > > > on
> > > > > > > > >> unit
> > > > > > > > >>> and E2E tests in parallel?
> > > > > > > > >>>
> > > > > > > > >>> Both in parallel sounds good to me. They do not
> depend
> > on
> > > > > each
> > > > > > > > other
> > > > > > > > >> (At
> > > > > > > > >>> least in my mind)
> > > > > > > > >>> -
> > > > > > > > >>> For E2E testing, is Playwright a good choice, or are
> > there
> > > > > > other
> > > > > > > > >>> preferred tools we should explore?
> > > > > > > > >>>
> > > > > > > > >>> I like playwright and I feel this is the modern
> > solution to
> > > > > go
> > > > > > > to.
> > > > > > > > >>> -
> > > > > > > > >>> If Playwright, should we use TypeScript or Python as
> > > > language
> > > > > > for
> > > > > > > > >> tests?
> > > > > > > > >>> Good arguments mentioned above for both sides. I
> > personally
> > > > > > think
> > > > > > > > >> (maybe
> > > > > > > > >>> wrongly), that ideally people maintaining / developing
> e2e
> > > > tests
> > > > > > will
> > > > > > > > be
> > > > > > > > >>> people with
> > > > > > > > >>> front-end knowledge. (events, identifiers, selectors,
> > css
> > > > > > > styling,
> > > > > > > > >>> browser local states, etc..), so I tend to think that
> > those
> > > > > people
> > > > > > > > will
> > > > > > > > >>> most likely come from the front-end community and
> > > > > > > > >>> having that in typescript would probably help them
> get
> > > > > started.
> > > > > > > > >>>
> > > > > > > > >>>
> > > > > > > > >>> On Wed, Sep 24, 2025 at 10:01 AM Amogh Desai <
> > > > > > amoghdesai@apache.org>
> > > > > > > > >>> wrote:
> > > > > > > > >>>
> > > > > > > > >>>> Thanks for starting this discussion, Rahul.
> > > > > > > > >>>>
> > > > > > > > >>>> I am aligned with your thoughts, we have seen many many
> UI
> > > > > issues
> > > > > > > > >> lately
> > > > > > > > >>>> which I believe we should prioritize fixing or at least
> > > > reducing
> > > > > > > going
> > > > > > > > >>>> forward.
> > > > > > > > >>>>
> > > > > > > > >>>> To answer your question(s) --
> > > > > > > > >>>>
> > > > > > > > >>>> * I think the unit tests and e2e / user flow tests as
> > > > > orthogonal,
> > > > > > so
> > > > > > > > >> they
> > > > > > > > >>>> can be
> > > > > > > > >>>> developed in parallel.
> > > > > > > > >>>>
> > > > > > > > >>>> * While I do not have real experience with Playwright
> but
> > from
> > > > > > what
> > > > > > > I
> > > > > > > > >>> hear
> > > > > > > > >>>> / read on reddit,
> > > > > > > > >>>> github, and related forums, it looks like Playwright can
> > do
> > > > > > similar
> > > > > > > > job
> > > > > > > > >>> as
> > > > > > > > >>>> alternatives such as
> > > > > > > > >>>> Selenium, Cypress, while being more beginner friendly,
> > which
> > > > > could
> > > > > > > > play
> > > > > > > > >>>> better in our favour.
> > > > > > > > >>>>
> > > > > > > > >>>> * Again, not very strong on this but I feel that having
> > > > > Typescript
> > > > > > > as
> > > > > > > > >> the
> > > > > > > > >>>> language of choice would limit
> > > > > > > > >>>> the number of contributors we get to help on this.
> Python
> > > > would
> > > > > be
> > > > > > > my
> > > > > > > > >>>> weapon of choice, given
> > > > > > > > >>>> the "testing" is not limited by it and it potentially
> > could
> > > > > > > encourage
> > > > > > > > >>>> broader participation.
> > > > > > > > >>>>
> > > > > > > > >>>> (Folks active on CI/CD(atleast me), are not very
> > comfortable
> > > > > with
> > > > > > > > >>>> Typescript tests compared to
> > > > > > > > >>>> Python)
> > > > > > > > >>>>
> > > > > > > > >>>> One additional point: It would be valuable if we extract
> > the
> > > > > e2e /
> > > > > > > > user
> > > > > > > > >>>> flow testing into a GH job that
> > > > > > > > >>>> a release manager could run as "sort" of last validation
> > > > before
> > > > > > > > >> releasing
> > > > > > > > >>>> to avoid any last
> > > > > > > > >>>> minute issues.
> > > > > > > > >>>>
> > > > > > > > >>>> Thanks & Regards,
> > > > > > > > >>>> Amogh Desai
> > > > > > > > >>>>
> > > > > > > > >>>>
> > > > > > > > >>>> On Tue, Sep 23, 2025 at 9:15 PM Brent Bovenzi via dev <
> > > > > > > > >>>> dev@airflow.apache.org> wrote:
> > > > > > > > >>>>
> > > > > > > > >>>>> I agree that we need to improve UI tests. Happy to make
> > that
> > > > a
> > > > > > > > >> priority
> > > > > > > > >>>>> during 3.1.x and 3.2 development.
> > > > > > > > >>>>>
> > > > > > > > >>>>> Personally, I will start with expanding unit tests
> > first. But
> > > > > > that
> > > > > > > > >>>> doesn't
> > > > > > > > >>>>> mean others can't get started on E2E tests.
> > > > > > > > >>>>>
> > > > > > > > >>>>> I lean towards using typescript for playwright. With
> E2E
> > we
> > > > are
> > > > > > > > >> testing
> > > > > > > > >>>> the
> > > > > > > > >>>>> UI more than we are testing the API so I think the code
> > > > should
> > > > > > sit
> > > > > > > > >>> closer
> > > > > > > > >>>>> to the UI code.
> > > > > > > > >>>>>
> > > > > > > > >>>>> On Tue, Sep 23, 2025 at 9:38 AM Vincent Beck <
> > > > > > vincbeck@apache.org>
> > > > > > > > >>>> wrote:
> > > > > > > > >>>>>> Not answering any of these questions (sorry) but
> strong
> > +1
> > > > on
> > > > > > > > >> adding
> > > > > > > > >>>>>> end-to-end (E2E) tests. This will significantly make
> > our UI
> > > > > more
> > > > > > > > >>> robust
> > > > > > > > >>>>> but
> > > > > > > > >>>>>> also will speed up some of our work. An example is
> > front-end
> > > > > > > > >>> dependency
> > > > > > > > >>>>>> upgrades by dependabot. Today we have no way of
> knowing
> > > > > whether
> > > > > > > > >> these
> > > > > > > > >>>> PRs
> > > > > > > > >>>>>> break the UI, in order to merge these PRs we need to
> > compile
> > > > > the
> > > > > > > > >>>>> front-end
> > > > > > > > >>>>>> assets and test manually whether the UI work. Having
> > > > > end-to-end
> > > > > > > > >> tests
> > > > > > > > >>>> on
> > > > > > > > >>>>>> the front-end and making them run in the CI would
> speed
> > up
> > > > > this
> > > > > > > > >> very
> > > > > > > > >>>>>> example (on top of increasing code robustness,
> increase
> > our
> > > > > > > > >>> confidence
> > > > > > > > >>>>> etc).
> > > > > > > > >>>>>> On 2025/09/23 11:03:40 Rahul Vats wrote:
> > > > > > > > >>>>>>> Hi all,
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>> With Airflow 3.0, the UI has undergone a complete
> > revamp,
> > > > and
> > > > > > new
> > > > > > > > >>>>> changes
> > > > > > > > >>>>>>> are naturally more prone to regressions. We've
> already
> > seen
> > > > > > more
> > > > > > > > >> UI
> > > > > > > > >>>>>>> regressions that are hard to catch with our current
> > limited
> > > > > > test
> > > > > > > > >>>>>> coverage.
> > > > > > > > >>>>>>> Manual testing is also challenging given the wide
> > variety
> > > > of
> > > > > > > > >>>> workflows
> > > > > > > > >>>>>> and
> > > > > > > > >>>>>>> screens in the UI.
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>> Currently, the Airflow UI has a few unit tests using
> > Vitest
> > > > > and
> > > > > > > > >>> React
> > > > > > > > >>>>>>> Testing Library, but coverage is quite limited — only
> > > > about 4
> > > > > > > > >> test
> > > > > > > > >>>>> files
> > > > > > > > >>>>>>> covering major pages like DagsList and TaskInstance.
> > Many
> > > > > > > > >> critical
> > > > > > > > >>>>>>> pages (Connections,
> > > > > > > > >>>>>>> Variables, Pools, Dashboard, etc.) have no test
> > coverage at
> > > > > > all.
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>> I’m exploring a two-pronged approach to improve UI
> > testing:
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>> 1.
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>> Expand unit test coverage for individual
> > components and
> > > > > > pages
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>> 2.
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>> Add end-to-end (E2E) tests using Playwright for
> > > > complete
> > > > > > user
> > > > > > > > >>>>>> workflows
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>> This would give us both component-level safety and
> full
> > > > > > > > >>> user-journey
> > > > > > > > >>>>>>> testing across browsers, helping catch regressions
> > early.
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>> A few questions:
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>> -
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>> Should we prioritize expanding unit test coverage
> > > > first,
> > > > > or
> > > > > > > > >> work
> > > > > > > > >>>> on
> > > > > > > > >>>>>> unit
> > > > > > > > >>>>>>> and E2E tests in parallel?
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>> -
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>> For E2E testing, is Playwright a good choice, or
> > are
> > > > > there
> > > > > > > > >> other
> > > > > > > > >>>>>>> preferred tools we should explore?
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>> -
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>> If Playwright, should we use TypeScript or Python
> > as
> > > > > > language
> > > > > > > > >>> for
> > > > > > > > >>>>>> tests?
> > > > > > > > >>>>>>> -
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>> Any other suggestions to strengthen our UI
> testing
> > > > > > strategy?
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>> I’d love to hear your thoughts on whether this would
> be
> > > > > > valuable
> > > > > > > > >>> for
> > > > > > > > >>>>> the
> > > > > > > > >>>>>>> project before diving into implementation details.
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>> Regards,
> > > > > > > > >>>>>>> Rahul Vats
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>
> > > > > > > > >>
> > > > > >
> > ---------------------------------------------------------------------
> > > > > > > > >>>>>> To unsubscribe, e-mail:
> > dev-unsubscribe@airflow.apache.org
> > > > > > > > >>>>>> For additional commands, e-mail:
> > > > dev-help@airflow.apache.org
> > > > > > > > >>>>>>
> > > > > > > > >>>>>>
> > > > > > > > >>
> > > > > > > >
> > > > > > > >
> > > > ---------------------------------------------------------------------
> > > > > > > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > > > > > > For additional commands, e-mail: dev-help@airflow.apache.org
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > For additional commands, e-mail: dev-help@airflow.apache.org
> >
> >
>
Re: [Discuss] Improving Airflow UI Testing Coverage
Posted by Jarek Potiuk <ja...@potiuk.com>.
Cool!
On Fri, Dec 12, 2025 at 4:34 PM Vincent Beck <vi...@apache.org> wrote:
> Nice! Thanks for driving it! Really excited about it!
>
> On 2025/12/12 10:02:18 Rahul Vats wrote:
> > Hi Everyone,
> >
> > Following up on our earlier discussions about UI E2E testing -- the
> initial
> > framework is now in place and ready for community contributions!
> >
> > *What's Been Merged:*
> > - Playwright framework with TypeScript (#58548
> > <https://github.com/apache/airflow/pull/58548> )
> > - CI workflow integrated (#58901
> > <https://github.com/apache/airflow/pull/58901>)
> > - Initial test covering login and DAG triggering
> >
> > *Meta Issue for Tracking:*
> > #59028 <https://github.com/apache/airflow/issues/59028>
> >
> > *Sub-Issues Ready to Pick Up*:
> >
> >
> > - https://github.com/apache/airflow/issues/59306
> > - https://github.com/apache/airflow/issues/59307
> > - https://github.com/apache/airflow/issues/59308
> > - https://github.com/apache/airflow/issues/59309
> > - https://github.com/apache/airflow/issues/59357
> > - https://github.com/apache/airflow/issues/59358
> >
> >
> > More sub-issues will be created and linked to the meta issue #59028
> > <https://github.com/apache/airflow/issues/59028>
> >
> > *Contribution Guide*
> >
> https://github.com/apache/airflow/blob/main/airflow-core/src/airflow/ui/tests/e2e/README.md#contributing-new-tests
> >
> > *Slack Channel:*
> > Join #sig-airflow-ui-e2e-tests
> > <https://apache-airflow.slack.com/archives/C0A2QT7UUGP> for questions
> and
> > discussions.
> >
> > Looking forward to contributions!
> >
> > Regards,
> > Rahul Vats
> >
> > On Wed, 15 Oct 2025 at 10:45, Amogh Desai <am...@apache.org> wrote:
> >
> > > Thanks for the summary, Rahul.
> > >
> > > Sounds like a good plan, you could coordinate the work better either by
> > > using
> > > a Github Board or Slack channel or both too.
> > >
> > > Thanks & Regards,
> > > Amogh Desai
> > >
> > >
> > > On Fri, Oct 10, 2025 at 9:11 PM Rahul Vats <ra...@gmail.com>
> wrote:
> > >
> > > > Hi Everyone,
> > > >
> > > > Thank you all for the fantastic feedback and strong support! The
> > > community
> > > > consensus is clear and encouraging:
> > > >
> > > > - High priority for improving UI testing
> > > > - Playwright + TypeScript for e2e testing
> > > > - Parallel development of unit and e2e tests
> > > > - Test pyramid approach with focused e2e scope
> > > > - Multiple volunteers ready to contribute!
> > > >
> > > >
> > > > *Addressing Key Concerns*
> > > > Important points on e2e complexity are well taken. I completely
> agree
> > > that
> > > > we should follow the test pyramid.
> > > >
> > > > - E2E tests should cover only critical workflows (~10-15 scenarios)
> > > > - Unit tests are the foundation and should be the primary focus and
> we
> > > > should have an enforcement so that every UI change comes with a unit
> > > test.
> > > > - CI integration needs careful planning to avoid slowing development.
> > > >
> > > > *Next Steps*
> > > >
> > > > - I plan to work on a POC for E2E tests and will raise a PR for
> review
> > > > soon. After that, I'll share a detailed implementation plan and
> > > coordinate
> > > > with volunteers who offered to help.
> > > > - In the meantime, I suggest we should continue expanding unit test
> > > > coverage as planned.
> > > >
> > > >
> > > > It was great meeting you all at the Airflow Summit. Momentum here is
> > > > fantastic, and I'm excited to work with everyone who offered to help
> make
> > > > Airflow 3 UI much more reliable!
> > > >
> > > > Regards,
> > > > Rahul Vats
> > > >
> > > > On Thu, 25 Sept 2025 at 07:49, Vikram Koka via dev <
> > > dev@airflow.apache.org
> > > > >
> > > > wrote:
> > > >
> > > > > Rahul,
> > > > > Thank you for starting this thread.
> > > > >
> > > > > Strong +1 from me as well.
> > > > > I not only agree with this, but I believe this needs to be a high
> > > > priority
> > > > > going forward and before 3.2.
> > > > >
> > > > > I also agree with the comments on the "test pyramid" approach, of
> there
> > > > > needing to be FAR more unit tests as compared to end to end tests.
> > > > > I also believe that a certain level of rigor around this i.e. some
> form
> > > > of
> > > > > enforcement to ensure that all new UI changes come with a unit
> test.
> > > > >
> > > > > I would also add the Core API changes (for UI) to the mix here,
> since a
> > > > > fair number of UI changes requires underlying API changes.
> > > > >
> > > > > Vikram
> > > > >
> > > > >
> > > > > On Thu, Sep 25, 2025 at 12:04 AM Amogh Desai <
> amoghdesai@apache.org>
> > > > > wrote:
> > > > >
> > > > > > Valid points.
> > > > > >
> > > > > > I was just expressing concern from a CI/CD failure perspective
> but in
> > > > no
> > > > > > way
> > > > > > was I objecting to usage of typescript.
> > > > > >
> > > > > > Anything that works for the team that owns most of the tests
> works by
> > > > me!
> > > > > >
> > > > > >
> > > > > > Thanks & Regards,
> > > > > > Amogh Desai
> > > > > >
> > > > > >
> > > > > > On Thu, Sep 25, 2025 at 1:39 AM Jens Scheffler <
> jscheffl@apache.org>
> > > > > > wrote:
> > > > > >
> > > > > > > Echoing a full agreement to all statement before.
> > > > > > >
> > > > > > > I also would emphasize the test pyramid. As we really have a
> low
> > > > > > > coverage we should maybe also have a metric. I would also
> propose
> > > > that
> > > > > > > we focus more on testing coverage == hopefully also improved
> > > quality
> > > > > > > preventing regression in 3.2. One option could a bit of force
> to
> > > > force
> > > > > > > all new changes applied to UI to request a unit test being
> added
> > > (all
> > > > > > > changes TS/x files need to have a corresponding .test.ts/x
> file)
> > > > except
> > > > > > > for cases where it does not make sense we allow exclusion of
> the
> > > rule
> > > > > by
> > > > > > > adding a label by maintainers. (just want to drop the idea,
> might
> > > be
> > > > > > > many will hate it)
> > > > > > >
> > > > > > > For language also have no strong preference but even though I
> am a
> > > TS
> > > > > > > noob I see that testing UI is closer to UI development, to
> using
> > > > Python
> > > > > > > for TS UI testing might make a knowledge gap also for potential
> > > > utility
> > > > > > > code. So as most voices would propose TS (maybe then finally I
> get
> > > to
> > > > > > > understand it)
> > > > > > >
> > > > > > > Jens
> > > > > > >
> > > > > > > On 24.09.25 18:33, Jarek Potiuk wrote:
> > > > > > > > Small poking at important things from holidays while having a
> > > > coffee
> > > > > > at a
> > > > > > > > place with internet :)
> > > > > > > >
> > > > > > > > +1 on having both. I have no strong opinion on the language
> and
> > > > > > > Playwright.
> > > > > > > >
> > > > > > > > But there are few things for consideration:
> > > > > > > >
> > > > > > > > We should have many, many more unit tests than e2e tests.
> This is
> > > > the
> > > > > > > > classic test pyramid case: The Practical Test Pyramid
> > > > > > > > https://share.google/YHVIt8EBptT1yYFur
> > > > > > > >
> > > > > > > > The e2e tests are slow, brittle and difficult to maintain
> and we
> > > > > should
> > > > > > > > have rather small number of those - mostly green path on all
> > > major
> > > > > > > > functionalities. Possibly with some cross browser checks -
> but
> > > > those
> > > > > > are
> > > > > > > > very difficult to get right.
> > > > > > > >
> > > > > > > > It will be a pain to maintain those. Like our k8s e2e tests -
> > > when
> > > > > they
> > > > > > > > break (rarely) people will have hard time to create and run
> the
> > > > tests
> > > > > > so
> > > > > > > > the tooling around that has to drive people who will be
> > > > reproducing
> > > > > > the
> > > > > > > > failures by hand - how to set it up and how to reproduce it
> and
> > > how
> > > > > to
> > > > > > > > iterate as fast as possible.
> > > > > > > >
> > > > > > > > Keeping the tests stable and side-effect free will be a
> > > challenge -
> > > > > so
> > > > > > > > likely a good 'reset the environment' before each test is a
> must.
> > > > > Also
> > > > > > > they
> > > > > > > > need to be run with all UI changes in CI which will make UI
> PR
> > > run
> > > > > > > longer.
> > > > > > > > It's an imperative to get them run not only in canary builds
> but
> > > in
> > > > > all
> > > > > > > PRs
> > > > > > > > for ui and API and likely whenever any core model change. It
> will
> > > > > slow
> > > > > > > down
> > > > > > > > UI development and create frustration for contributors
> unfamiliar
> > > > > with
> > > > > > > the
> > > > > > > > test framework so we need to have at least a few people
> > > proficient
> > > > in
> > > > > > > > intricacies of the test framework dedicated and willing to
> help
> > > > those
> > > > > > who
> > > > > > > > will struggle with issues.
> > > > > > > >
> > > > > > > > Language is secondary - e2e tests will require all stack -
> Python
> > > > > > > (backend)
> > > > > > > > and Typescript and react experiences anyway. Most likely
> people
> > > who
> > > > > > write
> > > > > > > > UI should be involved so Typescript is fine, and also even
> those
> > > > who
> > > > > do
> > > > > > > not
> > > > > > > > know TypeScript will be able to fix some obvious cases.
> > > > > > > >
> > > > > > > > J.
> > > > > > > >
> > > > > > > > śr., 24 wrz 2025, 09:50 użytkownik Shubham Raj <
> > > > > shubhamraj@apache.org>
> > > > > > > > napisał:
> > > > > > > >
> > > > > > > >> This addition will be really nice.
> > > > > > > >> Also, at this point of 3.x, I think UI has been relatively
> > > stable
> > > > > and
> > > > > > > >> writing e2e would be a good starting point. Unit tests can
> go
> > > > > > parallel.
> > > > > > > >> It's right that people maintaining CI/CD would be first
> persons
> > > to
> > > > > see
> > > > > > > any
> > > > > > > >> breakage, but I feel writing in ts will be easier for folks
> > > > working
> > > > > on
> > > > > > > >> frontend, and also for the new contributors who would like
> to
> > > > > > > contribute in
> > > > > > > >> this UI e2e tests coming from cypress or similar experience,
> > > since
> > > > > > > majority
> > > > > > > >> of other choices do come with ts/js support.
> > > > > > > >> I would be happy to help with the e2e tests. We should
> probably
> > > go
> > > > > in
> > > > > > > >> phases with P0/1/2.
> > > > > > > >>
> > > > > > > >> Thanks.
> > > > > > > >>
> > > > > > > >>
> > > > > > > >> On Wed, Sep 24, 2025 at 9:01 PM Pierre Jeambrun <
> > > > > > pierrejbrun@gmail.com>
> > > > > > > >> wrote:
> > > > > > > >>
> > > > > > > >>> I also think that having a good unit test coverage + e2e
> tests
> > > > > would
> > > > > > > >> really
> > > > > > > >>> help maintaining quality of releases and prevent
> regressions
> > > > early,
> > > > > > > >> making
> > > > > > > >>> all that much more manageable.
> > > > > > > >>>
> > > > > > > >>> My two cents regarding the questions above:
> > > > > > > >>>
> > > > > > > >>> -
> > > > > > > >>> Should we prioritize expanding unit test coverage
> first, or
> > > > > work
> > > > > > on
> > > > > > > >> unit
> > > > > > > >>> and E2E tests in parallel?
> > > > > > > >>>
> > > > > > > >>> Both in parallel sounds good to me. They do not depend
> on
> > > > each
> > > > > > > other
> > > > > > > >> (At
> > > > > > > >>> least in my mind)
> > > > > > > >>> -
> > > > > > > >>> For E2E testing, is Playwright a good choice, or are
> there
> > > > > other
> > > > > > > >>> preferred tools we should explore?
> > > > > > > >>>
> > > > > > > >>> I like playwright and I feel this is the modern
> solution to
> > > > go
> > > > > > to.
> > > > > > > >>> -
> > > > > > > >>> If Playwright, should we use TypeScript or Python as
> > > language
> > > > > for
> > > > > > > >> tests?
> > > > > > > >>> Good arguments mentioned above for both sides. I
> personally
> > > > > think
> > > > > > > >> (maybe
> > > > > > > >>> wrongly), that ideally people maintaining / developing e2e
> > > tests
> > > > > will
> > > > > > > be
> > > > > > > >>> people with
> > > > > > > >>> front-end knowledge. (events, identifiers, selectors,
> css
> > > > > > styling,
> > > > > > > >>> browser local states, etc..), so I tend to think that
> those
> > > > people
> > > > > > > will
> > > > > > > >>> most likely come from the front-end community and
> > > > > > > >>> having that in typescript would probably help them get
> > > > started.
> > > > > > > >>>
> > > > > > > >>>
> > > > > > > >>> On Wed, Sep 24, 2025 at 10:01 AM Amogh Desai <
> > > > > amoghdesai@apache.org>
> > > > > > > >>> wrote:
> > > > > > > >>>
> > > > > > > >>>> Thanks for starting this discussion, Rahul.
> > > > > > > >>>>
> > > > > > > >>>> I am aligned with your thoughts, we have seen many many UI
> > > > issues
> > > > > > > >> lately
> > > > > > > >>>> which I believe we should prioritize fixing or at least
> > > reducing
> > > > > > going
> > > > > > > >>>> forward.
> > > > > > > >>>>
> > > > > > > >>>> To answer your question(s) --
> > > > > > > >>>>
> > > > > > > >>>> * I think the unit tests and e2e / user flow tests as
> > > > orthogonal,
> > > > > so
> > > > > > > >> they
> > > > > > > >>>> can be
> > > > > > > >>>> developed in parallel.
> > > > > > > >>>>
> > > > > > > >>>> * While I do not have real experience with Playwright but
> from
> > > > > what
> > > > > > I
> > > > > > > >>> hear
> > > > > > > >>>> / read on reddit,
> > > > > > > >>>> github, and related forums, it looks like Playwright can
> do
> > > > > similar
> > > > > > > job
> > > > > > > >>> as
> > > > > > > >>>> alternatives such as
> > > > > > > >>>> Selenium, Cypress, while being more beginner friendly,
> which
> > > > could
> > > > > > > play
> > > > > > > >>>> better in our favour.
> > > > > > > >>>>
> > > > > > > >>>> * Again, not very strong on this but I feel that having
> > > > Typescript
> > > > > > as
> > > > > > > >> the
> > > > > > > >>>> language of choice would limit
> > > > > > > >>>> the number of contributors we get to help on this. Python
> > > would
> > > > be
> > > > > > my
> > > > > > > >>>> weapon of choice, given
> > > > > > > >>>> the "testing" is not limited by it and it potentially
> could
> > > > > > encourage
> > > > > > > >>>> broader participation.
> > > > > > > >>>>
> > > > > > > >>>> (Folks active on CI/CD(atleast me), are not very
> comfortable
> > > > with
> > > > > > > >>>> Typescript tests compared to
> > > > > > > >>>> Python)
> > > > > > > >>>>
> > > > > > > >>>> One additional point: It would be valuable if we extract
> the
> > > > e2e /
> > > > > > > user
> > > > > > > >>>> flow testing into a GH job that
> > > > > > > >>>> a release manager could run as "sort" of last validation
> > > before
> > > > > > > >> releasing
> > > > > > > >>>> to avoid any last
> > > > > > > >>>> minute issues.
> > > > > > > >>>>
> > > > > > > >>>> Thanks & Regards,
> > > > > > > >>>> Amogh Desai
> > > > > > > >>>>
> > > > > > > >>>>
> > > > > > > >>>> On Tue, Sep 23, 2025 at 9:15 PM Brent Bovenzi via dev <
> > > > > > > >>>> dev@airflow.apache.org> wrote:
> > > > > > > >>>>
> > > > > > > >>>>> I agree that we need to improve UI tests. Happy to make
> that
> > > a
> > > > > > > >> priority
> > > > > > > >>>>> during 3.1.x and 3.2 development.
> > > > > > > >>>>>
> > > > > > > >>>>> Personally, I will start with expanding unit tests
> first. But
> > > > > that
> > > > > > > >>>> doesn't
> > > > > > > >>>>> mean others can't get started on E2E tests.
> > > > > > > >>>>>
> > > > > > > >>>>> I lean towards using typescript for playwright. With E2E
> we
> > > are
> > > > > > > >> testing
> > > > > > > >>>> the
> > > > > > > >>>>> UI more than we are testing the API so I think the code
> > > should
> > > > > sit
> > > > > > > >>> closer
> > > > > > > >>>>> to the UI code.
> > > > > > > >>>>>
> > > > > > > >>>>> On Tue, Sep 23, 2025 at 9:38 AM Vincent Beck <
> > > > > vincbeck@apache.org>
> > > > > > > >>>> wrote:
> > > > > > > >>>>>> Not answering any of these questions (sorry) but strong
> +1
> > > on
> > > > > > > >> adding
> > > > > > > >>>>>> end-to-end (E2E) tests. This will significantly make
> our UI
> > > > more
> > > > > > > >>> robust
> > > > > > > >>>>> but
> > > > > > > >>>>>> also will speed up some of our work. An example is
> front-end
> > > > > > > >>> dependency
> > > > > > > >>>>>> upgrades by dependabot. Today we have no way of knowing
> > > > whether
> > > > > > > >> these
> > > > > > > >>>> PRs
> > > > > > > >>>>>> break the UI, in order to merge these PRs we need to
> compile
> > > > the
> > > > > > > >>>>> front-end
> > > > > > > >>>>>> assets and test manually whether the UI work. Having
> > > > end-to-end
> > > > > > > >> tests
> > > > > > > >>>> on
> > > > > > > >>>>>> the front-end and making them run in the CI would speed
> up
> > > > this
> > > > > > > >> very
> > > > > > > >>>>>> example (on top of increasing code robustness, increase
> our
> > > > > > > >>> confidence
> > > > > > > >>>>> etc).
> > > > > > > >>>>>> On 2025/09/23 11:03:40 Rahul Vats wrote:
> > > > > > > >>>>>>> Hi all,
> > > > > > > >>>>>>>
> > > > > > > >>>>>>> With Airflow 3.0, the UI has undergone a complete
> revamp,
> > > and
> > > > > new
> > > > > > > >>>>> changes
> > > > > > > >>>>>>> are naturally more prone to regressions. We've already
> seen
> > > > > more
> > > > > > > >> UI
> > > > > > > >>>>>>> regressions that are hard to catch with our current
> limited
> > > > > test
> > > > > > > >>>>>> coverage.
> > > > > > > >>>>>>> Manual testing is also challenging given the wide
> variety
> > > of
> > > > > > > >>>> workflows
> > > > > > > >>>>>> and
> > > > > > > >>>>>>> screens in the UI.
> > > > > > > >>>>>>>
> > > > > > > >>>>>>> Currently, the Airflow UI has a few unit tests using
> Vitest
> > > > and
> > > > > > > >>> React
> > > > > > > >>>>>>> Testing Library, but coverage is quite limited — only
> > > about 4
> > > > > > > >> test
> > > > > > > >>>>> files
> > > > > > > >>>>>>> covering major pages like DagsList and TaskInstance.
> Many
> > > > > > > >> critical
> > > > > > > >>>>>>> pages (Connections,
> > > > > > > >>>>>>> Variables, Pools, Dashboard, etc.) have no test
> coverage at
> > > > > all.
> > > > > > > >>>>>>>
> > > > > > > >>>>>>> I’m exploring a two-pronged approach to improve UI
> testing:
> > > > > > > >>>>>>>
> > > > > > > >>>>>>> 1.
> > > > > > > >>>>>>>
> > > > > > > >>>>>>> Expand unit test coverage for individual
> components and
> > > > > pages
> > > > > > > >>>>>>>
> > > > > > > >>>>>>> 2.
> > > > > > > >>>>>>>
> > > > > > > >>>>>>> Add end-to-end (E2E) tests using Playwright for
> > > complete
> > > > > user
> > > > > > > >>>>>> workflows
> > > > > > > >>>>>>>
> > > > > > > >>>>>>> This would give us both component-level safety and full
> > > > > > > >>> user-journey
> > > > > > > >>>>>>> testing across browsers, helping catch regressions
> early.
> > > > > > > >>>>>>>
> > > > > > > >>>>>>> A few questions:
> > > > > > > >>>>>>>
> > > > > > > >>>>>>> -
> > > > > > > >>>>>>>
> > > > > > > >>>>>>> Should we prioritize expanding unit test coverage
> > > first,
> > > > or
> > > > > > > >> work
> > > > > > > >>>> on
> > > > > > > >>>>>> unit
> > > > > > > >>>>>>> and E2E tests in parallel?
> > > > > > > >>>>>>>
> > > > > > > >>>>>>> -
> > > > > > > >>>>>>>
> > > > > > > >>>>>>> For E2E testing, is Playwright a good choice, or
> are
> > > > there
> > > > > > > >> other
> > > > > > > >>>>>>> preferred tools we should explore?
> > > > > > > >>>>>>>
> > > > > > > >>>>>>> -
> > > > > > > >>>>>>>
> > > > > > > >>>>>>> If Playwright, should we use TypeScript or Python
> as
> > > > > language
> > > > > > > >>> for
> > > > > > > >>>>>> tests?
> > > > > > > >>>>>>> -
> > > > > > > >>>>>>>
> > > > > > > >>>>>>> Any other suggestions to strengthen our UI testing
> > > > > strategy?
> > > > > > > >>>>>>>
> > > > > > > >>>>>>>
> > > > > > > >>>>>>> I’d love to hear your thoughts on whether this would be
> > > > > valuable
> > > > > > > >>> for
> > > > > > > >>>>> the
> > > > > > > >>>>>>> project before diving into implementation details.
> > > > > > > >>>>>>>
> > > > > > > >>>>>>> Regards,
> > > > > > > >>>>>>> Rahul Vats
> > > > > > > >>>>>>>
> > > > > > > >>>>>>
> > > > > > > >>
> > > > >
> ---------------------------------------------------------------------
> > > > > > > >>>>>> To unsubscribe, e-mail:
> dev-unsubscribe@airflow.apache.org
> > > > > > > >>>>>> For additional commands, e-mail:
> > > dev-help@airflow.apache.org
> > > > > > > >>>>>>
> > > > > > > >>>>>>
> > > > > > > >>
> > > > > > >
> > > > > > >
> > > ---------------------------------------------------------------------
> > > > > > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > > > > > For additional commands, e-mail: dev-help@airflow.apache.org
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> For additional commands, e-mail: dev-help@airflow.apache.org
>
>
Re: [Discuss] Improving Airflow UI Testing Coverage
Posted by Vincent Beck <vi...@apache.org>.
Nice! Thanks for driving it! Really excited about it!
On 2025/12/12 10:02:18 Rahul Vats wrote:
> Hi Everyone,
>
> Following up on our earlier discussions about UI E2E testing -- the initial
> framework is now in place and ready for community contributions!
>
> *What's Been Merged:*
> - Playwright framework with TypeScript (#58548
> <https://github.com/apache/airflow/pull/58548> )
> - CI workflow integrated (#58901
> <https://github.com/apache/airflow/pull/58901>)
> - Initial test covering login and DAG triggering
>
> *Meta Issue for Tracking:*
> #59028 <https://github.com/apache/airflow/issues/59028>
>
> *Sub-Issues Ready to Pick Up*:
>
>
> - https://github.com/apache/airflow/issues/59306
> - https://github.com/apache/airflow/issues/59307
> - https://github.com/apache/airflow/issues/59308
> - https://github.com/apache/airflow/issues/59309
> - https://github.com/apache/airflow/issues/59357
> - https://github.com/apache/airflow/issues/59358
>
>
> More sub-issues will be created and linked to the meta issue #59028
> <https://github.com/apache/airflow/issues/59028>
>
> *Contribution Guide*
> https://github.com/apache/airflow/blob/main/airflow-core/src/airflow/ui/tests/e2e/README.md#contributing-new-tests
>
> *Slack Channel:*
> Join #sig-airflow-ui-e2e-tests
> <https://apache-airflow.slack.com/archives/C0A2QT7UUGP> for questions and
> discussions.
>
> Looking forward to contributions!
>
> Regards,
> Rahul Vats
>
> On Wed, 15 Oct 2025 at 10:45, Amogh Desai <am...@apache.org> wrote:
>
> > Thanks for the summary, Rahul.
> >
> > Sounds like a good plan, you could coordinate the work better either by
> > using
> > a Github Board or Slack channel or both too.
> >
> > Thanks & Regards,
> > Amogh Desai
> >
> >
> > On Fri, Oct 10, 2025 at 9:11 PM Rahul Vats <ra...@gmail.com> wrote:
> >
> > > Hi Everyone,
> > >
> > > Thank you all for the fantastic feedback and strong support! The
> > community
> > > consensus is clear and encouraging:
> > >
> > > - High priority for improving UI testing
> > > - Playwright + TypeScript for e2e testing
> > > - Parallel development of unit and e2e tests
> > > - Test pyramid approach with focused e2e scope
> > > - Multiple volunteers ready to contribute!
> > >
> > >
> > > *Addressing Key Concerns*
> > > Important points on e2e complexity are well taken. I completely agree
> > that
> > > we should follow the test pyramid.
> > >
> > > - E2E tests should cover only critical workflows (~10-15 scenarios)
> > > - Unit tests are the foundation and should be the primary focus and we
> > > should have an enforcement so that every UI change comes with a unit
> > test.
> > > - CI integration needs careful planning to avoid slowing development.
> > >
> > > *Next Steps*
> > >
> > > - I plan to work on a POC for E2E tests and will raise a PR for review
> > > soon. After that, I'll share a detailed implementation plan and
> > coordinate
> > > with volunteers who offered to help.
> > > - In the meantime, I suggest we should continue expanding unit test
> > > coverage as planned.
> > >
> > >
> > > It was great meeting you all at the Airflow Summit. Momentum here is
> > > fantastic, and I'm excited to work with everyone who offered to help make
> > > Airflow 3 UI much more reliable!
> > >
> > > Regards,
> > > Rahul Vats
> > >
> > > On Thu, 25 Sept 2025 at 07:49, Vikram Koka via dev <
> > dev@airflow.apache.org
> > > >
> > > wrote:
> > >
> > > > Rahul,
> > > > Thank you for starting this thread.
> > > >
> > > > Strong +1 from me as well.
> > > > I not only agree with this, but I believe this needs to be a high
> > > priority
> > > > going forward and before 3.2.
> > > >
> > > > I also agree with the comments on the "test pyramid" approach, of there
> > > > needing to be FAR more unit tests as compared to end to end tests.
> > > > I also believe that a certain level of rigor around this i.e. some form
> > > of
> > > > enforcement to ensure that all new UI changes come with a unit test.
> > > >
> > > > I would also add the Core API changes (for UI) to the mix here, since a
> > > > fair number of UI changes requires underlying API changes.
> > > >
> > > > Vikram
> > > >
> > > >
> > > > On Thu, Sep 25, 2025 at 12:04 AM Amogh Desai <am...@apache.org>
> > > > wrote:
> > > >
> > > > > Valid points.
> > > > >
> > > > > I was just expressing concern from a CI/CD failure perspective but in
> > > no
> > > > > way
> > > > > was I objecting to usage of typescript.
> > > > >
> > > > > Anything that works for the team that owns most of the tests works by
> > > me!
> > > > >
> > > > >
> > > > > Thanks & Regards,
> > > > > Amogh Desai
> > > > >
> > > > >
> > > > > On Thu, Sep 25, 2025 at 1:39 AM Jens Scheffler <js...@apache.org>
> > > > > wrote:
> > > > >
> > > > > > Echoing a full agreement to all statement before.
> > > > > >
> > > > > > I also would emphasize the test pyramid. As we really have a low
> > > > > > coverage we should maybe also have a metric. I would also propose
> > > that
> > > > > > we focus more on testing coverage == hopefully also improved
> > quality
> > > > > > preventing regression in 3.2. One option could a bit of force to
> > > force
> > > > > > all new changes applied to UI to request a unit test being added
> > (all
> > > > > > changes TS/x files need to have a corresponding .test.ts/x file)
> > > except
> > > > > > for cases where it does not make sense we allow exclusion of the
> > rule
> > > > by
> > > > > > adding a label by maintainers. (just want to drop the idea, might
> > be
> > > > > > many will hate it)
> > > > > >
> > > > > > For language also have no strong preference but even though I am a
> > TS
> > > > > > noob I see that testing UI is closer to UI development, to using
> > > Python
> > > > > > for TS UI testing might make a knowledge gap also for potential
> > > utility
> > > > > > code. So as most voices would propose TS (maybe then finally I get
> > to
> > > > > > understand it)
> > > > > >
> > > > > > Jens
> > > > > >
> > > > > > On 24.09.25 18:33, Jarek Potiuk wrote:
> > > > > > > Small poking at important things from holidays while having a
> > > coffee
> > > > > at a
> > > > > > > place with internet :)
> > > > > > >
> > > > > > > +1 on having both. I have no strong opinion on the language and
> > > > > > Playwright.
> > > > > > >
> > > > > > > But there are few things for consideration:
> > > > > > >
> > > > > > > We should have many, many more unit tests than e2e tests. This is
> > > the
> > > > > > > classic test pyramid case: The Practical Test Pyramid
> > > > > > > https://share.google/YHVIt8EBptT1yYFur
> > > > > > >
> > > > > > > The e2e tests are slow, brittle and difficult to maintain and we
> > > > should
> > > > > > > have rather small number of those - mostly green path on all
> > major
> > > > > > > functionalities. Possibly with some cross browser checks - but
> > > those
> > > > > are
> > > > > > > very difficult to get right.
> > > > > > >
> > > > > > > It will be a pain to maintain those. Like our k8s e2e tests -
> > when
> > > > they
> > > > > > > break (rarely) people will have hard time to create and run the
> > > tests
> > > > > so
> > > > > > > the tooling around that has to drive people who will be
> > > reproducing
> > > > > the
> > > > > > > failures by hand - how to set it up and how to reproduce it and
> > how
> > > > to
> > > > > > > iterate as fast as possible.
> > > > > > >
> > > > > > > Keeping the tests stable and side-effect free will be a
> > challenge -
> > > > so
> > > > > > > likely a good 'reset the environment' before each test is a must.
> > > > Also
> > > > > > they
> > > > > > > need to be run with all UI changes in CI which will make UI PR
> > run
> > > > > > longer.
> > > > > > > It's an imperative to get them run not only in canary builds but
> > in
> > > > all
> > > > > > PRs
> > > > > > > for ui and API and likely whenever any core model change. It will
> > > > slow
> > > > > > down
> > > > > > > UI development and create frustration for contributors unfamiliar
> > > > with
> > > > > > the
> > > > > > > test framework so we need to have at least a few people
> > proficient
> > > in
> > > > > > > intricacies of the test framework dedicated and willing to help
> > > those
> > > > > who
> > > > > > > will struggle with issues.
> > > > > > >
> > > > > > > Language is secondary - e2e tests will require all stack - Python
> > > > > > (backend)
> > > > > > > and Typescript and react experiences anyway. Most likely people
> > who
> > > > > write
> > > > > > > UI should be involved so Typescript is fine, and also even those
> > > who
> > > > do
> > > > > > not
> > > > > > > know TypeScript will be able to fix some obvious cases.
> > > > > > >
> > > > > > > J.
> > > > > > >
> > > > > > > śr., 24 wrz 2025, 09:50 użytkownik Shubham Raj <
> > > > shubhamraj@apache.org>
> > > > > > > napisał:
> > > > > > >
> > > > > > >> This addition will be really nice.
> > > > > > >> Also, at this point of 3.x, I think UI has been relatively
> > stable
> > > > and
> > > > > > >> writing e2e would be a good starting point. Unit tests can go
> > > > > parallel.
> > > > > > >> It's right that people maintaining CI/CD would be first persons
> > to
> > > > see
> > > > > > any
> > > > > > >> breakage, but I feel writing in ts will be easier for folks
> > > working
> > > > on
> > > > > > >> frontend, and also for the new contributors who would like to
> > > > > > contribute in
> > > > > > >> this UI e2e tests coming from cypress or similar experience,
> > since
> > > > > > majority
> > > > > > >> of other choices do come with ts/js support.
> > > > > > >> I would be happy to help with the e2e tests. We should probably
> > go
> > > > in
> > > > > > >> phases with P0/1/2.
> > > > > > >>
> > > > > > >> Thanks.
> > > > > > >>
> > > > > > >>
> > > > > > >> On Wed, Sep 24, 2025 at 9:01 PM Pierre Jeambrun <
> > > > > pierrejbrun@gmail.com>
> > > > > > >> wrote:
> > > > > > >>
> > > > > > >>> I also think that having a good unit test coverage + e2e tests
> > > > would
> > > > > > >> really
> > > > > > >>> help maintaining quality of releases and prevent regressions
> > > early,
> > > > > > >> making
> > > > > > >>> all that much more manageable.
> > > > > > >>>
> > > > > > >>> My two cents regarding the questions above:
> > > > > > >>>
> > > > > > >>> -
> > > > > > >>> Should we prioritize expanding unit test coverage first, or
> > > > work
> > > > > on
> > > > > > >> unit
> > > > > > >>> and E2E tests in parallel?
> > > > > > >>>
> > > > > > >>> Both in parallel sounds good to me. They do not depend on
> > > each
> > > > > > other
> > > > > > >> (At
> > > > > > >>> least in my mind)
> > > > > > >>> -
> > > > > > >>> For E2E testing, is Playwright a good choice, or are there
> > > > other
> > > > > > >>> preferred tools we should explore?
> > > > > > >>>
> > > > > > >>> I like playwright and I feel this is the modern solution to
> > > go
> > > > > to.
> > > > > > >>> -
> > > > > > >>> If Playwright, should we use TypeScript or Python as
> > language
> > > > for
> > > > > > >> tests?
> > > > > > >>> Good arguments mentioned above for both sides. I personally
> > > > think
> > > > > > >> (maybe
> > > > > > >>> wrongly), that ideally people maintaining / developing e2e
> > tests
> > > > will
> > > > > > be
> > > > > > >>> people with
> > > > > > >>> front-end knowledge. (events, identifiers, selectors, css
> > > > > styling,
> > > > > > >>> browser local states, etc..), so I tend to think that those
> > > people
> > > > > > will
> > > > > > >>> most likely come from the front-end community and
> > > > > > >>> having that in typescript would probably help them get
> > > started.
> > > > > > >>>
> > > > > > >>>
> > > > > > >>> On Wed, Sep 24, 2025 at 10:01 AM Amogh Desai <
> > > > amoghdesai@apache.org>
> > > > > > >>> wrote:
> > > > > > >>>
> > > > > > >>>> Thanks for starting this discussion, Rahul.
> > > > > > >>>>
> > > > > > >>>> I am aligned with your thoughts, we have seen many many UI
> > > issues
> > > > > > >> lately
> > > > > > >>>> which I believe we should prioritize fixing or at least
> > reducing
> > > > > going
> > > > > > >>>> forward.
> > > > > > >>>>
> > > > > > >>>> To answer your question(s) --
> > > > > > >>>>
> > > > > > >>>> * I think the unit tests and e2e / user flow tests as
> > > orthogonal,
> > > > so
> > > > > > >> they
> > > > > > >>>> can be
> > > > > > >>>> developed in parallel.
> > > > > > >>>>
> > > > > > >>>> * While I do not have real experience with Playwright but from
> > > > what
> > > > > I
> > > > > > >>> hear
> > > > > > >>>> / read on reddit,
> > > > > > >>>> github, and related forums, it looks like Playwright can do
> > > > similar
> > > > > > job
> > > > > > >>> as
> > > > > > >>>> alternatives such as
> > > > > > >>>> Selenium, Cypress, while being more beginner friendly, which
> > > could
> > > > > > play
> > > > > > >>>> better in our favour.
> > > > > > >>>>
> > > > > > >>>> * Again, not very strong on this but I feel that having
> > > Typescript
> > > > > as
> > > > > > >> the
> > > > > > >>>> language of choice would limit
> > > > > > >>>> the number of contributors we get to help on this. Python
> > would
> > > be
> > > > > my
> > > > > > >>>> weapon of choice, given
> > > > > > >>>> the "testing" is not limited by it and it potentially could
> > > > > encourage
> > > > > > >>>> broader participation.
> > > > > > >>>>
> > > > > > >>>> (Folks active on CI/CD(atleast me), are not very comfortable
> > > with
> > > > > > >>>> Typescript tests compared to
> > > > > > >>>> Python)
> > > > > > >>>>
> > > > > > >>>> One additional point: It would be valuable if we extract the
> > > e2e /
> > > > > > user
> > > > > > >>>> flow testing into a GH job that
> > > > > > >>>> a release manager could run as "sort" of last validation
> > before
> > > > > > >> releasing
> > > > > > >>>> to avoid any last
> > > > > > >>>> minute issues.
> > > > > > >>>>
> > > > > > >>>> Thanks & Regards,
> > > > > > >>>> Amogh Desai
> > > > > > >>>>
> > > > > > >>>>
> > > > > > >>>> On Tue, Sep 23, 2025 at 9:15 PM Brent Bovenzi via dev <
> > > > > > >>>> dev@airflow.apache.org> wrote:
> > > > > > >>>>
> > > > > > >>>>> I agree that we need to improve UI tests. Happy to make that
> > a
> > > > > > >> priority
> > > > > > >>>>> during 3.1.x and 3.2 development.
> > > > > > >>>>>
> > > > > > >>>>> Personally, I will start with expanding unit tests first. But
> > > > that
> > > > > > >>>> doesn't
> > > > > > >>>>> mean others can't get started on E2E tests.
> > > > > > >>>>>
> > > > > > >>>>> I lean towards using typescript for playwright. With E2E we
> > are
> > > > > > >> testing
> > > > > > >>>> the
> > > > > > >>>>> UI more than we are testing the API so I think the code
> > should
> > > > sit
> > > > > > >>> closer
> > > > > > >>>>> to the UI code.
> > > > > > >>>>>
> > > > > > >>>>> On Tue, Sep 23, 2025 at 9:38 AM Vincent Beck <
> > > > vincbeck@apache.org>
> > > > > > >>>> wrote:
> > > > > > >>>>>> Not answering any of these questions (sorry) but strong +1
> > on
> > > > > > >> adding
> > > > > > >>>>>> end-to-end (E2E) tests. This will significantly make our UI
> > > more
> > > > > > >>> robust
> > > > > > >>>>> but
> > > > > > >>>>>> also will speed up some of our work. An example is front-end
> > > > > > >>> dependency
> > > > > > >>>>>> upgrades by dependabot. Today we have no way of knowing
> > > whether
> > > > > > >> these
> > > > > > >>>> PRs
> > > > > > >>>>>> break the UI, in order to merge these PRs we need to compile
> > > the
> > > > > > >>>>> front-end
> > > > > > >>>>>> assets and test manually whether the UI work. Having
> > > end-to-end
> > > > > > >> tests
> > > > > > >>>> on
> > > > > > >>>>>> the front-end and making them run in the CI would speed up
> > > this
> > > > > > >> very
> > > > > > >>>>>> example (on top of increasing code robustness, increase our
> > > > > > >>> confidence
> > > > > > >>>>> etc).
> > > > > > >>>>>> On 2025/09/23 11:03:40 Rahul Vats wrote:
> > > > > > >>>>>>> Hi all,
> > > > > > >>>>>>>
> > > > > > >>>>>>> With Airflow 3.0, the UI has undergone a complete revamp,
> > and
> > > > new
> > > > > > >>>>> changes
> > > > > > >>>>>>> are naturally more prone to regressions. We've already seen
> > > > more
> > > > > > >> UI
> > > > > > >>>>>>> regressions that are hard to catch with our current limited
> > > > test
> > > > > > >>>>>> coverage.
> > > > > > >>>>>>> Manual testing is also challenging given the wide variety
> > of
> > > > > > >>>> workflows
> > > > > > >>>>>> and
> > > > > > >>>>>>> screens in the UI.
> > > > > > >>>>>>>
> > > > > > >>>>>>> Currently, the Airflow UI has a few unit tests using Vitest
> > > and
> > > > > > >>> React
> > > > > > >>>>>>> Testing Library, but coverage is quite limited — only
> > about 4
> > > > > > >> test
> > > > > > >>>>> files
> > > > > > >>>>>>> covering major pages like DagsList and TaskInstance. Many
> > > > > > >> critical
> > > > > > >>>>>>> pages (Connections,
> > > > > > >>>>>>> Variables, Pools, Dashboard, etc.) have no test coverage at
> > > > all.
> > > > > > >>>>>>>
> > > > > > >>>>>>> I’m exploring a two-pronged approach to improve UI testing:
> > > > > > >>>>>>>
> > > > > > >>>>>>> 1.
> > > > > > >>>>>>>
> > > > > > >>>>>>> Expand unit test coverage for individual components and
> > > > pages
> > > > > > >>>>>>>
> > > > > > >>>>>>> 2.
> > > > > > >>>>>>>
> > > > > > >>>>>>> Add end-to-end (E2E) tests using Playwright for
> > complete
> > > > user
> > > > > > >>>>>> workflows
> > > > > > >>>>>>>
> > > > > > >>>>>>> This would give us both component-level safety and full
> > > > > > >>> user-journey
> > > > > > >>>>>>> testing across browsers, helping catch regressions early.
> > > > > > >>>>>>>
> > > > > > >>>>>>> A few questions:
> > > > > > >>>>>>>
> > > > > > >>>>>>> -
> > > > > > >>>>>>>
> > > > > > >>>>>>> Should we prioritize expanding unit test coverage
> > first,
> > > or
> > > > > > >> work
> > > > > > >>>> on
> > > > > > >>>>>> unit
> > > > > > >>>>>>> and E2E tests in parallel?
> > > > > > >>>>>>>
> > > > > > >>>>>>> -
> > > > > > >>>>>>>
> > > > > > >>>>>>> For E2E testing, is Playwright a good choice, or are
> > > there
> > > > > > >> other
> > > > > > >>>>>>> preferred tools we should explore?
> > > > > > >>>>>>>
> > > > > > >>>>>>> -
> > > > > > >>>>>>>
> > > > > > >>>>>>> If Playwright, should we use TypeScript or Python as
> > > > language
> > > > > > >>> for
> > > > > > >>>>>> tests?
> > > > > > >>>>>>> -
> > > > > > >>>>>>>
> > > > > > >>>>>>> Any other suggestions to strengthen our UI testing
> > > > strategy?
> > > > > > >>>>>>>
> > > > > > >>>>>>>
> > > > > > >>>>>>> I’d love to hear your thoughts on whether this would be
> > > > valuable
> > > > > > >>> for
> > > > > > >>>>> the
> > > > > > >>>>>>> project before diving into implementation details.
> > > > > > >>>>>>>
> > > > > > >>>>>>> Regards,
> > > > > > >>>>>>> Rahul Vats
> > > > > > >>>>>>>
> > > > > > >>>>>>
> > > > > > >>
> > > > ---------------------------------------------------------------------
> > > > > > >>>>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > > > > >>>>>> For additional commands, e-mail:
> > dev-help@airflow.apache.org
> > > > > > >>>>>>
> > > > > > >>>>>>
> > > > > > >>
> > > > > >
> > > > > >
> > ---------------------------------------------------------------------
> > > > > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > > > > For additional commands, e-mail: dev-help@airflow.apache.org
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
For additional commands, e-mail: dev-help@airflow.apache.org
Re: [Discuss] Improving Airflow UI Testing Coverage
Posted by Rahul Vats <ra...@gmail.com>.
Hi Everyone,
Following up on our earlier discussions about UI E2E testing -- the initial
framework is now in place and ready for community contributions!
*What's Been Merged:*
- Playwright framework with TypeScript (#58548
<https://github.com/apache/airflow/pull/58548> )
- CI workflow integrated (#58901
<https://github.com/apache/airflow/pull/58901>)
- Initial test covering login and DAG triggering
*Meta Issue for Tracking:*
#59028 <https://github.com/apache/airflow/issues/59028>
*Sub-Issues Ready to Pick Up*:
- https://github.com/apache/airflow/issues/59306
- https://github.com/apache/airflow/issues/59307
- https://github.com/apache/airflow/issues/59308
- https://github.com/apache/airflow/issues/59309
- https://github.com/apache/airflow/issues/59357
- https://github.com/apache/airflow/issues/59358
More sub-issues will be created and linked to the meta issue #59028
<https://github.com/apache/airflow/issues/59028>
*Contribution Guide*
https://github.com/apache/airflow/blob/main/airflow-core/src/airflow/ui/tests/e2e/README.md#contributing-new-tests
*Slack Channel:*
Join #sig-airflow-ui-e2e-tests
<https://apache-airflow.slack.com/archives/C0A2QT7UUGP> for questions and
discussions.
Looking forward to contributions!
Regards,
Rahul Vats
On Wed, 15 Oct 2025 at 10:45, Amogh Desai <am...@apache.org> wrote:
> Thanks for the summary, Rahul.
>
> Sounds like a good plan, you could coordinate the work better either by
> using
> a Github Board or Slack channel or both too.
>
> Thanks & Regards,
> Amogh Desai
>
>
> On Fri, Oct 10, 2025 at 9:11 PM Rahul Vats <ra...@gmail.com> wrote:
>
> > Hi Everyone,
> >
> > Thank you all for the fantastic feedback and strong support! The
> community
> > consensus is clear and encouraging:
> >
> > - High priority for improving UI testing
> > - Playwright + TypeScript for e2e testing
> > - Parallel development of unit and e2e tests
> > - Test pyramid approach with focused e2e scope
> > - Multiple volunteers ready to contribute!
> >
> >
> > *Addressing Key Concerns*
> > Important points on e2e complexity are well taken. I completely agree
> that
> > we should follow the test pyramid.
> >
> > - E2E tests should cover only critical workflows (~10-15 scenarios)
> > - Unit tests are the foundation and should be the primary focus and we
> > should have an enforcement so that every UI change comes with a unit
> test.
> > - CI integration needs careful planning to avoid slowing development.
> >
> > *Next Steps*
> >
> > - I plan to work on a POC for E2E tests and will raise a PR for review
> > soon. After that, I'll share a detailed implementation plan and
> coordinate
> > with volunteers who offered to help.
> > - In the meantime, I suggest we should continue expanding unit test
> > coverage as planned.
> >
> >
> > It was great meeting you all at the Airflow Summit. Momentum here is
> > fantastic, and I'm excited to work with everyone who offered to help make
> > Airflow 3 UI much more reliable!
> >
> > Regards,
> > Rahul Vats
> >
> > On Thu, 25 Sept 2025 at 07:49, Vikram Koka via dev <
> dev@airflow.apache.org
> > >
> > wrote:
> >
> > > Rahul,
> > > Thank you for starting this thread.
> > >
> > > Strong +1 from me as well.
> > > I not only agree with this, but I believe this needs to be a high
> > priority
> > > going forward and before 3.2.
> > >
> > > I also agree with the comments on the "test pyramid" approach, of there
> > > needing to be FAR more unit tests as compared to end to end tests.
> > > I also believe that a certain level of rigor around this i.e. some form
> > of
> > > enforcement to ensure that all new UI changes come with a unit test.
> > >
> > > I would also add the Core API changes (for UI) to the mix here, since a
> > > fair number of UI changes requires underlying API changes.
> > >
> > > Vikram
> > >
> > >
> > > On Thu, Sep 25, 2025 at 12:04 AM Amogh Desai <am...@apache.org>
> > > wrote:
> > >
> > > > Valid points.
> > > >
> > > > I was just expressing concern from a CI/CD failure perspective but in
> > no
> > > > way
> > > > was I objecting to usage of typescript.
> > > >
> > > > Anything that works for the team that owns most of the tests works by
> > me!
> > > >
> > > >
> > > > Thanks & Regards,
> > > > Amogh Desai
> > > >
> > > >
> > > > On Thu, Sep 25, 2025 at 1:39 AM Jens Scheffler <js...@apache.org>
> > > > wrote:
> > > >
> > > > > Echoing a full agreement to all statement before.
> > > > >
> > > > > I also would emphasize the test pyramid. As we really have a low
> > > > > coverage we should maybe also have a metric. I would also propose
> > that
> > > > > we focus more on testing coverage == hopefully also improved
> quality
> > > > > preventing regression in 3.2. One option could a bit of force to
> > force
> > > > > all new changes applied to UI to request a unit test being added
> (all
> > > > > changes TS/x files need to have a corresponding .test.ts/x file)
> > except
> > > > > for cases where it does not make sense we allow exclusion of the
> rule
> > > by
> > > > > adding a label by maintainers. (just want to drop the idea, might
> be
> > > > > many will hate it)
> > > > >
> > > > > For language also have no strong preference but even though I am a
> TS
> > > > > noob I see that testing UI is closer to UI development, to using
> > Python
> > > > > for TS UI testing might make a knowledge gap also for potential
> > utility
> > > > > code. So as most voices would propose TS (maybe then finally I get
> to
> > > > > understand it)
> > > > >
> > > > > Jens
> > > > >
> > > > > On 24.09.25 18:33, Jarek Potiuk wrote:
> > > > > > Small poking at important things from holidays while having a
> > coffee
> > > > at a
> > > > > > place with internet :)
> > > > > >
> > > > > > +1 on having both. I have no strong opinion on the language and
> > > > > Playwright.
> > > > > >
> > > > > > But there are few things for consideration:
> > > > > >
> > > > > > We should have many, many more unit tests than e2e tests. This is
> > the
> > > > > > classic test pyramid case: The Practical Test Pyramid
> > > > > > https://share.google/YHVIt8EBptT1yYFur
> > > > > >
> > > > > > The e2e tests are slow, brittle and difficult to maintain and we
> > > should
> > > > > > have rather small number of those - mostly green path on all
> major
> > > > > > functionalities. Possibly with some cross browser checks - but
> > those
> > > > are
> > > > > > very difficult to get right.
> > > > > >
> > > > > > It will be a pain to maintain those. Like our k8s e2e tests -
> when
> > > they
> > > > > > break (rarely) people will have hard time to create and run the
> > tests
> > > > so
> > > > > > the tooling around that has to drive people who will be
> > reproducing
> > > > the
> > > > > > failures by hand - how to set it up and how to reproduce it and
> how
> > > to
> > > > > > iterate as fast as possible.
> > > > > >
> > > > > > Keeping the tests stable and side-effect free will be a
> challenge -
> > > so
> > > > > > likely a good 'reset the environment' before each test is a must.
> > > Also
> > > > > they
> > > > > > need to be run with all UI changes in CI which will make UI PR
> run
> > > > > longer.
> > > > > > It's an imperative to get them run not only in canary builds but
> in
> > > all
> > > > > PRs
> > > > > > for ui and API and likely whenever any core model change. It will
> > > slow
> > > > > down
> > > > > > UI development and create frustration for contributors unfamiliar
> > > with
> > > > > the
> > > > > > test framework so we need to have at least a few people
> proficient
> > in
> > > > > > intricacies of the test framework dedicated and willing to help
> > those
> > > > who
> > > > > > will struggle with issues.
> > > > > >
> > > > > > Language is secondary - e2e tests will require all stack - Python
> > > > > (backend)
> > > > > > and Typescript and react experiences anyway. Most likely people
> who
> > > > write
> > > > > > UI should be involved so Typescript is fine, and also even those
> > who
> > > do
> > > > > not
> > > > > > know TypeScript will be able to fix some obvious cases.
> > > > > >
> > > > > > J.
> > > > > >
> > > > > > śr., 24 wrz 2025, 09:50 użytkownik Shubham Raj <
> > > shubhamraj@apache.org>
> > > > > > napisał:
> > > > > >
> > > > > >> This addition will be really nice.
> > > > > >> Also, at this point of 3.x, I think UI has been relatively
> stable
> > > and
> > > > > >> writing e2e would be a good starting point. Unit tests can go
> > > > parallel.
> > > > > >> It's right that people maintaining CI/CD would be first persons
> to
> > > see
> > > > > any
> > > > > >> breakage, but I feel writing in ts will be easier for folks
> > working
> > > on
> > > > > >> frontend, and also for the new contributors who would like to
> > > > > contribute in
> > > > > >> this UI e2e tests coming from cypress or similar experience,
> since
> > > > > majority
> > > > > >> of other choices do come with ts/js support.
> > > > > >> I would be happy to help with the e2e tests. We should probably
> go
> > > in
> > > > > >> phases with P0/1/2.
> > > > > >>
> > > > > >> Thanks.
> > > > > >>
> > > > > >>
> > > > > >> On Wed, Sep 24, 2025 at 9:01 PM Pierre Jeambrun <
> > > > pierrejbrun@gmail.com>
> > > > > >> wrote:
> > > > > >>
> > > > > >>> I also think that having a good unit test coverage + e2e tests
> > > would
> > > > > >> really
> > > > > >>> help maintaining quality of releases and prevent regressions
> > early,
> > > > > >> making
> > > > > >>> all that much more manageable.
> > > > > >>>
> > > > > >>> My two cents regarding the questions above:
> > > > > >>>
> > > > > >>> -
> > > > > >>> Should we prioritize expanding unit test coverage first, or
> > > work
> > > > on
> > > > > >> unit
> > > > > >>> and E2E tests in parallel?
> > > > > >>>
> > > > > >>> Both in parallel sounds good to me. They do not depend on
> > each
> > > > > other
> > > > > >> (At
> > > > > >>> least in my mind)
> > > > > >>> -
> > > > > >>> For E2E testing, is Playwright a good choice, or are there
> > > other
> > > > > >>> preferred tools we should explore?
> > > > > >>>
> > > > > >>> I like playwright and I feel this is the modern solution to
> > go
> > > > to.
> > > > > >>> -
> > > > > >>> If Playwright, should we use TypeScript or Python as
> language
> > > for
> > > > > >> tests?
> > > > > >>> Good arguments mentioned above for both sides. I personally
> > > think
> > > > > >> (maybe
> > > > > >>> wrongly), that ideally people maintaining / developing e2e
> tests
> > > will
> > > > > be
> > > > > >>> people with
> > > > > >>> front-end knowledge. (events, identifiers, selectors, css
> > > > styling,
> > > > > >>> browser local states, etc..), so I tend to think that those
> > people
> > > > > will
> > > > > >>> most likely come from the front-end community and
> > > > > >>> having that in typescript would probably help them get
> > started.
> > > > > >>>
> > > > > >>>
> > > > > >>> On Wed, Sep 24, 2025 at 10:01 AM Amogh Desai <
> > > amoghdesai@apache.org>
> > > > > >>> wrote:
> > > > > >>>
> > > > > >>>> Thanks for starting this discussion, Rahul.
> > > > > >>>>
> > > > > >>>> I am aligned with your thoughts, we have seen many many UI
> > issues
> > > > > >> lately
> > > > > >>>> which I believe we should prioritize fixing or at least
> reducing
> > > > going
> > > > > >>>> forward.
> > > > > >>>>
> > > > > >>>> To answer your question(s) --
> > > > > >>>>
> > > > > >>>> * I think the unit tests and e2e / user flow tests as
> > orthogonal,
> > > so
> > > > > >> they
> > > > > >>>> can be
> > > > > >>>> developed in parallel.
> > > > > >>>>
> > > > > >>>> * While I do not have real experience with Playwright but from
> > > what
> > > > I
> > > > > >>> hear
> > > > > >>>> / read on reddit,
> > > > > >>>> github, and related forums, it looks like Playwright can do
> > > similar
> > > > > job
> > > > > >>> as
> > > > > >>>> alternatives such as
> > > > > >>>> Selenium, Cypress, while being more beginner friendly, which
> > could
> > > > > play
> > > > > >>>> better in our favour.
> > > > > >>>>
> > > > > >>>> * Again, not very strong on this but I feel that having
> > Typescript
> > > > as
> > > > > >> the
> > > > > >>>> language of choice would limit
> > > > > >>>> the number of contributors we get to help on this. Python
> would
> > be
> > > > my
> > > > > >>>> weapon of choice, given
> > > > > >>>> the "testing" is not limited by it and it potentially could
> > > > encourage
> > > > > >>>> broader participation.
> > > > > >>>>
> > > > > >>>> (Folks active on CI/CD(atleast me), are not very comfortable
> > with
> > > > > >>>> Typescript tests compared to
> > > > > >>>> Python)
> > > > > >>>>
> > > > > >>>> One additional point: It would be valuable if we extract the
> > e2e /
> > > > > user
> > > > > >>>> flow testing into a GH job that
> > > > > >>>> a release manager could run as "sort" of last validation
> before
> > > > > >> releasing
> > > > > >>>> to avoid any last
> > > > > >>>> minute issues.
> > > > > >>>>
> > > > > >>>> Thanks & Regards,
> > > > > >>>> Amogh Desai
> > > > > >>>>
> > > > > >>>>
> > > > > >>>> On Tue, Sep 23, 2025 at 9:15 PM Brent Bovenzi via dev <
> > > > > >>>> dev@airflow.apache.org> wrote:
> > > > > >>>>
> > > > > >>>>> I agree that we need to improve UI tests. Happy to make that
> a
> > > > > >> priority
> > > > > >>>>> during 3.1.x and 3.2 development.
> > > > > >>>>>
> > > > > >>>>> Personally, I will start with expanding unit tests first. But
> > > that
> > > > > >>>> doesn't
> > > > > >>>>> mean others can't get started on E2E tests.
> > > > > >>>>>
> > > > > >>>>> I lean towards using typescript for playwright. With E2E we
> are
> > > > > >> testing
> > > > > >>>> the
> > > > > >>>>> UI more than we are testing the API so I think the code
> should
> > > sit
> > > > > >>> closer
> > > > > >>>>> to the UI code.
> > > > > >>>>>
> > > > > >>>>> On Tue, Sep 23, 2025 at 9:38 AM Vincent Beck <
> > > vincbeck@apache.org>
> > > > > >>>> wrote:
> > > > > >>>>>> Not answering any of these questions (sorry) but strong +1
> on
> > > > > >> adding
> > > > > >>>>>> end-to-end (E2E) tests. This will significantly make our UI
> > more
> > > > > >>> robust
> > > > > >>>>> but
> > > > > >>>>>> also will speed up some of our work. An example is front-end
> > > > > >>> dependency
> > > > > >>>>>> upgrades by dependabot. Today we have no way of knowing
> > whether
> > > > > >> these
> > > > > >>>> PRs
> > > > > >>>>>> break the UI, in order to merge these PRs we need to compile
> > the
> > > > > >>>>> front-end
> > > > > >>>>>> assets and test manually whether the UI work. Having
> > end-to-end
> > > > > >> tests
> > > > > >>>> on
> > > > > >>>>>> the front-end and making them run in the CI would speed up
> > this
> > > > > >> very
> > > > > >>>>>> example (on top of increasing code robustness, increase our
> > > > > >>> confidence
> > > > > >>>>> etc).
> > > > > >>>>>> On 2025/09/23 11:03:40 Rahul Vats wrote:
> > > > > >>>>>>> Hi all,
> > > > > >>>>>>>
> > > > > >>>>>>> With Airflow 3.0, the UI has undergone a complete revamp,
> and
> > > new
> > > > > >>>>> changes
> > > > > >>>>>>> are naturally more prone to regressions. We've already seen
> > > more
> > > > > >> UI
> > > > > >>>>>>> regressions that are hard to catch with our current limited
> > > test
> > > > > >>>>>> coverage.
> > > > > >>>>>>> Manual testing is also challenging given the wide variety
> of
> > > > > >>>> workflows
> > > > > >>>>>> and
> > > > > >>>>>>> screens in the UI.
> > > > > >>>>>>>
> > > > > >>>>>>> Currently, the Airflow UI has a few unit tests using Vitest
> > and
> > > > > >>> React
> > > > > >>>>>>> Testing Library, but coverage is quite limited — only
> about 4
> > > > > >> test
> > > > > >>>>> files
> > > > > >>>>>>> covering major pages like DagsList and TaskInstance. Many
> > > > > >> critical
> > > > > >>>>>>> pages (Connections,
> > > > > >>>>>>> Variables, Pools, Dashboard, etc.) have no test coverage at
> > > all.
> > > > > >>>>>>>
> > > > > >>>>>>> I’m exploring a two-pronged approach to improve UI testing:
> > > > > >>>>>>>
> > > > > >>>>>>> 1.
> > > > > >>>>>>>
> > > > > >>>>>>> Expand unit test coverage for individual components and
> > > pages
> > > > > >>>>>>>
> > > > > >>>>>>> 2.
> > > > > >>>>>>>
> > > > > >>>>>>> Add end-to-end (E2E) tests using Playwright for
> complete
> > > user
> > > > > >>>>>> workflows
> > > > > >>>>>>>
> > > > > >>>>>>> This would give us both component-level safety and full
> > > > > >>> user-journey
> > > > > >>>>>>> testing across browsers, helping catch regressions early.
> > > > > >>>>>>>
> > > > > >>>>>>> A few questions:
> > > > > >>>>>>>
> > > > > >>>>>>> -
> > > > > >>>>>>>
> > > > > >>>>>>> Should we prioritize expanding unit test coverage
> first,
> > or
> > > > > >> work
> > > > > >>>> on
> > > > > >>>>>> unit
> > > > > >>>>>>> and E2E tests in parallel?
> > > > > >>>>>>>
> > > > > >>>>>>> -
> > > > > >>>>>>>
> > > > > >>>>>>> For E2E testing, is Playwright a good choice, or are
> > there
> > > > > >> other
> > > > > >>>>>>> preferred tools we should explore?
> > > > > >>>>>>>
> > > > > >>>>>>> -
> > > > > >>>>>>>
> > > > > >>>>>>> If Playwright, should we use TypeScript or Python as
> > > language
> > > > > >>> for
> > > > > >>>>>> tests?
> > > > > >>>>>>> -
> > > > > >>>>>>>
> > > > > >>>>>>> Any other suggestions to strengthen our UI testing
> > > strategy?
> > > > > >>>>>>>
> > > > > >>>>>>>
> > > > > >>>>>>> I’d love to hear your thoughts on whether this would be
> > > valuable
> > > > > >>> for
> > > > > >>>>> the
> > > > > >>>>>>> project before diving into implementation details.
> > > > > >>>>>>>
> > > > > >>>>>>> Regards,
> > > > > >>>>>>> Rahul Vats
> > > > > >>>>>>>
> > > > > >>>>>>
> > > > > >>
> > > ---------------------------------------------------------------------
> > > > > >>>>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > > > >>>>>> For additional commands, e-mail:
> dev-help@airflow.apache.org
> > > > > >>>>>>
> > > > > >>>>>>
> > > > > >>
> > > > >
> > > > >
> ---------------------------------------------------------------------
> > > > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > > > For additional commands, e-mail: dev-help@airflow.apache.org
> > > > >
> > > > >
> > > >
> > >
> >
>
Re: [Discuss] Improving Airflow UI Testing Coverage
Posted by Amogh Desai <am...@apache.org>.
Thanks for the summary, Rahul.
Sounds like a good plan, you could coordinate the work better either by
using
a Github Board or Slack channel or both too.
Thanks & Regards,
Amogh Desai
On Fri, Oct 10, 2025 at 9:11 PM Rahul Vats <ra...@gmail.com> wrote:
> Hi Everyone,
>
> Thank you all for the fantastic feedback and strong support! The community
> consensus is clear and encouraging:
>
> - High priority for improving UI testing
> - Playwright + TypeScript for e2e testing
> - Parallel development of unit and e2e tests
> - Test pyramid approach with focused e2e scope
> - Multiple volunteers ready to contribute!
>
>
> *Addressing Key Concerns*
> Important points on e2e complexity are well taken. I completely agree that
> we should follow the test pyramid.
>
> - E2E tests should cover only critical workflows (~10-15 scenarios)
> - Unit tests are the foundation and should be the primary focus and we
> should have an enforcement so that every UI change comes with a unit test.
> - CI integration needs careful planning to avoid slowing development.
>
> *Next Steps*
>
> - I plan to work on a POC for E2E tests and will raise a PR for review
> soon. After that, I'll share a detailed implementation plan and coordinate
> with volunteers who offered to help.
> - In the meantime, I suggest we should continue expanding unit test
> coverage as planned.
>
>
> It was great meeting you all at the Airflow Summit. Momentum here is
> fantastic, and I'm excited to work with everyone who offered to help make
> Airflow 3 UI much more reliable!
>
> Regards,
> Rahul Vats
>
> On Thu, 25 Sept 2025 at 07:49, Vikram Koka via dev <dev@airflow.apache.org
> >
> wrote:
>
> > Rahul,
> > Thank you for starting this thread.
> >
> > Strong +1 from me as well.
> > I not only agree with this, but I believe this needs to be a high
> priority
> > going forward and before 3.2.
> >
> > I also agree with the comments on the "test pyramid" approach, of there
> > needing to be FAR more unit tests as compared to end to end tests.
> > I also believe that a certain level of rigor around this i.e. some form
> of
> > enforcement to ensure that all new UI changes come with a unit test.
> >
> > I would also add the Core API changes (for UI) to the mix here, since a
> > fair number of UI changes requires underlying API changes.
> >
> > Vikram
> >
> >
> > On Thu, Sep 25, 2025 at 12:04 AM Amogh Desai <am...@apache.org>
> > wrote:
> >
> > > Valid points.
> > >
> > > I was just expressing concern from a CI/CD failure perspective but in
> no
> > > way
> > > was I objecting to usage of typescript.
> > >
> > > Anything that works for the team that owns most of the tests works by
> me!
> > >
> > >
> > > Thanks & Regards,
> > > Amogh Desai
> > >
> > >
> > > On Thu, Sep 25, 2025 at 1:39 AM Jens Scheffler <js...@apache.org>
> > > wrote:
> > >
> > > > Echoing a full agreement to all statement before.
> > > >
> > > > I also would emphasize the test pyramid. As we really have a low
> > > > coverage we should maybe also have a metric. I would also propose
> that
> > > > we focus more on testing coverage == hopefully also improved quality
> > > > preventing regression in 3.2. One option could a bit of force to
> force
> > > > all new changes applied to UI to request a unit test being added (all
> > > > changes TS/x files need to have a corresponding .test.ts/x file)
> except
> > > > for cases where it does not make sense we allow exclusion of the rule
> > by
> > > > adding a label by maintainers. (just want to drop the idea, might be
> > > > many will hate it)
> > > >
> > > > For language also have no strong preference but even though I am a TS
> > > > noob I see that testing UI is closer to UI development, to using
> Python
> > > > for TS UI testing might make a knowledge gap also for potential
> utility
> > > > code. So as most voices would propose TS (maybe then finally I get to
> > > > understand it)
> > > >
> > > > Jens
> > > >
> > > > On 24.09.25 18:33, Jarek Potiuk wrote:
> > > > > Small poking at important things from holidays while having a
> coffee
> > > at a
> > > > > place with internet :)
> > > > >
> > > > > +1 on having both. I have no strong opinion on the language and
> > > > Playwright.
> > > > >
> > > > > But there are few things for consideration:
> > > > >
> > > > > We should have many, many more unit tests than e2e tests. This is
> the
> > > > > classic test pyramid case: The Practical Test Pyramid
> > > > > https://share.google/YHVIt8EBptT1yYFur
> > > > >
> > > > > The e2e tests are slow, brittle and difficult to maintain and we
> > should
> > > > > have rather small number of those - mostly green path on all major
> > > > > functionalities. Possibly with some cross browser checks - but
> those
> > > are
> > > > > very difficult to get right.
> > > > >
> > > > > It will be a pain to maintain those. Like our k8s e2e tests - when
> > they
> > > > > break (rarely) people will have hard time to create and run the
> tests
> > > so
> > > > > the tooling around that has to drive people who will be
> reproducing
> > > the
> > > > > failures by hand - how to set it up and how to reproduce it and how
> > to
> > > > > iterate as fast as possible.
> > > > >
> > > > > Keeping the tests stable and side-effect free will be a challenge -
> > so
> > > > > likely a good 'reset the environment' before each test is a must.
> > Also
> > > > they
> > > > > need to be run with all UI changes in CI which will make UI PR run
> > > > longer.
> > > > > It's an imperative to get them run not only in canary builds but in
> > all
> > > > PRs
> > > > > for ui and API and likely whenever any core model change. It will
> > slow
> > > > down
> > > > > UI development and create frustration for contributors unfamiliar
> > with
> > > > the
> > > > > test framework so we need to have at least a few people proficient
> in
> > > > > intricacies of the test framework dedicated and willing to help
> those
> > > who
> > > > > will struggle with issues.
> > > > >
> > > > > Language is secondary - e2e tests will require all stack - Python
> > > > (backend)
> > > > > and Typescript and react experiences anyway. Most likely people who
> > > write
> > > > > UI should be involved so Typescript is fine, and also even those
> who
> > do
> > > > not
> > > > > know TypeScript will be able to fix some obvious cases.
> > > > >
> > > > > J.
> > > > >
> > > > > śr., 24 wrz 2025, 09:50 użytkownik Shubham Raj <
> > shubhamraj@apache.org>
> > > > > napisał:
> > > > >
> > > > >> This addition will be really nice.
> > > > >> Also, at this point of 3.x, I think UI has been relatively stable
> > and
> > > > >> writing e2e would be a good starting point. Unit tests can go
> > > parallel.
> > > > >> It's right that people maintaining CI/CD would be first persons to
> > see
> > > > any
> > > > >> breakage, but I feel writing in ts will be easier for folks
> working
> > on
> > > > >> frontend, and also for the new contributors who would like to
> > > > contribute in
> > > > >> this UI e2e tests coming from cypress or similar experience, since
> > > > majority
> > > > >> of other choices do come with ts/js support.
> > > > >> I would be happy to help with the e2e tests. We should probably go
> > in
> > > > >> phases with P0/1/2.
> > > > >>
> > > > >> Thanks.
> > > > >>
> > > > >>
> > > > >> On Wed, Sep 24, 2025 at 9:01 PM Pierre Jeambrun <
> > > pierrejbrun@gmail.com>
> > > > >> wrote:
> > > > >>
> > > > >>> I also think that having a good unit test coverage + e2e tests
> > would
> > > > >> really
> > > > >>> help maintaining quality of releases and prevent regressions
> early,
> > > > >> making
> > > > >>> all that much more manageable.
> > > > >>>
> > > > >>> My two cents regarding the questions above:
> > > > >>>
> > > > >>> -
> > > > >>> Should we prioritize expanding unit test coverage first, or
> > work
> > > on
> > > > >> unit
> > > > >>> and E2E tests in parallel?
> > > > >>>
> > > > >>> Both in parallel sounds good to me. They do not depend on
> each
> > > > other
> > > > >> (At
> > > > >>> least in my mind)
> > > > >>> -
> > > > >>> For E2E testing, is Playwright a good choice, or are there
> > other
> > > > >>> preferred tools we should explore?
> > > > >>>
> > > > >>> I like playwright and I feel this is the modern solution to
> go
> > > to.
> > > > >>> -
> > > > >>> If Playwright, should we use TypeScript or Python as language
> > for
> > > > >> tests?
> > > > >>> Good arguments mentioned above for both sides. I personally
> > think
> > > > >> (maybe
> > > > >>> wrongly), that ideally people maintaining / developing e2e tests
> > will
> > > > be
> > > > >>> people with
> > > > >>> front-end knowledge. (events, identifiers, selectors, css
> > > styling,
> > > > >>> browser local states, etc..), so I tend to think that those
> people
> > > > will
> > > > >>> most likely come from the front-end community and
> > > > >>> having that in typescript would probably help them get
> started.
> > > > >>>
> > > > >>>
> > > > >>> On Wed, Sep 24, 2025 at 10:01 AM Amogh Desai <
> > amoghdesai@apache.org>
> > > > >>> wrote:
> > > > >>>
> > > > >>>> Thanks for starting this discussion, Rahul.
> > > > >>>>
> > > > >>>> I am aligned with your thoughts, we have seen many many UI
> issues
> > > > >> lately
> > > > >>>> which I believe we should prioritize fixing or at least reducing
> > > going
> > > > >>>> forward.
> > > > >>>>
> > > > >>>> To answer your question(s) --
> > > > >>>>
> > > > >>>> * I think the unit tests and e2e / user flow tests as
> orthogonal,
> > so
> > > > >> they
> > > > >>>> can be
> > > > >>>> developed in parallel.
> > > > >>>>
> > > > >>>> * While I do not have real experience with Playwright but from
> > what
> > > I
> > > > >>> hear
> > > > >>>> / read on reddit,
> > > > >>>> github, and related forums, it looks like Playwright can do
> > similar
> > > > job
> > > > >>> as
> > > > >>>> alternatives such as
> > > > >>>> Selenium, Cypress, while being more beginner friendly, which
> could
> > > > play
> > > > >>>> better in our favour.
> > > > >>>>
> > > > >>>> * Again, not very strong on this but I feel that having
> Typescript
> > > as
> > > > >> the
> > > > >>>> language of choice would limit
> > > > >>>> the number of contributors we get to help on this. Python would
> be
> > > my
> > > > >>>> weapon of choice, given
> > > > >>>> the "testing" is not limited by it and it potentially could
> > > encourage
> > > > >>>> broader participation.
> > > > >>>>
> > > > >>>> (Folks active on CI/CD(atleast me), are not very comfortable
> with
> > > > >>>> Typescript tests compared to
> > > > >>>> Python)
> > > > >>>>
> > > > >>>> One additional point: It would be valuable if we extract the
> e2e /
> > > > user
> > > > >>>> flow testing into a GH job that
> > > > >>>> a release manager could run as "sort" of last validation before
> > > > >> releasing
> > > > >>>> to avoid any last
> > > > >>>> minute issues.
> > > > >>>>
> > > > >>>> Thanks & Regards,
> > > > >>>> Amogh Desai
> > > > >>>>
> > > > >>>>
> > > > >>>> On Tue, Sep 23, 2025 at 9:15 PM Brent Bovenzi via dev <
> > > > >>>> dev@airflow.apache.org> wrote:
> > > > >>>>
> > > > >>>>> I agree that we need to improve UI tests. Happy to make that a
> > > > >> priority
> > > > >>>>> during 3.1.x and 3.2 development.
> > > > >>>>>
> > > > >>>>> Personally, I will start with expanding unit tests first. But
> > that
> > > > >>>> doesn't
> > > > >>>>> mean others can't get started on E2E tests.
> > > > >>>>>
> > > > >>>>> I lean towards using typescript for playwright. With E2E we are
> > > > >> testing
> > > > >>>> the
> > > > >>>>> UI more than we are testing the API so I think the code should
> > sit
> > > > >>> closer
> > > > >>>>> to the UI code.
> > > > >>>>>
> > > > >>>>> On Tue, Sep 23, 2025 at 9:38 AM Vincent Beck <
> > vincbeck@apache.org>
> > > > >>>> wrote:
> > > > >>>>>> Not answering any of these questions (sorry) but strong +1 on
> > > > >> adding
> > > > >>>>>> end-to-end (E2E) tests. This will significantly make our UI
> more
> > > > >>> robust
> > > > >>>>> but
> > > > >>>>>> also will speed up some of our work. An example is front-end
> > > > >>> dependency
> > > > >>>>>> upgrades by dependabot. Today we have no way of knowing
> whether
> > > > >> these
> > > > >>>> PRs
> > > > >>>>>> break the UI, in order to merge these PRs we need to compile
> the
> > > > >>>>> front-end
> > > > >>>>>> assets and test manually whether the UI work. Having
> end-to-end
> > > > >> tests
> > > > >>>> on
> > > > >>>>>> the front-end and making them run in the CI would speed up
> this
> > > > >> very
> > > > >>>>>> example (on top of increasing code robustness, increase our
> > > > >>> confidence
> > > > >>>>> etc).
> > > > >>>>>> On 2025/09/23 11:03:40 Rahul Vats wrote:
> > > > >>>>>>> Hi all,
> > > > >>>>>>>
> > > > >>>>>>> With Airflow 3.0, the UI has undergone a complete revamp, and
> > new
> > > > >>>>> changes
> > > > >>>>>>> are naturally more prone to regressions. We've already seen
> > more
> > > > >> UI
> > > > >>>>>>> regressions that are hard to catch with our current limited
> > test
> > > > >>>>>> coverage.
> > > > >>>>>>> Manual testing is also challenging given the wide variety of
> > > > >>>> workflows
> > > > >>>>>> and
> > > > >>>>>>> screens in the UI.
> > > > >>>>>>>
> > > > >>>>>>> Currently, the Airflow UI has a few unit tests using Vitest
> and
> > > > >>> React
> > > > >>>>>>> Testing Library, but coverage is quite limited — only about 4
> > > > >> test
> > > > >>>>> files
> > > > >>>>>>> covering major pages like DagsList and TaskInstance. Many
> > > > >> critical
> > > > >>>>>>> pages (Connections,
> > > > >>>>>>> Variables, Pools, Dashboard, etc.) have no test coverage at
> > all.
> > > > >>>>>>>
> > > > >>>>>>> I’m exploring a two-pronged approach to improve UI testing:
> > > > >>>>>>>
> > > > >>>>>>> 1.
> > > > >>>>>>>
> > > > >>>>>>> Expand unit test coverage for individual components and
> > pages
> > > > >>>>>>>
> > > > >>>>>>> 2.
> > > > >>>>>>>
> > > > >>>>>>> Add end-to-end (E2E) tests using Playwright for complete
> > user
> > > > >>>>>> workflows
> > > > >>>>>>>
> > > > >>>>>>> This would give us both component-level safety and full
> > > > >>> user-journey
> > > > >>>>>>> testing across browsers, helping catch regressions early.
> > > > >>>>>>>
> > > > >>>>>>> A few questions:
> > > > >>>>>>>
> > > > >>>>>>> -
> > > > >>>>>>>
> > > > >>>>>>> Should we prioritize expanding unit test coverage first,
> or
> > > > >> work
> > > > >>>> on
> > > > >>>>>> unit
> > > > >>>>>>> and E2E tests in parallel?
> > > > >>>>>>>
> > > > >>>>>>> -
> > > > >>>>>>>
> > > > >>>>>>> For E2E testing, is Playwright a good choice, or are
> there
> > > > >> other
> > > > >>>>>>> preferred tools we should explore?
> > > > >>>>>>>
> > > > >>>>>>> -
> > > > >>>>>>>
> > > > >>>>>>> If Playwright, should we use TypeScript or Python as
> > language
> > > > >>> for
> > > > >>>>>> tests?
> > > > >>>>>>> -
> > > > >>>>>>>
> > > > >>>>>>> Any other suggestions to strengthen our UI testing
> > strategy?
> > > > >>>>>>>
> > > > >>>>>>>
> > > > >>>>>>> I’d love to hear your thoughts on whether this would be
> > valuable
> > > > >>> for
> > > > >>>>> the
> > > > >>>>>>> project before diving into implementation details.
> > > > >>>>>>>
> > > > >>>>>>> Regards,
> > > > >>>>>>> Rahul Vats
> > > > >>>>>>>
> > > > >>>>>>
> > > > >>
> > ---------------------------------------------------------------------
> > > > >>>>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > > >>>>>> For additional commands, e-mail: dev-help@airflow.apache.org
> > > > >>>>>>
> > > > >>>>>>
> > > > >>
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > > For additional commands, e-mail: dev-help@airflow.apache.org
> > > >
> > > >
> > >
> >
>
Re: [Discuss] Improving Airflow UI Testing Coverage
Posted by Rahul Vats <ra...@gmail.com>.
Hi Everyone,
Thank you all for the fantastic feedback and strong support! The community
consensus is clear and encouraging:
- High priority for improving UI testing
- Playwright + TypeScript for e2e testing
- Parallel development of unit and e2e tests
- Test pyramid approach with focused e2e scope
- Multiple volunteers ready to contribute!
*Addressing Key Concerns*
Important points on e2e complexity are well taken. I completely agree that
we should follow the test pyramid.
- E2E tests should cover only critical workflows (~10-15 scenarios)
- Unit tests are the foundation and should be the primary focus and we
should have an enforcement so that every UI change comes with a unit test.
- CI integration needs careful planning to avoid slowing development.
*Next Steps*
- I plan to work on a POC for E2E tests and will raise a PR for review
soon. After that, I'll share a detailed implementation plan and coordinate
with volunteers who offered to help.
- In the meantime, I suggest we should continue expanding unit test
coverage as planned.
It was great meeting you all at the Airflow Summit. Momentum here is
fantastic, and I'm excited to work with everyone who offered to help make
Airflow 3 UI much more reliable!
Regards,
Rahul Vats
On Thu, 25 Sept 2025 at 07:49, Vikram Koka via dev <de...@airflow.apache.org>
wrote:
> Rahul,
> Thank you for starting this thread.
>
> Strong +1 from me as well.
> I not only agree with this, but I believe this needs to be a high priority
> going forward and before 3.2.
>
> I also agree with the comments on the "test pyramid" approach, of there
> needing to be FAR more unit tests as compared to end to end tests.
> I also believe that a certain level of rigor around this i.e. some form of
> enforcement to ensure that all new UI changes come with a unit test.
>
> I would also add the Core API changes (for UI) to the mix here, since a
> fair number of UI changes requires underlying API changes.
>
> Vikram
>
>
> On Thu, Sep 25, 2025 at 12:04 AM Amogh Desai <am...@apache.org>
> wrote:
>
> > Valid points.
> >
> > I was just expressing concern from a CI/CD failure perspective but in no
> > way
> > was I objecting to usage of typescript.
> >
> > Anything that works for the team that owns most of the tests works by me!
> >
> >
> > Thanks & Regards,
> > Amogh Desai
> >
> >
> > On Thu, Sep 25, 2025 at 1:39 AM Jens Scheffler <js...@apache.org>
> > wrote:
> >
> > > Echoing a full agreement to all statement before.
> > >
> > > I also would emphasize the test pyramid. As we really have a low
> > > coverage we should maybe also have a metric. I would also propose that
> > > we focus more on testing coverage == hopefully also improved quality
> > > preventing regression in 3.2. One option could a bit of force to force
> > > all new changes applied to UI to request a unit test being added (all
> > > changes TS/x files need to have a corresponding .test.ts/x file) except
> > > for cases where it does not make sense we allow exclusion of the rule
> by
> > > adding a label by maintainers. (just want to drop the idea, might be
> > > many will hate it)
> > >
> > > For language also have no strong preference but even though I am a TS
> > > noob I see that testing UI is closer to UI development, to using Python
> > > for TS UI testing might make a knowledge gap also for potential utility
> > > code. So as most voices would propose TS (maybe then finally I get to
> > > understand it)
> > >
> > > Jens
> > >
> > > On 24.09.25 18:33, Jarek Potiuk wrote:
> > > > Small poking at important things from holidays while having a coffee
> > at a
> > > > place with internet :)
> > > >
> > > > +1 on having both. I have no strong opinion on the language and
> > > Playwright.
> > > >
> > > > But there are few things for consideration:
> > > >
> > > > We should have many, many more unit tests than e2e tests. This is the
> > > > classic test pyramid case: The Practical Test Pyramid
> > > > https://share.google/YHVIt8EBptT1yYFur
> > > >
> > > > The e2e tests are slow, brittle and difficult to maintain and we
> should
> > > > have rather small number of those - mostly green path on all major
> > > > functionalities. Possibly with some cross browser checks - but those
> > are
> > > > very difficult to get right.
> > > >
> > > > It will be a pain to maintain those. Like our k8s e2e tests - when
> they
> > > > break (rarely) people will have hard time to create and run the tests
> > so
> > > > the tooling around that has to drive people who will be reproducing
> > the
> > > > failures by hand - how to set it up and how to reproduce it and how
> to
> > > > iterate as fast as possible.
> > > >
> > > > Keeping the tests stable and side-effect free will be a challenge -
> so
> > > > likely a good 'reset the environment' before each test is a must.
> Also
> > > they
> > > > need to be run with all UI changes in CI which will make UI PR run
> > > longer.
> > > > It's an imperative to get them run not only in canary builds but in
> all
> > > PRs
> > > > for ui and API and likely whenever any core model change. It will
> slow
> > > down
> > > > UI development and create frustration for contributors unfamiliar
> with
> > > the
> > > > test framework so we need to have at least a few people proficient in
> > > > intricacies of the test framework dedicated and willing to help those
> > who
> > > > will struggle with issues.
> > > >
> > > > Language is secondary - e2e tests will require all stack - Python
> > > (backend)
> > > > and Typescript and react experiences anyway. Most likely people who
> > write
> > > > UI should be involved so Typescript is fine, and also even those who
> do
> > > not
> > > > know TypeScript will be able to fix some obvious cases.
> > > >
> > > > J.
> > > >
> > > > śr., 24 wrz 2025, 09:50 użytkownik Shubham Raj <
> shubhamraj@apache.org>
> > > > napisał:
> > > >
> > > >> This addition will be really nice.
> > > >> Also, at this point of 3.x, I think UI has been relatively stable
> and
> > > >> writing e2e would be a good starting point. Unit tests can go
> > parallel.
> > > >> It's right that people maintaining CI/CD would be first persons to
> see
> > > any
> > > >> breakage, but I feel writing in ts will be easier for folks working
> on
> > > >> frontend, and also for the new contributors who would like to
> > > contribute in
> > > >> this UI e2e tests coming from cypress or similar experience, since
> > > majority
> > > >> of other choices do come with ts/js support.
> > > >> I would be happy to help with the e2e tests. We should probably go
> in
> > > >> phases with P0/1/2.
> > > >>
> > > >> Thanks.
> > > >>
> > > >>
> > > >> On Wed, Sep 24, 2025 at 9:01 PM Pierre Jeambrun <
> > pierrejbrun@gmail.com>
> > > >> wrote:
> > > >>
> > > >>> I also think that having a good unit test coverage + e2e tests
> would
> > > >> really
> > > >>> help maintaining quality of releases and prevent regressions early,
> > > >> making
> > > >>> all that much more manageable.
> > > >>>
> > > >>> My two cents regarding the questions above:
> > > >>>
> > > >>> -
> > > >>> Should we prioritize expanding unit test coverage first, or
> work
> > on
> > > >> unit
> > > >>> and E2E tests in parallel?
> > > >>>
> > > >>> Both in parallel sounds good to me. They do not depend on each
> > > other
> > > >> (At
> > > >>> least in my mind)
> > > >>> -
> > > >>> For E2E testing, is Playwright a good choice, or are there
> other
> > > >>> preferred tools we should explore?
> > > >>>
> > > >>> I like playwright and I feel this is the modern solution to go
> > to.
> > > >>> -
> > > >>> If Playwright, should we use TypeScript or Python as language
> for
> > > >> tests?
> > > >>> Good arguments mentioned above for both sides. I personally
> think
> > > >> (maybe
> > > >>> wrongly), that ideally people maintaining / developing e2e tests
> will
> > > be
> > > >>> people with
> > > >>> front-end knowledge. (events, identifiers, selectors, css
> > styling,
> > > >>> browser local states, etc..), so I tend to think that those people
> > > will
> > > >>> most likely come from the front-end community and
> > > >>> having that in typescript would probably help them get started.
> > > >>>
> > > >>>
> > > >>> On Wed, Sep 24, 2025 at 10:01 AM Amogh Desai <
> amoghdesai@apache.org>
> > > >>> wrote:
> > > >>>
> > > >>>> Thanks for starting this discussion, Rahul.
> > > >>>>
> > > >>>> I am aligned with your thoughts, we have seen many many UI issues
> > > >> lately
> > > >>>> which I believe we should prioritize fixing or at least reducing
> > going
> > > >>>> forward.
> > > >>>>
> > > >>>> To answer your question(s) --
> > > >>>>
> > > >>>> * I think the unit tests and e2e / user flow tests as orthogonal,
> so
> > > >> they
> > > >>>> can be
> > > >>>> developed in parallel.
> > > >>>>
> > > >>>> * While I do not have real experience with Playwright but from
> what
> > I
> > > >>> hear
> > > >>>> / read on reddit,
> > > >>>> github, and related forums, it looks like Playwright can do
> similar
> > > job
> > > >>> as
> > > >>>> alternatives such as
> > > >>>> Selenium, Cypress, while being more beginner friendly, which could
> > > play
> > > >>>> better in our favour.
> > > >>>>
> > > >>>> * Again, not very strong on this but I feel that having Typescript
> > as
> > > >> the
> > > >>>> language of choice would limit
> > > >>>> the number of contributors we get to help on this. Python would be
> > my
> > > >>>> weapon of choice, given
> > > >>>> the "testing" is not limited by it and it potentially could
> > encourage
> > > >>>> broader participation.
> > > >>>>
> > > >>>> (Folks active on CI/CD(atleast me), are not very comfortable with
> > > >>>> Typescript tests compared to
> > > >>>> Python)
> > > >>>>
> > > >>>> One additional point: It would be valuable if we extract the e2e /
> > > user
> > > >>>> flow testing into a GH job that
> > > >>>> a release manager could run as "sort" of last validation before
> > > >> releasing
> > > >>>> to avoid any last
> > > >>>> minute issues.
> > > >>>>
> > > >>>> Thanks & Regards,
> > > >>>> Amogh Desai
> > > >>>>
> > > >>>>
> > > >>>> On Tue, Sep 23, 2025 at 9:15 PM Brent Bovenzi via dev <
> > > >>>> dev@airflow.apache.org> wrote:
> > > >>>>
> > > >>>>> I agree that we need to improve UI tests. Happy to make that a
> > > >> priority
> > > >>>>> during 3.1.x and 3.2 development.
> > > >>>>>
> > > >>>>> Personally, I will start with expanding unit tests first. But
> that
> > > >>>> doesn't
> > > >>>>> mean others can't get started on E2E tests.
> > > >>>>>
> > > >>>>> I lean towards using typescript for playwright. With E2E we are
> > > >> testing
> > > >>>> the
> > > >>>>> UI more than we are testing the API so I think the code should
> sit
> > > >>> closer
> > > >>>>> to the UI code.
> > > >>>>>
> > > >>>>> On Tue, Sep 23, 2025 at 9:38 AM Vincent Beck <
> vincbeck@apache.org>
> > > >>>> wrote:
> > > >>>>>> Not answering any of these questions (sorry) but strong +1 on
> > > >> adding
> > > >>>>>> end-to-end (E2E) tests. This will significantly make our UI more
> > > >>> robust
> > > >>>>> but
> > > >>>>>> also will speed up some of our work. An example is front-end
> > > >>> dependency
> > > >>>>>> upgrades by dependabot. Today we have no way of knowing whether
> > > >> these
> > > >>>> PRs
> > > >>>>>> break the UI, in order to merge these PRs we need to compile the
> > > >>>>> front-end
> > > >>>>>> assets and test manually whether the UI work. Having end-to-end
> > > >> tests
> > > >>>> on
> > > >>>>>> the front-end and making them run in the CI would speed up this
> > > >> very
> > > >>>>>> example (on top of increasing code robustness, increase our
> > > >>> confidence
> > > >>>>> etc).
> > > >>>>>> On 2025/09/23 11:03:40 Rahul Vats wrote:
> > > >>>>>>> Hi all,
> > > >>>>>>>
> > > >>>>>>> With Airflow 3.0, the UI has undergone a complete revamp, and
> new
> > > >>>>> changes
> > > >>>>>>> are naturally more prone to regressions. We've already seen
> more
> > > >> UI
> > > >>>>>>> regressions that are hard to catch with our current limited
> test
> > > >>>>>> coverage.
> > > >>>>>>> Manual testing is also challenging given the wide variety of
> > > >>>> workflows
> > > >>>>>> and
> > > >>>>>>> screens in the UI.
> > > >>>>>>>
> > > >>>>>>> Currently, the Airflow UI has a few unit tests using Vitest and
> > > >>> React
> > > >>>>>>> Testing Library, but coverage is quite limited — only about 4
> > > >> test
> > > >>>>> files
> > > >>>>>>> covering major pages like DagsList and TaskInstance. Many
> > > >> critical
> > > >>>>>>> pages (Connections,
> > > >>>>>>> Variables, Pools, Dashboard, etc.) have no test coverage at
> all.
> > > >>>>>>>
> > > >>>>>>> I’m exploring a two-pronged approach to improve UI testing:
> > > >>>>>>>
> > > >>>>>>> 1.
> > > >>>>>>>
> > > >>>>>>> Expand unit test coverage for individual components and
> pages
> > > >>>>>>>
> > > >>>>>>> 2.
> > > >>>>>>>
> > > >>>>>>> Add end-to-end (E2E) tests using Playwright for complete
> user
> > > >>>>>> workflows
> > > >>>>>>>
> > > >>>>>>> This would give us both component-level safety and full
> > > >>> user-journey
> > > >>>>>>> testing across browsers, helping catch regressions early.
> > > >>>>>>>
> > > >>>>>>> A few questions:
> > > >>>>>>>
> > > >>>>>>> -
> > > >>>>>>>
> > > >>>>>>> Should we prioritize expanding unit test coverage first, or
> > > >> work
> > > >>>> on
> > > >>>>>> unit
> > > >>>>>>> and E2E tests in parallel?
> > > >>>>>>>
> > > >>>>>>> -
> > > >>>>>>>
> > > >>>>>>> For E2E testing, is Playwright a good choice, or are there
> > > >> other
> > > >>>>>>> preferred tools we should explore?
> > > >>>>>>>
> > > >>>>>>> -
> > > >>>>>>>
> > > >>>>>>> If Playwright, should we use TypeScript or Python as
> language
> > > >>> for
> > > >>>>>> tests?
> > > >>>>>>> -
> > > >>>>>>>
> > > >>>>>>> Any other suggestions to strengthen our UI testing
> strategy?
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>> I’d love to hear your thoughts on whether this would be
> valuable
> > > >>> for
> > > >>>>> the
> > > >>>>>>> project before diving into implementation details.
> > > >>>>>>>
> > > >>>>>>> Regards,
> > > >>>>>>> Rahul Vats
> > > >>>>>>>
> > > >>>>>>
> > > >>
> ---------------------------------------------------------------------
> > > >>>>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > >>>>>> For additional commands, e-mail: dev-help@airflow.apache.org
> > > >>>>>>
> > > >>>>>>
> > > >>
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > For additional commands, e-mail: dev-help@airflow.apache.org
> > >
> > >
> >
>
Re: [Discuss] Improving Airflow UI Testing Coverage
Posted by Vikram Koka via dev <de...@airflow.apache.org>.
Rahul,
Thank you for starting this thread.
Strong +1 from me as well.
I not only agree with this, but I believe this needs to be a high priority
going forward and before 3.2.
I also agree with the comments on the "test pyramid" approach, of there
needing to be FAR more unit tests as compared to end to end tests.
I also believe that a certain level of rigor around this i.e. some form of
enforcement to ensure that all new UI changes come with a unit test.
I would also add the Core API changes (for UI) to the mix here, since a
fair number of UI changes requires underlying API changes.
Vikram
On Thu, Sep 25, 2025 at 12:04 AM Amogh Desai <am...@apache.org> wrote:
> Valid points.
>
> I was just expressing concern from a CI/CD failure perspective but in no
> way
> was I objecting to usage of typescript.
>
> Anything that works for the team that owns most of the tests works by me!
>
>
> Thanks & Regards,
> Amogh Desai
>
>
> On Thu, Sep 25, 2025 at 1:39 AM Jens Scheffler <js...@apache.org>
> wrote:
>
> > Echoing a full agreement to all statement before.
> >
> > I also would emphasize the test pyramid. As we really have a low
> > coverage we should maybe also have a metric. I would also propose that
> > we focus more on testing coverage == hopefully also improved quality
> > preventing regression in 3.2. One option could a bit of force to force
> > all new changes applied to UI to request a unit test being added (all
> > changes TS/x files need to have a corresponding .test.ts/x file) except
> > for cases where it does not make sense we allow exclusion of the rule by
> > adding a label by maintainers. (just want to drop the idea, might be
> > many will hate it)
> >
> > For language also have no strong preference but even though I am a TS
> > noob I see that testing UI is closer to UI development, to using Python
> > for TS UI testing might make a knowledge gap also for potential utility
> > code. So as most voices would propose TS (maybe then finally I get to
> > understand it)
> >
> > Jens
> >
> > On 24.09.25 18:33, Jarek Potiuk wrote:
> > > Small poking at important things from holidays while having a coffee
> at a
> > > place with internet :)
> > >
> > > +1 on having both. I have no strong opinion on the language and
> > Playwright.
> > >
> > > But there are few things for consideration:
> > >
> > > We should have many, many more unit tests than e2e tests. This is the
> > > classic test pyramid case: The Practical Test Pyramid
> > > https://share.google/YHVIt8EBptT1yYFur
> > >
> > > The e2e tests are slow, brittle and difficult to maintain and we should
> > > have rather small number of those - mostly green path on all major
> > > functionalities. Possibly with some cross browser checks - but those
> are
> > > very difficult to get right.
> > >
> > > It will be a pain to maintain those. Like our k8s e2e tests - when they
> > > break (rarely) people will have hard time to create and run the tests
> so
> > > the tooling around that has to drive people who will be reproducing
> the
> > > failures by hand - how to set it up and how to reproduce it and how to
> > > iterate as fast as possible.
> > >
> > > Keeping the tests stable and side-effect free will be a challenge - so
> > > likely a good 'reset the environment' before each test is a must. Also
> > they
> > > need to be run with all UI changes in CI which will make UI PR run
> > longer.
> > > It's an imperative to get them run not only in canary builds but in all
> > PRs
> > > for ui and API and likely whenever any core model change. It will slow
> > down
> > > UI development and create frustration for contributors unfamiliar with
> > the
> > > test framework so we need to have at least a few people proficient in
> > > intricacies of the test framework dedicated and willing to help those
> who
> > > will struggle with issues.
> > >
> > > Language is secondary - e2e tests will require all stack - Python
> > (backend)
> > > and Typescript and react experiences anyway. Most likely people who
> write
> > > UI should be involved so Typescript is fine, and also even those who do
> > not
> > > know TypeScript will be able to fix some obvious cases.
> > >
> > > J.
> > >
> > > śr., 24 wrz 2025, 09:50 użytkownik Shubham Raj <sh...@apache.org>
> > > napisał:
> > >
> > >> This addition will be really nice.
> > >> Also, at this point of 3.x, I think UI has been relatively stable and
> > >> writing e2e would be a good starting point. Unit tests can go
> parallel.
> > >> It's right that people maintaining CI/CD would be first persons to see
> > any
> > >> breakage, but I feel writing in ts will be easier for folks working on
> > >> frontend, and also for the new contributors who would like to
> > contribute in
> > >> this UI e2e tests coming from cypress or similar experience, since
> > majority
> > >> of other choices do come with ts/js support.
> > >> I would be happy to help with the e2e tests. We should probably go in
> > >> phases with P0/1/2.
> > >>
> > >> Thanks.
> > >>
> > >>
> > >> On Wed, Sep 24, 2025 at 9:01 PM Pierre Jeambrun <
> pierrejbrun@gmail.com>
> > >> wrote:
> > >>
> > >>> I also think that having a good unit test coverage + e2e tests would
> > >> really
> > >>> help maintaining quality of releases and prevent regressions early,
> > >> making
> > >>> all that much more manageable.
> > >>>
> > >>> My two cents regarding the questions above:
> > >>>
> > >>> -
> > >>> Should we prioritize expanding unit test coverage first, or work
> on
> > >> unit
> > >>> and E2E tests in parallel?
> > >>>
> > >>> Both in parallel sounds good to me. They do not depend on each
> > other
> > >> (At
> > >>> least in my mind)
> > >>> -
> > >>> For E2E testing, is Playwright a good choice, or are there other
> > >>> preferred tools we should explore?
> > >>>
> > >>> I like playwright and I feel this is the modern solution to go
> to.
> > >>> -
> > >>> If Playwright, should we use TypeScript or Python as language for
> > >> tests?
> > >>> Good arguments mentioned above for both sides. I personally think
> > >> (maybe
> > >>> wrongly), that ideally people maintaining / developing e2e tests will
> > be
> > >>> people with
> > >>> front-end knowledge. (events, identifiers, selectors, css
> styling,
> > >>> browser local states, etc..), so I tend to think that those people
> > will
> > >>> most likely come from the front-end community and
> > >>> having that in typescript would probably help them get started.
> > >>>
> > >>>
> > >>> On Wed, Sep 24, 2025 at 10:01 AM Amogh Desai <am...@apache.org>
> > >>> wrote:
> > >>>
> > >>>> Thanks for starting this discussion, Rahul.
> > >>>>
> > >>>> I am aligned with your thoughts, we have seen many many UI issues
> > >> lately
> > >>>> which I believe we should prioritize fixing or at least reducing
> going
> > >>>> forward.
> > >>>>
> > >>>> To answer your question(s) --
> > >>>>
> > >>>> * I think the unit tests and e2e / user flow tests as orthogonal, so
> > >> they
> > >>>> can be
> > >>>> developed in parallel.
> > >>>>
> > >>>> * While I do not have real experience with Playwright but from what
> I
> > >>> hear
> > >>>> / read on reddit,
> > >>>> github, and related forums, it looks like Playwright can do similar
> > job
> > >>> as
> > >>>> alternatives such as
> > >>>> Selenium, Cypress, while being more beginner friendly, which could
> > play
> > >>>> better in our favour.
> > >>>>
> > >>>> * Again, not very strong on this but I feel that having Typescript
> as
> > >> the
> > >>>> language of choice would limit
> > >>>> the number of contributors we get to help on this. Python would be
> my
> > >>>> weapon of choice, given
> > >>>> the "testing" is not limited by it and it potentially could
> encourage
> > >>>> broader participation.
> > >>>>
> > >>>> (Folks active on CI/CD(atleast me), are not very comfortable with
> > >>>> Typescript tests compared to
> > >>>> Python)
> > >>>>
> > >>>> One additional point: It would be valuable if we extract the e2e /
> > user
> > >>>> flow testing into a GH job that
> > >>>> a release manager could run as "sort" of last validation before
> > >> releasing
> > >>>> to avoid any last
> > >>>> minute issues.
> > >>>>
> > >>>> Thanks & Regards,
> > >>>> Amogh Desai
> > >>>>
> > >>>>
> > >>>> On Tue, Sep 23, 2025 at 9:15 PM Brent Bovenzi via dev <
> > >>>> dev@airflow.apache.org> wrote:
> > >>>>
> > >>>>> I agree that we need to improve UI tests. Happy to make that a
> > >> priority
> > >>>>> during 3.1.x and 3.2 development.
> > >>>>>
> > >>>>> Personally, I will start with expanding unit tests first. But that
> > >>>> doesn't
> > >>>>> mean others can't get started on E2E tests.
> > >>>>>
> > >>>>> I lean towards using typescript for playwright. With E2E we are
> > >> testing
> > >>>> the
> > >>>>> UI more than we are testing the API so I think the code should sit
> > >>> closer
> > >>>>> to the UI code.
> > >>>>>
> > >>>>> On Tue, Sep 23, 2025 at 9:38 AM Vincent Beck <vi...@apache.org>
> > >>>> wrote:
> > >>>>>> Not answering any of these questions (sorry) but strong +1 on
> > >> adding
> > >>>>>> end-to-end (E2E) tests. This will significantly make our UI more
> > >>> robust
> > >>>>> but
> > >>>>>> also will speed up some of our work. An example is front-end
> > >>> dependency
> > >>>>>> upgrades by dependabot. Today we have no way of knowing whether
> > >> these
> > >>>> PRs
> > >>>>>> break the UI, in order to merge these PRs we need to compile the
> > >>>>> front-end
> > >>>>>> assets and test manually whether the UI work. Having end-to-end
> > >> tests
> > >>>> on
> > >>>>>> the front-end and making them run in the CI would speed up this
> > >> very
> > >>>>>> example (on top of increasing code robustness, increase our
> > >>> confidence
> > >>>>> etc).
> > >>>>>> On 2025/09/23 11:03:40 Rahul Vats wrote:
> > >>>>>>> Hi all,
> > >>>>>>>
> > >>>>>>> With Airflow 3.0, the UI has undergone a complete revamp, and new
> > >>>>> changes
> > >>>>>>> are naturally more prone to regressions. We've already seen more
> > >> UI
> > >>>>>>> regressions that are hard to catch with our current limited test
> > >>>>>> coverage.
> > >>>>>>> Manual testing is also challenging given the wide variety of
> > >>>> workflows
> > >>>>>> and
> > >>>>>>> screens in the UI.
> > >>>>>>>
> > >>>>>>> Currently, the Airflow UI has a few unit tests using Vitest and
> > >>> React
> > >>>>>>> Testing Library, but coverage is quite limited — only about 4
> > >> test
> > >>>>> files
> > >>>>>>> covering major pages like DagsList and TaskInstance. Many
> > >> critical
> > >>>>>>> pages (Connections,
> > >>>>>>> Variables, Pools, Dashboard, etc.) have no test coverage at all.
> > >>>>>>>
> > >>>>>>> I’m exploring a two-pronged approach to improve UI testing:
> > >>>>>>>
> > >>>>>>> 1.
> > >>>>>>>
> > >>>>>>> Expand unit test coverage for individual components and pages
> > >>>>>>>
> > >>>>>>> 2.
> > >>>>>>>
> > >>>>>>> Add end-to-end (E2E) tests using Playwright for complete user
> > >>>>>> workflows
> > >>>>>>>
> > >>>>>>> This would give us both component-level safety and full
> > >>> user-journey
> > >>>>>>> testing across browsers, helping catch regressions early.
> > >>>>>>>
> > >>>>>>> A few questions:
> > >>>>>>>
> > >>>>>>> -
> > >>>>>>>
> > >>>>>>> Should we prioritize expanding unit test coverage first, or
> > >> work
> > >>>> on
> > >>>>>> unit
> > >>>>>>> and E2E tests in parallel?
> > >>>>>>>
> > >>>>>>> -
> > >>>>>>>
> > >>>>>>> For E2E testing, is Playwright a good choice, or are there
> > >> other
> > >>>>>>> preferred tools we should explore?
> > >>>>>>>
> > >>>>>>> -
> > >>>>>>>
> > >>>>>>> If Playwright, should we use TypeScript or Python as language
> > >>> for
> > >>>>>> tests?
> > >>>>>>> -
> > >>>>>>>
> > >>>>>>> Any other suggestions to strengthen our UI testing strategy?
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> I’d love to hear your thoughts on whether this would be valuable
> > >>> for
> > >>>>> the
> > >>>>>>> project before diving into implementation details.
> > >>>>>>>
> > >>>>>>> Regards,
> > >>>>>>> Rahul Vats
> > >>>>>>>
> > >>>>>>
> > >> ---------------------------------------------------------------------
> > >>>>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > >>>>>> For additional commands, e-mail: dev-help@airflow.apache.org
> > >>>>>>
> > >>>>>>
> > >>
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > For additional commands, e-mail: dev-help@airflow.apache.org
> >
> >
>
Re: [Discuss] Improving Airflow UI Testing Coverage
Posted by Amogh Desai <am...@apache.org>.
Valid points.
I was just expressing concern from a CI/CD failure perspective but in no way
was I objecting to usage of typescript.
Anything that works for the team that owns most of the tests works by me!
Thanks & Regards,
Amogh Desai
On Thu, Sep 25, 2025 at 1:39 AM Jens Scheffler <js...@apache.org> wrote:
> Echoing a full agreement to all statement before.
>
> I also would emphasize the test pyramid. As we really have a low
> coverage we should maybe also have a metric. I would also propose that
> we focus more on testing coverage == hopefully also improved quality
> preventing regression in 3.2. One option could a bit of force to force
> all new changes applied to UI to request a unit test being added (all
> changes TS/x files need to have a corresponding .test.ts/x file) except
> for cases where it does not make sense we allow exclusion of the rule by
> adding a label by maintainers. (just want to drop the idea, might be
> many will hate it)
>
> For language also have no strong preference but even though I am a TS
> noob I see that testing UI is closer to UI development, to using Python
> for TS UI testing might make a knowledge gap also for potential utility
> code. So as most voices would propose TS (maybe then finally I get to
> understand it)
>
> Jens
>
> On 24.09.25 18:33, Jarek Potiuk wrote:
> > Small poking at important things from holidays while having a coffee at a
> > place with internet :)
> >
> > +1 on having both. I have no strong opinion on the language and
> Playwright.
> >
> > But there are few things for consideration:
> >
> > We should have many, many more unit tests than e2e tests. This is the
> > classic test pyramid case: The Practical Test Pyramid
> > https://share.google/YHVIt8EBptT1yYFur
> >
> > The e2e tests are slow, brittle and difficult to maintain and we should
> > have rather small number of those - mostly green path on all major
> > functionalities. Possibly with some cross browser checks - but those are
> > very difficult to get right.
> >
> > It will be a pain to maintain those. Like our k8s e2e tests - when they
> > break (rarely) people will have hard time to create and run the tests so
> > the tooling around that has to drive people who will be reproducing the
> > failures by hand - how to set it up and how to reproduce it and how to
> > iterate as fast as possible.
> >
> > Keeping the tests stable and side-effect free will be a challenge - so
> > likely a good 'reset the environment' before each test is a must. Also
> they
> > need to be run with all UI changes in CI which will make UI PR run
> longer.
> > It's an imperative to get them run not only in canary builds but in all
> PRs
> > for ui and API and likely whenever any core model change. It will slow
> down
> > UI development and create frustration for contributors unfamiliar with
> the
> > test framework so we need to have at least a few people proficient in
> > intricacies of the test framework dedicated and willing to help those who
> > will struggle with issues.
> >
> > Language is secondary - e2e tests will require all stack - Python
> (backend)
> > and Typescript and react experiences anyway. Most likely people who write
> > UI should be involved so Typescript is fine, and also even those who do
> not
> > know TypeScript will be able to fix some obvious cases.
> >
> > J.
> >
> > śr., 24 wrz 2025, 09:50 użytkownik Shubham Raj <sh...@apache.org>
> > napisał:
> >
> >> This addition will be really nice.
> >> Also, at this point of 3.x, I think UI has been relatively stable and
> >> writing e2e would be a good starting point. Unit tests can go parallel.
> >> It's right that people maintaining CI/CD would be first persons to see
> any
> >> breakage, but I feel writing in ts will be easier for folks working on
> >> frontend, and also for the new contributors who would like to
> contribute in
> >> this UI e2e tests coming from cypress or similar experience, since
> majority
> >> of other choices do come with ts/js support.
> >> I would be happy to help with the e2e tests. We should probably go in
> >> phases with P0/1/2.
> >>
> >> Thanks.
> >>
> >>
> >> On Wed, Sep 24, 2025 at 9:01 PM Pierre Jeambrun <pi...@gmail.com>
> >> wrote:
> >>
> >>> I also think that having a good unit test coverage + e2e tests would
> >> really
> >>> help maintaining quality of releases and prevent regressions early,
> >> making
> >>> all that much more manageable.
> >>>
> >>> My two cents regarding the questions above:
> >>>
> >>> -
> >>> Should we prioritize expanding unit test coverage first, or work on
> >> unit
> >>> and E2E tests in parallel?
> >>>
> >>> Both in parallel sounds good to me. They do not depend on each
> other
> >> (At
> >>> least in my mind)
> >>> -
> >>> For E2E testing, is Playwright a good choice, or are there other
> >>> preferred tools we should explore?
> >>>
> >>> I like playwright and I feel this is the modern solution to go to.
> >>> -
> >>> If Playwright, should we use TypeScript or Python as language for
> >> tests?
> >>> Good arguments mentioned above for both sides. I personally think
> >> (maybe
> >>> wrongly), that ideally people maintaining / developing e2e tests will
> be
> >>> people with
> >>> front-end knowledge. (events, identifiers, selectors, css styling,
> >>> browser local states, etc..), so I tend to think that those people
> will
> >>> most likely come from the front-end community and
> >>> having that in typescript would probably help them get started.
> >>>
> >>>
> >>> On Wed, Sep 24, 2025 at 10:01 AM Amogh Desai <am...@apache.org>
> >>> wrote:
> >>>
> >>>> Thanks for starting this discussion, Rahul.
> >>>>
> >>>> I am aligned with your thoughts, we have seen many many UI issues
> >> lately
> >>>> which I believe we should prioritize fixing or at least reducing going
> >>>> forward.
> >>>>
> >>>> To answer your question(s) --
> >>>>
> >>>> * I think the unit tests and e2e / user flow tests as orthogonal, so
> >> they
> >>>> can be
> >>>> developed in parallel.
> >>>>
> >>>> * While I do not have real experience with Playwright but from what I
> >>> hear
> >>>> / read on reddit,
> >>>> github, and related forums, it looks like Playwright can do similar
> job
> >>> as
> >>>> alternatives such as
> >>>> Selenium, Cypress, while being more beginner friendly, which could
> play
> >>>> better in our favour.
> >>>>
> >>>> * Again, not very strong on this but I feel that having Typescript as
> >> the
> >>>> language of choice would limit
> >>>> the number of contributors we get to help on this. Python would be my
> >>>> weapon of choice, given
> >>>> the "testing" is not limited by it and it potentially could encourage
> >>>> broader participation.
> >>>>
> >>>> (Folks active on CI/CD(atleast me), are not very comfortable with
> >>>> Typescript tests compared to
> >>>> Python)
> >>>>
> >>>> One additional point: It would be valuable if we extract the e2e /
> user
> >>>> flow testing into a GH job that
> >>>> a release manager could run as "sort" of last validation before
> >> releasing
> >>>> to avoid any last
> >>>> minute issues.
> >>>>
> >>>> Thanks & Regards,
> >>>> Amogh Desai
> >>>>
> >>>>
> >>>> On Tue, Sep 23, 2025 at 9:15 PM Brent Bovenzi via dev <
> >>>> dev@airflow.apache.org> wrote:
> >>>>
> >>>>> I agree that we need to improve UI tests. Happy to make that a
> >> priority
> >>>>> during 3.1.x and 3.2 development.
> >>>>>
> >>>>> Personally, I will start with expanding unit tests first. But that
> >>>> doesn't
> >>>>> mean others can't get started on E2E tests.
> >>>>>
> >>>>> I lean towards using typescript for playwright. With E2E we are
> >> testing
> >>>> the
> >>>>> UI more than we are testing the API so I think the code should sit
> >>> closer
> >>>>> to the UI code.
> >>>>>
> >>>>> On Tue, Sep 23, 2025 at 9:38 AM Vincent Beck <vi...@apache.org>
> >>>> wrote:
> >>>>>> Not answering any of these questions (sorry) but strong +1 on
> >> adding
> >>>>>> end-to-end (E2E) tests. This will significantly make our UI more
> >>> robust
> >>>>> but
> >>>>>> also will speed up some of our work. An example is front-end
> >>> dependency
> >>>>>> upgrades by dependabot. Today we have no way of knowing whether
> >> these
> >>>> PRs
> >>>>>> break the UI, in order to merge these PRs we need to compile the
> >>>>> front-end
> >>>>>> assets and test manually whether the UI work. Having end-to-end
> >> tests
> >>>> on
> >>>>>> the front-end and making them run in the CI would speed up this
> >> very
> >>>>>> example (on top of increasing code robustness, increase our
> >>> confidence
> >>>>> etc).
> >>>>>> On 2025/09/23 11:03:40 Rahul Vats wrote:
> >>>>>>> Hi all,
> >>>>>>>
> >>>>>>> With Airflow 3.0, the UI has undergone a complete revamp, and new
> >>>>> changes
> >>>>>>> are naturally more prone to regressions. We've already seen more
> >> UI
> >>>>>>> regressions that are hard to catch with our current limited test
> >>>>>> coverage.
> >>>>>>> Manual testing is also challenging given the wide variety of
> >>>> workflows
> >>>>>> and
> >>>>>>> screens in the UI.
> >>>>>>>
> >>>>>>> Currently, the Airflow UI has a few unit tests using Vitest and
> >>> React
> >>>>>>> Testing Library, but coverage is quite limited — only about 4
> >> test
> >>>>> files
> >>>>>>> covering major pages like DagsList and TaskInstance. Many
> >> critical
> >>>>>>> pages (Connections,
> >>>>>>> Variables, Pools, Dashboard, etc.) have no test coverage at all.
> >>>>>>>
> >>>>>>> I’m exploring a two-pronged approach to improve UI testing:
> >>>>>>>
> >>>>>>> 1.
> >>>>>>>
> >>>>>>> Expand unit test coverage for individual components and pages
> >>>>>>>
> >>>>>>> 2.
> >>>>>>>
> >>>>>>> Add end-to-end (E2E) tests using Playwright for complete user
> >>>>>> workflows
> >>>>>>>
> >>>>>>> This would give us both component-level safety and full
> >>> user-journey
> >>>>>>> testing across browsers, helping catch regressions early.
> >>>>>>>
> >>>>>>> A few questions:
> >>>>>>>
> >>>>>>> -
> >>>>>>>
> >>>>>>> Should we prioritize expanding unit test coverage first, or
> >> work
> >>>> on
> >>>>>> unit
> >>>>>>> and E2E tests in parallel?
> >>>>>>>
> >>>>>>> -
> >>>>>>>
> >>>>>>> For E2E testing, is Playwright a good choice, or are there
> >> other
> >>>>>>> preferred tools we should explore?
> >>>>>>>
> >>>>>>> -
> >>>>>>>
> >>>>>>> If Playwright, should we use TypeScript or Python as language
> >>> for
> >>>>>> tests?
> >>>>>>> -
> >>>>>>>
> >>>>>>> Any other suggestions to strengthen our UI testing strategy?
> >>>>>>>
> >>>>>>>
> >>>>>>> I’d love to hear your thoughts on whether this would be valuable
> >>> for
> >>>>> the
> >>>>>>> project before diving into implementation details.
> >>>>>>>
> >>>>>>> Regards,
> >>>>>>> Rahul Vats
> >>>>>>>
> >>>>>>
> >> ---------------------------------------------------------------------
> >>>>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> >>>>>> For additional commands, e-mail: dev-help@airflow.apache.org
> >>>>>>
> >>>>>>
> >>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> For additional commands, e-mail: dev-help@airflow.apache.org
>
>
Re: [Discuss] Improving Airflow UI Testing Coverage
Posted by Jens Scheffler <js...@apache.org>.
Echoing a full agreement to all statement before.
I also would emphasize the test pyramid. As we really have a low
coverage we should maybe also have a metric. I would also propose that
we focus more on testing coverage == hopefully also improved quality
preventing regression in 3.2. One option could a bit of force to force
all new changes applied to UI to request a unit test being added (all
changes TS/x files need to have a corresponding .test.ts/x file) except
for cases where it does not make sense we allow exclusion of the rule by
adding a label by maintainers. (just want to drop the idea, might be
many will hate it)
For language also have no strong preference but even though I am a TS
noob I see that testing UI is closer to UI development, to using Python
for TS UI testing might make a knowledge gap also for potential utility
code. So as most voices would propose TS (maybe then finally I get to
understand it)
Jens
On 24.09.25 18:33, Jarek Potiuk wrote:
> Small poking at important things from holidays while having a coffee at a
> place with internet :)
>
> +1 on having both. I have no strong opinion on the language and Playwright.
>
> But there are few things for consideration:
>
> We should have many, many more unit tests than e2e tests. This is the
> classic test pyramid case: The Practical Test Pyramid
> https://share.google/YHVIt8EBptT1yYFur
>
> The e2e tests are slow, brittle and difficult to maintain and we should
> have rather small number of those - mostly green path on all major
> functionalities. Possibly with some cross browser checks - but those are
> very difficult to get right.
>
> It will be a pain to maintain those. Like our k8s e2e tests - when they
> break (rarely) people will have hard time to create and run the tests so
> the tooling around that has to drive people who will be reproducing the
> failures by hand - how to set it up and how to reproduce it and how to
> iterate as fast as possible.
>
> Keeping the tests stable and side-effect free will be a challenge - so
> likely a good 'reset the environment' before each test is a must. Also they
> need to be run with all UI changes in CI which will make UI PR run longer.
> It's an imperative to get them run not only in canary builds but in all PRs
> for ui and API and likely whenever any core model change. It will slow down
> UI development and create frustration for contributors unfamiliar with the
> test framework so we need to have at least a few people proficient in
> intricacies of the test framework dedicated and willing to help those who
> will struggle with issues.
>
> Language is secondary - e2e tests will require all stack - Python (backend)
> and Typescript and react experiences anyway. Most likely people who write
> UI should be involved so Typescript is fine, and also even those who do not
> know TypeScript will be able to fix some obvious cases.
>
> J.
>
> śr., 24 wrz 2025, 09:50 użytkownik Shubham Raj <sh...@apache.org>
> napisał:
>
>> This addition will be really nice.
>> Also, at this point of 3.x, I think UI has been relatively stable and
>> writing e2e would be a good starting point. Unit tests can go parallel.
>> It's right that people maintaining CI/CD would be first persons to see any
>> breakage, but I feel writing in ts will be easier for folks working on
>> frontend, and also for the new contributors who would like to contribute in
>> this UI e2e tests coming from cypress or similar experience, since majority
>> of other choices do come with ts/js support.
>> I would be happy to help with the e2e tests. We should probably go in
>> phases with P0/1/2.
>>
>> Thanks.
>>
>>
>> On Wed, Sep 24, 2025 at 9:01 PM Pierre Jeambrun <pi...@gmail.com>
>> wrote:
>>
>>> I also think that having a good unit test coverage + e2e tests would
>> really
>>> help maintaining quality of releases and prevent regressions early,
>> making
>>> all that much more manageable.
>>>
>>> My two cents regarding the questions above:
>>>
>>> -
>>> Should we prioritize expanding unit test coverage first, or work on
>> unit
>>> and E2E tests in parallel?
>>>
>>> Both in parallel sounds good to me. They do not depend on each other
>> (At
>>> least in my mind)
>>> -
>>> For E2E testing, is Playwright a good choice, or are there other
>>> preferred tools we should explore?
>>>
>>> I like playwright and I feel this is the modern solution to go to.
>>> -
>>> If Playwright, should we use TypeScript or Python as language for
>> tests?
>>> Good arguments mentioned above for both sides. I personally think
>> (maybe
>>> wrongly), that ideally people maintaining / developing e2e tests will be
>>> people with
>>> front-end knowledge. (events, identifiers, selectors, css styling,
>>> browser local states, etc..), so I tend to think that those people will
>>> most likely come from the front-end community and
>>> having that in typescript would probably help them get started.
>>>
>>>
>>> On Wed, Sep 24, 2025 at 10:01 AM Amogh Desai <am...@apache.org>
>>> wrote:
>>>
>>>> Thanks for starting this discussion, Rahul.
>>>>
>>>> I am aligned with your thoughts, we have seen many many UI issues
>> lately
>>>> which I believe we should prioritize fixing or at least reducing going
>>>> forward.
>>>>
>>>> To answer your question(s) --
>>>>
>>>> * I think the unit tests and e2e / user flow tests as orthogonal, so
>> they
>>>> can be
>>>> developed in parallel.
>>>>
>>>> * While I do not have real experience with Playwright but from what I
>>> hear
>>>> / read on reddit,
>>>> github, and related forums, it looks like Playwright can do similar job
>>> as
>>>> alternatives such as
>>>> Selenium, Cypress, while being more beginner friendly, which could play
>>>> better in our favour.
>>>>
>>>> * Again, not very strong on this but I feel that having Typescript as
>> the
>>>> language of choice would limit
>>>> the number of contributors we get to help on this. Python would be my
>>>> weapon of choice, given
>>>> the "testing" is not limited by it and it potentially could encourage
>>>> broader participation.
>>>>
>>>> (Folks active on CI/CD(atleast me), are not very comfortable with
>>>> Typescript tests compared to
>>>> Python)
>>>>
>>>> One additional point: It would be valuable if we extract the e2e / user
>>>> flow testing into a GH job that
>>>> a release manager could run as "sort" of last validation before
>> releasing
>>>> to avoid any last
>>>> minute issues.
>>>>
>>>> Thanks & Regards,
>>>> Amogh Desai
>>>>
>>>>
>>>> On Tue, Sep 23, 2025 at 9:15 PM Brent Bovenzi via dev <
>>>> dev@airflow.apache.org> wrote:
>>>>
>>>>> I agree that we need to improve UI tests. Happy to make that a
>> priority
>>>>> during 3.1.x and 3.2 development.
>>>>>
>>>>> Personally, I will start with expanding unit tests first. But that
>>>> doesn't
>>>>> mean others can't get started on E2E tests.
>>>>>
>>>>> I lean towards using typescript for playwright. With E2E we are
>> testing
>>>> the
>>>>> UI more than we are testing the API so I think the code should sit
>>> closer
>>>>> to the UI code.
>>>>>
>>>>> On Tue, Sep 23, 2025 at 9:38 AM Vincent Beck <vi...@apache.org>
>>>> wrote:
>>>>>> Not answering any of these questions (sorry) but strong +1 on
>> adding
>>>>>> end-to-end (E2E) tests. This will significantly make our UI more
>>> robust
>>>>> but
>>>>>> also will speed up some of our work. An example is front-end
>>> dependency
>>>>>> upgrades by dependabot. Today we have no way of knowing whether
>> these
>>>> PRs
>>>>>> break the UI, in order to merge these PRs we need to compile the
>>>>> front-end
>>>>>> assets and test manually whether the UI work. Having end-to-end
>> tests
>>>> on
>>>>>> the front-end and making them run in the CI would speed up this
>> very
>>>>>> example (on top of increasing code robustness, increase our
>>> confidence
>>>>> etc).
>>>>>> On 2025/09/23 11:03:40 Rahul Vats wrote:
>>>>>>> Hi all,
>>>>>>>
>>>>>>> With Airflow 3.0, the UI has undergone a complete revamp, and new
>>>>> changes
>>>>>>> are naturally more prone to regressions. We've already seen more
>> UI
>>>>>>> regressions that are hard to catch with our current limited test
>>>>>> coverage.
>>>>>>> Manual testing is also challenging given the wide variety of
>>>> workflows
>>>>>> and
>>>>>>> screens in the UI.
>>>>>>>
>>>>>>> Currently, the Airflow UI has a few unit tests using Vitest and
>>> React
>>>>>>> Testing Library, but coverage is quite limited — only about 4
>> test
>>>>> files
>>>>>>> covering major pages like DagsList and TaskInstance. Many
>> critical
>>>>>>> pages (Connections,
>>>>>>> Variables, Pools, Dashboard, etc.) have no test coverage at all.
>>>>>>>
>>>>>>> I’m exploring a two-pronged approach to improve UI testing:
>>>>>>>
>>>>>>> 1.
>>>>>>>
>>>>>>> Expand unit test coverage for individual components and pages
>>>>>>>
>>>>>>> 2.
>>>>>>>
>>>>>>> Add end-to-end (E2E) tests using Playwright for complete user
>>>>>> workflows
>>>>>>>
>>>>>>> This would give us both component-level safety and full
>>> user-journey
>>>>>>> testing across browsers, helping catch regressions early.
>>>>>>>
>>>>>>> A few questions:
>>>>>>>
>>>>>>> -
>>>>>>>
>>>>>>> Should we prioritize expanding unit test coverage first, or
>> work
>>>> on
>>>>>> unit
>>>>>>> and E2E tests in parallel?
>>>>>>>
>>>>>>> -
>>>>>>>
>>>>>>> For E2E testing, is Playwright a good choice, or are there
>> other
>>>>>>> preferred tools we should explore?
>>>>>>>
>>>>>>> -
>>>>>>>
>>>>>>> If Playwright, should we use TypeScript or Python as language
>>> for
>>>>>> tests?
>>>>>>> -
>>>>>>>
>>>>>>> Any other suggestions to strengthen our UI testing strategy?
>>>>>>>
>>>>>>>
>>>>>>> I’d love to hear your thoughts on whether this would be valuable
>>> for
>>>>> the
>>>>>>> project before diving into implementation details.
>>>>>>>
>>>>>>> Regards,
>>>>>>> Rahul Vats
>>>>>>>
>>>>>>
>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>>>>>> For additional commands, e-mail: dev-help@airflow.apache.org
>>>>>>
>>>>>>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
For additional commands, e-mail: dev-help@airflow.apache.org
Re: [Discuss] Improving Airflow UI Testing Coverage
Posted by Jarek Potiuk <ja...@potiuk.com>.
Small poking at important things from holidays while having a coffee at a
place with internet :)
+1 on having both. I have no strong opinion on the language and Playwright.
But there are few things for consideration:
We should have many, many more unit tests than e2e tests. This is the
classic test pyramid case: The Practical Test Pyramid
https://share.google/YHVIt8EBptT1yYFur
The e2e tests are slow, brittle and difficult to maintain and we should
have rather small number of those - mostly green path on all major
functionalities. Possibly with some cross browser checks - but those are
very difficult to get right.
It will be a pain to maintain those. Like our k8s e2e tests - when they
break (rarely) people will have hard time to create and run the tests so
the tooling around that has to drive people who will be reproducing the
failures by hand - how to set it up and how to reproduce it and how to
iterate as fast as possible.
Keeping the tests stable and side-effect free will be a challenge - so
likely a good 'reset the environment' before each test is a must. Also they
need to be run with all UI changes in CI which will make UI PR run longer.
It's an imperative to get them run not only in canary builds but in all PRs
for ui and API and likely whenever any core model change. It will slow down
UI development and create frustration for contributors unfamiliar with the
test framework so we need to have at least a few people proficient in
intricacies of the test framework dedicated and willing to help those who
will struggle with issues.
Language is secondary - e2e tests will require all stack - Python (backend)
and Typescript and react experiences anyway. Most likely people who write
UI should be involved so Typescript is fine, and also even those who do not
know TypeScript will be able to fix some obvious cases.
J.
śr., 24 wrz 2025, 09:50 użytkownik Shubham Raj <sh...@apache.org>
napisał:
> This addition will be really nice.
> Also, at this point of 3.x, I think UI has been relatively stable and
> writing e2e would be a good starting point. Unit tests can go parallel.
> It's right that people maintaining CI/CD would be first persons to see any
> breakage, but I feel writing in ts will be easier for folks working on
> frontend, and also for the new contributors who would like to contribute in
> this UI e2e tests coming from cypress or similar experience, since majority
> of other choices do come with ts/js support.
> I would be happy to help with the e2e tests. We should probably go in
> phases with P0/1/2.
>
> Thanks.
>
>
> On Wed, Sep 24, 2025 at 9:01 PM Pierre Jeambrun <pi...@gmail.com>
> wrote:
>
> > I also think that having a good unit test coverage + e2e tests would
> really
> > help maintaining quality of releases and prevent regressions early,
> making
> > all that much more manageable.
> >
> > My two cents regarding the questions above:
> >
> > -
> > Should we prioritize expanding unit test coverage first, or work on
> unit
> > and E2E tests in parallel?
> >
> > Both in parallel sounds good to me. They do not depend on each other
> (At
> > least in my mind)
> > -
> > For E2E testing, is Playwright a good choice, or are there other
> > preferred tools we should explore?
> >
> > I like playwright and I feel this is the modern solution to go to.
> > -
> > If Playwright, should we use TypeScript or Python as language for
> tests?
> >
> > Good arguments mentioned above for both sides. I personally think
> (maybe
> > wrongly), that ideally people maintaining / developing e2e tests will be
> > people with
> > front-end knowledge. (events, identifiers, selectors, css styling,
> > browser local states, etc..), so I tend to think that those people will
> > most likely come from the front-end community and
> > having that in typescript would probably help them get started.
> >
> >
> > On Wed, Sep 24, 2025 at 10:01 AM Amogh Desai <am...@apache.org>
> > wrote:
> >
> > > Thanks for starting this discussion, Rahul.
> > >
> > > I am aligned with your thoughts, we have seen many many UI issues
> lately
> > > which I believe we should prioritize fixing or at least reducing going
> > > forward.
> > >
> > > To answer your question(s) --
> > >
> > > * I think the unit tests and e2e / user flow tests as orthogonal, so
> they
> > > can be
> > > developed in parallel.
> > >
> > > * While I do not have real experience with Playwright but from what I
> > hear
> > > / read on reddit,
> > > github, and related forums, it looks like Playwright can do similar job
> > as
> > > alternatives such as
> > > Selenium, Cypress, while being more beginner friendly, which could play
> > > better in our favour.
> > >
> > > * Again, not very strong on this but I feel that having Typescript as
> the
> > > language of choice would limit
> > > the number of contributors we get to help on this. Python would be my
> > > weapon of choice, given
> > > the "testing" is not limited by it and it potentially could encourage
> > > broader participation.
> > >
> > > (Folks active on CI/CD(atleast me), are not very comfortable with
> > > Typescript tests compared to
> > > Python)
> > >
> > > One additional point: It would be valuable if we extract the e2e / user
> > > flow testing into a GH job that
> > > a release manager could run as "sort" of last validation before
> releasing
> > > to avoid any last
> > > minute issues.
> > >
> > > Thanks & Regards,
> > > Amogh Desai
> > >
> > >
> > > On Tue, Sep 23, 2025 at 9:15 PM Brent Bovenzi via dev <
> > > dev@airflow.apache.org> wrote:
> > >
> > > > I agree that we need to improve UI tests. Happy to make that a
> priority
> > > > during 3.1.x and 3.2 development.
> > > >
> > > > Personally, I will start with expanding unit tests first. But that
> > > doesn't
> > > > mean others can't get started on E2E tests.
> > > >
> > > > I lean towards using typescript for playwright. With E2E we are
> testing
> > > the
> > > > UI more than we are testing the API so I think the code should sit
> > closer
> > > > to the UI code.
> > > >
> > > > On Tue, Sep 23, 2025 at 9:38 AM Vincent Beck <vi...@apache.org>
> > > wrote:
> > > >
> > > > > Not answering any of these questions (sorry) but strong +1 on
> adding
> > > > > end-to-end (E2E) tests. This will significantly make our UI more
> > robust
> > > > but
> > > > > also will speed up some of our work. An example is front-end
> > dependency
> > > > > upgrades by dependabot. Today we have no way of knowing whether
> these
> > > PRs
> > > > > break the UI, in order to merge these PRs we need to compile the
> > > > front-end
> > > > > assets and test manually whether the UI work. Having end-to-end
> tests
> > > on
> > > > > the front-end and making them run in the CI would speed up this
> very
> > > > > example (on top of increasing code robustness, increase our
> > confidence
> > > > etc).
> > > > >
> > > > > On 2025/09/23 11:03:40 Rahul Vats wrote:
> > > > > > Hi all,
> > > > > >
> > > > > > With Airflow 3.0, the UI has undergone a complete revamp, and new
> > > > changes
> > > > > > are naturally more prone to regressions. We've already seen more
> UI
> > > > > > regressions that are hard to catch with our current limited test
> > > > > coverage.
> > > > > > Manual testing is also challenging given the wide variety of
> > > workflows
> > > > > and
> > > > > > screens in the UI.
> > > > > >
> > > > > > Currently, the Airflow UI has a few unit tests using Vitest and
> > React
> > > > > > Testing Library, but coverage is quite limited — only about 4
> test
> > > > files
> > > > > > covering major pages like DagsList and TaskInstance. Many
> critical
> > > > > > pages (Connections,
> > > > > > Variables, Pools, Dashboard, etc.) have no test coverage at all.
> > > > > >
> > > > > > I’m exploring a two-pronged approach to improve UI testing:
> > > > > >
> > > > > > 1.
> > > > > >
> > > > > > Expand unit test coverage for individual components and pages
> > > > > >
> > > > > > 2.
> > > > > >
> > > > > > Add end-to-end (E2E) tests using Playwright for complete user
> > > > > workflows
> > > > > >
> > > > > >
> > > > > > This would give us both component-level safety and full
> > user-journey
> > > > > > testing across browsers, helping catch regressions early.
> > > > > >
> > > > > > A few questions:
> > > > > >
> > > > > > -
> > > > > >
> > > > > > Should we prioritize expanding unit test coverage first, or
> work
> > > on
> > > > > unit
> > > > > > and E2E tests in parallel?
> > > > > >
> > > > > > -
> > > > > >
> > > > > > For E2E testing, is Playwright a good choice, or are there
> other
> > > > > > preferred tools we should explore?
> > > > > >
> > > > > > -
> > > > > >
> > > > > > If Playwright, should we use TypeScript or Python as language
> > for
> > > > > tests?
> > > > > >
> > > > > > -
> > > > > >
> > > > > > Any other suggestions to strengthen our UI testing strategy?
> > > > > >
> > > > > >
> > > > > > I’d love to hear your thoughts on whether this would be valuable
> > for
> > > > the
> > > > > > project before diving into implementation details.
> > > > > >
> > > > > > Regards,
> > > > > > Rahul Vats
> > > > > >
> > > > >
> > > > >
> ---------------------------------------------------------------------
> > > > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > > > For additional commands, e-mail: dev-help@airflow.apache.org
> > > > >
> > > > >
> > > >
> > >
> >
>
>
Re: [Discuss] Improving Airflow UI Testing Coverage
Posted by Shubham Raj <sh...@apache.org>.
This addition will be really nice.
Also, at this point of 3.x, I think UI has been relatively stable and
writing e2e would be a good starting point. Unit tests can go parallel.
It's right that people maintaining CI/CD would be first persons to see any
breakage, but I feel writing in ts will be easier for folks working on
frontend, and also for the new contributors who would like to contribute in
this UI e2e tests coming from cypress or similar experience, since majority
of other choices do come with ts/js support.
I would be happy to help with the e2e tests. We should probably go in
phases with P0/1/2.
Thanks.
On Wed, Sep 24, 2025 at 9:01 PM Pierre Jeambrun <pi...@gmail.com>
wrote:
> I also think that having a good unit test coverage + e2e tests would really
> help maintaining quality of releases and prevent regressions early, making
> all that much more manageable.
>
> My two cents regarding the questions above:
>
> -
> Should we prioritize expanding unit test coverage first, or work on unit
> and E2E tests in parallel?
>
> Both in parallel sounds good to me. They do not depend on each other (At
> least in my mind)
> -
> For E2E testing, is Playwright a good choice, or are there other
> preferred tools we should explore?
>
> I like playwright and I feel this is the modern solution to go to.
> -
> If Playwright, should we use TypeScript or Python as language for tests?
>
> Good arguments mentioned above for both sides. I personally think (maybe
> wrongly), that ideally people maintaining / developing e2e tests will be
> people with
> front-end knowledge. (events, identifiers, selectors, css styling,
> browser local states, etc..), so I tend to think that those people will
> most likely come from the front-end community and
> having that in typescript would probably help them get started.
>
>
> On Wed, Sep 24, 2025 at 10:01 AM Amogh Desai <am...@apache.org>
> wrote:
>
> > Thanks for starting this discussion, Rahul.
> >
> > I am aligned with your thoughts, we have seen many many UI issues lately
> > which I believe we should prioritize fixing or at least reducing going
> > forward.
> >
> > To answer your question(s) --
> >
> > * I think the unit tests and e2e / user flow tests as orthogonal, so they
> > can be
> > developed in parallel.
> >
> > * While I do not have real experience with Playwright but from what I
> hear
> > / read on reddit,
> > github, and related forums, it looks like Playwright can do similar job
> as
> > alternatives such as
> > Selenium, Cypress, while being more beginner friendly, which could play
> > better in our favour.
> >
> > * Again, not very strong on this but I feel that having Typescript as the
> > language of choice would limit
> > the number of contributors we get to help on this. Python would be my
> > weapon of choice, given
> > the "testing" is not limited by it and it potentially could encourage
> > broader participation.
> >
> > (Folks active on CI/CD(atleast me), are not very comfortable with
> > Typescript tests compared to
> > Python)
> >
> > One additional point: It would be valuable if we extract the e2e / user
> > flow testing into a GH job that
> > a release manager could run as "sort" of last validation before releasing
> > to avoid any last
> > minute issues.
> >
> > Thanks & Regards,
> > Amogh Desai
> >
> >
> > On Tue, Sep 23, 2025 at 9:15 PM Brent Bovenzi via dev <
> > dev@airflow.apache.org> wrote:
> >
> > > I agree that we need to improve UI tests. Happy to make that a priority
> > > during 3.1.x and 3.2 development.
> > >
> > > Personally, I will start with expanding unit tests first. But that
> > doesn't
> > > mean others can't get started on E2E tests.
> > >
> > > I lean towards using typescript for playwright. With E2E we are testing
> > the
> > > UI more than we are testing the API so I think the code should sit
> closer
> > > to the UI code.
> > >
> > > On Tue, Sep 23, 2025 at 9:38 AM Vincent Beck <vi...@apache.org>
> > wrote:
> > >
> > > > Not answering any of these questions (sorry) but strong +1 on adding
> > > > end-to-end (E2E) tests. This will significantly make our UI more
> robust
> > > but
> > > > also will speed up some of our work. An example is front-end
> dependency
> > > > upgrades by dependabot. Today we have no way of knowing whether these
> > PRs
> > > > break the UI, in order to merge these PRs we need to compile the
> > > front-end
> > > > assets and test manually whether the UI work. Having end-to-end tests
> > on
> > > > the front-end and making them run in the CI would speed up this very
> > > > example (on top of increasing code robustness, increase our
> confidence
> > > etc).
> > > >
> > > > On 2025/09/23 11:03:40 Rahul Vats wrote:
> > > > > Hi all,
> > > > >
> > > > > With Airflow 3.0, the UI has undergone a complete revamp, and new
> > > changes
> > > > > are naturally more prone to regressions. We've already seen more UI
> > > > > regressions that are hard to catch with our current limited test
> > > > coverage.
> > > > > Manual testing is also challenging given the wide variety of
> > workflows
> > > > and
> > > > > screens in the UI.
> > > > >
> > > > > Currently, the Airflow UI has a few unit tests using Vitest and
> React
> > > > > Testing Library, but coverage is quite limited — only about 4 test
> > > files
> > > > > covering major pages like DagsList and TaskInstance. Many critical
> > > > > pages (Connections,
> > > > > Variables, Pools, Dashboard, etc.) have no test coverage at all.
> > > > >
> > > > > I’m exploring a two-pronged approach to improve UI testing:
> > > > >
> > > > > 1.
> > > > >
> > > > > Expand unit test coverage for individual components and pages
> > > > >
> > > > > 2.
> > > > >
> > > > > Add end-to-end (E2E) tests using Playwright for complete user
> > > > workflows
> > > > >
> > > > >
> > > > > This would give us both component-level safety and full
> user-journey
> > > > > testing across browsers, helping catch regressions early.
> > > > >
> > > > > A few questions:
> > > > >
> > > > > -
> > > > >
> > > > > Should we prioritize expanding unit test coverage first, or work
> > on
> > > > unit
> > > > > and E2E tests in parallel?
> > > > >
> > > > > -
> > > > >
> > > > > For E2E testing, is Playwright a good choice, or are there other
> > > > > preferred tools we should explore?
> > > > >
> > > > > -
> > > > >
> > > > > If Playwright, should we use TypeScript or Python as language
> for
> > > > tests?
> > > > >
> > > > > -
> > > > >
> > > > > Any other suggestions to strengthen our UI testing strategy?
> > > > >
> > > > >
> > > > > I’d love to hear your thoughts on whether this would be valuable
> for
> > > the
> > > > > project before diving into implementation details.
> > > > >
> > > > > Regards,
> > > > > Rahul Vats
> > > > >
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > > For additional commands, e-mail: dev-help@airflow.apache.org
> > > >
> > > >
> > >
> >
>
Re: [Discuss] Improving Airflow UI Testing Coverage
Posted by Pierre Jeambrun <pi...@gmail.com>.
I also think that having a good unit test coverage + e2e tests would really
help maintaining quality of releases and prevent regressions early, making
all that much more manageable.
My two cents regarding the questions above:
-
Should we prioritize expanding unit test coverage first, or work on unit
and E2E tests in parallel?
Both in parallel sounds good to me. They do not depend on each other (At
least in my mind)
-
For E2E testing, is Playwright a good choice, or are there other
preferred tools we should explore?
I like playwright and I feel this is the modern solution to go to.
-
If Playwright, should we use TypeScript or Python as language for tests?
Good arguments mentioned above for both sides. I personally think (maybe
wrongly), that ideally people maintaining / developing e2e tests will be
people with
front-end knowledge. (events, identifiers, selectors, css styling,
browser local states, etc..), so I tend to think that those people will
most likely come from the front-end community and
having that in typescript would probably help them get started.
On Wed, Sep 24, 2025 at 10:01 AM Amogh Desai <am...@apache.org> wrote:
> Thanks for starting this discussion, Rahul.
>
> I am aligned with your thoughts, we have seen many many UI issues lately
> which I believe we should prioritize fixing or at least reducing going
> forward.
>
> To answer your question(s) --
>
> * I think the unit tests and e2e / user flow tests as orthogonal, so they
> can be
> developed in parallel.
>
> * While I do not have real experience with Playwright but from what I hear
> / read on reddit,
> github, and related forums, it looks like Playwright can do similar job as
> alternatives such as
> Selenium, Cypress, while being more beginner friendly, which could play
> better in our favour.
>
> * Again, not very strong on this but I feel that having Typescript as the
> language of choice would limit
> the number of contributors we get to help on this. Python would be my
> weapon of choice, given
> the "testing" is not limited by it and it potentially could encourage
> broader participation.
>
> (Folks active on CI/CD(atleast me), are not very comfortable with
> Typescript tests compared to
> Python)
>
> One additional point: It would be valuable if we extract the e2e / user
> flow testing into a GH job that
> a release manager could run as "sort" of last validation before releasing
> to avoid any last
> minute issues.
>
> Thanks & Regards,
> Amogh Desai
>
>
> On Tue, Sep 23, 2025 at 9:15 PM Brent Bovenzi via dev <
> dev@airflow.apache.org> wrote:
>
> > I agree that we need to improve UI tests. Happy to make that a priority
> > during 3.1.x and 3.2 development.
> >
> > Personally, I will start with expanding unit tests first. But that
> doesn't
> > mean others can't get started on E2E tests.
> >
> > I lean towards using typescript for playwright. With E2E we are testing
> the
> > UI more than we are testing the API so I think the code should sit closer
> > to the UI code.
> >
> > On Tue, Sep 23, 2025 at 9:38 AM Vincent Beck <vi...@apache.org>
> wrote:
> >
> > > Not answering any of these questions (sorry) but strong +1 on adding
> > > end-to-end (E2E) tests. This will significantly make our UI more robust
> > but
> > > also will speed up some of our work. An example is front-end dependency
> > > upgrades by dependabot. Today we have no way of knowing whether these
> PRs
> > > break the UI, in order to merge these PRs we need to compile the
> > front-end
> > > assets and test manually whether the UI work. Having end-to-end tests
> on
> > > the front-end and making them run in the CI would speed up this very
> > > example (on top of increasing code robustness, increase our confidence
> > etc).
> > >
> > > On 2025/09/23 11:03:40 Rahul Vats wrote:
> > > > Hi all,
> > > >
> > > > With Airflow 3.0, the UI has undergone a complete revamp, and new
> > changes
> > > > are naturally more prone to regressions. We've already seen more UI
> > > > regressions that are hard to catch with our current limited test
> > > coverage.
> > > > Manual testing is also challenging given the wide variety of
> workflows
> > > and
> > > > screens in the UI.
> > > >
> > > > Currently, the Airflow UI has a few unit tests using Vitest and React
> > > > Testing Library, but coverage is quite limited — only about 4 test
> > files
> > > > covering major pages like DagsList and TaskInstance. Many critical
> > > > pages (Connections,
> > > > Variables, Pools, Dashboard, etc.) have no test coverage at all.
> > > >
> > > > I’m exploring a two-pronged approach to improve UI testing:
> > > >
> > > > 1.
> > > >
> > > > Expand unit test coverage for individual components and pages
> > > >
> > > > 2.
> > > >
> > > > Add end-to-end (E2E) tests using Playwright for complete user
> > > workflows
> > > >
> > > >
> > > > This would give us both component-level safety and full user-journey
> > > > testing across browsers, helping catch regressions early.
> > > >
> > > > A few questions:
> > > >
> > > > -
> > > >
> > > > Should we prioritize expanding unit test coverage first, or work
> on
> > > unit
> > > > and E2E tests in parallel?
> > > >
> > > > -
> > > >
> > > > For E2E testing, is Playwright a good choice, or are there other
> > > > preferred tools we should explore?
> > > >
> > > > -
> > > >
> > > > If Playwright, should we use TypeScript or Python as language for
> > > tests?
> > > >
> > > > -
> > > >
> > > > Any other suggestions to strengthen our UI testing strategy?
> > > >
> > > >
> > > > I’d love to hear your thoughts on whether this would be valuable for
> > the
> > > > project before diving into implementation details.
> > > >
> > > > Regards,
> > > > Rahul Vats
> > > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > For additional commands, e-mail: dev-help@airflow.apache.org
> > >
> > >
> >
>
Re: [Discuss] Improving Airflow UI Testing Coverage
Posted by Amogh Desai <am...@apache.org>.
Thanks for starting this discussion, Rahul.
I am aligned with your thoughts, we have seen many many UI issues lately
which I believe we should prioritize fixing or at least reducing going
forward.
To answer your question(s) --
* I think the unit tests and e2e / user flow tests as orthogonal, so they
can be
developed in parallel.
* While I do not have real experience with Playwright but from what I hear
/ read on reddit,
github, and related forums, it looks like Playwright can do similar job as
alternatives such as
Selenium, Cypress, while being more beginner friendly, which could play
better in our favour.
* Again, not very strong on this but I feel that having Typescript as the
language of choice would limit
the number of contributors we get to help on this. Python would be my
weapon of choice, given
the "testing" is not limited by it and it potentially could encourage
broader participation.
(Folks active on CI/CD(atleast me), are not very comfortable with
Typescript tests compared to
Python)
One additional point: It would be valuable if we extract the e2e / user
flow testing into a GH job that
a release manager could run as "sort" of last validation before releasing
to avoid any last
minute issues.
Thanks & Regards,
Amogh Desai
On Tue, Sep 23, 2025 at 9:15 PM Brent Bovenzi via dev <
dev@airflow.apache.org> wrote:
> I agree that we need to improve UI tests. Happy to make that a priority
> during 3.1.x and 3.2 development.
>
> Personally, I will start with expanding unit tests first. But that doesn't
> mean others can't get started on E2E tests.
>
> I lean towards using typescript for playwright. With E2E we are testing the
> UI more than we are testing the API so I think the code should sit closer
> to the UI code.
>
> On Tue, Sep 23, 2025 at 9:38 AM Vincent Beck <vi...@apache.org> wrote:
>
> > Not answering any of these questions (sorry) but strong +1 on adding
> > end-to-end (E2E) tests. This will significantly make our UI more robust
> but
> > also will speed up some of our work. An example is front-end dependency
> > upgrades by dependabot. Today we have no way of knowing whether these PRs
> > break the UI, in order to merge these PRs we need to compile the
> front-end
> > assets and test manually whether the UI work. Having end-to-end tests on
> > the front-end and making them run in the CI would speed up this very
> > example (on top of increasing code robustness, increase our confidence
> etc).
> >
> > On 2025/09/23 11:03:40 Rahul Vats wrote:
> > > Hi all,
> > >
> > > With Airflow 3.0, the UI has undergone a complete revamp, and new
> changes
> > > are naturally more prone to regressions. We've already seen more UI
> > > regressions that are hard to catch with our current limited test
> > coverage.
> > > Manual testing is also challenging given the wide variety of workflows
> > and
> > > screens in the UI.
> > >
> > > Currently, the Airflow UI has a few unit tests using Vitest and React
> > > Testing Library, but coverage is quite limited — only about 4 test
> files
> > > covering major pages like DagsList and TaskInstance. Many critical
> > > pages (Connections,
> > > Variables, Pools, Dashboard, etc.) have no test coverage at all.
> > >
> > > I’m exploring a two-pronged approach to improve UI testing:
> > >
> > > 1.
> > >
> > > Expand unit test coverage for individual components and pages
> > >
> > > 2.
> > >
> > > Add end-to-end (E2E) tests using Playwright for complete user
> > workflows
> > >
> > >
> > > This would give us both component-level safety and full user-journey
> > > testing across browsers, helping catch regressions early.
> > >
> > > A few questions:
> > >
> > > -
> > >
> > > Should we prioritize expanding unit test coverage first, or work on
> > unit
> > > and E2E tests in parallel?
> > >
> > > -
> > >
> > > For E2E testing, is Playwright a good choice, or are there other
> > > preferred tools we should explore?
> > >
> > > -
> > >
> > > If Playwright, should we use TypeScript or Python as language for
> > tests?
> > >
> > > -
> > >
> > > Any other suggestions to strengthen our UI testing strategy?
> > >
> > >
> > > I’d love to hear your thoughts on whether this would be valuable for
> the
> > > project before diving into implementation details.
> > >
> > > Regards,
> > > Rahul Vats
> > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > For additional commands, e-mail: dev-help@airflow.apache.org
> >
> >
>
Re: [Discuss] Improving Airflow UI Testing Coverage
Posted by Brent Bovenzi via dev <de...@airflow.apache.org>.
I agree that we need to improve UI tests. Happy to make that a priority
during 3.1.x and 3.2 development.
Personally, I will start with expanding unit tests first. But that doesn't
mean others can't get started on E2E tests.
I lean towards using typescript for playwright. With E2E we are testing the
UI more than we are testing the API so I think the code should sit closer
to the UI code.
On Tue, Sep 23, 2025 at 9:38 AM Vincent Beck <vi...@apache.org> wrote:
> Not answering any of these questions (sorry) but strong +1 on adding
> end-to-end (E2E) tests. This will significantly make our UI more robust but
> also will speed up some of our work. An example is front-end dependency
> upgrades by dependabot. Today we have no way of knowing whether these PRs
> break the UI, in order to merge these PRs we need to compile the front-end
> assets and test manually whether the UI work. Having end-to-end tests on
> the front-end and making them run in the CI would speed up this very
> example (on top of increasing code robustness, increase our confidence etc).
>
> On 2025/09/23 11:03:40 Rahul Vats wrote:
> > Hi all,
> >
> > With Airflow 3.0, the UI has undergone a complete revamp, and new changes
> > are naturally more prone to regressions. We've already seen more UI
> > regressions that are hard to catch with our current limited test
> coverage.
> > Manual testing is also challenging given the wide variety of workflows
> and
> > screens in the UI.
> >
> > Currently, the Airflow UI has a few unit tests using Vitest and React
> > Testing Library, but coverage is quite limited — only about 4 test files
> > covering major pages like DagsList and TaskInstance. Many critical
> > pages (Connections,
> > Variables, Pools, Dashboard, etc.) have no test coverage at all.
> >
> > I’m exploring a two-pronged approach to improve UI testing:
> >
> > 1.
> >
> > Expand unit test coverage for individual components and pages
> >
> > 2.
> >
> > Add end-to-end (E2E) tests using Playwright for complete user
> workflows
> >
> >
> > This would give us both component-level safety and full user-journey
> > testing across browsers, helping catch regressions early.
> >
> > A few questions:
> >
> > -
> >
> > Should we prioritize expanding unit test coverage first, or work on
> unit
> > and E2E tests in parallel?
> >
> > -
> >
> > For E2E testing, is Playwright a good choice, or are there other
> > preferred tools we should explore?
> >
> > -
> >
> > If Playwright, should we use TypeScript or Python as language for
> tests?
> >
> > -
> >
> > Any other suggestions to strengthen our UI testing strategy?
> >
> >
> > I’d love to hear your thoughts on whether this would be valuable for the
> > project before diving into implementation details.
> >
> > Regards,
> > Rahul Vats
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> For additional commands, e-mail: dev-help@airflow.apache.org
>
>
Re: [Discuss] Improving Airflow UI Testing Coverage
Posted by Vincent Beck <vi...@apache.org>.
Not answering any of these questions (sorry) but strong +1 on adding end-to-end (E2E) tests. This will significantly make our UI more robust but also will speed up some of our work. An example is front-end dependency upgrades by dependabot. Today we have no way of knowing whether these PRs break the UI, in order to merge these PRs we need to compile the front-end assets and test manually whether the UI work. Having end-to-end tests on the front-end and making them run in the CI would speed up this very example (on top of increasing code robustness, increase our confidence etc).
On 2025/09/23 11:03:40 Rahul Vats wrote:
> Hi all,
>
> With Airflow 3.0, the UI has undergone a complete revamp, and new changes
> are naturally more prone to regressions. We've already seen more UI
> regressions that are hard to catch with our current limited test coverage.
> Manual testing is also challenging given the wide variety of workflows and
> screens in the UI.
>
> Currently, the Airflow UI has a few unit tests using Vitest and React
> Testing Library, but coverage is quite limited — only about 4 test files
> covering major pages like DagsList and TaskInstance. Many critical
> pages (Connections,
> Variables, Pools, Dashboard, etc.) have no test coverage at all.
>
> I’m exploring a two-pronged approach to improve UI testing:
>
> 1.
>
> Expand unit test coverage for individual components and pages
>
> 2.
>
> Add end-to-end (E2E) tests using Playwright for complete user workflows
>
>
> This would give us both component-level safety and full user-journey
> testing across browsers, helping catch regressions early.
>
> A few questions:
>
> -
>
> Should we prioritize expanding unit test coverage first, or work on unit
> and E2E tests in parallel?
>
> -
>
> For E2E testing, is Playwright a good choice, or are there other
> preferred tools we should explore?
>
> -
>
> If Playwright, should we use TypeScript or Python as language for tests?
>
> -
>
> Any other suggestions to strengthen our UI testing strategy?
>
>
> I’d love to hear your thoughts on whether this would be valuable for the
> project before diving into implementation details.
>
> Regards,
> Rahul Vats
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
For additional commands, e-mail: dev-help@airflow.apache.org