You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "zach-overflow (via GitHub)" <gi...@apache.org> on 2026/09/24 12:20:55 UTC

[I] Add async support in `apache-airflow-client` [airflow]

zach-overflow opened a new issue, #73661:
URL: https://github.com/apache/airflow/issues/73661

   ### Description
   
   Currently, the `apache-airflow-client` package (generated from the `openapi-generator-cli`) uses the default `urllib3` client, which does not support async operations.
   
   The proposed feature is to generate and package `async` client code in the `apache-airflow-client` package, in addition to the existing synchronous client code. 
   Ideally, this **should not** alter the synchronous interface to ensure this is not a breaking change.
   
   ### Use case/motivation
   
   Async client usage is particularly helpful for users who either:
   * Need to perform independent API operations in a single process (such as a bulk "pause dags" operation).
   * Manage multiple Airflow instances, and require multi-plexing requests across instances (e.g. a management script which updates the same Airflow variables across multiple Airflow instances at once).
       * This is a situation I'm currently using async api calls for, but with httpx2 directly, rather than with the official Apache Airflow python client. It would be really nice to have the additional type safety the official client offers, and I'm sure this would benefit others as well.
   
   As noted above, `urllib3` is the current client used in the client generation tool, but the upcoming release of `openapi-generator-cli` version `7.26.0` will support for generating code which uses `httpx2` under the hood. `httpx2` supports both sync and async operations. Additionally, the openapi-generator-cli has configuration options to generate code with both sync and async code in the same output package.
   
   Both the `httpx2` and `async` code generation config options are described here: https://openapi-generator.tech/docs/generators/python/#config-options
   
   ### Related issues
   
   Tangentially related: https://github.com/apache/airflow/issues/70522 
   * context:`httpx` is apparently no longer actively maintained, and Pydantic's `httpx2` is the replacement that appears to have taken the designation as the python community's "accepted" replacement.
   
   ### Are you willing to submit a PR?
   
   - [x] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Add async support in `apache-airflow-client` [airflow]

Posted by "kadubhumika (via GitHub)" <gi...@apache.org>.
kadubhumika commented on issue #73661:
URL: https://github.com/apache/airflow/issues/73661#issuecomment-5839490555

   Hi! @zach-overflow  I’ve investigated #73661 and prototyped the async Python client generation locally.
   
   I was able to verify that the newer generator configuration can produce both:
   
   * the existing synchronous methods, e.g. `get_version()`
   * new async methods, e.g. `get_version_async()`
   I also tested the generated client imports successfully and verified that the generated package uses the `httpx2` transport.
   
   The current blocker is the OpenAPI Generator version: Airflow is pinned to 7.25.0, while the 7.26.0 line is currently still a SNAPSHOT. I don’t want to introduce an unreleased generator version into Airflow just to make the prototype work.
   I’ve therefore paused the implementation before opening a PR. Once the required generator support is available as a stable release, I can continue with the Breeze generation changes, templates, and tests.
   
   Could you let me know if this is the expected direction for #73661, or if you would prefer a different approach that works with the currently pinned generator?
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Add async support in `apache-airflow-client` [airflow]

Posted by "kadubhumika (via GitHub)" <gi...@apache.org>.
kadubhumika commented on issue #73661:
URL: https://github.com/apache/airflow/issues/73661#issuecomment-5892456360

   @zach-overflow  Thanks for clarifying! That makes sense. I’ll leave this on hold until 7.26.0 is released and can continue once the stable release is available.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Add async support in `apache-airflow-client` [airflow]

Posted by "zach-overflow (via GitHub)" <gi...@apache.org>.
zach-overflow commented on issue #73661:
URL: https://github.com/apache/airflow/issues/73661#issuecomment-5890350089

   @kadubhumika sorry for missing your comments. I realized after writing this issue that this is not doable until the next minor release (`7.26.0`) of the [`openapi-generator-cli`](https://github.com/OpenAPITools/openapi-generator). Until that release is available, Airflows' client generation would be running off a dev version of the CLI, which I don't think is a good situation. 
   
   There's also a bit of post-processing required to ensure both the async and sync generated clients operate on the same pydantic models since`openapi-generator-cli` will try to create duplicate models. This point isn't a huge deal, but worth flagging as well.
   
   I'll update the ticket description to highlight this comment for now.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Add async support in `apache-airflow-client` [airflow]

Posted by "kadubhumika (via GitHub)" <gi...@apache.org>.
kadubhumika commented on issue #73661:
URL: https://github.com/apache/airflow/issues/73661#issuecomment-5842845694

   What Works Locally:Using a newer generator line (7.26.0-SNAPSHOT), I confirmed that generating code using the httpx2 library works perfectly.It generates both the necessary httpx2.AsyncClient transport and the required async methods.Backward Compatibility: To avoid breaking changes where native generation swaps get_version() to async, I experimented with a custom api.mustache template. This successfully preserves the existing synchronous interface (get_version() -> sync) while cleanly appending the new async methods (get_version_async() -> async).The Blocker:Airflow's Breeze environment is currently pinned to OpenAPI Generator 7.25.0, which supports the older httpx but lacks native support for httpx2.The httpx2 engine is only available starting in the 7.26.0 release line, which is currently in a SNAPSHOT phase. I want to avoid introducing an unstable/unreleased generator version into Airflow’s build pipeline.How would you prefer to proceed?Hold for 7.26.0 Release: Pause this iss
 ue until OpenAPI Generator 7.26.0 has a stable release. Once released, I can submit a PR that updates the pinned generator version in Breeze, adds the api.mustache template, and adds the corresponding async unit tests.Fallback to standard httpx (if acceptable): If we want to ship async support sooner under the current 7.25.0 generator, we could use standard httpx. However, as noted in the description, the community is moving toward httpx2.Let me know your thoughts on the best direction forward!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Add async support in `apache-airflow-client` [airflow]

Posted by "kadubhumika (via GitHub)" <gi...@apache.org>.
kadubhumika commented on issue #73661:
URL: https://github.com/apache/airflow/issues/73661#issuecomment-5817380566

   Hi @zach-overflow , I'd be interested in working on this issue.
   I can investigate the current apache-airflow-client generation configuration and add async client support while keeping the existing synchronous interface unchanged. I'll also add/update the relevant tests to cover the generated async client.
   If this is still available, may I proceed with the implementation? Thanks!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Add async support in `apache-airflow-client` [airflow]

Posted by "kadubhumika (via GitHub)" <gi...@apache.org>.
kadubhumika commented on issue #73661:
URL: https://github.com/apache/airflow/issues/73661#issuecomment-6371551780

   @zach-overflow thank you forr the updates i will start working on this iisue fastly.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Add async support in `apache-airflow-client` [airflow]

Posted by "kadubhumika (via GitHub)" <gi...@apache.org>.
kadubhumika commented on issue #73661:
URL: https://github.com/apache/airflow/issues/73661#issuecomment-5855390346

   Should I start working on @zach-overflow ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Add async support in `apache-airflow-client` [airflow]

Posted by "zach-overflow (via GitHub)" <gi...@apache.org>.
zach-overflow commented on issue #73661:
URL: https://github.com/apache/airflow/issues/73661#issuecomment-6371043625

   @kadubhumika looks like [openapi-generator `7.26.0`](https://github.com/OpenAPITools/openapi-generator/releases/tag/v7.26.0) was released, which added `httpx2` support. Feel free to take a shot at it. Two things I think you'll need to watch out for and account for in your implementation:
   1. The openapi-generator, as far as I'm aware, does not natively have a way of merging the identical pydantic model class definitions from the sync and async parts of the generated library, so you will likely need to add a minor post-processing step which ensures both the sync and the async code imports the same model definitions. 
   2. Similarly, the openapi-generator might have some odd function and method renaming if you enable both sync and async pathways. If that's the case, you'll want to make sure that the sync methods/functions are not changed compared to the current python client offering. If the tool does change the function/method names, then you'll need to either adjust the openapi-generator-cli's templating options, or again do some post-processing on the output files.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org