You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Pierre Jeambrun <pi...@gmail.com> on 2026/07/06 09:16:15 UTC
Re: [DISCUSS] Configuration design for REST API & UI metrics tagging
Thanks for starting the discussion Henry,
I would be for option B as well, which seems to have more voices. (Also we
can alway offer extensibility later if that's proven to be requested).
To make this move forward I believe you can start a lazy consensus for
Option B and see what happens.
Cheers,
On Thu, Jun 25, 2026 at 11:54 PM Niko Oliveira <on...@apache.org> wrote:
> Late to the game here, but huge +1 to option B with Vincent's optimization
> as well.
>
> Cheers,
> Niko
>
> On Mon, Jun 22, 2026 at 7:34 AM Ash Berlin-Taylor <as...@apache.org> wrote:
>
> > As mentioned in the PR , I agree with Vincent and said almost word for
> > word in the PR.
> >
> > Option A feels like a massive complexity for almost no real use case.
> >
> > -ash
> >
> > > On 22 Jun 2026, at 14:37, Vincent Beck <vi...@apache.org> wrote:
> > >
> > > Hi Henry,
> > >
> > > On my side I lean towards option B, in my opinion users do not need to
> > configure such mapping, this is something Airflow can dictate. That would
> > avoid yet another option/complexity in Airflow. Also, going deeper in
> that
> > direction, do we actually need a mapping? Can we not just put the API
> path
> > as tag instead of having a mapping? These metrics are going to be read by
> > deployment manager, I do not know if providing a proxy name instead of
> the
> > API path would be actually helpful.
> > >
> > > On 2026/06/21 13:27:15 "Zhe-You(Jason) Liu" wrote:
> > >> Hi Henry,
> > >>
> > >> Thanks for working on this interesting feature for the upcoming 3.3
> > Airflow
> > >> release.
> > >> Yes, I lean toward option A.
> > >>
> > >> Additionally, we could even make the API server disable the metrics
> > >> middleware if user explicitly sets the `api_path_prefix_to_surface` as
> > >> None,
> > >> With option A plus the None-aware handling, we can support the metrics
> > >> middleware with the further extensibility and the feature toggling.
> > >>
> > >> Best,
> > >> Jason
> > >>
> > >> On Sun, Jun 21, 2026 at 3:26 PM Henry Chen <he...@gmail.com>
> > wrote:
> > >>
> > >>> Hi everyone,
> > >>>
> > >>> I would like to start a discussion regarding the configuration design
> > for
> > >>> the new REST API and UI metrics feature currently being proposed in
> PR
> > >>> #64523 <https://github.com/apache/airflow/pull/64523>.
> > >>>
> > >>> The core capability adds request monitoring (QPS, latency, and
> errors)
> > to
> > >>> Airflow’s FastAPI/Starlette stack. While there is general agreement
> > that
> > >>> having these metrics in production would be highly valuable, we have
> > hit a
> > >>> architectural design question regarding how users should configure
> > these
> > >>> metrics, and we’d love to get the community's feedback.
> > >>> The Core Question
> > >>>
> > >>> How flexible should the mapping between URL prefixes and metric tags
> (
> > >>> api_surface) be for end-users? Currently, we have two design
> > directions:
> > >>> Option A (Current PR Design): Fully User-Configurable via JSON
> > >>>
> > >>> Allows deployment managers to define a custom mapping in airflow.cfg
> > >>> (e.g., {"/api/v2":
> > >>> "public", "/ui": "ui", "/execution": "execution", "/my-plugin":
> > "plugin"}).
> > >>>
> > >>> -
> > >>>
> > >>> *Pros:* High flexibility. It allows users to gain metrics coverage
> > for
> > >>> additional custom mounted APIs, Execution APIs, or third-party
> plugin
> > >>> routes without needing Airflow core code changes.
> > >>> -
> > >>>
> > >>> *Cons:* Adds slightly more complexity to the configuration.
> > >>>
> > >>> Option B (Alternative Suggestion): Hardcoded in Code with Simple
> > Toggles
> > >>>
> > >>> Airflow maintainers hardcode the specific route-to-tag mappings
> (e.g.,
> > >>> strictly /api/v2 and /ui) directly in the codebase. Users would only
> > have a
> > >>> simple boolean flag to turn the metrics on or off, but cannot
> > customize the
> > >>> URL-to-tag mappings.
> > >>>
> > >>> -
> > >>>
> > >>> *Pros:* Simpler configuration, lower cognitive load for the average
> > >>> user.
> > >>> -
> > >>>
> > >>> *Cons:* Lacks extensibility for custom plugins or
> enterprise-specific
> > >>> API extensions.
> > >>>
> > >>> Discussion Context
> > >>>
> > >>> Jason suggested that allowing users to define the mapping themselves
> > >>> (Option A) provides the necessary extensibility for production
> > environments
> > >>> where custom plugins or additional mounted endpoints are heavily
> > utilized.
> > >>> On the other hand, ashb raised concerns about whether users actually
> > need
> > >>> this level of configurability and whether it duplicates existing
> metric
> > >>> filtering mechanisms.
> > >>> I really appreciate Jason, ashb, Pierre, and Jens taking the time to
> > share
> > >>> their insights on this.
> > >>>
> > >>> We would appreciate your thoughts on which approach makes more sense
> > for
> > >>> Airflow's architecture moving forward.
> > >>>
> > >>> You can find the full discussion and implementation details in the PR
> > here:
> > >>> https://github.com/apache/airflow/pull/64523
> > >>>
> > >>> Thanks,
> > >>> Henry
> > >>>
> > >>
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > For additional commands, e-mail: dev-help@airflow.apache.org
> > >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > For additional commands, e-mail: dev-help@airflow.apache.org
> >
> >
>
Re: [DISCUSS] Configuration design for REST API & UI metrics tagging
Posted by Jarek Potiuk <ja...@potiuk.com>.
+1 On B. It's a useful feature, but I think we should be opinionated
here. What we gain (small gain) in configurability for some users is
outweighed by the complexity of managing a wildly configurable
solution.
On Mon, Jul 6, 2026 at 11:17 AM Pierre Jeambrun <pi...@gmail.com> wrote:
>
> Thanks for starting the discussion Henry,
>
> I would be for option B as well, which seems to have more voices. (Also we
> can alway offer extensibility later if that's proven to be requested).
>
> To make this move forward I believe you can start a lazy consensus for
> Option B and see what happens.
>
> Cheers,
>
>
>
> On Thu, Jun 25, 2026 at 11:54 PM Niko Oliveira <on...@apache.org> wrote:
>
> > Late to the game here, but huge +1 to option B with Vincent's optimization
> > as well.
> >
> > Cheers,
> > Niko
> >
> > On Mon, Jun 22, 2026 at 7:34 AM Ash Berlin-Taylor <as...@apache.org> wrote:
> >
> > > As mentioned in the PR , I agree with Vincent and said almost word for
> > > word in the PR.
> > >
> > > Option A feels like a massive complexity for almost no real use case.
> > >
> > > -ash
> > >
> > > > On 22 Jun 2026, at 14:37, Vincent Beck <vi...@apache.org> wrote:
> > > >
> > > > Hi Henry,
> > > >
> > > > On my side I lean towards option B, in my opinion users do not need to
> > > configure such mapping, this is something Airflow can dictate. That would
> > > avoid yet another option/complexity in Airflow. Also, going deeper in
> > that
> > > direction, do we actually need a mapping? Can we not just put the API
> > path
> > > as tag instead of having a mapping? These metrics are going to be read by
> > > deployment manager, I do not know if providing a proxy name instead of
> > the
> > > API path would be actually helpful.
> > > >
> > > > On 2026/06/21 13:27:15 "Zhe-You(Jason) Liu" wrote:
> > > >> Hi Henry,
> > > >>
> > > >> Thanks for working on this interesting feature for the upcoming 3.3
> > > Airflow
> > > >> release.
> > > >> Yes, I lean toward option A.
> > > >>
> > > >> Additionally, we could even make the API server disable the metrics
> > > >> middleware if user explicitly sets the `api_path_prefix_to_surface` as
> > > >> None,
> > > >> With option A plus the None-aware handling, we can support the metrics
> > > >> middleware with the further extensibility and the feature toggling.
> > > >>
> > > >> Best,
> > > >> Jason
> > > >>
> > > >> On Sun, Jun 21, 2026 at 3:26 PM Henry Chen <he...@gmail.com>
> > > wrote:
> > > >>
> > > >>> Hi everyone,
> > > >>>
> > > >>> I would like to start a discussion regarding the configuration design
> > > for
> > > >>> the new REST API and UI metrics feature currently being proposed in
> > PR
> > > >>> #64523 <https://github.com/apache/airflow/pull/64523>.
> > > >>>
> > > >>> The core capability adds request monitoring (QPS, latency, and
> > errors)
> > > to
> > > >>> Airflow’s FastAPI/Starlette stack. While there is general agreement
> > > that
> > > >>> having these metrics in production would be highly valuable, we have
> > > hit a
> > > >>> architectural design question regarding how users should configure
> > > these
> > > >>> metrics, and we’d love to get the community's feedback.
> > > >>> The Core Question
> > > >>>
> > > >>> How flexible should the mapping between URL prefixes and metric tags
> > (
> > > >>> api_surface) be for end-users? Currently, we have two design
> > > directions:
> > > >>> Option A (Current PR Design): Fully User-Configurable via JSON
> > > >>>
> > > >>> Allows deployment managers to define a custom mapping in airflow.cfg
> > > >>> (e.g., {"/api/v2":
> > > >>> "public", "/ui": "ui", "/execution": "execution", "/my-plugin":
> > > "plugin"}).
> > > >>>
> > > >>> -
> > > >>>
> > > >>> *Pros:* High flexibility. It allows users to gain metrics coverage
> > > for
> > > >>> additional custom mounted APIs, Execution APIs, or third-party
> > plugin
> > > >>> routes without needing Airflow core code changes.
> > > >>> -
> > > >>>
> > > >>> *Cons:* Adds slightly more complexity to the configuration.
> > > >>>
> > > >>> Option B (Alternative Suggestion): Hardcoded in Code with Simple
> > > Toggles
> > > >>>
> > > >>> Airflow maintainers hardcode the specific route-to-tag mappings
> > (e.g.,
> > > >>> strictly /api/v2 and /ui) directly in the codebase. Users would only
> > > have a
> > > >>> simple boolean flag to turn the metrics on or off, but cannot
> > > customize the
> > > >>> URL-to-tag mappings.
> > > >>>
> > > >>> -
> > > >>>
> > > >>> *Pros:* Simpler configuration, lower cognitive load for the average
> > > >>> user.
> > > >>> -
> > > >>>
> > > >>> *Cons:* Lacks extensibility for custom plugins or
> > enterprise-specific
> > > >>> API extensions.
> > > >>>
> > > >>> Discussion Context
> > > >>>
> > > >>> Jason suggested that allowing users to define the mapping themselves
> > > >>> (Option A) provides the necessary extensibility for production
> > > environments
> > > >>> where custom plugins or additional mounted endpoints are heavily
> > > utilized.
> > > >>> On the other hand, ashb raised concerns about whether users actually
> > > need
> > > >>> this level of configurability and whether it duplicates existing
> > metric
> > > >>> filtering mechanisms.
> > > >>> I really appreciate Jason, ashb, Pierre, and Jens taking the time to
> > > share
> > > >>> their insights on this.
> > > >>>
> > > >>> We would appreciate your thoughts on which approach makes more sense
> > > for
> > > >>> Airflow's architecture moving forward.
> > > >>>
> > > >>> You can find the full discussion and implementation details in the PR
> > > here:
> > > >>> https://github.com/apache/airflow/pull/64523
> > > >>>
> > > >>> Thanks,
> > > >>> Henry
> > > >>>
> > > >>
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > > For additional commands, e-mail: dev-help@airflow.apache.org
> > > >
> > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > For additional commands, e-mail: dev-help@airflow.apache.org
> > >
> > >
> >
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
For additional commands, e-mail: dev-help@airflow.apache.org