You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by "Ferruzzi, Dennis" <fe...@amazon.com.INVALID> on 2023/03/09 21:42:09 UTC
[AIP-49 OTel] [Discussion] Metrics Provider packages
Hi folks. I am working on adding support for OpenTelemetry based on AIP-49 and I think we have come to a point where it is worth discussing options.
Currently `airflow/stats.py`[1] contains classes for a base/NoStats option as well as Statsd and DataDog, and I will be adding in another option for OTel. I think it is getting to the point where we may want to break these out like we do with provider packages and let the users install only the metrics backend(s) they want.
If we do, then where would they live? Should they fall in with the service provider packages at `airflow/providers/{statsd | datadog | otel}/`, or maybe a new location like `airflow/stats/providers/{statsd | datadog | otel}`? If we make the move, then we would also need to sort out how to handle the change. Perhaps the provider packages for the existing options should be bundled with core for now and later moved to fully-separated like all the other provide packages?
I'd love to hear what you folks think.
- ferruzzi
[1] https://github.com/apache/airflow/blob/main/airflow/stats.py
Re: [AIP-49 OTel] [Discussion] Metrics Provider packages
Posted by Pierre Jeambrun <pi...@gmail.com>.
I really like the idea of having OTEL as the metrics backbone for airflow.
The flexibility and abstraction layer it offers really opens a lot of
opportunities.
I share Jarek's feelings for the current Statsd/Datadog situation,
writing 2 custom collectors that will ensure backward compatibility seems
like a minimal effort to allow us to fully benefit from OTEL and have a
standardized way
for handling metrics across Airflow. (if someone needs to export statsd
metrics to somewhere else for instance)
Do we leave the responsibility to the user to configure valid endpoint for
the exporters, meaning that this is up to him to integrate with his own
environment (persist and observe metrics), or do we also support a
configurable option to bootstrap a backend for basic observability
within airflow ?
Best
Le ven. 17 mars 2023 à 21:50, Jarek Potiuk <ja...@potiuk.com> a écrit :
> Well. Not really "remove" but make it generated via "otel" API. What
> we really care for backwards compatibility is that the statsd metrics
> are produced. But how they are produced is really up to our decision -
> and if open-telemetry is our "go-to" solution, generating Statsd
> metrics as result of the open-telemetry API calls, sounds like a good
> idea (rather than opposite) IMHO.
>
> On Fri, Mar 17, 2023 at 9:39 PM Ferruzzi, Dennis
> <fe...@amazon.com.invalid> wrote:
> >
> > I wasn't expecting support to remove StatsD entirely, I thought it would
> be best to implement them side by side. Whoever implemented the StatsD
> did a nice job of implementing interfaces I can work with. I'd like to
> leave the discussion open a bit and see if anyone else chimes in. If we
> decide to remove StatsD-specific code entirely, then that would be
> basically a final clean-up stage of the project so it won't keep me from
> working while we discuss it.
> >
> >
> > ________________________________
> > From: Howard Yoo <ho...@gmail.com>
> > Sent: Wednesday, March 15, 2023 6:53 PM
> > To: dev@airflow.apache.org
> > Subject: RE: [EXTERNAL][AIP-49 OTel] [Discussion] Metrics Provider
> packages
> >
> > CAUTION: This email originated from outside of the organization. Do not
> click links or open attachments unless you can confirm the sender and know
> the content is safe.
> >
> >
> >
> > I never conceived of the idea of reverting everything in such a radical
> > fashion, because I never thought those actions would even be possible!
> >
> > So having the OTEL as the backbone of Airflow's metrics/logs/traces is
> > actually a great idea, in my opinion. With the usage of OTEL collector
> that
> > can then export the OTEL data into any external format that it can, we
> are
> > definitely safe from introducing complexity.
> >
> > As for the statsd and dogstatsd capability, we should obviously maintain
> it
> > for the backward compatibility, but I do agree that OTEL should be the
> > definite choice by default.
> >
> > On Sun, Mar 12, 2023 at 11:50 AM Jarek Potiuk <ja...@potiuk.com> wrote:
> >
> > > I think that is a good point but I have a conceptually different
> > > proposal. From the coding perspective it might be similar in terms of
> > > implementation and complexity, but long term it will much better
> > > reflect the way how "Airflow as a Platform" currently is implemented.
> > >
> > > My proposal is very close to what was at the center of our recent
> > > discussions on the OpenLineage integration. I think we should revert
> > > the architecture.
> > >
> > > IMHO rather than what we currently have as a common 'stats" interface
> > > should be deprecated and open-telemetry API should be THE common
> > > metrics API airflow should use as "common interface" for metrics (and
> > > anyone who would like to use Airflow metrics should implement). Why
> > > should we define our own "Stats" API, and chose which implementation
> > > should handle this? We already have Open Telemetry API and by Open
> > > Telemetry configuration we can use which collector to use in order to
> > > export the metrics.
> > >
> > > So just to rephrase it - our current Stats/Datadog Statsd
> > > implementation should be merely just our own simple custom
> > > OpenTelemetry Collectors - they should collect the metrics which
> > > airflow sends via OTEL API and send them to Stats/DatadogStats in the
> > > same way current metrics are sent. But those should only be used for
> > > backwards compatibility reasons - we should not aim to implement a
> > > generic "fully-featured/reusable" statsd or DataDog Statsd collector -
> > > just provide a bare minimum that mimics current behaviour for
> > > backwards compatibility. That should be considerably smalller task.
> > >
> > > If we do it this way, then it is rather simple, I think. OpenTelemetry
> > > provider might or might not be a separate provider (we might not need
> > > it eventually that should provide "open telemetry" functionality
> > > in-airflow core - initialization, configuration some common code
> > > etc.). This might well be in `airflow.otel` package - no need for
> > > separate provider there I think.
> > >
> > > Then any external entity (including provider packages) might provide a
> > > collector implementation that will collect the metrics and export
> > > them. In our case (for backported Statsd/DatadogStatsd)- those would
> > > be StatsD and DataDog providers that will provide a collector that
> > > might be configured as the collector used by Airflow's OpenTelemetry.
> > > But the assumption is that the existing collectors for those who
> > > already integrated with Open Telemetry (Grafana, New Relic, Amazon,
> > > Google) should already have collector, that the users should be able
> > > to just configure and use, so for exmaple I do not even expect
> > > anything in AWS provider to collect the metrics from airflow - there
> > > should be an existing CloudWatch OTEL collector, that should collect
> > > the metrics and send them to CloudWatch. The most that should be -
> > > possibly - in AWS provider is the documentation how to enable the
> > > collector for CloudWatch and dependency to pull the cloudwatch
> > > collector package.
> > >
> > > I hope what I am writing makes sense :).
> > >
> > > J.
> > >
> > > On Thu, Mar 9, 2023 at 10:42 PM Ferruzzi, Dennis
> > > <fe...@amazon.com.invalid> wrote:
> > > >
> > > > Hi folks. I am working on adding support for OpenTelemetry based on
> > > AIP-49 and I think we have come to a point where it is worth discussing
> > > options.
> > > >
> > > > Currently `airflow/stats.py`[1] contains classes for a base/NoStats
> > > option as well as Statsd and DataDog, and I will be adding in another
> > > option for OTel. I think it is getting to the point where we may want
> to
> > > break these out like we do with provider packages and let the users
> install
> > > only the metrics backend(s) they want.
> > > >
> > > >
> > > > If we do, then where would they live? Should they fall in with the
> > > service provider packages at `airflow/providers/{statsd | datadog |
> > > otel}/`, or maybe a new location like `airflow/stats/providers/{statsd
> |
> > > datadog | otel}`? If we make the move, then we would also need to
> sort out
> > > how to handle the change. Perhaps the provider packages for the
> existing
> > > options should be bundled with core for now and later moved to
> > > fully-separated like all the other provide packages?
> > > >
> > > > I'd love to hear what you folks think.
> > > > - ferruzzi
> > > >
> > > > [1] https://github.com/apache/airflow/blob/main/airflow/stats.py
> > > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > For additional commands, e-mail: dev-help@airflow.apache.org
> > >
> > >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> For additional commands, e-mail: dev-help@airflow.apache.org
>
>
Re: [AIP-49 OTel] [Discussion] Metrics Provider packages
Posted by Jarek Potiuk <ja...@potiuk.com>.
Well. Not really "remove" but make it generated via "otel" API. What
we really care for backwards compatibility is that the statsd metrics
are produced. But how they are produced is really up to our decision -
and if open-telemetry is our "go-to" solution, generating Statsd
metrics as result of the open-telemetry API calls, sounds like a good
idea (rather than opposite) IMHO.
On Fri, Mar 17, 2023 at 9:39 PM Ferruzzi, Dennis
<fe...@amazon.com.invalid> wrote:
>
> I wasn't expecting support to remove StatsD entirely, I thought it would be best to implement them side by side. Whoever implemented the StatsD did a nice job of implementing interfaces I can work with. I'd like to leave the discussion open a bit and see if anyone else chimes in. If we decide to remove StatsD-specific code entirely, then that would be basically a final clean-up stage of the project so it won't keep me from working while we discuss it.
>
>
> ________________________________
> From: Howard Yoo <ho...@gmail.com>
> Sent: Wednesday, March 15, 2023 6:53 PM
> To: dev@airflow.apache.org
> Subject: RE: [EXTERNAL][AIP-49 OTel] [Discussion] Metrics Provider packages
>
> CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
>
>
>
> I never conceived of the idea of reverting everything in such a radical
> fashion, because I never thought those actions would even be possible!
>
> So having the OTEL as the backbone of Airflow's metrics/logs/traces is
> actually a great idea, in my opinion. With the usage of OTEL collector that
> can then export the OTEL data into any external format that it can, we are
> definitely safe from introducing complexity.
>
> As for the statsd and dogstatsd capability, we should obviously maintain it
> for the backward compatibility, but I do agree that OTEL should be the
> definite choice by default.
>
> On Sun, Mar 12, 2023 at 11:50 AM Jarek Potiuk <ja...@potiuk.com> wrote:
>
> > I think that is a good point but I have a conceptually different
> > proposal. From the coding perspective it might be similar in terms of
> > implementation and complexity, but long term it will much better
> > reflect the way how "Airflow as a Platform" currently is implemented.
> >
> > My proposal is very close to what was at the center of our recent
> > discussions on the OpenLineage integration. I think we should revert
> > the architecture.
> >
> > IMHO rather than what we currently have as a common 'stats" interface
> > should be deprecated and open-telemetry API should be THE common
> > metrics API airflow should use as "common interface" for metrics (and
> > anyone who would like to use Airflow metrics should implement). Why
> > should we define our own "Stats" API, and chose which implementation
> > should handle this? We already have Open Telemetry API and by Open
> > Telemetry configuration we can use which collector to use in order to
> > export the metrics.
> >
> > So just to rephrase it - our current Stats/Datadog Statsd
> > implementation should be merely just our own simple custom
> > OpenTelemetry Collectors - they should collect the metrics which
> > airflow sends via OTEL API and send them to Stats/DatadogStats in the
> > same way current metrics are sent. But those should only be used for
> > backwards compatibility reasons - we should not aim to implement a
> > generic "fully-featured/reusable" statsd or DataDog Statsd collector -
> > just provide a bare minimum that mimics current behaviour for
> > backwards compatibility. That should be considerably smalller task.
> >
> > If we do it this way, then it is rather simple, I think. OpenTelemetry
> > provider might or might not be a separate provider (we might not need
> > it eventually that should provide "open telemetry" functionality
> > in-airflow core - initialization, configuration some common code
> > etc.). This might well be in `airflow.otel` package - no need for
> > separate provider there I think.
> >
> > Then any external entity (including provider packages) might provide a
> > collector implementation that will collect the metrics and export
> > them. In our case (for backported Statsd/DatadogStatsd)- those would
> > be StatsD and DataDog providers that will provide a collector that
> > might be configured as the collector used by Airflow's OpenTelemetry.
> > But the assumption is that the existing collectors for those who
> > already integrated with Open Telemetry (Grafana, New Relic, Amazon,
> > Google) should already have collector, that the users should be able
> > to just configure and use, so for exmaple I do not even expect
> > anything in AWS provider to collect the metrics from airflow - there
> > should be an existing CloudWatch OTEL collector, that should collect
> > the metrics and send them to CloudWatch. The most that should be -
> > possibly - in AWS provider is the documentation how to enable the
> > collector for CloudWatch and dependency to pull the cloudwatch
> > collector package.
> >
> > I hope what I am writing makes sense :).
> >
> > J.
> >
> > On Thu, Mar 9, 2023 at 10:42 PM Ferruzzi, Dennis
> > <fe...@amazon.com.invalid> wrote:
> > >
> > > Hi folks. I am working on adding support for OpenTelemetry based on
> > AIP-49 and I think we have come to a point where it is worth discussing
> > options.
> > >
> > > Currently `airflow/stats.py`[1] contains classes for a base/NoStats
> > option as well as Statsd and DataDog, and I will be adding in another
> > option for OTel. I think it is getting to the point where we may want to
> > break these out like we do with provider packages and let the users install
> > only the metrics backend(s) they want.
> > >
> > >
> > > If we do, then where would they live? Should they fall in with the
> > service provider packages at `airflow/providers/{statsd | datadog |
> > otel}/`, or maybe a new location like `airflow/stats/providers/{statsd |
> > datadog | otel}`? If we make the move, then we would also need to sort out
> > how to handle the change. Perhaps the provider packages for the existing
> > options should be bundled with core for now and later moved to
> > fully-separated like all the other provide packages?
> > >
> > > I'd love to hear what you folks think.
> > > - ferruzzi
> > >
> > > [1] https://github.com/apache/airflow/blob/main/airflow/stats.py
> > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > For additional commands, e-mail: dev-help@airflow.apache.org
> >
> >
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
For additional commands, e-mail: dev-help@airflow.apache.org
Re: [AIP-49 OTel] [Discussion] Metrics Provider packages
Posted by "Ferruzzi, Dennis" <fe...@amazon.com.INVALID>.
I wasn't expecting support to remove StatsD entirely, I thought it would be best to implement them side by side. Whoever implemented the StatsD did a nice job of implementing interfaces I can work with. I'd like to leave the discussion open a bit and see if anyone else chimes in. If we decide to remove StatsD-specific code entirely, then that would be basically a final clean-up stage of the project so it won't keep me from working while we discuss it.
________________________________
From: Howard Yoo <ho...@gmail.com>
Sent: Wednesday, March 15, 2023 6:53 PM
To: dev@airflow.apache.org
Subject: RE: [EXTERNAL][AIP-49 OTel] [Discussion] Metrics Provider packages
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
I never conceived of the idea of reverting everything in such a radical
fashion, because I never thought those actions would even be possible!
So having the OTEL as the backbone of Airflow's metrics/logs/traces is
actually a great idea, in my opinion. With the usage of OTEL collector that
can then export the OTEL data into any external format that it can, we are
definitely safe from introducing complexity.
As for the statsd and dogstatsd capability, we should obviously maintain it
for the backward compatibility, but I do agree that OTEL should be the
definite choice by default.
On Sun, Mar 12, 2023 at 11:50 AM Jarek Potiuk <ja...@potiuk.com> wrote:
> I think that is a good point but I have a conceptually different
> proposal. From the coding perspective it might be similar in terms of
> implementation and complexity, but long term it will much better
> reflect the way how "Airflow as a Platform" currently is implemented.
>
> My proposal is very close to what was at the center of our recent
> discussions on the OpenLineage integration. I think we should revert
> the architecture.
>
> IMHO rather than what we currently have as a common 'stats" interface
> should be deprecated and open-telemetry API should be THE common
> metrics API airflow should use as "common interface" for metrics (and
> anyone who would like to use Airflow metrics should implement). Why
> should we define our own "Stats" API, and chose which implementation
> should handle this? We already have Open Telemetry API and by Open
> Telemetry configuration we can use which collector to use in order to
> export the metrics.
>
> So just to rephrase it - our current Stats/Datadog Statsd
> implementation should be merely just our own simple custom
> OpenTelemetry Collectors - they should collect the metrics which
> airflow sends via OTEL API and send them to Stats/DatadogStats in the
> same way current metrics are sent. But those should only be used for
> backwards compatibility reasons - we should not aim to implement a
> generic "fully-featured/reusable" statsd or DataDog Statsd collector -
> just provide a bare minimum that mimics current behaviour for
> backwards compatibility. That should be considerably smalller task.
>
> If we do it this way, then it is rather simple, I think. OpenTelemetry
> provider might or might not be a separate provider (we might not need
> it eventually that should provide "open telemetry" functionality
> in-airflow core - initialization, configuration some common code
> etc.). This might well be in `airflow.otel` package - no need for
> separate provider there I think.
>
> Then any external entity (including provider packages) might provide a
> collector implementation that will collect the metrics and export
> them. In our case (for backported Statsd/DatadogStatsd)- those would
> be StatsD and DataDog providers that will provide a collector that
> might be configured as the collector used by Airflow's OpenTelemetry.
> But the assumption is that the existing collectors for those who
> already integrated with Open Telemetry (Grafana, New Relic, Amazon,
> Google) should already have collector, that the users should be able
> to just configure and use, so for exmaple I do not even expect
> anything in AWS provider to collect the metrics from airflow - there
> should be an existing CloudWatch OTEL collector, that should collect
> the metrics and send them to CloudWatch. The most that should be -
> possibly - in AWS provider is the documentation how to enable the
> collector for CloudWatch and dependency to pull the cloudwatch
> collector package.
>
> I hope what I am writing makes sense :).
>
> J.
>
> On Thu, Mar 9, 2023 at 10:42 PM Ferruzzi, Dennis
> <fe...@amazon.com.invalid> wrote:
> >
> > Hi folks. I am working on adding support for OpenTelemetry based on
> AIP-49 and I think we have come to a point where it is worth discussing
> options.
> >
> > Currently `airflow/stats.py`[1] contains classes for a base/NoStats
> option as well as Statsd and DataDog, and I will be adding in another
> option for OTel. I think it is getting to the point where we may want to
> break these out like we do with provider packages and let the users install
> only the metrics backend(s) they want.
> >
> >
> > If we do, then where would they live? Should they fall in with the
> service provider packages at `airflow/providers/{statsd | datadog |
> otel}/`, or maybe a new location like `airflow/stats/providers/{statsd |
> datadog | otel}`? If we make the move, then we would also need to sort out
> how to handle the change. Perhaps the provider packages for the existing
> options should be bundled with core for now and later moved to
> fully-separated like all the other provide packages?
> >
> > I'd love to hear what you folks think.
> > - ferruzzi
> >
> > [1] https://github.com/apache/airflow/blob/main/airflow/stats.py
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> For additional commands, e-mail: dev-help@airflow.apache.org
>
>
Re: [AIP-49 OTel] [Discussion] Metrics Provider packages
Posted by Howard Yoo <ho...@gmail.com>.
I never conceived of the idea of reverting everything in such a radical
fashion, because I never thought those actions would even be possible!
So having the OTEL as the backbone of Airflow's metrics/logs/traces is
actually a great idea, in my opinion. With the usage of OTEL collector that
can then export the OTEL data into any external format that it can, we are
definitely safe from introducing complexity.
As for the statsd and dogstatsd capability, we should obviously maintain it
for the backward compatibility, but I do agree that OTEL should be the
definite choice by default.
On Sun, Mar 12, 2023 at 11:50 AM Jarek Potiuk <ja...@potiuk.com> wrote:
> I think that is a good point but I have a conceptually different
> proposal. From the coding perspective it might be similar in terms of
> implementation and complexity, but long term it will much better
> reflect the way how "Airflow as a Platform" currently is implemented.
>
> My proposal is very close to what was at the center of our recent
> discussions on the OpenLineage integration. I think we should revert
> the architecture.
>
> IMHO rather than what we currently have as a common 'stats" interface
> should be deprecated and open-telemetry API should be THE common
> metrics API airflow should use as "common interface" for metrics (and
> anyone who would like to use Airflow metrics should implement). Why
> should we define our own "Stats" API, and chose which implementation
> should handle this? We already have Open Telemetry API and by Open
> Telemetry configuration we can use which collector to use in order to
> export the metrics.
>
> So just to rephrase it - our current Stats/Datadog Statsd
> implementation should be merely just our own simple custom
> OpenTelemetry Collectors - they should collect the metrics which
> airflow sends via OTEL API and send them to Stats/DatadogStats in the
> same way current metrics are sent. But those should only be used for
> backwards compatibility reasons - we should not aim to implement a
> generic "fully-featured/reusable" statsd or DataDog Statsd collector -
> just provide a bare minimum that mimics current behaviour for
> backwards compatibility. That should be considerably smalller task.
>
> If we do it this way, then it is rather simple, I think. OpenTelemetry
> provider might or might not be a separate provider (we might not need
> it eventually that should provide "open telemetry" functionality
> in-airflow core - initialization, configuration some common code
> etc.). This might well be in `airflow.otel` package - no need for
> separate provider there I think.
>
> Then any external entity (including provider packages) might provide a
> collector implementation that will collect the metrics and export
> them. In our case (for backported Statsd/DatadogStatsd)- those would
> be StatsD and DataDog providers that will provide a collector that
> might be configured as the collector used by Airflow's OpenTelemetry.
> But the assumption is that the existing collectors for those who
> already integrated with Open Telemetry (Grafana, New Relic, Amazon,
> Google) should already have collector, that the users should be able
> to just configure and use, so for exmaple I do not even expect
> anything in AWS provider to collect the metrics from airflow - there
> should be an existing CloudWatch OTEL collector, that should collect
> the metrics and send them to CloudWatch. The most that should be -
> possibly - in AWS provider is the documentation how to enable the
> collector for CloudWatch and dependency to pull the cloudwatch
> collector package.
>
> I hope what I am writing makes sense :).
>
> J.
>
> On Thu, Mar 9, 2023 at 10:42 PM Ferruzzi, Dennis
> <fe...@amazon.com.invalid> wrote:
> >
> > Hi folks. I am working on adding support for OpenTelemetry based on
> AIP-49 and I think we have come to a point where it is worth discussing
> options.
> >
> > Currently `airflow/stats.py`[1] contains classes for a base/NoStats
> option as well as Statsd and DataDog, and I will be adding in another
> option for OTel. I think it is getting to the point where we may want to
> break these out like we do with provider packages and let the users install
> only the metrics backend(s) they want.
> >
> >
> > If we do, then where would they live? Should they fall in with the
> service provider packages at `airflow/providers/{statsd | datadog |
> otel}/`, or maybe a new location like `airflow/stats/providers/{statsd |
> datadog | otel}`? If we make the move, then we would also need to sort out
> how to handle the change. Perhaps the provider packages for the existing
> options should be bundled with core for now and later moved to
> fully-separated like all the other provide packages?
> >
> > I'd love to hear what you folks think.
> > - ferruzzi
> >
> > [1] https://github.com/apache/airflow/blob/main/airflow/stats.py
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> For additional commands, e-mail: dev-help@airflow.apache.org
>
>
Re: [AIP-49 OTel] [Discussion] Metrics Provider packages
Posted by Jarek Potiuk <ja...@potiuk.com>.
I think that is a good point but I have a conceptually different
proposal. From the coding perspective it might be similar in terms of
implementation and complexity, but long term it will much better
reflect the way how "Airflow as a Platform" currently is implemented.
My proposal is very close to what was at the center of our recent
discussions on the OpenLineage integration. I think we should revert
the architecture.
IMHO rather than what we currently have as a common 'stats" interface
should be deprecated and open-telemetry API should be THE common
metrics API airflow should use as "common interface" for metrics (and
anyone who would like to use Airflow metrics should implement). Why
should we define our own "Stats" API, and chose which implementation
should handle this? We already have Open Telemetry API and by Open
Telemetry configuration we can use which collector to use in order to
export the metrics.
So just to rephrase it - our current Stats/Datadog Statsd
implementation should be merely just our own simple custom
OpenTelemetry Collectors - they should collect the metrics which
airflow sends via OTEL API and send them to Stats/DatadogStats in the
same way current metrics are sent. But those should only be used for
backwards compatibility reasons - we should not aim to implement a
generic "fully-featured/reusable" statsd or DataDog Statsd collector -
just provide a bare minimum that mimics current behaviour for
backwards compatibility. That should be considerably smalller task.
If we do it this way, then it is rather simple, I think. OpenTelemetry
provider might or might not be a separate provider (we might not need
it eventually that should provide "open telemetry" functionality
in-airflow core - initialization, configuration some common code
etc.). This might well be in `airflow.otel` package - no need for
separate provider there I think.
Then any external entity (including provider packages) might provide a
collector implementation that will collect the metrics and export
them. In our case (for backported Statsd/DatadogStatsd)- those would
be StatsD and DataDog providers that will provide a collector that
might be configured as the collector used by Airflow's OpenTelemetry.
But the assumption is that the existing collectors for those who
already integrated with Open Telemetry (Grafana, New Relic, Amazon,
Google) should already have collector, that the users should be able
to just configure and use, so for exmaple I do not even expect
anything in AWS provider to collect the metrics from airflow - there
should be an existing CloudWatch OTEL collector, that should collect
the metrics and send them to CloudWatch. The most that should be -
possibly - in AWS provider is the documentation how to enable the
collector for CloudWatch and dependency to pull the cloudwatch
collector package.
I hope what I am writing makes sense :).
J.
On Thu, Mar 9, 2023 at 10:42 PM Ferruzzi, Dennis
<fe...@amazon.com.invalid> wrote:
>
> Hi folks. I am working on adding support for OpenTelemetry based on AIP-49 and I think we have come to a point where it is worth discussing options.
>
> Currently `airflow/stats.py`[1] contains classes for a base/NoStats option as well as Statsd and DataDog, and I will be adding in another option for OTel. I think it is getting to the point where we may want to break these out like we do with provider packages and let the users install only the metrics backend(s) they want.
>
>
> If we do, then where would they live? Should they fall in with the service provider packages at `airflow/providers/{statsd | datadog | otel}/`, or maybe a new location like `airflow/stats/providers/{statsd | datadog | otel}`? If we make the move, then we would also need to sort out how to handle the change. Perhaps the provider packages for the existing options should be bundled with core for now and later moved to fully-separated like all the other provide packages?
>
> I'd love to hear what you folks think.
> - ferruzzi
>
> [1] https://github.com/apache/airflow/blob/main/airflow/stats.py
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
For additional commands, e-mail: dev-help@airflow.apache.org