You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Howard Yoo <ho...@gmail.com> on 2025/04/14 15:22:16 UTC
[DISCUSSION] Opentelemetry support for LOGS in Apache Airflow
Hi Airflow community members,
Apache Airflow has been moving steadily from conventional statsd based
telemetry into OpenTelemetry (opentelemetry.io), which has gained
popularity and support for the last few years. There has been AIP proposals
which had a 'phased' approach to incorporate
1. Metrics
2. Traces
3. Logs
Based on the current availability and maturity of Otel's support. Since the
metrics emission based on statsd were the fastest low-hanging fruit that we
could implement OTEL metrics support relatively easy, which we did as our
first try.
Second was the Traces. OTEL support on traces for Python was very mature
and well established. Even though Airflow did not have any prior
implementation on emitting Traces, we were able to instrument it and
produce traces on both Airflow specific processes (jobs) (e.g. schedulers,
triggers, executors), as well as task runs.
However, the area for Logs were not done yet, due at that time a relatively
early and immature state of logs support in Otel.
Now, it seems like Logging support on Python has become stable and reliable
that we can use it to emit log messages in Otel format. So, the timing
seems to be quite perfect for us to start up a discussion on it.
Logging support in Airflow is basically provided via 'provider' model where
Airflow itself does not have a specific implementation of how logging is
done, but requests contributors to provide provider that can plug-in their
logging support.
Would we want to move to that direction, of having 'opentelemetry' provider
?
Or do we want to have logging support as more of a native support way?
The reason why I would like to start this discussion thread was to gain
opinions from the community on whether Airflow has the interest to support
OTEL logging first, and then any opinions on specifics of what would be the
best way to support OTEL logging.
Please feel free to reply and add any comments to the thread to voice your
opinions. If there isn't any substantial interest in OTEL logging, well, we
can live without it.
Re: [DISCUSSION] Opentelemetry support for LOGS in Apache Airflow
Posted by Amogh Desai <am...@gmail.com>.
Thanks for sparking the discussion anyways, Howard!
Even though we do not have bandwidth right now I am glad you started this
discussion.
We will revisit this once we are in a better shape with AF3 (may or june :))
Thanks & Regards,
Amogh Desai
On Wed, Apr 16, 2025 at 3:32 AM Howard Yoo <ho...@gmail.com> wrote:
> Hey, absolutely!
> Given the needed focus and attention on Airflow 3.0, I am totally okay with
> revisiting this topic in May or June! I was just wondering if this was
> something that we want to eventually do, when the Airflow enters into the
> world of 3.0.
>
> Regards,
> Howard
>
> On Tue, Apr 15, 2025 at 12:14 PM Ferruzzi, Dennis
> <fe...@amazon.com.invalid> wrote:
>
> > +1 to what Ash, Jarek, Kaxil, and Vikram said above.
> >
> >
> > - ferruzzi
> >
> >
> > ________________________________
> > From: Vikram Koka <vi...@astronomer.io.INVALID>
> > Sent: Tuesday, April 15, 2025 6:57 AM
> > To: dev@airflow.apache.org
> > Subject: RE: [EXT] [DISCUSSION] Opentelemetry support for LOGS in Apache
> > Airflow
> >
> > CAUTION: This email originated from outside of the organization. Do not
> > click links or open attachments unless you can confirm the sender and
> know
> > the content is safe.
> >
> >
> >
> > AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe.
> > Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne
> pouvez
> > pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain
> que
> > le contenu ne présente aucun risque.
> >
> >
> >
> > +1 to what Ash, Jarek, and Kaxil said above.
> >
> > Let's chat in late May or June on this topic.
> >
> > On Tue, Apr 15, 2025 at 7:05 AM Kaxil Naik <ka...@gmail.com> wrote:
> >
> > > Yeah, no strength or bandwidth right now for a discussion. And would
> very
> > > much prefer we focus on getting 3.0.0 out and any bug fixes after that
> > >
> > > On Tue, 15 Apr 2025 at 14:08, Jarek Potiuk <ja...@potiuk.com> wrote:
> > >
> > > > Very much agree with Ash here.
> > > >
> > > > On Tue, Apr 15, 2025 at 10:30 AM Ash Berlin-Taylor <as...@apache.org>
> > > wrote:
> > > >
> > > > > Hi Howard,
> > > > >
> > > > > Your timing is…. Interesting given we are focusing on getting 3.0
> > > shipped
> > > > > and are in the final throws of that.
> > > > >
> > > > > Let's revisit this in May or June once the dust has settled from
> 3.0,
> > > any
> > > > > immediate bug fixes we might need, and give people a little bit of
> > time
> > > > to
> > > > > unwind after the crunch.
> > > > >
> > > > > -ash
> > > > >
> > > > > > On 14 Apr 2025, at 16:22, Howard Yoo <ho...@gmail.com>
> wrote:
> > > > > >
> > > > > > Hi Airflow community members,
> > > > > >
> > > > > > Apache Airflow has been moving steadily from conventional statsd
> > > based
> > > > > > telemetry into OpenTelemetry (opentelemetry.io), which has
> gained
> > > > > > popularity and support for the last few years. There has been AIP
> > > > > proposals
> > > > > > which had a 'phased' approach to incorporate
> > > > > >
> > > > > > 1. Metrics
> > > > > > 2. Traces
> > > > > > 3. Logs
> > > > > >
> > > > > > Based on the current availability and maturity of Otel's support.
> > > Since
> > > > > the
> > > > > > metrics emission based on statsd were the fastest low-hanging
> fruit
> > > > that
> > > > > we
> > > > > > could implement OTEL metrics support relatively easy, which we
> did
> > as
> > > > our
> > > > > > first try.
> > > > > >
> > > > > > Second was the Traces. OTEL support on traces for Python was very
> > > > mature
> > > > > > and well established. Even though Airflow did not have any prior
> > > > > > implementation on emitting Traces, we were able to instrument it
> > and
> > > > > > produce traces on both Airflow specific processes (jobs) (e.g.
> > > > > schedulers,
> > > > > > triggers, executors), as well as task runs.
> > > > > >
> > > > > > However, the area for Logs were not done yet, due at that time a
> > > > > relatively
> > > > > > early and immature state of logs support in Otel.
> > > > > >
> > > > > > Now, it seems like Logging support on Python has become stable
> and
> > > > > reliable
> > > > > > that we can use it to emit log messages in Otel format. So, the
> > > timing
> > > > > > seems to be quite perfect for us to start up a discussion on it.
> > > > > >
> > > > > > Logging support in Airflow is basically provided via 'provider'
> > model
> > > > > where
> > > > > > Airflow itself does not have a specific implementation of how
> > logging
> > > > is
> > > > > > done, but requests contributors to provide provider that can
> > plug-in
> > > > > their
> > > > > > logging support.
> > > > > >
> > > > > > Would we want to move to that direction, of having
> 'opentelemetry'
> > > > > provider
> > > > > > ?
> > > > > > Or do we want to have logging support as more of a native support
> > > way?
> > > > > >
> > > > > > The reason why I would like to start this discussion thread was
> to
> > > gain
> > > > > > opinions from the community on whether Airflow has the interest
> to
> > > > > support
> > > > > > OTEL logging first, and then any opinions on specifics of what
> > would
> > > be
> > > > > the
> > > > > > best way to support OTEL logging.
> > > > > >
> > > > > > Please feel free to reply and add any comments to the thread to
> > voice
> > > > > your
> > > > > > opinions. If there isn't any substantial interest in OTEL
> logging,
> > > > well,
> > > > > we
> > > > > > can live without it.
> > > > >
> > > > >
> > > > >
> ---------------------------------------------------------------------
> > > > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > > > For additional commands, e-mail: dev-help@airflow.apache.org
> > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSSION] Opentelemetry support for LOGS in Apache Airflow
Posted by Howard Yoo <ho...@gmail.com>.
Hey, absolutely!
Given the needed focus and attention on Airflow 3.0, I am totally okay with
revisiting this topic in May or June! I was just wondering if this was
something that we want to eventually do, when the Airflow enters into the
world of 3.0.
Regards,
Howard
On Tue, Apr 15, 2025 at 12:14 PM Ferruzzi, Dennis
<fe...@amazon.com.invalid> wrote:
> +1 to what Ash, Jarek, Kaxil, and Vikram said above.
>
>
> - ferruzzi
>
>
> ________________________________
> From: Vikram Koka <vi...@astronomer.io.INVALID>
> Sent: Tuesday, April 15, 2025 6:57 AM
> To: dev@airflow.apache.org
> Subject: RE: [EXT] [DISCUSSION] Opentelemetry support for LOGS in Apache
> Airflow
>
> CAUTION: This email originated from outside of the organization. Do not
> click links or open attachments unless you can confirm the sender and know
> the content is safe.
>
>
>
> AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe.
> Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez
> pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que
> le contenu ne présente aucun risque.
>
>
>
> +1 to what Ash, Jarek, and Kaxil said above.
>
> Let's chat in late May or June on this topic.
>
> On Tue, Apr 15, 2025 at 7:05 AM Kaxil Naik <ka...@gmail.com> wrote:
>
> > Yeah, no strength or bandwidth right now for a discussion. And would very
> > much prefer we focus on getting 3.0.0 out and any bug fixes after that
> >
> > On Tue, 15 Apr 2025 at 14:08, Jarek Potiuk <ja...@potiuk.com> wrote:
> >
> > > Very much agree with Ash here.
> > >
> > > On Tue, Apr 15, 2025 at 10:30 AM Ash Berlin-Taylor <as...@apache.org>
> > wrote:
> > >
> > > > Hi Howard,
> > > >
> > > > Your timing is…. Interesting given we are focusing on getting 3.0
> > shipped
> > > > and are in the final throws of that.
> > > >
> > > > Let's revisit this in May or June once the dust has settled from 3.0,
> > any
> > > > immediate bug fixes we might need, and give people a little bit of
> time
> > > to
> > > > unwind after the crunch.
> > > >
> > > > -ash
> > > >
> > > > > On 14 Apr 2025, at 16:22, Howard Yoo <ho...@gmail.com> wrote:
> > > > >
> > > > > Hi Airflow community members,
> > > > >
> > > > > Apache Airflow has been moving steadily from conventional statsd
> > based
> > > > > telemetry into OpenTelemetry (opentelemetry.io), which has gained
> > > > > popularity and support for the last few years. There has been AIP
> > > > proposals
> > > > > which had a 'phased' approach to incorporate
> > > > >
> > > > > 1. Metrics
> > > > > 2. Traces
> > > > > 3. Logs
> > > > >
> > > > > Based on the current availability and maturity of Otel's support.
> > Since
> > > > the
> > > > > metrics emission based on statsd were the fastest low-hanging fruit
> > > that
> > > > we
> > > > > could implement OTEL metrics support relatively easy, which we did
> as
> > > our
> > > > > first try.
> > > > >
> > > > > Second was the Traces. OTEL support on traces for Python was very
> > > mature
> > > > > and well established. Even though Airflow did not have any prior
> > > > > implementation on emitting Traces, we were able to instrument it
> and
> > > > > produce traces on both Airflow specific processes (jobs) (e.g.
> > > > schedulers,
> > > > > triggers, executors), as well as task runs.
> > > > >
> > > > > However, the area for Logs were not done yet, due at that time a
> > > > relatively
> > > > > early and immature state of logs support in Otel.
> > > > >
> > > > > Now, it seems like Logging support on Python has become stable and
> > > > reliable
> > > > > that we can use it to emit log messages in Otel format. So, the
> > timing
> > > > > seems to be quite perfect for us to start up a discussion on it.
> > > > >
> > > > > Logging support in Airflow is basically provided via 'provider'
> model
> > > > where
> > > > > Airflow itself does not have a specific implementation of how
> logging
> > > is
> > > > > done, but requests contributors to provide provider that can
> plug-in
> > > > their
> > > > > logging support.
> > > > >
> > > > > Would we want to move to that direction, of having 'opentelemetry'
> > > > provider
> > > > > ?
> > > > > Or do we want to have logging support as more of a native support
> > way?
> > > > >
> > > > > The reason why I would like to start this discussion thread was to
> > gain
> > > > > opinions from the community on whether Airflow has the interest to
> > > > support
> > > > > OTEL logging first, and then any opinions on specifics of what
> would
> > be
> > > > the
> > > > > best way to support OTEL logging.
> > > > >
> > > > > Please feel free to reply and add any comments to the thread to
> voice
> > > > your
> > > > > opinions. If there isn't any substantial interest in OTEL logging,
> > > well,
> > > > we
> > > > > can live without it.
> > > >
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > > For additional commands, e-mail: dev-help@airflow.apache.org
> > > >
> > > >
> > >
> >
>
Re: [DISCUSSION] Opentelemetry support for LOGS in Apache Airflow
Posted by "Ferruzzi, Dennis" <fe...@amazon.com.INVALID>.
+1 to what Ash, Jarek, Kaxil, and Vikram said above.
- ferruzzi
________________________________
From: Vikram Koka <vi...@astronomer.io.INVALID>
Sent: Tuesday, April 15, 2025 6:57 AM
To: dev@airflow.apache.org
Subject: RE: [EXT] [DISCUSSION] Opentelemetry support for LOGS in Apache Airflow
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe. Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que le contenu ne présente aucun risque.
+1 to what Ash, Jarek, and Kaxil said above.
Let's chat in late May or June on this topic.
On Tue, Apr 15, 2025 at 7:05 AM Kaxil Naik <ka...@gmail.com> wrote:
> Yeah, no strength or bandwidth right now for a discussion. And would very
> much prefer we focus on getting 3.0.0 out and any bug fixes after that
>
> On Tue, 15 Apr 2025 at 14:08, Jarek Potiuk <ja...@potiuk.com> wrote:
>
> > Very much agree with Ash here.
> >
> > On Tue, Apr 15, 2025 at 10:30 AM Ash Berlin-Taylor <as...@apache.org>
> wrote:
> >
> > > Hi Howard,
> > >
> > > Your timing is…. Interesting given we are focusing on getting 3.0
> shipped
> > > and are in the final throws of that.
> > >
> > > Let's revisit this in May or June once the dust has settled from 3.0,
> any
> > > immediate bug fixes we might need, and give people a little bit of time
> > to
> > > unwind after the crunch.
> > >
> > > -ash
> > >
> > > > On 14 Apr 2025, at 16:22, Howard Yoo <ho...@gmail.com> wrote:
> > > >
> > > > Hi Airflow community members,
> > > >
> > > > Apache Airflow has been moving steadily from conventional statsd
> based
> > > > telemetry into OpenTelemetry (opentelemetry.io), which has gained
> > > > popularity and support for the last few years. There has been AIP
> > > proposals
> > > > which had a 'phased' approach to incorporate
> > > >
> > > > 1. Metrics
> > > > 2. Traces
> > > > 3. Logs
> > > >
> > > > Based on the current availability and maturity of Otel's support.
> Since
> > > the
> > > > metrics emission based on statsd were the fastest low-hanging fruit
> > that
> > > we
> > > > could implement OTEL metrics support relatively easy, which we did as
> > our
> > > > first try.
> > > >
> > > > Second was the Traces. OTEL support on traces for Python was very
> > mature
> > > > and well established. Even though Airflow did not have any prior
> > > > implementation on emitting Traces, we were able to instrument it and
> > > > produce traces on both Airflow specific processes (jobs) (e.g.
> > > schedulers,
> > > > triggers, executors), as well as task runs.
> > > >
> > > > However, the area for Logs were not done yet, due at that time a
> > > relatively
> > > > early and immature state of logs support in Otel.
> > > >
> > > > Now, it seems like Logging support on Python has become stable and
> > > reliable
> > > > that we can use it to emit log messages in Otel format. So, the
> timing
> > > > seems to be quite perfect for us to start up a discussion on it.
> > > >
> > > > Logging support in Airflow is basically provided via 'provider' model
> > > where
> > > > Airflow itself does not have a specific implementation of how logging
> > is
> > > > done, but requests contributors to provide provider that can plug-in
> > > their
> > > > logging support.
> > > >
> > > > Would we want to move to that direction, of having 'opentelemetry'
> > > provider
> > > > ?
> > > > Or do we want to have logging support as more of a native support
> way?
> > > >
> > > > The reason why I would like to start this discussion thread was to
> gain
> > > > opinions from the community on whether Airflow has the interest to
> > > support
> > > > OTEL logging first, and then any opinions on specifics of what would
> be
> > > the
> > > > best way to support OTEL logging.
> > > >
> > > > Please feel free to reply and add any comments to the thread to voice
> > > your
> > > > opinions. If there isn't any substantial interest in OTEL logging,
> > well,
> > > we
> > > > can live without it.
> > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > For additional commands, e-mail: dev-help@airflow.apache.org
> > >
> > >
> >
>
Re: [DISCUSSION] Opentelemetry support for LOGS in Apache Airflow
Posted by Vikram Koka <vi...@astronomer.io.INVALID>.
+1 to what Ash, Jarek, and Kaxil said above.
Let's chat in late May or June on this topic.
On Tue, Apr 15, 2025 at 7:05 AM Kaxil Naik <ka...@gmail.com> wrote:
> Yeah, no strength or bandwidth right now for a discussion. And would very
> much prefer we focus on getting 3.0.0 out and any bug fixes after that
>
> On Tue, 15 Apr 2025 at 14:08, Jarek Potiuk <ja...@potiuk.com> wrote:
>
> > Very much agree with Ash here.
> >
> > On Tue, Apr 15, 2025 at 10:30 AM Ash Berlin-Taylor <as...@apache.org>
> wrote:
> >
> > > Hi Howard,
> > >
> > > Your timing is…. Interesting given we are focusing on getting 3.0
> shipped
> > > and are in the final throws of that.
> > >
> > > Let's revisit this in May or June once the dust has settled from 3.0,
> any
> > > immediate bug fixes we might need, and give people a little bit of time
> > to
> > > unwind after the crunch.
> > >
> > > -ash
> > >
> > > > On 14 Apr 2025, at 16:22, Howard Yoo <ho...@gmail.com> wrote:
> > > >
> > > > Hi Airflow community members,
> > > >
> > > > Apache Airflow has been moving steadily from conventional statsd
> based
> > > > telemetry into OpenTelemetry (opentelemetry.io), which has gained
> > > > popularity and support for the last few years. There has been AIP
> > > proposals
> > > > which had a 'phased' approach to incorporate
> > > >
> > > > 1. Metrics
> > > > 2. Traces
> > > > 3. Logs
> > > >
> > > > Based on the current availability and maturity of Otel's support.
> Since
> > > the
> > > > metrics emission based on statsd were the fastest low-hanging fruit
> > that
> > > we
> > > > could implement OTEL metrics support relatively easy, which we did as
> > our
> > > > first try.
> > > >
> > > > Second was the Traces. OTEL support on traces for Python was very
> > mature
> > > > and well established. Even though Airflow did not have any prior
> > > > implementation on emitting Traces, we were able to instrument it and
> > > > produce traces on both Airflow specific processes (jobs) (e.g.
> > > schedulers,
> > > > triggers, executors), as well as task runs.
> > > >
> > > > However, the area for Logs were not done yet, due at that time a
> > > relatively
> > > > early and immature state of logs support in Otel.
> > > >
> > > > Now, it seems like Logging support on Python has become stable and
> > > reliable
> > > > that we can use it to emit log messages in Otel format. So, the
> timing
> > > > seems to be quite perfect for us to start up a discussion on it.
> > > >
> > > > Logging support in Airflow is basically provided via 'provider' model
> > > where
> > > > Airflow itself does not have a specific implementation of how logging
> > is
> > > > done, but requests contributors to provide provider that can plug-in
> > > their
> > > > logging support.
> > > >
> > > > Would we want to move to that direction, of having 'opentelemetry'
> > > provider
> > > > ?
> > > > Or do we want to have logging support as more of a native support
> way?
> > > >
> > > > The reason why I would like to start this discussion thread was to
> gain
> > > > opinions from the community on whether Airflow has the interest to
> > > support
> > > > OTEL logging first, and then any opinions on specifics of what would
> be
> > > the
> > > > best way to support OTEL logging.
> > > >
> > > > Please feel free to reply and add any comments to the thread to voice
> > > your
> > > > opinions. If there isn't any substantial interest in OTEL logging,
> > well,
> > > we
> > > > can live without it.
> > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > For additional commands, e-mail: dev-help@airflow.apache.org
> > >
> > >
> >
>
Re: [DISCUSSION] Opentelemetry support for LOGS in Apache Airflow
Posted by Kaxil Naik <ka...@gmail.com>.
Yeah, no strength or bandwidth right now for a discussion. And would very
much prefer we focus on getting 3.0.0 out and any bug fixes after that
On Tue, 15 Apr 2025 at 14:08, Jarek Potiuk <ja...@potiuk.com> wrote:
> Very much agree with Ash here.
>
> On Tue, Apr 15, 2025 at 10:30 AM Ash Berlin-Taylor <as...@apache.org> wrote:
>
> > Hi Howard,
> >
> > Your timing is…. Interesting given we are focusing on getting 3.0 shipped
> > and are in the final throws of that.
> >
> > Let's revisit this in May or June once the dust has settled from 3.0, any
> > immediate bug fixes we might need, and give people a little bit of time
> to
> > unwind after the crunch.
> >
> > -ash
> >
> > > On 14 Apr 2025, at 16:22, Howard Yoo <ho...@gmail.com> wrote:
> > >
> > > Hi Airflow community members,
> > >
> > > Apache Airflow has been moving steadily from conventional statsd based
> > > telemetry into OpenTelemetry (opentelemetry.io), which has gained
> > > popularity and support for the last few years. There has been AIP
> > proposals
> > > which had a 'phased' approach to incorporate
> > >
> > > 1. Metrics
> > > 2. Traces
> > > 3. Logs
> > >
> > > Based on the current availability and maturity of Otel's support. Since
> > the
> > > metrics emission based on statsd were the fastest low-hanging fruit
> that
> > we
> > > could implement OTEL metrics support relatively easy, which we did as
> our
> > > first try.
> > >
> > > Second was the Traces. OTEL support on traces for Python was very
> mature
> > > and well established. Even though Airflow did not have any prior
> > > implementation on emitting Traces, we were able to instrument it and
> > > produce traces on both Airflow specific processes (jobs) (e.g.
> > schedulers,
> > > triggers, executors), as well as task runs.
> > >
> > > However, the area for Logs were not done yet, due at that time a
> > relatively
> > > early and immature state of logs support in Otel.
> > >
> > > Now, it seems like Logging support on Python has become stable and
> > reliable
> > > that we can use it to emit log messages in Otel format. So, the timing
> > > seems to be quite perfect for us to start up a discussion on it.
> > >
> > > Logging support in Airflow is basically provided via 'provider' model
> > where
> > > Airflow itself does not have a specific implementation of how logging
> is
> > > done, but requests contributors to provide provider that can plug-in
> > their
> > > logging support.
> > >
> > > Would we want to move to that direction, of having 'opentelemetry'
> > provider
> > > ?
> > > Or do we want to have logging support as more of a native support way?
> > >
> > > The reason why I would like to start this discussion thread was to gain
> > > opinions from the community on whether Airflow has the interest to
> > support
> > > OTEL logging first, and then any opinions on specifics of what would be
> > the
> > > best way to support OTEL logging.
> > >
> > > Please feel free to reply and add any comments to the thread to voice
> > your
> > > opinions. If there isn't any substantial interest in OTEL logging,
> well,
> > we
> > > can live without it.
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > For additional commands, e-mail: dev-help@airflow.apache.org
> >
> >
>
Re: [DISCUSSION] Opentelemetry support for LOGS in Apache Airflow
Posted by Jarek Potiuk <ja...@potiuk.com>.
Very much agree with Ash here.
On Tue, Apr 15, 2025 at 10:30 AM Ash Berlin-Taylor <as...@apache.org> wrote:
> Hi Howard,
>
> Your timing is…. Interesting given we are focusing on getting 3.0 shipped
> and are in the final throws of that.
>
> Let's revisit this in May or June once the dust has settled from 3.0, any
> immediate bug fixes we might need, and give people a little bit of time to
> unwind after the crunch.
>
> -ash
>
> > On 14 Apr 2025, at 16:22, Howard Yoo <ho...@gmail.com> wrote:
> >
> > Hi Airflow community members,
> >
> > Apache Airflow has been moving steadily from conventional statsd based
> > telemetry into OpenTelemetry (opentelemetry.io), which has gained
> > popularity and support for the last few years. There has been AIP
> proposals
> > which had a 'phased' approach to incorporate
> >
> > 1. Metrics
> > 2. Traces
> > 3. Logs
> >
> > Based on the current availability and maturity of Otel's support. Since
> the
> > metrics emission based on statsd were the fastest low-hanging fruit that
> we
> > could implement OTEL metrics support relatively easy, which we did as our
> > first try.
> >
> > Second was the Traces. OTEL support on traces for Python was very mature
> > and well established. Even though Airflow did not have any prior
> > implementation on emitting Traces, we were able to instrument it and
> > produce traces on both Airflow specific processes (jobs) (e.g.
> schedulers,
> > triggers, executors), as well as task runs.
> >
> > However, the area for Logs were not done yet, due at that time a
> relatively
> > early and immature state of logs support in Otel.
> >
> > Now, it seems like Logging support on Python has become stable and
> reliable
> > that we can use it to emit log messages in Otel format. So, the timing
> > seems to be quite perfect for us to start up a discussion on it.
> >
> > Logging support in Airflow is basically provided via 'provider' model
> where
> > Airflow itself does not have a specific implementation of how logging is
> > done, but requests contributors to provide provider that can plug-in
> their
> > logging support.
> >
> > Would we want to move to that direction, of having 'opentelemetry'
> provider
> > ?
> > Or do we want to have logging support as more of a native support way?
> >
> > The reason why I would like to start this discussion thread was to gain
> > opinions from the community on whether Airflow has the interest to
> support
> > OTEL logging first, and then any opinions on specifics of what would be
> the
> > best way to support OTEL logging.
> >
> > Please feel free to reply and add any comments to the thread to voice
> your
> > opinions. If there isn't any substantial interest in OTEL logging, well,
> we
> > can live without it.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> For additional commands, e-mail: dev-help@airflow.apache.org
>
>
Re: [DISCUSSION] Opentelemetry support for LOGS in Apache Airflow
Posted by Ash Berlin-Taylor <as...@apache.org>.
Hi Howard,
Your timing is…. Interesting given we are focusing on getting 3.0 shipped and are in the final throws of that.
Let's revisit this in May or June once the dust has settled from 3.0, any immediate bug fixes we might need, and give people a little bit of time to unwind after the crunch.
-ash
> On 14 Apr 2025, at 16:22, Howard Yoo <ho...@gmail.com> wrote:
>
> Hi Airflow community members,
>
> Apache Airflow has been moving steadily from conventional statsd based
> telemetry into OpenTelemetry (opentelemetry.io), which has gained
> popularity and support for the last few years. There has been AIP proposals
> which had a 'phased' approach to incorporate
>
> 1. Metrics
> 2. Traces
> 3. Logs
>
> Based on the current availability and maturity of Otel's support. Since the
> metrics emission based on statsd were the fastest low-hanging fruit that we
> could implement OTEL metrics support relatively easy, which we did as our
> first try.
>
> Second was the Traces. OTEL support on traces for Python was very mature
> and well established. Even though Airflow did not have any prior
> implementation on emitting Traces, we were able to instrument it and
> produce traces on both Airflow specific processes (jobs) (e.g. schedulers,
> triggers, executors), as well as task runs.
>
> However, the area for Logs were not done yet, due at that time a relatively
> early and immature state of logs support in Otel.
>
> Now, it seems like Logging support on Python has become stable and reliable
> that we can use it to emit log messages in Otel format. So, the timing
> seems to be quite perfect for us to start up a discussion on it.
>
> Logging support in Airflow is basically provided via 'provider' model where
> Airflow itself does not have a specific implementation of how logging is
> done, but requests contributors to provide provider that can plug-in their
> logging support.
>
> Would we want to move to that direction, of having 'opentelemetry' provider
> ?
> Or do we want to have logging support as more of a native support way?
>
> The reason why I would like to start this discussion thread was to gain
> opinions from the community on whether Airflow has the interest to support
> OTEL logging first, and then any opinions on specifics of what would be the
> best way to support OTEL logging.
>
> Please feel free to reply and add any comments to the thread to voice your
> opinions. If there isn't any substantial interest in OTEL logging, well, we
> can live without it.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
For additional commands, e-mail: dev-help@airflow.apache.org