[DISCUSS] Distributed tracing using OpenTelemetry

[Brian Putt <pu...@gmail.com>]

Hello Apache NiFi,

I'd like to discuss implementing NIFI-10110 which adds OpenTelemetry
integration into NiFi. Tracing will provide a way to identify
bottlenecks within various flows and propagate trace information to
downstream systems (whether they're another NiFi cluster or otherwise).

I wanted to get the community's feedback to see if this makes sense and if
it's something that you'd find valuable. If so, would anyone be interested
in discussing implementation details? We'd certainly be willing to
implement this feature, but feel it will need feedback from the community
as we add the integration.

Some key areas around implementation will involve where it makes sense to
add the OpenTelemetry wrapper because ideally, processors get tracing 'for
free' and this then leads to other questions as to sampling and how to
expose flowfile attributes into span tags that get emitted.

OpenTelemetry offers a Java library that's licensed under Apache 2.

Re: [DISCUSS] Distributed tracing using OpenTelemetry

[Pierre Villard <pi...@gmail.com>]

Hi,

Monitoring in NiFi is one of my favorite topics and I do think
OpenTelemetry would be of great value. There is never too much
observability in a software like NiFi when it comes to integrating anything
with anything. I'm more than happy to be part of the discussion in terms of
requirements/implementation, etc. Getting better insights from the
provenance data of NiFi is probably where I'd start.

Pierre

Le jeu. 16 juin 2022 à 10:44, Isha Lamboo <is...@virtualsciences.nl>
a écrit :

> Hi Brian,
>
> Thank you for bringing this up. I see value here for NiFi deployments in
> larger (enterprise) environments, where NiFi runs alongside other
> integration and data management solutions that also support (or plan to
> support) OpenTelemetry. Moving data between on-prem legacy applications and
> cloud data lakes is a good example. It currently takes a lot of work to
> correlate NiFi logging with that of Azure Data Factory (and friends), so in
> practice it often doesn't get done.
>
> I don't have the technical depth to really talk implementation, but from a
> functional perspective a generic wrapper that leverages existing
> flowfile/provenance data would make sense and I only would want the wrapper
> to look for an "opentelemetry.label" attribute to easily identify a
> specific flow and correlate it with the same "flow" in other systems.
>
> Regards,
>
> Isha
>
> -----Oorspronkelijk bericht-----
> Van: Brian Putt <pu...@gmail.com>
> Verzonden: woensdag 15 juni 2022 21:57
> Aan: dev@nifi.apache.org
> Onderwerp: [DISCUSS] Distributed tracing using OpenTelemetry
>
> Hello Apache NiFi,
>
> I'd like to discuss implementing NIFI-10110 which adds OpenTelemetry
> integration into NiFi. Tracing will provide a way to identify bottlenecks
> within various flows and propagate trace information to downstream systems
> (whether they're another NiFi cluster or otherwise).
>
> I wanted to get the community's feedback to see if this makes sense and if
> it's something that you'd find valuable. If so, would anyone be interested
> in discussing implementation details? We'd certainly be willing to
> implement this feature, but feel it will need feedback from the community
> as we add the integration.
>
> Some key areas around implementation will involve where it makes sense to
> add the OpenTelemetry wrapper because ideally, processors get tracing 'for
> free' and this then leads to other questions as to sampling and how to
> expose flowfile attributes into span tags that get emitted.
>
> OpenTelemetry offers a Java library that's licensed under Apache 2.
>

RE: [DISCUSS] Distributed tracing using OpenTelemetry

[Isha Lamboo <is...@virtualsciences.nl>]

Hi Brian,

Thank you for bringing this up. I see value here for NiFi deployments in larger (enterprise) environments, where NiFi runs alongside other integration and data management solutions that also support (or plan to support) OpenTelemetry. Moving data between on-prem legacy applications and cloud data lakes is a good example. It currently takes a lot of work to correlate NiFi logging with that of Azure Data Factory (and friends), so in practice it often doesn't get done.

I don't have the technical depth to really talk implementation, but from a functional perspective a generic wrapper that leverages existing flowfile/provenance data would make sense and I only would want the wrapper to look for an "opentelemetry.label" attribute to easily identify a specific flow and correlate it with the same "flow" in other systems.

Regards,

Isha

-----Oorspronkelijk bericht-----
Van: Brian Putt <pu...@gmail.com> 
Verzonden: woensdag 15 juni 2022 21:57
Aan: dev@nifi.apache.org
Onderwerp: [DISCUSS] Distributed tracing using OpenTelemetry

Hello Apache NiFi,

I'd like to discuss implementing NIFI-10110 which adds OpenTelemetry integration into NiFi. Tracing will provide a way to identify bottlenecks within various flows and propagate trace information to downstream systems (whether they're another NiFi cluster or otherwise).

I wanted to get the community's feedback to see if this makes sense and if it's something that you'd find valuable. If so, would anyone be interested in discussing implementation details? We'd certainly be willing to implement this feature, but feel it will need feedback from the community as we add the integration.

Some key areas around implementation will involve where it makes sense to add the OpenTelemetry wrapper because ideally, processors get tracing 'for free' and this then leads to other questions as to sampling and how to expose flowfile attributes into span tags that get emitted.

OpenTelemetry offers a Java library that's licensed under Apache 2.