Understanding Storm

[Edward Armes <ed...@gmail.com>]

Hi there,

I've been looking into trying to understand various aspects about Storm
including the metrics system. Would anyone here be able to confirm my
understanding.

- The current metrics for the daemons all use the v1 reporting and in the
next major release (2.3.x) will start to use the v2 reporting as well. But
currently and nor will they in future use the "metrics.reporters"
configuration.
- All metrics are collected (including worker metrics) are or will be
collected in the metrics store via a nimbus instance, and this is going
forwarded will be available to be used by the scheduler (and other systems)
to view metrics at a cluster level.
- Workers (Topologies) can be instrumented separately to send metrics to
external systems (Prometheus, Graphite, etc....).
- The metrics store can be an external system, but an a custom adapter is
needed
- Trident topologies are not a direct extension of normal Storm topologies
but are there own abstraction that is exposed in a Spark/Hadoop like way
that is then translated down to Storm components. It is currently however
not possible to include a Trident topology into a standard Storm topology
for places where you need exactly once processing.

Apologies for sending this to the dev list however the user list seems to
be very quiet at the moment.

Thanks

Edward

Re: Understanding Storm

[Aaron Gresch <ag...@gmail.com>]

Regarding metrics:

Daemons use the storm.daemon.metrics.reporter.plugins config for reporting
metrics.  I don't know of any plans to change daemon metrics in the 2.3.0
release.

Worker metrics are reported in both V1 and V2 depending on how the metrics
are registered.  A decent chunk (but not all) of the internal metrics have
been changed to V2.  Workers can still register and use V1 metrics, but
they have been deprecated.

See
https://github.com/apache/storm/blob/master/docs/Metrics.md
https://github.com/apache/storm/blob/master/docs/metrics_v2.md





On Thu, Apr 1, 2021 at 12:31 PM Edward Armes <ed...@gmail.com> wrote:

> Hi there,
>
> I've been looking into trying to understand various aspects about Storm
> including the metrics system. Would anyone here be able to confirm my
> understanding.
>
> - The current metrics for the daemons all use the v1 reporting and in the
> next major release (2.3.x) will start to use the v2 reporting as well. But
> currently and nor will they in future use the "metrics.reporters"
> configuration.
> - All metrics are collected (including worker metrics) are or will be
> collected in the metrics store via a nimbus instance, and this is going
> forwarded will be available to be used by the scheduler (and other systems)
> to view metrics at a cluster level.
> - Workers (Topologies) can be instrumented separately to send metrics to
> external systems (Prometheus, Graphite, etc....).
> - The metrics store can be an external system, but an a custom adapter is
> needed
> - Trident topologies are not a direct extension of normal Storm topologies
> but are there own abstraction that is exposed in a Spark/Hadoop like way
> that is then translated down to Storm components. It is currently however
> not possible to include a Trident topology into a standard Storm topology
> for places where you need exactly once processing.
>
> Apologies for sending this to the dev list however the user list seems to
> be very quiet at the moment.
>
> Thanks
>
> Edward
>