You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Simon Cooper <si...@featurespace.co.uk> on 2018/03/28 12:38:14 UTC
Dynamic metrics in V2 system
Hi,
We're looking at upgrading our topologies to use the new V2 metrics system, however one of our absolutely core requirements is the ability to dynamically create/assign metrics after the topology is started up - we don't know what metrics we'll be running beforehand, and the metrics can change during the lifetime of the topology. We handled this in the V1 system using IMetric wrappers, but the codehale-based metrics system doesn't seem to have the ability to do this.
If the V1 system is removed in a future version, this means we won't be able to use the storm metric system to report our metrics, which is a real shame, as we'll lose all the built-in info that storm provides. We'll need to implement our own system from scratch & basically duplicate all the existing infrastructure.
Is there a way we can create metrics dynamically in the V2 system?
Thanks,
Simon Cooper
This message, and any files/attachments transmitted together with it, is intended for the use only of the person (or persons) to whom it is addressed. It may contain information which is confidential and/or protected by legal privilege. Accordingly, any dissemination, distribution, copying or use of this message, or any part of it or anything sent together with it, other than by intended recipients, may constitute a breach of civil or criminal law and is hereby prohibited. Unless otherwise stated, any views expressed in this message are those of the person sending it and not the sender's employer. No responsibility, legal or otherwise, of whatever nature, is accepted as to the accuracy of the contents of this message or for the completeness of the message as received. Anyone who is not the intended recipient of this message is advised to make no use of it and is requested to contact Featurespace Limited as soon as possible. Any recipient of this message who has knowledge or suspects that it may have been the subject of unauthorised interception or alteration is also requested to contact Featurespace Limited.
RE: Dynamic metrics in V2 system
Posted by Simon Cooper <si...@featurespace.co.uk>.
Great, thanks for the info!
One thing I've noticed is that the storm configuration isn't the full config as provided on topology deployment, it's just the 'base' config as specified in storm.yaml. It also doesn't have any sort of TopologyContext object provided. I'll get some more info, then open a bug report.
Simon
From: Bobby Evans <bo...@apache.org>
Sent: 30 March 2018 14:56
To: user@storm.apache.org
Subject: Re: Dynamic metrics in V2 system
This should work for the most part. The new metrics API is just a thin wrapper around the codahale metrics API that has no restrictions on when you can add or remove metrics. It is separate from storm and it really is just a naming convention that is used to store which worker, component, etc, a metric is associated with.
We have not exposed any of the remove APIs so if this is something you really need please file a JIRA for it and we can look into supporting it too.
Thanks,
Bobby
On Thu, Mar 29, 2018 at 11:46 AM Simon Cooper <si...@featurespace.co.uk>> wrote:
Thanks! Does the new metrics system only create one reporter for the topology, and send the info between workers, or have one reporter per worker?
Simon
From: Jungtaek Lim <ka...@gmail.com>>
Sent: 29 March 2018 16:12
To: user@storm.apache.org<ma...@storm.apache.org>
Subject: Re: Dynamic metrics in V2 system
Hi Simon,
Actually I'm not familiar with codahale-based metrics system, but from the guide on codahale website, metrics can be registered even after starting reporters, which makes possible to have dynamic metrics.
Please refer "Complete getting started" in https://metrics.dropwizard.io/3.1.0/getting-started/<https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmetrics.dropwizard.io%2F3.1.0%2Fgetting-started%2F&data=02%7C01%7Csimon.cooper%40featurespace.co.uk%7C08937e98d90444458f4308d595876c84%7C19e863aab068484d9f9f990b545c5a0f%7C0%7C0%7C636579331230445822&sdata=58rMUWDUStdFsVjOtgikj%2BX%2BrD5nisGCmZBbqVSaQbg%3D&reserved=0>
Hope this helps.
Thanks,
Jungtaek Lim (HeartSaVioR)
2018년 3월 28일 (수) 오후 9:38, Simon Cooper <si...@featurespace.co.uk>>님이 작성:
Hi,
We're looking at upgrading our topologies to use the new V2 metrics system, however one of our absolutely core requirements is the ability to dynamically create/assign metrics after the topology is started up - we don't know what metrics we'll be running beforehand, and the metrics can change during the lifetime of the topology. We handled this in the V1 system using IMetric wrappers, but the codehale-based metrics system doesn't seem to have the ability to do this.
If the V1 system is removed in a future version, this means we won't be able to use the storm metric system to report our metrics, which is a real shame, as we’ll lose all the built-in info that storm provides. We'll need to implement our own system from scratch & basically duplicate all the existing infrastructure.
Is there a way we can create metrics dynamically in the V2 system?
Thanks,
Simon Cooper
This message, and any files/attachments transmitted together with it, is intended for the use only of the person (or persons) to whom it is addressed. It may contain information which is confidential and/or protected by legal privilege. Accordingly, any dissemination, distribution, copying or use of this message, or any part of it or anything sent together with it, other than by intended recipients, may constitute a breach of civil or criminal law and is hereby prohibited. Unless otherwise stated, any views expressed in this message are those of the person sending it and not the sender's employer. No responsibility, legal or otherwise, of whatever nature, is accepted as to the accuracy of the contents of this message or for the completeness of the message as received. Anyone who is not the intended recipient of this message is advised to make no use of it and is requested to contact Featurespace Limited as soon as possible. Any recipient of this message who has knowledge or suspects that it may have been the subject of unauthorised interception or alteration is also requested to contact Featurespace Limited.
Re: Dynamic metrics in V2 system
Posted by Bobby Evans <bo...@apache.org>.
This should work for the most part. The new metrics API is just a thin
wrapper around the codahale metrics API that has no restrictions on when
you can add or remove metrics. It is separate from storm and it really is
just a naming convention that is used to store which worker, component,
etc, a metric is associated with.
We have not exposed any of the remove APIs so if this is something you
really need please file a JIRA for it and we can look into supporting it
too.
Thanks,
Bobby
On Thu, Mar 29, 2018 at 11:46 AM Simon Cooper <
simon.cooper@featurespace.co.uk> wrote:
> Thanks! Does the new metrics system only create one reporter for the
> topology, and send the info between workers, or have one reporter per
> worker?
>
>
>
> Simon
>
>
>
> *From:* Jungtaek Lim <ka...@gmail.com>
> *Sent:* 29 March 2018 16:12
> *To:* user@storm.apache.org
> *Subject:* Re: Dynamic metrics in V2 system
>
>
>
> Hi Simon,
>
>
>
> Actually I'm not familiar with codahale-based metrics system, but from the
> guide on codahale website, metrics can be registered even after starting
> reporters, which makes possible to have dynamic metrics.
>
>
>
> Please refer "Complete getting started" in
> https://metrics.dropwizard.io/3.1.0/getting-started/
> <https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmetrics.dropwizard.io%2F3.1.0%2Fgetting-started%2F&data=02%7C01%7Csimon.cooper%40featurespace.co.uk%7C08937e98d90444458f4308d595876c84%7C19e863aab068484d9f9f990b545c5a0f%7C0%7C0%7C636579331230445822&sdata=58rMUWDUStdFsVjOtgikj%2BX%2BrD5nisGCmZBbqVSaQbg%3D&reserved=0>
>
>
>
> Hope this helps.
>
>
>
> Thanks,
>
> Jungtaek Lim (HeartSaVioR)
>
>
>
> 2018년 3월 28일 (수) 오후 9:38, Simon Cooper <si...@featurespace.co.uk>님이
> 작성:
>
> Hi,
>
>
>
> We're looking at upgrading our topologies to use the new V2 metrics
> system, however one of our absolutely core requirements is the ability to
> dynamically create/assign metrics after the topology is started up - we
> don't know what metrics we'll be running beforehand, and the metrics can
> change during the lifetime of the topology. We handled this in the V1
> system using IMetric wrappers, but the codehale-based metrics system
> doesn't seem to have the ability to do this.
>
>
>
> If the V1 system is removed in a future version, this means we won't be
> able to use the storm metric system to report our metrics, which is a real
> shame, as we’ll lose all the built-in info that storm provides. We'll need
> to implement our own system from scratch & basically duplicate all the
> existing infrastructure.
>
>
>
> Is there a way we can create metrics dynamically in the V2 system?
>
>
>
> Thanks,
>
> Simon Cooper
>
> This message, and any files/attachments transmitted together with it, is
> intended for the use only of the person (or persons) to whom it is
> addressed. It may contain information which is confidential and/or
> protected by legal privilege. Accordingly, any dissemination, distribution,
> copying or use of this message, or any part of it or anything sent together
> with it, other than by intended recipients, may constitute a breach of
> civil or criminal law and is hereby prohibited. Unless otherwise stated,
> any views expressed in this message are those of the person sending it and
> not the sender's employer. No responsibility, legal or otherwise, of
> whatever nature, is accepted as to the accuracy of the contents of this
> message or for the completeness of the message as received. Anyone who is
> not the intended recipient of this message is advised to make no use of it
> and is requested to contact Featurespace Limited as soon as possible. Any
> recipient of this message who has knowledge or suspects that it may have
> been the subject of unauthorised interception or alteration is also
> requested to contact Featurespace Limited.
>
>
RE: Dynamic metrics in V2 system
Posted by Simon Cooper <si...@featurespace.co.uk>.
Thanks! Does the new metrics system only create one reporter for the topology, and send the info between workers, or have one reporter per worker?
Simon
From: Jungtaek Lim <ka...@gmail.com>
Sent: 29 March 2018 16:12
To: user@storm.apache.org
Subject: Re: Dynamic metrics in V2 system
Hi Simon,
Actually I'm not familiar with codahale-based metrics system, but from the guide on codahale website, metrics can be registered even after starting reporters, which makes possible to have dynamic metrics.
Please refer "Complete getting started" in https://metrics.dropwizard.io/3.1.0/getting-started/<https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmetrics.dropwizard.io%2F3.1.0%2Fgetting-started%2F&data=02%7C01%7Csimon.cooper%40featurespace.co.uk%7C08937e98d90444458f4308d595876c84%7C19e863aab068484d9f9f990b545c5a0f%7C0%7C0%7C636579331230445822&sdata=58rMUWDUStdFsVjOtgikj%2BX%2BrD5nisGCmZBbqVSaQbg%3D&reserved=0>
Hope this helps.
Thanks,
Jungtaek Lim (HeartSaVioR)
2018년 3월 28일 (수) 오후 9:38, Simon Cooper <si...@featurespace.co.uk>>님이 작성:
Hi,
We're looking at upgrading our topologies to use the new V2 metrics system, however one of our absolutely core requirements is the ability to dynamically create/assign metrics after the topology is started up - we don't know what metrics we'll be running beforehand, and the metrics can change during the lifetime of the topology. We handled this in the V1 system using IMetric wrappers, but the codehale-based metrics system doesn't seem to have the ability to do this.
If the V1 system is removed in a future version, this means we won't be able to use the storm metric system to report our metrics, which is a real shame, as we’ll lose all the built-in info that storm provides. We'll need to implement our own system from scratch & basically duplicate all the existing infrastructure.
Is there a way we can create metrics dynamically in the V2 system?
Thanks,
Simon Cooper
This message, and any files/attachments transmitted together with it, is intended for the use only of the person (or persons) to whom it is addressed. It may contain information which is confidential and/or protected by legal privilege. Accordingly, any dissemination, distribution, copying or use of this message, or any part of it or anything sent together with it, other than by intended recipients, may constitute a breach of civil or criminal law and is hereby prohibited. Unless otherwise stated, any views expressed in this message are those of the person sending it and not the sender's employer. No responsibility, legal or otherwise, of whatever nature, is accepted as to the accuracy of the contents of this message or for the completeness of the message as received. Anyone who is not the intended recipient of this message is advised to make no use of it and is requested to contact Featurespace Limited as soon as possible. Any recipient of this message who has knowledge or suspects that it may have been the subject of unauthorised interception or alteration is also requested to contact Featurespace Limited.
Re: Dynamic metrics in V2 system
Posted by Jungtaek Lim <ka...@gmail.com>.
Hi Simon,
Actually I'm not familiar with codahale-based metrics system, but from the
guide on codahale website, metrics can be registered even after starting
reporters, which makes possible to have dynamic metrics.
Please refer "Complete getting started" in
https://metrics.dropwizard.io/3.1.0/getting-started/
Hope this helps.
Thanks,
Jungtaek Lim (HeartSaVioR)
2018년 3월 28일 (수) 오후 9:38, Simon Cooper <si...@featurespace.co.uk>님이
작성:
> Hi,
>
>
>
> We're looking at upgrading our topologies to use the new V2 metrics
> system, however one of our absolutely core requirements is the ability to
> dynamically create/assign metrics after the topology is started up - we
> don't know what metrics we'll be running beforehand, and the metrics can
> change during the lifetime of the topology. We handled this in the V1
> system using IMetric wrappers, but the codehale-based metrics system
> doesn't seem to have the ability to do this.
>
>
>
> If the V1 system is removed in a future version, this means we won't be
> able to use the storm metric system to report our metrics, which is a real
> shame, as we’ll lose all the built-in info that storm provides. We'll need
> to implement our own system from scratch & basically duplicate all the
> existing infrastructure.
>
>
>
> Is there a way we can create metrics dynamically in the V2 system?
>
>
>
> Thanks,
>
> Simon Cooper
> This message, and any files/attachments transmitted together with it, is
> intended for the use only of the person (or persons) to whom it is
> addressed. It may contain information which is confidential and/or
> protected by legal privilege. Accordingly, any dissemination, distribution,
> copying or use of this message, or any part of it or anything sent together
> with it, other than by intended recipients, may constitute a breach of
> civil or criminal law and is hereby prohibited. Unless otherwise stated,
> any views expressed in this message are those of the person sending it and
> not the sender's employer. No responsibility, legal or otherwise, of
> whatever nature, is accepted as to the accuracy of the contents of this
> message or for the completeness of the message as received. Anyone who is
> not the intended recipient of this message is advised to make no use of it
> and is requested to contact Featurespace Limited as soon as possible. Any
> recipient of this message who has knowledge or suspects that it may have
> been the subject of unauthorised interception or alteration is also
> requested to contact Featurespace Limited.
>