You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@storm.apache.org by GitBox <gi...@apache.org> on 2020/04/21 01:29:15 UTC
[GitHub] [storm] kishorvpatil opened a new pull request #3255: [STORM-3627] Add flag to use metrics shortname for topology
kishorvpatil opened a new pull request #3255:
URL: https://github.com/apache/storm/pull/3255
Adding topology configuration _topology.metrics.use.shortname_ to allow for use of shortnames in case metrics tick is enabled using _topology.enable.v2.metrics.tick_.
This allows for use of shortname instead of complete naming convention that includes, topology-id, worker, component details in the metrics name.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [storm] kishorvpatil commented on issue #3255: [STORM-3627] Add flag to use metrics shortname for topology
Posted by GitBox <gi...@apache.org>.
kishorvpatil commented on issue #3255:
URL: https://github.com/apache/storm/pull/3255#issuecomment-618576609
Closing this as it can cause duplicate reporting.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [storm] Ethanlm commented on a change in pull request #3255: [STORM-3627] Add flag to use metrics shortname for topology
Posted by GitBox <gi...@apache.org>.
Ethanlm commented on a change in pull request #3255:
URL: https://github.com/apache/storm/pull/3255#discussion_r412218182
##########
File path: storm-client/src/jvm/org/apache/storm/metrics2/StormMetricRegistry.java
##########
@@ -52,16 +64,19 @@ public JcMetrics jcMetrics(String name, String topologyId, String componentId, I
public Meter meter(String name, WorkerTopologyContext context, String componentId, Integer taskId, String streamId) {
String metricName = metricName(name, context.getStormId(), componentId, streamId, taskId, context.getThisWorkerPort());
+ LOG.warn("Adding meter: " + metricName);
return registry.meter(metricName);
}
public Counter counter(String name, WorkerTopologyContext context, String componentId, Integer taskId, String streamId) {
String metricName = metricName(name, context.getStormId(), componentId, streamId, taskId, context.getThisWorkerPort());
+ LOG.warn("Adding counter: " + metricName);
return registry.counter(metricName);
Review comment:
Since the registry is shared among tasks inside the same worker, all the metrics should have different name otherwise the same metric object will be returned: https://github.com/dropwizard/metrics/blob/4.1-development/metrics-core/src/main/java/com/codahale/metrics/MetricRegistry.java#L172-L181
So if `registery.counter("a-counter", xxxx)` is called more than once, starting from the second time, it returns the same metric object.
Which means all tasks will have the same metric objects in the worker. For example, all tasks have `acked`, `failed`, `emitted`, `transferred` metrics
https://github.com/apache/storm/blob/master/storm-client/src/jvm/org/apache/storm/metrics2/TaskMetrics.java#L21-L24
With this code change, it will return the same value for all the tasks inside the same worker.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [storm] agresch commented on a change in pull request #3255: [STORM-3627] Add flag to use metrics shortname for topology
Posted by GitBox <gi...@apache.org>.
agresch commented on a change in pull request #3255:
URL: https://github.com/apache/storm/pull/3255#discussion_r412219276
##########
File path: storm-client/src/jvm/org/apache/storm/metrics2/StormMetricRegistry.java
##########
@@ -36,10 +37,21 @@
private final MetricRegistry registry = new MetricRegistry();
private final List<StormReporter> reporters = new ArrayList<>();
private String hostName = null;
+ private boolean useShortName = false;
+
+ public StormMetricRegistry(boolean useShortName) {
Review comment:
I would pass the topology conf instead in case we want to add other future options to configure this class.
##########
File path: storm-client/src/jvm/org/apache/storm/metrics2/StormMetricRegistry.java
##########
@@ -36,10 +37,21 @@
private final MetricRegistry registry = new MetricRegistry();
private final List<StormReporter> reporters = new ArrayList<>();
private String hostName = null;
+ private boolean useShortName = false;
+
+ public StormMetricRegistry(boolean useShortName) {
+ this.useShortName = useShortName;
+ }
+
+ public StormMetricRegistry() {
+ this.useShortName = false;
+ LOG.warn("Constructing StormMetricsRegistry: " + ClusterUtils.stringifyError(new Exception("Stack trace")));
+ }
public <T> SimpleGauge<T> gauge(
T initialValue, String name, String topologyId, String componentId, Integer taskId, Integer port) {
String metricName = metricName(name, topologyId, componentId, taskId, port);
+ LOG.warn("Adding gauge: " + metricName);
Review comment:
Why warn here? Debug code?
##########
File path: storm-client/src/jvm/org/apache/storm/metrics2/StormMetricRegistry.java
##########
@@ -36,10 +37,21 @@
private final MetricRegistry registry = new MetricRegistry();
private final List<StormReporter> reporters = new ArrayList<>();
private String hostName = null;
+ private boolean useShortName = false;
+
+ public StormMetricRegistry(boolean useShortName) {
+ this.useShortName = useShortName;
+ }
+
+ public StormMetricRegistry() {
+ this.useShortName = false;
+ LOG.warn("Constructing StormMetricsRegistry: " + ClusterUtils.stringifyError(new Exception("Stack trace")));
Review comment:
Why is this a warning?
##########
File path: storm-client/src/jvm/org/apache/storm/metrics2/StormMetricRegistry.java
##########
@@ -111,6 +126,9 @@ public void stop() {
}
private String metricName(String name, String stormId, String componentId, String streamId, Integer taskId, Integer workerPort) {
+ if (this.useShortName) {
Review comment:
useShortWorkerName instead possibly?
##########
File path: storm-client/src/jvm/org/apache/storm/metrics2/StormMetricRegistry.java
##########
@@ -111,6 +126,9 @@ public void stop() {
}
private String metricName(String name, String stormId, String componentId, String streamId, Integer taskId, Integer workerPort) {
+ if (this.useShortName) {
+ return "storm.worker." + name;
Review comment:
Can we replace all of these with a constant?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [storm] Ethanlm commented on a change in pull request #3255: [STORM-3627] Add flag to use metrics shortname for topology
Posted by GitBox <gi...@apache.org>.
Ethanlm commented on a change in pull request #3255:
URL: https://github.com/apache/storm/pull/3255#discussion_r412218182
##########
File path: storm-client/src/jvm/org/apache/storm/metrics2/StormMetricRegistry.java
##########
@@ -52,16 +64,19 @@ public JcMetrics jcMetrics(String name, String topologyId, String componentId, I
public Meter meter(String name, WorkerTopologyContext context, String componentId, Integer taskId, String streamId) {
String metricName = metricName(name, context.getStormId(), componentId, streamId, taskId, context.getThisWorkerPort());
+ LOG.warn("Adding meter: " + metricName);
return registry.meter(metricName);
}
public Counter counter(String name, WorkerTopologyContext context, String componentId, Integer taskId, String streamId) {
String metricName = metricName(name, context.getStormId(), componentId, streamId, taskId, context.getThisWorkerPort());
+ LOG.warn("Adding counter: " + metricName);
return registry.counter(metricName);
Review comment:
Since the registry is shared among tasks inside the same worker, all the metrics should have different name otherwise the same metric object will be returned: https://github.com/dropwizard/metrics/blob/4.1-development/metrics-core/src/main/java/com/codahale/metrics/MetricRegistry.java#L172-L181
So if `registery.counter("a-counter", xxxx)` is called more than once, starting from the second time, it returns the same metric object.
Which means all tasks will have the same metric objects in the worker. For example, all tasks have `acked`, `failed`, `emitted`, `transferred` metrics
https://github.com/apache/storm/blob/master/storm-client/src/jvm/org/apache/storm/metrics2/TaskMetrics.java#L21-L24
With this code change, all the tasks will have the same metric values inside the same worker.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [storm] Ethanlm commented on a change in pull request #3255: [STORM-3627] Add flag to use metrics shortname for topology
Posted by GitBox <gi...@apache.org>.
Ethanlm commented on a change in pull request #3255:
URL: https://github.com/apache/storm/pull/3255#discussion_r412218182
##########
File path: storm-client/src/jvm/org/apache/storm/metrics2/StormMetricRegistry.java
##########
@@ -52,16 +64,19 @@ public JcMetrics jcMetrics(String name, String topologyId, String componentId, I
public Meter meter(String name, WorkerTopologyContext context, String componentId, Integer taskId, String streamId) {
String metricName = metricName(name, context.getStormId(), componentId, streamId, taskId, context.getThisWorkerPort());
+ LOG.warn("Adding meter: " + metricName);
return registry.meter(metricName);
}
public Counter counter(String name, WorkerTopologyContext context, String componentId, Integer taskId, String streamId) {
String metricName = metricName(name, context.getStormId(), componentId, streamId, taskId, context.getThisWorkerPort());
+ LOG.warn("Adding counter: " + metricName);
return registry.counter(metricName);
Review comment:
Since the registry is shared among tasks inside the same worker, all the metrics should have different name otherwise the same metric object will be returned: https://github.com/dropwizard/metrics/blob/4.1-development/metrics-core/src/main/java/com/codahale/metrics/MetricRegistry.java#L172-L181
So if `registery.counter("a-counter", xxxx)` is called more than once, starting from the second time, it returns the same metric object.
Which means all tasks will have the same metric objects in the worker. For example, all tasks have `acked`, `failed`, `emitted`, `transferred` metrics
https://github.com/apache/storm/blob/master/storm-client/src/jvm/org/apache/storm/metrics2/TaskMetrics.java#L21-L24
With this code change, all the tasks inside the same worker. will have the same metric values.
I think the right way to solve this `shortName`/`longName` problem is
inside https://github.com/apache/storm/pull/3251, when we cache the taskIdToNameToMetric, we cache the shortName.
But for dropwhizard registry perspective, we need to use the long name to avoid name collision.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org