You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@storm.apache.org by GitBox <gi...@apache.org> on 2020/08/04 19:38:27 UTC
[GitHub] [storm] RuiLi8080 opened a new pull request #3320: [STORM-3684] use real compId instead of constant _system for receive-queue V2 metrics
RuiLi8080 opened a new pull request #3320:
URL: https://github.com/apache/storm/pull/3320
## What is the purpose of the change
When registering receive-queue V2 metrics for each executor, we should use the its correspondent component Id instead of constant system component Id.
## How was the change tested
Launch a local cluster with a WordCount topology. Enable `org.apache.storm.metrics2.reporters.ConsoleStormReporter` to output metrics to worker log. We will see the long name as :
`storm.worker.<topoName>.<hostname>.<compId>.<taskId>.<port>.<metric-name>`
The compId part now reflects the correct component for the taskId.
```
2020-08-04 14:22:44.911 c.c.m.ConsoleReporter metrics-console-reporter-1-thread-1 [INFO] storm.worker.wc1-1-1596568923.192_168_0_19.count.10.6700-receive-queue-pct_full
2020-08-04 14:22:44.911 c.c.m.ConsoleReporter metrics-console-reporter-1-thread-1 [INFO] value = 0.0
2020-08-04 14:22:44.912 c.c.m.ConsoleReporter metrics-console-reporter-1-thread-1 [INFO] storm.worker.wc1-1-1596568923.192_168_0_19.count.10.6700-receive-queue-population
2020-08-04 14:22:44.912 c.c.m.ConsoleReporter metrics-console-reporter-1-thread-1 [INFO] value = 0
2020-08-04 14:22:44.912 c.c.m.ConsoleReporter metrics-console-reporter-1-thread-1 [INFO] storm.worker.wc1-1-1596568923.192_168_0_19.count.10.6700-receive-queue-sojourn_time_ms
2020-08-04 14:22:44.912 c.c.m.ConsoleReporter metrics-console-reporter-1-thread-1 [INFO] value = 0.0
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [storm] Ethanlm commented on a change in pull request #3320: [STORM-3684] use real compId instead of constant _system for receive-queue V2 metrics
Posted by GitBox <gi...@apache.org>.
Ethanlm commented on a change in pull request #3320:
URL: https://github.com/apache/storm/pull/3320#discussion_r465756938
##########
File path: storm-client/src/jvm/org/apache/storm/daemon/worker/WorkerState.java
##########
@@ -717,11 +719,19 @@ private Assignment getLocalAssignment(IStormClusterState stormClusterState, Stri
IWaitStrategy backPressureWaitStrategy = IWaitStrategy.createBackPressureWaitStrategy(topologyConf);
Map<List<Long>, JCQueue> receiveQueueMap = new HashMap<>();
+
for (List<Long> executor : executors) {
List<Integer> taskIds = StormCommon.executorIdToTasks(executor);
+ int taskId = taskIds.get(0);
+ String compId;
+ if (taskId == -1) {
Review comment:
I would use `SYSTEM_TASK_ID` instead of `-1`
##########
File path: storm-client/src/jvm/org/apache/storm/daemon/worker/WorkerState.java
##########
@@ -186,6 +186,9 @@ public WorkerState(Map<String, Object> conf,
this.isWorkerActive = new CountDownLatch(1);
this.isTopologyActive = new AtomicBoolean(false);
this.stormComponentToDebug = new AtomicReference<>();
+ this.topology = ConfigUtils.readSupervisorTopology(conf, topologyId, AdvancedFSOps.make(conf));
+ this.taskToComponent = StormCommon.stormTaskInfo(topology, topologyConf);
+ // mkReceiveQueueMap relies on taskToComponent which relies on topology above
Review comment:
I would change `mkReceiveQueueMap` method to include `taskToComponent` parameter to make this relationship more obvious
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [storm] Ethanlm commented on a change in pull request #3320: [STORM-3684] use real compId instead of constant _system for receive-queue V2 metrics
Posted by GitBox <gi...@apache.org>.
Ethanlm commented on a change in pull request #3320:
URL: https://github.com/apache/storm/pull/3320#discussion_r465301414
##########
File path: storm-client/src/jvm/org/apache/storm/daemon/worker/WorkerState.java
##########
@@ -199,7 +200,6 @@ public WorkerState(Map<String, Object> conf,
}
Collections.sort(localTaskIds);
this.topologyConf = topologyConf;
- this.topology = ConfigUtils.readSupervisorTopology(conf, topologyId, AdvancedFSOps.make(conf));
Review comment:
anything specific to this change?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [storm] Ethanlm commented on a change in pull request #3320: [STORM-3684] use real compId instead of constant _system for receive-queue V2 metrics
Posted by GitBox <gi...@apache.org>.
Ethanlm commented on a change in pull request #3320:
URL: https://github.com/apache/storm/pull/3320#discussion_r466033224
##########
File path: storm-client/src/jvm/org/apache/storm/daemon/worker/WorkerState.java
##########
@@ -704,7 +704,8 @@ private Assignment getLocalAssignment(IStormClusterState stormClusterState, Stri
}
}
- private Map<List<Long>, JCQueue> mkReceiveQueueMap(Map<String, Object> topologyConf, Set<List<Long>> executors) {
+ private Map<List<Long>, JCQueue> mkReceiveQueueMap(Map<String, Object> topologyConf,
+ Set<List<Long>> executors, Map<Integer, String> taskToComponent) {
Review comment:
nit: space
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [storm] RuiLi8080 commented on a change in pull request #3320: [STORM-3684] use real compId instead of constant _system for receive-queue V2 metrics
Posted by GitBox <gi...@apache.org>.
RuiLi8080 commented on a change in pull request #3320:
URL: https://github.com/apache/storm/pull/3320#discussion_r465490938
##########
File path: storm-client/src/jvm/org/apache/storm/daemon/worker/WorkerState.java
##########
@@ -199,7 +200,6 @@ public WorkerState(Map<String, Object> conf,
}
Collections.sort(localTaskIds);
this.topologyConf = topologyConf;
- this.topology = ConfigUtils.readSupervisorTopology(conf, topologyId, AdvancedFSOps.make(conf));
Review comment:
I actually updated. Basically mkReceiveQueueMap() function needs to know `this.taskToComponent` which relies on `this.topology`. So we need to move them up.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [storm] Ethanlm commented on a change in pull request #3320: [STORM-3684] use real compId instead of constant _system for receive-queue V2 metrics
Posted by GitBox <gi...@apache.org>.
Ethanlm commented on a change in pull request #3320:
URL: https://github.com/apache/storm/pull/3320#discussion_r465301242
##########
File path: storm-client/src/jvm/org/apache/storm/daemon/worker/WorkerState.java
##########
@@ -717,11 +718,20 @@ private Assignment getLocalAssignment(IStormClusterState stormClusterState, Stri
IWaitStrategy backPressureWaitStrategy = IWaitStrategy.createBackPressureWaitStrategy(topologyConf);
Map<List<Long>, JCQueue> receiveQueueMap = new HashMap<>();
+
+ Map<Integer, String> taskIdToCompId = StormCommon.stormTaskInfo(topology, topologyConf);
Review comment:
Is it possible to reuse `this.taskToComponent`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [storm] RuiLi8080 commented on a change in pull request #3320: [STORM-3684] use real compId instead of constant _system for receive-queue V2 metrics
Posted by GitBox <gi...@apache.org>.
RuiLi8080 commented on a change in pull request #3320:
URL: https://github.com/apache/storm/pull/3320#discussion_r465816418
##########
File path: storm-client/src/jvm/org/apache/storm/daemon/worker/WorkerState.java
##########
@@ -717,11 +719,19 @@ private Assignment getLocalAssignment(IStormClusterState stormClusterState, Stri
IWaitStrategy backPressureWaitStrategy = IWaitStrategy.createBackPressureWaitStrategy(topologyConf);
Map<List<Long>, JCQueue> receiveQueueMap = new HashMap<>();
+
for (List<Long> executor : executors) {
List<Integer> taskIds = StormCommon.executorIdToTasks(executor);
+ int taskId = taskIds.get(0);
+ String compId;
+ if (taskId == -1) {
Review comment:
Yes, updated
##########
File path: storm-client/src/jvm/org/apache/storm/daemon/worker/WorkerState.java
##########
@@ -186,6 +186,9 @@ public WorkerState(Map<String, Object> conf,
this.isWorkerActive = new CountDownLatch(1);
this.isTopologyActive = new AtomicBoolean(false);
this.stormComponentToDebug = new AtomicReference<>();
+ this.topology = ConfigUtils.readSupervisorTopology(conf, topologyId, AdvancedFSOps.make(conf));
+ this.taskToComponent = StormCommon.stormTaskInfo(topology, topologyConf);
+ // mkReceiveQueueMap relies on taskToComponent which relies on topology above
Review comment:
SUre
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [storm] RuiLi8080 commented on a change in pull request #3320: [STORM-3684] use real compId instead of constant _system for receive-queue V2 metrics
Posted by GitBox <gi...@apache.org>.
RuiLi8080 commented on a change in pull request #3320:
URL: https://github.com/apache/storm/pull/3320#discussion_r465864721
##########
File path: storm-client/src/jvm/org/apache/storm/daemon/worker/WorkerState.java
##########
@@ -717,11 +718,19 @@ private Assignment getLocalAssignment(IStormClusterState stormClusterState, Stri
IWaitStrategy backPressureWaitStrategy = IWaitStrategy.createBackPressureWaitStrategy(topologyConf);
Map<List<Long>, JCQueue> receiveQueueMap = new HashMap<>();
+
for (List<Long> executor : executors) {
List<Integer> taskIds = StormCommon.executorIdToTasks(executor);
+ int taskId = taskIds.get(0);
+ String compId;
+ if (taskId == Constants.SYSTEM_TASK_ID) {
+ compId = Constants.SYSTEM_COMPONENT_ID;
Review comment:
StormCommon.stormTaskInfoImpl() address taskID starting from 1.
WorkerState.readWorkerExecutors() additionally add Constants.SYSTEM_EXECUTOR_ID (-1) before reading from assignment. This system executor is not present in `taskToComponent` map.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [storm] RuiLi8080 commented on a change in pull request #3320: [STORM-3684] use real compId instead of constant _system for receive-queue V2 metrics
Posted by GitBox <gi...@apache.org>.
RuiLi8080 commented on a change in pull request #3320:
URL: https://github.com/apache/storm/pull/3320#discussion_r466495103
##########
File path: storm-client/src/jvm/org/apache/storm/daemon/worker/WorkerState.java
##########
@@ -704,7 +704,8 @@ private Assignment getLocalAssignment(IStormClusterState stormClusterState, Stri
}
}
- private Map<List<Long>, JCQueue> mkReceiveQueueMap(Map<String, Object> topologyConf, Set<List<Long>> executors) {
+ private Map<List<Long>, JCQueue> mkReceiveQueueMap(Map<String, Object> topologyConf,
+ Set<List<Long>> executors, Map<Integer, String> taskToComponent) {
Review comment:
updated
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [storm] Ethanlm commented on pull request #3320: [STORM-3684] use real compId instead of constant _system for receive-queue V2 metrics
Posted by GitBox <gi...@apache.org>.
Ethanlm commented on pull request #3320:
URL: https://github.com/apache/storm/pull/3320#issuecomment-670638763
travis failure seems unrelated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [storm] Ethanlm commented on a change in pull request #3320: [STORM-3684] use real compId instead of constant _system for receive-queue V2 metrics
Posted by GitBox <gi...@apache.org>.
Ethanlm commented on a change in pull request #3320:
URL: https://github.com/apache/storm/pull/3320#discussion_r465754981
##########
File path: storm-client/src/jvm/org/apache/storm/daemon/worker/WorkerState.java
##########
@@ -186,6 +186,9 @@ public WorkerState(Map<String, Object> conf,
this.isWorkerActive = new CountDownLatch(1);
this.isTopologyActive = new AtomicBoolean(false);
this.stormComponentToDebug = new AtomicReference<>();
+ this.topology = ConfigUtils.readSupervisorTopology(conf, topologyId, AdvancedFSOps.make(conf));
+ this.taskToComponent = StormCommon.stormTaskInfo(topology, topologyConf);
+ // mkReceiveQueueMap relies on taskToComponent which relies on topology above
Review comment:
I would change `mkReceiveQueueMap` method to include `taskToComponent` parameter to make this relationship more obvious (and then we can remove the comment)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [storm] Ethanlm merged pull request #3320: [STORM-3684] use real compId instead of constant _system for receive-queue V2 metrics
Posted by GitBox <gi...@apache.org>.
Ethanlm merged pull request #3320:
URL: https://github.com/apache/storm/pull/3320
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [storm] agresch commented on a change in pull request #3320: [STORM-3684] use real compId instead of constant _system for receive-queue V2 metrics
Posted by GitBox <gi...@apache.org>.
agresch commented on a change in pull request #3320:
URL: https://github.com/apache/storm/pull/3320#discussion_r465851962
##########
File path: storm-client/src/jvm/org/apache/storm/daemon/worker/WorkerState.java
##########
@@ -717,11 +718,19 @@ private Assignment getLocalAssignment(IStormClusterState stormClusterState, Stri
IWaitStrategy backPressureWaitStrategy = IWaitStrategy.createBackPressureWaitStrategy(topologyConf);
Map<List<Long>, JCQueue> receiveQueueMap = new HashMap<>();
+
for (List<Long> executor : executors) {
List<Integer> taskIds = StormCommon.executorIdToTasks(executor);
+ int taskId = taskIds.get(0);
+ String compId;
+ if (taskId == Constants.SYSTEM_TASK_ID) {
+ compId = Constants.SYSTEM_COMPONENT_ID;
Review comment:
Should StormCommon.stormTaskInfo() be addressing this mapping?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [storm] RuiLi8080 commented on a change in pull request #3320: [STORM-3684] use real compId instead of constant _system for receive-queue V2 metrics
Posted by GitBox <gi...@apache.org>.
RuiLi8080 commented on a change in pull request #3320:
URL: https://github.com/apache/storm/pull/3320#discussion_r465490556
##########
File path: storm-client/src/jvm/org/apache/storm/daemon/worker/WorkerState.java
##########
@@ -717,11 +718,20 @@ private Assignment getLocalAssignment(IStormClusterState stormClusterState, Stri
IWaitStrategy backPressureWaitStrategy = IWaitStrategy.createBackPressureWaitStrategy(topologyConf);
Map<List<Long>, JCQueue> receiveQueueMap = new HashMap<>();
+
+ Map<Integer, String> taskIdToCompId = StormCommon.stormTaskInfo(topology, topologyConf);
Review comment:
Yes, updated. Thanks for pointing out.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org