You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@storm.apache.org by GitBox <gi...@apache.org> on 2020/06/22 14:38:59 UTC
[GitHub] [storm] RuiLi8080 opened a new pull request #3289: [STORM-3654] remove executor id from JCQueue receive-queue name
RuiLi8080 opened a new pull request #3289:
URL: https://github.com/apache/storm/pull/3289
## What is the purpose of the change
This name is mainly used in two places:
1. JCQueue metric name: Add unique executor id to metric name could be bad if we want to aggregate metrics across multiple executors/components. Also, executor id is already present in metrics dimension info such as `metricRegistry.gauge(queueName + "-capacity", cap, topologyId, componentId, taskId, port);`.
2. JCQueue name: queue name was only used in debug level log in back-pressure usecase. It would be useful to know which executor is experiencing back-pressure when tuning topologies.
In order to retain this info, we add a new field called `owner` of the JCQueue as we know it is only used in worker transfer queue or executor receive queue. Owner will just be the executor id or worker id.
## How was the change tested
Tested with WordCount topology with LoggingMetricsConsumer. Executor id is removed from metric name.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [storm] Ethanlm merged pull request #3289: [STORM-3654] remove executor id from JCQueue receive-queue name
Posted by GitBox <gi...@apache.org>.
Ethanlm merged pull request #3289:
URL: https://github.com/apache/storm/pull/3289
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [storm] Ethanlm commented on a change in pull request #3289: [STORM-3654] remove executor id from JCQueue receive-queue name
Posted by GitBox <gi...@apache.org>.
Ethanlm commented on a change in pull request #3289:
URL: https://github.com/apache/storm/pull/3289#discussion_r445723573
##########
File path: examples/storm-perf/src/main/java/org/apache/storm/perf/queuetest/JCQueuePerfTest.java
##########
@@ -72,7 +72,7 @@ private static void producerFwdConsumer(int prodBatchSz) {
private static void oneProducer1Consumer(int prodBatchSz) {
- JCQueue q1 = new JCQueue("q1", 50_000, 0, prodBatchSz, new WaitStrategyPark(100), "test", "test",
+ JCQueue q1 = new JCQueue("q1", null, 50_000, 0, prodBatchSz, new WaitStrategyPark(100), "test", "test",
Review comment:
why is this `null` (Although I guess it doesn't matter)
##########
File path: storm-client/test/jvm/org/apache/storm/utils/JCQueueBackpressureTest.java
##########
@@ -22,7 +22,7 @@
public class JCQueueBackpressureTest {
private static JCQueue createQueue(String name, int queueSize) {
- return new JCQueue(name, queueSize, 0, 1, new WaitStrategyPark(0), "test", "test", Collections.singletonList(1000), 1000, new StormMetricRegistry());
+ return new JCQueue(name, null, queueSize, 0, 1, new WaitStrategyPark(0), "test", "test", Collections.singletonList(1000), 1000, new StormMetricRegistry());
Review comment:
is it better to use `name` instead of `null`?
##########
File path: storm-client/src/jvm/org/apache/storm/utils/JCQueue.java
##########
@@ -44,15 +44,16 @@
private final IWaitStrategy backPressureWaitStrategy;
private final String queueName;
- public JCQueue(String queueName, int size, int overflowLimit, int producerBatchSz, IWaitStrategy backPressureWaitStrategy,
- String topologyId, String componentId, List<Integer> taskIds, int port, StormMetricRegistry metricRegistry) {
+ public JCQueue(String queueName, String metricNamePrefix, int size, int overflowLimit, int producerBatchSz,
+ IWaitStrategy backPressureWaitStrategy, String topologyId, String componentId, List<Integer> taskIds,
+ int port, StormMetricRegistry metricRegistry) {
this.queueName = queueName;
this.overflowLimit = overflowLimit;
this.recvQueue = new MpscArrayQueue<>(size);
this.overflowQ = new MpscUnboundedArrayQueue<>(size);
for (Integer taskId : taskIds) {
- this.jcqMetrics.add(new JCQueueMetrics(queueName, topologyId, componentId, taskId, port,
+ this.jcqMetrics.add(new JCQueueMetrics(metricNamePrefix, topologyId, componentId, taskId, port,
Review comment:
Since JCQueueMetrics takes `metricNamePrefix` now, I think it is better to change the variable name in JCQueueMetrics from `queueName` to `metricNamePrefix` to avoid confusion.
https://github.com/apache/storm/blob/master/storm-client/src/jvm/org/apache/storm/utils/JCQueueMetrics.java#L35
##########
File path: storm-client/test/jvm/org/apache/storm/utils/JCQueueTest.java
##########
@@ -157,7 +157,7 @@ private JCQueue createQueue(String name, int queueSize) {
}
private JCQueue createQueue(String name, int batchSize, int queueSize) {
- return new JCQueue(name, queueSize, 0, batchSize, waitStrategy, "test", "test", Collections.singletonList(1000), 1000, new StormMetricRegistry());
+ return new JCQueue(name, null, queueSize, 0, batchSize, waitStrategy, "test", "test", Collections.singletonList(1000), 1000, new StormMetricRegistry());
Review comment:
is it better to use name instead of null?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [storm] RuiLi8080 commented on pull request #3289: [STORM-3654] remove executor id from JCQueue receive-queue name
Posted by GitBox <gi...@apache.org>.
RuiLi8080 commented on pull request #3289:
URL: https://github.com/apache/storm/pull/3289#issuecomment-649003061
@agresch Thanks for the approval. Talked to @Ethanlm offline, he suggested to use `metricNamePrefix` instead of `owner` which seems to be more confusing. I agreed since essentially we just want to prevent including executor id in metric name. This way we can still keep the old way of queue name for logging.
Just pushed new changes for review.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org