You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@storm.apache.org by bo...@apache.org on 2017/09/07 19:14:53 UTC
[12/18] storm git commit: Addressed review comments
Addressed review comments
Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/381387b2
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/381387b2
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/381387b2
Branch: refs/heads/master
Commit: 381387b2c89f11cc6206219634af53df0769850e
Parents: 0ef492e
Author: Robert (Bobby) Evans <ev...@yahoo-inc.com>
Authored: Fri Aug 25 12:10:41 2017 -0500
Committer: Robert (Bobby) Evans <ev...@yahoo-inc.com>
Committed: Fri Aug 25 12:10:41 2017 -0500
----------------------------------------------------------------------
examples/storm-loadgen/README.md | 6 +++---
.../java/org/apache/storm/loadgen/CaptureLoad.java | 14 ++++++--------
.../org/apache/storm/loadgen/EstimateThroughput.java | 10 ++++------
.../org/apache/storm/loadgen/TopologyLoadConf.java | 4 ++--
4 files changed, 15 insertions(+), 19 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/storm/blob/381387b2/examples/storm-loadgen/README.md
----------------------------------------------------------------------
diff --git a/examples/storm-loadgen/README.md b/examples/storm-loadgen/README.md
index 29d27ab..52f5358 100644
--- a/examples/storm-loadgen/README.md
+++ b/examples/storm-loadgen/README.md
@@ -5,14 +5,14 @@ A set of tools to place an artificial load on a storm cluster to compare against
## Methodology
The idea behind all of these tools is to measure the trade-offs between latency, throughput, and cost when processing data using Apache Storm.
-When processing data you typically will know a few things. First you will know about how much data you are going to be processing. This will typically be a range of values that change throughput the day. You also will have an idea of how quickly you need the data processed by. Often this is measured in terms of the latency it takes to process data at the some percentile or set of percentiles. This is because of most use cases the value of the data declines over time, and being able to react to the data quickly is more valuable. You probably also have a budget for how much you are willing to spend to be able to process this data. There are always trade-offs in how quickly you can process some data and how efficiently you can processes that data both in terms of resource usage (cost) and latency. These tools are designed to help you explore that space.
+When processing data you typically will know a few things. First you will know about how much data you are going to be processing. This will typically be a range of values that change throughput the day. You also will have an idea of how quickly you need the data processed by. Often this is measured in terms of the latency it takes to process data at some percentile or set of percentiles. This is because in most use cases the value of the data declines over time, and being able to react to the data quickly is more valuable. You probably also have a budget for how much you are willing to spend to be able to process this data. There are always trade-offs in how quickly you can process some data and how efficiently you can processes that data both in terms of resource usage (cost) and latency. These tools are designed to help you explore that space.
A note on how latency is measured. Storm typically measures latency from when a message is emitted by a spout until the point it is fully acked or failed (in many versions of storm it actually does this in the acker instead of the spout so it is trying to be a measure of how long it takes for the actual processing, removing as much of the acker overhead as possible). For these tools we do it differently. We simulate a throughput and measure the start time of the tuple from when it would have been emitted if the topology could keep up with the load. In the normal case this should not be an issue, but if the topology cannot keep up with the throughput you will see the latency grow very high compared to the latency reported by storm.
## Tools
### CaptureLoad
-`CaptureLoad` will look at the topologies on a running cluster and store the structure of and metrics about each of theses topologies storing them in a format that can be used later to reproduce a similar load on the cluster.
+`CaptureLoad` will look at the topologies on a running cluster and store the structure of and metrics about each in a format described below that can be used later to reproduce a similar load on the cluster.
#### Usage
```
@@ -49,7 +49,7 @@ storm jar storm-loadgen.jar org.apache.storm.loadgen.GenLoad [options] [capture_
| -w,--report-window <INTERVAL_SECS> | How long of a rolling window should be in each report. Will be rounded up to the next report interval boundary. default 30|
## ThroughputVsLatency
-This is a topology similar to `GenLoad` in most ways, except instead of simulating a load it runs a word count algorithm.
+A word count topology with metrics reporting like the `GenLoad` command.
### Usage
```
http://git-wip-us.apache.org/repos/asf/storm/blob/381387b2/examples/storm-loadgen/src/main/java/org/apache/storm/loadgen/CaptureLoad.java
----------------------------------------------------------------------
diff --git a/examples/storm-loadgen/src/main/java/org/apache/storm/loadgen/CaptureLoad.java b/examples/storm-loadgen/src/main/java/org/apache/storm/loadgen/CaptureLoad.java
index 649a4c0..11d64a1 100644
--- a/examples/storm-loadgen/src/main/java/org/apache/storm/loadgen/CaptureLoad.java
+++ b/examples/storm-loadgen/src/main/java/org/apache/storm/loadgen/CaptureLoad.java
@@ -25,6 +25,7 @@ import java.util.List;
import java.util.Map;
import java.util.function.Function;
import java.util.stream.Collectors;
+import java.util.stream.Stream;
import org.apache.commons.cli.CommandLine;
import org.apache.commons.cli.CommandLineParser;
import org.apache.commons.cli.DefaultParser;
@@ -76,8 +77,7 @@ public class CaptureLoad {
List<Double> subvalues = data.values().stream()
.map((subMap) -> subMap.get(id))
.filter((value) -> value != null)
- .mapToDouble((value) -> value.doubleValue())
- .boxed().collect(Collectors.toList());
+ .collect(Collectors.toList());
ret.addAll(subvalues);
}
}
@@ -325,16 +325,14 @@ public class CaptureLoad {
.build());
CommandLineParser parser = new DefaultParser();
CommandLine cmd = null;
- ParseException pe = null;
+ boolean printHelp = false;
try {
cmd = parser.parse(options, args);
} catch (ParseException e) {
- pe = e;
+ System.err.println("ERROR " + e.getMessage());
+ printHelp = true;
}
- if (pe != null || cmd.hasOption('h')) {
- if (pe != null) {
- System.err.println("ERROR " + pe.getMessage());
- }
+ if (printHelp || cmd.hasOption('h')) {
new HelpFormatter().printHelp("CaptureLoad [options] [topologyName]*", options);
return;
}
http://git-wip-us.apache.org/repos/asf/storm/blob/381387b2/examples/storm-loadgen/src/main/java/org/apache/storm/loadgen/EstimateThroughput.java
----------------------------------------------------------------------
diff --git a/examples/storm-loadgen/src/main/java/org/apache/storm/loadgen/EstimateThroughput.java b/examples/storm-loadgen/src/main/java/org/apache/storm/loadgen/EstimateThroughput.java
index 80ede37..fc0b4c7 100644
--- a/examples/storm-loadgen/src/main/java/org/apache/storm/loadgen/EstimateThroughput.java
+++ b/examples/storm-loadgen/src/main/java/org/apache/storm/loadgen/EstimateThroughput.java
@@ -55,16 +55,14 @@ public class EstimateThroughput {
.build());
CommandLineParser parser = new DefaultParser();
CommandLine cmd = null;
- ParseException pe = null;
+ boolean printHelp = false;
try {
cmd = parser.parse(options, args);
} catch (ParseException e) {
- pe = e;
+ System.err.println("ERROR " + e.getMessage());
+ printHelp = true;
}
- if (pe != null || cmd.hasOption('h')) {
- if (pe != null) {
- System.err.println("ERROR " + pe.getMessage());
- }
+ if (printHelp || cmd.hasOption('h')) {
new HelpFormatter().printHelp("EstimateThroughput [options] [topologyName]*", options);
return;
}
http://git-wip-us.apache.org/repos/asf/storm/blob/381387b2/examples/storm-loadgen/src/main/java/org/apache/storm/loadgen/TopologyLoadConf.java
----------------------------------------------------------------------
diff --git a/examples/storm-loadgen/src/main/java/org/apache/storm/loadgen/TopologyLoadConf.java b/examples/storm-loadgen/src/main/java/org/apache/storm/loadgen/TopologyLoadConf.java
index 38458c6..b171972 100644
--- a/examples/storm-loadgen/src/main/java/org/apache/storm/loadgen/TopologyLoadConf.java
+++ b/examples/storm-loadgen/src/main/java/org/apache/storm/loadgen/TopologyLoadConf.java
@@ -427,7 +427,7 @@ public class TopologyLoadConf {
public double getAllEmittedAggregate() {
double ret = getSpoutEmittedAggregate();
for (LoadCompConf bolt: bolts) {
- ret += bolt.getAllEmittedAggregate() * bolt.parallelism;
+ ret += bolt.getAllEmittedAggregate();
}
return ret;
}
@@ -439,7 +439,7 @@ public class TopologyLoadConf {
public double getSpoutEmittedAggregate() {
double ret = 0;
for (LoadCompConf spout: spouts) {
- ret += spout.getAllEmittedAggregate() * spout.parallelism;
+ ret += spout.getAllEmittedAggregate();
}
return ret;
}