You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@storm.apache.org by ka...@apache.org on 2015/07/13 03:00:06 UTC
[1/2] storm git commit: [STORM-934] Add doc descriptions for the
TOPOLOGY_ACKER_EXECUTORS
Repository: storm
Updated Branches:
refs/heads/master 9906f85d5 -> 7bccc6d3d
[STORM-934] Add doc descriptions for the TOPOLOGY_ACKER_EXECUTORS
Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/c27948df
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/c27948df
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/c27948df
Branch: refs/heads/master
Commit: c27948df35fd6ccbbe6fd19c3c2528c810c24699
Parents: 9906f85
Author: zhuol <zh...@yahoo-inc.com>
Authored: Fri Jul 10 13:38:06 2015 -0500
Committer: Jungtaek Lim <ka...@gmail.com>
Committed: Mon Jul 13 09:51:18 2015 +0900
----------------------------------------------------------------------
docs/documentation/Guaranteeing-message-processing.md | 4 ++--
.../documentation/Running-topologies-on-a-production-cluster.md | 4 ++--
storm-core/src/jvm/backtype/storm/Config.java | 5 +++--
3 files changed, 7 insertions(+), 6 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/storm/blob/c27948df/docs/documentation/Guaranteeing-message-processing.md
----------------------------------------------------------------------
diff --git a/docs/documentation/Guaranteeing-message-processing.md b/docs/documentation/Guaranteeing-message-processing.md
index 4ecc620..9ade3fa 100644
--- a/docs/documentation/Guaranteeing-message-processing.md
+++ b/docs/documentation/Guaranteeing-message-processing.md
@@ -136,7 +136,7 @@ As always in software design, the answer is "it depends." Storm 0.7.0 introduced
### How does Storm implement reliability in an efficient way?
-A Storm topology has a set of special "acker" tasks that track the DAG of tuples for every spout tuple. When an acker sees that a DAG is complete, it sends a message to the spout task that created the spout tuple to ack the message. You can set the number of acker tasks for a topology in the topology configuration using [Config.TOPOLOGY_ACKERS](/javadoc/apidocs/backtype/storm/Config.html#TOPOLOGY_ACKERS). Storm defaults TOPOLOGY_ACKERS to one task -- you will need to increase this number for topologies processing large amounts of messages.
+A Storm topology has a set of special "acker" tasks that track the DAG of tuples for every spout tuple. When an acker sees that a DAG is complete, it sends a message to the spout task that created the spout tuple to ack the message. You can set the number of acker executors for a topology in the topology configuration using [Config.TOPOLOGY_ACKER_EXECUTORS](/javadoc/apidocs/backtype/storm/Config.html#TOPOLOGY_ACKER_EXECUTORS). Storm defaults TOPOLOGY_ACKER_EXECUTORS to be equal to the number of workers configured in the topology -- you will need to increase this number for topologies processing large amounts of messages.
The best way to understand Storm's reliability implementation is to look at the lifecycle of tuples and tuple DAGs. When a tuple is created in a topology, whether in a spout or a bolt, it is given a random 64 bit id. These ids are used by ackers to track the tuple DAG for every spout tuple.
@@ -178,4 +178,4 @@ There are three ways to remove reliability. The first is to set Config.TOPOLOGY_
The second way is to remove reliability on a message by message basis. You can turn off tracking for an individual spout tuple by omitting a message id in the `SpoutOutputCollector.emit` method.
-Finally, if you don't care if a particular subset of the tuples downstream in the topology fail to be processed, you can emit them as unanchored tuples. Since they're not anchored to any spout tuples, they won't cause any spout tuples to fail if they aren't acked.
\ No newline at end of file
+Finally, if you don't care if a particular subset of the tuples downstream in the topology fail to be processed, you can emit them as unanchored tuples. Since they're not anchored to any spout tuples, they won't cause any spout tuples to fail if they aren't acked.
http://git-wip-us.apache.org/repos/asf/storm/blob/c27948df/docs/documentation/Running-topologies-on-a-production-cluster.md
----------------------------------------------------------------------
diff --git a/docs/documentation/Running-topologies-on-a-production-cluster.md b/docs/documentation/Running-topologies-on-a-production-cluster.md
index ebba452..b8cb892 100644
--- a/docs/documentation/Running-topologies-on-a-production-cluster.md
+++ b/docs/documentation/Running-topologies-on-a-production-cluster.md
@@ -50,7 +50,7 @@ You can find out how to configure your `storm` client to talk to a Storm cluster
There are a variety of configurations you can set per topology. A list of all the configurations you can set can be found [here](/javadoc/apidocs/backtype/storm/Config.html). The ones prefixed with "TOPOLOGY" can be overridden on a topology-specific basis (the other ones are cluster configurations and cannot be overridden). Here are some common ones that are set for a topology:
1. **Config.TOPOLOGY_WORKERS**: This sets the number of worker processes to use to execute the topology. For example, if you set this to 25, there will be 25 Java processes across the cluster executing all the tasks. If you had a combined 150 parallelism across all components in the topology, each worker process will have 6 tasks running within it as threads.
-2. **Config.TOPOLOGY_ACKERS**: This sets the number of tasks that will track tuple trees and detect when a spout tuple has been fully processed. Ackers are an integral part of Storm's reliability model and you can read more about them on [Guaranteeing message processing](Guaranteeing-message-processing.html).
+2. **Config.TOPOLOGY_ACKER_EXECUTORS**: This sets the number of executors that will track tuple trees and detect when a spout tuple has been fully processed. Ackers are an integral part of Storm's reliability model and you can read more about them on [Guaranteeing message processing](Guaranteeing-message-processing.html). By not setting this variable or setting it as null, Storm will set the number of acker executors to be equal to the number of workers configured for this topology. If this variable is set to 0, then Storm will immediately ack tuples as soon as they come off the spout, effectively disabling reliability.
3. **Config.TOPOLOGY_MAX_SPOUT_PENDING**: This sets the maximum number of spout tuples that can be pending on a single spout task at once (pending means the tuple has not been acked or failed yet). It is highly recommended you set this config to prevent queue explosion.
4. **Config.TOPOLOGY_MESSAGE_TIMEOUT_SECS**: This is the maximum amount of time a spout tuple has to be fully completed before it is considered failed. This value defaults to 30 seconds, which is sufficient for most topologies. See [Guaranteeing message processing](Guaranteeing-message-processing.html) for more information on how Storm's reliability model works.
5. **Config.TOPOLOGY_SERIALIZATIONS**: You can register more serializers to Storm using this config so that you can use custom types within tuples.
@@ -74,4 +74,4 @@ To update a running topology, the only option currently is to kill the current t
The best place to monitor a topology is using the Storm UI. The Storm UI provides information about errors happening in tasks and fine-grained stats on the throughput and latency performance of each component of each running topology.
-You can also look at the worker logs on the cluster machines.
\ No newline at end of file
+You can also look at the worker logs on the cluster machines.
http://git-wip-us.apache.org/repos/asf/storm/blob/c27948df/storm-core/src/jvm/backtype/storm/Config.java
----------------------------------------------------------------------
diff --git a/storm-core/src/jvm/backtype/storm/Config.java b/storm-core/src/jvm/backtype/storm/Config.java
index 063d446..f0ae8e7 100644
--- a/storm-core/src/jvm/backtype/storm/Config.java
+++ b/storm-core/src/jvm/backtype/storm/Config.java
@@ -1080,8 +1080,9 @@ public class Config extends HashMap<String, Object> {
/**
* How many executors to spawn for ackers.
*
- * <p>If this is set to 0, then Storm will immediately ack tuples as soon
- * as they come off the spout, effectively disabling reliability.</p>
+ * <p>By not setting this variable or setting it as null, Storm will set the number of acker executors
+ * to be equal to the number of workers configured for this topology. If this variable is set to 0,
+ * then Storm will immediately ack tuples as soon as they come off the spout, effectively disabling reliability.</p>
*/
public static final String TOPOLOGY_ACKER_EXECUTORS = "topology.acker.executors";
public static final Object TOPOLOGY_ACKER_EXECUTORS_SCHEMA = ConfigValidation.IntegerValidator;
[2/2] storm git commit: add STORM-934 to CHANGELOG.md
Posted by ka...@apache.org.
add STORM-934 to CHANGELOG.md
Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/7bccc6d3
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/7bccc6d3
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/7bccc6d3
Branch: refs/heads/master
Commit: 7bccc6d3d6a123f9192a586293fc5f15c940a9e1
Parents: c27948d
Author: Jungtaek Lim <ka...@gmail.com>
Authored: Mon Jul 13 09:59:42 2015 +0900
Committer: Jungtaek Lim <ka...@gmail.com>
Committed: Mon Jul 13 09:59:42 2015 +0900
----------------------------------------------------------------------
CHANGELOG.md | 1 +
1 file changed, 1 insertion(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/storm/blob/7bccc6d3/CHANGELOG.md
----------------------------------------------------------------------
diff --git a/CHANGELOG.md b/CHANGELOG.md
index f11c009..872b66f 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -8,6 +8,7 @@
* STORM-477: Add warning for invalid JAVA_HOME
* STORM-826: Update KafkaBolt to use the new kafka producer API
* STORM-912: Support SSL on Logviewer
+ * STORM-934: The current doc for topology ackers is outdated
## 0.10.0-beta2
* STORM-843: [storm-redis] Add Javadoc to storm-redis