You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@storm.apache.org by bo...@apache.org on 2016/03/17 03:10:06 UTC
svn commit: r1735360 [1/6] - in /storm/branches/bobby-versioned-site:
_includes/ releases/0.10.0/ releases/0.10.0/images/
Author: bobby
Date: Thu Mar 17 02:10:04 2016
New Revision: 1735360
URL: http://svn.apache.org/viewvc?rev=1735360&view=rev
Log:
More cleanup
Added:
storm/branches/bobby-versioned-site/releases/0.10.0/images/
storm/branches/bobby-versioned-site/releases/0.10.0/images/architecture.png (with props)
storm/branches/bobby-versioned-site/releases/0.10.0/images/architecture.svg
storm/branches/bobby-versioned-site/releases/0.10.0/images/bolt.png (with props)
storm/branches/bobby-versioned-site/releases/0.10.0/images/bolt.svg
storm/branches/bobby-versioned-site/releases/0.10.0/images/bullet.gif (with props)
storm/branches/bobby-versioned-site/releases/0.10.0/images/download.png (with props)
storm/branches/bobby-versioned-site/releases/0.10.0/images/incubator-logo.png (with props)
storm/branches/bobby-versioned-site/releases/0.10.0/images/mailinglist.png (with props)
storm/branches/bobby-versioned-site/releases/0.10.0/images/security.png (with props)
storm/branches/bobby-versioned-site/releases/0.10.0/images/security.svg
storm/branches/bobby-versioned-site/releases/0.10.0/images/spout.png (with props)
storm/branches/bobby-versioned-site/releases/0.10.0/images/spout.svg
storm/branches/bobby-versioned-site/releases/0.10.0/images/storm.svg
storm/branches/bobby-versioned-site/releases/0.10.0/images/storm_header.png (with props)
storm/branches/bobby-versioned-site/releases/0.10.0/images/storm_logo_tagline_color copy.png (with props)
storm/branches/bobby-versioned-site/releases/0.10.0/images/storm_logo_tagline_color.png (with props)
storm/branches/bobby-versioned-site/releases/0.10.0/images/top_bg.gif (with props)
storm/branches/bobby-versioned-site/releases/0.10.0/images/topology.png (with props)
storm/branches/bobby-versioned-site/releases/0.10.0/images/topology.svg
storm/branches/bobby-versioned-site/releases/0.10.0/images/topology_dark.png (with props)
storm/branches/bobby-versioned-site/releases/0.10.0/images/topology_dark.svg
storm/branches/bobby-versioned-site/releases/0.10.0/images/ui_topology_viz.png (with props)
Removed:
storm/branches/bobby-versioned-site/releases/0.10.0/Documentation.md
Modified:
storm/branches/bobby-versioned-site/_includes/footer.html
storm/branches/bobby-versioned-site/_includes/header.html
storm/branches/bobby-versioned-site/releases/0.10.0/Acking-framework-implementation.md
storm/branches/bobby-versioned-site/releases/0.10.0/Clojure-DSL.md
storm/branches/bobby-versioned-site/releases/0.10.0/Command-line-client.md
storm/branches/bobby-versioned-site/releases/0.10.0/Common-patterns.md
storm/branches/bobby-versioned-site/releases/0.10.0/Concepts.md
storm/branches/bobby-versioned-site/releases/0.10.0/Configuration.md
storm/branches/bobby-versioned-site/releases/0.10.0/Contributing-to-Storm.md
storm/branches/bobby-versioned-site/releases/0.10.0/Creating-a-new-Storm-project.md
storm/branches/bobby-versioned-site/releases/0.10.0/DSLs-and-multilang-adapters.md
storm/branches/bobby-versioned-site/releases/0.10.0/Defining-a-non-jvm-language-dsl-for-storm.md
storm/branches/bobby-versioned-site/releases/0.10.0/Distributed-RPC.md
storm/branches/bobby-versioned-site/releases/0.10.0/FAQ.md
storm/branches/bobby-versioned-site/releases/0.10.0/Fault-tolerance.md
storm/branches/bobby-versioned-site/releases/0.10.0/Guaranteeing-message-processing.md
storm/branches/bobby-versioned-site/releases/0.10.0/Hooks.md
storm/branches/bobby-versioned-site/releases/0.10.0/Implementation-docs.md
storm/branches/bobby-versioned-site/releases/0.10.0/Installing-native-dependencies.md
storm/branches/bobby-versioned-site/releases/0.10.0/Kestrel-and-Storm.md
storm/branches/bobby-versioned-site/releases/0.10.0/Lifecycle-of-a-topology.md
storm/branches/bobby-versioned-site/releases/0.10.0/Local-mode.md
storm/branches/bobby-versioned-site/releases/0.10.0/Maven.md
storm/branches/bobby-versioned-site/releases/0.10.0/Message-passing-implementation.md
storm/branches/bobby-versioned-site/releases/0.10.0/Metrics.md
storm/branches/bobby-versioned-site/releases/0.10.0/Multilang-protocol.md
storm/branches/bobby-versioned-site/releases/0.10.0/Rationale.md
storm/branches/bobby-versioned-site/releases/0.10.0/Running-topologies-on-a-production-cluster.md
storm/branches/bobby-versioned-site/releases/0.10.0/Serialization.md
storm/branches/bobby-versioned-site/releases/0.10.0/Serializers.md
storm/branches/bobby-versioned-site/releases/0.10.0/Setting-up-a-Storm-cluster.md
storm/branches/bobby-versioned-site/releases/0.10.0/Setting-up-development-environment.md
storm/branches/bobby-versioned-site/releases/0.10.0/Spout-implementations.md
storm/branches/bobby-versioned-site/releases/0.10.0/Structure-of-the-codebase.md
storm/branches/bobby-versioned-site/releases/0.10.0/Support-for-non-java-languages.md
storm/branches/bobby-versioned-site/releases/0.10.0/Transactional-topologies.md
storm/branches/bobby-versioned-site/releases/0.10.0/Trident-API-Overview.md
storm/branches/bobby-versioned-site/releases/0.10.0/Trident-spouts.md
storm/branches/bobby-versioned-site/releases/0.10.0/Trident-state.md
storm/branches/bobby-versioned-site/releases/0.10.0/Trident-tutorial.md
storm/branches/bobby-versioned-site/releases/0.10.0/Troubleshooting.md
storm/branches/bobby-versioned-site/releases/0.10.0/Tutorial.md
storm/branches/bobby-versioned-site/releases/0.10.0/Understanding-the-parallelism-of-a-Storm-topology.md
storm/branches/bobby-versioned-site/releases/0.10.0/Using-non-JVM-languages-with-Storm.md
storm/branches/bobby-versioned-site/releases/0.10.0/index.md
Modified: storm/branches/bobby-versioned-site/_includes/footer.html
URL: http://svn.apache.org/viewvc/storm/branches/bobby-versioned-site/_includes/footer.html?rev=1735360&r1=1735359&r2=1735360&view=diff
==============================================================================
--- storm/branches/bobby-versioned-site/_includes/footer.html (original)
+++ storm/branches/bobby-versioned-site/_includes/footer.html Thu Mar 17 02:10:04 2016
@@ -22,10 +22,10 @@
<div class="footer-widget">
<h5>First Look</h5>
<ul class="footer-list">
- <li><a href="releases/current/Rationale.html">Rationale</a></li>
- <li><a href="releases/current/tutorial.html">Tutorial</a></li>
- <li><a href="releases/current/Setting-up-development-environment.html">Setting up development environment</a></li>
- <li><a href="releases/current/Creating-a-new-Storm-project.html">Creating a new Storm project</a></li>
+ <li><a href="/releases/current/Rationale.html">Rationale</a></li>
+ <li><a href="/releases/current/tutorial.html">Tutorial</a></li>
+ <li><a href="/releases/current/Setting-up-development-environment.html">Setting up development environment</a></li>
+ <li><a href="/releases/current/Creating-a-new-Storm-project.html">Creating a new Storm project</a></li>
</ul>
</div>
</div>
@@ -33,10 +33,9 @@
<div class="footer-widget">
<h5>Documentation</h5>
<ul class="footer-list">
- <li><a href="releases/current/index.html">Index</a></li>
- <li><a href="releases/current/documentation.html">Manual</a></li>
- <li><a href="releases/current/javadocs/index.html">Javadoc</a></li>
- <li><a href="releases/current/FAQ.html">FAQ</a></li>
+ <li><a href="/releases/current/index.html">Index</a></li>
+ <li><a href="/releases/current/javadocs/index.html">Javadoc</a></li>
+ <li><a href="/releases/current/FAQ.html">FAQ</a></li>
</ul>
</div>
</div>
Modified: storm/branches/bobby-versioned-site/_includes/header.html
URL: http://svn.apache.org/viewvc/storm/branches/bobby-versioned-site/_includes/header.html?rev=1735360&r1=1735359&r2=1735360&view=diff
==============================================================================
--- storm/branches/bobby-versioned-site/_includes/header.html (original)
+++ storm/branches/bobby-versioned-site/_includes/header.html Thu Mar 17 02:10:04 2016
@@ -1,12 +1,17 @@
<header>
<div class="container-fluid">
- <div class="row">
- <div class="col-md-10">
- <a href="/index.html"><img src="/images/logo.png" class="logo" /></a>
- </div>
- <div class="col-md-2">
- <a href="/releases.html" class="btn-std btn-block btn-download">Download</a>
- </div>
+ <div class="row">
+ <div class="col-md-5">
+ <a href="/index.html"><img src="/images/logo.png" class="logo" /></a>
+ </div>
+ <div class="col-md-5">
+ {% if page.version %}
+ <h1>Version: {{page.version}}</h1>
+ {% endif %}
+ </div>
+ <div class="col-md-2">
+ <a href="/releases.html" class="btn-std btn-block btn-download">Download</a>
+ </div>
</div>
</div>
</header>
Modified: storm/branches/bobby-versioned-site/releases/0.10.0/Acking-framework-implementation.md
URL: http://svn.apache.org/viewvc/storm/branches/bobby-versioned-site/releases/0.10.0/Acking-framework-implementation.md?rev=1735360&r1=1735359&r2=1735360&view=diff
==============================================================================
--- storm/branches/bobby-versioned-site/releases/0.10.0/Acking-framework-implementation.md (original)
+++ storm/branches/bobby-versioned-site/releases/0.10.0/Acking-framework-implementation.md Thu Mar 17 02:10:04 2016
@@ -2,14 +2,15 @@
title: Acking Framework Implementation
layout: documentation
documentation: true
+version: v0.10.0
---
-[Storm's acker](https://github.com/apache/storm/blob/46c3ba7/storm-core/src/clj/backtype/storm/daemon/acker.clj#L28) tracks completion of each tupletree with a checksum hash: each time a tuple is sent, its value is XORed into the checksum, and each time a tuple is acked its value is XORed in again. If all tuples have been successfully acked, the checksum will be zero (the odds that the checksum will be zero otherwise are vanishingly small).
+[Storm's acker](https://github.com/apache/storm/blob/{{page.version}}/storm-core/src/clj/backtype/storm/daemon/acker.clj) tracks completion of each tupletree with a checksum hash: each time a tuple is sent, its value is XORed into the checksum, and each time a tuple is acked its value is XORed in again. If all tuples have been successfully acked, the checksum will be zero (the odds that the checksum will be zero otherwise are vanishingly small).
You can read a bit more about the [reliability mechanism](Guaranteeing-message-processing.html#what-is-storms-reliability-api) elsewhere on the wiki -- this explains the internal details.
### acker `execute()`
-The acker is actually a regular bolt, with its [execute method](https://github.com/apache/storm/blob/46c3ba7/storm-core/src/clj/backtype/storm/daemon/acker.clj#L36) defined withing `mk-acker-bolt`. When a new tupletree is born, the spout sends the XORed edge-ids of each tuple recipient, which the acker records in its `pending` ledger. Every time an executor acks a tuple, the acker receives a partial checksum that is the XOR of the tuple's own edge-id (clearing it from the ledger) and the edge-id of each downstream tuple the executor emitted (thus entering them into the ledger).
+The acker is actually a regular bolt, with its [execute method](https://github.com/apache/storm/blob/{{page.version}}/storm-core/src/clj/backtype/storm/daemon/acker.clj) defined withing `mk-acker-bolt`. When a new tupletree is born, the spout sends the XORed edge-ids of each tuple recipient, which the acker records in its `pending` ledger. Every time an executor acks a tuple, the acker receives a partial checksum that is the XOR of the tuple's own edge-id (clearing it from the ledger) and the edge-id of each downstream tuple the executor emitted (thus entering them into the ledger).
This is accomplished as follows.
@@ -19,7 +20,7 @@ On a tick tuple, just advance pending tu
* on ack: xor the partial checksum into the existing checksum value
* on fail: just mark it as failed
-Next, [put the record](https://github.com/apache/storm/blob/46c3ba7/storm-core/src/clj/backtype/storm/daemon/acker.clj#L50)), into the RotatingMap (thus resetting is countdown to expiry) and take action:
+Next, [put the record](https://github.com/apache/storm/blob/{{page.version}}/storm-core/src/clj/backtype/storm/daemon/acker.clj)), into the RotatingMap (thus resetting is countdown to expiry) and take action:
* if the total checksum is zero, the tupletree is complete: remove it from the pending collection and notify the spout of success
* if the tupletree has failed, it is also complete: remove it from the pending collection and notify the spout of failure
@@ -28,7 +29,7 @@ Finally, pass on an ack of our own.
### Pending tuples and the `RotatingMap`
-The acker stores pending tuples in a [`RotatingMap`](https://github.com/apache/storm/blob/master/storm-core/src/jvm/backtype/storm/utils/RotatingMap.java#L19), a simple device used in several places within Storm to efficiently time-expire a process.
+The acker stores pending tuples in a [`RotatingMap`](https://github.com/apache/storm/blob/{{page.version}}/storm-core/src/jvm/backtype/storm/utils/RotatingMap.java), a simple device used in several places within Storm to efficiently time-expire a process.
The RotatingMap behaves as a HashMap, and offers the same O(1) access guarantees.
Modified: storm/branches/bobby-versioned-site/releases/0.10.0/Clojure-DSL.md
URL: http://svn.apache.org/viewvc/storm/branches/bobby-versioned-site/releases/0.10.0/Clojure-DSL.md?rev=1735360&r1=1735359&r2=1735360&view=diff
==============================================================================
--- storm/branches/bobby-versioned-site/releases/0.10.0/Clojure-DSL.md (original)
+++ storm/branches/bobby-versioned-site/releases/0.10.0/Clojure-DSL.md Thu Mar 17 02:10:04 2016
@@ -2,8 +2,9 @@
title: Clojure DSL
layout: documentation
documentation: true
+version: v0.10.0
---
-Storm comes with a Clojure DSL for defining spouts, bolts, and topologies. The Clojure DSL has access to everything the Java API exposes, so if you're a Clojure user you can code Storm topologies without touching Java at all. The Clojure DSL is defined in the source in the [backtype.storm.clojure](https://github.com/apache/storm/blob/0.5.3/src/clj/backtype/storm/clojure.clj) namespace.
+Storm comes with a Clojure DSL for defining spouts, bolts, and topologies. The Clojure DSL has access to everything the Java API exposes, so if you're a Clojure user you can code Storm topologies without touching Java at all. The Clojure DSL is defined in the source in the [backtype.storm.clojure](https://github.com/apache/storm/blob/{{page.version}}/storm-core/src/clj/backtype/storm/clojure.clj) namespace.
This page outlines all the pieces of the Clojure DSL, including:
@@ -17,7 +18,7 @@ This page outlines all the pieces of the
To define a topology, use the `topology` function. `topology` takes in two arguments: a map of "spout specs" and a map of "bolt specs". Each spout and bolt spec wires the code for the component into the topology by specifying things like inputs and parallelism.
-Let's take a look at an example topology definition [from the storm-starter project](https://github.com/apache/storm/blob/master/examples/storm-starter/src/clj/storm/starter/clj/word_count.clj):
+Let's take a look at an example topology definition [from the storm-starter project](https://github.com/apache/storm/blob/{{page.version}}/examples/storm-starter/src/clj/storm/starter/clj/word_count.clj):
```clojure
(topology
@@ -38,11 +39,11 @@ The maps of spout and bolt specs are map
#### spout-spec
-`spout-spec` takes as arguments the spout implementation (an object that implements [IRichSpout](/javadoc/apidocs/backtype/storm/topology/IRichSpout.html)) and optional keyword arguments. The only option that exists currently is the `:p` option, which specifies the parallelism for the spout. If you omit `:p`, the spout will execute as a single task.
+`spout-spec` takes as arguments the spout implementation (an object that implements [IRichSpout](javadocs/backtype/storm/topology/IRichSpout.html)) and optional keyword arguments. The only option that exists currently is the `:p` option, which specifies the parallelism for the spout. If you omit `:p`, the spout will execute as a single task.
#### bolt-spec
-`bolt-spec` takes as arguments the input declaration for the bolt, the bolt implementation (an object that implements [IRichBolt](/javadoc/apidocs/backtype/storm/topology/IRichBolt.html)), and optional keyword arguments.
+`bolt-spec` takes as arguments the input declaration for the bolt, the bolt implementation (an object that implements [IRichBolt](javadocs/backtype/storm/topology/IRichBolt.html)), and optional keyword arguments.
The input declaration is a map from stream ids to stream groupings. A stream id can have one of two forms:
@@ -203,7 +204,7 @@ The signature for `defspout` looks like
If you leave out the option map, it defaults to {:prepare true}. The output declaration for `defspout` has the same syntax as `defbolt`.
-Here's an example `defspout` implementation from [storm-starter](https://github.com/apache/storm/blob/master/examples/storm-starter/src/clj/storm/starter/clj/word_count.clj):
+Here's an example `defspout` implementation from [storm-starter](https://github.com/apache/storm/blob/{{page.version}}/examples/storm-starter/src/clj/storm/starter/clj/word_count.clj):
```clojure
(defspout sentence-spout ["sentence"]
Modified: storm/branches/bobby-versioned-site/releases/0.10.0/Command-line-client.md
URL: http://svn.apache.org/viewvc/storm/branches/bobby-versioned-site/releases/0.10.0/Command-line-client.md?rev=1735360&r1=1735359&r2=1735360&view=diff
==============================================================================
--- storm/branches/bobby-versioned-site/releases/0.10.0/Command-line-client.md (original)
+++ storm/branches/bobby-versioned-site/releases/0.10.0/Command-line-client.md Thu Mar 17 02:10:04 2016
@@ -2,6 +2,7 @@
title: Command Line Client
layout: documentation
documentation: true
+version: v0.10.0
---
This page describes all the commands that are possible with the "storm" command line client. To learn how to set up your "storm" client to talk to a remote cluster, follow the instructions in [Setting up development environment](Setting-up-a-development-environment.html).
@@ -25,7 +26,7 @@ These commands are:
Syntax: `storm jar topology-jar-path class ...`
-Runs the main method of `class` with the specified arguments. The storm jars and configs in `~/.storm` are put on the classpath. The process is configured so that [StormSubmitter](/javadoc/apidocs/backtype/storm/StormSubmitter.html) will upload the jar at `topology-jar-path` when the topology is submitted.
+Runs the main method of `class` with the specified arguments. The storm jars and configs in `~/.storm` are put on the classpath. The process is configured so that [StormSubmitter](javadocs/backtype/storm/StormSubmitter.html) will upload the jar at `topology-jar-path` when the topology is submitted.
### kill
Modified: storm/branches/bobby-versioned-site/releases/0.10.0/Common-patterns.md
URL: http://svn.apache.org/viewvc/storm/branches/bobby-versioned-site/releases/0.10.0/Common-patterns.md?rev=1735360&r1=1735359&r2=1735360&view=diff
==============================================================================
--- storm/branches/bobby-versioned-site/releases/0.10.0/Common-patterns.md (original)
+++ storm/branches/bobby-versioned-site/releases/0.10.0/Common-patterns.md Thu Mar 17 02:10:04 2016
@@ -2,6 +2,7 @@
title: Common Topology Patterns
layout: documentation
documentation: true
+version: v0.10.0
---
This page lists a variety of common patterns in Storm topologies.
@@ -39,7 +40,7 @@ If you want reliability in your data pro
If the bolt emits tuples, then you may want to use multi-anchoring to ensure reliability. It all depends on the specific application. See [Guaranteeing message processing](Guaranteeing-message-processing.html) for more details on how reliability works.
### BasicBolt
-Many bolts follow a similar pattern of reading an input tuple, emitting zero or more tuples based on that input tuple, and then acking that input tuple immediately at the end of the execute method. Bolts that match this pattern are things like functions and filters. This is such a common pattern that Storm exposes an interface called [IBasicBolt](/javadoc/apidocs/backtype/storm/topology/IBasicBolt.html) that automates this pattern for you. See [Guaranteeing message processing](Guaranteeing-message-processing.html) for more information.
+Many bolts follow a similar pattern of reading an input tuple, emitting zero or more tuples based on that input tuple, and then acking that input tuple immediately at the end of the execute method. Bolts that match this pattern are things like functions and filters. This is such a common pattern that Storm exposes an interface called [IBasicBolt](javadocs/backtype/storm/topology/IBasicBolt.html) that automates this pattern for you. See [Guaranteeing message processing](Guaranteeing-message-processing.html) for more information.
### In-memory caching + fields grouping combo
@@ -70,7 +71,7 @@ builder.setBolt("merge", new MergeObject
.globalGrouping("rank");
```
-This pattern works because of the fields grouping done by the first bolt which gives the partitioning you need for this to be semantically correct. You can see an example of this pattern in storm-starter [here](https://github.com/apache/storm/blob/master/examples/storm-starter/src/jvm/storm/starter/RollingTopWords.java).
+This pattern works because of the fields grouping done by the first bolt which gives the partitioning you need for this to be semantically correct. You can see an example of this pattern in storm-starter [here](https://github.com/apache/storm/blob/{{page.version}}/examples/storm-starter/src/jvm/storm/starter/RollingTopWords.java).
If however you have a known skew in the data being processed it can be advantageous to use partialKeyGrouping instead of fieldsGrouping. This will distribute the load for each key between two downstream bolts instead of a single one.
@@ -83,15 +84,15 @@ builder.setBolt("merge", new MergeRanksO
.globalGrouping("rank");
```
-The topology needs an extra layer of processing to aggregate the partial counts from the upstream bolts but this only processes aggregated values now so the bolt it is not subject to the load caused by the skewed data. You can see an example of this pattern in storm-starter [here](https://github.com/apache/storm/blob/master/examples/storm-starter/src/jvm/storm/starter/SkewedRollingTopWords.java).
+The topology needs an extra layer of processing to aggregate the partial counts from the upstream bolts but this only processes aggregated values now so the bolt it is not subject to the load caused by the skewed data. You can see an example of this pattern in storm-starter [here](https://github.com/apache/storm/blob/{{page.version}}/examples/storm-starter/src/jvm/storm/starter/SkewedRollingTopWords.java).
### TimeCacheMap for efficiently keeping a cache of things that have been recently updated
-You sometimes want to keep a cache in memory of items that have been recently "active" and have items that have been inactive for some time be automatically expires. [TimeCacheMap](/javadoc/apidocs/backtype/storm/utils/TimeCacheMap.html) is an efficient data structure for doing this and provides hooks so you can insert callbacks whenever an item is expired.
+You sometimes want to keep a cache in memory of items that have been recently "active" and have items that have been inactive for some time be automatically expires. [TimeCacheMap](javadocs/backtype/storm/utils/TimeCacheMap.html) is an efficient data structure for doing this and provides hooks so you can insert callbacks whenever an item is expired.
### CoordinatedBolt and KeyedFairBolt for Distributed RPC
-When building distributed RPC applications on top of Storm, there are two common patterns that are usually needed. These are encapsulated by [CoordinatedBolt](/javadoc/apidocs/backtype/storm/task/CoordinatedBolt.html) and [KeyedFairBolt](/javadoc/apidocs/backtype/storm/task/KeyedFairBolt.html) which are part of the "standard library" that ships with the Storm codebase.
+When building distributed RPC applications on top of Storm, there are two common patterns that are usually needed. These are encapsulated by [CoordinatedBolt](javadocs/backtype/storm/task/CoordinatedBolt.html) and [KeyedFairBolt](javadocs/backtype/storm/task/KeyedFairBolt.html) which are part of the "standard library" that ships with the Storm codebase.
`CoordinatedBolt` wraps the bolt containing your logic and figures out when your bolt has received all the tuples for any given request. It makes heavy use of direct streams to do this.
Modified: storm/branches/bobby-versioned-site/releases/0.10.0/Concepts.md
URL: http://svn.apache.org/viewvc/storm/branches/bobby-versioned-site/releases/0.10.0/Concepts.md?rev=1735360&r1=1735359&r2=1735360&view=diff
==============================================================================
--- storm/branches/bobby-versioned-site/releases/0.10.0/Concepts.md (original)
+++ storm/branches/bobby-versioned-site/releases/0.10.0/Concepts.md Thu Mar 17 02:10:04 2016
@@ -2,6 +2,7 @@
title: Concepts
layout: documentation
documentation: true
+version: v0.10.0
---
This page lists the main concepts of Storm and links to resources where you can find more information. The concepts discussed are:
@@ -21,7 +22,7 @@ The logic for a realtime application is
**Resources:**
-* [TopologyBuilder](/javadoc/apidocs/backtype/storm/topology/TopologyBuilder.html): use this class to construct topologies in Java
+* [TopologyBuilder](javadocs/backtype/storm/topology/TopologyBuilder.html): use this class to construct topologies in Java
* [Running topologies on a production cluster](Running-topologies-on-a-production-cluster.html)
* [Local mode](Local-mode.html): Read this to learn how to develop and test topologies in local mode.
@@ -29,30 +30,28 @@ The logic for a realtime application is
The stream is the core abstraction in Storm. A stream is an unbounded sequence of tuples that is processed and created in parallel in a distributed fashion. Streams are defined with a schema that names the fields in the stream's tuples. By default, tuples can contain integers, longs, shorts, bytes, strings, doubles, floats, booleans, and byte arrays. You can also define your own serializers so that custom types can be used natively within tuples.
-Every stream is given an id when declared. Since single-stream spouts and bolts are so common, [OutputFieldsDeclarer](/javadoc/apidocs/backtype/storm/topology/OutputFieldsDeclarer.html) has convenience methods for declaring a single stream without specifying an id. In this case, the stream is given the default id of "default".
+Every stream is given an id when declared. Since single-stream spouts and bolts are so common, [OutputFieldsDeclarer](javadocs/backtype/storm/topology/OutputFieldsDeclarer.html) has convenience methods for declaring a single stream without specifying an id. In this case, the stream is given the default id of "default".
**Resources:**
-* [Tuple](/javadoc/apidocs/backtype/storm/tuple/Tuple.html): streams are composed of tuples
-* [OutputFieldsDeclarer](/javadoc/apidocs/backtype/storm/topology/OutputFieldsDeclarer.html): used to declare streams and their schemas
+* [Tuple](javadocs/backtype/storm/tuple/Tuple.html): streams are composed of tuples
+* [OutputFieldsDeclarer](javadocs/backtype/storm/topology/OutputFieldsDeclarer.html): used to declare streams and their schemas
* [Serialization](Serialization.html): Information about Storm's dynamic typing of tuples and declaring custom serializations
-* [ISerialization](/javadoc/apidocs/backtype/storm/serialization/ISerialization.html): custom serializers must implement this interface
-* [CONFIG.TOPOLOGY_SERIALIZATIONS](/javadoc/apidocs/backtype/storm/Config.html#TOPOLOGY_SERIALIZATIONS): custom serializers can be registered using this configuration
### Spouts
A spout is a source of streams in a topology. Generally spouts will read tuples from an external source and emit them into the topology (e.g. a Kestrel queue or the Twitter API). Spouts can either be __reliable__ or __unreliable__. A reliable spout is capable of replaying a tuple if it failed to be processed by Storm, whereas an unreliable spout forgets about the tuple as soon as it is emitted.
-Spouts can emit more than one stream. To do so, declare multiple streams using the `declareStream` method of [OutputFieldsDeclarer](/javadoc/apidocs/backtype/storm/topology/OutputFieldsDeclarer.html) and specify the stream to emit to when using the `emit` method on [SpoutOutputCollector](/javadoc/apidocs/backtype/storm/spout/SpoutOutputCollector.html).
+Spouts can emit more than one stream. To do so, declare multiple streams using the `declareStream` method of [OutputFieldsDeclarer](javadocs/backtype/storm/topology/OutputFieldsDeclarer.html) and specify the stream to emit to when using the `emit` method on [SpoutOutputCollector](javadocs/backtype/storm/spout/SpoutOutputCollector.html).
The main method on spouts is `nextTuple`. `nextTuple` either emits a new tuple into the topology or simply returns if there are no new tuples to emit. It is imperative that `nextTuple` does not block for any spout implementation, because Storm calls all the spout methods on the same thread.
-The other main methods on spouts are `ack` and `fail`. These are called when Storm detects that a tuple emitted from the spout either successfully completed through the topology or failed to be completed. `ack` and `fail` are only called for reliable spouts. See [the Javadoc](/javadoc/apidocs/backtype/storm/spout/ISpout.html) for more information.
+The other main methods on spouts are `ack` and `fail`. These are called when Storm detects that a tuple emitted from the spout either successfully completed through the topology or failed to be completed. `ack` and `fail` are only called for reliable spouts. See [the Javadoc](javadocs/backtype/storm/spout/ISpout.html) for more information.
**Resources:**
-* [IRichSpout](/javadoc/apidocs/backtype/storm/topology/IRichSpout.html): this is the interface that spouts must implement.
+* [IRichSpout](javadocs/backtype/storm/topology/IRichSpout.html): this is the interface that spouts must implement.
* [Guaranteeing message processing](Guaranteeing-message-processing.html)
### Bolts
@@ -61,26 +60,26 @@ All processing in topologies is done in
Bolts can do simple stream transformations. Doing complex stream transformations often requires multiple steps and thus multiple bolts. For example, transforming a stream of tweets into a stream of trending images requires at least two steps: a bolt to do a rolling count of retweets for each image, and one or more bolts to stream out the top X images (you can do this particular stream transformation in a more scalable way with three bolts than with two).
-Bolts can emit more than one stream. To do so, declare multiple streams using the `declareStream` method of [OutputFieldsDeclarer](/javadoc/apidocs/backtype/storm/topology/OutputFieldsDeclarer.html) and specify the stream to emit to when using the `emit` method on [OutputCollector](/javadoc/apidocs/backtype/storm/task/OutputCollector.html).
+Bolts can emit more than one stream. To do so, declare multiple streams using the `declareStream` method of [OutputFieldsDeclarer](javadocs/backtype/storm/topology/OutputFieldsDeclarer.html) and specify the stream to emit to when using the `emit` method on [OutputCollector](javadocs/backtype/storm/task/OutputCollector.html).
-When you declare a bolt's input streams, you always subscribe to specific streams of another component. If you want to subscribe to all the streams of another component, you have to subscribe to each one individually. [InputDeclarer](/javadoc/apidocs/backtype/storm/topology/InputDeclarer.html) has syntactic sugar for subscribing to streams declared on the default stream id. Saying `declarer.shuffleGrouping("1")` subscribes to the default stream on component "1" and is equivalent to `declarer.shuffleGrouping("1", DEFAULT_STREAM_ID)`.
+When you declare a bolt's input streams, you always subscribe to specific streams of another component. If you want to subscribe to all the streams of another component, you have to subscribe to each one individually. [InputDeclarer](javadocs/backtype/storm/topology/InputDeclarer.html) has syntactic sugar for subscribing to streams declared on the default stream id. Saying `declarer.shuffleGrouping("1")` subscribes to the default stream on component "1" and is equivalent to `declarer.shuffleGrouping("1", DEFAULT_STREAM_ID)`.
-The main method in bolts is the `execute` method which takes in as input a new tuple. Bolts emit new tuples using the [OutputCollector](/javadoc/apidocs/backtype/storm/task/OutputCollector.html) object. Bolts must call the `ack` method on the `OutputCollector` for every tuple they process so that Storm knows when tuples are completed (and can eventually determine that its safe to ack the original spout tuples). For the common case of processing an input tuple, emitting 0 or more tuples based on that tuple, and then acking the input tuple, Storm provides an [IBasicBolt](/javadoc/apidocs/backtype/storm/topology/IBasicBolt.html) interface which does the acking automatically.
+The main method in bolts is the `execute` method which takes in as input a new tuple. Bolts emit new tuples using the [OutputCollector](javadocs/backtype/storm/task/OutputCollector.html) object. Bolts must call the `ack` method on the `OutputCollector` for every tuple they process so that Storm knows when tuples are completed (and can eventually determine that its safe to ack the original spout tuples). For the common case of processing an input tuple, emitting 0 or more tuples based on that tuple, and then acking the input tuple, Storm provides an [IBasicBolt](javadocs/backtype/storm/topology/IBasicBolt.html) interface which does the acking automatically.
-Its perfectly fine to launch new threads in bolts that do processing asynchronously. [OutputCollector](/javadoc/apidocs/backtype/storm/task/OutputCollector.html) is thread-safe and can be called at any time.
+Its perfectly fine to launch new threads in bolts that do processing asynchronously. [OutputCollector](javadocs/backtype/storm/task/OutputCollector.html) is thread-safe and can be called at any time.
**Resources:**
-* [IRichBolt](/javadoc/apidocs/backtype/storm/topology/IRichBolt.html): this is general interface for bolts.
-* [IBasicBolt](/javadoc/apidocs/backtype/storm/topology/IBasicBolt.html): this is a convenience interface for defining bolts that do filtering or simple functions.
-* [OutputCollector](/javadoc/apidocs/backtype/storm/task/OutputCollector.html): bolts emit tuples to their output streams using an instance of this class
+* [IRichBolt](javadocs/backtype/storm/topology/IRichBolt.html): this is general interface for bolts.
+* [IBasicBolt](javadocs/backtype/storm/topology/IBasicBolt.html): this is a convenience interface for defining bolts that do filtering or simple functions.
+* [OutputCollector](javadocs/backtype/storm/task/OutputCollector.html): bolts emit tuples to their output streams using an instance of this class
* [Guaranteeing message processing](Guaranteeing-message-processing.html)
### Stream groupings
Part of defining a topology is specifying for each bolt which streams it should receive as input. A stream grouping defines how that stream should be partitioned among the bolt's tasks.
-There are seven built-in stream groupings in Storm, and you can implement a custom stream grouping by implementing the [CustomStreamGrouping](/javadoc/apidocs/backtype/storm/grouping/CustomStreamGrouping.html) interface:
+There are seven built-in stream groupings in Storm, and you can implement a custom stream grouping by implementing the [CustomStreamGrouping](javadocs/backtype/storm/grouping/CustomStreamGrouping.html) interface:
1. **Shuffle grouping**: Tuples are randomly distributed across the bolt's tasks in a way such that each bolt is guaranteed to get an equal number of tuples.
2. **Fields grouping**: The stream is partitioned by the fields specified in the grouping. For example, if the stream is grouped by the "user-id" field, tuples with the same "user-id" will always go to the same task, but tuples with different "user-id"'s may go to different tasks.
@@ -88,26 +87,25 @@ There are seven built-in stream grouping
4. **All grouping**: The stream is replicated across all the bolt's tasks. Use this grouping with care.
5. **Global grouping**: The entire stream goes to a single one of the bolt's tasks. Specifically, it goes to the task with the lowest id.
6. **None grouping**: This grouping specifies that you don't care how the stream is grouped. Currently, none groupings are equivalent to shuffle groupings. Eventually though, Storm will push down bolts with none groupings to execute in the same thread as the bolt or spout they subscribe from (when possible).
-7. **Direct grouping**: This is a special kind of grouping. A stream grouped this way means that the __producer__ of the tuple decides which task of the consumer will receive this tuple. Direct groupings can only be declared on streams that have been declared as direct streams. Tuples emitted to a direct stream must be emitted using one of the [emitDirect](/javadoc/apidocs/backtype/storm/task/OutputCollector.html#emitDirect(int, int, java.util.List) methods. A bolt can get the task ids of its consumers by either using the provided [TopologyContext](/javadoc/apidocs/backtype/storm/task/TopologyContext.html) or by keeping track of the output of the `emit` method in [OutputCollector](/javadoc/apidocs/backtype/storm/task/OutputCollector.html) (which returns the task ids that the tuple was sent to).
+7. **Direct grouping**: This is a special kind of grouping. A stream grouped this way means that the __producer__ of the tuple decides which task of the consumer will receive this tuple. Direct groupings can only be declared on streams that have been declared as direct streams. Tuples emitted to a direct stream must be emitted using one of the [emitDirect](javadocs/backtype/storm/task/OutputCollector.html#emitDirect(int, int, java.util.List) methods. A bolt can get the task ids of its consumers by either using the provided [TopologyContext](javadocs/backtype/storm/task/TopologyContext.html) or by keeping track of the output of the `emit` method in [OutputCollector](javadocs/backtype/storm/task/OutputCollector.html) (which returns the task ids that the tuple was sent to).
8. **Local or shuffle grouping**: If the target bolt has one or more tasks in the same worker process, tuples will be shuffled to just those in-process tasks. Otherwise, this acts like a normal shuffle grouping.
**Resources:**
-* [TopologyBuilder](/javadoc/apidocs/backtype/storm/topology/TopologyBuilder.html): use this class to define topologies
-* [InputDeclarer](/javadoc/apidocs/backtype/storm/topology/InputDeclarer.html): this object is returned whenever `setBolt` is called on `TopologyBuilder` and is used for declaring a bolt's input streams and how those streams should be grouped
-* [CoordinatedBolt](/javadoc/apidocs/backtype/storm/task/CoordinatedBolt.html): this bolt is useful for distributed RPC topologies and makes heavy use of direct streams and direct groupings
+* [TopologyBuilder](javadocs/backtype/storm/topology/TopologyBuilder.html): use this class to define topologies
+* [InputDeclarer](javadocs/backtype/storm/topology/InputDeclarer.html): this object is returned whenever `setBolt` is called on `TopologyBuilder` and is used for declaring a bolt's input streams and how those streams should be grouped
### Reliability
Storm guarantees that every spout tuple will be fully processed by the topology. It does this by tracking the tree of tuples triggered by every spout tuple and determining when that tree of tuples has been successfully completed. Every topology has a "message timeout" associated with it. If Storm fails to detect that a spout tuple has been completed within that timeout, then it fails the tuple and replays it later.
-To take advantage of Storm's reliability capabilities, you must tell Storm when new edges in a tuple tree are being created and tell Storm whenever you've finished processing an individual tuple. These are done using the [OutputCollector](/javadoc/apidocs/backtype/storm/task/OutputCollector.html) object that bolts use to emit tuples. Anchoring is done in the `emit` method, and you declare that you're finished with a tuple using the `ack` method.
+To take advantage of Storm's reliability capabilities, you must tell Storm when new edges in a tuple tree are being created and tell Storm whenever you've finished processing an individual tuple. These are done using the [OutputCollector](javadocs/backtype/storm/task/OutputCollector.html) object that bolts use to emit tuples. Anchoring is done in the `emit` method, and you declare that you're finished with a tuple using the `ack` method.
This is all explained in much more detail in [Guaranteeing message processing](Guaranteeing-message-processing.html).
### Tasks
-Each spout or bolt executes as many tasks across the cluster. Each task corresponds to one thread of execution, and stream groupings define how to send tuples from one set of tasks to another set of tasks. You set the parallelism for each spout or bolt in the `setSpout` and `setBolt` methods of [TopologyBuilder](/javadoc/apidocs/backtype/storm/topology/TopologyBuilder.html).
+Each spout or bolt executes as many tasks across the cluster. Each task corresponds to one thread of execution, and stream groupings define how to send tuples from one set of tasks to another set of tasks. You set the parallelism for each spout or bolt in the `setSpout` and `setBolt` methods of [TopologyBuilder](javadocs/backtype/storm/topology/TopologyBuilder.html).
### Workers
@@ -115,4 +113,4 @@ Topologies execute across one or more wo
**Resources:**
-* [Config.TOPOLOGY_WORKERS](/javadoc/apidocs/backtype/storm/Config.html#TOPOLOGY_WORKERS): this config sets the number of workers to allocate for executing the topology
+* [Config.TOPOLOGY_WORKERS](javadocs/backtype/storm/Config.html#TOPOLOGY_WORKERS): this config sets the number of workers to allocate for executing the topology
Modified: storm/branches/bobby-versioned-site/releases/0.10.0/Configuration.md
URL: http://svn.apache.org/viewvc/storm/branches/bobby-versioned-site/releases/0.10.0/Configuration.md?rev=1735360&r1=1735359&r2=1735360&view=diff
==============================================================================
--- storm/branches/bobby-versioned-site/releases/0.10.0/Configuration.md (original)
+++ storm/branches/bobby-versioned-site/releases/0.10.0/Configuration.md Thu Mar 17 02:10:04 2016
@@ -2,10 +2,11 @@
title: Configuration
layout: documentation
documentation: true
+version: v0.10.0
---
Storm has a variety of configurations for tweaking the behavior of nimbus, supervisors, and running topologies. Some configurations are system configurations and cannot be modified on a topology by topology basis, whereas other configurations can be modified per topology.
-Every configuration has a default value defined in [defaults.yaml](https://github.com/apache/storm/blob/master/conf/defaults.yaml) in the Storm codebase. You can override these configurations by defining a storm.yaml in the classpath of Nimbus and the supervisors. Finally, you can define a topology-specific configuration that you submit along with your topology when using [StormSubmitter](/javadoc/apidocs/backtype/storm/StormSubmitter.html). However, the topology-specific configuration can only override configs prefixed with "TOPOLOGY".
+Every configuration has a default value defined in [defaults.yaml](https://github.com/apache/storm/{{page.version}}/master/conf/defaults.yaml) in the Storm codebase. You can override these configurations by defining a storm.yaml in the classpath of Nimbus and the supervisors. Finally, you can define a topology-specific configuration that you submit along with your topology when using [StormSubmitter](javadocs/backtype/storm/StormSubmitter.html). However, the topology-specific configuration can only override configs prefixed with "TOPOLOGY".
Storm 0.7.0 and onwards lets you override configuration on a per-bolt/per-spout basis. The only configurations that can be overriden this way are:
@@ -25,7 +26,7 @@ The preference order for configuration v
**Resources:**
* [Config](javadocs/backtype/storm/Config.html): a listing of all configurations as well as a helper class for creating topology specific configurations
-* [defaults.yaml](https://github.com/apache/storm/blob/master/conf/defaults.yaml): the default values for all configurations
+* [defaults.yaml](https://github.com/apache/storm/blob/{{page.version}}/conf/defaults.yaml): the default values for all configurations
* [Setting up a Storm cluster](Setting-up-a-Storm-cluster.html): explains how to create and configure a Storm cluster
* [Running topologies on a production cluster](Running-topologies-on-a-production-cluster.html): lists useful configurations when running topologies on a cluster
* [Local mode](Local-mode.html): lists useful configurations when using local mode
Modified: storm/branches/bobby-versioned-site/releases/0.10.0/Contributing-to-Storm.md
URL: http://svn.apache.org/viewvc/storm/branches/bobby-versioned-site/releases/0.10.0/Contributing-to-Storm.md?rev=1735360&r1=1735359&r2=1735360&view=diff
==============================================================================
--- storm/branches/bobby-versioned-site/releases/0.10.0/Contributing-to-Storm.md (original)
+++ storm/branches/bobby-versioned-site/releases/0.10.0/Contributing-to-Storm.md Thu Mar 17 02:10:04 2016
@@ -2,6 +2,7 @@
title: Contributing
layout: documentation
documentation: true
+version: v0.10.0
---
### Getting started with contributing
Modified: storm/branches/bobby-versioned-site/releases/0.10.0/Creating-a-new-Storm-project.md
URL: http://svn.apache.org/viewvc/storm/branches/bobby-versioned-site/releases/0.10.0/Creating-a-new-Storm-project.md?rev=1735360&r1=1735359&r2=1735360&view=diff
==============================================================================
--- storm/branches/bobby-versioned-site/releases/0.10.0/Creating-a-new-Storm-project.md (original)
+++ storm/branches/bobby-versioned-site/releases/0.10.0/Creating-a-new-Storm-project.md Thu Mar 17 02:10:04 2016
@@ -2,17 +2,18 @@
title: Creating a New Storm Project
layout: documentation
documentation: true
+version: v0.10.0
---
This page outlines how to set up a Storm project for development. The steps are:
1. Add Storm jars to classpath
2. If using multilang, add multilang dir to classpath
-Follow along to see how to set up the [storm-starter](https://github.com/apache/storm/blob/master/examples/storm-starter) project in Eclipse.
+Follow along to see how to set up the [storm-starter](https://github.com/apache/storm/blob/{{page.version}}/examples/storm-starter) project in Eclipse.
### Add Storm jars to classpath
-You'll need the Storm jars on your classpath to develop Storm topologies. Using [Maven](Maven.html) is highly recommended. [Here's an example](https://github.com/apache/storm/blob/master/examples/storm-starter/pom.xml) of how to setup your pom.xml for a Storm project. If you don't want to use Maven, you can include the jars from the Storm release on your classpath.
+You'll need the Storm jars on your classpath to develop Storm topologies. Using [Maven](Maven.html) is highly recommended. [Here's an example](https://github.com/apache/storm/blob/{{page.version}}/examples/storm-starter/pom.xml) of how to setup your pom.xml for a Storm project. If you don't want to use Maven, you can include the jars from the Storm release on your classpath.
To set up the classpath in Eclipse, create a new Java project, include `src/jvm/` as a source path, and make sure all the jars in `lib/` and `lib/dev/` are in the `Referenced Libraries` section of the project.
Modified: storm/branches/bobby-versioned-site/releases/0.10.0/DSLs-and-multilang-adapters.md
URL: http://svn.apache.org/viewvc/storm/branches/bobby-versioned-site/releases/0.10.0/DSLs-and-multilang-adapters.md?rev=1735360&r1=1735359&r2=1735360&view=diff
==============================================================================
--- storm/branches/bobby-versioned-site/releases/0.10.0/DSLs-and-multilang-adapters.md (original)
+++ storm/branches/bobby-versioned-site/releases/0.10.0/DSLs-and-multilang-adapters.md Thu Mar 17 02:10:04 2016
@@ -2,6 +2,7 @@
title: Storm DSLs and Multi-Lang Adapters
layout: documentation
documentation: true
+version: v0.10.0
---
* [Scala DSL](https://github.com/velvia/ScalaStorm)
* [JRuby DSL](https://github.com/colinsurprenant/redstorm)
Modified: storm/branches/bobby-versioned-site/releases/0.10.0/Defining-a-non-jvm-language-dsl-for-storm.md
URL: http://svn.apache.org/viewvc/storm/branches/bobby-versioned-site/releases/0.10.0/Defining-a-non-jvm-language-dsl-for-storm.md?rev=1735360&r1=1735359&r2=1735360&view=diff
==============================================================================
--- storm/branches/bobby-versioned-site/releases/0.10.0/Defining-a-non-jvm-language-dsl-for-storm.md (original)
+++ storm/branches/bobby-versioned-site/releases/0.10.0/Defining-a-non-jvm-language-dsl-for-storm.md Thu Mar 17 02:10:04 2016
@@ -2,8 +2,9 @@
title: Defining a Non-JVM DSL for Storm
layout: documentation
documentation: true
+version: v0.10.0
---
-The right place to start to learn how to make a non-JVM DSL for Storm is [storm-core/src/storm.thrift](https://github.com/apache/storm/blob/master/storm-core/src/storm.thrift). Since Storm topologies are just Thrift structures, and Nimbus is a Thrift daemon, you can create and submit topologies in any language.
+The right place to start to learn how to make a non-JVM DSL for Storm is [storm-core/src/storm.thrift](https://github.com/apache/storm/blob/{{page.version}}/storm-core/src/storm.thrift). Since Storm topologies are just Thrift structures, and Nimbus is a Thrift daemon, you can create and submit topologies in any language.
When you create the Thrift structs for spouts and bolts, the code for the spout or bolt is specified in the ComponentObject struct:
@@ -35,4 +36,4 @@ Then you can connect to Nimbus using the
void submitTopology(1: string name, 2: string uploadedJarLocation, 3: string jsonConf, 4: StormTopology topology) throws (1: AlreadyAliveException e, 2: InvalidTopologyException ite);
```
-Finally, one of the key things to do in a non-JVM DSL is make it easy to define the entire topology in one file (the bolts, spouts, and the definition of the topology).
\ No newline at end of file
+Finally, one of the key things to do in a non-JVM DSL is make it easy to define the entire topology in one file (the bolts, spouts, and the definition of the topology).
Modified: storm/branches/bobby-versioned-site/releases/0.10.0/Distributed-RPC.md
URL: http://svn.apache.org/viewvc/storm/branches/bobby-versioned-site/releases/0.10.0/Distributed-RPC.md?rev=1735360&r1=1735359&r2=1735360&view=diff
==============================================================================
--- storm/branches/bobby-versioned-site/releases/0.10.0/Distributed-RPC.md (original)
+++ storm/branches/bobby-versioned-site/releases/0.10.0/Distributed-RPC.md Thu Mar 17 02:10:04 2016
@@ -2,6 +2,7 @@
title: Distributed RPC
layout: documentation
documentation: true
+version: v0.10.0
---
The idea behind distributed RPC (DRPC) is to parallelize the computation of really intense functions on the fly using Storm. The Storm topology takes in as input a stream of function arguments, and it emits an output stream of the results for each of those function calls.
@@ -24,7 +25,7 @@ A client sends the DRPC server the name
### LinearDRPCTopologyBuilder
-Storm comes with a topology builder called [LinearDRPCTopologyBuilder](/javadoc/apidocs/backtype/storm/drpc/LinearDRPCTopologyBuilder.html) that automates almost all the steps involved for doing DRPC. These include:
+Storm comes with a topology builder called [LinearDRPCTopologyBuilder](javadocs/backtype/storm/drpc/LinearDRPCTopologyBuilder.html) that automates almost all the steps involved for doing DRPC. These include:
1. Setting up the spout
2. Returning the results to the DRPC server
@@ -118,7 +119,7 @@ The reach of a URL is the number of uniq
A single reach computation can involve thousands of database calls and tens of millions of follower records during the computation. It's a really, really intense computation. As you're about to see, implementing this function on top of Storm is dead simple. On a single machine, reach can take minutes to compute; on a Storm cluster, you can compute reach for even the hardest URLs in a couple seconds.
-A sample reach topology is defined in storm-starter [here](https://github.com/apache/storm/blob/master/examples/storm-starter/src/jvm/storm/starter/ReachTopology.java). Here's how you define the reach topology:
+A sample reach topology is defined in storm-starter [here](https://github.com/apache/storm/blob/{{page.version}}/examples/storm-starter/src/jvm/storm/starter/ReachTopology.java). Here's how you define the reach topology:
```java
LinearDRPCTopologyBuilder builder = new LinearDRPCTopologyBuilder("reach");
@@ -196,4 +197,4 @@ The rest of the topology should be self-
### Advanced
* KeyedFairBolt for weaving the processing of multiple requests at the same time
-* How to use `CoordinatedBolt` directly
\ No newline at end of file
+* How to use `CoordinatedBolt` directly
Modified: storm/branches/bobby-versioned-site/releases/0.10.0/FAQ.md
URL: http://svn.apache.org/viewvc/storm/branches/bobby-versioned-site/releases/0.10.0/FAQ.md?rev=1735360&r1=1735359&r2=1735360&view=diff
==============================================================================
--- storm/branches/bobby-versioned-site/releases/0.10.0/FAQ.md (original)
+++ storm/branches/bobby-versioned-site/releases/0.10.0/FAQ.md Thu Mar 17 02:10:04 2016
@@ -2,6 +2,7 @@
title: FAQ
layout: documentation
documentation: true
+version: v0.10.0
---
## Best Practices
@@ -74,11 +75,11 @@ You should only store static data, and a
### How often is the 'emitPartitionBatchNew' function called?
-Since the MBC is the actual spout, all the tuples in a batch are just members of its tupletree. That means storm's "max spout pending" config effectively defines the number of concurrent batches trident runs. The MBC emits a new batch if it has fewer than max-spending tuples pending and if at least one [trident batch interval](https://github.com/apache/storm/blob/master/conf/defaults.yaml#L115)'s worth of seconds has passed since the last batch.
+Since the MBC is the actual spout, all the tuples in a batch are just members of its tupletree. That means storm's "max spout pending" config effectively defines the number of concurrent batches trident runs. The MBC emits a new batch if it has fewer than max-spending tuples pending and if at least one [trident batch interval](https://github.com/apache/storm/blob/{{page.version}}/conf/defaults.yaml#L115)'s worth of seconds has passed since the last batch.
### If nothing was emitted does Trident slow down the calls?
-Yes, there's a pluggable "spout wait strategy"; the default is to sleep for a [configurable amount of time](https://github.com/apache/storm/blob/master/conf/defaults.yaml#L110)
+Yes, there's a pluggable "spout wait strategy"; the default is to sleep for a [configurable amount of time](https://github.com/apache/storm/blob/{{page.version}}/conf/defaults.yaml#L110)
### OK, then what is the trident batch interval for?
@@ -120,4 +121,4 @@ You cannot know that all events are coll
* Set a time limit using domain knowledge
* Introduce a _punctuation_: a record known to come after all records in the given time bucket. Trident uses this scheme to know when a batch is complete. If you for instance receive records from a set of sensors, each in order for that sensor, then once all sensors have sent you a 3:02:xx or later timestamp lets you know you can commit.
* When possible, make your process incremental: each value that comes in makes the answer more an more true. A Trident ReducerAggregator is an operator that takes a prior result and a set of new records and returns a new result. This lets the result be cached and serialized to a datastore; if a server drops off line for a day and then comes back with a full day's worth of data in a rush, the old results will be calmly retrieved and updated.
-* Lambda architecture: Record all events into an archival store (S3, HBase, HDFS) on receipt. in the fast layer, once the time window is clear, process the bucket to get an actionable answer, and ignore everything older than the time window. Periodically run a global aggregation to calculate a "correct" answer.
\ No newline at end of file
+* Lambda architecture: Record all events into an archival store (S3, HBase, HDFS) on receipt. in the fast layer, once the time window is clear, process the bucket to get an actionable answer, and ignore everything older than the time window. Periodically run a global aggregation to calculate a "correct" answer.
Modified: storm/branches/bobby-versioned-site/releases/0.10.0/Fault-tolerance.md
URL: http://svn.apache.org/viewvc/storm/branches/bobby-versioned-site/releases/0.10.0/Fault-tolerance.md?rev=1735360&r1=1735359&r2=1735360&view=diff
==============================================================================
--- storm/branches/bobby-versioned-site/releases/0.10.0/Fault-tolerance.md (original)
+++ storm/branches/bobby-versioned-site/releases/0.10.0/Fault-tolerance.md Thu Mar 17 02:10:04 2016
@@ -2,6 +2,7 @@
title: Fault Tolerance
layout: documentation
documentation: true
+version: v0.10.0
---
This page explains the design details of Storm that make it a fault-tolerant system.
@@ -27,4 +28,4 @@ So the answer is that Nimbus is "sort of
## How does Storm guarantee data processing?
-Storm provides mechanisms to guarantee data processing even if nodes die or messages are lost. See [Guaranteeing message processing](Guaranteeing-message-processing.html) for the details.
\ No newline at end of file
+Storm provides mechanisms to guarantee data processing even if nodes die or messages are lost. See [Guaranteeing message processing](Guaranteeing-message-processing.html) for the details.
Modified: storm/branches/bobby-versioned-site/releases/0.10.0/Guaranteeing-message-processing.md
URL: http://svn.apache.org/viewvc/storm/branches/bobby-versioned-site/releases/0.10.0/Guaranteeing-message-processing.md?rev=1735360&r1=1735359&r2=1735360&view=diff
==============================================================================
--- storm/branches/bobby-versioned-site/releases/0.10.0/Guaranteeing-message-processing.md (original)
+++ storm/branches/bobby-versioned-site/releases/0.10.0/Guaranteeing-message-processing.md Thu Mar 17 02:10:04 2016
@@ -2,6 +2,7 @@
title: Guaranteeing Message Processing
layout: documentation
documentation: true
+version: v0.10.0
---
Storm guarantees that each message coming off a spout will be fully processed. This page describes how Storm accomplishes this guarantee and what you have to do as a user to benefit from Storm's reliability capabilities.
@@ -25,11 +26,11 @@ This topology reads sentences off of a K
-Storm considers a tuple coming off a spout "fully processed" when the tuple tree has been exhausted and every message in the tree has been processed. A tuple is considered failed when its tree of messages fails to be fully processed within a specified timeout. This timeout can be configured on a topology-specific basis using the [Config.TOPOLOGY_MESSAGE_TIMEOUT_SECS](/javadoc/apidocs/backtype/storm/Config.html#TOPOLOGY_MESSAGE_TIMEOUT_SECS) configuration and defaults to 30 seconds.
+Storm considers a tuple coming off a spout "fully processed" when the tuple tree has been exhausted and every message in the tree has been processed. A tuple is considered failed when its tree of messages fails to be fully processed within a specified timeout. This timeout can be configured on a topology-specific basis using the [Config.TOPOLOGY_MESSAGE_TIMEOUT_SECS](javadocs/backtype/storm/Config.html#TOPOLOGY_MESSAGE_TIMEOUT_SECS) configuration and defaults to 30 seconds.
### What happens if a message is fully processed or fails to be fully processed?
-To understand this question, let's take a look at the lifecycle of a tuple coming off of a spout. For reference, here is the interface that spouts implement (see the [Javadoc](/javadoc/apidocs/backtype/storm/spout/ISpout.html) for more information):
+To understand this question, let's take a look at the lifecycle of a tuple coming off of a spout. For reference, here is the interface that spouts implement (see the [Javadoc](javadocs/backtype/storm/spout/ISpout.html) for more information):
```java
public interface ISpout extends Serializable {
@@ -136,7 +137,7 @@ As always in software design, the answer
### How does Storm implement reliability in an efficient way?
-A Storm topology has a set of special "acker" tasks that track the DAG of tuples for every spout tuple. When an acker sees that a DAG is complete, it sends a message to the spout task that created the spout tuple to ack the message. You can set the number of acker tasks for a topology in the topology configuration using [Config.TOPOLOGY_ACKERS](/javadoc/apidocs/backtype/storm/Config.html#TOPOLOGY_ACKERS). Storm defaults TOPOLOGY_ACKERS to one task -- you will need to increase this number for topologies processing large amounts of messages.
+A Storm topology has a set of special "acker" tasks that track the DAG of tuples for every spout tuple. When an acker sees that a DAG is complete, it sends a message to the spout task that created the spout tuple to ack the message. You can set the number of acker tasks for a topology in the topology configuration using [Config.TOPOLOGY_ACKERS](javadocs/backtype/storm/Config.html#TOPOLOGY_ACKERS). Storm defaults TOPOLOGY_ACKERS to one task -- you will need to increase this number for topologies processing large amounts of messages.
The best way to understand Storm's reliability implementation is to look at the lifecycle of tuples and tuple DAGs. When a tuple is created in a topology, whether in a spout or a bolt, it is given a random 64 bit id. These ids are used by ackers to track the tuple DAG for every spout tuple.
@@ -178,4 +179,4 @@ There are three ways to remove reliabili
The second way is to remove reliability on a message by message basis. You can turn off tracking for an individual spout tuple by omitting a message id in the `SpoutOutputCollector.emit` method.
-Finally, if you don't care if a particular subset of the tuples downstream in the topology fail to be processed, you can emit them as unanchored tuples. Since they're not anchored to any spout tuples, they won't cause any spout tuples to fail if they aren't acked.
\ No newline at end of file
+Finally, if you don't care if a particular subset of the tuples downstream in the topology fail to be processed, you can emit them as unanchored tuples. Since they're not anchored to any spout tuples, they won't cause any spout tuples to fail if they aren't acked.
Modified: storm/branches/bobby-versioned-site/releases/0.10.0/Hooks.md
URL: http://svn.apache.org/viewvc/storm/branches/bobby-versioned-site/releases/0.10.0/Hooks.md?rev=1735360&r1=1735359&r2=1735360&view=diff
==============================================================================
--- storm/branches/bobby-versioned-site/releases/0.10.0/Hooks.md (original)
+++ storm/branches/bobby-versioned-site/releases/0.10.0/Hooks.md Thu Mar 17 02:10:04 2016
@@ -2,8 +2,9 @@
title: Hooks
layout: documentation
documentation: true
+version: v0.10.0
---
-Storm provides hooks with which you can insert custom code to run on any number of events within Storm. You create a hook by extending the [BaseTaskHook](/javadoc/apidocs/backtype/storm/hooks/BaseTaskHook.html) class and overriding the appropriate method for the event you want to catch. There are two ways to register your hook:
+Storm provides hooks with which you can insert custom code to run on any number of events within Storm. You create a hook by extending the [BaseTaskHook](javadocs/backtype/storm/hooks/BaseTaskHook.html) class and overriding the appropriate method for the event you want to catch. There are two ways to register your hook:
-1. In the open method of your spout or prepare method of your bolt using the [TopologyContext#addTaskHook](/javadoc/apidocs/backtype/storm/task/TopologyContext.html) method.
-2. Through the Storm configuration using the ["topology.auto.task.hooks"](/javadoc/apidocs/backtype/storm/Config.html#TOPOLOGY_AUTO_TASK_HOOKS) config. These hooks are automatically registered in every spout or bolt, and are useful for doing things like integrating with a custom monitoring system.
\ No newline at end of file
+1. In the open method of your spout or prepare method of your bolt using the [TopologyContext#addTaskHook](javadocs/backtype/storm/task/TopologyContext.html) method.
+2. Through the Storm configuration using the ["topology.auto.task.hooks"](javadocs/backtype/storm/Config.html#TOPOLOGY_AUTO_TASK_HOOKS) config. These hooks are automatically registered in every spout or bolt, and are useful for doing things like integrating with a custom monitoring system.
Modified: storm/branches/bobby-versioned-site/releases/0.10.0/Implementation-docs.md
URL: http://svn.apache.org/viewvc/storm/branches/bobby-versioned-site/releases/0.10.0/Implementation-docs.md?rev=1735360&r1=1735359&r2=1735360&view=diff
==============================================================================
--- storm/branches/bobby-versioned-site/releases/0.10.0/Implementation-docs.md (original)
+++ storm/branches/bobby-versioned-site/releases/0.10.0/Implementation-docs.md Thu Mar 17 02:10:04 2016
@@ -2,6 +2,7 @@
title: Storm Internal Implementation
layout: documentation
documentation: true
+version: v0.10.0
---
This section of the wiki is dedicated to explaining how Storm is implemented. You should have a good grasp of how to use Storm before reading these sections.
Modified: storm/branches/bobby-versioned-site/releases/0.10.0/Installing-native-dependencies.md
URL: http://svn.apache.org/viewvc/storm/branches/bobby-versioned-site/releases/0.10.0/Installing-native-dependencies.md?rev=1735360&r1=1735359&r2=1735360&view=diff
==============================================================================
--- storm/branches/bobby-versioned-site/releases/0.10.0/Installing-native-dependencies.md (original)
+++ storm/branches/bobby-versioned-site/releases/0.10.0/Installing-native-dependencies.md Thu Mar 17 02:10:04 2016
@@ -1,5 +1,6 @@
---
layout: documentation
+version: v0.10.0
---
The native dependencies are only needed on actual Storm clusters. When running Storm in local mode, Storm uses a pure Java messaging system so that you don't need to install native dependencies on your development machine.
@@ -35,4 +36,4 @@ To get the JZMQ build to work, you may n
3. Upgrade autoconf on your machine
4. Follow the instructions in [this blog post](http://blog.pmorelli.com/getting-zeromq-and-jzmq-running-on-mac-os-x)
-If you run into any errors when running `./configure`, [this thread](http://stackoverflow.com/questions/3522248/how-do-i-compile-jzmq-for-zeromq-on-osx) may provide a solution.
\ No newline at end of file
+If you run into any errors when running `./configure`, [this thread](http://stackoverflow.com/questions/3522248/how-do-i-compile-jzmq-for-zeromq-on-osx) may provide a solution.
Modified: storm/branches/bobby-versioned-site/releases/0.10.0/Kestrel-and-Storm.md
URL: http://svn.apache.org/viewvc/storm/branches/bobby-versioned-site/releases/0.10.0/Kestrel-and-Storm.md?rev=1735360&r1=1735359&r2=1735360&view=diff
==============================================================================
--- storm/branches/bobby-versioned-site/releases/0.10.0/Kestrel-and-Storm.md (original)
+++ storm/branches/bobby-versioned-site/releases/0.10.0/Kestrel-and-Storm.md Thu Mar 17 02:10:04 2016
@@ -2,12 +2,13 @@
title: Storm and Kestrel
layout: documentation
documentation: true
+version: v0.10.0
---
This page explains how to use to Storm to consume items from a Kestrel cluster.
## Preliminaries
### Storm
-This tutorial uses examples from the [storm-kestrel](https://github.com/nathanmarz/storm-kestrel) project and the [storm-starter](http://github.com/apache/storm/blob/master/examples/storm-starter) project. It's recommended that you clone those projects and follow along with the examples. Read [Setting up development environment](https://github.com/apache/storm/wiki/Setting-up-development-environment) and [Creating a new Storm project](https://github.com/apache/storm/wiki/Creating-a-new-Storm-project) to get your machine set up.
+This tutorial uses examples from the [storm-kestrel](https://github.com/nathanmarz/storm-kestrel) project and the [storm-starter](http://github.com/apache/storm/blob/{{page.version}}/examples/storm-starter) project. It's recommended that you clone those projects and follow along with the examples. Read [Setting up development environment](https://github.com/apache/storm/wiki/Setting-up-development-environment) and [Creating a new Storm project](https://github.com/apache/storm/wiki/Creating-a-new-Storm-project) to get your machine set up.
### Kestrel
It assumes you are able to run locally a Kestrel server as described [here](https://github.com/nathanmarz/storm-kestrel).
@@ -197,4 +198,4 @@ Than, wait about 5 seconds in order to a
Now execute the program to add items to the queue and launch the Storm topology. The order in which you launch the programs is of no importance.
-If you run the topology with TOPOLOGY_DEBUG you should see tuples being emitted in the topology.
\ No newline at end of file
+If you run the topology with TOPOLOGY_DEBUG you should see tuples being emitted in the topology.
Modified: storm/branches/bobby-versioned-site/releases/0.10.0/Lifecycle-of-a-topology.md
URL: http://svn.apache.org/viewvc/storm/branches/bobby-versioned-site/releases/0.10.0/Lifecycle-of-a-topology.md?rev=1735360&r1=1735359&r2=1735360&view=diff
==============================================================================
--- storm/branches/bobby-versioned-site/releases/0.10.0/Lifecycle-of-a-topology.md (original)
+++ storm/branches/bobby-versioned-site/releases/0.10.0/Lifecycle-of-a-topology.md Thu Mar 17 02:10:04 2016
@@ -2,6 +2,7 @@
title: Lifecycle of a Storm Topology
layout: documentation
documentation: true
+version: v0.10.0
---
(**NOTE**: this page is based on the 0.7.1 code; many things have changed since then, including a split between tasks and executors, and a reorganization of the code under `storm-core/src` rather than `src/`.)
@@ -79,4 +80,4 @@ First a couple of important notes about
- This causes the topology to be deactivated for the wait time before its actually shut down. This gives the topology a chance to finish processing what it's currently processing before shutting down the workers
- Changing the status during the kill transition ensures that the kill protocol is fault-tolerant to Nimbus crashing. On startup, if the status of the topology is "killed", Nimbus schedules the remove event to run "wait time seconds" in the future [code](https://github.com/apache/storm/blob/0.7.1/src/clj/backtype/storm/daemon/nimbus.clj#L111)
- Removing a topology cleans out the assignment and static information from ZK [code](https://github.com/apache/storm/blob/0.7.1/src/clj/backtype/storm/daemon/nimbus.clj#L116)
-- A separate cleanup thread runs the `do-cleanup` function which will clean up the heartbeat dir and the jars/configs stored locally. [code](https://github.com/apache/storm/blob/0.7.1/src/clj/backtype/storm/daemon/nimbus.clj#L577)
\ No newline at end of file
+- A separate cleanup thread runs the `do-cleanup` function which will clean up the heartbeat dir and the jars/configs stored locally. [code](https://github.com/apache/storm/blob/0.7.1/src/clj/backtype/storm/daemon/nimbus.clj#L577)
Modified: storm/branches/bobby-versioned-site/releases/0.10.0/Local-mode.md
URL: http://svn.apache.org/viewvc/storm/branches/bobby-versioned-site/releases/0.10.0/Local-mode.md?rev=1735360&r1=1735359&r2=1735360&view=diff
==============================================================================
--- storm/branches/bobby-versioned-site/releases/0.10.0/Local-mode.md (original)
+++ storm/branches/bobby-versioned-site/releases/0.10.0/Local-mode.md Thu Mar 17 02:10:04 2016
@@ -2,6 +2,7 @@
title: Local Mode
layout: documentation
documentation: true
+version: v0.10.0
---
Local mode simulates a Storm cluster in process and is useful for developing and testing topologies. Running topologies in local mode is similar to running topologies [on a cluster](Running-topologies-on-a-production-cluster.html).
@@ -13,7 +14,7 @@ import backtype.storm.LocalCluster;
LocalCluster cluster = new LocalCluster();
```
-You can then submit topologies using the `submitTopology` method on the `LocalCluster` object. Just like the corresponding method on [StormSubmitter](/javadoc/apidocs/backtype/storm/StormSubmitter.html), `submitTopology` takes a name, a topology configuration, and the topology object. You can then kill a topology using the `killTopology` method which takes the topology name as an argument.
+You can then submit topologies using the `submitTopology` method on the `LocalCluster` object. Just like the corresponding method on [StormSubmitter](javadocs/backtype/storm/StormSubmitter.html), `submitTopology` takes a name, a topology configuration, and the topology object. You can then kill a topology using the `killTopology` method which takes the topology name as an argument.
To shutdown a local cluster, simple call:
@@ -23,7 +24,7 @@ cluster.shutdown();
### Common configurations for local mode
-You can see a full list of configurations [here](/javadoc/apidocs/backtype/storm/Config.html).
+You can see a full list of configurations [here](javadocs/backtype/storm/Config.html).
1. **Config.TOPOLOGY_MAX_TASK_PARALLELISM**: This config puts a ceiling on the number of threads spawned for a single component. Oftentimes production topologies have a lot of parallelism (hundreds of threads) which places unreasonable load when trying to test the topology in local mode. This config lets you easy control that parallelism.
-2. **Config.TOPOLOGY_DEBUG**: When this is set to true, Storm will log a message every time a tuple is emitted from any spout or bolt. This is extremely useful for debugging.
\ No newline at end of file
+2. **Config.TOPOLOGY_DEBUG**: When this is set to true, Storm will log a message every time a tuple is emitted from any spout or bolt. This is extremely useful for debugging.
Modified: storm/branches/bobby-versioned-site/releases/0.10.0/Maven.md
URL: http://svn.apache.org/viewvc/storm/branches/bobby-versioned-site/releases/0.10.0/Maven.md?rev=1735360&r1=1735359&r2=1735360&view=diff
==============================================================================
--- storm/branches/bobby-versioned-site/releases/0.10.0/Maven.md (original)
+++ storm/branches/bobby-versioned-site/releases/0.10.0/Maven.md Thu Mar 17 02:10:04 2016
@@ -1,5 +1,7 @@
---
layout: documentation
+version: v0.10.0
+raw-version: 0.10.0
---
To develop topologies, you'll need the Storm jars on your classpath. You should either include the unpacked jars in the classpath for your project or use Maven to include Storm as a development dependency. Storm is hosted on Maven Central. To include Storm in your project as a development dependency, add the following to your pom.xml:
@@ -8,13 +10,13 @@ To develop topologies, you'll need the S
<dependency>
<groupId>org.apache.storm</groupId>
<artifactId>storm-core</artifactId>
- <version>0.9.3</version>
+ <version>{{page.raw-version}}</version>
<scope>provided</scope>
</dependency>
```
-[Here's an example](https://github.com/apache/storm/blob/master/examples/storm-starter/pom.xml) of a pom.xml for a Storm project.
+[Here's an example](https://github.com/apache/storm/blob/{{page.version}}/examples/storm-starter/pom.xml) of a pom.xml for a Storm project.
### Developing Storm
-Please refer to [DEVELOPER.md](https://github.com/apache/storm/blob/master/DEVELOPER.md) for more details.
\ No newline at end of file
+Please refer to [DEVELOPER.md](https://github.com/apache/storm/blob/{{page.version}}/DEVELOPER.md) for more details.
Modified: storm/branches/bobby-versioned-site/releases/0.10.0/Message-passing-implementation.md
URL: http://svn.apache.org/viewvc/storm/branches/bobby-versioned-site/releases/0.10.0/Message-passing-implementation.md?rev=1735360&r1=1735359&r2=1735360&view=diff
==============================================================================
--- storm/branches/bobby-versioned-site/releases/0.10.0/Message-passing-implementation.md (original)
+++ storm/branches/bobby-versioned-site/releases/0.10.0/Message-passing-implementation.md Thu Mar 17 02:10:04 2016
@@ -2,6 +2,7 @@
title: Message Passing Implementation
layout: documentation
documentation: true
+version: v0.10.0
---
(Note: this walkthrough is out of date as of 0.8.0. 0.8.0 revamped the message passing infrastructure to be based on the Disruptor)
Modified: storm/branches/bobby-versioned-site/releases/0.10.0/Metrics.md
URL: http://svn.apache.org/viewvc/storm/branches/bobby-versioned-site/releases/0.10.0/Metrics.md?rev=1735360&r1=1735359&r2=1735360&view=diff
==============================================================================
--- storm/branches/bobby-versioned-site/releases/0.10.0/Metrics.md (original)
+++ storm/branches/bobby-versioned-site/releases/0.10.0/Metrics.md Thu Mar 17 02:10:04 2016
@@ -2,6 +2,7 @@
title: Storm Metrics
layout: documentation
documentation: true
+version: v0.10.0
---
Storm exposes a metrics interface to report summary statistics across the full topology.
It's used internally to track the numbers you see in the Nimbus UI console: counts of executes and acks; average process latency per bolt; worker heap usage; and so forth.
@@ -12,13 +13,13 @@ Metrics have to implement just one metho
Storm gives you these metric types:
-* [AssignableMetric](https://github.com/apache/storm/blob/master/storm-core/src/jvm/backtype/storm/metric/api/AssignableMetric.java) -- set the metric to the explicit value you supply. Useful if it's an external value or in the case that you are already calculating the summary statistic yourself.
-* [CombinedMetric](https://github.com/apache/storm/blob/master/storm-core/src/jvm/backtype/storm/metric/api/CombinedMetric.java) -- generic interface for metrics that can be updated associatively.
-* [CountMetric](https://github.com/apache/storm/blob/master/storm-core/src/jvm/backtype/storm/metric/api/CountMetric.java) -- a running total of the supplied values. Call `incr()` to increment by one, `incrBy(n)` to add/subtract the given number.
- - [MultiCountMetric](https://github.com/apache/storm/blob/master/storm-core/src/jvm/backtype/storm/metric/api/MultiCountMetric.java) -- a hashmap of count metrics.
-* [ReducedMetric](https://github.com/apache/storm/blob/master/storm-core/src/jvm/backtype/storm/metric/api/ReducedMetric.java)
- - [MeanReducer](https://github.com/apache/storm/blob/master/storm-core/src/jvm/backtype/storm/metric/api/MeanReducer.java) -- track a running average of values given to its `reduce()` method. (It accepts `Double`, `Integer` or `Long` values, and maintains the internal average as a `Double`.) Despite his reputation, the MeanReducer is actually a pretty nice guy in person.
- - [MultiReducedMetric](https://github.com/apache/storm/blob/master/storm-core/src/jvm/backtype/storm/metric/api/MultiReducedMetric.java) -- a hashmap of reduced metrics.
+* [AssignableMetric](https://github.com/apache/storm/blob/{{page.version}}/storm-core/src/jvm/backtype/storm/metric/api/AssignableMetric.java) -- set the metric to the explicit value you supply. Useful if it's an external value or in the case that you are already calculating the summary statistic yourself.
+* [CombinedMetric](https://github.com/apache/storm/blob/{{page.version}}/storm-core/src/jvm/backtype/storm/metric/api/CombinedMetric.java) -- generic interface for metrics that can be updated associatively.
+* [CountMetric](https://github.com/apache/storm/blob/{{page.version}}/storm-core/src/jvm/backtype/storm/metric/api/CountMetric.java) -- a running total of the supplied values. Call `incr()` to increment by one, `incrBy(n)` to add/subtract the given number.
+ - [MultiCountMetric](https://github.com/apache/storm/blob/{{page.version}}/storm-core/src/jvm/backtype/storm/metric/api/MultiCountMetric.java) -- a hashmap of count metrics.
+* [ReducedMetric](https://github.com/apache/storm/blob/{{page.version}}/storm-core/src/jvm/backtype/storm/metric/api/ReducedMetric.java)
+ - [MeanReducer](https://github.com/apache/storm/blob/{{page.version}}/storm-core/src/jvm/backtype/storm/metric/api/MeanReducer.java) -- track a running average of values given to its `reduce()` method. (It accepts `Double`, `Integer` or `Long` values, and maintains the internal average as a `Double`.) Despite his reputation, the MeanReducer is actually a pretty nice guy in person.
+ - [MultiReducedMetric](https://github.com/apache/storm/blob/{{page.version}}/storm-core/src/jvm/backtype/storm/metric/api/MultiReducedMetric.java) -- a hashmap of reduced metrics.
### Metric Consumer
@@ -30,7 +31,7 @@ Storm gives you these metric types:
### Builtin Metrics
-The [builtin metrics](https://github.com/apache/storm/blob/46c3ba7/storm-core/src/clj/backtype/storm/daemon/builtin_metrics.clj) instrument Storm itself.
+The [builtin metrics](https://github.com/apache/storm/blob/{{page.version}}/storm-core/src/clj/backtype/storm/daemon/builtin_metrics.clj) instrument Storm itself.
-[builtin_metrics.clj](https://github.com/apache/storm/blob/46c3ba7/storm-core/src/clj/backtype/storm/daemon/builtin_metrics.clj) sets up data structures for the built-in metrics, and facade methods that the other framework components can use to update them. The metrics themselves are calculated in the calling code -- see for example [`ack-spout-msg`](https://github.com/apache/storm/blob/46c3ba7/storm-core/src/clj/backtype/storm/daemon/executor.clj#358) in `clj/b/s/daemon/daemon/executor.clj`
+[builtin_metrics.clj](https://github.com/apache/storm/blob/{{page.version}}/storm-core/src/clj/backtype/storm/daemon/builtin_metrics.clj) sets up data structures for the built-in metrics, and facade methods that the other framework components can use to update them. The metrics themselves are calculated in the calling code -- see for example [`ack-spout-msg`](https://github.com/apache/storm/blob/{{page.version}}/storm-core/src/clj/backtype/storm/daemon/executor.clj#358) in `clj/b/s/daemon/daemon/executor.clj`
Modified: storm/branches/bobby-versioned-site/releases/0.10.0/Multilang-protocol.md
URL: http://svn.apache.org/viewvc/storm/branches/bobby-versioned-site/releases/0.10.0/Multilang-protocol.md?rev=1735360&r1=1735359&r2=1735360&view=diff
==============================================================================
--- storm/branches/bobby-versioned-site/releases/0.10.0/Multilang-protocol.md (original)
+++ storm/branches/bobby-versioned-site/releases/0.10.0/Multilang-protocol.md Thu Mar 17 02:10:04 2016
@@ -2,6 +2,7 @@
title: Multi-Lang Protocol
layout: documentation
documentation: true
+version: v0.10.0
---
This page explains the multilang protocol as of Storm 0.7.1. Versions prior to 0.7.1 used a somewhat different protocol, documented [here](Storm-multi-language-protocol-(versions-0.7.0-and-below\).html).
Modified: storm/branches/bobby-versioned-site/releases/0.10.0/Rationale.md
URL: http://svn.apache.org/viewvc/storm/branches/bobby-versioned-site/releases/0.10.0/Rationale.md?rev=1735360&r1=1735359&r2=1735360&view=diff
==============================================================================
--- storm/branches/bobby-versioned-site/releases/0.10.0/Rationale.md (original)
+++ storm/branches/bobby-versioned-site/releases/0.10.0/Rationale.md Thu Mar 17 02:10:04 2016
@@ -2,6 +2,7 @@
title: Rationale
layout: documentation
documentation: true
+version: v0.10.0
---
The past decade has seen a revolution in data processing. MapReduce, Hadoop, and related technologies have made it possible to store and process data at scales previously unthinkable. Unfortunately, these data processing technologies are not realtime systems, nor are they meant to be. There's no hack that will turn Hadoop into a realtime system; realtime data processing has a fundamentally different set of requirements than batch processing.