You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@storm.apache.org by pt...@apache.org on 2014/02/04 17:51:32 UTC
svn commit: r1564375 - in /incubator/storm/site: ./ _layouts/ documentation/
Author: ptgoetz
Date: Tue Feb 4 16:51:30 2014
New Revision: 1564375
URL: http://svn.apache.org/r1564375
Log:
first pass at integrating github wiki content
Added:
incubator/storm/site/_layouts/documentation.html
Modified:
incubator/storm/site/_layouts/default.html
incubator/storm/site/community.html
incubator/storm/site/documentation/Acking-framework-implementation.md
incubator/storm/site/documentation/Clojure-DSL.md
incubator/storm/site/documentation/Command-line-client.md
incubator/storm/site/documentation/Common-patterns.md
incubator/storm/site/documentation/Concepts.md
incubator/storm/site/documentation/Configuration.md
incubator/storm/site/documentation/Contributing-to-Storm.md
incubator/storm/site/documentation/Creating-a-new-Storm-project.md
incubator/storm/site/documentation/DSLs-and-multilang-adapters.md
incubator/storm/site/documentation/Defining-a-non-jvm-language-dsl-for-storm.md
incubator/storm/site/documentation/Distributed-RPC.md
incubator/storm/site/documentation/Documentation.md
incubator/storm/site/documentation/FAQ.md
incubator/storm/site/documentation/Fault-tolerance.md
incubator/storm/site/documentation/Guaranteeing-message-processing.md
incubator/storm/site/documentation/Home.md
incubator/storm/site/documentation/Hooks.md
incubator/storm/site/documentation/Implementation-docs.md
incubator/storm/site/documentation/Installing-native-dependencies.md
incubator/storm/site/documentation/Kestrel-and-Storm.md
incubator/storm/site/documentation/Lifecycle-of-a-topology.md
incubator/storm/site/documentation/Local-mode.md
incubator/storm/site/documentation/Maven.md
incubator/storm/site/documentation/Message-passing-implementation.md
incubator/storm/site/documentation/Metrics.md
incubator/storm/site/documentation/Multilang-protocol.md
incubator/storm/site/documentation/Powered-By.md
incubator/storm/site/documentation/Project-ideas.md
incubator/storm/site/documentation/Rationale.md
incubator/storm/site/documentation/Running-topologies-on-a-production-cluster.md
incubator/storm/site/documentation/Serialization-(prior-to-0.6.0).md
incubator/storm/site/documentation/Serialization.md
incubator/storm/site/documentation/Serializers.md
incubator/storm/site/documentation/Setting-up-a-Storm-cluster.md
incubator/storm/site/documentation/Setting-up-development-environment.md
incubator/storm/site/documentation/Spout-implementations.md
incubator/storm/site/documentation/Storm-multi-language-protocol-(versions-0.7.0-and-below).md
incubator/storm/site/documentation/Structure-of-the-codebase.md
incubator/storm/site/documentation/Support-for-non-java-languages.md
incubator/storm/site/documentation/Transactional-topologies.md
incubator/storm/site/documentation/Trident-API-Overview.md
incubator/storm/site/documentation/Trident-spouts.md
incubator/storm/site/documentation/Trident-state.md
incubator/storm/site/documentation/Trident-tutorial.md
incubator/storm/site/documentation/Troubleshooting.md
incubator/storm/site/documentation/Tutorial.md
incubator/storm/site/documentation/Understanding-the-parallelism-of-a-Storm-topology.md
incubator/storm/site/documentation/Using-non-JVM-languages-with-Storm.md
incubator/storm/site/index.html
Modified: incubator/storm/site/_layouts/default.html
URL: http://svn.apache.org/viewvc/incubator/storm/site/_layouts/default.html?rev=1564375&r1=1564374&r2=1564375&view=diff
==============================================================================
--- incubator/storm/site/_layouts/default.html (original)
+++ incubator/storm/site/_layouts/default.html Tue Feb 4 16:51:30 2014
@@ -37,7 +37,7 @@
<li><a href="/blog.html">blog</a></li>
-->
-<li><a href="/documentation.html">documentation</a></li>
+<li><a href="/documentation/Home.html">documentation</a></li>
<li><a href="/blog.html">blog</a></li>
<li><a href="/downloads.html">downloads</a></li>
<li><a href="/community.html">community</a></li>
Added: incubator/storm/site/_layouts/documentation.html
URL: http://svn.apache.org/viewvc/incubator/storm/site/_layouts/documentation.html?rev=1564375&view=auto
==============================================================================
--- incubator/storm/site/_layouts/documentation.html (added)
+++ incubator/storm/site/_layouts/documentation.html Tue Feb 4 16:51:30 2014
@@ -0,0 +1,24 @@
+---
+layout: default
+title: Storm Documentation
+items:
+ -
+ - "/documentation/FAQ.html"
+ - "FAQ"
+ -
+ - "/about/simple-api.html"
+ - "Simple API"
+---
+
+<div id="content">
+<div id="sidebar">
+ <ul>
+ {% for post in page.items %}
+<li><a class="{% if page.url == post[0] then %}current{% else %}{% endif %}" href="{{ post[0] }}">{{ post[1] }}</a></li>
+ {% endfor %}
+ </ul>
+</div>
+<div id="aboutcontent">
+{{ content }}
+</div>
+</div>
\ No newline at end of file
Modified: incubator/storm/site/community.html
URL: http://svn.apache.org/viewvc/incubator/storm/site/community.html?rev=1564375&r1=1564374&r2=1564375&view=diff
==============================================================================
--- incubator/storm/site/community.html (original)
+++ incubator/storm/site/community.html Tue Feb 4 16:51:30 2014
@@ -19,7 +19,7 @@ title: Storm community
<li><a href="http://www.meetup.com/New-York-City-Storm-User-Group/">New York City Storm User Group</a></li>
</ul>
<h2>Contributing to Storm</h2>
- <p>Contributions to the Storm codebase are welcome and encouraged. Information about how to contribute to the Storm codebase can be found <a href="https://github.com/nathanmarz/storm/wiki/Contributing-to-Storm">here</a>.</p>
- <p>This website is hosted <a href="https://github.com/nathanmarz/storm-website">on Github</a>.</p>
+ <p>Contributions to the Storm codebase are welcome and encouraged. Information about how to contribute to the Storm codebase can be found <a href="documentation/Contributing-to-Storm.html">here</a>.</p>
+
</div>
Modified: incubator/storm/site/documentation/Acking-framework-implementation.md
URL: http://svn.apache.org/viewvc/incubator/storm/site/documentation/Acking-framework-implementation.md?rev=1564375&r1=1564374&r2=1564375&view=diff
==============================================================================
--- incubator/storm/site/documentation/Acking-framework-implementation.md (original)
+++ incubator/storm/site/documentation/Acking-framework-implementation.md Tue Feb 4 16:51:30 2014
@@ -1,3 +1,6 @@
+---
+layout: documentation
+---
[Storm's acker](https://github.com/nathanmarz/storm/blob/46c3ba7/storm-core/src/clj/backtype/storm/daemon/acker.clj#L28) tracks completion of each tupletree with a checksum hash: each time a tuple is sent, its value is XORed into the checksum, and each time a tuple is acked its value is XORed in again. If all tuples have been successfully acked, the checksum will be zero (the odds that the checksum will be zero otherwise are vanishingly small).
You can read a bit more about the [[reliability mechanism|Guaranteeing-message-processing#what-is-storms-reliability-api]] elsewhere on the wiki -- this explains the internal details.
Modified: incubator/storm/site/documentation/Clojure-DSL.md
URL: http://svn.apache.org/viewvc/incubator/storm/site/documentation/Clojure-DSL.md?rev=1564375&r1=1564374&r2=1564375&view=diff
==============================================================================
--- incubator/storm/site/documentation/Clojure-DSL.md (original)
+++ incubator/storm/site/documentation/Clojure-DSL.md Tue Feb 4 16:51:30 2014
@@ -1,3 +1,6 @@
+---
+layout: documentation
+---
Storm comes with a Clojure DSL for defining spouts, bolts, and topologies. The Clojure DSL has access to everything the Java API exposes, so if you're a Clojure user you can code Storm topologies without touching Java at all. The Clojure DSL is defined in the source in the [backtype.storm.clojure](https://github.com/nathanmarz/storm/blob/0.5.3/src/clj/backtype/storm/clojure.clj) namespace.
This page outlines all the pieces of the Clojure DSL, including:
Modified: incubator/storm/site/documentation/Command-line-client.md
URL: http://svn.apache.org/viewvc/incubator/storm/site/documentation/Command-line-client.md?rev=1564375&r1=1564374&r2=1564375&view=diff
==============================================================================
--- incubator/storm/site/documentation/Command-line-client.md (original)
+++ incubator/storm/site/documentation/Command-line-client.md Tue Feb 4 16:51:30 2014
@@ -1,3 +1,6 @@
+---
+layout: documentation
+---
This page describes all the commands that are possible with the "storm" command line client. To learn how to set up your "storm" client to talk to a remote cluster, follow the instructions in [[Setting up development environment]].
These commands are:
Modified: incubator/storm/site/documentation/Common-patterns.md
URL: http://svn.apache.org/viewvc/incubator/storm/site/documentation/Common-patterns.md?rev=1564375&r1=1564374&r2=1564375&view=diff
==============================================================================
--- incubator/storm/site/documentation/Common-patterns.md (original)
+++ incubator/storm/site/documentation/Common-patterns.md Tue Feb 4 16:51:30 2014
@@ -1,3 +1,7 @@
+---
+layout: documentation
+---
+
This page lists a variety of common patterns in Storm topologies.
1. Streaming joins
Modified: incubator/storm/site/documentation/Concepts.md
URL: http://svn.apache.org/viewvc/incubator/storm/site/documentation/Concepts.md?rev=1564375&r1=1564374&r2=1564375&view=diff
==============================================================================
--- incubator/storm/site/documentation/Concepts.md (original)
+++ incubator/storm/site/documentation/Concepts.md Tue Feb 4 16:51:30 2014
@@ -1,3 +1,7 @@
+---
+layout: documentation
+---
+
This page lists the main concepts of Storm and links to resources where you can find more information. The concepts discussed are:
1. Topologies
@@ -16,8 +20,8 @@ The logic for a realtime application is
**Resources:**
* [TopologyBuilder](http://nathanmarz.github.com/storm/doc/index.html): use this class to construct topologies in Java
-* [[Running topologies on a production cluster]]
-* [[Local mode]]: Read this to learn how to develop and test topologies in local mode.
+* [Running topologies on a production cluster](Running-topologies-on-a-production-cluster.html)
+* [Local mode](Local-mode.html): Read this to learn how to develop and test topologies in local mode.
### Streams
@@ -30,7 +34,7 @@ Every stream is given an id when declare
* [Tuple](http://nathanmarz.github.com/storm/doc/backtype/storm/tuple/Tuple.html): streams are composed of tuples
* [OutputFieldsDeclarer](http://nathanmarz.github.com/storm/doc/backtype/storm/topology/OutputFieldsDeclarer.html): used to declare streams and their schemas
-* [[Serialization]]: Information about Storm's dynamic typing of tuples and declaring custom serializations
+* [Serialization](Serialization.html): Information about Storm's dynamic typing of tuples and declaring custom serializations
* [ISerialization](http://nathanmarz.github.com/storm/doc/backtype/storm/serialization/ISerialization.html): custom serializers must implement this interface
* [CONFIG.TOPOLOGY_SERIALIZATIONS](http://nathanmarz.github.com/storm/doc/backtype/storm/Config.html#TOPOLOGY_SERIALIZATIONS): custom serializers can be registered using this configuration
@@ -47,7 +51,7 @@ The other main methods on spouts are `ac
**Resources:**
* [IRichSpout](http://nathanmarz.github.com/storm/doc/backtype/storm/topology/IRichSpout.html): this is the interface that spouts must implement.
-* [[Guaranteeing message processing]]
+* [Guaranteeing message processing](Guaranteeing-message-processing.html)
### Bolts
@@ -68,7 +72,7 @@ Its perfectly fine to launch new threads
* [IRichBolt](http://nathanmarz.github.com/storm/doc/backtype/storm/topology/IRichBolt.html): this is general interface for bolts.
* [IBasicBolt](http://nathanmarz.github.com/storm/doc/backtype/storm/topology/IBasicBolt.html): this is a convenience interface for defining bolts that do filtering or simple functions.
* [OutputCollector](http://nathanmarz.github.com/storm/doc/backtype/storm/task/OutputCollector.html): bolts emit tuples to their output streams using an instance of this class
-* [[Guaranteeing message processing]]
+* [Guaranteeing message processing](Guaranteeing-message-processing.html)
### Stream groupings
@@ -96,7 +100,7 @@ Storm guarantees that every spout tuple
To take advantage of Storm's reliability capabilities, you must tell Storm when new edges in a tuple tree are being created and tell Storm whenever you've finished processing an individual tuple. These are done using the [OutputCollector](http://nathanmarz.github.com/storm/doc/backtype/storm/task/OutputCollector.html) object that bolts use to emit tuples. Anchoring is done in the `emit` method, and you declare that you're finished with a tuple using the `ack` method.
-This is all explained in much more detail on [[Guaranteeing message processing]].
+This is all explained in much more detail in [Guaranteeing message processing](Guaranteeing-message-processing.html).
### Tasks
Modified: incubator/storm/site/documentation/Configuration.md
URL: http://svn.apache.org/viewvc/incubator/storm/site/documentation/Configuration.md?rev=1564375&r1=1564374&r2=1564375&view=diff
==============================================================================
--- incubator/storm/site/documentation/Configuration.md (original)
+++ incubator/storm/site/documentation/Configuration.md Tue Feb 4 16:51:30 2014
@@ -1,3 +1,6 @@
+---
+layout: documentation
+---
Storm has a variety of configurations for tweaking the behavior of nimbus, supervisors, and running topologies. Some configurations are system configurations and cannot be modified on a topology by topology basis, whereas other configurations can be modified per topology.
Every configuration has a default value defined in [defaults.yaml](https://github.com/nathanmarz/storm/blob/master/conf/defaults.yaml) in the Storm codebase. You can override these configurations by defining a storm.yaml in the classpath of Nimbus and the supervisors. Finally, you can define a topology-specific configuration that you submit along with your topology when using [StormSubmitter](http://nathanmarz.github.com/storm/doc/backtype/storm/StormSubmitter.html). However, the topology-specific configuration can only override configs prefixed with "TOPOLOGY".
@@ -7,7 +10,7 @@ Storm 0.7.0 and onwards lets you overrid
1. "topology.debug"
2. "topology.max.spout.pending"
3. "topology.max.task.parallelism"
-4. "topology.kryo.register": This works a little bit differently than the other ones, since the serializations will be available to all components in the topology. More details on [[Serialization]].
+4. "topology.kryo.register": This works a little bit differently than the other ones, since the serializations will be available to all components in the topology. More details on [[Serialization]().
The Java API lets you specify component specific configurations in two ways:
@@ -21,6 +24,6 @@ The preference order for configuration v
* [Config](http://nathanmarz.github.com/storm/doc/backtype/storm/Config.html): a listing of all configurations as well as a helper class for creating topology specific configurations
* [defaults.yaml](https://github.com/nathanmarz/storm/blob/master/conf/defaults.yaml): the default values for all configurations
-* [[Setting up a Storm cluster]]: explains how to create and configure a Storm cluster
-* [[Running topologies on a production cluster]]: lists useful configurations when running topologies on a cluster
-* [[Local mode]]: lists useful configurations when using local mode
\ No newline at end of file
+* [Setting up a Storm cluster](Setting-up-a-Storm-cluster.html): explains how to create and configure a Storm cluster
+* [Running topologies on a production cluster](Running-topologies-on-a-production-cluster.html): lists useful configurations when running topologies on a cluster
+* [Local mode](Local-mode.html): lists useful configurations when using local mode
\ No newline at end of file
Modified: incubator/storm/site/documentation/Contributing-to-Storm.md
URL: http://svn.apache.org/viewvc/incubator/storm/site/documentation/Contributing-to-Storm.md?rev=1564375&r1=1564374&r2=1564375&view=diff
==============================================================================
--- incubator/storm/site/documentation/Contributing-to-Storm.md (original)
+++ incubator/storm/site/documentation/Contributing-to-Storm.md Tue Feb 4 16:51:30 2014
@@ -1,10 +1,14 @@
+---
+layout: documentation
+---
+
### Getting started with contributing
-Some of the issues on the [issue tracker](https://github.com/nathanmarz/storm/issues) are marked with the "Newbie" label. If you're interesting in contributing to Storm but don't know where to begin, these are good issues to start with. These issues are a great way to get your feet wet with learning the codebase because they require learning about only an isolated portion of the codebase and are a relatively small amount of work.
+Some of the issues on the [issue tracker](https://issues.apache.org/jira/browse/STORM) are marked with the "Newbie" label. If you're interesting in contributing to Storm but don't know where to begin, these are good issues to start with. These issues are a great way to get your feet wet with learning the codebase because they require learning about only an isolated portion of the codebase and are a relatively small amount of work.
### Learning the codebase
-The [[Implementation docs]] section of the wiki gives detailed walkthroughs of the codebase. Reading through these docs is highly recommended to understand the codebase.
+The [Implementation docs](Implementation-docs.html) section of the wiki gives detailed walkthroughs of the codebase. Reading through these docs is highly recommended to understand the codebase.
### Contribution process
@@ -12,7 +16,7 @@ Contributions to the Storm codebase shou
For small patches, feel free to submit pull requests directly for them. For larger contributions, please use the following process. The idea behind this process is to prevent any wasted work and catch design issues early on:
-1. Open an issue on the [issue tracker](https://github.com/nathanmarz/storm/issues) if one doesn't exist already
+1. Open an issue on the [issue tracker](https://issues.apache.org/jira/browse/STORM) if one doesn't exist already
2. Comment on the issue with your plan for implementing the issue. Explain what pieces of the codebase you're going to touch and how everything is going to fit together.
3. Storm committers will iterate with you on the design to make sure you're on the right track
4. Implement your issue, submit a pull request, and iterate from there.
@@ -25,19 +29,3 @@ Modules built on top of Storm (like spou
Documentation contributions are very welcome! The best way to send contributions is as emails through the mailing list.
-### Contributor agreement
-
-To contribute to Storm, you will need to sign a contributor agreement. This is a standard agreement used by most major open source projects to provide legal protection to the project and community. Basically you just agree to share ownership of the code with the project.
-
-You can download the contributor agreement [here](https://dl.dropbox.com/u/133901206/storm-apache-style-cla.txt). The easiest way to fill it out is with [HelloFax](https://www.hellofax.com/), where you can fill in the fields, sign it, and email it back all digitally. The whole process should only take a couple minutes. Email the filled and signed agreement to nathan.marz@gmail.com.
-
-Make sure you fill in all the fields. These are:
-
- 1. Full Name
- 2. Username(s)
- 3. Mailing address
- 4. Country
- 5. Telephone
- 6. Email
- 7. Signature
- 8. Date
Modified: incubator/storm/site/documentation/Creating-a-new-Storm-project.md
URL: http://svn.apache.org/viewvc/incubator/storm/site/documentation/Creating-a-new-Storm-project.md?rev=1564375&r1=1564374&r2=1564375&view=diff
==============================================================================
--- incubator/storm/site/documentation/Creating-a-new-Storm-project.md (original)
+++ incubator/storm/site/documentation/Creating-a-new-Storm-project.md Tue Feb 4 16:51:30 2014
@@ -1,3 +1,6 @@
+---
+layout: documentation
+---
This page outlines how to set up a Storm project for development. The steps are:
1. Add Storm jars to classpath
@@ -7,7 +10,7 @@ Follow along to see how to set up the [s
### Add Storm jars to classpath
-You'll need the Storm jars on your classpath to develop Storm topologies. Using [[Maven]] is highly recommended. [Here's an example](https://github.com/nathanmarz/storm-starter/blob/master/m2-pom.xml) of how to setup your pom.xml for a Storm project. If you don't want to use Maven, you can include the jars from the Storm release on your classpath.
+You'll need the Storm jars on your classpath to develop Storm topologies. Using [Maven](Maven.html) is highly recommended. [Here's an example](https://github.com/nathanmarz/storm-starter/blob/master/m2-pom.xml) of how to setup your pom.xml for a Storm project. If you don't want to use Maven, you can include the jars from the Storm release on your classpath.
[storm-starter](http://github.com/nathanmarz/storm-starter) uses [Leiningen](http://github.com/technomancy/leiningen) for build and dependency resolution. You can install leiningen by downloading [this script](https://raw.github.com/technomancy/leiningen/stable/bin/lein), placing it on your path, and making it executable. To retrieve the dependencies for Storm, simply run `lein deps` in the project root.
Modified: incubator/storm/site/documentation/DSLs-and-multilang-adapters.md
URL: http://svn.apache.org/viewvc/incubator/storm/site/documentation/DSLs-and-multilang-adapters.md?rev=1564375&r1=1564374&r2=1564375&view=diff
==============================================================================
--- incubator/storm/site/documentation/DSLs-and-multilang-adapters.md (original)
+++ incubator/storm/site/documentation/DSLs-and-multilang-adapters.md Tue Feb 4 16:51:30 2014
@@ -1,6 +1,9 @@
+---
+layout: documentation
+---
* [Scala DSL](https://github.com/velvia/ScalaStorm)
* [JRuby DSL](https://github.com/colinsurprenant/redstorm)
-* [[Clojure DSL]]
+* [Clojure DSL](Clojure-DSL.html)
* [Storm/Esper integration](https://github.com/tomdz/storm-esper): Streaming SQL on top of Storm
* [io-storm](https://github.com/gphat/io-storm): Perl multilang adapter
* [storm-php](https://github.com/lazyshot/storm-php): PHP multilang adapter
\ No newline at end of file
Modified: incubator/storm/site/documentation/Defining-a-non-jvm-language-dsl-for-storm.md
URL: http://svn.apache.org/viewvc/incubator/storm/site/documentation/Defining-a-non-jvm-language-dsl-for-storm.md?rev=1564375&r1=1564374&r2=1564375&view=diff
==============================================================================
--- incubator/storm/site/documentation/Defining-a-non-jvm-language-dsl-for-storm.md (original)
+++ incubator/storm/site/documentation/Defining-a-non-jvm-language-dsl-for-storm.md Tue Feb 4 16:51:30 2014
@@ -1,3 +1,6 @@
+---
+layout: documentation
+---
The right place to start to learn how to make a non-JVM DSL for Storm is [storm-core/src/storm.thrift](https://github.com/nathanmarz/storm/blob/master/storm-core/src/storm.thrift). Since Storm topologies are just Thrift structures, and Nimbus is a Thrift daemon, you can create and submit topologies in any language.
When you create the Thrift structs for spouts and bolts, the code for the spout or bolt is specified in the ComponentObject struct:
Modified: incubator/storm/site/documentation/Distributed-RPC.md
URL: http://svn.apache.org/viewvc/incubator/storm/site/documentation/Distributed-RPC.md?rev=1564375&r1=1564374&r2=1564375&view=diff
==============================================================================
--- incubator/storm/site/documentation/Distributed-RPC.md (original)
+++ incubator/storm/site/documentation/Distributed-RPC.md Tue Feb 4 16:51:30 2014
@@ -1,3 +1,6 @@
+---
+layout: documentation
+---
The idea behind distributed RPC (DRPC) is to parallelize the computation of really intense functions on the fly using Storm. The Storm topology takes in as input a stream of function arguments, and it emits an output stream of the results for each of those function calls.
DRPC is not so much a feature of Storm as it is a pattern expressed from Storm's primitives of streams, spouts, bolts, and topologies. DRPC could have been packaged as a separate library from Storm, but it's so useful that it's bundled with Storm.
Modified: incubator/storm/site/documentation/Documentation.md
URL: http://svn.apache.org/viewvc/incubator/storm/site/documentation/Documentation.md?rev=1564375&r1=1564374&r2=1564375&view=diff
==============================================================================
--- incubator/storm/site/documentation/Documentation.md (original)
+++ incubator/storm/site/documentation/Documentation.md Tue Feb 4 16:51:30 2014
@@ -1,47 +1,50 @@
+---
+layout: documentation
+---
### Basics of Storm
* [Javadoc](http://nathanmarz.github.com/storm)
-* [[Concepts]]
-* [[Configuration]]
-* [[Guaranteeing message processing]]
-* [[Fault-tolerance]]
-* [[Command line client]]
-* [[Understanding the parallelism of a Storm topology]]
-* [[FAQ]]
+* [Concepts](Concepts.html)
+* [Configuration](Configuration.html)
+* [Guaranteeing message processing](Guaranteeing-message-processing.html)
+* [Fault-tolerance](Fault-tolerance.html)
+* [Command line client](Command-line-client.html)
+* [Understanding the parallelism of a Storm topology](Understanding-the-parallelism-of-a-storm-topology.html)
+* [FAQ](FAQ.html)
### Trident
Trident is an alternative interface to Storm. It provides exactly-once processing, "transactional" datastore persistence, and a set of common stream analytics operations.
-* [[Trident Tutorial]] -- basic concepts and walkthrough
-* [[Trident API Overview]] -- operations for transforming and orchestrating data
-* [[Trident State]] -- exactly-once processing and fast, persistent aggregation
-* [[Trident spouts]] -- transactional and non-transactional data intake
+* [Trident Tutorial](Trident-Tutorial.html) -- basic concepts and walkthrough
+* [Trident API Overview](Trident-API-Overview.html) -- operations for transforming and orchestrating data
+* [Trident State](Trident-State.html) -- exactly-once processing and fast, persistent aggregation
+* [Trident spouts](Trident-spouts.html) -- transactional and non-transactional data intake
### Setup and deploying
-* [[Setting up a Storm cluster]]
-* [[Local mode]]
-* [[Troubleshooting]]
-* [[Running topologies on a production cluster]]
-* [[Building Storm|Maven]] with Maven or Leiningen
+* [Setting up a Storm cluster](Setting-up-a-Storm-cluster.html)
+* [Local mode](Local-mode.html)
+* [Troubleshooting](Troubleshooting.html)
+* [Running topologies on a production cluster](Running-topologies-on-a-production-cluster.html)
+* [Building Storm](Maven.html) with Maven
### Intermediate
-* [[Serialization]]
-* [[Common patterns]]
-* [[Clojure DSL]]
-* [[Using non-JVM languages with Storm]]
-* [[Distributed RPC]]
-* [[Transactional topologies]]
-* [[Kestrel and Storm]]
-* [[Direct groupings]]
-* [[Hooks]]
-* [[Metrics]]
-* [[Lifecycle of a trident tuple]]
+* [Serialization](Serialization.html)
+* [Common patterns](Common-patterns.html)
+* [Clojure DSL](Clojure-DSL.html)
+* [Using non-JVM languages with Storm](Using-non-JVM-languages-with-Storm.html)
+* [Distributed RPC](Distributed-RPC.html)
+* [Transactional topologies](Transactional-topologies.html)
+* [Kestrel and Storm](Kestrel-and-Storm.html)
+* [Direct groupings](Direct-groupings.html)
+* [Hooks](Hooks.html)
+* [Metrics](Metrics.html)
+* [Lifecycle of a trident tuple]()
### Advanced
-* [[Defining a non-JVM language DSL for Storm]]
-* [[Multilang protocol]] (how to provide support for another language)
-* [[Implementation docs]]
\ No newline at end of file
+* [Defining a non-JVM language DSL for Storm](Defining-a-non-jvm-language-dsl-for-storm.html)
+* [Multilang protocol](Multilang-protocol.html) (how to provide support for another language)
+* [Implementation docs](Implementation-docs.html)
\ No newline at end of file
Modified: incubator/storm/site/documentation/FAQ.md
URL: http://svn.apache.org/viewvc/incubator/storm/site/documentation/FAQ.md?rev=1564375&r1=1564374&r2=1564375&view=diff
==============================================================================
--- incubator/storm/site/documentation/FAQ.md (original)
+++ incubator/storm/site/documentation/FAQ.md Tue Feb 4 16:51:30 2014
@@ -1,4 +1,6 @@
-This FAQ is being assembled OG-style, copy-pasting in answers to frequent questions on the mailing list.
+---
+layout: documentation
+---
## Best Practices
@@ -56,7 +58,7 @@ A Trident "each" operator returns a Stre
You can join streams with join, merge or multiReduce.
-At time of writing, you can't emit to multiple output streams from Trident -- see [issue #638](https://github.com/nathanmarz/storm/issues/638)
+At time of writing, you can't emit to multiple output streams from Trident -- see [STORM-68](https://issues.apache.org/jira/browse/STORM-68)
## Spouts
Modified: incubator/storm/site/documentation/Fault-tolerance.md
URL: http://svn.apache.org/viewvc/incubator/storm/site/documentation/Fault-tolerance.md?rev=1564375&r1=1564374&r2=1564375&view=diff
==============================================================================
--- incubator/storm/site/documentation/Fault-tolerance.md (original)
+++ incubator/storm/site/documentation/Fault-tolerance.md Tue Feb 4 16:51:30 2014
@@ -1,3 +1,6 @@
+---
+layout: documentation
+---
This page explains the design details of Storm that make it a fault-tolerant system.
## What happens when a worker dies?
Modified: incubator/storm/site/documentation/Guaranteeing-message-processing.md
URL: http://svn.apache.org/viewvc/incubator/storm/site/documentation/Guaranteeing-message-processing.md?rev=1564375&r1=1564374&r2=1564375&view=diff
==============================================================================
--- incubator/storm/site/documentation/Guaranteeing-message-processing.md (original)
+++ incubator/storm/site/documentation/Guaranteeing-message-processing.md Tue Feb 4 16:51:30 2014
@@ -1,3 +1,6 @@
+---
+layout: documentation
+---
Storm guarantees that each message coming off a spout will be fully processed. This page describes how Storm accomplishes this guarantee and what you have to do as a user to benefit from Storm's reliability capabilities.
### What does it mean for a message to be "fully processed"?
@@ -126,7 +129,7 @@ In contrast, bolts that do aggregations
### How do I make my applications work correctly given that tuples can be replayed?
-As always in software design, the answer is "it depends." Storm 0.7.0 introduced the "transactional topologies" feature, which enables you to get fully fault-tolerant exactly-once messaging semantics for most computations. Read more about transactional topologies [here](https://github.com/nathanmarz/storm/wiki/Transactional-topologies).
+As always in software design, the answer is "it depends." Storm 0.7.0 introduced the "transactional topologies" feature, which enables you to get fully fault-tolerant exactly-once messaging semantics for most computations. Read more about transactional topologies [here](Transactional-topologies.html).
### How does Storm implement reliability in an efficient way?
Modified: incubator/storm/site/documentation/Home.md
URL: http://svn.apache.org/viewvc/incubator/storm/site/documentation/Home.md?rev=1564375&r1=1564374&r2=1564375&view=diff
==============================================================================
--- incubator/storm/site/documentation/Home.md (original)
+++ incubator/storm/site/documentation/Home.md Tue Feb 4 16:51:30 2014
@@ -1,17 +1,20 @@
+---
+layout: documentation
+---
Storm is a distributed realtime computation system. Similar to how Hadoop provides a set of general primitives for doing batch processing, Storm provides a set of general primitives for doing realtime computation. Storm is simple, can be used with any programming language, [is used by many companies](https://github.com/nathanmarz/storm/wiki/Powered-By), and is a lot of fun to use!
### Read these first
-* [[Rationale]]
-* [[Tutorial]]
-* [[Setting up development environment]]
-* [[Creating a new Storm project]]
+* [Rationale](Rationale.html)
+* [Tutorial](Tutorial.html)
+* [Setting up development environment](Setting-up-development-environment.html)
+* [Creating a new Storm project](Creating-a-new-Storm-project.html)
### Documentation
-* [Manual](https://github.com/nathanmarz/storm/wiki/Documentation)
+* [Manual](Documentation.html)
* [Javadoc](http://nathanmarz.github.com/storm)
-* [[FAQ]]
+* [FAQ](FAQ.html)
### Getting help
@@ -50,15 +53,15 @@ You can also come to the #storm-user roo
* [storm-contrib](https://github.com/nathanmarz/storm-contrib)
* [storm-deploy](http://github.com/nathanmarz/storm-deploy): One click deploys for Storm clusters on AWS
-* [[Spout implementations]]
-* [[DSLs and multilang adapters]]
-* [[Serializers]]
+* [Spout implementations](Spout-implementations.html)
+* [DSLs and multilang adapters](DSLs-and-multilang-adapters.html)
+* [Serializers](Serializers.html)
### Contributing to Storm
-* [[Contributing to Storm]]
-* [[Project ideas]]
+* [Contributing to Storm](Contributing-to-Storm.html)
+* [Project ideas](Project-ideas.html)
### Powered by Storm
-[Companies and projects powered by Storm](https://github.com/nathanmarz/storm/wiki/Powered-By)
\ No newline at end of file
+[Companies and projects powered by Storm](Powered-By.html)
\ No newline at end of file
Modified: incubator/storm/site/documentation/Hooks.md
URL: http://svn.apache.org/viewvc/incubator/storm/site/documentation/Hooks.md?rev=1564375&r1=1564374&r2=1564375&view=diff
==============================================================================
--- incubator/storm/site/documentation/Hooks.md (original)
+++ incubator/storm/site/documentation/Hooks.md Tue Feb 4 16:51:30 2014
@@ -1,3 +1,6 @@
+---
+layout: documentation
+---
Storm provides hooks with which you can insert custom code to run on any number of events within Storm. You create a hook by extending the [BaseTaskHook](http://nathanmarz.github.com/storm/doc-0.7.1/backtype/storm/hooks/BaseTaskHook.html) class and overriding the appropriate method for the event you want to catch. There are two ways to register your hook:
1. In the open method of your spout or prepare method of your bolt using the [TopologyContext#addTaskHook](http://nathanmarz.github.com/storm/doc-0.7.1/backtype/storm/task/TopologyContext.html) method.
Modified: incubator/storm/site/documentation/Implementation-docs.md
URL: http://svn.apache.org/viewvc/incubator/storm/site/documentation/Implementation-docs.md?rev=1564375&r1=1564374&r2=1564375&view=diff
==============================================================================
--- incubator/storm/site/documentation/Implementation-docs.md (original)
+++ incubator/storm/site/documentation/Implementation-docs.md Tue Feb 4 16:51:30 2014
@@ -1,10 +1,13 @@
+---
+layout: documentation
+---
This section of the wiki is dedicated to explaining how Storm is implemented. You should have a good grasp of how to use Storm before reading these sections.
-- [[Structure of the codebase]]
-- [[Lifecycle of a topology]]
-- [[Message passing implementation]]
-- [[Acking framework implementation]]
-- [[Metrics]]
+- [Structure of the codebase](Structure-of-the-codebase.html)
+- [Lifecycle of a topology](Lifecycle-of-a-topology.html)
+- [Message passing implementation](Message-passing-implementation.html)
+- [Acking framework implementation](Acking-framework-implementation.html)
+- [Metrics](Metrics.html)
- How transactional topologies work
- subtopology for TransactionalSpout
- how state is stored in ZK
Modified: incubator/storm/site/documentation/Installing-native-dependencies.md
URL: http://svn.apache.org/viewvc/incubator/storm/site/documentation/Installing-native-dependencies.md?rev=1564375&r1=1564374&r2=1564375&view=diff
==============================================================================
--- incubator/storm/site/documentation/Installing-native-dependencies.md (original)
+++ incubator/storm/site/documentation/Installing-native-dependencies.md Tue Feb 4 16:51:30 2014
@@ -1,3 +1,6 @@
+---
+layout: documentation
+---
The native dependencies are only needed on actual Storm clusters. When running Storm in local mode, Storm uses a pure Java messaging system so that you don't need to install native dependencies on your development machine.
Installing ZeroMQ and JZMQ is usually straightforward. Sometimes, however, people run into issues with autoconf and get strange errors. If you run into any issues, please email the [Storm mailing list](http://groups.google.com/group/storm-user) or come get help in the #storm-user room on freenode.
Modified: incubator/storm/site/documentation/Kestrel-and-Storm.md
URL: http://svn.apache.org/viewvc/incubator/storm/site/documentation/Kestrel-and-Storm.md?rev=1564375&r1=1564374&r2=1564375&view=diff
==============================================================================
--- incubator/storm/site/documentation/Kestrel-and-Storm.md (original)
+++ incubator/storm/site/documentation/Kestrel-and-Storm.md Tue Feb 4 16:51:30 2014
@@ -1,3 +1,6 @@
+---
+layout: documentation
+---
This page explains how to use to Storm to consume items from a Kestrel cluster.
## Preliminaries
@@ -173,7 +176,7 @@ In order to stop it type a closing brack
```
## Using KestrelSpout
-This topology reads sentences off of a Kestrel queue using KestrelSpout, splits the sentences into its constituent words (Bolt: SplitSentence), and then emits for each word the number of times it has seen that word before (Bolt: WordCount). How data is processed is described in detail in [Guaranteeing message processing](https://github.com/nathanmarz/storm/wiki/Guaranteeing-message-processing).
+This topology reads sentences off of a Kestrel queue using KestrelSpout, splits the sentences into its constituent words (Bolt: SplitSentence), and then emits for each word the number of times it has seen that word before (Bolt: WordCount). How data is processed is described in detail in [Guaranteeing message processing](Guaranteeing-message-processing.html).
```
TopologyBuilder builder = new TopologyBuilder();
Modified: incubator/storm/site/documentation/Lifecycle-of-a-topology.md
URL: http://svn.apache.org/viewvc/incubator/storm/site/documentation/Lifecycle-of-a-topology.md?rev=1564375&r1=1564374&r2=1564375&view=diff
==============================================================================
--- incubator/storm/site/documentation/Lifecycle-of-a-topology.md (original)
+++ incubator/storm/site/documentation/Lifecycle-of-a-topology.md Tue Feb 4 16:51:30 2014
@@ -1,3 +1,6 @@
+---
+layout: documentation
+---
(**NOTE**: this page is based on the 0.7.1 code; many things have changed since then, including a split between tasks and executors, and a reorganization of the code under `storm-core/src` rather than `src/`.)
This page explains in detail the lifecycle of a topology from running the "storm jar" command to uploading the topology to Nimbus to the supervisors starting/stopping workers to workers and tasks setting themselves up. It also explains how Nimbus monitors topologies and how topologies are shutdown when they are killed.
Modified: incubator/storm/site/documentation/Local-mode.md
URL: http://svn.apache.org/viewvc/incubator/storm/site/documentation/Local-mode.md?rev=1564375&r1=1564374&r2=1564375&view=diff
==============================================================================
--- incubator/storm/site/documentation/Local-mode.md (original)
+++ incubator/storm/site/documentation/Local-mode.md Tue Feb 4 16:51:30 2014
@@ -1,4 +1,7 @@
-Local mode simulates a Storm cluster in process and is useful for developing and testing topologies. Running topologies in local mode is similar to running topologies [on a cluster](https://github.com/nathanmarz/storm/wiki/Running-topologies-on-a-production-cluster).
+---
+layout: documentation
+---
+Local mode simulates a Storm cluster in process and is useful for developing and testing topologies. Running topologies in local mode is similar to running topologies [on a cluster](Running-topologies-on-a-production-cluster.html).
To create an in-process cluster, simply use the `LocalCluster` class. For example:
Modified: incubator/storm/site/documentation/Maven.md
URL: http://svn.apache.org/viewvc/incubator/storm/site/documentation/Maven.md?rev=1564375&r1=1564374&r2=1564375&view=diff
==============================================================================
--- incubator/storm/site/documentation/Maven.md (original)
+++ incubator/storm/site/documentation/Maven.md Tue Feb 4 16:51:30 2014
@@ -1,3 +1,6 @@
+---
+layout: documentation
+---
To develop topologies, you'll need the Storm jars on your classpath. You should either include the unpacked jars in the classpath for your project or use Maven to include Storm as a development dependency. Storm is hosted on Clojars (a Maven repository). To include Storm in your project as a development dependency, add the following to your pom.xml:
```xml
Modified: incubator/storm/site/documentation/Message-passing-implementation.md
URL: http://svn.apache.org/viewvc/incubator/storm/site/documentation/Message-passing-implementation.md?rev=1564375&r1=1564374&r2=1564375&view=diff
==============================================================================
--- incubator/storm/site/documentation/Message-passing-implementation.md (original)
+++ incubator/storm/site/documentation/Message-passing-implementation.md Tue Feb 4 16:51:30 2014
@@ -1,3 +1,6 @@
+---
+layout: documentation
+---
(Note: this walkthrough is out of date as of 0.8.0. 0.8.0 revamped the message passing infrastructure to be based on the Disruptor)
This page walks through how emitting and transferring tuples works in Storm.
Modified: incubator/storm/site/documentation/Metrics.md
URL: http://svn.apache.org/viewvc/incubator/storm/site/documentation/Metrics.md?rev=1564375&r1=1564374&r2=1564375&view=diff
==============================================================================
--- incubator/storm/site/documentation/Metrics.md (original)
+++ incubator/storm/site/documentation/Metrics.md Tue Feb 4 16:51:30 2014
@@ -1,4 +1,6 @@
-
+---
+layout: documentation
+---
Storm exposes a metrics interface to report summary statistics across the full topology.
It's used internally to track the numbers you see in the Nimbus UI console: counts of executes and acks; average process latency per bolt; worker heap usage; and so forth.
Modified: incubator/storm/site/documentation/Multilang-protocol.md
URL: http://svn.apache.org/viewvc/incubator/storm/site/documentation/Multilang-protocol.md?rev=1564375&r1=1564374&r2=1564375&view=diff
==============================================================================
--- incubator/storm/site/documentation/Multilang-protocol.md (original)
+++ incubator/storm/site/documentation/Multilang-protocol.md Tue Feb 4 16:51:30 2014
@@ -1,4 +1,7 @@
-This page explains the multilang protocol as of Storm 0.7.1. Versions prior to 0.7.1 used a somewhat different protocol, documented [here](https://github.com/nathanmarz/storm/wiki/Storm-multi-language-protocol-(versions-0.7.0-and-below\)).
+---
+layout: documentation
+---
+fThis page explains the multilang protocol as of Storm 0.7.1. Versions prior to 0.7.1 used a somewhat different protocol, documented [here](Storm-multi-language-protocol-(versions-0.7.0-and-below\).html).
# Storm Multi-Language Protocol
Modified: incubator/storm/site/documentation/Powered-By.md
URL: http://svn.apache.org/viewvc/incubator/storm/site/documentation/Powered-By.md?rev=1564375&r1=1564374&r2=1564375&view=diff
==============================================================================
--- incubator/storm/site/documentation/Powered-By.md (original)
+++ incubator/storm/site/documentation/Powered-By.md Tue Feb 4 16:51:30 2014
@@ -1,3 +1,6 @@
+---
+layout: documentation
+---
Want to be added to this page? Send an email [here](mailto:nathan.marz@gmail.com).
<table>
Modified: incubator/storm/site/documentation/Project-ideas.md
URL: http://svn.apache.org/viewvc/incubator/storm/site/documentation/Project-ideas.md?rev=1564375&r1=1564374&r2=1564375&view=diff
==============================================================================
--- incubator/storm/site/documentation/Project-ideas.md (original)
+++ incubator/storm/site/documentation/Project-ideas.md Tue Feb 4 16:51:30 2014
@@ -1,3 +1,6 @@
+---
+layout: documentation
+---
* **DSLs for non-JVM languages:** These DSL's should be all-inclusive and not require any Java for the creation of topologies, spouts, or bolts. Since topologies are [Thrift](http://thrift.apache.org/) structs, Nimbus is a Thrift service, and bolts can be written in any language, this is possible.
* **Online machine learning algorithms:** Something like [Mahout](http://mahout.apache.org/) but for online algorithms
* **Suite of performance benchmarks:** These benchmarks should test Storm's performance on CPU and IO intensive workloads. There should be benchmarks for different classes of applications, such as stream processing (where throughput is the priority) and distributed RPC (where latency is the priority).
\ No newline at end of file
Modified: incubator/storm/site/documentation/Rationale.md
URL: http://svn.apache.org/viewvc/incubator/storm/site/documentation/Rationale.md?rev=1564375&r1=1564374&r2=1564375&view=diff
==============================================================================
--- incubator/storm/site/documentation/Rationale.md (original)
+++ incubator/storm/site/documentation/Rationale.md Tue Feb 4 16:51:30 2014
@@ -1,3 +1,6 @@
+---
+layout: documentation
+---
The past decade has seen a revolution in data processing. MapReduce, Hadoop, and related technologies have made it possible to store and process data at scales previously unthinkable. Unfortunately, these data processing technologies are not realtime systems, nor are they meant to be. There's no hack that will turn Hadoop into a realtime system; realtime data processing has a fundamentally different set of requirements than batch processing.
However, realtime data processing at massive scale is becoming more and more of a requirement for businesses. The lack of a "Hadoop of realtime" has become the biggest hole in the data processing ecosystem.
Modified: incubator/storm/site/documentation/Running-topologies-on-a-production-cluster.md
URL: http://svn.apache.org/viewvc/incubator/storm/site/documentation/Running-topologies-on-a-production-cluster.md?rev=1564375&r1=1564374&r2=1564375&view=diff
==============================================================================
--- incubator/storm/site/documentation/Running-topologies-on-a-production-cluster.md (original)
+++ incubator/storm/site/documentation/Running-topologies-on-a-production-cluster.md Tue Feb 4 16:51:30 2014
@@ -1,3 +1,6 @@
+---
+layout: documentation
+---
Running topologies on a production cluster is similar to running in [[Local mode]]. Here are the steps:
1) Define the topology (Use [TopologyBuilder](http://nathanmarz.github.com/storm/doc/backtype/storm/topology/TopologyBuilder.html) if defining using Java)
Modified: incubator/storm/site/documentation/Serialization-(prior-to-0.6.0).md
URL: http://svn.apache.org/viewvc/incubator/storm/site/documentation/Serialization-%28prior-to-0.6.0%29.md?rev=1564375&r1=1564374&r2=1564375&view=diff
==============================================================================
--- incubator/storm/site/documentation/Serialization-(prior-to-0.6.0).md (original)
+++ incubator/storm/site/documentation/Serialization-(prior-to-0.6.0).md Tue Feb 4 16:51:30 2014
@@ -1,3 +1,6 @@
+---
+layout: documentation
+---
Tuples can be comprised of objects of any types. Since Storm is a distributed system, it needs to know how to serialize and deserialize objects when they're passed between tasks. By default Storm can serialize ints, shorts, longs, floats, doubles, bools, bytes, strings, and byte arrays, but if you want to use another type in your tuples, you'll need to implement a custom serializer.
### Dynamic typing
Modified: incubator/storm/site/documentation/Serialization.md
URL: http://svn.apache.org/viewvc/incubator/storm/site/documentation/Serialization.md?rev=1564375&r1=1564374&r2=1564375&view=diff
==============================================================================
--- incubator/storm/site/documentation/Serialization.md (original)
+++ incubator/storm/site/documentation/Serialization.md Tue Feb 4 16:51:30 2014
@@ -1,3 +1,6 @@
+---
+layout: documentation
+---
This page is about how the serialization system in Storm works for versions 0.6.0 and onwards. Storm used a different serialization system prior to 0.6.0 which is documented on [[Serialization (prior to 0.6.0)]].
Tuples can be comprised of objects of any types. Since Storm is a distributed system, it needs to know how to serialize and deserialize objects when they're passed between tasks.
Modified: incubator/storm/site/documentation/Serializers.md
URL: http://svn.apache.org/viewvc/incubator/storm/site/documentation/Serializers.md?rev=1564375&r1=1564374&r2=1564375&view=diff
==============================================================================
--- incubator/storm/site/documentation/Serializers.md (original)
+++ incubator/storm/site/documentation/Serializers.md Tue Feb 4 16:51:30 2014
@@ -1 +1,4 @@
+---
+layout: documentation
+---
* [storm-json](https://github.com/rapportive-oss/storm-json): Simple JSON serializer for Storm
\ No newline at end of file
Modified: incubator/storm/site/documentation/Setting-up-a-Storm-cluster.md
URL: http://svn.apache.org/viewvc/incubator/storm/site/documentation/Setting-up-a-Storm-cluster.md?rev=1564375&r1=1564374&r2=1564375&view=diff
==============================================================================
--- incubator/storm/site/documentation/Setting-up-a-Storm-cluster.md (original)
+++ incubator/storm/site/documentation/Setting-up-a-Storm-cluster.md Tue Feb 4 16:51:30 2014
@@ -1,6 +1,9 @@
+---
+layout: documentation
+---
This page outlines the steps for getting a Storm cluster up and running. If you're on AWS, you should check out the [storm-deploy](https://github.com/nathanmarz/storm-deploy/wiki) project. [storm-deploy](https://github.com/nathanmarz/storm-deploy/wiki) completely automates the provisioning, configuration, and installation of Storm clusters on EC2. It also sets up Ganglia for you so you can monitor CPU, disk, and network usage.
-If you run into difficulties with your Storm cluster, first check for a solution is in the [[Troubleshooting]] page. Otherwise, email the [mailing list](https://groups.google.com/group/storm-user).
+If you run into difficulties with your Storm cluster, first check for a solution is in the [Troubleshooting](Troubleshooting.html) page. Otherwise, email the mailing list.
Here's a summary of the steps for setting up a Storm cluster:
@@ -23,11 +26,8 @@ A few notes about Zookeeper deployment:
Next you need to install Storm's dependencies on Nimbus and the worker machines. These are:
-1. [ZeroMQ 2.1.7](http://www.zeromq.org/area:download) - Note that you should **not** install version 2.1.10, as that version has some serious bugs that can cause strange issues for a Storm cluster. In some rare cases, users have reported an "IllegalArgumentException" bubbling up from the ZeroMQ code when using 2.1.7 â in these cases downgrading to 2.1.4 fixed the problem.
-2. [JZMQ](http://github.com/nathanmarz/jzmq)
-3. Java 6
-4. Python 2.6.6
-5. unzip
+1. Java 6
+2. Python 2.6.6
These are the versions of the dependencies that have been tested with Storm. Storm may or may not work with different versions of Java and/or Python.
@@ -57,15 +57,13 @@ If the port that your Zookeeper cluster
storm.local.dir: "/mnt/storm"
```
-3) **java.library.path**: This is the load path for the native libraries that Storm uses (ZeroMQ and JZMQ). The default of "/usr/local/lib:/opt/local/lib:/usr/lib" should be fine for most installations, so you probably don't need to set this config.
-
-4) **nimbus.host**: The worker nodes need to know which machine is the master in order to download topology jars and confs. For example:
+3) **nimbus.host**: The worker nodes need to know which machine is the master in order to download topology jars and confs. For example:
```yaml
nimbus.host: "111.222.333.44"
```
-5) **supervisor.slots.ports**: For each worker machine, you configure how many workers run on that machine with this config. Each worker uses a single port for receiving messages, and this setting defines which ports are open for use. If you define five ports here, then Storm will allocate up to five workers to run on this machine. If you define three ports, Storm will only run up to three. By default, this setting is configured to run 4 workers on the ports 6700, 6701, 6702, and 6703. For example:
+4) **supervisor.slots.ports**: For each worker machine, you configure how many workers run on that machine with this config. Each worker uses a single port for receiving messages, and this setting defines which ports are open for use. If you define five ports here, then Storm will allocate up to five workers to run on this machine. If you define three ports, Storm will only run up to three. By default, this setting is configured to run 4 workers on the ports 6700, 6701, 6702, and 6703. For example:
```yaml
supervisor.slots.ports:
Modified: incubator/storm/site/documentation/Setting-up-development-environment.md
URL: http://svn.apache.org/viewvc/incubator/storm/site/documentation/Setting-up-development-environment.md?rev=1564375&r1=1564374&r2=1564375&view=diff
==============================================================================
--- incubator/storm/site/documentation/Setting-up-development-environment.md (original)
+++ incubator/storm/site/documentation/Setting-up-development-environment.md Tue Feb 4 16:51:30 2014
@@ -1,6 +1,9 @@
+---
+layout: documentation
+---
This page outlines what you need to do to get a Storm development environment set up. In summary, the steps are:
-1. Download a [Storm release](http://storm-project.net/downloads.html) , unpack it, and put the unpacked `bin/` directory on your PATH
+1. Download a [Storm release](..//downloads.html) , unpack it, and put the unpacked `bin/` directory on your PATH
2. To be able to start and stop topologies on a remote cluster, put the cluster information in `~/.storm/storm.yaml`
More detail on each of these steps is below.
Modified: incubator/storm/site/documentation/Spout-implementations.md
URL: http://svn.apache.org/viewvc/incubator/storm/site/documentation/Spout-implementations.md?rev=1564375&r1=1564374&r2=1564375&view=diff
==============================================================================
--- incubator/storm/site/documentation/Spout-implementations.md (original)
+++ incubator/storm/site/documentation/Spout-implementations.md Tue Feb 4 16:51:30 2014
@@ -1,3 +1,6 @@
+---
+layout: documentation
+---
* [storm-kestrel](https://github.com/nathanmarz/storm-kestrel): Adapter to use Kestrel as a spout
* [storm-amqp-spout](https://github.com/rapportive-oss/storm-amqp-spout): Adapter to use AMQP source as a spout
* [storm-jms](https://github.com/ptgoetz/storm-jms): Adapter to use a JMS source as a spout
Modified: incubator/storm/site/documentation/Storm-multi-language-protocol-(versions-0.7.0-and-below).md
URL: http://svn.apache.org/viewvc/incubator/storm/site/documentation/Storm-multi-language-protocol-%28versions-0.7.0-and-below%29.md?rev=1564375&r1=1564374&r2=1564375&view=diff
==============================================================================
--- incubator/storm/site/documentation/Storm-multi-language-protocol-(versions-0.7.0-and-below).md (original)
+++ incubator/storm/site/documentation/Storm-multi-language-protocol-(versions-0.7.0-and-below).md Tue Feb 4 16:51:30 2014
@@ -1,3 +1,6 @@
+---
+layout: documentation
+---
This page explains the multilang protocol for versions 0.7.0 and below. The protocol changed in version 0.7.1.
# Storm Multi-Language Protocol
Modified: incubator/storm/site/documentation/Structure-of-the-codebase.md
URL: http://svn.apache.org/viewvc/incubator/storm/site/documentation/Structure-of-the-codebase.md?rev=1564375&r1=1564374&r2=1564375&view=diff
==============================================================================
--- incubator/storm/site/documentation/Structure-of-the-codebase.md (original)
+++ incubator/storm/site/documentation/Structure-of-the-codebase.md Tue Feb 4 16:51:30 2014
@@ -1,3 +1,6 @@
+---
+layout: documentation
+---
There are three distinct layers to Storm's codebase.
First, Storm was designed from the very beginning to be compatible with multiple languages. Nimbus is a Thrift service and topologies are defined as Thrift structures. The usage of Thrift allows Storm to be used from any language.
Modified: incubator/storm/site/documentation/Support-for-non-java-languages.md
URL: http://svn.apache.org/viewvc/incubator/storm/site/documentation/Support-for-non-java-languages.md?rev=1564375&r1=1564374&r2=1564375&view=diff
==============================================================================
--- incubator/storm/site/documentation/Support-for-non-java-languages.md (original)
+++ incubator/storm/site/documentation/Support-for-non-java-languages.md Tue Feb 4 16:51:30 2014
@@ -1,4 +1,7 @@
+---
+layout: documentation
+---
* [Scala DSL](https://github.com/velvia/ScalaStorm)
* [JRuby DSL](https://github.com/colinsurprenant/storm-jruby)
-* [[Clojure DSL]]
+* [Clojure DSL](Clojure-DSL.html)
* [io-storm](https://github.com/gphat/io-storm): Perl multilang adapter
Modified: incubator/storm/site/documentation/Transactional-topologies.md
URL: http://svn.apache.org/viewvc/incubator/storm/site/documentation/Transactional-topologies.md?rev=1564375&r1=1564374&r2=1564375&view=diff
==============================================================================
--- incubator/storm/site/documentation/Transactional-topologies.md (original)
+++ incubator/storm/site/documentation/Transactional-topologies.md Tue Feb 4 16:51:30 2014
@@ -1,12 +1,15 @@
-**NOTE**: Transactional topologies have been deprecated -- use the [[Trident|Trident-tutorial]] framework instead.
+---
+layout: documentation
+---
+**NOTE**: Transactional topologies have been deprecated -- use the [Trident](Trident-tutorial.html) framework instead.
__________________________________________________________________________
-Storm [guarantees data processing](https://github.com/nathanmarz/storm/wiki/Guaranteeing-message-processing) by providing an at least once processing guarantee. The most common question asked about Storm is "Given that tuples can be replayed, how do you do things like counting on top of Storm? Won't you overcount?"
+Storm [guarantees data processing](Guaranteeing-message-processing.html) by providing an at least once processing guarantee. The most common question asked about Storm is "Given that tuples can be replayed, how do you do things like counting on top of Storm? Won't you overcount?"
Storm 0.7.0 introduces transactional topologies, which enable you to get exactly once messaging semantics for pretty much any computation. So you can do things like counting in a fully-accurate, scalable, and fault-tolerant way.
-Like [Distributed RPC](https://github.com/nathanmarz/storm/wiki/Distributed-RPC), transactional topologies aren't so much a feature of Storm as they are a higher level abstraction built on top of Storm's primitives of streams, spouts, bolts, and topologies.
+Like [Distributed RPC](Distributed-RPC.html), transactional topologies aren't so much a feature of Storm as they are a higher level abstraction built on top of Storm's primitives of streams, spouts, bolts, and topologies.
This page explains the transactional topology abstraction, how to use the API, and provides details as to its implementation.
Modified: incubator/storm/site/documentation/Trident-API-Overview.md
URL: http://svn.apache.org/viewvc/incubator/storm/site/documentation/Trident-API-Overview.md?rev=1564375&r1=1564374&r2=1564375&view=diff
==============================================================================
--- incubator/storm/site/documentation/Trident-API-Overview.md (original)
+++ incubator/storm/site/documentation/Trident-API-Overview.md Tue Feb 4 16:51:30 2014
@@ -1,3 +1,6 @@
+---
+layout: documentation
+---
# Trident API overview
The core data model in Trident is the "Stream", processed as a series of batches. A stream is partitioned among the nodes in the cluster, and operations applied to a stream are applied in parallel across each partition.
@@ -228,7 +231,7 @@ This code will run the Count and Sum agg
### stateQuery and partitionPersist
-stateQuery and partitionPersist query and update sources of state, respectively. You can read about how to use them on [Trident state doc](https://github.com/nathanmarz/storm/wiki/Trident-state).
+stateQuery and partitionPersist query and update sources of state, respectively. You can read about how to use them on [Trident state doc](Trident-state.html).
### projection
@@ -274,7 +277,7 @@ The groupBy operation repartitions the s
-If you run aggregators on a grouped stream, the aggregation will be run within each group instead of against the whole batch. persistentAggregate can also be run on a GroupedStream, in which case the results will be stored in a [MapState](https://github.com/nathanmarz/storm/blob/master/storm-core/src/jvm/storm/trident/state/map/MapState.java) with the key being the grouping fields. You can read more about persistentAggregate in the [Trident state doc](https://github.com/nathanmarz/storm/wiki/Trident-state).
+If you run aggregators on a grouped stream, the aggregation will be run within each group instead of against the whole batch. persistentAggregate can also be run on a GroupedStream, in which case the results will be stored in a [MapState](https://github.com/nathanmarz/storm/blob/master/storm-core/src/jvm/storm/trident/state/map/MapState.java) with the key being the grouping fields. You can read more about persistentAggregate in the [Trident state doc](Trident-state.html).
Like regular streams, aggregators on grouped streams can be chained.
Modified: incubator/storm/site/documentation/Trident-spouts.md
URL: http://svn.apache.org/viewvc/incubator/storm/site/documentation/Trident-spouts.md?rev=1564375&r1=1564374&r2=1564375&view=diff
==============================================================================
--- incubator/storm/site/documentation/Trident-spouts.md (original)
+++ incubator/storm/site/documentation/Trident-spouts.md Tue Feb 4 16:51:30 2014
@@ -1,8 +1,11 @@
+---
+layout: documentation
+---
# Trident spouts
Like in the vanilla Storm API, spouts are the source of streams in a Trident topology. On top of the vanilla Storm spouts, Trident exposes additional APIs for more sophisticated spouts.
-There is an inextricable link between how you source your data streams and how you update state (e.g. databases) based on those data streams. See [Trident state doc](https://github.com/nathanmarz/storm/wiki/Trident-state) for an explanation of this â understanding this link is imperative for understanding the spout options available.
+There is an inextricable link between how you source your data streams and how you update state (e.g. databases) based on those data streams. See [Trident state doc](Trident-state.html) for an explanation of this â understanding this link is imperative for understanding the spout options available.
Regular Storm spouts will be non-transactional spouts in a Trident topology. To use a regular Storm IRichSpout, create the stream like this in a TridentTopology:
@@ -23,7 +26,7 @@ You can configure the Zookeeper storage
By default, Trident processes a single batch at a time, waiting for the batch to succeed or fail before trying another batch. You can get significantly higher throughput â and lower latency of processing of each batch â by pipelining the batches. You configure the maximum amount of batches to be processed simultaneously with the "topology.max.spout.pending" property.
-Even while processing multiple batches simultaneously, Trident will order any state updates taking place in the topology among batches. For example, suppose you're doing a global count aggregation into a database. The idea is that while you're updating the count in the database for batch 1, you can still be computing the partial counts for batches 2 through 10. Trident won't move on to the state updates for batch 2 until the state updates for batch 1 have succeeded. This is essential for achieving exactly-once processing semantics, as outline in [Trident state doc](https://github.com/nathanmarz/storm/wiki/Trident-state).
+Even while processing multiple batches simultaneously, Trident will order any state updates taking place in the topology among batches. For example, suppose you're doing a global count aggregation into a database. The idea is that while you're updating the count in the database for batch 1, you can still be computing the partial counts for batches 2 through 10. Trident won't move on to the state updates for batch 2 until the state updates for batch 1 have succeeded. This is essential for achieving exactly-once processing semantics, as outline in [Trident state doc](Trident-state.html).
## Trident spout types
Modified: incubator/storm/site/documentation/Trident-state.md
URL: http://svn.apache.org/viewvc/incubator/storm/site/documentation/Trident-state.md?rev=1564375&r1=1564374&r2=1564375&view=diff
==============================================================================
--- incubator/storm/site/documentation/Trident-state.md (original)
+++ incubator/storm/site/documentation/Trident-state.md Tue Feb 4 16:51:30 2014
@@ -1,3 +1,6 @@
+---
+layout: documentation
+---
# State in Trident
Trident has first-class abstractions for reading from and writing to stateful sources. The state can either be internal to the topology â e.g., kept in-memory and backed by HDFS â or externally stored in a database like Memcached or Cassandra. There's no difference in the Trident API for either case.
@@ -10,7 +13,7 @@ When failures occur, tuples will be repl
By just storing the count in the database, you have no idea whether or not this tuple has been processed before. So you need more information in order to make the right decision. Trident provides the following semantics which are sufficient for achieving exactly-once processing semantics:
-1. Tuples are processed as small batches (see [the tutorial](https://github.com/nathanmarz/storm/wiki/Trident-tutorial))
+1. Tuples are processed as small batches (see [the tutorial](Trident-tutorial.html))
2. Each batch of tuples is given a unique id called the "transaction id" (txid). If the batch is replayed, it is given the exact same txid.
3. State updates are ordered among batches. That is, the state updates for batch 3 won't be applied until the state updates for batch 2 have succeeded.
Modified: incubator/storm/site/documentation/Trident-tutorial.md
URL: http://svn.apache.org/viewvc/incubator/storm/site/documentation/Trident-tutorial.md?rev=1564375&r1=1564374&r2=1564375&view=diff
==============================================================================
--- incubator/storm/site/documentation/Trident-tutorial.md (original)
+++ incubator/storm/site/documentation/Trident-tutorial.md Tue Feb 4 16:51:30 2014
@@ -1,3 +1,6 @@
+---
+layout: documentation
+---
# Trident tutorial
Trident is a high-level abstraction for doing realtime computing on top of Storm. It allows you to seamlessly intermix high throughput (millions of messages per second), stateful stream processing with low latency distributed querying. If you're familiar with high level batch processing tools like Pig or Cascading, the concepts of Trident will be very familiar â Trident has joins, aggregations, grouping, functions, and filters. In addition to these, Trident adds primitives for doing stateful, incremental processing on top of any database or persistence store. Trident has consistent, exactly-once semantics, so it is easy to reason about Trident topologies.
Modified: incubator/storm/site/documentation/Troubleshooting.md
URL: http://svn.apache.org/viewvc/incubator/storm/site/documentation/Troubleshooting.md?rev=1564375&r1=1564374&r2=1564375&view=diff
==============================================================================
--- incubator/storm/site/documentation/Troubleshooting.md (original)
+++ incubator/storm/site/documentation/Troubleshooting.md Tue Feb 4 16:51:30 2014
@@ -1,3 +1,6 @@
+---
+layout: documentation
+---
## Troubleshooting
This page lists issues people have run into when using Storm along with their solutions.
Modified: incubator/storm/site/documentation/Tutorial.md
URL: http://svn.apache.org/viewvc/incubator/storm/site/documentation/Tutorial.md?rev=1564375&r1=1564374&r2=1564375&view=diff
==============================================================================
--- incubator/storm/site/documentation/Tutorial.md (original)
+++ incubator/storm/site/documentation/Tutorial.md Tue Feb 4 16:51:30 2014
@@ -1,3 +1,6 @@
+---
+layout: documentation
+---
In this tutorial, you'll learn how to create Storm topologies and deploy them to a Storm cluster. Java will be the main language used, but a few examples will use Python to illustrate Storm's multi-language capabilities.
## Preliminaries
@@ -52,7 +55,7 @@ A topology runs forever, or until you ki
## Data model
-Storm uses tuples as its data model. A tuple is a named list of values, and a field in a tuple can be an object of any type. Out of the box, Storm supports all the primitive types, strings, and byte arrays as tuple field values. To use an object of another type, you just need to implement [a serializer](https://github.com/nathanmarz/storm/wiki/Serialization) for the type.
+Storm uses tuples as its data model. A tuple is a named list of values, and a field in a tuple can be an object of any type. Out of the box, Storm supports all the primitive types, strings, and byte arrays as tuple field values. To use an object of another type, you just need to implement [a serializer](Serialization.html) for the type.
Every node in a topology must declare the output fields for the tuples it emits. For example, this bolt declares that it emits 2-tuples with the fields "double" and "triple":
@@ -296,11 +299,11 @@ Earlier on in this tutorial, we skipped
## Transactional topologies
-Storm guarantees that every message will be played through the topology at least once. A common question asked is "how do you do things like counting on top of Storm? Won't you overcount?" Storm has a feature called transactional topologies that let you achieve exactly-once messaging semantics for most computations. Read more about transactional topologies [here](https://github.com/nathanmarz/storm/wiki/Transactional-topologies).
+Storm guarantees that every message will be played through the topology at least once. A common question asked is "how do you do things like counting on top of Storm? Won't you overcount?" Storm has a feature called transactional topologies that let you achieve exactly-once messaging semantics for most computations. Read more about transactional topologies [here](Transactional-topologies.html).
## Distributed RPC
-This tutorial showed how to do basic stream processing on top of Storm. There's lots more things you can do with Storm's primitives. One of the most interesting applications of Storm is Distributed RPC, where you parallelize the computation of intense functions on the fly. Read more about Distributed RPC [here](https://github.com/nathanmarz/storm/wiki/Distributed-RPC).
+This tutorial showed how to do basic stream processing on top of Storm. There's lots more things you can do with Storm's primitives. One of the most interesting applications of Storm is Distributed RPC, where you parallelize the computation of intense functions on the fly. Read more about Distributed RPC [here](Distributed-RPC.html).
## Conclusion
Modified: incubator/storm/site/documentation/Understanding-the-parallelism-of-a-Storm-topology.md
URL: http://svn.apache.org/viewvc/incubator/storm/site/documentation/Understanding-the-parallelism-of-a-Storm-topology.md?rev=1564375&r1=1564374&r2=1564375&view=diff
==============================================================================
--- incubator/storm/site/documentation/Understanding-the-parallelism-of-a-Storm-topology.md (original)
+++ incubator/storm/site/documentation/Understanding-the-parallelism-of-a-Storm-topology.md Tue Feb 4 16:51:30 2014
@@ -1,3 +1,6 @@
+---
+layout: documentation
+---
# What makes a running topology: worker processes, executors and tasks
Storm distinguishes between the following three main entities that are used to actually run a topology in a Storm cluster:
Modified: incubator/storm/site/documentation/Using-non-JVM-languages-with-Storm.md
URL: http://svn.apache.org/viewvc/incubator/storm/site/documentation/Using-non-JVM-languages-with-Storm.md?rev=1564375&r1=1564374&r2=1564375&view=diff
==============================================================================
--- incubator/storm/site/documentation/Using-non-JVM-languages-with-Storm.md (original)
+++ incubator/storm/site/documentation/Using-non-JVM-languages-with-Storm.md Tue Feb 4 16:51:30 2014
@@ -1,3 +1,6 @@
+---
+layout: documentation
+---
- two pieces: creating topologies and implementing spouts and bolts in other languages
- creating topologies in another language is easy since topologies are just thrift structures (link to storm.thrift)
- implementing spouts and bolts in another language is called a "multilang components" or "shelling"
Modified: incubator/storm/site/index.html
URL: http://svn.apache.org/viewvc/incubator/storm/site/index.html?rev=1564375&r1=1564374&r2=1564375&view=diff
==============================================================================
--- incubator/storm/site/index.html (original)
+++ incubator/storm/site/index.html Tue Feb 4 16:51:30 2014
@@ -75,14 +75,18 @@ Storm has many use cases: realtime analy
</p>
<p>
-Storm <a href="/about/integrates.html">integrates</a> with the queueing and database technologies you already use. A Storm topology consumes streams of data and processes those streams in arbitrarily complex ways, repartitioning the streams between each stage of the computation however needed. Read more in <a href="https://github.com/nathanmarz/storm/wiki/Tutorial">the tutorial</a>.
+<a href="/documentation/FAQ.html">FAQ</a>
+</p>
+
+<p>
+Storm <a href="/about/integrates.html">integrates</a> with the queueing and database technologies you already use. A Storm topology consumes streams of data and processes those streams in arbitrarily complex ways, repartitioning the streams between each stage of the computation however needed. Read more in <a href="/documentation/Tutorial.html">the tutorial</a>.
</p>
<h2>Companies & Projects Using Storm</h2>
{% for logo in page.logos %}
<a href="{{ logo[0] }}"><img style="height: {{ logo[2] }}px; vertical-align: middle; margin-top: 4px; margin-left: 2px;" src="/images/logos/{{ logo[1]}}"/></a>
{% endfor %}
-<a href="https://github.com/nathanmarz/storm/wiki/Powered-By" style="font-size: 23px; padding: 11px;">and many others</a>
+<a href="/documentation/Powered-By.html" style="font-size: 23px; padding: 11px;">and many others</a>
</div>
<div id="right">
@@ -93,7 +97,7 @@ Storm <a href="/about/integrates.html">i
</p>
<h2>Source code</h2>
-<iframe src="http://ghbtns.com/github-btn.html?user=nathanmarz&repo=storm&type=watch&count=true&size=large"
+<iframe src="http://ghbtns.com/github-btn.html?user=apache&repo=incubator-storm&type=watch&count=true&size=large"
allowtransparency="true" frameborder="0" scrolling="0" width="170px" height="30px"></iframe>
<h2>Community</h2>