You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@storm.apache.org by bo...@apache.org on 2016/03/17 03:56:15 UTC
svn commit: r1735362 - in
/storm/branches/bobby-versioned-site/releases/0.10.0: ./ images/
Author: bobby
Date: Thu Mar 17 02:56:15 2016
New Revision: 1735362
URL: http://svn.apache.org/viewvc?rev=1735362&view=rev
Log:
Some more cleanup of some of the release specific pages
Added:
storm/branches/bobby-versioned-site/releases/0.10.0/Daemon-Fault-Tolerance.md
- copied, changed from r1735360, storm/branches/bobby-versioned-site/releases/0.10.0/Fault-tolerance.md
storm/branches/bobby-versioned-site/releases/0.10.0/images/ack_tree.png (with props)
storm/branches/bobby-versioned-site/releases/0.10.0/images/tuple-dag.png (with props)
storm/branches/bobby-versioned-site/releases/0.10.0/images/tuple_tree.png (with props)
Removed:
storm/branches/bobby-versioned-site/releases/0.10.0/Acking-framework-implementation.md
storm/branches/bobby-versioned-site/releases/0.10.0/Fault-tolerance.md
Modified:
storm/branches/bobby-versioned-site/releases/0.10.0/Guaranteeing-message-processing.md
storm/branches/bobby-versioned-site/releases/0.10.0/Implementation-docs.md
storm/branches/bobby-versioned-site/releases/0.10.0/index.md
Copied: storm/branches/bobby-versioned-site/releases/0.10.0/Daemon-Fault-Tolerance.md (from r1735360, storm/branches/bobby-versioned-site/releases/0.10.0/Fault-tolerance.md)
URL: http://svn.apache.org/viewvc/storm/branches/bobby-versioned-site/releases/0.10.0/Daemon-Fault-Tolerance.md?p2=storm/branches/bobby-versioned-site/releases/0.10.0/Daemon-Fault-Tolerance.md&p1=storm/branches/bobby-versioned-site/releases/0.10.0/Fault-tolerance.md&r1=1735360&r2=1735362&rev=1735362&view=diff
==============================================================================
--- storm/branches/bobby-versioned-site/releases/0.10.0/Fault-tolerance.md (original)
+++ storm/branches/bobby-versioned-site/releases/0.10.0/Daemon-Fault-Tolerance.md Thu Mar 17 02:56:15 2016
@@ -1,14 +1,14 @@
---
-title: Fault Tolerance
+title: Daemon Fault Tolerance
layout: documentation
documentation: true
version: v0.10.0
---
-This page explains the design details of Storm that make it a fault-tolerant system.
+Storm has several different daemon processes. Nimbus that schedules workers, supervisors that launch and kill workers, the log viewer that gives access to logs, and the UI that shows the status of a cluster.
## What happens when a worker dies?
-When a worker dies, the supervisor will restart it. If it continuously fails on startup and is unable to heartbeat to Nimbus, Nimbus will reassign the worker to another machine.
+When a worker dies, the supervisor will restart it. If it continuously fails on startup and is unable to heartbeat to Nimbus, Nimbus will reschedule the worker.
## What happens when a node dies?
Modified: storm/branches/bobby-versioned-site/releases/0.10.0/Guaranteeing-message-processing.md
URL: http://svn.apache.org/viewvc/storm/branches/bobby-versioned-site/releases/0.10.0/Guaranteeing-message-processing.md?rev=1735362&r1=1735361&r2=1735362&view=diff
==============================================================================
--- storm/branches/bobby-versioned-site/releases/0.10.0/Guaranteeing-message-processing.md (original)
+++ storm/branches/bobby-versioned-site/releases/0.10.0/Guaranteeing-message-processing.md Thu Mar 17 02:56:15 2016
@@ -4,7 +4,8 @@ layout: documentation
documentation: true
version: v0.10.0
---
-Storm guarantees that each message coming off a spout will be fully processed. This page describes how Storm accomplishes this guarantee and what you have to do as a user to benefit from Storm's reliability capabilities.
+Storm offers several different levels of guaranteed message processing, includeing best effort, at least once, and exactly once through [Trident](Trident-tutorial.html).
+This page describes how Storm can guarantee at least once processing.
### What does it mean for a message to be "fully processed"?
@@ -132,12 +133,11 @@ In contrast, bolts that do aggregations
### How do I make my applications work correctly given that tuples can be replayed?
-As always in software design, the answer is "it depends." Storm 0.7.0 introduced the "transactional topologies" feature, which enables you to get fully fault-tolerant exactly-once messaging semantics for most computations. Read more about transactional topologies [here](Transactional-topologies.html).
-
+As always in software design, the answer is "it depends." If you really want exactly once semantics use the [Trident](Trident-tutorial.html) API. In some cases, like with a lot of analytics, dropping data is OK so disabling the fault tolerance by setting the number of acker bolts to 0 [Config.TOPOLOGY_ACKERS](javadocs/backtype/storm/Config.html#TOPOLOGY_ACKERS). But in some cases you want to be sure that everything was processed at least once and nothing was dropped. This is especially useful if all operations are idenpotent or if deduping can happen aferwards.
### How does Storm implement reliability in an efficient way?
-A Storm topology has a set of special "acker" tasks that track the DAG of tuples for every spout tuple. When an acker sees that a DAG is complete, it sends a message to the spout task that created the spout tuple to ack the message. You can set the number of acker tasks for a topology in the topology configuration using [Config.TOPOLOGY_ACKERS](javadocs/backtype/storm/Config.html#TOPOLOGY_ACKERS). Storm defaults TOPOLOGY_ACKERS to one task -- you will need to increase this number for topologies processing large amounts of messages.
+A Storm topology has a set of special "acker" tasks that track the DAG of tuples for every spout tuple. When an acker sees that a DAG is complete, it sends a message to the spout task that created the spout tuple to ack the message. You can set the number of acker tasks for a topology in the topology configuration using [Config.TOPOLOGY_ACKERS](javadocs/backtype/storm/Config.html#TOPOLOGY_ACKERS). Storm defaults TOPOLOGY_ACKERS to one task per worker.
The best way to understand Storm's reliability implementation is to look at the lifecycle of tuples and tuple DAGs. When a tuple is created in a topology, whether in a spout or a bolt, it is given a random 64 bit id. These ids are used by ackers to track the tuple DAG for every spout tuple.
Modified: storm/branches/bobby-versioned-site/releases/0.10.0/Implementation-docs.md
URL: http://svn.apache.org/viewvc/storm/branches/bobby-versioned-site/releases/0.10.0/Implementation-docs.md?rev=1735362&r1=1735361&r2=1735362&view=diff
==============================================================================
--- storm/branches/bobby-versioned-site/releases/0.10.0/Implementation-docs.md (original)
+++ storm/branches/bobby-versioned-site/releases/0.10.0/Implementation-docs.md Thu Mar 17 02:56:15 2016
@@ -9,7 +9,6 @@ This section of the wiki is dedicated to
- [Structure of the codebase](Structure-of-the-codebase.html)
- [Lifecycle of a topology](Lifecycle-of-a-topology.html)
- [Message passing implementation](Message-passing-implementation.html)
-- [Acking framework implementation](Acking-framework-implementation.html)
- [Metrics](Metrics.html)
- How transactional topologies work
- subtopology for TransactionalSpout
Added: storm/branches/bobby-versioned-site/releases/0.10.0/images/ack_tree.png
URL: http://svn.apache.org/viewvc/storm/branches/bobby-versioned-site/releases/0.10.0/images/ack_tree.png?rev=1735362&view=auto
==============================================================================
Binary file - no diff available.
Propchange: storm/branches/bobby-versioned-site/releases/0.10.0/images/ack_tree.png
------------------------------------------------------------------------------
svn:mime-type = application/octet-stream
Added: storm/branches/bobby-versioned-site/releases/0.10.0/images/tuple-dag.png
URL: http://svn.apache.org/viewvc/storm/branches/bobby-versioned-site/releases/0.10.0/images/tuple-dag.png?rev=1735362&view=auto
==============================================================================
Binary file - no diff available.
Propchange: storm/branches/bobby-versioned-site/releases/0.10.0/images/tuple-dag.png
------------------------------------------------------------------------------
svn:mime-type = application/octet-stream
Added: storm/branches/bobby-versioned-site/releases/0.10.0/images/tuple_tree.png
URL: http://svn.apache.org/viewvc/storm/branches/bobby-versioned-site/releases/0.10.0/images/tuple_tree.png?rev=1735362&view=auto
==============================================================================
Binary file - no diff available.
Propchange: storm/branches/bobby-versioned-site/releases/0.10.0/images/tuple_tree.png
------------------------------------------------------------------------------
svn:mime-type = application/octet-stream
Modified: storm/branches/bobby-versioned-site/releases/0.10.0/index.md
URL: http://svn.apache.org/viewvc/storm/branches/bobby-versioned-site/releases/0.10.0/index.md?rev=1735362&r1=1735361&r2=1735362&view=diff
==============================================================================
--- storm/branches/bobby-versioned-site/releases/0.10.0/index.md (original)
+++ storm/branches/bobby-versioned-site/releases/0.10.0/index.md Thu Mar 17 02:56:15 2016
@@ -10,7 +10,7 @@ version: v0.10.0
* [Concepts](Concepts.html)
* [Configuration](Configuration.html)
* [Guaranteeing message processing](Guaranteeing-message-processing.html)
-* [Fault-tolerance](Fault-tolerance.html)
+* [Daemon Fault Tolerance](Daemon-Fault-Tolerance.html)
* [Command line client](Command-line-client.html)
* [REST API](STORM-UI-REST-API.html)
* [Understanding the parallelism of a Storm topology](Understanding-the-parallelism-of-a-Storm-topology.html)