You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@storm.apache.org by pt...@apache.org on 2016/01/20 02:06:52 UTC
[12/22] storm git commit: remove 'docs' directory
http://git-wip-us.apache.org/repos/asf/storm/blob/93e0d028/docs/documentation/Lifecycle-of-a-topology.md
----------------------------------------------------------------------
diff --git a/docs/documentation/Lifecycle-of-a-topology.md b/docs/documentation/Lifecycle-of-a-topology.md
deleted file mode 100644
index 68e5757..0000000
--- a/docs/documentation/Lifecycle-of-a-topology.md
+++ /dev/null
@@ -1,82 +0,0 @@
----
-title: Lifecycle of a Storm Topology
-layout: documentation
-documentation: true
----
-(**NOTE**: this page is based on the 0.7.1 code; many things have changed since then, including a split between tasks and executors, and a reorganization of the code under `storm-core/src` rather than `src/`.)
-
-This page explains in detail the lifecycle of a topology from running the "storm jar" command to uploading the topology to Nimbus to the supervisors starting/stopping workers to workers and tasks setting themselves up. It also explains how Nimbus monitors topologies and how topologies are shutdown when they are killed.
-
-First a couple of important notes about topologies:
-
-1. The actual topology that runs is different than the topology the user specifies. The actual topology has implicit streams and an implicit "acker" bolt added to manage the acking framework (used to guarantee data processing). The implicit topology is created via the [system-topology!](https://github.com/apache/storm/blob/0.7.1/src/clj/backtype/storm/daemon/common.clj#L188) function.
-2. `system-topology!` is used in two places:
- - when Nimbus is creating tasks for the topology [code](https://github.com/apache/storm/blob/0.7.1/src/clj/backtype/storm/daemon/nimbus.clj#L316)
- - in the worker so it knows where it needs to route messages to [code](https://github.com/apache/storm/blob/0.7.1/src/clj/backtype/storm/daemon/worker.clj#L90)
-
-## Starting a topology
-
-- "storm jar" command executes your class with the specified arguments. The only special thing that "storm jar" does is set the "storm.jar" environment variable for use by `StormSubmitter` later. [code](https://github.com/apache/storm/blob/0.7.1/bin/storm#L101)
-- When your code uses `StormSubmitter.submitTopology`, `StormSubmitter` takes the following actions:
- - First, `StormSubmitter` uploads the jar if it hasn't been uploaded before. [code](https://github.com/apache/storm/blob/0.7.1/src/jvm/backtype/storm/StormSubmitter.java#L83)
- - Jar uploading is done via Nimbus's Thrift interface [code](https://github.com/apache/storm/blob/0.7.1/src/storm.thrift#L200)
- - `beginFileUpload` returns a path in Nimbus's inbox
- - 15 kilobytes are uploaded at a time through `uploadChunk`
- - `finishFileUpload` is called when it's finished uploading
- - Here is Nimbus's implementation of those Thrift methods: [code](https://github.com/apache/storm/blob/0.7.1/src/clj/backtype/storm/daemon/nimbus.clj#L694)
- - Second, `StormSubmitter` calls `submitTopology` on the Nimbus thrift interface [code](https://github.com/apache/storm/blob/0.7.1/src/jvm/backtype/storm/StormSubmitter.java#L60)
- - The topology config is serialized using JSON (JSON is used so that writing DSL's in any language is as easy as possible)
- - Notice that the Thrift `submitTopology` call takes in the Nimbus inbox path where the jar was uploaded
-
-- Nimbus receives the topology submission. [code](https://github.com/apache/storm/blob/0.7.1/src/clj/backtype/storm/daemon/nimbus.clj#L639)
-- Nimbus normalizes the topology configuration. The main purpose of normalization is to ensure that every single task will have the same serialization registrations, which is critical for getting serialization working correctly. [code](https://github.com/apache/storm/blob/0.7.1/src/clj/backtype/storm/daemon/nimbus.clj#L557)
-- Nimbus sets up the static state for the topology [code](https://github.com/apache/storm/blob/0.7.1/src/clj/backtype/storm/daemon/nimbus.clj#L661)
- - Jars and configs are kept on local filesystem because they're too big for Zookeeper. The jar and configs are copied into the path {nimbus local dir}/stormdist/{topology id}
- - `setup-storm-static` writes task -> component mapping into ZK
- - `setup-heartbeats` creates a ZK "directory" in which tasks can heartbeat
-- Nimbus calls `mk-assignment` to assign tasks to machines [code](https://github.com/apache/storm/blob/0.7.1/src/clj/backtype/storm/daemon/nimbus.clj#L458)
- - Assignment record definition is here: [code](https://github.com/apache/storm/blob/0.7.1/src/clj/backtype/storm/daemon/common.clj#L25)
- - Assignment contains:
- - `master-code-dir`: used by supervisors to download the correct jars/configs for the topology from Nimbus
- - `task->node+port`: Map from a task id to the worker that task should be running on. (A worker is identified by a node/port pair)
- - `node->host`: A map from node id to hostname. This is used so workers know which machines to connect to to communicate with other workers. Node ids are used to identify supervisors so that multiple supervisors can be run on one machine. One place this is done is with Mesos integration.
- - `task->start-time-secs`: Contains a map from task id to the timestamp at which Nimbus launched that task. This is used by Nimbus when monitoring topologies, as tasks are given a longer timeout to heartbeat when they're first launched (the launch timeout is configured by "nimbus.task.launch.secs" config)
-- Once topologies are assigned, they're initially in a deactivated mode. `start-storm` writes data into Zookeeper so that the cluster knows the topology is active and can start emitting tuples from spouts. [code](https://github.com/apache/storm/blob/0.7.1/src/clj/backtype/storm/daemon/nimbus.clj#L504)
-
-- TODO cluster state diagram (show all nodes and what's kept everywhere)
-
-- Supervisor runs two functions in the background:
- - `synchronize-supervisor`: This is called whenever assignments in Zookeeper change and also every 10 seconds. [code](https://github.com/apache/storm/blob/0.7.1/src/clj/backtype/storm/daemon/supervisor.clj#L241)
- - Downloads code from Nimbus for topologies assigned to this machine for which it doesn't have the code yet. [code](https://github.com/apache/storm/blob/0.7.1/src/clj/backtype/storm/daemon/supervisor.clj#L258)
- - Writes into local filesystem what this node is supposed to be running. It writes a map from port -> LocalAssignment. LocalAssignment contains a topology id as well as the list of task ids for that worker. [code](https://github.com/apache/storm/blob/0.7.1/src/clj/backtype/storm/daemon/supervisor.clj#L13)
- - `sync-processes`: Reads from the LFS what `synchronize-supervisor` wrote and compares that to what's actually running on the machine. It then starts/stops worker processes as necessary to synchronize. [code](https://github.com/apache/storm/blob/0.7.1/src/clj/backtype/storm/daemon/supervisor.clj#L177)
-
-- Worker processes start up through the `mk-worker` function [code](https://github.com/apache/storm/blob/0.7.1/src/clj/backtype/storm/daemon/worker.clj#L67)
- - Worker connects to other workers and starts a thread to monitor for changes. So if a worker gets reassigned, the worker will automatically reconnect to the other worker's new location. [code](https://github.com/apache/storm/blob/0.7.1/src/clj/backtype/storm/daemon/worker.clj#L123)
- - Monitors whether a topology is active or not and stores that state in the `storm-active-atom` variable. This variable is used by tasks to determine whether or not to call `nextTuple` on the spouts. [code](https://github.com/apache/storm/blob/0.7.1/src/clj/backtype/storm/daemon/worker.clj#L155)
- - The worker launches the actual tasks as threads within it [code](https://github.com/apache/storm/blob/0.7.1/src/clj/backtype/storm/daemon/worker.clj#L178)
-- Tasks are set up through the `mk-task` function [code](https://github.com/apache/storm/blob/0.7.1/src/clj/backtype/storm/daemon/task.clj#L160)
- - Tasks set up routing function which takes in a stream and an output tuple and returns a list of task ids to send the tuple to [code](https://github.com/apache/storm/blob/0.7.1/src/clj/backtype/storm/daemon/task.clj#L207) (there's also a 3-arity version used for direct streams)
- - Tasks set up the spout-specific or bolt-specific code with [code](https://github.com/apache/storm/blob/0.7.1/src/clj/backtype/storm/daemon/task.clj#L241)
-
-## Topology Monitoring
-
-- Nimbus monitors the topology during its lifetime
- - Schedules recurring task on the timer thread to check the topologies [code](https://github.com/apache/storm/blob/0.7.1/src/clj/backtype/storm/daemon/nimbus.clj#L623)
- - Nimbus's behavior is represented as a finite state machine [code](https://github.com/apache/storm/blob/0.7.1/src/clj/backtype/storm/daemon/nimbus.clj#L98)
- - The "monitor" event is called on a topology every "nimbus.monitor.freq.secs", which calls `reassign-topology` through `reassign-transition` [code](https://github.com/apache/storm/blob/0.7.1/src/clj/backtype/storm/daemon/nimbus.clj#L497)
- - `reassign-topology` calls `mk-assignments`, the same function used to assign the topology the first time. `mk-assignments` is also capable of incrementally updating a topology
- - `mk-assignments` checks heartbeats and reassigns workers as necessary
- - Any reassignments change the state in ZK, which will trigger supervisors to synchronize and start/stop workers
-
-## Killing a topology
-
-- "storm kill" command runs this code which just calls the Nimbus Thrift interface to kill the topology: [code](https://github.com/apache/storm/blob/0.7.1/src/clj/backtype/storm/command/kill_topology.clj)
-- Nimbus receives the kill command [code](https://github.com/apache/storm/blob/0.7.1/src/clj/backtype/storm/daemon/nimbus.clj#L671)
-- Nimbus applies the "kill" transition to the topology [code](https://github.com/apache/storm/blob/0.7.1/src/clj/backtype/storm/daemon/nimbus.clj#L676)
-- The kill transition function changes the status of the topology to "killed" and schedules the "remove" event to run "wait time seconds" in the future. [code](https://github.com/apache/storm/blob/0.7.1/src/clj/backtype/storm/daemon/nimbus.clj#L63)
- - The wait time defaults to the topology message timeout but can be overridden with the -w flag in the "storm kill" command
- - This causes the topology to be deactivated for the wait time before its actually shut down. This gives the topology a chance to finish processing what it's currently processing before shutting down the workers
- - Changing the status during the kill transition ensures that the kill protocol is fault-tolerant to Nimbus crashing. On startup, if the status of the topology is "killed", Nimbus schedules the remove event to run "wait time seconds" in the future [code](https://github.com/apache/storm/blob/0.7.1/src/clj/backtype/storm/daemon/nimbus.clj#L111)
-- Removing a topology cleans out the assignment and static information from ZK [code](https://github.com/apache/storm/blob/0.7.1/src/clj/backtype/storm/daemon/nimbus.clj#L116)
-- A separate cleanup thread runs the `do-cleanup` function which will clean up the heartbeat dir and the jars/configs stored locally. [code](https://github.com/apache/storm/blob/0.7.1/src/clj/backtype/storm/daemon/nimbus.clj#L577)
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/storm/blob/93e0d028/docs/documentation/Local-mode.md
----------------------------------------------------------------------
diff --git a/docs/documentation/Local-mode.md b/docs/documentation/Local-mode.md
deleted file mode 100644
index 5ffd365..0000000
--- a/docs/documentation/Local-mode.md
+++ /dev/null
@@ -1,29 +0,0 @@
----
-title: Local Mode
-layout: documentation
-documentation: true
----
-Local mode simulates a Storm cluster in process and is useful for developing and testing topologies. Running topologies in local mode is similar to running topologies [on a cluster](Running-topologies-on-a-production-cluster.html).
-
-To create an in-process cluster, simply use the `LocalCluster` class. For example:
-
-```java
-import backtype.storm.LocalCluster;
-
-LocalCluster cluster = new LocalCluster();
-```
-
-You can then submit topologies using the `submitTopology` method on the `LocalCluster` object. Just like the corresponding method on [StormSubmitter](/javadoc/apidocs/backtype/storm/StormSubmitter.html), `submitTopology` takes a name, a topology configuration, and the topology object. You can then kill a topology using the `killTopology` method which takes the topology name as an argument.
-
-To shutdown a local cluster, simple call:
-
-```java
-cluster.shutdown();
-```
-
-### Common configurations for local mode
-
-You can see a full list of configurations [here](/javadoc/apidocs/backtype/storm/Config.html).
-
-1. **Config.TOPOLOGY_MAX_TASK_PARALLELISM**: This config puts a ceiling on the number of threads spawned for a single component. Oftentimes production topologies have a lot of parallelism (hundreds of threads) which places unreasonable load when trying to test the topology in local mode. This config lets you easy control that parallelism.
-2. **Config.TOPOLOGY_DEBUG**: When this is set to true, Storm will log a message every time a tuple is emitted from any spout or bolt. This is extremely useful for debugging.
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/storm/blob/93e0d028/docs/documentation/Logs.md
----------------------------------------------------------------------
diff --git a/docs/documentation/Logs.md b/docs/documentation/Logs.md
deleted file mode 100644
index 28e6693..0000000
--- a/docs/documentation/Logs.md
+++ /dev/null
@@ -1,30 +0,0 @@
----
-title: Storm Logs
-layout: documentation
-documentation: true
----
-Logs in Storm are essential for tracking the status, operations, error messages and debug information for all the
-daemons (e.g., nimbus, supervisor, logviewer, drpc, ui, pacemaker) and topologies' workers.
-
-### Location of the Logs
-All the daemon logs are placed under ${storm.log.dir} directory, which an administrator can set in the System properties or
-in the cluster configuration. By default, ${storm.log.dir} points to ${storm.home}/logs.
-
-All the worker logs are placed under the workers-artifacts directory in a hierarchical manner, e.g.,
-${workers-artifacts}/${topologyId}/${port}/worker.log. Users can set the workers-artifacts directory
-by configuring the variable "storm.workers.artifacts.dir". By default, workers-artifacts directory
-locates at ${storm.log.dir}/logs/workers-artifacts.
-
-### Using the Storm UI for Log View/Download and Log Search
-Daemon and worker logs are allowed to view and download through Storm UI by authorized users.
-
-To improve the debugging of Storm, we provide the Log Search feature.
-Log Search supports searching in a certain log file or in all of a topology's log files:
-
-String search in a log file: In the log page for a worker, a user can search a certain string, e.g., "Exception", in a certain worker log. This search can happen for both normal text log or rolled zip log files. In the results, the offset and matched lines will be displayed.
-
-
-
-Search in a topology: a user can also search a string for a certain topology by clicking the icon of magnifying lens at the top right corner of the UI page. This means the UI will try to search on all the supervisor nodes in a distributed way to find the matched string in all logs for this topology. The search can happen for either normal text log files or rolled zip log files by checking/unchecking the "Search archived logs:" box. Then the matched results can be shown on the UI with url links, directing the user to the certain logs on each supervisor node. This powerful feature is very helpful for users to find certain problematic supervisor nodes running this topology.
-
-
http://git-wip-us.apache.org/repos/asf/storm/blob/93e0d028/docs/documentation/Maven.md
----------------------------------------------------------------------
diff --git a/docs/documentation/Maven.md b/docs/documentation/Maven.md
deleted file mode 100644
index 83e1d1c..0000000
--- a/docs/documentation/Maven.md
+++ /dev/null
@@ -1,20 +0,0 @@
----
-layout: documentation
----
-To develop topologies, you'll need the Storm jars on your classpath. You should either include the unpacked jars in the classpath for your project or use Maven to include Storm as a development dependency. Storm is hosted on Maven Central. To include Storm in your project as a development dependency, add the following to your pom.xml:
-
-
-```xml
-<dependency>
- <groupId>org.apache.storm</groupId>
- <artifactId>storm-core</artifactId>
- <version>0.9.5</version>
- <scope>provided</scope>
-</dependency>
-```
-
-[Here's an example](https://github.com/apache/storm/blob/master/examples/storm-starter/pom.xml) of a pom.xml for a Storm project.
-
-### Developing Storm
-
-Please refer to [DEVELOPER.md](https://github.com/apache/storm/blob/master/DEVELOPER.md) for more details.
http://git-wip-us.apache.org/repos/asf/storm/blob/93e0d028/docs/documentation/Message-passing-implementation.md
----------------------------------------------------------------------
diff --git a/docs/documentation/Message-passing-implementation.md b/docs/documentation/Message-passing-implementation.md
deleted file mode 100644
index a17f66a..0000000
--- a/docs/documentation/Message-passing-implementation.md
+++ /dev/null
@@ -1,30 +0,0 @@
----
-title: Message Passing Implementation
-layout: documentation
-documentation: true
----
-(Note: this walkthrough is out of date as of 0.8.0. 0.8.0 revamped the message passing infrastructure to be based on the Disruptor)
-
-This page walks through how emitting and transferring tuples works in Storm.
-
-- Worker is responsible for message transfer
- - `refresh-connections` is called every "task.refresh.poll.secs" or whenever assignment in ZK changes. It manages connections to other workers and maintains a mapping from task -> worker [code](https://github.com/apache/storm/blob/0.7.1/src/clj/backtype/storm/daemon/worker.clj#L123)
- - Provides a "transfer function" that is used by tasks to send tuples to other tasks. The transfer function takes in a task id and a tuple, and it serializes the tuple and puts it onto a "transfer queue". There is a single transfer queue for each worker. [code](https://github.com/apache/storm/blob/0.7.1/src/clj/backtype/storm/daemon/worker.clj#L56)
- - The serializer is thread-safe [code](https://github.com/apache/storm/blob/0.7.1/src/jvm/backtype/storm/serialization/KryoTupleSerializer.java#L26)
- - The worker has a single thread which drains the transfer queue and sends the messages to other workers [code](https://github.com/apache/storm/blob/0.7.1/src/clj/backtype/storm/daemon/worker.clj#L185)
- - Message sending happens through this protocol: [code](https://github.com/apache/storm/blob/0.7.1/src/clj/backtype/storm/messaging/protocol.clj)
- - The implementation for distributed mode uses ZeroMQ [code](https://github.com/apache/storm/blob/0.7.1/src/clj/backtype/storm/messaging/zmq.clj)
- - The implementation for local mode uses in memory Java queues (so that it's easy to use Storm locally without needing to get ZeroMQ installed) [code](https://github.com/apache/storm/blob/0.7.1/src/clj/backtype/storm/messaging/local.clj)
-- Receiving messages in tasks works differently in local mode and distributed mode
- - In local mode, the tuple is sent directly to an in-memory queue for the receiving task [code](https://github.com/apache/storm/blob/0.7.1/src/clj/backtype/storm/messaging/local.clj#L21)
- - In distributed mode, each worker listens on a single TCP port for incoming messages and then routes those messages in-memory to tasks. The TCP port is called a "virtual port", because it receives [task id, message] and then routes it to the actual task. [code](https://github.com/apache/storm/blob/0.7.1/src/clj/backtype/storm/daemon/worker.clj#L204)
- - The virtual port implementation is here: [code](https://github.com/apache/storm/blob/0.7.1/src/clj/zilch/virtual_port.clj)
- - Tasks listen on an in-memory ZeroMQ port for messages from the virtual port [code](https://github.com/apache/storm/blob/0.7.1/src/clj/backtype/storm/daemon/task.clj#L201)
- - Bolts listen here: [code](https://github.com/apache/storm/blob/0.7.1/src/clj/backtype/storm/daemon/task.clj#L489)
- - Spouts listen here: [code](https://github.com/apache/storm/blob/0.7.1/src/clj/backtype/storm/daemon/task.clj#L382)
-- Tasks are responsible for message routing. A tuple is emitted either to a direct stream (where the task id is specified) or a regular stream. In direct streams, the message is only sent if that bolt subscribes to that direct stream. In regular streams, the stream grouping functions are used to determine the task ids to send the tuple to.
- - Tasks have a routing map from {stream id} -> {component id} -> {stream grouping function} [code](https://github.com/apache/storm/blob/0.7.1/src/clj/backtype/storm/daemon/task.clj#L198)
- - The "tasks-fn" returns the task ids to send the tuples to for either regular stream emit or direct stream emit [code](https://github.com/apache/storm/blob/0.7.1/src/clj/backtype/storm/daemon/task.clj#L207)
- - After getting the output task ids, bolts and spouts use the transfer-fn provided by the worker to actually transfer the tuples
- - Bolt transfer code here: [code](https://github.com/apache/storm/blob/0.7.1/src/clj/backtype/storm/daemon/task.clj#L429)
- - Spout transfer code here: [code](https://github.com/apache/storm/blob/0.7.1/src/clj/backtype/storm/daemon/task.clj#L329)
http://git-wip-us.apache.org/repos/asf/storm/blob/93e0d028/docs/documentation/Metrics.md
----------------------------------------------------------------------
diff --git a/docs/documentation/Metrics.md b/docs/documentation/Metrics.md
deleted file mode 100644
index a84649d..0000000
--- a/docs/documentation/Metrics.md
+++ /dev/null
@@ -1,36 +0,0 @@
----
-title: Storm Metrics
-layout: documentation
-documentation: true
----
-Storm exposes a metrics interface to report summary statistics across the full topology.
-It's used internally to track the numbers you see in the Nimbus UI console: counts of executes and acks; average process latency per bolt; worker heap usage; and so forth.
-
-### Metric Types
-
-Metrics have to implement just one method, `getValueAndReset` -- do any remaining work to find the summary value, and reset back to an initial state. For example, the MeanReducer divides the running total by its running count to find the mean, then initializes both values back to zero.
-
-Storm gives you these metric types:
-
-* [AssignableMetric](https://github.com/apache/storm/blob/master/storm-core/src/jvm/backtype/storm/metric/api/AssignableMetric.java) -- set the metric to the explicit value you supply. Useful if it's an external value or in the case that you are already calculating the summary statistic yourself.
-* [CombinedMetric](https://github.com/apache/storm/blob/master/storm-core/src/jvm/backtype/storm/metric/api/CombinedMetric.java) -- generic interface for metrics that can be updated associatively.
-* [CountMetric](https://github.com/apache/storm/blob/master/storm-core/src/jvm/backtype/storm/metric/api/CountMetric.java) -- a running total of the supplied values. Call `incr()` to increment by one, `incrBy(n)` to add/subtract the given number.
- - [MultiCountMetric](https://github.com/apache/storm/blob/master/storm-core/src/jvm/backtype/storm/metric/api/MultiCountMetric.java) -- a hashmap of count metrics.
-* [ReducedMetric](https://github.com/apache/storm/blob/master/storm-core/src/jvm/backtype/storm/metric/api/ReducedMetric.java)
- - [MeanReducer](https://github.com/apache/storm/blob/master/storm-core/src/jvm/backtype/storm/metric/api/MeanReducer.java) -- track a running average of values given to its `reduce()` method. (It accepts `Double`, `Integer` or `Long` values, and maintains the internal average as a `Double`.) Despite his reputation, the MeanReducer is actually a pretty nice guy in person.
- - [MultiReducedMetric](https://github.com/apache/storm/blob/master/storm-core/src/jvm/backtype/storm/metric/api/MultiReducedMetric.java) -- a hashmap of reduced metrics.
-
-
-### Metric Consumer
-
-
-### Build your own metric
-
-
-
-### Builtin Metrics
-
-The [builtin metrics](https://github.com/apache/storm/blob/46c3ba7/storm-core/src/clj/backtype/storm/daemon/builtin_metrics.clj) instrument Storm itself.
-
-[builtin_metrics.clj](https://github.com/apache/storm/blob/46c3ba7/storm-core/src/clj/backtype/storm/daemon/builtin_metrics.clj) sets up data structures for the built-in metrics, and facade methods that the other framework components can use to update them. The metrics themselves are calculated in the calling code -- see for example [`ack-spout-msg`](https://github.com/apache/storm/blob/46c3ba7/storm-core/src/clj/backtype/storm/daemon/executor.clj#358) in `clj/b/s/daemon/daemon/executor.clj`
-
http://git-wip-us.apache.org/repos/asf/storm/blob/93e0d028/docs/documentation/Multilang-protocol.md
----------------------------------------------------------------------
diff --git a/docs/documentation/Multilang-protocol.md b/docs/documentation/Multilang-protocol.md
deleted file mode 100644
index 2a90059..0000000
--- a/docs/documentation/Multilang-protocol.md
+++ /dev/null
@@ -1,287 +0,0 @@
----
-title: Multi-Lang Protocol
-layout: documentation
-documentation: true
----
-This page explains the multilang protocol as of Storm 0.7.1. Versions prior to 0.7.1 used a somewhat different protocol, documented [here](Storm-multi-language-protocol-(versions-0.7.0-and-below\).html).
-
-# Storm Multi-Language Protocol
-
-## Shell Components
-
-Support for multiple languages is implemented via the ShellBolt,
-ShellSpout, and ShellProcess classes. These classes implement the
-IBolt and ISpout interfaces and the protocol for executing a script or
-program via the shell using Java's ProcessBuilder class.
-
-## Output fields
-
-Output fields are part of the Thrift definition of the topology. This means that when you multilang in Java, you need to create a bolt that extends ShellBolt, implements IRichBolt, and declare the fields in `declareOutputFields` (similarly for ShellSpout).
-
-You can learn more about this on [Concepts](Concepts.html)
-
-## Protocol Preamble
-
-A simple protocol is implemented via the STDIN and STDOUT of the
-executed script or program. All data exchanged with the process is
-encoded in JSON, making support possible for pretty much any language.
-
-# Packaging Your Stuff
-
-To run a shell component on a cluster, the scripts that are shelled
-out to must be in the `resources/` directory within the jar submitted
-to the master.
-
-However, during development or testing on a local machine, the resources
-directory just needs to be on the classpath.
-
-## The Protocol
-
-Notes:
-
-* Both ends of this protocol use a line-reading mechanism, so be sure to
-trim off newlines from the input and to append them to your output.
-* All JSON inputs and outputs are terminated by a single line containing "end". Note that this delimiter is not itself JSON encoded.
-* The bullet points below are written from the perspective of the script writer's
-STDIN and STDOUT.
-
-### Initial Handshake
-
-The initial handshake is the same for both types of shell components:
-
-* STDIN: Setup info. This is a JSON object with the Storm configuration, a PID directory, and a topology context, like this:
-
-```
-{
- "conf": {
- "topology.message.timeout.secs": 3,
- // etc
- },
- "pidDir": "...",
- "context": {
- "task->component": {
- "1": "example-spout",
- "2": "__acker",
- "3": "example-bolt1",
- "4": "example-bolt2"
- },
- "taskid": 3,
- // Everything below this line is only available in Storm 0.10.0+
- "componentid": "example-bolt"
- "stream->target->grouping": {
- "default": {
- "example-bolt2": {
- "type": "SHUFFLE"}}},
- "streams": ["default"],
- "stream->outputfields": {"default": ["word"]},
- "source->stream->grouping": {
- "example-spout": {
- "default": {
- "type": "FIELDS",
- "fields": ["word"]
- }
- }
- }
- "source->stream->fields": {
- "example-spout": {
- "default": ["word"]
- }
- }
- }
-}
-```
-
-Your script should create an empty file named with its PID in this directory. e.g.
-the PID is 1234, so an empty file named 1234 is created in the directory. This
-file lets the supervisor know the PID so it can shutdown the process later on.
-
-As of Storm 0.10.0, the context sent by Storm to shell components has been
-enhanced substantially to include all aspects of the topology context available
-to JVM components. One key addition is the ability to determine a shell
-component's source and targets (i.e., inputs and outputs) in the topology via
-the `stream->target->grouping` and `source->stream->grouping` dictionaries. At
-the innermost level of these nested dictionaries, groupings are represented as
-a dictionary that minimally has a `type` key, but can also have a `fields` key
-to specify which fields are involved in a `FIELDS` grouping.
-
-* STDOUT: Your PID, in a JSON object, like `{"pid": 1234}`. The shell component will log the PID to its log.
-
-What happens next depends on the type of component:
-
-### Spouts
-
-Shell spouts are synchronous. The rest happens in a while(true) loop:
-
-* STDIN: Either a next, ack, or fail command.
-
-"next" is the equivalent of ISpout's `nextTuple`. It looks like:
-
-```
-{"command": "next"}
-```
-
-"ack" looks like:
-
-```
-{"command": "ack", "id": "1231231"}
-```
-
-"fail" looks like:
-
-```
-{"command": "fail", "id": "1231231"}
-```
-
-* STDOUT: The results of your spout for the previous command. This can
- be a sequence of emits and logs.
-
-An emit looks like:
-
-```
-{
- "command": "emit",
- // The id for the tuple. Leave this out for an unreliable emit. The id can
- // be a string or a number.
- "id": "1231231",
- // The id of the stream this tuple was emitted to. Leave this empty to emit to default stream.
- "stream": "1",
- // If doing an emit direct, indicate the task to send the tuple to
- "task": 9,
- // All the values in this tuple
- "tuple": ["field1", 2, 3]
-}
-```
-
-If not doing an emit direct, you will immediately receive the task ids to which the tuple was emitted on STDIN as a JSON array.
-
-A "log" will log a message in the worker log. It looks like:
-
-```
-{
- "command": "log",
- // the message to log
- "msg": "hello world!"
-}
-```
-
-* STDOUT: a "sync" command ends the sequence of emits and logs. It looks like:
-
-```
-{"command": "sync"}
-```
-
-After you sync, ShellSpout will not read your output until it sends another next, ack, or fail command.
-
-Note that, similarly to ISpout, all of the spouts in the worker will be locked up after a next, ack, or fail, until you sync. Also like ISpout, if you have no tuples to emit for a next, you should sleep for a small amount of time before syncing. ShellSpout will not automatically sleep for you.
-
-
-### Bolts
-
-The shell bolt protocol is asynchronous. You will receive tuples on STDIN as soon as they are available, and you may emit, ack, and fail, and log at any time by writing to STDOUT, as follows:
-
-* STDIN: A tuple! This is a JSON encoded structure like this:
-
-```
-{
- // The tuple's id - this is a string to support languages lacking 64-bit precision
- "id": "-6955786537413359385",
- // The id of the component that created this tuple
- "comp": "1",
- // The id of the stream this tuple was emitted to
- "stream": "1",
- // The id of the task that created this tuple
- "task": 9,
- // All the values in this tuple
- "tuple": ["snow white and the seven dwarfs", "field2", 3]
-}
-```
-
-* STDOUT: An ack, fail, emit, or log. Emits look like:
-
-```
-{
- "command": "emit",
- // The ids of the tuples this output tuples should be anchored to
- "anchors": ["1231231", "-234234234"],
- // The id of the stream this tuple was emitted to. Leave this empty to emit to default stream.
- "stream": "1",
- // If doing an emit direct, indicate the task to send the tuple to
- "task": 9,
- // All the values in this tuple
- "tuple": ["field1", 2, 3]
-}
-```
-
-If not doing an emit direct, you will receive the task ids to which
-the tuple was emitted on STDIN as a JSON array. Note that, due to the
-asynchronous nature of the shell bolt protocol, when you read after
-emitting, you may not receive the task ids. You may instead read the
-task ids for a previous emit or a new tuple to process. You will
-receive the task id lists in the same order as their corresponding
-emits, however.
-
-An ack looks like:
-
-```
-{
- "command": "ack",
- // the id of the tuple to ack
- "id": "123123"
-}
-```
-
-A fail looks like:
-
-```
-{
- "command": "fail",
- // the id of the tuple to fail
- "id": "123123"
-}
-```
-
-A "log" will log a message in the worker log. It looks like:
-
-```
-{
- "command": "log",
- // the message to log
- "msg": "hello world!"
-}
-```
-
-* Note that, as of version 0.7.1, there is no longer any need for a
- shell bolt to 'sync'.
-
-### Handling Heartbeats (0.9.3 and later)
-
-As of Storm 0.9.3, heartbeats have been between ShellSpout/ShellBolt and their
-multi-lang subprocesses to detect hanging/zombie subprocesses. Any libraries
-for interfacing with Storm via multi-lang must take the following actions
-regarding hearbeats:
-
-#### Spout
-
-Shell spouts are synchronous, so subprocesses always send `sync` commands at the
-end of `next()`, so you should not have to do much to support heartbeats for
-spouts. That said, you must not let subprocesses sleep more than the worker
-timeout during `next()`.
-
-#### Bolt
-
-Shell bolts are asynchronous, so a ShellBolt will send heartbeat tuples to its
-subprocess periodically. Heartbeat tuple looks like:
-
-```
-{
- "id": "-6955786537413359385",
- "comp": "1",
- "stream": "__heartbeat",
- // this shell bolt's system task id
- "task": -1,
- "tuple": []
-}
-```
-
-When subprocess receives heartbeat tuple, it must send a `sync` command back to
-ShellBolt.
http://git-wip-us.apache.org/repos/asf/storm/blob/93e0d028/docs/documentation/Pacemaker.md
----------------------------------------------------------------------
diff --git a/docs/documentation/Pacemaker.md b/docs/documentation/Pacemaker.md
deleted file mode 100644
index 24e3e7f..0000000
--- a/docs/documentation/Pacemaker.md
+++ /dev/null
@@ -1,108 +0,0 @@
-# Pacemaker
-
-### Introduction
-Pacemaker is a storm daemon designed to process heartbeats from workers. As Storm is scaled up, ZooKeeper begins to become a bottleneck due to high volumes of writes from workers doing heartbeats. Lots of writes to disk and too much traffic across the network is generated as ZooKeeper tries to maintain consistency.
-
-Because heartbeats are of an ephemeral nature, they do not need to be persisted to disk or synced across nodes; an in-memory store will do. This is the role of Pacemaker. Pacemaker functions as a simple in-memory key/value store with ZooKeeper-like, directory-style keys and byte array values.
-
-The corresponding Pacemaker client is a plugin for the `ClusterState` interface, `org.apache.storm.pacemaker.pacemaker_state_factory`. Heartbeat calls are funneled by the `ClusterState` produced by `pacemaker_state_factory` into the Pacemaker daemon, while other set/get operations are forwarded to ZooKeeper.
-
-------
-
-### Configuration
-
- - `pacemaker.host` : The host that the Pacemaker daemon is running on
- - `pacemaker.port` : The port that Pacemaker will listen on
- - `pacemaker.max.threads` : Maximum number of threads Pacemaker daemon will use to handle requests.
- - `pacemaker.childopts` : Any JVM parameters that need to go to the Pacemaker. (used by storm-deploy project)
- - `pacemaker.auth.method` : The authentication method that is used (more info below)
-
-#### Example
-
-To get Pacemaker up and running, set the following option in the cluster config on all nodes:
-```
-storm.cluster.state.store: "org.apache.storm.pacemaker.pacemaker_state_factory"
-```
-
-The Pacemaker host also needs to be set on all nodes:
-```
-pacemaker.host: somehost.mycompany.com
-```
-
-And then start all of your daemons
-
-(including Pacemaker):
-```
-$ storm pacemaker
-```
-
-The Storm cluster should now be pushing all worker heartbeats through Pacemaker.
-
-### Security
-
-Currently digest (password-based) and Kerberos security are supported. Security is currently only around reads, not writes. Writes may be performed by anyone, whereas reads may only be performed by authorized and authenticated users. This is an area for future development, as it leaves the cluster open to DoS attacks, but it prevents any sensitive information from reaching unauthorized eyes, which was the main goal.
-
-#### Digest
-To configure digest authentication, set `pacemaker.auth.method: DIGEST` in the cluster config on the nodes hosting Nimbus and Pacemaker.
-The nodes must also have `java.security.auth.login.config` set to point to a JAAS config file containing the following structure:
-```
-PacemakerDigest {
- username="some username"
- password="some password";
-};
-```
-
-Any node with these settings configured will be able to read from Pacemaker.
-Worker nodes need not have these configs set, and may keep `pacemaker.auth.method: NONE` set, since they do not need to read from the Pacemaker daemon.
-
-#### Kerberos
-To configure Kerberos authentication, set `pacemaker.auth.method: KERBEROS` in the cluster config on the nodes hosting Nimbus and Pacemaker.
-The nodes must also have `java.security.auth.login.config` set to point to a JAAS config.
-
-The JAAS config on Nimbus must look something like this:
-```
-PacemakerClient {
- com.sun.security.auth.module.Krb5LoginModule required
- useKeyTab=true
- keyTab="/etc/keytabs/nimbus.keytab"
- storeKey=true
- useTicketCache=false
- serviceName="pacemaker"
- principal="nimbus@MY.COMPANY.COM";
-};
-
-```
-
-The JAAS config on Pacemaker must look something like this:
-```
-PacemakerServer {
- com.sun.security.auth.module.Krb5LoginModule required
- useKeyTab=true
- keyTab="/etc/keytabs/pacemaker.keytab"
- storeKey=true
- useTicketCache=false
- principal="pacemaker@MY.COMPANY.COM";
-};
-```
-
- - The client's user principal in the `PacemakerClient` section on the Nimbus host must match the `nimbus.daemon.user` storm cluster config value.
- - The client's `serviceName` value must match the server's user principal in the `PacemakerServer` section on the Pacemaker host.
-
-
-### Fault Tolerance
-
-Pacemaker runs as a single daemon instance, making it a potential Single Point of Failure.
-
-If Pacemaker becomes unreachable by Nimbus, through crash or network partition, the workers will continue to run, and Nimbus will repeatedly attempt to reconnect. Nimbus functionality will be disrupted, but the topologies themselves will continue to run.
-In case of partition of the cluster where Nimbus and Pacemaker are on the same side of the partition, the workers that are on the other side of the partition will not be able to heartbeat, and Nimbus will reschedule the tasks elsewhere. This is probably what we want to happen anyway.
-
-
-### ZooKeeper Comparison
-Compared to ZooKeeper, Pacemaker uses less CPU, less memory, and of course no disk for the same load, thanks to lack of overhead from maintaining consistency between nodes.
-On Gigabit networking, there is a theoretical limit of about 6000 nodes. However, the real limit is likely around 2000-3000 nodes. These limits have not yet been tested.
-On a 270 supervisor cluster, fully scheduled with topologies, Pacemaker resource utilization was 70% of one core and nearly 1GiB of RAM on a machine with 4 `Intel(R) Xeon(R) CPU E5530 @ 2.40GHz` and 24GiB of RAM.
-
-
-There is an easy route to HA for Pacemaker. Unlike ZooKeeper, Pacemaker should be able to scale horizontally without overhead. By contrast, with ZooKeeper, there are diminishing returns when adding ZK nodes.
-
-In short, a single Pacemaker node should be able to handle many times the load that a ZooKeeper cluster can, and future HA work allowing horizontal scaling will increase that even farther.