You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@storm.apache.org by kn...@apache.org on 2016/03/28 18:43:33 UTC
[2/2] storm git commit: Adding documentation for Trident RAS API
Adding documentation for Trident RAS API
Signed-off-by: Kyle Nusbaum <Ky...@gmail.com>
Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/8fd4fbcd
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/8fd4fbcd
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/8fd4fbcd
Branch: refs/heads/1.x-branch
Commit: 8fd4fbcd1371c2da93e98152f7ce871ee2bfbff8
Parents: b0db246
Author: Kyle Nusbaum <Ky...@gmail.com>
Authored: Thu Mar 24 14:42:17 2016 -0500
Committer: Kyle Nusbaum <Ky...@gmail.com>
Committed: Mon Mar 28 11:43:20 2016 -0500
----------------------------------------------------------------------
docs/Resource_Aware_Scheduler_overview.md | 2 ++
docs/Trident-RAS-API.md | 49 ++++++++++++++++++++++++++
docs/index.md | 1 +
3 files changed, 52 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/storm/blob/8fd4fbcd/docs/Resource_Aware_Scheduler_overview.md
----------------------------------------------------------------------
diff --git a/docs/Resource_Aware_Scheduler_overview.md b/docs/Resource_Aware_Scheduler_overview.md
index 591b37c..30e0eeb 100644
--- a/docs/Resource_Aware_Scheduler_overview.md
+++ b/docs/Resource_Aware_Scheduler_overview.md
@@ -16,6 +16,8 @@ The user can switch to using the Resource Aware Scheduler by setting the followi
## API Overview
+For use with Trident, please see the [Trident RAS API](Trident-RAS-API.html)
+
For a Storm Topology, the user can now specify the amount of resources a topology component (i.e. Spout or Bolt) is required to run a single instance of the component. The user can specify the resource requirement for a topology component by using the following API calls.
### Setting Memory Requirement
http://git-wip-us.apache.org/repos/asf/storm/blob/8fd4fbcd/docs/Trident-RAS-API.md
----------------------------------------------------------------------
diff --git a/docs/Trident-RAS-API.md b/docs/Trident-RAS-API.md
new file mode 100644
index 0000000..30a3889
--- /dev/null
+++ b/docs/Trident-RAS-API.md
@@ -0,0 +1,49 @@
+---
+title: Trident RAS API
+layout: documentation
+documentation: true
+---
+
+## Trident RAS API
+
+The Trident RAS (Resource Aware Scheduler) API provides a mechanism to specify the resource consumption of their topology. The API looks exactly like the base RAS API, only it is called on Trident Streams instead of Bolts and Spouts.
+
+In order to avoid duplication and inconsistency in documentation, the purpose and effects of resource setting are not described here, but are instead found in the [Resource Aware Scheduler Overview](Resource_Aware_Scheduler_overview.html)
+
+### Use
+
+First an example:
+
+```java
+ TridentTopology topo = new TridentTopology();
+ TridentState wordCounts =
+ topology
+ .newStream("words", feeder)
+ .parallelismHint(5)
+ .setCPULoad(20)
+ .setMemoryLoad(512,256)
+ .each( new Fields("sentence"), new Split(), new Fields("word"))
+ .setCPULoad(10)
+ .setMemoryLoad(512)
+ .each(new Fields("word"), new BangAdder(), new Fields("word!"))
+ .parallelismHint(10)
+ .setCPULoad(50)
+ .setMemoryLoad(1024)
+ .groupBy(new Fields("word!"))
+ .persistentAggregate(new MemoryMapState.Factory(), new Count(), new Fields("count"))
+ .setCPULoad(100)
+ .setMemoryLoad(2048);
+```
+
+Resources can be set per operation (except for grouping, shuffling, partitioning).
+Operations that are combined by Trident into single Bolts have their resources summed.
+
+Every Bolt is given **at least** the default resources, regardless of user settings.
+
+In the above case, we end up with
+ * a spout and spout coordinator with a CPU load of 20% each, and a memory load of 512MiB on heap and 256MiB off heap.
+ * a bolt with 60% cpu load (10% + 50%) and a memory load of 1536MiB (1024 + 512) on heap from the combined `Split` and `BangAdder`
+ * a bolt with 100% cpu load and a memory load of 2048MiB.
+
+The methods can be called for every operation (or some of the operations) or used in the same manner as `parallelismHint()`.
+That is, resource declarations have the same *boundaries* as parallelismHints. They don't cross any groupings, shufflings, or any other kind of repartitioning.
http://git-wip-us.apache.org/repos/asf/storm/blob/8fd4fbcd/docs/index.md
----------------------------------------------------------------------
diff --git a/docs/index.md b/docs/index.md
index bfd68db..a3be26c 100644
--- a/docs/index.md
+++ b/docs/index.md
@@ -28,6 +28,7 @@ Trident is an alternative interface to Storm. It provides exactly-once processin
* [Trident API Overview](Trident-API-Overview.html) -- operations for transforming and orchestrating data
* [Trident State](Trident-state.html) -- exactly-once processing and fast, persistent aggregation
* [Trident spouts](Trident-spouts.html) -- transactional and non-transactional data intake
+* [Trident RAS API](Tridnet-RAS-API.html) -- using the Resource Aware Scheduler with Trident.
### Setup and Deploying