You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@storm.apache.org by kn...@apache.org on 2016/03/28 18:43:32 UTC
[1/2] storm git commit: Minor fix to RAS doc.
Repository: storm
Updated Branches:
refs/heads/1.x-branch b0db246ad -> fd2aefd4b
Minor fix to RAS doc.
Update Trident-RAS-API.md
Signed-off-by: Kyle Nusbaum <Ky...@gmail.com>
Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/fd2aefd4
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/fd2aefd4
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/fd2aefd4
Branch: refs/heads/1.x-branch
Commit: fd2aefd4b5fd2f3a106d6287438b8ef6e78a8591
Parents: 8fd4fbc
Author: Kyle Nusbaum <Ky...@gmail.com>
Authored: Mon Mar 28 11:21:31 2016 -0500
Committer: Kyle Nusbaum <Ky...@gmail.com>
Committed: Mon Mar 28 11:43:20 2016 -0500
----------------------------------------------------------------------
docs/Trident-RAS-API.md | 19 ++++++++++---------
1 file changed, 10 insertions(+), 9 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/storm/blob/fd2aefd4/docs/Trident-RAS-API.md
----------------------------------------------------------------------
diff --git a/docs/Trident-RAS-API.md b/docs/Trident-RAS-API.md
index 30a3889..7d8a1be 100644
--- a/docs/Trident-RAS-API.md
+++ b/docs/Trident-RAS-API.md
@@ -6,13 +6,13 @@ documentation: true
## Trident RAS API
-The Trident RAS (Resource Aware Scheduler) API provides a mechanism to specify the resource consumption of their topology. The API looks exactly like the base RAS API, only it is called on Trident Streams instead of Bolts and Spouts.
+The Trident RAS (Resource Aware Scheduler) API provides a mechanism to allow users to specify the resource consumption of a Trident topology. The API looks exactly like the base RAS API, only it is called on Trident Streams instead of Bolts and Spouts.
In order to avoid duplication and inconsistency in documentation, the purpose and effects of resource setting are not described here, but are instead found in the [Resource Aware Scheduler Overview](Resource_Aware_Scheduler_overview.html)
### Use
-First an example:
+First, an example:
```java
TridentTopology topo = new TridentTopology();
@@ -35,15 +35,16 @@ First an example:
.setMemoryLoad(2048);
```
-Resources can be set per operation (except for grouping, shuffling, partitioning).
-Operations that are combined by Trident into single Bolts have their resources summed.
+Resources can be set for each operation (except for grouping, shuffling, partitioning).
+Operations that are combined by Trident into single Bolts will have their resources summed.
Every Bolt is given **at least** the default resources, regardless of user settings.
In the above case, we end up with
- * a spout and spout coordinator with a CPU load of 20% each, and a memory load of 512MiB on heap and 256MiB off heap.
- * a bolt with 60% cpu load (10% + 50%) and a memory load of 1536MiB (1024 + 512) on heap from the combined `Split` and `BangAdder`
- * a bolt with 100% cpu load and a memory load of 2048MiB.
+ * a spout and spout coordinator with a CPU load of 20% each, and a memory load of 512MiB on-heap and 256MiB off-heap.
+ * a bolt with 60% cpu load (10% + 50%) and a memory load of 1536MiB (1024 + 512) on-heap from the combined `Split` and `BangAdder`
+ * a bolt with 100% cpu load and a memory load of 2048MiB on-heap, with default value for off-heap.
-The methods can be called for every operation (or some of the operations) or used in the same manner as `parallelismHint()`.
-That is, resource declarations have the same *boundaries* as parallelismHints. They don't cross any groupings, shufflings, or any other kind of repartitioning.
+The API can be called as many times as is desired.
+It may be called after every operation, after some of the operations, or used in the same manner as `parallelismHint()` to set resources for a whole section.
+Resource declarations have the same *boundaries* as parallelism hints. They don't cross any groupings, shufflings, or any other kind of repartitioning.
[2/2] storm git commit: Adding documentation for Trident RAS API
Posted by kn...@apache.org.
Adding documentation for Trident RAS API
Signed-off-by: Kyle Nusbaum <Ky...@gmail.com>
Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/8fd4fbcd
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/8fd4fbcd
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/8fd4fbcd
Branch: refs/heads/1.x-branch
Commit: 8fd4fbcd1371c2da93e98152f7ce871ee2bfbff8
Parents: b0db246
Author: Kyle Nusbaum <Ky...@gmail.com>
Authored: Thu Mar 24 14:42:17 2016 -0500
Committer: Kyle Nusbaum <Ky...@gmail.com>
Committed: Mon Mar 28 11:43:20 2016 -0500
----------------------------------------------------------------------
docs/Resource_Aware_Scheduler_overview.md | 2 ++
docs/Trident-RAS-API.md | 49 ++++++++++++++++++++++++++
docs/index.md | 1 +
3 files changed, 52 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/storm/blob/8fd4fbcd/docs/Resource_Aware_Scheduler_overview.md
----------------------------------------------------------------------
diff --git a/docs/Resource_Aware_Scheduler_overview.md b/docs/Resource_Aware_Scheduler_overview.md
index 591b37c..30e0eeb 100644
--- a/docs/Resource_Aware_Scheduler_overview.md
+++ b/docs/Resource_Aware_Scheduler_overview.md
@@ -16,6 +16,8 @@ The user can switch to using the Resource Aware Scheduler by setting the followi
## API Overview
+For use with Trident, please see the [Trident RAS API](Trident-RAS-API.html)
+
For a Storm Topology, the user can now specify the amount of resources a topology component (i.e. Spout or Bolt) is required to run a single instance of the component. The user can specify the resource requirement for a topology component by using the following API calls.
### Setting Memory Requirement
http://git-wip-us.apache.org/repos/asf/storm/blob/8fd4fbcd/docs/Trident-RAS-API.md
----------------------------------------------------------------------
diff --git a/docs/Trident-RAS-API.md b/docs/Trident-RAS-API.md
new file mode 100644
index 0000000..30a3889
--- /dev/null
+++ b/docs/Trident-RAS-API.md
@@ -0,0 +1,49 @@
+---
+title: Trident RAS API
+layout: documentation
+documentation: true
+---
+
+## Trident RAS API
+
+The Trident RAS (Resource Aware Scheduler) API provides a mechanism to specify the resource consumption of their topology. The API looks exactly like the base RAS API, only it is called on Trident Streams instead of Bolts and Spouts.
+
+In order to avoid duplication and inconsistency in documentation, the purpose and effects of resource setting are not described here, but are instead found in the [Resource Aware Scheduler Overview](Resource_Aware_Scheduler_overview.html)
+
+### Use
+
+First an example:
+
+```java
+ TridentTopology topo = new TridentTopology();
+ TridentState wordCounts =
+ topology
+ .newStream("words", feeder)
+ .parallelismHint(5)
+ .setCPULoad(20)
+ .setMemoryLoad(512,256)
+ .each( new Fields("sentence"), new Split(), new Fields("word"))
+ .setCPULoad(10)
+ .setMemoryLoad(512)
+ .each(new Fields("word"), new BangAdder(), new Fields("word!"))
+ .parallelismHint(10)
+ .setCPULoad(50)
+ .setMemoryLoad(1024)
+ .groupBy(new Fields("word!"))
+ .persistentAggregate(new MemoryMapState.Factory(), new Count(), new Fields("count"))
+ .setCPULoad(100)
+ .setMemoryLoad(2048);
+```
+
+Resources can be set per operation (except for grouping, shuffling, partitioning).
+Operations that are combined by Trident into single Bolts have their resources summed.
+
+Every Bolt is given **at least** the default resources, regardless of user settings.
+
+In the above case, we end up with
+ * a spout and spout coordinator with a CPU load of 20% each, and a memory load of 512MiB on heap and 256MiB off heap.
+ * a bolt with 60% cpu load (10% + 50%) and a memory load of 1536MiB (1024 + 512) on heap from the combined `Split` and `BangAdder`
+ * a bolt with 100% cpu load and a memory load of 2048MiB.
+
+The methods can be called for every operation (or some of the operations) or used in the same manner as `parallelismHint()`.
+That is, resource declarations have the same *boundaries* as parallelismHints. They don't cross any groupings, shufflings, or any other kind of repartitioning.
http://git-wip-us.apache.org/repos/asf/storm/blob/8fd4fbcd/docs/index.md
----------------------------------------------------------------------
diff --git a/docs/index.md b/docs/index.md
index bfd68db..a3be26c 100644
--- a/docs/index.md
+++ b/docs/index.md
@@ -28,6 +28,7 @@ Trident is an alternative interface to Storm. It provides exactly-once processin
* [Trident API Overview](Trident-API-Overview.html) -- operations for transforming and orchestrating data
* [Trident State](Trident-state.html) -- exactly-once processing and fast, persistent aggregation
* [Trident spouts](Trident-spouts.html) -- transactional and non-transactional data intake
+* [Trident RAS API](Tridnet-RAS-API.html) -- using the Resource Aware Scheduler with Trident.
### Setup and Deploying