You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@storm.apache.org by sr...@apache.org on 2018/05/12 16:06:35 UTC
[48/51] [partial] storm-site git commit: Rebuild content based on new
2.0.0-SNAPSHOT docs
http://git-wip-us.apache.org/repos/asf/storm-site/blob/0183bf7e/content/releases/2.0.0-SNAPSHOT/Performance.html
----------------------------------------------------------------------
diff --git a/content/releases/2.0.0-SNAPSHOT/Performance.html b/content/releases/2.0.0-SNAPSHOT/Performance.html
new file mode 100644
index 0000000..2224555
--- /dev/null
+++ b/content/releases/2.0.0-SNAPSHOT/Performance.html
@@ -0,0 +1,389 @@
+<!DOCTYPE html>
+<html>
+ <head>
+ <meta charset="utf-8">
+ <meta http-equiv="X-UA-Compatible" content="IE=edge">
+ <meta name="viewport" content="width=device-width, initial-scale=1">
+
+ <link rel="shortcut icon" href="/favicon.ico" type="image/x-icon">
+ <link rel="icon" href="/favicon.ico" type="image/x-icon">
+
+ <title>Performance Tuning</title>
+
+ <!-- Bootstrap core CSS -->
+ <link href="/assets/css/bootstrap.min.css" rel="stylesheet">
+ <!-- Bootstrap theme -->
+ <link href="/assets/css/bootstrap-theme.min.css" rel="stylesheet">
+
+ <!-- Custom styles for this template -->
+ <link rel="stylesheet" href="http://fortawesome.github.io/Font-Awesome/assets/font-awesome/css/font-awesome.css">
+ <link href="/css/style.css" rel="stylesheet">
+ <link href="/assets/css/owl.theme.css" rel="stylesheet">
+ <link href="/assets/css/owl.carousel.css" rel="stylesheet">
+ <script type="text/javascript" src="/assets/js/jquery.min.js"></script>
+ <script type="text/javascript" src="/assets/js/bootstrap.min.js"></script>
+ <script type="text/javascript" src="/assets/js/owl.carousel.min.js"></script>
+ <script type="text/javascript" src="/assets/js/storm.js"></script>
+ <!-- Just for debugging purposes. Don't actually copy these 2 lines! -->
+ <!--[if lt IE 9]><script src="../../assets/js/ie8-responsive-file-warning.js"></script><![endif]-->
+
+ <!-- HTML5 shim and Respond.js for IE8 support of HTML5 elements and media queries -->
+ <!--[if lt IE 9]>
+ <script src="https://oss.maxcdn.com/html5shiv/3.7.2/html5shiv.min.js"></script>
+ <script src="https://oss.maxcdn.com/respond/1.4.2/respond.min.js"></script>
+ <![endif]-->
+ </head>
+
+
+ <body>
+ <header>
+ <div class="container-fluid">
+ <div class="row">
+ <div class="col-md-5">
+ <a href="/index.html"><img src="/images/logo.png" class="logo" /></a>
+ </div>
+ <div class="col-md-5">
+
+ <h1>Version: 2.0.0-SNAPSHOT</h1>
+
+ </div>
+ <div class="col-md-2">
+ <a href="/downloads.html" class="btn-std btn-block btn-download">Download</a>
+ </div>
+ </div>
+ </div>
+</header>
+<!--Header End-->
+<!--Navigation Begin-->
+<div class="navbar" role="banner">
+ <div class="container-fluid">
+ <div class="navbar-header">
+ <button class="navbar-toggle" type="button" data-toggle="collapse" data-target=".bs-navbar-collapse">
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ </button>
+ </div>
+ <nav class="collapse navbar-collapse bs-navbar-collapse" role="navigation">
+ <ul class="nav navbar-nav">
+ <li><a href="/index.html" id="home">Home</a></li>
+ <li><a href="/getting-help.html" id="getting-help">Getting Help</a></li>
+ <li><a href="/about/integrates.html" id="project-info">Project Information</a></li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown" id="documentation">Documentation <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+
+
+ <li><a href="/releases/2.0.0-SNAPSHOT/index.html">2.0.0-SNAPSHOT</a></li>
+
+
+
+ <li><a href="/releases/1.2.1/index.html">1.2.1</a></li>
+
+
+
+ <li><a href="/releases/1.1.2/index.html">1.1.2</a></li>
+
+
+
+
+
+ <li><a href="/releases/1.0.6/index.html">1.0.6</a></li>
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ </ul>
+ </li>
+ <li><a href="/talksAndVideos.html">Talks and Slideshows</a></li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown" id="contribute">Community <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+ <li><a href="/contribute/Contributing-to-Storm.html">Contributing</a></li>
+ <li><a href="/contribute/People.html">People</a></li>
+ <li><a href="/contribute/BYLAWS.html">ByLaws</a></li>
+ </ul>
+ </li>
+ <li><a href="/2018/02/19/storm121-released.html" id="news">News</a></li>
+ </ul>
+ </nav>
+ </div>
+</div>
+
+
+
+ <div class="container-fluid">
+ <h1 class="page-title">Performance Tuning</h1>
+ <div class="row">
+ <div class="col-md-12">
+ <!-- Documentation -->
+
+<p class="post-meta"></p>
+
+<p>Latency, throughput and resource consumption are the three key dimensions involved in performance tuning.
+In the following sections we discuss the settings that can used to tune along these dimension and understand the trade-offs.</p>
+
+<p>It is important to understand that these settings can vary depending on the topology, the type of hardware and the number of hosts used by the topology.</p>
+
+<h2 id="1-buffer-size">1. Buffer Size</h2>
+
+<p>Spouts and Bolts operate asynchronously using message passing. Message queues used for this purpose are of fixed but configurable size. Buffer size
+refers to the size of these queues. Every consumer has its own receive queue. The messages wait in the queue until the consumer is ready to process them.
+The queue will typically be almost empty or almost full depending whether the consumer is operating faster or slower than the rate at which producers
+are generating messages for it. Storm queues always have a single consumer and potentially multiple producers. There are two buffer size settings
+of interest:</p>
+
+<ul>
+<li><code>topology.executor.receive.buffer.size</code> : This is the size of the message queue employed for each spout and bolt executor.</li>
+<li><code>topology.transfer.buffer.size</code> : This is the size of the outbound message queue used for inter-worker messaging. This queue is referred to as
+the <em>Worker Transfer Queue</em>.</li>
+</ul>
+
+<p><strong>Note:</strong> If the specified buffer size is not a power of 2, it is internally rounded up to the next power of 2.</p>
+
+<h4 id="guidance">Guidance</h4>
+
+<p>Very small message queues (size < 1024) are likely to hamper throughput by not providing enough isolation between the consumer and producer. This
+can affect the asynchronous nature of the processing as the producers are likely to find the downstream queue to be full.</p>
+
+<p>Very large message queues are also not desirable to deal with slow consumers. Better to employ more consumers (i.e. bolts) on additional CPU cores instead. If queues
+are large and often full, the messages will end up waiting longer in these queues at each step of the processing, leading to poor latency being
+reported on the Storm UI. Large queues also imply higher memory consumption especially if the queues are typically full.</p>
+
+<h2 id="2-batch-size">2. Batch Size</h2>
+
+<p>Producers can either write a batch of messages to the consumer's queue or write each message individually. This batch size can be configured.
+Inserting messages in batches to downstream queues helps reduce the number of synchronization operations required for the inserts. Consequently this helps achieve higher throughput. However,
+sometimes it may take a little time for the buffer to fill up, before it is flushed into the downstream queue. This implies that the buffered messages
+will take longer to become visible to the downstream consumer who is waiting to process them. This can increase the average end-to-end latency for
+these messages. The latency can get very bad if the batch sizes are large and the topology is not experiencing high traffic.</p>
+
+<ul>
+<li><p><code>topology.producer.batch.size</code> : The batch size for writes into the receive queue of any spout/bolt is controlled via this setting. This setting
+impacts the communication within a worker process. Each upstream producer maintains a separate batch to a component's receive queue. So if two spout
+instances are writing to the same downstream bolt instance, each of the spout instances will have maintain a separate batch.</p></li>
+<li><p><code>topology.transfer.batch.size</code> : Messages that are destined to a spout/bolt running on a different worker process, are sent to a queue called
+the <strong>Worker Transfer Queue</strong>. The Worker Transfer Thread is responsible for draining the messages in this queue and send them to the appropriate
+worker process over the network. This setting controls the batch size for writes into the Worker Transfer Queue. This impacts the communication
+between worker processes.</p></li>
+</ul>
+
+<h4 id="guidance">Guidance</h4>
+
+<p><strong>For Low latency:</strong> Set batch size to 1. This basically disables batching. This is likely to reduce peak sustainable throughput under heavy traffic, but
+not likely to impact throughput much under low/medium traffic situations.</p>
+
+<p><strong>For High throughput:</strong> Set batch size > 1. Try values like 10, 100, 1000 or even higher and see what yields the best throughput for the topology.
+Beyond a certain point the throughput is likely to get worse.</p>
+
+<p><strong>Varying throughput:</strong> Topologies often experience fluctuating amounts of incoming traffic over the day. Other topos may experience higher traffic in some
+paths and lower throughput in other paths simultaneously. If latency is not a concern, a small bach size (e.g. 10) and in conjunction with the right flush
+frequency may provide a reasonable compromise for such scenarios. For meeting stricter latency SLAs, consider setting it to 1.</p>
+
+<h2 id="3-flush-tuple-frequency">3. Flush Tuple Frequency</h2>
+
+<p>In low/medium traffic situations or when batch size is too large, the batches may take too long to fill up and consequently the messages could take unacceptably
+long time to become visible to downstream components. In such case, periodic flushing of batches is necessary to keep the messages moving and avoid compromising
+latencies when batching is enabled.</p>
+
+<p>When batching has been enabled, special messages called <em>flush tuples</em> are inserted periodically into the receive queues of all spout and bolt instances.
+This causes each spout/bolt instance to flush all its outstanding batches to their respective downstream components.</p>
+
+<p><code>topology.flush.tuple.freq.millis</code> : This setting controls how often the flush tuples are generated. Flush tuples are not generated if this configuration is
+set to 0 or if (<code>topology.producer.batch.size</code>=1 and <code>topology.transfer.batch.size</code>=1).</p>
+
+<h4 id="guidance">Guidance</h4>
+
+<p>Flushing interval can be used as tool to retain the higher throughput benefits of batching and avoid batched messages getting stuck for too long waiting for their.
+batch to fill. Preferably this value should be larger than the average execute latencies of the bolts in the topology. Trying to flush the queues more frequently than
+the amount of time it takes to produce the messages may hurt performance. Understanding the average execute latencies of each bolt will help determine the average
+number of messages in the queues between two flushes.</p>
+
+<p><strong>For Low latency:</strong> A smaller value helps achieve tighter latency SLAs.</p>
+
+<p><strong>For High throughput:</strong> When trying to maximize throughput under high traffic situations, the batches are likely to get filled and flushed automatically.
+To optimize for such cases, this value can be set to a higher number.</p>
+
+<p><strong>Varying throughput:</strong> If latency is not a concern, a larger value will optimize for high traffic situations. For meeting tighter SLAs set this to lower
+values.</p>
+
+<h2 id="4-wait-strategy">4. Wait Strategy</h2>
+
+<p>Wait strategies are used to conserve CPU usage by trading off some latency and throughput. They are applied for the following situations:</p>
+
+<p>4.1 <strong>Spout Wait:</strong> In low/no traffic situations, Spout's nextTuple() may not produce any new emits. To prevent invoking the Spout's nextTuple too often,
+this wait strategy is used between nextTuple() calls, allowing the spout's executor thread to idle and conserve CPU. Spout wait strategy is also used
+when the <code>topology.max.spout.pending</code> limit has been reached when ACKers are enabled. Select a strategy using <code>topology.spout.wait.strategy</code>. Configure the
+chosen wait strategy using one of the <code>topology.spout.wait.*</code> settings.</p>
+
+<p>4.2 <strong>Bolt Wait:</strong> : When a bolt polls it's receive queue for new messages to process, it is possible that the queue is empty. This typically happens
+in case of low/no traffic situations or when the upstream spout/bolt is inherently slower. This wait strategy is used in such cases. It avoids high CPU usage
+due to the bolt continuously checking on a typically empty queue. Select a strategy using <code>topology.bolt.wait.strategy</code>. The chosen strategy can be further configured
+using the <code>topology.bolt.wait.*</code> settings.</p>
+
+<p>4.3 <strong>Backpressure Wait</strong> : Select a strategy using <code>topology.backpressure.wait.strategy</code>. When a spout/bolt tries to write to a downstream component's receive queue,
+there is a possibility that the queue is full. In such cases the write needs to be retried. This wait strategy is used to induce some idling in-between re-attempts for
+conserving CPU. The chosen strategy can be further configured using the <code>topology.backpressure.wait.*</code> settings.</p>
+
+<h4 id="built-in-wait-strategies">Built-in wait strategies:</h4>
+
+<p>These wait strategies are availabe for use with all of the above mentioned wait situations.</p>
+
+<ul>
+<li><strong>ProgressiveWaitStrategy</strong> : This strategy can be used for Bolt Wait or Backpressure Wait situations. Set the strategy to 'org.apache.storm.policy.WaitStrategyProgressive' to
+select this wait strategy. This is a dynamic wait strategy that enters into progressively deeper states of CPU conservation if the Backpressure Wait or Bolt Wait situations persist.
+It has 3 levels of idling and allows configuring how long to stay at each level :</li>
+</ul>
+
+<ol>
+<li><p>Level1 / No Waiting - The first few times it will return immediately. This does not conserve any CPU. The number of times it remains in this state is configured using
+<code>topology.spout.wait.progressive.level1.count</code> or <code>topology.bolt.wait.progressive.level1.count</code> or <code>topology.backpressure.wait.progressive.level1.count</code> depending which
+situation it is being used.</p></li>
+<li><p>Level 2 / Park Nanos - In this state it disables the current thread for thread scheduling purposes, for 1 nano second using LockSupport.parkNanos(). This puts the CPU in a minimal
+conservation state. It remains in this state for <code>topology.spout.wait.progressive.level2.count</code> or <code>topology.bolt.wait.progressive.level2.count</code> or
+<code>topology.backpressure.wait.progressive.level2.count</code> iterations.</p></li>
+<li><p>Level 3 / Thread.sleep() - In this level it calls Thread.sleep() with the value specified in <code>topology.spout.wait.progressive.level3.sleep.millis</code> or
+<code>topology.bolt.wait.progressive.level3.sleep.millis</code> or <code>topology.backpressure.wait.progressive.level3.sleep.millis</code>. This is the most CPU conserving level and it remains in
+this level for the remaining iterations.</p></li>
+</ol>
+
+<ul>
+<li><strong>ParkWaitStrategy</strong> : This strategy can be used for Bolt Wait or Backpressure Wait situations. Set the strategy to <code>org.apache.storm.policy.WaitStrategyPark</code> to use this.
+This strategy disables the current thread for thread scheduling purposes by calling LockSupport.parkNanos(). The amount of park time is configured using either
+<code>topology.bolt.wait.park.microsec</code> or <code>topology.backpressure.wait.park.microsec</code> based on the wait situation it is used. Setting the park time to 0, effectively disables
+invocation of LockSupport.parkNanos and this mode can be used to achieve busy polling (which at the cost of high CPU utilization even when idle, may improve latency and/or throughput).</li>
+</ul>
+
+<h2 id="5-max-spout-pending">5. Max.spout.pending</h2>
+
+<p>The setting <code>topology.max.spout.pending</code> limits the number of un-ACKed tuples at the spout level. Once a spout reaches this limit, the spout's nextTuple()
+method will not be called until some ACKs are received for the outstanding emits. This setting does not have any affect if ACKing is disabled. It
+is a spout throttling mechanism which can impact throughput and latency. Setting it to null disables it for storm-core topologies. Impact on throughput
+is dependent on the topology and its concurrency (workers/executors), so experimentation is necessary to determine optimal setting. Latency and memory consumption
+is expected to typically increase with higher and higher values for this.</p>
+
+<h2 id="6-load-aware-messaging">6. Load Aware messaging</h2>
+
+<p>When load aware messaging is enabled (default), shuffle grouping takes additional factors into consideration for message routing.
+Impact of this on performance is dependent on the topology and its deployment footprint (i.e. distribution over process and machines).
+Consequently it is useful to assess the impact of setting <code>topology.disable.loadaware.messaging</code> to <code>true</code> or <code>false</code> for your
+specific case.</p>
+
+<h2 id="7-sampling-rate">7. Sampling Rate</h2>
+
+<p>Sampling rate is used to control how often certain metrics are computed on the Spout and Bolt executors. This is configured using <code>topology.stats.sample.rate</code>
+Setting it to 1 means, the stats are computed for every emitted message. As an example, to sample once every 1000 messages it can be set to 0.001. It may be
+possible to improve throughput and latency by reducing the sampling rate.</p>
+
+<h2 id="8-budgeting-cpu-cores-for-executors">8. Budgeting CPU cores for Executors</h2>
+
+<p>There are three main types of executors (i.e threads) to take into account when budgeting CPU cores for them. Spout Executors, Bolt Executors, Worker Transfer (handles outbound
+messages) and NettyWorker (handles inbound messages).
+The first two are used to run spout, bolt and acker instances. The Worker Transfer thread is used to serialize and send messages to other workers (in multi-worker mode).</p>
+
+<p>Executors that are expected to remain busy, either because they are handling a lot of messages, or because their processing is inherently CPU intensive, should be allocated
+1 physical core each. Allocating logical cores (instead of physical) or less than 1 physical core for CPU intensive executors increases CPU contention and performance can suffer.
+Executors that are not expected to be busy can be allocated a smaller fraction of the physical core (or even logical cores). It maybe not be economical to allocate a full physical
+core for executors that are not likely to saturate the CPU.</p>
+
+<p>The <em>system bolt</em> generally processes very few messages per second, and so requires very little cpu (typically less than 10% of a physical core).</p>
+
+
+
+ </div>
+ </div>
+ </div>
+<footer>
+ <div class="container-fluid">
+ <div class="row">
+ <div class="col-md-3">
+ <div class="footer-widget">
+ <h5>Meetups</h5>
+ <ul class="latest-news">
+
+ <li><a href="http://www.meetup.com/Apache-Storm-Apache-Kafka/">Apache Storm & Apache Kafka</a> <span class="small">(Sunnyvale, CA)</span></li>
+
+ <li><a href="http://www.meetup.com/Apache-Storm-Kafka-Users/">Apache Storm & Kafka Users</a> <span class="small">(Seattle, WA)</span></li>
+
+ <li><a href="http://www.meetup.com/New-York-City-Storm-User-Group/">NYC Storm User Group</a> <span class="small">(New York, NY)</span></li>
+
+ <li><a href="http://www.meetup.com/Bay-Area-Stream-Processing">Bay Area Stream Processing</a> <span class="small">(Emeryville, CA)</span></li>
+
+ <li><a href="http://www.meetup.com/Boston-Storm-Users/">Boston Realtime Data</a> <span class="small">(Boston, MA)</span></li>
+
+ <li><a href="http://www.meetup.com/storm-london">London Storm User Group</a> <span class="small">(London, UK)</span></li>
+
+ <!-- <li><a href="http://www.meetup.com/Apache-Storm-Kafka-Users/">Seatle, WA</a> <span class="small">(27 Jun 2015)</span></li> -->
+ </ul>
+ </div>
+ </div>
+ <div class="col-md-3">
+ <div class="footer-widget">
+ <h5>About Storm</h5>
+ <p>Storm integrates with any queueing system and any database system. Storm's spout abstraction makes it easy to integrate a new queuing system. Likewise, integrating Storm with database systems is easy.</p>
+ </div>
+ </div>
+ <div class="col-md-3">
+ <div class="footer-widget">
+ <h5>First Look</h5>
+ <ul class="footer-list">
+ <li><a href="/releases/current/Rationale.html">Rationale</a></li>
+ <li><a href="/releases/current/Tutorial.html">Tutorial</a></li>
+ <li><a href="/releases/current/Setting-up-development-environment.html">Setting up development environment</a></li>
+ <li><a href="/releases/current/Creating-a-new-Storm-project.html">Creating a new Storm project</a></li>
+ </ul>
+ </div>
+ </div>
+ <div class="col-md-3">
+ <div class="footer-widget">
+ <h5>Documentation</h5>
+ <ul class="footer-list">
+ <li><a href="/releases/current/index.html">Index</a></li>
+ <li><a href="/releases/current/javadocs/index.html">Javadoc</a></li>
+ <li><a href="/releases/current/FAQ.html">FAQ</a></li>
+ </ul>
+ </div>
+ </div>
+ </div>
+ <hr/>
+ <div class="row">
+ <div class="col-md-12">
+ <p align="center">Copyright © 2015 <a href="http://www.apache.org">Apache Software Foundation</a>. All Rights Reserved.
+ <br>Apache Storm, Apache, the Apache feather logo, and the Apache Storm project logos are trademarks of The Apache Software Foundation.
+ <br>All other marks mentioned may be trademarks or registered trademarks of their respective owners.</p>
+ </div>
+ </div>
+ </div>
+</footer>
+<!--Footer End-->
+<!-- Scroll to top -->
+<span class="totop"><a href="#"><i class="fa fa-angle-up"></i></a></span>
+
+</body>
+
+</html>
+
http://git-wip-us.apache.org/repos/asf/storm-site/blob/0183bf7e/content/releases/2.0.0-SNAPSHOT/Resource_Aware_Scheduler_overview.html
----------------------------------------------------------------------
diff --git a/content/releases/2.0.0-SNAPSHOT/Resource_Aware_Scheduler_overview.html b/content/releases/2.0.0-SNAPSHOT/Resource_Aware_Scheduler_overview.html
index c7702d3..814b5e2 100644
--- a/content/releases/2.0.0-SNAPSHOT/Resource_Aware_Scheduler_overview.html
+++ b/content/releases/2.0.0-SNAPSHOT/Resource_Aware_Scheduler_overview.html
@@ -158,6 +158,7 @@
<ol>
<li><a href="#Setting-Memory-Requirement">Setting Memory Requirement</a></li>
+<li><a href="#Setting-Shared-Memory">Shared Memory Requirement</a></li>
<li><a href="#Setting-CPU-Requirement">Setting CPU Requirement</a></li>
<li><a href="#Limiting-the-Heap-Size-per-Worker-(JVM)Process">Limiting the Heap Size per Worker (JVM) Process</a></li>
<li><a href="#Setting-Available-Resources-on-Node">Setting Available Resources on Node</a></li>
@@ -170,7 +171,6 @@
<li><a href="#Specifying-Topology-Priority">Specifying Topology Priority</a></li>
<li><a href="#Specifying-Scheduling-Strategy">Specifying Scheduling Strategy</a></li>
<li><a href="#Specifying-Topology-Prioritization-Strategy">Specifying Topology Prioritization Strategy</a></li>
-<li><a href="#Specifying-Eviction-Strategy">Specifying Eviction Strategy</a></li>
</ol></li>
<li><a href="#Profiling-Resource-Usage">Profiling Resource Usage</a></li>
<li><a href="#Enhancements-on-original-DefaultResourceAwareStrategy">Enhancements on original DefaultResourceAwareStrategy</a></li>
@@ -180,10 +180,11 @@
<h2 id="using-resource-aware-scheduler">Using Resource Aware Scheduler</h2>
-<p>The user can switch to using the Resource Aware Scheduler by setting the following in <em>conf/storm.yaml</em></p>
-<div class="highlight"><pre><code class="language-" data-lang="">storm.scheduler: “org.apache.storm.scheduler.resource.ResourceAwareScheduler”
-</code></pre></div>
-<p><div id='API-Overview'/></p>
+<p>The user can switch to using the Resource Aware Scheduler by setting the following in <em>conf/storm.yaml</em>
+<code>
+ storm.scheduler: “org.apache.storm.scheduler.resource.ResourceAwareScheduler”
+</code><br>
+<div id='API-Overview'/></p>
<h2 id="api-overview">API Overview</h2>
@@ -195,129 +196,165 @@
<h3 id="setting-memory-requirement">Setting Memory Requirement</h3>
-<p>API to set component memory requirement:</p>
-<div class="highlight"><pre><code class="language-" data-lang="">public T setMemoryLoad(Number onHeap, Number offHeap)
-</code></pre></div>
-<p>Parameters:
+<p>API to set component memory requirement:
+<code>
+ public T setMemoryLoad(Number onHeap, Number offHeap)
+</code>
+Parameters:
* Number onHeap – The amount of on heap memory an instance of this component will consume in megabytes
* Number offHeap – The amount of off heap memory an instance of this component will consume in megabytes</p>
-<p>The user also has to option to just specify the on heap memory requirement if the component does not have an off heap memory need.</p>
-<div class="highlight"><pre><code class="language-" data-lang="">public T setMemoryLoad(Number onHeap)
-</code></pre></div>
-<p>Parameters:
+<p>The user also has to option to just specify the on heap memory requirement if the component does not have an off heap memory need.
+<code>
+ public T setMemoryLoad(Number onHeap)
+</code>
+Parameters:
* Number onHeap – The amount of on heap memory an instance of this component will consume</p>
<p>If no value is provided for offHeap, 0.0 will be used. If no value is provided for onHeap, or if the API is never called for a component, the default value will be used.</p>
+<p>Example of Usage:
+<code>
+ SpoutDeclarer s1 = builder.setSpout("word", new TestWordSpout(), 10);
+ s1.setMemoryLoad(1024.0, 512.0);
+ builder.setBolt("exclaim1", new ExclamationBolt(), 3)
+ .shuffleGrouping("word").setMemoryLoad(512.0);
+</code>
+The entire memory requested for this topology is 16.5 GB. That is from 10 spouts with 1GB on heap memory and 0.5 GB off heap memory each and 3 bolts with 0.5 GB on heap memory each.</p>
+
+<p><div id='Setting-Shared-Memory'/></p>
+
+<h3 id="shared-memory">Shared Memory</h3>
+
+<p>In some cases you may have memory that is shared between components. It may be a as simple as a large static data structure, or as complex as static data that is memory mapped into a bolt and is shared across workers. In any case you can specify your shared memory request by
+creating one of <code>SharedOffHeapWithinNode</code>, <code>SharedOffHeapWithinWorker</code>, or <code>SharedOnHeap</code> and adding it to bolts and spouts that use that shared memory.</p>
+
<p>Example of Usage:</p>
-<div class="highlight"><pre><code class="language-" data-lang="">SpoutDeclarer s1 = builder.setSpout("word", new TestWordSpout(), 10);
-s1.setMemoryLoad(1024.0, 512.0);
-builder.setBolt("exclaim1", new ExclamationBolt(), 3)
- .shuffleGrouping("word").setMemoryLoad(512.0);
+<div class="highlight"><pre><code class="language-" data-lang=""> builder.setBolt("exclaim1", new ExclamationBolt(), 3).shuffleGrouping("word")
+ .addSharedMemory(new SharedOnHeap(100, "exclaim-cache"));
+</code></pre></div>
+<p>In the above example all of the "exclaim1" bolts in a worker will share 100MB of memory.</p>
+<div class="highlight"><pre><code class="language-" data-lang=""> builder.setBolt("lookup", new LookupBolt(), 3).shuffleGrouping("spout")
+ .addSharedMemory(new SharedOffHeapWithinNode(500, "static-lookup"));
</code></pre></div>
-<p>The entire memory requested for this topology is 16.5 GB. That is from 10 spouts with 1GB on heap memory and 0.5 GB off heap memory each and 3 bolts with 0.5 GB on heap memory each.</p>
+<p>In this example all "lookup" bolts on a given node will share 500 MB or memory off heap. </p>
<p><div id='Setting-CPU-Requirement'/></p>
<h3 id="setting-cpu-requirement">Setting CPU Requirement</h3>
-<p>API to set component CPU requirement:</p>
-<div class="highlight"><pre><code class="language-" data-lang="">public T setCPULoad(Double amount)
-</code></pre></div>
-<p>Parameters:
+<p>API to set component CPU requirement:
+<code>
+ public T setCPULoad(Double amount)
+</code>
+Parameters:
* Number amount – The amount of on CPU an instance of this component will consume.</p>
<p>Currently, the amount of CPU resources a component requires or is available on a node is represented by a point system. CPU usage is a difficult concept to define. Different CPU architectures perform differently depending on the task at hand. They are so complex that expressing all of that in a single precise portable number is impossible. Instead we take a convention over configuration approach and are primarily concerned with rough level of CPU usage while still providing the possibility to specify amounts more fine grained.</p>
<p>By convention a CPU core typically will get 100 points. If you feel that your processors are more or less powerful you can adjust this accordingly. Heavy tasks that are CPU bound will get 100 points, as they can consume an entire core. Medium tasks should get 50, light tasks 25, and tiny tasks 10. In some cases you have a task that spawns other threads to help with processing. These tasks may need to go above 100 points to express the amount of CPU they are using. If these conventions are followed the common case for a single threaded task the reported Capacity * 100 should be the number of CPU points that the task needs.</p>
-<p>Example of Usage:</p>
-<div class="highlight"><pre><code class="language-" data-lang="">SpoutDeclarer s1 = builder.setSpout("word", new TestWordSpout(), 10);
-s1.setCPULoad(15.0);
-builder.setBolt("exclaim1", new ExclamationBolt(), 3)
- .shuffleGrouping("word").setCPULoad(10.0);
-builder.setBolt("exclaim2", new HeavyBolt(), 1)
- .shuffleGrouping("exclaim1").setCPULoad(450.0);
-</code></pre></div>
+<p>Example of Usage:
+<code>
+ SpoutDeclarer s1 = builder.setSpout("word", new TestWordSpout(), 10);
+ s1.setCPULoad(15.0);
+ builder.setBolt("exclaim1", new ExclamationBolt(), 3)
+ .shuffleGrouping("word").setCPULoad(10.0);
+ builder.setBolt("exclaim2", new HeavyBolt(), 1)
+ .shuffleGrouping("exclaim1").setCPULoad(450.0);
+</code></p>
+
<p><div id='Limiting-the-Heap-Size-per-Worker-(JVM)Process'/></p>
<h3 id="limiting-the-heap-size-per-worker-jvm-process">Limiting the Heap Size per Worker (JVM) Process</h3>
-<div class="highlight"><pre><code class="language-" data-lang="">public void setTopologyWorkerMaxHeapSize(Number size)
+<div class="highlight"><pre><code class="language-" data-lang=""> public void setTopologyWorkerMaxHeapSize(Number size)
</code></pre></div>
<p>Parameters:
* Number size – The memory limit a worker process will be allocated in megabytes</p>
<p>The user can limit the amount of memory resources the resource aware scheduler allocates to a single worker on a per topology basis by using the above API. This API is in place so that the users can spread executors to multiple workers. However, spreading executors to multiple workers may increase the communication latency since executors will not be able to use Disruptor Queue for intra-process communication.</p>
-<p>Example of Usage:</p>
-<div class="highlight"><pre><code class="language-" data-lang="">Config conf = new Config();
-conf.setTopologyWorkerMaxHeapSize(512.0);
-</code></pre></div>
+<p>Example of Usage:
+<code>
+ Config conf = new Config();
+ conf.setTopologyWorkerMaxHeapSize(512.0);
+</code></p>
+
<p><div id='Setting-Available-Resources-on-Node'/></p>
<h3 id="setting-available-resources-on-node">Setting Available Resources on Node</h3>
<p>A storm administrator can specify node resource availability by modifying the <em>conf/storm.yaml</em> file located in the storm home directory of that node.</p>
-<p>A storm administrator can specify how much available memory a node has in megabytes adding the following to <em>storm.yaml</em></p>
-<div class="highlight"><pre><code class="language-" data-lang="">supervisor.memory.capacity.mb: [amount<Double>]
-</code></pre></div>
-<p>A storm administrator can also specify how much available CPU resources a node has available adding the following to <em>storm.yaml</em></p>
-<div class="highlight"><pre><code class="language-" data-lang="">supervisor.cpu.capacity: [amount<Double>]
-</code></pre></div>
+<p>A storm administrator can specify how much available memory a node has in megabytes adding the following to <em>storm.yaml</em>
+<code>
+ supervisor.memory.capacity.mb: [amount<Double>]
+</code>
+A storm administrator can also specify how much available CPU resources a node has available adding the following to <em>storm.yaml</em>
+<code>
+ supervisor.cpu.capacity: [amount<Double>]
+</code></p>
+
<p>Note: that the amount the user can specify for the available CPU is represented using a point system like discussed earlier.</p>
-<p>Example of Usage:</p>
-<div class="highlight"><pre><code class="language-" data-lang="">supervisor.memory.capacity.mb: 20480.0
-supervisor.cpu.capacity: 100.0
-</code></pre></div>
+<p>Example of Usage:
+<code>
+ supervisor.memory.capacity.mb: 20480.0
+ supervisor.cpu.capacity: 100.0
+</code></p>
+
<p><div id='Other-Configurations'/></p>
<h3 id="other-configurations">Other Configurations</h3>
<p>The user can set some default configurations for the Resource Aware Scheduler in <em>conf/storm.yaml</em>:</p>
-<div class="highlight"><pre><code class="language-" data-lang="">//default value if on heap memory requirement is not specified for a component
-topology.component.resources.onheap.memory.mb: 128.0
+<div class="highlight"><pre><code class="language-" data-lang=""> //default value if on heap memory requirement is not specified for a component
+ topology.component.resources.onheap.memory.mb: 128.0
-//default value if off heap memory requirement is not specified for a component
-topology.component.resources.offheap.memory.mb: 0.0
+ //default value if off heap memory requirement is not specified for a component
+ topology.component.resources.offheap.memory.mb: 0.0
-//default value if CPU requirement is not specified for a component
-topology.component.cpu.pcore.percent: 10.0
+ //default value if CPU requirement is not specified for a component
+ topology.component.cpu.pcore.percent: 10.0
-//default value for the max heap size for a worker
-topology.worker.max.heap.size.mb: 768.0
+ //default value for the max heap size for a worker
+ topology.worker.max.heap.size.mb: 768.0
</code></pre></div>
+<h3 id="warning">Warning</h3>
+
+<p>If Resource Aware Scheduling is enabled, it will dynamically calculate the number of workers and the <code>topology.workers</code> setting is ignored.</p>
+
<p><div id='Topology-Priorities-and-Per-User-Resource'/></p>
<h2 id="topology-priorities-and-per-user-resource">Topology Priorities and Per User Resource</h2>
-<p>The Resource Aware Scheduler or RAS also has multitenant capabilities since many Storm users typically share a Storm cluster. Resource Aware Scheduler can allocate resources on a per user basis. Each user can be guaranteed a certain amount of resources to run his or her topologies and the Resource Aware Scheduler will meet those guarantees when possible. When the Storm cluster has extra free resources, Resource Aware Scheduler will to be able allocate additional resources to user in a fair manner. The importance of topologies can also vary. Topologies can be used for actual production or just experimentation, thus Resource Aware Scheduler will take into account the importance of a topology when determining the order in which to schedule topologies or when to evict topologies</p>
+<p>The Resource Aware Scheduler or RAS also has multi-tenant capabilities since many Storm users typically share a Storm cluster. Resource Aware Scheduler can allocate resources on a per user basis. Each user can be guaranteed a certain amount of resources to run his or her topologies and the Resource Aware Scheduler will meet those guarantees when possible. When the Storm cluster has extra free resources, Resource Aware Scheduler will to be able allocate additional resources to user in a fair manner. The importance of topologies can also vary. Topologies can be used for actual production or just experimentation, thus Resource Aware Scheduler will take into account the importance of a topology when determining the order in which to schedule topologies or when to evict topologies</p>
<p><div id='Setup'/></p>
<h3 id="setup">Setup</h3>
-<p>The resource guarantees of a user can be specified <em>conf/user-resource-pools.yaml</em>. Specify the resource guarantees of a user in the following format:</p>
-<div class="highlight"><pre><code class="language-" data-lang="">resource.aware.scheduler.user.pools:
-[UserId]
- cpu: [Amount of Guarantee CPU Resources]
- memory: [Amount of Guarantee Memory Resources]
-</code></pre></div>
-<p>An example of what <em>user-resource-pools.yaml</em> can look like:</p>
-<div class="highlight"><pre><code class="language-" data-lang="">resource.aware.scheduler.user.pools:
- jerry:
- cpu: 1000
- memory: 8192.0
- derek:
- cpu: 10000.0
- memory: 32768
- bobby:
- cpu: 5000.0
- memory: 16384.0
-</code></pre></div>
-<p>Please note that the specified amount of Guaranteed CPU and Memory can be either a integer or double</p>
+<p>The resource guarantees of a user can be specified <em>conf/user-resource-pools.yaml</em>. Specify the resource guarantees of a user in the following format:
+<code>
+ resource.aware.scheduler.user.pools:
+ [UserId]
+ cpu: [Amount of Guarantee CPU Resources]
+ memory: [Amount of Guarantee Memory Resources]
+</code>
+An example of what <em>user-resource-pools.yaml</em> can look like:
+<code>
+ resource.aware.scheduler.user.pools:
+ jerry:
+ cpu: 1000
+ memory: 8192.0
+ derek:
+ cpu: 10000.0
+ memory: 32768
+ bobby:
+ cpu: 5000.0
+ memory: 16384.0
+</code>
+Please note that the specified amount of Guaranteed CPU and Memory can be either a integer or double</p>
<p><div id='Specifying-Topology-Priority'/></p>
@@ -329,10 +366,11 @@ For example we can create a priority level mapping:</p>
STAGING => 10 – 19
DEV => 20 – 29
</code></pre></div>
-<p>Thus, each priority level contains 10 sub priorities. Users can set the priority level of a topology by using the following API</p>
-<div class="highlight"><pre><code class="language-" data-lang="">conf.setTopologyPriority(int priority)
-</code></pre></div>
-<p>Parameters:
+<p>Thus, each priority level contains 10 sub priorities. Users can set the priority level of a topology by using the following API
+<code>
+ conf.setTopologyPriority(int priority)
+</code>
+Parameters:
* priority – an integer representing the priority of the topology</p>
<p>Please note that the 0-29 range is not a hard limit. Thus, a user can set a priority number that is higher than 29. However, the property of higher the priority number, lower the importance still holds</p>
@@ -341,16 +379,18 @@ DEV => 20 – 29
<h3 id="specifying-scheduling-strategy">Specifying Scheduling Strategy</h3>
-<p>A user can specify on a per topology basis what scheduling strategy to use. Users can implement the IStrategy interface and define new strategies to schedule specific topologies. This pluggable interface was created since we realize different topologies might have different scheduling needs. A user can set the topology strategy within the topology definition by using the API:</p>
-<div class="highlight"><pre><code class="language-" data-lang="">public void setTopologyStrategy(Class<? extends IStrategy> clazz)
-</code></pre></div>
-<p>Parameters:
+<p>A user can specify on a per topology basis what scheduling strategy to use. Users can implement the IStrategy interface and define new strategies to schedule specific topologies. This pluggable interface was created since we realize different topologies might have different scheduling needs. A user can set the topology strategy within the topology definition by using the API:
+<code>
+ public void setTopologyStrategy(Class<? extends IStrategy> clazz)
+</code>
+Parameters:
* clazz – The strategy class that implements the IStrategy interface</p>
-<p>Example Usage:</p>
-<div class="highlight"><pre><code class="language-" data-lang="">conf.setTopologyStrategy(org.apache.storm.scheduler.resource.strategies.scheduling.DefaultResourceAwareStrategy.class);
-</code></pre></div>
-<p>A default scheduling is provided. The DefaultResourceAwareStrategy is implemented based off the scheduling algorithm in the original paper describing resource aware scheduling in Storm:</p>
+<p>Example Usage:
+<code>
+ conf.setTopologyStrategy(org.apache.storm.scheduler.resource.strategies.scheduling.DefaultResourceAwareStrategy.class);
+</code>
+A default scheduling is provided. The DefaultResourceAwareStrategy is implemented based off the scheduling algorithm in the original paper describing resource aware scheduling in Storm:</p>
<p>Peng, Boyang, Mohammad Hosseini, Zhihao Hong, Reza Farivar, and Roy Campbell. "R-storm: Resource-aware scheduling in storm." In Proceedings of the 16th Annual Middleware Conference, pp. 149-161. ACM, 2015.</p>
@@ -362,64 +402,97 @@ DEV => 20 – 29
<h3 id="specifying-topology-prioritization-strategy">Specifying Topology Prioritization Strategy</h3>
-<p>The order of scheduling is a pluggable interface in which a user could define a strategy that prioritizes topologies. For a user to define his or her own prioritization strategy, he or she needs to implement the ISchedulingPriorityStrategy interface. A user can set the scheduling priority strategy by setting the <em>Config.RESOURCE_AWARE_SCHEDULER_PRIORITY_STRATEGY</em> to point to the class that implements the strategy. For instance:</p>
-<div class="highlight"><pre><code class="language-" data-lang="">resource.aware.scheduler.priority.strategy: "org.apache.storm.scheduler.resource.strategies.priority.DefaultSchedulingPriorityStrategy"
-</code></pre></div>
-<p>A default strategy will be provided. The following explains how the default scheduling priority strategy works.</p>
+<p>The order of scheduling and eviction is determined by a pluggable interface in which the cluster owner can define how topologies should be scheduled. For the owner to define his or her own prioritization strategy, she or he needs to implement the ISchedulingPriorityStrategy interface. A user can set the scheduling priority strategy by setting the <code>DaemonConfig.RESOURCE_AWARE_SCHEDULER_PRIORITY_STRATEGY</code> to point to the class that implements the strategy. For instance:
+<code>
+ resource.aware.scheduler.priority.strategy: "org.apache.storm.scheduler.resource.strategies.priority.DefaultSchedulingPriorityStrategy"
+</code></p>
+
+<p>Topologies are scheduled starting at the beginning of the list returned by this plugin. If there are not enough resources to schedule the topology others are evicted starting at the end of the list. Eviction stops when there are no lower priority topologies left to evict.</p>
<p><strong>DefaultSchedulingPriorityStrategy</strong></p>
-<p>The order of scheduling should be based on the distance between a user’s current resource allocation and his or her guaranteed allocation. We should prioritize the users who are the furthest away from their resource guarantee. The difficulty of this problem is that a user may have multiple resource guarantees, and another user can have another set of resource guarantees, so how can we compare them in a fair manner? Let's use the average percentage of resource guarantees satisfied as a method of comparison.</p>
+<p>In the past the order of scheduling was based on the distance between a user’s current resource allocation and his or her guaranteed allocation.</p>
+
+<p>We currently use a slightly different approach. We simulate scheduling the highest priority topology for each user and score the topology for each of the resources using the formula</p>
+<div class="highlight"><pre><code class="language-" data-lang="">(Requested + Assigned - Guaranteed)/Available
+</code></pre></div>
+<p>Where</p>
+
+<ul>
+<li><code>Requested</code> is the resource requested by this topology (or a approximation of it for complex requests like shared memory)</li>
+<li><code>Assigned</code> is the resources already assigned by the simulation.</li>
+<li><code>Guaranteed</code> is the resource guarantee for this user</li>
+<li><code>Available</code> is the amount of that resource currently available in the cluster.</li>
+</ul>
-<p>For example:</p>
+<p>This gives a score that is negative for guaranteed requests and a score that is positive for requests that are not within the guarantee.</p>
+
+<p>To combine different resources the maximum of all the individual resource scores is used. This guarantees that if a user would go over a guarantee for a single resource it would not be offset by being under guarantee on any other resources.</p>
+
+<p>For Example:</p>
+
+<p>Assume we have to schedule the following topologies.</p>
<table><thead>
<tr>
+<th>ID</th>
<th>User</th>
-<th>Resource Guarantee</th>
-<th>Resource Allocated</th>
+<th>CPU</th>
+<th>Memory</th>
+<th>Priority</th>
</tr>
</thead><tbody>
<tr>
+<td>A-1</td>
<td>A</td>
-<td><10 CPU, 50GB></td>
-<td><2 CPU, 40 GB></td>
+<td>100</td>
+<td>1,000</td>
+<td>1</td>
+</tr>
+<tr>
+<td>A-2</td>
+<td>A</td>
+<td>100</td>
+<td>1,000</td>
+<td>10</td>
+</tr>
+<tr>
+<td>B-1</td>
+<td>B</td>
+<td>100</td>
+<td>1,000</td>
+<td>1</td>
</tr>
<tr>
+<td>B-2</td>
<td>B</td>
-<td>< 20 CPU, 25GB></td>
-<td><15 CPU, 10 GB></td>
+<td>100</td>
+<td>1,000</td>
+<td>10</td>
</tr>
</tbody></table>
-<p>User A’s average percentage satisfied of resource guarantee: </p>
-
-<p>(2/10+40/50)/2 = 0.5</p>
-
-<p>User B’s average percentage satisfied of resource guarantee: </p>
-
-<p>(15/20+10/25)/2 = 0.575</p>
-
-<p>Thus, in this example User A has a smaller average percentage of his or her resource guarantee satisfied than User B. Thus, User A should get priority to be allocated more resource, i.e., schedule a topology submitted by User A.</p>
-
-<p>When scheduling, RAS sorts users by the average percentage satisfied of resource guarantee and schedule topologies from users based on that ordering starting from the users with the lowest average percentage satisfied of resource guarantee. When a user’s resource guarantee is completely satisfied, the user’s average percentage satisfied of resource guarantee will be greater than or equal to 1.</p>
-
-<p><div id='Specifying-Eviction-Strategy'/></p>
+<p>The cluster as a whole has 300 CPU and 4,000 Memory.</p>
-<h3 id="specifying-eviction-strategy">Specifying Eviction Strategy</h3>
-
-<p>The eviction strategy is used when there are not enough free resources in the cluster to schedule new topologies. If the cluster is full, we need a mechanism to evict topologies so that user resource guarantees can be met and additional resource can be shared fairly among users. The strategy for evicting topologies is also a pluggable interface in which the user can implement his or her own topology eviction strategy. For a user to implement his or her own eviction strategy, he or she needs to implement the IEvictionStrategy Interface and set <em>Config.RESOURCE_AWARE_SCHEDULER_EVICTION_STRATEGY</em> to point to the implemented strategy class. For instance:</p>
-<div class="highlight"><pre><code class="language-" data-lang="">resource.aware.scheduler.eviction.strategy: "org.apache.storm.scheduler.resource.strategies.eviction.DefaultEvictionStrategy"
+<p>User A is guaranteed 100 CPU and 1,000 Memory. User B is guaranteed 200 CPU and 1,500 Memory. The scores for the most important, lowest priority number, topologies for each user would be.</p>
+<div class="highlight"><pre><code class="language-" data-lang="">A-1 Score = max(CPU: (100 + 0 - 100)/300, MEM: (1,000 + 0 - 1,000)/4,000) = 0
+B-1 Score = max(CPU: (100 + 0 - 200)/300, MEM: (1,000 + 0 - 1,500)/4,000) = -0.125
+</code></pre></div>
+<p><code>B-1</code> has the lowest score so it would be the highest priority topology to schedule. In the next round the scores would be.</p>
+<div class="highlight"><pre><code class="language-" data-lang="">A-1 Score = max(CPU: (100 + 0 - 100)/200, MEM: (1,000 + 0 - 1,000)/3,000) = 0
+B-2 Score = max(CPU: (100 + 100 - 200)/200, MEM: (1,000 + 1,000 - 1,500)/3,000) = 0.167
</code></pre></div>
-<p>A default eviction strategy is provided. The following explains how the default topology eviction strategy works</p>
+<p><code>A-1</code> has the lowest score now so it would be the next highest priority topology to schedule.</p>
-<p><strong>DefaultEvictionStrategy</strong></p>
+<p>This process would be repeated until all of the topologies are ordered, even if there are no resources left on the cluster to schedule a topology.</p>
-<p>To determine if topology eviction should occur we should take into account the priority of the topology that we are trying to schedule and whether the resource guarantees for the owner of the topology have been met. </p>
+<p><strong>FIFOSchedulingPriorityStrategy</strong></p>
-<p>We should never evict a topology from a user that does not have his or her resource guarantees satisfied. The following flow chart should describe the logic for the eviction process.</p>
+<p>The FIFO strategy is intended more for test or staging clusters where users are running integration tests or doing development work. Topologies in these situations tend to be short lived and at times a user may forget that they are running a topology at all.</p>
-<p><img src="images/resource_aware_scheduler_default_eviction_strategy.png" alt="Viewing metrics with VisualVM"></p>
+<p>To try and be as fair as possible to users running short lived topologies the <code>FIFOSchedulingPriorityStrategy</code> extends the <code>DefaultSchedulingPriorityStrategy</code> so that any negative score (a.k.a. a topology that fits within a user's guarantees) would remain unchanged, but positive scores are replaced with the up-time of the topology.</p>
+
+<p>This respects the guarantees of a user, but at the same time it gives priority for the rest of the resources to the most recently launched topology. Older topologies, that have probably been forgotten about, are then least likely to get resources.</p>
<p><div id='Profiling-Resource-Usage'/></p>
@@ -428,27 +501,27 @@ DEV => 20 – 29
<p>Figuring out resource usage for your topology:</p>
<p>To get an idea of how much memory/CPU your topology is actually using you can add the following to your topology launch code.</p>
-<div class="highlight"><pre><code class="language-" data-lang="">//Log all storm metrics
-conf.registerMetricsConsumer(backtype.storm.metric.LoggingMetricsConsumer.class);
+<div class="highlight"><pre><code class="language-" data-lang=""> //Log all storm metrics
+ conf.registerMetricsConsumer(backtype.storm.metric.LoggingMetricsConsumer.class);
-//Add in per worker CPU measurement
-Map<String, String> workerMetrics = new HashMap<String, String>();
-workerMetrics.put("CPU", "org.apache.storm.metrics.sigar.CPUMetric");
-conf.put(Config.TOPOLOGY_WORKER_METRICS, workerMetrics);
+ //Add in per worker CPU measurement
+ Map<String, String> workerMetrics = new HashMap<String, String>();
+ workerMetrics.put("CPU", "org.apache.storm.metrics.sigar.CPUMetric");
+ conf.put(Config.TOPOLOGY_WORKER_METRICS, workerMetrics);
</code></pre></div>
<p>The CPU metrics will require you to add</p>
-<div class="highlight"><pre><code class="language-" data-lang=""><dependency>
- <groupId>org.apache.storm</groupId>
- <artifactId>storm-metrics</artifactId>
- <version>1.0.0</version>
-</dependency>
+<div class="highlight"><pre><code class="language-" data-lang=""> <dependency>
+ <groupId>org.apache.storm</groupId>
+ <artifactId>storm-metrics</artifactId>
+ <version>1.0.0</version>
+ </dependency>
</code></pre></div>
<p>as a topology dependency (1.0.0 or higher).</p>
<p>You can then go to your topology on the UI, turn on the system metrics, and find the log that the LoggingMetricsConsumer is writing to. It will output results in the log like.</p>
-<div class="highlight"><pre><code class="language-" data-lang="">1454526100 node1.nodes.com:6707 -1:__system CPU {user-ms=74480, sys-ms=10780}
-1454526100 node1.nodes.com:6707 -1:__system memory/nonHeap {unusedBytes=2077536, virtualFreeBytes=-64621729, initBytes=2555904, committedBytes=66699264, maxBytes=-1, usedBytes=64621728}
-1454526100 node1.nodes.com:6707 -1:__system memory/heap {unusedBytes=573861408, virtualFreeBytes=694644256, initBytes=805306368, committedBytes=657719296, maxBytes=778502144, usedBytes=83857888}
+<div class="highlight"><pre><code class="language-" data-lang=""> 1454526100 node1.nodes.com:6707 -1:__system CPU {user-ms=74480, sys-ms=10780}
+ 1454526100 node1.nodes.com:6707 -1:__system memory/nonHeap {unusedBytes=2077536, virtualFreeBytes=-64621729, initBytes=2555904, committedBytes=66699264, maxBytes=-1, usedBytes=64621728}
+ 1454526100 node1.nodes.com:6707 -1:__system memory/heap {unusedBytes=573861408, virtualFreeBytes=694644256, initBytes=805306368, committedBytes=657719296, maxBytes=778502144, usedBytes=83857888}
</code></pre></div>
<p>The metrics with -1:__system are generally metrics for the entire worker. In the example above that worker is running on node1.nodes.com:6707. These metrics are collected every 60 seconds. For the CPU you can see that over the 60 seconds this worker used 74480 + 10780 = 85260 ms of CPU time. This is equivalent to 85260/60000 or about 1.5 cores.</p>
@@ -456,7 +529,7 @@ conf.put(Config.TOPOLOGY_WORKER_METRICS, workerMetrics);
<p><div id='Enhancements-on-original-DefaultResourceAwareStrategy'/></p>
-<h2 id="enhancements-on-original-defaultresourceawarestrategy">* Enhancements on original DefaultResourceAwareStrategy *</h2>
+<h2 id="enhancements-on-original-defaultresourceawarestrategy">Enhancements on original DefaultResourceAwareStrategy</h2>
<p>The default resource aware scheduling strategy as described in the paper above has two main scheduling phases:</p>
@@ -469,11 +542,11 @@ conf.put(Config.TOPOLOGY_WORKER_METRICS, workerMetrics);
<h3 id="task-selection-enhancements">Task Selection Enhancements</h3>
-<p>Instead of using a breadth first traversal of the topology graph to create a ordering of components and its executors, a new heuristic is used that orders components by the number of in and out edges (potential connections) of the component. This is discovered to be a more effective way to colocate executors that communicate with each other and reduce the network latency.</p>
+<p>Instead of using a breadth first traversal of the topology graph to create a ordering of components and its executors, a new heuristic is used that orders components by the number of in and out edges (potential connections) of the component. This is discovered to be a more effective way to co-locate executors that communicate with each other and reduce the network latency.</p>
<h3 id="node-selection-enhancements">Node Selection Enhancements</h3>
-<p>Node selection comes down first selecting which rack (server rack) and then which node on that rack to choose. The gist of strategy in choosing a rack and node is finding the rack that has the "most" resource available and in that rack find the node with the "most" free resources. The assumption we are making for this strategy is that the node or rack with the most free resources will have the highest probability that allows us to schedule colocate the most number of executors on the node or rack to reduce network communication latency</p>
+<p>Node selection comes down first selecting which rack (server rack) and then which node on that rack to choose. The gist of strategy in choosing a rack and node is finding the rack that has the "most" resource available and in that rack find the node with the "most" free resources. The assumption we are making for this strategy is that the node or rack with the most free resources will have the highest probability that allows us to schedule co-locate the most number of executors on the node or rack to reduce network communication latency</p>
<p>Racks and nodes will be sorted from best choice to worst choice. When finding an executor, the strategy will iterate through all racks and nodes, starting from best to worst, before giving up. Racks and nodes will be sorted in the following matter:</p>
@@ -483,57 +556,58 @@ conf.put(Config.TOPOLOGY_WORKER_METRICS, workerMetrics);
<li><p>Subordinate resource availability or the amount "effective" resources on the rack or node<br>
-- Please refer the section on Subordinate Resource Availability</p></li>
<li><p>Average of the all the resource availability<br>
--- This is simply taking the average of the percent available (available resources on node or rack divied by theavailable resources on rack or cluster, repectively). This situation will only be used when "effective resources" for two objects (rack or node) are the same. Then we consider the average of all the percentages of resources as a metric for sorting. For example:</p>
-<div class="highlight"><pre><code class="language-" data-lang="">Avail Resources:
-node 1: CPU = 50 Memory = 1024 Slots = 20
-node 2: CPU = 50 Memory = 8192 Slots = 40
-node 3: CPU = 1000 Memory = 0 Slots = 0
-
-Effective resources for nodes:
+-- This is simply taking the average of the percent available (available resources on node or rack divided by the available resources on rack or cluster, respectively). This situation will only be used when "effective resources" for two objects (rack or node) are the same. Then we consider the average of all the percentages of resources as a metric for sorting. For example:
+```
+ Avail Resources:
+ node 1: CPU = 50 Memory = 1024 Slots = 20
+ node 2: CPU = 50 Memory = 8192 Slots = 40
+ node 3: CPU = 1000 Memory = 0 Slots = 0</p>
+<div class="highlight"><pre><code class="language-" data-lang="">Effective resources for nodes:
node 1 = 50 / (50+50+1000) = 0.045 (CPU bound)
node 2 = 50 / (50+50+1000) = 0.045 (CPU bound)
node 3 = 0 (memory and slots are 0)
+</code></pre></div><div class="highlight"><pre><code class="language-" data-lang="">ode 1 and node 2 have the same effective resources but clearly node 2 has more resources (memory and slots) than node 1 and we would want to pick node 2 first since there is a higher probability we will be able to schedule more executors on it. This is what the phase 2 averaging does
</code></pre></div></li>
</ol>
-<p>ode 1 and node 2 have the same effective resources but clearly node 2 has more resources (memory and slots) than node 1 and we would want to pick node 2 first since there is a higher probability we will be able to schedule more executors on it. This is what the phase 2 averaging does</p>
-
-<p>Thus the sorting follows the following progression. Compare based on 1) and if equal then compare based on 2) and if equal compare based on 3) and if equal we break ties by arbitrarly assigning ordering based on comparing the ids of the node or rack.</p>
+<p>Thus the sorting follows the following progression. Compare based on 1) and if equal then compare based on 2) and if equal compare based on 3) and if equal we break ties by arbitrarily assigning ordering based on comparing the ids of the node or rack.</p>
<p><strong>Subordinate Resource Availability</strong></p>
-<p>Originally the getBestClustering algorithm for RAS finds the "Best" rack based on which rack has the "most available" resources by finding the rack with the biggest sum of available memory + available across all nodes in the rack. This method is not very accurate since memory and cpu usage aree values on a different scale and the values are not normailized. This method is also not effective since it does not consider the number of slots available and it fails to identifying racks that are not schedulable due to the exhaustion of one of the resources either memory, cpu, or slots. Also the previous method does not consider failures of workers. When executors of a topology gets unassigned and needs to be scheduled again, the current logic in getBestClustering may be inadequate since it will likely return a cluster that is different from where the majority of executors from the topology is originally scheduling in.</p>
+<p>Originally the getBestClustering algorithm for RAS finds the "Best" rack based on which rack has the "most available" resources by finding the rack with the biggest sum of available memory + available across all nodes in the rack. This method is not very accurate since memory and cpu usage agree values on a different scale and the values are not normalized. This method is also not effective since it does not consider the number of slots available and it fails to identifying racks that are not schedulable due to the exhaustion of one of the resources either memory, cpu, or slots. Also the previous method does not consider failures of workers. When executors of a topology gets unassigned and needs to be scheduled again, the current logic in getBestClustering may be inadequate since it will likely return a cluster that is different from where the majority of executors from the topology is originally scheduling in.</p>
<p>The new strategy/algorithm to find the "best" rack or node, I dub subordinate resource availability ordering (inspired by Dominant Resource Fairness), sorts racks and nodes by the subordinate (not dominant) resource availability.</p>
-<p>For example given 4 racks with the following resource availabilities</p>
-<div class="highlight"><pre><code class="language-" data-lang="">//generate some that has alot of memory but little of cpu
-rack-3 Avail [ CPU 100.0 MEM 200000.0 Slots 40 ] Total [ CPU 100.0 MEM 200000.0 Slots 40 ]
-//generate some supervisors that are depleted of one resource
-rack-2 Avail [ CPU 0.0 MEM 80000.0 Slots 40 ] Total [ CPU 0.0 MEM 80000.0 Slots 40 ]
-//generate some that has a lot of cpu but little of memory
-rack-4 Avail [ CPU 6100.0 MEM 10000.0 Slots 40 ] Total [ CPU 6100.0 MEM 10000.0 Slots 40 ]
-//generate another rack of supervisors with less resources than rack-0
-rack-1 Avail [ CPU 2000.0 MEM 40000.0 Slots 40 ] Total [ CPU 2000.0 MEM 40000.0 Slots 40 ]
-//best rack to choose
-rack-0 Avail [ CPU 4000.0 MEM 80000.0 Slots 40( ] Total [ CPU 4000.0 MEM 80000.0 Slots 40 ]
-Cluster Overall Avail [ CPU 12200.0 MEM 410000.0 Slots 200 ] Total [ CPU 12200.0 MEM 410000.0 Slots 200 ]
-</code></pre></div>
-<p>It is clear that rack-0 is the best cluster since its the most balanced and can potentially schedule the most executors, while rack-2 is the worst rack since rack-2 is depleted of cpu resource thus rendering it unschedulable even though there are other resources available.</p>
+<p>For example given 4 racks with the following resource availabilities
+<code>
+ //generate some that has a lot of memory but little of cpu
+ rack-3 Avail [ CPU 100.0 MEM 200000.0 Slots 40 ] Total [ CPU 100.0 MEM 200000.0 Slots 40 ]
+ //generate some supervisors that are depleted of one resource
+ rack-2 Avail [ CPU 0.0 MEM 80000.0 Slots 40 ] Total [ CPU 0.0 MEM 80000.0 Slots 40 ]
+ //generate some that has a lot of cpu but little of memory
+ rack-4 Avail [ CPU 6100.0 MEM 10000.0 Slots 40 ] Total [ CPU 6100.0 MEM 10000.0 Slots 40 ]
+ //generate another rack of supervisors with less resources than rack-0
+ rack-1 Avail [ CPU 2000.0 MEM 40000.0 Slots 40 ] Total [ CPU 2000.0 MEM 40000.0 Slots 40 ]
+ //best rack to choose
+ rack-0 Avail [ CPU 4000.0 MEM 80000.0 Slots 40( ] Total [ CPU 4000.0 MEM 80000.0 Slots 40 ]
+ Cluster Overall Avail [ CPU 12200.0 MEM 410000.0 Slots 200 ] Total [ CPU 12200.0 MEM 410000.0 Slots 200 ]
+</code>
+It is clear that rack-0 is the best cluster since its the most balanced and can potentially schedule the most executors, while rack-2 is the worst rack since rack-2 is depleted of cpu resource thus rendering it unschedulable even though there are other resources available.</p>
<p>We first calculate the resource availability percentage of all the racks for each resource by computing:</p>
<div class="highlight"><pre><code class="language-" data-lang="">(resource available on rack) / (resource available in cluster)
</code></pre></div>
<p>We do this calculation to normalize the values otherwise the resource values would not be comparable.</p>
-<p>So for our example:</p>
-<div class="highlight"><pre><code class="language-" data-lang="">rack-3 Avail [ CPU 0.819672131147541% MEM 48.78048780487805% Slots 20.0% ] effective resources: 0.00819672131147541
-rack-2 Avail [ 0.0% MEM 19.51219512195122% Slots 20.0% ] effective resources: 0.0
-rack-4 Avail [ CPU 50.0% MEM 2.4390243902439024% Slots 20.0% ] effective resources: 0.024390243902439025
-rack-1 Avail [ CPU 16.39344262295082% MEM 9.75609756097561% Slots 20.0% ] effective resources: 0.0975609756097561
-rack-0 Avail [ CPU 32.78688524590164% MEM 19.51219512195122% Slots 20.0% ] effective resources: 0.1951219512195122
-</code></pre></div>
-<p>The effective resource of a rack, which is also the subordinate resource, is computed by: </p>
+<p>So for our example:
+<code>
+ rack-3 Avail [ CPU 0.819672131147541% MEM 48.78048780487805% Slots 20.0% ] effective resources: 0.00819672131147541
+ rack-2 Avail [ 0.0% MEM 19.51219512195122% Slots 20.0% ] effective resources: 0.0
+ rack-4 Avail [ CPU 50.0% MEM 2.4390243902439024% Slots 20.0% ] effective resources: 0.024390243902439025
+ rack-1 Avail [ CPU 16.39344262295082% MEM 9.75609756097561% Slots 20.0% ] effective resources: 0.0975609756097561
+ rack-0 Avail [ CPU 32.78688524590164% MEM 19.51219512195122% Slots 20.0% ] effective resources: 0.1951219512195122
+</code>
+The effective resource of a rack, which is also the subordinate resource, is computed by: </p>
<div class="highlight"><pre><code class="language-" data-lang="">MIN(resource availability percentage of {CPU, Memory, # of free Slots}).
</code></pre></div>
<p>Then we order the racks by the effective resource.</p>
@@ -564,7 +638,7 @@ rack-0 Avail [ CPU 32.78688524590164% MEM 19.51219512195122% Slots 20.0% ] effec
<p>For this network metric, the larger the number is number is the more potential network latency the topology will have for this scheduling. Two types of experiments are performed. One set experiments are performed with randomly generated topologies and randomly generate clusters. The other set of experiments are performed with a dataset containing all of the running topologies at yahoo and semi-randomly generated clusters based on the size of the topology. Both set of experiments are run millions of iterations until results converge. </p>
-<p>For the experiments involving randomly generated topologies, an optimal strategy is implemented that exhausively finds the most optimal solution if a solution exists. The topologies and clusters used in this experiment are relatively small so that the optimal strategy traverse to solution space to find a optimal solution in a reasonable amount of time. This strategy is not run with the Yahoo topologies since the topologies are large and would take unreasonable amount of time to run, since the solutions space is W^N (ordering doesn't matter within a worker) where W is the number of workers and N is the number of executors. The NextGenStrategy represents the scheduling strategy with these enhancements. The DefaultResourceAwareStrategy represents the original scheduling strategy. The RoundRobinStrategy represents a naive strategy that simply schedules executors in a round robin fashion while respecting the resource constraints. The graph below presents averages of the netwo
rk metric. A CDF graph is also presented further down.</p>
+<p>For the experiments involving randomly generated topologies, an optimal strategy is implemented that exhaustively finds the most optimal solution if a solution exists. The topologies and clusters used in this experiment are relatively small so that the optimal strategy traverse to solution space to find a optimal solution in a reasonable amount of time. This strategy is not run with the Yahoo topologies since the topologies are large and would take unreasonable amount of time to run, since the solutions space is W^N (ordering doesn't matter within a worker) where W is the number of workers and N is the number of executors. The NextGenStrategy represents the scheduling strategy with these enhancements. The DefaultResourceAwareStrategy represents the original scheduling strategy. The RoundRobinStrategy represents a naive strategy that simply schedules executors in a round robin fashion while respecting the resource constraints. The graph below presents averages of the netw
ork metric. A CDF graph is also presented further down.</p>
<table><thead>
<tr>
http://git-wip-us.apache.org/repos/asf/storm-site/blob/0183bf7e/content/releases/2.0.0-SNAPSHOT/Running-topologies-on-a-production-cluster.html
----------------------------------------------------------------------
diff --git a/content/releases/2.0.0-SNAPSHOT/Running-topologies-on-a-production-cluster.html b/content/releases/2.0.0-SNAPSHOT/Running-topologies-on-a-production-cluster.html
index 8f87ce3..0cfa1a1 100644
--- a/content/releases/2.0.0-SNAPSHOT/Running-topologies-on-a-production-cluster.html
+++ b/content/releases/2.0.0-SNAPSHOT/Running-topologies-on-a-production-cluster.html
@@ -154,7 +154,7 @@
<span class="n">conf</span><span class="o">.</span><span class="na">setMaxSpoutPending</span><span class="o">(</span><span class="mi">5000</span><span class="o">);</span>
<span class="n">StormSubmitter</span><span class="o">.</span><span class="na">submitTopology</span><span class="o">(</span><span class="s">"mytopology"</span><span class="o">,</span> <span class="n">conf</span><span class="o">,</span> <span class="n">topology</span><span class="o">);</span>
</code></pre></div>
-<p>3) Create a jar containing your code and all the dependencies of your code (except for Storm -- the Storm jars will be added to the classpath on the worker nodes).</p>
+<p>3) Create a JAR containing your topology code. You have the option to either bundle all of the dependencies of your code into that JAR (except for Storm -- the Storm JARs will be added to the classpath on the worker nodes), or you can leverage the <a href="Classpath-handling.html">Classpath handling</a> features in Storm for using external libraries without bundling them into your topology JAR.</p>
<p>If you're using Maven, the <a href="http://maven.apache.org/plugins/maven-assembly-plugin/">Maven Assembly Plugin</a> can do the packaging for you. Just add this to your pom.xml:</p>
<div class="highlight"><pre><code class="language-xml" data-lang="xml"> <span class="nt"><plugin></span>
http://git-wip-us.apache.org/repos/asf/storm-site/blob/0183bf7e/content/releases/2.0.0-SNAPSHOT/SECURITY.html
----------------------------------------------------------------------
diff --git a/content/releases/2.0.0-SNAPSHOT/SECURITY.html b/content/releases/2.0.0-SNAPSHOT/SECURITY.html
index 4901530..4ceb90e 100644
--- a/content/releases/2.0.0-SNAPSHOT/SECURITY.html
+++ b/content/releases/2.0.0-SNAPSHOT/SECURITY.html
@@ -157,6 +157,9 @@ Authentication and Authorization. But to do so usually requires
configuring your Operating System to restrict the operations that can be done.
This is generally a good idea even if you plan on running your cluster with Auth.</p>
+<p>Storm's OS level security relies on running Storm processes using OS accounts that have only the permissions they need.
+Note that workers run under the same OS account as the Supervisor daemon by default.</p>
+
<p>The exact detail of how to setup these precautions varies a lot and is beyond
the scope of this document.</p>
@@ -257,7 +260,8 @@ ui.filter.params:
"kerberos.keytab": "/vagrant/keytabs/http.keytab"
"kerberos.name.rules": "RULE:[2:$1@$0]([jt]t@.*EXAMPLE.COM)s/.*/$MAPRED_USER/ RULE:[2:$1@$0]([nd]n@.*EXAMPLE.COM)s/.*/$HDFS_USER/DEFAULT"
</code>
-make sure to create a principal 'HTTP/{hostname}' (here hostname should be the one where UI daemon runs</p>
+make sure to create a principal 'HTTP/{hostname}' (here hostname should be the one where UI daemon runs
+Be aware that the UI user <em>MUST</em> be HTTP.</p>
<p>Once configured users needs to do kinit before accessing UI.
Ex:
@@ -273,9 +277,9 @@ curl -i --negotiate -u:anyUser -b ~/cookiejar.txt -c ~/cookiejar.txt <a href=
<p><strong>Caution</strong>: In AD MIT Keberos setup the key size is bigger than the default UI jetty server request header size. Make sure you set ui.header.buffer.bytes to 65536 in storm.yaml. More details are on <a href="https://issues.apache.org/jira/browse/STORM-633">STORM-633</a></p>
-<h2 id="ui-drpc-ssl">UI / DRPC SSL</h2>
+<h2 id="ui-drpc-logviewer-ssl">UI / DRPC / LOGVIEWER SSL</h2>
-<p>Both UI and DRPC allows users to configure ssl .</p>
+<p>UI,DRPC and LOGVIEWER allows users to configure ssl .</p>
<h3 id="ui">UI</h3>
@@ -319,6 +323,27 @@ curl -i --negotiate -u:anyUser -b ~/cookiejar.txt -c ~/cookiejar.txt <a href=
9. drpc.https.want.client.auth (If this set to true server requests for client certifcate authentication, but keeps the connection if no authentication provided)
10. drpc.https.need.client.auth (If this set to true server requires client to provide authentication)</p>
+<h3 id="logviewer">LOGVIEWER</h3>
+
+<p>similarly to UI and DRPC , users need to configure following for LOGVIEWER</p>
+
+<ol>
+<li>logviewer.https.port </li>
+<li>logviewer.https.keystore.type (example "jks")</li>
+<li>logviewer.https.keystore.path (example "/etc/ssl/storm_keystore.jks")</li>
+<li>logviewer.https.keystore.password (keystore password)</li>
+<li>logviewer.https.key.password (private key password)</li>
+</ol>
+
+<p>optional config
+6. logviewer.https.truststore.path (example "/etc/ssl/storm_truststore.jks")
+7. logviewer.https.truststore.password (truststore password)
+8. logviewer.https.truststore.type (example "jks")</p>
+
+<p>If users want to setup 2-way auth
+9. logviewer.https.want.client.auth (If this set to true server requests for client certifcate authentication, but keeps the connection if no authentication provided)
+10. logviewer.https.need.client.auth (If this set to true server requires client to provide authentication)</p>
+
<h2 id="authentication-kerberos">Authentication (Kerberos)</h2>
<p>Storm offers pluggable authentication support through thrift and SASL. This
@@ -573,16 +598,20 @@ nimbus.impersonation.acl:
<h3 id="automatic-credentials-push-and-renewal">Automatic Credentials Push and Renewal</h3>
<p>Individual topologies have the ability to push credentials (tickets and tokens) to workers so that they can access secure services. Exposing this to all of the users can be a pain for them.
-To hide this from them in the common case plugins can be used to populate the credentials, unpack them on the other side into a java Subject, and also allow Nimbus to renew the credentials if needed.
-These are controlled by the following configs. topology.auto-credentials is a list of java plugins, all of which must implement IAutoCredentials interface, that populate the credentials on gateway
-and unpack them on the worker side. On a kerberos secure cluster they should be set by default to point to org.apache.storm.security.auth.kerberos.AutoTGT.<br>
-nimbus.credential.renewers.classes should also be set to this value so that nimbus can periodically renew the TGT on behalf of the user.</p>
+To hide this from them in the common case plugins can be used to populate the credentials, unpack them on the other side into a java Subject, and also allow Nimbus to renew the credentials if needed. These are controlled by the following configs.</p>
+
+<p><code>topology.auto-credentials</code> is a list of java plugins, all of which must implement the <code>IAutoCredentials</code> interface, that populate the credentials on gateway
+and unpack them on the worker side. On a kerberos secure cluster they should be set by default to point to <code>org.apache.storm.security.auth.kerberos.AutoTGT</code></p>
+
+<p><code>nimbus.credential.renewers.classes</code> should also be set to <code>org.apache.storm.security.auth.kerberos.AutoTGT</code> so that nimbus can periodically renew the TGT on behalf of the user. </p>
-<p>nimbus.credential.renewers.freq.secs controls how often the renewer will poll to see if anything needs to be renewed, but the default should be fine.</p>
+<p>All autocredential classes that desire to implement the IMetricsRegistrant interface can register metrics automatically for each topology. The AutoTGT class currently implements this interface and adds a metric named TGT-TimeToExpiryMsecs showing the remaining time until the TGT needs to be renewed.</p>
-<p>In addition Nimbus itself can be used to get credentials on behalf of the user submitting topologies. This can be configures using nimbus.autocredential.plugins.classes which is a list
-of fully qualified class names ,all of which must implement INimbusCredentialPlugin. Nimbus will invoke the populateCredentials method of all the configured implementation as part of topology
-submission. You should use this config with topology.auto-credentials and nimbus.credential.renewers.classes so the credentials can be populated on worker side and nimbus can automatically renew
+<p><code>nimbus.credential.renewers.freq.secs</code> controls how often the renewer will poll to see if anything needs to be renewed, but the default should be fine.</p>
+
+<p>In addition Nimbus itself can be used to get credentials on behalf of the user submitting topologies. This can be configured using <code>nimbus.autocredential.plugins.classes</code> which is a list
+of fully qualified class names, all of which must implement <code>INimbusCredentialPlugin</code>. Nimbus will invoke the populateCredentials method of all the configured implementation as part of topology
+submission. You should use this config with <code>topology.auto-credentials</code> and <code>nimbus.credential.renewers.classes</code> so the credentials can be populated on worker side and nimbus can automatically renew
them. Currently there are 2 examples of using this config, AutoHDFS and AutoHBase which auto populates hdfs and hbase delegation tokens for topology submitter so they don't have to distribute keytabs
on all possible worker hosts.</p>
@@ -642,7 +671,44 @@ on all possible worker hosts.</p>
</code></pre></div>
<h3 id="drpc">DRPC</h3>
-<p>Hopefully more on this soon</p>
+<p>Storm provides the Access Control List for the DRPC Authorizer.Users can see <a href="javadocs/org/apache/storm/security/auth/authorizer/DRPCSimpleACLAuthorizer.html">org.apache.storm.security.auth.authorizer.DRPCSimpleACLAuthorizer</a> for more details.</p>
+
+<p>There are several DRPC ACL related configurations.</p>
+
+<p>| YAML Setting | Description |
+ |------------|----------------------|
+ | drpc.authorizer.acl | A class that will perform authorization for DRPC operations. Set this to org.apache.storm.security.auth.authorizer.DRPCSimpleACLAuthorizer when using security.|
+ | drpc.authorizer.acl.filename | This is the name of a file that the ACLs will be loaded from. It is separate from storm.yaml to allow the file to be updated without bringing down a DRPC server. Defaults to drpc-auth-acl.yaml |
+ | drpc.authorizer.acl.strict| It is useful to set this to false for staging where users may want to experiment, but true for production where you want users to be secure. Defaults to false. |</p>
+
+<p>The file pointed to by drpc.authorizer.acl.filename will have only one config in it drpc.authorizer.acl this should be of the form</p>
+
+<p>drpc.authorizer.acl:
+ "functionName1":
+ "client.users":
+ - "alice"
+ - "bob"
+ "invocation.user": "bob"</p>
+
+<p>In this the users bob and alice as client.users are allowed to run DRPC requests against functionName1, but only bob as the invocation.user is allowed to run the topology that actually processes those requests.</p>
+
+<h2 id="cluster-zookeeper-authentication">Cluster Zookeeper Authentication</h2>
+
+<p>Users can implement cluster Zookeeper authentication by setting several configurations are shown below.</p>
+
+<p>| YAML Setting | Description |
+ |------------|----------------------|
+ | storm.zookeeper.auth.scheme | The cluster Zookeeper authentication scheme to use, e.g. "digest". Defaults to no authentication. |
+ | storm.zookeeper.auth.payload | A string representing the payload for cluster Zookeeper authentication.It should only be set in the storm-cluster-auth.yaml.Users can see storm-cluster-auth.yaml.example for more details. |</p>
+
+<p>Also,there are several configurations for topology Zookeeper authentication:</p>
+
+<p>| YAML Setting | Description |
+ |------------|----------------------|
+ | storm.zookeeper.topology.auth.scheme | The topology Zookeeper authentication scheme to use, e.g. "digest". It is the internal config and user shouldn't set it. |
+ | storm.zookeeper.topology.auth.payload | A string representing the payload for topology Zookeeper authentication. |</p>
+
+<p>Note: If storm.zookeeper.topology.auth.payload isn't set,storm will generate a ZooKeeper secret payload for MD5-digest with generateZookeeperDigestSecretPayload() method.</p>