You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@storm.apache.org by pt...@apache.org on 2017/09/15 17:00:21 UTC
[42/51] [partial] storm-site git commit: test git pubsub
http://git-wip-us.apache.org/repos/asf/storm-site/blob/3730cb62/content/releases/0.10.0/Rationale.html
----------------------------------------------------------------------
diff --git a/content/releases/0.10.0/Rationale.html b/content/releases/0.10.0/Rationale.html
deleted file mode 100644
index 484ab1d..0000000
--- a/content/releases/0.10.0/Rationale.html
+++ /dev/null
@@ -1,274 +0,0 @@
-<!DOCTYPE html>
-<html>
- <head>
- <meta charset="utf-8">
- <meta http-equiv="X-UA-Compatible" content="IE=edge">
- <meta name="viewport" content="width=device-width, initial-scale=1">
-
- <link rel="shortcut icon" href="/favicon.ico" type="image/x-icon">
- <link rel="icon" href="/favicon.ico" type="image/x-icon">
-
- <title>Rationale</title>
-
- <!-- Bootstrap core CSS -->
- <link href="/assets/css/bootstrap.min.css" rel="stylesheet">
- <!-- Bootstrap theme -->
- <link href="/assets/css/bootstrap-theme.min.css" rel="stylesheet">
-
- <!-- Custom styles for this template -->
- <link rel="stylesheet" href="http://fortawesome.github.io/Font-Awesome/assets/font-awesome/css/font-awesome.css">
- <link href="/css/style.css" rel="stylesheet">
- <link href="/assets/css/owl.theme.css" rel="stylesheet">
- <link href="/assets/css/owl.carousel.css" rel="stylesheet">
- <script type="text/javascript" src="/assets/js/jquery.min.js"></script>
- <script type="text/javascript" src="/assets/js/bootstrap.min.js"></script>
- <script type="text/javascript" src="/assets/js/owl.carousel.min.js"></script>
- <script type="text/javascript" src="/assets/js/storm.js"></script>
- <!-- Just for debugging purposes. Don't actually copy these 2 lines! -->
- <!--[if lt IE 9]><script src="../../assets/js/ie8-responsive-file-warning.js"></script><![endif]-->
-
- <!-- HTML5 shim and Respond.js for IE8 support of HTML5 elements and media queries -->
- <!--[if lt IE 9]>
- <script src="https://oss.maxcdn.com/html5shiv/3.7.2/html5shiv.min.js"></script>
- <script src="https://oss.maxcdn.com/respond/1.4.2/respond.min.js"></script>
- <![endif]-->
- </head>
-
-
- <body>
- <header>
- <div class="container-fluid">
- <div class="row">
- <div class="col-md-5">
- <a href="/index.html"><img src="/images/logo.png" class="logo" /></a>
- </div>
- <div class="col-md-5">
-
- <h1>Version: 0.10.0</h1>
-
- </div>
- <div class="col-md-2">
- <a href="/downloads.html" class="btn-std btn-block btn-download">Download</a>
- </div>
- </div>
- </div>
-</header>
-<!--Header End-->
-<!--Navigation Begin-->
-<div class="navbar" role="banner">
- <div class="container-fluid">
- <div class="navbar-header">
- <button class="navbar-toggle" type="button" data-toggle="collapse" data-target=".bs-navbar-collapse">
- <span class="icon-bar"></span>
- <span class="icon-bar"></span>
- <span class="icon-bar"></span>
- </button>
- </div>
- <nav class="collapse navbar-collapse bs-navbar-collapse" role="navigation">
- <ul class="nav navbar-nav">
- <li><a href="/index.html" id="home">Home</a></li>
- <li><a href="/getting-help.html" id="getting-help">Getting Help</a></li>
- <li><a href="/about/integrates.html" id="project-info">Project Information</a></li>
- <li class="dropdown">
- <a href="#" class="dropdown-toggle" data-toggle="dropdown" id="documentation">Documentation <b class="caret"></b></a>
- <ul class="dropdown-menu">
-
-
- <li><a href="/releases/2.0.0-SNAPSHOT/index.html">2.0.0-SNAPSHOT</a></li>
-
-
-
- <li><a href="/releases/1.1.0/index.html">1.1.0</a></li>
-
-
-
-
-
- <li><a href="/releases/1.0.4/index.html">1.0.4</a></li>
-
-
-
-
-
- <li><a href="/releases/1.0.3/index.html">1.0.3</a></li>
-
-
-
- <li><a href="/releases/1.0.2/index.html">1.0.2</a></li>
-
-
-
- <li><a href="/releases/1.0.1/index.html">1.0.1</a></li>
-
-
-
- <li><a href="/releases/1.0.0/index.html">1.0.0</a></li>
-
-
-
- <li><a href="/releases/0.10.2/index.html">0.10.2</a></li>
-
-
-
- <li><a href="/releases/0.10.1/index.html">0.10.1</a></li>
-
-
-
-
-
- <li><a href="/releases/0.10.0/index.html">0.10.0</a></li>
-
-
-
- <li><a href="/releases/0.9.7/index.html">0.9.7</a></li>
-
-
-
- <li><a href="/releases/0.9.6/index.html">0.9.6</a></li>
-
-
-
-
-
-
-
-
-
-
-
-
- </ul>
- </li>
- <li><a href="/talksAndVideos.html">Talks and Slideshows</a></li>
- <li class="dropdown">
- <a href="#" class="dropdown-toggle" data-toggle="dropdown" id="contribute">Community <b class="caret"></b></a>
- <ul class="dropdown-menu">
- <li><a href="/contribute/Contributing-to-Storm.html">Contributing</a></li>
- <li><a href="/contribute/People.html">People</a></li>
- <li><a href="/contribute/BYLAWS.html">ByLaws</a></li>
- </ul>
- </li>
- <li><a href="/2017/07/28/storm104-released.html" id="news">News</a></li>
- </ul>
- </nav>
- </div>
-</div>
-
-
-
- <div class="container-fluid">
- <h1 class="page-title">Rationale</h1>
- <div class="row">
- <div class="col-md-12">
- <!-- Documentation -->
-
-<p class="post-meta"></p>
-
-<p>The past decade has seen a revolution in data processing. MapReduce, Hadoop, and related technologies have made it possible to store and process data at scales previously unthinkable. Unfortunately, these data processing technologies are not realtime systems, nor are they meant to be. There's no hack that will turn Hadoop into a realtime system; realtime data processing has a fundamentally different set of requirements than batch processing.</p>
-
-<p>However, realtime data processing at massive scale is becoming more and more of a requirement for businesses. The lack of a "Hadoop of realtime" has become the biggest hole in the data processing ecosystem.</p>
-
-<p>Storm fills that hole.</p>
-
-<p>Before Storm, you would typically have to manually build a network of queues and workers to do realtime processing. Workers would process messages off a queue, update databases, and send new messages to other queues for further processing. Unfortunately, this approach has serious limitations:</p>
-
-<ol>
-<li><strong>Tedious</strong>: You spend most of your development time configuring where to send messages, deploying workers, and deploying intermediate queues. The realtime processing logic that you care about corresponds to a relatively small percentage of your codebase.</li>
-<li><strong>Brittle</strong>: There's little fault-tolerance. You're responsible for keeping each worker and queue up.</li>
-<li><strong>Painful to scale</strong>: When the message throughput get too high for a single worker or queue, you need to partition how the data is spread around. You need to reconfigure the other workers to know the new locations to send messages. This introduces moving parts and new pieces that can fail.</li>
-</ol>
-
-<p>Although the queues and workers paradigm breaks down for large numbers of messages, message processing is clearly the fundamental paradigm for realtime computation. The question is: how do you do it in a way that doesn't lose data, scales to huge volumes of messages, and is dead-simple to use and operate?</p>
-
-<p>Storm satisfies these goals. </p>
-
-<h2 id="why-storm-is-important">Why Storm is important</h2>
-
-<p>Storm exposes a set of primitives for doing realtime computation. Like how MapReduce greatly eases the writing of parallel batch processing, Storm's primitives greatly ease the writing of parallel realtime computation.</p>
-
-<p>The key properties of Storm are:</p>
-
-<ol>
-<li><strong>Extremely broad set of use cases</strong>: Storm can be used for processing messages and updating databases (stream processing), doing a continuous query on data streams and streaming the results into clients (continuous computation), parallelizing an intense query like a search query on the fly (distributed RPC), and more. Storm's small set of primitives satisfy a stunning number of use cases.</li>
-<li><strong>Scalable</strong>: Storm scales to massive numbers of messages per second. To scale a topology, all you have to do is add machines and increase the parallelism settings of the topology. As an example of Storm's scale, one of Storm's initial applications processed 1,000,000 messages per second on a 10 node cluster, including hundreds of database calls per second as part of the topology. Storm's usage of Zookeeper for cluster coordination makes it scale to much larger cluster sizes.</li>
-<li><strong>Guarantees no data loss</strong>: A realtime system must have strong guarantees about data being successfully processed. A system that drops data has a very limited set of use cases. Storm guarantees that every message will be processed, and this is in direct contrast with other systems like S4. </li>
-<li><strong>Extremely robust</strong>: Unlike systems like Hadoop, which are notorious for being difficult to manage, Storm clusters just work. It is an explicit goal of the Storm project to make the user experience of managing Storm clusters as painless as possible.</li>
-<li><strong>Fault-tolerant</strong>: If there are faults during execution of your computation, Storm will reassign tasks as necessary. Storm makes sure that a computation can run forever (or until you kill the computation).</li>
-<li><strong>Programming language agnostic</strong>: Robust and scalable realtime processing shouldn't be limited to a single platform. Storm topologies and processing components can be defined in any language, making Storm accessible to nearly anyone.</li>
-</ol>
-
-
-
- </div>
- </div>
- </div>
-<footer>
- <div class="container-fluid">
- <div class="row">
- <div class="col-md-3">
- <div class="footer-widget">
- <h5>Meetups</h5>
- <ul class="latest-news">
-
- <li><a href="http://www.meetup.com/Apache-Storm-Apache-Kafka/">Apache Storm & Apache Kafka</a> <span class="small">(Sunnyvale, CA)</span></li>
-
- <li><a href="http://www.meetup.com/Apache-Storm-Kafka-Users/">Apache Storm & Kafka Users</a> <span class="small">(Seattle, WA)</span></li>
-
- <li><a href="http://www.meetup.com/New-York-City-Storm-User-Group/">NYC Storm User Group</a> <span class="small">(New York, NY)</span></li>
-
- <li><a href="http://www.meetup.com/Bay-Area-Stream-Processing">Bay Area Stream Processing</a> <span class="small">(Emeryville, CA)</span></li>
-
- <li><a href="http://www.meetup.com/Boston-Storm-Users/">Boston Realtime Data</a> <span class="small">(Boston, MA)</span></li>
-
- <li><a href="http://www.meetup.com/storm-london">London Storm User Group</a> <span class="small">(London, UK)</span></li>
-
- <!-- <li><a href="http://www.meetup.com/Apache-Storm-Kafka-Users/">Seatle, WA</a> <span class="small">(27 Jun 2015)</span></li> -->
- </ul>
- </div>
- </div>
- <div class="col-md-3">
- <div class="footer-widget">
- <h5>About Storm</h5>
- <p>Storm integrates with any queueing system and any database system. Storm's spout abstraction makes it easy to integrate a new queuing system. Likewise, integrating Storm with database systems is easy.</p>
- </div>
- </div>
- <div class="col-md-3">
- <div class="footer-widget">
- <h5>First Look</h5>
- <ul class="footer-list">
- <li><a href="/releases/current/Rationale.html">Rationale</a></li>
- <li><a href="/releases/current/Tutorial.html">Tutorial</a></li>
- <li><a href="/releases/current/Setting-up-development-environment.html">Setting up development environment</a></li>
- <li><a href="/releases/current/Creating-a-new-Storm-project.html">Creating a new Storm project</a></li>
- </ul>
- </div>
- </div>
- <div class="col-md-3">
- <div class="footer-widget">
- <h5>Documentation</h5>
- <ul class="footer-list">
- <li><a href="/releases/current/index.html">Index</a></li>
- <li><a href="/releases/current/javadocs/index.html">Javadoc</a></li>
- <li><a href="/releases/current/FAQ.html">FAQ</a></li>
- </ul>
- </div>
- </div>
- </div>
- <hr/>
- <div class="row">
- <div class="col-md-12">
- <p align="center">Copyright © 2015 <a href="http://www.apache.org">Apache Software Foundation</a>. All Rights Reserved.
- <br>Apache Storm, Apache, the Apache feather logo, and the Apache Storm project logos are trademarks of The Apache Software Foundation.
- <br>All other marks mentioned may be trademarks or registered trademarks of their respective owners.</p>
- </div>
- </div>
- </div>
-</footer>
-<!--Footer End-->
-<!-- Scroll to top -->
-<span class="totop"><a href="#"><i class="fa fa-angle-up"></i></a></span>
-
-</body>
-
-</html>
-
http://git-wip-us.apache.org/repos/asf/storm-site/blob/3730cb62/content/releases/0.10.0/Running-topologies-on-a-production-cluster.html
----------------------------------------------------------------------
diff --git a/content/releases/0.10.0/Running-topologies-on-a-production-cluster.html b/content/releases/0.10.0/Running-topologies-on-a-production-cluster.html
deleted file mode 100644
index a1f43e6..0000000
--- a/content/releases/0.10.0/Running-topologies-on-a-production-cluster.html
+++ /dev/null
@@ -1,310 +0,0 @@
-<!DOCTYPE html>
-<html>
- <head>
- <meta charset="utf-8">
- <meta http-equiv="X-UA-Compatible" content="IE=edge">
- <meta name="viewport" content="width=device-width, initial-scale=1">
-
- <link rel="shortcut icon" href="/favicon.ico" type="image/x-icon">
- <link rel="icon" href="/favicon.ico" type="image/x-icon">
-
- <title>Running Topologies on a Production Cluster</title>
-
- <!-- Bootstrap core CSS -->
- <link href="/assets/css/bootstrap.min.css" rel="stylesheet">
- <!-- Bootstrap theme -->
- <link href="/assets/css/bootstrap-theme.min.css" rel="stylesheet">
-
- <!-- Custom styles for this template -->
- <link rel="stylesheet" href="http://fortawesome.github.io/Font-Awesome/assets/font-awesome/css/font-awesome.css">
- <link href="/css/style.css" rel="stylesheet">
- <link href="/assets/css/owl.theme.css" rel="stylesheet">
- <link href="/assets/css/owl.carousel.css" rel="stylesheet">
- <script type="text/javascript" src="/assets/js/jquery.min.js"></script>
- <script type="text/javascript" src="/assets/js/bootstrap.min.js"></script>
- <script type="text/javascript" src="/assets/js/owl.carousel.min.js"></script>
- <script type="text/javascript" src="/assets/js/storm.js"></script>
- <!-- Just for debugging purposes. Don't actually copy these 2 lines! -->
- <!--[if lt IE 9]><script src="../../assets/js/ie8-responsive-file-warning.js"></script><![endif]-->
-
- <!-- HTML5 shim and Respond.js for IE8 support of HTML5 elements and media queries -->
- <!--[if lt IE 9]>
- <script src="https://oss.maxcdn.com/html5shiv/3.7.2/html5shiv.min.js"></script>
- <script src="https://oss.maxcdn.com/respond/1.4.2/respond.min.js"></script>
- <![endif]-->
- </head>
-
-
- <body>
- <header>
- <div class="container-fluid">
- <div class="row">
- <div class="col-md-5">
- <a href="/index.html"><img src="/images/logo.png" class="logo" /></a>
- </div>
- <div class="col-md-5">
-
- <h1>Version: 0.10.0</h1>
-
- </div>
- <div class="col-md-2">
- <a href="/downloads.html" class="btn-std btn-block btn-download">Download</a>
- </div>
- </div>
- </div>
-</header>
-<!--Header End-->
-<!--Navigation Begin-->
-<div class="navbar" role="banner">
- <div class="container-fluid">
- <div class="navbar-header">
- <button class="navbar-toggle" type="button" data-toggle="collapse" data-target=".bs-navbar-collapse">
- <span class="icon-bar"></span>
- <span class="icon-bar"></span>
- <span class="icon-bar"></span>
- </button>
- </div>
- <nav class="collapse navbar-collapse bs-navbar-collapse" role="navigation">
- <ul class="nav navbar-nav">
- <li><a href="/index.html" id="home">Home</a></li>
- <li><a href="/getting-help.html" id="getting-help">Getting Help</a></li>
- <li><a href="/about/integrates.html" id="project-info">Project Information</a></li>
- <li class="dropdown">
- <a href="#" class="dropdown-toggle" data-toggle="dropdown" id="documentation">Documentation <b class="caret"></b></a>
- <ul class="dropdown-menu">
-
-
- <li><a href="/releases/2.0.0-SNAPSHOT/index.html">2.0.0-SNAPSHOT</a></li>
-
-
-
- <li><a href="/releases/1.1.0/index.html">1.1.0</a></li>
-
-
-
-
-
- <li><a href="/releases/1.0.4/index.html">1.0.4</a></li>
-
-
-
-
-
- <li><a href="/releases/1.0.3/index.html">1.0.3</a></li>
-
-
-
- <li><a href="/releases/1.0.2/index.html">1.0.2</a></li>
-
-
-
- <li><a href="/releases/1.0.1/index.html">1.0.1</a></li>
-
-
-
- <li><a href="/releases/1.0.0/index.html">1.0.0</a></li>
-
-
-
- <li><a href="/releases/0.10.2/index.html">0.10.2</a></li>
-
-
-
- <li><a href="/releases/0.10.1/index.html">0.10.1</a></li>
-
-
-
-
-
- <li><a href="/releases/0.10.0/index.html">0.10.0</a></li>
-
-
-
- <li><a href="/releases/0.9.7/index.html">0.9.7</a></li>
-
-
-
- <li><a href="/releases/0.9.6/index.html">0.9.6</a></li>
-
-
-
-
-
-
-
-
-
-
-
-
- </ul>
- </li>
- <li><a href="/talksAndVideos.html">Talks and Slideshows</a></li>
- <li class="dropdown">
- <a href="#" class="dropdown-toggle" data-toggle="dropdown" id="contribute">Community <b class="caret"></b></a>
- <ul class="dropdown-menu">
- <li><a href="/contribute/Contributing-to-Storm.html">Contributing</a></li>
- <li><a href="/contribute/People.html">People</a></li>
- <li><a href="/contribute/BYLAWS.html">ByLaws</a></li>
- </ul>
- </li>
- <li><a href="/2017/07/28/storm104-released.html" id="news">News</a></li>
- </ul>
- </nav>
- </div>
-</div>
-
-
-
- <div class="container-fluid">
- <h1 class="page-title">Running Topologies on a Production Cluster</h1>
- <div class="row">
- <div class="col-md-12">
- <!-- Documentation -->
-
-<p class="post-meta"></p>
-
-<p>Running topologies on a production cluster is similar to running in <a href="Local-mode.html">Local mode</a>. Here are the steps:</p>
-
-<p>1) Define the topology (Use <a href="javadocs/backtype/storm/topology/TopologyBuilder.html">TopologyBuilder</a> if defining using Java)</p>
-
-<p>2) Use <a href="javadocs/backtype/storm/StormSubmitter.html">StormSubmitter</a> to submit the topology to the cluster. <code>StormSubmitter</code> takes as input the name of the topology, a configuration for the topology, and the topology itself. For example:</p>
-<div class="highlight"><pre><code class="language-java" data-lang="java"><span class="n">Config</span> <span class="n">conf</span> <span class="o">=</span> <span class="k">new</span> <span class="n">Config</span><span class="o">();</span>
-<span class="n">conf</span><span class="o">.</span><span class="na">setNumWorkers</span><span class="o">(</span><span class="mi">20</span><span class="o">);</span>
-<span class="n">conf</span><span class="o">.</span><span class="na">setMaxSpoutPending</span><span class="o">(</span><span class="mi">5000</span><span class="o">);</span>
-<span class="n">StormSubmitter</span><span class="o">.</span><span class="na">submitTopology</span><span class="o">(</span><span class="s">"mytopology"</span><span class="o">,</span> <span class="n">conf</span><span class="o">,</span> <span class="n">topology</span><span class="o">);</span>
-</code></pre></div>
-<p>3) Create a jar containing your code and all the dependencies of your code (except for Storm -- the Storm jars will be added to the classpath on the worker nodes).</p>
-
-<p>If you're using Maven, the <a href="http://maven.apache.org/plugins/maven-assembly-plugin/">Maven Assembly Plugin</a> can do the packaging for you. Just add this to your pom.xml:</p>
-<div class="highlight"><pre><code class="language-xml" data-lang="xml"> <span class="nt"><plugin></span>
- <span class="nt"><artifactId></span>maven-assembly-plugin<span class="nt"></artifactId></span>
- <span class="nt"><configuration></span>
- <span class="nt"><descriptorRefs></span>
- <span class="nt"><descriptorRef></span>jar-with-dependencies<span class="nt"></descriptorRef></span>
- <span class="nt"></descriptorRefs></span>
- <span class="nt"><archive></span>
- <span class="nt"><manifest></span>
- <span class="nt"><mainClass></span>com.path.to.main.Class<span class="nt"></mainClass></span>
- <span class="nt"></manifest></span>
- <span class="nt"></archive></span>
- <span class="nt"></configuration></span>
- <span class="nt"></plugin></span>
-</code></pre></div>
-<p>Then run mvn assembly:assembly to get an appropriately packaged jar. Make sure you <a href="http://maven.apache.org/plugins/maven-assembly-plugin/examples/single/including-and-excluding-artifacts.html">exclude</a> the Storm jars since the cluster already has Storm on the classpath.</p>
-
-<p>4) Submit the topology to the cluster using the <code>storm</code> client, specifying the path to your jar, the classname to run, and any arguments it will use:</p>
-
-<p><code>storm jar path/to/allmycode.jar org.me.MyTopology arg1 arg2 arg3</code></p>
-
-<p><code>storm jar</code> will submit the jar to the cluster and configure the <code>StormSubmitter</code> class to talk to the right cluster. In this example, after uploading the jar <code>storm jar</code> calls the main function on <code>org.me.MyTopology</code> with the arguments "arg1", "arg2", and "arg3".</p>
-
-<p>You can find out how to configure your <code>storm</code> client to talk to a Storm cluster on <a href="Setting-up-development-environment.html">Setting up development environment</a>.</p>
-
-<h3 id="common-configurations">Common configurations</h3>
-
-<p>There are a variety of configurations you can set per topology. A list of all the configurations you can set can be found <a href="javadocs/backtype/storm/Config.html">here</a>. The ones prefixed with "TOPOLOGY" can be overridden on a topology-specific basis (the other ones are cluster configurations and cannot be overridden). Here are some common ones that are set for a topology:</p>
-
-<ol>
-<li><strong>Config.TOPOLOGY_WORKERS</strong>: This sets the number of worker processes to use to execute the topology. For example, if you set this to 25, there will be 25 Java processes across the cluster executing all the tasks. If you had a combined 150 parallelism across all components in the topology, each worker process will have 6 tasks running within it as threads.</li>
-<li><strong>Config.TOPOLOGY_ACKER_EXECUTORS</strong>: This sets the number of executors that will track tuple trees and detect when a spout tuple has been fully processed. Ackers are an integral part of Storm's reliability model and you can read more about them on <a href="Guaranteeing-message-processing.html">Guaranteeing message processing</a>. By not setting this variable or setting it as null, Storm will set the number of acker executors to be equal to the number of workers configured for this topology. If this variable is set to 0, then Storm will immediately ack tuples as soon as they come off the spout, effectively disabling reliability.</li>
-<li><strong>Config.TOPOLOGY_MAX_SPOUT_PENDING</strong>: This sets the maximum number of spout tuples that can be pending on a single spout task at once (pending means the tuple has not been acked or failed yet). It is highly recommended you set this config to prevent queue explosion.</li>
-<li><strong>Config.TOPOLOGY_MESSAGE_TIMEOUT_SECS</strong>: This is the maximum amount of time a spout tuple has to be fully completed before it is considered failed. This value defaults to 30 seconds, which is sufficient for most topologies. See <a href="Guaranteeing-message-processing.html">Guaranteeing message processing</a> for more information on how Storm's reliability model works.</li>
-<li><strong>Config.TOPOLOGY_SERIALIZATIONS</strong>: You can register more serializers to Storm using this config so that you can use custom types within tuples.</li>
-</ol>
-
-<h3 id="killing-a-topology">Killing a topology</h3>
-
-<p>To kill a topology, simply run:</p>
-
-<p><code>storm kill {stormname}</code></p>
-
-<p>Give the same name to <code>storm kill</code> as you used when submitting the topology.</p>
-
-<p>Storm won't kill the topology immediately. Instead, it deactivates all the spouts so that they don't emit any more tuples, and then Storm waits Config.TOPOLOGY_MESSAGE_TIMEOUT_SECS seconds before destroying all the workers. This gives the topology enough time to complete any tuples it was processing when it got killed.</p>
-
-<h3 id="updating-a-running-topology">Updating a running topology</h3>
-
-<p>To update a running topology, the only option currently is to kill the current topology and resubmit a new one. A planned feature is to implement a <code>storm swap</code> command that swaps a running topology with a new one, ensuring minimal downtime and no chance of both topologies processing tuples at the same time. </p>
-
-<h3 id="monitoring-topologies">Monitoring topologies</h3>
-
-<p>The best place to monitor a topology is using the Storm UI. The Storm UI provides information about errors happening in tasks and fine-grained stats on the throughput and latency performance of each component of each running topology.</p>
-
-<p>You can also look at the worker logs on the cluster machines.</p>
-
-
-
- </div>
- </div>
- </div>
-<footer>
- <div class="container-fluid">
- <div class="row">
- <div class="col-md-3">
- <div class="footer-widget">
- <h5>Meetups</h5>
- <ul class="latest-news">
-
- <li><a href="http://www.meetup.com/Apache-Storm-Apache-Kafka/">Apache Storm & Apache Kafka</a> <span class="small">(Sunnyvale, CA)</span></li>
-
- <li><a href="http://www.meetup.com/Apache-Storm-Kafka-Users/">Apache Storm & Kafka Users</a> <span class="small">(Seattle, WA)</span></li>
-
- <li><a href="http://www.meetup.com/New-York-City-Storm-User-Group/">NYC Storm User Group</a> <span class="small">(New York, NY)</span></li>
-
- <li><a href="http://www.meetup.com/Bay-Area-Stream-Processing">Bay Area Stream Processing</a> <span class="small">(Emeryville, CA)</span></li>
-
- <li><a href="http://www.meetup.com/Boston-Storm-Users/">Boston Realtime Data</a> <span class="small">(Boston, MA)</span></li>
-
- <li><a href="http://www.meetup.com/storm-london">London Storm User Group</a> <span class="small">(London, UK)</span></li>
-
- <!-- <li><a href="http://www.meetup.com/Apache-Storm-Kafka-Users/">Seatle, WA</a> <span class="small">(27 Jun 2015)</span></li> -->
- </ul>
- </div>
- </div>
- <div class="col-md-3">
- <div class="footer-widget">
- <h5>About Storm</h5>
- <p>Storm integrates with any queueing system and any database system. Storm's spout abstraction makes it easy to integrate a new queuing system. Likewise, integrating Storm with database systems is easy.</p>
- </div>
- </div>
- <div class="col-md-3">
- <div class="footer-widget">
- <h5>First Look</h5>
- <ul class="footer-list">
- <li><a href="/releases/current/Rationale.html">Rationale</a></li>
- <li><a href="/releases/current/Tutorial.html">Tutorial</a></li>
- <li><a href="/releases/current/Setting-up-development-environment.html">Setting up development environment</a></li>
- <li><a href="/releases/current/Creating-a-new-Storm-project.html">Creating a new Storm project</a></li>
- </ul>
- </div>
- </div>
- <div class="col-md-3">
- <div class="footer-widget">
- <h5>Documentation</h5>
- <ul class="footer-list">
- <li><a href="/releases/current/index.html">Index</a></li>
- <li><a href="/releases/current/javadocs/index.html">Javadoc</a></li>
- <li><a href="/releases/current/FAQ.html">FAQ</a></li>
- </ul>
- </div>
- </div>
- </div>
- <hr/>
- <div class="row">
- <div class="col-md-12">
- <p align="center">Copyright © 2015 <a href="http://www.apache.org">Apache Software Foundation</a>. All Rights Reserved.
- <br>Apache Storm, Apache, the Apache feather logo, and the Apache Storm project logos are trademarks of The Apache Software Foundation.
- <br>All other marks mentioned may be trademarks or registered trademarks of their respective owners.</p>
- </div>
- </div>
- </div>
-</footer>
-<!--Footer End-->
-<!-- Scroll to top -->
-<span class="totop"><a href="#"><i class="fa fa-angle-up"></i></a></span>
-
-</body>
-
-</html>
-
http://git-wip-us.apache.org/repos/asf/storm-site/blob/3730cb62/content/releases/0.10.0/SECURITY.html
----------------------------------------------------------------------
diff --git a/content/releases/0.10.0/SECURITY.html b/content/releases/0.10.0/SECURITY.html
deleted file mode 100644
index 5be784d..0000000
--- a/content/releases/0.10.0/SECURITY.html
+++ /dev/null
@@ -1,739 +0,0 @@
-<!DOCTYPE html>
-<html>
- <head>
- <meta charset="utf-8">
- <meta http-equiv="X-UA-Compatible" content="IE=edge">
- <meta name="viewport" content="width=device-width, initial-scale=1">
-
- <link rel="shortcut icon" href="/favicon.ico" type="image/x-icon">
- <link rel="icon" href="/favicon.ico" type="image/x-icon">
-
- <title>Running Apache Storm Securely</title>
-
- <!-- Bootstrap core CSS -->
- <link href="/assets/css/bootstrap.min.css" rel="stylesheet">
- <!-- Bootstrap theme -->
- <link href="/assets/css/bootstrap-theme.min.css" rel="stylesheet">
-
- <!-- Custom styles for this template -->
- <link rel="stylesheet" href="http://fortawesome.github.io/Font-Awesome/assets/font-awesome/css/font-awesome.css">
- <link href="/css/style.css" rel="stylesheet">
- <link href="/assets/css/owl.theme.css" rel="stylesheet">
- <link href="/assets/css/owl.carousel.css" rel="stylesheet">
- <script type="text/javascript" src="/assets/js/jquery.min.js"></script>
- <script type="text/javascript" src="/assets/js/bootstrap.min.js"></script>
- <script type="text/javascript" src="/assets/js/owl.carousel.min.js"></script>
- <script type="text/javascript" src="/assets/js/storm.js"></script>
- <!-- Just for debugging purposes. Don't actually copy these 2 lines! -->
- <!--[if lt IE 9]><script src="../../assets/js/ie8-responsive-file-warning.js"></script><![endif]-->
-
- <!-- HTML5 shim and Respond.js for IE8 support of HTML5 elements and media queries -->
- <!--[if lt IE 9]>
- <script src="https://oss.maxcdn.com/html5shiv/3.7.2/html5shiv.min.js"></script>
- <script src="https://oss.maxcdn.com/respond/1.4.2/respond.min.js"></script>
- <![endif]-->
- </head>
-
-
- <body>
- <header>
- <div class="container-fluid">
- <div class="row">
- <div class="col-md-5">
- <a href="/index.html"><img src="/images/logo.png" class="logo" /></a>
- </div>
- <div class="col-md-5">
-
- <h1>Version: 0.10.0</h1>
-
- </div>
- <div class="col-md-2">
- <a href="/downloads.html" class="btn-std btn-block btn-download">Download</a>
- </div>
- </div>
- </div>
-</header>
-<!--Header End-->
-<!--Navigation Begin-->
-<div class="navbar" role="banner">
- <div class="container-fluid">
- <div class="navbar-header">
- <button class="navbar-toggle" type="button" data-toggle="collapse" data-target=".bs-navbar-collapse">
- <span class="icon-bar"></span>
- <span class="icon-bar"></span>
- <span class="icon-bar"></span>
- </button>
- </div>
- <nav class="collapse navbar-collapse bs-navbar-collapse" role="navigation">
- <ul class="nav navbar-nav">
- <li><a href="/index.html" id="home">Home</a></li>
- <li><a href="/getting-help.html" id="getting-help">Getting Help</a></li>
- <li><a href="/about/integrates.html" id="project-info">Project Information</a></li>
- <li class="dropdown">
- <a href="#" class="dropdown-toggle" data-toggle="dropdown" id="documentation">Documentation <b class="caret"></b></a>
- <ul class="dropdown-menu">
-
-
- <li><a href="/releases/2.0.0-SNAPSHOT/index.html">2.0.0-SNAPSHOT</a></li>
-
-
-
- <li><a href="/releases/1.1.0/index.html">1.1.0</a></li>
-
-
-
-
-
- <li><a href="/releases/1.0.4/index.html">1.0.4</a></li>
-
-
-
-
-
- <li><a href="/releases/1.0.3/index.html">1.0.3</a></li>
-
-
-
- <li><a href="/releases/1.0.2/index.html">1.0.2</a></li>
-
-
-
- <li><a href="/releases/1.0.1/index.html">1.0.1</a></li>
-
-
-
- <li><a href="/releases/1.0.0/index.html">1.0.0</a></li>
-
-
-
- <li><a href="/releases/0.10.2/index.html">0.10.2</a></li>
-
-
-
- <li><a href="/releases/0.10.1/index.html">0.10.1</a></li>
-
-
-
-
-
- <li><a href="/releases/0.10.0/index.html">0.10.0</a></li>
-
-
-
- <li><a href="/releases/0.9.7/index.html">0.9.7</a></li>
-
-
-
- <li><a href="/releases/0.9.6/index.html">0.9.6</a></li>
-
-
-
-
-
-
-
-
-
-
-
-
- </ul>
- </li>
- <li><a href="/talksAndVideos.html">Talks and Slideshows</a></li>
- <li class="dropdown">
- <a href="#" class="dropdown-toggle" data-toggle="dropdown" id="contribute">Community <b class="caret"></b></a>
- <ul class="dropdown-menu">
- <li><a href="/contribute/Contributing-to-Storm.html">Contributing</a></li>
- <li><a href="/contribute/People.html">People</a></li>
- <li><a href="/contribute/BYLAWS.html">ByLaws</a></li>
- </ul>
- </li>
- <li><a href="/2017/07/28/storm104-released.html" id="news">News</a></li>
- </ul>
- </nav>
- </div>
-</div>
-
-
-
- <div class="container-fluid">
- <h1 class="page-title">Running Apache Storm Securely</h1>
- <div class="row">
- <div class="col-md-12">
- <!-- Documentation -->
-
-<p class="post-meta"></p>
-
-<h1 id="running-apache-storm-securely">Running Apache Storm Securely</h1>
-
-<p>Apache Storm offers a range of configuration options when trying to secure
-your cluster. By default all authentication and authorization is disabled but
-can be turned on as needed.</p>
-
-<h2 id="firewall-os-level-security">Firewall/OS level Security</h2>
-
-<p>You can still have a secure storm cluster without turning on formal
-Authentication and Authorization. But to do so usually requires
-configuring your Operating System to restrict the operations that can be done.
-This is generally a good idea even if you plan on running your cluster with Auth.</p>
-
-<p>The exact detail of how to setup these precautions varies a lot and is beyond
-the scope of this document.</p>
-
-<p>It is generally a good idea to enable a firewall and restrict incoming network
-connections to only those originating from the cluster itself and from trusted
-hosts and services, a complete list of ports storm uses are below. </p>
-
-<p>If the data your cluster is processing is sensitive it might be best to setup
-IPsec to encrypt all traffic being sent between the hosts in the cluster.</p>
-
-<h3 id="ports">Ports</h3>
-
-<table><thead>
-<tr>
-<th>Default Port</th>
-<th>Storm Config</th>
-<th>Client Hosts/Processes</th>
-<th>Server</th>
-</tr>
-</thead><tbody>
-<tr>
-<td>2181</td>
-<td><code>storm.zookeeper.port</code></td>
-<td>Nimbus, Supervisors, and Worker processes</td>
-<td>Zookeeper</td>
-</tr>
-<tr>
-<td>6627</td>
-<td><code>nimbus.thrift.port</code></td>
-<td>Storm clients, Supervisors, and UI</td>
-<td>Nimbus</td>
-</tr>
-<tr>
-<td>8080</td>
-<td><code>ui.port</code></td>
-<td>Client Web Browsers</td>
-<td>UI</td>
-</tr>
-<tr>
-<td>8000</td>
-<td><code>logviewer.port</code></td>
-<td>Client Web Browsers</td>
-<td>Logviewer</td>
-</tr>
-<tr>
-<td>3772</td>
-<td><code>drpc.port</code></td>
-<td>External DRPC Clients</td>
-<td>DRPC</td>
-</tr>
-<tr>
-<td>3773</td>
-<td><code>drpc.invocations.port</code></td>
-<td>Worker Processes</td>
-<td>DRPC</td>
-</tr>
-<tr>
-<td>3774</td>
-<td><code>drpc.http.port</code></td>
-<td>External HTTP DRPC Clients</td>
-<td>DRPC</td>
-</tr>
-<tr>
-<td>670{0,1,2,3}</td>
-<td><code>supervisor.slots.ports</code></td>
-<td>Worker Processes</td>
-<td>Worker Processes</td>
-</tr>
-</tbody></table>
-
-<h3 id="ui-logviewer">UI/Logviewer</h3>
-
-<p>The UI and logviewer processes provide a way to not only see what a cluster is
-doing, but also manipulate running topologies. In general these processes should
-not be exposed except to users of the cluster.</p>
-
-<p>Some form of Authentication is typically required, with using java servlet filters </p>
-<div class="highlight"><pre><code class="language-yaml" data-lang="yaml"><span class="s">ui.filter</span><span class="pi">:</span> <span class="s2">"</span><span class="s">filter.class"</span>
-<span class="s">ui.filter.params</span><span class="pi">:</span> <span class="s2">"</span><span class="s">param1":"value1"</span>
-</code></pre></div>
-<p>or by restricting the UI/log viewers ports to only accept connections from local
-hosts, and then front them with another web server, like Apache httpd, that can
-authenticate/authorize incoming connections and
-proxy the connection to the storm process. To make this work the ui process must have
-logviewer.port set to the port of the proxy in its storm.yaml, while the logviewers
-must have it set to the actual port that they are going to bind to.</p>
-
-<p>The servlet filters are preferred because it allows individual topologies to
-specificy who is and who is not allowed to access the pages associated with
-them. </p>
-
-<p>Storm UI can be configured to use AuthenticationFilter from hadoop-auth.
-<code>yaml
-ui.filter: "org.apache.hadoop.security.authentication.server.AuthenticationFilter"
-ui.filter.params:
- "type": "kerberos"
- "kerberos.principal": "HTTP/nimbus.witzend.com"
- "kerberos.keytab": "/vagrant/keytabs/http.keytab"
- "kerberos.name.rules": "RULE:[2:$1@$0]([jt]t@.*EXAMPLE.COM)s/.*/$MAPRED_USER/ RULE:[2:$1@$0]([nd]n@.*EXAMPLE.COM)s/.*/$HDFS_USER/DEFAULT"
-</code>
-make sure to create a principal 'HTTP/{hostname}' (here hostname should be the one where UI daemon runs</p>
-
-<p>Once configured users needs to do kinit before accessing UI.
-Ex:
-curl -i --negotiate -u:anyUser -b ~/cookiejar.txt -c ~/cookiejar.txt <a href="http://storm-ui-hostname:8080/api/v1/cluster/summary">http://storm-ui-hostname:8080/api/v1/cluster/summary</a></p>
-
-<ol>
-<li>Firefox: Goto about:config and search for network.negotiate-auth.trusted-uris double-click to add value "<a href="http://storm-ui-hostname:8080">http://storm-ui-hostname:8080</a>"</li>
-<li>Google-chrome: start from command line with: google-chrome --auth-server-whitelist="<em>storm-ui-hostname" --auth-negotiate-delegate-whitelist="</em>storm-ui-hostname"<br></li>
-<li>IE: Configure trusted websites to include "storm-ui-hostname" and allow negotiation for that website </li>
-</ol>
-
-<p><strong>Caution</strong>: In AD MIT Keberos setup the key size is bigger than the default UI jetty server request header size. Make sure you set ui.header.buffer.bytes to 65536 in storm.yaml. More details are on <a href="https://issues.apache.org/jira/browse/STORM-633">STORM-633</a></p>
-
-<h2 id="ui-drpc-ssl">UI / DRPC SSL</h2>
-
-<p>Both UI and DRPC allows users to configure ssl .</p>
-
-<h3 id="ui">UI</h3>
-
-<p>For UI users needs to set following config in storm.yaml. Generating keystores with proper keys and certs should be taken care by the user before this step.</p>
-
-<ol>
-<li>ui.https.port </li>
-<li>ui.https.keystore.type (example "jks")</li>
-<li>ui.https.keystore.path (example "/etc/ssl/storm_keystore.jks")</li>
-<li>ui.https.keystore.password (keystore password)</li>
-<li>ui.https.key.password (private key password)</li>
-</ol>
-
-<p>optional config
-6. ui.https.truststore.path (example "/etc/ssl/storm_truststore.jks")
-7. ui.https.truststore.password (truststore password)
-8. ui.https.truststore.type (example "jks")</p>
-
-<p>If users want to setup 2-way auth
-9. ui.https.want.client.auth (If this set to true server requests for client certifcate authentication, but keeps the connection if no authentication provided)
-10. ui.https.need.client.auth (If this set to true server requires client to provide authentication)</p>
-
-<h3 id="drpc">DRPC</h3>
-
-<p>similarly to UI , users need to configure following for DRPC</p>
-
-<ol>
-<li>drpc.https.port </li>
-<li>drpc.https.keystore.type (example "jks")</li>
-<li>drpc.https.keystore.path (example "/etc/ssl/storm_keystore.jks")</li>
-<li>drpc.https.keystore.password (keystore password)</li>
-<li>drpc.https.key.password (private key password)</li>
-</ol>
-
-<p>optional config
-6. drpc.https.truststore.path (example "/etc/ssl/storm_truststore.jks")
-7. drpc.https.truststore.password (truststore password)
-8. drpc.https.truststore.type (example "jks")</p>
-
-<p>If users want to setup 2-way auth
-9. drpc.https.want.client.auth (If this set to true server requests for client certifcate authentication, but keeps the connection if no authentication provided)
-10. drpc.https.need.client.auth (If this set to true server requires client to provide authentication)</p>
-
-<h2 id="authentication-kerberos">Authentication (Kerberos)</h2>
-
-<p>Storm offers pluggable authentication support through thrift and SASL. This
-example only goes off of Kerberos as it is a common setup for most big data
-projects.</p>
-
-<p>Setting up a KDC and configuring kerberos on each node is beyond the scope of
-this document and it is assumed that you have done that already.</p>
-
-<h3 id="create-headless-principals-and-keytabs">Create Headless Principals and keytabs</h3>
-
-<p>Each Zookeeper Server, Nimbus, and DRPC server will need a service principal, which, by convention, includes the FQDN of the host it will run on. Be aware that the zookeeper user <em>MUST</em> be zookeeper.<br>
-The supervisors and UI also need a principal to run as, but because they are outgoing connections they do not need to be service principals.
-The following is an example of how to setup kerberos principals, but the
-details may vary depending on your KDC and OS.</p>
-<div class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="c"># Zookeeper (Will need one of these for each box in teh Zk ensamble)</span>
-sudo kadmin.local -q <span class="s1">'addprinc zookeeper/zk1.example.com@STORM.EXAMPLE.COM'</span>
-sudo kadmin.local -q <span class="s2">"ktadd -k /tmp/zk.keytab zookeeper/zk1.example.com@STORM.EXAMPLE.COM"</span>
-<span class="c"># Nimbus and DRPC</span>
-sudo kadmin.local -q <span class="s1">'addprinc storm/storm.example.com@STORM.EXAMPLE.COM'</span>
-sudo kadmin.local -q <span class="s2">"ktadd -k /tmp/storm.keytab storm/storm.example.com@STORM.EXAMPLE.COM"</span>
-<span class="c"># All UI logviewer and Supervisors</span>
-sudo kadmin.local -q <span class="s1">'addprinc storm@STORM.EXAMPLE.COM'</span>
-sudo kadmin.local -q <span class="s2">"ktadd -k /tmp/storm.keytab storm@STORM.EXAMPLE.COM"</span>
-</code></pre></div>
-<p>be sure to distribute the keytab(s) to the appropriate boxes and set the FS permissions so that only the headless user running ZK, or storm has access to them.</p>
-
-<h4 id="storm-kerberos-configuration">Storm Kerberos Configuration</h4>
-
-<p>Both storm and Zookeeper use jaas configuration files to log the user in.
-Each jaas file may have multiple sections for different interfaces being used.</p>
-
-<p>To enable Kerberos authentication in storm you need to set the following storm.yaml configs
-<code>yaml
-storm.thrift.transport: "backtype.storm.security.auth.kerberos.KerberosSaslTransportPlugin"
-java.security.auth.login.config: "/path/to/jaas.conf"
-</code></p>
-
-<p>Nimbus and the supervisor processes will also connect to ZooKeeper(ZK) and we want to configure them to use Kerberos for authentication with ZK. To do this append
-<code>
--Djava.security.auth.login.config=/path/to/jaas.conf
-</code></p>
-
-<p>to the childopts of nimbus, ui, and supervisor. Here is an example given the default childopts settings at the time of writing:</p>
-<div class="highlight"><pre><code class="language-yaml" data-lang="yaml"><span class="s">nimbus.childopts</span><span class="pi">:</span> <span class="s2">"</span><span class="s">-Xmx1024m</span><span class="nv"> </span><span class="s">-Djava.security.auth.login.config=/path/to/jaas.conf"</span>
-<span class="s">ui.childopts</span><span class="pi">:</span> <span class="s2">"</span><span class="s">-Xmx768m</span><span class="nv"> </span><span class="s">-Djava.security.auth.login.config=/path/to/jaas.conf"</span>
-<span class="s">supervisor.childopts</span><span class="pi">:</span> <span class="s2">"</span><span class="s">-Xmx256m</span><span class="nv"> </span><span class="s">-Djava.security.auth.login.config=/path/to/jaas.conf"</span>
-</code></pre></div>
-<p>The jaas.conf file should look something like the following for the storm nodes.
-The StormServer section is used by nimbus and the DRPC Nodes. It does not need to be included on supervisor nodes.
-The StormClient section is used by all storm clients that want to talk to nimbus, including the ui, logviewer, and supervisor. We will use this section on the gateways as well but the structure of that will be a bit different.
-The Client section is used by processes wanting to talk to zookeeper and really only needs to be included with nimbus and the supervisors.
-The Server section is used by the zookeeper servers.
-Having unused sections in the jaas is not a problem.</p>
-<div class="highlight"><pre><code class="language-" data-lang="">StormServer {
- com.sun.security.auth.module.Krb5LoginModule required
- useKeyTab=true
- keyTab="$keytab"
- storeKey=true
- useTicketCache=false
- principal="$principal";
-};
-StormClient {
- com.sun.security.auth.module.Krb5LoginModule required
- useKeyTab=true
- keyTab="$keytab"
- storeKey=true
- useTicketCache=false
- serviceName="$nimbus_user"
- principal="$principal";
-};
-Client {
- com.sun.security.auth.module.Krb5LoginModule required
- useKeyTab=true
- keyTab="$keytab"
- storeKey=true
- useTicketCache=false
- serviceName="zookeeper"
- principal="$principal";
-};
-Server {
- com.sun.security.auth.module.Krb5LoginModule required
- useKeyTab=true
- keyTab="$keytab"
- storeKey=true
- useTicketCache=false
- principal="$principal";
-};
-</code></pre></div>
-<p>The following is an example based off of the keytabs generated
-<code>
-StormServer {
- com.sun.security.auth.module.Krb5LoginModule required
- useKeyTab=true
- keyTab="/keytabs/storm.keytab"
- storeKey=true
- useTicketCache=false
- principal="storm/storm.example.com@STORM.EXAMPLE.COM";
-};
-StormClient {
- com.sun.security.auth.module.Krb5LoginModule required
- useKeyTab=true
- keyTab="/keytabs/storm.keytab"
- storeKey=true
- useTicketCache=false
- serviceName="storm"
- principal="storm@STORM.EXAMPLE.COM";
-};
-Client {
- com.sun.security.auth.module.Krb5LoginModule required
- useKeyTab=true
- keyTab="/keytabs/storm.keytab"
- storeKey=true
- useTicketCache=false
- serviceName="zookeeper"
- principal="storm@STORM.EXAMPLE.COM";
-};
-Server {
- com.sun.security.auth.module.Krb5LoginModule required
- useKeyTab=true
- keyTab="/keytabs/zk.keytab"
- storeKey=true
- useTicketCache=false
- serviceName="zookeeper"
- principal="zookeeper/zk1.example.com@STORM.EXAMPLE.COM";
-};
-</code></p>
-
-<p>Nimbus also will translate the principal into a local user name, so that other services can use this name. To configure this for Kerberos authentication set</p>
-<div class="highlight"><pre><code class="language-" data-lang="">storm.principal.tolocal: "backtype.storm.security.auth.KerberosPrincipalToLocal"
-</code></pre></div>
-<p>This only needs to be done on nimbus, but it will not hurt on any node.
-We also need to inform the topology who the supervisor daemon and the nimbus daemon are running as from a ZooKeeper perspective.</p>
-<div class="highlight"><pre><code class="language-" data-lang="">storm.zookeeper.superACL: "sasl:${nimbus-user}"
-</code></pre></div>
-<p>Here <em>nimbus-user</em> is the Kerberos user that nimbus uses to authenticate with ZooKeeper. If ZooKeeeper is stripping host and realm then this needs to have host and realm stripped too.</p>
-
-<h4 id="zookeeper-ensemble">ZooKeeper Ensemble</h4>
-
-<p>Complete details of how to setup a secure ZK are beyond the scope of this document. But in general you want to enable SASL authentication on each server, and optionally strip off host and realm</p>
-<div class="highlight"><pre><code class="language-" data-lang="">authProvider.1 = org.apache.zookeeper.server.auth.SASLAuthenticationProvider
-kerberos.removeHostFromPrincipal = true
-kerberos.removeRealmFromPrincipal = true
-</code></pre></div>
-<p>And you want to include the jaas.conf on the command line when launching the server so it can use it can find the keytab.
-<code>
--Djava.security.auth.login.config=/jaas/zk_jaas.conf
-</code></p>
-
-<h4 id="gateways">Gateways</h4>
-
-<p>Ideally the end user will only need to run kinit before interacting with storm. To make this happen seamlessly we need the default jaas.conf on the gateways to be something like</p>
-<div class="highlight"><pre><code class="language-" data-lang="">StormClient {
- com.sun.security.auth.module.Krb5LoginModule required
- doNotPrompt=false
- useTicketCache=true
- serviceName="$nimbus_user";
-};
-</code></pre></div>
-<p>The end user can override this if they have a headless user that has a keytab.</p>
-
-<h3 id="authorization-setup">Authorization Setup</h3>
-
-<p><em>Authentication</em> does the job of verifying who the user is, but we also need <em>authorization</em> to do the job of enforcing what each user can do.</p>
-
-<p>The preferred authorization plug-in for nimbus is The <em>SimpleACLAuthorizer</em>. To use the <em>SimpleACLAuthorizer</em>, set the following:</p>
-<div class="highlight"><pre><code class="language-yaml" data-lang="yaml"><span class="s">nimbus.authorizer</span><span class="pi">:</span> <span class="s2">"</span><span class="s">backtype.storm.security.auth.authorizer.SimpleACLAuthorizer"</span>
-</code></pre></div>
-<p>DRPC has a separate authorizer configuration for it. Do not use SimpleACLAuthorizer for DRPC.</p>
-
-<p>The <em>SimpleACLAuthorizer</em> plug-in needs to know who the supervisor users are, and it needs to know about all of the administrator users, including the user running the ui daemon. </p>
-
-<p>These are set through <em>nimbus.supervisor.users</em> and <em>nimbus.admins</em> respectively. Each can either be a full Kerberos principal name, or the name of the user with host and realm stripped off.</p>
-
-<p>The Log servers have their own authorization configurations. These are set through <em>logs.users</em> and <em>logs.groups</em>. These should be set to the admin users or groups for all of the nodes in the cluster. </p>
-
-<p>When a topology is submitted, the submitting user can specify users in this list as well. The users and groups specified-in addition to the users in the cluster-wide setting-will be granted access to the submitted topology's worker logs in the logviewers.</p>
-
-<h3 id="supervisors-headless-user-and-group-setup">Supervisors headless User and group Setup</h3>
-
-<p>To ensure isolation of users in multi-tenancy, there is need to run supervisors and headless user and group unique to execution on the supervisor nodes. To enable this follow below steps.
-1. Add headlessuser to all supervisor hosts.
-2. Create unique group and make it the primary group for the headless user on the supervisor nodes.
-3. The set following properties on storm for these supervisor nodes.</p>
-
-<h3 id="multi-tenant-scheduler">Multi-tenant Scheduler</h3>
-
-<p>To support multi-tenancy better we have written a new scheduler. To enable this scheduler set.
-<code>yaml
-storm.scheduler: "backtype.storm.scheduler.multitenant.MultitenantScheduler"
-</code>
-Be aware that many of the features of this scheduler rely on storm authentication. Without them the scheduler will not know what the user is and will not isolate topologies properly.</p>
-
-<p>The goal of the multi-tenant scheduler is to provide a way to isolate topologies from one another, but to also limit the resources that an individual user can have in the cluster.</p>
-
-<p>The scheduler currently has one config that can be set either through =storm.yaml= or through a separate config file called =multitenant-scheduler.yaml= that should be placed in the same directory as =storm.yaml=. It is preferable to use =multitenant-scheduler.yaml= because it can be updated without needing to restart nimbus.</p>
-
-<p>There is currently only one config in =multitenant-scheduler.yaml=, =multitenant.scheduler.user.pools= is a map from the user name, to the maximum number of nodes that user is guaranteed to be able to use for their topologies.</p>
-
-<p>For example:</p>
-<div class="highlight"><pre><code class="language-yaml" data-lang="yaml"><span class="s">multitenant.scheduler.user.pools</span><span class="pi">:</span>
- <span class="s2">"</span><span class="s">evans"</span><span class="pi">:</span> <span class="s">10</span>
- <span class="s2">"</span><span class="s">derek"</span><span class="pi">:</span> <span class="s">10</span>
-</code></pre></div>
-<h3 id="run-worker-processes-as-user-who-submitted-the-topology">Run worker processes as user who submitted the topology</h3>
-
-<p>By default storm runs workers as the user that is running the supervisor. This is not ideal for security. To make storm run the topologies as the user that launched them set.</p>
-<div class="highlight"><pre><code class="language-yaml" data-lang="yaml"><span class="s">supervisor.run.worker.as.user</span><span class="pi">:</span> <span class="s">true</span>
-</code></pre></div>
-<p>There are several files that go along with this that are needed to be configured properly to make storm secure.</p>
-
-<p>The worker-launcher executable is a special program that allows the supervisor to launch workers as different users. For this to work it needs to be owned by root, but with the group set to be a group that only teh supervisor headless user is a part of.
-It also needs to have 6550 permissions.
-There is also a worker-launcher.cfg file, usually located under /etc/ that should look something like the following</p>
-<div class="highlight"><pre><code class="language-" data-lang="">storm.worker-launcher.group=$(worker_launcher_group)
-min.user.id=$(min_user_id)
-</code></pre></div>
-<p>where worker_launcher_group is the same group the supervisor is a part of, and min.user.id is set to the first real user id on the system.
-This config file also needs to be owned by root and not have world or group write permissions.</p>
-
-<h3 id="impersonating-a-user">Impersonating a user</h3>
-
-<p>A storm client may submit requests on behalf of another user. For example, if a <code>userX</code> submits an oozie workflow and as part of workflow execution if user <code>oozie</code> wants to submit a topology on behalf of <code>userX</code>
-it can do so by leveraging the impersonation feature.In order to submit topology as some other user , you can use <code>StormSubmitter.submitTopologyAs</code> API. Alternatively you can use <code>NimbusClient.getConfiguredClientAs</code>
-to get a nimbus client as some other user and perform any nimbus action(i.e. kill/rebalance/activate/deactivate) using this client. </p>
-
-<p>To ensure only authorized users can perform impersonation you should start nimbus with <code>nimbus.impersonation.authorizer</code> set to <code>backtype.storm.security.auth.authorizer.ImpersonationAuthorizer</code>.
-The <code>ImpersonationAuthorizer</code> uses <code>nimbus.impersonation.acl</code> as the acl to authorize users. Following is a sample nimbus config for supporting impersonation:</p>
-<div class="highlight"><pre><code class="language-yaml" data-lang="yaml"><span class="s">nimbus.impersonation.authorizer</span><span class="pi">:</span> <span class="s">backtype.storm.security.auth.authorizer.ImpersonationAuthorizer</span>
-<span class="s">nimbus.impersonation.acl</span><span class="pi">:</span>
- <span class="s">impersonating_user1</span><span class="pi">:</span>
- <span class="s">hosts</span><span class="pi">:</span>
- <span class="pi">[</span><span class="nv">comma separated list of hosts from which impersonating_user1 is allowed to impersonate other users</span><span class="pi">]</span>
- <span class="s">groups</span><span class="pi">:</span>
- <span class="pi">[</span><span class="nv">comma separated list of groups whose users impersonating_user1 is allowed to impersonate</span><span class="pi">]</span>
- <span class="s">impersonating_user2</span><span class="pi">:</span>
- <span class="s">hosts</span><span class="pi">:</span>
- <span class="pi">[</span><span class="nv">comma separated list of hosts from which impersonating_user2 is allowed to impersonate other users</span><span class="pi">]</span>
- <span class="s">groups</span><span class="pi">:</span>
- <span class="pi">[</span><span class="nv">comma separated list of groups whose users impersonating_user2 is allowed to impersonate</span><span class="pi">]</span>
-</code></pre></div>
-<p>To support the oozie use case following config can be supplied:
-<code>yaml
-nimbus.impersonation.acl:
- oozie:
- hosts:
- [oozie-host1, oozie-host2, 127.0.0.1]
- groups:
- [some-group-that-userX-is-part-of]
-</code></p>
-
-<h3 id="automatic-credentials-push-and-renewal">Automatic Credentials Push and Renewal</h3>
-
-<p>Individual topologies have the ability to push credentials (tickets and tokens) to workers so that they can access secure services. Exposing this to all of the users can be a pain for them.
-To hide this from them in the common case plugins can be used to populate the credentials, unpack them on the other side into a java Subject, and also allow Nimbus to renew the credentials if needed.
-These are controlled by the following configs. topology.auto-credentials is a list of java plugins, all of which must implement IAutoCredentials interface, that populate the credentials on gateway
-and unpack them on the worker side. On a kerberos secure cluster they should be set by default to point to backtype.storm.security.auth.kerberos.AutoTGT.<br>
-nimbus.credential.renewers.classes should also be set to this value so that nimbus can periodically renew the TGT on behalf of the user.</p>
-
-<p>nimbus.credential.renewers.freq.secs controls how often the renewer will poll to see if anything needs to be renewed, but the default should be fine.</p>
-
-<p>In addition Nimbus itself can be used to get credentials on behalf of the user submitting topologies. This can be configures using nimbus.autocredential.plugins.classes which is a list
-of fully qualified class names ,all of which must implement INimbusCredentialPlugin. Nimbus will invoke the populateCredentials method of all the configured implementation as part of topology
-submission. You should use this config with topology.auto-credentials and nimbus.credential.renewers.classes so the credentials can be populated on worker side and nimbus can automatically renew
-them. Currently there are 2 examples of using this config, AutoHDFS and AutoHBase which auto populates hdfs and hbase delegation tokens for topology submitter so they don't have to distribute keytabs
-on all possible worker hosts.</p>
-
-<h3 id="limits">Limits</h3>
-
-<p>By default storm allows any sized topology to be submitted. But ZK and others have limitations on how big a topology can actually be. The following configs allow you to limit the maximum size a topology can be.</p>
-
-<table><thead>
-<tr>
-<th>YAML Setting</th>
-<th>Description</th>
-</tr>
-</thead><tbody>
-<tr>
-<td>nimbus.slots.perTopology</td>
-<td>The maximum number of slots/workers a topology can use.</td>
-</tr>
-<tr>
-<td>nimbus.executors.perTopology</td>
-<td>The maximum number of executors/threads a topology can use.</td>
-</tr>
-</tbody></table>
-
-<h3 id="log-cleanup">Log Cleanup</h3>
-
-<p>The Logviewer daemon now is also responsible for cleaning up old log files for dead topologies.</p>
-
-<table><thead>
-<tr>
-<th>YAML Setting</th>
-<th>Description</th>
-</tr>
-</thead><tbody>
-<tr>
-<td>logviewer.cleanup.age.mins</td>
-<td>How old (by last modification time) must a worker's log be before that log is considered for clean-up. (Living workers' logs are never cleaned up by the logviewer: Their logs are rolled via logback.)</td>
-</tr>
-<tr>
-<td>logviewer.cleanup.interval.secs</td>
-<td>Interval of time in seconds that the logviewer cleans up worker logs.</td>
-</tr>
-</tbody></table>
-
-<h3 id="allowing-specific-users-or-groups-to-access-storm">Allowing specific users or groups to access storm</h3>
-
-<p>With SimpleACLAuthorizer any user with valid kerberos ticket can deploy a topology or do further operations such as activate, deactivate , access cluster information.
- One can restrict this access by specifying nimbus.users or nimbus.groups. If nimbus.users configured only the users in the list can deploy a topology or access cluster.
- Similarly nimbus.groups restrict storm cluster access to users who belong to those groups.</p>
-
-<p>To configure specify the following config in storm.yaml</p>
-<div class="highlight"><pre><code class="language-yaml" data-lang="yaml"><span class="s">nimbus.users</span><span class="pi">:</span>
- <span class="pi">-</span> <span class="s2">"</span><span class="s">testuser"</span>
-</code></pre></div>
-<p>or </p>
-<div class="highlight"><pre><code class="language-yaml" data-lang="yaml"><span class="s">nimbus.groups</span><span class="pi">:</span>
- <span class="pi">-</span> <span class="s2">"</span><span class="s">storm"</span>
-</code></pre></div>
-<h3 id="drpc">DRPC</h3>
-
-<p>Hopefully more on this soon</p>
-
-
-
- </div>
- </div>
- </div>
-<footer>
- <div class="container-fluid">
- <div class="row">
- <div class="col-md-3">
- <div class="footer-widget">
- <h5>Meetups</h5>
- <ul class="latest-news">
-
- <li><a href="http://www.meetup.com/Apache-Storm-Apache-Kafka/">Apache Storm & Apache Kafka</a> <span class="small">(Sunnyvale, CA)</span></li>
-
- <li><a href="http://www.meetup.com/Apache-Storm-Kafka-Users/">Apache Storm & Kafka Users</a> <span class="small">(Seattle, WA)</span></li>
-
- <li><a href="http://www.meetup.com/New-York-City-Storm-User-Group/">NYC Storm User Group</a> <span class="small">(New York, NY)</span></li>
-
- <li><a href="http://www.meetup.com/Bay-Area-Stream-Processing">Bay Area Stream Processing</a> <span class="small">(Emeryville, CA)</span></li>
-
- <li><a href="http://www.meetup.com/Boston-Storm-Users/">Boston Realtime Data</a> <span class="small">(Boston, MA)</span></li>
-
- <li><a href="http://www.meetup.com/storm-london">London Storm User Group</a> <span class="small">(London, UK)</span></li>
-
- <!-- <li><a href="http://www.meetup.com/Apache-Storm-Kafka-Users/">Seatle, WA</a> <span class="small">(27 Jun 2015)</span></li> -->
- </ul>
- </div>
- </div>
- <div class="col-md-3">
- <div class="footer-widget">
- <h5>About Storm</h5>
- <p>Storm integrates with any queueing system and any database system. Storm's spout abstraction makes it easy to integrate a new queuing system. Likewise, integrating Storm with database systems is easy.</p>
- </div>
- </div>
- <div class="col-md-3">
- <div class="footer-widget">
- <h5>First Look</h5>
- <ul class="footer-list">
- <li><a href="/releases/current/Rationale.html">Rationale</a></li>
- <li><a href="/releases/current/Tutorial.html">Tutorial</a></li>
- <li><a href="/releases/current/Setting-up-development-environment.html">Setting up development environment</a></li>
- <li><a href="/releases/current/Creating-a-new-Storm-project.html">Creating a new Storm project</a></li>
- </ul>
- </div>
- </div>
- <div class="col-md-3">
- <div class="footer-widget">
- <h5>Documentation</h5>
- <ul class="footer-list">
- <li><a href="/releases/current/index.html">Index</a></li>
- <li><a href="/releases/current/javadocs/index.html">Javadoc</a></li>
- <li><a href="/releases/current/FAQ.html">FAQ</a></li>
- </ul>
- </div>
- </div>
- </div>
- <hr/>
- <div class="row">
- <div class="col-md-12">
- <p align="center">Copyright © 2015 <a href="http://www.apache.org">Apache Software Foundation</a>. All Rights Reserved.
- <br>Apache Storm, Apache, the Apache feather logo, and the Apache Storm project logos are trademarks of The Apache Software Foundation.
- <br>All other marks mentioned may be trademarks or registered trademarks of their respective owners.</p>
- </div>
- </div>
- </div>
-</footer>
-<!--Footer End-->
-<!-- Scroll to top -->
-<span class="totop"><a href="#"><i class="fa fa-angle-up"></i></a></span>
-
-</body>
-
-</html>
-