You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@storm.apache.org by ka...@apache.org on 2018/06/09 13:42:49 UTC
[41/51] [partial] storm-site git commit: Update latest 1.x-branch
version to 1.2.2
http://git-wip-us.apache.org/repos/asf/storm-site/blob/6e122a12/content/releases/1.2.1/Multilang-protocol.html
----------------------------------------------------------------------
diff --git a/content/releases/1.2.1/Multilang-protocol.html b/content/releases/1.2.1/Multilang-protocol.html
deleted file mode 100644
index efc67fc..0000000
--- a/content/releases/1.2.1/Multilang-protocol.html
+++ /dev/null
@@ -1,514 +0,0 @@
-<!DOCTYPE html>
-<html>
- <head>
- <meta charset="utf-8">
- <meta http-equiv="X-UA-Compatible" content="IE=edge">
- <meta name="viewport" content="width=device-width, initial-scale=1">
-
- <link rel="shortcut icon" href="/favicon.ico" type="image/x-icon">
- <link rel="icon" href="/favicon.ico" type="image/x-icon">
-
- <title>Multi-Lang Protocol</title>
-
- <!-- Bootstrap core CSS -->
- <link href="/assets/css/bootstrap.min.css" rel="stylesheet">
- <!-- Bootstrap theme -->
- <link href="/assets/css/bootstrap-theme.min.css" rel="stylesheet">
-
- <!-- Custom styles for this template -->
- <link rel="stylesheet" href="http://fortawesome.github.io/Font-Awesome/assets/font-awesome/css/font-awesome.css">
- <link href="/css/style.css" rel="stylesheet">
- <link href="/assets/css/owl.theme.css" rel="stylesheet">
- <link href="/assets/css/owl.carousel.css" rel="stylesheet">
- <script type="text/javascript" src="/assets/js/jquery.min.js"></script>
- <script type="text/javascript" src="/assets/js/bootstrap.min.js"></script>
- <script type="text/javascript" src="/assets/js/owl.carousel.min.js"></script>
- <script type="text/javascript" src="/assets/js/storm.js"></script>
- <!-- Just for debugging purposes. Don't actually copy these 2 lines! -->
- <!--[if lt IE 9]><script src="../../assets/js/ie8-responsive-file-warning.js"></script><![endif]-->
-
- <!-- HTML5 shim and Respond.js for IE8 support of HTML5 elements and media queries -->
- <!--[if lt IE 9]>
- <script src="https://oss.maxcdn.com/html5shiv/3.7.2/html5shiv.min.js"></script>
- <script src="https://oss.maxcdn.com/respond/1.4.2/respond.min.js"></script>
- <![endif]-->
- </head>
-
-
- <body>
- <header>
- <div class="container-fluid">
- <div class="row">
- <div class="col-md-5">
- <a href="/index.html"><img src="/images/logo.png" class="logo" /></a>
- </div>
- <div class="col-md-5">
-
- <h1>Version: 1.2.1</h1>
-
- </div>
- <div class="col-md-2">
- <a href="/downloads.html" class="btn-std btn-block btn-download">Download</a>
- </div>
- </div>
- </div>
-</header>
-<!--Header End-->
-<!--Navigation Begin-->
-<div class="navbar" role="banner">
- <div class="container-fluid">
- <div class="navbar-header">
- <button class="navbar-toggle" type="button" data-toggle="collapse" data-target=".bs-navbar-collapse">
- <span class="icon-bar"></span>
- <span class="icon-bar"></span>
- <span class="icon-bar"></span>
- </button>
- </div>
- <nav class="collapse navbar-collapse bs-navbar-collapse" role="navigation">
- <ul class="nav navbar-nav">
- <li><a href="/index.html" id="home">Home</a></li>
- <li><a href="/getting-help.html" id="getting-help">Getting Help</a></li>
- <li><a href="/about/integrates.html" id="project-info">Project Information</a></li>
- <li class="dropdown">
- <a href="#" class="dropdown-toggle" data-toggle="dropdown" id="documentation">Documentation <b class="caret"></b></a>
- <ul class="dropdown-menu">
-
-
- <li><a href="/releases/2.0.0-SNAPSHOT/index.html">2.0.0-SNAPSHOT</a></li>
-
-
-
- <li><a href="/releases/1.2.1/index.html">1.2.1</a></li>
-
-
-
- <li><a href="/releases/1.1.2/index.html">1.1.2</a></li>
-
-
-
-
-
- <li><a href="/releases/1.0.6/index.html">1.0.6</a></li>
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
- </ul>
- </li>
- <li><a href="/talksAndVideos.html">Talks and Slideshows</a></li>
- <li class="dropdown">
- <a href="#" class="dropdown-toggle" data-toggle="dropdown" id="contribute">Community <b class="caret"></b></a>
- <ul class="dropdown-menu">
- <li><a href="/contribute/Contributing-to-Storm.html">Contributing</a></li>
- <li><a href="/contribute/People.html">People</a></li>
- <li><a href="/contribute/BYLAWS.html">ByLaws</a></li>
- </ul>
- </li>
- <li><a href="/2018/06/04/storm122-released.html" id="news">News</a></li>
- </ul>
- </nav>
- </div>
-</div>
-
-
-
- <div class="container-fluid">
- <h1 class="page-title">Multi-Lang Protocol</h1>
- <div class="row">
- <div class="col-md-12">
- <!-- Documentation -->
-
-<p class="post-meta"></p>
-
-<div class="documentation-content"><p>This page explains the multilang protocol as of Storm 0.7.1. Versions prior to 0.7.1 used a somewhat different protocol, documented [here](Storm-multi-language-protocol-(versions-0.7.0-and-below).html).</p>
-
-<h1 id="storm-multi-language-protocol">Storm Multi-Language Protocol</h1>
-
-<h2 id="shell-components">Shell Components</h2>
-
-<p>Support for multiple languages is implemented via the ShellBolt,
-ShellSpout, and ShellProcess classes. These classes implement the
-IBolt and ISpout interfaces and the protocol for executing a script or
-program via the shell using Java's ProcessBuilder class.</p>
-
-<h3 id="packaging-of-shell-scripts">Packaging of shell scripts</h3>
-
-<p>By default the ShellProcess assumes that your code is packaged inside of your topology jar under the resources subdirectory of your jar and by default will change the current working directory of
-the executable process to be that resources directory extracted from the jar.
-A jar file does not store permissions of the files in it. This includes the execute bit that would allow a shell script to be laoded and run by the operating systme.
-As such in most examples the scripts are of the form <code>python mybolt.py</code> because the python executable is already on the supervisor and mybolt is packaged in the resources directory of the jar.</p>
-
-<p>If you want to package something more complicated, like a new version of python itself, you need to instead use the blob store for this and a <code>.tgz</code> archive that does support permissions.</p>
-
-<p>See the docs on the <a href="distcache-blobstore.html">Blob Store</a> for more details on how to ship a jar.</p>
-
-<p>To make a ShellBolt/ShellSpout work with executables + scripts shipped in the blob store dist cache add</p>
-<div class="highlight"><pre><code class="language-" data-lang="">changeChildCWD(false);
-</code></pre></div>
-<p>in the constructor of your ShellBolt/ShellSpout. The shell command will then be relative to the cwd of the worker. Where the sym-links to the resources are.</p>
-
-<p>So if I shipped python with a symlink named <code>newPython</code> and a python ShellSpout I shipped into <code>shell_spout.py</code> I would have a something like</p>
-<div class="highlight"><pre><code class="language-" data-lang="">public MyShellSpout() {
- super("./newPython/bin/python", "./shell_spout.py");
- changeChildCWD(false);
-}
-</code></pre></div>
-<h2 id="output-fields">Output fields</h2>
-
-<p>Output fields are part of the Thrift definition of the topology. This means that when you multilang in Java, you need to create a bolt that extends ShellBolt, implements IRichBolt, and declare the fields in <code>declareOutputFields</code> (similarly for ShellSpout).</p>
-
-<p>You can learn more about this on <a href="Concepts.html">Concepts</a></p>
-
-<h2 id="protocol-preamble">Protocol Preamble</h2>
-
-<p>A simple protocol is implemented via the STDIN and STDOUT of the
-executed script or program. All data exchanged with the process is
-encoded in JSON, making support possible for pretty much any language.</p>
-
-<h1 id="packaging-your-stuff">Packaging Your Stuff</h1>
-
-<p>To run a shell component on a cluster, the scripts that are shelled
-out to must be in the <code>resources/</code> directory within the jar submitted
-to the master.</p>
-
-<p>However, during development or testing on a local machine, the resources
-directory just needs to be on the classpath.</p>
-
-<h2 id="the-protocol">The Protocol</h2>
-
-<p>Notes:</p>
-
-<ul>
-<li>Both ends of this protocol use a line-reading mechanism, so be sure to
-trim off newlines from the input and to append them to your output.</li>
-<li>All JSON inputs and outputs are terminated by a single line containing "end". Note that this delimiter is not itself JSON encoded.</li>
-<li>The bullet points below are written from the perspective of the script writer's
-STDIN and STDOUT.</li>
-</ul>
-
-<h3 id="initial-handshake">Initial Handshake</h3>
-
-<p>The initial handshake is the same for both types of shell components:</p>
-
-<ul>
-<li>STDIN: Setup info. This is a JSON object with the Storm configuration, a PID directory, and a topology context, like this:</li>
-</ul>
-<div class="highlight"><pre><code class="language-" data-lang="">{
- "conf": {
- "topology.message.timeout.secs": 3,
- // etc
- },
- "pidDir": "...",
- "context": {
- "task->component": {
- "1": "example-spout",
- "2": "__acker",
- "3": "example-bolt1",
- "4": "example-bolt2"
- },
- "taskid": 3,
- // Everything below this line is only available in Storm 0.10.0+
- "componentid": "example-bolt"
- "stream->target->grouping": {
- "default": {
- "example-bolt2": {
- "type": "SHUFFLE"}}},
- "streams": ["default"],
- "stream->outputfields": {"default": ["word"]},
- "source->stream->grouping": {
- "example-spout": {
- "default": {
- "type": "FIELDS",
- "fields": ["word"]
- }
- }
- }
- "source->stream->fields": {
- "example-spout": {
- "default": ["word"]
- }
- }
- }
-}
-</code></pre></div>
-<p>Your script should create an empty file named with its PID in this directory. e.g.
-the PID is 1234, so an empty file named 1234 is created in the directory. This
-file lets the supervisor know the PID so it can shutdown the process later on.</p>
-
-<p>As of Storm 0.10.0, the context sent by Storm to shell components has been
-enhanced substantially to include all aspects of the topology context available
-to JVM components. One key addition is the ability to determine a shell
-component's source and targets (i.e., inputs and outputs) in the topology via
-the <code>stream->target->grouping</code> and <code>source->stream->grouping</code> dictionaries. At
-the innermost level of these nested dictionaries, groupings are represented as
-a dictionary that minimally has a <code>type</code> key, but can also have a <code>fields</code> key
-to specify which fields are involved in a <code>FIELDS</code> grouping.</p>
-
-<ul>
-<li>STDOUT: Your PID, in a JSON object, like <code>{"pid": 1234}</code>. The shell component will log the PID to its log.</li>
-</ul>
-
-<p>What happens next depends on the type of component:</p>
-
-<h3 id="spouts">Spouts</h3>
-
-<p>Shell spouts are synchronous. The rest happens in a while(true) loop:</p>
-
-<ul>
-<li>STDIN: Either a next, ack, activate, deactivate or fail command.</li>
-</ul>
-
-<p>"next" is the equivalent of ISpout's <code>nextTuple</code>. It looks like:</p>
-<div class="highlight"><pre><code class="language-" data-lang="">{"command": "next"}
-</code></pre></div>
-<p>"ack" looks like:</p>
-<div class="highlight"><pre><code class="language-" data-lang="">{"command": "ack", "id": "1231231"}
-</code></pre></div>
-<p>"activate" is the equivalent of ISpout's <code>activate</code>:
-<code>
-{"command": "activate"}
-</code></p>
-
-<p>"deactivate" is the equivalent of ISpout's <code>deactivate</code>:
-<code>
-{"command": "deactivate"}
-</code></p>
-
-<p>"fail" looks like:</p>
-<div class="highlight"><pre><code class="language-" data-lang="">{"command": "fail", "id": "1231231"}
-</code></pre></div>
-<ul>
-<li>STDOUT: The results of your spout for the previous command. This can
-be a sequence of emits and logs.</li>
-</ul>
-
-<p>An emit looks like:</p>
-<div class="highlight"><pre><code class="language-" data-lang="">{
- "command": "emit",
- // The id for the tuple. Leave this out for an unreliable emit. The id can
- // be a string or a number.
- "id": "1231231",
- // The id of the stream this tuple was emitted to. Leave this empty to emit to default stream.
- "stream": "1",
- // If doing an emit direct, indicate the task to send the tuple to
- "task": 9,
- // All the values in this tuple
- "tuple": ["field1", 2, 3]
-}
-</code></pre></div>
-<p>If not doing an emit direct, you will immediately receive the task ids to which the tuple was emitted on STDIN as a JSON array.</p>
-
-<p>A "log" will log a message in the worker log. It looks like:</p>
-<div class="highlight"><pre><code class="language-" data-lang="">{
- "command": "log",
- // the message to log
- "msg": "hello world!"
-}
-</code></pre></div>
-<ul>
-<li>STDOUT: a "sync" command ends the sequence of emits and logs. It looks like:</li>
-</ul>
-<div class="highlight"><pre><code class="language-" data-lang="">{"command": "sync"}
-</code></pre></div>
-<p>After you sync, ShellSpout will not read your output until it sends another next, ack, or fail command.</p>
-
-<p>Note that, similarly to ISpout, all of the spouts in the worker will be locked up after a next, ack, or fail, until you sync. Also like ISpout, if you have no tuples to emit for a next, you should sleep for a small amount of time before syncing. ShellSpout will not automatically sleep for you.</p>
-
-<h3 id="bolts">Bolts</h3>
-
-<p>The shell bolt protocol is asynchronous. You will receive tuples on STDIN as soon as they are available, and you may emit, ack, and fail, and log at any time by writing to STDOUT, as follows:</p>
-
-<ul>
-<li>STDIN: A tuple! This is a JSON encoded structure like this:</li>
-</ul>
-<div class="highlight"><pre><code class="language-" data-lang="">{
- // The tuple's id - this is a string to support languages lacking 64-bit precision
- "id": "-6955786537413359385",
- // The id of the component that created this tuple
- "comp": "1",
- // The id of the stream this tuple was emitted to
- "stream": "1",
- // The id of the task that created this tuple
- "task": 9,
- // All the values in this tuple
- "tuple": ["snow white and the seven dwarfs", "field2", 3]
-}
-</code></pre></div>
-<ul>
-<li>STDOUT: An ack, fail, emit, or log. Emits look like:</li>
-</ul>
-<div class="highlight"><pre><code class="language-" data-lang="">{
- "command": "emit",
- // The ids of the tuples this output tuples should be anchored to
- "anchors": ["1231231", "-234234234"],
- // The id of the stream this tuple was emitted to. Leave this empty to emit to default stream.
- "stream": "1",
- // If doing an emit direct, indicate the task to send the tuple to
- "task": 9,
- // All the values in this tuple
- "tuple": ["field1", 2, 3]
-}
-</code></pre></div>
-<p>If not doing an emit direct, you will receive the task ids to which
-the tuple was emitted on STDIN as a JSON array. Note that, due to the
-asynchronous nature of the shell bolt protocol, when you read after
-emitting, you may not receive the task ids. You may instead read the
-task ids for a previous emit or a new tuple to process. You will
-receive the task id lists in the same order as their corresponding
-emits, however.</p>
-
-<p>An ack looks like:</p>
-<div class="highlight"><pre><code class="language-" data-lang="">{
- "command": "ack",
- // the id of the tuple to ack
- "id": "123123"
-}
-</code></pre></div>
-<p>A fail looks like:</p>
-<div class="highlight"><pre><code class="language-" data-lang="">{
- "command": "fail",
- // the id of the tuple to fail
- "id": "123123"
-}
-</code></pre></div>
-<p>A "log" will log a message in the worker log. It looks like:</p>
-<div class="highlight"><pre><code class="language-" data-lang="">{
- "command": "log",
- // the message to log
- "msg": "hello world!"
-}
-</code></pre></div>
-<ul>
-<li>Note that, as of version 0.7.1, there is no longer any need for a
-shell bolt to 'sync'.</li>
-</ul>
-
-<h3 id="handling-heartbeats-0-9-3-and-later">Handling Heartbeats (0.9.3 and later)</h3>
-
-<p>As of Storm 0.9.3, heartbeats have been between ShellSpout/ShellBolt and their
-multi-lang subprocesses to detect hanging/zombie subprocesses. Any libraries
-for interfacing with Storm via multi-lang must take the following actions
-regarding hearbeats:</p>
-
-<h4 id="spout">Spout</h4>
-
-<p>Shell spouts are synchronous, so subprocesses always send <code>sync</code> commands at the
-end of <code>next()</code>, so you should not have to do much to support heartbeats for
-spouts. That said, you must not let subprocesses sleep more than the worker
-timeout during <code>next()</code>.</p>
-
-<h4 id="bolt">Bolt</h4>
-
-<p>Shell bolts are asynchronous, so a ShellBolt will send heartbeat tuples to its
-subprocess periodically. Heartbeat tuple looks like:</p>
-<div class="highlight"><pre><code class="language-" data-lang="">{
- "id": "-6955786537413359385",
- "comp": "1",
- "stream": "__heartbeat",
- // this shell bolt's system task id
- "task": -1,
- "tuple": []
-}
-</code></pre></div>
-<p>When subprocess receives heartbeat tuple, it must send a <code>sync</code> command back to
-ShellBolt.</p>
-</div>
-
-
- </div>
- </div>
- </div>
-<footer>
- <div class="container-fluid">
- <div class="row">
- <div class="col-md-3">
- <div class="footer-widget">
- <h5>Meetups</h5>
- <ul class="latest-news">
-
- <li><a href="http://www.meetup.com/Apache-Storm-Apache-Kafka/">Apache Storm & Apache Kafka</a> <span class="small">(Sunnyvale, CA)</span></li>
-
- <li><a href="http://www.meetup.com/Apache-Storm-Kafka-Users/">Apache Storm & Kafka Users</a> <span class="small">(Seattle, WA)</span></li>
-
- <li><a href="http://www.meetup.com/New-York-City-Storm-User-Group/">NYC Storm User Group</a> <span class="small">(New York, NY)</span></li>
-
- <li><a href="http://www.meetup.com/Bay-Area-Stream-Processing">Bay Area Stream Processing</a> <span class="small">(Emeryville, CA)</span></li>
-
- <li><a href="http://www.meetup.com/Boston-Storm-Users/">Boston Realtime Data</a> <span class="small">(Boston, MA)</span></li>
-
- <li><a href="http://www.meetup.com/storm-london">London Storm User Group</a> <span class="small">(London, UK)</span></li>
-
- <!-- <li><a href="http://www.meetup.com/Apache-Storm-Kafka-Users/">Seatle, WA</a> <span class="small">(27 Jun 2015)</span></li> -->
- </ul>
- </div>
- </div>
- <div class="col-md-3">
- <div class="footer-widget">
- <h5>About Storm</h5>
- <p>Storm integrates with any queueing system and any database system. Storm's spout abstraction makes it easy to integrate a new queuing system. Likewise, integrating Storm with database systems is easy.</p>
- </div>
- </div>
- <div class="col-md-3">
- <div class="footer-widget">
- <h5>First Look</h5>
- <ul class="footer-list">
- <li><a href="/releases/current/Rationale.html">Rationale</a></li>
- <li><a href="/releases/current/Tutorial.html">Tutorial</a></li>
- <li><a href="/releases/current/Setting-up-development-environment.html">Setting up development environment</a></li>
- <li><a href="/releases/current/Creating-a-new-Storm-project.html">Creating a new Storm project</a></li>
- </ul>
- </div>
- </div>
- <div class="col-md-3">
- <div class="footer-widget">
- <h5>Documentation</h5>
- <ul class="footer-list">
- <li><a href="/releases/current/index.html">Index</a></li>
- <li><a href="/releases/current/javadocs/index.html">Javadoc</a></li>
- <li><a href="/releases/current/FAQ.html">FAQ</a></li>
- </ul>
- </div>
- </div>
- </div>
- <hr/>
- <div class="row">
- <div class="col-md-12">
- <p align="center">Copyright © 2015 <a href="http://www.apache.org">Apache Software Foundation</a>. All Rights Reserved.
- <br>Apache Storm, Apache, the Apache feather logo, and the Apache Storm project logos are trademarks of The Apache Software Foundation.
- <br>All other marks mentioned may be trademarks or registered trademarks of their respective owners.</p>
- </div>
- </div>
- </div>
-</footer>
-<!--Footer End-->
-<!-- Scroll to top -->
-<span class="totop"><a href="#"><i class="fa fa-angle-up"></i></a></span>
-
-</body>
-
-</html>
-
http://git-wip-us.apache.org/repos/asf/storm-site/blob/6e122a12/content/releases/1.2.1/Pacemaker.html
----------------------------------------------------------------------
diff --git a/content/releases/1.2.1/Pacemaker.html b/content/releases/1.2.1/Pacemaker.html
deleted file mode 100644
index 246c244..0000000
--- a/content/releases/1.2.1/Pacemaker.html
+++ /dev/null
@@ -1,336 +0,0 @@
-<!DOCTYPE html>
-<html>
- <head>
- <meta charset="utf-8">
- <meta http-equiv="X-UA-Compatible" content="IE=edge">
- <meta name="viewport" content="width=device-width, initial-scale=1">
-
- <link rel="shortcut icon" href="/favicon.ico" type="image/x-icon">
- <link rel="icon" href="/favicon.ico" type="image/x-icon">
-
- <title>Pacemaker</title>
-
- <!-- Bootstrap core CSS -->
- <link href="/assets/css/bootstrap.min.css" rel="stylesheet">
- <!-- Bootstrap theme -->
- <link href="/assets/css/bootstrap-theme.min.css" rel="stylesheet">
-
- <!-- Custom styles for this template -->
- <link rel="stylesheet" href="http://fortawesome.github.io/Font-Awesome/assets/font-awesome/css/font-awesome.css">
- <link href="/css/style.css" rel="stylesheet">
- <link href="/assets/css/owl.theme.css" rel="stylesheet">
- <link href="/assets/css/owl.carousel.css" rel="stylesheet">
- <script type="text/javascript" src="/assets/js/jquery.min.js"></script>
- <script type="text/javascript" src="/assets/js/bootstrap.min.js"></script>
- <script type="text/javascript" src="/assets/js/owl.carousel.min.js"></script>
- <script type="text/javascript" src="/assets/js/storm.js"></script>
- <!-- Just for debugging purposes. Don't actually copy these 2 lines! -->
- <!--[if lt IE 9]><script src="../../assets/js/ie8-responsive-file-warning.js"></script><![endif]-->
-
- <!-- HTML5 shim and Respond.js for IE8 support of HTML5 elements and media queries -->
- <!--[if lt IE 9]>
- <script src="https://oss.maxcdn.com/html5shiv/3.7.2/html5shiv.min.js"></script>
- <script src="https://oss.maxcdn.com/respond/1.4.2/respond.min.js"></script>
- <![endif]-->
- </head>
-
-
- <body>
- <header>
- <div class="container-fluid">
- <div class="row">
- <div class="col-md-5">
- <a href="/index.html"><img src="/images/logo.png" class="logo" /></a>
- </div>
- <div class="col-md-5">
-
- <h1>Version: 1.2.1</h1>
-
- </div>
- <div class="col-md-2">
- <a href="/downloads.html" class="btn-std btn-block btn-download">Download</a>
- </div>
- </div>
- </div>
-</header>
-<!--Header End-->
-<!--Navigation Begin-->
-<div class="navbar" role="banner">
- <div class="container-fluid">
- <div class="navbar-header">
- <button class="navbar-toggle" type="button" data-toggle="collapse" data-target=".bs-navbar-collapse">
- <span class="icon-bar"></span>
- <span class="icon-bar"></span>
- <span class="icon-bar"></span>
- </button>
- </div>
- <nav class="collapse navbar-collapse bs-navbar-collapse" role="navigation">
- <ul class="nav navbar-nav">
- <li><a href="/index.html" id="home">Home</a></li>
- <li><a href="/getting-help.html" id="getting-help">Getting Help</a></li>
- <li><a href="/about/integrates.html" id="project-info">Project Information</a></li>
- <li class="dropdown">
- <a href="#" class="dropdown-toggle" data-toggle="dropdown" id="documentation">Documentation <b class="caret"></b></a>
- <ul class="dropdown-menu">
-
-
- <li><a href="/releases/2.0.0-SNAPSHOT/index.html">2.0.0-SNAPSHOT</a></li>
-
-
-
- <li><a href="/releases/1.2.1/index.html">1.2.1</a></li>
-
-
-
- <li><a href="/releases/1.1.2/index.html">1.1.2</a></li>
-
-
-
-
-
- <li><a href="/releases/1.0.6/index.html">1.0.6</a></li>
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
- </ul>
- </li>
- <li><a href="/talksAndVideos.html">Talks and Slideshows</a></li>
- <li class="dropdown">
- <a href="#" class="dropdown-toggle" data-toggle="dropdown" id="contribute">Community <b class="caret"></b></a>
- <ul class="dropdown-menu">
- <li><a href="/contribute/Contributing-to-Storm.html">Contributing</a></li>
- <li><a href="/contribute/People.html">People</a></li>
- <li><a href="/contribute/BYLAWS.html">ByLaws</a></li>
- </ul>
- </li>
- <li><a href="/2018/06/04/storm122-released.html" id="news">News</a></li>
- </ul>
- </nav>
- </div>
-</div>
-
-
-
- <div class="container-fluid">
- <h1 class="page-title">Pacemaker</h1>
- <div class="row">
- <div class="col-md-12">
- <!-- Documentation -->
-
-<p class="post-meta"></p>
-
-<div class="documentation-content"><h3 id="introduction">Introduction</h3>
-
-<p>Pacemaker is a storm daemon designed to process heartbeats from workers. As Storm is scaled up, ZooKeeper begins to become a bottleneck due to high volumes of writes from workers doing heartbeats. Lots of writes to disk and too much traffic across the network is generated as ZooKeeper tries to maintain consistency.</p>
-
-<p>Because heartbeats are of an ephemeral nature, they do not need to be persisted to disk or synced across nodes; an in-memory store will do. This is the role of Pacemaker. Pacemaker functions as a simple in-memory key/value store with ZooKeeper-like, directory-style keys and byte array values.</p>
-
-<p>The corresponding Pacemaker client is a plugin for the <code>ClusterState</code> interface, <code>org.apache.storm.pacemaker.pacemaker_state_factory</code>. Heartbeat calls are funneled by the <code>ClusterState</code> produced by <code>pacemaker_state_factory</code> into the Pacemaker daemon, while other set/get operations are forwarded to ZooKeeper.</p>
-
-<hr>
-
-<h3 id="configuration">Configuration</h3>
-
-<ul>
-<li><code>pacemaker.host</code> : (deprecated) The host that the Pacemaker daemon is running on</li>
-<li><code>pacemaker.servers</code> : The hosts that the Pacemaker daemons are running on - This supercedes <code>pacemaker.host</code></li>
-<li><code>pacemaker.port</code> : The port that Pacemaker will listen on</li>
-<li><code>pacemaker.max.threads</code> : Maximum number of threads Pacemaker daemon will use to handle requests.</li>
-<li><code>pacemaker.childopts</code> : Any JVM parameters that need to go to the Pacemaker. (used by storm-deploy project)</li>
-<li><code>pacemaker.auth.method</code> : The authentication method that is used (more info below)</li>
-</ul>
-
-<h4 id="example">Example</h4>
-
-<p>To get Pacemaker up and running, set the following option in the cluster config on all nodes:
-<code>
-storm.cluster.state.store: "org.apache.storm.pacemaker.pacemaker_state_factory"
-</code></p>
-
-<p>The Pacemaker servers also need to be set on all nodes:
-<code>
-pacemaker.servers:
- - somehost.mycompany.com
- - someotherhost.mycompany.com
-</code>
-The pacemaker.host config still works for a single pacemaker, although it has been deprecated.
-<code>
-pacemaker.host: single_pacemaker.mycompany.com
-</code></p>
-
-<p>And then start all of your daemons</p>
-
-<p>(including Pacemaker):
-<code>
-$ storm pacemaker
-</code></p>
-
-<p>The Storm cluster should now be pushing all worker heartbeats through Pacemaker.</p>
-
-<h3 id="security">Security</h3>
-
-<p>Currently digest (password-based) and Kerberos security are supported. Security is currently only around reads, not writes. Writes may be performed by anyone, whereas reads may only be performed by authorized and authenticated users. This is an area for future development, as it leaves the cluster open to DoS attacks, but it prevents any sensitive information from reaching unauthorized eyes, which was the main goal.</p>
-
-<h4 id="digest">Digest</h4>
-
-<p>To configure digest authentication, set <code>pacemaker.auth.method: DIGEST</code> in the cluster config on the nodes hosting Nimbus and Pacemaker.
-The nodes must also have <code>java.security.auth.login.config</code> set to point to a JAAS config file containing the following structure:
-<code>
-PacemakerDigest {
- username="some username"
- password="some password";
-};
-</code></p>
-
-<p>Any node with these settings configured will be able to read from Pacemaker.
-Worker nodes need not have these configs set, and may keep <code>pacemaker.auth.method: NONE</code> set, since they do not need to read from the Pacemaker daemon.</p>
-
-<h4 id="kerberos">Kerberos</h4>
-
-<p>To configure Kerberos authentication, set <code>pacemaker.auth.method: KERBEROS</code> in the cluster config on the nodes hosting Nimbus and Pacemaker.
-The nodes must also have <code>java.security.auth.login.config</code> set to point to a JAAS config.</p>
-
-<p>The JAAS config on Nimbus must look something like this:
-```
-PacemakerClient {
- com.sun.security.auth.module.Krb5LoginModule required
- useKeyTab=true
- keyTab="/etc/keytabs/nimbus.keytab"
- storeKey=true
- useTicketCache=false
- serviceName="pacemaker"
- principal="<a href="mailto:nimbus@MY.COMPANY.COM">nimbus@MY.COMPANY.COM</a>";
-};</p>
-<div class="highlight"><pre><code class="language-" data-lang="">
-The JAAS config on Pacemaker must look something like this:
-</code></pre></div>
-<p>PacemakerServer {
- com.sun.security.auth.module.Krb5LoginModule required
- useKeyTab=true
- keyTab="/etc/keytabs/pacemaker.keytab"
- storeKey=true
- useTicketCache=false
- principal="<a href="mailto:pacemaker@MY.COMPANY.COM">pacemaker@MY.COMPANY.COM</a>";
-};
-```</p>
-
-<ul>
-<li>The client's user principal in the <code>PacemakerClient</code> section on the Nimbus host must match the <code>nimbus.daemon.user</code> storm cluster config value.</li>
-<li>The client's <code>serviceName</code> value must match the server's user principal in the <code>PacemakerServer</code> section on the Pacemaker host.</li>
-</ul>
-
-<h3 id="fault-tolerance">Fault Tolerance</h3>
-
-<p>Pacemaker runs as a single daemon instance, making it a potential Single Point of Failure.</p>
-
-<p>If Pacemaker becomes unreachable by Nimbus, through crash or network partition, the workers will continue to run, and Nimbus will repeatedly attempt to reconnect. Nimbus functionality will be disrupted, but the topologies themselves will continue to run.
-In case of partition of the cluster where Nimbus and Pacemaker are on the same side of the partition, the workers that are on the other side of the partition will not be able to heartbeat, and Nimbus will reschedule the tasks elsewhere. This is probably what we want to happen anyway.</p>
-
-<h3 id="zookeeper-comparison">ZooKeeper Comparison</h3>
-
-<p>Compared to ZooKeeper, Pacemaker uses less CPU, less memory, and of course no disk for the same load, thanks to lack of overhead from maintaining consistency between nodes.
-On Gigabit networking, there is a theoretical limit of about 6000 nodes. However, the real limit is likely around 2000-3000 nodes. These limits have not yet been tested.
-On a 270 supervisor cluster, fully scheduled with topologies, Pacemaker resource utilization was 70% of one core and nearly 1GiB of RAM on a machine with 4 <code>Intel(R) Xeon(R) CPU E5530 @ 2.40GHz</code> and 24GiB of RAM.</p>
-
-<p>Pacemaker now supports HA. Multiple Pacemaker instances can be used at once in a storm cluster to allow massive scalability. Just include the names of the Pacemaker hosts in the pacemaker.servers config and workers and Nimbus will start communicating with them. They're fault tolerant as well. The system keeps on working as long as there is at least one pacemaker left running - provided it can handle the load.</p>
-</div>
-
-
- </div>
- </div>
- </div>
-<footer>
- <div class="container-fluid">
- <div class="row">
- <div class="col-md-3">
- <div class="footer-widget">
- <h5>Meetups</h5>
- <ul class="latest-news">
-
- <li><a href="http://www.meetup.com/Apache-Storm-Apache-Kafka/">Apache Storm & Apache Kafka</a> <span class="small">(Sunnyvale, CA)</span></li>
-
- <li><a href="http://www.meetup.com/Apache-Storm-Kafka-Users/">Apache Storm & Kafka Users</a> <span class="small">(Seattle, WA)</span></li>
-
- <li><a href="http://www.meetup.com/New-York-City-Storm-User-Group/">NYC Storm User Group</a> <span class="small">(New York, NY)</span></li>
-
- <li><a href="http://www.meetup.com/Bay-Area-Stream-Processing">Bay Area Stream Processing</a> <span class="small">(Emeryville, CA)</span></li>
-
- <li><a href="http://www.meetup.com/Boston-Storm-Users/">Boston Realtime Data</a> <span class="small">(Boston, MA)</span></li>
-
- <li><a href="http://www.meetup.com/storm-london">London Storm User Group</a> <span class="small">(London, UK)</span></li>
-
- <!-- <li><a href="http://www.meetup.com/Apache-Storm-Kafka-Users/">Seatle, WA</a> <span class="small">(27 Jun 2015)</span></li> -->
- </ul>
- </div>
- </div>
- <div class="col-md-3">
- <div class="footer-widget">
- <h5>About Storm</h5>
- <p>Storm integrates with any queueing system and any database system. Storm's spout abstraction makes it easy to integrate a new queuing system. Likewise, integrating Storm with database systems is easy.</p>
- </div>
- </div>
- <div class="col-md-3">
- <div class="footer-widget">
- <h5>First Look</h5>
- <ul class="footer-list">
- <li><a href="/releases/current/Rationale.html">Rationale</a></li>
- <li><a href="/releases/current/Tutorial.html">Tutorial</a></li>
- <li><a href="/releases/current/Setting-up-development-environment.html">Setting up development environment</a></li>
- <li><a href="/releases/current/Creating-a-new-Storm-project.html">Creating a new Storm project</a></li>
- </ul>
- </div>
- </div>
- <div class="col-md-3">
- <div class="footer-widget">
- <h5>Documentation</h5>
- <ul class="footer-list">
- <li><a href="/releases/current/index.html">Index</a></li>
- <li><a href="/releases/current/javadocs/index.html">Javadoc</a></li>
- <li><a href="/releases/current/FAQ.html">FAQ</a></li>
- </ul>
- </div>
- </div>
- </div>
- <hr/>
- <div class="row">
- <div class="col-md-12">
- <p align="center">Copyright © 2015 <a href="http://www.apache.org">Apache Software Foundation</a>. All Rights Reserved.
- <br>Apache Storm, Apache, the Apache feather logo, and the Apache Storm project logos are trademarks of The Apache Software Foundation.
- <br>All other marks mentioned may be trademarks or registered trademarks of their respective owners.</p>
- </div>
- </div>
- </div>
-</footer>
-<!--Footer End-->
-<!-- Scroll to top -->
-<span class="totop"><a href="#"><i class="fa fa-angle-up"></i></a></span>
-
-</body>
-
-</html>
-