You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@storm.apache.org by bo...@apache.org on 2016/03/22 16:38:46 UTC
[20/51] [partial] storm git commit: STORM-1617: Versioned Docs
http://git-wip-us.apache.org/repos/asf/storm/blob/335bbf94/_site/documentation/getting-help.html
----------------------------------------------------------------------
diff --git a/_site/documentation/getting-help.html b/_site/documentation/getting-help.html
deleted file mode 100644
index 4cc2276..0000000
--- a/_site/documentation/getting-help.html
+++ /dev/null
@@ -1,198 +0,0 @@
-<!DOCTYPE html>
-<html>
- <head>
- <meta charset="utf-8">
- <meta http-equiv="X-UA-Compatible" content="IE=edge">
- <meta name="viewport" content="width=device-width, initial-scale=1">
-
- <link rel="shortcut icon" href="/favicon.ico" type="image/x-icon">
- <link rel="icon" href="/favicon.ico" type="image/x-icon">
-
- <title>Documentation</title>
-
- <!-- Bootstrap core CSS -->
- <link href="/assets/css/bootstrap.min.css" rel="stylesheet">
- <!-- Bootstrap theme -->
- <link href="/assets/css/bootstrap-theme.min.css" rel="stylesheet">
-
- <!-- Custom styles for this template -->
- <link rel="stylesheet" href="http://fortawesome.github.io/Font-Awesome/assets/font-awesome/css/font-awesome.css">
- <link href="/css/style.css" rel="stylesheet">
- <link href="/assets/css/owl.theme.css" rel="stylesheet">
- <link href="/assets/css/owl.carousel.css" rel="stylesheet">
- <script type="text/javascript" src="/assets/js/jquery.min.js"></script>
- <script type="text/javascript" src="/assets/js/bootstrap.min.js"></script>
- <script type="text/javascript" src="/assets/js/owl.carousel.min.js"></script>
- <script type="text/javascript" src="/assets/js/storm.js"></script>
- <!-- Just for debugging purposes. Don't actually copy these 2 lines! -->
- <!--[if lt IE 9]><script src="../../assets/js/ie8-responsive-file-warning.js"></script><![endif]-->
-
- <!-- HTML5 shim and Respond.js for IE8 support of HTML5 elements and media queries -->
- <!--[if lt IE 9]>
- <script src="https://oss.maxcdn.com/html5shiv/3.7.2/html5shiv.min.js"></script>
- <script src="https://oss.maxcdn.com/respond/1.4.2/respond.min.js"></script>
- <![endif]-->
- </head>
-
-
- <body>
- <header>
- <div class="container-fluid">
- <div class="row">
- <div class="col-md-10">
- <a href="/index.html"><img src="/images/logo.png" class="logo" /></a>
- </div>
- <div class="col-md-2">
- <a href="/downloads.html" class="btn-std btn-block btn-download">Download</a>
- </div>
- </div>
- </div>
-</header>
-<!--Header End-->
-<!--Navigation Begin-->
-<div class="navbar" role="banner">
- <div class="container-fluid">
- <div class="navbar-header">
- <button class="navbar-toggle" type="button" data-toggle="collapse" data-target=".bs-navbar-collapse">
- <span class="icon-bar"></span>
- <span class="icon-bar"></span>
- <span class="icon-bar"></span>
- </button>
- </div>
- <nav class="collapse navbar-collapse bs-navbar-collapse" role="navigation">
- <ul class="nav navbar-nav">
- <li><a href="/index.html" id="home">Home</a></li>
- <li><a href="/getting-help.html" id="getting-help">Getting Help</a></li>
- <li><a href="/about/integrates.html" id="project-info">Project Information</a></li>
- <li><a href="/documentation.html" id="documentation">Documentation</a></li>
- <li><a href="/talksAndVideos.html">Talks and Slideshows</a></li>
- <li class="dropdown">
- <a href="#" class="dropdown-toggle" data-toggle="dropdown" id="contribute">Community <b class="caret"></b></a>
- <ul class="dropdown-menu">
- <li><a href="/contribute/Contributing-to-Storm.html">Contributing</a></li>
- <li><a href="/contribute/People.html">People</a></li>
- <li><a href="/contribute/BYLAWS.html">ByLaws</a></li>
- </ul>
- </li>
- <li><a href="/2015/11/05/storm096-released.html" id="news">News</a></li>
- </ul>
- </nav>
- </div>
-</div>
-
-
-
- <div class="container-fluid">
- <h1 class="page-title">Documentation</h1>
- <div class="row">
- <div class="col-md-12">
- <h3 id="getting-help">Getting help</h3>
-
-<p><strong>NOTE:</strong> The google groups account <a href="mailto:storm-user@googlegroups.com">storm-user@googlegroups.com</a> is now officially deprecated in favor of the Apache-hosted user/dev mailing lists.</p>
-
-<h4 id="storm-users">Storm Users</h4>
-
-<p>Storm users should send messages and subscribe to <a href="mailto:user@storm.apache.org">user@storm.apache.org</a>.</p>
-
-<p>You can subscribe to this list by sending an email to <a href="mailto:user-subscribe@storm.apache.org">user-subscribe@storm.apache.org</a>. Likewise, you can cancel a subscription by sending an email to <a href="mailto:user-unsubscribe@storm.apache.org">user-unsubscribe@storm.apache.org</a>.</p>
-
-<p>You can view the archives of the mailing list <a href="http://mail-archives.apache.org/mod_mbox/storm-user/">here</a>.</p>
-
-<h4 id="storm-developers">Storm Developers</h4>
-
-<p>Storm developers should send messages and subscribe to <a href="mailto:dev@storm.apache.org">dev@storm.apache.org</a>.</p>
-
-<p>You can subscribe to this list by sending an email to <a href="mailto:dev-subscribe@storm.apache.org">dev-subscribe@storm.apache.org</a>. Likewise, you can cancel a subscription by sending an email to <a href="mailto:dev-unsubscribe@storm.apache.org">dev-unsubscribe@storm.apache.org</a>.</p>
-
-<p>You can view the archives of the mailing list <a href="http://mail-archives.apache.org/mod_mbox/storm-dev/">here</a>.</p>
-
-<h4 id="which-list-should-i-send-subscribe-to">Which list should I send/subscribe to?</h4>
-
-<p>If you are using a pre-built binary distribution of Storm, then chances are you should send questions, comments, storm-related announcements, etc. to <a href="user@storm.apache.org">user@storm.apache.org</a>. </p>
-
-<p>If you are building storm from source, developing new features, or otherwise hacking storm source code, then <a href="dev@storm.apache.org">dev@storm.apache.org</a> is more appropriate. </p>
-
-<h4 id="what-will-happen-with-storm-user-googlegroups-com">What will happen with <a href="mailto:storm-user@googlegroups.com">storm-user@googlegroups.com</a>?</h4>
-
-<p>All existing messages will remain archived there, and can be accessed/searched <a href="https://groups.google.com/forum/#!forum/storm-user">here</a>.</p>
-
-<p>New messages sent to <a href="mailto:storm-user@googlegroups.com">storm-user@googlegroups.com</a> will either be rejected/bounced or replied to with a message to direct the email to the appropriate Apache-hosted group.</p>
-
-<h4 id="irc">IRC</h4>
-
-<p>You can also come to the #storm-user room on <a href="http://freenode.net/">freenode</a>. You can usually find a Storm developer there to help you out.</p>
-
- </div>
- </div>
- </div>
-<footer>
- <div class="container-fluid">
- <div class="row">
- <div class="col-md-3">
- <div class="footer-widget">
- <h5>Meetups</h5>
- <ul class="latest-news">
-
- <li><a href="http://www.meetup.com/Apache-Storm-Apache-Kafka/">Apache Storm & Apache Kafka</a> <span class="small">(Sunnyvale, CA)</span></li>
-
- <li><a href="http://www.meetup.com/Apache-Storm-Kafka-Users/">Apache Storm & Kafka Users</a> <span class="small">(Seattle, WA)</span></li>
-
- <li><a href="http://www.meetup.com/New-York-City-Storm-User-Group/">NYC Storm User Group</a> <span class="small">(New York, NY)</span></li>
-
- <li><a href="http://www.meetup.com/Bay-Area-Stream-Processing">Bay Area Stream Processing</a> <span class="small">(Emeryville, CA)</span></li>
-
- <li><a href="http://www.meetup.com/Boston-Storm-Users/">Boston Realtime Data</a> <span class="small">(Boston, MA)</span></li>
-
- <li><a href="http://www.meetup.com/storm-london">London Storm User Group</a> <span class="small">(London, UK)</span></li>
-
- <!-- <li><a href="http://www.meetup.com/Apache-Storm-Kafka-Users/">Seatle, WA</a> <span class="small">(27 Jun 2015)</span></li> -->
- </ul>
- </div>
- </div>
- <div class="col-md-3">
- <div class="footer-widget">
- <h5>About Storm</h5>
- <p>Storm integrates with any queueing system and any database system. Storm's spout abstraction makes it easy to integrate a new queuing system. Likewise, integrating Storm with database systems is easy.</p>
- </div>
- </div>
- <div class="col-md-3">
- <div class="footer-widget">
- <h5>First Look</h5>
- <ul class="footer-list">
- <li><a href="/documentation/Rationale.html">Rationale</a></li>
- <li><a href="/tutorial.html">Tutorial</a></li>
- <li><a href="/documentation/Setting-up-development-environment.html">Setting up development environment</a></li>
- <li><a href="/documentation/Creating-a-new-Storm-project.html">Creating a new Storm project</a></li>
- </ul>
- </div>
- </div>
- <div class="col-md-3">
- <div class="footer-widget">
- <h5>Documentation</h5>
- <ul class="footer-list">
- <li><a href="/doc-index.html">Index</a></li>
- <li><a href="/documentation.html">Manual</a></li>
- <li><a href="https://storm.apache.org/javadoc/apidocs/index.html">Javadoc</a></li>
- <li><a href="/documentation/FAQ.html">FAQ</a></li>
- </ul>
- </div>
- </div>
- </div>
- <hr/>
- <div class="row">
- <div class="col-md-12">
- <p align="center">Copyright © 2015 <a href="http://www.apache.org">Apache Software Foundation</a>. All Rights Reserved.
- <br>Apache Storm, Apache, the Apache feather logo, and the Apache Storm project logos are trademarks of The Apache Software Foundation.
- <br>All other marks mentioned may be trademarks or registered trademarks of their respective owners.</p>
- </div>
- </div>
- </div>
-</footer>
-<!--Footer End-->
-<!-- Scroll to top -->
-<span class="totop"><a href="#"><i class="fa fa-angle-up"></i></a></span>
-
-</body>
-
-</html>
-
http://git-wip-us.apache.org/repos/asf/storm/blob/335bbf94/_site/documentation/images/ack_tree.png
----------------------------------------------------------------------
diff --git a/_site/documentation/images/ack_tree.png b/_site/documentation/images/ack_tree.png
deleted file mode 100644
index 2134cc8..0000000
Binary files a/_site/documentation/images/ack_tree.png and /dev/null differ
http://git-wip-us.apache.org/repos/asf/storm/blob/335bbf94/_site/documentation/images/batched-stream.png
----------------------------------------------------------------------
diff --git a/_site/documentation/images/batched-stream.png b/_site/documentation/images/batched-stream.png
deleted file mode 100644
index 1e6aa01..0000000
Binary files a/_site/documentation/images/batched-stream.png and /dev/null differ
http://git-wip-us.apache.org/repos/asf/storm/blob/335bbf94/_site/documentation/images/drpc-workflow.png
----------------------------------------------------------------------
diff --git a/_site/documentation/images/drpc-workflow.png b/_site/documentation/images/drpc-workflow.png
deleted file mode 100644
index 9905648..0000000
Binary files a/_site/documentation/images/drpc-workflow.png and /dev/null differ
http://git-wip-us.apache.org/repos/asf/storm/blob/335bbf94/_site/documentation/images/eclipse-project-properties.png
----------------------------------------------------------------------
diff --git a/_site/documentation/images/eclipse-project-properties.png b/_site/documentation/images/eclipse-project-properties.png
deleted file mode 100644
index 62f8d32..0000000
Binary files a/_site/documentation/images/eclipse-project-properties.png and /dev/null differ
http://git-wip-us.apache.org/repos/asf/storm/blob/335bbf94/_site/documentation/images/example-of-a-running-topology.png
----------------------------------------------------------------------
diff --git a/_site/documentation/images/example-of-a-running-topology.png b/_site/documentation/images/example-of-a-running-topology.png
deleted file mode 100644
index 1462b21..0000000
Binary files a/_site/documentation/images/example-of-a-running-topology.png and /dev/null differ
http://git-wip-us.apache.org/repos/asf/storm/blob/335bbf94/_site/documentation/images/grouping.png
----------------------------------------------------------------------
diff --git a/_site/documentation/images/grouping.png b/_site/documentation/images/grouping.png
deleted file mode 100644
index 3911286..0000000
Binary files a/_site/documentation/images/grouping.png and /dev/null differ
http://git-wip-us.apache.org/repos/asf/storm/blob/335bbf94/_site/documentation/images/ld-library-path-eclipse-linux.png
----------------------------------------------------------------------
diff --git a/_site/documentation/images/ld-library-path-eclipse-linux.png b/_site/documentation/images/ld-library-path-eclipse-linux.png
deleted file mode 100644
index b6fbbd9..0000000
Binary files a/_site/documentation/images/ld-library-path-eclipse-linux.png and /dev/null differ
http://git-wip-us.apache.org/repos/asf/storm/blob/335bbf94/_site/documentation/images/nimbus_ha_leader_election_and_failover.png
----------------------------------------------------------------------
diff --git a/_site/documentation/images/nimbus_ha_leader_election_and_failover.png b/_site/documentation/images/nimbus_ha_leader_election_and_failover.png
deleted file mode 100644
index 60cc1b7..0000000
Binary files a/_site/documentation/images/nimbus_ha_leader_election_and_failover.png and /dev/null differ
http://git-wip-us.apache.org/repos/asf/storm/blob/335bbf94/_site/documentation/images/nimbus_ha_topology_submission.png
----------------------------------------------------------------------
diff --git a/_site/documentation/images/nimbus_ha_topology_submission.png b/_site/documentation/images/nimbus_ha_topology_submission.png
deleted file mode 100644
index 7707e5a..0000000
Binary files a/_site/documentation/images/nimbus_ha_topology_submission.png and /dev/null differ
http://git-wip-us.apache.org/repos/asf/storm/blob/335bbf94/_site/documentation/images/relationships-worker-processes-executors-tasks.png
----------------------------------------------------------------------
diff --git a/_site/documentation/images/relationships-worker-processes-executors-tasks.png b/_site/documentation/images/relationships-worker-processes-executors-tasks.png
deleted file mode 100644
index ef6f3fd..0000000
Binary files a/_site/documentation/images/relationships-worker-processes-executors-tasks.png and /dev/null differ
http://git-wip-us.apache.org/repos/asf/storm/blob/335bbf94/_site/documentation/images/spout-vs-state.png
----------------------------------------------------------------------
diff --git a/_site/documentation/images/spout-vs-state.png b/_site/documentation/images/spout-vs-state.png
deleted file mode 100644
index b6b06b3..0000000
Binary files a/_site/documentation/images/spout-vs-state.png and /dev/null differ
http://git-wip-us.apache.org/repos/asf/storm/blob/335bbf94/_site/documentation/images/storm-cluster.png
----------------------------------------------------------------------
diff --git a/_site/documentation/images/storm-cluster.png b/_site/documentation/images/storm-cluster.png
deleted file mode 100644
index df2ddb8..0000000
Binary files a/_site/documentation/images/storm-cluster.png and /dev/null differ
http://git-wip-us.apache.org/repos/asf/storm/blob/335bbf94/_site/documentation/images/topology-tasks.png
----------------------------------------------------------------------
diff --git a/_site/documentation/images/topology-tasks.png b/_site/documentation/images/topology-tasks.png
deleted file mode 100644
index 0affaba..0000000
Binary files a/_site/documentation/images/topology-tasks.png and /dev/null differ
http://git-wip-us.apache.org/repos/asf/storm/blob/335bbf94/_site/documentation/images/topology.png
----------------------------------------------------------------------
diff --git a/_site/documentation/images/topology.png b/_site/documentation/images/topology.png
deleted file mode 100644
index a45c25c..0000000
Binary files a/_site/documentation/images/topology.png and /dev/null differ
http://git-wip-us.apache.org/repos/asf/storm/blob/335bbf94/_site/documentation/images/transactional-batches.png
----------------------------------------------------------------------
diff --git a/_site/documentation/images/transactional-batches.png b/_site/documentation/images/transactional-batches.png
deleted file mode 100644
index db2716b..0000000
Binary files a/_site/documentation/images/transactional-batches.png and /dev/null differ
http://git-wip-us.apache.org/repos/asf/storm/blob/335bbf94/_site/documentation/images/transactional-commit-flow.png
----------------------------------------------------------------------
diff --git a/_site/documentation/images/transactional-commit-flow.png b/_site/documentation/images/transactional-commit-flow.png
deleted file mode 100644
index 25131b0..0000000
Binary files a/_site/documentation/images/transactional-commit-flow.png and /dev/null differ
http://git-wip-us.apache.org/repos/asf/storm/blob/335bbf94/_site/documentation/images/transactional-design-2.png
----------------------------------------------------------------------
diff --git a/_site/documentation/images/transactional-design-2.png b/_site/documentation/images/transactional-design-2.png
deleted file mode 100644
index a0e0ebf..0000000
Binary files a/_site/documentation/images/transactional-design-2.png and /dev/null differ
http://git-wip-us.apache.org/repos/asf/storm/blob/335bbf94/_site/documentation/images/transactional-spout-structure.png
----------------------------------------------------------------------
diff --git a/_site/documentation/images/transactional-spout-structure.png b/_site/documentation/images/transactional-spout-structure.png
deleted file mode 100644
index ecf7def..0000000
Binary files a/_site/documentation/images/transactional-spout-structure.png and /dev/null differ
http://git-wip-us.apache.org/repos/asf/storm/blob/335bbf94/_site/documentation/images/trident-to-storm1.png
----------------------------------------------------------------------
diff --git a/_site/documentation/images/trident-to-storm1.png b/_site/documentation/images/trident-to-storm1.png
deleted file mode 100644
index b022776..0000000
Binary files a/_site/documentation/images/trident-to-storm1.png and /dev/null differ
http://git-wip-us.apache.org/repos/asf/storm/blob/335bbf94/_site/documentation/images/trident-to-storm2.png
----------------------------------------------------------------------
diff --git a/_site/documentation/images/trident-to-storm2.png b/_site/documentation/images/trident-to-storm2.png
deleted file mode 100644
index 6aa0fc5..0000000
Binary files a/_site/documentation/images/trident-to-storm2.png and /dev/null differ
http://git-wip-us.apache.org/repos/asf/storm/blob/335bbf94/_site/documentation/images/tuple-dag.png
----------------------------------------------------------------------
diff --git a/_site/documentation/images/tuple-dag.png b/_site/documentation/images/tuple-dag.png
deleted file mode 100644
index 34611d4..0000000
Binary files a/_site/documentation/images/tuple-dag.png and /dev/null differ
http://git-wip-us.apache.org/repos/asf/storm/blob/335bbf94/_site/documentation/images/tuple_tree.png
----------------------------------------------------------------------
diff --git a/_site/documentation/images/tuple_tree.png b/_site/documentation/images/tuple_tree.png
deleted file mode 100644
index b14f558..0000000
Binary files a/_site/documentation/images/tuple_tree.png and /dev/null differ
http://git-wip-us.apache.org/repos/asf/storm/blob/335bbf94/_site/documentation/nimbus-ha-design.md
----------------------------------------------------------------------
diff --git a/_site/documentation/nimbus-ha-design.md b/_site/documentation/nimbus-ha-design.md
deleted file mode 100644
index 672eece..0000000
--- a/_site/documentation/nimbus-ha-design.md
+++ /dev/null
@@ -1,217 +0,0 @@
-#Highly Available Nimbus design proposal
-##Problem Statement:
-Currently the storm master aka nimbus, is a process that runs on a single machine under supervision. In most cases the
-nimbus failure is transient and it is restarted by the supervisor. However sometimes when disks fail and networks
-partitions occur, nimbus goes down. Under these circumstances the topologies run normally but no new topologies can be
-submitted, no existing topologies can be killed/deactivated/activated and if a supervisor node fails then the
-reassignments are not performed resulting in performance degradation or topology failures. With this project we intend
-to resolve this problem by running nimbus in a primary backup mode to guarantee that even if a nimbus server fails one
-of the backups will take over.
-##Requirements:
-* Increase overall availability of nimbus.
-* Allow nimbus hosts to leave and join the cluster at will any time. A newly joined host should auto catch up and join
-the list of potential leaders automatically.
-* No topology resubmissions required in case of nimbus fail overs.
-* No active topology should ever be lost.
-
-##Leader Election:
-The nimbus server will use the following interface:
-
-```java
-public interface ILeaderElector {
- /**
- * queue up for leadership lock. The call returns immediately and the caller
- * must check isLeader() to perform any leadership action.
- */
- void addToLeaderLockQueue();
-
- /**
- * Removes the caller from the leader lock queue. If the caller is leader
- * also releases the lock.
- */
- void removeFromLeaderLockQueue();
-
- /**
- *
- * @return true if the caller currently has the leader lock.
- */
- boolean isLeader();
-
- /**
- *
- * @return the current leader's address , throws exception if noone has has lock.
- */
- InetSocketAddress getLeaderAddress();
-
- /**
- *
- * @return list of current nimbus addresses, includes leader.
- */
- List<InetSocketAddress> getAllNimbusAddresses();
-}
-```
-On startup nimbus will check if it has code for all active topologies available locally. Once it gets to this state it
-will call addToLeaderLockQueue() function. When a nimbus is notified to become a leader it will check if it has all the
-code locally before assuming the leadership role. If any active topology code is missing, the node will not accept the
-leadership role instead it will release the lock and wait till it has all the code before requeueing for leader lock.
-
-The first implementation will be Zookeeper based. If the zookeeper connection is lost/resetted resulting in loss of lock
-or the spot in queue the implementation will take care of updating the state such that isLeader() will reflect the
-current status.The leader like actions must finish in less than minimumOf(connectionTimeout, SessionTimeout) to ensure
-the lock was held by nimbus for the entire duration of the action (Not sure if we want to just state this expectation
-and ensure that zk configurations are set high enough which will result in higher failover time or we actually want to
-create some sort of rollback mechanism for all actions, the second option needs a lot of code). If a nimbus that is not
-leader receives a request that only a leader can perform it will throw a RunTimeException.
-
-Following steps describes a nimbus failover scenario:
-* Let’s say we have 4 topologies running with 3 nimbus nodes and code-replication-factor = 2. We assume that the
-invariant “The leader nimbus has code for all topologies locally” holds true at the beginning. nonleader-1 has code for
-the first 2 topologies and nonLeader-2 has code for the other 2 topologies.
-* Leader nimbus dies, hard disk failure so no recovery possible.
-* nonLeader-1 gets a zookeeper notification to indicate it is now the new leader. before accepting the leadership it
-checks if it has code available for all 4 topologies(these are topologies under /storm/storms/). It realizes it only has
-code for 2 topologies so it relinquishes the lock and looks under /storm/code-distributor/topologyId to find out from
-where can it download the code/metafile for the missing topologies. it finds entries for the leader nimbus and
-nonleader-2. It will try downloading from both as part of its retry mechanism.
-* nonLeader-2’s code sync thread also realizes that it is missing code for 2 topologies and follows the same process
-described in step-3 to download code for missing topologies.
-* eventually at least one of the nimbuses will have all the code locally and will accept leadership.
-This sequence diagram describes how leader election and failover would work with multiple components.
-
-
-
-##Nimbus state store:
-
-Currently the nimbus stores 2 kind of data
-* Meta information like supervisor info, assignment info which is stored in zookeeper
-* Actual topology configs and jars that is stored on nimbus host’s local disk.
-
-To achieve fail over from primary to backup servers nimbus state/data needs to be replicated across all nimbus hosts or
-needs to be stored in a distributed storage. Replicating the data correctly involves state management, consistency checks
-and it is hard to test for correctness.However many storm users do not want to take extra dependency on another replicated
-storage system like HDFS and still need high availability.Eventually, we want to move to the bittorrent protocol for code
-distribution given the size of the jars and to achieve better scaling when the total number of supervisors is very high.
-The current file system based model for code distribution works fine with systems that have file system like structure
-but it fails to support a non file system based approach like bit torrent. To support bit torrent and all the file
-system based replicated storage systems we propose the following interface:
-
-```java
-/**
- * Interface responsible to distribute code in the cluster.
- */
-public interface ICodeDistributor {
- /**
- * Prepare this code distributor.
- * @param conf
- */
- void prepare(Map conf);
-
- /**
- * This API will perform the actual upload of the code to the distributed implementation.
- * The API should return a Meta file which should have enough information for downloader
- * so it can download the code e.g. for bittorrent it will be a torrent file, in case of something
- * like HDFS or s3 it might have the actual directory or paths for files to be downloaded.
- * @param dirPath local directory where all the code to be distributed exists.
- * @param topologyId the topologyId for which the meta file needs to be created.
- * @return metaFile
- */
- File upload(Path dirPath, String topologyId);
-
- /**
- * Given the topologyId and metafile, download the actual code and return the downloaded file's list.
- * @param topologyid
- * @param metafile
- * @param destDirPath the folder where all the files will be downloaded.
- * @return
- */
- List<File> download(Path destDirPath, String topologyid, File metafile);
-
- /**
- * Given the topologyId, returns number of hosts where the code has been replicated.
- */
- int getReplicationCount(String topologyId);
-
- /**
- * Performs the cleanup.
- * @param topologyid
- */
- void cleanup(String topologyid);
-
- /**
- * Close this distributor.
- * @param conf
- */
- void close(Map conf);
-}
-```
-To support replication we will allow the user to define a code replication factor which would reflect number of nimbus
-hosts to which the code must be replicated before starting the topology. With replication comes the issue of consistency.
-We will treat zookeeper’s list of active topologies as our authority for topologies for which the code must exist on a
-nimbus host. Any nimbus host that does not have all the code for all the topologies which are marked as active in zookeeper
-will relinquish it’s lock so some other nimbus host could become leader. A background thread on all nimbus host will
-continuously try to sync code from other hosts where the code was successfully replicated so eventually at least one nimbus
-will accept leadership as long as at least one seed hosts exists for each active topology.
-
-Following steps describe code replication amongst nimbus hosts for a topology:
-* When client uploads jar, nothing changes.
-* When client submits a topology, leader nimbus calls code distributor’s upload function which will create a metafile stored
-locally on leader nimbus. Leader nimbus will write new entries under /storm/code-distributor/topologyId to notify all
-nonleader nimbuses that they should download this new code.
-* We wait on the leader nimbus to ensure at least N non leader nimbus has the code replicated, with a user configurable timeout.
-* When a non leader nimbus receives the notification about new code, it downloads the meta file from leader nimbus and then
-downloads the real code by calling code distributor’s download function with metafile as input.
-* Once non leader finishes downloading code, it will write an entry under /storm/code-distributor/topologyId to indicate
-it is one of the possible places to download the code/metafile in case the leader nimbus dies.
-* leader nimbus goes ahead and does all the usual things it does as part of submit topologies.
-
-The following sequence diagram describes the communication between different components involved in code distribution.
-
-
-
-##Thrift and Rest API
-In order to avoid workers/supervisors/ui talking to zookeeper for getting master nimbus address we are going to modify the
-`getClusterInfo` API so it can also return nimbus information. getClusterInfo currently returns `ClusterSummary` instance
-which has a list of `supervisorSummary` and a list of 'topologySummary` instances. We will add a list of `NimbusSummary`
-to the `ClusterSummary`. See the structures below:
-
-```thrift
-struct ClusterSummary {
- 1: required list<SupervisorSummary> supervisors;
- 3: required list<TopologySummary> topologies;
- 4: required list<NimbusSummary> nimbuses;
-}
-
-struct NimbusSummary {
- 1: required string host;
- 2: required i32 port;
- 3: required i32 uptime_secs;
- 4: required bool isLeader;
- 5: required string version;
-}
-```
-
-This will be used by StormSubmitter, Nimbus clients,supervisors and ui to discover the current leaders and participating
-nimbus hosts. Any nimbus host will be able to respond to these requests. The nimbus hosts can read this information once
-from zookeeper and cache it and keep updating the cache when the watchers are fired to indicate any changes,which should
-be rare in general case.
-
-## Configuration
-You can use nimbus ha with default configuration , however the default configuration assumes a single nimbus host so it
-trades off replication for lower topology submission latency. Depending on your use case you can adjust following configurations:
-* storm.codedistributor.class : This is a string representing fully qualified class name of a class that implements
-backtype.storm.codedistributor.ICodeDistributor. The default is set to "backtype.storm.codedistributor.LocalFileSystemCodeDistributor".
-This class leverages local file system to store both meta files and code/configs. This class adds extra load on zookeeper as even after
-downloading the code-distrbutor meta file it contacts zookeeper in order to figure out hosts from where it can download
-actual code/config and to get the current replication count. An alternative is to use
-"org.apache.storm.hdfs.ha.codedistributor.HDFSCodeDistributor" which relies on HDFS but does not add extra load on zookeeper and will
-make topology submission faster.
-* topology.min.replication.count : Minimum number of nimbus hosts where the code must be replicated before leader nimbus
-can mark the topology as active and create assignments. Default is 1.
-* topology.max.replication.wait.time.sec: Maximum wait time for the nimbus host replication to achieve the nimbus.min.replication.count.
-Once this time is elapsed nimbus will go ahead and perform topology activation tasks even if required nimbus.min.replication.count is not achieved.
-The default is 60 seconds, a value of -1 indicates to wait for ever.
-*nimbus.code.sync.freq.secs: frequency at which the background thread on nimbus which syncs code for locally missing topologies will run. default is 5 minutes.
-
-Note: Even though all nimbus hosts have watchers on zookeeper to be notified immediately as soon as a new topology is available for code
-download, the callback pretty much never results in code download. In practice we have observed that the desired replication is only achieved once the background-thread runs.
-So you should expect your topology submission time to be somewhere between 0 to (2 * nimbus.code.sync.freq.secs) for any nimbus.min.replication.count > 1.
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/storm/blob/335bbf94/_site/documentation/storm-eventhubs.html
----------------------------------------------------------------------
diff --git a/_site/documentation/storm-eventhubs.html b/_site/documentation/storm-eventhubs.html
deleted file mode 100644
index 49cb0aa..0000000
--- a/_site/documentation/storm-eventhubs.html
+++ /dev/null
@@ -1,203 +0,0 @@
-<!DOCTYPE html>
-<html>
- <head>
- <meta charset="utf-8">
- <meta http-equiv="X-UA-Compatible" content="IE=edge">
- <meta name="viewport" content="width=device-width, initial-scale=1">
-
- <link rel="shortcut icon" href="/favicon.ico" type="image/x-icon">
- <link rel="icon" href="/favicon.ico" type="image/x-icon">
-
- <title>Azue Event Hubs Integration</title>
-
- <!-- Bootstrap core CSS -->
- <link href="/assets/css/bootstrap.min.css" rel="stylesheet">
- <!-- Bootstrap theme -->
- <link href="/assets/css/bootstrap-theme.min.css" rel="stylesheet">
-
- <!-- Custom styles for this template -->
- <link rel="stylesheet" href="http://fortawesome.github.io/Font-Awesome/assets/font-awesome/css/font-awesome.css">
- <link href="/css/style.css" rel="stylesheet">
- <link href="/assets/css/owl.theme.css" rel="stylesheet">
- <link href="/assets/css/owl.carousel.css" rel="stylesheet">
- <script type="text/javascript" src="/assets/js/jquery.min.js"></script>
- <script type="text/javascript" src="/assets/js/bootstrap.min.js"></script>
- <script type="text/javascript" src="/assets/js/owl.carousel.min.js"></script>
- <script type="text/javascript" src="/assets/js/storm.js"></script>
- <!-- Just for debugging purposes. Don't actually copy these 2 lines! -->
- <!--[if lt IE 9]><script src="../../assets/js/ie8-responsive-file-warning.js"></script><![endif]-->
-
- <!-- HTML5 shim and Respond.js for IE8 support of HTML5 elements and media queries -->
- <!--[if lt IE 9]>
- <script src="https://oss.maxcdn.com/html5shiv/3.7.2/html5shiv.min.js"></script>
- <script src="https://oss.maxcdn.com/respond/1.4.2/respond.min.js"></script>
- <![endif]-->
- </head>
-
-
- <body>
- <header>
- <div class="container-fluid">
- <div class="row">
- <div class="col-md-10">
- <a href="/index.html"><img src="/images/logo.png" class="logo" /></a>
- </div>
- <div class="col-md-2">
- <a href="/downloads.html" class="btn-std btn-block btn-download">Download</a>
- </div>
- </div>
- </div>
-</header>
-<!--Header End-->
-<!--Navigation Begin-->
-<div class="navbar" role="banner">
- <div class="container-fluid">
- <div class="navbar-header">
- <button class="navbar-toggle" type="button" data-toggle="collapse" data-target=".bs-navbar-collapse">
- <span class="icon-bar"></span>
- <span class="icon-bar"></span>
- <span class="icon-bar"></span>
- </button>
- </div>
- <nav class="collapse navbar-collapse bs-navbar-collapse" role="navigation">
- <ul class="nav navbar-nav">
- <li><a href="/index.html" id="home">Home</a></li>
- <li><a href="/getting-help.html" id="getting-help">Getting Help</a></li>
- <li><a href="/about/integrates.html" id="project-info">Project Information</a></li>
- <li><a href="/documentation.html" id="documentation">Documentation</a></li>
- <li><a href="/talksAndVideos.html">Talks and Slideshows</a></li>
- <li class="dropdown">
- <a href="#" class="dropdown-toggle" data-toggle="dropdown" id="contribute">Community <b class="caret"></b></a>
- <ul class="dropdown-menu">
- <li><a href="/contribute/Contributing-to-Storm.html">Contributing</a></li>
- <li><a href="/contribute/People.html">People</a></li>
- <li><a href="/contribute/BYLAWS.html">ByLaws</a></li>
- </ul>
- </li>
- <li><a href="/2015/11/05/storm096-released.html" id="news">News</a></li>
- </ul>
- </nav>
- </div>
-</div>
-
-
-
- <div class="container-fluid">
- <h1 class="page-title">Azue Event Hubs Integration</h1>
- <div class="row">
- <div class="col-md-12">
- <!-- Documentation -->
-
-<p class="post-meta"></p>
-
-<p>Storm spout and bolt implementation for Microsoft Azure Eventhubs</p>
-
-<h3 id="build">build</h3>
-<div class="highlight"><pre><code class="language-" data-lang="">mvn clean package
-</code></pre></div>
-<h3 id="run-sample-topology">run sample topology</h3>
-
-<p>To run the sample topology, you need to modify the config.properties file with
-the eventhubs configurations. Here is an example:</p>
-<div class="highlight"><pre><code class="language-" data-lang="">eventhubspout.username = [username: policy name in EventHubs Portal]
-eventhubspout.password = [password: shared access key in EventHubs Portal]
-eventhubspout.namespace = [namespace]
-eventhubspout.entitypath = [entitypath]
-eventhubspout.partitions.count = [partitioncount]
-
-# if not provided, will use storm's zookeeper settings
-# zookeeper.connectionstring=zookeeper0:2181,zookeeper1:2181,zookeeper2:2181
-
-eventhubspout.checkpoint.interval = 10
-eventhub.receiver.credits = 1024
-</code></pre></div>
-<p>Then you can use storm.cmd to submit the sample topology:
- storm jar {jarfile} com.microsoft.eventhubs.samples.EventCount {topologyname} {spoutconffile}
- where the {jarfile} should be: eventhubs-storm-spout-{version}-jar-with-dependencies.jar</p>
-
-<h3 id="run-eventhubsendclient">Run EventHubSendClient</h3>
-
-<p>We have included a simple EventHubs send client for testing purpose. You can run the client like this:
- java -cp .\target\eventhubs-storm-spout-{version}-jar-with-dependencies.jar com.microsoft.eventhubs.client.EventHubSendClient
- [username] [password] [entityPath] [partitionId] [messageSize] [messageCount]
-If you want to send messages to all partitions, use "-1" as partitionId.</p>
-
-<h3 id="windows-azure-eventhubs">Windows Azure Eventhubs</h3>
-<div class="highlight"><pre><code class="language-" data-lang="">http://azure.microsoft.com/en-us/services/event-hubs/
-</code></pre></div>
-
-
- </div>
- </div>
- </div>
-<footer>
- <div class="container-fluid">
- <div class="row">
- <div class="col-md-3">
- <div class="footer-widget">
- <h5>Meetups</h5>
- <ul class="latest-news">
-
- <li><a href="http://www.meetup.com/Apache-Storm-Apache-Kafka/">Apache Storm & Apache Kafka</a> <span class="small">(Sunnyvale, CA)</span></li>
-
- <li><a href="http://www.meetup.com/Apache-Storm-Kafka-Users/">Apache Storm & Kafka Users</a> <span class="small">(Seattle, WA)</span></li>
-
- <li><a href="http://www.meetup.com/New-York-City-Storm-User-Group/">NYC Storm User Group</a> <span class="small">(New York, NY)</span></li>
-
- <li><a href="http://www.meetup.com/Bay-Area-Stream-Processing">Bay Area Stream Processing</a> <span class="small">(Emeryville, CA)</span></li>
-
- <li><a href="http://www.meetup.com/Boston-Storm-Users/">Boston Realtime Data</a> <span class="small">(Boston, MA)</span></li>
-
- <li><a href="http://www.meetup.com/storm-london">London Storm User Group</a> <span class="small">(London, UK)</span></li>
-
- <!-- <li><a href="http://www.meetup.com/Apache-Storm-Kafka-Users/">Seatle, WA</a> <span class="small">(27 Jun 2015)</span></li> -->
- </ul>
- </div>
- </div>
- <div class="col-md-3">
- <div class="footer-widget">
- <h5>About Storm</h5>
- <p>Storm integrates with any queueing system and any database system. Storm's spout abstraction makes it easy to integrate a new queuing system. Likewise, integrating Storm with database systems is easy.</p>
- </div>
- </div>
- <div class="col-md-3">
- <div class="footer-widget">
- <h5>First Look</h5>
- <ul class="footer-list">
- <li><a href="/documentation/Rationale.html">Rationale</a></li>
- <li><a href="/tutorial.html">Tutorial</a></li>
- <li><a href="/documentation/Setting-up-development-environment.html">Setting up development environment</a></li>
- <li><a href="/documentation/Creating-a-new-Storm-project.html">Creating a new Storm project</a></li>
- </ul>
- </div>
- </div>
- <div class="col-md-3">
- <div class="footer-widget">
- <h5>Documentation</h5>
- <ul class="footer-list">
- <li><a href="/doc-index.html">Index</a></li>
- <li><a href="/documentation.html">Manual</a></li>
- <li><a href="https://storm.apache.org/javadoc/apidocs/index.html">Javadoc</a></li>
- <li><a href="/documentation/FAQ.html">FAQ</a></li>
- </ul>
- </div>
- </div>
- </div>
- <hr/>
- <div class="row">
- <div class="col-md-12">
- <p align="center">Copyright © 2015 <a href="http://www.apache.org">Apache Software Foundation</a>. All Rights Reserved.
- <br>Apache Storm, Apache, the Apache feather logo, and the Apache Storm project logos are trademarks of The Apache Software Foundation.
- <br>All other marks mentioned may be trademarks or registered trademarks of their respective owners.</p>
- </div>
- </div>
- </div>
-</footer>
-<!--Footer End-->
-<!-- Scroll to top -->
-<span class="totop"><a href="#"><i class="fa fa-angle-up"></i></a></span>
-
-</body>
-
-</html>
-
http://git-wip-us.apache.org/repos/asf/storm/blob/335bbf94/_site/documentation/storm-hbase.html
----------------------------------------------------------------------
diff --git a/_site/documentation/storm-hbase.html b/_site/documentation/storm-hbase.html
deleted file mode 100644
index 5c638f8..0000000
--- a/_site/documentation/storm-hbase.html
+++ /dev/null
@@ -1,384 +0,0 @@
-<!DOCTYPE html>
-<html>
- <head>
- <meta charset="utf-8">
- <meta http-equiv="X-UA-Compatible" content="IE=edge">
- <meta name="viewport" content="width=device-width, initial-scale=1">
-
- <link rel="shortcut icon" href="/favicon.ico" type="image/x-icon">
- <link rel="icon" href="/favicon.ico" type="image/x-icon">
-
- <title>Storm HBase Integration</title>
-
- <!-- Bootstrap core CSS -->
- <link href="/assets/css/bootstrap.min.css" rel="stylesheet">
- <!-- Bootstrap theme -->
- <link href="/assets/css/bootstrap-theme.min.css" rel="stylesheet">
-
- <!-- Custom styles for this template -->
- <link rel="stylesheet" href="http://fortawesome.github.io/Font-Awesome/assets/font-awesome/css/font-awesome.css">
- <link href="/css/style.css" rel="stylesheet">
- <link href="/assets/css/owl.theme.css" rel="stylesheet">
- <link href="/assets/css/owl.carousel.css" rel="stylesheet">
- <script type="text/javascript" src="/assets/js/jquery.min.js"></script>
- <script type="text/javascript" src="/assets/js/bootstrap.min.js"></script>
- <script type="text/javascript" src="/assets/js/owl.carousel.min.js"></script>
- <script type="text/javascript" src="/assets/js/storm.js"></script>
- <!-- Just for debugging purposes. Don't actually copy these 2 lines! -->
- <!--[if lt IE 9]><script src="../../assets/js/ie8-responsive-file-warning.js"></script><![endif]-->
-
- <!-- HTML5 shim and Respond.js for IE8 support of HTML5 elements and media queries -->
- <!--[if lt IE 9]>
- <script src="https://oss.maxcdn.com/html5shiv/3.7.2/html5shiv.min.js"></script>
- <script src="https://oss.maxcdn.com/respond/1.4.2/respond.min.js"></script>
- <![endif]-->
- </head>
-
-
- <body>
- <header>
- <div class="container-fluid">
- <div class="row">
- <div class="col-md-10">
- <a href="/index.html"><img src="/images/logo.png" class="logo" /></a>
- </div>
- <div class="col-md-2">
- <a href="/downloads.html" class="btn-std btn-block btn-download">Download</a>
- </div>
- </div>
- </div>
-</header>
-<!--Header End-->
-<!--Navigation Begin-->
-<div class="navbar" role="banner">
- <div class="container-fluid">
- <div class="navbar-header">
- <button class="navbar-toggle" type="button" data-toggle="collapse" data-target=".bs-navbar-collapse">
- <span class="icon-bar"></span>
- <span class="icon-bar"></span>
- <span class="icon-bar"></span>
- </button>
- </div>
- <nav class="collapse navbar-collapse bs-navbar-collapse" role="navigation">
- <ul class="nav navbar-nav">
- <li><a href="/index.html" id="home">Home</a></li>
- <li><a href="/getting-help.html" id="getting-help">Getting Help</a></li>
- <li><a href="/about/integrates.html" id="project-info">Project Information</a></li>
- <li><a href="/documentation.html" id="documentation">Documentation</a></li>
- <li><a href="/talksAndVideos.html">Talks and Slideshows</a></li>
- <li class="dropdown">
- <a href="#" class="dropdown-toggle" data-toggle="dropdown" id="contribute">Community <b class="caret"></b></a>
- <ul class="dropdown-menu">
- <li><a href="/contribute/Contributing-to-Storm.html">Contributing</a></li>
- <li><a href="/contribute/People.html">People</a></li>
- <li><a href="/contribute/BYLAWS.html">ByLaws</a></li>
- </ul>
- </li>
- <li><a href="/2015/11/05/storm096-released.html" id="news">News</a></li>
- </ul>
- </nav>
- </div>
-</div>
-
-
-
- <div class="container-fluid">
- <h1 class="page-title">Storm HBase Integration</h1>
- <div class="row">
- <div class="col-md-12">
- <!-- Documentation -->
-
-<p class="post-meta"></p>
-
-<p>Storm/Trident integration for <a href="https://hbase.apache.org">Apache HBase</a></p>
-
-<h2 id="usage">Usage</h2>
-
-<p>The main API for interacting with HBase is the <code>org.apache.storm.hbase.bolt.mapper.HBaseMapper</code>
-interface:</p>
-<div class="highlight"><pre><code class="language-java" data-lang="java"><span class="kd">public</span> <span class="kd">interface</span> <span class="nc">HBaseMapper</span> <span class="kd">extends</span> <span class="n">Serializable</span> <span class="o">{</span>
- <span class="kt">byte</span><span class="o">[]</span> <span class="n">rowKey</span><span class="o">(</span><span class="n">Tuple</span> <span class="n">tuple</span><span class="o">);</span>
-
- <span class="n">ColumnList</span> <span class="n">columns</span><span class="o">(</span><span class="n">Tuple</span> <span class="n">tuple</span><span class="o">);</span>
-<span class="o">}</span>
-</code></pre></div>
-<p>The <code>rowKey()</code> method is straightforward: given a Storm tuple, return a byte array representing the
-row key.</p>
-
-<p>The <code>columns()</code> method defines what will be written to an HBase row. The <code>ColumnList</code> class allows you
-to add both standard HBase columns as well as HBase counter columns.</p>
-
-<p>To add a standard column, use one of the <code>addColumn()</code> methods:</p>
-<div class="highlight"><pre><code class="language-java" data-lang="java"><span class="n">ColumnList</span> <span class="n">cols</span> <span class="o">=</span> <span class="k">new</span> <span class="n">ColumnList</span><span class="o">();</span>
-<span class="n">cols</span><span class="o">.</span><span class="na">addColumn</span><span class="o">(</span><span class="k">this</span><span class="o">.</span><span class="na">columnFamily</span><span class="o">,</span> <span class="n">field</span><span class="o">.</span><span class="na">getBytes</span><span class="o">(),</span> <span class="n">toBytes</span><span class="o">(</span><span class="n">tuple</span><span class="o">.</span><span class="na">getValueByField</span><span class="o">(</span><span class="n">field</span><span class="o">)));</span>
-</code></pre></div>
-<p>To add a counter column, use one of the <code>addCounter()</code> methods:</p>
-<div class="highlight"><pre><code class="language-java" data-lang="java"><span class="n">ColumnList</span> <span class="n">cols</span> <span class="o">=</span> <span class="k">new</span> <span class="n">ColumnList</span><span class="o">();</span>
-<span class="n">cols</span><span class="o">.</span><span class="na">addCounter</span><span class="o">(</span><span class="k">this</span><span class="o">.</span><span class="na">columnFamily</span><span class="o">,</span> <span class="n">field</span><span class="o">.</span><span class="na">getBytes</span><span class="o">(),</span> <span class="n">toLong</span><span class="o">(</span><span class="n">tuple</span><span class="o">.</span><span class="na">getValueByField</span><span class="o">(</span><span class="n">field</span><span class="o">)));</span>
-</code></pre></div>
-<p>When the remote HBase is security enabled, a kerberos keytab and the corresponding principal name need to be
-provided for the storm-hbase connector. Specifically, the Config object passed into the topology should contain
-{(“storm.keytab.file”, “$keytab”), ("storm.kerberos.principal", “$principal”)}. Example:</p>
-<div class="highlight"><pre><code class="language-java" data-lang="java"><span class="n">Config</span> <span class="n">config</span> <span class="o">=</span> <span class="k">new</span> <span class="n">Config</span><span class="o">();</span>
-<span class="o">...</span>
-<span class="n">config</span><span class="o">.</span><span class="na">put</span><span class="o">(</span><span class="s">"storm.keytab.file"</span><span class="o">,</span> <span class="s">"$keytab"</span><span class="o">);</span>
-<span class="n">config</span><span class="o">.</span><span class="na">put</span><span class="o">(</span><span class="s">"storm.kerberos.principal"</span><span class="o">,</span> <span class="s">"$principle"</span><span class="o">);</span>
-<span class="n">StormSubmitter</span><span class="o">.</span><span class="na">submitTopology</span><span class="o">(</span><span class="s">"$topologyName"</span><span class="o">,</span> <span class="n">config</span><span class="o">,</span> <span class="n">builder</span><span class="o">.</span><span class="na">createTopology</span><span class="o">());</span>
-</code></pre></div>
-<h2 id="working-with-secure-hbase-using-delegation-tokens">Working with Secure HBASE using delegation tokens.</h2>
-
-<p>If your topology is going to interact with secure HBase, your bolts/states needs to be authenticated by HBase.
-The approach described above requires that all potential worker hosts have "storm.keytab.file" on them. If you have
-multiple topologies on a cluster , each with different hbase user, you will have to create multiple keytabs and distribute
-it to all workers. Instead of doing that you could use the following approach:</p>
-
-<p>Your administrator can configure nimbus to automatically get delegation tokens on behalf of the topology submitter user.
-The nimbus need to start with following configurations:</p>
-
-<p>nimbus.autocredential.plugins.classes : ["org.apache.storm.hbase.security.AutoHBase"]
-nimbus.credential.renewers.classes : ["org.apache.storm.hbase.security.AutoHBase"]
-hbase.keytab.file: "/path/to/keytab/on/nimbus" (This is the keytab of hbase super user that can impersonate other users.)
-hbase.kerberos.principal: "<a href="mailto:superuser@EXAMPLE.com">superuser@EXAMPLE.com</a>"
-nimbus.credential.renewers.freq.secs : 518400 (6 days, hbase tokens by default expire every 7 days and can not be renewed,
-if you have custom settings for hbase.auth.token.max.lifetime in hbase-site.xml than you should ensure this value is
-atleast 1 hour less then that.)</p>
-
-<p>Your topology configuration should have:
-topology.auto-credentials :["org.apache.storm.hbase.security.AutoHBase"] </p>
-
-<p>If nimbus did not have the above configuration you need to add it and then restart it. Ensure the hbase configuration
-files(core-site.xml,hdfs-site.xml and hbase-site.xml) and the storm-hbase jar with all the dependencies is present in nimbus's classpath.
-Nimbus will use the keytab and principal specified in the config to authenticate with HBase. From then on for every
-topology submission, nimbus will impersonate the topology submitter user and acquire delegation tokens on behalf of the
-topology submitter user. If topology was started with topology.auto-credentials set to AutoHBase, nimbus will push the
-delegation tokens to all the workers for your topology and the hbase bolt/state will authenticate with these tokens.</p>
-
-<p>As nimbus is impersonating topology submitter user, you need to ensure the user specified in storm.kerberos.principal
-has permissions to acquire tokens on behalf of other users. To achieve this you need to follow configuration directions
-listed on this link</p>
-
-<p><a href="http://hbase.apache.org/book/security.html#security.rest.gateway">http://hbase.apache.org/book/security.html#security.rest.gateway</a></p>
-
-<p>You can read about setting up secure HBase here:<a href="http://hbase.apache.org/book/security.html">http://hbase.apache.org/book/security.html</a>.</p>
-
-<h3 id="simplehbasemapper">SimpleHBaseMapper</h3>
-
-<p><code>storm-hbase</code> includes a general purpose <code>HBaseMapper</code> implementation called <code>SimpleHBaseMapper</code> that can map Storm
-tuples to both regular HBase columns as well as counter columns.</p>
-
-<p>To use <code>SimpleHBaseMapper</code>, you simply tell it which fields to map to which types of columns.</p>
-
-<p>The following code create a <code>SimpleHBaseMapper</code> instance that:</p>
-
-<ol>
-<li>Uses the <code>word</code> tuple value as a row key.</li>
-<li>Adds a standard HBase column for the tuple field <code>word</code>.</li>
-<li>Adds an HBase counter column for the tuple field <code>count</code>.</li>
-<li>Writes values to the <code>cf</code> column family.</li>
-</ol>
-<div class="highlight"><pre><code class="language-java" data-lang="java"><span class="n">SimpleHBaseMapper</span> <span class="n">mapper</span> <span class="o">=</span> <span class="k">new</span> <span class="n">SimpleHBaseMapper</span><span class="o">()</span>
- <span class="o">.</span><span class="na">withRowKeyField</span><span class="o">(</span><span class="s">"word"</span><span class="o">)</span>
- <span class="o">.</span><span class="na">withColumnFields</span><span class="o">(</span><span class="k">new</span> <span class="n">Fields</span><span class="o">(</span><span class="s">"word"</span><span class="o">))</span>
- <span class="o">.</span><span class="na">withCounterFields</span><span class="o">(</span><span class="k">new</span> <span class="n">Fields</span><span class="o">(</span><span class="s">"count"</span><span class="o">))</span>
- <span class="o">.</span><span class="na">withColumnFamily</span><span class="o">(</span><span class="s">"cf"</span><span class="o">);</span>
-</code></pre></div>
-<h3 id="hbasebolt">HBaseBolt</h3>
-
-<p>To use the <code>HBaseBolt</code>, construct it with the name of the table to write to, an a <code>HBaseMapper</code> implementation:</p>
-<div class="highlight"><pre><code class="language-java" data-lang="java"><span class="n">HBaseBolt</span> <span class="n">hbase</span> <span class="o">=</span> <span class="k">new</span> <span class="n">HBaseBolt</span><span class="o">(</span><span class="s">"WordCount"</span><span class="o">,</span> <span class="n">mapper</span><span class="o">);</span>
-</code></pre></div>
-<p>The <code>HBaseBolt</code> will delegate to the <code>mapper</code> instance to figure out how to persist tuple data to HBase.</p>
-
-<h3 id="hbasevaluemapper">HBaseValueMapper</h3>
-
-<p>This class allows you to transform the HBase lookup result into storm Values that will be emitted by the <code>HBaseLookupBolt</code>.</p>
-<div class="highlight"><pre><code class="language-java" data-lang="java"><span class="kd">public</span> <span class="kd">interface</span> <span class="nc">HBaseValueMapper</span> <span class="kd">extends</span> <span class="n">Serializable</span> <span class="o">{</span>
- <span class="kd">public</span> <span class="n">List</span><span class="o"><</span><span class="n">Values</span><span class="o">></span> <span class="n">toTuples</span><span class="o">(</span><span class="n">Result</span> <span class="n">result</span><span class="o">)</span> <span class="kd">throws</span> <span class="n">Exception</span><span class="o">;</span>
- <span class="kt">void</span> <span class="n">declareOutputFields</span><span class="o">(</span><span class="n">OutputFieldsDeclarer</span> <span class="n">declarer</span><span class="o">);</span>
-<span class="o">}</span>
-</code></pre></div>
-<p>The <code>toTuples</code> method takes in a HBase <code>Result</code> instance and expects a List of <code>Values</code> instant.
-Each of the value returned by this function will be emitted by the <code>HBaseLookupBolt</code>.</p>
-
-<p>The <code>declareOutputFields</code> should be used to declare the outputFields of the <code>HBaseLookupBolt</code>.</p>
-
-<p>There is an example implementation in <code>src/test/java</code> directory.</p>
-
-<h3 id="hbaseprojectioncriteria">HBaseProjectionCriteria</h3>
-
-<p>This class allows you to specify the projection criteria for your HBase Get function. This is optional parameter
-for the lookupBolt and if you do not specify this instance all the columns will be returned by <code>HBaseLookupBolt</code>.</p>
-<div class="highlight"><pre><code class="language-java" data-lang="java"><span class="kd">public</span> <span class="kd">class</span> <span class="nc">HBaseProjectionCriteria</span> <span class="kd">implements</span> <span class="n">Serializable</span> <span class="o">{</span>
- <span class="kd">public</span> <span class="n">HBaseProjectionCriteria</span> <span class="n">addColumnFamily</span><span class="o">(</span><span class="n">String</span> <span class="n">columnFamily</span><span class="o">);</span>
- <span class="kd">public</span> <span class="n">HBaseProjectionCriteria</span> <span class="n">addColumn</span><span class="o">(</span><span class="n">ColumnMetaData</span> <span class="n">column</span><span class="o">);</span>
-</code></pre></div>
-<p><code>addColumnFamily</code> takes in columnFamily. Setting this parameter means all columns for this family will be included
- in the projection.</p>
-
-<p><code>addColumn</code> takes in a columnMetaData instance. Setting this parameter means only this column from the column familty
- will be part of your projection.
-The following code creates a projectionCriteria which specifies a projection criteria that:</p>
-
-<ol>
-<li>includes count column from column family cf.</li>
-<li>includes all columns from column family cf2.</li>
-</ol>
-<div class="highlight"><pre><code class="language-java" data-lang="java"><span class="n">HBaseProjectionCriteria</span> <span class="n">projectionCriteria</span> <span class="o">=</span> <span class="k">new</span> <span class="n">HBaseProjectionCriteria</span><span class="o">()</span>
- <span class="o">.</span><span class="na">addColumn</span><span class="o">(</span><span class="k">new</span> <span class="n">HBaseProjectionCriteria</span><span class="o">.</span><span class="na">ColumnMetaData</span><span class="o">(</span><span class="s">"cf"</span><span class="o">,</span> <span class="s">"count"</span><span class="o">))</span>
- <span class="o">.</span><span class="na">addColumnFamily</span><span class="o">(</span><span class="s">"cf2"</span><span class="o">);</span>
-</code></pre></div>
-<h3 id="hbaselookupbolt">HBaseLookupBolt</h3>
-
-<p>To use the <code>HBaseLookupBolt</code>, Construct it with the name of the table to write to, an implementation of <code>HBaseMapper</code>
-and an implementation of <code>HBaseRowToStormValueMapper</code>. You can optionally specify a <code>HBaseProjectionCriteria</code>. </p>
-
-<p>The <code>HBaseLookupBolt</code> will use the mapper to get rowKey to lookup for. It will use the <code>HBaseProjectionCriteria</code> to
-figure out which columns to include in the result and it will leverage the <code>HBaseRowToStormValueMapper</code> to get the
-values to be emitted by the bolt.</p>
-
-<p>You can look at an example topology LookupWordCount.java under <code>src/test/java</code>.</p>
-
-<h2 id="example-persistent-word-count">Example: Persistent Word Count</h2>
-
-<p>A runnable example can be found in the <code>src/test/java</code> directory.</p>
-
-<h3 id="setup">Setup</h3>
-
-<p>The following steps assume you are running HBase locally, or there is an <code>hbase-site.xml</code> on the
-classpath pointing to your HBase cluster.</p>
-
-<p>Use the <code>hbase shell</code> command to create the schema:</p>
-<div class="highlight"><pre><code class="language-" data-lang="">> create 'WordCount', 'cf'
-</code></pre></div>
-<h3 id="execution">Execution</h3>
-
-<p>Run the <code>org.apache.storm.hbase.topology.PersistenWordCount</code> class (it will run the topology for 10 seconds, then exit).</p>
-
-<p>After (or while) the word count topology is running, run the <code>org.apache.storm.hbase.topology.WordCountClient</code> class
-to view the counter values stored in HBase. You should see something like to following:</p>
-<div class="highlight"><pre><code class="language-" data-lang="">Word: 'apple', Count: 6867
-Word: 'orange', Count: 6645
-Word: 'pineapple', Count: 6954
-Word: 'banana', Count: 6787
-Word: 'watermelon', Count: 6806
-</code></pre></div>
-<p>For reference, the sample topology is listed below:</p>
-<div class="highlight"><pre><code class="language-java" data-lang="java"><span class="kd">public</span> <span class="kd">class</span> <span class="nc">PersistentWordCount</span> <span class="o">{</span>
- <span class="kd">private</span> <span class="kd">static</span> <span class="kd">final</span> <span class="n">String</span> <span class="n">WORD_SPOUT</span> <span class="o">=</span> <span class="s">"WORD_SPOUT"</span><span class="o">;</span>
- <span class="kd">private</span> <span class="kd">static</span> <span class="kd">final</span> <span class="n">String</span> <span class="n">COUNT_BOLT</span> <span class="o">=</span> <span class="s">"COUNT_BOLT"</span><span class="o">;</span>
- <span class="kd">private</span> <span class="kd">static</span> <span class="kd">final</span> <span class="n">String</span> <span class="n">HBASE_BOLT</span> <span class="o">=</span> <span class="s">"HBASE_BOLT"</span><span class="o">;</span>
-
-
- <span class="kd">public</span> <span class="kd">static</span> <span class="kt">void</span> <span class="n">main</span><span class="o">(</span><span class="n">String</span><span class="o">[]</span> <span class="n">args</span><span class="o">)</span> <span class="kd">throws</span> <span class="n">Exception</span> <span class="o">{</span>
- <span class="n">Config</span> <span class="n">config</span> <span class="o">=</span> <span class="k">new</span> <span class="n">Config</span><span class="o">();</span>
-
- <span class="n">WordSpout</span> <span class="n">spout</span> <span class="o">=</span> <span class="k">new</span> <span class="n">WordSpout</span><span class="o">();</span>
- <span class="n">WordCounter</span> <span class="n">bolt</span> <span class="o">=</span> <span class="k">new</span> <span class="n">WordCounter</span><span class="o">();</span>
-
- <span class="n">SimpleHBaseMapper</span> <span class="n">mapper</span> <span class="o">=</span> <span class="k">new</span> <span class="n">SimpleHBaseMapper</span><span class="o">()</span>
- <span class="o">.</span><span class="na">withRowKeyField</span><span class="o">(</span><span class="s">"word"</span><span class="o">)</span>
- <span class="o">.</span><span class="na">withColumnFields</span><span class="o">(</span><span class="k">new</span> <span class="n">Fields</span><span class="o">(</span><span class="s">"word"</span><span class="o">))</span>
- <span class="o">.</span><span class="na">withCounterFields</span><span class="o">(</span><span class="k">new</span> <span class="n">Fields</span><span class="o">(</span><span class="s">"count"</span><span class="o">))</span>
- <span class="o">.</span><span class="na">withColumnFamily</span><span class="o">(</span><span class="s">"cf"</span><span class="o">);</span>
-
- <span class="n">HBaseBolt</span> <span class="n">hbase</span> <span class="o">=</span> <span class="k">new</span> <span class="n">HBaseBolt</span><span class="o">(</span><span class="s">"WordCount"</span><span class="o">,</span> <span class="n">mapper</span><span class="o">);</span>
-
-
- <span class="c1">// wordSpout ==> countBolt ==> HBaseBolt</span>
- <span class="n">TopologyBuilder</span> <span class="n">builder</span> <span class="o">=</span> <span class="k">new</span> <span class="n">TopologyBuilder</span><span class="o">();</span>
-
- <span class="n">builder</span><span class="o">.</span><span class="na">setSpout</span><span class="o">(</span><span class="n">WORD_SPOUT</span><span class="o">,</span> <span class="n">spout</span><span class="o">,</span> <span class="mi">1</span><span class="o">);</span>
- <span class="n">builder</span><span class="o">.</span><span class="na">setBolt</span><span class="o">(</span><span class="n">COUNT_BOLT</span><span class="o">,</span> <span class="n">bolt</span><span class="o">,</span> <span class="mi">1</span><span class="o">).</span><span class="na">shuffleGrouping</span><span class="o">(</span><span class="n">WORD_SPOUT</span><span class="o">);</span>
- <span class="n">builder</span><span class="o">.</span><span class="na">setBolt</span><span class="o">(</span><span class="n">HBASE_BOLT</span><span class="o">,</span> <span class="n">hbase</span><span class="o">,</span> <span class="mi">1</span><span class="o">).</span><span class="na">fieldsGrouping</span><span class="o">(</span><span class="n">COUNT_BOLT</span><span class="o">,</span> <span class="k">new</span> <span class="n">Fields</span><span class="o">(</span><span class="s">"word"</span><span class="o">));</span>
-
-
- <span class="k">if</span> <span class="o">(</span><span class="n">args</span><span class="o">.</span><span class="na">length</span> <span class="o">==</span> <span class="mi">0</span><span class="o">)</span> <span class="o">{</span>
- <span class="n">LocalCluster</span> <span class="n">cluster</span> <span class="o">=</span> <span class="k">new</span> <span class="n">LocalCluster</span><span class="o">();</span>
- <span class="n">cluster</span><span class="o">.</span><span class="na">submitTopology</span><span class="o">(</span><span class="s">"test"</span><span class="o">,</span> <span class="n">config</span><span class="o">,</span> <span class="n">builder</span><span class="o">.</span><span class="na">createTopology</span><span class="o">());</span>
- <span class="n">Thread</span><span class="o">.</span><span class="na">sleep</span><span class="o">(</span><span class="mi">10000</span><span class="o">);</span>
- <span class="n">cluster</span><span class="o">.</span><span class="na">killTopology</span><span class="o">(</span><span class="s">"test"</span><span class="o">);</span>
- <span class="n">cluster</span><span class="o">.</span><span class="na">shutdown</span><span class="o">();</span>
- <span class="n">System</span><span class="o">.</span><span class="na">exit</span><span class="o">(</span><span class="mi">0</span><span class="o">);</span>
- <span class="o">}</span> <span class="k">else</span> <span class="o">{</span>
- <span class="n">config</span><span class="o">.</span><span class="na">setNumWorkers</span><span class="o">(</span><span class="mi">3</span><span class="o">);</span>
- <span class="n">StormSubmitter</span><span class="o">.</span><span class="na">submitTopology</span><span class="o">(</span><span class="n">args</span><span class="o">[</span><span class="mi">0</span><span class="o">],</span> <span class="n">config</span><span class="o">,</span> <span class="n">builder</span><span class="o">.</span><span class="na">createTopology</span><span class="o">());</span>
- <span class="o">}</span>
- <span class="o">}</span>
-<span class="o">}</span>
-</code></pre></div>
-
-
- </div>
- </div>
- </div>
-<footer>
- <div class="container-fluid">
- <div class="row">
- <div class="col-md-3">
- <div class="footer-widget">
- <h5>Meetups</h5>
- <ul class="latest-news">
-
- <li><a href="http://www.meetup.com/Apache-Storm-Apache-Kafka/">Apache Storm & Apache Kafka</a> <span class="small">(Sunnyvale, CA)</span></li>
-
- <li><a href="http://www.meetup.com/Apache-Storm-Kafka-Users/">Apache Storm & Kafka Users</a> <span class="small">(Seattle, WA)</span></li>
-
- <li><a href="http://www.meetup.com/New-York-City-Storm-User-Group/">NYC Storm User Group</a> <span class="small">(New York, NY)</span></li>
-
- <li><a href="http://www.meetup.com/Bay-Area-Stream-Processing">Bay Area Stream Processing</a> <span class="small">(Emeryville, CA)</span></li>
-
- <li><a href="http://www.meetup.com/Boston-Storm-Users/">Boston Realtime Data</a> <span class="small">(Boston, MA)</span></li>
-
- <li><a href="http://www.meetup.com/storm-london">London Storm User Group</a> <span class="small">(London, UK)</span></li>
-
- <!-- <li><a href="http://www.meetup.com/Apache-Storm-Kafka-Users/">Seatle, WA</a> <span class="small">(27 Jun 2015)</span></li> -->
- </ul>
- </div>
- </div>
- <div class="col-md-3">
- <div class="footer-widget">
- <h5>About Storm</h5>
- <p>Storm integrates with any queueing system and any database system. Storm's spout abstraction makes it easy to integrate a new queuing system. Likewise, integrating Storm with database systems is easy.</p>
- </div>
- </div>
- <div class="col-md-3">
- <div class="footer-widget">
- <h5>First Look</h5>
- <ul class="footer-list">
- <li><a href="/documentation/Rationale.html">Rationale</a></li>
- <li><a href="/tutorial.html">Tutorial</a></li>
- <li><a href="/documentation/Setting-up-development-environment.html">Setting up development environment</a></li>
- <li><a href="/documentation/Creating-a-new-Storm-project.html">Creating a new Storm project</a></li>
- </ul>
- </div>
- </div>
- <div class="col-md-3">
- <div class="footer-widget">
- <h5>Documentation</h5>
- <ul class="footer-list">
- <li><a href="/doc-index.html">Index</a></li>
- <li><a href="/documentation.html">Manual</a></li>
- <li><a href="https://storm.apache.org/javadoc/apidocs/index.html">Javadoc</a></li>
- <li><a href="/documentation/FAQ.html">FAQ</a></li>
- </ul>
- </div>
- </div>
- </div>
- <hr/>
- <div class="row">
- <div class="col-md-12">
- <p align="center">Copyright © 2015 <a href="http://www.apache.org">Apache Software Foundation</a>. All Rights Reserved.
- <br>Apache Storm, Apache, the Apache feather logo, and the Apache Storm project logos are trademarks of The Apache Software Foundation.
- <br>All other marks mentioned may be trademarks or registered trademarks of their respective owners.</p>
- </div>
- </div>
- </div>
-</footer>
-<!--Footer End-->
-<!-- Scroll to top -->
-<span class="totop"><a href="#"><i class="fa fa-angle-up"></i></a></span>
-
-</body>
-
-</html>
-