You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@distributedlog.apache.org by si...@apache.org on 2016/09/13 08:59:58 UTC
[19/64] [partial] incubator-distributedlog git commit: delete the
content from old site
http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/1bd00e9a/content/docs/latest/start/quickstart.html
----------------------------------------------------------------------
diff --git a/content/docs/latest/start/quickstart.html b/content/docs/latest/start/quickstart.html
new file mode 100644
index 0000000..18b20b4
--- /dev/null
+++ b/content/docs/latest/start/quickstart.html
@@ -0,0 +1,459 @@
+<!DOCTYPE html>
+<html lang="en">
+
+ <head>
+ <meta charset="utf-8">
+ <meta http-equiv="X-UA-Compatible" content="IE=edge">
+ <meta name="viewport" content="width=device-width, initial-scale=1">
+
+ <title>Setup & Run Example</title>
+ <meta name="description" content="Apache DistributedLog is an high performance replicated log.
+">
+
+ <link rel="stylesheet" href="http://distributedlog.incubator.apache.org/docs/latest/styles/site.css">
+ <link rel="stylesheet" href="http://distributedlog.incubator.apache.org/docs/latest/css/theme.css">
+ <!-- JQuery -->
+ <script src="https://ajax.googleapis.com/ajax/libs/jquery/2.2.0/jquery.min.js"></script>
+ <script src="http://distributedlog.incubator.apache.org/docs/latest/js/bootstrap.min.js"></script>
+ <link rel="canonical" href="http://distributedlog.incubator.apache.org/docs/latest/start/quickstart.html" data-proofer-ignore>
+ <link rel="alternate" type="application/rss+xml" title="Apache DistributedLog (incubating)" href="http://distributedlog.incubator.apache.org/docs/latest/feed.xml">
+ <!-- Font Awesome -->
+ <script src="//cdnjs.cloudflare.com/ajax/libs/anchor-js/3.2.0/anchor.min.js"></script>
+ <!-- Google Analytics -->
+ <script>
+ (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
+ (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
+ m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
+ })(window,document,'script','https://www.google-analytics.com/analytics.js','ga');
+
+ ga('create', 'UA-83870961-1', 'auto');
+ ga('send', 'pageview');
+ </script>
+ <!-- End Google Analytics -->
+ <link rel="shortcut icon" type="image/x-icon" href="/images/favicon.ico">
+</head>
+
+
+ <body role="document">
+
+
+<nav class="navbar navbar-default navbar-fixed-top">
+ <div class="container">
+ <div class="navbar-header">
+ <a href="/" class="navbar-brand" >
+ <img alt="Brand" style="height: 28px" src="http://distributedlog.incubator.apache.org/docs/latest/images/distributedlog_logo_navbar.png">
+ </a>
+ <button type="button" class="navbar-toggle collapsed" data-toggle="collapse" data-target="#navbar" aria-expanded="false" aria-controls="navbar">
+ <span class="sr-only">Toggle navigation</span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ </button>
+ </div>
+ <div id="navbar" class="navbar-collapse collapse">
+ <ul class="nav navbar-nav">
+ <!-- Overview -->
+ <li><a href="http://distributedlog.incubator.apache.org/docs/latest/">V0.4.0</a></li>
+ <!-- Concepts -->
+ <li><a href="http://distributedlog.incubator.apache.org/docs/latest/basics/introduction">Concepts</a></li>
+ <!-- Quick Start -->
+ <li>
+ <a href="http://distributedlog.incubator.apache.org/docs/latest/start" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-expanded="false">Start<span class="caret"></span></a>
+ <ul class="dropdown-menu" role="menu">
+
+
+ <li>
+ <a href="http://distributedlog.incubator.apache.org/docs/latest/start/building.html">
+ Build DistributedLog from Source
+ </a>
+ </li>
+
+ <li>
+ <a href="http://distributedlog.incubator.apache.org/docs/latest/start/download.html">
+ Download Releases
+ </a>
+ </li>
+
+ <li role="separator" class="divider"></li>
+ <li class="dropdown-header"><strong>Quickstart</strong></li>
+
+
+ <li>
+ <a href="http://distributedlog.incubator.apache.org/docs/latest/start/quickstart.html">
+ Setup & Run Example
+ </a>
+ </li>
+
+ <li>
+ <a href="http://distributedlog.incubator.apache.org/docs/latest/tutorials/basic-1.html">
+ API - Write Records (via core library)
+ </a>
+ </li>
+
+ <li>
+ <a href="http://distributedlog.incubator.apache.org/docs/latest/tutorials/basic-2.html">
+ API - Write Records (via write proxy)
+ </a>
+ </li>
+
+ <li>
+ <a href="http://distributedlog.incubator.apache.org/docs/latest/tutorials/basic-5.html">
+ API - Read Records
+ </a>
+ </li>
+
+ <li role="separator" class="divider"></li>
+ <li class="dropdown-header"><strong>Deployment</strong></li>
+
+
+ <li>
+ <a href="http://distributedlog.incubator.apache.org/docs/latest/deployment/cluster.html">
+ Cluster Setup
+ </a>
+ </li>
+
+ <li>
+ <a href="http://distributedlog.incubator.apache.org/docs/latest/deployment/docker.html">
+ Docker
+ </a>
+ </li>
+
+ </ul>
+ </li>
+ <!-- User Guide -->
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">User Guide<span class="caret"></span></a>
+ <ul class="dropdown-menu">
+
+
+ <li>
+ <a href="http://distributedlog.incubator.apache.org/docs/latest/basics/introduction.html">
+ Introduction
+ </a>
+ </li>
+
+ <li>
+ <a href="http://distributedlog.incubator.apache.org/docs/latest/user_guide/considerations/main.html">
+ Considerations
+ </a>
+ </li>
+
+ <li>
+ <a href="http://distributedlog.incubator.apache.org/docs/latest/user_guide/architecture/main.html">
+ Architecture
+ </a>
+ </li>
+
+ <li>
+ <a href="http://distributedlog.incubator.apache.org/docs/latest/user_guide/api/main.html">
+ API
+ </a>
+ </li>
+
+ <li>
+ <a href="http://distributedlog.incubator.apache.org/docs/latest/user_guide/configuration/main.html">
+ Configuration
+ </a>
+ </li>
+
+ <li>
+ <a href="http://distributedlog.incubator.apache.org/docs/latest/user_guide/design/main.html">
+ Detail Design
+ </a>
+ </li>
+
+ <li>
+ <a href="http://distributedlog.incubator.apache.org/docs/latest/user_guide/globalreplicatedlog/main.html">
+ Global Replicated Log
+ </a>
+ </li>
+
+ <li>
+ <a href="http://distributedlog.incubator.apache.org/docs/latest/user_guide/implementation/main.html">
+ Implementation
+ </a>
+ </li>
+
+ <li>
+ <a href="http://distributedlog.incubator.apache.org/docs/latest/user_guide/references/main.html">
+ References
+ </a>
+ </li>
+
+ </ul>
+ </li>
+ <!-- Admin Guide -->
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Admin Guide<span class="caret"></span></a>
+ <ul class="dropdown-menu">
+ <li><a href="http://distributedlog.incubator.apache.org/docs/latest/deployment/cluster">Cluster Setup</a></li>
+
+
+ <li>
+ <a href="http://distributedlog.incubator.apache.org/docs/latest/admin_guide/operations.html">
+ Operations
+ </a>
+ </li>
+
+ <li>
+ <a href="http://distributedlog.incubator.apache.org/docs/latest/admin_guide/performance.html">
+ Performance Tuning
+ </a>
+ </li>
+
+ <li>
+ <a href="http://distributedlog.incubator.apache.org/docs/latest/admin_guide/hardware.html">
+ Hardware
+ </a>
+ </li>
+
+ <li>
+ <a href="http://distributedlog.incubator.apache.org/docs/latest/admin_guide/monitoring.html">
+ Monitoring
+ </a>
+ </li>
+
+ <li>
+ <a href="http://distributedlog.incubator.apache.org/docs/latest/admin_guide/zookeeper.html">
+ ZooKeeper
+ </a>
+ </li>
+
+ <li>
+ <a href="http://distributedlog.incubator.apache.org/docs/latest/admin_guide/bookkeeper.html">
+ BookKeeper
+ </a>
+ </li>
+
+ </ul>
+ </li>
+ <!-- Tutorials -->
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Tutorials<span class="caret"></span></a>
+ <ul class="dropdown-menu">
+ <li class="dropdown-header"><strong>Basic</strong></li>
+ <li><a href="http://distributedlog.incubator.apache.org/docs/latest/tutorials/basic-1">Write Records (via Core Library)</a></li>
+ <li><a href="http://distributedlog.incubator.apache.org/docs/latest/tutorials/basic-2">Write Records (via Write Proxy)</a></li>
+ <li><a href="http://distributedlog.incubator.apache.org/docs/latest/tutorials/basic-3">Write Records to multiple streams</a></li>
+ <li><a href="http://distributedlog.incubator.apache.org/docs/latest/tutorials/basic-4">Atomic Write Records</a></li>
+ <li><a href="http://distributedlog.incubator.apache.org/docs/latest/tutorials/basic-5">Tailing Read Records</a></li>
+ <li><a href="http://distributedlog.incubator.apache.org/docs/latest/tutorials/basic-6">Rewind Read Records</a></li>
+ <li role="separator" class="divider"></li>
+ <li class="dropdown-header"><strong>Messaging</strong></li>
+
+
+ <li>
+ <a href="http://distributedlog.incubator.apache.org/docs/latest/tutorials/messaging-1.html">
+ Write records to partitioned streams
+ </a>
+ </li>
+
+ <li>
+ <a href="http://distributedlog.incubator.apache.org/docs/latest/tutorials/messaging-2.html">
+ Write records to multiple streams (load balancer)
+ </a>
+ </li>
+
+ <li>
+ <a href="http://distributedlog.incubator.apache.org/docs/latest/tutorials/messaging-3.html">
+ At-least-once Processing
+ </a>
+ </li>
+
+ <li>
+ <a href="http://distributedlog.incubator.apache.org/docs/latest/tutorials/messaging-4.html">
+ Exact-Once Processing
+ </a>
+ </li>
+
+ <li>
+ <a href="http://distributedlog.incubator.apache.org/docs/latest/tutorials/messaging-5.html">
+ Implement a kafka-like pub/sub system
+ </a>
+ </li>
+
+ <li role="separator" class="divider"></li>
+ <li class="dropdown-header"><strong>Replicated State Machines</strong></li>
+
+
+ <li>
+ <a href="http://distributedlog.incubator.apache.org/docs/latest/tutorials/replicatedstatemachines.html">
+ Build replicated state machines
+ </a>
+ </li>
+
+ <li role="separator" class="divider"></li>
+ <li class="dropdown-header"><strong>Analytics</strong></li>
+ <li><a href="http://distributedlog.incubator.apache.org/docs/latest/tutorials/analytics-mapreduce">Process log streams using MapReduce</a></li>
+ </ul>
+ </li>
+ </ul>
+ </div><!--/.nav-collapse -->
+ </div>
+</nav>
+
+
+<link rel="stylesheet" href="">
+
+
+ <div class="container" role="main">
+
+ <div class="row">
+
+ <!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+<div class="col-md-8 col-md-offset-2">
+ <div class="contents topic" id="get-a-distributedlog-cluster-up-running-locally-and-run-the-example-program-in-a-few-simple-steps">
+<p class="topic-title first">Get a DistributedLog cluster up running locally and run the example program in a few simple steps.</p>
+<ul class="simple">
+<li><a class="reference internal" href="#quick-start" id="id1">Quick Start</a><ul>
+<li><a class="reference internal" href="#step-1-download-the-binary" id="id2">Step 1: Download the binary</a></li>
+<li><a class="reference internal" href="#step-2-start-zookeeper-bookkeeper" id="id3">Step 2: Start ZooKeeper & BookKeeper</a></li>
+<li><a class="reference internal" href="#step-3-create-a-distributedlog-namespace" id="id4">Step 3: Create a DistributedLog namespace</a></li>
+<li><a class="reference internal" href="#step-4-create-some-log-streams" id="id5">Step 4: Create some log streams</a></li>
+<li><a class="reference internal" href="#step-5-start-a-write-proxy" id="id6">Step 5: Start a write proxy</a></li>
+<li><a class="reference internal" href="#step-6-tail-reading-records" id="id7">Step 6: Tail reading records</a></li>
+<li><a class="reference internal" href="#step-7-write-some-records" id="id8">Step 7: Write some records</a></li>
+</ul>
+</li>
+</ul>
+</div>
+<div class="section" id="quick-start">
+<h2><a class="toc-backref" href="#id1">Quick Start</a></h2>
+<p>This tutorial assumes you are starting from fresh and have no existing BookKeeper or ZooKeeper data.
+If you already have an existing BookKeeper or ZooKeeper cluster, you can checkout the <a class="reference external" href="../operations/deployment">deploy</a> section
+for more details on how to deploy a production cluster.</p>
+<div class="section" id="step-1-download-the-binary">
+<h3><a class="toc-backref" href="#id2">Step 1: Download the binary</a></h3>
+<p><a class="reference external" href="../download">Download</a> the stable version of <cite>DistributedLog</cite> and un-zip it.</p>
+<pre class="literal-block">
+// Download the binary `distributedlog-all-${gitsha}.zip`
+> unzip distributedlog-all-${gitsha}.zip
+</pre>
+</div>
+<div class="section" id="step-2-start-zookeeper-bookkeeper">
+<h3><a class="toc-backref" href="#id3">Step 2: Start ZooKeeper & BookKeeper</a></h3>
+<p>DistributedLog uses <cite>ZooKeeper</cite> as the metadata store and <cite>BookKeeper</cite> as the log segment store. So
+you need to first start a zookeeper server and a few bookies if you don't already have one. You can
+use the <cite>dlog</cite> script in <cite>distributedlog-service</cite> package to get a standalone bookkeeper sandbox. It
+starts a zookeeper server and <cite>N</cite> bookies (N is 3 by default).</p>
+<pre class="literal-block">
+// Start the local sandbox instance at port `7000`
+> ./distributedlog-service/bin/dlog local 7000
+DistributedLog Sandbox is running now. You could access distributedlog://127.0.0.1:7000
+</pre>
+</div>
+<div class="section" id="step-3-create-a-distributedlog-namespace">
+<h3><a class="toc-backref" href="#id4">Step 3: Create a DistributedLog namespace</a></h3>
+<p>Before using distributedlog, you need to create a distributedlog namespace to store your own list of
+streams. The zkServer for the local sandbox is <cite>127.0.0.1:7000</cite> and the bookkeeper's ledgers path is
+<cite>/ledgers</cite>. You could create a namespace pointing to the corresponding bookkeeper cluster.</p>
+<pre class="literal-block">
+> ./distributedlog-service/bin/dlog admin bind -l /ledgers -s 127.0.0.1:7000 -c distributedlog://127.0.0.1:7000/messaging/my_namespace
+No bookkeeper is bound to distributedlog://127.0.0.1:7000/messaging/my_namespace
+Created binding on distributedlog://127.0.0.1:7000/messaging/my_namespace.
+</pre>
+<p>If you don't want to create a separated namespace, you could use the default namespace <cite>distributedlog://127.0.0.1:7000/messaging/distributedlog</cite>.</p>
+</div>
+<div class="section" id="step-4-create-some-log-streams">
+<h3><a class="toc-backref" href="#id5">Step 4: Create some log streams</a></h3>
+<p>Let's create 5 log streams, prefixed with <cite>messaging-stream-</cite>.</p>
+<pre class="literal-block">
+> ./distributedlog-service/bin/dlog tool create -u distributedlog://127.0.0.1:7000/messaging/my_namespace -r messaging-stream- -e 1-5
+</pre>
+<p>We can now see the streams if we run the <cite>list</cite> command from the tool.</p>
+<pre class="literal-block">
+> ./distributedlog-service/bin/dlog tool list -u distributedlog://127.0.0.1:7000/messaging/my_namespace
+Streams under distributedlog://127.0.0.1:7000/messaging/my_namespace :
+--------------------------------
+messaging-stream-1
+messaging-stream-3
+messaging-stream-2
+messaging-stream-4
+messaging-stream-5
+--------------------------------
+</pre>
+</div>
+<div class="section" id="step-5-start-a-write-proxy">
+<h3><a class="toc-backref" href="#id6">Step 5: Start a write proxy</a></h3>
+<p>Now, lets start a write proxy server that serves writes to distributedlog namespace <cite>distributedlog://127.0.0.1/messaging/my_namespace</cite>. The server listens on 8000 to accept fan-in write requests.</p>
+<pre class="literal-block">
+> ./distributedlog-service/bin/dlog-daemon.sh start writeproxy -p 8000 --shard-id 1 -sp 8001 -u distributedlog://127.0.0.1:7000/messaging/my_namespace -mx -c `pwd`/distributedlog-service/conf/distributedlog_proxy.conf
+</pre>
+<p>From 0.3.51-RC1 and onwards, use the below command to start the write proxy</p>
+<pre class="literal-block">
+> WP_SHARD_ID=1 WP_SERVICE_PORT=8000 WP_STATS_PORT=8001 WP_NAMESPACE='distributedlog://127.0.0.1:7000/messaging/my_namespace' ./distributedlog-service/bin/dlog-daemon.sh start writeproxy
+</pre>
+</div>
+<div class="section" id="step-6-tail-reading-records">
+<h3><a class="toc-backref" href="#id7">Step 6: Tail reading records</a></h3>
+<p>The distributedlog tutorial has a multi-streams reader that will dump out received records to standard output.</p>
+<pre class="literal-block">
+> ./distributedlog-tutorials/distributedlog-basic/bin/runner run com.twitter.distributedlog.basic.MultiReader distributedlog://127.0.0.1:7000/messaging/my_namespace messaging-stream-1,messaging-stream-2,messaging-stream-3,messaging-stream-4,messaging-stream-5
+</pre>
+</div>
+<div class="section" id="step-7-write-some-records">
+<h3><a class="toc-backref" href="#id8">Step 7: Write some records</a></h3>
+<p>The distributedlog tutorial also has a multi-streams writer that will take input from a console and write it out
+as records to the distributedlog write proxy. Each line will be sent as a separate record.</p>
+<p>Run the writer and type a few lines into the console to send to the server.</p>
+<pre class="literal-block">
+> ./distributedlog-tutorials/distributedlog-basic/bin/runner run com.twitter.distributedlog.basic.ConsoleProxyMultiWriter 'inet!127.0.0.1:8000' messaging-stream-1,messaging-stream-2,messaging-stream-3,messaging-stream-4,messaging-stream-5
+</pre>
+<p>If you have each of the above commands running in a different terminal then you should now be able to type messages into the writer terminal and see them appear in the reader terminal.</p>
+</div>
+</div>
+
+
+</div>
+
+
+
+ </div>
+
+
+ <hr>
+ <div class="row">
+ <div class="col-xs-12">
+ <footer>
+ <p class="text-center">© Copyright 2016
+ <a href="http://www.apache.org">The Apache Software Foundation.</a> All Rights Reserved.
+ </p>
+ <p class="text-center">
+ <a href="http://distributedlog.incubator.apache.org/docs/latest/feed.xml">RSS Feed</a>
+ </p>
+ </footer>
+ </div>
+ </div>
+ <!-- container div end -->
+</div>
+
+
+ <script>
+ (function () {
+ 'use strict';
+ anchors.options.placement = 'right';
+ anchors.add();
+ })();
+</script>
+
+ </body>
+
+</html>