You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@zookeeper.apache.org by bu...@apache.org on 2011/12/07 15:10:33 UTC
svn commit: r799871 [1/2] -
/websites/staging/zookeeper/trunk/content/bookkeeper/docs/r4.0.0/
Author: buildbot
Date: Wed Dec 7 14:10:32 2011
New Revision: 799871
Log:
Staging update by buildbot
Added:
websites/staging/zookeeper/trunk/content/bookkeeper/docs/r4.0.0/
websites/staging/zookeeper/trunk/content/bookkeeper/docs/r4.0.0/bookieRecovery.html
websites/staging/zookeeper/trunk/content/bookkeeper/docs/r4.0.0/bookkeeperConfig.html
websites/staging/zookeeper/trunk/content/bookkeeper/docs/r4.0.0/bookkeeperConfigParams.html
websites/staging/zookeeper/trunk/content/bookkeeper/docs/r4.0.0/bookkeeperInternals.html
websites/staging/zookeeper/trunk/content/bookkeeper/docs/r4.0.0/bookkeeperOverview.html
websites/staging/zookeeper/trunk/content/bookkeeper/docs/r4.0.0/bookkeeperProgrammer.html
websites/staging/zookeeper/trunk/content/bookkeeper/docs/r4.0.0/bookkeeperStarted.html
websites/staging/zookeeper/trunk/content/bookkeeper/docs/r4.0.0/bookkeeperStream.html
websites/staging/zookeeper/trunk/content/bookkeeper/docs/r4.0.0/doc.html
websites/staging/zookeeper/trunk/content/bookkeeper/docs/r4.0.0/hedwigBuild.html
websites/staging/zookeeper/trunk/content/bookkeeper/docs/r4.0.0/hedwigDesign.html
websites/staging/zookeeper/trunk/content/bookkeeper/docs/r4.0.0/hedwigUser.html
websites/staging/zookeeper/trunk/content/bookkeeper/docs/r4.0.0/index.html
Added: websites/staging/zookeeper/trunk/content/bookkeeper/docs/r4.0.0/bookieRecovery.html
==============================================================================
--- websites/staging/zookeeper/trunk/content/bookkeeper/docs/r4.0.0/bookieRecovery.html (added)
+++ websites/staging/zookeeper/trunk/content/bookkeeper/docs/r4.0.0/bookieRecovery.html Wed Dec 7 14:10:32 2011
@@ -0,0 +1,169 @@
+<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
+<html>
+<head>
+<!--
+
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE- 2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+ <link href="/bookkeeper/css/site.css" rel="stylesheet" type="text/css">
+ <link href="/bookkeeper/css/type-settings.css" rel="stylesheet" type="text/css">
+
+ <title>Apache BookKeeper - BookKeeper Bookie Recovery</title>
+
+ <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
+</head>
+<body>
+<div class="white_box">
+<div class="header">
+ <div class="header_l">
+ <div class="header_r">
+ </div>
+ </div>
+</div>
+<div class="content">
+ <div class="content_l">
+ <div class="content_r">
+ <div>
+
+<!-- Banner -->
+
+
+ <div id="header_background">
+ <div id="bookkeeper_logo">
+ <a href="/bookkeeper"><img src="/bookkeeper/images/bookkeeper_small.gif"/></a> <b><font size=+4 color=white>Apache BookKeeper™</font></b> <a href="http://apache.org"><img src="/bookkeeper/images/feather_small.gif"/></a>
+ </div>
+ </div>
+
+ <table border="0">
+ <tbody>
+ <tr>
+ <td style="overflow: hidden;" valign="top" width="100%">
+ <div class="wiki-content">
+ <h1>Bookie Recovery</h1>
+
+<p>When a bookie crashes, any ledgers with entries on the bookie potentially become underreplicated. For this reason, we provide a recovery tool which will ensure that all ledgers which had entries on the bookie are fully replicated. At the moment, this is not an automatic process. The administrator must run this tool manually when he sees that the bookie has died. </p>
+
+<p>To run recovery, with zk1.example.com as the zookeeper ensemble, and bk3.example.com as the failed bookie, do the following:</p>
+
+<p><code>bookkeeper-server/bin/bookkeeper org.apache.bookkeeper.tools.BookKeeperTools zk1.example.com:2181 bk3.example.com:3181</code></p>
+
+<p>It is necessary to specify the host and port portion of failed bookie, as this is how it identifies itself to zookeeper. It is possible to specify a third argument, which is the bookie to replicate to. If this is omitted, as in our example, a random bookie is chosen for each ledger fragment. A ledger fragment is a continous sequence of entries in a bookie, which share the same ensemble. </p>
+
+<p>The recovery process is as follows.</p>
+
+<ol>
+<li>The client reads the metadata of active ledgers from zookeeper;</li>
+<li>From this, the ledgers which contain fragments using the failed bookie in their ensemble are selected;</li>
+<li>A recovery process is initiated for each ledger in this list;<ol>
+<li>The client goes through all ledger fragments in the ledger, selecting those which contain the failed bookie;</li>
+<li>A recovery process is initiated for each ledger fragment in this list;<ol>
+<li>The client selects a bookie to which all entries in the ledger fragment will be replicated;</li>
+<li>the client reads entries that belong to the ledger fragment from other bookies in the ensemble and writes them to the selected bookie;</li>
+<li>Once all entries have been replicated, the zookeeper metadata for the fragment is updated to reflect the new ensemble;</li>
+<li>The fragment is marked as fully replicated in the recovery tool;</li>
+</ol>
+</li>
+<li>Once all ledger fragements are marked as fully replicated, the ledger is marked as fully replicated;</li>
+</ol>
+</li>
+<li>Once all ledgers are marked as fully replicated, bookie recovery is finished.</li>
+</ol>
+ </div>
+ </td>
+ <td valign="top">
+ <div class="navigation">
+ <div class="navigation_top">
+<div class="searchbox">
+<form action="http://search-hadoop.com/zookeeper" method="get">
+<input onFocus="getBlank (this, 'Search with Apache Solr');" size="25" name="q" id="query" type="text" value="Search with Apache Solr">
+ <input value="Search" type="submit">
+</form>
+</div>
+
+ <div class="navigation_bottom">
+<!-- sidenav -->
+<h3>Project</h3>
+
+<ul>
+<li><a href="/bookkeeper/releases.html#news">News</a></li>
+<li><a href="https://cwiki.apache.org/confluence/display/BOOKKEEPER/Index">Wiki</a></li>
+<li><a href="/bookkeeper/credits.html">Credits</a></li>
+<li><a href="/bylaws.html">Bylaws</a></li>
+<li><a href="http://www.apache.org/licenses/">License</a></li>
+<li><a href="/privacy.html">Privacy Policy</a></li>
+<li><a href="http://www.apache.org/foundation/sponsorship.html">Sponsorship</a></li>
+<li><a href="http://www.apache.org/foundation/thanks.html">Thanks</a></li>
+<li><a href="http://www.apache.org/security/">Security</a></li>
+</ul>
+
+
+<h3>Documentation</h3>
+
+<ul>
+<li><a href="/bookkeeper/docs/trunk">Trunk</a></li>
+</ul>
+
+<h3>Developers</h3>
+
+<ul>
+<li><a href="/bookkeeper/lists.html">Mailing Lists</a></li>
+<li><a href="/bookkeeper/irc.html"><span class="caps">IRC</span> Channel</a></li>
+<li><a href="/bookkeeper/svn.html">Version Control</a></li>
+<li><a href="https://issues.apache.org/jira/browse/BOOKKEEPER">Issue Tracker</a></li>
+</ul>
+<!-- sidenav -->
+ </div>
+ </div>
+ </div>
+ </td>
+ </tr>
+ </tbody>
+ </table>
+
+<!--
+ <div class="bottom_red_bar"></div>
+-->
+ </div>
+ </div>
+ </div>
+</div>
+<div class="black_box">
+<div class="footer">
+ <div class="footer_l">
+ <div class="footer_r">
+ <div>
+
+ </div>
+ </div>
+ </div>
+</div>
+</div>
+</div>
+<!--
+<div class="design_attribution">Page Template Design By Marc Prud'hommeaux based on <a href="http://activemq.apache.org/">ActiveMQ template</a></div>
+-->
+<div class="copyright_footer">
+<p>Copyright © 2010 The Apache Software Foundation, Licensed under the <a href="http://www.apache.org/licenses/LICENSE-2.0">Apache License, Version 2.0</a>.<br>Apache ZooKeeper, ZooKeeper, Apache, the Apache feather logo, and the Apache ZooKeeper project logo are trademarks of The Apache Software Foundation.</p>
+</div>
+<script src="skeleton_files/urchin.js" type="text/javascript">
+</script>
+<script type="text/javascript">
+_uacct = "UA-1940143-1";
+urchinTracker();
+</script>
+
+</body></html>
Added: websites/staging/zookeeper/trunk/content/bookkeeper/docs/r4.0.0/bookkeeperConfig.html
==============================================================================
--- websites/staging/zookeeper/trunk/content/bookkeeper/docs/r4.0.0/bookkeeperConfig.html (added)
+++ websites/staging/zookeeper/trunk/content/bookkeeper/docs/r4.0.0/bookkeeperConfig.html Wed Dec 7 14:10:32 2011
@@ -0,0 +1,183 @@
+<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
+<html>
+<head>
+<!--
+
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE- 2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+ <link href="/bookkeeper/css/site.css" rel="stylesheet" type="text/css">
+ <link href="/bookkeeper/css/type-settings.css" rel="stylesheet" type="text/css">
+
+ <title>Apache BookKeeper - BookKeeper Administrator's Guide</title>
+
+ <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
+</head>
+<body>
+<div class="white_box">
+<div class="header">
+ <div class="header_l">
+ <div class="header_r">
+ </div>
+ </div>
+</div>
+<div class="content">
+ <div class="content_l">
+ <div class="content_r">
+ <div>
+
+<!-- Banner -->
+
+
+ <div id="header_background">
+ <div id="bookkeeper_logo">
+ <a href="/bookkeeper"><img src="/bookkeeper/images/bookkeeper_small.gif"/></a> <b><font size=+4 color=white>Apache BookKeeper™</font></b> <a href="http://apache.org"><img src="/bookkeeper/images/feather_small.gif"/></a>
+ </div>
+ </div>
+
+ <table border="0">
+ <tbody>
+ <tr>
+ <td style="overflow: hidden;" valign="top" width="100%">
+ <div class="wiki-content">
+ <h1>Abstract</h1>
+
+<p>This document contains information about deploying, administering and mantaining BookKeeper. It also discusses best practices and common problems. </p>
+
+<h1>Running a BookKeeper instance</h1>
+
+<h2>System requirements</h2>
+
+<p>A typical BookKeeper installation comprises a set of bookies and a set of ZooKeeper replicas. The exact number of bookies depends on the quorum mode, desired throughput, and number of clients using this installation simultaneously. The minimum number of bookies is three for self-verifying (stores a message authentication code along with each entry) and four for generic (does not store a message authentication code with each entry), and there is no upper limit on the number of bookies. Increasing the number of bookies will, in fact, enable higher throughput. </p>
+
+<p>For performance, we require each server to have at least two disks. It is possible to run a bookie with a single disk, but performance will be significantly lower in this case.</p>
+
+<p>For ZooKeeper, there is no constraint with respect to the number of replicas. Having a single machine running ZooKeeper in standalone mode is sufficient for BookKeeper. For resilience purposes, it might be a good idea to run ZooKeeper in quorum mode with multiple servers. Please refer to the ZooKeeper documentation for detail on how to configure ZooKeeper with multiple replicas. </p>
+
+<h2>Running bookies</h2>
+
+<p>To run a bookie, we execute the following command: </p>
+
+<p><code>bookkeeper-server/bin/bookkeeper bookie</code></p>
+
+<p>The configuration parameters can be set in bookkeeper-server/conf/bk_server.conf. </p>
+
+<p>The important parameters are:<br />
+ * <code>bookiePort</code>, Port number that the bookie listens on; <br />
+ * <code>zkServers</code>, Comma separated list of ZooKeeper servers with a hostname:port format; <br />
+ * <code>journalDir</code>, Path for Log Device (stores bookie write-ahead log); <br />
+ * <code>ledgerDir</code>, Path for Ledger Device (stores ledger entries); </p>
+
+<p>Ideally, <code>journalDir</code> and <code>ledgerDir</code> are each in a different device. See <a href="./bookkeeperConfigParams.html">BookKeeper Configuration Parameters</a> for a full list of configuration parameters.</p>
+
+
+<h3>Logging</h3>
+
+<p>BookKeeper uses <a href="http://www.slf4j.org">slf4j</a> for logging, with the log4j bindings enabled by default. To enable logging from a bookie, create a log4j.properties file and point the environment variable <span class="caps">BOOKIE</span>_LOG_CONF to the configuration file. The path to the log4j.properties file must be absolute.</p>
+
+<p><code>export BOOKIE_LOG_CONF=/tmp/log4j.properties</code><br />
+<code>bookkeeper-server/bin/bookkeeper bookie</code></p>
+
+<h2>Setting up a test ensemble</h2>
+
+<p>Sometimes it is useful to run a ensemble of bookies on your local machine for testing. We provide a utility for doing this. It will set up N bookies, and a zookeeper instance locally. The data on these bookies and of the zookeeper instance are not persisted over restarts, so obviously this should never be used in a production environment. To run a test ensemble of 10 bookies, do the following.</p>
+
+<p><code>bookkeeper-server/bin/bookkeeper localbookie 10</code></p>
+ </div>
+ </td>
+ <td valign="top">
+ <div class="navigation">
+ <div class="navigation_top">
+<div class="searchbox">
+<form action="http://search-hadoop.com/zookeeper" method="get">
+<input onFocus="getBlank (this, 'Search with Apache Solr');" size="25" name="q" id="query" type="text" value="Search with Apache Solr">
+ <input value="Search" type="submit">
+</form>
+</div>
+
+ <div class="navigation_bottom">
+<!-- sidenav -->
+<h3>Project</h3>
+
+<ul>
+<li><a href="/bookkeeper/releases.html#news">News</a></li>
+<li><a href="https://cwiki.apache.org/confluence/display/BOOKKEEPER/Index">Wiki</a></li>
+<li><a href="/bookkeeper/credits.html">Credits</a></li>
+<li><a href="/bylaws.html">Bylaws</a></li>
+<li><a href="http://www.apache.org/licenses/">License</a></li>
+<li><a href="/privacy.html">Privacy Policy</a></li>
+<li><a href="http://www.apache.org/foundation/sponsorship.html">Sponsorship</a></li>
+<li><a href="http://www.apache.org/foundation/thanks.html">Thanks</a></li>
+<li><a href="http://www.apache.org/security/">Security</a></li>
+</ul>
+
+
+<h3>Documentation</h3>
+
+<ul>
+<li><a href="/bookkeeper/docs/trunk">Trunk</a></li>
+</ul>
+
+<h3>Developers</h3>
+
+<ul>
+<li><a href="/bookkeeper/lists.html">Mailing Lists</a></li>
+<li><a href="/bookkeeper/irc.html"><span class="caps">IRC</span> Channel</a></li>
+<li><a href="/bookkeeper/svn.html">Version Control</a></li>
+<li><a href="https://issues.apache.org/jira/browse/BOOKKEEPER">Issue Tracker</a></li>
+</ul>
+<!-- sidenav -->
+ </div>
+ </div>
+ </div>
+ </td>
+ </tr>
+ </tbody>
+ </table>
+
+<!--
+ <div class="bottom_red_bar"></div>
+-->
+ </div>
+ </div>
+ </div>
+</div>
+<div class="black_box">
+<div class="footer">
+ <div class="footer_l">
+ <div class="footer_r">
+ <div>
+
+ </div>
+ </div>
+ </div>
+</div>
+</div>
+</div>
+<!--
+<div class="design_attribution">Page Template Design By Marc Prud'hommeaux based on <a href="http://activemq.apache.org/">ActiveMQ template</a></div>
+-->
+<div class="copyright_footer">
+<p>Copyright © 2010 The Apache Software Foundation, Licensed under the <a href="http://www.apache.org/licenses/LICENSE-2.0">Apache License, Version 2.0</a>.<br>Apache ZooKeeper, ZooKeeper, Apache, the Apache feather logo, and the Apache ZooKeeper project logo are trademarks of The Apache Software Foundation.</p>
+</div>
+<script src="skeleton_files/urchin.js" type="text/javascript">
+</script>
+<script type="text/javascript">
+_uacct = "UA-1940143-1";
+urchinTracker();
+</script>
+
+</body></html>
Added: websites/staging/zookeeper/trunk/content/bookkeeper/docs/r4.0.0/bookkeeperConfigParams.html
==============================================================================
--- websites/staging/zookeeper/trunk/content/bookkeeper/docs/r4.0.0/bookkeeperConfigParams.html (added)
+++ websites/staging/zookeeper/trunk/content/bookkeeper/docs/r4.0.0/bookkeeperConfigParams.html Wed Dec 7 14:10:32 2011
@@ -0,0 +1,159 @@
+<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
+<html>
+<head>
+<!--
+
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE- 2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+ <link href="/bookkeeper/css/site.css" rel="stylesheet" type="text/css">
+ <link href="/bookkeeper/css/type-settings.css" rel="stylesheet" type="text/css">
+
+ <title>Apache BookKeeper - BookKeeper Configuration Parameters</title>
+
+ <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
+</head>
+<body>
+<div class="white_box">
+<div class="header">
+ <div class="header_l">
+ <div class="header_r">
+ </div>
+ </div>
+</div>
+<div class="content">
+ <div class="content_l">
+ <div class="content_r">
+ <div>
+
+<!-- Banner -->
+
+
+ <div id="header_background">
+ <div id="bookkeeper_logo">
+ <a href="/bookkeeper"><img src="/bookkeeper/images/bookkeeper_small.gif"/></a> <b><font size=+4 color=white>Apache BookKeeper™</font></b> <a href="http://apache.org"><img src="/bookkeeper/images/feather_small.gif"/></a>
+ </div>
+ </div>
+
+ <table border="0">
+ <tbody>
+ <tr>
+ <td style="overflow: hidden;" valign="top" width="100%">
+ <div class="wiki-content">
+ <h1>BookKeeper Configuration Parameters</h1>
+
+<p>This page contains detailed information about configuration parameters used for configuring a BookKeeper bookie. There is an example in "bookkeeper-server/conf/bk_server.conf". </p>
+
+<h3>General parameters</h3>
+
+<table><tr><td><code>zkServers</code></td><td>A list of one of more servers on which zookeeper is running. The server list can be comma separated values, e.g., zk1:2181,zk2:2181,zk3:2181</td></tr><tr><td><code>zkTimeout</code></td><td>ZooKeeper client session timeout in milliseconds. Bookie server will exit if it received <span class="caps">SESSION</span>_EXPIRED because it was partitioned off from ZooKeeper for more than the session timeout <span class="caps">JVM </span>garbage collection, disk I/O will cause <span class="caps">SESSION</span>_EXPIRED. Increment this value could help avoiding this issue. The default value is 10,000.</td></tr><tr><td><code>bookiePort</code></td><td>Port that bookie server listens on. The default value is 3181.</td></tr><tr><td><code>journalDir</code></td><td>Directory Bookkeeper outputs its write ahead log, ideally in a dedicated device. The default value is "/tmp/bk-txn".</td></tr><tr><td><code>ledgerDirs</code></td><td>Directory Bookkeeper
outputs ledger snapshots could define multiple directories to store snapshots, comma separated. For example: /tmp/bk1-data,/tmp/bk2-data. Ideally ledger dirs and journal dir are each in a differet device, which reduce the contention between random i/o and sequential write. It is possible to run with a single disk, but performance will be significantly lower.</td></tr><tr><td><code>logSizeLimit</code></td><td>Maximum file size of entry logger, in bytes. A new entry log file will be created when the old one reaches the file size limitation. The default value is 2GB.</td></tr><tr><td><code>journalMaxSizeMB</code></td><td>Maximum file size of journal file, in mega bytes. A new journal file will be created when the old one reaches the file size limitation. The default value is 2kB.</td></tr><tr><td><code>journalMaxBackups</code></td><td>Max number of old journal file to keep. Keeping a number of old journal files might help data recovery in some special cases. The default value
is 5.</td></tr><tr><td><code>gcWaitTime</code></td><td>Interval to trigger next garbage collection, in milliseconds. Since garbage collection is running in the background, running the garbage collector too frequently hurts performance. It is best to set its value high enough if there is sufficient disk capacity.</td></tr><tr><td><code>flushInterval</code></td><td>Interval to flush ledger index pages to disk, in milliseconds. Flushing index files will introduce random disk I/O. Consequently, it is important to have journal dir and ledger dirs each on different devices. However, if it necessary to have journal dir and ledger dirs on the same device, one option is to increment the flush interval to get higher performance. Upon a failure, the bookie will take longer to recover.</td></tr><tr><td><code>bookieDeathWatchInterval</code></td><td>Interval to check whether a bookie is dead or not, in milliseconds.</td></tr></table>
+
+<h3><span class="caps">NIO </span>server settings</h3>
+
+<table><tr><td><code>serverTcpNoDelay</code></td><td>This settings is used to enabled/disabled Nagle's algorithm, which is a means of improving the efficiency of <span class="caps">TCP</span>/IP networks by reducing the number of packets that need to be sent over the network. If you are sending many small messages, such that more than one can fit in a single IP packet, setting server.tcpnodelay to false to enable Nagle algorithm can provide better performance. Default value is true.</td></tr></table>
+
+<h3>Ledger cache settings</h3>
+
+<table><tr><td><code>openFileLimit</code></td><td>Maximum number of ledger index files that can be opened in a bookie. If the number of ledger index files reaches this limit, the bookie starts to flush some ledger indexes from memory to disk. If flushing happens too frequently, then performance is affected. You can tune this number to improve performance according.</td></tr><tr><td><code>pageSize</code></td><td>Size of an index page in ledger cache, in bytes. A larger index page can improve performance when writing page to disk, which is efficient when you have small number of ledgers and these ledgers have a similar number of entries. With a large number of ledgers and a few entries per ledger, a smaller index page would improves memory usage.</td></tr><tr><td><code>pageLimit</code></td><td>Maximum number of index pages to store in the ledger cache. If the number of index pages reaches this limit, bookie server starts to flush ledger indexes from memory to disk. Incrementin
g this value is an option when flushing becomes frequent. It is important to make sure, though, that pageLimit*pageSize is not more than <span class="caps">JVM </span>max memory limit; otherwise it will raise an OutOfMemoryException. In general, incrementing pageLimit, using smaller index page would gain better performance in the case of a large number of ledgers with few entries per ledger. If pageLimit is -1, a bookie uses 1/3 of the <span class="caps">JVM </span>memory to compute the maximum number of index pages.</td></tr></table>
+
+<h3>Ledger manager settings</h3>
+
+<table><tr><td><code>ledgerManagerType</code></td><td>What kind of ledger manager is used to manage how ledgers are stored, managed and garbage collected. See <a href="./bookkeeperInternals.html">BookKeeper Internals</a> for detailed info. Default is flat.</td></tr><tr><td><code>zkLedgersRootPath</code></td><td>Root zookeeper path to store ledger metadata. Default is /ledgers.</td></tr></table>
+ </div>
+ </td>
+ <td valign="top">
+ <div class="navigation">
+ <div class="navigation_top">
+<div class="searchbox">
+<form action="http://search-hadoop.com/zookeeper" method="get">
+<input onFocus="getBlank (this, 'Search with Apache Solr');" size="25" name="q" id="query" type="text" value="Search with Apache Solr">
+ <input value="Search" type="submit">
+</form>
+</div>
+
+ <div class="navigation_bottom">
+<!-- sidenav -->
+<h3>Project</h3>
+
+<ul>
+<li><a href="/bookkeeper/releases.html#news">News</a></li>
+<li><a href="https://cwiki.apache.org/confluence/display/BOOKKEEPER/Index">Wiki</a></li>
+<li><a href="/bookkeeper/credits.html">Credits</a></li>
+<li><a href="/bylaws.html">Bylaws</a></li>
+<li><a href="http://www.apache.org/licenses/">License</a></li>
+<li><a href="/privacy.html">Privacy Policy</a></li>
+<li><a href="http://www.apache.org/foundation/sponsorship.html">Sponsorship</a></li>
+<li><a href="http://www.apache.org/foundation/thanks.html">Thanks</a></li>
+<li><a href="http://www.apache.org/security/">Security</a></li>
+</ul>
+
+
+<h3>Documentation</h3>
+
+<ul>
+<li><a href="/bookkeeper/docs/trunk">Trunk</a></li>
+</ul>
+
+<h3>Developers</h3>
+
+<ul>
+<li><a href="/bookkeeper/lists.html">Mailing Lists</a></li>
+<li><a href="/bookkeeper/irc.html"><span class="caps">IRC</span> Channel</a></li>
+<li><a href="/bookkeeper/svn.html">Version Control</a></li>
+<li><a href="https://issues.apache.org/jira/browse/BOOKKEEPER">Issue Tracker</a></li>
+</ul>
+<!-- sidenav -->
+ </div>
+ </div>
+ </div>
+ </td>
+ </tr>
+ </tbody>
+ </table>
+
+<!--
+ <div class="bottom_red_bar"></div>
+-->
+ </div>
+ </div>
+ </div>
+</div>
+<div class="black_box">
+<div class="footer">
+ <div class="footer_l">
+ <div class="footer_r">
+ <div>
+
+ </div>
+ </div>
+ </div>
+</div>
+</div>
+</div>
+<!--
+<div class="design_attribution">Page Template Design By Marc Prud'hommeaux based on <a href="http://activemq.apache.org/">ActiveMQ template</a></div>
+-->
+<div class="copyright_footer">
+<p>Copyright © 2010 The Apache Software Foundation, Licensed under the <a href="http://www.apache.org/licenses/LICENSE-2.0">Apache License, Version 2.0</a>.<br>Apache ZooKeeper, ZooKeeper, Apache, the Apache feather logo, and the Apache ZooKeeper project logo are trademarks of The Apache Software Foundation.</p>
+</div>
+<script src="skeleton_files/urchin.js" type="text/javascript">
+</script>
+<script type="text/javascript">
+_uacct = "UA-1940143-1";
+urchinTracker();
+</script>
+
+</body></html>
Added: websites/staging/zookeeper/trunk/content/bookkeeper/docs/r4.0.0/bookkeeperInternals.html
==============================================================================
--- websites/staging/zookeeper/trunk/content/bookkeeper/docs/r4.0.0/bookkeeperInternals.html (added)
+++ websites/staging/zookeeper/trunk/content/bookkeeper/docs/r4.0.0/bookkeeperInternals.html Wed Dec 7 14:10:32 2011
@@ -0,0 +1,219 @@
+<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
+<html>
+<head>
+<!--
+
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE- 2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+ <link href="/bookkeeper/css/site.css" rel="stylesheet" type="text/css">
+ <link href="/bookkeeper/css/type-settings.css" rel="stylesheet" type="text/css">
+
+ <title>Apache BookKeeper - BookKeeper Internals</title>
+
+ <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
+</head>
+<body>
+<div class="white_box">
+<div class="header">
+ <div class="header_l">
+ <div class="header_r">
+ </div>
+ </div>
+</div>
+<div class="content">
+ <div class="content_l">
+ <div class="content_r">
+ <div>
+
+<!-- Banner -->
+
+
+ <div id="header_background">
+ <div id="bookkeeper_logo">
+ <a href="/bookkeeper"><img src="/bookkeeper/images/bookkeeper_small.gif"/></a> <b><font size=+4 color=white>Apache BookKeeper™</font></b> <a href="http://apache.org"><img src="/bookkeeper/images/feather_small.gif"/></a>
+ </div>
+ </div>
+
+ <table border="0">
+ <tbody>
+ <tr>
+ <td style="overflow: hidden;" valign="top" width="100%">
+ <div class="wiki-content">
+ <h2>Bookie Internals</h2>
+
+<p>Bookie server stores its data in multiple ledger directories and its journal files in a journal directory. Ideally, storing journal files in a separate directory than data files would increase throughput and decrease latency</p>
+
+<h3>The Bookie Journal</h3>
+
+<p>Journal directory has one kind of file in it:</p>
+
+<ul>
+<li><code>{timestamp}.txn</code> - holds transactions executed in the bookie server.</li>
+</ul>
+
+<p>Before persisting ledger index and data to disk, a bookie ensures that the transaction that represents the update is written to a journal in non-volatile storage. A new journal file is created using current timestamp when a bookie starts or an old journal file reaches its maximum size.</p>
+
+<p>A bookie supports journal rolling to remove old journal files. In order to remove old journal files safely, bookie server records LastLogMark in Ledger Device, which indicates all updates (including index and data) before LastLogMark has been persisted to the Ledger Device.</p>
+
+<p>LastLogMark contains two parts:</p>
+
+<ul>
+<li><code>LastLogId</code> - indicates which journal file the transaction persisted.</li>
+<li><code>LastLogPos</code> - indicates the position the transaction persisted in LastLogId journal file.</li>
+</ul>
+
+<p>You may use following settings to further fine tune the behavior of journalling on bookies:</p>
+
+<table><tr><td><code>journalMaxSizeMB</code></td><td>journal file size limitation. when a journal reaches this limitation, it will be closed and new journal file be created.</td></tr><tr><td><code>journalMaxBackups</code></td><td>how many old journal files whose id is less than LastLogMark 's journal id.</td></tr></table>
+
+<blockquote><p><span class="caps">NOTE</span>: keeping number of old journal files would be useful for manually recovery in special case.</p></blockquote>
+
+<h1>ZooKeeper Metadata</h1>
+
+<p>For BookKeeper, we require a ZooKeeper installation to store metadata, and to pass the list of ZooKeeper servers as parameter to the constructor of the BookKeeper class (<code>org.apache.bookkeeper.client.BookKeeper</code>). To setup ZooKeeper, please check the <a href="http://zookeeper.apache.org/doc/trunk/index.html">ZooKeeper documentation</a>. </p>
+
+<p>BookKeeper provides two mechanisms to organize its metadata in ZooKeeper. By default, the <code>FlatLedgerManager</code> is used, and 99% of users should never need to look at anything else. However, in cases where there are a lot of active ledgers concurrently, (> 50,000), <code>HierarchicalLedgerManager</code> should be used. For so many ledgers, a hierarchical approach is needed due to a limit ZooKeeper places on packet sizes <a href="https://issues.apache.org/jira/browse/BOOKKEEPER-39"><span class="caps">JIRA</span> Issue</a>.</p>
+
+<table><tr><td><code>FlatLedgerManager</code></td><td>All ledger metadata are placed as children in a single zookeeper path.</td></tr><tr><td><code>HierarchicalLedgerManager</code></td><td>All ledger metadata are partitioned into 2-level znodes.</td></tr></table>
+
+<h2>Flat Ledger Manager</h2>
+
+<p>All ledgers' metadata are put in a single zookeeper path, created using zookeeper sequential node, which can ensure uniqueness of ledger id. Each ledger node is prefixed with 'L'.</p>
+
+<p>Bookie server manages its owned active ledgers in a hash map. So it is easy for bookie server to find what ledgers are deleted from zookeeper and garbage collect them. And its garbage collection flow is described as below:</p>
+
+<ul>
+<li>Fetch all existing ledgers from zookeeper (<code>zkActiveLedgers</code>).</li>
+<li>Fetch all ledgers currently active within the Bookie (<code>bkActiveLedgers</code>).</li>
+<li>Loop over <code>bkActiveLedgers</code> to find those ledgers which do not exist in <code>zkActiveLedgers</code> and garbage collect them.</li>
+</ul>
+
+<h2>Hierarchical Ledger Manager</h2>
+
+<p><code>HierarchicalLedgerManager</code> first obtains a global unique id from ZooKeeper using a <span class="caps">EPHEMERAL</span>_SEQUENTIAL znode.</p>
+
+<p>Since ZooKeeper sequential counter has a format of %10d -- that is 10 digits with 0 (zero) padding, i.e. "<path>0000000001", <code>HierarchicalLedgerManager</code> splits the generated id into 3 parts :</p>
+
+<p><code>{level1 (2 digits)}{level2 (4 digits)}{level3 (4 digits)}</code></p>
+
+<p>These 3 parts are used to form the actual ledger node path used to store ledger metadata:</p>
+
+<p><code>{ledgers_root_path}/{level1}/{level2}/L{level3}</code></p>
+
+<p>E.g. Ledger 0000000001 is split into 3 parts 00, 0000, 00001, which is stored in znode /{ledgers_root_path}/00/0000/L0001. So each znode could have at most 10000 ledgers, which avoids the problem of the child list being larger than the maximum ZooKeeper packet size.</p>
+
+<p>Bookie server manages its active ledgers in a sorted map, which simplifies access to active ledgers in a particular (level1, level2) partition.</p>
+
+<p>Garbage collection in bookie server is processed node by node as follows:</p>
+
+<ul>
+<li>Fetching all level1 nodes, by calling zk#getChildren(ledgerRootPath).<ul>
+<li>For each level1 nodes, fetching their level2 nodes :</li>
+<li>For each partition (level1, level2) :<ul>
+<li>Fetch all existed ledgers from zookeeper belonging to partition (level1, level2) (<code>zkActiveLedgers</code>).</li>
+<li>Fetch all ledgers currently active in the bookie which belong to partition (level1, level2) (<code>bkActiveLedgers</code>).</li>
+<li>Loop over <code>bkActiveLedgers</code> to find those ledgers which do not exist in <code>zkActiveLedgers</code>, and garbage collect them.</li>
+</ul></li>
+</ul></li>
+</ul>
+
+<blockquote><p><span class="caps">NOTE</span>: Hierarchical Ledger Manager is more suitable to manage large number of ledgers existed in BookKeeper.</p></blockquote>
+ </div>
+ </td>
+ <td valign="top">
+ <div class="navigation">
+ <div class="navigation_top">
+<div class="searchbox">
+<form action="http://search-hadoop.com/zookeeper" method="get">
+<input onFocus="getBlank (this, 'Search with Apache Solr');" size="25" name="q" id="query" type="text" value="Search with Apache Solr">
+ <input value="Search" type="submit">
+</form>
+</div>
+
+ <div class="navigation_bottom">
+<!-- sidenav -->
+<h3>Project</h3>
+
+<ul>
+<li><a href="/bookkeeper/releases.html#news">News</a></li>
+<li><a href="https://cwiki.apache.org/confluence/display/BOOKKEEPER/Index">Wiki</a></li>
+<li><a href="/bookkeeper/credits.html">Credits</a></li>
+<li><a href="/bylaws.html">Bylaws</a></li>
+<li><a href="http://www.apache.org/licenses/">License</a></li>
+<li><a href="/privacy.html">Privacy Policy</a></li>
+<li><a href="http://www.apache.org/foundation/sponsorship.html">Sponsorship</a></li>
+<li><a href="http://www.apache.org/foundation/thanks.html">Thanks</a></li>
+<li><a href="http://www.apache.org/security/">Security</a></li>
+</ul>
+
+
+<h3>Documentation</h3>
+
+<ul>
+<li><a href="/bookkeeper/docs/trunk">Trunk</a></li>
+</ul>
+
+<h3>Developers</h3>
+
+<ul>
+<li><a href="/bookkeeper/lists.html">Mailing Lists</a></li>
+<li><a href="/bookkeeper/irc.html"><span class="caps">IRC</span> Channel</a></li>
+<li><a href="/bookkeeper/svn.html">Version Control</a></li>
+<li><a href="https://issues.apache.org/jira/browse/BOOKKEEPER">Issue Tracker</a></li>
+</ul>
+<!-- sidenav -->
+ </div>
+ </div>
+ </div>
+ </td>
+ </tr>
+ </tbody>
+ </table>
+
+<!--
+ <div class="bottom_red_bar"></div>
+-->
+ </div>
+ </div>
+ </div>
+</div>
+<div class="black_box">
+<div class="footer">
+ <div class="footer_l">
+ <div class="footer_r">
+ <div>
+
+ </div>
+ </div>
+ </div>
+</div>
+</div>
+</div>
+<!--
+<div class="design_attribution">Page Template Design By Marc Prud'hommeaux based on <a href="http://activemq.apache.org/">ActiveMQ template</a></div>
+-->
+<div class="copyright_footer">
+<p>Copyright © 2010 The Apache Software Foundation, Licensed under the <a href="http://www.apache.org/licenses/LICENSE-2.0">Apache License, Version 2.0</a>.<br>Apache ZooKeeper, ZooKeeper, Apache, the Apache feather logo, and the Apache ZooKeeper project logo are trademarks of The Apache Software Foundation.</p>
+</div>
+<script src="skeleton_files/urchin.js" type="text/javascript">
+</script>
+<script type="text/javascript">
+_uacct = "UA-1940143-1";
+urchinTracker();
+</script>
+
+</body></html>
Added: websites/staging/zookeeper/trunk/content/bookkeeper/docs/r4.0.0/bookkeeperOverview.html
==============================================================================
--- websites/staging/zookeeper/trunk/content/bookkeeper/docs/r4.0.0/bookkeeperOverview.html (added)
+++ websites/staging/zookeeper/trunk/content/bookkeeper/docs/r4.0.0/bookkeeperOverview.html Wed Dec 7 14:10:32 2011
@@ -0,0 +1,319 @@
+<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
+<html>
+<head>
+<!--
+
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE- 2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+ <link href="/bookkeeper/css/site.css" rel="stylesheet" type="text/css">
+ <link href="/bookkeeper/css/type-settings.css" rel="stylesheet" type="text/css">
+
+ <title>Apache BookKeeper - BookKeeper overview</title>
+
+ <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
+</head>
+<body>
+<div class="white_box">
+<div class="header">
+ <div class="header_l">
+ <div class="header_r">
+ </div>
+ </div>
+</div>
+<div class="content">
+ <div class="content_l">
+ <div class="content_r">
+ <div>
+
+<!-- Banner -->
+
+
+ <div id="header_background">
+ <div id="bookkeeper_logo">
+ <a href="/bookkeeper"><img src="/bookkeeper/images/bookkeeper_small.gif"/></a> <b><font size=+4 color=white>Apache BookKeeper™</font></b> <a href="http://apache.org"><img src="/bookkeeper/images/feather_small.gif"/></a>
+ </div>
+ </div>
+
+ <table border="0">
+ <tbody>
+ <tr>
+ <td style="overflow: hidden;" valign="top" width="100%">
+ <div class="wiki-content">
+ <h1>Abstract</h1>
+
+<p>This guide contains detailed information about using BookKeeper for logging. It discusses the basic operations BookKeeper supports, and how to create logs and perform basic read and write operations on these logs.</p>
+
+<h1>BookKeeper introduction</h1>
+
+<p>BookKeeper is a replicated service to reliably log streams of records. In BookKeeper, servers are "bookies", log streams are "ledgers", and each unit of a log (aka record) is a "ledger entry". BookKeeper is designed to be reliable; bookies, the servers that store ledgers, can crash, corrupt data, discard data, but as long as there are enough bookies behaving correctly the service as a whole behaves correctly. </p>
+
+<p>The initial motivation for BookKeeper comes from the namenode of <span class="caps">HDFS.</span> Namenodes have to log operations in a reliable fashion so that recovery is possible in the case of crashes. We have found the applications for BookKeeper extend far beyond <span class="caps">HDFS, </span>however. Essentially, any application that requires an append storage can replace their implementations with BookKeeper. BookKeeper has the advantage of writing efficiently, replicating for fault tolerance, and scaling throughput with the number of servers through striping. </p>
+
+<p>At a high level, a bookkeeper client receives entries from a client application and stores it to sets of bookies, and there are a few advantages in having such a service: </p>
+
+<ul>
+<li>We can use hardware that is optimized for such a service. We currently believe that such a system has to be optimized only for disk I/O; </li>
+<li>We can have a pool of servers implementing such a log system, and shared among a number of servers; </li>
+<li>We can have a higher degree of replication with such a pool, which makes sense if the hardware necessary for it is cheaper compared to the one the application uses. </li>
+</ul>
+
+
+<h1>In slightly more detail...</h1>
+
+<p>BookKeeper implements highly available logs, and it has been designed with write-ahead logging in mind. Besides high availability due to the replicated nature of the service, it provides high throughput due to striping. As we write entries in a subset of bookies of an ensemble and rotate writes across available quorums, we are able to increase throughput with the number of servers for both reads and writes. Scalability is a property that is possible to achieve in this case due to the use of quorums. Other replication techniques, such as state-machine replication, do not enable such a property. </p>
+
+<p>An application first creates a ledger before writing to bookies through a local BookKeeper client instance. Upon creating a ledger, a BookKeeper client writes metadata about the ledger to ZooKeeper. Each ledger currently has a single writer. This writer has to execute a close ledger operation before any other client can read from it. If the writer of a ledger does not close a ledger properly because, for example, it has crashed before having the opportunity of closing the ledger, then the next client that tries to open a ledger executes a procedure to recover it. As closing a ledger consists essentially of writing the last entry written to a ledger to ZooKeeper, the recovery procedure simply finds the last entry written correctly and writes it to ZooKeeper. </p>
+
+<p>Note that currently this recovery procedure is executed automatically upon trying to open a ledger and no explicit action is necessary. Although two clients may try to recover a ledger concurrently, only one will succeed, the first one that is able to create the close znode for the ledger. </p>
+
+<h1>Bookkeeper elements and concepts</h1>
+
+<p>BookKeeper uses four basic elements: </p>
+
+<ul>
+<li> <em>Ledger</em> : A ledger is a sequence of entries, and each entry is a sequence of bytes. Entries are written sequentially to a ledger and at most once. Consequently, ledgers have an append-only semantics; </li>
+<li> <em>BookKeeper client</em> : A client runs along with a BookKeeper application, and it enables applications to execute operations on ledgers, such as creating a ledger and writing to it; </li>
+<li> <em>Bookie</em> : A bookie is a BookKeeper storage server. Bookies store the content of ledgers. For any given ledger L, we call an <em>ensemble</em> the group of bookies storing the content of L. For performance, we store on each bookie of an ensemble only a fragment of a ledger. That is, we stripe when writing entries to a ledger such that each entry is written to sub-group of bookies of the ensemble. </li>
+<li> <em>Metadata storage service</em> : BookKeeper requires a metadata storage service to store information related to ledgers and available bookies. We currently use ZooKeeper for such a task. </li>
+</ul>
+
+
+<h1>Bookkeeper initial design</h1>
+
+<p>A set of bookies implements BookKeeper, and we use a quorum-based protocol to replicate data across the bookies. There are basically two operations to an existing ledger: read and append. Here is the complete <span class="caps">API </span>list (mode detail <a href="http://bookkeeperProgrammer.html">here</a>): </p>
+
+<ul>
+<li>Create ledger: creates a new empty ledger; </li>
+<li>Open ledger: opens an existing ledger for reading; </li>
+<li>Add entry: adds a record to a ledger either synchronously or asynchronously; </li>
+<li>Read entries: reads a sequence of entries from a ledger either synchronously or asynchronously </li>
+</ul>
+
+
+<p>There is only a single client that can write to a ledger. Once that ledger is closed or the client fails, no more entries can be added. (We take advantage of this behavior to provide our strong guarantees.) There will not be gaps in the ledger. Fingers get broken, people get roughed up or end up in prison when books are manipulated, so there is no deleting or changing of entries. </p>
+
+<p><img src="images/bk-overview.jpg" alt="" />
+p. A simple use of BooKeeper is to implement a write-ahead transaction log. A server maintains an in-memory data structure (with periodic snapshots for example) and logs changes to that structure before it applies the change. The application server creates a ledger at startup and store the ledger id and password in a well known place (ZooKeeper maybe). When it needs to make a change, the server adds an entry with the change information to a ledger and apply the change when BookKeeper adds the entry successfully. The server can even use asyncAddEntry to queue up many changes for high change throughput. BooKeeper meticulously logs the changes in order and call the completion functions in order. </p>
+
+<p>When the application server dies, a backup server will come online, get the last snapshot and then it will open the ledger of the old server and read all the entries from the time the snapshot was taken. (Since it doesn't know the last entry number it will use <span class="caps">MAX</span>_INTEGER). Once all the entries have been processed, it will close the ledger and start a new one for its use. </p>
+
+<p>A client library takes care of communicating with bookies and managing entry numbers. An entry has the following fields: </p>
+
+<table><tr><td>Field</td><td>Type</td><td>Description</td></tr><tr><td>Ledger number</td><td>long</td><td>The id of the ledger of this entry</td></tr><tr><td>Entry number</td><td>long</td><td>The id of this entry</td></tr><tr><td>last confirmed ( <em>LC</em> )</td><td>long</td><td>id of the last recorded entry</td></tr><tr><td>data</td><td>byte[]</td><td>the entry data (supplied by application)</td></tr><tr><td>authentication code</td><td>byte[]</td><td>Message authentication code that includes all other fields of the entry</td></tr></table>
+
+
+<p>The client library generates a ledger entry. None of the fields are modified by the bookies and only the first three fields are interpreted by the bookies. </p>
+
+<p>To add to a ledger, the client generates the entry above using the ledger number. The entry number will be one more than the last entry generated. The <em>LC</em> field contains the last entry that has been successfully recorded by BookKeeper. If the client writes entries one at a time, <em>LC</em> is the last entry id. But, if the client is using asyncAddEntry, there may be many entries in flight. An entry is considered recorded when both of the following conditions are met: </p>
+
+<ul>
+<li>the entry has been accepted by a quorum of bookies </li>
+<li>all entries with a lower entry id have been accepted by a quorum of bookies </li>
+</ul>
+
+
+<p> <em>LC</em> seems mysterious right now, but it is too early to explain how we use it; just smile and move on. </p>
+
+<p>Once all the other fields have been field in, the client generates an authentication code with all of the previous fields. The entry is then sent to a quorum of bookies to be recorded. Any failures will result in the entry being sent to a new quorum of bookies. </p>
+
+<p>To read, the client library initially contacts a bookie and starts requesting entries. If an entry is missing or invalid (a bad <span class="caps">MAC </span>for example), the client will make a request to a different bookie. By using quorum writes, as long as enough bookies are up we are guaranteed to eventually be able to read an entry. </p>
+
+<h1>Bookkeeper metadata management</h1>
+
+<p>There are some meta data that needs to be made available to BookKeeper clients: </p>
+
+<ul>
+<li>The available bookies; </li>
+<li>The list of ledgers; </li>
+<li>The list of bookies that have been used for a given ledger; </li>
+<li>The last entry of a ledger; </li>
+</ul>
+
+
+<p>We maintain this information in ZooKeeper. Bookies use ephemeral nodes to indicate their availability. Clients use znodes to track ledger creation and deletion and also to know the end of the ledger and the bookies that were used to store the ledger. Bookies also watch the ledger list so that they can cleanup ledgers that get deleted. </p>
+
+<h1>Closing out ledgers</h1>
+
+<p>The process of closing out the ledger and finding the last entry is difficult due to the durability guarantees of BookKeeper: </p>
+
+<ul>
+<li>If an entry has been successfully recorded, it must be readable. </li>
+<li>If an entry is read once, it must always be available to be read. </li>
+</ul>
+
+
+<p>If the ledger was closed gracefully, ZooKeeper will have the last entry and everything will work well. But, if the BookKeeper client that was writing the ledger dies, there is some recovery that needs to take place. </p>
+
+<p>The problematic entries are the ones at the end of the ledger. There can be entries in flight when a BookKeeper client dies. If the entry only gets to one bookie, the entry should not be readable since the entry will disappear if that bookie fails. If the entry is only on one bookie, that doesn't mean that the entry has not been recorded successfully; the other bookies that recorded the entry might have failed. </p>
+
+<p>The trick to making everything work is to have a correct idea of a last entry. We do it in roughly three steps: </p>
+
+<ol>
+<li>Find the entry with the highest last recorded entry, <em>LC</em> ; </li>
+<li>Find the highest consecutively recorded entry, <em>LR</em> ; </li>
+<li>Make sure that all entries between <em>LC</em> and <em>LR</em> are on a quorum of bookies; </li>
+</ol>
+
+<h1>Data Management in Bookies</h1>
+
+<p>This section gives an overview of how a bookie manages its ledger fragments. </p>
+
+<h2>Basic</h2>
+
+<p>Bookies manage data in a log-structured way, which is implemented using three kind of files:</p>
+
+<ul>
+<li><em>Journal</em> : A journal file contains the BookKeeper transaction logs. Before any update takes place, a bookie ensures that a transaction describing the update is written to non-volatile storage. A new journal file is created once the bookie starts or the older journal file reaches the journal file size threshold.</li>
+<li><em>Entry Log</em> : An entry log file manages the written entries received from BookKeeper clients. Entries from different ledgers are aggregated and written sequentially, while their offsets are kept as pointers in <em>LedgerCache</em> for fast lookup. A new entry log file is created once the bookie starts or the older entry log file reaches the entry log size threshold. Old entry log files are removed by the <em>Garbage Collector Thread</em> once they are not associated with any active ledger.</li>
+<li><em>Index File</em> : An index file is created for each ledger, which comprises a header and several fixed-length index pages, recording the offsets of data stored in entry log files. </li>
+</ul>
+
+<p>Since updating index files would introduce random disk I/O, for performance consideration, index files are updated lazily by a <em>Sync Thread</em> running in the background. Before index pages are persisted to disk, they are gathered in <em>LedgerCache</em> for lookup.</p>
+
+<ul>
+<li><em>LedgerCache</em> : A memory pool caches ledger index pages, which more efficiently manage disk head scheduling.</li>
+</ul>
+
+<h2>Add Entry</h2>
+
+<p>When a bookie receives entries from clients to be written, these entries will go through the following steps to be persisted to disk:</p>
+
+<ol>
+<li>Append the entry in <em>Entry Log</em>, return its position { logId , offset } ;</li>
+<li>Update the index of this entry in <em>Ledger Cache</em> ;</li>
+<li>Append a transaction corresponding to this entry update in <em>Journal</em> ;</li>
+<li>Respond to BookKeeper client ;</li>
+</ol>
+
+<ul>
+<li>For performance reasons, <em>Entry Log</em> buffers entries in memory and commit them in batches, while <em>Ledger Cache</em> holds index pages in memory and flushes them lazily. We will discuss data flush and how to ensure data integrity in the following section 'Data Flush'.</li>
+</ul>
+
+<h2>Data Flush</h2>
+
+<p>Ledger index pages are flushed to index files in the following two cases:</p>
+
+<ol>
+<li><em>LedgerCache</em> memory reaches its limit. There is no more space available to hold newer index pages. Dirty index pages will be evicted from <em>LedgerCache</em> and persisted to index files.</li>
+<li>A background thread <em>Sync Thread</em> is responsible for flushing index pages from <em>LedgerCache</em> to index files periodically.</li>
+</ol>
+
+<p>Besides flushing index pages, <em>Sync Thread</em> is responsible for rolling journal files in case that journal files use too much disk space. </p>
+
+<p>The data flush flow in <em>Sync Thread</em> is as follows:</p>
+
+<ol>
+<li>Records a <em>LastLogMark</em> in memory. The <em>LastLogMark</em> contains two parts: first one is <em>txnLogId</em> (file id of a journal) and the second one is <em>txnLogPos</em> (offset in a journal). The <em>LastLogMark</em> indicates that those entries before it have been persisted to both index and entry log files.</li>
+<li>Flushes dirty index pages from <em>LedgerCache</em> to index file, and flushes entry log files to ensure all buffered entries in entry log files are persisted to disk.<ol>
+<li>Ideally, a bookie just needs to flush index pages and entry log files that contains entries before <em>LastLogMark</em>. There is no such information in <em>LedgerCache</em> and <em>Entry Log</em> mapping to journal files, though. Consequently, the thread flushes <em>LedgerCache</em> and <em>Entry Log</em> entirely here, and may flush entries after the <em>LastLogMark</em>. Flushing more is not a problem, though, just redundant.</li>
+</ol>
+</li>
+<li>Persists <em>LastLogMark</em> to disk, which means entries added before <em>LastLogMark</em> whose entry data and index page were also persisted to disk. It is the time to safely remove journal files created earlier than <em>txnLogId</em>.<ol>
+<li>If the bookie has crashed before persisting <em>LastLogMark</em> to disk, it still has journal files containing entries for which index pages may not have been persisted. Consequently, when this bookie restarts, it inspects journal files to restore those entries; data isn't lost.</li>
+</ol></li>
+</ol>
+
+<p>Using the above data flush mechanism, it is safe for the <em>Sync Thread</em> to skip data flushing when the bookie shuts down. However, in <em>Entry Logger</em>, it uses <em>BufferedChannel</em> to write entries in batches and there might be data buffered in <em>BufferedChannel</em> upon a shut down. The bookie needs to ensure <em>Entry Logger</em> flushes its buffered data during shutting down. Otherwise, <em>Entry Log</em> files become corrupted with partial entries.</p>
+
+<p>As described above, <em>EntryLogger#flush</em> is invoked in the following two cases:<br />
+* in <em>Sync Thread</em> : used to ensure entries added before <em>LastLogMark</em> are persisted to disk.<br />
+* in <em>ShutDown</em> : used to ensure its buffered data persisted to disk to avoid data corruption with partial entries.</p>
+ </div>
+ </td>
+ <td valign="top">
+ <div class="navigation">
+ <div class="navigation_top">
+<div class="searchbox">
+<form action="http://search-hadoop.com/zookeeper" method="get">
+<input onFocus="getBlank (this, 'Search with Apache Solr');" size="25" name="q" id="query" type="text" value="Search with Apache Solr">
+ <input value="Search" type="submit">
+</form>
+</div>
+
+ <div class="navigation_bottom">
+<!-- sidenav -->
+<h3>Project</h3>
+
+<ul>
+<li><a href="/bookkeeper/releases.html#news">News</a></li>
+<li><a href="https://cwiki.apache.org/confluence/display/BOOKKEEPER/Index">Wiki</a></li>
+<li><a href="/bookkeeper/credits.html">Credits</a></li>
+<li><a href="/bylaws.html">Bylaws</a></li>
+<li><a href="http://www.apache.org/licenses/">License</a></li>
+<li><a href="/privacy.html">Privacy Policy</a></li>
+<li><a href="http://www.apache.org/foundation/sponsorship.html">Sponsorship</a></li>
+<li><a href="http://www.apache.org/foundation/thanks.html">Thanks</a></li>
+<li><a href="http://www.apache.org/security/">Security</a></li>
+</ul>
+
+
+<h3>Documentation</h3>
+
+<ul>
+<li><a href="/bookkeeper/docs/trunk">Trunk</a></li>
+</ul>
+
+<h3>Developers</h3>
+
+<ul>
+<li><a href="/bookkeeper/lists.html">Mailing Lists</a></li>
+<li><a href="/bookkeeper/irc.html"><span class="caps">IRC</span> Channel</a></li>
+<li><a href="/bookkeeper/svn.html">Version Control</a></li>
+<li><a href="https://issues.apache.org/jira/browse/BOOKKEEPER">Issue Tracker</a></li>
+</ul>
+<!-- sidenav -->
+ </div>
+ </div>
+ </div>
+ </td>
+ </tr>
+ </tbody>
+ </table>
+
+<!--
+ <div class="bottom_red_bar"></div>
+-->
+ </div>
+ </div>
+ </div>
+</div>
+<div class="black_box">
+<div class="footer">
+ <div class="footer_l">
+ <div class="footer_r">
+ <div>
+
+ </div>
+ </div>
+ </div>
+</div>
+</div>
+</div>
+<!--
+<div class="design_attribution">Page Template Design By Marc Prud'hommeaux based on <a href="http://activemq.apache.org/">ActiveMQ template</a></div>
+-->
+<div class="copyright_footer">
+<p>Copyright © 2010 The Apache Software Foundation, Licensed under the <a href="http://www.apache.org/licenses/LICENSE-2.0">Apache License, Version 2.0</a>.<br>Apache ZooKeeper, ZooKeeper, Apache, the Apache feather logo, and the Apache ZooKeeper project logo are trademarks of The Apache Software Foundation.</p>
+</div>
+<script src="skeleton_files/urchin.js" type="text/javascript">
+</script>
+<script type="text/javascript">
+_uacct = "UA-1940143-1";
+urchinTracker();
+</script>
+
+</body></html>
Added: websites/staging/zookeeper/trunk/content/bookkeeper/docs/r4.0.0/bookkeeperProgrammer.html
==============================================================================
--- websites/staging/zookeeper/trunk/content/bookkeeper/docs/r4.0.0/bookkeeperProgrammer.html (added)
+++ websites/staging/zookeeper/trunk/content/bookkeeper/docs/r4.0.0/bookkeeperProgrammer.html Wed Dec 7 14:10:32 2011
@@ -0,0 +1,257 @@
+<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
+<html>
+<head>
+<!--
+
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE- 2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+ <link href="/bookkeeper/css/site.css" rel="stylesheet" type="text/css">
+ <link href="/bookkeeper/css/type-settings.css" rel="stylesheet" type="text/css">
+
+ <title>Apache BookKeeper - BookKeeper Getting Started Guide</title>
+
+ <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
+</head>
+<body>
+<div class="white_box">
+<div class="header">
+ <div class="header_l">
+ <div class="header_r">
+ </div>
+ </div>
+</div>
+<div class="content">
+ <div class="content_l">
+ <div class="content_r">
+ <div>
+
+<!-- Banner -->
+
+
+ <div id="header_background">
+ <div id="bookkeeper_logo">
+ <a href="/bookkeeper"><img src="/bookkeeper/images/bookkeeper_small.gif"/></a> <b><font size=+4 color=white>Apache BookKeeper™</font></b> <a href="http://apache.org"><img src="/bookkeeper/images/feather_small.gif"/></a>
+ </div>
+ </div>
+
+ <table border="0">
+ <tbody>
+ <tr>
+ <td style="overflow: hidden;" valign="top" width="100%">
+ <div class="wiki-content">
+ <h1>Abstract</h1>
+
+<p>This guide contains detailed information about using BookKeeper for write ahead logging. It discusses the basic operations BookKeeper supports, and how to create logs and perform basic read and write operations on these logs. The main classes used by BookKeeper client are <a href="./apidocs/org/apache/bookkeeper/client/BookKeeper.html">BookKeeper</a> and <a href="./apidocs/org/apache/bookkeeper/client/LedgerHandle.html">LedgerHandle</a>. </p>
+
+<p>BookKeeper is the main client used to create, open and delete ledgers. A ledger is a log file in BookKeeper, which contains a sequence of entries. Only the client which creates a ledger can write to it. A LedgerHandle represents the ledger to the client, and allows the client to read and write entries. When the client is finished writing they can close the LedgerHandle. Once a ledger has been closed, all client who read from it are guaranteed to read the exact same entries in the exact same order. All methods of BookKeeper and LedgerHandle have synchronous and asynchronous versions. Internally the synchronous versions are implemented using the asynchronous.</p>
+
+<h1> Instantiating BookKeeper</h1>
+
+<p>To create a BookKeeper client, you need to create a configuration object and set the address of the ZooKeeper ensemble in use. For example, if you were using <code>zk1.example.com:2181,zk2.example.com:2181,zk3.example.com:2181</code> as your ensemble, you would create the BookKeeper client as follows.</p>
+
+
+
+<pre><code>
+ClientConfiguration conf = new ClientConfiguration();
+conf.setZkServers("zk1.example.com:2181,zk2.example.com:2181,zk3.example.com:2181");
+
+BookKeeper client = new BookKeeper(conf);
+</code></pre>
+
+
+
+<p>It is important to close the client once you are finished working with it. The set calls on ClientConfiguration are chainable, so instead of putting a set* call on a new line as above, it is possible to make a number of calls on the one line. For example;</p>
+
+
+
+<pre><code>
+ClientConfiguration conf = new ClientConfiguration().setZkServers("localhost:2181").setZkTimeout(5000);
+</code></pre>
+
+
+
+<p>There is also a useful shortcut constructor which allows you to pass the zookeeper ensemble string directly to BookKeeper.</p>
+
+
+<pre><code>
+BookKeeper client = new BookKeeper("localhost:2181");
+</code></pre>
+
+
+
+<p>See <a href="./apidocs/org/apache/bookkeeper/client/BookKeeper.html">BookKeeper</a> for the full api.</p>
+
+
+<h1> Creating a ledger</h1>
+
+<p>Before writing entries to BookKeeper, it is necessary to create a ledger. Before creating the ledger you must decide the ensemble size and the quorum size. </p>
+
+<p>The ensemble size is the number of Bookies over which entries will be striped. The quorum size is the number of bookies which an entry will be written to. Striping is done in a round robin fashion. For example, if you have an ensemble size of 3 (consisting of bk1, bk2 & bk3), and a quorum of 2, entry 1 will be written to bk1 & bk2, entry 2 will be written to bk2 & bk3, entry 3 will be written to bk3 & bk1 and so on.</p>
+
+<p>Ledgers are also created with a digest type and password. The digest type is used to generate a checksum so that when reading entries we can ensure that the content is the same as what was written. The password is used as an access control mechanism.</p>
+
+<p>To create a ledger, with ensemble size 3, quorum size 2, using a <span class="caps">CRC </span>to checksum and "foobar" as the password, do the following:</p>
+
+
+
+<pre><code>
+LedgerHandle lh = client.createLedger(3, 2, DigestType.CRC32, "foobar");
+</code></pre>
+
+
+
+<p>You can now write to this ledger handle. As you probably plan to read the ledger at some stage, now is a good time to store the id of the ledger somewhere. The ledger id is a long, and can be obtained with <code>lh.getId()</code>.</p>
+
+<h1> Adding entries to a ledger</h1>
+
+<p>Once you have obtained a ledger handle, you can start adding entries to it. Entries are simply arrays of bytes. As such, adding entries to the ledger is rather simple.</p>
+
+
+
+<pre><code>
+lh.addEntry("Hello World!".getBytes());
+</code></pre>
+
+
+
+<h1> Closing a ledger</h1>
+
+<p>Once a client is done writing, it can closes the ledger. Closing the ledger is a very important step in BookKeeper, as once a ledger is closed, all reading clients are guaranteed to read the same sequence of entries in the same order. Closing takes no parameters. </p>
+
+
+
+<pre><code>
+lh.close();
+</code></pre>
+
+
+
+<h1> Opening a ledger</h1>
+
+<p>To read from a ledger, a client must open it first. To open a ledger you must know its <span class="caps">ID, </span>which digest type was used when creating it, and its password. To open the ledger we created above, assuming it has ID 1;</p>
+
+
+
+<pre><code>
+LedgerHandle lh2 = client.openLedger(1, DigestType.CRC32, "foobar");
+</code></pre>
+
+
+
+<p>You can now read entries from the ledger. Any attempt to write to this handle will throw an exception.</p>
+
+<blockquote><p><span class="caps">NOTE</span>: Opening a ledger, which another client already has open for writing will prevent that client from writing any new entries to it. If you do not wish this to happen, you should use the openLedgerNoRecovery method. However, keep in mind that without recovery, you lose the guarantees of what entries are in the ledger. You should only use openLedgerNoRecovery if you know what you are doing.</p></blockquote>
+
+<h1>Reading entries from a ledger</h1>
+
+<p>Now that you have an open ledger, you can read entries from it. You can use <code>getLastAddConfirmed</code> to get the id of the last entry in the ledger.</p>
+
+
+
+<pre><code>
+long lastEntry = lh2.getLastAddConfirmed();
+Enumeration<LedgerEntry> entries = lh2.readEntries(0, 9);
+while (entries.hasMoreElements()) {
+ byte[] bytes = entries.nextElement().getEntry();
+ System.out.println(new String(bytes));
+}
+</code></pre>
+ </div>
+ </td>
+ <td valign="top">
+ <div class="navigation">
+ <div class="navigation_top">
+<div class="searchbox">
+<form action="http://search-hadoop.com/zookeeper" method="get">
+<input onFocus="getBlank (this, 'Search with Apache Solr');" size="25" name="q" id="query" type="text" value="Search with Apache Solr">
+ <input value="Search" type="submit">
+</form>
+</div>
+
+ <div class="navigation_bottom">
+<!-- sidenav -->
+<h3>Project</h3>
+
+<ul>
+<li><a href="/bookkeeper/releases.html#news">News</a></li>
+<li><a href="https://cwiki.apache.org/confluence/display/BOOKKEEPER/Index">Wiki</a></li>
+<li><a href="/bookkeeper/credits.html">Credits</a></li>
+<li><a href="/bylaws.html">Bylaws</a></li>
+<li><a href="http://www.apache.org/licenses/">License</a></li>
+<li><a href="/privacy.html">Privacy Policy</a></li>
+<li><a href="http://www.apache.org/foundation/sponsorship.html">Sponsorship</a></li>
+<li><a href="http://www.apache.org/foundation/thanks.html">Thanks</a></li>
+<li><a href="http://www.apache.org/security/">Security</a></li>
+</ul>
+
+
+<h3>Documentation</h3>
+
+<ul>
+<li><a href="/bookkeeper/docs/trunk">Trunk</a></li>
+</ul>
+
+<h3>Developers</h3>
+
+<ul>
+<li><a href="/bookkeeper/lists.html">Mailing Lists</a></li>
+<li><a href="/bookkeeper/irc.html"><span class="caps">IRC</span> Channel</a></li>
+<li><a href="/bookkeeper/svn.html">Version Control</a></li>
+<li><a href="https://issues.apache.org/jira/browse/BOOKKEEPER">Issue Tracker</a></li>
+</ul>
+<!-- sidenav -->
+ </div>
+ </div>
+ </div>
+ </td>
+ </tr>
+ </tbody>
+ </table>
+
+<!--
+ <div class="bottom_red_bar"></div>
+-->
+ </div>
+ </div>
+ </div>
+</div>
+<div class="black_box">
+<div class="footer">
+ <div class="footer_l">
+ <div class="footer_r">
+ <div>
+
+ </div>
+ </div>
+ </div>
+</div>
+</div>
+</div>
+<!--
+<div class="design_attribution">Page Template Design By Marc Prud'hommeaux based on <a href="http://activemq.apache.org/">ActiveMQ template</a></div>
+-->
+<div class="copyright_footer">
+<p>Copyright © 2010 The Apache Software Foundation, Licensed under the <a href="http://www.apache.org/licenses/LICENSE-2.0">Apache License, Version 2.0</a>.<br>Apache ZooKeeper, ZooKeeper, Apache, the Apache feather logo, and the Apache ZooKeeper project logo are trademarks of The Apache Software Foundation.</p>
+</div>
+<script src="skeleton_files/urchin.js" type="text/javascript">
+</script>
+<script type="text/javascript">
+_uacct = "UA-1940143-1";
+urchinTracker();
+</script>
+
+</body></html>
Added: websites/staging/zookeeper/trunk/content/bookkeeper/docs/r4.0.0/bookkeeperStarted.html
==============================================================================
--- websites/staging/zookeeper/trunk/content/bookkeeper/docs/r4.0.0/bookkeeperStarted.html (added)
+++ websites/staging/zookeeper/trunk/content/bookkeeper/docs/r4.0.0/bookkeeperStarted.html Wed Dec 7 14:10:32 2011
@@ -0,0 +1,234 @@
+<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
+<html>
+<head>
+<!--
+
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE- 2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+ <link href="/bookkeeper/css/site.css" rel="stylesheet" type="text/css">
+ <link href="/bookkeeper/css/type-settings.css" rel="stylesheet" type="text/css">
+
+ <title>Apache BookKeeper - BookKeeper Getting Started Guide</title>
+
+ <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
+</head>
+<body>
+<div class="white_box">
+<div class="header">
+ <div class="header_l">
+ <div class="header_r">
+ </div>
+ </div>
+</div>
+<div class="content">
+ <div class="content_l">
+ <div class="content_r">
+ <div>
+
+<!-- Banner -->
+
+
+ <div id="header_background">
+ <div id="bookkeeper_logo">
+ <a href="/bookkeeper"><img src="/bookkeeper/images/bookkeeper_small.gif"/></a> <b><font size=+4 color=white>Apache BookKeeper™</font></b> <a href="http://apache.org"><img src="/bookkeeper/images/feather_small.gif"/></a>
+ </div>
+ </div>
+
+ <table border="0">
+ <tbody>
+ <tr>
+ <td style="overflow: hidden;" valign="top" width="100%">
+ <div class="wiki-content">
+ <h1>Abstract</h1>
+
+<p>This guide contains detailed information about using BookKeeper for logging. It discusses the basic operations BookKeeper supports, and how to create logs and perform basic read and write operations on these logs.</p>
+
+<h1>Getting Started: Setting up BookKeeper to write logs.</h1>
+
+<p>This document contains information to get you started quickly with BookKeeper. It is aimed primarily at developers willing to try it out, and contains simple installation instructions for a simple BookKeeper installation and a simple programming example. For further programming detail, please refer to <a href="http://bookkeeperProgrammer.html">BookKeeper Programmer's Guide</a>. </p>
+
+<h1>Pre-requisites</h1>
+
+<p>See <a href="./bookkeeperConfig.html#bk_sysReqin">System Requirements</a> the Admin guide.</p>
+
+<h1>Download</h1>
+
+<p>BookKeeper trunk can be downloaded from subversion. See "Version Control:http://zookeeper.apache.org/bookkeeper/svn.html. </p>
+
+<h1>LocalBookKeeper</h1>
+
+<p>BookKeeper provides a utility program to start a standalone ZooKeeper ensemble and a number of bookies on a local machine. As this all runs on a local machine, throughput will be very low. It should only be used for testing.</p>
+
+<p>To start a local bookkeeper ensemble with 5 bookies:</p>
+
+<p> <code>bookkeeper-server/bin/bookkeeper localbookie 5</code></p>
+
+<h1>Setting up bookies</h1>
+
+<p>If you're bold and you want more than just running things locally, then you'll need to run bookies in different servers. You'll need at least three bookies to start with. </p>
+
+<p>For each bookie, we need to execute a command like the following: </p>
+
+<p> <code>bookkeeper-server/bin/bookkeeper bookie</code></p>
+
+<p>This command will use the default directories for storing ledgers and the write ahead log, and will look for a zookeeper server on localhost:2181. See the <a href="./bookkeeperConfig.html">Admin Guide</a> for more details.</p>
+
+<p>To see the default values of these configuration variables, run:</p>
+
+<p> <code>bookkeeper-server/bin/bookkeeper help</code></p>
+
+<h1>Setting up ZooKeeper</h1>
+
+<p>ZooKeeper stores metadata on behalf of BookKeeper clients and bookies. To get a minimal ZooKeeper installation to work with BookKeeper, we can set up one server running in standalone mode. Once we have the server running, we need to create a few znodes: </p>
+
+<ol>
+<li> <code>/ledgers </code> </li>
+<li> <code>/ledgers/available </code> </li>
+<li>For each bookie, we add one znode such that the name of the znode is the concatenation of the machine name and the port number that the bookie is listening on. For example, if a bookie is running on bookie.foo.com an is listening on port 3181, we add a znode <code>/ledgers/available/bookie.foo.com:3181</code> . </li>
+</ol>
+
+<h1>Example</h1>
+
+<p>In the following excerpt of code, we: </p>
+
+<ol>
+<li>Open a bookkeeper client;</li>
+<li>Create a ledger; </li>
+<li>Write to the ledger; </li>
+<li>Close the ledger; </li>
+<li>Open the same ledger for reading; </li>
+<li>Read from the ledger; </li>
+<li>Close the ledger again; </li>
+<li>Close the bookkeeper client.</li>
+</ol>
+
+
+
+<pre><code>
+BookKeeper bkc = new BookKeeper("localhost:2181");
+LedgerHandle lh = bkc.createLedger(ledgerPassword);
+ledgerId = lh.getId();
+ByteBuffer entry = ByteBuffer.allocate(4);
+
+for(int i = 0; i < 10; i++){
+ entry.putInt(i);
+ entry.position(0);
+ entries.add(entry.array());
+ lh.addEntry(entry.array());
+}
+lh.close();
+lh = bkc.openLedger(ledgerId, ledgerPassword);
+
+Enumeration<LedgerEntry> ls = lh.readEntries(0, 9);
+int i = 0;
+while(ls.hasMoreElements()){
+ ByteBuffer origbb = ByteBuffer.wrap(
+ entries.get(i++));
+ Integer origEntry = origbb.getInt();
+ ByteBuffer result = ByteBuffer.wrap(
+ ls.nextElement().getEntry());
+
+ Integer retrEntry = result.getInt();
+}
+lh.close();
+bkc.close();
+</code></pre>
+ </div>
+ </td>
+ <td valign="top">
+ <div class="navigation">
+ <div class="navigation_top">
+<div class="searchbox">
+<form action="http://search-hadoop.com/zookeeper" method="get">
+<input onFocus="getBlank (this, 'Search with Apache Solr');" size="25" name="q" id="query" type="text" value="Search with Apache Solr">
+ <input value="Search" type="submit">
+</form>
+</div>
+
+ <div class="navigation_bottom">
+<!-- sidenav -->
+<h3>Project</h3>
+
+<ul>
+<li><a href="/bookkeeper/releases.html#news">News</a></li>
+<li><a href="https://cwiki.apache.org/confluence/display/BOOKKEEPER/Index">Wiki</a></li>
+<li><a href="/bookkeeper/credits.html">Credits</a></li>
+<li><a href="/bylaws.html">Bylaws</a></li>
+<li><a href="http://www.apache.org/licenses/">License</a></li>
+<li><a href="/privacy.html">Privacy Policy</a></li>
+<li><a href="http://www.apache.org/foundation/sponsorship.html">Sponsorship</a></li>
+<li><a href="http://www.apache.org/foundation/thanks.html">Thanks</a></li>
+<li><a href="http://www.apache.org/security/">Security</a></li>
+</ul>
+
+
+<h3>Documentation</h3>
+
+<ul>
+<li><a href="/bookkeeper/docs/trunk">Trunk</a></li>
+</ul>
+
+<h3>Developers</h3>
+
+<ul>
+<li><a href="/bookkeeper/lists.html">Mailing Lists</a></li>
+<li><a href="/bookkeeper/irc.html"><span class="caps">IRC</span> Channel</a></li>
+<li><a href="/bookkeeper/svn.html">Version Control</a></li>
+<li><a href="https://issues.apache.org/jira/browse/BOOKKEEPER">Issue Tracker</a></li>
+</ul>
+<!-- sidenav -->
+ </div>
+ </div>
+ </div>
+ </td>
+ </tr>
+ </tbody>
+ </table>
+
+<!--
+ <div class="bottom_red_bar"></div>
+-->
+ </div>
+ </div>
+ </div>
+</div>
+<div class="black_box">
+<div class="footer">
+ <div class="footer_l">
+ <div class="footer_r">
+ <div>
+
+ </div>
+ </div>
+ </div>
+</div>
+</div>
+</div>
+<!--
+<div class="design_attribution">Page Template Design By Marc Prud'hommeaux based on <a href="http://activemq.apache.org/">ActiveMQ template</a></div>
+-->
+<div class="copyright_footer">
+<p>Copyright © 2010 The Apache Software Foundation, Licensed under the <a href="http://www.apache.org/licenses/LICENSE-2.0">Apache License, Version 2.0</a>.<br>Apache ZooKeeper, ZooKeeper, Apache, the Apache feather logo, and the Apache ZooKeeper project logo are trademarks of The Apache Software Foundation.</p>
+</div>
+<script src="skeleton_files/urchin.js" type="text/javascript">
+</script>
+<script type="text/javascript">
+_uacct = "UA-1940143-1";
+urchinTracker();
+</script>
+
+</body></html>