You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@zookeeper.apache.org by ma...@apache.org on 2011/11/23 09:49:11 UTC
svn commit: r799149 [2/2] - in /websites/production/zookeeper: ./ content/
content/bookkeeper/docs/trunk/ content/doc/r3.4.0/ content/doc/r3.4.0/api/
content/doc/r3.4.0/api/org/ content/doc/r3.4.0/api/org/apache/
content/doc/r3.4.0/api/org/apache/zooke...
Modified: websites/production/zookeeper/content/bookkeeper/docs/trunk/bookkeeperStarted.html
==============================================================================
--- websites/production/zookeeper/content/bookkeeper/docs/trunk/bookkeeperStarted.html (original)
+++ websites/production/zookeeper/content/bookkeeper/docs/trunk/bookkeeperStarted.html Wed Nov 23 08:49:02 2011
@@ -63,15 +63,19 @@
<h1>Pre-requisites</h1>
-<p>See <a href="http://bookkeeperConfig.html#bk_sysReqin">System Requirements</a> the Admin guide.</p>
+<p>See <a href="./bookkeeperConfig.html#bk_sysReqin">System Requirements</a> the Admin guide.</p>
<h1>Download</h1>
-<p>BookKeeper is distributed along with ZooKeeper. To get a ZooKeeper distribution, download a recent <a href="http://hadoop.apache.org/zookeeper/releases.htmlrelease">stable</a> from one of the Apache Download Mirrors.</p>
+<p>BookKeeper trunk can be downloaded from subversion. See "Version Control:http://zookeeper.apache.org/bookkeeper/svn.html. </p>
<h1>LocalBookKeeper</h1>
-<p>Under org.apache.bookkeeper.util, you'll find a java program called LocalBookKeeper.java that sets you up to run BookKeeper on a single machine. This is far from ideal from a performance perspective, but the program is useful for both test and educational purposes. </p>
+<p>BookKeeper provides a utility program to start a standalone ZooKeeper ensemble and a number of bookies on a local machine. As this all runs on a local machine, throughput will be very low. It should only be used for testing.</p>
+
+<p>To start a local bookkeeper ensemble with 5 bookies:</p>
+
+<p>@ bookkeeper-server/bin/bookkeeper localbookie 5</p>
<h1>Setting up bookies</h1>
@@ -79,9 +83,13 @@
<p>For each bookie, we need to execute a command like the following: </p>
-<p> <code>java -cp .:./zookeeper-<version>-bookkeeper.jar:./zookeeper-<version>.jar\ :lib/slf4j-api-1.6.1.jar:lib/slf4j-log4j12-1.6.1.jar:lib/log4j-1.2.15.jar -Dlog4j.configuration=log4j.properties\ org.apache.bookkeeper.proto.BookieServer 3181 127.0.0.1:2181 /path_to_log_device/\ /path_to_ledger_device/ </code> </p>
+<p> @ bookkeeper-server/bin/bookkeeper bookie</p>
+
+<p>This command will use the default directories for storing ledgers and the write ahead log, and will look for a zookeeper server on localhost:2181. To modify this you must edit bookkeeper-server/conf/bkenv.sh. <span class="caps">BOOKIE</span>_TXN_LOGDIR and <span class="caps">BOOKIE</span>_DATA_DIR should be configured to used different physical devices for best performance. <span class="caps">BOOKIE</span>_ZOOKEEPER should be a comma separated list of zookeeper servers.</p>
-<p>"/path_to_log_device/" and "/path_to_ledger_device/" are different paths. Also, port 3181 is the port that a bookie listens on for connection requests from clients. 127.0.0.1:2181 is the hostname:port for the ZooKeeper server. In this example, the standalone ZooKeeper server is running locally on port 2181. If we had multiple ZooKeeper servers, this parameter would be a comma separated list of all the hostname:port values corresponding to them. </p>
+<p>To see the default values of these configuration variables, run:</p>
+
+<p> @ bookkeeper-server/bin/bookkeeper help</p>
<h1>Setting up ZooKeeper</h1>
@@ -93,7 +101,6 @@
<li>For each bookie, we add one znode such that the name of the znode is the concatenation of the machine name and the port number that the bookie is listening on. For example, if a bookie is running on bookie.foo.com an is listening on port 3181, we add a znode <code>/ledgers/available/bookie.foo.com:3181</code> . </li>
</ol>
-
<h1>Example</h1>
<p>In the following excerpt of code, we: </p>
Modified: websites/production/zookeeper/content/bookkeeper/docs/trunk/hedwigBuild.html
==============================================================================
--- websites/production/zookeeper/content/bookkeeper/docs/trunk/hedwigBuild.html (original)
+++ websites/production/zookeeper/content/bookkeeper/docs/trunk/hedwigBuild.html Wed Nov 23 08:49:02 2011
@@ -61,57 +61,14 @@
<li><span class="caps">JDK</span> 6: <a href="http://java.sun.com/">http://java.sun.com/</a>. Ensure <code>$JAVA_HOME</code> is correctly set.</li>
<li>Maven 2: <a href="http://maven.apache.org/">http://maven.apache.org/</a>.</li>
<li>Protocol Buffers 2.3.0: <a href="http://protobuf.googlecode.com/">http://protobuf.googlecode.com/</a>.</li>
-<li>Zookeeper 3.4.0: <a href="http://hadoop.apache.org/zookeeper/">http://hadoop.apache.org/zookeeper/</a>. See below.</li>
-<li>Bookkeeper 3.4.0: <a href="http://hadoop.apache.org/zookeeper/">http://hadoop.apache.org/zookeeper/</a>. See below.</li>
+<li>Zookeeper 3.4.0: <a href="http://zookeeper.apache.org/">http://zookeeper.apache.org/</a>.</li>
</ul>
<p>Hedwig has been tested on Windows <span class="caps">XP,</span> Linux 2.6, and OS X.</p>
-<p>For the deployment and distributed support scripts in <code>hw.bash</code>:</p>
-
-<ul>
-<li>Ant: <a href="http://ant.apache.org/">http://ant.apache.org/</a>, if you want to build Zookeeper.</li>
-<li>Bash: <a href="http://www.gnu.org/software/bash/">http://www.gnu.org/software/bash/</a>.</li>
-<li>Coreutils: <a href="http://www.gnu.org/software/coreutils/">http://www.gnu.org/software/coreutils/</a>.</li>
-<li>Expect: <a href="http://expect.nist.gov/">http://expect.nist.gov/</a>, if you want <code>unbuffer</code>.</li>
-<li>Findutils: <a href="http://www.gnu.org/software/findutils/">http://www.gnu.org/software/findutils/</a>.</li>
-<li>OpenSSH: <a href="http://www.openssh.com/">http://www.openssh.com/</a>.</li>
-<li>Python 2.6: <a href="http://python.org/">http://python.org/</a>.</li>
-</ul>
-
-<h2>Protocol Buffers</h2>
-
-<p>Hedwig requires the use of the Java runtime libraries of Protocol Buffers 2.3.0. These libraries need to be installed into your local maven repository. (Maven allows multiple versions to be installed.) To install protocol buffels to your local repository, you have to download the tarball and follow the <span class="caps">README.</span>txt instructions. Note that you must first install the C++ package which contains the compiler (protoc) before you can build the java libraries. That will install the library jar's in the local maven repository where Hedwig is currently configured to point to.</p>
-
-<h2>Zookeeper and Bookkeeper</h2>
-
-<p>Hedwig currently requires the version of Bookkeeper maintained in Apache's current trunk <span class="caps">SVN </span>respository (version 3.4.0). This is not a released version yet but certain features needed for BookKeeper are only available there.</p>
-
-<p>Hedwig also depends on ZK testing code for its own testing code.</p>
-
-<p>Since Hedwig is a Maven project, all these dependencies must be made available as Maven artifacts. However, neither ZK nor BK are currently Mavenized. Hedwig provides some bash scripts to ease the installation of <span class="caps">ZK,</span> ZK tests, and <span class="caps">BK, </span>all as Maven artifacts.</p>
-
-<p>Currently, we have included the necessary ZooKeeper and BookKeeper jars in the Hedwig source itself in the $HEDWIG_DIR/server/lib directory. There is no need to retrieve them directly from the Apache download site as they are non-released trunk versions.</p>
-
-<h1>Not relevant right now since we already have the ZK jars already in the Hedwig source.</h1>
-
-<p>To fetch and build ZK 3.4.0 (and its tests) in the current directory, run:</p>
-
-<p>$HEDWIG_DIR/scripts/hw.bash get-zk</p>
-
-<h1>Not relevant right now, but when we start using the apache version of <span class="caps">BK, </span>to</h1>
-
-<p>build the local version of BK:</p>
-
-<p>$HEDWIG_DIR/scripts/hw.bash get-bk</p>
-
-<p>The $HEDWIG_DIR/server/lib directory contains all of the the class and source jars for <span class="caps">ZK,</span> ZK tests, and <span class="caps">BK.</span> To install these, go to that directory and run the following command to install them into your local maven repository:</p>
-
-<p>$HEDWIG_DIR/scripts/hw.bash install-zk-bk</p>
-
<h1>Command-Line Instructions</h1>
-<p>From the main Hedwig directory, run <code>mvn package</code>. This will produce the executable jars for both the client and server, as well as a server "assembly jar" containing all dependencies as well for easier deployment.</p>
+<p>From the top level bookkeeper directory, run <code>mvn package</code>. This will compile and package the jars necessary for running hedwig. </p>
<p>See the User's Guide for instructions on running and usage.</p>
@@ -130,26 +87,6 @@
</ol>
<p>You are now ready to run and debug the client and server code. See the User's Guide for instructions on running and usage.</p>
-
-<h1>Utilities</h1>
-
-<h2>Removing Conflicting Files in Jars</h2>
-
-<p>The Maven assembly plugin that produces the fat assembly jar may end up putting into the jar files with the same conflicting paths from multiple dependencies. This makes working with the files from certain tools (like <code>jar</code>) a bit jarring. In our case, these files are not things like class files, but rather <span class="caps">README </span>and <span class="caps">LICENSE </span>files, so we can safely remove conflicts by choosing an arbitrary winner. To do so, run:</p>
-
-<p>$HEDWIG_DIR/scripts/hw.bash strip-jar</p>
-
-<h2>Adjusting Logging</h2>
-
-<p>The logging level is something that is baked into the jar in the <code>log4j.properties</code> resource. However, it would be wasteful to go through a Maven build cycle to update and adjust this. If you're working from a source tree, it's also annoying to have to edit a source file to adjust the logging.</p>
-
-<p>We have a little script for tweaking the logging level. After running <code>strip-jar</code>, run:</p>
-
-<p>$HEDWIG_DIR/scripts/hw.bash set-logging <span class="caps">WARN</span></p>
-
-<p>To see what the current logging level is:</p>
-
-<p>$HEDWIG_DIR/scripts/hw.bash get-logging</p>
</div>
</td>
<td valign="top">
Modified: websites/production/zookeeper/content/bookkeeper/docs/trunk/hedwigDesign.html
==============================================================================
--- websites/production/zookeeper/content/bookkeeper/docs/trunk/hedwigDesign.html (original)
+++ websites/production/zookeeper/content/bookkeeper/docs/trunk/hedwigDesign.html Wed Nov 23 08:49:02 2011
@@ -75,133 +75,6 @@
<p>A pipeline implements the intercepting filter pattern. A pipeline is a sequence of handlers. Whenever a packet is read from the wire, it travels up the stream, stopping at each handler that can handle upstream events. Vice-versa for writes. Between each filter, control flows back through the centralized pipeline, and a linked list of contexts keeps track of where we are in the pipeline (one context object per handler).</p>
-<h1>Distributed Performance Evaluation</h1>
-
-<p>We've included some scripts to repeatedly run varying configurations of Hedwig on a distributed testbed and collect the resulting data. The experiments use the <code>org.apache.hedwig.client.App</code> client application and are driven by <code>scripts/hw.bash</code> (via the <code>app</code> command).</p>
-
-<p>Currently, we have two types of experiments: subscription benchmarks and publishing benchmarks.</p>
-
-<h2>Subscription Benchmarks</h2>
-
-<p>The subscription benchmark compares synchronous and asynchronous subscriptions. Because the synchronicity of subscriptions is a server configuration parameter, the servers must be restarted to change this. The benchmarks varies the maximum number of outstanding subscription requests.</p>
-
-<p>To run the subscription benchmark with wilbur6 as the subscriber and wilbur1 as its default hub:</p>
-
-<p>hosts=wilbur6 scripts/hw.bash sub-exp wilbur1</p>
-
-<p>This produces log files into the <code>sub</code> directory, which may then be analyzed using the analysis scripts.</p>
-
-<h2>Publishing Benchmarks</h2>
-
-<p>The publishing benchmark measures the throughput and latency of publishing messages within a <span class="caps">LAN </span>and across a <span class="caps">WAN.</span> It varies the following parameters:</p>
-
-<ul>
-<li>maximum number of outstanding publish requests</li>
-<li>number of publishers</li>
-<li>number of (local) receivers</li>
-</ul>
-
-<p>We vary each dimension separately (and have default settings) to avoid a combinatorial explosion in the number of configurations to explore.</p>
-
-<p>First, start a (standalone) instance:</p>
-
-<p>scripts/hw.bash start-region '' $hwhost $zkhost $bk1host $bk2host $bk3host</p>
-
-<p>To run this over <code>$host1</code> through <code>$host3</code>, with the number of publishers/subscribers varying linearly over this set:</p>
-
-<p>npars="20 40 60 80 100" scripts/hw.bash pub-exps "$host1 $host2 $host3" $hwhost $zkhost</p>
-
-<p>This will vary the number of outstanding publish requests as specified in <code>npars</code>.</p>
-
-<p>You may also optionally run this experiment with a second subscribing region:</p>
-
-<p>scripts/hw.bash start-zk-bks $zkhost $bk1host $bk2host <span class="math">bk3host npars="..." scripts/hw.bash pub-exps "</math>host1 $host2 $host3" $hwhost $zkhost $rrecv $rhwhost $rzkhost</p>
-
-<p>where the final three extra arguments specify the client receiver, Hedwig, and Zookeeper hosts, in that order.</p>
-
-<p>This command will produce files into <code>./pub/</code>, which can then be process using <code>analyze.py</code>.</p>
-
-<h1>Analysis and Visualization</h1>
-
-<p><code>scripts/analyze.py</code> produces plots from the collected experimental data. It has just a few immediate dependencies. In the following, the indentation signifies nested dependencies, like an upside-down tree:</p>
-
-<pre><code> component AAA that component AA requires
- component AAB that component AA requires
-component AA that component A requires
- component ABA that component AB requires
- component ABB that component AB requires
-component AB that component A requires</code></pre>
-
-
-<p>component A that analysis tools depend on component <span class="caps">BAA </span>that component BA requires component <span class="caps">BAB </span>that component BA requires component BA that component B requires component <span class="caps">BBA </span>that component BB requires component <span class="caps">BBB </span>that component BB requires component BB that component B requires component B that analysis tools depend on</p>
-
-<p>The reason the tree is upside-down is so that you can treat this whole thing as a chunk of bash script.</p>
-
-<p><a href="http://toastball.net/toast/">toast</a> is a utility that makes it a breeze to install all this software, but you do need to make sure your environment is set up correctly (e.g. <code>PKG_CONFIG_PATH</code> must point to <code>~/.toast/armed/lib/pkgconfig/</code>).</p>
-
-<p>Setup:</p>
-
-<p>wget <del>O</del> http://toastball.net/toast/toast|perl -x - arm toast</p>
-
-<p>toast arm "http://www.python.org/ftp/python/2.6.2/Python-2.6.2.tar.bz2"</p>
-
-<p>toast arm numpy</p>
-
-
-
-<pre>
- toast arm libpng
-
- toast arm pixman
-
- toast arm freetype
-
- toast arm 'ftp://xmlsoft.org/libxml2/libxml2-2.7.3.tar.gz'
-
- toast arm fontconfig
-
- toast arm cairo
-
-toast arm pycairo
-</pre>
-
-
-
-<p>hg clone https://yang@bitbucket.org/yang/pycha/ pycha/setup.bash -d -p $path_to_install_to</p>
-
-<p>svn co https://assorted.svn.sourceforge.net/svnroot/assorted/python-commons/trunk/ python-commons/ python-commons/setup.bash -d -p $path_to_install_to</p>
-
-<p>To analyze the publishing experiments, change to the <code>pub</code> data directory and run:</p>
-
-<p>scripts/analyze.py pub</p>
-
-<p>To analyze the subscription experiments, change to the <code>sub</code> data directory and run:</p>
-
-<p>scripts/analyze.py sub</p>
-
-<h1>Debugging</h1>
-
-<p>You can attach an Eclipse debugger (or any debugger) to a Java process running on a remote host, as long as it has been started with the appropriate <span class="caps">JVM </span>flags. (See the Building Hedwig document to set up your Eclipse environment.) To launch something using <code>hw.bash</code> with debugger attachment enabled, prefix the command with <code>attach=true</code>, e.g.:</p>
-
-<p>attach=true scripts/hw.bash start-regions myregions.cfg</p>
-
-<h1>Profiling</h1>
-
-<p>The scripts we have provided include ways for you to launch with YourKit profiling enabled.</p>
-
-<p>To deploy YourKit onto a number of machines:</p>
-
-<p>hosts="..." scripts/hw.bash setup-yjp $path_to_yjp</p>
-
-<p>where the path points to the <a href="http://www.yourkit.com/download/yjp-8.0.15.zip">YourKit Linux zip archive</a> (which is freely available and doesn't require any license to use).</p>
-
-<p>Now when using the scripts to run distributed experiments, to profile anything with YourKit, prefix the command with <code>use_yjp=true</code>. E.g.:</p>
-
-<p>use_yjp=true scripts/hw.bash start-regions regions.cfg</p>
-
-<p>Now you may start on your local machine the YourKit <span class="caps">GUI </span>and connect to the hosts that you're interested in.</p>
-
-<p>Note that you may want to disable the default set of filters in YourKit.</p>
<h1>Pseudocode</h1>
Modified: websites/production/zookeeper/content/bookkeeper/docs/trunk/hedwigUser.html
==============================================================================
--- websites/production/zookeeper/content/bookkeeper/docs/trunk/hedwigUser.html (original)
+++ websites/production/zookeeper/content/bookkeeper/docs/trunk/hedwigUser.html Wed Nov 23 08:49:02 2011
@@ -75,140 +75,33 @@
<p>The client is a library intended to be consumed by user applications. It takes a Commons Configuration object, for which the source/documentation is in <code>org.apache.hedwig.client.conf.ClientConfiguration</code>.</p>
-<p>We have provided a simple client application, <code>org.apache.hedwig.client.App</code>, that can drive a number of benchmarks. This also takes a single configuration file argument, which is fed to the client library.</p>
-
-<p>We've provided a number of scripts to faciliate running servers and clients in a variety of configurations, including over distributed hosts. These are all consolidated in <code>scripts/hw.bash</code>. Although the snippets in this documentation run the script from the hedwig main directory, you can run it from any location. Apologies in advance for these being bash scripts; time permitting, a more robust and maintainable support/tooling infrastructure would be ideal.</p>
-
<h1>Deployment</h1>
-<p>When ssh-ing into a new host, you are requested to verify and accept the host key. In order to automatically accept the host keys for many new hosts (dangerous), use:</p>
-
-<p>hosts="$host1 $host2 ..." scripts/hw.bash warmup</p>
-
-<p>The <code>hosts</code> variable is set here to the list of hosts that you would like to warm up.</p>
-
-<p>To set up <span class="caps">JDK6 </span>on some hosts, use:</p>
-
-<p>hosts="..." scripts/hw.bash setup-java $path_to_modified_jdk6</p>
-
-<p>The argument must point to a <span class="caps">JDK6 </span>binary self-extracting executable, but with the <code>more</code> command that displays the License agreement replaced with <code>cat</code>. Unfortunately, this step must be performed manually. This script will extract the <span class="caps">JDK </span>directly into the home directory and update <code>$PATH</code> in <code>~/.bashrc</code> (in an idempotent fashion).</p>
-
-<p>Because the current implementation uses a single socket per subscription, the Hedwig launching scripts all require a high <code>ulimit</code> on the number of open file descriptors. Non-root users can only use up to the limit specified in <code>/etc/security/limits.conf</code>; to raise this to 1024^2, run:</p>
-
-<p>hosts="..." scripts/hw.bash setup-limits</p>
-
-<p>This uses <code>ssh</code> so that you need to enter your password for <code>sudo</code> just once.</p>
-
-<p>For most of the commands presented in the next section, you may prefix the command with:</p>
-
-<p>push_jar=true ...</p>
-
-<p>to first push the assembly jar (assumed to be available in <code>server/target/</code>) to all hosts.</p>
-
-<h1>Running Servers</h1>
-
-<p>To start three BK bookies on ports 3181-3183 on localhost (directories must all exist):</p>
-
-<p>scripts/hw.bash bk 3181 $bk1_journal_dir $bk1_ledger_dir & scripts/hw.bash bk 3182 $bk2_journal_dir $bk2_ledger_dir & scripts/hw.bash bk 3183 $bk3_journal_dir $bk3_ledger_dir &</p>
-
-<p>To start a ZK on port 2181 (directory must exist):</p>
-
-<p>scripts/hw.bash zk 2181 /path/for/zk/ &</p>
-
-<p>To register the BKs with the ZK (so that Hedwig knows where to find the bookies):</p>
-
-<p>scripts/hw.bash setup-bk localhost:2181 <code>hostname</code>:3181 <code>hostname</code>:3182 <code>hostname</code>:3183</p>
-
-<p>Everything up to this point may be done using a single command over a set of hosts, with ZK on port 9877 and BK on port 9878. The following function takes 2 arguments. The first is the ZK host. The second is a string list of BK hosts:</p>
-
-<p>scripts/hw.bash start-zk-bks <span class="math">zkhost "</math>bk1host $bk2host $bk3host ..."</p>
-
-<p>Note that the hosts may be <span class="caps">SSH </span>profile aliases in your <code>~/.ssh/config</code>; the script will parse this file and look up their hostnames where necessary. This applies for the hosts specified in the other commands.</p>
-
-<p>Also, the scripts use the <code>bk-journal</code> and <code>bk-ledger</code> functions in <code>hw.bash</code> to determine where to place the BK journal and ledger, given a hostname.</p>
-
-<p>To start a Hedwig server locally:</p>
-
-<p>scripts/hw.bash hw server.conf &</p>
-
-<p>To start Hedwig servers on some hosts "$hw1host $hw2host $hw3host ..." on port 9876, using $zkhost as the ZK server:</p>
-
-<p>scripts/hw.bash start-hw '' "$hw1host $hw2host $hw3host ..." $zkhost</p>
-
-<p>Above, the first empty string argument is the list of default servers to each of the other regions. You may run multiple connected instances of Hedwig this way.<br />
-E.g., to start three regions each with a single Hedwig hub that talk to each other, and using the hw.bash default server ports of 9875 (non-SSL) and 9876 (SSL):</p>
-
-<p>scripts/hw.bash start-hw "$hw2host:9875:9876 <span class="math">hw3host:9875:9876" "</math>hw1host" <span class="math">zk1host scripts/hw.bash start-hw "</math>hw1host:9875:9876 <span class="math">hw3host:9875:9876" "</math>hw2host" <span class="math">zk2host scripts/hw.bash start-hw "</math>hw1host:9875:9876 <span class="math">hw2host:9875:9876" "</math>hw3host" $zk3host</p>
-
-<p>Everything up to this point may be done using a single command over a set of hosts:</p>
-
-<p>scripts/hw.bash start-region '' "$hw1host $hw2host $hw3host ..." <span class="math">zkhost "</math>bk1host $bk2host $bk3host ..."</p>
-
-<p>The first three arguments are the same as for <code>start-hw</code>.</p>
-
-<p>You may start multiple regions as well:</p>
-
-<p>scripts/hw.bash start-regions regions.cfg</p>
-
-<p>"regions.cfg" is a list of all regions, one per line, with each region having the following format:</p>
-
-<p>region=<Region name>, hub=<list of hub servers>, default=<single hub server>, zk=<single ZK server>, bk=<list of BK servers></p>
-
-<p>This will create all of the regions with an all-to-all topology. Each region is connected to the default hub server of every other region. The ", " delimiter is used to separate out the different parts of a region along with the hard-coded parameter names. There also needs to be a newline after the last region line. Here is an example file specifying three regions:</p>
-
-<p>region=wilbur, hub=wilbur90 wilbur91, default=wilbur90, zk=wilbur93, bk=wilbur93 wilbur94 wilbur95 region=re1, hub=sherpa7 sherpa8, default=sherpa7, zk=sherpa9, bk=sherpa9 sherpa10 sherpa11 region=peanuts, hub=peanuts1 peanuts2, default=peanuts2, zk=peanuts3, bk=peanuts3 peanuts4 peanuts5</p>
-
-<h1>Running the Client</h1>
-
-<p>To run the test client:</p>
-
-<p><span class="caps">JAVAFLAGS</span>="..." scripts/hw.bash hwc $conf_path</p>
-
-<p>where <code>$conf_path</code> is a client configuration file.</p>
-
-<p>To run the test client on some other hosts:</p>
-
-<p>hosts="..." <span class="caps">JAVAFLAGS</span>="..." scripts/hw.bash app $hwhost</p>
-
-<p>This will generate a simple configuration file assuming $hwhost is listening on the default <span class="caps">SSL </span>and non-SSL ports which are specified as global variables in hw.bash. Currently these are 9875 for non-SSL and 9876 for <span class="caps">SSL.</span></p>
-
-<p>Client usage is currently documented in the source. To run a subscription benchmark, set <code>JAVAFLAGS</code> to:</p>
-
-<p>-Dmode=sub -Dcount=10000 -Dnpar=100 -Dstart=5 -Dnwarmups=30</p>
-
-<p>This will first create 30 warm-up subscriptions to topics "warmup-5" through "warmup-34", then 10,000 benchmarked subscriptions to topics "topic-5" through "topic-10,004". It will have a pipeline depth of 100 requests, meaning that there will be at most 100 outstanding (unresponded) messages in flight at any moment.</p>
-
-<p>To run a publishing benchmark, set <code>JAVAFLAGS</code> to:</p>
-
-<p>-Dmode=pub -Dcount=10000 -Dnpar=100 -Dstart=5</p>
-
-<p>This will publish 10,000 messages to topic "topic-5", with a pipeline depth of 100 requests.</p>
-
-<p>At the end, the programs will print throughput and latency information.</p>
+<h2>Limits</h2>
-<h1>Utilities</h1>
+<p>Because the current implementation uses a single socket per subscription, the Hedwig requires a high <code>ulimit</code> on the number of open file descriptors. Non-root users can only use up to the limit specified in <code>/etc/security/limits.conf</code>; to raise this to 1024^2, as root, modify the "nofile" line in /etc/security/limits.conf on all hubs.</p>
-<p>To kill all the user's Java processes running on some machines, use:</p>
+<h2>Running Servers</h2>
-<p>hosts="..." scripts/hw.bash dkill</p>
+<p>Hedwig requires BookKeeper to run. For BookKeeper setup instructions see <a href="./bookkeeperStarted.html">BookKeeper Getting Started</a>.</p>
-<p>To check if any processes are running and are using ports of interest (esp. 9876-9878):</p>
+<p>To start a Hedwig hub server:</p>
-<p>hosts="..." scripts/hw.bash dstatus</p>
+<p> @ hedwig-server/bin/hedwig server</p>
-<p>Add an argument to <code>dstatus</code> (may be anything) to get a more detailed listing.</p>
+<p>Hedwig takes its configuration from hedwig-server/conf/hw_server.conf by default. To change location of the conf file, modify the <span class="caps">HEDWIG</span>_SERVER_CONF environment variable.</p>
-<p>To check if there's anything consuming the <span class="caps">CPU </span>on some machines:</p>
+<h1>Debugging</h1>
-<p>hosts="..." scripts/hw.bash tops</p>
+<p>You can attach an Eclipse debugger (or any debugger) to a Java process running on a remote host, as long as it has been started with the appropriate <span class="caps">JVM </span>flags. (See the Building Hedwig document to set up your Eclipse environment.) To launch something using <code>bin/hedwig</code> with debugger attachment enabled, prefix the command with <code>HEDWIG_EXTRA_OPTS=-agentlib:jdwp=transport=dt_socket,server=y,address=5000</code>, e.g.:</p>
-<p>To run an arbitrary command on multiple hosts in parallel:</p>
+<p>@ <span class="caps">HEDWIG</span>_EXTRA_OPTS=-agentlib:jdwp=transport=dt_socket,server=y,address=5000 hedwig-server/bin/hedwig server</p>
-<p>hosts="..." scripts/hw.bash parssh $command</p>
+<h1>Logging</h1>
-<p>To do this in sequence:</p>
+<p>To control the logging in Hedwig server, set <span class="caps">HEDWIG</span>_LOG_CONF to the location of a log4j.properties file.</p>
-<p>hosts="..." xargs= scripts/hw.bash parssh $command</p>
+<p>@ <span class="caps">HEDWIG</span>_LOG_CONF=../hw_testing/log4j.properties hedwig-server/bin/hedwig server</p>
</div>
</td>
<td valign="top">
Modified: websites/production/zookeeper/content/bylaws.html
==============================================================================
--- websites/production/zookeeper/content/bylaws.html (original)
+++ websites/production/zookeeper/content/bylaws.html Wed Nov 23 08:49:02 2011
@@ -302,6 +302,7 @@ actions will also be referred to the <sp
<h3>Documentation</h3>
<ul>
+<li><a href="./doc/r3.4.0">Release 3.4.0</a></li>
<li><a href="./doc/r3.3.3" title="current">Release 3.3.3</a></li>
<li><a href="./doc/r3.3.2">Release 3.3.2</a></li>
<li><a href="./doc/r3.2.2">Release 3.2.2</a></li>
Modified: websites/production/zookeeper/content/credits.html
==============================================================================
--- websites/production/zookeeper/content/credits.html (original)
+++ websites/production/zookeeper/content/credits.html Wed Nov 23 08:49:02 2011
@@ -114,6 +114,7 @@
<h3>Documentation</h3>
<ul>
+<li><a href="./doc/r3.4.0">Release 3.4.0</a></li>
<li><a href="./doc/r3.3.3" title="current">Release 3.3.3</a></li>
<li><a href="./doc/r3.3.2">Release 3.3.2</a></li>
<li><a href="./doc/r3.2.2">Release 3.2.2</a></li>
Modified: websites/production/zookeeper/content/doc/trunk/bookkeeperConfig.pdf
==============================================================================
Files websites/production/zookeeper/content/doc/trunk/bookkeeperConfig.pdf (original) and websites/production/zookeeper/content/doc/trunk/bookkeeperConfig.pdf Wed Nov 23 08:49:02 2011 differ
Modified: websites/production/zookeeper/content/doc/trunk/bookkeeperOverview.pdf
==============================================================================
Files websites/production/zookeeper/content/doc/trunk/bookkeeperOverview.pdf (original) and websites/production/zookeeper/content/doc/trunk/bookkeeperOverview.pdf Wed Nov 23 08:49:02 2011 differ
Modified: websites/production/zookeeper/content/doc/trunk/bookkeeperProgrammer.pdf
==============================================================================
Files websites/production/zookeeper/content/doc/trunk/bookkeeperProgrammer.pdf (original) and websites/production/zookeeper/content/doc/trunk/bookkeeperProgrammer.pdf Wed Nov 23 08:49:02 2011 differ
Modified: websites/production/zookeeper/content/doc/trunk/bookkeeperStarted.pdf
==============================================================================
Files websites/production/zookeeper/content/doc/trunk/bookkeeperStarted.pdf (original) and websites/production/zookeeper/content/doc/trunk/bookkeeperStarted.pdf Wed Nov 23 08:49:02 2011 differ
Modified: websites/production/zookeeper/content/doc/trunk/bookkeeperStream.pdf
==============================================================================
Files websites/production/zookeeper/content/doc/trunk/bookkeeperStream.pdf (original) and websites/production/zookeeper/content/doc/trunk/bookkeeperStream.pdf Wed Nov 23 08:49:02 2011 differ
Modified: websites/production/zookeeper/content/doc/trunk/index.pdf
==============================================================================
Files websites/production/zookeeper/content/doc/trunk/index.pdf (original) and websites/production/zookeeper/content/doc/trunk/index.pdf Wed Nov 23 08:49:02 2011 differ
Modified: websites/production/zookeeper/content/doc/trunk/javaExample.pdf
==============================================================================
Files websites/production/zookeeper/content/doc/trunk/javaExample.pdf (original) and websites/production/zookeeper/content/doc/trunk/javaExample.pdf Wed Nov 23 08:49:02 2011 differ
Modified: websites/production/zookeeper/content/doc/trunk/linkmap.pdf
==============================================================================
Files websites/production/zookeeper/content/doc/trunk/linkmap.pdf (original) and websites/production/zookeeper/content/doc/trunk/linkmap.pdf Wed Nov 23 08:49:02 2011 differ
Modified: websites/production/zookeeper/content/doc/trunk/recipes.html
==============================================================================
--- websites/production/zookeeper/content/doc/trunk/recipes.html (original)
+++ websites/production/zookeeper/content/doc/trunk/recipes.html Wed Nov 23 08:49:02 2011
@@ -454,7 +454,7 @@ document.write("Last Published: " + docu
<li>
<p>if <em>p</em> is the lowest process
- node in L, wait on highest process node in P</p>
+ node in L, wait on highest process node in L</p>
</li>
Modified: websites/production/zookeeper/content/doc/trunk/recipes.pdf
==============================================================================
Files websites/production/zookeeper/content/doc/trunk/recipes.pdf (original) and websites/production/zookeeper/content/doc/trunk/recipes.pdf Wed Nov 23 08:49:02 2011 differ
Modified: websites/production/zookeeper/content/doc/trunk/releasenotes.pdf
==============================================================================
Files websites/production/zookeeper/content/doc/trunk/releasenotes.pdf (original) and websites/production/zookeeper/content/doc/trunk/releasenotes.pdf Wed Nov 23 08:49:02 2011 differ
Modified: websites/production/zookeeper/content/doc/trunk/zookeeperAdmin.html
==============================================================================
--- websites/production/zookeeper/content/doc/trunk/zookeeperAdmin.html (original)
+++ websites/production/zookeeper/content/doc/trunk/zookeeperAdmin.html Wed Nov 23 08:49:02 2011
@@ -874,10 +874,11 @@ server.3=zoo3:2888:3888</pre>
logs.
</p>
<p>A ZooKeeper server <strong>will not remove
- old snapshots and log files</strong>, this is the
- responsibility of the operator. Every serving environment is
- different and therefore the requirements of managing these
- files may differ from install to install (backup for example).
+ old snapshots and log files</strong> when using the default
+ configuration (see autopurge below), this is the
+ responsibility of the operator. Every serving environment is
+ different and therefore the requirements of managing these
+ files may differ from install to install (backup for example).
</p>
<p>The PurgeTxnLog utility implements a simple retention
policy that administrators can use. The <a href="api/index.html">API docs</a> contains details on
@@ -891,6 +892,11 @@ server.3=zoo3:2888:3888</pre>
can be run as a cron job on the ZooKeeper server machines to
clean up the logs daily.</p>
<pre class="code"> java -cp zookeeper.jar:lib/slf4j-api-1.6.1.jar:lib/slf4j-log4j12-1.6.1.jar:lib/log4j-1.2.15.jar:conf org.apache.zookeeper.server.PurgeTxnLog <dataDir> <snapDir> -n <count></pre>
+<p>Automatic purging of the snapshots and corresponding
+ transaction logs was introduced in version 3.4.0 and can be
+ enabled via the following configuration parameters <strong>autopurge.snapRetainCount</strong> and <strong>autopurge.purgeInterval</strong>. For more on
+ this, see <a href="#sc_advancedConfiguration">Advanced Configuration</a>
+ below.</p>
<a name="Debug+Log+Cleanup+%28log4j%29"></a>
<h4>Debug Log Cleanup (log4j)</h4>
<p>See the section on <a href="#sc_logging">logging</a> in this document. It is
@@ -1148,6 +1154,34 @@ server.3=zoo3:2888:3888</pre>
will allow the client to negotiate. Defaults to 20 times
the <strong>tickTime</strong>.</p>
</dd>
+
+
+<dt>
+<term>autopurge.snapRetainCount</term>
+</dt>
+<dd>
+<p>(No Java system property)</p>
+<p>
+<strong>New in 3.4.0:</strong>
+ When enabled, ZooKeeper auto purge feature retains
+ the <strong>autopurge.snapRetainCount</strong> most
+ recent snapshots and the corresponding transaction logs in the
+ <strong>dataDir</strong> and <strong>dataLogDir</strong> respectively and deletes the rest.
+ Defaults to 3. Minimum value is 3.</p>
+</dd>
+
+
+<dt>
+<term>autopurge.purgeInterval</term>
+</dt>
+<dd>
+<p>(No Java system property)</p>
+<p>
+<strong>New in 3.4.0:</strong> The
+ time interval in hours for which the purge task has to
+ be triggered. Set to a positive integer (1 and above)
+ to enable the auto purging. Defaults to 0.</p>
+</dd>
</dl>
<a name="sc_clusterOptions"></a>
@@ -1171,13 +1205,10 @@ server.3=zoo3:2888:3888</pre>
<div class="label">Note</div>
<div class="content">
-<p> The implementations of leader election
- 1 and 2 are currently not supported, and we have the intention
- of deprecating them in the near future. Implementations 0 and 3 are
- currently supported, and we plan to keep supporting them in the near future.
- To avoid having to support multiple versions of leader election unecessarily,
- we may eventually consider deprecating algorithm 0 as well, but we will plan
- according to the needs of the community.
+<p> The implementations of leader election 0, 1, and 2 are now
+ <strong> deprecated </strong>. We have the intention
+ of removing them in the next release, at which point only the
+ FastLeaderElection will be available.
</p>
</div>
Modified: websites/production/zookeeper/content/doc/trunk/zookeeperAdmin.pdf
==============================================================================
Files websites/production/zookeeper/content/doc/trunk/zookeeperAdmin.pdf (original) and websites/production/zookeeper/content/doc/trunk/zookeeperAdmin.pdf Wed Nov 23 08:49:02 2011 differ
Modified: websites/production/zookeeper/content/doc/trunk/zookeeperHierarchicalQuorums.pdf
==============================================================================
Files websites/production/zookeeper/content/doc/trunk/zookeeperHierarchicalQuorums.pdf (original) and websites/production/zookeeper/content/doc/trunk/zookeeperHierarchicalQuorums.pdf Wed Nov 23 08:49:02 2011 differ
Modified: websites/production/zookeeper/content/doc/trunk/zookeeperInternals.pdf
==============================================================================
Files websites/production/zookeeper/content/doc/trunk/zookeeperInternals.pdf (original) and websites/production/zookeeper/content/doc/trunk/zookeeperInternals.pdf Wed Nov 23 08:49:02 2011 differ
Modified: websites/production/zookeeper/content/doc/trunk/zookeeperJMX.pdf
==============================================================================
Files websites/production/zookeeper/content/doc/trunk/zookeeperJMX.pdf (original) and websites/production/zookeeper/content/doc/trunk/zookeeperJMX.pdf Wed Nov 23 08:49:02 2011 differ
Modified: websites/production/zookeeper/content/doc/trunk/zookeeperObservers.pdf
==============================================================================
Files websites/production/zookeeper/content/doc/trunk/zookeeperObservers.pdf (original) and websites/production/zookeeper/content/doc/trunk/zookeeperObservers.pdf Wed Nov 23 08:49:02 2011 differ
Modified: websites/production/zookeeper/content/doc/trunk/zookeeperOver.pdf
==============================================================================
Files websites/production/zookeeper/content/doc/trunk/zookeeperOver.pdf (original) and websites/production/zookeeper/content/doc/trunk/zookeeperOver.pdf Wed Nov 23 08:49:02 2011 differ
Modified: websites/production/zookeeper/content/doc/trunk/zookeeperProgrammers.pdf
==============================================================================
Files websites/production/zookeeper/content/doc/trunk/zookeeperProgrammers.pdf (original) and websites/production/zookeeper/content/doc/trunk/zookeeperProgrammers.pdf Wed Nov 23 08:49:02 2011 differ
Modified: websites/production/zookeeper/content/doc/trunk/zookeeperQuotas.pdf
==============================================================================
Files websites/production/zookeeper/content/doc/trunk/zookeeperQuotas.pdf (original) and websites/production/zookeeper/content/doc/trunk/zookeeperQuotas.pdf Wed Nov 23 08:49:02 2011 differ
Modified: websites/production/zookeeper/content/doc/trunk/zookeeperStarted.pdf
==============================================================================
Files websites/production/zookeeper/content/doc/trunk/zookeeperStarted.pdf (original) and websites/production/zookeeper/content/doc/trunk/zookeeperStarted.pdf Wed Nov 23 08:49:02 2011 differ
Modified: websites/production/zookeeper/content/doc/trunk/zookeeperTutorial.pdf
==============================================================================
Files websites/production/zookeeper/content/doc/trunk/zookeeperTutorial.pdf (original) and websites/production/zookeeper/content/doc/trunk/zookeeperTutorial.pdf Wed Nov 23 08:49:02 2011 differ
Modified: websites/production/zookeeper/content/index.html
==============================================================================
--- websites/production/zookeeper/content/index.html (original)
+++ websites/production/zookeeper/content/index.html Wed Nov 23 08:49:02 2011
@@ -120,6 +120,7 @@
<h3>Documentation</h3>
<ul>
+<li><a href="./doc/r3.4.0">Release 3.4.0</a></li>
<li><a href="./doc/r3.3.3" title="current">Release 3.3.3</a></li>
<li><a href="./doc/r3.3.2">Release 3.3.2</a></li>
<li><a href="./doc/r3.2.2">Release 3.2.2</a></li>
Modified: websites/production/zookeeper/content/irc.html
==============================================================================
--- websites/production/zookeeper/content/irc.html (original)
+++ websites/production/zookeeper/content/irc.html Wed Nov 23 08:49:02 2011
@@ -96,6 +96,7 @@
<h3>Documentation</h3>
<ul>
+<li><a href="./doc/r3.4.0">Release 3.4.0</a></li>
<li><a href="./doc/r3.3.3" title="current">Release 3.3.3</a></li>
<li><a href="./doc/r3.3.2">Release 3.3.2</a></li>
<li><a href="./doc/r3.2.2">Release 3.2.2</a></li>
Modified: websites/production/zookeeper/content/lists.html
==============================================================================
--- websites/production/zookeeper/content/lists.html (original)
+++ websites/production/zookeeper/content/lists.html Wed Nov 23 08:49:02 2011
@@ -136,6 +136,7 @@ In order to post to the list, it is nece
<h3>Documentation</h3>
<ul>
+<li><a href="./doc/r3.4.0">Release 3.4.0</a></li>
<li><a href="./doc/r3.3.3" title="current">Release 3.3.3</a></li>
<li><a href="./doc/r3.3.2">Release 3.3.2</a></li>
<li><a href="./doc/r3.2.2">Release 3.2.2</a></li>
Modified: websites/production/zookeeper/content/privacy.html
==============================================================================
--- websites/production/zookeeper/content/privacy.html (original)
+++ websites/production/zookeeper/content/privacy.html Wed Nov 23 08:49:02 2011
@@ -108,6 +108,7 @@
<h3>Documentation</h3>
<ul>
+<li><a href="./doc/r3.4.0">Release 3.4.0</a></li>
<li><a href="./doc/r3.3.3" title="current">Release 3.3.3</a></li>
<li><a href="./doc/r3.3.2">Release 3.3.2</a></li>
<li><a href="./doc/r3.2.2">Release 3.2.2</a></li>
Modified: websites/production/zookeeper/content/releases.html
==============================================================================
--- websites/production/zookeeper/content/releases.html (original)
+++ websites/production/zookeeper/content/releases.html Wed Nov 23 08:49:02 2011
@@ -73,6 +73,11 @@
<h2 id="news">News</h2>
+<h3>22 Nov, 2011: release 3.4.0 available</h3>
+
+<p>This a major release coming after a long time. This has many features including security, multi update api's, rpm/deb support, C native windows support. See <br />
+"ZooKeeper 3.4.0 Release Notes": http://zookeeper.apache.org/doc/r3.4.0/releasenotes.html for details. Please note that this is beta release. The current stable release is 3.3.3.</p>
+
<h3>27 Feb, 2011: release 3.3.3 available</h3>
<p>The release fixes two critical bugs that could cause data corruption. It also addresses 12 other issues. See<br />
@@ -181,6 +186,7 @@
<h3>Documentation</h3>
<ul>
+<li><a href="./doc/r3.4.0">Release 3.4.0</a></li>
<li><a href="./doc/r3.3.3" title="current">Release 3.3.3</a></li>
<li><a href="./doc/r3.3.2">Release 3.3.2</a></li>
<li><a href="./doc/r3.2.2">Release 3.2.2</a></li>
Modified: websites/production/zookeeper/content/svn.html
==============================================================================
--- websites/production/zookeeper/content/svn.html (original)
+++ websites/production/zookeeper/content/svn.html Wed Nov 23 08:49:02 2011
@@ -106,6 +106,7 @@
<h3>Documentation</h3>
<ul>
+<li><a href="./doc/r3.4.0">Release 3.4.0</a></li>
<li><a href="./doc/r3.3.3" title="current">Release 3.3.3</a></li>
<li><a href="./doc/r3.3.2">Release 3.3.2</a></li>
<li><a href="./doc/r3.2.2">Release 3.2.2</a></li>