You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@distributedlog.apache.org by si...@apache.org on 2016/09/13 07:34:37 UTC
[19/23] incubator-distributedlog git commit: DL-3: Move
distributedlog website to apache
http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/b169db04/docs/admin_guide/hardware.rst
----------------------------------------------------------------------
diff --git a/docs/admin_guide/hardware.rst b/docs/admin_guide/hardware.rst
new file mode 100644
index 0000000..ab1dd91
--- /dev/null
+++ b/docs/admin_guide/hardware.rst
@@ -0,0 +1,138 @@
+---
+layout: default
+
+# Top navigation
+top-nav-group: admin-guide
+top-nav-pos: 3
+top-nav-title: Hardware
+
+# Sub-level navigation
+sub-nav-group: admin-guide
+sub-nav-parent: admin-guide
+sub-nav-id: hardware
+sub-nav-pos: 3
+sub-nav-title: Hardware
+---
+
+.. contents:: Hardware
+
+Hardware
+========
+
+Figure 1 describes the data flow of DistributedLog. Write traffic comes to `Write Proxy`
+and the data is replicated in `RF` (replication factor) ways to `BookKeeper`. BookKeeper
+stores the replicated data and keeps the data for a given retention period. The data is
+read by `Read Proxy` and fanout to readers.
+
+In such layered architecture, each layer has its own responsibilities and different resource
+requirements. It makes the capacity and cost model much clear and users could scale
+different layers independently.
+
+.. figure:: ../images/costmodel.png
+ :align: center
+
+ Figure 1. DistributedLog Cost Model
+
+Metrics
+~~~~~~~
+
+There are different metrics measuring the capability of a service instance in each layer
+(e.g a `write proxy` node, a `bookie` storage node, a `read proxy` node and such). These metrics
+can be `rps` (requests per second), `bps` (bits per second), `number of streams` that a instance
+can support, and latency requirements. `bps` is the best and simple factor on measuring the
+capability of current distributedlog architecture.
+
+Write Proxy
+~~~~~~~~~~~
+
+Write Proxy (WP) is a stateless serving service that writes and replicates fan-in traffic into BookKeeper.
+The capability of a write proxy instance is purely dominated by the *OUTBOUND* network bandwidth,
+which is reflected as incoming `Write Throughput` and `Replication Factor`.
+
+Calculating the capacity of Write Proxy (number of instances of write proxies) is pretty straightforward.
+The formula is listed as below.
+
+::
+
+ Number of Write Proxies = (Write Throughput) * (Replication Factor) / (Write Proxy Outbound Bandwidth)
+
+As it is bandwidth bound, we'd recommend using machines that have high network bandwith (e.g 10Gb NIC).
+
+The cost estimation is also straightforward.
+
+::
+
+ Bandwidth TCO ($/day/MB) = (Write Proxy TCO) / (Write Proxy Outbound Bandwidth)
+ Cost of write proxies = (Write Throughput) * (Replication Factor) / (Bandwidth TCO)
+
+CPUs
+^^^^
+
+DistributedLog is not CPU bound. You can run an instance with 8 or 12 cores just fine.
+
+Memories
+^^^^^^^^
+
+There's a fair bit of caching. Consider running with at least 8GB of memory.
+
+Disks
+^^^^^
+
+This is a stateless process, disk performances are not relevant.
+
+Network
+^^^^^^^
+
+Depending on your throughput, you might be better off running this with 10Gb NIC. In this scenario, you can easily achieves 350MBps of writes.
+
+
+BookKeeper
+~~~~~~~~~~
+
+BookKeeper is the log segment store, which is a stateful service. There are two factors to measure the
+capability of a Bookie instance: `bandwidth` and `storage`. The bandwidth is majorly dominated by the
+outbound traffic from write proxy, which is `(Write Throughput) * (Replication Factor)`. The storage is
+majorly dominated by the traffic and also `Retention Period`.
+
+Calculating the capacity of BookKeeper (number of instances of bookies) is a bit more complicated than Write
+Proxy. The total number of instances is the maximum number of the instances of bookies calculated using
+`bandwidth` and `storage`.
+
+::
+
+ Number of bookies based on bandwidth = (Write Throughput) * (Replication Factor) / (Bookie Inbound Bandwidth)
+ Number of bookies based on storage = (Write Throughput) * (Replication Factor) * (Replication Factor) / (Bookie disk space)
+ Number of bookies = maximum((number of bookies based on bandwidth), (number of bookies based on storage))
+
+We should consider both bandwidth and storage when choosing the hardware for bookies. There are several rules to follow:
+- A bookie should have multiple disks.
+- The number of disks used as journal disks should have similar I/O bandwidth as its *INBOUND* network bandwidth. For example, if you plan to use a disk for journal which I/O bandwidth is around 100MBps, a 1Gb NIC is a better choice than 10Gb NIC.
+- The number of disks used as ledger disks should be large enough to hold data if retention period is typical long.
+
+The cost estimation is straightforward based on the number of bookies estimated above.
+
+::
+
+ Cost of bookies = (Number of bookies) * (Bookie TCO)
+
+Read Proxy
+~~~~~~~~~~
+
+Similar as Write Proxy, Read Proxy is also dominated by *OUTBOUND* bandwidth, which is reflected as incoming `Write Throughput` and `Fanout Factor`.
+
+Calculating the capacity of Read Proxy (number of instances of read proxies) is also pretty straightforward.
+The formula is listed as below.
+
+::
+
+ Number of Read Proxies = (Write Throughput) * (Fanout Factor) / (Read Proxy Outbound Bandwidth)
+
+As it is bandwidth bound, we'd recommend using machines that have high network bandwith (e.g 10Gb NIC).
+
+The cost estimation is also straightforward.
+
+::
+
+ Bandwidth TCO ($/day/MB) = (Read Proxy TCO) / (Read Proxy Outbound Bandwidth)
+ Cost of read proxies = (Write Throughput) * (Fanout Factor) / (Bandwidth TCO)
+
http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/b169db04/docs/admin_guide/main.rst
----------------------------------------------------------------------
diff --git a/docs/admin_guide/main.rst b/docs/admin_guide/main.rst
new file mode 100644
index 0000000..5fb7a3a
--- /dev/null
+++ b/docs/admin_guide/main.rst
@@ -0,0 +1,13 @@
+---
+title: "Admin Guide"
+layout: guide
+
+# Sub-level navigation
+sub-nav-group: admin-guide
+sub-nav-id: admin-guide
+sub-nav-parent: _root_
+sub-nav-group-title: Admin Guide
+sub-nav-pos: 1
+sub-nav-title: Admin Guide
+
+---
http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/b169db04/docs/admin_guide/monitoring.rst
----------------------------------------------------------------------
diff --git a/docs/admin_guide/monitoring.rst b/docs/admin_guide/monitoring.rst
new file mode 100644
index 0000000..59645c1
--- /dev/null
+++ b/docs/admin_guide/monitoring.rst
@@ -0,0 +1,398 @@
+---
+layout: default
+
+# Top navigation
+top-nav-group: admin-guide
+top-nav-pos: 4
+top-nav-title: Monitoring
+
+# Sub-level navigation
+sub-nav-group: admin-guide
+sub-nav-parent: admin-guide
+sub-nav-id: monitoring
+sub-nav-pos: 4
+sub-nav-title: Monitoring
+---
+
+.. contents:: Monitoring
+
+Monitoring
+==========
+
+DistributedLog uses the stats library provided by Apache BookKeeper for reporting metrics in
+both the server and the client. This can be configured to report stats using pluggable stats
+provider to integrate with your monitoring system.
+
+Stats Provider
+~~~~~~~~~~~~~~
+
+`StatsProvider` is a provider that provides different kinds of stats logger for different scopes.
+The provider is also responsible for reporting its managed metrics.
+
+::
+
+ // Create the stats provider
+ StatsProvider statsProvider = ...;
+ // Start the stats provider
+ statsProvider.start(conf);
+ // Stop the stats provider
+ statsProvider.stop();
+
+Stats Logger
+____________
+
+A scoped `StatsLogger` is a stats logger that records 3 kinds of statistics
+under a given `scope`.
+
+A `StatsLogger` could be either created by obtaining from stats provider with
+the scope name:
+
+::
+
+ StatsProvider statsProvider = ...;
+ StatsLogger statsLogger = statsProvider.scope("test-scope");
+
+Or created by obtaining from a stats logger with a sub scope name:
+
+::
+
+ StatsLogger rootStatsLogger = ...;
+ StatsLogger subStatsLogger = rootStatsLogger.scope("sub-scope");
+
+All the metrics in a stats provider are managed in a hierarchical of scopes.
+
+::
+
+ // all stats recorded by `rootStatsLogger` are under 'root'
+ StatsLogger rootStatsLogger = statsProvider.scope("root");
+ // all stats recorded by 'subStatsLogger1` are under 'root/scope1'
+ StatsLogger subStatsLogger1 = statsProvider.scope("scope1");
+ // all stats recorded by 'subStatsLogger2` are under 'root/scope2'
+ StatsLogger subStatsLogger2 = statsProvider.scope("scope2");
+
+Counters
+++++++++
+
+A `Counter` is a cumulative metric that represents a single numerical value. A **counter**
+is typically used to count requests served, tasks completed, errors occurred, etc. Counters
+should not be used to expose current counts of items whose number can also go down, e.g.
+the number of currently running tasks. Use `Gauges` for this use case.
+
+To change a counter, use:
+
+::
+
+ StatsLogger statsLogger = ...;
+ Counter births = statsLogger.getCounter("births");
+ // increment the counter
+ births.inc();
+ // decrement the counter
+ births.dec();
+ // change the counter by delta
+ births.add(-10);
+ // reset the counter
+ births.reset();
+
+Gauges
+++++++
+
+A `Gauge` is a metric that represents a single numerical value that can arbitrarily go up and down.
+
+Gauges are typically used for measured values like temperatures or current memory usage, but also
+"counts" that can go up and down, like the number of running tasks.
+
+To define a gauge, stick the following code somewhere in the initialization:
+
+::
+
+ final AtomicLong numPendingRequests = new AtomicLong(0L);
+ StatsLogger statsLogger = ...;
+ statsLogger.registerGauge(
+ "num_pending_requests",
+ new Gauge<Number>() {
+ @Override
+ public Number getDefaultValue() {
+ return 0;
+ }
+ @Override
+ public Number getSample() {
+ return numPendingRequests.get();
+ }
+ });
+
+The gauge must always return a numerical value when sampling.
+
+Metrics (OpStats)
++++++++++++++++++
+
+A `OpStats` is a set of metrics that represents the statistics of an `operation`. Those metrics
+include `success` or `failure` of the operations and its distribution (also known as `Histogram`).
+It is usually used for timing.
+
+::
+
+ StatsLogger statsLogger = ...;
+ OpStatsLogger writeStats = statsLogger.getOpStatsLogger("writes");
+ long writeLatency = ...;
+
+ // register success op
+ writeStats.registerSuccessfulEvent(writeLatency);
+
+ // register failure op
+ writeStats.registerFailedEvent(writeLatency);
+
+Available Stats Providers
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+All the available stats providers are listed as below:
+
+* Twitter Science Stats (deprecated)
+* Twitter Ostrich Stats (deprecated)
+* Twitter Finagle Stats
+* Codahale Stats
+
+Twitter Science Stats
+_____________________
+
+Use following dependency to enable Twitter science stats provider.
+
+::
+
+ <dependency>
+ <groupId>org.apache.bookkeeper.stats</groupId>
+ <artifactId>twitter-science-provider</artifactId>
+ <version>${bookkeeper.version}</version>
+ </dependency>
+
+Construct the stats provider for clients.
+
+::
+
+ StatsProvider statsProvider = new TwitterStatsProvider();
+ DistributedLogConfiguration conf = ...;
+
+ // starts the stats provider (optional)
+ statsProvider.start(conf);
+
+ // all the dl related stats are exposed under "dlog"
+ StatsLogger statsLogger = statsProvider.getStatsLogger("dlog");
+ DistributedLogNamespace namespace = DistributedLogNamespaceBuilder.newBuilder()
+ .uri(...)
+ .conf(conf)
+ .statsLogger(statsLogger)
+ .build();
+
+ ...
+
+ // stop the stats provider (optional)
+ statsProvider.stop();
+
+
+Expose the stats collected by the stats provider by configuring following settings:
+
+::
+
+ // enable exporting the stats
+ statsExport=true
+ // exporting the stats at port 8080
+ statsHttpPort=8080
+
+
+If exporting stats is enabled, all the stats are exported by the http endpoint.
+You could curl the http endpoint to check the stats.
+
+::
+
+ curl -s <host>:8080/vars
+
+
+check ScienceStats_ for more details.
+
+.. _ScienceStats: https://github.com/twitter/commons/tree/master/src/java/com/twitter/common/stats
+
+Twitter Ostrich Stats
+_____________________
+
+Use following dependency to enable Twitter ostrich stats provider.
+
+::
+
+ <dependency>
+ <groupId>org.apache.bookkeeper.stats</groupId>
+ <artifactId>twitter-ostrich-provider</artifactId>
+ <version>${bookkeeper.version}</version>
+ </dependency>
+
+Construct the stats provider for clients.
+
+::
+
+ StatsProvider statsProvider = new TwitterOstrichProvider();
+ DistributedLogConfiguration conf = ...;
+
+ // starts the stats provider (optional)
+ statsProvider.start(conf);
+
+ // all the dl related stats are exposed under "dlog"
+ StatsLogger statsLogger = statsProvider.getStatsLogger("dlog");
+ DistributedLogNamespace namespace = DistributedLogNamespaceBuilder.newBuilder()
+ .uri(...)
+ .conf(conf)
+ .statsLogger(statsLogger)
+ .build();
+
+ ...
+
+ // stop the stats provider (optional)
+ statsProvider.stop();
+
+
+Expose the stats collected by the stats provider by configuring following settings:
+
+::
+
+ // enable exporting the stats
+ statsExport=true
+ // exporting the stats at port 8080
+ statsHttpPort=8080
+
+
+If exporting stats is enabled, all the stats are exported by the http endpoint.
+You could curl the http endpoint to check the stats.
+
+::
+
+ curl -s <host>:8080/stats.txt
+
+
+check Ostrich_ for more details.
+
+.. _Ostrich: https://github.com/twitter/ostrich
+
+Twitter Finagle Metrics
+_______________________
+
+Use following dependency to enable bridging finagle stats receiver to bookkeeper's stats provider.
+All the stats exposed by the stats provider will be collected by finagle stats receiver and exposed
+by Twitter's admin service.
+
+::
+
+ <dependency>
+ <groupId>org.apache.bookkeeper.stats</groupId>
+ <artifactId>twitter-finagle-provider</artifactId>
+ <version>${bookkeeper.version}</version>
+ </dependency>
+
+Construct the stats provider for clients.
+
+::
+
+ StatsReceiver statsReceiver = ...; // finagle stats receiver
+ StatsProvider statsProvider = new FinagleStatsProvider(statsReceiver);
+ DistributedLogConfiguration conf = ...;
+
+ // the stats provider does nothing on start.
+ statsProvider.start(conf);
+
+ // all the dl related stats are exposed under "dlog"
+ StatsLogger statsLogger = statsProvider.getStatsLogger("dlog");
+ DistributedLogNamespace namespace = DistributedLogNamespaceBuilder.newBuilder()
+ .uri(...)
+ .conf(conf)
+ .statsLogger(statsLogger)
+ .build();
+
+ ...
+
+ // the stats provider does nothing on stop.
+ statsProvider.stop();
+
+
+check `finagle metrics library`__ for more details on how to expose the stats.
+
+.. _TwitterServer: https://twitter.github.io/twitter-server/Migration.html
+
+__ TwitterServer_
+
+Codahale Metrics
+________________
+
+Use following dependency to enable Twitter ostrich stats provider.
+
+::
+
+ <dependency>
+ <groupId>org.apache.bookkeeper.stats</groupId>
+ <artifactId>codahale-metrics-provider</artifactId>
+ <version>${bookkeeper.version}</version>
+ </dependency>
+
+Construct the stats provider for clients.
+
+::
+
+ StatsProvider statsProvider = new CodahaleMetricsProvider();
+ DistributedLogConfiguration conf = ...;
+
+ // starts the stats provider (optional)
+ statsProvider.start(conf);
+
+ // all the dl related stats are exposed under "dlog"
+ StatsLogger statsLogger = statsProvider.getStatsLogger("dlog");
+ DistributedLogNamespace namespace = DistributedLogNamespaceBuilder.newBuilder()
+ .uri(...)
+ .conf(conf)
+ .statsLogger(statsLogger)
+ .build();
+
+ ...
+
+ // stop the stats provider (optional)
+ statsProvider.stop();
+
+
+Expose the stats collected by the stats provider in different ways by configuring following settings.
+Check Codehale_ on how to configuring report endpoints.
+
+::
+
+ // How frequent report the stats
+ codahaleStatsOutputFrequencySeconds=...
+ // The prefix string of codahale stats
+ codahaleStatsPrefix=...
+
+ //
+ // Report Endpoints
+ //
+
+ // expose the stats to Graphite
+ codahaleStatsGraphiteEndpoint=...
+ // expose the stats to CSV files
+ codahaleStatsCSVEndpoint=...
+ // expose the stats to Slf4j logging
+ codahaleStatsSlf4jEndpoint=...
+ // expose the stats to JMX endpoint
+ codahaleStatsJmxEndpoint=...
+
+
+check Codehale_ for more details.
+
+.. _Codehale: https://dropwizard.github.io/metrics/3.1.0/
+
+Enable Stats Provider on Bookie Servers
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The stats provider used by *Bookie Servers* is configured by setting the following option.
+
+::
+
+ // class of stats provider
+ statsProviderClass="org.apache.bookkeeper.stats.CodahaleMetricsProvider"
+
+Metrics
+~~~~~~~
+
+Check the Metrics_ reference page for the metrics exposed by DistributedLog.
+
+.. _Metrics: ../user_guide/references/metrics
http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/b169db04/docs/admin_guide/operations.rst
----------------------------------------------------------------------
diff --git a/docs/admin_guide/operations.rst b/docs/admin_guide/operations.rst
new file mode 100644
index 0000000..cbd30fe
--- /dev/null
+++ b/docs/admin_guide/operations.rst
@@ -0,0 +1,224 @@
+---
+layout: default
+
+# Top navigation
+top-nav-group: admin-guide
+top-nav-pos: 1
+top-nav-title: Operations
+
+# Sub-level navigation
+sub-nav-group: admin-guide
+sub-nav-parent: admin-guide
+sub-nav-id: operations
+sub-nav-pos: 1
+sub-nav-title: Operations
+---
+
+.. contents:: DistributedLog Operations
+
+DistributedLog Operations
+=========================
+
+Feature Provider
+~~~~~~~~~~~~~~~~
+
+DistributedLog uses a `feature-provider` library provided by Apache BookKeeper for managing features
+dynamically at runtime. It is a feature-flag_ system used to proportionally control what features
+are enabled for the system. In other words, it is a way of altering the control in a system without
+restarting it. It can be used during all stages of development, its most visible use case is on
+production. For instance, during a production release, you can enable or disable individual features,
+control the data flow through the system, thereby minimizing risk of system failure in real time.
+
+.. _feature-flag: https://en.wikipedia.org/wiki/Feature_toggle
+
+This `feature-provider` interface is pluggable and easy to integrate with any configuration management
+system.
+
+API
+___
+
+`FeatureProvider` is a provider that manages features under different scopes. The provider is responsible
+for loading features dynamically at runtime. A `Feature` is a numeric flag that control how much percentage
+of this feature will be available to the system - the number is called `availability`.
+
+::
+
+ Feature.name() => returns the name of this feature
+ Feature.availability() => returns the availability of this feature
+ Feature.isAvailable() => returns true if its availability is larger than 0; otherwise false
+
+
+It is easy to obtain a feature from the provider by just providing a feature name.
+
+::
+
+ FeatureProvider provider = ...;
+ Feature feature = provider.getFeature("feature1"); // returns the feature named 'feature1'
+
+
+The `FeatureProvider` is scopable to allow creating features in a hierarchical way. For example, if a system
+is comprised of two subsystems, one is *cache*, while the other one is *storage*. so the features belong to
+different subsystems can be created under different scopes.
+
+::
+
+ FeatureProvider provider = ...;
+ FeatureProvider cacheFeatureProvider = provider.scope("cache");
+ FeatureProvider storageFeatureProvider = provider.scope("storage");
+ Feature writeThroughFeature = cacheFeatureProvider.getFeature("write_through");
+ Feature duralWriteFeature = storageFeatureProvider.getFeature("dural_write");
+
+ // so the available features under `provider` are: (assume scopes are separated by '.')
+ // - 'cache.write_through'
+ // - 'storage.dural_write'
+
+
+The feature provider could be passed to `DistributedLogNamespaceBuilder` when building the namespace,
+thereby it would be used for controlling the features exposed under `DistributedLogNamespace`.
+
+::
+
+ FeatureProvider rootProvider = ...;
+ FeatureProvider dlFeatureProvider = rootProvider.scope("dlog");
+ DistributedLogNamespace namespace = DistributedLogNamespaceBuilder.newBuilder()
+ .uri(uri)
+ .conf(conf)
+ .featureProvider(dlFeatureProvider)
+ .build();
+
+
+The feature provider is loaded by reflection on distributedlog write proxy server. You could specify
+the feature provider class name as below. Otherwise it would use `DefaultFeatureProvider`, which disables
+all the features by default.
+
+::
+
+ featureProviderClass=com.twitter.distributedlog.feature.DynamicConfigurationFeatureProvider
+
+
+
+Configuration Based Feature Provider
+____________________________________
+
+Beside `DefaultFeatureProvider`, distributedlog also provides a file-based feature provider - it loads
+the features from properties files.
+
+All the features and their availabilities are configured in properties file format. For example,
+
+::
+
+ cache.write_through=100
+ storage.dural_write=0
+
+
+You could configure `featureProviderClass` in distributedlog configuration file by setting it to
+`com.twitter.distributedlog.feature.DynamicConfigurationFeatureProvider` to enable file-based feature
+provider. The feature provider will load the features from two files, one is base config file configured
+by `fileFeatureProviderBaseConfigPath`, while the other one is overlay config file configured by
+`fileFeatureProviderOverlayConfigPath`. Current implementation doesn't differentiate these two files
+too much other than the `overlay` config will override the settings in `base` config. It is recommended
+to have a base config file for storing the default availability values for your system and dynamically
+adjust the availability values in overlay config file.
+
+::
+
+ featureProviderClass=com.twitter.distributedlog.feature.DynamicConfigurationFeatureProvider
+ fileFeatureProviderBaseConfigPath=/path/to/base/config
+ fileFeatureProviderOverlayConfigPath=/path/to/overlay/config
+ // how frequent we reload the config files
+ dynamicConfigReloadIntervalSec=60
+
+
+Available Features
+__________________
+
+Check the Features_ reference page for the features exposed by DistributedLog.
+
+.. _Features: ../user_guide/references/features
+
+`dlog`
+~~~~~~
+
+A CLI is provided for inspecting DistributedLog streams and metadata.
+
+.. code:: bash
+
+ dlog
+ JMX enabled by default
+ Usage: dlog <command>
+ where command is one of:
+ local Run distributedlog sandbox
+ example Run distributedlog example
+ tool Run distributedlog tool
+ proxy_tool Run distributedlog proxy tool to interact with proxies
+ balancer Run distributedlog balancer
+ admin Run distributedlog admin tool
+ help This help message
+
+ or command is the full name of a class with a defined main() method.
+
+ Environment variables:
+ DLOG_LOG_CONF Log4j configuration file (default $HOME/src/distributedlog/distributedlog-service/conf/log4j.properties)
+ DLOG_EXTRA_OPTS Extra options to be passed to the jvm
+ DLOG_EXTRA_CLASSPATH Add extra paths to the dlog classpath
+
+These variable can also be set in conf/dlogenv.sh
+
+Create a stream
+_______________
+
+To create a stream:
+
+.. code:: bash
+
+ dlog tool create -u <DL URI> -r <STREAM PREFIX> -e <STREAM EXPRESSION>
+
+
+List the streams
+________________
+
+To list all the streams under a given DistributedLog namespace:
+
+.. code:: bash
+
+ dlog tool list -u <DL URI>
+
+Show stream's information
+_________________________
+
+To view the metadata associated with a stream:
+
+.. code:: bash
+
+ dlog tool show -u <DL URI> -s <STREAM NAME>
+
+
+Dump a stream
+_____________
+
+To dump the items inside a stream:
+
+.. code:: bash
+
+ dlog tool dump -u <DL URI> -s <STREAM NAME> -o <START TXN ID> -l <NUM RECORDS>
+
+Delete a stream
+_______________
+
+To delete a stream, run:
+
+.. code:: bash
+
+ dlog tool delete -u <DL URI> -s <STREAM NAME>
+
+
+Truncate a stream
+_________________
+
+Truncate the streams under a given DistributedLog namespace. You could specify a filter to match the streams that you want to truncate.
+
+There is a difference between the ``truncate`` and ``delete`` command. When you issue a ``truncate``, the data will be purge without removing the streams. A ``delete`` will delete the stream. You can pass the flag ``-delete`` to the ``truncate`` command to also delete the streams.
+
+.. code:: bash
+
+ dlog tool truncate -u <DL URI>
http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/b169db04/docs/admin_guide/performance.rst
----------------------------------------------------------------------
diff --git a/docs/admin_guide/performance.rst b/docs/admin_guide/performance.rst
new file mode 100644
index 0000000..fa36660
--- /dev/null
+++ b/docs/admin_guide/performance.rst
@@ -0,0 +1,22 @@
+---
+layout: default
+
+# Top navigation
+top-nav-group: admin-guide
+top-nav-pos: 2
+top-nav-title: Performance Tuning
+
+# Sub-level navigation
+sub-nav-group: admin-guide
+sub-nav-parent: admin-guide
+sub-nav-id: performance
+sub-nav-pos: 2
+sub-nav-title: Performance Tuning
+---
+
+.. contents:: Performance Tuning
+
+Performance Tuning
+==================
+
+(describe how to tune performance, critical settings)
http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/b169db04/docs/admin_guide/vagrant.rst
----------------------------------------------------------------------
diff --git a/docs/admin_guide/vagrant.rst b/docs/admin_guide/vagrant.rst
new file mode 100644
index 0000000..70e6fb0
--- /dev/null
+++ b/docs/admin_guide/vagrant.rst
@@ -0,0 +1,18 @@
+Sample Deployment using vagrant
+================================
+
+This file explains vagrant deployment.
+
+Prerequesites
+--------------
+1. Vagrant: From https://www.vagrantup.com/downloads.html
+2. vagrant-hostmanager plugin: From https://github.com/devopsgroup-io/vagrant-hostmanager
+
+Steps
+-----
+1. Create a snapshot using ./scripts/snapshot
+2. Run vagrant up from the root directory of the enlistment
+3. Vagrant brings up a zookeepers with machine names: zk1,zk2, .... with IP addresses 192.168.50.11,192.168.50.12,....
+4. Vagrant brings the bookies with machine names: node1,node2, ... with IP addresses 192.168.50.51,192.168.50.52,....
+5. The script will also start writeproxies at distributedlog://$PUBLIC_ZOOKEEPER_ADDRESSES/messaging/distributedlog/mynamespace as the namespace. If you want it to point to a different namespace/port/shard, please update the ./vagrant/bk.sh script.
+6. If you want to run the client on the host machine, please add these node names and their IP addresses to the /etc/hosts file on the host.
http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/b169db04/docs/admin_guide/zookeeper.rst
----------------------------------------------------------------------
diff --git a/docs/admin_guide/zookeeper.rst b/docs/admin_guide/zookeeper.rst
new file mode 100644
index 0000000..0fee9de
--- /dev/null
+++ b/docs/admin_guide/zookeeper.rst
@@ -0,0 +1,104 @@
+---
+layout: default
+
+# Top navigation
+top-nav-group: admin-guide
+top-nav-pos: 5
+top-nav-title: ZooKeeper
+
+# Sub-level navigation
+sub-nav-group: admin-guide
+sub-nav-parent: admin-guide
+sub-nav-id: zookeeper
+sub-nav-pos: 5
+sub-nav-title: ZooKeeper
+---
+
+ZooKeeper
+=========
+
+To run a DistributedLog ensemble, you'll need a set of Zookeeper
+nodes. There is no constraints on the number of Zookeeper nodes you
+need. One node is enough to run your cluster, but for reliability
+purpose, you should run at least 3 nodes.
+
+Version
+-------
+
+DistributedLog leverages zookeepr `multi` operations for metadata updates.
+So the minimum version of zookeeper is 3.4.*. We recommend to run stable
+zookeeper version `3.4.8`.
+
+Run ZooKeeper from distributedlog source
+----------------------------------------
+
+Since `zookeeper` is one of the dependency of `distributedlog-service`. You could simply
+run `zookeeper` servers using same set of scripts provided in `distributedlog-service`.
+In the following sections, we will describe how to run zookeeper using the scripts provided
+in `distributedlog-service`.
+
+Build
++++++
+
+First of all, build DistributedLog:
+
+.. code-block:: bash
+
+ $ mvn clean install -DskipTests
+
+Configuration
++++++++++++++
+
+The configuration file `zookeeper.conf.template` under `distributedlog-service/conf` is a template of
+production configuration to run a zookeeper node. Most of the configuration settings are good for
+production usage. You might need to configure following settings according to your environment and
+hardware platform.
+
+Ensemble
+^^^^^^^^
+
+You need to configure the zookeeper servers form this ensemble as below:
+
+::
+
+ server.1=127.0.0.1:2710:3710:participant;0.0.0.0:2181
+
+
+Please check zookeeper_ website for more configurations.
+
+Disks
+^^^^^
+
+You need to configure following settings according to the disk layout of your hardware.
+It is recommended to put `dataLogDir` under a separated disk from others for performance.
+
+::
+
+ # the directory where the snapshot is stored.
+ dataDir=/tmp/data/zookeeper
+
+ # where txlog are written
+ dataLogDir=/tmp/data/zookeeper/txlog
+
+
+Run
++++
+
+As `zookeeper` is shipped as part of `distributedlog-service`, you could use the `dlog-daemon.sh`
+script to start `zookeeper` as daemon thread.
+
+Start the zookeeper:
+
+.. code-block:: bash
+
+ $ ./distributedlog-service/bin/dlog-daemon.sh start zookeeper /path/to/zookeeper.conf
+
+Stop the zookeeper:
+
+.. code-block:: bash
+
+ $ ./distributedlog-service/bin/dlog-daemon.sh stop zookeeper
+
+Please check zookeeper_ website for more details.
+
+.. _zookeeper: http://zookeeper.apache.org/
http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/b169db04/docs/api/core.rst
----------------------------------------------------------------------
diff --git a/docs/api/core.rst b/docs/api/core.rst
deleted file mode 100644
index f2f0a24..0000000
--- a/docs/api/core.rst
+++ /dev/null
@@ -1,471 +0,0 @@
-Core Library API
-================
-
-The distributedlog core library interacts with namespaces and logs directly.
-It is written in Java.
-
-Namespace API
--------------
-
-A DL namespace is a collection of *log streams*. Applications could *create*
-or *delete* logs under a DL namespace.
-
-Namespace URI
-~~~~~~~~~~~~~
-
-An **URI** is used to locate the *namespace*. The *Namespace URI* is typically
-comprised of *3* components:
-
-* scheme: `distributedlog-<backend>`. The *backend* indicates what backend is used to store the log data.
-* domain name: the domain name that used to talk to the *backend*. In the example as below, the domain name part is *zookeeper server*, which is used to store log metadata in bookkeeper based backend implementation.
-* path: path points to the location that stores logs. In the example as below, it is a zookeeper path that points to the znode that stores all the logs metadata.
-
-::
-
- distributedlog-bk://<zookeeper-server>/path/to/stream
-
-The available backend is only bookkeeper based backend.
-The default `distributedlog` scheme is aliased to `distributedlog-bk`.
-
-Building a Namespace
-~~~~~~~~~~~~~~~~~~~~
-
-Once you have the *namespace uri*, you could build the namespace instance.
-The namespace instance will be used for operating streams under it.
-
-::
-
- // DistributedLog Configuration
- DistributedLogConfiguration conf = new DistributedLogConfiguration();
- // Namespace URI
- URI uri = ...; // create the namespace uri
- // create a builder to build namespace instances
- DistributedLogNamespaceBuilder builder = DistributedLogNamespaceBuilder.newBuilder();
- DistributedLogNamespace namespace = builder
- .conf(conf) // configuration that used by namespace
- .uri(uri) // namespace uri
- .statsLogger(...) // stats logger to log stats
- .featureProvider(...) // feature provider on controlling features
- .build();
-
-Create a Log
-~~~~~~~~~~~~
-
-Creating a log is pretty straight forward by calling `distributedlognamespace#createlog(logname)`.
-it only creates the log under the namespace but doesn't return any handle for operating the log.
-
-::
-
- DistributedLogNamespace namespace = ...; // namespace
- try {
- namespace.createLog("test-log");
- } catch (IOException ioe) {
- // handling the exception on creating a log
- }
-
-Open a Log
-~~~~~~~~~~
-
-A `DistributedLogManager` handle will be returned when opening a log by `#openLog(logName)`. The
-handle could be used for writing data to or reading data from the log. If the log doesn't exist
-and `createStreamIfNotExists` is set to true in the configuration, the log will be created
-automatically when writing first record.
-
-::
-
- DistributedLogConfiguration conf = new DistributedLogConfiguration();
- conf.setCreateStreamIfNotExists(true);
- DistributedLogNamespace namespace = DistributedLogNamespace.newBuilder()
- .conf(conf)
- ...
- .build();
- DistributedLogManager logManager = namespace.openLog("test-log");
- // use the log manager to open writer to write data or open reader to read data
- ...
-
-Sometimes, applications may open a log with different configuration settings. It could be done via
-a overloaded `#openLog` method, as below:
-
-::
-
- DistributedLogConfiguration conf = new DistributedLogConfiguration();
- // set the retention period hours to 24 hours.
- conf.setRetentionPeriodHours(24);
- URI uri = ...;
- DistributedLogNamespace namespace = DistributedLogNamespace.newBuilder()
- .conf(conf)
- .uri(uri)
- ...
- .build();
-
- // Per Log Configuration
- DistributedLogConfigration logConf = new DistributedLogConfiguration();
- // set the retention period hours to 12 hours for a single stream
- logConf.setRetentionPeriodHours(12);
-
- // open the log with overrided settings
- DistributedLogManager logManager = namespace.openLog("test-log",
- Optional.of(logConf),
- Optiona.absent());
-
-Delete a Log
-~~~~~~~~~~~~
-
-`DistributedLogNamespace#deleteLog(logName)` will deletes the log from the namespace. Deleting a log
-will attempt acquiring a lock before deletion. If a log is writing by an active writer, the lock
-would be already acquired by the writer. so the deleting will fail.
-
-::
-
- DistributedLogNamespace namespace = ...;
- try {
- namespace.deleteLog("test-log");
- } catch (IOException ioe) {
- // handle the exceptions
- }
-
-Log Existence
-~~~~~~~~~~~~~
-
-Applications could check whether a log exists in a namespace by calling `DistributedLogNamespace#logExists(logName)`.
-
-::
-
- DistributedLogNamespace namespace = ...;
- if (namespace.logExists("test-log")) {
- // actions when log exists
- } else {
- // actions when log doesn't exist
- }
-
-Get List of Logs
-~~~~~~~~~~~~~~~~
-
-Applications could list the logs under a namespace by calling `DistributedLogNamespace#getLogs()`.
-
-::
-
- DistributedLogNamespace namespace = ...;
- Iterator<String> logs = namespace.getLogs();
- while (logs.hasNext()) {
- String logName = logs.next();
- // ... process the log
- }
-
-Writer API
-----------
-
-There are two ways to write records into a log stream, one is using 'synchronous' `LogWriter`, while the other one is using
-asynchronous `AsyncLogWriter`.
-
-LogWriter
-~~~~~~~~~
-
-The first thing to write data into a log stream is to construct the writer instance. Please note that the distributedlog core library enforce single-writer
-semantic by deploying a zookeeper locking mechanism. If there is only an active writer, the subsequent calls to `#startLogSegmentNonPartitioned()` will
-fail with `OwnershipAcquireFailedException`.
-
-::
-
- DistributedLogNamespace namespace = ....;
- DistributedLogManager dlm = namespace.openLog("test-log");
- LogWriter writer = dlm.startLogSegmentNonPartitioned();
-
-.. _Construct Log Record:
-
-Log records are constructed to represent the data written to a log stream. Each log record is associated with application defined transaction id.
-The transaction id has to be non-decreasing otherwise writing the record will be rejected with `TransactionIdOutOfOrderException`. Application is allowed to
-bypass the transaction id sanity checking by setting `maxIdSanityCheck` to false in configuration. System time and atomic numbers are good candicates used for
-transaction id.
-
-::
-
- long txid = 1L;
- byte[] data = ...;
- LogRecord record = new LogRecord(txid, data);
-
-Application could either add a single record (via `#write(LogRecord)`) or a bunch of records (via `#writeBulk(List<LogRecord>)`) into the log stream.
-
-::
-
- writer.write(record);
- // or
- List<LogRecord> records = Lists.newArrayList();
- records.add(record);
- writer.writeBulk(records);
-
-The write calls return immediately after the records are added into the output buffer of writer. So the data isn't guaranteed to be durable until writer
-explicitly calls `#setReadyToFlush()` and `#flushAndSync()`. Those two calls will first transmit buffered data to backend, wait for transmit acknowledges
-and commit the written data to make them visible to readers.
-
-::
-
- // flush the records
- writer.setReadyToFlush();
- // commit the records to make them visible to readers
- writer.flushAndSync();
-
-The DL log streams are endless streams unless they are sealed. 'endless' means that writers keep writing records to the log streams, readers could keep
-tailing reading from the end of the streams and it never stops. Application could seal a log stream by calling `#markEndOfStream()`.
-
-::
-
- // seal the log stream
- writer.markEndOfStream();
-
-
-The complete example of writing records is showed as below.
-
-::
-
- DistributedLogNamespace namespace = ....;
- DistributedLogManager dlm = namespace.openLog("test-log");
-
- LogWriter writer = dlm.startLogSegmentNonPartitioned();
- for (long txid = 1L; txid <= 100L; txid++) {
- byte[] data = ...;
- LogRecord record = new LogRecord(txid, data);
- writer.write(record);
- }
- // flush the records
- writer.setReadyToFlush();
- // commit the records to make them visible to readers
- writer.flushAndSync();
-
- // seal the log stream
- writer.markEndOfStream();
-
-AsyncLogWriter
-~~~~~~~~~~~~~~
-
-Constructing an asynchronous `AsyncLogWriter` is as simple as synchronous `LogWriter`.
-
-::
-
- DistributedLogNamespace namespace = ....;
- DistributedLogManager dlm = namespace.openLog("test-log");
- AsyncLogWriter writer = dlm.startAsyncLogSegmentNonPartitioned();
-
-All the writes to `AsyncLogWriter` are asynchronous. The futures representing write results are only satisfied when the data are persisted in the stream durably.
-A DLSN (distributedlog sequence number) will be returned for each write, which is used to represent the position (aka offset) of the record in the log stream.
-All the records adding in order are guaranteed to be persisted in order.
-
-.. _Async Write Records:
-
-::
-
- List<Future<DLSN>> addFutures = Lists.newArrayList();
- for (long txid = 1L; txid <= 100L; txid++) {
- byte[] data = ...;
- LogRecord record = new LogRecord(txid, data);
- addFutures.add(writer.write(record));
- }
- List<DLSN> addResults = Await.result(Future.collect(addFutures));
-
-The `AsyncLogWriter` also provides the method to truncate a stream to a given DLSN. This is super helpful for building replicated state machines, who need
-explicit controls on when the data could be deleted.
-
-::
-
- DLSN truncateDLSN = ...;
- Future<DLSN> truncateFuture = writer.truncate(truncateDLSN);
- // wait for truncation result
- Await.result(truncateFuture);
-
-Reader API
-----------
-
-Sequence Numbers
-~~~~~~~~~~~~~~~~
-
-A log record is associated with sequence numbers. First of all, application can assign its own sequence number (called `TransactionID`)
-to the log record while writing it (see `Construct Log Record`_). Secondly, a log record will be assigned with an unique system generated sequence number
-`DLSN` (distributedlog sequence number) when it is written to a log (see `Async Write Records`_). Besides `DLSN` and `TransactionID`,
-a monotonically increasing 64-bits `SequenceId` is assigned to the record at read time, indicating its position within the log.
-
-:Transaction ID: Transaction ID is a positive 64-bits long number that is assigned by the application.
- Transaction ID is very helpful when application wants to organize the records and position the readers using their own sequencing method. A typical
- use case of `Transaction ID` is `DistributedLog Write Proxy`. The write proxy assigns non-decreasing timestamps to log records, which the timestamps
- could be used as `physical time` to implement `TTL` (Time To Live) feature in a strong consistent database.
-:DLSN: DLSN (DistributedLog Sequence Number) is the sequence number generated during written time.
- DLSN is comparable and could be used to figure out the order between records. A DLSN is comprised with 3 components. They are `Log Segment Sequence Number`,
- `Entry Id` and `Slot Id`. The DLSN is usually used for comparison, positioning and truncation.
-:Sequence ID: Sequence ID is introduced to address the drawback of `DLSN`, in favor of answering questions like `how many records written between two DLSNs`.
- Sequence ID is a 64-bits monotonic increasing number starting from zero. The sequence ids are computed during reading, and only accessible by readers.
- That means writers don't know the sequence ids of records at the point they wrote them.
-
-The readers could be positioned to start reading from any positions in the log, by using `DLSN` or `Transaction ID`.
-
-LogReader
-~~~~~~~~~
-
-`LogReader` is a 'synchronous' sequential reader reading records from a log stream starting from a given position. The position could be
-`DLSN` (via `#getInputStream(DLSN)`) or `Transaction ID` (via `#getInputStream(long)`). After the reader is open, it could call either
-`#readNext(boolean)` or `#readBulk(boolean, int)` to read records out of the log stream sequentially. Closing the reader (via `#close()`)
-will release all the resources occupied by this reader instance.
-
-Exceptions could be thrown during reading records due to various issues. Once the exception is thrown, the reader is set to an error state
-and it isn't usable anymore. It is the application's responsibility to handle the exceptions and re-create readers if necessary.
-
-::
-
- DistributedLogManager dlm = ...;
- long nextTxId = ...;
- LogReader reader = dlm.getInputStream(nextTxId);
-
- while (true) { // keep reading & processing records
- LogRecord record;
- try {
- record = reader.readNext(false);
- nextTxId = record.getTransactionId();
- // process the record
- ...
- } catch (IOException ioe) {
- // handle the exception
- ...
- reader = dlm.getInputStream(nextTxId + 1);
- }
- }
-
-Reading records from an endless log stream in `synchronous` way isn't as trivial as in `asynchronous` way. Because it lacks of callback mechanism.
-Instead, `LogReader` introduces a flag `nonBlocking` on controlling the waiting behavior on `synchronous` reads. Blocking (`nonBlocking = false`)
-means the reads will wait for records before returning read calls, while NonBlocking (`nonBlocking = true`) means the reads will only check readahead
-cache and return whatever records available in readahead cache.
-
-The `waiting` period varies in `blocking` mode. If the reader is catching up with writer (there are plenty of records in the log), the read call will
-wait until records are read and returned. If the reader is already caught up with writer (there are no more records in the log at read time), the read
-call will wait for a small period of time (defined in `DistributedLogConfiguration#getReadAheadWaitTime()`) and return whatever records available in
-readahead cache. In other words, if a reader sees no record on blocking reads, it means the reader is `caught-up` with the writer.
-
-See examples below on how to read records using `LogReader`.
-
-::
-
- // Read individual records
-
- LogReader reader = ...;
- // keep reading records in blocking mode until no records available in the log
- LogRecord record = reader.readNext(false);
- while (null != record) {
- // process the record
- ...
- // read next record
- record = reader.readNext(false);
- }
- ...
-
- // reader is caught up with writer, doing non-blocking reads to tail the log
- while (true) {
- record = reader.readNext(true);
- if (null == record) {
- // no record available yet. backoff ?
- ...
- } else {
- // process the new record
- ...
- }
- }
-
-::
-
- // Read records in batch
-
- LogReader reader = ...;
- int N = 10;
-
- // keep reading N records in blocking mode until no records available in the log
- List<LogRecord> records = reader.readBulk(false, N);
- while (!records.isEmpty()) {
- // process the list of records
- ...
- if (records.size() < N) { // no more records available in the log
- break;
- }
- // read next N records
- records = reader.readBulk(false, N);
- }
-
- ...
-
- // reader is caught up with writer, doing non-blocking reads to tail the log
- while (true) {
- records = reader.readBulk(true, N);
- // process the new records
- ...
- }
-
-
-AsyncLogReader
-~~~~~~~~~~~~~~
-
-Similar as `LogReader`, applications could open an `AsyncLogReader` by positioning to different positions, either `DLSN` or `Transaction ID`.
-
-::
-
- DistributedLogManager dlm = ...;
-
- Future<AsyncLogReader> openFuture;
-
- // position the reader to transaction id `999`
- openFuture = dlm.openAsyncLogReader(999L);
-
- // or position the reader to DLSN
- DLSN fromDLSN = ...;
- openFuture = dlm.openAsyncLogReader(fromDLSN);
-
- AsyncLogReader reader = Await.result(openFuture);
-
-Reading records from an `AsyncLogReader` is asynchronously. The future returned by `#readNext()`, `#readBulk(int)` or `#readBulk(int, long, TimeUnit)`
-represents the result of the read operation. The future is only satisfied when there are records available. Application could chain the futures
-to do sequential reads.
-
-Reading records one by one from an `AsyncLogReader`.
-
-::
-
- void readOneRecord(AsyncLogReader reader) {
- reader.readNext().addEventListener(new FutureEventListener<LogRecordWithDLSN>() {
- public void onSuccess(LogRecordWithDLSN record) {
- // process the record
- ...
- // read next
- readOneRecord(reader);
- }
- public void onFailure(Throwable cause) {
- // handle errors and re-create reader
- ...
- reader = ...;
- // read next
- readOneRecord(reader);
- }
- });
- }
-
- AsyncLogReader reader = ...;
- readOneRecord(reader);
-
-Reading records in batches from an `AsyncLogReader`.
-
-::
-
- void readBulk(AsyncLogReader reader, int N) {
- reader.readBulk(N).addEventListener(new FutureEventListener<List<LogRecordWithDLSN>>() {
- public void onSuccess(List<LogRecordWithDLSN> records) {
- // process the records
- ...
- // read next
- readBulk(reader, N);
- }
- public void onFailure(Throwable cause) {
- // handle errors and re-create reader
- ...
- reader = ...;
- // read next
- readBulk(reader, N);
- }
- });
- }
-
- AsyncLogReader reader = ...;
- readBulk(reader, N);
-
http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/b169db04/docs/api/main.rst
----------------------------------------------------------------------
diff --git a/docs/api/main.rst b/docs/api/main.rst
deleted file mode 100644
index 5c80053..0000000
--- a/docs/api/main.rst
+++ /dev/null
@@ -1,9 +0,0 @@
-API
-===
-
-.. toctree::
- :maxdepth: 1
-
- core
- proxy
- practice
http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/b169db04/docs/api/practice.rst
----------------------------------------------------------------------
diff --git a/docs/api/practice.rst b/docs/api/practice.rst
deleted file mode 100644
index 419151e..0000000
--- a/docs/api/practice.rst
+++ /dev/null
@@ -1,76 +0,0 @@
-Best Practices
-==============
-
-Write records using Fat Client or Thin Client
----------------------------------------------
-
-`Fat Client` is the writer in distributedlog core library which talks to ZooKeeper and BookKeeper directly,
-while `Thin Client` is the write proxy client which talks to write proxy service.
-
-It is strongly recommended that writing records via `Write Proxy` service rather than using core library directly.
-Because using `Thin Client` has following benefits:
-
-- `Thin Client` is purely thrift RPC based client. It doesn't talk to zookeeper and bookkeeper directly and less complicated.
-- `Write Proxy` manages ownerships of log writers. `Thin Client` doesn't have to deal with ownerships of log writers.
-- `Thin Client` is more upgrade-friendly than `Fat Client`.
-
-The only exception to use distributedlog core library directly is when you application requires:
-
-- Write Ordering. `Write Ordering` means all the writes issued by the writer should be written in a strict order
- in the log. `Write Proxy` service could only guarantee `Read Ordering`. `Read Ordering` means the write proxies will write
- the write requests in their receiving order and gurantee the data seen by all the readers in same order.
-- Ownership Management. If the application already has any kind of ownership management, like `master-slave`, it makes more
- sense that it uses distributedlog core library directly.
-
-How to position reader by time
-------------------------------
-
-Sometimes, application wants to read data by time, like read data from 2 hours ago. This could be done by positioning
-the reader using `Transaction ID`, if the `Transaction ID` is the timestamp (All the streams produced by `Write Proxy` use
-timestamp as `Transaction ID`).
-
-::
-
- DistributedLogManager dlm = ...;
-
- long timestamp = System.currentTimeMillis();
- long startTxId = timestamp - TimeUnit.MILLISECONDS.convert(2, TimeUnit.HOURS);
- AsyncLogReader reader = Await.result(dlm.openAsyncLogReader(startTxId));
- ...
-
-How to seal a stream
---------------------
-
-Typically, DistributedLog is used as endless streams. In some use cases, application wants to `seal` the stream. So writers
-can't write more data into the log stream and readers could receive notifications about the stream has been sealed.
-
-Write could seal a log stream as below:
-
-::
-
- DistributedLogManager dlm = ...;
-
- LogWriter writer = dlm.startLogSegmentNonPartitioned;
- // writer writes bunch of records
- ...
-
- // writer seals the stream
- writer.markEndOfStream();
-
-Reader could detect a stream has been sealed as below:
-
-::
-
- DistributedLogManager dlm = ...;
-
- LogReader reader = dlm.getInputStream(1L);
- LogRecord record;
- try {
- while ((record = reader.readNext(false)) != null) {
- // process the record
- ...
- }
- } catch (EndOfStreamException eos) {
- // the stream has been sealed
- ...
- }
http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/b169db04/docs/api/proxy.rst
----------------------------------------------------------------------
diff --git a/docs/api/proxy.rst b/docs/api/proxy.rst
deleted file mode 100644
index ad2b56b..0000000
--- a/docs/api/proxy.rst
+++ /dev/null
@@ -1,74 +0,0 @@
-Write Proxy Client API
-======================
-
-`Write Proxy` is a 'stateless' service on managing the ownerships of writers of log streams. It is used to
-accept to `fan-in` writes from different publishers.
-
-Build Client
-------------
-
-The first thing of using `Write Proxy` service is to build the write proxy client. The endpoint of a `Write Proxy` service
-is typically identified by `Finagle Name`_. Name strings must be supplied when constructing a `Write Proxy` client.
-
-.. _Finagle Name: http://twitter.github.io/finagle/guide/Names.html
-
-::
-
- // 1. Create a Finagle client builder. It would be used for building connection to write proxies.
- ClientBuilder clientBuilder = ClientBuilder.get()
- .hostConnectionLimit(1)
- .hostConnectionCoresize(1)
- .tcpConnectTimeout(Duration$.MODULE$.fromMilliseconds(200))
- .connectTimeout(Duration$.MODULE$.fromMilliseconds(200))
- .requestTimeout(Duration$.MODULE$.fromSeconds(2));
-
- // 2. Choose a client id to identify the client.
- ClientId clientId = ClientId$.MODULE$.apply("test");
-
- String finagleNameStr = "inet!127.0.0.1:8000";
-
- // 3. Create the write proxy client builder
- DistributedLogClientBuilder builder = DistributedLogClientBuilder.newBuilder()
- .name("test-writer")
- .clientId(clientId)
- .clientBuilder(clientBuilder)
- .statsReceiver(statsReceiver)
- .finagleNameStr(finagleNameStr);
-
- // 4. Build the client
- DistributedLogClient client = builder.build();
-
-Write Records
--------------
-
-Writing records to log streams via `Write Proxy` is much simpler than using the core library. The transaction id
-will be automatically assigned with `timestamp` by write proxies. The `timestamp` is guaranteed to be non-decreasing, which it
-could be treated as `physical time` within a log stream, and be used for implementing features like `TTL` in a strong consistent
-database.
-
-::
-
- DistributedLogClient client = ...;
-
- // Write a record to a stream
- String streamName = "test-stream";
- byte[] data = ...;
- Future<DLSN> writeFuture = client.write(streamName, ByteBuffer.wrap(data));
- Await.result(writeFuture);
-
-Truncate Streams
-----------------
-
-Client could issue truncation requests (via `#truncate(String, DLSN)`) to write proxies to truncate a log stream up to a given
-position.
-
-::
-
- DistributedLogClient client = ...;
-
- // Truncate a stream to DLSN
- String streamName = "test-stream";
- DLSN truncationDLSN = ...;
- Future<DLSN> truncateFuture = client.truncate(streamName, truncationDLSN);
- Await.result(truncateFuture);
-
http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/b169db04/docs/architecture/main.rst
----------------------------------------------------------------------
diff --git a/docs/architecture/main.rst b/docs/architecture/main.rst
deleted file mode 100644
index e66cdcb..0000000
--- a/docs/architecture/main.rst
+++ /dev/null
@@ -1,160 +0,0 @@
-Architecture
-============
-
-Data Model
-----------
-
-DistributedLog exposes the `log stream` as the unit of operations. A `log stream` is a totally ordered,
-immutable sequence of log records. A `log record` is a sequence of bytes. Log records are batched into `entries`
-and written into `log segments`. Figure 1 illustrates the logical elements of a log stream.
-
-.. figure:: ../images/datamodel.png
- :align: center
-
- Figure 1. Anatomy of a log stream
-
-Log Segments
-~~~~~~~~~~~~
-
-Although the application views the log stream as a continuous sequence of log records, it is physically stored as
-multiple `log segments`, where these segments are the unit of `manageability`. All the records in a log segment have
-the same replication configuration. The log segments are allocated, distributed and stored in a `log segment store`.
-As records are written to the log stream, DistributedLog switches to a new log segment based on a configured `rolling policy`.
-The rolling policy can be `time-based` i.e. based on a configured period of time (e.g. every 2 hours) or `size-based`
-i.e. based on a maximum log segment size (e.g. every 128MB). This allows the log segments to be distributed evenly
-across all the storage nodes. This helps evenly spread read traffic to avoid hot spots in the cluster.
-
-A log segment is also the unit of data retention. Log segments are deleted either by explicitly truncation or expiration.
-Old data is garbage collected by the log segment store once the log segments are deleted.
-
-Log Sequence Numbers
-~~~~~~~~~~~~~~~~~~~~
-
-`Log records` are written sequentially into a log stream, and assigned a unique sequence number called `DLSN`
-(DistributedLog Sequence Number). A DLSN is comprised of 3 components: a `Log Segment Sequence Number` (LSSN),
-the sequence number of the log segment that the record belongs to, an `Entry ID` (EID) - the entry id in this log segment
-that the record is in, and a `Slot ID` (SID) - the slot within the entry. Records can be ordered by DLSN.
-
-Besides DLSN, an application can assign a `Transaction ID`, a non-decreasing positive 64-bit integer, to each log record it writes.
-This facilitates application-specific sequencing of records and positioning of the reader. For example, a common use of the transaction ID
-is to store the timestamp of when the log record was added to the log stream. This transaction ID can then be used to rewind to a specific
-time in analytics applications.
-
-Namespace
-~~~~~~~~~
-
-Log streams that belong to same application are usually categorized and managed under a `namespace`. A `namespace` is used by applications
-to locate where the log streams are. Applications can `create` and `delete` streams under a namespace, and `truncate` a stream to given ID.
-
-Software Stack
---------------
-
-The software stack is shown in Figure 2. The software stack is divided into three layers, with each layer is responsible for
-different features of DistributedLog. These layers are `Persistent Storage`, `DistributedLog Core` and `Stateless Serving`.
-
-.. figure:: ../images/softwarestack.png
- :align: center
-
- Figure 2. DistributedLog Software Stack
-
-Persistent Storage
-~~~~~~~~~~~~~~~~~~
-
-DistributedLog provides the core features - `durability`, `availability` and `consistency` through the storage layer.
-The main components in storage layer are `Log Segment Store`, `Cold Storage` and `Metadata Store`.
-
-Log Segment Store
-+++++++++++++++++
-
-The Log segment store is responsible for storing the log segments as they are created and ensure they are durably replicated.
-We use `Apache BookKeeper` as the log segment store. BookKeeper helps us achieve low tail latencies for writes and reads as well as
-low delivery latency which is the end to end latency from when the record is generated until it is read by the reader - because bookkeeper's
-storage layout is optimized for I/O isolation for log workloads.
-
-In addition to storage layout optimization, the log segment store (via BookKeeper) also provides a built-in `fencing` mechanism for
-achieving strong consistency among multiple writers. We will discuss more about consistency in section `Design Details`.
-
-Cold Storage
-++++++++++++
-
-The data in the log segment store is eventually moved to a `cold storage`. Cold storage allows cost efficient storage of large volumes
-of log segments for extended period of time. Applications many want to have access to old data for application error recovery or debugging.
-As log segments are completed, they are proactively copied over to the cold storage, thereby providing a backup for disaster recovery or an
-operation error. We use HDFS as our cold storage.
-
-Metadata Store
-++++++++++++++
-
-The metadata in DistributedLog consists of the mapping from log streams to their constituent log segments as well as each log segment\u2019s metadata.
-The log segment metadata includes the `log segment ID`, `start and end transaction IDs`, `completion time`, and its `status`. The metadata store
-is required to provide metadata operations such as consistent read and write ordering to guarantee metadata consistency in the event of failures.
-Also the metadata store should provide a notification mechanism to support streaming reads. We use ZooKeeper as the metadata store, because it is
-a strongly consistent data store which provides versioned updates, strong ordering and notifications using watches.
-
-DistributedLog Core
-~~~~~~~~~~~~~~~~~~~
-
-DistributedLog builds its core functionality on top of the log segment store and the metadata store. It provides the core data model of log streams
-and its naming system, and provides a `single-writer-multiple-reader` access model.
-
-Writers write data into the logs of their choice. Writers sequence log records written to the log streams. Therefore there is only one active log
-segment for a given log stream at a time. Correctness and consistency are guaranteed using a fencing mechanism in the log segment store and
-versioned metadata updates to prevent two writers from writing to the same log stream after a network partition.
-
-Reading from a log stream starts by `positioning` a reader on a log record by specifying either a DLSN or a Transaction Id. Once a reader has been
-positioned, it receives all the log records in increasing order of the sequence numbers and each record is delivered exactly once. It is up to
-individual applications to choose an appropriate mechanism to record readers positions and provide this position when a new reader session begins
-(e.g restart from a failure). Applications can choose the appropriate method for storing positions based on the desired processing semantics.
-
-Stateless Serving
-~~~~~~~~~~~~~~~~~
-
-A stateless serving layer is built on top of the storage layer to support large number of writers and readers. The serving layer includes `Write Proxy`
-and `Read Proxy`. `Write Proxy` manages the ownerships of the log streams, forwards the write requests to storage via the core library and handles
-load balancing and failover. It allows sequencing writes from many clients (aka `Fan-in`). `Read Proxy` caches log records for multiple readers consuming
-the same log stream.
-
-Ownership Tracker
-+++++++++++++++++
-
-`Ownership Tracker` tracks the liveness of the owners of the log streams and fails over the ownership to other healthy write proxies when the current
-owner becomes unavailable. Since we already use zookeeper for metadata storage, we also use zookeeper for tracking the liveness of write proxies using
-`ephemeral znodes` and failover the ownership when zookeeper session expires.
-
-Routing Service
-+++++++++++++++
-Since readers read committed data and are strict followers, the read proxies do not have to track ownership of log streams. We use consistent hashing
-as a routing mechanism to route the readers to corresponding read proxies.
-
-Applications can either use a thin client that talks to the serving tier to access DistributedLog or embed the core library to talk to the storage directly
-when they require strict write ordering. Applications choose partitioning strategies and track their reader positions based on their specific requirements.
-
-Lifecyle of records
--------------------
-
-Figure 3 illustrates the lifecycle of a log record in DistributedLog as it flows from writers to readers and we discuss how different layers interact with
-each other.
-
-.. figure:: ../images/requestflow.png
- :align: center
-
- Figure 3. Lifecycle of a record
-
-The application constructs the log records and initiates write requests (step 1). The write requests will be forwarded to the write proxy that is the master
-of the log stream. The master writer proxy will write the records in the log stream\u2019s transmit buffer. Based on the configured transmit policy, records in
-the transmit buffer will be transmitted as a batched entry to log segment store (step 2). Application can trade latency for throughput by transmitting
-`immediately` (lowest latency), `periodically` (grouping records that appear within the transmit period) or when transmit buffer has accumulated more than
-`max-outstanding bytes`.
-
-The batched entry is transmitted to multiple bookies (storage nodes in bookkeeper cluster) in parallel (step 3). The log segment store will respond back to
-writer once the entry is persisted durably on disk. Once the write proxy receives confirmation of durability from a quorum of bookies, it will send an
-acknowledgement to the application (step 4).
-
-Although the writer knows that the record is guaranteed to be persisted and replicated in bookkeeper. Individual bookies do not necessarily know that the
-consensus agreement has been reached for this record. The writer must therefore record a `commit` to make this record visible to all the readers.
-This `commit` can piggyback on the next batch of records from the application. If no new application records are received within the specified SLA for
-persistence, the writer will issue a special `control log record` notifying the log segment store that the record can now be made visible to readers (step 5).
-
-The readers\u2019 request that is waiting for new data using `long polling` will now receive the recent committed log records (step 6). Speculative long poll reads will be sent to other replicas to archieve predictable low 99.9% percentile latency (step 7).
-
-The log records will be cached in read proxies (step 8) for fanout readers. The read clients also use similar long poll read mechanism to read data from read proxies (step 9).
http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/b169db04/docs/basics/introduction.rst
----------------------------------------------------------------------
diff --git a/docs/basics/introduction.rst b/docs/basics/introduction.rst
index e29bb28..07ac059 100644
--- a/docs/basics/introduction.rst
+++ b/docs/basics/introduction.rst
@@ -1,3 +1,21 @@
+---
+# Top navigation
+top-nav-group: user-guide
+top-nav-pos: 1
+top-nav-title: Introduction
+
+# Sub-level navigation
+sub-nav-group: user-guide
+sub-nav-parent: user-guide
+sub-nav-id: introduction
+sub-nav-pos: 1
+sub-nav-title: Introduction
+
+layout: default
+---
+
+.. contents:: DistributedLog Overview
+
Introduction
============
http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/b169db04/docs/basics/main.rst
----------------------------------------------------------------------
diff --git a/docs/basics/main.rst b/docs/basics/main.rst
deleted file mode 100644
index 06d0afd..0000000
--- a/docs/basics/main.rst
+++ /dev/null
@@ -1,8 +0,0 @@
-Getting Started
-===============
-
-.. toctree::
- :maxdepth: 1
-
- introduction
- quickstart
http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/b169db04/docs/basics/quickstart.rst
----------------------------------------------------------------------
diff --git a/docs/basics/quickstart.rst b/docs/basics/quickstart.rst
deleted file mode 100644
index 508e625..0000000
--- a/docs/basics/quickstart.rst
+++ /dev/null
@@ -1,111 +0,0 @@
-Quick Start
-===========
-
-This tutorial assumes you are starting from fresh and have no existing BookKeeper or ZooKeeper data.
-
-Step 1: Download the binary
-~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-:doc:`Download <../download>` the stable version of `DistributedLog` and un-zip it.
-
-::
-
- // Download the binary `distributedlog-all-${gitsha}.zip`
- > unzip distributedlog-all-${gitsha}.zip
-
-
-Step 2: Start ZooKeeper & BookKeeper
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-DistributedLog uses `ZooKeeper` as the metadata store and `BookKeeper` as the log segment store. So
-you need to first start a zookeeper server and a few bookies if you don't already have one. You can
-use the `dlog` script in `distributedlog-service` package to get a standalone bookkeeper sandbox. It
-starts a zookeeper server and `N` bookies (N is 3 by default).
-
-::
-
- // Start the local sandbox instance at port `7000`
- > ./distributedlog-service/bin/dlog local 7000
- DistributedLog Sandbox is running now. You could access distributedlog://127.0.0.1:7000
-
-
-Step 3: Create a DistributedLog namespace
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-Before using distributedlog, you need to create a distributedlog namespace to store your own list of
-streams. The zkServer for the local sandbox is `127.0.0.1:7000` and the bookkeeper's ledgers path is
-`/ledgers`. You could create a namespace pointing to the corresponding bookkeeper cluster.
-
-::
-
- > ./distributedlog-service/bin/dlog admin bind -l /ledgers -s 127.0.0.1:7000 -c distributedlog://127.0.0.1:7000/messaging/my_namespace
- No bookkeeper is bound to distributedlog://127.0.0.1:7000/messaging/my_namespace
- Created binding on distributedlog://127.0.0.1:7000/messaging/my_namespace.
-
-
-If you don't want to create a separated namespace, you could use the default namespace `distributedlog://127.0.0.1:7000/messaging/distributedlog`.
-
-
-Step 4: Create some log streams
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-Let's create 5 log streams, prefixed with `messaging-stream-`.
-
-::
-
- > ./distributedlog-service/bin/dlog tool create -u distributedlog://127.0.0.1:7000/messaging/my_namespace -r messaging-stream- -e 1-5
-
-
-We can now see the streams if we run the `list` command from the tool.
-
-::
-
- > ./distributedlog-service/bin/dlog tool list -u distributedlog://127.0.0.1:7000/messaging/my_namespace
- Streams under distributedlog://127.0.0.1:7000/messaging/my_namespace :
- --------------------------------
- messaging-stream-1
- messaging-stream-3
- messaging-stream-2
- messaging-stream-4
- messaging-stream-5
- --------------------------------
-
-
-Step 5: Start a write proxy
-~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-Now, lets start a write proxy server that serves writes to distributedlog namespace `distributedlog://127.0.0.1/messaging/my_namespace`. The server listens on 8000 to accept fan-in write requests.
-
-::
-
- > ./distributedlog-service/bin/dlog-daemon.sh start writeproxy -p 8000 --shard-id 1 -sp 8001 -u distributedlog://127.0.0.1:7000/messaging/my_namespace -mx -c `pwd`/distributedlog-service/conf/distributedlog_proxy.conf
-
-From 0.3.51-RC1 and onwards, use the below command to start the write proxy
-
-::
-
- > WP_SHARD_ID=1 WP_SERVICE_PORT=8000 WP_STATS_PORT=8001 WP_NAMESPACE='distributedlog://127.0.0.1:7000/messaging/my_namespace' ./distributedlog-service/bin/dlog-daemon.sh start writeproxy
-
-Step 6: Tail reading records
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-The distributedlog tutorial has a multi-streams reader that will dump out received records to standard output.
-
-::
-
- > ./distributedlog-tutorials/distributedlog-basic/bin/runner run com.twitter.distributedlog.basic.MultiReader distributedlog://127.0.0.1:7000/messaging/my_namespace messaging-stream-1,messaging-stream-2,messaging-stream-3,messaging-stream-4,messaging-stream-5
-
-
-Step 7: Write some records
-~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-The distributedlog tutorial also has a multi-streams writer that will take input from a console and write it out
-as records to the distributedlog write proxy. Each line will be sent as a separate record.
-
-Run the writer and type a few lines into the console to send to the server.
-
-::
-
- > ./distributedlog-tutorials/distributedlog-basic/bin/runner run com.twitter.distributedlog.basic.ConsoleProxyMultiWriter 'inet!127.0.0.1:8000' messaging-stream-1,messaging-stream-2,messaging-stream-3,messaging-stream-4,messaging-stream-5
-
-If you have each of the above commands running in a different terminal then you should now be able to type messages into the writer terminal and see them appear in the reader terminal.
http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/b169db04/docs/configuration/client.rst
----------------------------------------------------------------------
diff --git a/docs/configuration/client.rst b/docs/configuration/client.rst
deleted file mode 100644
index 6c0b6fb..0000000
--- a/docs/configuration/client.rst
+++ /dev/null
@@ -1,97 +0,0 @@
-Client Configuration
-====================
-
-This section describes the settings used by DistributedLog Write Proxy Client.
-
-Different from core library, the proxy client uses a builder to configure its settings.
-
-::
-
- DistributedLogClient client = DistributedLogClientBuilder.newBuilder()
- .name("test-client")
- .clientId("test-client-id")
- .finagleNameStr("inet!localhost:8080")
- .statsReceiver(statsReceiver)
- .build();
-
-Client Builder Settings
------------------------
-
-Common Settings
-~~~~~~~~~~~~~~~
-
-- *name(string)*: The name of the distributedlog client.
-- *clientId(string)*: The client id used for the underneath finagle client. It is a string identifier that server will
- use to identify who are the client. So the server can book keep and optionally reject unknown clients.
-- *requestTimeoutMs(int)*: The maximum time that a request could take before claiming it as failure, in milliseconds.
-- *thriftmux(boolean)*: The flag to enable or disable using ThriftMux_ on the underneath channels.
-- *streamFailfast(boolean)*: The flag to enable or disable failfast the requests when the server responds `stream-not-ready`.
- A stream would be treated as not ready when it is initializing or rolling log segments. The setting is only take effects
- when the write proxy also enables `failFastOnStreamNotReady`.
-
-.. _ThriftMux: http://twitter.github.io/finagle/guide/Protocols.html#mux
-
-Environment Settings
-~~~~~~~~~~~~~~~~~~~~
-
-DistributedLog uses finagle Names_ to identify the network locations of write proxies.
-Names must be supplied when building a distributedlog client through `finagleNameStr` or
-`finagleNameStrs`.
-
-.. _Names: http://twitter.github.io/finagle/guide/Names.html
-
-- *finagleNameStr(string)*: The finagle name to locate write proxies.
-- *finagleNameStrs(string, string...)*: A list of finagle names. It is typically used by the global replicated log wherever there
- are multiple regions of write proxies. The first parameter is the finagle name of local region; while the remaining parameters
- are the finagle names for remote regions.
-
-Redirection Settings
-~~~~~~~~~~~~~~~~~~~~
-
-DistributedLog client can redirect the requests to other write proxies when accessing a write proxy doesn't own the given stream.
-This section describes the settings related to redirection.
-
-- *redirectBackoffStartMs(int)*: The initial backoff for redirection, in milliseconds.
-- *redirectBackoffMaxMs(int)*: The maximum backoff for redirection, in milliseconds.
-- *maxRedirects(int)*: The maximum number of redirections that a request could take before claiming it as failure.
-
-Channel Settings
-~~~~~~~~~~~~~~~~
-
-DistributedLog client uses FinagleClient_ to establish the connections to the write proxy. A finagle client will be
-created via ClientBuilder_ for each write proxy.
-
-.. _FinagleClient: https://twitter.github.io/finagle/guide/Clients.html
-
-.. _ClientBuilder: http://twitter.github.io/finagle/docs/index.html#com.twitter.finagle.builder.ClientBuilder
-
-- *clientBuilder(ClientBuilder)*: The finagle client builder to build connection to each write proxy.
-
-Ownership Cache Settings
-~~~~~~~~~~~~~~~~~~~~~~~~
-
-DistributedLog client maintains a ownership cache locally to archieve stable deterministic request routing. Normally,
-the ownership cache is propagated after identified a new owner when performing stream related operations such as write.
-The client also does handshaking when initiating connections to a write proxy or periodically for fast failure detection.
-During handshaking, the client also pull the latest ownership mapping from write proxies to update its local cache, which
-it would help detecting ownership changes quickly, and avoid latency penalty introduced by redirection when ownership changes.
-
-- *handshakeWithClientInfo(boolean)*: The flag to enable or disable pulling ownership mapping during handshaking.
-- *periodicHandshakeIntervalMs(long)*: The periodic handshake interval in milliseconds. Every provided interval, the DL client
- will handshake with existing proxies. It would detect proxy failures during handshaking. If the interval is already greater than
- `periodicOwnershipSyncIntervalMs`, the handshake will pull the latest ownership mapping. Otherwise, it will not. The default
- value is 5 minutes. Setting it to 0 or negative number will disable this feature.
-- *periodicOwnershipSyncIntervalMs(long)*: The periodic ownership sync interval, in milliseconds. If periodic handshake is
- enabled, the handshake will sync ownership if the elapsed time is greater than the sync interval.
-- *streamNameRegex(string)*: The regex to match the stream names that the client cares about their ownerships.
-
-Constraint Settings
-~~~~~~~~~~~~~~~~~~~
-
-- *checksum(boolean)*: The flag to enable/disable checksum validation on requests that sent to proxy.
-
-Stats Settings
-~~~~~~~~~~~~~~
-
-- *statsReceiver(StatsReceiver)*: The stats receiver used for collecting stats exposed by this client.
-- *streamStatsReceiver(StatsReceiver)*: The stats receiver used for collecting per stream stats exposed by this client.