You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by pe...@apache.org on 2020/06/16 01:29:50 UTC

[pulsar] branch master updated: Rearrange the BookKeeper documentation, add decommissioning section. (#7230)

This is an automated email from the ASF dual-hosted git repository.

penghui pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git


The following commit(s) were added to refs/heads/master by this push:
     new 59f1607  Rearrange the BookKeeper documentation, add decommissioning section. (#7230)
59f1607 is described below

commit 59f160792a6335988c8604e15345e234ef06acb3
Author: Anup Ghatage <gh...@gmail.com>
AuthorDate: Mon Jun 15 18:29:37 2020 -0700

    Rearrange the BookKeeper documentation, add decommissioning section. (#7230)
    
    ### Motivation
    
    While discussing issue https://github.com/apache/pulsar/issues/7055#issuecomment-641038722 yesterday, @sijie suggested I add the documentation for decommissioning bookies to the Pulsar docs.
    
    ### Modifications
    
    While adding the documentation for decommissioning, I also saw the BookKeeper sub sections didn't read well. There were duplicate sections for configuration as well.
    So just rearranged them in a way that it would be relevant for work flow as well as added the decommissioning process details.
    
    Rearranged the sections as:
    - Hardware considerations
    - Configure BookKeeper
    - Deploy BookKeeper
    - Starting bookies manually
    - Decommissioning bookies cleanly
---
 site2/docs/administration-zk-bk.md | 92 ++++++++++++++++++++++++++------------
 1 file changed, 63 insertions(+), 29 deletions(-)

diff --git a/site2/docs/administration-zk-bk.md b/site2/docs/administration-zk-bk.md
index 87665f1..a59154b 100644
--- a/site2/docs/administration-zk-bk.md
+++ b/site2/docs/administration-zk-bk.md
@@ -162,17 +162,49 @@ BookKeeper is responsible for all durable message storage in Pulsar. BookKeeper
 
 > For a guide to managing message persistence, retention, and expiry in Pulsar, see [this cookbook](cookbooks-retention-expiry.md).
 
-### Deploy BookKeeper
+### Hardware considerations
 
-BookKeeper provides [persistent message storage](concepts-architecture-overview.md#persistent-storage) for Pulsar.
+Bookie hosts are responsible for storing message data on disk. In order for bookies to provide optimal performance, ensuring that the bookies have a suitable hardware configuration is essential. You can choose two key dimensions to bookie hardware capacity:
 
-Each Pulsar broker needs to have its own cluster of bookies. The BookKeeper cluster shares a local ZooKeeper quorum with the Pulsar cluster.
+* Disk I/O capacity read/write
+* Storage capacity
+
+Message entries written to bookies are always synced to disk before returning an acknowledgement to the Pulsar broker. To ensure low write latency, BookKeeper is designed to use multiple devices:
+
+* A **journal** to ensure durability. For sequential writes, having fast [fsync](https://linux.die.net/man/2/fsync) operations on bookie hosts is critical. Typically, small and fast [solid-state drives](https://en.wikipedia.org/wiki/Solid-state_drive) (SSDs) should suffice, or [hard disk drives](https://en.wikipedia.org/wiki/Hard_disk_drive) (HDDs) with a [RAID](https://en.wikipedia.org/wiki/RAID)s controller and a battery-backed write cache. Both solutions can reach fsync latency of ~0.4 ms.
+* A **ledger storage device** is where data is stored until all consumers have acknowledged the message. Writes happen in the background, so write I/O is not a big concern. Reads happen sequentially most of the time and the backlog is drained only in case of consumer drain. To store large amounts of data, a typical configuration involves multiple HDDs with a RAID controller.
 
-### Configure bookies
+### Configure BookKeeper
 
 You can configure BookKeeper bookies using the [`conf/bookkeeper.conf`](reference-configuration.md#bookkeeper) configuration file. The most important aspect of configuring each bookie is ensuring that the [`zkServers`](reference-configuration.md#bookkeeper-zkServers) parameter is set to the connection string for local ZooKeeper of the Pulsar cluster.
 
-### Start up bookies
+Minimum configuration changes required in `conf/bookkeeper.conf` are:
+
+```properties
+# Change to point to journal disk mount point
+journalDirectory=data/bookkeeper/journal
+
+# Point to ledger storage disk mount point
+ledgerDirectories=data/bookkeeper/ledgers
+
+# Point to local ZK quorum
+zkServers=zk1.example.com:2181,zk2.example.com:2181,zk3.example.com:2181
+
+# Change the ledger manager type
+ledgerManagerType=hierarchical
+```
+
+To change the zookeeper root path that Bookkeeper uses, use zkLedgersRootPath=/MY-PREFIX/ledgers instead of zkServers=localhost:2181/MY-PREFIX
+
+> Consult the official [BookKeeper docs](http://bookkeeper.apache.org) for more information about BookKeeper.
+
+### Deploy BookKeeper
+
+BookKeeper provides [persistent message storage](concepts-architecture-overview.md#persistent-storage) for Pulsar.
+
+Each Pulsar broker needs to have its own cluster of bookies. The BookKeeper cluster shares a local ZooKeeper quorum with the Pulsar cluster.
+
+### Starting bookies manually
 
 You can start up a bookie in two ways: in the foreground or as a background daemon.
 
@@ -196,44 +228,46 @@ $ bin/bookkeeper shell bookiesanity
 
 This command creates a new ledger on the local bookie, writes a few entries, reads them back and finally deletes the ledger.
 
-### Hardware considerations
+### Decommissioning bookies cleanly
 
-Bookie hosts are responsible for storing message data on disk. In order for bookies to provide optimal performance, ensuring that the bookies have a suitable hardware configuration is essential. You can choose two key dimensions to bookie hardware capacity:
 
-* Disk I/O capacity read/write
-* Storage capacity
+In case the user wants to decommission a bookie, the following process is useful to follow in order to verify if the
+decommissioning was safely done.
 
-Message entries written to bookies are always synced to disk before returning an acknowledgement to the Pulsar broker. To ensure low write latency, BookKeeper is designed to use multiple devices:
+#### Before we decommission
+1. Ensure state of your cluster can support the decommissioning of the target bookie.
+Check if `EnsembleSize >= Write Quorum >= Ack Quorum` stays true with one less bookie
 
-* A **journal** to ensure durability. For sequential writes, having fast [fsync](https://linux.die.net/man/2/fsync) operations on bookie hosts is critical. Typically, small and fast [solid-state drives](https://en.wikipedia.org/wiki/Solid-state_drive) (SSDs) should suffice, or [hard disk drives](https://en.wikipedia.org/wiki/Hard_disk_drive) (HDDs) with a [RAID](https://en.wikipedia.org/wiki/RAID)s controller and a battery-backed write cache. Both solutions can reach fsync latency of ~0.4 ms.
-* A **ledger storage device** is where data is stored until all consumers have acknowledged the message. Writes happen in the background, so write I/O is not a big concern. Reads happen sequentially most of the time and the backlog is drained only in case of consumer drain. To store large amounts of data, a typical configuration involves multiple HDDs with a RAID controller.
+2. Ensure target bookie shows up in the listbookies command.
 
+3. Ensure that there is no other process ongoing (upgrade etc).
 
+#### Process of Decommissioning
+1. Log on to the bookie node, check if there are underreplicated ledgers.
 
-### Configure BookKeeper
+If there are, the decommission command will force them to be replicated.
+`$ bin/bookkeeper shell listunderreplicated`
 
-you can find configurable parameters for BookKeeper bookies in the [`conf/bookkeeper.conf`](reference-configuration.md#bookkeeper) file.
+2. Stop the bookie by killing the bookie process. Make sure there are no liveness / readiness probes setup for the bookies to spin them back up if you are deployed in a kubernetes environment.
 
-Minimum configuration changes required in `conf/bookkeeper.conf` are:
+3. Run the decommission command.
+If you have logged onto the node you wish to decommission, you don't need to provide `-bookieid`
+If you are running the decommission command for target bookie node from another bookie node you should mention 
+the target bookie id in the arguments for `-bookieid`
+`$ bin/bookkeeper shell decommissionbookie`
+or
+`$ bin/bookkeeper shell decommissionbookie -bookieid <target bookieid>`
 
-```properties
-# Change to point to journal disk mount point
-journalDirectory=data/bookkeeper/journal
-
-# Point to ledger storage disk mount point
-ledgerDirectories=data/bookkeeper/ledgers
+4. Validate that there are no ledgers on decommissioned bookie
+`$ bin/bookkeeper shell listledgers -bookieid <target bookieid>`
 
-# Point to local ZK quorum
-zkServers=zk1.example.com:2181,zk2.example.com:2181,zk3.example.com:2181
+Last step to verify is you could run this command to check if the bookie you decommissioned doesn’t show up in list bookies:
 
-# Change the ledger manager type
-ledgerManagerType=hierarchical
+```bash
+./bookkeeper shell listbookies -rw -h
+./bookkeeper shell listbookies -ro -h
 ```
 
-To change the zookeeper root path that Bookkeeper uses, use zkLedgersRootPath=/MY-PREFIX/ledgers instead of zkServers=localhost:2181/MY-PREFIX
-
-> Consult the official [BookKeeper docs](http://bookkeeper.apache.org) for more information about BookKeeper.
-
 ## BookKeeper persistence policies
 
 In Pulsar, you can set *persistence policies*, at the namespace level, that determine how BookKeeper handles persistent storage of messages. Policies determine four things: