You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@bookkeeper.apache.org by si...@apache.org on 2018/02/06 06:12:54 UTC
[bookkeeper] branch master updated: BP-29: Metadata API module
This is an automated email from the ASF dual-hosted git repository.
sijie pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/bookkeeper.git
The following commit(s) were added to refs/heads/master by this push:
new 011bb65 BP-29: Metadata API module
011bb65 is described below
commit 011bb652363b471e52e082cd9a13ff6a31375525
Author: Sijie Guo <si...@apache.org>
AuthorDate: Mon Feb 5 22:12:46 2018 -0800
BP-29: Metadata API module
Descriptions of the changes in this PR:
Related to BP-28 (#1113), this proposal is to propose how we want to organize the metadata modules, to support different metadata storage implementation.
Master Ticket: #1123
Author: Sijie Guo <si...@apache.org>
Reviewers: Jia Zhai <None>, Sijie Guo <si...@apache.org>, Venkateswararao Jujjuri (JV) <None>
This closes #1117 from sijie/BP-29-metadata-store-api
---
site/bps/BP-29-metadata-store-api-module.md | 87 +++++++++++++++++++++++++++++
site/community/bookkeeper_proposals.md | 3 +-
2 files changed, 89 insertions(+), 1 deletion(-)
diff --git a/site/bps/BP-29-metadata-store-api-module.md b/site/bps/BP-29-metadata-store-api-module.md
new file mode 100644
index 0000000..9a9b887
--- /dev/null
+++ b/site/bps/BP-29-metadata-store-api-module.md
@@ -0,0 +1,87 @@
+---
+title: "BP-29: Metadata API module"
+issue: https://github.com/apache/bookkeeper/<issue-number>
+state: 'Accepted'
+release: "N/A"
+---
+
+### Motivation
+
+We have already abstracted all the metadata operations into interfaces. And all the bookkeeper implementations only reply on metadata interfaces,
+rather than depending on zookeeper. This proposal is to organize the metadata interfaces and its implementations in a separate module and make
+bookkeeper implementation only depends on metadata interfaces, not depends on zookeeper. This would a few benefits:
+
+- It allows supporting different metadata storages, without bringing in dependencies of metadata store implementation directly into
+ bookkeeper-server module. The development of different metadata storage can be done without interleaving with each other.
+- It would define a clean module dependency between bookkeeper implementation and metadata api, and how bookkeeper load a different metadata
+ implementation.
+
+### Public Interfaces
+
+A more generic setting `metadataServiceUri` is introduced for replacing implementation specific settings `zkServers` and `zkLedgersRootPath`.
+
+A metadata service uri defines the location of a metadata storage. In zookeeper based implementation, the metadata service url will be
+`zk://<zkServers>/<zkLedgersRootPath>`.
+
+This new setting in bookie configuration will be like as below:
+
+```
+metadataServiceUri=zk://127.0.0.1/ledgers
+```
+
+If we eventually support Etcd as one of the metadata storages. Then the setting in bookie configuration to use Etcd will be:
+
+```
+metadataServiceUri=etcd://<etcd_servers>/<etcd_key_prefix>
+```
+
+### Proposed Changes
+
+#### Configuration
+
+This BP proposes introducing a more generic metadata setting `metadataServiceUri` to replace implementation specific settings
+`zkServers` and `zkLedgersRootPath`. All implementation specific settings should be considered moving to implementation itself.
+
+The `metadataServiceUri` can also be used for replacing the need of configuring `ledgerManagerFactoryClass`, `registrationClientClass` and
+`registrationManagerClass`. It is unnecessarily complicated to configure multiple settings to load a specific metadata implementation.
+We can just use the `scheme` field in `metadataServiceUri` to resolve which metadata implementation to use. Using uri to resolve
+different driver or implementation is commonly seen at java world, for example, jdbc to support different database drivers. Also, distributedlog
+uses this pattern to load different metadata driver.
+
+So in zookeeper based metadata implementation, the metadata service uri can be:
+
+- `zk+flat://127.0.0.1/ledgers`: the scheme is "zk+flat". it means a zookeeper base metadata implementation and it uses flat ledger manager.
+- `zk+hierarchical://127.0.0.1/ledgers`: the scheme is "zk+hierarchical". it means a zookeeper base metadata implementation and it
+ uses hierarchical ledger manager.
+- `zk+longhierarchical://127.0.0.1/ledgers`: the scheme is "zk+longhierarchical". it means a zookeeper base metadata implementation and it
+ uses long hierarchical ledger manager.
+
+#### Metadata Stores
+
+Introduce a new directory called `metadata-stores` for storing all the metadata related modules. Under this directory, it will have following modules:
+
+- `api`: it is the metadata api module `metadata-store-api`. It contains all the files defining the metadata interfaces.
+- `zookeeper`: it is the zookeeper implementation module `metadata-store-zookeeper`. It contains all the files that implementing the metadata interfaces
+ using zookeeper.
+
+If a new metadata implementation is added, a new directory will be created under `metadata-stores` to contain the implementation. For example, if we
+are adding `Etcd` as the metadata store. A `Etcd` directory will be created under `metadata-stores/etcd` to store the files that implement metadata
+interfaces using etcd client. And its module is named as `metadata-store-etcd`.
+
+We then change bookkeeper-server to depend on `metadata-store-api` only.
+
+This approach is same as other pluggable modules `stats-api` and `http-server`.
+
+### Compatibility, Deprecation, and Migration Plan
+
+No compatibility concern at this moment. New setting is introduced and old settings will still continue to work.
+No immediate deprecation.
+No migration is needed.
+
+### Test Plan
+
+This proposal is mostly around refactor. So existing test cases would cover this.
+
+### Rejected Alternatives
+
+N/A
diff --git a/site/community/bookkeeper_proposals.md b/site/community/bookkeeper_proposals.md
index 754436e..25c586c 100644
--- a/site/community/bookkeeper_proposals.md
+++ b/site/community/bookkeeper_proposals.md
@@ -85,7 +85,7 @@ using Google Doc.
This section lists all the _bookkeeper proposals_ made to BookKeeper.
-*Next Proposal Number: 28*
+*Next Proposal Number: 30*
### Inprogress
@@ -99,6 +99,7 @@ Proposal | State
[BP-18: LedgerType, Flags and StorageHints](https://cwiki.apache.org/confluence/display/BOOKKEEPER/BP-18%3A+LedgerType%2C+Flags+and+StorageHints) | Accepted
[BP-26: Move distributedlog library as part of bookkeeper](../../bps/BP-26-move-distributedlog-core-library) | Accepted
[BP-27: New BookKeeper CLI](../../bps/BP-27-new-bookkeeper-cli) | Draft
+[BP-29: Metadata API module](../../bps/BP-29-metadata-store-api-module) | Draft
### Adopted
--
To stop receiving notification emails like this one, please contact
sijie@apache.org.