[GitHub] jvrao commented on a change in pull request #1138: BP-31 BookKeeper Durability Anchor

[GitBox <gi...@apache.org>]

jvrao commented on a change in pull request #1138: BP-31 BookKeeper Durability Anchor
URL: https://github.com/apache/bookkeeper/pull/1138#discussion_r168311020
 
 

 ##########
 File path: site/bps/BP-31-durability
 ##########
 @@ -0,0 +1,134 @@
+
+---
+title: "BP-31: BookKeeper Durability(Anchor)"
+issue: https://github.com/apache/bookkeeper/BP-31-durability
+state: 'Anchor BP'
+release: "x.y.z"
+---
+## Motivation
+Apache BookKeeper is transitioning into a full fledged distributed storage that can keep the data for long term. Durability is paramount to achieve the status of trusted store. This Anchor BP discusses many gaps and areas of improvement.  Each issue listed here will have another issue and this BP is expected to be updated when that issue is created.
+
+## Durability Contract
+1. **Maintain WQ copies all the time**. If any ledger falls into under replicated state, there needs to be an SLA on how quickly the replication levels can be brought back to normal levels.
+2. **Enforce Placement Policy** strictly during write and replication.
+3. **Protect the data** against corruption on the wire or at rest.
+
+## Work Grouping (In the order of priority)
+### Detect Durability Validation
+First step is to understand the areas of durability breaches. Design metrics that record durability contract violations. 
+* At the Creation: Validate durability contract the extent is being created
+* At the Deletion: Validate durability contract when extent is deleted
+* During lifetime: Validate durability contract during the lifetime of the extent.(periodic validator)
+* During Read: IO or Checksum errors in the read path
+### Delete Discipline
+* Build a single delete choke point with stringent validations
+* Archival bit in the metadata to assist Two phase Deletes
+* Stateful/Explicit Deletes
+### Metadata Recovery
+* Metadata recovery tool to reconstruct the metadata if the metadata server gets wiped out. This tool need to make sure that the data is readable even if we can't get all the metadata (ex: ctime) back.
 
 Review comment:
   I just gave an example there. There can be more; The intention of the tool is to make ledgers readable. 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services