You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@bookkeeper.apache.org by Ivan Kelly <iv...@apache.org> on 2013/03/13 22:12:29 UTC

Builtin metadata store proposal

Hi guys,

A few proposals(ManagedLedger, SuperLedger) recently have started
putting metadata in the bookies in a limited fashion to work around
the metadata issues we have for many many ledgers. Sijie and I
discussed this on Monday, and we came up with a proposal for putting a
metadata store in a bookie, which will scale horizontally as you add
more bookies. 

I've put the proposal on google docs.

https://docs.google.com/document/d/1DobUg5HdnTnKlNN1Yc_o_flShZvN8erG1KtAcrlYPK4/edit?usp=sharing

The design uses zk(helix) and bookkeeper for storage. This has the
advantage of not introducing a dependency on another system, which, in
the case of bookkeeper could lead to circular dependencies. For
example, if we put metadata in hbase, but then hbase uses bookkeeper
for wal. It also has the advantage that we can store enough
information in the bookies themselves to repair a cluster in case
zookeeper is lost. This was a major concern for one large user I met
this week.

-Ivan

RE: Builtin metadata store proposal

Posted by Uma Maheswara Rao G <ma...@huawei.com>.
This sounds like a great idea!
I will go through the proposal and come up with some points if I can understand quickly!
I am not able to access the link from my office NW. I will read it from home.

Regards,
Uma

________________________________________
From: Ivan Kelly [ivank@apache.org]
Sent: Thursday, March 14, 2013 2:42 AM
To: bookkeeper-dev@zookeeper.apache.org
Subject: Builtin metadata store proposal

Hi guys,

A few proposals(ManagedLedger, SuperLedger) recently have started
putting metadata in the bookies in a limited fashion to work around
the metadata issues we have for many many ledgers. Sijie and I
discussed this on Monday, and we came up with a proposal for putting a
metadata store in a bookie, which will scale horizontally as you add
more bookies.

I've put the proposal on google docs.

https://docs.google.com/document/d/1DobUg5HdnTnKlNN1Yc_o_flShZvN8erG1KtAcrlYPK4/edit?usp=sharing

The design uses zk(helix) and bookkeeper for storage. This has the
advantage of not introducing a dependency on another system, which, in
the case of bookkeeper could lead to circular dependencies. For
example, if we put metadata in hbase, but then hbase uses bookkeeper
for wal. It also has the advantage that we can store enough
information in the bookies themselves to repair a cluster in case
zookeeper is lost. This was a major concern for one large user I met
this week.

-Ivan