ZooKeeper-less - single bookie setup

[Enrico Olivelli <eo...@gmail.com>]

Hi,
I would like to share this idea with you before starting a real design
effort for this new configuration.

Sometimes I have to run Bookie + BookKeeper inside the same JVM,
essentially because I want my applications to run both in "cluster mode"
and in "single instance mode".
I have some other use case of embedding the Bookie, but this is not the
scope of this email.

In order to run BookKeeper we need ZooKeeper and running ZooKeeper inside
the same process is tricky and you know, it is not the suggested
configuration, due to several aspects (out of the scope of this email
too....)

I would like to make it run without ZooKeeper at all.
This idea will support my use case (Bookie + Client inside the same JVM)
and the single Bookie use case, which is a feature that could be useful in
general.
Some other projects, like HBase, have "single server" vs "multi server"
setup, and so I think it will be a good idea for the project and for
gathering more interest and use cases.

We are using ZooKeeper for (at least) these purposes:
- bookie discovery
- ledger metadata management
- auditor leader election
- global configuration settings  (like autorecovery enabled/disabled....)

my case is to have only one Bookie, so things are really simpler, no
auditor, no shared configuration, no underreplication/overreplication, no
need for 'discovery'

In order to address the bookie discovery we can just have some client-side
configuration to set the address of the Bookie and have some special
PlacementPolicy which only returns the same bookie.

The big thing is to implement the ledger metadata management, because both
the Bookie and the client must deal with it.
My idea is to add to the Bookie protocol all the functions to access
LedgersMetadata (create/list/update with CAS.....), this RPCs will call the
underlying LedgerManager

On the client side we will not use ZK but execute RPCs on the Bookie
endpoint (a special LedgerManager which do not use ZK but uses RPCs)

On the Bookie side we will have a local ledger metadata management which
persists data on disk, we could use existing Key-Value stores, like RocksDB
or MapDB, or what ever we prefer, we need something which supports CAS and
eventually let us put bounds on used memory.

Does this make sense to you ?

-- Enrico

Re: ZooKeeper-less - single bookie setup

[Enrico Olivelli <eo...@gmail.com>]

Il gio 27 lug 2017, 18:28 Sijie Guo <gu...@gmail.com> ha scritto:

> On Thu, Jul 27, 2017 at 8:48 AM, Enrico Olivelli <eo...@gmail.com>
> wrote:
>
> > Il gio 27 lug 2017, 17:27 Venkateswara Rao Jujjuri <ju...@gmail.com>
> ha
> > scritto:
> >
> > > We have inMemoryMetaStore.java already to store metadata locally isn't
> > it?
> > > there are some tests that make use of that too. So if you don't need ZK
> > for
> > > consensus
> > > and you can use this inMemoryMetaStore for metadata store.
> > >
> >
> > MetadataStore should share data between clients and server, as
> > InMemoryMetaStore  is only in memory it is not useful.
> > We need a metadatastore on the bookie, with data persistence and memory
> > bounds, and on the client side we will access those metadata thru the
> > bookie.
> >
>
> There are already a MSLedgerManager :
>
> https://github.com/apache/bookkeeper/blob/master/bookkeeper-server/src/main/java/org/apache/bookkeeper/meta/MSLedgerManagerFactory.java


This facility still uses ZK but it is a good example. Is this used in
production by someone or is here only for tests?

From your answers I understand that my idea is not so silly, I think that
in the next months I will go on and draft some design notes and a prototype

Thank you

Enrico

>
>
> You just need to implement the metastore api.
>
>
> https://github.com/apache/bookkeeper/blob/master/bookkeeper-server/src/main/java/org/apache/bookkeeper/metastore/MetastoreTable.java
>
> If your client and bookie are running on same JVM, you can just share the
> embedded k/v instance. It can be either in-memory or local k/v. you don't
> necessarily need any network protocol.
>
>
> >
> > Enrico
> >
> >
> >
> > > On Thu, Jul 27, 2017 at 12:42 AM, Sijie Guo <gu...@gmail.com>
> wrote:
> > >
> > > > it might be better to call it 'local model' or 'standalone mode',
> > rather
> > > > than 'zookeeper-less'.
> > > >
> > > > On Wed, Jul 26, 2017 at 8:26 PM, Enrico Olivelli <
> eolivelli@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi,
> > > > > I would like to share this idea with you before starting a real
> > design
> > > > > effort for this new configuration.
> > > >
> > > >
> > > > > Sometimes I have to run Bookie + BookKeeper inside the same JVM,
> > > > > essentially because I want my applications to run both in "cluster
> > > mode"
> > > > > and in "single instance mode".
> > > > > I have some other use case of embedding the Bookie, but this is not
> > the
> > > > > scope of this email.
> > > > >
> > > > > In order to run BookKeeper we need ZooKeeper and running ZooKeeper
> > > inside
> > > > > the same process is tricky and you know, it is not the suggested
> > > > > configuration, due to several aspects (out of the scope of this
> email
> > > > > too....)
> > > > >
> > > > > I would like to make it run without ZooKeeper at all.
> > > > > This idea will support my use case (Bookie + Client inside the same
> > > JVM)
> > > > > and the single Bookie use case, which is a feature that could be
> > useful
> > > > in
> > > > > general.
> > > > > Some other projects, like HBase, have "single server" vs "multi
> > server"
> > > > > setup, and so I think it will be a good idea for the project and
> for
> > > > > gathering more interest and use cases.
> > > > >
> > > > > We are using ZooKeeper for (at least) these purposes:
> > > > > - bookie discovery
> > > > > - ledger metadata management
> > > > > - auditor leader election
> > > > > - global configuration settings  (like autorecovery
> > > enabled/disabled....)
> > > > >
> > > > > my case is to have only one Bookie, so things are really simpler,
> no
> > > > > auditor, no shared configuration, no underreplication/overreplicati
> > on,
> > > > no
> > > > > need for 'discovery'
> > > > >
> > > > > In order to address the bookie discovery we can just have some
> > > > client-side
> > > > > configuration to set the address of the Bookie and have some
> special
> > > > > PlacementPolicy which only returns the same bookie.
> > > > >
> > > > > The big thing is to implement the ledger metadata management,
> because
> > > > both
> > > > > the Bookie and the client must deal with it.
> > > > > My idea is to add to the Bookie protocol all the functions to
> access
> > > > > LedgersMetadata (create/list/update with CAS.....), this RPCs will
> > call
> > > > the
> > > > > underlying LedgerManager
> > > > >
> > > >
> > > > I think this is in general very good approach to take. It will make
> the
> > > > bookie client not depends on metadata store directly and delegating
> the
> > > > heavy work to the server (bookie side). Once the client is thin, it
> is
> > > easy
> > > > to implement clients in other languages.
> > > >
> > > > However, I do not think it is necessarily for your use case. You can
> > make
> > > > this as a separate topic. For RPC, I would suggest exploring gPRC (it
> > is
> > > > protobuf + netty), that would be much better for metadata rpcs rather
> > > than
> > > > implementing the logic in bookie protocol.
> > > >
> > > >
> > > > >
> > > > > On the client side we will not use ZK but execute RPCs on the
> Bookie
> > > > > endpoint (a special LedgerManager which do not use ZK but uses
> RPCs)
> > > > >
> > > > > On the Bookie side we will have a local ledger metadata management
> > > which
> > > > > persists data on disk, we could use existing Key-Value stores, like
> > > > RocksDB
> > > > > or MapDB, or what ever we prefer, we need something which supports
> > CAS
> > > > and
> > > > > eventually let us put bounds on used memory.
> > > > >
> > > >
> > > > In your use case, you might be able to just implement a ledger
> manager
> > > > using an embedded k/v.
> > > >
> > > >
> > > > >
> > > > > Does this make sense to you ?
> > > > >
> > > > > -- Enrico
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Jvrao
> > > ---
> > > First they ignore you, then they laugh at you, then they fight you,
> then
> > > you win. - Mahatma Gandhi
> > >
> > --
> >
> >
> > -- Enrico Olivelli
> >
>
-- 


-- Enrico Olivelli

Re: ZooKeeper-less - single bookie setup

[Sijie Guo <gu...@gmail.com>]

On Thu, Jul 27, 2017 at 8:48 AM, Enrico Olivelli <eo...@gmail.com>
wrote:

> Il gio 27 lug 2017, 17:27 Venkateswara Rao Jujjuri <ju...@gmail.com> ha
> scritto:
>
> > We have inMemoryMetaStore.java already to store metadata locally isn't
> it?
> > there are some tests that make use of that too. So if you don't need ZK
> for
> > consensus
> > and you can use this inMemoryMetaStore for metadata store.
> >
>
> MetadataStore should share data between clients and server, as
> InMemoryMetaStore  is only in memory it is not useful.
> We need a metadatastore on the bookie, with data persistence and memory
> bounds, and on the client side we will access those metadata thru the
> bookie.
>

There are already a MSLedgerManager :
https://github.com/apache/bookkeeper/blob/master/bookkeeper-server/src/main/java/org/apache/bookkeeper/meta/MSLedgerManagerFactory.java

You just need to implement the metastore api.

https://github.com/apache/bookkeeper/blob/master/bookkeeper-server/src/main/java/org/apache/bookkeeper/metastore/MetastoreTable.java

If your client and bookie are running on same JVM, you can just share the
embedded k/v instance. It can be either in-memory or local k/v. you don't
necessarily need any network protocol.


>
> Enrico
>
>
>
> > On Thu, Jul 27, 2017 at 12:42 AM, Sijie Guo <gu...@gmail.com> wrote:
> >
> > > it might be better to call it 'local model' or 'standalone mode',
> rather
> > > than 'zookeeper-less'.
> > >
> > > On Wed, Jul 26, 2017 at 8:26 PM, Enrico Olivelli <eo...@gmail.com>
> > > wrote:
> > >
> > > > Hi,
> > > > I would like to share this idea with you before starting a real
> design
> > > > effort for this new configuration.
> > >
> > >
> > > > Sometimes I have to run Bookie + BookKeeper inside the same JVM,
> > > > essentially because I want my applications to run both in "cluster
> > mode"
> > > > and in "single instance mode".
> > > > I have some other use case of embedding the Bookie, but this is not
> the
> > > > scope of this email.
> > > >
> > > > In order to run BookKeeper we need ZooKeeper and running ZooKeeper
> > inside
> > > > the same process is tricky and you know, it is not the suggested
> > > > configuration, due to several aspects (out of the scope of this email
> > > > too....)
> > > >
> > > > I would like to make it run without ZooKeeper at all.
> > > > This idea will support my use case (Bookie + Client inside the same
> > JVM)
> > > > and the single Bookie use case, which is a feature that could be
> useful
> > > in
> > > > general.
> > > > Some other projects, like HBase, have "single server" vs "multi
> server"
> > > > setup, and so I think it will be a good idea for the project and for
> > > > gathering more interest and use cases.
> > > >
> > > > We are using ZooKeeper for (at least) these purposes:
> > > > - bookie discovery
> > > > - ledger metadata management
> > > > - auditor leader election
> > > > - global configuration settings  (like autorecovery
> > enabled/disabled....)
> > > >
> > > > my case is to have only one Bookie, so things are really simpler, no
> > > > auditor, no shared configuration, no underreplication/overreplicati
> on,
> > > no
> > > > need for 'discovery'
> > > >
> > > > In order to address the bookie discovery we can just have some
> > > client-side
> > > > configuration to set the address of the Bookie and have some special
> > > > PlacementPolicy which only returns the same bookie.
> > > >
> > > > The big thing is to implement the ledger metadata management, because
> > > both
> > > > the Bookie and the client must deal with it.
> > > > My idea is to add to the Bookie protocol all the functions to access
> > > > LedgersMetadata (create/list/update with CAS.....), this RPCs will
> call
> > > the
> > > > underlying LedgerManager
> > > >
> > >
> > > I think this is in general very good approach to take. It will make the
> > > bookie client not depends on metadata store directly and delegating the
> > > heavy work to the server (bookie side). Once the client is thin, it is
> > easy
> > > to implement clients in other languages.
> > >
> > > However, I do not think it is necessarily for your use case. You can
> make
> > > this as a separate topic. For RPC, I would suggest exploring gPRC (it
> is
> > > protobuf + netty), that would be much better for metadata rpcs rather
> > than
> > > implementing the logic in bookie protocol.
> > >
> > >
> > > >
> > > > On the client side we will not use ZK but execute RPCs on the Bookie
> > > > endpoint (a special LedgerManager which do not use ZK but uses RPCs)
> > > >
> > > > On the Bookie side we will have a local ledger metadata management
> > which
> > > > persists data on disk, we could use existing Key-Value stores, like
> > > RocksDB
> > > > or MapDB, or what ever we prefer, we need something which supports
> CAS
> > > and
> > > > eventually let us put bounds on used memory.
> > > >
> > >
> > > In your use case, you might be able to just implement a ledger manager
> > > using an embedded k/v.
> > >
> > >
> > > >
> > > > Does this make sense to you ?
> > > >
> > > > -- Enrico
> > > >
> > >
> >
> >
> >
> > --
> > Jvrao
> > ---
> > First they ignore you, then they laugh at you, then they fight you, then
> > you win. - Mahatma Gandhi
> >
> --
>
>
> -- Enrico Olivelli
>

Re: ZooKeeper-less - single bookie setup

[Enrico Olivelli <eo...@gmail.com>]

Il gio 27 lug 2017, 17:27 Venkateswara Rao Jujjuri <ju...@gmail.com> ha
scritto:

> We have inMemoryMetaStore.java already to store metadata locally isn't it?
> there are some tests that make use of that too. So if you don't need ZK for
> consensus
> and you can use this inMemoryMetaStore for metadata store.
>

MetadataStore should share data between clients and server, as
InMemoryMetaStore  is only in memory it is not useful.
We need a metadatastore on the bookie, with data persistence and memory
bounds, and on the client side we will access those metadata thru the
bookie.

Enrico



> On Thu, Jul 27, 2017 at 12:42 AM, Sijie Guo <gu...@gmail.com> wrote:
>
> > it might be better to call it 'local model' or 'standalone mode', rather
> > than 'zookeeper-less'.
> >
> > On Wed, Jul 26, 2017 at 8:26 PM, Enrico Olivelli <eo...@gmail.com>
> > wrote:
> >
> > > Hi,
> > > I would like to share this idea with you before starting a real design
> > > effort for this new configuration.
> >
> >
> > > Sometimes I have to run Bookie + BookKeeper inside the same JVM,
> > > essentially because I want my applications to run both in "cluster
> mode"
> > > and in "single instance mode".
> > > I have some other use case of embedding the Bookie, but this is not the
> > > scope of this email.
> > >
> > > In order to run BookKeeper we need ZooKeeper and running ZooKeeper
> inside
> > > the same process is tricky and you know, it is not the suggested
> > > configuration, due to several aspects (out of the scope of this email
> > > too....)
> > >
> > > I would like to make it run without ZooKeeper at all.
> > > This idea will support my use case (Bookie + Client inside the same
> JVM)
> > > and the single Bookie use case, which is a feature that could be useful
> > in
> > > general.
> > > Some other projects, like HBase, have "single server" vs "multi server"
> > > setup, and so I think it will be a good idea for the project and for
> > > gathering more interest and use cases.
> > >
> > > We are using ZooKeeper for (at least) these purposes:
> > > - bookie discovery
> > > - ledger metadata management
> > > - auditor leader election
> > > - global configuration settings  (like autorecovery
> enabled/disabled....)
> > >
> > > my case is to have only one Bookie, so things are really simpler, no
> > > auditor, no shared configuration, no underreplication/overreplication,
> > no
> > > need for 'discovery'
> > >
> > > In order to address the bookie discovery we can just have some
> > client-side
> > > configuration to set the address of the Bookie and have some special
> > > PlacementPolicy which only returns the same bookie.
> > >
> > > The big thing is to implement the ledger metadata management, because
> > both
> > > the Bookie and the client must deal with it.
> > > My idea is to add to the Bookie protocol all the functions to access
> > > LedgersMetadata (create/list/update with CAS.....), this RPCs will call
> > the
> > > underlying LedgerManager
> > >
> >
> > I think this is in general very good approach to take. It will make the
> > bookie client not depends on metadata store directly and delegating the
> > heavy work to the server (bookie side). Once the client is thin, it is
> easy
> > to implement clients in other languages.
> >
> > However, I do not think it is necessarily for your use case. You can make
> > this as a separate topic. For RPC, I would suggest exploring gPRC (it is
> > protobuf + netty), that would be much better for metadata rpcs rather
> than
> > implementing the logic in bookie protocol.
> >
> >
> > >
> > > On the client side we will not use ZK but execute RPCs on the Bookie
> > > endpoint (a special LedgerManager which do not use ZK but uses RPCs)
> > >
> > > On the Bookie side we will have a local ledger metadata management
> which
> > > persists data on disk, we could use existing Key-Value stores, like
> > RocksDB
> > > or MapDB, or what ever we prefer, we need something which supports CAS
> > and
> > > eventually let us put bounds on used memory.
> > >
> >
> > In your use case, you might be able to just implement a ledger manager
> > using an embedded k/v.
> >
> >
> > >
> > > Does this make sense to you ?
> > >
> > > -- Enrico
> > >
> >
>
>
>
> --
> Jvrao
> ---
> First they ignore you, then they laugh at you, then they fight you, then
> you win. - Mahatma Gandhi
>
-- 


-- Enrico Olivelli

Re: ZooKeeper-less - single bookie setup

[Venkateswara Rao Jujjuri <ju...@gmail.com>]

We have inMemoryMetaStore.java already to store metadata locally isn't it?
there are some tests that make use of that too. So if you don't need ZK for
consensus
and you can use this inMemoryMetaStore for metadata store.

On Thu, Jul 27, 2017 at 12:42 AM, Sijie Guo <gu...@gmail.com> wrote:

> it might be better to call it 'local model' or 'standalone mode', rather
> than 'zookeeper-less'.
>
> On Wed, Jul 26, 2017 at 8:26 PM, Enrico Olivelli <eo...@gmail.com>
> wrote:
>
> > Hi,
> > I would like to share this idea with you before starting a real design
> > effort for this new configuration.
>
>
> > Sometimes I have to run Bookie + BookKeeper inside the same JVM,
> > essentially because I want my applications to run both in "cluster mode"
> > and in "single instance mode".
> > I have some other use case of embedding the Bookie, but this is not the
> > scope of this email.
> >
> > In order to run BookKeeper we need ZooKeeper and running ZooKeeper inside
> > the same process is tricky and you know, it is not the suggested
> > configuration, due to several aspects (out of the scope of this email
> > too....)
> >
> > I would like to make it run without ZooKeeper at all.
> > This idea will support my use case (Bookie + Client inside the same JVM)
> > and the single Bookie use case, which is a feature that could be useful
> in
> > general.
> > Some other projects, like HBase, have "single server" vs "multi server"
> > setup, and so I think it will be a good idea for the project and for
> > gathering more interest and use cases.
> >
> > We are using ZooKeeper for (at least) these purposes:
> > - bookie discovery
> > - ledger metadata management
> > - auditor leader election
> > - global configuration settings  (like autorecovery enabled/disabled....)
> >
> > my case is to have only one Bookie, so things are really simpler, no
> > auditor, no shared configuration, no underreplication/overreplication,
> no
> > need for 'discovery'
> >
> > In order to address the bookie discovery we can just have some
> client-side
> > configuration to set the address of the Bookie and have some special
> > PlacementPolicy which only returns the same bookie.
> >
> > The big thing is to implement the ledger metadata management, because
> both
> > the Bookie and the client must deal with it.
> > My idea is to add to the Bookie protocol all the functions to access
> > LedgersMetadata (create/list/update with CAS.....), this RPCs will call
> the
> > underlying LedgerManager
> >
>
> I think this is in general very good approach to take. It will make the
> bookie client not depends on metadata store directly and delegating the
> heavy work to the server (bookie side). Once the client is thin, it is easy
> to implement clients in other languages.
>
> However, I do not think it is necessarily for your use case. You can make
> this as a separate topic. For RPC, I would suggest exploring gPRC (it is
> protobuf + netty), that would be much better for metadata rpcs rather than
> implementing the logic in bookie protocol.
>
>
> >
> > On the client side we will not use ZK but execute RPCs on the Bookie
> > endpoint (a special LedgerManager which do not use ZK but uses RPCs)
> >
> > On the Bookie side we will have a local ledger metadata management which
> > persists data on disk, we could use existing Key-Value stores, like
> RocksDB
> > or MapDB, or what ever we prefer, we need something which supports CAS
> and
> > eventually let us put bounds on used memory.
> >
>
> In your use case, you might be able to just implement a ledger manager
> using an embedded k/v.
>
>
> >
> > Does this make sense to you ?
> >
> > -- Enrico
> >
>



-- 
Jvrao
---
First they ignore you, then they laugh at you, then they fight you, then
you win. - Mahatma Gandhi

Re: ZooKeeper-less - single bookie setup

[Sijie Guo <gu...@gmail.com>]

it might be better to call it 'local model' or 'standalone mode', rather
than 'zookeeper-less'.

On Wed, Jul 26, 2017 at 8:26 PM, Enrico Olivelli <eo...@gmail.com>
wrote:

> Hi,
> I would like to share this idea with you before starting a real design
> effort for this new configuration.


> Sometimes I have to run Bookie + BookKeeper inside the same JVM,
> essentially because I want my applications to run both in "cluster mode"
> and in "single instance mode".
> I have some other use case of embedding the Bookie, but this is not the
> scope of this email.
>
> In order to run BookKeeper we need ZooKeeper and running ZooKeeper inside
> the same process is tricky and you know, it is not the suggested
> configuration, due to several aspects (out of the scope of this email
> too....)
>
> I would like to make it run without ZooKeeper at all.
> This idea will support my use case (Bookie + Client inside the same JVM)
> and the single Bookie use case, which is a feature that could be useful in
> general.
> Some other projects, like HBase, have "single server" vs "multi server"
> setup, and so I think it will be a good idea for the project and for
> gathering more interest and use cases.
>
> We are using ZooKeeper for (at least) these purposes:
> - bookie discovery
> - ledger metadata management
> - auditor leader election
> - global configuration settings  (like autorecovery enabled/disabled....)
>
> my case is to have only one Bookie, so things are really simpler, no
> auditor, no shared configuration, no underreplication/overreplication, no
> need for 'discovery'
>
> In order to address the bookie discovery we can just have some client-side
> configuration to set the address of the Bookie and have some special
> PlacementPolicy which only returns the same bookie.
>
> The big thing is to implement the ledger metadata management, because both
> the Bookie and the client must deal with it.
> My idea is to add to the Bookie protocol all the functions to access
> LedgersMetadata (create/list/update with CAS.....), this RPCs will call the
> underlying LedgerManager
>

I think this is in general very good approach to take. It will make the
bookie client not depends on metadata store directly and delegating the
heavy work to the server (bookie side). Once the client is thin, it is easy
to implement clients in other languages.

However, I do not think it is necessarily for your use case. You can make
this as a separate topic. For RPC, I would suggest exploring gPRC (it is
protobuf + netty), that would be much better for metadata rpcs rather than
implementing the logic in bookie protocol.


>
> On the client side we will not use ZK but execute RPCs on the Bookie
> endpoint (a special LedgerManager which do not use ZK but uses RPCs)
>
> On the Bookie side we will have a local ledger metadata management which
> persists data on disk, we could use existing Key-Value stores, like RocksDB
> or MapDB, or what ever we prefer, we need something which supports CAS and
> eventually let us put bounds on used memory.
>

In your use case, you might be able to just implement a ledger manager
using an embedded k/v.


>
> Does this make sense to you ?
>
> -- Enrico
>