You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@bookkeeper.apache.org by Enrico Olivelli - Diennea <en...@diennea.com> on 2016/06/17 10:00:18 UTC

Embedded Bookie without ZooKeepeer and without Network

Hi BookKeepers,
I'm running several tests using the new "VM-local transport" option, one of the intents of https://issues.apache.org/jira/browse/BOOKKEEPER-896 was to have the ability to leverage Apache BookKeeper features inside a single-process.

Usuallly I have to write a BookKeeper-based transaction log, for distributed execution, and a local-process transaction log which writes to local files for standalone deployments.

The first result of my bench it that using BookKeeper inside the JVM is really faster that using simple file-based transaction logs, the more parallelism the best results.  This is not a surprise, BookKeeper is made for performance.

But actually it seems to me that is very better to embed BookKeeper instead of re-writing simple transaction-log managers

The road to leverage Bookie features without network in my opinion is:

First step:
I'm going to submit a PR for  https://issues.apache.org/jira/browse/BOOKKEEPER-930 (Option to disable Bookie networking)
Can we add it to 4.5.0 ?

Second step:
we need ZooKeeper in order to run a Bookie and a Client, it would be great to have the ability to implement a replacement, a least for the single-machine, embbeded usage
Can I file a JIRA and start gathering ideas ?

alternatively we could as on ZooKeeper project to implement something like "VM-local transport"

What to you think ?


--
Enrico Olivelli
Software Development Manager @Diennea
Tel.: (+39) 0546 066100 - Int. 925
Viale G.Marconi 30/14 - 48018 Faenza (RA)

MagNews - E-mail Marketing Solutions
http://www.magnews.it
Diennea - Digital Marketing Solutions
http://www.diennea.com


________________________________

Iscriviti alla nostra newsletter per rimanere aggiornato su digital ed email marketing! http://www.magnews.it/newsletter/

The information in this email is confidential and may be legally privileged. If you are not the intended recipient please notify the sender immediately and destroy this email. Any unauthorized, direct or indirect, disclosure, copying, storage, distribution or other use is strictly forbidden.

________________________________

[http://www.magnews.it/wp-content/uploads/2016/06/img.jpg] <http://www.magnews.it/2016/02/22/global-summit-marketing-digital-2016/?utm_source=footer-email&utm_medium=email&utm_campaign=global-summit>

Re: Embedded Bookie without ZooKeepeer and without Network

Posted by Enrico Olivelli - Diennea <en...@diennea.com>.

Il giorno ven, 17/06/2016 alle 17.24 -0700, Sijie Guo ha scritto:

On Fri, Jun 17, 2016 at 3:00 AM, Enrico Olivelli - Diennea <en...@diennea.com>> wrote:
Hi BookKeepers,
I'm running several tests using the new "VM-local transport" option, one of the intents of https://issues.apache.org/jira/browse/BOOKKEEPER-896 was to have the ability to leverage Apache BookKeeper features inside a single-process.

Usuallly I have to write a BookKeeper-based transaction log, for distributed execution, and a local-process transaction log which writes to local files for standalone deployments.

The first result of my bench it that using BookKeeper inside the JVM is really faster that using simple file-based transaction logs, the more parallelism the best results. This is not a surprise, BookKeeper is made for performance.

But actually it seems to me that is very better to embed BookKeeper instead of re-writing simple transaction-log managers

The road to leverage Bookie features without network in my opinion is:

I liked the idea to run bookkeeper in 'embed' mode.

First step:
I'm going to submit a PR for https://issues.apache.org/jira/browse/BOOKKEEPER-930 (Option to disable Bookie networking)
Can we add it to 4.5.0 ?

+1 for this.

Second step:
we need ZooKeeper in order to run a Bookie and a Client, it would be great to have the ability to implement a replacement, a least for the single-machine, embbeded usage
Can I file a JIRA and start gathering ideas ?

The zookeeper is used for two parts in bookkeeper:

- one is service discovery: to discover how many bookies are available. Since it is local/embed mode, you don't actually need service discovery. There is a ticket to abstract the bookie registration, so we can hide zookeeper: https://issues.apache.org/jira/browse/BOOKKEEPER-628 If you are interested, you can pickup the work from Rakesh.

I'm very interested, I'll take a look at Rakesh's patch

thanks

- one is for metadata: all the metadata operations are hidden by LedgerManager interface. so it is easier to just implement a local LedgerManager. In-memory LedgerManager for local testing, or using an embed key/value db like rocks-db.

alternatively we could as on ZooKeeper project to implement something like "VM-local transport"

What to you think ?

--
Enrico Olivelli
Software Development Manager @Diennea
Tel.: (+39) 0546 066100<tel:%28%2B39%29%200546%20066100> - Int. 925
Viale G.Marconi 30/14 - 48018 Faenza (RA)

MagNews - E-mail Marketing Solutions
http://www.magnews.it
Diennea - Digital Marketing Solutions
http://www.diennea.com

________________________________

Iscriviti alla nostra newsletter per rimanere aggiornato su digital ed email marketing! http://www.magnews.it/newsletter/

The information in this email is confidential and may be legally privileged. If you are not the intended recipient please notify the sender immediately and destroy this email. Any unauthorized, direct or indirect, disclosure, copying, storage, distribution or other use is strictly forbidden.

________________________________

[http://www.magnews.it/wp-content/uploads/2016/06/img.jpg] <http://www.magnews.it/2016/02/22/global-summit-marketing-digital-2016/?utm_source=footer-email&utm_medium=email&utm_campaign=global-summit>

--
Enrico Olivelli
Software Development Manager @Diennea
Tel.: (+39) 0546 066100 - Int. 925
Viale G.Marconi 30/14 - 48018 Faenza (RA)

MagNews - E-mail Marketing Solutions
http://www.magnews.it
Diennea - Digital Marketing Solutions
http://www.diennea.com

________________________________

Iscriviti alla nostra newsletter per rimanere aggiornato su digital ed email marketing! http://www.magnews.it/newsletter/

________________________________

[http://www.magnews.it/wp-content/uploads/2016/06/img.jpg] <http://www.magnews.it/2016/02/22/global-summit-marketing-digital-2016/?utm_source=footer-email&utm_medium=email&utm_campaign=global-summit>

Re: Embedded Bookie without ZooKeepeer and without Network

Posted by Sijie Guo <si...@apache.org>.

On Fri, Jun 17, 2016 at 3:00 AM, Enrico Olivelli - Diennea <
enrico.olivelli@diennea.com> wrote:

> Hi BookKeepers,
> I'm running several tests using the new "VM-local transport" option, one
> of the intents of https://issues.apache.org/jira/browse/BOOKKEEPER-896
> was to have the ability to leverage Apache BookKeeper features inside a
> single-process.
>
> Usuallly I have to write a BookKeeper-based transaction log, for
> distributed execution, and a local-process transaction log which writes to
> local files for standalone deployments.
>
> The first result of my bench it that using BookKeeper inside the JVM is
> really faster that using simple file-based transaction logs, the more
> parallelism the best results.  This is not a surprise, BookKeeper is made
> for performance.
>
> But actually it seems to me that is very better to embed BookKeeper
> instead of re-writing simple transaction-log managers
>
> The road to leverage Bookie features without network in my opinion is:
>

I liked the idea to run bookkeeper in 'embed' mode.


>
> First step:
> I'm going to submit a PR for
> https://issues.apache.org/jira/browse/BOOKKEEPER-930 (Option to disable
> Bookie networking)
> Can we add it to 4.5.0 ?
>

+1 for this.


>
> Second step:
> we need ZooKeeper in order to run a Bookie and a Client, it would be great
> to have the ability to implement a replacement, a least for the
> single-machine, embbeded usage
> Can I file a JIRA and start gathering ideas ?
>

The zookeeper is used for two parts in bookkeeper:

- one is service discovery: to discover how many bookies are available.
Since it is local/embed mode, you don't actually need service discovery.
There is a ticket to abstract the bookie registration, so we can hide
zookeeper: https://issues.apache.org/jira/browse/BOOKKEEPER-628 If you are
interested, you can pickup the work from Rakesh.

- one is for metadata: all the metadata operations are hidden by
LedgerManager interface. so it is easier to just implement a local
LedgerManager. In-memory LedgerManager for local testing, or using an embed
key/value db like rocks-db.


>
> alternatively we could as on ZooKeeper project to implement something like
> "VM-local transport"
>
> What to you think ?
>
>
> --
> Enrico Olivelli
> Software Development Manager @Diennea
> Tel.: (+39) 0546 066100 - Int. 925
> Viale G.Marconi 30/14 - 48018 Faenza (RA)
>
> MagNews - E-mail Marketing Solutions
> http://www.magnews.it
> Diennea - Digital Marketing Solutions
> http://www.diennea.com
>
>
> ________________________________
>
> Iscriviti alla nostra newsletter per rimanere aggiornato su digital ed
> email marketing! http://www.magnews.it/newsletter/
>
> The information in this email is confidential and may be legally
> privileged. If you are not the intended recipient please notify the sender
> immediately and destroy this email. Any unauthorized, direct or indirect,
> disclosure, copying, storage, distribution or other use is strictly
> forbidden.
>
> ________________________________
>
> [http://www.magnews.it/wp-content/uploads/2016/06/img.jpg] <
> http://www.magnews.it/2016/02/22/global-summit-marketing-digital-2016/?utm_source=footer-email&utm_medium=email&utm_campaign=global-summit
> >
>
>