You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pulsar.apache.org by Matteo Merli <mm...@apache.org> on 2017/07/14 22:26:13 UTC
[DISCUSS] Make RocksDb dependency optional
As we have already talked about, the RocksDb library has been put in the
Category-X (Forbidden) by the ASF, due to licensing problems.
With my very limited understanding of the issue, it seems the problem
resides in the patent clauses that's added the license.
https://www.apache.org/legal/resolved.html#category-x
------------------------------------------------------------------------
Rocks DB License
The Rocks DB license includes a specification of a PATENTS file that passes
along risk to downstream consumers of our software imbalanced in favor of
the licensor, not the licensee, thereby violating our Apache legal policy
of being a universal donor <https://s.apache.org/4Uzg>. The terms of
ROCKSDB license are not a subset of those found in the ALv2, and they
cannot be sublicensed as ALv2.
------------------------------------------------------------------------
Pulsar is using RocksDb since that is being pulled in from the Yahoo branch
of BookKeeper, which includes an alternative LedgerStorage implementation
that stores the indexes for the entries in a RocksDB databases instead of
using individual index files as it's default for BookKeeper. That is
important to store many tens of thousands of ledgers per single Bookie node.
To be able to release "Apache Pulsar" we need to fix the dependency
problem. These are the options that I have been thinking of (feel free to
propose alternatives! ):
1. Configure out of the box Pulsar to use InterleavedLedgerStorage and
explicitly exclude the RocksDb dependency from BookKeeper (yahoo branch)
2. Provide an additional key-value storage implementation for the
DbLedgerStorage (eg: LevelDb) and make that the default
3. Wait until there's more clarity on the subject (that would mean to
delay the Pulsar release)
As I see it, I prefer option #1 for now, because I don't see any comparable
replacement for RocksDB. LevelDb works fine for small workloads but show
several problems at higher load, and if we include it, we would need to
support it.
The main concern I have is related to the out-of-the-box performance. We
need to document how to enable the RocksDb configuration and we should make
clear that the published results were accomplished with that config.
Additionally, since the ultimate goal for the DbLedgerStorage was to be
merged into BookKeeper, it would be good to understand what are the options
on that side.
For example, will it be fine to have RocksDb as an optional dependency
(needed to compile from source, but not used or included in binary
distribution) ?
That way we could include the KeyValueRocksDb java class that uses the API
and compile it. Then, since the dependency is optional, it would require
the user to explicitly include RocksDB to have it working.
What do you think?
Matteo
--
Matteo Merli
<mm...@apache.org>
Re: [DISCUSS] Make RocksDb dependency optional
Posted by Matteo Merli <mm...@apache.org>.
I have just created a PR to upgrade RocksDB and resolve the license issue.
https://github.com/apache/incubator-pulsar/pull/587
We don't need to wait on BK release, since we can force which RocksDB
version we want to include.
--
Matteo Merli
<mm...@apache.org>
On Tue, Jul 18, 2017 at 2:11 PM, Matteo Merli <mm...@apache.org> wrote:
> Dave,
> I think RocksDB is cutting a new release 5.5.4 with Apache License right
> now :
>
> https://github.com/facebook/rocksdb/commits/5.5.fb
>
> Thankfully they're moving quickly to solve the issue!
>
> Matteo
>
> --
> Matteo Merli
> <mm...@apache.org>
>
> On Tue, Jul 18, 2017 at 9:17 AM, Dave Fisher <da...@comcast.net>
> wrote:
>
>> Hi -
>>
>> Current guidance is to wait for the relicensed version of RocksDB before
>> it can be allowed. Projects currently have until August 31 to resolve the
>> issues in Apache releases. Pulsar does not yet have an Apache release.
>>
>> It becomes a question of timing within RocksDB, Bookkeeper and this
>> podling. If the podling gets to the point of releasing before the proper
>> licensed version is available then following Matteo’s suggestion is
>> probably needed. That wouldn’t stop the project from publishing benchmark
>> data based on. RocksDB.
>>
>> Regards,
>> Dave
>>
>> > On Jul 16, 2017, at 12:02 PM, Dave Fisher <da...@comcast.net>
>> wrote:
>> >
>> > Hi -
>> >
>> > Please see https://issues.apache.org/jira/plugins/servlet/mobile#issue/
>> LEGAL-303/comment/16088730
>> >
>> > It looks like the RocksDB community is resolving the issue!
>> >
>> > Regards,
>> > Dave
>> >
>> >
>> > Sent from my iPhone
>> >
>> >> On Jul 14, 2017, at 3:26 PM, Matteo Merli <mm...@apache.org> wrote:
>> >>
>> >> As we have already talked about, the RocksDb library has been put in
>> the
>> >> Category-X (Forbidden) by the ASF, due to licensing problems.
>> >>
>> >> With my very limited understanding of the issue, it seems the problem
>> >> resides in the patent clauses that's added the license.
>> >>
>> >> https://www.apache.org/legal/resolved.html#category-x
>> >> ------------------------------------------------------------
>> ------------
>> >> Rocks DB License
>> >>
>> >> The Rocks DB license includes a specification of a PATENTS file that
>> passes
>> >> along risk to downstream consumers of our software imbalanced in favor
>> of
>> >> the licensor, not the licensee, thereby violating our Apache legal
>> policy
>> >> of being a universal donor <https://s.apache.org/4Uzg>. The terms of
>> >> ROCKSDB license are not a subset of those found in the ALv2, and they
>> >> cannot be sublicensed as ALv2.
>> >> ------------------------------------------------------------
>> ------------
>> >>
>> >> Pulsar is using RocksDb since that is being pulled in from the Yahoo
>> branch
>> >> of BookKeeper, which includes an alternative LedgerStorage
>> implementation
>> >> that stores the indexes for the entries in a RocksDB databases instead
>> of
>> >> using individual index files as it's default for BookKeeper. That is
>> >> important to store many tens of thousands of ledgers per single Bookie
>> node.
>> >>
>> >> To be able to release "Apache Pulsar" we need to fix the dependency
>> >> problem. These are the options that I have been thinking of (feel free
>> to
>> >> propose alternatives! ):
>> >>
>> >> 1. Configure out of the box Pulsar to use InterleavedLedgerStorage and
>> >> explicitly exclude the RocksDb dependency from BookKeeper (yahoo
>> branch)
>> >>
>> >> 2. Provide an additional key-value storage implementation for the
>> >> DbLedgerStorage (eg: LevelDb) and make that the default
>> >>
>> >> 3. Wait until there's more clarity on the subject (that would mean to
>> >> delay the Pulsar release)
>> >>
>> >> As I see it, I prefer option #1 for now, because I don't see any
>> comparable
>> >> replacement for RocksDB. LevelDb works fine for small workloads but
>> show
>> >> several problems at higher load, and if we include it, we would need to
>> >> support it.
>> >>
>> >> The main concern I have is related to the out-of-the-box performance.
>> We
>> >> need to document how to enable the RocksDb configuration and we should
>> make
>> >> clear that the published results were accomplished with that config.
>> >>
>> >> Additionally, since the ultimate goal for the DbLedgerStorage was to be
>> >> merged into BookKeeper, it would be good to understand what are the
>> options
>> >> on that side.
>> >> For example, will it be fine to have RocksDb as an optional dependency
>> >> (needed to compile from source, but not used or included in binary
>> >> distribution) ?
>> >> That way we could include the KeyValueRocksDb java class that uses the
>> API
>> >> and compile it. Then, since the dependency is optional, it would
>> require
>> >> the user to explicitly include RocksDB to have it working.
>> >>
>> >>
>> >> What do you think?
>> >> Matteo
>> >>
>> >>
>> >> --
>> >> Matteo Merli
>> >> <mm...@apache.org>
>>
>>
>
Re: [DISCUSS] Make RocksDb dependency optional
Posted by Matteo Merli <mm...@apache.org>.
Dave,
I think RocksDB is cutting a new release 5.5.4 with Apache License right
now :
https://github.com/facebook/rocksdb/commits/5.5.fb
Thankfully they're moving quickly to solve the issue!
Matteo
--
Matteo Merli
<mm...@apache.org>
On Tue, Jul 18, 2017 at 9:17 AM, Dave Fisher <da...@comcast.net> wrote:
> Hi -
>
> Current guidance is to wait for the relicensed version of RocksDB before
> it can be allowed. Projects currently have until August 31 to resolve the
> issues in Apache releases. Pulsar does not yet have an Apache release.
>
> It becomes a question of timing within RocksDB, Bookkeeper and this
> podling. If the podling gets to the point of releasing before the proper
> licensed version is available then following Matteo’s suggestion is
> probably needed. That wouldn’t stop the project from publishing benchmark
> data based on. RocksDB.
>
> Regards,
> Dave
>
> > On Jul 16, 2017, at 12:02 PM, Dave Fisher <da...@comcast.net> wrote:
> >
> > Hi -
> >
> > Please see https://issues.apache.org/jira/plugins/servlet/mobile#
> issue/LEGAL-303/comment/16088730
> >
> > It looks like the RocksDB community is resolving the issue!
> >
> > Regards,
> > Dave
> >
> >
> > Sent from my iPhone
> >
> >> On Jul 14, 2017, at 3:26 PM, Matteo Merli <mm...@apache.org> wrote:
> >>
> >> As we have already talked about, the RocksDb library has been put in the
> >> Category-X (Forbidden) by the ASF, due to licensing problems.
> >>
> >> With my very limited understanding of the issue, it seems the problem
> >> resides in the patent clauses that's added the license.
> >>
> >> https://www.apache.org/legal/resolved.html#category-x
> >> ------------------------------------------------------------
> ------------
> >> Rocks DB License
> >>
> >> The Rocks DB license includes a specification of a PATENTS file that
> passes
> >> along risk to downstream consumers of our software imbalanced in favor
> of
> >> the licensor, not the licensee, thereby violating our Apache legal
> policy
> >> of being a universal donor <https://s.apache.org/4Uzg>. The terms of
> >> ROCKSDB license are not a subset of those found in the ALv2, and they
> >> cannot be sublicensed as ALv2.
> >> ------------------------------------------------------------
> ------------
> >>
> >> Pulsar is using RocksDb since that is being pulled in from the Yahoo
> branch
> >> of BookKeeper, which includes an alternative LedgerStorage
> implementation
> >> that stores the indexes for the entries in a RocksDB databases instead
> of
> >> using individual index files as it's default for BookKeeper. That is
> >> important to store many tens of thousands of ledgers per single Bookie
> node.
> >>
> >> To be able to release "Apache Pulsar" we need to fix the dependency
> >> problem. These are the options that I have been thinking of (feel free
> to
> >> propose alternatives! ):
> >>
> >> 1. Configure out of the box Pulsar to use InterleavedLedgerStorage and
> >> explicitly exclude the RocksDb dependency from BookKeeper (yahoo branch)
> >>
> >> 2. Provide an additional key-value storage implementation for the
> >> DbLedgerStorage (eg: LevelDb) and make that the default
> >>
> >> 3. Wait until there's more clarity on the subject (that would mean to
> >> delay the Pulsar release)
> >>
> >> As I see it, I prefer option #1 for now, because I don't see any
> comparable
> >> replacement for RocksDB. LevelDb works fine for small workloads but show
> >> several problems at higher load, and if we include it, we would need to
> >> support it.
> >>
> >> The main concern I have is related to the out-of-the-box performance. We
> >> need to document how to enable the RocksDb configuration and we should
> make
> >> clear that the published results were accomplished with that config.
> >>
> >> Additionally, since the ultimate goal for the DbLedgerStorage was to be
> >> merged into BookKeeper, it would be good to understand what are the
> options
> >> on that side.
> >> For example, will it be fine to have RocksDb as an optional dependency
> >> (needed to compile from source, but not used or included in binary
> >> distribution) ?
> >> That way we could include the KeyValueRocksDb java class that uses the
> API
> >> and compile it. Then, since the dependency is optional, it would require
> >> the user to explicitly include RocksDB to have it working.
> >>
> >>
> >> What do you think?
> >> Matteo
> >>
> >>
> >> --
> >> Matteo Merli
> >> <mm...@apache.org>
>
>
Re: [DISCUSS] Make RocksDb dependency optional
Posted by Dave Fisher <da...@comcast.net>.
Hi -
Current guidance is to wait for the relicensed version of RocksDB before it can be allowed. Projects currently have until August 31 to resolve the issues in Apache releases. Pulsar does not yet have an Apache release.
It becomes a question of timing within RocksDB, Bookkeeper and this podling. If the podling gets to the point of releasing before the proper licensed version is available then following Matteo’s suggestion is probably needed. That wouldn’t stop the project from publishing benchmark data based on. RocksDB.
Regards,
Dave
> On Jul 16, 2017, at 12:02 PM, Dave Fisher <da...@comcast.net> wrote:
>
> Hi -
>
> Please see https://issues.apache.org/jira/plugins/servlet/mobile#issue/LEGAL-303/comment/16088730
>
> It looks like the RocksDB community is resolving the issue!
>
> Regards,
> Dave
>
>
> Sent from my iPhone
>
>> On Jul 14, 2017, at 3:26 PM, Matteo Merli <mm...@apache.org> wrote:
>>
>> As we have already talked about, the RocksDb library has been put in the
>> Category-X (Forbidden) by the ASF, due to licensing problems.
>>
>> With my very limited understanding of the issue, it seems the problem
>> resides in the patent clauses that's added the license.
>>
>> https://www.apache.org/legal/resolved.html#category-x
>> ------------------------------------------------------------------------
>> Rocks DB License
>>
>> The Rocks DB license includes a specification of a PATENTS file that passes
>> along risk to downstream consumers of our software imbalanced in favor of
>> the licensor, not the licensee, thereby violating our Apache legal policy
>> of being a universal donor <https://s.apache.org/4Uzg>. The terms of
>> ROCKSDB license are not a subset of those found in the ALv2, and they
>> cannot be sublicensed as ALv2.
>> ------------------------------------------------------------------------
>>
>> Pulsar is using RocksDb since that is being pulled in from the Yahoo branch
>> of BookKeeper, which includes an alternative LedgerStorage implementation
>> that stores the indexes for the entries in a RocksDB databases instead of
>> using individual index files as it's default for BookKeeper. That is
>> important to store many tens of thousands of ledgers per single Bookie node.
>>
>> To be able to release "Apache Pulsar" we need to fix the dependency
>> problem. These are the options that I have been thinking of (feel free to
>> propose alternatives! ):
>>
>> 1. Configure out of the box Pulsar to use InterleavedLedgerStorage and
>> explicitly exclude the RocksDb dependency from BookKeeper (yahoo branch)
>>
>> 2. Provide an additional key-value storage implementation for the
>> DbLedgerStorage (eg: LevelDb) and make that the default
>>
>> 3. Wait until there's more clarity on the subject (that would mean to
>> delay the Pulsar release)
>>
>> As I see it, I prefer option #1 for now, because I don't see any comparable
>> replacement for RocksDB. LevelDb works fine for small workloads but show
>> several problems at higher load, and if we include it, we would need to
>> support it.
>>
>> The main concern I have is related to the out-of-the-box performance. We
>> need to document how to enable the RocksDb configuration and we should make
>> clear that the published results were accomplished with that config.
>>
>> Additionally, since the ultimate goal for the DbLedgerStorage was to be
>> merged into BookKeeper, it would be good to understand what are the options
>> on that side.
>> For example, will it be fine to have RocksDb as an optional dependency
>> (needed to compile from source, but not used or included in binary
>> distribution) ?
>> That way we could include the KeyValueRocksDb java class that uses the API
>> and compile it. Then, since the dependency is optional, it would require
>> the user to explicitly include RocksDB to have it working.
>>
>>
>> What do you think?
>> Matteo
>>
>>
>> --
>> Matteo Merli
>> <mm...@apache.org>
Re: [DISCUSS] Make RocksDb dependency optional
Posted by Dave Fisher <da...@comcast.net>.
Hi -
Please see https://issues.apache.org/jira/plugins/servlet/mobile#issue/LEGAL-303/comment/16088730
It looks like the RocksDB community is resolving the issue!
Regards,
Dave
Sent from my iPhone
> On Jul 14, 2017, at 3:26 PM, Matteo Merli <mm...@apache.org> wrote:
>
> As we have already talked about, the RocksDb library has been put in the
> Category-X (Forbidden) by the ASF, due to licensing problems.
>
> With my very limited understanding of the issue, it seems the problem
> resides in the patent clauses that's added the license.
>
> https://www.apache.org/legal/resolved.html#category-x
> ------------------------------------------------------------------------
> Rocks DB License
>
> The Rocks DB license includes a specification of a PATENTS file that passes
> along risk to downstream consumers of our software imbalanced in favor of
> the licensor, not the licensee, thereby violating our Apache legal policy
> of being a universal donor <https://s.apache.org/4Uzg>. The terms of
> ROCKSDB license are not a subset of those found in the ALv2, and they
> cannot be sublicensed as ALv2.
> ------------------------------------------------------------------------
>
> Pulsar is using RocksDb since that is being pulled in from the Yahoo branch
> of BookKeeper, which includes an alternative LedgerStorage implementation
> that stores the indexes for the entries in a RocksDB databases instead of
> using individual index files as it's default for BookKeeper. That is
> important to store many tens of thousands of ledgers per single Bookie node.
>
> To be able to release "Apache Pulsar" we need to fix the dependency
> problem. These are the options that I have been thinking of (feel free to
> propose alternatives! ):
>
> 1. Configure out of the box Pulsar to use InterleavedLedgerStorage and
> explicitly exclude the RocksDb dependency from BookKeeper (yahoo branch)
>
> 2. Provide an additional key-value storage implementation for the
> DbLedgerStorage (eg: LevelDb) and make that the default
>
> 3. Wait until there's more clarity on the subject (that would mean to
> delay the Pulsar release)
>
> As I see it, I prefer option #1 for now, because I don't see any comparable
> replacement for RocksDB. LevelDb works fine for small workloads but show
> several problems at higher load, and if we include it, we would need to
> support it.
>
> The main concern I have is related to the out-of-the-box performance. We
> need to document how to enable the RocksDb configuration and we should make
> clear that the published results were accomplished with that config.
>
> Additionally, since the ultimate goal for the DbLedgerStorage was to be
> merged into BookKeeper, it would be good to understand what are the options
> on that side.
> For example, will it be fine to have RocksDb as an optional dependency
> (needed to compile from source, but not used or included in binary
> distribution) ?
> That way we could include the KeyValueRocksDb java class that uses the API
> and compile it. Then, since the dependency is optional, it would require
> the user to explicitly include RocksDB to have it working.
>
>
> What do you think?
> Matteo
>
>
> --
> Matteo Merli
> <mm...@apache.org>
Re: [DISCUSS] Make RocksDb dependency optional
Posted by Matteo Merli <mm...@apache.org>.
FYI: License change in RocksDB is already happening:
https://github.com/facebook/rocksdb/pull/2589
--
Matteo Merli
<mm...@apache.org>
On Sat, Jul 15, 2017 at 3:56 PM, Matteo Merli <mm...@apache.org> wrote:
> There have been some updates on this story:
>
> 1. This morning a mail was sent out to all PMCs instructing to remove
> RocksDb before Aug 31st. That was only for projects that had released code
> using RocksDb and all the other project were prevented from starting using
> it.
> https://issues.apache.org/jira/browse/LEGAL-303?focusedCommentId=16088663&
> page=com.atlassian.jira.plugin.system.issuetabpanels:
> comment-tabpanel#comment-16088663
>
> 2. This afternoon in an update from Facebook, they announced they're going
> to switch to dual license RocksDb in ALv2 + GPL.
> https://issues.apache.org/jira/browse/LEGAL-303?focusedCommentId=16088730&
> page=com.atlassian.jira.plugin.system.issuetabpanels:
> comment-tabpanel#comment-16088730
>
>
> In light of this, we should continue as we are right now and we should be
> able to keep using RocksDB.
>
>
>
>
>
> On Fri, Jul 14, 2017 at 3:26 PM Matteo Merli <mm...@apache.org> wrote:
>
>> As we have already talked about, the RocksDb library has been put in the
>> Category-X (Forbidden) by the ASF, due to licensing problems.
>>
>> With my very limited understanding of the issue, it seems the problem
>> resides in the patent clauses that's added the license.
>>
>> https://www.apache.org/legal/resolved.html#category-x
>> ------------------------------------------------------------------------
>> Rocks DB License
>>
>> The Rocks DB license includes a specification of a PATENTS file that
>> passes along risk to downstream consumers of our software imbalanced in
>> favor of the licensor, not the licensee, thereby violating our Apache legal
>> policy of being a universal donor <https://s.apache.org/4Uzg>. The terms
>> of ROCKSDB license are not a subset of those found in the ALv2, and they
>> cannot be sublicensed as ALv2.
>> ------------------------------------------------------------------------
>>
>> Pulsar is using RocksDb since that is being pulled in from the Yahoo
>> branch of BookKeeper, which includes an alternative LedgerStorage
>> implementation that stores the indexes for the entries in a RocksDB
>> databases instead of using individual index files as it's default for
>> BookKeeper. That is important to store many tens of thousands of ledgers
>> per single Bookie node.
>>
>> To be able to release "Apache Pulsar" we need to fix the dependency
>> problem. These are the options that I have been thinking of (feel free to
>> propose alternatives! ):
>>
>> 1. Configure out of the box Pulsar to use InterleavedLedgerStorage and
>> explicitly exclude the RocksDb dependency from BookKeeper (yahoo branch)
>>
>> 2. Provide an additional key-value storage implementation for the
>> DbLedgerStorage (eg: LevelDb) and make that the default
>>
>> 3. Wait until there's more clarity on the subject (that would mean to
>> delay the Pulsar release)
>>
>> As I see it, I prefer option #1 for now, because I don't see any
>> comparable replacement for RocksDB. LevelDb works fine for small workloads
>> but show several problems at higher load, and if we include it, we would
>> need to support it.
>>
>> The main concern I have is related to the out-of-the-box performance. We
>> need to document how to enable the RocksDb configuration and we should make
>> clear that the published results were accomplished with that config.
>>
>> Additionally, since the ultimate goal for the DbLedgerStorage was to be
>> merged into BookKeeper, it would be good to understand what are the options
>> on that side.
>> For example, will it be fine to have RocksDb as an optional dependency
>> (needed to compile from source, but not used or included in binary
>> distribution) ?
>> That way we could include the KeyValueRocksDb java class that uses the
>> API and compile it. Then, since the dependency is optional, it would
>> require the user to explicitly include RocksDB to have it working.
>>
>>
>> What do you think?
>> Matteo
>>
>>
>> --
>> Matteo Merli
>> <mm...@apache.org>
>>
>
Re: [DISCUSS] Make RocksDb dependency optional
Posted by Matteo Merli <mm...@apache.org>.
There have been some updates on this story:
1. This morning a mail was sent out to all PMCs instructing to remove
RocksDb before Aug 31st. That was only for projects that had released code
using RocksDb and all the other project were prevented from starting using
it.
https://issues.apache.org/jira/browse/LEGAL-303?focusedCommentId=16088663&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16088663
2. This afternoon in an update from Facebook, they announced they're going
to switch to dual license RocksDb in ALv2 + GPL.
https://issues.apache.org/jira/browse/LEGAL-303?focusedCommentId=16088730&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16088730
In light of this, we should continue as we are right now and we should be
able to keep using RocksDB.
On Fri, Jul 14, 2017 at 3:26 PM Matteo Merli <mm...@apache.org> wrote:
> As we have already talked about, the RocksDb library has been put in the
> Category-X (Forbidden) by the ASF, due to licensing problems.
>
> With my very limited understanding of the issue, it seems the problem
> resides in the patent clauses that's added the license.
>
> https://www.apache.org/legal/resolved.html#category-x
> ------------------------------------------------------------------------
> Rocks DB License
>
> The Rocks DB license includes a specification of a PATENTS file that
> passes along risk to downstream consumers of our software imbalanced in
> favor of the licensor, not the licensee, thereby violating our Apache legal
> policy of being a universal donor <https://s.apache.org/4Uzg>. The terms
> of ROCKSDB license are not a subset of those found in the ALv2, and they
> cannot be sublicensed as ALv2.
> ------------------------------------------------------------------------
>
> Pulsar is using RocksDb since that is being pulled in from the Yahoo
> branch of BookKeeper, which includes an alternative LedgerStorage
> implementation that stores the indexes for the entries in a RocksDB
> databases instead of using individual index files as it's default for
> BookKeeper. That is important to store many tens of thousands of ledgers
> per single Bookie node.
>
> To be able to release "Apache Pulsar" we need to fix the dependency
> problem. These are the options that I have been thinking of (feel free to
> propose alternatives! ):
>
> 1. Configure out of the box Pulsar to use InterleavedLedgerStorage and
> explicitly exclude the RocksDb dependency from BookKeeper (yahoo branch)
>
> 2. Provide an additional key-value storage implementation for the
> DbLedgerStorage (eg: LevelDb) and make that the default
>
> 3. Wait until there's more clarity on the subject (that would mean to
> delay the Pulsar release)
>
> As I see it, I prefer option #1 for now, because I don't see any
> comparable replacement for RocksDB. LevelDb works fine for small workloads
> but show several problems at higher load, and if we include it, we would
> need to support it.
>
> The main concern I have is related to the out-of-the-box performance. We
> need to document how to enable the RocksDb configuration and we should make
> clear that the published results were accomplished with that config.
>
> Additionally, since the ultimate goal for the DbLedgerStorage was to be
> merged into BookKeeper, it would be good to understand what are the options
> on that side.
> For example, will it be fine to have RocksDb as an optional dependency
> (needed to compile from source, but not used or included in binary
> distribution) ?
> That way we could include the KeyValueRocksDb java class that uses the API
> and compile it. Then, since the dependency is optional, it would require
> the user to explicitly include RocksDB to have it working.
>
>
> What do you think?
> Matteo
>
>
> --
> Matteo Merli
> <mm...@apache.org>
>