Expected workload for DBLedgerStorage ?

[Enrico Olivelli <eo...@gmail.com>]

Hi,
I am experimenting DBLedgerStorage with some project.
With my workloads I don't see much benefit, or sometimes a performance
degradation, just by keeping the same bookie and application configuration
(just switching the storage manager classe name in the bookie
configuration).

Which is the expected workload for DBLedgerStorage ?
(Usually I am using the default SortedLedgerStorage)

Cheers
Enrico

Re: Expected workload for DBLedgerStorage ?

[Sijie Guo <gu...@gmail.com>]

On Wed, Jun 20, 2018 at 12:45 PM Enrico Olivelli <eo...@gmail.com>
wrote:

>
>
> Il mer 20 giu 2018, 16:52 Sijie Guo <gu...@gmail.com> ha scritto:
>
>> On Wed, Jun 20, 2018 at 4:09 AM Enrico Olivelli <eo...@gmail.com>
>> wrote:
>>
>>> Hi,
>>> I am experimenting DBLedgerStorage with some project.
>>> With my workloads I don't see much benefit, or sometimes a performance
>>> degradation,
>>>
>>
>> Do you have a performance comparison? Or can you describe what have you
>> observed?
>>
>
> No real 'sharable' numbers, just run a bunch of benchmarks of one
> application. My data has no real meaning.
>
>
>
>
>>
>>> just by keeping the same bookie and application configuration (just
>>> switching the storage manager classe name in the bookie configuration).
>>>
>>> Which is the expected workload for DBLedgerStorage ?
>>> (Usually I am using the default SortedLedgerStorage)
>>>
>>
>> Ideally it should be working out better when you have large number of
>> ledgers per bookie (let's say more than 10k ledgers).
>>
>
> My test created only a few ledger ( less then 10) so maybe this is why I
> see no benefit.
>
As far as I understand RocksDB is used for indexes and maybe with a few
> ledgers the overhead is greater than the gain in term of resource saving.
> I have to tune my test with more concurrency.
>

> In this project I will have concurrent readers and writers, but readers
> will access data at any point of ledgers, so no usual 'tailing reads'. I
> will be using ledgers more likes files than like streams.
>

Are you fetching data by entry ids or would you expect to fetch entries by
their *bytes* offset?

In other words, does this https://github.com/apache/bookkeeper/issues/1376
meet your requirement if you are looking for an interface to fetch entries
more about their "bytes" offset?


> So having rocks db to store offsets seems a very good chance for me.
> In real systems I will have thousands of active ledgers per bookie.
> I have to use a more complex suite of benchmarks for the use case.
>
> I will be back with more interesting comparisons
>
> Thank you
> Enrico
>
>
>
>
>>
>>>
>>> Cheers
>>> Enrico
>>>
>>>
>>>
>>> --
>
>
> -- Enrico Olivelli
>

Re: Expected workload for DBLedgerStorage ?

[Enrico Olivelli <eo...@gmail.com>]

Il mer 20 giu 2018, 16:52 Sijie Guo <gu...@gmail.com> ha scritto:

> On Wed, Jun 20, 2018 at 4:09 AM Enrico Olivelli <eo...@gmail.com>
> wrote:
>
>> Hi,
>> I am experimenting DBLedgerStorage with some project.
>> With my workloads I don't see much benefit, or sometimes a performance
>> degradation,
>>
>
> Do you have a performance comparison? Or can you describe what have you
> observed?
>

No real 'sharable' numbers, just run a bunch of benchmarks of one
application. My data has no real meaning.




>
>> just by keeping the same bookie and application configuration (just
>> switching the storage manager classe name in the bookie configuration).
>>
>> Which is the expected workload for DBLedgerStorage ?
>> (Usually I am using the default SortedLedgerStorage)
>>
>
> Ideally it should be working out better when you have large number of
> ledgers per bookie (let's say more than 10k ledgers).
>

My test created only a few ledger ( less then 10) so maybe this is why I
see no benefit.
As far as I understand RocksDB is used for indexes and maybe with a few
ledgers the overhead is greater than the gain in term of resource saving.
I have to tune my test with more concurrency.

In this project I will have concurrent readers and writers, but readers
will access data at any point of ledgers, so no usual 'tailing reads'. I
will be using ledgers more likes files than like streams.
So having rocks db to store offsets seems a very good chance for me.
In real systems I will have thousands of active ledgers per bookie.
I have to use a more complex suite of benchmarks for the use case.

I will be back with more interesting comparisons

Thank you
Enrico




>
>>
>> Cheers
>> Enrico
>>
>>
>>
>> --


-- Enrico Olivelli

Re: Expected workload for DBLedgerStorage ?

[Sijie Guo <gu...@gmail.com>]

On Wed, Jun 20, 2018 at 4:09 AM Enrico Olivelli <eo...@gmail.com> wrote:

> Hi,
> I am experimenting DBLedgerStorage with some project.
> With my workloads I don't see much benefit, or sometimes a performance
> degradation,
>

Do you have a performance comparison? Or can you describe what have you
observed?


> just by keeping the same bookie and application configuration (just
> switching the storage manager classe name in the bookie configuration).
>
> Which is the expected workload for DBLedgerStorage ?
> (Usually I am using the default SortedLedgerStorage)
>

Ideally it should be working out better when you have large number of
ledgers per bookie (let's say more than 10k ledgers).


>
> Cheers
> Enrico
>
>
>
>