KahaDB vs LevelDB

["James A. Robinson" <ji...@gmail.com>]

Is KahaDB considered the more robust backing store of the two options?

We just ran into a variation of

https://issues.apache.org/jira/browse/AMQ-5459

and I couldn't see any way to recover it.

Jim

Re: KahaDB vs LevelDB

[Tim Bain <tb...@alumni.duke.edu>]

I've never used any of the three (my use cases have always been for
non-persistent messages because we wanted speed and were willing to accept
losing messages if the broker crashed), but based on the past year and a
half of monitoring the mailing list, I agree with Chris's assessments.

JDBC is little-used, little-supported, and little-tuned from a performance
standpoint, and is generally held to be the slowest of the three.  I'd
consider switching a development broker to it briefly while troubleshooting
a problem because there are lots of good tools for querying an RDBMS if you
needed to know details about the messages it held at a particular point in
time, but I'd never use it in a production environment.  Use at your own
risk.

KahaDB is faster than JDBC, widely used, and generally stable.  I've seen
very few true bugs related to KahaDB on the mailing list in the past year
(notable exception: its ungraceful handling of the loss of a network
connection to shared storage and its inability to detect that it may no
longer be the master broker in a master/slave pair), though that's not to
say there weren't bugs that went straight to JIRA.  Anyone who's interested
can query JIRA to see.  Our use of KahaDB does, however, have a major
design flaw: KahaDB requires all messages to be consumed quickly, while
ActiveMQ recommends but doesn't require it.  The result is that when an old
message remains unconsumed, old journal files can't be deleted, and can't
be compacted, so a single unconsumed message can force the broker to keep
all late journals even if they're not needed.  Note that this isn't a
design flaw of KahaDB itself, since it's perfectly reasonable if somewhat
inflexible to support only use cases where all messages must be consumed
quickly, but rather of our decision to use it given the paradigm mismatch
between its capabilities and our needs.  (Of course, we wrote it to support
our needs, so you could make the argument that the flaw really is in
KahaDB.)

I've never seen anyone post a question or a bug about unreplicated LevelDB,
or for that matter about the LevelDB portion (i.e. the non-Zookeeper
portion) of replicated LevelDB.  (Or maybe I'm just not remembering them,
since Chris referenced corruption issues that I don't recall.)  I'm not
sure if that means unreplicated LevelDB is stable, or that it's not being
used much.  We claim on the wiki that LevelDB will be faster than KahaDB,
but we offer no proof and I've seen a StackOverflow post where someone
benchmarked both and found no meaningful speed difference.  But I've never
seen anything that indicates LevelDB is slower.  I'm not familiar enough
with it to know whether it deals any better with old messages that never
get deleted or whether our use of it has the same design flaw as our use of
KahaDB.

Replicated LevelDB, on the other hand, has had more bugs and questions than
the others over the past year and a half, and most of what I've seen seems
to involve Zookeeper in some way.  The volume of questions and frustrations
has seemed lower in the past six to nine months than before that, so
hopefully that means it's becoming more stable.  Replicated LevelDB offers
some really promising things, and whatever bugs remain will get worked out
sooner with more people using it, so I'd encourage anyone who thinks they
might want to use it to at least run it somewhere (like a dev environment)
to see if you're comfortable using it in production and to give us reports
if you see bugs.  One major disadvantage of both flavors of LevelDB is that
there's no expert on the list who's been willing to answer questions, so
most questions go unanswered.  If you already use LevelDB, even if you're
not an expert, here's your invitation to get involved and pay something
back for a product you got for free.

All three persistence store technologies have one other flaw, which is
really a flaw in the broker: at dispatch time, we ignore any message that
isn't in a cursor (in the memory store), and the memory store is
constrained by RAM so it's quiet possible that not all messages in the
persistence store will fit in the cursor, so it sometimes happens that we
only consider some of the messages for dispatch.  This potentially breaks
quite a few features: selectors, message priorities, and message groups,
off the top of my head.  We just tell everyone "that's the way it is," but
the reality is that this (the assumption that it's valid to consider only
as many messages as will fit in the cursor and that they should be the
oldest messages in the store) is a bug and we need to fix it.  I've been
meaning to write a bug in JIRA about it, so maybe I'll get to it this
weekend.

Tim
On Jan 29, 2016 6:03 AM, "Christopher Shannon" <
christopher.l.shannon@gmail.com> wrote:

> Keep in mind that KahaDB will be significantly faster than any JDBC
> persistence solution, so if you switch to JDBC than you will have a large
> performance hit.
>
> On Fri, Jan 29, 2016 at 3:34 AM, Klaus Pittig <
> klaus.pittig@futura4retail.com> wrote:
>
> > I agree. We also ran into several issues using LevelDB and decided to
> > switch back to KahaDB as the default persistence, even if it's slower.
> > Our efforts repairing LevelDB storages on many different machines were
> > only with moderate success.
> >
> > In contrast handling problems with KahaDB is a straightforward process
> > and works in most cases. To avoid even this inconvenient administrative
> > task we think about switching to a journaledJdbc persistence with a
> > local PostgreSQL instance or similar, because it simplifies support and
> > administration (due to the available people knowing their company DBMS).
> >
> >
> > Am 28.01.2016 um 19:24 schrieb Christopher Shannon:
> > > In my opinion KahaDB is more stable at this point than LevelDB.  KahaDB
> > > does not seem to suffer from some of the corruption problems and other
> > > issues that have been reported when using LevelDB.
> > >
> > > On Thu, Jan 28, 2016 at 12:52 PM, James A. Robinson <
> > jim.robinson@gmail.com>
> > > wrote:
> > >
> > >> Is KahaDB considered the more robust backing store of the two options?
> > >>
> > >> We just ran into a variation of
> > >>
> > >> https://issues.apache.org/jira/browse/AMQ-5459
> > >>
> > >> and I couldn't see any way to recover it.
> > >>
> > >> Jim
> > >>
> > >
> >
>

Re: KahaDB vs LevelDB

[Klaus Pittig <kl...@futura4retail.com>]

You are right, and I do prefer KahaDB due to its performance (as long as
LevelDB seems to be just "beta-stable").
Such a decision needs to be made after careful consideration of the
trade-offs.
On the strength of our past experience some customers have much less
traffic resp. no the highest performance requirements, so a
journaledJdbc Adapter could help for some time.

A permanently "self-healing" KahaDB would be a huge advantage to
simplify long-term operation in my point of view.


Am 29.01.2016 um 14:02 schrieb Christopher Shannon:
> Keep in mind that KahaDB will be significantly faster than any JDBC
> persistence solution, so if you switch to JDBC than you will have a large
> performance hit.
> 
> On Fri, Jan 29, 2016 at 3:34 AM, Klaus Pittig <
> klaus.pittig@futura4retail.com> wrote:
> 
>> I agree. We also ran into several issues using LevelDB and decided to
>> switch back to KahaDB as the default persistence, even if it's slower.
>> Our efforts repairing LevelDB storages on many different machines were
>> only with moderate success.
>>
>> In contrast handling problems with KahaDB is a straightforward process
>> and works in most cases. To avoid even this inconvenient administrative
>> task we think about switching to a journaledJdbc persistence with a
>> local PostgreSQL instance or similar, because it simplifies support and
>> administration (due to the available people knowing their company DBMS).
>>
>>
>> Am 28.01.2016 um 19:24 schrieb Christopher Shannon:
>>> In my opinion KahaDB is more stable at this point than LevelDB.  KahaDB
>>> does not seem to suffer from some of the corruption problems and other
>>> issues that have been reported when using LevelDB.
>>>
>>> On Thu, Jan 28, 2016 at 12:52 PM, James A. Robinson <
>> jim.robinson@gmail.com>
>>> wrote:
>>>
>>>> Is KahaDB considered the more robust backing store of the two options?
>>>>
>>>> We just ran into a variation of
>>>>
>>>> https://issues.apache.org/jira/browse/AMQ-5459
>>>>
>>>> and I couldn't see any way to recover it.
>>>>
>>>> Jim
>>>>
>>>
>>
> 

Re: KahaDB vs LevelDB

[Christopher Shannon <ch...@gmail.com>]

Keep in mind that KahaDB will be significantly faster than any JDBC
persistence solution, so if you switch to JDBC than you will have a large
performance hit.

On Fri, Jan 29, 2016 at 3:34 AM, Klaus Pittig <
klaus.pittig@futura4retail.com> wrote:

> I agree. We also ran into several issues using LevelDB and decided to
> switch back to KahaDB as the default persistence, even if it's slower.
> Our efforts repairing LevelDB storages on many different machines were
> only with moderate success.
>
> In contrast handling problems with KahaDB is a straightforward process
> and works in most cases. To avoid even this inconvenient administrative
> task we think about switching to a journaledJdbc persistence with a
> local PostgreSQL instance or similar, because it simplifies support and
> administration (due to the available people knowing their company DBMS).
>
>
> Am 28.01.2016 um 19:24 schrieb Christopher Shannon:
> > In my opinion KahaDB is more stable at this point than LevelDB.  KahaDB
> > does not seem to suffer from some of the corruption problems and other
> > issues that have been reported when using LevelDB.
> >
> > On Thu, Jan 28, 2016 at 12:52 PM, James A. Robinson <
> jim.robinson@gmail.com>
> > wrote:
> >
> >> Is KahaDB considered the more robust backing store of the two options?
> >>
> >> We just ran into a variation of
> >>
> >> https://issues.apache.org/jira/browse/AMQ-5459
> >>
> >> and I couldn't see any way to recover it.
> >>
> >> Jim
> >>
> >
>

Re: KahaDB vs LevelDB

[Klaus Pittig <kl...@futura4retail.com>]

I agree. We also ran into several issues using LevelDB and decided to
switch back to KahaDB as the default persistence, even if it's slower.
Our efforts repairing LevelDB storages on many different machines were
only with moderate success.

In contrast handling problems with KahaDB is a straightforward process
and works in most cases. To avoid even this inconvenient administrative
task we think about switching to a journaledJdbc persistence with a
local PostgreSQL instance or similar, because it simplifies support and
administration (due to the available people knowing their company DBMS).


Am 28.01.2016 um 19:24 schrieb Christopher Shannon:
> In my opinion KahaDB is more stable at this point than LevelDB.  KahaDB
> does not seem to suffer from some of the corruption problems and other
> issues that have been reported when using LevelDB.
> 
> On Thu, Jan 28, 2016 at 12:52 PM, James A. Robinson <ji...@gmail.com>
> wrote:
> 
>> Is KahaDB considered the more robust backing store of the two options?
>>
>> We just ran into a variation of
>>
>> https://issues.apache.org/jira/browse/AMQ-5459
>>
>> and I couldn't see any way to recover it.
>>
>> Jim
>>
> 

Re: KahaDB vs LevelDB

[Christopher Shannon <ch...@gmail.com>]

In my opinion KahaDB is more stable at this point than LevelDB.  KahaDB
does not seem to suffer from some of the corruption problems and other
issues that have been reported when using LevelDB.

On Thu, Jan 28, 2016 at 12:52 PM, James A. Robinson <ji...@gmail.com>
wrote:

> Is KahaDB considered the more robust backing store of the two options?
>
> We just ran into a variation of
>
> https://issues.apache.org/jira/browse/AMQ-5459
>
> and I couldn't see any way to recover it.
>
> Jim
>