Long pauses with leveldb, no pauses with kaha

[Attila Nagy <br...@fsn.hu>]

Hi,

I use ActiveMQ 5.9.0 on FreeBSD with java 1.7.0_25-b15 and native 
leveldb (default settings).
Four servers, fully meshed network of brokers, only two get effective load.

I publish from a small number of servers (around 30) to a virtual topic, 
then consume the aggregated messages from one STOMP client.
The average message size is around 250 bytes, the incoming message rate 
is around 8k+6k messages per second on the two servers.

What I see is the flow towards the consumer periodically pauses (maybe 
the incoming flow too, but I don't monitor that).
Measuring three of those pauses gave 34.01463485, 34.03087783 and 
53.02782011 seconds, during which no messages were delivered to the 
client at all. The first two are pretty suspicious.

Digging into it a little, "top" on the ActiveMQ server shows that one 
thread is active during the pauses, and it's mostly in pipe read state. WTF.
Doing a ktrace (strace-like FreeBSD tool) shows that it spawns multiple 
"ln" commands (yikes) which mess with the leveldb files (I guess it's 
the compaction job).

It seems others have performance issues with leveldb too (like 
http://symas.com/mdb/hyperdex/, maybe you should also switch to LMDB :) 
), so instead of playing with its tunable knobs (I've already failed 
with them, see my previous mail about kahadb being faster than leveldb), 
I've just switched to kahadb (also default settings) to check whether 
this causes the delays.

Yes, leveldb causes the long pauses in message deliveries. I'm running 
two hours since I've switched, and I haven't yet had a second with zero 
delivered messages, and the overall average rate of message deliveries 
per seconds is also smoothed out, no fluctuation.

Are you aware of this issue? Skimming through the list it seems there 
may be others hit by this, but I couldn't yet find an e-mail which would 
state that leveldb is bad for performance, you should a) avoid it, b) 
set these knobs to these, c) wait for the next revision, we'll fix this.

Thanks,

Re: Long pauses with leveldb, no pauses with kaha

[Hiram Chirino <hi...@hiramchirino.com>]

Yep, at least I've noted that this can happen under consistent load.
I think part of the problem is that leveldb is periodically doing
level compactions which can cause pauses in it's write operations.

HyperDex and RocksDB are both similar libs which we could swap out
leveldb for in the future which seem to promise a more consistent
write rate.


On Wed, Dec 4, 2013 at 5:24 AM, Attila Nagy <br...@fsn.hu> wrote:
> Hi,
>
> I use ActiveMQ 5.9.0 on FreeBSD with java 1.7.0_25-b15 and native leveldb
> (default settings).
> Four servers, fully meshed network of brokers, only two get effective load.
>
> I publish from a small number of servers (around 30) to a virtual topic,
> then consume the aggregated messages from one STOMP client.
> The average message size is around 250 bytes, the incoming message rate is
> around 8k+6k messages per second on the two servers.
>
> What I see is the flow towards the consumer periodically pauses (maybe the
> incoming flow too, but I don't monitor that).
> Measuring three of those pauses gave 34.01463485, 34.03087783 and
> 53.02782011 seconds, during which no messages were delivered to the client
> at all. The first two are pretty suspicious.
>
> Digging into it a little, "top" on the ActiveMQ server shows that one thread
> is active during the pauses, and it's mostly in pipe read state. WTF.
> Doing a ktrace (strace-like FreeBSD tool) shows that it spawns multiple "ln"
> commands (yikes) which mess with the leveldb files (I guess it's the
> compaction job).
>
> It seems others have performance issues with leveldb too (like
> http://symas.com/mdb/hyperdex/, maybe you should also switch to LMDB :) ),
> so instead of playing with its tunable knobs (I've already failed with them,
> see my previous mail about kahadb being faster than leveldb), I've just
> switched to kahadb (also default settings) to check whether this causes the
> delays.
>
> Yes, leveldb causes the long pauses in message deliveries. I'm running two
> hours since I've switched, and I haven't yet had a second with zero
> delivered messages, and the overall average rate of message deliveries per
> seconds is also smoothed out, no fluctuation.
>
> Are you aware of this issue? Skimming through the list it seems there may be
> others hit by this, but I couldn't yet find an e-mail which would state that
> leveldb is bad for performance, you should a) avoid it, b) set these knobs
> to these, c) wait for the next revision, we'll fix this.
>
> Thanks,



-- 
Hiram Chirino
Engineering | Red Hat, Inc.
hchirino@redhat.com | fusesource.com | redhat.com
skype: hiramchirino | twitter: @hiramchirino