You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by Oleg Dulin <ol...@gmail.com> on 2012/09/19 17:29:39 UTC
KahaDB disk utilization
Dear Distinguished Colleagues:
I've been trying to understand something.
My queues take Java object messages. Serialized into bytes, each Java
object is about 1.5K on average.
When I have a backlog of 10 million objects on one of the queues, the
disk utilization of kahadb is around 50gigs. But this math here:
groovy:000> 1500*10000000/1024/1024/1024;
===> 1.9698386192
means that 10M objects should only take up less than 2gigs.
Even if I had 3 queues going with the same objects, the total would be
6gigs max. So why the disk utilization overhead ? How much does a
single JMS message need ?
--
Regards,
Oleg Dulin
NYC Java Big Data Engineer
http://www.olegdulin.com/
Re: KahaDB disk utilization
Posted by Gary Tully <ga...@gmail.com>.
maybe work with a 5.7-SNAPSHOT for the moment, work on 5.7 is nearly
complete, so you can expect a release candidate early next week.
On 26 September 2012 18:23, DV <vi...@gmail.com> wrote:
> Though I really like the idea of mKahaDB, it's unusable to us right
> now because of the bug described in AMQ-3841 - we frequently encounter
> it with AMQ 5.6 and the fix isn't due until 5.7 is out :/
>
> - DV
>
> On Wed, Sep 26, 2012 at 2:54 AM, Claus Ibsen <cl...@gmail.com> wrote:
>> Hi
>>
>>
>>
>> On Wed, Sep 26, 2012 at 5:14 AM, DV <vi...@gmail.com> wrote:
>>> Are you absolutely certain that there are exactly 10M messages on the
>>> broker? I've seen a huge KahaDB store with only a few messages in the
>>> queue because the earlier published messages weren't de-queued
>>> ("stuck" messages).
>>>
>>
>> Yes and you can use multi KahaDB to distribute destinations in different files
>> which can help reduce disk space. You can read a bit about it here:
>> http://activemq.apache.org/kahadb.html
>>
>>
>>> On the other hand, maybe KahaDB sacrifices space for performance, kind
>>> of like MongoDB does.
>>>
>>> - DV
>>>
>>> On Wed, Sep 19, 2012 at 8:29 AM, Oleg Dulin <ol...@gmail.com> wrote:
>>>> Dear Distinguished Colleagues:
>>>>
>>>> I've been trying to understand something.
>>>>
>>>> My queues take Java object messages. Serialized into bytes, each Java object
>>>> is about 1.5K on average.
>>>>
>>>> When I have a backlog of 10 million objects on one of the queues, the disk
>>>> utilization of kahadb is around 50gigs. But this math here:
>>>>
>>>> groovy:000> 1500*10000000/1024/1024/1024;
>>>> ===> 1.9698386192
>>>>
>>>> means that 10M objects should only take up less than 2gigs.
>>>>
>>>> Even if I had 3 queues going with the same objects, the total would be 6gigs
>>>> max. So why the disk utilization overhead ? How much does a single JMS
>>>> message need ?
>>>>
>>>>
>>>> --
>>>> Regards,
>>>> Oleg Dulin
>>>> NYC Java Big Data Engineer
>>>> http://www.olegdulin.com/
>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> Best regards, Dmitriy V.
>>
>>
>>
>> --
>> Claus Ibsen
>> -----------------
>> Red Hat, Inc.
>> FuseSource is now part of Red Hat
>> Email: cibsen@redhat.com
>> Web: http://fusesource.com
>> Twitter: davsclaus
>> Blog: http://davsclaus.com
>> Author of Camel in Action: http://www.manning.com/ibsen
>
>
>
> --
> Best regards, Dmitriy V.
--
http://fusesource.com
http://blog.garytully.com
Re: KahaDB disk utilization
Posted by DV <vi...@gmail.com>.
Though I really like the idea of mKahaDB, it's unusable to us right
now because of the bug described in AMQ-3841 - we frequently encounter
it with AMQ 5.6 and the fix isn't due until 5.7 is out :/
- DV
On Wed, Sep 26, 2012 at 2:54 AM, Claus Ibsen <cl...@gmail.com> wrote:
> Hi
>
>
>
> On Wed, Sep 26, 2012 at 5:14 AM, DV <vi...@gmail.com> wrote:
>> Are you absolutely certain that there are exactly 10M messages on the
>> broker? I've seen a huge KahaDB store with only a few messages in the
>> queue because the earlier published messages weren't de-queued
>> ("stuck" messages).
>>
>
> Yes and you can use multi KahaDB to distribute destinations in different files
> which can help reduce disk space. You can read a bit about it here:
> http://activemq.apache.org/kahadb.html
>
>
>> On the other hand, maybe KahaDB sacrifices space for performance, kind
>> of like MongoDB does.
>>
>> - DV
>>
>> On Wed, Sep 19, 2012 at 8:29 AM, Oleg Dulin <ol...@gmail.com> wrote:
>>> Dear Distinguished Colleagues:
>>>
>>> I've been trying to understand something.
>>>
>>> My queues take Java object messages. Serialized into bytes, each Java object
>>> is about 1.5K on average.
>>>
>>> When I have a backlog of 10 million objects on one of the queues, the disk
>>> utilization of kahadb is around 50gigs. But this math here:
>>>
>>> groovy:000> 1500*10000000/1024/1024/1024;
>>> ===> 1.9698386192
>>>
>>> means that 10M objects should only take up less than 2gigs.
>>>
>>> Even if I had 3 queues going with the same objects, the total would be 6gigs
>>> max. So why the disk utilization overhead ? How much does a single JMS
>>> message need ?
>>>
>>>
>>> --
>>> Regards,
>>> Oleg Dulin
>>> NYC Java Big Data Engineer
>>> http://www.olegdulin.com/
>>>
>>>
>>
>>
>>
>> --
>> Best regards, Dmitriy V.
>
>
>
> --
> Claus Ibsen
> -----------------
> Red Hat, Inc.
> FuseSource is now part of Red Hat
> Email: cibsen@redhat.com
> Web: http://fusesource.com
> Twitter: davsclaus
> Blog: http://davsclaus.com
> Author of Camel in Action: http://www.manning.com/ibsen
--
Best regards, Dmitriy V.
Re: KahaDB disk utilization
Posted by Gary Tully <ga...@gmail.com>.
I provided some more detail on journal disk usage in a blog entry a while back:
http://blog.garytully.com/2011/11/activemq-multiple-kahadb-instances.html
On 26 September 2012 10:54, Claus Ibsen <cl...@gmail.com> wrote:
> Hi
>
>
>
> On Wed, Sep 26, 2012 at 5:14 AM, DV <vi...@gmail.com> wrote:
>> Are you absolutely certain that there are exactly 10M messages on the
>> broker? I've seen a huge KahaDB store with only a few messages in the
>> queue because the earlier published messages weren't de-queued
>> ("stuck" messages).
>>
>
> Yes and you can use multi KahaDB to distribute destinations in different files
> which can help reduce disk space. You can read a bit about it here:
> http://activemq.apache.org/kahadb.html
>
>
>> On the other hand, maybe KahaDB sacrifices space for performance, kind
>> of like MongoDB does.
>>
>> - DV
>>
>> On Wed, Sep 19, 2012 at 8:29 AM, Oleg Dulin <ol...@gmail.com> wrote:
>>> Dear Distinguished Colleagues:
>>>
>>> I've been trying to understand something.
>>>
>>> My queues take Java object messages. Serialized into bytes, each Java object
>>> is about 1.5K on average.
>>>
>>> When I have a backlog of 10 million objects on one of the queues, the disk
>>> utilization of kahadb is around 50gigs. But this math here:
>>>
>>> groovy:000> 1500*10000000/1024/1024/1024;
>>> ===> 1.9698386192
>>>
>>> means that 10M objects should only take up less than 2gigs.
>>>
>>> Even if I had 3 queues going with the same objects, the total would be 6gigs
>>> max. So why the disk utilization overhead ? How much does a single JMS
>>> message need ?
>>>
>>>
>>> --
>>> Regards,
>>> Oleg Dulin
>>> NYC Java Big Data Engineer
>>> http://www.olegdulin.com/
>>>
>>>
>>
>>
>>
>> --
>> Best regards, Dmitriy V.
>
>
>
> --
> Claus Ibsen
> -----------------
> Red Hat, Inc.
> FuseSource is now part of Red Hat
> Email: cibsen@redhat.com
> Web: http://fusesource.com
> Twitter: davsclaus
> Blog: http://davsclaus.com
> Author of Camel in Action: http://www.manning.com/ibsen
--
http://fusesource.com
http://blog.garytully.com
Re: KahaDB disk utilization
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
On Wed, Sep 26, 2012 at 5:14 AM, DV <vi...@gmail.com> wrote:
> Are you absolutely certain that there are exactly 10M messages on the
> broker? I've seen a huge KahaDB store with only a few messages in the
> queue because the earlier published messages weren't de-queued
> ("stuck" messages).
>
Yes and you can use multi KahaDB to distribute destinations in different files
which can help reduce disk space. You can read a bit about it here:
http://activemq.apache.org/kahadb.html
> On the other hand, maybe KahaDB sacrifices space for performance, kind
> of like MongoDB does.
>
> - DV
>
> On Wed, Sep 19, 2012 at 8:29 AM, Oleg Dulin <ol...@gmail.com> wrote:
>> Dear Distinguished Colleagues:
>>
>> I've been trying to understand something.
>>
>> My queues take Java object messages. Serialized into bytes, each Java object
>> is about 1.5K on average.
>>
>> When I have a backlog of 10 million objects on one of the queues, the disk
>> utilization of kahadb is around 50gigs. But this math here:
>>
>> groovy:000> 1500*10000000/1024/1024/1024;
>> ===> 1.9698386192
>>
>> means that 10M objects should only take up less than 2gigs.
>>
>> Even if I had 3 queues going with the same objects, the total would be 6gigs
>> max. So why the disk utilization overhead ? How much does a single JMS
>> message need ?
>>
>>
>> --
>> Regards,
>> Oleg Dulin
>> NYC Java Big Data Engineer
>> http://www.olegdulin.com/
>>
>>
>
>
>
> --
> Best regards, Dmitriy V.
--
Claus Ibsen
-----------------
Red Hat, Inc.
FuseSource is now part of Red Hat
Email: cibsen@redhat.com
Web: http://fusesource.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
Re: KahaDB disk utilization
Posted by DV <vi...@gmail.com>.
Are you absolutely certain that there are exactly 10M messages on the
broker? I've seen a huge KahaDB store with only a few messages in the
queue because the earlier published messages weren't de-queued
("stuck" messages).
On the other hand, maybe KahaDB sacrifices space for performance, kind
of like MongoDB does.
- DV
On Wed, Sep 19, 2012 at 8:29 AM, Oleg Dulin <ol...@gmail.com> wrote:
> Dear Distinguished Colleagues:
>
> I've been trying to understand something.
>
> My queues take Java object messages. Serialized into bytes, each Java object
> is about 1.5K on average.
>
> When I have a backlog of 10 million objects on one of the queues, the disk
> utilization of kahadb is around 50gigs. But this math here:
>
> groovy:000> 1500*10000000/1024/1024/1024;
> ===> 1.9698386192
>
> means that 10M objects should only take up less than 2gigs.
>
> Even if I had 3 queues going with the same objects, the total would be 6gigs
> max. So why the disk utilization overhead ? How much does a single JMS
> message need ?
>
>
> --
> Regards,
> Oleg Dulin
> NYC Java Big Data Engineer
> http://www.olegdulin.com/
>
>
--
Best regards, Dmitriy V.