You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by Kevin Burton <bu...@spinn3r.com> on 2014/09/14 22:46:10 UTC
Do I have to clean up leveldb log files? It’s using 50GB but should be using 50MB..
I have an ActiveMQ + leveldb instance that’s been running for 1-2 months
now.
It doesn’t have many messages … they’re scheduled messages that get
executed about ever 15 minutes. There are about 100k of them… so the
throughput is not very significant.
the *.index directory takes up 50MB … this makes sense of course and is
about the right amount of data.
The problem is that the .log files just keep accumulating and aren’t
getting GCd.
I have about 60GB of .log files now.
I can’t find any documentation on how to remove them. When can they be
cleaned up? Perhaps my leveldb isn’t configured correctly? Or this is a
bug?
Kevin
--
Founder/CEO Spinn3r.com
Location: *San Francisco, CA*
blog: http://burtonator.wordpress.com
… or check out my Google+ profile
<https://plus.google.com/102718274791889610666/posts>
<http://spinn3r.com>
Re: Do I have to clean up leveldb log files? It’s using 50GB but should be using 50MB..
Posted by Kevin Burton <bu...@spinn3r.com>.
Amazed there aren’t any comments here.. .You’d think LevelDB filling up the
whole volume would be a major issue others would be dealing with.
On Sun, Sep 14, 2014 at 2:18 PM, Kevin Burton <bu...@spinn3r.com> wrote:
> Looks like this is a bug. I understand the LevelDB and Bigtable design
> very well and assumed that the binary log would be closed out when
> converted to an SSTable.
>
> Looks like LevelDB writes data to the log and it’s backed in a memtable.
> Once the log hits 100MB it’s converted to a sstable.
>
> So, the .log files can be safely deleted at that point.
>
> On Sun, Sep 14, 2014 at 1:51 PM, Kevin Burton <bu...@spinn3r.com> wrote:
>
>> Could this have to do with:
>>
>> <tempUsage>
>> <tempUsage limit=“50 gb"/>
>> </tempUsage>
>>
>> … the *.log files right now are taking up 36GB… so I should probably set
>> this to like 500MB or 1GB or something.
>>
>>
>> On Sun, Sep 14, 2014 at 1:46 PM, Kevin Burton <bu...@spinn3r.com> wrote:
>>
>>> I have an ActiveMQ + leveldb instance that’s been running for 1-2 months
>>> now.
>>>
>>> It doesn’t have many messages … they’re scheduled messages that get
>>> executed about ever 15 minutes. There are about 100k of them… so the
>>> throughput is not very significant.
>>>
>>> the *.index directory takes up 50MB … this makes sense of course and is
>>> about the right amount of data.
>>>
>>> The problem is that the .log files just keep accumulating and aren’t
>>> getting GCd.
>>>
>>> I have about 60GB of .log files now.
>>>
>>> I can’t find any documentation on how to remove them. When can they be
>>> cleaned up? Perhaps my leveldb isn’t configured correctly? Or this is a
>>> bug?
>>>
>>> Kevin
>>>
>>> --
>>>
>>> Founder/CEO Spinn3r.com
>>> Location: *San Francisco, CA*
>>> blog: http://burtonator.wordpress.com
>>> … or check out my Google+ profile
>>> <https://plus.google.com/102718274791889610666/posts>
>>> <http://spinn3r.com>
>>>
>>>
>>
>>
>> --
>>
>> Founder/CEO Spinn3r.com
>> Location: *San Francisco, CA*
>> blog: http://burtonator.wordpress.com
>> … or check out my Google+ profile
>> <https://plus.google.com/102718274791889610666/posts>
>> <http://spinn3r.com>
>>
>>
>
>
> --
>
> Founder/CEO Spinn3r.com
> Location: *San Francisco, CA*
> blog: http://burtonator.wordpress.com
> … or check out my Google+ profile
> <https://plus.google.com/102718274791889610666/posts>
> <http://spinn3r.com>
>
>
--
Founder/CEO Spinn3r.com
Location: *San Francisco, CA*
blog: http://burtonator.wordpress.com
… or check out my Google+ profile
<https://plus.google.com/102718274791889610666/posts>
<http://spinn3r.com>
Re: Do I have to clean up leveldb log files? It’s using 50GB but should be using 50MB..
Posted by Kevin Burton <bu...@spinn3r.com>.
Looks like this is a bug. I understand the LevelDB and Bigtable design very
well and assumed that the binary log would be closed out when converted to
an SSTable.
Looks like LevelDB writes data to the log and it’s backed in a memtable.
Once the log hits 100MB it’s converted to a sstable.
So, the .log files can be safely deleted at that point.
On Sun, Sep 14, 2014 at 1:51 PM, Kevin Burton <bu...@spinn3r.com> wrote:
> Could this have to do with:
>
> <tempUsage>
> <tempUsage limit=“50 gb"/>
> </tempUsage>
>
> … the *.log files right now are taking up 36GB… so I should probably set
> this to like 500MB or 1GB or something.
>
>
> On Sun, Sep 14, 2014 at 1:46 PM, Kevin Burton <bu...@spinn3r.com> wrote:
>
>> I have an ActiveMQ + leveldb instance that’s been running for 1-2 months
>> now.
>>
>> It doesn’t have many messages … they’re scheduled messages that get
>> executed about ever 15 minutes. There are about 100k of them… so the
>> throughput is not very significant.
>>
>> the *.index directory takes up 50MB … this makes sense of course and is
>> about the right amount of data.
>>
>> The problem is that the .log files just keep accumulating and aren’t
>> getting GCd.
>>
>> I have about 60GB of .log files now.
>>
>> I can’t find any documentation on how to remove them. When can they be
>> cleaned up? Perhaps my leveldb isn’t configured correctly? Or this is a
>> bug?
>>
>> Kevin
>>
>> --
>>
>> Founder/CEO Spinn3r.com
>> Location: *San Francisco, CA*
>> blog: http://burtonator.wordpress.com
>> … or check out my Google+ profile
>> <https://plus.google.com/102718274791889610666/posts>
>> <http://spinn3r.com>
>>
>>
>
>
> --
>
> Founder/CEO Spinn3r.com
> Location: *San Francisco, CA*
> blog: http://burtonator.wordpress.com
> … or check out my Google+ profile
> <https://plus.google.com/102718274791889610666/posts>
> <http://spinn3r.com>
>
>
--
Founder/CEO Spinn3r.com
Location: *San Francisco, CA*
blog: http://burtonator.wordpress.com
… or check out my Google+ profile
<https://plus.google.com/102718274791889610666/posts>
<http://spinn3r.com>
Re: Do I have to clean up leveldb log files? It’s using 50GB but should be using 50MB..
Posted by Kevin Burton <bu...@spinn3r.com>.
Could this have to do with:
<tempUsage>
<tempUsage limit=“50 gb"/>
</tempUsage>
… the *.log files right now are taking up 36GB… so I should probably set
this to like 500MB or 1GB or something.
On Sun, Sep 14, 2014 at 1:46 PM, Kevin Burton <bu...@spinn3r.com> wrote:
> I have an ActiveMQ + leveldb instance that’s been running for 1-2 months
> now.
>
> It doesn’t have many messages … they’re scheduled messages that get
> executed about ever 15 minutes. There are about 100k of them… so the
> throughput is not very significant.
>
> the *.index directory takes up 50MB … this makes sense of course and is
> about the right amount of data.
>
> The problem is that the .log files just keep accumulating and aren’t
> getting GCd.
>
> I have about 60GB of .log files now.
>
> I can’t find any documentation on how to remove them. When can they be
> cleaned up? Perhaps my leveldb isn’t configured correctly? Or this is a
> bug?
>
> Kevin
>
> --
>
> Founder/CEO Spinn3r.com
> Location: *San Francisco, CA*
> blog: http://burtonator.wordpress.com
> … or check out my Google+ profile
> <https://plus.google.com/102718274791889610666/posts>
> <http://spinn3r.com>
>
>
--
Founder/CEO Spinn3r.com
Location: *San Francisco, CA*
blog: http://burtonator.wordpress.com
… or check out my Google+ profile
<https://plus.google.com/102718274791889610666/posts>
<http://spinn3r.com>