You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by kal123 <kp...@gmail.com> on 2013/10/09 23:31:06 UTC
Replicated LevelDB Store filling up memory
When I test with Replicated LevelDB Store and try to fill up the store
(queue) by just running producer without any consumers, the memory usage
keeps going up and VM runs out of memory.
Has anyone see this issue.. any conf. or workarounds for this..
--
View this message in context: http://activemq.2283324.n4.nabble.com/Replicated-LevelDB-Store-filling-up-memory-tp4672485.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Replicated LevelDB Store filling up memory
Posted by Hiram Chirino <hi...@hiramchirino.com>.
Hi Kal,
If your issue was being caused by a slow slave, I've introduced a
change to handle aggregating together replication events on the master
side so that the master's memory does not get impacted by slow slaves.
The change was in this commit:
https://github.com/apache/activemq/commit/4367ec1b829f46da60bd592a671ea1ebc8aedcd7
And it's available in this build, incase you want to test:
https://repository.apache.org/content/repositories/snapshots/org/apache/activemq/apache-activemq/5.10-SNAPSHOT/apache-activemq-5.10-20131106.134045-17-bin.tar.gz
On Thu, Oct 17, 2013 at 10:16 AM, kal123 <kp...@gmail.com> wrote:
> I have 2 slaves and 1 master running and using persistent messges. The
> master's memory usage is going high.
> The following are the objects i see building up:
> byte: 2.77G
> LinkedList:235MB (9.8 mil instances)
> org.apache.activemq.leveldb.replicated.FileTransferFrame:78.7M (2.45 mil.
> instances)
> ..ReplicationFrame:59.7M (2.49 mil instances)
>
> Could the number of FileTransferFrame indicate that its waiting to sync to
> slaves, issue with slaves?
>
> Why would slave be not getting synced fast enough.. or very slow to sync?
> we see same issue when quorum_disk is used as well?
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/Replicated-LevelDB-Store-filling-up-memory-tp4672485p4672941.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
--
Hiram Chirino
Engineering | Red Hat, Inc.
hchirino@redhat.com | fusesource.com | redhat.com
skype: hiramchirino | twitter: @hiramchirino
blog: Hiram Chirino's Bit Mojo
Re: Replicated LevelDB Store filling up memory
Posted by Hiram Chirino <hi...@hiramchirino.com>.
One of the slaves falling way behind is probably the reason your
seeing that. There is a JMX MBean (with object name attributes
'Service=PersistenceAdapter,view=Replication' that provides a 'slaves'
attribute which will tell you each slaves journal replication
'position' which you can then compare to the position attribute of the
master.
On Thu, Oct 17, 2013 at 10:16 AM, kal123 <kp...@gmail.com> wrote:
> I have 2 slaves and 1 master running and using persistent messges. The
> master's memory usage is going high.
> The following are the objects i see building up:
> byte: 2.77G
> LinkedList:235MB (9.8 mil instances)
> org.apache.activemq.leveldb.replicated.FileTransferFrame:78.7M (2.45 mil.
> instances)
> ..ReplicationFrame:59.7M (2.49 mil instances)
>
> Could the number of FileTransferFrame indicate that its waiting to sync to
> slaves, issue with slaves?
>
> Why would slave be not getting synced fast enough.. or very slow to sync?
> we see same issue when quorum_disk is used as well?
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/Replicated-LevelDB-Store-filling-up-memory-tp4672485p4672941.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
--
Hiram Chirino
Engineering | Red Hat, Inc.
hchirino@redhat.com | fusesource.com | redhat.com
skype: hiramchirino | twitter: @hiramchirino
blog: Hiram Chirino's Bit Mojo
Re: Replicated LevelDB Store filling up memory
Posted by kal123 <kp...@gmail.com>.
I have 2 slaves and 1 master running and using persistent messges. The
master's memory usage is going high.
The following are the objects i see building up:
byte: 2.77G
LinkedList:235MB (9.8 mil instances)
org.apache.activemq.leveldb.replicated.FileTransferFrame:78.7M (2.45 mil.
instances)
..ReplicationFrame:59.7M (2.49 mil instances)
Could the number of FileTransferFrame indicate that its waiting to sync to
slaves, issue with slaves?
Why would slave be not getting synced fast enough.. or very slow to sync?
we see same issue when quorum_disk is used as well?
--
View this message in context: http://activemq.2283324.n4.nabble.com/Replicated-LevelDB-Store-filling-up-memory-tp4672485p4672941.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Replicated LevelDB Store filling up memory
Posted by Christian Posta <ch...@gmail.com>.
give more details about your broker configuration (number of nodes in
cluster?)
are you sending non-persistent messages? is the master going OOM up or
slave?
On Wed, Oct 9, 2013 at 2:31 PM, kal123 <kp...@gmail.com> wrote:
> When I test with Replicated LevelDB Store and try to fill up the store
> (queue) by just running producer without any consumers, the memory usage
> keeps going up and VM runs out of memory.
> Has anyone see this issue.. any conf. or workarounds for this..
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Replicated-LevelDB-Store-filling-up-memory-tp4672485.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
--
*Christian Posta*
http://www.christianposta.com/blog
twitter: @christianposta
Re: Replicated LevelDB Store filling up memory
Posted by Graham Leggett <mi...@sharp.fm>.
On 09 Oct 2013, at 11:31 PM, kal123 <kp...@gmail.com> wrote:
> When I test with Replicated LevelDB Store and try to fill up the store
> (queue) by just running producer without any consumers, the memory usage
> keeps going up and VM runs out of memory.
> Has anyone see this issue.. any conf. or workarounds for this..
I've also been suffering the VM running out of memory and hanging, we haven't found a source of the leak yet: https://issues.apache.org/jira/browse/AMQ-4789. We tried kahadb and leveldb, neither with any success.
Regards,
Graham
--
Re: Replicated LevelDB Store filling up memory
Posted by kal123 <kp...@gmail.com>.
could this fix the memory issue?
http://activemq.2283324.n4.nabble.com/git-commit-Fixes-bug-in-replicated-leveldb-where-log-files-on-slaves-were-not-getting-GCed-td4671551.html
I will try this out...
--
View this message in context: http://activemq.2283324.n4.nabble.com/Replicated-LevelDB-Store-filling-up-memory-tp4672485p4672494.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.