You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by Shine <ac...@club.webhop.org> on 2015/12/16 20:54:48 UTC
Need help with a message broker configuration
Hi,
i need a broker configuration for this features
- protocols: OpenWire + SSL (NMS C#) on Windows; STOMP + SSL on iOS and
Android.
- every message has a time-to-live/expire date.
=> all expired messages need to be delete .. no rollback or other
operations on expired messages are needed .. no DeadLetter ...
- the Client decides if a message is persistent or not.
- each client has its own queue and subscribes it.
- (fast) dead client detection on the message broker; especially for the
mobile devices.
my current running message broker (ActiveMQ Version 5.11.1) has this
problems:
- the kahadb folder grows and grows; currently the folder contains about 400
files.
- a client has many(unnecessary) connections to this topic
"ActiveMQ.Advisory.TempQueue,ActiveMQ.Advisory.TempTopic"
thanks for help.
best regards
Shine
--
View this message in context: http://activemq.2283324.n4.nabble.com/Need-help-with-a-message-broker-configuration-tp4705074.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Need help with a message broker configuration
Posted by Shine <ac...@club.webhop.org>.
Hi,
is a Multi kahaDB persistence adapter a good way to get the growing of the
KahaDB folder under control?
best regards
Shine
--
View this message in context: http://activemq.2283324.n4.nabble.com/Need-help-with-a-message-broker-configuration-tp4705074p4705138.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Need help with a message broker configuration
Posted by Shine <ac...@club.webhop.org>.
Here the current configuration ...
any suggestion for improvement?
How many GB diskspace needs a "normal" kahaDB?
<broker xmlns="http://activemq.apache.org/schema/core"
brokerName="localhost" dataDirectory="${activemq.data}" useJmx="true"
schedulePeriodForDestinationPurge="600000" >
<destinationPolicy>
<policyMap>
<policyEntries>
<policyEntry topic=">" producerFlowControl="true">
<pendingMessageLimitStrategy>
<constantPendingMessageLimitStrategy limit="1000"/>
</pendingMessageLimitStrategy>
</policyEntry>
<policyEntry queue=">" producerFlowControl="true"
optimizedDispatch="true" gcInactiveDestinations="true"
inactiveTimoutBeforeGC="21600000">
<deadLetterStrategy>
<individualDeadLetterStrategy
queuePrefix="DLQ."
useQueueForQueueMessages="true"
processExpired="false"
processNonPersistent="false"/>
</deadLetterStrategy>
</policyEntry>
</policyEntries>
</policyMap>
</destinationPolicy>
<managementContext>
<managementContext createConnector="false"/>
</managementContext>
<plugins>
<timeStampingBrokerPlugin futureOnly="true" />
<simpleAuthenticationPlugin anonymousAccessAllowed="false">
<users>
<authenticationUser username="Broker324Connector"
password="xxxaaaax" groups="users,admins"/>
<authenticationUser username="Broker521Connector"
password="aaaxxxaaaa" groups="users,admins"/>
<authenticationUser username="admin"
password="ttttyyyyttt" groups="users,admins"/>
<authenticationUser username="Username111"
password="ggggeeeeggg" groups="users,admins"/>
<authenticationUser username="system"
password="${activemq.password}" groups="users,admins"/>
</users>
</simpleAuthenticationPlugin>
<authorizationPlugin>
<map>
<authorizationMap>
<authorizationEntries>
<authorizationEntry queue=">" read="admins" write="admins"
admin="admins" />
<authorizationEntry topic=">" read="admins" write="admins"
admin="admins" />
<authorizationEntry topic="ActiveMQ.Advisory.>"
read="guests,users" write="guests,users" admin="guests,users"/>
</authorizationEntries>
</authorizationMap>
</map>
</authorizationPlugin>
</plugins>
<persistenceAdapter>
<kahaDB directory="${activemq.data}/kahadb"/>
</persistenceAdapter>
<systemUsage>
<systemUsage>
<memoryUsage>
<memoryUsage percentOfJvmHeap="70" />
</memoryUsage>
<storeUsage>
<storeUsage limit="40 gb"/>
</storeUsage>
<tempUsage>
<tempUsage limit="40 gb"/>
</tempUsage>
</systemUsage>
</systemUsage>
<sslContext>
<sslContext
keyStore="file:${activemq.conf}/messagebroker-425.hoster.net.ks"
keyStorePassword="111xxx666" />
</sslContext>
<transportConnectors>
<transportConnector name="ssl"
uri="ssl://0.0.0.0:51617?maximumConnections=30000"/>
<transportConnector name="stomp+ssl"
uri="stomp+ssl://0.0.0.0:51618?maximumConnections=25000"/>
</transportConnectors>
<shutdownHooks>
<bean xmlns="http://www.springframework.org/schema/beans"
class="org.apache.activemq.hooks.SpringContextHook" />
</shutdownHooks>
</broker>
--
View this message in context: http://activemq.2283324.n4.nabble.com/Need-help-with-a-message-broker-configuration-tp4705074p4705137.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Need help with a message broker configuration
Posted by Tim Bain <tb...@gmail.com>.
ActiveMQ is not a database, and its performance is optimized for use cases
where all messages are consumed relatively soon after they're published.
There's nothing that says a message can't be consumed after some period of
time just because the consumer to whom it is destined isn't online.
You can have your own process that monitors for messages that are
unconsumed after a certain period of time, and then consumes only those
messages by storing them in a database. When an end consumer came back
online, they would need to query the database to look for old messages
before it consumes any new ones from the queue.
Maybe the additional complication of implementing that is better for you
than keeping lots of KahaDB files, and maybe it's not. That's your
trade-off to make; my point is simply that your current implementation is
not your only choice, and others might behave better in the aspect you
initially asked about, though they may also be worse in others.
Good luck!
Tim
On Dec 30, 2015 12:54 PM, "Shine" <ac...@club.webhop.org> wrote:
> Hello Tim,
>
> there is no file older than 3 weeks ...
> this is the oldest file (12 days old):
> -rw-rw-r-- 1 activemq activemq 33584906 Dez 18 15:22 db-484.log
>
> .. and i have never seen an older file than 3 weeks.
>
> Yes, you understood my experiment correctly. I did the experiment after
> midnight on a production broker .. so there were no "new" consumer at the
> second run which consumed the messages.
>
> May be I will do some tests on a test broker and consume all messages.
>
> Some infos to the messages and the long time-to-live. The message contains
> a
> user filled form for example. The receiver is a normal desktop computer and
> the computer is not 24/7 on(-line) or the user is on holyday ... that is
> the
> reason for such long time-to-live.
>
> I will spend more disk space.
>
> regards
> Shine
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Need-help-with-a-message-broker-configuration-tp4705074p4705490.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Re: Need help with a message broker configuration
Posted by Shine <ac...@club.webhop.org>.
Hello Tim,
there is no file older than 3 weeks ...
this is the oldest file (12 days old):
-rw-rw-r-- 1 activemq activemq 33584906 Dez 18 15:22 db-484.log
.. and i have never seen an older file than 3 weeks.
Yes, you understood my experiment correctly. I did the experiment after
midnight on a production broker .. so there were no "new" consumer at the
second run which consumed the messages.
May be I will do some tests on a test broker and consume all messages.
Some infos to the messages and the long time-to-live. The message contains a
user filled form for example. The receiver is a normal desktop computer and
the computer is not 24/7 on(-line) or the user is on holyday ... that is the
reason for such long time-to-live.
I will spend more disk space.
regards
Shine
--
View this message in context: http://activemq.2283324.n4.nabble.com/Need-help-with-a-message-broker-configuration-tp4705074p4705490.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Need help with a message broker configuration
Posted by Tim Bain <tb...@alumni.duke.edu>.
OK, so I want to recap and make sure I've got everything correct. Please
confirm that there's nothing I'm mis-stating.
Your KahaDB data directory contained lots of files that were older than the
age-off interval that all of your producers are using when they produce
messages, which is unexpected because every message in those files should
either have been successfully consumed or deleted due to expiration.
Although your first posted configuration doesn't list it, you have the
discarding DLQ plugin in use, so you're sure that none of those early
messages are in the DLQ.
Your use patterns involve consumers with dedicated queues, where the
consumer might disappear and never return and where no other consumer will
consume from the queue. So once the consumer for a given queue disconnects
for the very last time, any messages still in the queue or published later
will expire and there will never be another consumer connecting to the
queue.
Yesterday, you did an experiment where you browsed all the queues twice,
the first time using a selector you knew wouldn't match any messages, and
the second time using a selector that you knew would match some but not all
messages. After the first browse, there was no change in which KahaDB
files were in the data directory; after the second browse, a substantial
number of files were deleted.
Two things (maybe three) changed between your first and your second browse:
1. You used a selector that actually matched messages.
2. Time passed, which means it's possible that there were some messages
that were not eligible for expiration when you examined the data directory
after the first browse but that had been expired after the second browse.
3. If this was a production broker instance with consumers connected to
it, it's possible that consumers consumed old messages on their own.
Any of these three differences could have been responsible for data files
being deleted, but your message on December 23 said that the oldest file
has a modification date of December 8, which means that any messages in it
with a 3-week expiration period would be expiring yesterday, possibly while
you were running your experiment.
In my message on December 24, I recommended that you make a backup copy of
all of the data files, and then use a copy of that backup to stand up a
dev/test broker that you could use for experiments. To see if #2 is the
full explanation, take the backup copy (which should still have the
December 8 file in it) and place a copy of it into your dev/test broker's
data directory (while the broker is down) and restart the broker without
connecting any consumers. If the broker immediately deletes the data file,
then you know the deletion didn't require your browsing activities to
delete those files, and the broker is working as designed.
If not, repeat your browsing with the selector that matches important
messages, and see if that causes the data files to be deleted, which would
indicate that the broker isn't expiring messages when there's no activity
(consumption or browsing) on the queue, which would be a bug. But Tim Bish
has previously said (http://stackoverflow.com/a/19143643) that the broker
does periodic sweeps of all queues (even ones with no consumers) to discard
any expired messages, so I don't believe this will turn out to be the case.
You pointed out that when there are a few lingering unconsumed messages but
nearly all messages have been consumed, KahaDB's live-to-dead ratio is very
bad. Yes, this is a known limitation of the design. If you want KahaDB to
support compaction of journal files, feel free to vote for
https://issues.apache.org/jira/browse/AMQ-3978.
You also asked whether changing the data file sizes would improve things
for you. In theory, smaller files would increase the odds that any given
file doesn't contain acks for messages in an earlier files that's being
kept alive because of a small number of unconsumed messages, but in
practice it's entirely a dice roll about exactly what order your messages
are received by the broker compared to where file breaks occur. And since
your usage pattern seems to be one where messages often sit for a while
before being consumed, your odds that all acks for messages in a give file
are contained within that file are much lower than average, probably close
to zero. So although in theory having more small files increases the odds
that at least some of them can be eliminated, for your use case it might
not. But you can always give it a try and see. (Unfortunately, your
3-week message expiration period means that it'll take you almost a month
to run that experiment.)
But that's really just window dressing on the real problem: your usage
pattern is one that allows messages to sit unconsumed and not discarded for
three whole weeks at a time. This is the point where Art Naseef would
break out the refrain "ActiveMQ is not a database" and tell you that you
need to change your usage patterns so messages are dealt with (consumed or
discarded) much sooner, like maybe after one day instead of three weeks.
If you can live with clients losing messages that aren't consumed after a
day (or whatever shorter interval you choose), then just change the
JMSExpiration each producer uses and you're done. If not, you can use your
own processes to consume messages after the 1-day interval and store their
data into a database, from which consumers who reconnected later would read
before they start processing the live stream of backed-up messages. (You
could also write the messages into queues that are backed by a different
KahaDB instance; since this will only be the unconsumed messages, the disk
space used will be much smaller.) Another option would be to move to
LevelDB instead of KahaDB; recent email threads seem to indicate that
replicated LevelDB still has some bugs that are being (maybe have been)
worked out, but they all seem to be in the replication aspect so I'd expect
that non-replicated LevelDB would probably work without issue. Or you
could just leave things the way they are and allocate a lot of disk space
to KahaDB and just live with it.
Tim
On Wed, Dec 30, 2015 at 1:49 AM, Shine <ac...@club.webhop.org> wrote:
> Hi Tim,
>
> I subscribed every queue with a tool, but I used a selector which never
> match a message.
> => no changes in den data folder
>
> Than I used a selector which filters unimportant messages.
> => the server removed about 150 files from the data folder.
>
> I think the activemq server works fine and if you consume all messages than
> all files will be remove.
>
> Now 750 unconsumed messages are left on the broker. The size of each
> message
> is 3 KB.
> => 750 * 3 KB = 2,2 MB
> ==> the data folder uses 15,5 GB
>
> Is it a good idea to set the maximum size of the message log file to 8 MB
> or
> 4 MB or 2 MB instead 32 MB?
>
>
> regards
> Shine
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Need-help-with-a-message-broker-configuration-tp4705074p4705484.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Re: Need help with a message broker configuration
Posted by Shine <ac...@club.webhop.org>.
Hi Tim,
I subscribed every queue with a tool, but I used a selector which never
match a message.
=> no changes in den data folder
Than I used a selector which filters unimportant messages.
=> the server removed about 150 files from the data folder.
I think the activemq server works fine and if you consume all messages than
all files will be remove.
Now 750 unconsumed messages are left on the broker. The size of each message
is 3 KB.
=> 750 * 3 KB = 2,2 MB
==> the data folder uses 15,5 GB
Is it a good idea to set the maximum size of the message log file to 8 MB or
4 MB or 2 MB instead 32 MB?
regards
Shine
--
View this message in context: http://activemq.2283324.n4.nabble.com/Need-help-with-a-message-broker-configuration-tp4705074p4705484.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Need help with a message broker configuration
Posted by Tim Bain <tb...@alumni.duke.edu>.
Queue browsing just involves HTTP interactions with the Jetty web server,
and can be scripted if the volume of queues makes manual browsing
impossible.
After you browsed those queues, did any of the very old KahaDB data files
get deleted? That's the second question you're trying to answer by
browsing the queues: are there any messages that are past their expiration
date but haven't yet been expired by the broker, such that the pure act of
attempting to browse them triggers their deletion? (And note that you
can't answer that question until you've browsed all queues that have
unconsumed messages, because a KahaDB data file can't be deleted until
every single message in it is deleted, so browsing "a lot of queues" might
trigger a file deletion or might not, but it doesn't indicate anything
about whether that's our problem or not.)
I recommend you make a copy of your KahaDB data files and put them
somewhere you can keep coming back them, then make a copy of those files
and use the second copy to stand up a new broker in a dev/test environment,
with no consumers and no producers, so you can do some experiments without
affecting your production broker. Then purge every queue (again, this can
be scripted, with a little work) and see if all of your KahaDB files
(except the current one) get deleted. (Keep in mind that the deletion
thread runs periodically, so there may be a small delay, depending on your
configuration.) I expect that everything will get deleted once there are
no messages left in any queue, which would mean that KahaDB itself is
working correctly, and your problem is that you have old messages that
aren't being deleted.
Next, use your archived copy of the KahaDB data files to restore the test
broker to a state where all of those messages are present again, and browse
every queue. As you go, note whether any message has either no
JMSExpiration header set, or a JMSExpiration header that's more than your
intended timeout (3 weeks?) after the creation date (JMSExpiration -
JMSTimestamp > 3 weeks), which will tell you whether any of your
assumptions about the behavior of your producers are invalid and that your
perceived problem may in fact be caused by invalid expectations. When
you're done, see if any of the earliest KahaDB data files get deleted; if
it did, then your problem is that the broker isn't deleting expired
messages when there is no consumer for them.
Running those experiments should help to zero in on what's causing the
behavior you're seeing, and whether it's a bug in ActiveMQ or problems in
your software.
Also, keep in mind that the raw number of files is irrelevant; what should
be constant (after a warm-up period equal to the expiration interval all
producers are using) is the number of files *created* since the oldest live
file, of which some may have been deleted since then. KahaDB can delete
any file it's done with, and there's a lot of randomness about what
messages a given file contains and therefore whether it can be deleted, so
expecting a constant rate of file deletions isn't realistic. But since the
filenames have a one-up number in them, you can tell easily how many files
were created between the oldest file and the newest (current) file. All of
that assumes a constant rate of message production and consumption; in a
real-world application such as your own, there may be variability of data
file creation rates due to the variability of message
production/consumption rates, so the "constant" rate of file creation is
very much an average, not a fixed value.
The other thing that should be constant is that the oldest data file should
never be substantially older than your largest expiration interval (if your
producers don't all use the same value). All messages in the file should
be older than the last-modified timestamp of the file, so they should all
be candidates for expiration when the file passes the expiration interval.
If your oldest data file is not older than the max expiration interval,
then you don't have a problem.
What is the modification date/time of your oldest file? Please make sure
you answer this question now, before you go running the experiments I
recommended, since your answer could eliminate the need for you to run
those experiments.
Tim
On Wed, Dec 23, 2015 at 1:59 PM, Shine <ac...@club.webhop.org> wrote:
> Hi Tim,
>
> there are about 2500 queues and it is not so easy to browse all queues and
> unconsumed messages.
>
> I took a look to a lot of queues with unconsumed messages and all messages
> has an (valid = future date) expiration date.
>
>
> Now 561 files / 17,9 GB
>
> regards
> Shine
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Need-help-with-a-message-broker-configuration-tp4705074p4705414.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Re: Need help with a message broker configuration
Posted by Shine <ac...@club.webhop.org>.
Hi Tim,
there are about 2500 queues and it is not so easy to browse all queues and
unconsumed messages.
I took a look to a lot of queues with unconsumed messages and all messages
has an (valid = future date) expiration date.
Now 561 files / 17,9 GB
regards
Shine
--
View this message in context: http://activemq.2283324.n4.nabble.com/Need-help-with-a-message-broker-configuration-tp4705074p4705414.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Need help with a message broker configuration
Posted by Tim Bain <tb...@alumni.duke.edu>.
I'm looking forward to your answers to the questions I asked this morning,
and to hearing the results of your browsing the queues and inspecting the
oldest messages on each.
Tim
On Wed, Dec 23, 2015 at 6:32 AM, Shine <ac...@club.webhop.org> wrote:
> Hi Tim,
>
> the oldest file has the name db-159.log ... created on the 8th of december.
> ..
>
>
> regards
> Shine
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Need-help-with-a-message-broker-configuration-tp4705074p4705385.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Re: Need help with a message broker configuration
Posted by Shine <ac...@club.webhop.org>.
Hi Tim,
the oldest file has the name db-159.log ... created on the 8th of december.
..
regards
Shine
--
View this message in context: http://activemq.2283324.n4.nabble.com/Need-help-with-a-message-broker-configuration-tp4705074p4705385.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Need help with a message broker configuration
Posted by Shine <ac...@club.webhop.org>.
Hi Tim,
the folder contains 546 files now; folder size: 17,4 GB. ... 2 files removed
and ~ 40 new files since yesterday.
regards
Shine
--
View this message in context: http://activemq.2283324.n4.nabble.com/Need-help-with-a-message-broker-configuration-tp4705074p4705384.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Need help with a message broker configuration
Posted by Tim Bain <tb...@alumni.duke.edu>.
Which two files were deleted? Not their filenames, but where did they sit
in the overall order?
When you browse the oldest message on each queue that has no consumers,
look to see if the oldest message in each either has no JMSExpiration
header or has one that's further in the future than your expected range.
That's what my money's on: the broker can't remove the oldest KahaDB file
because it contains a message that can't be expired.
Tim
On Dec 22, 2015 1:43 PM, "Shine" <ac...@club.webhop.org> wrote:
> Hi Tim,
>
> in the folder are 512 files now ... and the folder size is about 16,5 GB.
>
> in about 4 hours the server removed 2 files and creates 11 new files, so
> the
> folder grows and grows.
>
> regards
> Shine
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Need-help-with-a-message-broker-configuration-tp4705074p4705366.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Re: Need help with a message broker configuration
Posted by Shine <ac...@club.webhop.org>.
Hi Tim,
in the folder are 512 files now ... and the folder size is about 16,5 GB.
in about 4 hours the server removed 2 files and creates 11 new files, so the
folder grows and grows.
regards
Shine
--
View this message in context: http://activemq.2283324.n4.nabble.com/Need-help-with-a-message-broker-configuration-tp4705074p4705366.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Need help with a message broker configuration
Posted by Tim Bain <tb...@alumni.duke.edu>.
Your original description said that log directory "grows and grows", and
now you're saying "the activemq server removes log files and creates new
files." Which is it? If you're still looking for help, I think you need
to spend a little time describing exactly what you're seeing happen with
that directory, in gory detail.
On Dec 22, 2015 8:58 AM, "Shine" <ac...@club.webhop.org> wrote:
> Hi Tim,
>
> thanks for you help.
>
> i monitored the kahadb folder for some days. the activemq server removes
> log
> files and creates new files. The folder contains 503 files currently. I
> think thats no problem!?
>
> Maybe I browse all 3000 Queues today and take a look at the kahadb folder
> again.
>
>
> regards
> Shine
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Need-help-with-a-message-broker-configuration-tp4705074p4705356.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Re: Need help with a message broker configuration
Posted by Shine <ac...@club.webhop.org>.
Hi Tim,
thanks for you help.
i monitored the kahadb folder for some days. the activemq server removes log
files and creates new files. The folder contains 503 files currently. I
think thats no problem!?
Maybe I browse all 3000 Queues today and take a look at the kahadb folder
again.
regards
Shine
--
View this message in context: http://activemq.2283324.n4.nabble.com/Need-help-with-a-message-broker-configuration-tp4705074p4705356.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Need help with a message broker configuration
Posted by Tim Bain <tb...@alumni.duke.edu>.
My primary goal in having you browse every queue with unconsumed messages
was to try to force the broker to evaluate whether those messages should be
expired, which would indicate that the problem was simply that the broker
wasn't evaluating message expiration when no consumer was connected. If
the oldest KahaDB data files suddenly disappeared after you did that, you'd
know that the problem was simply that messages that should have been
expired weren't being.
My goal was also to validate the unstated assumption you seem to be making
that every single message sent to your broker has a JMSExpiration value
set; if even one of your unconsumed messages was sent without a value for
that header, it's going to keep alive your KahaDB files.
Tim
On Sat, Dec 19, 2015 at 5:57 AM, Shine <ac...@club.webhop.org> wrote:
> Hi Tim
>
> i dont browse the queue .. i click on the "Queues" link in the WebConsole
> to
> refresh the list of queues. You see the enqueued messages and the dequeued
> messages .. if the time to live of an message is over, than the counter of
> the dequeued messages increase and enqueued messages decrease.
>
> regards
> Shine
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Need-help-with-a-message-broker-configuration-tp4705074p4705210.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Re: Need help with a message broker configuration
Posted by Shine <ac...@club.webhop.org>.
Hi Tim
i dont browse the queue .. i click on the "Queues" link in the WebConsole to
refresh the list of queues. You see the enqueued messages and the dequeued
messages .. if the time to live of an message is over, than the counter of
the dequeued messages increase and enqueued messages decrease.
regards
Shine
--
View this message in context: http://activemq.2283324.n4.nabble.com/Need-help-with-a-message-broker-configuration-tp4705074p4705210.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Need help with a message broker configuration
Posted by Tim Bain <tb...@alumni.duke.edu>.
For troubleshooting purposes, I'd recommend you use the web console or a
JMX viewer to find out which queues have unconsumed messages and no
consumers, and browse the messages on each such queue via the web console.
Browsing should (I think) run the messages through the expiration logic, so
once you've browsed all of the queues, you should have expired any old
messages. Then go to the KahaDB data directory and see if some of the
oldest files got deleted; if they did, then the problem is simply that you
need a mechanism for forcing expiration checks on messages when there's no
consumer. If not, then there's something else going on.
While you're browsing the messages, look at the oldest one on each queue
and confirm that it's got an expiration date header set; if some of your
publishers aren't setting that header, then you need to address that issue
first.
Tim
On Fri, Dec 18, 2015 at 11:46 AM, Tim Bain <tb...@alumni.duke.edu> wrote:
> Messages published to a topic are deleted from the message store when they
> are consumed by the last subscriber. When there are no subscribers, they
> are immediately deleted. So there's nothing for you to worry about there.
>
> Tim
>
> On Fri, Dec 18, 2015 at 11:23 AM, Shine <ac...@club.webhop.org> wrote:
>
>> Hi Tim.
>>
>> no queue with that name
>>
>> In the topic Tab is that entry .. SHINE is a queue name
>> ActiveMQ.Advisory.Expired.Queue.SHINER
>> Number Of Consumers: 0
>> Messages Enqueued: 2 <= expired Messages ... automatically remove from
>> queue
>> Messages Dequeued: 0
>>
>> .. but i cant remove via WebConsole .. maybe the messages are
>> automatically
>> deleted.
>>
>> regards
>> shine
>>
>>
>>
>> --
>> View this message in context:
>> http://activemq.2283324.n4.nabble.com/Need-help-with-a-message-broker-configuration-tp4705074p4705184.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
>
>
Re: Need help with a message broker configuration
Posted by Tim Bain <tb...@alumni.duke.edu>.
Messages published to a topic are deleted from the message store when they
are consumed by the last subscriber. When there are no subscribers, they
are immediately deleted. So there's nothing for you to worry about there.
Tim
On Fri, Dec 18, 2015 at 11:23 AM, Shine <ac...@club.webhop.org> wrote:
> Hi Tim.
>
> no queue with that name
>
> In the topic Tab is that entry .. SHINE is a queue name
> ActiveMQ.Advisory.Expired.Queue.SHINER
> Number Of Consumers: 0
> Messages Enqueued: 2 <= expired Messages ... automatically remove from
> queue
> Messages Dequeued: 0
>
> .. but i cant remove via WebConsole .. maybe the messages are automatically
> deleted.
>
> regards
> shine
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Need-help-with-a-message-broker-configuration-tp4705074p4705184.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Re: Need help with a message broker configuration
Posted by Shine <ac...@club.webhop.org>.
Hi Tim.
no queue with that name
In the topic Tab is that entry .. SHINE is a queue name
ActiveMQ.Advisory.Expired.Queue.SHINER
Number Of Consumers: 0
Messages Enqueued: 2 <= expired Messages ... automatically remove from
queue
Messages Dequeued: 0
.. but i cant remove via WebConsole .. maybe the messages are automatically
deleted.
regards
shine
--
View this message in context: http://activemq.2283324.n4.nabble.com/Need-help-with-a-message-broker-configuration-tp4705074p4705184.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Need help with a message broker configuration
Posted by Tim Bain <tb...@alumni.duke.edu>.
It would be in the Queues tab, with a name of DeadLetterQueue.
On Fri, Dec 18, 2015 at 10:40 AM, Shine <ac...@club.webhop.org> wrote:
> Hi,
>
> i use queues only. Each client has its own queue. It works like a chat, you
> send a message to queue .. if the client is online, it receives the message
> ... if the client is offline .. maybe 5 days, than the message(s) still in
> the queue (max. 2-3 weeks) until the client goes online.
>
> I cant find a DLQ Queue/Topic if i use the (old) ActiveMQ WebConsole. Where
> can i find the DLQ messages?
>
> regards
> Shine
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Need-help-with-a-message-broker-configuration-tp4705074p4705177.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Re: Need help with a message broker configuration
Posted by Shine <ac...@club.webhop.org>.
Hi,
i use queues only. Each client has its own queue. It works like a chat, you
send a message to queue .. if the client is online, it receives the message
... if the client is offline .. maybe 5 days, than the message(s) still in
the queue (max. 2-3 weeks) until the client goes online.
I cant find a DLQ Queue/Topic if i use the (old) ActiveMQ WebConsole. Where
can i find the DLQ messages?
regards
Shine
--
View this message in context: http://activemq.2283324.n4.nabble.com/Need-help-with-a-message-broker-configuration-tp4705074p4705177.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Need help with a message broker configuration
Posted by Tim Bain <tb...@alumni.duke.edu>.
How is it that those messages are not being consumed? Do you have no
consumers on the queue (or an offline durable topic subscriber)? Do you
have consumers, but with selectors that don't hit those messages?
Something else?
I haven't looked in detail at the message expiration code, but I believe
that messages are only checked for expiration when they're being considered
for dispatching (and again in the client when they're about to be
consumed), so I think they won't expire if there are no consumers at all.
Can anyone confirm or correct that? I'm not sure how/if consumers with
non-matching selectors affect whether messages are expired.
Incidentally, 2-3 weeks?? That's a long time...
Also, can you confirm that there are no messages at all in the DLQ? Old
DLQ messages will keep KahaDB files alive, so if you have any messages
there, that might explain what you're seeing...
Thanks,
Tim
On Thu, Dec 17, 2015 at 12:10 PM, Shine <ac...@club.webhop.org> wrote:
> Hi Tim,
>
> For problem #1: there are a lot of unconsumed messages, but each message
> havs an expiration time (max 2-3 weeks). Over the time the size of the
> KahaDB folder should be nearly constant.
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Need-help-with-a-message-broker-configuration-tp4705074p4705136.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Re: Need help with a message broker configuration
Posted by Shine <ac...@club.webhop.org>.
Hi Tim,
For problem #1: there are a lot of unconsumed messages, but each message
havs an expiration time (max 2-3 weeks). Over the time the size of the
KahaDB folder should be nearly constant.
--
View this message in context: http://activemq.2283324.n4.nabble.com/Need-help-with-a-message-broker-configuration-tp4705074p4705136.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Need help with a message broker configuration
Posted by Tim Bain <tb...@alumni.duke.edu>.
For problem #1, do you have any old but unconsumed messages? KahaDB can't
compact its journal files, so a single old message that can't be deleted
can cause KahaDB to keep a large number of files.
For problem #2, why is that a problem? You haven't said anything that
indicates that those advisory subscriptions are causing problems. But if
they are, you can disable advisory support (see the middle of
http://activemq.apache.org/advisory-message.html), since you don't have a
network of brokers.
Tim
On Dec 16, 2015 1:13 PM, "Shine" <ac...@club.webhop.org> wrote:
> Hi,
>
> i need a broker configuration for this features
>
> - protocols: OpenWire + SSL (NMS C#) on Windows; STOMP + SSL on iOS and
> Android.
> - every message has a time-to-live/expire date.
> => all expired messages need to be delete .. no rollback or other
> operations on expired messages are needed .. no DeadLetter ...
> - the Client decides if a message is persistent or not.
> - each client has its own queue and subscribes it.
> - (fast) dead client detection on the message broker; especially for the
> mobile devices.
>
> my current running message broker (ActiveMQ Version 5.11.1) has this
> problems:
> - the kahadb folder grows and grows; currently the folder contains about
> 400
> files.
> - a client has many(unnecessary) connections to this topic
> "ActiveMQ.Advisory.TempQueue,ActiveMQ.Advisory.TempTopic"
>
> thanks for help.
>
> best regards
> Shine
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Need-help-with-a-message-broker-configuration-tp4705074.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>