question: unexplained 1000 pending messages on a virtual topic

[Yatir Ben Shlomo <ya...@outbrain.com>]

Hi.
We have a multi-broker setup of AMQ 5.4.0 running on linux machines, under jetty.
On a certain virtual topic - let's call it VirtualTopic.V - we have a few Consuming Queues
One of these Queues - let's call it Consumer.C.VirtualTopic.V - has 25 Active consumer attached to it (each one is a different JVM process)
The queue is working extensively and ~800-1500 messages are being consumed every second.
But I noticed that there is a constant number of 1000 pending messages in this queue.
The thing that strikes me is that these pending messages remain constant, they are never consumed, they seem to be stuck there forever.
I tried deleting this Queue but once the queue is recreated within  seconds it grows back to 1000 pending messages which again remain constant, stuck there forever.
I also tried restarting the amq service but again -it just grew back to 100 constant messages stuck.
Any ideas ?



Here is our amq configuration

<broker xmlns="http://activemq.apache.org/schema/core" brokerName="amq" dataDirectory="${activemq.base}/data">

        <!-- Now configure the rest of the desired broker behavior -->

        <!-- Destination specific policies using destination names or wildcards -->
        <destinationPolicy>
            <policyMap>
                <policyEntries>
                    <policyEntry queue=">" producerFlowControl="true" memoryLimit="50mb">
                        <deadLetterStrategy>
                          <individualDeadLetterStrategy queuePrefix="DLQ." useQueueForQueueMessages="true" />
                        </deadLetterStrategy>
                    </policyEntry>
                    <policyEntry topic=">" producerFlowControl="true" memoryLimit="50mb">
                    </policyEntry>
                </policyEntries>
            </policyMap>
        </destinationPolicy>

        <!-- Use the following to configure how ActiveMQ is exposed in JMX -->
        <managementContext>
            <managementContext createConnector="true" connectorPort="1099"/>
        </managementContext>

                <!-- First let's configure connectors -->

        <!--
                         The store and forward broker networks ActiveMQ will listen to
                                     Create a duplex connector to the first broker
            -->
        <networkConnectors>
            <networkConnector dynamicOnly="true" networkTTL="6" name="lapool1" uri="static://(tcp://amq1:61616,tcp://amq2:61616)">
                <excludedDestinations>
                    <queue physicalName="com.outbrain.imagefetcher.download_request_queue"/>
                    <queue physicalName="Consumer.nydc1.>"/>
                                    <queue physicalName="Consumer.*.VirtualTopic.>"/>
                </excludedDestinations>
            </networkConnector>
        </networkConnectors>

        <persistenceAdapter>
            <amqPersistenceAdapter syncOnWrite="false" directory="${activemq.base}/data/broker-data" maxFileLength="20 mb"/>
        </persistenceAdapter>

        <!--  The maximum amount of space the broker will use before slowing down producers -->
        <systemUsage>
            <systemUsage>
                <memoryUsage>
                    <memoryUsage limit="3 gb"/>
                </memoryUsage>
                <storeUsage>
                    <storeUsage limit="1 gb" name="foo"/>
                </storeUsage>
                <tempUsage>
                    <tempUsage limit="10 gb"/>
                </tempUsage>
            </systemUsage>
        </systemUsage>

         <!-- The transport connectors ActiveMQ will listen to -->
        <transportConnectors>
            <transportConnector name="openwire" uri="tcp://0.0.0.0:61616?maximumConnections=4000"/>
        </transportConnectors>

    </broker>