You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@qpid.apache.org by Stephen <re...@inode.serveftp.com> on 2009/03/03 09:41:17 UTC
Stress C++ broker without flow control
Hi all,
I am stressing C++ broker with Java Client
In each test, the Java Client sends 50000 continuously.
If the deliverymode is msg.setJMSDeliveryMode(DeliveryMode.NON_PERSISTENT), the receiver looks OK.
but if using msg.setJMSDeliveryMode(DeliveryMode.PERSISTENT), the broker console throws the following exception
2009-mar-03 16:13:11 error Channel exception: not-attached: receiving Frame[BEbe; channel=1; {ExchangeDeclareBody: exchange=development; type=direct; alternate-exchange=; arguments={}; }]: channel 1 is not attached (qpid/amqp_0_10/SessionHandler.cpp:79)
2009-mar-03 16:13:11 error Channel exception: not-attached: receiving Frame[BEbe; channel=1; {QueueDeclareBody: queue=admin_in; alternate-exchange=; durable=1; arguments={}; }]: channel 1 is not attached (qpid/amqp_0_10/SessionHandler.cpp:79)
2009-mar-03 16:13:11 error Channel exception: not-attached: receiving Frame[BEbe; channel=1; {ExchangeBindBody: queue=admin_in; exchange=development; binding-key=admin_in; arguments={x-match:V2:3:str16(any)}; }]: channel 1 is not attached (qpid/amqp_0_10/SessionHandler.cpp:79)
2009-mar-03 16:13:11 error Channel exception: not-attached: receiving Frame[BEbe; channel=1; {MessageSubscribeBody: queue=admin_in; destination=2; accept-mode=0; acquire-mode=0; resume-id=; resume-ttl=0; arguments={}; }]: channel 1 is not attached (qpid/amqp_0_10/SessionHandler.cpp:79)
2009-mar-03 16:13:11 error Channel exception: not-attached: receiving Frame[BEbe; channel=1; {MessageSetFlowModeBody: destination=2; flow-mode=1; }]: channel 1 is not attached (qpid/amqp_0_10/SessionHandler.cpp:79)
2009-mar-03 16:13:11 error Channel exception: not-attached: receiving Frame[BEbe; channel=1; {MessageFlowBody: destination=2; unit=1; value=4294967295; }]: channel 1 is not attached (qpid/amqp_0_10/SessionHandler.cpp:79)
2009-mar-03 16:13:11 error Channel exception: not-attached: receiving Frame[BEbe; channel=1; {MessageFlowBody: destination=2; unit=0; value=5000; }]: channel 1 is not attached (qpid/amqp_0_10/SessionHandler.cpp:79)
and the client side sees the following error
Any suggestion?
Regards,
Stephen
-----Java Client Exception-----
161349.447 INFO QueueSender error at 32459
org.springframework.jms.UncategorizedJmsException: Uncategorized exception occured during JMS processing; nested exception is javax.jms.JMSException:
Exception when sending message; nested exception is org.apache.qpid.transport.SessionException: timed out waiting for session to become open (state=
DETACHED)
at org.springframework.jms.support.JmsUtils.convertJmsAccessException(JmsUtils.java:308)
at org.springframework.jms.support.JmsAccessor.convertJmsAccessException(JmsAccessor.java:168)
at org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:474)
at org.springframework.jms.core.JmsTemplate.send(JmsTemplate.java:539)
at com.inode.message.SimpleSender.send(SimpleSender.java:34)
at com.inode.test.QueueSender.testSend(QueueSender.java:34)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.springframework.test.context.junit4.SpringTestMethod.invoke(SpringTestMethod.java:198)
at org.springframework.test.context.junit4.SpringMethodRoadie.runTestMethod(SpringMethodRoadie.java:274)
at org.springframework.test.context.junit4.SpringMethodRoadie$2.run(SpringMethodRoadie.java:207)
at org.springframework.test.context.junit4.SpringMethodRoadie.runBeforesThenTestThenAfters(SpringMethodRoadie.java:254)
at org.springframework.test.context.junit4.SpringMethodRoadie.runWithRepetitions(SpringMethodRoadie.java:234)
at org.springframework.test.context.junit4.SpringMethodRoadie.runTest(SpringMethodRoadie.java:204)
at org.springframework.test.context.junit4.SpringMethodRoadie.run(SpringMethodRoadie.java:146)
at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.invokeTestMethod(SpringJUnit4ClassRunner.java:151)
at org.junit.internal.runners.JUnit4ClassRunner.runMethods(JUnit4ClassRunner.java:59)
at org.junit.internal.runners.JUnit4ClassRunner$1.run(JUnit4ClassRunner.java:52)
at org.junit.internal.runners.ClassRoadie.runUnprotected(ClassRoadie.java:34)
at org.junit.internal.runners.ClassRoadie.runProtected(ClassRoadie.java:44)
at org.junit.internal.runners.JUnit4ClassRunner.run(JUnit4ClassRunner.java:50)
at org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:62)
at org.apache.maven.surefire.suite.AbstractDirectoryTestSuite.executeTestSet(AbstractDirectoryTestSuite.java:140)
at org.apache.maven.surefire.suite.AbstractDirectoryTestSuite.execute(AbstractDirectoryTestSuite.java:127)
at org.apache.maven.surefire.Surefire.run(Surefire.java:177)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.maven.surefire.booter.SurefireBooter.runSuitesInProcess(SurefireBooter.java:338)
at org.apache.maven.surefire.booter.SurefireBooter.main(SurefireBooter.java:997)
Caused by: javax.jms.JMSException: Exception when sending message
at org.apache.qpid.client.BasicMessageProducer_0_10.sendMessage(BasicMessageProducer_0_10.java:177)
at org.apache.qpid.client.BasicMessageProducer.sendImpl(BasicMessageProducer.java:489)
at org.apache.qpid.client.BasicMessageProducer.sendImpl(BasicMessageProducer.java:444)
at org.apache.qpid.client.BasicMessageProducer.send(BasicMessageProducer.java:313)
at org.springframework.jms.connection.CachedMessageProducer.send(CachedMessageProducer.java:117)
at org.springframework.jms.core.JmsTemplate.doSend(JmsTemplate.java:597)
at org.springframework.jms.core.JmsTemplate.doSend(JmsTemplate.java:574)
at org.springframework.jms.core.JmsTemplate$3.doInJms(JmsTemplate.java:541)
at org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:471)
... 30 more
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 64.566 sec
Re: Stress C++ broker without flow control
Posted by Gordon Sim <gs...@redhat.com>.
Stephen wrote:
>
> The message is TextMessage of 1k size body
>
> I still cannot figure out when / how it dies - in terms of number of
> total messages, rate, duration, broker queue size (if configuration is
> possible)
The qpid-tool utility might be useful here. You can connect to your
broker (qpid-tool hostname:port), then list the queues (list queue),
then see the details of the queue you are interested in (show <id>,
where id is given in the previous listing).
This shows the number of enqueues and dequeues and the resulting queue
depth in messages and bytes.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: Stress C++ broker without flow control
Posted by Kim van der Riet <ki...@redhat.com>.
On Tue, 2009-03-03 at 19:25 +0800, Stephen wrote:
> The message is TextMessage of 1k size body
>
> I still cannot figure out when / how it dies - in terms of number of total
> messages, rate, duration, broker queue size (if configuration is possible)
>
>
Some background on the store and enqueue capacity issues:
The store is in effect a circular disk buffer which is split amongst
several files (8 default). For data security reasons, the store will not
overwrite any file which still contains enqueued records from a previous
write cycle - even if the enqueued record is right at the end of the
file.
In addition, a totally full journal is a fatal condition because
removing a record (dequeueing) requires the ability to write a dequeue
record. For this reason, the journal imposes an apporx. 80% capacity
limit on enqueues, but dequeues are not affected as these may free up
space to continue enqueueing.
When a persistent message is enqueued, the store looks ahead about 20%
of the total capacity of the journal files and makes sure that there are
no enqueued messages still in the file at that logical position. If
enqueued records exist in that file, then a enqueue capacity exception
is thrown and the enqueue denied. (This should not to be a fatal
condition; it should be possible to enqueue again once the blocking
enqueud record(s) have been dequeued.)
The size of the journal (both number and size of files) may be changed
by using appropriate configuration. However, if records are not being
dequeued properly or in a timely manner, then undequeued records can
become "stuck" and will force an enqueue capacity exception. Make sure
that records are being dequeued as they are being consumed.
There is an auto-expand mode in the works for the store; when enabled,
it will automatically expand the journal by inserting a new file between
the last writable file and the one containing the first blocking
enqueued record. This should be seen not so much as a convenience
feature, but as a last-resort action, as this is a performance-hitting
operation that will block operations on the queue for the time it takes
to create and format a large data file. Ideally the journal should be
sized correctly to begin with.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: Stress C++ broker without flow control
Posted by Stephen <re...@inode.serveftp.com>.
The message is TextMessage of 1k size body
I still cannot figure out when / how it dies - in terms of number of total
messages, rate, duration, broker queue size (if configuration is possible)
----- Original Message -----
From: "Gordon Sim" <gs...@redhat.com>
To: <us...@qpid.apache.org>
Sent: Tuesday, March 03, 2009 6:42 PM
Subject: Re: Stress C++ broker without flow control
> Stephen wrote:
>> Hi ,
>>
>> I just got the first few lines from console:
>>
>> l2009-mar-03 18:22:43 warning Journal "admin_in": Enqueue capacity
>> threshold exceeded on queue "admin_in".
>> 2009-mar-03 18:22:43 error Unexpected exception: Enqueue capacity
>> threshold exceeded on queue "admin_in". (JournalImpl.cpp:501)
>> 2009-mar-03 18:22:43 error Connection 172.16.25.95:3777 closed by error:
>> Enqueue capacity threshold exceeded on queue "admin_in".
>> (JournalImpl.cpp:501)(501)
>
> Great, thats what I was looking for. This is the root cause of the
> problem, though clearly the handling of this on the java side is also an
> issue.
>
>> I am using the following command to start C++ broker:
>> ./qpidd --num-jfiles 64 --jfile-size-pgs 128 --tpl-num-jfiles 64 --auth
>> no --port 5678
>>
>> can the queue be bigger??
>
> That should be 64 files of 128*32k(4M) each, i.e. about 262M total
> journal space. I think the exception is thrown at about 80% capacity or
> so to allow space for dequeue records to correct the situation.
>
> How large are your messages?
>
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project: http://qpid.apache.org
> Use/Interact: mailto:users-subscribe@qpid.apache.org
>
>
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: Stress C++ broker without flow control
Posted by Gordon Sim <gs...@redhat.com>.
Stephen wrote:
> Hi ,
>
> I still have question on messages stored in C++ broker, but not releasing.
>
> I am sending ~200k messages of 512bytes each from Java Client to C++ broker
> I found there is a limit in C++ broker of 100MB default of each queue,
> when the limit reach,
You can alter this with the --default-queue-limit option; setting it to
0 turns off any default limit.
> 2009-mar-05 11:22:31 notice Journal "TplStore": Created
> 2009-mar-05 11:22:31 notice Store module initialized;
> dir=/ebina/fno/qpid_c++_m4/qpid_store
> 2009-mar-05 11:22:31 notice SASL disabled: No Authentication Performed
> 2009-mar-05 11:22:31 notice Listening on TCP port 5678
> 2009-mar-05 11:22:31 notice Broker running
> 2009-mar-05 11:24:48 warning Message 204801 on
> _@development_983f81f0-272a-48f9-aecd-0f203ebd3848 cannot be released
> from memory as the queue is not durable
> 2009-mar-05 11:24:48 error Execution exception: resource-limit-exceeded:
> Policy exceeded on _@development_983f81f0-272a-48f9-aecd-0f203ebd3848 by
> message 204801 of size 512 , policy: size: max=104857600,
> current=104857094; count: unlimited; type=flow_to_disk
> (qpid/broker/QueuePolicy.cpp:90)
>
> Then I shutdown the producer and consumer, And use qpid-tool to check
> the exact queue which exceeded the limit
>
> Type Element 113 116
> 119 124 126
>
> ========================================================================================================================
>
> property vhostRef 103 103
> 103 103 103
> property name _@development_b9e6
> _@development_6d47b _@development_9e79 mgmt-HXXXXXP0XX repl-HXX
> property durable False False
> False False False
> property autoDelete False False
> False True True
> property exclusive True True
> True True True
> property arguments {} {}
> {} {} {}
> statistic msgTotalEnqueues 204801 messages 204800
> 204797 653 45
> statistic msgTotalDequeues 204801 0
> 0 653 45
> statistic msgTxnEnqueues 0 0
> 0 0 0
> statistic msgTxnDequeues 0 0
> 0 0 0
> statistic msgPersistEnqueues 204801 204800
> 204797 0 0
> statistic msgPersistDequeues 204801 0
> 0 0 0
> statistic msgDepth 0 204800
> 204797 0 0
> statistic byteDepth 0 octets 104857094
> 104856064 0 0
> statistic byteTotalEnqueues 104857606 104857094
> 104856064 86850 22238
> statistic byteTotalDequeues 104857606 0
> 0 86850 22238
> statistic byteTxnEnqueues 0 0
> 0 0 0
> statistic byteTxnDequeues 0 0
> 0 0 0
> statistic bytePersistEnqueues 104857606 104857094
> 104856064 0 0
> statistic bytePersistDequeues 104857606 0
> 0 0 0
>
> It seems that the messages are enqueue in those "queues" and never got
> deleted. And (it seems that) these queues exist forever, without manual
> actions.
>
> Any idea / suggestion about the msg stored in memory / queues? I should
> have "subscribed" / "consumed" them already
>
> Here are my settings:
> Java producer is sending msg using url:
> topic://development/admin_in?durable='false', with auto acknowledgement
> Java consumer is receiving msg from url:
> topic://development/*.*?durable='false', with client acknowledgement
>
> The binding and queue are created by C++ broker automatically.
> I have tried to increase the queue limit of C++ broker like this:
>
> ./qpid-config -a guest/guest@127.0.0.1:5678 add exchange topic development
> ./qpid-config -a guest/guest@127.0.0.1:5678 add queue linux.news
> --max-queue-size 209715200
> ./qpid-config -a guest/guest@127.0.0.1:5678 bind development linux.news
>
> but no use, as the msgs are enqueued in other queues and do not dequeue.
A few questions:
Are you using durable subscriptions in JMS?
Are you issuing acknowledgements when in client acknowledged mode?
Are you using a specific client id?
What connection URL are you using for producer and consumer?
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: Stress C++ broker without flow control
Posted by Carl Trieloff <cc...@redhat.com>.
My guess without looking it that spring-jms might be making us browse,
not dequeue. Can you run the
broker with -t and cut the section that subscribed and a few message
transfers, that will tell us.
Carl.
Stephen Lau wrote:
> I am checking the difference betweens integration of spring-jms and
> pure qpid-perftests
> The queues only cummulate when using spring-jms,, still checking the
> behaviour of default qpid-perftests
>
> Stephen
>
> ----- Original Message ----- From: "Stephen"
> <re...@inode.serveftp.com>
> To: <us...@qpid.apache.org>
> Sent: Thursday, March 05, 2009 3:03 PM
> Subject: Re: Stress C++ broker without flow control
>
>
>> Hi ,
>>
>> I still have question on messages stored in C++ broker, but not
>> releasing.
>>
>> I am sending ~200k messages of 512bytes each from Java Client to C++
>> broker
>> I found there is a limit in C++ broker of 100MB default of each
>> queue, when the limit reach,
>>
>> 2009-mar-05 11:22:31 notice Journal "TplStore": Created
>> 2009-mar-05 11:22:31 notice Store module initialized;
>> dir=/ebina/fno/qpid_c++_m4/qpid_store
>> 2009-mar-05 11:22:31 notice SASL disabled: No Authentication Performed
>> 2009-mar-05 11:22:31 notice Listening on TCP port 5678
>> 2009-mar-05 11:22:31 notice Broker running
>> 2009-mar-05 11:24:48 warning Message 204801 on
>> _@development_983f81f0-272a-48f9-aecd-0f203ebd3848 cannot be released
>> from memory as the queue is not durable
>> 2009-mar-05 11:24:48 error Execution exception:
>> resource-limit-exceeded: Policy exceeded on
>> _@development_983f81f0-272a-48f9-aecd-0f203ebd3848 by message 204801
>> of size 512 , policy: size: max=104857600, current=104857094; count:
>> unlimited; type=flow_to_disk (qpid/broker/QueuePolicy.cpp:90)
>>
>> Then I shutdown the producer and consumer, And use qpid-tool to check
>> the exact queue which exceeded the limit
>>
>> Type Element 113 116 119 124 126
>>
>> ========================================================================================================================
>>
>> property vhostRef 103 103 103 103 103
>> property name _@development_b9e6 _@development_6d47b
>> _@development_9e79 mgmt-HXXXXXP0XX repl-HXX
>> property durable False False False False False
>> property autoDelete False False False True True
>> property exclusive True True True True True
>> property arguments {} {} {} {} {}
>> statistic msgTotalEnqueues 204801 messages 204800 204797 653 45
>> statistic msgTotalDequeues 204801 0 0 653 45
>> statistic msgTxnEnqueues 0 0 0 0 0
>> statistic msgTxnDequeues 0 0 0 0 0
>> statistic msgPersistEnqueues 204801 204800 204797 0 0
>> statistic msgPersistDequeues 204801 0 0 0 0
>> statistic msgDepth 0 204800 204797 0 0
>> statistic byteDepth 0 octets 104857094 104856064 0 0
>> statistic byteTotalEnqueues 104857606 104857094 104856064 86850 22238
>> statistic byteTotalDequeues 104857606 0 0 86850 22238
>> statistic byteTxnEnqueues 0 0 0 0 0
>> statistic byteTxnDequeues 0 0 0 0 0
>> statistic bytePersistEnqueues 104857606 104857094 104856064 0 0
>> statistic bytePersistDequeues 104857606 0 0 0 0
>>
>> It seems that the messages are enqueue in those "queues" and never
>> got deleted. And (it seems that) these queues exist forever, without
>> manual actions.
>>
>> Any idea / suggestion about the msg stored in memory / queues? I
>> should have "subscribed" / "consumed" them already
>>
>> Here are my settings:
>> Java producer is sending msg using url:
>> topic://development/admin_in?durable='false', with auto acknowledgement
>> Java consumer is receiving msg from url:
>> topic://development/*.*?durable='false', with client acknowledgement
>>
>> The binding and queue are created by C++ broker automatically.
>> I have tried to increase the queue limit of C++ broker like this:
>>
>> ./qpid-config -a guest/guest@127.0.0.1:5678 add exchange topic
>> development
>> ./qpid-config -a guest/guest@127.0.0.1:5678 add queue linux.news
>> --max-queue-size 209715200
>> ./qpid-config -a guest/guest@127.0.0.1:5678 bind development linux.news
>>
>> but no use, as the msgs are enqueued in other queues and do not dequeue.
>>
>> Regards,
>> Stephen
>>
>>
>> ----- Original Message ----- From: "Rafael Schloming"
>> <ra...@redhat.com>
>> To: <us...@qpid.apache.org>
>> Sent: Wednesday, March 04, 2009 4:46 AM
>> Subject: Re: Stress C++ broker without flow control
>>
>>
>>> Gordon Sim wrote:
>>>> Stephen wrote:
>>>>> Hi ,
>>>>>
>>>>> I just got the first few lines from console:
>>>>>
>>>>> l2009-mar-03 18:22:43 warning Journal "admin_in": Enqueue capacity
>>>>> threshold exceeded on queue "admin_in".
>>>>> 2009-mar-03 18:22:43 error Unexpected exception: Enqueue capacity
>>>>> threshold exceeded on queue "admin_in". (JournalImpl.cpp:501)
>>>>> 2009-mar-03 18:22:43 error Connection 172.16.25.95:3777 closed by
>>>>> error:
>>>>> Enqueue capacity threshold exceeded on queue "admin_in".
>>>>> (JournalImpl.cpp:501)(501)
>>>>
>>>> Great, thats what I was looking for. This is the root cause of the
>>>> problem, though clearly the handling of this on the java side is
>>>> also an
>>>> issue.
>>>
>>> The Java exception handling should now be fixed on trunk.
>>>
>>> --Rafael
>>>
>>>
>>> ---------------------------------------------------------------------
>>> Apache Qpid - AMQP Messaging Implementation
>>> Project: http://qpid.apache.org
>>> Use/Interact: mailto:users-subscribe@qpid.apache.org
>>>
>>>
>>
>>
>> ---------------------------------------------------------------------
>> Apache Qpid - AMQP Messaging Implementation
>> Project: http://qpid.apache.org
>> Use/Interact: mailto:users-subscribe@qpid.apache.org
>>
>>
>
>
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project: http://qpid.apache.org
> Use/Interact: mailto:users-subscribe@qpid.apache.org
>
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: Stress C++ broker without flow control
Posted by Stephen Lau <st...@xml-asia.org>.
I am checking the difference betweens integration of spring-jms and pure
qpid-perftests
The queues only cummulate when using spring-jms,, still checking the
behaviour of default qpid-perftests
Stephen
----- Original Message -----
From: "Stephen" <re...@inode.serveftp.com>
To: <us...@qpid.apache.org>
Sent: Thursday, March 05, 2009 3:03 PM
Subject: Re: Stress C++ broker without flow control
> Hi ,
>
> I still have question on messages stored in C++ broker, but not releasing.
>
> I am sending ~200k messages of 512bytes each from Java Client to C++
> broker
> I found there is a limit in C++ broker of 100MB default of each queue,
> when the limit reach,
>
> 2009-mar-05 11:22:31 notice Journal "TplStore": Created
> 2009-mar-05 11:22:31 notice Store module initialized;
> dir=/ebina/fno/qpid_c++_m4/qpid_store
> 2009-mar-05 11:22:31 notice SASL disabled: No Authentication Performed
> 2009-mar-05 11:22:31 notice Listening on TCP port 5678
> 2009-mar-05 11:22:31 notice Broker running
> 2009-mar-05 11:24:48 warning Message 204801 on
> _@development_983f81f0-272a-48f9-aecd-0f203ebd3848 cannot be released from
> memory as the queue is not durable
> 2009-mar-05 11:24:48 error Execution exception: resource-limit-exceeded:
> Policy exceeded on _@development_983f81f0-272a-48f9-aecd-0f203ebd3848 by
> message 204801 of size 512 , policy: size: max=104857600,
> current=104857094; count: unlimited; type=flow_to_disk
> (qpid/broker/QueuePolicy.cpp:90)
>
> Then I shutdown the producer and consumer, And use qpid-tool to check the
> exact queue which exceeded the limit
>
> Type Element 113 116 119
> 124 126
>
> ========================================================================================================================
> property vhostRef 103 103 103
> 103 103
> property name _@development_b9e6
> _@development_6d47b _@development_9e79 mgmt-HXXXXXP0XX repl-HXX
> property durable False False False
> False False
> property autoDelete False False False
> True True
> property exclusive True True True
> True True
> property arguments {} {} {}
> {} {}
> statistic msgTotalEnqueues 204801 messages 204800 204797
> 653 45
> statistic msgTotalDequeues 204801 0 0
> 653 45
> statistic msgTxnEnqueues 0 0 0
> 0 0
> statistic msgTxnDequeues 0 0 0
> 0 0
> statistic msgPersistEnqueues 204801 204800 204797
> 0 0
> statistic msgPersistDequeues 204801 0 0
> 0 0
> statistic msgDepth 0 204800 204797
> 0 0
> statistic byteDepth 0 octets 104857094
> 104856064 0 0
> statistic byteTotalEnqueues 104857606 104857094
> 104856064 86850 22238
> statistic byteTotalDequeues 104857606 0 0
> 86850 22238
> statistic byteTxnEnqueues 0 0 0
> 0 0
> statistic byteTxnDequeues 0 0 0
> 0 0
> statistic bytePersistEnqueues 104857606 104857094
> 104856064 0 0
> statistic bytePersistDequeues 104857606 0 0
> 0 0
>
> It seems that the messages are enqueue in those "queues" and never got
> deleted. And (it seems that) these queues exist forever, without manual
> actions.
>
> Any idea / suggestion about the msg stored in memory / queues? I should
> have "subscribed" / "consumed" them already
>
> Here are my settings:
> Java producer is sending msg using url:
> topic://development/admin_in?durable='false', with auto acknowledgement
> Java consumer is receiving msg from url:
> topic://development/*.*?durable='false', with client acknowledgement
>
> The binding and queue are created by C++ broker automatically.
> I have tried to increase the queue limit of C++ broker like this:
>
> ./qpid-config -a guest/guest@127.0.0.1:5678 add exchange topic development
> ./qpid-config -a guest/guest@127.0.0.1:5678 add queue
> linux.news --max-queue-size 209715200
> ./qpid-config -a guest/guest@127.0.0.1:5678 bind development linux.news
>
> but no use, as the msgs are enqueued in other queues and do not dequeue.
>
> Regards,
> Stephen
>
>
> ----- Original Message -----
> From: "Rafael Schloming" <ra...@redhat.com>
> To: <us...@qpid.apache.org>
> Sent: Wednesday, March 04, 2009 4:46 AM
> Subject: Re: Stress C++ broker without flow control
>
>
>> Gordon Sim wrote:
>>> Stephen wrote:
>>>> Hi ,
>>>>
>>>> I just got the first few lines from console:
>>>>
>>>> l2009-mar-03 18:22:43 warning Journal "admin_in": Enqueue capacity
>>>> threshold exceeded on queue "admin_in".
>>>> 2009-mar-03 18:22:43 error Unexpected exception: Enqueue capacity
>>>> threshold exceeded on queue "admin_in". (JournalImpl.cpp:501)
>>>> 2009-mar-03 18:22:43 error Connection 172.16.25.95:3777 closed by
>>>> error:
>>>> Enqueue capacity threshold exceeded on queue "admin_in".
>>>> (JournalImpl.cpp:501)(501)
>>>
>>> Great, thats what I was looking for. This is the root cause of the
>>> problem, though clearly the handling of this on the java side is also an
>>> issue.
>>
>> The Java exception handling should now be fixed on trunk.
>>
>> --Rafael
>>
>>
>> ---------------------------------------------------------------------
>> Apache Qpid - AMQP Messaging Implementation
>> Project: http://qpid.apache.org
>> Use/Interact: mailto:users-subscribe@qpid.apache.org
>>
>>
>
>
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project: http://qpid.apache.org
> Use/Interact: mailto:users-subscribe@qpid.apache.org
>
>
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: Stress C++ broker without flow control
Posted by Stephen <re...@inode.serveftp.com>.
Hi ,
I still have question on messages stored in C++ broker, but not releasing.
I am sending ~200k messages of 512bytes each from Java Client to C++ broker
I found there is a limit in C++ broker of 100MB default of each queue, when
the limit reach,
2009-mar-05 11:22:31 notice Journal "TplStore": Created
2009-mar-05 11:22:31 notice Store module initialized;
dir=/ebina/fno/qpid_c++_m4/qpid_store
2009-mar-05 11:22:31 notice SASL disabled: No Authentication Performed
2009-mar-05 11:22:31 notice Listening on TCP port 5678
2009-mar-05 11:22:31 notice Broker running
2009-mar-05 11:24:48 warning Message 204801 on
_@development_983f81f0-272a-48f9-aecd-0f203ebd3848 cannot be released from
memory as the queue is not durable
2009-mar-05 11:24:48 error Execution exception: resource-limit-exceeded:
Policy exceeded on _@development_983f81f0-272a-48f9-aecd-0f203ebd3848 by
message 204801 of size 512 , policy: size: max=104857600, current=104857094;
count: unlimited; type=flow_to_disk (qpid/broker/QueuePolicy.cpp:90)
Then I shutdown the producer and consumer, And use qpid-tool to check the
exact queue which exceeded the limit
Type Element 113 116
119 124 126
========================================================================================================================
property vhostRef 103 103
103 103 103
property name _@development_b9e6
_@development_6d47b _@development_9e79 mgmt-HXXXXXP0XX repl-HXX
property durable False False
False False False
property autoDelete False False
False True True
property exclusive True True
True True True
property arguments {} {}
{} {} {}
statistic msgTotalEnqueues 204801 messages 204800
204797 653 45
statistic msgTotalDequeues 204801 0
0 653 45
statistic msgTxnEnqueues 0 0
0 0 0
statistic msgTxnDequeues 0 0
0 0 0
statistic msgPersistEnqueues 204801 204800
204797 0 0
statistic msgPersistDequeues 204801 0
0 0 0
statistic msgDepth 0 204800
204797 0 0
statistic byteDepth 0 octets 104857094
104856064 0 0
statistic byteTotalEnqueues 104857606 104857094
104856064 86850 22238
statistic byteTotalDequeues 104857606 0
0 86850 22238
statistic byteTxnEnqueues 0 0
0 0 0
statistic byteTxnDequeues 0 0
0 0 0
statistic bytePersistEnqueues 104857606 104857094
104856064 0 0
statistic bytePersistDequeues 104857606 0
0 0 0
It seems that the messages are enqueue in those "queues" and never got
deleted. And (it seems that) these queues exist forever, without manual
actions.
Any idea / suggestion about the msg stored in memory / queues? I should have
"subscribed" / "consumed" them already
Here are my settings:
Java producer is sending msg using url:
topic://development/admin_in?durable='false', with auto acknowledgement
Java consumer is receiving msg from url:
topic://development/*.*?durable='false', with client acknowledgement
The binding and queue are created by C++ broker automatically.
I have tried to increase the queue limit of C++ broker like this:
./qpid-config -a guest/guest@127.0.0.1:5678 add exchange topic development
./qpid-config -a guest/guest@127.0.0.1:5678 add queue
linux.news --max-queue-size 209715200
./qpid-config -a guest/guest@127.0.0.1:5678 bind development linux.news
but no use, as the msgs are enqueued in other queues and do not dequeue.
Regards,
Stephen
----- Original Message -----
From: "Rafael Schloming" <ra...@redhat.com>
To: <us...@qpid.apache.org>
Sent: Wednesday, March 04, 2009 4:46 AM
Subject: Re: Stress C++ broker without flow control
> Gordon Sim wrote:
>> Stephen wrote:
>>> Hi ,
>>>
>>> I just got the first few lines from console:
>>>
>>> l2009-mar-03 18:22:43 warning Journal "admin_in": Enqueue capacity
>>> threshold exceeded on queue "admin_in".
>>> 2009-mar-03 18:22:43 error Unexpected exception: Enqueue capacity
>>> threshold exceeded on queue "admin_in". (JournalImpl.cpp:501)
>>> 2009-mar-03 18:22:43 error Connection 172.16.25.95:3777 closed by error:
>>> Enqueue capacity threshold exceeded on queue "admin_in".
>>> (JournalImpl.cpp:501)(501)
>>
>> Great, thats what I was looking for. This is the root cause of the
>> problem, though clearly the handling of this on the java side is also an
>> issue.
>
> The Java exception handling should now be fixed on trunk.
>
> --Rafael
>
>
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project: http://qpid.apache.org
> Use/Interact: mailto:users-subscribe@qpid.apache.org
>
>
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: Stress C++ broker without flow control
Posted by Rafael Schloming <ra...@redhat.com>.
Gordon Sim wrote:
> Stephen wrote:
>> Hi ,
>>
>> I just got the first few lines from console:
>>
>> l2009-mar-03 18:22:43 warning Journal "admin_in": Enqueue capacity
>> threshold exceeded on queue "admin_in".
>> 2009-mar-03 18:22:43 error Unexpected exception: Enqueue capacity
>> threshold exceeded on queue "admin_in". (JournalImpl.cpp:501)
>> 2009-mar-03 18:22:43 error Connection 172.16.25.95:3777 closed by error:
>> Enqueue capacity threshold exceeded on queue "admin_in".
>> (JournalImpl.cpp:501)(501)
>
> Great, thats what I was looking for. This is the root cause of the
> problem, though clearly the handling of this on the java side is also an
> issue.
The Java exception handling should now be fixed on trunk.
--Rafael
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: Stress C++ broker without flow control
Posted by Gordon Sim <gs...@redhat.com>.
Stephen wrote:
> Hi ,
>
> I just got the first few lines from console:
>
> l2009-mar-03 18:22:43 warning Journal "admin_in": Enqueue capacity
> threshold exceeded on queue "admin_in".
> 2009-mar-03 18:22:43 error Unexpected exception: Enqueue capacity
> threshold exceeded on queue "admin_in". (JournalImpl.cpp:501)
> 2009-mar-03 18:22:43 error Connection 172.16.25.95:3777 closed by error:
> Enqueue capacity threshold exceeded on queue "admin_in".
> (JournalImpl.cpp:501)(501)
Great, thats what I was looking for. This is the root cause of the
problem, though clearly the handling of this on the java side is also an
issue.
> I am using the following command to start C++ broker:
> ./qpidd --num-jfiles 64 --jfile-size-pgs 128 --tpl-num-jfiles 64 --auth
> no --port 5678
>
> can the queue be bigger??
That should be 64 files of 128*32k(4M) each, i.e. about 262M total
journal space. I think the exception is thrown at about 80% capacity or
so to allow space for dequeue records to correct the situation.
How large are your messages?
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: Stress C++ broker without flow control
Posted by Stephen <re...@inode.serveftp.com>.
Hi ,
I just got the first few lines from console:
l2009-mar-03 18:22:43 warning Journal "admin_in": Enqueue capacity threshold
exceeded on queue "admin_in".
2009-mar-03 18:22:43 error Unexpected exception: Enqueue capacity threshold
exceeded on queue "admin_in". (JournalImpl.cpp:501)
2009-mar-03 18:22:43 error Connection 172.16.25.95:3777 closed by error:
Enqueue capacity threshold exceeded on queue "admin_in".
(JournalImpl.cpp:501)(501)
I am using the following command to start C++ broker:
./qpidd --num-jfiles 64 --jfile-size-pgs 128 --tpl-num-jfiles 64 --auth
no --port 5678
can the queue be bigger??
Stephen Lau
----- Original Message -----
From: "Gordon Sim" <gs...@redhat.com>
To: <us...@qpid.apache.org>
Sent: Tuesday, March 03, 2009 5:53 PM
Subject: Re: Stress C++ broker without flow control
> Stephen wrote:
>> Hi all,
>>
>> I am stressing C++ broker with Java Client
>> In each test, the Java Client sends 50000 continuously.
>> If the deliverymode is
>> msg.setJMSDeliveryMode(DeliveryMode.NON_PERSISTENT), the receiver looks
>> OK.
>> but if using msg.setJMSDeliveryMode(DeliveryMode.PERSISTENT), the broker
>> console throws the following exception
>
> Were there any errors prior to these in the broker log, or were these
> the only ones?
>
>>
>>
>> 2009-mar-03 16:13:11 error Channel exception: not-attached: receiving
>> Frame[BEbe; channel=1; {ExchangeDeclareBody: exchange=development;
>> type=direct; alternate-exchange=; arguments={}; }]: channel 1 is not
>> attached (qpid/amqp_0_10/SessionHandler.cpp:79)
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: Stress C++ broker without flow control
Posted by Gordon Sim <gs...@redhat.com>.
Stephen wrote:
> Hi all,
>
> I am stressing C++ broker with Java Client
> In each test, the Java Client sends 50000 continuously.
> If the deliverymode is msg.setJMSDeliveryMode(DeliveryMode.NON_PERSISTENT), the receiver looks OK.
> but if using msg.setJMSDeliveryMode(DeliveryMode.PERSISTENT), the broker console throws the following exception
Were there any errors prior to these in the broker log, or were these
the only ones?
>
>
> 2009-mar-03 16:13:11 error Channel exception: not-attached: receiving Frame[BEbe; channel=1; {ExchangeDeclareBody: exchange=development; type=direct; alternate-exchange=; arguments={}; }]: channel 1 is not attached (qpid/amqp_0_10/SessionHandler.cpp:79)
> 2009-mar-03 16:13:11 error Channel exception: not-attached: receiving Frame[BEbe; channel=1; {QueueDeclareBody: queue=admin_in; alternate-exchange=; durable=1; arguments={}; }]: channel 1 is not attached (qpid/amqp_0_10/SessionHandler.cpp:79)
> 2009-mar-03 16:13:11 error Channel exception: not-attached: receiving Frame[BEbe; channel=1; {ExchangeBindBody: queue=admin_in; exchange=development; binding-key=admin_in; arguments={x-match:V2:3:str16(any)}; }]: channel 1 is not attached (qpid/amqp_0_10/SessionHandler.cpp:79)
> 2009-mar-03 16:13:11 error Channel exception: not-attached: receiving Frame[BEbe; channel=1; {MessageSubscribeBody: queue=admin_in; destination=2; accept-mode=0; acquire-mode=0; resume-id=; resume-ttl=0; arguments={}; }]: channel 1 is not attached (qpid/amqp_0_10/SessionHandler.cpp:79)
> 2009-mar-03 16:13:11 error Channel exception: not-attached: receiving Frame[BEbe; channel=1; {MessageSetFlowModeBody: destination=2; flow-mode=1; }]: channel 1 is not attached (qpid/amqp_0_10/SessionHandler.cpp:79)
> 2009-mar-03 16:13:11 error Channel exception: not-attached: receiving Frame[BEbe; channel=1; {MessageFlowBody: destination=2; unit=1; value=4294967295; }]: channel 1 is not attached (qpid/amqp_0_10/SessionHandler.cpp:79)
> 2009-mar-03 16:13:11 error Channel exception: not-attached: receiving Frame[BEbe; channel=1; {MessageFlowBody: destination=2; unit=0; value=5000; }]: channel 1 is not attached (qpid/amqp_0_10/SessionHandler.cpp:79)
>
> and the client side sees the following error
>
> Any suggestion?
>
> Regards,
> Stephen
>
> -----Java Client Exception-----
>
> 161349.447 INFO QueueSender error at 32459
> org.springframework.jms.UncategorizedJmsException: Uncategorized exception occured during JMS processing; nested exception is javax.jms.JMSException:
> Exception when sending message; nested exception is org.apache.qpid.transport.SessionException: timed out waiting for session to become open (state=
> DETACHED)
> at org.springframework.jms.support.JmsUtils.convertJmsAccessException(JmsUtils.java:308)
> at org.springframework.jms.support.JmsAccessor.convertJmsAccessException(JmsAccessor.java:168)
> at org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:474)
> at org.springframework.jms.core.JmsTemplate.send(JmsTemplate.java:539)
> at com.inode.message.SimpleSender.send(SimpleSender.java:34)
> at com.inode.test.QueueSender.testSend(QueueSender.java:34)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.springframework.test.context.junit4.SpringTestMethod.invoke(SpringTestMethod.java:198)
> at org.springframework.test.context.junit4.SpringMethodRoadie.runTestMethod(SpringMethodRoadie.java:274)
> at org.springframework.test.context.junit4.SpringMethodRoadie$2.run(SpringMethodRoadie.java:207)
> at org.springframework.test.context.junit4.SpringMethodRoadie.runBeforesThenTestThenAfters(SpringMethodRoadie.java:254)
> at org.springframework.test.context.junit4.SpringMethodRoadie.runWithRepetitions(SpringMethodRoadie.java:234)
> at org.springframework.test.context.junit4.SpringMethodRoadie.runTest(SpringMethodRoadie.java:204)
> at org.springframework.test.context.junit4.SpringMethodRoadie.run(SpringMethodRoadie.java:146)
> at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.invokeTestMethod(SpringJUnit4ClassRunner.java:151)
> at org.junit.internal.runners.JUnit4ClassRunner.runMethods(JUnit4ClassRunner.java:59)
> at org.junit.internal.runners.JUnit4ClassRunner$1.run(JUnit4ClassRunner.java:52)
> at org.junit.internal.runners.ClassRoadie.runUnprotected(ClassRoadie.java:34)
> at org.junit.internal.runners.ClassRoadie.runProtected(ClassRoadie.java:44)
> at org.junit.internal.runners.JUnit4ClassRunner.run(JUnit4ClassRunner.java:50)
> at org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:62)
> at org.apache.maven.surefire.suite.AbstractDirectoryTestSuite.executeTestSet(AbstractDirectoryTestSuite.java:140)
> at org.apache.maven.surefire.suite.AbstractDirectoryTestSuite.execute(AbstractDirectoryTestSuite.java:127)
> at org.apache.maven.surefire.Surefire.run(Surefire.java:177)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.maven.surefire.booter.SurefireBooter.runSuitesInProcess(SurefireBooter.java:338)
> at org.apache.maven.surefire.booter.SurefireBooter.main(SurefireBooter.java:997)
> Caused by: javax.jms.JMSException: Exception when sending message
> at org.apache.qpid.client.BasicMessageProducer_0_10.sendMessage(BasicMessageProducer_0_10.java:177)
> at org.apache.qpid.client.BasicMessageProducer.sendImpl(BasicMessageProducer.java:489)
> at org.apache.qpid.client.BasicMessageProducer.sendImpl(BasicMessageProducer.java:444)
> at org.apache.qpid.client.BasicMessageProducer.send(BasicMessageProducer.java:313)
> at org.springframework.jms.connection.CachedMessageProducer.send(CachedMessageProducer.java:117)
> at org.springframework.jms.core.JmsTemplate.doSend(JmsTemplate.java:597)
> at org.springframework.jms.core.JmsTemplate.doSend(JmsTemplate.java:574)
> at org.springframework.jms.core.JmsTemplate$3.doInJms(JmsTemplate.java:541)
> at org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:471)
> ... 30 more
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 64.566 sec
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org