You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@qpid.apache.org by or...@apache.org on 2017/02/24 14:49:50 UTC
svn commit: r1784283 - in /qpid/java/trunk/doc/java-broker/src/docbkx:
concepts/Java-Broker-Concepts-Queues.xml
runtime/Java-Broker-Runtime-Disk-Space-Management-Producer-Flow-Control.xml
Author: orudyy
Date: Fri Feb 24 14:49:50 2017
New Revision: 1784283
URL: http://svn.apache.org/viewvc?rev=1784283&view=rev
Log:
QPID-7618: Add documentation for queue overflow policies
Modified:
qpid/java/trunk/doc/java-broker/src/docbkx/concepts/Java-Broker-Concepts-Queues.xml
qpid/java/trunk/doc/java-broker/src/docbkx/runtime/Java-Broker-Runtime-Disk-Space-Management-Producer-Flow-Control.xml
Modified: qpid/java/trunk/doc/java-broker/src/docbkx/concepts/Java-Broker-Concepts-Queues.xml
URL: http://svn.apache.org/viewvc/qpid/java/trunk/doc/java-broker/src/docbkx/concepts/Java-Broker-Concepts-Queues.xml?rev=1784283&r1=1784282&r2=1784283&view=diff
==============================================================================
--- qpid/java/trunk/doc/java-broker/src/docbkx/concepts/Java-Broker-Concepts-Queues.xml (original)
+++ qpid/java/trunk/doc/java-broker/src/docbkx/concepts/Java-Broker-Concepts-Queues.xml Fri Feb 24 14:49:50 2017
@@ -375,4 +375,236 @@ amqp://guest:guest@client1/development?m
</para>
</section>
</section>
+
+ <section xml:id="Java-Broker-Concepts-Queue-OverflowPolicy" role="h2">
+ <title>Controlling Queue Size</title>
+ <para>
+ <emphasis>Overflow Policy</emphasis>
+ can be configured on an individual <emphasis>Queue</emphasis> to limit the queue capacity.
+ The capacity limits can be specified as a <emphasis>maximum number of bytes</emphasis> or
+ <emphasis>maximum number of messages</emphasis> on a <emphasis>Queue</emphasis>.
+ The <emphasis>Overflow Policy</emphasis> defines the Queue behaviour when any of the limits is reached.
+ </para>
+ <para>
+ The following <emphasis>Overflow Policies</emphasis> are supported:
+ <itemizedlist>
+ <listitem>
+ <para>
+ <emphasis>None</emphasis>
+ - Queue is unbounded and the capacity limits are not applied.
+ This is a default policy applied implicitly when policy is not set explicitly.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <emphasis>Ring</emphasis>
+ - The oldest messages are removed. For a
+ <link linkend="Java-Broker-Concepts-Queues-Types-Priority">Priority Queue</link>
+ oldest messages with lowest priorities are removed. For a
+ <link linkend="Java-Broker-Concepts-Queues-Types-Sorted">Sorted Queue</link>
+ oldest messages with lowest keys are removed.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <emphasis>Producer Flow Control</emphasis>
+ -The producing sessions
+ are blocked until queue depth falls below the <emphasis>resume threshold</emphasis> set as a
+ context variable <emphasis>\${queue.queueFlowResumeLimit}</emphasis>
+ (specifying the percentage from the limit values. Default is 80%).
+ </para>
+ </listitem>
+ </itemizedlist>
+ </para>
+ <para>
+ <emphasis>Overflow Policy</emphasis> can be set using queue attribute <varname>overflowPolicy</varname>.
+ You can set it to <varname>RING</varname> to configure <emphasis>Ring</emphasis> overflow policy.
+ <varname>PRODUCER_FLOW_CONTROL</varname> sets the <emphasis>Overflow Policy</emphasis> to
+ <emphasis>Producer Flow Control</emphasis>. <varname>NONE</varname> can be used to set
+ <emphasis>None</emphasis> overflow policy explicitly.
+ <emphasis>Maximum number of messages</emphasis> can be set using queue attribute
+ <varname>maximumQueueDepthMessages</varname>.
+ <emphasis>Maximum number of bytes</emphasis> can be set using
+ queue attribute <varname>maximumQueueDepthBytes</varname>
+ <note>
+ <para>
+ Negative values for <emphasis>maximum number of messages</emphasis> and
+ <emphasis>maximum number of bytes</emphasis>
+ can be used to remove the upper limit (simply making queue unbounded by the limit where it was set).
+ </para>
+ </note>
+ </para>
+ <para>The <emphasis>Overflow Policy</emphasis> and the limits can be specified using
+ <emphasis>Address</emphasis>
+ based syntax as in the example below:
+ <screen><![CDATA[
+my-queue; {create: always, node: {x-declare: {arguments:{'qpid.max_count': 10000, 'qpid.max_size': 102400, 'qpid.policy_type: 'ring'}}}}
+]]></screen>
+ <itemizedlist>
+ <listitem>
+ <para>
+ Argument <varname>qpid.max_count</varname> is used to define <emphasis>maximum number of
+ messages</emphasis>.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ Argument <varname>qpid.max_size</varname> is used to define <emphasis>maximum number of
+ bytes</emphasis>.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ Argument <varname>qpid.policy_type</varname> is used to define <emphasis>Overflow
+ Policy</emphasis>.
+ </para>
+ </listitem>
+ </itemizedlist>
+ </para>
+ <para>
+ Alternatively, <emphasis>Producer Flow Control</emphasis> can be specified with
+ <emphasis>Address</emphasis> based syntax using arguments <varname>x-qpid-flow-resume-capacity</varname> and
+ <varname>x-qpid-capacity</varname>, for example:
+ <screen><![CDATA[
+my-queue; {create: always, node: {x-declare: {arguments:{'x-qpid-capacity': 102400, 'x-qpid-flow-resume-capacity': 8192000}}}}
+]]></screen>
+ <para>This is a deprecated syntax. It is limited to specifying the capacity and resume threshold in bytes.</para>
+ <itemizedlist>
+ <listitem>
+ <para>
+ Argument <varname>x-qpid-capacity</varname> is used to define <emphasis>maximum number of
+ bytes</emphasis>.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ Argument <varname>x-qpid-flow-resume-capacity</varname> is used to define flow resume threshold
+ in bytes.
+ </para>
+ </listitem>
+ </itemizedlist>
+ </para>
+
+ <note>
+ <para>
+ The <emphasis>Producer Flow Control</emphasis> can be triggered when a configured disk
+ quota is exceeded. For more details, please read
+ <xref linkend="Java-Broker-Runtime-Disk-Space-Management-Producer-Flow-Control"/>.
+ </para>
+ </note>
+ <note>
+ <para>
+ The impact of the <emphasis>Producer Flow Control</emphasis> on JMS Client is documented
+ <link xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="${qpidJmsClient08Book}/JMS-Client-0-8-Appendix-ProducerFlowControl-Impact.html">here</link>
+ </para>
+ </note>
+ <section role="h4">
+ <title>Overflow Policies Operational Log Messages</title>
+ <para>
+ When <emphasis>Ring</emphasis> overflow policy is configured and any of the limits is breached the
+ following operational log message is issued
+ <screen><![CDATA[
+QUE-1005 : Dropped : {0,number} messages, Depth : {1,number} bytes, {2,number} messages, Capacity : {3,number} bytes, {4,number} messages
+]]></screen>
+ <itemizedlist>
+ <listitem>
+ <para>
+ <varname>{0,number}</varname>
+ a placeholder to provide the number of deleted messages.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <varname>{1,number}</varname>
+ a placeholder to provide the queue depth in bytes.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <varname>{2,number}</varname>
+ a placeholder to provide the queue depth in messages.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <varname>{3,number}</varname>
+ a placeholder to provide <emphasis>maximum number of bytes</emphasis>.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <varname>{4,number}</varname>
+ a placeholder to provide the <emphasis>maximum number of messages</emphasis>.
+ </para>
+ </listitem>
+ </itemizedlist>
+ </para>
+ <para>
+ When <emphasis>Producer Flow Control</emphasis> overflow policy is configured and any of the capacity
+ limits is breached the following operational log message is issued
+ <screen><![CDATA[
+QUE-1003 : Overfull : Size : {0,number} bytes, Capacity : {1,number}, Messages : {2,number}, Message Capacity : {3,number}
+]]></screen>
+ <itemizedlist>
+ <listitem>
+ <para>
+ <varname>{0,number}</varname>
+ a placeholder to provide the queue depth in bytes.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <varname>{1,number}</varname>
+ a placeholder to provide the <emphasis>maximum number of bytes</emphasis>.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <varname>{2,number}</varname>
+ a placeholder to provide the queue depth in messages.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <varname>{3,number}</varname>
+ a placeholder to provide <emphasis>maximum number of messages</emphasis>.
+ </para>
+ </listitem>
+ </itemizedlist>
+ </para>
+ <para>
+ When <emphasis>Producer Flow Control</emphasis> overflow policy is configured and flow is blocked, the
+ following operational log message is issued on reaching the resume threshold
+ <screen><![CDATA[
+QUE-1004 : Underfull : Size : {0,number} bytes, Resume Capacity : {1,number}, Messages : {2,number}, Message Capacity : {3,number}
+]]></screen>
+ <itemizedlist>
+ <listitem>
+ <para>
+ <varname>{0,number}</varname>
+ a placeholder to provide the queue depth in bytes.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <varname>{1,number}</varname>
+ a placeholder to provide the <emphasis>maximum number of bytes</emphasis>.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <varname>{2,number}</varname>
+ a placeholder to provide the queue depth in messages.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <varname>{3,number}</varname>
+ a placeholder to provide <emphasis>maximum number of messages</emphasis>.
+ </para>
+ </listitem>
+ </itemizedlist>
+ </para>
+ </section>
</section>
+</section>
Modified: qpid/java/trunk/doc/java-broker/src/docbkx/runtime/Java-Broker-Runtime-Disk-Space-Management-Producer-Flow-Control.xml
URL: http://svn.apache.org/viewvc/qpid/java/trunk/doc/java-broker/src/docbkx/runtime/Java-Broker-Runtime-Disk-Space-Management-Producer-Flow-Control.xml?rev=1784283&r1=1784282&r2=1784283&view=diff
==============================================================================
--- qpid/java/trunk/doc/java-broker/src/docbkx/runtime/Java-Broker-Runtime-Disk-Space-Management-Producer-Flow-Control.xml (original)
+++ qpid/java/trunk/doc/java-broker/src/docbkx/runtime/Java-Broker-Runtime-Disk-Space-Management-Producer-Flow-Control.xml Fri Feb 24 14:49:50 2017
@@ -20,163 +20,63 @@
-->
-<section xmlns="http://docbook.org/ns/docbook" version="5.0" xml:id="Qpid-Producer-Flow-Control">
- <title>Producer Flow Control</title>
-
- <section role="h2" xml:id="Java-Broker-Runtime-Disk-Space-Management-Producer-Flow-Control-GeneralInformation">
- <title>General Information</title>
- <para>
- The Apache Qpid Broker for Java supports a flow control mechanism to which can be used to prevent either a single queue
- or a virtualhost exceeding configured limits. These two mechanisms are described
- next.
+<section xmlns="http://docbook.org/ns/docbook" version="5.0"
+ xml:id="Java-Broker-Runtime-Disk-Space-Management-Producer-Flow-Control">
+ <title>Disk quota-based flow control</title>
+ <para>
+ The Apache Qpid Broker for Java supports a flow control mechanism which is triggered when a configured
+ disk quota is exceeded. This is supported by the BDB and Derby virtualhosts.
+ </para>
+ <para>
+ This functionality blocks all producers on reaching the disk overflow limit. When consumers consume the
+ messages, causing disk space usage to falls below the underflow limit, the producers are unblocked and continue
+ working as normal.
+ </para>
+ <para>
+ Two limits can be configured:
+ </para>
+ <para>
+ overfull limit - the maximum space on disk (in bytes).
+ </para>
+ <para>
+ underfull limit - when the space on disk drops below this limit, producers are allowed to resume publishing.
+ </para>
+ <para>
+ The overfull and underful limit can be specified when a new virtualhost is created or an exiting virtualhost is
+ edited. This can be done using the Store Overflow and Store Underfull settings within the virtual host creation
+ and edit dialogue. If editing an existing virtualhost, the virtualhost must be restarted for the new values to
+ take effect.
+ </para>
+ <para>
+ The disk quota functionality is based on "best effort" principle. This means the broker cannot guarantee that
+ the disk space limit will not be exceeded. If several concurrent transactions are started before the limit is
+ reached, which collectively cause the limit to be exceeded, the broker may allow all of them to be committed.
+ </para>
+ <para>
+ The Broker will also impose flow control if the filesystem hosting a virtualhost exceeds a <link
+ linkend="Java-Broker-Management-Managing-Broker-Context-StoreFilesystemMaxUsagePercent">configured
+ percentage.</link>.
+ </para>
+ <note>
+ <para>
+ The <emphasis>Producer Flow Control</emphasis> can be configured on individual queue using
+ <emphasis>Producer Flow Control</emphasis> overflow policy.
+ For more details, please read<xref linkend="Java-Broker-Concepts-Queue-OverflowPolicy"/>.
+ </para>
+ </note>
+ <section role="h4">
+ <title>Broker Log Messages for quota flow control</title>
+ <para>
+ There are two broker log messages that may occur if flow control through disk quota limits is enabled. When
+ the virtual host is blocked due to exceeding of the disk quota limit the following message appears in the
+ broker log
+ <programlisting>
+ [vh(/test)/ms(BDBMessageStore)] MST-1008 : Store overfull, flow control will be enforced
+ </programlisting>
+ When virtual host is unblocked after cleaning the disk space the following message appears in the broker log
+ <programlisting>
+ [vh(/test)/ms(BDBMessageStore)] MST-1009 : Store overfull condition cleared
+ </programlisting>
</para>
</section>
- <section role="h2" xml:id="Java-Broker-Runtime-Disk-Space-Management-Producer-Flow-Control-ServerConfiguration">
- <title>Server Configuration</title>
- <section role="h3">
- <title>Configuring a Queue to use flow control</title>
- <para>
- Flow control is enabled on a producer when it sends a message to a Queue
- which is "overfull". The producer flow control will be rescinded when all
- Queues on which a producer is blocking become "underfull". A Queue is defined
- as overfull when the size (in bytes) of the messages on the queue exceeds the
- <emphasis>capacity</emphasis> of the Queue. A Queue becomes "underfull" when its
- size becomes less than the <emphasis>resume capacity</emphasis>.
- </para>
- <para>
- The capacity and resume capacity can be specified when the queue is created. This
- can be done using the Flow Control Settings within the Queue creation dialogue.
- </para>
- <section role="h4">
- <title>Broker Log Messages</title>
- <para>
- There are four Broker log messages that may occur if flow control through queue capacity limits is enabled.
- Firstly, when a capacity limited queue becomes overfull, a log message similar to the following is produced
- </para>
- <programlisting>
-MESSAGE [vh(/test)/qu(MyQueue)] [vh(/test)/qu(MyQueue)] QUE-1003 : Overfull : Size : 1,200 bytes, Capacity : 1,000
- </programlisting>
- <para>Then for each channel which becomes blocked upon the overful queue a log message similar to the following is produced:</para>
- <programlisting>
-MESSAGE [con:2(guest@anonymous(713889609)/test)/ch:1] [con:2(guest@anonymous(713889609)/test)/ch:1] CHN-1005 : Flow Control Enforced (Queue MyQueue)
- </programlisting>
- <para>When enough messages have been consumed from the queue that it becomes underfull, then the following log is generated: </para>
- <programlisting>
-MESSAGE [vh(/test)/qu(MyQueue)] [vh(/test)/qu(MyQueue)] QUE-1004 : Underfull : Size : 600 bytes, Resume Capacity : 800
- </programlisting>
- <para>And for every channel which becomes unblocked you will see a message similar to: </para>
- <programlisting>
-MESSAGE [con:2(guest@anonymous(713889609)/test)/ch:1] [con:2(guest@anonymous(713889609)/test)/ch:1] CHN-1006 : Flow Control Removed
- </programlisting>
- <para>Obviously the details of connection, virtual host, queue, size, capacity, etc would depend on the configuration in use.</para>
-
-
- </section><!-- Broker Log Messages -->
- </section><!-- Configuring a Queue to use flow control -->
-
- <section role="h3">
- <title>Disk quota-based flow control</title>
- <para>
- Flow control can also be triggered when a configured disk quota is exceeded. This is supported by the BDB and
- Derby virtualhosts.
- </para>
- <para>
- This functionality blocks all producers on reaching the disk overflow limit. When consumers
- consume the messages, causing disk space usage to falls below the underflow limit, the
- producers are unblocked and continue working as normal.
- </para>
- <para>
- Two limits can be configured:
- </para>
- <para>
- overfull limit - the maximum space on disk (in bytes).
- </para>
- <para>
- underfull limit - when the space on disk drops below this limit, producers are allowed to resume publishing.
- </para>
-
- <para>
- The overfull and underful limit can be specified when a new virtualhost is created or an exiting
- virtualhost is edited. This can be done using the Store Overflow and Store Underfull settings
- within the virtual host creation and edit dialogue. If editing an existing virtualhost, the virtualhost
- must be restarted for the new values to take effect.
- </para>
-
- <para>
- The disk quota functionality is based on "best effort" principle. This means the broker
- cannot guarantee that the disk space limit will not be exceeded. If several concurrent
- transactions are started before the limit is reached, which collectively cause the limit
- to be exceeded, the broker may allow all of them to be committed.
- </para>
-
- <para>
- The Broker will also impose flow control if the filesystem hosting a virtualhost
- exceeds a <link linkend="Java-Broker-Management-Managing-Broker-Context-StoreFilesystemMaxUsagePercent">
- configured percentage.</link>.
- </para>
-
- <section role="h4">
- <title>Broker Log Messages for quota flow control</title>
- <para>
- There are two broker log messages that may occur if flow control through disk quota limits is enabled.
- When the virtual host is blocked due to exceeding of the disk quota limit the following message
- appears in the broker log
- <programlisting>
-[vh(/test)/ms(BDBMessageStore)] MST-1008 : Store overfull, flow control will be enforced
- </programlisting>
- When virtual host is unblocked after cleaning the disk space the following message appears in the broker log
- <programlisting>
-[vh(/test)/ms(BDBMessageStore)] MST-1009 : Store overfull condition cleared
- </programlisting>
- </para>
- </section>
- </section><!-- Disk quota-based flow control -->
- </section><!-- Server configuration -->
-
-
- <section role="h2" xml:id="Java-Broker-Runtime-Disk-Space-Management-Producer-Flow-Control-ClientImpact">
- <title>Client impact and configuration</title>
- <para>
- If a producer sends to a queue which is overfull, the broker will respond by
- instructing the client not to send any more messages. The impact of this is
- that any future attempts to send will block until the broker rescinds the flow control order.
- </para>
- <para>
- While blocking the client will periodically log the fact that it is blocked waiting on flow control.
- </para>
- <programlisting>
-WARN Message send delayed by 5s due to broker enforced flow control
-WARN Message send delayed by 10s due to broker enforced flow control
- </programlisting>
- <para>
- After a set period the send will timeout and throw a JMSException to the calling code.
- </para>
- <para>
- If such a JMSException is thrown, the message will not be sent to the broker,
- however the underlying Session may still be active - in particular if the
- Session is transactional then the current transaction will not be automatically
- rolled back. Users may choose to either attempt to resend the message, or to
- roll back any transactional work and close the Session.
- </para>
- <para>
- Both the timeout delay and the periodicity of the warning messages can be set
- using Java system properties.
- </para>
- <para>
- The amount of time (in milliseconds) to wait before timing out
- is controlled by the property qpid.flow_control_wait_failure.
- </para>
- <para>
- The frequency at which the log message informing that the producer is flow
- controlled is sent is controlled by the system property qpid.flow_control_wait_notify_period.
- </para>
- <para>
- Adding the following to the command line to start the client would result in a timeout of one minute,
- with warning messages every ten seconds:
- </para>
- <programlisting>
--Dqpid.flow_control_wait_failure=60000
--Dqpid.flow_control_wait_notify_period=10000
- </programlisting>
- </section> <!-- Client impact and configuration -->
</section>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@qpid.apache.org
For additional commands, e-mail: commits-help@qpid.apache.org