You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@cxf.apache.org by "Paul Hadrosek (JIRA)" <ji...@apache.org> on 2009/07/16 13:28:14 UTC
[jira] Created: (CXF-2343) Improve Message Performance Under High
Volume with Low Latency Consumers
Improve Message Performance Under High Volume with Low Latency Consumers
------------------------------------------------------------------------
Key: CXF-2343
URL: https://issues.apache.org/jira/browse/CXF-2343
Project: CXF
Issue Type: Improvement
Components: Transports
Reporter: Paul Hadrosek
IT organizations are on constant pressures in improving enterprise messaging systems to deliver and process high volume message traffic. For example financial IT organizations are continually automate, integrate and optimized the transaction life cycle. The underlying messaging system must be able to support extremely low latency and very high message throughput with effective congestion control.
Message performance testing was performed to evaluate the CXF bogus message selector implementation as the best approach in improving message throttling in a long consumer service process with a very high JMS message traffic volume.
The intent of the bogus message selector implementation in CXF is to alleviate message throttling by delegating incoming messages to under performed consumer services. Testing was performed on the CXF bogus message selector implementation with a commercial JMS message broker. The test scenario simulated an ingest driver that produced high volume message traffic with a long running CXF consumer service which forced a backlog of messages in the request queue. The test results illustrated a dramatic degradation in the commercial JMS message broker processing performance from 12,000 messages per second to 100 messages per second (99.17% degradation of performance). Also it was observed the JMS receiveTimeout parameter was set to the default value of 1 which contributed to the degradation of performance. CXF was modified to expose the JMS receiveTimeout parameter as a CXF feature. The test scenario was rerun with a receiveTimeout set to 30 seconds which improved message processing from 100 messages per second to 4,918 messages per second.
To further improve ingest message processing performance; the bogus message selector implementation was replaced with a stop and start Spring JMS Listener Container (DefaultMessageListenerContainer) implementation via the CXF JMSContinuation class. The JMSContintuation class starts and stops the Spring Container based on the maximum suspended continuations parameter. The moment a stop is issued from the JMSContinuation class to the Container, no new messages are allowed to be accepted by the Container until the Container completes processing the suspended messages. To avoid message loss while the Container is in the stop state, CXF features must support the JMS acceptMessagesWhileStopping parameter. This parameter must be set to true to notify the Container not to reject messages while in the stop state. The test scenario was rerun using the CXF start/stop Container implementation with a commercial JMS message broker. The test results demonstrated that message processing performance improved from 4,918 messages per second to 8,333 messages per second.
Based on the test results, I concluded that the start/stop Container implementation is the recommended solution for improving message performance in high JMS message traffic with low latency consumers. The start/stop Container implementation improved message processing performance over the bogus message selector implementation by 28%.
See attachment for patch.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (CXF-2343) Improve Message Performance Under High
Volume with Low Latency Consumers
Posted by "Daniel Kulp (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CXF-2343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Kulp resolved CXF-2343.
------------------------------
Resolution: Fixed
Fix Version/s: 2.2.3
2.1.6
2.0.12
Assignee: Daniel Kulp
> Improve Message Performance Under High Volume with Low Latency Consumers
> ------------------------------------------------------------------------
>
> Key: CXF-2343
> URL: https://issues.apache.org/jira/browse/CXF-2343
> Project: CXF
> Issue Type: Improvement
> Components: Transports
> Reporter: Paul Hadrosek
> Assignee: Daniel Kulp
> Fix For: 2.0.12, 2.1.6, 2.2.3
>
> Attachments: cxf.patch
>
>
> IT organizations are on constant pressures in improving enterprise messaging systems to deliver and process high volume message traffic. For example financial IT organizations are continually automate, integrate and optimized the transaction life cycle. The underlying messaging system must be able to support extremely low latency and very high message throughput with effective congestion control.
> Message performance testing was performed to evaluate the CXF bogus message selector implementation as the best approach in improving message throttling in a long consumer service process with a very high JMS message traffic volume.
> The intent of the bogus message selector implementation in CXF is to alleviate message throttling by delegating incoming messages to under performed consumer services. Testing was performed on the CXF bogus message selector implementation with a commercial JMS message broker. The test scenario simulated an ingest driver that produced high volume message traffic with a long running CXF consumer service which forced a backlog of messages in the request queue. The test results illustrated a dramatic degradation in the commercial JMS message broker processing performance from 12,000 messages per second to 100 messages per second (99.17% degradation of performance). Also it was observed the JMS receiveTimeout parameter was set to the default value of 1 which contributed to the degradation of performance. CXF was modified to expose the JMS receiveTimeout parameter as a CXF feature. The test scenario was rerun with a receiveTimeout set to 30 seconds which improved message processing from 100 messages per second to 4,918 messages per second.
> To further improve ingest message processing performance; the bogus message selector implementation was replaced with a stop and start Spring JMS Listener Container (DefaultMessageListenerContainer) implementation via the CXF JMSContinuation class. The JMSContintuation class starts and stops the Spring Container based on the maximum suspended continuations parameter. The moment a stop is issued from the JMSContinuation class to the Container, no new messages are allowed to be accepted by the Container until the Container completes processing the suspended messages. To avoid message loss while the Container is in the stop state, CXF features must support the JMS acceptMessagesWhileStopping parameter. This parameter must be set to true to notify the Container not to reject messages while in the stop state. The test scenario was rerun using the CXF start/stop Container implementation with a commercial JMS message broker. The test results demonstrated that message processing performance improved from 4,918 messages per second to 8,333 messages per second.
> Based on the test results, I concluded that the start/stop Container implementation is the recommended solution for improving message performance in high JMS message traffic with low latency consumers. The start/stop Container implementation improved message processing performance over the bogus message selector implementation by 28%.
> See attachment for patch.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CXF-2343) Improve Message Performance Under High
Volume with Low Latency Consumers
Posted by "Daniel Kulp (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CXF-2343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12732106#action_12732106 ]
Daniel Kulp commented on CXF-2343:
----------------------------------
I've gone ahead and applied this patch. I have one question about another potential optimization that you might look into. Looking at the code, it looks like when the threshold is reached, it's stops/disconnects (which is good), but as soon as a single message is then processed, it would reconnect, get a message, which then exceeds the threshold, disconnect, etc.... Potentially, worst case, connecting/disconnecting per-message. Would it makes sense to add another tunable param about when the reconnect would happen. Maybe as a percentage of the MaxSuspendedContinuations? Example, if MaxSuspendedContinuations is 500 and a reconnect set for 75%, it would wait for the suspended count to get to 375, then connect, which would allow for another 125 messages to start processing before a disconnect again.
Does that make sense?
Anyway, it's just a thought. I'm not sure how expensive the connect/disconnect is for the client or for the broker. I assume there is some protocol version negotiation and such on connect that maybe, by not connecting so often, this suggestion might help with throughput even more.
> Improve Message Performance Under High Volume with Low Latency Consumers
> ------------------------------------------------------------------------
>
> Key: CXF-2343
> URL: https://issues.apache.org/jira/browse/CXF-2343
> Project: CXF
> Issue Type: Improvement
> Components: Transports
> Reporter: Paul Hadrosek
> Attachments: cxf.patch
>
>
> IT organizations are on constant pressures in improving enterprise messaging systems to deliver and process high volume message traffic. For example financial IT organizations are continually automate, integrate and optimized the transaction life cycle. The underlying messaging system must be able to support extremely low latency and very high message throughput with effective congestion control.
> Message performance testing was performed to evaluate the CXF bogus message selector implementation as the best approach in improving message throttling in a long consumer service process with a very high JMS message traffic volume.
> The intent of the bogus message selector implementation in CXF is to alleviate message throttling by delegating incoming messages to under performed consumer services. Testing was performed on the CXF bogus message selector implementation with a commercial JMS message broker. The test scenario simulated an ingest driver that produced high volume message traffic with a long running CXF consumer service which forced a backlog of messages in the request queue. The test results illustrated a dramatic degradation in the commercial JMS message broker processing performance from 12,000 messages per second to 100 messages per second (99.17% degradation of performance). Also it was observed the JMS receiveTimeout parameter was set to the default value of 1 which contributed to the degradation of performance. CXF was modified to expose the JMS receiveTimeout parameter as a CXF feature. The test scenario was rerun with a receiveTimeout set to 30 seconds which improved message processing from 100 messages per second to 4,918 messages per second.
> To further improve ingest message processing performance; the bogus message selector implementation was replaced with a stop and start Spring JMS Listener Container (DefaultMessageListenerContainer) implementation via the CXF JMSContinuation class. The JMSContintuation class starts and stops the Spring Container based on the maximum suspended continuations parameter. The moment a stop is issued from the JMSContinuation class to the Container, no new messages are allowed to be accepted by the Container until the Container completes processing the suspended messages. To avoid message loss while the Container is in the stop state, CXF features must support the JMS acceptMessagesWhileStopping parameter. This parameter must be set to true to notify the Container not to reject messages while in the stop state. The test scenario was rerun using the CXF start/stop Container implementation with a commercial JMS message broker. The test results demonstrated that message processing performance improved from 4,918 messages per second to 8,333 messages per second.
> Based on the test results, I concluded that the start/stop Container implementation is the recommended solution for improving message performance in high JMS message traffic with low latency consumers. The start/stop Container implementation improved message processing performance over the bogus message selector implementation by 28%.
> See attachment for patch.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CXF-2343) Improve Message Performance Under High
Volume with Low Latency Consumers
Posted by "Paul Hadrosek (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CXF-2343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Paul Hadrosek updated CXF-2343:
-------------------------------
Attachment: cxf.patch
Code to replace bogus message selector.
> Improve Message Performance Under High Volume with Low Latency Consumers
> ------------------------------------------------------------------------
>
> Key: CXF-2343
> URL: https://issues.apache.org/jira/browse/CXF-2343
> Project: CXF
> Issue Type: Improvement
> Components: Transports
> Reporter: Paul Hadrosek
> Attachments: cxf.patch
>
>
> IT organizations are on constant pressures in improving enterprise messaging systems to deliver and process high volume message traffic. For example financial IT organizations are continually automate, integrate and optimized the transaction life cycle. The underlying messaging system must be able to support extremely low latency and very high message throughput with effective congestion control.
> Message performance testing was performed to evaluate the CXF bogus message selector implementation as the best approach in improving message throttling in a long consumer service process with a very high JMS message traffic volume.
> The intent of the bogus message selector implementation in CXF is to alleviate message throttling by delegating incoming messages to under performed consumer services. Testing was performed on the CXF bogus message selector implementation with a commercial JMS message broker. The test scenario simulated an ingest driver that produced high volume message traffic with a long running CXF consumer service which forced a backlog of messages in the request queue. The test results illustrated a dramatic degradation in the commercial JMS message broker processing performance from 12,000 messages per second to 100 messages per second (99.17% degradation of performance). Also it was observed the JMS receiveTimeout parameter was set to the default value of 1 which contributed to the degradation of performance. CXF was modified to expose the JMS receiveTimeout parameter as a CXF feature. The test scenario was rerun with a receiveTimeout set to 30 seconds which improved message processing from 100 messages per second to 4,918 messages per second.
> To further improve ingest message processing performance; the bogus message selector implementation was replaced with a stop and start Spring JMS Listener Container (DefaultMessageListenerContainer) implementation via the CXF JMSContinuation class. The JMSContintuation class starts and stops the Spring Container based on the maximum suspended continuations parameter. The moment a stop is issued from the JMSContinuation class to the Container, no new messages are allowed to be accepted by the Container until the Container completes processing the suspended messages. To avoid message loss while the Container is in the stop state, CXF features must support the JMS acceptMessagesWhileStopping parameter. This parameter must be set to true to notify the Container not to reject messages while in the stop state. The test scenario was rerun using the CXF start/stop Container implementation with a commercial JMS message broker. The test results demonstrated that message processing performance improved from 4,918 messages per second to 8,333 messages per second.
> Based on the test results, I concluded that the start/stop Container implementation is the recommended solution for improving message performance in high JMS message traffic with low latency consumers. The start/stop Container implementation improved message processing performance over the bogus message selector implementation by 28%.
> See attachment for patch.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.