You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by "Vanshul.Chawla" <Va...@target.com> on 2015/06/08 11:14:16 UTC
JMS MQ performance improvement
Hello All,
We are using the mq jms settings like this:
<!-- Queue Manager Connection factory(s) -->
<bean id="wmqConnectionFactory" class="com.ibm.mq.jms.MQQueueConnectionFactory">
<property name="CCDTURL">
<bean class="com.tgt.tcom.il.financial.MQConnectionFactoryUtil"
factory-method="convert">
<argument value="${financial.WMQ.CCDTfile_Stage}" />
</bean>
</property>
<property name="queueManager" value="${financial.WMQ.QMName_Stage3}" />
</bean>
<!-- Cache Connection Factory -->
<bean id="cachingWmqConnectionFactory"
class="org.springframework.jms.connection.CachingConnectionFactory"
destroy-method="destroy">
<property name="targetConnectionFactory" ref="wmqConnectionFactory" />
<property name="cacheConsumers" value="true" />
<property name="reconnectOnException" value="true" />
<property name="sessionCacheSize" value="10" />
</bean>
<!--Spring Transaction Manager -->
<bean id="wmqTransactionManager"
class="org.springframework.jms.connection.JmsTransactionManager">
<property name="connectionFactory" ref="wmqConnectionFactory" />
</bean>
<!--JMS components used for Endpoints -->
<bean id="wmq-qm" class="org.apache.camel.component.jms.JmsComponent">
<property name="connectionFactory" ref="cachingWmqConnectionFactory" />
<property name="transactionManager" ref="wmqTransactionManager" />
<property name="transacted" value="false" />
<property name="cacheLevelName" value="CACHE_AUTO" />
<property name="concurrentConsumers" value="40" />
<property name="maxConcurrentConsumers" value="60" />
</bean>
My concurrentConsumers is 40 and max is 60. But issue is it never reaches even 40 and messages are consumed slowly. Ie my queue will have 5K messages but threads instantiated will reach max 25-33 range.
Any option to force min 40 threads at startup itself?
Thanks and Regards,
Vanshul Chawla
RE: JMS MQ performance improvement
Posted by "Vanshul.Chawla" <Va...@target.com>.
To increase performance, use caching of connection. We used the same and are able to process a huge volume.
<!-- Queue Manager Connection factory(s) -->
<bean id="wmqConnectionFactory1-Prod1-FIT" class="com.ibm.mq.jms.MQQueueConnectionFactory">
<property name="CCDTURL">
<bean class="com.tgt.tcom.il.financial.MQConnectionFactoryUtil"
factory-method="convert">
<argument value="${financial.WMQ.CCDTfile}" />
</bean>
</property>
<property name="queueManager" value="${financial.WMQ.QMGrpName1}" />
</bean>
<!-- Cache Connection Factory -->
<bean id="cachingWmqConnectionFactory1-Prod1-FIT"
class="org.springframework.jms.connection.CachingConnectionFactory"
destroy-method="destroy">
<property name="targetConnectionFactory" ref="wmqConnectionFactory1-Prod1-FIT" />
<property name="cacheConsumers" value="true" />
<property name="reconnectOnException" value="true" />
<property name="sessionCacheSize" value="10" />
</bean>
<!--Spring Transaction Manager -->
<bean id="wmqTransactionManager1-Prod1-FIT"
class="org.springframework.jms.connection.JmsTransactionManager">
<property name="connectionFactory" ref="wmqConnectionFactory1-Prod1-FIT" />
</bean>
<!--JMS components used for Endpoints -->
<bean id="wmq-qm1-Prod1-FIT" class="org.apache.camel.component.jms.JmsComponent">
<property name="connectionFactory" ref="cachingWmqConnectionFactory1-Prod1-FIT" />
<property name="transactionManager" ref="wmqTransactionManager1-Prod1-FIT" />
<property name="transacted" value="false" />
<property name="cacheLevelName" value="CACHE_AUTO" />
<property name="concurrentConsumers" value="${financial.WMQ.DWSub1.Consumers}" />
<property name="maxConcurrentConsumers" value="${financial.WMQ.DWSub1.MaxConsumers}" />
<property name="maxMessagesPerTask" value="${financial.WMQ.DWSub1.maxMessagesPerTask}" />
</bean>
Thanks and Regards,
Vanshul Chawla
-----Original Message-----
From: mytechnology [mailto:mytech1235@gmail.com]
Sent: Tuesday, July 28, 2015 6:05 PM
To: users@camel.apache.org
Subject: RE: JMS MQ performance improvement
Why is it so hard (so far impossible) to find a way of configuring connection pool with IBM MQ. All the documentation is with websphere MDBs or ActiveMQ and Camel.
Has any body used connection pool with IBM MQ and Camel?
--
View this message in context: http://camel.465427.n5.nabble.com/JMS-MQ-performance-improvement-tp5767963p5770010.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Netty4 HTTP client body type
Posted by Alex Soto <al...@envieta.com>.
Hello,
I am using Netty producer to implement an HTTP client:
Exchange response = testProducer.request(URL, new Processor() {
@Override
public void process(Exchange exchange) throws Exception {
exchange.getIn().setBody(body);
}
});
My problem is that the response is not available as InputStream or byte[]:
response.getOut().getBody(InputStream.class)
response.getOut().getBody(byte[].class);
Both return null. Looks like it comes back as a CompositeByteBuf, but isn’t Camel supposed to convert this automatically?
Best regards,
Alex soto
RE: JMS MQ performance improvement
Posted by David Karlsen <da...@gmail.com>.
IBM used to support pooling but have removed it in recent 7.x versions
28. jul. 2015 3.17 p.m. skrev "mytechnology" <my...@gmail.com>:
>
> Why is it so hard (so far impossible) to find a way of configuring
connection
> pool with IBM MQ. All the documentation is with websphere MDBs or ActiveMQ
> and Camel.
>
> Has any body used connection pool with IBM MQ and Camel?
>
>
>
> --
> View this message in context:
http://camel.465427.n5.nabble.com/JMS-MQ-performance-improvement-tp5767963p5770010.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
RE: JMS MQ performance improvement
Posted by mytechnology <my...@gmail.com>.
Why is it so hard (so far impossible) to find a way of configuring connection
pool with IBM MQ. All the documentation is with websphere MDBs or ActiveMQ
and Camel.
Has any body used connection pool with IBM MQ and Camel?
--
View this message in context: http://camel.465427.n5.nabble.com/JMS-MQ-performance-improvement-tp5767963p5770010.html
Sent from the Camel - Users mailing list archive at Nabble.com.
RE: JMS MQ performance improvement
Posted by yogu13 <yo...@gmail.com>.
Hello,
you could try followign things
1. As
<property name="transactionManager" ref="wmqTransactionManager" />
<property name="transacted" value="false" />
as transacted is false you can try removing transactionManager which is
being set
2. for
<property name="cacheLevelName" value="CACHE_AUTO" />
try setting the cacheLevelName to CACHE_CONSUMER or CACHE_SESSION for higher
throughput
3. Enable asyncConsumer if async processing of exchanges is acceptible
Regards,
-Yogesh
--
View this message in context: http://camel.465427.n5.nabble.com/JMS-MQ-performance-improvement-tp5767963p5768016.html
Sent from the Camel - Users mailing list archive at Nabble.com.
RE: JMS MQ performance improvement
Posted by contactreji <co...@gmail.com>.
Cool..
Can you try using pooled jms connections?
It may be possible that due to initialization of new connection each time
for each message, internally thread size may be limited to have optimized
thread context switching.
Following few code snipped might help you. We have used this in our project
and observed increase in throughput. Not sure if IBM product supports
connection pools. Check the IBM driver classes.
We were using Jboss ActiveMQ as broker.
<bean id="jmsConnectionFactory"
class="org.apache.activemq.ActiveMQConnectionFactory">
<property name="brokerURL" value="${brokerURL}" />
<property name="userName" value="${brokerUserName}" />
<property name="password" value="${brokerPassword}" />
</bean>
<bean id="pooledConnectionFactory"
class="org.apache.activemq.pool.PooledConnectionFactory"
init-method="start" destroy-method="stop">
<property name="maxConnections" value="8" />
<property name="connectionFactory" ref="jmsConnectionFactory" />
<property name="idleTimeout" value="0" />
</bean>
<bean id="jmsConfig"
class="org.apache.camel.component.jms.JmsConfiguration">
<property name="connectionFactory" ref="pooledConnectionFactory" />
<property name="concurrentConsumers" value="10" />
</bean>
<bean id="AMQbrokerConnector"
class="org.apache.activemq.camel.component.ActiveMQComponent">
<property name="configuration" ref="jmsConfig" />
</bean>
-----
Reji Mathews
Sr. Developer - Middleware Integration / SOA ( Open Source - Apache Camel & Jboss Fuse ESB | Mule ESB )
LinkedIn - http://in.linkedin.com/pub/reji-mathews/31/9a2/40a
Twitter - reji_mathews
--
View this message in context: http://camel.465427.n5.nabble.com/JMS-MQ-performance-improvement-tp5767963p5767973.html
Sent from the Camel - Users mailing list archive at Nabble.com.
RE: JMS MQ performance improvement
Posted by "Vanshul.Chawla" <Va...@target.com>.
When I increase to 100, it goes up to 80 max and then again comes down slowly to same range of 25-33.
Its non transacted only.
If u see now:
599 0.12% 500000 9 0 0 E E - PERF.CAMEL.TCOM.FINANCIALFEED.OMS.PUB.1
702 0.14% 500000 8 0 0 E E - PERF.CAMEL.TCOM.FINANCIALFEED.OMS.PUB.2
628 0.13% 500000 7 0 0 E E - PERF.CAMEL.TCOM.FINANCIALFEED.OMS.PUB.3
Mon Jun 8 04:32:13 CDT 2015
Only 9,8,7 threads and messages on queue are 599,702,628
I am pushing a continuous load. Pasting the part of blueprint(some java classes are used in real code. This code is edited for perf test run)
-------------------------
<?xml version="1.0" encoding="UTF-8"?>
<blueprint xmlns="http://www.osgi.org/xmlns/blueprint/v1.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:camel="http://camel.apache.org/schema/blueprint"
xmlns:cm="http://aries.apache.org/blueprint/xmlns/blueprint-cm/v1.1.0"
xsi:schemaLocation="
http://www.osgi.org/xmlns/blueprint/v1.0.0 http://www.osgi.org/xmlns/blueprint/v1.0.0/blueprint.xsd
http://aries.apache.org/blueprint/xmlns/blueprint-cm/v1.1.0 http://aries.apache.org/schemas/blueprint-cm/blueprint-cm-1.1.0.xsd">
<cm:property-placeholder id="fuse-oms-financial-feed-bundle-config"
persistent-id="com.tgt.rjf.oms.financial.feed" update-strategy="reload">
<cm:default-properties />
</cm:property-placeholder>
<bean id="myAggregationStrategy" class="com.tgt.tcom.il.aggregator.MyAggregationStrategy" />
<bean id="removeSpacesFromFile" class="com.tgt.tcom.il.financial.RemoveSpacesFromFile" />
<bean id="removeSpecialChar" class="com.tgt.tcom.il.financial.RemoveSpecialCharFromFile" />
<bean id="setHeaders" class="com.tgt.tcom.il.financial.SetHeadersForRoutes"></bean>
<bean id="executor" class="java.util.concurrent.Executors"
factory-method="newFixedThreadPool">
<argument index="0" value="16" />
</bean>
<bean id="exceptionmessage" class="com.tgt.tcom.il.error.ErrorHandler" />
<bean id="userException" class="java.lang.IllegalArgumentException">
<argument index="0" value="UserException" />
</bean>
<!-- Queue Manager Connection factory(s) -->
<bean id="wmqConnectionFactory" class="com.ibm.mq.jms.MQQueueConnectionFactory">
<property name="CCDTURL">
<bean class="com.tgt.tcom.il.financial.MQConnectionFactoryUtil"
factory-method="convert">
<argument value="${financial.WMQ.CCDTfile_Stage}" />
</bean>
</property>
<property name="queueManager" value="${financial.WMQ.QMName_Stage3}" />
</bean>
<!-- Cache Connection Factory -->
<bean id="cachingWmqConnectionFactory"
class="org.springframework.jms.connection.CachingConnectionFactory"
destroy-method="destroy">
<property name="targetConnectionFactory" ref="wmqConnectionFactory" />
<property name="cacheConsumers" value="true" />
<property name="reconnectOnException" value="true" />
<property name="sessionCacheSize" value="10" />
</bean>
<!--Spring Transaction Manager -->
<bean id="wmqTransactionManager"
class="org.springframework.jms.connection.JmsTransactionManager">
<property name="connectionFactory" ref="wmqConnectionFactory" />
</bean>
<!--JMS components used for Endpoints -->
<bean id="wmq-qm" class="org.apache.camel.component.jms.JmsComponent">
<property name="connectionFactory" ref="cachingWmqConnectionFactory" />
<property name="transactionManager" ref="wmqTransactionManager" />
<property name="transacted" value="false" />
<property name="cacheLevelName" value="CACHE_AUTO" />
<property name="concurrentConsumers" value="40" />
<property name="maxConcurrentConsumers" value="60" />
</bean>
<camelContext id="FinancialFeed-Pub" errorHandlerRef="EH"
xmlns="http://camel.apache.org/schema/blueprint">
<propertyPlaceholder id="properties"
location="blueprint:fuse-oms-financial-feed-bundle-config" />
<errorHandler type="DeadLetterChannel"
deadLetterUri="wmq-qm:queue:{{financial.Perf.WMQ.FinFeedErrorQueue}}"
id="EH">
</errorHandler>
<onException>
<description>
What to be done on exception
Send to Error queue
</description>
<exception>java.lang.Exception</exception>
<log
message="Message not moved to Queue successfully due to ${exception.message}"
loggingLevel="ERROR">
<description>
No redelivery attemmpts. For cases like HTTP 500,
socket, just increase
the value of maximumRedeliveries and delay
Logging error message
</description>
</log>
<process ref="exceptionmessage">
<description>
Original payload message in one queue like our ORIG
queues
Generate exception message in java code
</description>
</process>
<process ref="removeSpecialChar">
<description>
Original payload message in one queue like our ORIG
queues
Generate exception message in java code
</description>
</process>
<to uri="wmq-qm:queue:{{financial.Perf.WMQ.FinFeedErrorQueue}}" />
</onException>
<route customId="true" handleFault="true" id="FinFeedRouterMainRoute1">
<from uri="wmq-qm:queue:{{financial.Perf.WMQ.OMSPub1Queue}}" />
<process ref="setHeaders" />
<log loggingLevel="INFO" logName="FinFeedPubAuditLog"
message="Timestamp=${date:now:yyyy-MM-dd hh:mm:ss.SSS} : MessageType=${in.header.MessageType} : Action=${in.header.Action} : MessageSubType=${in.header.MessageSubType} : OrderNo=${in.header.OrderNo} : ReceiptNo=${in.header.ReceiptNo} : ShipmentNo=${in.header.ShipmentNo} : InvoiceNo=${in.header.InvoiceNo} " />
<to uri="wmq-qm:queue:{{financial.Perf.WMQ.DWSub1Queue}}" />
</route>
<route handleFault="true" id="FinFeedRouterMainRoute2">
<from uri="wmq-qm:queue:{{financial.Perf.WMQ.OMSPub2Queue}}" />
<process ref="setHeaders" />
<log loggingLevel="INFO" logName="FinFeedPubAuditLog"
message="Timestamp=${date:now:yyyy-MM-dd hh:mm:ss.SSS} : MessageType=${in.header.MessageType} : Action=${in.header.Action} : MessageSubType=${in.header.MessageSubType} : OrderNo=${in.header.OrderNo} : ReceiptNo=${in.header.ReceiptNo} : ShipmentNo=${in.header.ShipmentNo} : InvoiceNo=${in.header.InvoiceNo} " />
<to uri="wmq-qm:queue:{{financial.Perf.WMQ.DWSub2Queue}}" />
</route>
<route handleFault="true" id="FinFeedRouterMainRoute3">
<from uri="wmq-qm:queue:{{financial.Perf.WMQ.OMSPub3Queue}}" />
<process ref="setHeaders" />
<log loggingLevel="INFO" logName="FinFeedPubAuditLog"
message="Timestamp=${date:now:yyyy-MM-dd hh:mm:ss.SSS} : MessageType=${in.header.MessageType} : Action=${in.header.Action} : MessageSubType=${in.header.MessageSubType} : OrderNo=${in.header.OrderNo} : ReceiptNo=${in.header.ReceiptNo} : ShipmentNo=${in.header.ShipmentNo} : InvoiceNo=${in.header.InvoiceNo} " />
<to uri="wmq-qm:queue:{{financial.Perf.WMQ.DWSub3Queue}}" />
</route>
</camelContext>
</blueprint>
-------------------------------------------------
Thanks and Regards,
Vanshul Chawla
-----Original Message-----
From: contactreji [mailto:contactreji@gmail.com]
Sent: Monday, June 08, 2015 2:55 PM
To: users@camel.apache.org
Subject: Re: JMS MQ performance improvement
Interesting!
1) Can you observe how many threads initiate when your camel routes are not transacted?
2) And what if you further increase the value for maxConcurrentConsumers.
Say about 100?
Is the count still around 25-33 range itself? can you post your camel routes here.
Cheers
Reji
-----
Reji Mathews
Sr. Developer - Middleware Integration / SOA ( Open Source - Apache Camel & Jboss Fuse ESB | Mule ESB ) LinkedIn - http://in.linkedin.com/pub/reji-mathews/31/9a2/40a
Twitter - reji_mathews
--
View this message in context: http://camel.465427.n5.nabble.com/JMS-MQ-performance-improvement-tp5767963p5767965.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: JMS MQ performance improvement
Posted by contactreji <co...@gmail.com>.
Interesting!
1) Can you observe how many threads initiate when your camel routes are not
transacted?
2) And what if you further increase the value for maxConcurrentConsumers.
Say about 100?
Is the count still around 25-33 range itself? can you post your camel routes
here.
Cheers
Reji
-----
Reji Mathews
Sr. Developer - Middleware Integration / SOA ( Open Source - Apache Camel & Jboss Fuse ESB | Mule ESB )
LinkedIn - http://in.linkedin.com/pub/reji-mathews/31/9a2/40a
Twitter - reji_mathews
--
View this message in context: http://camel.465427.n5.nabble.com/JMS-MQ-performance-improvement-tp5767963p5767965.html
Sent from the Camel - Users mailing list archive at Nabble.com.