You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by helander <le...@gmail.com> on 2012/08/07 19:04:12 UTC
Connection pooling with camel-amqp
I have tried to get a working solution for connection pooling/caching with
camel-amqp.
I found some examples that used the Spring CachingConnectionFactory, but
when using that it get an NPE.
The first time I do a write to a queue, it works, but subsequent writes
causes an NPE.
It looks like the NPE is associated with finding and existing/cached
destination.
Any kind of help on this would be most valuable.
Thanks
Lars
--
View this message in context: http://camel.465427.n5.nabble.com/Connection-pooling-with-camel-amqp-tp5716936.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Connection pooling with camel-amqp
Posted by helander <le...@gmail.com>.
Hi Claus,
I will take a stab at updating the page. As you suggest I will file an ICLA.
Thanks for providing the details about editing pages.
/Lasse
--
View this message in context: http://camel.465427.n5.nabble.com/Connection-pooling-with-camel-amqp-tp5716936p5717837.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Connection pooling with camel-amqp
Posted by Claus Ibsen <cl...@gmail.com>.
Hi Lasse
Thanks for sharing this.
It would be lovely if the amqp doc page could be improved
http://camel.apache.org/amqp
Currently its lacking details.
So if anyone wanna help with this please contribute.
To be able to edit the docs yourself please read this
http://camel.apache.org/how-do-i-edit-the-website.html
If you dont have an ICLA on file at ASF then you can still contribute
doc updates by using the JIRA ticket and attach a file with [x] in
grant license to ASF. Although we encourage ppl to file an ICLA so you
can update the docs yourself. And also its one step closer to become a
committer at any Apache project, as an ICLA would be required.
On Wed, Aug 22, 2012 at 9:46 AM, helander <le...@gmail.com> wrote:
> I discovered that the most interestinf info was lost in my previous post (was
> using the raw tag in Nabble, and that seemed to be filtered by e-mail
> systems). So here it comes again:
>
>
> I have come up with a workaround to the NPE problem with Qpid.
>
> The root of the problem is that Qpid is not able to correctly compare two
> destinations that has not been fully qualified in their definitions. In
> other parts of the Qpid API, assumptions are made on how to interpret
> destinations that has not been fully qualified, but this is not reflected by
> the equals-method of AMQDestination.
>
> The "workaround" is to make sure that you provide fully qualified
> destinations, and one way I have found to do that in conjunction with Camel
> uri specifications are according to the following example:
>
> <to uri="amqpbroker:queue:BURL:direct://amq.direct//myqueue"
>
> where the name of the queue is <b>myqueue*.
>
> If you are using topics instead of queues, the corresponding syntax should
> be
>
>
> <to uri="amqpbroker:queue:BURL:topic://amq.topic//mytopic"
>
>
> I am not using topics at the moment, so I can not vouch for that the syntax
> is correct in that case.
>
>
> Sorry for any confusion created
>
> Lars
>
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Connection-pooling-with-camel-amqp-tp5716936p5717827.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
--
Claus Ibsen
-----------------
FuseSource
Email: cibsen@fusesource.com
Web: http://fusesource.com
Twitter: davsclaus, fusenews
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
RE: Connection pooling with camel-amqp
Posted by helander <le...@gmail.com>.
I discovered that the most interestinf info was lost in my previous post (was
using the raw tag in Nabble, and that seemed to be filtered by e-mail
systems). So here it comes again:
I have come up with a workaround to the NPE problem with Qpid.
The root of the problem is that Qpid is not able to correctly compare two
destinations that has not been fully qualified in their definitions. In
other parts of the Qpid API, assumptions are made on how to interpret
destinations that has not been fully qualified, but this is not reflected by
the equals-method of AMQDestination.
The "workaround" is to make sure that you provide fully qualified
destinations, and one way I have found to do that in conjunction with Camel
uri specifications are according to the following example:
<to uri="amqpbroker:queue:BURL:direct://amq.direct//myqueue"
where the name of the queue is <b>myqueue*.
If you are using topics instead of queues, the corresponding syntax should
be
<to uri="amqpbroker:queue:BURL:topic://amq.topic//mytopic"
I am not using topics at the moment, so I can not vouch for that the syntax
is correct in that case.
Sorry for any confusion created
Lars
--
View this message in context: http://camel.465427.n5.nabble.com/Connection-pooling-with-camel-amqp-tp5716936p5717827.html
Sent from the Camel - Users mailing list archive at Nabble.com.
RE: Connection pooling with camel-amqp
Posted by helander <le...@gmail.com>.
I have come up with a workaround to the NPE problem with Qpid.
The root of the problem is that Qpid is not able to correctly compare two
destinations that has not been fully qualified in their definitions. In
other parts of the Qpid API, assumptions are made on how to interpret
destinations that has not been fully qualified, but this is not reflected by
the equals-method of AMQDestination.
The "workaround" is to make sure that you provide fully qualified
destinations, and one way I have found to do that in conjunction with Camel
uri specifications are according to the following example:
where the name of the queue is *myqueue*.
If you are using topics instead of queues, the corresponding syntax should
be
I am not using topics at the moment, so I can not vouch for that the syntax
is correct in that case.
Thanks
Lars
--
View this message in context: http://camel.465427.n5.nabble.com/Connection-pooling-with-camel-amqp-tp5716936p5717826.html
Sent from the Camel - Users mailing list archive at Nabble.com.
RE: Connection pooling with camel-amqp
Posted by Zhemzhitsky Sergey <Se...@troika.ru>.
Hi Lars,
>> I have added a comment to QPID-3760.
Thanks a lot.
Best Regards,
Sergey Zhemzhitsky
-----Original Message-----
From: helander [mailto:lehswe@gmail.com]
Sent: Thursday, August 09, 2012 11:35 AM
To: users@camel.apache.org
Subject: Re: Connection pooling with camel-amqp
Sergey,
I have added a comment to QPID-3760.
As for your list (thank you btw):
1) tcp_nodelay is default (since 0.14) OK
2) no transacted session OK
3) non-persistent messages OK
4) cachingLevel = 3 OK
5) cachingLevel = 3 OK
6) no JEE server (OSGi) Not Applicable
/Lars
--
View this message in context: http://camel.465427.n5.nabble.com/Connection-pooling-with-camel-amqp-tp5716936p5717050.html
Sent from the Camel - Users mailing list archive at Nabble.com.
_______________________________________________________
The information contained in this message may be privileged and conf idential and protected from disclosure. If you are not the original intended recipient, you are hereby notified that any review, retransmission, dissemination, or other use of, or taking of any action in reliance upon, this information is prohibited. If you have received this communication in error, please notify the sender immediately by replying to this message and delete it from your computer. Thank you for your cooperation. Troika Dialog, Russia.
If you need assistance please contact our Contact Center (+7495) 258 0500 or go to www.troika.ru/eng/Contacts/system.wbp
Re: Connection pooling with camel-amqp
Posted by helander <le...@gmail.com>.
Sergey,
I have added a comment to QPID-3760.
As for your list (thank you btw):
1) tcp_nodelay is default (since 0.14) OK
2) no transacted session OK
3) non-persistent messages OK
4) cachingLevel = 3 OK
5) cachingLevel = 3 OK
6) no JEE server (OSGi) Not Applicable
/Lars
--
View this message in context: http://camel.465427.n5.nabble.com/Connection-pooling-with-camel-amqp-tp5716936p5717050.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Connection pooling with camel-amqp
Posted by Sergey Zhemzhitsky <sz...@gmail.com>.
Hi Lars,
Could you attach your patch to the https://issues.apache.org/jira/browse/QPID-3760 to speed up its resolution?
To have a better latency with qpid you probably have to make sure that:
1. tcp_nodelay (http://qpid.apache.org/books/0.16/Programming-In-Apache-Qpid/html/QpidJNDI.html#idp32879264)
option is set to true
2. session is not transacted
3. messages are non-persistent
4. on the producer side - producer, session and connection are cached
5. on the consumer side - consumer, session and connection are cached
6. if you're using any application server you could try qpid jca resource adapter
(http://svn.apache.org/repos/asf/qpid/trunk/qpid/java/jca/) to achieve "native" connection pooling
Regards,
Sergey Zhemzhitsky
> I have tested the fix in
> org.apache.qpid.client.AMQDestination.equals(), and
> now it works.
> I just made sure that no null valued references were used.
> Hopefully the qpid project will fix this soon. For my case, the fix I made
> helps, but I do not know if there may be situations where it could create
> problems, so I think that someone with real qpid knowledge should make the
> change in qpid.
> My goal with using caching/pooling of connections was to manage the "poor"
> performance (latency) I experienced, and latency was basically reduced by
> 50%. From my earlier measurements I had expected more (=even lower latency),
> so I probably have to study detailed logs from the caching solution.
> /Lars
> --
> View this message in context:
> http://camel.465427.n5.nabble.com/Connection-pooling-with-camel-amqp-tp5716936p5717032.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
--
Best regards,
Sergey mailto:szh.subs@gmail.com
Re: Connection pooling with camel-amqp
Posted by helander <le...@gmail.com>.
I have tested the fix in org.apache.qpid.client.AMQDestination.equals(), and
now it works.
I just made sure that no null valued references were used.
Hopefully the qpid project will fix this soon. For my case, the fix I made
helps, but I do not know if there may be situations where it could create
problems, so I think that someone with real qpid knowledge should make the
change in qpid.
My goal with using caching/pooling of connections was to manage the "poor"
performance (latency) I experienced, and latency was basically reduced by
50%. From my earlier measurements I had expected more (=even lower latency),
so I probably have to study detailed logs from the caching solution.
/Lars
--
View this message in context: http://camel.465427.n5.nabble.com/Connection-pooling-with-camel-amqp-tp5716936p5717032.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Connection pooling with camel-amqp
Posted by helander <le...@gmail.com>.
Hi Christian,
I found this JIRA earlier today, and most likely it is the same problem.
The equals() function does not seem to deal with all possible varaints of
destination definitions.
I am in the process of trying out a fix, but I have not yet been able to
build a new qpid-client. Will struggle on for a while
/Lars
--
View this message in context: http://camel.465427.n5.nabble.com/Connection-pooling-with-camel-amqp-tp5716936p5717019.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Connection pooling with camel-amqp
Posted by Christian Müller <ch...@gmail.com>.
I voted for https://issues.apache.org/jira/browse/QPID-3760.
May it speed up a fix...
Best,
Christian
On Wed, Aug 8, 2012 at 1:16 PM, helander <le...@gmail.com> wrote:
> Hi Michal,
>
> before I try that, I think I will try to investigate why they are null.
>
> /Lars
>
>
>
> --
> View this message in context:
> http://camel.465427.n5.nabble.com/Connection-pooling-with-camel-amqp-tp5716936p5716980.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
--
Re: Connection pooling with camel-amqp
Posted by helander <le...@gmail.com>.
Hi Michal,
before I try that, I think I will try to investigate why they are null.
/Lars
--
View this message in context: http://camel.465427.n5.nabble.com/Connection-pooling-with-camel-amqp-tp5716936p5716980.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Connection pooling with camel-amqp
Posted by "michal.warecki" <mi...@gmail.com>.
Hmm, I don't know... In class AMQDestination there is method equals, and in
case _exchangeClass or _exchangeName are null there is NPE thrown.
Maybe try to replace lines:
if (!_exchangeClass.equals(that._exchangeClass))
{
return false;
}
if (!_exchangeName.equals(that._exchangeName))
{
return false;
}
with this:
if(_exchangeClass == null && that._exchangeClass != null)
{
return false;
}
if(_exchangeClass != null && that._exchangeClass == null)
{
return false;
}
if(_exchangeClass != null && that._exchangeClass != null) {
if (!_exchangeClass.equals(that._exchangeClass))
{
return false;
}
}
if(_exchangeName == null && that._exchangeName != null)
{
return false;
}
if(_exchangeName != null && that._exchangeName == null)
{
return false;
}
if(_exchangeName != null && that._exchangeName != null) {
if (!_exchangeName.equals(that._exchangeName))
{
return false;
}
}
I don't know the details of AMQDestination so I don't know if this is
correct code...
--
View this message in context: http://camel.465427.n5.nabble.com/Connection-pooling-with-camel-amqp-tp5716936p5716977.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Connection pooling with camel-amqp
Posted by Lars-Erik Helander <le...@gmail.com>.
Michal,
my stack trace is more or less identical (java source line numbers are not identical).
Do you have any idea how to work around this, patch qpid code or other solution? Creation of new connection and destination validation on each operation introduces significant overhead. I turned on debug logging and went thru the timing - and found out how much overhead there is :(.
Thanks for finding the bug.
/Lars
Skickat från min iPhone
7 aug 2012 kl. 22:39 skrev "michal.warecki" <mi...@gmail.com>:
> Can you paste your stacktrace also?
>
> If your stacktrace looks like this:
> Exception in thread "Thread-5" java.lang.NullPointerException
> at org.apache.qpid.client.AMQDestination.equals(AMQDestination.java:577)
> at
> org.springframework.jms.connection.CachingConnectionFactory$DestinationCacheKey.destinationEquals(CachingConnectionFactory.java:467)
> at
> org.springframework.jms.connection.CachingConnectionFactory$DestinationCacheKey.equals(CachingConnectionFactory.java:474)
> at java.util.HashMap.get(HashMap.java:385)
> at
> org.springframework.jms.connection.CachingConnectionFactory$CachedSessionInvocationHandler.getCachedProducer(CachingConnectionFactory.java:353)
> at
> org.springframework.jms.connection.CachingConnectionFactory$CachedSessionInvocationHandler.invoke(CachingConnectionFactory.java:320)
> at $Proxy1.createProducer(Unknown Source)
>
> this is a QPID bug:
> https://issues.apache.org/jira/browse/QPID-3760
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Connection-pooling-with-camel-amqp-tp5716936p5716947.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Connection pooling with camel-amqp
Posted by "michal.warecki" <mi...@gmail.com>.
Can you paste your stacktrace also?
If your stacktrace looks like this:
Exception in thread "Thread-5" java.lang.NullPointerException
at org.apache.qpid.client.AMQDestination.equals(AMQDestination.java:577)
at
org.springframework.jms.connection.CachingConnectionFactory$DestinationCacheKey.destinationEquals(CachingConnectionFactory.java:467)
at
org.springframework.jms.connection.CachingConnectionFactory$DestinationCacheKey.equals(CachingConnectionFactory.java:474)
at java.util.HashMap.get(HashMap.java:385)
at
org.springframework.jms.connection.CachingConnectionFactory$CachedSessionInvocationHandler.getCachedProducer(CachingConnectionFactory.java:353)
at
org.springframework.jms.connection.CachingConnectionFactory$CachedSessionInvocationHandler.invoke(CachingConnectionFactory.java:320)
at $Proxy1.createProducer(Unknown Source)
this is a QPID bug:
https://issues.apache.org/jira/browse/QPID-3760
--
View this message in context: http://camel.465427.n5.nabble.com/Connection-pooling-with-camel-amqp-tp5716936p5716947.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Connection pooling with camel-amqp
Posted by helander <le...@gmail.com>.
This is part of an OSGi blueprint xml file.
<bean id="myConnectionFactory"
class="org.apache.qpid.client.AMQConnectionFactory">
<argument
value="amqp://guest:guest@clientid/test?brokerlist='tcp://localhost:5672'"
/>
</bean>
<bean id="cachingConnectionFactory"
class="org.springframework.jms.connection.CachingConnectionFactory">
<property name="targetConnectionFactory" ref="myConnectionFactory"/>
</bean>
<bean id="mybroker" class="org.apache.camel.component.amqp.AMQPComponent">
<property name="connectionFactory" ref="cachingConnectionFactory"/>
<property name="cacheLevel" value="2"/>
</bean>
<camelContext xmlns="http://camel.apache.org/schema/blueprint">
<endpoint id="queue23" uri="mybroker:queue:Q_23"/>
<route>
<from uri="direct-vm:q_23" />
<to ref="queue23" />
</route>
</camelContext>
/Lars
--
View this message in context: http://camel.465427.n5.nabble.com/Connection-pooling-with-camel-amqp-tp5716936p5716941.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Connection pooling with camel-amqp
Posted by "michal.warecki" <mi...@gmail.com>.
Hi helander,
Can you paste the code responsible for pooling?
--
View this message in context: http://camel.465427.n5.nabble.com/Connection-pooling-with-camel-amqp-tp5716936p5716939.html
Sent from the Camel - Users mailing list archive at Nabble.com.