You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@camel.apache.org by Claus Ibsen <cl...@gmail.com> on 2009/05/25 10:19:04 UTC
[DISCUSS - CAMEL 2.0] - About Producers and thread safety
Hi
A recent bug was reported in ticket CAMEL-1641
https://issues.apache.org/activemq/browse/CAMEL-1641
The issue is in Camel 2.0 the FTP producer is not thread safe, in the
sense that you cannot concurrently send at the same time to the same
FTP endpoint.
I have digged into this issue and discovered the issue lies in the
fact that the FtpProducer cannot be singleton scoped. It uses a local
FTPClient that is used for the FTP communication.
A given endpoint producer is however stored in either a producer cache
or as a class instance variable in SendProcessor.
This causes the issue that a non singleton producer will be reused for
concurrent threads.
So to remedy this I have done some code as a prototype that I usually
do during my findings.
We extend the concept of IsSingleton from Endpoint to Producer as
well. Then we can set it as false for FtpProducer.
We let the producer cache detect the IsSingleton and only cache singletons.
I have also added a ProducerCallback interface that resembles the
Spring resource patterns for its JmsTemplate, JdbcTemplate etc with
its doInXXXX
public interface ProducerCallback<T> {
/**
* Performs operation on the given producer to send the given exchange.
*
* @param producer the producer, is newer <tt>null</tt>
* @param exchange the exchange, can be <tt>null</tt> if so then
create a new exchange from the producer
* @param exchangePattern the exchange pattern, can be <tt>null</tt>
* @return the response
* @throws Exception if an internal processing error has occurred.
*/
T doInProducer(Producer producer, Exchange exchange,
ExchangePattern exchangePattern) throws Exception;
}
With this callback it allows a producer client to safely use the
producer without having to worry about how to start and stop the
resource whether its singleton or not.
And in the future we can incorporate a pooling as well so Camel can
leverage pooling with this pattern. So Camel can safely obtain a
producer from the pool, invoke this callback where the client/user can
use it to do what is needed, usually to send an exchange. And
afterwards Camel can safely return the producer back to the pool.
However I know this is kinda expanding Camel out of its current scope.
There is a tiny concept of IoC or resource/lifecycle handling in here.
Another idea is also to denote the ProducerCache as a higher role. We
could let CamelContext be able to hold a per context producer cache
and let it provide this cache where its needed in: ProducerTemplate,
SendToProcessor and a few other areas. This allows us to reuse this
cache instead of having a per local cache.
Well a few concepts to grasp. I will create a ticket for this and
attach a patch for people to look at.
Any thoughts?
--
Claus Ibsen
Apache Camel Committer
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus
Re: [DISCUSS - CAMEL 2.0] - About Producers and thread safety
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
Update on this one.
We implemented a generic pooling in Camel naming it Service Pool.
Details in the wiki page:
http://camel.apache.org/servicepool.html
On Mon, May 25, 2009 at 3:10 PM, Claus Ibsen <cl...@gmail.com> wrote:
> Hi
>
> I have looked into this a bit more.
>
> With this we can support correlation for concurrent request/response
> with mina as well. A feature that was generelly lacking there.
>
> So this is definitely a great step in a good direction for Camel.
>
> This feature has been tracked here:
> http://fusesource.com/issues/browse/MR-18
>
>
>
> On Mon, May 25, 2009 at 10:19 AM, Claus Ibsen <cl...@gmail.com> wrote:
>> Hi
>>
>> A recent bug was reported in ticket CAMEL-1641
>> https://issues.apache.org/activemq/browse/CAMEL-1641
>>
>> The issue is in Camel 2.0 the FTP producer is not thread safe, in the
>> sense that you cannot concurrently send at the same time to the same
>> FTP endpoint.
>>
>> I have digged into this issue and discovered the issue lies in the
>> fact that the FtpProducer cannot be singleton scoped. It uses a local
>> FTPClient that is used for the FTP communication.
>>
>> A given endpoint producer is however stored in either a producer cache
>> or as a class instance variable in SendProcessor.
>> This causes the issue that a non singleton producer will be reused for
>> concurrent threads.
>>
>> So to remedy this I have done some code as a prototype that I usually
>> do during my findings.
>>
>> We extend the concept of IsSingleton from Endpoint to Producer as
>> well. Then we can set it as false for FtpProducer.
>> We let the producer cache detect the IsSingleton and only cache singletons.
>>
>> I have also added a ProducerCallback interface that resembles the
>> Spring resource patterns for its JmsTemplate, JdbcTemplate etc with
>> its doInXXXX
>>
>> public interface ProducerCallback<T> {
>>
>> /**
>> * Performs operation on the given producer to send the given exchange.
>> *
>> * @param producer the producer, is newer <tt>null</tt>
>> * @param exchange the exchange, can be <tt>null</tt> if so then
>> create a new exchange from the producer
>> * @param exchangePattern the exchange pattern, can be <tt>null</tt>
>> * @return the response
>> * @throws Exception if an internal processing error has occurred.
>> */
>> T doInProducer(Producer producer, Exchange exchange,
>> ExchangePattern exchangePattern) throws Exception;
>> }
>>
>> With this callback it allows a producer client to safely use the
>> producer without having to worry about how to start and stop the
>> resource whether its singleton or not.
>>
>> And in the future we can incorporate a pooling as well so Camel can
>> leverage pooling with this pattern. So Camel can safely obtain a
>> producer from the pool, invoke this callback where the client/user can
>> use it to do what is needed, usually to send an exchange. And
>> afterwards Camel can safely return the producer back to the pool.
>>
>> However I know this is kinda expanding Camel out of its current scope.
>> There is a tiny concept of IoC or resource/lifecycle handling in here.
>>
>>
>> Another idea is also to denote the ProducerCache as a higher role. We
>> could let CamelContext be able to hold a per context producer cache
>> and let it provide this cache where its needed in: ProducerTemplate,
>> SendToProcessor and a few other areas. This allows us to reuse this
>> cache instead of having a per local cache.
>>
>>
>> Well a few concepts to grasp. I will create a ticket for this and
>> attach a patch for people to look at.
>>
>>
>> Any thoughts?
>>
>>
>>
>>
>>
>> --
>> Claus Ibsen
>> Apache Camel Committer
>>
>> Open Source Integration: http://fusesource.com
>> Blog: http://davsclaus.blogspot.com/
>> Twitter: http://twitter.com/davsclaus
>>
>
>
>
> --
> Claus Ibsen
> Apache Camel Committer
>
> Open Source Integration: http://fusesource.com
> Blog: http://davsclaus.blogspot.com/
> Twitter: http://twitter.com/davsclaus
>
--
Claus Ibsen
Apache Camel Committer
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus
Re: [DISCUSS - CAMEL 2.0] - About Producers and thread safety
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
I have looked into this a bit more.
With this we can support correlation for concurrent request/response
with mina as well. A feature that was generelly lacking there.
So this is definitely a great step in a good direction for Camel.
This feature has been tracked here:
http://fusesource.com/issues/browse/MR-18
On Mon, May 25, 2009 at 10:19 AM, Claus Ibsen <cl...@gmail.com> wrote:
> Hi
>
> A recent bug was reported in ticket CAMEL-1641
> https://issues.apache.org/activemq/browse/CAMEL-1641
>
> The issue is in Camel 2.0 the FTP producer is not thread safe, in the
> sense that you cannot concurrently send at the same time to the same
> FTP endpoint.
>
> I have digged into this issue and discovered the issue lies in the
> fact that the FtpProducer cannot be singleton scoped. It uses a local
> FTPClient that is used for the FTP communication.
>
> A given endpoint producer is however stored in either a producer cache
> or as a class instance variable in SendProcessor.
> This causes the issue that a non singleton producer will be reused for
> concurrent threads.
>
> So to remedy this I have done some code as a prototype that I usually
> do during my findings.
>
> We extend the concept of IsSingleton from Endpoint to Producer as
> well. Then we can set it as false for FtpProducer.
> We let the producer cache detect the IsSingleton and only cache singletons.
>
> I have also added a ProducerCallback interface that resembles the
> Spring resource patterns for its JmsTemplate, JdbcTemplate etc with
> its doInXXXX
>
> public interface ProducerCallback<T> {
>
> /**
> * Performs operation on the given producer to send the given exchange.
> *
> * @param producer the producer, is newer <tt>null</tt>
> * @param exchange the exchange, can be <tt>null</tt> if so then
> create a new exchange from the producer
> * @param exchangePattern the exchange pattern, can be <tt>null</tt>
> * @return the response
> * @throws Exception if an internal processing error has occurred.
> */
> T doInProducer(Producer producer, Exchange exchange,
> ExchangePattern exchangePattern) throws Exception;
> }
>
> With this callback it allows a producer client to safely use the
> producer without having to worry about how to start and stop the
> resource whether its singleton or not.
>
> And in the future we can incorporate a pooling as well so Camel can
> leverage pooling with this pattern. So Camel can safely obtain a
> producer from the pool, invoke this callback where the client/user can
> use it to do what is needed, usually to send an exchange. And
> afterwards Camel can safely return the producer back to the pool.
>
> However I know this is kinda expanding Camel out of its current scope.
> There is a tiny concept of IoC or resource/lifecycle handling in here.
>
>
> Another idea is also to denote the ProducerCache as a higher role. We
> could let CamelContext be able to hold a per context producer cache
> and let it provide this cache where its needed in: ProducerTemplate,
> SendToProcessor and a few other areas. This allows us to reuse this
> cache instead of having a per local cache.
>
>
> Well a few concepts to grasp. I will create a ticket for this and
> attach a patch for people to look at.
>
>
> Any thoughts?
>
>
>
>
>
> --
> Claus Ibsen
> Apache Camel Committer
>
> Open Source Integration: http://fusesource.com
> Blog: http://davsclaus.blogspot.com/
> Twitter: http://twitter.com/davsclaus
>
--
Claus Ibsen
Apache Camel Committer
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus