You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@camel.apache.org by Claus Ibsen <cl...@gmail.com> on 2009/06/30 07:24:10 UTC
[DISCUSS] - Camel 2.0 - About Endpoints with lenient properties
eating to much memory
Hi
Tickets
======
I am looking into two issues related to Camel eating memory
https://issues.apache.org/activemq/browse/CAMEL-1771
https://issues.apache.org/activemq/browse/CAMEL-1738
The problem
=========
It all boils down to using a lot of http endpoints with unique urls.
So over time Camel accumulate a lot of created endpoints in its
internal endpoint registry and as well as JMX Beans
That consumes memory, and for people with millions unique endpoints
over time that consumes to much memory
Solutions
=======
To remedy this I have addressed in two areas
a) failsafe with a limited size of 1000 entries
- Using the LRUCache for ProducerCache/ConsumerCache (verified by end
user that its much better)
- Using the LRUCache in CamelContext for its endpoint reigstry
I assume having more than 1000 unique endpoints, producers or
consumers is not within normal usage for a given CamelContext?
And it does not bring harm as Camel will just recreate the object if
not already cached.
b)
- Not registering endpoints with lenient properties in JMX or camel
context. That is if only they use properties that Camel does not know
about (= lenient properties)
as these endpoints is highly not reusable and short lived. And its
only a few endpoints that support lenient properties (http, restlet,
cxf, atom, rss)
So if you setup an endpoint with custom parameters (not Camel options)
then its a lenient property and Camel will not cache/register it.
For example:
http://myserver/mypath?id=1
http://myserver/mypath?id=2
http://myserver/mypath?id=3
...
where id is a lenient property that is not a Camel option.
So these will not be registered.
But if you use
http://myserver/mypath
then it will be registered as its does not contain any lenient
properties at all.
c)
- Only registering a single JMX bean.
Otherwise the JMX registry will be cluttered with millions endpoints.
So what we do not is to register the same parent endpoint for the
endpoints.
We use the new getEndpointKey() on Endpoint to provide the key to use
for JMX registration. This allows us to provide the same key for all
the http endpoints.
Questions
========
1)
Option a and c is already nearly done. I am running further unit tests
and do a bit of code polish.
2)
As option b is a bit controversial, I am wondering if that is
feasible. Should we just go ahead with the LRUCache being able to
filter out the eldest endpoints?
And I wonder if people just create a few endpoints with lenient
properties then we still have "room" to store them in the cache so I
wonder if we should do this at all?
I do think we could keep this as a thought for the future, and just
let the LRUCache handle it, so when people use millions of unqiue http
endpoints we let the chace
filter out the not used ones.
--
Claus Ibsen
Apache Camel Committer
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus
Re: [DISCUSS] - Camel 2.0 - About Endpoints with lenient properties
eating to much memory
Posted by Claus Ibsen <cl...@gmail.com>.
On Tue, Jun 30, 2009 at 11:15 AM, Willem Jiang<wi...@gmail.com> wrote:
> Hi Claus,
>
> I think we need to implements a method for get unique name from the
> endpoint. It will base on the endpoint's parameter not the customer
> parameter. With this method we could avoid to register to much temp endpoint
> with different customer parameter.
Yeah good idea. I kinda added that already with the getEndpointKey().
You can then override the default implementation on demand.
>
> In this case we need find a way to trim the customer parameter from the
> endpoint URI. If we could do that, it will save us lots room for the store
> the fake temp http endpoints (we could share the endpoints in the
> ProducerCache/ConsumerCache with different customer parameter) and make the
> LRUCache more effective.
The problem with the http endpoints is that they require the full uri
to be useable as the producer created by it will send the camel
message
to the full uri provided on the endpoint.
So if the http endpoint is configured as: http://someserver/somepath/foo?id=1234
Then we cannot create/reuse a single endpoint with the uri:
http://someserver/somepath/foo
As then we will loose the ?id=1234 parameter.
So there is no quick win to only cache 1 shared http endpoint and
creating producers that know to use ?id=1234 when sending to it.
But as James hinted we could add a IsCacheable method/interface to
mark a given endpoint/producer whatever as being capable of being
cached.
Then we could deny caching those http endpoints with lenient
properties, as they are highly not reuseable.
BTW: then end user confirmed that the memory issue is fixed in 1.6.2
with the patches committed today.
>
> Just my two cents.
>
> Willem
>
> Claus Ibsen wrote:
>>
>> Hi
>>
>> Tickets
>> ======
>> I am looking into two issues related to Camel eating memory
>> https://issues.apache.org/activemq/browse/CAMEL-1771
>> https://issues.apache.org/activemq/browse/CAMEL-1738
>>
>>
>> The problem
>> =========
>> It all boils down to using a lot of http endpoints with unique urls.
>> So over time Camel accumulate a lot of created endpoints in its
>> internal endpoint registry and as well as JMX Beans
>> That consumes memory, and for people with millions unique endpoints
>> over time that consumes to much memory
>>
>>
>> Solutions
>> =======
>> To remedy this I have addressed in two areas
>>
>> a) failsafe with a limited size of 1000 entries
>> - Using the LRUCache for ProducerCache/ConsumerCache (verified by end
>> user that its much better)
>> - Using the LRUCache in CamelContext for its endpoint reigstry
>>
>> I assume having more than 1000 unique endpoints, producers or
>> consumers is not within normal usage for a given CamelContext?
>> And it does not bring harm as Camel will just recreate the object if
>> not already cached.
>>
>>
>> b)
>> - Not registering endpoints with lenient properties in JMX or camel
>> context. That is if only they use properties that Camel does not know
>> about (= lenient properties)
>> as these endpoints is highly not reusable and short lived. And its
>> only a few endpoints that support lenient properties (http, restlet,
>> cxf, atom, rss)
>>
>> So if you setup an endpoint with custom parameters (not Camel options)
>> then its a lenient property and Camel will not cache/register it.
>> For example:
>>
>> http://myserver/mypath?id=1
>> http://myserver/mypath?id=2
>> http://myserver/mypath?id=3
>> ...
>>
>> where id is a lenient property that is not a Camel option.
>> So these will not be registered.
>>
>>
>> But if you use
>> http://myserver/mypath
>> then it will be registered as its does not contain any lenient
>> properties at all.
>>
>>
>> c)
>> - Only registering a single JMX bean.
>> Otherwise the JMX registry will be cluttered with millions endpoints.
>> So what we do not is to register the same parent endpoint for the
>> endpoints.
>> We use the new getEndpointKey() on Endpoint to provide the key to use
>> for JMX registration. This allows us to provide the same key for all
>> the http endpoints.
>>
>>
>> Questions
>> ========
>> 1)
>> Option a and c is already nearly done. I am running further unit tests
>> and do a bit of code polish.
>>
>> 2)
>> As option b is a bit controversial, I am wondering if that is
>> feasible. Should we just go ahead with the LRUCache being able to
>> filter out the eldest endpoints?
>> And I wonder if people just create a few endpoints with lenient
>> properties then we still have "room" to store them in the cache so I
>> wonder if we should do this at all?
>> I do think we could keep this as a thought for the future, and just
>> let the LRUCache handle it, so when people use millions of unqiue http
>> endpoints we let the chace
>> filter out the not used ones.
>>
>>
>>
>>
>
>
--
Claus Ibsen
Apache Camel Committer
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus
Re: [DISCUSS] - Camel 2.0 - About Endpoints with lenient properties
eating to much memory
Posted by Claus Ibsen <cl...@gmail.com>.
On Tue, Jun 30, 2009 at 1:50 PM, James Strachan<ja...@gmail.com> wrote:
> I'm liking a) and allowing c) (a single mbean for all endpoints of a
> certain type) make sense to me.
Just for the record. Option a + c is currently implemented and
reported as fixed the issues by the end user.
>
> Using an LRU cache for non-singleton endpoints makes sense too; though
> I wonder if an endpoint might want to mark itself as not cachable if
> it knows its not really worth the trouble (as it has no state?).
>
>
> 2009/6/30 Willem Jiang <wi...@gmail.com>:
>> Hi Claus,
>>
>> I think we need to implements a method for get unique name from the
>> endpoint. It will base on the endpoint's parameter not the customer
>> parameter. With this method we could avoid to register to much temp endpoint
>> with different customer parameter.
>>
>> In this case we need find a way to trim the customer parameter from the
>> endpoint URI. If we could do that, it will save us lots room for the store
>> the fake temp http endpoints (we could share the endpoints in the
>> ProducerCache/ConsumerCache with different customer parameter) and make the
>> LRUCache more effective.
>>
>> Just my two cents.
>>
>> Willem
>>
>> Claus Ibsen wrote:
>>>
>>> Hi
>>>
>>> Tickets
>>> ======
>>> I am looking into two issues related to Camel eating memory
>>> https://issues.apache.org/activemq/browse/CAMEL-1771
>>> https://issues.apache.org/activemq/browse/CAMEL-1738
>>>
>>>
>>> The problem
>>> =========
>>> It all boils down to using a lot of http endpoints with unique urls.
>>> So over time Camel accumulate a lot of created endpoints in its
>>> internal endpoint registry and as well as JMX Beans
>>> That consumes memory, and for people with millions unique endpoints
>>> over time that consumes to much memory
>>>
>>>
>>> Solutions
>>> =======
>>> To remedy this I have addressed in two areas
>>>
>>> a) failsafe with a limited size of 1000 entries
>>> - Using the LRUCache for ProducerCache/ConsumerCache (verified by end
>>> user that its much better)
>>> - Using the LRUCache in CamelContext for its endpoint reigstry
>>>
>>> I assume having more than 1000 unique endpoints, producers or
>>> consumers is not within normal usage for a given CamelContext?
>>> And it does not bring harm as Camel will just recreate the object if
>>> not already cached.
>>>
>>>
>>> b)
>>> - Not registering endpoints with lenient properties in JMX or camel
>>> context. That is if only they use properties that Camel does not know
>>> about (= lenient properties)
>>> as these endpoints is highly not reusable and short lived. And its
>>> only a few endpoints that support lenient properties (http, restlet,
>>> cxf, atom, rss)
>>>
>>> So if you setup an endpoint with custom parameters (not Camel options)
>>> then its a lenient property and Camel will not cache/register it.
>>> For example:
>>>
>>> http://myserver/mypath?id=1
>>> http://myserver/mypath?id=2
>>> http://myserver/mypath?id=3
>>> ...
>>>
>>> where id is a lenient property that is not a Camel option.
>>> So these will not be registered.
>>>
>>>
>>> But if you use
>>> http://myserver/mypath
>>> then it will be registered as its does not contain any lenient
>>> properties at all.
>>>
>>>
>>> c)
>>> - Only registering a single JMX bean.
>>> Otherwise the JMX registry will be cluttered with millions endpoints.
>>> So what we do not is to register the same parent endpoint for the
>>> endpoints.
>>> We use the new getEndpointKey() on Endpoint to provide the key to use
>>> for JMX registration. This allows us to provide the same key for all
>>> the http endpoints.
>>>
>>>
>>> Questions
>>> ========
>>> 1)
>>> Option a and c is already nearly done. I am running further unit tests
>>> and do a bit of code polish.
>>>
>>> 2)
>>> As option b is a bit controversial, I am wondering if that is
>>> feasible. Should we just go ahead with the LRUCache being able to
>>> filter out the eldest endpoints?
>>> And I wonder if people just create a few endpoints with lenient
>>> properties then we still have "room" to store them in the cache so I
>>> wonder if we should do this at all?
>>> I do think we could keep this as a thought for the future, and just
>>> let the LRUCache handle it, so when people use millions of unqiue http
>>> endpoints we let the chace
>>> filter out the not used ones.
>>>
>>>
>>>
>>>
>>
>>
>
>
>
> --
> James
> -------
> http://macstrac.blogspot.com/
>
> Open Source Integration
> http://fusesource.com/
>
--
Claus Ibsen
Apache Camel Committer
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus
Re: [DISCUSS] - Camel 2.0 - About Endpoints with lenient properties
eating to much memory
Posted by James Strachan <ja...@gmail.com>.
I'm liking a) and allowing c) (a single mbean for all endpoints of a
certain type) make sense to me.
Using an LRU cache for non-singleton endpoints makes sense too; though
I wonder if an endpoint might want to mark itself as not cachable if
it knows its not really worth the trouble (as it has no state?).
2009/6/30 Willem Jiang <wi...@gmail.com>:
> Hi Claus,
>
> I think we need to implements a method for get unique name from the
> endpoint. It will base on the endpoint's parameter not the customer
> parameter. With this method we could avoid to register to much temp endpoint
> with different customer parameter.
>
> In this case we need find a way to trim the customer parameter from the
> endpoint URI. If we could do that, it will save us lots room for the store
> the fake temp http endpoints (we could share the endpoints in the
> ProducerCache/ConsumerCache with different customer parameter) and make the
> LRUCache more effective.
>
> Just my two cents.
>
> Willem
>
> Claus Ibsen wrote:
>>
>> Hi
>>
>> Tickets
>> ======
>> I am looking into two issues related to Camel eating memory
>> https://issues.apache.org/activemq/browse/CAMEL-1771
>> https://issues.apache.org/activemq/browse/CAMEL-1738
>>
>>
>> The problem
>> =========
>> It all boils down to using a lot of http endpoints with unique urls.
>> So over time Camel accumulate a lot of created endpoints in its
>> internal endpoint registry and as well as JMX Beans
>> That consumes memory, and for people with millions unique endpoints
>> over time that consumes to much memory
>>
>>
>> Solutions
>> =======
>> To remedy this I have addressed in two areas
>>
>> a) failsafe with a limited size of 1000 entries
>> - Using the LRUCache for ProducerCache/ConsumerCache (verified by end
>> user that its much better)
>> - Using the LRUCache in CamelContext for its endpoint reigstry
>>
>> I assume having more than 1000 unique endpoints, producers or
>> consumers is not within normal usage for a given CamelContext?
>> And it does not bring harm as Camel will just recreate the object if
>> not already cached.
>>
>>
>> b)
>> - Not registering endpoints with lenient properties in JMX or camel
>> context. That is if only they use properties that Camel does not know
>> about (= lenient properties)
>> as these endpoints is highly not reusable and short lived. And its
>> only a few endpoints that support lenient properties (http, restlet,
>> cxf, atom, rss)
>>
>> So if you setup an endpoint with custom parameters (not Camel options)
>> then its a lenient property and Camel will not cache/register it.
>> For example:
>>
>> http://myserver/mypath?id=1
>> http://myserver/mypath?id=2
>> http://myserver/mypath?id=3
>> ...
>>
>> where id is a lenient property that is not a Camel option.
>> So these will not be registered.
>>
>>
>> But if you use
>> http://myserver/mypath
>> then it will be registered as its does not contain any lenient
>> properties at all.
>>
>>
>> c)
>> - Only registering a single JMX bean.
>> Otherwise the JMX registry will be cluttered with millions endpoints.
>> So what we do not is to register the same parent endpoint for the
>> endpoints.
>> We use the new getEndpointKey() on Endpoint to provide the key to use
>> for JMX registration. This allows us to provide the same key for all
>> the http endpoints.
>>
>>
>> Questions
>> ========
>> 1)
>> Option a and c is already nearly done. I am running further unit tests
>> and do a bit of code polish.
>>
>> 2)
>> As option b is a bit controversial, I am wondering if that is
>> feasible. Should we just go ahead with the LRUCache being able to
>> filter out the eldest endpoints?
>> And I wonder if people just create a few endpoints with lenient
>> properties then we still have "room" to store them in the cache so I
>> wonder if we should do this at all?
>> I do think we could keep this as a thought for the future, and just
>> let the LRUCache handle it, so when people use millions of unqiue http
>> endpoints we let the chace
>> filter out the not used ones.
>>
>>
>>
>>
>
>
--
James
-------
http://macstrac.blogspot.com/
Open Source Integration
http://fusesource.com/
Re: [DISCUSS] - Camel 2.0 - About Endpoints with lenient properties
eating to much memory
Posted by Willem Jiang <wi...@gmail.com>.
Hi Claus,
I think we need to implements a method for get unique name from the
endpoint. It will base on the endpoint's parameter not the customer
parameter. With this method we could avoid to register to much temp
endpoint with different customer parameter.
In this case we need find a way to trim the customer parameter from the
endpoint URI. If we could do that, it will save us lots room for the
store the fake temp http endpoints (we could share the endpoints in the
ProducerCache/ConsumerCache with different customer parameter) and make
the LRUCache more effective.
Just my two cents.
Willem
Claus Ibsen wrote:
> Hi
>
> Tickets
> ======
> I am looking into two issues related to Camel eating memory
> https://issues.apache.org/activemq/browse/CAMEL-1771
> https://issues.apache.org/activemq/browse/CAMEL-1738
>
>
> The problem
> =========
> It all boils down to using a lot of http endpoints with unique urls.
> So over time Camel accumulate a lot of created endpoints in its
> internal endpoint registry and as well as JMX Beans
> That consumes memory, and for people with millions unique endpoints
> over time that consumes to much memory
>
>
> Solutions
> =======
> To remedy this I have addressed in two areas
>
> a) failsafe with a limited size of 1000 entries
> - Using the LRUCache for ProducerCache/ConsumerCache (verified by end
> user that its much better)
> - Using the LRUCache in CamelContext for its endpoint reigstry
>
> I assume having more than 1000 unique endpoints, producers or
> consumers is not within normal usage for a given CamelContext?
> And it does not bring harm as Camel will just recreate the object if
> not already cached.
>
>
> b)
> - Not registering endpoints with lenient properties in JMX or camel
> context. That is if only they use properties that Camel does not know
> about (= lenient properties)
> as these endpoints is highly not reusable and short lived. And its
> only a few endpoints that support lenient properties (http, restlet,
> cxf, atom, rss)
>
> So if you setup an endpoint with custom parameters (not Camel options)
> then its a lenient property and Camel will not cache/register it.
> For example:
>
> http://myserver/mypath?id=1
> http://myserver/mypath?id=2
> http://myserver/mypath?id=3
> ...
>
> where id is a lenient property that is not a Camel option.
> So these will not be registered.
>
>
> But if you use
> http://myserver/mypath
> then it will be registered as its does not contain any lenient
> properties at all.
>
>
> c)
> - Only registering a single JMX bean.
> Otherwise the JMX registry will be cluttered with millions endpoints.
> So what we do not is to register the same parent endpoint for the
> endpoints.
> We use the new getEndpointKey() on Endpoint to provide the key to use
> for JMX registration. This allows us to provide the same key for all
> the http endpoints.
>
>
> Questions
> ========
> 1)
> Option a and c is already nearly done. I am running further unit tests
> and do a bit of code polish.
>
> 2)
> As option b is a bit controversial, I am wondering if that is
> feasible. Should we just go ahead with the LRUCache being able to
> filter out the eldest endpoints?
> And I wonder if people just create a few endpoints with lenient
> properties then we still have "room" to store them in the cache so I
> wonder if we should do this at all?
> I do think we could keep this as a thought for the future, and just
> let the LRUCache handle it, so when people use millions of unqiue http
> endpoints we let the chace
> filter out the not used ones.
>
>
>
>