You are viewing a plain text version of this content. The canonical link for it is here.
Posted to server-dev@james.apache.org by "Noel J. Bergman" <no...@devtech.com> on 2006/10/01 23:11:46 UTC
Caching in DNSJAVA
Brian,
Stefano wrote:
> Noel J. Bergman wrote:
> > I lowered maxcachesize to 5000 from the default of 50000
> > If nothing else, the smaller cache should help eliminate
> > that memory consumption from consideration.
> Unfortunately I think that in current DNSServer implementation we
> can only tune the "IN" cache and not SOA, PTR and other caches. So
> they will be 50000 anyway.
> I've not checked the code for this, but I'm almost sure I remember
> this is how it works.
dnsjava is consistently the largest user of memory in my JAMES heap, and
does not appear to be entirely bounded, although I am having surprising
difficulty keeping it running when heap dumps are enabled.
Can you comment on the caching behavior inside of dnsjava? We use dnsjava
via:
http://svn.apache.org/repos/asf/james/server/branches/v2.3/src/java/org/apac
he/james/dnsserver/DNSServer.java
As you can see, we setup:
cache = new Cache (DClass.IN);
cache.setMaxEntries(maxCacheSize);
Lookup.setDefaultCache(cache, DClass.IN);
On a relatively slow day, my server processes 100K+ connections, of which
anywhere from 45%-70% might be blocked by a DNSRBL.
--- Noel
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: Caching in DNSJAVA
Posted by Stefano Bagnara <ap...@bago.org>.
Brian Wellington wrote:
> On Mon, 9 Oct 2006, Stefano Bagnara wrote:
>> Can I suggest to make some of the Cache private field/method protected
>> or extract the Cache interface from the default implementation?
>
> Are you suggesting simply making CacheMap public, and making the data
> field accessible? I don't think data should be public, but adding
> accessor methods would be ok.
I have not checked it yet but from a fast review making CacheMap public
and data protected would anyway force as to use CacheMap extensions for
the custom implementation.
Maybe the easier way would be:
- change data field type from CacheMap to Map
- change data field visibility from private to protected
- add casts to Cache.(set|get)MaxEntries()
-------
public int getMaxEntries() {
return ((CacheMap) data).getMaxSize();
}
public void setMaxEntries(int entries) {
((CacheMap) data).setMaxSize(entries);
}
--------
This way it would be easier to provide a custom Map implementation by
overriding the constructors (creating a different map instead of
CacheMap) and overriding the get/set max entries to avoid the
classcastexception.
What do you think?
Stefano
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: Caching in DNSJAVA
Posted by Brian Wellington <bw...@xbill.org>.
On Mon, 9 Oct 2006, Stefano Bagnara wrote:
> Brian Wellington wrote:
>> On Sat, 7 Oct 2006, Noel J. Bergman wrote:
>>> Approximate memory as to how much each node consume?
>>
>> It depends on the data, mostly. <snip/>
>>
>> It's easy enough to get a rough measurement of how much 50000 nodes takes
>> with a small test program, but it really depends on the format of the
>> contents for it to be at all useful.
>
> Hi Brian,
>
> first of all, thank you for all your help!
>
> I just checked the dump and I extracted this numbers:
>
> 21384 SOARecord
> 20123 NSRecord
> 17744 ARecord
> 14354 MXRecord
> ----------------
> 73605 Total
>
> I don't know how to calculate them and why they are 73K and not 50K, but if
> my fast dump analysis is ok they are using almost 20MB (depth calc including
> byte data and Names), and this mean an average 280bytes per Record.
The 50K number is the number of nodes (names), not the number of records.
There can be multiple records with either the same or different types in
each node.
> Do this numbers make sense to you or is it better that I check in depth
> "what" is using "how much"?
That seems realistic, given the number of objects needed to represent that
much data.
> And what about the 73000 records against 50000 cache size? Is this because we
> can have multiple records for every node?
Yes.
> I know we can provide our own Cache object as the default Cache, so I started
> thinking about overriding it with a weak/soft cache so that it could not
> cause OOM even with big cache limits, but I see that data (CacheMap) is
> private and CacheMap is also private.
>
> Can I suggest to make some of the Cache private field/method protected or
> extract the Cache interface from the default implementation?
Are you suggesting simply making CacheMap public, and making the data
field accessible? I don't think data should be public, but adding
accessor methods would be ok.
Brian
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: Caching in DNSJAVA
Posted by Stefano Bagnara <ap...@bago.org>.
Brian Wellington wrote:
> On Sat, 7 Oct 2006, Noel J. Bergman wrote:
>> Approximate memory as to how much each node consume?
>
> It depends on the data, mostly. <snip/>
>
> It's easy enough to get a rough measurement of how much 50000 nodes
> takes with a small test program, but it really depends on the format of
> the contents for it to be at all useful.
Hi Brian,
first of all, thank you for all your help!
I just checked the dump and I extracted this numbers:
21384 SOARecord
20123 NSRecord
17744 ARecord
14354 MXRecord
----------------
73605 Total
I don't know how to calculate them and why they are 73K and not 50K, but
if my fast dump analysis is ok they are using almost 20MB (depth calc
including byte data and Names), and this mean an average 280bytes per
Record.
Do this numbers make sense to you or is it better that I check in depth
"what" is using "how much"?
And what about the 73000 records against 50000 cache size? Is this
because we can have multiple records for every node?
I know we can provide our own Cache object as the default Cache, so I
started thinking about overriding it with a weak/soft cache so that it
could not cause OOM even with big cache limits, but I see that data
(CacheMap) is private and CacheMap is also private.
Can I suggest to make some of the Cache private field/method protected
or extract the Cache interface from the default implementation?
Thank you again,
Stefano
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
RE: Caching in DNSJAVA
Posted by Brian Wellington <bw...@xbill.org>.
On Sat, 7 Oct 2006, Noel J. Bergman wrote:
> Brian,
>
>> The caching algorithm in dnsjava should be fairly simple. Calling
>> setMaxEntries() sets the number of DNS nodes (names) in the cache;
>> all information about individual records with the same name is stored
>> in one node. The data structure is a LinkedHashMap with LRU semantics,
>> to ensure that only a certain number of elements are retained.
>
> So does that mean that Stefano's concern:
>
>>> Unfortunately I think that in current DNSServer implementation we
>>> can only tune the "IN" cache and not SOA, PTR and other caches. So
>>> they will be 50000 anyway.
>
> is unfounded, because there is only one cache, which we prepare as:
>
>> cache = new Cache (DClass.IN);
>> cache.setMaxEntries(maxCacheSize);
>> Lookup.setDefaultCache(cache, DClass.IN);
Yes, there's one large (per-class) cache, and it includes data of all
types.
>> I can't think of any reason why the number of elements wouldn't be
>> properly bounded
>
> Neither can I.
>
>> if the problem is that maxCacheSize nodes takes a lot of memory,
>> that's not really fixable without changing the way nodes
>> are stored.
>
> Approximate memory as to how much each node consume?
It depends on the data, mostly. A records will take less space than PTR,
because the A data is stored as an int, and the PTR data is stored as a
Name. There's going to be some per-node overhead: the cost of the
LinkedHashMap entry, the amortized cost of the Map itself, the node's
name/type/class/ttl, etc. There are a number of optimizations to make the
common cases use less memory, but I don't know if they apply to your
cache.
It's easy enough to get a rough measurement of how much 50000 nodes takes
with a small test program, but it really depends on the format of the
contents for it to be at all useful.
Brian
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
RE: Caching in DNSJAVA
Posted by "Noel J. Bergman" <no...@devtech.com>.
Brian,
> The caching algorithm in dnsjava should be fairly simple. Calling
> setMaxEntries() sets the number of DNS nodes (names) in the cache;
> all information about individual records with the same name is stored
> in one node. The data structure is a LinkedHashMap with LRU semantics,
> to ensure that only a certain number of elements are retained.
So does that mean that Stefano's concern:
>> Unfortunately I think that in current DNSServer implementation we
>> can only tune the "IN" cache and not SOA, PTR and other caches. So
>> they will be 50000 anyway.
is unfounded, because there is only one cache, which we prepare as:
> cache = new Cache (DClass.IN);
> cache.setMaxEntries(maxCacheSize);
> Lookup.setDefaultCache(cache, DClass.IN);
> I can't think of any reason why the number of elements wouldn't be
> properly bounded
Neither can I.
> if the problem is that maxCacheSize nodes takes a lot of memory,
> that's not really fixable without changing the way nodes
> are stored.
Approximate memory as to how much each node consume?
--- Noel
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: Caching in DNSJAVA
Posted by Brian Wellington <bw...@xbill.org>.
On Sun, 1 Oct 2006, Noel J. Bergman wrote:
> Stefano wrote:
>> Noel J. Bergman wrote:
>>> I lowered maxcachesize to 5000 from the default of 50000
>>> If nothing else, the smaller cache should help eliminate
>>> that memory consumption from consideration.
>
>> Unfortunately I think that in current DNSServer implementation we
>> can only tune the "IN" cache and not SOA, PTR and other caches. So
>> they will be 50000 anyway.
>
>> I've not checked the code for this, but I'm almost sure I remember
>> this is how it works.
>
> dnsjava is consistently the largest user of memory in my JAMES heap, and
> does not appear to be entirely bounded, although I am having surprising
> difficulty keeping it running when heap dumps are enabled.
>
> Can you comment on the caching behavior inside of dnsjava? We use dnsjava
> via:
>
> http://svn.apache.org/repos/asf/james/server/branches/v2.3/src/java/org/apac
> he/james/dnsserver/DNSServer.java
>
> As you can see, we setup:
>
> cache = new Cache (DClass.IN);
> cache.setMaxEntries(maxCacheSize);
> Lookup.setDefaultCache(cache, DClass.IN);
>
> On a relatively slow day, my server processes 100K+ connections, of which
> anywhere from 45%-70% might be blocked by a DNSRBL.
The caching algorithm in dnsjava should be fairly simple. Calling
setMaxEntries() sets the number of DNS nodes (names) in the cache; all
information about individual records with the same name is stored in one
node. The data structure is a LinkedHashMap with LRU semantics, to ensure
that only a certain number of elements are retained.
I can't think of any reason why the number of elements wouldn't be
properly bounded; the use of the LinkedHashMap is trivial and copied from
its documentation. There's a Cache.getSize() method, which should tell
you how many nodes are in use. It calls the LinkedHashMap size function
directly, so should be accurate
If you can find a way to make the size exceed the maximum size, that
would be interesting, but if the problem is that maxCacheSize nodes takes
a lot of memory, that's not really fixable without changing the way nodes
are stored.
Brian
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org