You are viewing a plain text version of this content. The canonical link for it is here.
Posted to server-dev@james.apache.org by Serge Knystautas <sk...@gmail.com> on 2006/04/25 06:25:26 UTC
DNS xbill heap usage
Every now and then, James grinds to a halt (James 2.2.0, JDK 1.5.0_06,
linux). It ground to a halt this morning, and tests showed it had run
out of heap space (mx set to 256MB).
Unfortunately I couldn't get a snapshot then, but I took one this
evening... after running half a day, James was holding onto 52MB
already. I couldn't figure out what in James could use that much
heap. Guesses?... :)
Yes, the xbill DNS library had used up 44MB of heap space. Now, I use
a lot of spam blacklists, so that might be causing it, but it seems to
me to be a huge room for improving heap space. Is there anyway we can
cap this cache size or ideally send it to disk?
--
Serge Knystautas
Lokitech >> software . strategy . design >> http://www.lokitech.com
p. 301.656.5501
e. sergek@lokitech.com
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: DNS xbill heap usage
Posted by Stefano Bagnara <ap...@bago.org>.
Hi Serge,
AFAIK dnsjava should use weakreferences for its cache. So your
OutOfMemory should not be related to dnsjava itself.
The dnsjava bundled with James 2.2.0 is rather old, but dnsjava2 is not
compatible with older versions.
Maybe that the dnsjava 1.x bundled with James 2.2.0 has memory leaks,
but I don't think so, because I used James 2.2.0 for long time with no
memory issues.
I don't know if dnsjava has tuning parameters or other ways to control
its caching.
Stefano
Serge Knystautas wrote:
> Every now and then, James grinds to a halt (James 2.2.0, JDK 1.5.0_06,
> linux). It ground to a halt this morning, and tests showed it had run
> out of heap space (mx set to 256MB).
>
> Unfortunately I couldn't get a snapshot then, but I took one this
> evening... after running half a day, James was holding onto 52MB
> already. I couldn't figure out what in James could use that much
> heap. Guesses?... :)
>
> Yes, the xbill DNS library had used up 44MB of heap space. Now, I use
> a lot of spam blacklists, so that might be causing it, but it seems to
> me to be a huge room for improving heap space. Is there anyway we can
> cap this cache size or ideally send it to disk?
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: DNS xbill heap usage
Posted by Stefano Bagnara <ap...@bago.org>.
Serge Knystautas wrote:
> The YourKit memory snapshots do not include weak and soft references
> (unless you do something special when you started the JVM), so the
> 44MB I'm seeing allocated are strong references. Also looking at the
> changelog, the CacheCleaner thread in dnsjava uses weak references,
> but the cache itself does not use weak references.
>
> Actually, it gets a bit worse. Out of my 52MB heap that is not
> gc'able, here is what I see:
> - 44MB in an xbill cache held by Lookup code that is held by an avalon
> policy class holder.
> - 3.2MB in an xbill cache held by DNS server
> - 1.7MB in a cache behind java.net.InetAddress
dsnjava 2.0.1 currently caches up to 50000 records for each DNS type.
The number is hardcoded so we cannot change it.
We could set our own caches to dnsjava and size them as we prefer.
We should also remove the cache done in the DNSServer: why should we add
our own cache lookup (on dnsjava objects) when dnsjava already support this?
On the InetAddress side, I see that dnsjava is an SPI for dns lookups.
Adding "sun.net.spi.nameservice.provider.1=dns,dnsjava" we would make
all the dns calls to use dnsjava the way we configured it.
Should we use the spi approach or try to propagate the
DNSServer.staticMethods usage we currenlty use sometimes??
Stefano
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: DNS xbill heap usage
Posted by Alex Karasulu <ao...@bellsouth.net>.
Stefano Bagnara wrote:
> Serge Knystautas wrote:
>> I thought we had already gone the spi approach, or at least had
>> considered it at one point. I thought there was some issue with
>> different results, but I don't remember any specifics.
>
> Don't know.
>
>> In terms of caching, is there a way we can make this configurable,
>> i.e., how does the dnsjava Cache get instantiated? Is it static, or
>> is there a singleton we create somewhere to handle this? If we
>> instantiate the singleton in the DNSServer, we could expose the cache
>> impl or limitations from the dnsserver conf block. I wonder how the
>> spi would instantiate it.
>
> Static methods Lookup.setDefaultCache/getDefaultCache can be used to
> access the singleton Cache object.
> Cache has a setMaxEntries method.
Just an FYI,
Don't know if this is useful so please disregard the noise if the
recommendation is out of place.
You might want to look at using Jdbm for maintaining a single lookup
table backed by a B+Tree as a
disk based cache. It would be ideal for a disk based DNS cache if you
can get the SPI to use it.
Good luck,
Alex
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: DNS xbill heap usage
Posted by Stefano Bagnara <ap...@bago.org>.
Serge Knystautas wrote:
> I thought we had already gone the spi approach, or at least had
> considered it at one point. I thought there was some issue with
> different results, but I don't remember any specifics.
Don't know.
> In terms of caching, is there a way we can make this configurable,
> i.e., how does the dnsjava Cache get instantiated? Is it static, or
> is there a singleton we create somewhere to handle this? If we
> instantiate the singleton in the DNSServer, we could expose the cache
> impl or limitations from the dnsserver conf block. I wonder how the
> spi would instantiate it.
Static methods Lookup.setDefaultCache/getDefaultCache can be used to
access the singleton Cache object.
Cache has a setMaxEntries method.
Stefano
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: DNS xbill heap usage
Posted by Serge Knystautas <sk...@gmail.com>.
On 4/28/06, Stefano Bagnara <ap...@bago.org> wrote:
> dsnjava 2.0.1 currently caches up to 50000 records for each DNS type.
> The number is hardcoded so we cannot change it.
> We could set our own caches to dnsjava and size them as we prefer.
>
> We should also remove the cache done in the DNSServer: why should we add
> our own cache lookup (on dnsjava objects) when dnsjava already support this?
>
> On the InetAddress side, I see that dnsjava is an SPI for dns lookups.
> Adding "sun.net.spi.nameservice.provider.1=dns,dnsjava" we would make
> all the dns calls to use dnsjava the way we configured it.
>
> Should we use the spi approach or try to propagate the
> DNSServer.staticMethods usage we currenlty use sometimes??
I thought we had already gone the spi approach, or at least had
considered it at one point. I thought there was some issue with
different results, but I don't remember any specifics.
In terms of caching, is there a way we can make this configurable,
i.e., how does the dnsjava Cache get instantiated? Is it static, or
is there a singleton we create somewhere to handle this? If we
instantiate the singleton in the DNSServer, we could expose the cache
impl or limitations from the dnsserver conf block. I wonder how the
spi would instantiate it.
--
Serge Knystautas
Lokitech >> software . strategy . design >> http://www.lokitech.com
p. 301.656.5501
e. sergek@lokitech.com
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: DNS xbill heap usage
Posted by Serge Knystautas <sk...@gmail.com>.
On 4/25/06, Stefano Bagnara <ap...@bago.org> wrote:
> AFAIK dnsjava should use weakreferences for its cache. So your
> OutOfMemory should not be related to dnsjava itself.
The YourKit memory snapshots do not include weak and soft references
(unless you do something special when you started the JVM), so the
44MB I'm seeing allocated are strong references. Also looking at the
changelog, the CacheCleaner thread in dnsjava uses weak references,
but the cache itself does not use weak references.
Actually, it gets a bit worse. Out of my 52MB heap that is not
gc'able, here is what I see:
- 44MB in an xbill cache held by Lookup code that is held by an avalon
policy class holder.
- 3.2MB in an xbill cache held by DNS server
- 1.7MB in a cache behind java.net.InetAddress
So 49MB or 52MB heap that is not gc'able are from DNS cache entries. :)
Here's a link to some HTML reports:
http://www.lokitech.com/~sergek/james/dnsleak/
--
Serge Knystautas
Lokitech >> software . strategy . design >> http://www.lokitech.com
p. 301.656.5501
e. sergek@lokitech.com
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: DNS xbill heap usage
Posted by Stefano Bagnara <ap...@bago.org>.
Serge Knystautas wrote:
>> the case. Personally, I'd prefer to stick with a dnsjava, and can agree
>> with Stefano that we could expose the DNSServer service and use it,
>> rather
>> than call the static methods in dnsjava directly.
>
> I don't see how we could drop use of dnsjava. The SPI only allows
> standard hostname to IP address lookups to be handled by a 3rd party
> library. The whole reason we got dnsjava in the first place is to be
> able to query MX records, and AFAIK, we still cannot do that. Maybe
> JNDI, but I didn't think so.
We don't want to drop the use of dnsjava.
The proposal was only to avoid static calls to DNSServer by replacing it
with dependencies on the DNSServer component and non-static calls.
> Also, AFAIK SPF information in the TXT record, so dnsjava should have
> no trouble retrieving that record type. jSPF would largely just be
> parsing of that TXT record into the appropriate rule set, no? If it
> intends to do more, it would almost have to depend on dnsjava to be
> able to get those TXT records, I would think.
An SPF type has been approved, too.
In jspf we currently support the TXT version because dnsjava does not
handle the SPF record type (99).
Stefano
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: DNS xbill heap usage
Posted by Serge Knystautas <sk...@gmail.com>.
On 4/30/06, Stefano Bagnara <ap...@bago.org> wrote:
> We don't want to drop the use of dnsjava.
> The proposal was only to avoid static calls to DNSServer by replacing it
> with dependencies on the DNSServer component and non-static calls.
<snip />
> An SPF type has been approved, too.
> In jspf we currently support the TXT version because dnsjava does not
> handle the SPF record type (99).
Great, thanks for clarifying both these points!
--
Serge Knystautas
Lokitech >> software . strategy . design >> http://www.lokitech.com
p. 301.656.5501
e. sergek@lokitech.com
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: DNS xbill heap usage
Posted by Norman Maurer <nm...@byteaction.de>.
Am Sonntag, den 30.04.2006, 10:44 -0400 schrieb Serge Knystautas:
> On 4/30/06, Stefano Bagnara <ap...@bago.org> wrote:
> > We don't want to drop the use of dnsjava.
> > The proposal was only to avoid static calls to DNSServer by replacing it
> > with dependencies on the DNSServer component and non-static calls.
>
> <snip />
>
> > An SPF type has been approved, too.
> > In jspf we currently support the TXT version because dnsjava does not
> > handle the SPF record type (99).
>
> Great, thanks for clarifying both these points!
BTW, the SPF type is not supported yet by most DNSServers. Bind also
miss this record.
bye
Re: DNS xbill heap usage
Posted by Serge Knystautas <sk...@gmail.com>.
On 4/29/06, Noel J. Bergman <no...@devtech.com> wrote:
> Yes. Please see the patch that I just committed. Are you in a position to
> test the result in-situ?
Thanks for fixing this Noel! I cannot readily test because my "test"
is running it in production and seeing thousands of RBL cache entries
accumulate for spammers and zombies. :)
> > or ideally send it to disk?
>
> Actually, I suspect that from a performance perspective, this might be
> counter-productive. Let your local DNS servers provide additional caching,
> rather than get involved in disk I/O.
I would prefer to have dnsjava lib abstract the cache impl. That way
for 90% of the users out there, they could use the basic bundled impl
they have, or for the advanced users like the James project, we could
configure an oscache or jdbm (have to look into that) impl.
> Should see where this is coming from, since we should generally use
> InetAddress as little as possible, due to its naive implementation.
> Actually, I seem to recall running into some issues with the standard Java
> code sticking its figurative nose where it doesn't belong, and not giving us
> a clean chain down to the DNS SPI. We would have to see if that is (still)
> the case. Personally, I'd prefer to stick with a dnsjava, and can agree
> with Stefano that we could expose the DNSServer service and use it, rather
> than call the static methods in dnsjava directly.
I don't see how we could drop use of dnsjava. The SPI only allows
standard hostname to IP address lookups to be handled by a 3rd party
library. The whole reason we got dnsjava in the first place is to be
able to query MX records, and AFAIK, we still cannot do that. Maybe
JNDI, but I didn't think so.
Anyway, I think there were some optional lookups we converted from
InetAddress to dnsjava, but that would just to deal with the sun
leak... InetAddress always caches everything, meaning you're stuck
with a timed out dns entries (not to mention cache heap space used).
My memory is similarly foggy on whether those got in and the small bug
we saw.
Also, AFAIK SPF information in the TXT record, so dnsjava should have
no trouble retrieving that record type. jSPF would largely just be
parsing of that TXT record into the appropriate rule set, no? If it
intends to do more, it would almost have to depend on dnsjava to be
able to get those TXT records, I would think.
--
Serge Knystautas
Lokitech >> software . strategy . design >> http://www.lokitech.com
p. 301.656.5501
e. sergek@lokitech.com
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: DNS xbill heap usage
Posted by Stefano Bagnara <ap...@bago.org>.
In the dnsjava changelog I see this after the 1.6.2 release we bundle
with James 2.2.0:
4/3/2005 - Change the Cache from being periodically cleaned to being
size-bounded.
This change is part of dnsjava-2.0+. Maybe that james 2.3.0 will fix
this problem.
Otherwise you could try rebuilding James 2.2.0 with dnsjava2. I remember
I changed the DNSServer to do this upgrade but it should be an easy task.
Stefano
Serge Knystautas wrote:
> On 4/25/06, Serge Knystautas <sk...@gmail.com> wrote:
>> Actually, it gets a bit worse. Out of my 52MB heap that is not
>> gc'able, here is what I see:
>> - 44MB in an xbill cache held by Lookup code that is held by an avalon
>> policy class holder.
>> - 3.2MB in an xbill cache held by DNS server
>> - 1.7MB in a cache behind java.net.InetAddress
>>
>> So 49MB or 52MB heap that is not gc'able are from DNS cache entries. :)
>
> Just to keep score, I did a snapshot now and had the following stats:
> - 87.3MB total heap used
> - 75.6MB DNS.Lookup via that policy class holder.
> - 5.4MB DNS.Lookup via the James DNSServer
> - 2.7MB via java.net.InetAddress
>
> On the bright side is that non-DNS caches in James are still using
> only a tiny amount of heap (3.6MB or about 1/25th of the total heap
> space and 1% of what I had allocated).
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: DNS xbill heap usage
Posted by Serge Knystautas <sk...@gmail.com>.
On 4/25/06, Serge Knystautas <sk...@gmail.com> wrote:
> Actually, it gets a bit worse. Out of my 52MB heap that is not
> gc'able, here is what I see:
> - 44MB in an xbill cache held by Lookup code that is held by an avalon
> policy class holder.
> - 3.2MB in an xbill cache held by DNS server
> - 1.7MB in a cache behind java.net.InetAddress
>
> So 49MB or 52MB heap that is not gc'able are from DNS cache entries. :)
Just to keep score, I did a snapshot now and had the following stats:
- 87.3MB total heap used
- 75.6MB DNS.Lookup via that policy class holder.
- 5.4MB DNS.Lookup via the James DNSServer
- 2.7MB via java.net.InetAddress
On the bright side is that non-DNS caches in James are still using
only a tiny amount of heap (3.6MB or about 1/25th of the total heap
space and 1% of what I had allocated).
--
Serge Knystautas
Lokitech >> software . strategy . design >> http://www.lokitech.com
p. 301.656.5501
e. sergek@lokitech.com
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
RE: DNS xbill heap usage
Posted by "Noel J. Bergman" <no...@devtech.com>.
> IIRC we discussed it and agreed that 2.3.0 will be 1.4+ only.
Yes. I'm just pointing out that dnsjava is another enforcer for the change.
--- Noel
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: DNS xbill heap usage
Posted by Stefano Bagnara <ap...@bago.org>.
Noel J. Bergman wrote:
> Actually, not. dnsjava uses a subclass of java.util.LinkedHashMap for its
> cache. Which, by the way, requires Java 1.4+, so we have dropped Java 1.3
> compatibility. Not that I consider that under debate nor dispute; just
> pointing it out.
We already did this when we removed the SMIME compatibility libraries
for java 1.3 ;-).
IIRC we discussed it and agreed that 2.3.0 will be 1.4+ only.
In fact James website says that 2.2.0 requires 1.4, although we know it
works with 1.3.
Stefano
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
RE: DNS xbill heap usage
Posted by "Noel J. Bergman" <no...@devtech.com>.
Collating from the entire thread ...
Serge Knystautas wrote:
> Is there anyway we can cap this cache size
Yes. Please see the patch that I just committed. Are you in a position to
test the result in-situ?
> or ideally send it to disk?
Actually, I suspect that from a performance perspective, this might be
counter-productive. Let your local DNS servers provide additional caching,
rather than get involved in disk I/O.
Stefano Bagnara wrote:
> AFAIK dnsjava should use weakreferences for its cache
Actually, not. dnsjava uses a subclass of java.util.LinkedHashMap for its
cache. Which, by the way, requires Java 1.4+, so we have dropped Java 1.3
compatibility. Not that I consider that under debate nor dispute; just
pointing it out.
Serge added:
> > - 44MB in an xbill cache held by Lookup code that is held
> > by an avalon policy class holder.
> - 87.3MB total heap used
> - 75.6MB DNS.Lookup via that policy class holder.
Ok, so this is clearly the big one. The cause is that Lookup defaults to
maintaining its own cache, which is not shared with the one that we use
internally, even though we share the same resolver. I'll fix that, which
will give us more control.
> > - 3.2MB in an xbill cache held by DNS server
> - 5.4MB DNS.Lookup via the James DNSServer
This is the one we intend to be using. The space doesn't seem all that bad,
but is just for the lookups going through DNSServer. All of the ones that
directly involved Lookup were going to a separate, unmanaged (by us) cache.
> > - 1.7MB in a cache behind java.net.InetAddress
> - 2.7MB via java.net.InetAddress
Should see where this is coming from, since we should generally use
InetAddress as little as possible, due to its naive implementation.
Stefano added:
> We could set our own caches to dnsjava and size them as we prefer.
Yes.
> We should also remove the cache done in the DNSServer: why should we add
> our own cache lookup (on dnsjava objects) when dnsjava already support
> this?
Keep in mind that we don't call Lookup in DNSServer. We call the resolver
directly. Therefore, there is no caching unless we implement it.
Steve Brewin suggested:
> It would be good to revert to the SPI approach for V3,
> but for V2 its too late and too hard to gauge how many
> workarounds would need to be undone.
Actually, I seem to recall running into some issues with the standard Java
code sticking its figurative nose where it doesn't belong, and not giving us
a clean chain down to the DNS SPI. We would have to see if that is (still)
the case. Personally, I'd prefer to stick with a dnsjava, and can agree
with Stefano that we could expose the DNSServer service and use it, rather
than call the static methods in dnsjava directly.
--- Noel
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org