You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Jordan Michaels <jo...@viviotech.net> on 2010/12/03 19:39:25 UTC
SOT: JVM and DNS Failover
Hey Folks,
I apologize for the slightly off-topic post but I'm having trouble
researching this issue so I figured it might be worthwhile to make a
post where folks more familiar with the intricacies of the JVM might be
able to point me in the right direction.
We have an application running on top of Tomcat. Normally, the
application runs great, but recently one of the DNS servers that the OS
uses has been having trouble, and dies. From an OS point of view, it's
not a problem. The OS simply queries the secondary resolver instead and
goes happily on it's way.
However, the JVM doesn't seem to be doing that. Whenever this name
server goes offline, the application dies because something it's doing
(we didn't write it) needs to resolve in order for the site to load.
This makes it so that whenever this single resolver goes down, the
entire site goes down. It's incredibly frustrating because it should be
failing over to the secondary resolver just like the OS does and things
would be fine.
Would any of you be able to point me to some documentation on how the
JVM handles DNS resolving? I'm hoping there are JVM settings that can be
tweaked to help force the JVM to fail over to the secondary resolver.
I'm using version 1.6 JVM.
Any pointers anyone could offer on this would be most appreciated.
Thank you in advance!
-Jordan
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: SOT: JVM and DNS Failover
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Jordan,
On 12/3/2010 2:56 PM, Jordan Michaels wrote:
> It's what makes me think that the...
>
> sun.net.client.defaultConnectTimeout (default: -1)
> sun.net.client.defaultReadTimeout (default: -1)
>
> settings are involved somehow...
- From the docs
(http://download.oracle.com/javase/1.5.0/docs/guide/net/properties.html):
"
These properties specify the default connect and read timeout (resp.)
for the protocol handler used by java.net.URLConnection.
"
This has absolutely nothing at all to do with DNS resolution.
Is the problem that a hostname (somehow) resolves to NXDOMAIN and /that/
value is being cached? I'm pretty sure that the JVM delegates to the OS
to do DNS resolution, so the caching would only be a problem if a DNS
record /changed/ during the lifetime of the JVM.
IIRC, DNS shouldn't cache NXDOMAIN in general. You could easily try
setting the ttl configs mentioned in other posts to be 0 (zero: do not
cache) and see what happens.
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAkz5U2MACgkQ9CaO5/Lv0PBUvgCgg4dl9V9ybFW2c+KB4v3yoDAX
SvkAn3wqxUJbwj3r+PYi4Zrwrb3WJiD5
=S+sQ
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: SOT: JVM and DNS Failover
Posted by Jordan Michaels <jo...@viviotech.net>.
On 12/03/2010 11:31 AM, Michael Ludwig wrote:
> Moin Jordan,
>
> Jordan Michaels schrieb am 03.12.2010 um 10:39 (-0800):
>
>> Would any of you be able to point me to some documentation on how
>> the JVM handles DNS resolving? I'm hoping there are JVM settings
>> that can be tweaked to help force the JVM to fail over to the
>> secondary resolver.
>>
>> I'm using version 1.6 JVM.
>
> Networking Properties
> http://download.oracle.com/javase/6/docs/technotes/guides/net/properties.html#nct
>
> networkaddress.cache.ttl
>
> […] A value of -1 indicates "cache forever". The default behavior
> is to cache forever when a security manager is installed, and to
> cache for an implementation specific period of time, when a
> security manager is not installed.
>
> So have you installed a security manager?
>
> Wondering myself what the default value is?
>
> sun.net.inetaddr.ttl
>
> This is a sun private system property which corresponds to
> networkaddress.cache.ttl. It takes the same value and has the
> same meaning, but can be set as a command-line option. However,
> the preferred way is to use the security property mentioned
> above.
>
> Still wondering. So is the "implementation specific period of time"
> the value taken from the OS?
>
The Tomcat and JVM installs are very close to vanilla installs. We've
added a few classes to Tomcat and changed the JVM settings a little
(java_opts) but nothing that would effect any of these settings here.
The weird thing is that you'd think if caching was on, the site wouldn't
immediately die when the resolver died, but that's what's happening.
When you hit the site when the resolver is dead, it just hangs... a
white screen with no errors... like it's just waiting.
It's what makes me think that the...
sun.net.client.defaultConnectTimeout (default: -1)
sun.net.client.defaultReadTimeout (default: -1)
settings are involved somehow... I just don't know if those settings are
used for DNS requests as well as the HTTP and FTP connections.
You can still SSH to the OS (CentOS 5) and the OS does what it's
supposed to, so the JVM isn't doing it through the OS, that's for sure.
-Jordan
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: SOT: JVM and DNS Failover
Posted by Michael Ludwig <mi...@gmx.de>.
Michael Ludwig schrieb am 03.12.2010 um 20:51 (+0100):
> Michael Ludwig schrieb am 03.12.2010 um 20:31 (+0100):
>
> > networkaddress.cache.ttl
>
> > Wondering myself what the default value is?
> >
> > sun.net.inetaddr.ttl
>
> > So is the "implementation specific period of time" the value taken
> > from the OS?
>
> I think this rather means that it's up to the JVM vendor's gusto.
>
> In the absence of a security manger, the value can be tweaked here:
>
> C:\Programme\Java\jre6\lib\security\java.security
The comments in that file (at least in my copy of it) really leave you
wondering whether in the absence of a security manager, the default
value is FOREVER or 30 seconds. And whether you can tweak it when
running under a security manager.
\,,,/
(o o)
------oOOo-(_)-oOOo------
# The Java-level namelookup cache policy for successful lookups:
#
# any negative value: caching forever
# any positive value: the number of seconds to cache an address for
# zero: do not cache
#
# default value is forever (FOREVER). For security reasons, this
# caching is made forever when a security manager is set. When a security
# manager is not set, the default behavior is to cache for 30 seconds.
#
# NOTE: setting this to anything other than the default value can have
# serious security implications. Do not set it unless
# you are sure you are not exposed to DNS spoofing attack.
#
#networkaddress.cache.ttl=-1
--
Michael Ludwig
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: SOT: JVM and DNS Failover
Posted by Michael Ludwig <mi...@gmx.de>.
Michael Ludwig schrieb am 03.12.2010 um 20:31 (+0100):
> Networking Properties
> http://download.oracle.com/javase/6/docs/technotes/guides/net/properties.html#nct
>
> networkaddress.cache.ttl
> Wondering myself what the default value is?
>
> sun.net.inetaddr.ttl
> So is the "implementation specific period of time" the value taken
> from the OS?
I think this rather means that it's up to the JVM vendor's gusto.
In the absence of a security manger, the value can be tweaked here:
C:\Programme\Java\jre6\lib\security\java.security
Michael
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: SOT: JVM and DNS Failover
Posted by Michael Ludwig <mi...@gmx.de>.
Moin Jordan,
Jordan Michaels schrieb am 03.12.2010 um 10:39 (-0800):
> Would any of you be able to point me to some documentation on how
> the JVM handles DNS resolving? I'm hoping there are JVM settings
> that can be tweaked to help force the JVM to fail over to the
> secondary resolver.
>
> I'm using version 1.6 JVM.
Networking Properties
http://download.oracle.com/javase/6/docs/technotes/guides/net/properties.html#nct
networkaddress.cache.ttl
[…] A value of -1 indicates "cache forever". The default behavior
is to cache forever when a security manager is installed, and to
cache for an implementation specific period of time, when a
security manager is not installed.
So have you installed a security manager?
Wondering myself what the default value is?
sun.net.inetaddr.ttl
This is a sun private system property which corresponds to
networkaddress.cache.ttl. It takes the same value and has the
same meaning, but can be set as a command-line option. However,
the preferred way is to use the security property mentioned
above.
Still wondering. So is the "implementation specific period of time"
the value taken from the OS?
--
Michael Ludwig
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: JVM and DNS Failover
Posted by Jordan Michaels <jo...@viviotech.net>.
On 12/03/2010 11:11 AM, Caldarale, Charles R wrote:
>> From: Jordan Michaels [mailto:jordan@viviotech.net]
>> Subject: SOT: JVM and DNS Failover
>
>> However, the JVM doesn't seem to be doing that. Whenever this name
>> server goes offline, the application dies because something it's doing
>> (we didn't write it) needs to resolve in order for the site to load.
>
> Not sure if this is directly related, but it might point to some things to look into:
> http://java-monitor.com/forum/showthread.php?t=181
>
> However, if the app is doing its own DNS lookups (not sure how it would), the JVM isn't involved.
>
> - Chuck
Thanks Chuck.
I have been looking at those settings as well.
On this page:
http://download.oracle.com/javase/6/docs/technotes/guides/net/properties.html
Under "Sun implementation-specific properties", there are a couple
settings which seem like they might be likely culprits:
sun.net.client.defaultConnectTimeout (default: -1)
sun.net.client.defaultReadTimeout (default: -1)
The docs mention HTTP and FTP connections, but I wonder if it could
apply to DNS connections as well? And... if the default timeout is
"forever"... then maybe this could explain why it's not failing over?
-Jordan
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: JVM and DNS Failover
Posted by "Caldarale, Charles R" <Ch...@unisys.com>.
> From: Jordan Michaels [mailto:jordan@viviotech.net]
> Subject: SOT: JVM and DNS Failover
> However, the JVM doesn't seem to be doing that. Whenever this name
> server goes offline, the application dies because something it's doing
> (we didn't write it) needs to resolve in order for the site to load.
Not sure if this is directly related, but it might point to some things to look into:
http://java-monitor.com/forum/showthread.php?t=181
However, if the app is doing its own DNS lookups (not sure how it would), the JVM isn't involved.
- Chuck
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org