You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by הילה <hi...@gmail.com> on 2011/05/17 14:59:08 UTC
Application crash after Migrate to different ESX
Hey Guys,
My application runs on tomcat 6.0.29, win 2008 R2, which is a VM machine.
i use the JDBC application pool., with java 1.6.24 64bit.
I have a problem, in which after migrate of the VM machine to another ESX,
the application crashes, and doesn't recover by itself.
I know that after migrate, a disconnection is expected, but in my case - it
crashes completely, or alternately until it works (but don't know if it's
stable now).
when it's not working, i see via netstat a time_wait of the connection to
the DB.
this is what displayed in the log-
2011-05-17 11:57:49,959 ERROR [Request-Processor3]
org.hibernate.transaction.JDBCTransaction - *JDBC begin failed*
* *
*java.sql.SQLException: Invalid state, the Connection object is closed.*
at
net.sourceforge.jtds.jdbc.ConnectionJDBC2.checkOpen(ConnectionJDBC2.java:1699)
at
net.sourceforge.jtds.jdbc.ConnectionJDBC2.getAutoCommit(ConnectionJDBC2.java:2123)
at sun.reflect.GeneratedMethodAccessor93.invoke(Unknown
Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.tomcat.jdbc.pool.ProxyConnection.invoke(ProxyConnection.java:125)
at $Proxy0.getAutoCommit(Unknown Source)
at
org.hibernate.transaction.JDBCTransaction.begin(JDBCTransaction.java:49)
at
org.hibernate.transaction.JDBCTransactionFactory.beginTransaction(JDBCTransactionFactory.java:24)
at
org.hibernate.jdbc.JDBCContext.beginTransaction(JDBCContext.java:271)
at
org.hibernate.impl.SessionImpl.beginTransaction(SessionImpl.java:1079)
at
com.vsgaming.app.appContext.getDatabaseConnection(appContext.java:296)
at
com.vsgaming.system.SystemMethods.health(SystemMethods.java:383)
at
com.vsgaming.system.SystemMethods.processRequest(SystemMethods.java:146)
at
com.vsgaming.app.appServlet$RequestProcessor.processStandardCompositeRequest(appServlet.java:1547)
at
com.vsgaming.app.appServlet$RequestProcessor.processLoadBalancedCompositeRequest(appServlet.java:1727)
at
com.vsgaming.edge.appServlet$RequestProcessor.processCompositeRequest(appServlet.java:1449)
at
com.vsgaming.app.appServlet$RequestProcessor.processRequestThread(appServlet.java:1372)
at
com.vsgaming.app.appServlet$RequestProcessor.run(appServlet.java:1338)
at java.lang.Thread.run(Thread.java:662)
I also noticed that servers that were migrated that has the APR native dll
installed, reacted better than the one without the APR native dll (I don't
know if it's related, but this is the only difference between the servers
that I could think of).
has someone experienced this issue?
how can I solve it?
Thanks
Hila
Re: Application crash after Migrate to different ESX
Posted by André Warnier <aw...@ice-sa.com>.
הילה wrote:
> UPDATE-
>
> this behavior occur even when I'm disconnecting the Network card from the VM
> settings and connecting it again.
> afterwards, the keep alive is alternately unavailable. looks like it's not
> always trying to approach the DB opened wireshark and saw that ip.addr==DB
> IP ADDRESS is not always showing results when the keep alive is unavailble =
> he's not trying to get to the DB at all, I think.
>
> it all sorts out after I restart the tomcat service. this is why I think it
> does related to tomcat.
>
Hi again.
First, nobody denies that you are seeing a problem, at some level.
But just as your expertise is probably at the application level, and the expertise of your
in-house DBA team is at the database administration level, well the expertise of the
people on this list is mostly at the Tomcat and java level, and that is the level at which
this list may be able to help you resolve the problem, if the problem is at the Tomcat level.
In that sense, the fact that when you restart Tomcat (or rather, the JVM which runs
Tomcat) the problem goes away, does not mean that it is "Tomcat's fault". It just means
that when you restart Tomcat, everything starting with your application and all the way
down to the database network connections get reset and restarted from scratch. No wonder
everything works fine then of course.
Similarly, when you unplug and replug the network cable, everyone here would expect that
some network connections may be lost in the event, and that this may cause problems to
your application.
The issue here consists of determining what part, in the stack of pieces of code which are
involved, should detect this loss of connection, and try to recover from it.
And if it is really a part of Tomcat which should do that, and it does not, then people
here may be able to help, to the best of their abilities and (donated) free time.
And if it is not a part of Tomcat which should do that, but another piece of code, then
the people here can only tell you to go to the mailing list of that other piece of code
for help.
I am not a big Tomcat or java expert, so I am trying to look at this issue "top-down", in
a generic way, in the hope of helping to narrow down the issue and maybe save some time
for the real Tomcat/java experts here.
In your original stack trace, the only point where I see a Tomcat class involved is this one :
at
org.apache.tomcat.jdbc.pool.ProxyConnection.invoke(ProxyConnection.java:125)
Above that (in the stack), and apart from your own application, there are only Hibernate
methods (a separate Apache project, as far as I know not part of Tomcat); and below that
(in the stack), there are mostly jtds classes (a separate, non-Apache project).
So, from a naive point of view, what makes you think that when for some reason (good or
bad) a network connection to the underlying database system gets lost or becomes unstable,
it is the job of that single lone Tomcat class to solve the problem ?
I am not saying that it is not maybe a problem of that one Tomcat class; but what I am
asking is what makes you so sure that it is ?
On a separate level, you have still not really explained what your "smart keep alive" is,
or at what level it happens, or where that code comes from.
Except for this, which does not say much in terms of usable information :
"smart keep alive is a keep alive in its ordinary meaning. it called smart
cause it's not checking only the port and its availability, but also a
connection to the DB and functionality via DB query of some sort."
So at what level is that code running, and where does it come from ?
And can it be, in the logic of your hardware/software infrastructure, that it is that
piece of code which should handle a temporary DB connection loss, and recover from it ?
Or should it be the jtds driver code ?
Or should it really be that org.apache.tomcat.jdbc.pool.ProxyConnection Tomcat class ?
What do you think ?
(I personally don't know. I am just trying to help you narrowing dowen the problem, and
maybe explaining to the Tomcat experts here why you think that they could be helpful).
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Application crash after Migrate to different ESX
Posted by הילה <hi...@gmail.com>.
Excellent. thanks a lot for all of your help :)
2011/6/7 Christopher Schultz <ch...@christopherschultz.net>
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> הילה,
>
> On 6/6/2011 4:35 PM, הילה wrote:
> > I use Tomcat's JDBC pool (tomcat-jdbc.jar)
>
> Okay.
>
> > sorry, I haven't quite understood why shouldn't I use the
> > ValidationInterval="30000" (30 seconds) string on my configuration..
> > On tomcat's documentation, it says that testonborrow=true and
> > validationQuery=SELECT 1 were part of the DBCP , and the
> > ValidationInterval=30000 is introduced in the Tomcat's JDBC pool.
>
> DBCP implements a different kind of connection validation by running a
> background thread. tomcat-pool has different semantics that do not
> launch a background thread.
>
> Feel free to use validationInterval set to whatever value you'd like.
>
> - -chris
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.10 (MingW32)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>
> iEUEARECAAYFAk3ufjUACgkQ9CaO5/Lv0PBwTACYvwblavTu/2YARBtPr/ILogXo
> NgCeIyhyJMD1TF8yUjUoHGlzW0WhZN8=
> =oqS5
> -----END PGP SIGNATURE-----
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
Re: Application crash after Migrate to different ESX
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
הילה,
On 6/6/2011 4:35 PM, הילה wrote:
> I use Tomcat's JDBC pool (tomcat-jdbc.jar)
Okay.
> sorry, I haven't quite understood why shouldn't I use the
> ValidationInterval="30000" (30 seconds) string on my configuration..
> On tomcat's documentation, it says that testonborrow=true and
> validationQuery=SELECT 1 were part of the DBCP , and the
> ValidationInterval=30000 is introduced in the Tomcat's JDBC pool.
DBCP implements a different kind of connection validation by running a
background thread. tomcat-pool has different semantics that do not
launch a background thread.
Feel free to use validationInterval set to whatever value you'd like.
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEUEARECAAYFAk3ufjUACgkQ9CaO5/Lv0PBwTACYvwblavTu/2YARBtPr/ILogXo
NgCeIyhyJMD1TF8yUjUoHGlzW0WhZN8=
=oqS5
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Application crash after Migrate to different ESX
Posted by הילה <hi...@gmail.com>.
Hey,
I use Tomcat's JDBC pool (tomcat-jdbc.jar)
sorry, I haven't quite understood why shouldn't I use the
ValidationInterval="30000" (30 seconds) string on my configuration..
On tomcat's documentation, it says that testonborrow=true and
validationQuery=SELECT 1 were part of the DBCP , and the
ValidationInterval=30000 is introduced in the Tomcat's JDBC pool.
Thanks
Hils
2011/6/6 Christopher Schultz <ch...@christopherschultz.net>
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> הילה,
>
> On 6/6/2011 9:56 AM, הילה wrote:
> > I've set testonborrow, validationquery and validationinterval - and it
> did
> > the trick. After ESX migration or just disconnect and reconnect of the
> > network card, the application recovers by itself.
>
> Note that "validationInterval" is not a valid configuration option (if
> you're talking about DBCP... sorry, I can't keep track of your
> configuration... are you using DBCP or Tomcat's new, semi-experimental
> pool?).
>
> If you're using DBCP, you should really read this page in it's entirety:
>
> http://commons.apache.org/dbcp/configuration.html
>
> If you are using Tomcat's pool, you should read the documentation that
> comes with it.
>
> Any type of "interval" setting is likely to create a background thread
> that monitors the connections and tests them periodically.
>
> Unless you have a very strict response-time metric that you are trying
> to hit, I would stick with testOnBorrow="true" (along with an innocuous
> validationQuery) and leave everything else alone.
>
> - -chris
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.10 (MingW32)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>
> iEYEARECAAYFAk3s5cgACgkQ9CaO5/Lv0PCLgQCfTA/5t1Rr2uJegTtZf+btaJgC
> otMAn0ixB0LEoO0TSBRAUF4QE9Ssv1+S
> =b247
> -----END PGP SIGNATURE-----
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
Re: Application crash after Migrate to different ESX
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
הילה,
On 6/6/2011 9:56 AM, הילה wrote:
> I've set testonborrow, validationquery and validationinterval - and it did
> the trick. After ESX migration or just disconnect and reconnect of the
> network card, the application recovers by itself.
Note that "validationInterval" is not a valid configuration option (if
you're talking about DBCP... sorry, I can't keep track of your
configuration... are you using DBCP or Tomcat's new, semi-experimental
pool?).
If you're using DBCP, you should really read this page in it's entirety:
http://commons.apache.org/dbcp/configuration.html
If you are using Tomcat's pool, you should read the documentation that
comes with it.
Any type of "interval" setting is likely to create a background thread
that monitors the connections and tests them periodically.
Unless you have a very strict response-time metric that you are trying
to hit, I would stick with testOnBorrow="true" (along with an innocuous
validationQuery) and leave everything else alone.
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAk3s5cgACgkQ9CaO5/Lv0PCLgQCfTA/5t1Rr2uJegTtZf+btaJgC
otMAn0ixB0LEoO0TSBRAUF4QE9Ssv1+S
=b247
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Application crash after Migrate to different ESX
Posted by הילה <hi...@gmail.com>.
Thanks :]
I've set testonborrow, validationquery and validationinterval - and it did
the trick. After ESX migration or just disconnect and reconnect of the
network card, the application recovers by itself.
Thanks
Hila
2011/6/6 Christopher Schultz <ch...@christopherschultz.net>
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> הילה,
>
> On 6/1/2011 5:14 PM, הילה wrote:
> > when using the string
> > testonborrow="true"
> > do I need to use it alone?
>
> It is not effective alone. In fact, testOnBorrow="true" is the default,
> oddly enough.
>
> http://commons.apache.org/dbcp/configuration.html
>
> > can it be used alone? or do I have to put also
> > the validationInterval string (or any other string)?
>
> As per the documentation:
>
> "
> NOTE - for a true value to have any effect, the validationQuery
> parameter must be set to a non-null string.
> "
>
> - -chris
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.10 (MingW32)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>
> iEYEARECAAYFAk3s23oACgkQ9CaO5/Lv0PCcVgCgjxvBdBIsdXW8BIyQk6MVEiNE
> 1xAAn2sqI2dOBfA/sLE8LCxryjzs9FFB
> =2mpx
> -----END PGP SIGNATURE-----
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
Re: Application crash after Migrate to different ESX
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
הילה,
On 6/1/2011 5:14 PM, הילה wrote:
> when using the string
> testonborrow="true"
> do I need to use it alone?
It is not effective alone. In fact, testOnBorrow="true" is the default,
oddly enough.
http://commons.apache.org/dbcp/configuration.html
> can it be used alone? or do I have to put also
> the validationInterval string (or any other string)?
As per the documentation:
"
NOTE - for a true value to have any effect, the validationQuery
parameter must be set to a non-null string.
"
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAk3s23oACgkQ9CaO5/Lv0PCcVgCgjxvBdBIsdXW8BIyQk6MVEiNE
1xAAn2sqI2dOBfA/sLE8LCxryjzs9FFB
=2mpx
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Application crash after Migrate to different ESX
Posted by הילה <hi...@gmail.com>.
Hey,
Explaining the whole infrastructure may be even more confusing.
so I'll ask something else, simpler-
when using the string
testonborrow="true"
do I need to use it alone? can it be used alone? or do I have to put also
the validationInterval string (or any other string)?
Thanks
Hila
2011/5/20 André Warnier <aw...@ice-sa.com>
> הילה wrote:
> ...
>
>
>> So, if you are using that pool, I basically do not understand why you
>>> would
>>> need any additional mechanism to overcome the loss of a db connection
>>> when
>>> your VM is migrated.
>>> Particularly that "smart keep alive" you keep talking about, but never
>>> telling us where it comes from and where it is inserted in that
>>> architecture.
>>>
>>> I don't understand on what mechanism are you talking about. I don't want
>>> to
>>>
>> add anything (unless it can fix the problem)
>> I just stated the problem, and the environment we have (win 2008, tomcat
>> 6.0.29, jdbc pool.. These are the dry facts)
>>
>> The keep alive is an xslt file which contains actions to perform on the DB
>> to check that everything is alive.
>> if so, it returns an OK response, which can be viewed in HTML file on IE
>> browser.
>> The Load balancer samples the keep alive every 10 seconds to check the OK
>> state. if it's not OK for 3 times in a row- the LB takes it out from the
>> servers' pool and no one else can connect to it.
>>
>
> So now there is also a load balancer ? I've just gone through all your
> previous posts, and this is the first time it has been mentioned.
> And it appears that it is the load balancer which tests de DB server
> directly (?)
> Huh ? I'm like Chris now, just a bit confused again about your setup.
>
>
> so yes. we need this keep alive and can't dismiss it, because it's the
>> indication that the server is functioning properly.
>>
>
> Which server ?
>
>
> and if it's not function
>
>> - The LB can identify it and remove the server from the servers' pool so
>> no
>> one will try and approach it.
>>
>>
> Again, which server is taken out of the server pool ? The DB server, or
> the one running Tomcat ?
>
> Do you actually mean that the load balancer on one side, and Tomcat on the
> other side, are each accessing the DB server in parallel and by different
> channels ?
>
>
>
>>> we use the JTDS driver (I tried the Microsoft JDBC, but its performance
>>>> is
>>>> poor compared to the JTDS driver of sourceforge)
>>>> Someone in my company suggested that the problem can rely in either of
>>>> these
>>>> JARs.
>>>> so I will check with the tomcat-dbcp.jar as scenario 1, sql-jdbc.jar as
>>>> scenario2, and maybe both combined as scenario 3.
>>>> scenario 4 will be testing the behavior while validation is configured.
>>>>
>>>>
>>>> Sure, add some extra variables to the problem. That will make it a lot
>>> simpler to find out what happens.
>>>
>>> no need of sarcasm here. these aren't additional variables. we spoke on
>>> the
>>>
>> connection pool, so this is one of the things I can focus on to try and
>> fix
>> the problem.
>>
>> Yes, that was sarcasm. I was just getting a bit frustrated, because I am
> trying to help, but it seems impossible to get logical explanations here
> even about your exact configuration.
>
> So let me try again, graphically.
> As far as I can tell by your posts, your configuration is :
>
> hardware :
> - Vmware VM with your application and Tomcat and jdbc pool and jtds drivers
> - network
> - another machine with the DB
>
> and, somewhere, there is a load-balancer with a "smart keep alive" feature
> built-in.
>
> logical :
>
> Application <--> Tomcat <--> jdbc pool <--> jtds driver <--> network <-->
> database
>
> At the start, the jdbc pool contains for example 10 connections to the
> database.
> At some point, there is a network problem, and as a consequence 5 of these
> connections are broken. But the jdbc pool is not configured to detect this
> in advance, and as a consequence, when the application tries to use a DB
> connection, it may get one of the 5 pooled connections which are broken, and
> it then gets an exception and breaks down.
> Or it may get a pooled connection that is not broken, and then everything
> appears to work fine.
>
> Now can you tell us where in the above schema the "smart keep alive" fits
> in ?
>
> Or else, correct the above schema to tell us how things really work ?
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
Re: Application crash after Migrate to different ESX
Posted by André Warnier <aw...@ice-sa.com>.
הילה wrote:
...
>
>> So, if you are using that pool, I basically do not understand why you would
>> need any additional mechanism to overcome the loss of a db connection when
>> your VM is migrated.
>> Particularly that "smart keep alive" you keep talking about, but never
>> telling us where it comes from and where it is inserted in that
>> architecture.
>>
>> I don't understand on what mechanism are you talking about. I don't want to
> add anything (unless it can fix the problem)
> I just stated the problem, and the environment we have (win 2008, tomcat
> 6.0.29, jdbc pool.. These are the dry facts)
>
> The keep alive is an xslt file which contains actions to perform on the DB
> to check that everything is alive.
> if so, it returns an OK response, which can be viewed in HTML file on IE
> browser.
> The Load balancer samples the keep alive every 10 seconds to check the OK
> state. if it's not OK for 3 times in a row- the LB takes it out from the
> servers' pool and no one else can connect to it.
So now there is also a load balancer ? I've just gone through all your previous posts,
and this is the first time it has been mentioned.
And it appears that it is the load balancer which tests de DB server directly (?)
Huh ? I'm like Chris now, just a bit confused again about your setup.
> so yes. we need this keep alive and can't dismiss it, because it's the
> indication that the server is functioning properly.
Which server ?
and if it's not function
> - The LB can identify it and remove the server from the servers' pool so no
> one will try and approach it.
>
Again, which server is taken out of the server pool ? The DB server, or the one running
Tomcat ?
Do you actually mean that the load balancer on one side, and Tomcat on the other side, are
each accessing the DB server in parallel and by different channels ?
>>
>>> we use the JTDS driver (I tried the Microsoft JDBC, but its performance is
>>> poor compared to the JTDS driver of sourceforge)
>>> Someone in my company suggested that the problem can rely in either of
>>> these
>>> JARs.
>>> so I will check with the tomcat-dbcp.jar as scenario 1, sql-jdbc.jar as
>>> scenario2, and maybe both combined as scenario 3.
>>> scenario 4 will be testing the behavior while validation is configured.
>>>
>>>
>> Sure, add some extra variables to the problem. That will make it a lot
>> simpler to find out what happens.
>>
>> no need of sarcasm here. these aren't additional variables. we spoke on the
> connection pool, so this is one of the things I can focus on to try and fix
> the problem.
>
Yes, that was sarcasm. I was just getting a bit frustrated, because I am trying to help,
but it seems impossible to get logical explanations here even about your exact configuration.
So let me try again, graphically.
As far as I can tell by your posts, your configuration is :
hardware :
- Vmware VM with your application and Tomcat and jdbc pool and jtds drivers
- network
- another machine with the DB
and, somewhere, there is a load-balancer with a "smart keep alive" feature built-in.
logical :
Application <--> Tomcat <--> jdbc pool <--> jtds driver <--> network <--> database
At the start, the jdbc pool contains for example 10 connections to the database.
At some point, there is a network problem, and as a consequence 5 of these connections are
broken. But the jdbc pool is not configured to detect this in advance, and as a
consequence, when the application tries to use a DB connection, it may get one of the 5
pooled connections which are broken, and it then gets an exception and breaks down.
Or it may get a pooled connection that is not broken, and then everything appears to work
fine.
Now can you tell us where in the above schema the "smart keep alive" fits in ?
Or else, correct the above schema to tell us how things really work ?
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Application crash after Migrate to different ESX
Posted by chris derham <ch...@derham.me.uk>.
>
> #2 here blows my mind: an LB runs an XSLT that connects to a DB? WTF?
>
> Another thought that occurs to me is why are you using xslt. Why not code a
jsp page - java servlet whatever. At least in this, you could add some
proper logging, and possibly take some corrective action when an exception
occurs, like restarting the pool. Not quite sure how you do this
programatically, but hope someone else will chip in
>> The problem is clearly that the pool doesn't test the connections when it
>> lends them. You say that you are unwilling or unable to alter this by
>> configuring test on borrow and using dbcp. The alternative seems to catch
>> the exception in your custom code, and get it to re-initialize the
database
>> connection pool. You will effectively just re-implement the connection
>> pool,
>> but if you won't use the existing one doesn't seem much else to suggest
>>
> I haven't understood the part in purple. can you explain?
The part in purple came through in gmail and was "You will effectively just
re-implement the connection pool, but if you won't use the existing one
doesn't seem much else to suggest". I was assuming that upon detection of a
broken db connection in your "smart keep alive", you would try to take some
sort of corrective action. This was the part I meant about re-writing the
functions founds in dbcp connection pool. Sorry if I assumed too much - it
was what I would have done
Chris
Re: Application crash after Migrate to different ESX
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
הילה,
On 5/19/2011 4:06 PM, הילה wrote:
>> 1) You have tomcat 6.0.29 running on virtual machines running on win 2008
>> 2) You have a load balancer which calls a home made xslt transform, that
>> queries the database. If this fails 3 times in a row, the load balancer
>> treats the machine as dead
#2 here blows my mind: an LB runs an XSLT that connects to a DB? WTF?
> just want to add that sometimes it is dead, the keep alive get page 500 and
> stays like this (this is the behavior while disconnect and reconnect the
> network card when the server is out of the LB pool), and sometimes it's page
> 500, then ok state, then page 500 again, then ok - until I restart the
> tomcat and then it's settled (this is the behavior while disconnect and
> reconnect the network card / migrate to another esx , when the server is
> part of the LB pool).
Observing "success, fail, success, fail, success, fail" is usually what
I see when some of the connections in a connection pool are broken while
others are not. It's essentially random which connection you'll get from
the pool, so you may get a good one, then a bad one, then a good one, etc.
Depending on exactly how the connections were established, how they were
severed, and how much time elapsed during the interruption, some may be
able to reconnect using the underlying TCP/IP reconnect while others are
permanently broken. The only solution is to flush those broken
connections from the pool.
I was asking about which pool you were using because there is not a
single "JDBC connection pool". There are two provided by Tomcat and you
neither said that you were using one of them, nor said which one you
were using. Not everyone on the list keeps their notes from your past
posts on their desks just in case you ask a followup later on. It's very
helpful when starting a new thread to re-explain your setup and give as
much detail as possible. I've never seen you post a <Resource> element,
for instance.
>> The problem is clearly that the pool doesn't test the connections when it
>> lends them. You say that you are unwilling or unable to alter this by
>> configuring test on borrow and using dbcp. The alternative seems to catch
>> the exception in your custom code, and get it to re-initialize the database
>> connection pool. You will effectively just re-implement the connection
>> pool,
>> but if you won't use the existing one doesn't seem much else to suggest
>>
> I haven't understood the part in purple. can you explain?
This list sends messages in plain text. There is no purple. Could you
reply and indicate what you mean using words instead of colors?
> and for the sake of trying, I will configure test on borrow and see what
> happens.
I think that would be a good idea.
> even if the server is out of the LB pool, it still can connect to DB and the
> keep alive can still show OK (or not, if the server/DB is not functioning).
> the difference is surfing to keepalive.xml when it's in the LB pool, and
> surfing to _keepalive.xml when it's not in the LB pool.
Let me ask a question about this "keep alive" thing. From the above
summary (which you didn't write but did agree with), your LB runs an
XSLT (presumably, it hits a URL on each server that runs an XSLT) that
contacts a DB. What does it do when it contacts that DB? Does it run a
simple query like "SELECT 1 FROM DUAL" just to see if the database is
reachable? That kind of makes sense, other than the fact that it's using
XSLT to do that. Whatever... you know your environment better than I do.
If the above is true, are you saying that the XSLT reports that the node
is bad or that the node is good? Or that it responds randomly after
migrating a machine between ESX instances?
Does that XSLT use the /exact same connection pool/ that the webapp
does? Not just the same configuration but the /exact same object in
memory/? If not, then your XSLT is lying to you, or at least you are
drawing the wrong conclusions from it's reports.
Other than removing a broken server from the LB pool, what happens when
a server fails a "keep-alive" check (I would call this a "health check",
but you can call it whatever you'd like)? Does it restart the
server/container/anything? Or, does it page you and log an error? That
is, what is the standard operating procedure for a server falling-out of
the LB pool?
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAk3ViJAACgkQ9CaO5/Lv0PDQRQCfTkKb/RXfPOU/eVAA4ROlBF6D
nYoAn1GRNYH8Io6kIn12xd/fUE+QvuQG
=zsa+
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Application crash after Migrate to different ESX
Posted by הילה <hi...@gmail.com>.
>
> All,
>
> 16 emails into this chain - might be worth a summary
>
> 1) You have tomcat 6.0.29 running on virtual machines running on win 2008
> 2) You have a load balancer which calls a home made xslt transform, that
> queries the database. If this fails 3 times in a row, the load balancer
> treats the machine as dead
> 3) When you move one virtual machine to another machine, the custom code
> stops working, hence the load balancer assumes that the new virtual machine
> is dead, hence the issue
>
> IMO this covers all the salient facts so far stated.
>
Good summary. :]
just want to add that sometimes it is dead, the keep alive get page 500 and
stays like this (this is the behavior while disconnect and reconnect the
network card when the server is out of the LB pool), and sometimes it's page
500, then ok state, then page 500 again, then ok - until I restart the
tomcat and then it's settled (this is the behavior while disconnect and
reconnect the network card / migrate to another esx , when the server is
part of the LB pool).
> The problem is clearly that the pool doesn't test the connections when it
> lends them. You say that you are unwilling or unable to alter this by
> configuring test on borrow and using dbcp. The alternative seems to catch
> the exception in your custom code, and get it to re-initialize the database
> connection pool. You will effectively just re-implement the connection
> pool,
> but if you won't use the existing one doesn't seem much else to suggest
>
> I haven't understood the part in purple. can you explain?
and for the sake of trying, I will configure test on borrow and see what
happens.
> Personally I'd be interested in determining what specifically is wrong with
> dbcp for your environment. Works fine for me and many others on this list I
> would wager. Assuming that you can recreate the dbcp issue off production,
> what symptoms do you get? Could you post the full resource element from the
> context.xml section?
>
we used dbcp connection pool for several years now, but on November we got a
requirement to switch the user that runs the tomcat service, from local
system to a domain user. when did that, it generated a memory leak (not sure
if just because of it, or also because of the ntlmauth.dll use). we searched
for a solution to the memory leak for couple of months, until I ran into the
JDBC connection pool. I switched to using the JDBC connection pool, with
still using domain user to run the tomcat service, and the memory leak
disappeared.
No problem to do it on production, I can pull out the server from the LB and
test it. that's what I did to see if it happens only on migrate to another
esx, or even when disconnect and reconnect the network card.
even if the server is out of the LB pool, it still can connect to DB and the
keep alive can still show OK (or not, if the server/DB is not functioning).
the difference is surfing to keepalive.xml when it's in the LB pool, and
surfing to _keepalive.xml when it's not in the LB pool.
I will post the resource element from the xml file on Sunday (it's weekend
here, so we don't work).
On Sunday I'll also try adding testonborrow to the resource tag and see if
the issue still occurs. if so, I'll proceed to check the DBCP pool with this
issue.
> > Sure, add some extra variables to the problem. That will make it a lot
> > > simpler to find out what happens.
> > >
> > > no need of sarcasm here.
> >
> > Most|all the people on this list are doing this for the love of it - as I
> understand it nobody is paid to do it. I would heartily encourage all the
> humor possible to make this more fun for everyone :-)
>
Oh well.. :]
>
> Chris
>
Re: Application crash after Migrate to different ESX
Posted by chris derham <ch...@derham.me.uk>.
All,
16 emails into this chain - might be worth a summary
1) You have tomcat 6.0.29 running on virtual machines running on win 2008
2) You have a load balancer which calls a home made xslt transform, that
queries the database. If this fails 3 times in a row, the load balancer
treats the machine as dead
3) When you move one virtual machine to another machine, the custom code
stops working, hence the load balancer assumes that the new virtual machine
is dead, hence the issue
IMO this covers all the salient facts so far stated.
The problem is clearly that the pool doesn't test the connections when it
lends them. You say that you are unwilling or unable to alter this by
configuring test on borrow and using dbcp. The alternative seems to catch
the exception in your custom code, and get it to re-initialize the database
connection pool. You will effectively just re-implement the connection pool,
but if you won't use the existing one doesn't seem much else to suggest
Personally I'd be interested in determining what specifically is wrong with
dbcp for your environment. Works fine for me and many others on this list I
would wager. Assuming that you can recreate the dbcp issue off production,
what symptoms do you get? Could you post the full resource element from the
context.xml section?
>>
> > Sure, add some extra variables to the problem. That will make it a lot
> > simpler to find out what happens.
> >
> > no need of sarcasm here.
>
> Most|all the people on this list are doing this for the love of it - as I
understand it nobody is paid to do it. I would heartily encourage all the
humor possible to make this more fun for everyone :-)
Chris
Re: Application crash after Migrate to different ESX
Posted by הילה <hi...@gmail.com>.
Yes, but did you actually ever bother to read its documentation, e.g. at
http://people.apache.org/~fhanik/jdbc-pool/jdbc-pool.html ?
> (particularly starting with the attribute "testOnBorrow")
>
> I do not know much about these things, but just entering
> ""org.apache.tomcat.jdbc.pool" in Google, and reading that documentation
> leads me to believe :
> - that there are N database connections in the pool
> - that they can be checked regularly in the background for validity,
> - that a non-valid connection automatically gets dropped from the pool, and
> replaced by a new one when needed
> - that you can pretty much set any aspect of that mechanism by those
> attributes
> - and that you can log what happens
>
Yes, I've read the documentation.
connections can be checked - yes, I know, this is the validation string I
can add in the xml file, but as I said - It has affected our production
environment when using it (although we used it at the time with the dbcp
pool and not the JDBC pool), so I prefer not to go there.
log- I have written down the log I get in my initial post.
>
> So, if you are using that pool, I basically do not understand why you would
> need any additional mechanism to overcome the loss of a db connection when
> your VM is migrated.
> Particularly that "smart keep alive" you keep talking about, but never
> telling us where it comes from and where it is inserted in that
> architecture.
>
> I don't understand on what mechanism are you talking about. I don't want to
add anything (unless it can fix the problem)
I just stated the problem, and the environment we have (win 2008, tomcat
6.0.29, jdbc pool.. These are the dry facts)
The keep alive is an xslt file which contains actions to perform on the DB
to check that everything is alive.
if so, it returns an OK response, which can be viewed in HTML file on IE
browser.
The Load balancer samples the keep alive every 10 seconds to check the OK
state. if it's not OK for 3 times in a row- the LB takes it out from the
servers' pool and no one else can connect to it.
so yes. we need this keep alive and can't dismiss it, because it's the
indication that the server is functioning properly. and if it's not function
- The LB can identify it and remove the server from the servers' pool so no
one will try and approach it.
>
>
>> we use the JTDS driver (I tried the Microsoft JDBC, but its performance is
>> poor compared to the JTDS driver of sourceforge)
>> Someone in my company suggested that the problem can rely in either of
>> these
>> JARs.
>> so I will check with the tomcat-dbcp.jar as scenario 1, sql-jdbc.jar as
>> scenario2, and maybe both combined as scenario 3.
>> scenario 4 will be testing the behavior while validation is configured.
>>
>>
> Sure, add some extra variables to the problem. That will make it a lot
> simpler to find out what happens.
>
> no need of sarcasm here. these aren't additional variables. we spoke on the
connection pool, so this is one of the things I can focus on to try and fix
the problem.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
Re: Application crash after Migrate to different ESX
Posted by André Warnier <aw...@ice-sa.com>.
הילה wrote:
> Hey,
> All of you basically asked about the connection pool I'm using.
> I stated at the beginning that I'm using JDBC connection pool.
> I implement it in the xml file under conf\catalina\localhost, under resource
> tag-
> *factory="org.apache.tomcat.jdbc.pool.DataSourceFactory"*
> with placing tomcat-jdbc.jar in the catalina_home\lib folder.
Yes, but did you actually ever bother to read its documentation, e.g. at
http://people.apache.org/~fhanik/jdbc-pool/jdbc-pool.html ?
(particularly starting with the attribute "testOnBorrow")
I do not know much about these things, but just entering ""org.apache.tomcat.jdbc.pool" in
Google, and reading that documentation leads me to believe :
- that there are N database connections in the pool
- that they can be checked regularly in the background for validity,
- that a non-valid connection automatically gets dropped from the pool, and replaced by a
new one when needed
- that you can pretty much set any aspect of that mechanism by those attributes
- and that you can log what happens
So, if you are using that pool, I basically do not understand why you would need any
additional mechanism to overcome the loss of a db connection when your VM is migrated.
Particularly that "smart keep alive" you keep talking about, but never telling us where it
comes from and where it is inserted in that architecture.
>
> we use the JTDS driver (I tried the Microsoft JDBC, but its performance is
> poor compared to the JTDS driver of sourceforge)
> Someone in my company suggested that the problem can rely in either of these
> JARs.
> so I will check with the tomcat-dbcp.jar as scenario 1, sql-jdbc.jar as
> scenario2, and maybe both combined as scenario 3.
> scenario 4 will be testing the behavior while validation is configured.
>
Sure, add some extra variables to the problem. That will make it a lot simpler to find
out what happens.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Application crash after Migrate to different ESX
Posted by הילה <hi...@gmail.com>.
Hey,
All of you basically asked about the connection pool I'm using.
I stated at the beginning that I'm using JDBC connection pool.
I implement it in the xml file under conf\catalina\localhost, under resource
tag-
*factory="org.apache.tomcat.jdbc.pool.DataSourceFactory"*
with placing tomcat-jdbc.jar in the catalina_home\lib folder.
we use the JTDS driver (I tried the Microsoft JDBC, but its performance is
poor compared to the JTDS driver of sourceforge)
Someone in my company suggested that the problem can rely in either of these
JARs.
so I will check with the tomcat-dbcp.jar as scenario 1, sql-jdbc.jar as
scenario2, and maybe both combined as scenario 3.
scenario 4 will be testing the behavior while validation is configured.
Thanks
Hila
2011/5/18 Christopher Schultz <ch...@christopherschultz.net>
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> הילה,
>
> On 5/18/2011 4:20 AM, הילה wrote:
> > this behavior occur even when I'm disconnecting the Network card from the
> VM
> > settings and connecting it again.
>
> At least it's reproducible in a way that you're willing to try in
> production. What about reproducing it in development/test environments?
> That would certainly be safer...
>
> > afterwards, the keep alive is alternately unavailable.
>
> That doesn't sound like keep-alive to me.
>
> The JDBC protocol, to my knowledge, does not include any kind of
> keep-alive or similar feature. To get that, you'll have to use some
> other product in order to ensure that connections are valid before
> trying to use them, otherwise you'll get exceptions being thrown.
>
> In the case you posted to start this thread, it looked like the
> connection was closed when it was obtained from a connection pool. How
> does com.vsgaming.app.appContext.getDatabaseConnection get it's
> database connections? Are you using a connection pool? If so, which one?
>
> It sounds like you haven't enabled connection validation which is really
> the only way to ensure that connections are valid when you check them
> out of a pool. It's too bad that seems to cause crashes in your
> webapp... I've never heard of that before. Sure, it causes a bit of a
> performance drop but you can tune the validation so it doesn't run on
> every connection check-out to help mitigate that performance loss.
>
> I would look into why connection validation causes your webapp to crash.
> (Strictly speaking, nobody in the Java world refers to an exception
> being thrown as a "crash"... it's a relatively frequent occurrence even
> though the user might not like it. Crashing is *much* worse than a
> single request failing due to some exception.)
>
> > looks like it's not
> > always trying to approach the DB opened wireshark and saw that
> ip.addr==DB
> > IP ADDRESS is not always showing results when the keep alive is
> unavailble =
> > he's not trying to get to the DB at all, I think.
>
> I cannot understand what you are saying above. Can you re-phrase it?
>
> > it all sorts out after I restart the tomcat service. this is why I think
> it
> > does related to tomcat.
>
> Re-establishing all the database connections by any means will likely
> correct everything. It still might have nothing to do with Tomcat. If
> you tell us how you do database connection management, it might help.
>
> - -chris
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.10 (MingW32)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>
> iEYEARECAAYFAk3T6s0ACgkQ9CaO5/Lv0PB0ewCeJSe09OYCOGMMA3nDXwptBoQQ
> qE8An3ZJELMtiwlgFVMbBd/Q1MTtdadY
> =NkAO
> -----END PGP SIGNATURE-----
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
Re: Application crash after Migrate to different ESX
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
הילה,
On 5/18/2011 4:20 AM, הילה wrote:
> this behavior occur even when I'm disconnecting the Network card from the VM
> settings and connecting it again.
At least it's reproducible in a way that you're willing to try in
production. What about reproducing it in development/test environments?
That would certainly be safer...
> afterwards, the keep alive is alternately unavailable.
That doesn't sound like keep-alive to me.
The JDBC protocol, to my knowledge, does not include any kind of
keep-alive or similar feature. To get that, you'll have to use some
other product in order to ensure that connections are valid before
trying to use them, otherwise you'll get exceptions being thrown.
In the case you posted to start this thread, it looked like the
connection was closed when it was obtained from a connection pool. How
does com.vsgaming.app.appContext.getDatabaseConnection get it's
database connections? Are you using a connection pool? If so, which one?
It sounds like you haven't enabled connection validation which is really
the only way to ensure that connections are valid when you check them
out of a pool. It's too bad that seems to cause crashes in your
webapp... I've never heard of that before. Sure, it causes a bit of a
performance drop but you can tune the validation so it doesn't run on
every connection check-out to help mitigate that performance loss.
I would look into why connection validation causes your webapp to crash.
(Strictly speaking, nobody in the Java world refers to an exception
being thrown as a "crash"... it's a relatively frequent occurrence even
though the user might not like it. Crashing is *much* worse than a
single request failing due to some exception.)
> looks like it's not
> always trying to approach the DB opened wireshark and saw that ip.addr==DB
> IP ADDRESS is not always showing results when the keep alive is unavailble =
> he's not trying to get to the DB at all, I think.
I cannot understand what you are saying above. Can you re-phrase it?
> it all sorts out after I restart the tomcat service. this is why I think it
> does related to tomcat.
Re-establishing all the database connections by any means will likely
correct everything. It still might have nothing to do with Tomcat. If
you tell us how you do database connection management, it might help.
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAk3T6s0ACgkQ9CaO5/Lv0PB0ewCeJSe09OYCOGMMA3nDXwptBoQQ
qE8An3ZJELMtiwlgFVMbBd/Q1MTtdadY
=NkAO
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Application crash after Migrate to different ESX
Posted by chris derham <ch...@derham.me.uk>.
>
> this behavior occur even when I'm disconnecting the Network card from the
> VM
> settings and connecting it again.
> afterwards, the keep alive is alternately unavailable. looks like it's not
> always trying to approach the DB opened wireshark and saw that ip.addr==DB
> IP ADDRESS is not always showing results when the keep alive is unavailble
> =
> he's not trying to get to the DB at all, I think.
>
> it all sorts out after I restart the tomcat service. this is why I think it
> does related to tomcat.
>
> I bet if you pulled the power out of the machine, and then plugged it back
in, forcing a machine restart, the problem would also be resolved. However I
don't think that you can blame the electric company :-)
Sounds to me a lot like the problem is this smart keep alive. You said in
one post that it checks the db connection, port and runs some sql. You said
in another that you couldn't use "I haven't configure validation cause in
our production environment it has
proven to be harmful (freezes application)". I don't understand how one
library can do this check and it works (albeit causing the problem you ask
about) whilst a connection pool can't perform the exact same thing without
causing freezes. Sounds to me that you just need to configure validation
query, and remove smart keep alive, and the app would work across esx
migrations
In your first post you said you were using "i use the JDBC application pool"
- how do you configure this?
You haven't provided a clear explanation about this smart keep alive. Who
coded it? What is it? Where does it work - at which level?
HTH
Chris
Re: Application crash after Migrate to different ESX
Posted by הילה <hi...@gmail.com>.
UPDATE-
this behavior occur even when I'm disconnecting the Network card from the VM
settings and connecting it again.
afterwards, the keep alive is alternately unavailable. looks like it's not
always trying to approach the DB opened wireshark and saw that ip.addr==DB
IP ADDRESS is not always showing results when the keep alive is unavailble =
he's not trying to get to the DB at all, I think.
it all sorts out after I restart the tomcat service. this is why I think it
does related to tomcat.
Thanks
Hila
18 במאי 2011 00:06, From Hila <hi...@gmail.com>:
>
>
>>>> I'll try explaining it differently.
>>>>
>>>> we use smart keep alive with this application. the smart keep alive
>>> does a
>>> connection to the DB, query something that relates to the application,
>>> and
>>> if everything turned out good, it returns state OK. if not ok, as in this
>>> case, it returns error page 500 which means- unable to run the specified
>>> query, because unable to connect to DB due to server side issue, or
>>> because
>>> DB unavailable and such.
>>> so after migrate, it disconnect the users, and the keep alive is
>>> unavailable
>>> (page 500), then it's available (state ok), then after couple of seconds
>>> again unavailable. then it returns to state ok and stay like this
>>> (scenario
>>> 1) or stays unavailable (page 500) until I restart the tomcat service and
>>> then display state ok for good (scenario 2).
>>>
>>> ...
>> I have no idea what this "smart keep alive" is, of which you talk above.
>> Is that a part of Tomcat ? If it returns an "error page 500", then it
>> sounds more like a webserver in its own right.
>>
>> ..and also taking into account your other posts about this issue :
>>
>> We would of course try our best to help you here, but this is a mailing
>> list dedicated to Tomcat, and it sounds as if the problem you are having is
>> more about a more general problem of infrastructure and combining different
>> pieces together, of which only one is Tomcat.
>>
>> I believe that questions and answers on this list are not the appropriate
>> way to resolve that kind of problem, and I believe that what you really need
>> is some expert help, from someone who really understands the different
>> pieces and how they (should) fit together.
>> Maybe these external consultants to which you referred ?
>>
>> To give you an idea : you said initially that the problem occurs when a VM
>> is migrated (by Vmware) to another ESX server, and that when that happens
>> the network connections are lost. Well, I am not even sure that this is
>> supposed to happen, if Vmware is configured correctly. So you may be
>> looking in the wrong place for a solution, to start with.
>> better to call in an expert, you would probably save time and money in the
>> end.
>>
>> I talked to VM experts, they say that network disconnection can happen for
>> a sec, but the application shouldn't be affected by it as it is.
>>
> smart keep alive is a keep alive in its ordinary meaning. it called smart
> cause it's not checking only the port and its availability, but also a
> connection to the DB and functionality via DB query of some sort.
> I do believe it relates to tomcat since in other machines that we migrated
> and don't have tomcat, our application doesn't crash.
> I also don't think this happened in every migrate we ever did. so I don't
> understand why it began to happen.
>
> the guys here in the list are helpful, so i'll give it a try. if no one can
> help, i'll try different ways/methods/approach :]
>
> Thanks
>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
Re: Application crash after Migrate to different ESX
Posted by הילה <hi...@gmail.com>.
>
>>> I'll try explaining it differently.
>>>
>>> we use smart keep alive with this application. the smart keep alive does
>> a
>> connection to the DB, query something that relates to the application, and
>> if everything turned out good, it returns state OK. if not ok, as in this
>> case, it returns error page 500 which means- unable to run the specified
>> query, because unable to connect to DB due to server side issue, or
>> because
>> DB unavailable and such.
>> so after migrate, it disconnect the users, and the keep alive is
>> unavailable
>> (page 500), then it's available (state ok), then after couple of seconds
>> again unavailable. then it returns to state ok and stay like this
>> (scenario
>> 1) or stays unavailable (page 500) until I restart the tomcat service and
>> then display state ok for good (scenario 2).
>>
>> ...
> I have no idea what this "smart keep alive" is, of which you talk above.
> Is that a part of Tomcat ? If it returns an "error page 500", then it
> sounds more like a webserver in its own right.
>
> ..and also taking into account your other posts about this issue :
>
> We would of course try our best to help you here, but this is a mailing
> list dedicated to Tomcat, and it sounds as if the problem you are having is
> more about a more general problem of infrastructure and combining different
> pieces together, of which only one is Tomcat.
>
> I believe that questions and answers on this list are not the appropriate
> way to resolve that kind of problem, and I believe that what you really need
> is some expert help, from someone who really understands the different
> pieces and how they (should) fit together.
> Maybe these external consultants to which you referred ?
>
> To give you an idea : you said initially that the problem occurs when a VM
> is migrated (by Vmware) to another ESX server, and that when that happens
> the network connections are lost. Well, I am not even sure that this is
> supposed to happen, if Vmware is configured correctly. So you may be
> looking in the wrong place for a solution, to start with.
> better to call in an expert, you would probably save time and money in the
> end.
>
> I talked to VM experts, they say that network disconnection can happen for
> a sec, but the application shouldn't be affected by it as it is.
>
smart keep alive is a keep alive in its ordinary meaning. it called smart
cause it's not checking only the port and its availability, but also a
connection to the DB and functionality via DB query of some sort.
I do believe it relates to tomcat since in other machines that we migrated
and don't have tomcat, our application doesn't crash.
I also don't think this happened in every migrate we ever did. so I don't
understand why it began to happen.
the guys here in the list are helpful, so i'll give it a try. if no one can
help, i'll try different ways/methods/approach :]
Thanks
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
Re: Application crash after Migrate to different ESX
Posted by André Warnier <aw...@ice-sa.com>.
הילה wrote:
...
>>
>> I'll try explaining it differently.
>>
> we use smart keep alive with this application. the smart keep alive does a
> connection to the DB, query something that relates to the application, and
> if everything turned out good, it returns state OK. if not ok, as in this
> case, it returns error page 500 which means- unable to run the specified
> query, because unable to connect to DB due to server side issue, or because
> DB unavailable and such.
> so after migrate, it disconnect the users, and the keep alive is unavailable
> (page 500), then it's available (state ok), then after couple of seconds
> again unavailable. then it returns to state ok and stay like this (scenario
> 1) or stays unavailable (page 500) until I restart the tomcat service and
> then display state ok for good (scenario 2).
>
...
I have no idea what this "smart keep alive" is, of which you talk above. Is that a part
of Tomcat ? If it returns an "error page 500", then it sounds more like a webserver in its
own right.
..and also taking into account your other posts about this issue :
We would of course try our best to help you here, but this is a mailing list dedicated to
Tomcat, and it sounds as if the problem you are having is more about a more general
problem of infrastructure and combining different pieces together, of which only one is
Tomcat.
I believe that questions and answers on this list are not the appropriate way to resolve
that kind of problem, and I believe that what you really need is some expert help, from
someone who really understands the different pieces and how they (should) fit together.
Maybe these external consultants to which you referred ?
To give you an idea : you said initially that the problem occurs when a VM is migrated (by
Vmware) to another ESX server, and that when that happens the network connections are
lost. Well, I am not even sure that this is supposed to happen, if Vmware is configured
correctly. So you may be looking in the wrong place for a solution, to start with.
better to call in an expert, you would probably save time and money in the end.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Application crash after Migrate to different ESX
Posted by הילה <hi...@gmail.com>.
2011/5/17 André Warnier <aw...@ice-sa.com>
> הילה wrote:
>
>> Hey Guys,
>>
>> My application runs on tomcat 6.0.29, win 2008 R2, which is a VM machine.
>> i use the JDBC application pool., with java 1.6.24 64bit.
>>
>
> OK.
>
>
>
>> I have a problem, in which after migrate of the VM machine to another ESX,
>> the application crashes, and doesn't recover by itself.
>>
>
> Well, Vmware is good at that kind of thing, but it cannot do miracles, so
> one would kind of expect such behaviour. (I mean if an application dies,
> Vmware cannot resurrect it).
>
>
> I know that after migrate, a disconnection is expected, but in my case -
>> it
>> crashes completely, or alternately until it works (but don't know if it's
>> stable now).
>>
>
> Now that is a rather obscure way of explaining the issue. We might need
> Konstantin again here.
>
>
> when it's not working, i see via netstat a time_wait of the connection to
>> the DB.
>>
>
> That is a normal TCP state. See e.g. :
> http://www.ssfnet.org/Exchange/tcp/tcpTutorialNotes.html
> (TCP STATE TRANSITION DIAGRAM).
> It happens for a short while just before a socket is closed.
> It is not the /reason/ for the error below.
>
> So ok, you move a VM somewhere else, while an application is running. For
> some reason, this causes a TCP connection to close, and as a result your
> application is getting the exception below.
> So what is the problem ? Can your application not catch this, and retry ?
>
> I'll try explaining it differently.
>
we use smart keep alive with this application. the smart keep alive does a
connection to the DB, query something that relates to the application, and
if everything turned out good, it returns state OK. if not ok, as in this
case, it returns error page 500 which means- unable to run the specified
query, because unable to connect to DB due to server side issue, or because
DB unavailable and such.
so after migrate, it disconnect the users, and the keep alive is unavailable
(page 500), then it's available (state ok), then after couple of seconds
again unavailable. then it returns to state ok and stay like this (scenario
1) or stays unavailable (page 500) until I restart the tomcat service and
then display state ok for good (scenario 2).
>
>> this is what displayed in the log-
>>
>> 2011-05-17 11:57:49,959 ERROR [Request-Processor3]
>> org.hibernate.transaction.JDBCTransaction - *JDBC begin failed*
>> * *
>>
>> *java.sql.SQLException: Invalid state, the Connection object is closed.*
>>
>> at
>>
>> net.sourceforge.jtds.jdbc.ConnectionJDBC2.checkOpen(ConnectionJDBC2.java:1699)
>>
>> at
>>
>> net.sourceforge.jtds.jdbc.ConnectionJDBC2.getAutoCommit(ConnectionJDBC2.java:2123)
>>
>> at sun.reflect.GeneratedMethodAccessor93.invoke(Unknown
>> Source)
>>
>> at
>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>
>> at java.lang.reflect.Method.invoke(Method.java:597)
>>
>> at
>>
>> org.apache.tomcat.jdbc.pool.ProxyConnection.invoke(ProxyConnection.java:125)
>>
>> at $Proxy0.getAutoCommit(Unknown Source)
>>
>> at
>> org.hibernate.transaction.JDBCTransaction.begin(JDBCTransaction.java:49)
>>
>> at
>>
>> org.hibernate.transaction.JDBCTransactionFactory.beginTransaction(JDBCTransactionFactory.java:24)
>>
>> at
>> org.hibernate.jdbc.JDBCContext.beginTransaction(JDBCContext.java:271)
>>
>> at
>> org.hibernate.impl.SessionImpl.beginTransaction(SessionImpl.java:1079)
>>
>> at
>> com.vsgaming.app.appContext.getDatabaseConnection(appContext.java:296)
>>
>> at
>> com.vsgaming.system.SystemMethods.health(SystemMethods.java:383)
>>
>> at
>> com.vsgaming.system.SystemMethods.processRequest(SystemMethods.java:146)
>>
>> at
>>
>> com.vsgaming.app.appServlet$RequestProcessor.processStandardCompositeRequest(appServlet.java:1547)
>>
>> at
>>
>> com.vsgaming.app.appServlet$RequestProcessor.processLoadBalancedCompositeRequest(appServlet.java:1727)
>>
>> at
>>
>> com.vsgaming.edge.appServlet$RequestProcessor.processCompositeRequest(appServlet.java:1449)
>>
>> at
>>
>> com.vsgaming.app.appServlet$RequestProcessor.processRequestThread(appServlet.java:1372)
>>
>> at
>> com.vsgaming.app.appServlet$RequestProcessor.run(appServlet.java:1338)
>>
>> at java.lang.Thread.run(Thread.java:662)
>>
>>
>> I also noticed that servers that were migrated that has the APR native dll
>> installed, reacted better than the one without the APR native dll (I don't
>> know if it's related, but this is the only difference between the servers
>> that I could think of).
>>
>>
>>
>> has someone experienced this issue?
>>
>> how can I solve it?
>>
>>
>> Thanks
>>
>> Hila
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
Re: Application crash after Migrate to different ESX
Posted by הילה <hi...@gmail.com>.
>
> how can I abruptly break all connections between application and DB?
>> the only way I know is take the server out of the Load balancer so no new
>> users can get to the server + the LB kicks the users that are already
>> connected.
>>
>
> Pulling the network cable at the db server would do it.
>
> It's a production environment. I can't do it.
>
>
>>
>>> I suspect that
>>> a) at webapp end you will have a lot of unusable connections in the
>>> pool. If you did not forget to configure validation, they will be
>>> replaced by valid connections by the next time when you try to get
>>> them from the pool.
>>>
>>>
>> I haven't configure validation cause in our production environment it has
>> proven to be harmful (freezes application)
>>
>>
>>> b) at the DB end you will have a number of aborted transactions. How
>>> it handles them depends on your DB.
>>>
>>>
>> How can I see the aborted transactions?
>> how can I check the configuration that handles it?
>>
>
> That's something the db admin will have to tell you.
>
>
> Each time I come to them with specific questions about connections and how
> it handles them (stuff that is more application and less infrastructure),
> they don't know what to say, since they are DBA infra and not applicative.
> if this is the only way to check the issue, I will ask the applicative DBA's
> (they are external, so we don't approach them with every issue unless they
> are our last resort)
>
>>
>>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
Re: Application crash after Migrate to different ESX
Posted by David kerber <dc...@verizon.net>.
On 5/17/2011 2:39 PM, הילה wrote:
>>
>>>> I know that after migrate, a disconnection is expected, but in my case -
>>>> it
>>>> crashes completely, or alternately until it works (but don't know if
>> it's
>>>> stable now).
>>>
>>
>> It is not a problem, it is a feature. What is your question?
>>
>
> what is the feature here exactly?
>
>>
>> Anyway, what happens with your application if you abruptly break all
>> connections between your application and your DB? What happens with
>> your DB at the same time?
>>
>
> how can I abruptly break all connections between application and DB?
> the only way I know is take the server out of the Load balancer so no new
> users can get to the server + the LB kicks the users that are already
> connected.
Pulling the network cable at the db server would do it.
>
>>
>> I suspect that
>> a) at webapp end you will have a lot of unusable connections in the
>> pool. If you did not forget to configure validation, they will be
>> replaced by valid connections by the next time when you try to get
>> them from the pool.
>>
>
> I haven't configure validation cause in our production environment it has
> proven to be harmful (freezes application)
>
>>
>> b) at the DB end you will have a number of aborted transactions. How
>> it handles them depends on your DB.
>>
>
> How can I see the aborted transactions?
> how can I check the configuration that handles it?
That's something the db admin will have to tell you.
>
>>
>> c) at the users' end whatever they were writing to the database at
>> that moment was lost, because of b). That is why you see the
>> exceptions.
>>
>
> yeah, I know :]
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Application crash after Migrate to different ESX
Posted by הילה <hi...@gmail.com>.
>
> >> I know that after migrate, a disconnection is expected, but in my case -
> >> it
> >> crashes completely, or alternately until it works (but don't know if
> it's
> >> stable now).
> >
>
> It is not a problem, it is a feature. What is your question?
>
what is the feature here exactly?
>
> Anyway, what happens with your application if you abruptly break all
> connections between your application and your DB? What happens with
> your DB at the same time?
>
how can I abruptly break all connections between application and DB?
the only way I know is take the server out of the Load balancer so no new
users can get to the server + the LB kicks the users that are already
connected.
>
> I suspect that
> a) at webapp end you will have a lot of unusable connections in the
> pool. If you did not forget to configure validation, they will be
> replaced by valid connections by the next time when you try to get
> them from the pool.
>
I haven't configure validation cause in our production environment it has
proven to be harmful (freezes application)
>
> b) at the DB end you will have a number of aborted transactions. How
> it handles them depends on your DB.
>
How can I see the aborted transactions?
how can I check the configuration that handles it?
>
> c) at the users' end whatever they were writing to the database at
> that moment was lost, because of b). That is why you see the
> exceptions.
>
yeah, I know :]
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
Re: Application crash after Migrate to different ESX
Posted by Konstantin Kolinko <kn...@gmail.com>.
2011/5/17 André Warnier <aw...@ice-sa.com>:
> הילה wrote:
>>
>> Hey Guys,
>>
>> My application runs on tomcat 6.0.29, win 2008 R2, which is a VM machine.
>> i use the JDBC application pool., with java 1.6.24 64bit.
>
> OK.
>
>>
>> I have a problem, in which after migrate of the VM machine to another ESX,
>> the application crashes, and doesn't recover by itself.
>
> Well, Vmware is good at that kind of thing, but it cannot do miracles, so
> one would kind of expect such behaviour. (I mean if an application dies,
> Vmware cannot resurrect it).
>
>> I know that after migrate, a disconnection is expected, but in my case -
>> it
>> crashes completely, or alternately until it works (but don't know if it's
>> stable now).
>
It is not a problem, it is a feature. What is your question?
Anyway, what happens with your application if you abruptly break all
connections between your application and your DB? What happens with
your DB at the same time?
I suspect that
a) at webapp end you will have a lot of unusable connections in the
pool. If you did not forget to configure validation, they will be
replaced by valid connections by the next time when you try to get
them from the pool.
b) at the DB end you will have a number of aborted transactions. How
it handles them depends on your DB.
c) at the users' end whatever they were writing to the database at
that moment was lost, because of b). That is why you see the
exceptions.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Application crash after Migrate to different ESX
Posted by André Warnier <aw...@ice-sa.com>.
הילה wrote:
> Hey Guys,
>
> My application runs on tomcat 6.0.29, win 2008 R2, which is a VM machine.
> i use the JDBC application pool., with java 1.6.24 64bit.
OK.
>
> I have a problem, in which after migrate of the VM machine to another ESX,
> the application crashes, and doesn't recover by itself.
Well, Vmware is good at that kind of thing, but it cannot do miracles, so one would kind
of expect such behaviour. (I mean if an application dies, Vmware cannot resurrect it).
> I know that after migrate, a disconnection is expected, but in my case - it
> crashes completely, or alternately until it works (but don't know if it's
> stable now).
Now that is a rather obscure way of explaining the issue. We might need Konstantin again
here.
> when it's not working, i see via netstat a time_wait of the connection to
> the DB.
That is a normal TCP state. See e.g. :
http://www.ssfnet.org/Exchange/tcp/tcpTutorialNotes.html
(TCP STATE TRANSITION DIAGRAM).
It happens for a short while just before a socket is closed.
It is not the /reason/ for the error below.
So ok, you move a VM somewhere else, while an application is running. For some reason,
this causes a TCP connection to close, and as a result your application is getting the
exception below.
So what is the problem ? Can your application not catch this, and retry ?
>
> this is what displayed in the log-
>
> 2011-05-17 11:57:49,959 ERROR [Request-Processor3]
> org.hibernate.transaction.JDBCTransaction - *JDBC begin failed*
> * *
>
> *java.sql.SQLException: Invalid state, the Connection object is closed.*
>
> at
> net.sourceforge.jtds.jdbc.ConnectionJDBC2.checkOpen(ConnectionJDBC2.java:1699)
>
> at
> net.sourceforge.jtds.jdbc.ConnectionJDBC2.getAutoCommit(ConnectionJDBC2.java:2123)
>
> at sun.reflect.GeneratedMethodAccessor93.invoke(Unknown
> Source)
>
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>
> at java.lang.reflect.Method.invoke(Method.java:597)
>
> at
> org.apache.tomcat.jdbc.pool.ProxyConnection.invoke(ProxyConnection.java:125)
>
> at $Proxy0.getAutoCommit(Unknown Source)
>
> at
> org.hibernate.transaction.JDBCTransaction.begin(JDBCTransaction.java:49)
>
> at
> org.hibernate.transaction.JDBCTransactionFactory.beginTransaction(JDBCTransactionFactory.java:24)
>
> at
> org.hibernate.jdbc.JDBCContext.beginTransaction(JDBCContext.java:271)
>
> at
> org.hibernate.impl.SessionImpl.beginTransaction(SessionImpl.java:1079)
>
> at
> com.vsgaming.app.appContext.getDatabaseConnection(appContext.java:296)
>
> at
> com.vsgaming.system.SystemMethods.health(SystemMethods.java:383)
>
> at
> com.vsgaming.system.SystemMethods.processRequest(SystemMethods.java:146)
>
> at
> com.vsgaming.app.appServlet$RequestProcessor.processStandardCompositeRequest(appServlet.java:1547)
>
> at
> com.vsgaming.app.appServlet$RequestProcessor.processLoadBalancedCompositeRequest(appServlet.java:1727)
>
> at
> com.vsgaming.edge.appServlet$RequestProcessor.processCompositeRequest(appServlet.java:1449)
>
> at
> com.vsgaming.app.appServlet$RequestProcessor.processRequestThread(appServlet.java:1372)
>
> at
> com.vsgaming.app.appServlet$RequestProcessor.run(appServlet.java:1338)
>
> at java.lang.Thread.run(Thread.java:662)
>
>
> I also noticed that servers that were migrated that has the APR native dll
> installed, reacted better than the one without the APR native dll (I don't
> know if it's related, but this is the only difference between the servers
> that I could think of).
>
>
>
> has someone experienced this issue?
>
> how can I solve it?
>
>
> Thanks
>
> Hila
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org