Design Review for ajp13's changes: WAS problem w/ ajp13 - if Tomc at is shutdown

[Shinta Tjio <st...@broadjump.com>]

Hi, all,
I would like to propose some changes to eliminate the
requirement to restart Apache, when you restart Tomcat.
I'm willing to give the code to anyone who needs it, 
when I'm done testing.

But I need some help/suggestions so that I can put in 
the right code. If any of the proposed changes below
should not exists ever, I'm open to other suggestions.
This is my first time looking at mod_jk's ajp13 code.
So any clue to make these better would be appreciated.

Right now, if you use ajp13 and you restart Tomcat, you
have to also restart Apache. See details in previous
postings. For us, having to restart Apache is not a
feasible solution in our customers' environment.

After looking at the code, I have two possible solutions:

1. From mod_jk, I can detect that the socket has been
   closed by Tomcat. This is normally indicated by the
   recv() returning ECONNRESET. The recv() is called
   after the request has been sent to the socket. The
   send() unfortunately, doesn't give you an error.

   The proposed fixed is to check for errno ECONNRESET, 
   then set the is_recoverable_error flag to TRUE, in
   the service() function in jk_ajp13_worker.c. I also 
   add a code in mod_jk.c, to check for this flag, and
   call the service() method again if the flag is set 
   TRUE. The 2nd time the service() method is called, 
   it will reconnect to Tomcat like normal.

2. Another solution would be to put in a select() on the
   socket prior to send(), looking for the socket being
   read ready. Under normal condition, this select() 
   should return nothing. But if Tomcat shuts down 
   the socket, this select() should return the socket
   being read ready. When this happen, I issue a read()
   of 1 bytes. If the read() comes back with return code
   0, this should be an indication that the socket was
   closed on the remote end. Then I will proceed to close 
   the socket. The remaining logic already handles the 
   reconnect, etc.

I have both of these solution prototyped and minimally 
tested. They both Anyone care to comment which solution fits better 
with the overall code? Anyone voluteer to review the code?

thanks,
shinta

> -----Original Message----- 
> From: Shinta Tjio 
> To: tomcat-dev@jakarta.apache.org 
> Cc: 'Dan Milstein' 
> Sent: 3/6/01 7:01 PM 
> Subject: RE: FW: problem w/ ajp13 - if Tomcat is shutdown 
> 
> I am using Tomcat 3.2.1, Apache 1.3.14, running on 
> Solaris 2.8, Sun machines. 
> 
> After various attempts of debugging this, I have 
> more information. 
> 
> 1. Even though I'm setting the worker's property 
> cache_size to default (1), I'm finding there 
> are up to 6 connections opened from Apache to 
> Tomcat. I deduce this by looking at the mod_jk.conf 
> and by doing netstat. 
> 
> I finally find out, this is so because my Apache 
> is set to spawn minimum of 6 children and each 
> of those children are making separate connections 
> to Tomcat. 
> 
> This is very bad because, I ended up having to 
> reload 6 times before Tomcat starts serving me 
> the page again. Each time it uses a different 
> Apache children that has defunct socket. So the 
> more Apache children I have, the longer it takes 
> me to recover from this problem. 
> 
> 2. It seems when Tomcat dies & restarts, the send() 
> called by ajp13's jk_tcp_socket_sendfull() does not 
> get an error. But the recv() does get an error, with 
> errno ECONNRESET. After which, the socket is properly 
> closed. 
> 
> 3. When I shutdown Tomcat, those sockets that were 
> opened between Apache/Tomcat showed up in state 
> CLOSE_WAIT, and FIN_WAIT2. I think this is normally 
> solved by calling the shutdown() API after closing 
> the socket. However, this would have to be done from 
> the Tomcat side in Ajp13ConnectionHandler.java. 
> I can't find the corresponding method of Socket 
> in Java. 
> 
> So.. based on all of these, the only fix I can think 
> of putting is to make mod_jk retry the send() if 
> recv() comes back with an error ECONNRESET. The retry 
> should happen after the old socket is properly closed. 
> 
> Anyone wants to comment? 
> 
> shinta 
>   
> 
> > -----Original Message----- 
> > From: Dan Milstein [ mailto:danmil@shore.net 
> <ma...@shore.net> 
> ] 
> > Sent: Tuesday, March 06, 2001 12:00 PM 
> > To: tomcat-dev@jakarta.apache.org 
> > Subject: Re: FW: problem w/ ajp13 - if Tomcat is shutdown 
> > 
> > 
> > What version of TC are you using?  What version of Apache? 
> > 
> > I would look into the mod_jk docs -- I think this is the 
> > spec'd behavior 
> > (which, admittedly, is not great, but that makes it more of 
> a feature 
> > request than a bug ;-).  With ajp13, Apache opens up a 
> > persistent TCP/IP 
> > connection TC -- if TC restarts, I think that connection may 
> > just hang up 
> > and then timeout (since Apache doesn't know that TC has restarted). 
> > 
> > If anyone wants to work on this, you would have the undying 
> > thanks of the 
> > rest of the TC community -- having to restart Apache all the 
> > time bugs a 
> > *lot* of people. 
> > 
> > -Dan 
> > 
> > > Shinta Tjio wrote: 
> > > 
> > > I'm having problem with mod_jk if ajp13 is used. 
> > > 
> > > The problem is often reproduced when Tomcat is shut 
> > > down without Apache being shut down. When a request 
> > > is fired through Apache as soon as Tomcat starts, 
> > > I often get Internal Server Error. The mod_jk.log 
> > > will have the following: 
> > > 
> > > > [jk_uri_worker_map.c (344)]: Into 
> > jk_uri_worker_map_t::map_uri_to_worker 
> > > 
> > > > [jk_uri_worker_map.c (406)]: 
> > jk_uri_worker_map_t::map_uri_to_worker, 
> > > >   Found a match ajp13 
> > > > [jk_worker.c (123)]: Into wc_get_worker_for_name ajp13 
> > > > [jk_worker.c (127)]: wc_get_worker_for_name, done  
> found a worker 
> > > > [jk_ajp13_worker.c (651)]: Into jk_worker_t::get_endpoint 
> > > > [jk_ajp13_worker.c (536)]: Into jk_endpoint_t::service 
> > > > [jk_ajp13.c (346)]: Into ajp13_marshal_into_msgb 
> > > > [jk_ajp13.c (480)]: ajp13_marshal_into_msgb - Done 
> > > > [jk_ajp13_worker.c (203)]: connection_tcp_get_message: Error - 
> > > >    jk_tcp_socket_recvfull failed 
> > > > [jk_ajp13_worker.c (619)]: Error reading request 
> > > > [jk_ajp13_worker.c (489)]: Into jk_endpoint_t::done 
> > > 
> > > If I hit reload multiple times, eventually Tomcat will 
> > > serve the servlet fine. 
> > > 
> > > Did anyone see this problem before? Is there anyway 
> > > around this? 
> > > 
> > > shinta 
> > 
> > -- 
> > 
> > Dan Milstein // danmil@shore.net 
> > 
> 
  

Re: Design Review for ajp13's changes: WAS problem w/ ajp13 - if Tomc at is shutdown

[cm...@yahoo.com]

Hi Shinta,

It's sounds like a solution to a real problem, please send a patch,
I'm sure someone will read it. Dan and Henri are the best people to ask
about this, I can also help a bit ( I've been using RPMs lately, it's too
easy to get them and not worry about compile :-)

My only sugestion/concern is that the code should work on both Windows and
unix - or at least compile :-) 

Costin


> I would like to propose some changes to eliminate the
> requirement to restart Apache, when you restart Tomcat.
> I'm willing to give the code to anyone who needs it, 
> when I'm done testing.
> 
> But I need some help/suggestions so that I can put in 
> the right code. If any of the proposed changes below
> should not exists ever, I'm open to other suggestions.
> This is my first time looking at mod_jk's ajp13 code.
> So any clue to make these better would be appreciated.
> 
> Right now, if you use ajp13 and you restart Tomcat, you
> have to also restart Apache. See details in previous
> postings. For us, having to restart Apache is not a
> feasible solution in our customers' environment.
> 
> After looking at the code, I have two possible solutions:
> 
> 1. From mod_jk, I can detect that the socket has been
>    closed by Tomcat. This is normally indicated by the
>    recv() returning ECONNRESET. The recv() is called
>    after the request has been sent to the socket. The
>    send() unfortunately, doesn't give you an error.
> 
>    The proposed fixed is to check for errno ECONNRESET, 
>    then set the is_recoverable_error flag to TRUE, in
>    the service() function in jk_ajp13_worker.c. I also 
>    add a code in mod_jk.c, to check for this flag, and
>    call the service() method again if the flag is set 
>    TRUE. The 2nd time the service() method is called, 
>    it will reconnect to Tomcat like normal.
> 
> 2. Another solution would be to put in a select() on the
>    socket prior to send(), looking for the socket being
>    read ready. Under normal condition, this select() 
>    should return nothing. But if Tomcat shuts down 
>    the socket, this select() should return the socket
>    being read ready. When this happen, I issue a read()
>    of 1 bytes. If the read() comes back with return code
>    0, this should be an indication that the socket was
>    closed on the remote end. Then I will proceed to close 
>    the socket. The remaining logic already handles the 
>    reconnect, etc.
> 
> I have both of these solution prototyped and minimally 
> tested. They both Anyone care to comment which solution fits better 
> with the overall code? Anyone voluteer to review the code?
> 
> thanks,
> shinta
> 
> > -----Original Message----- 
> > From: Shinta Tjio 
> > To: tomcat-dev@jakarta.apache.org 
> > Cc: 'Dan Milstein' 
> > Sent: 3/6/01 7:01 PM 
> > Subject: RE: FW: problem w/ ajp13 - if Tomcat is shutdown 
> > 
> > I am using Tomcat 3.2.1, Apache 1.3.14, running on 
> > Solaris 2.8, Sun machines. 
> > 
> > After various attempts of debugging this, I have 
> > more information. 
> > 
> > 1. Even though I'm setting the worker's property 
> > cache_size to default (1), I'm finding there 
> > are up to 6 connections opened from Apache to 
> > Tomcat. I deduce this by looking at the mod_jk.conf 
> > and by doing netstat. 
> > 
> > I finally find out, this is so because my Apache 
> > is set to spawn minimum of 6 children and each 
> > of those children are making separate connections 
> > to Tomcat. 
> > 
> > This is very bad because, I ended up having to 
> > reload 6 times before Tomcat starts serving me 
> > the page again. Each time it uses a different 
> > Apache children that has defunct socket. So the 
> > more Apache children I have, the longer it takes 
> > me to recover from this problem. 
> > 
> > 2. It seems when Tomcat dies & restarts, the send() 
> > called by ajp13's jk_tcp_socket_sendfull() does not 
> > get an error. But the recv() does get an error, with 
> > errno ECONNRESET. After which, the socket is properly 
> > closed. 
> > 
> > 3. When I shutdown Tomcat, those sockets that were 
> > opened between Apache/Tomcat showed up in state 
> > CLOSE_WAIT, and FIN_WAIT2. I think this is normally 
> > solved by calling the shutdown() API after closing 
> > the socket. However, this would have to be done from 
> > the Tomcat side in Ajp13ConnectionHandler.java. 
> > I can't find the corresponding method of Socket 
> > in Java. 
> > 
> > So.. based on all of these, the only fix I can think 
> > of putting is to make mod_jk retry the send() if 
> > recv() comes back with an error ECONNRESET. The retry 
> > should happen after the old socket is properly closed. 
> > 
> > Anyone wants to comment? 
> > 
> > shinta 
> >   
> > 
> > > -----Original Message----- 
> > > From: Dan Milstein [ mailto:danmil@shore.net 
> > <ma...@shore.net> 
> > ] 
> > > Sent: Tuesday, March 06, 2001 12:00 PM 
> > > To: tomcat-dev@jakarta.apache.org 
> > > Subject: Re: FW: problem w/ ajp13 - if Tomcat is shutdown 
> > > 
> > > 
> > > What version of TC are you using?  What version of Apache? 
> > > 
> > > I would look into the mod_jk docs -- I think this is the 
> > > spec'd behavior 
> > > (which, admittedly, is not great, but that makes it more of 
> > a feature 
> > > request than a bug ;-).  With ajp13, Apache opens up a 
> > > persistent TCP/IP 
> > > connection TC -- if TC restarts, I think that connection may 
> > > just hang up 
> > > and then timeout (since Apache doesn't know that TC has restarted). 
> > > 
> > > If anyone wants to work on this, you would have the undying 
> > > thanks of the 
> > > rest of the TC community -- having to restart Apache all the 
> > > time bugs a 
> > > *lot* of people. 
> > > 
> > > -Dan 
> > > 
> > > > Shinta Tjio wrote: 
> > > > 
> > > > I'm having problem with mod_jk if ajp13 is used. 
> > > > 
> > > > The problem is often reproduced when Tomcat is shut 
> > > > down without Apache being shut down. When a request 
> > > > is fired through Apache as soon as Tomcat starts, 
> > > > I often get Internal Server Error. The mod_jk.log 
> > > > will have the following: 
> > > > 
> > > > > [jk_uri_worker_map.c (344)]: Into 
> > > jk_uri_worker_map_t::map_uri_to_worker 
> > > > 
> > > > > [jk_uri_worker_map.c (406)]: 
> > > jk_uri_worker_map_t::map_uri_to_worker, 
> > > > >   Found a match ajp13 
> > > > > [jk_worker.c (123)]: Into wc_get_worker_for_name ajp13 
> > > > > [jk_worker.c (127)]: wc_get_worker_for_name, done  
> > found a worker 
> > > > > [jk_ajp13_worker.c (651)]: Into jk_worker_t::get_endpoint 
> > > > > [jk_ajp13_worker.c (536)]: Into jk_endpoint_t::service 
> > > > > [jk_ajp13.c (346)]: Into ajp13_marshal_into_msgb 
> > > > > [jk_ajp13.c (480)]: ajp13_marshal_into_msgb - Done 
> > > > > [jk_ajp13_worker.c (203)]: connection_tcp_get_message: Error - 
> > > > >    jk_tcp_socket_recvfull failed 
> > > > > [jk_ajp13_worker.c (619)]: Error reading request 
> > > > > [jk_ajp13_worker.c (489)]: Into jk_endpoint_t::done 
> > > > 
> > > > If I hit reload multiple times, eventually Tomcat will 
> > > > serve the servlet fine. 
> > > > 
> > > > Did anyone see this problem before? Is there anyway 
> > > > around this? 
> > > > 
> > > > shinta 
> > > 
> > > -- 
> > > 
> > > Dan Milstein // danmil@shore.net 
> > > 
> > 
>   
>