Re: Design Review for ajp13's changes: WAS problem w/ ajp13 - if Tomc at is shutdown

[cm...@yahoo.com]

Hi Shinta,

It's sounds like a solution to a real problem, please send a patch,
I'm sure someone will read it. Dan and Henri are the best people to ask
about this, I can also help a bit ( I've been using RPMs lately, it's too
easy to get them and not worry about compile :-)

My only sugestion/concern is that the code should work on both Windows and
unix - or at least compile :-) 

Costin


> I would like to propose some changes to eliminate the
> requirement to restart Apache, when you restart Tomcat.
> I'm willing to give the code to anyone who needs it, 
> when I'm done testing.
> 
> But I need some help/suggestions so that I can put in 
> the right code. If any of the proposed changes below
> should not exists ever, I'm open to other suggestions.
> This is my first time looking at mod_jk's ajp13 code.
> So any clue to make these better would be appreciated.
> 
> Right now, if you use ajp13 and you restart Tomcat, you
> have to also restart Apache. See details in previous
> postings. For us, having to restart Apache is not a
> feasible solution in our customers' environment.
> 
> After looking at the code, I have two possible solutions:
> 
> 1. From mod_jk, I can detect that the socket has been
>    closed by Tomcat. This is normally indicated by the
>    recv() returning ECONNRESET. The recv() is called
>    after the request has been sent to the socket. The
>    send() unfortunately, doesn't give you an error.
> 
>    The proposed fixed is to check for errno ECONNRESET, 
>    then set the is_recoverable_error flag to TRUE, in
>    the service() function in jk_ajp13_worker.c. I also 
>    add a code in mod_jk.c, to check for this flag, and
>    call the service() method again if the flag is set 
>    TRUE. The 2nd time the service() method is called, 
>    it will reconnect to Tomcat like normal.
> 
> 2. Another solution would be to put in a select() on the
>    socket prior to send(), looking for the socket being
>    read ready. Under normal condition, this select() 
>    should return nothing. But if Tomcat shuts down 
>    the socket, this select() should return the socket
>    being read ready. When this happen, I issue a read()
>    of 1 bytes. If the read() comes back with return code
>    0, this should be an indication that the socket was
>    closed on the remote end. Then I will proceed to close 
>    the socket. The remaining logic already handles the 
>    reconnect, etc.
> 
> I have both of these solution prototyped and minimally 
> tested. They both Anyone care to comment which solution fits better 
> with the overall code? Anyone voluteer to review the code?
> 
> thanks,
> shinta
> 
> > -----Original Message----- 
> > From: Shinta Tjio 
> > To: tomcat-dev@jakarta.apache.org 
> > Cc: 'Dan Milstein' 
> > Sent: 3/6/01 7:01 PM 
> > Subject: RE: FW: problem w/ ajp13 - if Tomcat is shutdown 
> > 
> > I am using Tomcat 3.2.1, Apache 1.3.14, running on 
> > Solaris 2.8, Sun machines. 
> > 
> > After various attempts of debugging this, I have 
> > more information. 
> > 
> > 1. Even though I'm setting the worker's property 
> > cache_size to default (1), I'm finding there 
> > are up to 6 connections opened from Apache to 
> > Tomcat. I deduce this by looking at the mod_jk.conf 
> > and by doing netstat. 
> > 
> > I finally find out, this is so because my Apache 
> > is set to spawn minimum of 6 children and each 
> > of those children are making separate connections 
> > to Tomcat. 
> > 
> > This is very bad because, I ended up having to 
> > reload 6 times before Tomcat starts serving me 
> > the page again. Each time it uses a different 
> > Apache children that has defunct socket. So the 
> > more Apache children I have, the longer it takes 
> > me to recover from this problem. 
> > 
> > 2. It seems when Tomcat dies & restarts, the send() 
> > called by ajp13's jk_tcp_socket_sendfull() does not 
> > get an error. But the recv() does get an error, with 
> > errno ECONNRESET. After which, the socket is properly 
> > closed. 
> > 
> > 3. When I shutdown Tomcat, those sockets that were 
> > opened between Apache/Tomcat showed up in state 
> > CLOSE_WAIT, and FIN_WAIT2. I think this is normally 
> > solved by calling the shutdown() API after closing 
> > the socket. However, this would have to be done from 
> > the Tomcat side in Ajp13ConnectionHandler.java. 
> > I can't find the corresponding method of Socket 
> > in Java. 
> > 
> > So.. based on all of these, the only fix I can think 
> > of putting is to make mod_jk retry the send() if 
> > recv() comes back with an error ECONNRESET. The retry 
> > should happen after the old socket is properly closed. 
> > 
> > Anyone wants to comment? 
> > 
> > shinta 
> >   
> > 
> > > -----Original Message----- 
> > > From: Dan Milstein [ mailto:danmil@shore.net 
> > <ma...@shore.net> 
> > ] 
> > > Sent: Tuesday, March 06, 2001 12:00 PM 
> > > To: tomcat-dev@jakarta.apache.org 
> > > Subject: Re: FW: problem w/ ajp13 - if Tomcat is shutdown 
> > > 
> > > 
> > > What version of TC are you using?  What version of Apache? 
> > > 
> > > I would look into the mod_jk docs -- I think this is the 
> > > spec'd behavior 
> > > (which, admittedly, is not great, but that makes it more of 
> > a feature 
> > > request than a bug ;-).  With ajp13, Apache opens up a 
> > > persistent TCP/IP 
> > > connection TC -- if TC restarts, I think that connection may 
> > > just hang up 
> > > and then timeout (since Apache doesn't know that TC has restarted). 
> > > 
> > > If anyone wants to work on this, you would have the undying 
> > > thanks of the 
> > > rest of the TC community -- having to restart Apache all the 
> > > time bugs a 
> > > *lot* of people. 
> > > 
> > > -Dan 
> > > 
> > > > Shinta Tjio wrote: 
> > > > 
> > > > I'm having problem with mod_jk if ajp13 is used. 
> > > > 
> > > > The problem is often reproduced when Tomcat is shut 
> > > > down without Apache being shut down. When a request 
> > > > is fired through Apache as soon as Tomcat starts, 
> > > > I often get Internal Server Error. The mod_jk.log 
> > > > will have the following: 
> > > > 
> > > > > [jk_uri_worker_map.c (344)]: Into 
> > > jk_uri_worker_map_t::map_uri_to_worker 
> > > > 
> > > > > [jk_uri_worker_map.c (406)]: 
> > > jk_uri_worker_map_t::map_uri_to_worker, 
> > > > >   Found a match ajp13 
> > > > > [jk_worker.c (123)]: Into wc_get_worker_for_name ajp13 
> > > > > [jk_worker.c (127)]: wc_get_worker_for_name, done  
> > found a worker 
> > > > > [jk_ajp13_worker.c (651)]: Into jk_worker_t::get_endpoint 
> > > > > [jk_ajp13_worker.c (536)]: Into jk_endpoint_t::service 
> > > > > [jk_ajp13.c (346)]: Into ajp13_marshal_into_msgb 
> > > > > [jk_ajp13.c (480)]: ajp13_marshal_into_msgb - Done 
> > > > > [jk_ajp13_worker.c (203)]: connection_tcp_get_message: Error - 
> > > > >    jk_tcp_socket_recvfull failed 
> > > > > [jk_ajp13_worker.c (619)]: Error reading request 
> > > > > [jk_ajp13_worker.c (489)]: Into jk_endpoint_t::done 
> > > > 
> > > > If I hit reload multiple times, eventually Tomcat will 
> > > > serve the servlet fine. 
> > > > 
> > > > Did anyone see this problem before? Is there anyway 
> > > > around this? 
> > > > 
> > > > shinta 
> > > 
> > > -- 
> > > 
> > > Dan Milstein // danmil@shore.net 
> > > 
> > 
>   
>