Re: apache.org getting DoS'd? maxclients turned down

[gr...@apache.org]

Pier Fumagalli wrote:

> > gregames@apache.org wrote:
> >>
> >> Brian Behlendorf wrote:
> >>>
> >>> The load average on apache.org spiked into the 300's a couple times this
> >>> weekend.
> >
> >>> I bet there's some cutoff, maybe it's 800, or 900, where
> >>> the # of parallel connections is just too much and something goes
> >>> nonlinear between Apache and the kernel.
> >>
> >> That's possible.  I will nudge it upward and keep a close eye on the load
> >> averages.

> > OK, with MaxClients at 900, I saw the load averages start to jump.  
> > I'm still seeing erratic ssh response.  Is there some way to monitor network
> > bandwidth utilization in real time?  I wonder if we are maxing out our NICs
> > this morning?  That would make the big downloads take longer, using up more httpd
> > processes.  If that's our problem, maybe we should pursue Joshua Slive's idea
> > of redirecting requests for the most popular downloads to a reliable mirror, like
> > nagoya.
 
> Please, feel free to redirect
> 
> www.apache.org/dist/httpd/(.*)   -> nagoya.apache.org/dist/httpd/$1
> www.apache.org/dist/jakarta/(.*) -> nagoya.apache.org/dist/jakarta/$1
> jakarta.apache.org/builds/(.*)   -> nagoya.apache.org/dist/jakarta/$1
> www.apache.org/dist/xml/(.*)     -> nagoya.apache.org/dist/xml/$1
> Xml.apache.org/dist/(.*)         -> nagoya.apache.org/dist/xml/$1
> 
> This should take some load away from daedalus, and maybe give you a better
> chance to investigate the problem further... Right now it's kinda of
> unbearable on there...

Well, I don't feel brave enough to redirect all of that at once without knowing
what's going on in the xml or jarkarta projects, and without being able to
monitor nagoya very well.  But I did add:

Redirect /dist/httpd/binaries http://nagoya.apache.org/dist/httpd/binaries

...as well as changing old Redirects from /httpd.apache.org/dist/ to point to
the proper place on nagoya, and bumping MaxClients from 775 to 800.

Please let me know if there are problems due to the Redirects I added.  There is
a list of the top 20 bandwidth eaters from yesterday at
http://www.apache.org/~gregames/top20.bytes_served.10.20 .  I suppose it would
be safe to put in individual redirects for those on the top 20 list that aren't
already covered until daedalus's response is sufficiently good (whatever that
means - maybe sub-minute response for shift-reload of http://xml.apache.org/ ). 
It's getting better already.  

httpd folks: there is a buglet or two relating to changing MaxClients via
graceful restart.  If you look at daedalus's server-status, there's a bunch of
processes at the end of the display mostly in "C" state, occasionally "G". 
grep'ping for the pids in the the error log, I see "long lost child came home"
messages.  I think we're stopping too soon when we search for the child's pid
after getting SIGCHILD.  mod_status is displaying "requests being processed" too
high, as if those long lost children were really there.  Then after I bumped
MaxClients up from 775 to 800, the number of httpd's actually running according
to ps seemed to top out at 777.  The difference there seems to be that we still
have several processes in "G" state in the lower part of the scoreboard.

Greg

Re: apache.org getting DoS'd? maxclients turned down

[gr...@apache.org]

gregames@apache.org wrote:
> 
> Pier Fumagalli wrote:

> > > I'm still seeing erratic ssh response.  Is there some way to monitor network
> > > bandwidth utilization in real time?  I wonder if we are maxing out our NICs
> > > this morning?  That would make the big downloads take longer, using up more httpd
> > > processes.  If that's our problem, maybe we should pursue Joshua Slive's idea
> > > of redirecting requests for the most popular downloads to a reliable mirror, like
> > > nagoya.
> 
> > Please, feel free to redirect
> >
> > www.apache.org/dist/httpd/(.*)   -> nagoya.apache.org/dist/httpd/$1
> > www.apache.org/dist/jakarta/(.*) -> nagoya.apache.org/dist/jakarta/$1
> > jakarta.apache.org/builds/(.*)   -> nagoya.apache.org/dist/jakarta/$1
> > www.apache.org/dist/xml/(.*)     -> nagoya.apache.org/dist/xml/$1
> > Xml.apache.org/dist/(.*)         -> nagoya.apache.org/dist/xml/$1
> >
> > This should take some load away from daedalus, and maybe give you a better
> > chance to investigate the problem further... Right now it's kinda of
> > unbearable on there...
> 
> Well, I don't feel brave enough to redirect all of that at once without knowing
> what's going on in the xml or jarkarta projects, and without being able to
> monitor nagoya very well.  But I did add:
> 
> Redirect /dist/httpd/binaries http://nagoya.apache.org/dist/httpd/binaries

We are maxed out on httpds this morning, and httpd response is poor.  There
isn't much load:

load averages:  0.94,  1.35,  2.34

Even though MaxClients is 800, we only have 776 child processes.  I think this
is because we're not cleaning up the scoreboard properly after I dropped
MaxClients via graceful restart. 

Here's a plan: non-graceful restart with MaxClients at 820, and at the same time
redirect the current httpd source tarballs and
/jakarta.apache.org/builds/jakarta-tomcat-4.0/release/v4.1.12/ to nagoya.  That
should free up bandwidth on daedalus.  MaxClients 820 looked OK for quite a
while yesterday.  If it's too much, we can bring MaxClients down via graceful;
raising it with a dirty scoreboard is the piece that doesn't work.

I don't want to redirect the jakarta nightly builds though, or everything under
/www.apache.org/dist/httpd/ , because then people wouldn't be able to access
files between the time daedalus gets updated and the time nagoya mirrors the
update. 

Greg

Re: apache.org getting DoS'd? maxclients turned down

[Pier Fumagalli <pi...@betaversion.org>]

On 21/10/02 21:14, "gregames@apache.org" <gr...@apache.org> wrote:

> Well, I don't feel brave enough to redirect all of that at once without
> knowing what's going on in the xml or jarkarta projects, and without being
> able to monitor nagoya very well.

Greg, if you want I can open up you an account...

> But I did add:
> 
> Redirect /dist/httpd/binaries http://nagoya.apache.org/dist/httpd/binaries

And we're beautifully serving binaries at a rate of 20/30 per minute! :-)
I'll keep the situation under control as we go! :-) :-)

GO worker/MPM!

    Pier