You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@httpd.apache.org by kalin mintchev <ka...@el.net> on 2005/06/07 19:02:06 UTC
[users@httpd] weird httpd processes
hi all...
for the last 4 - 5 days i noticed something weird on one of the freebsd
boxes here. one httpd process -or sometimes 2 or 3 - take up all the cpu.
and it looks like this:
# top
CPU states: 33.1% user, 0.0% nice, 66.5% system, 0.4% interrupt, 0.0% idle
Mem: 476M Active, 295M Inact, 162M Wired, 61M Cache, 112M Buf, 7556K Free
Swap: 2032M Total, 4K Used, 2032M Free
PID USERNAME PRI NICE SIZE RES STATE TIME WCPU CPU COMMAND
85655 nobody 64 0 20632K 14880K RUN 126:03 97.07% 97.07% httpd
85654 nobody 2 0 19532K 13616K sbwait 0:16 0.24% 0.24% httpd
6081 root 28 0 2016K 1196K RUN 0:00 0.22% 0.10% top
# ps -auwx|grep httpd
.....................................
nobody 85648 0.0 1.3 19564 13848 ?? S 1:35PM 0:14.90
/etc/httpd/bin/httpd -DSSL
nobody 85647 0.0 1.2 18528 12720 ?? S 1:35PM 0:19.13
/etc/httpd/bin/httpd -DSSL
nobody 85646 0.0 1.3 19136 13320 ?? S 1:35PM 0:17.12
/etc/httpd/bin/httpd -DSSL
root 85645 0.0 0.6 12752 6472 ?? Ss 1:35PM 0:08.30
/etc/httpd/bin/httpd -DSSL
nobody 85642 0.0 0.2 3272 2096 ?? S 1:35PM 0:17.78
/usr/local/apache/bin/httpd -DSSL
nobody 1376 0.0 0.0 0 0 ?? Z 6:46AM 0:00.00 (httpd)
this last process is unexisting. to make it disappear i have to stop the
server and then killall -9 httpd it...
after i do that it's ok until i get the same. the time varies...
here for example from bsdsar from today:
#bsdsar -s
..................
09:40:
USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND
www 239 1.5 7.5 86200 78616 p0- S 28May05 121:08.51
WebCatalog -m
root 3368 0.0 0.0 480 316 ?? R 9:40AM 0:00.00 ps -auxwwr
root 3367 0.0 0.0 628 252 ?? S 9:40AM 0:00.00 sh -c ps
-auxwwr [PIPE] head -6
nobody 3363 0.0 0.6 12752 6496 ?? S 9:40AM 0:00.00
/etc/httpd/bin/httpd -DSSL
nobody 3362 0.0 0.6 12752 6496 ?? S 9:40AM 0:00.00
/etc/httpd/bin/httpd -DSSL
10:00:
USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND
nobody 85655 95.2 1.4 20632 14880 ?? R 1:35PM 11:27.22
/etc/httpd/bin/httpd -DSSL
www 239 1.5 7.5 86200 78616 p0- S 28May05 121:16.66
WebCatalog -m
root 3739 0.0 0.0 628 252 ?? S 10:00AM 0:00.00 sh -c ps
-auxwwr [PIPE] head -6
root 3725 0.0 0.1 1992 1464 ?? S 10:00AM 0:00.01
/usr/bin/perl /usr/local/bin/bsdsar_gather
root 3718 0.0 0.0 628 252 ?? Ss 10:00AM 0:00.00 /bin/sh
-c /usr/local/bin/bsdsar_gather
...........
this si the first time it appeared today. it says it started at 1:35pm but
that's not possible since it appears for the first at 10:00 am...
i was trying to figure out if it's a connection being opened from the same
client machine all the time but i can not tell from netstat which
connection corresponds to this process...
i can't find anything unusual in the logs...
Server version: Apache/1.3.33 (Unix)
FreeBSD 4.10-RELEASE
can somebody please help explain what and why is this happening...
thanks a lot...
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] weird httpd processes
Posted by Joshua Slive <js...@gmail.com>.
On 6/19/05, kalin mintchev <ka...@el.net> wrote:
> hi all...
>
> i started this a few weeks ago.. the situation hasn't been improving...
> i'm getting really frustrated. i've looked for scripts but i can not find
> anything wrong. besides there are about 150 domains on that machine and i
> can't go through each php or pearl file....
>
> i woke up today to about 20 or so stalled processes - the most ever. i do
> have server-status on but that just points to the machine name... what to
> do? i can't shut down domains until i figured where the hell is the
> offending script....
>
> right now i have one process running wild but top and server-status report
> differnte cpu usage for those processes. here:
>
> from top:
>
> 55 processes: 2 running, 53 sleeping
> CPU states: 16.0% user, 0.0% nice, 83.7% system, 0.4% interrupt, 0.0% idle
> Mem: 349M Active, 428M Inact, 177M Wired, 38M Cache, 112M Buf, 9944K Free
> Swap: 2032M Total, 100K Used, 2032M Free
>
> PID USERNAME PRI NICE SIZE RES STATE TIME WCPU CPU COMMAND
> 66696 nobody 59 0 16168K 10580K RUN 153:54 98.24% 98.24% httpd
> 9-0 66696 0/3151/3151 W 6.50 9007 0 0.0 62.23 62.23
> 207.214.244.4 guey.el.net GET / HTTP/1.0
Well, it looks like a script at http://guey.el.net/ is causing your
problems. If you need more info than that, you can attach to that
process with a debugger and get a backtrace to see exactly what it is
up to.
Joshua.
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] weird httpd processes
Posted by kalin mintchev <ka...@el.net>.
hi all...
i started this a few weeks ago.. the situation hasn't been improving...
i'm getting really frustrated. i've looked for scripts but i can not find
anything wrong. besides there are about 150 domains on that machine and i
can't go through each php or pearl file....
i woke up today to about 20 or so stalled processes - the most ever. i do
have server-status on but that just points to the machine name... what to
do? i can't shut down domains until i figured where the hell is the
offending script....
right now i have one process running wild but top and server-status report
differnte cpu usage for those processes. here:
from top:
55 processes: 2 running, 53 sleeping
CPU states: 16.0% user, 0.0% nice, 83.7% system, 0.4% interrupt, 0.0% idle
Mem: 349M Active, 428M Inact, 177M Wired, 38M Cache, 112M Buf, 9944K Free
Swap: 2032M Total, 100K Used, 2032M Free
PID USERNAME PRI NICE SIZE RES STATE TIME WCPU CPU COMMAND
66696 nobody 59 0 16168K 10580K RUN 153:54 98.24% 98.24% httpd
36782 qtss 2 0 6324K 4980K poll 59:51 0.00% 0.00%
DarwinStr
from server-status:
Srv PID Acc M CPU SS Req Conn Child Slot Host VHost Request
0-0 66687 0/4206/4206 _ 7.34 3 22 0.0 114.66 114.66
65.214.44.36 richardedwardsnyc.com GET /s_s_singles.w.02/09.html HTTP/1.0
1-0 66688 0/4341/4341 _ 9.20 0 0 0.0 111.73 111.73
24.149.136.172 fotofolio.com GET /images/blbackbttn.gif HTTP/1.1
2-0 66689 11/4136/4136 K 7.63 0 0 0.0 85.68 85.68
68.111.73.32 TimesSquare.com GET
/cut03/images/homecut/solo/frame_r4_c6.gif HTTP/1.1
3-0 66690 1/4426/4426 K 9.09 10 0 0.0 108.92 108.92
24.185.235.98 oldcove.org GET /images/juniorsailing.gif HTTP/1.1
4-0 66691 1/4416/4416 K 9.58 10 0 0.0 108.34 108.34
24.185.235.98 oldcove.org GET /images/T-juniorsailing.gif HTTP/1.1
5-0 66692 1/4552/4552 K 7.52 10 0 0.0 83.19 83.19
24.185.235.98 oldcove.org GET /images/spacer.gif HTTP/1.1
6-0 66693 0/4286/4286 W 8.07 111 0 0.0 122.70 122.70
86.127.1.145 giganticmusic.com GET /home.htm HTTP/1.1
7-0 66694 1/4737/4737 K 9.10 25 7817 99.8 77.52 77.52
68.111.73.32 TimesSquare.com POST /cut03/allResultsT.php HTTP/1.1
8-0 66695 0/4251/4251 _ 9.89 11 303 0.0 143.56 143.56
209.212.80.98 giganticmusic.com GET /smoker.swf HTTP/1.1
9-0 66696 0/3151/3151 W 6.50 9007 0 0.0 62.23 62.23
207.214.244.4 guey.el.net GET / HTTP/1.0
10-0 70405 0/1286/3848 _ 2.93 12 0 0.0 26.16 102.67
209.212.80.98 giganticmusic.com GET /favicon.ico HTTP/1.1
11-0 71906 11/22/3243 K 0.00 0 0 0.0 0.08 87.98
68.111.73.32 TimesSquare.com GET /cut03/images/rank2.0.gif HTTP/1.1
12-0 71921 12/14/2890 K 0.00 0 0 0.0 0.02 42.80
68.111.73.32 TimesSquare.com GET /cut03/images/rank2.5.gif HTTP/1.1
13-0 71923 14/14/2392 W 0.00 0 0 257.7 0.25 67.16
209.212.80.98 giganticmusic.com GET /images/shows/kjhlkjh.gif HTTP/1.1
14-0 71924 5/5/2025 K 0.00 0 0 15.7 0.02 22.99
209.212.80.98 giganticmusic.com GET
/images/artists/someaction/sacover.jpg HTTP/1.1
15-0 71925 2/2/1634 W 0.00 4 0 20.4 0.02 37.66
24.90.33.115 guey.el.net GET /server-status HTTP/1.1
etc..
i'd really appreciate some help... thanks....
> Hi,
>
> This looks like a script that is running out of control. We've had similar
> things here, except that our problem was the script writer failed to
close
> the mysql connections. The user was waiting for data which never came, hit
> cancel, then tried again. The scripts used pconnect calls. This
brought
> a
> 30,000 Euro server to its knees several times by eating all the
available
> sockets, at least until we found out what the problem was.
>
> In your case, set the max execution time for scripts to something short
like
> 10 seconds. Consider also disabling ini_set because we also had
problems
> with users playing with our settings and stressing our server. Be sure
that
> all uri includes are disabled too (XSS attack problems).
>
> To find the exact script is difficult. Something to note is your CPU usage
> has 65% system. Look for scripts that were uploaded around the times
your
> server started having problems, then try to run them is one possibility
to
> find out what's happening. Look in the databases for connections that
are
> open and idle. Also, if your server comes down off its high load, look
in
> the apache log files to see which scripts just completed. Unfortunately
apache writes to the logs only after the script or data has been
(un)successfully sent.
>
> Markus
>
>
>
> On Tuesday 07 June 2005 19:02, kalin mintchev wrote:
>> hi all...
>> for the last 4 - 5 days i noticed something weird on one of the freebsd
boxes here. one httpd process -or sometimes 2 or 3 - take up all the
cpu.
>> and it looks like this:
> 8<---
>
> ---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server
Project.
> See <URL:http://httpd.apache.org/userslist.html> for more info.
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> " from the digest: users-digest-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
>
>
--
--
"In a world without walls or fences, who needs Windows and Gates?"
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: Re: [users@httpd] weird httpd processes
Posted by Systems administrator <sy...@gvi.com.br>.
If I understood what you said, you are able to get the client's IP
address by using another resource, like netstat -a and you see the
connection is in CLOSE_WAIT state for ever. CLOSE_WAIT takes 12 hours to
timeout. It is a long while, but it happens. Maybe you could check
something in your kernel configuration and put this value a little lower.
Seeya,
Miguel.
kalin mintchev wrote:
> i found a client connected to the process but could not find the client's
>ip number in any of the logs of the server. so i assume its some remote
>application that is hitting directly the offending script. but how do i
>get to that?!? like somebody mentioned it's probably not written in the
>logs until the file is served and the process finished but if the process
>never finishes (CLOSE_WAIT) there is no trace in the log....
>
>here is what's right now...
>
># top
> PID USERNAME PRI NICE SIZE RES STATE TIME WCPU CPU COMMAND
>53452 nobody 55 0 15932K 10128K RUN 108:16 96.88% 96.88% httpd
>56877 root 28 0 1996K 1100K RUN 0:00 0.89% 0.29% top
>.............
>
># sockstat -4 | grep 53452
>nobody httpd 53452 4 tcp4 66.117.34.36:80 66.134.162.210:1625
>nobody httpd 53452 115 tcp4 *:443 *:*
>nobody httpd 53452 116 tcp4 *:80 *:*
>nobody httpd 53452 117 tcp4 66.117.34.36:80 *:*
>
># netstat -a
>......
>tcp4 0 0 server.http h-66-134-162-210.1625 CLOSE_WAIT
>.......
>
># grep -rl 66.134.162.210 /html/*/logs/*
>#
>
>there are about maybe 150 domains under /html.... so now what?
>
>i guess i'd have to go through each and every one of them?!? - i know the
>20 min time window when the process gets wild but i don't have much
>else...
>
>i'm going to turn the mod_status on - like suggested here - to see what it
>will come up with but i'm not holding my breath....
>
>
>thanks....
>
>
>
>
>
>
>
>
>---------------------------------------------------------------------
>The official User-To-User support forum of the Apache HTTP Server Project.
>See <URL:http://httpd.apache.org/userslist.html> for more info.
>To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> " from the digest: users-digest-unsubscribe@httpd.apache.org
>For additional commands, e-mail: users-help@httpd.apache.org
>
>
>
>
.
--
Andre Miguel
Administrador de sistemas
e-mail: suporte@gvi.com.br
GVI - Globalview Internet Services
www.gvi.com.br
Fone: 3351-4470
Fax: 3351-4488
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] weird httpd processes
Posted by kalin mintchev <ka...@el.net>.
i found a client connected to the process but could not find the client's
ip number in any of the logs of the server. so i assume its some remote
application that is hitting directly the offending script. but how do i
get to that?!? like somebody mentioned it's probably not written in the
logs until the file is served and the process finished but if the process
never finishes (CLOSE_WAIT) there is no trace in the log....
here is what's right now...
# top
PID USERNAME PRI NICE SIZE RES STATE TIME WCPU CPU COMMAND
53452 nobody 55 0 15932K 10128K RUN 108:16 96.88% 96.88% httpd
56877 root 28 0 1996K 1100K RUN 0:00 0.89% 0.29% top
.............
# sockstat -4 | grep 53452
nobody httpd 53452 4 tcp4 66.117.34.36:80 66.134.162.210:1625
nobody httpd 53452 115 tcp4 *:443 *:*
nobody httpd 53452 116 tcp4 *:80 *:*
nobody httpd 53452 117 tcp4 66.117.34.36:80 *:*
# netstat -a
......
tcp4 0 0 server.http h-66-134-162-210.1625 CLOSE_WAIT
.......
# grep -rl 66.134.162.210 /html/*/logs/*
#
there are about maybe 150 domains under /html.... so now what?
i guess i'd have to go through each and every one of them?!? - i know the
20 min time window when the process gets wild but i don't have much
else...
i'm going to turn the mod_status on - like suggested here - to see what it
will come up with but i'm not holding my breath....
thanks....
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] weird httpd processes
Posted by Joost de Heer <sa...@xs4all.nl>.
kalin mintchev said:
>
> could this be a log size problem too?
> one of the virtual server has almost a gig of access log - i'm going to
> see why it wasn't rotated - but can this be one of the reasons.
>
> right now i have another instance of httpd going wild - started sometime
> in the last hour...
Do you have mod_status on? Perhaps the status pages can show you something
useful....
Joost
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] weird httpd processes
Posted by kalin mintchev <ka...@el.net>.
could this be a log size problem too?
one of the virtual server has almost a gig of access log - i'm going to
see why it wasn't rotated - but can this be one of the reasons.
right now i have another instance of httpd going wild - started sometime
in the last hour...
# top
CPU states: 24.9% user, 0.0% nice, 74.7% system, 0.4% interrupt, 0.0% idle
Mem: 469M Active, 3056K Inact, 87M Wired, 4416K Cache, 112M Buf, 439M Free
Swap: 2032M Total, 340K Used, 2032M Free
PID USERNAME PRI NICE SIZE RES STATE TIME WCPU CPU COMMAND
6113 nobody 64 0 17980K 12204K RUN 80:54 97.80% 97.80% httpd
>
> thank you - i'll be monitoring i guess - so far all the php scripts
> modified in the last 2 weeks seem to be fine... i tried most of them...
>
>
>
>> Hi,
>>
>> This looks like a script that is running out of control. We've had
>> similar
>> things here, except that our problem was the script writer failed to
>> close
>> the mysql connections. The user was waiting for data which never came,
>> hit
>> cancel, then tried again. The scripts used pconnect calls. This
>> brought
>> a
>> 30,000 Euro server to its knees several times by eating all the
>> available
>> sockets, at least until we found out what the problem was.
>>
>> In your case, set the max execution time for scripts to something short
>> like
>> 10 seconds. Consider also disabling ini_set because we also had
>> problems
>> with users playing with our settings and stressing our server. Be sure
>> that
>> all uri includes are disabled too (XSS attack problems).
>>
>> To find the exact script is difficult. Something to note is your CPU
>> usage
>> has 65% system. Look for scripts that were uploaded around the times
>> your
>> server started having problems, then try to run them is one possibility
>> to
>> find out what's happening. Look in the databases for connections that
>> are
>> open and idle. Also, if your server comes down off its high load, look
>> in
>> the apache log files to see which scripts just completed. Unfortunately
>> apache writes to the logs only after the script or data has been
>> (un)succe
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] weird httpd processes
Posted by kalin mintchev <ka...@el.net>.
thank you - i'll be monitoring i guess - so far all the php scripts
modified in the last 2 weeks seem to be fine... i tried most of them...
> Hi,
>
> This looks like a script that is running out of control. We've had
> similar
> things here, except that our problem was the script writer failed to close
> the mysql connections. The user was waiting for data which never came,
> hit
> cancel, then tried again. The scripts used pconnect calls. This brought
> a
> 30,000 Euro server to its knees several times by eating all the available
> sockets, at least until we found out what the problem was.
>
> In your case, set the max execution time for scripts to something short
> like
> 10 seconds. Consider also disabling ini_set because we also had problems
> with users playing with our settings and stressing our server. Be sure
> that
> all uri includes are disabled too (XSS attack problems).
>
> To find the exact script is difficult. Something to note is your CPU
> usage
> has 65% system. Look for scripts that were uploaded around the times your
> server started having problems, then try to run them is one possibility to
> find out what's happening. Look in the databases for connections that are
> open and idle. Also, if your server comes down off its high load, look in
> the apache log files to see which scripts just completed. Unfortunately
> apache writes to the logs only after the script or data has been
> (un)successfully sent.
>
> Markus
>
>
>
> On Tuesday 07 June 2005 19:02, kalin mintchev wrote:
>> hi all...
>>
>> for the last 4 - 5 days i noticed something weird on one of the freebsd
>> boxes here. one httpd process -or sometimes 2 or 3 - take up all the
>> cpu.
>> and it looks like this:
> 8<---
>
> ---------------------------------------------------------------------
> The official User-To-User support forum of the Apache HTTP Server Project.
> See <URL:http://httpd.apache.org/userslist.html> for more info.
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> " from the digest: users-digest-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
>
>
--
--
"In a world without walls or fences, who needs Windows and Gates?"
Re: [users@httpd] weird httpd processes
Posted by Markus Mayer <my...@gmx.at>.
Hi,
This looks like a script that is running out of control. We've had similar
things here, except that our problem was the script writer failed to close
the mysql connections. The user was waiting for data which never came, hit
cancel, then tried again. The scripts used pconnect calls. This brought a
30,000 Euro server to its knees several times by eating all the available
sockets, at least until we found out what the problem was.
In your case, set the max execution time for scripts to something short like
10 seconds. Consider also disabling ini_set because we also had problems
with users playing with our settings and stressing our server. Be sure that
all uri includes are disabled too (XSS attack problems).
To find the exact script is difficult. Something to note is your CPU usage
has 65% system. Look for scripts that were uploaded around the times your
server started having problems, then try to run them is one possibility to
find out what's happening. Look in the databases for connections that are
open and idle. Also, if your server comes down off its high load, look in
the apache log files to see which scripts just completed. Unfortunately
apache writes to the logs only after the script or data has been
(un)successfully sent.
Markus
On Tuesday 07 June 2005 19:02, kalin mintchev wrote:
> hi all...
>
> for the last 4 - 5 days i noticed something weird on one of the freebsd
> boxes here. one httpd process -or sometimes 2 or 3 - take up all the cpu.
> and it looks like this:
8<---
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
[users@httpd] Re: weird httpd processes
Posted by kalin mintchev <ka...@el.net>.
> Does your httpd installation serve up any scripted content i.e. using
> perl, PHP, etc. I have seen apache spin out of control due to poorly,
> or incorrectly written scripts.
ok. yes. the machine has been running fine for months. how would i know
which is the offending script?!? it serves a lot of websites that use
different scripts - perl,php,etc...
>
> Nathan
>
--
--
"In a world without walls or fences, who needs Windows and Gates?"
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org