You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@httpd.apache.org by Asrai khn <as...@gmail.com> on 2007/12/17 20:24:52 UTC
[users@httpd] apache freezes the virtual machine.
Hi we are running our shared web server under vm (UML) distro FC7 its
working fine but once in 10-15 days apache shoots virtual machine load very
hight upto 20-30 and all swap been consumed. if some how we get into the vm
via ssh stop/start apache normalize the vm which clearly show that is apache
that freezing and shooting the load on vm.
Mem: 512M
swap: 1024M
here what i find in logs
Dec 17 12:50:51 xxxx kernel: httpd invoked oom-killer: gfp_mask=0x200d2,
order=0, oomkilladj=0
Dec 17 12:50:51 xxxx kernel: 1330fd68: [<08053cfc>] dump_stack+0x1c/0x20
Dec 17 12:50:51 xxxx kernel: 1330fd80: [<0808a4f8>] out_of_memory+0x6b/0xd3
Dec 17 12:50:51 xxxx kernel: 1330fda8: [<0808b356>]
__alloc_pages+0x1ed/0x26c
Dec 17 12:50:51 xxxx kernel: 1330fde8: [<080974b3>]
read_swap_cache_async+0x33/0x80
Dec 17 12:50:51 xxxx kernel: 1330fdfc: [<080917e8>]
swapin_readahead+0x3c/0x5f
Dec 17 12:50:51 xxxx kernel: 1330fe20: [<08091868>] do_swap_page+0x5d/0x1d4
Dec 17 12:50:51 xxxx kernel: 1330fe4c: [<08091fbb>]
__handle_mm_fault+0x10a/0x17e
Dec 17 12:50:51 xxxx kernel: 1330fe80: [<08054926>]
handle_page_fault+0xe2/0x1bc
Dec 17 12:50:51 xxxx kernel: 1330feb4: [<08054b5e>] segv+0xa2/0x240
Dec 17 12:50:51 xxxx kernel: 1330ff68: [<08063ae7>] handle_segv+0x32/0x3a
Dec 17 12:50:51 xxxx kernel: 1330ff94: [<08063f8f>] userspace+0x117/0x166
Dec 17 12:50:51 xxxx kernel: 1330ffdc: [<08055371>] fork_handler+0x74/0x7c
Dec 17 12:50:51 xxxx kernel: 1330fffc: [<a55a5a5a>] 0xa55a5a5a
Dec 17 12:50:51 xxxx kernel:
Dec 17 12:50:51 xxxx kernel: Mem-info:
Dec 17 12:50:51 xxxx kernel: Normal per-cpu:
Dec 17 12:51:23 xxxx kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 34
Cold: hi: 62, btch: 15 usd: 60
Dec 17 12:51:28 xxxx kernel: Active:73753 inactive:50187 dirty:0 writeback:4
unstable:0
Dec 17 12:51:29 xxxx kernel: free:702 slab:2098 mapped:670 pagetables:960
bounce:0
Dec 17 12:51:30 xxxx kernel: Normal free:2808kB min:2884kB low:3604kB
high:4324kB active:295012kB inactive:200748kB present:520192kB
pages_scanned:829898 a
ll_unreclaimable? yes
Any idea how to make this vm stable?
Askar.
Re: [users@httpd] apache freezes the virtual machine.
Posted by Asrai khn <as...@gmail.com>.
On Dec 24, 2007 9:05 PM, Joshua Slive <jo...@slive.ca> wrote:
>
> That's what you get with the checkforensic script?
>
Yes that's all i got with
/usr/sbin/check_forensic /var/log/httpd/forensic_log
>
> Anyway, what that tells you is that one of those scripts is likely the
> culprit that is gobbling down all the memory in your box. Take a look
> at all the scripts that were being requested and see if you can figure
> out what is going wrong.
>
>
While i was checking the web server again our monitoring system give high
load alert for web server , then i once again killed httpd processes and run
'check_forensic' script this time agian it was the same domain
sudo /usr/sbin/check_forensic /var/log/httpd/forensic_log
+606c:476e903d:453|GET
/forum/viewtopic.php?t=26170&highlight=&sid=f800cd6e1c579026b318dfce4d907b74
HTTP/1.1|Host:www.xxxx.com|Connection:Keep-alive|Accept:*/*|From:googlebot(at)googlebot.com|User-Agent:Mozilla/5.0
(compatible; Googlebot/2.1;
+http%3a//www.google.com/bot.html)|Accept-Encoding:gzip
Yes you are correct need to check the script under the htdocs of that
particular site, problem is that i am not php guy :-s
Anyhow i am thinking to move the site to our other shared web server which
is also running virtual 'xen'.
Thanks. Askar
Re: [users@httpd] apache freezes the virtual machine.
Posted by Joshua Slive <jo...@slive.ca>.
On Dec 26, 2007 1:12 PM, Asrai khn <as...@gmail.com> wrote:
>
>
>
> On Dec 24, 2007 9:05 PM, Joshua Slive <jo...@slive.ca> wrote:
> > Anyway, what that tells you is that one of those scripts is likely the
> > culprit that is gobbling down all the memory in your box. Take a look
> > at all the scripts that were being requested and see if you can figure
> > out what is going wrong.
>
> Joshua,
>
> Ok we are moving the faulty site off our shared web server to dedicated
> hosting.
>
> However Also, need to figure how to protect ourself better against this kind
> of
> thing on shared webs. Limiting resource usage of apache spawned user
> processes via apache config options?
Not when you are running scripting engines in the same process as the
web server.
Here are some notes on the general issue of privelege separation.
Resource limitting is a different, but quite related issue with
similar solutions.
http://wiki.apache.org/httpd/PrivilegeSeparation
In general, if you need resource limitting you should get the
scripting engines out of the web server process, either by using
cgi/fast-cgi, or by running multiple web server instances.
Joshua.
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] apache freezes the virtual machine.
Posted by Asrai khn <as...@gmail.com>.
On Dec 24, 2007 9:05 PM, Joshua Slive <jo...@slive.ca> wrote:
> Anyway, what that tells you is that one of those scripts is likely the
> culprit that is gobbling down all the memory in your box. Take a look
> at all the scripts that were being requested and see if you can figure
> out what is going wrong.
Joshua,
Ok we are moving the faulty site off our shared web server to dedicated
hosting.
However Also, need to figure how to protect ourself better against this kind
of
thing on shared webs. Limiting resource usage of apache spawned user
processes via apache config options?
Thanks. Askar
Re: [users@httpd] apache freezes the virtual machine.
Posted by Joshua Slive <jo...@slive.ca>.
On Dec 24, 2007 9:48 AM, Asrai khn <as...@gmail.com> wrote:
>
> >
> >
> >
> > Hi Joshua,
> >
> >
> > Please let me know what would be action strategy now.
> >
> > +10d8:476979f0:39a|POST /forum/posting.php
> HTTP/1.0|Accept:*/*|User-Agent:Mozilla/4.0 (compatible; MSIE 4.01; Digital
> AlphaServer 1000A 4/233; Windows NT; Powered By 64-Bit Alpha
> Processor)|Referer:http%3a//www.xxx.com/forum/posting.php?mode=newtopic&f=12&sid=dd4189290cb614c55071a463743df2cd|Content-Type:application/x-www-form-urlencoded|Host:
> www.xxx.com|Content-Length:59423|Pragma:no-cache|Cookie:phpbb2mysql_data=a%253A2%253A%257Bs%253A11%253A%2522autologinid%2522%253Bs%253A0%253A%2522%2522%253Bs%253A6%253A%2522userid%2522%253Bs%253A5%253A%252214567%2522%253B%257D
> ; phpbb2mysql_sid=dd4189290cb614c55071a463743df2cd
> > +133e:476c9678:3|GET
> /forum/viewtopic.php?p=51500&sid=578b164651a42205e13907ad29ee6c1f
> (...)
> For get to mention that www.xxx.com is a single domain and there wasn't
> any logs for any other domain.
That's what you get with the checkforensic script?
Anyway, what that tells you is that one of those scripts is likely the
culprit that is gobbling down all the memory in your box. Take a look
at all the scripts that were being requested and see if you can figure
out what is going wrong.
Joshua.
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] apache freezes the virtual machine.
Posted by Asrai khn <as...@gmail.com>.
>
>
> Hi Joshua,
>
> Please let me know what would be action strategy now.
>
> +10d8:476979f0:39a|POST /forum/posting.php
> HTTP/1.0|Accept:*/*|User-Agent:Mozilla/4.0 (compatible; MSIE 4.01; Digital
> AlphaServer 1000A 4/233; Windows NT; Powered By 64-Bit Alpha
> Processor)|Referer:http%3a//www.xxx.com/forum/posting.php?mode=newtopic&f=12&sid=dd4189290cb614c55071a463743df2cd|Content-Type:application/x-www-form-urlencoded|Host:
>
> www.xxx.com|Content-Length:59423|Pragma:no-cache|Cookie:phpbb2mysql_data=a%253A2%253A%257Bs%253A11%253A%2522autologinid%2522%253Bs%253A0%253A%2522%2522%253Bs%253A6%253A%2522userid%2522%253Bs%253A5%253A%252214567%2522%253B%257D; phpbb2mysql_sid=dd4189290cb614c55071a463743df2cd
> +133e:476c9678:3|GET
> /forum/viewtopic.php?p=51500&sid=578b164651a42205e13907ad29ee6c1f
>
(...)
For get to mention that www.xxx.com is a single domain and there wasn't
any logs for any other domain.
Askar.
Re: [users@httpd] apache freezes the virtual machine.
Posted by Asrai khn <as...@gmail.com>.
On Dec 18, 2007 7:36 PM, Joshua Slive <jo...@slive.ca> wrote:
>
> The point of mod_log_forensic is to run it after an apache crash to
> see what requests were in play at the time of the crash. One way for
> you to do that would be to monitor memory usage on the apache box and
> kill -9 the server when things look to be getting out of control. Then
> you run check_forensic to see if you can find the cause.
Hi Joshua,
Ok we got another crash and our shared web servers (running under UML) was
unaccessible, however this time mod_forensic logging to a file.
Below is the logs when httpd bring the vm to its knees then i have to
restart it, i have replace the Host with 'xxx' just to protect our company
hosting.
Please let me know what would be action strategy now.
+10d8:476979f0:39a|POST /forum/posting.php
HTTP/1.0|Accept:*/*|User-Agent:Mozilla/4.0 (compatible; MSIE 4.01; Digital
AlphaServer 1000A 4/233; Windows NT; Powered By 64-Bit Alpha
Processor)|Referer:http%3a//www.xxx.com/forum/posting.php?mode=newtopic&f=12&sid=dd4189290cb614c55071a463743df2cd|Content-Type:application/x-www-form-urlencoded|Host:
www.xxx.com|Content-Length:59423|Pragma:no-cache|Cookie:phpbb2mysql_data=a%253A2%253A%257Bs%253A11%253A%2522autologinid%2522%253Bs%253A0%253A%2522%2522%253Bs%253A6%253A%2522userid%2522%253Bs%253A5%253A%252214567%2522%253B%257D;
phpbb2mysql_sid=dd4189290cb614c55071a463743df2cd
+133e:476c9678:3|GET
/forum/viewtopic.php?p=51500&sid=578b164651a42205e13907ad29ee6c1f
HTTP/1.1|Host:www.xxx.com|Connection:Keep-alive|Accept:*/*|From:googlebot(at)googlebot.com|User-Agent:Mozilla/5.0
(compatible; Googlebot/2.1;
+http%3a//www.google.com/bot.html)|Accept-Encoding:gzip
+1341:476c9629:1|GET /app/cs.php?c=gb&pno=0 HTTP/1.0|Accept:image/gif,
image/x-xbitmap, image/jpeg, image/pjpeg, application/x-shockwave-flash,
*/*|Accept-Language:en|Accept-Encoding:gzip, deflate|User-Agent:Mozilla/4.0
(compatible; MSIE 6.0; Windows NT 5.1)|Host:www.xxx.com
+13c0:476c96a0:0|GET /app/cs.php?c=gb&pno=0 HTTP/1.1|Accept:image/gif,
image/x-xbitmap, image/jpeg, image/pjpeg, application/x-shockwave-flash,
*/*|Accept-Language:en|Accept-Encoding:gzip, deflate|User-Agent:Mozilla/4.0
(compatible; MSIE 6.0; Windows NT 5.1)|Host:
www.xxx.com|Cache-Control:max-stale=0|Connection:close|X-BlueCoat-Via:3EFBBB6A4CC354A7
+1bd5:47697260:1b0|POST /forum/posting.php
HTTP/1.0|Accept:*/*|User-Agent:Mozilla/4.0 (compatible; MSIE 6.0; Windows NT
5.1; SV1; .NET CLR 1.1.4322; XMPP Tiscali Communicator v.10.0.2; .NET CLR
2.0.50727
)|Referer:http%3a//www.xxx.com/forum/posting.php?mode=newtopic&f=12&sid=58b841154dc1bc186be53e6712dcf5d6|Content-Type:application/x-www-form-urlencoded|Host:
www.xxx.com|Content-Length:56738|Pragma:no-cache|Cookie:phpbb2mysql_data=a%253A2%253A%257Bs%253A11%253A%2522autologinid%2522%253Bs%253A0%253A%2522%2522%253Bs%253A6%253A%2522userid%2522%253Bs%253A5%253A%252214564%2522%253B%257D;
phpbb2mysql_sid=58b841154dc1bc186be53e6712dcf5d6
+30e0:476c95af:3f1|GET /app/cs.php?c=gb&pno=0 HTTP/1.0|Accept:image/gif,
image/x-xbitmap, image/jpeg, image/pjpeg, application/x-shockwave-flash,
*/*|Accept-Language:en|Accept-Encoding:gzip, deflate|User-Agent:Mozilla/4.0
(compatible; MSIE 6.0; Windows NT 5.1)|Host:www.xxx.com
+3751:476c9585:3a2|GET /app/cs.php?c=gb&pno=0 HTTP/1.0|Accept:image/gif,
image/x-xbitmap, image/jpeg, image/pjpeg, application/x-shockwave-flash,
*/*|Accept-Language:en|Accept-Encoding:gzip, deflate|User-Agent:Mozilla/4.0
(compatible; MSIE 6.0; Windows NT 5.1)|Host:www.xxx.com
+37b5:476c9681:3b9|GET /
HTTP/1.1|Accept:*/*|Referer:http%3a//click.betafoxsearch.com/click/?dT0xNS4yMy45MzQ2ODgyLjI0NC4xJTdDNzY4NCU3QzE2OTk5JTdDMDAxMDg5Mjk0NjU4MjclN0MyNjAwOTAwMyU3QyU3QyU3QzU4JTdDJm51bWJlcj00JTNBNCZhZHRpdGxlPUNyZWF0ZSUyMGElMjBmcmVlJTIwbmV3c2xldHRlciUyMGZvciUyMHlvdXIlMjBjb25kbyUyMG9yJTIwY28tb3AmYWR1cmw9aHR0cCUzQSUyRiUyRnd3dy5teWNvbmRvbmV3c2xldHRlci5jb20mYWRib2R5PU15Q29uZG9OZXdzbGV0dGVyLmNvbSUyMGlzJTIwYSUyMGZyZWUlMjBzZXJ2aWNlJTIwZm9yJTIwY29uZG8lMjBhbmQlMjBjby1vcCUyMHJlc2lkZW50cyUyMHRvJTIwaW50ZXJhY3QlMjBvbmxpbmUuJTIwV2UlMjBwcm92aWRlJTIwYSUyMGZyZWUlMjBuZXdzbGV0dGVyJTIwYW55JTIwcmVzaWRlbnQlMjBjYW4lMjBhY2Nlc3MlMjB3aXRob3V0JTIwY2hhcmdlLiY=|Accept-Language:en-us|Accept-Encoding:gzip,
deflate|User-Agent:Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1;
FunWebProducts; .NET CLR 1.1.4322)|Connection:Keep-Alive|Host:
www.mycondonewsletter.com
+3adb:476c95a3:385|GET /app/cs.php?c=gb&pno=0 HTTP/1.0|Accept:image/gif,
image/x-xbitmap, image/jpeg, image/pjpeg, application/x-shockwave-flash,
*/*|Accept-Language:en|Accept-Encoding:gzip, deflate|User-Agent:Mozilla/4.0
(compatible; MSIE 6.0; Windows NT 5.1)|Host:www.xxx.com
+4e1c:47680f0e:18c|GET /app/cs.php?c=mpage&id=1125&catid=2&mid=113
HTTP/1.1|Host:www.xxx.com|User-Agent:Mozilla/4.0 (compatible; MSIE 6.0;
Windows NT 5.1;
SV1)|Accept:text/xml,application/xml,application/xhtml+xml,text/html;q=0.9
,text/plain;q=0.8,image/png,*/*;q=0.5|Connection:close
+512b:47695866:464|POST /forum/posting.php
HTTP/1.0|Accept:*/*|User-Agent:Mozilla/4.0 (compatible; MSIE 6.0; Windows NT
5.1; .NET CLR 1.1.4322;
FDM)|Referer:http%3a//www.xxx.com/forum/posting.php?mode=newtopic&f=12&sid=dd19e39ca20bd404017e222fe1e06f04|Content-Type:application/x-www-form-urlencoded|Host:
www.xxx.com|Content-Length:58997|Pragma:no-cache|Cookie:phpbb2mysql_data=a%253A2%253A%257Bs%253A11%253A%2522autologinid%2522%253Bs%253A0%253A%2522%2522%253Bs%253A6%253A%2522userid%2522%253Bs%253A5%253A%252214553%2522%253B%257D;
phpbb2mysql_sid=dd19e39ca20bd404017e222fe1e06f04
+7e4e:476c95e5:c1|GET /app/cs.php?c=gb&pno=0 HTTP/1.1|Accept:image/gif,
image/x-xbitmap, image/jpeg, image/pjpeg, application/x-shockwave-flash,
*/*|Accept-Language:en|Accept-Encoding:gzip, deflate|User-Agent:Mozilla/4.0
(compatible; MSIE 6.0; Windows NT 5.1)|Host:
www.xxx.com|Cache-Control:max-stale=0|Connection:Keep-Alive|X-BlueCoat-Via:3EFBBB6A4CC354A7
+7f31:476c9682:b5|GET /app/cs.php?c=gb&pno=0 HTTP/1.0|Accept:image/gif,
image/x-xbitmap, image/jpeg, image/pjpeg, application/x-shockwave-flash,
*/*|Accept-Language:en|Accept-Encoding:gzip, deflate|User-Agent:Mozilla/4.0
(compatible; MSIE 6.0; Windows NT 5.1)|Host:www.xxx.com
Thanks.
Askar
Re: [users@httpd] apache freezes the virtual machine.
Posted by Joshua Slive <jo...@slive.ca>.
On Dec 18, 2007 9:04 AM, Asrai khn <as...@gmail.com> wrote:
>
>
>
> On Dec 18, 2007 2:30 AM, Joshua Slive <jo...@slive.ca> wrote:
> >
> >
> >
> >
> > No, if you put it outside of any <VirtualHost> it should log for all
> > virtual hosts.
> >
> >
> >
> >
> >
> >
>
> Hi Joshua
>
> Thanks for all the tips, i have enabled mod_log_forensic on our shared web
> server and and also get the 'check_forensic' script.
>
> sudo /usr/sbin/check_forensic /var/log/httpd/forensic_log
>
> atm running it gives nothing which is obvious.
>
> Any tip to run it for example every 10 minutes and then mail the 'output' if
> any, instead of running it manually each time?
The point of mod_log_forensic is to run it after an apache crash to
see what requests were in play at the time of the crash. One way for
you to do that would be to monitor memory usage on the apache box and
kill -9 the server when things look to be getting out of control. Then
you run check_forensic to see if you can find the cause.
Joshua.
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] apache freezes the virtual machine.
Posted by Asrai khn <as...@gmail.com>.
On Dec 18, 2007 2:30 AM, Joshua Slive <jo...@slive.ca> wrote:
>
> No, if you put it outside of any <VirtualHost> it should log for all
> virtual hosts.
>
>
>
Hi Joshua
Thanks for all the tips, i have enabled mod_log_forensic on our shared web
server and and also get the 'check_forensic' script.
sudo /usr/sbin/check_forensic /var/log/httpd/forensic_log
atm running it gives nothing which is obvious.
Any tip to run it for example every 10 minutes and then mail the 'output' if
any, instead of running it manually each time?
Thanks. Askar
Re: [users@httpd] apache freezes the virtual machine.
Posted by Joshua Slive <jo...@slive.ca>.
On Dec 17, 2007 3:34 PM, Asrai khn <as...@gmail.com> wrote:
>
>
>
> On Dec 18, 2007 12:33 AM, Joshua Slive <jo...@slive.ca> wrote:
> >
> >
> > The only thing we know here is that for some reason apache is sucking
> > up tons of memory under a specific circumstance. This is likely due to
> > a bug in one of your programs that is running under apache (a php
> > script, for example), but could also be due to a bug in apache httpd
> > intself or in some third-party module you are running.
> >
> > One useful tool to figure out what request is triggering the problem
> > is mod_log_forensic. But without more debugging, there is nothing we
> > can suggest.
> >
> >
>
> Hi Joshua
>
> Thanks for the reply ok will try to grab more data so that we can figure out
> the problem. Would you please tell me do i have to add "ForensicLog
> logs/forensic_log" to all v.host s configs under /etc/httpd/conf.d ?
No, if you put it outside of any <VirtualHost> it should log for all
virtual hosts.
Joshua.
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] apache freezes the virtual machine.
Posted by Asrai khn <as...@gmail.com>.
On Dec 18, 2007 12:33 AM, Joshua Slive <jo...@slive.ca> wrote:
>
> The only thing we know here is that for some reason apache is sucking
> up tons of memory under a specific circumstance. This is likely due to
> a bug in one of your programs that is running under apache (a php
> script, for example), but could also be due to a bug in apache httpd
> intself or in some third-party module you are running.
>
> One useful tool to figure out what request is triggering the problem
> is mod_log_forensic. But without more debugging, there is nothing we
> can suggest.
>
>
Hi Joshua
Thanks for the reply ok will try to grab more data so that we can figure out
the problem. Would you please tell me do i have to add "ForensicLog
logs/forensic_log" to all v.host s configs under /etc/httpd/conf.d ?
Thanks. Askar
Re: [users@httpd] apache freezes the virtual machine.
Posted by Joshua Slive <jo...@slive.ca>.
On Dec 17, 2007 2:24 PM, Asrai khn <as...@gmail.com> wrote:
> Hi we are running our shared web server under vm (UML) distro FC7 its
> working fine but once in 10-15 days apache shoots virtual machine load very
> hight upto 20-30 and all swap been consumed. if some how we get into the vm
> via ssh stop/start apache normalize the vm which clearly show that is apache
> that freezing and shooting the load on vm.
>
> Any idea how to make this vm stable?
The only thing we know here is that for some reason apache is sucking
up tons of memory under a specific circumstance. This is likely due to
a bug in one of your programs that is running under apache (a php
script, for example), but could also be due to a bug in apache httpd
intself or in some third-party module you are running.
One useful tool to figure out what request is triggering the problem
is mod_log_forensic. But without more debugging, there is nothing we
can suggest.
Joshua.
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org