You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@httpd.apache.org by Phong Thai <ph...@gmail.com> on 2026/02/25 04:11:35 UTC
[users@httpd] Apache prefork overload caused by Googlebot crawling many host-routed sites
Hello,
I’m looking for advice on handling crawler-driven overload in an Apache
prefork environment.
Environment:
- Apache httpd with prefork MPM
- CentOS 7.4
- ~2 CPU / 4 GB RAM
- prefork must remain in use
Architecture summary:
- Multiple main domains
- Tens of thousands of very small sites, each with its own hostname
- All hostnames are routed through a central VirtualHost using
vhost-level rewrite rules (no .htaccess)
- Each hostname maps dynamically to a directory such as:
/app/sites/{unique-sub-domain-slug}/
Under normal conditions the system behaves well.
Issue:
When Googlebot crawls these small sites, Apache load spikes severely
(load averages > 200). httpd processes grow rapidly and many sites
become unreachable until crawler activity subsides. Main domains remain
responsive during these events.
Steps already taken:
- All rewrite logic moved from .htaccess to VirtualHost
- AllowOverride disabled
- Conservative timeouts and connection limits applied
- Resources increased compared to previous smaller deployment
This same design handled ~150 sites reasonably well in the past. With a
much larger number of sites, overload now happens daily.
My questions:
- Is this a known failure mode of prefork under heavy crawler activity?
- Are there Apache-level techniques to limit crawler impact without
blocking Googlebot?
- In similar setups, what usually becomes the bottleneck first: rewrite
processing, filesystem checks, or process spawning?
Any insight or real-world experience would be greatly appreciated.
Thank you.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Apache prefork overload caused by Googlebot crawling many host-routed sites
Posted by Tatsuki Makino <ta...@hotmail.com>.
Hello.
We know, there are images that have been built up in the past :)
CGI is slow because it forks a process for each request.
If it is a module of the Apache HTTP server, it is always on standby and continues running. For example, mod_perl or mod_php.
Since when did FastCGI become such an important IPC? :)
Not currently... but in the version I was using when I enabled HTTP/2 on the Apache HTTP server, the prefork MPM could not be used for the module for HTTP/2.
It's not just load issues, there are increasing situations where prefork MPM cannot be used.
Regards.
On 2026/03/02 18:12, E.S. Rosenberg wrote:
> I may be wrong on this but I think people move to things like lighttpd for
> the HTTP/3 support.
>
> Apache mpm-event + php-fpm works great for my uses.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Apache prefork overload caused by Googlebot crawling many host-routed sites
Posted by "E.S. Rosenberg" <es...@gmail.com>.
I may be wrong on this but I think people move to things like lighttpd for
the HTTP/3 support.
Apache mpm-event + php-fpm works great for my uses.
HTH,
Eliyahu - אליהו
Op ma 2 mrt 2026 om 08:13 schreef Tatsuki Makino <tatsuki_makino@hotmail.com
>:
> Thank you very much for your reply.
>
> I had thought that one of the reasons why there has been a recent trend of
> migrating to other HTTP servers is that even mod_php adapted to
> multi-threaded operation cannot operate stably for a long period on
> multi-threaded MPM of Apache HTTP server.
> I have also already tried using php-fpm with lighttpd.
> However, if the use of mod_php is no longer recommended with the
> multi-threaded MPM of the Apache HTTP Server and the use of FastCGI is
> recommended, there is no need to change the front-end HTTP server.
> Is it correct to understand it that way?
>
> Regards.
>
> On 2026/02/27 10:01, Frank Gingras wrote:
> > On Thu, Feb 26, 2026 at 7:02 PM Tatsuki Makino <
> tatsuki_makino@hotmail.com>
> > wrote:
> >> When combined with PHP and the like, it sometimes causes unexpected
> >> terminations, doesn't it?
> >>
> > Mixing httpd with the mod_php DSO is really not the best way to deploy a
> > server. What you want instead is to use a threaded mpm like event, and
> > proxy to pools of php-fpm processes using proxy_fcgi.
> >
> > This allows for a very large number of httpd workers, and a relatively
> > limited number of fpm processes. Then, you can apply caching as needed
> to
> > prevent requests from tying up fpm workers for too long, and
> over-spawning
> > heavy processes.
> >
> > Back then when using prefork and mod_php, memory leaks did indeed occur
> > over time, so restarting the workers was needed. This is really no
> longer
> > a major concern.
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
>
>
Re: [users@httpd] Apache prefork overload caused by Googlebot crawling many host-routed sites
Posted by Tatsuki Makino <ta...@hotmail.com>.
Thank you very much for your reply.
I had thought that one of the reasons why there has been a recent trend of migrating to other HTTP servers is that even mod_php adapted to multi-threaded operation cannot operate stably for a long period on multi-threaded MPM of Apache HTTP server.
I have also already tried using php-fpm with lighttpd.
However, if the use of mod_php is no longer recommended with the multi-threaded MPM of the Apache HTTP Server and the use of FastCGI is recommended, there is no need to change the front-end HTTP server.
Is it correct to understand it that way?
Regards.
On 2026/02/27 10:01, Frank Gingras wrote:
> On Thu, Feb 26, 2026 at 7:02 PM Tatsuki Makino <ta...@hotmail.com>
> wrote:
>> When combined with PHP and the like, it sometimes causes unexpected
>> terminations, doesn't it?
>>
> Mixing httpd with the mod_php DSO is really not the best way to deploy a
> server. What you want instead is to use a threaded mpm like event, and
> proxy to pools of php-fpm processes using proxy_fcgi.
>
> This allows for a very large number of httpd workers, and a relatively
> limited number of fpm processes. Then, you can apply caching as needed to
> prevent requests from tying up fpm workers for too long, and over-spawning
> heavy processes.
>
> Back then when using prefork and mod_php, memory leaks did indeed occur
> over time, so restarting the workers was needed. This is really no longer
> a major concern.
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Apache prefork overload caused by Googlebot crawling many host-routed sites
Posted by Frank Gingras <th...@apache.org>.
On Thu, Feb 26, 2026 at 7:02 PM Tatsuki Makino <ta...@hotmail.com>
wrote:
> Hello.
> Sorry for cutting in from the side.
>
> On 2026/02/25 23:48, Frank Gingras wrote:
> > The solution is really to use the event mpm here - why are you bound to
> use
> > the prefork approach?
> When combined with PHP and the like, it sometimes causes unexpected
> terminations, doesn't it?
> It still seems to be unresolved, and the cause is unknown.
> It seems wise to set it to voluntarily shut down and restart after
> handling a certain number of requests.
> Prefork is what performs that for each request individually :)
>
> Regards.
>
>
Mixing httpd with the mod_php DSO is really not the best way to deploy a
server. What you want instead is to use a threaded mpm like event, and
proxy to pools of php-fpm processes using proxy_fcgi.
This allows for a very large number of httpd workers, and a relatively
limited number of fpm processes. Then, you can apply caching as needed to
prevent requests from tying up fpm workers for too long, and over-spawning
heavy processes.
Back then when using prefork and mod_php, memory leaks did indeed occur
over time, so restarting the workers was needed. This is really no longer
a major concern.
Re: [users@httpd] Apache prefork overload caused by Googlebot crawling many host-routed sites
Posted by Tatsuki Makino <ta...@hotmail.com>.
Hello.
Sorry for cutting in from the side.
On 2026/02/25 23:48, Frank Gingras wrote:
> The solution is really to use the event mpm here - why are you bound to use
> the prefork approach?
When combined with PHP and the like, it sometimes causes unexpected terminations, doesn't it?
It still seems to be unresolved, and the cause is unknown.
It seems wise to set it to voluntarily shut down and restart after handling a certain number of requests.
Prefork is what performs that for each request individually :)
Regards.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Apache prefork overload caused by Googlebot crawling many host-routed sites
Posted by Frank Gingras <th...@apache.org>.
On Wed, Feb 25, 2026 at 5:47 AM Phong Thai <ph...@gmail.com> wrote:
> Yes, I’m aware of fake Googlebot traffic and that is a valid concern.
>
> I’m verifying Googlebot using reverse DNS (crawl-*.googlebot.com),
> and I do see both real Googlebot and a significant amount of
> cloud-provider traffic (Azure/AWS) spoofing UA.
>
> I already filter a large portion of obvious bot traffic at the
> network / firewall level, but the overload still occurs specifically
> when legitimate crawlers hit many hostnames in parallel.
>
> The difficulty is that with prefork, processes are spawned very early
> during vhost and rewrite evaluation, so even valid crawlers can
> exhaust memory before any application-level throttling applies.
>
> I’m trying to understand if there are Apache-level techniques
> to reduce rewrite / vhost routing cost per request,
> without blocking or misleading real Googlebot.
>
> On Wed, Feb 25, 2026, 4:15 PM Marc <Ma...@f1-outsourcing.eu> wrote:
>
>> Are you sure it is googlebot and not fake bots. I have 400k requests per
>> day from shit Microsoft Azure mostly. Try and filter out the crap, so you
>> have more resources left for real traffic.
>>
>> I have a honeypot page with robots.txt,
>> 5m cron, everything >100 requests goes to ipset blacklist.
>> Everthing blacklisted is redicted to lightweight html only page.
>>
>> I think this only works on ipv4 as these are not abundant.
>>
>> PS. maybe crawl delay in robots?
>> PPS. upgrading also helps with performance
>>
>>
>> >
>> >
>> > Hello,
>> >
>> > I’m looking for advice on handling crawler-driven overload in an
>> > Apache
>> > prefork environment.
>> >
>> > Environment:
>> > - Apache httpd with prefork MPM
>> > - CentOS 7.4
>> > - ~2 CPU / 4 GB RAM
>> > - prefork must remain in use
>> >
>> > Architecture summary:
>> > - Multiple main domains
>> > - Tens of thousands of very small sites, each with its own
>> hostname
>> > - All hostnames are routed through a central VirtualHost using
>> > vhost-level rewrite rules (no .htaccess)
>> > - Each hostname maps dynamically to a directory such as:
>> > /app/sites/{unique-sub-domain-slug}/
>> >
>> > Under normal conditions the system behaves well.
>> >
>> > Issue:
>> > When Googlebot crawls these small sites, Apache load spikes
>> > severely
>> > (load averages > 200). httpd processes grow rapidly and many sites
>> > become unreachable until crawler activity subsides. Main domains
>> > remain
>> > responsive during these events.
>> >
>> > Steps already taken:
>> > - All rewrite logic moved from .htaccess to VirtualHost
>> > - AllowOverride disabled
>> > - Conservative timeouts and connection limits applied
>> > - Resources increased compared to previous smaller deployment
>> >
>> > This same design handled ~150 sites reasonably well in the past.
>> > With a
>> > much larger number of sites, overload now happens daily.
>> >
>> > My questions:
>> > - Is this a known failure mode of prefork under heavy crawler
>> > activity?
>> > - Are there Apache-level techniques to limit crawler impact
>> without
>> > blocking Googlebot?
>> > - In similar setups, what usually becomes the bottleneck first:
>> > rewrite
>> > processing, filesystem checks, or process spawning?
>> >
>> > Any insight or real-world experience would be greatly appreciated.
>>
>
The solution is really to use the event mpm here - why are you bound to use
the prefork approach?
With prefork, the only way to scale is to pre-spawn tons of workers up to
80% of your available memory for httpd, and make sure the processes are not
killed.
Re: [users@httpd] Apache prefork overload caused by Googlebot crawling many host-routed sites
Posted by Phong Thai <ph...@gmail.com>.
Yes, I’m aware of fake Googlebot traffic and that is a valid concern.
I’m verifying Googlebot using reverse DNS (crawl-*.googlebot.com),
and I do see both real Googlebot and a significant amount of
cloud-provider traffic (Azure/AWS) spoofing UA.
I already filter a large portion of obvious bot traffic at the
network / firewall level, but the overload still occurs specifically
when legitimate crawlers hit many hostnames in parallel.
The difficulty is that with prefork, processes are spawned very early
during vhost and rewrite evaluation, so even valid crawlers can
exhaust memory before any application-level throttling applies.
I’m trying to understand if there are Apache-level techniques
to reduce rewrite / vhost routing cost per request,
without blocking or misleading real Googlebot.
On Wed, Feb 25, 2026, 4:15 PM Marc <Ma...@f1-outsourcing.eu> wrote:
> Are you sure it is googlebot and not fake bots. I have 400k requests per
> day from shit Microsoft Azure mostly. Try and filter out the crap, so you
> have more resources left for real traffic.
>
> I have a honeypot page with robots.txt,
> 5m cron, everything >100 requests goes to ipset blacklist.
> Everthing blacklisted is redicted to lightweight html only page.
>
> I think this only works on ipv4 as these are not abundant.
>
> PS. maybe crawl delay in robots?
> PPS. upgrading also helps with performance
>
>
> >
> >
> > Hello,
> >
> > I’m looking for advice on handling crawler-driven overload in an
> > Apache
> > prefork environment.
> >
> > Environment:
> > - Apache httpd with prefork MPM
> > - CentOS 7.4
> > - ~2 CPU / 4 GB RAM
> > - prefork must remain in use
> >
> > Architecture summary:
> > - Multiple main domains
> > - Tens of thousands of very small sites, each with its own hostname
> > - All hostnames are routed through a central VirtualHost using
> > vhost-level rewrite rules (no .htaccess)
> > - Each hostname maps dynamically to a directory such as:
> > /app/sites/{unique-sub-domain-slug}/
> >
> > Under normal conditions the system behaves well.
> >
> > Issue:
> > When Googlebot crawls these small sites, Apache load spikes
> > severely
> > (load averages > 200). httpd processes grow rapidly and many sites
> > become unreachable until crawler activity subsides. Main domains
> > remain
> > responsive during these events.
> >
> > Steps already taken:
> > - All rewrite logic moved from .htaccess to VirtualHost
> > - AllowOverride disabled
> > - Conservative timeouts and connection limits applied
> > - Resources increased compared to previous smaller deployment
> >
> > This same design handled ~150 sites reasonably well in the past.
> > With a
> > much larger number of sites, overload now happens daily.
> >
> > My questions:
> > - Is this a known failure mode of prefork under heavy crawler
> > activity?
> > - Are there Apache-level techniques to limit crawler impact without
> > blocking Googlebot?
> > - In similar setups, what usually becomes the bottleneck first:
> > rewrite
> > processing, filesystem checks, or process spawning?
> >
> > Any insight or real-world experience would be greatly appreciated.
>
RE: [users@httpd] Apache prefork overload caused by Googlebot crawling many host-routed sites
Posted by Marc <Ma...@f1-outsourcing.eu>.
Are you sure it is googlebot and not fake bots. I have 400k requests per day from shit Microsoft Azure mostly. Try and filter out the crap, so you have more resources left for real traffic.
I have a honeypot page with robots.txt,
5m cron, everything >100 requests goes to ipset blacklist.
Everthing blacklisted is redicted to lightweight html only page.
I think this only works on ipv4 as these are not abundant.
PS. maybe crawl delay in robots?
PPS. upgrading also helps with performance
>
>
> Hello,
>
> I’m looking for advice on handling crawler-driven overload in an
> Apache
> prefork environment.
>
> Environment:
> - Apache httpd with prefork MPM
> - CentOS 7.4
> - ~2 CPU / 4 GB RAM
> - prefork must remain in use
>
> Architecture summary:
> - Multiple main domains
> - Tens of thousands of very small sites, each with its own hostname
> - All hostnames are routed through a central VirtualHost using
> vhost-level rewrite rules (no .htaccess)
> - Each hostname maps dynamically to a directory such as:
> /app/sites/{unique-sub-domain-slug}/
>
> Under normal conditions the system behaves well.
>
> Issue:
> When Googlebot crawls these small sites, Apache load spikes
> severely
> (load averages > 200). httpd processes grow rapidly and many sites
> become unreachable until crawler activity subsides. Main domains
> remain
> responsive during these events.
>
> Steps already taken:
> - All rewrite logic moved from .htaccess to VirtualHost
> - AllowOverride disabled
> - Conservative timeouts and connection limits applied
> - Resources increased compared to previous smaller deployment
>
> This same design handled ~150 sites reasonably well in the past.
> With a
> much larger number of sites, overload now happens daily.
>
> My questions:
> - Is this a known failure mode of prefork under heavy crawler
> activity?
> - Are there Apache-level techniques to limit crawler impact without
> blocking Googlebot?
> - In similar setups, what usually becomes the bottleneck first:
> rewrite
> processing, filesystem checks, or process spawning?
>
> Any insight or real-world experience would be greatly appreciated.
Re: [users@httpd] Apache prefork overload caused by Googlebot crawling many host-routed sites
Posted by Frank Gingras <th...@apache.org>.
On Tue, Feb 24, 2026 at 11:12 PM Phong Thai <ph...@gmail.com> wrote:
> Hello,
>
> I’m looking for advice on handling crawler-driven overload in an Apache
> prefork environment.
>
> Environment:
> - Apache httpd with prefork MPM
> - CentOS 7.4
> - ~2 CPU / 4 GB RAM
> - prefork must remain in use
>
> Architecture summary:
> - Multiple main domains
> - Tens of thousands of very small sites, each with its own hostname
> - All hostnames are routed through a central VirtualHost using
> vhost-level rewrite rules (no .htaccess)
> - Each hostname maps dynamically to a directory such as:
> /app/sites/{unique-sub-domain-slug}/
>
> Under normal conditions the system behaves well.
>
> Issue:
> When Googlebot crawls these small sites, Apache load spikes severely
> (load averages > 200). httpd processes grow rapidly and many sites
> become unreachable until crawler activity subsides. Main domains remain
> responsive during these events.
>
> Steps already taken:
> - All rewrite logic moved from .htaccess to VirtualHost
> - AllowOverride disabled
> - Conservative timeouts and connection limits applied
> - Resources increased compared to previous smaller deployment
>
> This same design handled ~150 sites reasonably well in the past. With a
> much larger number of sites, overload now happens daily.
>
> My questions:
> - Is this a known failure mode of prefork under heavy crawler activity?
> - Are there Apache-level techniques to limit crawler impact without
> blocking Googlebot?
> - In similar setups, what usually becomes the bottleneck first: rewrite
> processing, filesystem checks, or process spawning?
>
> Any insight or real-world experience would be greatly appreciated.
>
> Thank you.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
>
>
Why is prefork required here? What non-thread-safe module are you loading?
4GB of RAM with large-ish processes is a recipe for disaster, either way.
Re: [users@httpd] Apache prefork overload caused by Googlebot crawling many host-routed sites
Posted by Sean Conner <sp...@conman.org>.
It was thus said that the Great Phong Thai once stated:
> Hello,
>
> I’m looking for advice on handling crawler-driven overload in an Apache
> prefork environment.
...
> - Are there Apache-level techniques to limit crawler impact without
> blocking Googlebot?
I found mod_limitipconn that works well in my case. It limits the number
of concurrent connections from a single IP address to a configurable number.
I changed the code to return a 429 instead of 503, but that's the only
change I've made. I set globally the number of concurrent connections to 20
(again, from a single IP address).
-spc
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org