You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@httpd.apache.org by Michele Mase' <mi...@gmail.com> on 2017/01/24 18:10:06 UTC

[users@httpd] Unable to fork new process

All,
I've a apache web server with 600 web sites as reverse proxy. The average
of req. is more than 100 req/sec during all the day.
The apache is a 2.2.15 release 56.el6_8.3 (RedHat 6.x system up to date) +
mod_ssl.
Every 10 days, the system has problems, and in the error log i found dozen
of:

 [error] (12)Cannot allocate memory: fork: Unable to fork new process

The temporary solution for me is the following: after an alert mail from
the system monitoring server, I must restart the httpd using:
service httpd stop; killall httpd.worker; killall -9 httpd.worker; service
httpd start
Adding more ram to the system does not help.

Any suggestion?
Best regards

apachectl -l
Compiled in modules:
  core.c
  worker.c
  http_core.c
  mod_so.c

apachectl -M
Loaded Modules:
 core_module (static)
 mpm_worker_module (static)
 http_module (static)
 so_module (static)
 log_post_module (shared)
 evasive20_module (shared)
 auth_basic_module (shared)
 auth_digest_module (shared)
 authn_file_module (shared)
 authn_alias_module (shared)
 authn_anon_module (shared)
 authn_dbm_module (shared)
 authn_default_module (shared)
 authz_host_module (shared)
 authz_user_module (shared)
 authz_owner_module (shared)
 authz_groupfile_module (shared)
 authz_dbm_module (shared)
 authz_default_module (shared)
 ldap_module (shared)
 authnz_ldap_module (shared)
 include_module (shared)
 log_config_module (shared)
 logio_module (shared)
 env_module (shared)
 ext_filter_module (shared)
 mime_magic_module (shared)
 expires_module (shared)
 deflate_module (shared)
 headers_module (shared)
 usertrack_module (shared)
 setenvif_module (shared)
 mime_module (shared)
 dav_module (shared)
 status_module (shared)
 autoindex_module (shared)
 info_module (shared)
 dav_fs_module (shared)
 vhost_alias_module (shared)
 negotiation_module (shared)
 dir_module (shared)
 actions_module (shared)
 speling_module (shared)
 userdir_module (shared)
 alias_module (shared)
 substitute_module (shared)
 rewrite_module (shared)
 proxy_module (shared)
 proxy_balancer_module (shared)
 proxy_ftp_module (shared)
 proxy_http_module (shared)
 proxy_ajp_module (shared)
 proxy_connect_module (shared)
 cache_module (shared)
 suexec_module (shared)
 disk_cache_module (shared)
 cgi_module (shared)
 version_module (shared)
 unique_id_module (shared)
 ssl_module (shared)
 reqtimeout_module (shared)
Syntax OK

RE: [users@httpd] Unable to fork new process

Posted by Darryl Philip Baker <da...@northwestern.edu>.
Thanks Michele. I was going to investigate Apache HTTPD 2.4 for this issue since we use “systemctl reload httpd” every time we modify our configuration. I’m not sure if we will ever get to 10^6 reload before the system is rebooted for a new kernel or a new version of systemd.  We are using RHEL7 and the wonderfully well written systemd. :-P

Darryl Baker
Sr. System Administrator
Northwestern | Information Technology
www.it.northwestern.edu

From: Michele Mase' [mailto:michele.mase@gmail.com]
Sent: January 27, 2017 2:03 AM
To: Httpd Users List <us...@httpd.apache.org>
Subject: Re: [users@httpd] Unable to fork new process

(12)Cannot allocate memory: AH00283: fork: Unable to fork new process again!
I suppose that apache2.4 too shoul be affected by the same problem. Apache 2.4 crashed, in the same manner of apache2.2. The difference is between the number of reloads: there are more than 10^6 times of reloads to crash apache 2.4.x. I started the loop on Yesterday, at 13:49 and the system crashed at 20:03, after 6h14minutes.
The impact of apache 2.4.x could be moderate, it's difficult to plan 10^6 of httpd reloads.
[Thu Jan 26 13:49:33.450510 2017] [mpm_worker:notice] [pid 18097:tid 140259330607072] AH00295: caught SIGTERM, shutting down
[Thu Jan 26 13:56:47.388838 2017] [core:notice] [pid 18517:tid 140237258659808] SELinux policy enabled; httpd running as context unconfined_u:system_r:initrc_t:s0
[Thu Jan 26 13:56:47.442743 2017] [suexec:notice] [pid 18517:tid 140237258659808] AH01232: suEXEC mechanism enabled (wrapper: /opt/rh/httpd24/root/usr/sbin/suexec)
[Thu Jan 26 13:56:47.673152 2017] [ssl:warn] [pid 18517:tid 140237258659808] AH02292: Init: Name-based SSL virtual hosts only work for clients with TLS server name indication support (RFC 4366)
[Thu Jan 26 13:56:48.745210 2017] [auth_digest:notice] [pid 18825:tid 140237258659808] AH01757: generating secret for digest authentication ...
[Thu Jan 26 13:56:48.776278 2017] [lbmethod_heartbeat:notice] [pid 18825:tid 140237258659808] AH02282: No slotmem from mod_heartmonitor
[Thu Jan 26 13:56:48.835965 2017] [ssl:warn] [pid 18825:tid 140237258659808] AH01873: Init: Session Cache is not configured [hint: SSLSessionCache]
[Thu Jan 26 13:56:49.040532 2017] [ssl:warn] [pid 18825:tid 140237258659808] AH02292: Init: Name-based SSL virtual hosts only work for clients with TLS server name indication support (RFC 4366)
[Thu Jan 26 13:56:49.107329 2017] [mpm_worker:notice] [pid 18825:tid 140237258659808] AH00292: Apache/2.4.6 (Red Hat) OpenSSL/1.0.0-fips configured -- resuming normal operations
[Thu Jan 26 13:56:49.112274 2017] [core:notice] [pid 18825:tid 140237258659808] AH00094: Command line: '/opt/rh/httpd24/root/usr/sbin/httpd'
[root@kvm164 httpd]# tail -f /opt/rh/httpd24/root/var/log/httpd/error_log
[Thu Jan 26 13:49:33.450510 2017] [mpm_worker:notice] [pid 18097:tid 140259330607072] AH00295: caught SIGTERM, shutting down
......
and finally the crash
......
[Thu Jan 26 20:03:11.052290 2017] [mpm_worker:error] [pid 19594:tid 140421173676000] (12)Cannot allocate memory: AH00283: fork: Unable to fork new process
[Thu Jan 26 20:03:11.126645 2017] [mpm_worker:notice] [pid 19594:tid 140421173676000] AH00292: Apache/2.4.6 (Red Hat) OpenSSL/1.0.0-fips configured -- resuming normal operations
[Thu Jan 26 20:03:11.126958 2017] [core:notice] [pid 19594:tid 140421173676000] AH00094: Command line: '/opt/rh/httpd24/root/usr/sbin/httpd'
[Thu Jan 26 20:03:11.187777 2017] [mpm_worker:notice] [pid 19594:tid 140421173676000] AH00298: SIGHUP received.  Attempting to restart
(12)Cannot allocate memory: AH00104: unable to start piped log program '/usr/sbin/rotatelogs /etc/httpd/logs/www.example.com/www.example.com_ssl_access_log_192.168.122.165<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.example.com_www.example.com-5Fssl-5Faccess-5Flog-5F192.168.122.165&d=CwMFaQ&c=yHlS04HhBraes5BQ9ueu5zKhE7rtNXt_d012z2PA6ws&r=tdje61_VHSXp608oLofeaJnnzo2Sr9_Cdcp70bBEtQ8&m=8nB-oMzW5gkbLxVoul3ab53VPskpqeqBETOS0ByZ5YE&s=G4MibIaiEySNZXyQGhH7gIvKexXt9U_jrlqReVR_iBo&e=> 86400'
[Thu Jan 26 20:03:15.825822 2017] [:emerg] [pid 19594:tid 140421173676000] AH00019: Unable to open logs, exiting
Maybe somebody sould fix the code.
Best Regards
Michele MAsè

On Thu, Jan 26, 2017 at 5:58 PM, Michele Mase' <mi...@gmail.com>> wrote:
Ok, I've just tried the loop within apache2.4.x and I confirm it is not affected by the error.
So, what could be the better solution?
ASAP, I plan to migrate my system on apache 2.4.x.
In the meantime it would be nice that version 2.2 was corrected.
Best Regards
Michele MAsè

On Thu, Jan 26, 2017 at 1:27 AM, Yann Ylavic <yl...@gmail.com>> wrote:
Hi,

On Wed, Jan 25, 2017 at 10:33 PM, Michele Mase' <mi...@gmail.com>> wrote:
> <?php

I checked restarts with valgrind on latest 2.2.x and found this fixes:

Index: modules/ssl/mod_ssl.c
===================================================================
--- modules/ssl/mod_ssl.c    (revision 1778094)
+++ modules/ssl/mod_ssl.c    (working copy)
@@ -277,7 +277,12 @@ static apr_status_t ssl_cleanup_pre_config(void *d
     /* Don't call ERR_free_strings here; ERR_load_*_strings only
      * actually load the error strings once per process due to static
      * variable abuse in OpenSSL. */
+#if (OPENSSL_VERSION_NUMBER >= 0x00090805f)
+    ERR_free_strings();
+#endif

+    sk_SSL_COMP_free(SSL_COMP_get_compression_methods());
+
     /* Also don't call CRYPTO_cleanup_all_ex_data here; any registered
      * ex_data indices may have been cached in static variables in
      * OpenSSL; removing them may cause havoc.  Notably, with OpenSSL
Index: modules/ssl/ssl_util_ssl.c
===================================================================
--- modules/ssl/ssl_util_ssl.c    (revision 1778094)
+++ modules/ssl/ssl_util_ssl.c    (working copy)
@@ -311,6 +311,7 @@ BOOL SSL_X509_isSGC(X509 *cert)
                     break;
                 }
             }
+            EXTENDED_KEY_USAGE_free(sk);
         }
     }
     return is_sgc;
__

> Tomorrow, probably i'll open a ticket with redhat.

The first one (ERR_free_strings) is fixed in 2.4.x and seems to be
backported in redhat's 2.2.* already (at least in
"httpd-2.2.15-56.el6_8.3.src.rpm").

The second one (sk_SSL_COMP_free) is nowhere, neither in httpd nor
redhat (AFAICT) releases.

The last and biggest one (EXTENDED_KEY_USAGE_free) is 2.2 code only
(2.4 not concerned), and seems to affect both httpd-2.x and redhat's.

I won't be able to verify how it affects the different openssl
versions (hence commit anything) in the next few days, just wanted to
notify before being away...

Regards,
Yann.

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org<ma...@httpd.apache.org>
For additional commands, e-mail: users-help@httpd.apache.org<ma...@httpd.apache.org>



Re: [users@httpd] Unable to fork new process

Posted by Michele Mase' <mi...@gmail.com>.
(12)Cannot allocate memory: AH00283: fork: Unable to fork new process again!
I suppose that apache2.4 too shoul be affected by the same problem. Apache
2.4 crashed, in the same manner of apache2.2. The difference is between the
number of reloads: there are more than 10^6 times of reloads to crash
apache 2.4.x. I started the loop on Yesterday, at 13:49 and the system
crashed at 20:03, after 6h14minutes.
The impact of apache 2.4.x could be moderate, it's difficult to plan 10^6
of httpd reloads.
[Thu Jan 26 13:49:33.450510 2017] [mpm_worker:notice] [pid 18097:tid
140259330607072] AH00295: caught SIGTERM, shutting down
[Thu Jan 26 13:56:47.388838 2017] [core:notice] [pid 18517:tid
140237258659808] SELinux policy enabled; httpd running as context
unconfined_u:system_r:initrc_t:s0
[Thu Jan 26 13:56:47.442743 2017] [suexec:notice] [pid 18517:tid
140237258659808] AH01232: suEXEC mechanism enabled (wrapper:
/opt/rh/httpd24/root/usr/sbin/suexec)
[Thu Jan 26 13:56:47.673152 2017] [ssl:warn] [pid 18517:tid
140237258659808] AH02292: Init: Name-based SSL virtual hosts only work for
clients with TLS server name indication support (RFC 4366)
[Thu Jan 26 13:56:48.745210 2017] [auth_digest:notice] [pid 18825:tid
140237258659808] AH01757: generating secret for digest authentication ...
[Thu Jan 26 13:56:48.776278 2017] [lbmethod_heartbeat:notice] [pid
18825:tid 140237258659808] AH02282: No slotmem from mod_heartmonitor
[Thu Jan 26 13:56:48.835965 2017] [ssl:warn] [pid 18825:tid
140237258659808] AH01873: Init: Session Cache is not configured [hint:
SSLSessionCache]
[Thu Jan 26 13:56:49.040532 2017] [ssl:warn] [pid 18825:tid
140237258659808] AH02292: Init: Name-based SSL virtual hosts only work for
clients with TLS server name indication support (RFC 4366)
[Thu Jan 26 13:56:49.107329 2017] [mpm_worker:notice] [pid 18825:tid
140237258659808] AH00292: Apache/2.4.6 (Red Hat) OpenSSL/1.0.0-fips
configured -- resuming normal operations
[Thu Jan 26 13:56:49.112274 2017] [core:notice] [pid 18825:tid
140237258659808] AH00094: Command line:
'/opt/rh/httpd24/root/usr/sbin/httpd'
[root@kvm164 httpd]# tail -f /opt/rh/httpd24/root/var/log/httpd/error_log
[Thu Jan 26 13:49:33.450510 2017] [mpm_worker:notice] [pid 18097:tid
140259330607072] AH00295: caught SIGTERM, shutting down
......
and finally the crash
......
[Thu Jan 26 20:03:11.052290 2017] [mpm_worker:error] [pid 19594:tid
140421173676000] (12)Cannot allocate memory: AH00283: fork: Unable to fork
new process
[Thu Jan 26 20:03:11.126645 2017] [mpm_worker:notice] [pid 19594:tid
140421173676000] AH00292: Apache/2.4.6 (Red Hat) OpenSSL/1.0.0-fips
configured -- resuming normal operations
[Thu Jan 26 20:03:11.126958 2017] [core:notice] [pid 19594:tid
140421173676000] AH00094: Command line:
'/opt/rh/httpd24/root/usr/sbin/httpd'
[Thu Jan 26 20:03:11.187777 2017] [mpm_worker:notice] [pid 19594:tid
140421173676000] AH00298: SIGHUP received.  Attempting to restart
(12)Cannot allocate memory: AH00104: unable to start piped log program
'/usr/sbin/rotatelogs /etc/httpd/logs/
www.example.com/www.example.com_ssl_access_log_192.168.122.165 86400'
[Thu Jan 26 20:03:15.825822 2017] [:emerg] [pid 19594:tid 140421173676000]
AH00019: Unable to open logs, exiting

Maybe somebody sould fix the code.
Best Regards
Michele MAsè

On Thu, Jan 26, 2017 at 5:58 PM, Michele Mase' <mi...@gmail.com>
wrote:

> Ok, I've just tried the loop within apache2.4.x and I confirm it is not
> affected by the error.
> So, what could be the better solution?
> ASAP, I plan to migrate my system on apache 2.4.x.
> In the meantime it would be nice that version 2.2 was corrected.
> Best Regards
> Michele MAsè
>
> On Thu, Jan 26, 2017 at 1:27 AM, Yann Ylavic <yl...@gmail.com> wrote:
>
>> Hi,
>>
>> On Wed, Jan 25, 2017 at 10:33 PM, Michele Mase' <mi...@gmail.com>
>> wrote:
>> > <?php
>>
>> I checked restarts with valgrind on latest 2.2.x and found this fixes:
>>
>> Index: modules/ssl/mod_ssl.c
>> ===================================================================
>> --- modules/ssl/mod_ssl.c    (revision 1778094)
>> +++ modules/ssl/mod_ssl.c    (working copy)
>> @@ -277,7 +277,12 @@ static apr_status_t ssl_cleanup_pre_config(void *d
>>      /* Don't call ERR_free_strings here; ERR_load_*_strings only
>>       * actually load the error strings once per process due to static
>>       * variable abuse in OpenSSL. */
>> +#if (OPENSSL_VERSION_NUMBER >= 0x00090805f)
>> +    ERR_free_strings();
>> +#endif
>>
>> +    sk_SSL_COMP_free(SSL_COMP_get_compression_methods());
>> +
>>      /* Also don't call CRYPTO_cleanup_all_ex_data here; any registered
>>       * ex_data indices may have been cached in static variables in
>>       * OpenSSL; removing them may cause havoc.  Notably, with OpenSSL
>> Index: modules/ssl/ssl_util_ssl.c
>> ===================================================================
>> --- modules/ssl/ssl_util_ssl.c    (revision 1778094)
>> +++ modules/ssl/ssl_util_ssl.c    (working copy)
>> @@ -311,6 +311,7 @@ BOOL SSL_X509_isSGC(X509 *cert)
>>                      break;
>>                  }
>>              }
>> +            EXTENDED_KEY_USAGE_free(sk);
>>          }
>>      }
>>      return is_sgc;
>> __
>>
>> > Tomorrow, probably i'll open a ticket with redhat.
>>
>> The first one (ERR_free_strings) is fixed in 2.4.x and seems to be
>> backported in redhat's 2.2.* already (at least in
>> "httpd-2.2.15-56.el6_8.3.src.rpm").
>>
>> The second one (sk_SSL_COMP_free) is nowhere, neither in httpd nor
>> redhat (AFAICT) releases.
>>
>> The last and biggest one (EXTENDED_KEY_USAGE_free) is 2.2 code only
>> (2.4 not concerned), and seems to affect both httpd-2.x and redhat's.
>>
>> I won't be able to verify how it affects the different openssl
>> versions (hence commit anything) in the next few days, just wanted to
>> notify before being away...
>>
>> Regards,
>> Yann.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
>> For additional commands, e-mail: users-help@httpd.apache.org
>>
>>
>

Re: [users@httpd] Unable to fork new process

Posted by Michele Mase' <mi...@gmail.com>.
Ok, I've just tried the loop within apache2.4.x and I confirm it is not
affected by the error.
So, what could be the better solution?
ASAP, I plan to migrate my system on apache 2.4.x.
In the meantime it would be nice that version 2.2 was corrected.
Best Regards
Michele MAsè

On Thu, Jan 26, 2017 at 1:27 AM, Yann Ylavic <yl...@gmail.com> wrote:

> Hi,
>
> On Wed, Jan 25, 2017 at 10:33 PM, Michele Mase' <mi...@gmail.com>
> wrote:
> > <?php
>
> I checked restarts with valgrind on latest 2.2.x and found this fixes:
>
> Index: modules/ssl/mod_ssl.c
> ===================================================================
> --- modules/ssl/mod_ssl.c    (revision 1778094)
> +++ modules/ssl/mod_ssl.c    (working copy)
> @@ -277,7 +277,12 @@ static apr_status_t ssl_cleanup_pre_config(void *d
>      /* Don't call ERR_free_strings here; ERR_load_*_strings only
>       * actually load the error strings once per process due to static
>       * variable abuse in OpenSSL. */
> +#if (OPENSSL_VERSION_NUMBER >= 0x00090805f)
> +    ERR_free_strings();
> +#endif
>
> +    sk_SSL_COMP_free(SSL_COMP_get_compression_methods());
> +
>      /* Also don't call CRYPTO_cleanup_all_ex_data here; any registered
>       * ex_data indices may have been cached in static variables in
>       * OpenSSL; removing them may cause havoc.  Notably, with OpenSSL
> Index: modules/ssl/ssl_util_ssl.c
> ===================================================================
> --- modules/ssl/ssl_util_ssl.c    (revision 1778094)
> +++ modules/ssl/ssl_util_ssl.c    (working copy)
> @@ -311,6 +311,7 @@ BOOL SSL_X509_isSGC(X509 *cert)
>                      break;
>                  }
>              }
> +            EXTENDED_KEY_USAGE_free(sk);
>          }
>      }
>      return is_sgc;
> __
>
> > Tomorrow, probably i'll open a ticket with redhat.
>
> The first one (ERR_free_strings) is fixed in 2.4.x and seems to be
> backported in redhat's 2.2.* already (at least in
> "httpd-2.2.15-56.el6_8.3.src.rpm").
>
> The second one (sk_SSL_COMP_free) is nowhere, neither in httpd nor
> redhat (AFAICT) releases.
>
> The last and biggest one (EXTENDED_KEY_USAGE_free) is 2.2 code only
> (2.4 not concerned), and seems to affect both httpd-2.x and redhat's.
>
> I won't be able to verify how it affects the different openssl
> versions (hence commit anything) in the next few days, just wanted to
> notify before being away...
>
> Regards,
> Yann.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
>
>

Re: [users@httpd] Unable to fork new process

Posted by Yann Ylavic <yl...@gmail.com>.
Hi,

On Wed, Jan 25, 2017 at 10:33 PM, Michele Mase' <mi...@gmail.com> wrote:
> <?php

I checked restarts with valgrind on latest 2.2.x and found this fixes:

Index: modules/ssl/mod_ssl.c
===================================================================
--- modules/ssl/mod_ssl.c    (revision 1778094)
+++ modules/ssl/mod_ssl.c    (working copy)
@@ -277,7 +277,12 @@ static apr_status_t ssl_cleanup_pre_config(void *d
     /* Don't call ERR_free_strings here; ERR_load_*_strings only
      * actually load the error strings once per process due to static
      * variable abuse in OpenSSL. */
+#if (OPENSSL_VERSION_NUMBER >= 0x00090805f)
+    ERR_free_strings();
+#endif

+    sk_SSL_COMP_free(SSL_COMP_get_compression_methods());
+
     /* Also don't call CRYPTO_cleanup_all_ex_data here; any registered
      * ex_data indices may have been cached in static variables in
      * OpenSSL; removing them may cause havoc.  Notably, with OpenSSL
Index: modules/ssl/ssl_util_ssl.c
===================================================================
--- modules/ssl/ssl_util_ssl.c    (revision 1778094)
+++ modules/ssl/ssl_util_ssl.c    (working copy)
@@ -311,6 +311,7 @@ BOOL SSL_X509_isSGC(X509 *cert)
                     break;
                 }
             }
+            EXTENDED_KEY_USAGE_free(sk);
         }
     }
     return is_sgc;
__

> Tomorrow, probably i'll open a ticket with redhat.

The first one (ERR_free_strings) is fixed in 2.4.x and seems to be
backported in redhat's 2.2.* already (at least in
"httpd-2.2.15-56.el6_8.3.src.rpm").

The second one (sk_SSL_COMP_free) is nowhere, neither in httpd nor
redhat (AFAICT) releases.

The last and biggest one (EXTENDED_KEY_USAGE_free) is 2.2 code only
(2.4 not concerned), and seems to affect both httpd-2.x and redhat's.

I won't be able to verify how it affects the different openssl
versions (hence commit anything) in the next few days, just wanted to
notify before being away...

Regards,
Yann.

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org


Re: [users@httpd] Unable to fork new process

Posted by Michele Mase' <mi...@gmail.com>.
 <?php
$logpath='/var/log/apache2/error.log';
$pos=0;
if(is_file($logpath))
        $pos=filesize($logpath);
$count=0;
$start=time();
while(true) {
        system('/etc/init.d/apache2 reload');
        $count++;
        $fh=fopen($logpath,'r');
        fseek($fh,$pos);
        while(!feof($fh)) {
                $line=fgets($fh);
                if(strpos($line,'[error]')!==false) {
                        print $line;
                        print $count." reloads\n";
                        print (time()-$start)." seconds\n";
                        fclose($fh);
                        exit(0);
                }
        }
        $pos=ftell($fh);
        fclose($fh);
}
?>







Try it yourself (change the location of error log as needed), tomorrow I
will try with an apche 2.4.x version on redhat system. To test it is quite
simple. Start an apache on a test server; 10 virtualhosts, 100k reloads and
it hangs. 260vhosts, 1k reloads and it hangs. Let me know.
Tomorrow, probably i'll open a ticket with redhat.
Ciao




On Wed, Jan 25, 2017 at 7:51 PM, Darryl Philip Baker <
darryl.baker@northwestern.edu> wrote:

> I would look at /etc/security/limits.conf and /etc/security/limits.d and
> see if there is a special memory entry for the user httpd is running as.
> If there is you could change it but this just moves the point of failure.
> Have you opened a ticket with Red Hat? They may be able to find a the fix
> from a newer version of Apache and backport it.
>
>
>
> As to how Apache 2.4 handles reloads it gracefully stops the existing
> instances and spawns new ones I have not done an exhaustive test of your
> problem but I handle about 200 virtual hosts on the central web server here
> at the University. I do reloads often and have not had the issue with
> Apache 2.4.
>
>
>
> *Darryl Baker*
>
> *Sr. System Administrator*
>
> *Northwestern* | Information Technology
>
> www.it.northwestern.edu
>
>
>
> *From:* Michele Mase' [mailto:michele.mase@gmail.com]
> *Sent:* January 25, 2017 11:50 AM
> *To:* Httpd Users List <us...@httpd.apache.org>
> *Subject:* Re: [users@httpd] Unable to fork new process
>
>
>
> I've tried with httpd.event, same result. Using a demo host with 260
> vhosts, the httpd hangs after "only" 1000 reloads. The relationship seems
> to be the following:
> number of virtualhosts = x/number of reloads
>
> 10 virtualhosts = 100000 reloads
>
> 100 virtualhosts = 1000 reloads
>
> Tomorrow I 'll try to test httpd 2.4.x.
>
> Other better ideas?
>
> Regards
>
>
>
> On Wed, Jan 25, 2017 at 4:42 PM, Darryl Philip Baker <
> darryl.baker@northwestern.edu> wrote:
>
> The Red Hat software collection are Red Hat rpms they are just in a
> non-default repository. The have all the same support from Red Hat as the
> defaults. I’ve included a link to the latest version but Apache 2.4 has
> been in there since the very first release.
>
>
>
> https://www.redhat.com/en/about/press-releases/red-hat-
> software-collections-23-and-red-hat-developer-toolset-6-
> now-generally-available
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.redhat.com_en_about_press-2Dreleases_red-2Dhat-2Dsoftware-2Dcollections-2D23-2Dand-2Dred-2Dhat-2Ddeveloper-2Dtoolset-2D6-2Dnow-2Dgenerally-2Davailable&d=CwMFaQ&c=yHlS04HhBraes5BQ9ueu5zKhE7rtNXt_d012z2PA6ws&r=tdje61_VHSXp608oLofeaJnnzo2Sr9_Cdcp70bBEtQ8&m=POnf8iSgk-7Y5sDIvZay4HMtxJuPXaV2aPvP6CGf32Q&s=UwHKp55lJ5QwqSmAcLj2eCJS9G9QIjWSOxvn9k4hUZM&e=>
>
>
>
> Darryl Baker
>
> NIT - CI -DAPS
>
> X76674
>
>
>
> *From:* Michele Mase' [mailto:michele.mase@gmail.com]
> *Sent:* January 25, 2017 8:59 AM
> *To:* Httpd Users List <us...@httpd.apache.org>
> *Subject:* Re: [users@httpd] Unable to fork new process
>
>
>
>
>
> On Wed, Jan 25, 2017 at 3:39 PM, Darryl Philip Baker <
> darryl.baker@northwestern.edu> wrote:
>
> e
>
>
>
> Due to redhat subscription, we must use redhat's rpms. Now I'm trying the
> php loop against another web server. Then I will try with apache 2.4.x in a
> offline env. My question is the same:
> After xx reloads under yy load average in zz time the error will appears
> "every time".
> Any smarter suggestion?
>
>
>

RE: [users@httpd] Unable to fork new process

Posted by Darryl Philip Baker <da...@northwestern.edu>.
I would look at /etc/security/limits.conf and /etc/security/limits.d and see if there is a special memory entry for the user httpd is running as.  If there is you could change it but this just moves the point of failure. Have you opened a ticket with Red Hat? They may be able to find a the fix from a newer version of Apache and backport it.

As to how Apache 2.4 handles reloads it gracefully stops the existing instances and spawns new ones I have not done an exhaustive test of your problem but I handle about 200 virtual hosts on the central web server here at the University. I do reloads often and have not had the issue with Apache 2.4.

Darryl Baker
Sr. System Administrator
Northwestern | Information Technology
www.it.northwestern.edu

From: Michele Mase' [mailto:michele.mase@gmail.com]
Sent: January 25, 2017 11:50 AM
To: Httpd Users List <us...@httpd.apache.org>
Subject: Re: [users@httpd] Unable to fork new process

I've tried with httpd.event, same result. Using a demo host with 260 vhosts, the httpd hangs after "only" 1000 reloads. The relationship seems to be the following:
number of virtualhosts = x/number of reloads
10 virtualhosts = 100000 reloads
100 virtualhosts = 1000 reloads
Tomorrow I 'll try to test httpd 2.4.x.
Other better ideas?
Regards

On Wed, Jan 25, 2017 at 4:42 PM, Darryl Philip Baker <da...@northwestern.edu>> wrote:
The Red Hat software collection are Red Hat rpms they are just in a non-default repository. The have all the same support from Red Hat as the defaults. I’ve included a link to the latest version but Apache 2.4 has been in there since the very first release.

https://www.redhat.com/en/about/press-releases/red-hat-software-collections-23-and-red-hat-developer-toolset-6-now-generally-available<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.redhat.com_en_about_press-2Dreleases_red-2Dhat-2Dsoftware-2Dcollections-2D23-2Dand-2Dred-2Dhat-2Ddeveloper-2Dtoolset-2D6-2Dnow-2Dgenerally-2Davailable&d=CwMFaQ&c=yHlS04HhBraes5BQ9ueu5zKhE7rtNXt_d012z2PA6ws&r=tdje61_VHSXp608oLofeaJnnzo2Sr9_Cdcp70bBEtQ8&m=POnf8iSgk-7Y5sDIvZay4HMtxJuPXaV2aPvP6CGf32Q&s=UwHKp55lJ5QwqSmAcLj2eCJS9G9QIjWSOxvn9k4hUZM&e=>

Darryl Baker
NIT - CI -DAPS
X76674

From: Michele Mase' [mailto:michele.mase@gmail.com<ma...@gmail.com>]
Sent: January 25, 2017 8:59 AM
To: Httpd Users List <us...@httpd.apache.org>>
Subject: Re: [users@httpd] Unable to fork new process


On Wed, Jan 25, 2017 at 3:39 PM, Darryl Philip Baker <da...@northwestern.edu>> wrote:
e

Due to redhat subscription, we must use redhat's rpms. Now I'm trying the php loop against another web server. Then I will try with apache 2.4.x in a offline env. My question is the same:
After xx reloads under yy load average in zz time the error will appears "every time".
Any smarter suggestion?


Re: [users@httpd] Unable to fork new process

Posted by Michele Mase' <mi...@gmail.com>.
I've tried with httpd.event, same result. Using a demo host with 260
vhosts, the httpd hangs after "only" 1000 reloads. The relationship seems
to be the following:
number of virtualhosts = x/number of reloads
10 virtualhosts = 100000 reloads
100 virtualhosts = 1000 reloads
Tomorrow I 'll try to test httpd 2.4.x.
Other better ideas?
Regards

On Wed, Jan 25, 2017 at 4:42 PM, Darryl Philip Baker <
darryl.baker@northwestern.edu> wrote:

> The Red Hat software collection are Red Hat rpms they are just in a
> non-default repository. The have all the same support from Red Hat as the
> defaults. I’ve included a link to the latest version but Apache 2.4 has
> been in there since the very first release.
>
>
>
> https://www.redhat.com/en/about/press-releases/red-hat-
> software-collections-23-and-red-hat-developer-toolset-6-
> now-generally-available
>
>
>
> Darryl Baker
>
> NIT - CI -DAPS
>
> X76674
>
>
>
> *From:* Michele Mase' [mailto:michele.mase@gmail.com]
> *Sent:* January 25, 2017 8:59 AM
> *To:* Httpd Users List <us...@httpd.apache.org>
> *Subject:* Re: [users@httpd] Unable to fork new process
>
>
>
>
>
> On Wed, Jan 25, 2017 at 3:39 PM, Darryl Philip Baker <
> darryl.baker@northwestern.edu> wrote:
>
> e
>
>
>
> Due to redhat subscription, we must use redhat's rpms. Now I'm trying the
> php loop against another web server. Then I will try with apache 2.4.x in a
> offline env. My question is the same:
> After xx reloads under yy load average in zz time the error will appears
> "every time".
> Any smarter suggestion?
>

RE: [users@httpd] Unable to fork new process

Posted by Darryl Philip Baker <da...@northwestern.edu>.
The Red Hat software collection are Red Hat rpms they are just in a non-default repository. The have all the same support from Red Hat as the defaults. I’ve included a link to the latest version but Apache 2.4 has been in there since the very first release.

https://www.redhat.com/en/about/press-releases/red-hat-software-collections-23-and-red-hat-developer-toolset-6-now-generally-available

Darryl Baker
NIT - CI -DAPS
X76674

From: Michele Mase' [mailto:michele.mase@gmail.com]
Sent: January 25, 2017 8:59 AM
To: Httpd Users List <us...@httpd.apache.org>
Subject: Re: [users@httpd] Unable to fork new process


On Wed, Jan 25, 2017 at 3:39 PM, Darryl Philip Baker <da...@northwestern.edu>> wrote:
e

Due to redhat subscription, we must use redhat's rpms. Now I'm trying the php loop against another web server. Then I will try with apache 2.4.x in a offline env. My question is the same:
After xx reloads under yy load average in zz time the error will appears "every time".
Any smarter suggestion?

Re: [users@httpd] Unable to fork new process

Posted by Michele Mase' <mi...@gmail.com>.
On Wed, Jan 25, 2017 at 3:39 PM, Darryl Philip Baker <
darryl.baker@northwestern.edu> wrote:

> e


Due to redhat subscription, we must use redhat's rpms. Now I'm trying the
php loop against another web server. Then I will try with apache 2.4.x in a
offline env. My question is the same:
After xx reloads under yy load average in zz time the error will appears
"every time".
Any smarter suggestion?

RE: [users@httpd] Unable to fork new process

Posted by Darryl Philip Baker <da...@northwestern.edu>.
My first suggestion remains, is it possible for you upgrade to the Apache httpd 2.4 from the Red Hat Software Collections? It means adding a new repo to your yum configuration but you would still have Red Hat support as with the default httpd installation. They even put the whole tree in /opt/rh so a machine can run both versions so long as each listens on a different IP. I just did that on some of the environments I deal with without too much difficulty. Most of my configuration changes dealt with the change in permission syntax.

Darryl Baker
Sr. System Administrator
Northwestern | Information Technology
www.it.northwestern.edu

From: Michele Mase' [mailto:michele.mase@gmail.com]
Sent: January 25, 2017 8:30 AM
To: Httpd Users List <us...@httpd.apache.org>
Subject: Re: [users@httpd] Unable to fork new process

Ok. Now I try again with a smaller set of modules. Anyone could help me? Should I ask redhat to solve my problem?
The question is quite simple:
After xx reloads under yy load average in zz time the error will appears "every time".
Any smarter suggestion?
B.R.

On Wed, Jan 25, 2017 at 12:22 PM, Eric Covener <co...@gmail.com>> wrote:
On Wed, Jan 25, 2017 at 5:29 AM, Michele Mase' <mi...@gmail.com>> wrote:
> Any better suggestion? Anyone ever will correct the httpd code?

Not without a more specific/realistic bug report on a recent release,
and even then it's unlikely to show up in the 2.2.15 based server
distributed by redhat.

One PD step would be to reproduce with the smallest possible set of
modules to see if one can be implicated.   Another would be to
actually observe memory usage along the way.

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org<ma...@httpd.apache.org>
For additional commands, e-mail: users-help@httpd.apache.org<ma...@httpd.apache.org>


Re: [users@httpd] Unable to fork new process

Posted by Michele Mase' <mi...@gmail.com>.
Ok. Now I try again with a smaller set of modules. Anyone could help me?
Should I ask redhat to solve my problem?
The question is quite simple:
After xx reloads under yy load average in zz time the error will appears
"every time".
Any smarter suggestion?
B.R.

On Wed, Jan 25, 2017 at 12:22 PM, Eric Covener <co...@gmail.com> wrote:

> On Wed, Jan 25, 2017 at 5:29 AM, Michele Mase' <mi...@gmail.com>
> wrote:
> > Any better suggestion? Anyone ever will correct the httpd code?
>
> Not without a more specific/realistic bug report on a recent release,
> and even then it's unlikely to show up in the 2.2.15 based server
> distributed by redhat.
>
> One PD step would be to reproduce with the smallest possible set of
> modules to see if one can be implicated.   Another would be to
> actually observe memory usage along the way.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
>
>

Re: [users@httpd] Unable to fork new process

Posted by Eric Covener <co...@gmail.com>.
On Wed, Jan 25, 2017 at 5:29 AM, Michele Mase' <mi...@gmail.com> wrote:
> Any better suggestion? Anyone ever will correct the httpd code?

Not without a more specific/realistic bug report on a recent release,
and even then it's unlikely to show up in the 2.2.15 based server
distributed by redhat.

One PD step would be to reproduce with the smallest possible set of
modules to see if one can be implicated.   Another would be to
actually observe memory usage along the way.

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org


Re: [users@httpd] Unable to fork new process

Posted by Michele Mase' <mi...@gmail.com>.
There is the possibility of a memory leak: following an old post about the
same problem https://ubuntuforums.org/showthread.php?t=1505539, I tried the
experiment: using the php script to keep reloading apache until apache logs
an error (a simple loop that reloads apache), after 100883 reloads and 7621
seconds, the apache on the test machine hangs with the error "[error]
(12)Cannot allocate memory: fork: Unable to fork new process".
Notice that the test machine is absolutely empty, there is only the web
server with no activity.
On a medium/large prod. environment, some apache reload (due to config
change, like add or remove a virtualhost) together with the load of the web
sites causes the same error after some days. It seems to be like the
following algorithm: xx apache reloads + yy load average = zz time memory
leak.
I applied the sipliest solution:
changing the logrotate.d/httpd by replacin service httpd reload with
service httpd restart
But the memory leak stirr remains.
Any better suggestion? Anyone ever will correct the httpd code?
Best regards

On Tue, Jan 24, 2017 at 9:52 PM, Houser, Rick <ri...@jackson.com>
wrote:

> Where and what is the leak?  If it’s in the workers (the parts running as
> non-root), perhaps just set a request limit so they keep cycling out
> automatically within Apache and getting replaced seamlessly with new
> children?  If it’s in the higher level processes (the root owned ones),
> that’s not going to work.  I could be mistaken, but I thought a process or
> thread limit also gave a message similar to that, so double check your
> ulimit settings.  Adding memory not helping you further implies you are
> running up against a software limit of some kind.
>
>
>
>
>
> Rick Houser
>
> Web Administration
>
>
>
> *From:* Darryl Philip Baker [mailto:darryl.baker@northwestern.edu]
> *Sent:* Tuesday, January 24, 2017 14:08
> *To:* users@httpd.apache.org
> *Subject:* RE: [users@httpd] Unable to fork new process
>
>
>
> First I am assuming you are using the Red Hat distributed version. Is
> upgrading to Apache httpd 2.4 from the Red Hat Software Collection an
> option?
>
>
>
> I can think of several options to detect and automate the restart as a
> remediation. One would be to script a memory watcher using the free command
> and automatically executing the restart steps you do manually. The run the
> script out of cron every 5 minutes.
>
>
>
> For a fix you might want to peruse the release note for Apache versions of
> 2.2 after 2.2.15, and the mods, for memory leak fixes and see if Red Hat
> has backported them into their version. If you find one that wasn’t
> backported time to nudge Res Hat. Maybe someone on this list more in tune
> with the releases can point you at one or two of these.  Or just build a
> newer version of Apache 2.2 from source
>
>
>
> Darryl Baker
>
> NIT - CI -DAPS
>
> X76674
>
>
>
> *From:* Michele Mase' [mailto:michele.mase@gmail.com
> <mi...@gmail.com>]
> *Sent:* January 24, 2017 12:10 PM
> *To:* Httpd Users List <us...@httpd.apache.org>
> *Subject:* [users@httpd] Unable to fork new process
>
>
>
> All,
> I've a apache web server with 600 web sites as reverse proxy. The average
> of req. is more than 100 req/sec during all the day.
> The apache is a 2.2.15 release 56.el6_8.3 (RedHat 6.x system up to date) +
> mod_ssl.
> Every 10 days, the system has problems, and in the error log i found dozen
> of:
>
>  [error] (12)Cannot allocate memory: fork: Unable to fork new process
>
> The temporary solution for me is the following: after an alert mail from
> the system monitoring server, I must restart the httpd using:
>
> service httpd stop; killall httpd.worker; killall -9 httpd.worker; service
> httpd start
>
> Adding more ram to the system does not help.
>
>
>
> Any suggestion?
>
> Best regards
>
> apachectl -l
> Compiled in modules:
>   core.c
>   worker.c
>   http_core.c
>   mod_so.c
>
> apachectl -M
> Loaded Modules:
>  core_module (static)
>  mpm_worker_module (static)
>  http_module (static)
>  so_module (static)
>  log_post_module (shared)
>  evasive20_module (shared)
>  auth_basic_module (shared)
>  auth_digest_module (shared)
>  authn_file_module (shared)
>  authn_alias_module (shared)
>  authn_anon_module (shared)
>  authn_dbm_module (shared)
>  authn_default_module (shared)
>  authz_host_module (shared)
>  authz_user_module (shared)
>  authz_owner_module (shared)
>  authz_groupfile_module (shared)
>  authz_dbm_module (shared)
>  authz_default_module (shared)
>  ldap_module (shared)
>  authnz_ldap_module (shared)
>  include_module (shared)
>  log_config_module (shared)
>  logio_module (shared)
>  env_module (shared)
>  ext_filter_module (shared)
>  mime_magic_module (shared)
>  expires_module (shared)
>  deflate_module (shared)
>  headers_module (shared)
>  usertrack_module (shared)
>  setenvif_module (shared)
>  mime_module (shared)
>  dav_module (shared)
>  status_module (shared)
>  autoindex_module (shared)
>  info_module (shared)
>  dav_fs_module (shared)
>  vhost_alias_module (shared)
>  negotiation_module (shared)
>  dir_module (shared)
>  actions_module (shared)
>  speling_module (shared)
>  userdir_module (shared)
>  alias_module (shared)
>  substitute_module (shared)
>  rewrite_module (shared)
>  proxy_module (shared)
>  proxy_balancer_module (shared)
>  proxy_ftp_module (shared)
>  proxy_http_module (shared)
>  proxy_ajp_module (shared)
>  proxy_connect_module (shared)
>  cache_module (shared)
>  suexec_module (shared)
>  disk_cache_module (shared)
>  cgi_module (shared)
>  version_module (shared)
>  unique_id_module (shared)
>  ssl_module (shared)
>  reqtimeout_module (shared)
> Syntax OK
>

RE: [users@httpd] Unable to fork new process

Posted by "Houser, Rick" <ri...@jackson.com>.
Where and what is the leak?  If it’s in the workers (the parts running as non-root), perhaps just set a request limit so they keep cycling out automatically within Apache and getting replaced seamlessly with new children?  If it’s in the higher level processes (the root owned ones), that’s not going to work.  I could be mistaken, but I thought a process or thread limit also gave a message similar to that, so double check your ulimit settings.  Adding memory not helping you further implies you are running up against a software limit of some kind.


Rick Houser
Web Administration

From: Darryl Philip Baker [mailto:darryl.baker@northwestern.edu]
Sent: Tuesday, January 24, 2017 14:08
To: users@httpd.apache.org
Subject: RE: [users@httpd] Unable to fork new process

First I am assuming you are using the Red Hat distributed version. Is upgrading to Apache httpd 2.4 from the Red Hat Software Collection an option?

I can think of several options to detect and automate the restart as a remediation. One would be to script a memory watcher using the free command and automatically executing the restart steps you do manually. The run the script out of cron every 5 minutes.

For a fix you might want to peruse the release note for Apache versions of 2.2 after 2.2.15, and the mods, for memory leak fixes and see if Red Hat has backported them into their version. If you find one that wasn’t backported time to nudge Res Hat. Maybe someone on this list more in tune with the releases can point you at one or two of these.  Or just build a newer version of Apache 2.2 from source

Darryl Baker
NIT - CI -DAPS
X76674

From: Michele Mase' [mailto:michele.mase@gmail.com]
Sent: January 24, 2017 12:10 PM
To: Httpd Users List <us...@httpd.apache.org>>
Subject: [users@httpd] Unable to fork new process

All,
I've a apache web server with 600 web sites as reverse proxy. The average of req. is more than 100 req/sec during all the day.
The apache is a 2.2.15 release 56.el6_8.3 (RedHat 6.x system up to date) + mod_ssl.
Every 10 days, the system has problems, and in the error log i found dozen of:

 [error] (12)Cannot allocate memory: fork: Unable to fork new process
The temporary solution for me is the following: after an alert mail from the system monitoring server, I must restart the httpd using:
service httpd stop; killall httpd.worker; killall -9 httpd.worker; service httpd start
Adding more ram to the system does not help.

Any suggestion?
Best regards

apachectl -l
Compiled in modules:
  core.c
  worker.c
  http_core.c
  mod_so.c

apachectl -M
Loaded Modules:
 core_module (static)
 mpm_worker_module (static)
 http_module (static)
 so_module (static)
 log_post_module (shared)
 evasive20_module (shared)
 auth_basic_module (shared)
 auth_digest_module (shared)
 authn_file_module (shared)
 authn_alias_module (shared)
 authn_anon_module (shared)
 authn_dbm_module (shared)
 authn_default_module (shared)
 authz_host_module (shared)
 authz_user_module (shared)
 authz_owner_module (shared)
 authz_groupfile_module (shared)
 authz_dbm_module (shared)
 authz_default_module (shared)
 ldap_module (shared)
 authnz_ldap_module (shared)
 include_module (shared)
 log_config_module (shared)
 logio_module (shared)
 env_module (shared)
 ext_filter_module (shared)
 mime_magic_module (shared)
 expires_module (shared)
 deflate_module (shared)
 headers_module (shared)
 usertrack_module (shared)
 setenvif_module (shared)
 mime_module (shared)
 dav_module (shared)
 status_module (shared)
 autoindex_module (shared)
 info_module (shared)
 dav_fs_module (shared)
 vhost_alias_module (shared)
 negotiation_module (shared)
 dir_module (shared)
 actions_module (shared)
 speling_module (shared)
 userdir_module (shared)
 alias_module (shared)
 substitute_module (shared)
 rewrite_module (shared)
 proxy_module (shared)
 proxy_balancer_module (shared)
 proxy_ftp_module (shared)
 proxy_http_module (shared)
 proxy_ajp_module (shared)
 proxy_connect_module (shared)
 cache_module (shared)
 suexec_module (shared)
 disk_cache_module (shared)
 cgi_module (shared)
 version_module (shared)
 unique_id_module (shared)
 ssl_module (shared)
 reqtimeout_module (shared)
Syntax OK

RE: [users@httpd] Unable to fork new process

Posted by Darryl Philip Baker <da...@northwestern.edu>.
First I am assuming you are using the Red Hat distributed version. Is upgrading to Apache httpd 2.4 from the Red Hat Software Collection an option?

I can think of several options to detect and automate the restart as a remediation. One would be to script a memory watcher using the free command and automatically executing the restart steps you do manually. The run the script out of cron every 5 minutes.

For a fix you might want to peruse the release note for Apache versions of 2.2 after 2.2.15, and the mods, for memory leak fixes and see if Red Hat has backported them into their version. If you find one that wasn’t backported time to nudge Res Hat. Maybe someone on this list more in tune with the releases can point you at one or two of these.  Or just build a newer version of Apache 2.2 from source

Darryl Baker
NIT - CI -DAPS
X76674

From: Michele Mase' [mailto:michele.mase@gmail.com]
Sent: January 24, 2017 12:10 PM
To: Httpd Users List <us...@httpd.apache.org>
Subject: [users@httpd] Unable to fork new process

All,
I've a apache web server with 600 web sites as reverse proxy. The average of req. is more than 100 req/sec during all the day.
The apache is a 2.2.15 release 56.el6_8.3 (RedHat 6.x system up to date) + mod_ssl.
Every 10 days, the system has problems, and in the error log i found dozen of:

 [error] (12)Cannot allocate memory: fork: Unable to fork new process
The temporary solution for me is the following: after an alert mail from the system monitoring server, I must restart the httpd using:
service httpd stop; killall httpd.worker; killall -9 httpd.worker; service httpd start
Adding more ram to the system does not help.

Any suggestion?
Best regards

apachectl -l
Compiled in modules:
  core.c
  worker.c
  http_core.c
  mod_so.c

apachectl -M
Loaded Modules:
 core_module (static)
 mpm_worker_module (static)
 http_module (static)
 so_module (static)
 log_post_module (shared)
 evasive20_module (shared)
 auth_basic_module (shared)
 auth_digest_module (shared)
 authn_file_module (shared)
 authn_alias_module (shared)
 authn_anon_module (shared)
 authn_dbm_module (shared)
 authn_default_module (shared)
 authz_host_module (shared)
 authz_user_module (shared)
 authz_owner_module (shared)
 authz_groupfile_module (shared)
 authz_dbm_module (shared)
 authz_default_module (shared)
 ldap_module (shared)
 authnz_ldap_module (shared)
 include_module (shared)
 log_config_module (shared)
 logio_module (shared)
 env_module (shared)
 ext_filter_module (shared)
 mime_magic_module (shared)
 expires_module (shared)
 deflate_module (shared)
 headers_module (shared)
 usertrack_module (shared)
 setenvif_module (shared)
 mime_module (shared)
 dav_module (shared)
 status_module (shared)
 autoindex_module (shared)
 info_module (shared)
 dav_fs_module (shared)
 vhost_alias_module (shared)
 negotiation_module (shared)
 dir_module (shared)
 actions_module (shared)
 speling_module (shared)
 userdir_module (shared)
 alias_module (shared)
 substitute_module (shared)
 rewrite_module (shared)
 proxy_module (shared)
 proxy_balancer_module (shared)
 proxy_ftp_module (shared)
 proxy_http_module (shared)
 proxy_ajp_module (shared)
 proxy_connect_module (shared)
 cache_module (shared)
 suexec_module (shared)
 disk_cache_module (shared)
 cgi_module (shared)
 version_module (shared)
 unique_id_module (shared)
 ssl_module (shared)
 reqtimeout_module (shared)
Syntax OK