You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@perl.apache.org by Stas Bekman <st...@stason.org> on 2005/01/05 17:57:58 UTC

[Fwd: [PATCH] libperl leaks a THREAD_KEY each time it is reloaded]

Gisle, could you please provide us some more details about the crash? I'd 
like to write a test for it. I suppose you had it with mp2 worker mpm? 
Care to post the output of t/REPORT (in the mp2 source) or mp2bug? Thanks 
a lot!

Thank you.

-------- Original Message --------
Subject: [PATCH] libperl leaks a THREAD_KEY each time it is reloaded
Date: 05 Jan 2005 03:21:06 -0800
From: Gisle Aas <gi...@ActiveState.com>
To: perl5-porters@perl.org

For a threaded perl the thread key is allocated on the first call to
perl_alloc() but there is nothing that frees this key on Unix.  On
Windows we watch for the DLL_PROCESS_DETACH event and then release the
key.  This patch implements the same for other platforms using
compiler dependent destructors.  The patch has been tested on Linux
(gcc) and HP-UX (cc and gcc).

--- perl.c.orig	2005-01-01 12:56:44.000000000 -0800
+++ perl.c	2005-01-05 03:02:55.000000000 -0800
@@ -958,6 +958,31 @@
  #endif
  }

+#if defined(USE_5005THREADS) || defined(USE_ITHREADS)
+/* provide destructors to clean up the thread key when libperl is unloaded */
+#ifndef WIN32 /* handled during DLL_PROCESS_DETACH in win32/perllib.c */
+
+#if defined(__hpux) && !defined(__GNUC__)
+#pragma fini "perl_fini"
+#endif
+
+#if defined(__GNUC__) && defined(__attribute__)
+/* want to make sure __attribute__ works here even
+ * for -Dd_attribut=undef builds.
+ */
+#undef __attribute__
+#endif
+
+static void __attribute__((destructor))
+perl_fini()
+{
+    if (PL_curinterp)
+	FREE_THREAD_KEY;
+}
+
+#endif /* WIN32 */
+#endif /* THREADS */
+
  void
  Perl_call_atexit(pTHX_ ATEXIT_t fn, void *ptr)
  {


This patch has been applied to ActivePerl because a customer had
Apache with mod_perl crash after a certain number of reloads.  Perl
crashes when it can't allocate a new thread key and each time Apache
reloads it will dlclose() and dlopen() mod_perl.so.  That patch make
sure that perl's thread key get freed at dlclose() time.

Test script to demonstrate this failure is attached.

Regards,
Gisle



-- 
__________________________________________________________________
Stas Bekman            JAm_pH ------> Just Another mod_perl Hacker
http://stason.org/     mod_perl Guide ---> http://perl.apache.org
mailto:stas@stason.org http://use.perl.org http://apacheweek.com
http://modperlbook.org http://apache.org   http://ticketmaster.com

Re: MP_PERL_FREE_THREAD_KEY_WORKAROUND

Posted by "Philippe M. Chiasson" <go...@ectoplasm.org>.
Eric Cholet wrote:

> Le 16 janv. 05, à 11:58, Gisle Aas a écrit :
>
>> Stas,
>>
>> Could you apply this addition to the  
>> MP_PERL_FREE_THREAD_KEY_WORKAROUND?
>> This will avoid that the perl_fini patch now in ActivePerl blows up
>> because we end up with calling FREE_THREAD_KEY twice.
>>
>> --Gisle
>>
>> diff -ru  
>> modperl-2.0_20050116053812/src/modules/perl/modperl_perl_includes.h  
>> modperl-2.0-hack/src/modules/perl/modperl_perl_includes.h
>> ---  
>> modperl-2.0_20050116053812/src/modules/perl/modperl_perl_includes.h     
>> 2005-01-13 06:46:07.000000000 +0100
>> +++  modperl-2.0-hack/src/modules/perl/modperl_perl_includes.h    
>> 2005-01-16  11:51:09.986693975 +0100
>> @@ -146,6 +146,7 @@
>>  #define MP_PERL_FREE_THREAD_KEY_WORKAROUND      \
>>      if (PL_curinterp) {                         \
>>          FREE_THREAD_KEY;                        \
>> +        PL_curinterp = NULL;                    \
>>      }
>>  #else
>>  #define MP_PERL_FREE_THREAD_KEY_WORKAROUND
>
> also fixes darwin w/ threaded Perl + worker

Same here for me, that patches makes the segfault I reported dissapear on my
darwin box.

Patch is in r125401.
 
--------------------------------------------------------------------------------
Philippe M. Chiasson m/gozer\@(apache|cpan|ectoplasm)\.org/ GPG KeyID : 88C3A5A5 
http://gozer.ectoplasm.org/     F9BF E0C2 480E 7680 1AE5 3631 CB32 A107 88C3A5A5


Re: MP_PERL_FREE_THREAD_KEY_WORKAROUND

Posted by "Philippe M. Chiasson" <go...@ectoplasm.org>.
Gisle Aas wrote:
> Eric Cholet <ch...@logilune.com> writes:
> 
>>>diff -ru
>>>modperl-2.0_20050116053812/src/modules/perl/modperl_perl_includes.h
>>>modperl-2.0-hack/src/modules/perl/modperl_perl_includes.h
>>>---
>>>modperl-2.0_20050116053812/src/modules/perl/modperl_perl_includes.h
>>>2005-01-13 06:46:07.000000000 +0100
>>>+++
>>>modperl-2.0-hack/src/modules/perl/modperl_perl_includes.h
>>>2005-01-16  11:51:09.986693975 +0100
>>>@@ -146,6 +146,7 @@
>>> #define MP_PERL_FREE_THREAD_KEY_WORKAROUND      \
>>>     if (PL_curinterp) {                         \
>>>         FREE_THREAD_KEY;                        \
>>>+        PL_curinterp = NULL;                    \
>>>     }
>>> #else
>>> #define MP_PERL_FREE_THREAD_KEY_WORKAROUND
>>
>>also fixes darwin w/ threaded Perl + worker
> 
> This sounds suspicions to me.  Doesn't this mean that
> MP_PERL_FREE_THREAD_KEY_WORKAROUND runs twice in this setup.  Can you
> explain why this would happen?

Without that patch, on darwin with perl-5.8.1-ithreads, I get the following
backtrace on startup:

Program received signal EXC_BAD_ACCESS, Could not access memory.
0x02432910 in S_mess_alloc (my_perl=0x0) at util.c:818
818         if (!PL_dirty)
(gdb) bt
#0  0x02432910 in S_mess_alloc (my_perl=0x0) at util.c:818
#1  0x02432db4 in Perl_vmess (my_perl=0x0, pat=0x252fc5c "panic: pthread_setspecific", args=0xbffff2a4) at util.c:946
#2  0x02434260 in Perl_vcroak (my_perl=0x0, pat=0x252fc5c "panic: pthread_setspecific", args=0xbffff2a4) at util.c:1146
#3  0x02434bd4 in Perl_croak_nocontext (pat=0x252fc5c "panic: pthread_setspecific") at util.c:1216
#4  0x023b5a4c in perl_alloc () at perl.c:144
#5  0x02206490 in modperl_startup (s=0x1807fb8, p=0x1806218) at mod_perl.c:237
#6  0x022063c0 in modperl_startup (s=0x1807fb8, p=0x1806218) at mod_perl.c:207
#7  0x02207df4 in modperl_init (base_server=0x1807fb8, p=0x1806218) at mod_perl.c:453
#8  0x02208450 in modperl_hook_init (pconf=0x1806218, plog=0x0, ptemp=0x0, s=0x1807fb8) at mod_perl.c:613
#9  0x022084a8 in modperl_run () at mod_perl.c:624
#10 0x0221067c in modperl_cmd_load_module (parms=0xbffff850, mconfig=0x1843ff8, arg=0x1971b78 "TestCompat::apache_module") at modperl_cmd.c:650
#11 0x00020498 in invoke_cmd (cmd=0x2240be0, parms=0xbffff850, mconfig=0x1843ff8, args=0x1971b78 "TestCompat::apache_module") at config.c:796
#12 0x00020bc4 in ap_walk_config_sub (current=0x1971b58, parms=0xbffff850, section_vector=0x183de50) at config.c:1059
#13 0x00020c64 in ap_walk_config (current=0x2240be0, parms=0xbffff850, section_vector=0x1971b58) at config.c:1098
#14 0x00021a08 in ap_process_config_tree (s=0x1807fb8, conftree=0x183de50, p=0x2240be0, ptemp=0xbffff850) at config.c:1641
#15 0x0000ae18 in main (argc=11, argv=0xbffffa00) at main.c:595

--------------------------------------------------------------------------------
Philippe M. Chiasson m/gozer\@(apache|cpan|ectoplasm)\.org/ GPG KeyID : 88C3A5A5
http://gozer.ectoplasm.org/     F9BF E0C2 480E 7680 1AE5 3631 CB32 A107 88C3A5A5

Re: MP_PERL_FREE_THREAD_KEY_WORKAROUND

Posted by Gisle Aas <gi...@activestate.com>.
Eric Cholet <ch...@logilune.com> writes:

> > diff -ru
> > modperl-2.0_20050116053812/src/modules/perl/modperl_perl_includes.h
> > modperl-2.0-hack/src/modules/perl/modperl_perl_includes.h
> > ---
> > modperl-2.0_20050116053812/src/modules/perl/modperl_perl_includes.h
> > 2005-01-13 06:46:07.000000000 +0100
> > +++
> > modperl-2.0-hack/src/modules/perl/modperl_perl_includes.h
> > 2005-01-16  11:51:09.986693975 +0100
> > @@ -146,6 +146,7 @@
> >  #define MP_PERL_FREE_THREAD_KEY_WORKAROUND      \
> >      if (PL_curinterp) {                         \
> >          FREE_THREAD_KEY;                        \
> > +        PL_curinterp = NULL;                    \
> >      }
> >  #else
> >  #define MP_PERL_FREE_THREAD_KEY_WORKAROUND
> 
> also fixes darwin w/ threaded Perl + worker

This sounds suspicions to me.  Doesn't this mean that
MP_PERL_FREE_THREAD_KEY_WORKAROUND runs twice in this setup.  Can you
explain why this would happen?

--Gisle


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@perl.apache.org
For additional commands, e-mail: dev-help@perl.apache.org


Re: MP_PERL_FREE_THREAD_KEY_WORKAROUND

Posted by Eric Cholet <ch...@logilune.com>.
Le 16 janv. 05, à 11:58, Gisle Aas a écrit :

> Stas,
>
> Could you apply this addition to the  
> MP_PERL_FREE_THREAD_KEY_WORKAROUND?
> This will avoid that the perl_fini patch now in ActivePerl blows up
> because we end up with calling FREE_THREAD_KEY twice.
>
> --Gisle
>
>
> diff -ru  
> modperl-2.0_20050116053812/src/modules/perl/modperl_perl_includes.h  
> modperl-2.0-hack/src/modules/perl/modperl_perl_includes.h
> ---  
> modperl-2.0_20050116053812/src/modules/perl/modperl_perl_includes.h	 
> 2005-01-13 06:46:07.000000000 +0100
> +++  
> modperl-2.0-hack/src/modules/perl/modperl_perl_includes.h	2005-01-16  
> 11:51:09.986693975 +0100
> @@ -146,6 +146,7 @@
>  #define MP_PERL_FREE_THREAD_KEY_WORKAROUND      \
>      if (PL_curinterp) {                         \
>          FREE_THREAD_KEY;                        \
> +        PL_curinterp = NULL;                    \
>      }
>  #else
>  #define MP_PERL_FREE_THREAD_KEY_WORKAROUND

also fixes darwin w/ threaded Perl + worker

--
Eric Cholet


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@perl.apache.org
For additional commands, e-mail: dev-help@perl.apache.org


MP_PERL_FREE_THREAD_KEY_WORKAROUND

Posted by Gisle Aas <gi...@activestate.com>.
Stas, 

Could you apply this addition to the MP_PERL_FREE_THREAD_KEY_WORKAROUND?
This will avoid that the perl_fini patch now in ActivePerl blows up
because we end up with calling FREE_THREAD_KEY twice.

--Gisle


diff -ru modperl-2.0_20050116053812/src/modules/perl/modperl_perl_includes.h modperl-2.0-hack/src/modules/perl/modperl_perl_includes.h
--- modperl-2.0_20050116053812/src/modules/perl/modperl_perl_includes.h	2005-01-13 06:46:07.000000000 +0100
+++ modperl-2.0-hack/src/modules/perl/modperl_perl_includes.h	2005-01-16 11:51:09.986693975 +0100
@@ -146,6 +146,7 @@
 #define MP_PERL_FREE_THREAD_KEY_WORKAROUND      \
     if (PL_curinterp) {                         \
         FREE_THREAD_KEY;                        \
+        PL_curinterp = NULL;                    \
     }
 #else
 #define MP_PERL_FREE_THREAD_KEY_WORKAROUND


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@perl.apache.org
For additional commands, e-mail: dev-help@perl.apache.org


Re: [Fwd: [PATCH] libperl leaks a THREAD_KEY each time it is reloaded]

Posted by Stas Bekman <st...@stason.org>.
Joe Orton wrote:
> On Wed, Jan 05, 2005 at 02:09:52PM -0500, Stas Bekman wrote:
> 
>>Gisle Aas wrote:
>>
>>>Stas Bekman <st...@stason.org> writes:
>>>
>>>
>>>
>>>>How do I check the limit for pthread_key_create on linux?
>>>
>>>
>>>On my Gentoo box I find the limit here:
>>>
>>>    $ grep PTHREAD_KEYS_MAX /usr/include/bits/local_lim.h
>>>    #define PTHREAD_KEYS_MAX        1024
> 
> 
> Ah, very interesting.  We have a long-standing bug report that Apache
> reproducibly crashes on the 1020th [sic] SIGHUP with mod_perl loaded, I
> bet this is the same problem.
> 
> https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=73486
> 
> So that is a repro case worth trying, Stas.  It just takes a while...

Joe, you can close that bug as resolved, pointing to:

http://svn.apache.org/viewcvs?view=rev&rev=125036

beside the current svn, it should be out fixed by mod_perl-1.999.21 (or 
2.0.0 whenever it's released).


-- 
__________________________________________________________________
Stas Bekman            JAm_pH ------> Just Another mod_perl Hacker
http://stason.org/     mod_perl Guide ---> http://perl.apache.org
mailto:stas@stason.org http://use.perl.org http://apacheweek.com
http://modperlbook.org http://apache.org   http://ticketmaster.com

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@perl.apache.org
For additional commands, e-mail: dev-help@perl.apache.org


Re: [Fwd: [PATCH] libperl leaks a THREAD_KEY each time it is reloaded]

Posted by Stas Bekman <st...@stason.org>.
Gisle Aas wrote:
[...]
>>https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=73486
>>
>>So that is a repro case worth trying, Stas.  It just takes a while...
> 
> 
> You can speed it up my loading some other module that calls
> pthread_key_create in order to steal most of the available keys.

Thanks Gisle, the following reproduces the problem on the first restart, 
but as I've posted in the thread, I get segfaults if I try to delete the key.

Index: src/modules/perl/mod_perl.c
===================================================================
--- src/modules/perl/mod_perl.c (revision 124422)
+++ src/modules/perl/mod_perl.c (working copy)
@@ -273,6 +273,8 @@

      perl_run(perl);

+    Perl_warn(aTHX_ "ALLOC: 0x%lx\n", (unsigned long)aTHX);
+
  #ifdef USE_ITHREADS
      /* base server / virtual host w/ +Parent gets its own mip */
      modperl_interp_init(s, p, perl);
@@ -658,6 +660,20 @@
      dTHXa(scfg->mip->parent->perl);
  #endif

+    {
+        pthread_key_t key;
+        int i = 0;
+        for (i=0; i<500; i++) {
+            if (pthread_key_create(&key, 0) == 0) {
+                Perl_warn(aTHX_ "%d\n", i);
+            }
+            else {
+                Perl_croak(aTHX_ "failed to create the key");
+            }
+        }
+    }
+
+
      if (!modperl_post_config_require(s, pconf)) {
          exit(1);
      }

-- 
__________________________________________________________________
Stas Bekman            JAm_pH ------> Just Another mod_perl Hacker
http://stason.org/     mod_perl Guide ---> http://perl.apache.org
mailto:stas@stason.org http://use.perl.org http://apacheweek.com
http://modperlbook.org http://apache.org   http://ticketmaster.com

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@perl.apache.org
For additional commands, e-mail: dev-help@perl.apache.org


Re: [Fwd: [PATCH] libperl leaks a THREAD_KEY each time it is reloaded]

Posted by Gisle Aas <gi...@ActiveState.com>.
Joe Orton <jo...@redhat.com> writes:

> On Wed, Jan 05, 2005 at 02:09:52PM -0500, Stas Bekman wrote:
> > Gisle Aas wrote:
> > >Stas Bekman <st...@stason.org> writes:
> > >
> > >
> > >>How do I check the limit for pthread_key_create on linux?
> > >
> > >
> > >On my Gentoo box I find the limit here:
> > >
> > >     $ grep PTHREAD_KEYS_MAX /usr/include/bits/local_lim.h
> > >     #define PTHREAD_KEYS_MAX        1024
> 
> Ah, very interesting.  We have a long-standing bug report that Apache
> reproducibly crashes on the 1020th [sic] SIGHUP with mod_perl loaded, I
> bet this is the same problem.

I'm willing to bet on this as well.  The stack trace match exactly.

> https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=73486
> 
> So that is a repro case worth trying, Stas.  It just takes a while...

You can speed it up my loading some other module that calls
pthread_key_create in order to steal most of the available keys.

Regards,
Gisle

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@perl.apache.org
For additional commands, e-mail: dev-help@perl.apache.org


Re: [Fwd: [PATCH] libperl leaks a THREAD_KEY each time it is reloaded]

Posted by Joe Orton <jo...@redhat.com>.
On Wed, Jan 05, 2005 at 02:09:52PM -0500, Stas Bekman wrote:
> Gisle Aas wrote:
> >Stas Bekman <st...@stason.org> writes:
> >
> >
> >>How do I check the limit for pthread_key_create on linux?
> >
> >
> >On my Gentoo box I find the limit here:
> >
> >     $ grep PTHREAD_KEYS_MAX /usr/include/bits/local_lim.h
> >     #define PTHREAD_KEYS_MAX        1024

Ah, very interesting.  We have a long-standing bug report that Apache
reproducibly crashes on the 1020th [sic] SIGHUP with mod_perl loaded, I
bet this is the same problem.

https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=73486

So that is a repro case worth trying, Stas.  It just takes a while...

joe

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@perl.apache.org
For additional commands, e-mail: dev-help@perl.apache.org


Re: [Fwd: [PATCH] libperl leaks a THREAD_KEY each time it is reloaded]

Posted by Stas Bekman <st...@stason.org>.
Gisle Aas wrote:
> Stas Bekman <st...@stason.org> writes:
> 
> 
>>How do I check the limit for pthread_key_create on linux?
> 
> 
> On my Gentoo box I find the limit here:
> 
>      $ grep PTHREAD_KEYS_MAX /usr/include/bits/local_lim.h
>      #define PTHREAD_KEYS_MAX        1024
> 
> 
>>Is there a way to check that dynamically (i.e. how much is already
>>used)? I'll try to reproduce it.
> 
> 
> /* Something like this perhaps? */
> #include <stdio.h>
> #include <pthread.h>
> 
> int
> main(int argc, char* argv[])
> {
>     pthread_key_t key;
>     int i = 0;
>     while (pthread_key_create(&key, 0) == 0)
>         i++;
>     printf("%d\n", i);
> }
> 
> 
>>I suppose modperl could provide a workaround for older perls, based on
>>your patch?
> 
> 
> Yes, you can just drop in the perl_fini functions into mod_perl itself.

Thanks Gisle, I'll work on reproducing it and then patch modperl and 
report back. I'd also need to know which perl has it fixed. I'll monitor 
p5p for that.

-- 
__________________________________________________________________
Stas Bekman            JAm_pH ------> Just Another mod_perl Hacker
http://stason.org/     mod_perl Guide ---> http://perl.apache.org
mailto:stas@stason.org http://use.perl.org http://apacheweek.com
http://modperlbook.org http://apache.org   http://ticketmaster.com

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@perl.apache.org
For additional commands, e-mail: dev-help@perl.apache.org


Re: [Fwd: [PATCH] libperl leaks a THREAD_KEY each time it is reloaded]

Posted by Stas Bekman <st...@stason.org>.
Gisle Aas wrote:
> Stas Bekman <st...@stason.org> writes:
> 
> 
>>Gisle, could you please provide us some more details about the crash?
> 
> 
> I don't really know much more.  I have not tried to reproduce this
> problem myself with mod_perl and I don't even know exactly which
> version of httpd and mod_perl was in use.
> 
> This is the (only) information I got from HP (our customer) about this
> problem:
> ----------------------------------------------------------------------
> Our customer is using httpd, with mod_perl enabled.  Repeated restart
> of this httpd results in the "panic: pthread_key_create" error.
> 
> Working on it, we have found out that each time httpd is restarted it
> rereads the configuration file, httpd.conf. And everytime, the specified
> LoadModule directive is also read. This loads the DSO, mod_perl.so. mod_perl
> calls perl_alloc() from libperl.a, for allocating the memory for the
> Interpreter.

OK, so it looks like mp2 DSO. It's not clear if it's a worker or prefork 
(but most likely a worker, unless they decided to use interpreter pools 
with prefork, which is possible too).

> As Jan told me in the previous mail, thread private data is created
> only once for a process.  Therefore it is not deleted anywhere.
> 
> However, everytime mod_perl.so is reloaded, PL_curinterp gets initialised to
> 0. So, everytime the thread private data is created. When the limit to the
> number of thread private data to be created per process is crossed reached,
> it does exit(), making httpd to EXIT.
> 
> We have attached 2 files <test.c> and <t.c> with behaviours similar to
> mod_perl and httpd respectively.
> 
> Isn't there any way to free this storage allocated by the library call?
> ----------------------------------------------------------------------
> 
> The sited patch apparently made the customer happy and I did not ask
> more questions since we had the smaller test case to work from.
> 
> On my Linux box a process might allocate 1024 thread keys before
> pthread_key_create fails.  It means that you would have to restart
> httpd that many times before it crashes.  This customer was running
> HP-UX where the limit is 256 thread keys before the allocator starts
> to fail.

How do I check the limit for pthread_key_create on linux? Is there a way 
to check that dynamically (i.e. how much is already used)? I'll try to 
reproduce it.

I suppose modperl could provide a workaround for older perls, based on 
your patch?

At the very least I want to log it in the troubleshooting docs and point 
users to the perl version that has this bug fixed.


-- 
__________________________________________________________________
Stas Bekman            JAm_pH ------> Just Another mod_perl Hacker
http://stason.org/     mod_perl Guide ---> http://perl.apache.org
mailto:stas@stason.org http://use.perl.org http://apacheweek.com
http://modperlbook.org http://apache.org   http://ticketmaster.com

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@perl.apache.org
For additional commands, e-mail: dev-help@perl.apache.org