You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@apr.apache.org by Jeff Trawick <tr...@attglobal.net> on 2002/01/10 17:28:47 UTC

pool problems with HEAD??

CVS HEAD, Solaris, worker MPM, one process with many threads

<IfModule worker.c>
ServerLimit         1
ThreadLimit         500
StartServers         1
MaxClients         500
MinSpareThreads     25
MaxSpareThreads     75
ThreadsPerChild     500
MaxRequestsPerChild  0
</IfModule>

I had a client with many concurrent connections continually doing the
same request over and over.

After starting another client to get another 200 or so concurrent
connections, I quickly got a segfault in mod_log_config because
supposedly apr_palloc returned some bogus addresses (0 and 0x38).

#0  0x3fd84 in config_log_transaction (r=Cannot access memory at
 address 0xed5018f4
) at mod_log_config.c:823
823             strs[i] = process_item(r, orig, &items[i]);
(gdb) p strs
$1 = (char **) 0x0
(gdb) p format->nelts
$2 = 14
(gdb) p strl
$4 = (int *) 0x38
(gdb) p i
$5 = 0
(gdb)
(gdb) where
#0  0x3fd84 in config_log_transaction (r=0x6a62c8, cls=0x1add38,
 default_format=0x15cde8)
    at mod_log_config.c:823
#1  0x3ffd4 in multi_log_transaction (r=0x6a2618) at
 mod_log_config.c:881
#2  0xc3f98 in ap_run_log_transaction (r=0x6a2618) at protocol.c:1297
#3  0x53228 in ap_process_request (r=0x6a2618) at http_request.c:315
#4  0x4b150 in ap_process_http_connection (c=0x6a0758) at
 http_core.c:280
#5  0xba72c in ap_run_process_connection (c=0x6a0758) at
 connection.c:84
#6  0xbace4 in ap_process_connection (c=0x6a0758) at connection.c:230
#7  0xa08bc in process_socket (p=0x6a0638, sock=0x6a0670,
 my_child_num=0, my_thread_num=187)
    at worker.c:562
#8  0xa11f4 in worker_thread (thd=0x2bef88, dummy=0x1163b0) at
 worker.c:777
#9  0xff340990 in ?? ()

The funkiness about r changing in the traceback is (I suspect) because
in multi_log_transaction we walked through a list of r's using the
parameter as the variable.  Both r's look healthy.

Running a level of code from about a day before 2.0.30 on AIX this
morning, running about the same worker MPM config, I got a segfault
down in apr_palloc() because some heap structures were messed up.

Has anybody pounded on an Apache configured like that (one process,
many threads)?

-- 
Jeff Trawick | trawick@attglobal.net | PGP public key at web site:
       http://www.geocities.com/SiliconValley/Park/9289/
             Born in Roswell... married an alien...

Re: pool problems with HEAD??

Posted by Cliff Woolley <cl...@yahoo.com>.
On 10 Jan 2002, Jeff Trawick wrote:

> I attached to the child process's pid which interrupted sigwait, then
> did continue and in a separate terminal I banged on the server.  When the
> child process woke up I was still in the same place (__sigsuspend
> called by sigwait) but with the message
> 
>   ptrace in stop_thread: No such process.
> 
> The process had exited.

Ugh, yeah, I've seen that kind of thing happen a lot.  I don't have a
deterministic anything to tell you to get around it...  I had thought that 
attaching to the right pid would fix it, but maybe not.  I'll try it on my 
machine and see if I can come up with anything better (and if so, I'll be 
sure to tell you what I did).

> I configured the worker MPM to have 1 process and 100's of threads then
> used an ab-like tool to bang on it with many concurrent connections.

Okay, thanks... I'll try that.

--Cliff

--------------------------------------------------------------
   Cliff Woolley
   cliffwoolley@yahoo.com
   Charlottesville, VA



Re: pool problems with HEAD??

Posted by Jeff Trawick <tr...@attglobal.net>.
Cliff Woolley <cl...@yahoo.com> writes:

> > I've tried efence on Linux but I'm not sure what is supposed to
> > happen.
> 
> It's supposed to segfault if you attempt to read or write outside the 
> bounds of memory you allocated with malloc.
> 
> > It segfaults all over the place when I bang on it, but I don't get
> > coredumps when threads segfault on my level of kernel :(
> 
> Try running it directly under gdb if you haven't already... I frequently
> use efence on Linux, and have found running it directly under the debugger
> to be the most headache-free way to figure out what's going on.  Usually 
> you can start up the server and then attach to the child process's pid 
> (not one of an individual thread), and you should be good to go.

I attached to the child process's pid which interrupted sigwait, then
did continue and in a separate terminal I banged on the server.  When the
child process woke up I was still in the same place (__sigsuspend
called by sigwait) but with the message

  ptrace in stop_thread: No such process.

The process had exited.

Do you see anything obvious which I messed up?

> Sorry I missed the rest of the conversation, but what exactly do you have 
> to do to "bang on it" to see the segfault?  I'll try it myself if you give 
> me a hint as to what to try.

I configured the worker MPM to have 1 process and 100's of threads then
used an ab-like tool to bang on it with many concurrent connections.

Thanks,

Jeff

-- 
Jeff Trawick | trawick@attglobal.net | PGP public key at web site:
       http://www.geocities.com/SiliconValley/Park/9289/
             Born in Roswell... married an alien...

Re: pool problems with HEAD??

Posted by Cliff Woolley <cl...@yahoo.com>.
On 10 Jan 2002, Jeff Trawick wrote:

> Ian Holsman <ia...@apache.org> writes:
> 
> > can you run it with efence/purify on ?
> > this should catch the problem.
> 
> I wish I had purify.

I've got it.  I'll try running it on HEAD later today.

> I've tried efence on Linux but I'm not sure what is supposed to
> happen.

It's supposed to segfault if you attempt to read or write outside the 
bounds of memory you allocated with malloc.

> It segfaults all over the place when I bang on it, but I don't get
> coredumps when threads segfault on my level of kernel :(

Try running it directly under gdb if you haven't already... I frequently
use efence on Linux, and have found running it directly under the debugger
to be the most headache-free way to figure out what's going on.  Usually 
you can start up the server and then attach to the child process's pid 
(not one of an individual thread), and you should be good to go.

Sorry I missed the rest of the conversation, but what exactly do you have 
to do to "bang on it" to see the segfault?  I'll try it myself if you give 
me a hint as to what to try.

> Maybe I can get efence installed on AIX or Solaris where I know I can
> get coredumps.

I guess that's also worth doing.

--Cliff

--------------------------------------------------------------
   Cliff Woolley
   cliffwoolley@yahoo.com
   Charlottesville, VA



RE: pool problems with HEAD??

Posted by Sander Striker <st...@apache.org>.
> From: trawick@rdu163-40-092.nc.rr.com
> [mailto:trawick@rdu163-40-092.nc.rr.com]On Behalf Of Jeff Trawick
> Sent: 10 January 2002 19:57
> To: Ian Holsman
> Cc: dev@httpd.apache.org
> Subject: Re: pool problems with HEAD??
> 
> 
> Ian Holsman <ia...@apache.org> writes:
> 
> > can you run it with efence/purify on ?
> > this should catch the problem.
> 
> I wish I had purify.  I've tried efence on Linux but I'm not sure what
> is supposed to happen.

It is supposed to segfault at the first point where a bad memory
pointer is used.  IOW, run it in gdb.  Which might be a problem with
your repro recipe (500 threads/1 process)... Or isn't it?

>  It segfaults all over the place when I bang on
> it, but I don't get coredumps when threads segfault on my level of
> kernel :(
> 
> Maybe I can get efence installed on AIX or Solaris where I know I can
> get coredumps.

Sander

Re: pool problems with HEAD??

Posted by Jeff Trawick <tr...@attglobal.net>.
Ian Holsman <ia...@apache.org> writes:

> can you run it with efence/purify on ?
> this should catch the problem.

I wish I had purify.  I've tried efence on Linux but I'm not sure what
is supposed to happen.  It segfaults all over the place when I bang on
it, but I don't get coredumps when threads segfault on my level of
kernel :(

Maybe I can get efence installed on AIX or Solaris where I know I can
get coredumps.
-- 
Jeff Trawick | trawick@attglobal.net | PGP public key at web site:
       http://www.geocities.com/SiliconValley/Park/9289/
             Born in Roswell... married an alien...

Re: pool problems with HEAD??

Posted by Ian Holsman <ia...@apache.org>.
can you run it with efence/purify on ?
this should catch the problem.

Jeff Trawick wrote:

> Jeff Trawick <tr...@attglobal.net> writes:
> 
> 
>>Aaron Bannert <aa...@clove.org> writes:
>>
>>
>>>On Thu, Jan 10, 2002 at 11:28:47AM -0500, Jeff Trawick wrote:
>>>
>>>>CVS HEAD, Solaris, worker MPM, one process with many threads
>>>>
>>>Sander just suggested that you might try reverting David's
>>>plog/pconf patch. (Sander had to run and asked me to post this).
>>>
>>Thanks for the quick feedback guys!
>>
>>
>>>I'm looking into other possibilities of pool mismanagement in
>>>mod_log_config as well.
>>>
>>I'm going to try this sequence:
>>
>>(1) turn off mod_log_config/access_log and see what happens
>>
> 
> I commented out the one CustomLog and tried again (still Solaris,
> current code):
> 
> Loaded symbols for /usr/lib/libthread.so.1
> #0  0xff1618a8 in memset () from /usr/platform/SUNW,Ultra-250/lib/libc_psr.so.1
> (gdb) where
> #0  0xff1618a8 in memset () from /usr/platform/SUNW,Ultra-250/lib/libc_psr.so.1
> #1  0xff34c56c in apr_pcalloc (pool=0x5d9ce0, size=24) at apr_pools.c:473
> #2  0xff330880 in make_array_core (res=0xffffffff, p=0x5d9ce0, nelts=5, elt_size=4, clear=1) at apr_tables.c:103
> #3  0xff33092c in apr_array_make (p=0x5d9ce0, nelts=5, elt_size=4) at apr_tables.c:121
> #4  0x4e2cc in fixup_vary (r=0x7a00d0) at http_protocol.c:1099
> #5  0x4e630 in ap_http_header_filter (f=0x5e2310, b=0x5e2430) at http_protocol.c:1189
> #6  0xbe208 in ap_pass_brigade (next=0x5e2310, bb=0x5e2430) at util_filter.c:388
> #7  0xc2dd4 in ap_content_length_filter (f=0x5e22f8, b=0x5e2430) at protocol.c:1010
> #8  0xbe208 in ap_pass_brigade (next=0x5e22f8, bb=0x5e2430) at util_filter.c:388
> #9  0x51bc0 in ap_byterange_filter (f=0x5e22e0, bb=0x5e2430) at http_protocol.c:2414
> #10 0xbe208 in ap_pass_brigade (next=0x5e22e0, bb=0x5e2430) at util_filter.c:388
> #11 0xcb918 in default_handler (r=0x7a00d0) at core.c:2815
> #12 0xa5f6c in ap_run_handler (r=0x7a00d0) at config.c:185
> #13 0xa6be4 in ap_invoke_handler (r=0x7a00d0) at config.c:359
> #14 0x53908 in ap_internal_redirect (new_uri=0x7a00c0 "/index.html.en", r=0x5d9d18) at http_request.c:454
> #15 0x8bd34 in handle_map_file (r=0x5d9d18) at mod_negotiation.c:2838
> #16 0xa5f6c in ap_run_handler (r=0x5d9d18) at config.c:185
> #17 0xa6be4 in ap_invoke_handler (r=0x5d9d18) at config.c:359
> #18 0x531bc in ap_process_request (r=0x5d9d18) at http_request.c:292
> #19 0x4b150 in ap_process_http_connection (c=0x5d7e58) at http_core.c:280
> #20 0xba72c in ap_run_process_connection (c=0x5d7e58) at connection.c:84
> #21 0xbace4 in ap_process_connection (c=0x5d7e58) at connection.c:230
> #22 0xa08bc in process_socket (p=0x5d7d38, sock=0x5d7d70, my_child_num=0, my_thread_num=458) at worker.c:562
> #23 0xa11f4 in worker_thread (thd=0x469650, dummy=0x4e2fc0) at worker.c:777
> #24 0xff340990 in dummy_worker (opaque=0x469650) at thread.c:122
> (gdb)
> 
> 
>>(2) revert plog/pconf patch and see what happens
>>
> 
> I applied this patch:
> 
> Index: server/core.c
> ===================================================================
> RCS file: /cvs/apache/httpd-2.0/server/core.c,v
> retrieving revision 1.128
> diff -u -r1.128 core.c
> --- core.c      2002/01/08 17:07:19     1.128
> +++ core.c      2002/01/10 17:54:08
> @@ -3361,7 +3361,8 @@
> 
>  static int core_open_logs(apr_pool_t *pconf, apr_pool_t *plog,
> apr_pool_t *ptemp, server_rec *s)
>  {
> -    ap_open_logs(s, plog);
> +/* XXX */
> +    ap_open_logs(s, pconf);
>      return OK;
>  }
> 
> 
> Here is a backtrace from the coredump I got:
> 
> #0  0xff34c3d0 in apr_palloc (pool=0x7b98e0, size=24) at apr_pools.c:430
> 430         endp = active->first_avail + size;
> (gdb) where
> #0  0xff34c3d0 in apr_palloc (pool=0x7b98e0, size=24) at  apr_pools.c:430
> #1  0xff330ff8 in apr_table_make (p=0x7b98e0, nelts=5) at apr_tables.c:343
> #2  0xcdb18 in core_create_conn (ptrans=0x7b98e0, server=0x1b2ea0, csd=0x7b7910, conn_id=433, sbh=0x7b9880) at core.c:3474
> #3  0xba9d0 in ap_run_create_connection (p=0x7b98e0, server=0x1b2ea0, csd=0x7b7910, conn_id=433, sbh=0x7b9880) at connection.c:86
> #4  0xa089c in process_socket (p=0x7b98e0, sock=0x7b7910, my_child_num=0, my_thread_num=433) at worker.c:560
> #5  0xa11f4 in worker_thread (thd=0x469330, dummy=0x4e2d68) at worker.c:777
> #6  0xff340990 in dummy_worker (opaque=0x469330) at thread.c:122
> (gdb) p *active
> Cannot access memory at address 0x0
> (gdb)
> 
> 
> 
>>(3) revert pool debug stuff and see what happens
>>
> 
> I'll report back separately on this.
> 
> 




Re: pool problems with HEAD??

Posted by Aaron Bannert <aa...@clove.org>.
On Thu, Jan 10, 2002 at 10:14:55AM -0800, Aaron Bannert wrote:
> On Thu, Jan 10, 2002 at 12:53:26PM -0500, Jeff Trawick wrote:
> ...
> > > (2) revert plog/pconf patch and see what happens
> > 
> > I applied this patch:
 
...
> I think this patch is incomplete. I'll post the full reverse patch
> to the pconf changes in a reply to this...

I take it back, your reverse patch was correct. The other changes
were backed out.

-aaron

Re: pool problems with HEAD??

Posted by Aaron Bannert <aa...@clove.org>.
On Thu, Jan 10, 2002 at 10:14:55AM -0800, Aaron Bannert wrote:
> On Thu, Jan 10, 2002 at 12:53:26PM -0500, Jeff Trawick wrote:
> ...
> > > (2) revert plog/pconf patch and see what happens
> > 
> > I applied this patch:
 
...
> I think this patch is incomplete. I'll post the full reverse patch
> to the pconf changes in a reply to this...

I take it back, your reverse patch was correct. The other changes
were backed out.

-aaron

Re: pool problems with HEAD??

Posted by Aaron Bannert <aa...@clove.org>.
On Thu, Jan 10, 2002 at 12:53:26PM -0500, Jeff Trawick wrote:
...
> > (2) revert plog/pconf patch and see what happens
> 
> I applied this patch:
> 
> Index: server/core.c
> ===================================================================
> RCS file: /cvs/apache/httpd-2.0/server/core.c,v
> retrieving revision 1.128
> diff -u -r1.128 core.c
> --- core.c      2002/01/08 17:07:19     1.128
> +++ core.c      2002/01/10 17:54:08
> @@ -3361,7 +3361,8 @@
> 
>  static int core_open_logs(apr_pool_t *pconf, apr_pool_t *plog,
> apr_pool_t *ptemp, server_rec *s)
>  {
> -    ap_open_logs(s, plog);
> +/* XXX */
> +    ap_open_logs(s, pconf);
>      return OK;
>  }
> 
> 
> Here is a backtrace from the coredump I got:
> 
> #0  0xff34c3d0 in apr_palloc (pool=0x7b98e0, size=24) at apr_pools.c:430
> 430         endp = active->first_avail + size;
> (gdb) where
> #0  0xff34c3d0 in apr_palloc (pool=0x7b98e0, size=24) at  apr_pools.c:430
> #1  0xff330ff8 in apr_table_make (p=0x7b98e0, nelts=5) at apr_tables.c:343
> #2  0xcdb18 in core_create_conn (ptrans=0x7b98e0, server=0x1b2ea0, csd=0x7b7910, conn_id=433, sbh=0x7b9880) at core.c:3474
> #3  0xba9d0 in ap_run_create_connection (p=0x7b98e0, server=0x1b2ea0, csd=0x7b7910, conn_id=433, sbh=0x7b9880) at connection.c:86
> #4  0xa089c in process_socket (p=0x7b98e0, sock=0x7b7910, my_child_num=0, my_thread_num=433) at worker.c:560
> #5  0xa11f4 in worker_thread (thd=0x469330, dummy=0x4e2d68) at worker.c:777
> #6  0xff340990 in dummy_worker (opaque=0x469330) at thread.c:122
> (gdb) p *active
> Cannot access memory at address 0x0
> (gdb)

I think this patch is incomplete. I'll post the full reverse patch
to the pconf changes in a reply to this...

-aaron

Re: pool problems with HEAD??

Posted by Aaron Bannert <aa...@clove.org>.
On Thu, Jan 10, 2002 at 12:53:26PM -0500, Jeff Trawick wrote:
...
> > (2) revert plog/pconf patch and see what happens
> 
> I applied this patch:
> 
> Index: server/core.c
> ===================================================================
> RCS file: /cvs/apache/httpd-2.0/server/core.c,v
> retrieving revision 1.128
> diff -u -r1.128 core.c
> --- core.c      2002/01/08 17:07:19     1.128
> +++ core.c      2002/01/10 17:54:08
> @@ -3361,7 +3361,8 @@
> 
>  static int core_open_logs(apr_pool_t *pconf, apr_pool_t *plog,
> apr_pool_t *ptemp, server_rec *s)
>  {
> -    ap_open_logs(s, plog);
> +/* XXX */
> +    ap_open_logs(s, pconf);
>      return OK;
>  }
> 
> 
> Here is a backtrace from the coredump I got:
> 
> #0  0xff34c3d0 in apr_palloc (pool=0x7b98e0, size=24) at apr_pools.c:430
> 430         endp = active->first_avail + size;
> (gdb) where
> #0  0xff34c3d0 in apr_palloc (pool=0x7b98e0, size=24) at  apr_pools.c:430
> #1  0xff330ff8 in apr_table_make (p=0x7b98e0, nelts=5) at apr_tables.c:343
> #2  0xcdb18 in core_create_conn (ptrans=0x7b98e0, server=0x1b2ea0, csd=0x7b7910, conn_id=433, sbh=0x7b9880) at core.c:3474
> #3  0xba9d0 in ap_run_create_connection (p=0x7b98e0, server=0x1b2ea0, csd=0x7b7910, conn_id=433, sbh=0x7b9880) at connection.c:86
> #4  0xa089c in process_socket (p=0x7b98e0, sock=0x7b7910, my_child_num=0, my_thread_num=433) at worker.c:560
> #5  0xa11f4 in worker_thread (thd=0x469330, dummy=0x4e2d68) at worker.c:777
> #6  0xff340990 in dummy_worker (opaque=0x469330) at thread.c:122
> (gdb) p *active
> Cannot access memory at address 0x0
> (gdb)

I think this patch is incomplete. I'll post the full reverse patch
to the pconf changes in a reply to this...

-aaron

RE: pool problems with HEAD??

Posted by Sander Striker <st...@apache.org>.
[...]
> > (2) revert plog/pconf patch and see what happens
> 
> I applied this patch:
> 
> Index: server/core.c
> ===================================================================
> RCS file: /cvs/apache/httpd-2.0/server/core.c,v
> retrieving revision 1.128
> diff -u -r1.128 core.c
> --- core.c      2002/01/08 17:07:19     1.128
> +++ core.c      2002/01/10 17:54:08
> @@ -3361,7 +3361,8 @@
> 
>  static int core_open_logs(apr_pool_t *pconf, apr_pool_t *plog,
> apr_pool_t *ptemp, server_rec *s)
>  {
> -    ap_open_logs(s, plog);
> +/* XXX */
> +    ap_open_logs(s, pconf);
>      return OK;
>  }
> 
> 
> Here is a backtrace from the coredump I got:
> 
> #0  0xff34c3d0 in apr_palloc (pool=0x7b98e0, size=24) at apr_pools.c:430
> 430         endp = active->first_avail + size;
> (gdb) where
> #0  0xff34c3d0 in apr_palloc (pool=0x7b98e0, size=24) at  apr_pools.c:430
> #1  0xff330ff8 in apr_table_make (p=0x7b98e0, nelts=5) at apr_tables.c:343
> #2  0xcdb18 in core_create_conn (ptrans=0x7b98e0, server=0x1b2ea0, csd=0x7b7910, conn_id=433, sbh=0x7b9880) at core.c:3474
> #3  0xba9d0 in ap_run_create_connection (p=0x7b98e0, server=0x1b2ea0, csd=0x7b7910, conn_id=433, sbh=0x7b9880) at connection.c:86
> #4  0xa089c in process_socket (p=0x7b98e0, sock=0x7b7910, my_child_num=0, my_thread_num=433) at worker.c:560
> #5  0xa11f4 in worker_thread (thd=0x469330, dummy=0x4e2d68) at worker.c:777
> #6  0xff340990 in dummy_worker (opaque=0x469330) at thread.c:122
> (gdb) p *active
> Cannot access memory at address 0x0
> (gdb)

Hmmm, this is weird.  This means pool->active == NULL.  AFAIK this can only
happen when the pool is not really valid anymore (it got destroyed somewhere).
The reason I suspect this, is because that is the time the pool struct is
recycled (the node containing the struct is put on the freelist).  So I think
this is a lifetime problem.

Guess what?  We really need lifetime checking in debug mode.
Alternatively you could try running with APR_POOL_DEBUG turned on
and a third party tool like efence or purify.

> > (3) revert pool debug stuff and see what happens
> 
> I'll report back separately on this.
> 
> -- 
> Jeff Trawick

Thanks,

Sander


RE: pool problems with HEAD??

Posted by Sander Striker <st...@apache.org>.
[...]
> > (2) revert plog/pconf patch and see what happens
> 
> I applied this patch:
> 
> Index: server/core.c
> ===================================================================
> RCS file: /cvs/apache/httpd-2.0/server/core.c,v
> retrieving revision 1.128
> diff -u -r1.128 core.c
> --- core.c      2002/01/08 17:07:19     1.128
> +++ core.c      2002/01/10 17:54:08
> @@ -3361,7 +3361,8 @@
> 
>  static int core_open_logs(apr_pool_t *pconf, apr_pool_t *plog,
> apr_pool_t *ptemp, server_rec *s)
>  {
> -    ap_open_logs(s, plog);
> +/* XXX */
> +    ap_open_logs(s, pconf);
>      return OK;
>  }
> 
> 
> Here is a backtrace from the coredump I got:
> 
> #0  0xff34c3d0 in apr_palloc (pool=0x7b98e0, size=24) at apr_pools.c:430
> 430         endp = active->first_avail + size;
> (gdb) where
> #0  0xff34c3d0 in apr_palloc (pool=0x7b98e0, size=24) at  apr_pools.c:430
> #1  0xff330ff8 in apr_table_make (p=0x7b98e0, nelts=5) at apr_tables.c:343
> #2  0xcdb18 in core_create_conn (ptrans=0x7b98e0, server=0x1b2ea0, csd=0x7b7910, conn_id=433, sbh=0x7b9880) at core.c:3474
> #3  0xba9d0 in ap_run_create_connection (p=0x7b98e0, server=0x1b2ea0, csd=0x7b7910, conn_id=433, sbh=0x7b9880) at connection.c:86
> #4  0xa089c in process_socket (p=0x7b98e0, sock=0x7b7910, my_child_num=0, my_thread_num=433) at worker.c:560
> #5  0xa11f4 in worker_thread (thd=0x469330, dummy=0x4e2d68) at worker.c:777
> #6  0xff340990 in dummy_worker (opaque=0x469330) at thread.c:122
> (gdb) p *active
> Cannot access memory at address 0x0
> (gdb)

Hmmm, this is weird.  This means pool->active == NULL.  AFAIK this can only
happen when the pool is not really valid anymore (it got destroyed somewhere).
The reason I suspect this, is because that is the time the pool struct is
recycled (the node containing the struct is put on the freelist).  So I think
this is a lifetime problem.

Guess what?  We really need lifetime checking in debug mode.
Alternatively you could try running with APR_POOL_DEBUG turned on
and a third party tool like efence or purify.

> > (3) revert pool debug stuff and see what happens
> 
> I'll report back separately on this.
> 
> -- 
> Jeff Trawick

Thanks,

Sander


Re: pool problems with HEAD??

Posted by Jeff Trawick <tr...@attglobal.net>.
Jeff Trawick <tr...@attglobal.net> writes:

> Aaron Bannert <aa...@clove.org> writes:
> 
> > On Thu, Jan 10, 2002 at 11:28:47AM -0500, Jeff Trawick wrote:
> > > CVS HEAD, Solaris, worker MPM, one process with many threads
> > 
> > Sander just suggested that you might try reverting David's
> > plog/pconf patch. (Sander had to run and asked me to post this).
> 
> Thanks for the quick feedback guys!
> 
> > I'm looking into other possibilities of pool mismanagement in
> > mod_log_config as well.
> 
> I'm going to try this sequence:
> 
> (1) turn off mod_log_config/access_log and see what happens

I commented out the one CustomLog and tried again (still Solaris,
current code):

Loaded symbols for /usr/lib/libthread.so.1
#0  0xff1618a8 in memset () from /usr/platform/SUNW,Ultra-250/lib/libc_psr.so.1
(gdb) where
#0  0xff1618a8 in memset () from /usr/platform/SUNW,Ultra-250/lib/libc_psr.so.1
#1  0xff34c56c in apr_pcalloc (pool=0x5d9ce0, size=24) at apr_pools.c:473
#2  0xff330880 in make_array_core (res=0xffffffff, p=0x5d9ce0, nelts=5, elt_size=4, clear=1) at apr_tables.c:103
#3  0xff33092c in apr_array_make (p=0x5d9ce0, nelts=5, elt_size=4) at apr_tables.c:121
#4  0x4e2cc in fixup_vary (r=0x7a00d0) at http_protocol.c:1099
#5  0x4e630 in ap_http_header_filter (f=0x5e2310, b=0x5e2430) at http_protocol.c:1189
#6  0xbe208 in ap_pass_brigade (next=0x5e2310, bb=0x5e2430) at util_filter.c:388
#7  0xc2dd4 in ap_content_length_filter (f=0x5e22f8, b=0x5e2430) at protocol.c:1010
#8  0xbe208 in ap_pass_brigade (next=0x5e22f8, bb=0x5e2430) at util_filter.c:388
#9  0x51bc0 in ap_byterange_filter (f=0x5e22e0, bb=0x5e2430) at http_protocol.c:2414
#10 0xbe208 in ap_pass_brigade (next=0x5e22e0, bb=0x5e2430) at util_filter.c:388
#11 0xcb918 in default_handler (r=0x7a00d0) at core.c:2815
#12 0xa5f6c in ap_run_handler (r=0x7a00d0) at config.c:185
#13 0xa6be4 in ap_invoke_handler (r=0x7a00d0) at config.c:359
#14 0x53908 in ap_internal_redirect (new_uri=0x7a00c0 "/index.html.en", r=0x5d9d18) at http_request.c:454
#15 0x8bd34 in handle_map_file (r=0x5d9d18) at mod_negotiation.c:2838
#16 0xa5f6c in ap_run_handler (r=0x5d9d18) at config.c:185
#17 0xa6be4 in ap_invoke_handler (r=0x5d9d18) at config.c:359
#18 0x531bc in ap_process_request (r=0x5d9d18) at http_request.c:292
#19 0x4b150 in ap_process_http_connection (c=0x5d7e58) at http_core.c:280
#20 0xba72c in ap_run_process_connection (c=0x5d7e58) at connection.c:84
#21 0xbace4 in ap_process_connection (c=0x5d7e58) at connection.c:230
#22 0xa08bc in process_socket (p=0x5d7d38, sock=0x5d7d70, my_child_num=0, my_thread_num=458) at worker.c:562
#23 0xa11f4 in worker_thread (thd=0x469650, dummy=0x4e2fc0) at worker.c:777
#24 0xff340990 in dummy_worker (opaque=0x469650) at thread.c:122
(gdb)

> (2) revert plog/pconf patch and see what happens

I applied this patch:

Index: server/core.c
===================================================================
RCS file: /cvs/apache/httpd-2.0/server/core.c,v
retrieving revision 1.128
diff -u -r1.128 core.c
--- core.c      2002/01/08 17:07:19     1.128
+++ core.c      2002/01/10 17:54:08
@@ -3361,7 +3361,8 @@

 static int core_open_logs(apr_pool_t *pconf, apr_pool_t *plog,
apr_pool_t *ptemp, server_rec *s)
 {
-    ap_open_logs(s, plog);
+/* XXX */
+    ap_open_logs(s, pconf);
     return OK;
 }


Here is a backtrace from the coredump I got:

#0  0xff34c3d0 in apr_palloc (pool=0x7b98e0, size=24) at apr_pools.c:430
430         endp = active->first_avail + size;
(gdb) where
#0  0xff34c3d0 in apr_palloc (pool=0x7b98e0, size=24) at  apr_pools.c:430
#1  0xff330ff8 in apr_table_make (p=0x7b98e0, nelts=5) at apr_tables.c:343
#2  0xcdb18 in core_create_conn (ptrans=0x7b98e0, server=0x1b2ea0, csd=0x7b7910, conn_id=433, sbh=0x7b9880) at core.c:3474
#3  0xba9d0 in ap_run_create_connection (p=0x7b98e0, server=0x1b2ea0, csd=0x7b7910, conn_id=433, sbh=0x7b9880) at connection.c:86
#4  0xa089c in process_socket (p=0x7b98e0, sock=0x7b7910, my_child_num=0, my_thread_num=433) at worker.c:560
#5  0xa11f4 in worker_thread (thd=0x469330, dummy=0x4e2d68) at worker.c:777
#6  0xff340990 in dummy_worker (opaque=0x469330) at thread.c:122
(gdb) p *active
Cannot access memory at address 0x0
(gdb)


> (3) revert pool debug stuff and see what happens

I'll report back separately on this.

-- 
Jeff Trawick | trawick@attglobal.net | PGP public key at web site:
       http://www.geocities.com/SiliconValley/Park/9289/
             Born in Roswell... married an alien...

Re: pool problems with HEAD??

Posted by Jeff Trawick <tr...@attglobal.net>.
Jeff Trawick <tr...@attglobal.net> writes:

> Aaron Bannert <aa...@clove.org> writes:
> 
> > On Thu, Jan 10, 2002 at 11:28:47AM -0500, Jeff Trawick wrote:
> > > CVS HEAD, Solaris, worker MPM, one process with many threads
> > 
> > Sander just suggested that you might try reverting David's
> > plog/pconf patch. (Sander had to run and asked me to post this).
> 
> Thanks for the quick feedback guys!
> 
> > I'm looking into other possibilities of pool mismanagement in
> > mod_log_config as well.
> 
> I'm going to try this sequence:
> 
> (1) turn off mod_log_config/access_log and see what happens

I commented out the one CustomLog and tried again (still Solaris,
current code):

Loaded symbols for /usr/lib/libthread.so.1
#0  0xff1618a8 in memset () from /usr/platform/SUNW,Ultra-250/lib/libc_psr.so.1
(gdb) where
#0  0xff1618a8 in memset () from /usr/platform/SUNW,Ultra-250/lib/libc_psr.so.1
#1  0xff34c56c in apr_pcalloc (pool=0x5d9ce0, size=24) at apr_pools.c:473
#2  0xff330880 in make_array_core (res=0xffffffff, p=0x5d9ce0, nelts=5, elt_size=4, clear=1) at apr_tables.c:103
#3  0xff33092c in apr_array_make (p=0x5d9ce0, nelts=5, elt_size=4) at apr_tables.c:121
#4  0x4e2cc in fixup_vary (r=0x7a00d0) at http_protocol.c:1099
#5  0x4e630 in ap_http_header_filter (f=0x5e2310, b=0x5e2430) at http_protocol.c:1189
#6  0xbe208 in ap_pass_brigade (next=0x5e2310, bb=0x5e2430) at util_filter.c:388
#7  0xc2dd4 in ap_content_length_filter (f=0x5e22f8, b=0x5e2430) at protocol.c:1010
#8  0xbe208 in ap_pass_brigade (next=0x5e22f8, bb=0x5e2430) at util_filter.c:388
#9  0x51bc0 in ap_byterange_filter (f=0x5e22e0, bb=0x5e2430) at http_protocol.c:2414
#10 0xbe208 in ap_pass_brigade (next=0x5e22e0, bb=0x5e2430) at util_filter.c:388
#11 0xcb918 in default_handler (r=0x7a00d0) at core.c:2815
#12 0xa5f6c in ap_run_handler (r=0x7a00d0) at config.c:185
#13 0xa6be4 in ap_invoke_handler (r=0x7a00d0) at config.c:359
#14 0x53908 in ap_internal_redirect (new_uri=0x7a00c0 "/index.html.en", r=0x5d9d18) at http_request.c:454
#15 0x8bd34 in handle_map_file (r=0x5d9d18) at mod_negotiation.c:2838
#16 0xa5f6c in ap_run_handler (r=0x5d9d18) at config.c:185
#17 0xa6be4 in ap_invoke_handler (r=0x5d9d18) at config.c:359
#18 0x531bc in ap_process_request (r=0x5d9d18) at http_request.c:292
#19 0x4b150 in ap_process_http_connection (c=0x5d7e58) at http_core.c:280
#20 0xba72c in ap_run_process_connection (c=0x5d7e58) at connection.c:84
#21 0xbace4 in ap_process_connection (c=0x5d7e58) at connection.c:230
#22 0xa08bc in process_socket (p=0x5d7d38, sock=0x5d7d70, my_child_num=0, my_thread_num=458) at worker.c:562
#23 0xa11f4 in worker_thread (thd=0x469650, dummy=0x4e2fc0) at worker.c:777
#24 0xff340990 in dummy_worker (opaque=0x469650) at thread.c:122
(gdb)

> (2) revert plog/pconf patch and see what happens

I applied this patch:

Index: server/core.c
===================================================================
RCS file: /cvs/apache/httpd-2.0/server/core.c,v
retrieving revision 1.128
diff -u -r1.128 core.c
--- core.c      2002/01/08 17:07:19     1.128
+++ core.c      2002/01/10 17:54:08
@@ -3361,7 +3361,8 @@

 static int core_open_logs(apr_pool_t *pconf, apr_pool_t *plog,
apr_pool_t *ptemp, server_rec *s)
 {
-    ap_open_logs(s, plog);
+/* XXX */
+    ap_open_logs(s, pconf);
     return OK;
 }


Here is a backtrace from the coredump I got:

#0  0xff34c3d0 in apr_palloc (pool=0x7b98e0, size=24) at apr_pools.c:430
430         endp = active->first_avail + size;
(gdb) where
#0  0xff34c3d0 in apr_palloc (pool=0x7b98e0, size=24) at  apr_pools.c:430
#1  0xff330ff8 in apr_table_make (p=0x7b98e0, nelts=5) at apr_tables.c:343
#2  0xcdb18 in core_create_conn (ptrans=0x7b98e0, server=0x1b2ea0, csd=0x7b7910, conn_id=433, sbh=0x7b9880) at core.c:3474
#3  0xba9d0 in ap_run_create_connection (p=0x7b98e0, server=0x1b2ea0, csd=0x7b7910, conn_id=433, sbh=0x7b9880) at connection.c:86
#4  0xa089c in process_socket (p=0x7b98e0, sock=0x7b7910, my_child_num=0, my_thread_num=433) at worker.c:560
#5  0xa11f4 in worker_thread (thd=0x469330, dummy=0x4e2d68) at worker.c:777
#6  0xff340990 in dummy_worker (opaque=0x469330) at thread.c:122
(gdb) p *active
Cannot access memory at address 0x0
(gdb)


> (3) revert pool debug stuff and see what happens

I'll report back separately on this.

-- 
Jeff Trawick | trawick@attglobal.net | PGP public key at web site:
       http://www.geocities.com/SiliconValley/Park/9289/
             Born in Roswell... married an alien...

Re: pool problems with HEAD??

Posted by Jeff Trawick <tr...@attglobal.net>.
Jeff Trawick <tr...@attglobal.net> writes:

> (1) turn off mod_log_config/access_log and see what happens
> (2) revert plog/pconf patch and see what happens
> (3) revert pool debug stuff and see what happens

I went back to apr_pools.c/1.125 but still have problems:

Loaded symbols for /usr/lib/libthread.so.1
#0  0xff16067c in exit () from
/usr/platform/SUNW,Ultra-250/lib/libc_psr.so.1
(gdb) where
#0  0xff16067c in exit () from
/usr/platform/SUNW,Ultra-250/lib/libc_psr.so.1
#1  0xb2eb8 in ap_make_dirstr_parent (p=0x7cf8c0, s=0x7d0190 "/export/home/trawick/apacheinst/htdocs/index.html") at util.c:625
#2  0xd1b50 in ap_directory_walk (r=0x7cf8f8) at request.c:1111
#3  0xcb19c in core_map_to_storage (r=0x7cf8f8) at core.c:2659
#4  0xce710 in ap_run_map_to_storage (r=0x7cf8f8) at request.c:109
#5  0xcfb08 in ap_process_request_internal (r=0x7cf8f8) at request.c:174
#6  0xd2b64 in ap_sub_req_method_uri (method=0xf8900 "GET", new_file=0x171160 "index.html", r=0x6a9460, next_filter=0x0) at request.c:1554
#7  0xd2be4 in ap_sub_req_lookup_uri (new_file=0x171160 "index.html", r=0x6a9460, next_filter=0x0) at request.c:1565
#8  0x8c8c8 in fixup_dir (r=0x6a9460) at mod_dir.c:199
#9  0xcec18 in ap_run_fixups (r=0x6a9460) at request.c:113
#10 0xd00b4 in ap_process_request_internal (r=0x6a9460) at request.c:268
#11 0x5319c in ap_process_request (r=0x6a9460) at http_request.c:290
#12 0x4b150 in ap_process_http_connection (c=0x7d3a50) at http_core.c:280
#13 0xba72c in ap_run_process_connection (c=0x7d3a50) at connection.c:84
#14 0xbace4 in ap_process_connection (c=0x7d3a50) at connection.c:230
#15 0xa08bc in process_socket (p=0x7d3930, sock=0x7d3968, my_child_num=0, my_thread_num=330) at worker.c:562
#16 0xa11f4 in worker_thread (thd=0x268610, dummy=0x3dffa8) at worker.c:777
#17 0xff340990 in dummy_worker (opaque=0x268610) at thread.c:122
(gdb) up
#1  0xb2eb8 in ap_make_dirstr_parent (p=0x7cf8c0, s=0x7d0190 "/export/home/trawick/apacheinst/htdocs/index.html") at
util.c:625
625         memcpy(d, s, l);
(gdb) list
620         if (last_slash == NULL) {
621             return apr_pstrdup(p, "");
622         }
623         l = (last_slash - s) + 1;
624         d = apr_palloc(p, l + 1);
625         memcpy(d, s, l);
626         d[l] = 0;
627         return (d);
628     }
629
(gdb) p *d
Cannot access memory at address 0xffffffff
(gdb) p d
$1 = 0xffffffff <Address 0xffffffff out of bounds>
(gdb)


-- 
Jeff Trawick | trawick@attglobal.net | PGP public key at web site:
       http://www.geocities.com/SiliconValley/Park/9289/
             Born in Roswell... married an alien...

Re: pool problems with HEAD??

Posted by Jeff Trawick <tr...@attglobal.net>.
Jeff Trawick <tr...@attglobal.net> writes:

> (1) turn off mod_log_config/access_log and see what happens
> (2) revert plog/pconf patch and see what happens
> (3) revert pool debug stuff and see what happens

I went back to apr_pools.c/1.125 but still have problems:

Loaded symbols for /usr/lib/libthread.so.1
#0  0xff16067c in exit () from
/usr/platform/SUNW,Ultra-250/lib/libc_psr.so.1
(gdb) where
#0  0xff16067c in exit () from
/usr/platform/SUNW,Ultra-250/lib/libc_psr.so.1
#1  0xb2eb8 in ap_make_dirstr_parent (p=0x7cf8c0, s=0x7d0190 "/export/home/trawick/apacheinst/htdocs/index.html") at util.c:625
#2  0xd1b50 in ap_directory_walk (r=0x7cf8f8) at request.c:1111
#3  0xcb19c in core_map_to_storage (r=0x7cf8f8) at core.c:2659
#4  0xce710 in ap_run_map_to_storage (r=0x7cf8f8) at request.c:109
#5  0xcfb08 in ap_process_request_internal (r=0x7cf8f8) at request.c:174
#6  0xd2b64 in ap_sub_req_method_uri (method=0xf8900 "GET", new_file=0x171160 "index.html", r=0x6a9460, next_filter=0x0) at request.c:1554
#7  0xd2be4 in ap_sub_req_lookup_uri (new_file=0x171160 "index.html", r=0x6a9460, next_filter=0x0) at request.c:1565
#8  0x8c8c8 in fixup_dir (r=0x6a9460) at mod_dir.c:199
#9  0xcec18 in ap_run_fixups (r=0x6a9460) at request.c:113
#10 0xd00b4 in ap_process_request_internal (r=0x6a9460) at request.c:268
#11 0x5319c in ap_process_request (r=0x6a9460) at http_request.c:290
#12 0x4b150 in ap_process_http_connection (c=0x7d3a50) at http_core.c:280
#13 0xba72c in ap_run_process_connection (c=0x7d3a50) at connection.c:84
#14 0xbace4 in ap_process_connection (c=0x7d3a50) at connection.c:230
#15 0xa08bc in process_socket (p=0x7d3930, sock=0x7d3968, my_child_num=0, my_thread_num=330) at worker.c:562
#16 0xa11f4 in worker_thread (thd=0x268610, dummy=0x3dffa8) at worker.c:777
#17 0xff340990 in dummy_worker (opaque=0x268610) at thread.c:122
(gdb) up
#1  0xb2eb8 in ap_make_dirstr_parent (p=0x7cf8c0, s=0x7d0190 "/export/home/trawick/apacheinst/htdocs/index.html") at
util.c:625
625         memcpy(d, s, l);
(gdb) list
620         if (last_slash == NULL) {
621             return apr_pstrdup(p, "");
622         }
623         l = (last_slash - s) + 1;
624         d = apr_palloc(p, l + 1);
625         memcpy(d, s, l);
626         d[l] = 0;
627         return (d);
628     }
629
(gdb) p *d
Cannot access memory at address 0xffffffff
(gdb) p d
$1 = 0xffffffff <Address 0xffffffff out of bounds>
(gdb)


-- 
Jeff Trawick | trawick@attglobal.net | PGP public key at web site:
       http://www.geocities.com/SiliconValley/Park/9289/
             Born in Roswell... married an alien...

Re: pool problems with HEAD??

Posted by Jeff Trawick <tr...@attglobal.net>.
Aaron Bannert <aa...@clove.org> writes:

> On Thu, Jan 10, 2002 at 11:28:47AM -0500, Jeff Trawick wrote:
> > CVS HEAD, Solaris, worker MPM, one process with many threads
> 
> Sander just suggested that you might try reverting David's
> plog/pconf patch. (Sander had to run and asked me to post this).

Thanks for the quick feedback guys!

> I'm looking into other possibilities of pool mismanagement in
> mod_log_config as well.

I'm going to try this sequence:

(1) turn off mod_log_config/access_log and see what happens
(2) revert plog/pconf patch and see what happens
(3) revert pool debug stuff and see what happens

-- 
Jeff Trawick | trawick@attglobal.net | PGP public key at web site:
       http://www.geocities.com/SiliconValley/Park/9289/
             Born in Roswell... married an alien...

Re: pool problems with HEAD??

Posted by Jeff Trawick <tr...@attglobal.net>.
Aaron Bannert <aa...@clove.org> writes:

> On Thu, Jan 10, 2002 at 11:28:47AM -0500, Jeff Trawick wrote:
> > CVS HEAD, Solaris, worker MPM, one process with many threads
> 
> Sander just suggested that you might try reverting David's
> plog/pconf patch. (Sander had to run and asked me to post this).

Thanks for the quick feedback guys!

> I'm looking into other possibilities of pool mismanagement in
> mod_log_config as well.

I'm going to try this sequence:

(1) turn off mod_log_config/access_log and see what happens
(2) revert plog/pconf patch and see what happens
(3) revert pool debug stuff and see what happens

-- 
Jeff Trawick | trawick@attglobal.net | PGP public key at web site:
       http://www.geocities.com/SiliconValley/Park/9289/
             Born in Roswell... married an alien...

Re: pool problems with HEAD??

Posted by Aaron Bannert <aa...@clove.org>.
On Thu, Jan 10, 2002 at 11:28:47AM -0500, Jeff Trawick wrote:
> CVS HEAD, Solaris, worker MPM, one process with many threads

Sander just suggested that you might try reverting David's
plog/pconf patch. (Sander had to run and asked me to post this).

I'm looking into other possibilities of pool mismanagement in
mod_log_config as well.

-aaron


Re: pool problems with HEAD??

Posted by Aaron Bannert <aa...@clove.org>.
On Thu, Jan 10, 2002 at 11:28:47AM -0500, Jeff Trawick wrote:
> CVS HEAD, Solaris, worker MPM, one process with many threads

Sander just suggested that you might try reverting David's
plog/pconf patch. (Sander had to run and asked me to post this).

I'm looking into other possibilities of pool mismanagement in
mod_log_config as well.

-aaron


Re: pool problems with HEAD??

Posted by Jeff Trawick <tr...@attglobal.net>.
Jeff Trawick <tr...@attglobal.net> writes:

> . Current code (worker MPM on Linux, at least) will segfault like
>   crazy if you turn on APR_POOL_DEBUG and add "-lefence " to the
>   beginning of EXTRA_LIBS in config_vars.mk (I guess this is the
>   right way to use ElectricFence).

After sufficient sleep and/or caffiene, it occurred to me that I
should try a non-threaded build with efence and APR_POOL_DEBUG on this
Linux box.  Sure enough, plentiful segfaults, but more importantly
usable coredumps...

I just fixed one of the causes of segfaults (bad parm in mod_rewrite)
(another misstep on my part though...  in the hurry to try to find
the problem in different environments I neglected to modernize this
config file...  but at least we got a fix out of it :) ).

I'm still getting this segfault:

#0  0x80ceb33 in core_input_filter (f=0x40af7fec, b=0x40c13ff4,
 mode=AP_MODE_BLOCKING,
    readbytes=0xbffff9c8) at core.c:2882
2882        APR_BRIGADE_NORMALIZE(ctx->b);
(gdb) where
#0  0x80ceb33 in core_input_filter (f=0x40af7fec, b=0x40c13ff4,
 mode=AP_MODE_BLOCKING,
    readbytes=0xbffff9c8) at core.c:2882
#1  0x80c56d9 in ap_get_brigade (next=0x40af7fec, bb=0x40c13ff4,
 mode=AP_MODE_BLOCKING,
    readbytes=0xbffff9c8) at util_filter.c:362
#2  0x80cea2b in net_time_filter (f=0x40c2dfec, b=0x40c13ff4,
 mode=AP_MODE_BLOCKING,
    readbytes=0xbffff9c8) at core.c:2844
#3  0x80c56d9 in ap_get_brigade (next=0x40c2dfec, bb=0x40c13ff4,
 mode=AP_MODE_BLOCKING,
    readbytes=0xbffff9c8) at util_filter.c:362
#4  0x80c6c38 in ap_rgetline (s=0x40b1fe9c, n=8192, r=0x40b1fe84,
 fold=0) at protocol.c:225
#5  0x80c7227 in read_request_line (r=0x40b1fe84) at protocol.c:429
#6  0x80c789b in ap_read_request (conn=0x40aabfb8) at protocol.c:608
#7  0x807a2f9 in ap_process_http_connection (c=0x40aabfb8) at
 http_core.c:273
#8  0x80c2f1d in ap_run_process_connection (c=0x40aabfb8) at
 connection.c:84
#9  0x80c335d in ap_process_connection (c=0x40aabfb8) at
 connection.c:230
#10 0x80b49e0 in child_main (child_num_arg=28) at prefork.c:716
#11 0x80b4b79 in make_child (s=0x41af9f9c, slot=28) at prefork.c:806
#12 0x80b4e74 in perform_idle_server_maintenance (p=0x4021afd8) at
 prefork.c:947
#13 0x80b52be in ap_mpm_run (_pconf=0x4021afd8, plog=0x40797fd8,
 s=0x41af9f9c)
    at prefork.c:1118
#14 0x80bbd8e in main (argc=1, argv=0xbffffca4) at main.c:461
(gdb) p ctx
$1 = (core_ctx_t *) 0x40c43ffc
(gdb) p *ctx
$2 = {b = 0x40c49ff4}
(gdb)

This is fairly close to one of the segfaults I was trying to find
yesterday when I ended up going down other paths due to other bugs in
the server and due to missteps of my own.

That segfault has a similar traceback to the one shown above except
that it dies on this apr_bucket_read():

    /* we are reading a single LF line, e.g. the HTTP headers */
    while (!APR_BRIGADE_EMPTY(ctx->b)) {
        const char *pos;

        e = APR_BRIGADE_FIRST(ctx->b);
        rv = apr_bucket_read(e, &str, &len, mode);

        /* We should treat EAGAIN here the same as we do for EOF
        (brigade is

have coredump, will debug... 
-- 
Jeff Trawick | trawick@attglobal.net | PGP public key at web site:
       http://www.geocities.com/SiliconValley/Park/9289/
             Born in Roswell... married an alien...

Re: pool problems with HEAD??

Posted by Jeff Trawick <tr...@attglobal.net>.
After spending a number of hours on this, here is what I have learned:

. If you're going to test old code (e.g., checkout by date) to see if
  it exhibits a certain bad behavior that you have now, make sure you
  apply any patches after the magic date to fix known problems in
  that level.  Otherwise, you'll waste a lot of time :)  It helps
  to use an operating system where you can reliably get dumps, and
  it helps if you actually look at the ones you get to make sure the
  breakage is what you expect.

. "ulimit -n NNNN" is the way to bump up the file descriptor limit on
  some systems where it is too low :)

. If you *don't* do "ulimit -n", current code (worker MPM on Solaris, at
  least) will segfault like crazy.  This was easy to hit with my
  single-process worker configuration.

  I'll put something in the STATUS file for this.

. Current code will segfault (worker MPM on AIX and Solaris, at least)
  if you do apachectl graceful while banging on the server.

  I'll put something in the STATUS file for this.

. Current code (worker MPM on Linux, at least) will segfault like
  crazy if you turn on APR_POOL_DEBUG and add "-lefence " to the
  beginning of EXTRA_LIBS in config_vars.mk (I guess this is the
  right way to use ElectricFence).

  I won't put anything in the STATUS file for this, though it is
  likely a problem with our code somewhere.  I was not successful in
  looking at a dump or capturing a segfault in gdb.

Unfortunately, I went looking for a probable pool-related problem that
somebody told me about and I kept hitting this stuff :(

-- 
Jeff Trawick | trawick@attglobal.net | PGP public key at web site:
       http://www.geocities.com/SiliconValley/Park/9289/
             Born in Roswell... married an alien...

RE: pool problems with HEAD??

Posted by Sander Striker <st...@apache.org>.
Jeff, I'm going over the pools code right now.
I'm wondering if something slipped in when I checked in
the APR_POOL_DEBUG code (*grunt*)

Could you do me a favor and try with one rev earlier
to see if it has the same problem?  In the mean time
I'll try and track it down.

Sander


> -----Original Message-----
> From: trawick@rdu163-40-092.nc.rr.com
> [mailto:trawick@rdu163-40-092.nc.rr.com]On Behalf Of Jeff Trawick
> Sent: 10 January 2002 17:29
> To: dev@apr.apache.org; dev@httpd.apache.org
> Subject: pool problems with HEAD??
> 
> 
> CVS HEAD, Solaris, worker MPM, one process with many threads
> 
> <IfModule worker.c>
> ServerLimit         1
> ThreadLimit         500
> StartServers         1
> MaxClients         500
> MinSpareThreads     25
> MaxSpareThreads     75
> ThreadsPerChild     500
> MaxRequestsPerChild  0
> </IfModule>
> 
> I had a client with many concurrent connections continually doing the
> same request over and over.
> 
> After starting another client to get another 200 or so concurrent
> connections, I quickly got a segfault in mod_log_config because
> supposedly apr_palloc returned some bogus addresses (0 and 0x38).
> 
> #0  0x3fd84 in config_log_transaction (r=Cannot access memory at
>  address 0xed5018f4
> ) at mod_log_config.c:823
> 823             strs[i] = process_item(r, orig, &items[i]);
> (gdb) p strs
> $1 = (char **) 0x0
> (gdb) p format->nelts
> $2 = 14
> (gdb) p strl
> $4 = (int *) 0x38
> (gdb) p i
> $5 = 0
> (gdb)
> (gdb) where
> #0  0x3fd84 in config_log_transaction (r=0x6a62c8, cls=0x1add38,
>  default_format=0x15cde8)
>     at mod_log_config.c:823
> #1  0x3ffd4 in multi_log_transaction (r=0x6a2618) at
>  mod_log_config.c:881
> #2  0xc3f98 in ap_run_log_transaction (r=0x6a2618) at protocol.c:1297
> #3  0x53228 in ap_process_request (r=0x6a2618) at http_request.c:315
> #4  0x4b150 in ap_process_http_connection (c=0x6a0758) at
>  http_core.c:280
> #5  0xba72c in ap_run_process_connection (c=0x6a0758) at
>  connection.c:84
> #6  0xbace4 in ap_process_connection (c=0x6a0758) at connection.c:230
> #7  0xa08bc in process_socket (p=0x6a0638, sock=0x6a0670,
>  my_child_num=0, my_thread_num=187)
>     at worker.c:562
> #8  0xa11f4 in worker_thread (thd=0x2bef88, dummy=0x1163b0) at
>  worker.c:777
> #9  0xff340990 in ?? ()
> 
> The funkiness about r changing in the traceback is (I suspect) because
> in multi_log_transaction we walked through a list of r's using the
> parameter as the variable.  Both r's look healthy.
> 
> Running a level of code from about a day before 2.0.30 on AIX this
> morning, running about the same worker MPM config, I got a segfault
> down in apr_palloc() because some heap structures were messed up.
> 
> Has anybody pounded on an Apache configured like that (one process,
> many threads)?
> 
> -- 
> Jeff Trawick | trawick@attglobal.net | PGP public key at web site:
>        http://www.geocities.com/SiliconValley/Park/9289/
>              Born in Roswell... married an alien...
> 
> 

RE: pool problems with HEAD??

Posted by Sander Striker <st...@apache.org>.
Jeff, I'm going over the pools code right now.
I'm wondering if something slipped in when I checked in
the APR_POOL_DEBUG code (*grunt*)

Could you do me a favor and try with one rev earlier
to see if it has the same problem?  In the mean time
I'll try and track it down.

Sander


> -----Original Message-----
> From: trawick@rdu163-40-092.nc.rr.com
> [mailto:trawick@rdu163-40-092.nc.rr.com]On Behalf Of Jeff Trawick
> Sent: 10 January 2002 17:29
> To: dev@apr.apache.org; dev@httpd.apache.org
> Subject: pool problems with HEAD??
> 
> 
> CVS HEAD, Solaris, worker MPM, one process with many threads
> 
> <IfModule worker.c>
> ServerLimit         1
> ThreadLimit         500
> StartServers         1
> MaxClients         500
> MinSpareThreads     25
> MaxSpareThreads     75
> ThreadsPerChild     500
> MaxRequestsPerChild  0
> </IfModule>
> 
> I had a client with many concurrent connections continually doing the
> same request over and over.
> 
> After starting another client to get another 200 or so concurrent
> connections, I quickly got a segfault in mod_log_config because
> supposedly apr_palloc returned some bogus addresses (0 and 0x38).
> 
> #0  0x3fd84 in config_log_transaction (r=Cannot access memory at
>  address 0xed5018f4
> ) at mod_log_config.c:823
> 823             strs[i] = process_item(r, orig, &items[i]);
> (gdb) p strs
> $1 = (char **) 0x0
> (gdb) p format->nelts
> $2 = 14
> (gdb) p strl
> $4 = (int *) 0x38
> (gdb) p i
> $5 = 0
> (gdb)
> (gdb) where
> #0  0x3fd84 in config_log_transaction (r=0x6a62c8, cls=0x1add38,
>  default_format=0x15cde8)
>     at mod_log_config.c:823
> #1  0x3ffd4 in multi_log_transaction (r=0x6a2618) at
>  mod_log_config.c:881
> #2  0xc3f98 in ap_run_log_transaction (r=0x6a2618) at protocol.c:1297
> #3  0x53228 in ap_process_request (r=0x6a2618) at http_request.c:315
> #4  0x4b150 in ap_process_http_connection (c=0x6a0758) at
>  http_core.c:280
> #5  0xba72c in ap_run_process_connection (c=0x6a0758) at
>  connection.c:84
> #6  0xbace4 in ap_process_connection (c=0x6a0758) at connection.c:230
> #7  0xa08bc in process_socket (p=0x6a0638, sock=0x6a0670,
>  my_child_num=0, my_thread_num=187)
>     at worker.c:562
> #8  0xa11f4 in worker_thread (thd=0x2bef88, dummy=0x1163b0) at
>  worker.c:777
> #9  0xff340990 in ?? ()
> 
> The funkiness about r changing in the traceback is (I suspect) because
> in multi_log_transaction we walked through a list of r's using the
> parameter as the variable.  Both r's look healthy.
> 
> Running a level of code from about a day before 2.0.30 on AIX this
> morning, running about the same worker MPM config, I got a segfault
> down in apr_palloc() because some heap structures were messed up.
> 
> Has anybody pounded on an Apache configured like that (one process,
> many threads)?
> 
> -- 
> Jeff Trawick | trawick@attglobal.net | PGP public key at web site:
>        http://www.geocities.com/SiliconValley/Park/9289/
>              Born in Roswell... married an alien...
> 
>