You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@httpd.apache.org by "M. Alex Hankins" <lx...@fastmail.fm> on 2004/06/25 19:49:11 UTC

[users@httpd] Segmentation fault in ap_die at http_request.c:143

Does anyone have any experience with this?  Should I add this to
Bugzilla?

We've been getting the occasional segmentation fault in our error logs. 
We have two core files so far.  

* Apache httpd version 2.0.49 with mod_ssl patch from Apache bug 27945
* mod_jk 1.2.5
* Solaris 8
* Tomcat 4.1.24 (probably irrelevant)

gdb says that the cause of death for both was due to signal 9 (SIGKILL),
not 11 (SIGSEGV).  We've set up coreadm to generate core files on a
per-process basis on this machine, and the two core files we have
DEFINITELY match the PIDs and timestamps of the segmentation faults
listed in the error_log.  This is not what I expected.  I mention this
only because it may be a clue.

Here is the stack trace for core file #1:

(gdb) where
#0  0x2ae28 in ap_die (type=302, r=0x1dd6c8) at http_request.c:143
#1  0x2b010 in ap_process_request (r=0x1dd6c8) at http_request.c:259
#2  0x262b4 in ap_process_http_connection (c=0x13d258) at
http_core.c:250
#3  0x38440 in ap_run_process_connection (c=0x13d258) at connection.c:42
#4  0x38754 in ap_process_connection (c=0x13d258, csd=0x13d180)
    at connection.c:175
#5  0x2c5cc in child_main (child_num_arg=440320) at prefork.c:609
#6  0x2c74c in make_child (s=0x793d0, slot=1) at prefork.c:703
#7  0x2c99c in perform_idle_server_maintenance (p=0x77630) at
prefork.c:838
#8  0x2cdd4 in ap_mpm_run (_pconf=0x0, plog=0x52000, s=0x6e000)
    at prefork.c:1039
#9  0x33104 in main (argc=6, argv=0xffbef544) at main.c:617

Here is a deeper analysis of core file #2.  It died in the same place:

(gdb) where
#0  0x2ae28 in ap_die (type=302, r=0x1efa88) at http_request.c:143
#1  0x2b010 in ap_process_request (r=0x1efa88) at http_request.c:259
#2  0x262b4 in ap_process_http_connection (c=0x13d258) at
http_core.c:250
#3  0x38440 in ap_run_process_connection (c=0x13d258) at connection.c:42
#4  0x38754 in ap_process_connection (c=0x13d258, csd=0x13d180)
    at connection.c:175
#5  0x2c5cc in child_main (child_num_arg=440320) at prefork.c:609
#6  0x2c74c in make_child (s=0x793d0, slot=4) at prefork.c:703
#7  0x2c99c in perform_idle_server_maintenance (p=0x77630) at
prefork.c:838
#8  0x2cdd4 in ap_mpm_run (_pconf=0x0, plog=0x52000, s=0x6e000)
    at prefork.c:1039
#9  0x33104 in main (argc=6, argv=0xffbef544) at main.c:617

(gdb) print custom_response
$1 = 0x66476162 <Address 0x66476162 out of bounds>

Ergo segmentation violation.

(gdb) print custom_response[0]
Cannot access memory at address 0x66476162

(gdb) print *r
$3 = {pool = 0x1efa50, connection = 0x13d258, server = 0xcebd8, next =
0x0,
  prev = 0x0, main = 0x0,
  the_request = 0x1f0728 "GET
  /servlet/page?_pageid=60&_dad=portal30&_schema=PORTAL30 HTTP/1.0",
  assbackwards = 0, proxyreq = 0, header_only = 0,
  protocol = 0x1f0830 "HTTP/1.0", proto_num = 1000,
  hostname = 0x1f0c68 "www.intbeta.teamport.app.zzz.com",
  request_time = 1087490615544349, status_line = 0x0, status = 302,
  method = 0x1f0778 "GET", method_number = 0, allowed = 0,
  allowed_xmethods = 0x0, allowed_methods = 0x1efc10, sent_bodyct = 0,
  bytes_sent = 0, mtime = 0, chunked = 0, range = 0x0, clength = 0,
  remaining = 0, read_length = 0, read_body = 0, read_chunked = 0,
  expecting_100 = 0, headers_in = 0x1efc40, headers_out = 0x1f00d0,
  err_headers_out = 0x1f0278, subprocess_env = 0x1efe88, notes =
  0x1f03d0,
  content_type = 0x0, handler = 0x1f0cb8 "jakarta-servlet",
  content_encoding = 0x0, content_languages = 0x0, vlist_validator =
  0x0,
  user = 0x0, ap_auth_type = 0x0, no_cache = 0, no_local_copy = 0,
  unparsed_uri = 0x1f07b8
  "/servlet/page?_pageid=60&_dad=portal30&_schema=PORTAL30", uri =
  0x1f07f0 "/servlet/page", filename = 0x1f07f9 "page",
  canonical_filename = 0x0, path_info = 0x0,
  args = 0x1f0800 "_pageid=60&_dad=portal30&_schema=PORTAL30", finfo = {
    pool = 0x0, valid = 0, protection = 0, filetype = APR_NOFILE, user =
    0,
    group = 0, inode = 0, device = 0, nlink = 0, size = 0, csize = 0,
    atime = 0, mtime = 0, ctime = 0, fname = 0x0, name = 0x0, filehand =
    0x0},
  parsed_uri = {scheme = 0x0, hostinfo = 0x0, user = 0x0, password =
  0x0,
    hostname = 0x0, port_str = 0x0, path = 0x1f07f0 "/servlet/page",
    query = 0x1f0800 "_pageid=60&_dad=portal30&_schema=PORTAL30",
    fragment = 0x0, hostent = 0x0, port = 0, is_initialized = 1,
    dns_looked_up = 0, dns_resolved = 0}, used_path_info = 2,
  per_dir_config = 0xeb108, request_config = 0x1f0528, htaccess = 0x0,
  output_filters = 0x1f06b8, input_filters = 0x1f06a0,
  proto_output_filters = 0x1f06b8, proto_input_filters = 0x1f06a0,
  eos_sent = 0}

(gdb) print error_index
$4 = 2030216

Thanks in advance.

Sincerely,
Alex

---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
   "   from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org


Re: [users@httpd] Segmentation fault in ap_die at http_request.c:143

Posted by Paul Querna <ch...@force-elite.com>.
On Fri, 2004-06-25 at 13:49 -0400, M. Alex Hankins wrote:
> Does anyone have any experience with this?  Should I add this to
> Bugzilla?
> 
Yes, please file a report in Bugzilla.

Thanks,
-Paul Querna

> We've been getting the occasional segmentation fault in our error logs. 
> We have two core files so far.  
> 
> * Apache httpd version 2.0.49 with mod_ssl patch from Apache bug 27945
> * mod_jk 1.2.5
> * Solaris 8
> * Tomcat 4.1.24 (probably irrelevant)
> 
> gdb says that the cause of death for both was due to signal 9 (SIGKILL),
> not 11 (SIGSEGV).  We've set up coreadm to generate core files on a
> per-process basis on this machine, and the two core files we have
> DEFINITELY match the PIDs and timestamps of the segmentation faults
> listed in the error_log.  This is not what I expected.  I mention this
> only because it may be a clue.
> 
> Here is the stack trace for core file #1:
> 
> (gdb) where
> #0  0x2ae28 in ap_die (type=302, r=0x1dd6c8) at http_request.c:143
> #1  0x2b010 in ap_process_request (r=0x1dd6c8) at http_request.c:259
> #2  0x262b4 in ap_process_http_connection (c=0x13d258) at
> http_core.c:250
> #3  0x38440 in ap_run_process_connection (c=0x13d258) at connection.c:42
> #4  0x38754 in ap_process_connection (c=0x13d258, csd=0x13d180)
>     at connection.c:175
> #5  0x2c5cc in child_main (child_num_arg=440320) at prefork.c:609
> #6  0x2c74c in make_child (s=0x793d0, slot=1) at prefork.c:703
> #7  0x2c99c in perform_idle_server_maintenance (p=0x77630) at
> prefork.c:838
> #8  0x2cdd4 in ap_mpm_run (_pconf=0x0, plog=0x52000, s=0x6e000)
>     at prefork.c:1039
> #9  0x33104 in main (argc=6, argv=0xffbef544) at main.c:617
> 
> Here is a deeper analysis of core file #2.  It died in the same place:
> 
> (gdb) where
> #0  0x2ae28 in ap_die (type=302, r=0x1efa88) at http_request.c:143
> #1  0x2b010 in ap_process_request (r=0x1efa88) at http_request.c:259
> #2  0x262b4 in ap_process_http_connection (c=0x13d258) at
> http_core.c:250
> #3  0x38440 in ap_run_process_connection (c=0x13d258) at connection.c:42
> #4  0x38754 in ap_process_connection (c=0x13d258, csd=0x13d180)
>     at connection.c:175
> #5  0x2c5cc in child_main (child_num_arg=440320) at prefork.c:609
> #6  0x2c74c in make_child (s=0x793d0, slot=4) at prefork.c:703
> #7  0x2c99c in perform_idle_server_maintenance (p=0x77630) at
> prefork.c:838
> #8  0x2cdd4 in ap_mpm_run (_pconf=0x0, plog=0x52000, s=0x6e000)
>     at prefork.c:1039
> #9  0x33104 in main (argc=6, argv=0xffbef544) at main.c:617
> 
> (gdb) print custom_response
> $1 = 0x66476162 <Address 0x66476162 out of bounds>
> 
> Ergo segmentation violation.
> 
> (gdb) print custom_response[0]
> Cannot access memory at address 0x66476162
> 
> (gdb) print *r
> $3 = {pool = 0x1efa50, connection = 0x13d258, server = 0xcebd8, next =
> 0x0,
>   prev = 0x0, main = 0x0,
>   the_request = 0x1f0728 "GET
>   /servlet/page?_pageid=60&_dad=portal30&_schema=PORTAL30 HTTP/1.0",
>   assbackwards = 0, proxyreq = 0, header_only = 0,
>   protocol = 0x1f0830 "HTTP/1.0", proto_num = 1000,
>   hostname = 0x1f0c68 "www.intbeta.teamport.app.zzz.com",
>   request_time = 1087490615544349, status_line = 0x0, status = 302,
>   method = 0x1f0778 "GET", method_number = 0, allowed = 0,
>   allowed_xmethods = 0x0, allowed_methods = 0x1efc10, sent_bodyct = 0,
>   bytes_sent = 0, mtime = 0, chunked = 0, range = 0x0, clength = 0,
>   remaining = 0, read_length = 0, read_body = 0, read_chunked = 0,
>   expecting_100 = 0, headers_in = 0x1efc40, headers_out = 0x1f00d0,
>   err_headers_out = 0x1f0278, subprocess_env = 0x1efe88, notes =
>   0x1f03d0,
>   content_type = 0x0, handler = 0x1f0cb8 "jakarta-servlet",
>   content_encoding = 0x0, content_languages = 0x0, vlist_validator =
>   0x0,
>   user = 0x0, ap_auth_type = 0x0, no_cache = 0, no_local_copy = 0,
>   unparsed_uri = 0x1f07b8
>   "/servlet/page?_pageid=60&_dad=portal30&_schema=PORTAL30", uri =
>   0x1f07f0 "/servlet/page", filename = 0x1f07f9 "page",
>   canonical_filename = 0x0, path_info = 0x0,
>   args = 0x1f0800 "_pageid=60&_dad=portal30&_schema=PORTAL30", finfo = {
>     pool = 0x0, valid = 0, protection = 0, filetype = APR_NOFILE, user =
>     0,
>     group = 0, inode = 0, device = 0, nlink = 0, size = 0, csize = 0,
>     atime = 0, mtime = 0, ctime = 0, fname = 0x0, name = 0x0, filehand =
>     0x0},
>   parsed_uri = {scheme = 0x0, hostinfo = 0x0, user = 0x0, password =
>   0x0,
>     hostname = 0x0, port_str = 0x0, path = 0x1f07f0 "/servlet/page",
>     query = 0x1f0800 "_pageid=60&_dad=portal30&_schema=PORTAL30",
>     fragment = 0x0, hostent = 0x0, port = 0, is_initialized = 1,
>     dns_looked_up = 0, dns_resolved = 0}, used_path_info = 2,
>   per_dir_config = 0xeb108, request_config = 0x1f0528, htaccess = 0x0,
>   output_filters = 0x1f06b8, input_filters = 0x1f06a0,
>   proto_output_filters = 0x1f06b8, proto_input_filters = 0x1f06a0,
>   eos_sent = 0}
> 
> (gdb) print error_index
> $4 = 2030216
> 
> Thanks in advance.
> 
> Sincerely,
> Alex
> 
> ---------------------------------------------------------------------
> The official User-To-User support forum of the Apache HTTP Server Project.
> See <URL:http://httpd.apache.org/userslist.html> for more info.
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
>    "   from the digest: users-digest-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
> 


---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
   "   from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org


Re: [users@httpd] Segmentation fault in ap_die at http_request.c:143

Posted by Joe Orton <jo...@redhat.com>.
On Fri, Jun 25, 2004 at 01:49:11PM -0400, M. Alex Hankins wrote:
> (gdb) print custom_response
> $1 = 0x66476162 <Address 0x66476162 out of bounds>
> 
> Ergo segmentation violation.
> 
> (gdb) print custom_response[0]
> Cannot access memory at address 0x66476162

Looks like this could be the following problem which was fixed in
2.0.50:

  *) Fix memory corruption problem with ap_custom_response() function.
     The core per-dir config would later point to request pool data
     that would be reused for different purposes on different requests.
     [Jeff Trawick, based on an old 1.3 patch submitted by Will Lowe]

joe

---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
   "   from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org