You are viewing a plain text version of this content. The canonical link for it is here.
Posted to bugs@httpd.apache.org by bu...@apache.org on 2014/11/03 20:56:13 UTC
[Bug 57177] New: terminated by signal ILL in server/scoreboard.c
line 369
https://issues.apache.org/bugzilla/show_bug.cgi?id=57177
Bug ID: 57177
Summary: terminated by signal ILL in server/scoreboard.c line
369
Product: Apache httpd-2
Version: 2.4.10
Hardware: Sun
OS: Solaris
Status: NEW
Severity: normal
Priority: P2
Component: Core
Assignee: bugs@httpd.apache.org
Reporter: dclarke@blastwave.org
Tough to describe other than a core dump being thrown thus :
(dbx) where
current thread: t@27
[1] 0xffffffff75b010d8(0x1006a8c50, 0x18, 0x6, 0x1006a8c50, 0x1006c6030,
0x1006c6030), at 0xffffffff75b010d8
=>[2] ap_increment_counts(sb = 0x1006a8bb0, r = 0x1006c6030), line 369 in
"scoreboard.c"
[3] eor_bucket_cleanup(data = 0x1006aaa68), line 37 in "eor_bucket.c"
[4] run_cleanups(cref = 0x1006c6028), line 2352 in "apr_pools.c"
[5] apr_pool_destroy(pool = 0x1006c5fb8), line 804 in "apr_pools.c"
[6] eor_bucket_destroy(data = 0x1006c6030), line 90 in "eor_bucket.c"
[7] ssl_io_filter_output(f = 0x1006a92e8, bb = 0x1006aff58), line 1659 in
"ssl_engine_io.c"
[8] ap_pass_brigade(next = 0x1006a92e8, bb = 0x1006aff58), line 590 in
"util_filter.c"
[9] ssl_io_filter_coalesce(f = 0x1006a92c0, bb = 0x1006aff58), line 1558 in
"ssl_engine_io.c"
[10] ap_pass_brigade(next = 0x1006a92c0, bb = 0x1006aff58), line 590 in
"util_filter.c"
[11] ap_process_request_after_handler(r = 0x1006c6030), line 256 in
"http_request.c"
[12] ap_process_async_request(r = 0x1006c6030), line 353 in "http_request.c"
[13] ap_process_http_async_connection(c = 0x1006a8c50), line 143 in
"http_core.c"
[14] ap_process_http_connection(c = 0x1006a8c50), line 228 in "http_core.c"
[15] ap_run_process_connection(0x1006a8c50, 0x1006a88c0, 0x10009f5c8,
0x1001c3540, 0x1006a8bb0, 0x1006aa848), at 0x10007e1d8
[16] process_socket(thd = 0x100281a98, p = 0x1006a8838, sock = 0x1006a88c0,
cs = 0x1006a8bc8, my_child_num = 1, my_thread_num = 24), line 1035 in "event.c"
[17] worker_thread(thd = 0x100281a98, dummy = 0x10065b070), line 1875 in
"event.c"
[18] dummy_worker(opaque = 0x100281a98), line 142 in "thread.c"
(dbx) list
369 bytes = pfn_ap_logio_get_last_bytes(r->connection);
370 }
371 else if (r->method_number == M_GET && r->method[0] == 'H') {
372 bytes = 0;
373 }
374 else {
375 bytes = r->bytes_sent;
376 }
377
378 #ifdef HAVE_TIMES
Not sure how to proceed on this.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 57177] terminated by signal ILL in server/scoreboard.c line 369
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=57177
Yann Ylavic <yl...@gmail.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |ylavic.dev@gmail.com
--- Comment #8 from Yann Ylavic <yl...@gmail.com> ---
Created attachment 32188
--> https://issues.apache.org/bugzilla/attachment.cgi?id=32188&action=edit
Initialize the scoreboard API (not the data) on graceful restarts
One way to fix the issue.
This patch moves the static optional function(s) initialization from
ap_calc_scoreboard_size() to ap_init_scoreboard(), so that when the latter is
called with no data to initialize (NULL), it will only make its optional
function(s) point to the right location.
Hence each MPM is also modified to call ap_init_scoreboard(NULL) on graceful
restarts.
Dennis, can you please verify this works for you (mod_logio loaded still)?
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 57177] terminated by signal ILL in server/scoreboard.c line 369
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=57177
Yann Ylavic <yl...@gmail.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
Attachment #32188|0 |1
is obsolete| |
--- Comment #9 from Yann Ylavic <yl...@gmail.com> ---
Created attachment 32189
--> https://issues.apache.org/bugzilla/attachment.cgi?id=32189&action=edit
Initialize the scoreboard API (not the data) on graceful restarts
Minor change to call ap_init_scoreboard(NULL) from ap_create_scoreboard() when
the scoreboard image exists already (only).
This avoids a double APR_RETRIEVE_OPTIONAL_FN() call when the image is to be
created and allows for such initializations to be centralized in
ap_init_scoreboard().
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 57177] terminated by signal ILL in server/scoreboard.c line 369
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=57177
--- Comment #11 from Yann Ylavic <yl...@gmail.com> ---
Dennys, any news on this, did you try the patch (attachment 32189)?
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 57177] terminated by signal ILL in server/scoreboard.c line 369
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=57177
--- Comment #6 from Rainer Jung <ra...@kippdata.de> ---
There's two occurences of
pfn_ap_logio_get_last_bytes =
APR_RETRIEVE_OPTIONAL_FN(ap_logio_get_last_bytes);
in server/scoreboard.c. It looks like none of the two runs after your graceful
restart. Could you add a log line to each of the two and check, whether they
run or not after gracefully restarting?
The line retrieves the address of a callback function in mod_logio which is
called during the crash. If the load order of the modules changes duriung
graceful restart, then the old address doesn't point any longer to the correct
function. Therefore the address needs to be updated after restart.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 57177] terminated by signal ILL in server/scoreboard.c line 369
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=57177
--- Comment #12 from Dennis Clarke <dc...@blastwave.org> ---
Sorry for the delay. I have been distracted by work on oher items and that
includes this thing :
https://issues.apache.org/bugzilla/show_bug.cgi?id=56324
Also we have a large exploit in PHP today and a new OpenSSL major release and
CMAKE issues etc etc.
Promise I will get into this on this weekend.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 57177] terminated by signal ILL in server/scoreboard.c line 369
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=57177
--- Comment #3 from Dennis Clarke <dc...@blastwave.org> ---
I have isolated httpd into a forground process with a very similar to near
identical config as the server that throws the core dumps. Have not been
able to repeat the crash.
Thus far it seems to only happen when I do a graceful restart.
Theree after I will see twenty or thirty core files an hour.
I am not yet able to isolate what exactly is happening but I do have
core files and that is a good start.
I'll update this if I see something interesting. In the meanwhile it MAY
be a valid bug. Maybe.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 57177] terminated by signal ILL in server/scoreboard.c line 369
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=57177
--- Comment #10 from Dennis Clarke <dc...@blastwave.org> ---
entirely tangential comment :
Working on it. I keep getting interrupted by people and their interpretation
of "work".
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 57177] terminated by signal ILL in server/scoreboard.c line 369
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=57177
--- Comment #4 from Dennis Clarke <dc...@blastwave.org> ---
Looking at ap_sb_handle_t :
struct ap_sb_handle_t {
int child_num;
int thread_num;
};
and then :
AP_DECLARE(void) ap_increment_counts(ap_sb_handle_t *sb, request_rec *r);
Thread 27 crashes in ap_increment_counts with pointers :
(sb = 0x1006a8bb0, r = 0x1006c6030)
So looking at memory at *sb we should see two integers :
(dbx) x 0x1006a8bb0 / 16 x
0x00000001006a8bb0: 0x0000 0x0001 0x0000 0x0018 0x0000 0x0000 0x0000 0x0000
0x00000001006a8bc0: 0x0000 0x0001 0x006a 0x8bc8 0x0000 0x0001 0x006a 0x8bc8
(dbx) print sizeof(int)
sizeof(int ) = 4
So my guess is that child_num is 1 and thread_num is 0x18h = 24.
(dbx) print *sb
*sb = {
child_num = 1
thread_num = 24
}
Sure, looks fine.
Next we have that pesky request_rec r which will be a bit verbose :
(dbx) print r
r = 0x1006c6030
(dbx) print *r
*r = {
pool = 0x1006c5fb8
connection = 0x1006a8c50
server = 0x10028c1e0
next = 0x1006d0538
prev = (nil)
main = (nil)
the_request = 0x1006c7318 "GET / HTTP/1.0"
assbackwards = 0
proxyreq = 0
header_only = 0
proto_num = 1000
protocol = 0x1006c7348 "HTTP/1.0"
hostname = 0x1006c73e0 "dev.mydomain.ca"
request_time = 1415043364521974
status_line = (nil)
status = 401
method_number = 0
method = 0x1000b85b0 "GET"
allowed = 0
allowed_xmethods = (nil)
allowed_methods = 0x1006c62e8
sent_bodyct = 0
bytes_sent = 0
mtime = 0
range = (nil)
clength = 0
chunked = 0
read_body = 0
read_chunked = 0
expecting_100 = 0
kept_body = (nil)
body_table = (nil)
remaining = 0
read_length = 0
headers_in = 0x1006c6328
headers_out = 0x1006c6a28
err_headers_out = 0x1006c6c70
subprocess_env = 0x1006c66a8
notes = 0x1006c6e10
content_type = (nil)
handler = (nil)
content_encoding = (nil)
content_languages = (nil)
vlist_validator = (nil)
user = (nil)
ap_auth_type = 0x100293228 "basic"
unparsed_uri = 0x1006c7338 "/"
uri = 0x1006c7340 "/"
filename = 0x1006c7690 "/usr/local/www/data/htdocs/"
canonical_filename = 0x1006c7690 "/usr/local/www/data/htdocs/"
path_info = 0x1006c75eb ""
args = (nil)
used_path_info = 2
eos_sent = 1
per_dir_config = 0x1006c7a20
request_config = 0x1006c6fb0
log = 0x10028c200
log_id = (nil)
htaccess = 0x1006c79f8
output_filters = 0x1006c7218
input_filters = 0x1006c73f0
proto_output_filters = 0x1006c7218
proto_input_filters = 0x1006c73f0
no_cache = 0
no_local_copy = 1
invoke_mtx = 0x1006c74f0
parsed_uri = {
scheme = (nil)
hostinfo = (nil)
user = (nil)
password = (nil)
hostname = (nil)
port_str = (nil)
path = 0x1006c7340 "/"
query = (nil)
fragment = (nil)
hostent = (nil)
port = 0
is_initialized = 1U
dns_looked_up = 0
dns_resolved = 0
}
finfo = {
pool = 0x1006c5fb8
valid = 7598960
protection = 1909
filetype = APR_DIR
user = 16411
group = 20002
inode = 424070U
device = 1099511693323U
nlink = 18
size = 21
csize = 1536
atime = 1415040555872951
mtime = 1414893925405310
ctime = 1414893925405310
fname = 0x1006c75d0 "/usr/local/www/data/htdocs/"
name = (nil)
filehand = (nil)
}
useragent_addr = 0x1006a8a50
useragent_ip = 0x1006a90b0 "64.41.200.104"
}
(dbx)
Well I can confirm that the uid and gid of the target file fname
is correct :
devl $ ls -ladin /usr/local/www/data/htdocs
424070 drwxrwxr-x 18 16411 20002 21 Nov 2 02:05 /usr/local/www/data/htdocs
Here is where things get interesting. That ip address 64.41.200.104 is the
guys at Qualys where I use their SSL tester to verify my config. So whatever
they are fireing at this server causes it to toss a core file.
I'll try that again and see if I can get a full trace.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 57177] terminated by signal ILL in server/scoreboard.c line 369
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=57177
--- Comment #5 from Dennis Clarke <dc...@blastwave.org> ---
Am happy to report that re-running the SSL scan tests from Qualys SSL Labs does
not cause any ill effects UNTIL I issue an apachectl graceful. Then I get a
pile of core files tossed :
devl# cd /var/crash/dev/coredump/
devl# ls -1tr *.core
node_dev-host_sun4v-zone_z_001-time_1415058824-pid_14569-uid_80-gid_80-fid_httpd.core
node_dev-host_sun4v-zone_z_001-time_1415058857-pid_14567-uid_80-gid_80-fid_httpd.core
node_dev-host_sun4v-zone_z_001-time_1415058858-pid_14572-uid_80-gid_80-fid_httpd.core
node_dev-host_sun4v-zone_z_001-time_1415058858-pid_14566-uid_80-gid_80-fid_httpd.core
node_dev-host_sun4v-zone_z_001-time_1415058876-pid_14565-uid_80-gid_80-fid_httpd.core
I will check each one to see what is happening here after a graceful restart.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 57177] terminated by signal ILL in server/scoreboard.c line 369
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=57177
Eric Covener <co...@gmail.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|NEEDINFO |NEW
--- Comment #2 from Eric Covener <co...@gmail.com> ---
sorry, that change is much older than I thought
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 57177] terminated by signal ILL in server/scoreboard.c line 369
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=57177
--- Comment #7 from Rainer Jung <ra...@kippdata.de> ---
And: as a short term workaround: if you do not need mod_logio, you can disable
the module by not loading it. That should stop your problem (but doesn't fix
the underlying code bug).
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 57177] terminated by signal ILL in server/scoreboard.c line 369
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=57177
Eric Covener <co...@gmail.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|NEW |NEEDINFO
--- Comment #1 from Eric Covener <co...@gmail.com> ---
looks like http://svn.apache.org/viewvc?view=revision&revision=886324 can you
try the patch?
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org