You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@httpd.apache.org by Niklas Edmundsson <ni...@acc.umu.se> on 2013/02/03 20:32:11 UTC
mod_socache_shmcb segfaults
Hi all!
Something is definitely fishy with mod_socache_shmcb as shipped with
httpd 2.4.3. I'm hacking on a module that uses 16byte indexes (IP(v6)
addresses) to store 2byte counters (16bit uint), these sizes are given
as hints in the call to the init function.
After an hour or so with production load, which means
retrieve/store/remove for each connection, it consistently crashes in
mod_socache_shmcb.c
To keep the essence of this mail somewhat easily readable the output
from my poking and prodding with gdb at one of the corefiles produced
is at the bottom [1] of this mail.
I'm not really clued in on the exact way the internals of
mod_socache_shmcb is supposed to work, but I'm fairly sure that the
value of header->subcache_data_size is simply wrong. It seems to be
set at line 434 in mod_socache_shmcb.c:
header->subcache_data_size = header->subcache_size -
header->subcache_data_offset;
Doing this math from the values in the corefile:
8184-10936 which wraps around to 4294964544 on this platform (Ubuntu
12.04LTS amd64). Actually using this value to calculate an offset
not surprisingly causes a segfault.
I claim that there is something wrong with this logic, but I haven't
got enough knowledge on how it's really supposed to work to suggest a
fix yet.
Suggestions?
[1]
Program terminated with signal 11, Segmentation fault.
#0 __memcpy_ssse3 () at ../sysdeps/x86_64/multiarch/memcpy-ssse3.S:1810
1810 ../sysdeps/x86_64/multiarch/memcpy-ssse3.S: No such file or directory.
(gdb) bt full
#0 __memcpy_ssse3 () at ../sysdeps/x86_64/multiarch/memcpy-ssse3.S:1810
No locals.
#1 0x00007f229f931d10 in shmcb_cyclic_ntoc_memcpy (buf_size=4294964544,
data=0x7f22a1c13a18 " \001\tH", dest_offset=4294901760,
src=0x7f229804b638 " \001\vX", src_len=16) at mod_socache_shmcb.c:193
No locals.
#2 0x00007f229f934c03 in shmcb_subcache_store (s=0x1652050,
header=0x7f22a1bd1008, subcache=0x7f22a1c10f60,
data=0x7f229cfedd2e "\001", data_len=2, id=0x7f229804b638 " \001\vX",
id_len=16, expiry=1362495943271548) at mod_socache_shmcb.c:787
data_offset = 0
new_idx = 0
id_offset = 4294901760
idx = 0x7f229cfedd2e
total_len = 18
#3 0x00007f229f9331fb in socache_shmcb_store (ctx=0x16df588, s=0x1652050,
id=0x7f229804b638 " \001\vX", idlen=16, expiry=1362495943271548,
encoded=0x7f229cfedd2e "\001", len_encoded=2, p=0x7f229804b458)
at mod_socache_shmcb.c:493
header = 0x7f22a1bd1008
subcache = 0x7f22a1c10f60
tryreplace = -1
#4 0x00007f229dcb7454 in connlimit_pre_conn (c=0x7f229804b760,
csd=0x7f229804b4e0) at mod_connlimit.c:344
rv = 70015
val = 1
vallen = 2
ci = 0x7f229804bb40
cfg = 0x1645a30
#5 0x000000000045f679 in ap_run_pre_connection (c=0x7f229804b760,
csd=0x7f229804b4e0) at connection.c:42
pHook = 0x165baf8
n = 1
rv = 0
#6 0x00007f22a034d34d in process_socket (thd=0x164be98, p=0x7f229804b458,
sock=0x7f229804b4e0, cs=0x7f229804b6e8, my_child_num=4, my_thread_num=2)
at event.c:919
pt = 0x7f229804b6d8
c = 0x7f229804b760
conn_id = 258
rc = 32546
sbh = 0x7f229804b6d0
#7 0x00007f22a034fd25 in worker_thread (thd=0x164be98, dummy=0x7f2298000b40)
---Type <return> to continue, or q <return> to quit---
at event.c:1812
ti = 0x7f2298000b40
process_slot = 4
thread_slot = 2
csd = 0x7f229804b4e0
cs = 0x0
ptrans = 0x7f229804b458
rv = 0
is_idle = 0
te = 0x0
#8 0x00007f22a13cd02b in dummy_worker (opaque=0x164be98)
at threadproc/unix/thread.c:142
thread = 0x164be98
#9 0x00007f22a1189e9a in start_thread (arg=0x7f229cfee700)
at pthread_create.c:308
__res = <optimized out>
pd = 0x7f229cfee700
now = <optimized out>
unwind_buf = {cancel_jmp_buf = {{jmp_buf = {0, 938109062393044333,
23374784, 139786639567296, 0, 3, -918192516318945939,
-918286780132549267}, mask_was_saved = 0}}, priv = {pad = {
0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0,
canceltype = 0}}}
not_first_call = 0
pagesize_m1 = <optimized out>
sp = <optimized out>
freesize = <optimized out>
__PRETTY_FUNCTION__ = "start_thread"
#10 0x00007f22a0eb6cbd in clone ()
at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
No locals.
#11 0x0000000000000000 in ?? ()
No symbol table info available.
(gdb) frame 1
#1 0x00007f229f931d10 in shmcb_cyclic_ntoc_memcpy (buf_size=4294964544,
data=0x7f22a1c13a18 " \001\tH", dest_offset=4294901760,
src=0x7f229804b638 " \001\vX", src_len=16) at mod_socache_shmcb.c:193
193 memcpy(data + dest_offset, src, src_len);
(gdb) l
188 unsigned int dest_offset, const unsigned char *src,
189 unsigned int src_len)
190 {
191 if (dest_offset + src_len < buf_size)
192 /* It be copied all in one go */
193 memcpy(data + dest_offset, src, src_len);
194 else {
195 /* Copy the two splits */
196 memcpy(data + dest_offset, src, buf_size - dest_offset);
197 memcpy(data, src + buf_size - dest_offset,
(gdb) print data
$1 = (unsigned char *) 0x7f22a1c13a18 " \001\tH"
(gdb) print dest_offset
$2 = 4294901760
(gdb) print data+dest_offset
$3 = (unsigned char *) 0x7f23a1c03a18 <Address 0x7f23a1c03a18 out of bounds>
(gdb) print src
$4 = (const unsigned char *) 0x7f229804b638 " \001\vX"
(gdb) print src_len
$5 = 16
(gdb) frame 2
#2 0x00007f229f934c03 in shmcb_subcache_store (s=0x1652050,
header=0x7f22a1bd1008, subcache=0x7f22a1c10f60,
data=0x7f229cfedd2e "\001", data_len=2, id=0x7f229804b638 " \001\vX",
id_len=16, expiry=1362495943271548) at mod_socache_shmcb.c:787
787 shmcb_cyclic_ntoc_memcpy(header->subcache_data_size,
(gdb) l
782 * (c.f. FAQ "My log entries are not in date order!")
783 */
784 /* Insert the id */
785 id_offset = SHMCB_CYCLIC_INCREMENT(subcache->data_pos, subcache->data_used,
786 header->subcache_data_size);
787 shmcb_cyclic_ntoc_memcpy(header->subcache_data_size,
788 SHMCB_DATA(header, subcache), id_offset,
789 id, id_len);
790 subcache->data_used += id_len;
791 /* Insert the data */
(gdb) print *header
$6 = {stat_stores = 8345, stat_replaced = 6382, stat_expiries = 0,
stat_scrolled = 123, stat_retrieves_hit = 14583, stat_retrieves_miss = 8464,
stat_removes_hit = 8200, stat_removes_miss = 118, subcache_num = 64,
index_num = 455, subcache_size = 8184, subcache_data_offset = 10936,
subcache_data_size = 4294964544}
/Nikke
--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Niklas Edmundsson, Admin @ {acc,hpc2n}.umu.se | nikke@acc.umu.se
---------------------------------------------------------------------------
"Captain, could I play some jazz?" "Make it soul, No. 1."
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Re: mod_socache_shmcb segfaults
Posted by Niklas Edmundsson <ni...@acc.umu.se>.
Noone familiar enough with this code to have a clue/hint on what's
going on and what's really supposed to happen?
On Sun, 10 Feb 2013, Niklas Edmundsson wrote:
> On Fri, 8 Feb 2013, Joe Orton wrote:
>
>> On Sun, Feb 03, 2013 at 08:32:11PM +0100, Niklas Edmundsson wrote:
>>> Hi all!
>>>
>>> Something is definitely fishy with mod_socache_shmcb as shipped with
>>> httpd 2.4.3. I'm hacking on a module that uses 16byte indexes
>>> (IP(v6) addresses) to store 2byte counters (16bit uint), these sizes
>>> are given as hints in the call to the init function.
>>>
>>> After an hour or so with production load, which means
>>> retrieve/store/remove for each connection, it consistently crashes
>>> in mod_socache_shmcb.c
>>
>> Hi Niklas, thanks for the report and the analysis. You are passing
>> specifically avg_obj_size = 2, avg_id_len = 16 in the hints?
>
> Yup, that's the intention at least. The actual hint-struct is declared as:
>
> static struct ap_socache_hints iplimit_cache_hints = {
> 16, /* average length of IDs */
> sizeof(apr_uint16_t), /* avg size of objects */
> 300000000 /* interval between expiry runs */
> };
>
>> It looks like something goes wrong with the sizing heuristics with small
>> obj/id sizes. I can't work out exactly why but it's definitely a bug.
>
> I'm hoping for someone familiar with that code to chime in with ideas, as the
> fix for this isn't really obvious to me either...
>
> /Nikke
>
/Nikke
--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Niklas Edmundsson, Admin @ {acc,hpc2n}.umu.se | nikke@acc.umu.se
---------------------------------------------------------------------------
Buddhism means never having to say you're sorry.
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Re: mod_socache_shmcb segfaults
Posted by Niklas Edmundsson <ni...@acc.umu.se>.
On Fri, 8 Feb 2013, Joe Orton wrote:
> On Sun, Feb 03, 2013 at 08:32:11PM +0100, Niklas Edmundsson wrote:
>> Hi all!
>>
>> Something is definitely fishy with mod_socache_shmcb as shipped with
>> httpd 2.4.3. I'm hacking on a module that uses 16byte indexes
>> (IP(v6) addresses) to store 2byte counters (16bit uint), these sizes
>> are given as hints in the call to the init function.
>>
>> After an hour or so with production load, which means
>> retrieve/store/remove for each connection, it consistently crashes
>> in mod_socache_shmcb.c
>
> Hi Niklas, thanks for the report and the analysis. You are passing
> specifically avg_obj_size = 2, avg_id_len = 16 in the hints?
Yup, that's the intention at least. The actual hint-struct is declared
as:
static struct ap_socache_hints iplimit_cache_hints = {
16, /* average length of IDs */
sizeof(apr_uint16_t), /* avg size of objects */
300000000 /* interval between expiry runs */
};
> It looks like something goes wrong with the sizing heuristics with small
> obj/id sizes. I can't work out exactly why but it's definitely a bug.
I'm hoping for someone familiar with that code to chime in with ideas,
as the fix for this isn't really obvious to me either...
/Nikke
--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Niklas Edmundsson, Admin @ {acc,hpc2n}.umu.se | nikke@acc.umu.se
---------------------------------------------------------------------------
Frood: really amazingly together guy
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Re: mod_socache_shmcb segfaults
Posted by Joe Orton <jo...@redhat.com>.
On Sun, Feb 03, 2013 at 08:32:11PM +0100, Niklas Edmundsson wrote:
> Hi all!
>
> Something is definitely fishy with mod_socache_shmcb as shipped with
> httpd 2.4.3. I'm hacking on a module that uses 16byte indexes
> (IP(v6) addresses) to store 2byte counters (16bit uint), these sizes
> are given as hints in the call to the init function.
>
> After an hour or so with production load, which means
> retrieve/store/remove for each connection, it consistently crashes
> in mod_socache_shmcb.c
Hi Niklas, thanks for the report and the analysis. You are passing
specifically avg_obj_size = 2, avg_id_len = 16 in the hints?
It looks like something goes wrong with the sizing heuristics with small
obj/id sizes. I can't work out exactly why but it's definitely a bug.
Regards, Joe