You are viewing a plain text version of this content. The canonical link for it is here.
Posted to apreq-dev@httpd.apache.org by Bojan Smojver <bo...@rexursive.com> on 2006/05/30 04:13:09 UTC
Endless loop in split_on_bdry() of library/parser_multipart.c?
I'm seeing an endless loop in this part of the code:
------------------------------------------------
do {
apr_bucket *f = APR_BRIGADE_FIRST(in);
APR_BUCKET_REMOVE(f);
APR_BRIGADE_INSERT_TAIL(out, f);
} while (e != APR_BRIGADE_FIRST(in));
------------------------------------------------
Basically, I'm submitting a multipart/form-data in lynx and eating the
whole thing up by calling ap_discard_request_body() in my handler. Stack
looks like this:
------------------------------------------------
(gdb) bt
#0 split_on_bdry (out=0x840dc50, in=0x840dc70, pattern=0x840d820,
bdry=0x840d812 "\r\n--LYNX") at parser_multipart.c:166
#1 0x006f0b29 in apreq_parse_multipart (parser=0x840d5b0, t=0x840d630,
bb=0x840d5d0) at parser_multipart.c:580
#2 0x00229dec in apreq_filter (f=0x841e768, bb=0x840d550,
mode=AP_MODE_READBYTES, block=APR_BLOCK_READ, readbytes=8192)
at ../../include/apreq_parser.h:126
#3 0x0808f6a5 in ap_discard_request_body (r=0x841cb50) at
http_filters.c:1123
#4 0x00c83593 in handler (r=0x841cb50) at mod_spin.c:767
#5 0x08075179 in ap_run_handler (r=0x841cb50) at config.c:157
#6 0x08078401 in ap_invoke_handler (r=0x841cb50) at config.c:371
#7 0x0808f158 in ap_process_request (r=0x841cb50) at http_request.c:258
#8 0x0808c45b in ap_process_http_connection (c=0x8406ea0) at
http_core.c:172
#9 0x0807c2a9 in ap_run_process_connection (c=0x8406ea0) at
connection.c:43
#10 0x080a26e0 in child_main (child_num_arg=Variable "child_num_arg" is
not available.
) at prefork.c:640
#11 0x080a2943 in make_child (s=0x835bcb0, slot=0) at prefork.c:680
#12 0x080a3726 in ap_mpm_run (_pconf=0x83570a8, plog=0x83951a0,
s=0x835bcb0)
at prefork.c:956
#13 0x08062d35 in main (argc=137711912, argv=0x0) at main.c:717
------------------------------------------------
The files (two of them) I'm submitting through the form are 690 kB in
size.
The code never gets out of the above while loop. Anyone seen anything
similar?
--
Bojan
Re: Endless loop in split_on_bdry() of library/parser_multipart.c?
Posted by Bojan Smojver <bo...@rexursive.com>.
Quoting Joe Schaefer <jo...@sunstarsys.com>:
> do { ... } while (0) is better- it's no longer a loop.
Instead of "while (e != APR_BRIGADE_FIRST(in))"?
> What is the assembler now?
Similar:
-----------------------------------------------
.L19:
movq 40(%rsp), %rbp
.LVL27:
movq 8(%rsp), %rdi
movq %r13, %rdx
.LVL28:
movq %r14, %rsi
.LVL29:
subq %r15, %rbp
cmpq %r13, %rbp
cmovbe %rbp, %rdx
addq %r15, %rdi
call strncmp
testl %eax, %eax
je .L47
.loc 1 159 0
testq %r15, %r15
je .L27
.LBB3:
.LBB4:
.loc 1 169 0
movq 24(%rsp), %rax
xorl %r15d, %r15d
movq 8(%rax), %rsi
.LVL30:
cmpq %rsi, %r12
.p2align 4,,7
.L29:
.loc 1 170 0
movq 8(%r12), %rdx
.LVL31:
.loc 1 171 0
movq 8(%rsi), %rcx
movq (%rdx), %rax
movq %rax, (%rcx)
movq (%rdx), %rax
.loc 1 172 0
movq %rbx, (%rdx)
.loc 1 171 0
movq %rcx, 8(%rax)
.loc 1 172 0
movq 8(%rbx), %rax
movq %rax, 8(%rsi)
movq 8(%rbx), %rax
movq %rdx, 8(%rbx)
movq %rsi, (%rax)
.LBE4:
.loc 1 173 0
jne .L29
jmp .L19
-----------------------------------------------
--
Bojan
Re: Endless loop in split_on_bdry() of library/parser_multipart.c?
Posted by Joe Schaefer <jo...@sunstarsys.com>.
Bojan Smojver <bo...@rexursive.com> writes:
> Quoting Joe Schaefer <jo...@sunstarsys.com>:
>
>> This is the right approach, I think. But the person who'd be
>> in the best place to test/commit it is Bojan. Just be sure
>> to bump the patch level in apreq_version.h, and add a comment
>> to CHANGES.
>
> This code hangs the test (Joe Orton's original suggestion):
>
> -------------------------------
> do {
> apr_bucket *f = APR_RING_FIRST(&in->list);
> apr_bucket *l = APR_RING_LAST(&in->list);
> APR_RING_UNSPLICE(f, l, link);
> APR_RING_SPLICE_TAIL(&out->list, f, l, apr_bucket, link);
> } while (e != APR_BRIGADE_FIRST(in));
> -------------------------------
>
> This code also hangs the test (Joe Schaefer's subsequent suggestion):
>
> -------------------------------
> do {
> apr_bucket *f = APR_RING_FIRST(&in->list);
> apr_bucket *l = APR_BUCKET_PREV(e);
> APR_RING_UNSPLICE(f, l, link);
> APR_RING_SPLICE_TAIL(&out->list, f, l, apr_bucket, link);
> } while (e != APR_BRIGADE_FIRST(in));
> -------------------------------
>
> When compiled with -fno-strict-aliasing, Joe Schaefer's suggestion
> passes the test, Joe Orton's suggestion hangs the test.
>
> Or maybe I misunderstood what the code should actually be?
do { ... } while (0) is better- it's no longer a loop.
What is the assembler now?
--
Joe Schaefer
Re: Endless loop in split_on_bdry() of library/parser_multipart.c?
Posted by Bojan Smojver <bo...@rexursive.com>.
Quoting Joe Schaefer <jo...@sunstarsys.com>:
> This is the right approach, I think. But the person who'd be
> in the best place to test/commit it is Bojan. Just be sure
> to bump the patch level in apreq_version.h, and add a comment
> to CHANGES.
This code hangs the test (Joe Orton's original suggestion):
-------------------------------
do {
apr_bucket *f = APR_RING_FIRST(&in->list);
apr_bucket *l = APR_RING_LAST(&in->list);
APR_RING_UNSPLICE(f, l, link);
APR_RING_SPLICE_TAIL(&out->list, f, l, apr_bucket, link);
} while (e != APR_BRIGADE_FIRST(in));
-------------------------------
This code also hangs the test (Joe Schaefer's subsequent suggestion):
-------------------------------
do {
apr_bucket *f = APR_RING_FIRST(&in->list);
apr_bucket *l = APR_BUCKET_PREV(e);
APR_RING_UNSPLICE(f, l, link);
APR_RING_SPLICE_TAIL(&out->list, f, l, apr_bucket, link);
} while (e != APR_BRIGADE_FIRST(in));
-------------------------------
When compiled with -fno-strict-aliasing, Joe Schaefer's suggestion
passes the test, Joe Orton's suggestion hangs the test.
Or maybe I misunderstood what the code should actually be?
--
Bojan
Re: Endless loop in split_on_bdry() of library/parser_multipart.c?
Posted by Bojan Smojver <bo...@rexursive.com>.
Quoting Bojan Smojver <bo...@rexursive.com>:
> Looks like the offsetof() provided by the platform isn't being used.
> Which in turn causes a lot of casting all over the place, which
> creates the aliasing problem? Maybe?
Nah, it isn't that... Fails just the same with native offsetof() :-(
--
Bojan
Re: Endless loop in split_on_bdry() of library/parser_multipart.c?
Posted by Bojan Smojver <bo...@rexursive.com>.
Quoting Bojan Smojver <bo...@rexursive.com>:
> I can test the above in my setup and see what comes out, but I'm more
> worried about the incorrectly compiled code and why it happened. I'm not
> sure if it's due to bugs in gcc, or is it something that RING/BRIGADE
> macros are doing that really isn't quite right.
I just posted this to the RH bug report:
==========================================================
One thing seems strange in the intermediate code:
------------------------------
((long) (((char *) (&(((struct apr_bucket*)((void *)0))->link))) - ((char *)
((void *)0))))
------------------------------
The above code is most likely this from apr_general.h:
------------------------------
#define APR_OFFSET(p_type,field) \
((long) (((char *) (&(((p_type)NULL)->field))) - ((char *) NULL)))
------------------------------
I was under the impression that on Linux, there is offsetof that can be used.
So, I created a little test program:
------------------------------
#include <stdio.h>
#include <stddef.h>
int main(int argc,char **argv){
struct test{
int a;
char *b;
};
printf("%lu\n",offsetof(struct test,b));
return 0;
}
------------------------------
Compiled it:
------------------------------
gcc -Wall -save-temps -ansi -o test test.c
------------------------------
This give intermediate code:
------------------------------
int main(int argc,char **argv){
struct test{
int a;
char *b;
};
printf("%lu\n",__builtin_offsetof (struct test, b));
return 0;
}
------------------------------
Sooo, why isn't APR code using that instead? No idea...
BTW, the output of the program on an x86_64 box is 8.
==========================================================
Looks like the offsetof() provided by the platform isn't being used.
Which in turn causes a lot of casting all over the place, which
creates the aliasing problem? Maybe?
--
Bojan
Re: Endless loop in split_on_bdry() of library/parser_multipart.c?
Posted by Bojan Smojver <bo...@rexursive.com>.
On Sun, 2006-06-11 at 20:31 -0400, Joe Schaefer wrote:
> > APR_RING_UNSPLICE(f, l, link);
> > APR_RING_SPLICE_TAIL(&out->list, f, l, apr_bucket, link);
> >
>
> This is the right approach, I think. But the person who'd be
> in the best place to test/commit it is Bojan. Just be sure
> to bump the patch level in apreq_version.h, and add a comment
> to CHANGES.
I can test the above in my setup and see what comes out, but I'm more
worried about the incorrectly compiled code and why it happened. I'm not
sure if it's due to bugs in gcc, or is it something that RING/BRIGADE
macros are doing that really isn't quite right.
RH gcc guys know about it and they already put a few comments into the
bug report, but I'm guessing they are still working on a final verdict.
In any event, we know how to work around the issue on the platform in
question (Fedora Core), so the whole thing is under control. Well, sort
of anyway... :-)
--
Bojan
Re: Endless loop in split_on_bdry() of library/parser_multipart.c?
Posted by Bojan Smojver <bo...@rexursive.com>.
On Wed, 2008-06-04 at 21:35 -0700, Joe Schaefer wrote:
> Needs a comment in the source about why we're using
> volatile here, but otherwise +1.
Done on both the trunk and v2_10 branch.
Any comment regarding my other patch about strict aliasing warnings?
--
Bojan
Re: Endless loop in split_on_bdry() of library/parser_multipart.c?
Posted by Bojan Smojver <bo...@rexursive.com>.
On Wed, 2008-06-04 at 21:35 -0700, Joe Schaefer wrote:
> Needs a comment in the source about why we're using
> volatile here, but otherwise +1.
Ah, yes... Sorry. I also need to fix up the comment in the CHANGES file.
--
Bojan
Re: Endless loop in split_on_bdry() of library/parser_multipart.c?
Posted by Joe Schaefer <jo...@yahoo.com>.
--- Bojan Smojver <bo...@rexursive.com> wrote:
> I propose we fix this as attached. I tested this on
> Fedora 9 and it
> works OK. Opinions?
Needs a comment in the source about why we're using
volatile here, but otherwise +1.
Re: Endless loop in split_on_bdry() of library/parser_multipart.c?
Posted by Bojan Smojver <bo...@rexursive.com>.
I propose we fix this as attached. I tested this on Fedora 9 and it
works OK. Opinions?
--
Bojan
Re: Endless loop in split_on_bdry() of library/parser_multipart.c?
Posted by Bojan Smojver <bo...@rexursive.com>.
On Sun, 2006-06-11 at 20:31 -0400, Joe Schaefer wrote:
> This is the right approach, I think. But the person who'd be
> in the best place to test/commit it is Bojan. Just be sure
> to bump the patch level in apreq_version.h, and add a comment
> to CHANGES.
Get this. This code works:
------------------------------------------
do {
apr_bucket *f;
if (APR_BUCKET_PREV(e) == APR_BRIGADE_SENTINEL(in))
break;
f = APR_BRIGADE_FIRST(in);
APR_BUCKET_REMOVE(f);
APR_BRIGADE_INSERT_TAIL(out, f);
} while (APR_BUCKET_PREV(e) != APR_BRIGADE_SENTINEL(in));
------------------------------------------
This code doesn't (fails on different tests, but in the same loop):
------------------------------------------
do {
apr_bucket *f = APR_BRIGADE_FIRST(in);
APR_BUCKET_REMOVE(f);
APR_BRIGADE_INSERT_TAIL(out, f);
} while (APR_BUCKET_PREV(e) != APR_BRIGADE_SENTINEL(in));
------------------------------------------
I think I'm probably too tired right now to understand this one...
--
Bojan
Re: Endless loop in split_on_bdry() of library/parser_multipart.c?
Posted by Joe Schaefer <jo...@sunstarsys.com>.
"Philip M. Gollucci" <pg...@p6m7g8.com> writes:
> do {
> apr_bucket *f = APR_BRIGADE_FIRST(in);
> APR_BUCKET_REMOVE(f);
> APR_BRIGADE_INSERT_TAIL(out, f);
> } while (e != APR_BRIGADE_FIRST(in));
This loop appends all the buckets before e onto the end of out.
>
> apr_bucket *f = APR_RING_FIRST(&in->list);
> apr_bucket *l = APR_RING_LAST(&in->list);
That l is wrong, it should point at APR_BUCKET_PREV(e)
(which is not the sentinel).
>
> APR_RING_UNSPLICE(f, l, link);
> APR_RING_SPLICE_TAIL(&out->list, f, l, apr_bucket, link);
>
This is the right approach, I think. But the person who'd be
in the best place to test/commit it is Bojan. Just be sure
to bump the patch level in apreq_version.h, and add a comment
to CHANGES.
--
Joe Schaefer
Re: Endless loop in split_on_bdry() of library/parser_multipart.c?
Posted by "Philip M. Gollucci" <pg...@p6m7g8.com>.
Joe Orton wrote:
> A loop like this will not terminate if "e" points to the brigade
> sentinel on entry; not sure if this is possible in the context of this
> code, but it's easy enough to verify with an assertion.
else if (off > 0) {
/* prior (partial) strncmp failed,
* so we can move previous buckets across
* and retest buf against the full bdry.
*/
#include <assert.h>
assert(e != APR_BRIGADE_SENTINEL(in));
do {
apr_bucket *f = APR_BRIGADE_FIRST(in);
APR_BUCKET_REMOVE(f);
APR_BRIGADE_INSERT_TAIL(out, f);
} while (e != APR_BRIGADE_FIRST(in));
off = 0;
goto look_for_boundary_up_front;
}
Apparently that never happens ... at least in the cases the tests throw
at it.
> If all you want to do is move a section of buckets onto another brigade
> you can actually do it in a constant time operation, i.e. without having
> to iterate through all the buckets. To splice buckets from "a" to "b"
> inclusively onto brigade "output", do:
>
> APR_RING_UNSPLICE(a, b, link);
> APR_RING_SPLICE_TAIL(&output->list, a, b, apr_bucket, link);
>
> and APR_RING_SPLICE_HEAD can be used similarly. It's a good idea to use
> the _CHECK_CONSISTENCY macros after using any APR_RING_* macros directly
> so that debug builds run the sanity checks.
Apparently, I'm missing some knowlege. Aren't brigades implemented as
rings. Can you safely mix the use of the macros? APREQ used the
brigade versions throughout.
apr_bucket *f = APR_RING_FIRST(&in->list);
apr_bucket *l = APR_RING_LAST(&in->list);
APR_RING_UNSPLICE(f, l, link);
APR_RING_SPLICE_TAIL(&out->list, f, l, apr_bucket, link);
APR_BRIGADE_CHECK_CONSISTENCY(f);
APR_BRIGADE_CHECK_CONSISTENCY(l);
doesn't quite work, but I'm still adjusting to the API and need to read up.
--
------------------------------------------------------------------------
Philip M. Gollucci (pgollucci@p6m7g8.com) 323.219.4708
Consultant / http://p6m7g8.net/Resume/resume.shtml
Senior Software Engineer - TicketMaster - http://ticketmaster.com
1024D/A79997FA F357 0FDD 2301 6296 690F 6A47 D55A 7172 A799 97F
"It takes a minute to have a crush on someone, an hour to like someone,
and a day to love someone, but it takes a lifetime to forget someone..."
Re: Endless loop in split_on_bdry() of library/parser_multipart.c?
Posted by Joe Orton <jo...@redhat.com>.
[please CC me on replies]
On Tue, May 30, 2006 at 12:13:09PM +1000, Bojan Smojver wrote:
> I'm seeing an endless loop in this part of the code:
>
> ------------------------------------------------
> do {
> apr_bucket *f = APR_BRIGADE_FIRST(in);
> APR_BUCKET_REMOVE(f);
> APR_BRIGADE_INSERT_TAIL(out, f);
> } while (e != APR_BRIGADE_FIRST(in));
> ------------------------------------------------
A loop like this will not terminate if "e" points to the brigade
sentinel on entry; not sure if this is possible in the context of this
code, but it's easy enough to verify with an assertion.
If all you want to do is move a section of buckets onto another brigade
you can actually do it in a constant time operation, i.e. without having
to iterate through all the buckets. To splice buckets from "a" to "b"
inclusively onto brigade "output", do:
APR_RING_UNSPLICE(a, b, link);
APR_RING_SPLICE_TAIL(&output->list, a, b, apr_bucket, link);
and APR_RING_SPLICE_HEAD can be used similarly. It's a good idea to use
the _CHECK_CONSISTENCY macros after using any APR_RING_* macros directly
so that debug builds run the sanity checks.
Regards,
joe
Re: Endless loop in split_on_bdry() of library/parser_multipart.c?
Posted by Bojan Smojver <bo...@rexursive.com>.
On Tue, 2006-05-30 at 12:13 +1000, Bojan Smojver wrote:
> I'm seeing an endless loop in this part of the code:
Sorry... Forgot to mention that this is 2.08-RC2.
--
Bojan
Re: Endless loop in split_on_bdry() of library/parser_multipart.c?
Posted by Bojan Smojver <bo...@rexursive.com>.
Quoting "Philip M. Gollucci" <pg...@p6m7g8.com>:
> Seems to be Fedora Core X specific.
Happens on x84_64 as well and with 2.07. Rebuilding the package in
Fedora Extras 5 now.
--
Bojan
Re: Endless loop in split_on_bdry() of library/parser_multipart.c?
Posted by Bojan Smojver <bo...@rexursive.com>.
On Thu, 2006-06-01 at 02:34 -0700, Philip M. Gollucci wrote:
> Seems to be Fedora Core X specific.
Yeah, very possible.
> Should we figure out how to re-write the loop so it doesn't matter,
> add a prereq check in build/version_check.pl, or just leave it ?
I maintain Fedora Extras libapreq2 package and I already applied the
relevant flag. Let's see what compiler folks tell us and then we can
rewrite if needed.
--
Bojan
Re: Endless loop in split_on_bdry() of library/parser_multipart.c?
Posted by "Philip M. Gollucci" <pg...@p6m7g8.com>.
Bojan Smojver wrote:
> Quoting Joe Schaefer <jo...@sunstarsys.com>:
>
>> At least now it's a bit clearer why the no-strict-aliasing
>> optimization is getting confused ;-)
>
> Hey, speak for yourself ;-)
FreeBSD 6.1-RELEASE
gcc version 4.1.2 20060526 (prerelease)
gcc version 3.4.4 20050518
default maintainer mode CFLAGS work and I can also use combination of these:
-O3 -fno-strict-aliasing -fomit-frame-pointer
Seems to be Fedora Core X specific.
Should we figure out how to re-write the loop so it doesn't matter,
add a prereq check in build/version_check.pl, or just leave it ?
--
------------------------------------------------------------------------
Philip M. Gollucci (pgollucci@p6m7g8.com) 323.219.4708
Consultant / http://p6m7g8.net/Resume/resume.shtml
Senior Software Engineer - TicketMaster - http://ticketmaster.com
1024D/A79997FA F357 0FDD 2301 6296 690F 6A47 D55A 7172 A799 97F
"It takes a minute to have a crush on someone, an hour to like someone,
and a day to love someone, but it takes a lifetime to forget someone..."
Re: Endless loop in split_on_bdry() of library/parser_multipart.c?
Posted by Bojan Smojver <bo...@rexursive.com>.
Quoting Joe Schaefer <jo...@sunstarsys.com>:
> At least now it's a bit clearer why the no-strict-aliasing
> optimization is getting confused ;-)
Hey, speak for yourself ;-)
--
Bojan
Re: Endless loop in split_on_bdry() of library/parser_multipart.c?
Posted by Joe Schaefer <jo...@sunstarsys.com>.
Bojan Smojver <bo...@rexursive.com> writes:
> Quoting Joe Schaefer <jo...@sunstarsys.com>:
>
>> Are you sure the "intermediate code" is correct?
>> It's calling free and destroy, which makes me think
>> that's some other "delete buckets" loop, not this one.>
>
> Nice catch! The correct snippet is:
>
> -------------------------------
> do {
> apr_bucket *f = (&(in)->list)->next;
> do { (((((f))))->link.prev)->link.next = ((((f))))->link.next;
> (((((f))))->link.next)->link.prev = ((((f))))->link.prev; } while (0);
> do { apr_bucket *ap__b = (f); do { ((((ap__b))))->link.next =
> ((struct apr_bucket *)((char *)(((&(out)->list))) - ((long) (((char *)
> (&(((struct apr_bucket*)((void *)0))->link))) - ((char *) ((void *)0))))));
> ((((ap__b))))->link.prev = (((struct apr_bucket *)((char *)(((&(out)->list)))
> - ((long) (((char *) (&(((struct apr_bucket*)((void *)0))->link))) - ((char *)
> ((void *)0)))))))->link.prev; ((((struct apr_bucket *)((char
> *)(((&(out)->list))) - ((long) (((char *) (&(((struct apr_bucket*)((void
> *)0))->link))) - ((char *) ((void *)0)))))))->link.prev)->link.next =
> (((ap__b))); (((struct apr_bucket *)((char *)(((&(out)->list))) - ((long)
> (((char *) (&(((struct apr_bucket*)((void *)0))->link))) - ((char *) ((void
> *)0)))))))->link.prev = (((ap__b))); } while (0); ; } while (0);
> } while (e != (&(in)->list)->next);
> -------------------------------
At least now it's a bit clearer why the no-strict-aliasing
optimization is getting confused ;-)
--
Joe Schaefer
Re: Endless loop in split_on_bdry() of library/parser_multipart.c?
Posted by Bojan Smojver <bo...@rexursive.com>.
Quoting Joe Schaefer <jo...@sunstarsys.com>:
> Are you sure the "intermediate code" is correct?
> It's calling free and destroy, which makes me think
> that's some other "delete buckets" loop, not this one.>
Nice catch! The correct snippet is:
-------------------------------
do {
apr_bucket *f = (&(in)->list)->next;
do { (((((f))))->link.prev)->link.next =
((((f))))->link.next; (((((f))))->link.next)->link.prev =
((((f))))->link.prev; } while (0);
do { apr_bucket *ap__b = (f); do {
((((ap__b))))->link.next = ((struct apr_bucket *)((char
*)(((&(out)->list))) - ((long) (((char *) (&(((struct
apr_bucket*)((void *)0))->link))) - ((char *) ((void *)0))))));
((((ap__b))))->link.prev = (((struct apr_bucket *)((char
*)(((&(out)->list))) - ((long) (((char *) (&(((struct
apr_bucket*)((void *)0))->link))) - ((char *) ((void
*)0)))))))->link.prev; ((((struct apr_bucket *)((char
*)(((&(out)->list))) - ((long) (((char *) (&(((struct
apr_bucket*)((void *)0))->link))) - ((char *) ((void
*)0)))))))->link.prev)->link.next = (((ap__b))); (((struct apr_bucket
*)((char *)(((&(out)->list))) - ((long) (((char *) (&(((struct
apr_bucket*)((void *)0))->link))) - ((char *) ((void
*)0)))))))->link.prev = (((ap__b))); } while (0); ; } while (0);
} while (e != (&(in)->list)->next);
-------------------------------
Thanks Joe!
PS. Fixed Bugzilla too.
--
Bojan
Re: Endless loop in split_on_bdry() of library/parser_multipart.c?
Posted by Joe Schaefer <jo...@sunstarsys.com>.
Bojan Smojver <bo...@rexursive.com> writes:
> Quoting Bojan Smojver <bo...@rexursive.com>:
>
>> I will report this in Red Hat Bugzilla and we'll see what the problem
>> really is (i.e. is it something in gcc or maybe even APR).
>
> https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=193740
Are you sure the "intermediate code" is correct?
It's calling free and destroy, which makes me think
that's some other "delete buckets" loop, not this one.
--
Joe Schaefer
Re: Endless loop in split_on_bdry() of library/parser_multipart.c?
Posted by Bojan Smojver <bo...@rexursive.com>.
Quoting Bojan Smojver <bo...@rexursive.com>:
> I will report this in Red Hat Bugzilla and we'll see what the problem
> really is (i.e. is it something in gcc or maybe even APR).
https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=193740
--
Bojan
Re: Endless loop in split_on_bdry() of library/parser_multipart.c?
Posted by Bojan Smojver <bo...@rexursive.com>.
Quoting "Philip M. Gollucci" <pg...@p6m7g8.com>:
> gcc version 3.4.4 [FreeBSD] 20050518
That's probably why. Optimization was reworked in 4.1 completely.
--
Bojan
Re: Endless loop in split_on_bdry() of library/parser_multipart.c?
Posted by "Philip M. Gollucci" <pg...@p6m7g8.com>.
Bojan Smojver wrote:
> Quoting "Philip M. Gollucci" <pg...@p6m7g8.com>:
>
>> my CFLAGS on FreeBSD are
>> -g -O2 -Werror -Wall -Wmissing-prototypes -Wstrict-prototypes
>> -Wmissing-declarations -Wwrite-strings -Wcast-qual -Wfloat-equal
>> -Wshadow -Wpointer-arith -Wbad-function-cast -Wsign-compare
>> -Waggregate-return -Wmissing-noreturn -Wmissing-format-attribute
>> -Wpacked -Wredundant-decls -Wnested-externs -Wdisabled-optimization
>> -Wno-long-long -Wendif-labels -Wcast-align
>
> gcc 4.1.1?
pgollucci@pgollucci.office.tmcs /home/pgollucci/tm/webcore/lib/perl/TMCS/Util 82 0>gcc -v
Using built-in specs.
Configured with: FreeBSD/i386 system compiler
Thread model: posix
gcc version 3.4.4 [FreeBSD] 20050518
I'll try some combination of the following tonight.
gcc40/Makefile:PORTREVISION= 20060525
gcc40/Makefile:VERSIONSTRING= 4.0-${PORTREVISION}
gcc41/Makefile:PORTREVISION= 20060526
gcc41/Makefile:VERSIONSTRING= 4.1-${PORTREVISION}
gcc42/Makefile:PORTREVISION= 20060527
gcc42/Makefile:VERSIONSTRING= 4.2-${PORTREVISION}
--
------------------------------------------------------------------------
Philip M. Gollucci (pgollucci@p6m7g8.com) 323.219.4708
Consultant / http://p6m7g8.net/Resume/resume.shtml
Senior Software Engineer - TicketMaster - http://ticketmaster.com
1024D/A79997FA F357 0FDD 2301 6296 690F 6A47 D55A 7172 A799 97F
"It takes a minute to have a crush on someone, an hour to like someone,
and a day to love someone, but it takes a lifetime to forget someone..."
Re: Endless loop in split_on_bdry() of library/parser_multipart.c?
Posted by Bojan Smojver <bo...@rexursive.com>.
Quoting "Philip M. Gollucci" <pg...@p6m7g8.com>:
> my CFLAGS on FreeBSD are
> -g -O2 -Werror -Wall -Wmissing-prototypes -Wstrict-prototypes
> -Wmissing-declarations -Wwrite-strings -Wcast-qual -Wfloat-equal
> -Wshadow -Wpointer-arith -Wbad-function-cast -Wsign-compare
> -Waggregate-return -Wmissing-noreturn -Wmissing-format-attribute
> -Wpacked -Wredundant-decls -Wnested-externs -Wdisabled-optimization
> -Wno-long-long -Wendif-labels -Wcast-align
gcc 4.1.1?
BTW, I'm not 100% sure that -fno-strict-aliasing is the right flag to
use. I just know that by trying them in order (as described in gcc man
page) and above -O1, this was the one that started the problems. Then
I went the other way and did -O2 -fno-strict-aliasing (i.e. removed
only that one from the lot above -O1) and the assembly was still good.
The manual page of gcc seem to suggest that "type-punning" may cause
problems like this.
I will report this in Red Hat Bugzilla and we'll see what the problem
really is (i.e. is it something in gcc or maybe even APR).
--
Bojan
Re: Endless loop in split_on_bdry() of library/parser_multipart.c?
Posted by "Philip M. Gollucci" <pg...@p6m7g8.com>.
> With -fno-strict-aliasing, I get:
my CFLAGS on FreeBSD are
-g -O2 -Werror -Wall -Wmissing-prototypes -Wstrict-prototypes -Wmissing-declarations -Wwrite-strings -Wcast-qual
-Wfloat-equal -Wshadow -Wpointer-arith -Wbad-function-cast -Wsign-compare -Waggregate-return -Wmissing-noreturn
-Wmissing-format-attribute -Wpacked -Wredundant-decls -Wnested-externs -Wdisabled-optimization -Wno-long-long
-Wendif-labels -Wcast-align
Which seem to work and come from this:
./buildconf --with-perl=/usr/local/software/perl/5.8.8/bin/perl
setenv PERL5LIB /usr/local/software/mod_perl/trunk/lib
./configure \
--prefix=/usr/local/software/apreq/trunk \
--with-perl=/usr/local/software/perl/5.8.8/bin/perl \
--with-apache2-apxs=/usr/local/software/httpd/2.2.2/prefork/bin/apxs \
--with-expat=/usr/local \
--enable-maintainer-mode \
--enable-perl-glue
gmake all
gmake test
--
------------------------------------------------------------------------
Philip M. Gollucci (pgollucci@p6m7g8.com) 323.219.4708
Consultant / http://p6m7g8.net/Resume/resume.shtml
Senior Software Engineer - TicketMaster - http://ticketmaster.com
1024D/A79997FA F357 0FDD 2301 6296 690F 6A47 D55A 7172 A799 97F
"It takes a minute to have a crush on someone, an hour to like someone,
and a day to love someone, but it takes a lifetime to forget someone..."
Re: Endless loop in split_on_bdry() of library/parser_multipart.c?
Posted by Bojan Smojver <bo...@rexursive.com>.
On Wed, 2006-05-31 at 10:49 -0700, Philip M. Gollucci wrote:
> So, does library/t/parsers fail for you without the -fno-strict-aliasing flag under -02 ?
Yes. The output is:
----------------------------
version....ok
cookie.....ok
params.....ok
parsers....ok 1/542
----------------------------
With process lt-parsers consuming all CPU and having no strace output
(i.e. it's stuck in an infinite loop). Backtrace from gdb is:
----------------------------
(gdb) bt
#0 0x0019c514 in split_on_bdry (out=0x8a569e8, in=0x8a56a08,
pattern=0x8a563e0, bdry=0x8a563d1 "\r\n--AaB03x") at
parser_multipart.c:167
#1 0x0019cb29 in apreq_parse_multipart (parser=0x8a56330, t=0x8a561b8,
bb=0x8a56350) at parser_multipart.c:580
#2 0x0804ba87 in parse_multipart (AT=0x890e138)
at ../../include/apreq_parser.h:126
#3 0x0804c3da in at_run (AT=0x890e138, test=0xbff45a30) at at.c:337
#4 0x08049426 in main () at parsers.c:546
----------------------------
With -fno-strict-aliasing, I get:
----------------------------
version....ok
cookie.....ok
params.....ok
parsers....ok
error......ok
util.......ok
All tests successful.
----------------------------
--
Bojan
Re: Endless loop in split_on_bdry() of library/parser_multipart.c?
Posted by "Philip M. Gollucci" <pg...@p6m7g8.com>.
> Now, the same code compiled with -fno-strict-aliasing (and -O2) gives:
EGAD!
I'll look later... I don't think I'll do much better then you. Maybe we should forward to
the gcc-dev list ?
Also, is -fno-strict-aliasing support on all gcc versions ?
So, does library/t/parsers fail for you without the -fno-strict-aliasing flag under -02 ?
--
------------------------------------------------------------------------
Philip M. Gollucci (pgollucci@p6m7g8.com) 323.219.4708
Consultant / http://p6m7g8.net/Resume/resume.shtml
Senior Software Engineer - TicketMaster - http://ticketmaster.com
1024D/A79997FA F357 0FDD 2301 6296 690F 6A47 D55A 7172 A799 97F
"It takes a minute to have a crush on someone, an hour to like someone,
and a day to love someone, but it takes a lifetime to forget someone..."
Re: Endless loop in split_on_bdry() of library/parser_multipart.c?
Posted by Bojan Smojver <bo...@rexursive.com>.
On Wed, 2006-05-31 at 02:18 -0700, Philip M. Gollucci wrote:
> Looking at library/parser_multipart.c and associated tests t/parsers.c ->
> we definetely exercise that code block sucessfully.
Here is the good news: I don't think this is a problem in libapreq2 (or
at least not on all platforms/compilers).
And the bad news is: it does appear that the problem has something to do
with optimisation flags on Fedora Core 5 with gcc 4.1.1 (and I think
4.1.0) on i686, as I first suspected after the code worked when compiled
with -O0.
WARNING: Please take my attempt to explain with a grain of salt - I
really don't understand compilers and x86 assembly language all that
well.
As it turns out, when these line of code (164 - 168 in
library/parse_multipart.c) are compiled with -O2:
---------------------------
do {
apr_bucket *f = APR_BRIGADE_FIRST(in);
APR_BUCKET_REMOVE(f);
APR_BRIGADE_INSERT_TAIL(out, f);
} while (e != APR_BRIGADE_FIRST(in));
---------------------------
the assembly (on an i686 machine) looks like this:
---------------------------
.LBB4:
.loc 1 165 0
movl -56(%ebp), %edx
movl $0, -36(%ebp)
movl 4(%edx), %ecx
.LVL27:
cmpl %ecx, -44(%ebp)
.p2align 4,,7
.L30:
.loc 1 166 0
movl 4(%ecx), %edx
movl (%ecx), %eax
movl %eax, (%edx)
movl (%ecx), %eax
.LBB5:
.loc 1 167 0
movl %edi, (%ecx)
.LBE5:
.loc 1 166 0
movl %edx, 4(%eax)
.LBB6:
.loc 1 167 0
movl 4(%edi), %eax
movl %eax, 4(%ecx)
movl 4(%edi), %eax
movl %ecx, 4(%edi)
movl %ecx, (%eax)
.LBE6:
.LBE4:
.loc 1 168 0
jne .L30
jmp .L19
---------------------------
.L19 is the goto label (look_for_boundary_up_front). Note how the
comparison (i.e. cmpl) is done *before* .L30, which is where we jump
from jne. In other words, the loop is endless because the condition is
never evaluated again.
Now, the same code compiled with -fno-strict-aliasing (and -O2) gives:
---------------------------
.L28:
.LBB4:
.loc 1 165 0
movl -56(%ebp), %edx
movl 4(%edx), %eax
.LVL25:
.loc 1 166 0
movl 4(%eax), %ecx
movl (%eax), %edx
movl %edx, (%ecx)
movl (%eax), %edx
.LBB5:
.loc 1 167 0
movl %esi, (%eax)
.LBE5:
.loc 1 166 0
movl %ecx, 4(%edx)
.LBB6:
.loc 1 167 0
movl 4(%esi), %edx
movl %edx, 4(%eax)
movl 4(%esi), %edx
movl %eax, 4(%esi)
movl %eax, (%edx)
.LBE6:
.LBE4:
.loc 1 168 0
movl -56(%ebp), %ecx
movl -44(%ebp), %eax
.LVL26:
cmpl 4(%ecx), %eax
jne .L28
jmp .L43
---------------------------
.L43 is the goto label (look_for_boundary_up_front). .L28, as you can
see, is the beginning of the loop. This code works because it does cmpl
on every pass.
It would sure be useful to have compiler/assembly folks take a look at
this, as I'm not entirely sure that my findings are correct (never
studied compilers nor x86 assembly, I'm afraid).
Anyhow, I can avoid the infinite loop by compiling with
-fno-strict-aliasing.
PS. Intermediate C code is here (same in both cases):
---------------------------
do {
apr_bucket *f = (&(in)->list)->next;
do { do { (((((f))))->link.prev)->link.next = ((((f))))->link.next;
(((((f))))->link.next)->link.prev = ((((f))))->link.prev; } while (0);
do { (f)->type->destroy((f)->data); (f)->free(f); } while (0); } while
(0);
} while ((&(in)->list)->next != e);
---------------------------
--
Bojan
Re: Endless loop in split_on_bdry() of library/parser_multipart.c?
Posted by "Philip M. Gollucci" <pg...@p6m7g8.com>.
> Thanks for your help. I'll see I go with GDB first. If I get nowhwere,
> I'll try the .t.
Looking at library/parser_multipart.c and associated tests t/parsers.c ->
we definetely exercise that code block sucessfully.
If you copy httpd's .gdbinit file you can make use of
dump_brigade in
dump_brigade out
dump_bucket e
dump_bucket f
from within gdb, but I'm betting you are already using them since you're
using gdb.
static apr_status_t split_on_bdry(apr_bucket_brigade *out,
apr_bucket_brigade *in,
const apr_strmatch_pattern *pattern,
const char *bdry)
{
apr_bucket *e = APR_BRIGADE_FIRST(in);
apr_size_t blen = strlen(bdry), off = 0;
------> line ~121 while ( e != APR_BRIGADE_SENTINEL(in) ) {
or better yet, at your problem source:
do {
apr_bucket *f = APR_BRIGADE_FIRST(in);
APR_BUCKET_REMOVE(f);
APR_BRIGADE_INSERT_TAIL(out, f);
-----> line ~171 } while (e != APR_BRIGADE_FIRST(in));
and send a
bt full
with the dump_* commands from above after it gets stuck.
maybe something will jump out.
I looked at your mod_spin module (1.0) in src/mod_spin.c where you call
ap_discard_request_body. Seems sane to me.
In one of the tests that hits this do {} while, I see this
(gdb) bt full
#0 split_on_bdry (out=0x81f9438, in=0x81f9458, pattern=0x81f8e38,
bdry=0x81f8e29 "\r\n--AaB03x") at parser_multipart.c:171
idx = 136287324
len = 181
buf = 0x804d7d0 "\n\r\n--AaB03x\r\ncontent-disposition:
form-data; name=\"\"\r\ncontent-type: text/plain;\r\n
charset=windows-1250\r\ncontent-transfer-encoding:
quoted-printable\r\n\r\nJoe owes =80100.\r\n--AaB03x--\r\n"
s = 136146624
e = (apr_bucket *) 0x81d6e68
blen = 10
off = 1
#1 0x2807f6eb in apreq_parse_multipart (parser=0x81f8d88, t=0x81f8c10,
bb=0x81f8da8) at parser_multipart.c:583
param = (apreq_param_t *) 0x81f9998
pool = (apr_pool_t *) 0x8063018
ba = (apr_bucket_alloc_t *) 0x8075018
ctx = (struct mfd_ctx *) 0x81f8dc8
s = 5
This part confuses me why they are error buckets, let alone why that
symbol isn't defined for me... but thats a goof on my part, that I'll
have to figure out somehow....... Interestly, I don't think any of them
are none error buckets from tests 9 -> 494. (attached -verbose output)
(gdb) dump_bucket e
bucket=IMMORTAL (0x081d6e68) length=181 data=0x0804d6a0
No symbol "ap_bucket_type_error" in current context.
(gdb) dump_brigade in
dump of brigade 0x81f9458
| type (address) | length | data addr | contents
| rc
--------------------------------------------------------------------------------
0 | IMMORTAL (0x081d6e68) | 181 | 0x0804d6a0No symbol
"ap_bucket_type_error" in current context.
(gdb) dump_brigade out
dump of brigade 0x81f9438
| type (address) | length | data addr | contents
| rc
--------------------------------------------------------------------------------
0 | IMMORTAL (0x081d6ec0) | 29 | 0x0804d6a0No symbol
"ap_bucket_type_error" in current context.
Time for sleep.
--
------------------------------------------------------------------------
Philip M. Gollucci (pgollucci@p6m7g8.com) 323.219.4708
Consultant / http://p6m7g8.net/Resume/resume.shtml
Senior Software Engineer - TicketMaster - http://ticketmaster.com
1024D/A79997FA F357 0FDD 2301 6296 690F 6A47 D55A 7172 A799 97F
"It takes a minute to have a crush on someone, an hour to like someone,
and a day to love someone, but it takes a lifetime to forget someone..."
Re: Endless loop in split_on_bdry() of library/parser_multipart.c?
Posted by Bojan Smojver <bo...@rexursive.com>.
Quoting "Philip M. Gollucci" <pg...@p6m7g8.com>:
>> I'm guessing this is a Perl thing. I'm in C, so this probably isn't
>> going to be very useful. I'm watching the whole thing in gdb anyway.
> If you can write a *.t test for it, I can try and see if I A)
> duplicate, and B) help
> you some this evening.
Thanks for your help. I'll see I go with GDB first. If I get nowhwere,
I'll try the .t.
--
Bojan
Re: Endless loop in split_on_bdry() of library/parser_multipart.c?
Posted by "Philip M. Gollucci" <pg...@p6m7g8.com>.
> I'm guessing this is a Perl thing. I'm in C, so this probably isn't
> going to be very useful. I'm watching the whole thing in gdb anyway.
If you can write a *.t test for it, I can try and see if I A) duplicate, and B) help
you some this evening.
--
Philip M. Gollucci (pgollucci@p6m7g8.com) 323.219.4708
Consultant / http://p6m7g8.net/Resume/resume.shtml
Senior Software Engineer - TicketMaster - http://ticketmaster.com
1024D/A79997FA F357 0FDD 2301 6296 690F 6A47 D55A 7172 A799 97F
"It takes a minute to have a crush on someone, an hour to like someone,
and a day to love someone, but it takes a lifetime to forget someone..."
Re: Endless loop in split_on_bdry() of library/parser_multipart.c?
Posted by Bojan Smojver <bo...@rexursive.com>.
Quoting "Philip M. Gollucci" <pg...@p6m7g8.com>:
> You might try using Apache2::DebugFilter to see what buckets are
> getting passed, but I think
> the apreq filter is hooked before that.
I'm guessing this is a Perl thing. I'm in C, so this probably isn't
going to be very useful. I'm watching the whole thing in gdb anyway.
> Being that bucket brigades are doubly linked circular lists, and the
> code is checking it hasn't looped on the list (aka ring) yet, it must
> be missing it. Probably because it got removed from it (in). Maybe we
> need to save the first bucket of the list somewhere. and change the
> comparison.
This code is hit when there is a partial match. In this particular
case, the boundary is set to "\r\n--LYNX". The partial match is "\r"
and off is set to 1. However, the ending condition seems to be never
met in that while loop.
To make matters worse, it's not as straightforward as one would think.
The whole thing seems to depend on how the buckets are lined up inside
the brigade. And, compiler options appear to affect the whole thing as
well. With -O2, I can see the problem. With -O0, I can't.
I'll keep looking...
--
Bojan
Re: Endless loop in split_on_bdry() of library/parser_multipart.c?
Posted by "Philip M. Gollucci" <pg...@p6m7g8.com>.
Bojan Smojver wrote:
> I'm seeing an endless loop in this part of the code:
>
> ------------------------------------------------
> do {
> apr_bucket *f = APR_BRIGADE_FIRST(in);
> APR_BUCKET_REMOVE(f);
> APR_BRIGADE_INSERT_TAIL(out, f);
> } while (e != APR_BRIGADE_FIRST(in));
> ------------------------------------------------
You might try using Apache2::DebugFilter to see what buckets are getting passed, but I think
the apreq filter is hooked before that.
Being that bucket brigades are doubly linked circular lists, and the code is checking it hasn't looped on the list (aka
ring) yet, it must be missing it. Probably because it got removed from it (in). Maybe we need to save the first bucket
of the list somewhere. and change the comparison.
Sadly I'm not that familiar with this portion of the code yet.
--
------------------------------------------------------------------------
Philip M. Gollucci (pgollucci@p6m7g8.com) 323.219.4708
Consultant / http://p6m7g8.net/Resume/resume.shtml
Senior Software Engineer - TicketMaster - http://ticketmaster.com
1024D/A79997FA F357 0FDD 2301 6296 690F 6A47 D55A 7172 A799 97F
"It takes a minute to have a crush on someone, an hour to like someone,
and a day to love someone, but it takes a lifetime to forget someone..."