You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by Lieven Govaerts <lg...@apache.org> on 2012/11/08 13:01:24 UTC
ra_serf: truncated responses by low Timeout are not handled graceful
(was Re: Random serf checkout failures)
Philip,
On Tue, Nov 6, 2012 at 11:20 PM, Philip Martin
<ph...@wandisco.com> wrote:
> Lieven Govaerts <lg...@mobsol.be> writes:
>
>>> Philip or Ben, can you test that with this patch svn will always stop
>>> with a communication error?
>
> Using serf 1.1.x patched and Subversion 1406366 I get this new error:
>
> A wc/1.3.0-rc1/subversion/libsvn_subr/config.c
> ../src/subversion/svn/checkout-cmd.c:168: (apr_err=120105)
> ../src/subversion/libsvn_client/checkout.c:179: (apr_err=120105)
> ../src/subversion/libsvn_client/update.c:579: (apr_err=120105)
> ../src/subversion/libsvn_client/update.c:440: (apr_err=120105)
> ../src/subversion/libsvn_wc/adm_crawler.c:858: (apr_err=120105)
> ../src/subversion/libsvn_ra_serf/update.c:2564: (apr_err=120105)
> ../src/subversion/libsvn_ra_serf/util.c:2028: (apr_err=120105)
> ../src/subversion/libsvn_ra_serf/util.c:2009: (apr_err=120105)
> ../src/subversion/libsvn_ra_serf/update.c:989: (apr_err=120105)
> svn: E120105: ra_serf: The server sent an improper HTTP response
>
> but I still see this as well:
>
> A wc/1.3.0-rc1/subversion/bindings/java/javahl/src/org/tigris/subversion/javahl/SVNAdmin.java
> ../src/subversion/libsvn_ra_serf/update.c:680: (apr_err=235000)
> svn: E235000: In file '../src/subversion/libsvn_ra_serf/update.c' line 680: assertion failed (! dir->ref_count)
>
> Program received signal SIGABRT, Aborted.
> 0x00007ffff65ea475 in *__GI_raise (sig=<optimized out>)
> at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
> 64 ../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory.
> (gdb) p dir->ref_count
> $1 = 1
> (gdb) bt
> #0 0x00007ffff65ea475 in *__GI_raise (sig=<optimized out>)
> at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
> #1 0x00007ffff65ed6f0 in *__GI_abort () at abort.c:92
> #2 0x00007ffff6feb5fd in svn_error_abort_on_malfunction (can_return=1,
> file=0x7ffff5b2ff78 "../src/subversion/libsvn_ra_serf/update.c", line=680,
> expr=0x7ffff5b3001e "! dir->ref_count")
> at ../src/subversion/libsvn_subr/error.c:678
> #3 0x00007ffff6feb64e in svn_error__malfunction (can_return=1,
> file=0x7ffff5b2ff78 "../src/subversion/libsvn_ra_serf/update.c", line=680,
> expr=0x7ffff5b3001e "! dir->ref_count")
> at ../src/subversion/libsvn_subr/error.c:703
> #4 0x00007ffff5b1edc0 in close_all_dirs (dir=0x7ffff1e810c8)
> at ../src/subversion/libsvn_ra_serf/update.c:680
> #5 0x00007ffff5b1ed4a in close_all_dirs (dir=0x7ffff1e830c8)
> at ../src/subversion/libsvn_ra_serf/update.c:676
> #6 0x00007ffff5b1ed4a in close_all_dirs (dir=0x7ffff1e850c8)
> at ../src/subversion/libsvn_ra_serf/update.c:676
> #7 0x00007ffff5b1ed4a in close_all_dirs (dir=0x7ffff1e870c8)
> at ../src/subversion/libsvn_ra_serf/update.c:676
> #8 0x00007ffff5b1ed4a in close_all_dirs (dir=0x7ffff1e8d0c8)
> at ../src/subversion/libsvn_ra_serf/update.c:676
> #9 0x00007ffff5b1ed4a in close_all_dirs (dir=0x7ffff20090c8)
> at ../src/subversion/libsvn_ra_serf/update.c:676
> #10 0x00007ffff5b1ed4a in close_all_dirs (dir=0x7ffff200b0c8)
> at ../src/subversion/libsvn_ra_serf/update.c:676
> #11 0x00007ffff5b1ed4a in close_all_dirs (dir=0x7ffff20130c8)
> at ../src/subversion/libsvn_ra_serf/update.c:676
> #12 0x00007ffff5b1ed4a in close_all_dirs (dir=0x7ffff24b90c8)
> at ../src/subversion/libsvn_ra_serf/update.c:676
> #13 0x00007ffff5b1ed4a in close_all_dirs (dir=0x7ffff28f60c8)
> at ../src/subversion/libsvn_ra_serf/update.c:676
> #14 0x00007ffff5b1ed4a in close_all_dirs (dir=0x7ffff32a10c8)
> at ../src/subversion/libsvn_ra_serf/update.c:676
> #15 0x00007ffff5b2420f in finish_report (report_baton=0x7ffff7e45090,
> pool=0x7ffff7e42028) at ../src/subversion/libsvn_ra_serf/update.c:2746
> #16 0x00007ffff789bc6d in svn_wc_crawl_revisions5 (wc_ctx=0x7ffff32fd6b0,
> local_abspath=0x7ffff7e42168 "/home/pm/sw/subversion/obj/wc",
> reporter=0x7ffff5d394e0, report_baton=0x7ffff7e45090, restore_files=1,
> depth=svn_depth_unknown, honor_depth_exclude=1,
> depth_compatibility_trick=0, use_commit_times=0,
> cancel_func=0x416b43 <svn_cl__check_cancel>, cancel_baton=0x0,
> notify_func=0x41c467 <notify>, notify_baton=0x7ffff32fd9a0,
> scratch_pool=0x7ffff7e42028)
> at ../src/subversion/libsvn_wc/adm_crawler.c:858
> #17 0x00007ffff7bc7f6d in update_internal (result_rev=0x0,
> local_abspath=0x7ffff7e42168 "/home/pm/sw/subversion/obj/wc",
> anchor_abspath=0x7ffff7e438a0 "/home/pm/sw/subversion/obj/wc",
> revision=0x7fffffffe050, depth=svn_depth_unknown, depth_is_sticky=0,
> ignore_externals=0, allow_unver_obstructions=0, adds_as_modification=1,
> timestamp_sleep=0x7fffffffe15c, notify_summary=1, ctx=0x7ffff32fd5f8,
> pool=0x7ffff7e42028) at ../src/subversion/libsvn_client/update.c:427
>
I can't reproduce this myself. Since this happens in the middle of the
checkout (not explicitly mentioned in your mail but Ben showed me),
and the close_all_dirs function is only called at the very end, when
the full report has been parsed, this can be explained by a truncated
REPORT response.
Any truncated response should now result in an error from serf (since
serf trunk r1678). It's not clear to me if this error
SERF_ERROR_TRUNCATED_HTTP_RESPONSE was returned from serf but then
ignored in svn, or not returned at all. I'll look into that.
The proposed solution here is to report a decent error code and
message to the user when this sudden abort of the REPORT response
connection happens.
Attached patch does that, by checking for the error situation first,
before trying to close all directories. We then rely on pool cleanup
to handle mandatory closing actions.
Lieven
Re: ra_serf: truncated responses by low Timeout are not handled graceful
Posted by Philip Martin <ph...@wandisco.com>.
Lieven Govaerts <lg...@apache.org> writes:
> My expectation is that with serf trunk up to date r1685 you'll now
> only see the error "The server sent a truncated HTTP response body."
> instead of asserts, checksum or other svn errors.
I do see that error:
../src/subversion/svn/checkout-cmd.c:168: (apr_err=120106)
../src/subversion/libsvn_client/checkout.c:179: (apr_err=120106)
../src/subversion/libsvn_client/update.c:579: (apr_err=120106)
../src/subversion/libsvn_client/update.c:440: (apr_err=120106)
../src/subversion/libsvn_wc/adm_crawler.c:858: (apr_err=120106)
../src/subversion/libsvn_ra_serf/update.c:2564: (apr_err=120106)
../src/subversion/libsvn_ra_serf/util.c:2028: (apr_err=120106)
../src/subversion/libsvn_ra_serf/util.c:2009: (apr_err=120106)
../src/subversion/libsvn_ra_serf/util.c:1574: (apr_err=120106)
svn: E120106: ra_serf: The server sent a truncated HTTP response body.
I also see:
../src/subversion/svn/checkout-cmd.c:168: (apr_err=175009)
../src/subversion/libsvn_client/checkout.c:179: (apr_err=175009)
../src/subversion/libsvn_client/update.c:579: (apr_err=175009)
../src/subversion/libsvn_client/update.c:440: (apr_err=175009)
../src/subversion/libsvn_wc/adm_crawler.c:858: (apr_err=175009)
../src/subversion/libsvn_ra_serf/update.c:2564: (apr_err=175009)
../src/subversion/libsvn_ra_serf/util.c:2028: (apr_err=175009)
../src/subversion/libsvn_ra_serf/util.c:1850: (apr_err=175009)
svn: E175009: Premature EOF seen from server (http status=200)
../src/subversion/libsvn_ra_serf/util.c:1850: (apr_err=70014)
svn: E070014: End of file found
and with mod_deflate enabled I also see:
../src/subversion/svn/checkout-cmd.c:168: (apr_err=120104)
../src/subversion/libsvn_client/checkout.c:179: (apr_err=120104)
../src/subversion/libsvn_client/update.c:579: (apr_err=120104)
../src/subversion/libsvn_client/update.c:440: (apr_err=120104)
../src/subversion/libsvn_wc/adm_crawler.c:858: (apr_err=120104)
../src/subversion/libsvn_ra_serf/update.c:2564: (apr_err=120104)
../src/subversion/libsvn_ra_serf/util.c:2028: (apr_err=120104)
../src/subversion/libsvn_ra_serf/util.c:2009: (apr_err=120104)
../src/subversion/libsvn_ra_serf/update.c:989: (apr_err=120104)
svn: E120104: ra_serf: An error occurred during decompression
which is probably that same EOF error during inflation.
Using serf trunk@1685, subversion trunk@1407208.
--
Certified & Supported Apache Subversion Downloads:
http://www.wandisco.com/subversion/download
Re: ra_serf: truncated responses by low Timeout are not handled graceful
Posted by Lieven Govaerts <lg...@apache.org>.
Hi,
On Thu, Nov 8, 2012 at 1:34 PM, Philip Martin
<ph...@wandisco.com> wrote:
> Lieven Govaerts <lg...@apache.org> writes:
>
>> Any truncated response should now result in an error from serf (since
>> serf trunk r1678). It's not clear to me if this error
>> SERF_ERROR_TRUNCATED_HTTP_RESPONSE was returned from serf but then
>> ignored in svn, or not returned at all.
>
> I was using 1.1.x with the patches you and Ivan posted so it would not
> have been returning that error.
The patch I sent you earlier used another error code, but the result
should be the same. You saw that in your logs as error "The server
sent an improper HTTP response".
That patch only checked bodies for responses with Content-Length
header. The REPORT response is chunked encoded, hence no error was
reported by serf when the server aborted the REPORT response
connection.
I've added a similar check for chunked-encoded response bodies in serf r1685.
My expectation is that with serf trunk up to date r1685 you'll now
only see the error "The server sent a truncated HTTP response body."
instead of asserts, checksum or other svn errors.
Lieven
Re: ra_serf: truncated responses by low Timeout are not handled graceful
Posted by Philip Martin <ph...@wandisco.com>.
Lieven Govaerts <lg...@apache.org> writes:
> Any truncated response should now result in an error from serf (since
> serf trunk r1678). It's not clear to me if this error
> SERF_ERROR_TRUNCATED_HTTP_RESPONSE was returned from serf but then
> ignored in svn, or not returned at all.
I was using 1.1.x with the patches you and Ivan posted so it would not
have been returning that error.
--
Certified & Supported Apache Subversion Downloads:
http://www.wandisco.com/subversion/download