You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by Johan Corveleyn <jc...@gmail.com> on 2012/02/09 02:33:25 UTC

Failing svnrdump_tests.py#43 with 1.7.x on Windows (was: Re: 1.7.3 next week-ish?)

Having another look at this failure ...

So far, we know (anyone, correct me if I'm wrong):

- 'svnrdump load' in this test fails with: "svnrdump: E140001:
Unrecognized record type in stream". It seems the dumpfile contents of
the file D/H/psi is split incorrectly (property content is dumped
early, and text delta later).

- It only happens on Windows.

- It only happens with ra_serf (both 1.7.x. and trunk) talking to
mod_dav_svn from 1.7.x@>=1239697 (i.e. after backport of r1237720 and
r1239596 (stuff in mod_dav_svn/liveprops.c) from trunk).

- If 1.7.x is rolled back to before r1239697, the problem does not occur.

- The problem does not occur with mod_dav_svn from trunk, not even
when rolled back to (before or after) r1237720 or r1239596.

- So far, it has been reproduced by the buildbot svn-slik-w2k3-x64-ra,
by Stephen Butler, and by my own (32bit) WinXP box. Paul Burba doesn't
see the failure with his Windows build (debug and release).


I'm trying some more experiments. If anyone else has any ideas, shoot ...

--
Johan

Re: Failing svnrdump_tests.py#43 with 1.7.x on Windows (was: Re: 1.7.3 next week-ish?)

Posted by Johan Corveleyn <jc...@gmail.com>.
On Thu, Feb 9, 2012 at 7:23 PM, Paul Burba <pt...@gmail.com> wrote:

[ ... ]

> However, if I let the test suite start up httpd, then I *do*
> consistently see the same failure:

Wow, I wouldn't have thought of that as a discriminating factor. I'm
running the server in a separate command window, just running httpd.
So that seems to fit.

Nice to know that I'm no longer (almost) the only one seeing this :-).

[ ... ]

>> There was IRC discussion about this:
>> http://colabti.org/irclogger/irclogger_log/svn-dev?date=2012-02-09 (~4pm)
>>
>> It is suggested there that setting libsvn_ra_serf/update.c:MAX_NR_OF_CONNS
>> to "2" will prevent ra_serf from driving multiple window handlers
>> concurrently and thus avoid the bug.  (Presumably by causing the file to
>> finish on the right side of the errant close_directory() call.)
>
> I just tried MAX_NR_OF_CONNS=2 with
> ^/subversion/branches/1.7.x@1242422 and the test still fails:

Interesting ...

-- 
Johan

Re: Failing svnrdump_tests.py#43 with 1.7.x on Windows

Posted by Philip Martin <ph...@wandisco.com>.
Johan Corveleyn <jc...@gmail.com> writes:

> I'm finally starting to get somewhere:
>
> (all the below is with a trunk client with MAX_NR_OF_CONNS=1, vs. the
> affected mod_dav_svn)
>
> When processing 'psi', update_editor.c#handle_fetch exits early. It
> skips the last block of 'if (APR_STATUS_IS_EOF(status))' (containing
> the 'close_file' call), because status==730035. That seems to be
> WSAEWOULDBLOCK (WinError.h): "A non-blocking socket operation could
> not be completed immediately."
>
> After that, for some reason, the close_directory is driven first, and
> only later handle_fetch is run again to finish off psi.

Ah!  So it's using fetch_ctx->read_headers to allow multiple calls to
handle_fetch.  I understand that now.

> That might explain the Windows-ness a bit (specific socket behavior),
> combined with some randomness (sometimes it does "complete
> immediately", sometimes it doesn't). It still doesn't explain the
> relatedness to 1.7.x.
>
> Open questions:
>
> - What's the link with Philip's change in liveprops.c ('if (kind !=
> svn_node_file) return DAV_PROP_INSERT_NOTSUPP;') ?

None really.  That change causes the server to avoid sending a
checksum="###error###" line for direcories, but all that matters in this
context is that changing a server response affects the timing of all the
server responses.

svnrdump/serf falls over at random depending on the order in which it
receives responses from the the (multi-threaded/multi-process) server.
It can fall over using a 1.7.2 client against a 1.6 server.  By changing
the timing we cause svnrdump/serf to fall over at random in a different
place.

> - Why the WSAEWOULDBLOCK error? Maybe that's expected and normal, and
> shouldn't cause a problem in and of itself?

Yes. The response to a GET request could be large.  So the client may
have to loop to read it all.  That loop happens outside handle_fetch.

> - Why, after exiting handle_fetch early, does ra_serf first do the
> close_directory first? I don't know much about how ra_serf organizes
> these things.

That's where the server timing comes in.  After the first call to
handle_fetch for psi if the next event is the rest of the GET then we go
back into handle_fetch and things are OK.  But sometimes some other
event happens first and we do close_directory.

> (actually, the WSAEWOULDBLOCK error also happens earlier during the
> update-drive, when adding trunk/D/G/rho, but it doesn't cause a
> problem -- no close_directory interference, just a clean "rerun" of
> handle_fetch).

-- 
uberSVN: Apache Subversion Made Easy
http://www.uberSVN.com

Re: Failing svnrdump_tests.py#43 with 1.7.x on Windows (was: Re: 1.7.3 next week-ish?)

Posted by Johan Corveleyn <jc...@gmail.com>.
On Thu, Feb 9, 2012 at 9:55 PM, Greg Stein <gs...@gmail.com> wrote:
>
> On Feb 9, 2012 3:07 PM, "Paul Burba" <pt...@gmail.com> wrote:
>>
>> On Thu, Feb 9, 2012 at 2:50 PM, Johan Corveleyn <jc...@gmail.com> wrote:
>> > On Thu, Feb 9, 2012 at 7:59 PM, Greg Stein <gs...@gmail.com> wrote:
>> >>
>> >> On Feb 9, 2012 1:23 PM, "Paul Burba" <pt...@gmail.com> wrote:
>> >>>
>> >>> On Thu, Feb 9, 2012 at 11:49 AM, Daniel Shahaf <da...@elego.de>
>> >>> wrote:
>> >>> >...
>> >>
>> >>> > It is suggested there that setting
>> >>> > libsvn_ra_serf/update.c:MAX_NR_OF_CONNS
>> >>> > to "2" will prevent ra_serf from driving multiple window handlers
>> >>> > concurrently and thus avoid the bug.  (Presumably by causing the
>> >>> > file to
>> >>> > finish on the right side of the errant close_directory() call.)
>> >>>
>> >>> I just tried MAX_NR_OF_CONNS=2 with
>> >>> ^/subversion/branches/1.7.x@1242422 and the test still fails:
>> >>
>> >> Woah. Weird. I may have been mistaken. Please try 1.
>> >>
>> >> If that fails, then more investigation is needed.
>> >
>> > I just tried with MAX_NR_OF_CONNS=1 and it still fails.
>>
>> Same for me.
>
> Thanks, guys. It seems there is something wrong, probably independent of
> ordering issues.

I'm finally starting to get somewhere:

(all the below is with a trunk client with MAX_NR_OF_CONNS=1, vs. the
affected mod_dav_svn)

When processing 'psi', update_editor.c#handle_fetch exits early. It
skips the last block of 'if (APR_STATUS_IS_EOF(status))' (containing
the 'close_file' call), because status==730035. That seems to be
WSAEWOULDBLOCK (WinError.h): "A non-blocking socket operation could
not be completed immediately."

After that, for some reason, the close_directory is driven first, and
only later handle_fetch is run again to finish off psi.

That might explain the Windows-ness a bit (specific socket behavior),
combined with some randomness (sometimes it does "complete
immediately", sometimes it doesn't). It still doesn't explain the
relatedness to 1.7.x.

Open questions:

- What's the link with Philip's change in liveprops.c ('if (kind !=
svn_node_file) return DAV_PROP_INSERT_NOTSUPP;') ?

- Why the WSAEWOULDBLOCK error? Maybe that's expected and normal, and
shouldn't cause a problem in and of itself?

- Why, after exiting handle_fetch early, does ra_serf first do the
close_directory first? I don't know much about how ra_serf organizes
these things.
(actually, the WSAEWOULDBLOCK error also happens earlier during the
update-drive, when adding trunk/D/G/rho, but it doesn't cause a
problem -- no close_directory interference, just a clean "rerun" of
handle_fetch).

-- 
Johan

Re: Failing svnrdump_tests.py#43 with 1.7.x on Windows (was: Re: 1.7.3 next week-ish?)

Posted by Greg Stein <gs...@gmail.com>.
On Feb 9, 2012 3:07 PM, "Paul Burba" <pt...@gmail.com> wrote:
>
> On Thu, Feb 9, 2012 at 2:50 PM, Johan Corveleyn <jc...@gmail.com> wrote:
> > On Thu, Feb 9, 2012 at 7:59 PM, Greg Stein <gs...@gmail.com> wrote:
> >>
> >> On Feb 9, 2012 1:23 PM, "Paul Burba" <pt...@gmail.com> wrote:
> >>>
> >>> On Thu, Feb 9, 2012 at 11:49 AM, Daniel Shahaf <da...@elego.de>
wrote:
> >>> >...
> >>
> >>> > It is suggested there that setting
> >>> > libsvn_ra_serf/update.c:MAX_NR_OF_CONNS
> >>> > to "2" will prevent ra_serf from driving multiple window handlers
> >>> > concurrently and thus avoid the bug.  (Presumably by causing the
file to
> >>> > finish on the right side of the errant close_directory() call.)
> >>>
> >>> I just tried MAX_NR_OF_CONNS=2 with
> >>> ^/subversion/branches/1.7.x@1242422 and the test still fails:
> >>
> >> Woah. Weird. I may have been mistaken. Please try 1.
> >>
> >> If that fails, then more investigation is needed.
> >
> > I just tried with MAX_NR_OF_CONNS=1 and it still fails.
>
> Same for me.

Thanks, guys. It seems there is something wrong, probably independent of
ordering issues.

Sigh.
-g

Re: Failing svnrdump_tests.py#43 with 1.7.x on Windows (was: Re: 1.7.3 next week-ish?)

Posted by Paul Burba <pt...@gmail.com>.
On Thu, Feb 9, 2012 at 2:50 PM, Johan Corveleyn <jc...@gmail.com> wrote:
> On Thu, Feb 9, 2012 at 7:59 PM, Greg Stein <gs...@gmail.com> wrote:
>>
>> On Feb 9, 2012 1:23 PM, "Paul Burba" <pt...@gmail.com> wrote:
>>>
>>> On Thu, Feb 9, 2012 at 11:49 AM, Daniel Shahaf <da...@elego.de> wrote:
>>> >...
>>
>>> > It is suggested there that setting
>>> > libsvn_ra_serf/update.c:MAX_NR_OF_CONNS
>>> > to "2" will prevent ra_serf from driving multiple window handlers
>>> > concurrently and thus avoid the bug.  (Presumably by causing the file to
>>> > finish on the right side of the errant close_directory() call.)
>>>
>>> I just tried MAX_NR_OF_CONNS=2 with
>>> ^/subversion/branches/1.7.x@1242422 and the test still fails:
>>
>> Woah. Weird. I may have been mistaken. Please try 1.
>>
>> If that fails, then more investigation is needed.
>
> I just tried with MAX_NR_OF_CONNS=1 and it still fails.

Same for me.

Paul

Re: Failing svnrdump_tests.py#43 with 1.7.x on Windows (was: Re: 1.7.3 next week-ish?)

Posted by Johan Corveleyn <jc...@gmail.com>.
On Thu, Feb 9, 2012 at 7:59 PM, Greg Stein <gs...@gmail.com> wrote:
>
> On Feb 9, 2012 1:23 PM, "Paul Burba" <pt...@gmail.com> wrote:
>>
>> On Thu, Feb 9, 2012 at 11:49 AM, Daniel Shahaf <da...@elego.de> wrote:
>> >...
>
>> > It is suggested there that setting
>> > libsvn_ra_serf/update.c:MAX_NR_OF_CONNS
>> > to "2" will prevent ra_serf from driving multiple window handlers
>> > concurrently and thus avoid the bug.  (Presumably by causing the file to
>> > finish on the right side of the errant close_directory() call.)
>>
>> I just tried MAX_NR_OF_CONNS=2 with
>> ^/subversion/branches/1.7.x@1242422 and the test still fails:
>
> Woah. Weird. I may have been mistaken. Please try 1.
>
> If that fails, then more investigation is needed.

I just tried with MAX_NR_OF_CONNS=1 and it still fails.

This is the client-side where I need to adjust this setting, right?
The last failure I've seen was with a trunk@1242116+MAX_NR_OF_CONNS=1
client vs. 1.7.x@1239697 mod_dav_svn (r1239697 is the backport that
started breaking things).

-- 
Johan

Re: Failing svnrdump_tests.py#43 with 1.7.x on Windows (was: Re: 1.7.3 next week-ish?)

Posted by Greg Stein <gs...@gmail.com>.
On Feb 9, 2012 1:23 PM, "Paul Burba" <pt...@gmail.com> wrote:
>
> On Thu, Feb 9, 2012 at 11:49 AM, Daniel Shahaf <da...@elego.de> wrote:
> >...
> > It is suggested there that setting
libsvn_ra_serf/update.c:MAX_NR_OF_CONNS
> > to "2" will prevent ra_serf from driving multiple window handlers
> > concurrently and thus avoid the bug.  (Presumably by causing the file to
> > finish on the right side of the errant close_directory() call.)
>
> I just tried MAX_NR_OF_CONNS=2 with
> ^/subversion/branches/1.7.x@1242422 and the test still fails:

Woah. Weird. I may have been mistaken. Please try 1.

If that fails, then more investigation is needed.

Regardless, let's not let this hold up 1.7.3. As Mark pointed out, this
existed in 1.7.0.

Cheers,
-g

Re: Failing svnrdump_tests.py#43 with 1.7.x on Windows (was: Re: 1.7.3 next week-ish?)

Posted by Paul Burba <pt...@gmail.com>.
On Thu, Feb 9, 2012 at 11:49 AM, Daniel Shahaf <da...@elego.de> wrote:
> Johan Corveleyn wrote on Thu, Feb 09, 2012 at 02:33:25 +0100:
>> Having another look at this failure ...
>>
>> So far, we know (anyone, correct me if I'm wrong):
>>
>> - 'svnrdump load' in this test fails with: "svnrdump: E140001:
>> Unrecognized record type in stream". It seems the dumpfile contents of
>> the file D/H/psi is split incorrectly (property content is dumped
>> early, and text delta later).
>>
>> - It only happens on Windows.

I tried running svnrdump_tests.py 43 again and it passed as before
when running the test against an Apache server already running as a
service):

C:\SVN\src-branch-1.7.x>win-tests.py -d -c --test=svnrdump#43 --url
http://localhost --http-library serf
Testing Debug configuration on remote repository http://localhost.
Running tests in svnrdump_tests.py [1/1]......................success
Summary of test results:
  1 test PASSED
WARNING: no failures, but
'C:\SVN\src-branch-1.7.x\Debug\dav-fails.log' exists from a previous
run.

However, if I let the test suite start up httpd, then I *do*
consistently see the same failure:

C:\SVN\src-branch-1.7.x>win-tests.py -d --http-library serf
--httpd-dir C:\Apache22 --httpd-port 7829 -u http://127.0.0.1:7829 -c
--test=svnrdump#43
Adding password for user jrandom
Adding password for user jconstant
Starting httpd as daemon
['httpd.exe', '-f',
'C:\\SVN\\src-branch-1.7.x\\Debug\\subversion\\tests\\cmdline\\httpd\\httpd.conf']
Testing Debug configuration on remote repository http://127.0.0.1:7829.
Running tests in svnrdump_tests.py [1/1]......................FAILURE
At least one test FAILED, checking C:\SVN\src-branch-1.7.x\Debug\dav-tests.log
FAIL:  svnrdump_tests.py 43: don't drop mergeinfo revs in incremental svnrdump
Summary of test results:
  1 test FAILED
Stopping httpd.exe

>> - It only happens with ra_serf (both 1.7.x. and trunk) talking to
>> mod_dav_svn from 1.7.x@>=1239697 (i.e. after backport of r1237720 and
>> r1239596 (stuff in mod_dav_svn/liveprops.c) from trunk).

I confirmed what Johan observed: svnrdump test #43 passes with
^/subversion/branches/1.7.x@1239696 and fails with
^/subversion/branches/1.7.x@1239697.  Starting with r1239696 I
remerged 1237720 and the test passes.  Only when I merged r1239596 did
it fail.

> There was IRC discussion about this:
> http://colabti.org/irclogger/irclogger_log/svn-dev?date=2012-02-09 (~4pm)
>
> It is suggested there that setting libsvn_ra_serf/update.c:MAX_NR_OF_CONNS
> to "2" will prevent ra_serf from driving multiple window handlers
> concurrently and thus avoid the bug.  (Presumably by causing the file to
> finish on the right side of the errant close_directory() call.)

I just tried MAX_NR_OF_CONNS=2 with
^/subversion/branches/1.7.x@1242422 and the test still fails:

C:\SVN\src-branch-1.7.x>win-tests.py -d --http-library serf
--httpd-dir C:\Apache22 --httpd-daemon --httpd-port 7829 -u
http://127.0.0.1:7829 -c --test=svnrdump
#43 --log-to-stdout
Adding password for user jrandom
Adding password for user jconstant
Starting httpd as daemon
['httpd.exe', '-f',
'C:\\SVN\\src-branch-1.7.x\\Debug\\subversion\\tests\\cmdline\\httpd\\httpd.conf']
Testing Debug configuration on remote repository http://127.0.0.1:7829.
START: svnrdump_tests.py
.
<SNIP>
.
CMD: C:\SVN\src-branch-1.7.x\Debug\subversion\svnrdump\svnrdump.exe
load http://127.0.0.1:7829/svn-test-work/repositories/svnrdump_tests-43
--config-dir C:\SVN\
src-branch-1.7.x\Debug\subversion\tests\cmdline\svn-test-work\local_tmp\config
--password rayjandom --no-auth-cache --username jrandom exited with 1
<TIME = 0.246000>
..\..\..\subversion\svnrdump\svnrdump.c:460: (apr_err=140001)
..\..\..\subversion\libsvn_repos\load.c:542: (apr_err=140001)
svnrdump: E140001: Unrecognized record type in stream
Unexpected output
EXPECTED STDERR:
ACTUAL STDERR:
..\..\..\subversion\svnrdump\svnrdump.c:460: (apr_err=140001)
..\..\..\subversion\libsvn_repos\load.c:542: (apr_err=140001)
svnrdump: E140001: Unrecognized record type in stream
DIFF STDERR:
--- EXPECTED STDERR
+++ ACTUAL STDERR
@@ -1,0 +1,3 @@
+..\..\..\subversion\svnrdump\svnrdump.c:460: (apr_err=140001)
+..\..\..\subversion\libsvn_repos\load.c:542: (apr_err=140001)
+svnrdump: E140001: Unrecognized record type in stream
CWD: C:\SVN\src-branch-1.7.x\Debug\subversion\tests\cmdline
EXCEPTION: SVNLineUnequal
Traceback (most recent call last):
  File "C:\SVN\src-branch-1.7.x\subversion\tests\cmdline\svntest\main.py",
line 1316, in run
    rc = self.pred.run(sandbox)
  File "C:\SVN\src-branch-1.7.x\subversion\tests\cmdline\svntest\testcase.py",
line 176, in run
    return self.func(sandbox)
  File "C:\SVN\src-branch-1.7.x\subversion\tests\cmdline\svnrdump_tests.py",
line 545, in dont_drop_valid_mergeinfo_during_incremental_svnrdump_loads
    [], 0, 'load', sbox.repo_url)
  File "C:\SVN\src-branch-1.7.x\subversion\tests\cmdline\svntest\actions.py",
line 358, in run_and_verify_svnrdump
    expected_stdout, expected_stderr)
  File "C:\SVN\src-branch-1.7.x\subversion\tests\cmdline\svntest\verify.py",
line 388, in verify_outputs
    compare_and_display_lines(message, label, expected, actual, raisable)
  File "C:\SVN\src-branch-1.7.x\subversion\tests\cmdline\svntest\verify.py",
line 361, in compare_and_display_lines
    raise raisable
SVNLineUnequal
FAIL:  svnrdump_tests.py 43: don't drop mergeinfo revs in incremental svnrdump
END: svnrdump_tests.py
ELAPSED: svnrdump_tests.py 0:00:07.300000

Stopping httpd.exe

C:\SVN\src-branch-1.7.x>

Paul
>> - If 1.7.x is rolled back to before r1239697, the problem does not occur.
>>
>> - The problem does not occur with mod_dav_svn from trunk, not even
>> when rolled back to (before or after) r1237720 or r1239596.
>>
>> - So far, it has been reproduced by the buildbot svn-slik-w2k3-x64-ra,
>> by Stephen Butler, and by my own (32bit) WinXP box. Paul Burba doesn't
>> see the failure with his Windows build (debug and release).
>>
>>
>> I'm trying some more experiments. If anyone else has any ideas, shoot ...
>>
>> --
>> Johan

Re: Failing svnrdump_tests.py#43 with 1.7.x on Windows (was: Re: 1.7.3 next week-ish?)

Posted by Daniel Shahaf <da...@elego.de>.
Johan Corveleyn wrote on Thu, Feb 09, 2012 at 02:33:25 +0100:
> Having another look at this failure ...
> 
> So far, we know (anyone, correct me if I'm wrong):
> 
> - 'svnrdump load' in this test fails with: "svnrdump: E140001:
> Unrecognized record type in stream". It seems the dumpfile contents of
> the file D/H/psi is split incorrectly (property content is dumped
> early, and text delta later).
> 
> - It only happens on Windows.
> 
> - It only happens with ra_serf (both 1.7.x. and trunk) talking to
> mod_dav_svn from 1.7.x@>=1239697 (i.e. after backport of r1237720 and
> r1239596 (stuff in mod_dav_svn/liveprops.c) from trunk).

There was IRC discussion about this:
http://colabti.org/irclogger/irclogger_log/svn-dev?date=2012-02-09 (~4pm)

It is suggested there that setting libsvn_ra_serf/update.c:MAX_NR_OF_CONNS
to "2" will prevent ra_serf from driving multiple window handlers
concurrently and thus avoid the bug.  (Presumably by causing the file to
finish on the right side of the errant close_directory() call.)

> 
> - If 1.7.x is rolled back to before r1239697, the problem does not occur.
> 
> - The problem does not occur with mod_dav_svn from trunk, not even
> when rolled back to (before or after) r1237720 or r1239596.
> 
> - So far, it has been reproduced by the buildbot svn-slik-w2k3-x64-ra,
> by Stephen Butler, and by my own (32bit) WinXP box. Paul Burba doesn't
> see the failure with his Windows build (debug and release).
> 
> 
> I'm trying some more experiments. If anyone else has any ideas, shoot ...
> 
> --
> Johan

Re: Failing svnrdump_tests.py#43 with 1.7.x on Windows (was: Re: 1.7.3 next week-ish?)

Posted by Daniel Shahaf <da...@elego.de>.
Johan Corveleyn wrote on Thu, Feb 09, 2012 at 04:03:43 +0100:
> BTW: for some reason, the test now also fails with this last test-run
> (with mod_dav_svn@1.7.x@1239696), but with another failure:
> 
> "svnrdump: E140001: Dump stream contains a malformed header (with no
> ':') at 'ile_prop 0112B188'"

Where is the 'f' of 'file_prop'?

> 
> However, that piece of text is output from the DBG statements. For
> some reason it interferes with the stream that 'svnrdump load' is
> trying to read. So that may also be a problem, though probably
> unrelated to the one we're investigating here.
> 
> -- 
> Johan

Re: Failing svnrdump_tests.py#43 with 1.7.x on Windows

Posted by Daniel Shahaf <da...@elego.de>.
Johan Corveleyn wrote on Thu, Feb 09, 2012 at 11:44:52 +0100:
> On Thu, Feb 9, 2012 at 10:45 AM, Philip Martin
> <ph...@wandisco.com> wrote:
> > Johan Corveleyn <jc...@gmail.com> writes:
> >
> >> DBG: dump_editor.c: 756: close_file 0112B188
> >> DBG: dump_editor.c: 552: add_file trunk/D/H/psi
> >> DBG: dump_editor.c: 725: apply_textdelta 0112B188
> >> DBG: dump_editor.c: 633: change_dir_prop 01150F00
> >> DBG: dump_editor.c: 633: change_dir_prop 01150F00
> >> DBG: dump_editor.c: 633: change_dir_prop 01150F00
> >> DBG: dump_editor.c: 633: change_dir_prop 01150F00
> >> DBG: dump_editor.c: 633: change_dir_prop 01150F00
> >> DBG: dump_editor.c: 518: close_directory 01150F00
> >> DBG: dump_editor.c: 675: change_file_prop 0112B188
> >> DBG: dump_editor.c: 675: change_file_prop 0112B188
> >> DBG: dump_editor.c: 675: change_file_prop 0112B188
> >> DBG: dump_editor.c: 675: change_file_prop 0112B188
> >> DBG: dump_editor.c: 675: change_file_prop 0112B188
> >> DBG: dump_editor.c: 756: close_file 0112B188
> >
> >> DBG: dump_editor.c: 552: add_file trunk/D/H/psi
> >> DBG: dump_editor.c: 725: apply_textdelta 0112B188
> >> DBG: dump_editor.c: 675: change_file_prop 0112B188
> >> DBG: dump_editor.c: 675: change_file_prop 0112B188
> >> DBG: dump_editor.c: 675: change_file_prop 0112B188
> >> DBG: dump_editor.c: 675: change_file_prop 0112B188
> >> DBG: dump_editor.c: 675: change_file_prop 0112B188
> >> DBG: dump_editor.c: 756: close_file 0112B188
> >
> > So this is what I expected, the problem is the close_directory between
> > open_file and close_file.  Because svnrdump doesn't implement a file
> > baton but instead just uses the edit baton to handle a single file, the
> > close_directory causes the file properties to be written early while the
> > text is written later in close_file.
> >
> > What I don't understand is how close_directory gets called between
> > open_file and close_file.  When libsvn_ra_serf calls open_file it does
> > so from libsvn_ra_serf/update.c:handle_fetch and later in that same
> > close_file is called.  How does close_directory get invoked between the
> > two?  What is the callstack when close_directory is called?
> 
> I'll do some more experiments tonight.
> 
> I'm not very experienced in debugging C code ... any hints on how to
> get a hold of the callstack easily? I'd think of setting a breakpoint

Are you aware of the wrapper editor debug_editor.c?

> there, but it would be best to do that with a conditional breakpoint
> somehow, to avoid having to resume a dozen times. But I'm still using
> Visual C Express 2008 here (the "free" version), and I'm not sure if
> that supports conditional breakpoints. And besides, I'm not sure what
> condition I could use ...

Well, you could just print the call stack unconditionally (look for
VC-specific means to do that).  Otherwise, I suppose a global variable
(or an envvar) such as SEEN_THE_FIRST_close_file_CALL would do nicely.

> 
> Or can I simply write the callstack to stdout/stderr somehow at that
> point, without erroring out?
> 
> What's still very mysterious about all this:
> - Why doesn't this bug present itself with mod_dav_svn from trunk.
> - Why does it only show up on Windows.
> 

Yes, ra_serf-specific, branch-specific, windows-specific at the same
time.  Odd.

As another angle: why does the bug not trigger for Paul?  Is it due to
his build flags, or due to his version of (mod_dav|apr|Windows|libserf)?

> But I like a little mystery now and then, so ... we'll see.
> 
> -- 
> Johan

Re: Failing svnrdump_tests.py#43 with 1.7.x on Windows

Posted by Philip Martin <ph...@wandisco.com>.
Johan Corveleyn <jc...@gmail.com> writes:

> I'm not very experienced in debugging C code ... any hints on how to
> get a hold of the callstack easily? I'd think of setting a breakpoint
> there, but it would be best to do that with a conditional breakpoint
> somehow, to avoid having to resume a dozen times.

There will only be a few calls.  I'd set breakpoints on add_file and
close_directory in dump_editor.c and simply resume at each break.

-- 
uberSVN: Apache Subversion Made Easy
http://www.uberSVN.com

Re: Failing svnrdump_tests.py#43 with 1.7.x on Windows

Posted by Johan Corveleyn <jc...@gmail.com>.
On Thu, Feb 9, 2012 at 10:45 AM, Philip Martin
<ph...@wandisco.com> wrote:
> Johan Corveleyn <jc...@gmail.com> writes:
>
>> DBG: dump_editor.c: 756: close_file 0112B188
>> DBG: dump_editor.c: 552: add_file trunk/D/H/psi
>> DBG: dump_editor.c: 725: apply_textdelta 0112B188
>> DBG: dump_editor.c: 633: change_dir_prop 01150F00
>> DBG: dump_editor.c: 633: change_dir_prop 01150F00
>> DBG: dump_editor.c: 633: change_dir_prop 01150F00
>> DBG: dump_editor.c: 633: change_dir_prop 01150F00
>> DBG: dump_editor.c: 633: change_dir_prop 01150F00
>> DBG: dump_editor.c: 518: close_directory 01150F00
>> DBG: dump_editor.c: 675: change_file_prop 0112B188
>> DBG: dump_editor.c: 675: change_file_prop 0112B188
>> DBG: dump_editor.c: 675: change_file_prop 0112B188
>> DBG: dump_editor.c: 675: change_file_prop 0112B188
>> DBG: dump_editor.c: 675: change_file_prop 0112B188
>> DBG: dump_editor.c: 756: close_file 0112B188
>
>> DBG: dump_editor.c: 552: add_file trunk/D/H/psi
>> DBG: dump_editor.c: 725: apply_textdelta 0112B188
>> DBG: dump_editor.c: 675: change_file_prop 0112B188
>> DBG: dump_editor.c: 675: change_file_prop 0112B188
>> DBG: dump_editor.c: 675: change_file_prop 0112B188
>> DBG: dump_editor.c: 675: change_file_prop 0112B188
>> DBG: dump_editor.c: 675: change_file_prop 0112B188
>> DBG: dump_editor.c: 756: close_file 0112B188
>
> So this is what I expected, the problem is the close_directory between
> open_file and close_file.  Because svnrdump doesn't implement a file
> baton but instead just uses the edit baton to handle a single file, the
> close_directory causes the file properties to be written early while the
> text is written later in close_file.
>
> What I don't understand is how close_directory gets called between
> open_file and close_file.  When libsvn_ra_serf calls open_file it does
> so from libsvn_ra_serf/update.c:handle_fetch and later in that same
> close_file is called.  How does close_directory get invoked between the
> two?  What is the callstack when close_directory is called?

I'll do some more experiments tonight.

I'm not very experienced in debugging C code ... any hints on how to
get a hold of the callstack easily? I'd think of setting a breakpoint
there, but it would be best to do that with a conditional breakpoint
somehow, to avoid having to resume a dozen times. But I'm still using
Visual C Express 2008 here (the "free" version), and I'm not sure if
that supports conditional breakpoints. And besides, I'm not sure what
condition I could use ...

Or can I simply write the callstack to stdout/stderr somehow at that
point, without erroring out?

What's still very mysterious about all this:
- Why doesn't this bug present itself with mod_dav_svn from trunk.
- Why does it only show up on Windows.

But I like a little mystery now and then, so ... we'll see.

-- 
Johan

Re: Failing svnrdump_tests.py#43 with 1.7.x on Windows

Posted by Philip Martin <ph...@wandisco.com>.
Johan Corveleyn <jc...@gmail.com> writes:

> DBG: dump_editor.c: 756: close_file 0112B188
> DBG: dump_editor.c: 552: add_file trunk/D/H/psi
> DBG: dump_editor.c: 725: apply_textdelta 0112B188
> DBG: dump_editor.c: 633: change_dir_prop 01150F00
> DBG: dump_editor.c: 633: change_dir_prop 01150F00
> DBG: dump_editor.c: 633: change_dir_prop 01150F00
> DBG: dump_editor.c: 633: change_dir_prop 01150F00
> DBG: dump_editor.c: 633: change_dir_prop 01150F00
> DBG: dump_editor.c: 518: close_directory 01150F00
> DBG: dump_editor.c: 675: change_file_prop 0112B188
> DBG: dump_editor.c: 675: change_file_prop 0112B188
> DBG: dump_editor.c: 675: change_file_prop 0112B188
> DBG: dump_editor.c: 675: change_file_prop 0112B188
> DBG: dump_editor.c: 675: change_file_prop 0112B188
> DBG: dump_editor.c: 756: close_file 0112B188

> DBG: dump_editor.c: 552: add_file trunk/D/H/psi
> DBG: dump_editor.c: 725: apply_textdelta 0112B188
> DBG: dump_editor.c: 675: change_file_prop 0112B188
> DBG: dump_editor.c: 675: change_file_prop 0112B188
> DBG: dump_editor.c: 675: change_file_prop 0112B188
> DBG: dump_editor.c: 675: change_file_prop 0112B188
> DBG: dump_editor.c: 675: change_file_prop 0112B188
> DBG: dump_editor.c: 756: close_file 0112B188

So this is what I expected, the problem is the close_directory between
open_file and close_file.  Because svnrdump doesn't implement a file
baton but instead just uses the edit baton to handle a single file, the
close_directory causes the file properties to be written early while the
text is written later in close_file.

What I don't understand is how close_directory gets called between
open_file and close_file.  When libsvn_ra_serf calls open_file it does
so from libsvn_ra_serf/update.c:handle_fetch and later in that same
close_file is called.  How does close_directory get invoked between the
two?  What is the callstack when close_directory is called?

-- 
uberSVN: Apache Subversion Made Easy
http://www.uberSVN.com

Re: Failing svnrdump_tests.py#43 with 1.7.x on Windows (was: Re: 1.7.3 next week-ish?)

Posted by Johan Corveleyn <jc...@gmail.com>.
On Thu, Feb 9, 2012 at 2:33 AM, Johan Corveleyn <jc...@gmail.com> wrote:
> Having another look at this failure ...
>
> So far, we know (anyone, correct me if I'm wrong):
>
> - 'svnrdump load' in this test fails with: "svnrdump: E140001:
> Unrecognized record type in stream". It seems the dumpfile contents of
> the file D/H/psi is split incorrectly (property content is dumped
> early, and text delta later).
>
> - It only happens on Windows.
>
> - It only happens with ra_serf (both 1.7.x. and trunk) talking to
> mod_dav_svn from 1.7.x@>=1239697 (i.e. after backport of r1237720 and
> r1239596 (stuff in mod_dav_svn/liveprops.c) from trunk).
>
> - If 1.7.x is rolled back to before r1239697, the problem does not occur.
>
> - The problem does not occur with mod_dav_svn from trunk, not even
> when rolled back to (before or after) r1237720 or r1239596.
>
> - So far, it has been reproduced by the buildbot svn-slik-w2k3-x64-ra,
> by Stephen Butler, and by my own (32bit) WinXP box. Paul Burba doesn't
> see the failure with his Windows build (debug and release).
>
>
> I'm trying some more experiments. If anyone else has any ideas, shoot ...


On Tue, Feb 7, 2012 at 1:11 PM, Philip Martin
<ph...@wandisco.com> wrote:
> "roderich.schupp@googlemail.com" <ro...@googlemail.com>
> writes:
>
>> It's definitely an ordering problem, look at the whole series of operations:
>>
>> ...
>> add file trunk/D/H/omega
>> add file trunk/D/H/psi        <--- the one with the bogus split props/text
>> add dir branches
>> add dir trunk/C
>> add dir trunk/B/F
>> ...
>>
>> "branches" starts while "trunk/..." has not finished yet.
>> Also "trunk/B/.." continues though already "trunk/D/..." has started.
>
> On my Linux box neon gives me:
>
> DBG: dump_editor.c: 546: add_file trunk/D/H/omega
> DBG: dump_editor.c: 669: change_file_prop 0xf76820
> DBG: dump_editor.c: 669: change_file_prop 0xf76820
> DBG: dump_editor.c: 669: change_file_prop 0xf76820
> DBG: dump_editor.c: 669: change_file_prop 0xf76820
> DBG: dump_editor.c: 669: change_file_prop 0xf76820
> DBG: dump_editor.c: 719: apply_textdelta 0xf76820
> DBG: dump_editor.c: 750: close_file 0xf76820
> DBG: dump_editor.c: 546: add_file trunk/D/H/psi
> DBG: dump_editor.c: 669: change_file_prop 0x1333820
> DBG: dump_editor.c: 669: change_file_prop 0x1333820
> DBG: dump_editor.c: 669: change_file_prop 0x1333820
> DBG: dump_editor.c: 669: change_file_prop 0x1333820
> DBG: dump_editor.c: 669: change_file_prop 0x1333820
> DBG: dump_editor.c: 719: apply_textdelta 0x1333820
> DBG: dump_editor.c: 750: close_file 0x1333820
> DBG: dump_editor.c: 512: close_directory 0x1304148
> DBG: dump_editor.c: 512: close_directory 0x1303780
> DBG: dump_editor.c: 512: close_directory 0x13026b8
> DBG: dump_editor.c: 432: add_directory branches
>
> while serf gives me:
>
> DBG: dump_editor.c: 546: add_file trunk/D/H/omega
> DBG: dump_editor.c: 719: apply_textdelta 0x19be798
> DBG: dump_editor.c: 669: change_file_prop 0x19be798
> DBG: dump_editor.c: 669: change_file_prop 0x19be798
> DBG: dump_editor.c: 669: change_file_prop 0x19be798
> DBG: dump_editor.c: 669: change_file_prop 0x19be798
> DBG: dump_editor.c: 669: change_file_prop 0x19be798
> DBG: dump_editor.c: 750: close_file 0x19be798
> DBG: dump_editor.c: 546: add_file trunk/D/H/psi
> DBG: dump_editor.c: 719: apply_textdelta 0x1ff8798
> DBG: dump_editor.c: 669: change_file_prop 0x1ff8798
> DBG: dump_editor.c: 669: change_file_prop 0x1ff8798
> DBG: dump_editor.c: 669: change_file_prop 0x1ff8798
> DBG: dump_editor.c: 669: change_file_prop 0x1ff8798
> DBG: dump_editor.c: 669: change_file_prop 0x1ff8798
> DBG: dump_editor.c: 750: close_file 0x1ff8798
> DBG: dump_editor.c: 432: add_directory branches
>
> Both of those orders work.
>
> The dump editor used by svnrdump doesn't use an explicit file baton, it
> simply uses the editor baton to collect the data for the "current" file.
> Assuming our editor API allows multiple files to be open at the same
> time then svnrdump probably needs to grow a dedicated file baton and use
> that in close_file.

Ok, I also enabled "#define LDR_DBG(x) SVN_DBG(x)" in dump_editor.c,
and when running the test I see the following when the corrupt dump
file is produced (this happens with "svnrdump.exe -q dump -r1:10 ..."
in svnrdump_tests.py line 543):

With svnrdump@trunk vs. mod_dav_svn@1.7.x@HEAD:

...
DBG: dump_editor.c: 552: add_file trunk/D/H/omega
DBG: dump_editor.c: 725: apply_textdelta 0112B188
DBG: dump_editor.c: 675: change_file_prop 0112B188
DBG: dump_editor.c: 675: change_file_prop 0112B188
DBG: dump_editor.c: 675: change_file_prop 0112B188
DBG: dump_editor.c: 675: change_file_prop 0112B188
DBG: dump_editor.c: 675: change_file_prop 0112B188
DBG: dump_editor.c: 756: close_file 0112B188
DBG: dump_editor.c: 552: add_file trunk/D/H/psi
DBG: dump_editor.c: 725: apply_textdelta 0112B188
DBG: dump_editor.c: 633: change_dir_prop 01150F00
DBG: dump_editor.c: 633: change_dir_prop 01150F00
DBG: dump_editor.c: 633: change_dir_prop 01150F00
DBG: dump_editor.c: 633: change_dir_prop 01150F00
DBG: dump_editor.c: 633: change_dir_prop 01150F00
DBG: dump_editor.c: 518: close_directory 01150F00
DBG: dump_editor.c: 675: change_file_prop 0112B188
DBG: dump_editor.c: 675: change_file_prop 0112B188
DBG: dump_editor.c: 675: change_file_prop 0112B188
DBG: dump_editor.c: 675: change_file_prop 0112B188
DBG: dump_editor.c: 675: change_file_prop 0112B188
DBG: dump_editor.c: 756: close_file 0112B188
DBG: dump_editor.c: 633: change_dir_prop 01151240
DBG: dump_editor.c: 633: change_dir_prop 01151240
DBG: dump_editor.c: 633: change_dir_prop 01151240
DBG: dump_editor.c: 633: change_dir_prop 01151240
DBG: dump_editor.c: 633: change_dir_prop 01151240
DBG: dump_editor.c: 518: close_directory 01151240
DBG: dump_editor.c: 633: change_dir_prop 01150D18
DBG: dump_editor.c: 633: change_dir_prop 01150D18
DBG: dump_editor.c: 633: change_dir_prop 01150D18
DBG: dump_editor.c: 633: change_dir_prop 01150D18
DBG: dump_editor.c: 633: change_dir_prop 01150D18
DBG: dump_editor.c: 518: close_directory 01150D18
DBG: dump_editor.c: 438: add_directory branches
...


With svnrdump@trunk vs. mod_dav_svn@1.7.x@1239696 (no corruption):

...
DBG: dump_editor.c: 552: add_file trunk/D/H/omega
DBG: dump_editor.c: 725: apply_textdelta 0112B188
DBG: dump_editor.c: 675: change_file_prop 0112B188
DBG: dump_editor.c: 675: change_file_prop 0112B188
DBG: dump_editor.c: 675: change_file_prop 0112B188
DBG: dump_editor.c: 675: change_file_prop 0112B188
DBG: dump_editor.c: 675: change_file_prop 0112B188
DBG: dump_editor.c: 756: close_file 0112B188
DBG: dump_editor.c: 633: change_dir_prop 01150F00
DBG: dump_editor.c: 633: change_dir_prop 01150F00
DBG: dump_editor.c: 633: change_dir_prop 01150F00
DBG: dump_editor.c: 633: change_dir_prop 01150F00
DBG: dump_editor.c: 633: change_dir_prop 01150F00
DBG: dump_editor.c: 518: close_directory 01150F00
DBG: dump_editor.c: 552: add_file trunk/D/H/psi
DBG: dump_editor.c: 725: apply_textdelta 0112B188
DBG: dump_editor.c: 675: change_file_prop 0112B188
DBG: dump_editor.c: 675: change_file_prop 0112B188
DBG: dump_editor.c: 675: change_file_prop 0112B188
DBG: dump_editor.c: 675: change_file_prop 0112B188
DBG: dump_editor.c: 675: change_file_prop 0112B188
DBG: dump_editor.c: 756: close_file 0112B188
DBG: dump_editor.c: 633: change_dir_prop 01151240
DBG: dump_editor.c: 633: change_dir_prop 01151240
DBG: dump_editor.c: 633: change_dir_prop 01151240
DBG: dump_editor.c: 633: change_dir_prop 01151240
DBG: dump_editor.c: 633: change_dir_prop 01151240
DBG: dump_editor.c: 518: close_directory 01151240
DBG: dump_editor.c: 633: change_dir_prop 01150D18
DBG: dump_editor.c: 633: change_dir_prop 01150D18
DBG: dump_editor.c: 633: change_dir_prop 01150D18
DBG: dump_editor.c: 633: change_dir_prop 01150D18
DBG: dump_editor.c: 633: change_dir_prop 01150D18
DBG: dump_editor.c: 518: close_directory 01150D18
DBG: dump_editor.c: 438: add_directory branches
...


Any ideas?

BTW: for some reason, the test now also fails with this last test-run
(with mod_dav_svn@1.7.x@1239696), but with another failure:

"svnrdump: E140001: Dump stream contains a malformed header (with no
':') at 'ile_prop 0112B188'"

However, that piece of text is output from the DBG statements. For
some reason it interferes with the stream that 'svnrdump load' is
trying to read. So that may also be a problem, though probably
unrelated to the one we're investigating here.

-- 
Johan