You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by Julian Foad <ju...@apache.org> on 2018/03/14 07:58:36 UTC

Rolling 1.10.0-rc2

Julian Foad wrote on 2018-02-28:
> I'm happy to announce the release of Apache Subversion 1.10.0-rc1.

That was 2 weeks ago and we need an RC2.

I plan to roll 1.10.0-rc2 this week so we can release it by next 
Wednesday (3 weeks from RC1).

Please review the 'STATUS' file and report here if there's anything else 
I should take into account.

If anyone's willing to write a brief list of the changes since RC1, to 
include in the RC2 announcement, that would be an added bonus.

- Julian

Re: Rolling 1.10.0-rc2

Posted by Julian Foad <ju...@apache.org>.
Julian Foad wrote:
> We now need one more review.

Thanks, Bert!

Looks like we're "green" again: the other nominations in the 'STATUS' 
file are only "nice to have" in my opinion.

So I'll plan to roll RC2 ASAP -- which means around mid-week.

- Julian

Re: Rolling 1.10.0-rc2

Posted by Julian Foad <ju...@apache.org>.
Julian Foad wrote:
> Can anyone volunteer to review this important block-read fix?
> 
> (adding to CC: some people who might)

Thanks for your review, stefan2!

We now need one more review.

- Julian


> Philip Martin wrote:
>> Julian Foad writes:
>>> How do you feel about the amount of test coverage we are giving this
>>> block-read code so far -- is it sufficient for a 1.10.0 release? And
>>> should I roll RC2 ASAP or not?
>>
>> The block-read fix in STATUS still needs approval.
>>
>> As well as running the regression tests with block-read enabled I've
>> also been putting some real repositories through block-read mode.  I've
>> used the original Collab Subversion repository: 40,000 revisions and
>> about 0.5GB, the Subversion part of the ASF repository: 1.8M revisions
>> and about 1.5GB, and the GCC repository: 250,000 revisions and about
>> 9GB.  I've done both dump/load and svnsync, with svnsync writing over
>> http: and svn:.  Writing over network protocols puts the FSFS cache in
>> the server process and that allows me to access the same cache with read
>> requests such as 'svnbench null-export' while the commits are being
>> made.
>>
>> I've not had any further problems with these tests.
>>

Re: Rolling 1.10.0-rc2

Posted by Julian Foad <ju...@apache.org>.
Can anyone volunteer to review this important block-read fix?

(adding to CC: some people who might)

- Julian


Philip Martin wrote:
> Julian Foad writes:
>> How do you feel about the amount of test coverage we are giving this
>> block-read code so far -- is it sufficient for a 1.10.0 release? And
>> should I roll RC2 ASAP or not?
> 
> The block-read fix in STATUS still needs approval.
> 
> As well as running the regression tests with block-read enabled I've
> also been putting some real repositories through block-read mode.  I've
> used the original Collab Subversion repository: 40,000 revisions and
> about 0.5GB, the Subversion part of the ASF repository: 1.8M revisions
> and about 1.5GB, and the GCC repository: 250,000 revisions and about
> 9GB.  I've done both dump/load and svnsync, with svnsync writing over
> http: and svn:.  Writing over network protocols puts the FSFS cache in
> the server process and that allows me to access the same cache with read
> requests such as 'svnbench null-export' while the commits are being
> made.
> 
> I've not had any further problems with these tests.
> 

Re: Rolling 1.10.0-rc2

Posted by Philip Martin <ph...@codematters.co.uk>.
Julian Foad <ju...@apache.org> writes:

> How do you feel about the amount of test coverage we are giving this
> block-read code so far -- is it sufficient for a 1.10.0 release? And
> should I roll RC2 ASAP or not?

The block-read fix in STATUS still needs approval.

As well as running the regression tests with block-read enabled I've
also been putting some real repositories through block-read mode.  I've
used the original Collab Subversion repository: 40,000 revisions and
about 0.5GB, the Subversion part of the ASF repository: 1.8M revisions
and about 1.5GB, and the GCC repository: 250,000 revisions and about
9GB.  I've done both dump/load and svnsync, with svnsync writing over
http: and svn:.  Writing over network protocols puts the FSFS cache in
the server process and that allows me to access the same cache with read
requests such as 'svnbench null-export' while the commits are being
made.

I've not had any further problems with these tests.

-- 
Philip

Re: Rolling 1.10.0-rc2

Posted by Julian Foad <ju...@apache.org>.
Julian Foad wrote:
> Philip Martin wrote:
>> Julian Foad <ju...@apache.org> writes:
>>> AFAIK we still need to arrange adequate test coverage for it.
>>
>> Test coverage is provided by the other nominations in STATUS.  Apply
>> r1827105, r1827114 and run "make davautocheck BLOCK_READ=1" and see lots
>> of tests FAIL.  Apply the 4725 fix and they PASS.
> 
> Have we got any bots doing this?

What I really mean is:

Thank you very much for doing all this.

How do you feel about the amount of test coverage we are giving this 
block-read code so far -- is it sufficient for a 1.10.0 release? And 
should I roll RC2 ASAP or not?

My feeling is it's not yet been tested enough to be confident for a 
1.10.0 release, and we should get it tested by bots on most platforms, 
but I should probably go ahead with RC2 ASAP and then we can evaluate 
and test further at the same time.

If that's agreed, I'll try to roll RC2 around Wednesday this week.

- Julian

Re: Rolling 1.10.0-rc2

Posted by Julian Foad <ju...@apache.org>.
Philip Martin wrote:
> Julian Foad <ju...@apache.org> writes:
>> AFAIK we still need to arrange adequate test coverage for it.
> 
> Test coverage is provided by the other nominations in STATUS.  Apply
> r1827105, r1827114 and run "make davautocheck BLOCK_READ=1" and see lots
> of tests FAIL.  Apply the 4725 fix and they PASS.

Have we got any bots doing this?

>>    * revprop caching has changed for 1.10; isn't mentioned in the release notes
> 
> r1827201

Thanks.

- Julian


Re: Rolling 1.10.0-rc2

Posted by Philip Martin <ph...@codematters.co.uk>.
Julian Foad <ju...@apache.org> writes:

> AFAIK we still need to arrange adequate test coverage for it.

Test coverage is provided by the other nominations in STATUS.  Apply
r1827105, r1827114 and run "make davautocheck BLOCK_READ=1" and see lots
of tests FAIL.  Apply the 4725 fix and they PASS.

>   * revprop caching has changed for 1.10; isn't mentioned in the release notes

r1827201

-- 
Philip

Re: Rolling 1.10.0-rc2

Posted by Julian Foad <ju...@apache.org>.
Julian Foad wrote:
> I plan to roll 1.10.0-rc2 this week so we can release it by next 
> Wednesday (3 weeks from RC1).

Update: That plan is delayed by

   "svnadmin load error fsfs cache"
   https://issues.apache.org/jira/browse/SVN-4725
   "FSFS cache reading one svndiff window beyond end"
   https://issues.apache.org/jira/browse/SVN-4727
   (both relate to the same underlying problem)

Philip has already found and fixed the problem.

AFAIK we still need to arrange adequate test coverage for it.

Also noted:

   * revprop caching has changed for 1.10; isn't mentioned in the 
release notes

- Julian


> Please review the 'STATUS' file and report here if there's anything else 
> I should take into account.
> 
> If anyone's willing to write a brief list of the changes since RC1, to 
> include in the RC2 announcement, that would be an added bonus.


Re: svnadmin load error with 1.10 that seems to be fsfs cache related

Posted by Philip Martin <ph...@codematters.co.uk>.
Julian Foad <ju...@apache.org> writes:

> Philip Martin wrote:
>> cached_data.c:parse_raw_window() where the svndiff version is hard-coded
>> to 1 in the call to svn_txdelta_read_svndiff_window.  Changing that to 2
>> allows the regression test to pass, the question is where should the
>> correct value be obtained?
>
> The simple answer appears to be 'rs->ver' (where 'rs' needs to be
> passed in via the function's baton).

It's not that simple.  When we cache the data rs->ver is set to 1 by
init_rep_state().  When we retrieve the data rs->ver is set to -1 by
create_rep_state().  I don't know why these functions initialize rs
differently.  Neither appears to be the value we want.

-- 
Philip

Re: Testsuite options

Posted by Philip Martin <ph...@codematters.co.uk>.
Philip Martin <ph...@codematters.co.uk> writes:

> It is even more pointless than that.  Looking at the code in
> svn_fs_fs__initialize_caches() I see that revprop caching is now enabled
> unconditionally for all FSFS access.
>
> The config setting SVN_FS_CONFIG_FSFS_CACHE_REVPROPS is public in
> svn_fs.h so it has to remain for backward compatibility.  The
> mod_dav_svn SVNCacheRevprops apache directive also still exists.
> svnserve still has the --cache-revprops command line option.
>
> Oddly, we still have code in mod_dav_svn.c and svnserve.c that responds
> to the directive/option setting and changes the setting of
> SVN_FS_CONFIG_FSFS_CACHE_REVPROPS in the config hash.  That's pointless
> as there is no FSFS code that responds to that config setting.
>
> I suppose the mod_dav_svn directive and svnserve command line option
> have to remain for backward compatibility, but why do we retain the code
> that sets SVN_FS_CONFIG_FSFS_CACHE_REVPROPS?

I didn't realise that revprop caching had changed for 1.10, and since it
isn't mentioned in the release notes I guess that applies to other
people as well.

Setting

  SVNCacheRevprops on

in apache config is now a no-op, while setting

  SVNCacheRevprops off

is silently ignored.  Similarly, the command line option

  svnserve --cache-revprops on

is a no-op, and the command line option

  svnserve --cache-revprops off

is silently ignored.

Is documenting this change in the release notes sufficient?  Should 1.10
raise a warning/error if the user makes a futile attempt to disable
revprop caching?

-- 
Philip

Re: Testsuite options

Posted by Philip Martin <ph...@codematters.co.uk>.
Philip Martin <ph...@codematters.co.uk> writes:

> The testsuite already has support for testing with revprop-cache on.  It
> can be used with svnserveautocheck but is pointless with davautocheck as
> mod_dav_svn now enables it by default.

It is even more pointless than that.  Looking at the code in
svn_fs_fs__initialize_caches() I see that revprop caching is now enabled
unconditionally for all FSFS access.

The config setting SVN_FS_CONFIG_FSFS_CACHE_REVPROPS is public in
svn_fs.h so it has to remain for backward compatibility.  The
mod_dav_svn SVNCacheRevprops apache directive also still exists.
svnserve still has the --cache-revprops command line option.

Oddly, we still have code in mod_dav_svn.c and svnserve.c that responds
to the directive/option setting and changes the setting of
SVN_FS_CONFIG_FSFS_CACHE_REVPROPS in the config hash.  That's pointless
as there is no FSFS code that responds to that config setting.

I suppose the mod_dav_svn directive and svnserve command line option
have to remain for backward compatibility, but why do we retain the code
that sets SVN_FS_CONFIG_FSFS_CACHE_REVPROPS?

-- 
Philip

Testsuite options

Posted by Philip Martin <ph...@codematters.co.uk>.
Branko Čibej <br...@apache.org> writes:

> As long as we have tuning options, we should test them. It's not that
> hard to add buildslave configurations.
>
> As for eliminating them: it would be nice if the server could auto-tune
> these parameters, but that is probably not a worthwhile goal to spend
> our efforts at (and that would make testing behaviour with different
> options harder).

Looking at mod_dav_svn there are tuning options:

                   default

  txdelta-cache:     on
  fulltext-cache:    on
  nodeprop-cache:    on
  revprop-cache:     on
  block-read:        off
  compression-level: 5

in svnserve there are:

  cache-txdeltas:    on
  cache-fulltext:    on
  cache-nodeprops:   on
  cache-revprops:    off
  block-read:        off

and in db/fsfs.conf there are:

  rep-sharing:               true
  dir-deltification:         true
  props-deltification:       true
  max-deltification:         1023
  max-linear-deltification:    16
  compression:                lz4
  revprop-pack-szie:           16
  compress-packed-revprops: false
  block-size:                  64
  l2p-page-size:             8192
  p2l-page-size:             1024

The testsuite already has support for testing with revprop-cache on.  It
can be used with svnserveautocheck but is pointless with davautocheck as
mod_dav_svn now enables it by default.

I've just added davautocheck/svnserveautocheck support for testing with
block-read on.  Using it with davautocheck effectively tests with all
options enabled.  There is no support for turning off the other caching
options.

The testsuite has support for changing the FSFS shard size.  I'm not
aware of any option for modifying the deltification settings.

The testsuite also has support for testing for compatibility with older
repository formats -- that should allow testing FSFS compression other
than lz4.

-- 
Philip

Re: svnadmin load error with 1.10 that seems to be fsfs cache related

Posted by Branko Čibej <br...@apache.org>.
On 17.03.2018 13:57, Philip Martin wrote:
> Julian Foad <ju...@apache.org> writes:
>
>> But does that mean this code isn't being tested by any of our manual
>> or buildbot testing in the last 6 months?
> Yes, I think so.
>
> For svnserve the code only gets enabled when svnserve has block-read
> enabled and svnserveautocheck doesn't do that.  By invoking svnserve
> manually it is possible to enable it and lots of tests fail with the
> zlib error:
>
> subversion/svnserve/svnserve -Tdr subversion/tests/cmdline --block-read 1
> make check BASE_URL=svn://localhost
>
> For http the code only gets enabled when mod_dav_svn has block read
> enabled and davautocheck doesn't do that.  Patching davautocheck.sh
>
> Index: ../src/subversion/tests/cmdline/davautocheck.sh
> ===================================================================
> --- ../src/subversion/tests/cmdline/davautocheck.sh	(revision 1827068)
> +++ ../src/subversion/tests/cmdline/davautocheck.sh	(working copy)
> @@ -548,6 +548,7 @@
>    Require           valid-user
>    SVNAdvertiseV2Protocol ${ADVERTISE_V2_PROTOCOL}
>    SVNCacheRevProps  ${CACHE_REVPROPS_SETTING}
> +  SVNBlockRead on
>    ${SVN_PATH_AUTHZ_LINE}
>  </Location>
>
> causes lots of tests to fail with the zlib error.
>
> There doesn't seen to be a way to enable the block-read code for svn
> file:// access but it could be enabled for svnadmin by making the cache
> greater than 64MB.  The regression tests use the default 16MB so
> patching main.py:
>
> Index: ../src/subversion/tests/cmdline/svntest/actions.py
> ===================================================================
> --- ../src/subversion/tests/cmdline/svntest/actions.py	(revision 1827068)
> +++ ../src/subversion/tests/cmdline/svntest/actions.py	(working copy)
> @@ -378,7 +378,7 @@
>  
>  def run_and_verify_dump(repo_dir, deltas=False):
>    "Runs 'svnadmin dump' and reports any errors, returning the dump content."
> -  args = ()
> +  args = ('-M65')
>    if deltas:
>      args += ('--deltas',)
>    exit_code, output, errput = run_and_verify_svnadmin(
>
> causes tests to fail with the zlib error.
>
>> Extending our testing is one option. I would much prefer to apply the
>> principle of testable design: "if this code only runs sometimes,
>> change it so it runs always". Can we do that?
> There are a lot of tuning options for caching.  Do we want to add
> mechanisms to enable them individually in the testsuite?  Do we want to
> change the defaults in our binaries?  Do we want to eliminate the tuning
> options?

As long as we have tuning options, we should test them. It's not that
hard to add buildslave configurations.

As for eliminating them: it would be nice if the server could auto-tune
these parameters, but that is probably not a worthwhile goal to spend
our efforts at (and that would make testing behaviour with different
options harder).

-- Brane


Re: svnadmin load error with 1.10 that seems to be fsfs cache related

Posted by Philip Martin <ph...@codematters.co.uk>.
Julian Foad <ju...@apache.org> writes:

> But does that mean this code isn't being tested by any of our manual
> or buildbot testing in the last 6 months?

Yes, I think so.

For svnserve the code only gets enabled when svnserve has block-read
enabled and svnserveautocheck doesn't do that.  By invoking svnserve
manually it is possible to enable it and lots of tests fail with the
zlib error:

subversion/svnserve/svnserve -Tdr subversion/tests/cmdline --block-read 1
make check BASE_URL=svn://localhost

For http the code only gets enabled when mod_dav_svn has block read
enabled and davautocheck doesn't do that.  Patching davautocheck.sh

Index: ../src/subversion/tests/cmdline/davautocheck.sh
===================================================================
--- ../src/subversion/tests/cmdline/davautocheck.sh	(revision 1827068)
+++ ../src/subversion/tests/cmdline/davautocheck.sh	(working copy)
@@ -548,6 +548,7 @@
   Require           valid-user
   SVNAdvertiseV2Protocol ${ADVERTISE_V2_PROTOCOL}
   SVNCacheRevProps  ${CACHE_REVPROPS_SETTING}
+  SVNBlockRead on
   ${SVN_PATH_AUTHZ_LINE}
 </Location>

causes lots of tests to fail with the zlib error.

There doesn't seen to be a way to enable the block-read code for svn
file:// access but it could be enabled for svnadmin by making the cache
greater than 64MB.  The regression tests use the default 16MB so
patching main.py:

Index: ../src/subversion/tests/cmdline/svntest/actions.py
===================================================================
--- ../src/subversion/tests/cmdline/svntest/actions.py	(revision 1827068)
+++ ../src/subversion/tests/cmdline/svntest/actions.py	(working copy)
@@ -378,7 +378,7 @@
 
 def run_and_verify_dump(repo_dir, deltas=False):
   "Runs 'svnadmin dump' and reports any errors, returning the dump content."
-  args = ()
+  args = ('-M65')
   if deltas:
     args += ('--deltas',)
   exit_code, output, errput = run_and_verify_svnadmin(

causes tests to fail with the zlib error.

> Extending our testing is one option. I would much prefer to apply the
> principle of testable design: "if this code only runs sometimes,
> change it so it runs always". Can we do that?

There are a lot of tuning options for caching.  Do we want to add
mechanisms to enable them individually in the testsuite?  Do we want to
change the defaults in our binaries?  Do we want to eliminate the tuning
options?

-- 
Philip

Re: svnadmin load error with 1.10 that seems to be fsfs cache related

Posted by Julian Foad <ju...@apache.org>.
Philip Martin wrote:
> cached_data.c:parse_raw_window() where the svndiff version is hard-coded
> to 1 in the call to svn_txdelta_read_svndiff_window.  Changing that to 2
> allows the regression test to pass, the question is where should the
> correct value be obtained?

The simple answer appears to be 'rs->ver' (where 'rs' needs to be passed 
in via the function's baton).

But does that mean this code isn't being tested by any of our manual or 
buildbot testing in the last 6 months?

If so, we need to resolve that somehow before I am happy to release it.

Extending our testing is one option. I would much prefer to apply the 
principle of testable design: "if this code only runs sometimes, change 
it so it runs always". Can we do that?

- Julian

Re: svnadmin load error with 1.10 that seems to be fsfs cache related

Posted by Philip Martin <ph...@codematters.co.uk>.
Philip Martin <ph...@codematters.co.uk> writes:

> I've committed and nominated for 1.10.  However I have found another
> caching problem:
>
> https://issues.apache.org/jira/browse/SVN-4727
>
> Loading my copy of the original Collab Subversion repository (40515
> revisions) fails when the cache is large:
>
> svnadmin: E160004: Corrupt node-revision '9-965551.0-857792.r988076/12'
> svnadmin: E160004: Reading one svndiff window read beyond the end of
> the representation

I think I've found the problem.  The call to auto_read_diff_version()
that I added for the previous problem is in the loop in cache_windows().
If there are more than one window chunks, and the first is found in the
cache, then when auto_read_diff_version() is called on subsequent chunks
it will reset some of the rep_state_t.

-- 
Philip

Re: svnadmin load error with 1.10 that seems to be fsfs cache related

Posted by Philip Martin <ph...@codematters.co.uk>.
Philip Martin <ph...@codematters.co.uk> writes:

> With a small cache the load completes and verification works, but
> verfication fails with a large cache:
>
> * Error verifying revision 11826.
> svnadmin: E160004: Corrupt node-revision '5-385.0.r8127/80'
> svnadmin: E160004: Reading one svndiff window read beyond the end of
> the representation

Repeated runs show the same error but on different revisions.  Here is
the debug error stack (note different revision):

* Error verifying revision 4965.
../src/subversion/svnadmin/svnadmin.c:2254,
../src/subversion/libsvn_repos/dump.c:2532,
../src/subversion/libsvn_repos/dump.c:2425,
../src/subversion/svnadmin/svnadmin.c:1028,
../src/subversion/libsvn_repos/dump.c:2369,
../src/subversion/libsvn_fs/fs-loader.c:754,
../src/subversion/libsvn_fs_fs/tree.c:4815,
../src/subversion/libsvn_fs_fs/tree.c:4749,
../src/subversion/libsvn_fs_fs/tree.c:4749,
../src/subversion/libsvn_fs_fs/tree.c:4749,
../src/subversion/libsvn_fs_fs/tree.c:4749,
../src/subversion/libsvn_fs_fs/tree.c:4748,
../src/subversion/libsvn_fs_fs/dag.c:200,
../src/subversion/libsvn_fs_fs/dag.c:167,
../src/subversion/libsvn_fs_fs/cached_data.c:521: (apr_err=SVN_ERR_FS_CORRUPT)
svnadmin: E160004: Corrupt node-revision '1m-3485.0.r4965/4'
../src/subversion/libsvn_fs_fs/cached_data.c:482,
../src/subversion/libsvn_fs_fs/cached_data.c:3749,
../src/subversion/libsvn_fs_fs/cached_data.c:3477,
../src/subversion/libsvn_fs_fs/cached_data.c:3414,
../src/subversion/libsvn_fs_fs/cached_data.c:3343: (apr_err=SVN_ERR_FS_CORRUPT)
svnadmin: E160004: Reading one svndiff window read beyond the end of the representation


-- 
Philip

Re: svnadmin load error with 1.10 that seems to be fsfs cache related

Posted by Philip Martin <ph...@codematters.co.uk>.
Branko Čibej <br...@apache.org> writes:

> On 15.03.2018 20:38, Philip Martin wrote:
>> Philip Martin <ph...@codematters.co.uk> writes:
>>
>>> I think the raw
>>> window cache may have to be modified to include the svndiff version.
>> Experimental patch:
>
> Looks approximately correct to me. Hard-coding version 1 would have been
> wrong before LZ4, too, since version 0 does exist (although it's not
> often used these days).

I've committed and nominated for 1.10.  However I have found another
caching problem:

https://issues.apache.org/jira/browse/SVN-4727

Loading my copy of the original Collab Subversion repository (40515
revisions) fails when the cache is large:

svnadmin: E160004: Corrupt node-revision '9-965551.0-857792.r988076/12'
svnadmin: E160004: Reading one svndiff window read beyond the end of the representation

With a small cache the load completes and verification works, but
verfication fails with a large cache:

* Error verifying revision 11826.
svnadmin: E160004: Corrupt node-revision '5-385.0.r8127/80'
svnadmin: E160004: Reading one svndiff window read beyond the end of the representation

-- 
Philip

Re: svnadmin load error with 1.10 that seems to be fsfs cache related

Posted by Branko Čibej <br...@apache.org>.
On 15.03.2018 20:38, Philip Martin wrote:
> Philip Martin <ph...@codematters.co.uk> writes:
>
>> I think the raw
>> window cache may have to be modified to include the svndiff version.
> Experimental patch:

Looks approximately correct to me. Hard-coding version 1 would have been
wrong before LZ4, too, since version 0 does exist (although it's not
often used these days).

-- Brane


Re: svnadmin load error with 1.10 that seems to be fsfs cache related

Posted by Philip Martin <ph...@codematters.co.uk>.
Philip Martin <ph...@codematters.co.uk> writes:

> I think the raw
> window cache may have to be modified to include the svndiff version.

Experimental patch:

Index: subversion/libsvn_fs_fs/cached_data.c
===================================================================
--- subversion/libsvn_fs_fs/cached_data.c	(revision 1826834)
+++ subversion/libsvn_fs_fs/cached_data.c	(working copy)
@@ -1268,7 +1268,7 @@ parse_raw_window(void **out,
   stream = svn_stream_from_string(&raw_window, result_pool);
 
   /* parse it */
-  SVN_ERR(svn_txdelta_read_svndiff_window(&result->window, stream, 1,
+  SVN_ERR(svn_txdelta_read_svndiff_window(&result->window, stream, window->ver,
                                           result_pool));
 
   /* complete the window and return it */
@@ -3212,7 +3212,7 @@ init_rep_state(rep_state_t *rs,
   rs->start = entry->offset + rs->header_size;
   rs->current = rep_header->type == svn_fs_fs__rep_plain ? 0 : 4;
   rs->size = entry->size - rep_header->header_size - 7;
-  rs->ver = 1;
+  rs->ver = -1;
   rs->chunk_index = 0;
   rs->raw_window_cache = ffd->raw_window_cache;
   rs->window_cache = ffd->txdelta_window_cache;
@@ -3310,6 +3310,8 @@ cache_windows(svn_fs_t *fs,
           apr_size_t window_len;
           char *buf;
 
+          auto_read_diff_version(rs, iterpool);
+
           /* navigate to the current window */
           SVN_ERR(rs_aligned_seek(rs, NULL, start_offset, iterpool));
           SVN_ERR(svn_txdelta__read_raw_window_len(&window_len,
@@ -3330,6 +3332,7 @@ cache_windows(svn_fs_t *fs,
           window.end_offset = rs->current;
           window.window.len = window_len;
           window.window.data = buf;
+          window.ver = rs->ver;
 
           /* cache the window now */
           SVN_ERR(svn_cache__set(rs->raw_window_cache, &key, &window,
Index: subversion/libsvn_fs_fs/temp_serializer.h
===================================================================
--- subversion/libsvn_fs_fs/temp_serializer.h	(revision 1826834)
+++ subversion/libsvn_fs_fs/temp_serializer.h	(working copy)
@@ -60,6 +60,9 @@ typedef struct
 
   /* the offset within the representation right after reading the window */
   apr_off_t end_offset;
+
+  /* svndiff version */
+  int ver;
 } svn_fs_fs__raw_cached_window_t;
 
-- 
Philip

Re: svnadmin load error with 1.10 that seems to be fsfs cache related

Posted by Philip Martin <ph...@codematters.co.uk>.
Branko Čibej <br...@apache.org> writes:

> The svndiff version is embedded in the first 4 bytes of the window data
> and should be parsed from there.

When we store the data in the raw window cache the svndiff version isn't
part of the stored data.  When we subsequently retreive the data from
the cache the svndiff version is no longer available.  I think the raw
window cache may have to be modified to include the svndiff version.

-- 
Philip

Re: svnadmin load error with 1.10 that seems to be fsfs cache related

Posted by Branko Čibej <br...@apache.org>.
On 15.03.2018 18:50, Philip Martin wrote:
> Philip Martin <ph...@codematters.co.uk> writes:
>
>> Evgeny Kotkov <ev...@visualsvn.com> writes:
>>
>>> Philip Martin <ph...@codematters.co.uk> writes:
>>>
>>>> That works as expected, but vary the cache size of the load process and
>>>> it fails.  The load succeeds with -M64 and smaller but fails with -M65
>>>> and larger:
>>> [...]
>>>
>>> Maybe this behavior could be related to the cache size threshold in svnadmin
>>> that enables the block read feature in fsfs (currently set to 64 MB, as per
>>> svnadmin.c:BLOCK_READ_CACHE_THRESHOLD).
>> Sounds plausible.
> It causes different code to run, in particular the window cache is
> enabled.  I'm not familiar with this code but the problem seems to be in
> cached_data.c:parse_raw_window() where the svndiff version is hard-coded
> to 1 in the call to svn_txdelta_read_svndiff_window.  Changing that to 2
> allows the regression test to pass, the question is where should the
> correct value be obtained?

The svndiff version is embedded in the first 4 bytes of the window data
and should be parsed from there.

-- Brane


Re: svnadmin load error with 1.10 that seems to be fsfs cache related

Posted by Philip Martin <ph...@codematters.co.uk>.
Philip Martin <ph...@codematters.co.uk> writes:

> Evgeny Kotkov <ev...@visualsvn.com> writes:
>
>> Philip Martin <ph...@codematters.co.uk> writes:
>>
>>> That works as expected, but vary the cache size of the load process and
>>> it fails.  The load succeeds with -M64 and smaller but fails with -M65
>>> and larger:
>>
>> [...]
>>
>> Maybe this behavior could be related to the cache size threshold in svnadmin
>> that enables the block read feature in fsfs (currently set to 64 MB, as per
>> svnadmin.c:BLOCK_READ_CACHE_THRESHOLD).
>
> Sounds plausible.

It causes different code to run, in particular the window cache is
enabled.  I'm not familiar with this code but the problem seems to be in
cached_data.c:parse_raw_window() where the svndiff version is hard-coded
to 1 in the call to svn_txdelta_read_svndiff_window.  Changing that to 2
allows the regression test to pass, the question is where should the
correct value be obtained?

-- 
Philip

Re: svnadmin load error with 1.10 that seems to be fsfs cache related

Posted by Philip Martin <ph...@codematters.co.uk>.
Evgeny Kotkov <ev...@visualsvn.com> writes:

> Philip Martin <ph...@codematters.co.uk> writes:
>
>> That works as expected, but vary the cache size of the load process and
>> it fails.  The load succeeds with -M64 and smaller but fails with -M65
>> and larger:
>
> [...]
>
> Maybe this behavior could be related to the cache size threshold in svnadmin
> that enables the block read feature in fsfs (currently set to 64 MB, as per
> svnadmin.c:BLOCK_READ_CACHE_THRESHOLD).

Sounds plausible.

Here is a simpler reproduction:

svnadmin create repo
svn -mm mkdir file://`pwd`/repo/subversion ^/subversion/trunk ^/subversion/branches
svnadmin dump -r1 -M100 repo > /dev/null

-- 
Philip

Re: svnadmin load error with 1.10 that seems to be fsfs cache related

Posted by Evgeny Kotkov <ev...@visualsvn.com>.
Philip Martin <ph...@codematters.co.uk> writes:

> That works as expected, but vary the cache size of the load process and
> it fails.  The load succeeds with -M64 and smaller but fails with -M65
> and larger:

[...]

Maybe this behavior could be related to the cache size threshold in svnadmin
that enables the block read feature in fsfs (currently set to 64 MB, as per
svnadmin.c:BLOCK_READ_CACHE_THRESHOLD).


Regards,
Evgeny Kotkov

Re: svnadmin load error with 1.10 that seems to be fsfs cache related

Posted by Philip Martin <ph...@codematters.co.uk>.
Philip Martin <ph...@codematters.co.uk> writes:

> $ svnadmin dump -q repo | svnadmin load -q -M1000 repo2
> svnadmin: E140001: zlib (uncompress): corrupt data: Decompression of
> svndiff data failed

This is now issue https://issues.apache.org/jira/browse/SVN-4725

-- 
Philip

svnadmin load error with 1.10 that seems to be fsfs cache related

Posted by Philip Martin <ph...@codematters.co.uk>.
Julian Foad <ju...@apache.org> writes:

> Julian Foad wrote on 2018-02-28:
>> I'm happy to announce the release of Apache Subversion 1.10.0-rc1.
>
> That was 2 weeks ago and we need an RC2.

svnadmin create repo
svn -mm mkdir file://`pwd`/repo/subversion
svn -mm mkdir file://`pwd`/repo/subversion/trunk ^/subversion/branches
svn -mm mkdir file://`pwd`/repo/subversion/trunk/src
svnadmin create repo2
svnadmin dump -q repo | svnadmin load -q repo2

That works as expected, but vary the cache size of the load process and
it fails.  The load succeeds with -M64 and smaller but fails with -M65
and larger:

$ svnadmin dump -q repo | svnadmin load -q -M1000 repo2
svnadmin: warning: apr_err=SVN_ERR_STREAM_MALFORMED_DATA
svnadmin: warning: W140001: zlib (uncompress): corrupt data: Decompression of svndiff data failed
../src-1.10/subversion/libsvn_repos/load.c:794,
../src-1.10/subversion/libsvn_repos/load-fs-vtable.c:1118,
../src-1.10/subversion/libsvn_fs/fs-loader.c:885,
../src-1.10/subversion/libsvn_fs_fs/tree.c:2333,
../src-1.10/subversion/libsvn_fs_fs/transaction.c:3983,
../src-1.10/subversion/libsvn_fs_fs/fs_fs.c:367,
../src-1.10/subversion/libsvn_fs_fs/fs_fs.c:239,
../src-1.10/subversion/libsvn_fs_fs/fs_fs.c:227,
../src-1.10/subversion/libsvn_fs_fs/transaction.c:3812,
../src-1.10/subversion/libsvn_fs_fs/transaction.c:3185,
../src-1.10/subversion/libsvn_fs_fs/transaction.c:3203,
../src-1.10/subversion/libsvn_fs_fs/transaction.c:2964,
../src-1.10/subversion/libsvn_fs_fs/transaction.c:2767,
../src-1.10/subversion/libsvn_fs_fs/transaction.c:653,
../src-1.10/subversion/libsvn_fs_fs/transaction.c:634,
../src-1.10/subversion/libsvn_subr/stream.c:221,
../src-1.10/subversion/libsvn_fs_fs/transaction.c:2736,
../src-1.10/subversion/libsvn_subr/stream.c:221,
../src-1.10/subversion/libsvn_delta/text_delta.c:524,
../src-1.10/subversion/libsvn_subr/stream.c:194,
../src-1.10/subversion/libsvn_fs_fs/cached_data.c:2148,
../src-1.10/subversion/libsvn_fs_fs/cached_data.c:1885,
../src-1.10/subversion/libsvn_fs_fs/cached_data.c:1704,
../src-1.10/subversion/libsvn_fs_fs/cached_data.c:1564,
../src-1.10/subversion/libsvn_fs_fs/cached_data.c:1323,
../src-1.10/subversion/libsvn_subr/cache-membuffer.c:3070,
../src-1.10/subversion/libsvn_subr/cache-membuffer.c:2608,
../src-1.10/subversion/libsvn_fs_fs/cached_data.c:1272,
../src-1.10/subversion/libsvn_delta/svndiff.c:590,
../src-1.10/subversion/libsvn_subr/compress_zlib.c:166,
../src-1.10/subversion/libsvn_subr/io.c:3699: (apr_err=SVN_ERR_STREAM_MALFORMED_DATA)
svnadmin: E140001: zlib (uncompress): corrupt data: Decompression of svndiff data failed


-- 
Philip

Re: Rolling 1.10.0-rc2

Posted by Daniel Shahaf <d....@daniel.shahaf.name>.
Johan Corveleyn wrote on Thu, Mar 15, 2018 at 12:00:57 +0100:
> C:\>python tools\dist\release.py  write-changelog --include-unlabeled-summaries branches/1.10.x tags/1.10.0-rc1
>     * Merge r1825045 from trunk: (r1825285)
>     * Merge r1825215 from trunk: (r1825286)
>     * Merge r1825306 from trunk: (r1826169)
>     * Merge r1825721 from trunk: (r1826064)
>     * Merge the r1825024 group from trunk: (r1825287)
>     * Merge the r1825736 group from trunk: (r1825795)
> 
> Hmmm, write-changelog should be improved to "recurse" into the merged
> revision / group to get those revision's changelog lines instead, or
> to parse it from the subsequent lines of the merge commit itself (as
> it was filled in by svn-role which got the info when processing
> STATUS).

release.py could import backport.parser and sic it on the log message, but
beware of py2/py3 mismatches.  (I'm not sure whether release.py works under py3
nor whether backport.py works under py2.)

Re: Rolling 1.10.0-rc2

Posted by Johan Corveleyn <jc...@gmail.com>.
On Wed, Mar 14, 2018 at 1:07 PM, Johan Corveleyn <jc...@gmail.com> wrote:
> On Wed, Mar 14, 2018 at 1:06 PM, Johan Corveleyn <jc...@gmail.com> wrote:
>> On Wed, Mar 14, 2018 at 12:22 PM, Branko Čibej <br...@apache.org> wrote:
>>> On 14.03.2018 08:58, Julian Foad wrote:
>> ...
>>>> If anyone's willing to write a brief list of the changes since RC1, to
>>>> include in the RC2 announcement, that would be an added bonus.
>>>
>>> svn log -v STATUS | grep svn-role. But that wouldn't really work, would
>>> it. :)
>>
>> Maybe try:
>>
>> release.py write-changelog --include-unlabeled-summaries
>> branches/1.10.x tags/1.10.0-rc1

Bah, doesn't yield very good results:

[[[
C:\>python tools\dist\release.py  write-changelog
--include-unlabeled-summaries branches/1.10.x tags/1.10.0-rc1
    * Merge r1825045 from trunk: (r1825285)
    * Merge r1825215 from trunk: (r1825286)
    * Merge r1825306 from trunk: (r1826169)
    * Merge r1825721 from trunk: (r1826064)
    * Merge the r1825024 group from trunk: (r1825287)
    * Merge the r1825736 group from trunk: (r1825795)

 User-visible changes:

 Developer-visible changes:

]]]

Hmmm, write-changelog should be improved to "recurse" into the merged
revision / group to get those revision's changelog lines instead, or
to parse it from the subsequent lines of the merge commit itself (as
it was filled in by svn-role which got the info when processing
STATUS).

-- 
Johan

Re: Rolling 1.10.0-rc2

Posted by Johan Corveleyn <jc...@gmail.com>.
On Wed, Mar 14, 2018 at 1:06 PM, Johan Corveleyn <jc...@gmail.com> wrote:
> On Wed, Mar 14, 2018 at 12:22 PM, Branko Čibej <br...@apache.org> wrote:
>> On 14.03.2018 08:58, Julian Foad wrote:
> ...
>>> If anyone's willing to write a brief list of the changes since RC1, to
>>> include in the RC2 announcement, that would be an added bonus.
>>
>> svn log -v STATUS | grep svn-role. But that wouldn't really work, would
>> it. :)
>
> Maybe try:
>
> release.py write-changelog --include-unlabeled-summaries
> branches/1.10.x tags/1.10.0-rc1

(the above all on one line of course ... stupid automatic wrapping)

-- 
Johan

Re: Rolling 1.10.0-rc2

Posted by Johan Corveleyn <jc...@gmail.com>.
On Wed, Mar 14, 2018 at 12:22 PM, Branko Čibej <br...@apache.org> wrote:
> On 14.03.2018 08:58, Julian Foad wrote:
...
>> If anyone's willing to write a brief list of the changes since RC1, to
>> include in the RC2 announcement, that would be an added bonus.
>
> svn log -v STATUS | grep svn-role. But that wouldn't really work, would
> it. :)

Maybe try:

release.py write-changelog --include-unlabeled-summaries
branches/1.10.x tags/1.10.0-rc1

(not at my "svn pc" right now, so can't try it myself)

-- 
Johan

Re: Rolling 1.10.0-rc2

Posted by Branko Čibej <br...@apache.org>.
On 14.03.2018 08:58, Julian Foad wrote:
> Julian Foad wrote on 2018-02-28:
>> I'm happy to announce the release of Apache Subversion 1.10.0-rc1.
>
> That was 2 weeks ago and we need an RC2.
>
> I plan to roll 1.10.0-rc2 this week so we can release it by next
> Wednesday (3 weeks from RC1).
>
> Please review the 'STATUS' file and report here if there's anything
> else I should take into account.

All three entries currently there should be part of RC2 (and the final
release). I'll try to review them tonight, as they each only need one
more signature.


> If anyone's willing to write a brief list of the changes since RC1, to
> include in the RC2 announcement, that would be an added bonus.

svn log -v STATUS | grep svn-role. But that wouldn't really work, would
it. :)

-- Brane


Re: Rolling 1.10.0-rc2

Posted by Evgeny Kotkov <ev...@visualsvn.com>.
Julian Foad <ju...@apache.org> writes:

> Please review the 'STATUS' file and report here if there's anything else I
> should take into account.

I have found a bug (my fault) in the new svn_txdelta_to_svndiff_stream()
API that can result in failing commits over http://.

Should be fixed by https://svn.apache.org/r1826747 and proposed
for backport to 1.10.x.


Thanks,
Evgeny Kotkov