You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by Ivan Zhakov <iv...@visualsvn.com> on 2014/04/02 12:13:11 UTC
Re: [RFC/PATCH] svnadmin: recover/hotcopy erroring out for old FSFS repositories
On 31 January 2014 14:57, Ivan Zhakov <iv...@visualsvn.com> wrote:
> On 31 January 2014 05:50, Evgeny Kotkov <ev...@visualsvn.com> wrote:
>>> This only affects non-sharded repositories with rev-local IDs,
>>> i.e. those in SVN 1.4 format. For those, it is writing 3 files
>>> instead of 2 per rev now.
>>
>> I assume you are talking about r1560723 [1]. If I read the code correctly...
>> [[[
>> if ( (!max_files_per_dir || rev % max_files_per_dir == 0)
>> && dst_ffd->format >= SVN_FS_FS__MIN_NO_GLOBAL_IDS_FORMAT)
>> SVN_ERR(hotcopy_update_current(&dst_youngest, dst_fs, rev, iterpool));
>> ]]]
>>
>> ...hotcopy now checkpoints after every revision for all non-sharded repositories
>> with FSFS format >= 3. So, checkpointing happens for all FSFS format 1/2
>> repositories upgraded via 'svnadmin upgrade' (with linear layout), that were
>> not fsfs-reshard'ed or dump/loaded into a repository with newer format.
>>
>> I am not aware of how many people reshard or dump/load their old repositories
>> after upgrade, but I did a quick benchmark for a real-world repository and on my
>> machine it shows 7x performance degradation with checkpointing enabled. Is it
>> worth the ability to re-run the backup from a checkpoint upon cancellation?
>>
>> [[[
>> # svnrdump http://googletest.googlecode.com/svn/trunk/
>> # load the dump into a --compatible-version=1.3 repository
>> # upgrade the repository with the most recent svnadmin
>>
>> # disable checkpointing, benchmark making 100 hotcopy backups:
>> real 0m18.741s
>> user 0m2.552s
>> sys 0m13.432s
>>
>> # now enable checkpointing and repeat the benchmark:
>> real 2m5.793s
>> user 0m0.836s
>> sys 0m34.840s
>> ]]]
>>
>> [1] http://svn.apache.org/viewvc?view=revision&revision=r1560723
>>
> That's what I suspected. In this case I think r1560723 should not be
> backported to 1.8.x for the following reasons:
> 1. Significant performance degradation for 'svnadmin hotcopy' with
> '--incremental' flag.
> 2. The change doesn't fix original circular dependency problem in FSFS
> 3. 'svnadmin recover' writes fake value to CURRENT file which very bad
> practice IMHO and could lead undefined consequences.
>
Hi Stefan,
Is this problem fixed or it's better to revert r1560723 for reasons I
stated above?
--
Ivan Zhakov
Re: [RFC/PATCH] svnadmin: recover/hotcopy erroring out for old FSFS repositories
Posted by Ivan Zhakov <iv...@visualsvn.com>.
On 7 April 2014 20:38, Stefan Fuhrmann <st...@wandisco.com> wrote:
> On Sun, Apr 6, 2014 at 5:36 PM, Ivan Zhakov <iv...@visualsvn.com> wrote:
>>
>> On 6 April 2014 18:31, Stefan Fuhrmann <st...@wandisco.com>
>> wrote:
>> > On Wed, Apr 2, 2014 at 12:13 PM, Ivan Zhakov <iv...@visualsvn.com> wrote:
>> >>
>> >> On 31 January 2014 14:57, Ivan Zhakov <iv...@visualsvn.com> wrote:
>> >> > On 31 January 2014 05:50, Evgeny Kotkov <ev...@visualsvn.com>
>> >> > wrote:
>> >> >>> This only affects non-sharded repositories with rev-local IDs,
>> >> >>> i.e. those in SVN 1.4 format. For those, it is writing 3 files
>> >> >>> instead of 2 per rev now.
>> >> >>
>> >> >> I assume you are talking about r1560723 [1]. If I read the code
>> >> >> correctly...
>> >> >> [[[
>> >> >> if ( (!max_files_per_dir || rev % max_files_per_dir == 0)
>> >> >> && dst_ffd->format >= SVN_FS_FS__MIN_NO_GLOBAL_IDS_FORMAT)
>> >> >> SVN_ERR(hotcopy_update_current(&dst_youngest, dst_fs, rev,
>> >> >> iterpool));
>> >> >> ]]]
>> >> >>
>> >> >> ...hotcopy now checkpoints after every revision for all non-sharded
>> >> >> repositories
>> >> >> with FSFS format >= 3. So, checkpointing happens for all FSFS
>> >> >> format
>> >> >> 1/2
>> >> >> repositories upgraded via 'svnadmin upgrade' (with linear layout),
>> >> >> that
>> >> >> were
>> >> >> not fsfs-reshard'ed or dump/loaded into a repository with newer
>> >> >> format.
>> >> >>
>> >> >> I am not aware of how many people reshard or dump/load their old
>> >> >> repositories
>> >> >> after upgrade, but I did a quick benchmark for a real-world
>> >> >> repository
>> >> >> and on my
>> >> >> machine it shows 7x performance degradation with checkpointing
>> >> >> enabled.
>> >> >> Is it
>> >> >> worth the ability to re-run the backup from a checkpoint upon
>> >> >> cancellation?
>> >> >>
>> >> >> [[[
>> >> >> # svnrdump http://googletest.googlecode.com/svn/trunk/
>> >> >> # load the dump into a --compatible-version=1.3 repository
>> >> >> # upgrade the repository with the most recent svnadmin
>> >> >>
>> >> >> # disable checkpointing, benchmark making 100 hotcopy backups:
>> >> >> real 0m18.741s
>> >> >> user 0m2.552s
>> >> >> sys 0m13.432s
>> >> >>
>> >> >> # now enable checkpointing and repeat the benchmark:
>> >> >> real 2m5.793s
>> >> >> user 0m0.836s
>> >> >> sys 0m34.840s
>> >> >> ]]]
>> >> >>
>> >> >> [1] http://svn.apache.org/viewvc?view=revision&revision=r1560723
>> >> >>
>> >> > That's what I suspected. In this case I think r1560723 should not be
>> >> > backported to 1.8.x for the following reasons:
>> >> > 1. Significant performance degradation for 'svnadmin hotcopy' with
>> >> > '--incremental' flag.
>> >> > 2. The change doesn't fix original circular dependency problem in
>> >> > FSFS
>> >> > 3. 'svnadmin recover' writes fake value to CURRENT file which very
>> >> > bad
>> >> > practice IMHO and could lead undefined consequences.
>> >> >
>> >> Hi Stefan,
>> >>
>> >> Is this problem fixed or it's better to revert r1560723 for reasons I
>> >> stated above?
>> >
>> >
>> > Hi Ivan,
>> >
>> > Evgeny applied his patch a month ago as r1573744.
>> >
>> Hi Stefan,
>>
>> Yes, I know. But svnadmin hotcopy performance regression was not fixed
>> as far I know.
>
>
> Oh - sorry!
> When I read your post, I didn't expand the quote.
>
> I don't have string feelings either way. As it makes
> laggards happy, I'm fine with the revert.
>
> Reverted in r1585516.
>
Thanks a lot!
--
Ivan Zhakov
Re: [RFC/PATCH] svnadmin: recover/hotcopy erroring out for old FSFS repositories
Posted by Stefan Fuhrmann <st...@wandisco.com>.
On Sun, Apr 6, 2014 at 5:36 PM, Ivan Zhakov <iv...@visualsvn.com> wrote:
> On 6 April 2014 18:31, Stefan Fuhrmann <st...@wandisco.com>
> wrote:
> > On Wed, Apr 2, 2014 at 12:13 PM, Ivan Zhakov <iv...@visualsvn.com> wrote:
> >>
> >> On 31 January 2014 14:57, Ivan Zhakov <iv...@visualsvn.com> wrote:
> >> > On 31 January 2014 05:50, Evgeny Kotkov <ev...@visualsvn.com>
> >> > wrote:
> >> >>> This only affects non-sharded repositories with rev-local IDs,
> >> >>> i.e. those in SVN 1.4 format. For those, it is writing 3 files
> >> >>> instead of 2 per rev now.
> >> >>
> >> >> I assume you are talking about r1560723 [1]. If I read the code
> >> >> correctly...
> >> >> [[[
> >> >> if ( (!max_files_per_dir || rev % max_files_per_dir == 0)
> >> >> && dst_ffd->format >= SVN_FS_FS__MIN_NO_GLOBAL_IDS_FORMAT)
> >> >> SVN_ERR(hotcopy_update_current(&dst_youngest, dst_fs, rev,
> >> >> iterpool));
> >> >> ]]]
> >> >>
> >> >> ...hotcopy now checkpoints after every revision for all non-sharded
> >> >> repositories
> >> >> with FSFS format >= 3. So, checkpointing happens for all FSFS format
> >> >> 1/2
> >> >> repositories upgraded via 'svnadmin upgrade' (with linear layout),
> that
> >> >> were
> >> >> not fsfs-reshard'ed or dump/loaded into a repository with newer
> format.
> >> >>
> >> >> I am not aware of how many people reshard or dump/load their old
> >> >> repositories
> >> >> after upgrade, but I did a quick benchmark for a real-world
> repository
> >> >> and on my
> >> >> machine it shows 7x performance degradation with checkpointing
> enabled.
> >> >> Is it
> >> >> worth the ability to re-run the backup from a checkpoint upon
> >> >> cancellation?
> >> >>
> >> >> [[[
> >> >> # svnrdump http://googletest.googlecode.com/svn/trunk/
> >> >> # load the dump into a --compatible-version=1.3 repository
> >> >> # upgrade the repository with the most recent svnadmin
> >> >>
> >> >> # disable checkpointing, benchmark making 100 hotcopy backups:
> >> >> real 0m18.741s
> >> >> user 0m2.552s
> >> >> sys 0m13.432s
> >> >>
> >> >> # now enable checkpointing and repeat the benchmark:
> >> >> real 2m5.793s
> >> >> user 0m0.836s
> >> >> sys 0m34.840s
> >> >> ]]]
> >> >>
> >> >> [1] http://svn.apache.org/viewvc?view=revision&revision=r1560723
> >> >>
> >> > That's what I suspected. In this case I think r1560723 should not be
> >> > backported to 1.8.x for the following reasons:
> >> > 1. Significant performance degradation for 'svnadmin hotcopy' with
> >> > '--incremental' flag.
> >> > 2. The change doesn't fix original circular dependency problem in FSFS
> >> > 3. 'svnadmin recover' writes fake value to CURRENT file which very bad
> >> > practice IMHO and could lead undefined consequences.
> >> >
> >> Hi Stefan,
> >>
> >> Is this problem fixed or it's better to revert r1560723 for reasons I
> >> stated above?
> >
> >
> > Hi Ivan,
> >
> > Evgeny applied his patch a month ago as r1573744.
> >
> Hi Stefan,
>
> Yes, I know. But svnadmin hotcopy performance regression was not fixed
> as far I know.
>
Oh - sorry!
When I read your post, I didn't expand the quote.
I don't have string feelings either way. As it makes
laggards happy, I'm fine with the revert.
Reverted in r1585516.
-- Stefan^2.
Re: [RFC/PATCH] svnadmin: recover/hotcopy erroring out for old FSFS repositories
Posted by Ivan Zhakov <iv...@visualsvn.com>.
On 6 April 2014 18:31, Stefan Fuhrmann <st...@wandisco.com> wrote:
> On Wed, Apr 2, 2014 at 12:13 PM, Ivan Zhakov <iv...@visualsvn.com> wrote:
>>
>> On 31 January 2014 14:57, Ivan Zhakov <iv...@visualsvn.com> wrote:
>> > On 31 January 2014 05:50, Evgeny Kotkov <ev...@visualsvn.com>
>> > wrote:
>> >>> This only affects non-sharded repositories with rev-local IDs,
>> >>> i.e. those in SVN 1.4 format. For those, it is writing 3 files
>> >>> instead of 2 per rev now.
>> >>
>> >> I assume you are talking about r1560723 [1]. If I read the code
>> >> correctly...
>> >> [[[
>> >> if ( (!max_files_per_dir || rev % max_files_per_dir == 0)
>> >> && dst_ffd->format >= SVN_FS_FS__MIN_NO_GLOBAL_IDS_FORMAT)
>> >> SVN_ERR(hotcopy_update_current(&dst_youngest, dst_fs, rev,
>> >> iterpool));
>> >> ]]]
>> >>
>> >> ...hotcopy now checkpoints after every revision for all non-sharded
>> >> repositories
>> >> with FSFS format >= 3. So, checkpointing happens for all FSFS format
>> >> 1/2
>> >> repositories upgraded via 'svnadmin upgrade' (with linear layout), that
>> >> were
>> >> not fsfs-reshard'ed or dump/loaded into a repository with newer format.
>> >>
>> >> I am not aware of how many people reshard or dump/load their old
>> >> repositories
>> >> after upgrade, but I did a quick benchmark for a real-world repository
>> >> and on my
>> >> machine it shows 7x performance degradation with checkpointing enabled.
>> >> Is it
>> >> worth the ability to re-run the backup from a checkpoint upon
>> >> cancellation?
>> >>
>> >> [[[
>> >> # svnrdump http://googletest.googlecode.com/svn/trunk/
>> >> # load the dump into a --compatible-version=1.3 repository
>> >> # upgrade the repository with the most recent svnadmin
>> >>
>> >> # disable checkpointing, benchmark making 100 hotcopy backups:
>> >> real 0m18.741s
>> >> user 0m2.552s
>> >> sys 0m13.432s
>> >>
>> >> # now enable checkpointing and repeat the benchmark:
>> >> real 2m5.793s
>> >> user 0m0.836s
>> >> sys 0m34.840s
>> >> ]]]
>> >>
>> >> [1] http://svn.apache.org/viewvc?view=revision&revision=r1560723
>> >>
>> > That's what I suspected. In this case I think r1560723 should not be
>> > backported to 1.8.x for the following reasons:
>> > 1. Significant performance degradation for 'svnadmin hotcopy' with
>> > '--incremental' flag.
>> > 2. The change doesn't fix original circular dependency problem in FSFS
>> > 3. 'svnadmin recover' writes fake value to CURRENT file which very bad
>> > practice IMHO and could lead undefined consequences.
>> >
>> Hi Stefan,
>>
>> Is this problem fixed or it's better to revert r1560723 for reasons I
>> stated above?
>
>
> Hi Ivan,
>
> Evgeny applied his patch a month ago as r1573744.
>
Hi Stefan,
Yes, I know. But svnadmin hotcopy performance regression was not fixed
as far I know.
--
Ivan Zhakov
CTO | VisualSVN | http://www.visualsvn.com
Re: [RFC/PATCH] svnadmin: recover/hotcopy erroring out for old FSFS repositories
Posted by Stefan Fuhrmann <st...@wandisco.com>.
On Wed, Apr 2, 2014 at 12:13 PM, Ivan Zhakov <iv...@visualsvn.com> wrote:
> On 31 January 2014 14:57, Ivan Zhakov <iv...@visualsvn.com> wrote:
> > On 31 January 2014 05:50, Evgeny Kotkov <ev...@visualsvn.com>
> wrote:
> >>> This only affects non-sharded repositories with rev-local IDs,
> >>> i.e. those in SVN 1.4 format. For those, it is writing 3 files
> >>> instead of 2 per rev now.
> >>
> >> I assume you are talking about r1560723 [1]. If I read the code
> correctly...
> >> [[[
> >> if ( (!max_files_per_dir || rev % max_files_per_dir == 0)
> >> && dst_ffd->format >= SVN_FS_FS__MIN_NO_GLOBAL_IDS_FORMAT)
> >> SVN_ERR(hotcopy_update_current(&dst_youngest, dst_fs, rev,
> iterpool));
> >> ]]]
> >>
> >> ...hotcopy now checkpoints after every revision for all non-sharded
> repositories
> >> with FSFS format >= 3. So, checkpointing happens for all FSFS format
> 1/2
> >> repositories upgraded via 'svnadmin upgrade' (with linear layout), that
> were
> >> not fsfs-reshard'ed or dump/loaded into a repository with newer format.
> >>
> >> I am not aware of how many people reshard or dump/load their old
> repositories
> >> after upgrade, but I did a quick benchmark for a real-world repository
> and on my
> >> machine it shows 7x performance degradation with checkpointing enabled.
> Is it
> >> worth the ability to re-run the backup from a checkpoint upon
> cancellation?
> >>
> >> [[[
> >> # svnrdump http://googletest.googlecode.com/svn/trunk/
> >> # load the dump into a --compatible-version=1.3 repository
> >> # upgrade the repository with the most recent svnadmin
> >>
> >> # disable checkpointing, benchmark making 100 hotcopy backups:
> >> real 0m18.741s
> >> user 0m2.552s
> >> sys 0m13.432s
> >>
> >> # now enable checkpointing and repeat the benchmark:
> >> real 2m5.793s
> >> user 0m0.836s
> >> sys 0m34.840s
> >> ]]]
> >>
> >> [1] http://svn.apache.org/viewvc?view=revision&revision=r1560723
> >>
> > That's what I suspected. In this case I think r1560723 should not be
> > backported to 1.8.x for the following reasons:
> > 1. Significant performance degradation for 'svnadmin hotcopy' with
> > '--incremental' flag.
> > 2. The change doesn't fix original circular dependency problem in FSFS
> > 3. 'svnadmin recover' writes fake value to CURRENT file which very bad
> > practice IMHO and could lead undefined consequences.
> >
> Hi Stefan,
>
> Is this problem fixed or it's better to revert r1560723 for reasons I
> stated above?
>
Hi Ivan,
Evgeny applied his patch a month ago as r1573744.
-- Stefan^2.