You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@subversion.apache.org by Attila Nagy <br...@fsn.hu> on 2011/11/16 15:49:12 UTC
Re: Assertion failed and crash with 1.7.1
On 10/27/11 14:57, Mark Phippard wrote:
> On Thu, Oct 27, 2011 at 8:12 AM, Attila Nagy <bra@fsn.hu
> <ma...@fsn.hu>> wrote:
>
> ZFS.
> It it worth to make benchmarks with this WC with 1.6 and 1.7? I so,
> I can try to find the time for it.
>
>
> There are some pretty easy to run benchmarks you can run to see if 1.7
> is simply slower in your environment. See:
>
> https://ctf.open.collab.net/sf/projects/csvn/
>
> If those benchmarks show 1.7 is slower for you, then that would point to
> issues with the 1.7 changes in your environment. As I mentioned
> earlier, and as an example, these benchmarks show that 1.7 is slower
> with WC on a network mount.
>
> If the benchmarks show 1.7 is faster, then we could focus on scalability
> issues that your working copy raises. Maybe inefficient indexes or
> something like that.
It turned out that not updating which is much slower (however I can't
really comment on the exact numbers now), but the propgets.
To understand this: as I've mentioned, I store operating system images
in svn, so I have to reproduce owner and permission information too.
Therefore I store them in svn properties and re-create them as needed on
updates.
I use pysvn for this and basically the code looks like this (in python):
def update_perms():
for path in propchg:
proplist = svn.propget('file:permissions', path)
if not os.path.islink(path) and proplist.has_key(path):
set_perms(path, proplist[path])
svn.update(walkroot)
update_perms()
The svn update collects the changed entries (propchg) and update_perms
iterates on them and gets their file:permissions property and sets it in
the file system.
And this is what takes ages (literally), compared to 1.6.
Any ideas about what could be done in this topic?
I will try to make more specific numbers if needed.
Re: Assertion failed and crash with 1.7.1
Posted by Andreas Krey <a....@gmx.de>.
On Wed, 16 Nov 2011 17:40:55 +0000, Philip Martin wrote:
...
> It might be faster to run a recursive propget,
It might also be erroneous. I noticed in an 1.5/1.6 setup that
a recursive propget over a big subtree simply fails to yield *all*
relevant properties. That is, a
svn pg -R svn:externals .
does *not* report some of the externals that
svn pg -R svn:externals some/sub/path
reports.
It being a large (and proprietary) repo and only occurring when recursing
over big parts of it, I didn't go to try to reproduce it for a bug report.
Andreas
--
"Totally trivial. Famous last words."
From: Linus Torvalds <torvalds@*.org>
Date: Fri, 22 Jan 2010 07:29:21 -0800
Re: Assertion failed and crash with 1.7.1
Posted by Johan Corveleyn <jc...@gmail.com>.
On Thu, Nov 17, 2011 at 8:02 PM, Daniel Shahaf <d....@daniel.shahaf.name> wrote:
> Johan Corveleyn wrote on Thu, Nov 17, 2011 at 13:55:51 +0100:
>> On Thu, Nov 17, 2011 at 1:02 PM, Philip Martin
>> <ph...@wandisco.com> wrote:
>> > Attila Nagy <br...@fsn.hu> writes:
>> >
>> >> On 11/16/11 18:40, Philip Martin wrote:
>> >>> Attila Nagy<br...@fsn.hu> writes:
>> >>>
>> >>>> I use pysvn for this and basically the code looks like this (in python):
>> >>>> def update_perms():
>> >>>> for path in propchg:
>> >>>> proplist = svn.propget('file:permissions', path)
>> >>>> if not os.path.islink(path) and proplist.has_key(path):
>> >>>> set_perms(path, proplist[path])
>> >>>> svn.update(walkroot)
>> >>>> update_perms()
>> >>>>
>> >>>> The svn update collects the changed entries (propchg) and update_perms
>> >>>> iterates on them and gets their file:permissions property and sets it
>> >>>> in the file system.
>> >>>>
>> >>>> And this is what takes ages (literally), compared to 1.6.
>> >>>> Any ideas about what could be done in this topic?
>> >>>
>> >>> It might be faster to run a recursive propget, which is a single
>> >>> transaction, and discard the output if it doesn't match one of the
>> >>> changed paths.
>> >>>
>> >> I will try this. Should this be true even for 10+ million files?
>> >
>> > It depends on the ratio of changed files to total files. If there is
>> > only one changed file then the single propget will be faster. If most
>> > of the files are changed then the recursive propget will be faster.
>>
>> Yes, you'll need to test that a bit.
>>
>> I do something similar here with a script that fetches all the log
>> entries for merged revisions (cherrypick merges, which I look up with
>> 'svn mergeinfo --show-revs merged'): if the number of merged revisions
>> is low relative to their range, I perform a series of individual 'svn
>> log' requests. Otherwise, I do a single 'svn log -r<min>:<max>'
>> request, parse the output and discard the entries that are not
>> relevant. This made my script much faster in most cases.
>>
>> This has nothing to do with recursive propget (or even with 1.7, I'm
>> using this in a 1.5 environment), but I'm just noting the similarity
>> of the problem here.
>
> svn log -r N -r M -r P
>
> was implemented a few years ago...
Indeed. But that was 1.6+, and I still have to support some 1.5
clients with this script. So unfortunately I had to write this
workaround :-).
But thanks for pointing this out, even if only for completeness of the
archives :-).
--
Johan
Re: Assertion failed and crash with 1.7.1
Posted by Daniel Shahaf <d....@daniel.shahaf.name>.
Johan Corveleyn wrote on Thu, Nov 17, 2011 at 13:55:51 +0100:
> On Thu, Nov 17, 2011 at 1:02 PM, Philip Martin
> <ph...@wandisco.com> wrote:
> > Attila Nagy <br...@fsn.hu> writes:
> >
> >> On 11/16/11 18:40, Philip Martin wrote:
> >>> Attila Nagy<br...@fsn.hu> writes:
> >>>
> >>>> I use pysvn for this and basically the code looks like this (in python):
> >>>> def update_perms():
> >>>> for path in propchg:
> >>>> proplist = svn.propget('file:permissions', path)
> >>>> if not os.path.islink(path) and proplist.has_key(path):
> >>>> set_perms(path, proplist[path])
> >>>> svn.update(walkroot)
> >>>> update_perms()
> >>>>
> >>>> The svn update collects the changed entries (propchg) and update_perms
> >>>> iterates on them and gets their file:permissions property and sets it
> >>>> in the file system.
> >>>>
> >>>> And this is what takes ages (literally), compared to 1.6.
> >>>> Any ideas about what could be done in this topic?
> >>>
> >>> It might be faster to run a recursive propget, which is a single
> >>> transaction, and discard the output if it doesn't match one of the
> >>> changed paths.
> >>>
> >> I will try this. Should this be true even for 10+ million files?
> >
> > It depends on the ratio of changed files to total files. If there is
> > only one changed file then the single propget will be faster. If most
> > of the files are changed then the recursive propget will be faster.
>
> Yes, you'll need to test that a bit.
>
> I do something similar here with a script that fetches all the log
> entries for merged revisions (cherrypick merges, which I look up with
> 'svn mergeinfo --show-revs merged'): if the number of merged revisions
> is low relative to their range, I perform a series of individual 'svn
> log' requests. Otherwise, I do a single 'svn log -r<min>:<max>'
> request, parse the output and discard the entries that are not
> relevant. This made my script much faster in most cases.
>
> This has nothing to do with recursive propget (or even with 1.7, I'm
> using this in a 1.5 environment), but I'm just noting the similarity
> of the problem here.
svn log -r N -r M -r P
was implemented a few years ago...
> --
> Johan
Re: Assertion failed and crash with 1.7.1
Posted by Johan Corveleyn <jc...@gmail.com>.
On Thu, Nov 17, 2011 at 1:02 PM, Philip Martin
<ph...@wandisco.com> wrote:
> Attila Nagy <br...@fsn.hu> writes:
>
>> On 11/16/11 18:40, Philip Martin wrote:
>>> Attila Nagy<br...@fsn.hu> writes:
>>>
>>>> I use pysvn for this and basically the code looks like this (in python):
>>>> def update_perms():
>>>> for path in propchg:
>>>> proplist = svn.propget('file:permissions', path)
>>>> if not os.path.islink(path) and proplist.has_key(path):
>>>> set_perms(path, proplist[path])
>>>> svn.update(walkroot)
>>>> update_perms()
>>>>
>>>> The svn update collects the changed entries (propchg) and update_perms
>>>> iterates on them and gets their file:permissions property and sets it
>>>> in the file system.
>>>>
>>>> And this is what takes ages (literally), compared to 1.6.
>>>> Any ideas about what could be done in this topic?
>>>
>>> It might be faster to run a recursive propget, which is a single
>>> transaction, and discard the output if it doesn't match one of the
>>> changed paths.
>>>
>> I will try this. Should this be true even for 10+ million files?
>
> It depends on the ratio of changed files to total files. If there is
> only one changed file then the single propget will be faster. If most
> of the files are changed then the recursive propget will be faster.
Yes, you'll need to test that a bit.
I do something similar here with a script that fetches all the log
entries for merged revisions (cherrypick merges, which I look up with
'svn mergeinfo --show-revs merged'): if the number of merged revisions
is low relative to their range, I perform a series of individual 'svn
log' requests. Otherwise, I do a single 'svn log -r<min>:<max>'
request, parse the output and discard the entries that are not
relevant. This made my script much faster in most cases.
This has nothing to do with recursive propget (or even with 1.7, I'm
using this in a 1.5 environment), but I'm just noting the similarity
of the problem here.
But for your propget problem, there may also be another option: 'svn
propget' can take multiple targets:
usage: 1. propget PROPNAME [TARGET[@REV]...]
So if you collect the targets in a list, you can get the exact
information you want with a single propget.
That said, I'm not sure that 'svn propget file:permissions A B C' does
all its work in a single transaction, rather than using a transaction
per target (in which case this technique won't buy you much). Maybe
you can quickly verify this if it makes any performance impact ...
--
Johan
Re: Assertion failed and crash with 1.7.1
Posted by Philip Martin <ph...@wandisco.com>.
Attila Nagy <br...@fsn.hu> writes:
> On 11/16/11 18:40, Philip Martin wrote:
>> Attila Nagy<br...@fsn.hu> writes:
>>
>>> I use pysvn for this and basically the code looks like this (in python):
>>> def update_perms():
>>> for path in propchg:
>>> proplist = svn.propget('file:permissions', path)
>>> if not os.path.islink(path) and proplist.has_key(path):
>>> set_perms(path, proplist[path])
>>> svn.update(walkroot)
>>> update_perms()
>>>
>>> The svn update collects the changed entries (propchg) and update_perms
>>> iterates on them and gets their file:permissions property and sets it
>>> in the file system.
>>>
>>> And this is what takes ages (literally), compared to 1.6.
>>> Any ideas about what could be done in this topic?
>>
>> It might be faster to run a recursive propget, which is a single
>> transaction, and discard the output if it doesn't match one of the
>> changed paths.
>>
> I will try this. Should this be true even for 10+ million files?
It depends on the ratio of changed files to total files. If there is
only one changed file then the single propget will be faster. If most
of the files are changed then the recursive propget will be faster.
--
Philip
Re: Assertion failed and crash with 1.7.1
Posted by Attila Nagy <br...@fsn.hu>.
On 11/16/11 18:40, Philip Martin wrote:
> Attila Nagy<br...@fsn.hu> writes:
>
>> I use pysvn for this and basically the code looks like this (in python):
>> def update_perms():
>> for path in propchg:
>> proplist = svn.propget('file:permissions', path)
>> if not os.path.islink(path) and proplist.has_key(path):
>> set_perms(path, proplist[path])
>> svn.update(walkroot)
>> update_perms()
>>
>> The svn update collects the changed entries (propchg) and update_perms
>> iterates on them and gets their file:permissions property and sets it
>> in the file system.
>>
>> And this is what takes ages (literally), compared to 1.6.
>> Any ideas about what could be done in this topic?
>
> It might be faster to run a recursive propget, which is a single
> transaction, and discard the output if it doesn't match one of the
> changed paths.
>
I will try this. Should this be true even for 10+ million files? I don't
want to think about the memory usage... :)
Re: Assertion failed and crash with 1.7.1
Posted by Philip Martin <ph...@wandisco.com>.
Attila Nagy <br...@fsn.hu> writes:
> I use pysvn for this and basically the code looks like this (in python):
> def update_perms():
> for path in propchg:
> proplist = svn.propget('file:permissions', path)
> if not os.path.islink(path) and proplist.has_key(path):
> set_perms(path, proplist[path])
> svn.update(walkroot)
> update_perms()
>
> The svn update collects the changed entries (propchg) and update_perms
> iterates on them and gets their file:permissions property and sets it
> in the file system.
>
> And this is what takes ages (literally), compared to 1.6.
> Any ideas about what could be done in this topic?
It might be faster to run a recursive propget, which is a single
transaction, and discard the output if it doesn't match one of the
changed paths.
--
Philip