You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@subversion.apache.org by Attila Nagy <br...@fsn.hu> on 2011/10/26 11:01:40 UTC
Assertion failed and crash with 1.7.1
Hi,
I'm trying to update a working copy of some tens of GBs with svn 1.7.1
(upgraded the WC to the new -sadly much slower- format) and always get this:
$ svn up
Updating '.':
Skipped 'mail/20110624/usr/local/etc/blacklist' -- Node remains in conflict
svn: E235000: In file 'subversion/libsvn_wc/update_editor.c' line 1582:
assertion failed (action == svn_wc_conflict_action_edit || action ==
svn_wc_conflict_action_delete || action == svn_wc_conflict_action_replace)
Abort (core dumped)
If I delete that file, I can get past it, but receive another crash, now
without any clues about what should I remove:
$ svn up
Updating '.':
svn: E235000: In file 'subversion/libsvn_wc/update_editor.c' line 1582:
assertion failed (action == svn_wc_conflict_action_edit || action ==
svn_wc_conflict_action_delete || action == svn_wc_conflict_action_replace)
Abort (core dumped)
Bug or feature?
Re: Assertion failed and crash with 1.7.1
Posted by Attila Nagy <br...@fsn.hu>.
On 10/26/11 15:31, Mark Phippard wrote:
> On Wed, Oct 26, 2011 at 5:01 AM, Attila Nagy<br...@fsn.hu> wrote:
>
>> I'm trying to update a working copy of some tens of GBs with svn 1.7.1
>> (upgraded the WC to the new -sadly much slower- format) and always get this:
> Is your WC on a network drive? That is the only scenario I have seen
> where 1.7.x was not significantly faster than previous releases.
>
> Sorry for the sidebar away from your problem.
>
No, it's on a midrange FC storage, backed by SSDs and 15k RPM FC drives.
It seems that svn spins mainly in SQLite operations. The database
(wc.db) is about 1.2 GBs in size, and fits completely in memory (32GB).
Pristines take about 15-17 GB, while the WC size is about 40-50 GBs with
a lot of files (with many duplicates).
Basically there is no disk IO during svn operations.
Is there a profiler for SQLite operations to see what takes so much time?
Thanks,
Re: Assertion failed and crash with 1.7.1
Posted by Mark Phippard <ma...@gmail.com>.
On Wed, Oct 26, 2011 at 5:01 AM, Attila Nagy <br...@fsn.hu> wrote:
> I'm trying to update a working copy of some tens of GBs with svn 1.7.1
> (upgraded the WC to the new -sadly much slower- format) and always get this:
Is your WC on a network drive? That is the only scenario I have seen
where 1.7.x was not significantly faster than previous releases.
Sorry for the sidebar away from your problem.
--
Thanks
Mark Phippard
http://markphip.blogspot.com/
AW: Assertion failed and crash with 1.7.1
Posted by Markus Schaber <m....@3s-software.com>.
Hi, Stefan,
Von: Stefan Sperling [mailto:stsp@elego.de]
> On Thu, Oct 27, 2011 at 09:53:11AM +0000, Markus Schaber wrote:
> > If you have uncommitted property modifications, copies or moves in the
> working copy, a lossless transition (or repair) of the WC to 1.7 should be
> possible in the following ways:
> >
>
> You meant to say "*no* uncommitted property modifications, copies, or
> moves", right?
Right. :-)
Re: Assertion failed and crash with 1.7.1
Posted by Stefan Sperling <st...@elego.de>.
On Thu, Oct 27, 2011 at 09:53:11AM +0000, Markus Schaber wrote:
> Hi,
>
> > Von: Attila Nagy [mailto:bra@fsn.hu]
> > BTW, with the current (1.7) WC-implementation, what possible problems do you see where this would be needed?
> > Previously with the per-directory .svn directories it was of course much easier, I could checkout the WC and move the .svn directory to its original place.
> > Now I (would) have to do this for all directories...
>
> If you have uncommitted property modifications, copies or moves in the working copy, a lossless transition (or repair) of the WC to 1.7 should be possible in the following ways:
>
You meant to say "*no* uncommitted property modifications, copies, or moves",
right?
But even then, in general, what you suggest isn't safe.
The broken working copy might be a mixed-revision working copy and
refer to pristine files from various revisions. In which case there
is no way a new checkout can replicate the needed DB state.
> 1) Preparation: Delete all .svn subdirectories in the working copy
>
> 2) create a separate checkout.
>
> 3) move or copy the .svn subdirectory of the new checkout into the top level directory of the existing working copy.
>
> Or:
>
> 1) Preparation: Delete all .svn subdirectories in the working copy
>
> 2) "svn checkout --force" in the existing working copy.
>
> I'm not sure whether this will work fine with externals, however. :-)
AW: Assertion failed and crash with 1.7.1
Posted by Markus Schaber <m....@3s-software.com>.
Hi,
> Von: Attila Nagy [mailto:bra@fsn.hu]
> BTW, with the current (1.7) WC-implementation, what possible problems do you see where this would be needed?
> Previously with the per-directory .svn directories it was of course much easier, I could checkout the WC and move the .svn directory to its original place.
> Now I (would) have to do this for all directories...
If you have uncommitted property modifications, copies or moves in the working copy, a lossless transition (or repair) of the WC to 1.7 should be possible in the following ways:
1) Preparation: Delete all .svn subdirectories in the working copy
2) create a separate checkout.
3) move or copy the .svn subdirectory of the new checkout into the top level directory of the existing working copy.
Or:
1) Preparation: Delete all .svn subdirectories in the working copy
2) "svn checkout --force" in the existing working copy.
I'm not sure whether this will work fine with externals, however. :-)
Mit freundlichen Grüßen
Markus Schaber
--
___________________________
We software Automation.
3S-Smart Software Solutions GmbH
Markus Schaber | Entwicklung
Memminger Str. 151 | 87439 Kempten | Tel. +49-831-54031-0 | Fax +49-831-54031-50
Email: m.schaber@3s-software.com | Web: http://www.3s-software.com
CoDeSys Internet-Forum: http://forum.3s-software.com
Geschäftsführer: Dipl.Inf. Dieter Hess, Dipl.Inf. Manfred Werner | Handelsregister: Kempten HRB 6186 | USt-IDNr.: DE 167014915
Re: Assertion failed and crash with 1.7.1
Posted by Attila Nagy <br...@fsn.hu>.
On 10/27/11 11:28, Stefan Sperling wrote:
> On Thu, Oct 27, 2011 at 10:47:51AM +0200, Attila Nagy wrote:
>> On 10/26/11 15:37, Philip Martin wrote:
>>> Attila Nagy<br...@fsn.hu> writes:
>>>
>>>> I'm trying to update a working copy of some tens of GBs with svn 1.7.1
>>> Did you upgrade with 1.7.0 or 1.7.1?
>> I've upgraded the WC with 1.7.0, then switched to 1.7.1, which I'm
>> currently using.
>> The upgrade took nearly one week (I can't afford a clean checkout,
>> because I have to preserve the files' inode numbers),
> This is an interesting use case, but please keep in mind that this
> is not a normal use case for Subversion. Checkouts are considered
> disposable. There are scenarios where getting a new checkout is the
> best way to replace a broken working copy.
I hope I will never have to do that! :)
BTW, with the current (1.7) WC-implementation, what possible problems do
you see where this would be needed?
Previously with the per-directory .svn directories it was of course much
easier, I could checkout the WC and move the .svn directory to its
original place.
Now I (would) have to do this for all directories...
>
> You will have your reasons for using Subversion this way, but if you
> care about inode numbers you might want to look into tools that know
> more about internals of the filesystem you are using and preserve
> inode information.
There is no such a thing I guess. I don't care about inodes until they
don't change on existing files.
>> at the start
>> it was running very fast, and at the end of the week it could print
>> a new directory in every 8-10 seconds, while the svn processes CPU
>> usage was 100%.
> This sounds like a linear table scan. There might be room for
> optimisation by adding another index.
Maybe, but I don't know what to index, because it seems to be hard to
get running SQLite operations out of the system.
>
>>>> $ svn up
>>>> Updating '.':
>>>> svn: E235000: In file 'subversion/libsvn_wc/update_editor.c' line
>>>> 1582: assertion failed (action == svn_wc_conflict_action_edit ||
>>>> action == svn_wc_conflict_action_delete || action ==
>>>> svn_wc_conflict_action_replace)
>>>> Abort (core dumped)
>>> Can you show us a gdb stack trace?
>> I'm sorry, this is a production machine and this error caused a
>> major lockup in our business processes,
> I'm sorry to hear that.
>
> This assertion you ran into looks exactly like the one reported here:
> http://svn.haxx.se/dev/archive-2011-10/0305.shtml
>
> Is there a symlink in the working copy?
Sure, a lot of them. (I use full operating system copies)
> If so this is a known regression with updating symlinks. A fix was made
> and is scheduled for release in 1.7.2.
> The fix was committed in this revision:
> http://svn.apache.org/viewvc?view=revision&revision=r1186944
> If you apply that patch to your 1.7.1 installation the update
> should succeed.
I could finally resolve the problem with some deletes and svn resolve,
but that's great to hear!
>
>> so I've had to work around
>> it, and I lost the core file in the process.
>> BTW, there was one tree conflict somewhere, it seems that an svn
>> resolve solved the issue.
> The tree conflict was probably flagged because of the above bug.
> It is spurious and can be removed via 'svn resolve'.
Yes, this was the problem.
Thanks!
Re: Assertion failed and crash with 1.7.1
Posted by Stefan Sperling <st...@elego.de>.
On Thu, Oct 27, 2011 at 10:47:51AM +0200, Attila Nagy wrote:
> On 10/26/11 15:37, Philip Martin wrote:
> >Attila Nagy<br...@fsn.hu> writes:
> >
> >>I'm trying to update a working copy of some tens of GBs with svn 1.7.1
> >Did you upgrade with 1.7.0 or 1.7.1?
> I've upgraded the WC with 1.7.0, then switched to 1.7.1, which I'm
> currently using.
> The upgrade took nearly one week (I can't afford a clean checkout,
> because I have to preserve the files' inode numbers),
This is an interesting use case, but please keep in mind that this
is not a normal use case for Subversion. Checkouts are considered
disposable. There are scenarios where getting a new checkout is the
best way to replace a broken working copy.
You will have your reasons for using Subversion this way, but if you
care about inode numbers you might want to look into tools that know
more about internals of the filesystem you are using and preserve
inode information.
> at the start
> it was running very fast, and at the end of the week it could print
> a new directory in every 8-10 seconds, while the svn processes CPU
> usage was 100%.
This sounds like a linear table scan. There might be room for
optimisation by adding another index.
> >>$ svn up
> >>Updating '.':
> >>svn: E235000: In file 'subversion/libsvn_wc/update_editor.c' line
> >>1582: assertion failed (action == svn_wc_conflict_action_edit ||
> >>action == svn_wc_conflict_action_delete || action ==
> >>svn_wc_conflict_action_replace)
> >>Abort (core dumped)
> >Can you show us a gdb stack trace?
> I'm sorry, this is a production machine and this error caused a
> major lockup in our business processes,
I'm sorry to hear that.
This assertion you ran into looks exactly like the one reported here:
http://svn.haxx.se/dev/archive-2011-10/0305.shtml
Is there a symlink in the working copy?
If so this is a known regression with updating symlinks. A fix was made
and is scheduled for release in 1.7.2.
The fix was committed in this revision:
http://svn.apache.org/viewvc?view=revision&revision=r1186944
If you apply that patch to your 1.7.1 installation the update
should succeed.
> so I've had to work around
> it, and I lost the core file in the process.
> BTW, there was one tree conflict somewhere, it seems that an svn
> resolve solved the issue.
The tree conflict was probably flagged because of the above bug.
It is spurious and can be removed via 'svn resolve'.
Re: Assertion failed and crash with 1.7.1
Posted by Philip Martin <ph...@wandisco.com>.
Attila Nagy <br...@fsn.hu> writes:
> ZFS.
> It it worth to make benchmarks with this WC with 1.6 and 1.7? I so, I
> can try to find the time for it.
There are some reports that a mismatch between SQLite page size and ZFS
block size can cause performance problems and that better performance is
obtained when they match:
http://osdir.com/ml/sqlite-users/2010-10/msg00582.html
http://www.mail-archive.com/zfs-discuss@opensolaris.org/msg20395.html
If you build Subversion from source we could try changing the SQLite
page size.
--
Philip
Re: Assertion failed and crash with 1.7.1
Posted by Andreas Krey <a....@gmx.de>.
On Wed, 16 Nov 2011 17:40:55 +0000, Philip Martin wrote:
...
> It might be faster to run a recursive propget,
It might also be erroneous. I noticed in an 1.5/1.6 setup that
a recursive propget over a big subtree simply fails to yield *all*
relevant properties. That is, a
svn pg -R svn:externals .
does *not* report some of the externals that
svn pg -R svn:externals some/sub/path
reports.
It being a large (and proprietary) repo and only occurring when recursing
over big parts of it, I didn't go to try to reproduce it for a bug report.
Andreas
--
"Totally trivial. Famous last words."
From: Linus Torvalds <torvalds@*.org>
Date: Fri, 22 Jan 2010 07:29:21 -0800
Re: Assertion failed and crash with 1.7.1
Posted by Johan Corveleyn <jc...@gmail.com>.
On Thu, Nov 17, 2011 at 8:02 PM, Daniel Shahaf <d....@daniel.shahaf.name> wrote:
> Johan Corveleyn wrote on Thu, Nov 17, 2011 at 13:55:51 +0100:
>> On Thu, Nov 17, 2011 at 1:02 PM, Philip Martin
>> <ph...@wandisco.com> wrote:
>> > Attila Nagy <br...@fsn.hu> writes:
>> >
>> >> On 11/16/11 18:40, Philip Martin wrote:
>> >>> Attila Nagy<br...@fsn.hu> writes:
>> >>>
>> >>>> I use pysvn for this and basically the code looks like this (in python):
>> >>>> def update_perms():
>> >>>> for path in propchg:
>> >>>> proplist = svn.propget('file:permissions', path)
>> >>>> if not os.path.islink(path) and proplist.has_key(path):
>> >>>> set_perms(path, proplist[path])
>> >>>> svn.update(walkroot)
>> >>>> update_perms()
>> >>>>
>> >>>> The svn update collects the changed entries (propchg) and update_perms
>> >>>> iterates on them and gets their file:permissions property and sets it
>> >>>> in the file system.
>> >>>>
>> >>>> And this is what takes ages (literally), compared to 1.6.
>> >>>> Any ideas about what could be done in this topic?
>> >>>
>> >>> It might be faster to run a recursive propget, which is a single
>> >>> transaction, and discard the output if it doesn't match one of the
>> >>> changed paths.
>> >>>
>> >> I will try this. Should this be true even for 10+ million files?
>> >
>> > It depends on the ratio of changed files to total files. If there is
>> > only one changed file then the single propget will be faster. If most
>> > of the files are changed then the recursive propget will be faster.
>>
>> Yes, you'll need to test that a bit.
>>
>> I do something similar here with a script that fetches all the log
>> entries for merged revisions (cherrypick merges, which I look up with
>> 'svn mergeinfo --show-revs merged'): if the number of merged revisions
>> is low relative to their range, I perform a series of individual 'svn
>> log' requests. Otherwise, I do a single 'svn log -r<min>:<max>'
>> request, parse the output and discard the entries that are not
>> relevant. This made my script much faster in most cases.
>>
>> This has nothing to do with recursive propget (or even with 1.7, I'm
>> using this in a 1.5 environment), but I'm just noting the similarity
>> of the problem here.
>
> svn log -r N -r M -r P
>
> was implemented a few years ago...
Indeed. But that was 1.6+, and I still have to support some 1.5
clients with this script. So unfortunately I had to write this
workaround :-).
But thanks for pointing this out, even if only for completeness of the
archives :-).
--
Johan
Re: Assertion failed and crash with 1.7.1
Posted by Daniel Shahaf <d....@daniel.shahaf.name>.
Johan Corveleyn wrote on Thu, Nov 17, 2011 at 13:55:51 +0100:
> On Thu, Nov 17, 2011 at 1:02 PM, Philip Martin
> <ph...@wandisco.com> wrote:
> > Attila Nagy <br...@fsn.hu> writes:
> >
> >> On 11/16/11 18:40, Philip Martin wrote:
> >>> Attila Nagy<br...@fsn.hu> writes:
> >>>
> >>>> I use pysvn for this and basically the code looks like this (in python):
> >>>> def update_perms():
> >>>> for path in propchg:
> >>>> proplist = svn.propget('file:permissions', path)
> >>>> if not os.path.islink(path) and proplist.has_key(path):
> >>>> set_perms(path, proplist[path])
> >>>> svn.update(walkroot)
> >>>> update_perms()
> >>>>
> >>>> The svn update collects the changed entries (propchg) and update_perms
> >>>> iterates on them and gets their file:permissions property and sets it
> >>>> in the file system.
> >>>>
> >>>> And this is what takes ages (literally), compared to 1.6.
> >>>> Any ideas about what could be done in this topic?
> >>>
> >>> It might be faster to run a recursive propget, which is a single
> >>> transaction, and discard the output if it doesn't match one of the
> >>> changed paths.
> >>>
> >> I will try this. Should this be true even for 10+ million files?
> >
> > It depends on the ratio of changed files to total files. If there is
> > only one changed file then the single propget will be faster. If most
> > of the files are changed then the recursive propget will be faster.
>
> Yes, you'll need to test that a bit.
>
> I do something similar here with a script that fetches all the log
> entries for merged revisions (cherrypick merges, which I look up with
> 'svn mergeinfo --show-revs merged'): if the number of merged revisions
> is low relative to their range, I perform a series of individual 'svn
> log' requests. Otherwise, I do a single 'svn log -r<min>:<max>'
> request, parse the output and discard the entries that are not
> relevant. This made my script much faster in most cases.
>
> This has nothing to do with recursive propget (or even with 1.7, I'm
> using this in a 1.5 environment), but I'm just noting the similarity
> of the problem here.
svn log -r N -r M -r P
was implemented a few years ago...
> --
> Johan
Re: Assertion failed and crash with 1.7.1
Posted by Johan Corveleyn <jc...@gmail.com>.
On Thu, Nov 17, 2011 at 1:02 PM, Philip Martin
<ph...@wandisco.com> wrote:
> Attila Nagy <br...@fsn.hu> writes:
>
>> On 11/16/11 18:40, Philip Martin wrote:
>>> Attila Nagy<br...@fsn.hu> writes:
>>>
>>>> I use pysvn for this and basically the code looks like this (in python):
>>>> def update_perms():
>>>> for path in propchg:
>>>> proplist = svn.propget('file:permissions', path)
>>>> if not os.path.islink(path) and proplist.has_key(path):
>>>> set_perms(path, proplist[path])
>>>> svn.update(walkroot)
>>>> update_perms()
>>>>
>>>> The svn update collects the changed entries (propchg) and update_perms
>>>> iterates on them and gets their file:permissions property and sets it
>>>> in the file system.
>>>>
>>>> And this is what takes ages (literally), compared to 1.6.
>>>> Any ideas about what could be done in this topic?
>>>
>>> It might be faster to run a recursive propget, which is a single
>>> transaction, and discard the output if it doesn't match one of the
>>> changed paths.
>>>
>> I will try this. Should this be true even for 10+ million files?
>
> It depends on the ratio of changed files to total files. If there is
> only one changed file then the single propget will be faster. If most
> of the files are changed then the recursive propget will be faster.
Yes, you'll need to test that a bit.
I do something similar here with a script that fetches all the log
entries for merged revisions (cherrypick merges, which I look up with
'svn mergeinfo --show-revs merged'): if the number of merged revisions
is low relative to their range, I perform a series of individual 'svn
log' requests. Otherwise, I do a single 'svn log -r<min>:<max>'
request, parse the output and discard the entries that are not
relevant. This made my script much faster in most cases.
This has nothing to do with recursive propget (or even with 1.7, I'm
using this in a 1.5 environment), but I'm just noting the similarity
of the problem here.
But for your propget problem, there may also be another option: 'svn
propget' can take multiple targets:
usage: 1. propget PROPNAME [TARGET[@REV]...]
So if you collect the targets in a list, you can get the exact
information you want with a single propget.
That said, I'm not sure that 'svn propget file:permissions A B C' does
all its work in a single transaction, rather than using a transaction
per target (in which case this technique won't buy you much). Maybe
you can quickly verify this if it makes any performance impact ...
--
Johan
Re: Assertion failed and crash with 1.7.1
Posted by Philip Martin <ph...@wandisco.com>.
Attila Nagy <br...@fsn.hu> writes:
> On 11/16/11 18:40, Philip Martin wrote:
>> Attila Nagy<br...@fsn.hu> writes:
>>
>>> I use pysvn for this and basically the code looks like this (in python):
>>> def update_perms():
>>> for path in propchg:
>>> proplist = svn.propget('file:permissions', path)
>>> if not os.path.islink(path) and proplist.has_key(path):
>>> set_perms(path, proplist[path])
>>> svn.update(walkroot)
>>> update_perms()
>>>
>>> The svn update collects the changed entries (propchg) and update_perms
>>> iterates on them and gets their file:permissions property and sets it
>>> in the file system.
>>>
>>> And this is what takes ages (literally), compared to 1.6.
>>> Any ideas about what could be done in this topic?
>>
>> It might be faster to run a recursive propget, which is a single
>> transaction, and discard the output if it doesn't match one of the
>> changed paths.
>>
> I will try this. Should this be true even for 10+ million files?
It depends on the ratio of changed files to total files. If there is
only one changed file then the single propget will be faster. If most
of the files are changed then the recursive propget will be faster.
--
Philip
Re: Assertion failed and crash with 1.7.1
Posted by Attila Nagy <br...@fsn.hu>.
On 11/16/11 18:40, Philip Martin wrote:
> Attila Nagy<br...@fsn.hu> writes:
>
>> I use pysvn for this and basically the code looks like this (in python):
>> def update_perms():
>> for path in propchg:
>> proplist = svn.propget('file:permissions', path)
>> if not os.path.islink(path) and proplist.has_key(path):
>> set_perms(path, proplist[path])
>> svn.update(walkroot)
>> update_perms()
>>
>> The svn update collects the changed entries (propchg) and update_perms
>> iterates on them and gets their file:permissions property and sets it
>> in the file system.
>>
>> And this is what takes ages (literally), compared to 1.6.
>> Any ideas about what could be done in this topic?
>
> It might be faster to run a recursive propget, which is a single
> transaction, and discard the output if it doesn't match one of the
> changed paths.
>
I will try this. Should this be true even for 10+ million files? I don't
want to think about the memory usage... :)
Re: Assertion failed and crash with 1.7.1
Posted by Philip Martin <ph...@wandisco.com>.
Attila Nagy <br...@fsn.hu> writes:
> I use pysvn for this and basically the code looks like this (in python):
> def update_perms():
> for path in propchg:
> proplist = svn.propget('file:permissions', path)
> if not os.path.islink(path) and proplist.has_key(path):
> set_perms(path, proplist[path])
> svn.update(walkroot)
> update_perms()
>
> The svn update collects the changed entries (propchg) and update_perms
> iterates on them and gets their file:permissions property and sets it
> in the file system.
>
> And this is what takes ages (literally), compared to 1.6.
> Any ideas about what could be done in this topic?
It might be faster to run a recursive propget, which is a single
transaction, and discard the output if it doesn't match one of the
changed paths.
--
Philip
Re: Assertion failed and crash with 1.7.1
Posted by Attila Nagy <br...@fsn.hu>.
On 10/27/11 14:57, Mark Phippard wrote:
> On Thu, Oct 27, 2011 at 8:12 AM, Attila Nagy <bra@fsn.hu
> <ma...@fsn.hu>> wrote:
>
> ZFS.
> It it worth to make benchmarks with this WC with 1.6 and 1.7? I so,
> I can try to find the time for it.
>
>
> There are some pretty easy to run benchmarks you can run to see if 1.7
> is simply slower in your environment. See:
>
> https://ctf.open.collab.net/sf/projects/csvn/
>
> If those benchmarks show 1.7 is slower for you, then that would point to
> issues with the 1.7 changes in your environment. As I mentioned
> earlier, and as an example, these benchmarks show that 1.7 is slower
> with WC on a network mount.
>
> If the benchmarks show 1.7 is faster, then we could focus on scalability
> issues that your working copy raises. Maybe inefficient indexes or
> something like that.
It turned out that not updating which is much slower (however I can't
really comment on the exact numbers now), but the propgets.
To understand this: as I've mentioned, I store operating system images
in svn, so I have to reproduce owner and permission information too.
Therefore I store them in svn properties and re-create them as needed on
updates.
I use pysvn for this and basically the code looks like this (in python):
def update_perms():
for path in propchg:
proplist = svn.propget('file:permissions', path)
if not os.path.islink(path) and proplist.has_key(path):
set_perms(path, proplist[path])
svn.update(walkroot)
update_perms()
The svn update collects the changed entries (propchg) and update_perms
iterates on them and gets their file:permissions property and sets it in
the file system.
And this is what takes ages (literally), compared to 1.6.
Any ideas about what could be done in this topic?
I will try to make more specific numbers if needed.
Re: Assertion failed and crash with 1.7.1
Posted by Mark Phippard <ma...@gmail.com>.
On Thu, Oct 27, 2011 at 8:12 AM, Attila Nagy <br...@fsn.hu> wrote:
> ZFS.
> It it worth to make benchmarks with this WC with 1.6 and 1.7? I so, I can
> try to find the time for it.
>
There are some pretty easy to run benchmarks you can run to see if 1.7 is
simply slower in your environment. See:
https://ctf.open.collab.net/sf/projects/csvn/
If those benchmarks show 1.7 is slower for you, then that would point to
issues with the 1.7 changes in your environment. As I mentioned earlier,
and as an example, these benchmarks show that 1.7 is slower with WC on a
network mount.
If the benchmarks show 1.7 is faster, then we could focus on scalability
issues that your working copy raises. Maybe inefficient indexes or
something like that.
--
Thanks
Mark Phippard
http://markphip.blogspot.com/
Re: Assertion failed and crash with 1.7.1
Posted by Attila Nagy <br...@fsn.hu>.
On 10/27/11 12:58, Philip Martin wrote:
> Attila Nagy<br...@fsn.hu> writes:
>
>> On 10/26/11 15:37, Philip Martin wrote:
>>> Attila Nagy<br...@fsn.hu> writes:
>>>
>>>> I'm trying to update a working copy of some tens of GBs with svn 1.7.1
>>> Did you upgrade with 1.7.0 or 1.7.1?
>> I've upgraded the WC with 1.7.0, then switched to 1.7.1, which I'm
>> currently using.
>> The upgrade took nearly one week (I can't afford a clean checkout,
>> because I have to preserve the files' inode numbers), at the start it
>> was running very fast, and at the end of the week it could print a new
>> directory in every 8-10 seconds, while the svn processes CPU usage was
>> 100%.
>> The disks weren't touched, it seems that even with a database
>> completely in the OS's buffer cache the SQL operations are pretty slow
>> with a lot of files.
>> Could it be because some indexes are missing (or not optimized for
>> this amount of files), or it is what we could get from SQLite?
> There is one query/index fix in 1.7.1 but it doesn't affect upgrade.
>
> I've not upgraded a wc as large as yours but I have just upgraded a 5GB
> working copy on my desktop machine (3 years old, not hugely powerful).
> It took about 45 minutes and used about 30 minutes of CPU. There wasn't
> any noticeable slowdown as the update proceeded.
>
> I see that you are using Fibre Channel storage, perhaps there is an
> SQLite issue there? What sort of filesystem are you using?
>
ZFS.
It it worth to make benchmarks with this WC with 1.6 and 1.7? I so, I
can try to find the time for it.
Re: Assertion failed and crash with 1.7.1
Posted by Philip Martin <ph...@wandisco.com>.
Attila Nagy <br...@fsn.hu> writes:
> On 10/26/11 15:37, Philip Martin wrote:
>> Attila Nagy<br...@fsn.hu> writes:
>>
>>> I'm trying to update a working copy of some tens of GBs with svn 1.7.1
>> Did you upgrade with 1.7.0 or 1.7.1?
> I've upgraded the WC with 1.7.0, then switched to 1.7.1, which I'm
> currently using.
> The upgrade took nearly one week (I can't afford a clean checkout,
> because I have to preserve the files' inode numbers), at the start it
> was running very fast, and at the end of the week it could print a new
> directory in every 8-10 seconds, while the svn processes CPU usage was
> 100%.
> The disks weren't touched, it seems that even with a database
> completely in the OS's buffer cache the SQL operations are pretty slow
> with a lot of files.
> Could it be because some indexes are missing (or not optimized for
> this amount of files), or it is what we could get from SQLite?
There is one query/index fix in 1.7.1 but it doesn't affect upgrade.
I've not upgraded a wc as large as yours but I have just upgraded a 5GB
working copy on my desktop machine (3 years old, not hugely powerful).
It took about 45 minutes and used about 30 minutes of CPU. There wasn't
any noticeable slowdown as the update proceeded.
I see that you are using Fibre Channel storage, perhaps there is an
SQLite issue there? What sort of filesystem are you using?
--
Philip
Re: Assertion failed and crash with 1.7.1
Posted by Attila Nagy <br...@fsn.hu>.
On 10/27/11 13:47, Mark Phippard wrote:
> On Thu, Oct 27, 2011 at 4:47 AM, Attila Nagy <bra@fsn.hu
> <ma...@fsn.hu>> wrote:
>
> On 10/26/11 15:37, Philip Martin wrote:
>> Attila Nagy<br...@fsn.hu> <ma...@fsn.hu> writes:
>>
>>> I'm trying to update a working copy of some tens of GBs with svn 1.7.1
>> Did you upgrade with 1.7.0 or 1.7.1?
> I've upgraded the WC with 1.7.0, then switched to 1.7.1, which I'm
> currently using.
> The upgrade took nearly one week (I can't afford a clean checkout,
> because I have to preserve the files' inode numbers), at the start
> it was running very fast, and at the end of the week it could
> print a new directory in every 8-10 seconds, while the svn
> processes CPU usage was 100%.
> The disks weren't touched, it seems that even with a database
> completely in the OS's buffer cache the SQL operations are pretty
> slow with a lot of files.
> Could it be because some indexes are missing (or not optimized for
> this amount of files), or it is what we could get from SQLite?
>
>
> I can see where having all that RAM could help greatly when the
> database is being read, but I do not see how it would help when we are
> writing to the database. I would imagine the database does stuff to
> flush itself to disk whenever a transaction is committed. I thought
> this was why we see 1.7 slower than other versions via NFS, even on
> small working copies where the database would fit in anyones RAM.
>
> I would imagine svn upgrade is almost entirely writes and I recall it
> does quite a few transactions. So couldn't the linear slow down just
> be based on the growth in the amount of bytes that are written to disk
> each time?
>
I've checked the disk IO stat and I couldn't see any bottlenecks there.
In fact while upgrade-ing, I barely saw any disk IO.
Checking the process revealed a lot of random reads from the wc.db,
which the OS could serve out of memory.
BTW, I don't have any transaction-related files next to my wc.db (I
remember about wal, or journal named files, but I'm not a regular user
of SQLite), and definitely couldn't see too much fsyncs.
Re: Assertion failed and crash with 1.7.1
Posted by Attila Nagy <br...@fsn.hu>.
On 10/27/11 14:13, Philip Martin wrote:
> Mark Phippard<ma...@gmail.com> writes:
>
>> I would imagine svn upgrade is almost entirely writes and I recall it does
>> quite a few transactions. So couldn't the linear slow down just be based on
>> the growth in the amount of bytes that are written to disk each time?
> Yes, number of transactions matters a lot. However upgrade is a special
> case that runs in a single SQLite transaction as this makes most
> upgrades faster. Maybe for really large upgrades the huge transaction
> is a problem? My 100,000 node working copy upgrades in 45 minutes.
I don't have too much files/directories either, currently: 1772120,
that's only an order of magnitude bigger.
What I could see is that svn's memory usage has grown constantly during
the operation. It started on about 20M resident and went to
approximately 60-80M.
I don't know for sure because I lost the interest in watching it in
about 3-4 days, so I'm just extrapolation from the last seen value
(sixty something megabytes on day 3). :)
> Upgrade does a number of read queries as well as writes, but all within
> the same transaction.
>
I guess, you are right, maybe the big transaction caused the major
slowdown.
Re: Assertion failed and crash with 1.7.1
Posted by Philip Martin <ph...@wandisco.com>.
Mark Phippard <ma...@gmail.com> writes:
> I would imagine svn upgrade is almost entirely writes and I recall it does
> quite a few transactions. So couldn't the linear slow down just be based on
> the growth in the amount of bytes that are written to disk each time?
Yes, number of transactions matters a lot. However upgrade is a special
case that runs in a single SQLite transaction as this makes most
upgrades faster. Maybe for really large upgrades the huge transaction
is a problem? My 100,000 node working copy upgrades in 45 minutes.
Upgrade does a number of read queries as well as writes, but all within
the same transaction.
--
Philip
Re: Assertion failed and crash with 1.7.1
Posted by Mark Phippard <ma...@gmail.com>.
On Thu, Oct 27, 2011 at 4:47 AM, Attila Nagy <br...@fsn.hu> wrote:
> On 10/26/11 15:37, Philip Martin wrote:
>
> Attila Nagy <br...@fsn.hu> <br...@fsn.hu> writes:
>
>
> I'm trying to update a working copy of some tens of GBs with svn 1.7.1
>
> Did you upgrade with 1.7.0 or 1.7.1?
>
> I've upgraded the WC with 1.7.0, then switched to 1.7.1, which I'm
> currently using.
> The upgrade took nearly one week (I can't afford a clean checkout, because
> I have to preserve the files' inode numbers), at the start it was running
> very fast, and at the end of the week it could print a new directory in
> every 8-10 seconds, while the svn processes CPU usage was 100%.
> The disks weren't touched, it seems that even with a database completely in
> the OS's buffer cache the SQL operations are pretty slow with a lot of
> files.
> Could it be because some indexes are missing (or not optimized for this
> amount of files), or it is what we could get from SQLite?
>
I can see where having all that RAM could help greatly when the database is
being read, but I do not see how it would help when we are writing to the
database. I would imagine the database does stuff to flush itself to disk
whenever a transaction is committed. I thought this was why we see 1.7
slower than other versions via NFS, even on small working copies where the
database would fit in anyones RAM.
I would imagine svn upgrade is almost entirely writes and I recall it does
quite a few transactions. So couldn't the linear slow down just be based on
the growth in the amount of bytes that are written to disk each time?
--
Thanks
Mark Phippard
http://markphip.blogspot.com/
Re: Assertion failed and crash with 1.7.1
Posted by Attila Nagy <br...@fsn.hu>.
On 10/26/11 15:37, Philip Martin wrote:
> Attila Nagy<br...@fsn.hu> writes:
>
>> I'm trying to update a working copy of some tens of GBs with svn 1.7.1
> Did you upgrade with 1.7.0 or 1.7.1?
I've upgraded the WC with 1.7.0, then switched to 1.7.1, which I'm
currently using.
The upgrade took nearly one week (I can't afford a clean checkout,
because I have to preserve the files' inode numbers), at the start it
was running very fast, and at the end of the week it could print a new
directory in every 8-10 seconds, while the svn processes CPU usage was 100%.
The disks weren't touched, it seems that even with a database completely
in the OS's buffer cache the SQL operations are pretty slow with a lot
of files.
Could it be because some indexes are missing (or not optimized for this
amount of files), or it is what we could get from SQLite?
>
>> $ svn up
>> Updating '.':
>> svn: E235000: In file 'subversion/libsvn_wc/update_editor.c' line
>> 1582: assertion failed (action == svn_wc_conflict_action_edit ||
>> action == svn_wc_conflict_action_delete || action ==
>> svn_wc_conflict_action_replace)
>> Abort (core dumped)
> Can you show us a gdb stack trace?
I'm sorry, this is a production machine and this error caused a major
lockup in our business processes, so I've had to work around it, and I
lost the core file in the process.
BTW, there was one tree conflict somewhere, it seems that an svn resolve
solved the issue.
>
> How many working nodes do you have?
> sqlite3 .svn/wc.db "select count (*) from nodes where op_depth> 0"
>
The query say that I have one. (now, after the conflict resolution)
Re: Assertion failed and crash with 1.7.1
Posted by Philip Martin <ph...@wandisco.com>.
Attila Nagy <br...@fsn.hu> writes:
> I'm trying to update a working copy of some tens of GBs with svn 1.7.1
Did you upgrade with 1.7.0 or 1.7.1?
> $ svn up
> Updating '.':
> svn: E235000: In file 'subversion/libsvn_wc/update_editor.c' line
> 1582: assertion failed (action == svn_wc_conflict_action_edit ||
> action == svn_wc_conflict_action_delete || action ==
> svn_wc_conflict_action_replace)
> Abort (core dumped)
Can you show us a gdb stack trace?
How many working nodes do you have?
sqlite3 .svn/wc.db "select count (*) from nodes where op_depth > 0"
--
Philip