Beginnings of python script for annotate

[Daniel Berlin <db...@dberlin.org>]

Does all that one needs, except that it's slow.
The reason it's slow is because it uses python's difflib to do the diffs 
(I verified this is the large part of the current slowness.
On gdb's dwarf2read.c, where we have a few revisions (8, i think) of the 
actual file, and 700 revisions of the repository (that's all i felt like 
importing),not doing the diffs takes 3 seconds, doing them takes 20

I also couldn't remember how to get the oldest revision of a file.
It does two things that probably make up the majority of the 3 seconds:
1. It walks each revision from 1, checking when the path starts to exist, 
to get the oldest revision.
2. It uses fs_file_contents_changed to determine if the contents changed 
from one revision to another.
This is faster than using python diff to do it.

The code is the result of 30 minutes of hacking getfile.py , so it's ugly.

It's also not perfect, the line counting is off slightly somehow.

Since the diff results are a generator in 2.2.1, i can't get the length of 
it unless i add another for loop.

But the method is there.

--Dan

Re: Beginnings of python script for annotate

[Daniel Berlin <db...@dberlin.org>]

On Tue, 13 Aug 2002, Blair Zajac wrote:

> Daniel Berlin wrote:
> > 
> > Does all that one needs, except that it's slow.
> > The reason it's slow is because it uses python's difflib to do the diffs
> > (I verified this is the large part of the current slowness.
> > On gdb's dwarf2read.c, where we have a few revisions (8, i think) of the
> > actual file, and 700 revisions of the repository (that's all i felt like
> > importing),not doing the diffs takes 3 seconds, doing them takes 20
> > 
> > I also couldn't remember how to get the oldest revision of a file.
> > It does two things that probably make up the majority of the 3 seconds:
> > 1. It walks each revision from 1, checking when the path starts to exist,
> > to get the oldest revision.
> > 2. It uses fs_file_contents_changed to determine if the contents changed
> > from one revision to another.
> > This is faster than using python diff to do it.
> 
> Does it handle renames of the file?
Nope.
It's just an example.
To determine how fast/slow diffing fulltexts is.
But handling renames shouldn't be that tricky, that's a gathering phase 
thing, and shouldn't be but a few more lines of python.



---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org

Re: Beginnings of python script for annotate

[Blair Zajac <bl...@orcaware.com>]

Daniel Berlin wrote:
> 
> Does all that one needs, except that it's slow.
> The reason it's slow is because it uses python's difflib to do the diffs
> (I verified this is the large part of the current slowness.
> On gdb's dwarf2read.c, where we have a few revisions (8, i think) of the
> actual file, and 700 revisions of the repository (that's all i felt like
> importing),not doing the diffs takes 3 seconds, doing them takes 20
> 
> I also couldn't remember how to get the oldest revision of a file.
> It does two things that probably make up the majority of the 3 seconds:
> 1. It walks each revision from 1, checking when the path starts to exist,
> to get the oldest revision.
> 2. It uses fs_file_contents_changed to determine if the contents changed
> from one revision to another.
> This is faster than using python diff to do it.

Does it handle renames of the file?

Best,
Blair

-- 
Blair Zajac <bl...@orcaware.com>
Web and OS performance plots - http://www.orcaware.com/orca/

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org

Re: Beginnings of python script for annotate

[Karl Fogel <kf...@newton.ch.collab.net>]

Branko Čibej <br...@xbc.nu> writes:
> >You know what I'm thinking?  This might be best solved with C program
> >that uses Sander's internal diff library...
> 
> No, it would be best solved by a Python script that uses the SVN
> bindings to drive Sander's diff library from libsvn_delta. :-)

Ooooh!  Yeah, that'd be ideal.

Daniel, how much time do you have to put into this?  You seem to have
started down this path a bit already, that's why I ask.  (Btw, note
that you don't need to query each revision -- you can get just the
revisions in which a given file has changed; on the repository side by
using svn_fs_revisions_changed(), on the client side by "svn log file"
or the binding equivalent).

We still need to know how this kind of solution works on a file that
has 100s of revisions, though.  Daniel's earlier test was on a file
with appx 8 changes.

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org

Re: Beginnings of python script for annotate

[Greg Stein <gs...@lyra.org>]

On Tue, Aug 13, 2002 at 10:26:27PM +0200, Branko Cibej wrote:
> Karl Fogel wrote:
>...
> >You know what I'm thinking?  This might be best solved with C program
> >that uses Sander's internal diff library...
> 
> No, it would be best solved by a Python script that uses the SVN 
> bindings to drive Sander's diff library from libsvn_delta. :-)

You betcha! Woo!

:-)

-- 
Greg Stein, http://www.lyra.org/

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org

Re: Beginnings of python script for annotate

[Branko Čibej <br...@xbc.nu>]

Karl Fogel wrote:

>Daniel Berlin <db...@dberlin.org> writes:
>  
>
>>Does all that one needs, except that it's slow.
>>The reason it's slow is because it uses python's difflib to do the diffs 
>>(I verified this is the large part of the current slowness.
>>On gdb's dwarf2read.c, where we have a few revisions (8, i think) of the 
>>actual file, and 700 revisions of the repository (that's all i felt like 
>>importing),not doing the diffs takes 3 seconds, doing them takes 20
>>    
>>
>
>Python's difflib re-implements diff, instead of driving a diff
>subprocess.  Maybe these numbers would be somewhat better if we were
>calling out to a subprocess, though Greg points out that the degree of
>improvement would be system-dependent.
>
>You know what I'm thinking?  This might be best solved with C program
>that uses Sander's internal diff library...
>  
>

No, it would be best solved by a Python script that uses the SVN 
bindings to drive Sander's diff library from libsvn_delta. :-)

-- 
Brane Čibej   <br...@xbc.nu>   http://www.xbc.nu/brane/


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org

Re: Beginnings of python script for annotate

[Karl Fogel <kf...@newton.ch.collab.net>]

Daniel Berlin <db...@dberlin.org> writes:
> Does all that one needs, except that it's slow.
> The reason it's slow is because it uses python's difflib to do the diffs 
> (I verified this is the large part of the current slowness.
> On gdb's dwarf2read.c, where we have a few revisions (8, i think) of the 
> actual file, and 700 revisions of the repository (that's all i felt like 
> importing),not doing the diffs takes 3 seconds, doing them takes 20

Python's difflib re-implements diff, instead of driving a diff
subprocess.  Maybe these numbers would be somewhat better if we were
calling out to a subprocess, though Greg points out that the degree of
improvement would be system-dependent.

You know what I'm thinking?  This might be best solved with C program
that uses Sander's internal diff library...

-K

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org

Re: Beginnings of python script for annotate

[Daniel Berlin <db...@dberlin.org>]

On Wed, 14 Aug 2002, Greg Stein wrote:

> On Tue, Aug 13, 2002 at 01:33:10PM -0400, Daniel Berlin wrote:
> > Does all that one needs, except that it's slow.
> 
> I've committed it to /trunk/tools/example/blame.py with some minor mods.
> 
> Hopefully, we can get some iterations on it...
I figured out why it screws the line numbers up sometimes.

I forgot to remap the line numbers after we see a line delete (after a 
deletion, every line after the deletion has to be moved up one in the 
annotation array, or we need to track this and do it at printout time).
 > 
> Thanks,
> -g
> 
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org

Re: Beginnings of python script for annotate

[Greg Stein <gs...@lyra.org>]

On Tue, Aug 13, 2002 at 01:33:10PM -0400, Daniel Berlin wrote:
> Does all that one needs, except that it's slow.

I've committed it to /trunk/tools/example/blame.py with some minor mods.

Hopefully, we can get some iterations on it...

Thanks,
-g

-- 
Greg Stein, http://www.lyra.org/

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org