You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by Johan Corveleyn <jc...@gmail.com> on 2012/05/02 12:20:35 UTC
Re: Merge policies
On Fri, Apr 20, 2012 at 10:52 PM, Stefan Fuhrmann <eq...@web.de> wrote:
> Am 19.04.2012 14:03, schrieb Johan Corveleyn:
[snip]
>> I haven't read your mail in detail yet, but just a note in passing: in
>> my experience (with a 1.5 server with FSFS-on-NFS backend) propgetting
>> ('svnlook propget') every changed file during pre-commit can be very
>> expensive.
[snip]
> Random thought: is it possible to link the transactions
> folder to /dev/shm even if the repository is on NFS?
Ah yes, thanks for the suggestion. I should try that someday. Right
now we're working on moving our repo to a newer server, so it'll take
me a while to experiment with this.
While we're on the subject: do you know an easy way to do this on
Windows? It's possible that we'll be moving our server to Windows
(from Solaris). Not sure yet, it depends on some testing /
benchmarking.
The reason is that I can probably get faster I/O from our NAS when
running the server on Windows, because there I can work with iSCSI
(which is currently not supported by our sysadmins on Solaris). And I
think iSCSI will get me better performance than NFS (there is also a
third option: Solaris on ESX, which makes Solaris talk to the storage
as if it's a local disk (but underlying it's still NFS I think) ...
don't know whether that makes it faster). To be benchmarked ...
>> - maybe propgetting through one of the bindings is much faster than
>> through svnlook, and you intend for admins to set this up with some
>> language bindings? In that case, keep in mind that bindings are not
>> always installed. Svnlook usually is always there.
>
>
> The problem is probably that svnlook is at a hefty
> disadvantage over e.g. the server because it has
> to open the FSFS repo anew for each access. Don't
> know really what to do about that.
So the only way to get the "relevant property changes" really fast
with svnlook, would be with a command that gets everything with one
invocation, I guess.
Actually, there is probably a way to get all those propchanges
quicker, by parsing the output of 'svnlook diff -t TXN'. But I've not
done the effort to do that.
Besides, by invoking diff (and discarding the actuall text diffs) a
lot of needless work is done (and think of those edge cases where our
internal diff can get really slow), and needless bytes are being
written/read.
Maybe we'd need a props-only-diff, which gives you only the prop
changes? In some easily parseable way ...
Another alternative would be to make 'svnlook propget' support
multiple targets, so you can just propget all paths from 'svnlook
changed' (possibly trimmed to the Adds and PropMods) in one go.
--
Johan
Re: Merge policies
Posted by Daniel Shahaf <d....@daniel.shahaf.name>.
Johan Corveleyn wrote on Wed, May 02, 2012 at 12:20:35 +0200:
> On Fri, Apr 20, 2012 at 10:52 PM, Stefan Fuhrmann <eq...@web.de> wrote:
> > Am 19.04.2012 14:03, schrieb Johan Corveleyn:
>
> [snip]
>
> >> I haven't read your mail in detail yet, but just a note in passing: in
> >> my experience (with a 1.5 server with FSFS-on-NFS backend) propgetting
> >> ('svnlook propget') every changed file during pre-commit can be very
> >> expensive.
>
> [snip]
>
> > Random thought: is it possible to link the transactions
> > folder to /dev/shm even if the repository is on NFS?
>
> Ah yes, thanks for the suggestion. I should try that someday. Right
> now we're working on moving our repo to a newer server, so it'll take
> me a while to experiment with this.
>
> While we're on the subject: do you know an easy way to do this on
> Windows? It's possible that we'll be moving our server to Windows
> (from Solaris). Not sure yet, it depends on some testing /
> benchmarking.
>
> The reason is that I can probably get faster I/O from our NAS when
> running the server on Windows, because there I can work with iSCSI
> (which is currently not supported by our sysadmins on Solaris). And I
> think iSCSI will get me better performance than NFS (there is also a
> third option: Solaris on ESX, which makes Solaris talk to the storage
> as if it's a local disk (but underlying it's still NFS I think) ...
> don't know whether that makes it faster). To be benchmarked ...
>
> >> - maybe propgetting through one of the bindings is much faster than
> >> through svnlook, and you intend for admins to set this up with some
> >> language bindings? In that case, keep in mind that bindings are not
> >> always installed. Svnlook usually is always there.
> >
> >
> > The problem is probably that svnlook is at a hefty
> > disadvantage over e.g. the server because it has
> > to open the FSFS repo anew for each access. Don't
> > know really what to do about that.
>
> So the only way to get the "relevant property changes" really fast
> with svnlook, would be with a command that gets everything with one
> invocation, I guess.
>
> Actually, there is probably a way to get all those propchanges
> quicker, by parsing the output of 'svnlook diff -t TXN'. But I've not
> done the effort to do that.
>
> Besides, by invoking diff (and discarding the actuall text diffs) a
> lot of needless work is done (and think of those edge cases where our
> internal diff can get really slow), and needless bytes are being
> written/read.
>
> Maybe we'd need a props-only-diff, which gives you only the prop
> changes? In some easily parseable way ...
>
> Another alternative would be to make 'svnlook propget' support
> multiple targets, so you can just propget all paths from 'svnlook
> changed' (possibly trimmed to the Adds and PropMods) in one go.
What about some svnmucc-style interface?
svnmlcc /path/to/repos propget r1 trunk/README svn:eol-style revpropget r1 svn:author propsetf r2 svn:log =(echo danielsh)
This will open one FS handle and won't require a custom script via the
bindings.
Re: Merge policies
Posted by Johan Corveleyn <jc...@gmail.com>.
On Wed, May 2, 2012 at 12:47 PM, Philip Martin
<ph...@wandisco.com> wrote:
> Johan Corveleyn <jc...@gmail.com> writes:
>
>> On Fri, Apr 20, 2012 at 10:52 PM, Stefan Fuhrmann <eq...@web.de> wrote:
>>> Am 19.04.2012 14:03, schrieb Johan Corveleyn:
>>>> - maybe propgetting through one of the bindings is much faster than
>>>> through svnlook, and you intend for admins to set this up with some
>>>> language bindings? In that case, keep in mind that bindings are not
>>>> always installed. Svnlook usually is always there.
>>>
>>>
>>> The problem is probably that svnlook is at a hefty
>>> disadvantage over e.g. the server because it has
>>> to open the FSFS repo anew for each access. Don't
>>> know really what to do about that.
>>
>> So the only way to get the "relevant property changes" really fast
>> with svnlook, would be with a command that gets everything with one
>> invocation, I guess.
>>
>> Actually, there is probably a way to get all those propchanges
>> quicker, by parsing the output of 'svnlook diff -t TXN'. But I've not
>> done the effort to do that.
>
> As suggested above, the bindings are fast:
>
> $ svnadmin create repo
> $ svn import -mm ../src file://`pwd`/repo/trunk # import Subversion trunk
> $ svn co file://`pwd`/repo wc
> $ svn ps -R svn:mime-type foo/bar wc
> $ svn ci -mm wc
> $ ./x.py repo 2 | wc
> 2270 2270 141635
>
> Getting the mime-type for 2270 files takes less than 1 second.
>
>
> $ cat x.py
> #!/usr/bin/python
> import sys
> from svn import repos, fs
> r=repos.open(sys.argv[1])
> f=repos.fs(r)
> #t=fs.open_txn(f, sys.argv[2])
> t=fs.revision_root(f, int(sys.argv[2]))
> p=fs.paths_changed2(t)
> for key, val in p.iteritems():
> print key + ":" + str(fs.node_prop(t, key, 'svn:mime-type'))
Nice. Thanks for testing. That's good to know.
That said, I think an improvement for svnlook is still sorely needed.
A lot of installations out there don't have bindings installed, plus a
lot of existing (sometimes publicly available) hooks are written for
svnlook (to be as generically useful as possible). Like this very nice
configurable pre-commit hook:
https://github.com/qazwart/SVN-Precommit-Kitchen-Sink-Hook
Anyway, I don't have the cycles to work on this myself, but I just
wanted to mention this as an important aspect if more and more tools
are going to validate properties in pre-commit hooks (like for
enforcing merge policies).
--
Johan
Re: Merge policies
Posted by Philip Martin <ph...@wandisco.com>.
Johan Corveleyn <jc...@gmail.com> writes:
> On Fri, Apr 20, 2012 at 10:52 PM, Stefan Fuhrmann <eq...@web.de> wrote:
>> Am 19.04.2012 14:03, schrieb Johan Corveleyn:
>>> - maybe propgetting through one of the bindings is much faster than
>>> through svnlook, and you intend for admins to set this up with some
>>> language bindings? In that case, keep in mind that bindings are not
>>> always installed. Svnlook usually is always there.
>>
>>
>> The problem is probably that svnlook is at a hefty
>> disadvantage over e.g. the server because it has
>> to open the FSFS repo anew for each access. Don't
>> know really what to do about that.
>
> So the only way to get the "relevant property changes" really fast
> with svnlook, would be with a command that gets everything with one
> invocation, I guess.
>
> Actually, there is probably a way to get all those propchanges
> quicker, by parsing the output of 'svnlook diff -t TXN'. But I've not
> done the effort to do that.
As suggested above, the bindings are fast:
$ svnadmin create repo
$ svn import -mm ../src file://`pwd`/repo/trunk # import Subversion trunk
$ svn co file://`pwd`/repo wc
$ svn ps -R svn:mime-type foo/bar wc
$ svn ci -mm wc
$ ./x.py repo 2 | wc
2270 2270 141635
Getting the mime-type for 2270 files takes less than 1 second.
$ cat x.py
#!/usr/bin/python
import sys
from svn import repos, fs
r=repos.open(sys.argv[1])
f=repos.fs(r)
#t=fs.open_txn(f, sys.argv[2])
t=fs.revision_root(f, int(sys.argv[2]))
p=fs.paths_changed2(t)
for key, val in p.iteritems():
print key + ":" + str(fs.node_prop(t, key, 'svn:mime-type'))
--
uberSVN: Apache Subversion Made Easy
http://www.uberSVN.com