You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@subversion.apache.org by sv...@href.com on 2010/07/14 22:03:53 UTC
how to contribute feature of unknown popularity
Re: export --skipfilesmatchingsize
Prior thread link:
<http://svn.haxx.se/users/archive-2010-06/0040.shtml>http://svn.haxx.se/users/archive-2010-06/0040.shtml
Hi, I am the person who wanted this feature and worked on testing it
over recent months. It has been working well for some time, and I
use it often. I use it because I build a lot of deployment servers
where, on a daily or weekly basis, much (often more than 90%) of a
system stays as-is and a small amount needs to be brought in from
subversion. I use export (as opposed to update) to avoid having .svn
files scattered throughout production machines. The main reason I
like the skip feature is that it lets me deal with DLLs and other
files that are (a) unchanging and (b) in use at the time of
export. The normal export feature aborts when it encounters a
pre-existing file that cannot be overwritten because it is in use;
with the skip option, I can still export that branch of the
repository and pick up any new/additional/changed
files. Originally, I thought the skip feature would be a good way
to save bandwidth and time. In the end, that was not as critical for
me as being able to skip over in-use, unchanged DLLs.
I understand that many people do not need this feature, because they
do not use export often, or if they do, they export into empty target
folders, or they are not using Windows.
So my question is, even if this is an unpopular feature, is there
some way to contribute the source back to the main development trunk?
The modification consists of 4 .c files and 1 .h file. We started
from source from CollabNet, back in April 2010. We have not merged
in changes from the trunk since then, and I am starting to worry that
if we wait much longer, it will be unnecessarily difficult to merge
in our contribution.
I have Windows binaries available for anyone who would like to try
the feature.
I have tested it on Win2003 Server (32-bit) and Windows 7 Home
Premium 64-bit.
The flag name was designed long on purpose, (a) to avoid any
potential conflict with other flags and (b) to self-document the
feature. The feature does simply compare the byte size of each file,
and skips the export of that particular file when the size
matches. While it is arguable that this does not help with certain
types of files (bitmap images as an example) where a developer might
make a real change without a size change...... the other benefits
meant that it was still a very useful feature (to me).
Looking forward to some wisdom for contributing...
-Ann
Re: how to contribute feature of unknown popularity
Posted by Andy Levy <an...@gmail.com>.
On Wed, Jul 14, 2010 at 18:03, <sv...@href.com> wrote:
> So my question is, even if this is an unpopular feature, is there some way
> to contribute the source back to the main development trunk? The
> modification consists of 4 .c files and 1 .h file. We started from source
> from CollabNet, back in April 2010. We have not merged in changes from the
> trunk since then, and I am starting to worry that if we wait much longer, it
> will be unnecessarily difficult to merge in our contribution.
http://subversion.apache.org/contributing.html - Scroll down to
"Contributing source" and read everything, plus everything linked
there. You probably need to submit a patch against the current trunk,
or at least a more current release than April 2010.
That said, if the Subversion "core" folks don't feel that it's a
feature that Subversion ought to have, you'll be maintaining your own
fork for a while.
RE: how to contribute feature of unknown popularity
Posted by Bob Archer <Bo...@amsi.com>.
You might want to look at Robocopy. It is basically windows answer to rsync.
BOb
From: svnusertemp@href.com [mailto:svnusertemp@href.com]
Sent: Thursday, July 15, 2010 9:07 PM
To: users@subversion.apache.org
Subject: Re: how to contribute feature of unknown popularity
Thank you very much for the replies.
About export failing, it happened to me so often, I thought "everyone" would know the details. It occurs when Windows will not allow svn.exe export to write to the file, for any reason. The most common reason in my case was that the file was "in use", e.g. a DLL loaded in memory. Locking of a DLL is a Windows limitation and they probably think it is "by design." Another thing that "locks" files is the readonly attribute (which okay normally would not be there but I did run into it). I have only tested the CollabNet svn.exe (command line) in this regard. The whole thing is completely reproducible so if you want more info on that, let me know.
RSync, why not. Good question. Because it seemed completely overwhelming to learn another whole way of doing things, when I was just getting my head around moving 15 years of work from 2 other version control systems on into Subversion. Because I am only starting to run Linux servers and did not want to put RSync there without knowing enough how to admin it. Because when I read the info on using RSync on Windows, it seemed to be yet another product that did not really run on Windows, easily, simply (could be wrong about that... ), and I really dislike the idea of simulating Linux on Windows, not least because the instructions always assume linux know-how and if I had linux know-how, I'd already be running it on Linux. (End rant... just explaining why I was so turned off RSync). Another reason was that the files were already *in* Subversion repositories and it seemed natural to want to take them back out of there. Does RSync automatically work against an existing repository or would that involve having another copy of the files? Maybe "R" stands for Repository. Obviously I don't know anything about RSync other than it sync's something to something.
Last reason.... I thought that the spirit of CollabNet was inviting people to collaborate, and here I really thought I had something to contribute. And I say that because, before launching into all this, I read quite a few forum posts from other people over the years who also were wanting to use svn export as lazy-man's sync. It seemed so close to being exactly that, okay if we leave aside bitmaps, all it needed was the idea of selective export and voila, it was a sync. It really does not seem to be as bizarre as selective export of files containing java fragments.
As to the proposed solution of exporting everything to a temporary location and then copying selectively (with rsync or otherwise), I ruled that out because it needlessly duplicated all the files -- waste of disk space, processing time and effort. I did in fact do that (with xcopy) during the weeks before Paul coded the skip feature.
Here is a link to the modified source: http://greenbreen.com/svn_mod_source/ which implements the --skipfilesmatchingsize feature.
-Ann
At 11:43 AM 7/15/2010, Stefan Sperling wrote:
On Thu, Jul 15, 2010 at 07:27:57AM -0400, Nico Kadel-Garcia wrote:
> On Thu, Jul 15, 2010 at 6:34 AM, Stefan Sperling <st...@elego.de> wrote:
> > Why don't you use rsync, which was designed to solve exactly this problem?
> And, at last check as part of an attempt at an rsnapshot based backup
> system, it hangs on open files.
That is a Windows problem. Windows cannot handle deletions of files that
are open. The rsync windows ports should have special logic to deal with
this. If they don't, that needs to be fixed.
(Or even better, Windows should be fixed...)
> > I think that using a high-quality sync tool such as rsync is definitely
> > a better solution to the problem you're trying to solve.
>
> I'm mystified about using export on his core server. Do a checkout and
> svn update to a something like "SRCDIR" on the core server, to avoid
> the "this file is locked" issue, and use "rsync -av --exclude=.svn
> /SRCDIR/ REMOTEHOST:/TARGETDIR/" syntax to push only that updated
> code.
What is the "this file is locked" issue?
There is no difference between checkout and export, except that
export does not create the .svn directories.
Stefan
Re: how to contribute feature of unknown popularity
Posted by sv...@href.com.
Thank you very much for the replies.
About export failing, it happened to me so often, I thought
"everyone" would know the details. It occurs when Windows will not
allow svn.exe export to write to the file, for any reason. The most
common reason in my case was that the file was "in use", e.g. a DLL
loaded in memory. Locking of a DLL is a Windows limitation and they
probably think it is "by design." Another thing that "locks" files
is the readonly attribute (which okay normally would not be there but
I did run into it). I have only tested the CollabNet svn.exe (command
line) in this regard. The whole thing is completely reproducible so
if you want more info on that, let me know.
RSync, why not. Good question. Because it seemed completely
overwhelming to learn another whole way of doing things, when I was
just getting my head around moving 15 years of work from 2 other
version control systems on into Subversion. Because I am only
starting to run Linux servers and did not want to put RSync there
without knowing enough how to admin it. Because when I read the info
on using RSync on Windows, it seemed to be yet another product that
did not really run on Windows, easily, simply (could be wrong about
that... ), and I really dislike the idea of simulating Linux on
Windows, not least because the instructions always assume linux
know-how and if I had linux know-how, I'd already be running it on
Linux. (End rant... just explaining why I was so turned off
RSync). Another reason was that the files were already *in*
Subversion repositories and it seemed natural to want to take them
back out of there. Does RSync automatically work against an existing
repository or would that involve having another copy of the
files? Maybe "R" stands for Repository. Obviously I don't know
anything about RSync other than it sync's something to something.
Last reason.... I thought that the spirit of CollabNet was inviting
people to collaborate, and here I really thought I had something to
contribute. And I say that because, before launching into all this,
I read quite a few forum posts from other people over the years who
also were wanting to use svn export as lazy-man's sync. It seemed so
close to being exactly that, okay if we leave aside bitmaps, all it
needed was the idea of selective export and voila, it was a sync. It
really does not seem to be as bizarre as selective export of files
containing java fragments.
As to the proposed solution of exporting everything to a temporary
location and then copying selectively (with rsync or otherwise), I
ruled that out because it needlessly duplicated all the files --
waste of disk space, processing time and effort. I did in fact do
that (with xcopy) during the weeks before Paul coded the skip feature.
Here is a link to the modified source:
http://greenbreen.com/svn_mod_source/ which implements the
--skipfilesmatchingsize feature.
-Ann
At 11:43 AM 7/15/2010, Stefan Sperling wrote:
>On Thu, Jul 15, 2010 at 07:27:57AM -0400, Nico Kadel-Garcia wrote:
> > On Thu, Jul 15, 2010 at 6:34 AM, Stefan Sperling <st...@elego.de> wrote:
> > > Why don't you use rsync, which was designed to solve exactly
> this problem?
> > And, at last check as part of an attempt at an rsnapshot based backup
> > system, it hangs on open files.
>
>That is a Windows problem. Windows cannot handle deletions of files that
>are open. The rsync windows ports should have special logic to deal with
>this. If they don't, that needs to be fixed.
>(Or even better, Windows should be fixed...)
>
> > > I think that using a high-quality sync tool such as rsync is definitely
> > > a better solution to the problem you're trying to solve.
> >
> > I'm mystified about using export on his core server. Do a checkout and
> > svn update to a something like "SRCDIR" on the core server, to avoid
> > the "this file is locked" issue, and use "rsync -av --exclude=.svn
> > /SRCDIR/ REMOTEHOST:/TARGETDIR/" syntax to push only that updated
> > code.
>
>What is the "this file is locked" issue?
>
>There is no difference between checkout and export, except that
>export does not create the .svn directories.
>
>Stefan
Re: how to contribute feature of unknown popularity
Posted by Stefan Sperling <st...@elego.de>.
On Thu, Jul 15, 2010 at 07:27:57AM -0400, Nico Kadel-Garcia wrote:
> On Thu, Jul 15, 2010 at 6:34 AM, Stefan Sperling <st...@elego.de> wrote:
> > Why don't you use rsync, which was designed to solve exactly this problem?
> And, at last check as part of an attempt at an rsnapshot based backup
> system, it hangs on open files.
That is a Windows problem. Windows cannot handle deletions of files that
are open. The rsync windows ports should have special logic to deal with
this. If they don't, that needs to be fixed.
(Or even better, Windows should be fixed...)
> > I think that using a high-quality sync tool such as rsync is definitely
> > a better solution to the problem you're trying to solve.
>
> I'm mystified about using export on his core server. Do a checkout and
> svn update to a something like "SRCDIR" on the core server, to avoid
> the "this file is locked" issue, and use "rsync -av --exclude=.svn
> /SRCDIR/ REMOTEHOST:/TARGETDIR/" syntax to push only that updated
> code.
What is the "this file is locked" issue?
There is no difference between checkout and export, except that
export does not create the .svn directories.
Stefan
Re: how to contribute feature of unknown popularity
Posted by Les Mikesell <le...@gmail.com>.
Nico Kadel-Garcia wrote:
>
>> You can run rsync on windows using cygwin, or you can use one of the
>> Windows ports (google mentions DeltaCopy and cwrsync on the first page,
>> there are probably more). rsync is free software, licensed under the GPL.
>> See http://en.wikipedia.org/wiki/Rsync for more details.
>
> And, at last check as part of an attempt at an rsnapshot based backup
> system, it hangs on open files.
No it doesn't. A lot of people use it as the transport for backups in backuppc
and while it may skip files that the OS has locked, it doesn't hang when using
rsync in daemon mode.
> I had to do it my original way, which
> was to export the filesystem via CIFS, mount it and do the backups on
> the relevant rsnapshot server, because the hanging rsync issue was
> deadly.
There was an old (and long-standing) bug when using rsync under cygwin sshd that
would hang more or less at random. It is fixed in current versions. But even
in the older version you could issue the rsync command on the windows side and
use ssh with a linux/unix version under sshd on the remote side.
>> If you require encryption, rsync traffic can be encrypted using SSH or
>> stunnel (see stunnel.org). If your server runs Linux, SSH is probably
>> the easier solution. With a Windows server, stunnel might be the better
>> option since running an SSH server on Windows requires cygwin.
>
> There are some commercial SSH servers and toolkits as well. If you
> need both rsync and SSH services on the same windows server, CygWin is
> the way to go.
If you don't need encryption, you can just run rsync as a standalone daemon (use
double-colons in the rsync command for that mode). If you do need encryption
you might use OpenVPN to set up tunnels with the staging server - which might
also be useful for VNC or other control access too. Or with the current 1.7.x
version, cygwin sshd should work the way you expect.
>> I think that using a high-quality sync tool such as rsync is definitely
>> a better solution to the problem you're trying to solve.
>
> I'm mystified about using export on his core server. Do a checkout and
> svn update to a something like "SRCDIR" on the core server, to avoid
> the "this file is locked" issue, and use "rsync -av --exclude=.svn
> /SRCDIR/ REMOTEHOST:/TARGETDIR/" syntax to push only that updated
> code.
Rsync has the -C option for exactly this purpose - to exclude files that are
normally cvs or svn metadata and cruft left over from compiles. But look at the
list of the built in excludes to make sure it won't skip anything you need.
Rsync is also better than most means of copying into a live system because it
will construct a new copy of existing files under a temporary name and rename it
only when complete. That way, running programs never see inconsistent files and
they aren't left in an inconsistent state if the transfer is interrupted.
--
Les Mikesell
lesmikesell@gmail.com
Re: how to contribute feature of unknown popularity
Posted by Nico Kadel-Garcia <nk...@gmail.com>.
On Thu, Jul 15, 2010 at 6:34 AM, Stefan Sperling <st...@elego.de> wrote:
> On Wed, Jul 14, 2010 at 10:03:53PM +0000, svnusertemp@href.com wrote:
>> Re: export --skipfilesmatchingsize
>> Prior thread link: <http://svn.haxx.se/users/archive-2010-06/0040.shtml>http://svn.haxx.se/users/archive-2010-06/0040.shtml
>>
>>
>> Hi, I am the person who wanted this feature and worked on testing it
>> over recent months. It has been working well for some time, and I
>> use it often. I use it because I build a lot of deployment servers
>> where, on a daily or weekly basis, much (often more than 90%) of a
>> system stays as-is and a small amount needs to be brought in from
>> subversion. I use export (as opposed to update) to avoid having
>> .svn files scattered throughout production machines. The main
>> reason I like the skip feature is that it lets me deal with DLLs and
>> other files that are (a) unchanging and (b) in use at the time of
>> export. The normal export feature aborts when it encounters a
>> pre-existing file that cannot be overwritten because it is in use;
>> with the skip option, I can still export that branch of the
>> repository and pick up any new/additional/changed files.
>> Originally, I thought the skip feature would be a good way to save
>> bandwidth and time. In the end, that was not as critical for me as
>> being able to skip over in-use, unchanged DLLs.
>
> Hi Ann,
>
> Why don't you use rsync, which was designed to solve exactly this problem?
> rsync avoids unnecessary transfers of file data over the network,
> by transmitting only the changes necessary to sync two data sets.
>
> You could do a local export on the machine where the Subversion repository
> is located, and then use rsync to sync the exported result to your
> deployment servers.
>
> You can run rsync on windows using cygwin, or you can use one of the
> Windows ports (google mentions DeltaCopy and cwrsync on the first page,
> there are probably more). rsync is free software, licensed under the GPL.
> See http://en.wikipedia.org/wiki/Rsync for more details.
And, at last check as part of an attempt at an rsnapshot based backup
system, it hangs on open files. I had to do it my original way, which
was to export the filesystem via CIFS, mount it and do the backups on
the relevant rsnapshot server, because the hanging rsync issue was
deadly.
> If you require encryption, rsync traffic can be encrypted using SSH or
> stunnel (see stunnel.org). If your server runs Linux, SSH is probably
> the easier solution. With a Windows server, stunnel might be the better
> option since running an SSH server on Windows requires cygwin.
There are some commercial SSH servers and toolkits as well. If you
need both rsync and SSH services on the same windows server, CygWin is
the way to go.
>
> As I've said before, I object to a feature that is not general enough
> to be useful to a lot of our users. What if the next person wants svn
> export to skip files containing java code, for instance? By your logic,
> we'd then add another option called --skip-files-containing-java-code.
> Proliferation of option flags like this is a wrong approach in my opinion.
It also won't work against files that are open, for example in a text
editor, but don't happen to be ".dll" files.
> I think that using a high-quality sync tool such as rsync is definitely
> a better solution to the problem you're trying to solve.
I'm mystified about using export on his core server. Do a checkout and
svn update to a something like "SRCDIR" on the core server, to avoid
the "this file is locked" issue, and use "rsync -av --exclude=.svn
/SRCDIR/ REMOTEHOST:/TARGETDIR/" syntax to push only that updated
code.
>
> Thanks,
> Stefan
>
Re: how to contribute feature of unknown popularity
Posted by Stefan Sperling <st...@elego.de>.
On Wed, Jul 14, 2010 at 10:03:53PM +0000, svnusertemp@href.com wrote:
> Re: export --skipfilesmatchingsize
> Prior thread link: <http://svn.haxx.se/users/archive-2010-06/0040.shtml>http://svn.haxx.se/users/archive-2010-06/0040.shtml
>
>
> Hi, I am the person who wanted this feature and worked on testing it
> over recent months. It has been working well for some time, and I
> use it often. I use it because I build a lot of deployment servers
> where, on a daily or weekly basis, much (often more than 90%) of a
> system stays as-is and a small amount needs to be brought in from
> subversion. I use export (as opposed to update) to avoid having
> .svn files scattered throughout production machines. The main
> reason I like the skip feature is that it lets me deal with DLLs and
> other files that are (a) unchanging and (b) in use at the time of
> export. The normal export feature aborts when it encounters a
> pre-existing file that cannot be overwritten because it is in use;
> with the skip option, I can still export that branch of the
> repository and pick up any new/additional/changed files.
> Originally, I thought the skip feature would be a good way to save
> bandwidth and time. In the end, that was not as critical for me as
> being able to skip over in-use, unchanged DLLs.
Hi Ann,
Why don't you use rsync, which was designed to solve exactly this problem?
rsync avoids unnecessary transfers of file data over the network,
by transmitting only the changes necessary to sync two data sets.
You could do a local export on the machine where the Subversion repository
is located, and then use rsync to sync the exported result to your
deployment servers.
You can run rsync on windows using cygwin, or you can use one of the
Windows ports (google mentions DeltaCopy and cwrsync on the first page,
there are probably more). rsync is free software, licensed under the GPL.
See http://en.wikipedia.org/wiki/Rsync for more details.
If you require encryption, rsync traffic can be encrypted using SSH or
stunnel (see stunnel.org). If your server runs Linux, SSH is probably
the easier solution. With a Windows server, stunnel might be the better
option since running an SSH server on Windows requires cygwin.
As I've said before, I object to a feature that is not general enough
to be useful to a lot of our users. What if the next person wants svn
export to skip files containing java code, for instance? By your logic,
we'd then add another option called --skip-files-containing-java-code.
Proliferation of option flags like this is a wrong approach in my opinion.
I think that using a high-quality sync tool such as rsync is definitely
a better solution to the problem you're trying to solve.
Thanks,
Stefan