You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by Martin Furter <mf...@rola.ch> on 2006/02/07 22:30:52 UTC
svnbackup.py
Hello
Attached is a backup script for use in post-commit hooks and cron jobs.
I'd like to have it included in subversion under tools/hook-scripts (or
contrib if you prefer that).
Since i have partial commit access now i can commit it myself, but i need
permission first, and some input. And ofcourse i will continue maintaining
the script.
So let's start with the story first:
About two years ago i installed subversion on one of the servers at the
company where i work and moved the code i work on to svn. I knew the first
question will be "and what about backups?" so i wrote a shell script which
made weekly full dumps and transferred them onto another server.
Six month later almost all projects have been moved to subversion, and
about a 15 month later these full dumps grew to more than 1GB, which
caused the disks to fill up pretty fast.
So i had to write a better backup script: svnbackup.py.
Now i run the following command in the weekly cronjob:
svnbackup.py -c 1000 -z /srv/svn/src /srv/svn_backup
The -c N option is the count of revisions per dumpfile, -z is for gzip.
When started the first time and HEAD is at 2923 it creates the following
files:
src.000000-000999.svndmp.gz
src.001000-001999.svndmp.gz
src.002000-002923.svndmp.gz
Say the next time HEAD is at 3045 it creates these two files:
src.002000-002999.svndmp.gz
src.003000-003045.svndmp.gz
It has a few more options, for example -r for dumping single revision from
a post-commit hook or -t to transfer the created dumpfiles to another
server using FTP (not well tested) or SMB (well tested).
The script needs python 2.4 because it uses the package 'subprocess'. I
wrote a simple compatibility function for an other tool which could be
used here too to make it run on python 2.3 and below but i don't know if
it's worth the effort.
Another problem is that the script creates temporary dump files when using
the gzip option. It would be better to pipe the output of svnadmin dump
directly into gzip, if would consume less disk space and even run faster
on my solaris box.
I've never tested the script on windows so i guess it will not work there.
Does anyone object to commit it to tools/contrib (after adding the
collabnet license) ?
Any other suggestions or wishes?
Martin
Re: svn-backup-dumps.py [Re: svnbackup.py]
Posted by Masaru Tsuchiyama <ts...@asahi-net.email.ne.jp>.
Hello.
>
> Thank you for your suggestions.
> I implemented error reporting in r19841.
>
> If you find more cases where it outputs a trace instead of an error
> message please report them.
>
Thanks.
+1 to merge this commit to 1.4.x branch.
Masaru
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: svn-backup-dumps.py [Re: svnbackup.py]
Posted by Martin Furter <mf...@rola.ch>.
Thank you for your suggestions.
I implemented error reporting in r19841.
If you find more cases where it outputs a trace instead of an error
message please report them.
Martin
On Sat, 27 May 2006, Masaru Tsuchiyama wrote:
> Hello
>
> svn-backup-dumps.py is a very useful script.
>
> I have two suggestions for it.
>
> 1. If we runs it without arguments, it shows not-user-friently
> error message. I think it must show a help message.
>
> 2. If we runs it with non-existing directory
> as <dumpdir>, it shows not-user-friently error message too.
> I think it must create dumpdir automatically or show the
> message that you must create dumpdir in advance.
>
> Thank you for your useful script.
>
> Masaru
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
svn-backup-dumps.py [Re: svnbackup.py]
Posted by Masaru Tsuchiyama <ts...@asahi-net.email.ne.jp>.
Hello
svn-backup-dumps.py is a very useful script.
I have two suggestions for it.
1. If we runs it without arguments, it shows not-user-friently
error message. I think it must show a help message.
2. If we runs it with non-existing directory
as <dumpdir>, it shows not-user-friently error message too.
I think it must create dumpdir automatically or show the
message that you must create dumpdir in advance.
Thank you for your useful script.
Masaru
-----Original Message-----
On Mon, 13 Mar 2006 04:57:40 +0100 (CET)
Martin Furter <mf...@rola.ch> wrote:
>
> Hello
>
> Now i found some time to continue the work on svn-backup-dumps.py.
>
> I think I implemented all the suggestions made, so if noone objects I'm
> gonna add it in about two days to trunk/tools/server-side.
>
> (Yes, in my first mail I said I want to add it to tools/hook-scripts, but
> meantime the directory 'server-side' popped up and I think it makes more
> sense to put it there.)
>
> Alan Barrett wrote:
> > * Make the compression program (and filename extension) configurable.
> > (I find that bzip2 compresses dumps better than gzip does.)
>
> I just added the option -b for bzip2.
>
> > * Make the use of the svnadmin dump "--deltas" option configurable.
>
> Done.
>
> Chris Foote wrote:
> > Or, you could just use the python zipfile/gzip/bz2/tarfile modules.
>
> Done. Using the modules gzip and bz2.
>
>
> On Mon, 13 Feb 2006 kfogel@collab.net wrote:
> > Just a usage() function or comment at the top of the file or something
> > like that is fine.
>
> I added a huge comment at the top of the file.
>
> Additionally I added some compatibility code for subprocess.Popen to make
> it work on python 2.3 too.
>
> Thanks.
> Martin
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: svnbackup.py
Posted by Martin Furter <mf...@rola.ch>.
Hello
Now i found some time to continue the work on svn-backup-dumps.py.
I think I implemented all the suggestions made, so if noone objects I'm
gonna add it in about two days to trunk/tools/server-side.
(Yes, in my first mail I said I want to add it to tools/hook-scripts, but
meantime the directory 'server-side' popped up and I think it makes more
sense to put it there.)
Alan Barrett wrote:
> * Make the compression program (and filename extension) configurable.
> (I find that bzip2 compresses dumps better than gzip does.)
I just added the option -b for bzip2.
> * Make the use of the svnadmin dump "--deltas" option configurable.
Done.
Chris Foote wrote:
> Or, you could just use the python zipfile/gzip/bz2/tarfile modules.
Done. Using the modules gzip and bz2.
On Mon, 13 Feb 2006 kfogel@collab.net wrote:
> Just a usage() function or comment at the top of the file or something
> like that is fine.
I added a huge comment at the top of the file.
Additionally I added some compatibility code for subprocess.Popen to make
it work on python 2.3 too.
Thanks.
Martin
Re: svnbackup.py
Posted by kf...@collab.net.
Martin Furter <mf...@rola.ch> writes:
> Yes, more documentation would be good. I haven't found docs in the
> tools dir so I'm not sure where to put it. What about svnbackup.readme
> ?
Just a usage() function or comment at the top of the file or something
like that is fine.
> I agree, the name is not very good. I'd prefer something which
> includes the word 'dump'. What about "svn-backup-dumps.py" ?
Works for me.
Thanks!
-Karl
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: svnbackup.py
Posted by Martin Furter <mf...@rola.ch>.
On Mon, 13 Feb 2006 kfogel@collab.net wrote:
> Martin Furter <mf...@rola.ch> writes:
>> Does anyone object to commit it to tools/contrib (after adding the
>> collabnet license) ?
>> Any other suggestions or wishes?
>
> I think it needs more documentation -- something like the email you
> wrote would be good :-). Also, maybe "svn-range-backup.py" or
> something like that? There are many possible backup strategies for
> Subversion, we shouldn't imply one is canonical unless it's clearly
> best for the majority of cases. (While this script will be very
> useful for certain situations, I don't think it meets that criterion.)
Yes, more documentation would be good. I haven't found docs in the tools
dir so I'm not sure where to put it. What about svnbackup.readme ?
I agree, the name is not very good. I'd prefer something which includes
the word 'dump'. What about "svn-backup-dumps.py" ?
On Wed, 8 Feb 2006, Alan Barrett wrote:
> On Tue, 07 Feb 2006, Martin Furter wrote:
>> So i had to write a better backup script: svnbackup.py.
>
> This looks very useful. I have a few suggestions:
>
> * Make the compression program (and filename extension) configurable.
> (I find that bzip2 compresses dumps better than gzip does.)
>
> * Make the use of the svnadmin dump "--deltas" option configurable.
Good ideas, i'll implement both.
>> Another problem is that the script creates temporary dump files when
>> using the gzip option. It would be better to pipe the output of
>> svnadmin dump directly into gzip, if would consume less disk space and
>> even run faster on my solaris box.
>
> Yes, that would be better.
On Fri, 10 Feb 2006, Chris Foote wrote:
> Or, you could just use the python zipfile/gzip/bz2/tarfile modules.
When i wrote the script I wanted to do that but I haven't found a streamy
version of the gzip functions. I'll take another look into the python
docs...
Martin
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: svnbackup.py
Posted by kf...@collab.net.
Martin Furter <mf...@rola.ch> writes:
> Does anyone object to commit it to tools/contrib (after adding the
> collabnet license) ?
> Any other suggestions or wishes?
I think it needs more documentation -- something like the email you
wrote would be good :-). Also, maybe "svn-range-backup.py" or
something like that? There are many possible backup strategies for
Subversion, we shouldn't imply one is canonical unless it's clearly
best for the majority of cases. (While this script will be very
useful for certain situations, I don't think it meets that criterion.)
Thoughts?
-Karl
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: svnbackup.py
Posted by Chris Foote <cf...@v21.me.uk>.
From: "Alan Barrett" <ap...@cequrux.com.apb.alt.za>
Sent: Wednesday, February 08, 2006 8:16 PM
> On Tue, 07 Feb 2006, Martin Furter wrote:
> > So i had to write a better backup script: svnbackup.py.
>
> This looks very useful. I have a few suggestions:
>
> * Make the compression program (and filename extension) configurable.
> (I find that bzip2 compresses dumps better than gzip does.)
>
Or, you could just use the python zipfile/gzip/bz2/tarfile modules.
[snip]
> --apb (Alan Barrett)
>
Chris
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: svnbackup.py
Posted by Alan Barrett <ap...@cequrux.com.apb.alt.za>.
On Tue, 07 Feb 2006, Martin Furter wrote:
> So i had to write a better backup script: svnbackup.py.
This looks very useful. I have a few suggestions:
* Make the compression program (and filename extension) configurable.
(I find that bzip2 compresses dumps better than gzip does.)
* Make the use of the svnadmin dump "--deltas" option configurable.
> Another problem is that the script creates temporary dump files when
> using the gzip option. It would be better to pipe the output of
> svnadmin dump directly into gzip, if would consume less disk space and
> even run faster on my solaris box.
Yes, that would be better.
--apb (Alan Barrett)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org