You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@subversion.apache.org by "D. Alan Stewart" <d....@mac.com> on 2006/07/29 03:45:17 UTC
large imports and tmp files
Warning! I'm a svn newbie. I so have some user-level experience with
cvs.
I'd like to use svn to maintain histories of some large binary files
(Parallels virtual hard disk images). they are typically 10-30 GB
each. I have installed the svn server and client on my OS X system.
The working files are on my laptop's boot drive, and the repository
on a 160GB secondary Firewire drive that I plug in when I'm at home.
However, I cannot even import one of these files because 'svn
import' fills up all available disk space on the boot drive. The
available free space on this drive is less than the size of the file
I want to import. Is there any way to direct where svn writes its tmp
files? From earlier postings in this list I gather that svn writes
them in the source directory. I have plenty of space on the secondary
drive if I could point svn to some location there.
I'm also wondering if I *temporarily* move data of my boot drive to
accomplish the initial svn import, will I have the same problem when
the time comes to commit changes?
TIA,
Alan
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: large imports and tmp files
Posted by "D. Alan Stewart" <d....@mac.com>.
Parallels is similar to VMWare, but unfortunately does not yet have
that product's snapshot functionality. I was hoping to use svn to
backup my virtual machines and somewhat emulate the VMWare snapshot
functionality. These images are spare, though fragmentation does
create wasted space over time.
Apparently svn is simply not suitable for what I need to do,
especially if limiting the number of previous versions stored is
going to be a problem.
Thanks.
Alan
On Jul 29, 2006, at 8:02 AM, Nico Kadel-Garcia wrote:
> First: Buy another drive, seriously. If you're going to deal with
> files *THAT* big, you want a couple of hundred Gig to work with.
>
> Second: If I were working directly with you, I'd want to slap you
> for creating that disk image structure. Seriously, unless you
> absolutely need disk images, you'll be much happier having a
> compressed tarball of the file system itself and a "make" structure
> that generates and ungenerates the disk image from the tarball, and
> vice versa. Disk images are notmally built of mostly wasted space,
> since the blank or unused bits of the filesystem are also in the
> disk image, and it makes them very awkward to work with.There's an
> occasional reason to do it for things like VMWare disk images, but
> it's messy to cope with and will explode the size of your
> repository, since your binary check-ins are likely to be *HUGE* and
> there's no graceful way throw out previous versions of a binary
> under Subversion.
>
> Third. Compress the damn thing for storage and submission to
> Subversion! Unless the image is realy jam-packed with binary data,
> you'll get a reasonable compression benefit and help speed up your
> check-outs and check-ins.
>
> Fourth: I don't see any way for you to avoid the spare files
> in .svn at checkout. It might be feasible to do an "svn export" and
> "svn import" instead, but I see such other big problems for you
> with such big file images, I'd urge you to sit down with someone
> else competent in your setup and work out a way to shrink those.
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
> For additional commands, e-mail: users-help@subversion.tigris.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: large imports and tmp files
Posted by Nico Kadel-Garcia <nk...@comcast.net>.
D. Alan Stewart wrote:
> Exactly. The virtual Windows machine won't be running. I'll shut it
> down and then backup the virtual hard drive, which is just a big
> binary file from OS X's perspective.
>
> On Jul 29, 2006, at 5:23 PM, Les Mikesell wrote:
>
>>
>> His (virtual) disk images are contained in a huge file. He could
>> use other backup methods for contents of the running virtual
>> machine, but I think he just wants the whole image file from
>> the host's perspective but with less space consumption than
>> full copies after each small change.
This actually makes sense: I've done similar things with OS image tarballs
in Linux, but always stored them as tarballs (which compress well), not as
disk images (which really, really, really do not compress well, especially
if they've been used for a while).
Subversion is the wrong tool for this, I think. The binaries are huge, and
there's no good facility in Subversion for discarding unwanted previous
revisions of a file or of alternatively named, copied files, so it'll clog
any reasonably sized repository pretty fast. I'd actually be inclined to
have a big backup drive and use something like "rsnapshot" to keep a few
backup copies stashed on a daily/weekly/monthly basis.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: large imports and tmp files
Posted by "D. Alan Stewart" <d....@mac.com>.
Exactly. The virtual Windows machine won't be running. I'll shut it
down and then backup the virtual hard drive, which is just a big
binary file from OS X's perspective.
On Jul 29, 2006, at 5:23 PM, Les Mikesell wrote:
>
> His (virtual) disk images are contained in a huge file. He could
> use other backup methods for contents of the running virtual
> machine, but I think he just wants the whole image file from
> the host's perspective but with less space consumption than
> full copies after each small change.
>
> --
> Les Mikesell
> lesmikesell@gmail.com
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: large imports and tmp files
Posted by Les Mikesell <le...@gmail.com>.
On Sat, 2006-07-29 at 15:39, Nico Kadel-Garcia wrote:
> >
> >>
> >> If what you want are backups, you might like something like
> >> http://www.nongnu.org/rdiff-backup/features.html better than
> >> a version control system.
>
> Rdiff is useless for actual disk images: it compares the contents of the
> files, not of the filesystem,
His (virtual) disk images are contained in a huge file. He could
use other backup methods for contents of the running virtual
machine, but I think he just wants the whole image file from
the host's perspective but with less space consumption than
full copies after each small change.
--
Les Mikesell
lesmikesell@gmail.com
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: large imports and tmp files
Posted by Nico Kadel-Garcia <nk...@comcast.net>.
----- Original Message -----
From: "D. Alan Stewart" <d....@mac.com>
To: "Les Mikesell" <le...@gmail.com>
Cc: <us...@subversion.tigris.org>
Sent: Saturday, July 29, 2006 4:21 PM
Subject: Re: large imports and tmp files
> Well I would like to be able to go back to previous versions of the
> virtual machine. I'm doing Windows software development and testing, and
> could conceivably hose the registry. Also, it would be nice to have a
> base machine configuration and branch from there for different projects.
> Different software would be installed on different branches.
>
> Thanks for the tip, I will investigate.
>
> Alan
>
> On Jul 29, 2006, at 12:45 PM, Les Mikesell wrote:
>
>>
>> If what you want are backups, you might like something like
>> http://www.nongnu.org/rdiff-backup/features.html better than
>> a version control system.
Rdiff is useless for actual disk images: it compares the contents of the
files, not of the filesystem, and it does it from a UNIX or UNIX-like world.
The result is that you'll lose hard-links and symlinks and file ownership
with it. If you used a FAT32 filesystem, it could work, but not for the NTFS
so common to Windows filesystems.
The same applies to tarballs for Windows images: they work great for FAT32,
not for restoring file ownership information of NTFS. It may work someday,
but reading NTFS file permissions taks some.... different structures: it's
similar to the reasons that Subversion stores symlinks, but doesn't really
reproduce them for NTFS or FAT32.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: large imports and tmp files
Posted by "D. Alan Stewart" <d....@mac.com>.
Well I would like to be able to go back to previous versions of the
virtual machine. I'm doing Windows software development and testing,
and could conceivably hose the registry. Also, it would be nice to
have a base machine configuration and branch from there for
different projects. Different software would be installed on
different branches.
Thanks for the tip, I will investigate.
Alan
On Jul 29, 2006, at 12:45 PM, Les Mikesell wrote:
>
> If what you want are backups, you might like something like
> http://www.nongnu.org/rdiff-backup/features.html better than
> a version control system.
>
> --
> Les Mikesell
> lesmikesell@gmail.com
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
> For additional commands, e-mail: users-help@subversion.tigris.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: large imports and tmp files
Posted by Les Mikesell <le...@gmail.com>.
On Sat, 2006-07-29 at 10:39, D.AlanStewart wrote:
> Parallels is similar to VMWare, but unfortunately does not yet have
> that product's snapshot functionality. I was hoping to use svn to
> backup my virtual machines and somewhat emulate the VMWare snapshot
> functionality. These images are spare, though fragmentation does
> create wasted space over time.
>
> Apparently svn is simply not suitable for what I need to do,
> especially if limiting the number of previous versions stored is
> going to be a problem.
If what you want are backups, you might like something like
http://www.nongnu.org/rdiff-backup/features.html better than
a version control system.
--
Les Mikesell
lesmikesell@gmail.com
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: large imports and tmp files
Posted by "D. Alan Stewart" <d....@mac.com>.
Parallels is similar to VMWare, but unfortunately does not yet have
that product's snapshot functionality. I was hoping to use svn to
backup my virtual machines and somewhat emulate the VMWare snapshot
functionality. These images are spare, though fragmentation does
create wasted space over time.
Apparently svn is simply not suitable for what I need to do,
especially if limiting the number of previous versions stored is
going to be a problem.
Thanks.
Alan
On Jul 29, 2006, at 8:02 AM, Nico Kadel-Garcia wrote:
> First: Buy another drive, seriously. If you're going to deal with
> files *THAT* big, you want a couple of hundred Gig to work with.
>
> Second: If I were working directly with you, I'd want to slap you
> for creating that disk image structure. Seriously, unless you
> absolutely need disk images, you'll be much happier having a
> compressed tarball of the file system itself and a "make" structure
> that generates and ungenerates the disk image from the tarball, and
> vice versa. Disk images are notmally built of mostly wasted space,
> since the blank or unused bits of the filesystem are also in the
> disk image, and it makes them very awkward to work with.There's an
> occasional reason to do it for things like VMWare disk images, but
> it's messy to cope with and will explode the size of your
> repository, since your binary check-ins are likely to be *HUGE* and
> there's no graceful way throw out previous versions of a binary
> under Subversion.
>
> Third. Compress the damn thing for storage and submission to
> Subversion! Unless the image is realy jam-packed with binary data,
> you'll get a reasonable compression benefit and help speed up your
> check-outs and check-ins.
>
> Fourth: I don't see any way for you to avoid the spare files
> in .svn at checkout. It might be feasible to do an "svn export" and
> "svn import" instead, but I see such other big problems for you
> with such big file images, I'd urge you to sit down with someone
> else competent in your setup and work out a way to shrink those.
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
> For additional commands, e-mail: users-help@subversion.tigris.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: large imports and tmp files
Posted by Nico Kadel-Garcia <nk...@comcast.net>.
D. Alan Stewart wrote:
> Warning! I'm a svn newbie. I so have some user-level experience with
> cvs.
>
> I'd like to use svn to maintain histories of some large binary files
> (Parallels virtual hard disk images). they are typically 10-30 GB
> each. I have installed the svn server and client on my OS X system.
> The working files are on my laptop's boot drive, and the repository
> on a 160GB secondary Firewire drive that I plug in when I'm at home.
>
> However, I cannot even import one of these files because 'svn
> import' fills up all available disk space on the boot drive. The
> available free space on this drive is less than the size of the file
> I want to import. Is there any way to direct where svn writes its tmp
> files? From earlier postings in this list I gather that svn writes
> them in the source directory. I have plenty of space on the secondary
> drive if I could point svn to some location there.
>
> I'm also wondering if I *temporarily* move data of my boot drive to
> accomplish the initial svn import, will I have the same problem when
> the time comes to commit changes?
First: Buy another drive, seriously. If you're going to deal with files
*THAT* big, you want a couple of hundred Gig to work with.
Second: If I were working directly with you, I'd want to slap you for
creating that disk image structure. Seriously, unless you absolutely need
disk images, you'll be much happier having a compressed tarball of the file
system itself and a "make" structure that generates and ungenerates the disk
image from the tarball, and vice versa. Disk images are notmally built of
mostly wasted space, since the blank or unused bits of the filesystem are
also in the disk image, and it makes them very awkward to work with.There's
an occasional reason to do it for things like VMWare disk images, but it's
messy to cope with and will explode the size of your repository, since your
binary check-ins are likely to be *HUGE* and there's no graceful way throw
out previous versions of a binary under Subversion.
Third. Compress the damn thing for storage and submission to Subversion!
Unless the image is realy jam-packed with binary data, you'll get a
reasonable compression benefit and help speed up your check-outs and
check-ins.
Fourth: I don't see any way for you to avoid the spare files in .svn at
checkout. It might be feasible to do an "svn export" and "svn import"
instead, but I see such other big problems for you with such big file
images, I'd urge you to sit down with someone else competent in your setup
and work out a way to shrink those.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org