BitTorrent RA layer

[Ozzie Chan <oz...@gmail.com>]

Hi all,

I've recently been contemplating implementing an RA layer using the
bittorrent protocol in order to speed up large repository checkouts.

The primary impetus for this feature is to get a large development group,
geographically colocated up and running quickly. The code base is large
(~1gb), and initial checkouts are a major pain. If we could harness peer to
peer downloads, then most of this pain goes away.

Has anyone thought about this before? How difficult would it be? Is anyone
perhaps interested in coordinating an effort to do this?

Re: BitTorrent RA layer

[Peter Samuelson <pe...@p12n.org>]

[Ozzie Chan]
> I've recently been contemplating implementing an RA layer using the
> bittorrent protocol in order to speed up large repository checkouts.

I don't think it would fit the RA layer very well, honestly.  I think
what you'd do instead is seed a torrent of a full checkout, or perhaps
of a svn dump file.

Or you could come up with a protocol that is somewhat, but not
entirely, like bittorrent: each client seeks dumpfiles of all the
revisions in the repository.  They exchange these much like normal
bittorrent payloads, except that there's probably no way to come up
with the checksums in advance, so the clients would all have to trust
each other.  The repository would serve as the initial "seed", and each
client would use 'svnrdump' (a tool to generate a dumpfile over the RA
layer) to retrieve new revisions from the repository that are not
already in the BT network.

I note that this gives you a copy of the repository, which is a
superset of a checkout and may be many times larger.  Can be useful,
too, to set up a local write-through proxy via mod_dav_svn and keep it
up to date with svnsync.

Refer also to Luke Leighton's recent git->BT gateway proof of concept:

    http://lists.debian.org/debian-devel/2010/09/msg00009.html and following
    http://gitorious.org/python-libbittorrent/pybtlib

I note that git is probably better suited to the bittorrent gateway
concept than svn is, since it is changeset-oriented, and each changeset
contains and is uniquely identified by a cryptographic hash.

-- 
Peter Samuelson | org-tld!p12n!peter | http://p12n.org/