You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by Greg Stein <gs...@lyra.org> on 2001/04/02 06:49:20 UTC
Linux Kernel Summit
Hey all,
I went to the Linux Kernel Summit on Friday evening, to speak about
Subversion at one of the BOFs. Linus is interested in Subversion for a
number of reasons, but also wanted a counterpoint to BitKeeper.
Larry McVoy was there when I first arrived, but had to be back home by 7pm
(an hour away, so he should have left by 6pm), so he couldn't stay for a
scheduled 9pm BitKeeper BOF. He left somewhere around 8pm, so I'm sure he's
in trouble :-) ... but it was unfortunate because Linus wanted to see some
blood :-)
Ah well. Back to SVN itself. I talked about SVN, its model, the advantages,
some of the architecture, etc. I think the three basic points of feedback
that I received are:
* Use some kind of checksum on the wire and in the DB. Given that whatever
is in source control is probably quite important, it would be a Good Thing
if we hashed/checksummed the data. Even better if we periodically reviewed
the whole database for items whose checksum has mysteriously changed or
failed. The point is that bits can corrupt over the N year period that the
data resides in source control. Catching the corruption sooner rather than
later makes it easier to get back to a correct state. Imagine if you tried
to get a checkin from four years ago, but it had been corrupted three
years back? You probably don't have a valid version anywhere.
I suggested that the client could compute an MD5 hash, send it over the
wire using HTTP's Content-MD5 header, Apache would verify it (built in!),
and that we'd place the hash into the database with the data. When the
data is retrieved, we can send it back over the wire with Content-MD5 and
the client can verify the hash value.
* Diff formats. Our SVNDIFF format is neato, but it can only change a
specific input. It has no context, so it does not apply to the typical use
case of mailing changes around, for application against arbitrary inputs.
(Sam TH just emailed about this use case)
People were somewhat leery of the XML form (even asking whether we stored
things that way). If we intend to use it for passing around patches, then
we'd want to consider allowing for content/unified diff formats inside the
XML (see foonote 1). They weren't so much bothered by XML, as by its
readability. I believe that we can create a readable XML-based patch, but
we do need to use other diff formats in there.
Our diff story also needs to be clarified for people; it was quite a
lengthy discussion. XML and SVNDIFFs are for transforming specific trees.
Something else (what?) is for mailing contextual patches.
Linus said he had recently seen an extension to the PATCH format which
started off with things like "mv foo.c bar.c" to be able to capture tree
types of changes. We may want to try to find that format and make it an
option.
* Merging. People are definitely interested in some of the "genetic" merging
and being able to do merging right.
They'd also like to be able to see a patch format that can embody a
sequence of revision patches, each with their own log message. This kind
of patch would be useful for people developing code independently and then
merging changes (with history) into each others' trees. Consider it a
"Poor Man's Distributed Repository."
That's about it. I also got an earful from Larry about not trusting anything
(see fn 2). Not even the filesystem. I don't agree with his paranoia level,
but we can ameliorate some of that using the MD5 hashes.
And as Eric posted here already, I also got an earful from him about
metadata not in a textual/editable format (e.g. using DB). Jim has already
responsed quite well to that.
There was a short discussion about licenses, and I simply pointed out that
SVN uses the Apache License, that it is completely free, and it will always
be free. There was a question raised about whether we might possibly be
infringing on some patents, and I responded that we didn't know of any. The
person suggested that Larry/BitMover may have some patents in this area, but
someone else suggest that was a rumor, and I can find nothing on the
Internet (Larry?).
If you have any questions about the above stuff, want some more drill down,
etc, then (of course) just send an email.
Cheers,
-g
(1) diff type selection is one of the things that changing txdelta's
direction can do. The piece of code writing out the XML data can choose
whatever format is appropriate (or selected by the user).
(2) I found a posting from Larry that appears to state his position
reasonably well:
http://boudicca.tux.org/hypermail/linux-kernel/1999/1999week40/0318.html
--
Greg Stein, http://www.lyra.org/
Re: Linux Kernel Summit
Posted by Karl Fogel <kf...@collab.net>.
What he said, especially the last paragraph:
Jim Blandy <ji...@zwingli.cygnus.com> writes:
> I don't think Subversion should be involved in:
> 1) checking that bits were transmitted across the network without
> corruption, or
> 2) checking that bits were stored on disk without corruption.
>
> These are problems which users can address (if they like) by using
> RAID, using hard drives with extensive parity checking, using better
> network media, IPsec MAC facilities (which will detect accidental
> corruption just as well as deliberate corruption), etc. Completely
> generic solutions benefit Subversion as much as anything we could
> hard-code into Subversion itself. They're outside of Subversion's
> responsibility.
>
> Subversion *should* use checksums or error-correcting information
> where we might make mistakes ourselves --- as in the computation and
> application of text deltas. Which is, in fact, where we're using them
> now.
Re: Linux Kernel Summit
Posted by Jim Blandy <ji...@zwingli.cygnus.com>.
Lim Swee Tat <st...@stlim.net> writes:
> I made the suggestion in the hopes of seeing a rather inexpensive
> checksum that will perhaps perform error correction, if such needs
> arise.
> Is it possible to make this checksum transparently so that we can then
> have both data integrity and error correction? But I guess under the
> current implementation of the Berkeley DB, this rather difficult since it
> will involve slipping a layer under the BerkeleyDB, rite??
> I understand the need to keep it simple, but if the tradeoff for
> getting error correction system that is already taking time doing
> checksums is minor, I'm willing to trade speed for the error correction
> capability.
I don't think Subversion should be involved in:
1) checking that bits were transmitted across the network without
corruption, or
2) checking that bits were stored on disk without corruption.
These are problems which users can address (if they like) by using
RAID, using hard drives with extensive parity checking, using better
network media, IPsec MAC facilities (which will detect accidental
corruption just as well as deliberate corruption), etc. Completely
generic solutions benefit Subversion as much as anything we could
hard-code into Subversion itself. They're outside of Subversion's
responsibility.
Subversion *should* use checksums or error-correcting information
where we might make mistakes ourselves --- as in the computation and
application of text deltas. Which is, in fact, where we're using them
now.
Re: error correction (was: Re: Linux Kernel Summit)
Posted by Lim Swee Tat <st...@stlim.net>.
Roger.
At [Mon, Apr 02, 2001 at 11:50:16PM -0700-4513- %H:hrs], Greg Stein <gs...@lyra.org> wrote:
> Date: Mon, 2 Apr 2001 23:50:16 -0700
> From: Greg Stein <gs...@lyra.org>
> To: Lim Swee Tat <st...@stlim.net>
> Cc: dev@subversion.tigris.org
> Subject: error correction (was: Re: Linux Kernel Summit)
>
> The kinds of errors that we're talking about are likely to take down large
> chunks of data at a time. Error correcting for them is a somewhat complex
> process (as described by GregH below).
>
> Personally, I'd say our answer for 1.0 is simply "error correction is
> performed by using RAID drives and a solid backup methodology." We will
> implement error detection so that the admin knows that a restore from backup
> is necessary. This is a simple strategy which allows us to deliver the
> software when people need it: soon.
>
> Cheers,
> -g
>
> On Tue, Apr 03, 2001 at 02:40:42PM +0800, Lim Swee Tat wrote:
> > Hi,
> > I made the suggestion in the hopes of seeing a rather inexpensive
> > checksum that will perhaps perform error correction, if such needs
> > arise.
> > Is it possible to make this checksum transparently so that we can then
> > have both data integrity and error correction? But I guess under the
> > current implementation of the Berkeley DB, this rather difficult since it
> > will involve slipping a layer under the BerkeleyDB, rite??
> > I understand the need to keep it simple, but if the tradeoff for
> > getting error correction system that is already taking time doing
> > checksums is minor, I'm willing to trade speed for the error correction
> > capability.
> >
> > Ciao
> > ST Lim
> >
> > At [Tue, Apr 03, 2001 at 01:47:41AM ], Greg Hudson <gh...@MIT.EDU> wrote:
> > > To: Jim Blandy <ji...@zwingli.cygnus.com>
> > > cc: dev@subversion.tigris.org
> > > Subject: Re: Linux Kernel Summit
> > > Date: Tue, 03 Apr 2001 01:47:41 -0400
> > > From: Greg Hudson <gh...@MIT.EDU>
> > >
> > > (Note: I'm not advocating we actually do any of this.)
> > >
> > > > (Now, I haven't really thought this through, but it seems to me that
> > > > any error-correcting data would have to be proportional in length to
> > > > the thing it was capable of detecting and correcting errors in.
> > > > That would kind of defeat the purpose of using deltas to begin
> > > > with.)
> > >
> > > Not at all. Consider this "simple" approach (it's not simple or
> > > adviseable in practice because it would involve slipping a layer
> > > underneath the Berkeley DB, but it's still theoretically sound):
> > >
> > > * Include in each page of the database a checksum of the page,
> > > such that you know if a page has been corrupted.
> > >
> > > * After every block of N pages in the database, keep an extra
> > > "parity page" which contains a bitwise parity calculation of
> > > the block of pages.
> > >
> > > Now if any single page is corrupted, you blow away that page and
> > > recompute it using the parity page and the other pages in that block.
> > > Of course, corruption would probably happen across a swath of pages,
> > > so instead of having a parity block across a sequential group of
> > > blocks you'd keep it across a non-local group of blocks (blocks 1,
> > > 1001, 2001, ...), or across blocks which live on different spindles.
> > > The space cost is 1/N times the previous size of your database, plus
> > > the cost of the page checksums.
> > >
> > > This is all basic RAID stuff, with checksums thrown in because RAID
> > > normally assumes that disk failure are detectable. There exist more
> > > advanced error correction techniques which can recover from more
> > > complicated failures.
> > ------------------------------------------------------------------
> > | Lim Swee Tat | Office Automation: The use of computers |
> > | 3ui Pte Ltd | to improve efficiency in the office by |
> > | 10 Anson Road | removing anyone you would want to talk |
> > | International Plaza | with over coffee. |
> > | #05-17 | |
> > | Singapore 079903 | |
> > | +65-220-7529 ext 26 | |
> > ------------------------------------------------------------------
>
> --
> Greg Stein, http://www.lyra.org/
------------------------------------------------------------------
| Lim Swee Tat | Work consists of whatever a body is |
| 3ui Pte Ltd | obliged to do. Play consists of |
| 10 Anson Road | whatever a body is not obliged to |
| International Plaza | do. -- Mark Twain |
| #05-17 | |
| Singapore 079903 | |
| +65-220-7529 ext 26 | |
------------------------------------------------------------------
error correction (was: Re: Linux Kernel Summit)
Posted by Greg Stein <gs...@lyra.org>.
The kinds of errors that we're talking about are likely to take down large
chunks of data at a time. Error correcting for them is a somewhat complex
process (as described by GregH below).
Personally, I'd say our answer for 1.0 is simply "error correction is
performed by using RAID drives and a solid backup methodology." We will
implement error detection so that the admin knows that a restore from backup
is necessary. This is a simple strategy which allows us to deliver the
software when people need it: soon.
Cheers,
-g
On Tue, Apr 03, 2001 at 02:40:42PM +0800, Lim Swee Tat wrote:
> Hi,
> I made the suggestion in the hopes of seeing a rather inexpensive
> checksum that will perhaps perform error correction, if such needs
> arise.
> Is it possible to make this checksum transparently so that we can then
> have both data integrity and error correction? But I guess under the
> current implementation of the Berkeley DB, this rather difficult since it
> will involve slipping a layer under the BerkeleyDB, rite??
> I understand the need to keep it simple, but if the tradeoff for
> getting error correction system that is already taking time doing
> checksums is minor, I'm willing to trade speed for the error correction
> capability.
>
> Ciao
> ST Lim
>
> At [Tue, Apr 03, 2001 at 01:47:41AM ], Greg Hudson <gh...@MIT.EDU> wrote:
> > To: Jim Blandy <ji...@zwingli.cygnus.com>
> > cc: dev@subversion.tigris.org
> > Subject: Re: Linux Kernel Summit
> > Date: Tue, 03 Apr 2001 01:47:41 -0400
> > From: Greg Hudson <gh...@MIT.EDU>
> >
> > (Note: I'm not advocating we actually do any of this.)
> >
> > > (Now, I haven't really thought this through, but it seems to me that
> > > any error-correcting data would have to be proportional in length to
> > > the thing it was capable of detecting and correcting errors in.
> > > That would kind of defeat the purpose of using deltas to begin
> > > with.)
> >
> > Not at all. Consider this "simple" approach (it's not simple or
> > adviseable in practice because it would involve slipping a layer
> > underneath the Berkeley DB, but it's still theoretically sound):
> >
> > * Include in each page of the database a checksum of the page,
> > such that you know if a page has been corrupted.
> >
> > * After every block of N pages in the database, keep an extra
> > "parity page" which contains a bitwise parity calculation of
> > the block of pages.
> >
> > Now if any single page is corrupted, you blow away that page and
> > recompute it using the parity page and the other pages in that block.
> > Of course, corruption would probably happen across a swath of pages,
> > so instead of having a parity block across a sequential group of
> > blocks you'd keep it across a non-local group of blocks (blocks 1,
> > 1001, 2001, ...), or across blocks which live on different spindles.
> > The space cost is 1/N times the previous size of your database, plus
> > the cost of the page checksums.
> >
> > This is all basic RAID stuff, with checksums thrown in because RAID
> > normally assumes that disk failure are detectable. There exist more
> > advanced error correction techniques which can recover from more
> > complicated failures.
> ------------------------------------------------------------------
> | Lim Swee Tat | Office Automation: The use of computers |
> | 3ui Pte Ltd | to improve efficiency in the office by |
> | 10 Anson Road | removing anyone you would want to talk |
> | International Plaza | with over coffee. |
> | #05-17 | |
> | Singapore 079903 | |
> | +65-220-7529 ext 26 | |
> ------------------------------------------------------------------
--
Greg Stein, http://www.lyra.org/
Re: Linux Kernel Summit
Posted by Lim Swee Tat <st...@stlim.net>.
Hi,
I made the suggestion in the hopes of seeing a rather inexpensive
checksum that will perhaps perform error correction, if such needs
arise.
Is it possible to make this checksum transparently so that we can then
have both data integrity and error correction? But I guess under the
current implementation of the Berkeley DB, this rather difficult since it
will involve slipping a layer under the BerkeleyDB, rite??
I understand the need to keep it simple, but if the tradeoff for
getting error correction system that is already taking time doing
checksums is minor, I'm willing to trade speed for the error correction
capability.
Ciao
ST Lim
At [Tue, Apr 03, 2001 at 01:47:41AM ], Greg Hudson <gh...@MIT.EDU> wrote:
> To: Jim Blandy <ji...@zwingli.cygnus.com>
> cc: dev@subversion.tigris.org
> Subject: Re: Linux Kernel Summit
> Date: Tue, 03 Apr 2001 01:47:41 -0400
> From: Greg Hudson <gh...@MIT.EDU>
>
> (Note: I'm not advocating we actually do any of this.)
>
> > (Now, I haven't really thought this through, but it seems to me that
> > any error-correcting data would have to be proportional in length to
> > the thing it was capable of detecting and correcting errors in.
> > That would kind of defeat the purpose of using deltas to begin
> > with.)
>
> Not at all. Consider this "simple" approach (it's not simple or
> adviseable in practice because it would involve slipping a layer
> underneath the Berkeley DB, but it's still theoretically sound):
>
> * Include in each page of the database a checksum of the page,
> such that you know if a page has been corrupted.
>
> * After every block of N pages in the database, keep an extra
> "parity page" which contains a bitwise parity calculation of
> the block of pages.
>
> Now if any single page is corrupted, you blow away that page and
> recompute it using the parity page and the other pages in that block.
> Of course, corruption would probably happen across a swath of pages,
> so instead of having a parity block across a sequential group of
> blocks you'd keep it across a non-local group of blocks (blocks 1,
> 1001, 2001, ...), or across blocks which live on different spindles.
> The space cost is 1/N times the previous size of your database, plus
> the cost of the page checksums.
>
> This is all basic RAID stuff, with checksums thrown in because RAID
> normally assumes that disk failure are detectable. There exist more
> advanced error correction techniques which can recover from more
> complicated failures.
------------------------------------------------------------------
| Lim Swee Tat | Office Automation: The use of computers |
| 3ui Pte Ltd | to improve efficiency in the office by |
| 10 Anson Road | removing anyone you would want to talk |
| International Plaza | with over coffee. |
| #05-17 | |
| Singapore 079903 | |
| +65-220-7529 ext 26 | |
------------------------------------------------------------------
Re: Linux Kernel Summit
Posted by Jim Blandy <ji...@zwingli.cygnus.com>.
Lim Swee Tat <st...@stlim.net> writes:
> I'm not very sure what you propose the use of the checksum should be
> for. Transmission over wire?? or to verify data corruption on data that
> has been stored on disk.
The latter. Older revisions of texts are represented as deltas to be
applied to the fulltext of the next younger revision; the youngest
revision is always stored as fulltext. (There are many other ways to
structure this, with different time/space tradeoffs; we're keeping
things simple to begin with.)
This means that, as newer revisions are committed, the representation
of a particular revision gets "smeared" out over more and more younger
revisions: If the youngest revision is Y, the representation of an
older revision O depends on Y's text, and Y-O deltas. (This is the
way RCS and CVS work, too. SCCS (and thus Bitkeeper?) do things
differently.)
The checksum serves to reassure us that this smearing process hasn't
actually done any damage to the file contents. It's there to catch
bugs.
We could, of course, go much further with error-detecting and
-correcting techniques, but it seems to me like a waste of time. What
we need is some way to at least notice that something has gone wrong.
Our recovery strategy is for the sysadmin to go get the backups.
(Now, I haven't really thought this through, but it seems to me that
any error-correcting data would have to be proportional in length to
the thing it was capable of detecting and correcting errors in. That
would kind of defeat the purpose of using deltas to begin with.)
Re: Linux Kernel Summit
Posted by Lim Swee Tat <st...@stlim.net>.
Hi folks,
I've been following this thread with interest.
I'm not very sure what you propose the use of the checksum should be
for. Transmission over wire?? or to verify data corruption on data that
has been stored on disk.
If you are considering the latter, MD5 will only be able to perform
the verification of data corruption on the data. It will be unable to
restore data to the correct context in the event of data corruption. Is
there any other hashes that may be perhaps used instead of MD5 for data
on the disk?? In the event that such a hash proves to be sufficiently
useful, can we then use it to hash the data when transmitting over
wire??
Ciao
ST Lim
------------------------------------------------------------------
| Lim Swee Tat | Just because you like my stuff doesn't |
| 3ui Pte Ltd | mean I owe you anything. -- Bob Dylan |
| 10 Anson Road | |
| International Plaza | |
| #05-17 | |
| Singapore 079903 | |
| +65-220-7529 ext 26 | |
------------------------------------------------------------------
RE: Linux Kernel Summit
Posted by Gavin Thomas Nicol <gt...@ebt.com>.
> MD5 is indeed a perfectly good checksum algorithm -- in fact, the
> properties that make it good for crypto are exactly those that make it
> good for detecting data corruption.
Another good one is tiger...
http://www.cs.technion.ac.il/~biham/Reports/Tiger/
An implementation is in gnupg I think.
Re: Linux Kernel Summit
Posted by "Eric S. Raymond" <es...@thyrsus.com>.
Greg Hudson <gh...@MIT.EDU>:
> The terms [hash and checksum] are used interchangeably as far as I
> know. If there is a difference, it would relate to the use of such
> algorithms in authentication systems, which isn't what we're doing.
> For our purposes, MD5 makes a perfectly good checksum algorithm.
A checksum is a way of using a hash function -- the latter labels what
you're doing, the former why you're doing it.
MD5 is indeed a perfectly good checksum algorithm -- in fact, the
properties that make it good for crypto are exactly those that make it
good for detecting data corruption.
--
<a href="http://www.tuxedo.org/~esr/">Eric S. Raymond</a>
The end move in politics is always to pick up a gun.
-- R. Buckminster Fuller
Checksums vs. Hashes (was Re: Linux Kernel Summit)
Posted by "Deven T. Corzine" <de...@ties.org>.
On Mon, 2 Apr 2001, Greg Stein wrote:
> 2) MD5 is a hash algorithm rather than a checksum
> (although, now that I think of it, is any checksum algorithm anything but
> a hash? mebbe GregH can clarify the hash vs checksum terminology)
There's no reason not to use MD5 or SHA "hash algorithms" as checksums.
Any function with a smaller range than domain qualifies as a hash function,
even the useless "hash" function that returns a constant value. Thus, any
checksum algorithm qualifies as a hash function. Whether or not a given
hash function might be described as a "checksum algorithm" depends on what
the intended purpose of the function is, and how well it detects errors.
A checksum is generally designed to preserve data integrity by detecting
*accidental* corruption of the data. That's why the simplest checksum
algorithms just add together the bytes (tossing or wrapping the overflow)
and "check the sum" against the known value. Of course, CRC-32 became much
more common, because it did a better job of catching more errors than a
simplistic checksum. But they're all hash functions.
There are hash functions that would do a poor job of detecting corrupted
data, and hence would never be described as "checksum" algorithms. Suppose
you have a simplistic checksum routine that gives you an 8-bit additive
checksum. Then divide that 8-bit number by 11 (a prime number) and take
the remainder (modulo operation). The result is only in the range of 0-10,
and would have a 9% chance of returning the same result for corrupted data.
This would be a truly lousy checksum algorithm, but it might be acceptable
to select hash buckets in a small, simplistic hash table implementation --
it's also a hash function, but nobody in their right mind would call it a
"checksum" algorithm, simply because it would perform that role so poorly!
MD5, however, is designed to be a cryptographically-secure hash algorithm,
which means it is designed to detect *malicious* corruption of data as well
as accidental corruption. This is much more difficult, and there is always
some risk of weaknesses in the algorithm, as with encryption algorithms.
In fact, there are some potential weaknesses in MD5 that have been found,
but it hasn't been completely broken. I believe that SHA is considered
more secure than MD5 these days. A typical feature of cryptographically-
secure hash alrogithms is that even tiny changes in the original data will
often change a large proportion of the bits in the hash result. (Also,
cryptographically-secure hash algorithms often use 128 bits or more for the
result, while checksum algorithms more commonly use 32 bits or less.)
Since cryptographically-secure hash algorithms are generally used as a tool
to "fingerprint" data, the uses are a bit more broad than the traditional
error-checking purpose of a checksum, which is probably why they're not
generally called "checksum algorithms". Nevertheless, they should actually
serve quite well as checksums. The main downside of using MD5 (128 bits),
SHA (160 bits) or a similar algorithm, is the additional space needed
beyond traditional checksums like CRC-32 (32 bits). Also, analysis of the
algorithms is more likely to have focused on resistance to attack rather
than error-checking efficacy, but their larger size should make them more
effective than they would be with fewer bits.
MD5 or SHA should make a fine checksum, and may protect against malicious
changes as well. (SHA may be slightly preferable for that.)
Deven
Re: Linux Kernel Summit
Posted by Greg Stein <gs...@lyra.org>.
On Mon, Apr 02, 2001 at 06:24:00AM -0500, Jim Blandy wrote:
> Greg Stein <gs...@lyra.org> writes:
> > * Use some kind of checksum on the wire and in the DB. Given that whatever
>...
> I think someone pointed this out a while back. The skels for
> deltafied storage include checksums. There's no external FS interface
> for retrieving them at the moment, but that's a few minutes' work.
Ah. I see that in "structure". Two points there:
1) the checksum probably ought to go on both clear text and on deltas
2) MD5 is a hash algorithm rather than a checksum
(although, now that I think of it, is any checksum algorithm anything but
a hash? mebbe GregH can clarify the hash vs checksum terminology)
We would then want an API to set the checksum, and one to fetch. I'm not
sure that the FS needs to compute or verify the value at any point. The
client will compute it, Apache will verify it, and a (weekly) cron job can
scan the database verifying records against their stated checksums.
Cheers,
-g
--
Greg Stein, http://www.lyra.org/
Re: Linux Kernel Summit
Posted by Jim Blandy <ji...@zwingli.cygnus.com>.
Greg Stein <gs...@lyra.org> writes:
> * Use some kind of checksum on the wire and in the DB. Given that whatever
> is in source control is probably quite important, it would be a Good Thing
> if we hashed/checksummed the data. Even better if we periodically reviewed
> the whole database for items whose checksum has mysteriously changed or
> failed. The point is that bits can corrupt over the N year period that the
> data resides in source control. Catching the corruption sooner rather than
> later makes it easier to get back to a correct state. Imagine if you tried
> to get a checkin from four years ago, but it had been corrupted three
> years back? You probably don't have a valid version anywhere.
>
> I suggested that the client could compute an MD5 hash, send it over the
> wire using HTTP's Content-MD5 header, Apache would verify it (built in!),
> and that we'd place the hash into the database with the data. When the
> data is retrieved, we can send it back over the wire with Content-MD5 and
> the client can verify the hash value.
I think someone pointed this out a while back. The skels for
deltafied storage include checksums. There's no external FS interface
for retrieving them at the moment, but that's a few minutes' work.
Re: Linux Kernel Summit
Posted by Tripp Lilley <tl...@perspex.com>.
On Mon, 2 Apr 2001, Greg Stein wrote:
> I tend to agree here. But if it is cheap, using HTTP systems, then it
> shouldn't be a problem for us [to satisfy the overly paranoid]. It isn't
> going to help with TCP (which is already checksum'd), but will help with
> problems in an end-to-end connection (i.e. a proxy that mucks it up). As you
> point out, though, I'm not aware of any observances of this, nor of any
> broken TCP stacks that don't checksum and verify the result.
>
> [ hmm. actually, after I wrote that, I seem to recall there *was* a stack
> somewhere that didn't check the checksums... ]
Actually, at Interop we discovered a surprising number of "in the middle"
devices (proxies, caches, firewalls, traffic shapers) that ignored the TCP
checksum, or recomputed it incorrectly, or silently changed data -and-
recomputed the checksum (so the receiving end wasn't aware that anything
was wrong).
"surprising" in this context is evaluated as "more than zero", but I
neither remember exact numbers, nor remember nor would be allowed to say
which vendors or products.
But it happens. So it's best to guard against it. Which sucks.
--
Joy-Loving * Tripp Lilley * http://stargate.eheart.sg505.net/~tlilley/
------------------------------------------------------------------------------
"Fiber makes you poop." -- From <http://www.pvponline.com/bts_studio.php3>
Re: Linux Kernel Summit
Posted by Greg Stein <gs...@lyra.org>.
On Mon, Apr 02, 2001 at 12:32:32PM -0400, Greg Hudson wrote:
> > * Use some kind of checksum on the wire and in the DB. Given that whatever
> > is in source control is probably quite important, it would be a Good Thing
> > if we hashed/checksummed the data.
>
> For checksums on the wire: I am really hesitant to solve problems
> which (a) haven't been observed in the past (to my knowledge), and (b)
> are supposed to be solved by our transport layer. (I'm assuming here
> that CVS doesn't verify wire transfers with checksums; I could be
> wrong.) On the other hand, if it can be done purely using HTTP
> mechanisms, it wouldn't be the end of the world to do so.
I tend to agree here. But if it is cheap, using HTTP systems, then it
shouldn't be a problem for us [to satisfy the overly paranoid]. It isn't
going to help with TCP (which is already checksum'd), but will help with
problems in an end-to-end connection (i.e. a proxy that mucks it up). As you
point out, though, I'm not aware of any observances of this, nor of any
broken TCP stacks that don't checksum and verify the result.
[ hmm. actually, after I wrote that, I seem to recall there *was* a stack
somewhere that didn't check the checksums... ]
> For checksums in the DB: Larry makes a good argument that if you're
> storing very valuble data, you want checksums (or some other form of
> redundancy) in your permanent storage, and you want to periodically
> verify that none of the data is corrupt. On the other hand, we're not
> particularly targeting very expensive data; we're targeting the open
> source community, according to our web pages. So, I would say: we
> should ensure that our DB format is extensible enough to support
> adding per-revision fields, and we should treat this as a "some day"
> feature unless someone feels inspired to implement it right away.
Yup. And Jim just pointed out that we have something here already. I think
we should be able to do this with a pretty low impact.
> > * Diff formats.
>
> I feel like you're conflating the XML delta format with svndiff.
Um. I was trying to say that they *aren't* conflated. That we need to be
able to use arbitrary formats in there. And that SVNDIFF is useful in
certain cases (e.g. shipping diffs over the wire), but we need to have
multiple formats.
> We've always intended to use gnudiff format for passing stuff around,
> and it can be embedded in an XML delta just as easily as svndiff can.
> (Our code currently only deals with svndiffs, but that's a temporary
> state.)
All righty. I wasn't aware of our intent there. (thx!)
> I think you're correct, though, that the editor interface has to
> change in order to produce anything other than svndiffs. (It doesn't
> have to change to a pull model, but text delta windows aren't going to
> hack it.)
Yup. We could do it with something other than "pull", but pull solves other
problems for us. Specifically, it allows an editor implementor to use pull
*or* push, it allows Neon to pull data, it allows the editor to choose the
format that is "right" for it, etc.
Cheers,
-g
--
Greg Stein, http://www.lyra.org/
Re: Linux Kernel Summit
Posted by Joe Orton <jo...@btconnect.com>.
On Mon, Apr 02, 2001 at 12:32:32PM -0400, Greg Hudson wrote:
> > * Use some kind of checksum on the wire and in the DB. Given that whatever
> > is in source control is probably quite important, it would be a Good Thing
> > if we hashed/checksummed the data.
>
> For checksums on the wire: I am really hesitant to solve problems
> which (a) haven't been observed in the past (to my knowledge), and (b)
> are supposed to be solved by our transport layer. (I'm assuming here
> that CVS doesn't verify wire transfers with checksums; I could be
> wrong.) On the other hand, if it can be done purely using HTTP
> mechanisms, it wouldn't be the end of the world to do so.
HTTP digest authentication does this; it uses MD5 checksums for request
and response bodies.
joe