Are svn_fs_dir_t and svn_fs_file_t worth it?

[Jim Blandy <ji...@savonarola.red-bean.com>]

At the moment, the filesystem interface has three types that refer to
objects in the filesystem:

   svn_fs_dir_t --- a directory
   svn_fs_file_t --- a file
   svn_fs_node_t --- either a directory or a file

There is a set of systematically named functions for casting dirs and
files to nodes, and casting nodes to either dirs or files; the latter
functions indicate an error if the node is not actually a dir or file.

The original motivations for providing these three types in their
little hierarchy were these:
- In many cases, the compiler can tell you if you've passed a 
  directory when a file is required, or vice versa, at compile time.
- It makes function prototypes somewhat self-documenting.
- Even though every node is a directory or file, having a type for
  nodes lets you define a single operation that works on all kinds of
  nodes.

However, in the filesystem code itself, I've encountered the following
consequences:
- Calls to the casting functions are pretty frequent, even though they
  serve no function other than making the static typing work out.
  It's somewhat obscure.
- If I need to use functions specific to a given node kind, and also
  functions that work on any kind of node, I end up with making a
  `convenience' (heh.) copy of the variable:

      svn_fs_dir_t *foo_dir;
      svn_fs_node_t *foo_node = svn_fs_dir_to_node (foo_dir);

  It's pretty bureaucratic.

So I think I'd like to eliminate svn_fs_dir_t and svn_fs_file_t, and
use svn_fs_node_t throughout.  It would make svn_fs_node_t more like
Unix file descriptors --- a uniform way to reference whatever's out
there.

What do folks think?  Greg, I'm especially interested in your opinion,
since you've actually written code that uses the fs interface.

Re: dir/file plus more (was: Re: Are svn_fs_dir_t and svn_fs_file_t worth it?)

[Jim Blandy <ji...@zwingli.cygnus.com>]

> *) svn_fs__file_from_skel() appears to copy the file contents into memory.
>    This isn't going to scale to multi-megabyte (or gigabyte!) files. It
>    would be good to have a way that directly maps from DB3 to a seek/read
>    function. The ideal interface for me will allow me to seek to a point in
>    the "stream" and then read "n" bytes from it. Preferably, the read would
>    not allocate memory (say, if DB3 mmap'd the record, then I'd just get a
>    pointer into that mamp).
> 
>    Basically, I'm looking at a case where we have a gigabyte file stored
>    into Subversion. The client requests bytes 100-120 and bytes
>    2000000-2000100. For optimum Apache behavior, I could get just those
>    bytes without any memory allocation. (beyond what mmap does, or possibly
>    reading from a file descriptor into some allocated memory (of just the
>    right size))
> 
>    What kinds of mechanisms does DB3 support for reading large content? Can
>    you give me a pointer to the doc/API? (so that I can be a bit more
>    intelligent in my request here)

Berkeley DB does have mechanisms for reading and writing partial
records; I've included the page from the manual below.  However, these
don't really solve the problem.  If we have a billion-byte file stored
as a delta against another billion-byte file, we need our delta
application routines to support random access.  That should be fun.




 Berkeley DB Reference Guide:                           [Prev] [Ref][Next]
      Access Methods

                    Partial record storage and retrieval

It is possible to both store and retrieve parts of data items in all
Berkeley DB access methods. This is done by specifying the DB_DBT_PARTIAL
flag to the DBT structure passed to the Berkeley DB interface.

The DB_DBT_PARTIAL flag is based on the values of two elements of the DBT
structure, dlen and doff. The value of dlen is the number of bytes of the
record in which the application is interested. The value of doff is the
offset from the beginning of the data item where those bytes start.

For example, if the data item were ABCDEFGHIJKL, a doff value of 3 would
indicate that the bytes of interest started at D, and a dlen value of 4
would indicate that the bytes of interest were DEFG.

When retrieving a data item from a database, the dlen bytes starting doff
bytes from the beginning of the record are returned, as if they comprised
the entire record. If any or all of the specified bytes do not exist in the
record, the retrieval is still successful and the existing bytes or nul
bytes are returned.

When storing a data item into the database, the dlen bytes starting doff
bytes from the beginning of the specified key's data record are replaced by
the data specified by the data and size structure elements. If dlen is
smaller than size, the record will grow, and if dlen is larger than size,
the record will shrink. If the specified bytes do not exist, the record will
be extended using nul bytes as necessary, and the store call will still
succeed.

The following are various examples of the put case for the DB_DBT_PARTIAL
flag. In all examples, the initial data item is 20 bytes in length:

ABCDEFGHIJ0123456789

  1.      size = 20
          doff = 0
          dlen = 20
          data = abcdefghijabcdefghij

          Result: The 20 bytes at offset 0 are replaced by the 20 bytes of data,
          i.e., the entire record is replaced.

          ABCDEFGHIJ0123456789 -> abcdefghijabcdefghij

  2.      size = 10
          doff = 20
          dlen = 0
          data = abcdefghij

          Result: The 0 bytes at offset 20 are replaced by the 10 bytes of data,
          i.e., the record is extended by 10 bytes.

          ABCDEFGHIJ0123456789 -> ABCDEFGHIJ0123456789abcdefghij

  3.      size = 10
          doff = 10
          dlen = 5
          data = abcdefghij

          Result: The 5 bytes at offset 10 are replaced by the 10 bytes of data.

          ABCDEFGHIJ0123456789 -> ABCDEFGHIJabcdefghij56789

  4.      size = 10
          doff = 10
          dlen = 0
          data = abcdefghij

          Result: The 0 bytes at offset 10 are replaced by the 10 bytes of data,
          i.e., 10 bytes are inserted into the record.

          ABCDEFGHIJ0123456789 -> ABCDEFGHIJabcdefghij0123456789

  5.      size = 10
          doff = 2
          dlen = 15
          data = abcdefghij

          Result: The 15 bytes at offset 2 are replaced by the 10 bytes of data.

          ABCDEFGHIJ0123456789 -> ABabcdefghij789

  6.      size = 10
          doff = 0
          dlen = 0
          data = abcdefghij

          Result: The 0 bytes at offset 0 are replaced by the 10 bytes of data,
          i.e., the 10 bytes are inserted at the beginning of the record.

          ABCDEFGHIJ0123456789 -> abcdefghijABCDEFGHIJ0123456789

  7.      size = 0
          doff = 0
          dlen = 10
          data = ""

          Result: The 10 bytes at offset 0 are replaced by the 0 bytes of data,
          i.e., the first 10 bytes of the record are discarded.

          ABCDEFGHIJ0123456789 -> 0123456789

  8.      size = 10
          doff = 25
          dlen = 0
          data = abcdefghij

          Result: The 0 bytes at offset 25 are replaced by the 10 bytes of data,
          i.e., 10 bytes are inserted into the record past the end of the current
          data.

          ABCDEFGHIJ0123456789 -> ABCDEFGHIJ0123456789\0\0\0\0\0abcdefghij

                                                        [Prev] [Ref][Next]

Copyright Sleepycat Software

dir/file plus more (was: Re: Are svn_fs_dir_t and svn_fs_file_t worth it?)

[Greg Stein <gs...@lyra.org>]

On Wed, Nov 08, 2000 at 07:01:20PM -0500, Jim Blandy wrote:
>...
> So I think I'd like to eliminate svn_fs_dir_t and svn_fs_file_t, and
> use svn_fs_node_t throughout.  It would make svn_fs_node_t more like
> Unix file descriptors --- a uniform way to reference whatever's out
> there.
> 
> What do folks think?  Greg, I'm especially interested in your opinion,
> since you've actually written code that uses the fs interface.

This would be fine with me. There are points where I need a "file" to fetch
file-like information or content. Since these can already return an error,
there aren't any additional checks for me. But even better: I already know
whether it is a file or dir before I ever try to do that; therefore, I
shouldn't be messing up what I try to do with a node.

Of course, the svn_fs_node_is_dir() must stick around.


On a separate note, there are some gaps in the API that would be nice to
have and/or explained "how to do [because I'm missing it]":

*) fetch the svn_fs_id_t for a given node

*) when I call svn_fs_file_contents(), I get back a stream baton. is there
   some way to mark that as "no longer needed" ? Note that reading to EOF is
   not the answer, as I might read a portion of the stream. [is it relevant
   to mark it as unneeded?]

*) svn_fs__file_from_skel() appears to copy the file contents into memory.
   This isn't going to scale to multi-megabyte (or gigabyte!) files. It
   would be good to have a way that directly maps from DB3 to a seek/read
   function. The ideal interface for me will allow me to seek to a point in
   the "stream" and then read "n" bytes from it. Preferably, the read would
   not allocate memory (say, if DB3 mmap'd the record, then I'd just get a
   pointer into that mamp).

   Basically, I'm looking at a case where we have a gigabyte file stored
   into Subversion. The client requests bytes 100-120 and bytes
   2000000-2000100. For optimum Apache behavior, I could get just those
   bytes without any memory allocation. (beyond what mmap does, or possibly
   reading from a file descriptor into some allocated memory (of just the
   right size))

   What kinds of mechanisms does DB3 support for reading large content? Can
   you give me a pointer to the doc/API? (so that I can be a bit more
   intelligent in my request here)


thx,
-g

-- 
Greg Stein, http://www.lyra.org/

Re: Are svn_fs_dir_t and svn_fs_file_t worth it?

[Karl Fogel <kf...@galois.collab.net>]

Jim Blandy <ji...@savonarola.red-bean.com> writes:
> [...]
>
> So I think I'd like to eliminate svn_fs_dir_t and svn_fs_file_t, and
> use svn_fs_node_t throughout.  It would make svn_fs_node_t more like
> Unix file descriptors --- a uniform way to reference whatever's out
> there.
> 
> What do folks think?  Greg, I'm especially interested in your opinion,
> since you've actually written code that uses the fs interface.

I haven't had to use the code, so my opinion is more or less
groundless here :-), but what you propose sounds like a good idea.

There's no single answer in these situations, I guess.  When the
commonalities begin to outweigh the differences, at some point it
makes sense to always use the same data structure and just have a type
bit.  It sounds like that point has been reached with fs nodes.