Re: svn commit: r1172652 - in /subversion/trunk/subversion: include/private/svn_subr_private.h libsvn_subr/spillbuf.c

["C. Michael Pilato" <cm...@collab.net>]

On 09/19/2011 10:58 AM, gstein@apache.org wrote:
> Author: gstein
> Date: Mon Sep 19 14:58:47 2011
> New Revision: 1172652
> 
> URL: http://svn.apache.org/viewvc?rev=1172652&view=rev
> Log:
> Add a new "spill buffer" type and functions, extracted from ra_serf. This
> type will buffer content in memory until it reaches a specified size,
> where it will then spill additional content onto disk.

Awesome.  Wanna drop an exemplary use thereof into
subversion/libsvn_repos/reporter.c (for use with the
report_baton_t->tempfile member)?  I've been wanting to make that change for
some time now, and this would prove a good demonstration of your new logic.

> Added: subversion/trunk/subversion/include/private/svn_subr_private.h
> URL: http://svn.apache.org/viewvc/subversion/trunk/subversion/include/private/svn_subr_private.h?rev=1172652&view=auto
> ==============================================================================
> --- subversion/trunk/subversion/include/private/svn_subr_private.h (added)
> +++ subversion/trunk/subversion/include/private/svn_subr_private.h Mon Sep 19 14:58:47 2011
> @@ -0,0 +1,86 @@
> +/*
> + * svn_subr_privae.h : private definitions from libsvn_subr

                   ^^  Oops!

I didn't complete a full review, just a skim.  Looks good!

-- 
C. Michael Pilato <cm...@collab.net>
CollabNet   <>   www.collab.net   <>   Distributed Development On Demand

Re: svn commit: r1172652 - in /subversion/trunk/subversion: include/private/svn_subr_private.h libsvn_subr/spillbuf.c

[Greg Stein <gs...@gmail.com>]

On Mon, Sep 19, 2011 at 11:05, C. Michael Pilato <cm...@collab.net> wrote:
> On 09/19/2011 10:58 AM, gstein@apache.org wrote:
>> Author: gstein
>> Date: Mon Sep 19 14:58:47 2011
>> New Revision: 1172652
>>
>> URL: http://svn.apache.org/viewvc?rev=1172652&view=rev
>> Log:
>> Add a new "spill buffer" type and functions, extracted from ra_serf. This
>> type will buffer content in memory until it reaches a specified size,
>> where it will then spill additional content onto disk.
>
> Awesome.  Wanna drop an exemplary use thereof into
> subversion/libsvn_repos/reporter.c (for use with the
> report_baton_t->tempfile member)?  I've been wanting to make that change for
> some time now, and this would prove a good demonstration of your new logic.

I'm going to write a unit test for the code. That should provide a
good demonstration.

The code that I first checked in uses a callback to do the reading
(appropriate for the xml parsing in ra_serf), but I don't know if that
works for ra_neon. ie. the push vs pull problem. But that said, I've
just adjusted the code to *also* provide a pull option (rather than
push into a callback). Gonna commit that, and then work on the unit
test.

>...

Cheers,
-g

Re: svn commit: r1172652 - in /subversion/trunk/subversion: include/private/svn_subr_private.h libsvn_subr/spillbuf.c

[Greg Stein <gs...@gmail.com>]

On Mon, Sep 19, 2011 at 11:05, C. Michael Pilato <cm...@collab.net> wrote:
>...
> Awesome.  Wanna drop an exemplary use thereof into
> subversion/libsvn_repos/reporter.c (for use with the
> report_baton_t->tempfile member)?  I've been wanting to make that change for
> some time now, and this would prove a good demonstration of your new logic.

I took a quick look at this, and some extra logic is going to be
needed. The reporter wants to read N bytes from the "buffer", but the
spillbuf says "you have N bytes $here [to consume]". The spillbuf also
works best when you write the same number of bytes as the declared
"blocksize". But if you declare a blocksize of 1000 bytes, and write
just 3, then you waste 997 bytes. And you're not allowed to write >
1000 (ie. split it up).

[ I need to put that info into the header ]

It seems the answer is a secondary layer that is more
byte/stream-oriented that will buffer writes to the blocksize (and do
splits), and then manages the read buffer to enable the byte/stream
style of reading.

I'd say that the key point is that the reporter is kind of
"record-oriented" while the spillbuf (as written) is a plain stream of
bytes that were collected a block at a time.

Cheers,
-g