new svnfsfs tool (was: Re: svn commit: r1594860 [1/3] - in /subversion/trunk: ./ subversion/include/ tools/server-side/ tools/server-side/svnfsfs/)

[Stefan Sperling <st...@elego.de>]

On Thu, May 15, 2014 at 10:48:37AM -0000, stefan2@apache.org wrote:
> Author: stefan2
> Date: Thu May 15 10:48:37 2014
> New Revision: 1594860
> 
> URL: http://svn.apache.org/r1594860
> Log:
> Introduce FSFS expert tool 'svnfsfs'.  It is intended to grow various
> FSFS-specific commands in the future - like recreating repcache.db etc.
> 
> For now, it provides two commands to read (dump) and write (load)
> format 7 index information.  With these, corrupted repositories can
> be manipulated / fixed by hand or script.
> 
> The former fsfs-stats tool becomes a sub-command as well.

Wouldn't it make sense to integrate such functionality in svnadmin?

Re: new svnfsfs tool

[Stefan Fuhrmann <st...@wandisco.com>]

On Sat, May 17, 2014 at 10:10 AM, Branko Čibej <br...@wandisco.com> wrote:

>  On 16.05.2014 19:56, Stefan Sperling wrote:
>
> On Thu, May 15, 2014 at 10:48:37AM -0000, stefan2@apache.org wrote:
>
>  Author: stefan2
> Date: Thu May 15 10:48:37 2014
> New Revision: 1594860
>
> URL: http://svn.apache.org/r1594860
> Log:
> Introduce FSFS expert tool 'svnfsfs'.  It is intended to grow various
> FSFS-specific commands in the future - like recreating repcache.db etc.
>
> For now, it provides two commands to read (dump) and write (load)
> format 7 index information.  With these, corrupted repositories can
> be manipulated / fixed by hand or script.
>
> The former fsfs-stats tool becomes a sub-command as well.
>
>  Wouldn't it make sense to integrate such functionality in svnadmin?
>
>
> If the tool is strictly for fiddling with internals of FSFS repositories
> that cannot meaningfully apply to other backends, then no, probably not ...
> we really don't want to have backend-specific commands in svnadmin.
>

I agree.

But I also see Stefan's implicit point of having fewer tools
and doing things consistently over e.g. all backends. My
rationale is as follows: Is there a reasonable way to define
FS vtable functions that will provide the necessary data in
a mostly generic way?

For the "recreate rep-cache.db" mentioned in the log message,
it would certainly be possible to add an '--rebuild-caches' option.
This is fairly generic and should therefore go into svnadmin,
unless we needed some more specific option or UI interaction.

OTOH, it would be very hard to define a generic read/write
interface for backend-specific index data. So, dump-index
and load-index are better of in a separate tool that directly
calls into the FSFS lib.

The 'stats' sub-command is a borderline case. Lots of the
grouping etc. may work for FSX as well. However, the container-
based structure of FSX may require a different data structure
(1 additional level of indirection?) and it is too early to design
what the interface should look like. So, for now it's fine to have
it in the extra tool (at least dropping fsfs-stats in the process)
but later the code may be moved behind the FS API.

As for svn-bench, yes it would be conceivable to add a '--null'
option to the standard client UI. But there are a few issues
with that approach. First, we may not support all other options
of the respective command when in '--null' mode. That's not
terribly bad but there would be more to test or we may get
the occasional bug report.

Secondly, svn-bench is intended to be a minimalistic protocol
driver. It takes various shortcuts and even duplicates and
modifies code from the client lib (and possibly others) to
bypass client-side bottlenecks.

So again, having one tool instead of two would be nice but
the overall coding, testing and maintenance effort may then
bit higher than with the current setup.

-- Stefan^2.

Re: new svnfsfs tool

[Branko Čibej <br...@wandisco.com>]

On 17.05.2014 13:12, Stefan Sperling wrote:
> On Sat, May 17, 2014 at 10:10:32AM +0200, Branko Čibej wrote:
>> ... we really don't want to have backend-specific commands in svnadmin.
> Why not? Anyway, you're a bit late with this demand.
> There are svnadmin list-dblogs and list-unused-dblogs (BDB-only),
> there is hotcopy --incremental which only works on FSFS, some
> subcommands take a --memory-cache-size option which is FSFS-specific.
> I didn't look for more examples.

Yes, I know about that. So much for consistency. :)

> I think it's better to keep admin-facing user interfaces focused
> on the existing admin tools. In my experience, even experienced
> admins often don't know off-hand whether to use svnadmin or svnlook.
>
> Why should people suddenly care about the backend a repository is
> using if they just want to see some repository filesystem stats?
>
> Also, stuff in tools/ is often not packaged by packagers, and ends
> up missing on most installations even though it would be useful.
>
> svn-bench is another such example. We should just move this into
> 'svn' as far as I'm concerned.

Eh? Benchmarking in the client? Now that would really be confusing.

-- Brane


-- 
Branko Čibej | Director of Subversion
WANdisco // Non-Stop Data
e. brane@wandisco.com

Re: new svnfsfs tool

[Stefan Sperling <st...@elego.de>]

On Sat, May 17, 2014 at 10:10:32AM +0200, Branko Čibej wrote:
> ... we really don't want to have backend-specific commands in svnadmin.

Why not? Anyway, you're a bit late with this demand.
There are svnadmin list-dblogs and list-unused-dblogs (BDB-only),
there is hotcopy --incremental which only works on FSFS, some
subcommands take a --memory-cache-size option which is FSFS-specific.
I didn't look for more examples.

I think it's better to keep admin-facing user interfaces focused
on the existing admin tools. In my experience, even experienced
admins often don't know off-hand whether to use svnadmin or svnlook.

Why should people suddenly care about the backend a repository is
using if they just want to see some repository filesystem stats?

Also, stuff in tools/ is often not packaged by packagers, and ends
up missing on most installations even though it would be useful.

svn-bench is another such example. We should just move this into
'svn' as far as I'm concerned.

Re: new svnfsfs tool

[Branko Čibej <br...@wandisco.com>]

On 16.05.2014 19:56, Stefan Sperling wrote:
> On Thu, May 15, 2014 at 10:48:37AM -0000, stefan2@apache.org wrote:
>> Author: stefan2
>> Date: Thu May 15 10:48:37 2014
>> New Revision: 1594860
>>
>> URL: http://svn.apache.org/r1594860
>> Log:
>> Introduce FSFS expert tool 'svnfsfs'.  It is intended to grow various
>> FSFS-specific commands in the future - like recreating repcache.db etc.
>>
>> For now, it provides two commands to read (dump) and write (load)
>> format 7 index information.  With these, corrupted repositories can
>> be manipulated / fixed by hand or script.
>>
>> The former fsfs-stats tool becomes a sub-command as well.
> Wouldn't it make sense to integrate such functionality in svnadmin?

If the tool is strictly for fiddling with internals of FSFS repositories
that cannot meaningfully apply to other backends, then no, probably not
... we really don't want to have backend-specific commands in svnadmin.

-- Brane


-- 
Branko Čibej | Director of Subversion
WANdisco // Non-Stop Data
e. brane@wandisco.com