You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by Erik Huelsmann <eh...@gmail.com> on 2010/09/02 21:34:50 UTC
[PROPOSAL] WC-NG: merge NODE_DATA, WORKING_NODE and BASE_NODE into a
single table (NODES)
As described by Julian earlier this month, Julian, Philip and I observed
that the BASE_NODE, WORKING_NODE and NODE_DATA tables have many fields in
common. Notably, by introducing the NODE_DATA table, most fields from
BASE_NODE and WORKING_NODE already moved to a common table.
The remaining fields (after switching to NODE_DATA *and* SINGLE-DB) on the
side of WORKING_NODE are the 2 cache fields 'translated_size' and
'last_mod_time'. Apart from those two, there are the indexing fields wc_id,
local_relpath and parent_relpath.
In the end we're storing *lots* of bytes (wc_id, local_relpath and
parent_relpath) to store 2 64-bit values.
On the side of BASE_NODE, we end up storing dav_cache, repos_id, repos_path
and revision. The NODE_DATA table already has the fields original_repos_id,
original_repos_path and original_revision. When op_depth == 0, these are
guaranteed to be empty (null), since they are for working nodes with
copy/move source information. Renaming the three fields in NODE_DATA to
repos_id, repos_path and revision, generalizing their use to include
op_depth == 0 [ofcourse nicely documented in the table docs], BASE_NODE
would be reduced to a store of the dav_cache, translated_size and
last_mod_time fields.
By subsuming translated_size and last_mod_time into NODE_DATA, neither
WORKING_NODE nor BASE_NODE will need to store these values anymore. This
eliminates the entire reason of existence of WORKING_NODE. BASE_NODE then
only stores dav_cache. Here too, it's probably more efficient (in size) to
store dav_cache in NODE_DATA to prevent repeated storage of wc_id,
local_relpath and parent_relpath in BASE_NODE.
In addition to the eliminated storage overhead, we'd be making things a
little less complex for ourselves: UPDATE, INSERT and DELETE queries would
be operating only on a single table, removing the need to split updates
across multiple statements.
This week, I was discussing this change with Greg on IRC. We both have the
feeling this should work out well. The proposal here is to switch
(WORKING_NODE, NODE_DATA, BASE_NODE) into a single table --> NODES.
Comments? Fears? Enhancements?
Bye,
Erik.
Re: [PROPOSAL] WC-NG: merge NODE_DATA, WORKING_NODE and BASE_NODE
into a single table (NODES)
Posted by Erik Huelsmann <eh...@gmail.com>.
Given all the responses in the thread, I'd say we're moving to the single
table for BASE and WORKING node recording. There was a flurry of activity
from me yesterday and this morning regarding NODE_DATA: that was just me
flushing my queue of patches.
The work isn't completely irrelevant, as it identifies the spots where the
NODES table will be introduced, just like NODE_DATA had to.
Today, I'll go to draw up the NODES table and move over the queries which
had already been modified for NODE_DATA over to the NODES design. I hope to
get a very long way today already. If it's not done today, then I expect it
to be able to finish it this week.
Anyone wanting to join in: let's chat on IRC.
Bye,
Erik.
On Thu, Sep 2, 2010 at 11:34 PM, Erik Huelsmann <eh...@gmail.com> wrote:
>
>
> As described by Julian earlier this month, Julian, Philip and I observed
> that the BASE_NODE, WORKING_NODE and NODE_DATA tables have many fields in
> common. Notably, by introducing the NODE_DATA table, most fields from
> BASE_NODE and WORKING_NODE already moved to a common table.
>
> The remaining fields (after switching to NODE_DATA *and* SINGLE-DB) on the
> side of WORKING_NODE are the 2 cache fields 'translated_size' and
> 'last_mod_time'. Apart from those two, there are the indexing fields wc_id,
> local_relpath and parent_relpath.
>
> In the end we're storing *lots* of bytes (wc_id, local_relpath and
> parent_relpath) to store 2 64-bit values.
>
> On the side of BASE_NODE, we end up storing dav_cache, repos_id, repos_path
> and revision. The NODE_DATA table already has the fields original_repos_id,
> original_repos_path and original_revision. When op_depth == 0, these are
> guaranteed to be empty (null), since they are for working nodes with
> copy/move source information. Renaming the three fields in NODE_DATA to
> repos_id, repos_path and revision, generalizing their use to include
> op_depth == 0 [ofcourse nicely documented in the table docs], BASE_NODE
> would be reduced to a store of the dav_cache, translated_size and
> last_mod_time fields.
>
> By subsuming translated_size and last_mod_time into NODE_DATA, neither
> WORKING_NODE nor BASE_NODE will need to store these values anymore. This
> eliminates the entire reason of existence of WORKING_NODE. BASE_NODE then
> only stores dav_cache. Here too, it's probably more efficient (in size) to
> store dav_cache in NODE_DATA to prevent repeated storage of wc_id,
> local_relpath and parent_relpath in BASE_NODE.
>
> In addition to the eliminated storage overhead, we'd be making things a
> little less complex for ourselves: UPDATE, INSERT and DELETE queries would
> be operating only on a single table, removing the need to split updates
> across multiple statements.
>
>
> This week, I was discussing this change with Greg on IRC. We both have the
> feeling this should work out well. The proposal here is to switch
> (WORKING_NODE, NODE_DATA, BASE_NODE) into a single table --> NODES.
>
>
> Comments? Fears? Enhancements?
>
>
> Bye,
>
>
> Erik.
>
Re: [PROPOSAL] WC-NG: merge NODE_DATA, WORKING_NODE and BASE_NODE
into a single table (NODES)
Posted by "Hyrum K. Wright" <hy...@mail.utexas.edu>.
On Sat, Sep 4, 2010 at 4:54 AM, Bert Huijben <be...@qqmail.nl> wrote:
>
>
>> -----Original Message-----
>> From: Greg Stein [mailto:gstein@gmail.com]
>> Sent: zaterdag 4 september 2010 6:28
>> To: Bert Huijben
>> Cc: Julian Foad; Erik Huelsmann; dev@subversion.apache.org
>> Subject: Re: [PROPOSAL] WC-NG: merge NODE_DATA, WORKING_NODE and
>> BASE_NODE into a single table (NODES)
>
>> How confusing are you trying to make this? You're succeeding
>> wonderfully.
>
> Well, you are good at helping it more confusing for everybody.
>
...
>
> But maybe we should look at what Eric finds here, as he is the only one
> really working on this right now. (Or it must be that I missed your commits)
All,
This is a discussion about a tough problem. Ad hominem arguments
muddle the waters even more, and discourage contributions from
internal and external participants. Let's focus on the technical
problem, and not the failings of the participants.
Thanks,
-Hyrum
RE: [PROPOSAL] WC-NG: merge NODE_DATA, WORKING_NODE and BASE_NODE into a single table (NODES)
Posted by Bert Huijben <be...@qqmail.nl>.
> -----Original Message-----
> From: Greg Stein [mailto:gstein@gmail.com]
> Sent: zaterdag 4 september 2010 6:28
> To: Bert Huijben
> Cc: Julian Foad; Erik Huelsmann; dev@subversion.apache.org
> Subject: Re: [PROPOSAL] WC-NG: merge NODE_DATA, WORKING_NODE and
> BASE_NODE into a single table (NODES)
> How confusing are you trying to make this? You're succeeding
> wonderfully.
Well, you are good at helping it more confusing for everybody.
> We are talking about *adds* and their op_depth value. What the fuck
> does that have to do with commits?
>
> All adds are very simple commit operations. They can all have the same
> op_depth value. There are no restrictions on adding more or removing
> them. They are free floating nodes within the various restructuring
> that copies/moves may have done. You can independently revert any
> single add.
>
> Get back to the original question: adds do not require variant
> op_depth values. Very simple. Any add under an added parent just uses
> that parent's op_depth.
But not making it use a specific op_depth makes adding the add harder (has
to check the parent op), the revert harder (can't just take the op_depth to
see that it can be reverted by itself) and the commit (see previous mail).
Most of this work will probably be handled by _scan_addition() and specific
DB operations like svn_wc__db_(temp_)op_copy(),
svn_wc__db_temp_op_make_copy(), etc.
I really think all of these would be easier to implement with an explicit
(non-inherited) op_depth for each add as that makes it possible to identify
the impact of these operations without scanning up all the time. (Just
looking at the op_depth would show that scanning up is not needed at all;
using the same op_depth implies that you only have to scan one specific
parent)
But maybe we should look at what Eric finds here, as he is the only one
really working on this right now. (Or it must be that I missed your commits)
The problem can be solved both ways, but for one of these two use cases this
requires more scanning upwards. (Which isn't very cheap for generic use
unless we some transaction/lock around the group of statements)
Bert
Re: [PROPOSAL] WC-NG: merge NODE_DATA, WORKING_NODE and BASE_NODE
into a single table (NODES)
Posted by Greg Stein <gs...@gmail.com>.
On Fri, Sep 3, 2010 at 11:46, Bert Huijben <be...@qqmail.nl> wrote:
>> -----Original Message-----
>> From: Greg Stein [mailto:gstein@gmail.com]
>> Sent: vrijdag 3 september 2010 17:23
>> To: Bert Huijben
>> Cc: Julian Foad; Erik Huelsmann; dev@subversion.apache.org
>> Subject: Re: [PROPOSAL] WC-NG: merge NODE_DATA, WORKING_NODE and
>> BASE_NODE into a single table (NODES)
>>
>> On Fri, Sep 3, 2010 at 04:46, Bert Huijben <be...@qqmail.nl> wrote:
>> >> -----Original Message-----
>> >> From: Greg Stein [mailto:gstein@gmail.com]
>> >> Sent: vrijdag 3 september 2010 1:14
>> >> To: Julian Foad
>> >> Cc: Erik Huelsmann; dev@subversion.apache.org
>> >> Subject: Re: [PROPOSAL] WC-NG: merge NODE_DATA, WORKING_NODE and
>> >> BASE_NODE into a single table (NODES)
>> >>
>> >> On Thu, Sep 2, 2010 at 18:39, Julian Foad <ju...@wandisco.com>
>> >...
>> >> > We need to describe how the layering works for copies, deletes,
>> and
>> >> > adds. In particular I'm recalling something about how local adds
>> >> aren't
>> >> > recursive, unlike copies, so an additional change within an added
>> dir
>> >> > doesn't work the same way with regard to op_depth as it would
>> inside
>> >> a
>> >> > copied dir.
>> >>
>> >> Adds within other adds don't layer. You're simply adding more nodes
>> at
>> >> the parent's op_depth.
>> >>
>> >> Deleting an ancestor of a deleted node also does not create an
>> >> additional layer. Instead, we establish the new op_depth and relabel
>> >> all of the deleted children with that new op_depth so that it looks
>> >> like one big happy delete operation.
>> >>
>> >> Copies/moves always introduce a new layer.
>> >
>> > I'm not sure if I follow your reasoning here. (layering vs op_depths)
>> >
>> > I think local additions should always be their own op (and have their
>> own
>> > op_depth). That would simplify the revert and commit process: They
>> would
>> > only have to do restructuring changes(add/delete) when there is a new
>> > op_depth.
>>
>> A local add is not a restructuring change. It is just adding new nodes
>> into the tree. You can revert any of those nodes -- you are not
>> required to revert the root of an add.
>>
>> You *do* have to go to the root of a copy/move to revert it, however.
>>
>> There is no way to revert the deletion of a child, so (again) you must
>> revert a deletion at its op-root.
>>
>> > So the commit harvester would be as simple as walking the operations
>> (for
>> > restructuring operations) and checking present nodes for text and
>> prop mods.
>>
>> And this leads me to think that you're confusing "add" with
>> "copy/move". Adds never have text/prop mods. The text and props are
>> ALWAYS new.
>
> The commit harvester should be something like:
>
> Phase 1: Walk the tree (limited by depth) to find restructuring operations
> that must be committed. (Handle delete + copy + add (+ in 1.8 move)). Some
> of these operations will apply deeper then the limited depth (See issue
> #3699 and others), but only their operations are handled in these step: not
> local content changes.
>
> (This phase should see each local added directory/file as a separate
> operation, or a svn ci --depth empty DIR would commit all added children.
> Copies are one operation for the entire tree)
>
> Phase 2: Walk the tree (limited by depth) to find which nodes that have a
> pristine version with text or props that differs from the current version.
> (adds are ignored here; already handled in step 1).
>
> This would leave all operations outside the selected depth in the working
> copy for committing later; a much saner operation then the current behavior
> where an add or copy turns a depth limited operation in a recursive commit,
> including all the local modifications deeper in the tree.
How confusing are you trying to make this? You're succeeding wonderfully.
We are talking about *adds* and their op_depth value. What the fuck
does that have to do with commits?
All adds are very simple commit operations. They can all have the same
op_depth value. There are no restrictions on adding more or removing
them. They are free floating nodes within the various restructuring
that copies/moves may have done. You can independently revert any
single add.
Get back to the original question: adds do not require variant
op_depth values. Very simple. Any add under an added parent just uses
that parent's op_depth.
Cheers,
-g
RE: [PROPOSAL] WC-NG: merge NODE_DATA, WORKING_NODE and BASE_NODE into a single table (NODES)
Posted by Bert Huijben <be...@qqmail.nl>.
> -----Original Message-----
> From: Greg Stein [mailto:gstein@gmail.com]
> Sent: vrijdag 3 september 2010 17:23
> To: Bert Huijben
> Cc: Julian Foad; Erik Huelsmann; dev@subversion.apache.org
> Subject: Re: [PROPOSAL] WC-NG: merge NODE_DATA, WORKING_NODE and
> BASE_NODE into a single table (NODES)
>
> On Fri, Sep 3, 2010 at 04:46, Bert Huijben <be...@qqmail.nl> wrote:
> >> -----Original Message-----
> >> From: Greg Stein [mailto:gstein@gmail.com]
> >> Sent: vrijdag 3 september 2010 1:14
> >> To: Julian Foad
> >> Cc: Erik Huelsmann; dev@subversion.apache.org
> >> Subject: Re: [PROPOSAL] WC-NG: merge NODE_DATA, WORKING_NODE and
> >> BASE_NODE into a single table (NODES)
> >>
> >> On Thu, Sep 2, 2010 at 18:39, Julian Foad <ju...@wandisco.com>
> >...
> >> > We need to describe how the layering works for copies, deletes,
> and
> >> > adds. In particular I'm recalling something about how local adds
> >> aren't
> >> > recursive, unlike copies, so an additional change within an added
> dir
> >> > doesn't work the same way with regard to op_depth as it would
> inside
> >> a
> >> > copied dir.
> >>
> >> Adds within other adds don't layer. You're simply adding more nodes
> at
> >> the parent's op_depth.
> >>
> >> Deleting an ancestor of a deleted node also does not create an
> >> additional layer. Instead, we establish the new op_depth and relabel
> >> all of the deleted children with that new op_depth so that it looks
> >> like one big happy delete operation.
> >>
> >> Copies/moves always introduce a new layer.
> >
> > I'm not sure if I follow your reasoning here. (layering vs op_depths)
> >
> > I think local additions should always be their own op (and have their
> own
> > op_depth). That would simplify the revert and commit process: They
> would
> > only have to do restructuring changes(add/delete) when there is a new
> > op_depth.
>
> A local add is not a restructuring change. It is just adding new nodes
> into the tree. You can revert any of those nodes -- you are not
> required to revert the root of an add.
>
> You *do* have to go to the root of a copy/move to revert it, however.
>
> There is no way to revert the deletion of a child, so (again) you must
> revert a deletion at its op-root.
>
> > So the commit harvester would be as simple as walking the operations
> (for
> > restructuring operations) and checking present nodes for text and
> prop mods.
>
> And this leads me to think that you're confusing "add" with
> "copy/move". Adds never have text/prop mods. The text and props are
> ALWAYS new.
The commit harvester should be something like:
Phase 1: Walk the tree (limited by depth) to find restructuring operations
that must be committed. (Handle delete + copy + add (+ in 1.8 move)). Some
of these operations will apply deeper then the limited depth (See issue
#3699 and others), but only their operations are handled in these step: not
local content changes.
(This phase should see each local added directory/file as a separate
operation, or a svn ci --depth empty DIR would commit all added children.
Copies are one operation for the entire tree)
Phase 2: Walk the tree (limited by depth) to find which nodes that have a
pristine version with text or props that differs from the current version.
(adds are ignored here; already handled in step 1).
This would leave all operations outside the selected depth in the working
copy for committing later; a much saner operation then the current behavior
where an add or copy turns a depth limited operation in a recursive commit,
including all the local modifications deeper in the tree.
Bert
Re: [PROPOSAL] WC-NG: merge NODE_DATA, WORKING_NODE and BASE_NODE
into a single table (NODES)
Posted by Greg Stein <gs...@gmail.com>.
On Fri, Sep 3, 2010 at 04:46, Bert Huijben <be...@qqmail.nl> wrote:
>> -----Original Message-----
>> From: Greg Stein [mailto:gstein@gmail.com]
>> Sent: vrijdag 3 september 2010 1:14
>> To: Julian Foad
>> Cc: Erik Huelsmann; dev@subversion.apache.org
>> Subject: Re: [PROPOSAL] WC-NG: merge NODE_DATA, WORKING_NODE and
>> BASE_NODE into a single table (NODES)
>>
>> On Thu, Sep 2, 2010 at 18:39, Julian Foad <ju...@wandisco.com>
>...
>> > We need to describe how the layering works for copies, deletes, and
>> > adds. In particular I'm recalling something about how local adds
>> aren't
>> > recursive, unlike copies, so an additional change within an added dir
>> > doesn't work the same way with regard to op_depth as it would inside
>> a
>> > copied dir.
>>
>> Adds within other adds don't layer. You're simply adding more nodes at
>> the parent's op_depth.
>>
>> Deleting an ancestor of a deleted node also does not create an
>> additional layer. Instead, we establish the new op_depth and relabel
>> all of the deleted children with that new op_depth so that it looks
>> like one big happy delete operation.
>>
>> Copies/moves always introduce a new layer.
>
> I'm not sure if I follow your reasoning here. (layering vs op_depths)
>
> I think local additions should always be their own op (and have their own
> op_depth). That would simplify the revert and commit process: They would
> only have to do restructuring changes(add/delete) when there is a new
> op_depth.
A local add is not a restructuring change. It is just adding new nodes
into the tree. You can revert any of those nodes -- you are not
required to revert the root of an add.
You *do* have to go to the root of a copy/move to revert it, however.
There is no way to revert the deletion of a child, so (again) you must
revert a deletion at its op-root.
> So the commit harvester would be as simple as walking the operations (for
> restructuring operations) and checking present nodes for text and prop mods.
And this leads me to think that you're confusing "add" with
"copy/move". Adds never have text/prop mods. The text and props are
ALWAYS new.
IOW, fix your terminology and then I can tell whether we agree or not.
>...
Cheers,
-g
RE: [PROPOSAL] WC-NG: merge NODE_DATA, WORKING_NODE and BASE_NODE into a single table (NODES)
Posted by Bert Huijben <be...@qqmail.nl>.
> -----Original Message-----
> From: Greg Stein [mailto:gstein@gmail.com]
> Sent: vrijdag 3 september 2010 1:14
> To: Julian Foad
> Cc: Erik Huelsmann; dev@subversion.apache.org
> Subject: Re: [PROPOSAL] WC-NG: merge NODE_DATA, WORKING_NODE and
> BASE_NODE into a single table (NODES)
>
> On Thu, Sep 2, 2010 at 18:39, Julian Foad <ju...@wandisco.com>
> wrote:
> >...
> >> Comments? Fears? Enhancements?
> >
> > +1.
> >
> > It would be useful if you could post the latest version of the
> > description of the new format. Here's a bit of introductory text I
> > wrote, starting with a paragraph of yours from wc-metadata.sql:
> >
> > /* The NODE_DATA table describes the way working nodes are layered on
> top of
> > base nodes and on top of other working nodes, due to nested tree
> structure
> > changes. The layers are modelled using the "op_depth" column.
> >
> > An "operation depth" refers to the number of directory levels down
> from
> > the WC root at which a tree-change operation (delete, add, copy,
> move,
> > replace) was performed. It does NOT refer to the number of path
>
> There is no "replace" operation. There are deletes and then an
> add/copy/move.
>
> >...
> > As for the three columns holding cached values, it seems a bit impure
> > but pragmatically OK to move them into NODES, and certainly it would
> be
> > unhelpful to keep the existing BASE_NODE and WORKING_NODE tables with
> > their present names and only those columns in them.
>
> translated_size and last_mod_time would nominally go into ACTUAL_NODE.
> Hyrum and I decided against that, however, because it meant we would
> have to always create ACTUAL_NODE rows. With the values in *_NODE, we
> can defer creating rows in ACTUAL_NODE until something comes up.
>
> > We need to describe how the layering works for copies, deletes, and
> > adds. In particular I'm recalling something about how local adds
> aren't
> > recursive, unlike copies, so an additional change within an added dir
> > doesn't work the same way with regard to op_depth as it would inside
> a
> > copied dir.
>
> Adds within other adds don't layer. You're simply adding more nodes at
> the parent's op_depth.
>
> Deleting an ancestor of a deleted node also does not create an
> additional layer. Instead, we establish the new op_depth and relabel
> all of the deleted children with that new op_depth so that it looks
> like one big happy delete operation.
>
> Copies/moves always introduce a new layer.
I'm not sure if I follow your reasoning here. (layering vs op_depths)
I think local additions should always be their own op (and have their own
op_depth). That would simplify the revert and commit process: They would
only have to do restructuring changes(add/delete) when there is a new
op_depth.
So the commit harvester would be as simple as walking the operations (for
restructuring operations) and checking present nodes for text and prop mods.
If multiple levels of adds would be a single operation, we would have to
perform more checks to see if it is an operation that must be committed (or
requires restructuring on revert), while we could just compare the op_depth
with the number of '/'-s in the local_relpath if every local add is its own
op_depth.
Bert
> Cheers,
> -g
Re: [PROPOSAL] WC-NG: merge NODE_DATA, WORKING_NODE and BASE_NODE
into a single table (NODES)
Posted by Greg Stein <gs...@gmail.com>.
On Thu, Sep 2, 2010 at 18:39, Julian Foad <ju...@wandisco.com> wrote:
>...
>> Comments? Fears? Enhancements?
>
> +1.
>
> It would be useful if you could post the latest version of the
> description of the new format. Here's a bit of introductory text I
> wrote, starting with a paragraph of yours from wc-metadata.sql:
>
> /* The NODE_DATA table describes the way working nodes are layered on top of
> base nodes and on top of other working nodes, due to nested tree structure
> changes. The layers are modelled using the "op_depth" column.
>
> An "operation depth" refers to the number of directory levels down from
> the WC root at which a tree-change operation (delete, add, copy, move,
> replace) was performed. It does NOT refer to the number of path
There is no "replace" operation. There are deletes and then an add/copy/move.
>...
> As for the three columns holding cached values, it seems a bit impure
> but pragmatically OK to move them into NODES, and certainly it would be
> unhelpful to keep the existing BASE_NODE and WORKING_NODE tables with
> their present names and only those columns in them.
translated_size and last_mod_time would nominally go into ACTUAL_NODE.
Hyrum and I decided against that, however, because it meant we would
have to always create ACTUAL_NODE rows. With the values in *_NODE, we
can defer creating rows in ACTUAL_NODE until something comes up.
> We need to describe how the layering works for copies, deletes, and
> adds. In particular I'm recalling something about how local adds aren't
> recursive, unlike copies, so an additional change within an added dir
> doesn't work the same way with regard to op_depth as it would inside a
> copied dir.
Adds within other adds don't layer. You're simply adding more nodes at
the parent's op_depth.
Deleting an ancestor of a deleted node also does not create an
additional layer. Instead, we establish the new op_depth and relabel
all of the deleted children with that new op_depth so that it looks
like one big happy delete operation.
Copies/moves always introduce a new layer.
Cheers,
-g
Re: [PROPOSAL] WC-NG: merge NODE_DATA, WORKING_NODE and BASE_NODE
into a single table (NODES)
Posted by Julian Foad <ju...@wandisco.com>.
Erik Huelsmann wrote:
> As described by Julian earlier this month, Julian, Philip and I observed
> that the BASE_NODE, WORKING_NODE and NODE_DATA tables have many fields in
> common. Notably, by introducing the NODE_DATA table, most fields from
> BASE_NODE and WORKING_NODE already moved to a common table.
>
> The remaining fields (after switching to NODE_DATA *and* SINGLE-DB) on the
> side of WORKING_NODE are the 2 cache fields 'translated_size' and
> 'last_mod_time'. Apart from those two, there are the indexing fields wc_id,
> local_relpath and parent_relpath.
>
> In the end we're storing *lots* of bytes (wc_id, local_relpath and
> parent_relpath) to store 2 64-bit values.
>
> On the side of BASE_NODE, we end up storing dav_cache, repos_id, repos_path
> and revision. The NODE_DATA table already has the fields original_repos_id,
> original_repos_path and original_revision. When op_depth == 0, these are
> guaranteed to be empty (null), since they are for working nodes with
> copy/move source information. Renaming the three fields in NODE_DATA to
> repos_id, repos_path and revision, generalizing their use to include
> op_depth == 0 [ofcourse nicely documented in the table docs], BASE_NODE
> would be reduced to a store of the dav_cache, translated_size and
> last_mod_time fields.
>
> By subsuming translated_size and last_mod_time into NODE_DATA, neither
> WORKING_NODE nor BASE_NODE will need to store these values anymore. This
> eliminates the entire reason of existence of WORKING_NODE. BASE_NODE then
> only stores dav_cache. Here too, it's probably more efficient (in size) to
> store dav_cache in NODE_DATA to prevent repeated storage of wc_id,
> local_relpath and parent_relpath in BASE_NODE.
>
> In addition to the eliminated storage overhead, we'd be making things a
> little less complex for ourselves: UPDATE, INSERT and DELETE queries would
> be operating only on a single table, removing the need to split updates
> across multiple statements.
>
>
> This week, I was discussing this change with Greg on IRC. We both have the
> feeling this should work out well. The proposal here is to switch
> (WORKING_NODE, NODE_DATA, BASE_NODE) into a single table --> NODES.
>
>
> Comments? Fears? Enhancements?
+1.
It would be useful if you could post the latest version of the
description of the new format. Here's a bit of introductory text I
wrote, starting with a paragraph of yours from wc-metadata.sql:
/* The NODE_DATA table describes the way working nodes are layered on top of
base nodes and on top of other working nodes, due to nested tree structure
changes. The layers are modelled using the "op_depth" column.
An "operation depth" refers to the number of directory levels down from
the WC root at which a tree-change operation (delete, add, copy, move,
replace) was performed. It does NOT refer to the number of path
components in a node's own 'local_relpath', but rather to the depth of
one of the tree changes that affects that node.
The tree checked out of the repository and modified by "update", "switch"
and "commit" post-processing, is represented by rows with op_depth=0.
That "layer" of the NODE_DATA table corresponds to what was called the
BASE_NODE table.
If the WC root path is "." and already contains "./A/", and a directory
tree "^/B" in the repository is copied to "./A/B", then rows are created
for "./A/B" and for all the children beneath it, all with op_depth=2.
Each path in the WC has one or more rows, each at a different "op_depth",
depending on how many nested tree changes affect it. Rows also exist for
paths that are not a currently visible part of the WC but were a part of
one of the tree changes.
*/
Unifying the node-rev reference columns (repos_id, repos_relpath,
revision) across all layers makes perfect sense to me. Instead of
thinking of "based on this node-rev" in BASE being different from
"copied from this node-rev" in WORKING, we will treat both of them as
"the node in this layer reflects (is a copy of) this repository
node-rev".
As for the three columns holding cached values, it seems a bit impure
but pragmatically OK to move them into NODES, and certainly it would be
unhelpful to keep the existing BASE_NODE and WORKING_NODE tables with
their present names and only those columns in them.
We need to describe how the layering works for copies, deletes, and
adds. In particular I'm recalling something about how local adds aren't
recursive, unlike copies, so an additional change within an added dir
doesn't work the same way with regard to op_depth as it would inside a
copied dir.
- Julian
Re: [PROPOSAL] WC-NG: merge NODE_DATA, WORKING_NODE and BASE_NODE
into a single table (NODES)
Posted by Neels J Hofmeyr <ne...@elego.de>.
On 2010-09-03 00:24, Stefan Sperling wrote:
> On Thu, Sep 02, 2010 at 11:34:50PM +0200, Erik Huelsmann wrote:
>> Comments? Fears? Enhancements?
>
> I always like simplifications.
> I cannot judge the overall impact of this change, but I guess you guys
> will have figured out the consequences.
>
> It feels like with a single table we're going back to a flat data store,
> yet this time it's a smart database rather than an entries file we have
> to parse.
>
> Having only a single table to deal with might make it a bit easier
> to fix wrecked databases, should that ever happen.
>
> Neels told me liked the idea, too.
yes, I +1 the single table. I kept thinking 'this needs to be simpler' but
had not sat down to think it out. You guys figured it out.
Depending on what is needed, we can then slightly modify one and the same
SQL query to get either BASE, WORKING, the layers in-between or
the-topmost-layer. I like.
~Neels
>
> And that's about all I can say about this :)
>
> Stefan
Re: [PROPOSAL] WC-NG: merge NODE_DATA, WORKING_NODE and BASE_NODE
into a single table (NODES)
Posted by Stefan Sperling <st...@elego.de>.
On Thu, Sep 02, 2010 at 11:34:50PM +0200, Erik Huelsmann wrote:
> Comments? Fears? Enhancements?
I always like simplifications.
I cannot judge the overall impact of this change, but I guess you guys
will have figured out the consequences.
It feels like with a single table we're going back to a flat data store,
yet this time it's a smart database rather than an entries file we have
to parse.
Having only a single table to deal with might make it a bit easier
to fix wrecked databases, should that ever happen.
Neels told me liked the idea, too.
And that's about all I can say about this :)
Stefan
Re: [PROPOSAL] WC-NG: merge NODE_DATA, WORKING_NODE and BASE_NODE into a single table (NODES)
Posted by Philip Martin <ph...@wandisco.com>.
"Hyrum K. Wright" <hy...@mail.utexas.edu> writes:
> Sure, but is this "bring us back to parity with 1.6" work, or is it
> "new stuff we can do with wc-ng" work? If the former, it's certainly
> a release blocker. If the latter, I'm not so sure....
We currently have a regression from 1.6, wc-ng cannot record the
revert-base when a child of a copied directory is replaced:
svn cp wc/A wc/X
svn rm wc/X/f
svn cp wc/g wc/X/f # 1.7 has only one row to store text-base and revert-base
--
Philip
Re: [PROPOSAL] WC-NG: merge NODE_DATA, WORKING_NODE and BASE_NODE
into a single table (NODES)
Posted by "Hyrum K. Wright" <hy...@mail.utexas.edu>.
On Fri, Sep 3, 2010 at 10:41 AM, Greg Stein <gs...@gmail.com> wrote:
> On Thu, Sep 2, 2010 at 22:51, Hyrum K. Wright
> <hy...@mail.utexas.edu> wrote:
>>...
>> My one concern (and perhaps this comes from not following the
>> discussion closely enough) is how this impacts 1.7. This feels eerily
>> like an eleventh-hour redesign, and our track record with these in the
>
> Nope. This has been "on deck" for months, but just hasn't happened in
> force until now. This schema change is required to support an add
> within a copy. (along with enabling other stuff like better reverts)
Sure, but is this "bring us back to parity with 1.6" work, or is it
"new stuff we can do with wc-ng" work? If the former, it's certainly
a release blocker. If the latter, I'm not so sure....
-Hyrum
Re: [PROPOSAL] WC-NG: merge NODE_DATA, WORKING_NODE and BASE_NODE
into a single table (NODES)
Posted by Greg Stein <gs...@gmail.com>.
On Thu, Sep 2, 2010 at 22:51, Hyrum K. Wright
<hy...@mail.utexas.edu> wrote:
>...
> My one concern (and perhaps this comes from not following the
> discussion closely enough) is how this impacts 1.7. This feels eerily
> like an eleventh-hour redesign, and our track record with these in the
Nope. This has been "on deck" for months, but just hasn't happened in
force until now. This schema change is required to support an add
within a copy. (along with enabling other stuff like better reverts)
>...
Cheers,
-g
Re: [PROPOSAL] WC-NG: merge NODE_DATA, WORKING_NODE and BASE_NODE
into a single table (NODES)
Posted by "Hyrum K. Wright" <hy...@mail.utexas.edu>.
On Thu, Sep 2, 2010 at 4:34 PM, Erik Huelsmann <eh...@gmail.com> wrote:
> As described by Julian earlier this month, Julian, Philip and I observed
> that the BASE_NODE, WORKING_NODE and NODE_DATA tables have many fields in
> common. Notably, by introducing the NODE_DATA table, most fields from
> BASE_NODE and WORKING_NODE already moved to a common table.
>
> The remaining fields (after switching to NODE_DATA *and* SINGLE-DB) on the
> side of WORKING_NODE are the 2 cache fields 'translated_size' and
> 'last_mod_time'. Apart from those two, there are the indexing fields wc_id,
> local_relpath and parent_relpath.
>
> In the end we're storing *lots* of bytes (wc_id, local_relpath and
> parent_relpath) to store 2 64-bit values.
>
> On the side of BASE_NODE, we end up storing dav_cache, repos_id, repos_path
> and revision. The NODE_DATA table already has the fields original_repos_id,
> original_repos_path and original_revision. When op_depth == 0, these are
> guaranteed to be empty (null), since they are for working nodes with
> copy/move source information. Renaming the three fields in NODE_DATA to
> repos_id, repos_path and revision, generalizing their use to include
> op_depth == 0 [ofcourse nicely documented in the table docs], BASE_NODE
> would be reduced to a store of the dav_cache, translated_size and
> last_mod_time fields.
>
> By subsuming translated_size and last_mod_time into NODE_DATA, neither
> WORKING_NODE nor BASE_NODE will need to store these values anymore. This
> eliminates the entire reason of existence of WORKING_NODE. BASE_NODE then
> only stores dav_cache. Here too, it's probably more efficient (in size) to
> store dav_cache in NODE_DATA to prevent repeated storage of wc_id,
> local_relpath and parent_relpath in BASE_NODE.
>
> In addition to the eliminated storage overhead, we'd be making things a
> little less complex for ourselves: UPDATE, INSERT and DELETE queries would
> be operating only on a single table, removing the need to split updates
> across multiple statements.
>
>
> This week, I was discussing this change with Greg on IRC. We both have the
> feeling this should work out well. The proposal here is to switch
> (WORKING_NODE, NODE_DATA, BASE_NODE) into a single table --> NODES.
>
>
> Comments? Fears? Enhancements?
I'm for the change, generally, though I admit I haven't been following
the discussion closely enough to evaluate the technical merits.
My one concern (and perhaps this comes from not following the
discussion closely enough) is how this impacts 1.7. This feels eerily
like an eleventh-hour redesign, and our track record with these in the
past hasn't been too stellar. I realize we've made some significant
progress in the past week with single-db and all that, but this just
feels like a fundamental change which has the potential to postpone
all the other 1.7 features we're eager to get into our users' hands.
The schema can *always* be improved; at what point are we allowing the
perfect to be the enemy of the good?
All that being said, if there is an SMOP in there somewhere, I'm happy
to unleash my OCD on it. :P
-Hyrum