DM Rule #5: References considered harmful.

[David Nuescheler <da...@gmail.com>]

Explanation
---
References imply referential integrity. I find it important to
understand that references do not just add additional cost for the
repository managing the referential integrity, but they also are
costly from a content flexibility perspective.

Personally I make sure I only ever use references when I really cannot
deal with a dangling reference and otherwise use a path, a name or a
string UUID to refer to another node.

Example
---
Let's assume I allow "references" from a document (a) to another
document (b). If I model this relation using reference properties this
means that the two documents are linked on a repository level. I
cannot export/import document (a) individually, since the reference
property's target may not exist. Other operations like merge, update,
restore or clone are affected as well.

So I would either model those references as "weak-references" (in JCR
v1.0 his essentially boils down to string properties that contain the
uuid of the target node) or simply use a path. Sometimes the path is
more meaningful to begin with.

I think there are usecases where a system really can't work if a
reference is dangling, but I just can't come up with a good "real" yet
simple example from my direct experience.

Re: DM Rule #5: References considered harmful.

[Tako Schotanus <qu...@gmail.com>]

On 7/10/07, David Nuescheler <da...@gmail.com> wrote:
> Hi Tako,
>
> thanks for your comment and for the excellent example ;)
>
> > /myapp/config/users/joe
> > /myapp/config/users/jane
> > /myapp/config/users/tarzan
> >
> > /myapp/docs/some-doc1/@owner = 'jane'
> >
> > where I just refer to the specific nodes in the users section and to
> > find those users I have a special configuration section:
> >
> > /myapp/config/paths/users = '/myapp/config/users'
>
> This is precisely how I would have modeled it as well.
>
> If there is anything contradictory in the "rules", it is very likely a
> mistake and should be corrected. Is there anything in particular
> that you had in mind?

No no, just reflecting on the fact that I miss referential integrity
in JCR paths :-)

I'm one of those people who think that things like that should be
explicitly declared instead of having to figure out the constraints
from docs (yeah sure) or code.

Cheers,
 -Tako

Re: DM Rule #5: References considered harmful.

[David Nuescheler <da...@gmail.com>]

Hi Tako,

thanks for your comment and for the excellent example ;)

> /myapp/config/users/joe
> /myapp/config/users/jane
> /myapp/config/users/tarzan
>
> /myapp/docs/some-doc1/@owner = 'jane'
>
> where I just refer to the specific nodes in the users section and to
> find those users I have a special configuration section:
>
> /myapp/config/paths/users = '/myapp/config/users'

This is precisely how I would have modeled it as well.

If there is anything contradictory in the "rules", it is very likely a
mistake and should be corrected. Is there anything in particular
that you had in mind?

regards,
david

Re: DM Rule #5: References considered harmful.

[Tako Schotanus <qu...@gmail.com>]

I think this goes directly against the directive of using meaningful
path names. Meaningful path names are only useful if they actually
reflect the data they "point to" and with data normally not being
static this means that you should take into account that you will have
to change your node names or move your nodes regularly. The moment you
do this you could possibly have left your repository with lots of
dangling references. The only way to prevent this would be to use
generated node names and a hierarchy that doesn't reflect the actual
relationship between parent and child nodes, which would defeat the
purpose of having a hierarchical repository in the first place.

That doesn't mean I think you should UUIDs instead, I'm just pointing
out a problem that I encountered myself.

In my latest designs I just tend to do something like this:

/myapp/config/users/joe
/myapp/config/users/jane
/myapp/config/users/tarzan

/myapp/docs/some-doc1/@owner = 'jane'

where I just refer to the specific nodes in the users section and to
find those users I have a special configuration section:

/myapp/config/paths/users = '/myapp/config/users'

and I "bootstrap" the app by pointing it to "/myapp/config/paths".
This way my application is completely independent of paths because I
don't like the idea that I can't move things around.

I would rather have used paths, but without referential integrity I
don't see much use except for the fact that you know it's not just any
string.


On 7/7/07, David Nuescheler <da...@gmail.com> wrote:
> Explanation
> ---
> References imply referential integrity. I find it important to
> understand that references do not just add additional cost for the
> repository managing the referential integrity, but they also are
> costly from a content flexibility perspective.
>
> Personally I make sure I only ever use references when I really cannot
> deal with a dangling reference and otherwise use a path, a name or a
> string UUID to refer to another node.
>
> Example
> ---
> Let's assume I allow "references" from a document (a) to another
> document (b). If I model this relation using reference properties this
> means that the two documents are linked on a repository level. I
> cannot export/import document (a) individually, since the reference
> property's target may not exist. Other operations like merge, update,
> restore or clone are affected as well.
>
> So I would either model those references as "weak-references" (in JCR
> v1.0 his essentially boils down to string properties that contain the
> uuid of the target node) or simply use a path. Sometimes the path is
> more meaningful to begin with.
>
> I think there are usecases where a system really can't work if a
> reference is dangling, but I just can't come up with a good "real" yet
> simple example from my direct experience.
>

Re: DM Rule #5: References considered harmful.

[Jukka Zitting <ju...@gmail.com>]

Hi,

On 7/7/07, David Nuescheler <da...@gmail.com> wrote:
> References imply referential integrity. I find it important to
> understand that references do not just add additional cost for the
> repository managing the referential integrity, but they also are
> costly from a content flexibility perspective.
>
> Personally I make sure I only ever use references when I really cannot
> deal with a dangling reference and otherwise use a path, a name or a
> string UUID to refer to another node.

Since there are cases where references are really useful, I would
rather label this rule as "Moderate use of references" than implying
that all references are harmful.

I would actually argue that similar flexibility costs arise regardless
of whether you use hard or soft references. The main problem is that
you are going beyond the native hierarchy model, and this will always
have inherent costs. In fact, when you really need to do that, using
hard references is IMHO an acceptable and good solution.

However, here are my rules of thumb when dealing with references (be
they hard references, paths, UUID strings, or even just naming
conventions) between separate subtrees:

1) Don't do that. If there's a way to make all your content
hierarchical, use it. This applies for example to the blog post vs.
comments example in David's example #1.

2) If there's an alternative, use it. If there already is some natural
unique identifier in the target node (usernames are typical examples),
then it makes sense to use that instead of a reference or path
property (for example you want to store the username of a comment
author instead of the UUID or path of a user node).

3) When no alternatives exist, use references just in one direction.
If I have two subtrees A and B whose content I need to interlink, then
it makes sense to have all references going from A to B or vice versa
but not in both directions. This makes it possible to manage (backup,
migrate, etc.) at least the other subtree without worrying about the
other one. Note that this is how the version store works in JCR, the
version history references from normal content are always directed to
/jcr:system/jcr:versionStorage, but the version store has no
references back to normal content.

4) If you really need references in both directions between two
subtrees, manage those subtees as a part of a larger tree. If two
subtrees are hopelessly intertwined (and you can't find a way to merge
them), then it's better if you don't think of them as separate trees
but rather as branches of a bigger tree. This way you can still backup
and migrate the whole tree without problems due to dangling
references.

BR,

Jukka Zitting

Re: DM Rule #5: References considered harmful.

[Felix Meschberger <fm...@gmail.com>]

Hi David,

I have the issue of creating a caching system, where I have my cache
entries, which have dependencies of certain data. Of course, I would not
create REFERENCEs to the real data to not lock down the user managing
the data. But how about creating a mirror hierarchy of data to which the
cache entries REFERENCE.

In this case, when the real data is modified I can lookup the mirror and
find the referring cache entries. The nice thing of having a REFERENCE
property in this case is, that I can get the referents from the data
(actually the mirro of it) node without having to employ a heavy weight
search.

What do you think ?

Regards
Felix

Am Samstag, den 07.07.2007, 13:20 +0200 schrieb David Nuescheler:
> Explanation
> ---
> References imply referential integrity. I find it important to
> understand that references do not just add additional cost for the
> repository managing the referential integrity, but they also are
> costly from a content flexibility perspective.
> 
> Personally I make sure I only ever use references when I really cannot
> deal with a dangling reference and otherwise use a path, a name or a
> string UUID to refer to another node.
> 
> Example
> ---
> Let's assume I allow "references" from a document (a) to another
> document (b). If I model this relation using reference properties this
> means that the two documents are linked on a repository level. I
> cannot export/import document (a) individually, since the reference
> property's target may not exist. Other operations like merge, update,
> restore or clone are affected as well.
> 
> So I would either model those references as "weak-references" (in JCR
> v1.0 his essentially boils down to string properties that contain the
> uuid of the target node) or simply use a path. Sometimes the path is
> more meaningful to begin with.
> 
> I think there are usecases where a system really can't work if a
> reference is dangling, but I just can't come up with a good "real" yet
> simple example from my direct experience.

Re: DM Rule #5: References considered harmful.

[Mark Waschkowski <mw...@gmail.com>]

Hi David,

Thanks again for initiating this discussion through your posts, its just
excellent.

I was about to respond in the other thread about same name siblings, but
there are a lot of posts in that thread and my comment seems more suited to
this thread. However, I found the below (from the other thread) to be most
helpful:

"I think it is important to look at (a) a meaningful path that is
explanatory of the context that you are in and (b) stable and unique
identification as two totally separate issues."

I really like the idea of modeling contacts like (which is different that I
originally planned ie. SNS):
/contacts/david@ day.com
/contacts/dev@ jackrabbit.apache.org

This is simple, and has meaning. However, if the repository is used (at
least in part) to replace some database type functionality, then dangling
references are unacceptable. I believe the following is a use case for
needing stable identification:

User stores a bunch of contacts
User stores a bunch of buildings
User wants to link each building to a contact.
User wants to link reports to a contact.
User wants to link 'x' to a contact.

Different buildings may want to refer to same contact. In future, other
entities may also want to refer to contacts. Of course, we don't want to
link contacts to other things as we shouldn't have to touch a contact when
adding other entities to the system.

Therefore, I would think that it would make sense for the contacts to have a
uuid, and everything else (in this case buildings, but there could be many
other entities that need to refer to a contact) refer to the contact by the
uuid. Then, if the contact's email address changed, that particular node
could be moved, but its uuid wouldn't change so all the references would
remain intact. I believe that using a path and then having to refactor all
nodes that refer to the contact by its path would be error prone.

Thus, it seems to me that you get both a semantic path for the contact, and
the benefits of referential integrity when needed, without worrying about
error prone refactoring. What do you think?

Best,

Mark

On 7/10/07, David Nuescheler < david.nuescheler@gmail.com > wrote:
>
> Hi Mark,
>
> Thanks a lot for your comment.
>
> > /contacts/johndoe
> > /contacts/janedoe
> > etc.
> > Maybe just using
> > /contact/contact[1]
> > /contact/contact[2]
> > and using uuid's to reference individuals make sense for this?
>
> I think it is important to look at (a) a meaningful path that is
> explanatory of the context that you are in and (b) stable and unique
> identification as two totally separate issues.
>
> (a) Meaningful Path
> ------
> [which really belongs into the "drive your hierarchy" section]
>
> I mainly use SNS as a symptom to diagnose a hierarchy issue
> in a content model.
>
> People use SNS mostly for large, unordered collections,
> which it is definitely not what they are designed for.
>
> [note to self: propose a better way to address this problem in jsr-283]
>
> SNS means that if you for example remove contact[1] the
> paths of all other contacts and all their children will need to
> change.
>
> Keep in mind that you force the repository to keep track of the
> order of your contacts. Which you are actually not interested in
> I would assume.
>
> While I think that...
>
> /contacts/0
> /contacts/1
> /contacts/2
> /contacts/3
>
> ... is better from that perspective, since the repository does not
> need to keep track of sort order and you get better path stability,
>
> I would still try something like...
>
> /contacts/david@day.com
> /contacts/dev@ jackrabbit.apache.org
>
> ...which could be both meaningful and unique.
>
> The reason for that is that someone who browses the repository
> with any form of generic application ...
>
> (like http://jsr170tools.day.com/crx/browser/index.jsp )
>
> ... understands get a meaningful list of contacts and
> the path reveals information about the context of the
> information.
>
> Let's assume the path should ever surface, for example in a URL,
> then...
>
> /contacts/mark@gmail/ituneslibrary/Nelly Furtado/Loose/Say It Right
>
> will give the user much more context than ...
>
> /contacts/contact[42342]/ituneslib/artist[234]/album[2]/track[8]
>
> ..and really, i cannot see a drawback for naming
> nodes in a meaningful way. ;)
>
> ---------
> (b) Stable and Unique Identification
> ----
> If your contacts should be mix:referenceable and therefore expose
> a UUID so they can be referenced in my mind is decided by
> the needs of you application.
>
> If you application needs to reference users in a stable fashion and
> you don't have something like a unique stable user-id (eg. an email
> or a login...) mix:referenceable certainly would be the way to go.
>
> (Important: this does not mean that a hard reference property
> needs to be used to refer to or identify the node later)
>
> -----
>
> Please keep in mind, this is just how I would model the content,
> and I am convinced that someone could possibly come up with
> a usecase where even I would warm up to the idea of using SNS... ;)
>
> regards,
> david
>



-- 
Best,

Mark Waschkowski

On 7/10/07, David Nuescheler <david.nuescheler@gmail.com > wrote:
>
> Hi Mark,
>
> > What about when the repository is being used instead of a database?
> > User stores a bunch of contacts
> > User stores a bunch of buildings
> > User wants to link each building to a contact. Different buildings may
> want
> > to refer to same contact.
>
> From my gut feeling I would recommend something like...
>
> /buildings/mainbuilding
> /buildings/th14
> /buildings/th5
>
> and
>
> /contacts/jack
> /contacts/joe
> /contacts/jill
>
> where the contacts have a multivalue "name"
> property called "buildings" containing strings like "mainbuilding",
> "th14", ...
>
> This is working off the assumption that the buildings are more
> static/stable than contacts. So I would choose the "link" pointing
> to the buildings.
>
> The reason why I chose a "name" property rather than a "reference"
> is really because I think the application could gracefully fail a dangling
>
> link to a building and because i think that the list of buildings is
> very static.
> Opposed to a "uuid" stuck in a reference the string "mainbuilding" will
> mean something to someone looking at it.
>
> > Isn't this a perfect use case for using a reference?
> If your application cannot afford dangling references and
> the building names are dynamic I would employ references.
>
> I think it is important to think about the other options
> that a content repository offers to link content before defaulting
> to the most rigid and expensive one (which is a reference).
>
> regards,
> david
>



-- 
Best,

Mark Waschkowski

Re: DM Rule #5: References considered harmful.

[David Nuescheler <da...@gmail.com>]

Hi Mark,

> What about when the repository is being used instead of a database?
> User stores a bunch of contacts
> User stores a bunch of buildings
> User wants to link each building to a contact. Different buildings may want
> to refer to same contact.

>From my gut feeling I would recommend something like...

/buildings/mainbuilding
/buildings/th14
/buildings/th5

and

/contacts/jack
/contacts/joe
/contacts/jill

where the contacts have a multivalue "name"
property called "buildings" containing strings like "mainbuilding",
"th14", ...

This is working off the assumption that the buildings are more
static/stable than contacts. So I would choose the "link" pointing
to the buildings.

The reason why I chose a "name" property rather than a "reference"
is really because I think the application could gracefully fail a dangling
link to a building and because i think that the list of buildings is
very static.
Opposed to a "uuid" stuck in a reference the string "mainbuilding" will
mean something to someone looking at it.

> Isn't this a perfect use case for using a reference?
If your application cannot afford dangling references and
the building names are dynamic I would employ references.

I think it is important to think about the other options
that a content repository offers to link content before defaulting
to the most rigid and expensive one (which is a reference).

regards,
david

Re: DM Rule #5: References considered harmful.

[Mark Waschkowski <mw...@gmail.com>]

Hi David,

Really appreciate you bringing these comments to the mailing list, its
making me really think about how to do modeling within a repository, thanks!

What about when the repository is being used instead of a database?

ie.
User stores a bunch of contacts
User stores a bunch of buildings
User wants to link each building to a contact. Different buildings may want
to refer to same contact.

Isn't this a perfect use case for using a reference?

Cheers,

Mark

On 7/7/07, David Nuescheler <da...@gmail.com> wrote:
>
> Explanation
> ---
> References imply referential integrity. I find it important to
> understand that references do not just add additional cost for the
> repository managing the referential integrity, but they also are
> costly from a content flexibility perspective.
>
> Personally I make sure I only ever use references when I really cannot
> deal with a dangling reference and otherwise use a path, a name or a
> string UUID to refer to another node.
>
> Example
> ---
> Let's assume I allow "references" from a document (a) to another
> document (b). If I model this relation using reference properties this
> means that the two documents are linked on a repository level. I
> cannot export/import document (a) individually, since the reference
> property's target may not exist. Other operations like merge, update,
> restore or clone are affected as well.
>
> So I would either model those references as "weak-references" (in JCR
> v1.0 his essentially boils down to string properties that contain the
> uuid of the target node) or simply use a path. Sometimes the path is
> more meaningful to begin with.
>
> I think there are usecases where a system really can't work if a
> reference is dangling, but I just can't come up with a good "real" yet
> simple example from my direct experience.
>



-- 
Best,

Mark Waschkowski