You are viewing a plain text version of this content. The canonical link for it is here.
Posted to oak-dev@jackrabbit.apache.org by Tobias Bocanegra <tr...@apache.org> on 2013/10/15 02:39:37 UTC
content hash of a tree
Hi,
just wondering, how can I retrieve the content-hash of a subtree? shouldn't
that be a method on the tree interface?
regards, toby
Re: content hash of a tree
Posted by Thomas Mueller <mu...@adobe.com>.
Hi,
>With the current state of Oak, is Oak on Cassandra possible ?
I think yes. I don't know what would prevent us from using Cassandra /
HBase.
Regards,
Thomas
Re: content hash of a tree
Posted by Ian Boston <ie...@tfd.co.uk>.
On 15 October 2013 08:13, Thomas Mueller <mu...@adobe.com> wrote:
> Hi,
>
>>I thought this was one of the fundamental principles that
>>allow for quick diff/merge and help identify a commit within the MVCC
>>timeline.
>
> I'm not sure where you heard or read that, but a content hash is not
> needed for this; a revision number that is changed whenever there is a
> modification is enough. (For each change in this node or any direct or
> indirect child node.) For the MongoMK, the revision is a combination of
> the timestamp, the cluster node id, and a counter; very similar to the
> MongoDB object id: http://docs.mongodb.org/manual/reference/object-id/
>
>>what was the reason to abandon this idea?
>
> As for the MongoDB: performance and scalability. A node lookup by content
> hash would be bad for performance, as it would require an index on a
> randomly distributed data. See also
> http://fr.slideshare.net/daumdna/mongodb-scaling-write-performance - page
> 9 (the red line is with an index on the content hash, the green line
> without). But even without such an index: maintaining the content hash
> would be prohibitively expensive and would prevent scalable writes.
With the current state of Oak, is Oak on Cassandra possible ?
Slide 45.
Ian
>
> Regards,
> Thomas
>
Re: content hash of a tree
Posted by Tobias Bocanegra <tr...@apache.org>.
On Tue, Oct 15, 2013 at 11:53 AM, Jukka Zitting <ju...@gmail.com> wrote:
> Hi,
>
> On Tue, Oct 15, 2013 at 2:41 PM, Tobias Bocanegra <tr...@apache.org> wrote:
>> Maybe the jcr:etags need to be handled in a commit hook that also can be
>> configured to what to include in the etag-hash. for example, the
>> jcr:lastModified or jcr:created does not need to be included in the hash.
>
> Sounds good. Note that the JCR spec already defines what to include in
> the etag calculation:
> http://www.day.com/specs/jcr/2.0/3_Repository_Model.html#3.7.12%20Entity%20Tags
oh, thanks, forgot about that. so it's even easier. we just can use
the hashes of the binary properties of a node.
regards, toby
Re: content hash of a tree
Posted by Jukka Zitting <ju...@gmail.com>.
Hi,
On Tue, Oct 15, 2013 at 2:41 PM, Tobias Bocanegra <tr...@apache.org> wrote:
> Maybe the jcr:etags need to be handled in a commit hook that also can be
> configured to what to include in the etag-hash. for example, the
> jcr:lastModified or jcr:created does not need to be included in the hash.
Sounds good. Note that the JCR spec already defines what to include in
the etag calculation:
http://www.day.com/specs/jcr/2.0/3_Repository_Model.html#3.7.12%20Entity%20Tags
BR,
Jukka Zitting
Re: content hash of a tree
Posted by Tobias Bocanegra <tr...@apache.org>.
Hi,
On Tue, Oct 15, 2013 at 11:28 AM, Jukka Zitting <ju...@gmail.com>wrote:
> Hi,
>
> On Tue, Oct 15, 2013 at 1:53 PM, Tobias Bocanegra <tr...@apache.org>
> wrote:
> > * for example for NodeTypeImpl.equals() we could just compare the "hash"
> of
> > the substree storing the node type definition.
>
> I'd just compare the names of the node types. Or alternatively do a
> deep compare of the nodes, as the relevant trees aren't that big.
>
sure. the current implementation compares the CND of the 2 nodetypes.
>
> > * for example to provide jcr:etag
>
> That's a good idea, but unfortunately there's no easy way to expose
> content hashes without potentially breaking access controls. For
> example confidential information stored in a read-protected property
> could end up leaking through a content hash if the attacker knows the
> hashing algorithm, can read the rest of that node and has a list of
> likely alternatives for the confidential bits (or is just interested
> in the existence of such hidden details).
well, I think that's a bit far fetched :-)
> As another example exposing
> content hashes would make it possible for someone to infer the
> existence of a hidden subtree or the fact that such a subtree has been
> modified.
>
that's more likely. I agree.
Maybe the jcr:etags need to be handled in a commit hook that also can be
configured to what to include in the etag-hash. for example, the
jcr:lastModified or jcr:created does not need to be included in the hash.
> The best we can probably do is expose content hashes (or identifiers)
> on blobs, as there the partial access problem doesn't come up. The
> Blob interface actually already did expose a hash value, but I
> recently took it away since it wasn't yet used anywhere and its
> definition was too implementation-dependent. We can bring it back for
> example when we implement JackrabbitValue.
>
ok. thanks.
regards, toby
Re: content hash of a tree
Posted by Jukka Zitting <ju...@gmail.com>.
Hi,
On Tue, Oct 15, 2013 at 1:53 PM, Tobias Bocanegra <tr...@apache.org> wrote:
> * for example for NodeTypeImpl.equals() we could just compare the "hash" of
> the substree storing the node type definition.
I'd just compare the names of the node types. Or alternatively do a
deep compare of the nodes, as the relevant trees aren't that big.
> * for example to provide jcr:etag
That's a good idea, but unfortunately there's no easy way to expose
content hashes without potentially breaking access controls. For
example confidential information stored in a read-protected property
could end up leaking through a content hash if the attacker knows the
hashing algorithm, can read the rest of that node and has a list of
likely alternatives for the confidential bits (or is just interested
in the existence of such hidden details). As another example exposing
content hashes would make it possible for someone to infer the
existence of a hidden subtree or the fact that such a subtree has been
modified.
The best we can probably do is expose content hashes (or identifiers)
on blobs, as there the partial access problem doesn't come up. The
Blob interface actually already did expose a hash value, but I
recently took it away since it wasn't yet used anywhere and its
definition was too implementation-dependent. We can bring it back for
example when we implement JackrabbitValue.
BR,
Jukka Zitting
Re: content hash of a tree
Posted by Tobias Bocanegra <tr...@apache.org>.
Hi,
thanks for the update.
On Tue, Oct 15, 2013 at 9:52 AM, Jukka Zitting <ju...@gmail.com>wrote:
> Hi,
>
> On Tue, Oct 15, 2013 at 1:07 AM, Tobias Bocanegra <tr...@apache.org>
> wrote:
> > hmm, that's bad. I thought this was one of the fundamental principles
> that
> > allow for quick diff/merge and help identify a commit within the MVCC
> > timeline.
>
> As mentioned by others, we actually don't need content hashes for this
> and the alternatives have better performance/scalability aspects.
>
> What's the use case you have in mind?
>
* for example for NodeTypeImpl.equals() we could just compare the "hash" of
the substree storing the node type definition.
* for example to provide jcr:etag
Regards, Toby
Re: content hash of a tree
Posted by Jukka Zitting <ju...@gmail.com>.
Hi,
On Tue, Oct 15, 2013 at 1:07 AM, Tobias Bocanegra <tr...@apache.org> wrote:
> hmm, that's bad. I thought this was one of the fundamental principles that
> allow for quick diff/merge and help identify a commit within the MVCC
> timeline.
As mentioned by others, we actually don't need content hashes for this
and the alternatives have better performance/scalability aspects.
What's the use case you have in mind?
BR,
Jukka Zitting
RE: content hash of a tree
Posted by Marcel Reutegger <mr...@adobe.com>.
Hi,
> > You can't. The underlying MicroKernel/NodeStore does not necessarily
> > store content by hash. In fact neither the MongoMK nor the SegmentMK
> > do so.
> >
> hmm, that's bad. I thought this was one of the fundamental principles that
> allow for quick diff/merge and help identify a commit within the MVCC
> timeline.
> what was the reason to abandon this idea?
at least for the MongoMK we didn't want to serialize the commits across
multiple instances, which would be required with hashes.
regards
marcel
Re: content hash of a tree
Posted by Thomas Mueller <mu...@adobe.com>.
Hi,
>I thought this was one of the fundamental principles that
>allow for quick diff/merge and help identify a commit within the MVCC
>timeline.
I'm not sure where you heard or read that, but a content hash is not
needed for this; a revision number that is changed whenever there is a
modification is enough. (For each change in this node or any direct or
indirect child node.) For the MongoMK, the revision is a combination of
the timestamp, the cluster node id, and a counter; very similar to the
MongoDB object id: http://docs.mongodb.org/manual/reference/object-id/
>what was the reason to abandon this idea?
As for the MongoDB: performance and scalability. A node lookup by content
hash would be bad for performance, as it would require an index on a
randomly distributed data. See also
http://fr.slideshare.net/daumdna/mongodb-scaling-write-performance - page
9 (the red line is with an index on the content hash, the green line
without). But even without such an index: maintaining the content hash
would be prohibitively expensive and would prevent scalable writes.
Regards,
Thomas
Re: content hash of a tree
Posted by Tobias Bocanegra <tr...@apache.org>.
On Mon, Oct 14, 2013 at 7:08 PM, Jukka Zitting <ju...@gmail.com>wrote:
> Hi,
>
> On Mon, Oct 14, 2013 at 8:39 PM, Tobias Bocanegra <tr...@apache.org>
> wrote:
> > just wondering, how can I retrieve the content-hash of a subtree?
>
> You can't. The underlying MicroKernel/NodeStore does not necessarily
> store content by hash. In fact neither the MongoMK nor the SegmentMK
> do so.
>
hmm, that's bad. I thought this was one of the fundamental principles that
allow for quick diff/merge and help identify a commit within the MVCC
timeline.
what was the reason to abandon this idea?
regards, toby
Re: content hash of a tree
Posted by Jukka Zitting <ju...@gmail.com>.
Hi,
On Mon, Oct 14, 2013 at 8:39 PM, Tobias Bocanegra <tr...@apache.org> wrote:
> just wondering, how can I retrieve the content-hash of a subtree?
You can't. The underlying MicroKernel/NodeStore does not necessarily
store content by hash. In fact neither the MongoMK nor the SegmentMK
do so.
BR,
Jukka Zitting