You are viewing a plain text version of this content. The canonical link for it is here.
Posted to oak-dev@jackrabbit.apache.org by Tobias Bocanegra <tr...@apache.org> on 2013/10/31 02:30:20 UTC
WeakReferences Query
Hi,
I'm looking at a strange problem where a WeakRef reverse lookup
results in a Node that does not point to the respective reference:
org.apache.jackrabbit.oak.spi.query.Cursors$TraversingCursor Traversed
11000 nodes with filter Filter(query=SELECT * FROM [nt:base] WHERE
PROPERTY([ref], 'WeakReference') = $uuid, path=/*,
property=[ref=559f6184-5873-4899-85f0-40d9b24403d9]); consider
creating an index or changing the query
In this particular case, the node/row returned is a frozen node in the
version store, which does not have a ref property at all. I try to
come up with a test case.
also, from reading the javadoc at Node.getWeakReferences(String):
* In implementations that support versioning, this method does not return
* properties that are part of the frozen state of a version in version
* storage.
so even searching the version store is wrong.
However, I think that we should auto-generate property indexes for
WeakReferences, if this is possible.
WDYT?
Regards, Toby
Re: WeakReferences Query
Posted by Tobias Bocanegra <tr...@apache.org>.
Thanks. I think it's important to find a solution for this, so I set
https://issues.apache.org/jira/browse/OAK-1137 to critical.
Regards, Toby
On Thu, Oct 31, 2013 at 6:48 AM, Thomas Mueller <mu...@adobe.com> wrote:
> Hi,
>
>>>With the SegmentMK it would also be possible (and a bit more
>>>efficient) to use multi-valued property containing all the referencing
>>>paths, as the underlying list implementation can scale to millions of
>>>entries with O(log n) updates. But a tree structure should also work
>>>fine.
>>
>>Nice! Unfortunately, the MongoMK implementation doesn't support this.
>
> Sorry, to clairify: the MongoMK doesn't support very large multi-valued
> properties. It would need a tree structure.
>
> Regards,
> Thomas
>
Re: WeakReferences Query
Posted by Thomas Mueller <mu...@adobe.com>.
Hi,
>>With the SegmentMK it would also be possible (and a bit more
>>efficient) to use multi-valued property containing all the referencing
>>paths, as the underlying list implementation can scale to millions of
>>entries with O(log n) updates. But a tree structure should also work
>>fine.
>
>Nice! Unfortunately, the MongoMK implementation doesn't support this.
Sorry, to clairify: the MongoMK doesn't support very large multi-valued
properties. It would need a tree structure.
Regards,
Thomas
Re: WeakReferences Query
Posted by Thomas Mueller <mu...@adobe.com>.
Hi,
>With the SegmentMK it would also be possible (and a bit more
>efficient) to use multi-valued property containing all the referencing
>paths, as the underlying list implementation can scale to millions of
>entries with O(log n) updates. But a tree structure should also work
>fine.
Nice! Unfortunately, the MongoMK implementation doesn't support this.
Regards,
Thomas
Re: WeakReferences Query
Posted by Jukka Zitting <ju...@gmail.com>.
Hi,
On Thu, Oct 31, 2013 at 9:09 AM, Thomas Mueller <mu...@adobe.com> wrote:
> Instead, I suggest we support (both?) weak and hard references as follows:
> at the target node, add a hidden child node (called ":references" or so).
+1
> Within this node, store all references as a tree.
With the SegmentMK it would also be possible (and a bit more
efficient) to use multi-valued property containing all the referencing
paths, as the underlying list implementation can scale to millions of
entries with O(log n) updates. But a tree structure should also work
fine.
BR,
Jukka Zitting
Re: WeakReferences Query
Posted by Thomas Mueller <mu...@adobe.com>.
Hi,
In early versions of Oak, there was an index implementation that supported
indexing properties of certain types, such that queries with conditions
like "WHERE PROPERTY([ref], 'WeakReference') = $uuid" were fast. But this
index implementation was removed, so right now the query still works but
traverses the whole repository.
We could implement an index for property types. But I would rather not do
that, as it is quite a lot of work.
Instead, I suggest we support (both?) weak and hard references as follows:
at the target node, add a hidden child node (called ":references" or so).
Within this node, store all references as a tree. So for example if /a/b/c
references /x, then there would be a hidden node /x/:references/a/b/c. I
wonder whether removing a node (or one of its parents) that references
another node is supposed to remove the reference or if this should be done
lazily.
Regards,
Thomas
On 10/31/13 5:28 AM, "Tobias Bocanegra" <tr...@apache.org> wrote:
>Actually, I was wrong. I think the iterator is correct and only
>reports properties that actually point to the respective node. but due
>to an NPE in the permission evaluation in the version store, I thought
>that was an offending result. So I think the query is correct.
>
>Nevertheless, I think we should auto-index all (weak) references,
>because traversing all items in the repository is certainly not an
>option.
>Regards, Toby
>
>On Wed, Oct 30, 2013 at 6:30 PM, Tobias Bocanegra <tr...@apache.org>
>wrote:
>> Hi,
>>
>> I'm looking at a strange problem where a WeakRef reverse lookup
>> results in a Node that does not point to the respective reference:
>>
>> org.apache.jackrabbit.oak.spi.query.Cursors$TraversingCursor Traversed
>> 11000 nodes with filter Filter(query=SELECT * FROM [nt:base] WHERE
>> PROPERTY([ref], 'WeakReference') = $uuid, path=/*,
>> property=[ref=559f6184-5873-4899-85f0-40d9b24403d9]); consider
>> creating an index or changing the query
>>
>> In this particular case, the node/row returned is a frozen node in the
>> version store, which does not have a ref property at all. I try to
>> come up with a test case.
>>
>> also, from reading the javadoc at Node.getWeakReferences(String):
>>
>> * In implementations that support versioning, this method does not
>>return
>> * properties that are part of the frozen state of a version in
>>version
>> * storage.
>>
>> so even searching the version store is wrong.
>>
>> However, I think that we should auto-generate property indexes for
>> WeakReferences, if this is possible.
>>
>> WDYT?
>>
>> Regards, Toby
Re: WeakReferences Query
Posted by Tobias Bocanegra <tr...@apache.org>.
Actually, I was wrong. I think the iterator is correct and only
reports properties that actually point to the respective node. but due
to an NPE in the permission evaluation in the version store, I thought
that was an offending result. So I think the query is correct.
Nevertheless, I think we should auto-index all (weak) references,
because traversing all items in the repository is certainly not an
option.
Regards, Toby
On Wed, Oct 30, 2013 at 6:30 PM, Tobias Bocanegra <tr...@apache.org> wrote:
> Hi,
>
> I'm looking at a strange problem where a WeakRef reverse lookup
> results in a Node that does not point to the respective reference:
>
> org.apache.jackrabbit.oak.spi.query.Cursors$TraversingCursor Traversed
> 11000 nodes with filter Filter(query=SELECT * FROM [nt:base] WHERE
> PROPERTY([ref], 'WeakReference') = $uuid, path=/*,
> property=[ref=559f6184-5873-4899-85f0-40d9b24403d9]); consider
> creating an index or changing the query
>
> In this particular case, the node/row returned is a frozen node in the
> version store, which does not have a ref property at all. I try to
> come up with a test case.
>
> also, from reading the javadoc at Node.getWeakReferences(String):
>
> * In implementations that support versioning, this method does not return
> * properties that are part of the frozen state of a version in version
> * storage.
>
> so even searching the version store is wrong.
>
> However, I think that we should auto-generate property indexes for
> WeakReferences, if this is possible.
>
> WDYT?
>
> Regards, Toby