Custom index type

[rachna <ra...@telegraph.co.uk>]

Hi,  we have built a custom index type and created jackrabbit oak index using
this type, index gets populated with the correct data but now when I try to
read it using standard jcr nodes, it's not allowing me to read hidden nodes. 
I think I need to use NodeState and NodeBuilder classes to read this index,
but struggling to get root object on oak:index node.  Does anyone know how
can I do that?

Thanks



--
View this message in context: http://jackrabbit.510166.n4.nabble.com/Custom-index-type-tp4665031.html
Sent from the Jackrabbit - Users mailing list archive at Nabble.com.

Re: Custom index type

[Clay Ferguson <wc...@gmail.com>]

With all due respect it very much sounds like you are doing this hard way,
if not the impossible way. Here's an example of some of my search code:

*https://github.com/Clay-Ferguson/meta64/blob/master/src/main/java/com/meta64/mobile/service/NodeSearchService.java
<https://github.com/Clay-Ferguson/meta64/blob/master/src/main/java/com/meta64/mobile/service/NodeSearchService.java>*You
can see i'm using ISDESCENDANTNODE so limit search to only certain areas of
the tree.

It sounds like you may be worried about queriying at every level of the
tree when you shouldn't. Can you just say what you are doing like: "I'm
trying to find all nodes that have a certain property value under a certain
portion of the tree". or smoething like that just to put it in plain
english. Or even sent an example of the kinds of queries you are trying to
run. I still think it's likely you don't need a custom indexer. It sounds
like you just need to find an approach that keeps your query from being
"order of N" number of parameters just because the tree depth is "N"
parameters. Normally the tree depth is never needed in a query. Also if you
DO need to run specific queries at each level then run MULTIPLE queries.
Don't try to create one massive query that runs all levels of the tree at
the same time. ISDESCENDANT node may be able to help.

Best regards,
Clay Ferguson
wclayf@gmail.com


On Thu, Oct 13, 2016 at 11:02 AM, rachna <ra...@telegraph.co.uk>
wrote:

> Hi Clay & Thomas,
>
> Thanks for your replies.
>
> We have previously developed a feature that uses the tag management system
> inside of AEM. The current business logic performs a search against the
> repository using the tags associated with the content page. As part of this
> business logic, the decendants of the tags (and decendants of these tags
> until the bottom of the tree is reached) are also added to the query. This
> creates a large query that isn't very efficient. We've hit the 1024 limit
> on
> parameters for the query api.
>
> Our custom indexer analyses the decendants of the tags when the reindex
> operation is triggered so we can precompute this and store it efficently in
> the index.
> (We store the page path against the decendant tags in the index)
>
> Ideally we would want to query our index directly (e.g. native query) to
> return the results. From looking at the documentation for the lucene index,
> I can see this is possible.
> The documentation for the property index does not have the same query
> listed.
>
> Is it possible to run a native query against a custom indexer?
> Otherwise, can we programmtically access a new NodeState?
>
> We've hit the 1024 limit on parameters for the query api.  Also we've hit
> the 1024 limit on parameters for the query api.
>
>
>
>
>
> --
> View this message in context: http://jackrabbit.510166.n4.
> nabble.com/Custom-index-type-tp4665031p4665112.html
> Sent from the Jackrabbit - Users mailing list archive at Nabble.com.
>

Re: Custom index type

[Clay Ferguson <wc...@gmail.com>]

​Rachna,
I think the fulltext search idea you were trying to use should actually
work. You just need to be sure to configure the index properly for fulltext
search (I have exmaple of that in meta64.com on github), and be sure your
concatenation between "key:value" is something Lucene will see as a "normal
character" so that it won's split up your search and find "key" and "value"
as separate strings.  Good luck. ​

Best regards,
Clay Ferguson
wclayf@gmail.com


On Tue, Oct 18, 2016 at 3:08 AM, rachna <ra...@telegraph.co.uk>
wrote:

> Hi Clay & Thomas,
>
> Thanks for the reply, we are planning to create a Lucene property index on
> top of oak content, I will update the ticket with our Findings.
>
> Your help much appreciated.
>
> Many thanks,
> Rachna
>
>
>
> --
> View this message in context: http://jackrabbit.510166.n4.
> nabble.com/Custom-index-type-tp4665031p4665131.html
> Sent from the Jackrabbit - Users mailing list archive at Nabble.com.
>

Re: Custom index type

[rachna <ra...@telegraph.co.uk>]

Hi Clay & Thomas,

Thanks for the reply, we are planning to create a Lucene property index on
top of oak content, I will update the ticket with our Findings.

Your help much appreciated.

Many thanks,
Rachna



--
View this message in context: http://jackrabbit.510166.n4.nabble.com/Custom-index-type-tp4665031p4665131.html
Sent from the Jackrabbit - Users mailing list archive at Nabble.com.

Re: Custom index type

[Thomas Mueller <mu...@adobe.com>]

Hi,

Using "like" with a regular property index will not work well. What it
does is using the index as a "cq:tags is not null", so it will traverse
all entries.

What you could try is using a Lucene property index, in combination with
"like".

Regards,
Thomas


On 14/10/16 17:29, "rachna" <ra...@telegraph.co.uk> wrote:

>Thanks Clay & Thomas.
>
>Taking a step back from our problem has helped to look at it in a
>different
>way.
>
>The tag property also stores the values in a specific format that show the
>tree structure.
>
>cq:tags
>- location:europe
>- type:waterfalls
>
>Therefore instead of traversing the repository to identify the descendants
>of these tags, we could use a LIKE query.
>
>e.g. SELECT * FROM [cq:PageContent] AS b WHERE ISDESCENDANTNODE(b,
>[/content/guides]) AND ([cq:tags] LIKE 'location:europe%' OR [cq:tags]
>LIKE
>'type:waterfalls%') ORDER BY [cq:lastModified]
>
>However, since our repository contains a large number of items that match
>this criteria, we start to see warnings about traversing the index.
>
>org.apache.jackrabbit.oak.plugins.index.property.strategy.ContentMirrorSto
>reStrategy
>Traversed 210000 nodes (210164 index entries) using index jcr:primaryType
>with filter Filter(query=SELECT * FROM [cq:PageContent] AS b WHERE
>ISDESCENDANTNODE(b, [/content/guides]) AND ([cq:tags] LIKE
>'location:europe%' OR [cq:tags] LIKE 'type:waterfalls%') ORDER BY
>[cq:lastModified], path=/content/guides//*, property=[cq:tags=[is not
>null]])
>
>Instead, I created a lucene index that indexes the cq:tags (/w full text)
>and cq:lastModified (/w ordered support) property.
>
>e.g. SELECT [jcr:path] FROM [cq:PageContent] AS b WHERE
>ISDESCENDANTNODE(b,
>[/content/guides]) AND (CONTAINS([cq:tags], 'location:europe') OR
>CONTAINS([cq:tags], 'type:waterfalls')) ORDER BY [cq:lastModified]
>
>That seems to be much faster than using a property index and should solve
>most of the issues that we might have (hopefully avoiding creating a new
>index).
>
>Is there any support with the lucene index to use something like
>STARTSWITH
>rather CONTAINS?
>
>The maxClauseCount configuration parameter introduced the soft limit of
>1024
>which is part of Jackrabbit 2.
>We have been attempting to move to oak however our progress has been slow
>due to repository inconsistencies.
>I realise this value is configurable however constantly increasing it
>doesn't sound the right thing to do.
>
>Thanks,
>Rachna
>
>
>
>--
>View this message in context:
>http://jackrabbit.510166.n4.nabble.com/Custom-index-type-tp4665031p4665121
>.html
>Sent from the Jackrabbit - Users mailing list archive at Nabble.com.

Re: Custom index type

[Thomas Mueller <mu...@adobe.com>]

Hi,

> "What's the best way to query for LARGE NUMBERS of key/value pairs?"

As I wrote, I'm not aware of a limit (on the number of values or the query
length) in Oak. But sure, if it is possible to avoid having a large query,
then it should be avoided, for simplicity.

Regards,
Thomas




On 14/10/16 18:28, "Clay Ferguson" <wc...@gmail.com> wrote:

>The "Traversed 210000 nodes" warning is really telling you that it was
>unable to use your indexes to perform the search. (I think) This doesn't
>mean too many results were found, it just means you didn't create all the
>right indexes for the search. Just create an index for each property, and
>then search them the normal way (without the LIKE clause, but using '=')
>and I bet you will see good performance.  If you genuinely have thousands
>of key/value pairs to search it is possible that your full-text approach
>is
>the best performing solution, but I'm not sure.
>
>However your general question is: "What's the best way to query for LARGE
>NUMBERS of key/value pairs?"
>
>Maybe some experts who know more than me about Oak can reply to that
>simplified version of your question.
>
>Best regards,
>Clay Ferguson
>wclayf@gmail.com
>
>
>On Fri, Oct 14, 2016 at 10:29 AM, rachna <ra...@telegraph.co.uk>
>wrote:
>
>> Thanks Clay & Thomas.
>>
>> Taking a step back from our problem has helped to look at it in a
>>different
>> way.
>>
>> The tag property also stores the values in a specific format that show
>>the
>> tree structure.
>>
>> cq:tags
>> - location:europe
>> - type:waterfalls
>>
>> Therefore instead of traversing the repository to identify the
>>descendants
>> of these tags, we could use a LIKE query.
>>
>> e.g. SELECT * FROM [cq:PageContent] AS b WHERE ISDESCENDANTNODE(b,
>> [/content/guides]) AND ([cq:tags] LIKE 'location:europe%' OR [cq:tags]
>>LIKE
>> 'type:waterfalls%') ORDER BY [cq:lastModified]
>>
>> However, since our repository contains a large number of items that
>>match
>> this criteria, we start to see warnings about traversing the index.
>>
>> org.apache.jackrabbit.oak.plugins.index.property.strategy.
>> ContentMirrorStoreStrategy
>> Traversed 210000 nodes (210164 index entries) using index
>>jcr:primaryType
>> with filter Filter(query=SELECT * FROM [cq:PageContent] AS b WHERE
>> ISDESCENDANTNODE(b, [/content/guides]) AND ([cq:tags] LIKE
>> 'location:europe%' OR [cq:tags] LIKE 'type:waterfalls%') ORDER BY
>> [cq:lastModified], path=/content/guides//*, property=[cq:tags=[is not
>> null]])
>>
>> Instead, I created a lucene index that indexes the cq:tags (/w full
>>text)
>> and cq:lastModified (/w ordered support) property.
>>
>> e.g. SELECT [jcr:path] FROM [cq:PageContent] AS b WHERE
>>ISDESCENDANTNODE(b,
>> [/content/guides]) AND (CONTAINS([cq:tags], 'location:europe') OR
>> CONTAINS([cq:tags], 'type:waterfalls')) ORDER BY [cq:lastModified]
>>
>> That seems to be much faster than using a property index and should
>>solve
>> most of the issues that we might have (hopefully avoiding creating a new
>> index).
>>
>> Is there any support with the lucene index to use something like
>>STARTSWITH
>> rather CONTAINS?
>>
>> The maxClauseCount configuration parameter introduced the soft limit of
>> 1024
>> which is part of Jackrabbit 2.
>> We have been attempting to move to oak however our progress has been
>>slow
>> due to repository inconsistencies.
>> I realise this value is configurable however constantly increasing it
>> doesn't sound the right thing to do.
>>
>> Thanks,
>> Rachna
>>
>>
>>
>> --
>> View this message in context: http://jackrabbit.510166.n4.
>> nabble.com/Custom-index-type-tp4665031p4665121.html
>> Sent from the Jackrabbit - Users mailing list archive at Nabble.com.
>>

Re: Custom index type

[Clay Ferguson <wc...@gmail.com>]

The "Traversed 210000 nodes" warning is really telling you that it was
unable to use your indexes to perform the search. (I think) This doesn't
mean too many results were found, it just means you didn't create all the
right indexes for the search. Just create an index for each property, and
then search them the normal way (without the LIKE clause, but using '=')
and I bet you will see good performance.  If you genuinely have thousands
of key/value pairs to search it is possible that your full-text approach is
the best performing solution, but I'm not sure.

However your general question is: "What's the best way to query for LARGE
NUMBERS of key/value pairs?"

Maybe some experts who know more than me about Oak can reply to that
simplified version of your question.

Best regards,
Clay Ferguson
wclayf@gmail.com


On Fri, Oct 14, 2016 at 10:29 AM, rachna <ra...@telegraph.co.uk>
wrote:

> Thanks Clay & Thomas.
>
> Taking a step back from our problem has helped to look at it in a different
> way.
>
> The tag property also stores the values in a specific format that show the
> tree structure.
>
> cq:tags
> - location:europe
> - type:waterfalls
>
> Therefore instead of traversing the repository to identify the descendants
> of these tags, we could use a LIKE query.
>
> e.g. SELECT * FROM [cq:PageContent] AS b WHERE ISDESCENDANTNODE(b,
> [/content/guides]) AND ([cq:tags] LIKE 'location:europe%' OR [cq:tags] LIKE
> 'type:waterfalls%') ORDER BY [cq:lastModified]
>
> However, since our repository contains a large number of items that match
> this criteria, we start to see warnings about traversing the index.
>
> org.apache.jackrabbit.oak.plugins.index.property.strategy.
> ContentMirrorStoreStrategy
> Traversed 210000 nodes (210164 index entries) using index jcr:primaryType
> with filter Filter(query=SELECT * FROM [cq:PageContent] AS b WHERE
> ISDESCENDANTNODE(b, [/content/guides]) AND ([cq:tags] LIKE
> 'location:europe%' OR [cq:tags] LIKE 'type:waterfalls%') ORDER BY
> [cq:lastModified], path=/content/guides//*, property=[cq:tags=[is not
> null]])
>
> Instead, I created a lucene index that indexes the cq:tags (/w full text)
> and cq:lastModified (/w ordered support) property.
>
> e.g. SELECT [jcr:path] FROM [cq:PageContent] AS b WHERE ISDESCENDANTNODE(b,
> [/content/guides]) AND (CONTAINS([cq:tags], 'location:europe') OR
> CONTAINS([cq:tags], 'type:waterfalls')) ORDER BY [cq:lastModified]
>
> That seems to be much faster than using a property index and should solve
> most of the issues that we might have (hopefully avoiding creating a new
> index).
>
> Is there any support with the lucene index to use something like STARTSWITH
> rather CONTAINS?
>
> The maxClauseCount configuration parameter introduced the soft limit of
> 1024
> which is part of Jackrabbit 2.
> We have been attempting to move to oak however our progress has been slow
> due to repository inconsistencies.
> I realise this value is configurable however constantly increasing it
> doesn't sound the right thing to do.
>
> Thanks,
> Rachna
>
>
>
> --
> View this message in context: http://jackrabbit.510166.n4.
> nabble.com/Custom-index-type-tp4665031p4665121.html
> Sent from the Jackrabbit - Users mailing list archive at Nabble.com.
>

Re: Custom index type

[rachna <ra...@telegraph.co.uk>]

Thanks Clay & Thomas.

Taking a step back from our problem has helped to look at it in a different
way.

The tag property also stores the values in a specific format that show the
tree structure.

cq:tags
- location:europe
- type:waterfalls

Therefore instead of traversing the repository to identify the descendants
of these tags, we could use a LIKE query.

e.g. SELECT * FROM [cq:PageContent] AS b WHERE ISDESCENDANTNODE(b,
[/content/guides]) AND ([cq:tags] LIKE 'location:europe%' OR [cq:tags] LIKE
'type:waterfalls%') ORDER BY [cq:lastModified]

However, since our repository contains a large number of items that match
this criteria, we start to see warnings about traversing the index.

org.apache.jackrabbit.oak.plugins.index.property.strategy.ContentMirrorStoreStrategy
Traversed 210000 nodes (210164 index entries) using index jcr:primaryType
with filter Filter(query=SELECT * FROM [cq:PageContent] AS b WHERE
ISDESCENDANTNODE(b, [/content/guides]) AND ([cq:tags] LIKE
'location:europe%' OR [cq:tags] LIKE 'type:waterfalls%') ORDER BY
[cq:lastModified], path=/content/guides//*, property=[cq:tags=[is not
null]])

Instead, I created a lucene index that indexes the cq:tags (/w full text)
and cq:lastModified (/w ordered support) property.

e.g. SELECT [jcr:path] FROM [cq:PageContent] AS b WHERE ISDESCENDANTNODE(b,
[/content/guides]) AND (CONTAINS([cq:tags], 'location:europe') OR
CONTAINS([cq:tags], 'type:waterfalls')) ORDER BY [cq:lastModified]

That seems to be much faster than using a property index and should solve
most of the issues that we might have (hopefully avoiding creating a new
index).

Is there any support with the lucene index to use something like STARTSWITH
rather CONTAINS?

The maxClauseCount configuration parameter introduced the soft limit of 1024
which is part of Jackrabbit 2.
We have been attempting to move to oak however our progress has been slow
due to repository inconsistencies.
I realise this value is configurable however constantly increasing it
doesn't sound the right thing to do.

Thanks,
Rachna



--
View this message in context: http://jackrabbit.510166.n4.nabble.com/Custom-index-type-tp4665031p4665121.html
Sent from the Jackrabbit - Users mailing list archive at Nabble.com.

Re: Custom index type

[Thomas Mueller <mu...@adobe.com>]

Hi,

> We've hit the 1024 limit on parameters for the query api.

Do you know which limit this is? I'm not aware of such a limit in Oak.

Jut to ensure we talk about the same issue, I made an example, could you
check if this is correct?

Assuming you have 3 nodes:
/a @tag = 'color'
/b @tag = 'color/red'
/c @tag = 'color/blue'

Now you expect the following queries to return:

where @tag = 'color'
where @tag in ('color')
-> /a, /b, /c

where @tag = 'color/red'
where @tag in ('color', 'color/red')
-> /a, /b

where @tag = 'color/blue'
where @tag in ('color', 'color/blue')
-> /a, /c

Is this what you are looking for?

Regards,
Thomas






On 13/10/16 18:02, "rachna" <ra...@telegraph.co.uk> wrote:

>Hi Clay & Thomas,
>
>Thanks for your replies.
>
>We have previously developed a feature that uses the tag management system
>inside of AEM. The current business logic performs a search against the
>repository using the tags associated with the content page. As part of
>this
>business logic, the decendants of the tags (and decendants of these tags
>until the bottom of the tree is reached) are also added to the query. This
>creates a large query that isn't very efficient. We've hit the 1024 limit
>on
>parameters for the query api.
>
>Our custom indexer analyses the decendants of the tags when the reindex
>operation is triggered so we can precompute this and store it efficently
>in
>the index.
>(We store the page path against the decendant tags in the index)
>
>Ideally we would want to query our index directly (e.g. native query) to
>return the results. From looking at the documentation for the lucene
>index,
>I can see this is possible.
>The documentation for the property index does not have the same query
>listed.
>
>Is it possible to run a native query against a custom indexer?
>Otherwise, can we programmtically access a new NodeState?
>
>We've hit the 1024 limit on parameters for the query api.  Also we've hit
>the 1024 limit on parameters for the query api.
>
>
>
>
>
>--
>View this message in context:
>http://jackrabbit.510166.n4.nabble.com/Custom-index-type-tp4665031p4665112
>.html
>Sent from the Jackrabbit - Users mailing list archive at Nabble.com.

Re: Custom index type

[rachna <ra...@telegraph.co.uk>]

Hi Clay & Thomas,

Thanks for your replies.

We have previously developed a feature that uses the tag management system
inside of AEM. The current business logic performs a search against the
repository using the tags associated with the content page. As part of this
business logic, the decendants of the tags (and decendants of these tags
until the bottom of the tree is reached) are also added to the query. This
creates a large query that isn't very efficient. We've hit the 1024 limit on
parameters for the query api.

Our custom indexer analyses the decendants of the tags when the reindex
operation is triggered so we can precompute this and store it efficently in
the index.
(We store the page path against the decendant tags in the index)

Ideally we would want to query our index directly (e.g. native query) to
return the results. From looking at the documentation for the lucene index,
I can see this is possible.
The documentation for the property index does not have the same query
listed.

Is it possible to run a native query against a custom indexer?
Otherwise, can we programmtically access a new NodeState?

We've hit the 1024 limit on parameters for the query api.  Also we've hit
the 1024 limit on parameters for the query api.





--
View this message in context: http://jackrabbit.510166.n4.nabble.com/Custom-index-type-tp4665031p4665112.html
Sent from the Jackrabbit - Users mailing list archive at Nabble.com.

Re: Custom index type

[Thomas Mueller <mu...@adobe.com>]

Hi,

Please describe why you want to write your own index type, and can't use
the existing ones (property index, Lucene index, Solr).

Regards,
Thomas



On 11/10/16 15:49, "rachna" <ra...@telegraph.co.uk> wrote:

>Hi, 
>
>Just like property index type, I have got my own index type called custom,
>which is loaded with data according to requirement.  So query is not
>picking
>up this index as it's custom type, I want to write my own API function to
>read this index and return data, I am not getting any exception but
>neither
>data.
>When I use node.getNodes(), it doesn't return me any nodes from :index
>node
>onwards,
>When I use oak-run tool to open this index, I can see it's populated with
>the required data from :index onwards.
>
>
>
>--
>View this message in context:
>http://jackrabbit.510166.n4.nabble.com/Custom-index-type-tp4665031p4665105
>.html
>Sent from the Jackrabbit - Users mailing list archive at Nabble.com.

Re: Custom index type

[Clay Ferguson <wc...@gmail.com>]

​Again it seems to me like you are trying to read from the index directly.
You should never do that. Just pretend like the index doesn't even exist,
and write normal content nodes onto the normal non-index part of the tree
and then index will STILL be getting used automatically. It works just like
an RDBMS index. You create the index, and then forget about it. All your
searches will use the appropriate indexes if they can but you should never
try to look at the index itself. Just write code as if the index did not
exist. ​When you add nodes to the content tree, they will be "indexed"
automatically. If you ARE trying to store DATA in an index, then do not do
that. That is wrong. You should be storing data as nodes on the tree. The
index will take care of itself automatically.

Best regards,
Clay Ferguson
wclayf@gmail.com


On Tue, Oct 11, 2016 at 8:49 AM, rachna <ra...@telegraph.co.uk>
wrote:

> Hi,
>
> Just like property index type, I have got my own index type called custom,
> which is loaded with data according to requirement.  So query is not
> picking
> up this index as it's custom type, I want to write my own API function to
> read this index and return data, I am not getting any exception but neither
> data.
> When I use node.getNodes(), it doesn't return me any nodes from :index node
> onwards,
> When I use oak-run tool to open this index, I can see it's populated with
> the required data from :index onwards.
>
>
>
> --
> View this message in context: http://jackrabbit.510166.n4.
> nabble.com/Custom-index-type-tp4665031p4665105.html
> Sent from the Jackrabbit - Users mailing list archive at Nabble.com.
>

Re: Custom index type

[rachna <ra...@telegraph.co.uk>]

Hi, 

Just like property index type, I have got my own index type called custom,
which is loaded with data according to requirement.  So query is not picking
up this index as it's custom type, I want to write my own API function to
read this index and return data, I am not getting any exception but neither
data.
When I use node.getNodes(), it doesn't return me any nodes from :index node
onwards,
When I use oak-run tool to open this index, I can see it's populated with
the required data from :index onwards.



--
View this message in context: http://jackrabbit.510166.n4.nabble.com/Custom-index-type-tp4665031p4665105.html
Sent from the Jackrabbit - Users mailing list archive at Nabble.com.

Re: Custom index type

[Clay Ferguson <wc...@gmail.com>]

I'm not sure what you mean by "read the index". The repository consists of
nodes. The indexes just aid in searching, but you don't need to read the
index itself directly AFAIK. So it's all about the ownership on the node
you are trying to read. You must be using a session that is logged into by
a user who had read (and/or write) privileges on the node you are trying to
read. Are you getting an exception? Send us the stack trace if so. If not,
what method specifically are you calling that's giving unexpected results?

Best regards,
Clay Ferguson
wclayf@gmail.com


On Wed, Oct 5, 2016 at 3:11 AM, rachna <ra...@telegraph.co.uk>
wrote:

> Hi,  we have built a custom index type and created jackrabbit oak index
> using
> this type, index gets populated with the correct data but now when I try to
> read it using standard jcr nodes, it's not allowing me to read hidden
> nodes.
> I think I need to use NodeState and NodeBuilder classes to read this index,
> but struggling to get root object on oak:index node.  Does anyone know how
> can I do that?
>
> Thanks
>
>
>
> --
> View this message in context: http://jackrabbit.510166.n4.
> nabble.com/Custom-index-type-tp4665031.html
> Sent from the Jackrabbit - Users mailing list archive at Nabble.com.
>