You are viewing a plain text version of this content. The canonical link for it is here.
Posted to oak-dev@jackrabbit.apache.org by Sebastien Berthezene <sb...@gmail.com> on 2015/09/22 12:51:26 UTC
Jackrabbit OAK property index never used ?
I am trying to use property index with Jackrabbit but when i have many
thousands of nodes it seems that transversal mode is always chosen for
query execution.
For example, i have 10 000 nodes of type test:mytype under a single node
/mystore. Into each of these nodes i have a property test:myprop with 3
different possible values (nearly 3000 nodes for each value).
When i run the following query
select [jcr:uuid] from [test:mytype] where [test:myprop]='MyValue'
the query engine processor always use the transversing mode and do not use
the index i have created for test:myprop.
I tried to debug the code directly, i clearly see the query engine trying
to use the property index i have defined but do not use it because cost of
transversing cursor index seems to be always "100" and my property index
contains nearly "3000" nodes for each possible value. Query engine consider
that using transversal cursor will be more efficient, even if engine will
need to transverse 10000 nodes !
Did someone already faced similar problem ?
Regards
Re: Jackrabbit OAK property index never used ?
Posted by Sebastien Berthezene <sb...@gmail.com>.
Thanks Thomas but how you do that ? Here is what i have done :
session.getNodeByIdentifier("/oak:index/counter").setProperty(IndexConstants.REINDEX_PROPERTY_NAME,
true);
session.save();
It did not change anything, the execution plan still use traverse :
o.apache.jackrabbit.oak.query.QueryImpl : cost using filter
Filter(query=select....', path=*, property=[my:status=[Draft]])
...
2015-09-25 11:12:12 [DEBUG] - o.apache.jackrabbit.oak.query.QueryImpl :
cost for reference is Infinity
...
2015-09-25 11:12:12 [DEBUG] - o.a.j.o.p.index.property.PropertyIndex :
property cost for StatusIndex2 is 12702.0
o.apache.jackrabbit.oak.query.QueryImpl : cost for property is 12702.0
...
2015-09-25 11:12:12 [DEBUG] - o.apache.jackrabbit.oak.query.QueryImpl :
cost for ordered is Infinity
...
2015-09-25 11:12:12 [DEBUG] - o.apache.jackrabbit.oak.query.QueryImpl :
cost for nodeType is 25504.0
...
2015-09-25 11:12:12 [DEBUG] - o.apache.jackrabbit.oak.query.QueryImpl :
cost for traverse is 2000.0
...
2015-09-25 11:12:12 [DEBUG] - o.apache.jackrabbit.oak.query.QueryImpl :
query execute ....
2015-09-25 11:12:12 [DEBUG] - o.apache.jackrabbit.oak.query.QueryImpl :
query plan [my:document] as [doc] /* traverse "*" where [doc].[my:status] =
'Draft' */
Whatever the query i run, cost for traverse is always 2000 but i have
nearly 30K nodes.
2015-09-24 10:28 GMT+02:00 Thomas Mueller <mu...@adobe.com>:
> Hi,
>
> OAK-2852 is fixed now, this will be in the next release.
>
>
> If you don't want to wait for that, I suggest to reindex the counter
> index, that is, set "reindex" on that index to "true" (and save).
>
> Regards,
> Thomas
>
> On 23/09/15 16:55, "Sebastien Berthezene" <sb...@gmail.com> wrote:
>
> >Thanks for your help but I have the node "/oak:index/counter". Dump of
> >node
> >get with "session.getNodeByIdentifier("/oak:index/counter")" is the
> >following :
> >type = counter
> >async = async
> >jcr:primaryType = oak:QueryIndexDefinition
> >
> >From what i understand it seems that node count is not set (is it supposed
> >to be set somewhere ?) and so traversal index use a default hardcoded
> >value
> >corresponding to ApproximateCounter.COUNT_RESOLUTION * 20 ( = 2000) always
> >lower than node count managed by my property index. How could i fix this
> >node count supposed to be stored into a ":count" property ?
> >
> >Regards
> >
> >2015-09-23 12:15 GMT+02:00 Thomas Mueller <mu...@adobe.com>:
> >
> >> Hi,
> >>
> >> I think you are hitting OAK-2852.
> >>
> >> Regards,
> >> Thomas
> >>
> >>
> >>
> >> On 23/09/15 11:42, "Thomas Mueller" <mu...@adobe.com> wrote:
> >>
> >> >Hi,
> >> >
> >> >Do you have a node called /oak:index/counter ? Out of the-box, it
> >>should
> >> >be there (with a recent version of Oak). That is the approximate
> >>counter
> >> >index that is used to estimate how many nodes to traverse. As a
> >> >workaround, you probably have to re-index that one manually. I wonder
> >>why
> >> >that index is not updated in your case, it is a regular asynchronous
> >> >index.
> >> >
> >> >Regards,
> >> >Thomas
> >> >
> >> >
> >> >On 22/09/15 17:04, "Sebastien Berthezene" <sb...@gmail.com>
> >>wrote:
> >> >
> >> >>Because 2000 (not corresponding to reality) is lower than 12702 from
> >>my
> >> >>property index, the traversal mode is used.
> >> >
> >>
> >>
>
>
Re: Jackrabbit OAK property index never used ?
Posted by Thomas Mueller <mu...@adobe.com>.
Hi,
OAK-2852 is fixed now, this will be in the next release.
If you don't want to wait for that, I suggest to reindex the counter
index, that is, set "reindex" on that index to "true" (and save).
Regards,
Thomas
On 23/09/15 16:55, "Sebastien Berthezene" <sb...@gmail.com> wrote:
>Thanks for your help but I have the node "/oak:index/counter". Dump of
>node
>get with "session.getNodeByIdentifier("/oak:index/counter")" is the
>following :
>type = counter
>async = async
>jcr:primaryType = oak:QueryIndexDefinition
>
>From what i understand it seems that node count is not set (is it supposed
>to be set somewhere ?) and so traversal index use a default hardcoded
>value
>corresponding to ApproximateCounter.COUNT_RESOLUTION * 20 ( = 2000) always
>lower than node count managed by my property index. How could i fix this
>node count supposed to be stored into a ":count" property ?
>
>Regards
>
>2015-09-23 12:15 GMT+02:00 Thomas Mueller <mu...@adobe.com>:
>
>> Hi,
>>
>> I think you are hitting OAK-2852.
>>
>> Regards,
>> Thomas
>>
>>
>>
>> On 23/09/15 11:42, "Thomas Mueller" <mu...@adobe.com> wrote:
>>
>> >Hi,
>> >
>> >Do you have a node called /oak:index/counter ? Out of the-box, it
>>should
>> >be there (with a recent version of Oak). That is the approximate
>>counter
>> >index that is used to estimate how many nodes to traverse. As a
>> >workaround, you probably have to re-index that one manually. I wonder
>>why
>> >that index is not updated in your case, it is a regular asynchronous
>> >index.
>> >
>> >Regards,
>> >Thomas
>> >
>> >
>> >On 22/09/15 17:04, "Sebastien Berthezene" <sb...@gmail.com>
>>wrote:
>> >
>> >>Because 2000 (not corresponding to reality) is lower than 12702 from
>>my
>> >>property index, the traversal mode is used.
>> >
>>
>>
Re: Jackrabbit OAK property index never used ?
Posted by Sebastien Berthezene <sb...@gmail.com>.
Thanks for your help but I have the node "/oak:index/counter". Dump of node
get with "session.getNodeByIdentifier("/oak:index/counter")" is the
following :
type = counter
async = async
jcr:primaryType = oak:QueryIndexDefinition
>From what i understand it seems that node count is not set (is it supposed
to be set somewhere ?) and so traversal index use a default hardcoded value
corresponding to ApproximateCounter.COUNT_RESOLUTION * 20 ( = 2000) always
lower than node count managed by my property index. How could i fix this
node count supposed to be stored into a ":count" property ?
Regards
2015-09-23 12:15 GMT+02:00 Thomas Mueller <mu...@adobe.com>:
> Hi,
>
> I think you are hitting OAK-2852.
>
> Regards,
> Thomas
>
>
>
> On 23/09/15 11:42, "Thomas Mueller" <mu...@adobe.com> wrote:
>
> >Hi,
> >
> >Do you have a node called /oak:index/counter ? Out of the-box, it should
> >be there (with a recent version of Oak). That is the approximate counter
> >index that is used to estimate how many nodes to traverse. As a
> >workaround, you probably have to re-index that one manually. I wonder why
> >that index is not updated in your case, it is a regular asynchronous
> >index.
> >
> >Regards,
> >Thomas
> >
> >
> >On 22/09/15 17:04, "Sebastien Berthezene" <sb...@gmail.com> wrote:
> >
> >>Because 2000 (not corresponding to reality) is lower than 12702 from my
> >>property index, the traversal mode is used.
> >
>
>
Re: Jackrabbit OAK property index never used ?
Posted by Thomas Mueller <mu...@adobe.com>.
Hi,
I think you are hitting OAK-2852.
Regards,
Thomas
On 23/09/15 11:42, "Thomas Mueller" <mu...@adobe.com> wrote:
>Hi,
>
>Do you have a node called /oak:index/counter ? Out of the-box, it should
>be there (with a recent version of Oak). That is the approximate counter
>index that is used to estimate how many nodes to traverse. As a
>workaround, you probably have to re-index that one manually. I wonder why
>that index is not updated in your case, it is a regular asynchronous
>index.
>
>Regards,
>Thomas
>
>
>On 22/09/15 17:04, "Sebastien Berthezene" <sb...@gmail.com> wrote:
>
>>Because 2000 (not corresponding to reality) is lower than 12702 from my
>>property index, the traversal mode is used.
>
Re: Jackrabbit OAK property index never used ?
Posted by Thomas Mueller <mu...@adobe.com>.
Hi,
Do you have a node called /oak:index/counter ? Out of the-box, it should
be there (with a recent version of Oak). That is the approximate counter
index that is used to estimate how many nodes to traverse. As a
workaround, you probably have to re-index that one manually. I wonder why
that index is not updated in your case, it is a regular asynchronous index.
Regards,
Thomas
On 22/09/15 17:04, "Sebastien Berthezene" <sb...@gmail.com> wrote:
>Because 2000 (not corresponding to reality) is lower than 12702 from my
>property index, the traversal mode is used.
Re: Jackrabbit OAK property index never used ?
Posted by Sebastien Berthezene <sb...@gmail.com>.
I use Oak 1.3.2 version. I have done more test with other data batch. When
i debug the code and try to understand query plan, it seems to engine is
not really aware of real node count and decide to use traversal mode
instead of my property index.
EstimatedChildNodeCounts for "/,2" : <Nothing, it is empty>
EstimatedChildNodeCounts for "/oak:index", 3 :
/oak:index/StatusIndex/:index: 25500,
/oak:index/StatusIndex/:index/Approved: 12700,
/oak:index/StatusIndex/:index/Draft: 12700,
/oak:index/StatusIndex/:index/Effective: 6300,
/oak:index/nodetype/:index: 102300,
/oak:index/nodetype/:index/my%3Adocument: 25500,
/oak:index/nodetype/:index/my%3Aroom: 100,
/oak:index/nodetype/:index/mix%3Areferenceable: 25500,
/oak:index/nodetype/:index/mix%3Ashareable: 12700,
/oak:index/nodetype/:index/nt%3ApropertyDefinition: 100,
/oak:index/nodetype/:index/rep%3APropertyDefinition: 300,
/oak:index/nodetype/:index/rep%3APropertyDefinitions: 100,
/oak:index/uuid/:index: 25500
My query is :
select [jcr:uuid] from [my:document] where [my:status]='Draft'
It is supposed to use the index "StatusIndex" but instead of that uses
traversal mode.
When i debug the code :
- into the ApproximateCounter.getCountSync(index) from
ContentMirrorStoreStrategy.count(), returned count is 255000 because it
seems to go into the case where "entryCount" is not set. Not sure to
understand why. The problem could be here.
- Cost to use the property index is evaluated to 12702.
- The QueryImpl.getBestSelectorExecutionPlan then go to traversal index
evaluation and traversal.getCost(filter, rootState) return 2000 because
NodeCounter.getEstimatedNodeCount(rootState, testPath, true) returns 2000
because ":count" property is not found and then
"ApproximateCounter.COUNT_RESOLUTION * 20" is used as fallback. Why i do
not have the ":count" property ? Where does it come from ?
- Because 2000 (not corresponding to reality) is lower than 12702 from my
property index, the traversal mode is used.
I enter into code part // for this index, property "entryCount" is not set
Basically it seems that some metadata supposed to keep the count of indexed
values or node count are incorrect or empty. I do not understand why. Is
there any way to fix them ?
2015-09-22 14:19 GMT+02:00 Thomas Mueller <mu...@adobe.com>:
> Hi,
>
> Which version of Oak do you use?
>
> Could you get the estimated node count for the root node, and for this
> index? To get that, for example use the NodeCounter JMX bean
> (NodeCounterMBean), getEstimatedChildNodeCounts("/", 2) and
> getEstimatedChildNodeCounts("/oak:index", 3).
>
> Regards,
> Thomas
>
>
> On 22/09/15 12:51, "Sebastien Berthezene" <sb...@gmail.com> wrote:
>
> >I am trying to use property index with Jackrabbit but when i have many
> >thousands of nodes it seems that transversal mode is always chosen for
> >query execution.
> >
> >For example, i have 10 000 nodes of type test:mytype under a single node
> >/mystore. Into each of these nodes i have a property test:myprop with 3
> >different possible values (nearly 3000 nodes for each value).
> >
> >When i run the following query
> >
> >select [jcr:uuid] from [test:mytype] where [test:myprop]='MyValue'
> >
> >the query engine processor always use the transversing mode and do not use
> >the index i have created for test:myprop.
> >
> >I tried to debug the code directly, i clearly see the query engine trying
> >to use the property index i have defined but do not use it because cost of
> >transversing cursor index seems to be always "100" and my property index
> >contains nearly "3000" nodes for each possible value. Query engine
> >consider
> >that using transversal cursor will be more efficient, even if engine will
> >need to transverse 10000 nodes !
> >
> >Did someone already faced similar problem ?
> >
> >Regards
>
>
Re: Jackrabbit OAK property index never used ?
Posted by Thomas Mueller <mu...@adobe.com>.
Hi,
Which version of Oak do you use?
Could you get the estimated node count for the root node, and for this
index? To get that, for example use the NodeCounter JMX bean
(NodeCounterMBean), getEstimatedChildNodeCounts("/", 2) and
getEstimatedChildNodeCounts("/oak:index", 3).
Regards,
Thomas
On 22/09/15 12:51, "Sebastien Berthezene" <sb...@gmail.com> wrote:
>I am trying to use property index with Jackrabbit but when i have many
>thousands of nodes it seems that transversal mode is always chosen for
>query execution.
>
>For example, i have 10 000 nodes of type test:mytype under a single node
>/mystore. Into each of these nodes i have a property test:myprop with 3
>different possible values (nearly 3000 nodes for each value).
>
>When i run the following query
>
>select [jcr:uuid] from [test:mytype] where [test:myprop]='MyValue'
>
>the query engine processor always use the transversing mode and do not use
>the index i have created for test:myprop.
>
>I tried to debug the code directly, i clearly see the query engine trying
>to use the property index i have defined but do not use it because cost of
>transversing cursor index seems to be always "100" and my property index
>contains nearly "3000" nodes for each possible value. Query engine
>consider
>that using transversal cursor will be more efficient, even if engine will
>need to transverse 10000 nodes !
>
>Did someone already faced similar problem ?
>
>Regards