You are viewing a plain text version of this content. The canonical link for it is here.
Posted to oak-dev@jackrabbit.apache.org by Alvaro Cabrerizo <to...@gmail.com> on 2017/05/16 08:23:57 UTC
index selection when 0 or 1 property clause (or sort clause) matches
Hello,
I've been checking the code of the IndexPlanner (apache OAK 1.4.1) and I
was surprised because the costPerEntryFactor remains 1 in both cases:
- when no property indexed or sorted match any property clause or sort
clause from the query
- when only an indexed or sorted property matches a property clause or
sort clause from the query
Although this piece of code avoids a division by zero (see
org.apache.jackrabbit.oak.plugins.index.lucene.IndexPlanner lines 201-203)
if (costPerEntryFactor == 0){
costPerEntryFactor = 1;
}
It also avoids the boosting of indexes that match 1 query clause (or in
other words, it doesn't penalize indexes that don't match any clause). I'm
thinking about opening an issue. Although it is for the long-tern, actually
I would like to know which index is selected in case that more than one had
the same cost.
Regards.
Re: index selection when 0 or 1 property clause (or sort clause) matches
Posted by Alvaro Cabrerizo <to...@gmail.com>.
Good news is always welcome!
Many thanks Chetan.
On Fri, Jun 9, 2017 at 9:44 AM, Chetan Mehrotra <ch...@gmail.com>
wrote:
> On Wed, May 17, 2017 at 1:40 PM, Alvaro Cabrerizo <to...@gmail.com>
> wrote:
> > The main issue of delegating in entryCount, is that if the index contains
> > more than 1000 docs and the query does not contain fulltext clauses the
> > index planner will use the number *1000 *as the entryCount, ovewriting
> the
> > actual size of the index [Math.min(definition.getEntryCount(),
> getReader().
> > *numDocs())*].
>
> A very late reply here but I have opened OAK-6333 to remove this
> artificial limit of 1000 in cost estimates. Once we have consensus on
> the approach then I would fix and backport it to older branched
>
> Chetan Mehrotra
>
Re: index selection when 0 or 1 property clause (or sort clause) matches
Posted by Chetan Mehrotra <ch...@gmail.com>.
On Wed, May 17, 2017 at 1:40 PM, Alvaro Cabrerizo <to...@gmail.com> wrote:
> The main issue of delegating in entryCount, is that if the index contains
> more than 1000 docs and the query does not contain fulltext clauses the
> index planner will use the number *1000 *as the entryCount, ovewriting the
> actual size of the index [Math.min(definition.getEntryCount(), getReader().
> *numDocs())*].
A very late reply here but I have opened OAK-6333 to remove this
artificial limit of 1000 in cost estimates. Once we have consensus on
the approach then I would fix and backport it to older branched
Chetan Mehrotra
Re: index selection when 0 or 1 property clause (or sort clause) matches
Posted by Alvaro Cabrerizo <to...@gmail.com>.
Hello,
The main issue of delegating in entryCount, is that if the index contains
more than 1000 docs and the query does not contain fulltext clauses the
index planner will use the number *1000 *as the entryCount, ovewriting the
actual size of the index [Math.min(definition.getEntryCount(), getReader().
*numDocs())*]. Nowadays the current indexes involved in the collision in
our system, contain serveral million and a several thousand of documents
respectively. The cost, for both indexes performing most common
queries is *1001,
*yes* 1001* = costPerExecution + (costPerEntry * entryCount) = 1 +
(1*1000). Thereby we can't predict which index will be used.
Regards.
On Wed, May 17, 2017 at 9:40 AM, Chetan Mehrotra <ch...@gmail.com>
wrote:
> > Using the entryCount was our first option, but we decided to modify
> > costPerEntry instead. Basically these are the reasons:
>
> What I meant was not the use of "entryCount" property but just that
> the sub index /nodeA/nodeB/includedC being a subset of /nodeA/nodeB/
> it would have lower value for numDocs and hence any query having path
> restriction of /nodeA/nodeB/includedC would get to use that index (by
> virtue of numDocs being less than parent). So you would not need to do
> any tweaking
>
> > response we expect. It seems that in OAK, indexes are used to speed up
> > queries, but not as a complete information entity that must answer every
> > query clause. Well, I'm still figuring out the big picture of querying
>
> Yes the index may only be partially covering the query. So index job
> is to provide a super set of possible result paths and Query Engine
> would then evaluate the restriction for each path and at the
> repository state per current JCR Session (except fulltext constraints)
>
> Chetan Mehrotra
>
Re: index selection when 0 or 1 property clause (or sort clause) matches
Posted by Chetan Mehrotra <ch...@gmail.com>.
> Using the entryCount was our first option, but we decided to modify
> costPerEntry instead. Basically these are the reasons:
What I meant was not the use of "entryCount" property but just that
the sub index /nodeA/nodeB/includedC being a subset of /nodeA/nodeB/
it would have lower value for numDocs and hence any query having path
restriction of /nodeA/nodeB/includedC would get to use that index (by
virtue of numDocs being less than parent). So you would not need to do
any tweaking
> response we expect. It seems that in OAK, indexes are used to speed up
> queries, but not as a complete information entity that must answer every
> query clause. Well, I'm still figuring out the big picture of querying
Yes the index may only be partially covering the query. So index job
is to provide a super set of possible result paths and Query Engine
would then evaluate the restriction for each path and at the
repository state per current JCR Session (except fulltext constraints)
Chetan Mehrotra
Re: index selection when 0 or 1 property clause (or sort clause) matches
Posted by Alvaro Cabrerizo <to...@gmail.com>.
Hello,
Using the entryCount was our first option, but we decided to modify
costPerEntry instead. Basically these are the reasons:
- It can only be used to reduce the cost. It can't be used to increase
the index cost as the index-planner selects
Math.min(definition.getEntryCount(),
getReader().*numDocs()); *Thus it useless to have an entryCount value
bigger than the index size.
- The entry count for the rest of indexes competing will be, in general
1000, as it will not be defined and all of our indexes are contain more
that 100 documents. Remember that if the value, entryCount, is not defined
then *definition*.*getEntryCount() *will use th*e **DEFAULT_ENTRY_COUNT*
which is 1000 and the planner will take
Math.min(definition.getEntryCount(),
getReader().*numDocs())*. In other words the entryCount should be
defined smaller than 1000 to make it "very attractive".
Using costPerEntry allows us to increase the cost, thus avoiding the usage
of this index by every user. In this way, only selected applications,
performing Lucene native queries will use it.
In this apparent maze that is the planner it is still not clear for me:
- which index is selected in case of cost tie.
I guess that part of the misconception I have about how the queries work in
OAK, comes from my experience working with Solr and Lucene building search
engines. Basically, when we develop a search component using Solr or
Lucene, all the necessary data (all the documents and fields) is stored
within the index (or indexes). Thus every query is routed deliberately to
the index we want to use and that index is enough to get the complete
response we expect. It seems that in OAK, indexes are used to speed up
queries, but not as a complete information entity that must answer every
query clause. Well, I'm still figuring out the big picture of querying
process in OAK.
Regards.
On Tue, May 16, 2017 at 6:23 PM, Chetan Mehrotra <ch...@gmail.com>
wrote:
> Having same asset indexed twice would add overhead in terms of async
> indexing speed and space consumption by index. So if possible avoid
> such a setup
>
> > We could assume that we always add a path restriction, but I'm not sure
> how
> > index movement can help. I mean, both indexes contains documents under
> > /nodeA/nodeB/nodeC so any query under this path is satisfied by both
>
> One way it would help is that entryCount for second index would be
> smaller compared to first and hence queries with path restriction for
> second index would have lesser cost
>
> Chetan Mehrotra
>
Re: index selection when 0 or 1 property clause (or sort clause) matches
Posted by Chetan Mehrotra <ch...@gmail.com>.
Having same asset indexed twice would add overhead in terms of async
indexing speed and space consumption by index. So if possible avoid
such a setup
> We could assume that we always add a path restriction, but I'm not sure how
> index movement can help. I mean, both indexes contains documents under
> /nodeA/nodeB/nodeC so any query under this path is satisfied by both
One way it would help is that entryCount for second index would be
smaller compared to first and hence queries with path restriction for
second index would have lesser cost
Chetan Mehrotra
Re: index selection when 0 or 1 property clause (or sort clause) matches
Posted by Alvaro Cabrerizo <to...@gmail.com>.
Hello,
I'm not sure about the text extraction, as other team is in charge of it.
But the repository is quite big.
We could assume that we always add a path restriction, but I'm not sure how
index movement can help. I mean, both indexes contains documents under
/nodeA/nodeB/nodeC so any query under this path is satisfied by both
indexes.
Regards.
On Tue, May 16, 2017 at 5:18 PM, Chetan Mehrotra <ch...@gmail.com>
wrote:
> > - Rebuilding Assets index takes several days
>
> Is the time spent in text extraction?
>
> Would the code always specify path restriction in the queries for
> my:Asset? If yes then you can just move the index definition under
> respective paths
>
> Would that be an option
> Chetan Mehrotra
>
>
> On Tue, May 16, 2017 at 8:20 PM, Alvaro Cabrerizo <to...@gmail.com>
> wrote:
> > Hello,
> >
> > Yes, there are some reasons:
> >
> > - One team is working with all the assets under /nodeA/nodeB
> > - Other team only works with an small subset, only assets under
> > /nodeA/node/nodeC
> > - Both teams have different searching requirements
> > - Rebuilding Assets index takes several days
> > - Rebuilding deeperAsset takes a couple of hours. Thus merging both
> will
> > penalize the size and modification agility of deeperAsset
> >
> > One option we are evaluating is to cheat the deepAsset cost (using
> > costPerEntry property) and make that team work with lucene native queries
> > (any recommendation is welcome). Anyway, it is not clear which index is
> > selected in case of tie.
> >
> > Regards.
> >
> >
> > On Tue, May 16, 2017 at 4:36 PM, Chetan Mehrotra <
> chetan.mehrotra@gmail.com>
> > wrote:
> >
> >> Any reason for having separate definitions for same nodetype?
> >> Chetan Mehrotra
> >>
> >>
> >> On Tue, May 16, 2017 at 7:52 PM, Alvaro Cabrerizo <to...@gmail.com>
> >> wrote:
> >> > Hello,
> >> >
> >> > Actually, it is OAK-5449. Sorry, I hadn't seen it.
> >> >
> >> > On the other hand, having these two definitions under oak:index (just
> a
> >> > sketch):
> >> >
> >> >
> >> > - Asset
> >> > - evaluatePathRestrictions="true"
> >> > - type="lucene"
> >> > - includedPath="/nodeA/nodeB"
> >> > - indexRules
> >> > - my:Asset
> >> > - properties
> >> > - name="my:title"
> >> > - deeperAsset
> >> > - evaluatePathRestrictions="true"
> >> > - type="lucene"
> >> > - includedPath="/nodeA/nodeB/includedC"
> >> > - ndexRules
> >> > - my:Asset
> >> > - properties
> >> > - name="my:description"
> >> >
> >> > Made the system to return a cost of 1001 (for both indexes) when
> >> performing
> >> > these kind of queries:
> >> >
> >> > - SELECT * FROM [my:Asset] AS s WHERE ISDESCENDANTNODE(s,'/nodeA/
> >> nodeB')
> >> > - SELECT * FROM [my:Asset] AS s WHERE ISDESCENDANTNODE(s,'/nodeA/
> >> nodeB')
> >> > AND s.[my:title]='title'
> >> > - SELECT * FROM [my:Asset] AS s WHERE ISDESCENDANTNODE(s,'/nodeA/
> >> nodeB')
> >> > AND s.[my:description]='description'
> >> >
> >> > Once cost is assigned (equal for both indexes), it is not clear which
> >> index
> >> > will be selected.
> >> >
> >> > Regards.
> >> >
> >> > On Tue, May 16, 2017 at 3:50 PM, Chetan Mehrotra <
> >> chetan.mehrotra@gmail.com>
> >> > wrote:
> >> >
> >> >> This looks similar to OAK-5449 (not yet fixed). Can you give a sample
> >> >> index definition there and some usecase details which is leading to
> >> >> ambiguity in index selection.
> >> >>
> >> >> In general index selection should not have multiple competing index
> >> >> definitions hence interested in knowing setup details
> >> >> Chetan Mehrotra
> >> >>
> >> >>
> >> >> On Tue, May 16, 2017 at 1:53 PM, Alvaro Cabrerizo <
> toporniz@gmail.com>
> >> >> wrote:
> >> >> > Hello,
> >> >> >
> >> >> > I've been checking the code of the IndexPlanner (apache OAK 1.4.1)
> >> and I
> >> >> > was surprised because the costPerEntryFactor remains 1 in both
> cases:
> >> >> >
> >> >> > - when no property indexed or sorted match any property clause
> or
> >> sort
> >> >> > clause from the query
> >> >> > - when only an indexed or sorted property matches a property
> >> clause or
> >> >> > sort clause from the query
> >> >> >
> >> >> > Although this piece of code avoids a division by zero (see
> >> >> > org.apache.jackrabbit.oak.plugins.index.lucene.IndexPlanner lines
> >> >> 201-203)
> >> >> >
> >> >> > if (costPerEntryFactor == 0){
> >> >> > costPerEntryFactor = 1;
> >> >> > }
> >> >> >
> >> >> > It also avoids the boosting of indexes that match 1 query clause
> (or
> >> in
> >> >> > other words, it doesn't penalize indexes that don't match any
> clause).
> >> >> I'm
> >> >> > thinking about opening an issue. Although it is for the long-tern,
> >> >> actually
> >> >> > I would like to know which index is selected in case that more than
> >> one
> >> >> had
> >> >> > the same cost.
> >> >> >
> >> >> > Regards.
> >> >>
> >>
>
Re: index selection when 0 or 1 property clause (or sort clause) matches
Posted by Chetan Mehrotra <ch...@gmail.com>.
> - Rebuilding Assets index takes several days
Is the time spent in text extraction?
Would the code always specify path restriction in the queries for
my:Asset? If yes then you can just move the index definition under
respective paths
Would that be an option
Chetan Mehrotra
On Tue, May 16, 2017 at 8:20 PM, Alvaro Cabrerizo <to...@gmail.com> wrote:
> Hello,
>
> Yes, there are some reasons:
>
> - One team is working with all the assets under /nodeA/nodeB
> - Other team only works with an small subset, only assets under
> /nodeA/node/nodeC
> - Both teams have different searching requirements
> - Rebuilding Assets index takes several days
> - Rebuilding deeperAsset takes a couple of hours. Thus merging both will
> penalize the size and modification agility of deeperAsset
>
> One option we are evaluating is to cheat the deepAsset cost (using
> costPerEntry property) and make that team work with lucene native queries
> (any recommendation is welcome). Anyway, it is not clear which index is
> selected in case of tie.
>
> Regards.
>
>
> On Tue, May 16, 2017 at 4:36 PM, Chetan Mehrotra <ch...@gmail.com>
> wrote:
>
>> Any reason for having separate definitions for same nodetype?
>> Chetan Mehrotra
>>
>>
>> On Tue, May 16, 2017 at 7:52 PM, Alvaro Cabrerizo <to...@gmail.com>
>> wrote:
>> > Hello,
>> >
>> > Actually, it is OAK-5449. Sorry, I hadn't seen it.
>> >
>> > On the other hand, having these two definitions under oak:index (just a
>> > sketch):
>> >
>> >
>> > - Asset
>> > - evaluatePathRestrictions="true"
>> > - type="lucene"
>> > - includedPath="/nodeA/nodeB"
>> > - indexRules
>> > - my:Asset
>> > - properties
>> > - name="my:title"
>> > - deeperAsset
>> > - evaluatePathRestrictions="true"
>> > - type="lucene"
>> > - includedPath="/nodeA/nodeB/includedC"
>> > - ndexRules
>> > - my:Asset
>> > - properties
>> > - name="my:description"
>> >
>> > Made the system to return a cost of 1001 (for both indexes) when
>> performing
>> > these kind of queries:
>> >
>> > - SELECT * FROM [my:Asset] AS s WHERE ISDESCENDANTNODE(s,'/nodeA/
>> nodeB')
>> > - SELECT * FROM [my:Asset] AS s WHERE ISDESCENDANTNODE(s,'/nodeA/
>> nodeB')
>> > AND s.[my:title]='title'
>> > - SELECT * FROM [my:Asset] AS s WHERE ISDESCENDANTNODE(s,'/nodeA/
>> nodeB')
>> > AND s.[my:description]='description'
>> >
>> > Once cost is assigned (equal for both indexes), it is not clear which
>> index
>> > will be selected.
>> >
>> > Regards.
>> >
>> > On Tue, May 16, 2017 at 3:50 PM, Chetan Mehrotra <
>> chetan.mehrotra@gmail.com>
>> > wrote:
>> >
>> >> This looks similar to OAK-5449 (not yet fixed). Can you give a sample
>> >> index definition there and some usecase details which is leading to
>> >> ambiguity in index selection.
>> >>
>> >> In general index selection should not have multiple competing index
>> >> definitions hence interested in knowing setup details
>> >> Chetan Mehrotra
>> >>
>> >>
>> >> On Tue, May 16, 2017 at 1:53 PM, Alvaro Cabrerizo <to...@gmail.com>
>> >> wrote:
>> >> > Hello,
>> >> >
>> >> > I've been checking the code of the IndexPlanner (apache OAK 1.4.1)
>> and I
>> >> > was surprised because the costPerEntryFactor remains 1 in both cases:
>> >> >
>> >> > - when no property indexed or sorted match any property clause or
>> sort
>> >> > clause from the query
>> >> > - when only an indexed or sorted property matches a property
>> clause or
>> >> > sort clause from the query
>> >> >
>> >> > Although this piece of code avoids a division by zero (see
>> >> > org.apache.jackrabbit.oak.plugins.index.lucene.IndexPlanner lines
>> >> 201-203)
>> >> >
>> >> > if (costPerEntryFactor == 0){
>> >> > costPerEntryFactor = 1;
>> >> > }
>> >> >
>> >> > It also avoids the boosting of indexes that match 1 query clause (or
>> in
>> >> > other words, it doesn't penalize indexes that don't match any clause).
>> >> I'm
>> >> > thinking about opening an issue. Although it is for the long-tern,
>> >> actually
>> >> > I would like to know which index is selected in case that more than
>> one
>> >> had
>> >> > the same cost.
>> >> >
>> >> > Regards.
>> >>
>>
Re: index selection when 0 or 1 property clause (or sort clause) matches
Posted by Alvaro Cabrerizo <to...@gmail.com>.
Hello,
Yes, there are some reasons:
- One team is working with all the assets under /nodeA/nodeB
- Other team only works with an small subset, only assets under
/nodeA/node/nodeC
- Both teams have different searching requirements
- Rebuilding Assets index takes several days
- Rebuilding deeperAsset takes a couple of hours. Thus merging both will
penalize the size and modification agility of deeperAsset
One option we are evaluating is to cheat the deepAsset cost (using
costPerEntry property) and make that team work with lucene native queries
(any recommendation is welcome). Anyway, it is not clear which index is
selected in case of tie.
Regards.
On Tue, May 16, 2017 at 4:36 PM, Chetan Mehrotra <ch...@gmail.com>
wrote:
> Any reason for having separate definitions for same nodetype?
> Chetan Mehrotra
>
>
> On Tue, May 16, 2017 at 7:52 PM, Alvaro Cabrerizo <to...@gmail.com>
> wrote:
> > Hello,
> >
> > Actually, it is OAK-5449. Sorry, I hadn't seen it.
> >
> > On the other hand, having these two definitions under oak:index (just a
> > sketch):
> >
> >
> > - Asset
> > - evaluatePathRestrictions="true"
> > - type="lucene"
> > - includedPath="/nodeA/nodeB"
> > - indexRules
> > - my:Asset
> > - properties
> > - name="my:title"
> > - deeperAsset
> > - evaluatePathRestrictions="true"
> > - type="lucene"
> > - includedPath="/nodeA/nodeB/includedC"
> > - ndexRules
> > - my:Asset
> > - properties
> > - name="my:description"
> >
> > Made the system to return a cost of 1001 (for both indexes) when
> performing
> > these kind of queries:
> >
> > - SELECT * FROM [my:Asset] AS s WHERE ISDESCENDANTNODE(s,'/nodeA/
> nodeB')
> > - SELECT * FROM [my:Asset] AS s WHERE ISDESCENDANTNODE(s,'/nodeA/
> nodeB')
> > AND s.[my:title]='title'
> > - SELECT * FROM [my:Asset] AS s WHERE ISDESCENDANTNODE(s,'/nodeA/
> nodeB')
> > AND s.[my:description]='description'
> >
> > Once cost is assigned (equal for both indexes), it is not clear which
> index
> > will be selected.
> >
> > Regards.
> >
> > On Tue, May 16, 2017 at 3:50 PM, Chetan Mehrotra <
> chetan.mehrotra@gmail.com>
> > wrote:
> >
> >> This looks similar to OAK-5449 (not yet fixed). Can you give a sample
> >> index definition there and some usecase details which is leading to
> >> ambiguity in index selection.
> >>
> >> In general index selection should not have multiple competing index
> >> definitions hence interested in knowing setup details
> >> Chetan Mehrotra
> >>
> >>
> >> On Tue, May 16, 2017 at 1:53 PM, Alvaro Cabrerizo <to...@gmail.com>
> >> wrote:
> >> > Hello,
> >> >
> >> > I've been checking the code of the IndexPlanner (apache OAK 1.4.1)
> and I
> >> > was surprised because the costPerEntryFactor remains 1 in both cases:
> >> >
> >> > - when no property indexed or sorted match any property clause or
> sort
> >> > clause from the query
> >> > - when only an indexed or sorted property matches a property
> clause or
> >> > sort clause from the query
> >> >
> >> > Although this piece of code avoids a division by zero (see
> >> > org.apache.jackrabbit.oak.plugins.index.lucene.IndexPlanner lines
> >> 201-203)
> >> >
> >> > if (costPerEntryFactor == 0){
> >> > costPerEntryFactor = 1;
> >> > }
> >> >
> >> > It also avoids the boosting of indexes that match 1 query clause (or
> in
> >> > other words, it doesn't penalize indexes that don't match any clause).
> >> I'm
> >> > thinking about opening an issue. Although it is for the long-tern,
> >> actually
> >> > I would like to know which index is selected in case that more than
> one
> >> had
> >> > the same cost.
> >> >
> >> > Regards.
> >>
>
Re: index selection when 0 or 1 property clause (or sort clause) matches
Posted by Chetan Mehrotra <ch...@gmail.com>.
Any reason for having separate definitions for same nodetype?
Chetan Mehrotra
On Tue, May 16, 2017 at 7:52 PM, Alvaro Cabrerizo <to...@gmail.com> wrote:
> Hello,
>
> Actually, it is OAK-5449. Sorry, I hadn't seen it.
>
> On the other hand, having these two definitions under oak:index (just a
> sketch):
>
>
> - Asset
> - evaluatePathRestrictions="true"
> - type="lucene"
> - includedPath="/nodeA/nodeB"
> - indexRules
> - my:Asset
> - properties
> - name="my:title"
> - deeperAsset
> - evaluatePathRestrictions="true"
> - type="lucene"
> - includedPath="/nodeA/nodeB/includedC"
> - ndexRules
> - my:Asset
> - properties
> - name="my:description"
>
> Made the system to return a cost of 1001 (for both indexes) when performing
> these kind of queries:
>
> - SELECT * FROM [my:Asset] AS s WHERE ISDESCENDANTNODE(s,'/nodeA/nodeB')
> - SELECT * FROM [my:Asset] AS s WHERE ISDESCENDANTNODE(s,'/nodeA/nodeB')
> AND s.[my:title]='title'
> - SELECT * FROM [my:Asset] AS s WHERE ISDESCENDANTNODE(s,'/nodeA/nodeB')
> AND s.[my:description]='description'
>
> Once cost is assigned (equal for both indexes), it is not clear which index
> will be selected.
>
> Regards.
>
> On Tue, May 16, 2017 at 3:50 PM, Chetan Mehrotra <ch...@gmail.com>
> wrote:
>
>> This looks similar to OAK-5449 (not yet fixed). Can you give a sample
>> index definition there and some usecase details which is leading to
>> ambiguity in index selection.
>>
>> In general index selection should not have multiple competing index
>> definitions hence interested in knowing setup details
>> Chetan Mehrotra
>>
>>
>> On Tue, May 16, 2017 at 1:53 PM, Alvaro Cabrerizo <to...@gmail.com>
>> wrote:
>> > Hello,
>> >
>> > I've been checking the code of the IndexPlanner (apache OAK 1.4.1) and I
>> > was surprised because the costPerEntryFactor remains 1 in both cases:
>> >
>> > - when no property indexed or sorted match any property clause or sort
>> > clause from the query
>> > - when only an indexed or sorted property matches a property clause or
>> > sort clause from the query
>> >
>> > Although this piece of code avoids a division by zero (see
>> > org.apache.jackrabbit.oak.plugins.index.lucene.IndexPlanner lines
>> 201-203)
>> >
>> > if (costPerEntryFactor == 0){
>> > costPerEntryFactor = 1;
>> > }
>> >
>> > It also avoids the boosting of indexes that match 1 query clause (or in
>> > other words, it doesn't penalize indexes that don't match any clause).
>> I'm
>> > thinking about opening an issue. Although it is for the long-tern,
>> actually
>> > I would like to know which index is selected in case that more than one
>> had
>> > the same cost.
>> >
>> > Regards.
>>
Re: index selection when 0 or 1 property clause (or sort clause) matches
Posted by Alvaro Cabrerizo <to...@gmail.com>.
Hello,
Actually, it is OAK-5449. Sorry, I hadn't seen it.
On the other hand, having these two definitions under oak:index (just a
sketch):
- Asset
- evaluatePathRestrictions="true"
- type="lucene"
- includedPath="/nodeA/nodeB"
- indexRules
- my:Asset
- properties
- name="my:title"
- deeperAsset
- evaluatePathRestrictions="true"
- type="lucene"
- includedPath="/nodeA/nodeB/includedC"
- ndexRules
- my:Asset
- properties
- name="my:description"
Made the system to return a cost of 1001 (for both indexes) when performing
these kind of queries:
- SELECT * FROM [my:Asset] AS s WHERE ISDESCENDANTNODE(s,'/nodeA/nodeB')
- SELECT * FROM [my:Asset] AS s WHERE ISDESCENDANTNODE(s,'/nodeA/nodeB')
AND s.[my:title]='title'
- SELECT * FROM [my:Asset] AS s WHERE ISDESCENDANTNODE(s,'/nodeA/nodeB')
AND s.[my:description]='description'
Once cost is assigned (equal for both indexes), it is not clear which index
will be selected.
Regards.
On Tue, May 16, 2017 at 3:50 PM, Chetan Mehrotra <ch...@gmail.com>
wrote:
> This looks similar to OAK-5449 (not yet fixed). Can you give a sample
> index definition there and some usecase details which is leading to
> ambiguity in index selection.
>
> In general index selection should not have multiple competing index
> definitions hence interested in knowing setup details
> Chetan Mehrotra
>
>
> On Tue, May 16, 2017 at 1:53 PM, Alvaro Cabrerizo <to...@gmail.com>
> wrote:
> > Hello,
> >
> > I've been checking the code of the IndexPlanner (apache OAK 1.4.1) and I
> > was surprised because the costPerEntryFactor remains 1 in both cases:
> >
> > - when no property indexed or sorted match any property clause or sort
> > clause from the query
> > - when only an indexed or sorted property matches a property clause or
> > sort clause from the query
> >
> > Although this piece of code avoids a division by zero (see
> > org.apache.jackrabbit.oak.plugins.index.lucene.IndexPlanner lines
> 201-203)
> >
> > if (costPerEntryFactor == 0){
> > costPerEntryFactor = 1;
> > }
> >
> > It also avoids the boosting of indexes that match 1 query clause (or in
> > other words, it doesn't penalize indexes that don't match any clause).
> I'm
> > thinking about opening an issue. Although it is for the long-tern,
> actually
> > I would like to know which index is selected in case that more than one
> had
> > the same cost.
> >
> > Regards.
>
Re: index selection when 0 or 1 property clause (or sort clause) matches
Posted by Chetan Mehrotra <ch...@gmail.com>.
This looks similar to OAK-5449 (not yet fixed). Can you give a sample
index definition there and some usecase details which is leading to
ambiguity in index selection.
In general index selection should not have multiple competing index
definitions hence interested in knowing setup details
Chetan Mehrotra
On Tue, May 16, 2017 at 1:53 PM, Alvaro Cabrerizo <to...@gmail.com> wrote:
> Hello,
>
> I've been checking the code of the IndexPlanner (apache OAK 1.4.1) and I
> was surprised because the costPerEntryFactor remains 1 in both cases:
>
> - when no property indexed or sorted match any property clause or sort
> clause from the query
> - when only an indexed or sorted property matches a property clause or
> sort clause from the query
>
> Although this piece of code avoids a division by zero (see
> org.apache.jackrabbit.oak.plugins.index.lucene.IndexPlanner lines 201-203)
>
> if (costPerEntryFactor == 0){
> costPerEntryFactor = 1;
> }
>
> It also avoids the boosting of indexes that match 1 query clause (or in
> other words, it doesn't penalize indexes that don't match any clause). I'm
> thinking about opening an issue. Although it is for the long-tern, actually
> I would like to know which index is selected in case that more than one had
> the same cost.
>
> Regards.