You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by sp...@gmx.eu on 2012/02/13 20:17:43 UTC
OR-FilterQuery
Hi,
how efficent is such an query:
q=some text
fq=id:(1 OR 2 OR 3...)
Should I better use q:some text AND id:(1 OR 2 OR 3...)?
Is the Filter Cache used for the OR'ed fq?
Thank you
RE: OR-FilterQuery
Posted by sp...@gmx.eu.
> In other words, there's no attempt to decompose the fq clause
> and store parts of it in the cache, it's exact-match or
> nothing.
Ah ok, thank you.
Re: OR-FilterQuery
Posted by Erick Erickson <er...@gmail.com>.
bq: Is the Filter Cache used for the OR'ed fq?
The filter cache is actually pretty simple conceptually. It's
just a map where the key is the fq and the value is the set
of documents that satisfy that fq (we'll skip the implementation
here, just think of it as the list of all the docs that the fq selects).
Solr doesn't attempt to do much with the key, just think of it
as a single string. Whether or not an fq is reused from the
cache depends upon whether the key is in the map.
So fq=id:(1 OR 2 OR 3) will just look to see if
"id:(1 OR 2 OR 3)" is a key. If so, it'll just use the
document list stored in the cache.
It won't match
"id:(1 OR 2)"
or
"id:(2)
or
"id:1 OR id:2 OR id:3"
In other words, there's no attempt to decompose the fq clause
and store parts of it in the cache, it's exact-match or
nothing.
Hope that helps
Erick
On Mon, Feb 13, 2012 at 2:17 PM, <sp...@gmx.eu> wrote:
> Hi,
>
> how efficent is such an query:
>
> q=some text
> fq=id:(1 OR 2 OR 3...)
>
> Should I better use q:some text AND id:(1 OR 2 OR 3...)?
>
> Is the Filter Cache used for the OR'ed fq?
>
> Thank you
>
Re: OR-FilterQuery
Posted by Erick Erickson <er...@gmail.com>.
BTW, you're not the first person who would like this capability, see:
https://issues.apache.org/jira/browse/SOLR-1223
But the fact that this JIRA was originally opened in in June of 2009
and hasn't been implemented yet indicates that it's not super-high
priority.
Best
Erick
On Tue, Feb 14, 2012 at 4:33 PM, Erick Erickson <er...@gmail.com> wrote:
> Whoa!
>
> fq=id(1 OR 2)
> is not the same thing at all as
> fq=id:1&fq=id:2
>
> Assuming that any document had one and only one ID, the second clause
> would return exactly 0 documents, each and every time.
>
> Multiple fq clauses are essentially set intersections. So the first query is the
> set of all documents where id is 1 or 2
> the second is the intersection of two sets of documents, one set
> with an id of 1 and one with an id of 2. Not the same thing at all.
>
> There's no support for the concept of
> (fq=id:1 OR fq=id:2)
>
> Best
> Erick
>
> On Tue, Feb 14, 2012 at 2:13 PM, Em <ma...@yahoo.de> wrote:
>> Hi Mikhail,
>>
>> thanks for kicking in some brainstorming-code!
>> The given thread is almost a year old and I was working with Solr in my
>> freetime to see where it fails to behave/perform as I expect/wish.
>>
>> I found out that if you got a lot of different access-patterns for a
>> filter-query, you might end up with either a big cache to make things
>> fast or with lower performance (impact depends on usecase and
>> circumstances).
>>
>> Scenario:
>> You got a permission-field and the client is able to filter by one to
>> three permission-values.
>> That is:
>> fq=foo:user
>> fq=foo:moderator
>> fq=foo:manager
>>
>> If you can not control/guarantee the order of the fq's values, you could
>> end up with a lot of mess which all returns the same.
>>
>> Example:
>> fq=permission:user OR permission:moderator OR permission:manager
>> fq=permission:user OR permission:manager OR permission:moderator
>> fq=permission:moderator OR permission:user OR permission:manager
>> ...
>> They all return the same but where cached seperately which leads to the
>> fact that you are wasting memory a lot.
>>
>> Furthermore, if your access pattern will lead to a lot of different fq's
>> on a small set of distinct values, it may make more sense to cache each
>> filter-query for itself from a memory-consuming point of view (may cost
>> a little bit performance).
>>
>> That beeing said, if you cache a filter for foo:user, foo:moderator and
>> foo:manager you can combine those filters with AND, OR, NOT or whatever
>> without recomputing every filter over and over again which would be the
>> case if your filter-cache is not large enough.
>>
>> However, I never compared the performance differences (in terms of
>> speed) of a cached filter-query like
>> foo:bar OR foo:baz
>> With a combination of two cached filter-queries like
>> foo:bar
>> foo:baz
>> combined by a logical OR.
>>
>> That's how the background looks like.
>> Unfortunately I didn't had the time to implement this in the past.
>>
>> Back to your post:
>> Looks like a cool idea and is almost what I had in mind!
>>
>> I would formulate an easier syntax so that one is able to "parse" each
>> fq-clause on its own to cache the CachingWrapperFilter to reuse it again.
>>
>>> it will use per segment bitset at contrast to Solr's fq which caches for
>>> top level reader.
>> Could you explain why this bitset would be per-segment based, please?
>> I don't see a reason why this *have* to be so.
>> What is the benefit you are seeing?
>>
>> Kind regards,
>> Em
>>
>> Am 14.02.2012 19:33, schrieb Mikhail Khludnev:
>>> Hi Em,
>>>
>>> I briefly read the thread. Are you talking about combing of cached clauses
>>> of BooleanQuery, instead of evaluating whole BQ as a filter?
>>>
>>> I found something like that in API (but only in API)
>>> http://lucene.apache.org/solr/api/org/apache/solr/search/ExtendedQuery.html#setCacheSep(boolean)
>>>
>>> Am I get you right? Why do you need it, btw? If I'm ..
>>> I have idea how to do it in two mins:
>>>
>>> q=+f:text
>>> +(_query_:{!fq}id:1 _query_:{!fq}id:2 _query_:{!fq}id:3 _query_:{!fq}id:4)...
>>>
>>> Right leg will be a BooleanQuery with SHOULD clauses backed on cached
>>> queries (see below).
>>>
>>> if you are not scarred by the syntax yet you can implement trivial
>>> "fq"QParserPlugin, which will be just
>>>
>>> // lazily through User/Generic Cache
>>> q = new FilteredQuery (new MatchAllDocsQuery(), new
>>> CachingWrapperFilter(new
>>> QueryWrapperFilter(subQuery(localParams.get(QueryParsing.V)))));
>>> return q;
>>>
>>> it will use per segment bitset at contrast to Solr's fq which caches for
>>> top level reader.
>>>
>>> WDYT?
>>>
>>> On Mon, Feb 13, 2012 at 11:34 PM, Em <ma...@yahoo.de> wrote:
>>>
>>>> Hi,
>>>>
>>>> have a look at:
>>>> http://search-lucene.com/m/Z8lWGEiKoI
>>>>
>>>> I think not much had changed since then.
>>>>
>>>> Regards,
>>>> Em
>>>>
>>>> Am 13.02.2012 20:17, schrieb spring@gmx.eu:
>>>>> Hi,
>>>>>
>>>>> how efficent is such an query:
>>>>>
>>>>> q=some text
>>>>> fq=id:(1 OR 2 OR 3...)
>>>>>
>>>>> Should I better use q:some text AND id:(1 OR 2 OR 3...)?
>>>>>
>>>>> Is the Filter Cache used for the OR'ed fq?
>>>>>
>>>>> Thank you
>>>>>
>>>>>
>>>>
>>>
>>>
>>>
Re: OR-FilterQuery
Posted by Erick Erickson <er...@gmail.com>.
Ah, OK, I misread your post apparently. And yes, what you suggest
would result in some efficiencies, but at present I don't think there's any
syntax that allows one to combine filter queries as you suggest. There
was some discussion about it in the JIRA I referenced, but no action that
I could see.
That is, efficiencies in some circumstances, though I think it would be
hard to predict. For instance, imagine a set of 100 entries in an FQ. And
no, I'm not making things up, I've seen applications where this makes
sense. Splitting that out into 100 separate entries in the filterCache would
use up a lot of space. Likewise, I suspect that the actual process of
creating the heuristics that were able to analyze an incoming filter
query and "do the right thing" in terms of splitting it up and recombining
it would be pretty hairy. Local parameters for instance, and let's throw in
dereferencing too <G>...
So I suspect that this is one of those features that is quite easy to see
the benefits of in the simple case, but pretty quickly becomes a
nightmare to actually implement correctly, but that's mostly
a guess.
And before putting the work into it, I think modeling the actual
benefits would be wise, as well as convincing myself that there
are enough cases where this *would* be beneficial. I mean Solr
does a pretty reasonable job of caching these anyway, and with the
"non-cached" filters it's not clear to me that the benefits are
sufficient...
Good luck, though, if you want to tackle it!
Erick
On Tue, Feb 14, 2012 at 4:54 PM, Em <ma...@yahoo.de> wrote:
> Hi Erick,
>
>> Whoa!
>>
>> fq=id(1 OR 2)
>> is not the same thing at all as
>> fq=id:1&fq=id:2
> Ahm, who said they would be the same? :)
> I mean, you are completely right in what you are saying but it seems to
> me that we are talking about two different things.
>
> I was talking about caching each filter-criteria instead of the whole
> filter-query to recombine the cached filter-criteria based on the
> boolean-operators the client sends.
>
> In other words:
> currently
> fq=id:1 OR id:2
> results into ONE cached filter-entry.
>
> fq=id:2 OR id:1
> results into ANOTHER cached filter-entry
>
> fq=id:2 AND id:1
> results into (surprise, surprise) a third filter-entry (although this
> example does not make sense).
>
> My idea was to cache each filter-criteria, that means caching the bitset
> for id:1 and the bitset for id:2 to recombine both bitsets via AND, OR,
> NOT etc. whenever this is neccessary.
>
> This way one could save memory (and maybe computing-time as well) which
> definitely makes sense when you got a way smaller set of
> filter-criterias while having a much larger set of possible (and used)
> combinations of each filter-criteria with a small number of repetitions
> per combination (which would destroy the benefit of caching).
>
> Don't you agree?
>
> Kind regards,
> Em
>
>
> Am 14.02.2012 22:33, schrieb Erick Erickson:
>> Whoa!
>>
>> fq=id(1 OR 2)
>> is not the same thing at all as
>> fq=id:1&fq=id:2
>>
>> Assuming that any document had one and only one ID, the second clause
>> would return exactly 0 documents, each and every time.
>>
>> Multiple fq clauses are essentially set intersections. So the first query is the
>> set of all documents where id is 1 or 2
>> the second is the intersection of two sets of documents, one set
>> with an id of 1 and one with an id of 2. Not the same thing at all.
>>
>> There's no support for the concept of
>> (fq=id:1 OR fq=id:2)
>>
>> Best
>> Erick
>>
>> On Tue, Feb 14, 2012 at 2:13 PM, Em <ma...@yahoo.de> wrote:
>>> Hi Mikhail,
>>>
>>> thanks for kicking in some brainstorming-code!
>>> The given thread is almost a year old and I was working with Solr in my
>>> freetime to see where it fails to behave/perform as I expect/wish.
>>>
>>> I found out that if you got a lot of different access-patterns for a
>>> filter-query, you might end up with either a big cache to make things
>>> fast or with lower performance (impact depends on usecase and
>>> circumstances).
>>>
>>> Scenario:
>>> You got a permission-field and the client is able to filter by one to
>>> three permission-values.
>>> That is:
>>> fq=foo:user
>>> fq=foo:moderator
>>> fq=foo:manager
>>>
>>> If you can not control/guarantee the order of the fq's values, you could
>>> end up with a lot of mess which all returns the same.
>>>
>>> Example:
>>> fq=permission:user OR permission:moderator OR permission:manager
>>> fq=permission:user OR permission:manager OR permission:moderator
>>> fq=permission:moderator OR permission:user OR permission:manager
>>> ...
>>> They all return the same but where cached seperately which leads to the
>>> fact that you are wasting memory a lot.
>>>
>>> Furthermore, if your access pattern will lead to a lot of different fq's
>>> on a small set of distinct values, it may make more sense to cache each
>>> filter-query for itself from a memory-consuming point of view (may cost
>>> a little bit performance).
>>>
>>> That beeing said, if you cache a filter for foo:user, foo:moderator and
>>> foo:manager you can combine those filters with AND, OR, NOT or whatever
>>> without recomputing every filter over and over again which would be the
>>> case if your filter-cache is not large enough.
>>>
>>> However, I never compared the performance differences (in terms of
>>> speed) of a cached filter-query like
>>> foo:bar OR foo:baz
>>> With a combination of two cached filter-queries like
>>> foo:bar
>>> foo:baz
>>> combined by a logical OR.
>>>
>>> That's how the background looks like.
>>> Unfortunately I didn't had the time to implement this in the past.
>>>
>>> Back to your post:
>>> Looks like a cool idea and is almost what I had in mind!
>>>
>>> I would formulate an easier syntax so that one is able to "parse" each
>>> fq-clause on its own to cache the CachingWrapperFilter to reuse it again.
>>>
>>>> it will use per segment bitset at contrast to Solr's fq which caches for
>>>> top level reader.
>>> Could you explain why this bitset would be per-segment based, please?
>>> I don't see a reason why this *have* to be so.
>>> What is the benefit you are seeing?
>>>
>>> Kind regards,
>>> Em
>>>
>>> Am 14.02.2012 19:33, schrieb Mikhail Khludnev:
>>>> Hi Em,
>>>>
>>>> I briefly read the thread. Are you talking about combing of cached clauses
>>>> of BooleanQuery, instead of evaluating whole BQ as a filter?
>>>>
>>>> I found something like that in API (but only in API)
>>>> http://lucene.apache.org/solr/api/org/apache/solr/search/ExtendedQuery.html#setCacheSep(boolean)
>>>>
>>>> Am I get you right? Why do you need it, btw? If I'm ..
>>>> I have idea how to do it in two mins:
>>>>
>>>> q=+f:text
>>>> +(_query_:{!fq}id:1 _query_:{!fq}id:2 _query_:{!fq}id:3 _query_:{!fq}id:4)...
>>>>
>>>> Right leg will be a BooleanQuery with SHOULD clauses backed on cached
>>>> queries (see below).
>>>>
>>>> if you are not scarred by the syntax yet you can implement trivial
>>>> "fq"QParserPlugin, which will be just
>>>>
>>>> // lazily through User/Generic Cache
>>>> q = new FilteredQuery (new MatchAllDocsQuery(), new
>>>> CachingWrapperFilter(new
>>>> QueryWrapperFilter(subQuery(localParams.get(QueryParsing.V)))));
>>>> return q;
>>>>
>>>> it will use per segment bitset at contrast to Solr's fq which caches for
>>>> top level reader.
>>>>
>>>> WDYT?
>>>>
>>>> On Mon, Feb 13, 2012 at 11:34 PM, Em <ma...@yahoo.de> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> have a look at:
>>>>> http://search-lucene.com/m/Z8lWGEiKoI
>>>>>
>>>>> I think not much had changed since then.
>>>>>
>>>>> Regards,
>>>>> Em
>>>>>
>>>>> Am 13.02.2012 20:17, schrieb spring@gmx.eu:
>>>>>> Hi,
>>>>>>
>>>>>> how efficent is such an query:
>>>>>>
>>>>>> q=some text
>>>>>> fq=id:(1 OR 2 OR 3...)
>>>>>>
>>>>>> Should I better use q:some text AND id:(1 OR 2 OR 3...)?
>>>>>>
>>>>>> Is the Filter Cache used for the OR'ed fq?
>>>>>>
>>>>>> Thank you
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>>
>>
Re: OR-FilterQuery
Posted by Em <ma...@yahoo.de>.
Hi Erick,
> Whoa!
>
> fq=id(1 OR 2)
> is not the same thing at all as
> fq=id:1&fq=id:2
Ahm, who said they would be the same? :)
I mean, you are completely right in what you are saying but it seems to
me that we are talking about two different things.
I was talking about caching each filter-criteria instead of the whole
filter-query to recombine the cached filter-criteria based on the
boolean-operators the client sends.
In other words:
currently
fq=id:1 OR id:2
results into ONE cached filter-entry.
fq=id:2 OR id:1
results into ANOTHER cached filter-entry
fq=id:2 AND id:1
results into (surprise, surprise) a third filter-entry (although this
example does not make sense).
My idea was to cache each filter-criteria, that means caching the bitset
for id:1 and the bitset for id:2 to recombine both bitsets via AND, OR,
NOT etc. whenever this is neccessary.
This way one could save memory (and maybe computing-time as well) which
definitely makes sense when you got a way smaller set of
filter-criterias while having a much larger set of possible (and used)
combinations of each filter-criteria with a small number of repetitions
per combination (which would destroy the benefit of caching).
Don't you agree?
Kind regards,
Em
Am 14.02.2012 22:33, schrieb Erick Erickson:
> Whoa!
>
> fq=id(1 OR 2)
> is not the same thing at all as
> fq=id:1&fq=id:2
>
> Assuming that any document had one and only one ID, the second clause
> would return exactly 0 documents, each and every time.
>
> Multiple fq clauses are essentially set intersections. So the first query is the
> set of all documents where id is 1 or 2
> the second is the intersection of two sets of documents, one set
> with an id of 1 and one with an id of 2. Not the same thing at all.
>
> There's no support for the concept of
> (fq=id:1 OR fq=id:2)
>
> Best
> Erick
>
> On Tue, Feb 14, 2012 at 2:13 PM, Em <ma...@yahoo.de> wrote:
>> Hi Mikhail,
>>
>> thanks for kicking in some brainstorming-code!
>> The given thread is almost a year old and I was working with Solr in my
>> freetime to see where it fails to behave/perform as I expect/wish.
>>
>> I found out that if you got a lot of different access-patterns for a
>> filter-query, you might end up with either a big cache to make things
>> fast or with lower performance (impact depends on usecase and
>> circumstances).
>>
>> Scenario:
>> You got a permission-field and the client is able to filter by one to
>> three permission-values.
>> That is:
>> fq=foo:user
>> fq=foo:moderator
>> fq=foo:manager
>>
>> If you can not control/guarantee the order of the fq's values, you could
>> end up with a lot of mess which all returns the same.
>>
>> Example:
>> fq=permission:user OR permission:moderator OR permission:manager
>> fq=permission:user OR permission:manager OR permission:moderator
>> fq=permission:moderator OR permission:user OR permission:manager
>> ...
>> They all return the same but where cached seperately which leads to the
>> fact that you are wasting memory a lot.
>>
>> Furthermore, if your access pattern will lead to a lot of different fq's
>> on a small set of distinct values, it may make more sense to cache each
>> filter-query for itself from a memory-consuming point of view (may cost
>> a little bit performance).
>>
>> That beeing said, if you cache a filter for foo:user, foo:moderator and
>> foo:manager you can combine those filters with AND, OR, NOT or whatever
>> without recomputing every filter over and over again which would be the
>> case if your filter-cache is not large enough.
>>
>> However, I never compared the performance differences (in terms of
>> speed) of a cached filter-query like
>> foo:bar OR foo:baz
>> With a combination of two cached filter-queries like
>> foo:bar
>> foo:baz
>> combined by a logical OR.
>>
>> That's how the background looks like.
>> Unfortunately I didn't had the time to implement this in the past.
>>
>> Back to your post:
>> Looks like a cool idea and is almost what I had in mind!
>>
>> I would formulate an easier syntax so that one is able to "parse" each
>> fq-clause on its own to cache the CachingWrapperFilter to reuse it again.
>>
>>> it will use per segment bitset at contrast to Solr's fq which caches for
>>> top level reader.
>> Could you explain why this bitset would be per-segment based, please?
>> I don't see a reason why this *have* to be so.
>> What is the benefit you are seeing?
>>
>> Kind regards,
>> Em
>>
>> Am 14.02.2012 19:33, schrieb Mikhail Khludnev:
>>> Hi Em,
>>>
>>> I briefly read the thread. Are you talking about combing of cached clauses
>>> of BooleanQuery, instead of evaluating whole BQ as a filter?
>>>
>>> I found something like that in API (but only in API)
>>> http://lucene.apache.org/solr/api/org/apache/solr/search/ExtendedQuery.html#setCacheSep(boolean)
>>>
>>> Am I get you right? Why do you need it, btw? If I'm ..
>>> I have idea how to do it in two mins:
>>>
>>> q=+f:text
>>> +(_query_:{!fq}id:1 _query_:{!fq}id:2 _query_:{!fq}id:3 _query_:{!fq}id:4)...
>>>
>>> Right leg will be a BooleanQuery with SHOULD clauses backed on cached
>>> queries (see below).
>>>
>>> if you are not scarred by the syntax yet you can implement trivial
>>> "fq"QParserPlugin, which will be just
>>>
>>> // lazily through User/Generic Cache
>>> q = new FilteredQuery (new MatchAllDocsQuery(), new
>>> CachingWrapperFilter(new
>>> QueryWrapperFilter(subQuery(localParams.get(QueryParsing.V)))));
>>> return q;
>>>
>>> it will use per segment bitset at contrast to Solr's fq which caches for
>>> top level reader.
>>>
>>> WDYT?
>>>
>>> On Mon, Feb 13, 2012 at 11:34 PM, Em <ma...@yahoo.de> wrote:
>>>
>>>> Hi,
>>>>
>>>> have a look at:
>>>> http://search-lucene.com/m/Z8lWGEiKoI
>>>>
>>>> I think not much had changed since then.
>>>>
>>>> Regards,
>>>> Em
>>>>
>>>> Am 13.02.2012 20:17, schrieb spring@gmx.eu:
>>>>> Hi,
>>>>>
>>>>> how efficent is such an query:
>>>>>
>>>>> q=some text
>>>>> fq=id:(1 OR 2 OR 3...)
>>>>>
>>>>> Should I better use q:some text AND id:(1 OR 2 OR 3...)?
>>>>>
>>>>> Is the Filter Cache used for the OR'ed fq?
>>>>>
>>>>> Thank you
>>>>>
>>>>>
>>>>
>>>
>>>
>>>
>
Re: OR-FilterQuery
Posted by Erick Erickson <er...@gmail.com>.
Whoa!
fq=id(1 OR 2)
is not the same thing at all as
fq=id:1&fq=id:2
Assuming that any document had one and only one ID, the second clause
would return exactly 0 documents, each and every time.
Multiple fq clauses are essentially set intersections. So the first query is the
set of all documents where id is 1 or 2
the second is the intersection of two sets of documents, one set
with an id of 1 and one with an id of 2. Not the same thing at all.
There's no support for the concept of
(fq=id:1 OR fq=id:2)
Best
Erick
On Tue, Feb 14, 2012 at 2:13 PM, Em <ma...@yahoo.de> wrote:
> Hi Mikhail,
>
> thanks for kicking in some brainstorming-code!
> The given thread is almost a year old and I was working with Solr in my
> freetime to see where it fails to behave/perform as I expect/wish.
>
> I found out that if you got a lot of different access-patterns for a
> filter-query, you might end up with either a big cache to make things
> fast or with lower performance (impact depends on usecase and
> circumstances).
>
> Scenario:
> You got a permission-field and the client is able to filter by one to
> three permission-values.
> That is:
> fq=foo:user
> fq=foo:moderator
> fq=foo:manager
>
> If you can not control/guarantee the order of the fq's values, you could
> end up with a lot of mess which all returns the same.
>
> Example:
> fq=permission:user OR permission:moderator OR permission:manager
> fq=permission:user OR permission:manager OR permission:moderator
> fq=permission:moderator OR permission:user OR permission:manager
> ...
> They all return the same but where cached seperately which leads to the
> fact that you are wasting memory a lot.
>
> Furthermore, if your access pattern will lead to a lot of different fq's
> on a small set of distinct values, it may make more sense to cache each
> filter-query for itself from a memory-consuming point of view (may cost
> a little bit performance).
>
> That beeing said, if you cache a filter for foo:user, foo:moderator and
> foo:manager you can combine those filters with AND, OR, NOT or whatever
> without recomputing every filter over and over again which would be the
> case if your filter-cache is not large enough.
>
> However, I never compared the performance differences (in terms of
> speed) of a cached filter-query like
> foo:bar OR foo:baz
> With a combination of two cached filter-queries like
> foo:bar
> foo:baz
> combined by a logical OR.
>
> That's how the background looks like.
> Unfortunately I didn't had the time to implement this in the past.
>
> Back to your post:
> Looks like a cool idea and is almost what I had in mind!
>
> I would formulate an easier syntax so that one is able to "parse" each
> fq-clause on its own to cache the CachingWrapperFilter to reuse it again.
>
>> it will use per segment bitset at contrast to Solr's fq which caches for
>> top level reader.
> Could you explain why this bitset would be per-segment based, please?
> I don't see a reason why this *have* to be so.
> What is the benefit you are seeing?
>
> Kind regards,
> Em
>
> Am 14.02.2012 19:33, schrieb Mikhail Khludnev:
>> Hi Em,
>>
>> I briefly read the thread. Are you talking about combing of cached clauses
>> of BooleanQuery, instead of evaluating whole BQ as a filter?
>>
>> I found something like that in API (but only in API)
>> http://lucene.apache.org/solr/api/org/apache/solr/search/ExtendedQuery.html#setCacheSep(boolean)
>>
>> Am I get you right? Why do you need it, btw? If I'm ..
>> I have idea how to do it in two mins:
>>
>> q=+f:text
>> +(_query_:{!fq}id:1 _query_:{!fq}id:2 _query_:{!fq}id:3 _query_:{!fq}id:4)...
>>
>> Right leg will be a BooleanQuery with SHOULD clauses backed on cached
>> queries (see below).
>>
>> if you are not scarred by the syntax yet you can implement trivial
>> "fq"QParserPlugin, which will be just
>>
>> // lazily through User/Generic Cache
>> q = new FilteredQuery (new MatchAllDocsQuery(), new
>> CachingWrapperFilter(new
>> QueryWrapperFilter(subQuery(localParams.get(QueryParsing.V)))));
>> return q;
>>
>> it will use per segment bitset at contrast to Solr's fq which caches for
>> top level reader.
>>
>> WDYT?
>>
>> On Mon, Feb 13, 2012 at 11:34 PM, Em <ma...@yahoo.de> wrote:
>>
>>> Hi,
>>>
>>> have a look at:
>>> http://search-lucene.com/m/Z8lWGEiKoI
>>>
>>> I think not much had changed since then.
>>>
>>> Regards,
>>> Em
>>>
>>> Am 13.02.2012 20:17, schrieb spring@gmx.eu:
>>>> Hi,
>>>>
>>>> how efficent is such an query:
>>>>
>>>> q=some text
>>>> fq=id:(1 OR 2 OR 3...)
>>>>
>>>> Should I better use q:some text AND id:(1 OR 2 OR 3...)?
>>>>
>>>> Is the Filter Cache used for the OR'ed fq?
>>>>
>>>> Thank you
>>>>
>>>>
>>>
>>
>>
>>
Re: OR-FilterQuery
Posted by Em <ma...@yahoo.de>.
Hi Mikhail,
> it's just how org.apache.lucene.search.CachingWrapperFilter works. The
> first out-of-the box stuff which I've found.
Thanks for your explanation and snippets - I thought this was configurable.
Regards,
Em
Am 15.02.2012 06:16, schrieb Mikhail Khludnev:
> On Tue, Feb 14, 2012 at 11:13 PM, Em <ma...@yahoo.de> wrote:
>
>> Hi Mikhail,
>>
>>> it will use per segment bitset at contrast to Solr's fq which caches for
>>> top level reader.
>> Could you explain why this bitset would be per-segment based, please?
>
> I don't see a reason why this *have* to be so.
>>
> it's just how org.apache.lucene.search.CachingWrapperFilter works. The
> first out-of-the box stuff which I've found.
> as an top-level segment alternative we need
> org.apache.solr.search.SolrIndexSearcher.getDocSet(Query).
>
> btw, one more top-level snippet
>
> class FQParser extends QParser{
>
> Query parse(...){
> return new SolrConstantScoreQuery(
> solrIndexSearcher.getDocSet(
> subQuery(localParam.get(V))
> ).getTopFilter())
> }
> }
>
>
>
>> What is the benefit you are seeing?
>>
> It seems like two different POVs: Lucene prefer per segment caching to
> have fast incremental updates, but maybe 'because it's good but not in
> worst case' (I guess I've heard it there
> http://www.lucidimagination.com/devzone/events/conferences/ApacheLuceneEurocon2011/many-facets-apache-solr)
> Solr prefer top-reader caches.
>
>
>> Kind regards,
>> Em
>>
>> Am 14.02.2012 19:33, schrieb Mikhail Khludnev:
>>> Hi Em,
>>>
>>> I briefly read the thread. Are you talking about combing of cached
>> clauses
>>> of BooleanQuery, instead of evaluating whole BQ as a filter?
>>>
>>> I found something like that in API (but only in API)
>>>
>> http://lucene.apache.org/solr/api/org/apache/solr/search/ExtendedQuery.html#setCacheSep(boolean)
>>>
>>> Am I get you right? Why do you need it, btw? If I'm ..
>>> I have idea how to do it in two mins:
>>>
>>> q=+f:text
>>> +(_query_:{!fq}id:1 _query_:{!fq}id:2 _query_:{!fq}id:3
>> _query_:{!fq}id:4)...
>>>
>>> Right leg will be a BooleanQuery with SHOULD clauses backed on cached
>>> queries (see below).
>>>
>>> if you are not scarred by the syntax yet you can implement trivial
>>> "fq"QParserPlugin, which will be just
>>>
>>> // lazily through User/Generic Cache
>>> q = new FilteredQuery (new MatchAllDocsQuery(), new
>>> CachingWrapperFilter(new
>>> QueryWrapperFilter(subQuery(localParams.get(QueryParsing.V)))));
>>> return q;
>>>
>>> it will use per segment bitset at contrast to Solr's fq which caches for
>>> top level reader.
>>>
>>> WDYT?
>>>
>>> On Mon, Feb 13, 2012 at 11:34 PM, Em <ma...@yahoo.de>
>> wrote:
>>>
>>>> Hi,
>>>>
>>>> have a look at:
>>>> http://search-lucene.com/m/Z8lWGEiKoI
>>>>
>>>> I think not much had changed since then.
>>>>
>>>> Regards,
>>>> Em
>>>>
>>>> Am 13.02.2012 20:17, schrieb spring@gmx.eu:
>>>>> Hi,
>>>>>
>>>>> how efficent is such an query:
>>>>>
>>>>> q=some text
>>>>> fq=id:(1 OR 2 OR 3...)
>>>>>
>>>>> Should I better use q:some text AND id:(1 OR 2 OR 3...)?
>>>>>
>>>>> Is the Filter Cache used for the OR'ed fq?
>>>>>
>>>>> Thank you
>>>>>
>>>>>
>>>>
>>>
>>>
>>>
>>
>
>
>
Re: OR-FilterQuery
Posted by Mikhail Khludnev <mk...@griddynamics.com>.
On Tue, Feb 14, 2012 at 11:13 PM, Em <ma...@yahoo.de> wrote:
> Hi Mikhail,
>
> > it will use per segment bitset at contrast to Solr's fq which caches for
> > top level reader.
> Could you explain why this bitset would be per-segment based, please?
I don't see a reason why this *have* to be so.
>
it's just how org.apache.lucene.search.CachingWrapperFilter works. The
first out-of-the box stuff which I've found.
as an top-level segment alternative we need
org.apache.solr.search.SolrIndexSearcher.getDocSet(Query).
btw, one more top-level snippet
class FQParser extends QParser{
Query parse(...){
return new SolrConstantScoreQuery(
solrIndexSearcher.getDocSet(
subQuery(localParam.get(V))
).getTopFilter())
}
}
> What is the benefit you are seeing?
>
It seems like two different POVs: Lucene prefer per segment caching to
have fast incremental updates, but maybe 'because it's good but not in
worst case' (I guess I've heard it there
http://www.lucidimagination.com/devzone/events/conferences/ApacheLuceneEurocon2011/many-facets-apache-solr)
Solr prefer top-reader caches.
> Kind regards,
> Em
>
> Am 14.02.2012 19:33, schrieb Mikhail Khludnev:
> > Hi Em,
> >
> > I briefly read the thread. Are you talking about combing of cached
> clauses
> > of BooleanQuery, instead of evaluating whole BQ as a filter?
> >
> > I found something like that in API (but only in API)
> >
> http://lucene.apache.org/solr/api/org/apache/solr/search/ExtendedQuery.html#setCacheSep(boolean)
> >
> > Am I get you right? Why do you need it, btw? If I'm ..
> > I have idea how to do it in two mins:
> >
> > q=+f:text
> > +(_query_:{!fq}id:1 _query_:{!fq}id:2 _query_:{!fq}id:3
> _query_:{!fq}id:4)...
> >
> > Right leg will be a BooleanQuery with SHOULD clauses backed on cached
> > queries (see below).
> >
> > if you are not scarred by the syntax yet you can implement trivial
> > "fq"QParserPlugin, which will be just
> >
> > // lazily through User/Generic Cache
> > q = new FilteredQuery (new MatchAllDocsQuery(), new
> > CachingWrapperFilter(new
> > QueryWrapperFilter(subQuery(localParams.get(QueryParsing.V)))));
> > return q;
> >
> > it will use per segment bitset at contrast to Solr's fq which caches for
> > top level reader.
> >
> > WDYT?
> >
> > On Mon, Feb 13, 2012 at 11:34 PM, Em <ma...@yahoo.de>
> wrote:
> >
> >> Hi,
> >>
> >> have a look at:
> >> http://search-lucene.com/m/Z8lWGEiKoI
> >>
> >> I think not much had changed since then.
> >>
> >> Regards,
> >> Em
> >>
> >> Am 13.02.2012 20:17, schrieb spring@gmx.eu:
> >>> Hi,
> >>>
> >>> how efficent is such an query:
> >>>
> >>> q=some text
> >>> fq=id:(1 OR 2 OR 3...)
> >>>
> >>> Should I better use q:some text AND id:(1 OR 2 OR 3...)?
> >>>
> >>> Is the Filter Cache used for the OR'ed fq?
> >>>
> >>> Thank you
> >>>
> >>>
> >>
> >
> >
> >
>
--
Sincerely yours
Mikhail Khludnev
Lucid Certified
Apache Lucene/Solr Developer
Grid Dynamics
<http://www.griddynamics.com>
<mk...@griddynamics.com>
Re: OR-FilterQuery
Posted by Em <ma...@yahoo.de>.
Hi Mikhail,
thanks for kicking in some brainstorming-code!
The given thread is almost a year old and I was working with Solr in my
freetime to see where it fails to behave/perform as I expect/wish.
I found out that if you got a lot of different access-patterns for a
filter-query, you might end up with either a big cache to make things
fast or with lower performance (impact depends on usecase and
circumstances).
Scenario:
You got a permission-field and the client is able to filter by one to
three permission-values.
That is:
fq=foo:user
fq=foo:moderator
fq=foo:manager
If you can not control/guarantee the order of the fq's values, you could
end up with a lot of mess which all returns the same.
Example:
fq=permission:user OR permission:moderator OR permission:manager
fq=permission:user OR permission:manager OR permission:moderator
fq=permission:moderator OR permission:user OR permission:manager
...
They all return the same but where cached seperately which leads to the
fact that you are wasting memory a lot.
Furthermore, if your access pattern will lead to a lot of different fq's
on a small set of distinct values, it may make more sense to cache each
filter-query for itself from a memory-consuming point of view (may cost
a little bit performance).
That beeing said, if you cache a filter for foo:user, foo:moderator and
foo:manager you can combine those filters with AND, OR, NOT or whatever
without recomputing every filter over and over again which would be the
case if your filter-cache is not large enough.
However, I never compared the performance differences (in terms of
speed) of a cached filter-query like
foo:bar OR foo:baz
With a combination of two cached filter-queries like
foo:bar
foo:baz
combined by a logical OR.
That's how the background looks like.
Unfortunately I didn't had the time to implement this in the past.
Back to your post:
Looks like a cool idea and is almost what I had in mind!
I would formulate an easier syntax so that one is able to "parse" each
fq-clause on its own to cache the CachingWrapperFilter to reuse it again.
> it will use per segment bitset at contrast to Solr's fq which caches for
> top level reader.
Could you explain why this bitset would be per-segment based, please?
I don't see a reason why this *have* to be so.
What is the benefit you are seeing?
Kind regards,
Em
Am 14.02.2012 19:33, schrieb Mikhail Khludnev:
> Hi Em,
>
> I briefly read the thread. Are you talking about combing of cached clauses
> of BooleanQuery, instead of evaluating whole BQ as a filter?
>
> I found something like that in API (but only in API)
> http://lucene.apache.org/solr/api/org/apache/solr/search/ExtendedQuery.html#setCacheSep(boolean)
>
> Am I get you right? Why do you need it, btw? If I'm ..
> I have idea how to do it in two mins:
>
> q=+f:text
> +(_query_:{!fq}id:1 _query_:{!fq}id:2 _query_:{!fq}id:3 _query_:{!fq}id:4)...
>
> Right leg will be a BooleanQuery with SHOULD clauses backed on cached
> queries (see below).
>
> if you are not scarred by the syntax yet you can implement trivial
> "fq"QParserPlugin, which will be just
>
> // lazily through User/Generic Cache
> q = new FilteredQuery (new MatchAllDocsQuery(), new
> CachingWrapperFilter(new
> QueryWrapperFilter(subQuery(localParams.get(QueryParsing.V)))));
> return q;
>
> it will use per segment bitset at contrast to Solr's fq which caches for
> top level reader.
>
> WDYT?
>
> On Mon, Feb 13, 2012 at 11:34 PM, Em <ma...@yahoo.de> wrote:
>
>> Hi,
>>
>> have a look at:
>> http://search-lucene.com/m/Z8lWGEiKoI
>>
>> I think not much had changed since then.
>>
>> Regards,
>> Em
>>
>> Am 13.02.2012 20:17, schrieb spring@gmx.eu:
>>> Hi,
>>>
>>> how efficent is such an query:
>>>
>>> q=some text
>>> fq=id:(1 OR 2 OR 3...)
>>>
>>> Should I better use q:some text AND id:(1 OR 2 OR 3...)?
>>>
>>> Is the Filter Cache used for the OR'ed fq?
>>>
>>> Thank you
>>>
>>>
>>
>
>
>
Re: OR-FilterQuery
Posted by Mikhail Khludnev <mk...@griddynamics.com>.
Hi Em,
I briefly read the thread. Are you talking about combing of cached clauses
of BooleanQuery, instead of evaluating whole BQ as a filter?
I found something like that in API (but only in API)
http://lucene.apache.org/solr/api/org/apache/solr/search/ExtendedQuery.html#setCacheSep(boolean)
Am I get you right? Why do you need it, btw? If I'm ..
I have idea how to do it in two mins:
q=+f:text
+(_query_:{!fq}id:1 _query_:{!fq}id:2 _query_:{!fq}id:3 _query_:{!fq}id:4)...
Right leg will be a BooleanQuery with SHOULD clauses backed on cached
queries (see below).
if you are not scarred by the syntax yet you can implement trivial
"fq"QParserPlugin, which will be just
// lazily through User/Generic Cache
q = new FilteredQuery (new MatchAllDocsQuery(), new
CachingWrapperFilter(new
QueryWrapperFilter(subQuery(localParams.get(QueryParsing.V)))));
return q;
it will use per segment bitset at contrast to Solr's fq which caches for
top level reader.
WDYT?
On Mon, Feb 13, 2012 at 11:34 PM, Em <ma...@yahoo.de> wrote:
> Hi,
>
> have a look at:
> http://search-lucene.com/m/Z8lWGEiKoI
>
> I think not much had changed since then.
>
> Regards,
> Em
>
> Am 13.02.2012 20:17, schrieb spring@gmx.eu:
> > Hi,
> >
> > how efficent is such an query:
> >
> > q=some text
> > fq=id:(1 OR 2 OR 3...)
> >
> > Should I better use q:some text AND id:(1 OR 2 OR 3...)?
> >
> > Is the Filter Cache used for the OR'ed fq?
> >
> > Thank you
> >
> >
>
--
Sincerely yours
Mikhail Khludnev
Lucid Certified
Apache Lucene/Solr Developer
Grid Dynamics
<http://www.griddynamics.com>
<mk...@griddynamics.com>
Re: OR-FilterQuery
Posted by Em <ma...@yahoo.de>.
Hi,
have a look at:
http://search-lucene.com/m/Z8lWGEiKoI
I think not much had changed since then.
Regards,
Em
Am 13.02.2012 20:17, schrieb spring@gmx.eu:
> Hi,
>
> how efficent is such an query:
>
> q=some text
> fq=id:(1 OR 2 OR 3...)
>
> Should I better use q:some text AND id:(1 OR 2 OR 3...)?
>
> Is the Filter Cache used for the OR'ed fq?
>
> Thank you
>
>
RE: OR-FilterQuery
Posted by sp...@gmx.eu.
> > q=some text
> > fq=id:(1 OR 2 OR 3...)
> >
> > Should I better use q:some text AND id:(1 OR 2 OR 3...)?
> >
> 1. These two opts have the different scoring.
> 2. if you hit same fq=id:(1 OR 2 OR 3...) many times you have
> a benefit due
> to reading docset from heap instead of searching on disk.
OK, understood.
Thank you.
Re: OR-FilterQuery
Posted by Mikhail Khludnev <mk...@griddynamics.com>.
On Mon, Feb 13, 2012 at 11:17 PM, <sp...@gmx.eu> wrote:
> Hi,
>
> how efficent is such an query:
>
> q=some text
> fq=id:(1 OR 2 OR 3...)
>
> Should I better use q:some text AND id:(1 OR 2 OR 3...)?
>
1. These two opts have the different scoring.
2. if you hit same fq=id:(1 OR 2 OR 3...) many times you have a benefit due
to reading docset from heap instead of searching on disk.
>
> Is the Filter Cache used for the OR'ed fq?
>
Filter cache is used for whatever filter. I guess I didn't get you. Can't
you rephrase your question?
>
> Thank you
>
>
--
Sincerely yours
Mikhail Khludnev
Lucid Certified
Apache Lucene/Solr Developer
Grid Dynamics
<http://www.griddynamics.com>
<mk...@griddynamics.com>