You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Michael Owen <mi...@hotmail.com> on 2010/12/15 15:49:35 UTC
Lower level filtering
Hi all,
I'm currently using Solr and I've got a question about filtering on a lower level than filter queries.
We want to be able to restrict the documents that can possibly be returned to a users query. From another system we'll get a list of document unique ids for the user which is all the documents that they can possibly see (i.e. a base index list as such). The criteria for what document ids get returned is going to be quite flexible. As the number of ids can be up to index size - 1 (i.e. thousands) using a filter query doesn't seem right for entering a filter query which is so large.
Can something be done at a lower level - perhaps at a Lucene level - as I understand Lucene starts from a bitset of possible documents it can return - could we AND this with a filter bitset returned from the other system? Would this be a good way forward?
And then how would you do this in Solr with still keeping Solr's extra functionality it brings over Lucene. A new SearchHandler?
Thanks
Mike
RE: Lower level filtering
Posted by Michael Owen <mi...@hotmail.com>.
That was a quick response Steve!
Sounds all great! Much appreciated. Definitely think specifying a bit filter is something that many people many find useful.
I'll have a look at Solr-2052 too.
Thanks again,
Mike
> Date: Wed, 15 Dec 2010 09:57:54 -0500
> Subject: Re: Lower level filtering
> From: eelstretching@gmail.com
> To: solr-user@lucene.apache.org
>
> On Wed, Dec 15, 2010 at 9:49 AM, Michael Owen <mi...@hotmail.com> wrote:
> > I'm currently using Solr and I've got a question about filtering on a lower level than filter queries.
> > We want to be able to restrict the documents that can possibly be returned to a users query. From another system we'll get a list of document unique ids for the user which is all the documents that they can possibly see (i.e. a base index list as such). The criteria for what document ids get returned is going to be quite flexible. As the number of ids can be up to index size - 1 (i.e. thousands) using a filter query doesn't seem right for entering a filter query which is so large.
> > Can something be done at a lower level - perhaps at a Lucene level - as I understand Lucene starts from a bitset of possible documents it can return - could we AND this with a filter bitset returned from the other system? Would this be a good way forward?
> > And then how would you do this in Solr with still keeping Solr's extra functionality it brings over Lucene. A new SearchHandler?
>
> I actually submitted a patch a while ago in Solr-2052 that allows you
> to specify a bit filter and a filter query (you could specify either,
> but not both.)
>
> Otis pointed out that the patch can't be applied against the current
> source, so I need to go back and make it work with the current source
> (new job = no time). I'll see if I can find the time this weekend to
> do this.
>
> Steve
> --
> Stephen Green
> http://thesearchguy.wordpress.com
Re: Lower level filtering
Posted by Stephen Green <ee...@gmail.com>.
On Wed, Dec 15, 2010 at 9:57 AM, Stephen Green <ee...@gmail.com> wrote:
> Otis pointed out that the patch can't be applied against the current
> source, so I need to go back and make it work with the current source
> (new job = no time). I'll see if I can find the time this weekend to
> do this.
OK, I just submitted a patch that works against trunk.
Steve
--
Stephen Green
http://thesearchguy.wordpress.com
Re: Lower level filtering
Posted by Stephen Green <ee...@gmail.com>.
On Wed, Dec 15, 2010 at 9:49 AM, Michael Owen <mi...@hotmail.com> wrote:
> I'm currently using Solr and I've got a question about filtering on a lower level than filter queries.
> We want to be able to restrict the documents that can possibly be returned to a users query. From another system we'll get a list of document unique ids for the user which is all the documents that they can possibly see (i.e. a base index list as such). The criteria for what document ids get returned is going to be quite flexible. As the number of ids can be up to index size - 1 (i.e. thousands) using a filter query doesn't seem right for entering a filter query which is so large.
> Can something be done at a lower level - perhaps at a Lucene level - as I understand Lucene starts from a bitset of possible documents it can return - could we AND this with a filter bitset returned from the other system? Would this be a good way forward?
> And then how would you do this in Solr with still keeping Solr's extra functionality it brings over Lucene. A new SearchHandler?
I actually submitted a patch a while ago in Solr-2052 that allows you
to specify a bit filter and a filter query (you could specify either,
but not both.)
Otis pointed out that the patch can't be applied against the current
source, so I need to go back and make it work with the current source
(new job = no time). I'll see if I can find the time this weekend to
do this.
Steve
--
Stephen Green
http://thesearchguy.wordpress.com
RE: Lower level filtering
Posted by Michael Owen <mi...@hotmail.com>.
Good point - though the inverse could be true where only a few documents is allowed and then a big list still exists. Even in the middle ground, its still going to be a long list of thousands.
Thanks
Mike
> Date: Wed, 15 Dec 2010 14:58:33 +0000
> Subject: Re: Lower level filtering
> From: savvas.andreas.moysidis@googlemail.com
> To: solr-user@lucene.apache.org
>
> It might not be practical in your case, but is it possible to get from that
> other system, a list of ids the user is *not* allow to see and somehow
> invert the logic in the filter?
>
> Regards,
> -- Savvas.
>
> On 15 December 2010 14:49, Michael Owen <mi...@hotmail.com> wrote:
>
> >
> > Hi all,
> > I'm currently using Solr and I've got a question about filtering on a lower
> > level than filter queries.
> > We want to be able to restrict the documents that can possibly be returned
> > to a users query. From another system we'll get a list of document unique
> > ids for the user which is all the documents that they can possibly see (i.e.
> > a base index list as such). The criteria for what document ids get returned
> > is going to be quite flexible. As the number of ids can be up to index size
> > - 1 (i.e. thousands) using a filter query doesn't seem right for entering a
> > filter query which is so large.
> > Can something be done at a lower level - perhaps at a Lucene level - as I
> > understand Lucene starts from a bitset of possible documents it can return -
> > could we AND this with a filter bitset returned from the other system? Would
> > this be a good way forward?
> > And then how would you do this in Solr with still keeping Solr's extra
> > functionality it brings over Lucene. A new SearchHandler?
> > Thanks
> > Mike
> >
> >
> >
> >
> >
> >
Re: Lower level filtering
Posted by Savvas-Andreas Moysidis <sa...@googlemail.com>.
It might not be practical in your case, but is it possible to get from that
other system, a list of ids the user is *not* allow to see and somehow
invert the logic in the filter?
Regards,
-- Savvas.
On 15 December 2010 14:49, Michael Owen <mi...@hotmail.com> wrote:
>
> Hi all,
> I'm currently using Solr and I've got a question about filtering on a lower
> level than filter queries.
> We want to be able to restrict the documents that can possibly be returned
> to a users query. From another system we'll get a list of document unique
> ids for the user which is all the documents that they can possibly see (i.e.
> a base index list as such). The criteria for what document ids get returned
> is going to be quite flexible. As the number of ids can be up to index size
> - 1 (i.e. thousands) using a filter query doesn't seem right for entering a
> filter query which is so large.
> Can something be done at a lower level - perhaps at a Lucene level - as I
> understand Lucene starts from a bitset of possible documents it can return -
> could we AND this with a filter bitset returned from the other system? Would
> this be a good way forward?
> And then how would you do this in Solr with still keeping Solr's extra
> functionality it brings over Lucene. A new SearchHandler?
> Thanks
> Mike
>
>
>
>
>
>
Re: Lower level filtering
Posted by Erick Erickson <er...@gmail.com>.
Here's the problem with what you're outlining:
Solr/Lucene doc ids are NOT invariant, so
the doc IDs you get from "the other system"
will not be directly usable by in the filter. But
assuming the other system stores what you've
defined as <uniqueKey> you could walk the
index and get the doc IDs from that (See TermDocs
in the Lucene API documentation).
There's an extensive discussion of this here:
http://lucene.472066.n3.nabble.com/filter-query-from-external-list-of-Solr-unique-IDs-td1709060.html
But this sounds a lot like this patch (I've only skimmed the
comments, so I may be off base):
https://issues.apache.org/jira/browse/SOLR-2272
note that this is a patch that I don't think has been
committed yet, so you'd have to get the source, apply
the patch and then use it. See:
http://wiki.apache.org/solr/HowToContribute
for instructions here.
Best
Erick
On Wed, Dec 15, 2010 at 9:49 AM, Michael Owen <mi...@hotmail.com>wrote:
>
> Hi all,
> I'm currently using Solr and I've got a question about filtering on a lower
> level than filter queries.
> We want to be able to restrict the documents that can possibly be returned
> to a users query. From another system we'll get a list of document unique
> ids for the user which is all the documents that they can possibly see (i.e.
> a base index list as such). The criteria for what document ids get returned
> is going to be quite flexible. As the number of ids can be up to index size
> - 1 (i.e. thousands) using a filter query doesn't seem right for entering a
> filter query which is so large.
> Can something be done at a lower level - perhaps at a Lucene level - as I
> understand Lucene starts from a bitset of possible documents it can return -
> could we AND this with a filter bitset returned from the other system? Would
> this be a good way forward?
> And then how would you do this in Solr with still keeping Solr's extra
> functionality it brings over Lucene. A new SearchHandler?
> Thanks
> Mike
>
>
>
>
>
>