You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Patrick Schemitz <ps...@solute.de> on 2014/10/13 17:04:48 UTC
Solr: very slow custom Query/Weight/Scorer, "post filtering" vs sorting
Hi all,
I'm trying to implement an image similarity search using Solr 4.6.1.
I store an image descriptor in each document, and compare these with the
descriptor given in the query, resulting in an image similarity score.
This score is then used to filter documents (via a threshold), and to
sort the results.
I've written the boilerplate QParser and QParserPlugin around a custom
Query and its accompanying Weight and Scorer classes. The
Scorer.nextDoc() is where the actual image similarity computation (and
skipping of docs below the threshold) takes place.
This Query/Weight/Scorer construct is obviously very costly, so I don't
want it to leapfrog with the other - much faster - filters in the query
(especially when using a high threshold).
I've tried the "post filtering" mechanism described by Yonik here:
http://java.dzone.com/articles/advanced-filter-caching-solr
This speeds up things, but now the results are not sorted by image
similarity any more.
I guess what I actually need is a "post query", as opposed to a "post
filter".
How can I bring together post filtering and sorting?
Do I have to write and use a custom sort function, effectively computing
image similarities twice?
Any help appreciated!
Cheers, Patrick
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: Solr: very slow custom Query/Weight/Scorer, "post filtering"
vs sorting
Posted by Patrick Schemitz <ps...@solute.de>.
On 10/13/2014 05:09 PM, Shalin Shekhar Mangar wrote:
> That sounds like
> re-ranking?
> https://cwiki.apache.org/confluence/display/solr/Query+Re-Ranking
Interesting approach. So at least I don't have to implement an
additional custom sort function. However,
"If a document matches the original query, but does not match the
re-ranking query, the document's original score will remain."
I need the ImageSimilarityQuery to drop docs with dissimilar images from
the result set. As this apparently cannot be done in the re-rank query,
I'ld have to run the ImageSimilarityQuery twice: once as a post filter
to trim the recall, and then as a re-rank query for correct sorting.
As computing the image similarities is quite expensive, I'ld rather not
do this twice.
Any idea how I can resolve this?
Cheers, Patrick
> On Mon, Oct 13, 2014 at 8:34 PM, Patrick Schemitz <ps@solute.de
> <ma...@solute.de>> wrote:
>
> Hi all,
>
> I'm trying to implement an image similarity search using Solr 4.6.1.
>
> I store an image descriptor in each document, and compare these with the
> descriptor given in the query, resulting in an image similarity score.
> This score is then used to filter documents (via a threshold), and to
> sort the results.
>
> I've written the boilerplate QParser and QParserPlugin around a custom
> Query and its accompanying Weight and Scorer classes. The
> Scorer.nextDoc() is where the actual image similarity computation (and
> skipping of docs below the threshold) takes place.
>
> This Query/Weight/Scorer construct is obviously very costly, so I don't
> want it to leapfrog with the other - much faster - filters in the query
> (especially when using a high threshold).
>
> I've tried the "post filtering" mechanism described by Yonik here:
> http://java.dzone.com/articles/advanced-filter-caching-solr
>
> This speeds up things, but now the results are not sorted by image
> similarity any more.
>
> I guess what I actually need is a "post query", as opposed to a "post
> filter".
>
> How can I bring together post filtering and sorting?
>
> Do I have to write and use a custom sort function, effectively computing
> image similarities twice?
>
> Any help appreciated!
>
> Cheers, Patrick
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> <ma...@lucene.apache.org>
> For additional commands, e-mail: dev-help@lucene.apache.org
> <ma...@lucene.apache.org>
>
>
>
>
> --
> Regards,
> Shalin Shekhar Mangar.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: Solr: very slow custom Query/Weight/Scorer, "post filtering" vs sorting
Posted by Shalin Shekhar Mangar <sh...@gmail.com>.
That sounds like re-ranking?
https://cwiki.apache.org/confluence/display/solr/Query+Re-Ranking
On Mon, Oct 13, 2014 at 8:34 PM, Patrick Schemitz <ps...@solute.de> wrote:
> Hi all,
>
> I'm trying to implement an image similarity search using Solr 4.6.1.
>
> I store an image descriptor in each document, and compare these with the
> descriptor given in the query, resulting in an image similarity score.
> This score is then used to filter documents (via a threshold), and to
> sort the results.
>
> I've written the boilerplate QParser and QParserPlugin around a custom
> Query and its accompanying Weight and Scorer classes. The
> Scorer.nextDoc() is where the actual image similarity computation (and
> skipping of docs below the threshold) takes place.
>
> This Query/Weight/Scorer construct is obviously very costly, so I don't
> want it to leapfrog with the other - much faster - filters in the query
> (especially when using a high threshold).
>
> I've tried the "post filtering" mechanism described by Yonik here:
> http://java.dzone.com/articles/advanced-filter-caching-solr
>
> This speeds up things, but now the results are not sorted by image
> similarity any more.
>
> I guess what I actually need is a "post query", as opposed to a "post
> filter".
>
> How can I bring together post filtering and sorting?
>
> Do I have to write and use a custom sort function, effectively computing
> image similarities twice?
>
> Any help appreciated!
>
> Cheers, Patrick
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
>
--
Regards,
Shalin Shekhar Mangar.
Re: Solr: very slow custom Query/Weight/Scorer, "post filtering" vs sorting
Posted by "david.w.smiley@gmail.com" <da...@gmail.com>.
On Mon, Oct 13, 2014 at 11:04 AM, Patrick Schemitz <ps...@solute.de> wrote:
> This Query/Weight/Scorer construct is obviously very costly, so I don't
> want it to leapfrog with the other - much faster - filters in the query
> (especially when using a high threshold).
>
It’s leap-frogging with filters? That’s strange; filters should be running
first, assuming the default cache=true on them. If for some reason you
want to not cache filters or other queries, then you cold have your
QueryParser parse those queries as additional parameters, and then
construct a FilteredQuery configured with your query
and QUERY_FIRST_FILTER_STRATEGY.
Cheers,
David
Re: Solr: very slow custom Query/Weight/Scorer, "post filtering"
vs sorting
Posted by Patrick Schemitz <ps...@solute.de>.
> https://bitbucket.org/dermotte/liresolr
The LIRE Solr integration uses a value source, which can then be used as
a sort function. However, if I want to implement both sorting by image
similarity *and* discarding docs with dissimilar images above some
custom threshold (while retaining proper facet counts), I would have to
do something like:
q=category:123&fq={!frange l=0.0 u=100.0 cache=false
cost=1000}lirefunc(cl, "WHATEVERFEATUREVECTOR")&sort=lirefunc(cl,
"WHATEVERFEATUREVECTOR")+asc
So unless I am mistaken, I would end up evaluating feature vectors
TWICE, which is what I'm trying to avoid, as I wrote in my original post.
Cheers, Patrick
On 10/13/2014 05:41 PM, Shawn Heisey wrote:
> On 10/13/2014 9:04 AM, Patrick Schemitz wrote:
>> I'm trying to implement an image similarity search using Solr 4.6.1.
>
> You're inventing a wheel that has already been done a couple of times.
> This does not necessarily mean that you need to give up on your
> implementation, but it does mean you need to have the right reasons for
> continuing. This might even be good news for you, because you know that
> it *is* possible.
>
> There's one commercial solution that I know of (supporting up to Solr
> 4.9), and one that's open source:
>
> http://www.pixolution.de/
> https://bitbucket.org/dermotte/liresolr
>
> The company where I work has pixolution integrated into Solr. Test
> queries look really good, but we haven't yet integrated it into anything
> that faces our users. I haven't tried the open source solution.
>
> Thanks,
> Shawn
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: Solr: very slow custom Query/Weight/Scorer, "post filtering"
vs sorting
Posted by Shawn Heisey <ap...@elyograg.org>.
On 10/13/2014 9:04 AM, Patrick Schemitz wrote:
> I'm trying to implement an image similarity search using Solr 4.6.1.
You're inventing a wheel that has already been done a couple of times.
This does not necessarily mean that you need to give up on your
implementation, but it does mean you need to have the right reasons for
continuing. This might even be good news for you, because you know that
it *is* possible.
There's one commercial solution that I know of (supporting up to Solr
4.9), and one that's open source:
http://www.pixolution.de/
https://bitbucket.org/dermotte/liresolr
The company where I work has pixolution integrated into Solr. Test
queries look really good, but we haven't yet integrated it into anything
that faces our users. I haven't tried the open source solution.
Thanks,
Shawn
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org