You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by arian487 <ak...@tagged.com> on 2011/06/27 20:34:59 UTC
After the query component has the results, can I do more filtering
on them?
So I made a custom search component which runs right after the query
component and this custom component will update the score of each based on
some things (and no, I definitely can't use existing components). I didn't
see any easy way to just update the score so what I currently do is
something like this:
DocList docList = rb.getResults().docList;
float[] scores = new float[docList.size()];
int[] docs = new int[docList.size()];
int docCounter = 0;
int maxScore = 0;
while (docList.iterator().hasNext()) {
int userId = docList.iterator().nextDoc();
int score = userIdsToScore.get(userId);
scores[docCounter] = score;
docs[docCounter] = userId;
docCounter++;
if (maxScore < score) {
maxScore = score;
}
}
docList = new DocSlice(0, docCounter, docs, scores, 0, maxScore);
my userIdsToScore hashtable is how I'm determining the new score. There are
a few other things I'm doing but this is the gist. I'm also not sure how to
go about sorting this...but basically my question is, is this how I should
be updating the score of the documents?
--
View this message in context: http://lucene.472066.n3.nabble.com/After-the-query-component-has-the-results-can-I-do-more-filtering-on-them-tp3114775p3114775.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: After the query component has the results, can I do more
filtering on them?
Posted by arian487 <ak...@tagged.com>.
bump
--
View this message in context: http://lucene.472066.n3.nabble.com/After-the-query-component-has-the-results-can-I-do-more-filtering-on-them-tp3114775p3123502.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: After the query component has the results, can I do more
filtering on them?
Posted by arian487 <ak...@tagged.com>.
unfortunately the userIdsToScore updates very often. I'd get more Ids almost
every single query (hence why I made the new component). But I see the
problem of not being able to score the whole resultSet. I'd actually need
to do this now that I think about it. I want to get a whole whack of users
(lets say 10,000), score them using my system, and then 'remember' the top
3500 of these users in the result cache or something.
How would I go about operating on the whole resultSet rather then just the
'rows' I set. I wonder if I can set rows to be really large, score them in
the component, and then remember all of these results in the result cache
and then dynamically change rows in my component so not all 3500 (or w/e
number I choose) are returned.
--
View this message in context: http://lucene.472066.n3.nabble.com/After-the-query-component-has-the-results-can-I-do-more-filtering-on-them-tp3114775p3127560.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: After the query component has the results, can I do more
filtering on them?
Posted by arian487 <ak...@tagged.com>.
Sorry for being vague. Okay so these scores exist on an external server and
they change often enough. The score for each returned user is actually
dependent on the user doing the searching (if I'm making the request, and
you make the same request, the scores are different). So what I'm doing is
getting a bunch of scores from the external and aggregating that with the
current scores solr gave in my component. So heres the flow (all numbers
are arbitrary):
1) Get 10,000 results from solr from the query component
2) return a list of scores and ids from the external server (it'll return a
lot of them)
3) Out of this 10000, I take the top 3500 docs after aggregating the
external servers scores and netcons scores.
The problem is, the score for each doc is specific to the user making the
request. The algorithm in doing these scores is quite complex. I cannot
simply re-index with new scores, hence I've written this component which
runs after querycomponent and does the magic of filtering.
I've come up with a solution but it involved me changing a lot of solr code.
First and foremost, I've maed the queryResultCache public and developed a
small API in accessing and changing it. I've also changed the
QueryResultKey to include a Long userId in its hashCode and equals
functions. When a search is made, the QueryComponent caches its results,
and then in my custom component I go into that cache, get my superset,
filter it out from the scores in my external server, and throw it back into
cache. Of course none of this happens if my custom scored stuff is already
cached, so its actually decent.
If you have any suggestions and improvements I'd greatly appreciate it.
Sorry for the long response...I didn't want to be an XY problem again :D
--
View this message in context: http://lucene.472066.n3.nabble.com/After-the-query-component-has-the-results-can-I-do-more-filtering-on-them-tp3114775p3141652.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: After the query component has the results, can I do more filtering
on them?
Posted by Chris Hostetter <ho...@fucit.org>.
: Sorry for the double post but in this case, is it possible for me to access
: the queryResultCache in my component and play with it? Ideally what I want
: is this:
it could be possible to do what you're describing, but it would probabl be
fairly brittle.
i know you said earlier thta you can't use any eisting components, but i
strongly urge you to post the details on *what* you wnat to do (ie: where
are these scores coming from, how are the determined, how often do they
change, do all of them change or just some of them, etc..) instead of
*how* you want to do it (ie: modify the scores after the search)
Even if an existing tool (like ExternalFileField) can't be used directly
in your case, providing the full information about your use case may help
people suggest a completley differnet approach then the one you're
considering...
http://people.apache.org/~hossman/#xyproblem
XY Problem
Your question appears to be an "XY Problem" ... that is: you are dealing
with "X", you are assuming "Y" will help you, and you are asking about "Y"
without giving more details about the "X" so that we can understand the
full issue. Perhaps the best solution doesn't involve "Y" at all?
See Also: http://www.perlmonks.org/index.pl?node_id=542341
-Hoss
Re: After the query component has the results, can I do more
filtering on them?
Posted by arian487 <ak...@tagged.com>.
Sorry for the double post but in this case, is it possible for me to access
the queryResultCache in my component and play with it? Ideally what I want
is this:
1) I have 10000 (just a random large number) total results.
2) In my component I access all of these results, score them, and take the
top 3500 (a random smaller number) and drop the rest.
3) The 3500 I have now should end up going into the queryResultCache and
essentially replacing the other one.
4) The number returned to the user should then be rows and subsequent
queries which are the same just gets them from my new result cache.
I'm pretty noob about all if this so I'm hoping someone can help.
--
View this message in context: http://lucene.472066.n3.nabble.com/After-the-query-component-has-the-results-can-I-do-more-filtering-on-them-tp3114775p3127581.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: After the query component has the results, can I do more filtering on them?
Posted by Ahmet Arslan <io...@yahoo.com>.
> So I made a custom search component
> which runs right after the query
> component and this custom component will update the score
> of each based on
> some things (and no, I definitely can't use existing
> components). I didn't
> see any easy way to just update the score so what I
> currently do is
> something like this:
>
>
> DocList docList = rb.getResults().docList;
> float[]
> scores = new float[docList.size()];
> int[] docs = new
> int[docList.size()];
> int docCounter = 0;
> int maxScore = 0;
>
> while
> (docList.iterator().hasNext()) {
>
> int userId = docList.iterator().nextDoc();
>
> int score = userIdsToScore.get(userId);
>
>
> scores[docCounter] = score;
>
> docs[docCounter] = userId;
>
> docCounter++;
>
>
> if
> (maxScore < score) {
>
> maxScore = score;
> }
> }
> docList = new
> DocSlice(0, docCounter, docs, scores, 0, maxScore);
>
> my userIdsToScore hashtable is how I'm determining the new
> score. There are
> a few other things I'm doing but this is the gist.
> I'm also not sure how to
> go about sorting this...but basically my question is, is
> this how I should
> be updating the score of the documents?
This way you are just updating the scores cosmetically. i.e. they are not sorted by score anymore. Plus with this approach you can only process start + rows many documents at maximum. Obtaining the whole result set is not an option.
If you have some mapping like userIdsToScore, may be you can use ExternalFileField combined with FunctionQueries to influence score.