You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Darin Amos <da...@gmail.com> on 2014/11/28 02:28:11 UTC
Trying to get ALL scores from a previous search in a custom search component ("last-components")
Hello,
I am trying to implement a Rollup Search component on a version of SOLR that exists previously to the parent/child additions, so I am trying to implement my own. The searches will be executed exclusively against the child documents, and I want to “rollup” those child documents into the parent documents.
The interface is going to allow the user to add the following parameters to the SOLR query:
&rollup=true&rollup.parentField=id&rollup.childField=parentId
My code so far is below. What I have works so far, except my second parent query loses the order. I would like to be able to sort my parent query by the score of the previous child search. Perhaps I would take the highest score from all children (haven’t decided yet). My problem however is that I don’t know how I can get the score from all the hits in the original search, just what is returned. If my child query gets 10,000 hits, but only return 100 records, I can’t get all the scores I need.
Does anyone have any recommendations?
Thanks!!
Darin
//Loop through all the records and look for the parent reference field
Set<String> parentRefs = new HashSet<String>();
DocIterator docSetIterator = rb.getResults().docSet.iterator();
while(docSetIterator.hasNext()){
int docInt = docSetIterator.next();
String fieldValues[] = rb.req.getSearcher().doc(docInt).getValues(childFieldName);
for(String fieldValue : fieldValues){
if(fieldValue != null && fieldValue.length() > 0 && !parentRefs.contains(fieldValue)){
parentRefs.add(fieldValue);
}
}
}
//Build a boolean query of term queries
BooleanQuery parentQuery = new BooleanQuery();
Iterator<String> parentIdIterator = parentRefs.iterator();
while(parentIdIterator.hasNext()){
String parentId = parentIdIterator.next();
TermQuery termQuery = new TermQuery(new Term(parentFieldName, parentId));
parentQuery.add(termQuery, BooleanClause.Occur.SHOULD);
}
DocList parentList = searcher.getDocList(parentQuery, new ArrayList<Query>(), null, 0, 100, 1); //TODO: use correct start/end/flags later...
//Add parent results
ResultContext resultContext = new ResultContext();
resultContext.docs = parentList;
resultContext.query = parentQuery;
rb.rsp.add("parents", resultContext);
rb.rsp.getToLog().add("hits", parentList.matches());
Re: Trying to get ALL scores from a previous search in a custom
search component ("last-components")
Posted by Erick Erickson <er...@gmail.com>.
Note that what I said doesn't apply if you use the underlying
_index_ reading capabilities in Lucene. It's a bit more complicated
than just fetching the doc, but it's all memory-resident thus fast.
You need to unInvert the index though, and DocValues may
be helpful here.
Best,
Erick
On Fri, Nov 28, 2014 at 1:26 PM, Darin Amos <da...@gmail.com> wrote:
> Thanks for the advice,
>
> I will take a look to see if there is some tuning I can do here, I am not terribly concerned about that yet anyway.
>
> My concern still remains with how can I get the scores of the entire matched set. Maybe it is not possible, or perhaps I need to write my own query/match/scorer to do this.
>
> Curious is anyone else has tried to implement a similar rollup type search component before.
>
> Thanks!
>
> Darin
>
>> On Nov 28, 2014, at 4:11 PM, Erick Erickson <er...@gmail.com> wrote:
>>
>> Because you're fetching and decompressing the doc from disk. Grouping etc.
>> Do their work from _indexed_ terms, which are already in memory. Two
>> different things.
>>
>> If I'm reading this right on a quick scan...
>>
>> Best
>> Erick
>> On Nov 28, 2014 10:21 AM, "Darin Amos" <da...@gmail.com> wrote:
>>
>>> Hi Eric,
>>>
>>> I am curious why this would b considered an anti-patern to check a stored
>>> valued for every matching document. Is this not what the facet query
>>> component is doing anyway so it can get the total counts?
>>>
>>> Grouping doesn’t solve the issue because again, I will only see groups for
>>> the items returned, not all matching documents.
>>>
>>> I am using a product that has SOLR 4.3 embedded into it, so I cannot
>>> upgrade to 4.9 or 4.10 and take advantage of the parent/child feature added
>>> recently.
>>>
>>> Thanks!
>>>
>>> Darin
>>>
>>>> On Nov 28, 2014, at 12:03 PM, Erick Erickson <er...@gmail.com>
>>> wrote:
>>>>
>>>> Does grouping work for you here? Because
>>>> even if you solve this problem, if I'm reading
>>>> this right you're going to fetch stored values
>>>> for every doc that matches the query, which
>>>> is an anti-pattern big-time, consider *:*....
>>>>
>>>> Of course I did a very quick skim, so maybe
>>>> I'm all wet....
>>>>
>>>> Best,
>>>> Erick
>>>>
>>>> On Thu, Nov 27, 2014 at 5:28 PM, Darin Amos <da...@gmail.com> wrote:
>>>>> Hello,
>>>>>
>>>>> I am trying to implement a Rollup Search component on a version of SOLR
>>> that exists previously to the parent/child additions, so I am trying to
>>> implement my own. The searches will be executed exclusively against the
>>> child documents, and I want to “rollup” those child documents into the
>>> parent documents.
>>>>>
>>>>> The interface is going to allow the user to add the following
>>> parameters to the SOLR query:
>>>>>
>>>>> &rollup=true&rollup.parentField=id&rollup.childField=parentId
>>>>>
>>>>> My code so far is below. What I have works so far, except my second
>>> parent query loses the order. I would like to be able to sort my parent
>>> query by the score of the previous child search. Perhaps I would take the
>>> highest score from all children (haven’t decided yet). My problem however
>>> is that I don’t know how I can get the score from all the hits in the
>>> original search, just what is returned. If my child query gets 10,000 hits,
>>> but only return 100 records, I can’t get all the scores I need.
>>>>>
>>>>> Does anyone have any recommendations?
>>>>>
>>>>> Thanks!!
>>>>> Darin
>>>>>
>>>>>
>>>>> //Loop through all the records and look for the
>>> parent reference field
>>>>> Set<String> parentRefs = new HashSet<String>();
>>>>> DocIterator docSetIterator =
>>> rb.getResults().docSet.iterator();
>>>>> while(docSetIterator.hasNext()){
>>>>> int docInt = docSetIterator.next();
>>>>> String fieldValues[] =
>>> rb.req.getSearcher().doc(docInt).getValues(childFieldName);
>>>>>
>>>>> for(String fieldValue : fieldValues){
>>>>> if(fieldValue != null &&
>>> fieldValue.length() > 0 && !parentRefs.contains(fieldValue)){
>>>>>
>>> parentRefs.add(fieldValue);
>>>>> }
>>>>> }
>>>>> }
>>>>>
>>>>> //Build a boolean query of term queries
>>>>> BooleanQuery parentQuery = new BooleanQuery();
>>>>> Iterator<String> parentIdIterator =
>>> parentRefs.iterator();
>>>>> while(parentIdIterator.hasNext()){
>>>>> String parentId =
>>> parentIdIterator.next();
>>>>> TermQuery termQuery = new TermQuery(new
>>> Term(parentFieldName, parentId));
>>>>> parentQuery.add(termQuery,
>>> BooleanClause.Occur.SHOULD);
>>>>> }
>>>>>
>>>>>
>>>>> DocList parentList =
>>> searcher.getDocList(parentQuery, new ArrayList<Query>(), null, 0, 100, 1);
>>> //TODO: use correct start/end/flags later...
>>>>>
>>>>> //Add parent results
>>>>> ResultContext resultContext = new ResultContext();
>>>>> resultContext.docs = parentList;
>>>>> resultContext.query = parentQuery;
>>>>> rb.rsp.add("parents", resultContext);
>>>>> rb.rsp.getToLog().add("hits", parentList.matches());
>>>
>>>
>
Re: Trying to get ALL scores from a previous search in a custom search component ("last-components")
Posted by Darin Amos <da...@gmail.com>.
Thanks for the advice,
I will take a look to see if there is some tuning I can do here, I am not terribly concerned about that yet anyway.
My concern still remains with how can I get the scores of the entire matched set. Maybe it is not possible, or perhaps I need to write my own query/match/scorer to do this.
Curious is anyone else has tried to implement a similar rollup type search component before.
Thanks!
Darin
> On Nov 28, 2014, at 4:11 PM, Erick Erickson <er...@gmail.com> wrote:
>
> Because you're fetching and decompressing the doc from disk. Grouping etc.
> Do their work from _indexed_ terms, which are already in memory. Two
> different things.
>
> If I'm reading this right on a quick scan...
>
> Best
> Erick
> On Nov 28, 2014 10:21 AM, "Darin Amos" <da...@gmail.com> wrote:
>
>> Hi Eric,
>>
>> I am curious why this would b considered an anti-patern to check a stored
>> valued for every matching document. Is this not what the facet query
>> component is doing anyway so it can get the total counts?
>>
>> Grouping doesn’t solve the issue because again, I will only see groups for
>> the items returned, not all matching documents.
>>
>> I am using a product that has SOLR 4.3 embedded into it, so I cannot
>> upgrade to 4.9 or 4.10 and take advantage of the parent/child feature added
>> recently.
>>
>> Thanks!
>>
>> Darin
>>
>>> On Nov 28, 2014, at 12:03 PM, Erick Erickson <er...@gmail.com>
>> wrote:
>>>
>>> Does grouping work for you here? Because
>>> even if you solve this problem, if I'm reading
>>> this right you're going to fetch stored values
>>> for every doc that matches the query, which
>>> is an anti-pattern big-time, consider *:*....
>>>
>>> Of course I did a very quick skim, so maybe
>>> I'm all wet....
>>>
>>> Best,
>>> Erick
>>>
>>> On Thu, Nov 27, 2014 at 5:28 PM, Darin Amos <da...@gmail.com> wrote:
>>>> Hello,
>>>>
>>>> I am trying to implement a Rollup Search component on a version of SOLR
>> that exists previously to the parent/child additions, so I am trying to
>> implement my own. The searches will be executed exclusively against the
>> child documents, and I want to “rollup” those child documents into the
>> parent documents.
>>>>
>>>> The interface is going to allow the user to add the following
>> parameters to the SOLR query:
>>>>
>>>> &rollup=true&rollup.parentField=id&rollup.childField=parentId
>>>>
>>>> My code so far is below. What I have works so far, except my second
>> parent query loses the order. I would like to be able to sort my parent
>> query by the score of the previous child search. Perhaps I would take the
>> highest score from all children (haven’t decided yet). My problem however
>> is that I don’t know how I can get the score from all the hits in the
>> original search, just what is returned. If my child query gets 10,000 hits,
>> but only return 100 records, I can’t get all the scores I need.
>>>>
>>>> Does anyone have any recommendations?
>>>>
>>>> Thanks!!
>>>> Darin
>>>>
>>>>
>>>> //Loop through all the records and look for the
>> parent reference field
>>>> Set<String> parentRefs = new HashSet<String>();
>>>> DocIterator docSetIterator =
>> rb.getResults().docSet.iterator();
>>>> while(docSetIterator.hasNext()){
>>>> int docInt = docSetIterator.next();
>>>> String fieldValues[] =
>> rb.req.getSearcher().doc(docInt).getValues(childFieldName);
>>>>
>>>> for(String fieldValue : fieldValues){
>>>> if(fieldValue != null &&
>> fieldValue.length() > 0 && !parentRefs.contains(fieldValue)){
>>>>
>> parentRefs.add(fieldValue);
>>>> }
>>>> }
>>>> }
>>>>
>>>> //Build a boolean query of term queries
>>>> BooleanQuery parentQuery = new BooleanQuery();
>>>> Iterator<String> parentIdIterator =
>> parentRefs.iterator();
>>>> while(parentIdIterator.hasNext()){
>>>> String parentId =
>> parentIdIterator.next();
>>>> TermQuery termQuery = new TermQuery(new
>> Term(parentFieldName, parentId));
>>>> parentQuery.add(termQuery,
>> BooleanClause.Occur.SHOULD);
>>>> }
>>>>
>>>>
>>>> DocList parentList =
>> searcher.getDocList(parentQuery, new ArrayList<Query>(), null, 0, 100, 1);
>> //TODO: use correct start/end/flags later...
>>>>
>>>> //Add parent results
>>>> ResultContext resultContext = new ResultContext();
>>>> resultContext.docs = parentList;
>>>> resultContext.query = parentQuery;
>>>> rb.rsp.add("parents", resultContext);
>>>> rb.rsp.getToLog().add("hits", parentList.matches());
>>
>>
Re: Trying to get ALL scores from a previous search in a custom
search component ("last-components")
Posted by Erick Erickson <er...@gmail.com>.
Because you're fetching and decompressing the doc from disk. Grouping etc.
Do their work from _indexed_ terms, which are already in memory. Two
different things.
If I'm reading this right on a quick scan...
Best
Erick
On Nov 28, 2014 10:21 AM, "Darin Amos" <da...@gmail.com> wrote:
> Hi Eric,
>
> I am curious why this would b considered an anti-patern to check a stored
> valued for every matching document. Is this not what the facet query
> component is doing anyway so it can get the total counts?
>
> Grouping doesn’t solve the issue because again, I will only see groups for
> the items returned, not all matching documents.
>
> I am using a product that has SOLR 4.3 embedded into it, so I cannot
> upgrade to 4.9 or 4.10 and take advantage of the parent/child feature added
> recently.
>
> Thanks!
>
> Darin
>
> > On Nov 28, 2014, at 12:03 PM, Erick Erickson <er...@gmail.com>
> wrote:
> >
> > Does grouping work for you here? Because
> > even if you solve this problem, if I'm reading
> > this right you're going to fetch stored values
> > for every doc that matches the query, which
> > is an anti-pattern big-time, consider *:*....
> >
> > Of course I did a very quick skim, so maybe
> > I'm all wet....
> >
> > Best,
> > Erick
> >
> > On Thu, Nov 27, 2014 at 5:28 PM, Darin Amos <da...@gmail.com> wrote:
> >> Hello,
> >>
> >> I am trying to implement a Rollup Search component on a version of SOLR
> that exists previously to the parent/child additions, so I am trying to
> implement my own. The searches will be executed exclusively against the
> child documents, and I want to “rollup” those child documents into the
> parent documents.
> >>
> >> The interface is going to allow the user to add the following
> parameters to the SOLR query:
> >>
> >> &rollup=true&rollup.parentField=id&rollup.childField=parentId
> >>
> >> My code so far is below. What I have works so far, except my second
> parent query loses the order. I would like to be able to sort my parent
> query by the score of the previous child search. Perhaps I would take the
> highest score from all children (haven’t decided yet). My problem however
> is that I don’t know how I can get the score from all the hits in the
> original search, just what is returned. If my child query gets 10,000 hits,
> but only return 100 records, I can’t get all the scores I need.
> >>
> >> Does anyone have any recommendations?
> >>
> >> Thanks!!
> >> Darin
> >>
> >>
> >> //Loop through all the records and look for the
> parent reference field
> >> Set<String> parentRefs = new HashSet<String>();
> >> DocIterator docSetIterator =
> rb.getResults().docSet.iterator();
> >> while(docSetIterator.hasNext()){
> >> int docInt = docSetIterator.next();
> >> String fieldValues[] =
> rb.req.getSearcher().doc(docInt).getValues(childFieldName);
> >>
> >> for(String fieldValue : fieldValues){
> >> if(fieldValue != null &&
> fieldValue.length() > 0 && !parentRefs.contains(fieldValue)){
> >>
> parentRefs.add(fieldValue);
> >> }
> >> }
> >> }
> >>
> >> //Build a boolean query of term queries
> >> BooleanQuery parentQuery = new BooleanQuery();
> >> Iterator<String> parentIdIterator =
> parentRefs.iterator();
> >> while(parentIdIterator.hasNext()){
> >> String parentId =
> parentIdIterator.next();
> >> TermQuery termQuery = new TermQuery(new
> Term(parentFieldName, parentId));
> >> parentQuery.add(termQuery,
> BooleanClause.Occur.SHOULD);
> >> }
> >>
> >>
> >> DocList parentList =
> searcher.getDocList(parentQuery, new ArrayList<Query>(), null, 0, 100, 1);
> //TODO: use correct start/end/flags later...
> >>
> >> //Add parent results
> >> ResultContext resultContext = new ResultContext();
> >> resultContext.docs = parentList;
> >> resultContext.query = parentQuery;
> >> rb.rsp.add("parents", resultContext);
> >> rb.rsp.getToLog().add("hits", parentList.matches());
>
>
Re: Trying to get ALL scores from a previous search in a custom search component ("last-components")
Posted by Darin Amos <da...@gmail.com>.
Hi Eric,
I am curious why this would b considered an anti-patern to check a stored valued for every matching document. Is this not what the facet query component is doing anyway so it can get the total counts?
Grouping doesn’t solve the issue because again, I will only see groups for the items returned, not all matching documents.
I am using a product that has SOLR 4.3 embedded into it, so I cannot upgrade to 4.9 or 4.10 and take advantage of the parent/child feature added recently.
Thanks!
Darin
> On Nov 28, 2014, at 12:03 PM, Erick Erickson <er...@gmail.com> wrote:
>
> Does grouping work for you here? Because
> even if you solve this problem, if I'm reading
> this right you're going to fetch stored values
> for every doc that matches the query, which
> is an anti-pattern big-time, consider *:*....
>
> Of course I did a very quick skim, so maybe
> I'm all wet....
>
> Best,
> Erick
>
> On Thu, Nov 27, 2014 at 5:28 PM, Darin Amos <da...@gmail.com> wrote:
>> Hello,
>>
>> I am trying to implement a Rollup Search component on a version of SOLR that exists previously to the parent/child additions, so I am trying to implement my own. The searches will be executed exclusively against the child documents, and I want to “rollup” those child documents into the parent documents.
>>
>> The interface is going to allow the user to add the following parameters to the SOLR query:
>>
>> &rollup=true&rollup.parentField=id&rollup.childField=parentId
>>
>> My code so far is below. What I have works so far, except my second parent query loses the order. I would like to be able to sort my parent query by the score of the previous child search. Perhaps I would take the highest score from all children (haven’t decided yet). My problem however is that I don’t know how I can get the score from all the hits in the original search, just what is returned. If my child query gets 10,000 hits, but only return 100 records, I can’t get all the scores I need.
>>
>> Does anyone have any recommendations?
>>
>> Thanks!!
>> Darin
>>
>>
>> //Loop through all the records and look for the parent reference field
>> Set<String> parentRefs = new HashSet<String>();
>> DocIterator docSetIterator = rb.getResults().docSet.iterator();
>> while(docSetIterator.hasNext()){
>> int docInt = docSetIterator.next();
>> String fieldValues[] = rb.req.getSearcher().doc(docInt).getValues(childFieldName);
>>
>> for(String fieldValue : fieldValues){
>> if(fieldValue != null && fieldValue.length() > 0 && !parentRefs.contains(fieldValue)){
>> parentRefs.add(fieldValue);
>> }
>> }
>> }
>>
>> //Build a boolean query of term queries
>> BooleanQuery parentQuery = new BooleanQuery();
>> Iterator<String> parentIdIterator = parentRefs.iterator();
>> while(parentIdIterator.hasNext()){
>> String parentId = parentIdIterator.next();
>> TermQuery termQuery = new TermQuery(new Term(parentFieldName, parentId));
>> parentQuery.add(termQuery, BooleanClause.Occur.SHOULD);
>> }
>>
>>
>> DocList parentList = searcher.getDocList(parentQuery, new ArrayList<Query>(), null, 0, 100, 1); //TODO: use correct start/end/flags later...
>>
>> //Add parent results
>> ResultContext resultContext = new ResultContext();
>> resultContext.docs = parentList;
>> resultContext.query = parentQuery;
>> rb.rsp.add("parents", resultContext);
>> rb.rsp.getToLog().add("hits", parentList.matches());
Re: Trying to get ALL scores from a previous search in a custom
search component ("last-components")
Posted by Erick Erickson <er...@gmail.com>.
Does grouping work for you here? Because
even if you solve this problem, if I'm reading
this right you're going to fetch stored values
for every doc that matches the query, which
is an anti-pattern big-time, consider *:*....
Of course I did a very quick skim, so maybe
I'm all wet....
Best,
Erick
On Thu, Nov 27, 2014 at 5:28 PM, Darin Amos <da...@gmail.com> wrote:
> Hello,
>
> I am trying to implement a Rollup Search component on a version of SOLR that exists previously to the parent/child additions, so I am trying to implement my own. The searches will be executed exclusively against the child documents, and I want to “rollup” those child documents into the parent documents.
>
> The interface is going to allow the user to add the following parameters to the SOLR query:
>
> &rollup=true&rollup.parentField=id&rollup.childField=parentId
>
> My code so far is below. What I have works so far, except my second parent query loses the order. I would like to be able to sort my parent query by the score of the previous child search. Perhaps I would take the highest score from all children (haven’t decided yet). My problem however is that I don’t know how I can get the score from all the hits in the original search, just what is returned. If my child query gets 10,000 hits, but only return 100 records, I can’t get all the scores I need.
>
> Does anyone have any recommendations?
>
> Thanks!!
> Darin
>
>
> //Loop through all the records and look for the parent reference field
> Set<String> parentRefs = new HashSet<String>();
> DocIterator docSetIterator = rb.getResults().docSet.iterator();
> while(docSetIterator.hasNext()){
> int docInt = docSetIterator.next();
> String fieldValues[] = rb.req.getSearcher().doc(docInt).getValues(childFieldName);
>
> for(String fieldValue : fieldValues){
> if(fieldValue != null && fieldValue.length() > 0 && !parentRefs.contains(fieldValue)){
> parentRefs.add(fieldValue);
> }
> }
> }
>
> //Build a boolean query of term queries
> BooleanQuery parentQuery = new BooleanQuery();
> Iterator<String> parentIdIterator = parentRefs.iterator();
> while(parentIdIterator.hasNext()){
> String parentId = parentIdIterator.next();
> TermQuery termQuery = new TermQuery(new Term(parentFieldName, parentId));
> parentQuery.add(termQuery, BooleanClause.Occur.SHOULD);
> }
>
>
> DocList parentList = searcher.getDocList(parentQuery, new ArrayList<Query>(), null, 0, 100, 1); //TODO: use correct start/end/flags later...
>
> //Add parent results
> ResultContext resultContext = new ResultContext();
> resultContext.docs = parentList;
> resultContext.query = parentQuery;
> rb.rsp.add("parents", resultContext);
> rb.rsp.getToLog().add("hits", parentList.matches());