You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Mike Hugo <mi...@piragua.com> on 2012/01/26 18:04:50 UTC
Solr Join query with fq not correctly filtering results?
Hello,
I'm trying out the Solr JOIN query functionality on trunk. I have the
latest checkout, revision #1236272 - I did the following steps to get the
example up and running:
cd solr
ant example
java -jar start.jar
cd exampledocs
java -jar post.jar *.xml
Then I tried a few of the sample queries on the wiki page
http://wiki.apache.org/solr/Join. In particular, this is one that I'm
interest in
Find all manufacturer docs named "belkin", then join them against (product)
> docs and filter that list to only products with a price less than 12 dollars
>
> http://localhost:8983/solr/select?q={!join+from=id+to=manu_id_s}compName_s:Belkin&fq=price:%5B%2A+TO+12%5D
However, when I run that query, I get two results, one with a price of
19.95 and another with a price of 11.5 Because of the filter query, I'm
only expecting to see one result - the one with a price of 11.99.
I was also able to replicate this in a unit test added to
org.apache.solr.TestJoin:
@Test
public void testJoin_withFilterQuery() throws Exception {
assertU(add(doc("id", "1","name", "john", "title", "Director",
"dept_s","Engineering")));
assertU(add(doc("id", "2","name", "mark", "title", "VP",
"dept_s","Marketing")));
assertU(add(doc("id", "3","name", "nancy", "title", "MTS",
"dept_s","Sales")));
assertU(add(doc("id", "4","name", "dave", "title", "MTS",
"dept_s","Support", "dept_s","Engineering")));
assertU(add(doc("id", "5","name", "tina", "title", "VP",
"dept_s","Engineering")));
assertU(add(doc("id","10", "dept_id_s", "Engineering", "text","These
guys develop stuff")));
assertU(add(doc("id","11", "dept_id_s", "Marketing", "text","These guys
make you look good")));
assertU(add(doc("id","12", "dept_id_s", "Sales", "text","These guys
sell stuff")));
assertU(add(doc("id","13", "dept_id_s", "Support", "text","These guys
help customers")));
assertU(commit());
//***********
//This works as expected - the correct number of results are found
//***********
// find people that develop stuff
assertJQ(req("q","{!join from=dept_id_s to=dept_s}text:develop",
"fl","id")
,"/response=={'numFound':3,'start':0,'docs':[{'id':'1'},{'id':'4'},{'id':'5'}]}"
);
* //************
* // this fails - the response returned finds all three people - it
should only find John*
* // expected =/response=={"numFound":1,"start":0,"docs":[{"id":"1"}]}
*
* // response = {*
* // "responseHeader":{*
* // "status":0,*
* // "QTime":4},*
* // "response":{"numFound":3,"start":0,"docs":[*
* // {*
* // "id":"1"},*
* // {*
* // "id":"4"},*
* // {*
* // "id":"5"}]*
* // }}*
* //************
* // find people that develop stuff - but limit via filter query to a
name of "john"*
* assertJQ(req("q","{!join from=dept_id_s to=dept_s}text:develop",
"fl","id", "fq", "name:john")*
* ,"/response=={'numFound':1,'start':0,'docs':[{'id':'1'}]}"*
* );*
}
Interestingly, I know this worked at some point. I had a snapshot build in
my ivy cache from 10/2/2011 and it was working with that
build maven_artifacts/org/apache/solr/
solr/4.0-SNAPSHOT/solr-4.0-20111002.161157-1.pom"
Mike
Re: Solr Join query with fq not correctly filtering results?
Posted by Mike Hugo <mi...@piragua.com>.
Thanks Yonik!!
The join functionality is proving extremely useful for us in a specific use
case - we're really looking forward to join and other cool features coming
in Solr4!!
Mike
On Wed, Feb 1, 2012 at 3:30 PM, Yonik Seeley <yo...@lucidimagination.com>wrote:
> Thanks for your persistence in tracking this down Mike!
> I'm going to start looking into this now...
>
> -Yonik
> lucidimagination.com
>
>
>
> On Thu, Jan 26, 2012 at 11:06 PM, Mike Hugo <mi...@piragua.com> wrote:
> > I created issue https://issues.apache.org/jira/browse/SOLR-3062 for this
> > problem. I was able to track it down to something in this commit -
> > http://svn.apache.org/viewvc?view=revision&revision=1188624(LUCENE-1536:
> > Filters can now be applied down-low, if their DocIdSet implements a new
> > bits() method, returning all documents in a random access way
> > ) - before that commit the join / fq functionality works as expected /
> > documented on the wiki page. After that commit it's broken.
> >
> > Any assistance is greatly appreciated!
> >
> > Thanks,
> >
> > Mike
> >
> > On Thu, Jan 26, 2012 at 11:04 AM, Mike Hugo <mi...@piragua.com> wrote:
> >
> >> Hello,
> >>
> >> I'm trying out the Solr JOIN query functionality on trunk. I have the
> >> latest checkout, revision #1236272 - I did the following steps to get
> the
> >> example up and running:
> >>
> >> cd solr
> >> ant example
> >> java -jar start.jar
> >> cd exampledocs
> >> java -jar post.jar *.xml
> >>
> >> Then I tried a few of the sample queries on the wiki page
> >> http://wiki.apache.org/solr/Join. In particular, this is one that I'm
> >> interest in
> >>
> >> Find all manufacturer docs named "belkin", then join them against
> >>> (product) docs and filter that list to only products with a price less
> than
> >>> 12 dollars
> >>>
> >>>
> http://localhost:8983/solr/select?q={!join+from=id+to=manu_id_s}compName_s:Belkin&fq=price:%5B%2A+TO+12%5D
> <
> http://localhost:8983/solr/select?q=%7B!join+from=id+to=manu_id_s%7DcompName_s:Belkin&fq=price:%5B%2A+TO+12%5D
> >
> >>
> >>
> >> However, when I run that query, I get two results, one with a price of
> >> 19.95 and another with a price of 11.5 Because of the filter query, I'm
> >> only expecting to see one result - the one with a price of 11.99.
> >>
> >> I was also able to replicate this in a unit test added to
> >> org.apache.solr.TestJoin:
> >>
> >> @Test
> >> public void testJoin_withFilterQuery() throws Exception {
> >> assertU(add(doc("id", "1","name", "john", "title", "Director",
> >> "dept_s","Engineering")));
> >> assertU(add(doc("id", "2","name", "mark", "title", "VP",
> >> "dept_s","Marketing")));
> >> assertU(add(doc("id", "3","name", "nancy", "title", "MTS",
> >> "dept_s","Sales")));
> >> assertU(add(doc("id", "4","name", "dave", "title", "MTS",
> >> "dept_s","Support", "dept_s","Engineering")));
> >> assertU(add(doc("id", "5","name", "tina", "title", "VP",
> >> "dept_s","Engineering")));
> >>
> >> assertU(add(doc("id","10", "dept_id_s", "Engineering", "text","These
> >> guys develop stuff")));
> >> assertU(add(doc("id","11", "dept_id_s", "Marketing", "text","These
> >> guys make you look good")));
> >> assertU(add(doc("id","12", "dept_id_s", "Sales", "text","These guys
> >> sell stuff")));
> >> assertU(add(doc("id","13", "dept_id_s", "Support", "text","These
> guys
> >> help customers")));
> >>
> >> assertU(commit());
> >>
> >> //***********
> >> //This works as expected - the correct number of results are found
> >> //***********
> >> // find people that develop stuff
> >> assertJQ(req("q","{!join from=dept_id_s to=dept_s}text:develop",
> >> "fl","id")
> >>
> >>
> ,"/response=={'numFound':3,'start':0,'docs':[{'id':'1'},{'id':'4'},{'id':'5'}]}"
> >> );
> >>
> >> * //************
> >> * // this fails - the response returned finds all three people - it
> >> should only find John*
> >> * // expected
> >> =/response=={"numFound":1,"start":0,"docs":[{"id":"1"}]}*
> >> * // response = {*
> >> * // "responseHeader":{*
> >> * // "status":0,*
> >> * // "QTime":4},*
> >> * // "response":{"numFound":3,"start":0,"docs":[*
> >> * // {*
> >> * // "id":"1"},*
> >> * // {*
> >> * // "id":"4"},*
> >> * // {*
> >> * // "id":"5"}]*
> >> * // }}*
> >> * //************
> >> * // find people that develop stuff - but limit via filter query to a
> >> name of "john"*
> >> * assertJQ(req("q","{!join from=dept_id_s to=dept_s}text:develop",
> >> "fl","id", "fq", "name:john")*
> >> * ,"/response=={'numFound':1,'start':0,'docs':[{'id':'1'}]}"*
> >> * );*
> >>
> >> }
> >>
> >>
> >> Interestingly, I know this worked at some point. I had a snapshot build
> >> in my ivy cache from 10/2/2011 and it was working with that
> >> build maven_artifacts/org/apache/solr/
> >> solr/4.0-SNAPSHOT/solr-4.0-20111002.161157-1.pom"
> >>
> >>
> >> Mike
> >>
>
Re: Solr Join query with fq not correctly filtering results?
Posted by Yonik Seeley <yo...@lucidimagination.com>.
Thanks for your persistence in tracking this down Mike!
I'm going to start looking into this now...
-Yonik
lucidimagination.com
On Thu, Jan 26, 2012 at 11:06 PM, Mike Hugo <mi...@piragua.com> wrote:
> I created issue https://issues.apache.org/jira/browse/SOLR-3062 for this
> problem. I was able to track it down to something in this commit -
> http://svn.apache.org/viewvc?view=revision&revision=1188624 (LUCENE-1536:
> Filters can now be applied down-low, if their DocIdSet implements a new
> bits() method, returning all documents in a random access way
> ) - before that commit the join / fq functionality works as expected /
> documented on the wiki page. After that commit it's broken.
>
> Any assistance is greatly appreciated!
>
> Thanks,
>
> Mike
>
> On Thu, Jan 26, 2012 at 11:04 AM, Mike Hugo <mi...@piragua.com> wrote:
>
>> Hello,
>>
>> I'm trying out the Solr JOIN query functionality on trunk. I have the
>> latest checkout, revision #1236272 - I did the following steps to get the
>> example up and running:
>>
>> cd solr
>> ant example
>> java -jar start.jar
>> cd exampledocs
>> java -jar post.jar *.xml
>>
>> Then I tried a few of the sample queries on the wiki page
>> http://wiki.apache.org/solr/Join. In particular, this is one that I'm
>> interest in
>>
>> Find all manufacturer docs named "belkin", then join them against
>>> (product) docs and filter that list to only products with a price less than
>>> 12 dollars
>>>
>>> http://localhost:8983/solr/select?q={!join+from=id+to=manu_id_s}compName_s:Belkin&fq=price:%5B%2A+TO+12%5D<http://localhost:8983/solr/select?q=%7B!join+from=id+to=manu_id_s%7DcompName_s:Belkin&fq=price:%5B%2A+TO+12%5D>
>>
>>
>> However, when I run that query, I get two results, one with a price of
>> 19.95 and another with a price of 11.5 Because of the filter query, I'm
>> only expecting to see one result - the one with a price of 11.99.
>>
>> I was also able to replicate this in a unit test added to
>> org.apache.solr.TestJoin:
>>
>> @Test
>> public void testJoin_withFilterQuery() throws Exception {
>> assertU(add(doc("id", "1","name", "john", "title", "Director",
>> "dept_s","Engineering")));
>> assertU(add(doc("id", "2","name", "mark", "title", "VP",
>> "dept_s","Marketing")));
>> assertU(add(doc("id", "3","name", "nancy", "title", "MTS",
>> "dept_s","Sales")));
>> assertU(add(doc("id", "4","name", "dave", "title", "MTS",
>> "dept_s","Support", "dept_s","Engineering")));
>> assertU(add(doc("id", "5","name", "tina", "title", "VP",
>> "dept_s","Engineering")));
>>
>> assertU(add(doc("id","10", "dept_id_s", "Engineering", "text","These
>> guys develop stuff")));
>> assertU(add(doc("id","11", "dept_id_s", "Marketing", "text","These
>> guys make you look good")));
>> assertU(add(doc("id","12", "dept_id_s", "Sales", "text","These guys
>> sell stuff")));
>> assertU(add(doc("id","13", "dept_id_s", "Support", "text","These guys
>> help customers")));
>>
>> assertU(commit());
>>
>> //***********
>> //This works as expected - the correct number of results are found
>> //***********
>> // find people that develop stuff
>> assertJQ(req("q","{!join from=dept_id_s to=dept_s}text:develop",
>> "fl","id")
>>
>> ,"/response=={'numFound':3,'start':0,'docs':[{'id':'1'},{'id':'4'},{'id':'5'}]}"
>> );
>>
>> * //************
>> * // this fails - the response returned finds all three people - it
>> should only find John*
>> * // expected
>> =/response=={"numFound":1,"start":0,"docs":[{"id":"1"}]}*
>> * // response = {*
>> * // "responseHeader":{*
>> * // "status":0,*
>> * // "QTime":4},*
>> * // "response":{"numFound":3,"start":0,"docs":[*
>> * // {*
>> * // "id":"1"},*
>> * // {*
>> * // "id":"4"},*
>> * // {*
>> * // "id":"5"}]*
>> * // }}*
>> * //************
>> * // find people that develop stuff - but limit via filter query to a
>> name of "john"*
>> * assertJQ(req("q","{!join from=dept_id_s to=dept_s}text:develop",
>> "fl","id", "fq", "name:john")*
>> * ,"/response=={'numFound':1,'start':0,'docs':[{'id':'1'}]}"*
>> * );*
>>
>> }
>>
>>
>> Interestingly, I know this worked at some point. I had a snapshot build
>> in my ivy cache from 10/2/2011 and it was working with that
>> build maven_artifacts/org/apache/solr/
>> solr/4.0-SNAPSHOT/solr-4.0-20111002.161157-1.pom"
>>
>>
>> Mike
>>
Re: Solr Join query with fq not correctly filtering results?
Posted by Mike Hugo <mi...@piragua.com>.
I've been looking into this a bit further and am trying to figure out why
the FQ isn't getting applied.
Can anyone point me to a good spot in the code to start looking at how FQ
parameters are applied to query results in Solr4?
Thanks,
Mike
On Thu, Jan 26, 2012 at 10:06 PM, Mike Hugo <mi...@piragua.com> wrote:
> I created issue https://issues.apache.org/jira/browse/SOLR-3062 for this
> problem. I was able to track it down to something in this commit -
> http://svn.apache.org/viewvc?view=revision&revision=1188624 (LUCENE-1536:
> Filters can now be applied down-low, if their DocIdSet implements a new
> bits() method, returning all documents in a random access way
> ) - before that commit the join / fq functionality works as expected /
> documented on the wiki page. After that commit it's broken.
>
> Any assistance is greatly appreciated!
>
> Thanks,
>
> Mike
>
>
> On Thu, Jan 26, 2012 at 11:04 AM, Mike Hugo <mi...@piragua.com> wrote:
>
>> Hello,
>>
>> I'm trying out the Solr JOIN query functionality on trunk. I have the
>> latest checkout, revision #1236272 - I did the following steps to get the
>> example up and running:
>>
>> cd solr
>> ant example
>> java -jar start.jar
>> cd exampledocs
>> java -jar post.jar *.xml
>>
>> Then I tried a few of the sample queries on the wiki page
>> http://wiki.apache.org/solr/Join. In particular, this is one that I'm
>> interest in
>>
>> Find all manufacturer docs named "belkin", then join them against
>>> (product) docs and filter that list to only products with a price less than
>>> 12 dollars
>>>
>>> http://localhost:8983/solr/select?q={!join+from=id+to=manu_id_s}compName_s:Belkin&fq=price:%5B%2A+TO+12%5D<http://localhost:8983/solr/select?q=%7B!join+from=id+to=manu_id_s%7DcompName_s:Belkin&fq=price:%5B%2A+TO+12%5D>
>>
>>
>> However, when I run that query, I get two results, one with a price of
>> 19.95 and another with a price of 11.5 Because of the filter query, I'm
>> only expecting to see one result - the one with a price of 11.99.
>>
>> I was also able to replicate this in a unit test added to
>> org.apache.solr.TestJoin:
>>
>> @Test
>> public void testJoin_withFilterQuery() throws Exception {
>> assertU(add(doc("id", "1","name", "john", "title", "Director",
>> "dept_s","Engineering")));
>> assertU(add(doc("id", "2","name", "mark", "title", "VP",
>> "dept_s","Marketing")));
>> assertU(add(doc("id", "3","name", "nancy", "title", "MTS",
>> "dept_s","Sales")));
>> assertU(add(doc("id", "4","name", "dave", "title", "MTS",
>> "dept_s","Support", "dept_s","Engineering")));
>> assertU(add(doc("id", "5","name", "tina", "title", "VP",
>> "dept_s","Engineering")));
>>
>> assertU(add(doc("id","10", "dept_id_s", "Engineering", "text","These
>> guys develop stuff")));
>> assertU(add(doc("id","11", "dept_id_s", "Marketing", "text","These
>> guys make you look good")));
>> assertU(add(doc("id","12", "dept_id_s", "Sales", "text","These guys
>> sell stuff")));
>> assertU(add(doc("id","13", "dept_id_s", "Support", "text","These guys
>> help customers")));
>>
>> assertU(commit());
>>
>> //***********
>> //This works as expected - the correct number of results are found
>> //***********
>> // find people that develop stuff
>> assertJQ(req("q","{!join from=dept_id_s to=dept_s}text:develop",
>> "fl","id")
>>
>> ,"/response=={'numFound':3,'start':0,'docs':[{'id':'1'},{'id':'4'},{'id':'5'}]}"
>> );
>>
>> * //************
>> * // this fails - the response returned finds all three people - it
>> should only find John*
>> * // expected
>> =/response=={"numFound":1,"start":0,"docs":[{"id":"1"}]}*
>> * // response = {*
>> * // "responseHeader":{*
>> * // "status":0,*
>> * // "QTime":4},*
>> * // "response":{"numFound":3,"start":0,"docs":[*
>> * // {*
>> * // "id":"1"},*
>> * // {*
>> * // "id":"4"},*
>> * // {*
>> * // "id":"5"}]*
>> * // }}*
>> * //************
>> * // find people that develop stuff - but limit via filter query to a
>> name of "john"*
>> * assertJQ(req("q","{!join from=dept_id_s to=dept_s}text:develop",
>> "fl","id", "fq", "name:john")*
>> * ,"/response=={'numFound':1,'start':0,'docs':[{'id':'1'}]}"*
>> * );*
>>
>> }
>>
>>
>> Interestingly, I know this worked at some point. I had a snapshot build
>> in my ivy cache from 10/2/2011 and it was working with that
>> build maven_artifacts/org/apache/solr/
>> solr/4.0-SNAPSHOT/solr-4.0-20111002.161157-1.pom"
>>
>>
>> Mike
>>
>
>
Re: Solr Join query with fq not correctly filtering results?
Posted by Mike Hugo <mi...@piragua.com>.
I created issue https://issues.apache.org/jira/browse/SOLR-3062 for this
problem. I was able to track it down to something in this commit -
http://svn.apache.org/viewvc?view=revision&revision=1188624 (LUCENE-1536:
Filters can now be applied down-low, if their DocIdSet implements a new
bits() method, returning all documents in a random access way
) - before that commit the join / fq functionality works as expected /
documented on the wiki page. After that commit it's broken.
Any assistance is greatly appreciated!
Thanks,
Mike
On Thu, Jan 26, 2012 at 11:04 AM, Mike Hugo <mi...@piragua.com> wrote:
> Hello,
>
> I'm trying out the Solr JOIN query functionality on trunk. I have the
> latest checkout, revision #1236272 - I did the following steps to get the
> example up and running:
>
> cd solr
> ant example
> java -jar start.jar
> cd exampledocs
> java -jar post.jar *.xml
>
> Then I tried a few of the sample queries on the wiki page
> http://wiki.apache.org/solr/Join. In particular, this is one that I'm
> interest in
>
> Find all manufacturer docs named "belkin", then join them against
>> (product) docs and filter that list to only products with a price less than
>> 12 dollars
>>
>> http://localhost:8983/solr/select?q={!join+from=id+to=manu_id_s}compName_s:Belkin&fq=price:%5B%2A+TO+12%5D<http://localhost:8983/solr/select?q=%7B!join+from=id+to=manu_id_s%7DcompName_s:Belkin&fq=price:%5B%2A+TO+12%5D>
>
>
> However, when I run that query, I get two results, one with a price of
> 19.95 and another with a price of 11.5 Because of the filter query, I'm
> only expecting to see one result - the one with a price of 11.99.
>
> I was also able to replicate this in a unit test added to
> org.apache.solr.TestJoin:
>
> @Test
> public void testJoin_withFilterQuery() throws Exception {
> assertU(add(doc("id", "1","name", "john", "title", "Director",
> "dept_s","Engineering")));
> assertU(add(doc("id", "2","name", "mark", "title", "VP",
> "dept_s","Marketing")));
> assertU(add(doc("id", "3","name", "nancy", "title", "MTS",
> "dept_s","Sales")));
> assertU(add(doc("id", "4","name", "dave", "title", "MTS",
> "dept_s","Support", "dept_s","Engineering")));
> assertU(add(doc("id", "5","name", "tina", "title", "VP",
> "dept_s","Engineering")));
>
> assertU(add(doc("id","10", "dept_id_s", "Engineering", "text","These
> guys develop stuff")));
> assertU(add(doc("id","11", "dept_id_s", "Marketing", "text","These
> guys make you look good")));
> assertU(add(doc("id","12", "dept_id_s", "Sales", "text","These guys
> sell stuff")));
> assertU(add(doc("id","13", "dept_id_s", "Support", "text","These guys
> help customers")));
>
> assertU(commit());
>
> //***********
> //This works as expected - the correct number of results are found
> //***********
> // find people that develop stuff
> assertJQ(req("q","{!join from=dept_id_s to=dept_s}text:develop",
> "fl","id")
>
> ,"/response=={'numFound':3,'start':0,'docs':[{'id':'1'},{'id':'4'},{'id':'5'}]}"
> );
>
> * //************
> * // this fails - the response returned finds all three people - it
> should only find John*
> * // expected
> =/response=={"numFound":1,"start":0,"docs":[{"id":"1"}]}*
> * // response = {*
> * // "responseHeader":{*
> * // "status":0,*
> * // "QTime":4},*
> * // "response":{"numFound":3,"start":0,"docs":[*
> * // {*
> * // "id":"1"},*
> * // {*
> * // "id":"4"},*
> * // {*
> * // "id":"5"}]*
> * // }}*
> * //************
> * // find people that develop stuff - but limit via filter query to a
> name of "john"*
> * assertJQ(req("q","{!join from=dept_id_s to=dept_s}text:develop",
> "fl","id", "fq", "name:john")*
> * ,"/response=={'numFound':1,'start':0,'docs':[{'id':'1'}]}"*
> * );*
>
> }
>
>
> Interestingly, I know this worked at some point. I had a snapshot build
> in my ivy cache from 10/2/2011 and it was working with that
> build maven_artifacts/org/apache/solr/
> solr/4.0-SNAPSHOT/solr-4.0-20111002.161157-1.pom"
>
>
> Mike
>