You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Per Steffensen <pe...@gmail.com> on 2016/05/23 20:38:02 UTC
Using rows=-1 for "give me all"
Hi
Back when we used 4.4.0 I believe a query with rows=-1 returned all
matching documents. In 5.1.0 (the one we are using now) rows=-1 will
trigger a validation exception. If I remove the code that throws that
exception, it seems like rows=-1 behaves like rows=0. Has the support
for rows=-1 (give me all) been reintroduced in a release after 5.1.0? If
yes, which JIRA-ticket? If no, any plans to reintroduce it? Any good
reason for changing the rows=-1 behavior? Am I the only one that liked
it? :-)
Regards, Per Steffensen
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: Using rows=-1 for "give me all"
Posted by Ahmet Arslan <io...@yahoo.com.INVALID>.
Hi Steffensen,
Not sure about rows=-1, but retrieval engines are optimized to return top-N results.
However, there exists special commands for "give me all"
https://cwiki.apache.org/confluence/display/solr/Exporting+Result+Sets
Ahmet
On Monday, May 23, 2016 11:38 PM, Per Steffensen <pe...@gmail.com> wrote:
Hi
Back when we used 4.4.0 I believe a query with rows=-1 returned all
matching documents. In 5.1.0 (the one we are using now) rows=-1 will
trigger a validation exception. If I remove the code that throws that
exception, it seems like rows=-1 behaves like rows=0. Has the support
for rows=-1 (give me all) been reintroduced in a release after 5.1.0? If
yes, which JIRA-ticket? If no, any plans to reintroduce it? Any good
reason for changing the rows=-1 behavior? Am I the only one that liked
it? :-)
Regards, Per Steffensen
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
RE: Using rows=-1 for "give me all"
Posted by Uwe Schindler <uw...@thetaphi.de>.
Hi,
it could be that you hit some "bug", because of the missing check. Maybe the value overflowed and -1 was then interpreted like Integer.MAX_VALUE. This would make Lucene allocate a huuuuuuuuge array of 2 Giga * 4 bytes = 8 GiB for collecting the search results in a priority queue. And this is very bad memory-wise and expensive from computing power standpoint.
Uwe
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de
> -----Original Message-----
> From: Per Steffensen [mailto:persteff@gmail.com]
> Sent: Wednesday, May 25, 2016 9:12 AM
> To: dev@lucene.apache.org
> Subject: Re: Using rows=-1 for "give me all"
>
> On 25/05/16 01:11, Chris Hostetter wrote:
> > : > : Back when we used 4.4.0 I believe a query with rows=-1 returned all
> > : > matching
> >
> > : > Nope -- that's never been how rows=-1 behaved.
> > : Believe you are wrong. That was how it behaved in 4.4.0
> >
> > Nope -- not in a regular search it didn't...
> OK, thanks. I am confused now, because we have been using the "feature"
> a lot in the past - also for none-grouping queries. I will investigate
> more, but cannot promise I will come to a conclusion that is relevant here.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: Using rows=-1 for "give me all"
Posted by Per Steffensen <pe...@gmail.com>.
On 25/05/16 01:11, Chris Hostetter wrote:
> : > : Back when we used 4.4.0 I believe a query with rows=-1 returned all
> : > matching
>
> : > Nope -- that's never been how rows=-1 behaved.
> : Believe you are wrong. That was how it behaved in 4.4.0
>
> Nope -- not in a regular search it didn't...
OK, thanks. I am confused now, because we have been using the "feature"
a lot in the past - also for none-grouping queries. I will investigate
more, but cannot promise I will come to a conclusion that is relevant here.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: Using rows=-1 for "give me all"
Posted by Chris Hostetter <ho...@fucit.org>.
: > : Back when we used 4.4.0 I believe a query with rows=-1 returned all
: > matching
: > Nope -- that's never been how rows=-1 behaved.
: Believe you are wrong. That was how it behaved in 4.4.0
Nope -- not in a regular search it didn't...
hossman@tray:~/lucene/lucene_solr_4_4_0_tag/solr/example/exampledocs$ curl 'http://localhost:8983/solr/select?q=*:*&indent=true&rows=-1'
<?xml version="1.0" encoding="UTF-8"?>
<response>
<lst name="responseHeader">
<int name="status">0</int>
<int name="QTime">0</int>
<lst name="params">
<str name="q">*:*</str>
<str name="indent">true</str>
<str name="rows">-1</str>
</lst>
</lst>
<result name="response" numFound="32" start="0">
</result>
</response>
...your subsequent comments however suggest you may have been refering
solely to the *grouping* component -- but you didn't mention that in your
original email.
Based on the jira comments you linked to (SOLR-7254) i guess perhaps
that's how grouping worked in 4.4, but wether it was actually intended or
not and/or why that behavior may have changed since -- i have no idea.
And as for wether yonik did/didn't ever intend to support it in regular
searches: i have no idea -- If he says he planned on it then i guess maybe
he planned on it, but it's never actually been supported, and when
suggested in the past it's been argued against heavily as a "miss feature"
in the past (see SOLR-534) and discouraging people from expecting this to
ever work in general search results is one of the oldest FAQs we have...
https://wiki.apache.org/solr/FAQ#How_can_I_get_ALL_the_matching_documents_back.3F_..._How_can_I_return_an_unlimited_number_of_rows.3F
-Hoss
http://www.lucidworks.com/
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: Using rows=-1 for "give me all"
Posted by Per Steffensen <pe...@gmail.com>.
On 23/05/16 23:11, Chris Hostetter wrote:
> : Back when we used 4.4.0 I believe a query with rows=-1 returned all matching
> : documents. In 5.1.0 (the one we are using now) rows=-1 will trigger a
>
> Nope -- that's never been how rows=-1 behaved.
Believe you are wrong. That was how it behaved in 4.4.0
>
> The fact that it didn't return an error is some older versions of solr
> (and may have behaved similar to rows=0 in specific versions) was a bug of
> poor validation. but no "fetch all docs" behavior has ever been removed,
> it never existed in the first place other then in the special case support
> for the "/export" handler.
And I believe that was intended behavior. The javadoc for
SortSpec.getCount still says that -1 ought to do a "no cut off".
https://github.com/apache/lucene-solr/blob/branch_6_0/solr/core/src/java/org/apache/solr/search/SortSpec.java
Also see Yoniks comment around 23/Mar/15 here:
https://issues.apache.org/jira/browse/SOLR-7254
Regards, Per
Re: Using rows=-1 for "give me all"
Posted by Chris Hostetter <ho...@fucit.org>.
: Back when we used 4.4.0 I believe a query with rows=-1 returned all matching
: documents. In 5.1.0 (the one we are using now) rows=-1 will trigger a
Nope -- that's never been how rows=-1 behaved.
The fact that it didn't return an error is some older versions of solr
(and may have behaved similar to rows=0 in specific versions) was a bug of
poor validation. but no "fetch all docs" behavior has ever been removed,
it never existed in the first place other then in the special case support
for the "/export" handler.
: validation exception. If I remove the code that throws that exception, it
: seems like rows=-1 behaves like rows=0. Has the support for rows=-1 (give me
: all) been reintroduced in a release after 5.1.0? If yes, which JIRA-ticket? If
: no, any plans to reintroduce it? Any good reason for changing the rows=-1
: behavior? Am I the only one that liked it? :-)
:
: Regards, Per Steffensen
:
: ---------------------------------------------------------------------
: To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
: For additional commands, e-mail: dev-help@lucene.apache.org
:
:
-Hoss
http://www.lucidworks.com/
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org