You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by shb <su...@gmail.com> on 2009/07/23 04:45:15 UTC
how to get all the docIds in the search result?
When I use
SolrQuery query = new SolrQuery();
query.set("q", "issn:0002-9505");
query.setRows(10);
QueryResponse response = server.query(query);
I only can get the 10 ids in the response.
How can i get all the docIds in the search result? Thanks.
Re: how to get all the docIds in the search result?
Posted by Chris Hostetter <ho...@fucit.org>.
: Here id is indeed the uniqueKey of a document.
: I want to get all the ids for some other useage.
http://people.apache.org/~hossman/#xyproblem
XY Problem
Your question appears to be an "XY Problem" ... that is: you are dealing
with "X", you are assuming "Y" will help you, and you are asking about "Y"
without giving more details about the "X" so that we can understand the
full issue. Perhaps the best solution doesn't involve "Y" at all?
See Also: http://www.perlmonks.org/index.pl?node_id=542341
-Hoss
Re: how to get all the docIds in the search result?
Posted by Otis Gospodnetic <ot...@yahoo.com>.
You could pull the ID directly from the Lucene index, that may be a little faster.
You can also use Lucene's TermEnum to get to this.
And you should make sure that "id" field is the first field in your documents (when you index them).
But no matter what you do, this will not be subsecond for non-trivial indices - it's the equivalent of a full table scan in RDBMS world.
Otis
--
Sematext is hiring: http://sematext.com/about/jobs.html?mls
Lucene, Solr, Nutch, Katta, Hadoop, HBase, UIMA, NLP, NER, IR
----- Original Message ----
> From: shb <su...@gmail.com>
> To: solr-user@lucene.apache.org
> Sent: Thursday, July 23, 2009 5:35:29 AM
> Subject: Re: how to get all the docIds in the search result?
>
> I have tried the following code:
> query.setRows(Integer.MAX_VALUE);
> query.setFields("id");
>
> when it return 1000,000 records, it will take about 22s.
> This is very slow. Is there any other way?
>
>
> 2009/7/23 Toby Cole
>
> > Have you tried limiting the fields that you're requesting to just the ID?
> > Something along the line of:
> >
> > query.setRows(Integer.MAX_VALUE);
> > query.setFields("id");
> >
> > Might speed the query up a little.
> >
> >
> > On 23 Jul 2009, at 09:11, shb wrote:
> >
> > Here id is indeed the uniqueKey of a document.
> >> I want to get all the ids for some other useage.
> >>
> >>
> >> 2009/7/23 Shalin Shekhar Mangar
> >>
> >> On Thu, Jul 23, 2009 at 1:09 PM, shb wrote:
> >>>
> >>> if I use query.setRows(Integer.MAX_VALUE);
> >>>> the query will become very slow, because searcher will go
> >>>> to fetch the filed value in the index for all the returned
> >>>> document.
> >>>>
> >>>> So if I set query.setRows(10), is there any other ways to
> >>>> get all the ids? thanks
> >>>>
> >>>>
> >>> You should fetch as many rows as you need and not more. Why do you need
> >>> all
> >>> the ids? I'm assuming that by id you mean the uniqueKey of a document.
> >>>
> >>> --
> >>> Regards,
> >>> Shalin Shekhar Mangar.
> >>>
> >>>
> > --
> >
> > Toby Cole
> > Software Engineer, Semantico Limited
> >
> > Registered in England and Wales no. 03841410, VAT no. GB-744614334.
> > Registered office Lees House, 21-23 Dyke Road, Brighton BN1 3FE, UK.
> >
> > Check out all our latest news and thinking on the Discovery blog
> > http://blogs.semantico.com/discovery-blog/
> >
> >
Re: how to get all the docIds in the search result?
Posted by Otis Gospodnetic <ot...@yahoo.com>.
And if I may add another thing - if you are using Solr in this fashion, have a look at your caches, esp. document cache. If your "queries" of this type are repeated, you may benefit from large cache. Or, if they are not, you may completely disable some caches.
Otis
--
Sematext is hiring: http://sematext.com/about/jobs.html?mls
Lucene, Solr, Nutch, Katta, Hadoop, HBase, UIMA, NLP, NER, IR
----- Original Message ----
> From: Erik Hatcher <er...@ehatchersolutions.com>
> To: solr-user@lucene.apache.org
> Sent: Thursday, July 23, 2009 11:15:45 AM
> Subject: Re: how to get all the docIds in the search result?
>
> Rather than trying to get all document id's in one call to Solr, consider paging
> through the results. Set rows=1000 or probably larger, then check the numFound
> and continue making requests to Solr incrementing start parameter accordingly
> until done.
>
> Erik
>
> On Jul 23, 2009, at 5:35 AM, shb wrote:
>
> > I have tried the following code:
> > query.setRows(Integer.MAX_VALUE);
> > query.setFields("id");
> >
> > when it return 1000,000 records, it will take about 22s.
> > This is very slow. Is there any other way?
> >
> >
> > 2009/7/23 Toby Cole
> >
> >> Have you tried limiting the fields that you're requesting to just the ID?
> >> Something along the line of:
> >>
> >> query.setRows(Integer.MAX_VALUE);
> >> query.setFields("id");
> >>
> >> Might speed the query up a little.
> >>
> >>
> >> On 23 Jul 2009, at 09:11, shb wrote:
> >>
> >> Here id is indeed the uniqueKey of a document.
> >>> I want to get all the ids for some other useage.
> >>>
> >>>
> >>> 2009/7/23 Shalin Shekhar Mangar
> >>>
> >>> On Thu, Jul 23, 2009 at 1:09 PM, shb wrote:
> >>>>
> >>>> if I use query.setRows(Integer.MAX_VALUE);
> >>>>> the query will become very slow, because searcher will go
> >>>>> to fetch the filed value in the index for all the returned
> >>>>> document.
> >>>>>
> >>>>> So if I set query.setRows(10), is there any other ways to
> >>>>> get all the ids? thanks
> >>>>>
> >>>>>
> >>>> You should fetch as many rows as you need and not more. Why do you need
> >>>> all
> >>>> the ids? I'm assuming that by id you mean the uniqueKey of a document.
> >>>>
> >>>> --
> >>>> Regards,
> >>>> Shalin Shekhar Mangar.
> >>>>
> >>>>
> >> --
> >>
> >> Toby Cole
> >> Software Engineer, Semantico Limited
> >>
> >> Registered in England and Wales no. 03841410, VAT no. GB-744614334.
> >> Registered office Lees House, 21-23 Dyke Road, Brighton BN1 3FE, UK.
> >>
> >> Check out all our latest news and thinking on the Discovery blog
> >> http://blogs.semantico.com/discovery-blog/
> >>
> >>
Re: how to get all the docIds in the search result?
Posted by Erik Hatcher <er...@ehatchersolutions.com>.
Rather than trying to get all document id's in one call to Solr,
consider paging through the results. Set rows=1000 or probably
larger, then check the numFound and continue making requests to Solr
incrementing start parameter accordingly until done.
Erik
On Jul 23, 2009, at 5:35 AM, shb wrote:
> I have tried the following code:
> query.setRows(Integer.MAX_VALUE);
> query.setFields("id");
>
> when it return 1000,000 records, it will take about 22s.
> This is very slow. Is there any other way?
>
>
> 2009/7/23 Toby Cole <to...@semantico.com>
>
>> Have you tried limiting the fields that you're requesting to just
>> the ID?
>> Something along the line of:
>>
>> query.setRows(Integer.MAX_VALUE);
>> query.setFields("id");
>>
>> Might speed the query up a little.
>>
>>
>> On 23 Jul 2009, at 09:11, shb wrote:
>>
>> Here id is indeed the uniqueKey of a document.
>>> I want to get all the ids for some other useage.
>>>
>>>
>>> 2009/7/23 Shalin Shekhar Mangar <sh...@gmail.com>
>>>
>>> On Thu, Jul 23, 2009 at 1:09 PM, shb <su...@gmail.com> wrote:
>>>>
>>>> if I use query.setRows(Integer.MAX_VALUE);
>>>>> the query will become very slow, because searcher will go
>>>>> to fetch the filed value in the index for all the returned
>>>>> document.
>>>>>
>>>>> So if I set query.setRows(10), is there any other ways to
>>>>> get all the ids? thanks
>>>>>
>>>>>
>>>> You should fetch as many rows as you need and not more. Why do
>>>> you need
>>>> all
>>>> the ids? I'm assuming that by id you mean the uniqueKey of a
>>>> document.
>>>>
>>>> --
>>>> Regards,
>>>> Shalin Shekhar Mangar.
>>>>
>>>>
>> --
>>
>> Toby Cole
>> Software Engineer, Semantico Limited
>> <to...@semantico.com> <tel:+44 1273 358 238>
>> Registered in England and Wales no. 03841410, VAT no. GB-744614334.
>> Registered office Lees House, 21-23 Dyke Road, Brighton BN1 3FE, UK.
>>
>> Check out all our latest news and thinking on the Discovery blog
>> http://blogs.semantico.com/discovery-blog/
>>
>>
Re: how to get all the docIds in the search result?
Posted by Toby Cole <to...@semantico.com>.
That's the only way I can think of doing it through Solr.
What is the configuration of the handler you're calling? It could be
that highlighting or faceting are turned on and slowing down your query.
Toby.
On 23 Jul 2009, at 10:35, shb wrote:
> I have tried the following code:
> query.setRows(Integer.MAX_VALUE);
> query.setFields("id");
>
> when it return 1000,000 records, it will take about 22s.
> This is very slow. Is there any other way?
>
>
> 2009/7/23 Toby Cole <to...@semantico.com>
>
>> Have you tried limiting the fields that you're requesting to just
>> the ID?
>> Something along the line of:
>>
>> query.setRows(Integer.MAX_VALUE);
>> query.setFields("id");
>>
>> Might speed the query up a little.
>>
>>
>> On 23 Jul 2009, at 09:11, shb wrote:
>>
>> Here id is indeed the uniqueKey of a document.
>>> I want to get all the ids for some other useage.
>>>
>>>
>>> 2009/7/23 Shalin Shekhar Mangar <sh...@gmail.com>
>>>
>>> On Thu, Jul 23, 2009 at 1:09 PM, shb <su...@gmail.com> wrote:
>>>>
>>>> if I use query.setRows(Integer.MAX_VALUE);
>>>>> the query will become very slow, because searcher will go
>>>>> to fetch the filed value in the index for all the returned
>>>>> document.
>>>>>
>>>>> So if I set query.setRows(10), is there any other ways to
>>>>> get all the ids? thanks
>>>>>
>>>>>
>>>> You should fetch as many rows as you need and not more. Why do
>>>> you need
>>>> all
>>>> the ids? I'm assuming that by id you mean the uniqueKey of a
>>>> document.
>>>>
>>>> --
>>>> Regards,
>>>> Shalin Shekhar Mangar.
>>>>
>>>>
>> --
>>
>> Toby Cole
>> Software Engineer, Semantico Limited
>> <to...@semantico.com> <tel:+44 1273 358 238>
>> Registered in England and Wales no. 03841410, VAT no. GB-744614334.
>> Registered office Lees House, 21-23 Dyke Road, Brighton BN1 3FE, UK.
>>
>> Check out all our latest news and thinking on the Discovery blog
>> http://blogs.semantico.com/discovery-blog/
Re: how to get all the docIds in the search result?
Posted by shb <su...@gmail.com>.
I have tried the following code:
query.setRows(Integer.MAX_VALUE);
query.setFields("id");
when it return 1000,000 records, it will take about 22s.
This is very slow. Is there any other way?
2009/7/23 Toby Cole <to...@semantico.com>
> Have you tried limiting the fields that you're requesting to just the ID?
> Something along the line of:
>
> query.setRows(Integer.MAX_VALUE);
> query.setFields("id");
>
> Might speed the query up a little.
>
>
> On 23 Jul 2009, at 09:11, shb wrote:
>
> Here id is indeed the uniqueKey of a document.
>> I want to get all the ids for some other useage.
>>
>>
>> 2009/7/23 Shalin Shekhar Mangar <sh...@gmail.com>
>>
>> On Thu, Jul 23, 2009 at 1:09 PM, shb <su...@gmail.com> wrote:
>>>
>>> if I use query.setRows(Integer.MAX_VALUE);
>>>> the query will become very slow, because searcher will go
>>>> to fetch the filed value in the index for all the returned
>>>> document.
>>>>
>>>> So if I set query.setRows(10), is there any other ways to
>>>> get all the ids? thanks
>>>>
>>>>
>>> You should fetch as many rows as you need and not more. Why do you need
>>> all
>>> the ids? I'm assuming that by id you mean the uniqueKey of a document.
>>>
>>> --
>>> Regards,
>>> Shalin Shekhar Mangar.
>>>
>>>
> --
>
> Toby Cole
> Software Engineer, Semantico Limited
> <to...@semantico.com> <tel:+44 1273 358 238>
> Registered in England and Wales no. 03841410, VAT no. GB-744614334.
> Registered office Lees House, 21-23 Dyke Road, Brighton BN1 3FE, UK.
>
> Check out all our latest news and thinking on the Discovery blog
> http://blogs.semantico.com/discovery-blog/
>
>
Re: how to get all the docIds in the search result?
Posted by Toby Cole <to...@semantico.com>.
Have you tried limiting the fields that you're requesting to just the
ID?
Something along the line of:
query.setRows(Integer.MAX_VALUE);
query.setFields("id");
Might speed the query up a little.
On 23 Jul 2009, at 09:11, shb wrote:
> Here id is indeed the uniqueKey of a document.
> I want to get all the ids for some other useage.
>
>
> 2009/7/23 Shalin Shekhar Mangar <sh...@gmail.com>
>
>> On Thu, Jul 23, 2009 at 1:09 PM, shb <su...@gmail.com> wrote:
>>
>>> if I use query.setRows(Integer.MAX_VALUE);
>>> the query will become very slow, because searcher will go
>>> to fetch the filed value in the index for all the returned
>>> document.
>>>
>>> So if I set query.setRows(10), is there any other ways to
>>> get all the ids? thanks
>>>
>>
>> You should fetch as many rows as you need and not more. Why do you
>> need all
>> the ids? I'm assuming that by id you mean the uniqueKey of a
>> document.
>>
>> --
>> Regards,
>> Shalin Shekhar Mangar.
>>
--
Toby Cole
Software Engineer, Semantico Limited
<to...@semantico.com> <tel:+44 1273 358 238>
Registered in England and Wales no. 03841410, VAT no. GB-744614334.
Registered office Lees House, 21-23 Dyke Road, Brighton BN1 3FE, UK.
Check out all our latest news and thinking on the Discovery blog
http://blogs.semantico.com/discovery-blog/
Re: how to get all the docIds in the search result?
Posted by shb <su...@gmail.com>.
Here id is indeed the uniqueKey of a document.
I want to get all the ids for some other useage.
2009/7/23 Shalin Shekhar Mangar <sh...@gmail.com>
> On Thu, Jul 23, 2009 at 1:09 PM, shb <su...@gmail.com> wrote:
>
> > if I use query.setRows(Integer.MAX_VALUE);
> > the query will become very slow, because searcher will go
> > to fetch the filed value in the index for all the returned
> > document.
> >
> > So if I set query.setRows(10), is there any other ways to
> > get all the ids? thanks
> >
>
> You should fetch as many rows as you need and not more. Why do you need all
> the ids? I'm assuming that by id you mean the uniqueKey of a document.
>
> --
> Regards,
> Shalin Shekhar Mangar.
>
Re: how to get all the docIds in the search result?
Posted by Shalin Shekhar Mangar <sh...@gmail.com>.
On Thu, Jul 23, 2009 at 1:09 PM, shb <su...@gmail.com> wrote:
> if I use query.setRows(Integer.MAX_VALUE);
> the query will become very slow, because searcher will go
> to fetch the filed value in the index for all the returned
> document.
>
> So if I set query.setRows(10), is there any other ways to
> get all the ids? thanks
>
You should fetch as many rows as you need and not more. Why do you need all
the ids? I'm assuming that by id you mean the uniqueKey of a document.
--
Regards,
Shalin Shekhar Mangar.
Re: how to get all the docIds in the search result?
Posted by shb <su...@gmail.com>.
if I use query.setRows(Integer.MAX_VALUE);
the query will become very slow, because searcher will go
to fetch the filed value in the index for all the returned
document.
So if I set query.setRows(10), is there any other ways to
get all the ids? thanks
2009/7/23 Avlesh Singh <av...@gmail.com>
> query.setRows(Integer.MAX_VALUE);
>
> Cheers
> Avlesh
>
> On Thu, Jul 23, 2009 at 8:15 AM, shb <su...@gmail.com> wrote:
>
> > When I use
> > SolrQuery query = new SolrQuery();
> > query.set("q", "issn:0002-9505");
> > query.setRows(10);
> > QueryResponse response = server.query(query);
> > I only can get the 10 ids in the response.
> >
> > How can i get all the docIds in the search result? Thanks.
> >
>
Re: how to get all the docIds in the search result?
Posted by Avlesh Singh <av...@gmail.com>.
query.setRows(Integer.MAX_VALUE);
Cheers
Avlesh
On Thu, Jul 23, 2009 at 8:15 AM, shb <su...@gmail.com> wrote:
> When I use
> SolrQuery query = new SolrQuery();
> query.set("q", "issn:0002-9505");
> query.setRows(10);
> QueryResponse response = server.query(query);
> I only can get the 10 ids in the response.
>
> How can i get all the docIds in the search result? Thanks.
>