You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Pavel Belenkovich <Pa...@exlibrisgroup.com> on 2014/05/21 19:07:39 UTC
multiple queries in single request
Hi,
I have list of 1000 values for some field which is sort of id (essentially unique between documents)
(let's say firstname_lastmane).
I need to get the document for each id (to know which document is for which id, not just list of responses).
Is there some support for multiple queries in single Solr request?
I saw old posts requesting that but don't know if it's been implemented yet.
There are 2 methods I can think of to achieve the result:
1 - trivial - make separate request per value. I think it's very inefficient.
2- Perform single request with OR on all values.
Then loop over the responses and match them to requested values.
This would also require making the field stored.
Can you propose better option?
thanx,
Pavel
RE: multiple queries in single request
Posted by Pavel Belenkovich <Pa...@exlibrisgroup.com>.
Great, thanx Mikhail, I"ll try that out.
regards,
Pavel.
-----Original Message-----
From: Mikhail Khludnev [mailto:mkhludnev@griddynamics.com]
Sent: Thursday, May 22, 2014 11:49
To: solr-user
Subject: Re: multiple queries in single request
Pavel,
I suppose the benchmark matters, anyway. (when benchmark 1. don't forget to enable http connection pooling on the client side) Regarding the 2. - passing the long disjunction is a well known road block (it's not a really natural problem for search engines). Most of approaches is described at http://lucene.472066.n3.nabble.com/Solr-large-boolean-filter-td4070747.htmlthere
is even some code at github!
If you see that stored fields is a slowpoke, but only if you do, you can consider https://wiki.apache.org/solr/FieldCollapsing , however it's also known as a bottleneck.
On Thu, May 22, 2014 at 12:00 PM, Pavel Belenkovich < Pavel.Belenkovich@exlibrisgroup.com> wrote:
> Hi Jack!
>
> Thanx for the response!
>
> So you say that using method 2 below (single request with ORs and
> sorting results in client) is better than method 1 (separate requests)?
>
> regards,
> Pavel.
>
>
> -----Original Message-----
> From: Jack Krupansky [mailto:jack@basetechnology.com]
> Sent: Thursday, May 22, 2014 01:26
> To: solr-user@lucene.apache.org
> Subject: Re: multiple queries in single request
>
> Nothing special for this use case.
>
> This seems to be a use case that I would call "bulk data retrieval -
> based on ID".
>
> I would suggest "batching" your requests - limit each request query
> to, say,
> 50 or 100 IDs.
>
> -- Jack Krupansky
>
> -----Original Message-----
> From: Pavel Belenkovich
> Sent: Wednesday, May 21, 2014 1:07 PM
> To: solr-user@lucene.apache.org
> Subject: multiple queries in single request
>
> Hi,
>
> I have list of 1000 values for some field which is sort of id
> (essentially unique between documents) (let's say firstname_lastmane).
> I need to get the document for each id (to know which document is for
> which id, not just list of responses).
>
> Is there some support for multiple queries in single Solr request?
> I saw old posts requesting that but don't know if it's been
> implemented yet.
>
> There are 2 methods I can think of to achieve the result:
> 1 - trivial - make separate request per value. I think it's very
> inefficient.
> 2- Perform single request with OR on all values.
> Then loop over the responses and match them to requested values.
> This would also require making the field stored.
>
> Can you propose better option?
>
> thanx,
> Pavel
>
>
--
Sincerely yours
Mikhail Khludnev
Principal Engineer,
Grid Dynamics
<http://www.griddynamics.com>
<mk...@griddynamics.com>
Re: multiple queries in single request
Posted by Mikhail Khludnev <mk...@griddynamics.com>.
Pavel,
I suppose the benchmark matters, anyway. (when benchmark 1. don't forget to
enable http connection pooling on the client side)
Regarding the 2. - passing the long disjunction is a well known road block
(it's not a really natural problem for search engines). Most of approaches
is described at
http://lucene.472066.n3.nabble.com/Solr-large-boolean-filter-td4070747.htmlthere
is even some code at github!
If you see that stored fields is a slowpoke, but only if you do, you can
consider https://wiki.apache.org/solr/FieldCollapsing , however it's also
known as a bottleneck.
On Thu, May 22, 2014 at 12:00 PM, Pavel Belenkovich <
Pavel.Belenkovich@exlibrisgroup.com> wrote:
> Hi Jack!
>
> Thanx for the response!
>
> So you say that using method 2 below (single request with ORs and sorting
> results in client) is better than method 1 (separate requests)?
>
> regards,
> Pavel.
>
>
> -----Original Message-----
> From: Jack Krupansky [mailto:jack@basetechnology.com]
> Sent: Thursday, May 22, 2014 01:26
> To: solr-user@lucene.apache.org
> Subject: Re: multiple queries in single request
>
> Nothing special for this use case.
>
> This seems to be a use case that I would call "bulk data retrieval - based
> on ID".
>
> I would suggest "batching" your requests - limit each request query to,
> say,
> 50 or 100 IDs.
>
> -- Jack Krupansky
>
> -----Original Message-----
> From: Pavel Belenkovich
> Sent: Wednesday, May 21, 2014 1:07 PM
> To: solr-user@lucene.apache.org
> Subject: multiple queries in single request
>
> Hi,
>
> I have list of 1000 values for some field which is sort of id (essentially
> unique between documents) (let's say firstname_lastmane).
> I need to get the document for each id (to know which document is for
> which id, not just list of responses).
>
> Is there some support for multiple queries in single Solr request?
> I saw old posts requesting that but don't know if it's been implemented
> yet.
>
> There are 2 methods I can think of to achieve the result:
> 1 - trivial - make separate request per value. I think it's very
> inefficient.
> 2- Perform single request with OR on all values.
> Then loop over the responses and match them to requested values.
> This would also require making the field stored.
>
> Can you propose better option?
>
> thanx,
> Pavel
>
>
--
Sincerely yours
Mikhail Khludnev
Principal Engineer,
Grid Dynamics
<http://www.griddynamics.com>
<mk...@griddynamics.com>
RE: multiple queries in single request
Posted by Pavel Belenkovich <Pa...@exlibrisgroup.com>.
Thanx Jack!
Could someone please explain what "batching" means in this case?
(Assuming I have just 1-2 documents per requested id)
regards,
Pavel.
-----Original Message-----
From: Jack Krupansky [mailto:jack@basetechnology.com]
Sent: Thursday, May 22, 2014 15:51
To: solr-user@lucene.apache.org
Subject: Re: multiple queries in single request
No, I was rejecting BOTH methods 1 and 2. I was suggesting a different method. I'll leave it to somebody else to describe the method so that it is easier to understand.
-- Jack Krupansky
-----Original Message-----
From: Pavel Belenkovich
Sent: Thursday, May 22, 2014 4:00 AM
To: solr-user@lucene.apache.org
Subject: RE: multiple queries in single request
Hi Jack!
Thanx for the response!
So you say that using method 2 below (single request with ORs and sorting results in client) is better than method 1 (separate requests)?
regards,
Pavel.
-----Original Message-----
From: Jack Krupansky [mailto:jack@basetechnology.com]
Sent: Thursday, May 22, 2014 01:26
To: solr-user@lucene.apache.org
Subject: Re: multiple queries in single request
Nothing special for this use case.
This seems to be a use case that I would call "bulk data retrieval - based on ID".
I would suggest "batching" your requests - limit each request query to, say,
50 or 100 IDs.
-- Jack Krupansky
-----Original Message-----
From: Pavel Belenkovich
Sent: Wednesday, May 21, 2014 1:07 PM
To: solr-user@lucene.apache.org
Subject: multiple queries in single request
Hi,
I have list of 1000 values for some field which is sort of id (essentially unique between documents) (let's say firstname_lastmane).
I need to get the document for each id (to know which document is for which id, not just list of responses).
Is there some support for multiple queries in single Solr request?
I saw old posts requesting that but don't know if it's been implemented yet.
There are 2 methods I can think of to achieve the result:
1 - trivial - make separate request per value. I think it's very inefficient.
2- Perform single request with OR on all values.
Then loop over the responses and match them to requested values.
This would also require making the field stored.
Can you propose better option?
thanx,
Pavel
Re: multiple queries in single request
Posted by Jack Krupansky <ja...@basetechnology.com>.
No, I was rejecting BOTH methods 1 and 2. I was suggesting a different
method. I'll leave it to somebody else to describe the method so that it is
easier to understand.
-- Jack Krupansky
-----Original Message-----
From: Pavel Belenkovich
Sent: Thursday, May 22, 2014 4:00 AM
To: solr-user@lucene.apache.org
Subject: RE: multiple queries in single request
Hi Jack!
Thanx for the response!
So you say that using method 2 below (single request with ORs and sorting
results in client) is better than method 1 (separate requests)?
regards,
Pavel.
-----Original Message-----
From: Jack Krupansky [mailto:jack@basetechnology.com]
Sent: Thursday, May 22, 2014 01:26
To: solr-user@lucene.apache.org
Subject: Re: multiple queries in single request
Nothing special for this use case.
This seems to be a use case that I would call "bulk data retrieval - based
on ID".
I would suggest "batching" your requests - limit each request query to, say,
50 or 100 IDs.
-- Jack Krupansky
-----Original Message-----
From: Pavel Belenkovich
Sent: Wednesday, May 21, 2014 1:07 PM
To: solr-user@lucene.apache.org
Subject: multiple queries in single request
Hi,
I have list of 1000 values for some field which is sort of id (essentially
unique between documents) (let's say firstname_lastmane).
I need to get the document for each id (to know which document is for which
id, not just list of responses).
Is there some support for multiple queries in single Solr request?
I saw old posts requesting that but don't know if it's been implemented yet.
There are 2 methods I can think of to achieve the result:
1 - trivial - make separate request per value. I think it's very
inefficient.
2- Perform single request with OR on all values.
Then loop over the responses and match them to requested values.
This would also require making the field stored.
Can you propose better option?
thanx,
Pavel
RE: multiple queries in single request
Posted by Pavel Belenkovich <Pa...@exlibrisgroup.com>.
Hi Jack!
Thanx for the response!
So you say that using method 2 below (single request with ORs and sorting results in client) is better than method 1 (separate requests)?
regards,
Pavel.
-----Original Message-----
From: Jack Krupansky [mailto:jack@basetechnology.com]
Sent: Thursday, May 22, 2014 01:26
To: solr-user@lucene.apache.org
Subject: Re: multiple queries in single request
Nothing special for this use case.
This seems to be a use case that I would call "bulk data retrieval - based on ID".
I would suggest "batching" your requests - limit each request query to, say,
50 or 100 IDs.
-- Jack Krupansky
-----Original Message-----
From: Pavel Belenkovich
Sent: Wednesday, May 21, 2014 1:07 PM
To: solr-user@lucene.apache.org
Subject: multiple queries in single request
Hi,
I have list of 1000 values for some field which is sort of id (essentially unique between documents) (let's say firstname_lastmane).
I need to get the document for each id (to know which document is for which id, not just list of responses).
Is there some support for multiple queries in single Solr request?
I saw old posts requesting that but don't know if it's been implemented yet.
There are 2 methods I can think of to achieve the result:
1 - trivial - make separate request per value. I think it's very inefficient.
2- Perform single request with OR on all values.
Then loop over the responses and match them to requested values.
This would also require making the field stored.
Can you propose better option?
thanx,
Pavel
Re: multiple queries in single request
Posted by Jack Krupansky <ja...@basetechnology.com>.
Nothing special for this use case.
This seems to be a use case that I would call "bulk data retrieval - based
on ID".
I would suggest "batching" your requests - limit each request query to, say,
50 or 100 IDs.
-- Jack Krupansky
-----Original Message-----
From: Pavel Belenkovich
Sent: Wednesday, May 21, 2014 1:07 PM
To: solr-user@lucene.apache.org
Subject: multiple queries in single request
Hi,
I have list of 1000 values for some field which is sort of id (essentially
unique between documents)
(let's say firstname_lastmane).
I need to get the document for each id (to know which document is for which
id, not just list of responses).
Is there some support for multiple queries in single Solr request?
I saw old posts requesting that but don't know if it's been implemented yet.
There are 2 methods I can think of to achieve the result:
1 - trivial - make separate request per value. I think it's very
inefficient.
2- Perform single request with OR on all values.
Then loop over the responses and match them to requested values.
This would also require making the field stored.
Can you propose better option?
thanx,
Pavel