You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by henry cleland <hn...@gmail.com> on 2014/11/14 17:51:45 UTC
Restrict search to subset (a list of aprrox 40,000 ids from an
external service) of corpus
Hi guys,
How do I search only a subset of my corpus based on a large list of non
consecutive unique key ids (cannot do a range query).
Is there a way around doing this q=id:(id1 OR id2 OR id3 OR id4 ... OR
id40000 ) AND name:*
Also what is the limit of "OR"s i can apply on the query if that is the
only way out, i don't suppose it is infinity.
Thanks
Re: Restrict search to subset (a list of aprrox 40,000 ids from an
external service) of corpus
Posted by deviantcode <hn...@gmail.com>.
Thanks Eric, I will give that go and try to workout the number of ids i can
safely pass to {!terms}.
Also, is this to confirm that solr cannot execute such queries as those
three i listed earlier?
Thanks
--
View this message in context: http://lucene.472066.n3.nabble.com/Restrict-search-to-subset-a-list-of-aprrox-40-000-ids-from-an-external-service-of-corpus-tp4169210p4169477.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Restrict search to subset (a list of aprrox 40,000 ids from an external service) of corpus
Posted by Erik Hatcher <er...@gmail.com>.
&fq={!terms}... in theory ought to do the trick pretty performantly. How's that work for you?
Erik
> On Nov 17, 2014, at 08:33, deviantcode <hn...@gmail.com> wrote:
>
> Unfortunately no, the ids come back from an external service (spark) that
> performs complex aspects of the user query which as far is i can tell,
> cannot be easily done in solr.
> For example assuming, from the "family" schema described previously, the
> children docs record, "weight" "height", "year" and "month" fields.
> The queries to be performed are like the following:
> "find all children who have been gaining weight over the last 6 months"
> ,
> "find children who have recorded a 10% increase in weight over any two
> consecutive years",
> "find children of age X in the top 25% percentile of height from a
> particular school"
>
> The ids that come back match such queries. Is there a way to do these in
> solr?
>
> Thanks
>
>
> On Mon, Nov 17, 2014 at 12:09 PM, Alexandre Rafalovitch [via Lucene] <
> ml-node+s472066n4169455h5@n3.nabble.com> wrote:
>
>> On 17 November 2014 04:47, deviantcode <[hidden email]
>> <http://user/SendEmail.jtp?type=node&node=4169455&i=0>> wrote:
>>> Find all parent-ids from the list of 40,000 ids, (subset of all the
>> parent
>>> ids)
>>
>> And how do you calculate that subset? Is that absolutely not something
>> that translates into the rules that can be codified in Solr?
>>
>> Just passing 40,000 ids into Solr is going to be performance killer
>> all by itself. How often that subset changes? If not often, there
>> might be other ways to create a semi-permanent marker of some sort.
>>
>> Regards,
>> Alex.
>>
>>
>> Personal: http://www.outerthoughts.com/ and @arafalov
>> Solr resources and newsletter: http://www.solr-start.com/ and @solrstart
>> Solr popularizers community: https://www.linkedin.com/groups?gid=6713853
>>
>>
>> ------------------------------
>> If you reply to this email, your message will be added to the discussion
>> below:
>>
>> http://lucene.472066.n3.nabble.com/Restrict-search-to-subset-a-list-of-aprrox-40-000-ids-from-an-external-service-of-corpus-tp4169210p4169455.html
>> To unsubscribe from Restrict search to subset (a list of aprrox 40,000
>> ids from an external service) of corpus, click here
>> <http://lucene.472066.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=4169210&code=aG5vY2xlbGFuZEBnbWFpbC5jb218NDE2OTIxMHwxNDYwMjM1ODg1>
>> .
>> NAML
>> <http://lucene.472066.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
>
>
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Restrict-search-to-subset-a-list-of-aprrox-40-000-ids-from-an-external-service-of-corpus-tp4169210p4169466.html
> Sent from the Solr - User mailing list archive at Nabble.com.
Re: Restrict search to subset (a list of aprrox 40,000 ids from an
external service) of corpus
Posted by deviantcode <hn...@gmail.com>.
Unfortunately no, the ids come back from an external service (spark) that
performs complex aspects of the user query which as far is i can tell,
cannot be easily done in solr.
For example assuming, from the "family" schema described previously, the
children docs record, "weight" "height", "year" and "month" fields.
The queries to be performed are like the following:
"find all children who have been gaining weight over the last 6 months"
,
"find children who have recorded a 10% increase in weight over any two
consecutive years",
"find children of age X in the top 25% percentile of height from a
particular school"
The ids that come back match such queries. Is there a way to do these in
solr?
Thanks
On Mon, Nov 17, 2014 at 12:09 PM, Alexandre Rafalovitch [via Lucene] <
ml-node+s472066n4169455h5@n3.nabble.com> wrote:
> On 17 November 2014 04:47, deviantcode <[hidden email]
> <http://user/SendEmail.jtp?type=node&node=4169455&i=0>> wrote:
> > Find all parent-ids from the list of 40,000 ids, (subset of all the
> parent
> > ids)
>
> And how do you calculate that subset? Is that absolutely not something
> that translates into the rules that can be codified in Solr?
>
> Just passing 40,000 ids into Solr is going to be performance killer
> all by itself. How often that subset changes? If not often, there
> might be other ways to create a semi-permanent marker of some sort.
>
> Regards,
> Alex.
>
>
> Personal: http://www.outerthoughts.com/ and @arafalov
> Solr resources and newsletter: http://www.solr-start.com/ and @solrstart
> Solr popularizers community: https://www.linkedin.com/groups?gid=6713853
>
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
>
> http://lucene.472066.n3.nabble.com/Restrict-search-to-subset-a-list-of-aprrox-40-000-ids-from-an-external-service-of-corpus-tp4169210p4169455.html
> To unsubscribe from Restrict search to subset (a list of aprrox 40,000
> ids from an external service) of corpus, click here
> <http://lucene.472066.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=4169210&code=aG5vY2xlbGFuZEBnbWFpbC5jb218NDE2OTIxMHwxNDYwMjM1ODg1>
> .
> NAML
> <http://lucene.472066.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
--
View this message in context: http://lucene.472066.n3.nabble.com/Restrict-search-to-subset-a-list-of-aprrox-40-000-ids-from-an-external-service-of-corpus-tp4169210p4169466.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Restrict search to subset (a list of aprrox 40,000 ids from an
external service) of corpus
Posted by Alexandre Rafalovitch <ar...@gmail.com>.
On 17 November 2014 04:47, deviantcode <hn...@gmail.com> wrote:
> Find all parent-ids from the list of 40,000 ids, (subset of all the parent
> ids)
And how do you calculate that subset? Is that absolutely not something
that translates into the rules that can be codified in Solr?
Just passing 40,000 ids into Solr is going to be performance killer
all by itself. How often that subset changes? If not often, there
might be other ways to create a semi-permanent marker of some sort.
Regards,
Alex.
Personal: http://www.outerthoughts.com/ and @arafalov
Solr resources and newsletter: http://www.solr-start.com/ and @solrstart
Solr popularizers community: https://www.linkedin.com/groups?gid=6713853
Re: Restrict search to subset (a list of aprrox 40,000 ids from an
external service) of corpus
Posted by deviantcode <hn...@gmail.com>.
Hi Jurgen,
Thanks for the reply. There actually is a complex query which i
oversimplified in the post. as:
"q=id:(id1 OR id2 OR id3 OR id4 ... OR id40000 ) AND name:*" I am not
searching for docs with those ids,
i wish to restrict my actual search query to those ids similar to how an
'fq' works.
Perhaps a better example in a "family" schema =>
"Find all parent-ids from the list of 40,000 ids, (subset of all the parent
ids) that have children between ages "X" TO "Y" AND attend a schools in
locality "Z" where the parent themselves are bankers OR engineers AND earn
between 40-50k".
Its something like this im working with and only want to restirict to the
ids provided and not search the whole corpus.
Hope this clarifies it.
--
View this message in context: http://lucene.472066.n3.nabble.com/Restrict-search-to-subset-a-list-of-aprrox-40-000-ids-from-an-external-service-of-corpus-tp4169210p4169436.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Restrict search to subset (a list of aprrox 40,000 ids from an
external service) of corpus
Posted by "Jürgen Wagner (DVT)" <ju...@devoteam.com>.
Hi guy,
there's not much of a search operation here. Why not store the
documents in a key/value store and simply fetch them by matching ids?
Another approach: as there is no query, you could easily partition the
set of ids and fetch the results in multiple batches.
The maximum number of clauses should be 1024. You can set it to a higher
value using the respective method in
org.apache.lucene.search.BooleanQuery (I've never done that one before,
though).
Now, your mileage may vary. What is the idea behind this retrieval? You
really want to fetch objects by id? Check out MemcacheDB or Apache
Cassandra or Apache CouchDB, depending on your application and the type
of information you want to store.
Best regards,
--Jürgen
On 14.11.2014 17:51, henry cleland wrote:
> Hi guys,
> How do I search only a subset of my corpus based on a large list of non
> consecutive unique key ids (cannot do a range query).
> Is there a way around doing this q=id:(id1 OR id2 OR id3 OR id4 ... OR
> id40000 ) AND name:*
>
> Also what is the limit of "OR"s i can apply on the query if that is the
> only way out, i don't suppose it is infinity.
> Thanks
>
--
Mit freundlichen Grüßen/Kind regards/Cordialement vôtre/Atentamente/С
уважением
*i.A. Jürgen Wagner*
Head of Competence Center "Intelligence"
& Senior Cloud Consultant
Devoteam GmbH, Industriestr. 3, 70565 Stuttgart, Germany
Phone: +49 6151 868-8725, Fax: +49 711 13353-53, Mobile: +49 171 864 1543
E-Mail: juergen.wagner@devoteam.com
<ma...@devoteam.com>, URL: www.devoteam.de
<http://www.devoteam.de/>
------------------------------------------------------------------------
Managing Board: Jürgen Hatzipantelis (CEO)
Address of Record: 64331 Weiterstadt, Germany; Commercial Register:
Amtsgericht Darmstadt HRB 6450; Tax Number: DE 172 993 071
Re: Restrict search to subset (a list of aprrox 40,000 ids from an
external service) of corpus
Posted by deviantcode <hn...@gmail.com>.
Hi,
I am not sure i understand what you mean. Could you kindly elaborate
further?
Thanks
--
View this message in context: http://lucene.472066.n3.nabble.com/Restrict-search-to-subset-a-list-of-aprrox-40-000-ids-from-an-external-service-of-corpus-tp4169210p4169435.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Restrict search to subset (a list of aprrox 40,000 ids from an
external service) of corpus
Posted by Anurag Sharma <an...@gmail.com>.
Is it possible to add another integer dyanmicField to the selected doc ids?
If yes, further can add update incremental/same values to these docs now
search can be done to this subset using range/filter query.
On Fri, Nov 14, 2014 at 10:21 PM, henry cleland <hn...@gmail.com>
wrote:
> Hi guys,
> How do I search only a subset of my corpus based on a large list of non
> consecutive unique key ids (cannot do a range query).
> Is there a way around doing this q=id:(id1 OR id2 OR id3 OR id4 ... OR
> id40000 ) AND name:*
>
> Also what is the limit of "OR"s i can apply on the query if that is the
> only way out, i don't suppose it is infinity.
> Thanks
>
Re: Restrict search to subset (a list of aprrox 40,000 ids from an
external service) of corpus
Posted by deviantcode <hn...@gmail.com>.
Thanks will try with a POST
--
View this message in context: http://lucene.472066.n3.nabble.com/Restrict-search-to-subset-a-list-of-aprrox-40-000-ids-from-an-external-service-of-corpus-tp4169210p4169675.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Restrict search to subset (a list of aprrox 40,000 ids from an
external service) of corpus
Posted by Shawn Heisey <ap...@elyograg.org>.
On 11/14/2014 9:51 AM, henry cleland wrote:
> How do I search only a subset of my corpus based on a large list of non
> consecutive unique key ids (cannot do a range query).
> Is there a way around doing this q=id:(id1 OR id2 OR id3 OR id4 ... OR
> id40000 ) AND name:*
>
> Also what is the limit of "OR"s i can apply on the query if that is the
> only way out, i don't suppose it is infinity.
Very large boolean queries can be slow. You might want to use the
TermsQuery parser instead:
https://cwiki.apache.org/confluence/display/solr/Other+Parsers#OtherParsers-TermsQueryParser
For what you are actually trying to do, the number of OR clauses you can
have is determined by two things:
1) The maxBooleanClauses value in solrconfig.xml, which defaults to 1024.
2) The size of the query string allowed.
2a) If the request is a GET, the max http header size controls how long
your query (and the other URL paramters) can be. This almost always
defaults to 8K, or 8192 bytes, and includes other things, like the "GET"
command, the URL path, and the protocol (usually HTTP/1.1). This header
size is configurable in the servlet container config.
2b) If the request is a POST, the size limit in any recent Solr version
for the request will usually be 2MB. That limit is configurable in
solrconfig.xml.
I'm trying to get the maxBooleanClauses limit removed:
https://issues.apache.org/jira/browse/SOLR-4586
Thanks,
Shawn