You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Matt Stunfield <he...@gmail.com> on 2014/07/03 13:10:36 UTC
Query with many search items
Hi,
I'm new to Lucene/SOLR and I'm researching if SOLR would fit to our case
requirements. I would be very happy if You could help me :).
Environment:
We have a database storing some (mostly) text information. There are
elements containing multiple sections of information. Each section is
stored separately. This section content would be Solr document. Section
content can be any number of text paragraphs - usually it is less than 200
words. We have a dictionary of terms - currently 1600 items and there will
be more. Term might be a single word or phrase - 3 to 4 words.
Task: find if terms occur in single section. Found terms must be
distinguished. In query result there is (highlighted?) found terms
positions.
I know that Solr is perfect for searching items in large amounts of data,
but (1.) *will it cope (reasonably fast) with searching many items in a
small bit of information?*
Additional info: Solr instance will be accessed from one of .NET clients.
Query text and indexed text will be stemmed.
2. I suppose it isn’t possible to perform search with 1600 search items
like: “term_1” “term_2” … “term_N” at once, is it?
3. Is Lucene/Solr capable of performing simultaneous 1600 queries in
separate threads?
4. Can I store a query, so that its content won’t be stemmed again
unnecessarily? Terms dictionary won’t be changed frequently.
I would be glad for any help.
H.
Re: Query with many search items
Posted by Toke Eskildsen <te...@statsbiblioteket.dk>.
On Thu, 2014-07-03 at 13:10 +0200, Matt Stunfield wrote:
[Sections of text < 200 words, dictionary of terms & phrases]
> Task: find if terms occur in single section. Found terms must be
> distinguished. In query result there is (highlighted?) found terms
> positions.
I agree with Alexandre that this sounds like a Luwak-job.
> 2. I suppose it isn’t possible to perform search with 1600 search items
> like: “term_1” “term_2” … “term_N” at once, is it?
This should be possible with stock Solr, although it would not be a very
fast query. You might have to use POST instead of GET as you will have a
very big request.
At some point in time you had to ensure that the Solr config was set to
handle many boolean clauses
http://wiki.apache.org/solr/SolrConfigXml#The_Query_Section
but that does not seem to apply anymore
https://issues.apache.org/jira/browse/SOLR-4586
> 3. Is Lucene/Solr capable of performing simultaneous 1600 queries in
> separate threads?
Yes (hard limit is 10,000 with the supplied Jetty), although my guess is
that you would get better performance with less concurrent requests.
Anyway, that solution would mean that the results would have to be
merged in your end, which would require some work with merging the
highlight information for the matched terms.
- Toke Eskildsen, State and University Library, Denmark
Re: Query with many search items
Posted by Alexandre Rafalovitch <ar...@gmail.com>.
Luwak may be relevant here (https://github.com/flaxsearch/luwak)?
Or it may help to describe difference from Luwak's solution to further
fine-tune your requirement.
Regards,
Alex.
Personal website: http://www.outerthoughts.com/
Current project: http://www.solr-start.com/ - Accelerating your Solr proficiency
On Thu, Jul 3, 2014 at 6:10 PM, Matt Stunfield <he...@gmail.com> wrote:
> Hi,
>
> I'm new to Lucene/SOLR and I'm researching if SOLR would fit to our case
> requirements. I would be very happy if You could help me :).
>
> Environment:
> We have a database storing some (mostly) text information. There are
> elements containing multiple sections of information. Each section is
> stored separately. This section content would be Solr document. Section
> content can be any number of text paragraphs - usually it is less than 200
> words. We have a dictionary of terms - currently 1600 items and there will
> be more. Term might be a single word or phrase - 3 to 4 words.
>
> Task: find if terms occur in single section. Found terms must be
> distinguished. In query result there is (highlighted?) found terms
> positions.
>
> I know that Solr is perfect for searching items in large amounts of data,
> but (1.) *will it cope (reasonably fast) with searching many items in a
> small bit of information?*
>
> Additional info: Solr instance will be accessed from one of .NET clients.
> Query text and indexed text will be stemmed.
>
> 2. I suppose it isn’t possible to perform search with 1600 search items
> like: “term_1” “term_2” … “term_N” at once, is it?
> 3. Is Lucene/Solr capable of performing simultaneous 1600 queries in
> separate threads?
> 4. Can I store a query, so that its content won’t be stemmed again
> unnecessarily? Terms dictionary won’t be changed frequently.
>
> I would be glad for any help.
> H.