You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by al...@aim.com on 2013/03/15 19:41:34 UTC
structure of solr index
Hi,
I wondered if solr searches on indexed fields only or on entire index? In more detail, let say I have fields id, title and content, all indexed, stored. Will a search send all these fields to memory or only indexed part of these fields?
Thanks.
Alex.
Re: structure of solr index
Posted by Jack Krupansky <ja...@basetechnology.com>.
Certainly if you are actually going to reference stored values they will add
on to the TOTAL time and memory, but still have zero impact on the actual
search time or memory for search. Searching for documents and returning
results are two separate steps. Highlighting and faceting are other separate
steps that can add significantly to the TOTAL request processing time, but
still have ZERO impact on the actual search time or memory for search
itself.
Add &debugQuery=true to your query request and there will be a "timing"
section that breaks down some of the major steps. The QueryComponent is the
actual search time.
Just to be clear, QTime is neither the actual search time nor the total
request processing time, but somewhere in the middle. QTime will include
actual search time (the QueryComponent in the timing section), highlighting,
faceting etc., but not include the time to output the actual response or
transmit it over the network or parse it on the receiving end. Nor does
QTime include the time it takes to format the original request, send it to
Solr, or for Solr to parse the raw Solr request.
Whether it is better to simply return document identifiers and get the
stored values elsewhere depends completely on the specific nature of the
application. But, Solr is optimized around the application having complete
control over what subset of fields should be returned on any given query.
So, Solr gives you the best of both worlds.
You could also look at a commercial product like DataStax Enterprise (DSE)
which does in fact combine Cassandra for storing values and Solr for
Indexing. It does it transparently so that the app issues fairly standard
Solr queries, but the stored values come from Cassandra under the hood
rather than from Lucene even though Lucene's indexing is still used for
search on all fields. Under the hood, DSE is reading only the document
unique key field value from the index and then using that key to access the
field/column values from Cassandra and then merging them into a standard
Solr response.
See:
http://3.datastax.com/
-- Jack Krupansky
-----Original Message-----
From: alxsss@aim.com
Sent: Monday, March 18, 2013 1:48 PM
To: solr-user@lucene.apache.org
Subject: Re: structure of solr index
---So,"search" time is in no way impacting by the existence or non-existence
of
stored values,
What about memory? Would it require to increase memeory in order to have the
same Qtime as in the case of indexed only fields?
For example in the case of indexed fields only index size is 5GB, average
Qtime is 0.1 sec and memory is 10G.
In case when the same fields are indexed and stored index size is 50GB. Will
the Qtime be 0.1s + time for extracting of stored fields?
Another scenario is to store fields in hbase or cassandra, have only indexed
fields in Solr and after getting id field from solr extract stored values
from hbase or cassandra. Will this setup be faster than the one with stored
fields in Solr?
Thanks.
Alex.
-----Original Message-----
From: Jack Krupansky <ja...@basetechnology.com>
To: solr-user <so...@lucene.apache.org>
Sent: Sat, Mar 16, 2013 9:53 am
Subject: Re: structure of solr index
"Search" depends only on the "index". But... returning field values for each
of the matched documents does require access to the "stored" values. So,
"search" time is in no way impacting by the existence or non-existence of
stored values, but total query processing time would of course include both
search time and the time to access and format the stored field values.
-- Jack Krupansky
-----Original Message-----
From: alxsss@aim.com
Sent: Saturday, March 16, 2013 12:48 PM
To: solr-user@lucene.apache.org
Subject: Re: structure of solr index
Hi,
So, will search time be the same for the case when fields are indexed only
vs the case when they are indexed and stored?
Thanks.
Alex.
-----Original Message-----
From: Otis Gospodnetic <ot...@gmail.com>
To: solr-user <so...@lucene.apache.org>
Sent: Fri, Mar 15, 2013 8:09 pm
Subject: Re: structure of solr index
Hi,
I think you are asking if the original/raw content of those fields will be
read. No, it won't, not for the search itself. If you want to
retrieve/return those fields then, of course, they will be read for the
documents being returned.
Otis
--
Solr & ElasticSearch Support
http://sematext.com/
On Fri, Mar 15, 2013 at 2:41 PM, <al...@aim.com> wrote:
> Hi,
>
> I wondered if solr searches on indexed fields only or on entire index? In
> more detail, let say I have fields id, title and content, all indexed,
> stored. Will a search send all these fields to memory or only indexed part
> of these fields?
>
> Thanks.
> Alex.
>
>
>
Re: structure of solr index
Posted by al...@aim.com.
---So,"search" time is in no way impacting by the existence or non-existence of
stored values,
What about memory? Would it require to increase memeory in order to have the same Qtime as in the case of indexed only fields?
For example in the case of indexed fields only index size is 5GB, average Qtime is 0.1 sec and memory is 10G.
In case when the same fields are indexed and stored index size is 50GB. Will the Qtime be 0.1s + time for extracting of stored fields?
Another scenario is to store fields in hbase or cassandra, have only indexed fields in Solr and after getting id field from solr extract stored values from hbase or cassandra. Will this setup be faster than the one with stored fields in Solr?
Thanks.
Alex.
-----Original Message-----
From: Jack Krupansky <ja...@basetechnology.com>
To: solr-user <so...@lucene.apache.org>
Sent: Sat, Mar 16, 2013 9:53 am
Subject: Re: structure of solr index
"Search" depends only on the "index". But... returning field values for each
of the matched documents does require access to the "stored" values. So,
"search" time is in no way impacting by the existence or non-existence of
stored values, but total query processing time would of course include both
search time and the time to access and format the stored field values.
-- Jack Krupansky
-----Original Message-----
From: alxsss@aim.com
Sent: Saturday, March 16, 2013 12:48 PM
To: solr-user@lucene.apache.org
Subject: Re: structure of solr index
Hi,
So, will search time be the same for the case when fields are indexed only
vs the case when they are indexed and stored?
Thanks.
Alex.
-----Original Message-----
From: Otis Gospodnetic <ot...@gmail.com>
To: solr-user <so...@lucene.apache.org>
Sent: Fri, Mar 15, 2013 8:09 pm
Subject: Re: structure of solr index
Hi,
I think you are asking if the original/raw content of those fields will be
read. No, it won't, not for the search itself. If you want to
retrieve/return those fields then, of course, they will be read for the
documents being returned.
Otis
--
Solr & ElasticSearch Support
http://sematext.com/
On Fri, Mar 15, 2013 at 2:41 PM, <al...@aim.com> wrote:
> Hi,
>
> I wondered if solr searches on indexed fields only or on entire index? In
> more detail, let say I have fields id, title and content, all indexed,
> stored. Will a search send all these fields to memory or only indexed part
> of these fields?
>
> Thanks.
> Alex.
>
>
>
Re: structure of solr index
Posted by Jack Krupansky <ja...@basetechnology.com>.
"Search" depends only on the "index". But... returning field values for each
of the matched documents does require access to the "stored" values. So,
"search" time is in no way impacting by the existence or non-existence of
stored values, but total query processing time would of course include both
search time and the time to access and format the stored field values.
-- Jack Krupansky
-----Original Message-----
From: alxsss@aim.com
Sent: Saturday, March 16, 2013 12:48 PM
To: solr-user@lucene.apache.org
Subject: Re: structure of solr index
Hi,
So, will search time be the same for the case when fields are indexed only
vs the case when they are indexed and stored?
Thanks.
Alex.
-----Original Message-----
From: Otis Gospodnetic <ot...@gmail.com>
To: solr-user <so...@lucene.apache.org>
Sent: Fri, Mar 15, 2013 8:09 pm
Subject: Re: structure of solr index
Hi,
I think you are asking if the original/raw content of those fields will be
read. No, it won't, not for the search itself. If you want to
retrieve/return those fields then, of course, they will be read for the
documents being returned.
Otis
--
Solr & ElasticSearch Support
http://sematext.com/
On Fri, Mar 15, 2013 at 2:41 PM, <al...@aim.com> wrote:
> Hi,
>
> I wondered if solr searches on indexed fields only or on entire index? In
> more detail, let say I have fields id, title and content, all indexed,
> stored. Will a search send all these fields to memory or only indexed part
> of these fields?
>
> Thanks.
> Alex.
>
>
>
Re: structure of solr index
Posted by al...@aim.com.
Hi,
So, will search time be the same for the case when fields are indexed only vs the case when they are indexed and stored?
Thanks.
Alex.
-----Original Message-----
From: Otis Gospodnetic <ot...@gmail.com>
To: solr-user <so...@lucene.apache.org>
Sent: Fri, Mar 15, 2013 8:09 pm
Subject: Re: structure of solr index
Hi,
I think you are asking if the original/raw content of those fields will be
read. No, it won't, not for the search itself. If you want to
retrieve/return those fields then, of course, they will be read for the
documents being returned.
Otis
--
Solr & ElasticSearch Support
http://sematext.com/
On Fri, Mar 15, 2013 at 2:41 PM, <al...@aim.com> wrote:
> Hi,
>
> I wondered if solr searches on indexed fields only or on entire index? In
> more detail, let say I have fields id, title and content, all indexed,
> stored. Will a search send all these fields to memory or only indexed part
> of these fields?
>
> Thanks.
> Alex.
>
>
>
Re: structure of solr index
Posted by Otis Gospodnetic <ot...@gmail.com>.
Hi,
I think you are asking if the original/raw content of those fields will be
read. No, it won't, not for the search itself. If you want to
retrieve/return those fields then, of course, they will be read for the
documents being returned.
Otis
--
Solr & ElasticSearch Support
http://sematext.com/
On Fri, Mar 15, 2013 at 2:41 PM, <al...@aim.com> wrote:
> Hi,
>
> I wondered if solr searches on indexed fields only or on entire index? In
> more detail, let say I have fields id, title and content, all indexed,
> stored. Will a search send all these fields to memory or only indexed part
> of these fields?
>
> Thanks.
> Alex.
>
>
>