Replacing text fields with numeric fields for speed

[Gora Mohanty <go...@srijan.in>]

Hi,

  One of the things that we were thinking of doing in order to
speed up results from Solr search is to convert fixed-text fields
(such as values from a drop-down) into numeric fields. The thinking
behind this was that searching through numeric values would be
faster than searching through text. However, I now feel that we
were barking up the wrong tree, as Lucene is probably not doing a
text search per se.

  From some experiments, I see only a small difference between a
text search on a field, and a numeric search on the corresponding
numeric field. This difference can probably be attributed to the
additional processing on the text field. Could someone clarify on
whether one can expect a difference in speed between searching
through a fixed-text field, and its numeric equivalent?

  I am aware of the benefit of numeric fields for range queries.

Regards,
Gora

Re: Replacing text fields with numeric fields for speed

[Gora Mohanty <go...@srijan.in>]

On Fri, 23 Jul 2010 14:33:54 +0200
Peter Karich <pe...@yahoo.de> wrote:

> Gora,
> 
> just for my interests:
> does apache bench sends different queries, or from the logs, or
> always the same query?
> If it would be always the same query the cache of solr will come
> and make the response time super small.

Yes, the way that things are set up currently the query is always
the same. My reasoning was that the effect of the Solr cache should
be the same for both numeric, and text fields. I am going to be
trying some more rigorous tests, such as turning off Solr caching,
and pre-warming the query before running the tests.

> I would like to find a tool or script where I can send my logfile
> to solr and measure some things ... because at the moment we are
> using fastbench and I would like to replace it ;-)

Not sure what fastbench is, but using Solr logs as a tool to
measure search times for typical searches is an interesting idea.
Hmm, we will also need to do that, so maybe we can compare notes on
this.

Regards,
Gora

Re: Replacing text fields with numeric fields for speed

[Peter Karich <pe...@yahoo.de>]

Gora,

just for my interests:
does apache bench sends different queries, or from the logs, or always
the same query?
If it would be always the same query the cache of solr will come and
make the response time super small.

I would like to find a tool or script where I can send my logfile to solr
and measure some things ... because at the moment we are using fastbench
and I would like to replace it ;-)

Regards,
Peter.

> On Fri, 23 Jul 2010 14:44:32 +0530
> Gora Mohanty <go...@srijan.in> wrote:
> [...]
>   
>>   From some experiments, I see only a small difference between a
>> text search on a field, and a numeric search on the corresponding
>> numeric field.
>>     
> [...]
>
> Well, I take that back. Running more rigorous tests with Apache
> Bench shows a difference of slightly over a factor of 2 between the
> median search time on the numeric field, and on the text field. The
> search on the numeric field is, of course, faster. That much
> of a difference puzzles me. Would someone knowledgeable about
> Lucene indexes care to comment?
>
> Regards,
> Gora
>   

Re: Replacing text fields with numeric fields for speed

[Gora Mohanty <go...@srijan.in>]

On Fri, 23 Jul 2010 14:44:32 +0530
Gora Mohanty <go...@srijan.in> wrote:
[...]
>   From some experiments, I see only a small difference between a
> text search on a field, and a numeric search on the corresponding
> numeric field.
[...]

Well, I take that back. Running more rigorous tests with Apache
Bench shows a difference of slightly over a factor of 2 between the
median search time on the numeric field, and on the text field. The
search on the numeric field is, of course, faster. That much
of a difference puzzles me. Would someone knowledgeable about
Lucene indexes care to comment?

Regards,
Gora