question on Hits.doc

[ro...@xemaps.com]

Hey guys,

We were noticing some speed problems on our searches and after adding some
debug statements to the lucene source code, we have determined that the
Hits.doc(x) is the problem.  (BTW, we are using Lucene 1.2 [with plans to
upgrade]).  It seems that retrieving the actual Document from the search is
very slow.

We think it might be our "Message" field which stores a huge amount of text. 
We are currently running a test in which we won't "store" the "Message" field,
however, I was wondering if any of you guys would know if that would be the
reason why we're having the performance problems?  If so, could anyone also
please explain it?  It seemed that we weren't having these performance
problems before.  Has anyone else experienced this?  Our environment is NT 4,
JDK 1.4.2, and PIIIs.

I know that for large text fields, storing the field is not a good practice,
however, it held certain conveniences for us that I hope to not get rid of.

Roy.

---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org

Re: question on Hits.doc

[Otis Gospodnetic <ot...@yahoo.com>]

Hello Roy,

This sounds normal.  When you pull a Document from Hits, you are really
pulling it from the disk.  All fields are read from disk at that time
(i.e. no lazy loading of fields), so if you have large text fields,
this is going to result in a lot of disk IO.  You could try running
vmstat or sar (I'm assuming you are using a UNIX flavour) and look at
the bi/bo (really just bo) column (bo = blocks out -- data read from
disks).

There is not much you can do.  If you don't have to store the field,
they will probably help.  Some people are working on adding support for
field compression, so maybe that will help.

Otis

--- roy-lucene-user@xemaps.com wrote:

> Hey guys,
> 
> We were noticing some speed problems on our searches and after adding
> some
> debug statements to the lucene source code, we have determined that
> the
> Hits.doc(x) is the problem.  (BTW, we are using Lucene 1.2 [with
> plans to
> upgrade]).  It seems that retrieving the actual Document from the
> search is
> very slow.
> 
> We think it might be our "Message" field which stores a huge amount
> of text. 
> We are currently running a test in which we won't "store" the
> "Message" field,
> however, I was wondering if any of you guys would know if that would
> be the
> reason why we're having the performance problems?  If so, could
> anyone also
> please explain it?  It seemed that we weren't having these
> performance
> problems before.  Has anyone else experienced this?  Our environment
> is NT 4,
> JDK 1.4.2, and PIIIs.
> 
> I know that for large text fields, storing the field is not a good
> practice,
> however, it held certain conveniences for us that I hope to not get
> rid of.
> 
> Roy.
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: lucene-user-help@jakarta.apache.org
> 
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org