Severe QTime issues to do with updates

[Kaktu Chakarabati <ji...@gmail.com>]

Hey,
I've recently noticed that there is a very large spike in the QTime for
nodes serving queries, immediately after snappulling and snapinstalling.
The numbers i'm seeing there are obviously some kind of
lock-contention/concurrency issue, as I've monitored iostat/sar and its not
a disk IO
issue ( all the index is still mostly in the os caches, as is also
noticeable in the snappuller.log which runs very fast - the incremental
update to the
lucene index is minimal).
I'm using a strong machine (16GB 2xQuadCore, CentOS5) for this and from all
monitoring the CPU/Disk IO seems minimal through-out the day.
The update runs every 10 minutes, takes about 200seconds to complete (with
my current autoWarm settings)., and the index size
is about 20million documents (~6GB). queries/warmup involve mostly some
pre-fixed set of filters and facets.

I'm using a nightly build of solr from few months back ( nightly exported -
yonik - 2009-01-11 08:05:52 )  which should already have
the benefits of readonlyindexreader, NIOFS, UnInvertedIndex and
ConcurrentLRUCache (for filterCache).
I've read a related thread a guy called oleg posted (
http://mail-archives.apache.org/mod_mbox/lucene-solr-user/200901.mbox/%3C339777.57289.qm@web50307.mail.re2.yahoo.com%3E)
, but did not see the thread concluded with any definite conclusion.

Please advise!

Best regards,
-Chak

Re: Severe QTime issues to do with updates

[Rafał Kuć <ra...@alud.com.pl>]

Hello!

   First  of all I would suggest Solr user mailing list for a question
like this. Anyway, we were experiencing similar problem with the older
version of Solr 1.4 - in random situations, after index update on slaves
we  were  getting  response  times  from 1 to 20 seconds. After trying
everything  from  I/O  analysis  to  GC logging we updated SOLR to the
newest version available and that solved response time issues.

   Hope that helps.

-- 
Regards,
 Rafał Kuć


> Hey,
> I've recently noticed that there is a very large spike in the QTime for
> nodes serving queries, immediately after snappulling and snapinstalling.
> The numbers i'm seeing there are obviously some kind of
> lock-contention/concurrency issue, as I've monitored iostat/sar and its not
> a disk IO
> issue ( all the index is still mostly in the os caches, as is also
> noticeable in the snappuller.log which runs very fast - the incremental
> update to the
> lucene index is minimal).
> I'm using a strong machine (16GB 2xQuadCore, CentOS5) for this and from all
> monitoring the CPU/Disk IO seems minimal through-out the day.
> The update runs every 10 minutes, takes about 200seconds to complete (with
> my current autoWarm settings)., and the index size
> is about 20million documents (~6GB). queries/warmup involve mostly some
> pre-fixed set of filters and facets.

> I'm using a nightly build of solr from few months back ( nightly exported -
> yonik - 2009-01-11 08:05:52 )  which should already have
> the benefits of readonlyindexreader, NIOFS, UnInvertedIndex and
> ConcurrentLRUCache (for filterCache).
> I've read a related thread a guy called oleg posted (
> http://mail-archives.apache.org/mod_mbox/lucene-solr-user/200901.mbox/%3C339777.57289.qm@web50307.mail.re2.yahoo.com%3E)
> , but did not see the thread concluded with any definite conclusion.

> Please advise!

> Best regards,
> -Chak