You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by "michael.boom" <my...@yahoo.com> on 2013/10/18 17:37:20 UTC
Solr timeout after reboot
I have a SolrCloud environment with 4 shards, each having a replica and a
leader. The index size is about 70M docs and 60Gb, running with Jetty +
Zookeeper, on 2 EC2 instances, each with 4CPUs and 15G RAM.
I'm using SolrMeter for stress testing.
If I restart Jetty and then try to use SolrMeter to bomb an instance with
queries, using a query per minute rate of 3000 then that solr instance
somehow timesout and I need to restart it again.
If instead of using 3000 qpm i startup slowly with 200 for a minute or two,
then 1800 and then 3000 everything is good.
I assume this happens because Solr is not warmed up.
What settings could I tweak so that Solr doesn't time out anymore when
getting many requests? Is there a way to limit how many req it can serve?
-----
Thanks,
Michael
--
View this message in context: http://lucene.472066.n3.nabble.com/Solr-timeout-after-reboot-tp4096408.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr timeout after reboot
Posted by Erick Erickson <er...@gmail.com>.
Have you considered just putting some killer queries in the firstSearcher
and newSearcher
sections of solrconfig.xml? By "killer" I mean even single queries that
search something
on all the fields you care about, facet on a bunch of fields and sort by a
bunch of
fields. These can even be all in the same query, the point is to fill up
the lower-level
caches before serving searchers.
If you're wondering about the difference, firstSearcher queries are
executed when you
boot the server, by definition there won't be anything useful for autowarm
from the
caches in solrconfig.xml.
newSearcher are queries fired after a commit happens, thus there is
probably a bunch of
autowarming data that can be used from the cache "autowarm" counts.
FWIW,
Erick
On Mon, Oct 21, 2013 at 7:34 PM, Otis Gospodnetic <
otis.gospodnetic@gmail.com> wrote:
> Hi Michael,
>
> I agree with Shawn, don't listen to Peter ;) .... but only this once -
> he's a smart guy, as you can see in list archives.
> And I disagree with Shawn..... again, only just this once and only
> somewhat. :) Because:
>
> In general, Shawn's advice is correct, but we have no way of knowing
> your particular details. TO illustrate the point, let me use an
> extreme case where you have just one query that you hammer your
> servers with. Your Solr caches will be well utilized and your servers
> will not really need a lot of memory to cache your 100 GB index
> because only a small portion of it will ever be accessed. Of course,
> this is an extreme case and not realistic, but I think it helps one
> understands how as the number of distinct queries grows (and thus also
> the number of distinct documents being matched and returned), the need
> for more and more memory goes up. So the question is where exactly
> your particular application falls.
>
> You mentioned stress testing. Just like you, I am assuming, have a
> real index there, you need to have your real queries, too - real
> volume, real diversity, real rate, real complexity, real or as close
> to real everything.
>
> Since you as using SPM, you should be able to go to various graphs in
> SPM and look for a little ambulance icon above each graph. Use that
> to assemble a message with N graphs you want us to look at and we'll
> be able to help more. Graphs that may be of interest here are your
> Solr cache graphs, disk IO, and memory graphs -- taken during your
> realistic stress testing, of course. You can then send that message
> directly to solr-user, assuming your SPM account email address is
> subscribed to the list. Or you can paste it into a new email, up to
> you.
>
> Otis
> --
> Performance Monitoring * Log Analytics * Search Analytics
> Solr & Elasticsearch Support * http://sematext.com/
>
>
>
> On Mon, Oct 21, 2013 at 11:07 AM, Shawn Heisey <so...@elyograg.org> wrote:
> > On 10/21/2013 8:03 AM, michael.boom wrote:
> >> I'm using the m3.xlarge server with 15G RAM, but my index size is over
> 100G,
> >> so I guess putting running the above command would bite all available
> >> memory.
> >
> > With a 100GB index, I would want a minimum server memory size of 64GB,
> > and I would much prefer 128GB. If you shard your index, then each
> > machine will require less memory, because each one will have less of the
> > index onboard. Running a big Solr install is usually best handled on
> > bare metal, because it loves RAM, and getting a lot of memory in a
> > virtual environment is quite expensive. It's also expensive on bare
> > metal too, but unlike Amazon, more memory doesn't increase your monthly
> > cost.
> >
> > With only 15GB total RAM and an index that big, you're probably giving
> > at least half of your RAM to Solr, leaving *very* little for the OS disk
> > cache, compared to your index size. The ideal cache size is the same as
> > your index size, but you can almost always get away with less.
> >
> > http://wiki.apache.org/solr/SolrPerformanceProblems#OS_Disk_Cache
> >
> > If you try the "cat" trick with your numbers, it's going to take forever
> > every time you run it, it will kill your performance while it's
> > happening, and only the last few GB that it reads will remain in the OS
> > disk cache. Chances are that it will be the wrong part of the index,
> too.
> >
> > You only want to cat your entire index if you have enough free RAM to
> > *FIT* your entire index. If you *DO* have that much free memory (which
> > for you would require a total RAM size of about 128GB), then the first
> > time will take quite a while, but every time you do it after that, it
> > will happen nearly instantly, because it will not have to actually read
> > the disk at all.
> >
> > You could try only doing the cat on certain index files, but when you
> > don't have enough cache for the entire index, running queries will do a
> > better job of filling the cache intelligently. The first bunch of
> > queries will be slow.
> >
> > Summary: You need more RAM. Quite a bit more RAM.
> >
> > Thanks,
> > Shawn
> >
>
Re: Solr timeout after reboot
Posted by Otis Gospodnetic <ot...@gmail.com>.
Hi Michael,
I agree with Shawn, don't listen to Peter ;) .... but only this once -
he's a smart guy, as you can see in list archives.
And I disagree with Shawn..... again, only just this once and only
somewhat. :) Because:
In general, Shawn's advice is correct, but we have no way of knowing
your particular details. TO illustrate the point, let me use an
extreme case where you have just one query that you hammer your
servers with. Your Solr caches will be well utilized and your servers
will not really need a lot of memory to cache your 100 GB index
because only a small portion of it will ever be accessed. Of course,
this is an extreme case and not realistic, but I think it helps one
understands how as the number of distinct queries grows (and thus also
the number of distinct documents being matched and returned), the need
for more and more memory goes up. So the question is where exactly
your particular application falls.
You mentioned stress testing. Just like you, I am assuming, have a
real index there, you need to have your real queries, too - real
volume, real diversity, real rate, real complexity, real or as close
to real everything.
Since you as using SPM, you should be able to go to various graphs in
SPM and look for a little ambulance icon above each graph. Use that
to assemble a message with N graphs you want us to look at and we'll
be able to help more. Graphs that may be of interest here are your
Solr cache graphs, disk IO, and memory graphs -- taken during your
realistic stress testing, of course. You can then send that message
directly to solr-user, assuming your SPM account email address is
subscribed to the list. Or you can paste it into a new email, up to
you.
Otis
--
Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/
On Mon, Oct 21, 2013 at 11:07 AM, Shawn Heisey <so...@elyograg.org> wrote:
> On 10/21/2013 8:03 AM, michael.boom wrote:
>> I'm using the m3.xlarge server with 15G RAM, but my index size is over 100G,
>> so I guess putting running the above command would bite all available
>> memory.
>
> With a 100GB index, I would want a minimum server memory size of 64GB,
> and I would much prefer 128GB. If you shard your index, then each
> machine will require less memory, because each one will have less of the
> index onboard. Running a big Solr install is usually best handled on
> bare metal, because it loves RAM, and getting a lot of memory in a
> virtual environment is quite expensive. It's also expensive on bare
> metal too, but unlike Amazon, more memory doesn't increase your monthly
> cost.
>
> With only 15GB total RAM and an index that big, you're probably giving
> at least half of your RAM to Solr, leaving *very* little for the OS disk
> cache, compared to your index size. The ideal cache size is the same as
> your index size, but you can almost always get away with less.
>
> http://wiki.apache.org/solr/SolrPerformanceProblems#OS_Disk_Cache
>
> If you try the "cat" trick with your numbers, it's going to take forever
> every time you run it, it will kill your performance while it's
> happening, and only the last few GB that it reads will remain in the OS
> disk cache. Chances are that it will be the wrong part of the index, too.
>
> You only want to cat your entire index if you have enough free RAM to
> *FIT* your entire index. If you *DO* have that much free memory (which
> for you would require a total RAM size of about 128GB), then the first
> time will take quite a while, but every time you do it after that, it
> will happen nearly instantly, because it will not have to actually read
> the disk at all.
>
> You could try only doing the cat on certain index files, but when you
> don't have enough cache for the entire index, running queries will do a
> better job of filling the cache intelligently. The first bunch of
> queries will be slow.
>
> Summary: You need more RAM. Quite a bit more RAM.
>
> Thanks,
> Shawn
>
Re: Solr timeout after reboot
Posted by Shawn Heisey <so...@elyograg.org>.
On 10/21/2013 8:03 AM, michael.boom wrote:
> I'm using the m3.xlarge server with 15G RAM, but my index size is over 100G,
> so I guess putting running the above command would bite all available
> memory.
With a 100GB index, I would want a minimum server memory size of 64GB,
and I would much prefer 128GB. If you shard your index, then each
machine will require less memory, because each one will have less of the
index onboard. Running a big Solr install is usually best handled on
bare metal, because it loves RAM, and getting a lot of memory in a
virtual environment is quite expensive. It's also expensive on bare
metal too, but unlike Amazon, more memory doesn't increase your monthly
cost.
With only 15GB total RAM and an index that big, you're probably giving
at least half of your RAM to Solr, leaving *very* little for the OS disk
cache, compared to your index size. The ideal cache size is the same as
your index size, but you can almost always get away with less.
http://wiki.apache.org/solr/SolrPerformanceProblems#OS_Disk_Cache
If you try the "cat" trick with your numbers, it's going to take forever
every time you run it, it will kill your performance while it's
happening, and only the last few GB that it reads will remain in the OS
disk cache. Chances are that it will be the wrong part of the index, too.
You only want to cat your entire index if you have enough free RAM to
*FIT* your entire index. If you *DO* have that much free memory (which
for you would require a total RAM size of about 128GB), then the first
time will take quite a while, but every time you do it after that, it
will happen nearly instantly, because it will not have to actually read
the disk at all.
You could try only doing the cat on certain index files, but when you
don't have enough cache for the entire index, running queries will do a
better job of filling the cache intelligently. The first bunch of
queries will be slow.
Summary: You need more RAM. Quite a bit more RAM.
Thanks,
Shawn
Re: Solr timeout after reboot
Posted by François Schiettecatte <fs...@gmail.com>.
Well no, the OS is smarter than that, it manages file system cache along with other memory requirements. If applications need more memory then file system cache will likely be reduced.
The command is a cheap trick to get the OS to fill the file system cache as quickly as possible, not sure how much it will help though with a 100GB index on a 15GB machine. This might work if you 'cat' the index files other than the '.fdx' and '.fdt' files.
François
On Oct 21, 2013, at 10:03 AM, michael.boom <my...@yahoo.com> wrote:
> I'm using the m3.xlarge server with 15G RAM, but my index size is over 100G,
> so I guess putting running the above command would bite all available
> memory.
>
>
>
> -----
> Thanks,
> Michael
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Solr-timeout-after-reboot-tp4096408p4096827.html
> Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr timeout after reboot
Posted by "michael.boom" <my...@yahoo.com>.
I'm using the m3.xlarge server with 15G RAM, but my index size is over 100G,
so I guess putting running the above command would bite all available
memory.
-----
Thanks,
Michael
--
View this message in context: http://lucene.472066.n3.nabble.com/Solr-timeout-after-reboot-tp4096408p4096827.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr timeout after reboot
Posted by "michael.boom" <my...@yahoo.com>.
Thank you, Peter!
Last weekend I was up until 4am trying to understand why is Solr starting so
so sooo slow, when i had gave enough memory to fit the entire index.
And then I remembered your trick used on the m3.xlarge machines, tried it
and it worked like a charm!
Thank you again!
-----
Thanks,
Michael
--
View this message in context: http://lucene.472066.n3.nabble.com/Solr-timeout-after-reboot-tp4096408p4100254.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr timeout after reboot
Posted by Peter Keegan <pe...@gmail.com>.
I found this warming to be especially necessary after starting an instance
of those m3.xlarge servers, else the response times for the first minutes
was terrible.
Peter
On Mon, Oct 21, 2013 at 8:39 AM, François Schiettecatte <
fschiettecatte@gmail.com> wrote:
> To put the file data into file system cache which would make for faster
> access.
>
> François
>
>
> On Oct 21, 2013, at 8:33 AM, michael.boom <my...@yahoo.com> wrote:
>
> > Hmm, no, I haven't...
> >
> > What would be the effect of this ?
> >
> >
> >
> > -----
> > Thanks,
> > Michael
> > --
> > View this message in context:
> http://lucene.472066.n3.nabble.com/Solr-timeout-after-reboot-tp4096408p4096809.html
> > Sent from the Solr - User mailing list archive at Nabble.com.
>
>
Re: Solr timeout after reboot
Posted by François Schiettecatte <fs...@gmail.com>.
To put the file data into file system cache which would make for faster access.
François
On Oct 21, 2013, at 8:33 AM, michael.boom <my...@yahoo.com> wrote:
> Hmm, no, I haven't...
>
> What would be the effect of this ?
>
>
>
> -----
> Thanks,
> Michael
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Solr-timeout-after-reboot-tp4096408p4096809.html
> Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr timeout after reboot
Posted by "michael.boom" <my...@yahoo.com>.
Hmm, no, I haven't...
What would be the effect of this ?
-----
Thanks,
Michael
--
View this message in context: http://lucene.472066.n3.nabble.com/Solr-timeout-after-reboot-tp4096408p4096809.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr timeout after reboot
Posted by Peter Keegan <pe...@gmail.com>.
Have you tried this old trick to warm the FS cache?
cat .../<core>/data/index/* >/dev/null
Peter
On Mon, Oct 21, 2013 at 5:31 AM, michael.boom <my...@yahoo.com> wrote:
> Thank you, Otis!
>
> I've integrated the SPM on my Solr instances and now I have access to
> monitoring data.
> Could you give me some hints on which metrics should I watch?
>
> Below I've added my query configs. Is there anything I could tweak here?
>
> <query>
> <maxBooleanClauses>1024</maxBooleanClauses>
>
> <filterCache class="solr.FastLRUCache"
> size="1000"
> initialSize="1000"
> autowarmCount="0"/>
>
> <queryResultCache class="solr.LRUCache"
> size="1000"
> initialSize="1000"
> autowarmCount="0"/>
>
> <documentCache class="solr.LRUCache"
> size="1000"
> initialSize="1000"
> autowarmCount="0"/>
>
>
> <fieldValueCache class="solr.FastLRUCache"
> size="1000"
> initialSize="1000"
> autowarmCount="0" />
>
>
> <enableLazyFieldLoading>true</enableLazyFieldLoading>
>
> <queryResultWindowSize>20</queryResultWindowSize>
>
> <queryResultMaxDocsCached>100</queryResultMaxDocsCached>
>
> <listener event="firstSearcher" class="solr.QuerySenderListener">
> <arr name="queries">
> <lst>
> <str name="q">active:true</str>
> </lst>
> </arr>
> </listener>
>
> <useColdSearcher>false</useColdSearcher>
>
> <maxWarmingSearchers>10</maxWarmingSearchers>
>
> </query>
>
>
>
> -----
> Thanks,
> Michael
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Solr-timeout-after-reboot-tp4096408p4096780.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
Re: Solr timeout after reboot
Posted by "michael.boom" <my...@yahoo.com>.
Thank you, Otis!
I've integrated the SPM on my Solr instances and now I have access to
monitoring data.
Could you give me some hints on which metrics should I watch?
Below I've added my query configs. Is there anything I could tweak here?
<query>
<maxBooleanClauses>1024</maxBooleanClauses>
<filterCache class="solr.FastLRUCache"
size="1000"
initialSize="1000"
autowarmCount="0"/>
<queryResultCache class="solr.LRUCache"
size="1000"
initialSize="1000"
autowarmCount="0"/>
<documentCache class="solr.LRUCache"
size="1000"
initialSize="1000"
autowarmCount="0"/>
<fieldValueCache class="solr.FastLRUCache"
size="1000"
initialSize="1000"
autowarmCount="0" />
<enableLazyFieldLoading>true</enableLazyFieldLoading>
<queryResultWindowSize>20</queryResultWindowSize>
<queryResultMaxDocsCached>100</queryResultMaxDocsCached>
<listener event="firstSearcher" class="solr.QuerySenderListener">
<arr name="queries">
<lst>
<str name="q">active:true</str>
</lst>
</arr>
</listener>
<useColdSearcher>false</useColdSearcher>
<maxWarmingSearchers>10</maxWarmingSearchers>
</query>
-----
Thanks,
Michael
--
View this message in context: http://lucene.472066.n3.nabble.com/Solr-timeout-after-reboot-tp4096408p4096780.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr timeout after reboot
Posted by Otis Gospodnetic <ot...@gmail.com>.
Michael,
The servlet container controls timeouts, max threads and such. That's not a
high query rate, but yes, it could be solr or OS caches are cold. You will
ne able too see all this in SPM for Solr while you hammer your poor Solr
servers :)
Otis
Solr & ElasticSearch Support
http://sematext.com/
On Oct 18, 2013 11:38 AM, "michael.boom" <my...@yahoo.com> wrote:
> I have a SolrCloud environment with 4 shards, each having a replica and a
> leader. The index size is about 70M docs and 60Gb, running with Jetty +
> Zookeeper, on 2 EC2 instances, each with 4CPUs and 15G RAM.
>
> I'm using SolrMeter for stress testing.
> If I restart Jetty and then try to use SolrMeter to bomb an instance with
> queries, using a query per minute rate of 3000 then that solr instance
> somehow timesout and I need to restart it again.
> If instead of using 3000 qpm i startup slowly with 200 for a minute or two,
> then 1800 and then 3000 everything is good.
>
> I assume this happens because Solr is not warmed up.
> What settings could I tweak so that Solr doesn't time out anymore when
> getting many requests? Is there a way to limit how many req it can serve?
>
>
>
> -----
> Thanks,
> Michael
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Solr-timeout-after-reboot-tp4096408.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>