You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Daniel Andersson <da...@polkabrothers.com> on 2008/05/03 19:06:10 UTC
Help optimizing
Hi (again) people
We've now invested in a server with 8 GB of RAM after too many
OutOfMemory-errors.
Our database/index is 3.5 GB and contains 4,352,471 documents. Most
documents are less than 1 kb. When performing a search, the results
vary between 1.5 seconds up to 60 seconds.
I don't have a big problem with 1.5 seconds (even though below 1 would
be nice), but 60 seconds it just.. well, scary.
How do I optimize Solr to better use all the RAM? I'm using java6,
64bit version, and start Solr using:
java -Xmx7500M -Xms4096M -jar start.jar
But according to top it only seems to be using 7.7% of the memory
(around 600 MB).
Most queries are for make_id + model_id or city + state and almost all
of the queries are ordered by datetime_found (newest -> oldest).
from schema.xml:
<field name="ad_id" type="sint" indexed="true" stored="true"
required="true" />
<field name="make_id" type="sint" indexed="true"
multiValued="false" />
<field name="model_id" type="sint" indexed="true"
multiValued="false" />
<field name="model_group_id" type="sint" indexed="true"
multiValues="true" />
<field name="zip_code" type="sint" indexed="true"
multiValued="false" />
<field name="year" type="sint" indexed="true" multiValued="false" />
<field name="mileage" type="sint" indexed="true"
multiValued="false" />
<field name="cylinders" type="sint" indexed="true"
multiValued="false" />
<field name="doors" type="sint" indexed="true"
multiValued="false" />
<field name="price" type="sfloat" indexed="true"
multiValued="false" />
<field name="headline" type="text" indexed="true"
multiValued="false" />
<field name="description" type="text" indexed="true"
multiValued="false" />
<field name="condition" type="text" indexed="true"
multiValued="false" />
<field name="sell_type" type="text" indexed="true"
multiValued="false" />
<field name="city" type="text" indexed="true" multiValued="false" />
<field name="state" type="text" indexed="true"
multiValued="false" />
<field name="alt_location" type="text" indexed="true"
multiValued="false" />
<field name="address" type="text" indexed="true"
multiValued="false" />
<field name="image_exists" type="boolean" indexed="true"
multiValued="false" />
<field name="vehicle_type" type="text" indexed="true"
multiValued="false" />
<field name="datetime_found" type="date" indexed="true"
multiValued="false" />
<field name="transmission" type="text" indexed="true"
multiValued="false" />
<field name="color" type="text" indexed="true"
multiValued="false" />
<field name="interior" type="text" indexed="true"
multiValued="false" />
<field name="private_seller" type="boolean" indexed="true"
multiValued="false" />
<field name="fuel" type="text" indexed="true" multiValued="false" />
<field name="ac" type="boolean" indexed="true"
multiValued="false" />
<field name="cruise_control" type="boolean" indexed="true"
multiValued="false" />
<field name="lat" type="sdouble" indexed="true" stored="true" />
<field name="lng" type="sdouble" indexed="true" stored="true" />
<dynamicField name="_local*" type="sdouble" indexed="true"
stored="true" multiValued="false" />
<!-- catchall field, containing all other searchable text fields
(implemented
via copyField further on in this schema -->
<field name="text" type="text" indexed="true" stored="false"
multiValued="true"/>
<copyField source="headline" dest="text"/>
<copyField source="description" dest="text"/>
<copyField source="color" dest="text"/>
from solrconfig.xml:
<filterCache
class="solr.LRUCache"
size="4096"
initialSize="4096"
autowarmCount="2048"/>
<queryResultCache
class="solr.LRUCache"
size="4096"
initialSize="4096"
autowarmCount="2048"/>
<documentCache
class="solr.LRUCache"
size="4096"
initialSize="4096"
autowarmCount="0"/>
<queryResultWindowSize>50</queryResultWindowSize>
<queryResultMaxDocsCached>1000</queryResultMaxDocsCached>
<HashDocSet maxSize="2000" loadFactor="0.75"/>
<listener event="newSearcher" class="solr.QuerySenderListener">
<arr name="queries">
<lst> <str name="q">datetime_found</str> <str name="start">0</
str> <str name="rows">10</str> </lst>
</arr>
</listener>
<listener event="firstSearcher" class="solr.QuerySenderListener">
<arr name="queries">
<lst> <str name="q">datetime_found</str> <str name="start">0</
str> <str name="rows">10</str> </lst>
</arr>
</listener>
<useColdSearcher>true</useColdSearcher>
<maxWarmingSearchers>6</maxWarmingSearchers>
Any help much appreciated!
Kind regards,
Daniel
Re: Help optimizing
Posted by Daniel Andersson <da...@polkabrothers.com>.
On May 6, 2008, at 2:19 PM, Grant Ingersoll wrote:
> On May 3, 2008, at 1:06 PM, Daniel Andersson wrote:
>
>> When performing a search, the results vary between 1.5 seconds up
>> to 60 seconds.
>>
> Is this pure Solr time or overall application time? I ask, b/c it
> is often the case that people are measuring application time and the
> problem lies in the application, so I just want to clarify.
It's 1.5 to send the command to Solr, wait for it to search and get
the data back.
The web server is located in the US and the Solr-machine is in Sweden
(don't ask), so I can see it taking a while to send data back and
forth, so getting the searches below 1.5s is not something I'm
expecting. I "just" want to get away from the >5s searches.
Is there a way of getting Solr to output the total time spent on any
command? Just so I can eliminate some odd network problem/error.
> Also, have you done any profiling to see where the hotspots are?
I have not. Not a Java person, so not sure how to do this. Is there
something in the Solr admin that will allow me to do this? Have looked
around and read what I could find in the Wiki, but didn't find
anything that looked like profiling.
Cheers,
Daniel
Re: Help optimizing
Posted by Daniel Andersson <da...@polkabrothers.com>.
On May 6, 2008, at 7:26 PM, Lance Norskog wrote:
> One cause of out-of-memory is multiple simultaneous requests. If you
> limit
> the query stream to one or two simultaneous requests, you might fix
> this.
> No, Solr does not have an option for this. The servlet containers have
> controls for this that you have to dig very deep to find.
Unfortunately the website is still very small, in terms of visitors.
Was running MySQL, Apache and Solr on the same machine which only had
2 GB of RAM, so understandable if Solr throws an error or two at me.
Cheers,
Daniel
RE: Help optimizing
Posted by Lance Norskog <go...@gmail.com>.
One cause of out-of-memory is multiple simultaneous requests. If you limit
the query stream to one or two simultaneous requests, you might fix this.
No, Solr does not have an option for this. The servlet containers have
controls for this that you have to dig very deep to find.
Lance Norskog
-----Original Message-----
From: Grant Ingersoll [mailto:gsingers@apache.org]
Sent: Tuesday, May 06, 2008 5:19 AM
To: solr-user@lucene.apache.org
Subject: Re: Help optimizing
On May 3, 2008, at 1:06 PM, Daniel Andersson wrote:
> Hi (again) people
>
> We've now invested in a server with 8 GB of RAM after too many
> OutOfMemory-errors.
>
> Our database/index is 3.5 GB and contains 4,352,471 documents. Most
> documents are less than 1 kb. When performing a search, the results
> vary between 1.5 seconds up to 60 seconds.
>
> I don't have a big problem with 1.5 seconds (even though below 1 would
> be nice), but 60 seconds it just.. well, scary.
Is this pure Solr time or overall application time? I ask, b/c it is often
the case that people are measuring application time and the problem lies in
the application, so I just want to clarify.
Also, have you done any profiling to see where the hotspots are?
-Grant
Re: Help optimizing
Posted by Grant Ingersoll <gs...@apache.org>.
On May 3, 2008, at 1:06 PM, Daniel Andersson wrote:
> Hi (again) people
>
> We've now invested in a server with 8 GB of RAM after too many
> OutOfMemory-errors.
>
> Our database/index is 3.5 GB and contains 4,352,471 documents. Most
> documents are less than 1 kb. When performing a search, the results
> vary between 1.5 seconds up to 60 seconds.
>
> I don't have a big problem with 1.5 seconds (even though below 1
> would be nice), but 60 seconds it just.. well, scary.
Is this pure Solr time or overall application time? I ask, b/c it is
often the case that people are measuring application time and the
problem lies in the application, so I just want to clarify.
Also, have you done any profiling to see where the hotspots are?
-Grant
Re: Help optimizing
Posted by Daniel Andersson <da...@polkabrothers.com>.
On May 6, 2008, at 4:00 AM, Mike Klaas wrote:
> On 3-May-08, at 10:06 AM, Daniel Andersson wrote:
>>
>> How do I optimize Solr to better use all the RAM? I'm using java6,
>> 64bit version, and start Solr using:
>> java -Xmx7500M -Xms4096M -jar start.jar
>>
>> But according to top it only seems to be using 7.7% of the memory
>> (around 600 MB).
>
> Don't try to give Solr _all_ the memory on the system. Solr depends
> on the index existing in the OS's disk cache (this is "cached" in
> top). You should have at least 2 GB memory for a 3.5GB index,
> depending on how much of the index is stored (best is of course to
> have 3.5GB available so it can be cached completely).
>
> Solr will require a wide distribution of queries to "warm up" (get
> the index in the OS disk cache). This is automatically prioritize
> the "hot spots" in the index. If you want to load the whole thing
> 'cd datadir; cat * > /dev/null' works, but I don't recommend relying
> on that.
Ah. Have given it 4 GB of RAM now (Xmx=4 GB, Xms=2 GB)
>> Most queries are for make_id + model_id or city + state and almost
>> all of the queries are ordered by datetime_found (newest -> oldest).
>
> How many documents match, typically? How many documents are
> returned, typically? How often do you commit() [I suspect
> frequently, based on the problems you are having]?
Average documents matched/found: 6427
Only return 10 documents per page
Commit every 10,000 documents. Tried it at 100,000 with 2 GB of ram (1
GB dedicated to Solr) and it just gave me OutOfMemory every time.
Haven't tried increasing it since moving it to this new server.
Cheers,
Daniel
Re: Help optimizing
Posted by Mike Klaas <mi...@gmail.com>.
On 3-May-08, at 10:06 AM, Daniel Andersson wrote:
>
> Our database/index is 3.5 GB and contains 4,352,471 documents. Most
> documents are less than 1 kb. When performing a search, the results
> vary between 1.5 seconds up to 60 seconds.
>
> I don't have a big problem with 1.5 seconds (even though below 1
> would be nice), but 60 seconds it just.. well, scary.
That is too long, and shouldn't be happening.
>
> How do I optimize Solr to better use all the RAM? I'm using java6,
> 64bit version, and start Solr using:
> java -Xmx7500M -Xms4096M -jar start.jar
>
> But according to top it only seems to be using 7.7% of the memory
> (around 600 MB).
Don't try to give Solr _all_ the memory on the system. Solr depends
on the index existing in the OS's disk cache (this is "cached" in
top). You should have at least 2 GB memory for a 3.5GB index,
depending on how much of the index is stored (best is of course to
have 3.5GB available so it can be cached completely).
Solr will require a wide distribution of queries to "warm up" (get the
index in the OS disk cache). This is automatically prioritize the
"hot spots" in the index. If you want to load the whole thing 'cd
datadir; cat * > /dev/null' works, but I don't recommend relying on
that.
> Most queries are for make_id + model_id or city + state and almost
> all of the queries are ordered by datetime_found (newest -> oldest).
How many documents match, typically? How many documents are returned,
typically? How often do you commit() [I suspect frequently, based on
the problems you are having]?
-Mike