You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by James <lj...@163.com> on 2012/02/08 05:53:49 UTC
is there any practice to load index into RAM to accelerate solr
performance?
Is there any practice to load index into RAM to accelerate solr performance?
The over all documents is about 100 million. The search time around 100ms. I am seeking some method to accelerate the respond time for solr.
Just check that there is some practice use SSD disk. And SSD is also cost much, just want to know is there some method like to load the index file in RAM and keep the RAM index and disk index synchronized. Then I can search on the RAM index.
Re: is there any practice to load index into RAM to accelerate solr performance?
Posted by Patrick Plaatje <pp...@gmail.com>.
A start maybe to use a RAM disk for that. Mount is as a normal disk and
have the index files stored there. Have a read here:
http://en.wikipedia.org/wiki/RAM_disk
Cheers,
Patrick
2012/2/8 Ted Dunning <te...@gmail.com>
> This is true with Lucene as it stands. It would be much faster if there
> were a specialized in-memory index such as is typically used with high
> performance search engines.
>
> On Tue, Feb 7, 2012 at 9:50 PM, Lance Norskog <go...@gmail.com> wrote:
>
> > Experience has shown that it is much faster to run Solr with a small
> > amount of memory and let the rest of the ram be used by the operating
> > system "disk cache". That is, the OS is very good at keeping the right
> > disk blocks in memory, much better than Solr.
> >
> > How much RAM is in the server and how much RAM does the JVM get? How
> > big are the documents, and how large is the term index for your
> > searches? How many documents do you get with each search? And, do you
> > use filter queries- these are very powerful at limiting searches.
> >
> > 2012/2/7 James <lj...@163.com>:
> > > Is there any practice to load index into RAM to accelerate solr
> > performance?
> > > The over all documents is about 100 million. The search time around
> > 100ms. I am seeking some method to accelerate the respond time for solr.
> > > Just check that there is some practice use SSD disk. And SSD is also
> > cost much, just want to know is there some method like to load the index
> > file in RAM and keep the RAM index and disk index synchronized. Then I
> can
> > search on the RAM index.
> >
> >
> >
> > --
> > Lance Norskog
> > goksron@gmail.com
> >
>
--
Patrick Plaatje
Senior Consultant
<http://www.nmobile.nl/>
Re:Re: is there any practice to load index into RAM to accelerate
solr performance?
Posted by James <lj...@163.com>.
But the solr did not have the im-memory index, I am right?
At 2012-02-08 16:17:49,"Ted Dunning" <te...@gmail.com> wrote:
>This is true with Lucene as it stands. It would be much faster if there
>were a specialized in-memory index such as is typically used with high
>performance search engines.
>
>On Tue, Feb 7, 2012 at 9:50 PM, Lance Norskog <go...@gmail.com> wrote:
>
>> Experience has shown that it is much faster to run Solr with a small
>> amount of memory and let the rest of the ram be used by the operating
>> system "disk cache". That is, the OS is very good at keeping the right
>> disk blocks in memory, much better than Solr.
>>
>> How much RAM is in the server and how much RAM does the JVM get? How
>> big are the documents, and how large is the term index for your
>> searches? How many documents do you get with each search? And, do you
>> use filter queries- these are very powerful at limiting searches.
>>
>> 2012/2/7 James <lj...@163.com>:
>> > Is there any practice to load index into RAM to accelerate solr
>> performance?
>> > The over all documents is about 100 million. The search time around
>> 100ms. I am seeking some method to accelerate the respond time for solr.
>> > Just check that there is some practice use SSD disk. And SSD is also
>> cost much, just want to know is there some method like to load the index
>> file in RAM and keep the RAM index and disk index synchronized. Then I can
>> search on the RAM index.
>>
>>
>>
>> --
>> Lance Norskog
>> goksron@gmail.com
>>
Re: is there any practice to load index into RAM to accelerate solr performance?
Posted by Ted Dunning <te...@gmail.com>.
Add this as well:
http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.155.5030
On Wed, Feb 8, 2012 at 1:56 AM, Andrzej Bialecki <ab...@getopt.org> wrote:
> On 08/02/2012 09:17, Ted Dunning wrote:
>
>> This is true with Lucene as it stands. It would be much faster if there
>> were a specialized in-memory index such as is typically used with high
>> performance search engines.
>>
>
> This could be implemented in Lucene trunk as a Codec. The challenge though
> is to come up with the right data structures.
>
> There has been some interesting research on optimizations for in-memory
> inverted indexes, but it usually involves changing the query evaluation
> algos as well - for reference:
>
> http://digbib.ubka.uni-**karlsruhe.de/volltexte/**documents/1202502<http://digbib.ubka.uni-karlsruhe.de/volltexte/documents/1202502>
> http://www.siam.org/**proceedings/alenex/2008/alx08_**01transierf.pdf<http://www.siam.org/proceedings/alenex/2008/alx08_01transierf.pdf>
> http://research.google.com/**pubs/archive/37365.pdf<http://research.google.com/pubs/archive/37365.pdf>
>
> --
> Best regards,
> Andrzej Bialecki <><
> ___. ___ ___ ___ _ _ ______________________________**____
> [__ || __|__/|__||\/| Information Retrieval, Semantic Web
> ___|||__|| \| || | Embedded Unix, System Integration
> http://www.sigram.com Contact: info at sigram dot com
>
>
Re: is there any practice to load index into RAM to accelerate solr
performance?
Posted by Andrzej Bialecki <ab...@getopt.org>.
On 08/02/2012 09:17, Ted Dunning wrote:
> This is true with Lucene as it stands. It would be much faster if there
> were a specialized in-memory index such as is typically used with high
> performance search engines.
This could be implemented in Lucene trunk as a Codec. The challenge
though is to come up with the right data structures.
There has been some interesting research on optimizations for in-memory
inverted indexes, but it usually involves changing the query evaluation
algos as well - for reference:
http://digbib.ubka.uni-karlsruhe.de/volltexte/documents/1202502
http://www.siam.org/proceedings/alenex/2008/alx08_01transierf.pdf
http://research.google.com/pubs/archive/37365.pdf
--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com
Re: is there any practice to load index into RAM to accelerate solr performance?
Posted by Robert Stewart <bs...@gmail.com>.
I concur with this. As long as index segment files are cached in OS file cache performance is as about good as it gets. Pulling segment files into RAM inside JVM process may actually be slower, given Lucene's existing data structures and algorithms for reading segment file data. If you have very large index (much bigger than available RAM) then it will only be slow when accessing disk for uncached segment files. In that case you might consider sharding index across more than one server and using distributed searching (possibly SOLR cloud, etc.).
How large is your index in GB? You can also try making index files smaller by removing indexed/stored fields you dont need, compressing large stored fields, etc. Also maybe turn off storing norms, term frequencies, positions, vectors and stuff if you dont need them.
On Feb 8, 2012, at 3:17 AM, Ted Dunning wrote:
> This is true with Lucene as it stands. It would be much faster if there
> were a specialized in-memory index such as is typically used with high
> performance search engines.
>
> On Tue, Feb 7, 2012 at 9:50 PM, Lance Norskog <go...@gmail.com> wrote:
>
>> Experience has shown that it is much faster to run Solr with a small
>> amount of memory and let the rest of the ram be used by the operating
>> system "disk cache". That is, the OS is very good at keeping the right
>> disk blocks in memory, much better than Solr.
>>
>> How much RAM is in the server and how much RAM does the JVM get? How
>> big are the documents, and how large is the term index for your
>> searches? How many documents do you get with each search? And, do you
>> use filter queries- these are very powerful at limiting searches.
>>
>> 2012/2/7 James <lj...@163.com>:
>>> Is there any practice to load index into RAM to accelerate solr
>> performance?
>>> The over all documents is about 100 million. The search time around
>> 100ms. I am seeking some method to accelerate the respond time for solr.
>>> Just check that there is some practice use SSD disk. And SSD is also
>> cost much, just want to know is there some method like to load the index
>> file in RAM and keep the RAM index and disk index synchronized. Then I can
>> search on the RAM index.
>>
>>
>>
>> --
>> Lance Norskog
>> goksron@gmail.com
>>
Re: is there any practice to load index into RAM to accelerate solr performance?
Posted by Ted Dunning <te...@gmail.com>.
This is true with Lucene as it stands. It would be much faster if there
were a specialized in-memory index such as is typically used with high
performance search engines.
On Tue, Feb 7, 2012 at 9:50 PM, Lance Norskog <go...@gmail.com> wrote:
> Experience has shown that it is much faster to run Solr with a small
> amount of memory and let the rest of the ram be used by the operating
> system "disk cache". That is, the OS is very good at keeping the right
> disk blocks in memory, much better than Solr.
>
> How much RAM is in the server and how much RAM does the JVM get? How
> big are the documents, and how large is the term index for your
> searches? How many documents do you get with each search? And, do you
> use filter queries- these are very powerful at limiting searches.
>
> 2012/2/7 James <lj...@163.com>:
> > Is there any practice to load index into RAM to accelerate solr
> performance?
> > The over all documents is about 100 million. The search time around
> 100ms. I am seeking some method to accelerate the respond time for solr.
> > Just check that there is some practice use SSD disk. And SSD is also
> cost much, just want to know is there some method like to load the index
> file in RAM and keep the RAM index and disk index synchronized. Then I can
> search on the RAM index.
>
>
>
> --
> Lance Norskog
> goksron@gmail.com
>
Re: is there any practice to load index into RAM to accelerate solr performance?
Posted by Lance Norskog <go...@gmail.com>.
Experience has shown that it is much faster to run Solr with a small
amount of memory and let the rest of the ram be used by the operating
system "disk cache". That is, the OS is very good at keeping the right
disk blocks in memory, much better than Solr.
How much RAM is in the server and how much RAM does the JVM get? How
big are the documents, and how large is the term index for your
searches? How many documents do you get with each search? And, do you
use filter queries- these are very powerful at limiting searches.
2012/2/7 James <lj...@163.com>:
> Is there any practice to load index into RAM to accelerate solr performance?
> The over all documents is about 100 million. The search time around 100ms. I am seeking some method to accelerate the respond time for solr.
> Just check that there is some practice use SSD disk. And SSD is also cost much, just want to know is there some method like to load the index file in RAM and keep the RAM index and disk index synchronized. Then I can search on the RAM index.
--
Lance Norskog
goksron@gmail.com
Re: is there any practice to load index into RAM to accelerate solr performance?
Posted by Dmitry Kan <dm...@gmail.com>.
Hi,
This talk has some interesting details on setting up an Lucene index in RAM:
http://www.lucidimagination.com/devzone/events/conferences/revolution/2011/lucene-yelp
Would be great to hear your findings!
Dmitry
2012/2/8 James <lj...@163.com>
> Is there any practice to load index into RAM to accelerate solr
> performance?
> The over all documents is about 100 million. The search time around 100ms.
> I am seeking some method to accelerate the respond time for solr.
> Just check that there is some practice use SSD disk. And SSD is also cost
> much, just want to know is there some method like to load the index file in
> RAM and keep the RAM index and disk index synchronized. Then I can search
> on the RAM index.
>