You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Li Li <fa...@gmail.com> on 2010/07/14 08:38:27 UTC
Cache full text into memory
I want to cache full text into memory to improve performance.
Full text is only used to highlight in my application(But it's very
time consuming, My avg query time is about 250ms, I guess it will cost
about 50ms if I just get top 10 full text. Things get worse when get
more full text because in disk, it scatters erverywhere for a query.).
My full text per machine is about 200GB. The memory available for
store full text is about 10GB. So I want to compress it in memory.
Suppose compression ratio is 1:5, then I can load 1/4 full text in
memory. I need a Cache component for it. Has anyone faced the problem
before? I need some advice. Is it possbile using external tools such
as MemCached? Thank you.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Cache full text into memory
Posted by findbestopensource <fi...@gmail.com>.
I doubt about it. Caching system is a key value store. You have to use some
compression library to compress and decompress your data. Caching system
helps to retrieve fast. Anyways please take a look of each of the caching
system features.
Regards
Aditya
www.findbestopensource.com
On Wed, Jul 14, 2010 at 3:06 PM, Li Li <fa...@gmail.com> wrote:
> Thank you. I don't know which cache system to use. In my application,
> the cache system must support compression algorithm which has high
> compression ratio and fast decompression speed(because each time it
> get from cache, it must decompress).
>
> 2010/7/14 findbestopensource <fi...@gmail.com>:
> > I have just provided you two options. Since you already store as part of
> the
> > index, You could try external caching. Try using ehcache / Membase
> > http://www.findbestopensource.com/tagged/distributed-caching . The
> caching
> > system will do LRU and is much more efficient.
> >
> > On Wed, Jul 14, 2010 at 12:39 PM, Li Li <fa...@gmail.com> wrote:
> >
> >> I have already store it in lucene index. But it is in disk and When a
> >> query come, it must seek the disk to get it. I am not familiar with
> >> lucene cache. I just want to fully use my memory that load 10GB of it
> >> in memory and a LRU stragety when cache full. To load more into
> >> memory, I want to compress it "in memory". I don't care much about
> >> disk space so whether or not it's compressed in lucene .
> >>
> >> 2010/7/14 findbestopensource <fi...@gmail.com>:
> >> > You have two options
> >> > 1. Store the compressed text as part of stored field in Solr.
> >> > 2. Using external caching.
> >> > http://www.findbestopensource.com/tagged/distributed-caching
> >> > You could use ehcache / Memcache / Membase.
> >> >
> >> > The problem with external caching is you need to synchronize the
> >> deletions
> >> > and modification. Fetching the stored field from Solr is also faster.
> >> >
> >> > Regards
> >> > Aditya
> >> > www.findbestopensource.com
> >> >
> >> >
> >> > On Wed, Jul 14, 2010 at 12:08 PM, Li Li <fa...@gmail.com> wrote:
> >> >
> >> >> I want to cache full text into memory to improve performance.
> >> >> Full text is only used to highlight in my application(But it's very
> >> >> time consuming, My avg query time is about 250ms, I guess it will
> cost
> >> >> about 50ms if I just get top 10 full text. Things get worse when get
> >> >> more full text because in disk, it scatters erverywhere for a
> query.).
> >> >> My full text per machine is about 200GB. The memory available for
> >> >> store full text is about 10GB. So I want to compress it in memory.
> >> >> Suppose compression ratio is 1:5, then I can load 1/4 full text in
> >> >> memory. I need a Cache component for it. Has anyone faced the problem
> >> >> before? I need some advice. Is it possbile using external tools such
> >> >> as MemCached? Thank you.
> >> >>
> >> >
> >>
> >
>
Re: Cache full text into memory
Posted by Li Li <fa...@gmail.com>.
Thank you. I don't know which cache system to use. In my application,
the cache system must support compression algorithm which has high
compression ratio and fast decompression speed(because each time it
get from cache, it must decompress).
2010/7/14 findbestopensource <fi...@gmail.com>:
> I have just provided you two options. Since you already store as part of the
> index, You could try external caching. Try using ehcache / Membase
> http://www.findbestopensource.com/tagged/distributed-caching . The caching
> system will do LRU and is much more efficient.
>
> On Wed, Jul 14, 2010 at 12:39 PM, Li Li <fa...@gmail.com> wrote:
>
>> I have already store it in lucene index. But it is in disk and When a
>> query come, it must seek the disk to get it. I am not familiar with
>> lucene cache. I just want to fully use my memory that load 10GB of it
>> in memory and a LRU stragety when cache full. To load more into
>> memory, I want to compress it "in memory". I don't care much about
>> disk space so whether or not it's compressed in lucene .
>>
>> 2010/7/14 findbestopensource <fi...@gmail.com>:
>> > You have two options
>> > 1. Store the compressed text as part of stored field in Solr.
>> > 2. Using external caching.
>> > http://www.findbestopensource.com/tagged/distributed-caching
>> > You could use ehcache / Memcache / Membase.
>> >
>> > The problem with external caching is you need to synchronize the
>> deletions
>> > and modification. Fetching the stored field from Solr is also faster.
>> >
>> > Regards
>> > Aditya
>> > www.findbestopensource.com
>> >
>> >
>> > On Wed, Jul 14, 2010 at 12:08 PM, Li Li <fa...@gmail.com> wrote:
>> >
>> >> I want to cache full text into memory to improve performance.
>> >> Full text is only used to highlight in my application(But it's very
>> >> time consuming, My avg query time is about 250ms, I guess it will cost
>> >> about 50ms if I just get top 10 full text. Things get worse when get
>> >> more full text because in disk, it scatters erverywhere for a query.).
>> >> My full text per machine is about 200GB. The memory available for
>> >> store full text is about 10GB. So I want to compress it in memory.
>> >> Suppose compression ratio is 1:5, then I can load 1/4 full text in
>> >> memory. I need a Cache component for it. Has anyone faced the problem
>> >> before? I need some advice. Is it possbile using external tools such
>> >> as MemCached? Thank you.
>> >>
>> >
>>
>
Re: Cache full text into memory
Posted by findbestopensource <fi...@gmail.com>.
I have just provided you two options. Since you already store as part of the
index, You could try external caching. Try using ehcache / Membase
http://www.findbestopensource.com/tagged/distributed-caching . The caching
system will do LRU and is much more efficient.
On Wed, Jul 14, 2010 at 12:39 PM, Li Li <fa...@gmail.com> wrote:
> I have already store it in lucene index. But it is in disk and When a
> query come, it must seek the disk to get it. I am not familiar with
> lucene cache. I just want to fully use my memory that load 10GB of it
> in memory and a LRU stragety when cache full. To load more into
> memory, I want to compress it "in memory". I don't care much about
> disk space so whether or not it's compressed in lucene .
>
> 2010/7/14 findbestopensource <fi...@gmail.com>:
> > You have two options
> > 1. Store the compressed text as part of stored field in Solr.
> > 2. Using external caching.
> > http://www.findbestopensource.com/tagged/distributed-caching
> > You could use ehcache / Memcache / Membase.
> >
> > The problem with external caching is you need to synchronize the
> deletions
> > and modification. Fetching the stored field from Solr is also faster.
> >
> > Regards
> > Aditya
> > www.findbestopensource.com
> >
> >
> > On Wed, Jul 14, 2010 at 12:08 PM, Li Li <fa...@gmail.com> wrote:
> >
> >> I want to cache full text into memory to improve performance.
> >> Full text is only used to highlight in my application(But it's very
> >> time consuming, My avg query time is about 250ms, I guess it will cost
> >> about 50ms if I just get top 10 full text. Things get worse when get
> >> more full text because in disk, it scatters erverywhere for a query.).
> >> My full text per machine is about 200GB. The memory available for
> >> store full text is about 10GB. So I want to compress it in memory.
> >> Suppose compression ratio is 1:5, then I can load 1/4 full text in
> >> memory. I need a Cache component for it. Has anyone faced the problem
> >> before? I need some advice. Is it possbile using external tools such
> >> as MemCached? Thank you.
> >>
> >
>
Re: Cache full text into memory
Posted by Li Li <fa...@gmail.com>.
I have already store it in lucene index. But it is in disk and When a
query come, it must seek the disk to get it. I am not familiar with
lucene cache. I just want to fully use my memory that load 10GB of it
in memory and a LRU stragety when cache full. To load more into
memory, I want to compress it "in memory". I don't care much about
disk space so whether or not it's compressed in lucene .
2010/7/14 findbestopensource <fi...@gmail.com>:
> You have two options
> 1. Store the compressed text as part of stored field in Solr.
> 2. Using external caching.
> http://www.findbestopensource.com/tagged/distributed-caching
> You could use ehcache / Memcache / Membase.
>
> The problem with external caching is you need to synchronize the deletions
> and modification. Fetching the stored field from Solr is also faster.
>
> Regards
> Aditya
> www.findbestopensource.com
>
>
> On Wed, Jul 14, 2010 at 12:08 PM, Li Li <fa...@gmail.com> wrote:
>
>> I want to cache full text into memory to improve performance.
>> Full text is only used to highlight in my application(But it's very
>> time consuming, My avg query time is about 250ms, I guess it will cost
>> about 50ms if I just get top 10 full text. Things get worse when get
>> more full text because in disk, it scatters erverywhere for a query.).
>> My full text per machine is about 200GB. The memory available for
>> store full text is about 10GB. So I want to compress it in memory.
>> Suppose compression ratio is 1:5, then I can load 1/4 full text in
>> memory. I need a Cache component for it. Has anyone faced the problem
>> before? I need some advice. Is it possbile using external tools such
>> as MemCached? Thank you.
>>
>
Re: Cache full text into memory
Posted by findbestopensource <fi...@gmail.com>.
You have two options
1. Store the compressed text as part of stored field in Solr.
2. Using external caching.
http://www.findbestopensource.com/tagged/distributed-caching
You could use ehcache / Memcache / Membase.
The problem with external caching is you need to synchronize the deletions
and modification. Fetching the stored field from Solr is also faster.
Regards
Aditya
www.findbestopensource.com
On Wed, Jul 14, 2010 at 12:08 PM, Li Li <fa...@gmail.com> wrote:
> I want to cache full text into memory to improve performance.
> Full text is only used to highlight in my application(But it's very
> time consuming, My avg query time is about 250ms, I guess it will cost
> about 50ms if I just get top 10 full text. Things get worse when get
> more full text because in disk, it scatters erverywhere for a query.).
> My full text per machine is about 200GB. The memory available for
> store full text is about 10GB. So I want to compress it in memory.
> Suppose compression ratio is 1:5, then I can load 1/4 full text in
> memory. I need a Cache component for it. Has anyone faced the problem
> before? I need some advice. Is it possbile using external tools such
> as MemCached? Thank you.
>
Re: Cache full text into memory
Posted by findbestopensource <fi...@gmail.com>.
You have two options
1. Store the compressed text as part of stored field in Solr.
2. Using external caching.
http://www.findbestopensource.com/tagged/distributed-caching
You could use ehcache / Memcache / Membase.
The problem with external caching is you need to synchronize the deletions
and modification. Fetching the stored field from Solr is also faster.
Regards
Aditya
www.findbestopensource.com
On Wed, Jul 14, 2010 at 12:08 PM, Li Li <fa...@gmail.com> wrote:
> I want to cache full text into memory to improve performance.
> Full text is only used to highlight in my application(But it's very
> time consuming, My avg query time is about 250ms, I guess it will cost
> about 50ms if I just get top 10 full text. Things get worse when get
> more full text because in disk, it scatters erverywhere for a query.).
> My full text per machine is about 200GB. The memory available for
> store full text is about 10GB. So I want to compress it in memory.
> Suppose compression ratio is 1:5, then I can load 1/4 full text in
> memory. I need a Cache component for it. Has anyone faced the problem
> before? I need some advice. Is it possbile using external tools such
> as MemCached? Thank you.
>