You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by lucene user <lu...@gmail.com> on 2007/08/12 09:03:48 UTC
Amount of RAM needed to support a growing lucene index?
Hi, Folks -
Two quick questions - need to size a server to run our new index.
If I have an index with 111k articles and 90 million words indexed, how much
RAM should I have to get really fast access speeds?
If I have an index with 290k articles and 234 million words indexed, how
much RAM should I have to get really fast access speeds?
Any other advice about sizing a server?
What other info do you need to have to help size the server?
Does it matter if the server has a 64 bit processor?
Speed of processor important? Speed of disks?
Thanks!
Re: Amount of RAM needed to support a growing lucene index?
Posted by karl wettin <ka...@gmail.com>.
14 aug 2007 kl. 00.17 skrev lucene user:
> What if my concern is more in terms of having a large number of
> requests per
> second? When should I start to be worried and start thinking about
> more than
> low end hardware?
I have served one request every 10th millisecond, 24/7 on a single
machine
with a corpus similar to yours using Lucene, Solaris, dual core and
8GB RAM.
If you follow the recommendations available on the Wiki you do not
need to
worry about Lucene. You might need to consider tweaking your JVM with
some
memory related parameters though. Heap per thread, et c.
http://wiki.apache.org/lucene-java/BasicsOfPerformance
--
karl
>
> Thanks!
>
> On 8/12/07, karl wettin <ka...@gmail.com> wrote:
>>
>>
>> 12 aug 2007 kl. 14.01 skrev lucene user:
>>
>>> Do you know if 290k articles and 234 million words is a large
>>> lucene index
>>> or a medium one? Do people build them this big all the time?
>>
>> If the calculator in my head works you have 300k documents at 4k text
>> each.
>>
>> I say your corpus is borderline small.
>>
>>
>> --
>> karl
>>
>>> Thanks!
>>>
>>> On 8/12/07, karl wettin <ka...@gmail.com> wrote:
>>>>
>>>>
>>>> 12 aug 2007 kl. 09.03 skrev lucene user:
>>>>
>>>>> If I have an index with 111k articles and 90 million words
>>>>> indexed,
>>>>> how much
>>>>> RAM should I have to get really fast access speeds?
>>>>>
>>>>> If I have an index with 290k articles and 234 million words
>>>>> indexed, how
>>>>> much RAM should I have to get really fast access speeds?
>>>>
>>>> Define really fast.
>>>>
>>>> I say you need 1.3x as much RAM as the size of your FSDirectory to
>>>> ensure that the file system cache is never flushed out. But it also
>>>> depends on user load. Each thread consumes RAM and CPU.
>>>>
>>>> In order to really find out, setup the benchmarker to run on your
>>>> index, and limit the amount of memory your file system chache
>>>> and JVM
>>>> is allowed.
>>>>
>>>>> Any other advice about sizing a server?
>>>>> What other info do you need to have to help size the server?
>>>>
>>>> Sizing?
>>>>
>>>>> Does it matter if the server has a 64 bit processor?
>>>>
>>>> In a 64 bit environment a reference to an instance consumes
>>>> twice as
>>>> much RAM as in a 32 bit environment. It should not affect a file
>>>> centric Lucene store (Directory), Your OS and your application that
>>>> use Lucene might be consuming some more resources though. Again,
>>>> benchmark.
>>>>
>>>>> Speed of processor important?
>>>>
>>>> Yes.
>>>>
>>>>> Speed of disks?
>>>>
>>>> May or may not be intersting depending on how much RAM you have.
>>>>
>>>>
>>>>
>>>> --
>>>> karl
>>>>
>>>>
>>>> -------------------------------------------------------------------
>>>> --
>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>
>>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Amount of RAM needed to support a growing lucene index?
Posted by lucene user <lu...@gmail.com>.
That is wonderful to hear. (I love that I am not stressing the technology
near its limits.)
What if my concern is more in terms of having a large number of requests per
second? When should I start to be worried and start thinking about more than
low end hardware?
Thanks!
On 8/12/07, karl wettin <ka...@gmail.com> wrote:
>
>
> 12 aug 2007 kl. 14.01 skrev lucene user:
>
> > Do you know if 290k articles and 234 million words is a large
> > lucene index
> > or a medium one? Do people build them this big all the time?
>
> If the calculator in my head works you have 300k documents at 4k text
> each.
>
> I say your corpus is borderline small.
>
>
> --
> karl
>
> > Thanks!
> >
> > On 8/12/07, karl wettin <ka...@gmail.com> wrote:
> >>
> >>
> >> 12 aug 2007 kl. 09.03 skrev lucene user:
> >>
> >>> If I have an index with 111k articles and 90 million words indexed,
> >>> how much
> >>> RAM should I have to get really fast access speeds?
> >>>
> >>> If I have an index with 290k articles and 234 million words
> >>> indexed, how
> >>> much RAM should I have to get really fast access speeds?
> >>
> >> Define really fast.
> >>
> >> I say you need 1.3x as much RAM as the size of your FSDirectory to
> >> ensure that the file system cache is never flushed out. But it also
> >> depends on user load. Each thread consumes RAM and CPU.
> >>
> >> In order to really find out, setup the benchmarker to run on your
> >> index, and limit the amount of memory your file system chache and JVM
> >> is allowed.
> >>
> >>> Any other advice about sizing a server?
> >>> What other info do you need to have to help size the server?
> >>
> >> Sizing?
> >>
> >>> Does it matter if the server has a 64 bit processor?
> >>
> >> In a 64 bit environment a reference to an instance consumes twice as
> >> much RAM as in a 32 bit environment. It should not affect a file
> >> centric Lucene store (Directory), Your OS and your application that
> >> use Lucene might be consuming some more resources though. Again,
> >> benchmark.
> >>
> >>> Speed of processor important?
> >>
> >> Yes.
> >>
> >>> Speed of disks?
> >>
> >> May or may not be intersting depending on how much RAM you have.
> >>
> >>
> >>
> >> --
> >> karl
> >>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> >> For additional commands, e-mail: java-user-help@lucene.apache.org
> >>
> >>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
Re: Amount of RAM needed to support a growing lucene index?
Posted by karl wettin <ka...@gmail.com>.
12 aug 2007 kl. 14.01 skrev lucene user:
> Do you know if 290k articles and 234 million words is a large
> lucene index
> or a medium one? Do people build them this big all the time?
If the calculator in my head works you have 300k documents at 4k text
each.
I say your corpus is borderline small.
--
karl
> Thanks!
>
> On 8/12/07, karl wettin <ka...@gmail.com> wrote:
>>
>>
>> 12 aug 2007 kl. 09.03 skrev lucene user:
>>
>>> If I have an index with 111k articles and 90 million words indexed,
>>> how much
>>> RAM should I have to get really fast access speeds?
>>>
>>> If I have an index with 290k articles and 234 million words
>>> indexed, how
>>> much RAM should I have to get really fast access speeds?
>>
>> Define really fast.
>>
>> I say you need 1.3x as much RAM as the size of your FSDirectory to
>> ensure that the file system cache is never flushed out. But it also
>> depends on user load. Each thread consumes RAM and CPU.
>>
>> In order to really find out, setup the benchmarker to run on your
>> index, and limit the amount of memory your file system chache and JVM
>> is allowed.
>>
>>> Any other advice about sizing a server?
>>> What other info do you need to have to help size the server?
>>
>> Sizing?
>>
>>> Does it matter if the server has a 64 bit processor?
>>
>> In a 64 bit environment a reference to an instance consumes twice as
>> much RAM as in a 32 bit environment. It should not affect a file
>> centric Lucene store (Directory), Your OS and your application that
>> use Lucene might be consuming some more resources though. Again,
>> benchmark.
>>
>>> Speed of processor important?
>>
>> Yes.
>>
>>> Speed of disks?
>>
>> May or may not be intersting depending on how much RAM you have.
>>
>>
>>
>> --
>> karl
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Amount of RAM needed to support a growing lucene index?
Posted by lucene user <lu...@gmail.com>.
Thanks, Karl.
Do you know if 290k articles and 234 million words is a large lucene index
or a medium one? Do people build them this big all the time?
Thanks!
On 8/12/07, karl wettin <ka...@gmail.com> wrote:
>
>
> 12 aug 2007 kl. 09.03 skrev lucene user:
>
> > If I have an index with 111k articles and 90 million words indexed,
> > how much
> > RAM should I have to get really fast access speeds?
> >
> > If I have an index with 290k articles and 234 million words
> > indexed, how
> > much RAM should I have to get really fast access speeds?
>
> Define really fast.
>
> I say you need 1.3x as much RAM as the size of your FSDirectory to
> ensure that the file system cache is never flushed out. But it also
> depends on user load. Each thread consumes RAM and CPU.
>
> In order to really find out, setup the benchmarker to run on your
> index, and limit the amount of memory your file system chache and JVM
> is allowed.
>
> > Any other advice about sizing a server?
> > What other info do you need to have to help size the server?
>
> Sizing?
>
> > Does it matter if the server has a 64 bit processor?
>
> In a 64 bit environment a reference to an instance consumes twice as
> much RAM as in a 32 bit environment. It should not affect a file
> centric Lucene store (Directory), Your OS and your application that
> use Lucene might be consuming some more resources though. Again,
> benchmark.
>
> > Speed of processor important?
>
> Yes.
>
> > Speed of disks?
>
> May or may not be intersting depending on how much RAM you have.
>
>
>
> --
> karl
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
Re: Amount of RAM needed to support a growing lucene index?
Posted by karl wettin <ka...@gmail.com>.
12 aug 2007 kl. 09.03 skrev lucene user:
> If I have an index with 111k articles and 90 million words indexed,
> how much
> RAM should I have to get really fast access speeds?
>
> If I have an index with 290k articles and 234 million words
> indexed, how
> much RAM should I have to get really fast access speeds?
Define really fast.
I say you need 1.3x as much RAM as the size of your FSDirectory to
ensure that the file system cache is never flushed out. But it also
depends on user load. Each thread consumes RAM and CPU.
In order to really find out, setup the benchmarker to run on your
index, and limit the amount of memory your file system chache and JVM
is allowed.
> Any other advice about sizing a server?
> What other info do you need to have to help size the server?
Sizing?
> Does it matter if the server has a 64 bit processor?
In a 64 bit environment a reference to an instance consumes twice as
much RAM as in a 32 bit environment. It should not affect a file
centric Lucene store (Directory), Your OS and your application that
use Lucene might be consuming some more resources though. Again,
benchmark.
> Speed of processor important?
Yes.
> Speed of disks?
May or may not be intersting depending on how much RAM you have.
--
karl
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org