You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Sujatha Arun <su...@gmail.com> on 2011/10/19 11:04:24 UTC
Optimization /Commit memory
Do we require 2 or 3 Times OS RAM memory or Hard Disk Space while
performing Commit or Optimize or Both?
what is the requirement in terms of size of RAM and HD for commit and
Optimize
Regards
Sujatha
Re: Optimization /Commit memory
Posted by Sujatha Arun <su...@gmail.com>.
Thanks Simon and Jay .That was helpful .
So what we are looking at during optimize is 2 or 3 times free Disk Space
to recreate the index.
Regards
Sujatha
On Wed, Oct 26, 2011 at 12:26 AM, Simon Willnauer <
simon.willnauer@googlemail.com> wrote:
> RAM costs during optimize / merge is generally low. Optimize is
> basically a merge of all segments into one, however there are
> exceptions. Lucene streams existing segments from disk and serializes
> the new segment on the fly. When you optimize or in general when you
> merge segments you need disk space for the "source" segments and the
> "targed" (merged) segment.
>
> If you use CompoundFileSystem (CFS) you need to additional space once
> the merge is done and your files are packed into the CFS which is
> basically the size of the "target" (merged) segment. Once the merge is
> done lucene can free the diskspace unless you have an IndexReader open
> that references those segments (lucene keeps track of these files and
> frees diskspace once possible).
>
> That said, I think you should use optimize very very rarely. Usually
> if you document collection is rarely changing optimize is useful and
> reasonable once in a while. if you collection is constantly changing
> you should rely on the merge policy to balance the number of segments
> for you in the background. Lucene 3.4 has a nice improved
> TieredMergePolicy that does a great job. (previous version are also
> good - just saying)
>
> A commit is basically flushing the segment you have in memory
> (IndexWriter memory) to disk. compression ratio can be up to 30% of
> the ram cost or even more depending on your data. The actual commit
> doesn't need a notable amount of memory.
>
> hope this helps
>
> simon
>
> On Mon, Oct 24, 2011 at 7:38 PM, Jaeger, Jay - DOT
> <Ja...@dot.wi.gov> wrote:
> > I have not spent a lot of time researching it, but one would expect that
> the OS RAM requirement for optimization of an index to be minimal.
> >
> > My understanding is that during optimization an essentially new index is
> built. Once complete it switches out the indexes and will throw away the
> old one. (In Windows it may not throw away the old one until the next
> Commit).
> >
> > JRJ
> >
> > -----Original Message-----
> > From: Sujatha Arun [mailto:suja.arun@gmail.com]
> > Sent: Friday, October 21, 2011 12:10 AM
> > To: solr-user@lucene.apache.org
> > Subject: Re: Optimization /Commit memory
> >
> > Just one more thing ,when we are talking about Optimization , we
> > are referring to HD free space for replicating the index (2 or 3
> times
> > the index size ) .what is role of RAM (OS) here?
> >
> > Regards
> > Suajtha
> >
> > On Fri, Oct 21, 2011 at 10:12 AM, Sujatha Arun <su...@gmail.com>
> wrote:
> >
> >> Thanks that helps.
> >>
> >> Regards
> >> Sujatha
> >>
> >>
> >> On Thu, Oct 20, 2011 at 6:23 PM, Jaeger, Jay - DOT <
> Jay.Jaeger@dot.wi.gov>wrote:
> >>
> >>> Well, since the OS RAM includes the JVM RAM, that is part of your
> >>> requirement, yes? Aside from the JVM and normal OS requirements, all
> you
> >>> need OS RAM for is file caching. Thus, for updates, the OS RAM is not
> a
> >>> major factor. For searches, you want sufficient OS RAM to cache enough
> of
> >>> the index to get the query performance you need, and to cache queries
> inside
> >>> the JVM if you get a lot of repeat queries (see solrconfig.xml for the
> >>> various caches: we have not played with them much). So, the amount of
> RAM
> >>> necessary for that is very much dependent upon the size of your index,
> so I
> >>> cannot give you a simple number.
> >>>
> >>> You seem to believe that you have to have sufficient memory to have the
> >>> entire index in memory. Except where extremely high performance is
> >>> required, I have not found that to be the case.
> >>>
> >>> This is just one of those "your mileage may vary" things. There is not
> a
> >>> single answer or formula that fits every situation.
> >>>
> >>> JRJ
> >>>
> >>> -----Original Message-----
> >>> From: Sujatha Arun [mailto:suja.arun@gmail.com]
> >>> Sent: Wednesday, October 19, 2011 11:58 PM
> >>> To: solr-user@lucene.apache.org
> >>> Subject: Re: Optimization /Commit memory
> >>>
> >>> Thanks Jay ,
> >>>
> >>> I was trying to compute the *OS RAM requirement* *not JVM RAM* for a
> 14
> >>> GB
> >>> Index [cumulative Index size of all Instances].And I put it thus -
> >>>
> >>> Requirement of Operating System RAM for an Index of 14GB is - Index
> >>> Size
> >>> + 3 Times the maximum Index Size of Individual Instance for Optimize .
> >>>
> >>> That is to say ,I have several Instances ,combined Index Size is 14GB
> >>> .Maximum Individual Index Size is 2.5GB .so My requirement for OS RAM
> is
> >>> 14GB +3 * 2.5 GB ~ = 22GB.
> >>>
> >>> Correct?
> >>>
> >>> Regards
> >>> Sujatha
> >>>
> >>>
> >>>
> >>> On Thu, Oct 20, 2011 at 3:45 AM, Jaeger, Jay - DOT <
> Jay.Jaeger@dot.wi.gov
> >>> >wrote:
> >>>
> >>> > Commit does not particularly spike disk or memory usage, unless you
> are
> >>> > adding a very large number of documents between commits. A commit
> can
> >>> cause
> >>> > a need to merge indexes, which can increase disk space temporarily.
> An
> >>> > optimize is *likely* to merge indexes, which will usually increase
> disk
> >>> > space temporarily.
> >>> >
> >>> > How much disk space depends very much upon how big your index is in
> the
> >>> > first place. A 2 to 3 times factor of the sum of your peak index
> file
> >>> size
> >>> > seems safe, to me.
> >>> >
> >>> > Solr uses only modest amounts of memory for the JVM for this stuff.
> >>> >
> >>> > JRJ
> >>> >
> >>> > -----Original Message-----
> >>> > From: Sujatha Arun [mailto:suja.arun@gmail.com]
> >>> > Sent: Wednesday, October 19, 2011 4:04 AM
> >>> > To: solr-user@lucene.apache.org
> >>> > Subject: Optimization /Commit memory
> >>> >
> >>> > Do we require 2 or 3 Times OS RAM memory or Hard Disk Space while
> >>> > performing Commit or Optimize or Both?
> >>> >
> >>> > what is the requirement in terms of size of RAM and HD for commit
> and
> >>> > Optimize
> >>> >
> >>> > Regards
> >>> > Sujatha
> >>> >
> >>>
> >>
> >>
> >
>
Re: Optimization /Commit memory
Posted by Simon Willnauer <si...@googlemail.com>.
RAM costs during optimize / merge is generally low. Optimize is
basically a merge of all segments into one, however there are
exceptions. Lucene streams existing segments from disk and serializes
the new segment on the fly. When you optimize or in general when you
merge segments you need disk space for the "source" segments and the
"targed" (merged) segment.
If you use CompoundFileSystem (CFS) you need to additional space once
the merge is done and your files are packed into the CFS which is
basically the size of the "target" (merged) segment. Once the merge is
done lucene can free the diskspace unless you have an IndexReader open
that references those segments (lucene keeps track of these files and
frees diskspace once possible).
That said, I think you should use optimize very very rarely. Usually
if you document collection is rarely changing optimize is useful and
reasonable once in a while. if you collection is constantly changing
you should rely on the merge policy to balance the number of segments
for you in the background. Lucene 3.4 has a nice improved
TieredMergePolicy that does a great job. (previous version are also
good - just saying)
A commit is basically flushing the segment you have in memory
(IndexWriter memory) to disk. compression ratio can be up to 30% of
the ram cost or even more depending on your data. The actual commit
doesn't need a notable amount of memory.
hope this helps
simon
On Mon, Oct 24, 2011 at 7:38 PM, Jaeger, Jay - DOT
<Ja...@dot.wi.gov> wrote:
> I have not spent a lot of time researching it, but one would expect that the OS RAM requirement for optimization of an index to be minimal.
>
> My understanding is that during optimization an essentially new index is built. Once complete it switches out the indexes and will throw away the old one. (In Windows it may not throw away the old one until the next Commit).
>
> JRJ
>
> -----Original Message-----
> From: Sujatha Arun [mailto:suja.arun@gmail.com]
> Sent: Friday, October 21, 2011 12:10 AM
> To: solr-user@lucene.apache.org
> Subject: Re: Optimization /Commit memory
>
> Just one more thing ,when we are talking about Optimization , we
> are referring to HD free space for replicating the index (2 or 3 times
> the index size ) .what is role of RAM (OS) here?
>
> Regards
> Suajtha
>
> On Fri, Oct 21, 2011 at 10:12 AM, Sujatha Arun <su...@gmail.com> wrote:
>
>> Thanks that helps.
>>
>> Regards
>> Sujatha
>>
>>
>> On Thu, Oct 20, 2011 at 6:23 PM, Jaeger, Jay - DOT <Ja...@dot.wi.gov>wrote:
>>
>>> Well, since the OS RAM includes the JVM RAM, that is part of your
>>> requirement, yes? Aside from the JVM and normal OS requirements, all you
>>> need OS RAM for is file caching. Thus, for updates, the OS RAM is not a
>>> major factor. For searches, you want sufficient OS RAM to cache enough of
>>> the index to get the query performance you need, and to cache queries inside
>>> the JVM if you get a lot of repeat queries (see solrconfig.xml for the
>>> various caches: we have not played with them much). So, the amount of RAM
>>> necessary for that is very much dependent upon the size of your index, so I
>>> cannot give you a simple number.
>>>
>>> You seem to believe that you have to have sufficient memory to have the
>>> entire index in memory. Except where extremely high performance is
>>> required, I have not found that to be the case.
>>>
>>> This is just one of those "your mileage may vary" things. There is not a
>>> single answer or formula that fits every situation.
>>>
>>> JRJ
>>>
>>> -----Original Message-----
>>> From: Sujatha Arun [mailto:suja.arun@gmail.com]
>>> Sent: Wednesday, October 19, 2011 11:58 PM
>>> To: solr-user@lucene.apache.org
>>> Subject: Re: Optimization /Commit memory
>>>
>>> Thanks Jay ,
>>>
>>> I was trying to compute the *OS RAM requirement* *not JVM RAM* for a 14
>>> GB
>>> Index [cumulative Index size of all Instances].And I put it thus -
>>>
>>> Requirement of Operating System RAM for an Index of 14GB is - Index
>>> Size
>>> + 3 Times the maximum Index Size of Individual Instance for Optimize .
>>>
>>> That is to say ,I have several Instances ,combined Index Size is 14GB
>>> .Maximum Individual Index Size is 2.5GB .so My requirement for OS RAM is
>>> 14GB +3 * 2.5 GB ~ = 22GB.
>>>
>>> Correct?
>>>
>>> Regards
>>> Sujatha
>>>
>>>
>>>
>>> On Thu, Oct 20, 2011 at 3:45 AM, Jaeger, Jay - DOT <Jay.Jaeger@dot.wi.gov
>>> >wrote:
>>>
>>> > Commit does not particularly spike disk or memory usage, unless you are
>>> > adding a very large number of documents between commits. A commit can
>>> cause
>>> > a need to merge indexes, which can increase disk space temporarily. An
>>> > optimize is *likely* to merge indexes, which will usually increase disk
>>> > space temporarily.
>>> >
>>> > How much disk space depends very much upon how big your index is in the
>>> > first place. A 2 to 3 times factor of the sum of your peak index file
>>> size
>>> > seems safe, to me.
>>> >
>>> > Solr uses only modest amounts of memory for the JVM for this stuff.
>>> >
>>> > JRJ
>>> >
>>> > -----Original Message-----
>>> > From: Sujatha Arun [mailto:suja.arun@gmail.com]
>>> > Sent: Wednesday, October 19, 2011 4:04 AM
>>> > To: solr-user@lucene.apache.org
>>> > Subject: Optimization /Commit memory
>>> >
>>> > Do we require 2 or 3 Times OS RAM memory or Hard Disk Space while
>>> > performing Commit or Optimize or Both?
>>> >
>>> > what is the requirement in terms of size of RAM and HD for commit and
>>> > Optimize
>>> >
>>> > Regards
>>> > Sujatha
>>> >
>>>
>>
>>
>
RE: Optimization /Commit memory
Posted by "Jaeger, Jay - DOT" <Ja...@dot.wi.gov>.
I have not spent a lot of time researching it, but one would expect that the OS RAM requirement for optimization of an index to be minimal.
My understanding is that during optimization an essentially new index is built. Once complete it switches out the indexes and will throw away the old one. (In Windows it may not throw away the old one until the next Commit).
JRJ
-----Original Message-----
From: Sujatha Arun [mailto:suja.arun@gmail.com]
Sent: Friday, October 21, 2011 12:10 AM
To: solr-user@lucene.apache.org
Subject: Re: Optimization /Commit memory
Just one more thing ,when we are talking about Optimization , we
are referring to HD free space for replicating the index (2 or 3 times
the index size ) .what is role of RAM (OS) here?
Regards
Suajtha
On Fri, Oct 21, 2011 at 10:12 AM, Sujatha Arun <su...@gmail.com> wrote:
> Thanks that helps.
>
> Regards
> Sujatha
>
>
> On Thu, Oct 20, 2011 at 6:23 PM, Jaeger, Jay - DOT <Ja...@dot.wi.gov>wrote:
>
>> Well, since the OS RAM includes the JVM RAM, that is part of your
>> requirement, yes? Aside from the JVM and normal OS requirements, all you
>> need OS RAM for is file caching. Thus, for updates, the OS RAM is not a
>> major factor. For searches, you want sufficient OS RAM to cache enough of
>> the index to get the query performance you need, and to cache queries inside
>> the JVM if you get a lot of repeat queries (see solrconfig.xml for the
>> various caches: we have not played with them much). So, the amount of RAM
>> necessary for that is very much dependent upon the size of your index, so I
>> cannot give you a simple number.
>>
>> You seem to believe that you have to have sufficient memory to have the
>> entire index in memory. Except where extremely high performance is
>> required, I have not found that to be the case.
>>
>> This is just one of those "your mileage may vary" things. There is not a
>> single answer or formula that fits every situation.
>>
>> JRJ
>>
>> -----Original Message-----
>> From: Sujatha Arun [mailto:suja.arun@gmail.com]
>> Sent: Wednesday, October 19, 2011 11:58 PM
>> To: solr-user@lucene.apache.org
>> Subject: Re: Optimization /Commit memory
>>
>> Thanks Jay ,
>>
>> I was trying to compute the *OS RAM requirement* *not JVM RAM* for a 14
>> GB
>> Index [cumulative Index size of all Instances].And I put it thus -
>>
>> Requirement of Operating System RAM for an Index of 14GB is - Index
>> Size
>> + 3 Times the maximum Index Size of Individual Instance for Optimize .
>>
>> That is to say ,I have several Instances ,combined Index Size is 14GB
>> .Maximum Individual Index Size is 2.5GB .so My requirement for OS RAM is
>> 14GB +3 * 2.5 GB ~ = 22GB.
>>
>> Correct?
>>
>> Regards
>> Sujatha
>>
>>
>>
>> On Thu, Oct 20, 2011 at 3:45 AM, Jaeger, Jay - DOT <Jay.Jaeger@dot.wi.gov
>> >wrote:
>>
>> > Commit does not particularly spike disk or memory usage, unless you are
>> > adding a very large number of documents between commits. A commit can
>> cause
>> > a need to merge indexes, which can increase disk space temporarily. An
>> > optimize is *likely* to merge indexes, which will usually increase disk
>> > space temporarily.
>> >
>> > How much disk space depends very much upon how big your index is in the
>> > first place. A 2 to 3 times factor of the sum of your peak index file
>> size
>> > seems safe, to me.
>> >
>> > Solr uses only modest amounts of memory for the JVM for this stuff.
>> >
>> > JRJ
>> >
>> > -----Original Message-----
>> > From: Sujatha Arun [mailto:suja.arun@gmail.com]
>> > Sent: Wednesday, October 19, 2011 4:04 AM
>> > To: solr-user@lucene.apache.org
>> > Subject: Optimization /Commit memory
>> >
>> > Do we require 2 or 3 Times OS RAM memory or Hard Disk Space while
>> > performing Commit or Optimize or Both?
>> >
>> > what is the requirement in terms of size of RAM and HD for commit and
>> > Optimize
>> >
>> > Regards
>> > Sujatha
>> >
>>
>
>
Re: Optimization /Commit memory
Posted by Sujatha Arun <su...@gmail.com>.
Just one more thing ,when we are talking about Optimization , we
are referring to HD free space for replicating the index (2 or 3 times
the index size ) .what is role of RAM (OS) here?
Regards
Suajtha
On Fri, Oct 21, 2011 at 10:12 AM, Sujatha Arun <su...@gmail.com> wrote:
> Thanks that helps.
>
> Regards
> Sujatha
>
>
> On Thu, Oct 20, 2011 at 6:23 PM, Jaeger, Jay - DOT <Ja...@dot.wi.gov>wrote:
>
>> Well, since the OS RAM includes the JVM RAM, that is part of your
>> requirement, yes? Aside from the JVM and normal OS requirements, all you
>> need OS RAM for is file caching. Thus, for updates, the OS RAM is not a
>> major factor. For searches, you want sufficient OS RAM to cache enough of
>> the index to get the query performance you need, and to cache queries inside
>> the JVM if you get a lot of repeat queries (see solrconfig.xml for the
>> various caches: we have not played with them much). So, the amount of RAM
>> necessary for that is very much dependent upon the size of your index, so I
>> cannot give you a simple number.
>>
>> You seem to believe that you have to have sufficient memory to have the
>> entire index in memory. Except where extremely high performance is
>> required, I have not found that to be the case.
>>
>> This is just one of those "your mileage may vary" things. There is not a
>> single answer or formula that fits every situation.
>>
>> JRJ
>>
>> -----Original Message-----
>> From: Sujatha Arun [mailto:suja.arun@gmail.com]
>> Sent: Wednesday, October 19, 2011 11:58 PM
>> To: solr-user@lucene.apache.org
>> Subject: Re: Optimization /Commit memory
>>
>> Thanks Jay ,
>>
>> I was trying to compute the *OS RAM requirement* *not JVM RAM* for a 14
>> GB
>> Index [cumulative Index size of all Instances].And I put it thus -
>>
>> Requirement of Operating System RAM for an Index of 14GB is - Index
>> Size
>> + 3 Times the maximum Index Size of Individual Instance for Optimize .
>>
>> That is to say ,I have several Instances ,combined Index Size is 14GB
>> .Maximum Individual Index Size is 2.5GB .so My requirement for OS RAM is
>> 14GB +3 * 2.5 GB ~ = 22GB.
>>
>> Correct?
>>
>> Regards
>> Sujatha
>>
>>
>>
>> On Thu, Oct 20, 2011 at 3:45 AM, Jaeger, Jay - DOT <Jay.Jaeger@dot.wi.gov
>> >wrote:
>>
>> > Commit does not particularly spike disk or memory usage, unless you are
>> > adding a very large number of documents between commits. A commit can
>> cause
>> > a need to merge indexes, which can increase disk space temporarily. An
>> > optimize is *likely* to merge indexes, which will usually increase disk
>> > space temporarily.
>> >
>> > How much disk space depends very much upon how big your index is in the
>> > first place. A 2 to 3 times factor of the sum of your peak index file
>> size
>> > seems safe, to me.
>> >
>> > Solr uses only modest amounts of memory for the JVM for this stuff.
>> >
>> > JRJ
>> >
>> > -----Original Message-----
>> > From: Sujatha Arun [mailto:suja.arun@gmail.com]
>> > Sent: Wednesday, October 19, 2011 4:04 AM
>> > To: solr-user@lucene.apache.org
>> > Subject: Optimization /Commit memory
>> >
>> > Do we require 2 or 3 Times OS RAM memory or Hard Disk Space while
>> > performing Commit or Optimize or Both?
>> >
>> > what is the requirement in terms of size of RAM and HD for commit and
>> > Optimize
>> >
>> > Regards
>> > Sujatha
>> >
>>
>
>
Re: Optimization /Commit memory
Posted by Sujatha Arun <su...@gmail.com>.
Thanks that helps.
Regards
Sujatha
On Thu, Oct 20, 2011 at 6:23 PM, Jaeger, Jay - DOT <Ja...@dot.wi.gov>wrote:
> Well, since the OS RAM includes the JVM RAM, that is part of your
> requirement, yes? Aside from the JVM and normal OS requirements, all you
> need OS RAM for is file caching. Thus, for updates, the OS RAM is not a
> major factor. For searches, you want sufficient OS RAM to cache enough of
> the index to get the query performance you need, and to cache queries inside
> the JVM if you get a lot of repeat queries (see solrconfig.xml for the
> various caches: we have not played with them much). So, the amount of RAM
> necessary for that is very much dependent upon the size of your index, so I
> cannot give you a simple number.
>
> You seem to believe that you have to have sufficient memory to have the
> entire index in memory. Except where extremely high performance is
> required, I have not found that to be the case.
>
> This is just one of those "your mileage may vary" things. There is not a
> single answer or formula that fits every situation.
>
> JRJ
>
> -----Original Message-----
> From: Sujatha Arun [mailto:suja.arun@gmail.com]
> Sent: Wednesday, October 19, 2011 11:58 PM
> To: solr-user@lucene.apache.org
> Subject: Re: Optimization /Commit memory
>
> Thanks Jay ,
>
> I was trying to compute the *OS RAM requirement* *not JVM RAM* for a 14 GB
> Index [cumulative Index size of all Instances].And I put it thus -
>
> Requirement of Operating System RAM for an Index of 14GB is - Index Size
> + 3 Times the maximum Index Size of Individual Instance for Optimize .
>
> That is to say ,I have several Instances ,combined Index Size is 14GB
> .Maximum Individual Index Size is 2.5GB .so My requirement for OS RAM is
> 14GB +3 * 2.5 GB ~ = 22GB.
>
> Correct?
>
> Regards
> Sujatha
>
>
>
> On Thu, Oct 20, 2011 at 3:45 AM, Jaeger, Jay - DOT <Jay.Jaeger@dot.wi.gov
> >wrote:
>
> > Commit does not particularly spike disk or memory usage, unless you are
> > adding a very large number of documents between commits. A commit can
> cause
> > a need to merge indexes, which can increase disk space temporarily. An
> > optimize is *likely* to merge indexes, which will usually increase disk
> > space temporarily.
> >
> > How much disk space depends very much upon how big your index is in the
> > first place. A 2 to 3 times factor of the sum of your peak index file
> size
> > seems safe, to me.
> >
> > Solr uses only modest amounts of memory for the JVM for this stuff.
> >
> > JRJ
> >
> > -----Original Message-----
> > From: Sujatha Arun [mailto:suja.arun@gmail.com]
> > Sent: Wednesday, October 19, 2011 4:04 AM
> > To: solr-user@lucene.apache.org
> > Subject: Optimization /Commit memory
> >
> > Do we require 2 or 3 Times OS RAM memory or Hard Disk Space while
> > performing Commit or Optimize or Both?
> >
> > what is the requirement in terms of size of RAM and HD for commit and
> > Optimize
> >
> > Regards
> > Sujatha
> >
>
RE: Optimization /Commit memory
Posted by "Jaeger, Jay - DOT" <Ja...@dot.wi.gov>.
Well, since the OS RAM includes the JVM RAM, that is part of your requirement, yes? Aside from the JVM and normal OS requirements, all you need OS RAM for is file caching. Thus, for updates, the OS RAM is not a major factor. For searches, you want sufficient OS RAM to cache enough of the index to get the query performance you need, and to cache queries inside the JVM if you get a lot of repeat queries (see solrconfig.xml for the various caches: we have not played with them much). So, the amount of RAM necessary for that is very much dependent upon the size of your index, so I cannot give you a simple number.
You seem to believe that you have to have sufficient memory to have the entire index in memory. Except where extremely high performance is required, I have not found that to be the case.
This is just one of those "your mileage may vary" things. There is not a single answer or formula that fits every situation.
JRJ
-----Original Message-----
From: Sujatha Arun [mailto:suja.arun@gmail.com]
Sent: Wednesday, October 19, 2011 11:58 PM
To: solr-user@lucene.apache.org
Subject: Re: Optimization /Commit memory
Thanks Jay ,
I was trying to compute the *OS RAM requirement* *not JVM RAM* for a 14 GB
Index [cumulative Index size of all Instances].And I put it thus -
Requirement of Operating System RAM for an Index of 14GB is - Index Size
+ 3 Times the maximum Index Size of Individual Instance for Optimize .
That is to say ,I have several Instances ,combined Index Size is 14GB
.Maximum Individual Index Size is 2.5GB .so My requirement for OS RAM is
14GB +3 * 2.5 GB ~ = 22GB.
Correct?
Regards
Sujatha
On Thu, Oct 20, 2011 at 3:45 AM, Jaeger, Jay - DOT <Ja...@dot.wi.gov>wrote:
> Commit does not particularly spike disk or memory usage, unless you are
> adding a very large number of documents between commits. A commit can cause
> a need to merge indexes, which can increase disk space temporarily. An
> optimize is *likely* to merge indexes, which will usually increase disk
> space temporarily.
>
> How much disk space depends very much upon how big your index is in the
> first place. A 2 to 3 times factor of the sum of your peak index file size
> seems safe, to me.
>
> Solr uses only modest amounts of memory for the JVM for this stuff.
>
> JRJ
>
> -----Original Message-----
> From: Sujatha Arun [mailto:suja.arun@gmail.com]
> Sent: Wednesday, October 19, 2011 4:04 AM
> To: solr-user@lucene.apache.org
> Subject: Optimization /Commit memory
>
> Do we require 2 or 3 Times OS RAM memory or Hard Disk Space while
> performing Commit or Optimize or Both?
>
> what is the requirement in terms of size of RAM and HD for commit and
> Optimize
>
> Regards
> Sujatha
>
Re: Optimization /Commit memory
Posted by Sujatha Arun <su...@gmail.com>.
Thanks Jay ,
I was trying to compute the *OS RAM requirement* *not JVM RAM* for a 14 GB
Index [cumulative Index size of all Instances].And I put it thus -
Requirement of Operating System RAM for an Index of 14GB is - Index Size
+ 3 Times the maximum Index Size of Individual Instance for Optimize .
That is to say ,I have several Instances ,combined Index Size is 14GB
.Maximum Individual Index Size is 2.5GB .so My requirement for OS RAM is
14GB +3 * 2.5 GB ~ = 22GB.
Correct?
Regards
Sujatha
On Thu, Oct 20, 2011 at 3:45 AM, Jaeger, Jay - DOT <Ja...@dot.wi.gov>wrote:
> Commit does not particularly spike disk or memory usage, unless you are
> adding a very large number of documents between commits. A commit can cause
> a need to merge indexes, which can increase disk space temporarily. An
> optimize is *likely* to merge indexes, which will usually increase disk
> space temporarily.
>
> How much disk space depends very much upon how big your index is in the
> first place. A 2 to 3 times factor of the sum of your peak index file size
> seems safe, to me.
>
> Solr uses only modest amounts of memory for the JVM for this stuff.
>
> JRJ
>
> -----Original Message-----
> From: Sujatha Arun [mailto:suja.arun@gmail.com]
> Sent: Wednesday, October 19, 2011 4:04 AM
> To: solr-user@lucene.apache.org
> Subject: Optimization /Commit memory
>
> Do we require 2 or 3 Times OS RAM memory or Hard Disk Space while
> performing Commit or Optimize or Both?
>
> what is the requirement in terms of size of RAM and HD for commit and
> Optimize
>
> Regards
> Sujatha
>
RE: Optimization /Commit memory
Posted by "Jaeger, Jay - DOT" <Ja...@dot.wi.gov>.
Commit does not particularly spike disk or memory usage, unless you are adding a very large number of documents between commits. A commit can cause a need to merge indexes, which can increase disk space temporarily. An optimize is *likely* to merge indexes, which will usually increase disk space temporarily.
How much disk space depends very much upon how big your index is in the first place. A 2 to 3 times factor of the sum of your peak index file size seems safe, to me.
Solr uses only modest amounts of memory for the JVM for this stuff.
JRJ
-----Original Message-----
From: Sujatha Arun [mailto:suja.arun@gmail.com]
Sent: Wednesday, October 19, 2011 4:04 AM
To: solr-user@lucene.apache.org
Subject: Optimization /Commit memory
Do we require 2 or 3 Times OS RAM memory or Hard Disk Space while
performing Commit or Optimize or Both?
what is the requirement in terms of size of RAM and HD for commit and
Optimize
Regards
Sujatha