You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Robert Stewart <Ro...@INFONGEN.COM> on 2008/08/16 13:40:01 UTC
windows file system cache
Anyone else run on Windows? We have index around 26 GB in size. Seems file system cache ends up taking up nearly all available RAM (26 GB out of 32 GB on 64-bit box). Lucene process is around 5 GB, so very little left over for queries, etc, and box starts swapping during searches. I think changing file system cache size setting is needed. Anyone else have same issue?
RE: windows file system cache
Posted by Robert Stewart <Ro...@INFONGEN.COM>.
Thank you for the help. It seems that just changing "memory usage" setting to "programs" from default of "system cache" fixed the issue. Now it takes only about 4 GB of system cache instead of 26 GB, and search performance is back to normal (fast).
-----Original Message-----
From: Mark Miller [mailto:markrmiller@gmail.com]
Sent: Monday, August 18, 2008 10:03 AM
To: java-user@lucene.apache.org
Subject: Re: windows file system cache
Mark Miller wrote:
> Mark Miller wrote:
>> Robert Stewart wrote:
>>> Anyone else run on Windows? We have index around 26 GB in size.
>>> Seems file system cache ends up taking up nearly all available RAM
>>> (26 GB out of 32 GB on 64-bit box). Lucene process is around 5 GB,
>>> so very little left over for queries, etc, and box starts swapping
>>> during searches. I think changing file system cache size setting is
>>> needed. Anyone else have same issue?
>>>
>>>
>> Hmmm...get more ram :)
>>
>> Windows 64-bit upped the default file system cache size from 1 gig to
>> 1 terabyte. Your feeling the awesome effects of that upgrade I think.
>>
>> There is an API call ( SetSystemFilecache() ) to override this - so
>> perhaps code up a C app to set it before running your Lucene app?
>>
>> - Mark
> You may actually be able to do it from the registry as well:
> http://support.microsoft.com/kb/892589 (don't use windows anymore so
> havn't confirmed)
>
> Info showing the change from 1 gig to 1 terabyte:
> http://support.microsoft.com/kb/294418
>
> You just want to set it to a certain percentage of what you got -
> leaving enough to do whatever your lucene app needs to do.
Found a great page about the problem using Domino:
http://www-1.ibm.com/support/docview.wss?uid=swg21270452
They appear to have compiled all the little bits of info that I have
seen elsewhere, and describe the problem being fixed just as you did.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: windows file system cache
Posted by Mark Miller <ma...@gmail.com>.
Mark Miller wrote:
> Mark Miller wrote:
>> Robert Stewart wrote:
>>> Anyone else run on Windows? We have index around 26 GB in size.
>>> Seems file system cache ends up taking up nearly all available RAM
>>> (26 GB out of 32 GB on 64-bit box). Lucene process is around 5 GB,
>>> so very little left over for queries, etc, and box starts swapping
>>> during searches. I think changing file system cache size setting is
>>> needed. Anyone else have same issue?
>>>
>>>
>> Hmmm...get more ram :)
>>
>> Windows 64-bit upped the default file system cache size from 1 gig to
>> 1 terabyte. Your feeling the awesome effects of that upgrade I think.
>>
>> There is an API call ( SetSystemFilecache() ) to override this - so
>> perhaps code up a C app to set it before running your Lucene app?
>>
>> - Mark
> You may actually be able to do it from the registry as well:
> http://support.microsoft.com/kb/892589 (don't use windows anymore so
> havn't confirmed)
>
> Info showing the change from 1 gig to 1 terabyte:
> http://support.microsoft.com/kb/294418
>
> You just want to set it to a certain percentage of what you got -
> leaving enough to do whatever your lucene app needs to do.
Found a great page about the problem using Domino:
http://www-1.ibm.com/support/docview.wss?uid=swg21270452
They appear to have compiled all the little bits of info that I have
seen elsewhere, and describe the problem being fixed just as you did.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: windows file system cache
Posted by Mark Miller <ma...@gmail.com>.
Mark Miller wrote:
> Robert Stewart wrote:
>> Anyone else run on Windows? We have index around 26 GB in size.
>> Seems file system cache ends up taking up nearly all available RAM
>> (26 GB out of 32 GB on 64-bit box). Lucene process is around 5 GB,
>> so very little left over for queries, etc, and box starts swapping
>> during searches. I think changing file system cache size setting is
>> needed. Anyone else have same issue?
>>
>>
> Hmmm...get more ram :)
>
> Windows 64-bit upped the default file system cache size from 1 gig to
> 1 terabyte. Your feeling the awesome effects of that upgrade I think.
>
> There is an API call ( SetSystemFilecache() ) to override this - so
> perhaps code up a C app to set it before running your Lucene app?
>
> - Mark
You may actually be able to do it from the registry as well:
http://support.microsoft.com/kb/892589 (don't use windows anymore so
havn't confirmed)
Info showing the change from 1 gig to 1 terabyte:
http://support.microsoft.com/kb/294418
You just want to set it to a certain percentage of what you got -
leaving enough to do whatever your lucene app needs to do.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: windows file system cache
Posted by Mark Miller <ma...@gmail.com>.
Robert Stewart wrote:
> Anyone else run on Windows? We have index around 26 GB in size. Seems file system cache ends up taking up nearly all available RAM (26 GB out of 32 GB on 64-bit box). Lucene process is around 5 GB, so very little left over for queries, etc, and box starts swapping during searches. I think changing file system cache size setting is needed. Anyone else have same issue?
>
>
Hmmm...get more ram :)
Windows 64-bit upped the default file system cache size from 1 gig to 1
terabyte. Your feeling the awesome effects of that upgrade I think.
There is an API call ( SetSystemFilecache() ) to override this - so
perhaps code up a C app to set it before running your Lucene app?
- Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: windows file system cache
Posted by Michael McCandless <lu...@mikemccandless.com>.
Toke Eskildsen wrote:
>> Lucene process is around 5 GB, so very little left over for queries,
>> etc, and box starts swapping during searches.
>
> Not so fine and also unexpected. Are you sure that what you're
> seeing is
> swapping and not just flushing of the write-cache? Are you observing
> the
> disk-activity only when your switching from writer to reader or does
> it
> continue to be a problem with subsequent searches?
On Linux, at least, the OS will choose to evict "real" pages (swap
them out to disk) and then use the RAM for the IO cache instead. This
then causes lots of paging when less-often-used parts of your app need
to access memory.
You can tune the "swappiness" kernel parameter (http://kerneltrap.org/node/3000
) to change how aggressively it does this. But I've never heard of a
corresponding way to tune Windows.
Mike
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: windows file system cache
Posted by Toke Eskildsen <te...@statsbiblioteket.dk>.
On Sat, 2008-08-16 at 07:40 -0400, Robert Stewart wrote:
> Anyone else run on Windows? We have index around 26 GB in size.
> Seems file system cache ends up taking up nearly all available RAM
> (26 GB out of 32 GB on 64-bit box).
Sounds fine so far. If the RAM isn't used for anything else, the system
should use it for caching.
> Lucene process is around 5 GB, so very little left over for queries,
> etc, and box starts swapping during searches.
Not so fine and also unexpected. Are you sure that what you're seeing is
swapping and not just flushing of the write-cache? Are you observing the
disk-activity only when your switching from writer to reader or does it
continue to be a problem with subsequent searches?
Have you profiled the JVM memory usage? Do you have the same problem if
you specify 6GB as the ceiling?
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org