Re: Java caching of low-level index data?

[Earwin Burrfoot <ea...@gmail.com>]

> I'm curious if anyone has thought about (or even tried) caching the low-level index data in Java, rather than
> in the OS.  For example, at the IndexInput level there could be an LRU cache of byte[] blocks, similar to
> how a RDBMS caches index pages.  (Conveniently, BufferedIndexInput already reads in 1k chunks.) You
> would reverse the advice above and instead make your JVM heap as large as possible (or at least large
> enough to achieve a desired speed/space tradeoff).
I did something along these lines. It sucks. Having big Java heaps
ends you up with insane GC times. Loading GB-sized files into a bunch
of byte[1024] also wastes memory. Best bet by now is to rely on
mmap/file cache.

> I think swappiness is exactly the configuration that tells Linux just
> how happily it should swapp out application memory for IO cache vs
> other IO cache for new IO cache.
swappiness is roughly the percentage of free memory after which OS
starts searching for pages suitable for paging out. If set to low
values, OS wakes up in near-OOM conditions. If set to high values, as
soon as OS decides (according to some heuristics) that page is
eligible for page-out, it goes to disk.


-- 
Kirill Zakharenko/Кирилл Захаренко (earwin@gmail.com)
Home / Mobile: +7 (495) 683-567-4 / +7 (903) 5-888-423
ICQ: 104465785

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org

Re: Java caching of low-level index data?

[Nigel <ni...@gmail.com>]

Thanks, that's good feedback!

Chris

On Mon, Aug 3, 2009 at 8:50 AM, Earwin Burrfoot <ea...@gmail.com> wrote:

> > I'm curious if anyone has thought about (or even tried) caching the
> low-level index data in Java, rather than
> > in the OS.  For example, at the IndexInput level there could be an LRU
> cache of byte[] blocks, similar to
> > how a RDBMS caches index pages.  (Conveniently, BufferedIndexInput
> already reads in 1k chunks.) You
> > would reverse the advice above and instead make your JVM heap as large as
> possible (or at least large
> > enough to achieve a desired speed/space tradeoff).
> I did something along these lines. It sucks. Having big Java heaps
> ends you up with insane GC times. Loading GB-sized files into a bunch
> of byte[1024] also wastes memory. Best bet by now is to rely on
> mmap/file cache.
>
> > I think swappiness is exactly the configuration that tells Linux just
> > how happily it should swapp out application memory for IO cache vs
> > other IO cache for new IO cache.
> swappiness is roughly the percentage of free memory after which OS
> starts searching for pages suitable for paging out. If set to low
> values, OS wakes up in near-OOM conditions. If set to high values, as
> soon as OS decides (according to some heuristics) that page is
> eligible for page-out, it goes to disk.
>
>