how lucene search works in memory

[su...@zapak.co.in]

Hello,


I am trying for optimizing the searching by putting indexes onto
memory. RAMDirectory is not option for me, as I am transferring indexes
built to slave system to use.
So if u could let me know that how to place my indexes onto memory(m
thinking of using mmap) .For this I wanna know how exactly the lucene
searches indexes. Does it put all of the indexes onto memory, when we open
index searcher instance and then does searching.

Curious to know how the lucene searches actually work.Can u get me the
link.


Thanks and regards,
Suman




---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

Re: how lucene search works in memory

[Erick Erickson <er...@gmail.com>]

First, I'd be sure you need to. See the following:
http://wiki.apache.org/lucene-java/ImproveSearchingSpeed

<http://wiki.apache.org/lucene-java/ImproveSearchingSpeed>A lot of very
bright people have worked very hard at
optimizing Lucene's search *and* the op system
caching. I'd carefully examine my system first to see
if I'm making optimal use of what already exists before
assuming that a custom solution (that I have to maintain)
is the correct answer...

FWIW
Erick

On Tue, Mar 23, 2010 at 6:21 AM, <su...@zapak.co.in> wrote:

> Hello,
>
>
> I am trying for optimizing the searching by putting indexes onto
> memory. RAMDirectory is not option for me, as I am transferring indexes
> built to slave system to use.
> So if u could let me know that how to place my indexes onto memory(m
> thinking of using mmap) .For this I wanna know how exactly the lucene
> searches indexes. Does it put all of the indexes onto memory, when we open
> index searcher instance and then does searching.
>
> Curious to know how the lucene searches actually work.Can u get me the
> link.
>
>
> Thanks and regards,
> Suman
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

Re: how lucene search works in memory

[Anshum <an...@gmail.com>]

Hi Suman,
I couldn't find a link but talking about approaches to load an index into
memory would be :
1. Create a tmpfs partition and copy your index into the partition, open the
index reader/searcher from the tmpfs.
  * You would have to handle the copying/management of indexes in this case.
  * It is volatile i.e. would get wiped off in case the machine restarts.
  * Lets you create a partition of any size and therefore gives you
the flexibility to have an index of > 4G even in case of a 32 bit
machine/JVM architecture.
2. Use RAMFs, which would mean that the JVM copies the opened index onto its
memory(using and sharing the JVM's memory and address space)
  * Index management would reduce.
  * Index size would be limited to the maximum address space available to
the JVM process (In case of a 32 bit machine, it'd be 4G)
  * Would get wiped off when JVM goes off (Not affecting the Disk copy in
any manner though)

--
Anshum Gupta
Naukri Labs!
http://ai-cafe.blogspot.com

The facts expressed here belong to everybody, the opinions to me. The
distinction is yours to draw............


On Tue, Mar 23, 2010 at 3:51 PM, <su...@zapak.co.in> wrote:

> Hello,
>
>
> I am trying for optimizing the searching by putting indexes onto
> memory. RAMDirectory is not option for me, as I am transferring indexes
> built to slave system to use.
> So if u could let me know that how to place my indexes onto memory(m
> thinking of using mmap) .For this I wanna know how exactly the lucene
> searches indexes. Does it put all of the indexes onto memory, when we open
> index searcher instance and then does searching.
>
> Curious to know how the lucene searches actually work.Can u get me the
> link.
>
>
> Thanks and regards,
> Suman
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>