[jira] Updated: (LUCENE-1313) Realtime Search

["Jason Rutherglen (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/LUCENE-1313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jason Rutherglen updated LUCENE-1313:
-------------------------------------

    Attachment: LUCENE-1313.patch

* RAM buffer size is stored in the writer rather than set into
DocumentsWriter. This is due to the actual ram buffer limit in
NRT changing depending on the size of the ramdir. 

* NRTMergePolicy and IW.resolveRAMSegments merges all ram dir
segments to primaryDir (i.e. disk) when the ramDir is over
totalMax, or any new merges would put ramDir over totalMax.

* In DocumentsWriter we have a set limit on the buffer size
which is (tempMax - ramDirSize)/2. This keeps the total ram used
under the totalMax (or IW.maxBufferSize), while also keeping our
temporary ram usage under the tempMax amount. When DW.ramBuffer
limit is reached, it's auto flushed to the ramDir.

* All tests pass except TestIndexWriterRAMDir.testFSDirectory.
Will look into this further. When flushToRAM is on by default,
there seems to be deadlock in
org.apache.lucene.TestMergeSchedulerExternal, however when I
tried to see if there is any via jconsole by setting
ANT_OPTS="-Dcom.sun.management.jmxremote" I didn't see any. I'm
not sure if this is due to not connecting to the right process?
Or something else.

* Added testReadDocuments which insures we can read documents
we've flushed to disk. This essentially tests our ability to
simultaneously read and write documents to and from the
docstore. It seemd to work on Windows.

* I think there's more that can be done to more accurately
manage the RAM however I think the way it works is a good
starting point.



> Realtime Search
> ---------------
>
>                 Key: LUCENE-1313
>                 URL: https://issues.apache.org/jira/browse/LUCENE-1313
>             Project: Lucene - Java
>          Issue Type: New Feature
>          Components: Index
>    Affects Versions: 2.4.1
>            Reporter: Jason Rutherglen
>            Priority: Minor
>             Fix For: 2.9
>
>         Attachments: LUCENE-1313.jar, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, lucene-1313.patch, lucene-1313.patch, lucene-1313.patch, lucene-1313.patch
>
>
> Enable near realtime search in Lucene without external
> dependencies. When RAM NRT is enabled, the implementation adds a
> RAMDirectory to IndexWriter. Flushes go to the ramdir unless
> there is no available space. Merges are completed in the ram
> dir until there is no more available ram. 
> IW.optimize and IW.commit flush the ramdir to the primary
> directory, all other operations try to keep segments in ram
> until there is no more space.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org