Re: DO NOT REPLY [Bug 35838] New: - Java NIO patch against Lucene 1.9

[DM Smith <dm...@gmail.com>]

I tried to use bugzilla to record the comment, but I haven't the id
yet. See below for comment:

On 7/23/05, bugzilla@apache.org <bu...@apache.org> wrote:
> DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG·
> RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT
> <http://issues.apache.org/bugzilla/show_bug.cgi?id=35838>.
> ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND·
> INSERTED IN THE BUG DATABASE.
> 
> http://issues.apache.org/bugzilla/show_bug.cgi?id=35838
> 
>            Summary: Java NIO patch against Lucene 1.9
>            Product: Lucene
>            Version: unspecified
>           Platform: All
>         OS/Version: All
>             Status: NEW
>           Severity: normal
>           Priority: P2
>          Component: Store
>         AssignedTo: java-dev@lucene.apache.org
>         ReportedBy: clamprecht@gmail.com
> 
> 
> Robert Engels previously submitted a patch against Lucene 1.4 for a Java NIO-
> based Directory implementation.  It also included some changes to FSDirectory
> to allow better concurrency when searching from multiple threads.  The
> complete thread is at:
> 
> http://mail-archives.apache.org/mod_mbox/lucene-java-dev/200505.mbox/%
> 3cLMENLAOACIBLMOIILNNNEEOEEPAA.rengels@ix.netcom.com%3e
> 
> This thread ended with Doug Cutting suggesting that someone port Robert's
> changes to the SVN trunk.  This is what I've done in this patch.
> 
> There are two parts to the patch.  The first part modifies FieldsReader,
> CompoundFileReader, and SegmentReader, to allow better concurrency when
> reading an index.  The second part includes the new NioFSDirectory
> implementation, and makes small changes to FSDirectory and IndexInput to
> accomodate this change.  I'll put a more detailed outline of the changes to
> each file in a separate message.
> 
> To use the new NioFSDirectory, set the system property
> org.apache.lucene.FSDirectory.class to
> org.apache.lucene.store.NioFSDirectory.  This will cause
> FSDirectory.getDirectory() to return an NioFSDirectory instance.  By default,
> NioFile limits the number of concurrent channels to 4, but you can override
> this by setting the system property org.apache.lucene.nio.channels.

It has been noted in another thread that System properties are being replaced.
Should this mechanism be used here or not?
Wouldn't a static setter on FSDirectory work as well?

> 
> I did some performance tests with these patches.  The biggest improvement came
> from the concurrency improvements.  NioFSDirectory performed about the same as
> FSDirectory (with the concurrency improvements).
> 
> I ran my tests under Fedora Core 1; uname -a reports:
> Linux myhost 2.4.22-1.2199.nptlsmp #1 SMP Wed Aug 4 11:48:29 EDT 2004 i686
> i686 i386 GNU/Linux
> 
> The machine is a dual xeon 2.8GHz with 4GB RAM, and the tests were run against
> a 9GB compound index file.  The tests were run "hot" -- with everything
> already cached by linux's filesystem cache.  The numbers are:
> 
> FSDirectory without patch:          13.3 searches per second
> FSDirectory WITH concurrency patch: 14.3 searches per second
> 
> Both tests were run with 6 concurrent threads, which gave the highest numbers
> in each case.  I suspect that the concurrency improvements would make a bigger
> difference on a more realistic test where the index isn't all cached in RAM
> already, since the I/O happens whild holding the sychronized lock.  Patches to
> follow...
> 
> Thoughts?
> 
> --
> Configure bugmail: http://issues.apache.org/bugzilla/userprefs.cgi?tab=email
> ------- You are receiving this mail because: -------
> You are the assignee for the bug, or are watching the assignee.
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
> 
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org