You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Brian Pinkerton (JIRA)" <ji...@apache.org> on 2007/12/10 01:24:43 UTC
[jira] Commented: (LUCENE-753) Use NIO positional read to avoid
synchronization in FSIndexInput
[ https://issues.apache.org/jira/browse/LUCENE-753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12549869 ]
Brian Pinkerton commented on LUCENE-753:
----------------------------------------
Most of my workloads would benefit by removing the synchronization in FSIndexInput, so I took a closer look at this issue. I found exactly the opposite results that Yonik did on two platforms that I use frequently in production (Solaris and Linux), and by a significant margin. I even get the same behavior on the Mac, though I'm not running Java6 there.
# uname -a
Linux xxx 2.6.9-22.0.1.ELsmp #1 SMP Tue Oct 18 18:39:27 EDT 2005 i686 i686 i386 GNU/Linux
# java -version
java version "1.6.0_02"
Java(TM) SE Runtime Environment (build 1.6.0_02-b05)
Java HotSpot(TM) Client VM (build 1.6.0_02-b05, mixed mode, sharing)
config: impl=ChannelPread serial=false nThreads=200 iterations=10 bufsize=1024 filelen=10485760
answer=0, ms=88543, MB/sec=236.85124741650947
config: impl=ClassicFile serial=false nThreads=200 iterations=10 bufsize=1024 filelen=10485760
answer=0, ms=150560, MB/sec=139.29011689691816
# uname -a
SunOS xxx 5.10 Generic_118844-26 i86pc i386 i86pc
# java -version
java version "1.6.0"
Java(TM) SE Runtime Environment (build 1.6.0-b105)
Java HotSpot(TM) Server VM (build 1.6.0-b105, mixed mode)
config: impl=ChannelPread serial=false nThreads=200 iterations=10 bufsize=1024 filelen=10485760
answer=0, ms=39621, MB/sec=529.3031473208652
config: impl=ClassicFile serial=false nThreads=200 iterations=10 bufsize=1024 filelen=10485760
answer=0, ms=119057, MB/sec=176.14688762525515
> Use NIO positional read to avoid synchronization in FSIndexInput
> ----------------------------------------------------------------
>
> Key: LUCENE-753
> URL: https://issues.apache.org/jira/browse/LUCENE-753
> Project: Lucene - Java
> Issue Type: New Feature
> Components: Store
> Reporter: Yonik Seeley
> Attachments: FileReadTest.java, FSIndexInput.patch, FSIndexInput.patch
>
>
> As suggested by Doug, we could use NIO pread to avoid synchronization on the underlying file.
> This could mitigate any MT performance drop caused by reducing the number of files in the index format.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: [jira] Commented: (LUCENE-753) Use NIO positional read to avoid synchronization in FSIndexInput
Posted by robert engels <re...@ix.netcom.com>.
I posted a bug on this to Sun a long while back.
NIO preads actually sync behind the scenes on some platforms. Using
multiple file descriptors is much faster.
See bug http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6265734
On Dec 9, 2007, at 6:24 PM, Brian Pinkerton (JIRA) wrote:
>
> [ https://issues.apache.org/jira/browse/LUCENE-753?
> page=com.atlassian.jira.plugin.system.issuetabpanels:comment-
> tabpanel#action_12549869 ]
>
> Brian Pinkerton commented on LUCENE-753:
> ----------------------------------------
>
> Most of my workloads would benefit by removing the synchronization
> in FSIndexInput, so I took a closer look at this issue. I found
> exactly the opposite results that Yonik did on two platforms that I
> use frequently in production (Solaris and Linux), and by a
> significant margin. I even get the same behavior on the Mac,
> though I'm not running Java6 there.
>
> # uname -a
> Linux xxx 2.6.9-22.0.1.ELsmp #1 SMP Tue Oct 18 18:39:27 EDT 2005
> i686 i686 i386 GNU/Linux
> # java -version
> java version "1.6.0_02"
> Java(TM) SE Runtime Environment (build 1.6.0_02-b05)
> Java HotSpot(TM) Client VM (build 1.6.0_02-b05, mixed mode, sharing)
>
> config: impl=ChannelPread serial=false nThreads=200 iterations=10
> bufsize=1024 filelen=10485760
> answer=0, ms=88543, MB/sec=236.85124741650947
> config: impl=ClassicFile serial=false nThreads=200 iterations=10
> bufsize=1024 filelen=10485760
> answer=0, ms=150560, MB/sec=139.29011689691816
>
>
>
> # uname -a
> SunOS xxx 5.10 Generic_118844-26 i86pc i386 i86pc
> # java -version
> java version "1.6.0"
> Java(TM) SE Runtime Environment (build 1.6.0-b105)
> Java HotSpot(TM) Server VM (build 1.6.0-b105, mixed mode)
>
> config: impl=ChannelPread serial=false nThreads=200 iterations=10
> bufsize=1024 filelen=10485760
> answer=0, ms=39621, MB/sec=529.3031473208652
>
> config: impl=ClassicFile serial=false nThreads=200 iterations=10
> bufsize=1024 filelen=10485760
> answer=0, ms=119057, MB/sec=176.14688762525515
>
>
>
>> Use NIO positional read to avoid synchronization in FSIndexInput
>> ----------------------------------------------------------------
>>
>> Key: LUCENE-753
>> URL: https://issues.apache.org/jira/browse/LUCENE-753
>> Project: Lucene - Java
>> Issue Type: New Feature
>> Components: Store
>> Reporter: Yonik Seeley
>> Attachments: FileReadTest.java, FSIndexInput.patch,
>> FSIndexInput.patch
>>
>>
>> As suggested by Doug, we could use NIO pread to avoid
>> synchronization on the underlying file.
>> This could mitigate any MT performance drop caused by reducing the
>> number of files in the index format.
>
> --
> This message is automatically generated by JIRA.
> -
> You can reply to this email to add a comment to the issue online.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org