You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by John Wang <jo...@gmail.com> on 2010/05/02 00:38:09 UTC
Re: Nasty NIO behavior makes NIOFSDirectory silently close channel
We are seeing this issue as well in your production. (using Zoie on top of
lucene 2.9.1)
After some performance comparisons, we do NOT see performance gain with NIO,
rather these nasty ClosedChannelExceptions.
I think the performance gains ppl are seeing with 2.9.1 can be due to many
different things. From what we seen, they are not related to NIOFSDirectory.
Our solution is to avoid calling FSDirectory.open(), instead just call new
SimpleFSDirectory(). Is this safe?
-John
On Fri, Jan 29, 2010 at 12:32 PM, Mark Miller <ma...@gmail.com> wrote:
> Perhaps - one of the things they are supposed to be addressing is
> extendability.
>
> nio2 does have FileSystemProvider, which would actually allow you to
> create a custom channel !
>
> I have not dug in enough to know much more than that though.
>
> *But*, another really interesting thing is that in Java 7,
> FileDescriptors are ref counted ! (though users can't inc/dec).
>
> But, FileInputStream and OutputStream have a new constructor that takes
> a FileDescriptor.
>
> So possibly, you could just make one that sits around to keep the
> FileDescriptor valid, and get your channel off
> FileInputStream/FileOutputStream?
>
> And then if it goes down, make a new one using the FileDescriptor which
> was not actually closed because there was a still a ref to it.
>
> Possibly .... ;)
>
> Michael McCandless wrote:
> > Does anyone know if nio2 has improved this...?
> >
> > Mike
> >
> > On Fri, Jan 29, 2010 at 2:00 PM, Jason Rutherglen
> > <ja...@gmail.com> wrote:
> >
> >> Defaulting NIOFSDir could account for some of the recent speed
> >> improvements users have been reporting in Lucene 2.9. So removing it
> >> as a default could reverse those and people could then report Lucene
> >> 3.X has slowed...
> >>
> >> On Thu, Jan 28, 2010 at 5:24 AM, Michael McCandless
> >> <lu...@mikemccandless.com> wrote:
> >>
> >>> Bummer.
> >>>
> >>> So the only viable workarounds are 1) don't use Thread.interrupt (nor,
> >>> things like Future.cancel, which in turn use Thread.interrupt) with
> >>> NIOFSDir, or 2) we fix NIOFSDir to reopen the channel AND the app must
> >>> make a deletion policy that keeps a commit alive if any reader is
> >>> using it. Or, 3) don't use NIOFSDir!
> >>>
> >>> Mike
> >>>
> >>> On Thu, Jan 28, 2010 at 7:29 AM, Simon Willnauer
> >>> <si...@googlemail.com> wrote:
> >>>
> >>>> On Thu, Jan 28, 2010 at 12:43 PM, Michael McCandless
> >>>> <lu...@mikemccandless.com> wrote:
> >>>>
> >>>>> On Thu, Jan 28, 2010 at 6:38 AM, Uwe Schindler <uw...@thetaphi.de>
> wrote:
> >>>>>
> >>>>>
> >>>>>> So I checked the code of NIOFSIndexInput, my last comment was not
> really correct:
> >>>>>> NIOFSIndexInput extends SimpleFSIndexInput and that opens the RAF.
> In the ctor RAF.getChannel() is called. The RAF keeps open until the file is
> closed (and also the channel).
> >>>>>>
> >>>>>> So it's really simple to fix in my opinion, just call getChannel()
> again on this exception. Because the RAF should still be open?
> >>>>>>
> >>>> Short answer:
> >>>> public final FileChannel getChannel() {
> >>>> synchronized (this) {
> >>>> if (channel == null)
> >>>> channel = FileChannelImpl.open(fd, true, rw, this);
> >>>> return channel;
> >>>> }
> >>>> }
> >>>>
> >>>> this is not gonna work I tried it before. The RandomAccessFile buffers
> >>>> the channel!!
> >>>>
> >>>> simon
> >>>>
> >>>>> I think we need a definitive answer on what happens to the RAF when
> >>>>> the FileChannel was closed by Thread.Interrupt. Simon can you test
> >>>>> this?
> >>>>>
> >>>>> Mike
> >>>>>
> >>>>> ---------------------------------------------------------------------
> >>>>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> >>>>> For additional commands, e-mail: java-dev-help@lucene.apache.org
> >>>>>
> >>>>>
> >>>>>
> >>>> ---------------------------------------------------------------------
> >>>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> >>>> For additional commands, e-mail: java-dev-help@lucene.apache.org
> >>>>
> >>>>
> >>>>
> >>> ---------------------------------------------------------------------
> >>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> >>> For additional commands, e-mail: java-dev-help@lucene.apache.org
> >>>
> >>>
> >>>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> >> For additional commands, e-mail: java-dev-help@lucene.apache.org
> >>
> >>
> >>
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-dev-help@lucene.apache.org
> >
> >
>
>
> --
> - Mark
>
> http://www.lucidimagination.com
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>
Re: Nasty NIO behavior makes NIOFSDirectory silently close channel
Posted by John Wang <jo...@gmail.com>.
Re : HW concurrency.
8 core machine, avg latency around 30ms, so with 12 threads, more than
enough room.
-John
On Mon, May 3, 2010 at 10:44 AM, John Wang <jo...@gmail.com> wrote:
> Pretty much pure searching. Not reopening readers. Keeping the reader open
> for the test.
>
> -John
>
>
> On Mon, May 3, 2010 at 2:47 AM, Michael McCandless <
> lucene@mikemccandless.com> wrote:
>
>> This is strange -- it's completely opposite of what others have seen!
>>
>> Other have seen sizable concurrency gains when switching from
>> SimpleFSDir -> NIOFSDir, except on Windows where Sun's long standing
>> JVM bug (http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6265734)
>> prevents concurrency gains.
>>
>> There must be something different about the testing.
>>
>> John are you testing pure searching? Are you reopening readers as
>> well? How much HW concurrency do your machines have?
>>
>> Mike
>>
>> On Sun, May 2, 2010 at 11:53 PM, John Wang <jo...@gmail.com> wrote:
>> > Solaris and MacOs
>> >
>> > On Sun, May 2, 2010 at 4:47 PM, Mark Miller <ma...@gmail.com>
>> wrote:
>> >>
>> >> Interesting ... whats the OS?
>> >>
>> >> On Sun, May 2, 2010 at 7:12 PM, John Wang <jo...@gmail.com> wrote:
>> >>>
>> >>> 12 concurrent search threads in a stress environment. (with about 5M
>> doc
>> >>> index)
>> >>> -John
>> >>>
>> >>> On Sun, May 2, 2010 at 3:13 PM, Mark Miller <ma...@gmail.com>
>> >>> wrote:
>> >>>>
>> >>>> Perfectly safe to use SimpleFSDirectory. When you measure the
>> >>>> performance, are you measuring a lot of concurrent requests? That's
>> where
>> >>>> you should see the win.
>> >>>>
>> >>>> - Mark
>> >>>>
>> >>>> On 5/1/10 6:38 PM, John Wang wrote:
>> >>>>>
>> >>>>> We are seeing this issue as well in your production. (using Zoie on
>> top
>> >>>>> of lucene 2.9.1)
>> >>>>> After some performance comparisons, we do NOT see performance gain
>> with
>> >>>>> NIO, rather these nasty ClosedChannelExceptions.
>> >>>>>
>> >>>>> I think the performance gains ppl are seeing with 2.9.1 can be due
>> to
>> >>>>> many different things. From what we seen, they are not related to
>> >>>>> NIOFSDirectory.
>> >>>>>
>> >>>>> Our solution is to avoid calling FSDirectory.open(), instead just
>> call
>> >>>>> new SimpleFSDirectory(). Is this safe?
>> >>>>>
>> >>>>> -John
>> >>>>>
>> >>>>> On Fri, Jan 29, 2010 at 12:32 PM, Mark Miller <
>> markrmiller@gmail.com
>> >>>>> <ma...@gmail.com>> wrote:
>> >>>>>
>> >>>>> Perhaps - one of the things they are supposed to be addressing is
>> >>>>> extendability.
>> >>>>>
>> >>>>> nio2 does have FileSystemProvider, which would actually allow you
>> to
>> >>>>> create a custom channel !
>> >>>>>
>> >>>>> I have not dug in enough to know much more than that though.
>> >>>>>
>> >>>>> *But*, another really interesting thing is that in Java 7,
>> >>>>> FileDescriptors are ref counted ! (though users can't inc/dec).
>> >>>>>
>> >>>>> But, FileInputStream and OutputStream have a new constructor that
>> >>>>> takes
>> >>>>> a FileDescriptor.
>> >>>>>
>> >>>>> So possibly, you could just make one that sits around to keep the
>> >>>>> FileDescriptor valid, and get your channel off
>> >>>>> FileInputStream/FileOutputStream?
>> >>>>>
>> >>>>> And then if it goes down, make a new one using the FileDescriptor
>> >>>>> which
>> >>>>> was not actually closed because there was a still a ref to it.
>> >>>>>
>> >>>>> Possibly .... ;)
>> >>>>>
>> >>>>> Michael McCandless wrote:
>> >>>>> > Does anyone know if nio2 has improved this...?
>> >>>>> >
>> >>>>> > Mike
>> >>>>> >
>> >>>>> > On Fri, Jan 29, 2010 at 2:00 PM, Jason Rutherglen
>> >>>>> > <jason.rutherglen@gmail.com <mailto:
>> jason.rutherglen@gmail.com>>
>> >>>>> wrote:
>> >>>>> >
>> >>>>> >> Defaulting NIOFSDir could account for some of the recent
>> speed
>> >>>>> >> improvements users have been reporting in Lucene 2.9. So
>> >>>>> removing it
>> >>>>> >> as a default could reverse those and people could then report
>> >>>>> Lucene
>> >>>>> >> 3.X has slowed...
>> >>>>> >>
>> >>>>> >> On Thu, Jan 28, 2010 at 5:24 AM, Michael McCandless
>> >>>>> >> <lucene@mikemccandless.com <mailto:lucene@mikemccandless.com
>> >>
>> >>>>> wrote:
>> >>>>> >>
>> >>>>> >>> Bummer.
>> >>>>> >>>
>> >>>>> >>> So the only viable workarounds are 1) don't use
>> >>>>> Thread.interrupt (nor,
>> >>>>> >>> things like Future.cancel, which in turn use
>> Thread.interrupt)
>> >>>>> with
>> >>>>> >>> NIOFSDir, or 2) we fix NIOFSDir to reopen the channel AND
>> the
>> >>>>> app must
>> >>>>> >>> make a deletion policy that keeps a commit alive if any
>> reader
>> >>>>> is
>> >>>>> >>> using it. Or, 3) don't use NIOFSDir!
>> >>>>> >>>
>> >>>>> >>> Mike
>> >>>>> >>>
>> >>>>> >>> On Thu, Jan 28, 2010 at 7:29 AM, Simon Willnauer
>> >>>>> >>> <simon.willnauer@googlemail.com
>> >>>>> <ma...@googlemail.com>> wrote:
>> >>>>> >>>
>> >>>>> >>>> On Thu, Jan 28, 2010 at 12:43 PM, Michael McCandless
>> >>>>> >>>> <lucene@mikemccandless.com <mailto:
>> lucene@mikemccandless.com>>
>> >>>>> wrote:
>> >>>>> >>>>
>> >>>>> >>>>> On Thu, Jan 28, 2010 at 6:38 AM, Uwe Schindler
>> >>>>> <uwe@thetaphi.de <ma...@thetaphi.de>> wrote:
>> >>>>> >>>>>
>> >>>>> >>>>>
>> >>>>> >>>>>> So I checked the code of NIOFSIndexInput, my last comment
>> >>>>> was not really correct:
>> >>>>> >>>>>> NIOFSIndexInput extends SimpleFSIndexInput and that opens
>> >>>>> the RAF. In the ctor RAF.getChannel() is called. The RAF keeps
>> open
>> >>>>> until the file is closed (and also the channel).
>> >>>>> >>>>>>
>> >>>>> >>>>>> So it's really simple to fix in my opinion, just call
>> >>>>> getChannel() again on this exception. Because the RAF should
>> still
>> >>>>> be open?
>> >>>>> >>>>>>
>> >>>>> >>>> Short answer:
>> >>>>> >>>> public final FileChannel getChannel() {
>> >>>>> >>>> synchronized (this) {
>> >>>>> >>>> if (channel == null)
>> >>>>> >>>> channel = FileChannelImpl.open(fd, true, rw,
>> >>>>> this);
>> >>>>> >>>> return channel;
>> >>>>> >>>> }
>> >>>>> >>>> }
>> >>>>> >>>>
>> >>>>> >>>> this is not gonna work I tried it before. The
>> RandomAccessFile
>> >>>>> buffers
>> >>>>> >>>> the channel!!
>> >>>>> >>>>
>> >>>>> >>>> simon
>> >>>>> >>>>
>> >>>>> >>>>> I think we need a definitive answer on what happens to the
>> >>>>> RAF when
>> >>>>> >>>>> the FileChannel was closed by Thread.Interrupt. Simon can
>> >>>>> you test
>> >>>>> >>>>> this?
>> >>>>> >>>>>
>> >>>>> >>>>> Mike
>> >>>>> >>>>>
>> >>>>> >>>>>
>> >>>>>
>> >>>>>
>> ---------------------------------------------------------------------
>> >>>>> >>>>> To unsubscribe, e-mail:
>> >>>>> java-dev-unsubscribe@lucene.apache.org
>> >>>>> <ma...@lucene.apache.org>
>> >>>>> >>>>> For additional commands, e-mail:
>> >>>>> java-dev-help@lucene.apache.org
>> >>>>> <ma...@lucene.apache.org>
>> >>>>> >>>>>
>> >>>>> >>>>>
>> >>>>> >>>>>
>> >>>>> >>>>
>> >>>>>
>> >>>>>
>> ---------------------------------------------------------------------
>> >>>>> >>>> To unsubscribe, e-mail:
>> java-dev-unsubscribe@lucene.apache.org
>> >>>>> <ma...@lucene.apache.org>
>> >>>>> >>>> For additional commands, e-mail:
>> >>>>> java-dev-help@lucene.apache.org
>> >>>>> <ma...@lucene.apache.org>
>> >>>>> >>>>
>> >>>>> >>>>
>> >>>>> >>>>
>> >>>>> >>>
>> >>>>>
>> >>>>>
>> ---------------------------------------------------------------------
>> >>>>> >>> To unsubscribe, e-mail:
>> java-dev-unsubscribe@lucene.apache.org
>> >>>>> <ma...@lucene.apache.org>
>> >>>>> >>> For additional commands, e-mail:
>> >>>>> java-dev-help@lucene.apache.org
>> >>>>> <ma...@lucene.apache.org>
>> >>>>> >>>
>> >>>>> >>>
>> >>>>> >>>
>> >>>>> >>
>> >>>>>
>> >>>>>
>> ---------------------------------------------------------------------
>> >>>>> >> To unsubscribe, e-mail:
>> java-dev-unsubscribe@lucene.apache.org
>> >>>>> <ma...@lucene.apache.org>
>> >>>>> >> For additional commands, e-mail:
>> java-dev-help@lucene.apache.org
>> >>>>> <ma...@lucene.apache.org>
>> >>>>> >>
>> >>>>> >>
>> >>>>> >>
>> >>>>> >
>> >>>>> >
>> >>>>>
>> ---------------------------------------------------------------------
>> >>>>> > To unsubscribe, e-mail:
>> java-dev-unsubscribe@lucene.apache.org
>> >>>>> <ma...@lucene.apache.org>
>> >>>>> > For additional commands, e-mail:
>> java-dev-help@lucene.apache.org
>> >>>>> <ma...@lucene.apache.org>
>> >>>>> >
>> >>>>> >
>> >>>>>
>> >>>>>
>> >>>>> --
>> >>>>> - Mark
>> >>>>>
>> >>>>> http://www.lucidimagination.com
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>>
>> ---------------------------------------------------------------------
>> >>>>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>> >>>>> <ma...@lucene.apache.org>
>> >>>>> For additional commands, e-mail: java-dev-help@lucene.apache.org
>> >>>>> <ma...@lucene.apache.org>
>> >>>>>
>> >>>>>
>> >>>>
>> >>>>
>> >>>> --
>> >>>> - Mark
>> >>>>
>> >>>> http://www.lucidimagination.com
>> >>>>
>> >>>> ---------------------------------------------------------------------
>> >>>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>> >>>> For additional commands, e-mail: dev-help@lucene.apache.org
>> >>>>
>> >>>
>> >>
>> >>
>> >>
>> >> --
>> >> - Mark
>> >>
>> >> http://www.lucidimagination.com
>> >>
>> >
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: dev-help@lucene.apache.org
>>
>>
>
Re: Nasty NIO behavior makes NIOFSDirectory silently close channel
Posted by John Wang <jo...@gmail.com>.
Pretty much pure searching. Not reopening readers. Keeping the reader open
for the test.
-John
On Mon, May 3, 2010 at 2:47 AM, Michael McCandless <
lucene@mikemccandless.com> wrote:
> This is strange -- it's completely opposite of what others have seen!
>
> Other have seen sizable concurrency gains when switching from
> SimpleFSDir -> NIOFSDir, except on Windows where Sun's long standing
> JVM bug (http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6265734)
> prevents concurrency gains.
>
> There must be something different about the testing.
>
> John are you testing pure searching? Are you reopening readers as
> well? How much HW concurrency do your machines have?
>
> Mike
>
> On Sun, May 2, 2010 at 11:53 PM, John Wang <jo...@gmail.com> wrote:
> > Solaris and MacOs
> >
> > On Sun, May 2, 2010 at 4:47 PM, Mark Miller <ma...@gmail.com>
> wrote:
> >>
> >> Interesting ... whats the OS?
> >>
> >> On Sun, May 2, 2010 at 7:12 PM, John Wang <jo...@gmail.com> wrote:
> >>>
> >>> 12 concurrent search threads in a stress environment. (with about 5M
> doc
> >>> index)
> >>> -John
> >>>
> >>> On Sun, May 2, 2010 at 3:13 PM, Mark Miller <ma...@gmail.com>
> >>> wrote:
> >>>>
> >>>> Perfectly safe to use SimpleFSDirectory. When you measure the
> >>>> performance, are you measuring a lot of concurrent requests? That's
> where
> >>>> you should see the win.
> >>>>
> >>>> - Mark
> >>>>
> >>>> On 5/1/10 6:38 PM, John Wang wrote:
> >>>>>
> >>>>> We are seeing this issue as well in your production. (using Zoie on
> top
> >>>>> of lucene 2.9.1)
> >>>>> After some performance comparisons, we do NOT see performance gain
> with
> >>>>> NIO, rather these nasty ClosedChannelExceptions.
> >>>>>
> >>>>> I think the performance gains ppl are seeing with 2.9.1 can be due to
> >>>>> many different things. From what we seen, they are not related to
> >>>>> NIOFSDirectory.
> >>>>>
> >>>>> Our solution is to avoid calling FSDirectory.open(), instead just
> call
> >>>>> new SimpleFSDirectory(). Is this safe?
> >>>>>
> >>>>> -John
> >>>>>
> >>>>> On Fri, Jan 29, 2010 at 12:32 PM, Mark Miller <markrmiller@gmail.com
> >>>>> <ma...@gmail.com>> wrote:
> >>>>>
> >>>>> Perhaps - one of the things they are supposed to be addressing is
> >>>>> extendability.
> >>>>>
> >>>>> nio2 does have FileSystemProvider, which would actually allow you
> to
> >>>>> create a custom channel !
> >>>>>
> >>>>> I have not dug in enough to know much more than that though.
> >>>>>
> >>>>> *But*, another really interesting thing is that in Java 7,
> >>>>> FileDescriptors are ref counted ! (though users can't inc/dec).
> >>>>>
> >>>>> But, FileInputStream and OutputStream have a new constructor that
> >>>>> takes
> >>>>> a FileDescriptor.
> >>>>>
> >>>>> So possibly, you could just make one that sits around to keep the
> >>>>> FileDescriptor valid, and get your channel off
> >>>>> FileInputStream/FileOutputStream?
> >>>>>
> >>>>> And then if it goes down, make a new one using the FileDescriptor
> >>>>> which
> >>>>> was not actually closed because there was a still a ref to it.
> >>>>>
> >>>>> Possibly .... ;)
> >>>>>
> >>>>> Michael McCandless wrote:
> >>>>> > Does anyone know if nio2 has improved this...?
> >>>>> >
> >>>>> > Mike
> >>>>> >
> >>>>> > On Fri, Jan 29, 2010 at 2:00 PM, Jason Rutherglen
> >>>>> > <jason.rutherglen@gmail.com <mailto:jason.rutherglen@gmail.com
> >>
> >>>>> wrote:
> >>>>> >
> >>>>> >> Defaulting NIOFSDir could account for some of the recent speed
> >>>>> >> improvements users have been reporting in Lucene 2.9. So
> >>>>> removing it
> >>>>> >> as a default could reverse those and people could then report
> >>>>> Lucene
> >>>>> >> 3.X has slowed...
> >>>>> >>
> >>>>> >> On Thu, Jan 28, 2010 at 5:24 AM, Michael McCandless
> >>>>> >> <lucene@mikemccandless.com <mailto:lucene@mikemccandless.com
> >>
> >>>>> wrote:
> >>>>> >>
> >>>>> >>> Bummer.
> >>>>> >>>
> >>>>> >>> So the only viable workarounds are 1) don't use
> >>>>> Thread.interrupt (nor,
> >>>>> >>> things like Future.cancel, which in turn use
> Thread.interrupt)
> >>>>> with
> >>>>> >>> NIOFSDir, or 2) we fix NIOFSDir to reopen the channel AND the
> >>>>> app must
> >>>>> >>> make a deletion policy that keeps a commit alive if any
> reader
> >>>>> is
> >>>>> >>> using it. Or, 3) don't use NIOFSDir!
> >>>>> >>>
> >>>>> >>> Mike
> >>>>> >>>
> >>>>> >>> On Thu, Jan 28, 2010 at 7:29 AM, Simon Willnauer
> >>>>> >>> <simon.willnauer@googlemail.com
> >>>>> <ma...@googlemail.com>> wrote:
> >>>>> >>>
> >>>>> >>>> On Thu, Jan 28, 2010 at 12:43 PM, Michael McCandless
> >>>>> >>>> <lucene@mikemccandless.com <mailto:
> lucene@mikemccandless.com>>
> >>>>> wrote:
> >>>>> >>>>
> >>>>> >>>>> On Thu, Jan 28, 2010 at 6:38 AM, Uwe Schindler
> >>>>> <uwe@thetaphi.de <ma...@thetaphi.de>> wrote:
> >>>>> >>>>>
> >>>>> >>>>>
> >>>>> >>>>>> So I checked the code of NIOFSIndexInput, my last comment
> >>>>> was not really correct:
> >>>>> >>>>>> NIOFSIndexInput extends SimpleFSIndexInput and that opens
> >>>>> the RAF. In the ctor RAF.getChannel() is called. The RAF keeps
> open
> >>>>> until the file is closed (and also the channel).
> >>>>> >>>>>>
> >>>>> >>>>>> So it's really simple to fix in my opinion, just call
> >>>>> getChannel() again on this exception. Because the RAF should still
> >>>>> be open?
> >>>>> >>>>>>
> >>>>> >>>> Short answer:
> >>>>> >>>> public final FileChannel getChannel() {
> >>>>> >>>> synchronized (this) {
> >>>>> >>>> if (channel == null)
> >>>>> >>>> channel = FileChannelImpl.open(fd, true, rw,
> >>>>> this);
> >>>>> >>>> return channel;
> >>>>> >>>> }
> >>>>> >>>> }
> >>>>> >>>>
> >>>>> >>>> this is not gonna work I tried it before. The
> RandomAccessFile
> >>>>> buffers
> >>>>> >>>> the channel!!
> >>>>> >>>>
> >>>>> >>>> simon
> >>>>> >>>>
> >>>>> >>>>> I think we need a definitive answer on what happens to the
> >>>>> RAF when
> >>>>> >>>>> the FileChannel was closed by Thread.Interrupt. Simon can
> >>>>> you test
> >>>>> >>>>> this?
> >>>>> >>>>>
> >>>>> >>>>> Mike
> >>>>> >>>>>
> >>>>> >>>>>
> >>>>>
> >>>>>
> ---------------------------------------------------------------------
> >>>>> >>>>> To unsubscribe, e-mail:
> >>>>> java-dev-unsubscribe@lucene.apache.org
> >>>>> <ma...@lucene.apache.org>
> >>>>> >>>>> For additional commands, e-mail:
> >>>>> java-dev-help@lucene.apache.org
> >>>>> <ma...@lucene.apache.org>
> >>>>> >>>>>
> >>>>> >>>>>
> >>>>> >>>>>
> >>>>> >>>>
> >>>>>
> >>>>>
> ---------------------------------------------------------------------
> >>>>> >>>> To unsubscribe, e-mail:
> java-dev-unsubscribe@lucene.apache.org
> >>>>> <ma...@lucene.apache.org>
> >>>>> >>>> For additional commands, e-mail:
> >>>>> java-dev-help@lucene.apache.org
> >>>>> <ma...@lucene.apache.org>
> >>>>> >>>>
> >>>>> >>>>
> >>>>> >>>>
> >>>>> >>>
> >>>>>
> >>>>>
> ---------------------------------------------------------------------
> >>>>> >>> To unsubscribe, e-mail:
> java-dev-unsubscribe@lucene.apache.org
> >>>>> <ma...@lucene.apache.org>
> >>>>> >>> For additional commands, e-mail:
> >>>>> java-dev-help@lucene.apache.org
> >>>>> <ma...@lucene.apache.org>
> >>>>> >>>
> >>>>> >>>
> >>>>> >>>
> >>>>> >>
> >>>>>
> >>>>>
> ---------------------------------------------------------------------
> >>>>> >> To unsubscribe, e-mail:
> java-dev-unsubscribe@lucene.apache.org
> >>>>> <ma...@lucene.apache.org>
> >>>>> >> For additional commands, e-mail:
> java-dev-help@lucene.apache.org
> >>>>> <ma...@lucene.apache.org>
> >>>>> >>
> >>>>> >>
> >>>>> >>
> >>>>> >
> >>>>> >
> >>>>> ---------------------------------------------------------------------
> >>>>> > To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> >>>>> <ma...@lucene.apache.org>
> >>>>> > For additional commands, e-mail:
> java-dev-help@lucene.apache.org
> >>>>> <ma...@lucene.apache.org>
> >>>>> >
> >>>>> >
> >>>>>
> >>>>>
> >>>>> --
> >>>>> - Mark
> >>>>>
> >>>>> http://www.lucidimagination.com
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> ---------------------------------------------------------------------
> >>>>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> >>>>> <ma...@lucene.apache.org>
> >>>>> For additional commands, e-mail: java-dev-help@lucene.apache.org
> >>>>> <ma...@lucene.apache.org>
> >>>>>
> >>>>>
> >>>>
> >>>>
> >>>> --
> >>>> - Mark
> >>>>
> >>>> http://www.lucidimagination.com
> >>>>
> >>>> ---------------------------------------------------------------------
> >>>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> >>>> For additional commands, e-mail: dev-help@lucene.apache.org
> >>>>
> >>>
> >>
> >>
> >>
> >> --
> >> - Mark
> >>
> >> http://www.lucidimagination.com
> >>
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
>
Re: Nasty NIO behavior makes NIOFSDirectory silently close channel
Posted by Michael McCandless <lu...@mikemccandless.com>.
This is strange -- it's completely opposite of what others have seen!
Other have seen sizable concurrency gains when switching from
SimpleFSDir -> NIOFSDir, except on Windows where Sun's long standing
JVM bug (http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6265734)
prevents concurrency gains.
There must be something different about the testing.
John are you testing pure searching? Are you reopening readers as
well? How much HW concurrency do your machines have?
Mike
On Sun, May 2, 2010 at 11:53 PM, John Wang <jo...@gmail.com> wrote:
> Solaris and MacOs
>
> On Sun, May 2, 2010 at 4:47 PM, Mark Miller <ma...@gmail.com> wrote:
>>
>> Interesting ... whats the OS?
>>
>> On Sun, May 2, 2010 at 7:12 PM, John Wang <jo...@gmail.com> wrote:
>>>
>>> 12 concurrent search threads in a stress environment. (with about 5M doc
>>> index)
>>> -John
>>>
>>> On Sun, May 2, 2010 at 3:13 PM, Mark Miller <ma...@gmail.com>
>>> wrote:
>>>>
>>>> Perfectly safe to use SimpleFSDirectory. When you measure the
>>>> performance, are you measuring a lot of concurrent requests? That's where
>>>> you should see the win.
>>>>
>>>> - Mark
>>>>
>>>> On 5/1/10 6:38 PM, John Wang wrote:
>>>>>
>>>>> We are seeing this issue as well in your production. (using Zoie on top
>>>>> of lucene 2.9.1)
>>>>> After some performance comparisons, we do NOT see performance gain with
>>>>> NIO, rather these nasty ClosedChannelExceptions.
>>>>>
>>>>> I think the performance gains ppl are seeing with 2.9.1 can be due to
>>>>> many different things. From what we seen, they are not related to
>>>>> NIOFSDirectory.
>>>>>
>>>>> Our solution is to avoid calling FSDirectory.open(), instead just call
>>>>> new SimpleFSDirectory(). Is this safe?
>>>>>
>>>>> -John
>>>>>
>>>>> On Fri, Jan 29, 2010 at 12:32 PM, Mark Miller <markrmiller@gmail.com
>>>>> <ma...@gmail.com>> wrote:
>>>>>
>>>>> Perhaps - one of the things they are supposed to be addressing is
>>>>> extendability.
>>>>>
>>>>> nio2 does have FileSystemProvider, which would actually allow you to
>>>>> create a custom channel !
>>>>>
>>>>> I have not dug in enough to know much more than that though.
>>>>>
>>>>> *But*, another really interesting thing is that in Java 7,
>>>>> FileDescriptors are ref counted ! (though users can't inc/dec).
>>>>>
>>>>> But, FileInputStream and OutputStream have a new constructor that
>>>>> takes
>>>>> a FileDescriptor.
>>>>>
>>>>> So possibly, you could just make one that sits around to keep the
>>>>> FileDescriptor valid, and get your channel off
>>>>> FileInputStream/FileOutputStream?
>>>>>
>>>>> And then if it goes down, make a new one using the FileDescriptor
>>>>> which
>>>>> was not actually closed because there was a still a ref to it.
>>>>>
>>>>> Possibly .... ;)
>>>>>
>>>>> Michael McCandless wrote:
>>>>> > Does anyone know if nio2 has improved this...?
>>>>> >
>>>>> > Mike
>>>>> >
>>>>> > On Fri, Jan 29, 2010 at 2:00 PM, Jason Rutherglen
>>>>> > <jason.rutherglen@gmail.com <ma...@gmail.com>>
>>>>> wrote:
>>>>> >
>>>>> >> Defaulting NIOFSDir could account for some of the recent speed
>>>>> >> improvements users have been reporting in Lucene 2.9. So
>>>>> removing it
>>>>> >> as a default could reverse those and people could then report
>>>>> Lucene
>>>>> >> 3.X has slowed...
>>>>> >>
>>>>> >> On Thu, Jan 28, 2010 at 5:24 AM, Michael McCandless
>>>>> >> <lucene@mikemccandless.com <ma...@mikemccandless.com>>
>>>>> wrote:
>>>>> >>
>>>>> >>> Bummer.
>>>>> >>>
>>>>> >>> So the only viable workarounds are 1) don't use
>>>>> Thread.interrupt (nor,
>>>>> >>> things like Future.cancel, which in turn use Thread.interrupt)
>>>>> with
>>>>> >>> NIOFSDir, or 2) we fix NIOFSDir to reopen the channel AND the
>>>>> app must
>>>>> >>> make a deletion policy that keeps a commit alive if any reader
>>>>> is
>>>>> >>> using it. Or, 3) don't use NIOFSDir!
>>>>> >>>
>>>>> >>> Mike
>>>>> >>>
>>>>> >>> On Thu, Jan 28, 2010 at 7:29 AM, Simon Willnauer
>>>>> >>> <simon.willnauer@googlemail.com
>>>>> <ma...@googlemail.com>> wrote:
>>>>> >>>
>>>>> >>>> On Thu, Jan 28, 2010 at 12:43 PM, Michael McCandless
>>>>> >>>> <lucene@mikemccandless.com <ma...@mikemccandless.com>>
>>>>> wrote:
>>>>> >>>>
>>>>> >>>>> On Thu, Jan 28, 2010 at 6:38 AM, Uwe Schindler
>>>>> <uwe@thetaphi.de <ma...@thetaphi.de>> wrote:
>>>>> >>>>>
>>>>> >>>>>
>>>>> >>>>>> So I checked the code of NIOFSIndexInput, my last comment
>>>>> was not really correct:
>>>>> >>>>>> NIOFSIndexInput extends SimpleFSIndexInput and that opens
>>>>> the RAF. In the ctor RAF.getChannel() is called. The RAF keeps open
>>>>> until the file is closed (and also the channel).
>>>>> >>>>>>
>>>>> >>>>>> So it's really simple to fix in my opinion, just call
>>>>> getChannel() again on this exception. Because the RAF should still
>>>>> be open?
>>>>> >>>>>>
>>>>> >>>> Short answer:
>>>>> >>>> public final FileChannel getChannel() {
>>>>> >>>> synchronized (this) {
>>>>> >>>> if (channel == null)
>>>>> >>>> channel = FileChannelImpl.open(fd, true, rw,
>>>>> this);
>>>>> >>>> return channel;
>>>>> >>>> }
>>>>> >>>> }
>>>>> >>>>
>>>>> >>>> this is not gonna work I tried it before. The RandomAccessFile
>>>>> buffers
>>>>> >>>> the channel!!
>>>>> >>>>
>>>>> >>>> simon
>>>>> >>>>
>>>>> >>>>> I think we need a definitive answer on what happens to the
>>>>> RAF when
>>>>> >>>>> the FileChannel was closed by Thread.Interrupt. Simon can
>>>>> you test
>>>>> >>>>> this?
>>>>> >>>>>
>>>>> >>>>> Mike
>>>>> >>>>>
>>>>> >>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> >>>>> To unsubscribe, e-mail:
>>>>> java-dev-unsubscribe@lucene.apache.org
>>>>> <ma...@lucene.apache.org>
>>>>> >>>>> For additional commands, e-mail:
>>>>> java-dev-help@lucene.apache.org
>>>>> <ma...@lucene.apache.org>
>>>>> >>>>>
>>>>> >>>>>
>>>>> >>>>>
>>>>> >>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> >>>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>>>>> <ma...@lucene.apache.org>
>>>>> >>>> For additional commands, e-mail:
>>>>> java-dev-help@lucene.apache.org
>>>>> <ma...@lucene.apache.org>
>>>>> >>>>
>>>>> >>>>
>>>>> >>>>
>>>>> >>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> >>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>>>>> <ma...@lucene.apache.org>
>>>>> >>> For additional commands, e-mail:
>>>>> java-dev-help@lucene.apache.org
>>>>> <ma...@lucene.apache.org>
>>>>> >>>
>>>>> >>>
>>>>> >>>
>>>>> >>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> >> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>>>>> <ma...@lucene.apache.org>
>>>>> >> For additional commands, e-mail: java-dev-help@lucene.apache.org
>>>>> <ma...@lucene.apache.org>
>>>>> >>
>>>>> >>
>>>>> >>
>>>>> >
>>>>> >
>>>>> ---------------------------------------------------------------------
>>>>> > To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>>>>> <ma...@lucene.apache.org>
>>>>> > For additional commands, e-mail: java-dev-help@lucene.apache.org
>>>>> <ma...@lucene.apache.org>
>>>>> >
>>>>> >
>>>>>
>>>>>
>>>>> --
>>>>> - Mark
>>>>>
>>>>> http://www.lucidimagination.com
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>>>>> <ma...@lucene.apache.org>
>>>>> For additional commands, e-mail: java-dev-help@lucene.apache.org
>>>>> <ma...@lucene.apache.org>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> - Mark
>>>>
>>>> http://www.lucidimagination.com
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>>>> For additional commands, e-mail: dev-help@lucene.apache.org
>>>>
>>>
>>
>>
>>
>> --
>> - Mark
>>
>> http://www.lucidimagination.com
>>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: Nasty NIO behavior makes NIOFSDirectory silently close channel
Posted by John Wang <jo...@gmail.com>.
Solaris and MacOs
On Sun, May 2, 2010 at 4:47 PM, Mark Miller <ma...@gmail.com> wrote:
> Interesting ... whats the OS?
>
>
> On Sun, May 2, 2010 at 7:12 PM, John Wang <jo...@gmail.com> wrote:
>
>> 12 concurrent search threads in a stress environment. (with about 5M doc
>> index)
>>
>> -John
>>
>> On Sun, May 2, 2010 at 3:13 PM, Mark Miller <ma...@gmail.com>wrote:
>>
>>> Perfectly safe to use SimpleFSDirectory. When you measure the
>>> performance, are you measuring a lot of concurrent requests? That's where
>>> you should see the win.
>>>
>>> - Mark
>>>
>>>
>>> On 5/1/10 6:38 PM, John Wang wrote:
>>>
>>>> We are seeing this issue as well in your production. (using Zoie on top
>>>> of lucene 2.9.1)
>>>> After some performance comparisons, we do NOT see performance gain with
>>>> NIO, rather these nasty ClosedChannelExceptions.
>>>>
>>>> I think the performance gains ppl are seeing with 2.9.1 can be due to
>>>> many different things. From what we seen, they are not related to
>>>> NIOFSDirectory.
>>>>
>>>> Our solution is to avoid calling FSDirectory.open(), instead just call
>>>> new SimpleFSDirectory(). Is this safe?
>>>>
>>>> -John
>>>>
>>>> On Fri, Jan 29, 2010 at 12:32 PM, Mark Miller <markrmiller@gmail.com
>>>> <ma...@gmail.com>> wrote:
>>>>
>>>> Perhaps - one of the things they are supposed to be addressing is
>>>> extendability.
>>>>
>>>> nio2 does have FileSystemProvider, which would actually allow you to
>>>> create a custom channel !
>>>>
>>>> I have not dug in enough to know much more than that though.
>>>>
>>>> *But*, another really interesting thing is that in Java 7,
>>>> FileDescriptors are ref counted ! (though users can't inc/dec).
>>>>
>>>> But, FileInputStream and OutputStream have a new constructor that
>>>> takes
>>>> a FileDescriptor.
>>>>
>>>> So possibly, you could just make one that sits around to keep the
>>>> FileDescriptor valid, and get your channel off
>>>> FileInputStream/FileOutputStream?
>>>>
>>>> And then if it goes down, make a new one using the FileDescriptor
>>>> which
>>>> was not actually closed because there was a still a ref to it.
>>>>
>>>> Possibly .... ;)
>>>>
>>>> Michael McCandless wrote:
>>>> > Does anyone know if nio2 has improved this...?
>>>> >
>>>> > Mike
>>>> >
>>>> > On Fri, Jan 29, 2010 at 2:00 PM, Jason Rutherglen
>>>> > <jason.rutherglen@gmail.com <ma...@gmail.com>>
>>>>
>>>> wrote:
>>>> >
>>>> >> Defaulting NIOFSDir could account for some of the recent speed
>>>> >> improvements users have been reporting in Lucene 2.9. So
>>>> removing it
>>>> >> as a default could reverse those and people could then report
>>>> Lucene
>>>> >> 3.X has slowed...
>>>> >>
>>>> >> On Thu, Jan 28, 2010 at 5:24 AM, Michael McCandless
>>>> >> <lucene@mikemccandless.com <ma...@mikemccandless.com>>
>>>>
>>>> wrote:
>>>> >>
>>>> >>> Bummer.
>>>> >>>
>>>> >>> So the only viable workarounds are 1) don't use
>>>> Thread.interrupt (nor,
>>>> >>> things like Future.cancel, which in turn use Thread.interrupt)
>>>> with
>>>> >>> NIOFSDir, or 2) we fix NIOFSDir to reopen the channel AND the
>>>> app must
>>>> >>> make a deletion policy that keeps a commit alive if any reader
>>>> is
>>>> >>> using it. Or, 3) don't use NIOFSDir!
>>>> >>>
>>>> >>> Mike
>>>> >>>
>>>> >>> On Thu, Jan 28, 2010 at 7:29 AM, Simon Willnauer
>>>> >>> <simon.willnauer@googlemail.com
>>>> <ma...@googlemail.com>> wrote:
>>>> >>>
>>>> >>>> On Thu, Jan 28, 2010 at 12:43 PM, Michael McCandless
>>>> >>>> <lucene@mikemccandless.com <ma...@mikemccandless.com>>
>>>>
>>>> wrote:
>>>> >>>>
>>>> >>>>> On Thu, Jan 28, 2010 at 6:38 AM, Uwe Schindler
>>>> <uwe@thetaphi.de <ma...@thetaphi.de>> wrote:
>>>> >>>>>
>>>> >>>>>
>>>> >>>>>> So I checked the code of NIOFSIndexInput, my last comment
>>>> was not really correct:
>>>> >>>>>> NIOFSIndexInput extends SimpleFSIndexInput and that opens
>>>> the RAF. In the ctor RAF.getChannel() is called. The RAF keeps open
>>>> until the file is closed (and also the channel).
>>>> >>>>>>
>>>> >>>>>> So it's really simple to fix in my opinion, just call
>>>> getChannel() again on this exception. Because the RAF should still
>>>> be open?
>>>> >>>>>>
>>>> >>>> Short answer:
>>>> >>>> public final FileChannel getChannel() {
>>>> >>>> synchronized (this) {
>>>> >>>> if (channel == null)
>>>> >>>> channel = FileChannelImpl.open(fd, true, rw,
>>>> this);
>>>> >>>> return channel;
>>>> >>>> }
>>>> >>>> }
>>>> >>>>
>>>> >>>> this is not gonna work I tried it before. The RandomAccessFile
>>>> buffers
>>>> >>>> the channel!!
>>>> >>>>
>>>> >>>> simon
>>>> >>>>
>>>> >>>>> I think we need a definitive answer on what happens to the
>>>> RAF when
>>>> >>>>> the FileChannel was closed by Thread.Interrupt. Simon can
>>>> you test
>>>> >>>>> this?
>>>> >>>>>
>>>> >>>>> Mike
>>>> >>>>>
>>>> >>>>>
>>>> ---------------------------------------------------------------------
>>>> >>>>> To unsubscribe, e-mail:
>>>> java-dev-unsubscribe@lucene.apache.org
>>>> <ma...@lucene.apache.org>
>>>>
>>>> >>>>> For additional commands, e-mail:
>>>> java-dev-help@lucene.apache.org <mailto:
>>>> java-dev-help@lucene.apache.org>
>>>>
>>>> >>>>>
>>>> >>>>>
>>>> >>>>>
>>>> >>>>
>>>> ---------------------------------------------------------------------
>>>> >>>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>>>> <ma...@lucene.apache.org>
>>>>
>>>> >>>> For additional commands, e-mail:
>>>> java-dev-help@lucene.apache.org <mailto:
>>>> java-dev-help@lucene.apache.org>
>>>>
>>>> >>>>
>>>> >>>>
>>>> >>>>
>>>> >>>
>>>> ---------------------------------------------------------------------
>>>> >>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>>>> <ma...@lucene.apache.org>
>>>>
>>>> >>> For additional commands, e-mail:
>>>> java-dev-help@lucene.apache.org <mailto:
>>>> java-dev-help@lucene.apache.org>
>>>>
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>
>>>> ---------------------------------------------------------------------
>>>> >> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>>>> <ma...@lucene.apache.org>
>>>>
>>>> >> For additional commands, e-mail: java-dev-help@lucene.apache.org
>>>> <ma...@lucene.apache.org>
>>>>
>>>> >>
>>>> >>
>>>> >>
>>>> >
>>>> >
>>>> ---------------------------------------------------------------------
>>>> > To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>>>> <ma...@lucene.apache.org>
>>>>
>>>> > For additional commands, e-mail: java-dev-help@lucene.apache.org
>>>> <ma...@lucene.apache.org>
>>>>
>>>> >
>>>> >
>>>>
>>>>
>>>> --
>>>> - Mark
>>>>
>>>> http://www.lucidimagination.com
>>>>
>>>>
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>>>> <ma...@lucene.apache.org>
>>>>
>>>> For additional commands, e-mail: java-dev-help@lucene.apache.org
>>>> <ma...@lucene.apache.org>
>>>>
>>>>
>>>>
>>>
>>> --
>>> - Mark
>>>
>>> http://www.lucidimagination.com
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: dev-help@lucene.apache.org
>>>
>>>
>>
>
>
> --
> - Mark
>
> http://www.lucidimagination.com
>
>
Re: Nasty NIO behavior makes NIOFSDirectory silently close channel
Posted by Mark Miller <ma...@gmail.com>.
Interesting ... whats the OS?
On Sun, May 2, 2010 at 7:12 PM, John Wang <jo...@gmail.com> wrote:
> 12 concurrent search threads in a stress environment. (with about 5M doc
> index)
>
> -John
>
> On Sun, May 2, 2010 at 3:13 PM, Mark Miller <ma...@gmail.com> wrote:
>
>> Perfectly safe to use SimpleFSDirectory. When you measure the performance,
>> are you measuring a lot of concurrent requests? That's where you should see
>> the win.
>>
>> - Mark
>>
>>
>> On 5/1/10 6:38 PM, John Wang wrote:
>>
>>> We are seeing this issue as well in your production. (using Zoie on top
>>> of lucene 2.9.1)
>>> After some performance comparisons, we do NOT see performance gain with
>>> NIO, rather these nasty ClosedChannelExceptions.
>>>
>>> I think the performance gains ppl are seeing with 2.9.1 can be due to
>>> many different things. From what we seen, they are not related to
>>> NIOFSDirectory.
>>>
>>> Our solution is to avoid calling FSDirectory.open(), instead just call
>>> new SimpleFSDirectory(). Is this safe?
>>>
>>> -John
>>>
>>> On Fri, Jan 29, 2010 at 12:32 PM, Mark Miller <markrmiller@gmail.com
>>> <ma...@gmail.com>> wrote:
>>>
>>> Perhaps - one of the things they are supposed to be addressing is
>>> extendability.
>>>
>>> nio2 does have FileSystemProvider, which would actually allow you to
>>> create a custom channel !
>>>
>>> I have not dug in enough to know much more than that though.
>>>
>>> *But*, another really interesting thing is that in Java 7,
>>> FileDescriptors are ref counted ! (though users can't inc/dec).
>>>
>>> But, FileInputStream and OutputStream have a new constructor that
>>> takes
>>> a FileDescriptor.
>>>
>>> So possibly, you could just make one that sits around to keep the
>>> FileDescriptor valid, and get your channel off
>>> FileInputStream/FileOutputStream?
>>>
>>> And then if it goes down, make a new one using the FileDescriptor
>>> which
>>> was not actually closed because there was a still a ref to it.
>>>
>>> Possibly .... ;)
>>>
>>> Michael McCandless wrote:
>>> > Does anyone know if nio2 has improved this...?
>>> >
>>> > Mike
>>> >
>>> > On Fri, Jan 29, 2010 at 2:00 PM, Jason Rutherglen
>>> > <jason.rutherglen@gmail.com <ma...@gmail.com>>
>>>
>>> wrote:
>>> >
>>> >> Defaulting NIOFSDir could account for some of the recent speed
>>> >> improvements users have been reporting in Lucene 2.9. So
>>> removing it
>>> >> as a default could reverse those and people could then report
>>> Lucene
>>> >> 3.X has slowed...
>>> >>
>>> >> On Thu, Jan 28, 2010 at 5:24 AM, Michael McCandless
>>> >> <lucene@mikemccandless.com <ma...@mikemccandless.com>>
>>>
>>> wrote:
>>> >>
>>> >>> Bummer.
>>> >>>
>>> >>> So the only viable workarounds are 1) don't use
>>> Thread.interrupt (nor,
>>> >>> things like Future.cancel, which in turn use Thread.interrupt)
>>> with
>>> >>> NIOFSDir, or 2) we fix NIOFSDir to reopen the channel AND the
>>> app must
>>> >>> make a deletion policy that keeps a commit alive if any reader is
>>> >>> using it. Or, 3) don't use NIOFSDir!
>>> >>>
>>> >>> Mike
>>> >>>
>>> >>> On Thu, Jan 28, 2010 at 7:29 AM, Simon Willnauer
>>> >>> <simon.willnauer@googlemail.com
>>> <ma...@googlemail.com>> wrote:
>>> >>>
>>> >>>> On Thu, Jan 28, 2010 at 12:43 PM, Michael McCandless
>>> >>>> <lucene@mikemccandless.com <ma...@mikemccandless.com>>
>>>
>>> wrote:
>>> >>>>
>>> >>>>> On Thu, Jan 28, 2010 at 6:38 AM, Uwe Schindler
>>> <uwe@thetaphi.de <ma...@thetaphi.de>> wrote:
>>> >>>>>
>>> >>>>>
>>> >>>>>> So I checked the code of NIOFSIndexInput, my last comment
>>> was not really correct:
>>> >>>>>> NIOFSIndexInput extends SimpleFSIndexInput and that opens
>>> the RAF. In the ctor RAF.getChannel() is called. The RAF keeps open
>>> until the file is closed (and also the channel).
>>> >>>>>>
>>> >>>>>> So it's really simple to fix in my opinion, just call
>>> getChannel() again on this exception. Because the RAF should still
>>> be open?
>>> >>>>>>
>>> >>>> Short answer:
>>> >>>> public final FileChannel getChannel() {
>>> >>>> synchronized (this) {
>>> >>>> if (channel == null)
>>> >>>> channel = FileChannelImpl.open(fd, true, rw,
>>> this);
>>> >>>> return channel;
>>> >>>> }
>>> >>>> }
>>> >>>>
>>> >>>> this is not gonna work I tried it before. The RandomAccessFile
>>> buffers
>>> >>>> the channel!!
>>> >>>>
>>> >>>> simon
>>> >>>>
>>> >>>>> I think we need a definitive answer on what happens to the
>>> RAF when
>>> >>>>> the FileChannel was closed by Thread.Interrupt. Simon can
>>> you test
>>> >>>>> this?
>>> >>>>>
>>> >>>>> Mike
>>> >>>>>
>>> >>>>>
>>> ---------------------------------------------------------------------
>>> >>>>> To unsubscribe, e-mail:
>>> java-dev-unsubscribe@lucene.apache.org
>>> <ma...@lucene.apache.org>
>>>
>>> >>>>> For additional commands, e-mail:
>>> java-dev-help@lucene.apache.org <mailto:
>>> java-dev-help@lucene.apache.org>
>>>
>>> >>>>>
>>> >>>>>
>>> >>>>>
>>> >>>>
>>> ---------------------------------------------------------------------
>>> >>>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>>> <ma...@lucene.apache.org>
>>>
>>> >>>> For additional commands, e-mail:
>>> java-dev-help@lucene.apache.org <mailto:
>>> java-dev-help@lucene.apache.org>
>>>
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>
>>> ---------------------------------------------------------------------
>>> >>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>>> <ma...@lucene.apache.org>
>>>
>>> >>> For additional commands, e-mail:
>>> java-dev-help@lucene.apache.org <mailto:
>>> java-dev-help@lucene.apache.org>
>>>
>>> >>>
>>> >>>
>>> >>>
>>> >>
>>> ---------------------------------------------------------------------
>>> >> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>>> <ma...@lucene.apache.org>
>>>
>>> >> For additional commands, e-mail: java-dev-help@lucene.apache.org
>>> <ma...@lucene.apache.org>
>>>
>>> >>
>>> >>
>>> >>
>>> >
>>> >
>>> ---------------------------------------------------------------------
>>> > To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>>> <ma...@lucene.apache.org>
>>>
>>> > For additional commands, e-mail: java-dev-help@lucene.apache.org
>>> <ma...@lucene.apache.org>
>>>
>>> >
>>> >
>>>
>>>
>>> --
>>> - Mark
>>>
>>> http://www.lucidimagination.com
>>>
>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>>> <ma...@lucene.apache.org>
>>>
>>> For additional commands, e-mail: java-dev-help@lucene.apache.org
>>> <ma...@lucene.apache.org>
>>>
>>>
>>>
>>
>> --
>> - Mark
>>
>> http://www.lucidimagination.com
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: dev-help@lucene.apache.org
>>
>>
>
--
- Mark
http://www.lucidimagination.com
Re: Nasty NIO behavior makes NIOFSDirectory silently close channel
Posted by John Wang <jo...@gmail.com>.
12 concurrent search threads in a stress environment. (with about 5M doc
index)
-John
On Sun, May 2, 2010 at 3:13 PM, Mark Miller <ma...@gmail.com> wrote:
> Perfectly safe to use SimpleFSDirectory. When you measure the performance,
> are you measuring a lot of concurrent requests? That's where you should see
> the win.
>
> - Mark
>
>
> On 5/1/10 6:38 PM, John Wang wrote:
>
>> We are seeing this issue as well in your production. (using Zoie on top
>> of lucene 2.9.1)
>> After some performance comparisons, we do NOT see performance gain with
>> NIO, rather these nasty ClosedChannelExceptions.
>>
>> I think the performance gains ppl are seeing with 2.9.1 can be due to
>> many different things. From what we seen, they are not related to
>> NIOFSDirectory.
>>
>> Our solution is to avoid calling FSDirectory.open(), instead just call
>> new SimpleFSDirectory(). Is this safe?
>>
>> -John
>>
>> On Fri, Jan 29, 2010 at 12:32 PM, Mark Miller <markrmiller@gmail.com
>> <ma...@gmail.com>> wrote:
>>
>> Perhaps - one of the things they are supposed to be addressing is
>> extendability.
>>
>> nio2 does have FileSystemProvider, which would actually allow you to
>> create a custom channel !
>>
>> I have not dug in enough to know much more than that though.
>>
>> *But*, another really interesting thing is that in Java 7,
>> FileDescriptors are ref counted ! (though users can't inc/dec).
>>
>> But, FileInputStream and OutputStream have a new constructor that takes
>> a FileDescriptor.
>>
>> So possibly, you could just make one that sits around to keep the
>> FileDescriptor valid, and get your channel off
>> FileInputStream/FileOutputStream?
>>
>> And then if it goes down, make a new one using the FileDescriptor which
>> was not actually closed because there was a still a ref to it.
>>
>> Possibly .... ;)
>>
>> Michael McCandless wrote:
>> > Does anyone know if nio2 has improved this...?
>> >
>> > Mike
>> >
>> > On Fri, Jan 29, 2010 at 2:00 PM, Jason Rutherglen
>> > <jason.rutherglen@gmail.com <ma...@gmail.com>>
>>
>> wrote:
>> >
>> >> Defaulting NIOFSDir could account for some of the recent speed
>> >> improvements users have been reporting in Lucene 2.9. So
>> removing it
>> >> as a default could reverse those and people could then report
>> Lucene
>> >> 3.X has slowed...
>> >>
>> >> On Thu, Jan 28, 2010 at 5:24 AM, Michael McCandless
>> >> <lucene@mikemccandless.com <ma...@mikemccandless.com>>
>>
>> wrote:
>> >>
>> >>> Bummer.
>> >>>
>> >>> So the only viable workarounds are 1) don't use
>> Thread.interrupt (nor,
>> >>> things like Future.cancel, which in turn use Thread.interrupt)
>> with
>> >>> NIOFSDir, or 2) we fix NIOFSDir to reopen the channel AND the
>> app must
>> >>> make a deletion policy that keeps a commit alive if any reader is
>> >>> using it. Or, 3) don't use NIOFSDir!
>> >>>
>> >>> Mike
>> >>>
>> >>> On Thu, Jan 28, 2010 at 7:29 AM, Simon Willnauer
>> >>> <simon.willnauer@googlemail.com
>> <ma...@googlemail.com>> wrote:
>> >>>
>> >>>> On Thu, Jan 28, 2010 at 12:43 PM, Michael McCandless
>> >>>> <lucene@mikemccandless.com <ma...@mikemccandless.com>>
>>
>> wrote:
>> >>>>
>> >>>>> On Thu, Jan 28, 2010 at 6:38 AM, Uwe Schindler
>> <uwe@thetaphi.de <ma...@thetaphi.de>> wrote:
>> >>>>>
>> >>>>>
>> >>>>>> So I checked the code of NIOFSIndexInput, my last comment
>> was not really correct:
>> >>>>>> NIOFSIndexInput extends SimpleFSIndexInput and that opens
>> the RAF. In the ctor RAF.getChannel() is called. The RAF keeps open
>> until the file is closed (and also the channel).
>> >>>>>>
>> >>>>>> So it's really simple to fix in my opinion, just call
>> getChannel() again on this exception. Because the RAF should still
>> be open?
>> >>>>>>
>> >>>> Short answer:
>> >>>> public final FileChannel getChannel() {
>> >>>> synchronized (this) {
>> >>>> if (channel == null)
>> >>>> channel = FileChannelImpl.open(fd, true, rw,
>> this);
>> >>>> return channel;
>> >>>> }
>> >>>> }
>> >>>>
>> >>>> this is not gonna work I tried it before. The RandomAccessFile
>> buffers
>> >>>> the channel!!
>> >>>>
>> >>>> simon
>> >>>>
>> >>>>> I think we need a definitive answer on what happens to the
>> RAF when
>> >>>>> the FileChannel was closed by Thread.Interrupt. Simon can
>> you test
>> >>>>> this?
>> >>>>>
>> >>>>> Mike
>> >>>>>
>> >>>>>
>> ---------------------------------------------------------------------
>> >>>>> To unsubscribe, e-mail:
>> java-dev-unsubscribe@lucene.apache.org
>> <ma...@lucene.apache.org>
>>
>> >>>>> For additional commands, e-mail:
>> java-dev-help@lucene.apache.org <mailto:
>> java-dev-help@lucene.apache.org>
>>
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>
>> ---------------------------------------------------------------------
>> >>>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>> <ma...@lucene.apache.org>
>>
>> >>>> For additional commands, e-mail:
>> java-dev-help@lucene.apache.org <mailto:
>> java-dev-help@lucene.apache.org>
>>
>> >>>>
>> >>>>
>> >>>>
>> >>>
>> ---------------------------------------------------------------------
>> >>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>> <ma...@lucene.apache.org>
>>
>> >>> For additional commands, e-mail:
>> java-dev-help@lucene.apache.org <mailto:
>> java-dev-help@lucene.apache.org>
>>
>> >>>
>> >>>
>> >>>
>> >>
>> ---------------------------------------------------------------------
>> >> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>> <ma...@lucene.apache.org>
>>
>> >> For additional commands, e-mail: java-dev-help@lucene.apache.org
>> <ma...@lucene.apache.org>
>>
>> >>
>> >>
>> >>
>> >
>> >
>> ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>> <ma...@lucene.apache.org>
>>
>> > For additional commands, e-mail: java-dev-help@lucene.apache.org
>> <ma...@lucene.apache.org>
>>
>> >
>> >
>>
>>
>> --
>> - Mark
>>
>> http://www.lucidimagination.com
>>
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>> <ma...@lucene.apache.org>
>>
>> For additional commands, e-mail: java-dev-help@lucene.apache.org
>> <ma...@lucene.apache.org>
>>
>>
>>
>
> --
> - Mark
>
> http://www.lucidimagination.com
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
>
Re: Nasty NIO behavior makes NIOFSDirectory silently close channel
Posted by Mark Miller <ma...@gmail.com>.
Perfectly safe to use SimpleFSDirectory. When you measure the
performance, are you measuring a lot of concurrent requests? That's
where you should see the win.
- Mark
On 5/1/10 6:38 PM, John Wang wrote:
> We are seeing this issue as well in your production. (using Zoie on top
> of lucene 2.9.1)
> After some performance comparisons, we do NOT see performance gain with
> NIO, rather these nasty ClosedChannelExceptions.
>
> I think the performance gains ppl are seeing with 2.9.1 can be due to
> many different things. From what we seen, they are not related to
> NIOFSDirectory.
>
> Our solution is to avoid calling FSDirectory.open(), instead just call
> new SimpleFSDirectory(). Is this safe?
>
> -John
>
> On Fri, Jan 29, 2010 at 12:32 PM, Mark Miller <markrmiller@gmail.com
> <ma...@gmail.com>> wrote:
>
> Perhaps - one of the things they are supposed to be addressing is
> extendability.
>
> nio2 does have FileSystemProvider, which would actually allow you to
> create a custom channel !
>
> I have not dug in enough to know much more than that though.
>
> *But*, another really interesting thing is that in Java 7,
> FileDescriptors are ref counted ! (though users can't inc/dec).
>
> But, FileInputStream and OutputStream have a new constructor that takes
> a FileDescriptor.
>
> So possibly, you could just make one that sits around to keep the
> FileDescriptor valid, and get your channel off
> FileInputStream/FileOutputStream?
>
> And then if it goes down, make a new one using the FileDescriptor which
> was not actually closed because there was a still a ref to it.
>
> Possibly .... ;)
>
> Michael McCandless wrote:
> > Does anyone know if nio2 has improved this...?
> >
> > Mike
> >
> > On Fri, Jan 29, 2010 at 2:00 PM, Jason Rutherglen
> > <jason.rutherglen@gmail.com <ma...@gmail.com>>
> wrote:
> >
> >> Defaulting NIOFSDir could account for some of the recent speed
> >> improvements users have been reporting in Lucene 2.9. So
> removing it
> >> as a default could reverse those and people could then report Lucene
> >> 3.X has slowed...
> >>
> >> On Thu, Jan 28, 2010 at 5:24 AM, Michael McCandless
> >> <lucene@mikemccandless.com <ma...@mikemccandless.com>>
> wrote:
> >>
> >>> Bummer.
> >>>
> >>> So the only viable workarounds are 1) don't use
> Thread.interrupt (nor,
> >>> things like Future.cancel, which in turn use Thread.interrupt) with
> >>> NIOFSDir, or 2) we fix NIOFSDir to reopen the channel AND the
> app must
> >>> make a deletion policy that keeps a commit alive if any reader is
> >>> using it. Or, 3) don't use NIOFSDir!
> >>>
> >>> Mike
> >>>
> >>> On Thu, Jan 28, 2010 at 7:29 AM, Simon Willnauer
> >>> <simon.willnauer@googlemail.com
> <ma...@googlemail.com>> wrote:
> >>>
> >>>> On Thu, Jan 28, 2010 at 12:43 PM, Michael McCandless
> >>>> <lucene@mikemccandless.com <ma...@mikemccandless.com>>
> wrote:
> >>>>
> >>>>> On Thu, Jan 28, 2010 at 6:38 AM, Uwe Schindler
> <uwe@thetaphi.de <ma...@thetaphi.de>> wrote:
> >>>>>
> >>>>>
> >>>>>> So I checked the code of NIOFSIndexInput, my last comment
> was not really correct:
> >>>>>> NIOFSIndexInput extends SimpleFSIndexInput and that opens
> the RAF. In the ctor RAF.getChannel() is called. The RAF keeps open
> until the file is closed (and also the channel).
> >>>>>>
> >>>>>> So it's really simple to fix in my opinion, just call
> getChannel() again on this exception. Because the RAF should still
> be open?
> >>>>>>
> >>>> Short answer:
> >>>> public final FileChannel getChannel() {
> >>>> synchronized (this) {
> >>>> if (channel == null)
> >>>> channel = FileChannelImpl.open(fd, true, rw, this);
> >>>> return channel;
> >>>> }
> >>>> }
> >>>>
> >>>> this is not gonna work I tried it before. The RandomAccessFile
> buffers
> >>>> the channel!!
> >>>>
> >>>> simon
> >>>>
> >>>>> I think we need a definitive answer on what happens to the
> RAF when
> >>>>> the FileChannel was closed by Thread.Interrupt. Simon can
> you test
> >>>>> this?
> >>>>>
> >>>>> Mike
> >>>>>
> >>>>>
> ---------------------------------------------------------------------
> >>>>> To unsubscribe, e-mail:
> java-dev-unsubscribe@lucene.apache.org
> <ma...@lucene.apache.org>
> >>>>> For additional commands, e-mail:
> java-dev-help@lucene.apache.org <ma...@lucene.apache.org>
> >>>>>
> >>>>>
> >>>>>
> >>>>
> ---------------------------------------------------------------------
> >>>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> <ma...@lucene.apache.org>
> >>>> For additional commands, e-mail:
> java-dev-help@lucene.apache.org <ma...@lucene.apache.org>
> >>>>
> >>>>
> >>>>
> >>>
> ---------------------------------------------------------------------
> >>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> <ma...@lucene.apache.org>
> >>> For additional commands, e-mail:
> java-dev-help@lucene.apache.org <ma...@lucene.apache.org>
> >>>
> >>>
> >>>
> >>
> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> <ma...@lucene.apache.org>
> >> For additional commands, e-mail: java-dev-help@lucene.apache.org
> <ma...@lucene.apache.org>
> >>
> >>
> >>
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> <ma...@lucene.apache.org>
> > For additional commands, e-mail: java-dev-help@lucene.apache.org
> <ma...@lucene.apache.org>
> >
> >
>
>
> --
> - Mark
>
> http://www.lucidimagination.com
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> <ma...@lucene.apache.org>
> For additional commands, e-mail: java-dev-help@lucene.apache.org
> <ma...@lucene.apache.org>
>
>
--
- Mark
http://www.lucidimagination.com
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org