You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Simon Willnauer <si...@googlemail.com> on 2010/01/28 09:51:50 UTC
Nasty NIO behavior makes NIOFSDirectory silently close channel
Last week I ran into nasty hardly reproducible ChannelClosedExceptions
thrown by NIOFSDirectory#readInternal() during term position reading.
I had a hard time to figure out what causes the channel to be closed
as the IndexReader where I obtained the Indexinput or rather
TermPositions from was still untouched and open. Yet I was able to
perform actions on any other files except the proximity file. Even
more weirdness this only happened after upgrading to 3.0 from 2.3.
Yet, it turned out that the case of the problem is Java's FileChannel
behavior if the thread is interrupted while a read happens.
FileChannel (http://java.sun.com/j2se/1.5.0/docs/api/java/nio/channels/FileChannel.html)
throws an ClosedByInterruptException
(http://java.sun.com/j2se/1.5.0/docs/api/java/nio/channels/ClosedByInterruptException.html)
if the any thread is interrupted during an IO operation. This causes
the index input to be invalid and puts the IndexReader into an invalid
state.
Yet, I'm really surprised that this has never occured to somebody else
or at least not on the list. This behavior makes the usage of
NIOFSDirectory extremely dangerous especially in combination with
j.u.c.Future and their cancel capabilities. Since, NIO is the new
default for anything except of windows we should at least document
this behavior in BIG RED LETTERS or rethink the default. I don't see a
way to recover from this internal close without really reopen the
IndexReader again but I want to leave this to the discussion.
I was able to isolate the behavior into a testcase and will open an
issue soon. The list seemed to be a better place to initially inform
people.
simon
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Nasty NIO behavior makes NIOFSDirectory silently close channel
Posted by Michael McCandless <lu...@mikemccandless.com>.
Yeah, so not great (doubling your file descriptor usage to protect
against this). Still we could provide that as an option to
NIOFSDir...
We could at least attempt the reopen, in case the file is present.
This way, at least, one could make a (custom, app dependent) deletion
policy that ensure any commit point still being used by a reader is
not deleted. Together that would make a viable (but, app dependent)
workaround.
The other obvious workaround is to not use Thread.interrupt (or things
that in turn use it, eg Future.cancel) with NIOFSDir.
Mike
On Thu, Jan 28, 2010 at 6:28 AM, Uwe Schindler <uw...@thetaphi.de> wrote:
> From JavaDocs:
> The state of a file channel is intimately connected to that of the object whose getChannel method returned the channel. Changing the channel's position, whether explicitly or by reading or writing bytes, will change the file position of the originating object, and vice versa. Changing the file's length via the file channel will change the length seen via the originating object, and vice versa. Changing the file's content by writing bytes will change the content seen by the originating object, and vice versa.
>
> This can only work if the RandomAccessFile and the corresponding channel use the same descriptor. So if you create a RandomAccessFile in the ctor and call getChannel() later, just keep the RandomAccessFile open and close it at close.
>
> The current code uses File.openChannel().
>
> -----
> Uwe Schindler
> H.-H.-Meier-Allee 63, D-28213 Bremen
> http://www.thetaphi.de
> eMail: uwe@thetaphi.de
>
>
>> -----Original Message-----
>> From: Uwe Schindler [mailto:uwe@thetaphi.de]
>> Sent: Thursday, January 28, 2010 12:23 PM
>> To: java-dev@lucene.apache.org
>> Subject: RE: Nasty NIO behavior makes NIOFSDirectory silently close
>> channel
>>
>> I am not sure, because RandomAccessFile has a getChannel method, so I
>> suspect that they share the descriptor. Maybe check it with lsof?
>>
>> Possibly we could wait until Simon provides a testcase that fails.
>>
>> -----
>> Uwe Schindler
>> H.-H.-Meier-Allee 63, D-28213 Bremen
>> http://www.thetaphi.de
>> eMail: uwe@thetaphi.de
>>
>>
>> > -----Original Message-----
>> > From: Michael McCandless [mailto:lucene@mikemccandless.com]
>> > Sent: Thursday, January 28, 2010 12:08 PM
>> > To: java-dev@lucene.apache.org
>> > Subject: Re: Nasty NIO behavior makes NIOFSDirectory silently close
>> > channel
>> >
>> > On Thu, Jan 28, 2010 at 5:59 AM, Uwe Schindler <uw...@thetaphi.de>
>> wrote:
>> > > But if you keep the underlying RandomAccessFile open?
>> >
>> > We could do that but... won't this consume 2 file descriptors?
>> >
>> > Mike
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>> > For additional commands, e-mail: java-dev-help@lucene.apache.org
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Nasty NIO behavior makes NIOFSDirectory silently close channel
Posted by John Wang <jo...@gmail.com>.
Re : HW concurrency.
8 core machine, avg latency around 30ms, so with 12 threads, more than
enough room.
-John
On Mon, May 3, 2010 at 10:44 AM, John Wang <jo...@gmail.com> wrote:
> Pretty much pure searching. Not reopening readers. Keeping the reader open
> for the test.
>
> -John
>
>
> On Mon, May 3, 2010 at 2:47 AM, Michael McCandless <
> lucene@mikemccandless.com> wrote:
>
>> This is strange -- it's completely opposite of what others have seen!
>>
>> Other have seen sizable concurrency gains when switching from
>> SimpleFSDir -> NIOFSDir, except on Windows where Sun's long standing
>> JVM bug (http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6265734)
>> prevents concurrency gains.
>>
>> There must be something different about the testing.
>>
>> John are you testing pure searching? Are you reopening readers as
>> well? How much HW concurrency do your machines have?
>>
>> Mike
>>
>> On Sun, May 2, 2010 at 11:53 PM, John Wang <jo...@gmail.com> wrote:
>> > Solaris and MacOs
>> >
>> > On Sun, May 2, 2010 at 4:47 PM, Mark Miller <ma...@gmail.com>
>> wrote:
>> >>
>> >> Interesting ... whats the OS?
>> >>
>> >> On Sun, May 2, 2010 at 7:12 PM, John Wang <jo...@gmail.com> wrote:
>> >>>
>> >>> 12 concurrent search threads in a stress environment. (with about 5M
>> doc
>> >>> index)
>> >>> -John
>> >>>
>> >>> On Sun, May 2, 2010 at 3:13 PM, Mark Miller <ma...@gmail.com>
>> >>> wrote:
>> >>>>
>> >>>> Perfectly safe to use SimpleFSDirectory. When you measure the
>> >>>> performance, are you measuring a lot of concurrent requests? That's
>> where
>> >>>> you should see the win.
>> >>>>
>> >>>> - Mark
>> >>>>
>> >>>> On 5/1/10 6:38 PM, John Wang wrote:
>> >>>>>
>> >>>>> We are seeing this issue as well in your production. (using Zoie on
>> top
>> >>>>> of lucene 2.9.1)
>> >>>>> After some performance comparisons, we do NOT see performance gain
>> with
>> >>>>> NIO, rather these nasty ClosedChannelExceptions.
>> >>>>>
>> >>>>> I think the performance gains ppl are seeing with 2.9.1 can be due
>> to
>> >>>>> many different things. From what we seen, they are not related to
>> >>>>> NIOFSDirectory.
>> >>>>>
>> >>>>> Our solution is to avoid calling FSDirectory.open(), instead just
>> call
>> >>>>> new SimpleFSDirectory(). Is this safe?
>> >>>>>
>> >>>>> -John
>> >>>>>
>> >>>>> On Fri, Jan 29, 2010 at 12:32 PM, Mark Miller <
>> markrmiller@gmail.com
>> >>>>> <ma...@gmail.com>> wrote:
>> >>>>>
>> >>>>> Perhaps - one of the things they are supposed to be addressing is
>> >>>>> extendability.
>> >>>>>
>> >>>>> nio2 does have FileSystemProvider, which would actually allow you
>> to
>> >>>>> create a custom channel !
>> >>>>>
>> >>>>> I have not dug in enough to know much more than that though.
>> >>>>>
>> >>>>> *But*, another really interesting thing is that in Java 7,
>> >>>>> FileDescriptors are ref counted ! (though users can't inc/dec).
>> >>>>>
>> >>>>> But, FileInputStream and OutputStream have a new constructor that
>> >>>>> takes
>> >>>>> a FileDescriptor.
>> >>>>>
>> >>>>> So possibly, you could just make one that sits around to keep the
>> >>>>> FileDescriptor valid, and get your channel off
>> >>>>> FileInputStream/FileOutputStream?
>> >>>>>
>> >>>>> And then if it goes down, make a new one using the FileDescriptor
>> >>>>> which
>> >>>>> was not actually closed because there was a still a ref to it.
>> >>>>>
>> >>>>> Possibly .... ;)
>> >>>>>
>> >>>>> Michael McCandless wrote:
>> >>>>> > Does anyone know if nio2 has improved this...?
>> >>>>> >
>> >>>>> > Mike
>> >>>>> >
>> >>>>> > On Fri, Jan 29, 2010 at 2:00 PM, Jason Rutherglen
>> >>>>> > <jason.rutherglen@gmail.com <mailto:
>> jason.rutherglen@gmail.com>>
>> >>>>> wrote:
>> >>>>> >
>> >>>>> >> Defaulting NIOFSDir could account for some of the recent
>> speed
>> >>>>> >> improvements users have been reporting in Lucene 2.9. So
>> >>>>> removing it
>> >>>>> >> as a default could reverse those and people could then report
>> >>>>> Lucene
>> >>>>> >> 3.X has slowed...
>> >>>>> >>
>> >>>>> >> On Thu, Jan 28, 2010 at 5:24 AM, Michael McCandless
>> >>>>> >> <lucene@mikemccandless.com <mailto:lucene@mikemccandless.com
>> >>
>> >>>>> wrote:
>> >>>>> >>
>> >>>>> >>> Bummer.
>> >>>>> >>>
>> >>>>> >>> So the only viable workarounds are 1) don't use
>> >>>>> Thread.interrupt (nor,
>> >>>>> >>> things like Future.cancel, which in turn use
>> Thread.interrupt)
>> >>>>> with
>> >>>>> >>> NIOFSDir, or 2) we fix NIOFSDir to reopen the channel AND
>> the
>> >>>>> app must
>> >>>>> >>> make a deletion policy that keeps a commit alive if any
>> reader
>> >>>>> is
>> >>>>> >>> using it. Or, 3) don't use NIOFSDir!
>> >>>>> >>>
>> >>>>> >>> Mike
>> >>>>> >>>
>> >>>>> >>> On Thu, Jan 28, 2010 at 7:29 AM, Simon Willnauer
>> >>>>> >>> <simon.willnauer@googlemail.com
>> >>>>> <ma...@googlemail.com>> wrote:
>> >>>>> >>>
>> >>>>> >>>> On Thu, Jan 28, 2010 at 12:43 PM, Michael McCandless
>> >>>>> >>>> <lucene@mikemccandless.com <mailto:
>> lucene@mikemccandless.com>>
>> >>>>> wrote:
>> >>>>> >>>>
>> >>>>> >>>>> On Thu, Jan 28, 2010 at 6:38 AM, Uwe Schindler
>> >>>>> <uwe@thetaphi.de <ma...@thetaphi.de>> wrote:
>> >>>>> >>>>>
>> >>>>> >>>>>
>> >>>>> >>>>>> So I checked the code of NIOFSIndexInput, my last comment
>> >>>>> was not really correct:
>> >>>>> >>>>>> NIOFSIndexInput extends SimpleFSIndexInput and that opens
>> >>>>> the RAF. In the ctor RAF.getChannel() is called. The RAF keeps
>> open
>> >>>>> until the file is closed (and also the channel).
>> >>>>> >>>>>>
>> >>>>> >>>>>> So it's really simple to fix in my opinion, just call
>> >>>>> getChannel() again on this exception. Because the RAF should
>> still
>> >>>>> be open?
>> >>>>> >>>>>>
>> >>>>> >>>> Short answer:
>> >>>>> >>>> public final FileChannel getChannel() {
>> >>>>> >>>> synchronized (this) {
>> >>>>> >>>> if (channel == null)
>> >>>>> >>>> channel = FileChannelImpl.open(fd, true, rw,
>> >>>>> this);
>> >>>>> >>>> return channel;
>> >>>>> >>>> }
>> >>>>> >>>> }
>> >>>>> >>>>
>> >>>>> >>>> this is not gonna work I tried it before. The
>> RandomAccessFile
>> >>>>> buffers
>> >>>>> >>>> the channel!!
>> >>>>> >>>>
>> >>>>> >>>> simon
>> >>>>> >>>>
>> >>>>> >>>>> I think we need a definitive answer on what happens to the
>> >>>>> RAF when
>> >>>>> >>>>> the FileChannel was closed by Thread.Interrupt. Simon can
>> >>>>> you test
>> >>>>> >>>>> this?
>> >>>>> >>>>>
>> >>>>> >>>>> Mike
>> >>>>> >>>>>
>> >>>>> >>>>>
>> >>>>>
>> >>>>>
>> ---------------------------------------------------------------------
>> >>>>> >>>>> To unsubscribe, e-mail:
>> >>>>> java-dev-unsubscribe@lucene.apache.org
>> >>>>> <ma...@lucene.apache.org>
>> >>>>> >>>>> For additional commands, e-mail:
>> >>>>> java-dev-help@lucene.apache.org
>> >>>>> <ma...@lucene.apache.org>
>> >>>>> >>>>>
>> >>>>> >>>>>
>> >>>>> >>>>>
>> >>>>> >>>>
>> >>>>>
>> >>>>>
>> ---------------------------------------------------------------------
>> >>>>> >>>> To unsubscribe, e-mail:
>> java-dev-unsubscribe@lucene.apache.org
>> >>>>> <ma...@lucene.apache.org>
>> >>>>> >>>> For additional commands, e-mail:
>> >>>>> java-dev-help@lucene.apache.org
>> >>>>> <ma...@lucene.apache.org>
>> >>>>> >>>>
>> >>>>> >>>>
>> >>>>> >>>>
>> >>>>> >>>
>> >>>>>
>> >>>>>
>> ---------------------------------------------------------------------
>> >>>>> >>> To unsubscribe, e-mail:
>> java-dev-unsubscribe@lucene.apache.org
>> >>>>> <ma...@lucene.apache.org>
>> >>>>> >>> For additional commands, e-mail:
>> >>>>> java-dev-help@lucene.apache.org
>> >>>>> <ma...@lucene.apache.org>
>> >>>>> >>>
>> >>>>> >>>
>> >>>>> >>>
>> >>>>> >>
>> >>>>>
>> >>>>>
>> ---------------------------------------------------------------------
>> >>>>> >> To unsubscribe, e-mail:
>> java-dev-unsubscribe@lucene.apache.org
>> >>>>> <ma...@lucene.apache.org>
>> >>>>> >> For additional commands, e-mail:
>> java-dev-help@lucene.apache.org
>> >>>>> <ma...@lucene.apache.org>
>> >>>>> >>
>> >>>>> >>
>> >>>>> >>
>> >>>>> >
>> >>>>> >
>> >>>>>
>> ---------------------------------------------------------------------
>> >>>>> > To unsubscribe, e-mail:
>> java-dev-unsubscribe@lucene.apache.org
>> >>>>> <ma...@lucene.apache.org>
>> >>>>> > For additional commands, e-mail:
>> java-dev-help@lucene.apache.org
>> >>>>> <ma...@lucene.apache.org>
>> >>>>> >
>> >>>>> >
>> >>>>>
>> >>>>>
>> >>>>> --
>> >>>>> - Mark
>> >>>>>
>> >>>>> http://www.lucidimagination.com
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>>
>> ---------------------------------------------------------------------
>> >>>>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>> >>>>> <ma...@lucene.apache.org>
>> >>>>> For additional commands, e-mail: java-dev-help@lucene.apache.org
>> >>>>> <ma...@lucene.apache.org>
>> >>>>>
>> >>>>>
>> >>>>
>> >>>>
>> >>>> --
>> >>>> - Mark
>> >>>>
>> >>>> http://www.lucidimagination.com
>> >>>>
>> >>>> ---------------------------------------------------------------------
>> >>>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>> >>>> For additional commands, e-mail: dev-help@lucene.apache.org
>> >>>>
>> >>>
>> >>
>> >>
>> >>
>> >> --
>> >> - Mark
>> >>
>> >> http://www.lucidimagination.com
>> >>
>> >
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: dev-help@lucene.apache.org
>>
>>
>
Re: Nasty NIO behavior makes NIOFSDirectory silently close channel
Posted by John Wang <jo...@gmail.com>.
Pretty much pure searching. Not reopening readers. Keeping the reader open
for the test.
-John
On Mon, May 3, 2010 at 2:47 AM, Michael McCandless <
lucene@mikemccandless.com> wrote:
> This is strange -- it's completely opposite of what others have seen!
>
> Other have seen sizable concurrency gains when switching from
> SimpleFSDir -> NIOFSDir, except on Windows where Sun's long standing
> JVM bug (http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6265734)
> prevents concurrency gains.
>
> There must be something different about the testing.
>
> John are you testing pure searching? Are you reopening readers as
> well? How much HW concurrency do your machines have?
>
> Mike
>
> On Sun, May 2, 2010 at 11:53 PM, John Wang <jo...@gmail.com> wrote:
> > Solaris and MacOs
> >
> > On Sun, May 2, 2010 at 4:47 PM, Mark Miller <ma...@gmail.com>
> wrote:
> >>
> >> Interesting ... whats the OS?
> >>
> >> On Sun, May 2, 2010 at 7:12 PM, John Wang <jo...@gmail.com> wrote:
> >>>
> >>> 12 concurrent search threads in a stress environment. (with about 5M
> doc
> >>> index)
> >>> -John
> >>>
> >>> On Sun, May 2, 2010 at 3:13 PM, Mark Miller <ma...@gmail.com>
> >>> wrote:
> >>>>
> >>>> Perfectly safe to use SimpleFSDirectory. When you measure the
> >>>> performance, are you measuring a lot of concurrent requests? That's
> where
> >>>> you should see the win.
> >>>>
> >>>> - Mark
> >>>>
> >>>> On 5/1/10 6:38 PM, John Wang wrote:
> >>>>>
> >>>>> We are seeing this issue as well in your production. (using Zoie on
> top
> >>>>> of lucene 2.9.1)
> >>>>> After some performance comparisons, we do NOT see performance gain
> with
> >>>>> NIO, rather these nasty ClosedChannelExceptions.
> >>>>>
> >>>>> I think the performance gains ppl are seeing with 2.9.1 can be due to
> >>>>> many different things. From what we seen, they are not related to
> >>>>> NIOFSDirectory.
> >>>>>
> >>>>> Our solution is to avoid calling FSDirectory.open(), instead just
> call
> >>>>> new SimpleFSDirectory(). Is this safe?
> >>>>>
> >>>>> -John
> >>>>>
> >>>>> On Fri, Jan 29, 2010 at 12:32 PM, Mark Miller <markrmiller@gmail.com
> >>>>> <ma...@gmail.com>> wrote:
> >>>>>
> >>>>> Perhaps - one of the things they are supposed to be addressing is
> >>>>> extendability.
> >>>>>
> >>>>> nio2 does have FileSystemProvider, which would actually allow you
> to
> >>>>> create a custom channel !
> >>>>>
> >>>>> I have not dug in enough to know much more than that though.
> >>>>>
> >>>>> *But*, another really interesting thing is that in Java 7,
> >>>>> FileDescriptors are ref counted ! (though users can't inc/dec).
> >>>>>
> >>>>> But, FileInputStream and OutputStream have a new constructor that
> >>>>> takes
> >>>>> a FileDescriptor.
> >>>>>
> >>>>> So possibly, you could just make one that sits around to keep the
> >>>>> FileDescriptor valid, and get your channel off
> >>>>> FileInputStream/FileOutputStream?
> >>>>>
> >>>>> And then if it goes down, make a new one using the FileDescriptor
> >>>>> which
> >>>>> was not actually closed because there was a still a ref to it.
> >>>>>
> >>>>> Possibly .... ;)
> >>>>>
> >>>>> Michael McCandless wrote:
> >>>>> > Does anyone know if nio2 has improved this...?
> >>>>> >
> >>>>> > Mike
> >>>>> >
> >>>>> > On Fri, Jan 29, 2010 at 2:00 PM, Jason Rutherglen
> >>>>> > <jason.rutherglen@gmail.com <mailto:jason.rutherglen@gmail.com
> >>
> >>>>> wrote:
> >>>>> >
> >>>>> >> Defaulting NIOFSDir could account for some of the recent speed
> >>>>> >> improvements users have been reporting in Lucene 2.9. So
> >>>>> removing it
> >>>>> >> as a default could reverse those and people could then report
> >>>>> Lucene
> >>>>> >> 3.X has slowed...
> >>>>> >>
> >>>>> >> On Thu, Jan 28, 2010 at 5:24 AM, Michael McCandless
> >>>>> >> <lucene@mikemccandless.com <mailto:lucene@mikemccandless.com
> >>
> >>>>> wrote:
> >>>>> >>
> >>>>> >>> Bummer.
> >>>>> >>>
> >>>>> >>> So the only viable workarounds are 1) don't use
> >>>>> Thread.interrupt (nor,
> >>>>> >>> things like Future.cancel, which in turn use
> Thread.interrupt)
> >>>>> with
> >>>>> >>> NIOFSDir, or 2) we fix NIOFSDir to reopen the channel AND the
> >>>>> app must
> >>>>> >>> make a deletion policy that keeps a commit alive if any
> reader
> >>>>> is
> >>>>> >>> using it. Or, 3) don't use NIOFSDir!
> >>>>> >>>
> >>>>> >>> Mike
> >>>>> >>>
> >>>>> >>> On Thu, Jan 28, 2010 at 7:29 AM, Simon Willnauer
> >>>>> >>> <simon.willnauer@googlemail.com
> >>>>> <ma...@googlemail.com>> wrote:
> >>>>> >>>
> >>>>> >>>> On Thu, Jan 28, 2010 at 12:43 PM, Michael McCandless
> >>>>> >>>> <lucene@mikemccandless.com <mailto:
> lucene@mikemccandless.com>>
> >>>>> wrote:
> >>>>> >>>>
> >>>>> >>>>> On Thu, Jan 28, 2010 at 6:38 AM, Uwe Schindler
> >>>>> <uwe@thetaphi.de <ma...@thetaphi.de>> wrote:
> >>>>> >>>>>
> >>>>> >>>>>
> >>>>> >>>>>> So I checked the code of NIOFSIndexInput, my last comment
> >>>>> was not really correct:
> >>>>> >>>>>> NIOFSIndexInput extends SimpleFSIndexInput and that opens
> >>>>> the RAF. In the ctor RAF.getChannel() is called. The RAF keeps
> open
> >>>>> until the file is closed (and also the channel).
> >>>>> >>>>>>
> >>>>> >>>>>> So it's really simple to fix in my opinion, just call
> >>>>> getChannel() again on this exception. Because the RAF should still
> >>>>> be open?
> >>>>> >>>>>>
> >>>>> >>>> Short answer:
> >>>>> >>>> public final FileChannel getChannel() {
> >>>>> >>>> synchronized (this) {
> >>>>> >>>> if (channel == null)
> >>>>> >>>> channel = FileChannelImpl.open(fd, true, rw,
> >>>>> this);
> >>>>> >>>> return channel;
> >>>>> >>>> }
> >>>>> >>>> }
> >>>>> >>>>
> >>>>> >>>> this is not gonna work I tried it before. The
> RandomAccessFile
> >>>>> buffers
> >>>>> >>>> the channel!!
> >>>>> >>>>
> >>>>> >>>> simon
> >>>>> >>>>
> >>>>> >>>>> I think we need a definitive answer on what happens to the
> >>>>> RAF when
> >>>>> >>>>> the FileChannel was closed by Thread.Interrupt. Simon can
> >>>>> you test
> >>>>> >>>>> this?
> >>>>> >>>>>
> >>>>> >>>>> Mike
> >>>>> >>>>>
> >>>>> >>>>>
> >>>>>
> >>>>>
> ---------------------------------------------------------------------
> >>>>> >>>>> To unsubscribe, e-mail:
> >>>>> java-dev-unsubscribe@lucene.apache.org
> >>>>> <ma...@lucene.apache.org>
> >>>>> >>>>> For additional commands, e-mail:
> >>>>> java-dev-help@lucene.apache.org
> >>>>> <ma...@lucene.apache.org>
> >>>>> >>>>>
> >>>>> >>>>>
> >>>>> >>>>>
> >>>>> >>>>
> >>>>>
> >>>>>
> ---------------------------------------------------------------------
> >>>>> >>>> To unsubscribe, e-mail:
> java-dev-unsubscribe@lucene.apache.org
> >>>>> <ma...@lucene.apache.org>
> >>>>> >>>> For additional commands, e-mail:
> >>>>> java-dev-help@lucene.apache.org
> >>>>> <ma...@lucene.apache.org>
> >>>>> >>>>
> >>>>> >>>>
> >>>>> >>>>
> >>>>> >>>
> >>>>>
> >>>>>
> ---------------------------------------------------------------------
> >>>>> >>> To unsubscribe, e-mail:
> java-dev-unsubscribe@lucene.apache.org
> >>>>> <ma...@lucene.apache.org>
> >>>>> >>> For additional commands, e-mail:
> >>>>> java-dev-help@lucene.apache.org
> >>>>> <ma...@lucene.apache.org>
> >>>>> >>>
> >>>>> >>>
> >>>>> >>>
> >>>>> >>
> >>>>>
> >>>>>
> ---------------------------------------------------------------------
> >>>>> >> To unsubscribe, e-mail:
> java-dev-unsubscribe@lucene.apache.org
> >>>>> <ma...@lucene.apache.org>
> >>>>> >> For additional commands, e-mail:
> java-dev-help@lucene.apache.org
> >>>>> <ma...@lucene.apache.org>
> >>>>> >>
> >>>>> >>
> >>>>> >>
> >>>>> >
> >>>>> >
> >>>>> ---------------------------------------------------------------------
> >>>>> > To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> >>>>> <ma...@lucene.apache.org>
> >>>>> > For additional commands, e-mail:
> java-dev-help@lucene.apache.org
> >>>>> <ma...@lucene.apache.org>
> >>>>> >
> >>>>> >
> >>>>>
> >>>>>
> >>>>> --
> >>>>> - Mark
> >>>>>
> >>>>> http://www.lucidimagination.com
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> ---------------------------------------------------------------------
> >>>>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> >>>>> <ma...@lucene.apache.org>
> >>>>> For additional commands, e-mail: java-dev-help@lucene.apache.org
> >>>>> <ma...@lucene.apache.org>
> >>>>>
> >>>>>
> >>>>
> >>>>
> >>>> --
> >>>> - Mark
> >>>>
> >>>> http://www.lucidimagination.com
> >>>>
> >>>> ---------------------------------------------------------------------
> >>>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> >>>> For additional commands, e-mail: dev-help@lucene.apache.org
> >>>>
> >>>
> >>
> >>
> >>
> >> --
> >> - Mark
> >>
> >> http://www.lucidimagination.com
> >>
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
>
Re: Nasty NIO behavior makes NIOFSDirectory silently close channel
Posted by Michael McCandless <lu...@mikemccandless.com>.
This is strange -- it's completely opposite of what others have seen!
Other have seen sizable concurrency gains when switching from
SimpleFSDir -> NIOFSDir, except on Windows where Sun's long standing
JVM bug (http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6265734)
prevents concurrency gains.
There must be something different about the testing.
John are you testing pure searching? Are you reopening readers as
well? How much HW concurrency do your machines have?
Mike
On Sun, May 2, 2010 at 11:53 PM, John Wang <jo...@gmail.com> wrote:
> Solaris and MacOs
>
> On Sun, May 2, 2010 at 4:47 PM, Mark Miller <ma...@gmail.com> wrote:
>>
>> Interesting ... whats the OS?
>>
>> On Sun, May 2, 2010 at 7:12 PM, John Wang <jo...@gmail.com> wrote:
>>>
>>> 12 concurrent search threads in a stress environment. (with about 5M doc
>>> index)
>>> -John
>>>
>>> On Sun, May 2, 2010 at 3:13 PM, Mark Miller <ma...@gmail.com>
>>> wrote:
>>>>
>>>> Perfectly safe to use SimpleFSDirectory. When you measure the
>>>> performance, are you measuring a lot of concurrent requests? That's where
>>>> you should see the win.
>>>>
>>>> - Mark
>>>>
>>>> On 5/1/10 6:38 PM, John Wang wrote:
>>>>>
>>>>> We are seeing this issue as well in your production. (using Zoie on top
>>>>> of lucene 2.9.1)
>>>>> After some performance comparisons, we do NOT see performance gain with
>>>>> NIO, rather these nasty ClosedChannelExceptions.
>>>>>
>>>>> I think the performance gains ppl are seeing with 2.9.1 can be due to
>>>>> many different things. From what we seen, they are not related to
>>>>> NIOFSDirectory.
>>>>>
>>>>> Our solution is to avoid calling FSDirectory.open(), instead just call
>>>>> new SimpleFSDirectory(). Is this safe?
>>>>>
>>>>> -John
>>>>>
>>>>> On Fri, Jan 29, 2010 at 12:32 PM, Mark Miller <markrmiller@gmail.com
>>>>> <ma...@gmail.com>> wrote:
>>>>>
>>>>> Perhaps - one of the things they are supposed to be addressing is
>>>>> extendability.
>>>>>
>>>>> nio2 does have FileSystemProvider, which would actually allow you to
>>>>> create a custom channel !
>>>>>
>>>>> I have not dug in enough to know much more than that though.
>>>>>
>>>>> *But*, another really interesting thing is that in Java 7,
>>>>> FileDescriptors are ref counted ! (though users can't inc/dec).
>>>>>
>>>>> But, FileInputStream and OutputStream have a new constructor that
>>>>> takes
>>>>> a FileDescriptor.
>>>>>
>>>>> So possibly, you could just make one that sits around to keep the
>>>>> FileDescriptor valid, and get your channel off
>>>>> FileInputStream/FileOutputStream?
>>>>>
>>>>> And then if it goes down, make a new one using the FileDescriptor
>>>>> which
>>>>> was not actually closed because there was a still a ref to it.
>>>>>
>>>>> Possibly .... ;)
>>>>>
>>>>> Michael McCandless wrote:
>>>>> > Does anyone know if nio2 has improved this...?
>>>>> >
>>>>> > Mike
>>>>> >
>>>>> > On Fri, Jan 29, 2010 at 2:00 PM, Jason Rutherglen
>>>>> > <jason.rutherglen@gmail.com <ma...@gmail.com>>
>>>>> wrote:
>>>>> >
>>>>> >> Defaulting NIOFSDir could account for some of the recent speed
>>>>> >> improvements users have been reporting in Lucene 2.9. So
>>>>> removing it
>>>>> >> as a default could reverse those and people could then report
>>>>> Lucene
>>>>> >> 3.X has slowed...
>>>>> >>
>>>>> >> On Thu, Jan 28, 2010 at 5:24 AM, Michael McCandless
>>>>> >> <lucene@mikemccandless.com <ma...@mikemccandless.com>>
>>>>> wrote:
>>>>> >>
>>>>> >>> Bummer.
>>>>> >>>
>>>>> >>> So the only viable workarounds are 1) don't use
>>>>> Thread.interrupt (nor,
>>>>> >>> things like Future.cancel, which in turn use Thread.interrupt)
>>>>> with
>>>>> >>> NIOFSDir, or 2) we fix NIOFSDir to reopen the channel AND the
>>>>> app must
>>>>> >>> make a deletion policy that keeps a commit alive if any reader
>>>>> is
>>>>> >>> using it. Or, 3) don't use NIOFSDir!
>>>>> >>>
>>>>> >>> Mike
>>>>> >>>
>>>>> >>> On Thu, Jan 28, 2010 at 7:29 AM, Simon Willnauer
>>>>> >>> <simon.willnauer@googlemail.com
>>>>> <ma...@googlemail.com>> wrote:
>>>>> >>>
>>>>> >>>> On Thu, Jan 28, 2010 at 12:43 PM, Michael McCandless
>>>>> >>>> <lucene@mikemccandless.com <ma...@mikemccandless.com>>
>>>>> wrote:
>>>>> >>>>
>>>>> >>>>> On Thu, Jan 28, 2010 at 6:38 AM, Uwe Schindler
>>>>> <uwe@thetaphi.de <ma...@thetaphi.de>> wrote:
>>>>> >>>>>
>>>>> >>>>>
>>>>> >>>>>> So I checked the code of NIOFSIndexInput, my last comment
>>>>> was not really correct:
>>>>> >>>>>> NIOFSIndexInput extends SimpleFSIndexInput and that opens
>>>>> the RAF. In the ctor RAF.getChannel() is called. The RAF keeps open
>>>>> until the file is closed (and also the channel).
>>>>> >>>>>>
>>>>> >>>>>> So it's really simple to fix in my opinion, just call
>>>>> getChannel() again on this exception. Because the RAF should still
>>>>> be open?
>>>>> >>>>>>
>>>>> >>>> Short answer:
>>>>> >>>> public final FileChannel getChannel() {
>>>>> >>>> synchronized (this) {
>>>>> >>>> if (channel == null)
>>>>> >>>> channel = FileChannelImpl.open(fd, true, rw,
>>>>> this);
>>>>> >>>> return channel;
>>>>> >>>> }
>>>>> >>>> }
>>>>> >>>>
>>>>> >>>> this is not gonna work I tried it before. The RandomAccessFile
>>>>> buffers
>>>>> >>>> the channel!!
>>>>> >>>>
>>>>> >>>> simon
>>>>> >>>>
>>>>> >>>>> I think we need a definitive answer on what happens to the
>>>>> RAF when
>>>>> >>>>> the FileChannel was closed by Thread.Interrupt. Simon can
>>>>> you test
>>>>> >>>>> this?
>>>>> >>>>>
>>>>> >>>>> Mike
>>>>> >>>>>
>>>>> >>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> >>>>> To unsubscribe, e-mail:
>>>>> java-dev-unsubscribe@lucene.apache.org
>>>>> <ma...@lucene.apache.org>
>>>>> >>>>> For additional commands, e-mail:
>>>>> java-dev-help@lucene.apache.org
>>>>> <ma...@lucene.apache.org>
>>>>> >>>>>
>>>>> >>>>>
>>>>> >>>>>
>>>>> >>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> >>>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>>>>> <ma...@lucene.apache.org>
>>>>> >>>> For additional commands, e-mail:
>>>>> java-dev-help@lucene.apache.org
>>>>> <ma...@lucene.apache.org>
>>>>> >>>>
>>>>> >>>>
>>>>> >>>>
>>>>> >>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> >>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>>>>> <ma...@lucene.apache.org>
>>>>> >>> For additional commands, e-mail:
>>>>> java-dev-help@lucene.apache.org
>>>>> <ma...@lucene.apache.org>
>>>>> >>>
>>>>> >>>
>>>>> >>>
>>>>> >>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> >> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>>>>> <ma...@lucene.apache.org>
>>>>> >> For additional commands, e-mail: java-dev-help@lucene.apache.org
>>>>> <ma...@lucene.apache.org>
>>>>> >>
>>>>> >>
>>>>> >>
>>>>> >
>>>>> >
>>>>> ---------------------------------------------------------------------
>>>>> > To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>>>>> <ma...@lucene.apache.org>
>>>>> > For additional commands, e-mail: java-dev-help@lucene.apache.org
>>>>> <ma...@lucene.apache.org>
>>>>> >
>>>>> >
>>>>>
>>>>>
>>>>> --
>>>>> - Mark
>>>>>
>>>>> http://www.lucidimagination.com
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>>>>> <ma...@lucene.apache.org>
>>>>> For additional commands, e-mail: java-dev-help@lucene.apache.org
>>>>> <ma...@lucene.apache.org>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> - Mark
>>>>
>>>> http://www.lucidimagination.com
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>>>> For additional commands, e-mail: dev-help@lucene.apache.org
>>>>
>>>
>>
>>
>>
>> --
>> - Mark
>>
>> http://www.lucidimagination.com
>>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: Nasty NIO behavior makes NIOFSDirectory silently close channel
Posted by John Wang <jo...@gmail.com>.
Solaris and MacOs
On Sun, May 2, 2010 at 4:47 PM, Mark Miller <ma...@gmail.com> wrote:
> Interesting ... whats the OS?
>
>
> On Sun, May 2, 2010 at 7:12 PM, John Wang <jo...@gmail.com> wrote:
>
>> 12 concurrent search threads in a stress environment. (with about 5M doc
>> index)
>>
>> -John
>>
>> On Sun, May 2, 2010 at 3:13 PM, Mark Miller <ma...@gmail.com>wrote:
>>
>>> Perfectly safe to use SimpleFSDirectory. When you measure the
>>> performance, are you measuring a lot of concurrent requests? That's where
>>> you should see the win.
>>>
>>> - Mark
>>>
>>>
>>> On 5/1/10 6:38 PM, John Wang wrote:
>>>
>>>> We are seeing this issue as well in your production. (using Zoie on top
>>>> of lucene 2.9.1)
>>>> After some performance comparisons, we do NOT see performance gain with
>>>> NIO, rather these nasty ClosedChannelExceptions.
>>>>
>>>> I think the performance gains ppl are seeing with 2.9.1 can be due to
>>>> many different things. From what we seen, they are not related to
>>>> NIOFSDirectory.
>>>>
>>>> Our solution is to avoid calling FSDirectory.open(), instead just call
>>>> new SimpleFSDirectory(). Is this safe?
>>>>
>>>> -John
>>>>
>>>> On Fri, Jan 29, 2010 at 12:32 PM, Mark Miller <markrmiller@gmail.com
>>>> <ma...@gmail.com>> wrote:
>>>>
>>>> Perhaps - one of the things they are supposed to be addressing is
>>>> extendability.
>>>>
>>>> nio2 does have FileSystemProvider, which would actually allow you to
>>>> create a custom channel !
>>>>
>>>> I have not dug in enough to know much more than that though.
>>>>
>>>> *But*, another really interesting thing is that in Java 7,
>>>> FileDescriptors are ref counted ! (though users can't inc/dec).
>>>>
>>>> But, FileInputStream and OutputStream have a new constructor that
>>>> takes
>>>> a FileDescriptor.
>>>>
>>>> So possibly, you could just make one that sits around to keep the
>>>> FileDescriptor valid, and get your channel off
>>>> FileInputStream/FileOutputStream?
>>>>
>>>> And then if it goes down, make a new one using the FileDescriptor
>>>> which
>>>> was not actually closed because there was a still a ref to it.
>>>>
>>>> Possibly .... ;)
>>>>
>>>> Michael McCandless wrote:
>>>> > Does anyone know if nio2 has improved this...?
>>>> >
>>>> > Mike
>>>> >
>>>> > On Fri, Jan 29, 2010 at 2:00 PM, Jason Rutherglen
>>>> > <jason.rutherglen@gmail.com <ma...@gmail.com>>
>>>>
>>>> wrote:
>>>> >
>>>> >> Defaulting NIOFSDir could account for some of the recent speed
>>>> >> improvements users have been reporting in Lucene 2.9. So
>>>> removing it
>>>> >> as a default could reverse those and people could then report
>>>> Lucene
>>>> >> 3.X has slowed...
>>>> >>
>>>> >> On Thu, Jan 28, 2010 at 5:24 AM, Michael McCandless
>>>> >> <lucene@mikemccandless.com <ma...@mikemccandless.com>>
>>>>
>>>> wrote:
>>>> >>
>>>> >>> Bummer.
>>>> >>>
>>>> >>> So the only viable workarounds are 1) don't use
>>>> Thread.interrupt (nor,
>>>> >>> things like Future.cancel, which in turn use Thread.interrupt)
>>>> with
>>>> >>> NIOFSDir, or 2) we fix NIOFSDir to reopen the channel AND the
>>>> app must
>>>> >>> make a deletion policy that keeps a commit alive if any reader
>>>> is
>>>> >>> using it. Or, 3) don't use NIOFSDir!
>>>> >>>
>>>> >>> Mike
>>>> >>>
>>>> >>> On Thu, Jan 28, 2010 at 7:29 AM, Simon Willnauer
>>>> >>> <simon.willnauer@googlemail.com
>>>> <ma...@googlemail.com>> wrote:
>>>> >>>
>>>> >>>> On Thu, Jan 28, 2010 at 12:43 PM, Michael McCandless
>>>> >>>> <lucene@mikemccandless.com <ma...@mikemccandless.com>>
>>>>
>>>> wrote:
>>>> >>>>
>>>> >>>>> On Thu, Jan 28, 2010 at 6:38 AM, Uwe Schindler
>>>> <uwe@thetaphi.de <ma...@thetaphi.de>> wrote:
>>>> >>>>>
>>>> >>>>>
>>>> >>>>>> So I checked the code of NIOFSIndexInput, my last comment
>>>> was not really correct:
>>>> >>>>>> NIOFSIndexInput extends SimpleFSIndexInput and that opens
>>>> the RAF. In the ctor RAF.getChannel() is called. The RAF keeps open
>>>> until the file is closed (and also the channel).
>>>> >>>>>>
>>>> >>>>>> So it's really simple to fix in my opinion, just call
>>>> getChannel() again on this exception. Because the RAF should still
>>>> be open?
>>>> >>>>>>
>>>> >>>> Short answer:
>>>> >>>> public final FileChannel getChannel() {
>>>> >>>> synchronized (this) {
>>>> >>>> if (channel == null)
>>>> >>>> channel = FileChannelImpl.open(fd, true, rw,
>>>> this);
>>>> >>>> return channel;
>>>> >>>> }
>>>> >>>> }
>>>> >>>>
>>>> >>>> this is not gonna work I tried it before. The RandomAccessFile
>>>> buffers
>>>> >>>> the channel!!
>>>> >>>>
>>>> >>>> simon
>>>> >>>>
>>>> >>>>> I think we need a definitive answer on what happens to the
>>>> RAF when
>>>> >>>>> the FileChannel was closed by Thread.Interrupt. Simon can
>>>> you test
>>>> >>>>> this?
>>>> >>>>>
>>>> >>>>> Mike
>>>> >>>>>
>>>> >>>>>
>>>> ---------------------------------------------------------------------
>>>> >>>>> To unsubscribe, e-mail:
>>>> java-dev-unsubscribe@lucene.apache.org
>>>> <ma...@lucene.apache.org>
>>>>
>>>> >>>>> For additional commands, e-mail:
>>>> java-dev-help@lucene.apache.org <mailto:
>>>> java-dev-help@lucene.apache.org>
>>>>
>>>> >>>>>
>>>> >>>>>
>>>> >>>>>
>>>> >>>>
>>>> ---------------------------------------------------------------------
>>>> >>>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>>>> <ma...@lucene.apache.org>
>>>>
>>>> >>>> For additional commands, e-mail:
>>>> java-dev-help@lucene.apache.org <mailto:
>>>> java-dev-help@lucene.apache.org>
>>>>
>>>> >>>>
>>>> >>>>
>>>> >>>>
>>>> >>>
>>>> ---------------------------------------------------------------------
>>>> >>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>>>> <ma...@lucene.apache.org>
>>>>
>>>> >>> For additional commands, e-mail:
>>>> java-dev-help@lucene.apache.org <mailto:
>>>> java-dev-help@lucene.apache.org>
>>>>
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>
>>>> ---------------------------------------------------------------------
>>>> >> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>>>> <ma...@lucene.apache.org>
>>>>
>>>> >> For additional commands, e-mail: java-dev-help@lucene.apache.org
>>>> <ma...@lucene.apache.org>
>>>>
>>>> >>
>>>> >>
>>>> >>
>>>> >
>>>> >
>>>> ---------------------------------------------------------------------
>>>> > To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>>>> <ma...@lucene.apache.org>
>>>>
>>>> > For additional commands, e-mail: java-dev-help@lucene.apache.org
>>>> <ma...@lucene.apache.org>
>>>>
>>>> >
>>>> >
>>>>
>>>>
>>>> --
>>>> - Mark
>>>>
>>>> http://www.lucidimagination.com
>>>>
>>>>
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>>>> <ma...@lucene.apache.org>
>>>>
>>>> For additional commands, e-mail: java-dev-help@lucene.apache.org
>>>> <ma...@lucene.apache.org>
>>>>
>>>>
>>>>
>>>
>>> --
>>> - Mark
>>>
>>> http://www.lucidimagination.com
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: dev-help@lucene.apache.org
>>>
>>>
>>
>
>
> --
> - Mark
>
> http://www.lucidimagination.com
>
>
Re: Nasty NIO behavior makes NIOFSDirectory silently close channel
Posted by Mark Miller <ma...@gmail.com>.
Interesting ... whats the OS?
On Sun, May 2, 2010 at 7:12 PM, John Wang <jo...@gmail.com> wrote:
> 12 concurrent search threads in a stress environment. (with about 5M doc
> index)
>
> -John
>
> On Sun, May 2, 2010 at 3:13 PM, Mark Miller <ma...@gmail.com> wrote:
>
>> Perfectly safe to use SimpleFSDirectory. When you measure the performance,
>> are you measuring a lot of concurrent requests? That's where you should see
>> the win.
>>
>> - Mark
>>
>>
>> On 5/1/10 6:38 PM, John Wang wrote:
>>
>>> We are seeing this issue as well in your production. (using Zoie on top
>>> of lucene 2.9.1)
>>> After some performance comparisons, we do NOT see performance gain with
>>> NIO, rather these nasty ClosedChannelExceptions.
>>>
>>> I think the performance gains ppl are seeing with 2.9.1 can be due to
>>> many different things. From what we seen, they are not related to
>>> NIOFSDirectory.
>>>
>>> Our solution is to avoid calling FSDirectory.open(), instead just call
>>> new SimpleFSDirectory(). Is this safe?
>>>
>>> -John
>>>
>>> On Fri, Jan 29, 2010 at 12:32 PM, Mark Miller <markrmiller@gmail.com
>>> <ma...@gmail.com>> wrote:
>>>
>>> Perhaps - one of the things they are supposed to be addressing is
>>> extendability.
>>>
>>> nio2 does have FileSystemProvider, which would actually allow you to
>>> create a custom channel !
>>>
>>> I have not dug in enough to know much more than that though.
>>>
>>> *But*, another really interesting thing is that in Java 7,
>>> FileDescriptors are ref counted ! (though users can't inc/dec).
>>>
>>> But, FileInputStream and OutputStream have a new constructor that
>>> takes
>>> a FileDescriptor.
>>>
>>> So possibly, you could just make one that sits around to keep the
>>> FileDescriptor valid, and get your channel off
>>> FileInputStream/FileOutputStream?
>>>
>>> And then if it goes down, make a new one using the FileDescriptor
>>> which
>>> was not actually closed because there was a still a ref to it.
>>>
>>> Possibly .... ;)
>>>
>>> Michael McCandless wrote:
>>> > Does anyone know if nio2 has improved this...?
>>> >
>>> > Mike
>>> >
>>> > On Fri, Jan 29, 2010 at 2:00 PM, Jason Rutherglen
>>> > <jason.rutherglen@gmail.com <ma...@gmail.com>>
>>>
>>> wrote:
>>> >
>>> >> Defaulting NIOFSDir could account for some of the recent speed
>>> >> improvements users have been reporting in Lucene 2.9. So
>>> removing it
>>> >> as a default could reverse those and people could then report
>>> Lucene
>>> >> 3.X has slowed...
>>> >>
>>> >> On Thu, Jan 28, 2010 at 5:24 AM, Michael McCandless
>>> >> <lucene@mikemccandless.com <ma...@mikemccandless.com>>
>>>
>>> wrote:
>>> >>
>>> >>> Bummer.
>>> >>>
>>> >>> So the only viable workarounds are 1) don't use
>>> Thread.interrupt (nor,
>>> >>> things like Future.cancel, which in turn use Thread.interrupt)
>>> with
>>> >>> NIOFSDir, or 2) we fix NIOFSDir to reopen the channel AND the
>>> app must
>>> >>> make a deletion policy that keeps a commit alive if any reader is
>>> >>> using it. Or, 3) don't use NIOFSDir!
>>> >>>
>>> >>> Mike
>>> >>>
>>> >>> On Thu, Jan 28, 2010 at 7:29 AM, Simon Willnauer
>>> >>> <simon.willnauer@googlemail.com
>>> <ma...@googlemail.com>> wrote:
>>> >>>
>>> >>>> On Thu, Jan 28, 2010 at 12:43 PM, Michael McCandless
>>> >>>> <lucene@mikemccandless.com <ma...@mikemccandless.com>>
>>>
>>> wrote:
>>> >>>>
>>> >>>>> On Thu, Jan 28, 2010 at 6:38 AM, Uwe Schindler
>>> <uwe@thetaphi.de <ma...@thetaphi.de>> wrote:
>>> >>>>>
>>> >>>>>
>>> >>>>>> So I checked the code of NIOFSIndexInput, my last comment
>>> was not really correct:
>>> >>>>>> NIOFSIndexInput extends SimpleFSIndexInput and that opens
>>> the RAF. In the ctor RAF.getChannel() is called. The RAF keeps open
>>> until the file is closed (and also the channel).
>>> >>>>>>
>>> >>>>>> So it's really simple to fix in my opinion, just call
>>> getChannel() again on this exception. Because the RAF should still
>>> be open?
>>> >>>>>>
>>> >>>> Short answer:
>>> >>>> public final FileChannel getChannel() {
>>> >>>> synchronized (this) {
>>> >>>> if (channel == null)
>>> >>>> channel = FileChannelImpl.open(fd, true, rw,
>>> this);
>>> >>>> return channel;
>>> >>>> }
>>> >>>> }
>>> >>>>
>>> >>>> this is not gonna work I tried it before. The RandomAccessFile
>>> buffers
>>> >>>> the channel!!
>>> >>>>
>>> >>>> simon
>>> >>>>
>>> >>>>> I think we need a definitive answer on what happens to the
>>> RAF when
>>> >>>>> the FileChannel was closed by Thread.Interrupt. Simon can
>>> you test
>>> >>>>> this?
>>> >>>>>
>>> >>>>> Mike
>>> >>>>>
>>> >>>>>
>>> ---------------------------------------------------------------------
>>> >>>>> To unsubscribe, e-mail:
>>> java-dev-unsubscribe@lucene.apache.org
>>> <ma...@lucene.apache.org>
>>>
>>> >>>>> For additional commands, e-mail:
>>> java-dev-help@lucene.apache.org <mailto:
>>> java-dev-help@lucene.apache.org>
>>>
>>> >>>>>
>>> >>>>>
>>> >>>>>
>>> >>>>
>>> ---------------------------------------------------------------------
>>> >>>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>>> <ma...@lucene.apache.org>
>>>
>>> >>>> For additional commands, e-mail:
>>> java-dev-help@lucene.apache.org <mailto:
>>> java-dev-help@lucene.apache.org>
>>>
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>
>>> ---------------------------------------------------------------------
>>> >>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>>> <ma...@lucene.apache.org>
>>>
>>> >>> For additional commands, e-mail:
>>> java-dev-help@lucene.apache.org <mailto:
>>> java-dev-help@lucene.apache.org>
>>>
>>> >>>
>>> >>>
>>> >>>
>>> >>
>>> ---------------------------------------------------------------------
>>> >> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>>> <ma...@lucene.apache.org>
>>>
>>> >> For additional commands, e-mail: java-dev-help@lucene.apache.org
>>> <ma...@lucene.apache.org>
>>>
>>> >>
>>> >>
>>> >>
>>> >
>>> >
>>> ---------------------------------------------------------------------
>>> > To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>>> <ma...@lucene.apache.org>
>>>
>>> > For additional commands, e-mail: java-dev-help@lucene.apache.org
>>> <ma...@lucene.apache.org>
>>>
>>> >
>>> >
>>>
>>>
>>> --
>>> - Mark
>>>
>>> http://www.lucidimagination.com
>>>
>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>>> <ma...@lucene.apache.org>
>>>
>>> For additional commands, e-mail: java-dev-help@lucene.apache.org
>>> <ma...@lucene.apache.org>
>>>
>>>
>>>
>>
>> --
>> - Mark
>>
>> http://www.lucidimagination.com
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: dev-help@lucene.apache.org
>>
>>
>
--
- Mark
http://www.lucidimagination.com
Re: Nasty NIO behavior makes NIOFSDirectory silently close channel
Posted by John Wang <jo...@gmail.com>.
12 concurrent search threads in a stress environment. (with about 5M doc
index)
-John
On Sun, May 2, 2010 at 3:13 PM, Mark Miller <ma...@gmail.com> wrote:
> Perfectly safe to use SimpleFSDirectory. When you measure the performance,
> are you measuring a lot of concurrent requests? That's where you should see
> the win.
>
> - Mark
>
>
> On 5/1/10 6:38 PM, John Wang wrote:
>
>> We are seeing this issue as well in your production. (using Zoie on top
>> of lucene 2.9.1)
>> After some performance comparisons, we do NOT see performance gain with
>> NIO, rather these nasty ClosedChannelExceptions.
>>
>> I think the performance gains ppl are seeing with 2.9.1 can be due to
>> many different things. From what we seen, they are not related to
>> NIOFSDirectory.
>>
>> Our solution is to avoid calling FSDirectory.open(), instead just call
>> new SimpleFSDirectory(). Is this safe?
>>
>> -John
>>
>> On Fri, Jan 29, 2010 at 12:32 PM, Mark Miller <markrmiller@gmail.com
>> <ma...@gmail.com>> wrote:
>>
>> Perhaps - one of the things they are supposed to be addressing is
>> extendability.
>>
>> nio2 does have FileSystemProvider, which would actually allow you to
>> create a custom channel !
>>
>> I have not dug in enough to know much more than that though.
>>
>> *But*, another really interesting thing is that in Java 7,
>> FileDescriptors are ref counted ! (though users can't inc/dec).
>>
>> But, FileInputStream and OutputStream have a new constructor that takes
>> a FileDescriptor.
>>
>> So possibly, you could just make one that sits around to keep the
>> FileDescriptor valid, and get your channel off
>> FileInputStream/FileOutputStream?
>>
>> And then if it goes down, make a new one using the FileDescriptor which
>> was not actually closed because there was a still a ref to it.
>>
>> Possibly .... ;)
>>
>> Michael McCandless wrote:
>> > Does anyone know if nio2 has improved this...?
>> >
>> > Mike
>> >
>> > On Fri, Jan 29, 2010 at 2:00 PM, Jason Rutherglen
>> > <jason.rutherglen@gmail.com <ma...@gmail.com>>
>>
>> wrote:
>> >
>> >> Defaulting NIOFSDir could account for some of the recent speed
>> >> improvements users have been reporting in Lucene 2.9. So
>> removing it
>> >> as a default could reverse those and people could then report
>> Lucene
>> >> 3.X has slowed...
>> >>
>> >> On Thu, Jan 28, 2010 at 5:24 AM, Michael McCandless
>> >> <lucene@mikemccandless.com <ma...@mikemccandless.com>>
>>
>> wrote:
>> >>
>> >>> Bummer.
>> >>>
>> >>> So the only viable workarounds are 1) don't use
>> Thread.interrupt (nor,
>> >>> things like Future.cancel, which in turn use Thread.interrupt)
>> with
>> >>> NIOFSDir, or 2) we fix NIOFSDir to reopen the channel AND the
>> app must
>> >>> make a deletion policy that keeps a commit alive if any reader is
>> >>> using it. Or, 3) don't use NIOFSDir!
>> >>>
>> >>> Mike
>> >>>
>> >>> On Thu, Jan 28, 2010 at 7:29 AM, Simon Willnauer
>> >>> <simon.willnauer@googlemail.com
>> <ma...@googlemail.com>> wrote:
>> >>>
>> >>>> On Thu, Jan 28, 2010 at 12:43 PM, Michael McCandless
>> >>>> <lucene@mikemccandless.com <ma...@mikemccandless.com>>
>>
>> wrote:
>> >>>>
>> >>>>> On Thu, Jan 28, 2010 at 6:38 AM, Uwe Schindler
>> <uwe@thetaphi.de <ma...@thetaphi.de>> wrote:
>> >>>>>
>> >>>>>
>> >>>>>> So I checked the code of NIOFSIndexInput, my last comment
>> was not really correct:
>> >>>>>> NIOFSIndexInput extends SimpleFSIndexInput and that opens
>> the RAF. In the ctor RAF.getChannel() is called. The RAF keeps open
>> until the file is closed (and also the channel).
>> >>>>>>
>> >>>>>> So it's really simple to fix in my opinion, just call
>> getChannel() again on this exception. Because the RAF should still
>> be open?
>> >>>>>>
>> >>>> Short answer:
>> >>>> public final FileChannel getChannel() {
>> >>>> synchronized (this) {
>> >>>> if (channel == null)
>> >>>> channel = FileChannelImpl.open(fd, true, rw,
>> this);
>> >>>> return channel;
>> >>>> }
>> >>>> }
>> >>>>
>> >>>> this is not gonna work I tried it before. The RandomAccessFile
>> buffers
>> >>>> the channel!!
>> >>>>
>> >>>> simon
>> >>>>
>> >>>>> I think we need a definitive answer on what happens to the
>> RAF when
>> >>>>> the FileChannel was closed by Thread.Interrupt. Simon can
>> you test
>> >>>>> this?
>> >>>>>
>> >>>>> Mike
>> >>>>>
>> >>>>>
>> ---------------------------------------------------------------------
>> >>>>> To unsubscribe, e-mail:
>> java-dev-unsubscribe@lucene.apache.org
>> <ma...@lucene.apache.org>
>>
>> >>>>> For additional commands, e-mail:
>> java-dev-help@lucene.apache.org <mailto:
>> java-dev-help@lucene.apache.org>
>>
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>
>> ---------------------------------------------------------------------
>> >>>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>> <ma...@lucene.apache.org>
>>
>> >>>> For additional commands, e-mail:
>> java-dev-help@lucene.apache.org <mailto:
>> java-dev-help@lucene.apache.org>
>>
>> >>>>
>> >>>>
>> >>>>
>> >>>
>> ---------------------------------------------------------------------
>> >>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>> <ma...@lucene.apache.org>
>>
>> >>> For additional commands, e-mail:
>> java-dev-help@lucene.apache.org <mailto:
>> java-dev-help@lucene.apache.org>
>>
>> >>>
>> >>>
>> >>>
>> >>
>> ---------------------------------------------------------------------
>> >> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>> <ma...@lucene.apache.org>
>>
>> >> For additional commands, e-mail: java-dev-help@lucene.apache.org
>> <ma...@lucene.apache.org>
>>
>> >>
>> >>
>> >>
>> >
>> >
>> ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>> <ma...@lucene.apache.org>
>>
>> > For additional commands, e-mail: java-dev-help@lucene.apache.org
>> <ma...@lucene.apache.org>
>>
>> >
>> >
>>
>>
>> --
>> - Mark
>>
>> http://www.lucidimagination.com
>>
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>> <ma...@lucene.apache.org>
>>
>> For additional commands, e-mail: java-dev-help@lucene.apache.org
>> <ma...@lucene.apache.org>
>>
>>
>>
>
> --
> - Mark
>
> http://www.lucidimagination.com
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
>
Re: Nasty NIO behavior makes NIOFSDirectory silently close channel
Posted by Mark Miller <ma...@gmail.com>.
Perfectly safe to use SimpleFSDirectory. When you measure the
performance, are you measuring a lot of concurrent requests? That's
where you should see the win.
- Mark
On 5/1/10 6:38 PM, John Wang wrote:
> We are seeing this issue as well in your production. (using Zoie on top
> of lucene 2.9.1)
> After some performance comparisons, we do NOT see performance gain with
> NIO, rather these nasty ClosedChannelExceptions.
>
> I think the performance gains ppl are seeing with 2.9.1 can be due to
> many different things. From what we seen, they are not related to
> NIOFSDirectory.
>
> Our solution is to avoid calling FSDirectory.open(), instead just call
> new SimpleFSDirectory(). Is this safe?
>
> -John
>
> On Fri, Jan 29, 2010 at 12:32 PM, Mark Miller <markrmiller@gmail.com
> <ma...@gmail.com>> wrote:
>
> Perhaps - one of the things they are supposed to be addressing is
> extendability.
>
> nio2 does have FileSystemProvider, which would actually allow you to
> create a custom channel !
>
> I have not dug in enough to know much more than that though.
>
> *But*, another really interesting thing is that in Java 7,
> FileDescriptors are ref counted ! (though users can't inc/dec).
>
> But, FileInputStream and OutputStream have a new constructor that takes
> a FileDescriptor.
>
> So possibly, you could just make one that sits around to keep the
> FileDescriptor valid, and get your channel off
> FileInputStream/FileOutputStream?
>
> And then if it goes down, make a new one using the FileDescriptor which
> was not actually closed because there was a still a ref to it.
>
> Possibly .... ;)
>
> Michael McCandless wrote:
> > Does anyone know if nio2 has improved this...?
> >
> > Mike
> >
> > On Fri, Jan 29, 2010 at 2:00 PM, Jason Rutherglen
> > <jason.rutherglen@gmail.com <ma...@gmail.com>>
> wrote:
> >
> >> Defaulting NIOFSDir could account for some of the recent speed
> >> improvements users have been reporting in Lucene 2.9. So
> removing it
> >> as a default could reverse those and people could then report Lucene
> >> 3.X has slowed...
> >>
> >> On Thu, Jan 28, 2010 at 5:24 AM, Michael McCandless
> >> <lucene@mikemccandless.com <ma...@mikemccandless.com>>
> wrote:
> >>
> >>> Bummer.
> >>>
> >>> So the only viable workarounds are 1) don't use
> Thread.interrupt (nor,
> >>> things like Future.cancel, which in turn use Thread.interrupt) with
> >>> NIOFSDir, or 2) we fix NIOFSDir to reopen the channel AND the
> app must
> >>> make a deletion policy that keeps a commit alive if any reader is
> >>> using it. Or, 3) don't use NIOFSDir!
> >>>
> >>> Mike
> >>>
> >>> On Thu, Jan 28, 2010 at 7:29 AM, Simon Willnauer
> >>> <simon.willnauer@googlemail.com
> <ma...@googlemail.com>> wrote:
> >>>
> >>>> On Thu, Jan 28, 2010 at 12:43 PM, Michael McCandless
> >>>> <lucene@mikemccandless.com <ma...@mikemccandless.com>>
> wrote:
> >>>>
> >>>>> On Thu, Jan 28, 2010 at 6:38 AM, Uwe Schindler
> <uwe@thetaphi.de <ma...@thetaphi.de>> wrote:
> >>>>>
> >>>>>
> >>>>>> So I checked the code of NIOFSIndexInput, my last comment
> was not really correct:
> >>>>>> NIOFSIndexInput extends SimpleFSIndexInput and that opens
> the RAF. In the ctor RAF.getChannel() is called. The RAF keeps open
> until the file is closed (and also the channel).
> >>>>>>
> >>>>>> So it's really simple to fix in my opinion, just call
> getChannel() again on this exception. Because the RAF should still
> be open?
> >>>>>>
> >>>> Short answer:
> >>>> public final FileChannel getChannel() {
> >>>> synchronized (this) {
> >>>> if (channel == null)
> >>>> channel = FileChannelImpl.open(fd, true, rw, this);
> >>>> return channel;
> >>>> }
> >>>> }
> >>>>
> >>>> this is not gonna work I tried it before. The RandomAccessFile
> buffers
> >>>> the channel!!
> >>>>
> >>>> simon
> >>>>
> >>>>> I think we need a definitive answer on what happens to the
> RAF when
> >>>>> the FileChannel was closed by Thread.Interrupt. Simon can
> you test
> >>>>> this?
> >>>>>
> >>>>> Mike
> >>>>>
> >>>>>
> ---------------------------------------------------------------------
> >>>>> To unsubscribe, e-mail:
> java-dev-unsubscribe@lucene.apache.org
> <ma...@lucene.apache.org>
> >>>>> For additional commands, e-mail:
> java-dev-help@lucene.apache.org <ma...@lucene.apache.org>
> >>>>>
> >>>>>
> >>>>>
> >>>>
> ---------------------------------------------------------------------
> >>>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> <ma...@lucene.apache.org>
> >>>> For additional commands, e-mail:
> java-dev-help@lucene.apache.org <ma...@lucene.apache.org>
> >>>>
> >>>>
> >>>>
> >>>
> ---------------------------------------------------------------------
> >>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> <ma...@lucene.apache.org>
> >>> For additional commands, e-mail:
> java-dev-help@lucene.apache.org <ma...@lucene.apache.org>
> >>>
> >>>
> >>>
> >>
> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> <ma...@lucene.apache.org>
> >> For additional commands, e-mail: java-dev-help@lucene.apache.org
> <ma...@lucene.apache.org>
> >>
> >>
> >>
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> <ma...@lucene.apache.org>
> > For additional commands, e-mail: java-dev-help@lucene.apache.org
> <ma...@lucene.apache.org>
> >
> >
>
>
> --
> - Mark
>
> http://www.lucidimagination.com
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> <ma...@lucene.apache.org>
> For additional commands, e-mail: java-dev-help@lucene.apache.org
> <ma...@lucene.apache.org>
>
>
--
- Mark
http://www.lucidimagination.com
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: Nasty NIO behavior makes NIOFSDirectory silently close channel
Posted by John Wang <jo...@gmail.com>.
We are seeing this issue as well in your production. (using Zoie on top of
lucene 2.9.1)
After some performance comparisons, we do NOT see performance gain with NIO,
rather these nasty ClosedChannelExceptions.
I think the performance gains ppl are seeing with 2.9.1 can be due to many
different things. From what we seen, they are not related to NIOFSDirectory.
Our solution is to avoid calling FSDirectory.open(), instead just call new
SimpleFSDirectory(). Is this safe?
-John
On Fri, Jan 29, 2010 at 12:32 PM, Mark Miller <ma...@gmail.com> wrote:
> Perhaps - one of the things they are supposed to be addressing is
> extendability.
>
> nio2 does have FileSystemProvider, which would actually allow you to
> create a custom channel !
>
> I have not dug in enough to know much more than that though.
>
> *But*, another really interesting thing is that in Java 7,
> FileDescriptors are ref counted ! (though users can't inc/dec).
>
> But, FileInputStream and OutputStream have a new constructor that takes
> a FileDescriptor.
>
> So possibly, you could just make one that sits around to keep the
> FileDescriptor valid, and get your channel off
> FileInputStream/FileOutputStream?
>
> And then if it goes down, make a new one using the FileDescriptor which
> was not actually closed because there was a still a ref to it.
>
> Possibly .... ;)
>
> Michael McCandless wrote:
> > Does anyone know if nio2 has improved this...?
> >
> > Mike
> >
> > On Fri, Jan 29, 2010 at 2:00 PM, Jason Rutherglen
> > <ja...@gmail.com> wrote:
> >
> >> Defaulting NIOFSDir could account for some of the recent speed
> >> improvements users have been reporting in Lucene 2.9. So removing it
> >> as a default could reverse those and people could then report Lucene
> >> 3.X has slowed...
> >>
> >> On Thu, Jan 28, 2010 at 5:24 AM, Michael McCandless
> >> <lu...@mikemccandless.com> wrote:
> >>
> >>> Bummer.
> >>>
> >>> So the only viable workarounds are 1) don't use Thread.interrupt (nor,
> >>> things like Future.cancel, which in turn use Thread.interrupt) with
> >>> NIOFSDir, or 2) we fix NIOFSDir to reopen the channel AND the app must
> >>> make a deletion policy that keeps a commit alive if any reader is
> >>> using it. Or, 3) don't use NIOFSDir!
> >>>
> >>> Mike
> >>>
> >>> On Thu, Jan 28, 2010 at 7:29 AM, Simon Willnauer
> >>> <si...@googlemail.com> wrote:
> >>>
> >>>> On Thu, Jan 28, 2010 at 12:43 PM, Michael McCandless
> >>>> <lu...@mikemccandless.com> wrote:
> >>>>
> >>>>> On Thu, Jan 28, 2010 at 6:38 AM, Uwe Schindler <uw...@thetaphi.de>
> wrote:
> >>>>>
> >>>>>
> >>>>>> So I checked the code of NIOFSIndexInput, my last comment was not
> really correct:
> >>>>>> NIOFSIndexInput extends SimpleFSIndexInput and that opens the RAF.
> In the ctor RAF.getChannel() is called. The RAF keeps open until the file is
> closed (and also the channel).
> >>>>>>
> >>>>>> So it's really simple to fix in my opinion, just call getChannel()
> again on this exception. Because the RAF should still be open?
> >>>>>>
> >>>> Short answer:
> >>>> public final FileChannel getChannel() {
> >>>> synchronized (this) {
> >>>> if (channel == null)
> >>>> channel = FileChannelImpl.open(fd, true, rw, this);
> >>>> return channel;
> >>>> }
> >>>> }
> >>>>
> >>>> this is not gonna work I tried it before. The RandomAccessFile buffers
> >>>> the channel!!
> >>>>
> >>>> simon
> >>>>
> >>>>> I think we need a definitive answer on what happens to the RAF when
> >>>>> the FileChannel was closed by Thread.Interrupt. Simon can you test
> >>>>> this?
> >>>>>
> >>>>> Mike
> >>>>>
> >>>>> ---------------------------------------------------------------------
> >>>>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> >>>>> For additional commands, e-mail: java-dev-help@lucene.apache.org
> >>>>>
> >>>>>
> >>>>>
> >>>> ---------------------------------------------------------------------
> >>>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> >>>> For additional commands, e-mail: java-dev-help@lucene.apache.org
> >>>>
> >>>>
> >>>>
> >>> ---------------------------------------------------------------------
> >>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> >>> For additional commands, e-mail: java-dev-help@lucene.apache.org
> >>>
> >>>
> >>>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> >> For additional commands, e-mail: java-dev-help@lucene.apache.org
> >>
> >>
> >>
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-dev-help@lucene.apache.org
> >
> >
>
>
> --
> - Mark
>
> http://www.lucidimagination.com
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>
Re: Nasty NIO behavior makes NIOFSDirectory silently close channel
Posted by Yonik Seeley <yo...@lucidimagination.com>.
On Thu, Jan 28, 2010 at 8:24 AM, Michael McCandless
<lu...@mikemccandless.com> wrote:
> Bummer.
>
> So the only viable workarounds are 1) don't use Thread.interrupt (nor,
> things like Future.cancel, which in turn use Thread.interrupt) with
> NIOFSDir, or 2) we fix NIOFSDir to reopen the channel AND the app must
> make a deletion policy that keeps a commit alive if any reader is
> using it. Or, 3) don't use NIOFSDir!
All of these options suck.... (venom directed at Sun... not you Mike :-)
Lucene is used embedded in so many places - it would be OK for Lucene
itself not to use Thread.interrupt() or Future.cancel(), but *any*
other component or application that happens to include Lucene? And I
know that Solr already uses Future.cancel() - which means we need to
go carefully look to see that nothing that could be canceled calls
lucene APIs that could read/write the index.
Other options include hard hacks like bytecode modification to stop
the close (assuming what does the code is even in Java).
In the meantime - perhaps Lucene should revert and not use NIOFSDir by default?
-Yonik
http://www.lucidimagination.com
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Nasty NIO behavior makes NIOFSDirectory silently close channel
Posted by Simon Willnauer <si...@googlemail.com>.
FYI - I created an issue for this
https://issues.apache.org/jira/browse/LUCENE-2239
simon
On Fri, Jan 29, 2010 at 2:31 PM, Grant Ingersoll <gs...@apache.org> wrote:
>
> On Jan 28, 2010, at 4:38 PM, Yonik Seeley wrote:
>
>> On Thu, Jan 28, 2010 at 3:49 PM, Grant Ingersoll <gs...@apache.org> wrote:
>>> Could we get the Channel (and other necessary classes) implementation from Apache Harmony
>>
>> Public JDK methods don't have the low level stuff to implement our
>> own, so rolling our own or using Harmony would require native code...
>> not a good option.
>
> Yeah, I thought so, but figured it might be worth looking.
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Nasty NIO behavior makes NIOFSDirectory silently close channel
Posted by Grant Ingersoll <gs...@apache.org>.
On Jan 28, 2010, at 4:38 PM, Yonik Seeley wrote:
> On Thu, Jan 28, 2010 at 3:49 PM, Grant Ingersoll <gs...@apache.org> wrote:
>> Could we get the Channel (and other necessary classes) implementation from Apache Harmony
>
> Public JDK methods don't have the low level stuff to implement our
> own, so rolling our own or using Harmony would require native code...
> not a good option.
Yeah, I thought so, but figured it might be worth looking.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Nasty NIO behavior makes NIOFSDirectory silently close channel
Posted by Yonik Seeley <yo...@lucidimagination.com>.
On Thu, Jan 28, 2010 at 3:49 PM, Grant Ingersoll <gs...@apache.org> wrote:
> Could we get the Channel (and other necessary classes) implementation from Apache Harmony
Public JDK methods don't have the low level stuff to implement our
own, so rolling our own or using Harmony would require native code...
not a good option.
-Yonik
http://www.lucidimagination.com
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Nasty NIO behavior makes NIOFSDirectory silently close channel
Posted by Grant Ingersoll <gs...@apache.org>.
Could we get the Channel (and other necessary classes) implementation from Apache Harmony and provide our own implementation? Not ideal, but may solve the problem until it can be resolved by other means.
Just a thought, I haven't looked into Harmony's implementation to know whether that is viable or not.
-Grant
On Jan 28, 2010, at 8:24 AM, Michael McCandless wrote:
> Bummer.
>
> So the only viable workarounds are 1) don't use Thread.interrupt (nor,
> things like Future.cancel, which in turn use Thread.interrupt) with
> NIOFSDir, or 2) we fix NIOFSDir to reopen the channel AND the app must
> make a deletion policy that keeps a commit alive if any reader is
> using it. Or, 3) don't use NIOFSDir!
>
> Mike
>
> On Thu, Jan 28, 2010 at 7:29 AM, Simon Willnauer
> <si...@googlemail.com> wrote:
>> On Thu, Jan 28, 2010 at 12:43 PM, Michael McCandless
>> <lu...@mikemccandless.com> wrote:
>>> On Thu, Jan 28, 2010 at 6:38 AM, Uwe Schindler <uw...@thetaphi.de> wrote:
>>>
>>>> So I checked the code of NIOFSIndexInput, my last comment was not really correct:
>>>> NIOFSIndexInput extends SimpleFSIndexInput and that opens the RAF. In the ctor RAF.getChannel() is called. The RAF keeps open until the file is closed (and also the channel).
>>>>
>>>> So it's really simple to fix in my opinion, just call getChannel() again on this exception. Because the RAF should still be open?
>>
>> Short answer:
>> public final FileChannel getChannel() {
>> synchronized (this) {
>> if (channel == null)
>> channel = FileChannelImpl.open(fd, true, rw, this);
>> return channel;
>> }
>> }
>>
>> this is not gonna work I tried it before. The RandomAccessFile buffers
>> the channel!!
>>
>> simon
>>>
>>> I think we need a definitive answer on what happens to the RAF when
>>> the FileChannel was closed by Thread.Interrupt. Simon can you test
>>> this?
>>>
>>> Mike
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-dev-help@lucene.apache.org
>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-dev-help@lucene.apache.org
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Nasty NIO behavior makes NIOFSDirectory silently close channel
Posted by Mark Miller <ma...@gmail.com>.
Perhaps - one of the things they are supposed to be addressing is
extendability.
nio2 does have FileSystemProvider, which would actually allow you to
create a custom channel !
I have not dug in enough to know much more than that though.
*But*, another really interesting thing is that in Java 7,
FileDescriptors are ref counted ! (though users can't inc/dec).
But, FileInputStream and OutputStream have a new constructor that takes
a FileDescriptor.
So possibly, you could just make one that sits around to keep the
FileDescriptor valid, and get your channel off
FileInputStream/FileOutputStream?
And then if it goes down, make a new one using the FileDescriptor which
was not actually closed because there was a still a ref to it.
Possibly .... ;)
Michael McCandless wrote:
> Does anyone know if nio2 has improved this...?
>
> Mike
>
> On Fri, Jan 29, 2010 at 2:00 PM, Jason Rutherglen
> <ja...@gmail.com> wrote:
>
>> Defaulting NIOFSDir could account for some of the recent speed
>> improvements users have been reporting in Lucene 2.9. So removing it
>> as a default could reverse those and people could then report Lucene
>> 3.X has slowed...
>>
>> On Thu, Jan 28, 2010 at 5:24 AM, Michael McCandless
>> <lu...@mikemccandless.com> wrote:
>>
>>> Bummer.
>>>
>>> So the only viable workarounds are 1) don't use Thread.interrupt (nor,
>>> things like Future.cancel, which in turn use Thread.interrupt) with
>>> NIOFSDir, or 2) we fix NIOFSDir to reopen the channel AND the app must
>>> make a deletion policy that keeps a commit alive if any reader is
>>> using it. Or, 3) don't use NIOFSDir!
>>>
>>> Mike
>>>
>>> On Thu, Jan 28, 2010 at 7:29 AM, Simon Willnauer
>>> <si...@googlemail.com> wrote:
>>>
>>>> On Thu, Jan 28, 2010 at 12:43 PM, Michael McCandless
>>>> <lu...@mikemccandless.com> wrote:
>>>>
>>>>> On Thu, Jan 28, 2010 at 6:38 AM, Uwe Schindler <uw...@thetaphi.de> wrote:
>>>>>
>>>>>
>>>>>> So I checked the code of NIOFSIndexInput, my last comment was not really correct:
>>>>>> NIOFSIndexInput extends SimpleFSIndexInput and that opens the RAF. In the ctor RAF.getChannel() is called. The RAF keeps open until the file is closed (and also the channel).
>>>>>>
>>>>>> So it's really simple to fix in my opinion, just call getChannel() again on this exception. Because the RAF should still be open?
>>>>>>
>>>> Short answer:
>>>> public final FileChannel getChannel() {
>>>> synchronized (this) {
>>>> if (channel == null)
>>>> channel = FileChannelImpl.open(fd, true, rw, this);
>>>> return channel;
>>>> }
>>>> }
>>>>
>>>> this is not gonna work I tried it before. The RandomAccessFile buffers
>>>> the channel!!
>>>>
>>>> simon
>>>>
>>>>> I think we need a definitive answer on what happens to the RAF when
>>>>> the FileChannel was closed by Thread.Interrupt. Simon can you test
>>>>> this?
>>>>>
>>>>> Mike
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>>>>> For additional commands, e-mail: java-dev-help@lucene.apache.org
>>>>>
>>>>>
>>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>>>> For additional commands, e-mail: java-dev-help@lucene.apache.org
>>>>
>>>>
>>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-dev-help@lucene.apache.org
>>>
>>>
>>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-dev-help@lucene.apache.org
>>
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>
--
- Mark
http://www.lucidimagination.com
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Nasty NIO behavior makes NIOFSDirectory silently close channel
Posted by Michael McCandless <lu...@mikemccandless.com>.
Does anyone know if nio2 has improved this...?
Mike
On Fri, Jan 29, 2010 at 2:00 PM, Jason Rutherglen
<ja...@gmail.com> wrote:
> Defaulting NIOFSDir could account for some of the recent speed
> improvements users have been reporting in Lucene 2.9. So removing it
> as a default could reverse those and people could then report Lucene
> 3.X has slowed...
>
> On Thu, Jan 28, 2010 at 5:24 AM, Michael McCandless
> <lu...@mikemccandless.com> wrote:
>> Bummer.
>>
>> So the only viable workarounds are 1) don't use Thread.interrupt (nor,
>> things like Future.cancel, which in turn use Thread.interrupt) with
>> NIOFSDir, or 2) we fix NIOFSDir to reopen the channel AND the app must
>> make a deletion policy that keeps a commit alive if any reader is
>> using it. Or, 3) don't use NIOFSDir!
>>
>> Mike
>>
>> On Thu, Jan 28, 2010 at 7:29 AM, Simon Willnauer
>> <si...@googlemail.com> wrote:
>>> On Thu, Jan 28, 2010 at 12:43 PM, Michael McCandless
>>> <lu...@mikemccandless.com> wrote:
>>>> On Thu, Jan 28, 2010 at 6:38 AM, Uwe Schindler <uw...@thetaphi.de> wrote:
>>>>
>>>>> So I checked the code of NIOFSIndexInput, my last comment was not really correct:
>>>>> NIOFSIndexInput extends SimpleFSIndexInput and that opens the RAF. In the ctor RAF.getChannel() is called. The RAF keeps open until the file is closed (and also the channel).
>>>>>
>>>>> So it's really simple to fix in my opinion, just call getChannel() again on this exception. Because the RAF should still be open?
>>>
>>> Short answer:
>>> public final FileChannel getChannel() {
>>> synchronized (this) {
>>> if (channel == null)
>>> channel = FileChannelImpl.open(fd, true, rw, this);
>>> return channel;
>>> }
>>> }
>>>
>>> this is not gonna work I tried it before. The RandomAccessFile buffers
>>> the channel!!
>>>
>>> simon
>>>>
>>>> I think we need a definitive answer on what happens to the RAF when
>>>> the FileChannel was closed by Thread.Interrupt. Simon can you test
>>>> this?
>>>>
>>>> Mike
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>>>> For additional commands, e-mail: java-dev-help@lucene.apache.org
>>>>
>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-dev-help@lucene.apache.org
>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-dev-help@lucene.apache.org
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Nasty NIO behavior makes NIOFSDirectory silently close channel
Posted by Jason Rutherglen <ja...@gmail.com>.
Defaulting NIOFSDir could account for some of the recent speed
improvements users have been reporting in Lucene 2.9. So removing it
as a default could reverse those and people could then report Lucene
3.X has slowed...
On Thu, Jan 28, 2010 at 5:24 AM, Michael McCandless
<lu...@mikemccandless.com> wrote:
> Bummer.
>
> So the only viable workarounds are 1) don't use Thread.interrupt (nor,
> things like Future.cancel, which in turn use Thread.interrupt) with
> NIOFSDir, or 2) we fix NIOFSDir to reopen the channel AND the app must
> make a deletion policy that keeps a commit alive if any reader is
> using it. Or, 3) don't use NIOFSDir!
>
> Mike
>
> On Thu, Jan 28, 2010 at 7:29 AM, Simon Willnauer
> <si...@googlemail.com> wrote:
>> On Thu, Jan 28, 2010 at 12:43 PM, Michael McCandless
>> <lu...@mikemccandless.com> wrote:
>>> On Thu, Jan 28, 2010 at 6:38 AM, Uwe Schindler <uw...@thetaphi.de> wrote:
>>>
>>>> So I checked the code of NIOFSIndexInput, my last comment was not really correct:
>>>> NIOFSIndexInput extends SimpleFSIndexInput and that opens the RAF. In the ctor RAF.getChannel() is called. The RAF keeps open until the file is closed (and also the channel).
>>>>
>>>> So it's really simple to fix in my opinion, just call getChannel() again on this exception. Because the RAF should still be open?
>>
>> Short answer:
>> public final FileChannel getChannel() {
>> synchronized (this) {
>> if (channel == null)
>> channel = FileChannelImpl.open(fd, true, rw, this);
>> return channel;
>> }
>> }
>>
>> this is not gonna work I tried it before. The RandomAccessFile buffers
>> the channel!!
>>
>> simon
>>>
>>> I think we need a definitive answer on what happens to the RAF when
>>> the FileChannel was closed by Thread.Interrupt. Simon can you test
>>> this?
>>>
>>> Mike
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-dev-help@lucene.apache.org
>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-dev-help@lucene.apache.org
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Nasty NIO behavior makes NIOFSDirectory silently close channel
Posted by Michael McCandless <lu...@mikemccandless.com>.
Bummer.
So the only viable workarounds are 1) don't use Thread.interrupt (nor,
things like Future.cancel, which in turn use Thread.interrupt) with
NIOFSDir, or 2) we fix NIOFSDir to reopen the channel AND the app must
make a deletion policy that keeps a commit alive if any reader is
using it. Or, 3) don't use NIOFSDir!
Mike
On Thu, Jan 28, 2010 at 7:29 AM, Simon Willnauer
<si...@googlemail.com> wrote:
> On Thu, Jan 28, 2010 at 12:43 PM, Michael McCandless
> <lu...@mikemccandless.com> wrote:
>> On Thu, Jan 28, 2010 at 6:38 AM, Uwe Schindler <uw...@thetaphi.de> wrote:
>>
>>> So I checked the code of NIOFSIndexInput, my last comment was not really correct:
>>> NIOFSIndexInput extends SimpleFSIndexInput and that opens the RAF. In the ctor RAF.getChannel() is called. The RAF keeps open until the file is closed (and also the channel).
>>>
>>> So it's really simple to fix in my opinion, just call getChannel() again on this exception. Because the RAF should still be open?
>
> Short answer:
> public final FileChannel getChannel() {
> synchronized (this) {
> if (channel == null)
> channel = FileChannelImpl.open(fd, true, rw, this);
> return channel;
> }
> }
>
> this is not gonna work I tried it before. The RandomAccessFile buffers
> the channel!!
>
> simon
>>
>> I think we need a definitive answer on what happens to the RAF when
>> the FileChannel was closed by Thread.Interrupt. Simon can you test
>> this?
>>
>> Mike
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-dev-help@lucene.apache.org
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Nasty NIO behavior makes NIOFSDirectory silently close channel
Posted by Simon Willnauer <si...@googlemail.com>.
On Thu, Jan 28, 2010 at 12:43 PM, Michael McCandless
<lu...@mikemccandless.com> wrote:
> On Thu, Jan 28, 2010 at 6:38 AM, Uwe Schindler <uw...@thetaphi.de> wrote:
>
>> So I checked the code of NIOFSIndexInput, my last comment was not really correct:
>> NIOFSIndexInput extends SimpleFSIndexInput and that opens the RAF. In the ctor RAF.getChannel() is called. The RAF keeps open until the file is closed (and also the channel).
>>
>> So it's really simple to fix in my opinion, just call getChannel() again on this exception. Because the RAF should still be open?
Short answer:
public final FileChannel getChannel() {
synchronized (this) {
if (channel == null)
channel = FileChannelImpl.open(fd, true, rw, this);
return channel;
}
}
this is not gonna work I tried it before. The RandomAccessFile buffers
the channel!!
simon
>
> I think we need a definitive answer on what happens to the RAF when
> the FileChannel was closed by Thread.Interrupt. Simon can you test
> this?
>
> Mike
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Nasty NIO behavior makes NIOFSDirectory silently close channel
Posted by Michael McCandless <lu...@mikemccandless.com>.
On Thu, Jan 28, 2010 at 6:38 AM, Uwe Schindler <uw...@thetaphi.de> wrote:
> So I checked the code of NIOFSIndexInput, my last comment was not really correct:
> NIOFSIndexInput extends SimpleFSIndexInput and that opens the RAF. In the ctor RAF.getChannel() is called. The RAF keeps open until the file is closed (and also the channel).
>
> So it's really simple to fix in my opinion, just call getChannel() again on this exception. Because the RAF should still be open?
I think we need a definitive answer on what happens to the RAF when
the FileChannel was closed by Thread.Interrupt. Simon can you test
this?
Mike
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
RE: Nasty NIO behavior makes NIOFSDirectory silently close channel
Posted by Uwe Schindler <uw...@thetaphi.de>.
Hi again,
So I checked the code of NIOFSIndexInput, my last comment was not really correct:
NIOFSIndexInput extends SimpleFSIndexInput and that opens the RAF. In the ctor RAF.getChannel() is called. The RAF keeps open until the file is closed (and also the channel).
So it's really simple to fix in my opinion, just call getChannel() again on this exception. Because the RAF should still be open?
> The current code uses File.openChannel().
This was wrong - sorry.
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de
> -----Original Message-----
> From: Uwe Schindler [mailto:uwe@thetaphi.de]
> Sent: Thursday, January 28, 2010 12:29 PM
> To: java-dev@lucene.apache.org
> Subject: RE: Nasty NIO behavior makes NIOFSDirectory silently close
> channel
>
> From JavaDocs:
> The state of a file channel is intimately connected to that of the
> object whose getChannel method returned the channel. Changing the
> channel's position, whether explicitly or by reading or writing bytes,
> will change the file position of the originating object, and vice
> versa. Changing the file's length via the file channel will change the
> length seen via the originating object, and vice versa. Changing the
> file's content by writing bytes will change the content seen by the
> originating object, and vice versa.
>
> This can only work if the RandomAccessFile and the corresponding
> channel use the same descriptor. So if you create a RandomAccessFile in
> the ctor and call getChannel() later, just keep the RandomAccessFile
> open and close it at close.
>
> The current code uses File.openChannel().
>
> -----
> Uwe Schindler
> H.-H.-Meier-Allee 63, D-28213 Bremen
> http://www.thetaphi.de
> eMail: uwe@thetaphi.de
>
>
> > -----Original Message-----
> > From: Uwe Schindler [mailto:uwe@thetaphi.de]
> > Sent: Thursday, January 28, 2010 12:23 PM
> > To: java-dev@lucene.apache.org
> > Subject: RE: Nasty NIO behavior makes NIOFSDirectory silently close
> > channel
> >
> > I am not sure, because RandomAccessFile has a getChannel method, so I
> > suspect that they share the descriptor. Maybe check it with lsof?
> >
> > Possibly we could wait until Simon provides a testcase that fails.
> >
> > -----
> > Uwe Schindler
> > H.-H.-Meier-Allee 63, D-28213 Bremen
> > http://www.thetaphi.de
> > eMail: uwe@thetaphi.de
> >
> >
> > > -----Original Message-----
> > > From: Michael McCandless [mailto:lucene@mikemccandless.com]
> > > Sent: Thursday, January 28, 2010 12:08 PM
> > > To: java-dev@lucene.apache.org
> > > Subject: Re: Nasty NIO behavior makes NIOFSDirectory silently close
> > > channel
> > >
> > > On Thu, Jan 28, 2010 at 5:59 AM, Uwe Schindler <uw...@thetaphi.de>
> > wrote:
> > > > But if you keep the underlying RandomAccessFile open?
> > >
> > > We could do that but... won't this consume 2 file descriptors?
> > >
> > > Mike
> > >
> > > -------------------------------------------------------------------
> --
> > > To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> > > For additional commands, e-mail: java-dev-help@lucene.apache.org
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
RE: Nasty NIO behavior makes NIOFSDirectory silently close channel
Posted by Uwe Schindler <uw...@thetaphi.de>.
>From JavaDocs:
The state of a file channel is intimately connected to that of the object whose getChannel method returned the channel. Changing the channel's position, whether explicitly or by reading or writing bytes, will change the file position of the originating object, and vice versa. Changing the file's length via the file channel will change the length seen via the originating object, and vice versa. Changing the file's content by writing bytes will change the content seen by the originating object, and vice versa.
This can only work if the RandomAccessFile and the corresponding channel use the same descriptor. So if you create a RandomAccessFile in the ctor and call getChannel() later, just keep the RandomAccessFile open and close it at close.
The current code uses File.openChannel().
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de
> -----Original Message-----
> From: Uwe Schindler [mailto:uwe@thetaphi.de]
> Sent: Thursday, January 28, 2010 12:23 PM
> To: java-dev@lucene.apache.org
> Subject: RE: Nasty NIO behavior makes NIOFSDirectory silently close
> channel
>
> I am not sure, because RandomAccessFile has a getChannel method, so I
> suspect that they share the descriptor. Maybe check it with lsof?
>
> Possibly we could wait until Simon provides a testcase that fails.
>
> -----
> Uwe Schindler
> H.-H.-Meier-Allee 63, D-28213 Bremen
> http://www.thetaphi.de
> eMail: uwe@thetaphi.de
>
>
> > -----Original Message-----
> > From: Michael McCandless [mailto:lucene@mikemccandless.com]
> > Sent: Thursday, January 28, 2010 12:08 PM
> > To: java-dev@lucene.apache.org
> > Subject: Re: Nasty NIO behavior makes NIOFSDirectory silently close
> > channel
> >
> > On Thu, Jan 28, 2010 at 5:59 AM, Uwe Schindler <uw...@thetaphi.de>
> wrote:
> > > But if you keep the underlying RandomAccessFile open?
> >
> > We could do that but... won't this consume 2 file descriptors?
> >
> > Mike
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
RE: Nasty NIO behavior makes NIOFSDirectory silently close channel
Posted by Uwe Schindler <uw...@thetaphi.de>.
I am not sure, because RandomAccessFile has a getChannel method, so I suspect that they share the descriptor. Maybe check it with lsof?
Possibly we could wait until Simon provides a testcase that fails.
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de
> -----Original Message-----
> From: Michael McCandless [mailto:lucene@mikemccandless.com]
> Sent: Thursday, January 28, 2010 12:08 PM
> To: java-dev@lucene.apache.org
> Subject: Re: Nasty NIO behavior makes NIOFSDirectory silently close
> channel
>
> On Thu, Jan 28, 2010 at 5:59 AM, Uwe Schindler <uw...@thetaphi.de> wrote:
> > But if you keep the underlying RandomAccessFile open?
>
> We could do that but... won't this consume 2 file descriptors?
>
> Mike
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Nasty NIO behavior makes NIOFSDirectory silently close channel
Posted by Simon Willnauer <si...@googlemail.com>.
for sure it will consume 2 file descriptors and it requires some
workaround that checks if we are hitting this exception.
kind of ugly anyway.
On Thu, Jan 28, 2010 at 12:07 PM, Michael McCandless
<lu...@mikemccandless.com> wrote:
> On Thu, Jan 28, 2010 at 5:59 AM, Uwe Schindler <uw...@thetaphi.de> wrote:
>> But if you keep the underlying RandomAccessFile open?
>
> We could do that but... won't this consume 2 file descriptors?
>
> Mike
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Nasty NIO behavior makes NIOFSDirectory silently close channel
Posted by Michael McCandless <lu...@mikemccandless.com>.
On Thu, Jan 28, 2010 at 5:59 AM, Uwe Schindler <uw...@thetaphi.de> wrote:
> But if you keep the underlying RandomAccessFile open?
We could do that but... won't this consume 2 file descriptors?
Mike
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
RE: Nasty NIO behavior makes NIOFSDirectory silently close channel
Posted by Uwe Schindler <uw...@thetaphi.de>.
But if you keep the underlying RandomAccessFile open?
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de
> -----Original Message-----
> From: Michael McCandless [mailto:lucene@mikemccandless.com]
> Sent: Thursday, January 28, 2010 11:47 AM
> To: java-dev@lucene.apache.org
> Cc: simon.willnauer@gmail.com
> Subject: Re: Nasty NIO behavior makes NIOFSDirectory silently close
> channel
>
> On Thu, Jan 28, 2010 at 5:20 AM, Uwe Schindler <uw...@thetaphi.de> wrote:
>
> > Can we fix NIOFSIndexInput to simply reopen the channel when the
> exception occurs?
>
> The problem is that the file may have been deleted in the meantime.
>
> This is quite a nasty behavior of FileChannel.
>
> Mike
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Nasty NIO behavior makes NIOFSDirectory silently close channel
Posted by Michael McCandless <lu...@mikemccandless.com>.
On Thu, Jan 28, 2010 at 5:20 AM, Uwe Schindler <uw...@thetaphi.de> wrote:
> Can we fix NIOFSIndexInput to simply reopen the channel when the exception occurs?
The problem is that the file may have been deleted in the meantime.
This is quite a nasty behavior of FileChannel.
Mike
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
RE: Nasty NIO behavior makes NIOFSDirectory silently close channel
Posted by Uwe Schindler <uw...@thetaphi.de>.
Can we fix NIOFSIndexInput to simply reopen the channel when the exception occurs?
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de
> -----Original Message-----
> From: Simon Willnauer [mailto:simon.willnauer@googlemail.com]
> Sent: Thursday, January 28, 2010 9:52 AM
> To: java-dev
> Subject: Nasty NIO behavior makes NIOFSDirectory silently close channel
>
> Last week I ran into nasty hardly reproducible ChannelClosedExceptions
> thrown by NIOFSDirectory#readInternal() during term position reading.
> I had a hard time to figure out what causes the channel to be closed
> as the IndexReader where I obtained the Indexinput or rather
> TermPositions from was still untouched and open. Yet I was able to
> perform actions on any other files except the proximity file. Even
> more weirdness this only happened after upgrading to 3.0 from 2.3.
> Yet, it turned out that the case of the problem is Java's FileChannel
> behavior if the thread is interrupted while a read happens.
> FileChannel
> (http://java.sun.com/j2se/1.5.0/docs/api/java/nio/channels/FileChannel.
> html)
> throws an ClosedByInterruptException
> (http://java.sun.com/j2se/1.5.0/docs/api/java/nio/channels/ClosedByInte
> rruptException.html)
> if the any thread is interrupted during an IO operation. This causes
> the index input to be invalid and puts the IndexReader into an invalid
> state.
> Yet, I'm really surprised that this has never occured to somebody else
> or at least not on the list. This behavior makes the usage of
> NIOFSDirectory extremely dangerous especially in combination with
> j.u.c.Future and their cancel capabilities. Since, NIO is the new
> default for anything except of windows we should at least document
> this behavior in BIG RED LETTERS or rethink the default. I don't see a
> way to recover from this internal close without really reopen the
> IndexReader again but I want to leave this to the discussion.
>
> I was able to isolate the behavior into a testcase and will open an
> issue soon. The list seemed to be a better place to initially inform
> people.
>
> simon
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org