You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Shai Erera (JIRA)" <ji...@apache.org> on 2009/06/21 04:36:07 UTC
[jira] Created: (LUCENE-1707) Don't use ensureOpen() excessively in
IndexReader and IndexWriter
Don't use ensureOpen() excessively in IndexReader and IndexWriter
-----------------------------------------------------------------
Key: LUCENE-1707
URL: https://issues.apache.org/jira/browse/LUCENE-1707
Project: Lucene - Java
Issue Type: Improvement
Reporter: Shai Erera
A spin off from here: http://www.nabble.com/Excessive-use-of-ensureOpen()-td24127806.html.
We should stop calling this method when it's not necessary for any internal Lucene code. Currently, this code seems to hurt properly written apps, unnecessarily.
Will post a patch soon
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: [jira] Commented: (LUCENE-1707) Don't use ensureOpen()
excessively in IndexReader and IndexWriter
Posted by Michael McCandless <lu...@mikemccandless.com>.
On Mon, Jul 6, 2009 at 11:40 AM, Uwe Schindler<uw...@thetaphi.de> wrote:
> Wonderful, and the tests (TestRussianStems) pass?
Yup!
Mike
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
RE: [jira] Commented: (LUCENE-1707) Don't use ensureOpen() excessively in IndexReader and IndexWriter
Posted by Uwe Schindler <uw...@thetaphi.de>.
Wonderful, and the tests (TestRussianStems) pass?
Thanks,
Uwe
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de
> -----Original Message-----
> From: Michael McCandless [mailto:lucene@mikemccandless.com]
> Sent: Monday, July 06, 2009 5:37 PM
> To: java-dev@lucene.apache.org
> Subject: Re: [jira] Commented: (LUCENE-1707) Don't use ensureOpen()
> excessively in IndexReader and IndexWriter
>
> contrib/analyzers/src/test/org/apache/lucene/analysis/ru/stemsUTF8.txt
> looks right on OpenSolaris (unix EOLs).
>
> Mike
>
> On Mon, Jul 6, 2009 at 9:53 AM, Uwe Schindler<uw...@thetaphi.de> wrote:
> > I fixed the encoding problem by convertig the test files to UTF-8 and
> > changed the Reader charset parameter to UTF-8. All files now have old-
> style
> > native again. Could somebody check if in unix, the files only have LF
> (and
> > in windows the files have CRLF, which is the state how I committed it)?
> >
> > The overall strange/incorrect charset conversion is not touched at all,
> but
> > I strongly agree to remove it (and only keep UnicodeRussian as charset
> > parmeter allowed to the analyzer) or remove the analyzer at all.
> >
> > -----
> > Uwe Schindler
> > H.-H.-Meier-Allee 63, D-28213 Bremen
> > http://www.thetaphi.de
> > eMail: uwe@thetaphi.de
> >
> >> -----Original Message-----
> >> From: Robert Muir [mailto:rcmuir@gmail.com]
> >> Sent: Monday, July 06, 2009 3:26 PM
> >> To: java-dev@lucene.apache.org
> >> Subject: Re: [jira] Commented: (LUCENE-1707) Don't use ensureOpen()
> >> excessively in IndexReader and IndexWriter
> >>
> >> uwe I completely agree.
> >>
> >> to add the icing on the cake the entire analyzer appears to be just a
> >> duplication of the contrib/snowball Russian functionality...!
> >>
> >> On Mon, Jul 6, 2009 at 9:19 AM, Uwe Schindler<uw...@thetaphi.de> wrote:
> >> > The whole russian analyzer is very strange and works against all
> >> > charset/unicode conventions. It defines own "charsets" (the only
> valid
> >> one
> >> > is UNICODE), which are all applied to standard java 16 bit chars. The
> >> test
> >> > shows, how this works: It open a text file in KOI8 using the "ISO-
> 88591-
> >> 1"
> >> > charset (just to not modify the codepoints when converting to 16bit
> java
> >> > chars (in principle it does a deprecated "new String(byte[],0)").
> These
> >> > completely wrong java chars are then handled by an analyzers's
> internal
> >> > charset conversion (working on the 16 bit chars).
> >> >
> >> > The only correct usage of this package is:
> >> > - open file with correct encoding (when instantiating the Reader, so
> >> specify
> >> > KOI8 or windows1251 to the Reader). The string is then correctly UTF-
> 16
> >> > encoded java chars. On this string the "pseudo-charset" UNICODE of
> this
> >> > analyzer can be used.
> >> >
> >> > In my opinion, this invalid usage of java chars should be deprecated,
> >> the
> >> > only correct pseudo-charset should be the one specified by UNICODE
> and
> >> all
> >> > charset conversions should be done using the Reader.
> >> >
> >> > Uwe
> >> >
> >> > -----
> >> > Uwe Schindler
> >> > H.-H.-Meier-Allee 63, D-28213 Bremen
> >> > http://www.thetaphi.de
> >> > eMail: uwe@thetaphi.de
> >> >
> >> >> -----Original Message-----
> >> >> From: Robert Muir [mailto:rcmuir@gmail.com]
> >> >> Sent: Monday, July 06, 2009 3:08 PM
> >> >> To: java-dev@lucene.apache.org
> >> >> Subject: Re: [jira] Commented: (LUCENE-1707) Don't use ensureOpen()
> >> >> excessively in IndexReader and IndexWriter
> >> >>
> >> >> Uwe, I think so too. This way it will not be prone to breakage
> again.
> >> >>
> >> >> On Mon, Jul 6, 2009 at 8:38 AM, Uwe Schindler<uw...@thetaphi.de>
> wrote:
> >> >> > In my opinion, these files should be converted to UTF-8 and
> committed
> >> >> again
> >> >> > (and the Reader in the test recondigured for UTF-8). Then they can
> be
> >> >> native
> >> >> > EOL style again. The problem is that SVN can only handle the EOL
> >> style
> >> >> for
> >> >> > one-byte-per-char and UTF-8 files.
> >> >> >
> >> >> > I give it a try here (and I have a converter).
> >> >> >
> >> >> > -----
> >> >> > Uwe Schindler
> >> >> > H.-H.-Meier-Allee 63, D-28213 Bremen
> >> >> > http://www.thetaphi.de
> >> >> > eMail: uwe@thetaphi.de
> >> >> >
> >> >> >> -----Original Message-----
> >> >> >> From: Robert Muir [mailto:rcmuir@gmail.com]
> >> >> >> Sent: Monday, July 06, 2009 1:11 PM
> >> >> >> To: java-dev@lucene.apache.org
> >> >> >> Subject: Re: [jira] Commented: (LUCENE-1707) Don't use
> ensureOpen()
> >> >> >> excessively in IndexReader and IndexWriter
> >> >> >>
> >> >> >> yeah, its fixed now.
> >> >> >>
> >> >> >> On Mon, Jul 6, 2009 at 7:06 AM, Michael
> >> >> >> McCandless<lu...@mikemccandless.com> wrote:
> >> >> >> > Is this the native vs LF svn:eol-style that Uwe already fixed?
> >> >> >> >
> >> >> >> > Mike
> >> >> >> >
> >> >> >> > On Thu, Jul 2, 2009 at 10:03 AM, Shai Erera<se...@gmail.com>
> >> wrote:
> >> >> >> >> Can somebody try to revert the change and test it on Windows?
> >> >> >> >>
> >> >> >> >> On Thu, Jul 2, 2009 at 4:44 PM, Robert Muir <rc...@gmail.com>
> >> >> wrote:
> >> >> >> >>>
> >> >> >> >>> well then I have no idea why it doesn't fail. Except that
> >> perhaps
> >> >> its
> >> >> >> >>> EOL-related (as Shai said), and that the failure is somehow
> >> >> >> >>> platform-dependent due to newline differences between windows
> >> and
> >> >> unix
> >> >> >> >>> (and the way these are encoded in UTF-16/stored in SVN)?
> >> >> >> >>>
> >> >> >> >>> I don't do really any work with files in UTF-16 so this is
> just
> >> a
> >> >> >> theory.
> >> >> >> >>>
> >> >> >> >>> On Thu, Jul 2, 2009 at 9:40 AM, Mark
> >> Miller<ma...@gmail.com>
> >> >> >> wrote:
> >> >> >> >>> > Hudson runs all the tests and emails java-dev if any of
> them
> >> >> fail.
> >> >> >> >>> >
> >> >> >> >>> > On Thu, Jul 2, 2009 at 9:37 AM, Robert Muir (JIRA)
> >> >> <ji...@apache.org>
> >> >> >> >>> > wrote:
> >> >> >> >>> >>
> >> >> >> >>> >> [
> >> >> >> >>> >>
> >> >> >> >>> >> https://issues.apache.org/jira/browse/LUCENE-
> >> >> >>
> 1707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-
> >> >> >> tabpanel&focusedCommentId=12726479#action_12726479
> >> >> >> >>> >> ]
> >> >> >> >>> >>
> >> >> >> >>> >> Robert Muir commented on LUCENE-1707:
> >> >> >> >>> >> -------------------------------------
> >> >> >> >>> >>
> >> >> >> >>> >> bq. Why doesn't Hudson encounter this problem?
> >> >> >> >>> >>
> >> >> >> >>> >> Forgive my ignorance, does hudson also run tests or just
> >> verify
> >> >> >> build?
> >> >> >> >>> >> These files are only used in tests!
> >> >> >> >>> >>
> >> >> >> >>> >> I agree we should correct it, and perhaps to prevent other
> >> >> problems
> >> >> >> >>> >> these
> >> >> >> >>> >> files should be converted to UTF-8.
> >> >> >> >>> >>
> >> >> >> >>> >> For the record I am still confused about these java-code
> >> >> analyzers
> >> >> >> that
> >> >> >> >>> >> implement snowball algorithms, why do they exist when the
> >> same
> >> >> >> >>> >> functionality
> >> >> >> >>> >> is in contrib/snowball?
> >> >> >> >>> >>
> >> >> >> >>> >>
> >> >> >> >>> >> > Don't use ensureOpen() excessively in IndexReader and
> >> >> IndexWriter
> >> >> >> >>> >> > --------------------------------------------------------
> ---
> >> ---
> >> >> ---
> >> >> >> >>> >> >
> >> >> >> >>> >> > Key: LUCENE-1707
> >> >> >> >>> >> > URL:
> >> >> >> >>> >> > https://issues.apache.org/jira/browse/LUCENE-1707
> >> >> >> >>> >> > Project: Lucene - Java
> >> >> >> >>> >> > Issue Type: Improvement
> >> >> >> >>> >> > Components: Index
> >> >> >> >>> >> > Reporter: Shai Erera
> >> >> >> >>> >> > Fix For: 2.9
> >> >> >> >>> >> >
> >> >> >> >>> >> > Attachments: LUCENE-1707.patch, LUCENE-
> 1707.patch
> >> >> >> >>> >> >
> >> >> >> >>> >> >
> >> >> >> >>> >> > A spin off from here:
> >> >> >> >>> >> > http://www.nabble.com/Excessive-use-of-ensureOpen()-
> >> >> >> td24127806.html.
> >> >> >> >>> >> > We should stop calling this method when it's not
> necessary
> >> for
> >> >> >> any
> >> >> >> >>> >> > internal Lucene code. Currently, this code seems to hurt
> >> >> properly
> >> >> >> >>> >> > written
> >> >> >> >>> >> > apps, unnecessarily.
> >> >> >> >>> >> > Will post a patch soon
> >> >> >> >>> >>
> >> >> >> >>> >> --
> >> >> >> >>> >> This message is automatically generated by JIRA.
> >> >> >> >>> >> -
> >> >> >> >>> >> You can reply to this email to add a comment to the issue
> >> >> online.
> >> >> >> >>> >>
> >> >> >> >>> >>
> >> >> >> >>> >> ----------------------------------------------------------
> ---
> >> ---
> >> >> ---
> >> >> >> --
> >> >> >> >>> >> To unsubscribe, e-mail: java-dev-
> >> unsubscribe@lucene.apache.org
> >> >> >> >>> >> For additional commands, e-mail: java-dev-
> >> help@lucene.apache.org
> >> >> >> >>> >>
> >> >> >> >>> >
> >> >> >> >>> >
> >> >> >> >>> >
> >> >> >> >>> > --
> >> >> >> >>> > --
> >> >> >> >>> > - Mark
> >> >> >> >>> >
> >> >> >> >>> > http://www.lucidimagination.com
> >> >> >> >>> >
> >> >> >> >>> >
> >> >> >> >>>
> >> >> >> >>>
> >> >> >> >>>
> >> >> >> >>> --
> >> >> >> >>> Robert Muir
> >> >> >> >>> rcmuir@gmail.com
> >> >> >> >>>
> >> >> >> >>> -------------------------------------------------------------
> ---
> >> ---
> >> >> --
> >> >> >> >>> To unsubscribe, e-mail: java-dev-
> unsubscribe@lucene.apache.org
> >> >> >> >>> For additional commands, e-mail: java-dev-
> help@lucene.apache.org
> >> >> >> >>>
> >> >> >> >>
> >> >> >> >>
> >> >> >> >
> >> >> >> > ---------------------------------------------------------------
> ---
> >> ---
> >> >> >> > To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> >> >> >> > For additional commands, e-mail: java-dev-
> help@lucene.apache.org
> >> >> >> >
> >> >> >> >
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >> --
> >> >> >> Robert Muir
> >> >> >> rcmuir@gmail.com
> >> >> >>
> >> >> >> -----------------------------------------------------------------
> ---
> >> -
> >> >> >> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> >> >> >> For additional commands, e-mail: java-dev-help@lucene.apache.org
> >> >> >
> >> >> >
> >> >> >
> >> >> > ------------------------------------------------------------------
> ---
> >> >> > To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> >> >> > For additional commands, e-mail: java-dev-help@lucene.apache.org
> >> >> >
> >> >> >
> >> >>
> >> >>
> >> >>
> >> >> --
> >> >> Robert Muir
> >> >> rcmuir@gmail.com
> >> >>
> >> >> --------------------------------------------------------------------
> -
> >> >> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> >> >> For additional commands, e-mail: java-dev-help@lucene.apache.org
> >> >
> >> >
> >> >
> >> > ---------------------------------------------------------------------
> >> > To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> >> > For additional commands, e-mail: java-dev-help@lucene.apache.org
> >> >
> >> >
> >>
> >>
> >>
> >> --
> >> Robert Muir
> >> rcmuir@gmail.com
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> >> For additional commands, e-mail: java-dev-help@lucene.apache.org
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-dev-help@lucene.apache.org
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: [jira] Commented: (LUCENE-1707) Don't use ensureOpen()
excessively in IndexReader and IndexWriter
Posted by Michael McCandless <lu...@mikemccandless.com>.
contrib/analyzers/src/test/org/apache/lucene/analysis/ru/stemsUTF8.txt
looks right on OpenSolaris (unix EOLs).
Mike
On Mon, Jul 6, 2009 at 9:53 AM, Uwe Schindler<uw...@thetaphi.de> wrote:
> I fixed the encoding problem by convertig the test files to UTF-8 and
> changed the Reader charset parameter to UTF-8. All files now have old-style
> native again. Could somebody check if in unix, the files only have LF (and
> in windows the files have CRLF, which is the state how I committed it)?
>
> The overall strange/incorrect charset conversion is not touched at all, but
> I strongly agree to remove it (and only keep UnicodeRussian as charset
> parmeter allowed to the analyzer) or remove the analyzer at all.
>
> -----
> Uwe Schindler
> H.-H.-Meier-Allee 63, D-28213 Bremen
> http://www.thetaphi.de
> eMail: uwe@thetaphi.de
>
>> -----Original Message-----
>> From: Robert Muir [mailto:rcmuir@gmail.com]
>> Sent: Monday, July 06, 2009 3:26 PM
>> To: java-dev@lucene.apache.org
>> Subject: Re: [jira] Commented: (LUCENE-1707) Don't use ensureOpen()
>> excessively in IndexReader and IndexWriter
>>
>> uwe I completely agree.
>>
>> to add the icing on the cake the entire analyzer appears to be just a
>> duplication of the contrib/snowball Russian functionality...!
>>
>> On Mon, Jul 6, 2009 at 9:19 AM, Uwe Schindler<uw...@thetaphi.de> wrote:
>> > The whole russian analyzer is very strange and works against all
>> > charset/unicode conventions. It defines own "charsets" (the only valid
>> one
>> > is UNICODE), which are all applied to standard java 16 bit chars. The
>> test
>> > shows, how this works: It open a text file in KOI8 using the "ISO-88591-
>> 1"
>> > charset (just to not modify the codepoints when converting to 16bit java
>> > chars (in principle it does a deprecated "new String(byte[],0)"). These
>> > completely wrong java chars are then handled by an analyzers's internal
>> > charset conversion (working on the 16 bit chars).
>> >
>> > The only correct usage of this package is:
>> > - open file with correct encoding (when instantiating the Reader, so
>> specify
>> > KOI8 or windows1251 to the Reader). The string is then correctly UTF-16
>> > encoded java chars. On this string the "pseudo-charset" UNICODE of this
>> > analyzer can be used.
>> >
>> > In my opinion, this invalid usage of java chars should be deprecated,
>> the
>> > only correct pseudo-charset should be the one specified by UNICODE and
>> all
>> > charset conversions should be done using the Reader.
>> >
>> > Uwe
>> >
>> > -----
>> > Uwe Schindler
>> > H.-H.-Meier-Allee 63, D-28213 Bremen
>> > http://www.thetaphi.de
>> > eMail: uwe@thetaphi.de
>> >
>> >> -----Original Message-----
>> >> From: Robert Muir [mailto:rcmuir@gmail.com]
>> >> Sent: Monday, July 06, 2009 3:08 PM
>> >> To: java-dev@lucene.apache.org
>> >> Subject: Re: [jira] Commented: (LUCENE-1707) Don't use ensureOpen()
>> >> excessively in IndexReader and IndexWriter
>> >>
>> >> Uwe, I think so too. This way it will not be prone to breakage again.
>> >>
>> >> On Mon, Jul 6, 2009 at 8:38 AM, Uwe Schindler<uw...@thetaphi.de> wrote:
>> >> > In my opinion, these files should be converted to UTF-8 and committed
>> >> again
>> >> > (and the Reader in the test recondigured for UTF-8). Then they can be
>> >> native
>> >> > EOL style again. The problem is that SVN can only handle the EOL
>> style
>> >> for
>> >> > one-byte-per-char and UTF-8 files.
>> >> >
>> >> > I give it a try here (and I have a converter).
>> >> >
>> >> > -----
>> >> > Uwe Schindler
>> >> > H.-H.-Meier-Allee 63, D-28213 Bremen
>> >> > http://www.thetaphi.de
>> >> > eMail: uwe@thetaphi.de
>> >> >
>> >> >> -----Original Message-----
>> >> >> From: Robert Muir [mailto:rcmuir@gmail.com]
>> >> >> Sent: Monday, July 06, 2009 1:11 PM
>> >> >> To: java-dev@lucene.apache.org
>> >> >> Subject: Re: [jira] Commented: (LUCENE-1707) Don't use ensureOpen()
>> >> >> excessively in IndexReader and IndexWriter
>> >> >>
>> >> >> yeah, its fixed now.
>> >> >>
>> >> >> On Mon, Jul 6, 2009 at 7:06 AM, Michael
>> >> >> McCandless<lu...@mikemccandless.com> wrote:
>> >> >> > Is this the native vs LF svn:eol-style that Uwe already fixed?
>> >> >> >
>> >> >> > Mike
>> >> >> >
>> >> >> > On Thu, Jul 2, 2009 at 10:03 AM, Shai Erera<se...@gmail.com>
>> wrote:
>> >> >> >> Can somebody try to revert the change and test it on Windows?
>> >> >> >>
>> >> >> >> On Thu, Jul 2, 2009 at 4:44 PM, Robert Muir <rc...@gmail.com>
>> >> wrote:
>> >> >> >>>
>> >> >> >>> well then I have no idea why it doesn't fail. Except that
>> perhaps
>> >> its
>> >> >> >>> EOL-related (as Shai said), and that the failure is somehow
>> >> >> >>> platform-dependent due to newline differences between windows
>> and
>> >> unix
>> >> >> >>> (and the way these are encoded in UTF-16/stored in SVN)?
>> >> >> >>>
>> >> >> >>> I don't do really any work with files in UTF-16 so this is just
>> a
>> >> >> theory.
>> >> >> >>>
>> >> >> >>> On Thu, Jul 2, 2009 at 9:40 AM, Mark
>> Miller<ma...@gmail.com>
>> >> >> wrote:
>> >> >> >>> > Hudson runs all the tests and emails java-dev if any of them
>> >> fail.
>> >> >> >>> >
>> >> >> >>> > On Thu, Jul 2, 2009 at 9:37 AM, Robert Muir (JIRA)
>> >> <ji...@apache.org>
>> >> >> >>> > wrote:
>> >> >> >>> >>
>> >> >> >>> >> [
>> >> >> >>> >>
>> >> >> >>> >> https://issues.apache.org/jira/browse/LUCENE-
>> >> >> 1707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-
>> >> >> tabpanel&focusedCommentId=12726479#action_12726479
>> >> >> >>> >> ]
>> >> >> >>> >>
>> >> >> >>> >> Robert Muir commented on LUCENE-1707:
>> >> >> >>> >> -------------------------------------
>> >> >> >>> >>
>> >> >> >>> >> bq. Why doesn't Hudson encounter this problem?
>> >> >> >>> >>
>> >> >> >>> >> Forgive my ignorance, does hudson also run tests or just
>> verify
>> >> >> build?
>> >> >> >>> >> These files are only used in tests!
>> >> >> >>> >>
>> >> >> >>> >> I agree we should correct it, and perhaps to prevent other
>> >> problems
>> >> >> >>> >> these
>> >> >> >>> >> files should be converted to UTF-8.
>> >> >> >>> >>
>> >> >> >>> >> For the record I am still confused about these java-code
>> >> analyzers
>> >> >> that
>> >> >> >>> >> implement snowball algorithms, why do they exist when the
>> same
>> >> >> >>> >> functionality
>> >> >> >>> >> is in contrib/snowball?
>> >> >> >>> >>
>> >> >> >>> >>
>> >> >> >>> >> > Don't use ensureOpen() excessively in IndexReader and
>> >> IndexWriter
>> >> >> >>> >> > -----------------------------------------------------------
>> ---
>> >> ---
>> >> >> >>> >> >
>> >> >> >>> >> > Key: LUCENE-1707
>> >> >> >>> >> > URL:
>> >> >> >>> >> > https://issues.apache.org/jira/browse/LUCENE-1707
>> >> >> >>> >> > Project: Lucene - Java
>> >> >> >>> >> > Issue Type: Improvement
>> >> >> >>> >> > Components: Index
>> >> >> >>> >> > Reporter: Shai Erera
>> >> >> >>> >> > Fix For: 2.9
>> >> >> >>> >> >
>> >> >> >>> >> > Attachments: LUCENE-1707.patch, LUCENE-1707.patch
>> >> >> >>> >> >
>> >> >> >>> >> >
>> >> >> >>> >> > A spin off from here:
>> >> >> >>> >> > http://www.nabble.com/Excessive-use-of-ensureOpen()-
>> >> >> td24127806.html.
>> >> >> >>> >> > We should stop calling this method when it's not necessary
>> for
>> >> >> any
>> >> >> >>> >> > internal Lucene code. Currently, this code seems to hurt
>> >> properly
>> >> >> >>> >> > written
>> >> >> >>> >> > apps, unnecessarily.
>> >> >> >>> >> > Will post a patch soon
>> >> >> >>> >>
>> >> >> >>> >> --
>> >> >> >>> >> This message is automatically generated by JIRA.
>> >> >> >>> >> -
>> >> >> >>> >> You can reply to this email to add a comment to the issue
>> >> online.
>> >> >> >>> >>
>> >> >> >>> >>
>> >> >> >>> >> -------------------------------------------------------------
>> ---
>> >> ---
>> >> >> --
>> >> >> >>> >> To unsubscribe, e-mail: java-dev-
>> unsubscribe@lucene.apache.org
>> >> >> >>> >> For additional commands, e-mail: java-dev-
>> help@lucene.apache.org
>> >> >> >>> >>
>> >> >> >>> >
>> >> >> >>> >
>> >> >> >>> >
>> >> >> >>> > --
>> >> >> >>> > --
>> >> >> >>> > - Mark
>> >> >> >>> >
>> >> >> >>> > http://www.lucidimagination.com
>> >> >> >>> >
>> >> >> >>> >
>> >> >> >>>
>> >> >> >>>
>> >> >> >>>
>> >> >> >>> --
>> >> >> >>> Robert Muir
>> >> >> >>> rcmuir@gmail.com
>> >> >> >>>
>> >> >> >>> ----------------------------------------------------------------
>> ---
>> >> --
>> >> >> >>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>> >> >> >>> For additional commands, e-mail: java-dev-help@lucene.apache.org
>> >> >> >>>
>> >> >> >>
>> >> >> >>
>> >> >> >
>> >> >> > ------------------------------------------------------------------
>> ---
>> >> >> > To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>> >> >> > For additional commands, e-mail: java-dev-help@lucene.apache.org
>> >> >> >
>> >> >> >
>> >> >>
>> >> >>
>> >> >>
>> >> >> --
>> >> >> Robert Muir
>> >> >> rcmuir@gmail.com
>> >> >>
>> >> >> --------------------------------------------------------------------
>> -
>> >> >> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>> >> >> For additional commands, e-mail: java-dev-help@lucene.apache.org
>> >> >
>> >> >
>> >> >
>> >> > ---------------------------------------------------------------------
>> >> > To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>> >> > For additional commands, e-mail: java-dev-help@lucene.apache.org
>> >> >
>> >> >
>> >>
>> >>
>> >>
>> >> --
>> >> Robert Muir
>> >> rcmuir@gmail.com
>> >>
>> >> ---------------------------------------------------------------------
>> >> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>> >> For additional commands, e-mail: java-dev-help@lucene.apache.org
>> >
>> >
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>> > For additional commands, e-mail: java-dev-help@lucene.apache.org
>> >
>> >
>>
>>
>>
>> --
>> Robert Muir
>> rcmuir@gmail.com
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
RE: [jira] Commented: (LUCENE-1707) Don't use ensureOpen() excessively in IndexReader and IndexWriter
Posted by Uwe Schindler <uw...@thetaphi.de>.
I fixed the encoding problem by convertig the test files to UTF-8 and
changed the Reader charset parameter to UTF-8. All files now have old-style
native again. Could somebody check if in unix, the files only have LF (and
in windows the files have CRLF, which is the state how I committed it)?
The overall strange/incorrect charset conversion is not touched at all, but
I strongly agree to remove it (and only keep UnicodeRussian as charset
parmeter allowed to the analyzer) or remove the analyzer at all.
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de
> -----Original Message-----
> From: Robert Muir [mailto:rcmuir@gmail.com]
> Sent: Monday, July 06, 2009 3:26 PM
> To: java-dev@lucene.apache.org
> Subject: Re: [jira] Commented: (LUCENE-1707) Don't use ensureOpen()
> excessively in IndexReader and IndexWriter
>
> uwe I completely agree.
>
> to add the icing on the cake the entire analyzer appears to be just a
> duplication of the contrib/snowball Russian functionality...!
>
> On Mon, Jul 6, 2009 at 9:19 AM, Uwe Schindler<uw...@thetaphi.de> wrote:
> > The whole russian analyzer is very strange and works against all
> > charset/unicode conventions. It defines own "charsets" (the only valid
> one
> > is UNICODE), which are all applied to standard java 16 bit chars. The
> test
> > shows, how this works: It open a text file in KOI8 using the "ISO-88591-
> 1"
> > charset (just to not modify the codepoints when converting to 16bit java
> > chars (in principle it does a deprecated "new String(byte[],0)"). These
> > completely wrong java chars are then handled by an analyzers's internal
> > charset conversion (working on the 16 bit chars).
> >
> > The only correct usage of this package is:
> > - open file with correct encoding (when instantiating the Reader, so
> specify
> > KOI8 or windows1251 to the Reader). The string is then correctly UTF-16
> > encoded java chars. On this string the "pseudo-charset" UNICODE of this
> > analyzer can be used.
> >
> > In my opinion, this invalid usage of java chars should be deprecated,
> the
> > only correct pseudo-charset should be the one specified by UNICODE and
> all
> > charset conversions should be done using the Reader.
> >
> > Uwe
> >
> > -----
> > Uwe Schindler
> > H.-H.-Meier-Allee 63, D-28213 Bremen
> > http://www.thetaphi.de
> > eMail: uwe@thetaphi.de
> >
> >> -----Original Message-----
> >> From: Robert Muir [mailto:rcmuir@gmail.com]
> >> Sent: Monday, July 06, 2009 3:08 PM
> >> To: java-dev@lucene.apache.org
> >> Subject: Re: [jira] Commented: (LUCENE-1707) Don't use ensureOpen()
> >> excessively in IndexReader and IndexWriter
> >>
> >> Uwe, I think so too. This way it will not be prone to breakage again.
> >>
> >> On Mon, Jul 6, 2009 at 8:38 AM, Uwe Schindler<uw...@thetaphi.de> wrote:
> >> > In my opinion, these files should be converted to UTF-8 and committed
> >> again
> >> > (and the Reader in the test recondigured for UTF-8). Then they can be
> >> native
> >> > EOL style again. The problem is that SVN can only handle the EOL
> style
> >> for
> >> > one-byte-per-char and UTF-8 files.
> >> >
> >> > I give it a try here (and I have a converter).
> >> >
> >> > -----
> >> > Uwe Schindler
> >> > H.-H.-Meier-Allee 63, D-28213 Bremen
> >> > http://www.thetaphi.de
> >> > eMail: uwe@thetaphi.de
> >> >
> >> >> -----Original Message-----
> >> >> From: Robert Muir [mailto:rcmuir@gmail.com]
> >> >> Sent: Monday, July 06, 2009 1:11 PM
> >> >> To: java-dev@lucene.apache.org
> >> >> Subject: Re: [jira] Commented: (LUCENE-1707) Don't use ensureOpen()
> >> >> excessively in IndexReader and IndexWriter
> >> >>
> >> >> yeah, its fixed now.
> >> >>
> >> >> On Mon, Jul 6, 2009 at 7:06 AM, Michael
> >> >> McCandless<lu...@mikemccandless.com> wrote:
> >> >> > Is this the native vs LF svn:eol-style that Uwe already fixed?
> >> >> >
> >> >> > Mike
> >> >> >
> >> >> > On Thu, Jul 2, 2009 at 10:03 AM, Shai Erera<se...@gmail.com>
> wrote:
> >> >> >> Can somebody try to revert the change and test it on Windows?
> >> >> >>
> >> >> >> On Thu, Jul 2, 2009 at 4:44 PM, Robert Muir <rc...@gmail.com>
> >> wrote:
> >> >> >>>
> >> >> >>> well then I have no idea why it doesn't fail. Except that
> perhaps
> >> its
> >> >> >>> EOL-related (as Shai said), and that the failure is somehow
> >> >> >>> platform-dependent due to newline differences between windows
> and
> >> unix
> >> >> >>> (and the way these are encoded in UTF-16/stored in SVN)?
> >> >> >>>
> >> >> >>> I don't do really any work with files in UTF-16 so this is just
> a
> >> >> theory.
> >> >> >>>
> >> >> >>> On Thu, Jul 2, 2009 at 9:40 AM, Mark
> Miller<ma...@gmail.com>
> >> >> wrote:
> >> >> >>> > Hudson runs all the tests and emails java-dev if any of them
> >> fail.
> >> >> >>> >
> >> >> >>> > On Thu, Jul 2, 2009 at 9:37 AM, Robert Muir (JIRA)
> >> <ji...@apache.org>
> >> >> >>> > wrote:
> >> >> >>> >>
> >> >> >>> >> [
> >> >> >>> >>
> >> >> >>> >> https://issues.apache.org/jira/browse/LUCENE-
> >> >> 1707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-
> >> >> tabpanel&focusedCommentId=12726479#action_12726479
> >> >> >>> >> ]
> >> >> >>> >>
> >> >> >>> >> Robert Muir commented on LUCENE-1707:
> >> >> >>> >> -------------------------------------
> >> >> >>> >>
> >> >> >>> >> bq. Why doesn't Hudson encounter this problem?
> >> >> >>> >>
> >> >> >>> >> Forgive my ignorance, does hudson also run tests or just
> verify
> >> >> build?
> >> >> >>> >> These files are only used in tests!
> >> >> >>> >>
> >> >> >>> >> I agree we should correct it, and perhaps to prevent other
> >> problems
> >> >> >>> >> these
> >> >> >>> >> files should be converted to UTF-8.
> >> >> >>> >>
> >> >> >>> >> For the record I am still confused about these java-code
> >> analyzers
> >> >> that
> >> >> >>> >> implement snowball algorithms, why do they exist when the
> same
> >> >> >>> >> functionality
> >> >> >>> >> is in contrib/snowball?
> >> >> >>> >>
> >> >> >>> >>
> >> >> >>> >> > Don't use ensureOpen() excessively in IndexReader and
> >> IndexWriter
> >> >> >>> >> > -----------------------------------------------------------
> ---
> >> ---
> >> >> >>> >> >
> >> >> >>> >> > Key: LUCENE-1707
> >> >> >>> >> > URL:
> >> >> >>> >> > https://issues.apache.org/jira/browse/LUCENE-1707
> >> >> >>> >> > Project: Lucene - Java
> >> >> >>> >> > Issue Type: Improvement
> >> >> >>> >> > Components: Index
> >> >> >>> >> > Reporter: Shai Erera
> >> >> >>> >> > Fix For: 2.9
> >> >> >>> >> >
> >> >> >>> >> > Attachments: LUCENE-1707.patch, LUCENE-1707.patch
> >> >> >>> >> >
> >> >> >>> >> >
> >> >> >>> >> > A spin off from here:
> >> >> >>> >> > http://www.nabble.com/Excessive-use-of-ensureOpen()-
> >> >> td24127806.html.
> >> >> >>> >> > We should stop calling this method when it's not necessary
> for
> >> >> any
> >> >> >>> >> > internal Lucene code. Currently, this code seems to hurt
> >> properly
> >> >> >>> >> > written
> >> >> >>> >> > apps, unnecessarily.
> >> >> >>> >> > Will post a patch soon
> >> >> >>> >>
> >> >> >>> >> --
> >> >> >>> >> This message is automatically generated by JIRA.
> >> >> >>> >> -
> >> >> >>> >> You can reply to this email to add a comment to the issue
> >> online.
> >> >> >>> >>
> >> >> >>> >>
> >> >> >>> >> -------------------------------------------------------------
> ---
> >> ---
> >> >> --
> >> >> >>> >> To unsubscribe, e-mail: java-dev-
> unsubscribe@lucene.apache.org
> >> >> >>> >> For additional commands, e-mail: java-dev-
> help@lucene.apache.org
> >> >> >>> >>
> >> >> >>> >
> >> >> >>> >
> >> >> >>> >
> >> >> >>> > --
> >> >> >>> > --
> >> >> >>> > - Mark
> >> >> >>> >
> >> >> >>> > http://www.lucidimagination.com
> >> >> >>> >
> >> >> >>> >
> >> >> >>>
> >> >> >>>
> >> >> >>>
> >> >> >>> --
> >> >> >>> Robert Muir
> >> >> >>> rcmuir@gmail.com
> >> >> >>>
> >> >> >>> ----------------------------------------------------------------
> ---
> >> --
> >> >> >>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> >> >> >>> For additional commands, e-mail: java-dev-help@lucene.apache.org
> >> >> >>>
> >> >> >>
> >> >> >>
> >> >> >
> >> >> > ------------------------------------------------------------------
> ---
> >> >> > To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> >> >> > For additional commands, e-mail: java-dev-help@lucene.apache.org
> >> >> >
> >> >> >
> >> >>
> >> >>
> >> >>
> >> >> --
> >> >> Robert Muir
> >> >> rcmuir@gmail.com
> >> >>
> >> >> --------------------------------------------------------------------
> -
> >> >> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> >> >> For additional commands, e-mail: java-dev-help@lucene.apache.org
> >> >
> >> >
> >> >
> >> > ---------------------------------------------------------------------
> >> > To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> >> > For additional commands, e-mail: java-dev-help@lucene.apache.org
> >> >
> >> >
> >>
> >>
> >>
> >> --
> >> Robert Muir
> >> rcmuir@gmail.com
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> >> For additional commands, e-mail: java-dev-help@lucene.apache.org
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-dev-help@lucene.apache.org
> >
> >
>
>
>
> --
> Robert Muir
> rcmuir@gmail.com
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: [jira] Commented: (LUCENE-1707) Don't use ensureOpen()
excessively in IndexReader and IndexWriter
Posted by Robert Muir <rc...@gmail.com>.
uwe I completely agree.
to add the icing on the cake the entire analyzer appears to be just a
duplication of the contrib/snowball Russian functionality...!
On Mon, Jul 6, 2009 at 9:19 AM, Uwe Schindler<uw...@thetaphi.de> wrote:
> The whole russian analyzer is very strange and works against all
> charset/unicode conventions. It defines own "charsets" (the only valid one
> is UNICODE), which are all applied to standard java 16 bit chars. The test
> shows, how this works: It open a text file in KOI8 using the "ISO-88591-1"
> charset (just to not modify the codepoints when converting to 16bit java
> chars (in principle it does a deprecated "new String(byte[],0)"). These
> completely wrong java chars are then handled by an analyzers's internal
> charset conversion (working on the 16 bit chars).
>
> The only correct usage of this package is:
> - open file with correct encoding (when instantiating the Reader, so specify
> KOI8 or windows1251 to the Reader). The string is then correctly UTF-16
> encoded java chars. On this string the "pseudo-charset" UNICODE of this
> analyzer can be used.
>
> In my opinion, this invalid usage of java chars should be deprecated, the
> only correct pseudo-charset should be the one specified by UNICODE and all
> charset conversions should be done using the Reader.
>
> Uwe
>
> -----
> Uwe Schindler
> H.-H.-Meier-Allee 63, D-28213 Bremen
> http://www.thetaphi.de
> eMail: uwe@thetaphi.de
>
>> -----Original Message-----
>> From: Robert Muir [mailto:rcmuir@gmail.com]
>> Sent: Monday, July 06, 2009 3:08 PM
>> To: java-dev@lucene.apache.org
>> Subject: Re: [jira] Commented: (LUCENE-1707) Don't use ensureOpen()
>> excessively in IndexReader and IndexWriter
>>
>> Uwe, I think so too. This way it will not be prone to breakage again.
>>
>> On Mon, Jul 6, 2009 at 8:38 AM, Uwe Schindler<uw...@thetaphi.de> wrote:
>> > In my opinion, these files should be converted to UTF-8 and committed
>> again
>> > (and the Reader in the test recondigured for UTF-8). Then they can be
>> native
>> > EOL style again. The problem is that SVN can only handle the EOL style
>> for
>> > one-byte-per-char and UTF-8 files.
>> >
>> > I give it a try here (and I have a converter).
>> >
>> > -----
>> > Uwe Schindler
>> > H.-H.-Meier-Allee 63, D-28213 Bremen
>> > http://www.thetaphi.de
>> > eMail: uwe@thetaphi.de
>> >
>> >> -----Original Message-----
>> >> From: Robert Muir [mailto:rcmuir@gmail.com]
>> >> Sent: Monday, July 06, 2009 1:11 PM
>> >> To: java-dev@lucene.apache.org
>> >> Subject: Re: [jira] Commented: (LUCENE-1707) Don't use ensureOpen()
>> >> excessively in IndexReader and IndexWriter
>> >>
>> >> yeah, its fixed now.
>> >>
>> >> On Mon, Jul 6, 2009 at 7:06 AM, Michael
>> >> McCandless<lu...@mikemccandless.com> wrote:
>> >> > Is this the native vs LF svn:eol-style that Uwe already fixed?
>> >> >
>> >> > Mike
>> >> >
>> >> > On Thu, Jul 2, 2009 at 10:03 AM, Shai Erera<se...@gmail.com> wrote:
>> >> >> Can somebody try to revert the change and test it on Windows?
>> >> >>
>> >> >> On Thu, Jul 2, 2009 at 4:44 PM, Robert Muir <rc...@gmail.com>
>> wrote:
>> >> >>>
>> >> >>> well then I have no idea why it doesn't fail. Except that perhaps
>> its
>> >> >>> EOL-related (as Shai said), and that the failure is somehow
>> >> >>> platform-dependent due to newline differences between windows and
>> unix
>> >> >>> (and the way these are encoded in UTF-16/stored in SVN)?
>> >> >>>
>> >> >>> I don't do really any work with files in UTF-16 so this is just a
>> >> theory.
>> >> >>>
>> >> >>> On Thu, Jul 2, 2009 at 9:40 AM, Mark Miller<ma...@gmail.com>
>> >> wrote:
>> >> >>> > Hudson runs all the tests and emails java-dev if any of them
>> fail.
>> >> >>> >
>> >> >>> > On Thu, Jul 2, 2009 at 9:37 AM, Robert Muir (JIRA)
>> <ji...@apache.org>
>> >> >>> > wrote:
>> >> >>> >>
>> >> >>> >> [
>> >> >>> >>
>> >> >>> >> https://issues.apache.org/jira/browse/LUCENE-
>> >> 1707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-
>> >> tabpanel&focusedCommentId=12726479#action_12726479
>> >> >>> >> ]
>> >> >>> >>
>> >> >>> >> Robert Muir commented on LUCENE-1707:
>> >> >>> >> -------------------------------------
>> >> >>> >>
>> >> >>> >> bq. Why doesn't Hudson encounter this problem?
>> >> >>> >>
>> >> >>> >> Forgive my ignorance, does hudson also run tests or just verify
>> >> build?
>> >> >>> >> These files are only used in tests!
>> >> >>> >>
>> >> >>> >> I agree we should correct it, and perhaps to prevent other
>> problems
>> >> >>> >> these
>> >> >>> >> files should be converted to UTF-8.
>> >> >>> >>
>> >> >>> >> For the record I am still confused about these java-code
>> analyzers
>> >> that
>> >> >>> >> implement snowball algorithms, why do they exist when the same
>> >> >>> >> functionality
>> >> >>> >> is in contrib/snowball?
>> >> >>> >>
>> >> >>> >>
>> >> >>> >> > Don't use ensureOpen() excessively in IndexReader and
>> IndexWriter
>> >> >>> >> > --------------------------------------------------------------
>> ---
>> >> >>> >> >
>> >> >>> >> > Key: LUCENE-1707
>> >> >>> >> > URL:
>> >> >>> >> > https://issues.apache.org/jira/browse/LUCENE-1707
>> >> >>> >> > Project: Lucene - Java
>> >> >>> >> > Issue Type: Improvement
>> >> >>> >> > Components: Index
>> >> >>> >> > Reporter: Shai Erera
>> >> >>> >> > Fix For: 2.9
>> >> >>> >> >
>> >> >>> >> > Attachments: LUCENE-1707.patch, LUCENE-1707.patch
>> >> >>> >> >
>> >> >>> >> >
>> >> >>> >> > A spin off from here:
>> >> >>> >> > http://www.nabble.com/Excessive-use-of-ensureOpen()-
>> >> td24127806.html.
>> >> >>> >> > We should stop calling this method when it's not necessary for
>> >> any
>> >> >>> >> > internal Lucene code. Currently, this code seems to hurt
>> properly
>> >> >>> >> > written
>> >> >>> >> > apps, unnecessarily.
>> >> >>> >> > Will post a patch soon
>> >> >>> >>
>> >> >>> >> --
>> >> >>> >> This message is automatically generated by JIRA.
>> >> >>> >> -
>> >> >>> >> You can reply to this email to add a comment to the issue
>> online.
>> >> >>> >>
>> >> >>> >>
>> >> >>> >> ----------------------------------------------------------------
>> ---
>> >> --
>> >> >>> >> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>> >> >>> >> For additional commands, e-mail: java-dev-help@lucene.apache.org
>> >> >>> >>
>> >> >>> >
>> >> >>> >
>> >> >>> >
>> >> >>> > --
>> >> >>> > --
>> >> >>> > - Mark
>> >> >>> >
>> >> >>> > http://www.lucidimagination.com
>> >> >>> >
>> >> >>> >
>> >> >>>
>> >> >>>
>> >> >>>
>> >> >>> --
>> >> >>> Robert Muir
>> >> >>> rcmuir@gmail.com
>> >> >>>
>> >> >>> -------------------------------------------------------------------
>> --
>> >> >>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>> >> >>> For additional commands, e-mail: java-dev-help@lucene.apache.org
>> >> >>>
>> >> >>
>> >> >>
>> >> >
>> >> > ---------------------------------------------------------------------
>> >> > To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>> >> > For additional commands, e-mail: java-dev-help@lucene.apache.org
>> >> >
>> >> >
>> >>
>> >>
>> >>
>> >> --
>> >> Robert Muir
>> >> rcmuir@gmail.com
>> >>
>> >> ---------------------------------------------------------------------
>> >> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>> >> For additional commands, e-mail: java-dev-help@lucene.apache.org
>> >
>> >
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>> > For additional commands, e-mail: java-dev-help@lucene.apache.org
>> >
>> >
>>
>>
>>
>> --
>> Robert Muir
>> rcmuir@gmail.com
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>
--
Robert Muir
rcmuir@gmail.com
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
RE: [jira] Commented: (LUCENE-1707) Don't use ensureOpen() excessively in IndexReader and IndexWriter
Posted by Uwe Schindler <uw...@thetaphi.de>.
The whole russian analyzer is very strange and works against all
charset/unicode conventions. It defines own "charsets" (the only valid one
is UNICODE), which are all applied to standard java 16 bit chars. The test
shows, how this works: It open a text file in KOI8 using the "ISO-88591-1"
charset (just to not modify the codepoints when converting to 16bit java
chars (in principle it does a deprecated "new String(byte[],0)"). These
completely wrong java chars are then handled by an analyzers's internal
charset conversion (working on the 16 bit chars).
The only correct usage of this package is:
- open file with correct encoding (when instantiating the Reader, so specify
KOI8 or windows1251 to the Reader). The string is then correctly UTF-16
encoded java chars. On this string the "pseudo-charset" UNICODE of this
analyzer can be used.
In my opinion, this invalid usage of java chars should be deprecated, the
only correct pseudo-charset should be the one specified by UNICODE and all
charset conversions should be done using the Reader.
Uwe
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de
> -----Original Message-----
> From: Robert Muir [mailto:rcmuir@gmail.com]
> Sent: Monday, July 06, 2009 3:08 PM
> To: java-dev@lucene.apache.org
> Subject: Re: [jira] Commented: (LUCENE-1707) Don't use ensureOpen()
> excessively in IndexReader and IndexWriter
>
> Uwe, I think so too. This way it will not be prone to breakage again.
>
> On Mon, Jul 6, 2009 at 8:38 AM, Uwe Schindler<uw...@thetaphi.de> wrote:
> > In my opinion, these files should be converted to UTF-8 and committed
> again
> > (and the Reader in the test recondigured for UTF-8). Then they can be
> native
> > EOL style again. The problem is that SVN can only handle the EOL style
> for
> > one-byte-per-char and UTF-8 files.
> >
> > I give it a try here (and I have a converter).
> >
> > -----
> > Uwe Schindler
> > H.-H.-Meier-Allee 63, D-28213 Bremen
> > http://www.thetaphi.de
> > eMail: uwe@thetaphi.de
> >
> >> -----Original Message-----
> >> From: Robert Muir [mailto:rcmuir@gmail.com]
> >> Sent: Monday, July 06, 2009 1:11 PM
> >> To: java-dev@lucene.apache.org
> >> Subject: Re: [jira] Commented: (LUCENE-1707) Don't use ensureOpen()
> >> excessively in IndexReader and IndexWriter
> >>
> >> yeah, its fixed now.
> >>
> >> On Mon, Jul 6, 2009 at 7:06 AM, Michael
> >> McCandless<lu...@mikemccandless.com> wrote:
> >> > Is this the native vs LF svn:eol-style that Uwe already fixed?
> >> >
> >> > Mike
> >> >
> >> > On Thu, Jul 2, 2009 at 10:03 AM, Shai Erera<se...@gmail.com> wrote:
> >> >> Can somebody try to revert the change and test it on Windows?
> >> >>
> >> >> On Thu, Jul 2, 2009 at 4:44 PM, Robert Muir <rc...@gmail.com>
> wrote:
> >> >>>
> >> >>> well then I have no idea why it doesn't fail. Except that perhaps
> its
> >> >>> EOL-related (as Shai said), and that the failure is somehow
> >> >>> platform-dependent due to newline differences between windows and
> unix
> >> >>> (and the way these are encoded in UTF-16/stored in SVN)?
> >> >>>
> >> >>> I don't do really any work with files in UTF-16 so this is just a
> >> theory.
> >> >>>
> >> >>> On Thu, Jul 2, 2009 at 9:40 AM, Mark Miller<ma...@gmail.com>
> >> wrote:
> >> >>> > Hudson runs all the tests and emails java-dev if any of them
> fail.
> >> >>> >
> >> >>> > On Thu, Jul 2, 2009 at 9:37 AM, Robert Muir (JIRA)
> <ji...@apache.org>
> >> >>> > wrote:
> >> >>> >>
> >> >>> >> [
> >> >>> >>
> >> >>> >> https://issues.apache.org/jira/browse/LUCENE-
> >> 1707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-
> >> tabpanel&focusedCommentId=12726479#action_12726479
> >> >>> >> ]
> >> >>> >>
> >> >>> >> Robert Muir commented on LUCENE-1707:
> >> >>> >> -------------------------------------
> >> >>> >>
> >> >>> >> bq. Why doesn't Hudson encounter this problem?
> >> >>> >>
> >> >>> >> Forgive my ignorance, does hudson also run tests or just verify
> >> build?
> >> >>> >> These files are only used in tests!
> >> >>> >>
> >> >>> >> I agree we should correct it, and perhaps to prevent other
> problems
> >> >>> >> these
> >> >>> >> files should be converted to UTF-8.
> >> >>> >>
> >> >>> >> For the record I am still confused about these java-code
> analyzers
> >> that
> >> >>> >> implement snowball algorithms, why do they exist when the same
> >> >>> >> functionality
> >> >>> >> is in contrib/snowball?
> >> >>> >>
> >> >>> >>
> >> >>> >> > Don't use ensureOpen() excessively in IndexReader and
> IndexWriter
> >> >>> >> > --------------------------------------------------------------
> ---
> >> >>> >> >
> >> >>> >> > Key: LUCENE-1707
> >> >>> >> > URL:
> >> >>> >> > https://issues.apache.org/jira/browse/LUCENE-1707
> >> >>> >> > Project: Lucene - Java
> >> >>> >> > Issue Type: Improvement
> >> >>> >> > Components: Index
> >> >>> >> > Reporter: Shai Erera
> >> >>> >> > Fix For: 2.9
> >> >>> >> >
> >> >>> >> > Attachments: LUCENE-1707.patch, LUCENE-1707.patch
> >> >>> >> >
> >> >>> >> >
> >> >>> >> > A spin off from here:
> >> >>> >> > http://www.nabble.com/Excessive-use-of-ensureOpen()-
> >> td24127806.html.
> >> >>> >> > We should stop calling this method when it's not necessary for
> >> any
> >> >>> >> > internal Lucene code. Currently, this code seems to hurt
> properly
> >> >>> >> > written
> >> >>> >> > apps, unnecessarily.
> >> >>> >> > Will post a patch soon
> >> >>> >>
> >> >>> >> --
> >> >>> >> This message is automatically generated by JIRA.
> >> >>> >> -
> >> >>> >> You can reply to this email to add a comment to the issue
> online.
> >> >>> >>
> >> >>> >>
> >> >>> >> ----------------------------------------------------------------
> ---
> >> --
> >> >>> >> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> >> >>> >> For additional commands, e-mail: java-dev-help@lucene.apache.org
> >> >>> >>
> >> >>> >
> >> >>> >
> >> >>> >
> >> >>> > --
> >> >>> > --
> >> >>> > - Mark
> >> >>> >
> >> >>> > http://www.lucidimagination.com
> >> >>> >
> >> >>> >
> >> >>>
> >> >>>
> >> >>>
> >> >>> --
> >> >>> Robert Muir
> >> >>> rcmuir@gmail.com
> >> >>>
> >> >>> -------------------------------------------------------------------
> --
> >> >>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> >> >>> For additional commands, e-mail: java-dev-help@lucene.apache.org
> >> >>>
> >> >>
> >> >>
> >> >
> >> > ---------------------------------------------------------------------
> >> > To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> >> > For additional commands, e-mail: java-dev-help@lucene.apache.org
> >> >
> >> >
> >>
> >>
> >>
> >> --
> >> Robert Muir
> >> rcmuir@gmail.com
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> >> For additional commands, e-mail: java-dev-help@lucene.apache.org
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-dev-help@lucene.apache.org
> >
> >
>
>
>
> --
> Robert Muir
> rcmuir@gmail.com
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: [jira] Commented: (LUCENE-1707) Don't use ensureOpen()
excessively in IndexReader and IndexWriter
Posted by Robert Muir <rc...@gmail.com>.
Uwe, I think so too. This way it will not be prone to breakage again.
On Mon, Jul 6, 2009 at 8:38 AM, Uwe Schindler<uw...@thetaphi.de> wrote:
> In my opinion, these files should be converted to UTF-8 and committed again
> (and the Reader in the test recondigured for UTF-8). Then they can be native
> EOL style again. The problem is that SVN can only handle the EOL style for
> one-byte-per-char and UTF-8 files.
>
> I give it a try here (and I have a converter).
>
> -----
> Uwe Schindler
> H.-H.-Meier-Allee 63, D-28213 Bremen
> http://www.thetaphi.de
> eMail: uwe@thetaphi.de
>
>> -----Original Message-----
>> From: Robert Muir [mailto:rcmuir@gmail.com]
>> Sent: Monday, July 06, 2009 1:11 PM
>> To: java-dev@lucene.apache.org
>> Subject: Re: [jira] Commented: (LUCENE-1707) Don't use ensureOpen()
>> excessively in IndexReader and IndexWriter
>>
>> yeah, its fixed now.
>>
>> On Mon, Jul 6, 2009 at 7:06 AM, Michael
>> McCandless<lu...@mikemccandless.com> wrote:
>> > Is this the native vs LF svn:eol-style that Uwe already fixed?
>> >
>> > Mike
>> >
>> > On Thu, Jul 2, 2009 at 10:03 AM, Shai Erera<se...@gmail.com> wrote:
>> >> Can somebody try to revert the change and test it on Windows?
>> >>
>> >> On Thu, Jul 2, 2009 at 4:44 PM, Robert Muir <rc...@gmail.com> wrote:
>> >>>
>> >>> well then I have no idea why it doesn't fail. Except that perhaps its
>> >>> EOL-related (as Shai said), and that the failure is somehow
>> >>> platform-dependent due to newline differences between windows and unix
>> >>> (and the way these are encoded in UTF-16/stored in SVN)?
>> >>>
>> >>> I don't do really any work with files in UTF-16 so this is just a
>> theory.
>> >>>
>> >>> On Thu, Jul 2, 2009 at 9:40 AM, Mark Miller<ma...@gmail.com>
>> wrote:
>> >>> > Hudson runs all the tests and emails java-dev if any of them fail.
>> >>> >
>> >>> > On Thu, Jul 2, 2009 at 9:37 AM, Robert Muir (JIRA) <ji...@apache.org>
>> >>> > wrote:
>> >>> >>
>> >>> >> [
>> >>> >>
>> >>> >> https://issues.apache.org/jira/browse/LUCENE-
>> 1707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-
>> tabpanel&focusedCommentId=12726479#action_12726479
>> >>> >> ]
>> >>> >>
>> >>> >> Robert Muir commented on LUCENE-1707:
>> >>> >> -------------------------------------
>> >>> >>
>> >>> >> bq. Why doesn't Hudson encounter this problem?
>> >>> >>
>> >>> >> Forgive my ignorance, does hudson also run tests or just verify
>> build?
>> >>> >> These files are only used in tests!
>> >>> >>
>> >>> >> I agree we should correct it, and perhaps to prevent other problems
>> >>> >> these
>> >>> >> files should be converted to UTF-8.
>> >>> >>
>> >>> >> For the record I am still confused about these java-code analyzers
>> that
>> >>> >> implement snowball algorithms, why do they exist when the same
>> >>> >> functionality
>> >>> >> is in contrib/snowball?
>> >>> >>
>> >>> >>
>> >>> >> > Don't use ensureOpen() excessively in IndexReader and IndexWriter
>> >>> >> > -----------------------------------------------------------------
>> >>> >> >
>> >>> >> > Key: LUCENE-1707
>> >>> >> > URL:
>> >>> >> > https://issues.apache.org/jira/browse/LUCENE-1707
>> >>> >> > Project: Lucene - Java
>> >>> >> > Issue Type: Improvement
>> >>> >> > Components: Index
>> >>> >> > Reporter: Shai Erera
>> >>> >> > Fix For: 2.9
>> >>> >> >
>> >>> >> > Attachments: LUCENE-1707.patch, LUCENE-1707.patch
>> >>> >> >
>> >>> >> >
>> >>> >> > A spin off from here:
>> >>> >> > http://www.nabble.com/Excessive-use-of-ensureOpen()-
>> td24127806.html.
>> >>> >> > We should stop calling this method when it's not necessary for
>> any
>> >>> >> > internal Lucene code. Currently, this code seems to hurt properly
>> >>> >> > written
>> >>> >> > apps, unnecessarily.
>> >>> >> > Will post a patch soon
>> >>> >>
>> >>> >> --
>> >>> >> This message is automatically generated by JIRA.
>> >>> >> -
>> >>> >> You can reply to this email to add a comment to the issue online.
>> >>> >>
>> >>> >>
>> >>> >> -------------------------------------------------------------------
>> --
>> >>> >> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>> >>> >> For additional commands, e-mail: java-dev-help@lucene.apache.org
>> >>> >>
>> >>> >
>> >>> >
>> >>> >
>> >>> > --
>> >>> > --
>> >>> > - Mark
>> >>> >
>> >>> > http://www.lucidimagination.com
>> >>> >
>> >>> >
>> >>>
>> >>>
>> >>>
>> >>> --
>> >>> Robert Muir
>> >>> rcmuir@gmail.com
>> >>>
>> >>> ---------------------------------------------------------------------
>> >>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>> >>> For additional commands, e-mail: java-dev-help@lucene.apache.org
>> >>>
>> >>
>> >>
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>> > For additional commands, e-mail: java-dev-help@lucene.apache.org
>> >
>> >
>>
>>
>>
>> --
>> Robert Muir
>> rcmuir@gmail.com
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>
--
Robert Muir
rcmuir@gmail.com
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
RE: [jira] Commented: (LUCENE-1707) Don't use ensureOpen() excessively in IndexReader and IndexWriter
Posted by Uwe Schindler <uw...@thetaphi.de>.
In my opinion, these files should be converted to UTF-8 and committed again
(and the Reader in the test recondigured for UTF-8). Then they can be native
EOL style again. The problem is that SVN can only handle the EOL style for
one-byte-per-char and UTF-8 files.
I give it a try here (and I have a converter).
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de
> -----Original Message-----
> From: Robert Muir [mailto:rcmuir@gmail.com]
> Sent: Monday, July 06, 2009 1:11 PM
> To: java-dev@lucene.apache.org
> Subject: Re: [jira] Commented: (LUCENE-1707) Don't use ensureOpen()
> excessively in IndexReader and IndexWriter
>
> yeah, its fixed now.
>
> On Mon, Jul 6, 2009 at 7:06 AM, Michael
> McCandless<lu...@mikemccandless.com> wrote:
> > Is this the native vs LF svn:eol-style that Uwe already fixed?
> >
> > Mike
> >
> > On Thu, Jul 2, 2009 at 10:03 AM, Shai Erera<se...@gmail.com> wrote:
> >> Can somebody try to revert the change and test it on Windows?
> >>
> >> On Thu, Jul 2, 2009 at 4:44 PM, Robert Muir <rc...@gmail.com> wrote:
> >>>
> >>> well then I have no idea why it doesn't fail. Except that perhaps its
> >>> EOL-related (as Shai said), and that the failure is somehow
> >>> platform-dependent due to newline differences between windows and unix
> >>> (and the way these are encoded in UTF-16/stored in SVN)?
> >>>
> >>> I don't do really any work with files in UTF-16 so this is just a
> theory.
> >>>
> >>> On Thu, Jul 2, 2009 at 9:40 AM, Mark Miller<ma...@gmail.com>
> wrote:
> >>> > Hudson runs all the tests and emails java-dev if any of them fail.
> >>> >
> >>> > On Thu, Jul 2, 2009 at 9:37 AM, Robert Muir (JIRA) <ji...@apache.org>
> >>> > wrote:
> >>> >>
> >>> >> [
> >>> >>
> >>> >> https://issues.apache.org/jira/browse/LUCENE-
> 1707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-
> tabpanel&focusedCommentId=12726479#action_12726479
> >>> >> ]
> >>> >>
> >>> >> Robert Muir commented on LUCENE-1707:
> >>> >> -------------------------------------
> >>> >>
> >>> >> bq. Why doesn't Hudson encounter this problem?
> >>> >>
> >>> >> Forgive my ignorance, does hudson also run tests or just verify
> build?
> >>> >> These files are only used in tests!
> >>> >>
> >>> >> I agree we should correct it, and perhaps to prevent other problems
> >>> >> these
> >>> >> files should be converted to UTF-8.
> >>> >>
> >>> >> For the record I am still confused about these java-code analyzers
> that
> >>> >> implement snowball algorithms, why do they exist when the same
> >>> >> functionality
> >>> >> is in contrib/snowball?
> >>> >>
> >>> >>
> >>> >> > Don't use ensureOpen() excessively in IndexReader and IndexWriter
> >>> >> > -----------------------------------------------------------------
> >>> >> >
> >>> >> > Key: LUCENE-1707
> >>> >> > URL:
> >>> >> > https://issues.apache.org/jira/browse/LUCENE-1707
> >>> >> > Project: Lucene - Java
> >>> >> > Issue Type: Improvement
> >>> >> > Components: Index
> >>> >> > Reporter: Shai Erera
> >>> >> > Fix For: 2.9
> >>> >> >
> >>> >> > Attachments: LUCENE-1707.patch, LUCENE-1707.patch
> >>> >> >
> >>> >> >
> >>> >> > A spin off from here:
> >>> >> > http://www.nabble.com/Excessive-use-of-ensureOpen()-
> td24127806.html.
> >>> >> > We should stop calling this method when it's not necessary for
> any
> >>> >> > internal Lucene code. Currently, this code seems to hurt properly
> >>> >> > written
> >>> >> > apps, unnecessarily.
> >>> >> > Will post a patch soon
> >>> >>
> >>> >> --
> >>> >> This message is automatically generated by JIRA.
> >>> >> -
> >>> >> You can reply to this email to add a comment to the issue online.
> >>> >>
> >>> >>
> >>> >> -------------------------------------------------------------------
> --
> >>> >> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> >>> >> For additional commands, e-mail: java-dev-help@lucene.apache.org
> >>> >>
> >>> >
> >>> >
> >>> >
> >>> > --
> >>> > --
> >>> > - Mark
> >>> >
> >>> > http://www.lucidimagination.com
> >>> >
> >>> >
> >>>
> >>>
> >>>
> >>> --
> >>> Robert Muir
> >>> rcmuir@gmail.com
> >>>
> >>> ---------------------------------------------------------------------
> >>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> >>> For additional commands, e-mail: java-dev-help@lucene.apache.org
> >>>
> >>
> >>
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-dev-help@lucene.apache.org
> >
> >
>
>
>
> --
> Robert Muir
> rcmuir@gmail.com
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: [jira] Commented: (LUCENE-1707) Don't use ensureOpen()
excessively in IndexReader and IndexWriter
Posted by Robert Muir <rc...@gmail.com>.
yeah, its fixed now.
On Mon, Jul 6, 2009 at 7:06 AM, Michael
McCandless<lu...@mikemccandless.com> wrote:
> Is this the native vs LF svn:eol-style that Uwe already fixed?
>
> Mike
>
> On Thu, Jul 2, 2009 at 10:03 AM, Shai Erera<se...@gmail.com> wrote:
>> Can somebody try to revert the change and test it on Windows?
>>
>> On Thu, Jul 2, 2009 at 4:44 PM, Robert Muir <rc...@gmail.com> wrote:
>>>
>>> well then I have no idea why it doesn't fail. Except that perhaps its
>>> EOL-related (as Shai said), and that the failure is somehow
>>> platform-dependent due to newline differences between windows and unix
>>> (and the way these are encoded in UTF-16/stored in SVN)?
>>>
>>> I don't do really any work with files in UTF-16 so this is just a theory.
>>>
>>> On Thu, Jul 2, 2009 at 9:40 AM, Mark Miller<ma...@gmail.com> wrote:
>>> > Hudson runs all the tests and emails java-dev if any of them fail.
>>> >
>>> > On Thu, Jul 2, 2009 at 9:37 AM, Robert Muir (JIRA) <ji...@apache.org>
>>> > wrote:
>>> >>
>>> >> [
>>> >>
>>> >> https://issues.apache.org/jira/browse/LUCENE-1707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12726479#action_12726479
>>> >> ]
>>> >>
>>> >> Robert Muir commented on LUCENE-1707:
>>> >> -------------------------------------
>>> >>
>>> >> bq. Why doesn't Hudson encounter this problem?
>>> >>
>>> >> Forgive my ignorance, does hudson also run tests or just verify build?
>>> >> These files are only used in tests!
>>> >>
>>> >> I agree we should correct it, and perhaps to prevent other problems
>>> >> these
>>> >> files should be converted to UTF-8.
>>> >>
>>> >> For the record I am still confused about these java-code analyzers that
>>> >> implement snowball algorithms, why do they exist when the same
>>> >> functionality
>>> >> is in contrib/snowball?
>>> >>
>>> >>
>>> >> > Don't use ensureOpen() excessively in IndexReader and IndexWriter
>>> >> > -----------------------------------------------------------------
>>> >> >
>>> >> > Key: LUCENE-1707
>>> >> > URL:
>>> >> > https://issues.apache.org/jira/browse/LUCENE-1707
>>> >> > Project: Lucene - Java
>>> >> > Issue Type: Improvement
>>> >> > Components: Index
>>> >> > Reporter: Shai Erera
>>> >> > Fix For: 2.9
>>> >> >
>>> >> > Attachments: LUCENE-1707.patch, LUCENE-1707.patch
>>> >> >
>>> >> >
>>> >> > A spin off from here:
>>> >> > http://www.nabble.com/Excessive-use-of-ensureOpen()-td24127806.html.
>>> >> > We should stop calling this method when it's not necessary for any
>>> >> > internal Lucene code. Currently, this code seems to hurt properly
>>> >> > written
>>> >> > apps, unnecessarily.
>>> >> > Will post a patch soon
>>> >>
>>> >> --
>>> >> This message is automatically generated by JIRA.
>>> >> -
>>> >> You can reply to this email to add a comment to the issue online.
>>> >>
>>> >>
>>> >> ---------------------------------------------------------------------
>>> >> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>>> >> For additional commands, e-mail: java-dev-help@lucene.apache.org
>>> >>
>>> >
>>> >
>>> >
>>> > --
>>> > --
>>> > - Mark
>>> >
>>> > http://www.lucidimagination.com
>>> >
>>> >
>>>
>>>
>>>
>>> --
>>> Robert Muir
>>> rcmuir@gmail.com
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-dev-help@lucene.apache.org
>>>
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>
--
Robert Muir
rcmuir@gmail.com
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: [jira] Commented: (LUCENE-1707) Don't use ensureOpen()
excessively in IndexReader and IndexWriter
Posted by Michael McCandless <lu...@mikemccandless.com>.
Is this the native vs LF svn:eol-style that Uwe already fixed?
Mike
On Thu, Jul 2, 2009 at 10:03 AM, Shai Erera<se...@gmail.com> wrote:
> Can somebody try to revert the change and test it on Windows?
>
> On Thu, Jul 2, 2009 at 4:44 PM, Robert Muir <rc...@gmail.com> wrote:
>>
>> well then I have no idea why it doesn't fail. Except that perhaps its
>> EOL-related (as Shai said), and that the failure is somehow
>> platform-dependent due to newline differences between windows and unix
>> (and the way these are encoded in UTF-16/stored in SVN)?
>>
>> I don't do really any work with files in UTF-16 so this is just a theory.
>>
>> On Thu, Jul 2, 2009 at 9:40 AM, Mark Miller<ma...@gmail.com> wrote:
>> > Hudson runs all the tests and emails java-dev if any of them fail.
>> >
>> > On Thu, Jul 2, 2009 at 9:37 AM, Robert Muir (JIRA) <ji...@apache.org>
>> > wrote:
>> >>
>> >> [
>> >>
>> >> https://issues.apache.org/jira/browse/LUCENE-1707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12726479#action_12726479
>> >> ]
>> >>
>> >> Robert Muir commented on LUCENE-1707:
>> >> -------------------------------------
>> >>
>> >> bq. Why doesn't Hudson encounter this problem?
>> >>
>> >> Forgive my ignorance, does hudson also run tests or just verify build?
>> >> These files are only used in tests!
>> >>
>> >> I agree we should correct it, and perhaps to prevent other problems
>> >> these
>> >> files should be converted to UTF-8.
>> >>
>> >> For the record I am still confused about these java-code analyzers that
>> >> implement snowball algorithms, why do they exist when the same
>> >> functionality
>> >> is in contrib/snowball?
>> >>
>> >>
>> >> > Don't use ensureOpen() excessively in IndexReader and IndexWriter
>> >> > -----------------------------------------------------------------
>> >> >
>> >> > Key: LUCENE-1707
>> >> > URL:
>> >> > https://issues.apache.org/jira/browse/LUCENE-1707
>> >> > Project: Lucene - Java
>> >> > Issue Type: Improvement
>> >> > Components: Index
>> >> > Reporter: Shai Erera
>> >> > Fix For: 2.9
>> >> >
>> >> > Attachments: LUCENE-1707.patch, LUCENE-1707.patch
>> >> >
>> >> >
>> >> > A spin off from here:
>> >> > http://www.nabble.com/Excessive-use-of-ensureOpen()-td24127806.html.
>> >> > We should stop calling this method when it's not necessary for any
>> >> > internal Lucene code. Currently, this code seems to hurt properly
>> >> > written
>> >> > apps, unnecessarily.
>> >> > Will post a patch soon
>> >>
>> >> --
>> >> This message is automatically generated by JIRA.
>> >> -
>> >> You can reply to this email to add a comment to the issue online.
>> >>
>> >>
>> >> ---------------------------------------------------------------------
>> >> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>> >> For additional commands, e-mail: java-dev-help@lucene.apache.org
>> >>
>> >
>> >
>> >
>> > --
>> > --
>> > - Mark
>> >
>> > http://www.lucidimagination.com
>> >
>> >
>>
>>
>>
>> --
>> Robert Muir
>> rcmuir@gmail.com
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-dev-help@lucene.apache.org
>>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: [jira] Commented: (LUCENE-1707) Don't use ensureOpen()
excessively in IndexReader and IndexWriter
Posted by Shai Erera <se...@gmail.com>.
Can somebody try to revert the change and test it on Windows?
On Thu, Jul 2, 2009 at 4:44 PM, Robert Muir <rc...@gmail.com> wrote:
> well then I have no idea why it doesn't fail. Except that perhaps its
> EOL-related (as Shai said), and that the failure is somehow
> platform-dependent due to newline differences between windows and unix
> (and the way these are encoded in UTF-16/stored in SVN)?
>
> I don't do really any work with files in UTF-16 so this is just a theory.
>
> On Thu, Jul 2, 2009 at 9:40 AM, Mark Miller<ma...@gmail.com> wrote:
> > Hudson runs all the tests and emails java-dev if any of them fail.
> >
> > On Thu, Jul 2, 2009 at 9:37 AM, Robert Muir (JIRA) <ji...@apache.org>
> wrote:
> >>
> >> [
> >>
> https://issues.apache.org/jira/browse/LUCENE-1707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12726479#action_12726479
> >> ]
> >>
> >> Robert Muir commented on LUCENE-1707:
> >> -------------------------------------
> >>
> >> bq. Why doesn't Hudson encounter this problem?
> >>
> >> Forgive my ignorance, does hudson also run tests or just verify build?
> >> These files are only used in tests!
> >>
> >> I agree we should correct it, and perhaps to prevent other problems
> these
> >> files should be converted to UTF-8.
> >>
> >> For the record I am still confused about these java-code analyzers that
> >> implement snowball algorithms, why do they exist when the same
> functionality
> >> is in contrib/snowball?
> >>
> >>
> >> > Don't use ensureOpen() excessively in IndexReader and IndexWriter
> >> > -----------------------------------------------------------------
> >> >
> >> > Key: LUCENE-1707
> >> > URL:
> https://issues.apache.org/jira/browse/LUCENE-1707
> >> > Project: Lucene - Java
> >> > Issue Type: Improvement
> >> > Components: Index
> >> > Reporter: Shai Erera
> >> > Fix For: 2.9
> >> >
> >> > Attachments: LUCENE-1707.patch, LUCENE-1707.patch
> >> >
> >> >
> >> > A spin off from here:
> >> > http://www.nabble.com/Excessive-use-of-ensureOpen()-td24127806.html<http://www.nabble.com/Excessive-use-of-ensureOpen%28%29-td24127806.html>
> .
> >> > We should stop calling this method when it's not necessary for any
> >> > internal Lucene code. Currently, this code seems to hurt properly
> written
> >> > apps, unnecessarily.
> >> > Will post a patch soon
> >>
> >> --
> >> This message is automatically generated by JIRA.
> >> -
> >> You can reply to this email to add a comment to the issue online.
> >>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> >> For additional commands, e-mail: java-dev-help@lucene.apache.org
> >>
> >
> >
> >
> > --
> > --
> > - Mark
> >
> > http://www.lucidimagination.com
> >
> >
>
>
>
> --
> Robert Muir
> rcmuir@gmail.com
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>
Re: [jira] Commented: (LUCENE-1707) Don't use ensureOpen()
excessively in IndexReader and IndexWriter
Posted by Robert Muir <rc...@gmail.com>.
well then I have no idea why it doesn't fail. Except that perhaps its
EOL-related (as Shai said), and that the failure is somehow
platform-dependent due to newline differences between windows and unix
(and the way these are encoded in UTF-16/stored in SVN)?
I don't do really any work with files in UTF-16 so this is just a theory.
On Thu, Jul 2, 2009 at 9:40 AM, Mark Miller<ma...@gmail.com> wrote:
> Hudson runs all the tests and emails java-dev if any of them fail.
>
> On Thu, Jul 2, 2009 at 9:37 AM, Robert Muir (JIRA) <ji...@apache.org> wrote:
>>
>> [
>> https://issues.apache.org/jira/browse/LUCENE-1707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12726479#action_12726479
>> ]
>>
>> Robert Muir commented on LUCENE-1707:
>> -------------------------------------
>>
>> bq. Why doesn't Hudson encounter this problem?
>>
>> Forgive my ignorance, does hudson also run tests or just verify build?
>> These files are only used in tests!
>>
>> I agree we should correct it, and perhaps to prevent other problems these
>> files should be converted to UTF-8.
>>
>> For the record I am still confused about these java-code analyzers that
>> implement snowball algorithms, why do they exist when the same functionality
>> is in contrib/snowball?
>>
>>
>> > Don't use ensureOpen() excessively in IndexReader and IndexWriter
>> > -----------------------------------------------------------------
>> >
>> > Key: LUCENE-1707
>> > URL: https://issues.apache.org/jira/browse/LUCENE-1707
>> > Project: Lucene - Java
>> > Issue Type: Improvement
>> > Components: Index
>> > Reporter: Shai Erera
>> > Fix For: 2.9
>> >
>> > Attachments: LUCENE-1707.patch, LUCENE-1707.patch
>> >
>> >
>> > A spin off from here:
>> > http://www.nabble.com/Excessive-use-of-ensureOpen()-td24127806.html.
>> > We should stop calling this method when it's not necessary for any
>> > internal Lucene code. Currently, this code seems to hurt properly written
>> > apps, unnecessarily.
>> > Will post a patch soon
>>
>> --
>> This message is automatically generated by JIRA.
>> -
>> You can reply to this email to add a comment to the issue online.
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-dev-help@lucene.apache.org
>>
>
>
>
> --
> --
> - Mark
>
> http://www.lucidimagination.com
>
>
--
Robert Muir
rcmuir@gmail.com
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: [jira] Commented: (LUCENE-1707) Don't use ensureOpen()
excessively in IndexReader and IndexWriter
Posted by Mark Miller <ma...@gmail.com>.
Hudson runs all the tests and emails java-dev if any of them fail.
On Thu, Jul 2, 2009 at 9:37 AM, Robert Muir (JIRA) <ji...@apache.org> wrote:
>
> [
> https://issues.apache.org/jira/browse/LUCENE-1707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12726479#action_12726479]
>
> Robert Muir commented on LUCENE-1707:
> -------------------------------------
>
> bq. Why doesn't Hudson encounter this problem?
>
> Forgive my ignorance, does hudson also run tests or just verify build?
> These files are only used in tests!
>
> I agree we should correct it, and perhaps to prevent other problems these
> files should be converted to UTF-8.
>
> For the record I am still confused about these java-code analyzers that
> implement snowball algorithms, why do they exist when the same functionality
> is in contrib/snowball?
>
>
> > Don't use ensureOpen() excessively in IndexReader and IndexWriter
> > -----------------------------------------------------------------
> >
> > Key: LUCENE-1707
> > URL: https://issues.apache.org/jira/browse/LUCENE-1707
> > Project: Lucene - Java
> > Issue Type: Improvement
> > Components: Index
> > Reporter: Shai Erera
> > Fix For: 2.9
> >
> > Attachments: LUCENE-1707.patch, LUCENE-1707.patch
> >
> >
> > A spin off from here:
> http://www.nabble.com/Excessive-use-of-ensureOpen()-td24127806.html<http://www.nabble.com/Excessive-use-of-ensureOpen%28%29-td24127806.html>
> .
> > We should stop calling this method when it's not necessary for any
> internal Lucene code. Currently, this code seems to hurt properly written
> apps, unnecessarily.
> > Will post a patch soon
>
> --
> This message is automatically generated by JIRA.
> -
> You can reply to this email to add a comment to the issue online.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>
--
--
- Mark
http://www.lucidimagination.com
[jira] Commented: (LUCENE-1707) Don't use ensureOpen() excessively
in IndexReader and IndexWriter
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12726479#action_12726479 ]
Robert Muir commented on LUCENE-1707:
-------------------------------------
bq. Why doesn't Hudson encounter this problem?
Forgive my ignorance, does hudson also run tests or just verify build? These files are only used in tests!
I agree we should correct it, and perhaps to prevent other problems these files should be converted to UTF-8.
For the record I am still confused about these java-code analyzers that implement snowball algorithms, why do they exist when the same functionality is in contrib/snowball?
> Don't use ensureOpen() excessively in IndexReader and IndexWriter
> -----------------------------------------------------------------
>
> Key: LUCENE-1707
> URL: https://issues.apache.org/jira/browse/LUCENE-1707
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Shai Erera
> Fix For: 2.9
>
> Attachments: LUCENE-1707.patch, LUCENE-1707.patch
>
>
> A spin off from here: http://www.nabble.com/Excessive-use-of-ensureOpen()-td24127806.html.
> We should stop calling this method when it's not necessary for any internal Lucene code. Currently, this code seems to hurt properly written apps, unnecessarily.
> Will post a patch soon
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-1707) Don't use ensureOpen() excessively in
IndexReader and IndexWriter
Posted by "Shai Erera (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Shai Erera updated LUCENE-1707:
-------------------------------
Component/s: Index
Fix Version/s: 2.9
Lucene Fields: [New, Patch Available] (was: [New])
> Don't use ensureOpen() excessively in IndexReader and IndexWriter
> -----------------------------------------------------------------
>
> Key: LUCENE-1707
> URL: https://issues.apache.org/jira/browse/LUCENE-1707
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Shai Erera
> Fix For: 2.9
>
> Attachments: LUCENE-1707.patch
>
>
> A spin off from here: http://www.nabble.com/Excessive-use-of-ensureOpen()-td24127806.html.
> We should stop calling this method when it's not necessary for any internal Lucene code. Currently, this code seems to hurt properly written apps, unnecessarily.
> Will post a patch soon
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1707) Don't use ensureOpen() excessively
in IndexReader and IndexWriter
Posted by "Shai Erera (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12726343#action_12726343 ]
Shai Erera commented on LUCENE-1707:
------------------------------------
I haven't tried since then ... I think it's an encoding issue (one of the .txt files was checked out using incorrect encoding). I did change my SVN config file to add eol-style:native to a bunch of extensions, but I'm not sure if that's related. Anyway, this seemed to happen after these changes.
> Don't use ensureOpen() excessively in IndexReader and IndexWriter
> -----------------------------------------------------------------
>
> Key: LUCENE-1707
> URL: https://issues.apache.org/jira/browse/LUCENE-1707
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Shai Erera
> Fix For: 2.9
>
> Attachments: LUCENE-1707.patch, LUCENE-1707.patch
>
>
> A spin off from here: http://www.nabble.com/Excessive-use-of-ensureOpen()-td24127806.html.
> We should stop calling this method when it's not necessary for any internal Lucene code. Currently, this code seems to hurt properly written apps, unnecessarily.
> Will post a patch soon
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1707) Don't use ensureOpen() excessively
in IndexReader and IndexWriter
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12724237#action_12724237 ]
Michael McCandless commented on LUCENE-1707:
--------------------------------------------
bq. Except making refCount not volatile
OK I just committed that. How crazy to have a one-line change :) Such contrast!
bq. BTW, Yonik has just sent an email to java-dev on the semantics of close(). Maybe, if things need to change, we can do it as part of this issue.
Let's do that ("don't allow double-closing") under a new issue?
> Don't use ensureOpen() excessively in IndexReader and IndexWriter
> -----------------------------------------------------------------
>
> Key: LUCENE-1707
> URL: https://issues.apache.org/jira/browse/LUCENE-1707
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Shai Erera
> Fix For: 2.9
>
> Attachments: LUCENE-1707.patch, LUCENE-1707.patch
>
>
> A spin off from here: http://www.nabble.com/Excessive-use-of-ensureOpen()-td24127806.html.
> We should stop calling this method when it's not necessary for any internal Lucene code. Currently, this code seems to hurt properly written apps, unnecessarily.
> Will post a patch soon
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1707) Don't use ensureOpen() excessively
in IndexReader and IndexWriter
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12726143#action_12726143 ]
Robert Muir commented on LUCENE-1707:
-------------------------------------
Shai, I am curious: did you resolve your issue with TestRussianStem?
Although I do not experience this problem, it is unclear why RussianAnalyzer exists when it appears to be a duplicate of snowball functionality (in contrib/snowball)
{noformat}
/**
* Russian stemming algorithm implementation (see http://snowball.sourceforge.net for detailed description).
*
{noformat}
> Don't use ensureOpen() excessively in IndexReader and IndexWriter
> -----------------------------------------------------------------
>
> Key: LUCENE-1707
> URL: https://issues.apache.org/jira/browse/LUCENE-1707
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Shai Erera
> Fix For: 2.9
>
> Attachments: LUCENE-1707.patch, LUCENE-1707.patch
>
>
> A spin off from here: http://www.nabble.com/Excessive-use-of-ensureOpen()-td24127806.html.
> We should stop calling this method when it's not necessary for any internal Lucene code. Currently, this code seems to hurt properly written apps, unnecessarily.
> Will post a patch soon
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1707) Don't use ensureOpen() excessively
in IndexReader and IndexWriter
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12724153#action_12724153 ]
Michael McCandless commented on LUCENE-1707:
--------------------------------------------
Patch looks good; thanks Shai!
bq. Why is it volatile then?
Good question. Only the ctor (which I believe is fine) and ensureOpen() access it un-synchronized, so I think we should make it un-volatile; I'll change it before committing.
bq. What good does "not hiding that bug" do?
EG, say we're working on SegmentReader an accidentally create a bug that sometimes over decRefs. By no longer throwing an exception on close (with this change) unit tests might happily pass when previously they would fail and the bug could go undetected for longer. (Hopefully some tests would still catch the bug, but maybe not). I think catching a bug sooner rather than later is better.
Calling close many times is allowed (not a bug), and we handle that by only doing one decRef. But too many calls to decRef is a real bug and I think we shouldn't mask/delay it.
> Don't use ensureOpen() excessively in IndexReader and IndexWriter
> -----------------------------------------------------------------
>
> Key: LUCENE-1707
> URL: https://issues.apache.org/jira/browse/LUCENE-1707
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Shai Erera
> Fix For: 2.9
>
> Attachments: LUCENE-1707.patch, LUCENE-1707.patch
>
>
> A spin off from here: http://www.nabble.com/Excessive-use-of-ensureOpen()-td24127806.html.
> We should stop calling this method when it's not necessary for any internal Lucene code. Currently, this code seems to hurt properly written apps, unnecessarily.
> Will post a patch soon
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1707) Don't use ensureOpen() excessively
in IndexReader and IndexWriter
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12726462#action_12726462 ]
Robert Muir commented on LUCENE-1707:
-------------------------------------
my code was a little out of date. I am now experiencing this problem too.
I think you hit the nail on the head: the related files: wordsUnicode.txt and stemsUnicode.txt are in UTF-16 encoding. this might be causing the issue, because EOL is very different in that encoding!
> Don't use ensureOpen() excessively in IndexReader and IndexWriter
> -----------------------------------------------------------------
>
> Key: LUCENE-1707
> URL: https://issues.apache.org/jira/browse/LUCENE-1707
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Shai Erera
> Fix For: 2.9
>
> Attachments: LUCENE-1707.patch, LUCENE-1707.patch
>
>
> A spin off from here: http://www.nabble.com/Excessive-use-of-ensureOpen()-td24127806.html.
> We should stop calling this method when it's not necessary for any internal Lucene code. Currently, this code seems to hurt properly written apps, unnecessarily.
> Will post a patch soon
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-1707) Don't use ensureOpen() excessively in
IndexReader and IndexWriter
Posted by "Shai Erera (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Shai Erera updated LUCENE-1707:
-------------------------------
Attachment: LUCENE-1707.patch
Patch includes:
# remove all calls to ensureOpen in IndexReader and deprecated ensureOpen().
# remove most of the calls to ensureOpen in IndexWriter (those that happen at the beginning of methods).
# added isOpen to both IndexReader and IndexWriter.
# deleted some test cases that checked this functionality.
# fixed tag (removed test cases)
# CHANGES
All tests pass, except for TestRussianStem under contrib/analyzers. But that does not seem to be related to this patch, as it fails on a clean trunk checkout also. I think it has to do w/ encoding, but I don't know if I'm the only one that experiences that.
> Don't use ensureOpen() excessively in IndexReader and IndexWriter
> -----------------------------------------------------------------
>
> Key: LUCENE-1707
> URL: https://issues.apache.org/jira/browse/LUCENE-1707
> Project: Lucene - Java
> Issue Type: Improvement
> Reporter: Shai Erera
> Attachments: LUCENE-1707.patch
>
>
> A spin off from here: http://www.nabble.com/Excessive-use-of-ensureOpen()-td24127806.html.
> We should stop calling this method when it's not necessary for any internal Lucene code. Currently, this code seems to hurt properly written apps, unnecessarily.
> Will post a patch soon
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1707) Don't use ensureOpen() excessively
in IndexReader and IndexWriter
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12724112#action_12724112 ]
Michael McCandless commented on LUCENE-1707:
--------------------------------------------
bq. IndexReader already has a 'closed' member which is set to true in close(). Why not use it?
Ahh perfect let's use that!
bq. In fact, it's strange - close() decreases refCount and set 'closed' to true, but if decRef() is called from another reader (it is a public method, so maybe even from an outside app), and refCount is 0, close() will call decRef() again, and fail on assert refCount > 0.
If decRef() is called and refCount is 0, doClose() (not close()) is called. I think this is right?
> Don't use ensureOpen() excessively in IndexReader and IndexWriter
> -----------------------------------------------------------------
>
> Key: LUCENE-1707
> URL: https://issues.apache.org/jira/browse/LUCENE-1707
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Shai Erera
> Fix For: 2.9
>
> Attachments: LUCENE-1707.patch
>
>
> A spin off from here: http://www.nabble.com/Excessive-use-of-ensureOpen()-td24127806.html.
> We should stop calling this method when it's not necessary for any internal Lucene code. Currently, this code seems to hurt properly written apps, unnecessarily.
> Will post a patch soon
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1707) Don't use ensureOpen() excessively
in IndexReader and IndexWriter
Posted by "Shai Erera (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12724055#action_12724055 ]
Shai Erera commented on LUCENE-1707:
------------------------------------
bq. I'd actually rather not remove the ensureOpen calls (I think it's better when the user sees an AlreadyClosedException than an NPE from somewhere deeper in IndexReader), but instead change it to a best effort impl (ie don't use volatile).
I agree that it's better to see ACE than NPE. However, I thought we agreed that nearly all applications do not rely on ACE. And even if we make it super fast, I still don't see why for a properly written application, which knows it does not access a closed reader/writer we should check ensureOpen(). And, our merge code calls document() while reading from an IndexReader, which obviously cannot be close, so why check there?
True, ACE helps developers of good apps, which "forgot" they closed the reader few lines above, track the problem faster ... But then this issue becomes just changing refCount to not volatile? And nothing to do on IndexWriter? If that was the intention, then I don't think an issue is needed, and a committer can just commit this one line change?
> Don't use ensureOpen() excessively in IndexReader and IndexWriter
> -----------------------------------------------------------------
>
> Key: LUCENE-1707
> URL: https://issues.apache.org/jira/browse/LUCENE-1707
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Shai Erera
> Fix For: 2.9
>
> Attachments: LUCENE-1707.patch
>
>
> A spin off from here: http://www.nabble.com/Excessive-use-of-ensureOpen()-td24127806.html.
> We should stop calling this method when it's not necessary for any internal Lucene code. Currently, this code seems to hurt properly written apps, unnecessarily.
> Will post a patch soon
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-1707) Don't use ensureOpen() excessively in
IndexReader and IndexWriter
Posted by "Shai Erera (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Shai Erera updated LUCENE-1707:
-------------------------------
Attachment: LUCENE-1707.patch
Patch with protecting against decRef() and close(). Lines 28-30 include this particular change, so if you disapprove, just delete these lines (assuming everything else is good :) ).
> Don't use ensureOpen() excessively in IndexReader and IndexWriter
> -----------------------------------------------------------------
>
> Key: LUCENE-1707
> URL: https://issues.apache.org/jira/browse/LUCENE-1707
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Shai Erera
> Fix For: 2.9
>
> Attachments: LUCENE-1707.patch, LUCENE-1707.patch
>
>
> A spin off from here: http://www.nabble.com/Excessive-use-of-ensureOpen()-td24127806.html.
> We should stop calling this method when it's not necessary for any internal Lucene code. Currently, this code seems to hurt properly written apps, unnecessarily.
> Will post a patch soon
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1707) Don't use ensureOpen() excessively
in IndexReader and IndexWriter
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12724186#action_12724186 ]
Michael McCandless commented on LUCENE-1707:
--------------------------------------------
bq. So calling decRef() twice in a raw would hit either an assert exception (tests) or ACE (assertions disabled). I think?
Let's say I open a reader, I incRef it once somewhere.... then the reader is passed around and some other code somewhere incorrectly calls decRef twice, then finally at the end I close the reader.
Today, that trips an assertion, because you decRef'd too many times, and one of them "stole" the ref you got from open.
But with this change, we would not trip an assertion.
bq. If we change refCount to not volatile, is this patch still needed? I.e., what's wrong w/ checking in ensureOpen if refCount <= 0, same as before?
Duh, right, I think we in fact have nothing to do here...
> Don't use ensureOpen() excessively in IndexReader and IndexWriter
> -----------------------------------------------------------------
>
> Key: LUCENE-1707
> URL: https://issues.apache.org/jira/browse/LUCENE-1707
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Shai Erera
> Fix For: 2.9
>
> Attachments: LUCENE-1707.patch, LUCENE-1707.patch
>
>
> A spin off from here: http://www.nabble.com/Excessive-use-of-ensureOpen()-td24127806.html.
> We should stop calling this method when it's not necessary for any internal Lucene code. Currently, this code seems to hurt properly written apps, unnecessarily.
> Will post a patch soon
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1707) Don't use ensureOpen() excessively
in IndexReader and IndexWriter
Posted by "Shai Erera (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12724131#action_12724131 ]
Shai Erera commented on LUCENE-1707:
------------------------------------
bq. But, if refCount hits 0 and closed is false then there's some bug lurking (in the app code or the Lucene code)? Ie, someone did an extra decRef. I'd rather things fail then hide the bug in that case.
By that you're saying that calling decRef() and then close() is wrong, which I agree, but still possible. Why not protect against it, by setting closed=true in decRef() if refCount drops to 0? What good does "not hiding that bug" do?
Currently the tests pass whether I protect against it or not, so our code works fine (no surprises here). But I just think that decRef() and close() are public, which doesn't prevent anyone from calling them in whatever order one wants :).
If you don't think it's necessary to protect against it, I'll post a patch w/o it.
> Don't use ensureOpen() excessively in IndexReader and IndexWriter
> -----------------------------------------------------------------
>
> Key: LUCENE-1707
> URL: https://issues.apache.org/jira/browse/LUCENE-1707
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Shai Erera
> Fix For: 2.9
>
> Attachments: LUCENE-1707.patch
>
>
> A spin off from here: http://www.nabble.com/Excessive-use-of-ensureOpen()-td24127806.html.
> We should stop calling this method when it's not necessary for any internal Lucene code. Currently, this code seems to hurt properly written apps, unnecessarily.
> Will post a patch soon
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1707) Don't use ensureOpen() excessively
in IndexReader and IndexWriter
Posted by "Shai Erera (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12724220#action_12724220 ]
Shai Erera commented on LUCENE-1707:
------------------------------------
Ok I won't argue about that - these are hypothetical things anyway :).
bq. Duh, right, I think we in fact have nothing to do here...
Except making refCount not volatile :).
BTW, Yonik has just sent an email to java-dev on the semantics of close(). Maybe, if things need to change, we can do it as part of this issue.
> Don't use ensureOpen() excessively in IndexReader and IndexWriter
> -----------------------------------------------------------------
>
> Key: LUCENE-1707
> URL: https://issues.apache.org/jira/browse/LUCENE-1707
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Shai Erera
> Fix For: 2.9
>
> Attachments: LUCENE-1707.patch, LUCENE-1707.patch
>
>
> A spin off from here: http://www.nabble.com/Excessive-use-of-ensureOpen()-td24127806.html.
> We should stop calling this method when it's not necessary for any internal Lucene code. Currently, this code seems to hurt properly written apps, unnecessarily.
> Will post a patch soon
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1707) Don't use ensureOpen() excessively
in IndexReader and IndexWriter
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12723782#action_12723782 ]
Michael McCandless commented on LUCENE-1707:
--------------------------------------------
I'd actually rather not remove the ensureOpen calls (I think it's
better when the user sees an AlreadyClosedException than an NPE from
somewhere deeper in IndexReader), but instead change it to a best
effort impl (ie don't use volatile).
I think for nearly all the methods where we have ensureOpen today, the
cost (once we stop checking volatile) is tiny (this coming from
someone who is normally a freak about performance!).
> Don't use ensureOpen() excessively in IndexReader and IndexWriter
> -----------------------------------------------------------------
>
> Key: LUCENE-1707
> URL: https://issues.apache.org/jira/browse/LUCENE-1707
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Shai Erera
> Fix For: 2.9
>
> Attachments: LUCENE-1707.patch
>
>
> A spin off from here: http://www.nabble.com/Excessive-use-of-ensureOpen()-td24127806.html.
> We should stop calling this method when it's not necessary for any internal Lucene code. Currently, this code seems to hurt properly written apps, unnecessarily.
> Will post a patch soon
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1707) Don't use ensureOpen() excessively
in IndexReader and IndexWriter
Posted by "Shai Erera (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12724371#action_12724371 ]
Shai Erera commented on LUCENE-1707:
------------------------------------
sure, especially if == 0 and <= 0 perform the same in modern CPUs
> Don't use ensureOpen() excessively in IndexReader and IndexWriter
> -----------------------------------------------------------------
>
> Key: LUCENE-1707
> URL: https://issues.apache.org/jira/browse/LUCENE-1707
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Shai Erera
> Fix For: 2.9
>
> Attachments: LUCENE-1707.patch, LUCENE-1707.patch
>
>
> A spin off from here: http://www.nabble.com/Excessive-use-of-ensureOpen()-td24127806.html.
> We should stop calling this method when it's not necessary for any internal Lucene code. Currently, this code seems to hurt properly written apps, unnecessarily.
> Will post a patch soon
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Resolved: (LUCENE-1707) Don't use ensureOpen() excessively
in IndexReader and IndexWriter
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael McCandless resolved LUCENE-1707.
----------------------------------------
Resolution: Fixed
> Don't use ensureOpen() excessively in IndexReader and IndexWriter
> -----------------------------------------------------------------
>
> Key: LUCENE-1707
> URL: https://issues.apache.org/jira/browse/LUCENE-1707
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Shai Erera
> Fix For: 2.9
>
> Attachments: LUCENE-1707.patch, LUCENE-1707.patch
>
>
> A spin off from here: http://www.nabble.com/Excessive-use-of-ensureOpen()-td24127806.html.
> We should stop calling this method when it's not necessary for any internal Lucene code. Currently, this code seems to hurt properly written apps, unnecessarily.
> Will post a patch soon
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1707) Don't use ensureOpen() excessively
in IndexReader and IndexWriter
Posted by "Shai Erera (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12724291#action_12724291 ]
Shai Erera commented on LUCENE-1707:
------------------------------------
bq. if an app over-decRef's and refCount becomes -1 but doesn't trip the assert
Following assert we call ensureOpen, so I don't think refCount will ever be less than 0. Right?
> Don't use ensureOpen() excessively in IndexReader and IndexWriter
> -----------------------------------------------------------------
>
> Key: LUCENE-1707
> URL: https://issues.apache.org/jira/browse/LUCENE-1707
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Shai Erera
> Fix For: 2.9
>
> Attachments: LUCENE-1707.patch, LUCENE-1707.patch
>
>
> A spin off from here: http://www.nabble.com/Excessive-use-of-ensureOpen()-td24127806.html.
> We should stop calling this method when it's not necessary for any internal Lucene code. Currently, this code seems to hurt properly written apps, unnecessarily.
> Will post a patch soon
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1707) Don't use ensureOpen() excessively
in IndexReader and IndexWriter
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12724296#action_12724296 ]
Michael McCandless commented on LUCENE-1707:
--------------------------------------------
bq. Following assert we call ensureOpen, so I don't think refCount will ever be less than 0. Right?
Heh, you are correct! The only place where we change refCount, we also always call ensureOpen. So there's [currently] no way for it to slip past zero without ensureOpen noticing.
Still I'd prefer to leave it, defensively, eg in case things ever change such that refCount could become negative.
> Don't use ensureOpen() excessively in IndexReader and IndexWriter
> -----------------------------------------------------------------
>
> Key: LUCENE-1707
> URL: https://issues.apache.org/jira/browse/LUCENE-1707
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Shai Erera
> Fix For: 2.9
>
> Attachments: LUCENE-1707.patch, LUCENE-1707.patch
>
>
> A spin off from here: http://www.nabble.com/Excessive-use-of-ensureOpen()-td24127806.html.
> We should stop calling this method when it's not necessary for any internal Lucene code. Currently, this code seems to hurt properly written apps, unnecessarily.
> Will post a patch soon
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1707) Don't use ensureOpen() excessively
in IndexReader and IndexWriter
Posted by "Shai Erera (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12724138#action_12724138 ]
Shai Erera commented on LUCENE-1707:
------------------------------------
I see all access to refCount is synchronized. Why is it volatile then? Just because of ensureOpen()?
> Don't use ensureOpen() excessively in IndexReader and IndexWriter
> -----------------------------------------------------------------
>
> Key: LUCENE-1707
> URL: https://issues.apache.org/jira/browse/LUCENE-1707
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Shai Erera
> Fix For: 2.9
>
> Attachments: LUCENE-1707.patch, LUCENE-1707.patch
>
>
> A spin off from here: http://www.nabble.com/Excessive-use-of-ensureOpen()-td24127806.html.
> We should stop calling this method when it's not necessary for any internal Lucene code. Currently, this code seems to hurt properly written apps, unnecessarily.
> Will post a patch soon
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1707) Don't use ensureOpen() excessively
in IndexReader and IndexWriter
Posted by "Shai Erera (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12724103#action_12724103 ]
Shai Erera commented on LUCENE-1707:
------------------------------------
IndexReader already has a 'closed' member which is set to true in close(). Why not use it?
In fact, it's strange - close() decreases refCount and set 'closed' to true, but if decRef() is called from another reader (it is a public method, so maybe even from an outside app), and refCount is 0, close() will call decRef() again, and fail on assert refCount > 0. Shouldn't that be fixed?
> Don't use ensureOpen() excessively in IndexReader and IndexWriter
> -----------------------------------------------------------------
>
> Key: LUCENE-1707
> URL: https://issues.apache.org/jira/browse/LUCENE-1707
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Shai Erera
> Fix For: 2.9
>
> Attachments: LUCENE-1707.patch
>
>
> A spin off from here: http://www.nabble.com/Excessive-use-of-ensureOpen()-td24127806.html.
> We should stop calling this method when it's not necessary for any internal Lucene code. Currently, this code seems to hurt properly written apps, unnecessarily.
> Will post a patch soon
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1707) Don't use ensureOpen() excessively
in IndexReader and IndexWriter
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12724127#action_12724127 ]
Michael McCandless commented on LUCENE-1707:
--------------------------------------------
bq. So I'm not sure closed can be used.
Woops, sorry closed cannot be used: it simply reflects whether the referrer who had opened the reader, has closed it. So even once they've closed it, the reader can stibe open because others are holding references. We need a separate member that's set as soon as refCount hits 0.
bq. If you call decRef() and then close(), it'll be wrong? I think the solution is to set closed = true in decRef() after doClose() has returned?
But, if refCount hits 0 and closed is false then there's some bug lurking (in the app code or the Lucene code)? Ie, someone did an extra decRef. I'd rather things fail then hide the bug in that case.
> Don't use ensureOpen() excessively in IndexReader and IndexWriter
> -----------------------------------------------------------------
>
> Key: LUCENE-1707
> URL: https://issues.apache.org/jira/browse/LUCENE-1707
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Shai Erera
> Fix For: 2.9
>
> Attachments: LUCENE-1707.patch
>
>
> A spin off from here: http://www.nabble.com/Excessive-use-of-ensureOpen()-td24127806.html.
> We should stop calling this method when it's not necessary for any internal Lucene code. Currently, this code seems to hurt properly written apps, unnecessarily.
> Will post a patch soon
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1707) Don't use ensureOpen() excessively
in IndexReader and IndexWriter
Posted by "Shai Erera (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12724121#action_12724121 ]
Shai Erera commented on LUCENE-1707:
------------------------------------
bq. If decRef() is called and refCount is 0, doClose() (not close()) is called. I think this is right?
If you call decRef() and then close(), it'll be wrong? I think the solution is to set closed = true in decRef() after doClose() has returned?
> Don't use ensureOpen() excessively in IndexReader and IndexWriter
> -----------------------------------------------------------------
>
> Key: LUCENE-1707
> URL: https://issues.apache.org/jira/browse/LUCENE-1707
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Shai Erera
> Fix For: 2.9
>
> Attachments: LUCENE-1707.patch
>
>
> A spin off from here: http://www.nabble.com/Excessive-use-of-ensureOpen()-td24127806.html.
> We should stop calling this method when it's not necessary for any internal Lucene code. Currently, this code seems to hurt properly written apps, unnecessarily.
> Will post a patch soon
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1707) Don't use ensureOpen() excessively
in IndexReader and IndexWriter
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12724090#action_12724090 ]
Michael McCandless commented on LUCENE-1707:
--------------------------------------------
bq. However, I thought we agreed that nearly all applications do not rely on ACE
It's not that apps are relying on ACE, it's that the unlucky user who has problems is informed much more clearly that the problem is in their code and not a bug in Lucene. It's a trap.
In the past there were a good number of users posting with confusing exceptions coming out Lucene, thinking they had hit a bug. These users spent time, others spent time reading it, answering it, etc. Now, I haven't seen one of those emails in quite a while.
bq. But then this issue becomes just changing refCount to not volatile?
I think we should leave refCount volatile, but just assign false to a new write-once "isOpen" (that's not volatile) when refCount drops to false.
> Don't use ensureOpen() excessively in IndexReader and IndexWriter
> -----------------------------------------------------------------
>
> Key: LUCENE-1707
> URL: https://issues.apache.org/jira/browse/LUCENE-1707
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Shai Erera
> Fix For: 2.9
>
> Attachments: LUCENE-1707.patch
>
>
> A spin off from here: http://www.nabble.com/Excessive-use-of-ensureOpen()-td24127806.html.
> We should stop calling this method when it's not necessary for any internal Lucene code. Currently, this code seems to hurt properly written apps, unnecessarily.
> Will post a patch soon
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1707) Don't use ensureOpen() excessively
in IndexReader and IndexWriter
Posted by "Shai Erera (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12724118#action_12724118 ]
Shai Erera commented on LUCENE-1707:
------------------------------------
I tried using closed, but TestIndexReaderReopen fails. It appears that closed is used mainly to not execute close() twice (i.e., after you call it once, calling it again will do nothing) on the same instance. Even after I set closed to false wherever it should be (wherever refCount is increased or set to greater than 0), and set it to true in decRef()'s "if (refCount == 1)" section, the test fails. If I come up w/ isOpen, the test passes.
I think it's because IndexWriter.ReadersPool closes a reader, and the test then attempts to close the reader again, then comparing the ref counts. So I'm not sure closed can be used.
> Don't use ensureOpen() excessively in IndexReader and IndexWriter
> -----------------------------------------------------------------
>
> Key: LUCENE-1707
> URL: https://issues.apache.org/jira/browse/LUCENE-1707
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Shai Erera
> Fix For: 2.9
>
> Attachments: LUCENE-1707.patch
>
>
> A spin off from here: http://www.nabble.com/Excessive-use-of-ensureOpen()-td24127806.html.
> We should stop calling this method when it's not necessary for any internal Lucene code. Currently, this code seems to hurt properly written apps, unnecessarily.
> Will post a patch soon
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1707) Don't use ensureOpen() excessively
in IndexReader and IndexWriter
Posted by "Shai Erera (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12724169#action_12724169 ]
Shai Erera commented on LUCENE-1707:
------------------------------------
bq. Patch looks good; thanks Shai!
Note that decRef() sets closed=true if refCount = 0. This will protect against the other bug (calling close() then decRef() or vice versa). So if you don't want it there, we should remove it before committing.
bq. so I think we should make it un-volatile; I'll change it before committing.
If we change refCount to not volatile, is this patch still needed? I.e., what's wrong w/ checking in ensureOpen if refCount <= 0, same as before?
bq. But too many calls to decRef is a real bug and I think we shouldn't mask/delay it.
That would not be possible, since decRef() asserts refCount > 0 (and asserts are enabled in tests) and calls ensureOpen(). So calling decRef() twice in a raw would hit either an assert exception (tests) or ACE (assertions disabled). I think?
> Don't use ensureOpen() excessively in IndexReader and IndexWriter
> -----------------------------------------------------------------
>
> Key: LUCENE-1707
> URL: https://issues.apache.org/jira/browse/LUCENE-1707
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Shai Erera
> Fix For: 2.9
>
> Attachments: LUCENE-1707.patch, LUCENE-1707.patch
>
>
> A spin off from here: http://www.nabble.com/Excessive-use-of-ensureOpen()-td24127806.html.
> We should stop calling this method when it's not necessary for any internal Lucene code. Currently, this code seems to hurt properly written apps, unnecessarily.
> Will post a patch soon
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1707) Don't use ensureOpen() excessively
in IndexReader and IndexWriter
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12724254#action_12724254 ]
Michael McCandless commented on LUCENE-1707:
--------------------------------------------
Alas I don't think we should do that. Eg if an app over-decRef's and refCount becomes -1 but doesn't trip the assert because assertions are off, we want to still hit the ACE.
> Don't use ensureOpen() excessively in IndexReader and IndexWriter
> -----------------------------------------------------------------
>
> Key: LUCENE-1707
> URL: https://issues.apache.org/jira/browse/LUCENE-1707
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Shai Erera
> Fix For: 2.9
>
> Attachments: LUCENE-1707.patch, LUCENE-1707.patch
>
>
> A spin off from here: http://www.nabble.com/Excessive-use-of-ensureOpen()-td24127806.html.
> We should stop calling this method when it's not necessary for any internal Lucene code. Currently, this code seems to hurt properly written apps, unnecessarily.
> Will post a patch soon
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1707) Don't use ensureOpen() excessively
in IndexReader and IndexWriter
Posted by "Yonik Seeley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12724283#action_12724283 ]
Yonik Seeley commented on LUCENE-1707:
--------------------------------------
I also just double-checked and confirmed that the macro-op fusion in intel's latest processors treats CMP followed by JNE/JE the same as if followed by JA/JAE/JB/JBE.
> Don't use ensureOpen() excessively in IndexReader and IndexWriter
> -----------------------------------------------------------------
>
> Key: LUCENE-1707
> URL: https://issues.apache.org/jira/browse/LUCENE-1707
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Shai Erera
> Fix For: 2.9
>
> Attachments: LUCENE-1707.patch, LUCENE-1707.patch
>
>
> A spin off from here: http://www.nabble.com/Excessive-use-of-ensureOpen()-td24127806.html.
> We should stop calling this method when it's not necessary for any internal Lucene code. Currently, this code seems to hurt properly written apps, unnecessarily.
> Will post a patch soon
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1707) Don't use ensureOpen() excessively
in IndexReader and IndexWriter
Posted by "Yonik Seeley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12724256#action_12724256 ]
Yonik Seeley commented on LUCENE-1707:
--------------------------------------
bq. Do you think that changing ensureOpen() to check "if (refCount == 0)" will speed things up a bit?
Nope - not on any common CPUs. There is normally a test instruction which sets a bunch of status flags - branching on those status flags for less-or-equal is the same exact cycle count as equal.
> Don't use ensureOpen() excessively in IndexReader and IndexWriter
> -----------------------------------------------------------------
>
> Key: LUCENE-1707
> URL: https://issues.apache.org/jira/browse/LUCENE-1707
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Shai Erera
> Fix For: 2.9
>
> Attachments: LUCENE-1707.patch, LUCENE-1707.patch
>
>
> A spin off from here: http://www.nabble.com/Excessive-use-of-ensureOpen()-td24127806.html.
> We should stop calling this method when it's not necessary for any internal Lucene code. Currently, this code seems to hurt properly written apps, unnecessarily.
> Will post a patch soon
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1707) Don't use ensureOpen() excessively
in IndexReader and IndexWriter
Posted by "Shai Erera (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12726475#action_12726475 ]
Shai Erera commented on LUCENE-1707:
------------------------------------
so do we need to change those back to not eol-style:native? I know Mike updated a lot of files recently to eol-style:native (I think one of those two was updated as well). Why doesn't Hudson encounter this problem? I've commented out the .txt line in my SVN config. Previously I didn't have it, but I followed a page from Solr wiki (http://wiki.apache.org/solr/CommitterInfo#head-849f78497222f424339b79417056f4e510349fcb) which recommended to put it.
I'll try to checkout again, though I'm not sure if that will help. I think we need to revert the change that was done on r787424. I see that on r153431 (Feb 2005) the comment is "change eol style to LF" - maybe it was for a reason? :)
> Don't use ensureOpen() excessively in IndexReader and IndexWriter
> -----------------------------------------------------------------
>
> Key: LUCENE-1707
> URL: https://issues.apache.org/jira/browse/LUCENE-1707
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Shai Erera
> Fix For: 2.9
>
> Attachments: LUCENE-1707.patch, LUCENE-1707.patch
>
>
> A spin off from here: http://www.nabble.com/Excessive-use-of-ensureOpen()-td24127806.html.
> We should stop calling this method when it's not necessary for any internal Lucene code. Currently, this code seems to hurt properly written apps, unnecessarily.
> Will post a patch soon
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1707) Don't use ensureOpen() excessively
in IndexReader and IndexWriter
Posted by "Shai Erera (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12724251#action_12724251 ]
Shai Erera commented on LUCENE-1707:
------------------------------------
Do you think that changing ensureOpen() to check "if (refCount == 0)" will speed things up a bit?
> Don't use ensureOpen() excessively in IndexReader and IndexWriter
> -----------------------------------------------------------------
>
> Key: LUCENE-1707
> URL: https://issues.apache.org/jira/browse/LUCENE-1707
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Shai Erera
> Fix For: 2.9
>
> Attachments: LUCENE-1707.patch, LUCENE-1707.patch
>
>
> A spin off from here: http://www.nabble.com/Excessive-use-of-ensureOpen()-td24127806.html.
> We should stop calling this method when it's not necessary for any internal Lucene code. Currently, this code seems to hurt properly written apps, unnecessarily.
> Will post a patch soon
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org