You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Michael McCandless (JIRA)" <ji...@apache.org> on 2008/02/25 22:26:51 UTC

[jira] Created: (LUCENE-1191) If IndexWriter hits OutOfMemoryError it should not commit

If IndexWriter hits OutOfMemoryError it should not commit
---------------------------------------------------------

                 Key: LUCENE-1191
                 URL: https://issues.apache.org/jira/browse/LUCENE-1191
             Project: Lucene - Java
          Issue Type: Improvement
          Components: Index
    Affects Versions: 2.3.1, 2.3, 2.2, 2.1, 2.0.0, 1.9, 2.4
            Reporter: Michael McCandless
            Assignee: Michael McCandless
            Priority: Minor
             Fix For: 2.4
         Attachments: LUCENE-1191.patch

While progress has been made making IndexWriter robust to OOME, I
think there is still a real risk that an OOME at a bad time could put
IndexWriter into a bad state such that if close() is called and
somehow it succeeds without hitting another OOME, it risks
introducing messing up the index.

I'd like to detect if OOME has been hit in any of the methods that
alter IW's state, and if so, do not commit changes to the index.  If
close is called after hitting OOME, I think writer should instead
abort.

Attached patch just adds try/catch clauses to catch OOME, note that
it was hit, and re-throw it.  Then, sync() refuses to commit a new
segments_N if OOME was hit, and close instead calls abort when OOME
was hit.  All tests pass.  I plan to commit in a day or two.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


[jira] Commented: (LUCENE-1191) If IndexWriter hits OutOfMemoryError it should not commit

Posted by "Karl Wettin (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/LUCENE-1191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12572265#action_12572265 ] 

Karl Wettin commented on LUCENE-1191:
-------------------------------------

I think I hit that problem this weekend. One of my writer threads died due to an OOME, but the other threads kept going as memory was released from the dying thread. I then hit this SegmentMerger#appendPostings exception each flush: 

{code}
        if (doc < 0 || (df > 0 && doc <= lastDoc))
          throw new CorruptIndexException("docs out of order (" + doc +
              " <= " + lastDoc + " )");
{code}

> If IndexWriter hits OutOfMemoryError it should not commit
> ---------------------------------------------------------
>
>                 Key: LUCENE-1191
>                 URL: https://issues.apache.org/jira/browse/LUCENE-1191
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Index
>    Affects Versions: 1.9, 2.0.0, 2.1, 2.2, 2.3, 2.3.1, 2.4
>            Reporter: Michael McCandless
>            Assignee: Michael McCandless
>            Priority: Minor
>             Fix For: 2.4
>
>         Attachments: LUCENE-1191.patch
>
>
> While progress has been made making IndexWriter robust to OOME, I
> think there is still a real risk that an OOME at a bad time could put
> IndexWriter into a bad state such that if close() is called and
> somehow it succeeds without hitting another OOME, it risks
> introducing messing up the index.
> I'd like to detect if OOME has been hit in any of the methods that
> alter IW's state, and if so, do not commit changes to the index.  If
> close is called after hitting OOME, I think writer should instead
> abort.
> Attached patch just adds try/catch clauses to catch OOME, note that
> it was hit, and re-throw it.  Then, sync() refuses to commit a new
> segments_N if OOME was hit, and close instead calls abort when OOME
> was hit.  All tests pass.  I plan to commit in a day or two.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


[jira] Commented: (LUCENE-1191) If IndexWriter hits OutOfMemoryError it should not commit

Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/LUCENE-1191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12572277#action_12572277 ] 

Michael McCandless commented on LUCENE-1191:
--------------------------------------------

OK.  LUCENE-1171 fixed certain OOM cases after 2.3.0, and was backported to 2.3.1, and it may have fixed this although I didn't think there were cases that would lead to corruption of the posting lists.  If you also see this happen on 2.3.1 please report back!

> If IndexWriter hits OutOfMemoryError it should not commit
> ---------------------------------------------------------
>
>                 Key: LUCENE-1191
>                 URL: https://issues.apache.org/jira/browse/LUCENE-1191
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Index
>    Affects Versions: 1.9, 2.0.0, 2.1, 2.2, 2.3, 2.3.1, 2.4
>            Reporter: Michael McCandless
>            Assignee: Michael McCandless
>            Priority: Minor
>             Fix For: 2.4
>
>         Attachments: LUCENE-1191.patch
>
>
> While progress has been made making IndexWriter robust to OOME, I
> think there is still a real risk that an OOME at a bad time could put
> IndexWriter into a bad state such that if close() is called and
> somehow it succeeds without hitting another OOME, it risks
> introducing messing up the index.
> I'd like to detect if OOME has been hit in any of the methods that
> alter IW's state, and if so, do not commit changes to the index.  If
> close is called after hitting OOME, I think writer should instead
> abort.
> Attached patch just adds try/catch clauses to catch OOME, note that
> it was hit, and re-throw it.  Then, sync() refuses to commit a new
> segments_N if OOME was hit, and close instead calls abort when OOME
> was hit.  All tests pass.  I plan to commit in a day or two.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


[jira] Commented: (LUCENE-1191) If IndexWriter hits OutOfMemoryError it should not commit

Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/LUCENE-1191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12572270#action_12572270 ] 

Michael McCandless commented on LUCENE-1191:
--------------------------------------------

Karl, was that with trunk, 2.3.0 or 2.3.1?

> If IndexWriter hits OutOfMemoryError it should not commit
> ---------------------------------------------------------
>
>                 Key: LUCENE-1191
>                 URL: https://issues.apache.org/jira/browse/LUCENE-1191
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Index
>    Affects Versions: 1.9, 2.0.0, 2.1, 2.2, 2.3, 2.3.1, 2.4
>            Reporter: Michael McCandless
>            Assignee: Michael McCandless
>            Priority: Minor
>             Fix For: 2.4
>
>         Attachments: LUCENE-1191.patch
>
>
> While progress has been made making IndexWriter robust to OOME, I
> think there is still a real risk that an OOME at a bad time could put
> IndexWriter into a bad state such that if close() is called and
> somehow it succeeds without hitting another OOME, it risks
> introducing messing up the index.
> I'd like to detect if OOME has been hit in any of the methods that
> alter IW's state, and if so, do not commit changes to the index.  If
> close is called after hitting OOME, I think writer should instead
> abort.
> Attached patch just adds try/catch clauses to catch OOME, note that
> it was hit, and re-throw it.  Then, sync() refuses to commit a new
> segments_N if OOME was hit, and close instead calls abort when OOME
> was hit.  All tests pass.  I plan to commit in a day or two.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


[jira] Resolved: (LUCENE-1191) If IndexWriter hits OutOfMemoryError it should not commit

Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/LUCENE-1191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Michael McCandless resolved LUCENE-1191.
----------------------------------------

    Resolution: Fixed

> If IndexWriter hits OutOfMemoryError it should not commit
> ---------------------------------------------------------
>
>                 Key: LUCENE-1191
>                 URL: https://issues.apache.org/jira/browse/LUCENE-1191
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Index
>    Affects Versions: 1.9, 2.0.0, 2.1, 2.2, 2.3, 2.3.1, 2.4
>            Reporter: Michael McCandless
>            Assignee: Michael McCandless
>            Priority: Minor
>             Fix For: 2.4
>
>         Attachments: LUCENE-1191.patch
>
>
> While progress has been made making IndexWriter robust to OOME, I
> think there is still a real risk that an OOME at a bad time could put
> IndexWriter into a bad state such that if close() is called and
> somehow it succeeds without hitting another OOME, it risks
> introducing messing up the index.
> I'd like to detect if OOME has been hit in any of the methods that
> alter IW's state, and if so, do not commit changes to the index.  If
> close is called after hitting OOME, I think writer should instead
> abort.
> Attached patch just adds try/catch clauses to catch OOME, note that
> it was hit, and re-throw it.  Then, sync() refuses to commit a new
> segments_N if OOME was hit, and close instead calls abort when OOME
> was hit.  All tests pass.  I plan to commit in a day or two.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


[jira] Updated: (LUCENE-1191) If IndexWriter hits OutOfMemoryError it should not commit

Posted by "Hoss Man (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/LUCENE-1191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hoss Man updated LUCENE-1191:
-----------------------------

    Fix Version/s: 2.3.2

targeted for 2.3.2 bug fix release

> If IndexWriter hits OutOfMemoryError it should not commit
> ---------------------------------------------------------
>
>                 Key: LUCENE-1191
>                 URL: https://issues.apache.org/jira/browse/LUCENE-1191
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Index
>    Affects Versions: 1.9, 2.0.0, 2.1, 2.2, 2.3, 2.3.1, 2.4
>            Reporter: Michael McCandless
>            Assignee: Michael McCandless
>            Priority: Minor
>             Fix For: 2.3.2, 2.4
>
>         Attachments: LUCENE-1191.patch
>
>
> While progress has been made making IndexWriter robust to OOME, I
> think there is still a real risk that an OOME at a bad time could put
> IndexWriter into a bad state such that if close() is called and
> somehow it succeeds without hitting another OOME, it risks
> introducing messing up the index.
> I'd like to detect if OOME has been hit in any of the methods that
> alter IW's state, and if so, do not commit changes to the index.  If
> close is called after hitting OOME, I think writer should instead
> abort.
> Attached patch just adds try/catch clauses to catch OOME, note that
> it was hit, and re-throw it.  Then, sync() refuses to commit a new
> segments_N if OOME was hit, and close instead calls abort when OOME
> was hit.  All tests pass.  I plan to commit in a day or two.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


[jira] Commented: (LUCENE-1191) If IndexWriter hits OutOfMemoryError it should not commit

Posted by "Karl Wettin (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/LUCENE-1191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12572273#action_12572273 ] 

Karl Wettin commented on LUCENE-1191:
-------------------------------------

2.3.0

> If IndexWriter hits OutOfMemoryError it should not commit
> ---------------------------------------------------------
>
>                 Key: LUCENE-1191
>                 URL: https://issues.apache.org/jira/browse/LUCENE-1191
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Index
>    Affects Versions: 1.9, 2.0.0, 2.1, 2.2, 2.3, 2.3.1, 2.4
>            Reporter: Michael McCandless
>            Assignee: Michael McCandless
>            Priority: Minor
>             Fix For: 2.4
>
>         Attachments: LUCENE-1191.patch
>
>
> While progress has been made making IndexWriter robust to OOME, I
> think there is still a real risk that an OOME at a bad time could put
> IndexWriter into a bad state such that if close() is called and
> somehow it succeeds without hitting another OOME, it risks
> introducing messing up the index.
> I'd like to detect if OOME has been hit in any of the methods that
> alter IW's state, and if so, do not commit changes to the index.  If
> close is called after hitting OOME, I think writer should instead
> abort.
> Attached patch just adds try/catch clauses to catch OOME, note that
> it was hit, and re-throw it.  Then, sync() refuses to commit a new
> segments_N if OOME was hit, and close instead calls abort when OOME
> was hit.  All tests pass.  I plan to commit in a day or two.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


[jira] Updated: (LUCENE-1191) If IndexWriter hits OutOfMemoryError it should not commit

Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/LUCENE-1191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Michael McCandless updated LUCENE-1191:
---------------------------------------

    Attachment: LUCENE-1191.patch

> If IndexWriter hits OutOfMemoryError it should not commit
> ---------------------------------------------------------
>
>                 Key: LUCENE-1191
>                 URL: https://issues.apache.org/jira/browse/LUCENE-1191
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Index
>    Affects Versions: 1.9, 2.0.0, 2.1, 2.2, 2.3, 2.3.1, 2.4
>            Reporter: Michael McCandless
>            Assignee: Michael McCandless
>            Priority: Minor
>             Fix For: 2.4
>
>         Attachments: LUCENE-1191.patch
>
>
> While progress has been made making IndexWriter robust to OOME, I
> think there is still a real risk that an OOME at a bad time could put
> IndexWriter into a bad state such that if close() is called and
> somehow it succeeds without hitting another OOME, it risks
> introducing messing up the index.
> I'd like to detect if OOME has been hit in any of the methods that
> alter IW's state, and if so, do not commit changes to the index.  If
> close is called after hitting OOME, I think writer should instead
> abort.
> Attached patch just adds try/catch clauses to catch OOME, note that
> it was hit, and re-throw it.  Then, sync() refuses to commit a new
> segments_N if OOME was hit, and close instead calls abort when OOME
> was hit.  All tests pass.  I plan to commit in a day or two.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org