You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Shivender Devarakonda (JIRA)" <ji...@apache.org> on 2010/04/07 04:51:33 UTC
[jira] Created: (LUCENE-2376) java.lang.OutOfMemoryError:Java heap
space
java.lang.OutOfMemoryError:Java heap space
------------------------------------------
Key: LUCENE-2376
URL: https://issues.apache.org/jira/browse/LUCENE-2376
Project: Lucene - Java
Issue Type: Bug
Components: Index
Affects Versions: 2.9.1
Environment: Windows
Reporter: Shivender Devarakonda
I see an OutOfMemory error in our product and it is happening when we have some data objects on which we built the index. I see the following OutOfmemory error, this is happening after we call Indexwriter.optimize():
4/06/10 02:03:42.160 PM PDT [ERROR] [Lucene Merge Thread #12] In thread Lucene Merge Thread #12 and the message is org.apache.lucene.index.MergePolicy$MergeException: java.lang.OutOfMemoryError: Java heap space
4/06/10 02:03:42.207 PM PDT [VERBOSE] [Lucene Merge Thread #12] [Manager] Uncaught Exception in thread Lucene Merge Thread #12
org.apache.lucene.index.MergePolicy$MergeException: java.lang.OutOfMemoryError: Java heap space
at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:351)
at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:315)
Caused by: java.lang.OutOfMemoryError: Java heap space
at java.util.HashMap.resize(HashMap.java:462)
at java.util.HashMap.addEntry(HashMap.java:755)
at java.util.HashMap.put(HashMap.java:385)
at org.apache.lucene.index.FieldInfos.addInternal(FieldInfos.java:256)
at org.apache.lucene.index.FieldInfos.read(FieldInfos.java:366)
at org.apache.lucene.index.FieldInfos.<init>(FieldInfos.java:71)
at org.apache.lucene.index.SegmentReader$CoreReaders.<init>(SegmentReader.java:116)
at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:638)
at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:608)
at org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:686)
at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4979)
at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4614)
at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:235)
at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:291)
4/06/10 02:03:42.895 PM PDT [ERROR] this writer hit an OutOfMemoryError; cannot complete optimize
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: [jira] Created: (LUCENE-2376) java.lang.OutOfMemoryError:Java
heap space
Posted by Erick Erickson <er...@gmail.com>.
What kind of JVM settings are you using? Lots of people index lots of
documents
without running into this, can you provide more specifics about your
indexing
settings?
On Tue, Apr 6, 2010 at 10:51 PM, Shivender Devarakonda (JIRA) <
jira@apache.org> wrote:
> java.lang.OutOfMemoryError:Java heap space
> ------------------------------------------
>
> Key: LUCENE-2376
> URL: https://issues.apache.org/jira/browse/LUCENE-2376
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Affects Versions: 2.9.1
> Environment: Windows
> Reporter: Shivender Devarakonda
>
>
> I see an OutOfMemory error in our product and it is happening when we have
> some data objects on which we built the index. I see the following
> OutOfmemory error, this is happening after we call Indexwriter.optimize():
>
>
> 4/06/10 02:03:42.160 PM PDT [ERROR] [Lucene Merge Thread #12] In thread
> Lucene Merge Thread #12 and the message is
> org.apache.lucene.index.MergePolicy$MergeException:
> java.lang.OutOfMemoryError: Java heap space
> 4/06/10 02:03:42.207 PM PDT [VERBOSE] [Lucene Merge Thread #12] [Manager]
> Uncaught Exception in thread Lucene Merge Thread #12
> org.apache.lucene.index.MergePolicy$MergeException:
> java.lang.OutOfMemoryError: Java heap space
> at
> org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:351)
> at
> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:315)
> Caused by: java.lang.OutOfMemoryError: Java heap space
> at java.util.HashMap.resize(HashMap.java:462)
> at java.util.HashMap.addEntry(HashMap.java:755)
> at java.util.HashMap.put(HashMap.java:385)
> at
> org.apache.lucene.index.FieldInfos.addInternal(FieldInfos.java:256)
> at org.apache.lucene.index.FieldInfos.read(FieldInfos.java:366)
> at org.apache.lucene.index.FieldInfos.<init>(FieldInfos.java:71)
> at
> org.apache.lucene.index.SegmentReader$CoreReaders.<init>(SegmentReader.java:116)
> at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:638)
> at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:608)
> at
> org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:686)
> at
> org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4979)
> at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4614)
> at
> org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:235)
> at
> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:291)
> 4/06/10 02:03:42.895 PM PDT [ERROR] this writer hit an OutOfMemoryError;
> cannot complete optimize
>
>
> --
> This message is automatically generated by JIRA.
> -
> You can reply to this email to add a comment to the issue online.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>
[jira] Commented: (LUCENE-2376) java.lang.OutOfMemoryError:Java
heap space
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12854876#action_12854876 ]
Michael McCandless commented on LUCENE-2376:
--------------------------------------------
OK but I suspect the root cause is the same here -- your index seems to have a truly massive number of fields. Can you post the CheckIndex output?
IW re-uses per-field objects internally, so that many docs with the same field can be indexed more efficiently. However, when IW sweeps to free up RAM, if it notices an allocated field object hasn't been used recently, because that field name has not occurred in recently added docs, it frees up that memory and logs that "purge field". So from this output I can see you have at least 43K unique field names.
If you have not disabled norms on these fields you'll certainly run out of memory. Even if you disable norms, Lucene is in general not optimized for a tremendous number of unique fields and you'll likely hit other issues.
> java.lang.OutOfMemoryError:Java heap space
> ------------------------------------------
>
> Key: LUCENE-2376
> URL: https://issues.apache.org/jira/browse/LUCENE-2376
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Affects Versions: 2.9.1
> Environment: Windows
> Reporter: Shivender Devarakonda
> Attachments: InfoStreamOutput.txt
>
>
> I see an OutOfMemory error in our product and it is happening when we have some data objects on which we built the index. I see the following OutOfmemory error, this is happening after we call Indexwriter.optimize():
> 4/06/10 02:03:42.160 PM PDT [ERROR] [Lucene Merge Thread #12] In thread Lucene Merge Thread #12 and the message is org.apache.lucene.index.MergePolicy$MergeException: java.lang.OutOfMemoryError: Java heap space
> 4/06/10 02:03:42.207 PM PDT [VERBOSE] [Lucene Merge Thread #12] [Manager] Uncaught Exception in thread Lucene Merge Thread #12
> org.apache.lucene.index.MergePolicy$MergeException: java.lang.OutOfMemoryError: Java heap space
> at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:351)
> at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:315)
> Caused by: java.lang.OutOfMemoryError: Java heap space
> at java.util.HashMap.resize(HashMap.java:462)
> at java.util.HashMap.addEntry(HashMap.java:755)
> at java.util.HashMap.put(HashMap.java:385)
> at org.apache.lucene.index.FieldInfos.addInternal(FieldInfos.java:256)
> at org.apache.lucene.index.FieldInfos.read(FieldInfos.java:366)
> at org.apache.lucene.index.FieldInfos.<init>(FieldInfos.java:71)
> at org.apache.lucene.index.SegmentReader$CoreReaders.<init>(SegmentReader.java:116)
> at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:638)
> at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:608)
> at org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:686)
> at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4979)
> at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4614)
> at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:235)
> at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:291)
> 4/06/10 02:03:42.895 PM PDT [ERROR] this writer hit an OutOfMemoryError; cannot complete optimize
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2376) java.lang.OutOfMemoryError:Java
heap space
Posted by "Earwin Burrfoot (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12855615#action_12855615 ]
Earwin Burrfoot commented on LUCENE-2376:
-----------------------------------------
A field is basically an index in itself. As such, it has a bunch of bookkeeping data associated with it, which is always loaded into memory, unlike field values, which are mostly kept to disk. Also, like I explained in LUCENE-2361, field names are interned as they take part in string comparisons very often internally.
So creating an insane amount of unique fields is like begging for either OOM or OOPermGen. Can you elaborate, why do you need to store unique number in field name rather then field value?
> java.lang.OutOfMemoryError:Java heap space
> ------------------------------------------
>
> Key: LUCENE-2376
> URL: https://issues.apache.org/jira/browse/LUCENE-2376
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Affects Versions: 2.9.1
> Environment: Windows
> Reporter: Shivender Devarakonda
> Attachments: CheckIndex_JavaHeapOOM.txt, CheckIndex_PermGenSpaceOOM.txt, InfoStreamOutput.txt
>
>
> I see an OutOfMemory error in our product and it is happening when we have some data objects on which we built the index. I see the following OutOfmemory error, this is happening after we call Indexwriter.optimize():
> 4/06/10 02:03:42.160 PM PDT [ERROR] [Lucene Merge Thread #12] In thread Lucene Merge Thread #12 and the message is org.apache.lucene.index.MergePolicy$MergeException: java.lang.OutOfMemoryError: Java heap space
> 4/06/10 02:03:42.207 PM PDT [VERBOSE] [Lucene Merge Thread #12] [Manager] Uncaught Exception in thread Lucene Merge Thread #12
> org.apache.lucene.index.MergePolicy$MergeException: java.lang.OutOfMemoryError: Java heap space
> at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:351)
> at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:315)
> Caused by: java.lang.OutOfMemoryError: Java heap space
> at java.util.HashMap.resize(HashMap.java:462)
> at java.util.HashMap.addEntry(HashMap.java:755)
> at java.util.HashMap.put(HashMap.java:385)
> at org.apache.lucene.index.FieldInfos.addInternal(FieldInfos.java:256)
> at org.apache.lucene.index.FieldInfos.read(FieldInfos.java:366)
> at org.apache.lucene.index.FieldInfos.<init>(FieldInfos.java:71)
> at org.apache.lucene.index.SegmentReader$CoreReaders.<init>(SegmentReader.java:116)
> at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:638)
> at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:608)
> at org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:686)
> at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4979)
> at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4614)
> at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:235)
> at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:291)
> 4/06/10 02:03:42.895 PM PDT [ERROR] this writer hit an OutOfMemoryError; cannot complete optimize
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2376) java.lang.OutOfMemoryError:Java
heap space
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12855548#action_12855548 ]
Michael McCandless commented on LUCENE-2376:
--------------------------------------------
Yes total unique fields are 4 in that case. I suspect it's the fields like Cor414D5120514D475231202020202020204A94834B21C22E6E that are causing your problems.
> java.lang.OutOfMemoryError:Java heap space
> ------------------------------------------
>
> Key: LUCENE-2376
> URL: https://issues.apache.org/jira/browse/LUCENE-2376
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Affects Versions: 2.9.1
> Environment: Windows
> Reporter: Shivender Devarakonda
> Attachments: CheckIndex_JavaHeapOOM.txt, CheckIndex_PermGenSpaceOOM.txt, InfoStreamOutput.txt
>
>
> I see an OutOfMemory error in our product and it is happening when we have some data objects on which we built the index. I see the following OutOfmemory error, this is happening after we call Indexwriter.optimize():
> 4/06/10 02:03:42.160 PM PDT [ERROR] [Lucene Merge Thread #12] In thread Lucene Merge Thread #12 and the message is org.apache.lucene.index.MergePolicy$MergeException: java.lang.OutOfMemoryError: Java heap space
> 4/06/10 02:03:42.207 PM PDT [VERBOSE] [Lucene Merge Thread #12] [Manager] Uncaught Exception in thread Lucene Merge Thread #12
> org.apache.lucene.index.MergePolicy$MergeException: java.lang.OutOfMemoryError: Java heap space
> at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:351)
> at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:315)
> Caused by: java.lang.OutOfMemoryError: Java heap space
> at java.util.HashMap.resize(HashMap.java:462)
> at java.util.HashMap.addEntry(HashMap.java:755)
> at java.util.HashMap.put(HashMap.java:385)
> at org.apache.lucene.index.FieldInfos.addInternal(FieldInfos.java:256)
> at org.apache.lucene.index.FieldInfos.read(FieldInfos.java:366)
> at org.apache.lucene.index.FieldInfos.<init>(FieldInfos.java:71)
> at org.apache.lucene.index.SegmentReader$CoreReaders.<init>(SegmentReader.java:116)
> at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:638)
> at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:608)
> at org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:686)
> at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4979)
> at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4614)
> at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:235)
> at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:291)
> 4/06/10 02:03:42.895 PM PDT [ERROR] this writer hit an OutOfMemoryError; cannot complete optimize
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Issue Comment Edited: (LUCENE-2376)
java.lang.OutOfMemoryError:Java heap space
Posted by "Shivender Devarakonda (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12858047#action_12858047 ]
Shivender Devarakonda edited comment on LUCENE-2376 at 4/16/10 6:54 PM:
------------------------------------------------------------------------
Hi,
I reached to the development team that is actually creating these unique fields. they may revisit their redesign. I have a question what is the entry in the "CheckIndex" output refers to Unique index fields?
Thanks,
Shivender
was (Author: shivenderd):
Hi,
I reached to the development team that is actually creating these unique fields. they may revisit their redesign. I have a question what is the entry in the checkoutput text refers to Unique index fields?
Thanks,
Shivender
> java.lang.OutOfMemoryError:Java heap space
> ------------------------------------------
>
> Key: LUCENE-2376
> URL: https://issues.apache.org/jira/browse/LUCENE-2376
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Affects Versions: 2.9.1
> Environment: Windows
> Reporter: Shivender Devarakonda
> Attachments: CheckIndex_JavaHeapOOM.txt, CheckIndex_PermGenSpaceOOM.txt, InfoStreamOutput.txt
>
>
> I see an OutOfMemory error in our product and it is happening when we have some data objects on which we built the index. I see the following OutOfmemory error, this is happening after we call Indexwriter.optimize():
> 4/06/10 02:03:42.160 PM PDT [ERROR] [Lucene Merge Thread #12] In thread Lucene Merge Thread #12 and the message is org.apache.lucene.index.MergePolicy$MergeException: java.lang.OutOfMemoryError: Java heap space
> 4/06/10 02:03:42.207 PM PDT [VERBOSE] [Lucene Merge Thread #12] [Manager] Uncaught Exception in thread Lucene Merge Thread #12
> org.apache.lucene.index.MergePolicy$MergeException: java.lang.OutOfMemoryError: Java heap space
> at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:351)
> at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:315)
> Caused by: java.lang.OutOfMemoryError: Java heap space
> at java.util.HashMap.resize(HashMap.java:462)
> at java.util.HashMap.addEntry(HashMap.java:755)
> at java.util.HashMap.put(HashMap.java:385)
> at org.apache.lucene.index.FieldInfos.addInternal(FieldInfos.java:256)
> at org.apache.lucene.index.FieldInfos.read(FieldInfos.java:366)
> at org.apache.lucene.index.FieldInfos.<init>(FieldInfos.java:71)
> at org.apache.lucene.index.SegmentReader$CoreReaders.<init>(SegmentReader.java:116)
> at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:638)
> at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:608)
> at org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:686)
> at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4979)
> at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4614)
> at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:235)
> at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:291)
> 4/06/10 02:03:42.895 PM PDT [ERROR] this writer hit an OutOfMemoryError; cannot complete optimize
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2376) java.lang.OutOfMemoryError:Java
heap space
Posted by "Shivender Devarakonda (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12855606#action_12855606 ]
Shivender Devarakonda commented on LUCENE-2376:
-----------------------------------------------
Thanks for your inpput on this.
Could you please assist me in understanding more on this issue?:
For every object on which we index will have one or more CorXXXXXXX entry(XXXXX is a uniquenumber), Example:
Object 1's document will have " field A" "Cor12345" "Cor2345"
Object 2's documebt will hava "field A" "Cor4567" "Cor8902"
and so on...
These COrXXX are unique fields for each document so does that mean each document will contain empty holders or holes for the other fields that exists in other object's document but not in it's document?
How is that becoming reason for OOM?
Thanks,
Shivender
> java.lang.OutOfMemoryError:Java heap space
> ------------------------------------------
>
> Key: LUCENE-2376
> URL: https://issues.apache.org/jira/browse/LUCENE-2376
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Affects Versions: 2.9.1
> Environment: Windows
> Reporter: Shivender Devarakonda
> Attachments: CheckIndex_JavaHeapOOM.txt, CheckIndex_PermGenSpaceOOM.txt, InfoStreamOutput.txt
>
>
> I see an OutOfMemory error in our product and it is happening when we have some data objects on which we built the index. I see the following OutOfmemory error, this is happening after we call Indexwriter.optimize():
> 4/06/10 02:03:42.160 PM PDT [ERROR] [Lucene Merge Thread #12] In thread Lucene Merge Thread #12 and the message is org.apache.lucene.index.MergePolicy$MergeException: java.lang.OutOfMemoryError: Java heap space
> 4/06/10 02:03:42.207 PM PDT [VERBOSE] [Lucene Merge Thread #12] [Manager] Uncaught Exception in thread Lucene Merge Thread #12
> org.apache.lucene.index.MergePolicy$MergeException: java.lang.OutOfMemoryError: Java heap space
> at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:351)
> at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:315)
> Caused by: java.lang.OutOfMemoryError: Java heap space
> at java.util.HashMap.resize(HashMap.java:462)
> at java.util.HashMap.addEntry(HashMap.java:755)
> at java.util.HashMap.put(HashMap.java:385)
> at org.apache.lucene.index.FieldInfos.addInternal(FieldInfos.java:256)
> at org.apache.lucene.index.FieldInfos.read(FieldInfos.java:366)
> at org.apache.lucene.index.FieldInfos.<init>(FieldInfos.java:71)
> at org.apache.lucene.index.SegmentReader$CoreReaders.<init>(SegmentReader.java:116)
> at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:638)
> at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:608)
> at org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:686)
> at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4979)
> at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4614)
> at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:235)
> at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:291)
> 4/06/10 02:03:42.895 PM PDT [ERROR] this writer hit an OutOfMemoryError; cannot complete optimize
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2376) java.lang.OutOfMemoryError:Java
heap space
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12854398#action_12854398 ]
Michael McCandless commented on LUCENE-2376:
--------------------------------------------
Is this the same issue as LUCENE-2361? Or is something different here? Are you using the same settings in IndexWriter?
The infoStream output is now much more complete -- eg I see where the OOME happens at the end.
It does look like the cause here is an insane number of fields:
{code}
purge field=Cor414D5120514D475231202020202020204A94834B21C22E6B
purge field=Cor414D5120514D475231202020202020204A94834B21C22E6C
purge field=Cor414D5120514D475231202020202020204A94834B21C22E6D
purge field=Cor414D5120514D475231202020202020204A94834B21C22E6E
purge field=Cor414D5120514D475231202020202020204A94834B21C22E70
purge field=Cor414D5120514D475231202020202020204A94834B21C22E72
{code}
I count 43K purge fields lines in your log (and that's an undercount since you've pruned some). Are you disabling norms for these fields (if not, this will certainly lead to OOME)? In your design, is there any limit to the number of fields? Or will each document have a new field name?
Can you run CheckIndex on the index and post the output?
> java.lang.OutOfMemoryError:Java heap space
> ------------------------------------------
>
> Key: LUCENE-2376
> URL: https://issues.apache.org/jira/browse/LUCENE-2376
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Affects Versions: 2.9.1
> Environment: Windows
> Reporter: Shivender Devarakonda
> Attachments: InfoStreamOutput.txt
>
>
> I see an OutOfMemory error in our product and it is happening when we have some data objects on which we built the index. I see the following OutOfmemory error, this is happening after we call Indexwriter.optimize():
> 4/06/10 02:03:42.160 PM PDT [ERROR] [Lucene Merge Thread #12] In thread Lucene Merge Thread #12 and the message is org.apache.lucene.index.MergePolicy$MergeException: java.lang.OutOfMemoryError: Java heap space
> 4/06/10 02:03:42.207 PM PDT [VERBOSE] [Lucene Merge Thread #12] [Manager] Uncaught Exception in thread Lucene Merge Thread #12
> org.apache.lucene.index.MergePolicy$MergeException: java.lang.OutOfMemoryError: Java heap space
> at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:351)
> at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:315)
> Caused by: java.lang.OutOfMemoryError: Java heap space
> at java.util.HashMap.resize(HashMap.java:462)
> at java.util.HashMap.addEntry(HashMap.java:755)
> at java.util.HashMap.put(HashMap.java:385)
> at org.apache.lucene.index.FieldInfos.addInternal(FieldInfos.java:256)
> at org.apache.lucene.index.FieldInfos.read(FieldInfos.java:366)
> at org.apache.lucene.index.FieldInfos.<init>(FieldInfos.java:71)
> at org.apache.lucene.index.SegmentReader$CoreReaders.<init>(SegmentReader.java:116)
> at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:638)
> at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:608)
> at org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:686)
> at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4979)
> at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4614)
> at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:235)
> at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:291)
> 4/06/10 02:03:42.895 PM PDT [ERROR] this writer hit an OutOfMemoryError; cannot complete optimize
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2376) java.lang.OutOfMemoryError:Java
heap space
Posted by "Uwe Schindler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12854396#action_12854396 ]
Uwe Schindler commented on LUCENE-2376:
---------------------------------------
You mean insane amount of fields with norms...?
> java.lang.OutOfMemoryError:Java heap space
> ------------------------------------------
>
> Key: LUCENE-2376
> URL: https://issues.apache.org/jira/browse/LUCENE-2376
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Affects Versions: 2.9.1
> Environment: Windows
> Reporter: Shivender Devarakonda
> Attachments: InfoStreamOutput.txt
>
>
> I see an OutOfMemory error in our product and it is happening when we have some data objects on which we built the index. I see the following OutOfmemory error, this is happening after we call Indexwriter.optimize():
> 4/06/10 02:03:42.160 PM PDT [ERROR] [Lucene Merge Thread #12] In thread Lucene Merge Thread #12 and the message is org.apache.lucene.index.MergePolicy$MergeException: java.lang.OutOfMemoryError: Java heap space
> 4/06/10 02:03:42.207 PM PDT [VERBOSE] [Lucene Merge Thread #12] [Manager] Uncaught Exception in thread Lucene Merge Thread #12
> org.apache.lucene.index.MergePolicy$MergeException: java.lang.OutOfMemoryError: Java heap space
> at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:351)
> at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:315)
> Caused by: java.lang.OutOfMemoryError: Java heap space
> at java.util.HashMap.resize(HashMap.java:462)
> at java.util.HashMap.addEntry(HashMap.java:755)
> at java.util.HashMap.put(HashMap.java:385)
> at org.apache.lucene.index.FieldInfos.addInternal(FieldInfos.java:256)
> at org.apache.lucene.index.FieldInfos.read(FieldInfos.java:366)
> at org.apache.lucene.index.FieldInfos.<init>(FieldInfos.java:71)
> at org.apache.lucene.index.SegmentReader$CoreReaders.<init>(SegmentReader.java:116)
> at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:638)
> at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:608)
> at org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:686)
> at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4979)
> at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4614)
> at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:235)
> at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:291)
> 4/06/10 02:03:42.895 PM PDT [ERROR] this writer hit an OutOfMemoryError; cannot complete optimize
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2376) java.lang.OutOfMemoryError:Java
heap space
Posted by "Shivender Devarakonda (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12858047#action_12858047 ]
Shivender Devarakonda commented on LUCENE-2376:
-----------------------------------------------
Hi,
I reached to the development team that is actually creating these unique fields. they may revisit their redesign. I have a question what is the entry in the checkoutput text refers to Unique index fields?
Thanks,
Shivender
> java.lang.OutOfMemoryError:Java heap space
> ------------------------------------------
>
> Key: LUCENE-2376
> URL: https://issues.apache.org/jira/browse/LUCENE-2376
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Affects Versions: 2.9.1
> Environment: Windows
> Reporter: Shivender Devarakonda
> Attachments: CheckIndex_JavaHeapOOM.txt, CheckIndex_PermGenSpaceOOM.txt, InfoStreamOutput.txt
>
>
> I see an OutOfMemory error in our product and it is happening when we have some data objects on which we built the index. I see the following OutOfmemory error, this is happening after we call Indexwriter.optimize():
> 4/06/10 02:03:42.160 PM PDT [ERROR] [Lucene Merge Thread #12] In thread Lucene Merge Thread #12 and the message is org.apache.lucene.index.MergePolicy$MergeException: java.lang.OutOfMemoryError: Java heap space
> 4/06/10 02:03:42.207 PM PDT [VERBOSE] [Lucene Merge Thread #12] [Manager] Uncaught Exception in thread Lucene Merge Thread #12
> org.apache.lucene.index.MergePolicy$MergeException: java.lang.OutOfMemoryError: Java heap space
> at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:351)
> at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:315)
> Caused by: java.lang.OutOfMemoryError: Java heap space
> at java.util.HashMap.resize(HashMap.java:462)
> at java.util.HashMap.addEntry(HashMap.java:755)
> at java.util.HashMap.put(HashMap.java:385)
> at org.apache.lucene.index.FieldInfos.addInternal(FieldInfos.java:256)
> at org.apache.lucene.index.FieldInfos.read(FieldInfos.java:366)
> at org.apache.lucene.index.FieldInfos.<init>(FieldInfos.java:71)
> at org.apache.lucene.index.SegmentReader$CoreReaders.<init>(SegmentReader.java:116)
> at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:638)
> at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:608)
> at org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:686)
> at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4979)
> at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4614)
> at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:235)
> at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:291)
> 4/06/10 02:03:42.895 PM PDT [ERROR] this writer hit an OutOfMemoryError; cannot complete optimize
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2376) java.lang.OutOfMemoryError:Java
heap space
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12855336#action_12855336 ]
Michael McCandless commented on LUCENE-2376:
--------------------------------------------
Hmm indeed you have a great many unique fields. A number of your segments have 100K-200K unique fields. And it doesn't help that these field names are looong.
I suspect you're going to have to change your design to not create such a huge number of fields.
How does your app use these fields?
> java.lang.OutOfMemoryError:Java heap space
> ------------------------------------------
>
> Key: LUCENE-2376
> URL: https://issues.apache.org/jira/browse/LUCENE-2376
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Affects Versions: 2.9.1
> Environment: Windows
> Reporter: Shivender Devarakonda
> Attachments: CheckIndex_JavaHeapOOM.txt, CheckIndex_PermGenSpaceOOM.txt, InfoStreamOutput.txt
>
>
> I see an OutOfMemory error in our product and it is happening when we have some data objects on which we built the index. I see the following OutOfmemory error, this is happening after we call Indexwriter.optimize():
> 4/06/10 02:03:42.160 PM PDT [ERROR] [Lucene Merge Thread #12] In thread Lucene Merge Thread #12 and the message is org.apache.lucene.index.MergePolicy$MergeException: java.lang.OutOfMemoryError: Java heap space
> 4/06/10 02:03:42.207 PM PDT [VERBOSE] [Lucene Merge Thread #12] [Manager] Uncaught Exception in thread Lucene Merge Thread #12
> org.apache.lucene.index.MergePolicy$MergeException: java.lang.OutOfMemoryError: Java heap space
> at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:351)
> at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:315)
> Caused by: java.lang.OutOfMemoryError: Java heap space
> at java.util.HashMap.resize(HashMap.java:462)
> at java.util.HashMap.addEntry(HashMap.java:755)
> at java.util.HashMap.put(HashMap.java:385)
> at org.apache.lucene.index.FieldInfos.addInternal(FieldInfos.java:256)
> at org.apache.lucene.index.FieldInfos.read(FieldInfos.java:366)
> at org.apache.lucene.index.FieldInfos.<init>(FieldInfos.java:71)
> at org.apache.lucene.index.SegmentReader$CoreReaders.<init>(SegmentReader.java:116)
> at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:638)
> at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:608)
> at org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:686)
> at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4979)
> at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4614)
> at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:235)
> at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:291)
> 4/06/10 02:03:42.895 PM PDT [ERROR] this writer hit an OutOfMemoryError; cannot complete optimize
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2376) java.lang.OutOfMemoryError:Java
heap space
Posted by "Shivender Devarakonda (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12854749#action_12854749 ]
Shivender Devarakonda commented on LUCENE-2376:
-----------------------------------------------
Thanks for looking into this.
This issue is different from LUCENE-2361. The LUCENE-2361 is about the OOM on permgenspace and this issue is about OOM due to Java heapspace.
1 - LUCENE-2361 is happenining when we start ourproduct with the index directory that was already created. After few minutes, the product is throwing OOM-Permgenspace(As I described in the 2361)
2 - This issue happens when we start the product without any index directory. The product has the functionality to rebuild the complete index if index directory does not exist. You can see the infostream output in the attached file, this comes when it tries to index the Objects and we have huge data on which we are indexing.
1 - Can you tell me why do we see "purge field" entries in infostream output?
2 - We always have limited number of field entries to Document but there is a case where we can get different number of fields for different documents, do you think that will cause the problem?
> java.lang.OutOfMemoryError:Java heap space
> ------------------------------------------
>
> Key: LUCENE-2376
> URL: https://issues.apache.org/jira/browse/LUCENE-2376
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Affects Versions: 2.9.1
> Environment: Windows
> Reporter: Shivender Devarakonda
> Attachments: InfoStreamOutput.txt
>
>
> I see an OutOfMemory error in our product and it is happening when we have some data objects on which we built the index. I see the following OutOfmemory error, this is happening after we call Indexwriter.optimize():
> 4/06/10 02:03:42.160 PM PDT [ERROR] [Lucene Merge Thread #12] In thread Lucene Merge Thread #12 and the message is org.apache.lucene.index.MergePolicy$MergeException: java.lang.OutOfMemoryError: Java heap space
> 4/06/10 02:03:42.207 PM PDT [VERBOSE] [Lucene Merge Thread #12] [Manager] Uncaught Exception in thread Lucene Merge Thread #12
> org.apache.lucene.index.MergePolicy$MergeException: java.lang.OutOfMemoryError: Java heap space
> at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:351)
> at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:315)
> Caused by: java.lang.OutOfMemoryError: Java heap space
> at java.util.HashMap.resize(HashMap.java:462)
> at java.util.HashMap.addEntry(HashMap.java:755)
> at java.util.HashMap.put(HashMap.java:385)
> at org.apache.lucene.index.FieldInfos.addInternal(FieldInfos.java:256)
> at org.apache.lucene.index.FieldInfos.read(FieldInfos.java:366)
> at org.apache.lucene.index.FieldInfos.<init>(FieldInfos.java:71)
> at org.apache.lucene.index.SegmentReader$CoreReaders.<init>(SegmentReader.java:116)
> at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:638)
> at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:608)
> at org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:686)
> at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4979)
> at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4614)
> at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:235)
> at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:291)
> 4/06/10 02:03:42.895 PM PDT [ERROR] this writer hit an OutOfMemoryError; cannot complete optimize
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2376) java.lang.OutOfMemoryError:Java
heap space
Posted by "Shivender Devarakonda (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12855496#action_12855496 ]
Shivender Devarakonda commented on LUCENE-2376:
-----------------------------------------------
I have a question on this, if we have two documents in the index:
Document entrry 1 contains "field1" , "field2" "field3"
Document entry 2 contains "field1" , "field2", field4"
Shall I assume that the total unique fields in the index are 4?
We have a set of key and value pairs that we add them as fields to the document. The set of keys must be same in most cases but might be different in some scenarios. I need to see what actually our data contains? I will do some research on that.
Thanks,
Shivender
> java.lang.OutOfMemoryError:Java heap space
> ------------------------------------------
>
> Key: LUCENE-2376
> URL: https://issues.apache.org/jira/browse/LUCENE-2376
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Affects Versions: 2.9.1
> Environment: Windows
> Reporter: Shivender Devarakonda
> Attachments: CheckIndex_JavaHeapOOM.txt, CheckIndex_PermGenSpaceOOM.txt, InfoStreamOutput.txt
>
>
> I see an OutOfMemory error in our product and it is happening when we have some data objects on which we built the index. I see the following OutOfmemory error, this is happening after we call Indexwriter.optimize():
> 4/06/10 02:03:42.160 PM PDT [ERROR] [Lucene Merge Thread #12] In thread Lucene Merge Thread #12 and the message is org.apache.lucene.index.MergePolicy$MergeException: java.lang.OutOfMemoryError: Java heap space
> 4/06/10 02:03:42.207 PM PDT [VERBOSE] [Lucene Merge Thread #12] [Manager] Uncaught Exception in thread Lucene Merge Thread #12
> org.apache.lucene.index.MergePolicy$MergeException: java.lang.OutOfMemoryError: Java heap space
> at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:351)
> at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:315)
> Caused by: java.lang.OutOfMemoryError: Java heap space
> at java.util.HashMap.resize(HashMap.java:462)
> at java.util.HashMap.addEntry(HashMap.java:755)
> at java.util.HashMap.put(HashMap.java:385)
> at org.apache.lucene.index.FieldInfos.addInternal(FieldInfos.java:256)
> at org.apache.lucene.index.FieldInfos.read(FieldInfos.java:366)
> at org.apache.lucene.index.FieldInfos.<init>(FieldInfos.java:71)
> at org.apache.lucene.index.SegmentReader$CoreReaders.<init>(SegmentReader.java:116)
> at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:638)
> at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:608)
> at org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:686)
> at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4979)
> at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4614)
> at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:235)
> at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:291)
> 4/06/10 02:03:42.895 PM PDT [ERROR] this writer hit an OutOfMemoryError; cannot complete optimize
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-2376) java.lang.OutOfMemoryError:Java heap
space
Posted by "Shivender Devarakonda (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Shivender Devarakonda updated LUCENE-2376:
------------------------------------------
Attachment: CheckIndex_PermGenSpaceOOM.txt
If we start our product with already generated index content then we see and permgenspace OOM. I generated the CheckIndex on this index folder.
Please let me know your thoughts on these output files.
> java.lang.OutOfMemoryError:Java heap space
> ------------------------------------------
>
> Key: LUCENE-2376
> URL: https://issues.apache.org/jira/browse/LUCENE-2376
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Affects Versions: 2.9.1
> Environment: Windows
> Reporter: Shivender Devarakonda
> Attachments: CheckIndex_JavaHeapOOM.txt, CheckIndex_PermGenSpaceOOM.txt, InfoStreamOutput.txt
>
>
> I see an OutOfMemory error in our product and it is happening when we have some data objects on which we built the index. I see the following OutOfmemory error, this is happening after we call Indexwriter.optimize():
> 4/06/10 02:03:42.160 PM PDT [ERROR] [Lucene Merge Thread #12] In thread Lucene Merge Thread #12 and the message is org.apache.lucene.index.MergePolicy$MergeException: java.lang.OutOfMemoryError: Java heap space
> 4/06/10 02:03:42.207 PM PDT [VERBOSE] [Lucene Merge Thread #12] [Manager] Uncaught Exception in thread Lucene Merge Thread #12
> org.apache.lucene.index.MergePolicy$MergeException: java.lang.OutOfMemoryError: Java heap space
> at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:351)
> at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:315)
> Caused by: java.lang.OutOfMemoryError: Java heap space
> at java.util.HashMap.resize(HashMap.java:462)
> at java.util.HashMap.addEntry(HashMap.java:755)
> at java.util.HashMap.put(HashMap.java:385)
> at org.apache.lucene.index.FieldInfos.addInternal(FieldInfos.java:256)
> at org.apache.lucene.index.FieldInfos.read(FieldInfos.java:366)
> at org.apache.lucene.index.FieldInfos.<init>(FieldInfos.java:71)
> at org.apache.lucene.index.SegmentReader$CoreReaders.<init>(SegmentReader.java:116)
> at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:638)
> at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:608)
> at org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:686)
> at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4979)
> at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4614)
> at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:235)
> at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:291)
> 4/06/10 02:03:42.895 PM PDT [ERROR] this writer hit an OutOfMemoryError; cannot complete optimize
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-2376) java.lang.OutOfMemoryError:Java heap
space
Posted by "Shivender Devarakonda (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Shivender Devarakonda updated LUCENE-2376:
------------------------------------------
Attachment: CheckIndex_JavaHeapOOM.txt
CheckIndex output for JavaHeapOOM error. As I specified earlier, We saw OOM when it is indexing the data. I ran the checkIndex on the partially generated index folder.
> java.lang.OutOfMemoryError:Java heap space
> ------------------------------------------
>
> Key: LUCENE-2376
> URL: https://issues.apache.org/jira/browse/LUCENE-2376
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Affects Versions: 2.9.1
> Environment: Windows
> Reporter: Shivender Devarakonda
> Attachments: CheckIndex_JavaHeapOOM.txt, InfoStreamOutput.txt
>
>
> I see an OutOfMemory error in our product and it is happening when we have some data objects on which we built the index. I see the following OutOfmemory error, this is happening after we call Indexwriter.optimize():
> 4/06/10 02:03:42.160 PM PDT [ERROR] [Lucene Merge Thread #12] In thread Lucene Merge Thread #12 and the message is org.apache.lucene.index.MergePolicy$MergeException: java.lang.OutOfMemoryError: Java heap space
> 4/06/10 02:03:42.207 PM PDT [VERBOSE] [Lucene Merge Thread #12] [Manager] Uncaught Exception in thread Lucene Merge Thread #12
> org.apache.lucene.index.MergePolicy$MergeException: java.lang.OutOfMemoryError: Java heap space
> at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:351)
> at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:315)
> Caused by: java.lang.OutOfMemoryError: Java heap space
> at java.util.HashMap.resize(HashMap.java:462)
> at java.util.HashMap.addEntry(HashMap.java:755)
> at java.util.HashMap.put(HashMap.java:385)
> at org.apache.lucene.index.FieldInfos.addInternal(FieldInfos.java:256)
> at org.apache.lucene.index.FieldInfos.read(FieldInfos.java:366)
> at org.apache.lucene.index.FieldInfos.<init>(FieldInfos.java:71)
> at org.apache.lucene.index.SegmentReader$CoreReaders.<init>(SegmentReader.java:116)
> at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:638)
> at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:608)
> at org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:686)
> at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4979)
> at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4614)
> at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:235)
> at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:291)
> 4/06/10 02:03:42.895 PM PDT [ERROR] this writer hit an OutOfMemoryError; cannot complete optimize
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2376) java.lang.OutOfMemoryError:Java
heap space
Posted by "Earwin Burrfoot (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12854393#action_12854393 ]
Earwin Burrfoot commented on LUCENE-2376:
-----------------------------------------
That's the duplicate of LUCENE-2361.
It seems to me you guys have a segment with insane amount of fields. Well, or your index is corrupt and this segment just parses as one having insane amount of fields.
> java.lang.OutOfMemoryError:Java heap space
> ------------------------------------------
>
> Key: LUCENE-2376
> URL: https://issues.apache.org/jira/browse/LUCENE-2376
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Affects Versions: 2.9.1
> Environment: Windows
> Reporter: Shivender Devarakonda
> Attachments: InfoStreamOutput.txt
>
>
> I see an OutOfMemory error in our product and it is happening when we have some data objects on which we built the index. I see the following OutOfmemory error, this is happening after we call Indexwriter.optimize():
> 4/06/10 02:03:42.160 PM PDT [ERROR] [Lucene Merge Thread #12] In thread Lucene Merge Thread #12 and the message is org.apache.lucene.index.MergePolicy$MergeException: java.lang.OutOfMemoryError: Java heap space
> 4/06/10 02:03:42.207 PM PDT [VERBOSE] [Lucene Merge Thread #12] [Manager] Uncaught Exception in thread Lucene Merge Thread #12
> org.apache.lucene.index.MergePolicy$MergeException: java.lang.OutOfMemoryError: Java heap space
> at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:351)
> at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:315)
> Caused by: java.lang.OutOfMemoryError: Java heap space
> at java.util.HashMap.resize(HashMap.java:462)
> at java.util.HashMap.addEntry(HashMap.java:755)
> at java.util.HashMap.put(HashMap.java:385)
> at org.apache.lucene.index.FieldInfos.addInternal(FieldInfos.java:256)
> at org.apache.lucene.index.FieldInfos.read(FieldInfos.java:366)
> at org.apache.lucene.index.FieldInfos.<init>(FieldInfos.java:71)
> at org.apache.lucene.index.SegmentReader$CoreReaders.<init>(SegmentReader.java:116)
> at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:638)
> at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:608)
> at org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:686)
> at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4979)
> at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4614)
> at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:235)
> at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:291)
> 4/06/10 02:03:42.895 PM PDT [ERROR] this writer hit an OutOfMemoryError; cannot complete optimize
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-2376) java.lang.OutOfMemoryError:Java heap
space
Posted by "Shivender Devarakonda (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Shivender Devarakonda updated LUCENE-2376:
------------------------------------------
Attachment: InfoStreamOutput.txt
I attached the info stream output from the IndexWriter.
Basically I copied the output from the log file. The actual log contains more "PurgeFiled" statements than what I copied here.
> java.lang.OutOfMemoryError:Java heap space
> ------------------------------------------
>
> Key: LUCENE-2376
> URL: https://issues.apache.org/jira/browse/LUCENE-2376
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Affects Versions: 2.9.1
> Environment: Windows
> Reporter: Shivender Devarakonda
> Attachments: InfoStreamOutput.txt
>
>
> I see an OutOfMemory error in our product and it is happening when we have some data objects on which we built the index. I see the following OutOfmemory error, this is happening after we call Indexwriter.optimize():
> 4/06/10 02:03:42.160 PM PDT [ERROR] [Lucene Merge Thread #12] In thread Lucene Merge Thread #12 and the message is org.apache.lucene.index.MergePolicy$MergeException: java.lang.OutOfMemoryError: Java heap space
> 4/06/10 02:03:42.207 PM PDT [VERBOSE] [Lucene Merge Thread #12] [Manager] Uncaught Exception in thread Lucene Merge Thread #12
> org.apache.lucene.index.MergePolicy$MergeException: java.lang.OutOfMemoryError: Java heap space
> at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:351)
> at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:315)
> Caused by: java.lang.OutOfMemoryError: Java heap space
> at java.util.HashMap.resize(HashMap.java:462)
> at java.util.HashMap.addEntry(HashMap.java:755)
> at java.util.HashMap.put(HashMap.java:385)
> at org.apache.lucene.index.FieldInfos.addInternal(FieldInfos.java:256)
> at org.apache.lucene.index.FieldInfos.read(FieldInfos.java:366)
> at org.apache.lucene.index.FieldInfos.<init>(FieldInfos.java:71)
> at org.apache.lucene.index.SegmentReader$CoreReaders.<init>(SegmentReader.java:116)
> at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:638)
> at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:608)
> at org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:686)
> at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4979)
> at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4614)
> at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:235)
> at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:291)
> 4/06/10 02:03:42.895 PM PDT [ERROR] this writer hit an OutOfMemoryError; cannot complete optimize
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2376) java.lang.OutOfMemoryError:Java
heap space
Posted by "Shivender Devarakonda (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12855306#action_12855306 ]
Shivender Devarakonda commented on LUCENE-2376:
-----------------------------------------------
Please find the attached CheckIndex output when it was run against the index directorries for both permgent and heapOOM issues
> java.lang.OutOfMemoryError:Java heap space
> ------------------------------------------
>
> Key: LUCENE-2376
> URL: https://issues.apache.org/jira/browse/LUCENE-2376
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Affects Versions: 2.9.1
> Environment: Windows
> Reporter: Shivender Devarakonda
> Attachments: CheckIndex_JavaHeapOOM.txt, CheckIndex_PermGenSpaceOOM.txt, InfoStreamOutput.txt
>
>
> I see an OutOfMemory error in our product and it is happening when we have some data objects on which we built the index. I see the following OutOfmemory error, this is happening after we call Indexwriter.optimize():
> 4/06/10 02:03:42.160 PM PDT [ERROR] [Lucene Merge Thread #12] In thread Lucene Merge Thread #12 and the message is org.apache.lucene.index.MergePolicy$MergeException: java.lang.OutOfMemoryError: Java heap space
> 4/06/10 02:03:42.207 PM PDT [VERBOSE] [Lucene Merge Thread #12] [Manager] Uncaught Exception in thread Lucene Merge Thread #12
> org.apache.lucene.index.MergePolicy$MergeException: java.lang.OutOfMemoryError: Java heap space
> at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:351)
> at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:315)
> Caused by: java.lang.OutOfMemoryError: Java heap space
> at java.util.HashMap.resize(HashMap.java:462)
> at java.util.HashMap.addEntry(HashMap.java:755)
> at java.util.HashMap.put(HashMap.java:385)
> at org.apache.lucene.index.FieldInfos.addInternal(FieldInfos.java:256)
> at org.apache.lucene.index.FieldInfos.read(FieldInfos.java:366)
> at org.apache.lucene.index.FieldInfos.<init>(FieldInfos.java:71)
> at org.apache.lucene.index.SegmentReader$CoreReaders.<init>(SegmentReader.java:116)
> at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:638)
> at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:608)
> at org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:686)
> at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4979)
> at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4614)
> at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:235)
> at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:291)
> 4/06/10 02:03:42.895 PM PDT [ERROR] this writer hit an OutOfMemoryError; cannot complete optimize
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org