You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Shivender Devarakonda (JIRA)" <ji...@apache.org> on 2010/04/02 20:40:27 UTC
[jira] Created: (LUCENE-2361) OutOfMemoryException while Indexing
OutOfMemoryException while Indexing
-----------------------------------
Key: LUCENE-2361
URL: https://issues.apache.org/jira/browse/LUCENE-2361
Project: Lucene - Java
Issue Type: Bug
Components: Index
Affects Versions: 2.9.1
Environment: Windows
Reporter: Shivender Devarakonda
Hi,
We use lucene 2.9.1 version.
We see the following OutOfMemory error in our environment, I think This is happening at a significant high load. Have you observed this anytime? Please let me know your thoughts on this.
org.apache.lucene.index.MergePolicy$MergeException: java.lang.OutOfMemoryError: PermGen space
at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:351)
at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:315)
Caused by: java.lang.OutOfMemoryError: PermGen space
at java.lang.String.$$YJP$$intern(Native Method)
at java.lang.String.intern(Unknown Source)
at org.apache.lucene.util.SimpleStringInterner.intern(SimpleStringInterner.java:74)
at org.apache.lucene.util.StringHelper.intern(StringHelper.java:36)
at org.apache.lucene.index.FieldInfos.read(FieldInfos.java:356)
at org.apache.lucene.index.FieldInfos.<init>(FieldInfos.java:71)
at org.apache.lucene.index.SegmentReader$CoreReaders.<init>(SegmentReader.java:116)
at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:638)
at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:608)
at org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:686)
at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4979)
at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4614)
at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:235)
at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:291)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: [jira] Commented: (LUCENE-2361) OutOfMemoryException while
Indexing
Posted by Lars Grote <la...@gmail.com>.
Hi,
are you loading a lot of Classes? I've seen this error "PermGem" when having
a huge classpath.
As far as I know GC is not cleaning this part of Memory.
You can tweak the size like this: -XX:PermSize=64M -XX:MaxPermSize=128M
Not sure if this helps in anyway.
Regards Lars
On Sat, Apr 3, 2010 at 10:31 AM, Michael McCandless (JIRA)
<ji...@apache.org>wrote:
>
> [
> https://issues.apache.org/jira/browse/LUCENE-2361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12853110#action_12853110]
>
> Michael McCandless commented on LUCENE-2361:
> --------------------------------------------
>
> Can you share some details on how you're using Lucene? What's IndexWriters
> ramBufferSizeMB? How much heap are you giving the JVM? Are you calling
> commit (or closing/opening a new IndexWriter) frequently or rarely?
>
> Can you enable IndexWriter's infoStream and post the output?
>
> > OutOfMemoryException while Indexing
> > -----------------------------------
> >
> > Key: LUCENE-2361
> > URL: https://issues.apache.org/jira/browse/LUCENE-2361
> > Project: Lucene - Java
> > Issue Type: Bug
> > Components: Index
> > Affects Versions: 2.9.1
> > Environment: Windows
> > Reporter: Shivender Devarakonda
> >
> > Hi,
> > We use lucene 2.9.1 version.
> > We see the following OutOfMemory error in our environment, I think This
> is happening at a significant high load. Have you observed this anytime?
> Please let me know your thoughts on this.
> > org.apache.lucene.index.MergePolicy$MergeException:
> java.lang.OutOfMemoryError: PermGen space
> > at
> org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:351)
> > at
> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:315)
> > Caused by: java.lang.OutOfMemoryError: PermGen space
> > at java.lang.String.$$YJP$$intern(Native Method)
> > at java.lang.String.intern(Unknown Source)
> > at
> org.apache.lucene.util.SimpleStringInterner.intern(SimpleStringInterner.java:74)
> > at org.apache.lucene.util.StringHelper.intern(StringHelper.java:36)
> > at org.apache.lucene.index.FieldInfos.read(FieldInfos.java:356)
> > at org.apache.lucene.index.FieldInfos.<init>(FieldInfos.java:71)
> > at
> org.apache.lucene.index.SegmentReader$CoreReaders.<init>(SegmentReader.java:116)
> > at
> org.apache.lucene.index.SegmentReader.get(SegmentReader.java:638)
> > at
> org.apache.lucene.index.SegmentReader.get(SegmentReader.java:608)
> > at
> org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:686)
> > at
> org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4979)
> > at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4614)
> > at
> org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:235)
> > at
> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:291)
>
> --
> This message is automatically generated by JIRA.
> -
> You can reply to this email to add a comment to the issue online.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>
[jira] Commented: (LUCENE-2361) OutOfMemoryException while Indexing
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12853811#action_12853811 ]
Michael McCandless commented on LUCENE-2361:
--------------------------------------------
Hmm are you sure you're setting infoStream on all writers you create? The above output just shows an IW that was opened, not used for anything (no deletions, additions), and then closed.
And eg the OOME in the opening here during a merge should result in some details being posted to the infoStream.
> OutOfMemoryException while Indexing
> -----------------------------------
>
> Key: LUCENE-2361
> URL: https://issues.apache.org/jira/browse/LUCENE-2361
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Affects Versions: 2.9.1
> Environment: Windows
> Reporter: Shivender Devarakonda
>
> Hi,
> We use lucene 2.9.1 version.
> We see the following OutOfMemory error in our environment, I think This is happening at a significant high load. Have you observed this anytime? Please let me know your thoughts on this.
> org.apache.lucene.index.MergePolicy$MergeException: java.lang.OutOfMemoryError: PermGen space
> at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:351)
> at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:315)
> Caused by: java.lang.OutOfMemoryError: PermGen space
> at java.lang.String.$$YJP$$intern(Native Method)
> at java.lang.String.intern(Unknown Source)
> at org.apache.lucene.util.SimpleStringInterner.intern(SimpleStringInterner.java:74)
> at org.apache.lucene.util.StringHelper.intern(StringHelper.java:36)
> at org.apache.lucene.index.FieldInfos.read(FieldInfos.java:356)
> at org.apache.lucene.index.FieldInfos.<init>(FieldInfos.java:71)
> at org.apache.lucene.index.SegmentReader$CoreReaders.<init>(SegmentReader.java:116)
> at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:638)
> at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:608)
> at org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:686)
> at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4979)
> at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4614)
> at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:235)
> at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:291)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2361) OutOfMemoryException while Indexing
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12853110#action_12853110 ]
Michael McCandless commented on LUCENE-2361:
--------------------------------------------
Can you share some details on how you're using Lucene? What's IndexWriters ramBufferSizeMB? How much heap are you giving the JVM? Are you calling commit (or closing/opening a new IndexWriter) frequently or rarely?
Can you enable IndexWriter's infoStream and post the output?
> OutOfMemoryException while Indexing
> -----------------------------------
>
> Key: LUCENE-2361
> URL: https://issues.apache.org/jira/browse/LUCENE-2361
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Affects Versions: 2.9.1
> Environment: Windows
> Reporter: Shivender Devarakonda
>
> Hi,
> We use lucene 2.9.1 version.
> We see the following OutOfMemory error in our environment, I think This is happening at a significant high load. Have you observed this anytime? Please let me know your thoughts on this.
> org.apache.lucene.index.MergePolicy$MergeException: java.lang.OutOfMemoryError: PermGen space
> at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:351)
> at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:315)
> Caused by: java.lang.OutOfMemoryError: PermGen space
> at java.lang.String.$$YJP$$intern(Native Method)
> at java.lang.String.intern(Unknown Source)
> at org.apache.lucene.util.SimpleStringInterner.intern(SimpleStringInterner.java:74)
> at org.apache.lucene.util.StringHelper.intern(StringHelper.java:36)
> at org.apache.lucene.index.FieldInfos.read(FieldInfos.java:356)
> at org.apache.lucene.index.FieldInfos.<init>(FieldInfos.java:71)
> at org.apache.lucene.index.SegmentReader$CoreReaders.<init>(SegmentReader.java:116)
> at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:638)
> at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:608)
> at org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:686)
> at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4979)
> at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4614)
> at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:235)
> at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:291)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2361) OutOfMemoryException while Indexing
Posted by "Shivender Devarakonda (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12854091#action_12854091 ]
Shivender Devarakonda commented on LUCENE-2361:
-----------------------------------------------
Yes I set infostream on all writers.
> OutOfMemoryException while Indexing
> -----------------------------------
>
> Key: LUCENE-2361
> URL: https://issues.apache.org/jira/browse/LUCENE-2361
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Affects Versions: 2.9.1
> Environment: Windows
> Reporter: Shivender Devarakonda
>
> Hi,
> We use lucene 2.9.1 version.
> We see the following OutOfMemory error in our environment, I think This is happening at a significant high load. Have you observed this anytime? Please let me know your thoughts on this.
> org.apache.lucene.index.MergePolicy$MergeException: java.lang.OutOfMemoryError: PermGen space
> at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:351)
> at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:315)
> Caused by: java.lang.OutOfMemoryError: PermGen space
> at java.lang.String.$$YJP$$intern(Native Method)
> at java.lang.String.intern(Unknown Source)
> at org.apache.lucene.util.SimpleStringInterner.intern(SimpleStringInterner.java:74)
> at org.apache.lucene.util.StringHelper.intern(StringHelper.java:36)
> at org.apache.lucene.index.FieldInfos.read(FieldInfos.java:356)
> at org.apache.lucene.index.FieldInfos.<init>(FieldInfos.java:71)
> at org.apache.lucene.index.SegmentReader$CoreReaders.<init>(SegmentReader.java:116)
> at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:638)
> at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:608)
> at org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:686)
> at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4979)
> at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4614)
> at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:235)
> at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:291)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2361) OutOfMemoryException while Indexing
Posted by "Earwin Burrfoot (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12852996#action_12852996 ]
Earwin Burrfoot commented on LUCENE-2361:
-----------------------------------------
Also, are you running under profiler? Does that happen without it?
> OutOfMemoryException while Indexing
> -----------------------------------
>
> Key: LUCENE-2361
> URL: https://issues.apache.org/jira/browse/LUCENE-2361
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Affects Versions: 2.9.1
> Environment: Windows
> Reporter: Shivender Devarakonda
>
> Hi,
> We use lucene 2.9.1 version.
> We see the following OutOfMemory error in our environment, I think This is happening at a significant high load. Have you observed this anytime? Please let me know your thoughts on this.
> org.apache.lucene.index.MergePolicy$MergeException: java.lang.OutOfMemoryError: PermGen space
> at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:351)
> at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:315)
> Caused by: java.lang.OutOfMemoryError: PermGen space
> at java.lang.String.$$YJP$$intern(Native Method)
> at java.lang.String.intern(Unknown Source)
> at org.apache.lucene.util.SimpleStringInterner.intern(SimpleStringInterner.java:74)
> at org.apache.lucene.util.StringHelper.intern(StringHelper.java:36)
> at org.apache.lucene.index.FieldInfos.read(FieldInfos.java:356)
> at org.apache.lucene.index.FieldInfos.<init>(FieldInfos.java:71)
> at org.apache.lucene.index.SegmentReader$CoreReaders.<init>(SegmentReader.java:116)
> at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:638)
> at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:608)
> at org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:686)
> at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4979)
> at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4614)
> at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:235)
> at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:291)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Issue Comment Edited: (LUCENE-2361) OutOfMemoryException
while Indexing
Posted by "Shivender Devarakonda (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12853154#action_12853154 ]
Shivender Devarakonda edited comment on LUCENE-2361 at 4/3/10 4:26 PM:
-----------------------------------------------------------------------
Thanks for helping us on this.
We create a Daemon Thread that does the index updating functionality.
1- Create a IndexDir like this:
fDirectoryName = new File(tracesDir, relativeIndexDir);
fIndexDir = FSDirectory.open(fDirectoryName, NoLockFactory.getNoLockFactory());
fIndexDir.setReadChunkSize(8192);
2- The Index inserter thread does the following functionality, The new writer is invoked for a batch size of 100.
IndexWriter writer = null;
try
{
writer = new IndexWriter(fIndexDir, this.getAnalyzer(), !this.indexExists(), IndexWriter.MaxFieldLength.LIMITED);
writer.setMaxBufferedDocs(batchSize);
writer.addDocument(doc);
// while we're open and there's more in the queue, add more
numInserted += addMore(writer, luceneQueue, batchSize);
}
finally
{
Utils.close(writer);
insertionTime = MasterClock.currentTimeMillis() - startTime;
}
We do not set any RAMBufferSize in this case.
Please let me know if you need anything on this.
was (Author: shivenderd):
Thanks for helping us on this.
We create a Daemon Thread that does the index updating functionality.
1- Create a IndexDir like this:
fDirectoryName = new File(tracesDir, relativeIndexDir);
fIndexDir = FSDirectory.open(fDirectoryName, NoLockFactory.getNoLockFactory());
fIndexDir.setReadChunkSize(8192);
2- The Index inserter thread does the following functionality, The new writer is invoked for a batch size of 100.
IndexWriter writer = null;
try
{
writer = new IndexWriter(fIndexDir, this.getAnalyzer(), !this.indexExists(), IndexWriter.MaxFieldLength.LIMITED);
writer.setMaxBufferedDocs(batchSize);
writer.addDocument(doc);
// while we're open and there's more in the queue, add more
numInserted += addMore(writer, luceneQueue, batchSize);
}
finally
{
Utils.close(writer);
insertionTime = MasterClock.currentTimeMillis() - startTime;
}
We do not set any RAMBufferSize in this case.
Please let me know if you need anything on this.
> OutOfMemoryException while Indexing
> -----------------------------------
>
> Key: LUCENE-2361
> URL: https://issues.apache.org/jira/browse/LUCENE-2361
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Affects Versions: 2.9.1
> Environment: Windows
> Reporter: Shivender Devarakonda
>
> Hi,
> We use lucene 2.9.1 version.
> We see the following OutOfMemory error in our environment, I think This is happening at a significant high load. Have you observed this anytime? Please let me know your thoughts on this.
> org.apache.lucene.index.MergePolicy$MergeException: java.lang.OutOfMemoryError: PermGen space
> at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:351)
> at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:315)
> Caused by: java.lang.OutOfMemoryError: PermGen space
> at java.lang.String.$$YJP$$intern(Native Method)
> at java.lang.String.intern(Unknown Source)
> at org.apache.lucene.util.SimpleStringInterner.intern(SimpleStringInterner.java:74)
> at org.apache.lucene.util.StringHelper.intern(StringHelper.java:36)
> at org.apache.lucene.index.FieldInfos.read(FieldInfos.java:356)
> at org.apache.lucene.index.FieldInfos.<init>(FieldInfos.java:71)
> at org.apache.lucene.index.SegmentReader$CoreReaders.<init>(SegmentReader.java:116)
> at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:638)
> at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:608)
> at org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:686)
> at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4979)
> at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4614)
> at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:235)
> at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:291)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2361) OutOfMemoryException while Indexing
Posted by "Shivender Devarakonda (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12853025#action_12853025 ]
Shivender Devarakonda commented on LUCENE-2361:
-----------------------------------------------
I am sure we have less than 20 fields in the index
> OutOfMemoryException while Indexing
> -----------------------------------
>
> Key: LUCENE-2361
> URL: https://issues.apache.org/jira/browse/LUCENE-2361
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Affects Versions: 2.9.1
> Environment: Windows
> Reporter: Shivender Devarakonda
>
> Hi,
> We use lucene 2.9.1 version.
> We see the following OutOfMemory error in our environment, I think This is happening at a significant high load. Have you observed this anytime? Please let me know your thoughts on this.
> org.apache.lucene.index.MergePolicy$MergeException: java.lang.OutOfMemoryError: PermGen space
> at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:351)
> at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:315)
> Caused by: java.lang.OutOfMemoryError: PermGen space
> at java.lang.String.$$YJP$$intern(Native Method)
> at java.lang.String.intern(Unknown Source)
> at org.apache.lucene.util.SimpleStringInterner.intern(SimpleStringInterner.java:74)
> at org.apache.lucene.util.StringHelper.intern(StringHelper.java:36)
> at org.apache.lucene.index.FieldInfos.read(FieldInfos.java:356)
> at org.apache.lucene.index.FieldInfos.<init>(FieldInfos.java:71)
> at org.apache.lucene.index.SegmentReader$CoreReaders.<init>(SegmentReader.java:116)
> at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:638)
> at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:608)
> at org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:686)
> at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4979)
> at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4614)
> at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:235)
> at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:291)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2361) OutOfMemoryException while Indexing
Posted by "Shivender Devarakonda (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12854149#action_12854149 ]
Shivender Devarakonda commented on LUCENE-2361:
-----------------------------------------------
I did not see any infostream output when it had gone OOM. I see the same exception trace that I posted above. Do you think I missed anything?
> OutOfMemoryException while Indexing
> -----------------------------------
>
> Key: LUCENE-2361
> URL: https://issues.apache.org/jira/browse/LUCENE-2361
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Affects Versions: 2.9.1
> Environment: Windows
> Reporter: Shivender Devarakonda
>
> Hi,
> We use lucene 2.9.1 version.
> We see the following OutOfMemory error in our environment, I think This is happening at a significant high load. Have you observed this anytime? Please let me know your thoughts on this.
> org.apache.lucene.index.MergePolicy$MergeException: java.lang.OutOfMemoryError: PermGen space
> at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:351)
> at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:315)
> Caused by: java.lang.OutOfMemoryError: PermGen space
> at java.lang.String.$$YJP$$intern(Native Method)
> at java.lang.String.intern(Unknown Source)
> at org.apache.lucene.util.SimpleStringInterner.intern(SimpleStringInterner.java:74)
> at org.apache.lucene.util.StringHelper.intern(StringHelper.java:36)
> at org.apache.lucene.index.FieldInfos.read(FieldInfos.java:356)
> at org.apache.lucene.index.FieldInfos.<init>(FieldInfos.java:71)
> at org.apache.lucene.index.SegmentReader$CoreReaders.<init>(SegmentReader.java:116)
> at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:638)
> at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:608)
> at org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:686)
> at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4979)
> at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4614)
> at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:235)
> at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:291)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2361) OutOfMemoryException while Indexing
Posted by "Shivender Devarakonda (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12853562#action_12853562 ]
Shivender Devarakonda commented on LUCENE-2361:
-----------------------------------------------
I saw the following infostream ouptut when I started our product but after that there was no infostream output and I saw the same OOM error again. Did I miss anything?
IFD [main]: setInfoStream deletionPolicy=org.apache.lucene.index.KeepOnlyLastCommitDeletionPolicy@1cbf655
IW 0 [main]: setInfoStream: dir=org.apache.lucene.store.SimpleFSDirectory@C:\Program Files (x86)\CA Wily\Introscope9.0d595\traces\index autoCommit=false mergePolicy=org.apache.lucene.index.LogByteSizeMergePolicy@1996d53 mergeScheduler=org.apache.lucene.index.ConcurrentMergeScheduler@1d882b6 ramBufferSizeMB=16.0 maxBufferedDocs=-1 maxBuffereDeleteTerms=-1 maxFieldLength=10000 index=_2r5:c16187 _570:c40197 _7mt:c41371 _9yq:c42466 _cml:c45324 _fgk:c46186 _i63:c45931 _kz8:c46568 _ntr:c47071 _qiq:c46080 _qin:c18 _qip:c1 _qir:c4 _qit:c19 _qiv:c1 _qix:c51 _qiz:c2
IW 0 [main]: now flush at close
IW 0 [main]: flush: segment=null docStoreSegment=null docStoreOffset=0 flushDocs=false flushDeletes=true flushDocStores=false numDocs=0 numBufDelTerms=0
IW 0 [main]: index before flush _2r5:c16187 _570:c40197 _7mt:c41371 _9yq:c42466 _cml:c45324 _fgk:c46186 _i63:c45931 _kz8:c46568 _ntr:c47071 _qiq:c46080 _qin:c18 _qip:c1 _qir:c4 _qit:c19 _qiv:c1 _qix:c51 _qiz:c2
IW 0 [main]: CMS: now merge
IW 0 [main]: CMS: index: _2r5:c16187 _570:c40197 _7mt:c41371 _9yq:c42466 _cml:c45324 _fgk:c46186 _i63:c45931 _kz8:c46568 _ntr:c47071 _qiq:c46080 _qin:c18 _qip:c1 _qir:c4 _qit:c19 _qiv:c1 _qix:c51 _qiz:c2
IW 0 [main]: CMS: no more merges pending; now return
IW 0 [main]: now call final commit()
IW 0 [main]: startCommit(): start sizeInBytes=0
IW 0 [main]: skip startCommit(): no changes pending
IW 0 [main]: commit: pendingCommit == null; skip
IW 0 [main]: commit: done
IW 0 [main]: at close: _2r5:c16187 _570:c40197 _7mt:c41371 _9yq:c42466 _cml:c45324 _fgk:c46186 _i63:c45931 _kz8:c46568 _ntr:c47071 _qiq:c46080 _qin:c18 _qip:c1 _qir:c4 _qit:c19 _qiv:c1 _qix:c51 _qiz:c2
> OutOfMemoryException while Indexing
> -----------------------------------
>
> Key: LUCENE-2361
> URL: https://issues.apache.org/jira/browse/LUCENE-2361
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Affects Versions: 2.9.1
> Environment: Windows
> Reporter: Shivender Devarakonda
>
> Hi,
> We use lucene 2.9.1 version.
> We see the following OutOfMemory error in our environment, I think This is happening at a significant high load. Have you observed this anytime? Please let me know your thoughts on this.
> org.apache.lucene.index.MergePolicy$MergeException: java.lang.OutOfMemoryError: PermGen space
> at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:351)
> at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:315)
> Caused by: java.lang.OutOfMemoryError: PermGen space
> at java.lang.String.$$YJP$$intern(Native Method)
> at java.lang.String.intern(Unknown Source)
> at org.apache.lucene.util.SimpleStringInterner.intern(SimpleStringInterner.java:74)
> at org.apache.lucene.util.StringHelper.intern(StringHelper.java:36)
> at org.apache.lucene.index.FieldInfos.read(FieldInfos.java:356)
> at org.apache.lucene.index.FieldInfos.<init>(FieldInfos.java:71)
> at org.apache.lucene.index.SegmentReader$CoreReaders.<init>(SegmentReader.java:116)
> at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:638)
> at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:608)
> at org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:686)
> at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4979)
> at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4614)
> at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:235)
> at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:291)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2361) OutOfMemoryException while Indexing
Posted by "Shivender Devarakonda (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12853209#action_12853209 ]
Shivender Devarakonda commented on LUCENE-2361:
-----------------------------------------------
1 - Yes it is 100.
2 - I am not sure on this statement, I need to reachout to the original author of this code, will update you once I get a reply from the author. Do you think it will caues any issue?
3- We had some OOM issues realted to SUNJVM issue that is where we have used this call.
4- This issue is not always reproducible, I will post the infostream output once I get it reproduced again.
> OutOfMemoryException while Indexing
> -----------------------------------
>
> Key: LUCENE-2361
> URL: https://issues.apache.org/jira/browse/LUCENE-2361
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Affects Versions: 2.9.1
> Environment: Windows
> Reporter: Shivender Devarakonda
>
> Hi,
> We use lucene 2.9.1 version.
> We see the following OutOfMemory error in our environment, I think This is happening at a significant high load. Have you observed this anytime? Please let me know your thoughts on this.
> org.apache.lucene.index.MergePolicy$MergeException: java.lang.OutOfMemoryError: PermGen space
> at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:351)
> at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:315)
> Caused by: java.lang.OutOfMemoryError: PermGen space
> at java.lang.String.$$YJP$$intern(Native Method)
> at java.lang.String.intern(Unknown Source)
> at org.apache.lucene.util.SimpleStringInterner.intern(SimpleStringInterner.java:74)
> at org.apache.lucene.util.StringHelper.intern(StringHelper.java:36)
> at org.apache.lucene.index.FieldInfos.read(FieldInfos.java:356)
> at org.apache.lucene.index.FieldInfos.<init>(FieldInfos.java:71)
> at org.apache.lucene.index.SegmentReader$CoreReaders.<init>(SegmentReader.java:116)
> at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:638)
> at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:608)
> at org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:686)
> at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4979)
> at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4614)
> at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:235)
> at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:291)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2361) OutOfMemoryException while Indexing
Posted by "Shivender Devarakonda (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12853154#action_12853154 ]
Shivender Devarakonda commented on LUCENE-2361:
-----------------------------------------------
Thanks for helping us on this.
We create a Daemon Thread that does the index updating functionality.
1- Create a IndexDir like this:
fDirectoryName = new File(tracesDir, relativeIndexDir);
fIndexDir = FSDirectory.open(fDirectoryName, NoLockFactory.getNoLockFactory());
fIndexDir.setReadChunkSize(8192);
2- The Index inserter thread does the following functionality, The new writer is invoked for a batch size of 100.
IndexWriter writer = null;
try
{
writer = new IndexWriter(fIndexDir, this.getAnalyzer(), !this.indexExists(), IndexWriter.MaxFieldLength.LIMITED);
writer.setMaxBufferedDocs(batchSize);
writer.addDocument(doc);
// while we're open and there's more in the queue, add more
numInserted += addMore(writer, luceneQueue, batchSize);
}
finally
{
Utils.close(writer);
insertionTime = MasterClock.currentTimeMillis() - startTime;
}
We do not set any RAMBufferSize in this case.
Please let me know if you need anything on this.
> OutOfMemoryException while Indexing
> -----------------------------------
>
> Key: LUCENE-2361
> URL: https://issues.apache.org/jira/browse/LUCENE-2361
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Affects Versions: 2.9.1
> Environment: Windows
> Reporter: Shivender Devarakonda
>
> Hi,
> We use lucene 2.9.1 version.
> We see the following OutOfMemory error in our environment, I think This is happening at a significant high load. Have you observed this anytime? Please let me know your thoughts on this.
> org.apache.lucene.index.MergePolicy$MergeException: java.lang.OutOfMemoryError: PermGen space
> at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:351)
> at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:315)
> Caused by: java.lang.OutOfMemoryError: PermGen space
> at java.lang.String.$$YJP$$intern(Native Method)
> at java.lang.String.intern(Unknown Source)
> at org.apache.lucene.util.SimpleStringInterner.intern(SimpleStringInterner.java:74)
> at org.apache.lucene.util.StringHelper.intern(StringHelper.java:36)
> at org.apache.lucene.index.FieldInfos.read(FieldInfos.java:356)
> at org.apache.lucene.index.FieldInfos.<init>(FieldInfos.java:71)
> at org.apache.lucene.index.SegmentReader$CoreReaders.<init>(SegmentReader.java:116)
> at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:638)
> at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:608)
> at org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:686)
> at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4979)
> at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4614)
> at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:235)
> at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:291)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2361) OutOfMemoryException while Indexing
Posted by "Earwin Burrfoot (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12853033#action_12853033 ]
Earwin Burrfoot commented on LUCENE-2361:
-----------------------------------------
20 is absolutely ok. Does exception always have this stacktrace? Maybe someone else litters in PermGen?
> OutOfMemoryException while Indexing
> -----------------------------------
>
> Key: LUCENE-2361
> URL: https://issues.apache.org/jira/browse/LUCENE-2361
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Affects Versions: 2.9.1
> Environment: Windows
> Reporter: Shivender Devarakonda
>
> Hi,
> We use lucene 2.9.1 version.
> We see the following OutOfMemory error in our environment, I think This is happening at a significant high load. Have you observed this anytime? Please let me know your thoughts on this.
> org.apache.lucene.index.MergePolicy$MergeException: java.lang.OutOfMemoryError: PermGen space
> at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:351)
> at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:315)
> Caused by: java.lang.OutOfMemoryError: PermGen space
> at java.lang.String.$$YJP$$intern(Native Method)
> at java.lang.String.intern(Unknown Source)
> at org.apache.lucene.util.SimpleStringInterner.intern(SimpleStringInterner.java:74)
> at org.apache.lucene.util.StringHelper.intern(StringHelper.java:36)
> at org.apache.lucene.index.FieldInfos.read(FieldInfos.java:356)
> at org.apache.lucene.index.FieldInfos.<init>(FieldInfos.java:71)
> at org.apache.lucene.index.SegmentReader$CoreReaders.<init>(SegmentReader.java:116)
> at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:638)
> at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:608)
> at org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:686)
> at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4979)
> at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4614)
> at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:235)
> at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:291)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2361) OutOfMemoryException while Indexing
Posted by "Earwin Burrfoot (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12853021#action_12853021 ]
Earwin Burrfoot commented on LUCENE-2361:
-----------------------------------------
Basically, my bet is you're using a truckload of unique field names. Lucene interns them for speed, as it does quite a lot of field name comparisons, and the whole thing explodes.
Unless you fill up your PermGen in some other manner, and Lucene just happens to hit the consequences.
The strange thing is - last time I tested String.intern(), it failed to cause OOMs being bombarded by random strings, like it did in java 1.4.something.
> OutOfMemoryException while Indexing
> -----------------------------------
>
> Key: LUCENE-2361
> URL: https://issues.apache.org/jira/browse/LUCENE-2361
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Affects Versions: 2.9.1
> Environment: Windows
> Reporter: Shivender Devarakonda
>
> Hi,
> We use lucene 2.9.1 version.
> We see the following OutOfMemory error in our environment, I think This is happening at a significant high load. Have you observed this anytime? Please let me know your thoughts on this.
> org.apache.lucene.index.MergePolicy$MergeException: java.lang.OutOfMemoryError: PermGen space
> at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:351)
> at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:315)
> Caused by: java.lang.OutOfMemoryError: PermGen space
> at java.lang.String.$$YJP$$intern(Native Method)
> at java.lang.String.intern(Unknown Source)
> at org.apache.lucene.util.SimpleStringInterner.intern(SimpleStringInterner.java:74)
> at org.apache.lucene.util.StringHelper.intern(StringHelper.java:36)
> at org.apache.lucene.index.FieldInfos.read(FieldInfos.java:356)
> at org.apache.lucene.index.FieldInfos.<init>(FieldInfos.java:71)
> at org.apache.lucene.index.SegmentReader$CoreReaders.<init>(SegmentReader.java:116)
> at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:638)
> at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:608)
> at org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:686)
> at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4979)
> at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4614)
> at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:235)
> at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:291)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2361) OutOfMemoryException while Indexing
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12853186#action_12853186 ]
Michael McCandless commented on LUCENE-2361:
--------------------------------------------
OK, thanks. Is batchSize always 100?
Why are you using NoLockFactory? That's generally very dangerous.
Also, why do you call FSDir's setReadChunkSizeMB?
Can you post the infoStream output?
> OutOfMemoryException while Indexing
> -----------------------------------
>
> Key: LUCENE-2361
> URL: https://issues.apache.org/jira/browse/LUCENE-2361
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Affects Versions: 2.9.1
> Environment: Windows
> Reporter: Shivender Devarakonda
>
> Hi,
> We use lucene 2.9.1 version.
> We see the following OutOfMemory error in our environment, I think This is happening at a significant high load. Have you observed this anytime? Please let me know your thoughts on this.
> org.apache.lucene.index.MergePolicy$MergeException: java.lang.OutOfMemoryError: PermGen space
> at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:351)
> at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:315)
> Caused by: java.lang.OutOfMemoryError: PermGen space
> at java.lang.String.$$YJP$$intern(Native Method)
> at java.lang.String.intern(Unknown Source)
> at org.apache.lucene.util.SimpleStringInterner.intern(SimpleStringInterner.java:74)
> at org.apache.lucene.util.StringHelper.intern(StringHelper.java:36)
> at org.apache.lucene.index.FieldInfos.read(FieldInfos.java:356)
> at org.apache.lucene.index.FieldInfos.<init>(FieldInfos.java:71)
> at org.apache.lucene.index.SegmentReader$CoreReaders.<init>(SegmentReader.java:116)
> at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:638)
> at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:608)
> at org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:686)
> at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4979)
> at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4614)
> at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:235)
> at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:291)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2361) OutOfMemoryException while Indexing
Posted by "Earwin Burrfoot (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12852994#action_12852994 ]
Earwin Burrfoot commented on LUCENE-2361:
-----------------------------------------
Which JVM is that?
> OutOfMemoryException while Indexing
> -----------------------------------
>
> Key: LUCENE-2361
> URL: https://issues.apache.org/jira/browse/LUCENE-2361
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Affects Versions: 2.9.1
> Environment: Windows
> Reporter: Shivender Devarakonda
>
> Hi,
> We use lucene 2.9.1 version.
> We see the following OutOfMemory error in our environment, I think This is happening at a significant high load. Have you observed this anytime? Please let me know your thoughts on this.
> org.apache.lucene.index.MergePolicy$MergeException: java.lang.OutOfMemoryError: PermGen space
> at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:351)
> at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:315)
> Caused by: java.lang.OutOfMemoryError: PermGen space
> at java.lang.String.$$YJP$$intern(Native Method)
> at java.lang.String.intern(Unknown Source)
> at org.apache.lucene.util.SimpleStringInterner.intern(SimpleStringInterner.java:74)
> at org.apache.lucene.util.StringHelper.intern(StringHelper.java:36)
> at org.apache.lucene.index.FieldInfos.read(FieldInfos.java:356)
> at org.apache.lucene.index.FieldInfos.<init>(FieldInfos.java:71)
> at org.apache.lucene.index.SegmentReader$CoreReaders.<init>(SegmentReader.java:116)
> at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:638)
> at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:608)
> at org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:686)
> at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4979)
> at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4614)
> at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:235)
> at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:291)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2361) OutOfMemoryException while Indexing
Posted by "Shivender Devarakonda (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12853001#action_12853001 ]
Shivender Devarakonda commented on LUCENE-2361:
-----------------------------------------------
java 6 update 18. Yes it happens without the profiler also.
Thanks
> OutOfMemoryException while Indexing
> -----------------------------------
>
> Key: LUCENE-2361
> URL: https://issues.apache.org/jira/browse/LUCENE-2361
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Affects Versions: 2.9.1
> Environment: Windows
> Reporter: Shivender Devarakonda
>
> Hi,
> We use lucene 2.9.1 version.
> We see the following OutOfMemory error in our environment, I think This is happening at a significant high load. Have you observed this anytime? Please let me know your thoughts on this.
> org.apache.lucene.index.MergePolicy$MergeException: java.lang.OutOfMemoryError: PermGen space
> at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:351)
> at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:315)
> Caused by: java.lang.OutOfMemoryError: PermGen space
> at java.lang.String.$$YJP$$intern(Native Method)
> at java.lang.String.intern(Unknown Source)
> at org.apache.lucene.util.SimpleStringInterner.intern(SimpleStringInterner.java:74)
> at org.apache.lucene.util.StringHelper.intern(StringHelper.java:36)
> at org.apache.lucene.index.FieldInfos.read(FieldInfos.java:356)
> at org.apache.lucene.index.FieldInfos.<init>(FieldInfos.java:71)
> at org.apache.lucene.index.SegmentReader$CoreReaders.<init>(SegmentReader.java:116)
> at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:638)
> at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:608)
> at org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:686)
> at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4979)
> at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4614)
> at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:235)
> at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:291)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Issue Comment Edited: (LUCENE-2361) OutOfMemoryException
while Indexing
Posted by "Shivender Devarakonda (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12853025#action_12853025 ]
Shivender Devarakonda edited comment on LUCENE-2361 at 4/3/10 12:27 AM:
------------------------------------------------------------------------
I am sure we have less than 20 fields in the document. Do you think it still causes this issue?
was (Author: shivenderd):
I am sure we have less than 20 fields in the index
> OutOfMemoryException while Indexing
> -----------------------------------
>
> Key: LUCENE-2361
> URL: https://issues.apache.org/jira/browse/LUCENE-2361
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Affects Versions: 2.9.1
> Environment: Windows
> Reporter: Shivender Devarakonda
>
> Hi,
> We use lucene 2.9.1 version.
> We see the following OutOfMemory error in our environment, I think This is happening at a significant high load. Have you observed this anytime? Please let me know your thoughts on this.
> org.apache.lucene.index.MergePolicy$MergeException: java.lang.OutOfMemoryError: PermGen space
> at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:351)
> at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:315)
> Caused by: java.lang.OutOfMemoryError: PermGen space
> at java.lang.String.$$YJP$$intern(Native Method)
> at java.lang.String.intern(Unknown Source)
> at org.apache.lucene.util.SimpleStringInterner.intern(SimpleStringInterner.java:74)
> at org.apache.lucene.util.StringHelper.intern(StringHelper.java:36)
> at org.apache.lucene.index.FieldInfos.read(FieldInfos.java:356)
> at org.apache.lucene.index.FieldInfos.<init>(FieldInfos.java:71)
> at org.apache.lucene.index.SegmentReader$CoreReaders.<init>(SegmentReader.java:116)
> at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:638)
> at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:608)
> at org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:686)
> at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4979)
> at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4614)
> at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:235)
> at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:291)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2361) OutOfMemoryException while Indexing
Posted by "Shivender Devarakonda (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12853074#action_12853074 ]
Shivender Devarakonda commented on LUCENE-2361:
-----------------------------------------------
The stack trace is the same in other cases too. Basically, We have load that continuously pushes specific objects and we index each incoming object. . If we do not put that load then it is working fine.
> OutOfMemoryException while Indexing
> -----------------------------------
>
> Key: LUCENE-2361
> URL: https://issues.apache.org/jira/browse/LUCENE-2361
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Affects Versions: 2.9.1
> Environment: Windows
> Reporter: Shivender Devarakonda
>
> Hi,
> We use lucene 2.9.1 version.
> We see the following OutOfMemory error in our environment, I think This is happening at a significant high load. Have you observed this anytime? Please let me know your thoughts on this.
> org.apache.lucene.index.MergePolicy$MergeException: java.lang.OutOfMemoryError: PermGen space
> at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:351)
> at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:315)
> Caused by: java.lang.OutOfMemoryError: PermGen space
> at java.lang.String.$$YJP$$intern(Native Method)
> at java.lang.String.intern(Unknown Source)
> at org.apache.lucene.util.SimpleStringInterner.intern(SimpleStringInterner.java:74)
> at org.apache.lucene.util.StringHelper.intern(StringHelper.java:36)
> at org.apache.lucene.index.FieldInfos.read(FieldInfos.java:356)
> at org.apache.lucene.index.FieldInfos.<init>(FieldInfos.java:71)
> at org.apache.lucene.index.SegmentReader$CoreReaders.<init>(SegmentReader.java:116)
> at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:638)
> at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:608)
> at org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:686)
> at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4979)
> at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4614)
> at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:235)
> at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:291)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2361) OutOfMemoryException while Indexing
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12854120#action_12854120 ]
Michael McCandless commented on LUCENE-2361:
--------------------------------------------
Hmm but the above infoStream output shows no exception... I'm confused.
> OutOfMemoryException while Indexing
> -----------------------------------
>
> Key: LUCENE-2361
> URL: https://issues.apache.org/jira/browse/LUCENE-2361
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Affects Versions: 2.9.1
> Environment: Windows
> Reporter: Shivender Devarakonda
>
> Hi,
> We use lucene 2.9.1 version.
> We see the following OutOfMemory error in our environment, I think This is happening at a significant high load. Have you observed this anytime? Please let me know your thoughts on this.
> org.apache.lucene.index.MergePolicy$MergeException: java.lang.OutOfMemoryError: PermGen space
> at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:351)
> at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:315)
> Caused by: java.lang.OutOfMemoryError: PermGen space
> at java.lang.String.$$YJP$$intern(Native Method)
> at java.lang.String.intern(Unknown Source)
> at org.apache.lucene.util.SimpleStringInterner.intern(SimpleStringInterner.java:74)
> at org.apache.lucene.util.StringHelper.intern(StringHelper.java:36)
> at org.apache.lucene.index.FieldInfos.read(FieldInfos.java:356)
> at org.apache.lucene.index.FieldInfos.<init>(FieldInfos.java:71)
> at org.apache.lucene.index.SegmentReader$CoreReaders.<init>(SegmentReader.java:116)
> at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:638)
> at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:608)
> at org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:686)
> at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4979)
> at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4614)
> at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:235)
> at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:291)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org