You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Otis Gospodnetic (JIRA)" <ji...@apache.org> on 2008/09/11 19:47:45 UTC
[jira] Commented: (LUCENE-1381) Hanging while indexing/digesting on
multiple threads
[ https://issues.apache.org/jira/browse/LUCENE-1381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12630270#action_12630270 ]
Otis Gospodnetic commented on LUCENE-1381:
------------------------------------------
David, why not bring this up on java-user list first?
Are your 4 IndexWriters writing to the same index?
> Hanging while indexing/digesting on multiple threads
> ----------------------------------------------------
>
> Key: LUCENE-1381
> URL: https://issues.apache.org/jira/browse/LUCENE-1381
> Project: Lucene - Java
> Issue Type: Bug
> Components: Analysis
> Affects Versions: 2.3.2
> Environment: Java HotSpot(TM) 64-Bit Server VM (1.5.0_16-b02 mixed mode) on 2.6.9-78.0.1.ELsmp #1 SMP x86_64 x86_64 x86_64 GNU/Linux
> Reporter: David Fertig
>
> With several older lucene projects already running and "stable", I have recently written a multi-threading indexer using to the 2.3.2 release.
> My volume is in the millions of documents indexed daily and I have been stress testing for a while now. My current setup has 3 JVMs, each running 6 threads indexing different documents, with 1 IndexWriter per JVM. For stability testing, the indexer shutsdown and exits every 5-10 minutes, with a new JVM is started again for a clean restart. At this rate, I have noticed an rare, but eventually consistent internal hang/deadlock in all indexer threads while parsing documents. My 'manager' thread is alive and regularly polling the indexer threads and displaying their state variables, but the indexer threads themselves appear not to be making progress while using up nearly 100% of available CPU. Memory usage is relativly low and stable at 481m out of 2048m available.
> Most stack traces, and STAY in this state even after repeated inspections: (pressing CTRL-\ in active JVM window)
> ----------
> Full thread dump Java HotSpot(TM) 64-Bit Server VM (1.5.0_16-b02 mixed mode):
> "Thread-6" prio=1 tid=0x0000002b25750920 nid=0x34f6 runnable [0x0000000041465000..0x0000000041465db0]
> at java.util.WeakHashMap.eq(WeakHashMap.java:254)
> at java.util.WeakHashMap.get(WeakHashMap.java:345)
> at org.apache.commons.beanutils.MethodUtils.getMatchingAccessibleMethod(MethodUtils.java:530)
> at org.apache.commons.beanutils.MethodUtils.invokeMethod(MethodUtils.java:209)
> at org.apache.commons.digester.CallMethodRule.end(CallMethodRule.java:625)
> at org.apache.commons.digester.Rule.end(Rule.java:230)
> at org.apache.commons.digester.Digester.endElement(Digester.java:1130)
> at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(AbstractSAXParser.java:633)
> at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEndElement(XMLDocumentFragmentScannerImpl.java:1241)
> at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(XMLDocumentFragmentScannerImpl.java:1685)
> at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:368)
> at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:834)
> at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:764)
> at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:148)
> at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1242)
> at org.apache.commons.digester.Digester.parse(Digester.java:1685)
> ...
> "Thread-5" prio=1 tid=0x0000002b25754eb0 nid=0x34f5 runnable [0x0000000041364000..0x0000000041364d30]
> at java.lang.String.equals(String.java:858)
> at org.apache.commons.beanutils.MethodUtils$MethodDescriptor.equals(MethodUtils.java:833)
> at java.util.WeakHashMap.eq(WeakHashMap.java:254)
> at java.util.WeakHashMap.get(WeakHashMap.java:345)
> at org.apache.commons.beanutils.MethodUtils.getMatchingAccessibleMethod(MethodUtils.java:530)
> at org.apache.commons.beanutils.MethodUtils.invokeMethod(MethodUtils.java:209)
> at org.apache.commons.digester.CallMethodRule.end(CallMethodRule.java:625)
> at org.apache.commons.digester.Rule.end(Rule.java:230)
> at org.apache.commons.digester.Digester.endElement(Digester.java:1130)
> at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(AbstractSAXParser.java:633)
> at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEndElement(XMLDocumentFragmentScannerImpl.java:1241)
> at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(XMLDocumentFragmentScannerImpl.java:1685)
> at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:368)
> at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:834)
> at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:764)
> at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:148)
> at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1242)
> at org.apache.commons.digester.Digester.parse(Digester.java:1685)
> ...
> "Thread-4" prio=1 tid=0x0000002b25754860 nid=0x34f4 runnable [0x0000000041263000..0x0000000041263cb0]
> at java.lang.String.equals(String.java:858)
> at org.apache.commons.beanutils.MethodUtils$MethodDescriptor.equals(MethodUtils.java:833)
> at java.util.WeakHashMap.eq(WeakHashMap.java:254)
> at java.util.WeakHashMap.get(WeakHashMap.java:345)
> at org.apache.commons.beanutils.MethodUtils.getMatchingAccessibleMethod(MethodUtils.java:530)
> at org.apache.commons.beanutils.MethodUtils.invokeMethod(MethodUtils.java:209)
> at org.apache.commons.digester.CallMethodRule.end(CallMethodRule.java:625)
> at org.apache.commons.digester.Rule.end(Rule.java:230)
> at org.apache.commons.digester.Digester.endElement(Digester.java:1130)
> at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(AbstractSAXParser.java:633)
> at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEndElement(XMLDocumentFragmentScannerImpl.java:1241)
> at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(XMLDocumentFragmentScannerImpl.java:1685)
> at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:368)
> at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:834)
> at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:764)
> at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:148)
> at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1242)
> at org.apache.commons.digester.Digester.parse(Digester.java:1685)
> ...
> "Thread-3" prio=1 tid=0x0000002b2509d360 nid=0x34f3 runnable [0x0000000041162000..0x0000000041162c30]
> at java.lang.String.equals(String.java:858)
> at org.apache.commons.beanutils.MethodUtils$MethodDescriptor.equals(MethodUtils.java:833)
> at java.util.WeakHashMap.eq(WeakHashMap.java:254)
> at java.util.WeakHashMap.get(WeakHashMap.java:345)
> at org.apache.commons.beanutils.MethodUtils.getMatchingAccessibleMethod(MethodUtils.java:530)
> at org.apache.commons.beanutils.MethodUtils.invokeMethod(MethodUtils.java:209)
> at org.apache.commons.digester.CallMethodRule.end(CallMethodRule.java:625)
> at org.apache.commons.digester.Rule.end(Rule.java:230)
> at org.apache.commons.digester.Digester.endElement(Digester.java:1130)
> at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(AbstractSAXParser.java:633)
> at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEndElement(XMLDocumentFragmentScannerImpl.java:1241)
> at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(XMLDocumentFragmentScannerImpl.java:1685)
> at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:368)
> at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:834)
> at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:764)
> at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:148)
> at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1242)
> at org.apache.commons.digester.Digester.parse(Digester.java:1685)
> ...
> "Thread-2" prio=1 tid=0x0000002b25083e00 nid=0x34f2 runnable [0x0000000041061000..0x0000000041061bb0]
> at java.util.WeakHashMap.expungeStaleEntries(WeakHashMap.java:289)
> at java.util.WeakHashMap.getTable(WeakHashMap.java:297)
> at java.util.WeakHashMap.get(WeakHashMap.java:341)
> at org.apache.commons.beanutils.MethodUtils.getMatchingAccessibleMethod(MethodUtils.java:530)
> at org.apache.commons.beanutils.MethodUtils.invokeMethod(MethodUtils.java:209)
> at org.apache.commons.digester.CallMethodRule.end(CallMethodRule.java:625)
> at org.apache.commons.digester.Rule.end(Rule.java:230)
> at org.apache.commons.digester.Digester.endElement(Digester.java:1130)
> at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(AbstractSAXParser.java:633)
> at com.sun.org.apache.xerces.internal.parsers.AbstractXMLDocumentParser.emptyElement(AbstractXMLDocumentParser.java:221)
> at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanStartElement(XMLDocumentFragmentScannerImpl.java:872)
> at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(XMLDocumentFragmentScannerImpl.java:1693)
> at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:368)
> at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:834)
> at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:764)
> at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:148)
> at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1242)
> at org.apache.commons.digester.Digester.parse(Digester.java:1685)
> ...
> "Thread-1" prio=1 tid=0x0000002b2509a4a0 nid=0x34f1 runnable [0x0000000040f60000..0x0000000040f60b30]
> at java.util.WeakHashMap.expungeStaleEntries(WeakHashMap.java:289)
> at java.util.WeakHashMap.getTable(WeakHashMap.java:297)
> at java.util.WeakHashMap.get(WeakHashMap.java:341)
> at org.apache.commons.beanutils.MethodUtils.getMatchingAccessibleMethod(MethodUtils.java:530)
> at org.apache.commons.beanutils.MethodUtils.invokeMethod(MethodUtils.java:209)
> at org.apache.commons.digester.CallMethodRule.end(CallMethodRule.java:625)
> at org.apache.commons.digester.Rule.end(Rule.java:230)
> at org.apache.commons.digester.Digester.endElement(Digester.java:1130)
> at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(AbstractSAXParser.java:633)
> at com.sun.org.apache.xerces.internal.parsers.AbstractXMLDocumentParser.emptyElement(AbstractXMLDocumentParser.java:221)
> at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanStartElement(XMLDocumentFragmentScannerImpl.java:872)
> at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(XMLDocumentFragmentScannerImpl.java:1693)
> at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:368)
> at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:834)
> at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:764)
> at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:148)
> at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1242)
> at org.apache.commons.digester.Digester.parse(Digester.java:1685)
> ...
> "Low Memory Detector" daemon prio=1 tid=0x0000002b25002430 nid=0x34ef runnable [0x0000000000000000..0x0000000000000000]
> "CompilerThread1" daemon prio=1 tid=0x0000002b250009d0 nid=0x34ee waiting on condition [0x0000000000000000..0x0000000040c5c6d0]
> "CompilerThread0" daemon prio=1 tid=0x0000002b220c2ce0 nid=0x34ed waiting on condition [0x0000000000000000..0x0000000040b5b650]
> "AdapterThread" daemon prio=1 tid=0x0000002b220c1750 nid=0x34ec waiting on condition [0x0000000000000000..0x0000000000000000]
> "Signal Dispatcher" daemon prio=1 tid=0x0000002b220c02d0 nid=0x34eb runnable [0x0000000000000000..0x0000000000000000]
> "Finalizer" daemon prio=1 tid=0x0000002b220ae1e0 nid=0x34ea in Object.wait() [0x0000000040859000..0x0000000040859bb0]
> at java.lang.Object.wait(Native Method)
> - waiting on <0x0000002b0a8d7408> (a java.lang.ref.ReferenceQueue$Lock)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:120)
> - locked <0x0000002b0a8d7408> (a java.lang.ref.ReferenceQueue$Lock)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:136)
> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
> "Reference Handler" daemon prio=1 tid=0x0000002b220ab980 nid=0x34e9 in Object.wait() [0x0000000040758000..0x0000000040758b30]
> at java.lang.Object.wait(Native Method)
> - waiting on <0x0000002b0a7da3b0> (a java.lang.ref.Reference$Lock)
> at java.lang.Object.wait(Object.java:474)
> at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
> - locked <0x0000002b0a7da3b0> (a java.lang.ref.Reference$Lock)
> "main" prio=1 tid=0x0000000040115ea0 nid=0x34e3 waiting on condition [0x0000007fbfffc000..0x0000007fbfffd280]
> at java.lang.Thread.sleep(Native Method)
> ...
> "VM Thread" prio=1 tid=0x0000002b220a7480 nid=0x34e8 runnable
> "GC task thread#0 (ParallelGC)" prio=1 tid=0x0000000040133fe0 nid=0x34e4 runnable
> "GC task thread#1 (ParallelGC)" prio=1 tid=0x0000000040135630 nid=0x34e5 runnable
> "GC task thread#2 (ParallelGC)" prio=1 tid=0x0000000040136480 nid=0x34e6 runnable
> "GC task thread#3 (ParallelGC)" prio=1 tid=0x0000000040137300 nid=0x34e7 runnable
> "VM Periodic Task Thread" prio=1 tid=0x0000002b25004e10 nid=0x34f0 waiting on condition
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org