XML parsing hang

[Ken Krugler <kk...@transpac.com>]

Hi all,

Just a heads-up that we tracked down a serious issue we were having,  
while parsing about 100M docs.

A handful of these documents caused Tika's parsing to hang. We've got  
a FutureTask that we use to detect and (try to) terminate hung parses.

But for some of these parse attempts, we'd get lingering threads that  
were chewing up huge CPU cycles. They all look like:

> "Thread-14524974" prio=10 tid=0x00002aab18650800 nid=0x6c36 runnable  
> [0x0000000043c2a000]
>    java.lang.Thread.State: RUNNABLE
> 	at org.apache.xerces.impl.XMLScanner.scanExternalID(Unknown Source)
> 	at  
> org 
> .apache.xerces.impl.XMLDocumentScannerImpl.scanDoctypeDecl(Unknown  
> Source)
> 	at org.apache.xerces.impl.XMLDocumentScannerImpl 
> $PrologDispatcher.dispatch(Unknown Source)
> 	at  
> org 
> .apache 
> .xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown  
> Source)
> 	at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
> 	at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
> 	at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
> 	at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source)
> 	at org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown  
> Source)
> 	at org.apache.xerces.jaxp.SAXParserImpl.parse(Unknown Source)
> 	at javax.xml.parsers.SAXParser.parse(SAXParser.java:198)
> 	at  
> org 
> .apache 
> .tika 
> .detect.XmlRootExtractor.extractRootElement(XmlRootExtractor.java:47)
> 	at org.apache.tika.mime.MimeTypes.getMimeType(MimeTypes.java:236)
> 	at org.apache.tika.mime.MimeTypes.detect(MimeTypes.java:536)
> 	at  
> org.apache.tika.parser.AutoDetectParser.parse(AutoDetectParser.java: 
> 128)
> 	at bixo.parser.TikaCallable.call(TikaCallable.java:62)
> 	at bixo.parser.TikaCallable.call(TikaCallable.java:23)
> 	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> 	at java.lang.Thread.run(Thread.java:619)

We traced it to a bug in XercesImpl (we're using 2.9.1), where the  
scanExternalID() method can loop if it gets to the end of the document  
without getting a matching quote character for a literal.

Unfortunately we don't know the exact documents that caused these  
problems, but my guess is that they're not really XML docs, or they're  
badly broken.

This looks like it's been fixed in Xerces 2.10.0, potentially as a  
side effect of the fix for https://issues.apache.org/jira/browse/XERCESJ-1357

-- Ken

PS - Unfortunately 2.10.0 hasn't been pushed to Maven central yet.

--------------------------
Ken Krugler
+1 530-210-6378
http://bixolabs.com
e l a s t i c   w e b   m i n i n g







--------------------------
Ken Krugler
+1 530-210-6378
http://bixolabs.com
e l a s t i c   w e b   m i n i n g