[jira] [Commented] (TIKA-2109) OutOfMemory when parsing 5MB word document

["Tim Allison (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/TIKA-2109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15828228#comment-15828228 ] 

Tim Allison commented on TIKA-2109:
-----------------------------------

One feature of this document is that it is highly repetitive, which means that it compresses dramatically.  If you decompress the document.xml file, it weighs in at 70MB.  Our classic docx parser relies on beans and DOM, which means that it loads the entire thing into memory and has overhead for the beans.  I was able to parse it with 1.5g...but, y, that is bloaty.

If you switch to the new experimental SAX parser for docx (TIKA-1321), the file requires a far lower memory footprint.

{noformat}
java -jar -Xmx64m tika-app-1.15-SNAPSHOT.jar --config=tika_config.xml TIKA-2109.docx > testout.txt
{noformat}

where tika_config.xml is:
{noformat}
<?xml version="1.0" encoding="UTF-8"?>
<properties>
  <parsers>
    <parser class="org.apache.tika.parser.DefaultParser">
      <!--<parser-exclude class="org.apache.tika.parser.ocr.TesseractOCRParser"/>-->
    </parser>
        <parser class="org.apache.tika.parser.microsoft.ooxml.OOXMLParser">
            <params>
                <param name="useSAXDocxExtractor" type="bool">true</param>
                <param name="includeDeletedContent" type="bool">true</param>
                <param name="includeMoveFromContent" type="bool">true</param>
            </params>
        </parser>
  </parsers>
</properties>
{noformat}

> OutOfMemory when parsing 5MB word document
> ------------------------------------------
>
>                 Key: TIKA-2109
>                 URL: https://issues.apache.org/jira/browse/TIKA-2109
>             Project: Tika
>          Issue Type: Bug
>    Affects Versions: 1.13
>         Environment: openjdk version "1.8.0_91"
> OpenJDK Runtime Environment (build 1.8.0_91-8u91-b14-0ubuntu4~14.04-b14)
> OpenJDK 64-Bit Server VM (build 25.91-b14, mixed mode)
>            Reporter: Julian
>         Attachments: zafar-bug-9.docx
>
>
> When I run the following command to extract text from the attached 5MB word document, I get the OOM error below.
> java -jar tika-app-1.13.jar --text '/vagrant/zafar-bug-9.docx'
> The problem goes away if I set -Xms2G -Xmx2G, but I'm reluctant to specify such a high setting for my use case for what seems like a small file? Also I don't see this error with other files of similar size.
> Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
> 	at com.sun.org.apache.xerces.internal.dom.DeferredDocumentImpl.getNodeObject(DeferredDocumentImpl.java:972)
> 	at com.sun.org.apache.xerces.internal.dom.DeferredElementNSImpl.synchronizeData(DeferredElementNSImpl.java:126)
> 	at com.sun.org.apache.xerces.internal.dom.ElementNSImpl.getNamespaceURI(ElementNSImpl.java:250)
> 	at org.apache.xmlbeans.impl.store.Locale.loadNode(Locale.java:1420)
> 	at org.apache.xmlbeans.impl.store.Locale.loadNodeChildren(Locale.java:1403)
> 	at org.apache.xmlbeans.impl.store.Locale.loadNode(Locale.java:1445)
> 	at org.apache.xmlbeans.impl.store.Locale.loadNodeChildren(Locale.java:1403)
> 	at org.apache.xmlbeans.impl.store.Locale.loadNode(Locale.java:1445)
> 	at org.apache.xmlbeans.impl.store.Locale.loadNodeChildren(Locale.java:1403)
> 	at org.apache.xmlbeans.impl.store.Locale.loadNode(Locale.java:1445)
> 	at org.apache.xmlbeans.impl.store.Locale.loadNodeChildren(Locale.java:1403)
> 	at org.apache.xmlbeans.impl.store.Locale.loadNode(Locale.java:1445)
> 	at org.apache.xmlbeans.impl.store.Locale.loadNodeChildren(Locale.java:1403)
> 	at org.apache.xmlbeans.impl.store.Locale.loadNode(Locale.java:1445)
> 	at org.apache.xmlbeans.impl.store.Locale.parseToXmlObject(Locale.java:1385)
> 	at org.apache.xmlbeans.impl.store.Locale.parseToXmlObject(Locale.java:1370)
> 	at org.apache.xmlbeans.impl.schema.SchemaTypeLoaderBase.parse(SchemaTypeLoaderBase.java:370)
> 	at org.apache.poi.POIXMLTypeLoader.parse(POIXMLTypeLoader.java:117)
> 	at org.openxmlformats.schemas.wordprocessingml.x2006.main.DocumentDocument$Factory.parse(Unknown Source)
> 	at org.apache.poi.xwpf.usermodel.XWPFDocument.onDocumentRead(XWPFDocument.java:164)
> 	at org.apache.poi.POIXMLDocument.load(POIXMLDocument.java:160)
> 	at org.apache.poi.xwpf.usermodel.XWPFDocument.<init>(XWPFDocument.java:124)
> 	at org.apache.poi.xwpf.extractor.XWPFWordExtractor.<init>(XWPFWordExtractor.java:58)
> 	at org.apache.poi.extractor.ExtractorFactory.createExtractor(ExtractorFactory.java:237)
> 	at org.apache.tika.parser.microsoft.ooxml.OOXMLExtractorFactory.parse(OOXMLExtractorFactory.java:86)
> 	at org.apache.tika.parser.microsoft.ooxml.OOXMLParser.parse(OOXMLParser.java:87)
> 	at org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:280)
> 	at org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:280)
> 	at org.apache.tika.parser.AutoDetectParser.parse(AutoDetectParser.java:120)
> 	at org.apache.tika.cli.TikaCLI$OutputType.process(TikaCLI.java:191)
> 	at org.apache.tika.cli.TikaCLI.process(TikaCLI.java:480)
> 	at org.apache.tika.cli.TikaCLI.main(TikaCLI.java:145)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)