You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ws.apache.org by "Andreas Veithen (JIRA)" <ji...@apache.org> on 2016/01/08 11:11:39 UTC
[jira] [Comment Edited] (AXIOM-478) Solution for parsing large XML
[ https://issues.apache.org/jira/browse/AXIOM-478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15086795#comment-15086795 ]
Andreas Veithen edited comment on AXIOM-478 at 1/8/16 10:11 AM:
----------------------------------------------------------------
I tested getTextAsStream() but it still consume large memory. Here's the test code:
{noformat}
InputStream is = getClass().getResourceAsStream("sample2.txt");
OMXMLParserWrapper builder = OMXMLBuilderFactory.createOMBuilder(is);
OMElement document = builder.getDocumentElement();
Iterator<OMElement> entryIt = document.getChildElements();
Iterator<OMElement> contentIt = null;
OMElement contentElement = null;
OMElement base64Element = null;
Reader reader = null;
while (entryIt.hasNext()) {
contentElement = entryIt.next();
if ("content".equals(contentElement.getLocalName())) {
contentIt = contentElement.getChildElements();
while (contentIt.hasNext()) {
base64Element = contentIt.next();
if ("base64".equals(base64Element.getLocalName())) {
reader = base64Element.getTextAsStream(false);
int byteCount = 0;
for (int character = reader.read(); character != -1; character = reader.read()) {
byteCount++;
if (byteCount == 1) {
System.out.println("breakpoint");
}
if (byteCount == 24553800) {
System.out.println("breakpoint");
}
}
System.out.println("Base 64 Encoded Bytes: " + byteCount);
}
}
}
}
{noformat}
was (Author: lujie):
I tested getTextAsStream() but it still consume large memory. Here's the test code:
InputStream is = getClass().getResourceAsStream("sample2.txt");
OMXMLParserWrapper builder = OMXMLBuilderFactory.createOMBuilder(is);
OMElement document = builder.getDocumentElement();
Iterator<OMElement> entryIt = document.getChildElements();
Iterator<OMElement> contentIt = null;
OMElement contentElement = null;
OMElement base64Element = null;
Reader reader = null;
while (entryIt.hasNext()) {
contentElement = entryIt.next();
if ("content".equals(contentElement.getLocalName())) {
contentIt = contentElement.getChildElements();
while (contentIt.hasNext()) {
base64Element = contentIt.next();
if ("base64".equals(base64Element.getLocalName())) {
reader = base64Element.getTextAsStream(false);
int byteCount = 0;
for (int character = reader.read(); character != -1; character = reader.read()) {
byteCount++;
if (byteCount == 1) {
System.out.println("breakpoint");
}
if (byteCount == 24553800) {
System.out.println("breakpoint");
}
}
System.out.println("Base 64 Encoded Bytes: " + byteCount);
}
}
}
}
> Solution for parsing large XML
> ------------------------------
>
> Key: AXIOM-478
> URL: https://issues.apache.org/jira/browse/AXIOM-478
> Project: Axiom
> Issue Type: Question
> Reporter: LU Jie
>
> This is LU Jie from IBM. We use axiom to parse Atom in our project.
> One of our CMIS API will attach file content to the XML. If the file size is large, we will get a large atom.
> If we use Entry.getExtension(QName) to parse the content, it will allocate a large memory(around 5-6 times of the file size).
> We need you help to clarify if we can use DOM-like API of axiom to get the text of a certain element as stream. That is without allocating a large object in memory.
> Or is there an alternative solution for this use case?
> We DO know that we can use pull-parser to parse the XML as stream. But we need help to investigate if axiom has already provided an API or solution to avoid writing parser by ourselves.
> Here's the sample XML. We need to parse the text of cmisra:base64 element:
> {noformat}
> <atom:entry
> xmlns:atom="http://www.w3.org/2005/Atom"
> xmlns:cmisra="http://docs.oasis-open.org/ns/cmis/restatom/200908/"
> xmlns:chemistry="http://chemistry.apache.org/"
> xmlns:cmis="http://docs.oasis-open.org/ns/cmis/core/200908/">
> <atom:id
> xmlns:atom="http://www.w3.org/2005/Atom">urn:uuid:00000000-0000-0000-0000-00000000000
> </atom:id>
> <atom:title
> xmlns:atom="http://www.w3.org/2005/Atom" type="text">doucment1446016556658.txt
> </atom:title>
> <atom:updated
> xmlns:atom="http://www.w3.org/2005/Atom">2015-10-28T07:15:57.594Z
> </atom:updated>
> <cmisra:content
> xmlns:cmisra="http://docs.oasis-open.org/ns/cmis/restatom/200908/">
> <cmisra:mediatype
> xmlns:cmisra="http://docs.oasis-open.org/ns/cmis/restatom/200908/">text/plain
> </cmisra:mediatype>
> <chemistry:filename
> xmlns:chemistry="http://chemistry.apache.org/">doucment1446016556658.txt
> </chemistry:filename>
> <cmisra:base64
> xmlns:cmisra="http://docs.oasis-open.org/ns/cmis/restatom/200908/">Base64 encoded content of large file
> </cmisra:base64>
> </cmisra:content>
> <cmisra:object
> xmlns:cmisra="http://docs.oasis-open.org/ns/cmis/restatom/200908/">
> <cmis:properties
> xmlns:cmis="http://docs.oasis-open.org/ns/cmis/core/200908/">
> <cmis:propertyId
> xmlns:cmis="http://docs.oasis-open.org/ns/cmis/core/200908/" propertyDefinitionId="cmis:objectTypeId">
> <cmis:value
> xmlns:cmis="http://docs.oasis-open.org/ns/cmis/core/200908/">snx:file
> </cmis:value>
> </cmis:propertyId>
> <cmis:propertyString
> xmlns:cmis="http://docs.oasis-open.org/ns/cmis/core/200908/" propertyDefinitionId="cmis:name">
> <cmis:value
> xmlns:cmis="http://docs.oasis-open.org/ns/cmis/core/200908/">doucment1446016556658.txt
> </cmis:value>
> </cmis:propertyString>
> </cmis:properties>
> </cmisra:object>
> </atom:entry>
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@ws.apache.org
For additional commands, e-mail: dev-help@ws.apache.org