You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ws.apache.org by "Andreas Veithen (JIRA)" <ji...@apache.org> on 2016/01/08 11:11:39 UTC

[jira] [Comment Edited] (AXIOM-478) Solution for parsing large XML

    [ https://issues.apache.org/jira/browse/AXIOM-478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15086795#comment-15086795 ] 

Andreas Veithen edited comment on AXIOM-478 at 1/8/16 10:11 AM:
----------------------------------------------------------------

I tested getTextAsStream() but it still consume large memory. Here's the test code:

{noformat}
		InputStream is = getClass().getResourceAsStream("sample2.txt");
		OMXMLParserWrapper builder = OMXMLBuilderFactory.createOMBuilder(is);
		OMElement document = builder.getDocumentElement();

		Iterator<OMElement> entryIt = document.getChildElements();
		Iterator<OMElement> contentIt = null;
		OMElement contentElement = null;
		OMElement base64Element = null;
		Reader reader = null;
		
		while (entryIt.hasNext()) {
			contentElement = entryIt.next();
			if ("content".equals(contentElement.getLocalName())) {
				contentIt = contentElement.getChildElements();
				while (contentIt.hasNext()) {
					base64Element = contentIt.next();
					if ("base64".equals(base64Element.getLocalName())) {
						reader = base64Element.getTextAsStream(false);
						int byteCount = 0;
						for (int character = reader.read(); character != -1; character = reader.read()) {
							byteCount++;
							if (byteCount == 1) {
								System.out.println("breakpoint");
							}
							if (byteCount == 24553800) {
								System.out.println("breakpoint");
							}
						}
						System.out.println("Base 64 Encoded Bytes: " + byteCount);
					}
				}
			}
		}
{noformat}


was (Author: lujie):
I tested getTextAsStream() but it still consume large memory. Here's the test code:

		InputStream is = getClass().getResourceAsStream("sample2.txt");
		OMXMLParserWrapper builder = OMXMLBuilderFactory.createOMBuilder(is);
		OMElement document = builder.getDocumentElement();

		Iterator<OMElement> entryIt = document.getChildElements();
		Iterator<OMElement> contentIt = null;
		OMElement contentElement = null;
		OMElement base64Element = null;
		Reader reader = null;
		
		while (entryIt.hasNext()) {
			contentElement = entryIt.next();
			if ("content".equals(contentElement.getLocalName())) {
				contentIt = contentElement.getChildElements();
				while (contentIt.hasNext()) {
					base64Element = contentIt.next();
					if ("base64".equals(base64Element.getLocalName())) {
						reader = base64Element.getTextAsStream(false);
						int byteCount = 0;
						for (int character = reader.read(); character != -1; character = reader.read()) {
							byteCount++;
							if (byteCount == 1) {
								System.out.println("breakpoint");
							}
							if (byteCount == 24553800) {
								System.out.println("breakpoint");
							}
						}
						System.out.println("Base 64 Encoded Bytes: " + byteCount);
					}
				}
			}
		}

> Solution for parsing large XML
> ------------------------------
>
>                 Key: AXIOM-478
>                 URL: https://issues.apache.org/jira/browse/AXIOM-478
>             Project: Axiom
>          Issue Type: Question
>            Reporter: LU Jie
>
> This is LU Jie from IBM. We use axiom to parse Atom in our project. 
> One of our CMIS API will attach file content to the XML. If the file size is large, we will get a large atom.
> If we use Entry.getExtension(QName) to parse the content, it will allocate a large memory(around 5-6 times of the file size).
> We need you help to clarify if we can use DOM-like API of axiom to get the text of a certain element as stream. That is without allocating a large object in memory.
> Or is there an alternative solution for this use case?
> We DO know that we can use pull-parser to parse the XML as stream. But we need help to investigate if axiom has already provided an API or solution to avoid writing parser by ourselves.
> Here's the sample XML. We need to parse the text of cmisra:base64 element:
> {noformat}
> <atom:entry
>     xmlns:atom="http://www.w3.org/2005/Atom"
>     xmlns:cmisra="http://docs.oasis-open.org/ns/cmis/restatom/200908/"
>     xmlns:chemistry="http://chemistry.apache.org/"
>     xmlns:cmis="http://docs.oasis-open.org/ns/cmis/core/200908/">
>     <atom:id
>         xmlns:atom="http://www.w3.org/2005/Atom">urn:uuid:00000000-0000-0000-0000-00000000000
>     </atom:id>
>     <atom:title
>         xmlns:atom="http://www.w3.org/2005/Atom" type="text">doucment1446016556658.txt
>     </atom:title>
>     <atom:updated
>         xmlns:atom="http://www.w3.org/2005/Atom">2015-10-28T07:15:57.594Z
>     </atom:updated>
>     <cmisra:content
>         xmlns:cmisra="http://docs.oasis-open.org/ns/cmis/restatom/200908/">
>         <cmisra:mediatype
>             xmlns:cmisra="http://docs.oasis-open.org/ns/cmis/restatom/200908/">text/plain
>         </cmisra:mediatype>
>         <chemistry:filename
>             xmlns:chemistry="http://chemistry.apache.org/">doucment1446016556658.txt
>         </chemistry:filename>
>         <cmisra:base64
>             xmlns:cmisra="http://docs.oasis-open.org/ns/cmis/restatom/200908/">Base64 encoded content of large file
>         </cmisra:base64>
>     </cmisra:content>
>     <cmisra:object
>         xmlns:cmisra="http://docs.oasis-open.org/ns/cmis/restatom/200908/">
>         <cmis:properties
>             xmlns:cmis="http://docs.oasis-open.org/ns/cmis/core/200908/">
>             <cmis:propertyId
>                 xmlns:cmis="http://docs.oasis-open.org/ns/cmis/core/200908/" propertyDefinitionId="cmis:objectTypeId">
>                 <cmis:value
>                     xmlns:cmis="http://docs.oasis-open.org/ns/cmis/core/200908/">snx:file
>                 </cmis:value>
>             </cmis:propertyId>
>             <cmis:propertyString
>                 xmlns:cmis="http://docs.oasis-open.org/ns/cmis/core/200908/" propertyDefinitionId="cmis:name">
>                 <cmis:value
>                     xmlns:cmis="http://docs.oasis-open.org/ns/cmis/core/200908/">doucment1446016556658.txt
>                 </cmis:value>
>             </cmis:propertyString>
>         </cmis:properties>
>     </cmisra:object>
> </atom:entry>
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@ws.apache.org
For additional commands, e-mail: dev-help@ws.apache.org