You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cocoon.apache.org by Christofer Dutz <ma...@c-ware.de> on 2007/10/22 12:25:37 UTC
Re: [Exist-open] problem while processing large xmls
Hi,
I think you might be experiencing a problem with the memorymanager of
your java virtual machine.
The problem with this ist, that everytime the VM increases the amount of
allocated memory, it expects this to be in one block.
Whenever there is no unfragmented block able to contain all the VM
memory, an OutOfMemory exception occurs.
I would suggest increasing the Xms and seting the Xmx value to the same
value as Xms. This way the entire block is allocated
during VM startup and OutOfMemory exceptions should no longer occur.
In general when transforming very large Xml documents, I would suggest
using Sax to do the transformation. When doing a Dom
transformation of a 20MB Xml document could result in a memory
consumption of about 50-50MB depending on the document
structure and the transformation.
Hope this Helps ;)
Chris
Richa Gupta ,Gurgaon schrieb:
> Thanks for you suggestion.
>
> We looked on forum for OutOfMemoryError. And made changes for JVM cache and heap size in config.xml & startup.sh file.
>
> In conf.xml, values for db-connection are:
> <db-connection cacheSize="48M" collectionCacheSize="128" database="native" files="webapp/WEB-INF/data" free_mem_min="5" pageSize="4096">
>
> In startup.sh, we have
> if [ -z "$JAVA_OPTIONS" ]; then
> export JAVA_OPTIONS="-Xms16000k -Xmx768000k -Dfile.encoding=UTF-8"
>
> But we are still getting the same error. Please suggest a way to solve this problem.
>
> We are using xmldb's XQueryService to run transform xquery.
>
> Thanks,
> Richa
>
>
> -----Original Message-----
> From: Pierrick Brihaye [mailto:pierrick.brihaye@free.fr]
> Sent: Monday, October 22, 2007 11:15 AM
> To: Richa Gupta ,Gurgaon
> Cc: Exist-open@lists.sourceforge.net
> Subject: Re: [Exist-open] problem while processing large xmls
>
> Hi,
>
> Richa Gupta ,Gurgaon a écrit :
>
>
>> While applying xsl on a large xml file (say 20 mb), we are getting
>> OutOfMemoryError.
>>
>> _org.xmldb.api.base.XMLDBException_: _java.lang.Exception_:
>> java.lang.OutOfMemoryError: Java heap space
>>
>
>
>> Please suggest Is this some limitation of eXist?
>>
>
> Probably not. Rather a limitation of your Java Virtual Machine.
>
>
>> Also provide some solution to solve this problem.
>>
>
> Please ?
>
> Search the mailing-list archives : this question if a frequently asked one.
>
> Cheers,
>
> p.b.
>
> DISCLAIMER:
> -----------------------------------------------------------------------------------------------------------------------
>
> The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only.
> It shall not attach any liability on the originator or HCL or its affiliates. Any views or opinions presented in
> this email are solely those of the author and may not necessarily reflect the opinions of HCL or its affiliates.
> Any form of reproduction, dissemination, copying, disclosure, modification, distribution and / or publication of
> this message without the prior written consent of the author of this e-mail is strictly prohibited. If you have
> received this email in error please delete it and notify the sender immediately. Before opening any mail and
> attachments please check them for viruses and defect.
>
> -----------------------------------------------------------------------------------------------------------------------
>
> -------------------------------------------------------------------------
> This SF.net email is sponsored by: Splunk Inc.
> Still grepping through log files to find problems? Stop.
> Now Search log events and configuration files using AJAX and a browser.
> Download your FREE copy of Splunk now >> http://get.splunk.com/
> _______________________________________________
> Exist-open mailing list
> Exist-open@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/exist-open
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@cocoon.apache.org
For additional commands, e-mail: users-help@cocoon.apache.org
Re: [Exist-open] problem while processing large xmls
Posted by Christofer Dutz <ma...@c-ware.de>.
Uuups ... sorry for posting to the wrong mailinglist. Seems that I setup
my mail-filters a litte wrong ;)
Christofer Dutz schrieb:
> Hi,
>
> I think you might be experiencing a problem with the memorymanager of
> your java virtual machine.
> The problem with this ist, that everytime the VM increases the amount of
> allocated memory, it expects this to be in one block.
> Whenever there is no unfragmented block able to contain all the VM
> memory, an OutOfMemory exception occurs.
> I would suggest increasing the Xms and seting the Xmx value to the same
> value as Xms. This way the entire block is allocated
> during VM startup and OutOfMemory exceptions should no longer occur.
>
> In general when transforming very large Xml documents, I would suggest
> using Sax to do the transformation. When doing a Dom
> transformation of a 20MB Xml document could result in a memory
> consumption of about 50-50MB depending on the document
> structure and the transformation.
>
> Hope this Helps ;)
>
> Chris
>
>
>
> Richa Gupta ,Gurgaon schrieb:
>> Thanks for you suggestion.
>>
>> We looked on forum for OutOfMemoryError. And made changes for JVM
>> cache and heap size in config.xml & startup.sh file.
>>
>> In conf.xml, values for db-connection are: <db-connection
>> cacheSize="48M" collectionCacheSize="128" database="native"
>> files="webapp/WEB-INF/data" free_mem_min="5" pageSize="4096">
>>
>> In startup.sh, we have if [ -z "$JAVA_OPTIONS" ]; then
>> export JAVA_OPTIONS="-Xms16000k -Xmx768000k -Dfile.encoding=UTF-8"
>>
>> But we are still getting the same error. Please suggest a way to
>> solve this problem.
>>
>> We are using xmldb's XQueryService to run transform xquery.
>>
>> Thanks,
>> Richa
>>
>>
>> -----Original Message-----
>> From: Pierrick Brihaye [mailto:pierrick.brihaye@free.fr] Sent:
>> Monday, October 22, 2007 11:15 AM
>> To: Richa Gupta ,Gurgaon
>> Cc: Exist-open@lists.sourceforge.net
>> Subject: Re: [Exist-open] problem while processing large xmls
>>
>> Hi,
>>
>> Richa Gupta ,Gurgaon a écrit :
>>
>>
>>> While applying xsl on a large xml file (say 20 mb), we are getting
>>> OutOfMemoryError.
>>>
>>> _org.xmldb.api.base.XMLDBException_: _java.lang.Exception_:
>>> java.lang.OutOfMemoryError: Java heap space
>>>
>>
>>
>>> Please suggest Is this some limitation of eXist?
>>>
>>
>> Probably not. Rather a limitation of your Java Virtual Machine.
>>
>>
>>> Also provide some solution to solve this problem.
>>
>> Please ?
>>
>> Search the mailing-list archives : this question if a frequently
>> asked one.
>>
>> Cheers,
>>
>> p.b.
>>
>> DISCLAIMER:
>> -----------------------------------------------------------------------------------------------------------------------
>>
>>
>> The contents of this e-mail and any attachment(s) are confidential
>> and intended for the named recipient(s) only.
>> It shall not attach any liability on the originator or HCL or its
>> affiliates. Any views or opinions presented in this email are solely
>> those of the author and may not necessarily reflect the opinions of
>> HCL or its affiliates.
>> Any form of reproduction, dissemination, copying, disclosure,
>> modification, distribution and / or publication of this message
>> without the prior written consent of the author of this e-mail is
>> strictly prohibited. If you have received this email in error please
>> delete it and notify the sender immediately. Before opening any mail
>> and attachments please check them for viruses and defect.
>>
>> -----------------------------------------------------------------------------------------------------------------------
>>
>>
>> -------------------------------------------------------------------------
>>
>> This SF.net email is sponsored by: Splunk Inc.
>> Still grepping through log files to find problems? Stop.
>> Now Search log events and configuration files using AJAX and a browser.
>> Download your FREE copy of Splunk now >> http://get.splunk.com/
>> _______________________________________________
>> Exist-open mailing list
>> Exist-open@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/exist-open
>>
>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@cocoon.apache.org
> For additional commands, e-mail: users-help@cocoon.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@cocoon.apache.org
For additional commands, e-mail: users-help@cocoon.apache.org