You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@xalan.apache.org by Patrick Glur <gl...@Nagra-Kudelski.ch> on 2000/04/10 15:33:57 UTC
Memory usage when processing big files
I am using Xalan Java 1.0.0 with Xerces Java 1.0.3 on Compaq Tru64 Unix.
The Java VM uses a lot of memory when Xalan is processing a big XML input
file.
My input file is the following
<?xml version="1.0" encoding="UTF-8"?>
<Schedule providerId="Nagra" creationDate="2000-04-10T10:17:56Z">
<Service id="1">
<Period start="2000-04-11T06:00:00Z" end="2000-04-18T06:00:00Z"/>
<Event id="4000000" type="S">
<Period start="2000-04-11T06:00:00Z" end="2000-04-11T07:00:00Z"
duration="3600"/>
<EpgProduction>
<EpgText language="eng">
<ShortName>name 4000000</ShortName>
<Name>event title 4000000</Name>
<Description>description of event 4000000</Description>
<Actors>actors of event 4000000</Actors>
</EpgText>
<EpgText language="fra">
<ShortName>nom 4000000</ShortName>
<Name>titre de l'evenement 4000000</Name>
<Description>description de l'evenement 4000000</Description>
<Highlights>critique de l'evenement 4000000</Highlights>
</EpgText>
<ExtendedRating language="3" nudity="1" sex="4" violence="2"
foul="5"/>
<DvbContent c1="A" c2="1" u1="3" u2="0"/>
<DvbContent c1="B" c2="2" u1="4" u2="1"/>
<DvbContent c1="C" c2="3" u1="5" u2="2"/>
</EpgProduction>
</Event>
...
</Service>
</Schedule>
There are 200 services in the schedule. Each service is composed of 168
events, and each events is composed of 17 nodes. Therefore, there are 571601
nodes (1 + 200 * 2 + 200 * 168 * 17) in my input file. The file has 706204
lines and has a size of 30.65 MB.
I want to produce an ASCII output file. Each service is independant.
The stylesheet is the following:
<?xml version="1.0" encoding='UTF-8'?>
<!DOCTYPE xsl:stylesheet SYSTEM "proto5_master.ent">
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
version="1.0"
xmlns:lxslt="http://xml.apache.org/xslt"
xmlns:sog-ext="sog-ext1"
extension-element-prefixes="sog-ext">
<lxslt:component prefix="sog-ext">
<lxslt:script lang="javaclass" src="proto5_class"/>
</lxslt:component>
<xsl:output omit-xml-declaration="yes" method="text"/>
<xsl:variable name="HeaderRecord">0</xsl:variable>
<xsl:variable name="ServiceRecord">1</xsl:variable>
<xsl:variable name="EventRecord">2</xsl:variable>
<xsl:variable name="Sep">,</xsl:variable>
<xsl:template match="Schedule">
<xsl:value-of select="$HeaderRecord"/>
<xsl:value-of select="$Sep"/>
<xsl:value-of select="sog-ext:quoteStr(string(attribute::providerId))"/>
<xsl:value-of select="$Sep"/>
<xsl:value-of select="sog-ext:oracleDate(string(@creationDate))"/>
<xsl:value-of select="$Sep"/>
<xsl:value-of select="sog-ext:formatNumber(count(Service), 4)"/>
<xsl:text>&nl;</xsl:text>
<xsl:apply-templates select="Service"/>
</xsl:template>
<xsl:template match="Service">
<xsl:value-of select="$ServiceRecord"/>
<xsl:value-of select="$Sep"/>
<xsl:value-of select="sog-ext:formatNumber(number(@id), 5)"/>
<xsl:value-of select="$Sep"/>
<xsl:value-of select="sog-ext:oracleDate(string(Period/@start))"/>
<xsl:value-of select="$Sep"/>
<xsl:value-of select="sog-ext:oracleDate(string(Period/@end))"/>
<xsl:value-of select="$Sep"/>
<xsl:value-of select="sog-ext:formatNumber(count(Event), 4)"/>
<xsl:text>&nl;</xsl:text>
<xsl:apply-templates select="Event"/>
</xsl:template>
<xsl:template match="Event">
<xsl:value-of select="$EventRecord"/>
<xsl:value-of select="$Sep"/>
<xsl:value-of select="sog-ext:formatNumber(number(attribute::id), 12)"/>
<xsl:value-of select="$Sep"/>
<xsl:variable name="EvName" select="EpgProduction/EpgText/Name"/>
<xsl:value-of select="sog-ext:quoteStr(string($EvName))"/>
<xsl:value-of select="$Sep"/>
<xsl:variable name="EvShortName"
select="EpgProduction/EpgText/ShortName"/>
<xsl:value-of select="sog-ext:quoteStr(string($EvShortName))"/>
<xsl:value-of select="$Sep"/>
<xsl:variable name="EvDescr"
select="EpgProduction/EpgText/Description"/>
<xsl:value-of select="sog-ext:quoteTruncStr(string($EvDescr), 22)"/>
<xsl:value-of select="$Sep"/>
<xsl:value-of
select="sog-ext:quoteStr(string(EpgProduction/EpgText/@languag
e))"/>
<xsl:value-of select="$Sep"/>
<xsl:variable name="EvActors" select="EpgProduction/EpgText/Actors"/>
<xsl:value-of select="sog-ext:quoteStr(string($EvActors))"/>
<xsl:value-of select="$Sep"/>
<xsl:value-of select="sog-ext:oracleDate(string(Period/@start))"/>
<xsl:value-of select="$Sep"/>
<xsl:value-of select="sog-ext:oracleDate(string(Period/@end))"/>
<xsl:value-of select="$Sep"/>
<xsl:value-of
select="sog-ext:formatDuration(number(Period/@duration))"/>
<xsl:value-of select="$Sep"/>
<xsl:value-of select="EpgProduction/ExtendedRating/@sex"/>
<xsl:value-of select="$Sep"/>
<xsl:value-of select="EpgProduction/ExtendedRating/@violence"/>
<xsl:text>&nl;</xsl:text>
</xsl:template>
</xsl:stylesheet>
The processing duration is about 22 minutes. The memory usage of the Java VM
was increasing while processing the file. Once the file is parsed, already
100 MB are used, and just before the end of the processing 247 MB were used.
The output file has 33801 lines and a size of 4.81 MB.
I have try another solution that consists in splitting the original input
file in 200 files (one file per service). I then write a master XML input
file that contains the list of the other XML files.
The master input file looks like the following:
<?xml version="1.0" encoding="UTF-8"?>
<Schedule providerId="Nagra" creationDate="2000-04-10T10:17:56Z">
<FileList>
<File>proto6_200_168_1.xml</File>
<File>proto6_200_168_2.xml</File>
<File>proto6_200_168_3.xml</File>
....
<File>proto6_200_168_200.xml</File>
</FileList>
</Schedule>
The stylesheet is then updated with the document() function:
<!DOCTYPE xsl:stylesheet SYSTEM "proto6_master.ent">
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
version="1.0"
xmlns:lxslt="http://xml.apache.org/xslt"
xmlns:sog-ext="sog-ext1"
extension-element-prefixes="sog-ext">
<lxslt:component prefix="sog-ext">
<lxslt:script lang="javaclass" src="proto6_class"/>
</lxslt:component>
<xsl:output omit-xml-declaration="yes" method="text"/>
<xsl:variable name="HeaderRecord">0</xsl:variable>
<xsl:variable name="ServiceRecord">1</xsl:variable>
<xsl:variable name="EventRecord">2</xsl:variable>
<xsl:variable name="Sep">,</xsl:variable>
<xsl:template match="Schedule">
<xsl:value-of select="$HeaderRecord"/>
<xsl:value-of select="$Sep"/>
<xsl:value-of select="sog-ext:quoteStr(string(attribute::providerId))"/>
<xsl:value-of select="$Sep"/>
<xsl:value-of select="sog-ext:oracleDate(string(@creationDate))"/>
<xsl:value-of select="$Sep"/>
<xsl:value-of select="format-number(count(FileList/File), '0000')"/>
<xsl:text>&nl;</xsl:text>
<xsl:apply-templates select="FileList"/>
</xsl:template>
<xsl:template match="FileList">
<xsl:apply-templates select="File"/>
</xsl:template>
<xsl:template match="File">
<xsl:param name="path" select="."/>
<xsl:for-each select="document($path)">
<xsl:apply-templates select="Service"/>
</xsl:for-each>
</xsl:template>
<xsl:template match="Service">
<xsl:value-of select="$ServiceRecord"/>
<xsl:value-of select="$Sep"/>
<xsl:value-of select="sog-ext:formatNumber(number(@id), 5)"/>
<xsl:value-of select="$Sep"/>
<xsl:value-of select="sog-ext:oracleDate(string(Period/@start))"/>
<xsl:value-of select="$Sep"/>
<xsl:value-of select="sog-ext:oracleDate(string(Period/@end))"/>
<xsl:value-of select="$Sep"/>
<xsl:value-of select="sog-ext:formatNumber(count(Event), 4)"/>
<xsl:text>&nl;</xsl:text>
<xsl:apply-templates select="Event"/>
</xsl:template>
<xsl:template match="Event">
<xsl:value-of select="$EventRecord"/>
<xsl:value-of select="$Sep"/>
<xsl:value-of select="sog-ext:formatNumber(number(attribute::id), 12)"/>
<xsl:value-of select="$Sep"/>
<xsl:variable name="EvName" select="EpgProduction/EpgText/Name"/>
<xsl:value-of select="sog-ext:quoteStr(string($EvName))"/>
<xsl:value-of select="$Sep"/>
<xsl:variable name="EvShortName"
select="EpgProduction/EpgText/ShortName"/>
<xsl:value-of select="sog-ext:quoteStr(string($EvShortName))"/>
<xsl:value-of select="$Sep"/>
<xsl:variable name="EvDescr"
select="EpgProduction/EpgText/Description"/>
<xsl:value-of select="sog-ext:quoteTruncStr(string($EvDescr), 22)"/>
<xsl:value-of select="$Sep"/>
<xsl:value-of
select="sog-ext:quoteStr(string(EpgProduction/EpgText/@languag
e))"/>
<xsl:value-of select="$Sep"/>
<xsl:variable name="EvActors" select="EpgProduction/EpgText/Actors"/>
<xsl:value-of select="sog-ext:quoteStr(string($EvActors))"/>
<xsl:value-of select="$Sep"/>
<xsl:value-of select="sog-ext:oracleDate(string(Period/@start))"/>
<xsl:value-of select="$Sep"/>
<xsl:value-of select="sog-ext:oracleDate(string(Period/@end))"/>
<xsl:value-of select="$Sep"/>
<xsl:value-of
select="sog-ext:formatDuration(number(Period/@duration))"/>
<xsl:value-of select="$Sep"/>
<xsl:value-of select="EpgProduction/ExtendedRating/@sex"/>
<xsl:value-of select="$Sep"/>
<xsl:value-of select="EpgProduction/ExtendedRating/@violence"/>
<xsl:text>&nl;</xsl:text>
</xsl:template>
</xsl:stylesheet>
With this second solution, the parsing of the XML input file is much more
faster, because the XML input file is smaller and only contains the list of
the other input files. Unfortunately, just before the end of the processing,
the Java VM is still using more of less the same size of memory (230 MB).
What is the reason for this ? Does I use correctly the document() function ?
Why is the memory not releazed after each input file processed. I know that
with Java, the memory is not released by the programmer but by a garbage
collector, and that the time when the garbage collector is performing is
works cannot be controlled in the code. Is there something to do with my
problem ?
I cannot afford to use so much memory just for this process. Another
alternative may be to process each of these 200 input files one after each
other and to concatenate each output into one unique file, but this solution
will be the one I will use only if there isn't any other alternative.
Kind regards,
Patrick Glur
Tel: ++41.21.732.05.05
Fax: ++41.21.732.01.00
E-mail : glur@nagra-kudelski.ch