You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cocoon.apache.org by "Boisvert, Éric" <eb...@nrcan.gc.ca> on 2005/10/28 17:30:50 UTC
processing large files
Hi all
I need to process large xml file and as I tested with increasingly larger
file, the time to process suddently increased a lot. For instance, 200 K
files took 0.8 seconds, 400 K file 2.5 sec and when I get near 1 Meg, it
jumps to 30 seconds (nearly 10 times, for twice the size).. I played with
the pipeline caching, outputBufferSize, etc.. even boosted CATALINA_OPTS to
512 Megs, nothing helped. I guess this is related to the fact that at some
point the incoming document cannot be loaded entirely in memory.
Anyone has an idea to fix this ?
Cheers and thanks
================================================================
Eric Boisvert
Spécialiste TI-GI / IT-IM specialist
Eric.Boisvert@rncan.gc.ca, 418-654-3705, facsimile/télécopieur
418-654-2615
490, rue de la Couronne, Québec (Québec), G1K 9A9
490, rue de la Couronne, Quebec, Quebec, G1K 9A9
Laboratoire de cartographie numérique et de photogrammétrie (LCNP)
Digital Cartography and Photogrammetry Laboratory (DCPL)
Commission géologique du Canada (Québec) / Geological Survey of Canada
(Quebec)
Ressources naturelles Canada / Natural Resources Canada
Gouvernement du Canada / Government of Canada
http://www.cgcq.rncan.gc.ca/lcnp
http://www.nrcan.gc.ca/gsc
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@cocoon.apache.org
For additional commands, e-mail: users-help@cocoon.apache.org
Re: processing large files
Posted by Ralph Goers <Ra...@dslextreme.com>.
Boisvert, Éric wrote:
>Hi all
>
>I need to process large xml file and as I tested with increasingly larger
>file, the time to process suddently increased a lot. For instance, 200 K
>files took 0.8 seconds, 400 K file 2.5 sec and when I get near 1 Meg, it
>jumps to 30 seconds (nearly 10 times, for twice the size).. I played with
>the pipeline caching, outputBufferSize, etc.. even boosted CATALINA_OPTS to
>512 Megs, nothing helped. I guess this is related to the fact that at some
>point the incoming document cannot be loaded entirely in memory.
>
>Anyone has an idea to fix this ?
>
>Cheers and thanks
>
You will probably also want to listen to it.
http://cocoongt.hippo12.castaserver.com/cocoongt/audio/gt-11-nico.mp3
Ralph
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@cocoon.apache.org
For additional commands, e-mail: users-help@cocoon.apache.org
Re: processing large files
Posted by Grzegorz Tañczyk <go...@polzone.pl>.
Hello Éric,
If this file is not very complex and You don't need XPath to be
happy then You should to consider writing custom Transformer or
Generator. SAXParser interface is easy to use.
Friday, October 28, 2005, 5:30:50 PM, you wrote:
BÉ> Hi all
BÉ> I need to process large xml file and as I tested with increasingly larger
BÉ> file, the time to process suddently increased a lot. For instance, 200 K
BÉ> files took 0.8 seconds, 400 K file 2.5 sec and when I get near 1 Meg, it
BÉ> jumps to 30 seconds (nearly 10 times, for twice the size).. I played with
BÉ> the pipeline caching, outputBufferSize, etc.. even boosted CATALINA_OPTS to
BÉ> 512 Megs, nothing helped. I guess this is related to the fact that at some
BÉ> point the incoming document cannot be loaded entirely in memory.
BÉ> Anyone has an idea to fix this ?
BÉ> Cheers and thanks
BÉ> ================================================================
BÉ> Eric Boisvert
BÉ> Spécialiste TI-GI / IT-IM specialist
BÉ> Eric.Boisvert@rncan.gc.ca, 418-654-3705, facsimile/télécopieur
BÉ> 418-654-2615
BÉ> 490, rue de la Couronne, Québec (Québec), G1K 9A9
BÉ> 490, rue de la Couronne, Quebec, Quebec, G1K 9A9
BÉ> Laboratoire de cartographie numérique et de photogrammétrie (LCNP)
BÉ> Digital Cartography and Photogrammetry Laboratory (DCPL)
BÉ> Commission géologique du Canada (Québec) / Geological Survey of Canada
BÉ> (Quebec)
BÉ> Ressources naturelles Canada / Natural Resources Canada
BÉ> Gouvernement du Canada / Government of Canada
BÉ> http://www.cgcq.rncan.gc.ca/lcnp
BÉ> http://www.nrcan.gc.ca/gsc
BÉ> ---------------------------------------------------------------------
BÉ> To unsubscribe, e-mail: users-unsubscribe@cocoon.apache.org
BÉ> For additional commands, e-mail: users-help@cocoon.apache.org
--
Best regards,
Grzegorz mailto:goliatus@polzone.pl
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@cocoon.apache.org
For additional commands, e-mail: users-help@cocoon.apache.org
Re: processing large files
Posted by Ralph Goers <Ra...@dslextreme.com>.
Boisvert, Éric wrote:
>Hi all
>
>I need to process large xml file and as I tested with increasingly larger
>file, the time to process suddently increased a lot. For instance, 200 K
>files took 0.8 seconds, 400 K file 2.5 sec and when I get near 1 Meg, it
>jumps to 30 seconds (nearly 10 times, for twice the size).. I played with
>the pipeline caching, outputBufferSize, etc.. even boosted CATALINA_OPTS to
>512 Megs, nothing helped. I guess this is related to the fact that at some
>point the incoming document cannot be loaded entirely in memory.
>
>Anyone has an idea to fix this ?
>
>Cheers and thanks
>
>
>
This was the subject of one of the presentations at the Cocoon
GetTogether (http://www.cocoongt.org). Here is a link to the
presentation.
http://cocoongt.hippo12.castaserver.com/cocoongt/nico-verwer-performance.pdf
Ralph
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@cocoon.apache.org
For additional commands, e-mail: users-help@cocoon.apache.org