[SOLVED] processing large files

["Boisvert, Éric" <eb...@nrcan.gc.ca>]

Thanks all for your suggestions, Nico's MultiFragmentTraxTransformer is what
I needed. I started from his component and created a new one specific for my
needs and improve 4-5 fold the performance.

Thanks all.

Eric

-----Message d'origine-----
De : Grzegorz Tañczyk [mailto:goliatus@polzone.pl]
Envoyé : 28 octobre, 2005 21:22
À : Boisvert, Éric
Objet : Re: processing large files


Hello Éric,

  If this file is not very complex and You don't need XPath to be
  happy then You should to consider writing custom Transformer or
  Generator. SAXParser interface is easy to use.

Friday, October 28, 2005, 5:30:50 PM, you wrote:
BÉ> Hi all

BÉ> I need to process large xml file and as I tested with increasingly
larger
BÉ> file, the time to process suddently increased a lot.  For instance, 200
K
BÉ> files took 0.8 seconds, 400 K file 2.5 sec and when I get near 1 Meg, it
BÉ> jumps to 30 seconds (nearly 10 times, for twice the size).. I played
with
BÉ> the pipeline caching, outputBufferSize, etc.. even boosted CATALINA_OPTS
to
BÉ> 512 Megs, nothing helped.  I guess this is related to the fact that at
some
BÉ> point the incoming document cannot be loaded entirely in memory.

BÉ> Anyone has an idea to fix this ?

BÉ> Cheers and thanks

BÉ> ================================================================
BÉ> Eric Boisvert
BÉ> Spécialiste TI-GI / IT-IM specialist
BÉ> Eric.Boisvert@rncan.gc.ca, 418-654-3705, facsimile/télécopieur 
BÉ> 418-654-2615
BÉ> 490, rue de la Couronne, Québec (Québec), G1K 9A9
BÉ> 490, rue de la Couronne, Quebec, Quebec, G1K 9A9

BÉ> Laboratoire de cartographie numérique et de photogrammétrie (LCNP)
BÉ> Digital Cartography and Photogrammetry Laboratory (DCPL)
BÉ> Commission géologique du Canada (Québec) / Geological Survey of Canada
BÉ> (Quebec)
BÉ> Ressources naturelles Canada / Natural Resources Canada
BÉ> Gouvernement du Canada / Government of Canada
BÉ> http://www.cgcq.rncan.gc.ca/lcnp
BÉ> http://www.nrcan.gc.ca/gsc



BÉ> ---------------------------------------------------------------------
BÉ> To unsubscribe, e-mail: users-unsubscribe@cocoon.apache.org
BÉ> For additional commands, e-mail: users-help@cocoon.apache.org



-- 
Best regards,
 Grzegorz                            mailto:goliatus@polzone.pl


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@cocoon.apache.org
For additional commands, e-mail: users-help@cocoon.apache.org

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@cocoon.apache.org
For additional commands, e-mail: users-help@cocoon.apache.org