unmarshal().gzip() question

[briane80 <be...@ccea.org.uk>]

Hi,

I'm dealing with some large xml files (300mb) that are gzipped (to around
3mb) in a folder.

I have a file route that reads from this folder then calls
unmarshal().gzip() to get the xml file for processing and the next stage of
the route is a beanRef.

My question is does the unmarshal().gzip() decompress the whole file in
memory?  I get java OOM exceptions when running on tomcat with Xmx1024m but
not if i increase it to Xmx4g.

Would a better way be to direct the gzip to a temporary folder then read
from the folder to a beanRef?



--
View this message in context: http://camel.465427.n5.nabble.com/unmarshal-gzip-question-tp5727782.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: unmarshal().gzip() question

[Claus Ibsen <cl...@gmail.com>]

On Tue, Feb 19, 2013 at 11:52 AM, briane80 <be...@ccea.org.uk> wrote:
> Hi,
>
> I'm dealing with some large xml files (300mb) that are gzipped (to around
> 3mb) in a folder.
>
> I have a file route that reads from this folder then calls
> unmarshal().gzip() to get the xml file for processing and the next stage of
> the route is a beanRef.
>
> My question is does the unmarshal().gzip() decompress the whole file in
> memory?  I get java OOM exceptions when running on tomcat with Xmx1024m but
> not if i increase it to Xmx4g.
>

The gzip data format is pure in-memory based.

We could possible look into enhancing it to allow to configure a work
directory / to stream to a file directly.
And then use a FileInputStream as the message body. And then when the
exchange is done, remembering to delete the file from the work dir.


> Would a better way be to direct the gzip to a temporary folder then read
> from the folder to a beanRef?
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/unmarshal-gzip-question-tp5727782.html
> Sent from the Camel - Users mailing list archive at Nabble.com.



-- 
Claus Ibsen
-----------------
Red Hat, Inc.
FuseSource is now part of Red Hat
Email: cibsen@redhat.com
Web: http://fusesource.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen

Re: unmarshal().gzip() question

[briane80 <be...@ccea.org.uk>]

Instead of using .unmarshall().gzip() I passed the message to a beanRef that
decompresses the payload to a temp file using a buffer.  Then created
another route to read from the temp file.  This gave much better memory
usage and I was able to handle 300mb -> 4mb compressed files without OOM
exceptions.

Is there an option to buffer/stream automatically when it comes to
unmarshalling?

@Handler
public void processMessage(Exchange message) throws Exception
{
			
	//Decompress the file to a temp folder
	File file = new File(message.getIn().getHeader("CamelFilePath",
String.class));
		
	GZIPInputStream in = null;
	OutputStream out = null;
	File target = null;
	try 
	{
		//Open the compressed file
		in = new GZIPInputStream(new FileInputStream(file));

		//Open the output file
		target = new File("C:\\A2CMessaging\\Temp",file.getName());
		target.createNewFile();
		out = new FileOutputStream(target);

		//Transfer bytes from the compressed file to the output file
		//Buffer of 1 mb.
		byte[] buf = new byte[1048576];
		int len;
		while ((len = in.read(buf)) > 0) 
		{
			out.write(buf, 0, len);
		}

		in.close();
		out.close();
	} 
	....... try/catch/finally etc
		
}





--
View this message in context: http://camel.465427.n5.nabble.com/unmarshal-gzip-question-tp5727782p5727804.html
Sent from the Camel - Users mailing list archive at Nabble.com.