Robustly streaming messages to archive

[Baris Acar <ba...@acar.org.uk>]

Hi,
We use camel in a standalone app which responds to a user request by sending some messages via JMS. To aid with troubleshooting/support, we also save these messages to disk using the 'file:' endpoint. 

Unfortunately, since there are many tens of thousands of smallish (5-10KiB) messages per user request, the folders are quite unwieldy to deal with. e.g. transferring the saved messages from a network share is particularly slow compared to transferring a single larger file.

We tried aggregating all such messages for a user request and creating a zip archive on completion; however this has the drawback that nothing is logged if the process crashes or is forcibly terminated partway through a request.

Is there a camel-y solution to this problem? At the moment we are thinking of writing the messages into a single file via a tar stream, and possibly subsequently a gzip stream. My understanding is that, unlike .zip, both .tar and .gz support streaming (i.e. can tolerate partially complete archives), although I'm not an expert. I was wondering how the community has solved this problem?

Thanks,
Barış