is it better to use aggregator to write a large file hourly using messages from active mq queue?

[beschi <be...@gmail.com>]

I have a requirement of creating a large file hourly, the content of the
files will be the records/line from active mq queue.

I am using the following code, is there any better option?

		<route id="largeFileWritter">
			<from uri="activemq:queue:so.file.req.in"/>
			<log loggingLevel="INFO" message="Adding record to  File ${body}" />
			<camel:setHeader headerName="fileName">
				<camel:simple>Test File_${date:now:yyyyMMddHHmmssSSS}.txt</camel:simple>
			</camel:setHeader>
			<aggregate strategyRef="aggregatorStrategy" forceCompletionOnStop="true"
completionInterval="8600000">
            	<correlationExpression>
                	<constant>true</constant>
            	</correlationExpression>
            	<to
uri="file://d:/temp/temp1?fileExist=Append&amp;fileName=${headers.fileName}"
/>
        	</aggregate>
		</route>



--
View this message in context: http://camel.465427.n5.nabble.com/is-it-better-to-use-aggregator-to-write-a-large-file-hourly-using-messages-from-active-mq-queue-tp5791784.html
Sent from the Camel Development mailing list archive at Nabble.com.

Re: is it better to use aggregator to write a large file hourly using messages from active mq queue?

[Quinn Stevenson <qu...@pronoia-solutions.com>]

For a requirement such as this, I would probably schedule a route to start hourly and write the file rather than use an aggregator.  The reason for this is I normally can’t lose any messages and if I use an aggregator I need to have a persistent aggregation repository as well to ensure I don’t lose any message.  Since my messages are already persisted in the JMS store, I don’t really see much value in moving them from one store to another, just to hold them until I write the file.  

For this use case, you may also be able to just append to a file since your filename has the timestamp in it - i.e use a fileName pattern like yyyyMMddHH.txt and just append the records to it as they come in.  I don’t know if that would satisfy your use case or not.

As far as whether or not it’s better to use an aggregator - it depends on the use case.  The use case you’ve given can be accomplished without an aggregator, but I’ve had others that require an aggregator.  For me, its really a matter of what method fits the use case better.

HTH

> On Dec 20, 2016, at 10:42 PM, beschi <be...@gmail.com> wrote:
> 
> I have a requirement of creating a large file hourly, the content of the
> files will be the records/line from active mq queue.
> 
> I am using the following code, is there any better option?
> 
> 		<route id="largeFileWritter">
> 			<from uri="activemq:queue:so.file.req.in"/>
> 			<log loggingLevel="INFO" message="Adding record to  File ${body}" />
> 			<camel:setHeader headerName="fileName">
> 				<camel:simple>Test File_${date:now:yyyyMMddHHmmssSSS}.txt</camel:simple>
> 			</camel:setHeader>
> 			<aggregate strategyRef="aggregatorStrategy" forceCompletionOnStop="true"
> completionInterval="8600000">
>            	<correlationExpression>
>                	<constant>true</constant>
>            	</correlationExpression>
>            	<to
> uri="file://d:/temp/temp1?fileExist=Append&amp;fileName=${headers.fileName}"
> />
>        	</aggregate>
> 		</route>
> 
> 
> 
> --
> View this message in context: http://camel.465427.n5.nabble.com/is-it-better-to-use-aggregator-to-write-a-large-file-hourly-using-messages-from-active-mq-queue-tp5791784.html
> Sent from the Camel Development mailing list archive at Nabble.com.