You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by "kraythe ." <kr...@gmail.com> on 2013/12/06 02:36:21 UTC
File aggregation and transaction.
I have a route that I will past below. The route is transacted and I want
it to read off a queue and then batch the records into a file. However, if
someone murders the server with kill -9 prior to the file being written, I
would like all of those records to end up back in the queue to be processed
next time the server goes up. That is not the behavior we are seeing. If we
kill the server any records in transit are gobbled and dropped and the file
not written. What are we doing wrong? Note this is a method that generates
routes for us when we call it. Thanks a bunch:
protected void publishExcelFileForRecordQueue(final String queue,
finalString routeID,
final String path,
final long batchcompletionTimeout, final Comparator<Record> sorter,
final MappedExcelDataFormat medf, final String fileNameExpr) {
from(endpointAMQ(queue)) // Consume routed cases
.routeId(routeID) // Route Id for this route.
.onException(Exception.class).useOriginalMessage().handled(true).maximumRedeliveries(0)
// catch exceptions
.setHeader(Exchange.FAILURE_ROUTE_ID, property(Exchange.
FAILURE_ROUTE_ID)) // set route that errored
.setHeader(Exchange.EXCEPTION_CAUGHT, simple(
"${exception.stacktrace}")) // Store reason for error
.to(ExchangePattern.InOnly,
endpointAMQ(QUEUE_CASE_AUTOMATION_DEAD)).end()
// to DLQ
.transacted(KEY_TXPOLICY_REQUIRED) // make the route transacted
.unmarshal(new JacksonDataFormat(this.jacksonMapper, Record.class)) //
unmarshal from JSON
.aggregate(constant(true), newRecordListAggregationStrategy(sorter)) //
Aggregate and sort records.
.completionTimeout(batchcompletionTimeout) // Make as single report
.marshal(medf) // Write the agregated exchanges to excel
.to("file:" + path + "?fileName=" + fileNameExpr +
"&tempPrefix=.medf.").id("file_writer") // write the data to the file.
.end();
}
}
*Robert Simmons Jr. MSc. - Lead Java Architect @ EA*
*Author of: Hardcore Java (2003) and Maintainable Java (2012)*
*LinkedIn: **http://www.linkedin.com/pub/robert-simmons/40/852/a39
<http://www.linkedin.com/pub/robert-simmons/40/852/a39>*
Re: File aggregation and transaction.
Posted by "kraythe ." <kr...@gmail.com>.
I am actually using activemq component.
- Robert
On Friday, December 6, 2013, Claus Ibsen wrote:
> camel-jms using spring jms does not support batch for transactions.
> Its consuming a single message at a time.
>
> camel-sjms has batch support on the consumer. You may take a look at
> http://camel.apache.org/sjms
>
>
> On Fri, Dec 6, 2013 at 2:36 AM, kraythe . <kraythe@gmail.com<javascript:;>>
> wrote:
> > I have a route that I will past below. The route is transacted and I want
> > it to read off a queue and then batch the records into a file. However,
> if
> > someone murders the server with kill -9 prior to the file being written,
> I
> > would like all of those records to end up back in the queue to be
> processed
> > next time the server goes up. That is not the behavior we are seeing. If
> we
> > kill the server any records in transit are gobbled and dropped and the
> file
> > not written. What are we doing wrong? Note this is a method that
> generates
> > routes for us when we call it. Thanks a bunch:
> >
> > protected void publishExcelFileForRecordQueue(final String queue,
> > finalString routeID,
> > final String path,
> >
> > final long batchcompletionTimeout, final Comparator<Record> sorter,
> > final MappedExcelDataFormat medf, final String fileNameExpr) {
> >
> > from(endpointAMQ(queue)) // Consume routed cases
> >
> > .routeId(routeID) // Route Id for this route.
> >
> >
> .onException(Exception.class).useOriginalMessage().handled(true).maximumRedeliveries(0)
> > // catch exceptions
> >
> > .setHeader(Exchange.FAILURE_ROUTE_ID, property(Exchange.
> > FAILURE_ROUTE_ID)) // set route that errored
> >
> > .setHeader(Exchange.EXCEPTION_CAUGHT, simple(
> > "${exception.stacktrace}")) // Store reason for error
> >
> > .to(ExchangePattern.InOnly,
> > endpointAMQ(QUEUE_CASE_AUTOMATION_DEAD)).end()
> > // to DLQ
> >
> > .transacted(KEY_TXPOLICY_REQUIRED) // make the route transacted
> >
> > .unmarshal(new JacksonDataFormat(this.jacksonMapper,
> Record.class)) //
> > unmarshal from JSON
> >
> > .aggregate(constant(true),
> newRecordListAggregationStrategy(sorter)) //
> > Aggregate and sort records.
> >
> > .completionTimeout(batchcompletionTimeout) // Make as single
> report
> >
> > .marshal(medf) // Write the agregated exchanges to excel
> >
> > .to("file:" + path + "?fileName=" + fileNameExpr +
> > "&tempPrefix=.medf.").id("file_writer") // write the data to the file.
> >
> > .end();
> >
> > }
> >
> > }
> >
> > *Robert Simmons Jr. MSc. - Lead Java Architect @ EA*
> > *Author of: Hardcore Java (2003) and Maintainable Java (2012)*
> > *LinkedIn: **http://www.linkedin.com/pub/robert-simmons/40/852/a39
> > <http://www.linkedin.com/pub/robert-simmons/40/852/a39>*
>
>
>
> --
> Claus Ibsen
> -----------------
> Red Hat, Inc.
> Email: cibsen@redhat.com <javascript:;>
> Twitter: davsclaus
> Blog: http://davsclaus.com
> Author of Camel in Action: http://www.manning.com/ibsen
>
--
*Robert Simmons Jr. MSc. - Lead Java Architect @ EA*
*Author of: Hardcore Java (2003) and Maintainable Java (2012)*
*LinkedIn: **http://www.linkedin.com/pub/robert-simmons/40/852/a39
<http://www.linkedin.com/pub/robert-simmons/40/852/a39>*
Re: File aggregation and transaction.
Posted by Claus Ibsen <cl...@gmail.com>.
camel-jms using spring jms does not support batch for transactions.
Its consuming a single message at a time.
camel-sjms has batch support on the consumer. You may take a look at
http://camel.apache.org/sjms
On Fri, Dec 6, 2013 at 2:36 AM, kraythe . <kr...@gmail.com> wrote:
> I have a route that I will past below. The route is transacted and I want
> it to read off a queue and then batch the records into a file. However, if
> someone murders the server with kill -9 prior to the file being written, I
> would like all of those records to end up back in the queue to be processed
> next time the server goes up. That is not the behavior we are seeing. If we
> kill the server any records in transit are gobbled and dropped and the file
> not written. What are we doing wrong? Note this is a method that generates
> routes for us when we call it. Thanks a bunch:
>
> protected void publishExcelFileForRecordQueue(final String queue,
> finalString routeID,
> final String path,
>
> final long batchcompletionTimeout, final Comparator<Record> sorter,
> final MappedExcelDataFormat medf, final String fileNameExpr) {
>
> from(endpointAMQ(queue)) // Consume routed cases
>
> .routeId(routeID) // Route Id for this route.
>
> .onException(Exception.class).useOriginalMessage().handled(true).maximumRedeliveries(0)
> // catch exceptions
>
> .setHeader(Exchange.FAILURE_ROUTE_ID, property(Exchange.
> FAILURE_ROUTE_ID)) // set route that errored
>
> .setHeader(Exchange.EXCEPTION_CAUGHT, simple(
> "${exception.stacktrace}")) // Store reason for error
>
> .to(ExchangePattern.InOnly,
> endpointAMQ(QUEUE_CASE_AUTOMATION_DEAD)).end()
> // to DLQ
>
> .transacted(KEY_TXPOLICY_REQUIRED) // make the route transacted
>
> .unmarshal(new JacksonDataFormat(this.jacksonMapper, Record.class)) //
> unmarshal from JSON
>
> .aggregate(constant(true), newRecordListAggregationStrategy(sorter)) //
> Aggregate and sort records.
>
> .completionTimeout(batchcompletionTimeout) // Make as single report
>
> .marshal(medf) // Write the agregated exchanges to excel
>
> .to("file:" + path + "?fileName=" + fileNameExpr +
> "&tempPrefix=.medf.").id("file_writer") // write the data to the file.
>
> .end();
>
> }
>
> }
>
> *Robert Simmons Jr. MSc. - Lead Java Architect @ EA*
> *Author of: Hardcore Java (2003) and Maintainable Java (2012)*
> *LinkedIn: **http://www.linkedin.com/pub/robert-simmons/40/852/a39
> <http://www.linkedin.com/pub/robert-simmons/40/852/a39>*
--
Claus Ibsen
-----------------
Red Hat, Inc.
Email: cibsen@redhat.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen