You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by AndyBell <an...@dstglobalsolutions.com> on 2014/06/12 16:42:32 UTC
Logical Units of Work
Hi
I've not used Camel for a number of years and now have a (to me) non-trivial
task which I wonder if it can be done in Camel using standard camel
component:
I have a CSV file with structured data in it:
F1,xxx,,xxx,xxx
F2,XXX,XXX
F2,XXX,XXX
F1,xxx,,xxx,xxx
F2,XXX,XXX
F2,XXX,XXX
F2,XXX,XXX
There is a logical connection between the rows starting with F1 and the
subsequent F2 rows. So, in the example above, the first 3 rows would form a
logical unit of work and the final 4 rows would form a logical unit of work.
Can camel be configured to read in and process this data in LUW groups and
fail the whole LUW should any one of the rows in the LUW be in error?
Andy
--
View this message in context: http://camel.465427.n5.nabble.com/Logical-Units-of-Work-tp5752206.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: AW: Logical Units of Work
Posted by AndyBell <an...@dstglobalsolutions.com>.
Thank you both for the replies!
Andy
--
View this message in context: http://camel.465427.n5.nabble.com/Logical-Units-of-Work-tp5752206p5752243.html
Sent from the Camel - Users mailing list archive at Nabble.com.
AW: Logical Units of Work
Posted by "Jan Matèrne (jhm)" <ap...@materne.de>.
I did a sample.
Jan
/**
* user@camel.a.o (2014-06-12):
*
* I have a CSV file with structured data in it:
*
* F1,xxx,,xxx,xxx
* F2,XXX,XXX
* F2,XXX,XXX
* F1,xxx,,xxx,xxx
* F2,XXX,XXX
* F2,XXX,XXX
* F2,XXX,XXX
*
* There is a logical connection between the rows starting
* with F1 and the subsequent F2 rows. So, in the example above,
* the first 3 rows would form a logical unit of work and the
* final 4 rows would form a logical unit of work.
*
* Can camel be configured to read in and process this data in
* LUW groups and fail the whole LUW should any one of the
* rows in the LUW be in error?
*
* Andy
*
* @author Jan
*
*/
public class LogicalUnitOfWorkTest extends CamelTestSupport {
/*
* Base idea is adding an id to each line and changing the
* id value every time when an "F1" occurred.
* Then you could use an aggregator to collect them.
*/
@Override
protected RouteBuilder createRouteBuilder() throws Exception {
return new RouteBuilder() {
@Override
public void configure() throws Exception {
from("file:src/test/resources?fileName=logical-unit-of-work-data.csv&move=.done")
.split(body(String.class).tokenize("\n"))
.process(new WorkIdGenerator())
.aggregate(
header(WorkIdGenerator.WORK_ID),
AggregationStrategies.bean(LineAggregationStrategy.class, "append")
)
.completionTimeout(1000)
.to("mock:out")
//TODO try something with failing the whole logical-unit-of-work.
;
}
};
}
@Test
public void check() throws InterruptedException {
// 3 Exchanges: line0, line1-3, line4-9
getMockEndpoint("mock:out").expectedMessageCount(3);
File dir = new File("src/test/resources");
File bak = new File(dir, ".done");
File data = new File(bak, "logical-unit-of-work-data.csv");
try {
FileUtils.moveFileToDirectory(data, dir, false);
} catch (IOException e) {
//ignore
}
// Wait so file-component could scan.
Thread.sleep(5000);
assertMockEndpointsSatisfied();
}
}
/**
* Previously a splits the csv-file into lines.
* Use this strategy to combine the data with same
* work-id.
*/
public class LineAggregationStrategy {
public String append(String s1, String s2) {
return s1 + "\n" + s2;
}
}
/**
* Store a 'work-id' on each exchange.
* Value of that id is changed every time if a new
* "logical unit of work" starts (means, starts with 'F1').
*/
final class WorkIdGenerator implements Processor {
public static final String WORK_ID = "WorkID";
String id = next();
public String next() {
id = UUID.randomUUID().toString();
return id;
}
@Override
public void process(Exchange exchange) throws Exception {
if (exchange.getIn().getBody(String.class).startsWith("F1,")) {
next();
}
exchange.getIn().setHeader(WORK_ID, id);
}
}
> -----Ursprüngliche Nachricht-----
> Von: kraythe . [mailto:kraythe@gmail.com]
> Gesendet: Donnerstag, 12. Juni 2014 17:06
> An: Camel Users List
> Betreff: Re: Logical Units of Work
>
> You could use an aggregator to group the rows or use scatter gather
> (well gather since you are already scattered) EIP in order to correlate
> the rows.
> As for failing "in error" that rather depends on the error. If an
> exception is thrown that is not that hard. Gather all of the rows,
> split, process, aggregate. On the splitter share the unit of work and
> any exception in that unit of work will fail the exchange. The question
> is what happens after the exchange fails.
>
> *Robert Simmons Jr. MSc. - Lead Java Architect @ EA* *Author of:
> Hardcore Java (2003) and Maintainable Java (2012)*
> *LinkedIn: **http://www.linkedin.com/pub/robert-simmons/40/852/a39
> <http://www.linkedin.com/pub/robert-simmons/40/852/a39>*
>
>
> On Thu, Jun 12, 2014 at 9:42 AM, AndyBell
> <an...@dstglobalsolutions.com>
> wrote:
>
> > Hi
> >
> > I've not used Camel for a number of years and now have a (to me)
> > non-trivial task which I wonder if it can be done in Camel using
> > standard camel
> > component:
> >
> > I have a CSV file with structured data in it:
> >
> > F1,xxx,,xxx,xxx
> > F2,XXX,XXX
> > F2,XXX,XXX
> > F1,xxx,,xxx,xxx
> > F2,XXX,XXX
> > F2,XXX,XXX
> > F2,XXX,XXX
> >
> > There is a logical connection between the rows starting with F1 and
> > the subsequent F2 rows. So, in the example above, the first 3 rows
> > would form a logical unit of work and the final 4 rows would form a
> > logical unit of work.
> >
> > Can camel be configured to read in and process this data in LUW
> groups
> > and fail the whole LUW should any one of the rows in the LUW be in
> error?
> >
> > Andy
> >
> >
> >
> > --
> > View this message in context:
> > http://camel.465427.n5.nabble.com/Logical-Units-of-Work-
> tp5752206.html
> > Sent from the Camel - Users mailing list archive at Nabble.com.
> >
Re: Logical Units of Work
Posted by "kraythe ." <kr...@gmail.com>.
You could use an aggregator to group the rows or use scatter gather (well
gather since you are already scattered) EIP in order to correlate the rows.
As for failing "in error" that rather depends on the error. If an exception
is thrown that is not that hard. Gather all of the rows, split, process,
aggregate. On the splitter share the unit of work and any exception in that
unit of work will fail the exchange. The question is what happens after the
exchange fails.
*Robert Simmons Jr. MSc. - Lead Java Architect @ EA*
*Author of: Hardcore Java (2003) and Maintainable Java (2012)*
*LinkedIn: **http://www.linkedin.com/pub/robert-simmons/40/852/a39
<http://www.linkedin.com/pub/robert-simmons/40/852/a39>*
On Thu, Jun 12, 2014 at 9:42 AM, AndyBell <an...@dstglobalsolutions.com>
wrote:
> Hi
>
> I've not used Camel for a number of years and now have a (to me)
> non-trivial
> task which I wonder if it can be done in Camel using standard camel
> component:
>
> I have a CSV file with structured data in it:
>
> F1,xxx,,xxx,xxx
> F2,XXX,XXX
> F2,XXX,XXX
> F1,xxx,,xxx,xxx
> F2,XXX,XXX
> F2,XXX,XXX
> F2,XXX,XXX
>
> There is a logical connection between the rows starting with F1 and the
> subsequent F2 rows. So, in the example above, the first 3 rows would form a
> logical unit of work and the final 4 rows would form a logical unit of
> work.
>
> Can camel be configured to read in and process this data in LUW groups and
> fail the whole LUW should any one of the rows in the LUW be in error?
>
> Andy
>
>
>
> --
> View this message in context:
> http://camel.465427.n5.nabble.com/Logical-Units-of-Work-tp5752206.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>