Flatpack and aggregation

["Tom.Fornoville" <to...@roots.be>]

'm relatively new to Camel and guessing my problem can be solved with a
combination of FlatPack and an aggregation but I'm not seeing how I can make
it work without writing a lot of code.

It starts with a fixed width file with 2 columns (actually more but only 2
are relevant): action_id and organisation_id. The file contains a couple
thousand lines and can contain duplicates.

2 things needed:

filter out lines for a specific action_id (only action_id 70 is of interest
for us)
deduplicate based on organisation_id (the same organisation can have appear
on multiple lines)
It will be greatly appreciated if anyone has a solution or can point me in
the right direction .

Cheers, Tom



--
View this message in context: http://camel.465427.n5.nabble.com/Flatpack-and-aggregation-tp5736837.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: Flatpack and aggregation

["Tom.Fornoville" <to...@roots.be>]

Hi Zemian,

Thanks for putting me on the right track with the idempotentConsumer!

Since Flatpack puts the records in a Java Map in the Body I had to do a
convertTo and ended up with the following:

<bean id="organisationMemoryRepository"
class="org.apache.camel.processor.idempotent.MemoryIdempotentRepository" />
...
<route id="process-line">
<from uri="flatpack:fixed:kbo.pzmap.xml" />
<convertBodyTo type="java.util.Map"/>
<filter>
<ognl>request.body["ACTION"] == "070"</ognl>
<idempotentConsumer messageIdRepositoryRef="organisationMemoryRepository">
<ognl>request.body["ORGANISATION"]</ognl>
<to uri="log:be.roots.organisation?level=INFO" />
<!-- TODO: send it to the webservice -->
</idempotentConsumer>
</filter>
</route>

This seems to work as expected :-)

Cheers,
Tom

Tom Fornoville
Senior Developer
m: +32 478 65 86 51
www.roots.be


On Wed, Aug 7, 2013 at 3:46 AM, saltnlight5 [via Camel] <
ml-node+s465427n5736874h87@n5.nabble.com> wrote:

> Hello Tom,
>
> I think you can try flatpack with one of the idempotent consumer (
> http://camel.apache.org/idempotent-consumer.html) to eliminate duplicated
> messages. For example, maybe something like this:
>
>   from("file:inbox")
>     .to("flatpack:fixed:yourdata.pzmap.xml")
>     .filter(header("action_id"))
>     .idempotentConsumer(header("organisation_id"),
> MemoryIdempotentRepository.memoryIdempotentRepository(200))
>
> Also alternative to flatpack, you may also checkout Camel bindy:
> http://camel.apache.org/bindy.html to parse fixed length records.
>
> Have fun,
> Zemian
>
> ------------------------------
>  If you reply to this email, your message will be added to the discussion
> below:
>
> http://camel.465427.n5.nabble.com/Flatpack-and-aggregation-tp5736837p5736874.html
>  To unsubscribe from Flatpack and aggregation, click here<http://camel.465427.n5.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=5736837&code=dG9tLmZvcm5vdmlsbGVAcm9vdHMuYmV8NTczNjgzN3wxNzA0MjEyMTky>
> .
> NAML<http://camel.465427.n5.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>




--
View this message in context: http://camel.465427.n5.nabble.com/Flatpack-and-aggregation-tp5736837p5736914.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: Flatpack and aggregation

[saltnlight5 <sa...@gmail.com>]

Hello Tom,

I think you can try flatpack with one of the idempotent consumer
(http://camel.apache.org/idempotent-consumer.html) to eliminate duplicated
messages. For example, maybe something like this:

  from("file:inbox")
    .to("flatpack:fixed:yourdata.pzmap.xml")
    .filter(header("action_id"))
    .idempotentConsumer(header("organisation_id"),
MemoryIdempotentRepository.memoryIdempotentRepository(200))

Also alternative to flatpack, you may also checkout Camel bindy:
http://camel.apache.org/bindy.html to parse fixed length records.

Have fun,
Zemian



--
View this message in context: http://camel.465427.n5.nabble.com/Flatpack-and-aggregation-tp5736837p5736874.html
Sent from the Camel - Users mailing list archive at Nabble.com.