Apache Camel as ETL

[contactreji <co...@gmail.com>]

Hi Folks

We are trying to implement some ETL ( batch mode ) use cases. These jobs
should be running at certain scheduled time and will be triggered from
Chronos scheduling framework in Marathon.

Once the job is finished, it should somehow intimate the Chronos as well so
that it can terminate the application as well.

How do we do achieve this? Any clue? Cos for me camel looks like something
which is engineered for more of real time integration flows with routes up
and running and listening for an external event data input for execution.

My requirement is - When triggered from Chronos, camel application should do
some processing on some file and once the fileprocessing is completed and
output is generated, chronos should be able to detect this and send a app
termination.

Cheers
Reji



-----
Reji Mathews
Sr. Developer - Middleware Integration / SOA ( Open Source - Apache Camel & Jboss Fuse ESB | Mule ESB )
LinkedIn - http://in.linkedin.com/pub/reji-mathews/31/9a2/40a
Twitter - reji_mathews
--
View this message in context: http://camel.465427.n5.nabble.com/Apache-Camel-as-ETL-tp5775073.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: Apache Camel as ETL

[souciance <so...@gmail.com>]

In general I would say integration is usually event-based and hopefully
asynchronous. It makes everything much easier to handle.

In your case it is batch oriented and you need to take action afterwards.

I am sure it can be achieved in Camel and you'd probably have to write
several customised code. The question is, whether Camel is the right tool
for this or whether it is better to use something that is designed for ETL
usage. There is the ETL pattern stated here:
http://camel.apache.org/etl-example.html

You can also off course try https://nifi.apache.org/ and see if having
another tool helps. I would say if you plan to do a lot of ETL it is better
to have a specific tool for this.



--
View this message in context: http://camel.465427.n5.nabble.com/Apache-Camel-as-ETL-tp5775073p5775126.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: Apache Camel as ETL

[Bilgin Ibryam <bi...@gmail.com>]

This may give you some ideas
http://stackoverflow.com/questions/20638680/camel-stop-when-there-are-no-files-in-the-folder

B.

On 15 December 2015 at 08:26, contactreji <co...@gmail.com> wrote:

> Hi Folks
>
> We are trying to implement some ETL ( batch mode ) use cases. These jobs
> should be running at certain scheduled time and will be triggered from
> Chronos scheduling framework in Marathon.
>
> Once the job is finished, it should somehow intimate the Chronos as well so
> that it can terminate the application as well.
>
> How do we do achieve this? Any clue? Cos for me camel looks like something
> which is engineered for more of real time integration flows with routes up
> and running and listening for an external event data input for execution.
>
> My requirement is - When triggered from Chronos, camel application should
> do
> some processing on some file and once the fileprocessing is completed and
> output is generated, chronos should be able to detect this and send a app
> termination.
>
> Cheers
> Reji
>
>
>
> -----
> Reji Mathews
> Sr. Developer - Middleware Integration / SOA ( Open Source - Apache Camel
> & Jboss Fuse ESB | Mule ESB )
> LinkedIn - http://in.linkedin.com/pub/reji-mathews/31/9a2/40a
> Twitter - reji_mathews
> --
> View this message in context:
> http://camel.465427.n5.nabble.com/Apache-Camel-as-ETL-tp5775073.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>



-- 
Bilgin Ibryam

Red Hat, Inc.
Apache Camel & Apache OFBiz committer
Blog: ofbizian.com
Twitter: @bibryam <https://twitter.com/bibryam>

Author of Instant Apache Camel Message Routing
http://www.amazon.com/dp/1783283475