You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by thabach <ba...@evolootion.ch> on 2009/03/07 15:31:20 UTC
Camel in Batch Processing.
Heya
would you recommend using Camel in a Batch Processing scenario involving the
reading of some files at the end of a business day, combining data and
producing some output file ?
Is there a sample, tutorial or any pointers on how such batch processing
could be bootstrapped (with reference data for validation, enhancement) and
triggered ?
Thanks, Christian
--
View this message in context: http://www.nabble.com/Camel-in-Batch-Processing.-tp22388071p22388071.html
Sent from the Camel - Users (activemq) mailing list archive at Nabble.com.
Re: Camel in Batch Processing.
Posted by Adrian Trenaman <tr...@progress.com>.
Hi there Christian,
Sounds like the "event" that you're processing is not the appearance
of a single file, but rather a nightly occurrence which then requires
the processing of many files. Perhaps you could trigger the event
using a timer or quartz endpoint, and then do the work inside a custom
processor? Something like
from ("timer://....")
.process(new Processor() {
public void process(Exchange ex) {
// Read in all my files
// Do some work
// Write to DB
}
});
Of course, you could break up the custom processor into a number of
processors if that made it easier.
Just a thought!
/Ade
On 10 Mar 2009, at 17:27, thabach wrote:
>
> Hi Adrian
>
> I watched your presentation, very nice indeed, but my scenario is more
> "batch" in nature. The piece of software I envision implementing in
> Camel is
> pretty stateful, one could say. On every trigger (initiated by a
> timer or by
> other means) the integration logic is supposed to bootstrap itself
> with
> reference data from some file(s). After having done that, it is
> supposed to
> correlate data served from some other files and do some calculations
> on the
> "loaded" data based on some criteria of the previously loaded
> reference
> data. In general processing can only start after all sources have been
> loaded into memory entirely. Does it make sense to try to model such a
> scenario with flows (and thus Camel) at all ? Still I was
> envisioning to
> use all the nice Camel integration endpoints in interacting with the
> outside
> world :)
>
> Cheers and thank you for your advice, I appreciate.
>
> Christian.
>
>
>
> Adrian Trenaman-2 wrote:
>>
>> Hi Christian,
>>
>> I've got a sample of a batch file/ftp processor in my demo up on
>>
>>
>> http://fusesource.com/wiki/display/ProdInfo/FUSE+ESB+4+Getting+Started+Tutorial+Downloads
>>
>> Maybe that might help?
>>
>> /Ade
>>
>> On 7 Mar 2009, at 14:31, thabach wrote:
>>
>>>
>>> Heya
>>>
>>> would you recommend using Camel in a Batch Processing scenario
>>> involving the
>>> reading of some files at the end of a business day, combining data
>>> and
>>> producing some output file ?
>>>
>>> Is there a sample, tutorial or any pointers on how such batch
>>> processing
>>> could be bootstrapped (with reference data for validation,
>>> enhancement) and
>>> triggered ?
>>>
>>> Thanks, Christian
>>> --
>>> View this message in context:
>>> http://www.nabble.com/Camel-in-Batch-Processing.-tp22388071p22388071.html
>>> Sent from the Camel - Users (activemq) mailing list archive at
>>> Nabble.com.
>>>
>>
>> ---
>> Adrian Trenaman, Consultant Fellow, PS - Opensource Center of
>> Competence
>> Progress Software Corp
>> Shelbourne Road, Dublin 4, Ireland
>> ---
>> +353-1-637-2659 (Office)
>> +353-1-637-2882 (Fax)
>> +353-86-6051026 (Mobile)
>> adrian.trenaman (Skype)
>> ----
>> Blog: http://trenaman.blogspot.com
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>
> --
> View this message in context: http://www.nabble.com/Camel-in-Batch-Processing.-tp22388071p22439382.html
> Sent from the Camel - Users (activemq) mailing list archive at
> Nabble.com.
>
---
Adrian Trenaman, Consultant Fellow, PS - Opensource Center of Competence
Progress Software Corp
Shelbourne Road, Dublin 4, Ireland
---
+353-1-637-2659 (Office)
+353-1-637-2882 (Fax)
+353-86-6051026 (Mobile)
adrian.trenaman (Skype)
----
Blog: http://trenaman.blogspot.com
Re: Camel in Batch Processing.
Posted by thabach <ba...@evolootion.ch>.
Hi Adrian
I watched your presentation, very nice indeed, but my scenario is more
"batch" in nature. The piece of software I envision implementing in Camel is
pretty stateful, one could say. On every trigger (initiated by a timer or by
other means) the integration logic is supposed to bootstrap itself with
reference data from some file(s). After having done that, it is supposed to
correlate data served from some other files and do some calculations on the
"loaded" data based on some criteria of the previously loaded reference
data. In general processing can only start after all sources have been
loaded into memory entirely. Does it make sense to try to model such a
scenario with flows (and thus Camel) at all ? Still I was envisioning to
use all the nice Camel integration endpoints in interacting with the outside
world :)
Cheers and thank you for your advice, I appreciate.
Christian.
Adrian Trenaman-2 wrote:
>
> Hi Christian,
>
> I've got a sample of a batch file/ftp processor in my demo up on
>
>
> http://fusesource.com/wiki/display/ProdInfo/FUSE+ESB+4+Getting+Started+Tutorial+Downloads
>
> Maybe that might help?
>
> /Ade
>
> On 7 Mar 2009, at 14:31, thabach wrote:
>
>>
>> Heya
>>
>> would you recommend using Camel in a Batch Processing scenario
>> involving the
>> reading of some files at the end of a business day, combining data and
>> producing some output file ?
>>
>> Is there a sample, tutorial or any pointers on how such batch
>> processing
>> could be bootstrapped (with reference data for validation,
>> enhancement) and
>> triggered ?
>>
>> Thanks, Christian
>> --
>> View this message in context:
>> http://www.nabble.com/Camel-in-Batch-Processing.-tp22388071p22388071.html
>> Sent from the Camel - Users (activemq) mailing list archive at
>> Nabble.com.
>>
>
> ---
> Adrian Trenaman, Consultant Fellow, PS - Opensource Center of Competence
> Progress Software Corp
> Shelbourne Road, Dublin 4, Ireland
> ---
> +353-1-637-2659 (Office)
> +353-1-637-2882 (Fax)
> +353-86-6051026 (Mobile)
> adrian.trenaman (Skype)
> ----
> Blog: http://trenaman.blogspot.com
>
>
>
>
>
>
>
>
>
>
--
View this message in context: http://www.nabble.com/Camel-in-Batch-Processing.-tp22388071p22439382.html
Sent from the Camel - Users (activemq) mailing list archive at Nabble.com.
Re: Camel in Batch Processing.
Posted by Adrian Trenaman <tr...@progress.com>.
Hi Christian,
I've got a sample of a batch file/ftp processor in my demo up on
http://fusesource.com/wiki/display/ProdInfo/FUSE+ESB+4+Getting+Started+Tutorial+Downloads
Maybe that might help?
/Ade
On 7 Mar 2009, at 14:31, thabach wrote:
>
> Heya
>
> would you recommend using Camel in a Batch Processing scenario
> involving the
> reading of some files at the end of a business day, combining data and
> producing some output file ?
>
> Is there a sample, tutorial or any pointers on how such batch
> processing
> could be bootstrapped (with reference data for validation,
> enhancement) and
> triggered ?
>
> Thanks, Christian
> --
> View this message in context: http://www.nabble.com/Camel-in-Batch-Processing.-tp22388071p22388071.html
> Sent from the Camel - Users (activemq) mailing list archive at
> Nabble.com.
>
---
Adrian Trenaman, Consultant Fellow, PS - Opensource Center of Competence
Progress Software Corp
Shelbourne Road, Dublin 4, Ireland
---
+353-1-637-2659 (Office)
+353-1-637-2882 (Fax)
+353-86-6051026 (Mobile)
adrian.trenaman (Skype)
----
Blog: http://trenaman.blogspot.com