You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by "Mills, Gary (GE Energy Management, consultant)" <ga...@ge.com> on 2015/06/30 23:57:36 UTC
newbie implementation question
Hello,
Thanks in advance, forgive me as I a learning. I do try to look these things up also, but with user feedback this greatly helps point me in the right direction.
I have 4 files to pick up from FTP. I need to join these files together to achieve 1 single file. Each file has a unique header. These Headers need to be aligned. So the aggregation would need to join the files at the end and beginning of each header. The content of the 1 single file I need to sort by 2 elements, each row is terminated by new line \n. sort content by a numeric Id value and its timestamp value( field in record ). I am looking and it appears that camel can accomplish this without much coding. Am I correct or do I need to code this?
I am thinking:
Import and save the files on the file system. Make sure I have all 4 files. If I don't have all 4 then wait until next poll and hopefully get the remaining files. The files are said to be pretty much guaranteed to come in at specific times. I just want to code for the unexpected event that they are not all present.
So, add FTP endpoint, add FTP route, poll at 4 pm est, and at 4 am est ( I don't know how to do this or if it can be done in camel ). Then through the route save the files on the file system. Do another route with aggregation strategy that will complete on 4 files and the append of the files will need to start with each header ( I do not know how to do this or if it is possible with camel ).
Msg 1 header A B C
1 2 3 ( data )
1 2 3
Msg 2 header D E F
4 5 6
4 5 6
Msg 3 header G H I
7 8 9
7 8 9
Msg 4 header J K L
10 11 12
10 11 12
RESULT ===========> A B C D E F G H I J K L
1 2 3 4 5 6 7 8 9 10 11 12
1 2 3 4 5 6 7 8 9 10 11 12
1 2 3 4 5 6 7 8 9 10 11 12
Then with another route perform a sort on 2 fields, 1 == asset Id and the 2nd a timestamp.
Please offer suggestions or let me know if this is possible with Camel with little coding. Thanks again!!!
Gary
RE: newbie implementation question
Posted by "Mills, Gary (GE Energy Management, consultant)" <ga...@ge.com>.
Thank you Claus, greatly appreciated.
Have a great day!
-----Original Message-----
From: Claus Ibsen [mailto:claus.ibsen@gmail.com]
Sent: Wednesday, July 01, 2015 10:39 AM
To: users@camel.apache.org
Subject: Re: newbie implementation question
Hi
Yeah split into 2 or more parts. The ftp that downloads those needed files into some directory.
And then another route that only pickup the files when you have all 4.
As the aggregation is a bit complicated you may just write some java code that puts the data correct in the right order.
On Wed, Jul 1, 2015 at 12:12 PM, Walzer, Thomas <th...@integratix.net> wrote:
> Hi,
>
> scheduled polling is possible. However I would poll all the time.
> For aggregating & sorting I would use a database.
>
> Cheers, Thomas.
>
>> Am 30.06.2015 um 23:57 schrieb Mills, Gary (GE Energy Management, consultant) <ga...@ge.com>:
>>
>> Hello,
>>
>> Thanks in advance, forgive me as I a learning. I do try to look these things up also, but with user feedback this greatly helps point me in the right direction.
>>
>> I have 4 files to pick up from FTP. I need to join these files together to achieve 1 single file. Each file has a unique header. These Headers need to be aligned. So the aggregation would need to join the files at the end and beginning of each header. The content of the 1 single file I need to sort by 2 elements, each row is terminated by new line \n. sort content by a numeric Id value and its timestamp value( field in record ). I am looking and it appears that camel can accomplish this without much coding. Am I correct or do I need to code this?
>>
>> I am thinking:
>> Import and save the files on the file system. Make sure I have all 4 files. If I don't have all 4 then wait until next poll and hopefully get the remaining files. The files are said to be pretty much guaranteed to come in at specific times. I just want to code for the unexpected event that they are not all present.
>> So, add FTP endpoint, add FTP route, poll at 4 pm est, and at 4 am est ( I don't know how to do this or if it can be done in camel ). Then through the route save the files on the file system. Do another route with aggregation strategy that will complete on 4 files and the append of the files will need to start with each header ( I do not know how to do this or if it is possible with camel ).
>> Msg 1 header A B C
>> 1 2 3 ( data )
>> 1 2 3
>> Msg 2 header D E F
>> 4 5 6
>> 4 5 6
>> Msg 3 header G H I
>> 7 8 9
>> 7 8 9
>> Msg 4 header J K L
>> 10 11 12
>> 10 11 12
>> RESULT ===========> A B C D E F G H I J K L
>> 1 2 3 4 5 6 7 8 9 10 11 12
>> 1 2 3 4 5 6 7 8 9 10 11 12
>> 1 2 3 4 5 6 7 8 9 10
>> 11 12 Then with another route perform a sort on 2 fields, 1 == asset Id and the 2nd a timestamp.
>>
>> Please offer suggestions or let me know if this is possible with Camel with little coding. Thanks again!!!
>> Gary
>
--
Claus Ibsen
-----------------
Red Hat, Inc.
Email: cibsen@redhat.com
Twitter: davsclaus
Blog: https://urldefense.proofpoint.com/v2/url?u=http-3A__davsclaus.com&d=AwIFaQ&c=IV_clAzoPDE253xZdHuilRgztyh_RiV3wUrLrDQYWSI&r=Hb1nXb7biSqY9NYScX39HfYaeuqILlgaLYiYNWt8oJo&m=68DXTF-6uUXEf-PFubobEy472ONuuSKOKEqmOyy_d-Q&s=2iWsGi_a3CmTJIKKDRTqup6gI8WA133Fz65kKFwzK6I&e=
Author of Camel in Action: https://urldefense.proofpoint.com/v2/url?u=http-3A__www.manning.com_ibsen&d=AwIFaQ&c=IV_clAzoPDE253xZdHuilRgztyh_RiV3wUrLrDQYWSI&r=Hb1nXb7biSqY9NYScX39HfYaeuqILlgaLYiYNWt8oJo&m=68DXTF-6uUXEf-PFubobEy472ONuuSKOKEqmOyy_d-Q&s=aEI9b0cgifVIZ2hB3dpiCxMiZcz6l6l7UzdPPaCLC-o&e=
hawtio: https://urldefense.proofpoint.com/v2/url?u=http-3A__hawt.io_&d=AwIFaQ&c=IV_clAzoPDE253xZdHuilRgztyh_RiV3wUrLrDQYWSI&r=Hb1nXb7biSqY9NYScX39HfYaeuqILlgaLYiYNWt8oJo&m=68DXTF-6uUXEf-PFubobEy472ONuuSKOKEqmOyy_d-Q&s=I1Q-xc5pY_xx-3IB8JrWcYepKSvN0Tbz_RLPae64GEQ&e=
fabric8: https://urldefense.proofpoint.com/v2/url?u=http-3A__fabric8.io_&d=AwIFaQ&c=IV_clAzoPDE253xZdHuilRgztyh_RiV3wUrLrDQYWSI&r=Hb1nXb7biSqY9NYScX39HfYaeuqILlgaLYiYNWt8oJo&m=68DXTF-6uUXEf-PFubobEy472ONuuSKOKEqmOyy_d-Q&s=fOhkAFIEV_u5H9SfLWQlpkCfkSWp1WwaymqanqI5vdY&e=
Re: newbie implementation question
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
Yeah split into 2 or more parts. The ftp that downloads those needed
files into some directory.
And then another route that only pickup the files when you have all 4.
As the aggregation is a bit complicated you may just write some java
code that puts the data correct in the right order.
On Wed, Jul 1, 2015 at 12:12 PM, Walzer, Thomas
<th...@integratix.net> wrote:
> Hi,
>
> scheduled polling is possible. However I would poll all the time.
> For aggregating & sorting I would use a database.
>
> Cheers, Thomas.
>
>> Am 30.06.2015 um 23:57 schrieb Mills, Gary (GE Energy Management, consultant) <ga...@ge.com>:
>>
>> Hello,
>>
>> Thanks in advance, forgive me as I a learning. I do try to look these things up also, but with user feedback this greatly helps point me in the right direction.
>>
>> I have 4 files to pick up from FTP. I need to join these files together to achieve 1 single file. Each file has a unique header. These Headers need to be aligned. So the aggregation would need to join the files at the end and beginning of each header. The content of the 1 single file I need to sort by 2 elements, each row is terminated by new line \n. sort content by a numeric Id value and its timestamp value( field in record ). I am looking and it appears that camel can accomplish this without much coding. Am I correct or do I need to code this?
>>
>> I am thinking:
>> Import and save the files on the file system. Make sure I have all 4 files. If I don't have all 4 then wait until next poll and hopefully get the remaining files. The files are said to be pretty much guaranteed to come in at specific times. I just want to code for the unexpected event that they are not all present.
>> So, add FTP endpoint, add FTP route, poll at 4 pm est, and at 4 am est ( I don't know how to do this or if it can be done in camel ). Then through the route save the files on the file system. Do another route with aggregation strategy that will complete on 4 files and the append of the files will need to start with each header ( I do not know how to do this or if it is possible with camel ).
>> Msg 1 header A B C
>> 1 2 3 ( data )
>> 1 2 3
>> Msg 2 header D E F
>> 4 5 6
>> 4 5 6
>> Msg 3 header G H I
>> 7 8 9
>> 7 8 9
>> Msg 4 header J K L
>> 10 11 12
>> 10 11 12
>> RESULT ===========> A B C D E F G H I J K L
>> 1 2 3 4 5 6 7 8 9 10 11 12
>> 1 2 3 4 5 6 7 8 9 10 11 12
>> 1 2 3 4 5 6 7 8 9 10 11 12
>> Then with another route perform a sort on 2 fields, 1 == asset Id and the 2nd a timestamp.
>>
>> Please offer suggestions or let me know if this is possible with Camel with little coding. Thanks again!!!
>> Gary
>
--
Claus Ibsen
-----------------
Red Hat, Inc.
Email: cibsen@redhat.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
hawtio: http://hawt.io/
fabric8: http://fabric8.io/
Re: newbie implementation question
Posted by "Walzer, Thomas" <th...@integratix.net>.
Hi,
scheduled polling is possible. However I would poll all the time.
For aggregating & sorting I would use a database.
Cheers, Thomas.
> Am 30.06.2015 um 23:57 schrieb Mills, Gary (GE Energy Management, consultant) <ga...@ge.com>:
>
> Hello,
>
> Thanks in advance, forgive me as I a learning. I do try to look these things up also, but with user feedback this greatly helps point me in the right direction.
>
> I have 4 files to pick up from FTP. I need to join these files together to achieve 1 single file. Each file has a unique header. These Headers need to be aligned. So the aggregation would need to join the files at the end and beginning of each header. The content of the 1 single file I need to sort by 2 elements, each row is terminated by new line \n. sort content by a numeric Id value and its timestamp value( field in record ). I am looking and it appears that camel can accomplish this without much coding. Am I correct or do I need to code this?
>
> I am thinking:
> Import and save the files on the file system. Make sure I have all 4 files. If I don't have all 4 then wait until next poll and hopefully get the remaining files. The files are said to be pretty much guaranteed to come in at specific times. I just want to code for the unexpected event that they are not all present.
> So, add FTP endpoint, add FTP route, poll at 4 pm est, and at 4 am est ( I don't know how to do this or if it can be done in camel ). Then through the route save the files on the file system. Do another route with aggregation strategy that will complete on 4 files and the append of the files will need to start with each header ( I do not know how to do this or if it is possible with camel ).
> Msg 1 header A B C
> 1 2 3 ( data )
> 1 2 3
> Msg 2 header D E F
> 4 5 6
> 4 5 6
> Msg 3 header G H I
> 7 8 9
> 7 8 9
> Msg 4 header J K L
> 10 11 12
> 10 11 12
> RESULT ===========> A B C D E F G H I J K L
> 1 2 3 4 5 6 7 8 9 10 11 12
> 1 2 3 4 5 6 7 8 9 10 11 12
> 1 2 3 4 5 6 7 8 9 10 11 12
> Then with another route perform a sort on 2 fields, 1 == asset Id and the 2nd a timestamp.
>
> Please offer suggestions or let me know if this is possible with Camel with little coding. Thanks again!!!
> Gary