You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by CY Kan <ma...@gmail.com> on 2009/06/27 12:26:25 UTC
SFTP Handshake
Hi,
I'm trying Camel 2.0-M2 and can't figure out how to realize the scenario
below:
1. System A exports records to a file named records.txt in an sftp server
folder.
2. to avoid read/write concurrency issue, System A creates another file
named "completed" in the folder after records.txt is made. In other words,
seeing "completed" means ready for file download.
3. System B polls the sftp server folder for "completed".
4. If "completed" is found, System B donwloads records.txt.
5. System B archives records.txt in another folder and deletes "completed"
after download complete.
I think it's something like Aggregator + Filter but failed. The problem is
the download of records.txt needs postponed until "completed" is found.
Could you suggest if Camel is good at this sort of thing or any other idea?
Thank you in advance.
--
View this message in context: http://www.nabble.com/SFTP-Handshake-tp24232050p24232050.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: SFTP Handshake
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
I updated the wiki page for file2 as the option - processStrategy -
was missing and this is the option you need.
http://cwiki.apache.org/confluence/display/CAMEL/File2
from("fp://someserver/inbox?username=foo&password=bar&processStrategy=#myReadyFileStrategy").to("bean:handleFile");
And then you can declare a bean in spring XML (if using this)
<bean id="myReadyFileStrategy" class="com.mycompany.MyReadyFileStrategy"/>
Notice the # syntax on the URI parameter. It tells Camel to go look
for it in the registry.
On Sun, Jun 28, 2009 at 12:49 PM, Claus Ibsen<cl...@gmail.com> wrote:
> Hi
>
> Yeah that strategy used by good old mainframes? Easier to write a new
> filer rather than rename/move the actual file.
>
> Camel can do this but it would require a little additional coding as
> you need to implement your own process strategy.
> With 2.0m2 you implement you own GenericFileProcessStrategy interface.
>
> And in the begin method you use the listFiles() method on the
> operations to see if the "complete.txt" file is there.
> Return true | false depending on the file exists.
>
> In the commit operation you move the actual file and delete the
> complete.txt file.
> The operations interface have methods for that.
>
> The only issue I see is that the file consumer will log WARN if begin
> returns false.
> I think we should reduce that to DEBUG as it is a valid strategy for
> you. I will do that in my next commit.
>
>
> On Sat, Jun 27, 2009 at 12:26 PM, CY Kan<ma...@gmail.com> wrote:
>>
>> Hi,
>>
>> I'm trying Camel 2.0-M2 and can't figure out how to realize the scenario
>> below:
>>
>> 1. System A exports records to a file named records.txt in an sftp server
>> folder.
>> 2. to avoid read/write concurrency issue, System A creates another file
>> named "completed" in the folder after records.txt is made. In other words,
>> seeing "completed" means ready for file download.
>> 3. System B polls the sftp server folder for "completed".
>> 4. If "completed" is found, System B donwloads records.txt.
>> 5. System B archives records.txt in another folder and deletes "completed"
>> after download complete.
>>
>> I think it's something like Aggregator + Filter but failed. The problem is
>> the download of records.txt needs postponed until "completed" is found.
>> Could you suggest if Camel is good at this sort of thing or any other idea?
>>
>> Thank you in advance.
>> --
>> View this message in context: http://www.nabble.com/SFTP-Handshake-tp24232050p24232050.html
>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>
>>
>
>
>
> --
> Claus Ibsen
> Apache Camel Committer
>
> Open Source Integration: http://fusesource.com
> Blog: http://davsclaus.blogspot.com/
> Twitter: http://twitter.com/davsclaus
>
--
Claus Ibsen
Apache Camel Committer
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus
Re: SFTP Handshake
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
How did it go for you? Did you get it working?
On Sun, Jun 28, 2009 at 4:54 PM, Claus Ibsen<cl...@gmail.com> wrote:
> On Sun, Jun 28, 2009 at 4:44 PM, CY Kan<ma...@gmail.com> wrote:
>>
>> Hi,
>>
>> Thank you for quick reply. Yes, it's to receive data files from mainframe.
>>
>> I tried the following code and had some problem encountered.
>>
>> public class MyReadyFileStrategy<T> extends
>> GenericFileProcessStrategySupport<T> {
>>
>> @Override
>> public boolean begin(GenericFileOperations<T> arg0, GenericFileEndpoint<T>
>> arg1,
>> GenericFileExchange<T> arg2, GenericFile<T> arg3) throws Exception {
>> super.begin(arg0, arg1, arg2, arg3);
>>
>> List<GenericFile<T>> files = (List<GenericFile<T>>
>> arg0.listFiles();
>> AssertNotNull( "No GenericFileOperations!", arg0 );
>> AssertNotNull( "No files!", files );
>> ...
>>
>> The first assert is true while the second isn't. The method listFiles()
>> returns nothing and so the existence of "completed.txt" cannot be checked.
>> Is there any problem on listFiles()?
>
> Ah if the file is polled in sub folder or the likes, you might need to
> use listFiles(folder) where folder is the same folder as where the
> actual file is.
> The path is in stored in the GenericFile.
>
>
>
>>
>> Thank you again.
>>
>>
>> Claus Ibsen-2 wrote:
>>>
>>> Hi
>>>
>>> Yeah that strategy used by good old mainframes? Easier to write a new
>>> filer rather than rename/move the actual file.
>>>
>>> Camel can do this but it would require a little additional coding as
>>> you need to implement your own process strategy.
>>> With 2.0m2 you implement you own GenericFileProcessStrategy interface.
>>>
>>> And in the begin method you use the listFiles() method on the
>>> operations to see if the "complete.txt" file is there.
>>> Return true | false depending on the file exists.
>>>
>>> In the commit operation you move the actual file and delete the
>>> complete.txt file.
>>> The operations interface have methods for that.
>>>
>>> The only issue I see is that the file consumer will log WARN if begin
>>> returns false.
>>> I think we should reduce that to DEBUG as it is a valid strategy for
>>> you. I will do that in my next commit.
>>>
>>>
>>> On Sat, Jun 27, 2009 at 12:26 PM, CY Kan<ma...@gmail.com> wrote:
>>>>
>>>> Hi,
>>>>
>>>> I'm trying Camel 2.0-M2 and can't figure out how to realize the scenario
>>>> below:
>>>>
>>>> 1. System A exports records to a file named records.txt in an sftp server
>>>> folder.
>>>> 2. to avoid read/write concurrency issue, System A creates another file
>>>> named "completed" in the folder after records.txt is made. In other
>>>> words,
>>>> seeing "completed" means ready for file download.
>>>> 3. System B polls the sftp server folder for "completed".
>>>> 4. If "completed" is found, System B donwloads records.txt.
>>>> 5. System B archives records.txt in another folder and deletes
>>>> "completed"
>>>> after download complete.
>>>>
>>>> I think it's something like Aggregator + Filter but failed. The problem
>>>> is
>>>> the download of records.txt needs postponed until "completed" is found.
>>>> Could you suggest if Camel is good at this sort of thing or any other
>>>> idea?
>>>>
>>>> Thank you in advance.
>>>> --
>>>> View this message in context:
>>>> http://www.nabble.com/SFTP-Handshake-tp24232050p24232050.html
>>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> Claus Ibsen
>>> Apache Camel Committer
>>>
>>> Open Source Integration: http://fusesource.com
>>> Blog: http://davsclaus.blogspot.com/
>>> Twitter: http://twitter.com/davsclaus
>>>
>>>
>>
>> --
>> View this message in context: http://www.nabble.com/SFTP-Handshake-tp24232050p24242127.html
>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>
>>
>
>
>
> --
> Claus Ibsen
> Apache Camel Committer
>
> Open Source Integration: http://fusesource.com
> Blog: http://davsclaus.blogspot.com/
> Twitter: http://twitter.com/davsclaus
>
--
Claus Ibsen
Apache Camel Committer
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus
Re: SFTP Handshake
Posted by Claus Ibsen <cl...@gmail.com>.
On Sun, Jun 28, 2009 at 4:44 PM, CY Kan<ma...@gmail.com> wrote:
>
> Hi,
>
> Thank you for quick reply. Yes, it's to receive data files from mainframe.
>
> I tried the following code and had some problem encountered.
>
> public class MyReadyFileStrategy<T> extends
> GenericFileProcessStrategySupport<T> {
>
> @Override
> public boolean begin(GenericFileOperations<T> arg0, GenericFileEndpoint<T>
> arg1,
> GenericFileExchange<T> arg2, GenericFile<T> arg3) throws Exception {
> super.begin(arg0, arg1, arg2, arg3);
>
> List<GenericFile<T>> files = (List<GenericFile<T>>
> arg0.listFiles();
> AssertNotNull( "No GenericFileOperations!", arg0 );
> AssertNotNull( "No files!", files );
> ...
>
> The first assert is true while the second isn't. The method listFiles()
> returns nothing and so the existence of "completed.txt" cannot be checked.
> Is there any problem on listFiles()?
Ah if the file is polled in sub folder or the likes, you might need to
use listFiles(folder) where folder is the same folder as where the
actual file is.
The path is in stored in the GenericFile.
>
> Thank you again.
>
>
> Claus Ibsen-2 wrote:
>>
>> Hi
>>
>> Yeah that strategy used by good old mainframes? Easier to write a new
>> filer rather than rename/move the actual file.
>>
>> Camel can do this but it would require a little additional coding as
>> you need to implement your own process strategy.
>> With 2.0m2 you implement you own GenericFileProcessStrategy interface.
>>
>> And in the begin method you use the listFiles() method on the
>> operations to see if the "complete.txt" file is there.
>> Return true | false depending on the file exists.
>>
>> In the commit operation you move the actual file and delete the
>> complete.txt file.
>> The operations interface have methods for that.
>>
>> The only issue I see is that the file consumer will log WARN if begin
>> returns false.
>> I think we should reduce that to DEBUG as it is a valid strategy for
>> you. I will do that in my next commit.
>>
>>
>> On Sat, Jun 27, 2009 at 12:26 PM, CY Kan<ma...@gmail.com> wrote:
>>>
>>> Hi,
>>>
>>> I'm trying Camel 2.0-M2 and can't figure out how to realize the scenario
>>> below:
>>>
>>> 1. System A exports records to a file named records.txt in an sftp server
>>> folder.
>>> 2. to avoid read/write concurrency issue, System A creates another file
>>> named "completed" in the folder after records.txt is made. In other
>>> words,
>>> seeing "completed" means ready for file download.
>>> 3. System B polls the sftp server folder for "completed".
>>> 4. If "completed" is found, System B donwloads records.txt.
>>> 5. System B archives records.txt in another folder and deletes
>>> "completed"
>>> after download complete.
>>>
>>> I think it's something like Aggregator + Filter but failed. The problem
>>> is
>>> the download of records.txt needs postponed until "completed" is found.
>>> Could you suggest if Camel is good at this sort of thing or any other
>>> idea?
>>>
>>> Thank you in advance.
>>> --
>>> View this message in context:
>>> http://www.nabble.com/SFTP-Handshake-tp24232050p24232050.html
>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>
>>>
>>
>>
>>
>> --
>> Claus Ibsen
>> Apache Camel Committer
>>
>> Open Source Integration: http://fusesource.com
>> Blog: http://davsclaus.blogspot.com/
>> Twitter: http://twitter.com/davsclaus
>>
>>
>
> --
> View this message in context: http://www.nabble.com/SFTP-Handshake-tp24232050p24242127.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
>
--
Claus Ibsen
Apache Camel Committer
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus
Re: SFTP Handshake
Posted by CY Kan <ma...@gmail.com>.
Hi,
Thank you for quick reply. Yes, it's to receive data files from mainframe.
I tried the following code and had some problem encountered.
public class MyReadyFileStrategy<T> extends
GenericFileProcessStrategySupport<T> {
@Override
public boolean begin(GenericFileOperations<T> arg0, GenericFileEndpoint<T>
arg1,
GenericFileExchange<T> arg2, GenericFile<T> arg3) throws Exception {
super.begin(arg0, arg1, arg2, arg3);
List<GenericFile<T>> files = (List<GenericFile<T>>
arg0.listFiles();
AssertNotNull( "No GenericFileOperations!", arg0 );
AssertNotNull( "No files!", files );
...
The first assert is true while the second isn't. The method listFiles()
returns nothing and so the existence of "completed.txt" cannot be checked.
Is there any problem on listFiles()?
Thank you again.
Claus Ibsen-2 wrote:
>
> Hi
>
> Yeah that strategy used by good old mainframes? Easier to write a new
> filer rather than rename/move the actual file.
>
> Camel can do this but it would require a little additional coding as
> you need to implement your own process strategy.
> With 2.0m2 you implement you own GenericFileProcessStrategy interface.
>
> And in the begin method you use the listFiles() method on the
> operations to see if the "complete.txt" file is there.
> Return true | false depending on the file exists.
>
> In the commit operation you move the actual file and delete the
> complete.txt file.
> The operations interface have methods for that.
>
> The only issue I see is that the file consumer will log WARN if begin
> returns false.
> I think we should reduce that to DEBUG as it is a valid strategy for
> you. I will do that in my next commit.
>
>
> On Sat, Jun 27, 2009 at 12:26 PM, CY Kan<ma...@gmail.com> wrote:
>>
>> Hi,
>>
>> I'm trying Camel 2.0-M2 and can't figure out how to realize the scenario
>> below:
>>
>> 1. System A exports records to a file named records.txt in an sftp server
>> folder.
>> 2. to avoid read/write concurrency issue, System A creates another file
>> named "completed" in the folder after records.txt is made. In other
>> words,
>> seeing "completed" means ready for file download.
>> 3. System B polls the sftp server folder for "completed".
>> 4. If "completed" is found, System B donwloads records.txt.
>> 5. System B archives records.txt in another folder and deletes
>> "completed"
>> after download complete.
>>
>> I think it's something like Aggregator + Filter but failed. The problem
>> is
>> the download of records.txt needs postponed until "completed" is found.
>> Could you suggest if Camel is good at this sort of thing or any other
>> idea?
>>
>> Thank you in advance.
>> --
>> View this message in context:
>> http://www.nabble.com/SFTP-Handshake-tp24232050p24232050.html
>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>
>>
>
>
>
> --
> Claus Ibsen
> Apache Camel Committer
>
> Open Source Integration: http://fusesource.com
> Blog: http://davsclaus.blogspot.com/
> Twitter: http://twitter.com/davsclaus
>
>
--
View this message in context: http://www.nabble.com/SFTP-Handshake-tp24232050p24242127.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: SFTP Handshake
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
Yeah that strategy used by good old mainframes? Easier to write a new
filer rather than rename/move the actual file.
Camel can do this but it would require a little additional coding as
you need to implement your own process strategy.
With 2.0m2 you implement you own GenericFileProcessStrategy interface.
And in the begin method you use the listFiles() method on the
operations to see if the "complete.txt" file is there.
Return true | false depending on the file exists.
In the commit operation you move the actual file and delete the
complete.txt file.
The operations interface have methods for that.
The only issue I see is that the file consumer will log WARN if begin
returns false.
I think we should reduce that to DEBUG as it is a valid strategy for
you. I will do that in my next commit.
On Sat, Jun 27, 2009 at 12:26 PM, CY Kan<ma...@gmail.com> wrote:
>
> Hi,
>
> I'm trying Camel 2.0-M2 and can't figure out how to realize the scenario
> below:
>
> 1. System A exports records to a file named records.txt in an sftp server
> folder.
> 2. to avoid read/write concurrency issue, System A creates another file
> named "completed" in the folder after records.txt is made. In other words,
> seeing "completed" means ready for file download.
> 3. System B polls the sftp server folder for "completed".
> 4. If "completed" is found, System B donwloads records.txt.
> 5. System B archives records.txt in another folder and deletes "completed"
> after download complete.
>
> I think it's something like Aggregator + Filter but failed. The problem is
> the download of records.txt needs postponed until "completed" is found.
> Could you suggest if Camel is good at this sort of thing or any other idea?
>
> Thank you in advance.
> --
> View this message in context: http://www.nabble.com/SFTP-Handshake-tp24232050p24232050.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
>
--
Claus Ibsen
Apache Camel Committer
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus