You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by David Capwell <dc...@gmail.com> on 2012/06/28 02:15:36 UTC
unable to control ftp delay
I am trying to use sftp (camel 2.9.2) and notice that every download is
spread out to one every 10 seconds (files are around 10kb). Based off the
examples in http://camel.apache.org/ftp2.html it looks like there are delay
options (they are not listened under options, but used in example) but when
I use them nothing changes. How can I have the client pull files faster?
Another question, if I am trying to write the ftp files locally, is there
any way to avoid the file download if the local file is around?
I append fileExist=Ignore in the to() but it seems that the files still
get written again.
Here is the route I am using:
public void ftpToLocal() throws Exception {
runRoutes(new RouteBuilder() {
@Override
public void configure() throws Exception {
Registry registry = getContext().getRegistry();
String user = (String) registry.lookup("ftpUser");
String pass = (String) registry.lookup("ftpPass");
// we use a delay of 60 minutes (eg. once pr. hour we poll the FTP
server
// long delay = 60 * 60 * 1000L;
// long delay = TimeUnit.SECONDS.toMillis(1);
long delay = 200;
from("sftp://"+user+"@example.com/path/to/dir?password=" + pass +
"&binary=true&delay="+delay+"&consumer.delay=" + delay)
.to("file://data/ftp?fileExist=Ignore");
}
});
}
Thanks for your time reading this email.
Re: unable to control ftp delay
Posted by Claus Ibsen <cl...@gmail.com>.
On Fri, Jun 29, 2012 at 5:52 PM, David Capwell <dc...@gmail.com> wrote:
> I have added idempotent=true and this seems to work the way i wont but
> doesn't persist. So to make sure I don't download the same file even after
> reboot the best option is the filter, correct? If so what am I missing
> from getting my code to work?
>
There are details about persistent idempotent repositories in the Camel docs.
http://camel.apache.org/file2.html
And the EIP has some details as well
http://camel.apache.org/idempotent-consumer.html
Its just that the file/ftp component has baked in idempotent consumer,
so you do not
need to use .idempotentConsumer in the route. This is an optimization to avoid
consuming the file if really not needed.
An alternative is as you said a custom filter, where you can return
true|false, whether
to consume the file or not.
> thanks for all the help i have been given so far!
>
> On Fri, Jun 29, 2012 at 8:45 AM, David Capwell <dc...@gmail.com> wrote:
>
>> Sorry, I seemed to miss the exception that I am getting:
>>
>> Caused by: java.lang.IllegalArgumentException: Could not find a suitable
>> setter for property: filter as there isn't a setter method with same type:
>> java.lang.String nor type conversion possible: No type converter available
>> to convert from type: java.lang.String to the required type:
>> org.apache.camel.component.file.GenericFileFilter with value
>> #fileExistsFilter
>>
>> Not sure why but it seems that the filter is not getting the param from
>> the registry but trying to pass the value in as a normal string. Based on
>> the last comment it sounds like this should be working?
>>
>>
>> On Thu, Jun 28, 2012 at 11:21 PM, Claus Ibsen <cl...@gmail.com>wrote:
>>
>>> On Fri, Jun 29, 2012 at 12:32 AM, David Capwell <dc...@gmail.com>
>>> wrote:
>>> > Playing with the delay fields defined in file2 seems to improve download
>>> > speeds. Now that this is working better I want to be able to skip any
>>> file
>>> > that has already been processed (should persist on restart). It looked
>>> to
>>> > me that I should be able to set this up by using the filter option, but
>>> I
>>> > am having a hard time figuring out how to use this option without using
>>> > spring.
>>> >
>>>
>>> Yeah you can either use the filter or the idempotent consumer. The
>>> latter is the EIP pattern that fits this problem.
>>> The file/ftp component has baked in of both. And the latter has
>>> plugins for various persistent storages.
>>>
>>> The code below seems okay, if you configure the endpoint from the uri,
>>> then you need to refer to your
>>> bean that implements the filter using the # notation, as you do.
>>>
>>> Then you need to enlist that bean in the registry. In Spring XML that
>>> happens automatic when you do a <bean>.
>>> When you use pure Java, you need to pass in a registry implementation
>>> when you create the camel context,
>>> as you do with the code below using the SimpleRegistry.
>>>
>>> Then its just a matter of putting the bean in the simple registry, as
>>> shown in the code.
>>>
>>>
>>> The unit test of the camel-ftp component has plenty of examples as well
>>>
>>> https://svn.apache.org/repos/asf/camel/trunk/components/camel-ftp/src/test/java/
>>>
>>>
>>> > here is the code I am using (might not compile cause its copy/paste from
>>> > diff places):
>>> >
>>> > SimpleRegistry registry = new SimpleRegistry();
>>> > registry.put("fileExistsFilter", new CopyFilter());
>>> >
>>> > CamelContext context = new DefaultCamelContext(registry);
>>> > context.addRoutes(new RouteBuilder() {
>>> > @Override
>>> > public void configure() throws Exception {
>>> > from("sftp://example.com/path/to/file?password=" + pass +
>>> > "&binary=true&delay="+delay+ "&initialDelay=" + delay +
>>> > "&filter=#fileExistsFilter")
>>> > .to("file://data/ftp?fileExist=Ignore");
>>> >
>>> > }
>>> > });
>>> > context.start();
>>> >
>>> > ...
>>> > public static class CopyFilter implements GenericFileFilter {
>>> >
>>> > @Override
>>> > public boolean accept(GenericFile genericFile) {
>>> > // check to see if file exists locally
>>> > File localFile = new File("data/ftp", genericFile.getFileName());
>>> > if(localFile.exists()) {
>>> > // check to make sure the sizes match
>>> > if(localFile.length() == genericFile.getFileLength()) {
>>> > // file has already been copied, ignore
>>> > return false;
>>> > }
>>> > }
>>> > return true;
>>> > }
>>> > }
>>> >
>>> >
>>> > Thanks for your time reading this email
>>> >
>>> > On Thu, Jun 28, 2012 at 6:07 AM, David Capwell <dc...@gmail.com>
>>> wrote:
>>> >
>>> >> I'll take a look at file2 to see if that helps.
>>> >>
>>> >> My goal is to push the data from FTP to s3. Copying locally so I know
>>> >> what has been processed,
>>> >> On Jun 28, 2012 12:48 AM, "Claus Ibsen" <cl...@gmail.com> wrote:
>>> >>
>>> >>> Hi
>>> >>>
>>> >>> As mentioned on the ftp2 page
>>> >>> http://camel.apache.org/ftp2
>>> >>>
>>> >>> See the file2 page for more options as the ftp component inherits
>>> these
>>> >>> options
>>> >>> http://camel.apache.org/file2.html
>>> >>>
>>> >>> The delay and consumer.delay option is the same. delay is just
>>> >>> shorthand for consumer.delay.
>>> >>>
>>> >>> And no you cannot avoid the download if you later want to upload it
>>> >>> someplace else, and the file exists.
>>> >>> The ftp consumer don't know about this.
>>> >>>
>>> >>> A tricky improvement could be to add a new option to only download the
>>> >>> file on demand, but that is a bit tricky
>>> >>> as you would need a live connection and the ftp client to still be
>>> >>> around and connected.
>>> >>>
>>> >>>
>>> >>> What are you trying to do? To sync files between 2 ftp servers?
>>> >>>
>>> >>>
>>> >>>
>>> >>> On Thu, Jun 28, 2012 at 2:15 AM, David Capwell <dc...@gmail.com>
>>> >>> wrote:
>>> >>> > I am trying to use sftp (camel 2.9.2) and notice that every
>>> download is
>>> >>> > spread out to one every 10 seconds (files are around 10kb). Based
>>> off
>>> >>> the
>>> >>> > examples in http://camel.apache.org/ftp2.html it looks like there
>>> are
>>> >>> delay
>>> >>> > options (they are not listened under options, but used in example)
>>> but
>>> >>> when
>>> >>> > I use them nothing changes. How can I have the client pull files
>>> >>> faster?
>>> >>> >
>>> >>> > Another question, if I am trying to write the ftp files locally, is
>>> >>> there
>>> >>> > any way to avoid the file download if the local file is around?
>>> >>> > I append fileExist=Ignore in the to() but it seems that the files
>>> still
>>> >>> > get written again.
>>> >>> >
>>> >>> > Here is the route I am using:
>>> >>> >
>>> >>> > public void ftpToLocal() throws Exception {
>>> >>> > runRoutes(new RouteBuilder() {
>>> >>> > @Override
>>> >>> > public void configure() throws Exception {
>>> >>> > Registry registry = getContext().getRegistry();
>>> >>> > String user = (String) registry.lookup("ftpUser");
>>> >>> > String pass = (String) registry.lookup("ftpPass");
>>> >>> >
>>> >>> > // we use a delay of 60 minutes (eg. once pr. hour we poll
>>> the
>>> >>> FTP
>>> >>> > server
>>> >>> > // long delay = 60 * 60 * 1000L;
>>> >>> > // long delay = TimeUnit.SECONDS.toMillis(1);
>>> >>> > long delay = 200;
>>> >>> >
>>> >>> > from("sftp://"+user+"@example.com/path/to/dir?password=" +
>>> pass
>>> >>> +
>>> >>> > "&binary=true&delay="+delay+"&consumer.delay=" + delay)
>>> >>> > .to("file://data/ftp?fileExist=Ignore");
>>> >>> > }
>>> >>> > });
>>> >>> > }
>>> >>> >
>>> >>> >
>>> >>> > Thanks for your time reading this email.
>>> >>>
>>> >>>
>>> >>>
>>> >>> --
>>> >>> Claus Ibsen
>>> >>> -----------------
>>> >>> FuseSource
>>> >>> Email: cibsen@fusesource.com
>>> >>> Web: http://fusesource.com
>>> >>> Twitter: davsclaus, fusenews
>>> >>> Blog: http://davsclaus.com
>>> >>> Author of Camel in Action: http://www.manning.com/ibsen
>>> >>>
>>> >>
>>>
>>>
>>>
>>> --
>>> Claus Ibsen
>>> -----------------
>>> FuseSource
>>> Email: cibsen@fusesource.com
>>> Web: http://fusesource.com
>>> Twitter: davsclaus, fusenews
>>> Blog: http://davsclaus.com
>>> Author of Camel in Action: http://www.manning.com/ibsen
>>>
>>
>>
--
Claus Ibsen
-----------------
FuseSource
Email: cibsen@fusesource.com
Web: http://fusesource.com
Twitter: davsclaus, fusenews
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
Re: unable to control ftp delay
Posted by David Capwell <dc...@gmail.com>.
I have added idempotent=true and this seems to work the way i wont but
doesn't persist. So to make sure I don't download the same file even after
reboot the best option is the filter, correct? If so what am I missing
from getting my code to work?
thanks for all the help i have been given so far!
On Fri, Jun 29, 2012 at 8:45 AM, David Capwell <dc...@gmail.com> wrote:
> Sorry, I seemed to miss the exception that I am getting:
>
> Caused by: java.lang.IllegalArgumentException: Could not find a suitable
> setter for property: filter as there isn't a setter method with same type:
> java.lang.String nor type conversion possible: No type converter available
> to convert from type: java.lang.String to the required type:
> org.apache.camel.component.file.GenericFileFilter with value
> #fileExistsFilter
>
> Not sure why but it seems that the filter is not getting the param from
> the registry but trying to pass the value in as a normal string. Based on
> the last comment it sounds like this should be working?
>
>
> On Thu, Jun 28, 2012 at 11:21 PM, Claus Ibsen <cl...@gmail.com>wrote:
>
>> On Fri, Jun 29, 2012 at 12:32 AM, David Capwell <dc...@gmail.com>
>> wrote:
>> > Playing with the delay fields defined in file2 seems to improve download
>> > speeds. Now that this is working better I want to be able to skip any
>> file
>> > that has already been processed (should persist on restart). It looked
>> to
>> > me that I should be able to set this up by using the filter option, but
>> I
>> > am having a hard time figuring out how to use this option without using
>> > spring.
>> >
>>
>> Yeah you can either use the filter or the idempotent consumer. The
>> latter is the EIP pattern that fits this problem.
>> The file/ftp component has baked in of both. And the latter has
>> plugins for various persistent storages.
>>
>> The code below seems okay, if you configure the endpoint from the uri,
>> then you need to refer to your
>> bean that implements the filter using the # notation, as you do.
>>
>> Then you need to enlist that bean in the registry. In Spring XML that
>> happens automatic when you do a <bean>.
>> When you use pure Java, you need to pass in a registry implementation
>> when you create the camel context,
>> as you do with the code below using the SimpleRegistry.
>>
>> Then its just a matter of putting the bean in the simple registry, as
>> shown in the code.
>>
>>
>> The unit test of the camel-ftp component has plenty of examples as well
>>
>> https://svn.apache.org/repos/asf/camel/trunk/components/camel-ftp/src/test/java/
>>
>>
>> > here is the code I am using (might not compile cause its copy/paste from
>> > diff places):
>> >
>> > SimpleRegistry registry = new SimpleRegistry();
>> > registry.put("fileExistsFilter", new CopyFilter());
>> >
>> > CamelContext context = new DefaultCamelContext(registry);
>> > context.addRoutes(new RouteBuilder() {
>> > @Override
>> > public void configure() throws Exception {
>> > from("sftp://example.com/path/to/file?password=" + pass +
>> > "&binary=true&delay="+delay+ "&initialDelay=" + delay +
>> > "&filter=#fileExistsFilter")
>> > .to("file://data/ftp?fileExist=Ignore");
>> >
>> > }
>> > });
>> > context.start();
>> >
>> > ...
>> > public static class CopyFilter implements GenericFileFilter {
>> >
>> > @Override
>> > public boolean accept(GenericFile genericFile) {
>> > // check to see if file exists locally
>> > File localFile = new File("data/ftp", genericFile.getFileName());
>> > if(localFile.exists()) {
>> > // check to make sure the sizes match
>> > if(localFile.length() == genericFile.getFileLength()) {
>> > // file has already been copied, ignore
>> > return false;
>> > }
>> > }
>> > return true;
>> > }
>> > }
>> >
>> >
>> > Thanks for your time reading this email
>> >
>> > On Thu, Jun 28, 2012 at 6:07 AM, David Capwell <dc...@gmail.com>
>> wrote:
>> >
>> >> I'll take a look at file2 to see if that helps.
>> >>
>> >> My goal is to push the data from FTP to s3. Copying locally so I know
>> >> what has been processed,
>> >> On Jun 28, 2012 12:48 AM, "Claus Ibsen" <cl...@gmail.com> wrote:
>> >>
>> >>> Hi
>> >>>
>> >>> As mentioned on the ftp2 page
>> >>> http://camel.apache.org/ftp2
>> >>>
>> >>> See the file2 page for more options as the ftp component inherits
>> these
>> >>> options
>> >>> http://camel.apache.org/file2.html
>> >>>
>> >>> The delay and consumer.delay option is the same. delay is just
>> >>> shorthand for consumer.delay.
>> >>>
>> >>> And no you cannot avoid the download if you later want to upload it
>> >>> someplace else, and the file exists.
>> >>> The ftp consumer don't know about this.
>> >>>
>> >>> A tricky improvement could be to add a new option to only download the
>> >>> file on demand, but that is a bit tricky
>> >>> as you would need a live connection and the ftp client to still be
>> >>> around and connected.
>> >>>
>> >>>
>> >>> What are you trying to do? To sync files between 2 ftp servers?
>> >>>
>> >>>
>> >>>
>> >>> On Thu, Jun 28, 2012 at 2:15 AM, David Capwell <dc...@gmail.com>
>> >>> wrote:
>> >>> > I am trying to use sftp (camel 2.9.2) and notice that every
>> download is
>> >>> > spread out to one every 10 seconds (files are around 10kb). Based
>> off
>> >>> the
>> >>> > examples in http://camel.apache.org/ftp2.html it looks like there
>> are
>> >>> delay
>> >>> > options (they are not listened under options, but used in example)
>> but
>> >>> when
>> >>> > I use them nothing changes. How can I have the client pull files
>> >>> faster?
>> >>> >
>> >>> > Another question, if I am trying to write the ftp files locally, is
>> >>> there
>> >>> > any way to avoid the file download if the local file is around?
>> >>> > I append fileExist=Ignore in the to() but it seems that the files
>> still
>> >>> > get written again.
>> >>> >
>> >>> > Here is the route I am using:
>> >>> >
>> >>> > public void ftpToLocal() throws Exception {
>> >>> > runRoutes(new RouteBuilder() {
>> >>> > @Override
>> >>> > public void configure() throws Exception {
>> >>> > Registry registry = getContext().getRegistry();
>> >>> > String user = (String) registry.lookup("ftpUser");
>> >>> > String pass = (String) registry.lookup("ftpPass");
>> >>> >
>> >>> > // we use a delay of 60 minutes (eg. once pr. hour we poll
>> the
>> >>> FTP
>> >>> > server
>> >>> > // long delay = 60 * 60 * 1000L;
>> >>> > // long delay = TimeUnit.SECONDS.toMillis(1);
>> >>> > long delay = 200;
>> >>> >
>> >>> > from("sftp://"+user+"@example.com/path/to/dir?password=" +
>> pass
>> >>> +
>> >>> > "&binary=true&delay="+delay+"&consumer.delay=" + delay)
>> >>> > .to("file://data/ftp?fileExist=Ignore");
>> >>> > }
>> >>> > });
>> >>> > }
>> >>> >
>> >>> >
>> >>> > Thanks for your time reading this email.
>> >>>
>> >>>
>> >>>
>> >>> --
>> >>> Claus Ibsen
>> >>> -----------------
>> >>> FuseSource
>> >>> Email: cibsen@fusesource.com
>> >>> Web: http://fusesource.com
>> >>> Twitter: davsclaus, fusenews
>> >>> Blog: http://davsclaus.com
>> >>> Author of Camel in Action: http://www.manning.com/ibsen
>> >>>
>> >>
>>
>>
>>
>> --
>> Claus Ibsen
>> -----------------
>> FuseSource
>> Email: cibsen@fusesource.com
>> Web: http://fusesource.com
>> Twitter: davsclaus, fusenews
>> Blog: http://davsclaus.com
>> Author of Camel in Action: http://www.manning.com/ibsen
>>
>
>
Re: unable to control ftp delay
Posted by David Capwell <dc...@gmail.com>.
Sorry, I seemed to miss the exception that I am getting:
Caused by: java.lang.IllegalArgumentException: Could not find a suitable
setter for property: filter as there isn't a setter method with same type:
java.lang.String nor type conversion possible: No type converter available
to convert from type: java.lang.String to the required type:
org.apache.camel.component.file.GenericFileFilter with value
#fileExistsFilter
Not sure why but it seems that the filter is not getting the param from the
registry but trying to pass the value in as a normal string. Based on the
last comment it sounds like this should be working?
On Thu, Jun 28, 2012 at 11:21 PM, Claus Ibsen <cl...@gmail.com> wrote:
> On Fri, Jun 29, 2012 at 12:32 AM, David Capwell <dc...@gmail.com>
> wrote:
> > Playing with the delay fields defined in file2 seems to improve download
> > speeds. Now that this is working better I want to be able to skip any
> file
> > that has already been processed (should persist on restart). It looked
> to
> > me that I should be able to set this up by using the filter option, but I
> > am having a hard time figuring out how to use this option without using
> > spring.
> >
>
> Yeah you can either use the filter or the idempotent consumer. The
> latter is the EIP pattern that fits this problem.
> The file/ftp component has baked in of both. And the latter has
> plugins for various persistent storages.
>
> The code below seems okay, if you configure the endpoint from the uri,
> then you need to refer to your
> bean that implements the filter using the # notation, as you do.
>
> Then you need to enlist that bean in the registry. In Spring XML that
> happens automatic when you do a <bean>.
> When you use pure Java, you need to pass in a registry implementation
> when you create the camel context,
> as you do with the code below using the SimpleRegistry.
>
> Then its just a matter of putting the bean in the simple registry, as
> shown in the code.
>
>
> The unit test of the camel-ftp component has plenty of examples as well
>
> https://svn.apache.org/repos/asf/camel/trunk/components/camel-ftp/src/test/java/
>
>
> > here is the code I am using (might not compile cause its copy/paste from
> > diff places):
> >
> > SimpleRegistry registry = new SimpleRegistry();
> > registry.put("fileExistsFilter", new CopyFilter());
> >
> > CamelContext context = new DefaultCamelContext(registry);
> > context.addRoutes(new RouteBuilder() {
> > @Override
> > public void configure() throws Exception {
> > from("sftp://example.com/path/to/file?password=" + pass +
> > "&binary=true&delay="+delay+ "&initialDelay=" + delay +
> > "&filter=#fileExistsFilter")
> > .to("file://data/ftp?fileExist=Ignore");
> >
> > }
> > });
> > context.start();
> >
> > ...
> > public static class CopyFilter implements GenericFileFilter {
> >
> > @Override
> > public boolean accept(GenericFile genericFile) {
> > // check to see if file exists locally
> > File localFile = new File("data/ftp", genericFile.getFileName());
> > if(localFile.exists()) {
> > // check to make sure the sizes match
> > if(localFile.length() == genericFile.getFileLength()) {
> > // file has already been copied, ignore
> > return false;
> > }
> > }
> > return true;
> > }
> > }
> >
> >
> > Thanks for your time reading this email
> >
> > On Thu, Jun 28, 2012 at 6:07 AM, David Capwell <dc...@gmail.com>
> wrote:
> >
> >> I'll take a look at file2 to see if that helps.
> >>
> >> My goal is to push the data from FTP to s3. Copying locally so I know
> >> what has been processed,
> >> On Jun 28, 2012 12:48 AM, "Claus Ibsen" <cl...@gmail.com> wrote:
> >>
> >>> Hi
> >>>
> >>> As mentioned on the ftp2 page
> >>> http://camel.apache.org/ftp2
> >>>
> >>> See the file2 page for more options as the ftp component inherits these
> >>> options
> >>> http://camel.apache.org/file2.html
> >>>
> >>> The delay and consumer.delay option is the same. delay is just
> >>> shorthand for consumer.delay.
> >>>
> >>> And no you cannot avoid the download if you later want to upload it
> >>> someplace else, and the file exists.
> >>> The ftp consumer don't know about this.
> >>>
> >>> A tricky improvement could be to add a new option to only download the
> >>> file on demand, but that is a bit tricky
> >>> as you would need a live connection and the ftp client to still be
> >>> around and connected.
> >>>
> >>>
> >>> What are you trying to do? To sync files between 2 ftp servers?
> >>>
> >>>
> >>>
> >>> On Thu, Jun 28, 2012 at 2:15 AM, David Capwell <dc...@gmail.com>
> >>> wrote:
> >>> > I am trying to use sftp (camel 2.9.2) and notice that every download
> is
> >>> > spread out to one every 10 seconds (files are around 10kb). Based
> off
> >>> the
> >>> > examples in http://camel.apache.org/ftp2.html it looks like there
> are
> >>> delay
> >>> > options (they are not listened under options, but used in example)
> but
> >>> when
> >>> > I use them nothing changes. How can I have the client pull files
> >>> faster?
> >>> >
> >>> > Another question, if I am trying to write the ftp files locally, is
> >>> there
> >>> > any way to avoid the file download if the local file is around?
> >>> > I append fileExist=Ignore in the to() but it seems that the files
> still
> >>> > get written again.
> >>> >
> >>> > Here is the route I am using:
> >>> >
> >>> > public void ftpToLocal() throws Exception {
> >>> > runRoutes(new RouteBuilder() {
> >>> > @Override
> >>> > public void configure() throws Exception {
> >>> > Registry registry = getContext().getRegistry();
> >>> > String user = (String) registry.lookup("ftpUser");
> >>> > String pass = (String) registry.lookup("ftpPass");
> >>> >
> >>> > // we use a delay of 60 minutes (eg. once pr. hour we poll the
> >>> FTP
> >>> > server
> >>> > // long delay = 60 * 60 * 1000L;
> >>> > // long delay = TimeUnit.SECONDS.toMillis(1);
> >>> > long delay = 200;
> >>> >
> >>> > from("sftp://"+user+"@example.com/path/to/dir?password=" +
> pass
> >>> +
> >>> > "&binary=true&delay="+delay+"&consumer.delay=" + delay)
> >>> > .to("file://data/ftp?fileExist=Ignore");
> >>> > }
> >>> > });
> >>> > }
> >>> >
> >>> >
> >>> > Thanks for your time reading this email.
> >>>
> >>>
> >>>
> >>> --
> >>> Claus Ibsen
> >>> -----------------
> >>> FuseSource
> >>> Email: cibsen@fusesource.com
> >>> Web: http://fusesource.com
> >>> Twitter: davsclaus, fusenews
> >>> Blog: http://davsclaus.com
> >>> Author of Camel in Action: http://www.manning.com/ibsen
> >>>
> >>
>
>
>
> --
> Claus Ibsen
> -----------------
> FuseSource
> Email: cibsen@fusesource.com
> Web: http://fusesource.com
> Twitter: davsclaus, fusenews
> Blog: http://davsclaus.com
> Author of Camel in Action: http://www.manning.com/ibsen
>
Re: unable to control ftp delay
Posted by Claus Ibsen <cl...@gmail.com>.
On Fri, Jun 29, 2012 at 12:32 AM, David Capwell <dc...@gmail.com> wrote:
> Playing with the delay fields defined in file2 seems to improve download
> speeds. Now that this is working better I want to be able to skip any file
> that has already been processed (should persist on restart). It looked to
> me that I should be able to set this up by using the filter option, but I
> am having a hard time figuring out how to use this option without using
> spring.
>
Yeah you can either use the filter or the idempotent consumer. The
latter is the EIP pattern that fits this problem.
The file/ftp component has baked in of both. And the latter has
plugins for various persistent storages.
The code below seems okay, if you configure the endpoint from the uri,
then you need to refer to your
bean that implements the filter using the # notation, as you do.
Then you need to enlist that bean in the registry. In Spring XML that
happens automatic when you do a <bean>.
When you use pure Java, you need to pass in a registry implementation
when you create the camel context,
as you do with the code below using the SimpleRegistry.
Then its just a matter of putting the bean in the simple registry, as
shown in the code.
The unit test of the camel-ftp component has plenty of examples as well
https://svn.apache.org/repos/asf/camel/trunk/components/camel-ftp/src/test/java/
> here is the code I am using (might not compile cause its copy/paste from
> diff places):
>
> SimpleRegistry registry = new SimpleRegistry();
> registry.put("fileExistsFilter", new CopyFilter());
>
> CamelContext context = new DefaultCamelContext(registry);
> context.addRoutes(new RouteBuilder() {
> @Override
> public void configure() throws Exception {
> from("sftp://example.com/path/to/file?password=" + pass +
> "&binary=true&delay="+delay+ "&initialDelay=" + delay +
> "&filter=#fileExistsFilter")
> .to("file://data/ftp?fileExist=Ignore");
>
> }
> });
> context.start();
>
> ...
> public static class CopyFilter implements GenericFileFilter {
>
> @Override
> public boolean accept(GenericFile genericFile) {
> // check to see if file exists locally
> File localFile = new File("data/ftp", genericFile.getFileName());
> if(localFile.exists()) {
> // check to make sure the sizes match
> if(localFile.length() == genericFile.getFileLength()) {
> // file has already been copied, ignore
> return false;
> }
> }
> return true;
> }
> }
>
>
> Thanks for your time reading this email
>
> On Thu, Jun 28, 2012 at 6:07 AM, David Capwell <dc...@gmail.com> wrote:
>
>> I'll take a look at file2 to see if that helps.
>>
>> My goal is to push the data from FTP to s3. Copying locally so I know
>> what has been processed,
>> On Jun 28, 2012 12:48 AM, "Claus Ibsen" <cl...@gmail.com> wrote:
>>
>>> Hi
>>>
>>> As mentioned on the ftp2 page
>>> http://camel.apache.org/ftp2
>>>
>>> See the file2 page for more options as the ftp component inherits these
>>> options
>>> http://camel.apache.org/file2.html
>>>
>>> The delay and consumer.delay option is the same. delay is just
>>> shorthand for consumer.delay.
>>>
>>> And no you cannot avoid the download if you later want to upload it
>>> someplace else, and the file exists.
>>> The ftp consumer don't know about this.
>>>
>>> A tricky improvement could be to add a new option to only download the
>>> file on demand, but that is a bit tricky
>>> as you would need a live connection and the ftp client to still be
>>> around and connected.
>>>
>>>
>>> What are you trying to do? To sync files between 2 ftp servers?
>>>
>>>
>>>
>>> On Thu, Jun 28, 2012 at 2:15 AM, David Capwell <dc...@gmail.com>
>>> wrote:
>>> > I am trying to use sftp (camel 2.9.2) and notice that every download is
>>> > spread out to one every 10 seconds (files are around 10kb). Based off
>>> the
>>> > examples in http://camel.apache.org/ftp2.html it looks like there are
>>> delay
>>> > options (they are not listened under options, but used in example) but
>>> when
>>> > I use them nothing changes. How can I have the client pull files
>>> faster?
>>> >
>>> > Another question, if I am trying to write the ftp files locally, is
>>> there
>>> > any way to avoid the file download if the local file is around?
>>> > I append fileExist=Ignore in the to() but it seems that the files still
>>> > get written again.
>>> >
>>> > Here is the route I am using:
>>> >
>>> > public void ftpToLocal() throws Exception {
>>> > runRoutes(new RouteBuilder() {
>>> > @Override
>>> > public void configure() throws Exception {
>>> > Registry registry = getContext().getRegistry();
>>> > String user = (String) registry.lookup("ftpUser");
>>> > String pass = (String) registry.lookup("ftpPass");
>>> >
>>> > // we use a delay of 60 minutes (eg. once pr. hour we poll the
>>> FTP
>>> > server
>>> > // long delay = 60 * 60 * 1000L;
>>> > // long delay = TimeUnit.SECONDS.toMillis(1);
>>> > long delay = 200;
>>> >
>>> > from("sftp://"+user+"@example.com/path/to/dir?password=" + pass
>>> +
>>> > "&binary=true&delay="+delay+"&consumer.delay=" + delay)
>>> > .to("file://data/ftp?fileExist=Ignore");
>>> > }
>>> > });
>>> > }
>>> >
>>> >
>>> > Thanks for your time reading this email.
>>>
>>>
>>>
>>> --
>>> Claus Ibsen
>>> -----------------
>>> FuseSource
>>> Email: cibsen@fusesource.com
>>> Web: http://fusesource.com
>>> Twitter: davsclaus, fusenews
>>> Blog: http://davsclaus.com
>>> Author of Camel in Action: http://www.manning.com/ibsen
>>>
>>
--
Claus Ibsen
-----------------
FuseSource
Email: cibsen@fusesource.com
Web: http://fusesource.com
Twitter: davsclaus, fusenews
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
Re: unable to control ftp delay
Posted by David Capwell <dc...@gmail.com>.
Playing with the delay fields defined in file2 seems to improve download
speeds. Now that this is working better I want to be able to skip any file
that has already been processed (should persist on restart). It looked to
me that I should be able to set this up by using the filter option, but I
am having a hard time figuring out how to use this option without using
spring.
here is the code I am using (might not compile cause its copy/paste from
diff places):
SimpleRegistry registry = new SimpleRegistry();
registry.put("fileExistsFilter", new CopyFilter());
CamelContext context = new DefaultCamelContext(registry);
context.addRoutes(new RouteBuilder() {
@Override
public void configure() throws Exception {
from("sftp://example.com/path/to/file?password=" + pass +
"&binary=true&delay="+delay+ "&initialDelay=" + delay +
"&filter=#fileExistsFilter")
.to("file://data/ftp?fileExist=Ignore");
}
});
context.start();
...
public static class CopyFilter implements GenericFileFilter {
@Override
public boolean accept(GenericFile genericFile) {
// check to see if file exists locally
File localFile = new File("data/ftp", genericFile.getFileName());
if(localFile.exists()) {
// check to make sure the sizes match
if(localFile.length() == genericFile.getFileLength()) {
// file has already been copied, ignore
return false;
}
}
return true;
}
}
Thanks for your time reading this email
On Thu, Jun 28, 2012 at 6:07 AM, David Capwell <dc...@gmail.com> wrote:
> I'll take a look at file2 to see if that helps.
>
> My goal is to push the data from FTP to s3. Copying locally so I know
> what has been processed,
> On Jun 28, 2012 12:48 AM, "Claus Ibsen" <cl...@gmail.com> wrote:
>
>> Hi
>>
>> As mentioned on the ftp2 page
>> http://camel.apache.org/ftp2
>>
>> See the file2 page for more options as the ftp component inherits these
>> options
>> http://camel.apache.org/file2.html
>>
>> The delay and consumer.delay option is the same. delay is just
>> shorthand for consumer.delay.
>>
>> And no you cannot avoid the download if you later want to upload it
>> someplace else, and the file exists.
>> The ftp consumer don't know about this.
>>
>> A tricky improvement could be to add a new option to only download the
>> file on demand, but that is a bit tricky
>> as you would need a live connection and the ftp client to still be
>> around and connected.
>>
>>
>> What are you trying to do? To sync files between 2 ftp servers?
>>
>>
>>
>> On Thu, Jun 28, 2012 at 2:15 AM, David Capwell <dc...@gmail.com>
>> wrote:
>> > I am trying to use sftp (camel 2.9.2) and notice that every download is
>> > spread out to one every 10 seconds (files are around 10kb). Based off
>> the
>> > examples in http://camel.apache.org/ftp2.html it looks like there are
>> delay
>> > options (they are not listened under options, but used in example) but
>> when
>> > I use them nothing changes. How can I have the client pull files
>> faster?
>> >
>> > Another question, if I am trying to write the ftp files locally, is
>> there
>> > any way to avoid the file download if the local file is around?
>> > I append fileExist=Ignore in the to() but it seems that the files still
>> > get written again.
>> >
>> > Here is the route I am using:
>> >
>> > public void ftpToLocal() throws Exception {
>> > runRoutes(new RouteBuilder() {
>> > @Override
>> > public void configure() throws Exception {
>> > Registry registry = getContext().getRegistry();
>> > String user = (String) registry.lookup("ftpUser");
>> > String pass = (String) registry.lookup("ftpPass");
>> >
>> > // we use a delay of 60 minutes (eg. once pr. hour we poll the
>> FTP
>> > server
>> > // long delay = 60 * 60 * 1000L;
>> > // long delay = TimeUnit.SECONDS.toMillis(1);
>> > long delay = 200;
>> >
>> > from("sftp://"+user+"@example.com/path/to/dir?password=" + pass
>> +
>> > "&binary=true&delay="+delay+"&consumer.delay=" + delay)
>> > .to("file://data/ftp?fileExist=Ignore");
>> > }
>> > });
>> > }
>> >
>> >
>> > Thanks for your time reading this email.
>>
>>
>>
>> --
>> Claus Ibsen
>> -----------------
>> FuseSource
>> Email: cibsen@fusesource.com
>> Web: http://fusesource.com
>> Twitter: davsclaus, fusenews
>> Blog: http://davsclaus.com
>> Author of Camel in Action: http://www.manning.com/ibsen
>>
>
Re: unable to control ftp delay
Posted by David Capwell <dc...@gmail.com>.
I'll take a look at file2 to see if that helps.
My goal is to push the data from FTP to s3. Copying locally so I know what
has been processed,
On Jun 28, 2012 12:48 AM, "Claus Ibsen" <cl...@gmail.com> wrote:
> Hi
>
> As mentioned on the ftp2 page
> http://camel.apache.org/ftp2
>
> See the file2 page for more options as the ftp component inherits these
> options
> http://camel.apache.org/file2.html
>
> The delay and consumer.delay option is the same. delay is just
> shorthand for consumer.delay.
>
> And no you cannot avoid the download if you later want to upload it
> someplace else, and the file exists.
> The ftp consumer don't know about this.
>
> A tricky improvement could be to add a new option to only download the
> file on demand, but that is a bit tricky
> as you would need a live connection and the ftp client to still be
> around and connected.
>
>
> What are you trying to do? To sync files between 2 ftp servers?
>
>
>
> On Thu, Jun 28, 2012 at 2:15 AM, David Capwell <dc...@gmail.com> wrote:
> > I am trying to use sftp (camel 2.9.2) and notice that every download is
> > spread out to one every 10 seconds (files are around 10kb). Based off
> the
> > examples in http://camel.apache.org/ftp2.html it looks like there are
> delay
> > options (they are not listened under options, but used in example) but
> when
> > I use them nothing changes. How can I have the client pull files faster?
> >
> > Another question, if I am trying to write the ftp files locally, is there
> > any way to avoid the file download if the local file is around?
> > I append fileExist=Ignore in the to() but it seems that the files still
> > get written again.
> >
> > Here is the route I am using:
> >
> > public void ftpToLocal() throws Exception {
> > runRoutes(new RouteBuilder() {
> > @Override
> > public void configure() throws Exception {
> > Registry registry = getContext().getRegistry();
> > String user = (String) registry.lookup("ftpUser");
> > String pass = (String) registry.lookup("ftpPass");
> >
> > // we use a delay of 60 minutes (eg. once pr. hour we poll the FTP
> > server
> > // long delay = 60 * 60 * 1000L;
> > // long delay = TimeUnit.SECONDS.toMillis(1);
> > long delay = 200;
> >
> > from("sftp://"+user+"@example.com/path/to/dir?password=" + pass +
> > "&binary=true&delay="+delay+"&consumer.delay=" + delay)
> > .to("file://data/ftp?fileExist=Ignore");
> > }
> > });
> > }
> >
> >
> > Thanks for your time reading this email.
>
>
>
> --
> Claus Ibsen
> -----------------
> FuseSource
> Email: cibsen@fusesource.com
> Web: http://fusesource.com
> Twitter: davsclaus, fusenews
> Blog: http://davsclaus.com
> Author of Camel in Action: http://www.manning.com/ibsen
>
Re: unable to control ftp delay
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
As mentioned on the ftp2 page
http://camel.apache.org/ftp2
See the file2 page for more options as the ftp component inherits these options
http://camel.apache.org/file2.html
The delay and consumer.delay option is the same. delay is just
shorthand for consumer.delay.
And no you cannot avoid the download if you later want to upload it
someplace else, and the file exists.
The ftp consumer don't know about this.
A tricky improvement could be to add a new option to only download the
file on demand, but that is a bit tricky
as you would need a live connection and the ftp client to still be
around and connected.
What are you trying to do? To sync files between 2 ftp servers?
On Thu, Jun 28, 2012 at 2:15 AM, David Capwell <dc...@gmail.com> wrote:
> I am trying to use sftp (camel 2.9.2) and notice that every download is
> spread out to one every 10 seconds (files are around 10kb). Based off the
> examples in http://camel.apache.org/ftp2.html it looks like there are delay
> options (they are not listened under options, but used in example) but when
> I use them nothing changes. How can I have the client pull files faster?
>
> Another question, if I am trying to write the ftp files locally, is there
> any way to avoid the file download if the local file is around?
> I append fileExist=Ignore in the to() but it seems that the files still
> get written again.
>
> Here is the route I am using:
>
> public void ftpToLocal() throws Exception {
> runRoutes(new RouteBuilder() {
> @Override
> public void configure() throws Exception {
> Registry registry = getContext().getRegistry();
> String user = (String) registry.lookup("ftpUser");
> String pass = (String) registry.lookup("ftpPass");
>
> // we use a delay of 60 minutes (eg. once pr. hour we poll the FTP
> server
> // long delay = 60 * 60 * 1000L;
> // long delay = TimeUnit.SECONDS.toMillis(1);
> long delay = 200;
>
> from("sftp://"+user+"@example.com/path/to/dir?password=" + pass +
> "&binary=true&delay="+delay+"&consumer.delay=" + delay)
> .to("file://data/ftp?fileExist=Ignore");
> }
> });
> }
>
>
> Thanks for your time reading this email.
--
Claus Ibsen
-----------------
FuseSource
Email: cibsen@fusesource.com
Web: http://fusesource.com
Twitter: davsclaus, fusenews
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen