is FTP2 asynchrounus?

[awold <aw...@hotmail.com>]

I have a business requirement to retrieve and process approx 4000 files per
day, each up to 5MB in size.  I would envision using the FTP2 to do a
schedule polling and then passing the each file through a number of
processor completing at an end point.  But I have concerns that this would
be a synchronous process such that the next file would not be processed
until the previous file was processed fully to the end point.  Is the FTP2
component synchronous only?

If my assumption is correct, it would seem that I might need to instead
implement a seda solution, where the first step is simply to put the
filename to a seda queue, and then asynchronous processing could ebing from
there.


-- 
View this message in context: http://camel.465427.n5.nabble.com/is-FTP2-asynchrounus-tp3261083p3261083.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: is FTP2 asynchrounus?

[awold <aw...@hotmail.com>]

I did notice the localWorkDirectory option and gave that some thought.  But
since I am dealing with medical data there HIPPA concerns and then I would
have to factor in security concerns related to storing these files on a
local directory.  I think at this point I am will to pay the performance
penalty of each entire message being contained within a message.

So ultimately it seems like my best option is the two routes.  One to pull
via ftp, the other as SEDA for subsequent concurrent processing.

But I noticed that setting concurrentConsumers=# will always force # number
of threads to be created.  I need more of a thread pooling scheme and maybe
that's available.  I just have not yet had the time to explore the
documentation or the book regarding that.
-- 
View this message in context: http://camel.465427.n5.nabble.com/is-FTP2-asynchrounus-tp3261083p3262236.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: is FTP2 asynchrounus?

[Claus Ibsen <cl...@gmail.com>]

Hi

You can have it use a localWorkDirectory which means it will stream
directly to file.
But then you should not have to do any deletion/move the file afterwards.

However if you need to move / delete the downloaded file then you
should store it to a file
And then have a 2nd route which pickup those files which they can process async.

So the 1st route is sync. The 2nd route is async / concurrent, but it
consumes the local files instead of the FTP.
Then you can use the seda / threads to turn the route into async from
that point forward.



On Thu, Nov 11, 2010 at 10:28 PM, awold <aw...@hotmail.com> wrote:
>
> Opps, I think the answer was staring me right in the face in the
> documentation.
>
>
>
> FTP Consumer does not support concurrency
> The FTP consumer (with the same endpoint) does not support concurrency (the
> backing FTP client is not thread safe).
> You can use multiple FTP consumers to poll from different endpoints. It is
> only a single endpoint that does not support concurrent consumers.
> The FTP producer does not have this issue, it supports concurrency.
>
> In the future we will add consumer pooling to Camel to allow this consumer
> to support concurrency as well.
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/is-FTP2-asynchrounus-tp3261083p3261140.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>



-- 
Claus Ibsen
-----------------
FuseSource
Email: cibsen@fusesource.com
Web: http://fusesource.com
Twitter: davsclaus
Blog: http://davsclaus.blogspot.com/
Author of Camel in Action: http://www.manning.com/ibsen/

Re: is FTP2 asynchrounus?

[awold <aw...@hotmail.com>]

Opps, I think the answer was staring me right in the face in the
documentation.



FTP Consumer does not support concurrency
The FTP consumer (with the same endpoint) does not support concurrency (the
backing FTP client is not thread safe).
You can use multiple FTP consumers to poll from different endpoints. It is
only a single endpoint that does not support concurrent consumers. 
The FTP producer does not have this issue, it supports concurrency.

In the future we will add consumer pooling to Camel to allow this consumer
to support concurrency as well.

-- 
View this message in context: http://camel.465427.n5.nabble.com/is-FTP2-asynchrounus-tp3261083p3261140.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: is FTP2 asynchrounus?

[Richard Kettelerij <ri...@gmail.com>]

I think it's synchronous.

In Camel 2.4 a lot of effort was put into asynchronous routing support. As
you can see on http://camel.apache.org/asynchronous-routing-engine.html
there are a number of components that support this new async model. FTP is
not on this list. 

Also Camel uses Apache Commons Net FTPClient under the hood. As far as I
know this client doesn't support asynchronous file transfers.
-- 
View this message in context: http://camel.465427.n5.nabble.com/is-FTP2-asynchrounus-tp3261083p3261175.html
Sent from the Camel - Users mailing list archive at Nabble.com.