You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by alexey-s <al...@mail.ru> on 2015/04/03 13:48:01 UTC
camel-ftp 2.15.1 return ftp error code 421
On the new version of Apache Camel 2.15.1 started to happen strange errors.
On this version 2.15.0 there is no error.
--
View this message in context: http://camel.465427.n5.nabble.com/camel-ftp-2-15-1-return-ftp-error-code-421-tp5765272.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: camel-ftp 2.15.1 return ftp error code 421
Posted by alexey-s <al...@mail.ru>.
A short excerpt from karaf.log.xxx file
--
View this message in context: http://camel.465427.n5.nabble.com/camel-ftp-2-15-1-return-ftp-error-code-421-tp5765272p5765368.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: camel-ftp 2.15.1 return ftp error code 421
Posted by alexey-s <al...@mail.ru>.
Yes, I tried to find an answer to the question why the process ceases to take
files from ftp server.
The most simple - revert to the old assembly Camel. Lo and behold, the files
were taken.
But just a couple of days. Server file number increased by several hundred.
The system stops to pick up the files. Began to appear regularly ftp error
421.
Well, I have a filter on the list of directories. In the filter have
multiple directories.
A miracle occurred. The system is working properly. But I have to handle no
small part, and all the directories. And then the session expires.
I do not see the solution, how to raise the connection and to a certain
point of the interrupted process, not the beginning.
----
Aleksey
--
View this message in context: http://camel.465427.n5.nabble.com/camel-ftp-2-15-1-return-ftp-error-code-421-tp5765272p5765509.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: camel-ftp 2.15.1 return ftp error code 421
Posted by "Willem.Jiang" <wi...@gmail.com>.
The first connection is used to check if the ftp server is still alive before
starting to pull the directory.
If the RemoteFileConsumer cannot connect to the server, it just return the
pulling process and wait for another pulling.
--
View this message in context: http://camel.465427.n5.nabble.com/camel-ftp-2-15-1-return-ftp-error-code-421-tp5765272p5765842.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: camel-ftp 2.15.1 return ftp error code 421
Posted by "Willem.Jiang" <wi...@gmail.com>.
I think you may need to find another way to pull the sub directory instead of
pulling root directory directly.
--
View this message in context: http://camel.465427.n5.nabble.com/camel-ftp-2-15-1-return-ftp-error-code-421-tp5765272p5765843.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: camel-ftp 2.15.1 return ftp error code 421
Posted by alexey-s <al...@mail.ru>.
An exemplary work plan Ftp consumer.
As we can see, at the beginning of a connection to the server.
FTPClient establishes a new connection for a list of files in each
directory.
At the end of the first connection is switched off.
Why hold the first connection?
Another question.
Some directories may not be available. Directory visible, and get a list of
files is impossible.
How can I go on to the next directory?
---
Aleksey
--
View this message in context: http://camel.465427.n5.nabble.com/camel-ftp-2-15-1-return-ftp-error-code-421-tp5765272p5765763.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: camel-ftp 2.15.1 return ftp error code 421
Posted by Willem Jiang <wi...@gmail.com>.
I just try to guess what’s the cause of issue of camel-ftp 2.15.1 by looking up the change log. I’m not sure if changing the receiving buffer could make the difference.
It looks like you just find the key of the issue, am I right?
--
Willem Jiang
Red Hat, Inc.
Web: http://www.redhat.com
Blog: http://willemjiang.blogspot.com (English)
http://jnn.iteye.com (Chinese)
Twitter: willemjiang
Weibo: 姜宁willem
On April 8, 2015 at 10:29:12 PM, alexey-s (alexins@mail.ru) wrote:
> It looks like quite a different matter.
> On the server, there is a restriction session. After 10 minutes, the server
> connection tearing.
> No problem in download speed of large files. Disconnection is the fault of
> the strategy routePolicyStopTime. But it's not scary.
>
> The main problem is to read a large number of file names.
> It is necessary to go into the database idempotentRepository. There check
> the name of a previously processed file.
> The total scan time a large number of file names transferred abroad in 10
> minutes.
> FtpConsumer not time to start downloading the file contents.
>
> Parameter maxMessagesPerPoll I do not override. By default, it is 1000.
>
> Current indicators idempotentRepository repository size: 194159 files.
>
> --
> Aleksey
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/camel-ftp-2-15-1-return-ftp-error-code-421-tp5765272p5765482.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
Re: camel-ftp 2.15.1 return ftp error code 421
Posted by alexey-s <al...@mail.ru>.
It looks like quite a different matter.
On the server, there is a restriction session. After 10 minutes, the server
connection tearing.
No problem in download speed of large files. Disconnection is the fault of
the strategy routePolicyStopTime. But it's not scary.
The main problem is to read a large number of file names.
It is necessary to go into the database idempotentRepository. There check
the name of a previously processed file.
The total scan time a large number of file names transferred abroad in 10
minutes.
FtpConsumer not time to start downloading the file contents.
Parameter maxMessagesPerPoll I do not override. By default, it is 1000.
Current indicators idempotentRepository repository size: 194159 files.
--
Aleksey
--
View this message in context: http://camel.465427.n5.nabble.com/camel-ftp-2-15-1-return-ftp-error-code-421-tp5765272p5765482.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: camel-ftp 2.15.1 return ftp error code 421
Posted by Willem Jiang <wi...@gmail.com>.
OK, here is another change CAMEL-8513[1] in Camel 2.15.1 which improves the download speed. I guess it may cause the ftp server complain about the server load. So the workaround could be set the receiveBufferSize option to 1024 to slow down the processing speed.
[1]https://issues.apache.org/jira/browse/CAMEL-8513
--
Willem Jiang
Red Hat, Inc.
Web: http://www.redhat.com
Blog: http://willemjiang.blogspot.com (English)
http://jnn.iteye.com (Chinese)
Twitter: willemjiang
Weibo: 姜宁willem
On April 6, 2015 at 10:35:40 PM, alexey-s (alexins@mail.ru) wrote:
> Honestly. 2.15.0 On this effect is much less than 2.15.1.
> I used to
> CronScheduledRoutePolicy ftpDocReadRoutePolicy = new
> CronScheduledRoutePolicy();
> ftpDocReadRoutePolicy.setRouteStartTime(routePolicyStartTime);
> ftpDocReadRoutePolicy.setRouteStopTime(routePolicyStopTime);
> ftpDocReadRoutePolicy.setRouteStopGracePeriod(routePolicyStopGracePeriod);
>
> I used to have broke the connection when receiving the contents of files
> from ftp server.
> Now the volume of data to be processed has increased by 50 times.
> To combat system start time and stop very large. No problem.
> To test the interval between the start and stop up to 10 minutes.
> The strangest thing that 2.15.1 error occurs frequently.
> For a couple of hours of tests in 2.15.0 fix one mistake.
> 2.15.1 At the same time, for the problem persists for more than 10 times.
>
>
> And here's remarks.
>
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/camel-ftp-2-15-1-return-ftp-error-code-421-tp5765272p5765367.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
Re: camel-ftp 2.15.1 return ftp error code 421
Posted by alexey-s <al...@mail.ru>.
Honestly. 2.15.0 On this effect is much less than 2.15.1.
I used to
CronScheduledRoutePolicy ftpDocReadRoutePolicy = new
CronScheduledRoutePolicy();
ftpDocReadRoutePolicy.setRouteStartTime(routePolicyStartTime);
ftpDocReadRoutePolicy.setRouteStopTime(routePolicyStopTime);
ftpDocReadRoutePolicy.setRouteStopGracePeriod(routePolicyStopGracePeriod);
I used to have broke the connection when receiving the contents of files
from ftp server.
Now the volume of data to be processed has increased by 50 times.
To combat system start time and stop very large. No problem.
To test the interval between the start and stop up to 10 minutes.
The strangest thing that 2.15.1 error occurs frequently.
For a couple of hours of tests in 2.15.0 fix one mistake.
2.15.1 At the same time, for the problem persists for more than 10 times.
And here's remarks.
--
View this message in context: http://camel.465427.n5.nabble.com/camel-ftp-2-15-1-return-ftp-error-code-421-tp5765272p5765367.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: camel-ftp 2.15.1 return ftp error code 421
Posted by Willem Jiang <wi...@gmail.com>.
There are only few changes between Camel 2.15.1 and Camel 2.15.0.
I’m not sure if CAMEL-7565[1] is related to issue that you faced.
[1]https://issues.apache.org/jira/browse/CAMEL-7565
--
Willem Jiang
Red Hat, Inc.
Web: http://www.redhat.com
Blog: http://willemjiang.blogspot.com (English)
http://jnn.iteye.com (Chinese)
Twitter: willemjiang
Weibo: 姜宁willem
On April 3, 2015 at 9:21:02 PM, alexey-s (alexins@mail.ru) wrote:
> Imagine that your FTP directory shows the "C:\Program Files" on Windows or
> "/usr/lib" on Linux. There is a very large number of files. Files can not be
> deleted.
> Process periodically scans all files in all folders, filters, and sorts.
> Uses a database table to store the names of files previously treated
> (IdempotentReposiotry).
>
> It is understood that when preprocessing is large.
> Here then is manifested error with ftp server. He believes that his
> overloaded work and an error 421.
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/camel-ftp-2-15-1-return-ftp-error-code-421-tp5765272p5765276.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
Re: camel-ftp 2.15.1 return ftp error code 421
Posted by alexey-s <al...@mail.ru>.
Imagine that your FTP directory shows the "C:\Program Files" on Windows or
"/usr/lib" on Linux. There is a very large number of files. Files can not be
deleted.
Process periodically scans all files in all folders, filters, and sorts.
Uses a database table to store the names of files previously treated
(IdempotentReposiotry).
It is understood that when preprocessing is large.
Here then is manifested error with ftp server. He believes that his
overloaded work and an error 421.
--
View this message in context: http://camel.465427.n5.nabble.com/camel-ftp-2-15-1-return-ftp-error-code-421-tp5765272p5765276.html
Sent from the Camel - Users mailing list archive at Nabble.com.