You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by Michele <mi...@finconsgroup.com> on 2016/05/13 08:33:27 UTC
FTP Consumer has strange behaviour
Hi Everyoune,
I configure my route like this:
from("ftp://username@host/IN?passiveMode=true&localWorkDirectory=C:/tmp&password=xxxxxx&include=DDT_DA_SAP_[0-9]{8}_[0-9]{6}.txt&readLock=changed&maxMessagesPerPoll=1&charset=iso-8859-1")
.id("FileReader")
.split().tokenize("\n", 1000).streaming()
.log("${body}")
.to("seda:grouped")
.end();
in log file, I see this and no content:
-rw-r--r-- 1 ftp ftp 6070120 May 13 10:16
DDT_DA_SAP_20151123_123003.txt
Please help me.
Thanks in advance
Best Regards
Michele
--
View this message in context: http://camel.465427.n5.nabble.com/FTP-Consumer-has-strange-behaviour-tp5782537.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: FTP Consumer has strange behaviour
Posted by Brad Johnson <br...@mediadriver.com>.
Very cool. Glad to hear it's working.
On May 16, 2016 1:16 AM, "Michele" <mi...@finconsgroup.com>
wrote:
> Hi Brad,
>
> first of all, thank you for all.
>
> If I understood, I changed route configuration like this (doneFileName
> Policy on Local Producer and Consumer):
>
> <route id="RemoteFileReader">
> <from
> uri="ftp://username@host/IN?password=xxxxxx&move=Save&stepwise=false">
> <to uri="file:/tmp/localWork?doneFileName=${file:name}.done" />
> </route>
>
> <route id="LocalFileReader">
> <from
>
> uri="file:/tmp/localWork?doneFileName=${file:name}.done&delete=true&readLock=changed&readLockCheckInterval=5000&readLockTimeout=15000">
> ... process it
> </route>
> // OnException
> <route id="ReportBuilderRoute">
> <from uri="direct:messageDiscarded" />
> <marshal ref="LoggerDataFormat" />
> <setHeader headerName="OutputUriReport">
>
>
> <simple>{{local.work.directory.report.log}}?fileName=ERR_${file:name.noext}.csv&fileExist=Append</simple>
> </setHeader>
> <recipientList>
> <header>OutputUriReport</header>
> </recipientList>
> </route>
> // Upload outcome on FTP
> <route id="ReportFileUploader_Route">
> <from
>
> uri="{{local.work.directory.report.log}}?readLock=changed&readLockCheckInterval=5000&readLockTimeout=15000"
> />
> <log message="Tranfering report file ${in.header.CamelFileName}" />
> <to
> uri="ftp://username@host
> /IN/Errors?fileName=${file:name.noext}_${date:now:yyyyMMddHHmmss}.csv&password=xxxxx"
> />
> </route>
>
> doneFileName Policy didn't work on report builder and upload it on FTP
> (Route configured in previous post). I changed it introducing only readLock
> Policy on consumer (ReportFileUploader_Route) and when the whole report
> will
> be ready, it will move on FTP Errors folder. I hope it goes well. I tested
> it more than once and it work fine.
>
> Thank you for your support ;).
>
> Best Regards
> Michele
>
>
>
>
>
> --
> View this message in context:
> http://camel.465427.n5.nabble.com/FTP-Consumer-has-strange-behaviour-tp5782537p5782598.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
Re: FTP Consumer has strange behaviour
Posted by Michele <mi...@finconsgroup.com>.
Hi Brad,
first of all, thank you for all.
If I understood, I changed route configuration like this (doneFileName
Policy on Local Producer and Consumer):
<route id="RemoteFileReader">
<from
uri="ftp://username@host/IN?password=xxxxxx&move=Save&stepwise=false">
<to uri="file:/tmp/localWork?doneFileName=${file:name}.done" />
</route>
<route id="LocalFileReader">
<from
uri="file:/tmp/localWork?doneFileName=${file:name}.done&delete=true&readLock=changed&readLockCheckInterval=5000&readLockTimeout=15000">
... process it
</route>
// OnException
<route id="ReportBuilderRoute">
<from uri="direct:messageDiscarded" />
<marshal ref="LoggerDataFormat" />
<setHeader headerName="OutputUriReport">
<simple>{{local.work.directory.report.log}}?fileName=ERR_${file:name.noext}.csv&fileExist=Append</simple>
</setHeader>
<recipientList>
<header>OutputUriReport</header>
</recipientList>
</route>
// Upload outcome on FTP
<route id="ReportFileUploader_Route">
<from
uri="{{local.work.directory.report.log}}?readLock=changed&readLockCheckInterval=5000&readLockTimeout=15000"
/>
<log message="Tranfering report file ${in.header.CamelFileName}" />
<to
uri="ftp://username@host/IN/Errors?fileName=${file:name.noext}_${date:now:yyyyMMddHHmmss}.csv&password=xxxxx"
/>
</route>
doneFileName Policy didn't work on report builder and upload it on FTP
(Route configured in previous post). I changed it introducing only readLock
Policy on consumer (ReportFileUploader_Route) and when the whole report will
be ready, it will move on FTP Errors folder. I hope it goes well. I tested
it more than once and it work fine.
Thank you for your support ;).
Best Regards
Michele
--
View this message in context: http://camel.465427.n5.nabble.com/FTP-Consumer-has-strange-behaviour-tp5782537p5782598.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: FTP Consumer has strange behaviour
Posted by Brad Johnson <br...@mediadriver.com>.
I'd have to look up the exact FTP route settings for Camel as I don't
recall them. In FTP protocol the rename command is also a move command.
So you can FTP the entire file into a temporary directory and then do the
rename to move it to your processing directory. That means that the entire
file will be there already before it starts processing it. Another way
this is commonly done is your endpoint for file processing might look for
something like files ending with *.xml but when you FTP into that directory
you append .tmp to the end of the file. So the file processor endpoint
doesn't "see" the file until after the file rename which changes it from
.tmp to .xml.
Brad
On Sat, May 14, 2016 at 4:21 AM, Michele <mi...@finconsgroup.com>
wrote:
> Hi,
>
> Your route was working fine when you were reading from the local file
> system...
> It is right.
>
> In this case you'd leave that all alone and simply put in a new route to
> read from the FTP location,
> write it to a local location, and then rename the file. Ok I tryed this and
> it works fine.
>
> I have another problem: for each row processed, i write a failed outcome in
> csv file and then upload it on FTP Errors folder.
> I follow this link
>
> http://www.davsclaus.com/2010/12/camel-26-using-done-files-with-fileftp.html
> to do this, but after processing the whole file the upload of file doesn't
> work.
>
> Route is:
>
> <route id="RemoteFileReader">
> <from
> uri="ftp://username@host/IN?password=xxxxxx&move=Save&stepwise=false">
> <to uri="file:/tmp/localWork" />
> </route>
>
> <route id="LocalFileReader">
> <from
>
> uri="file:/tmp/localWork?delete=true&readLock=changed&readLockCheckInterval=5000&readLockTimeout=15000">
> ... process it
> </route>
> // OnException
> <route id="ReportBuilderRoute">
> <from uri="direct:messageDiscarded" />
> <marshal ref="LoggerDataFormat" />
> <setHeader headerName="OutputUriReport">
>
>
> <simple>{{local.work.directory.report.log}}?fileName=ERR_${file:name.noext}.csv&fileExist=Append&doneFileName=ERR_${file:name.noext}.csv.done</simple>
> </setHeader>
> <recipientList>
> <header>OutputUriReport</header>
> </recipientList>
> </route>
> // Upload outcome on FTP
> <route id="ReportFileUploader_Route">
> <from
>
> uri="{{local.work.directory.report.log}}?doneFileName=ERR_${file:name.noext}.csv.done&readLock=changed&readLockCheckInterval=5000&readLockTimeout=15000"
> />
> <to
> uri="ftp://username@host
> /IN/Errors?fileName=${file:name.noext}_${date:now:yyyyMMddHHmmss}.csv&password=xxxxx"
> />
> </route>
>
> The route is blocked...Any idea? where is the mistake?
>
> Thank you in advance.
>
> Best regards
>
> Michele
>
>
>
>
>
>
> --
> View this message in context:
> http://camel.465427.n5.nabble.com/FTP-Consumer-has-strange-behaviour-tp5782537p5782584.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
Re: FTP Consumer has strange behaviour
Posted by Michele <mi...@finconsgroup.com>.
Hi,
Your route was working fine when you were reading from the local file
system...
It is right.
In this case you'd leave that all alone and simply put in a new route to
read from the FTP location,
write it to a local location, and then rename the file. Ok I tryed this and
it works fine.
I have another problem: for each row processed, i write a failed outcome in
csv file and then upload it on FTP Errors folder.
I follow this link
http://www.davsclaus.com/2010/12/camel-26-using-done-files-with-fileftp.html
to do this, but after processing the whole file the upload of file doesn't
work.
Route is:
<route id="RemoteFileReader">
<from
uri="ftp://username@host/IN?password=xxxxxx&move=Save&stepwise=false">
<to uri="file:/tmp/localWork" />
</route>
<route id="LocalFileReader">
<from
uri="file:/tmp/localWork?delete=true&readLock=changed&readLockCheckInterval=5000&readLockTimeout=15000">
... process it
</route>
// OnException
<route id="ReportBuilderRoute">
<from uri="direct:messageDiscarded" />
<marshal ref="LoggerDataFormat" />
<setHeader headerName="OutputUriReport">
<simple>{{local.work.directory.report.log}}?fileName=ERR_${file:name.noext}.csv&fileExist=Append&doneFileName=ERR_${file:name.noext}.csv.done</simple>
</setHeader>
<recipientList>
<header>OutputUriReport</header>
</recipientList>
</route>
// Upload outcome on FTP
<route id="ReportFileUploader_Route">
<from
uri="{{local.work.directory.report.log}}?doneFileName=ERR_${file:name.noext}.csv.done&readLock=changed&readLockCheckInterval=5000&readLockTimeout=15000"
/>
<to
uri="ftp://username@host/IN/Errors?fileName=${file:name.noext}_${date:now:yyyyMMddHHmmss}.csv&password=xxxxx"
/>
</route>
The route is blocked...Any idea? where is the mistake?
Thank you in advance.
Best regards
Michele
--
View this message in context: http://camel.465427.n5.nabble.com/FTP-Consumer-has-strange-behaviour-tp5782537p5782584.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: FTP Consumer has strange behaviour
Posted by Brad Johnson <br...@mediadriver.com>.
Michelle,
One thing you should probably do when working with FTP files is make sure
your actions are atomic. I'm not sure if this is what you're running into
but it's a good idea anyway. FTP the file to a tmp directory and then move
the file via rename to a directory that you process in. That ensures that
the entire file is present before you start processing it. I don't know if
that will solve the problem or not I'm just saying its a good practice. Or
if you are using the same file name all the time then you can FTP the file
in with a .tmp extension so your file reader route doesn't detect and then
do a rename of the file to remove the .tmp and now your file reader sees
the new file and it is all there.
I've had issues when using FTP in anything but an atomic fashion like that.
Read/write locking is problematic in Java and with ancient technologies
like FTP and heterogeneous OSes. Your route was working fine when you were
reading from the local file system I believe. In this case you'd leave
that all alone and simply put in a new route to read from the FTP location,
write it to a local location, and then rename the file.
On Fri, May 13, 2016 at 4:13 AM, Michele <mi...@finconsgroup.com>
wrote:
> Hi Claus,
>
> I replace
> .split().tokenize("\n", 1000).streaming()
> with
> .split(body(String.class).tokenize("\n")).streaming()
>
> and the file content is read correctly.
>
> Why? Is wrong the FTP consumer configuration?
>
> I use Camel version 2.15.1 with JBoss Fuse 6.2.
>
> Thanks in advance.
>
> Best regards
>
> Michele
>
>
>
> --
> View this message in context:
> http://camel.465427.n5.nabble.com/FTP-Consumer-has-strange-behaviour-tp5782537p5782540.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
Re: FTP Consumer has strange behaviour
Posted by Michele <mi...@finconsgroup.com>.
Hi Claus,
I replace
.split().tokenize("\n", 1000).streaming()
with
.split(body(String.class).tokenize("\n")).streaming()
and the file content is read correctly.
Why? Is wrong the FTP consumer configuration?
I use Camel version 2.15.1 with JBoss Fuse 6.2.
Thanks in advance.
Best regards
Michele
--
View this message in context: http://camel.465427.n5.nabble.com/FTP-Consumer-has-strange-behaviour-tp5782537p5782540.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: FTP Consumer has strange behaviour
Posted by Michele <mi...@finconsgroup.com>.
Hi,
I remove the include parameters from options and I retry it.
The result is the same.
In log file
-rw-r--r-- 1 ftp ftp 6070120 May 13 10:16
DDT_DA_SAP_20151123_123003.txt
Thanks for your support.
Best regard
Michele
--
View this message in context: http://camel.465427.n5.nabble.com/FTP-Consumer-has-strange-behaviour-tp5782537p5782539.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: FTP Consumer has strange behaviour
Posted by Claus Ibsen <cl...@gmail.com>.
Your include is wrong, as its Java regular expression based, so be
careful with .txt as that does not match a .txt file, as you need to
do special for dots in regular expressions.
On Fri, May 13, 2016 at 10:33 AM, Michele
<mi...@finconsgroup.com> wrote:
> Hi Everyoune,
>
> I configure my route like this:
>
> from("ftp://username@host/IN?passiveMode=true&localWorkDirectory=C:/tmp&password=xxxxxx&include=DDT_DA_SAP_[0-9]{8}_[0-9]{6}.txt&readLock=changed&maxMessagesPerPoll=1&charset=iso-8859-1")
> .id("FileReader")
> .split().tokenize("\n", 1000).streaming()
> .log("${body}")
> .to("seda:grouped")
> .end();
>
> in log file, I see this and no content:
> -rw-r--r-- 1 ftp ftp 6070120 May 13 10:16
> DDT_DA_SAP_20151123_123003.txt
>
> Please help me.
>
> Thanks in advance
>
> Best Regards
> Michele
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/FTP-Consumer-has-strange-behaviour-tp5782537.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
--
Claus Ibsen
-----------------
http://davsclaus.com @davsclaus
Camel in Action 2: https://www.manning.com/ibsen2