You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by "BURN, James" <Ja...@oup.com> on 2015/06/23 14:25:45 UTC
File2: readLock/ready file alternatives
Hi
I'm using Camel 2.13.2 on a Linux VM to collect/process files on a Windows server.
This is done with a Linux filemount onto a folder on the Windows server.
The issue we're getting is large (37Mb) files are often collected incomplete.
I tried using the readLock, as per:
<from uri="file:/mnt/DEVELOPMENT?delete=true&localWorkDirectory=/tmp&include=.*&readLock=rename&readLockTimeout=20000&readLockCheckInterval=7000"/>
however, this didn't work and I can understand that perhaps the Windows filelock methods are not supported (as per readLock notes in http://camel.apache.org/file2.html)
I also tried fileLock=rename, but again ended up with an incomplete file.
Getting a "ready" file written after the data file isn't an option as we have no access to the script which writes the data files.
Are there any other tricks we can use here. I'm wondering (and this is what I thought readLock did when I first found it) whether my Camel route can poll any new files for xx seconds to see if a file has stopped increasing in size. If so then it is collected. Is this what exclusiveReadLockStrategy would allow? If so, how to I set this up?
Thoughts most welcome.
Thanks
James
Oxford University Press (UK) Disclaimer
This message is confidential. You should not copy it or disclose its contents to anyone. You may use and apply the information for the intended purpose only. OUP does not accept legal responsibility for the contents of this message. Any views or opinions presented are those of the author only and not of OUP. If this email has come to you in error, please delete it, along with any attachments. Please note that OUP may intercept incoming and outgoing email communications.
RE: File2: readLock/ready file alternatives
Posted by jamesburn <Ja...@oup.com>.
Hi Claus
Thanks for the tips. TRACE logging did help see the innards of what was happening with the Readlock. I'm also embarrassed to admit that it was my file-browser which wasn't refreshing the filesize hence I thought the files weren't being completely copied!
I now see they are - Camel File2 ReadLock is working fine and am happy with the ability to tune the ReadLock timeouts.
Thanks for your help and Camel in general.
James
From: Claus Ibsen-2 [via Camel] [mailto:ml-node+s465427n5768503h27@n5.nabble.com]
Sent: 23 June 2015 14:34
To: BURN, James
Subject: Re: File2: readLock/ready file alternatives
Hi
Maybe try with longer timeout values.
You can enable DEBUG/TRACE logging on
org.apache.camel.component.file.strategy to see what it logs
The code is
https://github.com/apache/camel/blob/master/camel-core/src/main/java/org/apache/camel/component/file/strategy/FileChangedExclusiveReadLockStrategy.java
On Tue, Jun 23, 2015 at 3:06 PM, jamesburn <[hidden email]</user/SendEmail.jtp?type=node&node=5768503&i=0>> wrote:
> Hi
>
> Thanks Claus for your prompt response!
>
> I did try readLock=changed first:
> <from uri="file:/mnt/DEVELOPMENT?delete=true&localWorkDirectory=/tmp&include=.*&readLock=changed&readLockTimeout=20000&readLockCheckInterval=7000"/>
>
> but am still getting incomplete files delivered (mostly).
>
> I can see a zero byte .camelLock file written whilst the file is being collected. Can I monitor what is actually happening with readLock whilst it is working?
>
> Cheers
>
> James
>
> From: Claus Ibsen-2 [via Camel] [mailto:[hidden email]</user/SendEmail.jtp?type=node&node=5768503&i=1>]
> Sent: 23 June 2015 13:46
> To: BURN, James
> Subject: Re: File2: readLock/ready file alternatives
>
> Hi
>
> There is a readLock=change that check for file size / date
> modifications over a time window
>
> On Tue, Jun 23, 2015 at 2:25 PM, BURN, James <[hidden email]</user/SendEmail.jtp?type=node&node=5768500&i=0>> wrote:
>
>> Hi
>>
>> I'm using Camel 2.13.2 on a Linux VM to collect/process files on a Windows server.
>>
>> This is done with a Linux filemount onto a folder on the Windows server.
>>
>> The issue we're getting is large (37Mb) files are often collected incomplete.
>>
>> I tried using the readLock, as per:
>> <from uri="file:/mnt/DEVELOPMENT?delete=true&localWorkDirectory=/tmp&include=.*&readLock=rename&readLockTimeout=20000&readLockCheckInterval=7000"/>
>>
>> however, this didn't work and I can understand that perhaps the Windows filelock methods are not supported (as per readLock notes in http://camel.apache.org/file2.html)
>>
>> I also tried fileLock=rename, but again ended up with an incomplete file.
>>
>> Getting a "ready" file written after the data file isn't an option as we have no access to the script which writes the data files.
>>
>> Are there any other tricks we can use here. I'm wondering (and this is what I thought readLock did when I first found it) whether my Camel route can poll any new files for xx seconds to see if a file has stopped increasing in size. If so then it is collected. Is this what exclusiveReadLockStrategy would allow? If so, how to I set this up?
>>
>> Thoughts most welcome.
>>
>> Thanks
>>
>> James
>>
>>
>> Oxford University Press (UK) Disclaimer
>>
>> This message is confidential. You should not copy it or disclose its contents to anyone. You may use and apply the information for the intended purpose only. OUP does not accept legal responsibility for the contents of this message. Any views or opinions presented are those of the author only and not of OUP. If this email has come to you in error, please delete it, along with any attachments. Please note that OUP may intercept incoming and outgoing email communications.
>
>
>
> --
> Claus Ibsen
> -----------------
> Red Hat, Inc.
> Email: [hidden email]</user/SendEmail.jtp?type=node&node=5768500&i=1>
> Twitter: davsclaus
> Blog: http://davsclaus.com
> Author of Camel in Action: http://www.manning.com/ibsen
> hawtio: http://hawt.io/
> fabric8: http://fabric8.io/
>
> ________________________________
> If you reply to this email, your message will be added to the discussion below:
> http://camel.465427.n5.nabble.com/File2-readLock-ready-file-alternatives-tp5768499p5768500.html
> To start a new topic under Camel - Users, email [hidden email]</user/SendEmail.jtp?type=node&node=5768503&i=2><mailto:[hidden email]</user/SendEmail.jtp?type=node&node=5768503&i=3>>
> To unsubscribe from Camel - Users, click here<
> NAML<http://camel.465427.n5.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
> Oxford University Press (UK) Disclaimer
>
> This message is confidential. You should not copy it or disclose its contents to anyone. You may use and apply the information for the intended purpose only. OUP does not accept legal responsibility for the contents of this message. Any views or opinions presented are those of the author only and not of OUP. If this email has come to you in error, please delete it, along with any attachments. Please note that OUP may intercept incoming and outgoing email communications.
>
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/File2-readLock-ready-file-alternatives-tp5768499p5768502.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
--
Claus Ibsen
-----------------
Red Hat, Inc.
Email: [hidden email]</user/SendEmail.jtp?type=node&node=5768503&i=4>
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
hawtio: http://hawt.io/
fabric8: http://fabric8.io/
________________________________
If you reply to this email, your message will be added to the discussion below:
http://camel.465427.n5.nabble.com/File2-readLock-ready-file-alternatives-tp5768499p5768503.html
To start a new topic under Camel - Users, email ml-node+s465427n465428h2@n5.nabble.com<ma...@n5.nabble.com>
To unsubscribe from Camel - Users, click here<http://camel.465427.n5.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=465428&code=SmFtZXMuQnVybkBvdXAuY29tfDQ2NTQyOHw5Njc4MTEwNDY=>.
NAML<http://camel.465427.n5.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
Oxford University Press (UK) Disclaimer
This message is confidential. You should not copy it or disclose its contents to anyone. You may use and apply the information for the intended purpose only. OUP does not accept legal responsibility for the contents of this message. Any views or opinions presented are those of the author only and not of OUP. If this email has come to you in error, please delete it, along with any attachments. Please note that OUP may intercept incoming and outgoing email communications.
--
View this message in context: http://camel.465427.n5.nabble.com/File2-readLock-ready-file-alternatives-tp5768499p5768656.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: File2: readLock/ready file alternatives
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
Maybe try with longer timeout values.
You can enable DEBUG/TRACE logging on
org.apache.camel.component.file.strategy to see what it logs
The code is
https://github.com/apache/camel/blob/master/camel-core/src/main/java/org/apache/camel/component/file/strategy/FileChangedExclusiveReadLockStrategy.java
On Tue, Jun 23, 2015 at 3:06 PM, jamesburn <Ja...@oup.com> wrote:
> Hi
>
> Thanks Claus for your prompt response!
>
> I did try readLock=changed first:
> <from uri="file:/mnt/DEVELOPMENT?delete=true&localWorkDirectory=/tmp&include=.*&readLock=changed&readLockTimeout=20000&readLockCheckInterval=7000"/>
>
> but am still getting incomplete files delivered (mostly).
>
> I can see a zero byte .camelLock file written whilst the file is being collected. Can I monitor what is actually happening with readLock whilst it is working?
>
> Cheers
>
> James
>
> From: Claus Ibsen-2 [via Camel] [mailto:ml-node+s465427n5768500h34@n5.nabble.com]
> Sent: 23 June 2015 13:46
> To: BURN, James
> Subject: Re: File2: readLock/ready file alternatives
>
> Hi
>
> There is a readLock=change that check for file size / date
> modifications over a time window
>
> On Tue, Jun 23, 2015 at 2:25 PM, BURN, James <[hidden email]</user/SendEmail.jtp?type=node&node=5768500&i=0>> wrote:
>
>> Hi
>>
>> I'm using Camel 2.13.2 on a Linux VM to collect/process files on a Windows server.
>>
>> This is done with a Linux filemount onto a folder on the Windows server.
>>
>> The issue we're getting is large (37Mb) files are often collected incomplete.
>>
>> I tried using the readLock, as per:
>> <from uri="file:/mnt/DEVELOPMENT?delete=true&localWorkDirectory=/tmp&include=.*&readLock=rename&readLockTimeout=20000&readLockCheckInterval=7000"/>
>>
>> however, this didn't work and I can understand that perhaps the Windows filelock methods are not supported (as per readLock notes in http://camel.apache.org/file2.html)
>>
>> I also tried fileLock=rename, but again ended up with an incomplete file.
>>
>> Getting a "ready" file written after the data file isn't an option as we have no access to the script which writes the data files.
>>
>> Are there any other tricks we can use here. I'm wondering (and this is what I thought readLock did when I first found it) whether my Camel route can poll any new files for xx seconds to see if a file has stopped increasing in size. If so then it is collected. Is this what exclusiveReadLockStrategy would allow? If so, how to I set this up?
>>
>> Thoughts most welcome.
>>
>> Thanks
>>
>> James
>>
>>
>> Oxford University Press (UK) Disclaimer
>>
>> This message is confidential. You should not copy it or disclose its contents to anyone. You may use and apply the information for the intended purpose only. OUP does not accept legal responsibility for the contents of this message. Any views or opinions presented are those of the author only and not of OUP. If this email has come to you in error, please delete it, along with any attachments. Please note that OUP may intercept incoming and outgoing email communications.
>
>
>
> --
> Claus Ibsen
> -----------------
> Red Hat, Inc.
> Email: [hidden email]</user/SendEmail.jtp?type=node&node=5768500&i=1>
> Twitter: davsclaus
> Blog: http://davsclaus.com
> Author of Camel in Action: http://www.manning.com/ibsen
> hawtio: http://hawt.io/
> fabric8: http://fabric8.io/
>
> ________________________________
> If you reply to this email, your message will be added to the discussion below:
> http://camel.465427.n5.nabble.com/File2-readLock-ready-file-alternatives-tp5768499p5768500.html
> To start a new topic under Camel - Users, email ml-node+s465427n465428h2@n5.nabble.com<ma...@n5.nabble.com>
> To unsubscribe from Camel - Users, click here<http://camel.465427.n5.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=465428&code=SmFtZXMuQnVybkBvdXAuY29tfDQ2NTQyOHw5Njc4MTEwNDY=>.
> NAML<http://camel.465427.n5.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
> Oxford University Press (UK) Disclaimer
>
> This message is confidential. You should not copy it or disclose its contents to anyone. You may use and apply the information for the intended purpose only. OUP does not accept legal responsibility for the contents of this message. Any views or opinions presented are those of the author only and not of OUP. If this email has come to you in error, please delete it, along with any attachments. Please note that OUP may intercept incoming and outgoing email communications.
>
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/File2-readLock-ready-file-alternatives-tp5768499p5768502.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
--
Claus Ibsen
-----------------
Red Hat, Inc.
Email: cibsen@redhat.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
hawtio: http://hawt.io/
fabric8: http://fabric8.io/
RE: File2: readLock/ready file alternatives
Posted by jamesburn <Ja...@oup.com>.
Hi
Thanks Claus for your prompt response!
I did try readLock=changed first:
<from uri="file:/mnt/DEVELOPMENT?delete=true&localWorkDirectory=/tmp&include=.*&readLock=changed&readLockTimeout=20000&readLockCheckInterval=7000"/>
but am still getting incomplete files delivered (mostly).
I can see a zero byte .camelLock file written whilst the file is being collected. Can I monitor what is actually happening with readLock whilst it is working?
Cheers
James
From: Claus Ibsen-2 [via Camel] [mailto:ml-node+s465427n5768500h34@n5.nabble.com]
Sent: 23 June 2015 13:46
To: BURN, James
Subject: Re: File2: readLock/ready file alternatives
Hi
There is a readLock=change that check for file size / date
modifications over a time window
On Tue, Jun 23, 2015 at 2:25 PM, BURN, James <[hidden email]</user/SendEmail.jtp?type=node&node=5768500&i=0>> wrote:
> Hi
>
> I'm using Camel 2.13.2 on a Linux VM to collect/process files on a Windows server.
>
> This is done with a Linux filemount onto a folder on the Windows server.
>
> The issue we're getting is large (37Mb) files are often collected incomplete.
>
> I tried using the readLock, as per:
> <from uri="file:/mnt/DEVELOPMENT?delete=true&localWorkDirectory=/tmp&include=.*&readLock=rename&readLockTimeout=20000&readLockCheckInterval=7000"/>
>
> however, this didn't work and I can understand that perhaps the Windows filelock methods are not supported (as per readLock notes in http://camel.apache.org/file2.html)
>
> I also tried fileLock=rename, but again ended up with an incomplete file.
>
> Getting a "ready" file written after the data file isn't an option as we have no access to the script which writes the data files.
>
> Are there any other tricks we can use here. I'm wondering (and this is what I thought readLock did when I first found it) whether my Camel route can poll any new files for xx seconds to see if a file has stopped increasing in size. If so then it is collected. Is this what exclusiveReadLockStrategy would allow? If so, how to I set this up?
>
> Thoughts most welcome.
>
> Thanks
>
> James
>
>
> Oxford University Press (UK) Disclaimer
>
> This message is confidential. You should not copy it or disclose its contents to anyone. You may use and apply the information for the intended purpose only. OUP does not accept legal responsibility for the contents of this message. Any views or opinions presented are those of the author only and not of OUP. If this email has come to you in error, please delete it, along with any attachments. Please note that OUP may intercept incoming and outgoing email communications.
--
Claus Ibsen
-----------------
Red Hat, Inc.
Email: [hidden email]</user/SendEmail.jtp?type=node&node=5768500&i=1>
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
hawtio: http://hawt.io/
fabric8: http://fabric8.io/
________________________________
If you reply to this email, your message will be added to the discussion below:
http://camel.465427.n5.nabble.com/File2-readLock-ready-file-alternatives-tp5768499p5768500.html
To start a new topic under Camel - Users, email ml-node+s465427n465428h2@n5.nabble.com<ma...@n5.nabble.com>
To unsubscribe from Camel - Users, click here<http://camel.465427.n5.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=465428&code=SmFtZXMuQnVybkBvdXAuY29tfDQ2NTQyOHw5Njc4MTEwNDY=>.
NAML<http://camel.465427.n5.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
Oxford University Press (UK) Disclaimer
This message is confidential. You should not copy it or disclose its contents to anyone. You may use and apply the information for the intended purpose only. OUP does not accept legal responsibility for the contents of this message. Any views or opinions presented are those of the author only and not of OUP. If this email has come to you in error, please delete it, along with any attachments. Please note that OUP may intercept incoming and outgoing email communications.
--
View this message in context: http://camel.465427.n5.nabble.com/File2-readLock-ready-file-alternatives-tp5768499p5768502.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: File2: readLock/ready file alternatives
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
There is a readLock=change that check for file size / date
modifications over a time window
On Tue, Jun 23, 2015 at 2:25 PM, BURN, James <Ja...@oup.com> wrote:
> Hi
>
> I'm using Camel 2.13.2 on a Linux VM to collect/process files on a Windows server.
>
> This is done with a Linux filemount onto a folder on the Windows server.
>
> The issue we're getting is large (37Mb) files are often collected incomplete.
>
> I tried using the readLock, as per:
> <from uri="file:/mnt/DEVELOPMENT?delete=true&localWorkDirectory=/tmp&include=.*&readLock=rename&readLockTimeout=20000&readLockCheckInterval=7000"/>
>
> however, this didn't work and I can understand that perhaps the Windows filelock methods are not supported (as per readLock notes in http://camel.apache.org/file2.html)
>
> I also tried fileLock=rename, but again ended up with an incomplete file.
>
> Getting a "ready" file written after the data file isn't an option as we have no access to the script which writes the data files.
>
> Are there any other tricks we can use here. I'm wondering (and this is what I thought readLock did when I first found it) whether my Camel route can poll any new files for xx seconds to see if a file has stopped increasing in size. If so then it is collected. Is this what exclusiveReadLockStrategy would allow? If so, how to I set this up?
>
> Thoughts most welcome.
>
> Thanks
>
> James
>
>
> Oxford University Press (UK) Disclaimer
>
> This message is confidential. You should not copy it or disclose its contents to anyone. You may use and apply the information for the intended purpose only. OUP does not accept legal responsibility for the contents of this message. Any views or opinions presented are those of the author only and not of OUP. If this email has come to you in error, please delete it, along with any attachments. Please note that OUP may intercept incoming and outgoing email communications.
--
Claus Ibsen
-----------------
Red Hat, Inc.
Email: cibsen@redhat.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
hawtio: http://hawt.io/
fabric8: http://fabric8.io/