You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by mcrive <mc...@optasportsdata.com> on 2010/02/10 17:17:21 UTC

GenericFileProducer behavior when temp file is used

Hi,
looking into the source code of GenericFileProducer I found out that when
using the tempFileName option following happens:

- if final file exists it gets deleted
- temp file gets written
- temp file gets renamed into final file

Deleting the final file before uploading the new file seems unsafe.
Suppose there is a network error (FTP) and the upload fails, then the final
file would be lost.
Uploading big files will result on missing final file for a while.

Would it be possible to change this behavior or add an option to change it?

I would do following:
- upload temp file
- delete final file
- rename temp file into final file

-- 
View this message in context: http://old.nabble.com/GenericFileProducer-behavior-when-temp-file-is-used-tp27533791p27533791.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: GenericFileProducer behavior when temp file is used

Posted by Claus Ibsen <cl...@gmail.com>.
On Fri, Feb 12, 2010 at 9:45 AM, mcrive <mc...@optasportsdata.com> wrote:
>
> I' ve opened a ticket for it
> https://issues.apache.org/activemq/browse/CAMEL-2466
>
> I will try to produce a patch as well but I don't have a very big experience
> in Java...
> where should I post it?
>

See this wiki page
http://activemq.apache.org/contributing.html

eg attach it to the JIRA ticket. Remember to ticket [x] in grant
license to Apache or what the wording is.


>
>
> Claus Ibsen-2 wrote:
>>
>> On Thu, Feb 11, 2010 at 6:00 PM, mcrive <mc...@optasportsdata.com> wrote:
>>>
>>> In our use case we are sending feeds which gets updated quite often,
>>> this means almost the time the file won't be a new file but an update of
>>> an
>>> already existing one.
>>> While uploading the update the final file will be completely missing in
>>> the
>>> target system.
>>>
>>> In such a situation the way you perform file upload is not safe.
>>>
>>
>> However uploading / overriding directly to a live system from a remote
>> system is probably not the best solution.
>> What if the file uploaded is corrupt or whatnot. If you have a system
>> in place that sanity check the uploaded file
>> before its exposes as live file.
>>
>> Also http based solutions often have cache build in which can return
>> cached content while the live file is being updated.
>>
>> That said you are welcome to create a JIRA ticket.
>>
>> And as always patches is welcome, and we are more likely to accept
>> them if there is unit test included.
>>
>>
>>> Regards,
>>> Marco
>>>
>>>
>>> Claus Ibsen-2 wrote:
>>>>
>>>> On Wed, Feb 10, 2010 at 5:17 PM, mcrive <mc...@optasportsdata.com>
>>>> wrote:
>>>>>
>>>>> Hi,
>>>>> looking into the source code of GenericFileProducer I found out that
>>>>> when
>>>>> using the tempFileName option following happens:
>>>>>
>>>>> - if final file exists it gets deleted
>>>>> - temp file gets written
>>>>> - temp file gets renamed into final file
>>>>>
>>>>> Deleting the final file before uploading the new file seems unsafe.
>>>>> Suppose there is a network error (FTP) and the upload fails, then the
>>>>> final
>>>>> file would be lost.
>>>>> Uploading big files will result on missing final file for a while.
>>>>>
>>>>> Would it be possible to change this behavior or add an option to change
>>>>> it?
>>>>>
>>>>> I would do following:
>>>>> - upload temp file
>>>>> - delete final file
>>>>> - rename temp file into final file
>>>>>
>>>>
>>>> Not better IMHO as its better to fail fast. Deleting the final file is
>>>> a fail safe check. In general the final file should not already exists
>>>> on the FTP server.
>>>>
>>>> The current behavior is the mot correct IMHO
>>>>
>>>>> --
>>>>> View this message in context:
>>>>> http://old.nabble.com/GenericFileProducer-behavior-when-temp-file-is-used-tp27533791p27533791.html
>>>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Claus Ibsen
>>>> Apache Camel Committer
>>>>
>>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>>> Open Source Integration: http://fusesource.com
>>>> Blog: http://davsclaus.blogspot.com/
>>>> Twitter: http://twitter.com/davsclaus
>>>>
>>>>
>>>
>>> --
>>> View this message in context:
>>> http://old.nabble.com/GenericFileProducer-behavior-when-temp-file-is-used-tp27533791p27550723.html
>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>
>>>
>>
>>
>>
>> --
>> Claus Ibsen
>> Apache Camel Committer
>>
>> Author of Camel in Action: http://www.manning.com/ibsen/
>> Open Source Integration: http://fusesource.com
>> Blog: http://davsclaus.blogspot.com/
>> Twitter: http://twitter.com/davsclaus
>>
>>
>
> --
> View this message in context: http://old.nabble.com/GenericFileProducer-behavior-when-temp-file-is-used-tp27533791p27560390.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
>



-- 
Claus Ibsen
Apache Camel Committer

Author of Camel in Action: http://www.manning.com/ibsen/
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus

Re: GenericFileProducer behavior when temp file is used

Posted by mcrive <mc...@optasportsdata.com>.
I' ve opened a ticket for it
https://issues.apache.org/activemq/browse/CAMEL-2466

I will try to produce a patch as well but I don't have a very big experience
in Java...
where should I post it?



Claus Ibsen-2 wrote:
> 
> On Thu, Feb 11, 2010 at 6:00 PM, mcrive <mc...@optasportsdata.com> wrote:
>>
>> In our use case we are sending feeds which gets updated quite often,
>> this means almost the time the file won't be a new file but an update of
>> an
>> already existing one.
>> While uploading the update the final file will be completely missing in
>> the
>> target system.
>>
>> In such a situation the way you perform file upload is not safe.
>>
> 
> However uploading / overriding directly to a live system from a remote
> system is probably not the best solution.
> What if the file uploaded is corrupt or whatnot. If you have a system
> in place that sanity check the uploaded file
> before its exposes as live file.
> 
> Also http based solutions often have cache build in which can return
> cached content while the live file is being updated.
> 
> That said you are welcome to create a JIRA ticket.
> 
> And as always patches is welcome, and we are more likely to accept
> them if there is unit test included.
> 
> 
>> Regards,
>> Marco
>>
>>
>> Claus Ibsen-2 wrote:
>>>
>>> On Wed, Feb 10, 2010 at 5:17 PM, mcrive <mc...@optasportsdata.com>
>>> wrote:
>>>>
>>>> Hi,
>>>> looking into the source code of GenericFileProducer I found out that
>>>> when
>>>> using the tempFileName option following happens:
>>>>
>>>> - if final file exists it gets deleted
>>>> - temp file gets written
>>>> - temp file gets renamed into final file
>>>>
>>>> Deleting the final file before uploading the new file seems unsafe.
>>>> Suppose there is a network error (FTP) and the upload fails, then the
>>>> final
>>>> file would be lost.
>>>> Uploading big files will result on missing final file for a while.
>>>>
>>>> Would it be possible to change this behavior or add an option to change
>>>> it?
>>>>
>>>> I would do following:
>>>> - upload temp file
>>>> - delete final file
>>>> - rename temp file into final file
>>>>
>>>
>>> Not better IMHO as its better to fail fast. Deleting the final file is
>>> a fail safe check. In general the final file should not already exists
>>> on the FTP server.
>>>
>>> The current behavior is the mot correct IMHO
>>>
>>>> --
>>>> View this message in context:
>>>> http://old.nabble.com/GenericFileProducer-behavior-when-temp-file-is-used-tp27533791p27533791.html
>>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> Claus Ibsen
>>> Apache Camel Committer
>>>
>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>> Open Source Integration: http://fusesource.com
>>> Blog: http://davsclaus.blogspot.com/
>>> Twitter: http://twitter.com/davsclaus
>>>
>>>
>>
>> --
>> View this message in context:
>> http://old.nabble.com/GenericFileProducer-behavior-when-temp-file-is-used-tp27533791p27550723.html
>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>
>>
> 
> 
> 
> -- 
> Claus Ibsen
> Apache Camel Committer
> 
> Author of Camel in Action: http://www.manning.com/ibsen/
> Open Source Integration: http://fusesource.com
> Blog: http://davsclaus.blogspot.com/
> Twitter: http://twitter.com/davsclaus
> 
> 

-- 
View this message in context: http://old.nabble.com/GenericFileProducer-behavior-when-temp-file-is-used-tp27533791p27560390.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: GenericFileProducer behavior when temp file is used

Posted by Claus Ibsen <cl...@gmail.com>.
On Thu, Feb 11, 2010 at 6:00 PM, mcrive <mc...@optasportsdata.com> wrote:
>
> In our use case we are sending feeds which gets updated quite often,
> this means almost the time the file won't be a new file but an update of an
> already existing one.
> While uploading the update the final file will be completely missing in the
> target system.
>
> In such a situation the way you perform file upload is not safe.
>

However uploading / overriding directly to a live system from a remote
system is probably not the best solution.
What if the file uploaded is corrupt or whatnot. If you have a system
in place that sanity check the uploaded file
before its exposes as live file.

Also http based solutions often have cache build in which can return
cached content while the live file is being updated.

That said you are welcome to create a JIRA ticket.

And as always patches is welcome, and we are more likely to accept
them if there is unit test included.


> Regards,
> Marco
>
>
> Claus Ibsen-2 wrote:
>>
>> On Wed, Feb 10, 2010 at 5:17 PM, mcrive <mc...@optasportsdata.com> wrote:
>>>
>>> Hi,
>>> looking into the source code of GenericFileProducer I found out that when
>>> using the tempFileName option following happens:
>>>
>>> - if final file exists it gets deleted
>>> - temp file gets written
>>> - temp file gets renamed into final file
>>>
>>> Deleting the final file before uploading the new file seems unsafe.
>>> Suppose there is a network error (FTP) and the upload fails, then the
>>> final
>>> file would be lost.
>>> Uploading big files will result on missing final file for a while.
>>>
>>> Would it be possible to change this behavior or add an option to change
>>> it?
>>>
>>> I would do following:
>>> - upload temp file
>>> - delete final file
>>> - rename temp file into final file
>>>
>>
>> Not better IMHO as its better to fail fast. Deleting the final file is
>> a fail safe check. In general the final file should not already exists
>> on the FTP server.
>>
>> The current behavior is the mot correct IMHO
>>
>>> --
>>> View this message in context:
>>> http://old.nabble.com/GenericFileProducer-behavior-when-temp-file-is-used-tp27533791p27533791.html
>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>
>>>
>>
>>
>>
>> --
>> Claus Ibsen
>> Apache Camel Committer
>>
>> Author of Camel in Action: http://www.manning.com/ibsen/
>> Open Source Integration: http://fusesource.com
>> Blog: http://davsclaus.blogspot.com/
>> Twitter: http://twitter.com/davsclaus
>>
>>
>
> --
> View this message in context: http://old.nabble.com/GenericFileProducer-behavior-when-temp-file-is-used-tp27533791p27550723.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
>



-- 
Claus Ibsen
Apache Camel Committer

Author of Camel in Action: http://www.manning.com/ibsen/
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus

Re: GenericFileProducer behavior when temp file is used

Posted by mcrive <mc...@optasportsdata.com>.
In our use case we are sending feeds which gets updated quite often, 
this means almost the time the file won't be a new file but an update of an
already existing one.
While uploading the update the final file will be completely missing in the
target system.

In such a situation the way you perform file upload is not safe.

Regards,
Marco


Claus Ibsen-2 wrote:
> 
> On Wed, Feb 10, 2010 at 5:17 PM, mcrive <mc...@optasportsdata.com> wrote:
>>
>> Hi,
>> looking into the source code of GenericFileProducer I found out that when
>> using the tempFileName option following happens:
>>
>> - if final file exists it gets deleted
>> - temp file gets written
>> - temp file gets renamed into final file
>>
>> Deleting the final file before uploading the new file seems unsafe.
>> Suppose there is a network error (FTP) and the upload fails, then the
>> final
>> file would be lost.
>> Uploading big files will result on missing final file for a while.
>>
>> Would it be possible to change this behavior or add an option to change
>> it?
>>
>> I would do following:
>> - upload temp file
>> - delete final file
>> - rename temp file into final file
>>
> 
> Not better IMHO as its better to fail fast. Deleting the final file is
> a fail safe check. In general the final file should not already exists
> on the FTP server.
> 
> The current behavior is the mot correct IMHO
> 
>> --
>> View this message in context:
>> http://old.nabble.com/GenericFileProducer-behavior-when-temp-file-is-used-tp27533791p27533791.html
>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>
>>
> 
> 
> 
> -- 
> Claus Ibsen
> Apache Camel Committer
> 
> Author of Camel in Action: http://www.manning.com/ibsen/
> Open Source Integration: http://fusesource.com
> Blog: http://davsclaus.blogspot.com/
> Twitter: http://twitter.com/davsclaus
> 
> 

-- 
View this message in context: http://old.nabble.com/GenericFileProducer-behavior-when-temp-file-is-used-tp27533791p27550723.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: GenericFileProducer behavior when temp file is used

Posted by Claus Ibsen <cl...@gmail.com>.
On Wed, Feb 10, 2010 at 5:17 PM, mcrive <mc...@optasportsdata.com> wrote:
>
> Hi,
> looking into the source code of GenericFileProducer I found out that when
> using the tempFileName option following happens:
>
> - if final file exists it gets deleted
> - temp file gets written
> - temp file gets renamed into final file
>
> Deleting the final file before uploading the new file seems unsafe.
> Suppose there is a network error (FTP) and the upload fails, then the final
> file would be lost.
> Uploading big files will result on missing final file for a while.
>
> Would it be possible to change this behavior or add an option to change it?
>
> I would do following:
> - upload temp file
> - delete final file
> - rename temp file into final file
>

Not better IMHO as its better to fail fast. Deleting the final file is
a fail safe check. In general the final file should not already exists
on the FTP server.

The current behavior is the mot correct IMHO

> --
> View this message in context: http://old.nabble.com/GenericFileProducer-behavior-when-temp-file-is-used-tp27533791p27533791.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
>



-- 
Claus Ibsen
Apache Camel Committer

Author of Camel in Action: http://www.manning.com/ibsen/
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus