You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by "Sam (Stephen Samuel)" <sa...@gmail.com> on 2012/07/06 11:17:08 UTC
Re: IdempotentRepository
"This operation is used if the option eager has been enabled."
Take a look at that, that might be why confirm is not being called.
On Fri, Jul 6, 2012 at 7:17 AM, Christian Lipp <Ch...@xion.at>wrote:
> Hello!
>
> I am fetching files from a FTP server (severall GB for the next years).
> The files are produced daily in directories which correspond to the date,
> like
>
> - 20120501
> - 20120502
> - ...
>
> I have only read rights and I am not the only consumer. This means that
> they keep the last month or so on the server and I fetch on a daily base.
> To avoid that I am fetching files twice I want to use an
> IdempotentRepository implementation. I don't want to save each file in a
> database or in a text file because the service will run for years and this
> is just unnecessary data.
>
> What I want to store is the last processed date only. This handles just
> the directories and would mean that I need some other strategy for the
> files. I could combine this approach with the default in memory based
> store. But let just stick to the directories:
>
> I read the directory sorted by file name. The IdempotentRepository is
> called by the FtpConsumer with
>
> - start()
> - contains() for every directory and file
> - add() for files only
>
> and that's it. No stop(), no confirm(). When I have errors, sometimes
> remove() is called. Since the repository is called only with String (the
> full path) I have not information if I deal with directories or files. I
> know it from the structure, but I am not able to implement a generic
> solution.
>
> Anyway the idea is:
>
> - Store the LastProcessedDate inside the repository
> - contains(): if the path contains an already processed date
> (<LastProcessedDate) then I skip it (return true) otherwise return false.
> - add(): if add() jumps to the next directory I set the LastProcessedDate
> to the directory before
>
> The only problem is the last processed directory: even if it is finished I
> do not get the chance to mark it as processed (set LastProcessedDate to its
> value).
>
> So finally my questions: do you think this approach makes sense and if
> yes: how would you deal with the last processed directory?
> If no, how would you solve it?
>
> Thanks and kind regards, Christian
>
--
-Sam
AW: IdempotentRepository
Posted by Christian Lipp <Ch...@xion.at>.
My problem is that the number (date) is in the directory, not in the file.
So I only know that I am finished when it moves to a new directory which doesn't work at the end.
Regards, Christian
-----Ursprüngliche Nachricht-----
Von: Marco Westermann [mailto:MarWestermann@gmx.de]
Gesendet: Freitag, 06. Juli 2012 12:10
An: users@camel.apache.org
Betreff: Re: IdempotentRepository
Hi,
I had a very similar problem. I wrote a little bean which writes the last number processed to a file. Another method gives me the last number processed.
regards, Marco
Am 06.07.2012 11:53, schrieb Christian Lipp:
> Do you mean " eagerMaxMessagesPerPoll" ?
> I set it to false, but nothing changed.
>
> In the camel code I could only find
>
> contains in GenericFileConsumer
> Add, remove in GenericFileOnCompletion
>
> Regards, CL
>
> -----Ursprüngliche Nachricht-----
> Von: Sam (Stephen Samuel) [mailto:samspade79@gmail.com]
> Gesendet: Freitag, 06. Juli 2012 11:17
> An: users@camel.apache.org
> Betreff: Re: IdempotentRepository
>
> "This operation is used if the option eager has been enabled."
>
> Take a look at that, that might be why confirm is not being called.
>
> On Fri, Jul 6, 2012 at 7:17 AM, Christian Lipp<Ch...@xion.at>wrote:
>
>> Hello!
>>
>> I am fetching files from a FTP server (severall GB for the next years).
>> The files are produced daily in directories which correspond to the
>> date, like
>>
>> - 20120501
>> - 20120502
>> - ...
>>
>> I have only read rights and I am not the only consumer. This means
>> that they keep the last month or so on the server and I fetch on a daily base.
>> To avoid that I am fetching files twice I want to use an
>> IdempotentRepository implementation. I don't want to save each file
>> in a database or in a text file because the service will run for
>> years and this is just unnecessary data.
>>
>> What I want to store is the last processed date only. This handles
>> just the directories and would mean that I need some other strategy
>> for the files. I could combine this approach with the default in
>> memory based store. But let just stick to the directories:
>>
>> I read the directory sorted by file name. The IdempotentRepository is
>> called by the FtpConsumer with
>>
>> - start()
>> - contains() for every directory and file
>> - add() for files only
>>
>> and that's it. No stop(), no confirm(). When I have errors, sometimes
>> remove() is called. Since the repository is called only with String
>> (the full path) I have not information if I deal with directories or
>> files. I know it from the structure, but I am not able to implement a
>> generic solution.
>>
>> Anyway the idea is:
>>
>> - Store the LastProcessedDate inside the repository
>> - contains(): if the path contains an already processed date
>> (<LastProcessedDate) then I skip it (return true) otherwise return false.
>> - add(): if add() jumps to the next directory I set the
>> LastProcessedDate to the directory before
>>
>> The only problem is the last processed directory: even if it is
>> finished I do not get the chance to mark it as processed (set
>> LastProcessedDate to its value).
>>
>> So finally my questions: do you think this approach makes sense and
>> if
>> yes: how would you deal with the last processed directory?
>> If no, how would you solve it?
>>
>> Thanks and kind regards, Christian
>>
>
>
> --
> -Sam
>
Re: IdempotentRepository
Posted by Marco Westermann <Ma...@gmx.de>.
Hi,
I had a very similar problem. I wrote a little bean which writes the
last number processed to a file. Another method gives me the last number
processed.
regards, Marco
Am 06.07.2012 11:53, schrieb Christian Lipp:
> Do you mean " eagerMaxMessagesPerPoll" ?
> I set it to false, but nothing changed.
>
> In the camel code I could only find
>
> contains in GenericFileConsumer
> Add, remove in GenericFileOnCompletion
>
> Regards, CL
>
> -----Ursprüngliche Nachricht-----
> Von: Sam (Stephen Samuel) [mailto:samspade79@gmail.com]
> Gesendet: Freitag, 06. Juli 2012 11:17
> An: users@camel.apache.org
> Betreff: Re: IdempotentRepository
>
> "This operation is used if the option eager has been enabled."
>
> Take a look at that, that might be why confirm is not being called.
>
> On Fri, Jul 6, 2012 at 7:17 AM, Christian Lipp<Ch...@xion.at>wrote:
>
>> Hello!
>>
>> I am fetching files from a FTP server (severall GB for the next years).
>> The files are produced daily in directories which correspond to the
>> date, like
>>
>> - 20120501
>> - 20120502
>> - ...
>>
>> I have only read rights and I am not the only consumer. This means
>> that they keep the last month or so on the server and I fetch on a daily base.
>> To avoid that I am fetching files twice I want to use an
>> IdempotentRepository implementation. I don't want to save each file in
>> a database or in a text file because the service will run for years
>> and this is just unnecessary data.
>>
>> What I want to store is the last processed date only. This handles
>> just the directories and would mean that I need some other strategy
>> for the files. I could combine this approach with the default in
>> memory based store. But let just stick to the directories:
>>
>> I read the directory sorted by file name. The IdempotentRepository is
>> called by the FtpConsumer with
>>
>> - start()
>> - contains() for every directory and file
>> - add() for files only
>>
>> and that's it. No stop(), no confirm(). When I have errors, sometimes
>> remove() is called. Since the repository is called only with String
>> (the full path) I have not information if I deal with directories or
>> files. I know it from the structure, but I am not able to implement a
>> generic solution.
>>
>> Anyway the idea is:
>>
>> - Store the LastProcessedDate inside the repository
>> - contains(): if the path contains an already processed date
>> (<LastProcessedDate) then I skip it (return true) otherwise return false.
>> - add(): if add() jumps to the next directory I set the
>> LastProcessedDate to the directory before
>>
>> The only problem is the last processed directory: even if it is
>> finished I do not get the chance to mark it as processed (set
>> LastProcessedDate to its value).
>>
>> So finally my questions: do you think this approach makes sense and if
>> yes: how would you deal with the last processed directory?
>> If no, how would you solve it?
>>
>> Thanks and kind regards, Christian
>>
>
>
> --
> -Sam
>
AW: IdempotentRepository
Posted by Christian Lipp <Ch...@xion.at>.
eagerMaxMessagesPerPoll is true by default, so it doesn't change anything to set it.
There was no difference for true/false.
Regards, Christian
-----Ursprüngliche Nachricht-----
Von: Sam (Stephen Samuel) [mailto:samspade79@gmail.com]
Gesendet: Freitag, 06. Juli 2012 13:13
An: users@camel.apache.org
Betreff: Re: IdempotentRepository
eagerMaxMessagesPerPoll needs to be set to true.
On Fri, Jul 6, 2012 at 10:53 AM, Christian Lipp <Ch...@xion.at>wrote:
> Do you mean " eagerMaxMessagesPerPoll" ?
> I set it to false, but nothing changed.
>
> In the camel code I could only find
>
> contains in GenericFileConsumer
> Add, remove in GenericFileOnCompletion
>
> Regards, CL
>
> -----Ursprüngliche Nachricht-----
> Von: Sam (Stephen Samuel) [mailto:samspade79@gmail.com]
> Gesendet: Freitag, 06. Juli 2012 11:17
> An: users@camel.apache.org
> Betreff: Re: IdempotentRepository
>
> "This operation is used if the option eager has been enabled."
>
> Take a look at that, that might be why confirm is not being called.
>
> On Fri, Jul 6, 2012 at 7:17 AM, Christian Lipp <Christian.Lipp@xion.at
> >wrote:
>
> > Hello!
> >
> > I am fetching files from a FTP server (severall GB for the next years).
> > The files are produced daily in directories which correspond to the
> > date, like
> >
> > - 20120501
> > - 20120502
> > - ...
> >
> > I have only read rights and I am not the only consumer. This means
> > that they keep the last month or so on the server and I fetch on a
> > daily
> base.
> > To avoid that I am fetching files twice I want to use an
> > IdempotentRepository implementation. I don't want to save each file
> > in a database or in a text file because the service will run for
> > years and this is just unnecessary data.
> >
> > What I want to store is the last processed date only. This handles
> > just the directories and would mean that I need some other strategy
> > for the files. I could combine this approach with the default in
> > memory based store. But let just stick to the directories:
> >
> > I read the directory sorted by file name. The IdempotentRepository
> > is called by the FtpConsumer with
> >
> > - start()
> > - contains() for every directory and file
> > - add() for files only
> >
> > and that's it. No stop(), no confirm(). When I have errors,
> > sometimes
> > remove() is called. Since the repository is called only with String
> > (the full path) I have not information if I deal with directories or
> > files. I know it from the structure, but I am not able to implement
> > a generic solution.
> >
> > Anyway the idea is:
> >
> > - Store the LastProcessedDate inside the repository
> > - contains(): if the path contains an already processed date
> > (<LastProcessedDate) then I skip it (return true) otherwise return false.
> > - add(): if add() jumps to the next directory I set the
> > LastProcessedDate to the directory before
> >
> > The only problem is the last processed directory: even if it is
> > finished I do not get the chance to mark it as processed (set
> > LastProcessedDate to its value).
> >
> > So finally my questions: do you think this approach makes sense and
> > if
> > yes: how would you deal with the last processed directory?
> > If no, how would you solve it?
> >
> > Thanks and kind regards, Christian
> >
>
>
>
> --
> -Sam
>
--
-Sam
Re: IdempotentRepository
Posted by "Sam (Stephen Samuel)" <sa...@gmail.com>.
eagerMaxMessagesPerPoll needs to be set to true.
On Fri, Jul 6, 2012 at 10:53 AM, Christian Lipp <Ch...@xion.at>wrote:
> Do you mean " eagerMaxMessagesPerPoll" ?
> I set it to false, but nothing changed.
>
> In the camel code I could only find
>
> contains in GenericFileConsumer
> Add, remove in GenericFileOnCompletion
>
> Regards, CL
>
> -----Ursprüngliche Nachricht-----
> Von: Sam (Stephen Samuel) [mailto:samspade79@gmail.com]
> Gesendet: Freitag, 06. Juli 2012 11:17
> An: users@camel.apache.org
> Betreff: Re: IdempotentRepository
>
> "This operation is used if the option eager has been enabled."
>
> Take a look at that, that might be why confirm is not being called.
>
> On Fri, Jul 6, 2012 at 7:17 AM, Christian Lipp <Christian.Lipp@xion.at
> >wrote:
>
> > Hello!
> >
> > I am fetching files from a FTP server (severall GB for the next years).
> > The files are produced daily in directories which correspond to the
> > date, like
> >
> > - 20120501
> > - 20120502
> > - ...
> >
> > I have only read rights and I am not the only consumer. This means
> > that they keep the last month or so on the server and I fetch on a daily
> base.
> > To avoid that I am fetching files twice I want to use an
> > IdempotentRepository implementation. I don't want to save each file in
> > a database or in a text file because the service will run for years
> > and this is just unnecessary data.
> >
> > What I want to store is the last processed date only. This handles
> > just the directories and would mean that I need some other strategy
> > for the files. I could combine this approach with the default in
> > memory based store. But let just stick to the directories:
> >
> > I read the directory sorted by file name. The IdempotentRepository is
> > called by the FtpConsumer with
> >
> > - start()
> > - contains() for every directory and file
> > - add() for files only
> >
> > and that's it. No stop(), no confirm(). When I have errors, sometimes
> > remove() is called. Since the repository is called only with String
> > (the full path) I have not information if I deal with directories or
> > files. I know it from the structure, but I am not able to implement a
> > generic solution.
> >
> > Anyway the idea is:
> >
> > - Store the LastProcessedDate inside the repository
> > - contains(): if the path contains an already processed date
> > (<LastProcessedDate) then I skip it (return true) otherwise return false.
> > - add(): if add() jumps to the next directory I set the
> > LastProcessedDate to the directory before
> >
> > The only problem is the last processed directory: even if it is
> > finished I do not get the chance to mark it as processed (set
> > LastProcessedDate to its value).
> >
> > So finally my questions: do you think this approach makes sense and if
> > yes: how would you deal with the last processed directory?
> > If no, how would you solve it?
> >
> > Thanks and kind regards, Christian
> >
>
>
>
> --
> -Sam
>
--
-Sam
AW: IdempotentRepository
Posted by Christian Lipp <Ch...@xion.at>.
Do you mean " eagerMaxMessagesPerPoll" ?
I set it to false, but nothing changed.
In the camel code I could only find
contains in GenericFileConsumer
Add, remove in GenericFileOnCompletion
Regards, CL
-----Ursprüngliche Nachricht-----
Von: Sam (Stephen Samuel) [mailto:samspade79@gmail.com]
Gesendet: Freitag, 06. Juli 2012 11:17
An: users@camel.apache.org
Betreff: Re: IdempotentRepository
"This operation is used if the option eager has been enabled."
Take a look at that, that might be why confirm is not being called.
On Fri, Jul 6, 2012 at 7:17 AM, Christian Lipp <Ch...@xion.at>wrote:
> Hello!
>
> I am fetching files from a FTP server (severall GB for the next years).
> The files are produced daily in directories which correspond to the
> date, like
>
> - 20120501
> - 20120502
> - ...
>
> I have only read rights and I am not the only consumer. This means
> that they keep the last month or so on the server and I fetch on a daily base.
> To avoid that I am fetching files twice I want to use an
> IdempotentRepository implementation. I don't want to save each file in
> a database or in a text file because the service will run for years
> and this is just unnecessary data.
>
> What I want to store is the last processed date only. This handles
> just the directories and would mean that I need some other strategy
> for the files. I could combine this approach with the default in
> memory based store. But let just stick to the directories:
>
> I read the directory sorted by file name. The IdempotentRepository is
> called by the FtpConsumer with
>
> - start()
> - contains() for every directory and file
> - add() for files only
>
> and that's it. No stop(), no confirm(). When I have errors, sometimes
> remove() is called. Since the repository is called only with String
> (the full path) I have not information if I deal with directories or
> files. I know it from the structure, but I am not able to implement a
> generic solution.
>
> Anyway the idea is:
>
> - Store the LastProcessedDate inside the repository
> - contains(): if the path contains an already processed date
> (<LastProcessedDate) then I skip it (return true) otherwise return false.
> - add(): if add() jumps to the next directory I set the
> LastProcessedDate to the directory before
>
> The only problem is the last processed directory: even if it is
> finished I do not get the chance to mark it as processed (set
> LastProcessedDate to its value).
>
> So finally my questions: do you think this approach makes sense and if
> yes: how would you deal with the last processed directory?
> If no, how would you solve it?
>
> Thanks and kind regards, Christian
>
--
-Sam