You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by webber78 <ki...@gmail.com> on 2012/09/17 17:28:51 UTC
Custom Idempotent Jdbc repository + Using lastmodified time
Hi,
I have implemented a custom jdbc idempotent repository(extanding
AbstractJdbcMessageIdRepository) to track sftp file downloads. Based on the
example from documenation, the only file properties I am able to store in
the new table are the messageId (which is the filename). I would like the
updated files (with same name) to get downloaded and processed.
But it appears there is no way to get the last modified time using the
messageId alone. If such a thing was possible, i could make the key in the
table a combination of filename and timestamp and that would solve the
problem.
Please let me know if anyone has a solution/workaround for this problem.
Thanks.
--
View this message in context: http://camel.465427.n5.nabble.com/Custom-Idempotent-Jdbc-repository-Using-lastmodified-time-tp5719480.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Custom Idempotent Jdbc repository + Using lastmodified time
Posted by Claus Ibsen <cl...@gmail.com>.
On Tue, Sep 18, 2012 at 9:11 AM, Claus Ibsen <cl...@gmail.com> wrote:
> On Tue, Sep 18, 2012 at 7:25 AM, Siano, Stephan <st...@sap.com> wrote:
>> There are actually two potential usages for idempotent repositories in conjunction with (S)FTP endpoints: usage in the endpoint itself or with an idempotent consumer in the pipeline (as you describe). If you have the idempotent repository directly in the endpoint this has the advantage that it is applied before the file is even fetched from the server, however AFAIK the only available information for this is the filename. With idempotent consumers in the pipeline, you are free to do whatever you want, but this is after you have fetched the file from the server.
>>
>> So to answer the original question: AFAIK there is no way to put any exchange properties to an idempotent repository that is attached to the endpoint using the idempotentRepository parameter in the file/FTP/SFTP endpoint, because there is actually no exchange at the time it is used. If you are willing to download all files and then select which ones you want to process, the idempotent consumer Christian described is the way to go.
>>
>
> Yes the built-in idempotent consumer is based on the file name only.
> We could enhance this with more details if that would be possible.
>
> Fell free to log a JIRA ticket.
>
I logged a ticket
https://issues.apache.org/jira/browse/CAMEL-5639
>> Best regards
>> Stephan
>>
>> -----Original Message-----
>> From: Christian Müller [mailto:christian.mueller@gmail.com]
>> Sent: Montag, 17. September 2012 23:57
>> To: users@camel.apache.org
>> Subject: Re: Custom Idempotent Jdbc repository + Using lastmodified time
>>
>> Checkout: http://camel.apache.org/idempotent-consumer.html and
>> http://camel.apache.org/simple.html
>>
>> Instead of
>> from("direct:a")
>> .idempotentConsumer(header("myMessageId"), ...
>>
>> try something like
>> from("direct:a")
>>
>> .idempotentConsumer(simple("${header.CamelFileNameOnly}-${header.CamelFileLastModified}"),
>> ...
>>
>> Best,
>> Christian
>>
>> On Mon, Sep 17, 2012 at 11:34 PM, webber78 <ki...@gmail.com> wrote:
>>
>>> Thanks for your reply. Is there anyway I can get the header you described
>>> in
>>> a subclass of AbstractJdbcMessageIdRepository. I saw a few examples of
>>> getting headers using the Exchange object, but not sure how to get access
>>> to
>>> these from an idempotent repository.
>>>
>>> Thanks,
>>> Kiran
>>>
>>>
>>>
>>> --
>>> View this message in context:
>>> http://camel.465427.n5.nabble.com/Custom-Idempotent-Jdbc-repository-Using-lastmodified-time-tp5719480p5719495.html
>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>
>>
>>
>>
>> --
>
>
>
> --
> Claus Ibsen
> -----------------
> FuseSource
> Email: cibsen@fusesource.com
> Web: http://fusesource.com
> Twitter: davsclaus, fusenews
> Blog: http://davsclaus.com
> Author of Camel in Action: http://www.manning.com/ibsen
--
Claus Ibsen
-----------------
Red Hat, Inc.
FuseSource is now part of Red Hat
Email: cibsen@redhat.com
Web: http://fusesource.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
Re: Custom Idempotent Jdbc repository + Using lastmodified time
Posted by Claus Ibsen <cl...@gmail.com>.
On Tue, Sep 18, 2012 at 7:25 AM, Siano, Stephan <st...@sap.com> wrote:
> There are actually two potential usages for idempotent repositories in conjunction with (S)FTP endpoints: usage in the endpoint itself or with an idempotent consumer in the pipeline (as you describe). If you have the idempotent repository directly in the endpoint this has the advantage that it is applied before the file is even fetched from the server, however AFAIK the only available information for this is the filename. With idempotent consumers in the pipeline, you are free to do whatever you want, but this is after you have fetched the file from the server.
>
> So to answer the original question: AFAIK there is no way to put any exchange properties to an idempotent repository that is attached to the endpoint using the idempotentRepository parameter in the file/FTP/SFTP endpoint, because there is actually no exchange at the time it is used. If you are willing to download all files and then select which ones you want to process, the idempotent consumer Christian described is the way to go.
>
Yes the built-in idempotent consumer is based on the file name only.
We could enhance this with more details if that would be possible.
Fell free to log a JIRA ticket.
> Best regards
> Stephan
>
> -----Original Message-----
> From: Christian Müller [mailto:christian.mueller@gmail.com]
> Sent: Montag, 17. September 2012 23:57
> To: users@camel.apache.org
> Subject: Re: Custom Idempotent Jdbc repository + Using lastmodified time
>
> Checkout: http://camel.apache.org/idempotent-consumer.html and
> http://camel.apache.org/simple.html
>
> Instead of
> from("direct:a")
> .idempotentConsumer(header("myMessageId"), ...
>
> try something like
> from("direct:a")
>
> .idempotentConsumer(simple("${header.CamelFileNameOnly}-${header.CamelFileLastModified}"),
> ...
>
> Best,
> Christian
>
> On Mon, Sep 17, 2012 at 11:34 PM, webber78 <ki...@gmail.com> wrote:
>
>> Thanks for your reply. Is there anyway I can get the header you described
>> in
>> a subclass of AbstractJdbcMessageIdRepository. I saw a few examples of
>> getting headers using the Exchange object, but not sure how to get access
>> to
>> these from an idempotent repository.
>>
>> Thanks,
>> Kiran
>>
>>
>>
>> --
>> View this message in context:
>> http://camel.465427.n5.nabble.com/Custom-Idempotent-Jdbc-repository-Using-lastmodified-time-tp5719480p5719495.html
>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>
>
>
>
> --
--
Claus Ibsen
-----------------
FuseSource
Email: cibsen@fusesource.com
Web: http://fusesource.com
Twitter: davsclaus, fusenews
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
RE: Custom Idempotent Jdbc repository + Using lastmodified time
Posted by "Siano, Stephan" <st...@sap.com>.
There are actually two potential usages for idempotent repositories in conjunction with (S)FTP endpoints: usage in the endpoint itself or with an idempotent consumer in the pipeline (as you describe). If you have the idempotent repository directly in the endpoint this has the advantage that it is applied before the file is even fetched from the server, however AFAIK the only available information for this is the filename. With idempotent consumers in the pipeline, you are free to do whatever you want, but this is after you have fetched the file from the server.
So to answer the original question: AFAIK there is no way to put any exchange properties to an idempotent repository that is attached to the endpoint using the idempotentRepository parameter in the file/FTP/SFTP endpoint, because there is actually no exchange at the time it is used. If you are willing to download all files and then select which ones you want to process, the idempotent consumer Christian described is the way to go.
Best regards
Stephan
-----Original Message-----
From: Christian Müller [mailto:christian.mueller@gmail.com]
Sent: Montag, 17. September 2012 23:57
To: users@camel.apache.org
Subject: Re: Custom Idempotent Jdbc repository + Using lastmodified time
Checkout: http://camel.apache.org/idempotent-consumer.html and
http://camel.apache.org/simple.html
Instead of
from("direct:a")
.idempotentConsumer(header("myMessageId"), ...
try something like
from("direct:a")
.idempotentConsumer(simple("${header.CamelFileNameOnly}-${header.CamelFileLastModified}"),
...
Best,
Christian
On Mon, Sep 17, 2012 at 11:34 PM, webber78 <ki...@gmail.com> wrote:
> Thanks for your reply. Is there anyway I can get the header you described
> in
> a subclass of AbstractJdbcMessageIdRepository. I saw a few examples of
> getting headers using the Exchange object, but not sure how to get access
> to
> these from an idempotent repository.
>
> Thanks,
> Kiran
>
>
>
> --
> View this message in context:
> http://camel.465427.n5.nabble.com/Custom-Idempotent-Jdbc-repository-Using-lastmodified-time-tp5719480p5719495.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
--
Re: Custom Idempotent Jdbc repository + Using lastmodified time
Posted by Christian Müller <ch...@gmail.com>.
Checkout: http://camel.apache.org/idempotent-consumer.html and
http://camel.apache.org/simple.html
Instead of
from("direct:a")
.idempotentConsumer(header("myMessageId"), ...
try something like
from("direct:a")
.idempotentConsumer(simple("${header.CamelFileNameOnly}-${header.CamelFileLastModified}"),
...
Best,
Christian
On Mon, Sep 17, 2012 at 11:34 PM, webber78 <ki...@gmail.com> wrote:
> Thanks for your reply. Is there anyway I can get the header you described
> in
> a subclass of AbstractJdbcMessageIdRepository. I saw a few examples of
> getting headers using the Exchange object, but not sure how to get access
> to
> these from an idempotent repository.
>
> Thanks,
> Kiran
>
>
>
> --
> View this message in context:
> http://camel.465427.n5.nabble.com/Custom-Idempotent-Jdbc-repository-Using-lastmodified-time-tp5719480p5719495.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
--
Re: Custom Idempotent Jdbc repository + Using lastmodified time
Posted by webber78 <ki...@gmail.com>.
Thanks for your reply. Is there anyway I can get the header you described in
a subclass of AbstractJdbcMessageIdRepository. I saw a few examples of
getting headers using the Exchange object, but not sure how to get access to
these from an idempotent repository.
Thanks,
Kiran
--
View this message in context: http://camel.465427.n5.nabble.com/Custom-Idempotent-Jdbc-repository-Using-lastmodified-time-tp5719480p5719495.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Custom Idempotent Jdbc repository + Using lastmodified time
Posted by Christian Müller <ch...@gmail.com>.
The FTP consumer (which extends the file consumer) should set the header
"CamelFileLastModified". Isn't it?
You can read more about it at [1] and [2].
[1] http://camel.apache.org/file2.html
[2] http://camel.apache.org/ftp2.html
Best,
Christian
On Mon, Sep 17, 2012 at 5:28 PM, webber78 <ki...@gmail.com> wrote:
> Hi,
>
> I have implemented a custom jdbc idempotent repository(extanding
> AbstractJdbcMessageIdRepository) to track sftp file downloads. Based on the
> example from documenation, the only file properties I am able to store in
> the new table are the messageId (which is the filename). I would like the
> updated files (with same name) to get downloaded and processed.
>
> But it appears there is no way to get the last modified time using the
> messageId alone. If such a thing was possible, i could make the key in the
> table a combination of filename and timestamp and that would solve the
> problem.
>
> Please let me know if anyone has a solution/workaround for this problem.
>
> Thanks.
>
>
>
> --
> View this message in context:
> http://camel.465427.n5.nabble.com/Custom-Idempotent-Jdbc-repository-Using-lastmodified-time-tp5719480.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
--