You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by Onder SEZGIN <on...@gmail.com> on 2021/08/05 15:22:46 UTC
Re: ondemand file processing with pollEnrich not grabbing files from
a directory
Hi,
My question should have been placed in users list in the first place sorry
for the confusion caused in dev list possibly.
Anyway i was able to get over such issue with Consumer/Producer templates.
I would expect pollEnrich would the same though but it did not.
i will check that later.
Hope it helps.
Thanks
On Thu, Aug 5, 2021 at 12:48 PM Onder SEZGIN <on...@gmail.com> wrote:
> extra info:
>
> it keeps logging No files found.. skipping to send to S3
> which possibly means that file is not getting picked up.
> and
> camel version is 3.7.5
>
> thanks
>
> On Thu, Aug 5, 2021 at 12:17 PM Onder SEZGIN <on...@gmail.com>
> wrote:
>
>> Hello,
>>
>> After a while i am taking a ride with newer versions of camel, and i have
>> a usecase to create a restful endpoint which will trigger uploading a file
>> from a preconfigured directory to s3 bucket.
>>
>> so below if we endpoint is called and there is no file in the directory
>> it logs nothing to upload but later if i place a file under the directory
>> and make an api call to upload endpoint again, it does not pick up the file?
>>
>> Any idea is appreciated, i kind of doubt about default aggregation part
>> of pollEnrich pattern but i am not sure.
>>
>> Thanks
>> Onder
>>
>> rest()
>> .path("/v1/upload")
>> .consumes("application/json")
>> .produces("application/text")
>> .post()
>> .type(Void.class)
>> .to("direct:uploadFiles");
>>
>> from("direct:uploadFiles")
>> .pollEnrich("file:{{shared-folder}}?include=.*.gz&delete=true&sendEmptyMessageWhenIdle=true")
>> .choice()
>> .when(body().isNull()).log("No files found.. skipping to send to S3")
>> .otherwise().to("direct:toAwsS3")
>> .endChoice();
>>
>>
Re: ondemand file processing with pollEnrich not grabbing files from
a directory
Posted by Claus Ibsen <cl...@gmail.com>.
consumer template and poll enrich is ONE exchange at a time. So if
there are 100 files in a folder, then only 1 of those files is picked
up.
There is only caching in regards to idempotent that it has built in
the file component, but its for files that it has previously picked
up.
You can turn that off or change the cache size etc.
You should likely use sendEmptyMessageWhenIdle=false, and then use a
timeout value instead if there are no files.
On Fri, Aug 6, 2021 at 12:00 PM Onder SEZGIN <on...@gmail.com> wrote:
>
> hello, just to make you aware about my progress on the same matter.
> I was able to get over the issue but the way i was able to sort it out was
> by passing specific file to consumerTemplate / pollEnrich.
> Otherwise, when pollEnrich or consumerTemplate is used with file expression
> as part of `include` param of file endpoint, i think consumerTemplate
> applies caching on endpoint level and after finding no files under a
> directory for a given expression to pick up files are not really picking up
> further files in the directory.
>
> My another expectation was that when pollEnrich or consumerTemplate would
> be able to pick up multiple files when a regexp is used as part of
> `include` param. however, both is picking up only one file..
>
>
> are these known stuff in regards to use of file component with pollEnrich
> and consumerTemplate, could they be bugs?
>
> Please advise.
>
> Thanks
> Onder
>
> On Thu, Aug 5, 2021 at 5:14 PM Onder SEZGIN <on...@gmail.com> wrote:
>
> > Noo, i think i was wrong in my previous email.
> >
> > exactly the same thing is happening in consumer/producer template way.
> >
> > it is not working as it was expected.
> >
> > On Thu, Aug 5, 2021 at 4:22 PM Onder SEZGIN <on...@gmail.com> wrote:
> >
> >> Hi,
> >>
> >> My question should have been placed in users list in the first place
> >> sorry for the confusion caused in dev list possibly.
> >> Anyway i was able to get over such issue with Consumer/Producer templates.
> >>
> >> I would expect pollEnrich would the same though but it did not.
> >> i will check that later.
> >>
> >> Hope it helps.
> >>
> >> Thanks
> >>
> >> On Thu, Aug 5, 2021 at 12:48 PM Onder SEZGIN <on...@gmail.com>
> >> wrote:
> >>
> >>> extra info:
> >>>
> >>> it keeps logging No files found.. skipping to send to S3
> >>> which possibly means that file is not getting picked up.
> >>> and
> >>> camel version is 3.7.5
> >>>
> >>> thanks
> >>>
> >>> On Thu, Aug 5, 2021 at 12:17 PM Onder SEZGIN <on...@gmail.com>
> >>> wrote:
> >>>
> >>>> Hello,
> >>>>
> >>>> After a while i am taking a ride with newer versions of camel, and i
> >>>> have a usecase to create a restful endpoint which will trigger uploading a
> >>>> file from a preconfigured directory to s3 bucket.
> >>>>
> >>>> so below if we endpoint is called and there is no file in the directory
> >>>> it logs nothing to upload but later if i place a file under the directory
> >>>> and make an api call to upload endpoint again, it does not pick up the file?
> >>>>
> >>>> Any idea is appreciated, i kind of doubt about default aggregation part
> >>>> of pollEnrich pattern but i am not sure.
> >>>>
> >>>> Thanks
> >>>> Onder
> >>>>
> >>>> rest()
> >>>> .path("/v1/upload")
> >>>> .consumes("application/json")
> >>>> .produces("application/text")
> >>>> .post()
> >>>> .type(Void.class)
> >>>> .to("direct:uploadFiles");
> >>>>
> >>>> from("direct:uploadFiles")
> >>>> .pollEnrich("file:{{shared-folder}}?include=.*.gz&delete=true&sendEmptyMessageWhenIdle=true")
> >>>> .choice()
> >>>> .when(body().isNull()).log("No files found.. skipping to send to S3")
> >>>> .otherwise().to("direct:toAwsS3")
> >>>> .endChoice();
> >>>>
> >>>>
--
Claus Ibsen
-----------------
http://davsclaus.com @davsclaus
Camel in Action 2: https://www.manning.com/ibsen2
Re: ondemand file processing with pollEnrich not grabbing files from
a directory
Posted by Onder SEZGIN <on...@gmail.com>.
hello, just to make you aware about my progress on the same matter.
I was able to get over the issue but the way i was able to sort it out was
by passing specific file to consumerTemplate / pollEnrich.
Otherwise, when pollEnrich or consumerTemplate is used with file expression
as part of `include` param of file endpoint, i think consumerTemplate
applies caching on endpoint level and after finding no files under a
directory for a given expression to pick up files are not really picking up
further files in the directory.
My another expectation was that when pollEnrich or consumerTemplate would
be able to pick up multiple files when a regexp is used as part of
`include` param. however, both is picking up only one file..
are these known stuff in regards to use of file component with pollEnrich
and consumerTemplate, could they be bugs?
Please advise.
Thanks
Onder
On Thu, Aug 5, 2021 at 5:14 PM Onder SEZGIN <on...@gmail.com> wrote:
> Noo, i think i was wrong in my previous email.
>
> exactly the same thing is happening in consumer/producer template way.
>
> it is not working as it was expected.
>
> On Thu, Aug 5, 2021 at 4:22 PM Onder SEZGIN <on...@gmail.com> wrote:
>
>> Hi,
>>
>> My question should have been placed in users list in the first place
>> sorry for the confusion caused in dev list possibly.
>> Anyway i was able to get over such issue with Consumer/Producer templates.
>>
>> I would expect pollEnrich would the same though but it did not.
>> i will check that later.
>>
>> Hope it helps.
>>
>> Thanks
>>
>> On Thu, Aug 5, 2021 at 12:48 PM Onder SEZGIN <on...@gmail.com>
>> wrote:
>>
>>> extra info:
>>>
>>> it keeps logging No files found.. skipping to send to S3
>>> which possibly means that file is not getting picked up.
>>> and
>>> camel version is 3.7.5
>>>
>>> thanks
>>>
>>> On Thu, Aug 5, 2021 at 12:17 PM Onder SEZGIN <on...@gmail.com>
>>> wrote:
>>>
>>>> Hello,
>>>>
>>>> After a while i am taking a ride with newer versions of camel, and i
>>>> have a usecase to create a restful endpoint which will trigger uploading a
>>>> file from a preconfigured directory to s3 bucket.
>>>>
>>>> so below if we endpoint is called and there is no file in the directory
>>>> it logs nothing to upload but later if i place a file under the directory
>>>> and make an api call to upload endpoint again, it does not pick up the file?
>>>>
>>>> Any idea is appreciated, i kind of doubt about default aggregation part
>>>> of pollEnrich pattern but i am not sure.
>>>>
>>>> Thanks
>>>> Onder
>>>>
>>>> rest()
>>>> .path("/v1/upload")
>>>> .consumes("application/json")
>>>> .produces("application/text")
>>>> .post()
>>>> .type(Void.class)
>>>> .to("direct:uploadFiles");
>>>>
>>>> from("direct:uploadFiles")
>>>> .pollEnrich("file:{{shared-folder}}?include=.*.gz&delete=true&sendEmptyMessageWhenIdle=true")
>>>> .choice()
>>>> .when(body().isNull()).log("No files found.. skipping to send to S3")
>>>> .otherwise().to("direct:toAwsS3")
>>>> .endChoice();
>>>>
>>>>
Re: ondemand file processing with pollEnrich not grabbing files from
a directory
Posted by Onder SEZGIN <on...@gmail.com>.
Noo, i think i was wrong in my previous email.
exactly the same thing is happening in consumer/producer template way.
it is not working as it was expected.
On Thu, Aug 5, 2021 at 4:22 PM Onder SEZGIN <on...@gmail.com> wrote:
> Hi,
>
> My question should have been placed in users list in the first place sorry
> for the confusion caused in dev list possibly.
> Anyway i was able to get over such issue with Consumer/Producer templates.
>
> I would expect pollEnrich would the same though but it did not.
> i will check that later.
>
> Hope it helps.
>
> Thanks
>
> On Thu, Aug 5, 2021 at 12:48 PM Onder SEZGIN <on...@gmail.com>
> wrote:
>
>> extra info:
>>
>> it keeps logging No files found.. skipping to send to S3
>> which possibly means that file is not getting picked up.
>> and
>> camel version is 3.7.5
>>
>> thanks
>>
>> On Thu, Aug 5, 2021 at 12:17 PM Onder SEZGIN <on...@gmail.com>
>> wrote:
>>
>>> Hello,
>>>
>>> After a while i am taking a ride with newer versions of camel, and i
>>> have a usecase to create a restful endpoint which will trigger uploading a
>>> file from a preconfigured directory to s3 bucket.
>>>
>>> so below if we endpoint is called and there is no file in the directory
>>> it logs nothing to upload but later if i place a file under the directory
>>> and make an api call to upload endpoint again, it does not pick up the file?
>>>
>>> Any idea is appreciated, i kind of doubt about default aggregation part
>>> of pollEnrich pattern but i am not sure.
>>>
>>> Thanks
>>> Onder
>>>
>>> rest()
>>> .path("/v1/upload")
>>> .consumes("application/json")
>>> .produces("application/text")
>>> .post()
>>> .type(Void.class)
>>> .to("direct:uploadFiles");
>>>
>>> from("direct:uploadFiles")
>>> .pollEnrich("file:{{shared-folder}}?include=.*.gz&delete=true&sendEmptyMessageWhenIdle=true")
>>> .choice()
>>> .when(body().isNull()).log("No files found.. skipping to send to S3")
>>> .otherwise().to("direct:toAwsS3")
>>> .endChoice();
>>>
>>>