You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by bwest <bw...@gmail.com> on 2016/05/31 22:37:35 UTC
File2 Consumer - Multiple file problem
Greetings,
I have the following question, I will try to lay the scenario as clearly as
possible. If this isn't clear, my apologies up front.
I have a directory, that will have multiple files dropped into it. The file
are text files and as part of this generation process they will have a guid
generated and place as part of the filename. Additionally as part of these
.txt files, a corresponding "control file" will also be dropped.
This control file is what I want to drive my camel consumer process with.
Below is conceptually what i'm thinking about.
*For example the directory could look like below:*
file-a-123abc456.txt
file-a-123abc456.txt
file-a-123abc456.xml
What I'd like to do is configure the file consumer to be triggered by the
.xml control file. Once the consumer file process is triggered, I'd then
like to somehow read all the corresponding payload files (.txts) and insert
them onto the exchange.
So with the above being said, a couple of questions"
1) Can I collect the multiple .txts and insert them "manually" on the
exchange? Mimicking the camel file consumer process?
2) If number one is possible, I'd like to then pass the exchange onto an
FTP producer. Is this the most efficient way to ftp multiple files?
I'm a bit new to Camel and if these are incorrect assumptions, please
forgive. I'm open to any and all suggestions as to the best way of handling
the above scenario.
Thanks,
--
View this message in context: http://camel.465427.n5.nabble.com/File2-Consumer-Multiple-file-problem-tp5783329.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: File2 Consumer - Multiple file problem
Posted by Ranx <br...@mediadriver.com>.
That depends on a lot of other processing issues. Are you actually reading
the contents of the text files or just ftp'ing them to a remote location?
Do you have any control over the remote location? The reason I ask is you
could have two routes set up. The first might look for *.txt files and FTP
them to a temporary directory on the remote site. The second route would
look for the *.xml and then FTP that to the temporary remote directory and
then do an FTP rename to move those from the temporary remote directory to
their final location.
But that depends on a lot of things like how may files are you talking about
is it thousands of files per hour or just a few every night? Once the files
have been streamed to the remote temporary directory the rename function
will move them on the remote drive for you in an atomic fashion.
But that's a lot of guessing on my part without knowing exactly what you're
trying to accomplish.
--
View this message in context: http://camel.465427.n5.nabble.com/File2-Consumer-Multiple-file-problem-tp5783329p5783373.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: File2 Consumer - Multiple file problem
Posted by bwest <bw...@gmail.com>.
Thanks for the reply! Although I do have one question. The .done file
sounds like a good way to trigger a batch. However, if I have multiple
processes writing files and .done files into my polled directory. Does
camel lock the .done file so that another process cannot come in and
overwrite it? If not, is there a way in the camel to lock the .done file
until it's fully processed and deleted?
On Wed, Jun 1, 2016 at 12:38 AM, souciance [via Camel] <
ml-node+s465427n5783334h59@n5.nabble.com> wrote:
> 1. You can collect the multiple txt files by placing a pattern i.e.
> (*.txt) in your file consumer URI. But first you need to configure a .done
> file in your file consumer route. Basically the .done means that only when
> the xml file is generate should you consume the txt files.
> 2. I am guessing the txt files will have different names? The main
> problem is to write a good aggregation strategy if you want to combine all
> the payload and then send to the FTP or decide it is ok to send them one by
> one.
>
> On Wed, Jun 1, 2016 at 12:37 AM, bwest [via Camel] <[hidden email]
> <http:///user/SendEmail.jtp?type=node&node=5783334&i=0>> wrote:
>
>> Greetings,
>>
>> I have the following question, I will try to lay the scenario as clearly
>> as possible. If this isn't clear, my apologies up front.
>>
>> I have a directory, that will have multiple files dropped into it. The
>> file are text files and as part of this generation process they will have a
>> guid generated and place as part of the filename. Additionally as part of
>> these .txt files, a corresponding "control file" will also be dropped.
>>
>> This control file is what I want to drive my camel consumer process
>> with. Below is conceptually what i'm thinking about.
>>
>> *For example the directory could look like below:*
>> file-a-123abc456.txt
>> file-a-123abc456.txt
>> file-a-123abc456.xml
>>
>> What I'd like to do is configure the file consumer to be triggered by the
>> .xml control file. Once the consumer file process is triggered, I'd then
>> like to somehow read all the corresponding payload files (.txts) and insert
>> them onto the exchange.
>>
>> So with the above being said, a couple of questions"
>>
>> 1) Can I collect the multiple .txts and insert them "manually" on the
>> exchange? Mimicking the camel file consumer process?
>>
>> 2) If number one is possible, I'd like to then pass the exchange onto an
>> FTP producer. Is this the most efficient way to ftp multiple files?
>>
>> I'm a bit new to Camel and if these are incorrect assumptions, please
>> forgive. I'm open to any and all suggestions as to the best way of
>> handling the above scenario.
>>
>>
>>
>> Thanks,
>>
>> ------------------------------
>> If you reply to this email, your message will be added to the discussion
>> below:
>>
>> http://camel.465427.n5.nabble.com/File2-Consumer-Multiple-file-problem-tp5783329.html
>> To start a new topic under Camel - Users, email [hidden email]
>> <http:///user/SendEmail.jtp?type=node&node=5783334&i=1>
>> To unsubscribe from Camel - Users, click here.
>> NAML
>> <http://camel.465427.n5.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>
>
>
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
>
> http://camel.465427.n5.nabble.com/File2-Consumer-Multiple-file-problem-tp5783329p5783334.html
> To unsubscribe from File2 Consumer - Multiple file problem, click here
> <http://camel.465427.n5.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=5783329&code=Yndlc3RlcmZpZWxkQGdtYWlsLmNvbXw1NzgzMzI5fDEwNTI2NDM2MzE=>
> .
> NAML
> <http://camel.465427.n5.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
--
View this message in context: http://camel.465427.n5.nabble.com/File2-Consumer-Multiple-file-problem-tp5783329p5783369.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: File2 Consumer - Multiple file problem
Posted by souciance <so...@gmail.com>.
1. You can collect the multiple txt files by placing a pattern i.e. (*.txt)
in your file consumer URI. But first you need to configure a .done file in
your file consumer route. Basically the .done means that only when the xml
file is generate should you consume the txt files.
2. I am guessing the txt files will have different names? The main problem
is to write a good aggregation strategy if you want to combine all the
payload and then send to the FTP or decide it is ok to send them one by one.
On Wed, Jun 1, 2016 at 12:37 AM, bwest [via Camel] <
ml-node+s465427n5783329h62@n5.nabble.com> wrote:
> Greetings,
>
> I have the following question, I will try to lay the scenario as clearly
> as possible. If this isn't clear, my apologies up front.
>
> I have a directory, that will have multiple files dropped into it. The
> file are text files and as part of this generation process they will have a
> guid generated and place as part of the filename. Additionally as part of
> these .txt files, a corresponding "control file" will also be dropped.
>
> This control file is what I want to drive my camel consumer process with.
> Below is conceptually what i'm thinking about.
>
> *For example the directory could look like below:*
> file-a-123abc456.txt
> file-a-123abc456.txt
> file-a-123abc456.xml
>
> What I'd like to do is configure the file consumer to be triggered by the
> .xml control file. Once the consumer file process is triggered, I'd then
> like to somehow read all the corresponding payload files (.txts) and insert
> them onto the exchange.
>
> So with the above being said, a couple of questions"
>
> 1) Can I collect the multiple .txts and insert them "manually" on the
> exchange? Mimicking the camel file consumer process?
>
> 2) If number one is possible, I'd like to then pass the exchange onto an
> FTP producer. Is this the most efficient way to ftp multiple files?
>
> I'm a bit new to Camel and if these are incorrect assumptions, please
> forgive. I'm open to any and all suggestions as to the best way of
> handling the above scenario.
>
>
>
> Thanks,
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
>
> http://camel.465427.n5.nabble.com/File2-Consumer-Multiple-file-problem-tp5783329.html
> To start a new topic under Camel - Users, email
> ml-node+s465427n465428h31@n5.nabble.com
> To unsubscribe from Camel - Users, click here
> <http://camel.465427.n5.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=465428&code=c291Y2lhbmNlLmVxZGFtLnJhc2h0aUBnbWFpbC5jb218NDY1NDI4fDE1MzI5MTE2NTY=>
> .
> NAML
> <http://camel.465427.n5.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
--
View this message in context: http://camel.465427.n5.nabble.com/File2-Consumer-Multiple-file-problem-tp5783329p5783334.html
Sent from the Camel - Users mailing list archive at Nabble.com.