You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by vishal1981 <vi...@ericsson.com> on 2012/07/16 08:29:34 UTC
Preventing reading of a large XML file in memory
Hi,
I am using Camel to transfer a file from my app server to an FTP server
before converting it to a different format. The incoming file can be
potentially huge (60 to 100 gbs) in size and I want to prevent Camel from
reading the whole file in memory. I would like to call a bean which takes
the input file location and an output file location as a parameter and not
the message body.
Any pointers on how I can achieve this without reading the whole file in
memory?
Thanks in advance.
--
View this message in context: http://camel.465427.n5.nabble.com/Preventing-reading-of-a-large-XML-file-in-memory-tp5716064.html
Sent from the Camel - Users mailing list archive at Nabble.com.
RE: Preventing reading of a large XML file in memory
Posted by vishal1981 <vi...@ericsson.com>.
Hi,
Sorry to bug you again...but..
Now I have my route configuration as File -> MyBean ->Ftp
Now MyBean converts the input file to a different format. I want to avoid doing a exchange.getOut().setBody(<read new file>) since the new file is huge. But somehow I want to convey to the FTP producer that FTP over the new file and not the one via message.getIn().getBody().
I tried by just setting the message header .setHeader(Exchange.FILE_NAME, "new file path") but I think the FTP producer just does the getIn().getBody() irrespective of the file name.
Is there a way to do this?
Thanks for your help.
________________________________
From: Claus Ibsen-2 [via Camel] [mailto:ml-node+s465427n5716115h91@n5.nabble.com]
Sent: Monday, July 16, 2012 2:54 PM
To: Vishal Changrani
Subject: Re: Preventing reading of a large XML file in memory
Sorry it was the ftp endpoint i meant
On Monday, July 16, 2012, vishal1981 wrote:
> Thanks for the reply.
> So is it compulsory to use a JMS Endpoint to prevent Camel from reading the
> whole message in memory?
> What is to my bean I dont pass the message body as a parameter but just
> then
> filename? Will that prevent Camel from loading the whole message in memory
> before calling my bean method?
> i.e. from(ftp.....).to(bean(MyBean.class,
> "doSomething(${header.CamelFileName})")
> (am using Camel 2.9)
>
>
> --
> View this message in context:
> http://camel.465427.n5.nabble.com/Preventing-reading-of-a-large-XML-file-in-memory-tp5716064p5716083.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
--
Claus Ibsen
-----------------
FuseSource
Email: [hidden email]</user/SendEmail.jtp?type=node&node=5716115&i=0>
Web: http://fusesource.com
Twitter: davsclaus, fusenews
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
________________________________
If you reply to this email, your message will be added to the discussion below:
http://camel.465427.n5.nabble.com/Preventing-reading-of-a-large-XML-file-in-memory-tp5716064p5716115.html
To unsubscribe from Preventing reading of a large XML file in memory, click here<http://camel.465427.n5.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=5716064&code=dmlzaGFsLmNoYW5ncmFuaUBlcmljc3Nvbi5jb218NTcxNjA2NHwxNTgwMjU0NDcx>.
NAML<http://camel.465427.n5.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
--
View this message in context: http://camel.465427.n5.nabble.com/Preventing-reading-of-a-large-XML-file-in-memory-tp5716064p5716205.html
Sent from the Camel - Users mailing list archive at Nabble.com.
RE: Preventing reading of a large XML file in memory
Posted by vishal1981 <vi...@ericsson.com>.
Hi,
Actually Camel already has support for what I was looking for. All I did was set the body of the message in my bean to a File object...when I pass this message down to a file endpoint it knows its a file handle and automatically reads from it. I am not sure which endpoints understand file object as message body and which dont but I suppose File and FTP atleast do. (by understand mean they realize that they are suppose to read from the File object and that the message is not just a string object).
thanks
________________________________
From: Willem.Jiang [via Camel] [mailto:ml-node+s465427n5716226h32@n5.nabble.com]
Sent: Wednesday, July 18, 2012 10:10 PM
To: Vishal Changrani
Subject: Re: Preventing reading of a large XML file in memory
Hi,
How did you process the old file?
Maybe you need some file cache to avoid loading the whole XML file into
the memory.
On 7/18/12 10:52 PM, vishal1981 wrote:
> Hi,
> Sorry to bug you again...but..
> Now I have my route configuration as File -> MyBean ->Ftp
> Now MyBean converts the input file to a different format. I want to avoid
> doing a exchange.getOut().setBody(<read new file>) since the new file is
> huge. But somehow I want to convey to the FTP producer that FTP over the new
> file and not the one via message.getIn().getBody().
>
> I tried by just setting the message header .setHeader(Exchange.FILE_NAME,
> "new file path") but I think the FTP producer just does the
> getIn().getBody() irrespective of the file name.
>
> Is there a way to do this?
>
> Thanks for your help.
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Preventing-reading-of-a-large-XML-file-in-memory-tp5716064p5716204.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
> .
>
--
Willem
----------------------------------
FuseSource
Web: http://www.fusesource.com
Blog: http://willemjiang.blogspot.com (English)
http://jnn.javaeye.com (Chinese)
Twitter: willemjiang
Weibo: willemjiang
________________________________
If you reply to this email, your message will be added to the discussion below:
http://camel.465427.n5.nabble.com/Preventing-reading-of-a-large-XML-file-in-memory-tp5716064p5716226.html
To unsubscribe from Preventing reading of a large XML file in memory, click here<http://camel.465427.n5.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=5716064&code=dmlzaGFsLmNoYW5ncmFuaUBlcmljc3Nvbi5jb218NTcxNjA2NHwxNTgwMjU0NDcx>.
NAML<http://camel.465427.n5.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
--
View this message in context: http://camel.465427.n5.nabble.com/Preventing-reading-of-a-large-XML-file-in-memory-tp5716064p5716249.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Preventing reading of a large XML file in memory
Posted by Willem Jiang <wi...@gmail.com>.
Hi,
How did you process the old file?
Maybe you need some file cache to avoid loading the whole XML file into
the memory.
On 7/18/12 10:52 PM, vishal1981 wrote:
> Hi,
> Sorry to bug you again...but..
> Now I have my route configuration as File -> MyBean ->Ftp
> Now MyBean converts the input file to a different format. I want to avoid
> doing a exchange.getOut().setBody(<read new file>) since the new file is
> huge. But somehow I want to convey to the FTP producer that FTP over the new
> file and not the one via message.getIn().getBody().
>
> I tried by just setting the message header .setHeader(Exchange.FILE_NAME,
> "new file path") but I think the FTP producer just does the
> getIn().getBody() irrespective of the file name.
>
> Is there a way to do this?
>
> Thanks for your help.
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Preventing-reading-of-a-large-XML-file-in-memory-tp5716064p5716204.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
> .
>
--
Willem
----------------------------------
FuseSource
Web: http://www.fusesource.com
Blog: http://willemjiang.blogspot.com (English)
http://jnn.javaeye.com (Chinese)
Twitter: willemjiang
Weibo: willemjiang
RE: Preventing reading of a large XML file in memory
Posted by vishal1981 <vi...@ericsson.com>.
Hi,
Sorry to bug you again...but..
Now I have my route configuration as File -> MyBean ->Ftp
Now MyBean converts the input file to a different format. I want to avoid
doing a exchange.getOut().setBody(<read new file>) since the new file is
huge. But somehow I want to convey to the FTP producer that FTP over the new
file and not the one via message.getIn().getBody().
I tried by just setting the message header .setHeader(Exchange.FILE_NAME,
"new file path") but I think the FTP producer just does the
getIn().getBody() irrespective of the file name.
Is there a way to do this?
Thanks for your help.
--
View this message in context: http://camel.465427.n5.nabble.com/Preventing-reading-of-a-large-XML-file-in-memory-tp5716064p5716204.html
Sent from the Camel - Users mailing list archive at Nabble.com.
RE: Preventing reading of a large XML file in memory
Posted by vishal1981 <vi...@ericsson.com>.
sweet..thanks a ton!
________________________________
From: Claus Ibsen-2 [via Camel] [mailto:ml-node+s465427n5716143h59@n5.nabble.com]
Sent: Tuesday, July 17, 2012 1:10 AM
To: Vishal Changrani
Subject: Re: Preventing reading of a large XML file in memory
On Tue, Jul 17, 2012 at 6:47 AM, vishal1981
<[hidden email]</user/SendEmail.jtp?type=node&node=5716143&i=0>> wrote:
> Thanks again.
> I am using something like the following,
>
> from(file.......).to(ejb:cms/CamelToPluginAdaptor/local?method=process(${header.CamelFileAbsolutePath},${properties:outbox}/${header.PROVIDER_ID}.xml,${header.PROVIDER_ID})
>
> Where cms/CamelToPluginAdaptor /local is my EJB.
>
> Will that prevent reading the whole file in memory?
>
Yes the parameters for the method call, does not require to content of
the file, so Camel will not load it.
>
>
> ________________________________
> From: Claus Ibsen-2 [via Camel] [mailto:[hidden email]</user/SendEmail.jtp?type=node&node=5716143&i=1>]
> Sent: Tuesday, July 17, 2012 12:43 AM
> To: Vishal Changrani
> Subject: Re: Preventing reading of a large XML file in memory
>
> On Tue, Jul 17, 2012 at 6:18 AM, vishal1981
> <[hidden email]</user/SendEmail.jtp?type=node&node=5716140&i=0>> wrote:
>> Thanks. I read about the working directory and thats exactly what I was looking for. thank you.
>> Yet to try it though.
>> However, what if I want the same behaviour for File. i.e. File ->Bean (no FTP). I dont want the file component to read the whole file just pass along the file path to the bean?
>> Is there something similiar for File component?
>>
>
> Yes as the file component only stores the handle to the file. And only
> read it on demand.
> So in your bean, just pass in java.io.File, or a FileInputStream,
> InputStream etc.
>
>>
>> ________________________________
>> From: Claus Ibsen-2 [via Camel] [mailto:[hidden email]</user/SendEmail.jtp?type=node&node=5716140&i=1>]
>> Sent: Monday, July 16, 2012 2:54 PM
>> To: Vishal Changrani
>> Subject: Re: Preventing reading of a large XML file in memory
>>
>> Sorry it was the ftp endpoint i meant
>>
>> On Monday, July 16, 2012, vishal1981 wrote:
>>
>>> Thanks for the reply.
>>> So is it compulsory to use a JMS Endpoint to prevent Camel from reading the
>>> whole message in memory?
>>> What is to my bean I dont pass the message body as a parameter but just
>>> then
>>> filename? Will that prevent Camel from loading the whole message in memory
>>> before calling my bean method?
>>> i.e. from(ftp.....).to(bean(MyBean.class,
>>> "doSomething(${header.CamelFileName})")
>>> (am using Camel 2.9)
>>>
>>>
>>> --
>>> View this message in context:
>>> http://camel.465427.n5.nabble.com/Preventing-reading-of-a-large-XML-file-in-memory-tp5716064p5716083.html
>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>
>>
>>
>> --
>> Claus Ibsen
>> -----------------
>> FuseSource
>> Email: [hidden email]</user/SendEmail.jtp?type=node&node=5716115&i=0>
>> Web: http://fusesource.com
>> Twitter: davsclaus, fusenews
>> Blog: http://davsclaus.com
>> Author of Camel in Action: http://www.manning.com/ibsen
>>
>>
>> ________________________________
>> If you reply to this email, your message will be added to the discussion below:
>>
>> NAML<http://camel.465427.n5.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>
>>
>> --
>> View this message in context: http://camel.465427.n5.nabble.com/Preventing-reading-of-a-large-XML-file-in-memory-tp5716064p5716139.html
>> Sent from the Camel - Users mailing list archive at Nabble.com.
>
>
>
> --
> Claus Ibsen
> -----------------
> FuseSource
> Email: [hidden email]</user/SendEmail.jtp?type=node&node=5716140&i=2>
> Web: http://fusesource.com
> Twitter: davsclaus, fusenews
> Blog: http://davsclaus.com
> Author of Camel in Action: http://www.manning.com/ibsen
>
>
> ________________________________
> If you reply to this email, your message will be added to the discussion below:
>
> NAML<http://camel.465427.n5.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Preventing-reading-of-a-large-XML-file-in-memory-tp5716064p5716141.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
--
Claus Ibsen
-----------------
FuseSource
Email: [hidden email]</user/SendEmail.jtp?type=node&node=5716143&i=2>
Web: http://fusesource.com
Twitter: davsclaus, fusenews
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
________________________________
If you reply to this email, your message will be added to the discussion below:
http://camel.465427.n5.nabble.com/Preventing-reading-of-a-large-XML-file-in-memory-tp5716064p5716143.html
To unsubscribe from Preventing reading of a large XML file in memory, click here<http://camel.465427.n5.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=5716064&code=dmlzaGFsLmNoYW5ncmFuaUBlcmljc3Nvbi5jb218NTcxNjA2NHwxNTgwMjU0NDcx>.
NAML<http://camel.465427.n5.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
--
View this message in context: http://camel.465427.n5.nabble.com/Preventing-reading-of-a-large-XML-file-in-memory-tp5716064p5716144.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Preventing reading of a large XML file in memory
Posted by Claus Ibsen <cl...@gmail.com>.
On Tue, Jul 17, 2012 at 6:47 AM, vishal1981
<vi...@ericsson.com> wrote:
> Thanks again.
> I am using something like the following,
>
> from(file.......).to(ejb:cms/CamelToPluginAdaptor/local?method=process(${header.CamelFileAbsolutePath},${properties:outbox}/${header.PROVIDER_ID}.xml,${header.PROVIDER_ID})
>
> Where cms/CamelToPluginAdaptor /local is my EJB.
>
> Will that prevent reading the whole file in memory?
>
Yes the parameters for the method call, does not require to content of
the file, so Camel will not load it.
>
>
> ________________________________
> From: Claus Ibsen-2 [via Camel] [mailto:ml-node+s465427n5716140h66@n5.nabble.com]
> Sent: Tuesday, July 17, 2012 12:43 AM
> To: Vishal Changrani
> Subject: Re: Preventing reading of a large XML file in memory
>
> On Tue, Jul 17, 2012 at 6:18 AM, vishal1981
> <[hidden email]</user/SendEmail.jtp?type=node&node=5716140&i=0>> wrote:
>> Thanks. I read about the working directory and thats exactly what I was looking for. thank you.
>> Yet to try it though.
>> However, what if I want the same behaviour for File. i.e. File ->Bean (no FTP). I dont want the file component to read the whole file just pass along the file path to the bean?
>> Is there something similiar for File component?
>>
>
> Yes as the file component only stores the handle to the file. And only
> read it on demand.
> So in your bean, just pass in java.io.File, or a FileInputStream,
> InputStream etc.
>
>>
>> ________________________________
>> From: Claus Ibsen-2 [via Camel] [mailto:[hidden email]</user/SendEmail.jtp?type=node&node=5716140&i=1>]
>> Sent: Monday, July 16, 2012 2:54 PM
>> To: Vishal Changrani
>> Subject: Re: Preventing reading of a large XML file in memory
>>
>> Sorry it was the ftp endpoint i meant
>>
>> On Monday, July 16, 2012, vishal1981 wrote:
>>
>>> Thanks for the reply.
>>> So is it compulsory to use a JMS Endpoint to prevent Camel from reading the
>>> whole message in memory?
>>> What is to my bean I dont pass the message body as a parameter but just
>>> then
>>> filename? Will that prevent Camel from loading the whole message in memory
>>> before calling my bean method?
>>> i.e. from(ftp.....).to(bean(MyBean.class,
>>> "doSomething(${header.CamelFileName})")
>>> (am using Camel 2.9)
>>>
>>>
>>> --
>>> View this message in context:
>>> http://camel.465427.n5.nabble.com/Preventing-reading-of-a-large-XML-file-in-memory-tp5716064p5716083.html
>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>
>>
>>
>> --
>> Claus Ibsen
>> -----------------
>> FuseSource
>> Email: [hidden email]</user/SendEmail.jtp?type=node&node=5716115&i=0>
>> Web: http://fusesource.com
>> Twitter: davsclaus, fusenews
>> Blog: http://davsclaus.com
>> Author of Camel in Action: http://www.manning.com/ibsen
>>
>>
>> ________________________________
>> If you reply to this email, your message will be added to the discussion below:
>>
>> NAML<http://camel.465427.n5.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>
>>
>> --
>> View this message in context: http://camel.465427.n5.nabble.com/Preventing-reading-of-a-large-XML-file-in-memory-tp5716064p5716139.html
>> Sent from the Camel - Users mailing list archive at Nabble.com.
>
>
>
> --
> Claus Ibsen
> -----------------
> FuseSource
> Email: [hidden email]</user/SendEmail.jtp?type=node&node=5716140&i=2>
> Web: http://fusesource.com
> Twitter: davsclaus, fusenews
> Blog: http://davsclaus.com
> Author of Camel in Action: http://www.manning.com/ibsen
>
>
> ________________________________
> If you reply to this email, your message will be added to the discussion below:
> http://camel.465427.n5.nabble.com/Preventing-reading-of-a-large-XML-file-in-memory-tp5716064p5716140.html
> To unsubscribe from Preventing reading of a large XML file in memory, click here<http://camel.465427.n5.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=5716064&code=dmlzaGFsLmNoYW5ncmFuaUBlcmljc3Nvbi5jb218NTcxNjA2NHwxNTgwMjU0NDcx>.
> NAML<http://camel.465427.n5.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Preventing-reading-of-a-large-XML-file-in-memory-tp5716064p5716141.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
--
Claus Ibsen
-----------------
FuseSource
Email: cibsen@fusesource.com
Web: http://fusesource.com
Twitter: davsclaus, fusenews
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
RE: Preventing reading of a large XML file in memory
Posted by vishal1981 <vi...@ericsson.com>.
Thanks again.
I am using something like the following,
from(file.......).to(ejb:cms/CamelToPluginAdaptor/local?method=process(${header.CamelFileAbsolutePath},${properties:outbox}/${header.PROVIDER_ID}.xml,${header.PROVIDER_ID})
Where cms/CamelToPluginAdaptor /local is my EJB.
Will that prevent reading the whole file in memory?
________________________________
From: Claus Ibsen-2 [via Camel] [mailto:ml-node+s465427n5716140h66@n5.nabble.com]
Sent: Tuesday, July 17, 2012 12:43 AM
To: Vishal Changrani
Subject: Re: Preventing reading of a large XML file in memory
On Tue, Jul 17, 2012 at 6:18 AM, vishal1981
<[hidden email]</user/SendEmail.jtp?type=node&node=5716140&i=0>> wrote:
> Thanks. I read about the working directory and thats exactly what I was looking for. thank you.
> Yet to try it though.
> However, what if I want the same behaviour for File. i.e. File ->Bean (no FTP). I dont want the file component to read the whole file just pass along the file path to the bean?
> Is there something similiar for File component?
>
Yes as the file component only stores the handle to the file. And only
read it on demand.
So in your bean, just pass in java.io.File, or a FileInputStream,
InputStream etc.
>
> ________________________________
> From: Claus Ibsen-2 [via Camel] [mailto:[hidden email]</user/SendEmail.jtp?type=node&node=5716140&i=1>]
> Sent: Monday, July 16, 2012 2:54 PM
> To: Vishal Changrani
> Subject: Re: Preventing reading of a large XML file in memory
>
> Sorry it was the ftp endpoint i meant
>
> On Monday, July 16, 2012, vishal1981 wrote:
>
>> Thanks for the reply.
>> So is it compulsory to use a JMS Endpoint to prevent Camel from reading the
>> whole message in memory?
>> What is to my bean I dont pass the message body as a parameter but just
>> then
>> filename? Will that prevent Camel from loading the whole message in memory
>> before calling my bean method?
>> i.e. from(ftp.....).to(bean(MyBean.class,
>> "doSomething(${header.CamelFileName})")
>> (am using Camel 2.9)
>>
>>
>> --
>> View this message in context:
>> http://camel.465427.n5.nabble.com/Preventing-reading-of-a-large-XML-file-in-memory-tp5716064p5716083.html
>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>
>
>
> --
> Claus Ibsen
> -----------------
> FuseSource
> Email: [hidden email]</user/SendEmail.jtp?type=node&node=5716115&i=0>
> Web: http://fusesource.com
> Twitter: davsclaus, fusenews
> Blog: http://davsclaus.com
> Author of Camel in Action: http://www.manning.com/ibsen
>
>
> ________________________________
> If you reply to this email, your message will be added to the discussion below:
>
> NAML<http://camel.465427.n5.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Preventing-reading-of-a-large-XML-file-in-memory-tp5716064p5716139.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
--
Claus Ibsen
-----------------
FuseSource
Email: [hidden email]</user/SendEmail.jtp?type=node&node=5716140&i=2>
Web: http://fusesource.com
Twitter: davsclaus, fusenews
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
________________________________
If you reply to this email, your message will be added to the discussion below:
http://camel.465427.n5.nabble.com/Preventing-reading-of-a-large-XML-file-in-memory-tp5716064p5716140.html
To unsubscribe from Preventing reading of a large XML file in memory, click here<http://camel.465427.n5.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=5716064&code=dmlzaGFsLmNoYW5ncmFuaUBlcmljc3Nvbi5jb218NTcxNjA2NHwxNTgwMjU0NDcx>.
NAML<http://camel.465427.n5.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
--
View this message in context: http://camel.465427.n5.nabble.com/Preventing-reading-of-a-large-XML-file-in-memory-tp5716064p5716141.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Preventing reading of a large XML file in memory
Posted by Claus Ibsen <cl...@gmail.com>.
On Tue, Jul 17, 2012 at 6:18 AM, vishal1981
<vi...@ericsson.com> wrote:
> Thanks. I read about the working directory and thats exactly what I was looking for. thank you.
> Yet to try it though.
> However, what if I want the same behaviour for File. i.e. File ->Bean (no FTP). I dont want the file component to read the whole file just pass along the file path to the bean?
> Is there something similiar for File component?
>
Yes as the file component only stores the handle to the file. And only
read it on demand.
So in your bean, just pass in java.io.File, or a FileInputStream,
InputStream etc.
>
> ________________________________
> From: Claus Ibsen-2 [via Camel] [mailto:ml-node+s465427n5716115h91@n5.nabble.com]
> Sent: Monday, July 16, 2012 2:54 PM
> To: Vishal Changrani
> Subject: Re: Preventing reading of a large XML file in memory
>
> Sorry it was the ftp endpoint i meant
>
> On Monday, July 16, 2012, vishal1981 wrote:
>
>> Thanks for the reply.
>> So is it compulsory to use a JMS Endpoint to prevent Camel from reading the
>> whole message in memory?
>> What is to my bean I dont pass the message body as a parameter but just
>> then
>> filename? Will that prevent Camel from loading the whole message in memory
>> before calling my bean method?
>> i.e. from(ftp.....).to(bean(MyBean.class,
>> "doSomething(${header.CamelFileName})")
>> (am using Camel 2.9)
>>
>>
>> --
>> View this message in context:
>> http://camel.465427.n5.nabble.com/Preventing-reading-of-a-large-XML-file-in-memory-tp5716064p5716083.html
>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>
>
>
> --
> Claus Ibsen
> -----------------
> FuseSource
> Email: [hidden email]</user/SendEmail.jtp?type=node&node=5716115&i=0>
> Web: http://fusesource.com
> Twitter: davsclaus, fusenews
> Blog: http://davsclaus.com
> Author of Camel in Action: http://www.manning.com/ibsen
>
>
> ________________________________
> If you reply to this email, your message will be added to the discussion below:
> http://camel.465427.n5.nabble.com/Preventing-reading-of-a-large-XML-file-in-memory-tp5716064p5716115.html
> To unsubscribe from Preventing reading of a large XML file in memory, click here<http://camel.465427.n5.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=5716064&code=dmlzaGFsLmNoYW5ncmFuaUBlcmljc3Nvbi5jb218NTcxNjA2NHwxNTgwMjU0NDcx>.
> NAML<http://camel.465427.n5.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Preventing-reading-of-a-large-XML-file-in-memory-tp5716064p5716139.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
--
Claus Ibsen
-----------------
FuseSource
Email: cibsen@fusesource.com
Web: http://fusesource.com
Twitter: davsclaus, fusenews
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
RE: Preventing reading of a large XML file in memory
Posted by vishal1981 <vi...@ericsson.com>.
Thanks. I read about the working directory and thats exactly what I was looking for. thank you.
Yet to try it though.
However, what if I want the same behaviour for File. i.e. File ->Bean (no FTP). I dont want the file component to read the whole file just pass along the file path to the bean?
Is there something similiar for File component?
________________________________
From: Claus Ibsen-2 [via Camel] [mailto:ml-node+s465427n5716115h91@n5.nabble.com]
Sent: Monday, July 16, 2012 2:54 PM
To: Vishal Changrani
Subject: Re: Preventing reading of a large XML file in memory
Sorry it was the ftp endpoint i meant
On Monday, July 16, 2012, vishal1981 wrote:
> Thanks for the reply.
> So is it compulsory to use a JMS Endpoint to prevent Camel from reading the
> whole message in memory?
> What is to my bean I dont pass the message body as a parameter but just
> then
> filename? Will that prevent Camel from loading the whole message in memory
> before calling my bean method?
> i.e. from(ftp.....).to(bean(MyBean.class,
> "doSomething(${header.CamelFileName})")
> (am using Camel 2.9)
>
>
> --
> View this message in context:
> http://camel.465427.n5.nabble.com/Preventing-reading-of-a-large-XML-file-in-memory-tp5716064p5716083.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
--
Claus Ibsen
-----------------
FuseSource
Email: [hidden email]</user/SendEmail.jtp?type=node&node=5716115&i=0>
Web: http://fusesource.com
Twitter: davsclaus, fusenews
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
________________________________
If you reply to this email, your message will be added to the discussion below:
http://camel.465427.n5.nabble.com/Preventing-reading-of-a-large-XML-file-in-memory-tp5716064p5716115.html
To unsubscribe from Preventing reading of a large XML file in memory, click here<http://camel.465427.n5.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=5716064&code=dmlzaGFsLmNoYW5ncmFuaUBlcmljc3Nvbi5jb218NTcxNjA2NHwxNTgwMjU0NDcx>.
NAML<http://camel.465427.n5.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
--
View this message in context: http://camel.465427.n5.nabble.com/Preventing-reading-of-a-large-XML-file-in-memory-tp5716064p5716139.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Preventing reading of a large XML file in memory
Posted by Claus Ibsen <cl...@gmail.com>.
Sorry it was the ftp endpoint i meant
On Monday, July 16, 2012, vishal1981 wrote:
> Thanks for the reply.
> So is it compulsory to use a JMS Endpoint to prevent Camel from reading the
> whole message in memory?
> What is to my bean I dont pass the message body as a parameter but just
> then
> filename? Will that prevent Camel from loading the whole message in memory
> before calling my bean method?
> i.e. from(ftp.....).to(bean(MyBean.class,
> "doSomething(${header.CamelFileName})")
> (am using Camel 2.9)
>
>
> --
> View this message in context:
> http://camel.465427.n5.nabble.com/Preventing-reading-of-a-large-XML-file-in-memory-tp5716064p5716083.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
--
Claus Ibsen
-----------------
FuseSource
Email: cibsen@fusesource.com
Web: http://fusesource.com
Twitter: davsclaus, fusenews
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
Re: Preventing reading of a large XML file in memory
Posted by vishal1981 <vi...@ericsson.com>.
Thanks for the reply.
So is it compulsory to use a JMS Endpoint to prevent Camel from reading the
whole message in memory?
What is to my bean I dont pass the message body as a parameter but just then
filename? Will that prevent Camel from loading the whole message in memory
before calling my bean method?
i.e. from(ftp.....).to(bean(MyBean.class,
"doSomething(${header.CamelFileName})")
(am using Camel 2.9)
--
View this message in context: http://camel.465427.n5.nabble.com/Preventing-reading-of-a-large-XML-file-in-memory-tp5716064p5716083.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Preventing reading of a large XML file in memory
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
Use the localWorkDirectory option on the JMS endpoint to stream
directly to a file when downloading the ftp file.
Then the message body is a GenericFile type (also a
org.apache.camel.WrappedFile from Camel 2.9 onwards) where you can
access the file using a stream etc.
So if you want to
from ftp
to bean (figure out file name)
to file
Then in the bean you need to set the header Exchange.FILE_NAME to
whatever you want the file to be saved as.
For that you can use the setHeader EIP and then use a method call
expression (eg like a method call on a bean).
On Mon, Jul 16, 2012 at 8:29 AM, vishal1981
<vi...@ericsson.com> wrote:
> Hi,
> I am using Camel to transfer a file from my app server to an FTP server
> before converting it to a different format. The incoming file can be
> potentially huge (60 to 100 gbs) in size and I want to prevent Camel from
> reading the whole file in memory. I would like to call a bean which takes
> the input file location and an output file location as a parameter and not
> the message body.
> Any pointers on how I can achieve this without reading the whole file in
> memory?
> Thanks in advance.
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Preventing-reading-of-a-large-XML-file-in-memory-tp5716064.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
--
Claus Ibsen
-----------------
FuseSource
Email: cibsen@fusesource.com
Web: http://fusesource.com
Twitter: davsclaus, fusenews
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen