You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by "Karthik.Pothuru" <Ka...@target.com> on 2014/04/18 14:32:26 UTC
Out of Memory For huge Json Response
Dear Team,
We are facing Out of memory issue when receiving a huge json response (around 1 GB ) when hitting a REST webservice with in Target Enterprise. We are currently using camel Http component .
Due to huge file, the entire data gets loaded into HEAP and the process is failing due to that.
Our requirement is to read this response and store it into a file which will later be parsed for further data processing.
Due to this issue, we are unable to go the next step of parsing. The same code works fine when small volume of data is received through the API. (at present we remember processing a response size of 150 MB and it was successful).
Kindly need your input how to read response with lesser memory foot print.
Thanks
Karthik Pothuru
Mob: +91 91766 97533
RE: Out of Memory For huge Json Response
Posted by "Venkatesh.Krishnamoorthy" <Ve...@target.com>.
Thanks much Fernando for your support. We were able to resolve this with streamcaching and readLock check.
Regards,
Venkat
Venkatesh Krishnamoorthy | Service Reinvention – Integration Development Lead | • Target - [Description: Description: Description: cid:image001.png@01CE948A.B9662580] TCS
Work: 612.304.7278 | Mobile: 612.205.4931
From: Fernando Ribeiro [mailto:fernando.ribeiro@upic.com.br]
Sent: Friday, April 18, 2014 10:19 AM
To: users@camel.apache.org
Cc: Rajbarath.Govindan; Venkatesh.Krishnamoorthy
Subject: Re: Out of Memory For huge Json Response
Why is the payload so big?
On Fri, Apr 18, 2014 at 9:32 AM, Karthik.Pothuru <Ka...@target.com>> wrote:
Dear Team,
We are facing Out of memory issue when receiving a huge json response (around 1 GB ) when hitting a REST webservice with in Target Enterprise. We are currently using camel Http component .
Due to huge file, the entire data gets loaded into HEAP and the process is failing due to that.
Our requirement is to read this response and store it into a file which will later be parsed for further data processing.
Due to this issue, we are unable to go the next step of parsing. The same code works fine when small volume of data is received through the API. (at present we remember processing a response size of 150 MB and it was successful).
Kindly need your input how to read response with lesser memory foot print.
Thanks
Karthik Pothuru
Mob: +91 91766 97533<tel:%2B91%2091766%2097533>
--
Fernando Ribeiro
RE: Out of Memory For huge Json Response
Posted by "Venkatesh.Krishnamoorthy" <Ve...@target.com>.
Hi Fernando,
We get one time load of all data from the source system and this will happen from time to time on an on-demand basis. The source system will push all data – huge json response over http. The current camel code is trying to load this data on limited heap instead of in chunks and stream it into a file. We want to avoid increase in jvm heap with increase in the response size by using the best Camel method to stream data in chunks.
Appreciate your quick response and looking forward to a resolution.
Thanks & Regards,
Venkat
From: Fernando Ribeiro [mailto:fernando.ribeiro@upic.com.br]
Sent: Friday, April 18, 2014 10:19 AM
To: users@camel.apache.org
Cc: Rajbarath.Govindan; Venkatesh.Krishnamoorthy
Subject: Re: Out of Memory For huge Json Response
Why is the payload so big?
On Fri, Apr 18, 2014 at 9:32 AM, Karthik.Pothuru <Ka...@target.com>> wrote:
Dear Team,
We are facing Out of memory issue when receiving a huge json response (around 1 GB ) when hitting a REST webservice with in Target Enterprise. We are currently using camel Http component .
Due to huge file, the entire data gets loaded into HEAP and the process is failing due to that.
Our requirement is to read this response and store it into a file which will later be parsed for further data processing.
Due to this issue, we are unable to go the next step of parsing. The same code works fine when small volume of data is received through the API. (at present we remember processing a response size of 150 MB and it was successful).
Kindly need your input how to read response with lesser memory foot print.
Thanks
Karthik Pothuru
Mob: +91 91766 97533<tel:%2B91%2091766%2097533>
--
Fernando Ribeiro
Re: Out of Memory For huge Json Response
Posted by Fernando Ribeiro <fe...@upic.com.br>.
Why is the payload so big?
On Fri, Apr 18, 2014 at 9:32 AM, Karthik.Pothuru <Karthik.Pothuru@target.com
> wrote:
> Dear Team,
>
> We are facing Out of memory issue when receiving a huge json response
> (around 1 GB ) when hitting a REST webservice with in Target Enterprise. We
> are currently using camel Http component .
> Due to huge file, the entire data gets loaded into HEAP and the process is
> failing due to that.
>
> Our requirement is to read this response and store it into a file which
> will later be parsed for further data processing.
>
> Due to this issue, we are unable to go the next step of parsing. The same
> code works fine when small volume of data is received through the API. (at
> present we remember processing a response size of 150 MB and it was
> successful).
>
> Kindly need your input how to read response with lesser memory foot print.
>
> Thanks
> Karthik Pothuru
> Mob: +91 91766 97533
>
--
Fernando Ribeiro
Re: Out of Memory For huge Json Response
Posted by Jason Stell <js...@gmail.com>.
If you can't allocate enough java heap to support the 1GB JSON document,
you'll have to switch to a Streaming JSON parser approach. Jackson supports
this : http://wiki.fasterxml.com/JacksonStreamingApi
Regards
On Fri, Apr 18, 2014 at 7:32 AM, Karthik.Pothuru <Karthik.Pothuru@target.com
> wrote:
> Dear Team,
>
> We are facing Out of memory issue when receiving a huge json response
> (around 1 GB ) when hitting a REST webservice with in Target Enterprise. We
> are currently using camel Http component .
> Due to huge file, the entire data gets loaded into HEAP and the process is
> failing due to that.
>
> Our requirement is to read this response and store it into a file which
> will later be parsed for further data processing.
>
> Due to this issue, we are unable to go the next step of parsing. The same
> code works fine when small volume of data is received through the API. (at
> present we remember processing a response size of 150 MB and it was
> successful).
>
> Kindly need your input how to read response with lesser memory foot print.
>
> Thanks
> Karthik Pothuru
> Mob: +91 91766 97533
>
Re: Out of Memory For huge Json Response
Posted by "kraythe ." <kr...@gmail.com>.
Holy moses! A 1 gig JSON doc? Isn't that a criminal offense somewhere?
Seriously though, you are hitting a JVM issue more than a camel one. I
would see if I could brew up something to shatter the document and have it
streamed in and read it piecemeal. You need something like SAX but for
JSON. My thinking takes me towards using the Jackson JSON parser in a
stream mode. I googled "jackson stream large document" and got some
promising hits.
*Robert Simmons Jr. MSc. - Lead Java Architect @ EA*
*Author of: Hardcore Java (2003) and Maintainable Java (2012)*
*LinkedIn: **http://www.linkedin.com/pub/robert-simmons/40/852/a39
<http://www.linkedin.com/pub/robert-simmons/40/852/a39>*
On Sun, Apr 20, 2014 at 8:49 PM, Willem Jiang <wi...@gmail.com>wrote:
> Can I have a look at your camel route?
> camel-http is using the stream by default and it can spool big streams to
> files.
> I guess there could be something wrong after you get the response from the
> RSET service, So I need to know more detail about your camel route.
>
> --
> Willem Jiang
>
> Red Hat, Inc.
> Web: http://www.redhat.com
> Blog: http://willemjiang.blogspot.com (English)
> http://jnn.iteye.com (Chinese)
> Twitter: willemjiang
> Weibo: 姜宁willem
>
>
>
> On April 18, 2014 at 11:02:51 PM, Karthik.Pothuru (
> karthik.pothuru@target.com) wrote:
> > Dear Team,
> >
> > We are facing Out of memory issue when receiving a huge json response
> (around 1 GB ) when
> > hitting a REST webservice with in Target Enterprise. We are currently
> using camel Http
> > component .
> > Due to huge file, the entire data gets loaded into HEAP and the process
> is failing due to
> > that.
> >
> > Our requirement is to read this response and store it into a file which
> will later be parsed
> > for further data processing.
> >
> > Due to this issue, we are unable to go the next step of parsing. The
> same code works fine
> > when small volume of data is received through the API. (at present we
> remember processing
> > a response size of 150 MB and it was successful).
> >
> > Kindly need your input how to read response with lesser memory foot
> print.
> >
> > Thanks
> > Karthik Pothuru
> > Mob: +91 91766 97533
> >
>
>
Re: Out of Memory For huge Json Response
Posted by Willem Jiang <wi...@gmail.com>.
Can I have a look at your camel route?
camel-http is using the stream by default and it can spool big streams to files.
I guess there could be something wrong after you get the response from the RSET service, So I need to know more detail about your camel route.
--
Willem Jiang
Red Hat, Inc.
Web: http://www.redhat.com
Blog: http://willemjiang.blogspot.com (English)
http://jnn.iteye.com (Chinese)
Twitter: willemjiang
Weibo: 姜宁willem
On April 18, 2014 at 11:02:51 PM, Karthik.Pothuru (karthik.pothuru@target.com) wrote:
> Dear Team,
>
> We are facing Out of memory issue when receiving a huge json response (around 1 GB ) when
> hitting a REST webservice with in Target Enterprise. We are currently using camel Http
> component .
> Due to huge file, the entire data gets loaded into HEAP and the process is failing due to
> that.
>
> Our requirement is to read this response and store it into a file which will later be parsed
> for further data processing.
>
> Due to this issue, we are unable to go the next step of parsing. The same code works fine
> when small volume of data is received through the API. (at present we remember processing
> a response size of 150 MB and it was successful).
>
> Kindly need your input how to read response with lesser memory foot print.
>
> Thanks
> Karthik Pothuru
> Mob: +91 91766 97533
>