You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by arshid <ay...@gmail.com> on 2019/10/07 11:25:13 UTC
too many objects
Hi all,
I am reading a csv file around 300 gb and then splitting it and parsing it to json and sending it to rest api using http4 provider.
The problem whenever I use http4 provider it creates lots of objects of type TempfileManager. Which results in slow down after running for few hours.
I have attached the screenshot of jvisualvm
Can anyone help me understand in what is happening
Sent from Mail for Windows 10
Re: too many objects
Posted by Willem Jiang <wi...@gmail.com>.
For the first part of route, it should be fine to remove the
streamCaching setting.
For the second part I'm not sure if we need to enable streamCaching
when message sending failed, maybe you can do a simple test for it.
Willem Jiang
Twitter: willemjiang
Weibo: 姜宁willem
On Thu, Oct 10, 2019 at 12:56 AM arshid <ay...@gmail.com> wrote:
>
> from(fileURI)
> .to(dataformatURL)
> .split(body())
> .streaming()
> .parallelProcessing()
> .marshal().json(JsonLibrary.Jackson)
> .to("direct:api");
>
>
>
> from("direct:api")
> .streamCaching()
> .onException(Exception.class)
> .handled(true)
> .to(errorFormat)
> .end()
> .loadBalance()
> .failover(maxAttempts, false, true)
> .to(hostsArray)
> end()
>
>
> above is stripped down version of my route.
>
>
> If I remove streamcaching the route works fine
>
> Sent from Mail for Windows 10
>
> From: Willem Jiang
> Sent: 09 October 2019 05:58
> To: users@camel.apache.org
> Subject: Re: too many objects
>
> The TempfileManager is used in the CachedOutputStream.java[1].
> If you can provide your route, we can see if disable the stream
> cache[2] have any side effects on your route.
>
> [1]https://github.com/apache/camel/blob/master/core/camel-support/src/main/java/org/apache/camel/converter/stream/CachedOutputStream.java
> [2]https://camel.apache.org/manual/latest/stream-caching.html
>
> Willem Jiang
>
> Twitter: willemjiang
> Weibo: 姜宁willem
>
> On Mon, Oct 7, 2019 at 7:34 PM arshid <ay...@gmail.com> wrote:
> >
> >
> >
> > Hi all,
> >
> >
> >
> > I am reading a csv file around 300 gb and then splitting it and parsing it to json and sending it to rest api using http4 provider.
> >
> > The problem whenever I use http4 provider it creates lots of objects of type TempfileManager. Which results in slow down after running for few hours.
> >
> > I have attached the screenshot of jvisualvm
> >
> >
> >
> > Can anyone help me understand in what is happening
> >
> >
> >
> >
> >
> >
> >
> > Sent from Mail for Windows 10
> >
> >
>
RE: too many objects
Posted by arshid <ay...@gmail.com>.
from(fileURI)
.to(dataformatURL)
.split(body())
.streaming()
.parallelProcessing()
.marshal().json(JsonLibrary.Jackson)
.to("direct:api");
from("direct:api")
.streamCaching()
.onException(Exception.class)
.handled(true)
.to(errorFormat)
.end()
.loadBalance()
.failover(maxAttempts, false, true)
.to(hostsArray)
end()
above is stripped down version of my route.
If I remove streamcaching the route works fine
Sent from Mail for Windows 10
From: Willem Jiang
Sent: 09 October 2019 05:58
To: users@camel.apache.org
Subject: Re: too many objects
The TempfileManager is used in the CachedOutputStream.java[1].
If you can provide your route, we can see if disable the stream
cache[2] have any side effects on your route.
[1]https://github.com/apache/camel/blob/master/core/camel-support/src/main/java/org/apache/camel/converter/stream/CachedOutputStream.java
[2]https://camel.apache.org/manual/latest/stream-caching.html
Willem Jiang
Twitter: willemjiang
Weibo: 姜宁willem
On Mon, Oct 7, 2019 at 7:34 PM arshid <ay...@gmail.com> wrote:
>
>
>
> Hi all,
>
>
>
> I am reading a csv file around 300 gb and then splitting it and parsing it to json and sending it to rest api using http4 provider.
>
> The problem whenever I use http4 provider it creates lots of objects of type TempfileManager. Which results in slow down after running for few hours.
>
> I have attached the screenshot of jvisualvm
>
>
>
> Can anyone help me understand in what is happening
>
>
>
>
>
>
>
> Sent from Mail for Windows 10
>
>
Re: too many objects
Posted by Willem Jiang <wi...@gmail.com>.
The TempfileManager is used in the CachedOutputStream.java[1].
If you can provide your route, we can see if disable the stream
cache[2] have any side effects on your route.
[1]https://github.com/apache/camel/blob/master/core/camel-support/src/main/java/org/apache/camel/converter/stream/CachedOutputStream.java
[2]https://camel.apache.org/manual/latest/stream-caching.html
Willem Jiang
Twitter: willemjiang
Weibo: 姜宁willem
On Mon, Oct 7, 2019 at 7:34 PM arshid <ay...@gmail.com> wrote:
>
>
>
> Hi all,
>
>
>
> I am reading a csv file around 300 gb and then splitting it and parsing it to json and sending it to rest api using http4 provider.
>
> The problem whenever I use http4 provider it creates lots of objects of type TempfileManager. Which results in slow down after running for few hours.
>
> I have attached the screenshot of jvisualvm
>
>
>
> Can anyone help me understand in what is happening
>
>
>
>
>
>
>
> Sent from Mail for Windows 10
>
>
Re: too many objects
Posted by Mantas Gridinas <mg...@gmail.com>.
What is the actual route definition you're using? I would try swapping
out http4 component for regular http component.
On Mon, Oct 7, 2019 at 2:34 PM arshid <ay...@gmail.com> wrote:
>
>
>
> Hi all,
>
>
>
> I am reading a csv file around 300 gb and then splitting it and parsing it to json and sending it to rest api using http4 provider.
>
> The problem whenever I use http4 provider it creates lots of objects of type TempfileManager. Which results in slow down after running for few hours.
>
> I have attached the screenshot of jvisualvm
>
>
>
> Can anyone help me understand in what is happening
>
>
>
>
>
>
>
> Sent from Mail for Windows 10
>
>