You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by bbk <br...@hotmail.com> on 2014/04/05 17:41:38 UTC
Java heap space with FILE producer and larges files
Hi,
Since I upgraded Camel 2.6.0 to 2.10.2, the move large files fail using FILE
producer past a certain size (over 200mo). The Camel error is during convert
the body GenericFile to String.
Failed delivery
org.apache.camel.TypeConversionException: Error during type conversion from
type: org.apache.camel.component.file.GenericFile to the required type:
java.lang.String with value GenericFile due java.lang.OutOfMemoryError: Java
heap space
Since Camel 2.9.3, the method genericFileToString in class
GenericFileConverter modified with adding lines:
// use reader first as it supports the file charset
BufferedReader reader = genericFileToReader(file, exchange);
if (reader != null) {
return IOConverter.toString(reader);
}
So, I noticed that my error is due to "IOConverter.toString" causing Java
heap space.
My question is: why my file producer fail in 2.10.2 and not in 2.6.0 and how
I can resolve it?
--
View this message in context: http://camel.465427.n5.nabble.com/Java-heap-space-with-FILE-producer-and-larges-files-tp5749845.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Java heap space with FILE producer and larges files
Posted by bbk <br...@hotmail.com>.
I have created the JIRA for this issue.
https://issues.apache.org/jira/browse/CAMEL-7376
--
View this message in context: http://camel.465427.n5.nabble.com/Java-heap-space-with-FILE-producer-and-larges-files-tp5749845p5750353.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Java heap space with FILE producer and larges files
Posted by bbk <br...@hotmail.com>.
I implemented my own tracer formatter but I have the same problem:
Tracer tracer = new Tracer();
tracer.setFormatter(new CustomMessageFormatter());
....
public class CustomMessageFormatter implements TraceFormatter {
@Override
public Object format(TraceInterceptor interceptor,
ProcessorDefinition<?> node, Exchange exchange) {
return "example";
}
I think the probleme is in the DefaultTraceEventMessage.class.
Like we can see in the trace, the constructor call:
this.body = MessageHelper.extractBodyAsString(in)
and that is the problem...
--
View this message in context: http://camel.465427.n5.nabble.com/Java-heap-space-with-FILE-producer-and-larges-files-tp5749845p5750116.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Java heap space with FILE producer and larges files
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
You would either have to wait for the issue to be fixed and upgrade to
that Camel release.
Or try implement your own tracer formatter where you do not read the
data if its a file.
See the section_Using Custom Formatter_ at
http://camel.apache.org/tracer
On Thu, Apr 10, 2014 at 3:59 PM, bbk <br...@hotmail.com> wrote:
> You're right.
>
> The problem is when I do:
>
> tracer.setDestinationUri("direct:traced");
> this.from("direct:traced")
> .to("file://....");
>
> How can I resolve the problem please?
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Java-heap-space-with-FILE-producer-and-larges-files-tp5749845p5750054.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
--
Claus Ibsen
-----------------
Red Hat, Inc.
Email: cibsen@redhat.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
hawtio: http://hawt.io/
fabric8: http://fabric8.io/
Re: Java heap space with FILE producer and larges files
Posted by bbk <br...@hotmail.com>.
You're right.
The problem is when I do:
tracer.setDestinationUri("direct:traced");
this.from("direct:traced")
.to("file://....");
How can I resolve the problem please?
--
View this message in context: http://camel.465427.n5.nabble.com/Java-heap-space-with-FILE-producer-and-larges-files-tp5749845p5750054.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Java heap space with FILE producer and larges files
Posted by Claus Ibsen <cl...@gmail.com>.
On Thu, Apr 10, 2014 at 8:20 AM, Claus Ibsen <cl...@gmail.com> wrote:
> That is because you have enabled tracing, you should turn that off.
> http://camel.apache.org/tracer.html
>
I logged a ticket to improve the tracer, so it wont load the file into
memory by deafult
https://issues.apache.org/jira/browse/CAMEL-7358
>
>
> On Wed, Apr 9, 2014 at 4:08 PM, bbk <br...@hotmail.com> wrote:
>> Thank you for your response.
>>
>> But I have the same problem but somewhere and I don't find that is causing.
>> The trace is:
>>
>> Error during type conversion from type:
>> org.apache.camel.component.file.Generi
>> cFile to the required type: java.lang.String with value GenericFile[...] due
>> java.lang.OutOfMemoryError: Java heap space
>> at
>> org.apache.camel.impl.converter.BaseTypeConverterRegistry.convertTo(BaseTypeConverterRegistry.java:130)
>> at
>> org.apache.camel.core.osgi.OsgiTypeConverter.convertTo(OsgiTypeConverter.java:102)
>> at
>> org.apache.camel.impl.MessageSupport.getBody(MessageSupport.java:72)
>> at
>> org.apache.camel.impl.MessageSupport.getBody(MessageSupport.java:47)
>> at
>> org.apache.camel.util.MessageHelper.extractBodyAsString(MessageHelper.java:67)
>> at
>> org.apache.camel.processor.interceptor.DefaultTraceEventMessage.<init>(DefaultTraceEventMessage.java:75)
>> at
>> org.apache.camel.processor.interceptor.DefaultTraceEventHandler.traceExchange(DefaultTraceEventHandler.java:8
>> 6)
>> at
>> org.apache.camel.processor.interceptor.TraceInterceptor.traceExchange(TraceInterceptor.java:291)
>> at
>> org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:151)
>> ...
>>
>> Thank you for your help.
>>
>>
>>
>> --
>> View this message in context: http://camel.465427.n5.nabble.com/Java-heap-space-with-FILE-producer-and-larges-files-tp5749845p5749999.html
>> Sent from the Camel - Users mailing list archive at Nabble.com.
>
>
>
> --
> Claus Ibsen
> -----------------
> Red Hat, Inc.
> Email: cibsen@redhat.com
> Twitter: davsclaus
> Blog: http://davsclaus.com
> Author of Camel in Action: http://www.manning.com/ibsen
> hawtio: http://hawt.io/
> fabric8: http://fabric8.io/
--
Claus Ibsen
-----------------
Red Hat, Inc.
Email: cibsen@redhat.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
hawtio: http://hawt.io/
fabric8: http://fabric8.io/
Re: Java heap space with FILE producer and larges files
Posted by Claus Ibsen <cl...@gmail.com>.
That is because you have enabled tracing, you should turn that off.
http://camel.apache.org/tracer.html
On Wed, Apr 9, 2014 at 4:08 PM, bbk <br...@hotmail.com> wrote:
> Thank you for your response.
>
> But I have the same problem but somewhere and I don't find that is causing.
> The trace is:
>
> Error during type conversion from type:
> org.apache.camel.component.file.Generi
> cFile to the required type: java.lang.String with value GenericFile[...] due
> java.lang.OutOfMemoryError: Java heap space
> at
> org.apache.camel.impl.converter.BaseTypeConverterRegistry.convertTo(BaseTypeConverterRegistry.java:130)
> at
> org.apache.camel.core.osgi.OsgiTypeConverter.convertTo(OsgiTypeConverter.java:102)
> at
> org.apache.camel.impl.MessageSupport.getBody(MessageSupport.java:72)
> at
> org.apache.camel.impl.MessageSupport.getBody(MessageSupport.java:47)
> at
> org.apache.camel.util.MessageHelper.extractBodyAsString(MessageHelper.java:67)
> at
> org.apache.camel.processor.interceptor.DefaultTraceEventMessage.<init>(DefaultTraceEventMessage.java:75)
> at
> org.apache.camel.processor.interceptor.DefaultTraceEventHandler.traceExchange(DefaultTraceEventHandler.java:8
> 6)
> at
> org.apache.camel.processor.interceptor.TraceInterceptor.traceExchange(TraceInterceptor.java:291)
> at
> org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:151)
> ...
>
> Thank you for your help.
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Java-heap-space-with-FILE-producer-and-larges-files-tp5749845p5749999.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
--
Claus Ibsen
-----------------
Red Hat, Inc.
Email: cibsen@redhat.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
hawtio: http://hawt.io/
fabric8: http://fabric8.io/
Re: Java heap space with FILE producer and larges files
Posted by bbk <br...@hotmail.com>.
Thank you for your response.
But I have the same problem but somewhere and I don't find that is causing.
The trace is:
Error during type conversion from type:
org.apache.camel.component.file.Generi
cFile to the required type: java.lang.String with value GenericFile[...] due
java.lang.OutOfMemoryError: Java heap space
at
org.apache.camel.impl.converter.BaseTypeConverterRegistry.convertTo(BaseTypeConverterRegistry.java:130)
at
org.apache.camel.core.osgi.OsgiTypeConverter.convertTo(OsgiTypeConverter.java:102)
at
org.apache.camel.impl.MessageSupport.getBody(MessageSupport.java:72)
at
org.apache.camel.impl.MessageSupport.getBody(MessageSupport.java:47)
at
org.apache.camel.util.MessageHelper.extractBodyAsString(MessageHelper.java:67)
at
org.apache.camel.processor.interceptor.DefaultTraceEventMessage.<init>(DefaultTraceEventMessage.java:75)
at
org.apache.camel.processor.interceptor.DefaultTraceEventHandler.traceExchange(DefaultTraceEventHandler.java:8
6)
at
org.apache.camel.processor.interceptor.TraceInterceptor.traceExchange(TraceInterceptor.java:291)
at
org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:151)
...
Thank you for your help.
--
View this message in context: http://camel.465427.n5.nabble.com/Java-heap-space-with-FILE-producer-and-larges-files-tp5749845p5749999.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Java heap space with FILE producer and larges files
Posted by Claus Ibsen <cl...@gmail.com>.
On Wed, Apr 9, 2014 at 11:27 AM, bbk <br...@hotmail.com> wrote:
> Hi
>
> The problem is when I do on the large files:
> - exchange.getIn().getBody(InputStream.class)
> - exchange.getIn().getBody(String.class)
here is your problem as you convert the payload to a String which
reads in the entire file to memory. Do NOT do that.
Use the other one where you convert it to an input stream, then you
can access the file using streaming mode.
> ...
>
> So, how can I resolve this problem?
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Java-heap-space-with-FILE-producer-and-larges-files-tp5749845p5749990.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
--
Claus Ibsen
-----------------
Red Hat, Inc.
Email: cibsen@redhat.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
hawtio: http://hawt.io/
fabric8: http://fabric8.io/
Re: Java heap space with FILE producer and larges files
Posted by bbk <br...@hotmail.com>.
Hi
The problem is when I do on the large files:
- exchange.getIn().getBody(InputStream.class)
- exchange.getIn().getBody(String.class)
...
So, how can I resolve this problem?
--
View this message in context: http://camel.465427.n5.nabble.com/Java-heap-space-with-FILE-producer-and-larges-files-tp5749845p5749990.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Java heap space with FILE producer and larges files
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
What are you doing in Camel. It seems your issue is maybe that
somehow/somewhere you force a type conversion from file -> String and
that is causing to load the entire file into memory. And therefor you
should find that problem and fix that instead.
On Tue, Apr 8, 2014 at 9:04 PM, bbk <br...@hotmail.com> wrote:
> I found my problem: https://issues.apache.org/jira/browse/CAMEL-5172
> So with 2.6.0, I didn't see exception but null.
>
> I want to use the old behavior. So I implemented TypeConverter and custom
> genericFileToString with the old behavior. My converter return null but
> after org.apache.camel.component.file.genericFileToString is used.
>
> Is it possible to remove org.apache.camel.component.file.genericFileToString
> converter?
>
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Java-heap-space-with-FILE-producer-and-larges-files-tp5749845p5749974.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
--
Claus Ibsen
-----------------
Red Hat, Inc.
Email: cibsen@redhat.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
hawtio: http://hawt.io/
fabric8: http://fabric8.io/
Re: Java heap space with FILE producer and larges files
Posted by bbk <br...@hotmail.com>.
I found my problem: https://issues.apache.org/jira/browse/CAMEL-5172
So with 2.6.0, I didn't see exception but null.
I want to use the old behavior. So I implemented TypeConverter and custom
genericFileToString with the old behavior. My converter return null but
after org.apache.camel.component.file.genericFileToString is used.
Is it possible to remove org.apache.camel.component.file.genericFileToString
converter?
--
View this message in context: http://camel.465427.n5.nabble.com/Java-heap-space-with-FILE-producer-and-larges-files-tp5749845p5749974.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Java heap space with FILE producer and larges files
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
See the how to get help on this page, and the bullets
http://camel.apache.org/support
For example
> try upgrading to the latest release and see if it's fixed there
Also if you convert a file -> String then the file is being read into
memory. And reading 200mb into memory takes up memory.
On Sun, Apr 6, 2014 at 9:35 PM, bbk <br...@hotmail.com> wrote:
> Thank you for your response.
>
> I don't see the differences in GenricFileConverter and IOConverter source
> class between 2.10.2 and 2.13.0.
>
> What are the differences between 2.10.2 and 2.13.0 that will resolve the
> problem?
>
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Java-heap-space-with-FILE-producer-and-larges-files-tp5749845p5749863.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
--
Claus Ibsen
-----------------
Red Hat, Inc.
Email: cibsen@redhat.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
Make your Camel applications look hawt, try: http://hawt.io
Re: Java heap space with FILE producer and larges files
Posted by bbk <br...@hotmail.com>.
Thank you for your response.
I don't see the differences in GenricFileConverter and IOConverter source
class between 2.10.2 and 2.13.0.
What are the differences between 2.10.2 and 2.13.0 that will resolve the
problem?
--
View this message in context: http://camel.465427.n5.nabble.com/Java-heap-space-with-FILE-producer-and-larges-files-tp5749845p5749863.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Java heap space with FILE producer and larges files
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
Can you try with Camel 2.13.0 release
On Sat, Apr 5, 2014 at 5:41 PM, bbk <br...@hotmail.com> wrote:
> Hi,
>
> Since I upgraded Camel 2.6.0 to 2.10.2, the move large files fail using FILE
> producer past a certain size (over 200mo). The Camel error is during convert
> the body GenericFile to String.
>
> Failed delivery
> org.apache.camel.TypeConversionException: Error during type conversion from
> type: org.apache.camel.component.file.GenericFile to the required type:
> java.lang.String with value GenericFile due java.lang.OutOfMemoryError: Java
> heap space
>
> Since Camel 2.9.3, the method genericFileToString in class
> GenericFileConverter modified with adding lines:
>
> // use reader first as it supports the file charset
> BufferedReader reader = genericFileToReader(file, exchange);
> if (reader != null) {
> return IOConverter.toString(reader);
> }
>
> So, I noticed that my error is due to "IOConverter.toString" causing Java
> heap space.
>
> My question is: why my file producer fail in 2.10.2 and not in 2.6.0 and how
> I can resolve it?
>
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Java-heap-space-with-FILE-producer-and-larges-files-tp5749845.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
--
Claus Ibsen
-----------------
Red Hat, Inc.
Email: cibsen@redhat.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
Make your Camel applications look hawt, try: http://hawt.io