You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by _Jens <je...@icw.de> on 2009/03/03 10:59:02 UTC
StreamCaching in Camel 1.6
Hi,
I have run into a problem with stream caching in Camel 1.6. I have a route
that reads data via streaming from an Http request. In some cases this can
be a large amount of data, so I cannot stream the data completely into
memory. I wrote this little test to demonstrate my problem:
@Test
public void testUploadFailsForHugeFile() throws Exception {
CamelContext context = new DefaultCamelContext();
context.addRoutes(new RouteBuilder() {
@Override
public void configure() throws Exception {
noStreamCaching();
from("jetty:http://localhost:8989/bigfile")
.noStreamCaching()
.process(new Processor() {
public void process(Exchange exchange) throws
Exception {
InputStream inputStream =
exchange.getIn().getBody(InputStream.class);
while (inputStream.read() != -1) {}
}
});
}
});
context.start();
HttpClient client = new HttpClient();
PostMethod method = new PostMethod("http://localhost:8989/bigfile");
File file = new File("c:\\temp\\test.bin.ok2");
method.setRequestEntity(new FileRequestEntity(file,
"unknown/unknown"));
assertEquals(200, client.executeMethod(method));
}
This fails with an OutOfMemoryError if test.bin.ok2 is a larger file because
the StreamCachingInterceptor reads the input stream into memory. Now I
understand that stream caching is turned on by default in Camel 1.6.
However, as you can see I tried to disable it in the route builder. This
seems not to make any difference.
Now, my question is if there is some other way to disable stream caching or
if I'm doing something wrong here.
Thanks in advance
Jens
--
View this message in context: http://www.nabble.com/StreamCaching-in-Camel-1.6-tp22305654p22305654.html
Sent from the Camel - Users (activemq) mailing list archive at Nabble.com.
Re: StreamCaching in Camel 1.6
Posted by _Jens <je...@icw.de>.
Hi Claus,
yes, it seems to have an effect because the lifecycle-strategy is different
when disabling JMX. The Constructor of the DefaultCamelContext uses either
DefaultLifecycleStrategy or InstrumentationLifecycleStrategy. This also has
influence on the error handling strategy.
Jens
--
View this message in context: http://www.nabble.com/StreamCaching-in-Camel-1.6-tp22305654p22554404.html
Sent from the Camel - Users (activemq) mailing list archive at Nabble.com.
Re: StreamCaching in Camel 1.6
Posted by _Jens <je...@icw.de>.
Wow nice job! I'll make sure that I try 1.6.1 soon.
Thanks a lot Claus.
--
View this message in context: http://www.nabble.com/StreamCaching-in-Camel-1.6-tp22305654p22581500.html
Sent from the Camel - Users (activemq) mailing list archive at Nabble.com.
Re: StreamCaching in Camel 1.6
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
Just wanted to note that the bug is now fixed in the next release of
Camel - 1.6.1 and 2.0
Thanks for reporting.
On Tue, Mar 17, 2009 at 9:28 AM, Claus Ibsen <cl...@gmail.com> wrote:
> Hi
>
> A quick test shows:
>
> JMX disabled:
> EventDrivenConsumerRoute[Endpoint[direct:start] ->
> UnitOfWork(StreamCachingInterceptor(choice{when body contains Hello:
> Pipeline[DeadLetterChannel[sendTo(Endpoint[mock:hello]),
> log:org.apache.camel.DeadLetterChannel?level=error],
> DeadLetterChannel[StreamCachingInterceptor(choice{when body contains
> Bye: DeadLetterChannel[StreamCachingInterceptor(sendTo(Endpoint[mock:bye])),
> log:org.apache.camel.DeadLetterChannel?level=error]}),
> log:org.apache.camel.DeadLetterChannel?level=error]]}))]
>
>
> JMX enabled:
> EventDrivenConsumerRoute[Endpoint[direct:start] ->
> UnitOfWork(Instrumentation(Instrumentation(choice{when body contains
> Hello: Pipeline[sendTo(Endpoint[mock:hello]), choice{when body
> contains Bye: sendTo(Endpoint[mock:bye])}]})))]
>
> So yeah there is a bug in Camel that is affected by the JMX.
> I will log a JIRA for this.
> https://issues.apache.org/activemq/browse/CAMEL-1463
>
>
>
> On Tue, Mar 17, 2009 at 9:15 AM, Claus Ibsen <cl...@gmail.com> wrote:
>> On Fri, Mar 6, 2009 at 1:46 PM, _Jens <je...@icw.de> wrote:
>>>
>>> Hi,
>>>
>>> first of all, thanks to both of you, Gert and Willem, for the quick replies.
>>> I saw that the jira was already fixed, great job.
>>>
>>> Does this mean that the methods for disabling stream caching will be removed
>>> in the future?
>>>
>>> I have tried to use "noErrorHandler" but there was a special case that used
>>> the default error handler at some point and I wasn't able to change this. A
>>> drilled down version of my scenario is shown in the test case below.
>>>
>>> It has to do with the choice and when elements in the route. They seem to
>>> introduce the standard error handler, as they don't inherit the configured
>>> one. The standard error handling then configures the route to use stream
>>> caching again. For some reason this only seems to happen if the JMX agent is
>>> disabled in the CamelContext, which I simulate by setting the system
>>> property. Of course, in our real scenario we simply disable it in the
>>> application context.
>> So you are saying that with/without JMX it has an influence on your route?
>> That is kinda odd, but there could be a gremlin.
>>
>> Need to check into this.
>>
>>
>>>
>>> @Test
>>> public void testUploadFailsForBigFileWithNoErrorHandler() throws
>>> Exception {
>>> System.setProperty(JmxSystemPropertyKeys.DISABLED, "true");
>>> DefaultCamelContext context = new DefaultCamelContext();
>>>
>>> context.addRoutes(new RouteBuilder() {
>>> @Override
>>> public void configure() throws Exception {
>>> errorHandler(noErrorHandler());
>>>
>>> from("jetty:http://localhost:8989/bigfile")
>>>
>>> .choice().when(header("foo").isEqualTo("bar")).to("direct:end")
>>> .otherwise().end();
>>>
>>> from("direct:end")
>>> .process(new Processor() {
>>> public void process(Exchange exchange) throws
>>> Exception {
>>> InputStream inputStream =
>>> exchange.getIn().getBody(InputStream.class);
>>> while (inputStream.read() != -1) {}
>>> }
>>> });
>>> }
>>> });
>>>
>>> context.start();
>>>
>>> HttpClient client = new HttpClient();
>>> PostMethod method = new PostMethod("http://localhost:8989/bigfile");
>>> File file = new File("c:\\temp\\test.bin.ok2");
>>> method.setRequestEntity(new FileRequestEntity(file,
>>> "unknown/unknown"));
>>> assertEquals(200, client.executeMethod(method));
>>> }
>>>
>>> --
>>> View this message in context: http://www.nabble.com/StreamCaching-in-Camel-1.6-tp22305654p22371815.html
>>> Sent from the Camel - Users (activemq) mailing list archive at Nabble.com.
>>>
>>>
>>
>>
>>
>> --
>> Claus Ibsen
>> Apache Camel Committer
>>
>> Open Source Integration: http://fusesource.com
>> Blog: http://davsclaus.blogspot.com/
>>
>
>
>
> --
> Claus Ibsen
> Apache Camel Committer
>
> Open Source Integration: http://fusesource.com
> Blog: http://davsclaus.blogspot.com/
>
--
Claus Ibsen
Apache Camel Committer
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Re: StreamCaching in Camel 1.6
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
A quick test shows:
JMX disabled:
EventDrivenConsumerRoute[Endpoint[direct:start] ->
UnitOfWork(StreamCachingInterceptor(choice{when body contains Hello:
Pipeline[DeadLetterChannel[sendTo(Endpoint[mock:hello]),
log:org.apache.camel.DeadLetterChannel?level=error],
DeadLetterChannel[StreamCachingInterceptor(choice{when body contains
Bye: DeadLetterChannel[StreamCachingInterceptor(sendTo(Endpoint[mock:bye])),
log:org.apache.camel.DeadLetterChannel?level=error]}),
log:org.apache.camel.DeadLetterChannel?level=error]]}))]
JMX enabled:
EventDrivenConsumerRoute[Endpoint[direct:start] ->
UnitOfWork(Instrumentation(Instrumentation(choice{when body contains
Hello: Pipeline[sendTo(Endpoint[mock:hello]), choice{when body
contains Bye: sendTo(Endpoint[mock:bye])}]})))]
So yeah there is a bug in Camel that is affected by the JMX.
I will log a JIRA for this.
https://issues.apache.org/activemq/browse/CAMEL-1463
On Tue, Mar 17, 2009 at 9:15 AM, Claus Ibsen <cl...@gmail.com> wrote:
> On Fri, Mar 6, 2009 at 1:46 PM, _Jens <je...@icw.de> wrote:
>>
>> Hi,
>>
>> first of all, thanks to both of you, Gert and Willem, for the quick replies.
>> I saw that the jira was already fixed, great job.
>>
>> Does this mean that the methods for disabling stream caching will be removed
>> in the future?
>>
>> I have tried to use "noErrorHandler" but there was a special case that used
>> the default error handler at some point and I wasn't able to change this. A
>> drilled down version of my scenario is shown in the test case below.
>>
>> It has to do with the choice and when elements in the route. They seem to
>> introduce the standard error handler, as they don't inherit the configured
>> one. The standard error handling then configures the route to use stream
>> caching again. For some reason this only seems to happen if the JMX agent is
>> disabled in the CamelContext, which I simulate by setting the system
>> property. Of course, in our real scenario we simply disable it in the
>> application context.
> So you are saying that with/without JMX it has an influence on your route?
> That is kinda odd, but there could be a gremlin.
>
> Need to check into this.
>
>
>>
>> @Test
>> public void testUploadFailsForBigFileWithNoErrorHandler() throws
>> Exception {
>> System.setProperty(JmxSystemPropertyKeys.DISABLED, "true");
>> DefaultCamelContext context = new DefaultCamelContext();
>>
>> context.addRoutes(new RouteBuilder() {
>> @Override
>> public void configure() throws Exception {
>> errorHandler(noErrorHandler());
>>
>> from("jetty:http://localhost:8989/bigfile")
>>
>> .choice().when(header("foo").isEqualTo("bar")).to("direct:end")
>> .otherwise().end();
>>
>> from("direct:end")
>> .process(new Processor() {
>> public void process(Exchange exchange) throws
>> Exception {
>> InputStream inputStream =
>> exchange.getIn().getBody(InputStream.class);
>> while (inputStream.read() != -1) {}
>> }
>> });
>> }
>> });
>>
>> context.start();
>>
>> HttpClient client = new HttpClient();
>> PostMethod method = new PostMethod("http://localhost:8989/bigfile");
>> File file = new File("c:\\temp\\test.bin.ok2");
>> method.setRequestEntity(new FileRequestEntity(file,
>> "unknown/unknown"));
>> assertEquals(200, client.executeMethod(method));
>> }
>>
>> --
>> View this message in context: http://www.nabble.com/StreamCaching-in-Camel-1.6-tp22305654p22371815.html
>> Sent from the Camel - Users (activemq) mailing list archive at Nabble.com.
>>
>>
>
>
>
> --
> Claus Ibsen
> Apache Camel Committer
>
> Open Source Integration: http://fusesource.com
> Blog: http://davsclaus.blogspot.com/
>
--
Claus Ibsen
Apache Camel Committer
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Re: StreamCaching in Camel 1.6
Posted by Claus Ibsen <cl...@gmail.com>.
On Fri, Mar 6, 2009 at 1:46 PM, _Jens <je...@icw.de> wrote:
>
> Hi,
>
> first of all, thanks to both of you, Gert and Willem, for the quick replies.
> I saw that the jira was already fixed, great job.
>
> Does this mean that the methods for disabling stream caching will be removed
> in the future?
>
> I have tried to use "noErrorHandler" but there was a special case that used
> the default error handler at some point and I wasn't able to change this. A
> drilled down version of my scenario is shown in the test case below.
>
> It has to do with the choice and when elements in the route. They seem to
> introduce the standard error handler, as they don't inherit the configured
> one. The standard error handling then configures the route to use stream
> caching again. For some reason this only seems to happen if the JMX agent is
> disabled in the CamelContext, which I simulate by setting the system
> property. Of course, in our real scenario we simply disable it in the
> application context.
So you are saying that with/without JMX it has an influence on your route?
That is kinda odd, but there could be a gremlin.
Need to check into this.
>
> @Test
> public void testUploadFailsForBigFileWithNoErrorHandler() throws
> Exception {
> System.setProperty(JmxSystemPropertyKeys.DISABLED, "true");
> DefaultCamelContext context = new DefaultCamelContext();
>
> context.addRoutes(new RouteBuilder() {
> @Override
> public void configure() throws Exception {
> errorHandler(noErrorHandler());
>
> from("jetty:http://localhost:8989/bigfile")
>
> .choice().when(header("foo").isEqualTo("bar")).to("direct:end")
> .otherwise().end();
>
> from("direct:end")
> .process(new Processor() {
> public void process(Exchange exchange) throws
> Exception {
> InputStream inputStream =
> exchange.getIn().getBody(InputStream.class);
> while (inputStream.read() != -1) {}
> }
> });
> }
> });
>
> context.start();
>
> HttpClient client = new HttpClient();
> PostMethod method = new PostMethod("http://localhost:8989/bigfile");
> File file = new File("c:\\temp\\test.bin.ok2");
> method.setRequestEntity(new FileRequestEntity(file,
> "unknown/unknown"));
> assertEquals(200, client.executeMethod(method));
> }
>
> --
> View this message in context: http://www.nabble.com/StreamCaching-in-Camel-1.6-tp22305654p22371815.html
> Sent from the Camel - Users (activemq) mailing list archive at Nabble.com.
>
>
--
Claus Ibsen
Apache Camel Committer
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Re: StreamCaching in Camel 1.6
Posted by _Jens <je...@icw.de>.
Hi,
first of all, thanks to both of you, Gert and Willem, for the quick replies.
I saw that the jira was already fixed, great job.
Does this mean that the methods for disabling stream caching will be removed
in the future?
I have tried to use "noErrorHandler" but there was a special case that used
the default error handler at some point and I wasn't able to change this. A
drilled down version of my scenario is shown in the test case below.
It has to do with the choice and when elements in the route. They seem to
introduce the standard error handler, as they don't inherit the configured
one. The standard error handling then configures the route to use stream
caching again. For some reason this only seems to happen if the JMX agent is
disabled in the CamelContext, which I simulate by setting the system
property. Of course, in our real scenario we simply disable it in the
application context.
@Test
public void testUploadFailsForBigFileWithNoErrorHandler() throws
Exception {
System.setProperty(JmxSystemPropertyKeys.DISABLED, "true");
DefaultCamelContext context = new DefaultCamelContext();
context.addRoutes(new RouteBuilder() {
@Override
public void configure() throws Exception {
errorHandler(noErrorHandler());
from("jetty:http://localhost:8989/bigfile")
.choice().when(header("foo").isEqualTo("bar")).to("direct:end")
.otherwise().end();
from("direct:end")
.process(new Processor() {
public void process(Exchange exchange) throws
Exception {
InputStream inputStream =
exchange.getIn().getBody(InputStream.class);
while (inputStream.read() != -1) {}
}
});
}
});
context.start();
HttpClient client = new HttpClient();
PostMethod method = new PostMethod("http://localhost:8989/bigfile");
File file = new File("c:\\temp\\test.bin.ok2");
method.setRequestEntity(new FileRequestEntity(file,
"unknown/unknown"));
assertEquals(200, client.executeMethod(method));
}
--
View this message in context: http://www.nabble.com/StreamCaching-in-Camel-1.6-tp22305654p22371815.html
Sent from the Camel - Users (activemq) mailing list archive at Nabble.com.
Re: StreamCaching in Camel 1.6
Posted by "willem.jiang" <wi...@gmail.com>.
Hi
Just as Gert said.
In CXF , we will cache the big message into the File instead of the memory.
I think we could do the same thing in Camel.
I just filled a JIRA[2] for it.
[1] http://www.nabble.com/Re%3A-StreamCaching-in-Camel-1.6-to22305858.html
[2] https://issues.apache.org/activemq/browse/CAMEL-1413
Willem
_Jens wrote:
>
> Hi,
>
> I have run into a problem with stream caching in Camel 1.6. I have a route
> that reads data via streaming from an Http request. In some cases this can
> be a large amount of data, so I cannot stream the data completely into
> memory. I wrote this little test to demonstrate my problem:
>
> @Test
> public void testUploadFailsForHugeFile() throws Exception {
> CamelContext context = new DefaultCamelContext();
>
> context.addRoutes(new RouteBuilder() {
> @Override
> public void configure() throws Exception {
> noStreamCaching();
> from("jetty:http://localhost:8989/bigfile")
> .noStreamCaching()
> .process(new Processor() {
> public void process(Exchange exchange) throws
> Exception {
> InputStream inputStream =
> exchange.getIn().getBody(InputStream.class);
> while (inputStream.read() != -1) {}
> }
> });
> }
> });
>
> context.start();
>
> HttpClient client = new HttpClient();
> PostMethod method = new
> PostMethod("http://localhost:8989/bigfile");
> File file = new File("c:\\temp\\test.bin.ok2");
> method.setRequestEntity(new FileRequestEntity(file,
> "unknown/unknown"));
> assertEquals(200, client.executeMethod(method));
> }
>
> This fails with an OutOfMemoryError if test.bin.ok2 is a larger file
> because the StreamCachingInterceptor reads the input stream into memory.
> Now I understand that stream caching is turned on by default in Camel 1.6.
> However, as you can see I tried to disable it in the route builder. This
> seems not to make any difference.
>
> Now, my question is if there is some other way to disable stream caching
> or if I'm doing something wrong here.
>
> Thanks in advance
> Jens
>
--
View this message in context: http://www.nabble.com/StreamCaching-in-Camel-1.6-tp22305654p22306407.html
Sent from the Camel - Users (activemq) mailing list archive at Nabble.com.
Re: StreamCaching in Camel 1.6
Posted by Gert Vanthienen <ge...@gmail.com>.
Jens,
Stream caching is turned on automatically when you use a
DeadLetterChannel as the error handler. Because the Exchange might be
redelivered or sent to the DLQ destination, we have to cache it before
we send it into the DLC. Adding an errorHandler(noErrorHandler()); to
the configure() method should fix that for you.
Regards,
Gert Vanthienen
------------------------
Open Source SOA: http://fusesource.com
Blog: http://gertvanthienen.blogspot.com/
2009/3/3 _Jens <je...@icw.de>:
>
> Hi,
>
> I have run into a problem with stream caching in Camel 1.6. I have a route
> that reads data via streaming from an Http request. In some cases this can
> be a large amount of data, so I cannot stream the data completely into
> memory. I wrote this little test to demonstrate my problem:
>
> @Test
> public void testUploadFailsForHugeFile() throws Exception {
> CamelContext context = new DefaultCamelContext();
>
> context.addRoutes(new RouteBuilder() {
> @Override
> public void configure() throws Exception {
> noStreamCaching();
> from("jetty:http://localhost:8989/bigfile")
> .noStreamCaching()
> .process(new Processor() {
> public void process(Exchange exchange) throws
> Exception {
> InputStream inputStream =
> exchange.getIn().getBody(InputStream.class);
> while (inputStream.read() != -1) {}
> }
> });
> }
> });
>
> context.start();
>
> HttpClient client = new HttpClient();
> PostMethod method = new PostMethod("http://localhost:8989/bigfile");
> File file = new File("c:\\temp\\test.bin.ok2");
> method.setRequestEntity(new FileRequestEntity(file,
> "unknown/unknown"));
> assertEquals(200, client.executeMethod(method));
> }
>
> This fails with an OutOfMemoryError if test.bin.ok2 is a larger file because
> the StreamCachingInterceptor reads the input stream into memory. Now I
> understand that stream caching is turned on by default in Camel 1.6.
> However, as you can see I tried to disable it in the route builder. This
> seems not to make any difference.
>
> Now, my question is if there is some other way to disable stream caching or
> if I'm doing something wrong here.
>
> Thanks in advance
> Jens
> --
> View this message in context: http://www.nabble.com/StreamCaching-in-Camel-1.6-tp22305654p22305654.html
> Sent from the Camel - Users (activemq) mailing list archive at Nabble.com.
>
>