You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by "Oleg.Kasian" <o-...@yandex.ru> on 2012/10/19 09:24:53 UTC

Problems when forwarding large post with jetty

I am facing the problem when forwarding post requests with large data using
camel-jetty component. I am not sure what is the reason, stack trace is:

2012-10-19 10:04:30.991:WARN:oejs.HttpConnection:header full:
java.lang.RuntimeException: Header>6144
2012-10-19 10:04:30.992:WARN:oejs.Response:Committed before 500 null
2012-10-19 10:04:30.992:WARN:oejs.HttpConnection:/forward
java.lang.IllegalStateException: Committed
	at org.eclipse.jetty.server.Response.resetBuffer(Response.java:1059)
        ...

camel version 2.10.0
jetty version 7.5.4.v20111024
The problem first occured when running camel based app on karaf 2.2.9,
however I can reproduce it with this unit test:

import java.io.InputStream;
import java.io.UnsupportedEncodingException;
import java.util.Random;
import org.apache.camel.builder.RouteBuilder;
import org.apache.camel.test.junit4.CamelTestSupport;
import org.apache.commons.httpclient.HttpClient;
import org.apache.commons.httpclient.NameValuePair;
import org.apache.commons.httpclient.methods.PostMethod;
import org.apache.http.HttpStatus;
import org.junit.Test;


public class JettyPostBigDataTest extends CamelTestSupport {

    private static final int LENGTH = 5 * 1024;

    @Test
    public void testPostParameter() throws Exception {
        final String body = generateBody(LENGTH);
        final NameValuePair[] data = {
            new NameValuePair("param", body)
        };
        final HttpClient client = new HttpClient();
        final PostMethod post = new
PostMethod("http://localhost:5757/test");
        post.setRequestBody(data);
        client.executeMethod(post);
        final InputStream response = post.getResponseBodyAsStream();
        final String out =
context.getTypeConverter().convertTo(String.class, response);
        assertEquals("ERROR OCCURED: \n" + out, HttpStatus.SC_OK,
post.getStatusCode());
    }

    private String generateBody(final int length) throws
UnsupportedEncodingException {
        final Random rnd = new Random();
        byte[] bytes = new byte[length];
        rnd.nextBytes(bytes);
        return new String(bytes);
    }

    @Override
    protected RouteBuilder createRouteBuilder() throws Exception {
        return new RouteBuilder() {
            public void configure() {

               
from("jetty:http://localhost:5757/test").to("http://localhost:5757/forward?bridgeEndpoint=true");

                from("jetty:http://localhost:5757/forward")
                        .setBody(simple("${in.header.param[0]}"))
                        .convertBodyTo(String.class)
                        .to("mock:result");

            }
        };
    }
}



--
View this message in context: http://camel.465427.n5.nabble.com/Problems-when-forwarding-large-post-with-jetty-tp5721278.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: Problems when forwarding large post with jetty

Posted by Claus Ibsen <cl...@gmail.com>.
On Fri, Oct 19, 2012 at 10:50 AM, Oleg.Kasian <o-...@yandex.ru> wrote:
> It is actually in body, it is application/x-www-form-urlencoded. Seems like,
> when jetty component receives such request, it parses request body to
> exchange headers, passing them to endpoint. In this case, simple forwarding
> like this:
>
> <from uri="jetty://http://0.0.0.0:7777/ws/predmine/controls" />
> <to uri="http://10.1.193.197:8080/predmine/boards.xml?bridgeEndpoint=true"
> />
>
> fails on large data. This doesn't work for requests sent directly from
> browsers either.
> Thanks to Raul, I was able to overcome the problem modifying my route:
>
> <from uri="jetty://http://0.0.0.0:7777/ws/predmine/controls" />
> <removeHeaders pattern="*" />
> <setHeader headerName="content-type">
>   <simple>application/x-www-form-urlencoded</simple>
> </setHeader>
> <to uri="http://10.1.193.197:8080/predmine/boards.xml?bridgeEndpoint=true"
> />
>

Ah seems like we should add some logic to support this use-case. As
the application/x-www-form-urlencoded content-type should IMHO be
parsed straight to the message body.

Fell free to log a JIRA ticket.


>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Problems-when-forwarding-large-post-with-jetty-tp5721278p5721286.html
> Sent from the Camel - Users mailing list archive at Nabble.com.



-- 
Claus Ibsen
-----------------
Red Hat, Inc.
FuseSource is now part of Red Hat
Email: cibsen@redhat.com
Web: http://fusesource.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen

Re: Problems when forwarding large post with jetty

Posted by "Oleg.Kasian" <o-...@yandex.ru>.
It is actually in body, it is application/x-www-form-urlencoded. Seems like,
when jetty component receives such request, it parses request body to
exchange headers, passing them to endpoint. In this case, simple forwarding
like this:

<from uri="jetty://http://0.0.0.0:7777/ws/predmine/controls" />
<to uri="http://10.1.193.197:8080/predmine/boards.xml?bridgeEndpoint=true"
/>

fails on large data. This doesn't work for requests sent directly from
browsers either.
Thanks to Raul, I was able to overcome the problem modifying my route:

<from uri="jetty://http://0.0.0.0:7777/ws/predmine/controls" />
<removeHeaders pattern="*" />
<setHeader headerName="content-type">
  <simple>application/x-www-form-urlencoded</simple>
</setHeader>
<to uri="http://10.1.193.197:8080/predmine/boards.xml?bridgeEndpoint=true"
/>



--
View this message in context: http://camel.465427.n5.nabble.com/Problems-when-forwarding-large-post-with-jetty-tp5721278p5721286.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: Problems when forwarding large post with jetty

Posted by Willem jiang <wi...@gmail.com>.
Yes, it is not a good practice to send the big message in the http request header.
Why don't put it into the http request body?


-- 
Willem Jiang

Red Hat, Inc.
FuseSource is now part of Red Hat
Web: http://www.fusesource.com | http://www.redhat.com
Blog: http://willemjiang.blogspot.com (http://willemjiang.blogspot.com/) (English)
          http://jnn.javaeye.com (http://jnn.javaeye.com/) (Chinese)
Twitter: willemjiang 
Weibo: willemjiang




On Friday, October 19, 2012 at 3:34 PM, Raul Kripalani wrote:

> By default, Camel will propagate all message headers as HTTP headers to the
> client when the response is sent back.
> 
> Jetty doesn't like headers larger than 6kb - for a good reason, as it's not
> a good practice to send such large headers. All Web servers have some
> limitation in this respect I think.
> 
> In the first route, just after you send to the second one, use a
> removeHeaders() DSL to remove the 'param' header, and you should be okay.
> 
> If you want a more automatic approach, you can consider setting exclusion
> patterns in the Header Filter Strategy.
> 
> Regards.
> 
> Sent from a mobile device
> On 19 Oct 2012 09:25, "Oleg.Kasian" <o-kasian@yandex.ru (mailto:o-kasian@yandex.ru)> wrote:
> 
> > I am facing the problem when forwarding post requests with large data using
> > camel-jetty component. I am not sure what is the reason, stack trace is:
> > 
> > 2012-10-19 10:04:30.991:WARN:oejs.HttpConnection:header full:
> > java.lang.RuntimeException: Header>6144
> > 2012-10-19 10:04:30.992:WARN:oejs.Response:Committed before 500 null
> > 2012-10-19 10:04:30.992:WARN:oejs.HttpConnection:/forward
> > java.lang.IllegalStateException: Committed
> > at
> > org.eclipse.jetty.server.Response.resetBuffer(Response.java:1059)
> > ...
> > 
> > camel version 2.10.0
> > jetty version 7.5.4.v20111024
> > The problem first occured when running camel based app on karaf 2.2.9,
> > however I can reproduce it with this unit test:
> > 
> > import java.io.InputStream;
> > import java.io.UnsupportedEncodingException;
> > import java.util.Random;
> > import org.apache.camel.builder.RouteBuilder;
> > import org.apache.camel.test.junit4.CamelTestSupport;
> > import org.apache.commons.httpclient.HttpClient;
> > import org.apache.commons.httpclient.NameValuePair;
> > import org.apache.commons.httpclient.methods.PostMethod;
> > import org.apache.http.HttpStatus;
> > import org.junit.Test;
> > 
> > 
> > public class JettyPostBigDataTest extends CamelTestSupport {
> > 
> > private static final int LENGTH = 5 * 1024;
> > 
> > @Test
> > public void testPostParameter() throws Exception {
> > final String body = generateBody(LENGTH);
> > final NameValuePair[] data = {
> > new NameValuePair("param", body)
> > };
> > final HttpClient client = new HttpClient();
> > final PostMethod post = new
> > PostMethod("http://localhost:5757/test");
> > post.setRequestBody(data);
> > client.executeMethod(post);
> > final InputStream response = post.getResponseBodyAsStream();
> > final String out =
> > context.getTypeConverter().convertTo(String.class, response);
> > assertEquals("ERROR OCCURED: \n" + out, HttpStatus.SC_OK,
> > post.getStatusCode());
> > }
> > 
> > private String generateBody(final int length) throws
> > UnsupportedEncodingException {
> > final Random rnd = new Random();
> > byte[] bytes = new byte[length];
> > rnd.nextBytes(bytes);
> > return new String(bytes);
> > }
> > 
> > @Override
> > protected RouteBuilder createRouteBuilder() throws Exception {
> > return new RouteBuilder() {
> > public void configure() {
> > 
> > 
> > from("jetty:http://localhost:5757/test").to("
> > http://localhost:5757/forward?bridgeEndpoint=true");
> > 
> > from("jetty:http://localhost:5757/forward")
> > .setBody(simple("${in.header.param[0]}"))
> > .convertBodyTo(String.class)
> > .to("mock:result");
> > 
> > }
> > };
> > }
> > }
> > 
> > 
> > 
> > --
> > View this message in context:
> > http://camel.465427.n5.nabble.com/Problems-when-forwarding-large-post-with-jetty-tp5721278.html
> > Sent from the Camel - Users mailing list archive at Nabble.com (http://Nabble.com).
> 




Re: Problems when forwarding large post with jetty

Posted by Raul Kripalani <ra...@evosent.com>.
By default, Camel will propagate all message headers as HTTP headers to the
client when the response is sent back.

Jetty doesn't like headers larger than 6kb - for a good reason, as it's not
a good practice to send such large headers. All Web servers have some
limitation in this respect I think.

In the first route, just after you send to the second one, use a
removeHeaders() DSL to remove the 'param' header, and you should be okay.

If you want a more automatic approach, you can consider setting exclusion
patterns in the Header Filter Strategy.

Regards.

Sent from a mobile device
On 19 Oct 2012 09:25, "Oleg.Kasian" <o-...@yandex.ru> wrote:

> I am facing the problem when forwarding post requests with large data using
> camel-jetty component. I am not sure what is the reason, stack trace is:
>
> 2012-10-19 10:04:30.991:WARN:oejs.HttpConnection:header full:
> java.lang.RuntimeException: Header>6144
> 2012-10-19 10:04:30.992:WARN:oejs.Response:Committed before 500 null
> 2012-10-19 10:04:30.992:WARN:oejs.HttpConnection:/forward
> java.lang.IllegalStateException: Committed
>         at
> org.eclipse.jetty.server.Response.resetBuffer(Response.java:1059)
>         ...
>
> camel version 2.10.0
> jetty version 7.5.4.v20111024
> The problem first occured when running camel based app on karaf 2.2.9,
> however I can reproduce it with this unit test:
>
> import java.io.InputStream;
> import java.io.UnsupportedEncodingException;
> import java.util.Random;
> import org.apache.camel.builder.RouteBuilder;
> import org.apache.camel.test.junit4.CamelTestSupport;
> import org.apache.commons.httpclient.HttpClient;
> import org.apache.commons.httpclient.NameValuePair;
> import org.apache.commons.httpclient.methods.PostMethod;
> import org.apache.http.HttpStatus;
> import org.junit.Test;
>
>
> public class JettyPostBigDataTest extends CamelTestSupport {
>
>     private static final int LENGTH = 5 * 1024;
>
>     @Test
>     public void testPostParameter() throws Exception {
>         final String body = generateBody(LENGTH);
>         final NameValuePair[] data = {
>             new NameValuePair("param", body)
>         };
>         final HttpClient client = new HttpClient();
>         final PostMethod post = new
> PostMethod("http://localhost:5757/test");
>         post.setRequestBody(data);
>         client.executeMethod(post);
>         final InputStream response = post.getResponseBodyAsStream();
>         final String out =
> context.getTypeConverter().convertTo(String.class, response);
>         assertEquals("ERROR OCCURED: \n" + out, HttpStatus.SC_OK,
> post.getStatusCode());
>     }
>
>     private String generateBody(final int length) throws
> UnsupportedEncodingException {
>         final Random rnd = new Random();
>         byte[] bytes = new byte[length];
>         rnd.nextBytes(bytes);
>         return new String(bytes);
>     }
>
>     @Override
>     protected RouteBuilder createRouteBuilder() throws Exception {
>         return new RouteBuilder() {
>             public void configure() {
>
>
> from("jetty:http://localhost:5757/test").to("
> http://localhost:5757/forward?bridgeEndpoint=true");
>
>                 from("jetty:http://localhost:5757/forward")
>                         .setBody(simple("${in.header.param[0]}"))
>                         .convertBodyTo(String.class)
>                         .to("mock:result");
>
>             }
>         };
>     }
> }
>
>
>
> --
> View this message in context:
> http://camel.465427.n5.nabble.com/Problems-when-forwarding-large-post-with-jetty-tp5721278.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>