You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Kohei Nozaki <ko...@nozaki.me> on 2018/12/16 03:01:46 UTC
Re: Tomcat occasionally returns a response without HTTP headers
Hello,
It turned out that the "sjsxp" library which JAX-WS RI v2.1.3 uses makes Tomcat behave this way. I tried a different version of JAX-WS RI (v2.1.7) which doesn't use the "sjsxp" library anymore and it solved the issue.
A very similar issue posted on the Metro mailing list: http://metro.1045641.n5.nabble.com/JAX-WS-RI-2-1-5-returning-malformed-response-tp1063518.html
It's surprising that a bug of a framework which is built on the Servlet API can make such an issue happen, but anyway thank you very much everyone who helped me out.
Regards,
Kohei
> On Dec 4, 2018, at 3:39, Christopher Schultz <ch...@christopherschultz.net> wrote:
>
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> Kohei,
>
> On 12/1/18 10:06, Kohei Nozaki wrote:
>> Hello Konstantin, thanks for sharing the valuable information.
>>
>>> On Dec 1, 2018, at 19:00, Konstantin Kolinko
>>> <kn...@gmail.com> wrote:
>>>
>>>> * Our downstream Nginx instance (The client of our Tomcat
>>>> instance) recorded the error "upstream sent no valid HTTP/1.0
>>>> header while reading response header from upstream" at that
>>>> time and the error makes perfect sense concerning the response
>>>> which has neither HTTP status line nor HTTP headers.
>>>
>>> 1. See the official FAQ / Troubleshoting page:
>>> https://wiki.apache.org/tomcat/FAQ/Troubleshooting_and_Diagnostics
>>>
>>>
>>>
> Especially pay attention to
>>> 1) configuring an access log 2) setting system property
>>> org.apache.catalina.connector.RECYCLE_FACADES=true
>>
>> I've investigated "RECYCLE_FACADES" and understand that an
>> improperly implemented application which keeps a reference to
>> Request or Response objects outside of their lifecycle can make
>> such an issue like mine happen (please correct me if I'm wrong..).
>>
>>
>> But I still don't quite understand what does "RECYCLE_FACADES=true"
>> do. The Wiki page says "This makes it easier to spot illegal access
>> when it happens, instead of waiting until side effects of such
>> access become visible" but how does it make easier? Does this
>> property make Tomcat produce an Exception or make Tomcat produce
>> some warning message to a log file or something when such accesses
>> happen, for example?
>
> Tomcat usually handles requests something like this. Imagine a
> single-threaded server where Tomcat only accepts a single connection
> at a time (just to simplify the code to the point where it fits into a
> ML post).
>
> Many of these methods are made-up. There is no
> TomcatHttpServletRequest class or a .setRequestLine method in it
> (though there are ... siilar concepts in there, way down deep). The
> point is how the objects are used, or rather *re* used.
>
> HttpServletRequest request = new TomcatHttpServletRequest();
> HttpServletResponse response = new TomcatHttpServletResponse();
>
> Connection conn = null;
>
> while(null != (conn = socket.acceptConnection()) {
> request.setRequestLine(conn.getRequestLine());
> request.setInputStream(conn.getInputStream());
> response.setOutputStream(conn.getOutputStream());
>
> Servlet servlet = getServletForRequest(request);
> if(null == servlet)
> servlet = defaultServlet;
>
> servlet.service(request, response);
>
> request.reset();
> response.reset();
> }
>
> In "real" Tomcat, each Connection object holds its own Request and
> Response objects ad manages them in a similar way, and of course,
> Tomcat can accept multiple simultaneous connections -- including
> multiple requests over a single connection -- simultaneously -- in the
> case of HTTP/2.
>
> If you enable the RECYCLE_FACADES in Tomcat, the code changes to
> behave like this:
>
> Connection conn = null;
>
> while(null != (conn = socket.acceptConnection()) {
> HttpServletRequest request = new TomcatHttpServletRequest();
> HttpServletResponse response = new TomcatHttpServletResponse();
> request.setRequestLine(conn.getRequestLine());
> request.setInputStream(conn.getInputStream());
> response.setOutputStream(conn.getOutputStream());
>
> Servlet servlet = getServletForRequest(request);
> if(null == servlet)
> servlet = defaultServlet;
>
> servlet.service(request, response);
>
> request.dispose();
> response.dispose();
> }
>
> Note how the request and response objects are no longer re-used across
> requests. This represents a trade-off between security/stability
> (always getting a fresh object) versus performance (less
> garbage-collection for a given request). An application can do things
> to the request or response that can break the way the server works, so
> an untrusted application should always be run with RECYCLE_FACADES set
> to ON.
>
> If the application keeps a reference to a request or response object
> after the request has completed (as defined by returning from
> servlet.service()), then Bad Things can happen. If you write to that
> response's OutputStream, for example, you might write into the
> response of *another request* that is being handled after your code
> should be finished.
>
>>
>> p.s. Speaking of the possibility of an improper implementation,
>> I've found some discussions which seem to have some similarity to
>> my case like the following:
>>
>> *
>> http://tomcat.10.x6.nabble.com/NullPointerException-in-MimeHeaders-td2
> 054107.html
>> : I've seen some similar Exceptions at the "MimeHeaders" class
>> which were recorded in my Tomcat's log too
>>
>> *
>> http://tomcat.10.x6.nabble.com/Tomcat-occasionally-duplicating-respons
> es-td5034710.html
>> : A case where a bug in a Filter make Tomcat produce an invalid
>> HTTP response
>>
>> Those discussions made me think of this possibility where my app
>> might be improperly implemented. I'm going to check the application
>> code from this perspective on Monday.
>
> When you use RECYCLE_FACADES, your application will only retain a
> reference to a useless object.... one which will likely NPE at some
> point when your code calls it. This makes the application safer
> (because one misbehaving servlet won't damage requests/responses
> currently in-use by subsequent requests) and also helps you find the
> error in your own code (because the problem will usually manifest
> itself as an earlier NPE instead of some "weirdness" which cannot
> really be readily explained because of the complexities of how objects
> are re-used, etc.).
>
> Hope that helps,
> - -chris
> -----BEGIN PGP SIGNATURE-----
> Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ <https://www.enigmail.net/>
>
> iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAlwFeG4ACgkQHPApP6U8
> pFgxWxAAmG1UgpXnQUf2WdI6KQrgMa+Xx6nCkrr5XU/QK2EHvGO1jAE6Oc/qBmh4
> ak25KzQJbC25BDCar/JdFuUlxAVwt2qht7M3RoCVqKm/BoZuamnnVXO5fq8a15cW
> NWMXFm9Unmh2wJGI12U8ZuuaPwte0rs4MSn+84gIzI81iaPmQB2TD6ohrl4mtJAT
> LsZQQks0IJKpu6/kpx2z0DPcqRTIynRIeegBzkvUjXOHFzu5wF99ZnS2etsTEbr6
> bqKmEiSHSp9OjfiwggWMiHmnIPuZNwXziNj8cPpMxTWGLwMNXP1LUe7JSQ+UTKj4
> tF5HYZovYW95h9vRHV/SkiJULmzi2nR/TGCe6noz0cNYon3y7kQJdG3i4GPaQ+bs
> +aww7homCjIceWREQbUKIp+/WUNXmtN8f8EwRjq94Av73CTi5inT+gv377D0Y6fG
> ui/p2vxkENsRDn/E6khTywxYjxOTruT0cmeYJ4Ee45k4EqMTb7142fxwblq1sgUy
> Of3ziwYVW7i6gz5zJaxv6oHxYCKVlHwj8aaqaETon5clGE7yFrRyAsilsLhu86OO
> 0ZwFLGUw7JSsiNs/KPODhmsY7XhNevmUu7ejhyfPe/TMle+TQyUmjTrdnjc+yX3C
> NwNUnZQUoQhkCseMaC2tp+6I1fo4/iFBLYdfCTrZpffQy4L0Okg=
> =uLmM
> -----END PGP SIGNATURE-----
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org <ma...@tomcat.apache.org>
> For additional commands, e-mail: users-help@tomcat.apache.org <ma...@tomcat.apache.org>
Re: Tomcat occasionally returns a response without HTTP headers
Posted by Mark Thomas <ma...@apache.org>.
On 16/12/2018 03:01, Kohei Nozaki wrote:
> Hello,
>
> It turned out that the "sjsxp" library which JAX-WS RI v2.1.3 uses makes Tomcat behave this way. I tried a different version of JAX-WS RI (v2.1.7) which doesn't use the "sjsxp" library anymore and it solved the issue.
Thanks for reporting back when you found a solution.
> A very similar issue posted on the Metro mailing list: http://metro.1045641.n5.nabble.com/JAX-WS-RI-2-1-5-returning-malformed-response-tp1063518.html
>
> It's surprising that a bug of a framework which is built on the Servlet API can make such an issue happen, but anyway thank you very much everyone who helped me out.
I'm guessing it held onto a reference and re-used it when it shouldn't.
That can cause all sorts of chaos.
Mark
>
> Regards,
> Kohei
>
>> On Dec 4, 2018, at 3:39, Christopher Schultz <ch...@christopherschultz.net> wrote:
>>
> Kohei,
>
> On 12/1/18 10:06, Kohei Nozaki wrote:
>>>> Hello Konstantin, thanks for sharing the valuable information.
>>>>
>>>>> On Dec 1, 2018, at 19:00, Konstantin Kolinko
>>>>> <kn...@gmail.com> wrote:
>>>>>
>>>>>> * Our downstream Nginx instance (The client of our Tomcat
>>>>>> instance) recorded the error "upstream sent no valid HTTP/1.0
>>>>>> header while reading response header from upstream" at that
>>>>>> time and the error makes perfect sense concerning the response
>>>>>> which has neither HTTP status line nor HTTP headers.
>>>>>
>>>>> 1. See the official FAQ / Troubleshoting page:
>>>>> https://wiki.apache.org/tomcat/FAQ/Troubleshooting_and_Diagnostics
>>>>>
>>>>>
>>>>>
> Especially pay attention to
>>>>> 1) configuring an access log 2) setting system property
>>>>> org.apache.catalina.connector.RECYCLE_FACADES=true
>>>>
>>>> I've investigated "RECYCLE_FACADES" and understand that an
>>>> improperly implemented application which keeps a reference to
>>>> Request or Response objects outside of their lifecycle can make
>>>> such an issue like mine happen (please correct me if I'm wrong..).
>>>>
>>>>
>>>> But I still don't quite understand what does "RECYCLE_FACADES=true"
>>>> do. The Wiki page says "This makes it easier to spot illegal access
>>>> when it happens, instead of waiting until side effects of such
>>>> access become visible" but how does it make easier? Does this
>>>> property make Tomcat produce an Exception or make Tomcat produce
>>>> some warning message to a log file or something when such accesses
>>>> happen, for example?
>
> Tomcat usually handles requests something like this. Imagine a
> single-threaded server where Tomcat only accepts a single connection
> at a time (just to simplify the code to the point where it fits into a
> ML post).
>
> Many of these methods are made-up. There is no
> TomcatHttpServletRequest class or a .setRequestLine method in it
> (though there are ... siilar concepts in there, way down deep). The
> point is how the objects are used, or rather *re* used.
>
> HttpServletRequest request = new TomcatHttpServletRequest();
> HttpServletResponse response = new TomcatHttpServletResponse();
>
> Connection conn = null;
>
> while(null != (conn = socket.acceptConnection()) {
> request.setRequestLine(conn.getRequestLine());
> request.setInputStream(conn.getInputStream());
> response.setOutputStream(conn.getOutputStream());
>
> Servlet servlet = getServletForRequest(request);
> if(null == servlet)
> servlet = defaultServlet;
>
> servlet.service(request, response);
>
> request.reset();
> response.reset();
> }
>
> In "real" Tomcat, each Connection object holds its own Request and
> Response objects ad manages them in a similar way, and of course,
> Tomcat can accept multiple simultaneous connections -- including
> multiple requests over a single connection -- simultaneously -- in the
> case of HTTP/2.
>
> If you enable the RECYCLE_FACADES in Tomcat, the code changes to
> behave like this:
>
> Connection conn = null;
>
> while(null != (conn = socket.acceptConnection()) {
> HttpServletRequest request = new TomcatHttpServletRequest();
> HttpServletResponse response = new TomcatHttpServletResponse();
> request.setRequestLine(conn.getRequestLine());
> request.setInputStream(conn.getInputStream());
> response.setOutputStream(conn.getOutputStream());
>
> Servlet servlet = getServletForRequest(request);
> if(null == servlet)
> servlet = defaultServlet;
>
> servlet.service(request, response);
>
> request.dispose();
> response.dispose();
> }
>
> Note how the request and response objects are no longer re-used across
> requests. This represents a trade-off between security/stability
> (always getting a fresh object) versus performance (less
> garbage-collection for a given request). An application can do things
> to the request or response that can break the way the server works, so
> an untrusted application should always be run with RECYCLE_FACADES set
> to ON.
>
> If the application keeps a reference to a request or response object
> after the request has completed (as defined by returning from
> servlet.service()), then Bad Things can happen. If you write to that
> response's OutputStream, for example, you might write into the
> response of *another request* that is being handled after your code
> should be finished.
>
>>>>
>>>> p.s. Speaking of the possibility of an improper implementation,
>>>> I've found some discussions which seem to have some similarity to
>>>> my case like the following:
>>>>
>>>> *
>>>> http://tomcat.10.x6.nabble.com/NullPointerException-in-MimeHeaders-td2
> 054107.html
>>>> : I've seen some similar Exceptions at the "MimeHeaders" class
>>>> which were recorded in my Tomcat's log too
>>>>
>>>> *
>>>> http://tomcat.10.x6.nabble.com/Tomcat-occasionally-duplicating-respons
> es-td5034710.html
>>>> : A case where a bug in a Filter make Tomcat produce an invalid
>>>> HTTP response
>>>>
>>>> Those discussions made me think of this possibility where my app
>>>> might be improperly implemented. I'm going to check the application
>>>> code from this perspective on Monday.
>
> When you use RECYCLE_FACADES, your application will only retain a
> reference to a useless object.... one which will likely NPE at some
> point when your code calls it. This makes the application safer
> (because one misbehaving servlet won't damage requests/responses
> currently in-use by subsequent requests) and also helps you find the
> error in your own code (because the problem will usually manifest
> itself as an earlier NPE instead of some "weirdness" which cannot
> really be readily explained because of the complexities of how objects
> are re-used, etc.).
>
> Hope that helps,
> -chris
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org <ma...@tomcat.apache.org>
>> For additional commands, e-mail: users-help@tomcat.apache.org <ma...@tomcat.apache.org>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org