You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Jerry Malcolm <te...@malcolms.com> on 2019/12/08 05:08:05 UTC
ServletRequest Obj Randomly not Processing x-www-form-urlencoded
parms
I have ajax code that sends requests to TC in a REST-style process. I
send the parms url-encoded in the body. This has worked untouched
literally for years. I have some new data objects in my db that "should
be" sending the same type of requests through the same javascript
routines. But for some inexplicable reason, the HttpServletRequest
object is randomly deciding to not process the parms. When I try to
enumerate the parms, I get none. Any parm I request comes back not
found. I added some code to read the body myself (request.getReader(),
etc). When the parms are available as it normally works, the reader is
empty, which is what I would expect since it's been read by the request
obj. But when the request object tells me I have no parms, I can read
the entire url-encoded parm string from the reader, which if I
understand things, means the request object never tried to read the
stream, unless it somehow restores the stream after a read (??). But
the important point I determined is that the parms are indeed present in
the body... just not processed.
I know this has to be something I'm doing wrong. It only occurs on one
or two new 'product items' on my web app. But it is pretty consistent
on those. I'll refresh 4 or 5 times, and the next time I get the
no-parms situation. I hit refresh again, and it works again. I've
analyzed all the parms and parm values to see if anything strange sticks
out. Nothing looks out of the ordinary. It's a pretty large parm set
(~158 parms). But I don't think that's anywhere near the largest set
I've sent over the years. I've dumped all of the headers for both
success and fail. No differences. The fail still shows the correct
content type and correct content length. Just no parms. I also checked
the logs to see if any exceptions are being thrown. Nothing. And all
of my code executes as normal until it can't find necessary parms and
then it spirals.... Hard drive has tons of space. And this doesn't
sound like a heap problem since it fails consistently on 2 out of 200+
data objects and works fine on all of the other objects. Those two
objects should only differ from the others in
name/description/price/etc. Obviously, there's something else
different. I just can't figure it out.
My production env is TC 8.5.xx on AWS Linux. Just to test, I fired up
my Windows laptop TC 9.x and got the exact same symptoms.
I'm not expecting anyone to solve my problem off the top of their head
(but I won't refuse that if someone really knows what's happening....).
What I really need is just a general overview of how the request object
works regarding url-encoded body parameters (and the impl class TC
uses). Is there simply some code in the impl that checks for
application/x-www-form-urlencoded and reads the buffer, parses the
parms, and stores them in a hash? Where does that occur in the
lifecycle of the request object? If I can just find the code that does
this function in the source, perhaps I can figure out what might make it
randomly decide to ignore this specific set of parms. Or maybe I can
fire everything up in an Eclipse environment and do some debugging. But
a java source file name for the request object impl class and a bit of
functional understanding will really help me along the way.
Thx
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: ServletRequest Obj Randomly not Processing x-www-form-urlencoded
parms
Posted by Mark Thomas <ma...@apache.org>.
On 09/12/2019 00:44, Jerry Malcolm wrote:
<snip/>
> Mark, thank you so much for the info. I downloaded the TC source and
> got everything up and running in Eclipse. After getting familiar with
> the paths when it worked normally, I was able to single-step a failure.
> I added a request.getParameter( "abc" ) as the first line of the JSP.
> It steps into the RequestFacade then to the Catalina Request object. I
> see the lazy parsing you described. It checks "parametersParsed". In
> this error case, parametersParsed is already set to true, so it skips
> parsing. However this is the first time here for this request, and the
> parameters have not been parsed, and the getParameter returns null.
>
> I grepped the source tree to see if there were any other places where
> parametersParsed is being set to true. The parseParameters method is
> the only place. Being the first line in the JSP, I'm not aware of any
> other code above the JSP that would be causing the parsing. And it
> appears the only way out of the parser method is with an error condition
> or successful parse, which neither appears to happen, implying it's
> never being called in the error condition... leading to the possibility
> that parametersParsed is somehow incorrectly set to true coming in.
>
> I saw some code about recycling the object instance. I haven't dug into
> any of that code. Is there any possible way that a request object could
> be recycled without being wiped clean first?
That should not be possible.
> I see the recycle method
> where everything is cleaned out. But I guess it could somehow throw an
> exception while cleaning it out. I realize this is a long shot. But I'm
> just grasping at straws as to why the object would indicate with no
> errors that the parameters had been parsed when they hadn't.
You could out a try/catch around the recycle method but I have a hard
time trying to thing of a scenario that would throw an Exception there.
> I guess my next step is to figure out how to breakpoint when request
> objects are allocated from the pool and trace it all the way up to my
> JSP line. Can you give me a pointer to the class(es) that handle
> request object pool mgmt?
The reference chain is:
Processor -> CoyoteRequest -> Request.
Processors are pooled and allocated starting here:
https://github.com/apache/tomcat/blob/master/java/org/apache/coyote/AbstractProtocol.java#L772
The CoyoteRequest and Response object are final fields on the Processor:
https://github.com/apache/tomcat/blob/master/java/org/apache/coyote/AbstractProcessor.java#L64
The Request and Response are held as notes on the corresponding Coyote
objects:
https://github.com/apache/tomcat/blob/master/java/org/apache/catalina/connector/CoyoteAdapter.java#L303
> Also, I've seen several places in the code where it says "if debug" or
> something like that. I've set logging to FINEST for everything, but I
> believe there are debug statements that aren't showing. Is there
> another way to enable 'debug'?
That should be sufficient. Can you provide a specific example?
> I'm resisting setting up a full rebuild environment of TC where I can
> put in println statements. But I guess if that's required, I can do
> it. If you'd like to talk me out that, I'm listening....
That is often where I end up investigating issues such as this. I
usually try and turn those println statements into useful debug logging
so I need less of them next time.
Issues like this can be caused if a reference to a request or response
is retained longer than it should be. You can try setting:
-Dorg.apache.catalina.connector.RECYCLE_FACADES=true
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: ServletRequest Obj Randomly not Processing x-www-form-urlencoded
parms
Posted by Jerry Malcolm <te...@malcolms.com>.
On 12/8/2019 6:23 AM, Mark Thomas wrote:
> On 08/12/2019 12:19, Mark Thomas wrote:
>> Jerry,
>>
>> From your description, it sounds as if there is something about those
>> requests that either isn't quite right or triggers an edge vase bug in
>> Tomcat's request parsing.
>>
>> I'm happy to provide some pointers to help you debug this. I'll use
>> 9.0.x links since I assume you prefer to debug this locally.
>>
>> Your starting point for debugging should probably be this class:
>>
>> Parameter parsing is lazy. It doesn't kick in until the application
>> requests a parametyer
Mark, thank you so much for the info. I downloaded the TC source and
got everything up and running in Eclipse. After getting familiar with
the paths when it worked normally, I was able to single-step a failure.
I added a request.getParameter( "abc" ) as the first line of the JSP.
It steps into the RequestFacade then to the Catalina Request object. I
see the lazy parsing you described. It checks "parametersParsed". In
this error case, parametersParsed is already set to true, so it skips
parsing. However this is the first time here for this request, and the
parameters have not been parsed, and the getParameter returns null.
I grepped the source tree to see if there were any other places where
parametersParsed is being set to true. The parseParameters method is
the only place. Being the first line in the JSP, I'm not aware of any
other code above the JSP that would be causing the parsing. And it
appears the only way out of the parser method is with an error condition
or successful parse, which neither appears to happen, implying it's
never being called in the error condition... leading to the possibility
that parametersParsed is somehow incorrectly set to true coming in.
I saw some code about recycling the object instance. I haven't dug into
any of that code. Is there any possible way that a request object could
be recycled without being wiped clean first? I see the recycle method
where everything is cleaned out. But I guess it could somehow throw an
exception while cleaning it out. I realize this is a long shot. But I'm
just grasping at straws as to why the object would indicate with no
errors that the parameters had been parsed when they hadn't.
I guess my next step is to figure out how to breakpoint when request
objects are allocated from the pool and trace it all the way up to my
JSP line. Can you give me a pointer to the class(es) that handle
request object pool mgmt?
Also, I've seen several places in the code where it says "if debug" or
something like that. I've set logging to FINEST for everything, but I
believe there are debug statements that aren't showing. Is there
another way to enable 'debug'?
I'm resisting setting up a full rebuild environment of TC where I can
put in println statements. But I guess if that's required, I can do
it. If you'd like to talk me out that, I'm listening....
Thx again.
Jerry
> Please ignore this. I managed to hit send halfway though. A complete
> response is on the way...
>
> Mark
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: ServletRequest Obj Randomly not Processing x-www-form-urlencoded
parms
Posted by Mark Thomas <ma...@apache.org>.
On 08/12/2019 12:19, Mark Thomas wrote:
> Jerry,
>
> From your description, it sounds as if there is something about those
> requests that either isn't quite right or triggers an edge vase bug in
> Tomcat's request parsing.
>
> I'm happy to provide some pointers to help you debug this. I'll use
> 9.0.x links since I assume you prefer to debug this locally.
>
> Your starting point for debugging should probably be this class:
>
> Parameter parsing is lazy. It doesn't kick in until the application
> requests a parametyer
Please ignore this. I managed to hit send halfway though. A complete
response is on the way...
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: ServletRequest Obj Randomly not Processing x-www-form-urlencoded
parms
Posted by Mark Thomas <ma...@homeinbox.net>.
Jerry,
From your description, it sounds as if there is something about those
requests that either isn't quite right or triggers an edge vase bug in
Tomcat's request parsing.
I'm happy to provide some pointers to help you debug this. I'll use
9.0.x links since I assume you prefer to debug this locally.
Your starting point for debugging should probably be this class:
Parameter parsing is lazy. It doesn't kick in until the application
requests a parametyer
On 08/12/2019 05:08, Jerry Malcolm wrote:
> I have ajax code that sends requests to TC in a REST-style process. I
> send the parms url-encoded in the body. This has worked untouched
> literally for years. I have some new data objects in my db that "should
> be" sending the same type of requests through the same javascript
> routines. But for some inexplicable reason, the HttpServletRequest
> object is randomly deciding to not process the parms. When I try to
> enumerate the parms, I get none. Any parm I request comes back not
> found. I added some code to read the body myself (request.getReader(),
> etc). When the parms are available as it normally works, the reader is
> empty, which is what I would expect since it's been read by the request
> obj. But when the request object tells me I have no parms, I can read
> the entire url-encoded parm string from the reader, which if I
> understand things, means the request object never tried to read the
> stream, unless it somehow restores the stream after a read (??). But
> the important point I determined is that the parms are indeed present in
> the body... just not processed.
>
> I know this has to be something I'm doing wrong. It only occurs on one
> or two new 'product items' on my web app. But it is pretty consistent
> on those. I'll refresh 4 or 5 times, and the next time I get the
> no-parms situation. I hit refresh again, and it works again. I've
> analyzed all the parms and parm values to see if anything strange sticks
> out. Nothing looks out of the ordinary. It's a pretty large parm set
> (~158 parms). But I don't think that's anywhere near the largest set
> I've sent over the years. I've dumped all of the headers for both
> success and fail. No differences. The fail still shows the correct
> content type and correct content length. Just no parms. I also checked
> the logs to see if any exceptions are being thrown. Nothing. And all
> of my code executes as normal until it can't find necessary parms and
> then it spirals.... Hard drive has tons of space. And this doesn't
> sound like a heap problem since it fails consistently on 2 out of 200+
> data objects and works fine on all of the other objects. Those two
> objects should only differ from the others in
> name/description/price/etc. Obviously, there's something else
> different. I just can't figure it out.
>
> My production env is TC 8.5.xx on AWS Linux. Just to test, I fired up
> my Windows laptop TC 9.x and got the exact same symptoms.
>
> I'm not expecting anyone to solve my problem off the top of their head
> (but I won't refuse that if someone really knows what's happening....).
> What I really need is just a general overview of how the request object
> works regarding url-encoded body parameters (and the impl class TC
> uses). Is there simply some code in the impl that checks for
> application/x-www-form-urlencoded and reads the buffer, parses the
> parms, and stores them in a hash? Where does that occur in the
> lifecycle of the request object? If I can just find the code that does
> this function in the source, perhaps I can figure out what might make it
> randomly decide to ignore this specific set of parms. Or maybe I can
> fire everything up in an Eclipse environment and do some debugging. But
> a java source file name for the request object impl class and a bit of
> functional understanding will really help me along the way.
>
> Thx
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: ServletRequest Obj Randomly not Processing x-www-form-urlencoded
parms
Posted by Mark Thomas <ma...@apache.org>.
Jerry,
From your description, it sounds as if there is something about those
requests that either isn't quite right or triggers an edge case bug in
Tomcat's request parsing.
I'm happy to provide some pointers to help you debug this. I'll use
9.0.x links since I assume you prefer to debug this locally.
Your starting point for debugging should probably be this class:
https://github.com/apache/tomcat/blob/master/java/org/apache/catalina/connector/Request.java
Parameter parsing is lazy. It doesn't kick in until the application
calls one of the methods that needs the parameters to be parsed.
Parsing is then performed in this method:
https://github.com/apache/tomcat/blob/master/java/org/apache/catalina/connector/Request.java#L3147
You'll see there are various tests at the beginning of that method. Best
guess, one of them is failing.
(The code in 8.5.x is almost identical if you prefer to work with 8.5.x)
HTH,
Mark
On 08/12/2019 05:08, Jerry Malcolm wrote:
> I have ajax code that sends requests to TC in a REST-style process. I
> send the parms url-encoded in the body. This has worked untouched
> literally for years. I have some new data objects in my db that "should
> be" sending the same type of requests through the same javascript
> routines. But for some inexplicable reason, the HttpServletRequest
> object is randomly deciding to not process the parms. When I try to
> enumerate the parms, I get none. Any parm I request comes back not
> found. I added some code to read the body myself (request.getReader(),
> etc). When the parms are available as it normally works, the reader is
> empty, which is what I would expect since it's been read by the request
> obj. But when the request object tells me I have no parms, I can read
> the entire url-encoded parm string from the reader, which if I
> understand things, means the request object never tried to read the
> stream, unless it somehow restores the stream after a read (??). But
> the important point I determined is that the parms are indeed present in
> the body... just not processed.
>
> I know this has to be something I'm doing wrong. It only occurs on one
> or two new 'product items' on my web app. But it is pretty consistent
> on those. I'll refresh 4 or 5 times, and the next time I get the
> no-parms situation. I hit refresh again, and it works again. I've
> analyzed all the parms and parm values to see if anything strange sticks
> out. Nothing looks out of the ordinary. It's a pretty large parm set
> (~158 parms). But I don't think that's anywhere near the largest set
> I've sent over the years. I've dumped all of the headers for both
> success and fail. No differences. The fail still shows the correct
> content type and correct content length. Just no parms. I also checked
> the logs to see if any exceptions are being thrown. Nothing. And all
> of my code executes as normal until it can't find necessary parms and
> then it spirals.... Hard drive has tons of space. And this doesn't
> sound like a heap problem since it fails consistently on 2 out of 200+
> data objects and works fine on all of the other objects. Those two
> objects should only differ from the others in
> name/description/price/etc. Obviously, there's something else
> different. I just can't figure it out.
>
> My production env is TC 8.5.xx on AWS Linux. Just to test, I fired up
> my Windows laptop TC 9.x and got the exact same symptoms.
>
> I'm not expecting anyone to solve my problem off the top of their head
> (but I won't refuse that if someone really knows what's happening....).
> What I really need is just a general overview of how the request object
> works regarding url-encoded body parameters (and the impl class TC
> uses). Is there simply some code in the impl that checks for
> application/x-www-form-urlencoded and reads the buffer, parses the
> parms, and stores them in a hash? Where does that occur in the
> lifecycle of the request object? If I can just find the code that does
> this function in the source, perhaps I can figure out what might make it
> randomly decide to ignore this specific set of parms. Or maybe I can
> fire everything up in an Eclipse environment and do some debugging. But
> a java source file name for the request object impl class and a bit of
> functional understanding will really help me along the way.
>
> Thx
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: ServletRequest Obj Randomly not Processing x-www-form-urlencoded parms
Posted by Rémy Maucherat <re...@apache.org>.
On Tue, Dec 10, 2019 at 4:36 PM Mark Thomas <ma...@apache.org> wrote:
> On 10/12/2019 14:27, Christopher Schultz wrote:
>
> <snip/>
>
> > Would using org.apache.catalina.connector.RECYCLE_FACADES=true have
> > made this problem go away? Or would the behavior have been the same,
> > just less dangerous?
>
> I think it would have triggered some NPEs in the background thread.
>
> > I'm wondering if Tomcat could or should have another safety feature to
> > help catch this sort of thing in development. In all my development
> > environments, I have the JDBC connection pool size set to a fixed
> > maximum of 1 connection. This means that any potential deadlocks in
> > the application due to sloppy connection-management will cause pretty
> > early because we'll get pool-fetch timeouts, missing-return-connection
> > errors, etc.
> >
> > Request object reuse has a measurable positive effect on performance
> > in production,
>
> It would be worth confirming that is still the case for the Request and
> Response objects. I suspect it is but it would be good to get some
> recent, hard data.
>
It would generate more GC per request and it is measurably slower on a best
case scenario with a static file (the worst would be starting to use
charsets and char input/output, or "large" Servlet buffers - the default is
8KB but people often increase it).
Quick test: add request = null; line 305 of CoyoteAdapter. -10% with ab -k
on tomcat.gif
>
> > but in development probably doesn't matter quite so
> > much. In the same way that WebappClassLoader becomes inert when the
> > application has been stopped, perhaps we could "shut-down" request /
> > response / session objects that have been loaned to a
> > request-processor thread.
> >
> > Something like this:
>
> I'm not sure why we need this over and above the RequestFacade object.
>
Rémy
Re: ServletRequest Obj Randomly not Processing x-www-form-urlencoded
parms
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Mark,
On 12/12/19 04:38, Mark Thomas wrote:
> On 11/12/2019 19:17, Christopher Schultz wrote:
>> On 12/11/19 13:01, Mark Thomas wrote:
>
> <snip/>
>
>>> Setting RECYCLE_FACADES=true does exactly that. Continued used
>>> by the app triggers an NPE since the underlying
>>> request/response is no longer there. Hence my question.
>>
>> RequestFacade currently has a number of methods that perform
>> null-checks on the "request" member, but not all methods do this.
>> Exampl e:
>>
>> @Override public ServletContext getServletContext() { if (request
>> == null) { throw new IllegalStateException(
>> sm.getString("requestFacade.nullRequest")); }
>>
>> return request.getServletContext(); }
>>
>> Negative example:
>>
>> @Override public AsyncContext startAsync() throws
>> IllegalStateException { return request.startAsync(); }
>>
>> Should these be aligned?
>
> Probably. It will only change the exception from an NPE to ISE but
> I guess we should be consistent here.
>
>> It seems to me that if recycleFacades="false" then there really
>> isn't any reason to wrap the request (response, etc.) in a facade
>> object at all, is there?
>
> It makes it harder for a malicious app to get at the internal
> object.
That's what I thought. So, if you trust the webapp and there are no
held-reference bugs, then the facades are completely superfluous.
Right now, the only option is either to re-use facades (which is
potentially dangerous if the webapp has held-reference bugs) or
recycle them (which is wasteful, GC-wise). There is no option to NOT
use them.
I'm just wondering if there is an opportunity for simpler execution in
a "known trusted" configuration -- which is very common.
- -chris
-----BEGIN PGP SIGNATURE-----
Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/
iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl3yXBAACgkQHPApP6U8
pFj9HhAAg3UuJ93OPeDggzdTXHj1uafJUVttma50XsQqHGw9IzXjKp0y68j1c/ro
Ii2Rm+/HnGyQG4tMk0ILTkkI6b9235Y8ywk7tTCVeiZppIgcnGH29MkARKakjXyW
Q95OAOKoUQ2Bo0z7Hl8jJCi/Ri+VAyV6O98q4nVKh2sx0a99wE+Q2GNBxqH2621r
AAGg9Bm9/GOQYOx04Lr6nEwLyWQJy8MyD9rYX5lr0wmh16AFj4l4Tey8oEFEsNuW
AUMflBgxuQV5BIICQPwSmHvZ4F0Poo4qxtSbO8N5vryecIxk2xxPvNMuK5GnQ87N
PepT0i9Pi/UAqK1P7MH6djc9Kz2ps2KyGU9kbYri0EqhST0lefTiBBXPUwxyv9PM
eDOvwAnk80Fp3Jt/KG9ygVuujmmm4qLWFk2CcauZhB08XQSUGxmIqNgTNLwFut0O
Kj5JhzJ5Patr+0nohywwDLQ1Cr3DYqvbz0hCjmWIAM0OIVMbIX4aGyQpetyU/imL
b5RNLcWeMqt5uM+gtQ7aQbBhwbuZpMVWXqPV4AREpPI0ek7HJlgNPZlTHVr6KrVo
i9lAZfYvtbkLDNs2JJitYDhv4TqbvH3X3jh9lXmgzuUirrmq1R+9tP92N/MUepHE
rlwyx4DESz5CiutWZksjzJHRK4aqLmUnFB0vZ8pMfyeLXXzUcfc=
=DZCT
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: ServletRequest Obj Randomly not Processing x-www-form-urlencoded
parms
Posted by Mark Thomas <ma...@apache.org>.
On 11/12/2019 19:17, Christopher Schultz wrote:
> On 12/11/19 13:01, Mark Thomas wrote:
<snip/>
>> Setting RECYCLE_FACADES=true does exactly that. Continued used by
>> the app triggers an NPE since the underlying request/response is no
>> longer there. Hence my question.
>
> RequestFacade currently has a number of methods that perform
> null-checks on the "request" member, but not all methods do this. Exampl
> e:
>
> @Override
> public ServletContext getServletContext() {
> if (request == null) {
> throw new IllegalStateException(
> sm.getString("requestFacade.nullRequest"));
> }
>
> return request.getServletContext();
> }
>
> Negative example:
>
> @Override
> public AsyncContext startAsync() throws IllegalStateException {
> return request.startAsync();
> }
>
> Should these be aligned?
Probably. It will only change the exception from an NPE to ISE but I
guess we should be consistent here.
> It seems to me that if recycleFacades="false" then there really isn't
> any reason to wrap the request (response, etc.) in a facade object at
> all, is there?
It makes it harder for a malicious app to get at the internal object.
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: ServletRequest Obj Randomly not Processing x-www-form-urlencoded
parms
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Mark,
On 12/11/19 13:01, Mark Thomas wrote:
> On 11/12/2019 17:43, Christopher Schultz wrote:
>
> <snip/>
>
>>> I'm not sure why we need this over and above the RequestFacade
>>> object.
>>
>> The RequestFacade is intended to protect the container from the
>> application, right?
>
> It can cut both ways.
>
>> And they are (usually) re-used. If a RequestFacade object is
>> retained by the application and used later, a later
>> client-request may be confused with the one the application was
>> expectin g> In the solution I presented, the facades are one-time
>> use and will stay dead once they are killed. The application
>> can't make a mistake and read a later client-request or write to
>> a later client-response.
>
> Setting RECYCLE_FACADES=true does exactly that. Continued used by
> the app triggers an NPE since the underlying request/response is no
> longer there. Hence my question.
RequestFacade currently has a number of methods that perform
null-checks on the "request" member, but not all methods do this. Exampl
e:
@Override
public ServletContext getServletContext() {
if (request == null) {
throw new IllegalStateException(
sm.getString("requestFacade.nullRequest"));
}
return request.getServletContext();
}
Negative example:
@Override
public AsyncContext startAsync() throws IllegalStateException {
return request.startAsync();
}
Should these be aligned?
It seems to me that if recycleFacades="false" then there really isn't
any reason to wrap the request (response, etc.) in a facade object at
all, is there? Is there any opportunity to further reduce memory
footprint by just NOT creating these objects at all?
- -chris
-----BEGIN PGP SIGNATURE-----
Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/
iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl3xQK4ACgkQHPApP6U8
pFjeYg/+MBc+iKgAtAmlf6D6MzgODhoGmGwibpPxsyssQ4jcmLPGUtwF/SVR5wtD
5QI+Nk3waVO0m9AUas0dSbEZnqYXrj0/yIqg2f8IpjKhwfayHfc0XQ7h+NMffK3o
WFpgFmIHb6L1nKBz7UCdLh4aAP4+AYzVH/yaw7aHNa1etZ4BH8CFcnEuzeaezU34
X5pYvP1vKtGae5kv5TuDKPll6UeA3xBSHhj6IJI8CSSwazPnOiwOITMPftCiAQ/3
qE4wR9AcYtdG8w+szAgpSIDkwqkEWGCTDtwxaTpfb4QZoxSCAcLRI8kYoUDkwV/u
lPgV647lM619r5yeEJ/aLaH/b11mr3dgWustu1+N7xupyosCLpgaRdbsr6ZBu+WY
ITalE2BSNMfjrV9RI45SVGfnWQ8RHzbmt9d5+oOyzo434zIx9pYY7IUEk8mSQ513
eb/D4Ap5BI2M2niOrVs6RnV9WEkvHdYq0/R23xO8LsmyyuAlBS3HuNrnK15z7Yva
a2FhZSSD7guPUXQWgB78HhabfhpvYSwVL/T5oghYMe3Q13Q6NMoY/lbzb7GEmZyr
FZy1sTnPXb0ScApqHVhKkXUgmWck/x/WDtsGf7NeGZV0uNQsLrdSURxCv1kbwlND
GajyDd4stFGPlJx5ycb7ci6vZqVqNGCTWgED/jPQNkGHGbiIgU4=
=Ygte
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: ServletRequest Obj Randomly not Processing x-www-form-urlencoded
parms
Posted by Mark Thomas <ma...@apache.org>.
On 11/12/2019 17:43, Christopher Schultz wrote:
<snip/>
>> I'm not sure why we need this over and above the RequestFacade
>> object.
>
> The RequestFacade is intended to protect the container from the
> application, right?
It can cut both ways.
> And they are (usually) re-used. If a RequestFacade
> object is retained by the application and used later, a later
> client-request may be confused with the one the application was expectin
> g>
> In the solution I presented, the facades are one-time use and will
> stay dead once they are killed. The application can't make a mistake
> and read a later client-request or write to a later client-response.
Setting RECYCLE_FACADES=true does exactly that. Continued used by the
app triggers an NPE since the underlying request/response is no longer
there. Hence my question.
Mark
>
> - -chris
> -----BEGIN PGP SIGNATURE-----
> Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/
>
> iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl3xKskACgkQHPApP6U8
> pFhZyBAAtLxars/ZWmm5pRUkAg4yamIGLOflCBWE38j01NzGVmfibczkToyOkrIX
> qLRSzlLWQsX3x3559I5gD5o9KAwixBCO0M+oCCGzOAmnnmXVs7dPvTnLYJKrJMW0
> U811tR9cNpNAN+rrdLhafYe+O8gHNsbuAOwLOnePc3lt/Q1ze7/kNYLsKDnrt8GG
> lQKLE5Tv6mYr6tDnio9I9GjYnXP+G4swSvIL74o4swhzTLG5dzMAELOWtvBTBlPR
> MoZvM6T1vIizhYf8xMMOCJuyMVxHm4kbesAlgAVdaff1pCqf9UjezrM/j+g7H0m9
> 5t6bC13EB/uKKGvz6sb1zJHr3yPiX+KRdiRaL59X96oMlKEZhBlySD1fk7KldU6W
> 62N9u6l+TrrGG3aKOrIEPYLdIsMNrbPlPDirissnCygkFPMDRKNOjH9xs5CZqV1G
> aXI45uC9gUtC4wCdNnSYE+f+uosDdY7WYTWdru0GuDDod/iOOOxNfPbJAyqE8t1w
> G8KF+22rJvVxmehd78UL2LQtsYeOjLsZ2HJQAa7LwRZN+TCRhv2efJMIWncPbRuP
> GjVxnoRBrkvFSnGNoP2ZKqP0lN7Jrf470NWr4wQHD1r1P5p8Z4suilViS77gxpSK
> LQlO+bpl08BrOMY4f7+oRjeLA0FhwA8rNNc7LeTdxmud0LPN9Ro=
> =gHas
> -----END PGP SIGNATURE-----
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: ServletRequest Obj Randomly not Processing x-www-form-urlencoded
parms
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Mark,
On 12/10/19 10:36, Mark Thomas wrote:
> On 10/12/2019 14:27, Christopher Schultz wrote:
>
> <snip/>
>
>> Would using org.apache.catalina.connector.RECYCLE_FACADES=true
>> have made this problem go away? Or would the behavior have been
>> the same, just less dangerous?
>
> I think it would have triggered some NPEs in the background
> thread.
>
>> I'm wondering if Tomcat could or should have another safety
>> feature to help catch this sort of thing in development. In all
>> my development environments, I have the JDBC connection pool size
>> set to a fixed maximum of 1 connection. This means that any
>> potential deadlocks in the application due to sloppy
>> connection-management will cause pretty early because we'll get
>> pool-fetch timeouts, missing-return-connection errors, etc.
>>
>> Request object reuse has a measurable positive effect on
>> performance in production,
>
> It would be worth confirming that is still the case for the Request
> and Response objects. I suspect it is but it would be good to get
> some recent, hard data.
>
>> but in development probably doesn't matter quite so much. In the
>> same way that WebappClassLoader becomes inert when the
>> application has been stopped, perhaps we could "shut-down"
>> request / response / session objects that have been loaned to a
>> request-processor thread.
>>
>> Something like this:
>
> I'm not sure why we need this over and above the RequestFacade
> object.
The RequestFacade is intended to protect the container from the
application, right? And they are (usually) re-used. If a RequestFacade
object is retained by the application and used later, a later
client-request may be confused with the one the application was expectin
g.
In the solution I presented, the facades are one-time use and will
stay dead once they are killed. The application can't make a mistake
and read a later client-request or write to a later client-response.
- -chris
-----BEGIN PGP SIGNATURE-----
Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/
iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl3xKskACgkQHPApP6U8
pFhZyBAAtLxars/ZWmm5pRUkAg4yamIGLOflCBWE38j01NzGVmfibczkToyOkrIX
qLRSzlLWQsX3x3559I5gD5o9KAwixBCO0M+oCCGzOAmnnmXVs7dPvTnLYJKrJMW0
U811tR9cNpNAN+rrdLhafYe+O8gHNsbuAOwLOnePc3lt/Q1ze7/kNYLsKDnrt8GG
lQKLE5Tv6mYr6tDnio9I9GjYnXP+G4swSvIL74o4swhzTLG5dzMAELOWtvBTBlPR
MoZvM6T1vIizhYf8xMMOCJuyMVxHm4kbesAlgAVdaff1pCqf9UjezrM/j+g7H0m9
5t6bC13EB/uKKGvz6sb1zJHr3yPiX+KRdiRaL59X96oMlKEZhBlySD1fk7KldU6W
62N9u6l+TrrGG3aKOrIEPYLdIsMNrbPlPDirissnCygkFPMDRKNOjH9xs5CZqV1G
aXI45uC9gUtC4wCdNnSYE+f+uosDdY7WYTWdru0GuDDod/iOOOxNfPbJAyqE8t1w
G8KF+22rJvVxmehd78UL2LQtsYeOjLsZ2HJQAa7LwRZN+TCRhv2efJMIWncPbRuP
GjVxnoRBrkvFSnGNoP2ZKqP0lN7Jrf470NWr4wQHD1r1P5p8Z4suilViS77gxpSK
LQlO+bpl08BrOMY4f7+oRjeLA0FhwA8rNNc7LeTdxmud0LPN9Ro=
=gHas
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: ServletRequest Obj Randomly not Processing x-www-form-urlencoded
parms
Posted by Mark Thomas <ma...@apache.org>.
On 10/12/2019 14:27, Christopher Schultz wrote:
<snip/>
> Would using org.apache.catalina.connector.RECYCLE_FACADES=true have
> made this problem go away? Or would the behavior have been the same,
> just less dangerous?
I think it would have triggered some NPEs in the background thread.
> I'm wondering if Tomcat could or should have another safety feature to
> help catch this sort of thing in development. In all my development
> environments, I have the JDBC connection pool size set to a fixed
> maximum of 1 connection. This means that any potential deadlocks in
> the application due to sloppy connection-management will cause pretty
> early because we'll get pool-fetch timeouts, missing-return-connection
> errors, etc.
>
> Request object reuse has a measurable positive effect on performance
> in production,
It would be worth confirming that is still the case for the Request and
Response objects. I suspect it is but it would be good to get some
recent, hard data.
> but in development probably doesn't matter quite so
> much. In the same way that WebappClassLoader becomes inert when the
> application has been stopped, perhaps we could "shut-down" request /
> response / session objects that have been loaned to a
> request-processor thread.
>
> Something like this:
I'm not sure why we need this over and above the RequestFacade object.
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: ServletRequest Obj Randomly not Processing x-www-form-urlencoded
parms
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Mark,
On 12/10/19 03:26, Mark Thomas wrote:
> On 09/12/2019 19:54, Jerry Malcolm wrote:
>
> <snip/>
>
>> Thanks to all of the responders on this problem. From the
>> beginning of this problem, I was convinced that I was doing
>> something bad wrong to cause this. I think now I have finally
>> identified the 'bad thing'. Periodically the model code that runs
>> in the request determines that I need to spin off another thread
>> to do some background work. That thread needs a few parameters
>> from the request object. So I pass the request object into the
>> thread object. I realize now that doing that is a horrible thing
>> to do. I was not aware until yesterday that request objects are
>> pooled and recycled. So I figured holding onto the request
>> object a little longer before it is discarded for that thread to
>> use was not going to be a problem. Obviously, that was
>> incorrect.
>
> Yep, that'll do it.
>
>> From what I can deduce, the main request thread finishes and
>> returns. The request object recycle code runs and the object is
>> returned to the pool. However, then the async thread later gets
>> a parm and must apparently still be able to use the request
>> object even though it's back in the pool. The
>> 'parametersProcessed' gets set to true. Then the next request
>> comes in, and that request object is assigned with
>> parametersProcessed=true.
>
> I think this analysis is correct.
>
>> I'm assuming the fix is to get everything I need out of the
>> request object and pass those parm values to the thread before
>> returning instead of passing the request object itself to the
>> thread.
>
> Correct.
>
>> Does all of this sound plausible as the source for this. The
>> good news is that I'm still teachable... :-)
>
> Very plausible.
>
> Kudos for figuring this out so quickly - and for the very clear
> write up.
Would using org.apache.catalina.connector.RECYCLE_FACADES=true have
made this problem go away? Or would the behavior have been the same,
just less dangerous?
I'm wondering if Tomcat could or should have another safety feature to
help catch this sort of thing in development. In all my development
environments, I have the JDBC connection pool size set to a fixed
maximum of 1 connection. This means that any potential deadlocks in
the application due to sloppy connection-management will cause pretty
early because we'll get pool-fetch timeouts, missing-return-connection
errors, etc.
Request object reuse has a measurable positive effect on performance
in production, but in development probably doesn't matter quite so
much. In the same way that WebappClassLoader becomes inert when the
application has been stopped, perhaps we could "shut-down" request /
response / session objects that have been loaned to a
request-processor thread.
Something like this:
Request realRequest = ... // fetch pooled request object
HttpServletRequest req;
RecyclableWrapper<HttpServletRequest> wrapper;
if(developmentSafetyMode) {
wrapper = new RecyclableWrapper<HttpServletRequest>(realRequest);
req =
(HttpServletRequest)Proxy.newProxyInstance(MigrateConfiguration.class.ge
tClassLoader(),
new Class[] { HttpServletRequest.class }, wrapper);
} else {
req = realRequest;
}
// dispatch request
if(null != wrapper) {
wrapper.recycle();
}
And then the invocation handler class:
static class RecyclableWrapper<T>
implements InvocationHandler
{
private T target;
private boolean recycled;
public RecyclableWrapper(T target) {
this.target = target;
}
public void recycle() {
recycled = true;
}
@Override
public Object invoke(Object proxy, Method method, Object[] args)
throws Throwable {
if(recycled)
throw new InvocationTargetException((Throwable)null,
"Object has been recycled.");
return method.invoke(target, args);
}
}
This would catch these kinds of errors very early, and we could use a
very specific error message that is easy to find using e.g. Google. OR
just put a URL to a wiki page explaining the problem directly in the
error message.
I suppose this could even be done as a Valve (or Filter) so it's
completely optional and trivially configurable. If a Valve, we'd have
to wrap the Request instead of HttpServletRequest.
I've only shown how to wrap the request above, but of course the
response, session, and maybe even the streams should be wrapped, too,
to be completely safe.
Is anyone interested in this kind of thing being packaged with Tomcat?
- -chris
-----BEGIN PGP SIGNATURE-----
Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/
iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl3vqzkACgkQHPApP6U8
pFg2qg/+PhgRgGxELAoqv/ZHPBjpv5IutL1X+uLa52Nl3NyuIPSWTFV7SB+ORE0l
5x8DVjW7MR7OUuu4Ptg2RPVmrufWDSANfujtJ9RS/ugzAii33O+3MIY2zHAZgfMk
AKmcIQ3eGK4Ep1SByPTvD7nK3nmIlSzn7Z6d4bBeSawOFYcb1i1zs026gxfgTnnq
x98+bl52YCDi9wCZk8GrglQ1eRy036K7WvmiEAVNs9SZNIhoho1tzvJOYmZ5WT3D
OQ0xOwpecf3S84Mi5BdNppNMOlBCGSgYhN99/LtMHA+vFV61SYFQhTxXt+151o4s
9pgFlZfdUFGqoA0EBaKX8/42NKOmifeSjirsoeOhSY91x1ZMix9IX0s8dlzvKEQZ
XYytge9BUsIhcKAT9Cw4+YGXeFD3HPIDpjfQrRPlWEjPmGv9eOtuCRySdrGxEea3
njr5LWO54ISD0vqeXVY3enWkHeoDUJORBoERHBGF33duUGy/nQzgPrOTaMwOkUWK
evU7pGe/LbBUaSBqXFXWdA8aLHLqzFJ8aM+8mPnvZjBH/jQh6QOtthTcZZlOw8ZH
ndzeDGLeBXK+cfX+87AymE5VmV2RShsafs8omjX+LLz8DMry99wEkVAIBYPAMV6I
mZaH5ad4f6wuex//vcLdDIAuFDchfymtt95mJIAP+y/uBNxek2A=
=XZM9
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: ServletRequest Obj Randomly not Processing x-www-form-urlencoded
parms
Posted by Mark Thomas <ma...@apache.org>.
On 09/12/2019 19:54, Jerry Malcolm wrote:
<snip/>
> Thanks to all of the responders on this problem. From the beginning of
> this problem, I was convinced that I was doing something bad wrong to
> cause this. I think now I have finally identified the 'bad thing'.
> Periodically the model code that runs in the request determines that I
> need to spin off another thread to do some background work. That thread
> needs a few parameters from the request object. So I pass the request
> object into the thread object. I realize now that doing that is a
> horrible thing to do. I was not aware until yesterday that request
> objects are pooled and recycled. So I figured holding onto the request
> object a little longer before it is discarded for that thread to use was
> not going to be a problem. Obviously, that was incorrect.
Yep, that'll do it.
> From what I can deduce, the main request thread finishes and returns.
> The request object recycle code runs and the object is returned to the
> pool. However, then the async thread later gets a parm and must
> apparently still be able to use the request object even though it's back
> in the pool. The 'parametersProcessed' gets set to true. Then the next
> request comes in, and that request object is assigned with
> parametersProcessed=true.
I think this analysis is correct.
> I'm assuming the fix is to get everything I need out of the request
> object and pass those parm values to the thread before returning instead
> of passing the request object itself to the thread.
Correct.
> Does all of this sound plausible as the source for this. The good news
> is that I'm still teachable... :-)
Very plausible.
Kudos for figuring this out so quickly - and for the very clear write up.
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: ServletRequest Obj Randomly not Processing x-www-form-urlencoded
parms
Posted by Jerry Malcolm <te...@malcolms.com>.
On 12/9/2019 5:39 AM, Konstantin Kolinko wrote:
> вс, 8 дек. 2019 г. в 08:09, Jerry Malcolm <te...@malcolms.com>:
>> I have ajax code that sends requests to TC in a REST-style process. I
>> send the parms url-encoded in the body. This has worked untouched
>> literally for years. I have some new data objects in my db that "should
>> be" sending the same type of requests through the same javascript
>> routines. But for some inexplicable reason, the HttpServletRequest
>> object is randomly deciding to not process the parms. When I try to
>> enumerate the parms, I get none. Any parm I request comes back not
>> found. I added some code to read the body myself (request.getReader(),
>> etc). When the parms are available as it normally works, the reader is
>> empty, which is what I would expect since it's been read by the request
>> obj. But when the request object tells me I have no parms, I can read
>> the entire url-encoded parm string from the reader, which if I
>> understand things, means the request object never tried to read the
>> stream, unless it somehow restores the stream after a read (??). But
>> the important point I determined is that the parms are indeed present in
>> the body... just not processed.
>>
>> [...]
Thanks to all of the responders on this problem. From the beginning of
this problem, I was convinced that I was doing something bad wrong to
cause this. I think now I have finally identified the 'bad thing'.
Periodically the model code that runs in the request determines that I
need to spin off another thread to do some background work. That thread
needs a few parameters from the request object. So I pass the request
object into the thread object. I realize now that doing that is a
horrible thing to do. I was not aware until yesterday that request
objects are pooled and recycled. So I figured holding onto the request
object a little longer before it is discarded for that thread to use was
not going to be a problem. Obviously, that was incorrect.
From what I can deduce, the main request thread finishes and returns.
The request object recycle code runs and the object is returned to the
pool. However, then the async thread later gets a parm and must
apparently still be able to use the request object even though it's back
in the pool. The 'parametersProcessed' gets set to true. Then the next
request comes in, and that request object is assigned with
parametersProcessed=true.
I'm assuming the fix is to get everything I need out of the request
object and pass those parm values to the thread before returning instead
of passing the request object itself to the thread.
Does all of this sound plausible as the source for this. The good news
is that I'm still teachable... :-)
Jerry
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: ServletRequest Obj Randomly not Processing x-www-form-urlencoded parms
Posted by Konstantin Kolinko <kn...@gmail.com>.
вс, 8 дек. 2019 г. в 08:09, Jerry Malcolm <te...@malcolms.com>:
>
> I have ajax code that sends requests to TC in a REST-style process. I
> send the parms url-encoded in the body. This has worked untouched
> literally for years. I have some new data objects in my db that "should
> be" sending the same type of requests through the same javascript
> routines. But for some inexplicable reason, the HttpServletRequest
> object is randomly deciding to not process the parms. When I try to
> enumerate the parms, I get none. Any parm I request comes back not
> found. I added some code to read the body myself (request.getReader(),
> etc). When the parms are available as it normally works, the reader is
> empty, which is what I would expect since it's been read by the request
> obj. But when the request object tells me I have no parms, I can read
> the entire url-encoded parm string from the reader, which if I
> understand things, means the request object never tried to read the
> stream, unless it somehow restores the stream after a read (??). But
> the important point I determined is that the parms are indeed present in
> the body... just not processed.
>
> [...]
I usually have the following in the pattern of AccessLogValve in my
configurations:
[%{org.apache.catalina.parameter_parse_failed}r
%{org.apache.catalina.parameter_parse_failed_reason}r]
Those request attributes are set in Tomcat whenever a problem is
encountered by parameter parsing, e.g. an IOException if a client
aborts the request. (The methods to process parameters in Servlet API
do not have a way to report any errors). Those attributes can be used
by org.apache.catalina.filters.FailedRequestFilter
http://tomcat.apache.org/tomcat-9.0-doc/config/filter.html#Failed_Request_Filter
You may look where those attributes are set in the source code.
Mark wrote:
> Issues like this can be caused if a reference to a request or response
> is retained longer than it should be. You can try setting:
> -Dorg.apache.catalina.connector.RECYCLE_FACADES=true
+1.
https://cwiki.apache.org/confluence/x/yColBg
https://cwiki.apache.org/confluence/display/TOMCAT/Troubleshooting+and+Diagnostics#TroubleshootingandDiagnostics-CommonTroubleshootingScenario
> I fired up my Windows laptop TC 9.x and got the exact same symptoms.
You may also try with Tomcat 9.0.30 - release candidate is available
and is currently being voted - see dev@ mailing list.
Best regards,
Konstantin Kolinko
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org