You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Manuel Dominguez Sarmiento <ma...@renxo.com> on 2020/03/28 14:00:56 UTC
Re: HttpServletRequest.getRemoteAddr() sometimes returns NULL on
Tomcat 9.0.30 and HTTP/2 secure requests
Hi Mark, we're now on the latest 9.0.33 release and we still see this
issue intermittently in our logs. Only on HTTP/2 secure requests.
Please see the attached access logs (these represent all the cases for
one whole day in one single high-volume server).
Some of the following request fields are NULL (or -1) in these examples:
- remoteAddr
- remotePort
- serverPort
- requestURI
- User-Agent
Some requests are missing some of the fields, some of the requests are
missing others. What is particularly interesting is that the errors are
clustered around particular timestamps, pointing to some likely issue
regarding object sharing across several requests.
Please note that this is not just an issue at the AccessLogValve level.
These fields contain invalid data while the request is being processed,
so that is causing unexpected exceptions in our production code. The
cases are few and isolated, but still this should be looked into.
Any thoughts?
*Manuel Dominguez Sarmiento*
On 05/02/2020 14:12, Manuel Dominguez Sarmiento wrote:
> Our filter is not doing anything fancy (and it has always worked
> correctly before we ran into this bug). In pseudo-code:
>
> public doFilter(request, response) {
>
> String ip = request.getRemoteAddr();
> boolean isProxy = isProxy(ip);
> if (isProxy) {
> String unwrappedIP = unwrapXForwardedFor(request);
> chain.doFilter(new
> MobileProxyHidingServletRequestWrapper(request, unwrappedIP), response);
> } else {
> chain.doFilter(request, response);
> }
> }
>
> All that MobileProxyHidingServletRequestWrapper is override
> getRemoteAddr() returning unwrappedIP instead of delegating to the
> actual request, while unwrapXForwardedFor() does what the name
> suggests, which is processing X-Forwarded-For to obtain the
> originating IP before it hit the detected proxy.
>
> *Manuel Dominguez Sarmiento*
>
> On 05/02/2020 10:28, Mark Thomas wrote:
>> On 04/02/2020 22:27, Manuel Dominguez Sarmiento wrote:
>>> We are getting the NPEs in a top-of-the-chain servlet filter which
>>> decorates HttpServletRequest.getRemoteAddr() before actual servlet
>>> processing. Only on HTTP/2 and in a very small number of cases. Perhaps
>>> we should test 9.0.31 and see what happens. When is this new version due
>>> for release?
>> I'm just working through back-porting some changes and then I'll be
>> starting the release process. It 9.0.31 should be available towards the
>> beginning of next week.
>>
>> Can you expand on what your filter is doing? When is the call made to
>> HttpServletRequest.getRemoteAddr() on the original request?
>>
>> Mark
>>
Re: HttpServletRequest.getRemoteAddr() sometimes returns NULL on
Tomcat 9.0.30 and HTTP/2 secure requests
Posted by Mark Thomas <ma...@apache.org>.
On 31/03/2020 14:36, Manuel Dominguez Sarmiento wrote:
> Thanks, so we would be able to log this with AccessLogValve using the
> following patterns, right?
> %{org.apache.coyote.connectionID}r
> %{org.apache.coyote.streamID}r
Correct.
Mark
>
> *Manuel Dominguez Sarmiento*
>
> On 31/03/2020 10:28, Mark Thomas wrote:
>> On 29/03/2020 16:16, Mark Thomas wrote:
>>> On 29/03/2020 15:58, Manuel Dominguez Sarmiento wrote:
>>>>> Do the streams with errors all share the same connection ID? I suspect
>>>>> an error on one stream is triggering the closure of the connection and
>>>>> what you are seeing is the fallout of closing the connection.
>>>> That makes sense and is kind of what we're suspecting. How can we check
>>>> the connection/stream id? Can this be logged by AccessLogValve?
>>> Not easily at the moment. That looks like something we should fix. I'll
>>> see what I can do for 9.0.34.
>> From 8.5.54, 9.0.34 and 10.0.0-M4 the following request attributes will
>> be available for HTTP/2
>>
>> org.apache.coyote.connectionID
>> org.apache.coyote.streamID
>>
>> Mark
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: HttpServletRequest.getRemoteAddr() sometimes returns NULL on
Tomcat 9.0.30 and HTTP/2 secure requests
Posted by Manuel Dominguez Sarmiento <ma...@renxo.com>.
Thanks, so we would be able to log this with AccessLogValve using the
following patterns, right?
%{org.apache.coyote.connectionID}r
%{org.apache.coyote.streamID}r
*Manuel Dominguez Sarmiento*
On 31/03/2020 10:28, Mark Thomas wrote:
> On 29/03/2020 16:16, Mark Thomas wrote:
>> On 29/03/2020 15:58, Manuel Dominguez Sarmiento wrote:
>>>> Do the streams with errors all share the same connection ID? I suspect
>>>> an error on one stream is triggering the closure of the connection and
>>>> what you are seeing is the fallout of closing the connection.
>>> That makes sense and is kind of what we're suspecting. How can we check
>>> the connection/stream id? Can this be logged by AccessLogValve?
>> Not easily at the moment. That looks like something we should fix. I'll
>> see what I can do for 9.0.34.
> From 8.5.54, 9.0.34 and 10.0.0-M4 the following request attributes will
> be available for HTTP/2
>
> org.apache.coyote.connectionID
> org.apache.coyote.streamID
>
> Mark
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
Re: HttpServletRequest.getRemoteAddr() sometimes returns NULL on
Tomcat 9.0.30 and HTTP/2 secure requests
Posted by Mark Thomas <ma...@apache.org>.
On 29/03/2020 16:16, Mark Thomas wrote:
> On 29/03/2020 15:58, Manuel Dominguez Sarmiento wrote:
>>
>>> Do the streams with errors all share the same connection ID? I suspect
>>> an error on one stream is triggering the closure of the connection and
>>> what you are seeing is the fallout of closing the connection.
>> That makes sense and is kind of what we're suspecting. How can we check
>> the connection/stream id? Can this be logged by AccessLogValve?
>
> Not easily at the moment. That looks like something we should fix. I'll
> see what I can do for 9.0.34.
From 8.5.54, 9.0.34 and 10.0.0-M4 the following request attributes will
be available for HTTP/2
org.apache.coyote.connectionID
org.apache.coyote.streamID
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: HttpServletRequest.getRemoteAddr() sometimes returns NULL on
Tomcat 9.0.30 and HTTP/2 secure requests
Posted by Mark Thomas <ma...@apache.org>.
On 29/03/2020 15:58, Manuel Dominguez Sarmiento wrote:
>
>> Do the streams with errors all share the same connection ID? I suspect
>> an error on one stream is triggering the closure of the connection and
>> what you are seeing is the fallout of closing the connection.
> That makes sense and is kind of what we're suspecting. How can we check
> the connection/stream id? Can this be logged by AccessLogValve?
Not easily at the moment. That looks like something we should fix. I'll
see what I can do for 9.0.34.
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: HttpServletRequest.getRemoteAddr() sometimes returns NULL on
Tomcat 9.0.30 and HTTP/2 secure requests
Posted by Manuel Dominguez Sarmiento <ma...@renxo.com>.
> Do the streams with errors all share the same connection ID? I suspect
> an error on one stream is triggering the closure of the connection and
> what you are seeing is the fallout of closing the connection.
That makes sense and is kind of what we're suspecting. How can we check
the connection/stream id? Can this be logged by AccessLogValve?
*Manuel Dominguez Sarmiento
*
Re: HttpServletRequest.getRemoteAddr() sometimes returns NULL on
Tomcat 9.0.30 and HTTP/2 secure requests
Posted by Mark Thomas <ma...@apache.org>.
On 28/03/2020 14:00, Manuel Dominguez Sarmiento wrote:
> Hi Mark, we're now on the latest 9.0.33 release and we still see this
> issue intermittently in our logs. Only on HTTP/2 secure requests.
>
> Please see the attached access logs (these represent all the cases for
> one whole day in one single high-volume server).
> Some of the following request fields are NULL (or -1) in these examples:
> - remoteAddr
> - remotePort
> - serverPort
> - requestURI
> - User-Agent
>
> Some requests are missing some of the fields, some of the requests are
> missing others. What is particularly interesting is that the errors are
> clustered around particular timestamps, pointing to some likely issue
> regarding object sharing across several requests.
Do the streams with errors all share the same connection ID? I suspect
an error on one stream is triggering the closure of the connection and
what you are seeing is the fallout of closing the connection.
Mark
>
> Please note that this is not just an issue at the AccessLogValve level.
> These fields contain invalid data while the request is being processed,
> so that is causing unexpected exceptions in our production code. The
> cases are few and isolated, but still this should be looked into.
>
> Any thoughts?
>
> *Manuel Dominguez Sarmiento*
>
> On 05/02/2020 14:12, Manuel Dominguez Sarmiento wrote:
>> Our filter is not doing anything fancy (and it has always worked
>> correctly before we ran into this bug). In pseudo-code:
>>
>> public doFilter(request, response) {
>>
>> String ip = request.getRemoteAddr();
>> boolean isProxy = isProxy(ip);
>> if (isProxy) {
>> String unwrappedIP = unwrapXForwardedFor(request);
>> chain.doFilter(new
>> MobileProxyHidingServletRequestWrapper(request, unwrappedIP), response);
>> } else {
>> chain.doFilter(request, response);
>> }
>> }
>>
>> All that MobileProxyHidingServletRequestWrapper is override
>> getRemoteAddr() returning unwrappedIP instead of delegating to the
>> actual request, while unwrapXForwardedFor() does what the name
>> suggests, which is processing X-Forwarded-For to obtain the
>> originating IP before it hit the detected proxy.
>>
>> *Manuel Dominguez Sarmiento*
>>
>> On 05/02/2020 10:28, Mark Thomas wrote:
>>> On 04/02/2020 22:27, Manuel Dominguez Sarmiento wrote:
>>>> We are getting the NPEs in a top-of-the-chain servlet filter which
>>>> decorates HttpServletRequest.getRemoteAddr() before actual servlet
>>>> processing. Only on HTTP/2 and in a very small number of cases. Perhaps
>>>> we should test 9.0.31 and see what happens. When is this new version due
>>>> for release?
>>> I'm just working through back-porting some changes and then I'll be
>>> starting the release process. It 9.0.31 should be available towards the
>>> beginning of next week.
>>>
>>> Can you expand on what your filter is doing? When is the call made to
>>> HttpServletRequest.getRemoteAddr() on the original request?
>>>
>>> Mark
>>>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org