You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Mark Thomas <ma...@apache.org> on 2023/03/06 16:33:49 UTC
Re: health check return 404 after upgrade from 70 to tomcat 9.0.71
On 25/02/2023 17:57, Mark Thomas wrote:
>
>
> On 25/02/2023 15:47, Rui wrote:
>> Hi
>> recently upgraded tomcat to 9.0.71 from 9.0.70
>> but saw 404 in our EKS cluster(with istio installed)
>>
>> Received [GET /actuator HTTP/1.1
>> Host: xxxxx:8079
>> User-Agent: kube-probe/1.23+
>> Accept: */*
>> Connection: close
>> Accept-Encoding: gzip
>> ]
>> Incoming request /health with originalRemoteAddr xxxx
>>
>> in 9.0.70 I can see the below following message but not in 9.0.71
>> o.a.c.authenticator.AuthenticatorBase : Security checking request GET
>> /health
>>
>> seems the processing has stopped somewhere and the pod health check
>> didn't
>> pass.
>>
>> I also noticed there was also a 404 issue but don't know if it is
>> relevant.
>> https://lists.apache.org/thread/gr814rmrlbk9rrqxqjrh4p3x0bfvv1g9
>>
>> I have tested it locally with curl or jmeter, but can't reproduce
>> the problem.
>>
>> but when I step by step debug the spring app, I found the undecodedURI
>> type
>> is T_STR
>> in CoyoteAdapter.java(in 70 it supposed to be T_BYTES), then
>> decodedURI is
>> uninitialized and the uri can't find the mapping in internalMap of
>> Mapper.java, which will cause 404(guess it thinks the uri is malformed)
>>
>> MessageBytes decodedURI = req.decodedURI();
>>
>> if (undecodedURI.getType() == MessageBytes.T_BYTES) {
>> // Copy the raw URI to the decodedURI
>> decodedURI.duplicate(undecodedURI);
>>
>>
>> 404 example: (when I debug step by step to check unndecodedURI)
>>
>> curl http://localhost:8080/actuator
>>
>> <!doctype html><html lang="en"><head><title>HTTP Status 404 – Not
>> Found</title><style type="text/css">body
>> {font-family:Tahoma,Arial,sans-serif;} h1, h2, h3, b
>> {color:white;background-color:#525D76;} h1 {font-size:22px;} h2
>> {font-size:16px;} h3 {font-size:14px;} p {font-size:12px;} a
>> {color:black;}
>> .line
>> {height:1px;background-color:#525D76;border:none;}</style></head><body><h1>HTTP
>> Status 404 – Not Found</h1></body></html>%
>>
>>
>> However, it runs well without breakpoint! I am quite confused...my
>> intellij
>> has problem?
>> but anyway, summary:
>> 9.0.69 and 70 are good in the AWS/EKS(kubernetes) cluster with istio
>> 9.0.71 and 72 return 404 when health check by the EKS cluster.
>
> Looks like an instance of:
> https://bz.apache.org/bugzilla/show_bug.cgi?id=66488
If not that then maybe
https://bz.apache.org/bugzilla/show_bug.cgi?id=66512
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: health check return 404 after upgrade from 70 to tomcat 9.0.71
Posted by Rui <wi...@gmail.com>.
Hi user group:
just found open-telemetry has fixing for it:
https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/instrumentation/tomcat/tomcat-common/javaagent/src/main/java/io/opentelemetry/javaagent/instrumentation/tomcat/common/TomcatHelper.java#L71
```
static String messageBytesToString(MessageBytes messageBytes) {
// on tomcat 10.1.0 MessageBytes.toString() has a side effect. Calling
it caches the string
// value and changes type of the MessageBytes from T_BYTES to T_STR
which breaks request
// processing in CoyoteAdapter.postParseRequest when it is called on
MessageBytes from
// request.requestURI().
if (messageBytes.getType() == MessageBytes.T_BYTES) {
return messageBytes.getByteChunk().toString();
}
return messageBytes.toString();
}
}
```
so I will update our telemetry-agent first
thanks!
Zhou Rui
On Sat, 18 Mar 2023 at 11:33, Rui <wi...@gmail.com> wrote:
> Hi Mark
>
> I can't find any Valve setting in the spring boot application(search by
> valve), would you mind sharing more info?
>
> Regarding the mb state and debugging, yes aware that my local intellij
> debugger could change the mb state, so in the the debug lib I call
> mb.getType() only
>
> I got more interesting logs today:
>
> step 1. a stack trace before it hits CoyoteAdapter.java#L677
> <https://github.com/apache/tomcat/blob/9.0.71/java/org/apache/catalina/connector/CoyoteAdapter.java#L677>
>
> ``` I type manually, can't copy it
> mb.toString()
>
> io.opentelemetry.javaagent.instrumentation.tomcat.common.TomcatHttpAttributesGetter.target(TomcatHttpAttributesGetter.java:26)
> ...
> ```
> refer to
>
> https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/instrumentation/tomcat/tomcat-common/javaagent/src/main/java/io/opentelemetry/javaagent/instrumentation/tomcat/common/TomcatHttpAttributesGetter.java#L28
>
> so the thing is telemetry-agent adds instrumentation code to call
> mb.toString() before it hits tomcat, which will change mb type(to T_STR)
>
>
> step 2.
> https://github.com/apache/tomcat/blob/9.0.71/java/org/apache/catalina/connector/CoyoteAdapter.java#L677
> mb type was changed at step1 so the following processing is wrong.
>
> Questions:
> 1. In mb.toString() it will change the type to (T_STR), is it a right
> approach?(compare with 9.0.70)
> 2. It may not be tomcat bug, but I feel a bit confused. The type will be
> changed to T_STR implicitly when call mb.toString(), it is kind of API
> change will impact 3rd libs such as open-telemetry?
> 3. what is your suggestion to solve this issue?
>
>
> thanks
> Zhou Rui
>
> On Fri, 17 Mar 2023 at 22:41, Mark Thomas <ma...@apache.org> wrote:
>
>> On 17/03/2023 14:27, Rui wrote:
>> > Hi user group:
>> >
>> > I added some debug info to the 9.0.71 baseline
>> >
>> > from the logs it hit this (supposed not inn 9.0.70)
>> >
>> https://github.com/apache/tomcat/blob/9.0.71/java/org/apache/catalina/connector/CoyoteAdapter.java#L677
>>
>> That suggests you are using the Rewrite Valve? Is that correct?
>>
>> If so, that suggests that whatever is going wrong, is going wrong in
>> that Valve.
>>
>> If not, it is likely that your debugging is corrupting the state of the
>> MessageByte instances. You need to be really careful when logging values
>> from them.
>>
>> Mark
>>
>> >
>> > means undecodedURI type is T_CHARS or T_STR, however, decodedURI type is
>> > T_NULL at this time,
>> > so decodedURI.toChars(); actually will recycle its buffer.
>> >
>> > I can reproduce it locally with debug mode, but don't know how
>> reproduce it
>> > in the company EKS cluster
>> >
>> > thanks.
>> > Zhou Rui
>> >
>> >
>> >
>> > On Fri, 17 Mar 2023 at 00:50, Rui <wi...@gmail.com> wrote:
>> >
>> >> I did some tests with several different commits in 9.0.71, I think my
>> >> issue is caused by this
>> >>
>> >>
>> https://github.com/apache/tomcat/commit/10a1a6d46d952bab4dfde44c3c0de12b0330da79
>> >> the "toBytesSimple" change has not been added to the repo yet, so the
>> >> change in 9.073 doesn't solve the problem.
>> >> Next I will go through these codes, but any clue?
>> >>
>> >> thanks
>> >> Zhou Rui
>> >>
>> >>
>> >> On Tue, 7 Mar 2023 at 00:34, Mark Thomas <ma...@apache.org> wrote:
>> >>
>> >>> On 25/02/2023 17:57, Mark Thomas wrote:
>> >>>>
>> >>>>
>> >>>> On 25/02/2023 15:47, Rui wrote:
>> >>>>> Hi
>> >>>>> recently upgraded tomcat to 9.0.71 from 9.0.70
>> >>>>> but saw 404 in our EKS cluster(with istio installed)
>> >>>>>
>> >>>>> Received [GET /actuator HTTP/1.1
>> >>>>> Host: xxxxx:8079
>> >>>>> User-Agent: kube-probe/1.23+
>> >>>>> Accept: */*
>> >>>>> Connection: close
>> >>>>> Accept-Encoding: gzip
>> >>>>> ]
>> >>>>> Incoming request /health with originalRemoteAddr xxxx
>> >>>>>
>> >>>>> in 9.0.70 I can see the below following message but not in 9.0.71
>> >>>>> o.a.c.authenticator.AuthenticatorBase : Security checking request
>> >>> GET
>> >>>>> /health
>> >>>>>
>> >>>>> seems the processing has stopped somewhere and the pod health check
>> >>>>> didn't
>> >>>>> pass.
>> >>>>>
>> >>>>> I also noticed there was also a 404 issue but don't know if it is
>> >>>>> relevant.
>> >>>>> https://lists.apache.org/thread/gr814rmrlbk9rrqxqjrh4p3x0bfvv1g9
>> >>>>>
>> >>>>> I have tested it locally with curl or jmeter, but can't reproduce
>> >>>>> the problem.
>> >>>>>
>> >>>>> but when I step by step debug the spring app, I found the
>> undecodedURI
>> >>>>> type
>> >>>>> is T_STR
>> >>>>> in CoyoteAdapter.java(in 70 it supposed to be T_BYTES), then
>> >>>>> decodedURI is
>> >>>>> uninitialized and the uri can't find the mapping in internalMap of
>> >>>>> Mapper.java, which will cause 404(guess it thinks the uri is
>> malformed)
>> >>>>>
>> >>>>> MessageBytes decodedURI = req.decodedURI();
>> >>>>>
>> >>>>> if (undecodedURI.getType() == MessageBytes.T_BYTES) {
>> >>>>> // Copy the raw URI to the decodedURI
>> >>>>> decodedURI.duplicate(undecodedURI);
>> >>>>>
>> >>>>>
>> >>>>> 404 example: (when I debug step by step to check unndecodedURI)
>> >>>>>
>> >>>>> curl http://localhost:8080/actuator
>> >>>>>
>> >>>>> <!doctype html><html lang="en"><head><title>HTTP Status 404 – Not
>> >>>>> Found</title><style type="text/css">body
>> >>>>> {font-family:Tahoma,Arial,sans-serif;} h1, h2, h3, b
>> >>>>> {color:white;background-color:#525D76;} h1 {font-size:22px;} h2
>> >>>>> {font-size:16px;} h3 {font-size:14px;} p {font-size:12px;} a
>> >>>>> {color:black;}
>> >>>>> .line
>> >>>>>
>> >>>
>> {height:1px;background-color:#525D76;border:none;}</style></head><body><h1>HTTP
>> >>>>> Status 404 – Not Found</h1></body></html>%
>> >>>>>
>> >>>>>
>> >>>>> However, it runs well without breakpoint! I am quite confused...my
>> >>>>> intellij
>> >>>>> has problem?
>> >>>>> but anyway, summary:
>> >>>>> 9.0.69 and 70 are good in the AWS/EKS(kubernetes) cluster with
>> istio
>> >>>>> 9.0.71 and 72 return 404 when health check by the EKS cluster.
>> >>>>
>> >>>> Looks like an instance of:
>> >>>> https://bz.apache.org/bugzilla/show_bug.cgi?id=66488
>> >>>
>> >>> If not that then maybe
>> >>>
>> >>> https://bz.apache.org/bugzilla/show_bug.cgi?id=66512
>> >>>
>> >>> Mark
>> >>>
>> >>> ---------------------------------------------------------------------
>> >>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> >>> For additional commands, e-mail: users-help@tomcat.apache.org
>> >>>
>> >>>
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
Re: health check return 404 after upgrade from 70 to tomcat 9.0.71
Posted by Rui <wi...@gmail.com>.
Hi Mark
I can't find any Valve setting in the spring boot application(search by
valve), would you mind sharing more info?
Regarding the mb state and debugging, yes aware that my local intellij
debugger could change the mb state, so in the the debug lib I call
mb.getType() only
I got more interesting logs today:
step 1. a stack trace before it hits CoyoteAdapter.java#L677
<https://github.com/apache/tomcat/blob/9.0.71/java/org/apache/catalina/connector/CoyoteAdapter.java#L677>
``` I type manually, can't copy it
mb.toString()
io.opentelemetry.javaagent.instrumentation.tomcat.common.TomcatHttpAttributesGetter.target(TomcatHttpAttributesGetter.java:26)
...
```
refer to
https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/instrumentation/tomcat/tomcat-common/javaagent/src/main/java/io/opentelemetry/javaagent/instrumentation/tomcat/common/TomcatHttpAttributesGetter.java#L28
so the thing is telemetry-agent adds instrumentation code to call
mb.toString() before it hits tomcat, which will change mb type(to T_STR)
step 2.
https://github.com/apache/tomcat/blob/9.0.71/java/org/apache/catalina/connector/CoyoteAdapter.java#L677
mb type was changed at step1 so the following processing is wrong.
Questions:
1. In mb.toString() it will change the type to (T_STR), is it a right
approach?(compare with 9.0.70)
2. It may not be tomcat bug, but I feel a bit confused. The type will be
changed to T_STR implicitly when call mb.toString(), it is kind of API
change will impact 3rd libs such as open-telemetry?
3. what is your suggestion to solve this issue?
thanks
Zhou Rui
On Fri, 17 Mar 2023 at 22:41, Mark Thomas <ma...@apache.org> wrote:
> On 17/03/2023 14:27, Rui wrote:
> > Hi user group:
> >
> > I added some debug info to the 9.0.71 baseline
> >
> > from the logs it hit this (supposed not inn 9.0.70)
> >
> https://github.com/apache/tomcat/blob/9.0.71/java/org/apache/catalina/connector/CoyoteAdapter.java#L677
>
> That suggests you are using the Rewrite Valve? Is that correct?
>
> If so, that suggests that whatever is going wrong, is going wrong in
> that Valve.
>
> If not, it is likely that your debugging is corrupting the state of the
> MessageByte instances. You need to be really careful when logging values
> from them.
>
> Mark
>
> >
> > means undecodedURI type is T_CHARS or T_STR, however, decodedURI type is
> > T_NULL at this time,
> > so decodedURI.toChars(); actually will recycle its buffer.
> >
> > I can reproduce it locally with debug mode, but don't know how reproduce
> it
> > in the company EKS cluster
> >
> > thanks.
> > Zhou Rui
> >
> >
> >
> > On Fri, 17 Mar 2023 at 00:50, Rui <wi...@gmail.com> wrote:
> >
> >> I did some tests with several different commits in 9.0.71, I think my
> >> issue is caused by this
> >>
> >>
> https://github.com/apache/tomcat/commit/10a1a6d46d952bab4dfde44c3c0de12b0330da79
> >> the "toBytesSimple" change has not been added to the repo yet, so the
> >> change in 9.073 doesn't solve the problem.
> >> Next I will go through these codes, but any clue?
> >>
> >> thanks
> >> Zhou Rui
> >>
> >>
> >> On Tue, 7 Mar 2023 at 00:34, Mark Thomas <ma...@apache.org> wrote:
> >>
> >>> On 25/02/2023 17:57, Mark Thomas wrote:
> >>>>
> >>>>
> >>>> On 25/02/2023 15:47, Rui wrote:
> >>>>> Hi
> >>>>> recently upgraded tomcat to 9.0.71 from 9.0.70
> >>>>> but saw 404 in our EKS cluster(with istio installed)
> >>>>>
> >>>>> Received [GET /actuator HTTP/1.1
> >>>>> Host: xxxxx:8079
> >>>>> User-Agent: kube-probe/1.23+
> >>>>> Accept: */*
> >>>>> Connection: close
> >>>>> Accept-Encoding: gzip
> >>>>> ]
> >>>>> Incoming request /health with originalRemoteAddr xxxx
> >>>>>
> >>>>> in 9.0.70 I can see the below following message but not in 9.0.71
> >>>>> o.a.c.authenticator.AuthenticatorBase : Security checking request
> >>> GET
> >>>>> /health
> >>>>>
> >>>>> seems the processing has stopped somewhere and the pod health check
> >>>>> didn't
> >>>>> pass.
> >>>>>
> >>>>> I also noticed there was also a 404 issue but don't know if it is
> >>>>> relevant.
> >>>>> https://lists.apache.org/thread/gr814rmrlbk9rrqxqjrh4p3x0bfvv1g9
> >>>>>
> >>>>> I have tested it locally with curl or jmeter, but can't reproduce
> >>>>> the problem.
> >>>>>
> >>>>> but when I step by step debug the spring app, I found the
> undecodedURI
> >>>>> type
> >>>>> is T_STR
> >>>>> in CoyoteAdapter.java(in 70 it supposed to be T_BYTES), then
> >>>>> decodedURI is
> >>>>> uninitialized and the uri can't find the mapping in internalMap of
> >>>>> Mapper.java, which will cause 404(guess it thinks the uri is
> malformed)
> >>>>>
> >>>>> MessageBytes decodedURI = req.decodedURI();
> >>>>>
> >>>>> if (undecodedURI.getType() == MessageBytes.T_BYTES) {
> >>>>> // Copy the raw URI to the decodedURI
> >>>>> decodedURI.duplicate(undecodedURI);
> >>>>>
> >>>>>
> >>>>> 404 example: (when I debug step by step to check unndecodedURI)
> >>>>>
> >>>>> curl http://localhost:8080/actuator
> >>>>>
> >>>>> <!doctype html><html lang="en"><head><title>HTTP Status 404 – Not
> >>>>> Found</title><style type="text/css">body
> >>>>> {font-family:Tahoma,Arial,sans-serif;} h1, h2, h3, b
> >>>>> {color:white;background-color:#525D76;} h1 {font-size:22px;} h2
> >>>>> {font-size:16px;} h3 {font-size:14px;} p {font-size:12px;} a
> >>>>> {color:black;}
> >>>>> .line
> >>>>>
> >>>
> {height:1px;background-color:#525D76;border:none;}</style></head><body><h1>HTTP
> >>>>> Status 404 – Not Found</h1></body></html>%
> >>>>>
> >>>>>
> >>>>> However, it runs well without breakpoint! I am quite confused...my
> >>>>> intellij
> >>>>> has problem?
> >>>>> but anyway, summary:
> >>>>> 9.0.69 and 70 are good in the AWS/EKS(kubernetes) cluster with
> istio
> >>>>> 9.0.71 and 72 return 404 when health check by the EKS cluster.
> >>>>
> >>>> Looks like an instance of:
> >>>> https://bz.apache.org/bugzilla/show_bug.cgi?id=66488
> >>>
> >>> If not that then maybe
> >>>
> >>> https://bz.apache.org/bugzilla/show_bug.cgi?id=66512
> >>>
> >>> Mark
> >>>
> >>> ---------------------------------------------------------------------
> >>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> >>> For additional commands, e-mail: users-help@tomcat.apache.org
> >>>
> >>>
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
Re: health check return 404 after upgrade from 70 to tomcat 9.0.71
Posted by Mark Thomas <ma...@apache.org>.
On 17/03/2023 14:27, Rui wrote:
> Hi user group:
>
> I added some debug info to the 9.0.71 baseline
>
> from the logs it hit this (supposed not inn 9.0.70)
> https://github.com/apache/tomcat/blob/9.0.71/java/org/apache/catalina/connector/CoyoteAdapter.java#L677
That suggests you are using the Rewrite Valve? Is that correct?
If so, that suggests that whatever is going wrong, is going wrong in
that Valve.
If not, it is likely that your debugging is corrupting the state of the
MessageByte instances. You need to be really careful when logging values
from them.
Mark
>
> means undecodedURI type is T_CHARS or T_STR, however, decodedURI type is
> T_NULL at this time,
> so decodedURI.toChars(); actually will recycle its buffer.
>
> I can reproduce it locally with debug mode, but don't know how reproduce it
> in the company EKS cluster
>
> thanks.
> Zhou Rui
>
>
>
> On Fri, 17 Mar 2023 at 00:50, Rui <wi...@gmail.com> wrote:
>
>> I did some tests with several different commits in 9.0.71, I think my
>> issue is caused by this
>>
>> https://github.com/apache/tomcat/commit/10a1a6d46d952bab4dfde44c3c0de12b0330da79
>> the "toBytesSimple" change has not been added to the repo yet, so the
>> change in 9.073 doesn't solve the problem.
>> Next I will go through these codes, but any clue?
>>
>> thanks
>> Zhou Rui
>>
>>
>> On Tue, 7 Mar 2023 at 00:34, Mark Thomas <ma...@apache.org> wrote:
>>
>>> On 25/02/2023 17:57, Mark Thomas wrote:
>>>>
>>>>
>>>> On 25/02/2023 15:47, Rui wrote:
>>>>> Hi
>>>>> recently upgraded tomcat to 9.0.71 from 9.0.70
>>>>> but saw 404 in our EKS cluster(with istio installed)
>>>>>
>>>>> Received [GET /actuator HTTP/1.1
>>>>> Host: xxxxx:8079
>>>>> User-Agent: kube-probe/1.23+
>>>>> Accept: */*
>>>>> Connection: close
>>>>> Accept-Encoding: gzip
>>>>> ]
>>>>> Incoming request /health with originalRemoteAddr xxxx
>>>>>
>>>>> in 9.0.70 I can see the below following message but not in 9.0.71
>>>>> o.a.c.authenticator.AuthenticatorBase : Security checking request
>>> GET
>>>>> /health
>>>>>
>>>>> seems the processing has stopped somewhere and the pod health check
>>>>> didn't
>>>>> pass.
>>>>>
>>>>> I also noticed there was also a 404 issue but don't know if it is
>>>>> relevant.
>>>>> https://lists.apache.org/thread/gr814rmrlbk9rrqxqjrh4p3x0bfvv1g9
>>>>>
>>>>> I have tested it locally with curl or jmeter, but can't reproduce
>>>>> the problem.
>>>>>
>>>>> but when I step by step debug the spring app, I found the undecodedURI
>>>>> type
>>>>> is T_STR
>>>>> in CoyoteAdapter.java(in 70 it supposed to be T_BYTES), then
>>>>> decodedURI is
>>>>> uninitialized and the uri can't find the mapping in internalMap of
>>>>> Mapper.java, which will cause 404(guess it thinks the uri is malformed)
>>>>>
>>>>> MessageBytes decodedURI = req.decodedURI();
>>>>>
>>>>> if (undecodedURI.getType() == MessageBytes.T_BYTES) {
>>>>> // Copy the raw URI to the decodedURI
>>>>> decodedURI.duplicate(undecodedURI);
>>>>>
>>>>>
>>>>> 404 example: (when I debug step by step to check unndecodedURI)
>>>>>
>>>>> curl http://localhost:8080/actuator
>>>>>
>>>>> <!doctype html><html lang="en"><head><title>HTTP Status 404 – Not
>>>>> Found</title><style type="text/css">body
>>>>> {font-family:Tahoma,Arial,sans-serif;} h1, h2, h3, b
>>>>> {color:white;background-color:#525D76;} h1 {font-size:22px;} h2
>>>>> {font-size:16px;} h3 {font-size:14px;} p {font-size:12px;} a
>>>>> {color:black;}
>>>>> .line
>>>>>
>>> {height:1px;background-color:#525D76;border:none;}</style></head><body><h1>HTTP
>>>>> Status 404 – Not Found</h1></body></html>%
>>>>>
>>>>>
>>>>> However, it runs well without breakpoint! I am quite confused...my
>>>>> intellij
>>>>> has problem?
>>>>> but anyway, summary:
>>>>> 9.0.69 and 70 are good in the AWS/EKS(kubernetes) cluster with istio
>>>>> 9.0.71 and 72 return 404 when health check by the EKS cluster.
>>>>
>>>> Looks like an instance of:
>>>> https://bz.apache.org/bugzilla/show_bug.cgi?id=66488
>>>
>>> If not that then maybe
>>>
>>> https://bz.apache.org/bugzilla/show_bug.cgi?id=66512
>>>
>>> Mark
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>
>>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: health check return 404 after upgrade from 70 to tomcat 9.0.71
Posted by Rui <wi...@gmail.com>.
Hi user group:
I added some debug info to the 9.0.71 baseline
from the logs it hit this (supposed not inn 9.0.70)
https://github.com/apache/tomcat/blob/9.0.71/java/org/apache/catalina/connector/CoyoteAdapter.java#L677
means undecodedURI type is T_CHARS or T_STR, however, decodedURI type is
T_NULL at this time,
so decodedURI.toChars(); actually will recycle its buffer.
I can reproduce it locally with debug mode, but don't know how reproduce it
in the company EKS cluster
thanks.
Zhou Rui
On Fri, 17 Mar 2023 at 00:50, Rui <wi...@gmail.com> wrote:
> I did some tests with several different commits in 9.0.71, I think my
> issue is caused by this
>
> https://github.com/apache/tomcat/commit/10a1a6d46d952bab4dfde44c3c0de12b0330da79
> the "toBytesSimple" change has not been added to the repo yet, so the
> change in 9.073 doesn't solve the problem.
> Next I will go through these codes, but any clue?
>
> thanks
> Zhou Rui
>
>
> On Tue, 7 Mar 2023 at 00:34, Mark Thomas <ma...@apache.org> wrote:
>
>> On 25/02/2023 17:57, Mark Thomas wrote:
>> >
>> >
>> > On 25/02/2023 15:47, Rui wrote:
>> >> Hi
>> >> recently upgraded tomcat to 9.0.71 from 9.0.70
>> >> but saw 404 in our EKS cluster(with istio installed)
>> >>
>> >> Received [GET /actuator HTTP/1.1
>> >> Host: xxxxx:8079
>> >> User-Agent: kube-probe/1.23+
>> >> Accept: */*
>> >> Connection: close
>> >> Accept-Encoding: gzip
>> >> ]
>> >> Incoming request /health with originalRemoteAddr xxxx
>> >>
>> >> in 9.0.70 I can see the below following message but not in 9.0.71
>> >> o.a.c.authenticator.AuthenticatorBase : Security checking request
>> GET
>> >> /health
>> >>
>> >> seems the processing has stopped somewhere and the pod health check
>> >> didn't
>> >> pass.
>> >>
>> >> I also noticed there was also a 404 issue but don't know if it is
>> >> relevant.
>> >> https://lists.apache.org/thread/gr814rmrlbk9rrqxqjrh4p3x0bfvv1g9
>> >>
>> >> I have tested it locally with curl or jmeter, but can't reproduce
>> >> the problem.
>> >>
>> >> but when I step by step debug the spring app, I found the undecodedURI
>> >> type
>> >> is T_STR
>> >> in CoyoteAdapter.java(in 70 it supposed to be T_BYTES), then
>> >> decodedURI is
>> >> uninitialized and the uri can't find the mapping in internalMap of
>> >> Mapper.java, which will cause 404(guess it thinks the uri is malformed)
>> >>
>> >> MessageBytes decodedURI = req.decodedURI();
>> >>
>> >> if (undecodedURI.getType() == MessageBytes.T_BYTES) {
>> >> // Copy the raw URI to the decodedURI
>> >> decodedURI.duplicate(undecodedURI);
>> >>
>> >>
>> >> 404 example: (when I debug step by step to check unndecodedURI)
>> >>
>> >> curl http://localhost:8080/actuator
>> >>
>> >> <!doctype html><html lang="en"><head><title>HTTP Status 404 – Not
>> >> Found</title><style type="text/css">body
>> >> {font-family:Tahoma,Arial,sans-serif;} h1, h2, h3, b
>> >> {color:white;background-color:#525D76;} h1 {font-size:22px;} h2
>> >> {font-size:16px;} h3 {font-size:14px;} p {font-size:12px;} a
>> >> {color:black;}
>> >> .line
>> >>
>> {height:1px;background-color:#525D76;border:none;}</style></head><body><h1>HTTP
>> >> Status 404 – Not Found</h1></body></html>%
>> >>
>> >>
>> >> However, it runs well without breakpoint! I am quite confused...my
>> >> intellij
>> >> has problem?
>> >> but anyway, summary:
>> >> 9.0.69 and 70 are good in the AWS/EKS(kubernetes) cluster with istio
>> >> 9.0.71 and 72 return 404 when health check by the EKS cluster.
>> >
>> > Looks like an instance of:
>> > https://bz.apache.org/bugzilla/show_bug.cgi?id=66488
>>
>> If not that then maybe
>>
>> https://bz.apache.org/bugzilla/show_bug.cgi?id=66512
>>
>> Mark
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
Re: health check return 404 after upgrade from 70 to tomcat 9.0.71
Posted by Rui <wi...@gmail.com>.
I did some tests with several different commits in 9.0.71, I think my issue
is caused by this
https://github.com/apache/tomcat/commit/10a1a6d46d952bab4dfde44c3c0de12b0330da79
the "toBytesSimple" change has not been added to the repo yet, so the
change in 9.073 doesn't solve the problem.
Next I will go through these codes, but any clue?
thanks
Zhou Rui
On Tue, 7 Mar 2023 at 00:34, Mark Thomas <ma...@apache.org> wrote:
> On 25/02/2023 17:57, Mark Thomas wrote:
> >
> >
> > On 25/02/2023 15:47, Rui wrote:
> >> Hi
> >> recently upgraded tomcat to 9.0.71 from 9.0.70
> >> but saw 404 in our EKS cluster(with istio installed)
> >>
> >> Received [GET /actuator HTTP/1.1
> >> Host: xxxxx:8079
> >> User-Agent: kube-probe/1.23+
> >> Accept: */*
> >> Connection: close
> >> Accept-Encoding: gzip
> >> ]
> >> Incoming request /health with originalRemoteAddr xxxx
> >>
> >> in 9.0.70 I can see the below following message but not in 9.0.71
> >> o.a.c.authenticator.AuthenticatorBase : Security checking request GET
> >> /health
> >>
> >> seems the processing has stopped somewhere and the pod health check
> >> didn't
> >> pass.
> >>
> >> I also noticed there was also a 404 issue but don't know if it is
> >> relevant.
> >> https://lists.apache.org/thread/gr814rmrlbk9rrqxqjrh4p3x0bfvv1g9
> >>
> >> I have tested it locally with curl or jmeter, but can't reproduce
> >> the problem.
> >>
> >> but when I step by step debug the spring app, I found the undecodedURI
> >> type
> >> is T_STR
> >> in CoyoteAdapter.java(in 70 it supposed to be T_BYTES), then
> >> decodedURI is
> >> uninitialized and the uri can't find the mapping in internalMap of
> >> Mapper.java, which will cause 404(guess it thinks the uri is malformed)
> >>
> >> MessageBytes decodedURI = req.decodedURI();
> >>
> >> if (undecodedURI.getType() == MessageBytes.T_BYTES) {
> >> // Copy the raw URI to the decodedURI
> >> decodedURI.duplicate(undecodedURI);
> >>
> >>
> >> 404 example: (when I debug step by step to check unndecodedURI)
> >>
> >> curl http://localhost:8080/actuator
> >>
> >> <!doctype html><html lang="en"><head><title>HTTP Status 404 – Not
> >> Found</title><style type="text/css">body
> >> {font-family:Tahoma,Arial,sans-serif;} h1, h2, h3, b
> >> {color:white;background-color:#525D76;} h1 {font-size:22px;} h2
> >> {font-size:16px;} h3 {font-size:14px;} p {font-size:12px;} a
> >> {color:black;}
> >> .line
> >>
> {height:1px;background-color:#525D76;border:none;}</style></head><body><h1>HTTP
> >> Status 404 – Not Found</h1></body></html>%
> >>
> >>
> >> However, it runs well without breakpoint! I am quite confused...my
> >> intellij
> >> has problem?
> >> but anyway, summary:
> >> 9.0.69 and 70 are good in the AWS/EKS(kubernetes) cluster with istio
> >> 9.0.71 and 72 return 404 when health check by the EKS cluster.
> >
> > Looks like an instance of:
> > https://bz.apache.org/bugzilla/show_bug.cgi?id=66488
>
> If not that then maybe
>
> https://bz.apache.org/bugzilla/show_bug.cgi?id=66512
>
> Mark
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
RE: health check return 404 after upgrade from 70 to tomcat 9.0.71
Posted by jo...@wellsfargo.com.INVALID.
> -----Original Message-----
> From: Chuck Caldarale <n8...@gmail.com>
> Sent: Monday, March 6, 2023 1:12 PM
> To: Tomcat Users List <us...@tomcat.apache.org>
> Subject: Re: health check return 404 after upgrade from 70 to tomcat 9.0.71
>
>
> > On Mar 6, 2023, at 12:03, <jo...@wellsfargo.com.invalid>
> <jo...@wellsfargo.com.INVALID> wrote:
> >
> > I believe we may have figured out what was causing the issue, but not why
> it was causing it.
> >
> > In the mod-proxy-http configuration for the balancer, we had been using
> the fully qualified host name in the balancer url. When we switched to IP
> address, things got tremendously faster to the point we aren't seeing the
> health check fail any longer. What is perplexing is why removing a simple DNS
> lookup by going from host name to IP address had such a dramatic affect. Is
> this something that adding ProxyHost and ProxyPort to the connector would
> possibly cure? We have not tried that currently.
>
> Once upon a time (before I retired), we had slow DNS lookups due to a
> misconfigured /etc/resolv.conf file still including a disabled DNS box as the
> first in the list. Might want to check the DNS configurations on the systems of
> interest and insure that all listed DNS boxes are functional.
>
> - Chuck
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
Thank you Chuck!
Dream * Excel * Explore * Inspire
Jon McAlexander
Senior Infrastructure Engineer
Asst. Vice President
He/His
Middleware Product Engineering
Enterprise CIO | EAS | Middleware | Infrastructure Solutions
8080 Cobblestone Rd | Urbandale, IA 50322
MAC: F4469-010
Tel 515-988-2508 | Cell 515-988-2508
jonmcalexander@wellsfargo.com
This message may contain confidential and/or privileged information. If you are not the addressee or authorized to receive this for the addressee, you must not use, copy, disclose, or take any action based on this message or any information herein. If you have received this message in error, please advise the sender immediately by reply e-mail and delete this message. Thank you for your cooperation.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: health check return 404 after upgrade from 70 to tomcat 9.0.71
Posted by Chuck Caldarale <n8...@gmail.com>.
> On Mar 6, 2023, at 12:03, <jo...@wellsfargo.com.invalid> <jo...@wellsfargo.com.INVALID> wrote:
>
> I believe we may have figured out what was causing the issue, but not why it was causing it.
>
> In the mod-proxy-http configuration for the balancer, we had been using the fully qualified host name in the balancer url. When we switched to IP address, things got tremendously faster to the point we aren't seeing the health check fail any longer. What is perplexing is why removing a simple DNS lookup by going from host name to IP address had such a dramatic affect. Is this something that adding ProxyHost and ProxyPort to the connector would possibly cure? We have not tried that currently.
Once upon a time (before I retired), we had slow DNS lookups due to a misconfigured /etc/resolv.conf file still including a disabled DNS box as the first in the list. Might want to check the DNS configurations on the systems of interest and insure that all listed DNS boxes are functional.
- Chuck
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: health check return 404 after upgrade from 70 to tomcat 9.0.71
Posted by jo...@wellsfargo.com.INVALID.
> -----Original Message-----
> From: Mark Thomas <ma...@apache.org>
> Sent: Monday, March 6, 2023 10:34 AM
> To: users@tomcat.apache.org
> Subject: Re: health check return 404 after upgrade from 70 to tomcat 9.0.71
>
> On 25/02/2023 17:57, Mark Thomas wrote:
> >
> >
> > On 25/02/2023 15:47, Rui wrote:
> >> Hi
> >> recently upgraded tomcat to 9.0.71 from 9.0.70 but saw 404 in our EKS
> >> cluster(with istio installed)
> >>
> >> Received [GET /actuator HTTP/1.1
> >> Host: xxxxx:8079
> >> User-Agent: kube-probe/1.23+
> >> Accept: */*
> >> Connection: close
> >> Accept-Encoding: gzip
> >> ]
> >> Incoming request /health with originalRemoteAddr xxxx
> >>
> >> in 9.0.70 I can see the below following message but not in 9.0.71
> >> o.a.c.authenticator.AuthenticatorBase : Security checking request
> >> GET /health
> >>
> >> seems the processing has stopped somewhere and the pod health check
> >> didn't pass.
> >>
> >> I also noticed there was also a 404 issue but don't know if it is
> >> relevant.
> >> https://urldefense.com/v3/__https://lists.apache.org/thread/gr814rmrl
> >> bk9rrqxqjrh4p3x0bfvv1g9__;!!F9svGWnIaVPGSwU!uiVBZpR54vzNpwI-
> 1lyRTgM4j
> >> KczN4Bobrx34KVhfSRZo-W3EsxLU7epxoIiW_-
> gA2CtJmE_dFUEWwJPZkCMDg$
> >>
> >> I have tested it locally with curl or jmeter, but can't reproduce the
> >> problem.
> >>
> >> but when I step by step debug the spring app, I found the
> >> undecodedURI type is T_STR in CoyoteAdapter.java(in 70 it supposed to
> >> be T_BYTES), then decodedURI is uninitialized and the uri can't find
> >> the mapping in internalMap of Mapper.java, which will cause 404(guess
> >> it thinks the uri is malformed)
> >>
> >> MessageBytes decodedURI = req.decodedURI();
> >>
> >> if (undecodedURI.getType() == MessageBytes.T_BYTES) {
> >> // Copy the raw URI to the decodedURI
> >> decodedURI.duplicate(undecodedURI);
> >>
> >>
> >> 404 example: (when I debug step by step to check unndecodedURI)
> >>
> >> curl
> >>
> https://urldefense.com/v3/__http://localhost:8080/actuator__;!!F9svGW
> >> nIaVPGSwU!uiVBZpR54vzNpwI-1lyRTgM4jKczN4Bobrx34KVhfSRZo-
> W3EsxLU7epxoI
> >> iW_-gA2CtJmE_dFUEWwKRB2lDBg$
> >>
> >> <!doctype html><html lang="en"><head><title>HTTP Status 404 – Not
> >> Found</title><style type="text/css">body
> >> {font-family:Tahoma,Arial,sans-serif;} h1, h2, h3, b
> >> {color:white;background-color:#525D76;} h1 {font-size:22px;} h2
> >> {font-size:16px;} h3 {font-size:14px;} p {font-size:12px;} a
> >> {color:black;} .line
> >> {height:1px;background-
> color:#525D76;border:none;}</style></head><bod
> >> y><h1>HTTP Status 404 – Not Found</h1></body></html>%
> >>
> >>
> >> However, it runs well without breakpoint! I am quite confused...my
> >> intellij has problem?
> >> but anyway, summary:
> >> 9.0.69 and 70 are good in the AWS/EKS(kubernetes) cluster with
> >> istio
> >> 9.0.71 and 72 return 404 when health check by the EKS cluster.
> >
> > Looks like an instance of:
> > https://urldefense.com/v3/__https://bz.apache.org/bugzilla/show_bug.cg
> > i?id=66488__;!!F9svGWnIaVPGSwU!uiVBZpR54vzNpwI-
> 1lyRTgM4jKczN4Bobrx34KV
> > hfSRZo-W3EsxLU7epxoIiW_-gA2CtJmE_dFUEWwJcG2lXgg$
>
> If not that then maybe
>
> https://urldefense.com/v3/__https://bz.apache.org/bugzilla/show_bug.cgi?
> id=66512__;!!F9svGWnIaVPGSwU!uiVBZpR54vzNpwI-
> 1lyRTgM4jKczN4Bobrx34KVhfSRZo-W3EsxLU7epxoIiW_-
> gA2CtJmE_dFUEWwIhYU5xdQ$
>
> Mark
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
I believe we may have figured out what was causing the issue, but not why it was causing it.
In the mod-proxy-http configuration for the balancer, we had been using the fully qualified host name in the balancer url. When we switched to IP address, things got tremendously faster to the point we aren't seeing the health check fail any longer. What is perplexing is why removing a simple DNS lookup by going from host name to IP address had such a dramatic affect. Is this something that adding ProxyHost and ProxyPort to the connector would possibly cure? We have not tried that currently.
Thanks,
Dream * Excel * Explore * Inspire
Jon McAlexander
Senior Infrastructure Engineer
Asst. Vice President
He/His
Middleware Product Engineering
Enterprise CIO | EAS | Middleware | Infrastructure Solutions
8080 Cobblestone Rd | Urbandale, IA 50322
MAC: F4469-010
Tel 515-988-2508 | Cell 515-988-2508
jonmcalexander@wellsfargo.com
This message may contain confidential and/or privileged information. If you are not the addressee or authorized to receive this for the addressee, you must not use, copy, disclose, or take any action based on this message or any information herein. If you have received this message in error, please advise the sender immediately by reply e-mail and delete this message. Thank you for your cooperation.