Re: AJP error using mod_proxy__ajp (conclusion: user error)

[Christopher Schultz <ch...@christopherschultz.net>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

All,

On 6/29/20 12:22, Christopher Schultz wrote:
> I believe I have determined the cause (or at least the fix) for
> this:
>
> Despite the mod_proxy_ajp documentation, it is using packets
> larger than 8192 bytes.
>
> Setting this on the Tomcat <Connector> resolves the problem:
>
> packetSize="65536"
>
> I don't see anywhere you can set the maximum packet size on
> mod_proxy_ajp so I think this is the only recourse.

That's because this is a property of mod_proxy and not mod_proxy_ajp.

ProxyIOBufferSize [1]

> Is anyone familiar enough with the mod_proxy_ajp code to know how
> to look for this?
>
> It was trivial to reproduce: just send a large POST message
> through mod_proxy_ajp to Tomcat.

Evidently, I've been down this road before and I had *intentionally*
set the high-water mark of ProxyIOBufferSize to 65535.

So this was entirely self-inflicted.

Thanks,
- -chris

[1] https://httpd.apache.org/docs/2.4/mod/mod_proxy.html#proxyiobuffersi
ze

> On 6/25/20 10:28, Christopher Schultz wrote:
>> All,
>
>> This issue is apparently trivially reproducible in my dev
>> environment.
>
>> Do I have to get a protocol-trace to get any more helpful
>> information?
>
>> Thanks, -chris
>
>> On 6/24/20 10:46, Christopher Schultz wrote:
>>> All,
>
>>> On 6/24/20 10:29, Christopher Schultz wrote:
>>>> All,
>
>>>> I'm slowly switching from mod_jk to mod_proxy_ajp and I have
>>>> a development environment where I'm getting Bad Gateway
>>>> responses sent to clients along with this exception in my
>>>> Tomcat log file:
>
>>>> java.lang.IllegalArgumentException: Header message of length
>>>> [8,194] received but the packetSize is only [8,192] at
>>>> org.apache.coyote.ajp.AjpProcessor.readMessage(AjpProcessor.java:68
5
>
>>>>
)
>
>>>>
>>>>
>
>>> at
>>>> org.apache.coyote.ajp.AjpProcessor.receive(AjpProcessor.java:626)
>>>>
>>>>
>
>>>>
>>>>
> at
>>>> org.apache.coyote.ajp.AjpProcessor.refillReadBuffer(AjpProcessor.ja
v
>
>>>>
a
>
>>>>
> :
>
>>>>
>> 73
>
>
>>> 4)
>>>> at
>>>> org.apache.coyote.ajp.AjpProcessor$SocketInputBuffer.doRead(AjpProc
e
>
>>>>
s
>
>>>>
> s
>
>>>>
>> or
>
>
>>> .java:1456)
>>>> at org.apache.coyote.Request.doRead(Request.java:581) at
>>>> org.apache.catalina.connector.InputBuffer.realReadBytes(InputBuffer
.
>
>>>>
j
>
>>>>
> a
>
>>>>
>> va
>
>
>>> :344)
>>>> at
>>>> org.apache.catalina.connector.InputBuffer.checkByteBufferEof(InputB
u
>
>>>>
f
>
>>>>
> f
>
>>>>
>> er
>
>
>>> .java:663)
>>>> at
>>>> org.apache.catalina.connector.InputBuffer.readByte(InputBuffer.java
:
>
>>>>
3
>
>>>>
> 5
>
>>>>
>> 8)
>
>
>>> at
>>>> org.apache.catalina.connector.CoyoteInputStream.read(CoyoteInputStr
e
>
>>>>
a
>
>>>>
> m
>
>>>>
>> .j
>
>
>>> ava:93)
>>>> at
>>>> org.apache.commons.io.input.ProxyInputStream.read(ProxyInputStream.
j
>
>>>>
a
>
>>>>
> v
>
>>>>
>> a:
>
>
>>> 53)
>>>> at
>>>> org.apache.commons.io.input.TeeInputStream.read(TeeInputStream.java
:
>
>>>>
1
>
>>>>
> 0
>
>>>>
>> 6)
>
>
>>> at java.io.FilterInputStream.read(FilterInputStream.java:83)
>>>> at my.product.MacInputStream.read(MacInputStream.java:29) at
>>>> java.io.FilterInputStream.read(FilterInputStream.java:83) at
>>>> com.sun.org.apache.xerces.internal.impl.XMLEntityManager$Rewindable
I
>
>>>>
n
>
>>>>
> p
>
>>>>
>> ut
>
>
>>> Stream.read(XMLEntityManager.java:2890)
>>>> at
>>>> com.sun.org.apache.xerces.internal.impl.XMLEntityManager.setupCurre
n
>
>>>>
t
>
>>>>
> E
>
>>>>
>> nt
>
>
>>> ity(XMLEntityManager.java:674)
>>>> at
>>>> com.sun.org.apache.xerces.internal.impl.XMLVersionDetector.determin
e
>
>>>>
D
>
>>>>
> o
>
>>>>
>> cV
>
>
>>> ersion(XMLVersionDetector.java:148)
>>>> at
>>>> com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse
(
>
>>>>
X
>
>>>>
> M
>
>>>>
>> L1
>
>
>>> 1Configuration.java:806)
>>>> at
>>>> com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse
(
>
>>>>
X
>
>>>>
> M
>
>>>>
>> L1
>
>
>>> 1Configuration.java:771)
>>>> at
>>>> com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParse
r
>
>>>>
.
>
>>>>
> j
>
>>>>
>> av
>
>
>>> a:141)
>>>> at
>>>> com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(
A
>
>>>>
b
>
>>>>
> s
>
>>>>
>> tr
>
>
>>> actSAXParser.java:1213)
>>>> at
>>>> com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser
.
>
>>>>
p
>
>>>>
> a
>
>>>>
>> rs
>
>
>>> e(SAXParserImpl.java:643)
>>>> at
>>>> com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl.parse(SAXPars
e
>
>>>>
r
>
>>>>
> I
>
>>>>
>> mp
>
>
>>> l.java:327)
>>>> at javax.xml.parsers.SAXParser.parse(SAXParser.java:195)
>
>>>> This is a web service which is reading the request with a
>>>> SAXParser. It's been running in production (and dev!) for
>>>> years without any issues. It''s been running for a few
>>>> months in development, now, with mod_proxy_ajp without any
>>>> errors.
>
>>>> I know about the "max packet size" and the default is 8192
>>>> bytes. I haven't changed the default. Here's my <Connector>
>>>> configuration:
>
>>>> <Connector port="8245" address="127.0.0.1"
>>>> secretRequired="false" redirectPort="443" protocol="AJP/1.3"
>>>> URIEncoding="UTF-8" executor="tomcatThreadPool" />
>
>>>> Here's the configuration in httpd.conf:
>
>>>> <Proxy "balancer://my-api"> BalancerMember
>>>> "ajp://localhost:8245" timeout=300 ping=5 ttl=60 </Proxy>
>
>>>> ProxyPass "/my-api/" "balancer://my-api/my-api/"
>>>> ProxyPassReverse "/my-api/" "balancer://my-api/my-api/"
>
>>>> The documentation for mod_proxy_ajp[1] seems to indicate
>>>> that the "Packet Size" for AJP is fixed at 8192 bytes:
>
>>>> " Packet Size
>
>>>> According to much of the code, the max packet size is 8 *
>>>> 1024 bytes (8K). The actual length of the packet is encoded
>>>> in the header.
>
>>>> Packet Headers
>
>>>> Packets sent from the server to the container begin with
>>>> 0x1234. Packets sent from the container to the server begin
>>>> with AB (that's the ASCII code for A followed by the ASCII
>>>> code for B). After those first two bytes, there is an
>>>> integer (encoded as above) with the length of the payload.
>>>> Although this might suggest that the maximum payload could be
>>>> as large as 2^16, in fact, *the code sets the maximum to be
>>>> 8K*. " (emphasis mine)
>
>>>> Does anyone know under what circumstances mod_proxy_ajp
>>>> might send more than 8192 bytes? It looks like mod_proxy_ajp
>>>> doesn't have any way to set the max packet size like mod_jk
>>>> does.
>
>>>> I should probably be able to set the max packet size on the
>>>> Tomcat side to something higher than 8192 to catch this kind
>>>> of thing... but it looks like it might be a bug in
>>>> mod_proxy_ajp.
>
>>>> Versions are Apache httpd 2.4.25 (Debian) and Tomcat
>>>> 8.5.trunk (8.5.55). mod_jk is not being used.
>
>>>> Any ideas?
>
>>>> -chris
>
>>>> [1] https://httpd.apache.org/docs/2.4/mod/mod_proxy_ajp.html
>
>
>>> Some additional information:
>
>>> 1. The headers of the HTTP request seem to be arriving in a
>>> correct packet before this error occurs. The headers are only
>>> a few hundred bytes (~340) and the request line should be
>>> relatively short (~50 bytes or so). Method is POST, protocol
>>> is HTTP/1.1.
>
>>> 2. Apache httpd is terminating TLS. I have no configuration
>>> for forwarding TLS information over to Tomcat, so I'm assuming
>>> it's not being sent as part of the first packet.
>
>>> 3. Before I get the packet-too-large error, I get another
>>> error:
>
>>> org.apache.catalina.connector.ClientAbortException:
>>> java.io.IOException: Invalid message received with length [-1]
>>> at
>>> org.apache.catalina.connector.InputBuffer.realReadBytes(InputBuffer.
j
>
>>>
a
>
>>>
> va
>
>
>> :348)
>>> at
>>> org.apache.catalina.connector.InputBuffer.checkByteBufferEof(InputBu
f
>
>>>
f
>
>>>
> er
>
>
>> .java:663)
>>> at
>>> org.apache.catalina.connector.InputBuffer.read(InputBuffer.java:370)
>
>>>
>>>
>
>> at
>>> org.apache.catalina.connector.CoyoteInputStream.read(CoyoteInputStre
a
>
>>>
m
>
>>>
> .j
>
>
>> ava:183)
>>> at
>>> org.apache.commons.io.input.ProxyInputStream.read(ProxyInputStream.j
a
>
>>>
v
>
>>>
> a:
>
>
>> 75)
>>> at
>>> org.apache.commons.io.input.TeeInputStream.read(TeeInputStream.java:
1
>
>>>
2
>
>>>
> 4)
>
>
>> at java.io.FilterInputStream.read(FilterInputStream.java:133)
>>> at my.product.MacInputStream.read(MacInputStream.java:49) at
>>> java.io.FilterInputStream.read(FilterInputStream.java:107) at
>>> my.product.XMLMessageProcessor.validate(XMLMessageProcessor.java:326
)
>
>>>
>>>
>
>> at com.chadis.servlet.APIServlet.doPost(APIServlet.java:291)
>>> at
>>> javax.servlet.http.HttpServlet.service(HttpServlet.java:660) at
>>> javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
>
>>> I haven't changed any configuration, yet. But if the first
>>> error is size=-1 then it's unlikely that the problem will be
>>> solved by increasing my max packet size on the Tomcat end.
>
>>> I'm working with my client to see if we can reproduce this
>>> problem reliably. I'm sure I can get more information once I
>>> do that.
>
>>> -chris
>
>
>
-----BEGIN PGP SIGNATURE-----
Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/

iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl76RyIACgkQHPApP6U8
pFhRbA//YNTn0vsQ5TKUUmkpCEaG99n6rm4aF2Tb7NbPoZqjuCQtNxmiKRp8Q+SP
rIF3m1AN6TUaaXosGXxGVUoOLRbZESC5QAXW/oU36zL8ioAf+XUyMLfq5lJxzokQ
lfyPNspYUKk37+CUr3ocVtaatgH2LX1Z81bMuY4LgU2eliNv9byUpe+bu+4hpEVj
r5EJalctwKw34A3ffUNspIXfUNlg0NUuzODcFiWlhE3MKujg6dEXjTY0fKwbz3W2
iGSkfofD8BDHDdBjjF3/pBodwunqSTsv0R/R/fDotm+CC5i59DPmfJf1j+JM4XUr
XeXZv4UTzVVScS7o3AaAELWnck1DUEG9eFC7Pv6hN3DbNgDqybu7tzVUH4/CCand
11GFzCFFqOTF7A+VbKY6D2zB206+nSbhJBXdOpJTJAHOqgFepItEHlLompoIIs7t
uuBIA5uzGUwi/b/UTPZdzMRh3qCvkWlz42GMKcCoDF47PGnV2+VpYP2qRwfucaMX
eY6kzq76Qbd0tCqr/O+Aqr0375afEPPLu10dlRu48uOcvqLWFlF1R7mAaSDWdRab
kweKjyj9P0+c51+2MiSZuNqEb2ayu7+is1o58WBCL9eVkoWDfWuCbgHn4O3vMkxj
ZRe8WITYJhXwutDU0scRWaCqE7W4r2IrFtPm2F8Q6IPhMnR7ySU=
=tH2L
-----END PGP SIGNATURE-----

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org