You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Larry Reisler <la...@vringo.com> on 2007/09/24 22:41:03 UTC
AJP Flush Packet causing text/plain output
We recently switched our development JBOSS instance from 4.05GA to 4.21GA, where we are have been using mod_jk for connecting an Apache front end server (2.2) to the Tomcat AppServer. We have noticed periodic times when the apache web server will return data with a content-type of plain/text (the server default) instead of the real content-type, and a chunked encoding (even if the AppServer was producing a non-chunked content). We tried switch to mod_proxy_ajp, but got the same result.
A bit of sleuthing (2 days X 2 engineers) revealed that the AJP connection on Tomcat is sending a SEND_BODY_CHUNK (which I assume is a flush packet) periodically that seem to be confusing mod_jk. We can see in the mod_jk and mod_proxy_ajp debug log that the correct headers are coming from the Tomcat server. However, it seems like mod_jk and mod_proxy_ajp are losing all the header information regarding the packet (including the Powered by headers, content-type, content-length, cookies, etc.).
In normal operation, for a sample small transaction, we would see this sequence in response:
SEND_HEADERS
SEND_BODY_CHUNK
END_RESPONSE
When the output would come out as text/plain, we would see this sequence:
SEND_BODY_CHUNK
SEND_HEADERS
SEND_BODY_CHUNK
END_RESPONSE
Is there something we should have configured differently so that mod_jk or mod_proxy_ajp will behave better?
RE: AJP Flush Packet causing text/plain output
Posted by Larry Reisler <la...@vringo.com>.
I understand your hesitation. I agree that my method was poor in terms of encapsulation, but was elegant in terms of its size, that it did not change any of the function signatures, and that it was easy to understand.
The holiday has begun here, and I am off for this week -- I will try to test the patch ASAP.
Thanks so much for your help.
Larry.
-----Original Message-----
From: Rainer Jung [mailto:rainer.jung@kippdata.de]
Sent: Tuesday, September 25, 2007 8:58 PM
To: Tomcat Users List
Subject: Re: AJP Flush Packet causing text/plain output
Hi Lary,
Larry Reisler wrote:
> Thanks for your reply. There is a holiday here right now, so I'm not
> sure if I will be able to get to file the Bugzilla issue before the
> holiday break.
Maybe this time we are faster than our request ticketing ...
> We tried a few different versions of mod_jk (the early ones had other
> issues), but all the latest ones showed the problem. We are
> currently using mod_jk 1.2.25.
>
> In desperation, I was able to create a patch to mod_jk that seems to
> prevent the symptom from occurring. Here is the diff for the file
> jk_ajp_common.c: 1742,1751d1741
< // Removing extra flush buffer if we do not need it.
< if (headeratclient == JK_FALSE) {
< int code = (int)jk_b_pget_byte(op->reply,0);
< unsigned int len = (unsigned int)jk_b_pget_int(op->reply,1);
< if ((code == JK_AJP13_SEND_BODY_CHUNK) && (len == 0)) {
< jk_log(l, JK_LOG_DEBUG, "Received flushbuffer -- ignoring");
< continue;
< }
< }
<
I'm a little reluctant to use your approach. It should work, but the
peeking for the message code and length does not belong into the get
reply function. The best place for the patch would be in the callback
handling, but there we don't know, if we already received the headers.
So either we pass some additional info to the callback handling (status
info about the request processing stage), or we prevent the flushing
later. I decided to move an already existing flag "response_started"
from the web server private data to the public service struct and use
it inside the callback handler.
This patch was committted
(http://svn.apache.org/viewvc?view=rev&revision=579349). You can find a
simple download form of it under
http://people.apache.org/~rjung/mod_jk-dev/patches/flush-before-header.patch
The patch looks lengthy, but in fact did only small changes.
It would be helpful, if you could try it with your reproduction scenario.
Regards,
Rainer
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: AJP Flush Packet causing text/plain output
Posted by Bill Barker <wb...@wilshire.com>.
This has been given the bug #
http://issues.apache.org/bugzilla/show_bug.cgi?id=43478
"Rainer Jung" <ra...@kippdata.de> wrote in message
news:46F95A2B.8030307@kippdata.de...
> Hi Lary,
>
> Larry Reisler wrote:
>> Thanks for your reply. There is a holiday here right now, so I'm not
>> sure if I will be able to get to file the Bugzilla issue before the
>> holiday break.
>
> Maybe this time we are faster than our request ticketing ...
>
>> We tried a few different versions of mod_jk (the early ones had other
>> issues), but all the latest ones showed the problem. We are
>> currently using mod_jk 1.2.25.
>>
>> In desperation, I was able to create a patch to mod_jk that seems to
>> prevent the symptom from occurring. Here is the diff for the file
>> jk_ajp_common.c: 1742,1751d1741
>
> < // Removing extra flush buffer if we do not need it.
> < if (headeratclient == JK_FALSE) {
> < int code = (int)jk_b_pget_byte(op->reply,0);
> < unsigned int len = (unsigned int)jk_b_pget_int(op->reply,1);
> < if ((code == JK_AJP13_SEND_BODY_CHUNK) && (len == 0)) {
> < jk_log(l, JK_LOG_DEBUG, "Received flushbuffer -- ignoring");
> < continue;
> < }
> < }
> <
>
> I'm a little reluctant to use your approach. It should work, but the
> peeking for the message code and length does not belong into the get reply
> function. The best place for the patch would be in the callback handling,
> but there we don't know, if we already received the headers.
>
> So either we pass some additional info to the callback handling (status
> info about the request processing stage), or we prevent the flushing
> later. I decided to move an already existing flag "response_started" from
> the web server private data to the public service struct and use it
> inside the callback handler.
>
> This patch was committted
> (http://svn.apache.org/viewvc?view=rev&revision=579349). You can find a
> simple download form of it under
>
> http://people.apache.org/~rjung/mod_jk-dev/patches/flush-before-header.patch
>
> The patch looks lengthy, but in fact did only small changes.
>
> It would be helpful, if you could try it with your reproduction scenario.
>
> Regards,
>
> Rainer
>
> ---------------------------------------------------------------------
> To start a new topic, e-mail: users@tomcat.apache.org
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: AJP Flush Packet causing text/plain output
Posted by Rainer Jung <ra...@kippdata.de>.
Hi Lary,
Larry Reisler wrote:
> Thanks for your reply. There is a holiday here right now, so I'm not
> sure if I will be able to get to file the Bugzilla issue before the
> holiday break.
Maybe this time we are faster than our request ticketing ...
> We tried a few different versions of mod_jk (the early ones had other
> issues), but all the latest ones showed the problem. We are
> currently using mod_jk 1.2.25.
>
> In desperation, I was able to create a patch to mod_jk that seems to
> prevent the symptom from occurring. Here is the diff for the file
> jk_ajp_common.c: 1742,1751d1741
< // Removing extra flush buffer if we do not need it.
< if (headeratclient == JK_FALSE) {
< int code = (int)jk_b_pget_byte(op->reply,0);
< unsigned int len = (unsigned int)jk_b_pget_int(op->reply,1);
< if ((code == JK_AJP13_SEND_BODY_CHUNK) && (len == 0)) {
< jk_log(l, JK_LOG_DEBUG, "Received flushbuffer -- ignoring");
< continue;
< }
< }
<
I'm a little reluctant to use your approach. It should work, but the
peeking for the message code and length does not belong into the get
reply function. The best place for the patch would be in the callback
handling, but there we don't know, if we already received the headers.
So either we pass some additional info to the callback handling (status
info about the request processing stage), or we prevent the flushing
later. I decided to move an already existing flag "response_started"
from the web server private data to the public service struct and use
it inside the callback handler.
This patch was committted
(http://svn.apache.org/viewvc?view=rev&revision=579349). You can find a
simple download form of it under
http://people.apache.org/~rjung/mod_jk-dev/patches/flush-before-header.patch
The patch looks lengthy, but in fact did only small changes.
It would be helpful, if you could try it with your reproduction scenario.
Regards,
Rainer
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: AJP Flush Packet causing text/plain output
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Martin,
Martin Gainty wrote:
> On a related note
> Has there been any jk connector releases for Apache HTTP 2.2?
Any reasonably recent version of mod_jk will work for Apache httpd 2.2.
As always, building your own binary is the best strategy, but binaries
are also available from the Apache website.
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iD8DBQFG+Vha9CaO5/Lv0PARAjonAJ0SmPXZVzsUMX99pHdAC0H/nTi+HQCfVAaw
Wrkv/L3B2a+vl/pbayeSPHE=
=LUO9
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: AJP Flush Packet causing text/plain output
Posted by Mladen Turk <ml...@gmail.com>.
Jim Jagielski wrote:
>>
>> Thanks for the patch. I'll check and likely commit. The core problem
>> maybe is inside the Tomcat AJP connector (shouldn't send flush
>> packets before sending headers) or in the webapp (which might trigger
>> flushes to early), but it will be definitely good to be tolerant on
>> the JK side.
>>
>
> +1... I'll look at the mod_proxy_ajp side.
>
Right, the problem is in the webapp probably. The first flush shouldn't
occur before initial packet is send.
However the explicit flushes (send_body with size 0) are very important.
Think the patch is safe, although not tested.
Regards,
Mladen
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: AJP Flush Packet causing text/plain output
Posted by Jim Jagielski <ji...@jaguNET.com>.
On Sep 25, 2007, at 6:39 AM, Rainer Jung wrote:
> Larry Reisler wrote:
>> Thanks for your reply. There is a holiday here right now, so I'm
>> not sure if I will be able to get to file the Bugzilla issue
>> before the holiday break.
>> We tried a few different versions of mod_jk (the early ones had
>> other issues), but all the latest ones showed the problem. We are
>> currently using mod_jk 1.2.25.
>> In desperation, I was able to create a patch to mod_jk that seems
>> to prevent the symptom from occurring. Here is the diff for the
>> file jk_ajp_common.c:
>> 1742,1751d1741
>> < // Removing extra flush buffer if we do not need it.
>> < if (headeratclient == JK_FALSE) {
>> < int code = (int)jk_b_pget_byte(op->reply,0);
>> < unsigned int len = (unsigned int)jk_b_pget_int(op-
>> >reply,1);
>> < if ((code == JK_AJP13_SEND_BODY_CHUNK) && (len == 0)) {
>> < jk_log(l, JK_LOG_DEBUG, "Received flushbuffer --
>> ignoring");
>> < continue;
>> < }
>> < }
>> <
>
> Thanks for the patch. I'll check and likely commit. The core
> problem maybe is inside the Tomcat AJP connector (shouldn't send
> flush packets before sending headers) or in the webapp (which might
> trigger flushes to early), but it will be definitely good to be
> tolerant on the JK side.
>
+1... I'll look at the mod_proxy_ajp side.
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: AJP Flush Packet causing text/plain output
Posted by Rainer Jung <ra...@kippdata.de>.
Larry Reisler wrote:
> Thanks for your reply. There is a holiday here right now, so I'm not sure if I will be able to get to file the Bugzilla issue before the holiday break.
>
> We tried a few different versions of mod_jk (the early ones had other issues), but all the latest ones showed the problem. We are currently using mod_jk 1.2.25.
>
> In desperation, I was able to create a patch to mod_jk that seems to prevent the symptom from occurring. Here is the diff for the file jk_ajp_common.c:
> 1742,1751d1741
> < // Removing extra flush buffer if we do not need it.
> < if (headeratclient == JK_FALSE) {
> < int code = (int)jk_b_pget_byte(op->reply,0);
> < unsigned int len = (unsigned int)jk_b_pget_int(op->reply,1);
> < if ((code == JK_AJP13_SEND_BODY_CHUNK) && (len == 0)) {
> < jk_log(l, JK_LOG_DEBUG, "Received flushbuffer -- ignoring");
> < continue;
> < }
> < }
> <
Thanks for the patch. I'll check and likely commit. The core problem
maybe is inside the Tomcat AJP connector (shouldn't send flush packets
before sending headers) or in the webapp (which might trigger flushes to
early), but it will be definitely good to be tolerant on the JK side.
Thanks!
Regards,
Rainer
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: AJP Flush Packet causing text/plain output
Posted by Larry Reisler <la...@vringo.com>.
Thanks for your reply. There is a holiday here right now, so I'm not sure if I will be able to get to file the Bugzilla issue before the holiday break.
We tried a few different versions of mod_jk (the early ones had other issues), but all the latest ones showed the problem. We are currently using mod_jk 1.2.25.
In desperation, I was able to create a patch to mod_jk that seems to prevent the symptom from occurring. Here is the diff for the file jk_ajp_common.c:
1742,1751d1741
< // Removing extra flush buffer if we do not need it.
< if (headeratclient == JK_FALSE) {
< int code = (int)jk_b_pget_byte(op->reply,0);
< unsigned int len = (unsigned int)jk_b_pget_int(op->reply,1);
< if ((code == JK_AJP13_SEND_BODY_CHUNK) && (len == 0)) {
< jk_log(l, JK_LOG_DEBUG, "Received flushbuffer -- ignoring");
< continue;
< }
< }
<
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: AJP Flush Packet causing text/plain output
Posted by Rainer Jung <ra...@kippdata.de>.
Hi Larry,
Larry Reisler wrote:
> We recently switched our development JBOSS instance from 4.05GA to
> 4.21GA, where we are have been using mod_jk for connecting an Apache
> front end server (2.2) to the Tomcat AppServer. We have noticed
Would you mind telling us the JK version?
> periodic times when the apache web server will return data with a
> content-type of plain/text (the server default) instead of the real
> content-type, and a chunked encoding (even if the AppServer was
> producing a non-chunked content). We tried switch to mod_proxy_ajp,
> but got the same result.
>
> A bit of sleuthing (2 days X 2 engineers) revealed that the AJP
> connection on Tomcat is sending a SEND_BODY_CHUNK (which I assume is
> a flush packet) periodically that seem to be confusing mod_jk. We can
> see in the mod_jk and mod_proxy_ajp debug log that the correct
> headers are coming from the Tomcat server. However, it seems like
> mod_jk and mod_proxy_ajp are losing all the header information
> regarding the packet (including the Powered by headers, content-type,
> content-length, cookies, etc.).
>
> In normal operation, for a sample small transaction, we would see
> this sequence in response: SEND_HEADERS SEND_BODY_CHUNK END_RESPONSE
>
> When the output would come out as text/plain, we would see this
> sequence: SEND_BODY_CHUNK SEND_HEADERS SEND_BODY_CHUNK END_RESPONSE
>
> Is there something we should have configured differently so that
> mod_jk or mod_proxy_ajp will behave better?
I'll have to check, but I wouldn't be to surprised, if mod_jk expects
SEND_HEADERS first. Could you please open an issue with bugzilla? That
helps us keeping track.
There's no option at the moment, and I think we need to fix this. At the
moment I'm not sure about where this needs fixing, in JK or in the AJP
connector on the Tomcat side (or both).
It would be very helpful, if you could attach your JK debug log, the JK
configuration and the Tomcat connector configuration to the bug entry.
Is there a chance to get a simple reproduction case (small webapp)?
Regards,
Rainer
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: AJP Flush Packet causing text/plain output
Posted by Rémy Maucherat <re...@gmail.com>.
On 9/24/07, Larry Reisler <la...@vringo.com> wrote:
> We recently switched our development JBOSS instance from 4.05GA to 4.21GA
Which AJP connector is used in JBoss exactly ?
Normally, the two connectors in org.apache.coyote.ajp cannot be doing
what you see.
Rémy
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: AJP Flush Packet causing text/plain output
Posted by Filip Hanik - Dev Lists <de...@hanik.com>.
this is probably what you run into
http://issues.apache.org/bugzilla/show_bug.cgi?id=43478
Filip
Mark Deneen wrote:
> I encountered a similar problem, where one servlet had a bug where the
> headers would be set _after_ the data had been sent. The result was
> that unrelated responses would come back as text/plain. Once I found
> the offending code and fixed it, the issue has not surfaced again.
>
> Mark
>
> On 9/24/07, Larry Reisler <la...@vringo.com> wrote:
>
>> We recently switched our development JBOSS instance from 4.05GA to 4.21GA, where we are have been using mod_jk for connecting an Apache front end server (2.2) to the Tomcat AppServer. We have noticed periodic times when the apache web server will return data with a content-type of plain/text (the server default) instead of the real content-type, and a chunked encoding (even if the AppServer was producing a non-chunked content). We tried switch to mod_proxy_ajp, but got the same result.
>>
>> A bit of sleuthing (2 days X 2 engineers) revealed that the AJP connection on Tomcat is sending a SEND_BODY_CHUNK (which I assume is a flush packet) periodically that seem to be confusing mod_jk. We can see in the mod_jk and mod_proxy_ajp debug log that the correct headers are coming from the Tomcat server. However, it seems like mod_jk and mod_proxy_ajp are losing all the header information regarding the packet (including the Powered by headers, content-type, content-length, cookies, etc.).
>>
>> In normal operation, for a sample small transaction, we would see this sequence in response:
>> SEND_HEADERS
>> SEND_BODY_CHUNK
>> END_RESPONSE
>>
>> When the output would come out as text/plain, we would see this sequence:
>> SEND_BODY_CHUNK
>> SEND_HEADERS
>> SEND_BODY_CHUNK
>> END_RESPONSE
>>
>> Is there something we should have configured differently so that mod_jk or mod_proxy_ajp will behave better?
>>
>>
>>
>>
>>
>
> ---------------------------------------------------------------------
> To start a new topic, e-mail: users@tomcat.apache.org
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
>
>
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: AJP Flush Packet causing text/plain output
Posted by Mark Deneen <md...@gmail.com>.
I encountered a similar problem, where one servlet had a bug where the
headers would be set _after_ the data had been sent. The result was
that unrelated responses would come back as text/plain. Once I found
the offending code and fixed it, the issue has not surfaced again.
Mark
On 9/24/07, Larry Reisler <la...@vringo.com> wrote:
> We recently switched our development JBOSS instance from 4.05GA to 4.21GA, where we are have been using mod_jk for connecting an Apache front end server (2.2) to the Tomcat AppServer. We have noticed periodic times when the apache web server will return data with a content-type of plain/text (the server default) instead of the real content-type, and a chunked encoding (even if the AppServer was producing a non-chunked content). We tried switch to mod_proxy_ajp, but got the same result.
>
> A bit of sleuthing (2 days X 2 engineers) revealed that the AJP connection on Tomcat is sending a SEND_BODY_CHUNK (which I assume is a flush packet) periodically that seem to be confusing mod_jk. We can see in the mod_jk and mod_proxy_ajp debug log that the correct headers are coming from the Tomcat server. However, it seems like mod_jk and mod_proxy_ajp are losing all the header information regarding the packet (including the Powered by headers, content-type, content-length, cookies, etc.).
>
> In normal operation, for a sample small transaction, we would see this sequence in response:
> SEND_HEADERS
> SEND_BODY_CHUNK
> END_RESPONSE
>
> When the output would come out as text/plain, we would see this sequence:
> SEND_BODY_CHUNK
> SEND_HEADERS
> SEND_BODY_CHUNK
> END_RESPONSE
>
> Is there something we should have configured differently so that mod_jk or mod_proxy_ajp will behave better?
>
>
>
>
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org