You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by declan harrison <ha...@gmail.com> on 2010/11/24 19:06:39 UTC
Tomcat 6.0.24 - Should send body As Chunked Response but sometimes
sends it back Un-Chunked
Hi
This is my first post to tomcat user mailing list.
I'm using Tomcat 6.0.24 on Linux RHELS release 5.3 (Tikanga) on a 64 bit OS.
I have deployed a Web application within Tomcat. The Web App in the
main sends dynamically created video content to a client over the
HTTP1.1 Blocking Connector.
I have configured Tomcat to use the Blocking IO HTTP Connector. Main
Configuration parameters are the following maxThreads="1000"
minSpareThreads="300"
The use case is the following. HTTP Client uses HTTP Post to send
video content to a Web App in Tomcat. Tomcat Web App Dynamically
re-codes this video content and sends this video payload in the body
of the HTTP Response (response to HTTP Post) to the client.
As the Web App doesn't know the final size of the video content it
sends so it does not add the HTTP "Content-Length" header in the
HttpServletResponse. The Tomcat HTTP Connector (I presume) correctly
knows to send this content as a HTTP Chunked response, i.e.
Transfer-Encoding: Chunked".
So this work well nearly all of the time, i.e. correct headers are
sent and all the data is sent as chunked response. However when I am
putting a load through the system e.g 40 or 50 concurrent threads
processing content and I make another single HTTP request to the Web
App that is normally rejected with a 500. However before the Tomcat
thread finalizes it response to this specific request I terminate the
client and its request (ctrl-c), then the next time this same Tomcat
thread is re-used it will send the video in the HTTP response body as
Un-Chunked and not Chunked as expected but the HTTP status code is 200
as expected. Also the "Content-Length" header is missing on this
response and all the other HTTP headers that the Web App inserted are
not sent. This then causes problems for the HTTP Client requesting
said content.
However the next time after this that we re-use this thread (3rd time
used) then its back to working normally again sending HTTP Chunked
response as exepcted.
So the normal response is like this. Note chunked response and chunk
size is "2000" hex about 8K.
HTTP/1.1 200 OK
Server: Apache-Coyote/1.1
x-my-session-id: 1000128946853721720696116
Content-Type: video/x-flv
Transfer-Encoding: chunked
Date: Thu, 11 Nov 2010 09:42:17 GMT
2000
onMetaData
totalduration
canseekontime
bytelength
The problems response looks this this. Note no Content-length header
or Chunked response.
HTTP/1.1 200 OK
Server: Apache-Coyote/1.1
Date: Wed, 10 Nov 2010 14:32:25 GMT
Connection: close
FLV...............
To replicate this issue is very much dependant on when I terminate the
client request. If I terminate too late or too early I dont see this
issue. It some sort of race condition.
I upgraded to Tomcat 6.0.29 as I had seen some security issues
relating to Transfer-Encodings, here
http://tomcat.apache.org/security-6.html. However it didnt make any
difference.
I have also read the bug list and cannot see this same issue.
Any ideas what the issue is here and is there anything I can do to mitigate it?
Thanks
Declan
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat 6.0.24 - Should send body As Chunked Response but
sometimes sends it back Un-Chunked
Posted by declan harrison <ha...@gmail.com>.
On Wed, Nov 24, 2010 at 7:59 PM, Christopher Schultz
<ch...@christopherschultz.net> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Pid,
>
> On 11/24/2010 2:32 PM, Pid * wrote:
>> That sounds like the thread is only being reset properly after it's
>> been used a second time.
>
> +1
>
> This can probably be replicated in a test by having a dummy servlet
> simply throw a ClientAbortException (or whatever it is that gets thrown
> when you try to write to a socket whose client has already hung up the
> phone).
>
> Declan, could you try to build a simple servlet that replicates these
> conditions and post it? That will go a long way towards getting this
> investigated and, if it's a problem with Tomcat, ultimately fixed.
>
Hi Christopher
Yes, I can look to build a servlet to do this; to throw an exception
when performing IO on the ServletoutputStream thats been tore down.
Hopefully will get back tomorrow
> - -chris
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.10 (MingW32)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>
> iEYEARECAAYFAkztbrsACgkQ9CaO5/Lv0PAH2gCdH/u7XLeA1Mh6uORcU7PlJQj1
> lIQAn1mhJnnFc2Mo6lH5r0dYU90X4LRK
> =S+Ki
> -----END PGP SIGNATURE-----
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat 6.0.24 - Should send body As Chunked Response but sometimes
sends it back Un-Chunked
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Pid,
On 11/24/2010 2:32 PM, Pid * wrote:
> That sounds like the thread is only being reset properly after it's
> been used a second time.
+1
This can probably be replicated in a test by having a dummy servlet
simply throw a ClientAbortException (or whatever it is that gets thrown
when you try to write to a socket whose client has already hung up the
phone).
Declan, could you try to build a simple servlet that replicates these
conditions and post it? That will go a long way towards getting this
investigated and, if it's a problem with Tomcat, ultimately fixed.
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAkztbrsACgkQ9CaO5/Lv0PAH2gCdH/u7XLeA1Mh6uORcU7PlJQj1
lIQAn1mhJnnFc2Mo6lH5r0dYU90X4LRK
=S+Ki
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat 6.0.24 - Should send body As Chunked Response but
sometimes sends it back Un-Chunked
Posted by declan harrison <ha...@gmail.com>.
On Thu, Nov 25, 2010 at 6:48 PM, Pid <pi...@pidster.com> wrote:
> On 25/11/2010 17:49, declan harrison wrote:
>>> > So far you've shown me that you wrap the output stream, and how the
>>> > write method works.
>>> >
>>> > You haven't actually posted the part of the code where you write your
>>> > bit of the data to the wrapped output stream.
>>> >
>>> >
>>> > p
>> I just call wrapper.write(buf, 0, bug.lenght). Are you looking more
>> source code here?
>
> You are reporting a problem with the response.
>
> You are writing to the output stream, and maybe setting headers on the
> response object.
>
> If you can post each bit of code that interacts with the response object
> we can stop guessing what's wrong and start trying to resolve it.
>
>
> p
>
Apologises about not posting enough of the code, see below for the
methods that sets the respos headers etc
In the method writeResponseHdrs (shown below) we end up reject the
request as bit rate too low and end up executing this code
Log.AdaptVideoError("Video bitrate ["+inputBitRate+"] is
not within range", "Will not encode this video.");
throw new InitialisationException("Video bitrate
["+inputBitRate+"] is not within range. Will not encode this video");
This exception is caught higher up the stack in doPost method which we
then set the status with the following
bufInStream = new AcceleratorInputStream(request.getInputStream(), 20480);
bufInStream.setHeaderRead();
bufOutStream = new AcceleratorOutputStream(response.getOutputStream(),20480);
try {
converter = createVideoAdapter(request, response,
bufInStream,bufOutStream,
request.getContentType(), true,
qualityPolicy);
} catch (Exception ex) {
// caught the exception ok, make sure we have commited the response
response.sendError(HttpServletResponse.SC_INTERNAL_SERVER_ERROR );
====== status code set here and sent in do Post
return;
}
So this is the request that I terminate on the client end before the
respone is actually sent by Tomcat, Im not exactly sure were in Tomcat
the code is when I terminate client request. If I terminate at the
correct point then the next request that re-uses that thread is wrong.
/**
* Write the initial response to the client.
*
* Must write headers before ressponse is committed.
* @throws InitialisationException
* @throws IOException
*
*/
private void writeResponseHdrs() throws InitialisationException,
IOException {
//
// Lets iterate thru the hdrs in the response and
if (responseSet == false) {
setResponseHeaders();
// lets set flag to indicate that hdr have been set
responseSet = true;
//inputStream.setHeaderRead();
outputStream.setHeaderWritten();
// set the status code as well
//
//
if (validateVideoBitate(inputBitRate) == false ) {
//Bitrate check failed - return duration to VR to facilitate JIT
if (writeResponseHdrs == true) {
streamDuration = mIContainer.getDuration();
response.setContentType( mimeType );
if (metaDuration > 0.0d) {
response.addHeader("x-duration",
String.valueOf(metaDuration));
Log.AdaptVideo("Adding video duration header on
BR failure [x-duration: "+
String.valueOf(metaDuration)+"]
extracted from onMetaDataTag.");
} else {
// in microseconds
response.addHeader("x-duration",
String.valueOf(streamDuration/1000000));
Log.AdaptVideo("Adding video duration header on
BR failure [x-duration: "+
String.valueOf(streamDuration/1000000)+"] extracted from container
API.");
}
outputStream.setHeaderWritten();
}
Log.AdaptVideoError("Video bitrate ["+inputBitRate+"] is
not within range", "Will not encode this video.");
throw new InitialisationException("Video bitrate
["+inputBitRate+"] is not within range. Will not encode this video");
}
}
}
private void setResponseHeaders() throws InitialisationException {
Log.AdaptVideoInfo("Adding Http Response headers to help caller");
response.setContentType( mimeType );
// add the following headers for caller
// so that they know what we plan to do
// with the output
// this after initial work
String origLen = request.getHeader("Content-Length");
if (origLen != null) {
response.addHeader("x-orig-content-length", origLen);
Log.AdaptVideo("sending the original length hdr
x-orig-content-length: "+origLen);
} else {
Log.AdaptVideo("Content-Length hdr not present in the request");
}
// add session to manage state for caller e.g. RBA etc
response.addHeader(SESSION_ID, videoSession.key());
Log.AdaptVideo("have added hdr - "+ SESSION_ID +":
"+videoSession.key());
if (metaDuration > 0.0d) {
response.addHeader("x-duration", String.valueOf(metaDuration));
Log.AdaptVideo("video duration header [x-duration: "+
String.valueOf(metaDuration)+"] extracted from
onMetaDataTag.");
} else {
// in microseconds
response.addHeader("x-duration",
String.valueOf(streamDuration/1000000));
Log.AdaptVideo("video duration header [x-duration: "+
String.valueOf(streamDuration/1000000)+"]
extracted from container API.");
}
response.addHeader("x-in-video-codec",getCodecString(mInVideoCodecId));
response.addHeader("x-in-video-fps", String.valueOf(inFrameRate));
response.addHeader("x-in-video-bitrate",
String.valueOf(mInVideoBitRate));
response.addHeader("x-in-audio-codec", getCodecString(mInAudioCodecId));
response.addHeader("x-in-audio-bitrate", String.valueOf(mInAudioBRate));
response.addHeader("x-in-audio-asr", String.valueOf(mInSampleRate));
response.addHeader("x--in-audio-channels",
String.valueOf(mInNumChannels));
}
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat 6.0.24 - Should send body As Chunked Response but sometimes
sends it back Un-Chunked
Posted by Pid <pi...@pidster.com>.
On 25/11/2010 17:49, declan harrison wrote:
>> > So far you've shown me that you wrap the output stream, and how the
>> > write method works.
>> >
>> > You haven't actually posted the part of the code where you write your
>> > bit of the data to the wrapped output stream.
>> >
>> >
>> > p
> I just call wrapper.write(buf, 0, bug.lenght). Are you looking more
> source code here?
You are reporting a problem with the response.
You are writing to the output stream, and maybe setting headers on the
response object.
If you can post each bit of code that interacts with the response object
we can stop guessing what's wrong and start trying to resolve it.
p
Re: Tomcat 6.0.24 - Should send body As Chunked Response but
sometimes sends it back Un-Chunked
Posted by declan harrison <ha...@gmail.com>.
On Thu, Nov 25, 2010 at 11:49 AM, Pid <pi...@pidster.com> wrote:
> On 25/11/2010 11:14, declan harrison wrote:
>> On Thu, Nov 25, 2010 at 10:37 AM, Pid <pi...@pidster.com> wrote:
>>> On 25/11/2010 10:22, declan harrison wrote:
>>>>> ... and how are you writing to the BufferedOutputStream?
>>>>>>
>>>> So I end up calling this method for most of the writes that are
>>>> performed for binary IO.
>>>> @Override
>>>> public synchronized void write(byte[] b, int off, int len)
>>>> throws IOException {
>>>>
>>>> long bTime = System.currentTimeMillis();
>>>> super.write(b, off, len);
>>>> long aTime = System.currentTimeMillis();
>>>> timeDelta += aTime-bTime;
>>>> ++numWrites;
>>>>
>>>> updateSNMPValue(bWrote);
>>>> }
>>>>
>>>
>>> Sorry, how does this answer my question?
>>>
>>>
>> I must have misunderstood your question. What do you mean exactly by
>> the question "and how are you writing to the BufferedOutputStream?"
>>
>> I call the write method above when writing out the binary data in the
>> response body. Is this the information you require or am I missing
>> something? Apoligises if Im being silly on this.
>
> So far you've shown me that you wrap the output stream, and how the
> write method works.
>
> You haven't actually posted the part of the code where you write your
> bit of the data to the wrapped output stream.
>
>
> p
I just call wrapper.write(buf, 0, bug.lenght). Are you looking more
source code here?
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat 6.0.24 - Should send body As Chunked Response but sometimes
sends it back Un-Chunked
Posted by Pid <pi...@pidster.com>.
On 25/11/2010 11:14, declan harrison wrote:
> On Thu, Nov 25, 2010 at 10:37 AM, Pid <pi...@pidster.com> wrote:
>> On 25/11/2010 10:22, declan harrison wrote:
>>>> ... and how are you writing to the BufferedOutputStream?
>>>>>
>>> So I end up calling this method for most of the writes that are
>>> performed for binary IO.
>>> @Override
>>> public synchronized void write(byte[] b, int off, int len)
>>> throws IOException {
>>>
>>> long bTime = System.currentTimeMillis();
>>> super.write(b, off, len);
>>> long aTime = System.currentTimeMillis();
>>> timeDelta += aTime-bTime;
>>> ++numWrites;
>>>
>>> updateSNMPValue(bWrote);
>>> }
>>>
>>
>> Sorry, how does this answer my question?
>>
>>
> I must have misunderstood your question. What do you mean exactly by
> the question "and how are you writing to the BufferedOutputStream?"
>
> I call the write method above when writing out the binary data in the
> response body. Is this the information you require or am I missing
> something? Apoligises if Im being silly on this.
So far you've shown me that you wrap the output stream, and how the
write method works.
You haven't actually posted the part of the code where you write your
bit of the data to the wrapped output stream.
p
Re: Tomcat 6.0.24 - Should send body As Chunked Response but
sometimes sends it back Un-Chunked
Posted by declan harrison <ha...@gmail.com>.
On Thu, Nov 25, 2010 at 10:37 AM, Pid <pi...@pidster.com> wrote:
> On 25/11/2010 10:22, declan harrison wrote:
>>> ... and how are you writing to the BufferedOutputStream?
>>> >
>> So I end up calling this method for most of the writes that are
>> performed for binary IO.
>> @Override
>> public synchronized void write(byte[] b, int off, int len)
>> throws IOException {
>>
>> long bTime = System.currentTimeMillis();
>> super.write(b, off, len);
>> long aTime = System.currentTimeMillis();
>> timeDelta += aTime-bTime;
>> ++numWrites;
>>
>> updateSNMPValue(bWrote);
>> }
>>
>
> Sorry, how does this answer my question?
>
>
I must have misunderstood your question. What do you mean exactly by
the question "and how are you writing to the BufferedOutputStream?"
I call the write method above when writing out the binary data in the
response body. Is this the information you require or am I missing
something? Apoligises if Im being silly on this.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat 6.0.24 - Should send body As Chunked Response but sometimes
sends it back Un-Chunked
Posted by Pid <pi...@pidster.com>.
On 25/11/2010 10:22, declan harrison wrote:
>> ... and how are you writing to the BufferedOutputStream?
>> >
> So I end up calling this method for most of the writes that are
> performed for binary IO.
> @Override
> public synchronized void write(byte[] b, int off, int len)
> throws IOException {
>
> long bTime = System.currentTimeMillis();
> super.write(b, off, len);
> long aTime = System.currentTimeMillis();
> timeDelta += aTime-bTime;
> ++numWrites;
>
> updateSNMPValue(bWrote);
> }
>
Sorry, how does this answer my question?
p
Re: Tomcat 6.0.24 - Should send body As Chunked Response but
sometimes sends it back Un-Chunked
Posted by declan harrison <ha...@gmail.com>.
On Thu, Nov 25, 2010 at 9:45 AM, Pid <pi...@pidster.com> wrote:
> On 24/11/2010 22:11, declan harrison wrote:
>>> How are you writing to the output of the response?
>>> >
>> Im wrapping the response ServletOutputStream in a BufferedOutputStream;
>> Something like this bufOutStream = new
>> BufferedOutputStream(response.getOutputStream(),20480);
>
> ... and how are you writing to the BufferedOutputStream?
>
So I end up calling this method for most of the writes that are
performed for binary IO.
@Override
public synchronized void write(byte[] b, int off, int len)
throws IOException {
long bTime = System.currentTimeMillis();
super.write(b, off, len);
long aTime = System.currentTimeMillis();
timeDelta += aTime-bTime;
++numWrites;
updateSNMPValue(bWrote);
}
Im not clear when I call response.addHeader and response.sendError,
what or if the BufferedOutput stream IO is called by Tomcat Coyote.
When I debug the Web App I see that BufferedOutputStream underlying
OutputStream (protected) is in fact a CoyoteOutputStream (I can access
this object) and it contains a protected object called
org.apache.catalina.connector.OutputBuffer. I would dearly like to
get at this objects data but I cannot as its protected, this contains
the headers and details about the stream and its state.
Is there a way to dump out this object OutputBuffer details. Is there
any debug logs I can enable here?
Thanks
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat 6.0.24 - Should send body As Chunked Response but sometimes
sends it back Un-Chunked
Posted by Pid <pi...@pidster.com>.
On 24/11/2010 22:11, declan harrison wrote:
>> How are you writing to the output of the response?
>> >
> Im wrapping the response ServletOutputStream in a BufferedOutputStream;
> Something like this bufOutStream = new
> BufferedOutputStream(response.getOutputStream(),20480);
... and how are you writing to the BufferedOutputStream?
p
Re: Tomcat 6.0.24 - Should send body As Chunked Response but
sometimes sends it back Un-Chunked
Posted by declan harrison <ha...@gmail.com>.
On Thu, Nov 25, 2010 at 3:08 PM, Caldarale, Charles R
<Ch...@unisys.com> wrote:
>> From: declan harrison [mailto:harrison.declan@gmail.com]
>> Subject: Re: Tomcat 6.0.24 - Should send body As Chunked Response but sometimes sends it back Un-Chunked
>
>> it does look like it to me that state is leaking between
>> these two request on this same thread.
>
> That hints of a ThreadLocal being used somewhere in your code (or a 3rd-party library). That's usually not a good idea in a pooled environment.
>
Must be in 3rd party code as I dont use any ThreadLocal data explicitly myself.
>> Is there a way of obliterating a thread on Tomcat
>
> Not that I'm aware of. You could temporarily set clearReferencesThreadLocals to true in your <Context> element to see if that takes care of the problem, but you probably don't want to do that in production (not really thread-safe).
Yea not sure. I was wondering it might be worth trying to use the APR
Connector and see if the issue is the present there also or hopefully
not. you think its worth a shot.
Im also looking at trying to produce a simple servlet where I can
reproduce this problem and send it to the mailing list. I'm hopeful I
can get such a simple servlet
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Tomcat 6.0.24 - Should send body As Chunked Response but
sometimes sends it back Un-Chunked
Posted by "Caldarale, Charles R" <Ch...@unisys.com>.
> From: declan harrison [mailto:harrison.declan@gmail.com]
> Subject: Re: Tomcat 6.0.24 - Should send body As Chunked Response but sometimes sends it back Un-Chunked
> it does look like it to me that state is leaking between
> these two request on this same thread.
That hints of a ThreadLocal being used somewhere in your code (or a 3rd-party library). That's usually not a good idea in a pooled environment.
> Is there a way of obliterating a thread on Tomcat
Not that I'm aware of. You could temporarily set clearReferencesThreadLocals to true in your <Context> element to see if that takes care of the problem, but you probably don't want to do that in production (not really thread-safe).
- Chuck
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat 6.0.24 - Should send body As Chunked Response but
sometimes sends it back Un-Chunked
Posted by declan harrison <ha...@gmail.com>.
On Wed, Nov 24, 2010 at 10:39 PM, Caldarale, Charles R
<Ch...@unisys.com> wrote:
>> From: declan harrison [mailto:harrison.declan@gmail.com]
>> Subject: Re: Tomcat 6.0.24 - Should send body As Chunked Response but sometimes sends it back Un-Chunked
>
>> I meant I have local variable that reference the response
>> output stream.
>
> Ok, you're off the hook on that one. Anything stored in the session that might allow intermingling of requests or responses?
>
Chuck
I have reviewed the code this morning and I canny see anything that
would allow the intermingling of the request or response between
separate HTTP requests but it does look like it to me that state is
leaking between these two request on this same thread.
Is there a way of obliterating a thread on Tomcat, this way I could
verify that this indeed is the issue.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Tomcat 6.0.24 - Should send body As Chunked Response but
sometimes sends it back Un-Chunked
Posted by "Caldarale, Charles R" <Ch...@unisys.com>.
> From: declan harrison [mailto:harrison.declan@gmail.com]
> Subject: Re: Tomcat 6.0.24 - Should send body As Chunked Response but sometimes sends it back Un-Chunked
> I meant I have local variable that reference the response
> output stream.
Ok, you're off the hook on that one. Anything stored in the session that might allow intermingling of requests or responses?
- Chuck
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat 6.0.24 - Should send body As Chunked Response but
sometimes sends it back Un-Chunked
Posted by declan harrison <ha...@gmail.com>.
On Wed, Nov 24, 2010 at 10:23 PM, Caldarale, Charles R
<Ch...@unisys.com> wrote:
> From: declan harrison [mailto:harrison.declan@gmail.com]
>> Subject: Re: Tomcat 6.0.24 - Should send body As Chunked Response but sometimes sends it back Un-Chunked
>
>> > Do you have any part of the response stored as an
>> > instance field in your servlet?
>
>> Yes I have a reference to the HttpServletResponse,
>> response.getOutputStream(), which I close when Im done.
>
> Fatal error - don't ever do that. Multiple threads running through your servlet will walk over each other. It's fine to have local variables pointing to objects related to a request or response, but having instance or static variables do so is guaranteed to cause problems. Remember that each servlet (or filter) is instantiated *once* for the lifetime of the webapp, not once per request.
>
> - Chuck
Hi Chuck
Sorry for my misleading statement I meant I have local variable that
reference the response output stream. I dont have a servlet attribute
that holds a reference to the response stream
>
>
> THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Tomcat 6.0.24 - Should send body As Chunked Response but
sometimes sends it back Un-Chunked
Posted by "Caldarale, Charles R" <Ch...@unisys.com>.
From: declan harrison [mailto:harrison.declan@gmail.com]
> Subject: Re: Tomcat 6.0.24 - Should send body As Chunked Response but sometimes sends it back Un-Chunked
> > Do you have any part of the response stored as an
> > instance field in your servlet?
> Yes I have a reference to the HttpServletResponse,
> response.getOutputStream(), which I close when Im done.
Fatal error - don't ever do that. Multiple threads running through your servlet will walk over each other. It's fine to have local variables pointing to objects related to a request or response, but having instance or static variables do so is guaranteed to cause problems. Remember that each servlet (or filter) is instantiated *once* for the lifetime of the webapp, not once per request.
- Chuck
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat 6.0.24 - Should send body As Chunked Response but
sometimes sends it back Un-Chunked
Posted by declan harrison <ha...@gmail.com>.
On Wed, Nov 24, 2010 at 7:32 PM, Pid * <pi...@pidster.com> wrote:
> On 24 Nov 2010, at 18:07, declan harrison <ha...@gmail.com> wrote:
>
>> Hi
>>
>> This is my first post to tomcat user mailing list.
>
> Hello.
Hi there
>
>> I'm using Tomcat 6.0.24 on Linux RHELS release 5.3 (Tikanga) on a 64 bit OS.
>
> Which JVM version?
java version "1.6.0_18"
Java(TM) SE Runtime Environment (build 1.6.0_18-b07)
Java HotSpot(TM) 64-Bit Server VM (build 16.0-b13, mixed mode)
>
>> I have deployed a Web application within Tomcat. The Web App in the
>> main sends dynamically created video content to a client over the
>> HTTP1.1 Blocking Connector.
>
> How are you writing to the output of the response?
>
Im wrapping the response ServletOutputStream in a BufferedOutputStream;
Something like this bufOutStream = new
BufferedOutputStream(response.getOutputStream(),20480);
>> I have configured Tomcat to use the Blocking IO HTTP Connector. Main
>> Configuration parameters are the following maxThreads="1000"
>> minSpareThreads="300"
>>
>> The use case is the following. HTTP Client uses HTTP Post to send
>> video content to a Web App in Tomcat. Tomcat Web App Dynamically
>> re-codes this video content and sends this video payload in the body
>> of the HTTP Response (response to HTTP Post) to the client.
>>
>> As the Web App doesn't know the final size of the video content it
>> sends so it does not add the HTTP "Content-Length" header in the
>> HttpServletResponse. The Tomcat HTTP Connector (I presume) correctly
>> knows to send this content as a HTTP Chunked response, i.e.
>> Transfer-Encoding: Chunked".
>>
>> So this work well nearly all of the time, i.e. correct headers are
>> sent and all the data is sent as chunked response. However when I am
>> putting a load through the system e.g 40 or 50 concurrent threads
>> processing content and I make another single HTTP request to the Web
>> App that is normally rejected with a 500.
>
> What us the cause, and is there an exception stacktrace in the logs?
It is expected for my Web App to reject this request with HTTP status
code 500. The input request content is invalid. So typically the Web
App will log the problem with the input content and then call
response.sendError(HttpServletResponse.SC_INTERNAL_SERVER_ERROR );
I dont have an error code setup in the web.xml file.
After having setup some HTTP response headers and then return from the
doPost method.
There is no exception backtrace as such, logged by tomcat as far as I can see.
do you think that Tomcat itself is logging a stacktrace somewhere.
Theres nothing in my log file apart from the expected warning/error
message that the Web App log explicitly.
>
>> However before the Tomcat
>> thread finalizes it response to this specific request I terminate the
>> client and its request (ctrl-c), then the next time this same Tomcat
>> thread is re-used it will send the video in the HTTP response body as
>> Un-Chunked and not Chunked as expected but the HTTP status code is 200
>
> How are you determining that it is the same thread?
So in my Web App log file i log the client IP address and Port. So
for the client request which I terminate manually I perform this from
a different IP address than the one were all the other requests are
made from. The log file also logs the ThreadId of each log event e.g.
Nov 22 17:54:19 hostname [ProcessName][1001][26348][tomcat-exec-243]
INFO Bla Bla...
Here you can see "tomcat-exec-243" is the thread Id.
So then the next time a log event is performed that matches this
threadId I see that it fails.
>
>> as expected. Also the "Content-Length" header is missing on this
>> response and all the other HTTP headers that the Web App inserted are
>> not sent. This then causes problems for the HTTP Client requesting
>> said content.
>>
>> However the next time after this that we re-use this thread (3rd time
>> used) then its back to working normally again sending HTTP Chunked
>> response as exepcted.
>
> That sounds like the thread is only being reset properly after it's
> been used a second time.
>
Yes that correct.
It appears to me that the state of the output stream in the thread is
all setup to send a non chunked response as it would with the 500
error it didnt send and so sends the video response non chunked and
its only after that does it resset its state.
Does each thread maintain state between different HTTP requests.
I also tried increasing linger time on each socket. So I tried 5
seconds and right enough it took 5 or more seconds before Tomcat
re-used this thread but the bad result was the same.
>> So the normal response is like this. Note chunked response and chunk
>> size is "2000" hex about 8K.
>>
>> HTTP/1.1 200 OK
>> Server: Apache-Coyote/1.1
>> x-my-session-id: 1000128946853721720696116
>> Content-Type: video/x-flv
>> Transfer-Encoding: chunked
>> Date: Thu, 11 Nov 2010 09:42:17 GMT
>>
>> 2000
>> onMetaData
>> totalduration
>> canseekontime
>> bytelength
>>
>> The problems response looks this this. Note no Content-length header
>> or Chunked response.
>> HTTP/1.1 200 OK
>> Server: Apache-Coyote/1.1
>> Date: Wed, 10 Nov 2010 14:32:25 GMT
>> Connection: close
>>
>> FLV...............
>>
>> To replicate this issue is very much dependant on when I terminate the
>> client request. If I terminate too late or too early I dont see this
>> issue. It some sort of race condition.
>
> Do you still see a 500 when you terminate, if so is it the same stacktrace?
I dont get a stack trace theres no unexpected error and the 500 status
isnt actually returned to the client because I have terminated the
request before response sent by Web App. Im not sure where in the
Tomcat code we are when the request is tore down.
>
> Do you have any part of the response stored as an instance field in
> your servlet?
Yes I have a reference to the HttpServletResponse,
response.getOutputStream(), which I close when Im done.
>
>
> p
>
>> I upgraded to Tomcat 6.0.29 as I had seen some security issues
>> relating to Transfer-Encodings, here
>> http://tomcat.apache.org/security-6.html. However it didnt make any
>> difference.
>>
>> I have also read the bug list and cannot see this same issue.
>>
>> Any ideas what the issue is here and is there anything I can do to mitigate it?
>>
>> Thanks
>> Declan
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat 6.0.24 - Should send body As Chunked Response but
sometimes sends it back Un-Chunked
Posted by Pid * <pi...@pidster.com>.
On 24 Nov 2010, at 18:07, declan harrison <ha...@gmail.com> wrote:
> Hi
>
> This is my first post to tomcat user mailing list.
Hello.
> I'm using Tomcat 6.0.24 on Linux RHELS release 5.3 (Tikanga) on a 64 bit OS.
Which JVM version?
> I have deployed a Web application within Tomcat. The Web App in the
> main sends dynamically created video content to a client over the
> HTTP1.1 Blocking Connector.
How are you writing to the output of the response?
> I have configured Tomcat to use the Blocking IO HTTP Connector. Main
> Configuration parameters are the following maxThreads="1000"
> minSpareThreads="300"
>
> The use case is the following. HTTP Client uses HTTP Post to send
> video content to a Web App in Tomcat. Tomcat Web App Dynamically
> re-codes this video content and sends this video payload in the body
> of the HTTP Response (response to HTTP Post) to the client.
>
> As the Web App doesn't know the final size of the video content it
> sends so it does not add the HTTP "Content-Length" header in the
> HttpServletResponse. The Tomcat HTTP Connector (I presume) correctly
> knows to send this content as a HTTP Chunked response, i.e.
> Transfer-Encoding: Chunked".
>
> So this work well nearly all of the time, i.e. correct headers are
> sent and all the data is sent as chunked response. However when I am
> putting a load through the system e.g 40 or 50 concurrent threads
> processing content and I make another single HTTP request to the Web
> App that is normally rejected with a 500.
What us the cause, and is there an exception stacktrace in the logs?
> However before the Tomcat
> thread finalizes it response to this specific request I terminate the
> client and its request (ctrl-c), then the next time this same Tomcat
> thread is re-used it will send the video in the HTTP response body as
> Un-Chunked and not Chunked as expected but the HTTP status code is 200
How are you determining that it is the same thread?
> as expected. Also the "Content-Length" header is missing on this
> response and all the other HTTP headers that the Web App inserted are
> not sent. This then causes problems for the HTTP Client requesting
> said content.
>
> However the next time after this that we re-use this thread (3rd time
> used) then its back to working normally again sending HTTP Chunked
> response as exepcted.
That sounds like the thread is only being reset properly after it's
been used a second time.
> So the normal response is like this. Note chunked response and chunk
> size is "2000" hex about 8K.
>
> HTTP/1.1 200 OK
> Server: Apache-Coyote/1.1
> x-my-session-id: 1000128946853721720696116
> Content-Type: video/x-flv
> Transfer-Encoding: chunked
> Date: Thu, 11 Nov 2010 09:42:17 GMT
>
> 2000
> onMetaData
> totalduration
> canseekontime
> bytelength
>
> The problems response looks this this. Note no Content-length header
> or Chunked response.
> HTTP/1.1 200 OK
> Server: Apache-Coyote/1.1
> Date: Wed, 10 Nov 2010 14:32:25 GMT
> Connection: close
>
> FLV...............
>
> To replicate this issue is very much dependant on when I terminate the
> client request. If I terminate too late or too early I dont see this
> issue. It some sort of race condition.
Do you still see a 500 when you terminate, if so is it the same stacktrace?
Do you have any part of the response stored as an instance field in
your servlet?
p
> I upgraded to Tomcat 6.0.29 as I had seen some security issues
> relating to Transfer-Encodings, here
> http://tomcat.apache.org/security-6.html. However it didnt make any
> difference.
>
> I have also read the bug list and cannot see this same issue.
>
> Any ideas what the issue is here and is there anything I can do to mitigate it?
>
> Thanks
> Declan
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org