You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mina.apache.org by Tuure Laurinolli <tu...@indagon.com> on 2007/12/12 03:19:42 UTC
HttpResponseEncoder and streaming data
Good morning,
Currently HttpResponseEncoder only seems to support HttpResponses that
contain all the data that is going to be in the response. This clearly
does not work for streaming data.
I needed immediate support for streaming data, so I added a
ChunkedHttpResponseEncoder that supports sending arbitrary data as part
of the response after the actual HttpResponse-object is sent. I think my
interface is ugly, so hopefully someone can come up with a better idea.
It seems clear to me that streaming data should be supported.
Description of my encoder:
ChunkedHttpResponseEncoder first expects an HttpResponse, and if the
HttpResponse has Transfer-encoding: chunked set, switches to expecting
HttpChunks instead. HttpChunks are simply containers for IoBuffers, and
are simply written to the Iosession, prefixed with the appropriate chunk
header. The response is terminated with a special END_TOKEN objec. When
the encoder receives an END_TOKEN, it switches back to expecting
HttpResponses.
If the encoder receives an inappropriate object for its current state
(END_TOKEN or HttpChunk when expecting HttpResponse, or HttpResponse
when expecting HttpChunk or END_TOKEN, it throws an exception).
Tuure Laurinolli
Re: HttpResponseEncoder and streaming data
Posted by Francesca Milan <fr...@newvision.it>.
Bogdan Ciprian Pistol ha scritto:
>> Chunked messages are standard :)
>>
>
> :), yes indeed. I should refer to non chunked messages by another
> name, ... but I cannot recall how the authors of RFC2616 are referring
> to "standard" http.
>
> Bogdan
>
Hi,
this is a HTTP request:
POST /open/1 HTTP/1.1
Content-Type: application/x-fcs
User-Agent: Shockwave Flash
Host: 127.0.0.1:8088
Content-Length: 1
Connection: Keep-Alive
Cache-Control: no-cache
...if you're studying RTMPT protocol ;-)
--
Francesca
Re: HttpResponseEncoder and streaming data
Posted by Bogdan Ciprian Pistol <bo...@gmail.com>.
> Chunked messages are standard :)
:), yes indeed. I should refer to non chunked messages by another
name, ... but I cannot recall how the authors of RFC2616 are referring
to "standard" http.
Bogdan
Re: HttpResponseEncoder and streaming data
Posted by Tuure Laurinolli <tu...@indagon.com>.
Bogdan Ciprian Pistol wrote:
> Tuure, why standard HTTP messages cannot be used for streaming? Why
> chunked messages are more appropriate?
Chunked messages are standard :) My understanding is that each HTTP
response is a response to a specific request. Thus streaming of data
over long periods of time (for delivery of a large file, for example)
cannot be achieved with multiple responses unless the client is prepared
to make multiple requests.
For sending large files, the current interface would be sufficient. With
a custom IoBuffer implementation, backed by a file. However, for
streaming of data that is produced on the server, and the exact size of
which is unknown, this isn't useful.
Re: HttpResponseEncoder and streaming data
Posted by Bogdan Ciprian Pistol <bo...@gmail.com>.
Hi Trustin,
> We could make our current class hierarchy more granular like the following:
>
> * HttpMessage
> ** HttpRequest - no getContent()
> *** MergedHttpRequest - getContent() here
> *** ChunkedHttpRequest - once written, encoder expects HttpChunks,
> other HttpRequest or disconnection.
> ** HttpResponse - no getContent()
> *** MergedHttpResponse - getContent() here
> *** ChunkedHttpResponse - once written, encoder expects HttpChunks,
> other HttpResponse or disconnection.
> * HttpChunk - getContent() here
Looks great.
Why is MergedHttpRequest needed? If someone wants to write a chunked
message he writes some ChunkedHttpRequests messages.
MergedHttpResponse could be returned as a normal HTTP message with
getContent( ).
If the application is interested about the merged HTTP chunks then it
is interested about the hole message so it could receive the message
as a standard HTTP message (from de decoder).
The application will not care if it's merged HTTP or standard HTTP in
this case, IMO.
I like the idea to first send ChunkedHttpResponse and subsequent
chunks as HttpChunks, (in my decoder I used only one class).
Bogdan
Re: HttpResponseEncoder and streaming data
Posted by Trustin Lee <tr...@gmail.com>.
On Dec 16, 2007 1:51 AM, Bogdan Ciprian Pistol <bo...@gmail.com> wrote:
> Hi all,
>
> On Dec 13, 2007 7:50 PM, Trustin Lee <tr...@gmail.com> wrote:
> > It seems like we now have some reasonable approaches on streaming
> > large object now. I think all of them are valid, but I also would
> > like you to think about receiving a large object. We need to resolve
> > both encoding and decoding and the resolution should be symmetric so
> > that it's easy to learn. Consequently, Dave's idea is providing some
> > clue to improve the ideas of Tuure and Bogdan. It would be really
> > exciting to see the best combination of the three ideas via
> > cooperation.
>
> I wrote only the server part (the encoder), the server is accessed by
> standard web browsers. :)
>
> This is how I see a client receiving chunked messages:
>
> The client receiving the chunked message can wait until the hole
> chunked message is received (how the browsers are handling chunked
> messages)
> and then the decoder would pass the message as a normal HTTP message
> to the application.
> This assumes that for the application the message has a meaning as a
> hole (all the chunked messages together), individual chunks are
> meaningless.
>
> The client could also decode every chunked message as an individual
> part of a stream of messages and the decoder will pass to the
> application every chunked message.
> This assumes that the application can handle every chunk individually
> and every chunk has a meaning (it can be individually evaluated by the
> application).
>
> The two ways of handling chunked messages should be configurable. An
> application should manifest it's interest in a what way to handle the
> received chunked messages.
Makes a lot of sense. We need to make it configurable and retain the
current behavior unless it's configured differently from the default.
> Tuure, why standard HTTP messages cannot be used for streaming? Why
> chunked messages are more appropriate?
> If you send chunk after chunk in the same HTTP response or if you send
> multiple standard HTTP responses it is almost the same (a standard
> HTTP message has just some optional additional headers and that first
> status line).
We could make our current class hierarchy more granular like the following:
* HttpMessage
** HttpRequest - no getContent()
*** MergedHttpRequest - getContent() here
*** ChunkedHttpRequest - once written, encoder expects HttpChunks,
other HttpRequest or disconnection.
** HttpResponse - no getContent()
*** MergedHttpResponse - getContent() here
*** ChunkedHttpResponse - once written, encoder expects HttpChunks,
other HttpResponse or disconnection.
* HttpChunk - getContent() here
Most users will be OK with MergedHttpRequest and MergedHttpResponse
and some people who wants to stream a big message will use
ChunkedHttpRequest, ChunkedResponse and HttpChunk. Additionally, we
won't need END_TOKEN because the encoder implementation can determine
the end of the chunks easily.
For decoding, if the decoder is configured for chunking, the decoder
will produce ChunkedHttpRequest and HttpChunks or ChunkedHttpResponse
and HttpChunks. If not configured for chunking, then just
MergedHttpRequest or MergedHttpResponse will be produced, which is the
default behavior.
I am not sure about the class name prefix 'Merged' though. Any better
name suggestion is appreciated.
What do you think about this solution?
Trustin
--
what we call human nature is actually human habit
--
http://gleamynode.net/
--
PGP Key ID: 0x0255ECA6
Re: HttpResponseEncoder and streaming data
Posted by Bogdan Ciprian Pistol <bo...@gmail.com>.
Hi all,
On Dec 13, 2007 7:50 PM, Trustin Lee <tr...@gmail.com> wrote:
> It seems like we now have some reasonable approaches on streaming
> large object now. I think all of them are valid, but I also would
> like you to think about receiving a large object. We need to resolve
> both encoding and decoding and the resolution should be symmetric so
> that it's easy to learn. Consequently, Dave's idea is providing some
> clue to improve the ideas of Tuure and Bogdan. It would be really
> exciting to see the best combination of the three ideas via
> cooperation.
I wrote only the server part (the encoder), the server is accessed by
standard web browsers. :)
This is how I see a client receiving chunked messages:
The client receiving the chunked message can wait until the hole
chunked message is received (how the browsers are handling chunked
messages)
and then the decoder would pass the message as a normal HTTP message
to the application.
This assumes that for the application the message has a meaning as a
hole (all the chunked messages together), individual chunks are
meaningless.
The client could also decode every chunked message as an individual
part of a stream of messages and the decoder will pass to the
application every chunked message.
This assumes that the application can handle every chunk individually
and every chunk has a meaning (it can be individually evaluated by the
application).
The two ways of handling chunked messages should be configurable. An
application should manifest it's interest in a what way to handle the
received chunked messages.
Tuure, why standard HTTP messages cannot be used for streaming? Why
chunked messages are more appropriate?
If you send chunk after chunk in the same HTTP response or if you send
multiple standard HTTP responses it is almost the same (a standard
HTTP message has just some optional additional headers and that first
status line).
Bogdan
Re: HttpResponseEncoder and streaming data
Posted by Trustin Lee <tr...@gmail.com>.
Hi folks,
It seems like we now have some reasonable approaches on streaming
large object now. I think all of them are valid, but I also would
like you to think about receiving a large object. We need to resolve
both encoding and decoding and the resolution should be symmetric so
that it's easy to learn. Consequently, Dave's idea is providing some
clue to improve the ideas of Tuure and Bogdan. It would be really
exciting to see the best combination of the three ideas via
cooperation.
Cheers,
Trustin
On Dec 13, 2007 3:16 PM, Bogdan Ciprian Pistol <bo...@gmail.com> wrote:
> On Dec 12, 2007 4:19 AM, Tuure Laurinolli <tu...@indagon.com> wrote:
> > Description of my encoder:
> >
> > ChunkedHttpResponseEncoder first expects an HttpResponse, and if the
> > HttpResponse has Transfer-encoding: chunked set, switches to expecting
> > HttpChunks instead. HttpChunks are simply containers for IoBuffers, and
> > are simply written to the Iosession, prefixed with the appropriate chunk
> > header. The response is terminated with a special END_TOKEN objec. When
> > the encoder receives an END_TOKEN, it switches back to expecting
> > HttpResponses.
> >
> > If the encoder receives an inappropriate object for its current state
> > (END_TOKEN or HttpChunk when expecting HttpResponse, or HttpResponse
> > when expecting HttpChunk or END_TOKEN, it throws an exception).
> >
> > Tuure Laurinolli
> >
>
> I wrote several weeks ago an encoder, named HttpEncoder, that supports
> standard HTTP and chunked HTTP messages (among others :) ).
> It can handle ChunkedHttpMessage or HttpMessage classes.
> So when you send a HttpMessage then it just sends that message, if you
> send a ChunkedHttpMessage it uses ChunkedHttpMessage.isFirstChunk( )
> to handle the chunked protocol correctly.
> The last chunk is not a token but an empty ChunkedHttpMessage (with a
> null body), and when HttpEncoder sends this empty last chunk it is as
> the HTTP specification says that the last chunked should be a zero
> body size chunk.
>
> It is the ChunkedHttpMessage that keeps its state, whether it's the
> first message or not, if the encoder would do this then I guess that
> you couldn't simultaneously send standard HTTP and chunked HTTP with
> the same encoder (when multiple sessions are using the same encoder),
> if you use different encoders then I guess you could send them
> simultaneously.
>
> Bogdan
>
--
what we call human nature is actually human habit
--
http://gleamynode.net/
--
PGP Key ID: 0x0255ECA6
Re: HttpResponseEncoder and streaming data
Posted by Tuure Laurinolli <tu...@indagon.com>.
Bogdan Ciprian Pistol wrote:
> On Dec 12, 2007 4:19 AM, Tuure Laurinolli <tu...@indagon.com> wrote:
> the HTTP specification says that the last chunked should be a zero
> body size chunk.
This is how END_TOKEN is handled by me.
> It is the ChunkedHttpMessage that keeps its state, whether it's the
> first message or not, if the encoder would do this then I guess that
> you couldn't simultaneously send standard HTTP and chunked HTTP with
> the same encoder
I was a bit unclear in my original post. The state of the encoding is
naturally kept in the IoSession, not the encoder.
On the whole your whole implementation sounds fairly similar to mine.
Instead of such a trivial implementation, I think the ideas in
ASYNCWEB-21 would provide some nice abstractions (like the possibility
of IoFilter-like functionality for decoding protocol messages from the
HTTP bodies). Care would have to be taken not to complicate the common
case of simple requests/responses that don't need streaming needlessly,
though.
Tuure Laurinolli
Re: HttpResponseEncoder and streaming data
Posted by Bogdan Ciprian Pistol <bo...@gmail.com>.
On Dec 12, 2007 4:19 AM, Tuure Laurinolli <tu...@indagon.com> wrote:
> Description of my encoder:
>
> ChunkedHttpResponseEncoder first expects an HttpResponse, and if the
> HttpResponse has Transfer-encoding: chunked set, switches to expecting
> HttpChunks instead. HttpChunks are simply containers for IoBuffers, and
> are simply written to the Iosession, prefixed with the appropriate chunk
> header. The response is terminated with a special END_TOKEN objec. When
> the encoder receives an END_TOKEN, it switches back to expecting
> HttpResponses.
>
> If the encoder receives an inappropriate object for its current state
> (END_TOKEN or HttpChunk when expecting HttpResponse, or HttpResponse
> when expecting HttpChunk or END_TOKEN, it throws an exception).
>
> Tuure Laurinolli
>
I wrote several weeks ago an encoder, named HttpEncoder, that supports
standard HTTP and chunked HTTP messages (among others :) ).
It can handle ChunkedHttpMessage or HttpMessage classes.
So when you send a HttpMessage then it just sends that message, if you
send a ChunkedHttpMessage it uses ChunkedHttpMessage.isFirstChunk( )
to handle the chunked protocol correctly.
The last chunk is not a token but an empty ChunkedHttpMessage (with a
null body), and when HttpEncoder sends this empty last chunk it is as
the HTTP specification says that the last chunked should be a zero
body size chunk.
It is the ChunkedHttpMessage that keeps its state, whether it's the
first message or not, if the encoder would do this then I guess that
you couldn't simultaneously send standard HTTP and chunked HTTP with
the same encoder (when multiple sessions are using the same encoder),
if you use different encoders then I guess you could send them
simultaneously.
Bogdan
RE: HttpResponseEncoder and streaming data
Posted by "Irving, Dave" <da...@bankofamerica.com>.
Tuure Laurinolli wrote:
> Good morning,
> Currently HttpResponseEncoder only seems to support HttpResponses
> that contain all the data that is going to be in the response.
> This clearly does not work for streaming data.
I had some ideas on this a while back. Although I think most of the info
is related to -requests-, the same ideas could be applied to
responses...
http://jira.safehaus.org/browse/ASYNCWEB-21
> ...
> Tuure Laurinolli
Dave