You are viewing a plain text version of this content. The canonical link for it is here.
Posted to httpclient-users@hc.apache.org by Chun Tat David Chu <be...@gmail.com> on 2013/10/09 19:58:04 UTC
Consuming chunk encoded data
Hi all,
Is there an utility or a technique in HTTP Client that allows you to
consume chunk encoded data whenever one chunk arrived on the wire? Perhaps
I overlooked this but I couldn't find anything related with this topic.
Here's what coming in over the wire.
HttpResponseHandlerImpl.setContentType(): application/json
HttpResponseHandlerImpl.setContentLength(): -1
58495 [main] DEBUG org.apache.http.wire - << "24[\r][\n]"
58495 [main] DEBUG org.apache.http.wire - <<
"{"firstName":"David","lastName":"0"}"
58495 [main] DEBUG org.apache.http.wire - << "[\r][\n]"
58495 [main] DEBUG org.apache.http.wire - << "24[\r][\n]"
58495 [main] DEBUG org.apache.http.wire - <<
"{"firstName":"David","lastName":"1"}"
58495 [main] DEBUG org.apache.http.wire - << "[\r][\n]"
58495 [main] DEBUG org.apache.http.wire - << "24[\r][\n]"
58495 [main] DEBUG org.apache.http.wire - <<
"{"firstName":"David","lastName":"2"}"
58501 [main] DEBUG org.apache.http.wire - << "[\r][\n]"
58501 [main] DEBUG org.apache.http.wire - << "0[\r][\n]"
58501 [main] DEBUG org.apache.http.wire - << "[\r][\n]"
What I want to do is to process a chunk at a time. Is there any helper
method that allows you to do this? or I just need to get the InputStream
and do this manually? Manually as in get the chunk size then read the
buffer based on the chunk size.
Thanks!
David
Re: Consuming chunk encoded data
Posted by Chun Tat David Chu <be...@gmail.com>.
Hi Oleg,
Thanks for your explanation. Instead of doing what I was asking for which
is take a complete chunk at a time. I ended up with using a JsonParser
from Jackson to process the stream since I know all the response from the
payload would be JSON objects.
Thanks,
David
On Thu, Oct 10, 2013 at 7:56 AM, Oleg Kalnichevski <ol...@apache.org> wrote:
> On Thu, 2013-10-10 at 16:47 +0700, Alexey Panchenko wrote:
> > Hi,
> >
> > Would it be possible to use HttpAsyncClient for this task, e.g. like here
> >
> http://hc.apache.org/httpcomponents-asyncclient-dev/httpasyncclient/examples/org/apache/http/examples/nio/client/AsyncClientHttpExchangeStreaming.java
> > ?
> >
> > Regards,
> > Alex
> >
>
> I do not think so. While I _suspect_ this _might_ work for smaller
> chunks that fit into the session input buffer in their entirety I am
> absolutely sure this approach would fail for large chunks. At any rate
> the design would be too brittle to be used for anything remotely
> serious.
>
> Oleg
>
> >
> >
> > On Thu, Oct 10, 2013 at 3:21 PM, Oleg Kalnichevski <ol...@apache.org>
> wrote:
> >
> > > On Wed, 2013-10-09 at 13:58 -0400, Chun Tat David Chu wrote:
> > > > Hi all,
> > > >
> > > > Is there an utility or a technique in HTTP Client that allows you to
> > > > consume chunk encoded data whenever one chunk arrived on the wire?
> > > Perhaps
> > > > I overlooked this but I couldn't find anything related with this
> topic.
> > > >
> > > > Here's what coming in over the wire.
> > > > HttpResponseHandlerImpl.setContentType(): application/json
> > > > HttpResponseHandlerImpl.setContentLength(): -1
> > > > 58495 [main] DEBUG org.apache.http.wire - << "24[\r][\n]"
> > > > 58495 [main] DEBUG org.apache.http.wire - <<
> > > > "{"firstName":"David","lastName":"0"}"
> > > > 58495 [main] DEBUG org.apache.http.wire - << "[\r][\n]"
> > > > 58495 [main] DEBUG org.apache.http.wire - << "24[\r][\n]"
> > > > 58495 [main] DEBUG org.apache.http.wire - <<
> > > > "{"firstName":"David","lastName":"1"}"
> > > > 58495 [main] DEBUG org.apache.http.wire - << "[\r][\n]"
> > > > 58495 [main] DEBUG org.apache.http.wire - << "24[\r][\n]"
> > > > 58495 [main] DEBUG org.apache.http.wire - <<
> > > > "{"firstName":"David","lastName":"2"}"
> > > > 58501 [main] DEBUG org.apache.http.wire - << "[\r][\n]"
> > > > 58501 [main] DEBUG org.apache.http.wire - << "0[\r][\n]"
> > > > 58501 [main] DEBUG org.apache.http.wire - << "[\r][\n]"
> > > >
> > > > What I want to do is to process a chunk at a time. Is there any
> helper
> > > > method that allows you to do this? or I just need to get the
> InputStream
> > > > and do this manually? Manually as in get the chunk size then read
> the
> > > > buffer based on the chunk size.
> > > >
> > > > Thanks!
> > > >
> > > > David
> > >
> > > David,
> > > Chunk coding as well as other transfer encoding schemes are supposed to
> > > be completely transparent to the application layer and therefore
> > > HttpClient exposes no API related to transfer encodings.
> > >
> > > If you are absolutely convinced that you have no other option but to
> > > misuse / abuse HTTP protocol in this way, you will have to implement
> > > your own APIs that interact directly with the SessionInputBuffer of the
> > > underlying HTTP connection.
> > >
> > > Oleg
> > >
> > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> > >
> > >
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
Re: Consuming chunk encoded data
Posted by Oleg Kalnichevski <ol...@apache.org>.
On Thu, 2013-10-10 at 16:47 +0700, Alexey Panchenko wrote:
> Hi,
>
> Would it be possible to use HttpAsyncClient for this task, e.g. like here
> http://hc.apache.org/httpcomponents-asyncclient-dev/httpasyncclient/examples/org/apache/http/examples/nio/client/AsyncClientHttpExchangeStreaming.java
> ?
>
> Regards,
> Alex
>
I do not think so. While I _suspect_ this _might_ work for smaller
chunks that fit into the session input buffer in their entirety I am
absolutely sure this approach would fail for large chunks. At any rate
the design would be too brittle to be used for anything remotely
serious.
Oleg
>
>
> On Thu, Oct 10, 2013 at 3:21 PM, Oleg Kalnichevski <ol...@apache.org> wrote:
>
> > On Wed, 2013-10-09 at 13:58 -0400, Chun Tat David Chu wrote:
> > > Hi all,
> > >
> > > Is there an utility or a technique in HTTP Client that allows you to
> > > consume chunk encoded data whenever one chunk arrived on the wire?
> > Perhaps
> > > I overlooked this but I couldn't find anything related with this topic.
> > >
> > > Here's what coming in over the wire.
> > > HttpResponseHandlerImpl.setContentType(): application/json
> > > HttpResponseHandlerImpl.setContentLength(): -1
> > > 58495 [main] DEBUG org.apache.http.wire - << "24[\r][\n]"
> > > 58495 [main] DEBUG org.apache.http.wire - <<
> > > "{"firstName":"David","lastName":"0"}"
> > > 58495 [main] DEBUG org.apache.http.wire - << "[\r][\n]"
> > > 58495 [main] DEBUG org.apache.http.wire - << "24[\r][\n]"
> > > 58495 [main] DEBUG org.apache.http.wire - <<
> > > "{"firstName":"David","lastName":"1"}"
> > > 58495 [main] DEBUG org.apache.http.wire - << "[\r][\n]"
> > > 58495 [main] DEBUG org.apache.http.wire - << "24[\r][\n]"
> > > 58495 [main] DEBUG org.apache.http.wire - <<
> > > "{"firstName":"David","lastName":"2"}"
> > > 58501 [main] DEBUG org.apache.http.wire - << "[\r][\n]"
> > > 58501 [main] DEBUG org.apache.http.wire - << "0[\r][\n]"
> > > 58501 [main] DEBUG org.apache.http.wire - << "[\r][\n]"
> > >
> > > What I want to do is to process a chunk at a time. Is there any helper
> > > method that allows you to do this? or I just need to get the InputStream
> > > and do this manually? Manually as in get the chunk size then read the
> > > buffer based on the chunk size.
> > >
> > > Thanks!
> > >
> > > David
> >
> > David,
> > Chunk coding as well as other transfer encoding schemes are supposed to
> > be completely transparent to the application layer and therefore
> > HttpClient exposes no API related to transfer encodings.
> >
> > If you are absolutely convinced that you have no other option but to
> > misuse / abuse HTTP protocol in this way, you will have to implement
> > your own APIs that interact directly with the SessionInputBuffer of the
> > underlying HTTP connection.
> >
> > Oleg
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> >
> >
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: Consuming chunk encoded data
Posted by Alexey Panchenko <al...@gmail.com>.
Hi,
Would it be possible to use HttpAsyncClient for this task, e.g. like here
http://hc.apache.org/httpcomponents-asyncclient-dev/httpasyncclient/examples/org/apache/http/examples/nio/client/AsyncClientHttpExchangeStreaming.java
?
Regards,
Alex
On Thu, Oct 10, 2013 at 3:21 PM, Oleg Kalnichevski <ol...@apache.org> wrote:
> On Wed, 2013-10-09 at 13:58 -0400, Chun Tat David Chu wrote:
> > Hi all,
> >
> > Is there an utility or a technique in HTTP Client that allows you to
> > consume chunk encoded data whenever one chunk arrived on the wire?
> Perhaps
> > I overlooked this but I couldn't find anything related with this topic.
> >
> > Here's what coming in over the wire.
> > HttpResponseHandlerImpl.setContentType(): application/json
> > HttpResponseHandlerImpl.setContentLength(): -1
> > 58495 [main] DEBUG org.apache.http.wire - << "24[\r][\n]"
> > 58495 [main] DEBUG org.apache.http.wire - <<
> > "{"firstName":"David","lastName":"0"}"
> > 58495 [main] DEBUG org.apache.http.wire - << "[\r][\n]"
> > 58495 [main] DEBUG org.apache.http.wire - << "24[\r][\n]"
> > 58495 [main] DEBUG org.apache.http.wire - <<
> > "{"firstName":"David","lastName":"1"}"
> > 58495 [main] DEBUG org.apache.http.wire - << "[\r][\n]"
> > 58495 [main] DEBUG org.apache.http.wire - << "24[\r][\n]"
> > 58495 [main] DEBUG org.apache.http.wire - <<
> > "{"firstName":"David","lastName":"2"}"
> > 58501 [main] DEBUG org.apache.http.wire - << "[\r][\n]"
> > 58501 [main] DEBUG org.apache.http.wire - << "0[\r][\n]"
> > 58501 [main] DEBUG org.apache.http.wire - << "[\r][\n]"
> >
> > What I want to do is to process a chunk at a time. Is there any helper
> > method that allows you to do this? or I just need to get the InputStream
> > and do this manually? Manually as in get the chunk size then read the
> > buffer based on the chunk size.
> >
> > Thanks!
> >
> > David
>
> David,
> Chunk coding as well as other transfer encoding schemes are supposed to
> be completely transparent to the application layer and therefore
> HttpClient exposes no API related to transfer encodings.
>
> If you are absolutely convinced that you have no other option but to
> misuse / abuse HTTP protocol in this way, you will have to implement
> your own APIs that interact directly with the SessionInputBuffer of the
> underlying HTTP connection.
>
> Oleg
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
Re: Consuming chunk encoded data
Posted by Oleg Kalnichevski <ol...@apache.org>.
On Wed, 2013-10-09 at 13:58 -0400, Chun Tat David Chu wrote:
> Hi all,
>
> Is there an utility or a technique in HTTP Client that allows you to
> consume chunk encoded data whenever one chunk arrived on the wire? Perhaps
> I overlooked this but I couldn't find anything related with this topic.
>
> Here's what coming in over the wire.
> HttpResponseHandlerImpl.setContentType(): application/json
> HttpResponseHandlerImpl.setContentLength(): -1
> 58495 [main] DEBUG org.apache.http.wire - << "24[\r][\n]"
> 58495 [main] DEBUG org.apache.http.wire - <<
> "{"firstName":"David","lastName":"0"}"
> 58495 [main] DEBUG org.apache.http.wire - << "[\r][\n]"
> 58495 [main] DEBUG org.apache.http.wire - << "24[\r][\n]"
> 58495 [main] DEBUG org.apache.http.wire - <<
> "{"firstName":"David","lastName":"1"}"
> 58495 [main] DEBUG org.apache.http.wire - << "[\r][\n]"
> 58495 [main] DEBUG org.apache.http.wire - << "24[\r][\n]"
> 58495 [main] DEBUG org.apache.http.wire - <<
> "{"firstName":"David","lastName":"2"}"
> 58501 [main] DEBUG org.apache.http.wire - << "[\r][\n]"
> 58501 [main] DEBUG org.apache.http.wire - << "0[\r][\n]"
> 58501 [main] DEBUG org.apache.http.wire - << "[\r][\n]"
>
> What I want to do is to process a chunk at a time. Is there any helper
> method that allows you to do this? or I just need to get the InputStream
> and do this manually? Manually as in get the chunk size then read the
> buffer based on the chunk size.
>
> Thanks!
>
> David
David,
Chunk coding as well as other transfer encoding schemes are supposed to
be completely transparent to the application layer and therefore
HttpClient exposes no API related to transfer encodings.
If you are absolutely convinced that you have no other option but to
misuse / abuse HTTP protocol in this way, you will have to implement
your own APIs that interact directly with the SessionInputBuffer of the
underlying HTTP connection.
Oleg
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org