You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by Tim Dudgeon <td...@gmail.com> on 2019/01/16 16:46:42 UTC
HTTP response is not being chunked
I'm hitting a strange problem with a route that is using the servlet
component to send a large stream of data in its response.
This is part of a route that uses the REST DSL which is set up like this:
restConfiguration().component("servlet").host("0.0.0.0");
The route sets the response body as an PipedInputStream and then (in a
separate thread) writes to that InputStream and eventually closes it.
The expectation is that as the response size is unknown the data will be
chunked and response returned to the client immediately and the client
will start to consume the data immediately.
When using curl as the client it seems to be working correctly, and I
see a 'Transfer-Encoding: chunked' header in the response.
However my real client is Apache HttpAsyncClient and with that what I'm
seeing is that the response is not returned until all the data is
written, and instead of a 'Transfer-Encoding: chunked' header I see a
'Content-Length: 19196336' header suggesting that Camel has waited for
the entire content to be written so that it can set the Content-Length.
I tried manually setting the 'Transfer-Encoding: chunked' in the
response message but it did not appear and instead I saw the
Content-Length header.
I've got HttpAsyncClient working in a test example to prove that it can
received the response asynchronously, so I think (but can't be sure)
that the problem is on the server (Camel) side.
What can explain this different behaviour from the same service when
using different clients?
Re: HTTP response is not being chunked
Posted by Tim Dudgeon <td...@gmail.com>.
So on further reflection it seems that there should be an option that
allows to control this - it's not reasonable that when you turn on gzip
compression for a response that the entire content gets cached so that
the Content-Length can be determined, especially when you've specified
that you want Transfer-Encoding to be chunked.
So either I'm not aware of any option that allows this, or alternatively
it could really be considered a bug.
Can any Camel guru comment on this?
BTW - I'm using Camel 2.18.0 which I know is a little old.
Tim
On 17/01/2019 07:44, Tim Dudgeon wrote:
> Yes, you are right - the 'Accept-Encoding: gzip' header is the cause.
> Adding that to the curl request causes the response to block, while
> removing it from Apache HTTP client request prevents the response
> blocking.
>
> But I'm not sure why this would be the case. Gzip compression is
> something that can be done of a streaming basis so there should be no
> need to read the entire dataset.
>
> Turning off compression will have a big performance impact!
>
> Tim
>
> On 16/01/2019 18:48, Luiz Eduardo Valmont wrote:
>> Hi Tim!
>>
>> Sorry about the delay. I was summoned into a meeting that took a while.
>> Plus my PC is in the process of upgrading from Debian 9 to Debian
>> 10/testing.
>>
>> On the important matter.
>>
>> There are certainly differences when it comes to headers. But I
>> suspect the
>> culprit is the accept-encoding. I don't think it's the Camel headers.
>>
>> I believe because it has to compress the content, it has to read
>> everything
>> beforehand. Try adding "-H 'Accept-Encoding: gzip'" (double quotes
>> excluded) to your curl command and see if it changes the actual
>> behaviour.
>>
>> If it does, then you'll have to configure your client accordingly.
>>
>> In any case, you can emulate the Async request in full by configuring
>> the
>> headers with curl's "-H" parameter.
>>
>> HTH
>>
>>
>> On Wed, 16 Jan 2019, 15:17 Tim Dudgeon <tdudgeon.ml@gmail.com wrote:
>>
>>> Hi Luiz,
>>>
>>> here are the request headers. For Apache HTTPClient:
>>>
>>> accept -> chemical/x-mdl-sdfile
>>> accept-encoding -> gzip
>>> breadcrumbId -> ID-6667e3dab68a-38877-1547658276076-0-15
>>> CamelHttpMethod -> POST
>>> CamelHttpPath ->
>>> CamelHttpQuery -> null
>>> CamelHttpServletRequest ->
>>> org.apache.catalina.connector.RequestFacade@7551baa3
>>> CamelHttpServletResponse ->
>>> org.apache.catalina.connector.ResponseFacade@60e05546
>>> CamelHttpUri ->
>>> /chem-services-cdk-basic/rest/v1/converters/dataset_to_sdf
>>> CamelHttpUrl ->
>>>
>>> http://localhost:8092/chem-services-cdk-basic/rest/v1/converters/dataset_to_sdf
>>>
>>> CamelServletContextPath -> /v1/converters/dataset_to_sdf
>>> connection -> Keep-Alive
>>> content-encoding -> gzip
>>> Content-Type -> application/x-squonk-dataset-molecule+json
>>> host -> localhost:8092
>>> transfer-encoding -> chunked
>>> user-agent -> Apache-HttpAsyncClient/4.1.4 (Java/1.8.0_191)
>>>
>>> For curl:
>>>
>>> accept -> chemical/x-mdl-sdfile
>>> breadcrumbId -> ID-09b7601451a2-34027-1547658868812-0-15
>>> CamelHttpMethod -> POST
>>> CamelHttpPath ->
>>> CamelHttpQuery -> null
>>> CamelHttpServletRequest ->
>>> org.apache.catalina.connector.RequestFacade@3af9ee39
>>> CamelHttpServletResponse ->
>>> org.apache.catalina.connector.ResponseFacade@1f33d20b
>>> CamelHttpUri ->
>>> /chem-services-cdk-basic/rest/v1/converters/dataset_to_sdf
>>> CamelHttpUrl ->
>>>
>>> http://localhost:8092/chem-services-cdk-basic/rest/v1/converters/dataset_to_sdf
>>>
>>> CamelServletContextPath -> /v1/converters/dataset_to_sdf
>>> content-encoding -> gzip
>>> content-length -> 19397182
>>> Content-Type -> application/x-squonk-dataset-molecule+json
>>> expect -> 100-continue
>>> host -> localhost:8092
>>> user-agent -> curl/7.55.1
>>>
>>>
>>> On 16/01/2019 16:55, Luiz Eduardo Valmont wrote:
>>>> Hi, Tim!
>>>>
>>>> I'd suggest trying to intercept / dump the HttpAsyncClient request as
>>> well
>>>> as the curl one. A simple "nc -l -p 8888" in an *NIX OS will do.
>>>>
>>>> My guess is that the perceived difference lies in the request headers.
>>>> Might be in the Accept.
>>>>
>>>> Can you post both requests in full, body excluded?
>>>>
>>>> Thanks!
>>>>
>>>> On Wed, 16 Jan 2019, 14:46 Tim Dudgeon <tdudgeon.ml@gmail.com wrote:
>>>>
>>>>> I'm hitting a strange problem with a route that is using the servlet
>>>>> component to send a large stream of data in its response.
>>>>> This is part of a route that uses the REST DSL which is set up like
>>> this:
>>>>> restConfiguration().component("servlet").host("0.0.0.0");
>>>>>
>>>>> The route sets the response body as an PipedInputStream and then
>>>>> (in a
>>>>> separate thread) writes to that InputStream and eventually closes it.
>>>>> The expectation is that as the response size is unknown the data
>>>>> will be
>>>>> chunked and response returned to the client immediately and the
>>>>> client
>>>>> will start to consume the data immediately.
>>>>>
>>>>> When using curl as the client it seems to be working correctly, and I
>>>>> see a 'Transfer-Encoding: chunked' header in the response.
>>>>>
>>>>> However my real client is Apache HttpAsyncClient and with that
>>>>> what I'm
>>>>> seeing is that the response is not returned until all the data is
>>>>> written, and instead of a 'Transfer-Encoding: chunked' header I see a
>>>>> 'Content-Length: 19196336' header suggesting that Camel has waited
>>>>> for
>>>>> the entire content to be written so that it can set the
>>>>> Content-Length.
>>>>> I tried manually setting the 'Transfer-Encoding: chunked' in the
>>>>> response message but it did not appear and instead I saw the
>>>>> Content-Length header.
>>>>> I've got HttpAsyncClient working in a test example to prove that
>>>>> it can
>>>>> received the response asynchronously, so I think (but can't be sure)
>>>>> that the problem is on the server (Camel) side.
>>>>>
>>>>> What can explain this different behaviour from the same service when
>>>>> using different clients?
>>>>>
>>>>>
Re: HTTP response is not being chunked
Posted by Tim Dudgeon <td...@gmail.com>.
Yes, you are right - the 'Accept-Encoding: gzip' header is the cause.
Adding that to the curl request causes the response to block, while
removing it from Apache HTTP client request prevents the response blocking.
But I'm not sure why this would be the case. Gzip compression is
something that can be done of a streaming basis so there should be no
need to read the entire dataset.
Turning off compression will have a big performance impact!
Tim
On 16/01/2019 18:48, Luiz Eduardo Valmont wrote:
> Hi Tim!
>
> Sorry about the delay. I was summoned into a meeting that took a while.
> Plus my PC is in the process of upgrading from Debian 9 to Debian
> 10/testing.
>
> On the important matter.
>
> There are certainly differences when it comes to headers. But I suspect the
> culprit is the accept-encoding. I don't think it's the Camel headers.
>
> I believe because it has to compress the content, it has to read everything
> beforehand. Try adding "-H 'Accept-Encoding: gzip'" (double quotes
> excluded) to your curl command and see if it changes the actual behaviour.
>
> If it does, then you'll have to configure your client accordingly.
>
> In any case, you can emulate the Async request in full by configuring the
> headers with curl's "-H" parameter.
>
> HTH
>
>
> On Wed, 16 Jan 2019, 15:17 Tim Dudgeon <tdudgeon.ml@gmail.com wrote:
>
>> Hi Luiz,
>>
>> here are the request headers. For Apache HTTPClient:
>>
>> accept -> chemical/x-mdl-sdfile
>> accept-encoding -> gzip
>> breadcrumbId -> ID-6667e3dab68a-38877-1547658276076-0-15
>> CamelHttpMethod -> POST
>> CamelHttpPath ->
>> CamelHttpQuery -> null
>> CamelHttpServletRequest ->
>> org.apache.catalina.connector.RequestFacade@7551baa3
>> CamelHttpServletResponse ->
>> org.apache.catalina.connector.ResponseFacade@60e05546
>> CamelHttpUri ->
>> /chem-services-cdk-basic/rest/v1/converters/dataset_to_sdf
>> CamelHttpUrl ->
>>
>> http://localhost:8092/chem-services-cdk-basic/rest/v1/converters/dataset_to_sdf
>> CamelServletContextPath -> /v1/converters/dataset_to_sdf
>> connection -> Keep-Alive
>> content-encoding -> gzip
>> Content-Type -> application/x-squonk-dataset-molecule+json
>> host -> localhost:8092
>> transfer-encoding -> chunked
>> user-agent -> Apache-HttpAsyncClient/4.1.4 (Java/1.8.0_191)
>>
>> For curl:
>>
>> accept -> chemical/x-mdl-sdfile
>> breadcrumbId -> ID-09b7601451a2-34027-1547658868812-0-15
>> CamelHttpMethod -> POST
>> CamelHttpPath ->
>> CamelHttpQuery -> null
>> CamelHttpServletRequest ->
>> org.apache.catalina.connector.RequestFacade@3af9ee39
>> CamelHttpServletResponse ->
>> org.apache.catalina.connector.ResponseFacade@1f33d20b
>> CamelHttpUri ->
>> /chem-services-cdk-basic/rest/v1/converters/dataset_to_sdf
>> CamelHttpUrl ->
>>
>> http://localhost:8092/chem-services-cdk-basic/rest/v1/converters/dataset_to_sdf
>> CamelServletContextPath -> /v1/converters/dataset_to_sdf
>> content-encoding -> gzip
>> content-length -> 19397182
>> Content-Type -> application/x-squonk-dataset-molecule+json
>> expect -> 100-continue
>> host -> localhost:8092
>> user-agent -> curl/7.55.1
>>
>>
>> On 16/01/2019 16:55, Luiz Eduardo Valmont wrote:
>>> Hi, Tim!
>>>
>>> I'd suggest trying to intercept / dump the HttpAsyncClient request as
>> well
>>> as the curl one. A simple "nc -l -p 8888" in an *NIX OS will do.
>>>
>>> My guess is that the perceived difference lies in the request headers.
>>> Might be in the Accept.
>>>
>>> Can you post both requests in full, body excluded?
>>>
>>> Thanks!
>>>
>>> On Wed, 16 Jan 2019, 14:46 Tim Dudgeon <tdudgeon.ml@gmail.com wrote:
>>>
>>>> I'm hitting a strange problem with a route that is using the servlet
>>>> component to send a large stream of data in its response.
>>>> This is part of a route that uses the REST DSL which is set up like
>> this:
>>>> restConfiguration().component("servlet").host("0.0.0.0");
>>>>
>>>> The route sets the response body as an PipedInputStream and then (in a
>>>> separate thread) writes to that InputStream and eventually closes it.
>>>> The expectation is that as the response size is unknown the data will be
>>>> chunked and response returned to the client immediately and the client
>>>> will start to consume the data immediately.
>>>>
>>>> When using curl as the client it seems to be working correctly, and I
>>>> see a 'Transfer-Encoding: chunked' header in the response.
>>>>
>>>> However my real client is Apache HttpAsyncClient and with that what I'm
>>>> seeing is that the response is not returned until all the data is
>>>> written, and instead of a 'Transfer-Encoding: chunked' header I see a
>>>> 'Content-Length: 19196336' header suggesting that Camel has waited for
>>>> the entire content to be written so that it can set the Content-Length.
>>>> I tried manually setting the 'Transfer-Encoding: chunked' in the
>>>> response message but it did not appear and instead I saw the
>>>> Content-Length header.
>>>> I've got HttpAsyncClient working in a test example to prove that it can
>>>> received the response asynchronously, so I think (but can't be sure)
>>>> that the problem is on the server (Camel) side.
>>>>
>>>> What can explain this different behaviour from the same service when
>>>> using different clients?
>>>>
>>>>
Re: HTTP response is not being chunked
Posted by Tim Dudgeon <td...@gmail.com>.
For the record, I created this issue to try resolve this:
https://issues.apache.org/jira/browse/CAMEL-13092
On 16/01/2019 18:48, Luiz Eduardo Valmont wrote:
> Hi Tim!
>
> Sorry about the delay. I was summoned into a meeting that took a while.
> Plus my PC is in the process of upgrading from Debian 9 to Debian
> 10/testing.
>
> On the important matter.
>
> There are certainly differences when it comes to headers. But I suspect the
> culprit is the accept-encoding. I don't think it's the Camel headers.
>
> I believe because it has to compress the content, it has to read everything
> beforehand. Try adding "-H 'Accept-Encoding: gzip'" (double quotes
> excluded) to your curl command and see if it changes the actual behaviour.
>
> If it does, then you'll have to configure your client accordingly.
>
> In any case, you can emulate the Async request in full by configuring the
> headers with curl's "-H" parameter.
>
> HTH
>
>
> On Wed, 16 Jan 2019, 15:17 Tim Dudgeon <tdudgeon.ml@gmail.com wrote:
>
>> Hi Luiz,
>>
>> here are the request headers. For Apache HTTPClient:
>>
>> accept -> chemical/x-mdl-sdfile
>> accept-encoding -> gzip
>> breadcrumbId -> ID-6667e3dab68a-38877-1547658276076-0-15
>> CamelHttpMethod -> POST
>> CamelHttpPath ->
>> CamelHttpQuery -> null
>> CamelHttpServletRequest ->
>> org.apache.catalina.connector.RequestFacade@7551baa3
>> CamelHttpServletResponse ->
>> org.apache.catalina.connector.ResponseFacade@60e05546
>> CamelHttpUri ->
>> /chem-services-cdk-basic/rest/v1/converters/dataset_to_sdf
>> CamelHttpUrl ->
>>
>> http://localhost:8092/chem-services-cdk-basic/rest/v1/converters/dataset_to_sdf
>> CamelServletContextPath -> /v1/converters/dataset_to_sdf
>> connection -> Keep-Alive
>> content-encoding -> gzip
>> Content-Type -> application/x-squonk-dataset-molecule+json
>> host -> localhost:8092
>> transfer-encoding -> chunked
>> user-agent -> Apache-HttpAsyncClient/4.1.4 (Java/1.8.0_191)
>>
>> For curl:
>>
>> accept -> chemical/x-mdl-sdfile
>> breadcrumbId -> ID-09b7601451a2-34027-1547658868812-0-15
>> CamelHttpMethod -> POST
>> CamelHttpPath ->
>> CamelHttpQuery -> null
>> CamelHttpServletRequest ->
>> org.apache.catalina.connector.RequestFacade@3af9ee39
>> CamelHttpServletResponse ->
>> org.apache.catalina.connector.ResponseFacade@1f33d20b
>> CamelHttpUri ->
>> /chem-services-cdk-basic/rest/v1/converters/dataset_to_sdf
>> CamelHttpUrl ->
>>
>> http://localhost:8092/chem-services-cdk-basic/rest/v1/converters/dataset_to_sdf
>> CamelServletContextPath -> /v1/converters/dataset_to_sdf
>> content-encoding -> gzip
>> content-length -> 19397182
>> Content-Type -> application/x-squonk-dataset-molecule+json
>> expect -> 100-continue
>> host -> localhost:8092
>> user-agent -> curl/7.55.1
>>
>>
>> On 16/01/2019 16:55, Luiz Eduardo Valmont wrote:
>>> Hi, Tim!
>>>
>>> I'd suggest trying to intercept / dump the HttpAsyncClient request as
>> well
>>> as the curl one. A simple "nc -l -p 8888" in an *NIX OS will do.
>>>
>>> My guess is that the perceived difference lies in the request headers.
>>> Might be in the Accept.
>>>
>>> Can you post both requests in full, body excluded?
>>>
>>> Thanks!
>>>
>>> On Wed, 16 Jan 2019, 14:46 Tim Dudgeon <tdudgeon.ml@gmail.com wrote:
>>>
>>>> I'm hitting a strange problem with a route that is using the servlet
>>>> component to send a large stream of data in its response.
>>>> This is part of a route that uses the REST DSL which is set up like
>> this:
>>>> restConfiguration().component("servlet").host("0.0.0.0");
>>>>
>>>> The route sets the response body as an PipedInputStream and then (in a
>>>> separate thread) writes to that InputStream and eventually closes it.
>>>> The expectation is that as the response size is unknown the data will be
>>>> chunked and response returned to the client immediately and the client
>>>> will start to consume the data immediately.
>>>>
>>>> When using curl as the client it seems to be working correctly, and I
>>>> see a 'Transfer-Encoding: chunked' header in the response.
>>>>
>>>> However my real client is Apache HttpAsyncClient and with that what I'm
>>>> seeing is that the response is not returned until all the data is
>>>> written, and instead of a 'Transfer-Encoding: chunked' header I see a
>>>> 'Content-Length: 19196336' header suggesting that Camel has waited for
>>>> the entire content to be written so that it can set the Content-Length.
>>>> I tried manually setting the 'Transfer-Encoding: chunked' in the
>>>> response message but it did not appear and instead I saw the
>>>> Content-Length header.
>>>> I've got HttpAsyncClient working in a test example to prove that it can
>>>> received the response asynchronously, so I think (but can't be sure)
>>>> that the problem is on the server (Camel) side.
>>>>
>>>> What can explain this different behaviour from the same service when
>>>> using different clients?
>>>>
>>>>
Re: HTTP response is not being chunked
Posted by Luiz Eduardo Valmont <va...@alis-sol.com.br>.
Hi Tim!
Sorry about the delay. I was summoned into a meeting that took a while.
Plus my PC is in the process of upgrading from Debian 9 to Debian
10/testing.
On the important matter.
There are certainly differences when it comes to headers. But I suspect the
culprit is the accept-encoding. I don't think it's the Camel headers.
I believe because it has to compress the content, it has to read everything
beforehand. Try adding "-H 'Accept-Encoding: gzip'" (double quotes
excluded) to your curl command and see if it changes the actual behaviour.
If it does, then you'll have to configure your client accordingly.
In any case, you can emulate the Async request in full by configuring the
headers with curl's "-H" parameter.
HTH
On Wed, 16 Jan 2019, 15:17 Tim Dudgeon <tdudgeon.ml@gmail.com wrote:
> Hi Luiz,
>
> here are the request headers. For Apache HTTPClient:
>
> accept -> chemical/x-mdl-sdfile
> accept-encoding -> gzip
> breadcrumbId -> ID-6667e3dab68a-38877-1547658276076-0-15
> CamelHttpMethod -> POST
> CamelHttpPath ->
> CamelHttpQuery -> null
> CamelHttpServletRequest ->
> org.apache.catalina.connector.RequestFacade@7551baa3
> CamelHttpServletResponse ->
> org.apache.catalina.connector.ResponseFacade@60e05546
> CamelHttpUri ->
> /chem-services-cdk-basic/rest/v1/converters/dataset_to_sdf
> CamelHttpUrl ->
>
> http://localhost:8092/chem-services-cdk-basic/rest/v1/converters/dataset_to_sdf
> CamelServletContextPath -> /v1/converters/dataset_to_sdf
> connection -> Keep-Alive
> content-encoding -> gzip
> Content-Type -> application/x-squonk-dataset-molecule+json
> host -> localhost:8092
> transfer-encoding -> chunked
> user-agent -> Apache-HttpAsyncClient/4.1.4 (Java/1.8.0_191)
>
> For curl:
>
> accept -> chemical/x-mdl-sdfile
> breadcrumbId -> ID-09b7601451a2-34027-1547658868812-0-15
> CamelHttpMethod -> POST
> CamelHttpPath ->
> CamelHttpQuery -> null
> CamelHttpServletRequest ->
> org.apache.catalina.connector.RequestFacade@3af9ee39
> CamelHttpServletResponse ->
> org.apache.catalina.connector.ResponseFacade@1f33d20b
> CamelHttpUri ->
> /chem-services-cdk-basic/rest/v1/converters/dataset_to_sdf
> CamelHttpUrl ->
>
> http://localhost:8092/chem-services-cdk-basic/rest/v1/converters/dataset_to_sdf
> CamelServletContextPath -> /v1/converters/dataset_to_sdf
> content-encoding -> gzip
> content-length -> 19397182
> Content-Type -> application/x-squonk-dataset-molecule+json
> expect -> 100-continue
> host -> localhost:8092
> user-agent -> curl/7.55.1
>
>
> On 16/01/2019 16:55, Luiz Eduardo Valmont wrote:
> > Hi, Tim!
> >
> > I'd suggest trying to intercept / dump the HttpAsyncClient request as
> well
> > as the curl one. A simple "nc -l -p 8888" in an *NIX OS will do.
> >
> > My guess is that the perceived difference lies in the request headers.
> > Might be in the Accept.
> >
> > Can you post both requests in full, body excluded?
> >
> > Thanks!
> >
> > On Wed, 16 Jan 2019, 14:46 Tim Dudgeon <tdudgeon.ml@gmail.com wrote:
> >
> >> I'm hitting a strange problem with a route that is using the servlet
> >> component to send a large stream of data in its response.
> >> This is part of a route that uses the REST DSL which is set up like
> this:
> >>
> >> restConfiguration().component("servlet").host("0.0.0.0");
> >>
> >> The route sets the response body as an PipedInputStream and then (in a
> >> separate thread) writes to that InputStream and eventually closes it.
> >> The expectation is that as the response size is unknown the data will be
> >> chunked and response returned to the client immediately and the client
> >> will start to consume the data immediately.
> >>
> >> When using curl as the client it seems to be working correctly, and I
> >> see a 'Transfer-Encoding: chunked' header in the response.
> >>
> >> However my real client is Apache HttpAsyncClient and with that what I'm
> >> seeing is that the response is not returned until all the data is
> >> written, and instead of a 'Transfer-Encoding: chunked' header I see a
> >> 'Content-Length: 19196336' header suggesting that Camel has waited for
> >> the entire content to be written so that it can set the Content-Length.
> >> I tried manually setting the 'Transfer-Encoding: chunked' in the
> >> response message but it did not appear and instead I saw the
> >> Content-Length header.
> >> I've got HttpAsyncClient working in a test example to prove that it can
> >> received the response asynchronously, so I think (but can't be sure)
> >> that the problem is on the server (Camel) side.
> >>
> >> What can explain this different behaviour from the same service when
> >> using different clients?
> >>
> >>
>
Re: HTTP response is not being chunked
Posted by Tim Dudgeon <td...@gmail.com>.
Hi Luiz,
here are the request headers. For Apache HTTPClient:
accept -> chemical/x-mdl-sdfile
accept-encoding -> gzip
breadcrumbId -> ID-6667e3dab68a-38877-1547658276076-0-15
CamelHttpMethod -> POST
CamelHttpPath ->
CamelHttpQuery -> null
CamelHttpServletRequest ->
org.apache.catalina.connector.RequestFacade@7551baa3
CamelHttpServletResponse ->
org.apache.catalina.connector.ResponseFacade@60e05546
CamelHttpUri ->
/chem-services-cdk-basic/rest/v1/converters/dataset_to_sdf
CamelHttpUrl ->
http://localhost:8092/chem-services-cdk-basic/rest/v1/converters/dataset_to_sdf
CamelServletContextPath -> /v1/converters/dataset_to_sdf
connection -> Keep-Alive
content-encoding -> gzip
Content-Type -> application/x-squonk-dataset-molecule+json
host -> localhost:8092
transfer-encoding -> chunked
user-agent -> Apache-HttpAsyncClient/4.1.4 (Java/1.8.0_191)
For curl:
accept -> chemical/x-mdl-sdfile
breadcrumbId -> ID-09b7601451a2-34027-1547658868812-0-15
CamelHttpMethod -> POST
CamelHttpPath ->
CamelHttpQuery -> null
CamelHttpServletRequest ->
org.apache.catalina.connector.RequestFacade@3af9ee39
CamelHttpServletResponse ->
org.apache.catalina.connector.ResponseFacade@1f33d20b
CamelHttpUri ->
/chem-services-cdk-basic/rest/v1/converters/dataset_to_sdf
CamelHttpUrl ->
http://localhost:8092/chem-services-cdk-basic/rest/v1/converters/dataset_to_sdf
CamelServletContextPath -> /v1/converters/dataset_to_sdf
content-encoding -> gzip
content-length -> 19397182
Content-Type -> application/x-squonk-dataset-molecule+json
expect -> 100-continue
host -> localhost:8092
user-agent -> curl/7.55.1
On 16/01/2019 16:55, Luiz Eduardo Valmont wrote:
> Hi, Tim!
>
> I'd suggest trying to intercept / dump the HttpAsyncClient request as well
> as the curl one. A simple "nc -l -p 8888" in an *NIX OS will do.
>
> My guess is that the perceived difference lies in the request headers.
> Might be in the Accept.
>
> Can you post both requests in full, body excluded?
>
> Thanks!
>
> On Wed, 16 Jan 2019, 14:46 Tim Dudgeon <tdudgeon.ml@gmail.com wrote:
>
>> I'm hitting a strange problem with a route that is using the servlet
>> component to send a large stream of data in its response.
>> This is part of a route that uses the REST DSL which is set up like this:
>>
>> restConfiguration().component("servlet").host("0.0.0.0");
>>
>> The route sets the response body as an PipedInputStream and then (in a
>> separate thread) writes to that InputStream and eventually closes it.
>> The expectation is that as the response size is unknown the data will be
>> chunked and response returned to the client immediately and the client
>> will start to consume the data immediately.
>>
>> When using curl as the client it seems to be working correctly, and I
>> see a 'Transfer-Encoding: chunked' header in the response.
>>
>> However my real client is Apache HttpAsyncClient and with that what I'm
>> seeing is that the response is not returned until all the data is
>> written, and instead of a 'Transfer-Encoding: chunked' header I see a
>> 'Content-Length: 19196336' header suggesting that Camel has waited for
>> the entire content to be written so that it can set the Content-Length.
>> I tried manually setting the 'Transfer-Encoding: chunked' in the
>> response message but it did not appear and instead I saw the
>> Content-Length header.
>> I've got HttpAsyncClient working in a test example to prove that it can
>> received the response asynchronously, so I think (but can't be sure)
>> that the problem is on the server (Camel) side.
>>
>> What can explain this different behaviour from the same service when
>> using different clients?
>>
>>
Re: HTTP response is not being chunked
Posted by Luiz Eduardo Valmont <va...@alis-sol.com.br>.
Hi, Tim!
I'd suggest trying to intercept / dump the HttpAsyncClient request as well
as the curl one. A simple "nc -l -p 8888" in an *NIX OS will do.
My guess is that the perceived difference lies in the request headers.
Might be in the Accept.
Can you post both requests in full, body excluded?
Thanks!
On Wed, 16 Jan 2019, 14:46 Tim Dudgeon <tdudgeon.ml@gmail.com wrote:
> I'm hitting a strange problem with a route that is using the servlet
> component to send a large stream of data in its response.
> This is part of a route that uses the REST DSL which is set up like this:
>
> restConfiguration().component("servlet").host("0.0.0.0");
>
> The route sets the response body as an PipedInputStream and then (in a
> separate thread) writes to that InputStream and eventually closes it.
> The expectation is that as the response size is unknown the data will be
> chunked and response returned to the client immediately and the client
> will start to consume the data immediately.
>
> When using curl as the client it seems to be working correctly, and I
> see a 'Transfer-Encoding: chunked' header in the response.
>
> However my real client is Apache HttpAsyncClient and with that what I'm
> seeing is that the response is not returned until all the data is
> written, and instead of a 'Transfer-Encoding: chunked' header I see a
> 'Content-Length: 19196336' header suggesting that Camel has waited for
> the entire content to be written so that it can set the Content-Length.
> I tried manually setting the 'Transfer-Encoding: chunked' in the
> response message but it did not appear and instead I saw the
> Content-Length header.
> I've got HttpAsyncClient working in a test example to prove that it can
> received the response asynchronously, so I think (but can't be sure)
> that the problem is on the server (Camel) side.
>
> What can explain this different behaviour from the same service when
> using different clients?
>
>