Camel Stream caching unable to capture the entire stream

["calyan.bandi" <ca...@gmail.com>]

Hi,

I have a camel route which invokes a webservice, and tries to processes the
response. I am using cxfrs component for the invocation. The response i am
receiving is very huge and hence most of the content is removed from the
end. 

To remedy this, i am using stream caching -
org.apache.camel.spi.StreamCachingStrategy. These options are not helping me
in the route as i still see that the response is not fully captured and the
end of the response is getting purged. Below are the settings i am using the
camel context and the streamCache is enabled per route. I have tried to set
the streamCache value per route and on the whole context, it didn't make any
difference. Hence i am setting this value per route.

<bean id="myResponseStreamStrategy"
class="org.apache.camel.impl.DefaultStreamCachingStrategy">
		<property name="spoolDirectory" value="${tempSpoolDirectory}"></property>
		<property name="spoolThreshold" value="131072"></property>
		<property name="spoolUsedHeapMemoryThreshold">
			<value type="java.lang.Integer">30</value>
		</property>
		<property name="removeSpoolDirectoryWhenStopping" value="true"></property>
	</bean>

And in the logs i can see the settings being applied as:
[main] 2015-12-14 06:47:22 INFO  AbstractCamelContextFactoryBean:892 - Using
custom StreamCachingStrategy:
DefaultStreamCachingStrategy[spoolDirectory=/home/testing/tmp,
spoolChiper=null, spoolThreshold=131072, spoolUsedHeapMemoryThreshold=30,
bufferSize=4096, anySpoolRules=false]
[main] 2015-12-14 06:47:22 INFO  CamelContextFactoryBean:273 - Bridging
Camel and Spring property placeholder configurer with id:
bridgePropertyPlaceholder
[main] 2015-12-14 06:47:22 INFO  SpringCamelContext:1637 - Apache Camel
2.13.2 (CamelContext: context) is starting
[main] 2015-12-14 06:47:22 INFO  ManagedManagementStrategy:187 - JMX is
enabled
[main] 2015-12-14 06:47:22 INFO  DefaultTypeConverter:50 - Loaded 206 type
converters
[main] 2015-12-14 06:47:23 INFO  SpringCamelContext:1831 -
AllowUseOriginalMessage is enabled. If access to the original message is not
needed, then its recommended to turn this option off as it may improve
performance.
[main] 2015-12-14 06:47:23 INFO  DefaultStreamCachingStrategy:314 -
StreamCaching in use with spool directory: /home/testing/tmp and rules:
[Spool > 128K body size, Spool > 30% used of Max heap memory]
 

I ran a java utility seperately to capture the entire response with which i
captured the full JSON response and the size of the entire string is around
71172 bytes. 

Any pointers around this issue, kindly suggest.

Thanks,
Kalyan




--
View this message in context: http://camel.465427.n5.nabble.com/Camel-Stream-caching-unable-to-capture-the-entire-stream-tp5775055.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: Camel Stream caching unable to capture the entire stream

[Franz Paul Forsthofer <em...@googlemail.com>]

Hi Kalyan,

this issue is known. See
https://issues.apache.org/jira/browse/CAMEL-8419 . CAMEL-8419 contains
even a patch. But this patch is more a workaround, therefore this
patch was never merged.

Best Regards Franz

On Wed, Dec 16, 2015 at 12:35 PM, calyan.bandi <ca...@gmail.com> wrote:
> Hi,
>
> I found the anwer to the problem in apache cxf forums.
>
> http://cxf.547215.n5.nabble.com/How-to-increase-the-max-size-of-a-REST-response-td563077.html
>
> This is more of a issue in the CXF component than camel. But still for
> referene, i am sharing this information.
>
> At the server side, we saw the below response headers for each of the
> requests.
>
> ID: 2
> Response-Code: 200
> Content-Type: application/json
> Headers: {Content-Type=[application/json], Date=[Wed, 16 Dec 2015 11:21:34
> GMT]}
> Messages: Outbound Message (saved to tmp file):
> Filename: C:\Users\Temp\cxf-tmp-309797\cos8635606649006238176tmp
> (message truncated to 65536 bytes)
>
> And from the client logs, whatever the payload that is received from the
> server, we are seeing only the first chunk in the logs (calculated the
> response payload size from the logs and it is 65536 bytes). This is only a
> concern during logging and in my camel routes at the client side the entire
> response is getting unmarshalled from JSON and things are working fine.
>
> Thanks,
> Kalyan
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Camel-Stream-caching-unable-to-capture-the-entire-stream-tp5775055p5775145.html
> Sent from the Camel - Users mailing list archive at Nabble.com.

Re: Camel Stream caching unable to capture the entire stream

["calyan.bandi" <ca...@gmail.com>]

Hi,

I found the anwer to the problem in apache cxf forums. 

http://cxf.547215.n5.nabble.com/How-to-increase-the-max-size-of-a-REST-response-td563077.html

This is more of a issue in the CXF component than camel. But still for
referene, i am sharing this information. 

At the server side, we saw the below response headers for each of the
requests.

ID: 2
Response-Code: 200
Content-Type: application/json
Headers: {Content-Type=[application/json], Date=[Wed, 16 Dec 2015 11:21:34
GMT]}
Messages: Outbound Message (saved to tmp file):
Filename: C:\Users\Temp\cxf-tmp-309797\cos8635606649006238176tmp
(message truncated to 65536 bytes)

And from the client logs, whatever the payload that is received from the
server, we are seeing only the first chunk in the logs (calculated the
response payload size from the logs and it is 65536 bytes). This is only a
concern during logging and in my camel routes at the client side the entire
response is getting unmarshalled from JSON and things are working fine.      

Thanks,
Kalyan



--
View this message in context: http://camel.465427.n5.nabble.com/Camel-Stream-caching-unable-to-capture-the-entire-stream-tp5775055p5775145.html
Sent from the Camel - Users mailing list archive at Nabble.com.