You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cxf.apache.org by Thomas Engelschmidt <te...@zama.org> on 2008/12/16 15:03:52 UTC
Howto consume huge amount of data with a cxf client
Hi
I'm implementing a cxf ws client, that's going to consume at huge amount of
data (500gb a month). Since its not possible to restrict the response size.
I would like to implement an interceptor, that gets the inputstream and
redirects it to a file, and if possible returns another xml object instead.
Since memory usage is the issue here. Therefor the interception, must happen
during reception of response data.
I have considered looking at the LoggingInInterceptor, but I'm not sure if
it intercepts at the correct phase.
1) Is it possible to redirect input to a file using InInterceptors ?
2) Which phase should such FileInInterceptor use ?
3) Is it possible to inject another xml response, that gets unmarchalled
into a jaxb pojo ?
4) Is this the right approach ?
Cheers
Thomas Engelschmidt
Re: Howto consume huge amount of data with a cxf client
Posted by Andrew Clegg <an...@gmail.com>.
2008/12/16 Thomas Engelschmidt <te...@zama.org>:
> I only have control over the client side. MTOM I would imagine requires
> configurtion on the server side.
I misread your mail as well -- so ignore what I said about Provider
services -- but you could build a Dispatch<StreamSource> client to
stream large amounts of data from a server to a file at the client
end.
When you said in the original mail:
"Is it possible to inject another xml response, that gets unmarchalled
into a jaxb pojo ?"
what exactly is it that you need the client to do?
It's probably a bit roundabout to have some Java code that generates
XML on the fly purely for the sake of unmarshalling back to JAXB
objects. Why not have your client construct the appropriate objects
directly, based on data read off the stream, and forget about JAXB?
Andrew.
Re: Howto consume huge amount of data with a cxf client
Posted by Thomas Engelschmidt <te...@zama.org>.
I only have control over the client side. MTOM I would imagine
requires configurtion on the server side.
On Dec 16, 2008, at 15:17 , Idar Borlaug wrote:
> Have you considered using MTOM, then you get a stream object in java,
> which you can redirect to file.
>
> 2008/12/16 Thomas Engelschmidt <te...@zama.org>:
>> Hi
>>
>> I'm implementing a cxf ws client, that's going to consume at huge
>> amount of
>> data (500gb a month). Since its not possible to restrict the
>> response size.
>>
>> I would like to implement an interceptor, that gets the inputstream
>> and
>> redirects it to a file, and if possible returns another xml object
>> instead.
>> Since memory usage is the issue here. Therefor the interception,
>> must happen
>> during reception of response data.
>> I have considered looking at the LoggingInInterceptor, but I'm not
>> sure if
>> it intercepts at the correct phase.
>>
>>
>> 1) Is it possible to redirect input to a file using InInterceptors ?
>>
>> 2) Which phase should such FileInInterceptor use ?
>>
>> 3) Is it possible to inject another xml response, that gets
>> unmarchalled
>> into a jaxb pojo ?
>>
>> 4) Is this the right approach ?
>>
>> Cheers
>>
>> Thomas Engelschmidt
>>
>
>
>
> --
> Idar
Re: Howto consume huge amount of data with a cxf client
Posted by Idar Borlaug <id...@gmail.com>.
Have you considered using MTOM, then you get a stream object in java,
which you can redirect to file.
2008/12/16 Thomas Engelschmidt <te...@zama.org>:
> Hi
>
> I'm implementing a cxf ws client, that's going to consume at huge amount of
> data (500gb a month). Since its not possible to restrict the response size.
>
> I would like to implement an interceptor, that gets the inputstream and
> redirects it to a file, and if possible returns another xml object instead.
> Since memory usage is the issue here. Therefor the interception, must happen
> during reception of response data.
> I have considered looking at the LoggingInInterceptor, but I'm not sure if
> it intercepts at the correct phase.
>
>
> 1) Is it possible to redirect input to a file using InInterceptors ?
>
> 2) Which phase should such FileInInterceptor use ?
>
> 3) Is it possible to inject another xml response, that gets unmarchalled
> into a jaxb pojo ?
>
> 4) Is this the right approach ?
>
> Cheers
>
> Thomas Engelschmidt
>
--
Idar
Re: Howto consume huge amount of data with a cxf client
Posted by Daniel Kulp <dk...@apache.org>.
If you cannot use MTOM (which would do this automatically), then something
similar to the LoggingInInterceptor is probably the correct approach. I'd
suggest using our CachedOutputStream. It provides temp files for large
messages, but uses byte[] for short messages to avoid file IO. It also
takes care of cleaning up the temp files and such. Something like:
super(Phase.RECEIVE);
public void handleMessage(Message message) throws Fault {
InputStream ins = message.getContent(InputStream.class);
int threashold = 64 * 1024; // <64K keep in memory
CachedOutputStream fout = new CachedOutputStream(threashold);
IOUtils.copy(ins, fout);
fout.flush();
message.setContent(InputStream.class, fout.getInputStream());
fout.close();
ins.close();
}
Dan
On Tuesday 16 December 2008 9:03:52 am Thomas Engelschmidt wrote:
> I'm implementing a cxf ws client, that's going to consume at huge amount of
> data (500gb a month). Since its not possible to restrict the response size.
>
> I would like to implement an interceptor, that gets the inputstream and
> redirects it to a file, and if possible returns another xml object instead.
> Since memory usage is the issue here. Therefor the interception, must
> happen during reception of response data.
> I have considered looking at the LoggingInInterceptor, but I'm not sure if
> it intercepts at the correct phase.
>
>
> 1) Is it possible to redirect input to a file using InInterceptors ?
>
> 2) Which phase should such FileInInterceptor use ?
>
> 3) Is it possible to inject another xml response, that gets unmarchalled
> into a jaxb pojo ?
>
> 4) Is this the right approach ?
>
> Cheers
>
> Thomas Engelschmidt
--
Daniel Kulp
dkulp@apache.org
http://dankulp.com/blog
Re: Howto consume huge amount of data with a cxf client
Posted by Andrew Clegg <an...@gmail.com>.
You could use a Provider<StreamSource> service that just writes the
stream directly to disk, and then invokes another service (or just
calls a method in your back-end application directly).
http://cwiki.apache.org/CXF20DOC/provider-services.html
Andrew.
2008/12/16 Thomas Engelschmidt <te...@zama.org>:
> Hi
>
> I'm implementing a cxf ws client, that's going to consume at huge amount of
> data (500gb a month). Since its not possible to restrict the response size.
>
> I would like to implement an interceptor, that gets the inputstream and
> redirects it to a file, and if possible returns another xml object instead.
> Since memory usage is the issue here. Therefor the interception, must happen
> during reception of response data.
> I have considered looking at the LoggingInInterceptor, but I'm not sure if
> it intercepts at the correct phase.
>
>
> 1) Is it possible to redirect input to a file using InInterceptors ?
>
> 2) Which phase should such FileInInterceptor use ?
>
> 3) Is it possible to inject another xml response, that gets unmarchalled
> into a jaxb pojo ?
>
> 4) Is this the right approach ?
>
> Cheers
>
> Thomas Engelschmidt
>