You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cxf.apache.org by dp-isp <dp...@csi.it> on 2009/07/27 11:50:00 UTC
Streaming large attachments by linking input and output stream
avoiding temp files
Hello everybody,
We have a CXF webservice that is a kind of "repository service" and a webapp
working as frontend for that repository.
Those two components are on separate hosts.
We would need to realize a continuous stream of data when downloading
attachments from CXF webservice to the users' browser through the frontend
webapp.
The webapp is basically a servlet that invokes the webservice, receives the
attachment datahandler and copies datahandler InputStream to servlet
response OutputStream (servlet code is at the end of this email).
Our issue: is it possible to directly link datahandler InputStream to
servlet response OutputStream without re-building the whole attachment in
memoroy or in a temporary file? (as can be configured by setting
org.apache.cxf.io.CachedOutputStream.Threshold,
org.apache.cxf.io.CachedOutputStream.OutputDirectory)
Going more into details, is there a way to send to CXF the output buffer to
write the attachment on? In such case we would send to CXF the sevlet's
output buffer.
Alternatively, is there a way to manage CXF working asynchronously with
attachments, giving back control to the servlet before attachment downolad
completion?
Many thanks,
Marco
---
This is the servlet's involved code:
...
StreamingService_Service ss = new StreamingService_Service(new
URL(serviceURL));
StreamingService port = ss.getServiceImplPort(new MTOMFeature());
Resume myFile = port.downloadMethod(fileName, fileType, usr, pwd, repo);
DataHandler handler = myFile.getResume();
response.setHeader("Pragma", "No-cache");
response.setContentType("application/octet-stream");
response.setHeader("content-disposition", "attachment; filename=" + fileName
+ "." + fileType);
try {
InputStream is = handler.getInputStream();
try {
OutputStream os = response.getOutputStream();
try {
byte[] b = new byte[100000];
int bytesRead = 0;
while ((bytesRead = is.read(b)) != -1) {
os.write(b, 0, bytesRead);
}
System.out.println("Download del file " + myFile.getCandidateName() + "."
+ myFile.getResumeFileType() + " completato.");
} finally {
os.flush();
os.close();
}
} finally {
is.close();
}
} catch (Exception e) {
e.printStackTrace();
}
...
--
View this message in context: http://www.nabble.com/Streaming-large-attachments-by-linking-input-and-output-stream-avoiding-temp-files-tp24677330p24677330.html
Sent from the cxf-user mailing list archive at Nabble.com.
Re: Streaming large attachments by linking input and output stream
avoiding temp files
Posted by dp-isp <dp...@csi.it>.
Ok Dan,
I was making some mass in my project, sorry for the alert, doublechecking
was the solution :-)
Now streaming works perfectly on Jboss in the conditions you described.
On WLS we still have some problem due to the fact that Weblogic uses its own
stax implementation.
org.apache.cxf.interceptor.Fault: Marshalling Error: The uri may not be
theempty string.
at
org.apache.cxf.jaxb.JAXBEncoderDecoder.marshall(JAXBEncoderDecoder.java:161)
...
Caused by: java.lang.IllegalArgumentException: The uri may not be theempty
string.
at
weblogic.xml.stax.util.NamespaceContextImpl.getPrefix(NamespaceContextImpl.java:76)
...
We only uses ear to deploy, so
<prefer-web-inf-classes>true</prefer-web-inf-classes>
in our weblogic.xml it isn't enough. But we have to use a filtering
classloader.
We still have some problem to tune it correctly, I'll update the post when
we'll manage it.
Thanks for your support,
Bye,
Marco
dkulp wrote:
>
>
> I think you are still picking up an older version of CXF from someplace.
> The
> stack trace you have below is impossible with the newer code.
>
>> at org.apache.cxf.helpers.IOUtils.copy(IOUtils.java:75)
>> at
>> org.apache.cxf.attachment.AttachmentDataSource.<init>(AttachmentDataSource.
>>java:39) at
>
> That's older code. Neither of the constructors of AttachmentDataSource
> call
> IOUtils anymore. Line 39 is just:
> this.ct = ctParam;
>
> I'd double check your libs.
>
> Dan
>
>
>
> On Thu July 30 2009 4:48:40 am dp-isp wrote:
>> I tried the 2.2.3 release candidate as you suggested with the same
>> implementation of my first post, but nothing changes. I also explicitly
>> disabled schema-validation inside cxf.xml of the service (even if it
>> should
>> be the default if I well understood.)
>> Is there a subversion URL or something similar, where I can get diffs
>> between version?
>>
>>
>> This is the thread dump on JBoss where the servlet (WS client) is
>> deployed
>> on. In that moment this thread is actually downloading the file from the
>> WS
>> server a caching it in /tmp/cxf-tmp-960634/cos44564tmp:
>>
>> "http-dev-jboss3-dspstd.csi.it%2F10.102.40.71-13610-1" daemon prio=1
>> tid=0x0837c568 nid=0x2bcc runnable [0x7ec8c000..0x7ec8dda0]
>> at java.net.SocketInputStream.socketRead0(Native Method)
>> at java.net.SocketInputStream.read(SocketInputStream.java:129)
>> at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
>> at
>> java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
>> at java.io.BufferedInputStream.read(BufferedInputStream.java:313)
>> - locked <0xaabfa010> (a java.io.BufferedInputStream)
>> at
>> sun.net.www.http.ChunkedInputStream.fastRead(ChunkedInputStream.java:221)
>> at
>> sun.net.www.http.ChunkedInputStream.read(ChunkedInputStream.java:662)
>> - locked <0xaabfc9c8> (a sun.net.www.http.ChunkedInputStream)
>> at java.io.FilterInputStream.read(FilterInputStream.java:111)
>> at
>> sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(HttpURLCon
>>nection.java:2172) at
>> java.io.FilterInputStream.read(FilterInputStream.java:111) at
>> java.io.PushbackInputStream.read(PushbackInputStream.java:161) at
>> org.apache.cxf.attachment.MimeBodyPartInputStream.read(MimeBodyPartInputStr
>>eam.java:74) at java.io.InputStream.read(InputStream.java:89)
>> at
>> org.apache.cxf.attachment.DelegatingInputStream.read(DelegatingInputStream.
>>java:77) at org.apache.cxf.helpers.IOUtils.copy(IOUtils.java:112)
>> at org.apache.cxf.helpers.IOUtils.copy(IOUtils.java:75)
>> at
>> org.apache.cxf.attachment.AttachmentDataSource.<init>(AttachmentDataSource.
>>java:39) at
>> org.apache.cxf.attachment.AttachmentUtil.createAttachment(AttachmentUtil.ja
>>va:168) at
>> org.apache.cxf.attachment.AttachmentDeserializer.createAttachment(Attachmen
>>tDeserializer.java:283) at
>> org.apache.cxf.attachment.AttachmentDeserializer.readNext(AttachmentDeseria
>>lizer.java:194) at
>> org.apache.cxf.attachment.LazyAttachmentCollection$1.hasNext(LazyAttachment
>>Collection.java:73) at
>> org.apache.cxf.attachment.LazyDataSource.load(LazyDataSource.java:49)
>> - locked <0xaac08660> (a
>> org.apache.cxf.attachment.LazyDataSource)
>> at
>> org.apache.cxf.attachment.LazyDataSource.getContentType(LazyDataSource.java
>>:59) at javax.activation.DataHandler.<init>(DataHandler.java:44) at
>> org.apache.cxf.jaxb.attachment.JAXBAttachmentUnmarshaller.getAttachmentAsDa
>>taHandler(JAXBAttachmentUnmarshaller.java:55) at
>> com.sun.xml.bind.v2.runtime.unmarshaller.MTOMDecorator.startElement(MTOMDec
>>orator.java:96) at
>> com.sun.xml.bind.v2.runtime.unmarshaller.StAXStreamConnector.handleStartEle
>>ment(StAXStreamConnector.java:275) at
>> com.sun.xml.bind.v2.runtime.unmarshaller.StAXStreamConnector.bridge(StAXStr
>>eamConnector.java:209) at
>> com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallerImpl.unmarshal0(Unmars
>>hallerImpl.java:360) at
>> com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallerImpl.unmarshal(Unmarsh
>>allerImpl.java:339) at
>> org.apache.cxf.jaxb.JAXBEncoderDecoder.unmarshall(JAXBEncoderDecoder.java:6
>>07) at
>> org.apache.cxf.jaxb.JAXBEncoderDecoder.unmarshall(JAXBEncoderDecoder.java:5
>>27) at
>> org.apache.cxf.jaxb.io.DataReaderImpl.read(DataReaderImpl.java:114)
>> at
>> org.apache.cxf.interceptor.DocLiteralInInterceptor.handleMessage(DocLiteral
>>InInterceptor.java:99) at
>> org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChai
>>n.java:236) - locked <0xaabffa20> (a
>> org.apache.cxf.phase.PhaseInterceptorChain) at
>> org.apache.cxf.endpoint.ClientImpl.onMessage(ClientImpl.java:645) at
>> org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleRespons
>>eInternal(HTTPConduit.java:2132) at
>> org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleRespons
>>e(HTTPConduit.java:2015) at
>> org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HTTPCon
>>duit.java:1940) at
>> org.apache.cxf.transport.AbstractConduit.close(AbstractConduit.java:66)
>> at
>> org.apache.cxf.transport.http.HTTPConduit.close(HTTPConduit.java:627)
>> at
>> org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInte
>>rceptor.handleMessage(MessageSenderInterceptor.java:62) at
>> org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChai
>>n.java:236) - locked <0xaabda040> (a
>> org.apache.cxf.phase.PhaseInterceptorChain) at
>> org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:471) at
>> org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:301) at
>> org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:253) at
>> org.apache.cxf.frontend.ClientProxy.invokeSync(ClientProxy.java:73)
>> at
>> org.apache.cxf.jaxws.JaxWsClientProxy.invoke(JaxWsClientProxy.java:121)
>> at $Proxy338.downloadMethod(Unknown Source)
>> at
>> it.csi.mtomsample.streamingclient.presentation.wsclient.DownloadServletBrow
>>ser.doPost(Unknown Source)
>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:710)
>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
>> at
>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Applicatio
>>nFilterChain.java:290) at
>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterC
>>hain.java:206) at
>> org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.j
>>ava:96) at
>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Applicatio
>>nFilterChain.java:235) at
>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterC
>>hain.java:206) at
>> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.j
>>ava:230) at
>> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.j
>>ava:175) at
>> org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssoc
>>iationValve.java:179) at
>> org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java
>>:84) at
>> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:12
>>8) at
>> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:10
>>4) at
>> org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnect
>>ionValve.java:157) at
>> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.jav
>>a:109) at
>> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:241)
>> at
>> org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)
>> at
>> org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Htt
>>p11Protocol.java:580) at
>> org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
>> at java.lang.Thread.run(Thread.java:595)
>>
>>
>>
>> This is the thread dump on WebLogic where the WS service is deployed on.
>> In
>> that moment this thread is actually providing the file to the client.
>>
>>
>> "[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default
>> (self-tuning)'" id=17 idx=0x5c tid=17758 prio=5 alive, in native, daemon
>> at
>> jrockit/net/SocketNativeIO.writeBytesPinned(Ljava/io/FileDescriptor;[BII)V(
>>Native Method)
>> at jrockit/net/SocketNativeIO.socketWrite(SocketNativeIO.java:45)
>> at
>> java/net/SocketOutputStream.socketWrite0(Ljava/io/FileDescriptor;[BII)V(Soc
>>ketOutputStream.java) at
>> java/net/SocketOutputStream.socketWrite(SocketOutputStream.java:92) at
>> java/net/SocketOutputStream.write(SocketOutputStream.java:136) at
>> weblogic/servlet/internal/ChunkOutput.writeChunkTransfer(ChunkOutput.java:5
>>25) at
>> weblogic/servlet/internal/ChunkOutput.writeChunks(ChunkOutput.java:504)
>> at weblogic/servlet/internal/ChunkOutput.flush(ChunkOutput.java:382)
>> at
>> weblogic/servlet/internal/CharsetChunkOutput.flush(CharsetChunkOutput.java:
>>249) at
>> weblogic/servlet/internal/ChunkOutput.checkForFlush(ChunkOutput.java:469)
>> at
>> weblogic/servlet/internal/CharsetChunkOutput.implWrite(CharsetChunkOutput.j
>>ava:396) at
>> weblogic/servlet/internal/CharsetChunkOutput.write(CharsetChunkOutput.java:
>>198) at
>> weblogic/servlet/internal/ChunkOutputWrapper.write(ChunkOutputWrapper.java:
>>133) at
>> weblogic/servlet/internal/ServletOutputStreamImpl.write(ServletOutputStream
>>Impl.java:169) at
>> org/apache/cxf/io/AbstractWrappedOutputStream.write(AbstractWrappedOutputSt
>>ream.java:46) at
javax/activation/DataHandler.writeTo(DataHandler.java:290)
>> at
>> org/apache/cxf/attachment/AttachmentSerializer.writeAttachments(AttachmentS
>>erializer.java:153) at
>> org/apache/cxf/interceptor/AttachmentOutInterceptor$AttachmentOutEndingInte
>>rceptor.handleMessage(AttachmentOutInterceptor.java:80) at
>> org/apache/cxf/phase/PhaseInterceptorChain.doIntercept(PhaseInterceptorChai
>>n.java:236) ^-- Holding lock:
>> org/apache/cxf/phase/PhaseInterceptorChain@0x102dafd8[thin lock]
>> at
>> org/apache/cxf/interceptor/OutgoingChainInterceptor.handleMessage(OutgoingC
>>hainInterceptor.java:74) at
>> org/apache/cxf/phase/PhaseInterceptorChain.doIntercept(PhaseInterceptorChai
>>n.java:236) ^-- Holding lock:
>> org/apache/cxf/phase/PhaseInterceptorChain@0x102cef90[thin lock]
>> at
>> org/apache/cxf/transport/ChainInitiationObserver.onMessage(ChainInitiationO
>>bserver.java:89) at
>> org/apache/cxf/transport/servlet/ServletDestination.invoke(ServletDestinati
>>on.java:99) at
>> org/apache/cxf/transport/servlet/ServletController.invokeDestination(Servle
>>tController.java:368) at
>> org/apache/cxf/transport/servlet/ServletController.invoke(ServletController
>>.java:183) at
>> org/apache/cxf/transport/servlet/AbstractCXFServlet.invoke(AbstractCXFServl
>>et.java:163) at
>> org/apache/cxf/transport/servlet/AbstractCXFServlet.doPost(AbstractCXFServl
>>et.java:141) at
javax/servlet/http/HttpServlet.service(HttpServlet.java:763)
>> at javax/servlet/http/HttpServlet.service(HttpServlet.java:856) at
>> weblogic/servlet/internal/StubSecurityHelper$ServletServiceAction.run(StubS
>>ecurityHelper.java:227) at
>> weblogic/servlet/internal/StubSecurityHelper.invokeServlet(StubSecurityHelp
>>er.java:125) at
>> weblogic/servlet/internal/ServletStubImpl.execute(ServletStubImpl.java:283)
>> at
>> weblogic/servlet/internal/ServletStubImpl.execute(ServletStubImpl.java:175)
>> at
>> weblogic/servlet/internal/WebAppServletContext$ServletInvocationAction.run(
>>WebAppServletContext.java:3231) at
>> weblogic/security/acl/internal/AuthenticatedSubject.doAs(AuthenticatedSubje
>>ct.java:321) at
>> weblogic/security/service/SecurityManager.runAs(SecurityManager.java:121)
>> at
>> weblogic/servlet/internal/WebAppServletContext.securedExecute(WebAppServlet
>>Context.java:2002) at
>> weblogic/servlet/internal/WebAppServletContext.execute(WebAppServletContext
>>.java:1908) at
>> weblogic/servlet/internal/ServletRequestImpl.run(ServletRequestImpl.java:13
>>62) at weblogic/work/ExecuteThread.execute(ExecuteThread.java:209) at
>> weblogic/work/ExecuteThread.run(ExecuteThread.java:181)
>> at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
>> -- end of trace
>>
>>
>> Bye,
>> Marco
>>
>> dkulp wrote:
>> > On Wed July 29 2009 5:30:13 am dp-isp wrote:
>> >> Sorry if my question is trivial... in order to test your fixes, can I
>> >> download the latest snapshot
>> >>
>> (http://repository.apache.org/snapshots/org/apache/cxf/apache-cxf/2.2.3-
>> >>SNA PSHOT/) or there is a better way?
>> >
>> > Yea. But testing out the actual 2.2.3 release candidate that we are
>> > voting on
>> > would be good as well:
>> >
>> https://repository.apache.org/content/repositories/cxf-staging-003/org/ap
>> >ache/cxf/apache-cxf/2.2.3
>> >
>> > Dan
>> >
>> >> Thanks,
>> >> Marco
>> >>
>> >> dkulp wrote:
>> >> > Well, I THOUGHT this would have been easily doable, but when I
>> >> > checked, it turns out the attachments were not streaming properly in
>> >> > MTOM mode.
>> >> >
>> >> > :-(
>> >> >
>> >> > I've gone through and fixed a BUNCH of things related to the
>> >> > attachment handling and streaming and such so some of this is now
>> >> > possible.
>> >> >
>> >> > Now, with the latest code, when you get the DataSource, if at all
>> >> > possible,
>> >> > the input stream that is returned will properly be streaming off the
>> >>
>> >> wire
>> >>
>> >> > instead of cached. Thus, you can copy directly from there to the
>> >>
>> >> output
>> >>
>> >> > stream.
>> >> >
>> >> > There are cases where this WON'T work stream directly, notably when
>> >> > schema validation is turned on. In that case, to work around
>> issues
>> >>
>> >> in
>> >>
>> >> > JAXB, we buffer the content. Also, if there are multiple
>> >>
>> >> attachments,
>> >>
>> >> > only the last
>> >> > one can really be streamed this way. The others get buffered.
>> >> >
>> >> > Dan
>> >> >
>> >> > On Mon July 27 2009 5:50:00 am dp-isp wrote:
>> >> >> Hello everybody,
>> >> >>
>> >> >> We have a CXF webservice that is a kind of "repository service" and
>> a
>> >> >> webapp working as frontend for that repository.
>> >> >> Those two components are on separate hosts.
>> >> >>
>> >> >> We would need to realize a continuous stream of data when
>> downloading
>> >> >> attachments from CXF webservice to the users' browser through the
>> >> >> frontend
>> >> >> webapp.
>> >> >>
>> >> >> The webapp is basically a servlet that invokes the webservice,
>> >>
>> >> receives
>> >>
>> >> >> the
>> >> >> attachment datahandler and copies datahandler InputStream to
>> servlet
>> >> >> response OutputStream (servlet code is at the end of this email).
>> >> >>
>> >> >> Our issue: is it possible to directly link datahandler InputStream
>> to
>> >> >> servlet response OutputStream without re-building the whole
>> >> >> attachment in memoroy or in a temporary file? (as can be configured
>> >> >> by setting org.apache.cxf.io.CachedOutputStream.Threshold,
>> >> >> org.apache.cxf.io.CachedOutputStream.OutputDirectory)
>> >> >>
>> >> >> Going more into details, is there a way to send to CXF the output
>> >>
>> >> buffer
>> >>
>> >> >> to
>> >> >> write the attachment on? In such case we would send to CXF the
>> >>
>> >> sevlet's
>> >>
>> >> >> output buffer.
>> >> >>
>> >> >> Alternatively, is there a way to manage CXF working asynchronously
>> >>
>> >> with
>> >>
>> >> >> attachments, giving back control to the servlet before attachment
>> >> >> downolad
>> >> >> completion?
>> >> >>
>> >> >> Many thanks,
>> >> >> Marco
>> >> >>
>> >> >> ---
>> >> >>
>> >> >> This is the servlet's involved code:
>> >> >>
>> >> >> ...
>> >> >> StreamingService_Service ss = new StreamingService_Service(new
>> >> >> URL(serviceURL));
>> >> >> StreamingService port = ss.getServiceImplPort(new MTOMFeature());
>> >> >>
>> >> >> Resume myFile = port.downloadMethod(fileName, fileType, usr, pwd,
>> >>
>> >> repo);
>> >>
>> >> >> DataHandler handler = myFile.getResume();
>> >> >>
>> >> >> response.setHeader("Pragma", "No-cache");
>> >> >> response.setContentType("application/octet-stream");
>> >> >> response.setHeader("content-disposition", "attachment; filename=" +
>> >> >> fileName + "." + fileType);
>> >> >> try {
>> >> >> InputStream is = handler.getInputStream();
>> >> >> try {
>> >> >> OutputStream os = response.getOutputStream();
>> >> >> try {
>> >> >> byte[] b = new byte[100000];
>> >> >> int bytesRead = 0;
>> >> >> while ((bytesRead = is.read(b)) != -1) {
>> >> >> os.write(b, 0, bytesRead);
>> >> >> }
>> >> >> System.out.println("Download del file " +
>> >> >> myFile.getCandidateName()
>> >>
>> >> +
>> >>
>> >> >> "." + myFile.getResumeFileType() + " completato.");
>> >> >> } finally {
>> >> >> os.flush();
>> >> >> os.close();
>> >> >> }
>> >> >> } finally {
>> >> >> is.close();
>> >> >> }
>> >> >>
>> >> >> } catch (Exception e) {
>> >> >> e.printStackTrace();
>> >> >> }
>> >> >> ...
>> >> >
>> >> > --
>> >> > Daniel Kulp
>> >> > dkulp@apache.org
>> >> > http://www.dankulp.com/blog
>> >
>> > --
>> > Daniel Kulp
>> > dkulp@apache.org
>> > http://www.dankulp.com/blog
>
> --
> Daniel Kulp
> dkulp@apache.org
> http://www.dankulp.com/blog
>
>
--
View this message in context: http://www.nabble.com/Streaming-large-attachments-by-linking-input-and-output-stream-avoiding-temp-files-tp24677330p24743365.html
Sent from the cxf-user mailing list archive at Nabble.com.
Re: Streaming large attachments by linking input and output stream avoiding temp files
Posted by Daniel Kulp <dk...@apache.org>.
I think you are still picking up an older version of CXF from someplace. The
stack trace you have below is impossible with the newer code.
> at org.apache.cxf.helpers.IOUtils.copy(IOUtils.java:75)
> at
> org.apache.cxf.attachment.AttachmentDataSource.<init>(AttachmentDataSource.
>java:39) at
That's older code. Neither of the constructors of AttachmentDataSource call
IOUtils anymore. Line 39 is just:
this.ct = ctParam;
I'd double check your libs.
Dan
On Thu July 30 2009 4:48:40 am dp-isp wrote:
> I tried the 2.2.3 release candidate as you suggested with the same
> implementation of my first post, but nothing changes. I also explicitly
> disabled schema-validation inside cxf.xml of the service (even if it should
> be the default if I well understood.)
> Is there a subversion URL or something similar, where I can get diffs
> between version?
>
>
> This is the thread dump on JBoss where the servlet (WS client) is deployed
> on. In that moment this thread is actually downloading the file from the WS
> server a caching it in /tmp/cxf-tmp-960634/cos44564tmp:
>
> "http-dev-jboss3-dspstd.csi.it%2F10.102.40.71-13610-1" daemon prio=1
> tid=0x0837c568 nid=0x2bcc runnable [0x7ec8c000..0x7ec8dda0]
> at java.net.SocketInputStream.socketRead0(Native Method)
> at java.net.SocketInputStream.read(SocketInputStream.java:129)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:313)
> - locked <0xaabfa010> (a java.io.BufferedInputStream)
> at
> sun.net.www.http.ChunkedInputStream.fastRead(ChunkedInputStream.java:221)
> at
> sun.net.www.http.ChunkedInputStream.read(ChunkedInputStream.java:662)
> - locked <0xaabfc9c8> (a sun.net.www.http.ChunkedInputStream)
> at java.io.FilterInputStream.read(FilterInputStream.java:111)
> at
> sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(HttpURLCon
>nection.java:2172) at
> java.io.FilterInputStream.read(FilterInputStream.java:111) at
> java.io.PushbackInputStream.read(PushbackInputStream.java:161) at
> org.apache.cxf.attachment.MimeBodyPartInputStream.read(MimeBodyPartInputStr
>eam.java:74) at java.io.InputStream.read(InputStream.java:89)
> at
> org.apache.cxf.attachment.DelegatingInputStream.read(DelegatingInputStream.
>java:77) at org.apache.cxf.helpers.IOUtils.copy(IOUtils.java:112)
> at org.apache.cxf.helpers.IOUtils.copy(IOUtils.java:75)
> at
> org.apache.cxf.attachment.AttachmentDataSource.<init>(AttachmentDataSource.
>java:39) at
> org.apache.cxf.attachment.AttachmentUtil.createAttachment(AttachmentUtil.ja
>va:168) at
> org.apache.cxf.attachment.AttachmentDeserializer.createAttachment(Attachmen
>tDeserializer.java:283) at
> org.apache.cxf.attachment.AttachmentDeserializer.readNext(AttachmentDeseria
>lizer.java:194) at
> org.apache.cxf.attachment.LazyAttachmentCollection$1.hasNext(LazyAttachment
>Collection.java:73) at
> org.apache.cxf.attachment.LazyDataSource.load(LazyDataSource.java:49)
> - locked <0xaac08660> (a org.apache.cxf.attachment.LazyDataSource)
> at
> org.apache.cxf.attachment.LazyDataSource.getContentType(LazyDataSource.java
>:59) at javax.activation.DataHandler.<init>(DataHandler.java:44) at
> org.apache.cxf.jaxb.attachment.JAXBAttachmentUnmarshaller.getAttachmentAsDa
>taHandler(JAXBAttachmentUnmarshaller.java:55) at
> com.sun.xml.bind.v2.runtime.unmarshaller.MTOMDecorator.startElement(MTOMDec
>orator.java:96) at
> com.sun.xml.bind.v2.runtime.unmarshaller.StAXStreamConnector.handleStartEle
>ment(StAXStreamConnector.java:275) at
> com.sun.xml.bind.v2.runtime.unmarshaller.StAXStreamConnector.bridge(StAXStr
>eamConnector.java:209) at
> com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallerImpl.unmarshal0(Unmars
>hallerImpl.java:360) at
> com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallerImpl.unmarshal(Unmarsh
>allerImpl.java:339) at
> org.apache.cxf.jaxb.JAXBEncoderDecoder.unmarshall(JAXBEncoderDecoder.java:6
>07) at
> org.apache.cxf.jaxb.JAXBEncoderDecoder.unmarshall(JAXBEncoderDecoder.java:5
>27) at
> org.apache.cxf.jaxb.io.DataReaderImpl.read(DataReaderImpl.java:114)
> at
> org.apache.cxf.interceptor.DocLiteralInInterceptor.handleMessage(DocLiteral
>InInterceptor.java:99) at
> org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChai
>n.java:236) - locked <0xaabffa20> (a
> org.apache.cxf.phase.PhaseInterceptorChain) at
> org.apache.cxf.endpoint.ClientImpl.onMessage(ClientImpl.java:645) at
> org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleRespons
>eInternal(HTTPConduit.java:2132) at
> org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleRespons
>e(HTTPConduit.java:2015) at
> org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HTTPCon
>duit.java:1940) at
> org.apache.cxf.transport.AbstractConduit.close(AbstractConduit.java:66)
> at
> org.apache.cxf.transport.http.HTTPConduit.close(HTTPConduit.java:627)
> at
> org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInte
>rceptor.handleMessage(MessageSenderInterceptor.java:62) at
> org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChai
>n.java:236) - locked <0xaabda040> (a
> org.apache.cxf.phase.PhaseInterceptorChain) at
> org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:471) at
> org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:301) at
> org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:253) at
> org.apache.cxf.frontend.ClientProxy.invokeSync(ClientProxy.java:73)
> at
> org.apache.cxf.jaxws.JaxWsClientProxy.invoke(JaxWsClientProxy.java:121)
> at $Proxy338.downloadMethod(Unknown Source)
> at
> it.csi.mtomsample.streamingclient.presentation.wsclient.DownloadServletBrow
>ser.doPost(Unknown Source)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:710)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Applicatio
>nFilterChain.java:290) at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterC
>hain.java:206) at
> org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.j
>ava:96) at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Applicatio
>nFilterChain.java:235) at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterC
>hain.java:206) at
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.j
>ava:230) at
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.j
>ava:175) at
> org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssoc
>iationValve.java:179) at
> org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java
>:84) at
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:12
>8) at
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:10
>4) at
> org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnect
>ionValve.java:157) at
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.jav
>a:109) at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:241)
> at
> org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)
> at
> org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Htt
>p11Protocol.java:580) at
> org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
> at java.lang.Thread.run(Thread.java:595)
>
>
>
> This is the thread dump on WebLogic where the WS service is deployed on. In
> that moment this thread is actually providing the file to the client.
>
>
> "[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default
> (self-tuning)'" id=17 idx=0x5c tid=17758 prio=5 alive, in native, daemon
> at
> jrockit/net/SocketNativeIO.writeBytesPinned(Ljava/io/FileDescriptor;[BII)V(
>Native Method)
> at jrockit/net/SocketNativeIO.socketWrite(SocketNativeIO.java:45)
> at
> java/net/SocketOutputStream.socketWrite0(Ljava/io/FileDescriptor;[BII)V(Soc
>ketOutputStream.java) at
> java/net/SocketOutputStream.socketWrite(SocketOutputStream.java:92) at
> java/net/SocketOutputStream.write(SocketOutputStream.java:136) at
> weblogic/servlet/internal/ChunkOutput.writeChunkTransfer(ChunkOutput.java:5
>25) at
> weblogic/servlet/internal/ChunkOutput.writeChunks(ChunkOutput.java:504)
> at weblogic/servlet/internal/ChunkOutput.flush(ChunkOutput.java:382)
> at
> weblogic/servlet/internal/CharsetChunkOutput.flush(CharsetChunkOutput.java:
>249) at
> weblogic/servlet/internal/ChunkOutput.checkForFlush(ChunkOutput.java:469)
> at
> weblogic/servlet/internal/CharsetChunkOutput.implWrite(CharsetChunkOutput.j
>ava:396) at
> weblogic/servlet/internal/CharsetChunkOutput.write(CharsetChunkOutput.java:
>198) at
> weblogic/servlet/internal/ChunkOutputWrapper.write(ChunkOutputWrapper.java:
>133) at
> weblogic/servlet/internal/ServletOutputStreamImpl.write(ServletOutputStream
>Impl.java:169) at
> org/apache/cxf/io/AbstractWrappedOutputStream.write(AbstractWrappedOutputSt
>ream.java:46) at javax/activation/DataHandler.writeTo(DataHandler.java:290)
> at
> org/apache/cxf/attachment/AttachmentSerializer.writeAttachments(AttachmentS
>erializer.java:153) at
> org/apache/cxf/interceptor/AttachmentOutInterceptor$AttachmentOutEndingInte
>rceptor.handleMessage(AttachmentOutInterceptor.java:80) at
> org/apache/cxf/phase/PhaseInterceptorChain.doIntercept(PhaseInterceptorChai
>n.java:236) ^-- Holding lock:
> org/apache/cxf/phase/PhaseInterceptorChain@0x102dafd8[thin lock]
> at
> org/apache/cxf/interceptor/OutgoingChainInterceptor.handleMessage(OutgoingC
>hainInterceptor.java:74) at
> org/apache/cxf/phase/PhaseInterceptorChain.doIntercept(PhaseInterceptorChai
>n.java:236) ^-- Holding lock:
> org/apache/cxf/phase/PhaseInterceptorChain@0x102cef90[thin lock]
> at
> org/apache/cxf/transport/ChainInitiationObserver.onMessage(ChainInitiationO
>bserver.java:89) at
> org/apache/cxf/transport/servlet/ServletDestination.invoke(ServletDestinati
>on.java:99) at
> org/apache/cxf/transport/servlet/ServletController.invokeDestination(Servle
>tController.java:368) at
> org/apache/cxf/transport/servlet/ServletController.invoke(ServletController
>.java:183) at
> org/apache/cxf/transport/servlet/AbstractCXFServlet.invoke(AbstractCXFServl
>et.java:163) at
> org/apache/cxf/transport/servlet/AbstractCXFServlet.doPost(AbstractCXFServl
>et.java:141) at javax/servlet/http/HttpServlet.service(HttpServlet.java:763)
> at javax/servlet/http/HttpServlet.service(HttpServlet.java:856) at
> weblogic/servlet/internal/StubSecurityHelper$ServletServiceAction.run(StubS
>ecurityHelper.java:227) at
> weblogic/servlet/internal/StubSecurityHelper.invokeServlet(StubSecurityHelp
>er.java:125) at
> weblogic/servlet/internal/ServletStubImpl.execute(ServletStubImpl.java:283)
> at
> weblogic/servlet/internal/ServletStubImpl.execute(ServletStubImpl.java:175)
> at
> weblogic/servlet/internal/WebAppServletContext$ServletInvocationAction.run(
>WebAppServletContext.java:3231) at
> weblogic/security/acl/internal/AuthenticatedSubject.doAs(AuthenticatedSubje
>ct.java:321) at
> weblogic/security/service/SecurityManager.runAs(SecurityManager.java:121)
> at
> weblogic/servlet/internal/WebAppServletContext.securedExecute(WebAppServlet
>Context.java:2002) at
> weblogic/servlet/internal/WebAppServletContext.execute(WebAppServletContext
>.java:1908) at
> weblogic/servlet/internal/ServletRequestImpl.run(ServletRequestImpl.java:13
>62) at weblogic/work/ExecuteThread.execute(ExecuteThread.java:209) at
> weblogic/work/ExecuteThread.run(ExecuteThread.java:181)
> at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
> -- end of trace
>
>
> Bye,
> Marco
>
> dkulp wrote:
> > On Wed July 29 2009 5:30:13 am dp-isp wrote:
> >> Sorry if my question is trivial... in order to test your fixes, can I
> >> download the latest snapshot
> >> (http://repository.apache.org/snapshots/org/apache/cxf/apache-cxf/2.2.3-
> >>SNA PSHOT/) or there is a better way?
> >
> > Yea. But testing out the actual 2.2.3 release candidate that we are
> > voting on
> > would be good as well:
> > https://repository.apache.org/content/repositories/cxf-staging-003/org/ap
> >ache/cxf/apache-cxf/2.2.3
> >
> > Dan
> >
> >> Thanks,
> >> Marco
> >>
> >> dkulp wrote:
> >> > Well, I THOUGHT this would have been easily doable, but when I
> >> > checked, it turns out the attachments were not streaming properly in
> >> > MTOM mode.
> >> >
> >> > :-(
> >> >
> >> > I've gone through and fixed a BUNCH of things related to the
> >> > attachment handling and streaming and such so some of this is now
> >> > possible.
> >> >
> >> > Now, with the latest code, when you get the DataSource, if at all
> >> > possible,
> >> > the input stream that is returned will properly be streaming off the
> >>
> >> wire
> >>
> >> > instead of cached. Thus, you can copy directly from there to the
> >>
> >> output
> >>
> >> > stream.
> >> >
> >> > There are cases where this WON'T work stream directly, notably when
> >> > schema validation is turned on. In that case, to work around issues
> >>
> >> in
> >>
> >> > JAXB, we buffer the content. Also, if there are multiple
> >>
> >> attachments,
> >>
> >> > only the last
> >> > one can really be streamed this way. The others get buffered.
> >> >
> >> > Dan
> >> >
> >> > On Mon July 27 2009 5:50:00 am dp-isp wrote:
> >> >> Hello everybody,
> >> >>
> >> >> We have a CXF webservice that is a kind of "repository service" and a
> >> >> webapp working as frontend for that repository.
> >> >> Those two components are on separate hosts.
> >> >>
> >> >> We would need to realize a continuous stream of data when downloading
> >> >> attachments from CXF webservice to the users' browser through the
> >> >> frontend
> >> >> webapp.
> >> >>
> >> >> The webapp is basically a servlet that invokes the webservice,
> >>
> >> receives
> >>
> >> >> the
> >> >> attachment datahandler and copies datahandler InputStream to servlet
> >> >> response OutputStream (servlet code is at the end of this email).
> >> >>
> >> >> Our issue: is it possible to directly link datahandler InputStream to
> >> >> servlet response OutputStream without re-building the whole
> >> >> attachment in memoroy or in a temporary file? (as can be configured
> >> >> by setting org.apache.cxf.io.CachedOutputStream.Threshold,
> >> >> org.apache.cxf.io.CachedOutputStream.OutputDirectory)
> >> >>
> >> >> Going more into details, is there a way to send to CXF the output
> >>
> >> buffer
> >>
> >> >> to
> >> >> write the attachment on? In such case we would send to CXF the
> >>
> >> sevlet's
> >>
> >> >> output buffer.
> >> >>
> >> >> Alternatively, is there a way to manage CXF working asynchronously
> >>
> >> with
> >>
> >> >> attachments, giving back control to the servlet before attachment
> >> >> downolad
> >> >> completion?
> >> >>
> >> >> Many thanks,
> >> >> Marco
> >> >>
> >> >> ---
> >> >>
> >> >> This is the servlet's involved code:
> >> >>
> >> >> ...
> >> >> StreamingService_Service ss = new StreamingService_Service(new
> >> >> URL(serviceURL));
> >> >> StreamingService port = ss.getServiceImplPort(new MTOMFeature());
> >> >>
> >> >> Resume myFile = port.downloadMethod(fileName, fileType, usr, pwd,
> >>
> >> repo);
> >>
> >> >> DataHandler handler = myFile.getResume();
> >> >>
> >> >> response.setHeader("Pragma", "No-cache");
> >> >> response.setContentType("application/octet-stream");
> >> >> response.setHeader("content-disposition", "attachment; filename=" +
> >> >> fileName + "." + fileType);
> >> >> try {
> >> >> InputStream is = handler.getInputStream();
> >> >> try {
> >> >> OutputStream os = response.getOutputStream();
> >> >> try {
> >> >> byte[] b = new byte[100000];
> >> >> int bytesRead = 0;
> >> >> while ((bytesRead = is.read(b)) != -1) {
> >> >> os.write(b, 0, bytesRead);
> >> >> }
> >> >> System.out.println("Download del file " +
> >> >> myFile.getCandidateName()
> >>
> >> +
> >>
> >> >> "." + myFile.getResumeFileType() + " completato.");
> >> >> } finally {
> >> >> os.flush();
> >> >> os.close();
> >> >> }
> >> >> } finally {
> >> >> is.close();
> >> >> }
> >> >>
> >> >> } catch (Exception e) {
> >> >> e.printStackTrace();
> >> >> }
> >> >> ...
> >> >
> >> > --
> >> > Daniel Kulp
> >> > dkulp@apache.org
> >> > http://www.dankulp.com/blog
> >
> > --
> > Daniel Kulp
> > dkulp@apache.org
> > http://www.dankulp.com/blog
--
Daniel Kulp
dkulp@apache.org
http://www.dankulp.com/blog
Re: Streaming large attachments by linking input and output stream
avoiding temp files
Posted by dp-isp <dp...@csi.it>.
I tried the 2.2.3 release candidate as you suggested with the same
implementation of my first post, but nothing changes. I also explicitly
disabled schema-validation inside cxf.xml of the service (even if it should
be the default if I well understood.)
Is there a subversion URL or something similar, where I can get diffs
between version?
This is the thread dump on JBoss where the servlet (WS client) is deployed
on. In that moment this thread is actually downloading the file from the WS
server a caching it in /tmp/cxf-tmp-960634/cos44564tmp:
"http-dev-jboss3-dspstd.csi.it%2F10.102.40.71-13610-1" daemon prio=1
tid=0x0837c568 nid=0x2bcc runnable [0x7ec8c000..0x7ec8dda0]
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
at java.io.BufferedInputStream.read(BufferedInputStream.java:313)
- locked <0xaabfa010> (a java.io.BufferedInputStream)
at
sun.net.www.http.ChunkedInputStream.fastRead(ChunkedInputStream.java:221)
at
sun.net.www.http.ChunkedInputStream.read(ChunkedInputStream.java:662)
- locked <0xaabfc9c8> (a sun.net.www.http.ChunkedInputStream)
at java.io.FilterInputStream.read(FilterInputStream.java:111)
at
sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(HttpURLConnection.java:2172)
at java.io.FilterInputStream.read(FilterInputStream.java:111)
at java.io.PushbackInputStream.read(PushbackInputStream.java:161)
at
org.apache.cxf.attachment.MimeBodyPartInputStream.read(MimeBodyPartInputStream.java:74)
at java.io.InputStream.read(InputStream.java:89)
at
org.apache.cxf.attachment.DelegatingInputStream.read(DelegatingInputStream.java:77)
at org.apache.cxf.helpers.IOUtils.copy(IOUtils.java:112)
at org.apache.cxf.helpers.IOUtils.copy(IOUtils.java:75)
at
org.apache.cxf.attachment.AttachmentDataSource.<init>(AttachmentDataSource.java:39)
at
org.apache.cxf.attachment.AttachmentUtil.createAttachment(AttachmentUtil.java:168)
at
org.apache.cxf.attachment.AttachmentDeserializer.createAttachment(AttachmentDeserializer.java:283)
at
org.apache.cxf.attachment.AttachmentDeserializer.readNext(AttachmentDeserializer.java:194)
at
org.apache.cxf.attachment.LazyAttachmentCollection$1.hasNext(LazyAttachmentCollection.java:73)
at
org.apache.cxf.attachment.LazyDataSource.load(LazyDataSource.java:49)
- locked <0xaac08660> (a org.apache.cxf.attachment.LazyDataSource)
at
org.apache.cxf.attachment.LazyDataSource.getContentType(LazyDataSource.java:59)
at javax.activation.DataHandler.<init>(DataHandler.java:44)
at
org.apache.cxf.jaxb.attachment.JAXBAttachmentUnmarshaller.getAttachmentAsDataHandler(JAXBAttachmentUnmarshaller.java:55)
at
com.sun.xml.bind.v2.runtime.unmarshaller.MTOMDecorator.startElement(MTOMDecorator.java:96)
at
com.sun.xml.bind.v2.runtime.unmarshaller.StAXStreamConnector.handleStartElement(StAXStreamConnector.java:275)
at
com.sun.xml.bind.v2.runtime.unmarshaller.StAXStreamConnector.bridge(StAXStreamConnector.java:209)
at
com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallerImpl.unmarshal0(UnmarshallerImpl.java:360)
at
com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallerImpl.unmarshal(UnmarshallerImpl.java:339)
at
org.apache.cxf.jaxb.JAXBEncoderDecoder.unmarshall(JAXBEncoderDecoder.java:607)
at
org.apache.cxf.jaxb.JAXBEncoderDecoder.unmarshall(JAXBEncoderDecoder.java:527)
at
org.apache.cxf.jaxb.io.DataReaderImpl.read(DataReaderImpl.java:114)
at
org.apache.cxf.interceptor.DocLiteralInInterceptor.handleMessage(DocLiteralInInterceptor.java:99)
at
org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:236)
- locked <0xaabffa20> (a org.apache.cxf.phase.PhaseInterceptorChain)
at org.apache.cxf.endpoint.ClientImpl.onMessage(ClientImpl.java:645)
at
org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleResponseInternal(HTTPConduit.java:2132)
at
org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleResponse(HTTPConduit.java:2015)
at
org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HTTPConduit.java:1940)
at
org.apache.cxf.transport.AbstractConduit.close(AbstractConduit.java:66)
at
org.apache.cxf.transport.http.HTTPConduit.close(HTTPConduit.java:627)
at
org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor.handleMessage(MessageSenderInterceptor.java:62)
at
org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:236)
- locked <0xaabda040> (a org.apache.cxf.phase.PhaseInterceptorChain)
at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:471)
at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:301)
at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:253)
at
org.apache.cxf.frontend.ClientProxy.invokeSync(ClientProxy.java:73)
at
org.apache.cxf.jaxws.JaxWsClientProxy.invoke(JaxWsClientProxy.java:121)
at $Proxy338.downloadMethod(Unknown Source)
at
it.csi.mtomsample.streamingclient.presentation.wsclient.DownloadServletBrowser.doPost(Unknown
Source)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:710)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at
org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
at
org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:179)
at
org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:84)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:104)
at
org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:157)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:241)
at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)
at
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:580)
at
org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
at java.lang.Thread.run(Thread.java:595)
This is the thread dump on WebLogic where the WS service is deployed on. In
that moment this thread is actually providing the file to the client.
"[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default
(self-tuning)'" id=17 idx=0x5c tid=17758 prio=5 alive, in native, daemon
at
jrockit/net/SocketNativeIO.writeBytesPinned(Ljava/io/FileDescriptor;[BII)V(Native
Method)
at jrockit/net/SocketNativeIO.socketWrite(SocketNativeIO.java:45)
at
java/net/SocketOutputStream.socketWrite0(Ljava/io/FileDescriptor;[BII)V(SocketOutputStream.java)
at java/net/SocketOutputStream.socketWrite(SocketOutputStream.java:92)
at java/net/SocketOutputStream.write(SocketOutputStream.java:136)
at
weblogic/servlet/internal/ChunkOutput.writeChunkTransfer(ChunkOutput.java:525)
at
weblogic/servlet/internal/ChunkOutput.writeChunks(ChunkOutput.java:504)
at weblogic/servlet/internal/ChunkOutput.flush(ChunkOutput.java:382)
at
weblogic/servlet/internal/CharsetChunkOutput.flush(CharsetChunkOutput.java:249)
at
weblogic/servlet/internal/ChunkOutput.checkForFlush(ChunkOutput.java:469)
at
weblogic/servlet/internal/CharsetChunkOutput.implWrite(CharsetChunkOutput.java:396)
at
weblogic/servlet/internal/CharsetChunkOutput.write(CharsetChunkOutput.java:198)
at
weblogic/servlet/internal/ChunkOutputWrapper.write(ChunkOutputWrapper.java:133)
at
weblogic/servlet/internal/ServletOutputStreamImpl.write(ServletOutputStreamImpl.java:169)
at
org/apache/cxf/io/AbstractWrappedOutputStream.write(AbstractWrappedOutputStream.java:46)
at javax/activation/DataHandler.writeTo(DataHandler.java:290)
at
org/apache/cxf/attachment/AttachmentSerializer.writeAttachments(AttachmentSerializer.java:153)
at
org/apache/cxf/interceptor/AttachmentOutInterceptor$AttachmentOutEndingInterceptor.handleMessage(AttachmentOutInterceptor.java:80)
at
org/apache/cxf/phase/PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:236)
^-- Holding lock:
org/apache/cxf/phase/PhaseInterceptorChain@0x102dafd8[thin lock]
at
org/apache/cxf/interceptor/OutgoingChainInterceptor.handleMessage(OutgoingChainInterceptor.java:74)
at
org/apache/cxf/phase/PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:236)
^-- Holding lock:
org/apache/cxf/phase/PhaseInterceptorChain@0x102cef90[thin lock]
at
org/apache/cxf/transport/ChainInitiationObserver.onMessage(ChainInitiationObserver.java:89)
at
org/apache/cxf/transport/servlet/ServletDestination.invoke(ServletDestination.java:99)
at
org/apache/cxf/transport/servlet/ServletController.invokeDestination(ServletController.java:368)
at
org/apache/cxf/transport/servlet/ServletController.invoke(ServletController.java:183)
at
org/apache/cxf/transport/servlet/AbstractCXFServlet.invoke(AbstractCXFServlet.java:163)
at
org/apache/cxf/transport/servlet/AbstractCXFServlet.doPost(AbstractCXFServlet.java:141)
at javax/servlet/http/HttpServlet.service(HttpServlet.java:763)
at javax/servlet/http/HttpServlet.service(HttpServlet.java:856)
at
weblogic/servlet/internal/StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227)
at
weblogic/servlet/internal/StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125)
at
weblogic/servlet/internal/ServletStubImpl.execute(ServletStubImpl.java:283)
at
weblogic/servlet/internal/ServletStubImpl.execute(ServletStubImpl.java:175)
at
weblogic/servlet/internal/WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3231)
at
weblogic/security/acl/internal/AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
at
weblogic/security/service/SecurityManager.runAs(SecurityManager.java:121)
at
weblogic/servlet/internal/WebAppServletContext.securedExecute(WebAppServletContext.java:2002)
at
weblogic/servlet/internal/WebAppServletContext.execute(WebAppServletContext.java:1908)
at
weblogic/servlet/internal/ServletRequestImpl.run(ServletRequestImpl.java:1362)
at weblogic/work/ExecuteThread.execute(ExecuteThread.java:209)
at weblogic/work/ExecuteThread.run(ExecuteThread.java:181)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
Bye,
Marco
dkulp wrote:
>
> On Wed July 29 2009 5:30:13 am dp-isp wrote:
>> Sorry if my question is trivial... in order to test your fixes, can I
>> download the latest snapshot
>> (http://repository.apache.org/snapshots/org/apache/cxf/apache-cxf/2.2.3-SNA
>>PSHOT/) or there is a better way?
>
> Yea. But testing out the actual 2.2.3 release candidate that we are
> voting on
> would be good as well:
> https://repository.apache.org/content/repositories/cxf-staging-003/org/apache/cxf/apache-cxf/2.2.3
>
> Dan
>
>> Thanks,
>> Marco
>>
>> dkulp wrote:
>> > Well, I THOUGHT this would have been easily doable, but when I checked,
>> > it turns out the attachments were not streaming properly in MTOM mode.
>> > :-(
>> >
>> > I've gone through and fixed a BUNCH of things related to the attachment
>> > handling and streaming and such so some of this is now possible.
>> >
>> > Now, with the latest code, when you get the DataSource, if at all
>> > possible,
>> > the input stream that is returned will properly be streaming off the
>> wire
>> > instead of cached. Thus, you can copy directly from there to the
>> output
>> > stream.
>> >
>> > There are cases where this WON'T work stream directly, notably when
>> > schema validation is turned on. In that case, to work around issues
>> in
>> > JAXB, we buffer the content. Also, if there are multiple
>> attachments,
>> > only the last
>> > one can really be streamed this way. The others get buffered.
>> >
>> > Dan
>> >
>> > On Mon July 27 2009 5:50:00 am dp-isp wrote:
>> >> Hello everybody,
>> >>
>> >> We have a CXF webservice that is a kind of "repository service" and a
>> >> webapp working as frontend for that repository.
>> >> Those two components are on separate hosts.
>> >>
>> >> We would need to realize a continuous stream of data when downloading
>> >> attachments from CXF webservice to the users' browser through the
>> >> frontend
>> >> webapp.
>> >>
>> >> The webapp is basically a servlet that invokes the webservice,
>> receives
>> >> the
>> >> attachment datahandler and copies datahandler InputStream to servlet
>> >> response OutputStream (servlet code is at the end of this email).
>> >>
>> >> Our issue: is it possible to directly link datahandler InputStream to
>> >> servlet response OutputStream without re-building the whole attachment
>> >> in memoroy or in a temporary file? (as can be configured by setting
>> >> org.apache.cxf.io.CachedOutputStream.Threshold,
>> >> org.apache.cxf.io.CachedOutputStream.OutputDirectory)
>> >>
>> >> Going more into details, is there a way to send to CXF the output
>> buffer
>> >> to
>> >> write the attachment on? In such case we would send to CXF the
>> sevlet's
>> >> output buffer.
>> >>
>> >> Alternatively, is there a way to manage CXF working asynchronously
>> with
>> >> attachments, giving back control to the servlet before attachment
>> >> downolad
>> >> completion?
>> >>
>> >> Many thanks,
>> >> Marco
>> >>
>> >> ---
>> >>
>> >> This is the servlet's involved code:
>> >>
>> >> ...
>> >> StreamingService_Service ss = new StreamingService_Service(new
>> >> URL(serviceURL));
>> >> StreamingService port = ss.getServiceImplPort(new MTOMFeature());
>> >>
>> >> Resume myFile = port.downloadMethod(fileName, fileType, usr, pwd,
>> repo);
>> >> DataHandler handler = myFile.getResume();
>> >>
>> >> response.setHeader("Pragma", "No-cache");
>> >> response.setContentType("application/octet-stream");
>> >> response.setHeader("content-disposition", "attachment; filename=" +
>> >> fileName + "." + fileType);
>> >> try {
>> >> InputStream is = handler.getInputStream();
>> >> try {
>> >> OutputStream os = response.getOutputStream();
>> >> try {
>> >> byte[] b = new byte[100000];
>> >> int bytesRead = 0;
>> >> while ((bytesRead = is.read(b)) != -1) {
>> >> os.write(b, 0, bytesRead);
>> >> }
>> >> System.out.println("Download del file " + myFile.getCandidateName()
>> +
>> >> "." + myFile.getResumeFileType() + " completato.");
>> >> } finally {
>> >> os.flush();
>> >> os.close();
>> >> }
>> >> } finally {
>> >> is.close();
>> >> }
>> >>
>> >> } catch (Exception e) {
>> >> e.printStackTrace();
>> >> }
>> >> ...
>> >
>> > --
>> > Daniel Kulp
>> > dkulp@apache.org
>> > http://www.dankulp.com/blog
>
> --
> Daniel Kulp
> dkulp@apache.org
> http://www.dankulp.com/blog
>
>
--
View this message in context: http://www.nabble.com/Streaming-large-attachments-by-linking-input-and-output-stream-avoiding-temp-files-tp24677330p24734552.html
Sent from the cxf-user mailing list archive at Nabble.com.
Re: Streaming large attachments by linking input and output stream avoiding temp files
Posted by Daniel Kulp <dk...@apache.org>.
On Wed July 29 2009 5:30:13 am dp-isp wrote:
> Sorry if my question is trivial... in order to test your fixes, can I
> download the latest snapshot
> (http://repository.apache.org/snapshots/org/apache/cxf/apache-cxf/2.2.3-SNA
>PSHOT/) or there is a better way?
Yea. But testing out the actual 2.2.3 release candidate that we are voting on
would be good as well:
https://repository.apache.org/content/repositories/cxf-staging-003/org/apache/cxf/apache-cxf/2.2.3
Dan
> Thanks,
> Marco
>
> dkulp wrote:
> > Well, I THOUGHT this would have been easily doable, but when I checked,
> > it turns out the attachments were not streaming properly in MTOM mode.
> > :-(
> >
> > I've gone through and fixed a BUNCH of things related to the attachment
> > handling and streaming and such so some of this is now possible.
> >
> > Now, with the latest code, when you get the DataSource, if at all
> > possible,
> > the input stream that is returned will properly be streaming off the wire
> > instead of cached. Thus, you can copy directly from there to the output
> > stream.
> >
> > There are cases where this WON'T work stream directly, notably when
> > schema validation is turned on. In that case, to work around issues in
> > JAXB, we buffer the content. Also, if there are multiple attachments,
> > only the last
> > one can really be streamed this way. The others get buffered.
> >
> > Dan
> >
> > On Mon July 27 2009 5:50:00 am dp-isp wrote:
> >> Hello everybody,
> >>
> >> We have a CXF webservice that is a kind of "repository service" and a
> >> webapp working as frontend for that repository.
> >> Those two components are on separate hosts.
> >>
> >> We would need to realize a continuous stream of data when downloading
> >> attachments from CXF webservice to the users' browser through the
> >> frontend
> >> webapp.
> >>
> >> The webapp is basically a servlet that invokes the webservice, receives
> >> the
> >> attachment datahandler and copies datahandler InputStream to servlet
> >> response OutputStream (servlet code is at the end of this email).
> >>
> >> Our issue: is it possible to directly link datahandler InputStream to
> >> servlet response OutputStream without re-building the whole attachment
> >> in memoroy or in a temporary file? (as can be configured by setting
> >> org.apache.cxf.io.CachedOutputStream.Threshold,
> >> org.apache.cxf.io.CachedOutputStream.OutputDirectory)
> >>
> >> Going more into details, is there a way to send to CXF the output buffer
> >> to
> >> write the attachment on? In such case we would send to CXF the sevlet's
> >> output buffer.
> >>
> >> Alternatively, is there a way to manage CXF working asynchronously with
> >> attachments, giving back control to the servlet before attachment
> >> downolad
> >> completion?
> >>
> >> Many thanks,
> >> Marco
> >>
> >> ---
> >>
> >> This is the servlet's involved code:
> >>
> >> ...
> >> StreamingService_Service ss = new StreamingService_Service(new
> >> URL(serviceURL));
> >> StreamingService port = ss.getServiceImplPort(new MTOMFeature());
> >>
> >> Resume myFile = port.downloadMethod(fileName, fileType, usr, pwd, repo);
> >> DataHandler handler = myFile.getResume();
> >>
> >> response.setHeader("Pragma", "No-cache");
> >> response.setContentType("application/octet-stream");
> >> response.setHeader("content-disposition", "attachment; filename=" +
> >> fileName + "." + fileType);
> >> try {
> >> InputStream is = handler.getInputStream();
> >> try {
> >> OutputStream os = response.getOutputStream();
> >> try {
> >> byte[] b = new byte[100000];
> >> int bytesRead = 0;
> >> while ((bytesRead = is.read(b)) != -1) {
> >> os.write(b, 0, bytesRead);
> >> }
> >> System.out.println("Download del file " + myFile.getCandidateName() +
> >> "." + myFile.getResumeFileType() + " completato.");
> >> } finally {
> >> os.flush();
> >> os.close();
> >> }
> >> } finally {
> >> is.close();
> >> }
> >>
> >> } catch (Exception e) {
> >> e.printStackTrace();
> >> }
> >> ...
> >
> > --
> > Daniel Kulp
> > dkulp@apache.org
> > http://www.dankulp.com/blog
--
Daniel Kulp
dkulp@apache.org
http://www.dankulp.com/blog
Re: Streaming large attachments by linking input and output stream
avoiding temp files
Posted by dp-isp <dp...@csi.it>.
Sorry if my question is trivial... in order to test your fixes, can I
download the latest snapshot
(http://repository.apache.org/snapshots/org/apache/cxf/apache-cxf/2.2.3-SNAPSHOT/)
or there is a better way?
Thanks,
Marco
dkulp wrote:
>
>
> Well, I THOUGHT this would have been easily doable, but when I checked, it
> turns out the attachments were not streaming properly in MTOM mode. :-(
>
> I've gone through and fixed a BUNCH of things related to the attachment
> handling and streaming and such so some of this is now possible.
>
> Now, with the latest code, when you get the DataSource, if at all
> possible,
> the input stream that is returned will properly be streaming off the wire
> instead of cached. Thus, you can copy directly from there to the output
> stream.
>
> There are cases where this WON'T work stream directly, notably when schema
> validation is turned on. In that case, to work around issues in JAXB, we
> buffer the content. Also, if there are multiple attachments, only the
> last
> one can really be streamed this way. The others get buffered.
>
> Dan
>
>
>
> On Mon July 27 2009 5:50:00 am dp-isp wrote:
>> Hello everybody,
>>
>> We have a CXF webservice that is a kind of "repository service" and a
>> webapp working as frontend for that repository.
>> Those two components are on separate hosts.
>>
>> We would need to realize a continuous stream of data when downloading
>> attachments from CXF webservice to the users' browser through the
>> frontend
>> webapp.
>>
>> The webapp is basically a servlet that invokes the webservice, receives
>> the
>> attachment datahandler and copies datahandler InputStream to servlet
>> response OutputStream (servlet code is at the end of this email).
>>
>> Our issue: is it possible to directly link datahandler InputStream to
>> servlet response OutputStream without re-building the whole attachment in
>> memoroy or in a temporary file? (as can be configured by setting
>> org.apache.cxf.io.CachedOutputStream.Threshold,
>> org.apache.cxf.io.CachedOutputStream.OutputDirectory)
>>
>> Going more into details, is there a way to send to CXF the output buffer
>> to
>> write the attachment on? In such case we would send to CXF the sevlet's
>> output buffer.
>>
>> Alternatively, is there a way to manage CXF working asynchronously with
>> attachments, giving back control to the servlet before attachment
>> downolad
>> completion?
>>
>> Many thanks,
>> Marco
>>
>> ---
>>
>> This is the servlet's involved code:
>>
>> ...
>> StreamingService_Service ss = new StreamingService_Service(new
>> URL(serviceURL));
>> StreamingService port = ss.getServiceImplPort(new MTOMFeature());
>>
>> Resume myFile = port.downloadMethod(fileName, fileType, usr, pwd, repo);
>> DataHandler handler = myFile.getResume();
>>
>> response.setHeader("Pragma", "No-cache");
>> response.setContentType("application/octet-stream");
>> response.setHeader("content-disposition", "attachment; filename=" +
>> fileName + "." + fileType);
>> try {
>> InputStream is = handler.getInputStream();
>> try {
>> OutputStream os = response.getOutputStream();
>> try {
>> byte[] b = new byte[100000];
>> int bytesRead = 0;
>> while ((bytesRead = is.read(b)) != -1) {
>> os.write(b, 0, bytesRead);
>> }
>> System.out.println("Download del file " + myFile.getCandidateName() +
>> "." + myFile.getResumeFileType() + " completato.");
>> } finally {
>> os.flush();
>> os.close();
>> }
>> } finally {
>> is.close();
>> }
>>
>> } catch (Exception e) {
>> e.printStackTrace();
>> }
>> ...
>
> --
> Daniel Kulp
> dkulp@apache.org
> http://www.dankulp.com/blog
>
>
--
View this message in context: http://www.nabble.com/Streaming-large-attachments-by-linking-input-and-output-stream-avoiding-temp-files-tp24677330p24715639.html
Sent from the cxf-user mailing list archive at Nabble.com.
Re: Streaming large attachments by linking input and output stream avoiding temp files
Posted by Daniel Kulp <dk...@apache.org>.
I know a BUNCH of the TCK tests related to mtom also use the
SAAJInInterceptor. That's how I discovered that the SAAJ stuff needed to
have them cached. I'm not sure if any of them use attachments large enough
to trip through to the file output stuff. They may all be smaller than 64K.
Can you log a JIRA and maybe attach a test case. I can then try and take a
look.
Dan
On Wed September 23 2009 7:11:56 pm K1001010 wrote:
> Thank you for the quick reply. After hearing that the changes should be in
> 2.2.3 I dug further into my code and found a client-side interceptor which
> was causing all attachments to be cached.
>
> I fixed that, and now see a server-side problem when sending attachments.
> The server is using SSAJInInterceptor (via WSS4JInInterceptor) and on line
> 117 of the first there is this:
>
> for (Attachment a : atts) {
> if (a.getDataHandler().getDataSource() instanceof
> AttachmentDataSource) {
> try {
>
> ((AttachmentDataSource)a.getDataHandler().getDataSource()).cache();
>
>
> So the attachment is always being cached. But more problematic is this:
>
> After the above call the “inmem” flag of the CachedOutputStream is
> correctly set to false. By the time my application code calls
> CachedOutputStram.getInputStream() however, the flag is true, and hence an
> empty ByteArrayInputStream is returned.
>
> Is there an apache CXF test case with MTOM enabled, WSS4JInInterceptor
> used, and attachment larger than 64k? The one I wrote seems to
> consistently fail—the server side application code always getting a
> non-null, but empty, stream, when the attachment is larger than 64k.
>
> dkulp wrote:
> > On Wed September 16 2009 12:31:57 pm K1001010 wrote:
> >> Did these changes make it into the 2.2.3 release?
> >
> > It looks like it, yea.
> >
> >> I am testing with a 150MB attachment, and still see it being streamed to
> >> disk
> >
> > Hmm... I hate to ask this as I don't want a 150MB test case, but is
> > there
> > any way you could create a "small" test case?
> >
> > There are a few cases where we do have to buffer it ahead of time. If
> > there
> > are any SAAJ related things, we need to buffer. Schema validation is
> > another.
> > If there are multiple attachments, it's likely the others may need to be
> > buffered, etc... A testcase would help figure out what is going on.
> >
> > If you can debug it by setting a breakpoint in
> > org.apache.cxf.attachment.AttachementDataSource.cache() and determine why
> > cache is being called, that would also be helpful.
> >
> > Dan
> >
> >> I have this code:
> >>
> >>
> >> Picture picture = serviceProxy.getPictuer();
> >> DataHandler handler = picture.getImageData();
> >> InputStream is = handler.getInputStream();
> >>
> >>
> >> In version 2.2.2 the entire 150BM file would be streamed to temp file on
> >> line 3, which would return a file input stream (or a delegate too).
> >>
> >> In version 2.2.3 the entire 150MB file is streamed to temp file before
> >> line
> >> 1 returns.
> >>
> >> I disable schema validation using this system property:
> >>
> >> org.apache.spring.validation.mode=VALIDATION_NONE
> >>
> >> Is that correct? Anything else required to stream directly from the
> >> wire?
> >>
> >> Thanks for any help.
> >>
> >> dkulp wrote:
> >> > Well, I THOUGHT this would have been easily doable, but when I
> >> > checked, it turns out the attachments were not streaming properly in
> >> > MTOM mode.
> >> >
> >> > :-(
> >> >
> >> > I've gone through and fixed a BUNCH of things related to the
> >> > attachment handling and streaming and such so some of this is now
> >> > possible.
> >> >
> >> > Now, with the latest code, when you get the DataSource, if at all
> >> > possible,
> >> > the input stream that is returned will properly be streaming off the
> >>
> >> wire
> >>
> >> > instead of cached. Thus, you can copy directly from there to the
> >>
> >> output
> >>
> >> > stream.
> >> >
> >> > There are cases where this WON'T work stream directly, notably when
> >> > schema validation is turned on. In that case, to work around issues
> >>
> >> in
> >>
> >> > JAXB, we buffer the content. Also, if there are multiple
> >>
> >> attachments,
> >>
> >> > only the last
> >> > one can really be streamed this way. The others get buffered.
> >> >
> >> > Dan
> >> >
> >> > On Mon July 27 2009 5:50:00 am dp-isp wrote:
> >> >> Hello everybody,
> >> >>
> >> >> We have a CXF webservice that is a kind of "repository service" and a
> >> >> webapp working as frontend for that repository.
> >> >> Those two components are on separate hosts.
> >> >>
> >> >> We would need to realize a continuous stream of data when downloading
> >> >> attachments from CXF webservice to the users' browser through the
> >> >> frontend
> >> >> webapp.
> >> >>
> >> >> The webapp is basically a servlet that invokes the webservice,
> >>
> >> receives
> >>
> >> >> the
> >> >> attachment datahandler and copies datahandler InputStream to servlet
> >> >> response OutputStream (servlet code is at the end of this email).
> >> >>
> >> >> Our issue: is it possible to directly link datahandler InputStream to
> >> >> servlet response OutputStream without re-building the whole
> >> >> attachment in memoroy or in a temporary file? (as can be configured
> >> >> by setting org.apache.cxf.io.CachedOutputStream.Threshold,
> >> >> org.apache.cxf.io.CachedOutputStream.OutputDirectory)
> >> >>
> >> >> Going more into details, is there a way to send to CXF the output
> >>
> >> buffer
> >>
> >> >> to
> >> >> write the attachment on? In such case we would send to CXF the
> >>
> >> sevlet's
> >>
> >> >> output buffer.
> >> >>
> >> >> Alternatively, is there a way to manage CXF working asynchronously
> >>
> >> with
> >>
> >> >> attachments, giving back control to the servlet before attachment
> >> >> downolad
> >> >> completion?
> >> >>
> >> >> Many thanks,
> >> >> Marco
> >> >>
> >> >> ---
> >> >>
> >> >> This is the servlet's involved code:
> >> >>
> >> >> ...
> >> >> StreamingService_Service ss = new StreamingService_Service(new
> >> >> URL(serviceURL));
> >> >> StreamingService port = ss.getServiceImplPort(new MTOMFeature());
> >> >>
> >> >> Resume myFile = port.downloadMethod(fileName, fileType, usr, pwd,
> >>
> >> repo);
> >>
> >> >> DataHandler handler = myFile.getResume();
> >> >>
> >> >> response.setHeader("Pragma", "No-cache");
> >> >> response.setContentType("application/octet-stream");
> >> >> response.setHeader("content-disposition", "attachment; filename=" +
> >> >> fileName + "." + fileType);
> >> >> try {
> >> >> InputStream is = handler.getInputStream();
> >> >> try {
> >> >> OutputStream os = response.getOutputStream();
> >> >> try {
> >> >> byte[] b = new byte[100000];
> >> >> int bytesRead = 0;
> >> >> while ((bytesRead = is.read(b)) != -1) {
> >> >> os.write(b, 0, bytesRead);
> >> >> }
> >> >> System.out.println("Download del file " +
> >> >> myFile.getCandidateName()
> >>
> >> +
> >>
> >> >> "." + myFile.getResumeFileType() + " completato.");
> >> >> } finally {
> >> >> os.flush();
> >> >> os.close();
> >> >> }
> >> >> } finally {
> >> >> is.close();
> >> >> }
> >> >>
> >> >> } catch (Exception e) {
> >> >> e.printStackTrace();
> >> >> }
> >> >> ...
>
--
Daniel Kulp
dkulp@apache.org
http://www.dankulp.com/blog
Re: Streaming large attachments by linking input and output stream avoiding temp files
Posted by Daniel Kulp <dk...@apache.org>.
Just to let you know, I was finally able to reproduce this, but it wasn't
easy. I added the SAAJ interceptor to our mtom test case and increased the
data size to 250K, but I still couldn't reproduce the error in eclipse.
However, when I ran mvn on the command line it failed, but not with Java 6,
only Java 5. I really have no idea why I cannot reliably reproduce it,
especially in eclipse. Doesn't matter what JDK I use in eclipse, I cannot
reproduce it there. Makes debugging hard.
In anycase, by adding a bunch of Thread.dumpStacks and printlns and such
throughout the code, I THINK I've tracked down what is causing this. I'll
run some more tests and and get changes committed up today, I hope. I
mostly need to go through and remove all my dumpStacks and printlns now as
well. :-(
Dan
On Wed September 23 2009 7:11:56 pm K1001010 wrote:
> Thank you for the quick reply. After hearing that the changes should be in
> 2.2.3 I dug further into my code and found a client-side interceptor which
> was causing all attachments to be cached.
>
> I fixed that, and now see a server-side problem when sending attachments.
> The server is using SSAJInInterceptor (via WSS4JInInterceptor) and on line
> 117 of the first there is this:
>
> for (Attachment a : atts) {
> if (a.getDataHandler().getDataSource() instanceof
> AttachmentDataSource) {
> try {
>
> ((AttachmentDataSource)a.getDataHandler().getDataSource()).cache();
>
>
> So the attachment is always being cached. But more problematic is this:
>
> After the above call the “inmem” flag of the CachedOutputStream is
> correctly set to false. By the time my application code calls
> CachedOutputStram.getInputStream() however, the flag is true, and hence an
> empty ByteArrayInputStream is returned.
>
> Is there an apache CXF test case with MTOM enabled, WSS4JInInterceptor
> used, and attachment larger than 64k? The one I wrote seems to
> consistently fail—the server side application code always getting a
> non-null, but empty, stream, when the attachment is larger than 64k.
>
> dkulp wrote:
> > On Wed September 16 2009 12:31:57 pm K1001010 wrote:
> >> Did these changes make it into the 2.2.3 release?
> >
> > It looks like it, yea.
> >
> >> I am testing with a 150MB attachment, and still see it being streamed to
> >> disk
> >
> > Hmm... I hate to ask this as I don't want a 150MB test case, but is
> > there
> > any way you could create a "small" test case?
> >
> > There are a few cases where we do have to buffer it ahead of time. If
> > there
> > are any SAAJ related things, we need to buffer. Schema validation is
> > another.
> > If there are multiple attachments, it's likely the others may need to be
> > buffered, etc... A testcase would help figure out what is going on.
> >
> > If you can debug it by setting a breakpoint in
> > org.apache.cxf.attachment.AttachementDataSource.cache() and determine why
> > cache is being called, that would also be helpful.
> >
> > Dan
> >
> >> I have this code:
> >>
> >>
> >> Picture picture = serviceProxy.getPictuer();
> >> DataHandler handler = picture.getImageData();
> >> InputStream is = handler.getInputStream();
> >>
> >>
> >> In version 2.2.2 the entire 150BM file would be streamed to temp file on
> >> line 3, which would return a file input stream (or a delegate too).
> >>
> >> In version 2.2.3 the entire 150MB file is streamed to temp file before
> >> line
> >> 1 returns.
> >>
> >> I disable schema validation using this system property:
> >>
> >> org.apache.spring.validation.mode=VALIDATION_NONE
> >>
> >> Is that correct? Anything else required to stream directly from the
> >> wire?
> >>
> >> Thanks for any help.
> >>
> >> dkulp wrote:
> >> > Well, I THOUGHT this would have been easily doable, but when I
> >> > checked, it turns out the attachments were not streaming properly in
> >> > MTOM mode.
> >> >
> >> > :-(
> >> >
> >> > I've gone through and fixed a BUNCH of things related to the
> >> > attachment handling and streaming and such so some of this is now
> >> > possible.
> >> >
> >> > Now, with the latest code, when you get the DataSource, if at all
> >> > possible,
> >> > the input stream that is returned will properly be streaming off the
> >>
> >> wire
> >>
> >> > instead of cached. Thus, you can copy directly from there to the
> >>
> >> output
> >>
> >> > stream.
> >> >
> >> > There are cases where this WON'T work stream directly, notably when
> >> > schema validation is turned on. In that case, to work around issues
> >>
> >> in
> >>
> >> > JAXB, we buffer the content. Also, if there are multiple
> >>
> >> attachments,
> >>
> >> > only the last
> >> > one can really be streamed this way. The others get buffered.
> >> >
> >> > Dan
> >> >
> >> > On Mon July 27 2009 5:50:00 am dp-isp wrote:
> >> >> Hello everybody,
> >> >>
> >> >> We have a CXF webservice that is a kind of "repository service" and a
> >> >> webapp working as frontend for that repository.
> >> >> Those two components are on separate hosts.
> >> >>
> >> >> We would need to realize a continuous stream of data when downloading
> >> >> attachments from CXF webservice to the users' browser through the
> >> >> frontend
> >> >> webapp.
> >> >>
> >> >> The webapp is basically a servlet that invokes the webservice,
> >>
> >> receives
> >>
> >> >> the
> >> >> attachment datahandler and copies datahandler InputStream to servlet
> >> >> response OutputStream (servlet code is at the end of this email).
> >> >>
> >> >> Our issue: is it possible to directly link datahandler InputStream to
> >> >> servlet response OutputStream without re-building the whole
> >> >> attachment in memoroy or in a temporary file? (as can be configured
> >> >> by setting org.apache.cxf.io.CachedOutputStream.Threshold,
> >> >> org.apache.cxf.io.CachedOutputStream.OutputDirectory)
> >> >>
> >> >> Going more into details, is there a way to send to CXF the output
> >>
> >> buffer
> >>
> >> >> to
> >> >> write the attachment on? In such case we would send to CXF the
> >>
> >> sevlet's
> >>
> >> >> output buffer.
> >> >>
> >> >> Alternatively, is there a way to manage CXF working asynchronously
> >>
> >> with
> >>
> >> >> attachments, giving back control to the servlet before attachment
> >> >> downolad
> >> >> completion?
> >> >>
> >> >> Many thanks,
> >> >> Marco
> >> >>
> >> >> ---
> >> >>
> >> >> This is the servlet's involved code:
> >> >>
> >> >> ...
> >> >> StreamingService_Service ss = new StreamingService_Service(new
> >> >> URL(serviceURL));
> >> >> StreamingService port = ss.getServiceImplPort(new MTOMFeature());
> >> >>
> >> >> Resume myFile = port.downloadMethod(fileName, fileType, usr, pwd,
> >>
> >> repo);
> >>
> >> >> DataHandler handler = myFile.getResume();
> >> >>
> >> >> response.setHeader("Pragma", "No-cache");
> >> >> response.setContentType("application/octet-stream");
> >> >> response.setHeader("content-disposition", "attachment; filename=" +
> >> >> fileName + "." + fileType);
> >> >> try {
> >> >> InputStream is = handler.getInputStream();
> >> >> try {
> >> >> OutputStream os = response.getOutputStream();
> >> >> try {
> >> >> byte[] b = new byte[100000];
> >> >> int bytesRead = 0;
> >> >> while ((bytesRead = is.read(b)) != -1) {
> >> >> os.write(b, 0, bytesRead);
> >> >> }
> >> >> System.out.println("Download del file " +
> >> >> myFile.getCandidateName()
> >>
> >> +
> >>
> >> >> "." + myFile.getResumeFileType() + " completato.");
> >> >> } finally {
> >> >> os.flush();
> >> >> os.close();
> >> >> }
> >> >> } finally {
> >> >> is.close();
> >> >> }
> >> >>
> >> >> } catch (Exception e) {
> >> >> e.printStackTrace();
> >> >> }
> >> >> ...
>
--
Daniel Kulp
dkulp@apache.org
http://www.dankulp.com/blog
Re: Streaming large attachments by linking input and output stream
avoiding temp files
Posted by Jakob Günther <ja...@yahoo.de>.
Yeah, i have the same problem.
Regards, Jakob
K1001010 schrieb:
> Thank you for the quick reply. After hearing that the changes should be in
> 2.2.3 I dug further into my code and found a client-side interceptor which
> was causing all attachments to be cached.
>
> I fixed that, and now see a server-side problem when sending attachments.
> The server is using SSAJInInterceptor (via WSS4JInInterceptor) and on line
> 117 of the first there is this:
>
> for (Attachment a : atts) {
> if (a.getDataHandler().getDataSource() instanceof AttachmentDataSource)
> {
> try {
>
> ((AttachmentDataSource)a.getDataHandler().getDataSource()).cache();
>
>
> So the attachment is always being cached. But more problematic is this:
>
> After the above call the “inmem” flag of the CachedOutputStream is correctly
> set to false. By the time my application code calls
> CachedOutputStram.getInputStream() however, the flag is true, and hence an
> empty ByteArrayInputStream is returned.
>
> Is there an apache CXF test case with MTOM enabled, WSS4JInInterceptor used,
> and attachment larger than 64k? The one I wrote seems to consistently
> fail—the server side application code always getting a non-null, but empty,
> stream, when the attachment is larger than 64k.
>
>
>
>
>
>
>
>
>
>
>
> dkulp wrote:
>> On Wed September 16 2009 12:31:57 pm K1001010 wrote:
>>> Did these changes make it into the 2.2.3 release?
>> It looks like it, yea.
>>
>>
>>> I am testing with a 150MB attachment, and still see it being streamed to
>>> disk
>> Hmm... I hate to ask this as I don't want a 150MB test case, but is
>> there
>> any way you could create a "small" test case?
>>
>> There are a few cases where we do have to buffer it ahead of time. If
>> there
>> are any SAAJ related things, we need to buffer. Schema validation is
>> another.
>> If there are multiple attachments, it's likely the others may need to be
>> buffered, etc... A testcase would help figure out what is going on.
>>
>> If you can debug it by setting a breakpoint in
>> org.apache.cxf.attachment.AttachementDataSource.cache() and determine why
>> cache is being called, that would also be helpful.
>>
>> Dan
>>
>>
>>
>>> I have this code:
>>>
>>>
>>> Picture picture = serviceProxy.getPictuer();
>>> DataHandler handler = picture.getImageData();
>>> InputStream is = handler.getInputStream();
>>>
>>>
>>> In version 2.2.2 the entire 150BM file would be streamed to temp file on
>>> line 3, which would return a file input stream (or a delegate too).
>>>
>>> In version 2.2.3 the entire 150MB file is streamed to temp file before
>>> line
>>> 1 returns.
>>>
>>> I disable schema validation using this system property:
>>>
>>> org.apache.spring.validation.mode=VALIDATION_NONE
>>>
>>> Is that correct? Anything else required to stream directly from the
>>> wire?
>>>
>>> Thanks for any help.
>>>
>>> dkulp wrote:
>>>> Well, I THOUGHT this would have been easily doable, but when I checked,
>>>> it turns out the attachments were not streaming properly in MTOM mode.
>>>> :-(
>>>>
>>>> I've gone through and fixed a BUNCH of things related to the attachment
>>>> handling and streaming and such so some of this is now possible.
>>>>
>>>> Now, with the latest code, when you get the DataSource, if at all
>>>> possible,
>>>> the input stream that is returned will properly be streaming off the
>>> wire
>>>> instead of cached. Thus, you can copy directly from there to the
>>> output
>>>> stream.
>>>>
>>>> There are cases where this WON'T work stream directly, notably when
>>>> schema validation is turned on. In that case, to work around issues
>>> in
>>>> JAXB, we buffer the content. Also, if there are multiple
>>> attachments,
>>>> only the last
>>>> one can really be streamed this way. The others get buffered.
>>>>
>>>> Dan
>>>>
>>>> On Mon July 27 2009 5:50:00 am dp-isp wrote:
>>>>> Hello everybody,
>>>>>
>>>>> We have a CXF webservice that is a kind of "repository service" and a
>>>>> webapp working as frontend for that repository.
>>>>> Those two components are on separate hosts.
>>>>>
>>>>> We would need to realize a continuous stream of data when downloading
>>>>> attachments from CXF webservice to the users' browser through the
>>>>> frontend
>>>>> webapp.
>>>>>
>>>>> The webapp is basically a servlet that invokes the webservice,
>>> receives
>>>>> the
>>>>> attachment datahandler and copies datahandler InputStream to servlet
>>>>> response OutputStream (servlet code is at the end of this email).
>>>>>
>>>>> Our issue: is it possible to directly link datahandler InputStream to
>>>>> servlet response OutputStream without re-building the whole attachment
>>>>> in memoroy or in a temporary file? (as can be configured by setting
>>>>> org.apache.cxf.io.CachedOutputStream.Threshold,
>>>>> org.apache.cxf.io.CachedOutputStream.OutputDirectory)
>>>>>
>>>>> Going more into details, is there a way to send to CXF the output
>>> buffer
>>>>> to
>>>>> write the attachment on? In such case we would send to CXF the
>>> sevlet's
>>>>> output buffer.
>>>>>
>>>>> Alternatively, is there a way to manage CXF working asynchronously
>>> with
>>>>> attachments, giving back control to the servlet before attachment
>>>>> downolad
>>>>> completion?
>>>>>
>>>>> Many thanks,
>>>>> Marco
>>>>>
>>>>> ---
>>>>>
>>>>> This is the servlet's involved code:
>>>>>
>>>>> ...
>>>>> StreamingService_Service ss = new StreamingService_Service(new
>>>>> URL(serviceURL));
>>>>> StreamingService port = ss.getServiceImplPort(new MTOMFeature());
>>>>>
>>>>> Resume myFile = port.downloadMethod(fileName, fileType, usr, pwd,
>>> repo);
>>>>> DataHandler handler = myFile.getResume();
>>>>>
>>>>> response.setHeader("Pragma", "No-cache");
>>>>> response.setContentType("application/octet-stream");
>>>>> response.setHeader("content-disposition", "attachment; filename=" +
>>>>> fileName + "." + fileType);
>>>>> try {
>>>>> InputStream is = handler.getInputStream();
>>>>> try {
>>>>> OutputStream os = response.getOutputStream();
>>>>> try {
>>>>> byte[] b = new byte[100000];
>>>>> int bytesRead = 0;
>>>>> while ((bytesRead = is.read(b)) != -1) {
>>>>> os.write(b, 0, bytesRead);
>>>>> }
>>>>> System.out.println("Download del file " + myFile.getCandidateName()
>>> +
>>>>> "." + myFile.getResumeFileType() + " completato.");
>>>>> } finally {
>>>>> os.flush();
>>>>> os.close();
>>>>> }
>>>>> } finally {
>>>>> is.close();
>>>>> }
>>>>>
>>>>> } catch (Exception e) {
>>>>> e.printStackTrace();
>>>>> }
>>>>> ...
>> --
>> Daniel Kulp
>> dkulp@apache.org
>> http://www.dankulp.com/blog
>>
>>
>
Re: Streaming large attachments by linking input and output stream
avoiding temp files
Posted by K1001010 <ke...@monsanto.com>.
Thank you for the quick reply. After hearing that the changes should be in
2.2.3 I dug further into my code and found a client-side interceptor which
was causing all attachments to be cached.
I fixed that, and now see a server-side problem when sending attachments.
The server is using SSAJInInterceptor (via WSS4JInInterceptor) and on line
117 of the first there is this:
for (Attachment a : atts) {
if (a.getDataHandler().getDataSource() instanceof AttachmentDataSource)
{
try {
((AttachmentDataSource)a.getDataHandler().getDataSource()).cache();
So the attachment is always being cached. But more problematic is this:
After the above call the “inmem” flag of the CachedOutputStream is correctly
set to false. By the time my application code calls
CachedOutputStram.getInputStream() however, the flag is true, and hence an
empty ByteArrayInputStream is returned.
Is there an apache CXF test case with MTOM enabled, WSS4JInInterceptor used,
and attachment larger than 64k? The one I wrote seems to consistently
fail—the server side application code always getting a non-null, but empty,
stream, when the attachment is larger than 64k.
dkulp wrote:
>
> On Wed September 16 2009 12:31:57 pm K1001010 wrote:
>> Did these changes make it into the 2.2.3 release?
>
> It looks like it, yea.
>
>
>> I am testing with a 150MB attachment, and still see it being streamed to
>> disk
>
> Hmm... I hate to ask this as I don't want a 150MB test case, but is
> there
> any way you could create a "small" test case?
>
> There are a few cases where we do have to buffer it ahead of time. If
> there
> are any SAAJ related things, we need to buffer. Schema validation is
> another.
> If there are multiple attachments, it's likely the others may need to be
> buffered, etc... A testcase would help figure out what is going on.
>
> If you can debug it by setting a breakpoint in
> org.apache.cxf.attachment.AttachementDataSource.cache() and determine why
> cache is being called, that would also be helpful.
>
> Dan
>
>
>
>>
>> I have this code:
>>
>>
>> Picture picture = serviceProxy.getPictuer();
>> DataHandler handler = picture.getImageData();
>> InputStream is = handler.getInputStream();
>>
>>
>> In version 2.2.2 the entire 150BM file would be streamed to temp file on
>> line 3, which would return a file input stream (or a delegate too).
>>
>> In version 2.2.3 the entire 150MB file is streamed to temp file before
>> line
>> 1 returns.
>>
>> I disable schema validation using this system property:
>>
>> org.apache.spring.validation.mode=VALIDATION_NONE
>>
>> Is that correct? Anything else required to stream directly from the
>> wire?
>>
>> Thanks for any help.
>>
>> dkulp wrote:
>> > Well, I THOUGHT this would have been easily doable, but when I checked,
>> > it turns out the attachments were not streaming properly in MTOM mode.
>> > :-(
>> >
>> > I've gone through and fixed a BUNCH of things related to the attachment
>> > handling and streaming and such so some of this is now possible.
>> >
>> > Now, with the latest code, when you get the DataSource, if at all
>> > possible,
>> > the input stream that is returned will properly be streaming off the
>> wire
>> > instead of cached. Thus, you can copy directly from there to the
>> output
>> > stream.
>> >
>> > There are cases where this WON'T work stream directly, notably when
>> > schema validation is turned on. In that case, to work around issues
>> in
>> > JAXB, we buffer the content. Also, if there are multiple
>> attachments,
>> > only the last
>> > one can really be streamed this way. The others get buffered.
>> >
>> > Dan
>> >
>> > On Mon July 27 2009 5:50:00 am dp-isp wrote:
>> >> Hello everybody,
>> >>
>> >> We have a CXF webservice that is a kind of "repository service" and a
>> >> webapp working as frontend for that repository.
>> >> Those two components are on separate hosts.
>> >>
>> >> We would need to realize a continuous stream of data when downloading
>> >> attachments from CXF webservice to the users' browser through the
>> >> frontend
>> >> webapp.
>> >>
>> >> The webapp is basically a servlet that invokes the webservice,
>> receives
>> >> the
>> >> attachment datahandler and copies datahandler InputStream to servlet
>> >> response OutputStream (servlet code is at the end of this email).
>> >>
>> >> Our issue: is it possible to directly link datahandler InputStream to
>> >> servlet response OutputStream without re-building the whole attachment
>> >> in memoroy or in a temporary file? (as can be configured by setting
>> >> org.apache.cxf.io.CachedOutputStream.Threshold,
>> >> org.apache.cxf.io.CachedOutputStream.OutputDirectory)
>> >>
>> >> Going more into details, is there a way to send to CXF the output
>> buffer
>> >> to
>> >> write the attachment on? In such case we would send to CXF the
>> sevlet's
>> >> output buffer.
>> >>
>> >> Alternatively, is there a way to manage CXF working asynchronously
>> with
>> >> attachments, giving back control to the servlet before attachment
>> >> downolad
>> >> completion?
>> >>
>> >> Many thanks,
>> >> Marco
>> >>
>> >> ---
>> >>
>> >> This is the servlet's involved code:
>> >>
>> >> ...
>> >> StreamingService_Service ss = new StreamingService_Service(new
>> >> URL(serviceURL));
>> >> StreamingService port = ss.getServiceImplPort(new MTOMFeature());
>> >>
>> >> Resume myFile = port.downloadMethod(fileName, fileType, usr, pwd,
>> repo);
>> >> DataHandler handler = myFile.getResume();
>> >>
>> >> response.setHeader("Pragma", "No-cache");
>> >> response.setContentType("application/octet-stream");
>> >> response.setHeader("content-disposition", "attachment; filename=" +
>> >> fileName + "." + fileType);
>> >> try {
>> >> InputStream is = handler.getInputStream();
>> >> try {
>> >> OutputStream os = response.getOutputStream();
>> >> try {
>> >> byte[] b = new byte[100000];
>> >> int bytesRead = 0;
>> >> while ((bytesRead = is.read(b)) != -1) {
>> >> os.write(b, 0, bytesRead);
>> >> }
>> >> System.out.println("Download del file " + myFile.getCandidateName()
>> +
>> >> "." + myFile.getResumeFileType() + " completato.");
>> >> } finally {
>> >> os.flush();
>> >> os.close();
>> >> }
>> >> } finally {
>> >> is.close();
>> >> }
>> >>
>> >> } catch (Exception e) {
>> >> e.printStackTrace();
>> >> }
>> >> ...
>>
>
> --
> Daniel Kulp
> dkulp@apache.org
> http://www.dankulp.com/blog
>
>
--
View this message in context: http://www.nabble.com/Streaming-large-attachments-by-linking-input-and-output-stream-avoiding-temp-files-tp24677330p25531325.html
Sent from the cxf-user mailing list archive at Nabble.com.
RE: Streaming large attachments by linking input and output stream avoiding temp files
Posted by "WINKLER, KEITH DAVID [AG-Contractor/1000]" <ke...@monsanto.com>.
Thank you for the quick reply. After hearing that the changes should be
in 2.2.3 I dug further into my code and found a client-side interceptor
which was causing all attachments to be cached.
Is there an apache CXF test case with MTOM enabled, WSS4JInInterceptor
used, and attachment larger than 64k? The one I wrote seems to
consistently fail-the server side application code always getting a
non-null, but empty, stream when the attachment is larger than 64k.
The server is using SSAJInInterceptor (via WSS4JInInterceptor) and on
line 117 of the first there is this:
for (Attachment a : atts) {
if (a.getDataHandler().getDataSource() instanceof
AttachmentDataSource) {
try {
((AttachmentDataSource)a.getDataHandler().getDataSource()).cache();
So the attachment is always being cached. But more problematic is this:
After the above call the "inmem" flag of the CachedOutputStream is
correctly set to false. By the time my application code calls
CachedOutputStram.getInputStream() however, the flag is true, and hence
an empty ByteArrayInputStream is returned.
Thanks for any help.
-----Original Message-----
From: Daniel Kulp [mailto:dkulp@apache.org]
Sent: Wednesday, September 16, 2009 9:33 PM
To: users@cxf.apache.org
Cc: WINKLER, KEITH DAVID [AG-Contractor/1000]
Subject: Re: Streaming large attachments by linking input and output
stream avoiding temp files
On Wed September 16 2009 12:31:57 pm K1001010 wrote:
> Did these changes make it into the 2.2.3 release?
It looks like it, yea.
> I am testing with a 150MB attachment, and still see it being streamed
to
> disk
Hmm... I hate to ask this as I don't want a 150MB test case, but is
there
any way you could create a "small" test case?
There are a few cases where we do have to buffer it ahead of time. If
there
are any SAAJ related things, we need to buffer. Schema validation is
another.
If there are multiple attachments, it's likely the others may need to be
buffered, etc... A testcase would help figure out what is going on.
If you can debug it by setting a breakpoint in
org.apache.cxf.attachment.AttachementDataSource.cache() and determine
why
cache is being called, that would also be helpful.
Dan
>
> I have this code:
>
>
> Picture picture = serviceProxy.getPictuer();
> DataHandler handler = picture.getImageData();
> InputStream is = handler.getInputStream();
>
>
> In version 2.2.2 the entire 150BM file would be streamed to temp file
on
> line 3, which would return a file input stream (or a delegate too).
>
> In version 2.2.3 the entire 150MB file is streamed to temp file before
line
> 1 returns.
>
> I disable schema validation using this system property:
>
> org.apache.spring.validation.mode=VALIDATION_NONE
>
> Is that correct? Anything else required to stream directly from the
wire?
>
> Thanks for any help.
>
> dkulp wrote:
> > Well, I THOUGHT this would have been easily doable, but when I
checked,
> > it turns out the attachments were not streaming properly in MTOM
mode.
> > :-(
> >
> > I've gone through and fixed a BUNCH of things related to the
attachment
> > handling and streaming and such so some of this is now possible.
> >
> > Now, with the latest code, when you get the DataSource, if at all
> > possible,
> > the input stream that is returned will properly be streaming off the
wire
> > instead of cached. Thus, you can copy directly from there to the
output
> > stream.
> >
> > There are cases where this WON'T work stream directly, notably when
> > schema validation is turned on. In that case, to work around
issues in
> > JAXB, we buffer the content. Also, if there are multiple
attachments,
> > only the last
> > one can really be streamed this way. The others get buffered.
> >
> > Dan
> >
> > On Mon July 27 2009 5:50:00 am dp-isp wrote:
> >> Hello everybody,
> >>
> >> We have a CXF webservice that is a kind of "repository service" and
a
> >> webapp working as frontend for that repository.
> >> Those two components are on separate hosts.
> >>
> >> We would need to realize a continuous stream of data when
downloading
> >> attachments from CXF webservice to the users' browser through the
> >> frontend
> >> webapp.
> >>
> >> The webapp is basically a servlet that invokes the webservice,
receives
> >> the
> >> attachment datahandler and copies datahandler InputStream to
servlet
> >> response OutputStream (servlet code is at the end of this email).
> >>
> >> Our issue: is it possible to directly link datahandler InputStream
to
> >> servlet response OutputStream without re-building the whole
attachment
> >> in memoroy or in a temporary file? (as can be configured by setting
> >> org.apache.cxf.io.CachedOutputStream.Threshold,
> >> org.apache.cxf.io.CachedOutputStream.OutputDirectory)
> >>
> >> Going more into details, is there a way to send to CXF the output
buffer
> >> to
> >> write the attachment on? In such case we would send to CXF the
sevlet's
> >> output buffer.
> >>
> >> Alternatively, is there a way to manage CXF working asynchronously
with
> >> attachments, giving back control to the servlet before attachment
> >> downolad
> >> completion?
> >>
> >> Many thanks,
> >> Marco
> >>
> >> ---
> >>
> >> This is the servlet's involved code:
> >>
> >> ...
> >> StreamingService_Service ss = new StreamingService_Service(new
> >> URL(serviceURL));
> >> StreamingService port = ss.getServiceImplPort(new MTOMFeature());
> >>
> >> Resume myFile = port.downloadMethod(fileName, fileType, usr, pwd,
repo);
> >> DataHandler handler = myFile.getResume();
> >>
> >> response.setHeader("Pragma", "No-cache");
> >> response.setContentType("application/octet-stream");
> >> response.setHeader("content-disposition", "attachment; filename=" +
> >> fileName + "." + fileType);
> >> try {
> >> InputStream is = handler.getInputStream();
> >> try {
> >> OutputStream os = response.getOutputStream();
> >> try {
> >> byte[] b = new byte[100000];
> >> int bytesRead = 0;
> >> while ((bytesRead = is.read(b)) != -1) {
> >> os.write(b, 0, bytesRead);
> >> }
> >> System.out.println("Download del file " +
myFile.getCandidateName() +
> >> "." + myFile.getResumeFileType() + " completato.");
> >> } finally {
> >> os.flush();
> >> os.close();
> >> }
> >> } finally {
> >> is.close();
> >> }
> >>
> >> } catch (Exception e) {
> >> e.printStackTrace();
> >> }
> >> ...
>
--
Daniel Kulp
dkulp@apache.org
http://www.dankulp.com/blog
---------------------------------------------------------------------------------------------------------
This e-mail message may contain privileged and/or confidential information, and is intended to be received only by persons entitled to receive such information. If you have received this e-mail in error, please notify the sender immediately. Please delete it and all attachments from any servers, hard drives or any other media. Other use of this e-mail by you is strictly prohibited.
All e-mails and attachments sent and received are subject to monitoring, reading and archival by Monsanto, including its subsidiaries. The recipient of this e-mail is solely responsible for checking for the presence of "Viruses" or other "Malware". Monsanto, along with its subsidiaries, accepts no liability for any damage caused by any such code transmitted by or accompanying this e-mail or any attachment.
---------------------------------------------------------------------------------------------------------
Re: Streaming large attachments by linking input and output stream avoiding temp files
Posted by Daniel Kulp <dk...@apache.org>.
On Wed September 16 2009 12:31:57 pm K1001010 wrote:
> Did these changes make it into the 2.2.3 release?
It looks like it, yea.
> I am testing with a 150MB attachment, and still see it being streamed to
> disk
Hmm... I hate to ask this as I don't want a 150MB test case, but is there
any way you could create a "small" test case?
There are a few cases where we do have to buffer it ahead of time. If there
are any SAAJ related things, we need to buffer. Schema validation is another.
If there are multiple attachments, it's likely the others may need to be
buffered, etc... A testcase would help figure out what is going on.
If you can debug it by setting a breakpoint in
org.apache.cxf.attachment.AttachementDataSource.cache() and determine why
cache is being called, that would also be helpful.
Dan
>
> I have this code:
>
>
> Picture picture = serviceProxy.getPictuer();
> DataHandler handler = picture.getImageData();
> InputStream is = handler.getInputStream();
>
>
> In version 2.2.2 the entire 150BM file would be streamed to temp file on
> line 3, which would return a file input stream (or a delegate too).
>
> In version 2.2.3 the entire 150MB file is streamed to temp file before line
> 1 returns.
>
> I disable schema validation using this system property:
>
> org.apache.spring.validation.mode=VALIDATION_NONE
>
> Is that correct? Anything else required to stream directly from the wire?
>
> Thanks for any help.
>
> dkulp wrote:
> > Well, I THOUGHT this would have been easily doable, but when I checked,
> > it turns out the attachments were not streaming properly in MTOM mode.
> > :-(
> >
> > I've gone through and fixed a BUNCH of things related to the attachment
> > handling and streaming and such so some of this is now possible.
> >
> > Now, with the latest code, when you get the DataSource, if at all
> > possible,
> > the input stream that is returned will properly be streaming off the wire
> > instead of cached. Thus, you can copy directly from there to the output
> > stream.
> >
> > There are cases where this WON'T work stream directly, notably when
> > schema validation is turned on. In that case, to work around issues in
> > JAXB, we buffer the content. Also, if there are multiple attachments,
> > only the last
> > one can really be streamed this way. The others get buffered.
> >
> > Dan
> >
> > On Mon July 27 2009 5:50:00 am dp-isp wrote:
> >> Hello everybody,
> >>
> >> We have a CXF webservice that is a kind of "repository service" and a
> >> webapp working as frontend for that repository.
> >> Those two components are on separate hosts.
> >>
> >> We would need to realize a continuous stream of data when downloading
> >> attachments from CXF webservice to the users' browser through the
> >> frontend
> >> webapp.
> >>
> >> The webapp is basically a servlet that invokes the webservice, receives
> >> the
> >> attachment datahandler and copies datahandler InputStream to servlet
> >> response OutputStream (servlet code is at the end of this email).
> >>
> >> Our issue: is it possible to directly link datahandler InputStream to
> >> servlet response OutputStream without re-building the whole attachment
> >> in memoroy or in a temporary file? (as can be configured by setting
> >> org.apache.cxf.io.CachedOutputStream.Threshold,
> >> org.apache.cxf.io.CachedOutputStream.OutputDirectory)
> >>
> >> Going more into details, is there a way to send to CXF the output buffer
> >> to
> >> write the attachment on? In such case we would send to CXF the sevlet's
> >> output buffer.
> >>
> >> Alternatively, is there a way to manage CXF working asynchronously with
> >> attachments, giving back control to the servlet before attachment
> >> downolad
> >> completion?
> >>
> >> Many thanks,
> >> Marco
> >>
> >> ---
> >>
> >> This is the servlet's involved code:
> >>
> >> ...
> >> StreamingService_Service ss = new StreamingService_Service(new
> >> URL(serviceURL));
> >> StreamingService port = ss.getServiceImplPort(new MTOMFeature());
> >>
> >> Resume myFile = port.downloadMethod(fileName, fileType, usr, pwd, repo);
> >> DataHandler handler = myFile.getResume();
> >>
> >> response.setHeader("Pragma", "No-cache");
> >> response.setContentType("application/octet-stream");
> >> response.setHeader("content-disposition", "attachment; filename=" +
> >> fileName + "." + fileType);
> >> try {
> >> InputStream is = handler.getInputStream();
> >> try {
> >> OutputStream os = response.getOutputStream();
> >> try {
> >> byte[] b = new byte[100000];
> >> int bytesRead = 0;
> >> while ((bytesRead = is.read(b)) != -1) {
> >> os.write(b, 0, bytesRead);
> >> }
> >> System.out.println("Download del file " + myFile.getCandidateName() +
> >> "." + myFile.getResumeFileType() + " completato.");
> >> } finally {
> >> os.flush();
> >> os.close();
> >> }
> >> } finally {
> >> is.close();
> >> }
> >>
> >> } catch (Exception e) {
> >> e.printStackTrace();
> >> }
> >> ...
>
--
Daniel Kulp
dkulp@apache.org
http://www.dankulp.com/blog
Re: Streaming large attachments by linking input and output stream
avoiding temp files
Posted by K1001010 <ke...@monsanto.com>.
Did these changes make it into the 2.2.3 release?
I am testing with a 150MB attachment, and still see it being streamed to
disk.
I have this code:
Picture picture = serviceProxy.getPictuer();
DataHandler handler = picture.getImageData();
InputStream is = handler.getInputStream();
In version 2.2.2 the entire 150BM file would be streamed to temp file on
line 3, which would return a file input stream (or a delegate too).
In version 2.2.3 the entire 150MB file is streamed to temp file before line
1 returns.
I disable schema validation using this system property:
org.apache.spring.validation.mode=VALIDATION_NONE
Is that correct? Anything else required to stream directly from the wire?
Thanks for any help.
dkulp wrote:
>
>
> Well, I THOUGHT this would have been easily doable, but when I checked, it
> turns out the attachments were not streaming properly in MTOM mode. :-(
>
> I've gone through and fixed a BUNCH of things related to the attachment
> handling and streaming and such so some of this is now possible.
>
> Now, with the latest code, when you get the DataSource, if at all
> possible,
> the input stream that is returned will properly be streaming off the wire
> instead of cached. Thus, you can copy directly from there to the output
> stream.
>
> There are cases where this WON'T work stream directly, notably when schema
> validation is turned on. In that case, to work around issues in JAXB, we
> buffer the content. Also, if there are multiple attachments, only the
> last
> one can really be streamed this way. The others get buffered.
>
> Dan
>
>
>
> On Mon July 27 2009 5:50:00 am dp-isp wrote:
>> Hello everybody,
>>
>> We have a CXF webservice that is a kind of "repository service" and a
>> webapp working as frontend for that repository.
>> Those two components are on separate hosts.
>>
>> We would need to realize a continuous stream of data when downloading
>> attachments from CXF webservice to the users' browser through the
>> frontend
>> webapp.
>>
>> The webapp is basically a servlet that invokes the webservice, receives
>> the
>> attachment datahandler and copies datahandler InputStream to servlet
>> response OutputStream (servlet code is at the end of this email).
>>
>> Our issue: is it possible to directly link datahandler InputStream to
>> servlet response OutputStream without re-building the whole attachment in
>> memoroy or in a temporary file? (as can be configured by setting
>> org.apache.cxf.io.CachedOutputStream.Threshold,
>> org.apache.cxf.io.CachedOutputStream.OutputDirectory)
>>
>> Going more into details, is there a way to send to CXF the output buffer
>> to
>> write the attachment on? In such case we would send to CXF the sevlet's
>> output buffer.
>>
>> Alternatively, is there a way to manage CXF working asynchronously with
>> attachments, giving back control to the servlet before attachment
>> downolad
>> completion?
>>
>> Many thanks,
>> Marco
>>
>> ---
>>
>> This is the servlet's involved code:
>>
>> ...
>> StreamingService_Service ss = new StreamingService_Service(new
>> URL(serviceURL));
>> StreamingService port = ss.getServiceImplPort(new MTOMFeature());
>>
>> Resume myFile = port.downloadMethod(fileName, fileType, usr, pwd, repo);
>> DataHandler handler = myFile.getResume();
>>
>> response.setHeader("Pragma", "No-cache");
>> response.setContentType("application/octet-stream");
>> response.setHeader("content-disposition", "attachment; filename=" +
>> fileName + "." + fileType);
>> try {
>> InputStream is = handler.getInputStream();
>> try {
>> OutputStream os = response.getOutputStream();
>> try {
>> byte[] b = new byte[100000];
>> int bytesRead = 0;
>> while ((bytesRead = is.read(b)) != -1) {
>> os.write(b, 0, bytesRead);
>> }
>> System.out.println("Download del file " + myFile.getCandidateName() +
>> "." + myFile.getResumeFileType() + " completato.");
>> } finally {
>> os.flush();
>> os.close();
>> }
>> } finally {
>> is.close();
>> }
>>
>> } catch (Exception e) {
>> e.printStackTrace();
>> }
>> ...
>
> --
> Daniel Kulp
> dkulp@apache.org
> http://www.dankulp.com/blog
>
>
--
View this message in context: http://www.nabble.com/Streaming-large-attachments-by-linking-input-and-output-stream-avoiding-temp-files-tp24677330p25476020.html
Sent from the cxf-user mailing list archive at Nabble.com.
Re: Streaming large attachments by linking input and output stream avoiding temp files
Posted by Daniel Kulp <dk...@apache.org>.
Well, I THOUGHT this would have been easily doable, but when I checked, it
turns out the attachments were not streaming properly in MTOM mode. :-(
I've gone through and fixed a BUNCH of things related to the attachment
handling and streaming and such so some of this is now possible.
Now, with the latest code, when you get the DataSource, if at all possible,
the input stream that is returned will properly be streaming off the wire
instead of cached. Thus, you can copy directly from there to the output
stream.
There are cases where this WON'T work stream directly, notably when schema
validation is turned on. In that case, to work around issues in JAXB, we
buffer the content. Also, if there are multiple attachments, only the last
one can really be streamed this way. The others get buffered.
Dan
On Mon July 27 2009 5:50:00 am dp-isp wrote:
> Hello everybody,
>
> We have a CXF webservice that is a kind of "repository service" and a
> webapp working as frontend for that repository.
> Those two components are on separate hosts.
>
> We would need to realize a continuous stream of data when downloading
> attachments from CXF webservice to the users' browser through the frontend
> webapp.
>
> The webapp is basically a servlet that invokes the webservice, receives the
> attachment datahandler and copies datahandler InputStream to servlet
> response OutputStream (servlet code is at the end of this email).
>
> Our issue: is it possible to directly link datahandler InputStream to
> servlet response OutputStream without re-building the whole attachment in
> memoroy or in a temporary file? (as can be configured by setting
> org.apache.cxf.io.CachedOutputStream.Threshold,
> org.apache.cxf.io.CachedOutputStream.OutputDirectory)
>
> Going more into details, is there a way to send to CXF the output buffer to
> write the attachment on? In such case we would send to CXF the sevlet's
> output buffer.
>
> Alternatively, is there a way to manage CXF working asynchronously with
> attachments, giving back control to the servlet before attachment downolad
> completion?
>
> Many thanks,
> Marco
>
> ---
>
> This is the servlet's involved code:
>
> ...
> StreamingService_Service ss = new StreamingService_Service(new
> URL(serviceURL));
> StreamingService port = ss.getServiceImplPort(new MTOMFeature());
>
> Resume myFile = port.downloadMethod(fileName, fileType, usr, pwd, repo);
> DataHandler handler = myFile.getResume();
>
> response.setHeader("Pragma", "No-cache");
> response.setContentType("application/octet-stream");
> response.setHeader("content-disposition", "attachment; filename=" +
> fileName + "." + fileType);
> try {
> InputStream is = handler.getInputStream();
> try {
> OutputStream os = response.getOutputStream();
> try {
> byte[] b = new byte[100000];
> int bytesRead = 0;
> while ((bytesRead = is.read(b)) != -1) {
> os.write(b, 0, bytesRead);
> }
> System.out.println("Download del file " + myFile.getCandidateName() +
> "." + myFile.getResumeFileType() + " completato.");
> } finally {
> os.flush();
> os.close();
> }
> } finally {
> is.close();
> }
>
> } catch (Exception e) {
> e.printStackTrace();
> }
> ...
--
Daniel Kulp
dkulp@apache.org
http://www.dankulp.com/blog