Downloading huge binary files using MTOM

[Stadelmann Josef <jo...@axa-winterthur.ch>]

Hi,

I have successfully developed and we use it since years, a Binary File Service, based on JAXWS/JAXB, using MTOM to download large binary data. At the service side there is a DataHandler involved which reads the bytes into a byte[] called ImageDepot. (The axis2 MTOM example). The problem is now that the size of this byte array might have become too small. At least I get that impression at the client side, which is a WCF client using a proxy generated by the svcutil asking for the WSDL data from the running SpezplaFS (File Service).

In short: all works fine until at the client the byte[] array becomes too small to keep more 2^31 bytes of data.

In short: How can I use MTOM and stream bytes from huge binary files, files with a size up to long.MaxValue, from a service to a client by using a64K byte buffer?

The client should be able to call this service, the stub, only once without the need to manage junks.

At the client I would see something like

BufferdInputStream bis = new BufferedInputStream(new ... what ever stream would talk with  my MTOM streaming service ...);
BufferedOutputStream bos = new BufferedOutputStream(new FileOutputStream(datafile));
Byte[] buffer;

While (! bis.wead(buffer, 0, Size)== 0) {
    bos.write(buffer, 0, Size);
}
If (!bis==null) bis.Close();
If (!bos==null) bos.Close();

And pointers to a working example are very welcome
Thank you
Josef

AW: Downloading huge binary files using MTOM

[Stadelmann Josef <jo...@axa-winterthur.ch>]

Hi

thanks a lot, I will think about what you describe.

Meanwhile I found out that the first exception occurs at the svcutil.exe generated WCF client proxy.
After that, my SpezplaFS (FileService) just raises a "broken pipe" exception.

Looking closer says me that the service already has a stream in effect and does not buffer the whole file.
So the service seams to behave properly, at least for downloads.

My focus needs to be at the client side and that is making a proxy without using a svcutil.exe generated code.
That is to say, I shall have a stream from the WCF framework and feed that, buffered or not, into a file.

How to do that is still a bit of a mystery for me.
And I think for many others too because seldom you find someone knowing the ( Axis2 / JAXWS / JAXB ) and
the other part, the WCF side very well.

Josef




Von: Okken,Brett [mailto:BOKKEN@CERNER.COM]
Gesendet: Donnerstag, 23. April 2015 15:38
An: java-user@axis.apache.org; axis-user@ws.apache.org
Betreff: RE: Downloading huge binary files using MTOM

This should be possible by building a custom javax.activation.DataSource implementation which is capable of generating the InputStream on demand (and potentially multiple times).

You may run into problems with some extensions (such as rampart for JAX-WS) that may require converting to a DOM representation, which forces the binary content into the xml itself.

Brett Okken | CAMM Platform Services | Knowledge Architect | 816.201.6112 | www.cerner.com<http://www.cerner.com> | bokken@cerner.com<ma...@cerner.com>

From: Stadelmann Josef [mailto:josef.stadelmann@axa-winterthur.ch]
Sent: Thursday, April 23, 2015 5:58 AM
To: axis-user@ws.apache.org<ma...@ws.apache.org>
Subject: Downloading huge binary files using MTOM

Hi,

I have successfully developed and we use it since years, a Binary File Service, based on JAXWS/JAXB, using MTOM to download large binary data. At the service side there is a DataHandler involved which reads the bytes into a byte[] called ImageDepot. (The axis2 MTOM example). The problem is now that the size of this byte array might have become too small. At least I get that impression at the client side, which is a WCF client using a proxy generated by the svcutil asking for the WSDL data from the running SpezplaFS (File Service).

In short: all works fine until at the client the byte[] array becomes too small to keep more 2^31 bytes of data.

In short: How can I use MTOM and stream bytes from huge binary files, files with a size up to long.MaxValue, from a service to a client by using a64K byte buffer?

The client should be able to call this service, the stub, only once without the need to manage junks.

At the client I would see something like

BufferdInputStream bis = new BufferedInputStream(new ... what ever stream would talk with  my MTOM streaming service ...);
BufferedOutputStream bos = new BufferedOutputStream(new FileOutputStream(datafile));
Byte[] buffer;

While (! bis.wead(buffer, 0, Size)== 0) {
    bos.write(buffer, 0, Size);
}
If (!bis==null) bis.Close();
If (!bos==null) bos.Close();

And pointers to a working example are very welcome
Thank you
Josef


CONFIDENTIALITY NOTICE This message and any included attachments are from Cerner Corporation and are intended only for the addressee. The information contained in this message is confidential and may constitute inside or non-public information under international, federal, or state securities laws. Unauthorized forwarding, printing, copying, distribution, or use of such information is strictly prohibited and may be unlawful. If you are not the addressee, please promptly delete this message and notify the sender of the delivery error by e-mail or you may call Cerner's corporate offices in Kansas City, Missouri, U.S.A at (+1) (816)221-1024.

RE: Downloading huge binary files using MTOM

["Okken,Brett" <BO...@CERNER.COM>]

This should be possible by building a custom javax.activation.DataSource implementation which is capable of generating the InputStream on demand (and potentially multiple times).

You may run into problems with some extensions (such as rampart for JAX-WS) that may require converting to a DOM representation, which forces the binary content into the xml itself.

Brett Okken | CAMM Platform Services | Knowledge Architect | 816.201.6112 | www.cerner.com<http://www.cerner.com> | bokken@cerner.com<ma...@cerner.com>

From: Stadelmann Josef [mailto:josef.stadelmann@axa-winterthur.ch]
Sent: Thursday, April 23, 2015 5:58 AM
To: axis-user@ws.apache.org
Subject: Downloading huge binary files using MTOM

Hi,

I have successfully developed and we use it since years, a Binary File Service, based on JAXWS/JAXB, using MTOM to download large binary data. At the service side there is a DataHandler involved which reads the bytes into a byte[] called ImageDepot. (The axis2 MTOM example). The problem is now that the size of this byte array might have become too small. At least I get that impression at the client side, which is a WCF client using a proxy generated by the svcutil asking for the WSDL data from the running SpezplaFS (File Service).

In short: all works fine until at the client the byte[] array becomes too small to keep more 2^31 bytes of data.

In short: How can I use MTOM and stream bytes from huge binary files, files with a size up to long.MaxValue, from a service to a client by using a64K byte buffer?

The client should be able to call this service, the stub, only once without the need to manage junks.

At the client I would see something like

BufferdInputStream bis = new BufferedInputStream(new ... what ever stream would talk with  my MTOM streaming service ...);
BufferedOutputStream bos = new BufferedOutputStream(new FileOutputStream(datafile));
Byte[] buffer;

While (! bis.wead(buffer, 0, Size)== 0) {
    bos.write(buffer, 0, Size);
}
If (!bis==null) bis.Close();
If (!bos==null) bos.Close();

And pointers to a working example are very welcome
Thank you
Josef


CONFIDENTIALITY NOTICE This message and any included attachments are from Cerner Corporation and are intended only for the addressee. The information contained in this message is confidential and may constitute inside or non-public information under international, federal, or state securities laws. Unauthorized forwarding, printing, copying, distribution, or use of such information is strictly prohibited and may be unlawful. If you are not the addressee, please promptly delete this message and notify the sender of the delivery error by e-mail or you may call Cerner's corporate offices in Kansas City, Missouri, U.S.A at (+1) (816)221-1024.