You are viewing a plain text version of this content. The canonical link for it is here.

Posted to soap-user@ws.apache.org by Nige White <ni...@forward-comp.co.uk> on 2004/11/10 14:26:46 UTC

Compressing SOAP requests and using SSL

My boss has just looked at the SOAP client/server system I wrote. I 
substituted my generated SOAP java client classes in for the 
Synergy-created java client classes, (I made sure all the method 
signatures were identical) and ran the web app. It talked to my SOAP 
server just fine.

Looking at the network traffic, it's obvious that SOAP is quite 
bandwidth-intensive. The Synergy RPC protocol is primitive but very 
compact, and my boss has basically vetoed the move to SOAP because of 
the bandwidth constraints (We host our customers' web apps in-house, and 
those web apps use RPC back to the customer's servers to access their 
data!!! Weird... don't ask - it's the way "they" want it)

My question is, is it possible to tell Apache SOAP to compress the HTTP 
request? With it being textual data, we could achieve quite good 
compression rates.

Also, can Apache SOAP use SSL? Can I just specify an "https://..." URL 
in my Call.invoke()?

Thanks for any help.

Nige

_____________________________________________________________________
This message has been checked for all known viruses. Virus scanning
powered by Messagelabs http://www.messagelabs.com For more information
e-mail : hostmaster@forward-comp.co.uk

Re: Compressing SOAP requests and using SSL

Posted by Nige White <ni...@forward-comp.co.uk>.

Scott Nichol wrote:

>The "nightly" build can gzip and will un-gzip a gzipped response.  The most recent one was posted 6/22/2004 at http://cvs.apache.org/dist/soap/nightly/2004-06-22/.
>
>To use SSL from the client, your only code change is to use the new endpoint URL (with https).  However, as has been pointed out, you must also have your JVM configuration set up for SSL, with cert authorities, keystore, etc.
>
>Scott Nichol
>
>  
>

Thanks for the helpful responses! I've downloaded the latest nightly 
build, and yes, it has setGzip(). Fantastic. Bandwidth is my first 
hurdle with the bosses here, in my quest to drag them out of the 1970s.

I've added the encodingStyle, I'm just about to test it. No doubt you'll 
hear from me if I can't get it to work.

Regards,

Nigel

_____________________________________________________________________
This message has been checked for all known viruses. Virus scanning
powered by Messagelabs http://www.messagelabs.com For more information
e-mail : hostmaster@forward-comp.co.uk

Re: Compressing SOAP requests

Posted by Scott Nichol <sn...@scottnichol.com>.

Your compressed response should have HTTP headers

Content-Type: text/xml
Content-Encoding: gzip

Scott Nichol

Do not send e-mail directly to this e-mail address,
because it is filtered to accept only mail from
specific mail lists.
----- Original Message ----- 
From: "Nige White" <ni...@forward-comp.co.uk>
To: <so...@ws.apache.org>
Sent: Monday, November 15, 2004 1:22 PM
Subject: Re: Compressing SOAP requests


> Scott Nichol wrote:
> 
> >FYI, deflate follows Zlib as described in RFC 1950 (http://www.ietf.org/rfc/rfc1950.txt?number=1950).  The Zlib DEFLATE method of compression is described in RFC 1951, while gzip is described in RFC 1952 (http://www.ietf.org/rfc/rfc1952.txt?number=1952).  RFC 1952 looks like it specifies 10 bytes for the header, so I am not sure why 8 bytes works....
> >
> >  
> >
> Well, I've been battling with it.
> 
> Why 8 bytes works...
> 
> zlib already has a 2 byte flag prefix which the gzip compressed data 
> (after its 10 byte header) does not use, so it *looks* like it works 
> appending the 8 bytes.
> 
> Here's what I do:
> 
> When recieving gzip data:
> 
> Remove the first 10 bytes, they are the gzip-specific 10 byte header
> Remove the last 8 bytes, they consist of a CRC-32 checksum of the 
> original data, and original data length
> Prepend the zlib 2 byte flag: 0x78 0x9c
> Call zlib-inflate, and ignore error -5. THis is probably because the 
> buffer ends abruptly with no Adler checksum
> 
> When sending data as gzip:
> 
> Call zlib-deflate
> Remove the 2 zlib flag prefix bytes.
> Prepend the 10 byte gzip-specific header
> Append a CRC-32 checksum of the original data (Calculated in "DBL" which 
> has no unsigned arithmetic, and no bit-shift operators!)
> Append the original data length
> 
> This works in that I can compress data, and output it to a file which 
> gunzip will happily inflate to the original data.
> 
> I can also see (in my logfile) the correctly inflated SOAP packets 
> coming in to my DBL server.
> 
> There does seem to be a problem when SOAP receives by compressed packet. 
> It is throwing "Content is not allowed in prolog."
> 
> I still set the Content-Type header to "text/html". Should I change this 
> to indicate to SOAP that it is gzip data coming back?
> 
> I'll be back tomorrow, home time now.
> 
> Cheers,
> 
> Nigel
> 
> _____________________________________________________________________
> This message has been checked for all known viruses. Virus scanning
> powered by Messagelabs http://www.messagelabs.com For more information
> e-mail : hostmaster@forward-comp.co.uk
> 
> 
>

Re: Compressing SOAP requests

Posted by Nige White <ni...@forward-comp.co.uk>.

Scott Nichol wrote:

>FYI, deflate follows Zlib as described in RFC 1950 (http://www.ietf.org/rfc/rfc1950.txt?number=1950). The Zlib DEFLATE method of compression is described in RFC 1951, while gzip is described in RFC 1952 (http://www.ietf.org/rfc/rfc1952.txt?number=1952). RFC 1952 looks like it specifies 10 bytes for the header, so I am not sure why 8 bytes works....
>
>
>
Well, I've been battling with it.

Why 8 bytes works...

zlib already has a 2 byte flag prefix which the gzip compressed data
(after its 10 byte header) does not use, so it *looks* like it works
appending the 8 bytes.

Here's what I do:

When recieving gzip data:

Remove the first 10 bytes, they are the gzip-specific 10 byte header
Remove the last 8 bytes, they consist of a CRC-32 checksum of the
original data, and original data length
Prepend the zlib 2 byte flag: 0x78 0x9c
Call zlib-inflate, and ignore error -5. THis is probably because the
buffer ends abruptly with no Adler checksum

When sending data as gzip:

Call zlib-deflate
Remove the 2 zlib flag prefix bytes.
Prepend the 10 byte gzip-specific header
Append a CRC-32 checksum of the original data (Calculated in "DBL" which
has no unsigned arithmetic, and no bit-shift operators!)
Append the original data length

This works in that I can compress data, and output it to a file which
gunzip will happily inflate to the original data.

I can also see (in my logfile) the correctly inflated SOAP packets
coming in to my DBL server.

There does seem to be a problem when SOAP receives by compressed packet.
It is throwing "Content is not allowed in prolog."

I still set the Content-Type header to "text/html". Should I change this
to indicate to SOAP that it is gzip data coming back?

I'll be back tomorrow, home time now.

Cheers,

Nigel

_____________________________________________________________________
This message has been checked for all known viruses. Virus scanning
powered by Messagelabs http://www.messagelabs.com For more information
e-mail : hostmaster@forward-comp.co.uk

Re: Compressing SOAP requests

Posted by Scott Nichol <sn...@scottnichol.com>.

FYI, deflate follows Zlib as described in RFC 1950 (http://www.ietf.org/rfc/rfc1950.txt?number=1950).  The Zlib DEFLATE method of compression is described in RFC 1951, while gzip is described in RFC 1952 (http://www.ietf.org/rfc/rfc1952.txt?number=1952).  RFC 1952 looks like it specifies 10 bytes for the header, so I am not sure why 8 bytes works....

Scott Nichol

Do not send e-mail directly to this e-mail address,
because it is filtered to accept only mail from
specific mail lists.
----- Original Message ----- 
From: "Scott Nichol" <sn...@scottnichol.com>
To: <so...@ws.apache.org>
Sent: Friday, November 12, 2004 1:08 PM
Subject: Re: Compressing SOAP requests


Try it without any CRC: I think gzip may be willing to decompress it, anyway.

Scott Nichol

Do not send e-mail directly to this e-mail address,
because it is filtered to accept only mail from
specific mail lists.
----- Original Message ----- 
From: "Nige White" <ni...@forward-comp.co.uk>
To: <so...@ws.apache.org>
Sent: Friday, November 12, 2004 12:48 PM
Subject: Re: Compressing SOAP requests


> Scott Nichol wrote:
> 
> >The magic header is 8 bytes: "\x1f\x8b\x08\x00\x00\x00\x00\x00".
> >
> >  
> >
> Interesting. I've read elsewhere that the header is 10 bytes, but...
> 
> I've tried writing those 8 bytes followed by the zlib-deflated data, 
> followed by 32 bits of zero.
> 
> Now, where gunzip -t  used to say "invalid compressed data--format violated"
> 
> it says "invalid compressed data--crc error"
> 
> Any idea how to calculate the CRC?
> 
> _____________________________________________________________________
> This message has been checked for all known viruses. Virus scanning
> powered by Messagelabs http://www.messagelabs.com For more information
> e-mail : hostmaster@forward-comp.co.uk
> 
> 
>

Re: Compressing SOAP requests

Posted by Scott Nichol <sn...@scottnichol.com>.

Try it without any CRC: I think gzip may be willing to decompress it, anyway.

Scott Nichol

Do not send e-mail directly to this e-mail address,
because it is filtered to accept only mail from
specific mail lists.
----- Original Message ----- 
From: "Nige White" <ni...@forward-comp.co.uk>
To: <so...@ws.apache.org>
Sent: Friday, November 12, 2004 12:48 PM
Subject: Re: Compressing SOAP requests


> Scott Nichol wrote:
> 
> >The magic header is 8 bytes: "\x1f\x8b\x08\x00\x00\x00\x00\x00".
> >
> >  
> >
> Interesting. I've read elsewhere that the header is 10 bytes, but...
> 
> I've tried writing those 8 bytes followed by the zlib-deflated data, 
> followed by 32 bits of zero.
> 
> Now, where gunzip -t  used to say "invalid compressed data--format violated"
> 
> it says "invalid compressed data--crc error"
> 
> Any idea how to calculate the CRC?
> 
> _____________________________________________________________________
> This message has been checked for all known viruses. Virus scanning
> powered by Messagelabs http://www.messagelabs.com For more information
> e-mail : hostmaster@forward-comp.co.uk
> 
> 
>

Re: Compressing SOAP requests

Posted by Nige White <ni...@forward-comp.co.uk>.

Scott Nichol wrote:

>The magic header is 8 bytes: "\x1f\x8b\x08\x00\x00\x00\x00\x00".
>
>  
>
Interesting. I've read elsewhere that the header is 10 bytes, but...

I've tried writing those 8 bytes followed by the zlib-deflated data, 
followed by 32 bits of zero.

Now, where gunzip -t  used to say "invalid compressed data--format violated"

it says "invalid compressed data--crc error"

Any idea how to calculate the CRC?

_____________________________________________________________________
This message has been checked for all known viruses. Virus scanning
powered by Messagelabs http://www.messagelabs.com For more information
e-mail : hostmaster@forward-comp.co.uk

Re: Compressing SOAP requests

Posted by Scott Nichol <sn...@scottnichol.com>.

The magic header is 8 bytes: "\x1f\x8b\x08\x00\x00\x00\x00\x00".

Scott Nichol

Do not send e-mail directly to this e-mail address,
because it is filtered to accept only mail from
specific mail lists.
----- Original Message ----- 
From: "Scott Nichol" <sn...@scottnichol.com>
To: <so...@ws.apache.org>
Sent: Friday, November 12, 2004 11:44 AM
Subject: Re: Compressing SOAP requests

Standard Zlib is equivalent to HTTP Content-Encoding header value 'deflate'.  This differs from 'gzip', in that gzip includes some extra bytes (a header and checksum) in the compressed data.  Apache SOAP does not support deflate right now.  My experience working with PHP Zlib is that many (most?) gzip are satisfied with 4 0x00 bytes as the checksum, and some constant string of (I think) 10 bytes at the start for the header.  I don't recall the values, but you might poke around php.net for the gzinflate/gzdeflate/gzcompress/gzuncompress functions for clues (they would be in the notes after the "formal" documentation), or check the IEFT RFCs.

In short, I *think* Zlib can inflate a gzip payload if the first ten (I think) bytes of the payload are stripped, as well as the last 4.  You can package up a Zlib deflate buffer as a gzip payload if you prepend the magic 10 bytes and stick an all-zero dummy checksum at the end.

Scott Nichol

Do not send e-mail directly to this e-mail address,
because it is filtered to accept only mail from
specific mail lists.
----- Original Message ----- 
From: "Nige White" <ni...@forward-comp.co.uk>
To: <so...@ws.apache.org>
Sent: Friday, November 12, 2004 11:11 AM
Subject: Compressing SOAP requests

> I've specified setGzip(true), and the incoming data is encoded.
> 
> I'm having trouble uncompressing it using the zlib library from 
> http://www.gzip.org/zlib/ (I can call C routines from this "DBL" language)
> 
> I can compress and decompress data in DBL using the zlib inflateInit(), 
> inflate(), and inflateEnd() routines, and I get back the data I started 
> with, so the zlib routines I have are consistent with themselves. The 
> good news is that they produce a 3:1 compression ratio on my test string!
> 
> Unfortunately, they won't uncompress the data I'm receiving from Apache 
> SOAP. Although if I write the data I receive out to a .gz file, then 
> gunzip will see it just fine.
> 
> So my question is, why won't the zlib routines from http://www.gzip.org/ 
> work? Is there an older version with a different compression algorithm I 
> should use? Should I do anything to the data before sending it into the 
> inflate() routine?
> 
> Hope you can help!
> 
> Nigel
> 
> _____________________________________________________________________
> This message has been checked for all known viruses. Virus scanning
> powered by Messagelabs http://www.messagelabs.com For more information
> e-mail : hostmaster@forward-comp.co.uk
> 
> 
>

Re: Compressing SOAP requests

Posted by Scott Nichol <sn...@scottnichol.com>.

Standard Zlib is equivalent to HTTP Content-Encoding header value 'deflate'.  This differs from 'gzip', in that gzip includes some extra bytes (a header and checksum) in the compressed data.  Apache SOAP does not support deflate right now.  My experience working with PHP Zlib is that many (most?) gzip are satisfied with 4 0x00 bytes as the checksum, and some constant string of (I think) 10 bytes at the start for the header.  I don't recall the values, but you might poke around php.net for the gzinflate/gzdeflate/gzcompress/gzuncompress functions for clues (they would be in the notes after the "formal" documentation), or check the IEFT RFCs.

In short, I *think* Zlib can inflate a gzip payload if the first ten (I think) bytes of the payload are stripped, as well as the last 4.  You can package up a Zlib deflate buffer as a gzip payload if you prepend the magic 10 bytes and stick an all-zero dummy checksum at the end.

Scott Nichol

Do not send e-mail directly to this e-mail address,
because it is filtered to accept only mail from
specific mail lists.
----- Original Message ----- 
From: "Nige White" <ni...@forward-comp.co.uk>
To: <so...@ws.apache.org>
Sent: Friday, November 12, 2004 11:11 AM
Subject: Compressing SOAP requests


> I've specified setGzip(true), and the incoming data is encoded.
> 
> I'm having trouble uncompressing it using the zlib library from 
> http://www.gzip.org/zlib/ (I can call C routines from this "DBL" language)
> 
> I can compress and decompress data in DBL using the zlib inflateInit(), 
> inflate(), and inflateEnd() routines, and I get back the data I started 
> with, so the zlib routines I have are consistent with themselves. The 
> good news is that they produce a 3:1 compression ratio on my test string!
> 
> Unfortunately, they won't uncompress the data I'm receiving from Apache 
> SOAP. Although if I write the data I receive out to a .gz file, then 
> gunzip will see it just fine.
> 
> So my question is, why won't the zlib routines from http://www.gzip.org/ 
> work? Is there an older version with a different compression algorithm I 
> should use? Should I do anything to the data before sending it into the 
> inflate() routine?
> 
> Hope you can help!
> 
> Nigel
> 
> _____________________________________________________________________
> This message has been checked for all known viruses. Virus scanning
> powered by Messagelabs http://www.messagelabs.com For more information
> e-mail : hostmaster@forward-comp.co.uk
> 
> 
>