You are viewing a plain text version of this content. The canonical link for it is here.
Posted to soap-user@ws.apache.org by Nige White <ni...@forward-comp.co.uk> on 2004/11/12 17:11:18 UTC
Compressing SOAP requests
I've specified setGzip(true), and the incoming data is encoded.
I'm having trouble uncompressing it using the zlib library from
http://www.gzip.org/zlib/ (I can call C routines from this "DBL" language)
I can compress and decompress data in DBL using the zlib inflateInit(),
inflate(), and inflateEnd() routines, and I get back the data I started
with, so the zlib routines I have are consistent with themselves. The
good news is that they produce a 3:1 compression ratio on my test string!
Unfortunately, they won't uncompress the data I'm receiving from Apache
SOAP. Although if I write the data I receive out to a .gz file, then
gunzip will see it just fine.
So my question is, why won't the zlib routines from http://www.gzip.org/
work? Is there an older version with a different compression algorithm I
should use? Should I do anything to the data before sending it into the
inflate() routine?
Hope you can help!
Nigel
_____________________________________________________________________
This message has been checked for all known viruses. Virus scanning
powered by Messagelabs http://www.messagelabs.com For more information
e-mail : hostmaster@forward-comp.co.uk
Re: Compressing SOAP requests
Posted by Scott Nichol <sn...@scottnichol.com>.
Your compressed response should have HTTP headers
Content-Type: text/xml
Content-Encoding: gzip
Scott Nichol
Do not send e-mail directly to this e-mail address,
because it is filtered to accept only mail from
specific mail lists.
----- Original Message -----
From: "Nige White" <ni...@forward-comp.co.uk>
To: <so...@ws.apache.org>
Sent: Monday, November 15, 2004 1:22 PM
Subject: Re: Compressing SOAP requests
> Scott Nichol wrote:
>
> >FYI, deflate follows Zlib as described in RFC 1950 (http://www.ietf.org/rfc/rfc1950.txt?number=1950). The Zlib DEFLATE method of compression is described in RFC 1951, while gzip is described in RFC 1952 (http://www.ietf.org/rfc/rfc1952.txt?number=1952). RFC 1952 looks like it specifies 10 bytes for the header, so I am not sure why 8 bytes works....
> >
> >
> >
> Well, I've been battling with it.
>
> Why 8 bytes works...
>
> zlib already has a 2 byte flag prefix which the gzip compressed data
> (after its 10 byte header) does not use, so it *looks* like it works
> appending the 8 bytes.
>
> Here's what I do:
>
> When recieving gzip data:
>
> Remove the first 10 bytes, they are the gzip-specific 10 byte header
> Remove the last 8 bytes, they consist of a CRC-32 checksum of the
> original data, and original data length
> Prepend the zlib 2 byte flag: 0x78 0x9c
> Call zlib-inflate, and ignore error -5. THis is probably because the
> buffer ends abruptly with no Adler checksum
>
> When sending data as gzip:
>
> Call zlib-deflate
> Remove the 2 zlib flag prefix bytes.
> Prepend the 10 byte gzip-specific header
> Append a CRC-32 checksum of the original data (Calculated in "DBL" which
> has no unsigned arithmetic, and no bit-shift operators!)
> Append the original data length
>
> This works in that I can compress data, and output it to a file which
> gunzip will happily inflate to the original data.
>
> I can also see (in my logfile) the correctly inflated SOAP packets
> coming in to my DBL server.
>
> There does seem to be a problem when SOAP receives by compressed packet.
> It is throwing "Content is not allowed in prolog."
>
> I still set the Content-Type header to "text/html". Should I change this
> to indicate to SOAP that it is gzip data coming back?
>
> I'll be back tomorrow, home time now.
>
> Cheers,
>
> Nigel
>
> _____________________________________________________________________
> This message has been checked for all known viruses. Virus scanning
> powered by Messagelabs http://www.messagelabs.com For more information
> e-mail : hostmaster@forward-comp.co.uk
>
>
>
Re: Compressing SOAP requests
Posted by Nige White <ni...@forward-comp.co.uk>.
Scott Nichol wrote:
>FYI, deflate follows Zlib as described in RFC 1950 (http://www.ietf.org/rfc/rfc1950.txt?number=1950). The Zlib DEFLATE method of compression is described in RFC 1951, while gzip is described in RFC 1952 (http://www.ietf.org/rfc/rfc1952.txt?number=1952). RFC 1952 looks like it specifies 10 bytes for the header, so I am not sure why 8 bytes works....
>
>
>
Well, I've been battling with it.
Why 8 bytes works...
zlib already has a 2 byte flag prefix which the gzip compressed data
(after its 10 byte header) does not use, so it *looks* like it works
appending the 8 bytes.
Here's what I do:
When recieving gzip data:
Remove the first 10 bytes, they are the gzip-specific 10 byte header
Remove the last 8 bytes, they consist of a CRC-32 checksum of the
original data, and original data length
Prepend the zlib 2 byte flag: 0x78 0x9c
Call zlib-inflate, and ignore error -5. THis is probably because the
buffer ends abruptly with no Adler checksum
When sending data as gzip:
Call zlib-deflate
Remove the 2 zlib flag prefix bytes.
Prepend the 10 byte gzip-specific header
Append a CRC-32 checksum of the original data (Calculated in "DBL" which
has no unsigned arithmetic, and no bit-shift operators!)
Append the original data length
This works in that I can compress data, and output it to a file which
gunzip will happily inflate to the original data.
I can also see (in my logfile) the correctly inflated SOAP packets
coming in to my DBL server.
There does seem to be a problem when SOAP receives by compressed packet.
It is throwing "Content is not allowed in prolog."
I still set the Content-Type header to "text/html". Should I change this
to indicate to SOAP that it is gzip data coming back?
I'll be back tomorrow, home time now.
Cheers,
Nigel
_____________________________________________________________________
This message has been checked for all known viruses. Virus scanning
powered by Messagelabs http://www.messagelabs.com For more information
e-mail : hostmaster@forward-comp.co.uk
Re: Compressing SOAP requests
Posted by Scott Nichol <sn...@scottnichol.com>.
FYI, deflate follows Zlib as described in RFC 1950 (http://www.ietf.org/rfc/rfc1950.txt?number=1950). The Zlib DEFLATE method of compression is described in RFC 1951, while gzip is described in RFC 1952 (http://www.ietf.org/rfc/rfc1952.txt?number=1952). RFC 1952 looks like it specifies 10 bytes for the header, so I am not sure why 8 bytes works....
Scott Nichol
Do not send e-mail directly to this e-mail address,
because it is filtered to accept only mail from
specific mail lists.
----- Original Message -----
From: "Scott Nichol" <sn...@scottnichol.com>
To: <so...@ws.apache.org>
Sent: Friday, November 12, 2004 1:08 PM
Subject: Re: Compressing SOAP requests
Try it without any CRC: I think gzip may be willing to decompress it, anyway.
Scott Nichol
Do not send e-mail directly to this e-mail address,
because it is filtered to accept only mail from
specific mail lists.
----- Original Message -----
From: "Nige White" <ni...@forward-comp.co.uk>
To: <so...@ws.apache.org>
Sent: Friday, November 12, 2004 12:48 PM
Subject: Re: Compressing SOAP requests
> Scott Nichol wrote:
>
> >The magic header is 8 bytes: "\x1f\x8b\x08\x00\x00\x00\x00\x00".
> >
> >
> >
> Interesting. I've read elsewhere that the header is 10 bytes, but...
>
> I've tried writing those 8 bytes followed by the zlib-deflated data,
> followed by 32 bits of zero.
>
> Now, where gunzip -t used to say "invalid compressed data--format violated"
>
> it says "invalid compressed data--crc error"
>
> Any idea how to calculate the CRC?
>
> _____________________________________________________________________
> This message has been checked for all known viruses. Virus scanning
> powered by Messagelabs http://www.messagelabs.com For more information
> e-mail : hostmaster@forward-comp.co.uk
>
>
>
Re: Compressing SOAP requests
Posted by Scott Nichol <sn...@scottnichol.com>.
Try it without any CRC: I think gzip may be willing to decompress it, anyway.
Scott Nichol
Do not send e-mail directly to this e-mail address,
because it is filtered to accept only mail from
specific mail lists.
----- Original Message -----
From: "Nige White" <ni...@forward-comp.co.uk>
To: <so...@ws.apache.org>
Sent: Friday, November 12, 2004 12:48 PM
Subject: Re: Compressing SOAP requests
> Scott Nichol wrote:
>
> >The magic header is 8 bytes: "\x1f\x8b\x08\x00\x00\x00\x00\x00".
> >
> >
> >
> Interesting. I've read elsewhere that the header is 10 bytes, but...
>
> I've tried writing those 8 bytes followed by the zlib-deflated data,
> followed by 32 bits of zero.
>
> Now, where gunzip -t used to say "invalid compressed data--format violated"
>
> it says "invalid compressed data--crc error"
>
> Any idea how to calculate the CRC?
>
> _____________________________________________________________________
> This message has been checked for all known viruses. Virus scanning
> powered by Messagelabs http://www.messagelabs.com For more information
> e-mail : hostmaster@forward-comp.co.uk
>
>
>
Re: Compressing SOAP requests
Posted by Nige White <ni...@forward-comp.co.uk>.
Scott Nichol wrote:
>The magic header is 8 bytes: "\x1f\x8b\x08\x00\x00\x00\x00\x00".
>
>
>
Interesting. I've read elsewhere that the header is 10 bytes, but...
I've tried writing those 8 bytes followed by the zlib-deflated data,
followed by 32 bits of zero.
Now, where gunzip -t used to say "invalid compressed data--format violated"
it says "invalid compressed data--crc error"
Any idea how to calculate the CRC?
_____________________________________________________________________
This message has been checked for all known viruses. Virus scanning
powered by Messagelabs http://www.messagelabs.com For more information
e-mail : hostmaster@forward-comp.co.uk
Re: Compressing SOAP requests
Posted by Scott Nichol <sn...@scottnichol.com>.
The magic header is 8 bytes: "\x1f\x8b\x08\x00\x00\x00\x00\x00".
Scott Nichol
Do not send e-mail directly to this e-mail address,
because it is filtered to accept only mail from
specific mail lists.
----- Original Message -----
From: "Scott Nichol" <sn...@scottnichol.com>
To: <so...@ws.apache.org>
Sent: Friday, November 12, 2004 11:44 AM
Subject: Re: Compressing SOAP requests
Standard Zlib is equivalent to HTTP Content-Encoding header value 'deflate'. This differs from 'gzip', in that gzip includes some extra bytes (a header and checksum) in the compressed data. Apache SOAP does not support deflate right now. My experience working with PHP Zlib is that many (most?) gzip are satisfied with 4 0x00 bytes as the checksum, and some constant string of (I think) 10 bytes at the start for the header. I don't recall the values, but you might poke around php.net for the gzinflate/gzdeflate/gzcompress/gzuncompress functions for clues (they would be in the notes after the "formal" documentation), or check the IEFT RFCs.
In short, I *think* Zlib can inflate a gzip payload if the first ten (I think) bytes of the payload are stripped, as well as the last 4. You can package up a Zlib deflate buffer as a gzip payload if you prepend the magic 10 bytes and stick an all-zero dummy checksum at the end.
Scott Nichol
Do not send e-mail directly to this e-mail address,
because it is filtered to accept only mail from
specific mail lists.
----- Original Message -----
From: "Nige White" <ni...@forward-comp.co.uk>
To: <so...@ws.apache.org>
Sent: Friday, November 12, 2004 11:11 AM
Subject: Compressing SOAP requests
> I've specified setGzip(true), and the incoming data is encoded.
>
> I'm having trouble uncompressing it using the zlib library from
> http://www.gzip.org/zlib/ (I can call C routines from this "DBL" language)
>
> I can compress and decompress data in DBL using the zlib inflateInit(),
> inflate(), and inflateEnd() routines, and I get back the data I started
> with, so the zlib routines I have are consistent with themselves. The
> good news is that they produce a 3:1 compression ratio on my test string!
>
> Unfortunately, they won't uncompress the data I'm receiving from Apache
> SOAP. Although if I write the data I receive out to a .gz file, then
> gunzip will see it just fine.
>
> So my question is, why won't the zlib routines from http://www.gzip.org/
> work? Is there an older version with a different compression algorithm I
> should use? Should I do anything to the data before sending it into the
> inflate() routine?
>
> Hope you can help!
>
> Nigel
>
> _____________________________________________________________________
> This message has been checked for all known viruses. Virus scanning
> powered by Messagelabs http://www.messagelabs.com For more information
> e-mail : hostmaster@forward-comp.co.uk
>
>
>
Re: Compressing SOAP requests
Posted by Scott Nichol <sn...@scottnichol.com>.
Standard Zlib is equivalent to HTTP Content-Encoding header value 'deflate'. This differs from 'gzip', in that gzip includes some extra bytes (a header and checksum) in the compressed data. Apache SOAP does not support deflate right now. My experience working with PHP Zlib is that many (most?) gzip are satisfied with 4 0x00 bytes as the checksum, and some constant string of (I think) 10 bytes at the start for the header. I don't recall the values, but you might poke around php.net for the gzinflate/gzdeflate/gzcompress/gzuncompress functions for clues (they would be in the notes after the "formal" documentation), or check the IEFT RFCs.
In short, I *think* Zlib can inflate a gzip payload if the first ten (I think) bytes of the payload are stripped, as well as the last 4. You can package up a Zlib deflate buffer as a gzip payload if you prepend the magic 10 bytes and stick an all-zero dummy checksum at the end.
Scott Nichol
Do not send e-mail directly to this e-mail address,
because it is filtered to accept only mail from
specific mail lists.
----- Original Message -----
From: "Nige White" <ni...@forward-comp.co.uk>
To: <so...@ws.apache.org>
Sent: Friday, November 12, 2004 11:11 AM
Subject: Compressing SOAP requests
> I've specified setGzip(true), and the incoming data is encoded.
>
> I'm having trouble uncompressing it using the zlib library from
> http://www.gzip.org/zlib/ (I can call C routines from this "DBL" language)
>
> I can compress and decompress data in DBL using the zlib inflateInit(),
> inflate(), and inflateEnd() routines, and I get back the data I started
> with, so the zlib routines I have are consistent with themselves. The
> good news is that they produce a 3:1 compression ratio on my test string!
>
> Unfortunately, they won't uncompress the data I'm receiving from Apache
> SOAP. Although if I write the data I receive out to a .gz file, then
> gunzip will see it just fine.
>
> So my question is, why won't the zlib routines from http://www.gzip.org/
> work? Is there an older version with a different compression algorithm I
> should use? Should I do anything to the data before sending it into the
> inflate() routine?
>
> Hope you can help!
>
> Nigel
>
> _____________________________________________________________________
> This message has been checked for all known viruses. Virus scanning
> powered by Messagelabs http://www.messagelabs.com For more information
> e-mail : hostmaster@forward-comp.co.uk
>
>
>