[jira] [Updated] (THRIFT-1103) TZlibTransport for python, a zlib compressed transport

["Will Pierce (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/THRIFT-1103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Will Pierce updated THRIFT-1103:
--------------------------------

    Attachment: THRIFT-1103.tzlibtransport_for_python_v1.patch

Patch attached.  Adds TZlibTransport.py into ./lib/py/src/transport/ and adds TZlibTransport into the transport/__init__.py module's __all__ list.

I tested this on python 2.4 and 2.7.  The zlib module is present and provides the same API in python 2.4 as 2.7 for our needs.

If the patch for THRIFT-1094 is good and can be commited, then it would make it easier for me to extend the RunClientServer.py/TestServer.py/TestClient.py code to include testing that exercises the TZlibTransport code.  (I did it locally in my copy of thrift-svn/trunk to test this code, but didn't want to submit a patch that requires another patch ( THRIFT-1094 ) which hasn't been approved yet.)



> TZlibTransport for python, a zlib compressed transport
> ------------------------------------------------------
>
>                 Key: THRIFT-1103
>                 URL: https://issues.apache.org/jira/browse/THRIFT-1103
>             Project: Thrift
>          Issue Type: New Feature
>          Components: Python - Library
>            Reporter: Will Pierce
>            Assignee: Will Pierce
>         Attachments: THRIFT-1103.tzlibtransport_for_python_v1.patch
>
>
> New implementation of zlib compressed transport for python.
> The attached patch provides a zlib compressed transport wrapper for python.  It is similar to the TFramedTransport, in that it wraps another transport, implementing the data compression as a transformation layer on top of the underlying transport that it wraps.
> The compression level is configurable in the constructor, from 0 (none) to 9 (best) and defaults to 9 for best compression.  The way this works is that every write() to the transport appends more data to the internal cStringIO write buffer.  When the transport's flush() method is called, the buffered bytes are then passed to a zlib Compressor object and flush()ed with zlib.Z_SYNC_FLUSH.
> Because the thrift API calls the transport's flush() after writeMessageEnd(), this means very small thrift RPC calls don't get compressed well.  This transport works best on thrift protocols where the payload contains strings longer than 10 characters.  As with all data compression, the more redundancy in the uncompressed input, the greater the resulting compression.
> The TZlibTransport class also implements some basic statistics that track the number of raw bytes written and read, versus the decompressed equivalent.  The getCompRatio() method returns a tuple of (readCompressionRatio,writeCompressionRatio) where ratio is computed using: compressed_bytes/uncompressed_bytes.  (So 10 compression is 0.10, meaning smaller numbers are better.)  The getCompSavings() method returns the actual number of (saved_read_bytes,saved_write_bytes) which might be negative when the compression of non-compressible data ends up expanding the data.  So hopefully, anyone who uses this transport will be able to tell whether the compression is saving bandwidth or not.
> I will add the patch in a few minutes.
> I haven't tested this against the C++ TZlibTransport, only against itself.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira