You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@arrow.apache.org by "Amol Umbarkar (Jira)" <ji...@apache.org> on 2020/05/18 09:21:00 UTC

[jira] [Created] (ARROW-8845) Selective compression on the wire

Amol Umbarkar created ARROW-8845:
------------------------------------

             Summary: Selective compression on the wire
                 Key: ARROW-8845
                 URL: https://issues.apache.org/jira/browse/ARROW-8845
             Project: Apache Arrow
          Issue Type: Improvement
          Components: FlightRPC
            Reporter: Amol Umbarkar


Dask seems to be selectively do compression if it is found to be useful. They sort of pick 10kb of sample upfront to calculate compression and if the results are good then the whole batch is compressed. This seems to save de-compression effort on receiver side.
 
Please take a look at [https://blog.dask.org/2016/04/14/dask-distributed-optimizing-protocol#problem-3-unwanted-compression]
 
Thought this could be relevant to arrow batch transfers as well. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)