[jira] [Resolved] (IMPALA-3766) Optionally compress spilled data before writing it to disk

["Tim Armstrong (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/IMPALA-3766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tim Armstrong resolved IMPALA-3766.
-----------------------------------
    Fix Version/s: Impala 4.0
       Resolution: Fixed

> Optionally compress spilled data before writing it to disk
> ----------------------------------------------------------
>
>                 Key: IMPALA-3766
>                 URL: https://issues.apache.org/jira/browse/IMPALA-3766
>             Project: IMPALA
>          Issue Type: New Feature
>          Components: Backend
>    Affects Versions: Impala 2.7.0
>            Reporter: Mostafa Mokhtar
>            Assignee: Tim Armstrong
>            Priority: Minor
>              Labels: performance
>             Fix For: Impala 4.0
>
>
> Evaluate compressing the buffers before writing them to disk for spilling operators. 
> Applying LZ4 on row batches before sending them over the network as part of exchange provides around 2x compression. 
> {code}
>          - BytesSent: 612.87 MB (642635712)
>          - NetworkThroughput(*): 1.88 GB/sec
>          - OverallThroughput: 1.21 GB/sec
>          - PeakMemoryUsage: 51.00 KB (52224)
>          - RowsReturned: 360.00K (360000)
>          - SerializeBatchTime: 176.002ms
>          - TransmitDataRPCTime: 319.005ms
>          - UncompressedRowBatchSize: 1.47 GB (1573356320)
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org