You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "Tim Armstrong (Jira)" <ji...@apache.org> on 2020/03/18 19:01:00 UTC
[jira] [Commented] (IMPALA-3766) LZ4 compress spill data before
writing it do disk
[ https://issues.apache.org/jira/browse/IMPALA-3766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17061997#comment-17061997 ]
Tim Armstrong commented on IMPALA-3766:
---------------------------------------
I'm revisiting this with the aim of reducing scratch space requirements.
> LZ4 compress spill data before writing it do disk
> -------------------------------------------------
>
> Key: IMPALA-3766
> URL: https://issues.apache.org/jira/browse/IMPALA-3766
> Project: IMPALA
> Issue Type: New Feature
> Components: Backend
> Affects Versions: Impala 2.7.0
> Reporter: Mostafa Mokhtar
> Assignee: Tim Armstrong
> Priority: Minor
> Labels: performance
>
> Evaluate compressing the buffers before writing them to disk for spilling operators.
> Applying LZ4 on row batches before sending them over the network as part of exchange provides around 2x compression.
> {code}
> - BytesSent: 612.87 MB (642635712)
> - NetworkThroughput(*): 1.88 GB/sec
> - OverallThroughput: 1.21 GB/sec
> - PeakMemoryUsage: 51.00 KB (52224)
> - RowsReturned: 360.00K (360000)
> - SerializeBatchTime: 176.002ms
> - TransmitDataRPCTime: 319.005ms
> - UncompressedRowBatchSize: 1.47 GB (1573356320)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org