[jira] [Resolved] (IMPALA-12430) Optimize sending rows within the same process

["Michael Smith (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/IMPALA-12430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Michael Smith resolved IMPALA-12430.
------------------------------------
    Fix Version/s: Impala 4.3.0
       Resolution: Fixed

> Optimize sending rows within the same process
> ---------------------------------------------
>
>                 Key: IMPALA-12430
>                 URL: https://issues.apache.org/jira/browse/IMPALA-12430
>             Project: IMPALA
>          Issue Type: Improvement
>          Components: Backend
>            Reporter: Csaba Ringhofer
>            Priority: Major
>              Labels: performance
>             Fix For: Impala 4.3.0
>
>
> Currently sending row batches to exchange nodes always goes through KRPC even if the sender and receiver are within the same process.
> This means that the following work is done without actually being necessary:
> sender:
> 1. serialize RowBatch to a single buffer
> 2. compress the buffer with LZ4
> 3. send the buffer as a sidecar in KRPC
> receiver:
> 4. fetch buffer from KRPC
> 5. decompress the buffer
> 6. convert the buffer to RowBatch
> Ideally a single deep copy from the sender's RowBatch to the destination's RowBatch is enough (this is needed to cleanup the memory referenced in the original RowBatch during send).
> The most expensive part is 2, the compression with LZ4 (decompression is much faster) and can be avoided with minimal changes.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org