[jira] [Assigned] (IMPALA-15512) Increase value of data_stream_sender_buffer_size

["Zoltán Borók-Nagy (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/IMPALA-15512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Zoltán Borók-Nagy reassigned IMPALA-15512:
------------------------------------------

    Assignee:     (was: Zoltán Borók-Nagy)

> Increase value of data_stream_sender_buffer_size
> ------------------------------------------------
>
>                 Key: IMPALA-15512
>                 URL: https://issues.apache.org/jira/browse/IMPALA-15512
>             Project: IMPALA
>          Issue Type: Improvement
>          Components: Backend
>            Reporter: Zoltán Borók-Nagy
>            Priority: Major
>              Labels: performance
>
> The current value (16KiB) of data_stream_sender_buffer_size originates from a commit {*}14 years ago{*}:
> [https://github.com/apache/impala/blame/bc08241ffbac0eec70b12ffc36929c9dbc97550c/be/src/exec/data-sink.cc#L29]
> Today's hardware has significantly more RAM so it makes sense to re-evaluate a proper value for it.
> Impala 5 also provides a good opportunity to increase its value.
> Increasing its value can significantly improve query performance, even 25% of performance gain can be achieved on TPC-DS queries. (measured on a 5 node cluster with
> --data_stream_sender_buffer_size=128K, TPC-DS at scale 100). Setting it to 128K might be a bold move, but at least 32K and 64K should be reasonable.
> h3. Opus 5.5: More detail on the 25%, with two caveats.
> *Setup:* 5 nodes, AWS m6id.4xlarge (16 vCPUs, 64 GB), TPC-DS SF100 Parquet, 104 queries of tpcds-unmodified run serially, sum of best-of-2. Master 1f34ea915e, MT_DOP=0.
> ||data_stream_sender_buffer_size||Total||
> |16K (default)|383.6 s|
> |64K|304.2 s (-21%)|
> |128K|286.9 s (-25%)|
> *The gain comes from round trips, not bandwidth.*
> * Hash exchanges send 10.2M TransmitData RPCs at 16K, 2.52M at 64K and 1.26M at 128K.
> * Network time per RPC stays at 390-430 us while the payload grows 8x.
> * A channel allows one RPC in flight, so a sender's throughput is one batch per round trip (see IMPALA-15519, in-flight RPCs).
> * Batches are about 2x the flag value (5x for rows with wide strings), because CalculateRowBatchCapacity() ignores var-len data (krpc-data-stream-sender.cc:392).
> *Caveat 1: network topology.*
> * This cluster has no placement group. 2 of the 5 hosts are 0.5-0.7 ms away from the other 3, against 0.1-0.15 ms within a group.
> * About 70% of the 64K gain comes from q72 and q78, whose large exchanges cross those links. The median query is unchanged.
> * Total CPU time falls 1-2% at 64K and 5% at 128K, from fewer RPCs and less serialization.
> * On a low-latency network the wall-clock gain will be smaller.
> *Caveat 2: memory at MT_DOP>0.*
> * At MT_DOP=0 the largest per-host sum of sender peaks is 42 / 74 / 111 MB at 16K / 64K / 128K (q14_1).
> * At MT_DOP=8 the channel count per sender is 8x higher. Modelled per-host sender memory for q67a goes from 306 MB at 16K to 2.8 GB expected (5.2 GB worst case) at a flat 128K. q14_1's worst case is 12.5 GB.
> * At MT_DOP=8 the gain is also much smaller: 128K + 2 in-flight RPCs + no LZ4 together give only -5% (187.0 -> 176.9 s).
> * The planner's sender estimate does not follow the flag: DataStreamSink assumes 1024 rows x row size. At 128K, 400 of 674 hash senders exceeded the estimate, by up to 8.5 MB per instance.
> *Suggestion.* A flat 64K is the low-risk default. For more, size the buffer per channel from a per-sender budget:
> * Target = clamp(budget / (2 x channels), 16K, 256K), flushed by bytes including var-len data, and never smaller than today.
> * A prototype with a 4 MB budget gave 286.1 s at MT_DOP=0, the same as a flat 128K.
> * At MT_DOP=8 it keeps sender memory close to today's: q67a about 657 MB per host expected, against 2.76 GB for a flat 128K.
> * The planner estimate in DataStreamSink must use the same rule.
> * A 1-2 MB budget is more conservative and still reaches the 64K level at MT_DOP=0.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org