You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org> on 2026/10/08 16:20:41 UTC
[Impala-ASF-CR] PROTOTYPE IMPALA-15525: Make exchange row batch compression configurable
Zoltan Borok-Nagy has uploaded this change for review. ( http://gerrit.cloudera.org:8080/25073
Change subject: PROTOTYPE IMPALA-15525: Make exchange row batch compression configurable
......................................................................
PROTOTYPE IMPALA-15525: Make exchange row batch compression configurable
Every row batch sent to a remote receiver is LZ4-compressed on the
sender's fragment thread, and there is no way to turn it off. On a
fast network the achieved ratio (1.5-2.1x on TPC-DS) is not worth the
CPU, and once senders are no longer limited by round trips
(IMPALA-15512, IMPALA-15519), LZ4 becomes their bottleneck.
Adds --exchange_compression=lz4|none|auto (default lz4, as before).
With 'auto', a sender always compresses its first 4 batches, then
keeps compressing while its cumulative ratio is at least
--exchange_compression_min_ratio (2). Otherwise it sends the next
--exchange_compression_skip_batches (64) batches uncompressed and
probes again with one compressed batch. A broadcast batch is decided
once for all receivers. Senders report the skipped and attempted
batches and the bytes before and after compression in new counters.
Testing:
- TPC-DS SF100, 5 executors, MT_DOP=0: 'none' -6.6% total (network
bytes ~1.85x), 'auto' -4.5%. Results matched master, except q56 and
q65, whose LIMIT ties vary between runs. No new tests.
Change-Id: I92c0da10ee436e53ba0178c68cdfea746165ba18
Assisted-by: Claude Opus 5.5 (Claude Code)
---
M be/src/runtime/krpc-data-stream-sender.cc
M be/src/runtime/krpc-data-stream-sender.h
2 files changed, 129 insertions(+), 1 deletion(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/73/25073/1
--
To view, visit http://gerrit.cloudera.org:8080/25073
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I92c0da10ee436e53ba0178c68cdfea746165ba18
Gerrit-Change-Number: 25073
Gerrit-PatchSet: 1
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
[Impala-ASF-CR] PROTOTYPE IMPALA-15525: Make exchange row batch compression configurable
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/25073 )
Change subject: PROTOTYPE IMPALA-15525: Make exchange row batch compression configurable
......................................................................
Patch Set 1:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/24429/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/25073
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I92c0da10ee436e53ba0178c68cdfea746165ba18
Gerrit-Change-Number: 25073
Gerrit-PatchSet: 1
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Thu, 08 Oct 2026 17:14:56 +0000
Gerrit-HasComments: No