You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Joe McDonnell (Code Review)" <ge...@cloudera.org> on 2026/10/20 03:20:53 UTC
[Impala-ASF-CR] WIP IMPALA-15512: Use data stream sender buffer size consistently and raise it to 64KB
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/25141 )
Change subject: WIP IMPALA-15512: Use data_stream_sender_buffer_size consistently and raise it to 64KB
......................................................................
Patch Set 3:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/25141/3/be/src/runtime/krpc-data-stream-sender-ir.cc
File be/src/runtime/krpc-data-stream-sender-ir.cc:
http://gerrit.cloudera.org:8080/#/c/25141/3/be/src/runtime/krpc-data-stream-sender-ir.cc@58
PS3, Line 58: if (UNLIKELY(collector_batch_->GetAppendedSize() >= buffer_size_)) {
This logic is clearly much better than our current logic, so this comment shouldn't hold up this particular change:
One thought here is whether it would be worth it to have a strict approach where it stops if it doesn't fit. It's harder to do that, because we have to handle backing out partial rows if we hit the limit. It would also require special handling of large rows.
If the bound is strict, it does open up options for memory reuse to avoid memory alloc/dealloc cycles. When memory allocations are powers of two, we can use BufferPool memory easily. BufferPool's Suballocator makes it possible to back these buffers with huge pages (though even simpler structures also work).
One thing I did in https://gerrit.cloudera.org/#/c/24126/ was to have the limit be the buffer size minus the expected row size. Something like that might avoid extra resizes from going over the buffer size. The downside is that when the expected row size is overestimated, we're missing out on some of the buffer space.
http://gerrit.cloudera.org:8080/#/c/25141/3/be/src/runtime/outbound-row-batch.cc
File be/src/runtime/outbound-row-batch.cc:
http://gerrit.cloudera.org:8080/#/c/25141/3/be/src/runtime/outbound-row-batch.cc@105
PS3, Line 105: Status OutboundRowBatch::GrowTupleData(
Do we know how often we resize buffers? Do we grow once and then keep it larger?
--
To view, visit http://gerrit.cloudera.org:8080/25141
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie60c1a020698cb5e2fe2ab60ed7586b897f15f18
Gerrit-Change-Number: 25141
Gerrit-PatchSet: 3
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 20 Oct 2026 03:20:53 +0000
Gerrit-HasComments: Yes