You are viewing a plain text version of this content. The canonical link for it is here.

Posted to reviews@kudu.apache.org by "Ashwani Raina (Code Review)" <ge...@cloudera.org> on 2026/05/20 09:07:04 UTC

[kudu-CR] [WIP] KUDU-3429 Fix rowset compaction memory accounting (2/n)

Ashwani Raina has uploaded this change for review. ( http://gerrit.cloudera.org:8080/24330


Change subject: [WIP] KUDU-3429 Fix rowset compaction memory accounting (2/n)
......................................................................

[WIP] KUDU-3429 Fix rowset compaction memory accounting (2/n)

The patch fixes the following:
- 'delta_blocks_mem_size()' does double accounting of the CFile block
  buffer by adding Slice::val again when it is already accounted for
  inside DeltaFileIterator::delta_blocks_mem_size_. The Slice::val field
  is a zero-copy view of CFile block buffer and not a separate memory
  area where block data is stored. Remove additional buffer sizes.
- Keep memory accounting calculation for prepare_deltas_ separate from
  delta_blocks_mem_size_ to avoid scenarios where delta block buffers
  are freed (inside FreeDeltaBlocks) while struct overhead for entries
  in prepared_deltas_ remain until preparation of next batch starts.
  So, even after clearing of delta blocks , memory_footprint() still
  reflects surviving PreparedDelta struct overhead.
- 'peak_mem_usage' doesn't show the correct peak of memory usage of a
  rowset compaction operation. It leaves out base-data arena and
  ApplyMutations arena contributions that the tracker does capture.
  MergeCompactionInput::max_memory_usage_ is sampled at FinishBlock()
  time when all the DRS delta blocks are not residing in memory
  simultaneously. Instead, make use of tracker's peak_consumption() that
  takes into consideration all the big consumers of memory during a
  rowset operation except base row data arena that is significantly low
  in memory usage as compared to other consumers.

TODO:
- Explore including base data area memory consumption, either in current
  or a separate patch.

Change-Id: Icc2133d38ed766f6834392a2e335d9246b511456
---
M src/kudu/tablet/delta_store.h
M src/kudu/tablet/deltafile.cc
M src/kudu/tablet/deltafile.h
M src/kudu/tablet/tablet.cc
4 files changed, 21 insertions(+), 15 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/30/24330/1
-- 
To view, visit http://gerrit.cloudera.org:8080/24330
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Icc2133d38ed766f6834392a2e335d9246b511456
Gerrit-Change-Number: 24330
Gerrit-PatchSet: 1
Gerrit-Owner: Ashwani Raina <ar...@gmail.com>

[kudu-CR] KUDU-3429 Fix rowset compaction memory accounting (2/n)

Posted by "Marton Greber (Code Review)" <ge...@cloudera.org>.

Marton Greber has removed a vote on this change.

Change subject: KUDU-3429 Fix rowset compaction memory accounting (2/n)
......................................................................


Removed Verified-1 by Kudu Jenkins (120)
-- 
To view, visit http://gerrit.cloudera.org:8080/24330
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: deleteVote
Gerrit-Change-Id: Icc2133d38ed766f6834392a2e335d9246b511456
Gerrit-Change-Number: 24330
Gerrit-PatchSet: 3
Gerrit-Owner: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Reviewer: Zoltan Martonka <zm...@cloudera.com>

[kudu-CR] KUDU-3429 Fix rowset compaction memory accounting (2/n)

Posted by "Marton Greber (Code Review)" <ge...@cloudera.org>.

Marton Greber has posted comments on this change. ( http://gerrit.cloudera.org:8080/24330 )

Change subject: KUDU-3429 Fix rowset compaction memory accounting (2/n)
......................................................................


Patch Set 3: Verified+1 Code-Review+1

unrelated test failures:
asan: tablet_server-test.2
tsan: catalog_manager-test.0


-- 
To view, visit http://gerrit.cloudera.org:8080/24330
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icc2133d38ed766f6834392a2e335d9246b511456
Gerrit-Change-Number: 24330
Gerrit-PatchSet: 3
Gerrit-Owner: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Reviewer: Zoltan Martonka <zm...@cloudera.com>
Gerrit-Comment-Date: Wed, 03 Jun 2026 13:25:30 +0000
Gerrit-HasComments: No

[kudu-CR] KUDU-3429 Fix rowset compaction memory accounting (2/n)

Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.

Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24330 )

Change subject: KUDU-3429 Fix rowset compaction memory accounting (2/n)
......................................................................


Patch Set 3:

Build Started http://jenkins.kudu.apache.org/job/pre_commit/2627/


-- 
To view, visit http://gerrit.cloudera.org:8080/24330
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icc2133d38ed766f6834392a2e335d9246b511456
Gerrit-Change-Number: 24330
Gerrit-PatchSet: 3
Gerrit-Owner: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Reviewer: Zoltan Martonka <zm...@cloudera.com>
Gerrit-Comment-Date: Tue, 02 Jun 2026 07:28:26 +0000
Gerrit-HasComments: No

[kudu-CR] KUDU-3429 Fix rowset compaction memory accounting (2/n)

Posted by "Zoltan Martonka (Code Review)" <ge...@cloudera.org>.

Zoltan Martonka has posted comments on this change. ( http://gerrit.cloudera.org:8080/24330 )

Change subject: KUDU-3429 Fix rowset compaction memory accounting (2/n)
......................................................................


Patch Set 3: Code-Review+1


-- 
To view, visit http://gerrit.cloudera.org:8080/24330
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icc2133d38ed766f6834392a2e335d9246b511456
Gerrit-Change-Number: 24330
Gerrit-PatchSet: 3
Gerrit-Owner: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Reviewer: Zoltan Martonka <zm...@cloudera.com>
Gerrit-Comment-Date: Fri, 12 Jun 2026 08:01:27 +0000
Gerrit-HasComments: No

[kudu-CR] KUDU-3429 Fix rowset compaction memory accounting (2/n)

Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.

Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24330 )

Change subject: KUDU-3429 Fix rowset compaction memory accounting (2/n)
......................................................................


Patch Set 2:

Build Started http://jenkins.kudu.apache.org/job/pre_commit/2622/


-- 
To view, visit http://gerrit.cloudera.org:8080/24330
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icc2133d38ed766f6834392a2e335d9246b511456
Gerrit-Change-Number: 24330
Gerrit-PatchSet: 2
Gerrit-Owner: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Zoltan Martonka <zm...@cloudera.com>
Gerrit-Comment-Date: Mon, 01 Jun 2026 12:31:47 +0000
Gerrit-HasComments: No

[kudu-CR] KUDU-3429 Fix rowset compaction memory accounting (2/n)

Posted by "Ashwani Raina (Code Review)" <ge...@cloudera.org>.

Hello Marton Greber, Zoltan Martonka, Kudu Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/24330

to look at the new patch set (#3).

Change subject: KUDU-3429 Fix rowset compaction memory accounting (2/n)
......................................................................

KUDU-3429 Fix rowset compaction memory accounting (2/n)

The patch fixes the following:
- 'delta_blocks_mem_size()' does double accounting of the CFile block
  buffer by adding Slice::val again when it is already accounted for
  inside DeltaFileIterator::delta_blocks_mem_size_. The Slice::val field
  is a zero-copy view of CFile block buffer and not a separate memory
  area where block data is stored. Remove additional buffer sizes.
- Keep memory accounting calculation for prepare_deltas_ separate from
  delta_blocks_mem_size_ to avoid scenarios where delta block buffers
  are freed (inside FreeDeltaBlocks) while struct overhead for entries
  in prepared_deltas_ remain until preparation of next batch starts.
  So, even after clearing of delta blocks , memory_footprint() still
  reflects surviving PreparedDelta struct overhead.
- 'peak_mem_usage' doesn't show the correct peak of memory usage of a
  rowset compaction operation. It leaves out base-data arena and
  ApplyMutations arena contributions that the tracker does capture.
  MergeCompactionInput::max_memory_usage_ is sampled at FinishBlock()
  time when all the DRS delta blocks are not residing in memory
  simultaneously. Instead, make use of tracker's peak_consumption() that
  takes into consideration all the big consumers of memory during a
  rowset merge compaction operation.
- After fixing the double-counting in delta_blocks_mem_size(), tracker
  value is lower and the old '3 MB' constant is no longer reliably
  exceeded for tests in compaction-highmem-test. Set the threshold
  (compaction_mem_usage_approx) to a value that is reliably below the
  minimum tracked allocation during compaction run.
- Pre-account for the arena's initial component so that the first
  PrepareBlock's Reset() produces a correctly-signed delta.

Change-Id: Icc2133d38ed766f6834392a2e335d9246b511456
---
M src/kudu/tablet/compaction-highmem-test.cc
M src/kudu/tablet/compaction.cc
M src/kudu/tablet/delta_store.h
M src/kudu/tablet/deltafile.cc
M src/kudu/tablet/deltafile.h
M src/kudu/tablet/tablet.cc
6 files changed, 44 insertions(+), 23 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/30/24330/3
-- 
To view, visit http://gerrit.cloudera.org:8080/24330
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Icc2133d38ed766f6834392a2e335d9246b511456
Gerrit-Change-Number: 24330
Gerrit-PatchSet: 3
Gerrit-Owner: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Reviewer: Zoltan Martonka <zm...@cloudera.com>