[kudu-CR] KUDU-3812 Test ordered scan memory usage with large cells.

["Zoltan Martonka (Code Review)" <ge...@cloudera.org>]

Hello Marton Greber, Alexey Serbin, Ashwani Raina, Kudu Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/24731

to look at the new patch set (#5).

Change subject: KUDU-3812 Test ordered scan memory usage with large cells.
......................................................................

KUDU-3812 Test ordered scan memory usage with large cells.

Currently sorted scans have a burned in 1024 size buffer.
With 1 MB of cells this can cause large, untracked memory usage.

This test does the following:
+ Start a cluster with 1 tserver and create a tablet.
+ Disable compaction for worst case result (compaction falls behind).
+ Write some rows with 1 MB cells into a tablet.
+ Disable flushing then keep writing until memory limit is hit.
+ Start 10 parallel ordered full scans. They are started one-by-one with a
  gap between them, so they probably request other parts of the data at
  the same time.
+ Check the peak memory usage.

The test increases RPC and transaction limits, but the whole scanning still
happens when they are set to default. So the memory usage would still happen.

One run gave me:
Peak tmalloc memory during test :  18697 MiB

We will need to dynamically downsize batches when cells are really large,
so the memory usage won't skyrocket and rpc/transaction limits can remain
untouched.

Change-Id: I75ceaebac1793c2421302f8b205ba85ad32a141a
---
M src/kudu/integration-tests/CMakeLists.txt
A src/kudu/integration-tests/ordered_scan_large_cells-itest.cc
2 files changed, 431 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/31/24731/5
-- 
To view, visit http://gerrit.cloudera.org:8080/24731
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I75ceaebac1793c2421302f8b205ba85ad32a141a
Gerrit-Change-Number: 24731
Gerrit-PatchSet: 5
Gerrit-Owner: Zoltan Martonka <zm...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Reviewer: Zoltan Martonka <zm...@cloudera.com>