[Impala-ASF-CR] IMPALA-3202: implement spill-to-disk in new buffer pool

["Tim Armstrong (Code Review)" <ge...@cloudera.org>]

Tim Armstrong has uploaded a new patch set (#7).

Change subject: IMPALA-3202: implement spill-to-disk in new buffer pool
......................................................................

IMPALA-3202: implement spill-to-disk in new buffer pool

See https://goo.gl/0zuy97 for a high-level summary of the design.

Unpinned pages can now be written to disk to free up memory. After
unpinning, pages enter a "dirty" state. Each client initiates
asynchronous writes for dirty page to free up memory to allocate
more buffers. After the write completes, pages are "clean" and can
be evicted from memory by any client that needs the buffer. This
is implemented by moving pages between lists in ClientImpl
(a new internal class that stores the client's state) and BufferPool.

I/O:
----
The mechanics of I/O to scratch files is handled by the TmpFileMgr
mechanisms introduced in earlier IMPALA-3202 patches.

The decision to start a write I/O is based on two factors. First,
each client maintains the invariant that bytes of the dirty
pages should not exceed the unused reservation. This is to ensure
that any client's reservation can always be fulfilled by evicting
a clean page. Second, additional writes are initiated to proactively
write data to disk, so that clean pages are available when needed.
The buffer pool tries to keep enough writes in flight to keep all
disks busy.

The buffer pool avoids read I/O whenever possible: if an unpinned
page is pinned again and its buffer is still in memory, no I/O
is required to read back the data.

Locking:
--------
Concurrency is managed using client, page, and clean page list locks.
The design uses intrusive doubly-linked lists to track pages. This
patch adds a LockType parameter to InternalQueue and a FakeLock type
to support a non-concurrent InternalList type that is more convenient
for the buffer pool's locking scheme.

Testing:
--------
Added some basic unit tests to BufferPoolTest that exercise the new
code paths. More test coverage will be added later with system tests
and porting of tests from BufferedBlockMgrTest - see the draft
test plan for IMPALA-3200.

Change-Id: I8c6119a4557da853fefa02e17c49d8be0e9fbe01
---
M be/src/common/compiler-util.h
A be/src/runtime/bufferpool/buffer-pool-internal.h
M be/src/runtime/bufferpool/buffer-pool-test.cc
M be/src/runtime/bufferpool/buffer-pool.cc
M be/src/runtime/bufferpool/buffer-pool.h
M be/src/runtime/bufferpool/suballocator-test.cc
M be/src/runtime/tmp-file-mgr.h
M be/src/util/aligned-new.h
A be/src/util/fake-lock.h
M be/src/util/internal-queue.h
10 files changed, 1,009 insertions(+), 297 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/84/5584/7
-- 
To view, visit http://gerrit.cloudera.org:8080/5584
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I8c6119a4557da853fefa02e17c49d8be0e9fbe01
Gerrit-PatchSet: 7
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>