[kudu-CR] KUDU-2612 p11: persist txn metadata in superblock

["Andrew Wong (Code Review)" <ge...@cloudera.org>]

Hello Alexey Serbin, Kudu Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/16492

to look at the new patch set (#8).

Change subject: KUDU-2612 p11: persist txn metadata in superblock
......................................................................

KUDU-2612 p11: persist txn metadata in superblock

Currently, transaction metadata is entirely stored in the WALs; once a
transaction is begun, it anchors WALs indefinitely. This is untenable,
as any tablet that participates in a transaction would never be able to
GC its WALs.

This patch addresses this by storing transaction metadata in the tablet
superblock for the following participant ops:
- BEGIN_TXN: an empty record is added to the superblock for the given
  transaction ID. In the future, this can be used to store the owner of
  the transaction.
- FINALIZE_COMMIT: the commit timestamp is added to the superblock for
  the given transaction ID. The commit timestamp of a transaction will
  be used when scanning over mutations applied to a tablet instead of
  the apply timestamp.
- ABORT_TXN: an abort bit is added to the superblock for the given
  transaction ID.

Upon applying each of these ops, the op's OpId is anchored in the WALs
until the tablet metadata is successfully flushed, ensuring that if we
don't flush the tablet metadata before restarting, we can rebuild any
ops that whose mutations had not been persisted by replaying the WALs.

What about BEGIN_COMMIT ops? While these ops will still need to be
anchored, BEGIN_COMMIT ops don't need to persist any long-term
information. As such, their anchoring is slightly different -- rather
than mutating the tablet metadata and unanchoring on a metadata flush,
BEGIN_COMMIT ops will update the in-memory state for the in-flight Txn,
and unanchor once the corresponding FINALIZE_COMMIT or ABORT_TXN begins
applying.

On bootstrap, the following rules are applied:
- If we've persisted terminal information (i.e. abort or commit) about a
  transaction ID, we can skip replaying any participant op for that
  transaction.
- If we otherwise have a persisted record for the transaction ID, we
  should start an open transaction and replay all participant ops for
  that transaction.
- If we have no record of a transaction persisted, we must replay all
  participant ops for the given transaction.

While storing things in the superblock is not ideal, there are further
improvements that can be made to reduce its size, e.g. after finalizing
a transaction, the transactions mutations may be merged in with the rest
of the tablet; once all mutations of a given transaction have been
merged, the transaction metadata may be removed from the metadata.

Change-Id: I2f32808aef10484f4e0ad3942bb005f61fbdb34a
---
M src/kudu/integration-tests/txn_participant-itest.cc
M src/kudu/tablet/metadata.proto
M src/kudu/tablet/ops/participant_op.cc
M src/kudu/tablet/ops/participant_op.h
M src/kudu/tablet/tablet.cc
M src/kudu/tablet/tablet.h
M src/kudu/tablet/tablet_bootstrap.cc
M src/kudu/tablet/tablet_metadata-test.cc
M src/kudu/tablet/tablet_metadata.cc
M src/kudu/tablet/tablet_metadata.h
M src/kudu/tablet/txn_participant-test.cc
M src/kudu/tablet/txn_participant.cc
M src/kudu/tablet/txn_participant.h
13 files changed, 719 insertions(+), 131 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/92/16492/8
-- 
To view, visit http://gerrit.cloudera.org:8080/16492
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I2f32808aef10484f4e0ad3942bb005f61fbdb34a
Gerrit-Change-Number: 16492
Gerrit-PatchSet: 8
Gerrit-Owner: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)