You are viewing a plain text version of this content. The canonical link for it is here.

Posted to reviews@kudu.apache.org by "Andrew Wong (Code Review)" <ge...@cloudera.org> on 2020/09/23 02:03:52 UTC

[kudu-CR] KUDU-2612 p11: persist txn metadata in superblock

Andrew Wong has uploaded this change for review. ( http://gerrit.cloudera.org:8080/16492


Change subject: KUDU-2612 p11: persist txn metadata in superblock
......................................................................

KUDU-2612 p11: persist txn metadata in superblock

Currently, transaction metadata is entirely stored in the WALs; once a
transaction is begun, it anchors WALs indefinitely. This is untenable,
as any tablet that participates in a transaction would never be able to
GC its WALs.

This patch addresses this by storing transaction metadata in the tablet
superblock for the following participant ops:
- BEGIN_TXN: an empty record is added to the superblock for the given
  transaction ID. In the future, this can be used to store the owner of
  the transaction.
- FINALIZE_COMMIT: the commit timestamp is added to the superblock for
  the given transaction ID.
- ABORT_TXN: an abort bit is added to the superblock for the given
  transaction ID.

Upon applying each of these ops, the op's OpId is anchored in the WALs
until the tablet metadata is successfully flushed, ensuring that if we
don't flush the tablet metadata before restarting, we can rebuild any
ops that whose mutations had not been persisted by replaying the WALs.

What about BEGIN_COMMIT ops? While these ops will still need to be
anchored, BEGIN_COMMIT ops don't need to persist any long-term
information. As such, their anchoring is slightly different -- rather
than mutating the tablet metadata and unanchoring on a metadata flush,
BEGIN_COMMIT ops will update the in-memory state for the in-flight Txn,
and unanchor once the corresponding FINALIZE_COMMIT or ABORT_TXN begins
applying.

On bootstrap, the following rules are applied:
- If we've persisted terminal information (i.e. abort or commit) about a
  transaction ID, we can skip replaying any participant op for that
  transaction.
- If we otherwise have a persisted record for the transaction ID, we
  should start an open transaction and replay all participant ops for
  that transaction.
- If we have no record of a transaction persisted, we must replay all
  participant ops for the given transaction.

While storing things in the superblock is not ideal, there are further
improvements that can be made to reduce its size, e.g. after finalizing
a transaction, the transactions mutations may be merged in with the rest
of the tablet; once all mutations of a given transaction have been
merged, the transaction metadata may be removed from the metadata.

Change-Id: I2f32808aef10484f4e0ad3942bb005f61fbdb34a
---
M src/kudu/integration-tests/txn_participant-itest.cc
M src/kudu/tablet/metadata.proto
M src/kudu/tablet/ops/participant_op.cc
M src/kudu/tablet/ops/participant_op.h
M src/kudu/tablet/tablet.cc
M src/kudu/tablet/tablet.h
M src/kudu/tablet/tablet_bootstrap.cc
M src/kudu/tablet/tablet_metadata-test.cc
M src/kudu/tablet/tablet_metadata.cc
M src/kudu/tablet/tablet_metadata.h
M src/kudu/tablet/txn_participant-test.cc
M src/kudu/tablet/txn_participant.cc
M src/kudu/tablet/txn_participant.h
13 files changed, 675 insertions(+), 119 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/92/16492/1
-- 
To view, visit http://gerrit.cloudera.org:8080/16492
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I2f32808aef10484f4e0ad3942bb005f61fbdb34a
Gerrit-Change-Number: 16492
Gerrit-PatchSet: 1
Gerrit-Owner: Andrew Wong <aw...@cloudera.com>