You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@kudu.apache.org by "Mike Percy (Code Review)" <ge...@cloudera.org> on 2017/05/22 22:01:45 UTC

[kudu-CR] KUDU-871 (part 1). Refactor to make cmeta a first class object

Hello David Ribeiro Alves, Todd Lipcon,

I'd like you to do a code review.  Please visit

    http://gerrit.cloudera.org:8080/6957

to review the following change.

Change subject: KUDU-871 (part 1). Refactor to make cmeta a first class object
......................................................................

KUDU-871 (part 1). Refactor to make cmeta a first class object

This patch is primarily a refactor to TabletReplica and RaftConsensus to
ensure that they are always constructed with a ConsensusMetadata object.
The ConsensusMetadata file is required for tombstoned voting as the
record of a replica's votes.

Changes in this patch:

1) Allow RaftConsensus to operate while not initialized or in a shutdown
   state. Also pass ConsensusMetadata into the RaftConsensus
   constructor.

2) Refactor TabletReplica to create a RaftConsensus instance at
   construction time. This will allow us to incorporate tombstoned
   voting later by allowing RaftConsensus to be accessible from a
   TabletReplica that comes up fresh in the TABLET_DATA_TOMBSTONED
   state. Also, do not destroy the Consensus object on
   TabletPeer::Shutdown() for the same reasons.

3) Add an additional on-disk data state (TABLET_DATA_FRESH) to allow us
   to identify when we crashed between writing the superblock and the
   ConsensusMetadata file. If a tablet is in that state then it has
   never voted so is legal to fully delete instead of attempt to recover
   at startup time.

Other changes:

* Remove unused Consensus::State enum.
* Get rid of TabletStatusListener
* bug fix: RaftConsensus must hold lock before snoozing FD

TODO:

* Add additional lifecycle runtime assertions in TabletReplica and
  RaftConsensus.
* Consider adding additional ReplicaState::State enum value for
  kInitialized but not kRunning.
* Finish writing new crash recovery tests.

Change-Id: Ia27a091d27b3996d37009d5ec866e744f9608388
---
M src/kudu/consensus/consensus.h
M src/kudu/consensus/raft_consensus.cc
M src/kudu/consensus/raft_consensus.h
M src/kudu/consensus/raft_consensus_quorum-test.cc
M src/kudu/integration-tests/ts_recovery-itest.cc
M src/kudu/master/sys_catalog.cc
M src/kudu/master/sys_catalog.h
M src/kudu/tablet/metadata.proto
M src/kudu/tablet/tablet_bootstrap-test.cc
M src/kudu/tablet/tablet_bootstrap.cc
M src/kudu/tablet/tablet_bootstrap.h
M src/kudu/tablet/tablet_replica-test.cc
M src/kudu/tablet/tablet_replica.cc
M src/kudu/tablet/tablet_replica.h
M src/kudu/tserver/tablet_copy_client.cc
M src/kudu/tserver/tablet_copy_client.h
M src/kudu/tserver/tablet_copy_source_session-test.cc
M src/kudu/tserver/ts_tablet_manager.cc
M src/kudu/tserver/ts_tablet_manager.h
M src/kudu/tserver/tserver-path-handlers.cc
20 files changed, 411 insertions(+), 381 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/57/6957/1
-- 
To view, visit http://gerrit.cloudera.org:8080/6957
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Ia27a091d27b3996d37009d5ec866e744f9608388
Gerrit-PatchSet: 1
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Mike Percy <mp...@apache.org>
Gerrit-Reviewer: David Ribeiro Alves <da...@gmail.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>

[kudu-CR] KUDU-871 (part 1). Refactor to make cmeta a first class object

Posted by "Mike Percy (Code Review)" <ge...@cloudera.org>.
Mike Percy has posted comments on this change.

Change subject: KUDU-871 (part 1). Refactor to make cmeta a first class object
......................................................................


Patch Set 1:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/6957/1//COMMIT_MSG
Commit Message:

PS1, Line 16: 1) Allow RaftConsensus to operate while not initialized or in a shutdown
            :    state. Also pass ConsensusMetadata into the RaftConsensus
            :    constructor.
> not 100% following this.
The problem is that I need to provide a long lived object (see patch 3 in this series) in order to avoid racy handling of the ConsensusMetadata by TabletCopy / tombstoned voting for example. The long lived object I chose was ConsensusMetadata because making both TabletReplica and RaftConsensus immortal is a much bigger change.

The compromise I landed on was ensuring that TabletReplica and RaftConsensus were always alive, however they would be replaced on TabletCopy. So the only immortal things end up being TabletMetadata and ConsensusMetadata.

The upshot is that we always get a RaftConsensus object whenever we have a TabletReplica object, but we don't have to support a cyclical lifecycle DAG on either object (going from Shutdown() -> Init() -> Start())


PS1, Line 41: * Add additional lifecycle runtime assertions in TabletReplica and
            :   RaftConsensus.
> Can you also clearly document the lifecycle for RaftConsensus? I'm not enti
The RaftConsensus::Shutdown() is similar to what it was before but you make a fair point that since RaftConsensus isn't destroyed on TabletPeer::Shutdown() anymore then it should be as aggressive as possible reclaiming memory when it's shut down. I'll check on that.

I tried to give a summary of the lifecycle in my above comment. Let me know if I need to clarify anything and I'll add more details as a comment in the next rev.


-- 
To view, visit http://gerrit.cloudera.org:8080/6957
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ia27a091d27b3996d37009d5ec866e744f9608388
Gerrit-PatchSet: 1
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Mike Percy <mp...@apache.org>
Gerrit-Reviewer: David Ribeiro Alves <da...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Mike Percy <mp...@apache.org>
Gerrit-Reviewer: Tidy Bot
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-HasComments: Yes

[kudu-CR] WIP: Ensure ConsensusMetadata exists, even for tombstoned tablets

Posted by "Mike Percy (Code Review)" <ge...@cloudera.org>.
Hello Kudu Jenkins,

I'd like you to reexamine a change.  Please visit

    http://gerrit.cloudera.org:8080/6957

to look at the new patch set (#3).

Change subject: WIP: Ensure ConsensusMetadata exists, even for tombstoned tablets
......................................................................

WIP: Ensure ConsensusMetadata exists, even for tombstoned tablets

Adds a TABLET_DATA_FRESH data state and a NOT_INITIALIZED run state.

The ConsensusMetadata file is required for tombstoned voting as the
record of a replica's votes.

Changes:

1) Add an additional on-disk data state (TABLET_DATA_FRESH) to allow us
   to identify when we crashed between writing the superblock and the
   ConsensusMetadata file. If a tablet is in that state then it has
   never voted so is legal to fully delete instead of attempt to recover
   at startup time.

2) Add an additional run state to TabletReplica (NOT_INITIALIZED) for
   lifecycle validation purposes.

3) Refactor TabletReplica to create a RaftConsensus instance at Init()
   time. This will allow us to incorporate tombstoned voting later by
   allowing RaftConsensus to be accessible from a TabletReplica that
   comes up fresh in the TABLET_DATA_TOMBSTONED state.

Tests:

* Adds a test for TABLET_DATA_FRESH crash recovery behavior at tablet
  creation time: TsRecoveryITest.TestCrashBeforeWritingConsensusMetadata
* Adds a tablet copy fault test as well:
  TabletCopyITest.TestTabletCopyFresh

TODO:

* TabletCopyITest.TestTabletCopyFresh is currently failing.

Change-Id: Ia27a091d27b3996d37009d5ec866e744f9608388
---
M src/kudu/integration-tests/tablet_copy-itest.cc
M src/kudu/integration-tests/ts_recovery-itest.cc
M src/kudu/master/sys_catalog.cc
M src/kudu/tablet/metadata.proto
M src/kudu/tablet/tablet_replica-test.cc
M src/kudu/tablet/tablet_replica.cc
M src/kudu/tablet/tablet_replica.h
M src/kudu/tserver/tablet_copy_client.cc
M src/kudu/tserver/tablet_copy_client.h
M src/kudu/tserver/tablet_copy_source_session-test.cc
M src/kudu/tserver/ts_tablet_manager.cc
M src/kudu/tserver/ts_tablet_manager.h
12 files changed, 251 insertions(+), 51 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/57/6957/3
-- 
To view, visit http://gerrit.cloudera.org:8080/6957
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ia27a091d27b3996d37009d5ec866e744f9608388
Gerrit-PatchSet: 3
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Mike Percy <mp...@apache.org>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: David Ribeiro Alves <da...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Mike Percy <mp...@apache.org>
Gerrit-Reviewer: Tidy Bot
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>

[kudu-CR] WIP: Ensure ConsensusMetadata exists, even for tombstoned tablets

Posted by "Mike Percy (Code Review)" <ge...@cloudera.org>.
Hello David Ribeiro Alves, Alexey Serbin, Kudu Jenkins,

I'd like you to reexamine a change.  Please visit

    http://gerrit.cloudera.org:8080/6957

to look at the new patch set (#2).

Change subject: WIP: Ensure ConsensusMetadata exists, even for tombstoned tablets
......................................................................

WIP: Ensure ConsensusMetadata exists, even for tombstoned tablets

Adds a TABLET_DATA_FRESH data state and a NOT_INITIALIZED run state.

The ConsensusMetadata file is required for tombstoned voting as the
record of a replica's votes.

Changes:

1) Add an additional on-disk data state (TABLET_DATA_FRESH) to allow us
   to identify when we crashed between writing the superblock and the
   ConsensusMetadata file. If a tablet is in that state then it has
   never voted so is legal to fully delete instead of attempt to recover
   at startup time.

2) Add an additional run state to TabletReplica (NOT_INITIALIZED) for
   lifecycle validation purposes.

3) Refactor TabletReplica to create a RaftConsensus instance at Init()
   time. This will allow us to incorporate tombstoned voting later by
   allowing RaftConsensus to be accessible from a TabletReplica that
   comes up fresh in the TABLET_DATA_TOMBSTONED state.

Tests:

* Adds a test for TABLET_DATA_FRESH crash recovery behavior at tablet
  creation time: TsRecoveryITest.TestCrashBeforeWritingConsensusMetadata
* Adds a tablet copy fault test as well:
  TabletCopyITest.TestTabletCopyFresh

TODO:

* TabletCopyITest.TestTabletCopyFresh is currently failing.

Change-Id: Ia27a091d27b3996d37009d5ec866e744f9608388
---
M src/kudu/integration-tests/tablet_copy-itest.cc
M src/kudu/integration-tests/ts_recovery-itest.cc
M src/kudu/master/sys_catalog.cc
M src/kudu/tablet/metadata.proto
M src/kudu/tablet/tablet_replica-test.cc
M src/kudu/tablet/tablet_replica.cc
M src/kudu/tablet/tablet_replica.h
M src/kudu/tserver/tablet_copy_client.cc
M src/kudu/tserver/tablet_copy_client.h
M src/kudu/tserver/tablet_copy_source_session-test.cc
M src/kudu/tserver/ts_tablet_manager.cc
M src/kudu/tserver/ts_tablet_manager.h
12 files changed, 253 insertions(+), 52 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/57/6957/2
-- 
To view, visit http://gerrit.cloudera.org:8080/6957
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ia27a091d27b3996d37009d5ec866e744f9608388
Gerrit-PatchSet: 2
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Mike Percy <mp...@apache.org>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: David Ribeiro Alves <da...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Mike Percy <mp...@apache.org>
Gerrit-Reviewer: Tidy Bot
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>

[kudu-CR] KUDU-871 (part 1). Refactor to make cmeta a first class object

Posted by "Todd Lipcon (Code Review)" <ge...@cloudera.org>.
Todd Lipcon has posted comments on this change.

Change subject: KUDU-871 (part 1). Refactor to make cmeta a first class object
......................................................................


Patch Set 1:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/6957/1//COMMIT_MSG
Commit Message:

PS1, Line 16: 1) Allow RaftConsensus to operate while not initialized or in a shutdown
            :    state. Also pass ConsensusMetadata into the RaftConsensus
            :    constructor.
not 100% following this.

I thought when discussing this, the idea was to do one or the other of these changes, not both?

i.e., if you make it so you can instantiate RaftConsensus even for tombstoned tablet replicas, then why isn't it sufficient to still keep ConsensusMeta confined to being within the RaftConsensus class?

Or alternatively, if you make TabletReplica aware of ConsensusMetadata, why do you need to make changes to the RaftConsensus lifecycle?


PS1, Line 41: * Add additional lifecycle runtime assertions in TabletReplica and
            :   RaftConsensus.
Can you also clearly document the lifecycle for RaftConsensus? I'm not entirely following it from the patch. eg when you first create a RaftConsensus object, it seems to exist but have a lot of nullptr members. But then when you do Shutdown() on it, it doesn't re-nullify those members. Does that imply that a RaftConsensus instance that was tombstoned on a live server has different state than one that was tombstoned at startup? Does that have impact on thread lifecycle or memory consumption?

Put another way, the above commit message tries to describe the changes, but I'm not sure what the new "state of the world" is, and the class docs here aren't clarified enough to make me feel confident that the new state of the world is correct.


-- 
To view, visit http://gerrit.cloudera.org:8080/6957
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ia27a091d27b3996d37009d5ec866e744f9608388
Gerrit-PatchSet: 1
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Mike Percy <mp...@apache.org>
Gerrit-Reviewer: David Ribeiro Alves <da...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-HasComments: Yes

[kudu-CR] WIP: Ensure ConsensusMetadata exists, even for tombstoned tablets

Posted by "David Ribeiro Alves (Code Review)" <ge...@cloudera.org>.
David Ribeiro Alves has posted comments on this change.

Change subject: WIP: Ensure ConsensusMetadata exists, even for tombstoned tablets
......................................................................


Patch Set 1:

(5 comments)

just browsed this one since it seems like it needs updating. the approach in general seems sound. I particularly like the avoidance of re-vivifying the replica.

http://gerrit.cloudera.org:8080/#/c/6957/1//COMMIT_MSG
Commit Message:

PS1, Line 16: 1) Allow RaftConsensus to operate while not initialized or in a shutdown
            :    state. Also pass ConsensusMetadata into the RaftConsensus
            :    constructor.
> The problem is that I need to provide a long lived object (see patch 3 in t
Mike: can you move some of this info to the commit message?


http://gerrit.cloudera.org:8080/#/c/6957/3/src/kudu/integration-tests/tablet_copy-itest.cc
File src/kudu/integration-tests/tablet_copy-itest.cc:

PS3, Line 900:  all of that data over to the 2nd tablet se
would the test fail if we kept it running? if so say why, if not consider leaving it running


PS3, Line 919: for (const Status& s : statuses) {
check the returned status


http://gerrit.cloudera.org:8080/#/c/6957/3/src/kudu/master/sys_catalog.cc
File src/kudu/master/sys_catalog.cc:

PS3, Line 317:                                          
didn't you get rid of this method? still working on propagating those changes?


http://gerrit.cloudera.org:8080/#/c/6957/1/src/kudu/tserver/ts_tablet_manager.cc
File src/kudu/tserver/ts_tablet_manager.cc:

PS1, Line 1061:  // If we are missing the ConsensusMetadata file then we will ultimately log
              :     // a FATAL.
not sure what this means?


-- 
To view, visit http://gerrit.cloudera.org:8080/6957
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ia27a091d27b3996d37009d5ec866e744f9608388
Gerrit-PatchSet: 1
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Mike Percy <mp...@apache.org>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: David Ribeiro Alves <da...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Mike Percy <mp...@apache.org>
Gerrit-Reviewer: Tidy Bot
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-HasComments: Yes

[kudu-CR] WIP: Ensure ConsensusMetadata exists, even for tombstoned tablets

Posted by "Mike Percy (Code Review)" <ge...@cloudera.org>.
Mike Percy has posted comments on this change.

Change subject: WIP: Ensure ConsensusMetadata exists, even for tombstoned tablets
......................................................................


Patch Set 3:

Just to explain why we don't need FRESH: if we can create "blank" ConsensusMetadata at-will then a failure to create or copy it initially after the superblock isn't a problem. We just assume it's empty and auto-create it as necessary.

-- 
To view, visit http://gerrit.cloudera.org:8080/6957
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ia27a091d27b3996d37009d5ec866e744f9608388
Gerrit-PatchSet: 3
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Mike Percy <mp...@apache.org>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: David Ribeiro Alves <da...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Mike Percy <mp...@apache.org>
Gerrit-Reviewer: Tidy Bot
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-HasComments: No

[kudu-CR] WIP: Ensure ConsensusMetadata exists, even for tombstoned tablets

Posted by "Mike Percy (Code Review)" <ge...@cloudera.org>.
Mike Percy has abandoned this change.

Change subject: WIP: Ensure ConsensusMetadata exists, even for tombstoned tablets
......................................................................


Abandoned

I talked to Todd about this and we think it's safe to autovivify replicas. Since we've become convinced that's safe, we can avoid having the additional data state TABLET_DATA_FRESH.

-- 
To view, visit http://gerrit.cloudera.org:8080/6957
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: abandon
Gerrit-Change-Id: Ia27a091d27b3996d37009d5ec866e744f9608388
Gerrit-PatchSet: 3
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Mike Percy <mp...@apache.org>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: David Ribeiro Alves <da...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Mike Percy <mp...@apache.org>
Gerrit-Reviewer: Tidy Bot
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>

[kudu-CR] KUDU-871 (part 1). Refactor to make cmeta a first class object

Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change.

Change subject: KUDU-871 (part 1). Refactor to make cmeta a first class object
......................................................................


Patch Set 1:

(6 comments)

some nits while skimming through

http://gerrit.cloudera.org:8080/#/c/6957/1//COMMIT_MSG
Commit Message:

PS1, Line 35: Remove unused Consensus::State enum.
nit: maybe, it makes sense to publish this change as separate tiny clean-up changelist?


PS1, Line 37: * bug fix: RaftConsensus must hold lock before snoozing FD
nit: if it's a bug fix, would it make sense to separate it and make sure all current tests pass before introducing such a major refactoring?  Or it becomes a bug only when combined with the new refactored code?


http://gerrit.cloudera.org:8080/#/c/6957/1/src/kudu/tablet/metadata.proto
File src/kudu/tablet/metadata.proto:

PS1, Line 77: TABLET_DATA_FRESH
Given the description I would expect this is a broader counterpart of TABLET_DATA_COPYING.  Is that true or there are some additional nits regarding voting, etc.?


http://gerrit.cloudera.org:8080/#/c/6957/1/src/kudu/tserver/ts_tablet_manager.cc
File src/kudu/tserver/ts_tablet_manager.cc:

PS1, Line 244: $0Unable
typo?  it seems space is missing


PS1, Line 593: *tablet_replica = tmp_replica;
Would it make sense to allow the 'tablet_replica' parameter be nullptr?  It seems in a couple of places in this file the result tablet replica output by this method is not used at this method's call sites.


PS1, Line 1064: $0Unable
typo?


-- 
To view, visit http://gerrit.cloudera.org:8080/6957
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ia27a091d27b3996d37009d5ec866e744f9608388
Gerrit-PatchSet: 1
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Mike Percy <mp...@apache.org>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: David Ribeiro Alves <da...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Mike Percy <mp...@apache.org>
Gerrit-Reviewer: Tidy Bot
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-HasComments: Yes