You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@kudu.apache.org by "Alexey Serbin (Code Review)" <ge...@cloudera.org> on 2017/02/08 02:00:46 UTC

[kudu-CR] WIP [security] load TSK from system table

Alexey Serbin has uploaded a new change for review.

  http://gerrit.cloudera.org:8080/5935

Change subject: WIP [security] load TSK from system table
......................................................................

WIP [security] load TSK from system table

Change-Id: Ie91d4129bda0ca49e81988c28385895a2abcd201
---
M src/kudu/master/authn_token_manager.h
M src/kudu/master/catalog_manager.cc
M src/kudu/master/catalog_manager.h
M src/kudu/master/master.cc
M src/kudu/master/master.proto
M src/kudu/master/sys_catalog.cc
M src/kudu/master/sys_catalog.h
M src/kudu/security/token_signer.cc
M src/kudu/security/token_signer.h
9 files changed, 263 insertions(+), 10 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/35/5935/1
-- 
To view, visit http://gerrit.cloudera.org:8080/5935
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Ie91d4129bda0ca49e81988c28385895a2abcd201
Gerrit-PatchSet: 1
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>

[kudu-CR] [security] load/store public TSK in the system table

Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Hello Kudu Jenkins,

I'd like you to reexamine a change.  Please visit

    http://gerrit.cloudera.org:8080/5935

to look at the new patch set (#10).

Change subject: [security] load/store public TSK in the system table
......................................................................

[security] load/store public TSK in the system table

Non-expired, already existing TSK records are read from the system
catalog table upon the 'elected-as-leader' callback and fed into the
TokenSigner.  New token signing keys are generated and written
into the system catalog table by a periodic task.

The expired entries are removed from the system table
upon the 'elected-as-leader' callback.

Added integration tests for cluster start/restart and switching
master leadership role into token_signer-itest.cc.

Change-Id: Ie91d4129bda0ca49e81988c28385895a2abcd201
---
M src/kudu/client/client-test.cc
M src/kudu/integration-tests/CMakeLists.txt
A src/kudu/integration-tests/token_signer-itest.cc
M src/kudu/master/catalog_manager.cc
M src/kudu/master/catalog_manager.h
M src/kudu/master/master.cc
M src/kudu/master/master.proto
M src/kudu/master/sys_catalog.cc
M src/kudu/master/sys_catalog.h
9 files changed, 463 insertions(+), 34 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/35/5935/10
-- 
To view, visit http://gerrit.cloudera.org:8080/5935
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie91d4129bda0ca49e81988c28385895a2abcd201
Gerrit-PatchSet: 10
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot

[kudu-CR] [security] load/store public TSK in the system table

Posted by "Todd Lipcon (Code Review)" <ge...@cloudera.org>.
Todd Lipcon has posted comments on this change.

Change subject: [security] load/store public TSK in the system table
......................................................................


Patch Set 18:

(2 comments)

Didn't look through the whole patch but spotted some issues

http://gerrit.cloudera.org:8080/#/c/5935/22/src/kudu/master/catalog_manager.cc
File src/kudu/master/catalog_manager.cc:

Line 442:     CatalogManager::ScopedLeaderSharedLock l(catalog_manager_);
this lock is now being held while in Wait() which isn't desirable


PS22, Line 477:             // skip further actions on this iteration
              :             break;
isn't this now breaking out of the whole loop, which would break background tasks from futher progress?


-- 
To view, visit http://gerrit.cloudera.org:8080/5935
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ie91d4129bda0ca49e81988c28385895a2abcd201
Gerrit-PatchSet: 18
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: Tidy Bot
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-HasComments: Yes

[kudu-CR] [security] load/store public TSK in the system table

Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change.

Change subject: [security] load/store public TSK in the system table
......................................................................


Patch Set 23:

> (2 comments)
 > 
 > I took the liberty of updating Alexey's patch to fix my two
 > comments. Haven't done a full re-review, just wanted to fix these
 > as they were blocking my work on the Java side.

yup, those were typos -- I should have inspected the code after remove the 'while' cycle.  Thank you for fixing those.  I think that was the reason for some test breakage.

-- 
To view, visit http://gerrit.cloudera.org:8080/5935
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ie91d4129bda0ca49e81988c28385895a2abcd201
Gerrit-PatchSet: 23
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-HasComments: No

[kudu-CR] [security] load/store public TSK in the system table

Posted by "Adar Dembo (Code Review)" <ge...@cloudera.org>.
Adar Dembo has posted comments on this change.

Change subject: [security] load/store public TSK in the system table
......................................................................


Patch Set 23:

(5 comments)

> Looks like we're triggering another shutdown deadlock like
 > https://issues.apache.org/jira/browse/KUDU-1863. Perhaps we should
 > just switch over the master replication itest to use External
 > MiniCluster instead of the internal one, to avoid having to deal
 > with the shutdown races.

Back when I was doing more multi-master work, I also observed that master_replication-itest was the only multi-master test to use an internal MiniCluster, but I drew the opposite conclusion: I thought this difference in coverage was valuable (though at-times frustrating). I have some fears about switching it:
1. The timing of an internal MiniCluster is somewhat different than an external one, such that it can expose different races.
2. A working shutdown path itself may be useful if/when we implement "graceful shutdown" functionality, and if we lose multi-master shutdown coverage, it may continue to bitrot such that when we want to enable it, we'll face a much longer uphill climb.

I'm open to being convinced, but if action is going to be taken, I suggest it is done outside of this (already long) code review.

http://gerrit.cloudera.org:8080/#/c/5935/23/src/kudu/integration-tests/registration-test.cc
File src/kudu/integration-tests/registration-test.cc:

Line 104:       SleepFor(MonoDelta::FromMilliseconds(1));
Why move this up? Can't we optimistically assume that the first call (without sleep) may succeed?


PS23, Line 111:         if (!ls.ok()) {
              :           if (ls.IsServiceUnavailable()) {
              :             // ServiceUnavailable means catalog manager is not yet ready
              :             // to serve requests -- try again at a later time.
              :             continue;
              :           }
              :           return ls;
              :         }
FWIW, I find the following more idiomatic:

  if (ls.IsServiceUnavailable() {
    continue;
  }
  RETURN_NOT_OK(ls);

Less nesting too.


http://gerrit.cloudera.org:8080/#/c/5935/13/src/kudu/master/catalog_manager.cc
File src/kudu/master/catalog_manager.cc:

PS13, Line 3281: SysTskEntryPB sys_entry;
               :     sys_entry.mutable_tsk()->Swap
> Unfortunately there is no assign for std::map, and inserting elements one-b
There's no std::map here, but your point is well-taken: there's no std::set::assign().

Would you find std::set::insert() more agreeable? I think there's a variant that allows range insertion, so you could do expired_entry_ids->insert(...begin(), ...end()). Could precede it with a expired_entry_ids->clear() if you're worried about the caller passing in a populated set.


http://gerrit.cloudera.org:8080/#/c/5935/23/src/kudu/master/catalog_manager.cc
File src/kudu/master/catalog_manager.cc:

PS23, Line 212: using security::TokenSigner;
              : using security::TokenSigningPrivateKey;
              : using security::TokenSigningPrivateKeyPB;
Nit: sort order (with other using statements).


Line 3293:   leader_lock_.AssertAcquiredForReading();
> I know you and Adar talked about this a bit on Friday on slack, but this se
Yes, we discussed it, but I think we came to two different conclusions. :)

I originally suggested to Alexey to assert on the lock here, without noticing that it was called from both a read and a write context. When Alexey pointed that out, I thought something like RWMutex::AssertAcquiredForReadOrWrite wasn't worth the complexity, and suggested that Alexey just ignore my feedback.

So, I'm not an advocate for this new approach, which I also think has a high "jarring to helping" ratio. I think dropping the assertion altogether is fine. AssertAcquiredForReadOrWrite can be done, but it's annoying to have to do it for such a minor use case.


-- 
To view, visit http://gerrit.cloudera.org:8080/5935
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ie91d4129bda0ca49e81988c28385895a2abcd201
Gerrit-PatchSet: 23
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-HasComments: Yes

[kudu-CR] [security] load/store public TSK in the system table

Posted by "Adar Dembo (Code Review)" <ge...@cloudera.org>.
Adar Dembo has posted comments on this change.

Change subject: [security] load/store public TSK in the system table
......................................................................


Patch Set 24: Code-Review+1

Looks good to me, but I'll defer to Todd since he had more substantive feedback in the last round.

-- 
To view, visit http://gerrit.cloudera.org:8080/5935
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ie91d4129bda0ca49e81988c28385895a2abcd201
Gerrit-PatchSet: 24
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-HasComments: No

[kudu-CR] [security] load/store public TSK in the system table

Posted by "Todd Lipcon (Code Review)" <ge...@cloudera.org>.
Todd Lipcon has posted comments on this change.

Change subject: [security] load/store public TSK in the system table
......................................................................


Patch Set 24: Code-Review+2

-- 
To view, visit http://gerrit.cloudera.org:8080/5935
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ie91d4129bda0ca49e81988c28385895a2abcd201
Gerrit-PatchSet: 24
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-HasComments: No

[kudu-CR] [security] load/store public TSK in the system table

Posted by "Todd Lipcon (Code Review)" <ge...@cloudera.org>.
Todd Lipcon has posted comments on this change.

Change subject: [security] load/store public TSK in the system table
......................................................................


Patch Set 13:

(6 comments)

http://gerrit.cloudera.org:8080/#/c/5935/13/src/kudu/integration-tests/token_signer-itest.cc
File src/kudu/integration-tests/token_signer-itest.cc:

PS13, Line 148: public part of TSK
              : // is persisted in the system catalog table
isn't the whole TSK persisted now?


http://gerrit.cloudera.org:8080/#/c/5935/13/src/kudu/master/catalog_manager.cc
File src/kudu/master/catalog_manager.cc:

Line 438:     do {
hrm, I think it would be clearer to pull this inner block into a function like CatalogManagerBgTasks::RunOneIteration or something


PS13, Line 454:             LOG(ERROR) << "Error processing TSK entry, "
              :                           "aborting the current task: " << s.ToString();
              :  
does this mean if we start getting a TSK-related error then we won't do any tablet processing? that seems like an unnecessary coupling, and we could instead just WARN and continue here on to doing the pending assignments, etc


PS13, Line 847:       CHECK_OK(CheckGenerateNewTsk());
              :       CHECK_OK(DeleteTskEntries(expired_entry_ids));
isn't it possible that we would lose leadership here? eg we propose the writes to the syscatalog and then they fail, wouldn't it be safe enough to just skip it? (under the assumption that we're about to become non-leader again)


http://gerrit.cloudera.org:8080/#/c/5935/13/src/kudu/master/sys_catalog.cc
File src/kudu/master/sys_catalog.cc:

PS13, Line 522: et
kEntryType


http://gerrit.cloudera.org:8080/#/c/5935/13/src/kudu/master/sys_catalog.h
File src/kudu/master/sys_catalog.h:

PS13, Line 92:   // Length of the TSK entry identifier in the system catalog: all keys have
             :   // the same length and are string representation of a number in decimal
             :   // notation padded with zeroes in the beginning. This is to have natural
             :   // ordering of the records while doing scan of TSK entries.
             :   static const int kSysTskEntryKeySize = 20;
hrm, why not use big-endian int64? is this more convenient? (no need to change it, just curious)


-- 
To view, visit http://gerrit.cloudera.org:8080/5935
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ie91d4129bda0ca49e81988c28385895a2abcd201
Gerrit-PatchSet: 13
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-HasComments: Yes

[kudu-CR] [security] load/store public TSK in the system table

Posted by "Todd Lipcon (Code Review)" <ge...@cloudera.org>.
Hello Dan Burkert, Alexey Serbin,

I'd like you to reexamine a change.  Please visit

    http://gerrit.cloudera.org:8080/5935

to look at the new patch set (#23).

Change subject: [security] load/store public TSK in the system table
......................................................................

[security] load/store public TSK in the system table

Non-expired, already existing TSK records are read from the system
catalog table upon the 'elected-as-leader' callback and fed into the
TokenSigner.  New token signing keys are generated and written
into the system catalog table by a periodic task.

The expired entries are removed from the system table
upon the 'elected-as-leader' callback.

Added integration tests for cluster start/restart and switching
master leadership role into token_signer-itest.cc.

Change-Id: Ie91d4129bda0ca49e81988c28385895a2abcd201
---
M src/kudu/client/client-test.cc
M src/kudu/integration-tests/CMakeLists.txt
M src/kudu/integration-tests/registration-test.cc
A src/kudu/integration-tests/token_signer-itest.cc
M src/kudu/integration-tests/ts_itest-base.h
M src/kudu/master/catalog_manager.cc
M src/kudu/master/catalog_manager.h
M src/kudu/master/master.cc
M src/kudu/master/master.proto
M src/kudu/master/sys_catalog.cc
M src/kudu/master/sys_catalog.h
11 files changed, 522 insertions(+), 49 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/35/5935/23
-- 
To view, visit http://gerrit.cloudera.org:8080/5935
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie91d4129bda0ca49e81988c28385895a2abcd201
Gerrit-PatchSet: 23
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: Tidy Bot
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>

[kudu-CR] [security] load/store public TSK in the system table

Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change.

Change subject: [security] load/store public TSK in the system table
......................................................................


Patch Set 13:

(4 comments)

http://gerrit.cloudera.org:8080/#/c/5935/14/src/kudu/integration-tests/token_signer-itest.cc
File src/kudu/integration-tests/token_signer-itest.cc:

Line 68:   virtual void SetUp() OVERRIDE {
> void SetUp() override {
Done


http://gerrit.cloudera.org:8080/#/c/5935/14/src/kudu/master/catalog_manager.cc
File src/kudu/master/catalog_manager.cc:

PS14, Line 324: Public
> Should this be private key now?
Done


http://gerrit.cloudera.org:8080/#/c/5935/13/src/kudu/master/sys_catalog.h
File src/kudu/master/sys_catalog.h:

PS13, Line 92:   // Length of the TSK entry identifier in the system catalog: all keys have
             :   // the same length and are string representation of a number in decimal
             :   // notation padded with zeroes in the beginning. This is to have natural
             :   // ordering of the records while doing scan of TSK entries.
             :   static const int kSysTskEntryKeySize = 20;
> Agreed, this could easily use the key encoder / decoder to do this instead 
Done


PS13, Line 92:   // Length of the TSK entry identifier in the system catalog: all keys have
             :   // the same length and are string representation of a number in decimal
             :   // notation padded with zeroes in the beginning. This is to have natural
             :   // ordering of the records while doing scan of TSK entries.
             :   static const int kSysTskEntryKeySize = 20;
> It has the added benefit of handling negatives correctly, and allowing us t
Done


-- 
To view, visit http://gerrit.cloudera.org:8080/5935
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ie91d4129bda0ca49e81988c28385895a2abcd201
Gerrit-PatchSet: 13
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-HasComments: Yes

[kudu-CR] [security] load/store public TSK in the system table

Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change.

Change subject: [security] load/store public TSK in the system table
......................................................................


Patch Set 22: Verified+1

Unrelated failure:

org.apache.kudu.client.NonRecoverableException: Table startswith already exists with id ca83c01aa54944dcbacd9a3af278a97e

-- 
To view, visit http://gerrit.cloudera.org:8080/5935
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ie91d4129bda0ca49e81988c28385895a2abcd201
Gerrit-PatchSet: 22
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-HasComments: No

[kudu-CR] [security] load/store public TSK in the system table

Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Hello Kudu Jenkins,

I'd like you to reexamine a change.  Please visit

    http://gerrit.cloudera.org:8080/5935

to look at the new patch set (#21).

Change subject: [security] load/store public TSK in the system table
......................................................................

[security] load/store public TSK in the system table

Non-expired, already existing TSK records are read from the system
catalog table upon the 'elected-as-leader' callback and fed into the
TokenSigner.  New token signing keys are generated and written
into the system catalog table by a periodic task.

The expired entries are removed from the system table
upon the 'elected-as-leader' callback.

Added integration tests for cluster start/restart and switching
master leadership role into token_signer-itest.cc.

Change-Id: Ie91d4129bda0ca49e81988c28385895a2abcd201
---
M src/kudu/client/client-test.cc
M src/kudu/integration-tests/CMakeLists.txt
M src/kudu/integration-tests/registration-test.cc
A src/kudu/integration-tests/token_signer-itest.cc
M src/kudu/integration-tests/ts_itest-base.h
M src/kudu/master/catalog_manager.cc
M src/kudu/master/catalog_manager.h
M src/kudu/master/master.cc
M src/kudu/master/master.proto
M src/kudu/master/sys_catalog.cc
M src/kudu/master/sys_catalog.h
11 files changed, 511 insertions(+), 41 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/35/5935/21
-- 
To view, visit http://gerrit.cloudera.org:8080/5935
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie91d4129bda0ca49e81988c28385895a2abcd201
Gerrit-PatchSet: 21
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>

[kudu-CR] [security] load/store public TSK in the system table

Posted by "Adar Dembo (Code Review)" <ge...@cloudera.org>.
Adar Dembo has posted comments on this change.

Change subject: [security] load/store public TSK in the system table
......................................................................


Patch Set 23:

> > Now that I'm seeing the TSK deletion code alongside the insertion
 > > code, I think it'd be a good idea to perform both operations in a
 > > single sys catalog write. So the flow could look something like
 > > this:
 > >
 > > <read all TSKs, keeping track of expired ones>
 > > tsk_actions.tsks_to_delete = expired_TSKs
 > > <check and generate new TSK>
 > > if new TSK needed:
 > > tsk_actions.tsks_to_add = new_TSK
 > > Write(tsk_actions)
 > >
 > > What do you think?
 > 
 > This sounds good from the performance perspective, but it
 > introduces additional coupling.  Also, imagine a situation when the
 > delete ops of the combined request succeeds, but the inserts fail. 
 > Then we could end up in a situation when there is no TSK entries in
 > the system table at all, and we have non-persisted key in run-time.
 >  The former is bad because the sequence of TSKs might be disrupted
 > when leadership switches no another master instance (its
 > TokenSigner will start with key sequence number 0 when there is TSK
 > seq_num 100500 in the system.  The latter is also not so good
 > because the system could start using non-persisted TSK, and when
 > leadership switched to another master, that valid key could no
 > longer be seen.
 > 
 > I would opt to keep it as is.

Hmm. I agree that there's more coupling, though in this case it seems warranted. The way I look at it, all of this falls under "TSK maintenance", a task done at leader-switch time. Viewed that way, combining the two operations makes sense as they're semantically related. I don't think the failure you're describing is possible; isn't the schema designed in such a way that INSERT failures (due to duplicate keys) are impossible?

Anyway, I don't feel strongly enough about it to warrant a change.

-- 
To view, visit http://gerrit.cloudera.org:8080/5935
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ie91d4129bda0ca49e81988c28385895a2abcd201
Gerrit-PatchSet: 23
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-HasComments: No

[kudu-CR] [security] load/store public TSK in the system table

Posted by "Dan Burkert (Code Review)" <ge...@cloudera.org>.
Dan Burkert has posted comments on this change.

Change subject: [security] load/store public TSK in the system table
......................................................................


Patch Set 13:

(3 comments)

http://gerrit.cloudera.org:8080/#/c/5935/14/src/kudu/integration-tests/token_signer-itest.cc
File src/kudu/integration-tests/token_signer-itest.cc:

Line 68:   virtual void SetUp() OVERRIDE {
void SetUp() override {


http://gerrit.cloudera.org:8080/#/c/5935/14/src/kudu/master/catalog_manager.cc
File src/kudu/master/catalog_manager.cc:

PS14, Line 324: Public
Should this be private key now?


http://gerrit.cloudera.org:8080/#/c/5935/13/src/kudu/master/sys_catalog.h
File src/kudu/master/sys_catalog.h:

PS13, Line 92:   // Length of the TSK entry identifier in the system catalog: all keys have
             :   // the same length and are string representation of a number in decimal
             :   // notation padded with zeroes in the beginning. This is to have natural
             :   // ordering of the records while doing scan of TSK entries.
             :   static const int kSysTskEntryKeySize = 20;
> hrm, why not use big-endian int64? is this more convenient? (no need to cha
Agreed, this could easily use the key encoder / decoder to do this instead of padding.

    int64_t key = 132;
    string buf;
    KeyEncoderTraits<DataType::INT64, string>::Encode(key, &buf);
    Slice encoded_key = buf;
    int64_t decoded_key;
    ASSERT_EQ(8, encoded_key.size());
    Status s = KeyEncoderTraits<DataType::INT64, string>::DecodeKeyPortion(&encoded_key, true, nullptr, reinterpret_cast<uint8_t*>(&decoded_key));
    ASSERT_OK(s);
    ASSERT_EQ(key, decoded_key);


-- 
To view, visit http://gerrit.cloudera.org:8080/5935
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ie91d4129bda0ca49e81988c28385895a2abcd201
Gerrit-PatchSet: 13
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-HasComments: Yes

[kudu-CR] [security] load/store public TSK in the system table

Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Hello Kudu Jenkins,

I'd like you to reexamine a change.  Please visit

    http://gerrit.cloudera.org:8080/5935

to look at the new patch set (#19).

Change subject: [security] load/store public TSK in the system table
......................................................................

[security] load/store public TSK in the system table

Non-expired, already existing TSK records are read from the system
catalog table upon the 'elected-as-leader' callback and fed into the
TokenSigner.  New token signing keys are generated and written
into the system catalog table by a periodic task.

The expired entries are removed from the system table
upon the 'elected-as-leader' callback.

Added integration tests for cluster start/restart and switching
master leadership role into token_signer-itest.cc.

Change-Id: Ie91d4129bda0ca49e81988c28385895a2abcd201
---
M src/kudu/client/client-test.cc
M src/kudu/integration-tests/CMakeLists.txt
M src/kudu/integration-tests/registration-test.cc
A src/kudu/integration-tests/token_signer-itest.cc
M src/kudu/integration-tests/ts_itest-base.h
M src/kudu/master/catalog_manager.cc
M src/kudu/master/catalog_manager.h
M src/kudu/master/master.cc
M src/kudu/master/master.proto
M src/kudu/master/sys_catalog.cc
M src/kudu/master/sys_catalog.h
11 files changed, 510 insertions(+), 41 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/35/5935/19
-- 
To view, visit http://gerrit.cloudera.org:8080/5935
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie91d4129bda0ca49e81988c28385895a2abcd201
Gerrit-PatchSet: 19
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>

[kudu-CR] [security] load/store public TSK in the system table

Posted by "Todd Lipcon (Code Review)" <ge...@cloudera.org>.
Todd Lipcon has posted comments on this change.

Change subject: [security] load/store public TSK in the system table
......................................................................


Patch Set 23:

(5 comments)

Looks like we're triggering another shutdown deadlock like https://issues.apache.org/jira/browse/KUDU-1863. Perhaps we should just switch over the master replication itest to use External MiniCluster instead of the internal one, to avoid having to deal with the shutdown races.

http://gerrit.cloudera.org:8080/#/c/5935/23/src/kudu/master/catalog_manager.cc
File src/kudu/master/catalog_manager.cc:

Line 333:   explicit TskEntryLoader(int64_t entry_expiration_seconds)
do we ever use anything here except for WallTime_Now()? probably better to just set it in the ctor instead of passing it in.


PS23, Line 452:       } else {
              :         if (l.leader_status().ok()) {
hm, why'd this change from else if to else { if { ?


PS23, Line 839: if (!s.ok()) {
              :         LOG(ERROR) << "Failed to intialize IPKI certificate info: "
              :                    << s.ToString();
              :         return;
I'm a little worried that, if one of these things fails, we'll return from this function but then be stuck in a disfunctional state (we're the leader master but the CA is not initialized, or we don't have the TSKs loaded, etc).

I wonder if we can separate the read-only things (which shouldn't ever fail and could be CHECK) vs the write-back things like generating the next TSK (which could fail due to losing leadership, and are probably OK to fail). We could move the write-only things (expiring old keys and generating a new one) outside of holding the transition lock, after setting leader_ready_term_.


Line 3293:   leader_lock_.AssertAcquiredForReading();
I know you and Adar talked about this a bit on Friday on slack, but this seems kind of ugly to me. I think a leader_lock_.AssertAcquiredForReadOrWrite would probably be better even if less specific.


http://gerrit.cloudera.org:8080/#/c/5935/23/src/kudu/master/master.cc
File src/kudu/master/master.cc:

PS23, Line 121: mananger
nit: typo


-- 
To view, visit http://gerrit.cloudera.org:8080/5935
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ie91d4129bda0ca49e81988c28385895a2abcd201
Gerrit-PatchSet: 23
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-HasComments: Yes

[kudu-CR] [security] load/store public TSK in the system table

Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change.

Change subject: [security] load/store public TSK in the system table
......................................................................


Patch Set 9:

> (34 comments)
 > 
 > I did a first pass. I think the most important question to settle
 > is whether followers need to maintain an up-to-date TSK soft state.
 > I'd prefer the answer to be "no", as it'd simplify this code a lot.

Thank you for the review.  Per our Slack discussion on kudu-security we decided that followers do not maintain up-to-date TSK soft state (i.e. no system table polling from follower masters).

That and the decision to store private part of the key in the table as well simplified the code, indeed.

-- 
To view, visit http://gerrit.cloudera.org:8080/5935
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ie91d4129bda0ca49e81988c28385895a2abcd201
Gerrit-PatchSet: 9
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot
Gerrit-HasComments: No

[kudu-CR] [security] load/store public TSK in the system table

Posted by "Todd Lipcon (Code Review)" <ge...@cloudera.org>.
Todd Lipcon has posted comments on this change.

Change subject: [security] load/store public TSK in the system table
......................................................................


Patch Set 23:

(2 comments)

I took the liberty of updating Alexey's patch to fix my two comments. Haven't done a full re-review, just wanted to fix these as they were blocking my work on the Java side.

http://gerrit.cloudera.org:8080/#/c/5935/22/src/kudu/master/catalog_manager.cc
File src/kudu/master/catalog_manager.cc:

Line 442:     {
> this lock is now being held while in Wait() which isn't desirable
done


PS22, Line 477:                   "aborting the current task: " << s.ToString();
              :             }
> isn't this now breaking out of the whole loop, which would break background
done


-- 
To view, visit http://gerrit.cloudera.org:8080/5935
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ie91d4129bda0ca49e81988c28385895a2abcd201
Gerrit-PatchSet: 23
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-HasComments: Yes

[kudu-CR] [security] load/store public TSK in the system table

Posted by "Dan Burkert (Code Review)" <ge...@cloudera.org>.
Dan Burkert has posted comments on this change.

Change subject: [security] load/store public TSK in the system table
......................................................................


Patch Set 13:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/5935/13/src/kudu/master/sys_catalog.h
File src/kudu/master/sys_catalog.h:

PS13, Line 92:   // Length of the TSK entry identifier in the system catalog: all keys have
             :   // the same length and are string representation of a number in decimal
             :   // notation padded with zeroes in the beginning. This is to have natural
             :   // ordering of the records while doing scan of TSK entries.
             :   static const int kSysTskEntryKeySize = 20;
> Agreed, this could easily use the key encoder / decoder to do this instead 
It has the added benefit of handling negatives correctly, and allowing us to add fields to the key backwards-compatibly if we ever need to.


-- 
To view, visit http://gerrit.cloudera.org:8080/5935
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ie91d4129bda0ca49e81988c28385895a2abcd201
Gerrit-PatchSet: 13
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-HasComments: Yes

[kudu-CR] WIP [security] load/store public TSK in the system table

Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Hello Kudu Jenkins,

I'd like you to reexamine a change.  Please visit

    http://gerrit.cloudera.org:8080/5935

to look at the new patch set (#4).

Change subject: WIP [security] load/store public TSK in the system table
......................................................................

WIP [security] load/store public TSK in the system table

Non-expired, already existing TSK records are read from the system
catalog table upon startup of the leader master and fed into the
TokenSigner.  Also, a new token signing key is generated on start-up
and its public part is written into the system catalog table as well.

Added integration tests for cluster start/restart and switching
master leadership role into token_signer-itest.cc.

It's a WIP because some tests are still failing -- need to troubleshoot
those breakages.

Change-Id: Ie91d4129bda0ca49e81988c28385895a2abcd201
---
M src/kudu/integration-tests/CMakeLists.txt
A src/kudu/integration-tests/token_signer-itest.cc
M src/kudu/master/authn_token_manager.cc
M src/kudu/master/catalog_manager.cc
M src/kudu/master/catalog_manager.h
M src/kudu/master/master.cc
M src/kudu/master/master.proto
M src/kudu/master/sys_catalog.cc
M src/kudu/master/sys_catalog.h
M src/kudu/security/token_signer.h
10 files changed, 578 insertions(+), 46 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/35/5935/4
-- 
To view, visit http://gerrit.cloudera.org:8080/5935
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie91d4129bda0ca49e81988c28385895a2abcd201
Gerrit-PatchSet: 4
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot

[kudu-CR] [security] load/store public TSK in the system table

Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Hello Kudu Jenkins,

I'd like you to reexamine a change.  Please visit

    http://gerrit.cloudera.org:8080/5935

to look at the new patch set (#11).

Change subject: [security] load/store public TSK in the system table
......................................................................

[security] load/store public TSK in the system table

Non-expired, already existing TSK records are read from the system
catalog table upon the 'elected-as-leader' callback and fed into the
TokenSigner.  New token signing keys are generated and written
into the system catalog table by a periodic task.

The expired entries are removed from the system table
upon the 'elected-as-leader' callback.

Added integration tests for cluster start/restart and switching
master leadership role into token_signer-itest.cc.

Change-Id: Ie91d4129bda0ca49e81988c28385895a2abcd201
---
M src/kudu/client/client-test.cc
M src/kudu/integration-tests/CMakeLists.txt
A src/kudu/integration-tests/token_signer-itest.cc
M src/kudu/master/catalog_manager.cc
M src/kudu/master/catalog_manager.h
M src/kudu/master/master.cc
M src/kudu/master/master.proto
M src/kudu/master/sys_catalog.cc
M src/kudu/master/sys_catalog.h
9 files changed, 461 insertions(+), 35 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/35/5935/11
-- 
To view, visit http://gerrit.cloudera.org:8080/5935
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie91d4129bda0ca49e81988c28385895a2abcd201
Gerrit-PatchSet: 11
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot

[kudu-CR] [security] load/store public TSK in the system table

Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Hello Kudu Jenkins,

I'd like you to reexamine a change.  Please visit

    http://gerrit.cloudera.org:8080/5935

to look at the new patch set (#8).

Change subject: [security] load/store public TSK in the system table
......................................................................

[security] load/store public TSK in the system table

Non-expired, already existing TSK records are read from the system
catalog table upon the 'elected-as-leader' callback and fed into the
TokenSigner.  New token signing keys are generated and written
into the system catalog table by a periodic task.

The expired entries are removed from the system table
upon the 'elected-as-leader' callback.

Added integration tests for cluster start/restart and switching
master leadership role into token_signer-itest.cc.

Change-Id: Ie91d4129bda0ca49e81988c28385895a2abcd201
---
M src/kudu/client/client-test.cc
M src/kudu/integration-tests/CMakeLists.txt
A src/kudu/integration-tests/token_signer-itest.cc
M src/kudu/master/catalog_manager.cc
M src/kudu/master/catalog_manager.h
M src/kudu/master/master.cc
M src/kudu/master/master.proto
M src/kudu/master/sys_catalog.cc
M src/kudu/master/sys_catalog.h
9 files changed, 472 insertions(+), 37 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/35/5935/8
-- 
To view, visit http://gerrit.cloudera.org:8080/5935
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie91d4129bda0ca49e81988c28385895a2abcd201
Gerrit-PatchSet: 8
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot

[kudu-CR] [security] load/store public TSK in the system table

Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Hello Kudu Jenkins,

I'd like you to reexamine a change.  Please visit

    http://gerrit.cloudera.org:8080/5935

to look at the new patch set (#24).

Change subject: [security] load/store public TSK in the system table
......................................................................

[security] load/store public TSK in the system table

Non-expired, already existing TSK records are read from the system
catalog table upon the 'elected-as-leader' callback and fed into the
TokenSigner.  New token signing keys are generated and written
into the system catalog table by a periodic task.

The expired entries are removed from the system table
upon the 'elected-as-leader' callback.

Added integration tests for cluster start/restart and switching
master leadership role into token_signer-itest.cc.

Change-Id: Ie91d4129bda0ca49e81988c28385895a2abcd201
---
M src/kudu/client/client-test.cc
M src/kudu/integration-tests/CMakeLists.txt
M src/kudu/integration-tests/registration-test.cc
A src/kudu/integration-tests/token_signer-itest.cc
M src/kudu/integration-tests/ts_itest-base.h
M src/kudu/master/catalog_manager.cc
M src/kudu/master/catalog_manager.h
M src/kudu/master/master.cc
M src/kudu/master/master.proto
M src/kudu/master/sys_catalog.cc
M src/kudu/master/sys_catalog.h
11 files changed, 587 insertions(+), 42 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/35/5935/24
-- 
To view, visit http://gerrit.cloudera.org:8080/5935
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie91d4129bda0ca49e81988c28385895a2abcd201
Gerrit-PatchSet: 24
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>

[kudu-CR] WIP [security] load/store public TSK in the system table

Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Hello Kudu Jenkins,

I'd like you to reexamine a change.  Please visit

    http://gerrit.cloudera.org:8080/5935

to look at the new patch set (#2).

Change subject: WIP [security] load/store public TSK in the system table
......................................................................

WIP [security] load/store public TSK in the system table

Non-expired, already existing TSK records are read from the system
catalog table upon startup of the leader master and fed into the
TokenSigner.  Also, a new token signing key is generated on start-up
and its public part is written into the system catalog table as well.

Added integration tests for cluster start/restart and switching
master leadership role into token_signer-itest.cc.

Change-Id: Ie91d4129bda0ca49e81988c28385895a2abcd201
---
M src/kudu/integration-tests/CMakeLists.txt
A src/kudu/integration-tests/token_signer-itest.cc
M src/kudu/master/catalog_manager.cc
M src/kudu/master/catalog_manager.h
M src/kudu/master/master.cc
M src/kudu/master/master.proto
M src/kudu/master/sys_catalog.cc
M src/kudu/master/sys_catalog.h
M src/kudu/security/token_signer.h
9 files changed, 602 insertions(+), 26 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/35/5935/2
-- 
To view, visit http://gerrit.cloudera.org:8080/5935
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie91d4129bda0ca49e81988c28385895a2abcd201
Gerrit-PatchSet: 2
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot

[kudu-CR] [security] load/store public TSK in the system table

Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Hello Kudu Jenkins,

I'd like you to reexamine a change.  Please visit

    http://gerrit.cloudera.org:8080/5935

to look at the new patch set (#15).

Change subject: [security] load/store public TSK in the system table
......................................................................

[security] load/store public TSK in the system table

Non-expired, already existing TSK records are read from the system
catalog table upon the 'elected-as-leader' callback and fed into the
TokenSigner.  New token signing keys are generated and written
into the system catalog table by a periodic task.

The expired entries are removed from the system table
upon the 'elected-as-leader' callback.

Added integration tests for cluster start/restart and switching
master leadership role into token_signer-itest.cc.

Change-Id: Ie91d4129bda0ca49e81988c28385895a2abcd201
---
M src/kudu/client/client-test.cc
M src/kudu/integration-tests/CMakeLists.txt
A src/kudu/integration-tests/token_signer-itest.cc
M src/kudu/master/catalog_manager.cc
M src/kudu/master/catalog_manager.h
M src/kudu/master/master.cc
M src/kudu/master/master.proto
M src/kudu/master/sys_catalog.cc
M src/kudu/master/sys_catalog.h
9 files changed, 494 insertions(+), 32 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/35/5935/15
-- 
To view, visit http://gerrit.cloudera.org:8080/5935
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie91d4129bda0ca49e81988c28385895a2abcd201
Gerrit-PatchSet: 15
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>

[kudu-CR] [security] load/store public TSK in the system table

Posted by "Adar Dembo (Code Review)" <ge...@cloudera.org>.
Adar Dembo has posted comments on this change.

Change subject: [security] load/store public TSK in the system table
......................................................................


Patch Set 13:

(4 comments)

Now that I'm seeing the TSK deletion code alongside the insertion code, I think it'd be a good idea to perform both operations in a single sys catalog write. So the flow could look something like this:

  <read all TSKs, keeping track of expired ones>
  tsk_actions.tsks_to_delete = expired_TSKs
  <check and generate new TSK>
  if new TSK needed:
    tsk_actions.tsks_to_add = new_TSK
  Write(tsk_actions)

What do you think?

http://gerrit.cloudera.org:8080/#/c/5935/13/src/kudu/integration-tests/token_signer-itest.cc
File src/kudu/integration-tests/token_signer-itest.cc:

Line 22: #include <gflags/gflags_declare.h>
Nit: should precede gtest.


http://gerrit.cloudera.org:8080/#/c/5935/13/src/kudu/master/catalog_manager.cc
File src/kudu/master/catalog_manager.cc:

PS13, Line 847:       CHECK_OK(CheckGenerateNewTsk());
              :       CHECK_OK(DeleteTskEntries(expired_entry_ids));
> isn't it possible that we would lose leadership here? eg we propose the wri
Agreed with Todd: the reason CHECK_OK() is reasonable on VisitTablesAndTabletsUnlocked() is because that's a read-only operation, so even if this master loses leadership, it should succeed. But these operations (and CheckInitCertAuthority() too, right?) may write, and write failures shouldn't crash the master.


Line 3255: Status CatalogManager::CheckGenerateNewTsk() {
Should assert that the leader_lock_ is held here.


PS13, Line 3281: set<string> ref(loader.expired_entry_ids());
               :     expired_entry_ids->swap(ref);
Can't this be combined?

  expired_entry_ids->assign(loader.expired_entry_ids().begin(),
                            loader.expired_entry_ids().end());

Something like that?

Alternatively, you can declare an empty set locally, pass it by pointer to TskEntryLoader, call VisitTskEntries(), and swap the set directly with expired_entry_ids.


-- 
To view, visit http://gerrit.cloudera.org:8080/5935
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ie91d4129bda0ca49e81988c28385895a2abcd201
Gerrit-PatchSet: 13
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-HasComments: Yes

[kudu-CR] [security] load/store public TSK in the system table

Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change.

Change subject: [security] load/store public TSK in the system table
......................................................................


Patch Set 13:

(10 comments)

> (4 comments)
 > 
 > Now that I'm seeing the TSK deletion code alongside the insertion
 > code, I think it'd be a good idea to perform both operations in a
 > single sys catalog write. So the flow could look something like
 > this:
 > 
 > <read all TSKs, keeping track of expired ones>
 > tsk_actions.tsks_to_delete = expired_TSKs
 > <check and generate new TSK>
 > if new TSK needed:
 > tsk_actions.tsks_to_add = new_TSK
 > Write(tsk_actions)
 > 
 > What do you think?

This sounds good from the performance perspective, but it introduces additional coupling.  Also, imagine a situation when the delete ops of the combined request succeeds, but the inserts fail.  Then we could end up in a situation when there is no TSK entries in the system table at all, and we have non-persisted key in run-time.  The former is bad because the sequence of TSKs might be disrupted when leadership switches no another master instance (its TokenSigner will start with key sequence number 0 when there is TSK seq_num 100500 in the system.  The latter is also not so good because the system could start using non-persisted TSK, and when leadership switched to another master, that valid key could no longer be seen.

I would opt to keep it as is.

http://gerrit.cloudera.org:8080/#/c/5935/13/src/kudu/integration-tests/token_signer-itest.cc
File src/kudu/integration-tests/token_signer-itest.cc:

Line 22: #include <gflags/gflags_declare.h>
> Nit: should precede gtest.
Done


PS13, Line 148: public part of TSK
              : // is persisted in the system catalog table
> isn't the whole TSK persisted now?
Done


http://gerrit.cloudera.org:8080/#/c/5935/13/src/kudu/master/catalog_manager.cc
File src/kudu/master/catalog_manager.cc:

Line 438:     do {
> hrm, I think it would be clearer to pull this inner block into a function l
This is no longer needed as soon as the comment below is addressed.


PS13, Line 454:             LOG(ERROR) << "Error processing TSK entry, "
              :                           "aborting the current task: " << s.ToString();
              :  
> does this mean if we start getting a TSK-related error then we won't do any
Yes, it sounds reasonable to continue here in case of an error.  For some reason I tried to preserve the same pattern as for the tablet metadata processing.


PS13, Line 847:       CHECK_OK(CheckGenerateNewTsk());
              :       CHECK_OK(DeleteTskEntries(expired_entry_ids));
> Agreed with Todd: the reason CHECK_OK() is reasonable on VisitTablesAndTabl
Thank you for the clarification: now I understand the reasoning for CHECK_OK() usage at VisitTablesAndTablets() callsite.

Yep, CheckInitAuthority() also does similar things.  I'll update this accordingly.


PS13, Line 847:       CHECK_OK(CheckGenerateNewTsk());
              :       CHECK_OK(DeleteTskEntries(expired_entry_ids));
> isn't it possible that we would lose leadership here? eg we propose the wri
It's a good observation.


Line 3255: Status CatalogManager::CheckGenerateNewTsk() {
> Should assert that the leader_lock_ is held here.
Done


PS13, Line 3281: set<string> ref(loader.expired_entry_ids());
               :     expired_entry_ids->swap(ref);
> Can't this be combined?
Unfortunately there is no assign for std::map, and inserting elements one-by-one is costly for tree maps.

The latter variant does not look good to me: it implies you pass its constructor some sink where one of class methods writes.  Does not smell good enough to me.

Probably, using vectors instead of sets might be an option, but then it would be necessary to sort them if we want to remove duplicates.

I would leave this as is for a while.


http://gerrit.cloudera.org:8080/#/c/5935/13/src/kudu/master/sys_catalog.cc
File src/kudu/master/sys_catalog.cc:

PS13, Line 522: et
> kEntryType
Done


http://gerrit.cloudera.org:8080/#/c/5935/13/src/kudu/master/sys_catalog.h
File src/kudu/master/sys_catalog.h:

PS13, Line 92:   // Length of the TSK entry identifier in the system catalog: all keys have
             :   // the same length and are string representation of a number in decimal
             :   // notation padded with zeroes in the beginning. This is to have natural
             :   // ordering of the records while doing scan of TSK entries.
             :   static const int kSysTskEntryKeySize = 20;
> hrm, why not use big-endian int64? is this more convenient? (no need to cha
I'm not sure I follow: you mean just print 8 bytes of 64-bit number as they are (in hexadecimal)?  Yep, that would work as well.  I thought having them in decimal would be easier for comprehension if ever looking at them.

The idea was to have entry_id identifier which would allow ordered scans over those entries (in ascending order), but now it's not crucial.


-- 
To view, visit http://gerrit.cloudera.org:8080/5935
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ie91d4129bda0ca49e81988c28385895a2abcd201
Gerrit-PatchSet: 13
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-HasComments: Yes

[kudu-CR] [security] load/store public TSK in the system table

Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change.

Change subject: [security] load/store public TSK in the system table
......................................................................


Patch Set 23:

(10 comments)

http://gerrit.cloudera.org:8080/#/c/5935/23/src/kudu/integration-tests/registration-test.cc
File src/kudu/integration-tests/registration-test.cc:

Line 104:       SleepFor(MonoDelta::FromMilliseconds(1));
> Why move this up? Can't we optimistically assume that the first call (witho
It seemed to me the easiest way to amend the code to handle ServiceUnavailable.  I agree it's not the best way.  I'll fix that.


PS23, Line 111:         if (!ls.ok()) {
              :           if (ls.IsServiceUnavailable()) {
              :             // ServiceUnavailable means catalog manager is not yet ready
              :             // to serve requests -- try again at a later time.
              :             continue;
              :           }
              :           return ls;
              :         }
> FWIW, I find the following more idiomatic:
That's much better, thanks.


http://gerrit.cloudera.org:8080/#/c/5935/23/src/kudu/master/catalog_manager.cc
File src/kudu/master/catalog_manager.cc:

PS23, Line 212: using security::TokenSigner;
              : using security::TokenSigningPrivateKey;
              : using security::TokenSigningPrivateKeyPB;
> Nit: sort order (with other using statements).
Done


Line 333:   explicit TskEntryLoader(int64_t entry_expiration_seconds)
> do we ever use anything here except for WallTime_Now()? probably better to 
Good idea.


PS23, Line 452:       } else {
              :         if (l.leader_status().ok()) {
> hm, why'd this change from else if to else { if { ?
uh-oh -- that's just another artifact from my previous change (with do { ...}  while() cycle) which has been rolled back.

Good catch -- I will fix this.


PS23, Line 839: if (!s.ok()) {
              :         LOG(ERROR) << "Failed to intialize IPKI certificate info: "
              :                    << s.ToString();
              :         return;
> I'm a little worried that, if one of these things fails, we'll return from 
I separated the loading and generation part.

The interesting part of this process is that it's necessary to make sure the generated entries are written prior to loading  those into memory.  If doing so, I'm not quite sure it's safe to do so outside of the transition lock.


Line 3293:   leader_lock_.AssertAcquiredForReading();
> Yes, we discussed it, but I think we came to two different conclusions. :)
Done


Line 3293:   leader_lock_.AssertAcquiredForReading();
> I know you and Adar talked about this a bit on Friday on slack, but this se
All right, decided to drop the assertions.  It looks ugly, indeed.


Line 3293:   leader_lock_.AssertAcquiredForReading();
> That's fair, I'm also fine with dropping it and relying on TSAN to check fo
Done


http://gerrit.cloudera.org:8080/#/c/5935/23/src/kudu/master/master.cc
File src/kudu/master/master.cc:

PS23, Line 121: mananger
> nit: typo
Done


-- 
To view, visit http://gerrit.cloudera.org:8080/5935
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ie91d4129bda0ca49e81988c28385895a2abcd201
Gerrit-PatchSet: 23
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-HasComments: Yes

[kudu-CR] WIP [security] load/store public TSK in the system table

Posted by "Adar Dembo (Code Review)" <ge...@cloudera.org>.
Adar Dembo has posted comments on this change.

Change subject: WIP [security] load/store public TSK in the system table
......................................................................


Patch Set 5:

(34 comments)

I did a first pass. I think the most important question to settle is whether followers need to maintain an up-to-date TSK soft state. I'd prefer the answer to be "no", as it'd simplify this code a lot.

http://gerrit.cloudera.org:8080/#/c/5935/5/src/kudu/client/client-internal.cc
File src/kudu/client/client-internal.cc:

Line 192:     if (s.IsNetworkError() || s.IsServiceUnavailable()) {
Probably doesn't belong in this patch, right? I think you put this in a different one?


http://gerrit.cloudera.org:8080/#/c/5935/5/src/kudu/integration-tests/token_signer-itest.cc
File src/kudu/integration-tests/token_signer-itest.cc:

Line 85:   SignedTokenPB MakeToken() {
Declare as static?


Line 125:   gscoped_ptr<MiniCluster> cluster_;
unique_ptr


Line 141:   SleepFor(MonoDelta::FromSeconds(1));
Why is this needed?


PS5, Line 145: have
has


PS5, Line 145: folower
follower


http://gerrit.cloudera.org:8080/#/c/5935/5/src/kudu/master/catalog_manager.cc
File src/kudu/master/catalog_manager.cc:

Line 204: DEFINE_int64(catalog_manager_sys_table_propagation_seconds, 60,
Not really sure what this means.


Line 208: // TODO(PKI): add flag tags
Indeed.


Line 342:     CHECK(token_signer_);
could use CHECK_NOTNULL in the initializer list above.


Line 345:   // The entry_id corresponds to the sequence number of the TSK key.
But we're ignoring the entry_id, so why is this comment useful?


Line 359:     LOG(INFO) << "Imported TSPK " << tspk.key_seq_num() << " from system table";
This may be too noisy. How many non-expired entries would we typically expect?


Line 365:   const set<int64_t>& loaded_key_seq_numbers() const {
We only ever get the first number, so change this method to expose that rather than the entire set.

Alternatively, track the earliest one explicitly (i.e. initialize it to -1, use std::min() or whatever to update it with each visit) and forgo the set entirely.


PS5, Line 371:  // Epoch time
What does this mean?


PS5, Line 469: firther
further


PS5, Line 489: firther
further


Line 497:           catalog_manager_->LoadTskEntries(&last_seen_tsk_entry_id);
This runs every second. Isn't that super wasteful?


Line 500:     } while (false);
Why this change? We can have an extra scope (for lock acquisition/release) without a do/while.


PS5, Line 3269: There's a bit of subtlety here:
Yeah, this is complicated. I'll take another look at this later; maybe ask Dan or Todd to review it too.


Line 3290:       RETURN_NOT_OK(signer->GenerateSigningKey(&next_tsk_));
Why are we writing to next_tsk_ before persisting, then clearing it on failure? Why not just write to a local variable and promote to next_tsk_ on success? Seems more straight-forward.


Line 3296:       const Status s = sys_catalog_->AddTskEntries({sys_entry});
Can we use security::TokenSigningPublicKeyPB in the interface to this function, rather than SysTskEntryPB?

Also, if we only ever add one at a time, let's not use a vector-based interface.


Line 3320:   TskEntryLoader loader(signer, WallTime_Now());
A certain amount of imprecision is OK here, right? Masters' clocks may not be perfectly synchronized, so on leader election it's possible that a TSK can either "lose" or "gain" some expiration time, depending on the clock difference between the previous leader master and the new one. That's OK, right?


Line 3322:   const set<int64_t> seq_numbers(loader.loaded_key_seq_numbers());
Could be a cref.


http://gerrit.cloudera.org:8080/#/c/5935/5/src/kudu/master/catalog_manager.h
File src/kudu/master/catalog_manager.h:

Line 565:   //Status LoadTokenSigningInfo(std::vector<SysTskEntryPB>* new_entries);
Why is this commented out?


PS5, Line 773: pubic
public


PS5, Line 773: follower master
follower masters


http://gerrit.cloudera.org:8080/#/c/5935/5/src/kudu/master/master.proto
File src/kudu/master/master.proto:

Line 200: // The on-disk entry in the sys.catalog table ("metadata" column) to represent
This isn't used (and is commented out); can it be removed? I think it'd be simpler to figure out the "newest" SysTskEntryPB by scanning all of them; storing a pointer to the latest feels like a premature optimization.


Line 204: // be single entry of this type in the sys.catalog table. 
Trailing whitespace here.


http://gerrit.cloudera.org:8080/#/c/5935/5/src/kudu/master/sys_catalog.cc
File src/kudu/master/sys_catalog.cc:

Line 478: // TSK-related methods
This isn't using the same format as the other boundaries, plus it's splitting the "table-related methods" section into multiple pieces.


Line 490: void SysCatalogTable::ReqAddTskEntry(WriteRequestPB* req,
Could be inlined into AddTskEntries(), since it's just used in that one place.


Line 511: void SysCatalogTable::ReqDeleteTskEntry(WriteRequestPB* req,
Not used?


Line 520:   CHECK_OK(row.SetStringNoCopy(
Is NoCopy() correct? Doesn't the string from TskSeqNumberToEntryId() go out of scope the moment the call completes?


Line 568:   unique_ptr<Arena> range_arena;
Can't reuse the arena below?


Line 576:     const string key_upper_boundary = key_low_boundary + ".";
Confused by this; if we want to scan from key_low_boundary and up, shouldn't we just pass 'nullptr' as upper?

Also, key_util:: has methods for incrementing cell values.


http://gerrit.cloudera.org:8080/#/c/5935/5/src/kudu/master/sys_catalog.h
File src/kudu/master/sys_catalog.h:

PS5, Line 148: If the 'low_entry_id'
             :   // is not empty, it is used as the scan's exclusive low boundary for the
             :   // 'entry_id' column.
This feels premature. The number of TSK rows is pretty low, right? Can't we just scan them all and filter out the ones we don't care about.


-- 
To view, visit http://gerrit.cloudera.org:8080/5935
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ie91d4129bda0ca49e81988c28385895a2abcd201
Gerrit-PatchSet: 5
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot
Gerrit-HasComments: Yes

[kudu-CR] [security] load/store public TSK in the system table

Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Hello Kudu Jenkins,

I'd like you to reexamine a change.  Please visit

    http://gerrit.cloudera.org:8080/5935

to look at the new patch set (#9).

Change subject: [security] load/store public TSK in the system table
......................................................................

[security] load/store public TSK in the system table

Non-expired, already existing TSK records are read from the system
catalog table upon the 'elected-as-leader' callback and fed into the
TokenSigner.  New token signing keys are generated and written
into the system catalog table by a periodic task.

The expired entries are removed from the system table
upon the 'elected-as-leader' callback.

Added integration tests for cluster start/restart and switching
master leadership role into token_signer-itest.cc.

Change-Id: Ie91d4129bda0ca49e81988c28385895a2abcd201
---
M src/kudu/client/client-test.cc
M src/kudu/integration-tests/CMakeLists.txt
A src/kudu/integration-tests/token_signer-itest.cc
M src/kudu/master/catalog_manager.cc
M src/kudu/master/catalog_manager.h
M src/kudu/master/master.cc
M src/kudu/master/master.proto
M src/kudu/master/sys_catalog.cc
M src/kudu/master/sys_catalog.h
9 files changed, 478 insertions(+), 34 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/35/5935/9
-- 
To view, visit http://gerrit.cloudera.org:8080/5935
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie91d4129bda0ca49e81988c28385895a2abcd201
Gerrit-PatchSet: 9
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot

[kudu-CR] [security] load/store public TSK in the system table

Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Hello Kudu Jenkins,

I'd like you to reexamine a change.  Please visit

    http://gerrit.cloudera.org:8080/5935

to look at the new patch set (#16).

Change subject: [security] load/store public TSK in the system table
......................................................................

[security] load/store public TSK in the system table

Non-expired, already existing TSK records are read from the system
catalog table upon the 'elected-as-leader' callback and fed into the
TokenSigner.  New token signing keys are generated and written
into the system catalog table by a periodic task.

The expired entries are removed from the system table
upon the 'elected-as-leader' callback.

Added integration tests for cluster start/restart and switching
master leadership role into token_signer-itest.cc.

Change-Id: Ie91d4129bda0ca49e81988c28385895a2abcd201
---
M src/kudu/client/client-test.cc
M src/kudu/integration-tests/CMakeLists.txt
A src/kudu/integration-tests/token_signer-itest.cc
M src/kudu/master/catalog_manager.cc
M src/kudu/master/catalog_manager.h
M src/kudu/master/master.cc
M src/kudu/master/master.proto
M src/kudu/master/sys_catalog.cc
M src/kudu/master/sys_catalog.h
9 files changed, 490 insertions(+), 35 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/35/5935/16
-- 
To view, visit http://gerrit.cloudera.org:8080/5935
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie91d4129bda0ca49e81988c28385895a2abcd201
Gerrit-PatchSet: 16
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>

[kudu-CR] [security] load/store public TSK in the system table

Posted by "Todd Lipcon (Code Review)" <ge...@cloudera.org>.
Todd Lipcon has posted comments on this change.

Change subject: [security] load/store public TSK in the system table
......................................................................


Patch Set 24: Verified+1

Seems like the known shutdown deadlock discussed above hit us again. Overriding, will address in a follow-on.

-- 
To view, visit http://gerrit.cloudera.org:8080/5935
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ie91d4129bda0ca49e81988c28385895a2abcd201
Gerrit-PatchSet: 24
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-HasComments: No

[kudu-CR] [security] load/store public TSK in the system table

Posted by "Todd Lipcon (Code Review)" <ge...@cloudera.org>.
Todd Lipcon has posted comments on this change.

Change subject: [security] load/store public TSK in the system table
......................................................................


Patch Set 23:

(1 comment)

Agreed that addressing the shutdown deadlock should be done separately from this patch (I based a bunch of patches on top of this one and none of them hit the deadlock, so I dont think it's overly common anyway)

http://gerrit.cloudera.org:8080/#/c/5935/23/src/kudu/master/catalog_manager.cc
File src/kudu/master/catalog_manager.cc:

Line 3293:   leader_lock_.AssertAcquiredForReading();
> Yes, we discussed it, but I think we came to two different conclusions. :)
That's fair, I'm also fine with dropping it and relying on TSAN to check for us.


-- 
To view, visit http://gerrit.cloudera.org:8080/5935
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ie91d4129bda0ca49e81988c28385895a2abcd201
Gerrit-PatchSet: 23
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-HasComments: Yes

[kudu-CR] [security] load/store public TSK in the system table

Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Hello Kudu Jenkins,

I'd like you to reexamine a change.  Please visit

    http://gerrit.cloudera.org:8080/5935

to look at the new patch set (#20).

Change subject: [security] load/store public TSK in the system table
......................................................................

[security] load/store public TSK in the system table

Non-expired, already existing TSK records are read from the system
catalog table upon the 'elected-as-leader' callback and fed into the
TokenSigner.  New token signing keys are generated and written
into the system catalog table by a periodic task.

The expired entries are removed from the system table
upon the 'elected-as-leader' callback.

Added integration tests for cluster start/restart and switching
master leadership role into token_signer-itest.cc.

Change-Id: Ie91d4129bda0ca49e81988c28385895a2abcd201
---
M src/kudu/client/client-test.cc
M src/kudu/integration-tests/CMakeLists.txt
M src/kudu/integration-tests/registration-test.cc
A src/kudu/integration-tests/token_signer-itest.cc
M src/kudu/integration-tests/ts_itest-base.h
M src/kudu/master/catalog_manager.cc
M src/kudu/master/catalog_manager.h
M src/kudu/master/master.cc
M src/kudu/master/master.proto
M src/kudu/master/sys_catalog.cc
M src/kudu/master/sys_catalog.h
11 files changed, 510 insertions(+), 41 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/35/5935/20
-- 
To view, visit http://gerrit.cloudera.org:8080/5935
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie91d4129bda0ca49e81988c28385895a2abcd201
Gerrit-PatchSet: 20
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>

[kudu-CR] [security] load/store public TSK in the system table

Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Hello Kudu Jenkins,

I'd like you to reexamine a change.  Please visit

    http://gerrit.cloudera.org:8080/5935

to look at the new patch set (#6).

Change subject: [security] load/store public TSK in the system table
......................................................................

[security] load/store public TSK in the system table

Non-expired, already existing TSK records are read from the system
catalog table upon the 'elected-as-leader' callback and fed into the
TokenSigner.  New token signing keys are generated and written
into the system catalog table by a periodic task.

The expired entries are removed from the system table
upon the 'elected-as-leader' callback.

Added integration tests for cluster start/restart and switching
master leadership role into token_signer-itest.cc.

Change-Id: Ie91d4129bda0ca49e81988c28385895a2abcd201
---
M src/kudu/client/client-test.cc
M src/kudu/integration-tests/CMakeLists.txt
A src/kudu/integration-tests/token_signer-itest.cc
M src/kudu/master/catalog_manager.cc
M src/kudu/master/catalog_manager.h
M src/kudu/master/master.cc
M src/kudu/master/master.proto
M src/kudu/master/sys_catalog.cc
M src/kudu/master/sys_catalog.h
9 files changed, 479 insertions(+), 26 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/35/5935/6
-- 
To view, visit http://gerrit.cloudera.org:8080/5935
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie91d4129bda0ca49e81988c28385895a2abcd201
Gerrit-PatchSet: 6
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot

[kudu-CR] WIP [security] load/store public TSK in the system table

Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Hello Kudu Jenkins,

I'd like you to reexamine a change.  Please visit

    http://gerrit.cloudera.org:8080/5935

to look at the new patch set (#3).

Change subject: WIP [security] load/store public TSK in the system table
......................................................................

WIP [security] load/store public TSK in the system table

Non-expired, already existing TSK records are read from the system
catalog table upon startup of the leader master and fed into the
TokenSigner.  Also, a new token signing key is generated on start-up
and its public part is written into the system catalog table as well.

Added integration tests for cluster start/restart and switching
master leadership role into token_signer-itest.cc.

Change-Id: Ie91d4129bda0ca49e81988c28385895a2abcd201
---
M src/kudu/integration-tests/CMakeLists.txt
A src/kudu/integration-tests/token_signer-itest.cc
M src/kudu/master/catalog_manager.cc
M src/kudu/master/catalog_manager.h
M src/kudu/master/master.cc
M src/kudu/master/master.proto
M src/kudu/master/sys_catalog.cc
M src/kudu/master/sys_catalog.h
M src/kudu/security/token_signer.h
9 files changed, 570 insertions(+), 29 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/35/5935/3
-- 
To view, visit http://gerrit.cloudera.org:8080/5935
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie91d4129bda0ca49e81988c28385895a2abcd201
Gerrit-PatchSet: 3
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot

[kudu-CR] [security] load/store public TSK in the system table

Posted by "Todd Lipcon (Code Review)" <ge...@cloudera.org>.
Todd Lipcon has submitted this change and it was merged.

Change subject: [security] load/store public TSK in the system table
......................................................................


[security] load/store public TSK in the system table

Non-expired, already existing TSK records are read from the system
catalog table upon the 'elected-as-leader' callback and fed into the
TokenSigner.  New token signing keys are generated and written
into the system catalog table by a periodic task.

The expired entries are removed from the system table
upon the 'elected-as-leader' callback.

Added integration tests for cluster start/restart and switching
master leadership role into token_signer-itest.cc.

Change-Id: Ie91d4129bda0ca49e81988c28385895a2abcd201
Reviewed-on: http://gerrit.cloudera.org:8080/5935
Reviewed-by: Adar Dembo <ad...@cloudera.com>
Reviewed-by: Todd Lipcon <to...@apache.org>
Tested-by: Todd Lipcon <to...@apache.org>
---
M src/kudu/client/client-test.cc
M src/kudu/integration-tests/CMakeLists.txt
M src/kudu/integration-tests/registration-test.cc
A src/kudu/integration-tests/token_signer-itest.cc
M src/kudu/integration-tests/ts_itest-base.h
M src/kudu/master/catalog_manager.cc
M src/kudu/master/catalog_manager.h
M src/kudu/master/master.cc
M src/kudu/master/master.proto
M src/kudu/master/sys_catalog.cc
M src/kudu/master/sys_catalog.h
11 files changed, 587 insertions(+), 42 deletions(-)

Approvals:
  Adar Dembo: Looks good to me, but someone else must approve
  Todd Lipcon: Looks good to me, approved; Verified



-- 
To view, visit http://gerrit.cloudera.org:8080/5935
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Ie91d4129bda0ca49e81988c28385895a2abcd201
Gerrit-PatchSet: 25
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: Tidy Bot
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>

[kudu-CR] WIP [security] load/store public TSK in the system table

Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Hello Kudu Jenkins,

I'd like you to reexamine a change.  Please visit

    http://gerrit.cloudera.org:8080/5935

to look at the new patch set (#5).

Change subject: WIP [security] load/store public TSK in the system table
......................................................................

WIP [security] load/store public TSK in the system table

Non-expired, already existing TSK records are read from the system
catalog table upon startup of the leader master and fed into the
TokenSigner.  Also, a new token signing key is generated on start-up
and its public part is written into the system catalog table as well.

Added integration tests for cluster start/restart and switching
master leadership role into token_signer-itest.cc.

It's a WIP because some tests are still failing -- need to troubleshoot
those breakages.

Change-Id: Ie91d4129bda0ca49e81988c28385895a2abcd201
---
M src/kudu/client/client-internal.cc
M src/kudu/integration-tests/CMakeLists.txt
A src/kudu/integration-tests/token_signer-itest.cc
M src/kudu/master/authn_token_manager.cc
M src/kudu/master/catalog_manager.cc
M src/kudu/master/catalog_manager.h
M src/kudu/master/master.cc
M src/kudu/master/master.proto
M src/kudu/master/sys_catalog.cc
M src/kudu/master/sys_catalog.h
M src/kudu/security/token_signer.h
11 files changed, 582 insertions(+), 50 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/35/5935/5
-- 
To view, visit http://gerrit.cloudera.org:8080/5935
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie91d4129bda0ca49e81988c28385895a2abcd201
Gerrit-PatchSet: 5
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot

[kudu-CR] [security] load/store public TSK in the system table

Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Hello Kudu Jenkins,

I'd like you to reexamine a change.  Please visit

    http://gerrit.cloudera.org:8080/5935

to look at the new patch set (#13).

Change subject: [security] load/store public TSK in the system table
......................................................................

[security] load/store public TSK in the system table

Non-expired, already existing TSK records are read from the system
catalog table upon the 'elected-as-leader' callback and fed into the
TokenSigner.  New token signing keys are generated and written
into the system catalog table by a periodic task.

The expired entries are removed from the system table
upon the 'elected-as-leader' callback.

Added integration tests for cluster start/restart and switching
master leadership role into token_signer-itest.cc.

Change-Id: Ie91d4129bda0ca49e81988c28385895a2abcd201
---
M src/kudu/client/client-test.cc
M src/kudu/integration-tests/CMakeLists.txt
A src/kudu/integration-tests/token_signer-itest.cc
M src/kudu/master/catalog_manager.cc
M src/kudu/master/catalog_manager.h
M src/kudu/master/master.cc
M src/kudu/master/master.proto
M src/kudu/master/sys_catalog.cc
M src/kudu/master/sys_catalog.h
9 files changed, 461 insertions(+), 35 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/35/5935/13
-- 
To view, visit http://gerrit.cloudera.org:8080/5935
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie91d4129bda0ca49e81988c28385895a2abcd201
Gerrit-PatchSet: 13
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot

[kudu-CR] [security] load/store public TSK in the system table

Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Hello Kudu Jenkins,

I'd like you to reexamine a change.  Please visit

    http://gerrit.cloudera.org:8080/5935

to look at the new patch set (#7).

Change subject: [security] load/store public TSK in the system table
......................................................................

[security] load/store public TSK in the system table

Non-expired, already existing TSK records are read from the system
catalog table upon the 'elected-as-leader' callback and fed into the
TokenSigner.  New token signing keys are generated and written
into the system catalog table by a periodic task.

The expired entries are removed from the system table
upon the 'elected-as-leader' callback.

Added integration tests for cluster start/restart and switching
master leadership role into token_signer-itest.cc.

Change-Id: Ie91d4129bda0ca49e81988c28385895a2abcd201
---
M src/kudu/client/client-test.cc
M src/kudu/integration-tests/CMakeLists.txt
A src/kudu/integration-tests/token_signer-itest.cc
M src/kudu/master/catalog_manager.cc
M src/kudu/master/catalog_manager.h
M src/kudu/master/master.cc
M src/kudu/master/master.proto
M src/kudu/master/sys_catalog.cc
M src/kudu/master/sys_catalog.h
9 files changed, 476 insertions(+), 28 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/35/5935/7
-- 
To view, visit http://gerrit.cloudera.org:8080/5935
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie91d4129bda0ca49e81988c28385895a2abcd201
Gerrit-PatchSet: 7
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot

[kudu-CR] [security] load/store public TSK in the system table

Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Hello Kudu Jenkins,

I'd like you to reexamine a change.  Please visit

    http://gerrit.cloudera.org:8080/5935

to look at the new patch set (#22).

Change subject: [security] load/store public TSK in the system table
......................................................................

[security] load/store public TSK in the system table

Non-expired, already existing TSK records are read from the system
catalog table upon the 'elected-as-leader' callback and fed into the
TokenSigner.  New token signing keys are generated and written
into the system catalog table by a periodic task.

The expired entries are removed from the system table
upon the 'elected-as-leader' callback.

Added integration tests for cluster start/restart and switching
master leadership role into token_signer-itest.cc.

Change-Id: Ie91d4129bda0ca49e81988c28385895a2abcd201
---
M src/kudu/client/client-test.cc
M src/kudu/integration-tests/CMakeLists.txt
M src/kudu/integration-tests/registration-test.cc
A src/kudu/integration-tests/token_signer-itest.cc
M src/kudu/integration-tests/ts_itest-base.h
M src/kudu/master/catalog_manager.cc
M src/kudu/master/catalog_manager.h
M src/kudu/master/master.cc
M src/kudu/master/master.proto
M src/kudu/master/sys_catalog.cc
M src/kudu/master/sys_catalog.h
11 files changed, 520 insertions(+), 46 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/35/5935/22
-- 
To view, visit http://gerrit.cloudera.org:8080/5935
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie91d4129bda0ca49e81988c28385895a2abcd201
Gerrit-PatchSet: 22
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>

[kudu-CR] [security] load/store public TSK in the system table

Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change.

Change subject: [security] load/store public TSK in the system table
......................................................................


Patch Set 5:

(38 comments)

http://gerrit.cloudera.org:8080/#/c/5935/5/src/kudu/client/client-internal.cc
File src/kudu/client/client-internal.cc:

Line 192:     if (s.IsNetworkError() || s.IsServiceUnavailable()) {
> Probably doesn't belong in this patch, right? I think you put this in a dif
Right, this part has been removed from this patch and is committed as a part of some other patch.


http://gerrit.cloudera.org:8080/#/c/5935/5/src/kudu/integration-tests/token_signer-itest.cc
File src/kudu/integration-tests/token_signer-itest.cc:

Line 42: using std::shared_ptr;
> warning: using decl 'shared_ptr' is unused [misc-unused-using-decls]
Done


Line 85:   SignedTokenPB MakeToken() {
> Declare as static?
Done


Line 125:   gscoped_ptr<MiniCluster> cluster_;
> unique_ptr
Done


Line 141:   SleepFor(MonoDelta::FromSeconds(1));
> Why is this needed?
Done


PS5, Line 145: have
> has
Done


PS5, Line 145: folower
> follower
Done


http://gerrit.cloudera.org:8080/#/c/5935/5/src/kudu/master/catalog_manager.cc
File src/kudu/master/catalog_manager.cc:

Line 204: DEFINE_int64(catalog_manager_sys_table_propagation_seconds, 60,
> Not really sure what this means.
This has been updated and moved.


Line 208: // TODO(PKI): add flag tags
> Indeed.
Done


Line 211: using std::remove_if;
> warning: using decl 'remove_if' is unused [misc-unused-using-decls]
Done


Line 342:     CHECK(token_signer_);
> could use CHECK_NOTNULL in the initializer list above.
Done


Line 345:   // The entry_id corresponds to the sequence number of the TSK key.
> But we're ignoring the entry_id, so why is this comment useful?
removed


Line 359:     LOG(INFO) << "Imported TSPK " << tspk.key_seq_num() << " from system table";
> This may be too noisy. How many non-expired entries would we typically expe
Usually not more than 1K.  Agree -- that would be too much.  Will change to VLOG(1) instead.


Line 365:   const set<int64_t>& loaded_key_seq_numbers() const {
> We only ever get the first number, so change this method to expose that rat
Luckily, this is no longer needed.


PS5, Line 371:  // Epoch time
> What does this mean?
I tried to add a description saying key_exp_time_seconds_ is absolute time, in seconds, representing Epoch time.  The value
 is to define cut-off time for TSK entries being loaded: entries with expiration time less than the specified value are not imported into TokenSigner.
 
I fill add this comment into the code.


PS5, Line 469: firther
> further
Done


Line 485:               // TODO Add tests for this in the revision that makes
> warning: missing username/bug in TODO [google-readability-todo]
Done


PS5, Line 489: firther
> further
Done


Line 497:           catalog_manager_->LoadTskEntries(&last_seen_tsk_entry_id);
> This runs every second. Isn't that super wasteful?
This is removed.


Line 500:     } while (false);
> Why this change? We can have an extra scope (for lock acquisition/release) 
That's to handle those 'brake' statements when it's found the instance is no longer a leader master.  Simply calling 'continue' would not work here -- it's necessary to go to that Wait() directive in the end of the cycle.  Using 'goto' was not considered as an option.


PS5, Line 3269: There's a bit of subtlety here:
> Yeah, this is complicated. I'll take another look at this later; maybe ask 
yup, this is simplified now.


Line 3290:       RETURN_NOT_OK(signer->GenerateSigningKey(&next_tsk_));
> Why are we writing to next_tsk_ before persisting, then clearing it on fail
This is gone.


Line 3296:       const Status s = sys_catalog_->AddTskEntries({sys_entry});
> Can we use security::TokenSigningPublicKeyPB in the interface to this funct
Done


Line 3320:   TskEntryLoader loader(signer, WallTime_Now());
> A certain amount of imprecision is OK here, right? Masters' clocks may not 
That's fine -- the propagation interval and other parameters are supposed to have bigger margins.


Line 3322:   const set<int64_t> seq_numbers(loader.loaded_key_seq_numbers());
> Could be a cref.
Good catch.  This has changed, though.


http://gerrit.cloudera.org:8080/#/c/5935/5/src/kudu/master/catalog_manager.h
File src/kudu/master/catalog_manager.h:

Line 565:   //Status LoadTokenSigningInfo(std::vector<SysTskEntryPB>* new_entries);
> Why is this commented out?
It seems it's no longer needed.


Line 627:   Status LoadTskEntries(std::string* tsk_entry_id);
> warning: function 'kudu::master::CatalogManager::LoadTskEntries' has a defi
Done


PS5, Line 773: follower master
> follower masters
this part is gone.


PS5, Line 773: pubic
> public
ugh


http://gerrit.cloudera.org:8080/#/c/5935/5/src/kudu/master/master.proto
File src/kudu/master/master.proto:

Line 200: // The on-disk entry in the sys.catalog table ("metadata" column) to represent
> This isn't used (and is commented out); can it be removed? I think it'd be 
Done


Line 204: // be single entry of this type in the sys.catalog table. 
> Trailing whitespace here.
Done


http://gerrit.cloudera.org:8080/#/c/5935/5/src/kudu/master/sys_catalog.cc
File src/kudu/master/sys_catalog.cc:

Line 478: // TSK-related methods
> This isn't using the same format as the other boundaries, plus it's splitti
Done


Line 490: void SysCatalogTable::ReqAddTskEntry(WriteRequestPB* req,
> Could be inlined into AddTskEntries(), since it's just used in that one pla
That's a good idea, thanks.


Line 511: void SysCatalogTable::ReqDeleteTskEntry(WriteRequestPB* req,
> Not used?
It seems this patch is missing the part which removes expired TSK entries from the system table.  I'll add that missed part in the next revision.


Line 520:   CHECK_OK(row.SetStringNoCopy(
> Is NoCopy() correct? Doesn't the string from TskSeqNumberToEntryId() go out
Good catch -- it should be replaced with SetString().


Line 568:   unique_ptr<Arena> range_arena;
> Can't reuse the arena below?
I was not sure whether it would be legal to do so.  Anyway, this part is going away since we are going to read all the TSK entries from the system table during our scans.


Line 576:     const string key_upper_boundary = key_low_boundary + ".";
> Confused by this; if we want to scan from key_low_boundary and up, shouldn'
For some reason I didn't see it was possible to pass nullptr as the condition for the range (I assumed it would be possible and tried to look at that).  Probably, I ended up looking into some wrong place or something else.

Thank you for pointing at this -- at least now I can see that passing nullptr either for upper or lower boundary is acceptable.


http://gerrit.cloudera.org:8080/#/c/5935/5/src/kudu/master/sys_catalog.h
File src/kudu/master/sys_catalog.h:

PS5, Line 148: If the 'low_entry_id'
             :   // is not empty, it is used as the scan's exclusive low boundary for the
             :   // 'entry_id' column.
> This feels premature. The number of TSK rows is pretty low, right? Can't we
Good idea.  Todd is advocating for this simpler approach as well.  I'll simplify it, thanks.


-- 
To view, visit http://gerrit.cloudera.org:8080/5935
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ie91d4129bda0ca49e81988c28385895a2abcd201
Gerrit-PatchSet: 5
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot
Gerrit-HasComments: Yes