You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@kudu.apache.org by "Dan Burkert (Code Review)" <ge...@cloudera.org> on 2017/10/18 02:19:46 UTC

[kudu-CR] KUDU-2191 (4/n): WIP: Hive Metastore catalog manager integration

Dan Burkert has uploaded this change for review. ( http://gerrit.cloudera.org:8080/8312


Change subject: KUDU-2191 (4/n): WIP: Hive Metastore catalog manager integration
......................................................................

KUDU-2191 (4/n): WIP: Hive Metastore catalog manager integration

This commit adds a basic integration with the Hive Metastore, which can
be enabled with a new flag: 'hive-metastore-addresses'. When the flag is
provided, the Master will push DDL changes in its catalog to the Hive
Metastore. In particular:

- Create table, delete table, and alter table operations in the Kudu
  master will synchronously update the corresponding entry in the HMS.

- New table and column names are required to be valid according to the
  Hive identifier rules, which are much stricter than Kudu's native
  identifier rules.

WIP: This patch doesn't yet have sufficient error handling logic, nor
test coverage of errors.  Here are the test cases I intend to add to
master_hms-itest before this is committed:

- Rename a Kudu table to a new name which already has an entry in the HMS
  (should fail).
- Rename a Kudu table when the original name doesn't have an entry in
  the HMS (should create an entry with the new name).
- Rename a Kudu table when the HMS is unreachable (should fail).
- Create a Kudu table with a name which already exists in the HMS
  (should fail).
- Create a Kudu table when the HMS is unreachable (should fail).
- Drop a Kudu table when the corresponding entry in the HMS does not
  exist (should succeed).
- Drop a Kudu table when the HMS is unreachable (should fail?).

Additionally, this patch as written overwrites the entire HMS table
entry during table rename operations. This is bad because it will wipe
out any stats that query engines like Impala have attached to the table.

Change-Id: I1b8f360408b05d6ac4f7359a8ac0dd889b196baf
---
M src/kudu/integration-tests/CMakeLists.txt
A src/kudu/integration-tests/master_hms-itest.cc
M src/kudu/master/CMakeLists.txt
M src/kudu/master/catalog_manager.cc
M src/kudu/master/catalog_manager.h
A src/kudu/master/hms_catalog.cc
A src/kudu/master/hms_catalog.h
M src/kudu/master/master.proto
M src/kudu/mini-cluster/external_mini_cluster.cc
M src/kudu/util/net/net_util.cc
M src/kudu/util/net/net_util.h
11 files changed, 664 insertions(+), 0 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/12/8312/1
-- 
To view, visit http://gerrit.cloudera.org:8080/8312
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I1b8f360408b05d6ac4f7359a8ac0dd889b196baf
Gerrit-Change-Number: 8312
Gerrit-PatchSet: 1
Gerrit-Owner: Dan Burkert <da...@apache.org>

[kudu-CR] KUDU-2191 (6/n): Hive Metastore catalog manager integration

Posted by "Dan Burkert (Code Review)" <ge...@cloudera.org>.
Dan Burkert has posted comments on this change. ( http://gerrit.cloudera.org:8080/8312 )

Change subject: KUDU-2191 (6/n): Hive Metastore catalog manager integration
......................................................................


Patch Set 4:

(7 comments)

http://gerrit.cloudera.org:8080/#/c/8312/2/src/kudu/integration-tests/CMakeLists.txt
File src/kudu/integration-tests/CMakeLists.txt:

http://gerrit.cloudera.org:8080/#/c/8312/2/src/kudu/integration-tests/CMakeLists.txt@79
PS2, Line 79: ADD_KUDU_TEST(fuzz-itest RUN_SERIAL true)
> I think you missed this one.
Done


http://gerrit.cloudera.org:8080/#/c/8312/2/src/kudu/master/catalog_manager.cc
File src/kudu/master/catalog_manager.cc:

http://gerrit.cloudera.org:8080/#/c/8312/2/src/kudu/master/catalog_manager.cc@1066
PS2, Line 1066:     return s;
> I think you missed this one.
I just did it in reverse order of Init(), I'm not aware of any ordering issues per se, as long as Shutdown() isn't called concurrently with any other use of the hms catalog.


http://gerrit.cloudera.org:8080/#/c/8312/2/src/kudu/master/catalog_manager.cc@1598
PS2, Line 1598:   }
> Hmm, the new behavior treats HMS table entry dropping as fatal, so this new
Done


http://gerrit.cloudera.org:8080/#/c/8312/2/src/kudu/master/catalog_manager.cc@2092
PS2, Line 2092:       case AlterTableRequestPB::DROP_RANGE_PARTITION: {
> I think you missed this.
Done


http://gerrit.cloudera.org:8080/#/c/8312/4/src/kudu/master/catalog_manager.cc
File src/kudu/master/catalog_manager.cc:

http://gerrit.cloudera.org:8080/#/c/8312/4/src/kudu/master/catalog_manager.cc@1561
PS4, Line 1561:       // TODO(dan): figure out how to test this.
> You can inject faults into SysCatalogTable::SyncWrite. Maybe that will help
That's a good idea, I'm going to leave these here for now until I can write some fault tests.  Until then, I've changed MasterFailoverTest to have the HMS enabled and it does appear to be triggering these codepaths.


http://gerrit.cloudera.org:8080/#/c/8312/3/src/kudu/master/hms_catalog.h
File src/kudu/master/hms_catalog.h:

http://gerrit.cloudera.org:8080/#/c/8312/3/src/kudu/master/hms_catalog.h@29
PS3, Line 29: #include "kudu/util/net/net_util.h"
> Not using optional anymore?
right, but HmsClient is now a field so I don't think it can be forward declared.


http://gerrit.cloudera.org:8080/#/c/8312/3/src/kudu/master/hms_catalog.h@51
PS3, Line 51:   ~HmsCatalog();
> Please doc the class and its methods. I'm especially interested in the sync
I've documented the public api in the .h, there are already pretty extensive notes in the .cc about synchronization, retry behavior, etc.



-- 
To view, visit http://gerrit.cloudera.org:8080/8312
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1b8f360408b05d6ac4f7359a8ac0dd889b196baf
Gerrit-Change-Number: 8312
Gerrit-PatchSet: 4
Gerrit-Owner: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Thu, 08 Mar 2018 00:47:12 +0000
Gerrit-HasComments: Yes

[kudu-CR] KUDU-2191 (6/n): Hive Metastore catalog manager integration

Posted by "Dan Burkert (Code Review)" <ge...@cloudera.org>.
Dan Burkert has abandoned this change. ( http://gerrit.cloudera.org:8080/8312 )

Change subject: KUDU-2191 (6/n): Hive Metastore catalog manager integration
......................................................................


Abandoned

split into smaller chunks
-- 
To view, visit http://gerrit.cloudera.org:8080/8312
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: abandon
Gerrit-Change-Id: I1b8f360408b05d6ac4f7359a8ac0dd889b196baf
Gerrit-Change-Number: 8312
Gerrit-PatchSet: 10
Gerrit-Owner: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>

[kudu-CR] KUDU-2191 (6/n): Hive Metastore catalog manager integration

Posted by "Dan Burkert (Code Review)" <ge...@cloudera.org>.
Dan Burkert has posted comments on this change. ( http://gerrit.cloudera.org:8080/8312 )

Change subject: KUDU-2191 (6/n): Hive Metastore catalog manager integration
......................................................................


Patch Set 10:

(8 comments)

http://gerrit.cloudera.org:8080/#/c/8312/8/src/kudu/integration-tests/master_hms-itest.cc
File src/kudu/integration-tests/master_hms-itest.cc:

http://gerrit.cloudera.org:8080/#/c/8312/8/src/kudu/integration-tests/master_hms-itest.cc@162
PS8, Line 162:   }
             : 
             :   // Checks that a table does not exist in the Kudu and HMS catalogs.
             :   void CheckCatalogsNegative(const string& database_n
> No good ideas here short of changing the MiniHMS lifecycle to outlive its m
I have spent time on trying to speed up HMS startup, to a certain extent.  That resulted in trimming a bunch of jars out of the Hadoop and Hive tarballs we vendor. [1] and [2] were the resulting artifacts.

[1] https://github.com/apache/kudu/blob/master/thirdparty/package-hadoop.sh
[2] https://github.com/apache/kudu/blob/master/thirdparty/package-hive.sh


http://gerrit.cloudera.org:8080/#/c/8312/8/src/kudu/integration-tests/master_hms-itest.cc@243
PS8, Line 243:   ASSERT_OK(hms_client_->CreateTable(external_table));
             : 
             :   // Attempt to rename the Kudu table to the external table name.
             :   string external_table_name = Substitute("$0.$1", hms_database_name, hms_external_table_name);
             :   unique_ptr<KuduTableAlterer> table_alterer(client_->NewTableAlterer(table_name));
             :   Status s = table_alterer->RenameTo(external_table_name)->Alter();
> Would be a good negative test for creation too, right?
Done


http://gerrit.cloudera.org:8080/#/c/8312/8/src/kudu/integration-tests/master_hms-itest.cc@268
PS8, Line 268:   ASSERT_TRUE(s.IsIllegalState()) << s.ToString();
             :   ASSERT_STR_CONTAINS(s.ToString(), "
> But isn't the CheckCatalogs() call on L277 ensuring that the Kudu table (re
1. Done
2. I agree.  I'm considering removing all of the const variables and using string literals everywhere instead.  WDYT?


http://gerrit.cloudera.org:8080/#/c/8312/9/src/kudu/master/catalog_manager.cc
File src/kudu/master/catalog_manager.cc:

http://gerrit.cloudera.org:8080/#/c/8312/9/src/kudu/master/catalog_manager.cc@1755
PS9, Line 1755:         Schema schema;
> Hmm, I do think it's somewhat unintuitive; I could see someone else reading
Done


http://gerrit.cloudera.org:8080/#/c/8312/9/src/kudu/master/catalog_manager.cc@2272
PS9, Line 2272:         Schema schema;
> Same thing.
Done


http://gerrit.cloudera.org:8080/#/c/8312/9/src/kudu/master/hms_catalog.cc
File src/kudu/master/hms_catalog.cc:

http://gerrit.cloudera.org:8080/#/c/8312/9/src/kudu/master/hms_catalog.cc@201
PS9, Line 201: HMS table entry: "
             :                 << s.ToString();
             :         return create_table();
> What's stopping another HMS client (perhaps another Kudu cluster?) from renaming the entry from the original name to some other name in between L172 and L207? If that happened, we'd tell our client that the rename went through just fine when in fact it didn't happen at all.

This codepath won't be active for a rename, since orig_table and table will have different names they wont be equal.  What this codepath is protecting against is sending unnecessary add/drop column or other such alterations.

To your general point, there's absolutely nothing we can or should do to try and protect against concurrent table alterations.  Even if we sent the (no-op) alter table operation to the HMS, some other actor can come in and alter it just afterwords, before the master can ack to the Kudu client that the alter is done.  But that's fine; the operations are still strictly serialized in the HMS.


http://gerrit.cloudera.org:8080/#/c/8312/9/src/kudu/master/hms_catalog.cc@258
PS9, Line 258:   // would be helpful. Perhaps a TRACE message and/or a TRACE_COUNTER_INCREMENT
> It's still an issue if Reconnect() succeeds (i.e. hms_client_->Start() succ
Ack


http://gerrit.cloudera.org:8080/#/c/8312/9/src/kudu/master/hms_catalog.cc@300
PS9, Line 300:         // Attempt to reconnect.
> No, just that in my experience, failures tend to cascade, so the first fail
Done



-- 
To view, visit http://gerrit.cloudera.org:8080/8312
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1b8f360408b05d6ac4f7359a8ac0dd889b196baf
Gerrit-Change-Number: 8312
Gerrit-PatchSet: 10
Gerrit-Owner: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Tue, 03 Apr 2018 17:15:00 +0000
Gerrit-HasComments: Yes

[kudu-CR] KUDU-2191 (6/n): Hive Metastore catalog manager integration

Posted by "Dan Burkert (Code Review)" <ge...@cloudera.org>.
Hello Tidy Bot, Kudu Jenkins, Adar Dembo, Hao Hao, Todd Lipcon, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/8312

to look at the new patch set (#9).

Change subject: KUDU-2191 (6/n): Hive Metastore catalog manager integration
......................................................................

KUDU-2191 (6/n): Hive Metastore catalog manager integration

This commit adds integration with the Hive Metastore to CatalogManager,
which can be enabled with a new flag: 'hive-metastore-addresses'. When
the flag is provided, the Master will push DDL changes in its catalog to
the Hive Metastore. In particular:

- Create table, delete table, and alter table operations in the Kudu
  master will synchronously update the corresponding entry in the HMS.

- New table and column names are required to be valid according to the
  Hive identifier rules, which are much stricter than Kudu's existing
  identifier rules.

I think the code-paths in hms_catalog are pretty well covered by the
tests added in this commit, however there is a dearth of stress and
chaos tests covering the actual CatalogManager integration. In
particular I've left some TODOs of roll-back code that could benefit
from more coverage, as well as a TODO in master-stress-test about
enabling HMS integration.

Change-Id: I1b8f360408b05d6ac4f7359a8ac0dd889b196baf
---
M src/kudu/hms/hms_client-test.cc
M src/kudu/hms/hms_client.cc
M src/kudu/hms/hms_client.h
M src/kudu/hms/mini_hms.cc
M src/kudu/integration-tests/CMakeLists.txt
M src/kudu/integration-tests/alter_table-randomized-test.cc
M src/kudu/integration-tests/master-stress-test.cc
M src/kudu/integration-tests/master_failover-itest.cc
A src/kudu/integration-tests/master_hms-itest.cc
M src/kudu/master/CMakeLists.txt
M src/kudu/master/catalog_manager.cc
M src/kudu/master/catalog_manager.h
A src/kudu/master/hms_catalog-test.cc
A src/kudu/master/hms_catalog.cc
A src/kudu/master/hms_catalog.h
M src/kudu/master/master.proto
M src/kudu/mini-cluster/external_mini_cluster-test.cc
M src/kudu/mini-cluster/external_mini_cluster.cc
M src/kudu/util/net/net_util.cc
M src/kudu/util/net/net_util.h
20 files changed, 1,309 insertions(+), 42 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/12/8312/9
-- 
To view, visit http://gerrit.cloudera.org:8080/8312
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I1b8f360408b05d6ac4f7359a8ac0dd889b196baf
Gerrit-Change-Number: 8312
Gerrit-PatchSet: 9
Gerrit-Owner: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>

[kudu-CR] KUDU-2191 (4/n): WIP: Hive Metastore catalog manager integration

Posted by "Adar Dembo (Code Review)" <ge...@cloudera.org>.
Adar Dembo has posted comments on this change. ( http://gerrit.cloudera.org:8080/8312 )

Change subject: KUDU-2191 (4/n): WIP: Hive Metastore catalog manager integration
......................................................................


Patch Set 3:

(36 comments)

http://gerrit.cloudera.org:8080/#/c/8312/2/src/kudu/integration-tests/CMakeLists.txt
File src/kudu/integration-tests/CMakeLists.txt:

http://gerrit.cloudera.org:8080/#/c/8312/2/src/kudu/integration-tests/CMakeLists.txt@79
PS2, Line 79: ADD_KUDU_TEST(master_hms-itest)
Nit: sort order.


http://gerrit.cloudera.org:8080/#/c/8312/2/src/kudu/integration-tests/master_hms-itest.cc
File src/kudu/integration-tests/master_hms-itest.cc:

http://gerrit.cloudera.org:8080/#/c/8312/2/src/kudu/integration-tests/master_hms-itest.cc@75
PS2, Line 75: e = true
Maybe use ExternalMiniClusterITestBase?


http://gerrit.cloudera.org:8080/#/c/8312/2/src/kudu/integration-tests/master_hms-itest.cc@79
PS2, Line 79:     ASSERT_OK(cluster_->Start());
Can this be omitted?


http://gerrit.cloudera.org:8080/#/c/8312/2/src/kudu/integration-tests/master_hms-itest.cc@144
PS2, Line 144:   ASSERT_OK(hms_client_->GetAllTables(hms_database_name, &tables));
I think you can use ContainsKey here, from map-util.h.


http://gerrit.cloudera.org:8080/#/c/8312/2/src/kudu/integration-tests/master_hms-itest.cc@155
PS2, Line 155: } // namespace kudu
ContainsKey here too.


http://gerrit.cloudera.org:8080/#/c/8312/2/src/kudu/master/catalog_manager.h
File src/kudu/master/catalog_manager.h:

http://gerrit.cloudera.org:8080/#/c/8312/2/src/kudu/master/catalog_manager.h@33
PS2, Line 33: #include <boost/optional/optional.hpp>
Not used?


http://gerrit.cloudera.org:8080/#/c/8312/2/src/kudu/master/catalog_manager.cc
File src/kudu/master/catalog_manager.cc:

http://gerrit.cloudera.org:8080/#/c/8312/2/src/kudu/master/catalog_manager.cc@231
PS2, Line 231:               "Addresses of Hive Metastore");
Should probably beef this up a bit to explain what setting this actually does.


http://gerrit.cloudera.org:8080/#/c/8312/2/src/kudu/master/catalog_manager.cc@634
PS2, Line 634:     CHECK_OK(HostPort::ParseStrings(FLAGS_hive_metastore_addresses, 9083, &addresses));
9083 is the default HMS port? Probably deserves more visibility than inlined in catalog_manager.cc. Maybe as a constant in hms_client.h?

Also, could you add a comment justifying the CHECK_OK? It's guaranteed because of the gflag validator, right?


http://gerrit.cloudera.org:8080/#/c/8312/2/src/kudu/master/catalog_manager.cc@1066
PS2, Line 1066:   if (hms_catalog_) {
The order of operations in CatalogManager::Shutdown is tricky and has bitten us before in various ways. How did you arrive at this precise placement of stopping hms_catalog? What constraints are there (if any) about where it should happen?


http://gerrit.cloudera.org:8080/#/c/8312/2/src/kudu/master/catalog_manager.cc@1415
PS2, Line 1415:   // Create the table in the HMS.
It's worth noting what happens to this table if step e fails.


http://gerrit.cloudera.org:8080/#/c/8312/2/src/kudu/master/catalog_manager.cc@1598
PS2, Line 1598:   if (hms_catalog_) {
Why is it OK if this succeeds but step 3 fails? I can take my answer in the form of a code comment.


http://gerrit.cloudera.org:8080/#/c/8312/2/src/kudu/master/catalog_manager.cc@2092
PS2, Line 2092:     Status s = hms_catalog_->AlterTable(table->id(), table_name, new_name, new_schema);
Shouldn't some aspect of this be conditioned on one (or several) of has_foo_changes? For example, if neither the table name nor the schema have changed, why send the alter to HMS?


http://gerrit.cloudera.org:8080/#/c/8312/3/src/kudu/master/hms_catalog.h
File src/kudu/master/hms_catalog.h:

http://gerrit.cloudera.org:8080/#/c/8312/3/src/kudu/master/hms_catalog.h@29
PS3, Line 29: #include "kudu/hms/hms_client.h"
Seems like you may be able to forward declare HmsClient.


http://gerrit.cloudera.org:8080/#/c/8312/3/src/kudu/master/hms_catalog.h@42
PS3, Line 42: struct Rpc {
Maybe define this as a private nested class of HmsCatalog, to make it clear that public users of HmsCatalog aren't expected to use it?


http://gerrit.cloudera.org:8080/#/c/8312/3/src/kudu/master/hms_catalog.h@51
PS3, Line 51: class HmsCatalog {
Please doc the class and its methods. I'm especially interested in the synchronization, the reason for a dedicated thread, etc.


http://gerrit.cloudera.org:8080/#/c/8312/3/src/kudu/master/hms_catalog.h@70
PS3, Line 70: Schema
Can you get away with forward declaring Schema?


http://gerrit.cloudera.org:8080/#/c/8312/3/src/kudu/master/hms_catalog.h@90
PS3, Line 90:   // Only mutated by the worker thread.
Maybe it shouldn't be a class member then? It could be declared on the stack in Run().


http://gerrit.cloudera.org:8080/#/c/8312/3/src/kudu/master/hms_catalog.h@91
PS3, Line 91:   boost::optional<hms::HmsClient> client_;
Worth a comment explaining why this is optional.


http://gerrit.cloudera.org:8080/#/c/8312/3/src/kudu/master/hms_catalog.cc
File src/kudu/master/hms_catalog.cc:

http://gerrit.cloudera.org:8080/#/c/8312/3/src/kudu/master/hms_catalog.cc@31
PS3, Line 31: #include "kudu/master/catalog_manager.h"
            : #include "kudu/master/sys_catalog.h"
Why do you need these?


http://gerrit.cloudera.org:8080/#/c/8312/3/src/kudu/master/hms_catalog.cc@34
PS3, Line 34: #include "kudu/util/flag_tags.h"
Not used?


http://gerrit.cloudera.org:8080/#/c/8312/3/src/kudu/master/hms_catalog.cc@38
PS3, Line 38: using rapidjson::Document;
            : using rapidjson::Value;
These aren't used.


http://gerrit.cloudera.org:8080/#/c/8312/3/src/kudu/master/hms_catalog.cc@73
PS3, Line 73:     worker = worker_.get();
Should this also null out worker_? What's the expected behavior of multiple Stop() calls?


http://gerrit.cloudera.org:8080/#/c/8312/3/src/kudu/master/hms_catalog.cc@97
PS3, Line 97:   Synchronizer s;
            :   rpc.callback = s.AsStdStatusCallback();
            : 
            :   {
            :     MutexLock lock(lock_);
            :     CHECK(running_);
            :     rpc_queue_.emplace_back(std::move(rpc));
            :     if (rpc_queue_.size() == 1) {
            :       cond_.Signal();
            :     }
            :   }
            : 
            :   return s.Wait();
Encapsulate this in a common method shared by CreateTable, DropTable, etc?


http://gerrit.cloudera.org:8080/#/c/8312/3/src/kudu/master/hms_catalog.cc@122
PS3, Line 122:     return client->DropTableWithContext(hms_database, hms_table, env_ctx);
Just curious, why doesn't dropping a table use a hive::Table as input?


http://gerrit.cloudera.org:8080/#/c/8312/3/src/kudu/master/hms_catalog.cc@131
PS3, Line 131:     if (rpc_queue_.size() == 1) {
             :       cond_.Signal();
             :     }
Why bother checking the rpc_queue_ size? Isn't always signalling correct too, and simpler?


http://gerrit.cloudera.org:8080/#/c/8312/3/src/kudu/master/hms_catalog.cc@184
PS3, Line 184: string("TODO")
Can you add a TODO comment explaining what's missing here?


http://gerrit.cloudera.org:8080/#/c/8312/3/src/kudu/master/hms_catalog.cc@191
PS3, Line 191: Status HmsCatalog::ParseTableName(const string& table,
Would love to see a barrage of tests on this method. I think it can be static, so shouldn't be too hard to write tests for it.


http://gerrit.cloudera.org:8080/#/c/8312/3/src/kudu/master/hms_catalog.cc@199
PS3, Line 199: a
Maybe mention that there should be exactly one database/table separator.


http://gerrit.cloudera.org:8080/#/c/8312/3/src/kudu/master/hms_catalog.cc@229
PS3, Line 229:     client_ = hms::HmsClient(address);
So we try to connect to the HMS on each address until we find one that works? Add a comment explaining this; on first glance I thought the code as written was buggy.


http://gerrit.cloudera.org:8080/#/c/8312/3/src/kudu/master/hms_catalog.cc@237
PS3, Line 237:     LOG(WARNING) << "Unable to connect to the Hive Metastore ("
             :                  << address.ToString() << "): " << s.ToString();
Can save some space with WARN_NOT_OK here.


http://gerrit.cloudera.org:8080/#/c/8312/3/src/kudu/master/hms_catalog.cc@248
PS3, Line 248:       MutexLock lock(lock_);
             :       rpc_queue_.swap(rpcs);
Nit: indentation.


http://gerrit.cloudera.org:8080/#/c/8312/3/src/kudu/master/hms_catalog.cc@263
PS3, Line 263: lock
Nit: stylistic, but I like to use really short variable names (like 'l') for lock guards, to reflect their short scope. In this case it's also a useful way to disambiguate between 'lock' and 'lock_', which are quite similar looking.


http://gerrit.cloudera.org:8080/#/c/8312/3/src/kudu/master/hms_catalog.cc@264
PS3, Line 264:       if (rpc_queue_.empty()) {
Spurious wake-ups are possible; this should be a while rather than an if.


http://gerrit.cloudera.org:8080/#/c/8312/3/src/kudu/master/hms_catalog.cc@306
PS3, Line 306:       // If it's an IO error, assume the connection is toast.
What other interesting error classes would we expect to see? Why is IOError the only one that merits resetting the connection?


http://gerrit.cloudera.org:8080/#/c/8312/2/src/kudu/util/net/net_util.h
File src/kudu/util/net/net_util.h:

http://gerrit.cloudera.org:8080/#/c/8312/2/src/kudu/util/net/net_util.h@133
PS2, Line 133: bool ValidateAddressListFlag(const char* flag_name, const std::string& addr_list);
> warning: function 'kudu::ValidateAddressListFlag' has a definition with dif
Doc please. Would be good to explain that the somewhat odd signature (i.e. not returning a Status) is so that it can be used directly as a gflag validator.


http://gerrit.cloudera.org:8080/#/c/8312/3/src/kudu/util/net/net_util.cc
File src/kudu/util/net/net_util.cc:

http://gerrit.cloudera.org:8080/#/c/8312/3/src/kudu/util/net/net_util.cc@279
PS3, Line 279:   WARN_NOT_OK(s, Substitute("invalid flag $0", flag_name));
I think ERROR is a more appropriate loglevel.



-- 
To view, visit http://gerrit.cloudera.org:8080/8312
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1b8f360408b05d6ac4f7359a8ac0dd889b196baf
Gerrit-Change-Number: 8312
Gerrit-PatchSet: 3
Gerrit-Owner: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot
Gerrit-Comment-Date: Thu, 19 Oct 2017 00:20:30 +0000
Gerrit-HasComments: Yes

[kudu-CR] KUDU-2191 (6/n): Hive Metastore catalog manager integration

Posted by "Adar Dembo (Code Review)" <ge...@cloudera.org>.
Adar Dembo has posted comments on this change. ( http://gerrit.cloudera.org:8080/8312 )

Change subject: KUDU-2191 (6/n): Hive Metastore catalog manager integration
......................................................................


Patch Set 4:

(10 comments)

Just responding to your responses, will defer a rereview until the tests pass (it's been long enough since my first review that I've forgotten all of the new code).

http://gerrit.cloudera.org:8080/#/c/8312/2/src/kudu/integration-tests/CMakeLists.txt
File src/kudu/integration-tests/CMakeLists.txt:

http://gerrit.cloudera.org:8080/#/c/8312/2/src/kudu/integration-tests/CMakeLists.txt@79
PS2, Line 79: ADD_KUDU_TEST(fuzz-itest RUN_SERIAL true)
> Nit: sort order.
I think you missed this one.


http://gerrit.cloudera.org:8080/#/c/8312/2/src/kudu/integration-tests/master_hms-itest.cc
File src/kudu/integration-tests/master_hms-itest.cc:

http://gerrit.cloudera.org:8080/#/c/8312/2/src/kudu/integration-tests/master_hms-itest.cc@75
PS2, Line 75: op());
> Maybe use ExternalMiniClusterITestBase?
Done


http://gerrit.cloudera.org:8080/#/c/8312/2/src/kudu/integration-tests/master_hms-itest.cc@79
PS2, Line 79:   Status StopHms() {
> Can this be omitted?
Done


http://gerrit.cloudera.org:8080/#/c/8312/2/src/kudu/integration-tests/master_hms-itest.cc@112
PS2, Line 112:     b.AddColumn("decimal32_val")->Type(KuduColumnSchema::DECIMAL)
> ???
The table_name() function expects a cref string, so std::move(table_name) has no effect.


http://gerrit.cloudera.org:8080/#/c/8312/2/src/kudu/master/catalog_manager.cc
File src/kudu/master/catalog_manager.cc:

http://gerrit.cloudera.org:8080/#/c/8312/2/src/kudu/master/catalog_manager.cc@1066
PS2, Line 1066:     return s;
> The order of operations in CatalogManager::Shutdown is tricky and has bitte
I think you missed this one.


http://gerrit.cloudera.org:8080/#/c/8312/2/src/kudu/master/catalog_manager.cc@1598
PS2, Line 1598:   }
> Why is it OK if this succeeds but step 3 fails? I can take my answer in the
Hmm, the new behavior treats HMS table entry dropping as fatal, so this new comment is no longer correct.


http://gerrit.cloudera.org:8080/#/c/8312/2/src/kudu/master/catalog_manager.cc@2092
PS2, Line 2092:       case AlterTableRequestPB::DROP_RANGE_PARTITION: {
> Shouldn't some aspect of this be conditioned on one (or several) of has_foo
I think you missed this.


http://gerrit.cloudera.org:8080/#/c/8312/4/src/kudu/master/catalog_manager.cc
File src/kudu/master/catalog_manager.cc:

http://gerrit.cloudera.org:8080/#/c/8312/4/src/kudu/master/catalog_manager.cc@1561
PS4, Line 1561:       // TODO(dan): figure out how to test this.
You can inject faults into SysCatalogTable::SyncWrite. Maybe that will help?


http://gerrit.cloudera.org:8080/#/c/8312/4/src/kudu/master/catalog_manager.cc@2253
PS4, Line 2253:       // TODO(dan): figure out how to test this.
See an earlier comment about injecting failures into syscatalog writes.


http://gerrit.cloudera.org:8080/#/c/8312/3/src/kudu/master/hms_catalog.h
File src/kudu/master/hms_catalog.h:

http://gerrit.cloudera.org:8080/#/c/8312/3/src/kudu/master/hms_catalog.h@29
PS3, Line 29: #include "kudu/util/net/net_util.h"
> optional seems to require the include in order to compile.
Not using optional anymore?



-- 
To view, visit http://gerrit.cloudera.org:8080/8312
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1b8f360408b05d6ac4f7359a8ac0dd889b196baf
Gerrit-Change-Number: 8312
Gerrit-PatchSet: 4
Gerrit-Owner: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Tue, 06 Mar 2018 19:48:33 +0000
Gerrit-HasComments: Yes

[kudu-CR] KUDU-2191 (6/n): Hive Metastore catalog manager integration

Posted by "Adar Dembo (Code Review)" <ge...@cloudera.org>.
Adar Dembo has posted comments on this change. ( http://gerrit.cloudera.org:8080/8312 )

Change subject: KUDU-2191 (6/n): Hive Metastore catalog manager integration
......................................................................


Patch Set 10:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/8312/8/src/kudu/integration-tests/master_hms-itest.cc
File src/kudu/integration-tests/master_hms-itest.cc:

http://gerrit.cloudera.org:8080/#/c/8312/8/src/kudu/integration-tests/master_hms-itest.cc@268
PS8, Line 268:   ASSERT_TRUE(s.IsIllegalState()) << s.ToString();
             :   ASSERT_STR_CONTAINS(s.ToString(), "
> Do you have a suggestion for how to rename?  We have three table name local
Hmm. The "hms" prefix is doubly confusing in that it is used for hms_client_ too. So I'd replace that with "full" or some other descriptive prefix. And instead of omitting the prefix for Kudu names, maybe prefix with "kudu" instead. It's also possible that a scheme like "table1", "table2", "table3", etc. would be easier in that it's easier to search and track.



-- 
To view, visit http://gerrit.cloudera.org:8080/8312
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1b8f360408b05d6ac4f7359a8ac0dd889b196baf
Gerrit-Change-Number: 8312
Gerrit-PatchSet: 10
Gerrit-Owner: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Tue, 03 Apr 2018 18:04:05 +0000
Gerrit-HasComments: Yes

[kudu-CR] KUDU-2191 (6/n): Hive Metastore catalog manager integration

Posted by "Dan Burkert (Code Review)" <ge...@cloudera.org>.
Hello Tidy Bot, Kudu Jenkins, Adar Dembo, Hao Hao, Todd Lipcon, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/8312

to look at the new patch set (#10).

Change subject: KUDU-2191 (6/n): Hive Metastore catalog manager integration
......................................................................

KUDU-2191 (6/n): Hive Metastore catalog manager integration

This commit adds integration with the Hive Metastore to CatalogManager,
which can be enabled with a new flag: 'hive-metastore-addresses'. When
the flag is provided, the Master will push DDL changes in its catalog to
the Hive Metastore. In particular:

- Create table, delete table, and alter table operations in the Kudu
  master will synchronously update the corresponding entry in the HMS.

- New table and column names are required to be valid according to the
  Hive identifier rules, which are much stricter than Kudu's existing
  identifier rules.

I think the code-paths in hms_catalog are pretty well covered by the
tests added in this commit, however there is a dearth of stress and
chaos tests covering the actual CatalogManager integration. In
particular I've left some TODOs of roll-back code that could benefit
from more coverage, as well as a TODO in master-stress-test about
enabling HMS integration.

Change-Id: I1b8f360408b05d6ac4f7359a8ac0dd889b196baf
---
M java/kudu-hive/src/test/java/org/apache/kudu/hive/metastore/TestKuduMetastorePlugin.java
M src/kudu/hms/hms_client-test.cc
M src/kudu/hms/hms_client.cc
M src/kudu/hms/hms_client.h
M src/kudu/hms/mini_hms.cc
M src/kudu/integration-tests/CMakeLists.txt
M src/kudu/integration-tests/alter_table-randomized-test.cc
M src/kudu/integration-tests/master-stress-test.cc
M src/kudu/integration-tests/master_failover-itest.cc
A src/kudu/integration-tests/master_hms-itest.cc
M src/kudu/master/CMakeLists.txt
M src/kudu/master/catalog_manager.cc
M src/kudu/master/catalog_manager.h
A src/kudu/master/hms_catalog-test.cc
A src/kudu/master/hms_catalog.cc
A src/kudu/master/hms_catalog.h
M src/kudu/master/master.proto
M src/kudu/mini-cluster/external_mini_cluster.cc
M src/kudu/util/net/net_util.cc
M src/kudu/util/net/net_util.h
20 files changed, 1,405 insertions(+), 58 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/12/8312/10
-- 
To view, visit http://gerrit.cloudera.org:8080/8312
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I1b8f360408b05d6ac4f7359a8ac0dd889b196baf
Gerrit-Change-Number: 8312
Gerrit-PatchSet: 10
Gerrit-Owner: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>

[kudu-CR] KUDU-2191 (6/n): Hive Metastore catalog manager integration

Posted by "Dan Burkert (Code Review)" <ge...@cloudera.org>.
Dan Burkert has posted comments on this change. ( http://gerrit.cloudera.org:8080/8312 )

Change subject: KUDU-2191 (6/n): Hive Metastore catalog manager integration
......................................................................


Patch Set 10:

As discussed on Slack I'm going to split this commit in two, but I wanted to put up a revision with the fixes made in response to Adar's comments.


-- 
To view, visit http://gerrit.cloudera.org:8080/8312
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1b8f360408b05d6ac4f7359a8ac0dd889b196baf
Gerrit-Change-Number: 8312
Gerrit-PatchSet: 10
Gerrit-Owner: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Wed, 28 Mar 2018 23:24:18 +0000
Gerrit-HasComments: No

[kudu-CR] KUDU-2191 (6/n): Hive Metastore catalog manager integration

Posted by "Dan Burkert (Code Review)" <ge...@cloudera.org>.
Dan Burkert has posted comments on this change. ( http://gerrit.cloudera.org:8080/8312 )

Change subject: KUDU-2191 (6/n): Hive Metastore catalog manager integration
......................................................................


Patch Set 4:

(39 comments)

http://gerrit.cloudera.org:8080/#/c/8312/2/src/kudu/integration-tests/master_hms-itest.cc
File src/kudu/integration-tests/master_hms-itest.cc:

http://gerrit.cloudera.org:8080/#/c/8312/2/src/kudu/integration-tests/master_hms-itest.cc@144
PS2, Line 144:   db.name = hms_database_name;
> I think you can use ContainsKey here, from map-util.h.
ContainsKey doesn't work on vector.


http://gerrit.cloudera.org:8080/#/c/8312/2/src/kudu/integration-tests/master_hms-itest.cc@155
PS2, Line 155:   s = CreateTable(hms_database_name, hms_table_name);
> ContainsKey here too.
Ack


http://gerrit.cloudera.org:8080/#/c/8312/3/src/kudu/integration-tests/master_hms-itest.cc
File src/kudu/integration-tests/master_hms-itest.cc:

http://gerrit.cloudera.org:8080/#/c/8312/3/src/kudu/integration-tests/master_hms-itest.cc@102
PS3, Line 102:     b.AddColumn("int8_val")->Type(KuduColumnSchema::INT8);
> warning: passing result of std::move() as a const reference argument; no mo
Done


http://gerrit.cloudera.org:8080/#/c/8312/2/src/kudu/master/catalog_manager.h
File src/kudu/master/catalog_manager.h:

http://gerrit.cloudera.org:8080/#/c/8312/2/src/kudu/master/catalog_manager.h@33
PS2, Line 33: #include <glog/logging.h>
> Not used?
Done


http://gerrit.cloudera.org:8080/#/c/8312/2/src/kudu/master/catalog_manager.cc
File src/kudu/master/catalog_manager.cc:

http://gerrit.cloudera.org:8080/#/c/8312/2/src/kudu/master/catalog_manager.cc@231
PS2, Line 231: TAG_FLAG(catalog_manager_inject_latency_load_ca_info_ms, hidden);
> Should probably beef this up a bit to explain what setting this actually do
Done


http://gerrit.cloudera.org:8080/#/c/8312/2/src/kudu/master/catalog_manager.cc@634
PS2, Line 634: // Propagate the 'read_default' to the 'write_default' in 'col',
> 9083 is the default HMS port? Probably deserves more visibility than inline
Done


http://gerrit.cloudera.org:8080/#/c/8312/2/src/kudu/master/catalog_manager.cc@1415
PS2, Line 1415: 
> It's worth noting what happens to this table if step e fails.
Done


http://gerrit.cloudera.org:8080/#/c/8312/3/src/kudu/master/catalog_manager.cc
File src/kudu/master/catalog_manager.cc:

http://gerrit.cloudera.org:8080/#/c/8312/3/src/kudu/master/catalog_manager.cc@231
PS3, Line 231: TAG_FLAG(catalog_manager_inject_latency_load_ca_info_ms, hidden);
> a bit more docs here, eg what the multiple addrs mean, etc. Also does the H
Done


http://gerrit.cloudera.org:8080/#/c/8312/3/src/kudu/master/catalog_manager.cc@1419
PS3, Line 1419:   }
> I think it's worth logging such errors as well, since it may be an end user
Done


http://gerrit.cloudera.org:8080/#/c/8312/3/src/kudu/master/hms_catalog.h
File src/kudu/master/hms_catalog.h:

http://gerrit.cloudera.org:8080/#/c/8312/3/src/kudu/master/hms_catalog.h@29
PS3, Line 29: #include "kudu/util/net/net_util.h"
> Seems like you may be able to forward declare HmsClient.
optional seems to require the include in order to compile.


http://gerrit.cloudera.org:8080/#/c/8312/3/src/kudu/master/hms_catalog.h@42
PS3, Line 42: //
> Maybe define this as a private nested class of HmsCatalog, to make it clear
Done


http://gerrit.cloudera.org:8080/#/c/8312/3/src/kudu/master/hms_catalog.h@70
PS3, Line 70: st, Te
> Can you get away with forward declaring Schema?
Done


http://gerrit.cloudera.org:8080/#/c/8312/3/src/kudu/master/hms_catalog.h@90
PS3, Line 90:                                std::string* hms_table) WARN_UNUSED_RESULT;
> Maybe it shouldn't be a class member then? It could be declared on the stac
Done


http://gerrit.cloudera.org:8080/#/c/8312/3/src/kudu/master/hms_catalog.h@91
PS3, Line 91: 
> Worth a comment explaining why this is optional.
Done


http://gerrit.cloudera.org:8080/#/c/8312/3/src/kudu/master/hms_catalog.cc
File src/kudu/master/hms_catalog.cc:

http://gerrit.cloudera.org:8080/#/c/8312/3/src/kudu/master/hms_catalog.cc@31
PS3, Line 31: #include "kudu/common/schema.h"
            : #include "kudu/gutil/strings/split.h
> Why do you need these?
Done


http://gerrit.cloudera.org:8080/#/c/8312/3/src/kudu/master/hms_catalog.cc@34
PS3, Line 34: #include "kudu/hms/hive_metastore_constants.h"
> Not used?
Done


http://gerrit.cloudera.org:8080/#/c/8312/3/src/kudu/master/hms_catalog.cc@38
PS3, Line 38: #include "kudu/util/net/net_util.h"
            : #include "kudu/util/thr
> These aren't used.
Done


http://gerrit.cloudera.org:8080/#/c/8312/3/src/kudu/master/hms_catalog.cc@73
PS3, Line 73:   RETURN_NOT_OK(ThreadPoolBuilder("hms-catalog")
> Should this also null out worker_? What's the expected behavior of multiple
Stop() should be idempotent, which I think it is here, assuming ThreadJoiner::Join is idempotent (which it appears to be from reading its source).


http://gerrit.cloudera.org:8080/#/c/8312/3/src/kudu/master/hms_catalog.cc@88
PS3, Line 88:                                const string& name,
> worth a CHECK(running_) in these functions? otherwise it seems like they wo
Problem is that running_ needs to be mutex protected.  I went back and forth on this and decided not to add it.  If you feel strongly I can, though.


http://gerrit.cloudera.org:8080/#/c/8312/3/src/kudu/master/hms_catalog.cc@97
PS3, Line 97: 
            : Status HmsCatalog::DropTable(const string& id, const string& name) {
            :   string hms_database;
            :   string hms_table;
            : 
            :   if (!ParseTableName(name, &hms_database, &hms_table).ok()) {
            :     // This is a legacy table; it can't be in the HMS, so return success.
            :     VLOG(1) << "Ignoring missing HMS table entry while dropping table "
            :             << name;
            :     return Status::OK();
            :   }
            : 
            :   hive::Environmen
> Encapsulate this in a common method shared by CreateTable, DropTable, etc?
Done


http://gerrit.cloudera.org:8080/#/c/8312/3/src/kudu/master/hms_catalog.cc@122
PS3, Line 122: 
> Just curious, why doesn't dropping a table use a hive::Table as input?
Hive drops tables by name.  hive::Table is the equivalent of Kudu's Schema, so in those terms we do the same thing.


http://gerrit.cloudera.org:8080/#/c/8312/3/src/kudu/master/hms_catalog.cc@131
PS3, Line 131:       // the fields, and write it back. Otherwise we'd overwrite metadata fields
             :       // that other tools put into place, such as table statistics. We do
             :      
> Why bother checking the rpc_queue_ size? Isn't always signalling correct to
I figured it'd be better to do 'edge-triggering' when the queue goes from empty to non-empty, rather than 'level-triggering' every time an item is added.  Maybe I'm missing an edge case or consideration, though?

Edit: this concern has gone away since moving to the threadpool approach.


http://gerrit.cloudera.org:8080/#/c/8312/3/src/kudu/master/hms_catalog.cc@177
PS3, Line 177: 
> warning: parameter 'schema' is unused [misc-unused-parameters]
Done


http://gerrit.cloudera.org:8080/#/c/8312/3/src/kudu/master/hms_catalog.cc@184
PS3, Line 184: duTableIdKey] 
> Can you add a TODO comment explaining what's missing here?
Done


http://gerrit.cloudera.org:8080/#/c/8312/3/src/kudu/master/hms_catalog.cc@191
PS3, Line 191:           return create_table();
> Would love to see a barrage of tests on this method. I think it can be stat
Done


http://gerrit.cloudera.org:8080/#/c/8312/3/src/kudu/master/hms_catalog.cc@198
PS3, Line 198: 
> no digits allowed? that's surprising
Done


http://gerrit.cloudera.org:8080/#/c/8312/3/src/kudu/master/hms_catalog.cc@199
PS3, Line 199:  
> Maybe mention that there should be exactly one database/table separator.
Done


http://gerrit.cloudera.org:8080/#/c/8312/3/src/kudu/master/hms_catalog.cc@228
PS3, Line 228:     Status s;
> I'd think we might need to store an index in the class so that we swap to t
Done


http://gerrit.cloudera.org:8080/#/c/8312/3/src/kudu/master/hms_catalog.cc@229
PS3, Line 229: 
> So we try to connect to the HMS on each address until we find one that work
Done


http://gerrit.cloudera.org:8080/#/c/8312/3/src/kudu/master/hms_catalog.cc@237
PS3, Line 237:     // essentially a single iteration of a run loop which handles HMS client
             :     // reconnection and task processing.
> Can save some space with WARN_NOT_OK here.
Can you be more specific? Having trouble seeing that.


http://gerrit.cloudera.org:8080/#/c/8312/3/src/kudu/master/hms_catalog.cc@248
PS3, Line 248:     // run during this backoff period are also immediately failed.
             :     //
> Nit: indentation.
Done


http://gerrit.cloudera.org:8080/#/c/8312/3/src/kudu/master/hms_catalog.cc@263
PS3, Line 263: pt t
> Nit: stylistic, but I like to use really short variable names (like 'l') fo
Done


http://gerrit.cloudera.org:8080/#/c/8312/3/src/kudu/master/hms_catalog.cc@264
PS3, Line 264:         Status s = Reconnect();
> Spurious wake-ups are possible; this should be a while rather than an if.
Done


http://gerrit.cloudera.org:8080/#/c/8312/3/src/kudu/master/hms_catalog.cc@289
PS3, Line 289:       // A fatal error occurred. Tear down the connection, and try again. We
> worth some logging here?
Done


http://gerrit.cloudera.org:8080/#/c/8312/3/src/kudu/master/hms_catalog.cc@304
PS3, Line 304: Status HmsCatalog::Reconnect() {
> I think wrapping this  in a TRACE_EVENT scope and a   LOG_IF_SLOW and such 
I've added a TODO with the text you wrote.  Note that the individual operations in HmsClient already are wrapped in LOG_IF_SLOW blocks, so I don't think that's such a big deal.  Having it hooked into the event tracing framework will be nice, though.


http://gerrit.cloudera.org:8080/#/c/8312/3/src/kudu/master/hms_catalog.cc@306
PS3, Line 306: 
> What other interesting error classes would we expect to see? Why is IOError
This has been reworked into an explicit IsFatalError function.


http://gerrit.cloudera.org:8080/#/c/8312/3/src/kudu/master/hms_catalog.cc@308
PS3, Line 308:   // succeeds. In order to avoid getting 'stuck' on a partially failed HMS, we
> is  there a more explicit Shutdown we should do on the client? or does call
Done


http://gerrit.cloudera.org:8080/#/c/8312/3/src/kudu/util/net/net_util.h
File src/kudu/util/net/net_util.h:

http://gerrit.cloudera.org:8080/#/c/8312/3/src/kudu/util/net/net_util.h@133
PS3, Line 133: // Validates a gflag value containing a comma-separated list of addresses.
> doc this? specifically I think this is meant to be used as a gflag VALIDATO
Done


http://gerrit.cloudera.org:8080/#/c/8312/3/src/kudu/util/net/net_util.cc
File src/kudu/util/net/net_util.cc:

http://gerrit.cloudera.org:8080/#/c/8312/3/src/kudu/util/net/net_util.cc@279
PS3, Line 279: bool ValidateAddressListFlag(const char* flag_name, const string& addr_list) {
> I think ERROR is a more appropriate loglevel.
Done



-- 
To view, visit http://gerrit.cloudera.org:8080/8312
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1b8f360408b05d6ac4f7359a8ac0dd889b196baf
Gerrit-Change-Number: 8312
Gerrit-PatchSet: 4
Gerrit-Owner: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Tue, 06 Mar 2018 01:48:21 +0000
Gerrit-HasComments: Yes

[kudu-CR] KUDU-2191 (4/n): WIP: Hive Metastore catalog manager integration

Posted by "Dan Burkert (Code Review)" <ge...@cloudera.org>.
Dan Burkert has posted comments on this change. ( http://gerrit.cloudera.org:8080/8312 )

Change subject: KUDU-2191 (4/n): WIP: Hive Metastore catalog manager integration
......................................................................


Patch Set 3:

(14 comments)

This is going to be significantly reworked in the near future.  Leaving open for now because I do want to integrate the feedback.

http://gerrit.cloudera.org:8080/#/c/8312/2/src/kudu/integration-tests/master_hms-itest.cc
File src/kudu/integration-tests/master_hms-itest.cc:

http://gerrit.cloudera.org:8080/#/c/8312/2/src/kudu/integration-tests/master_hms-itest.cc@58
PS2, Line 58: using hms::HmsClient;
> warning: using decl 'KuduTable' is unused [misc-unused-using-decls]
Done


http://gerrit.cloudera.org:8080/#/c/8312/2/src/kudu/integration-tests/master_hms-itest.cc@65
PS2, Line 65: class MasterHmsTest : public KuduTest {
> warning: using decl 'MiniHms' is unused [misc-unused-using-decls]
Done


http://gerrit.cloudera.org:8080/#/c/8312/2/src/kudu/integration-tests/master_hms-itest.cc@67
PS2, Line 67:  public:
> warning: using decl 'set' is unused [misc-unused-using-decls]
Done


http://gerrit.cloudera.org:8080/#/c/8312/2/src/kudu/integration-tests/master_hms-itest.cc@71
PS2, Line 71:   void SetUp() override {
> warning: using decl 'Split' is unused [misc-unused-using-decls]
Done


http://gerrit.cloudera.org:8080/#/c/8312/2/src/kudu/integration-tests/master_hms-itest.cc@72
PS2, Line 72:     KuduTest::SetUp();
> warning: using decl 'Substitute' is unused [misc-unused-using-decls]
Done


http://gerrit.cloudera.org:8080/#/c/8312/2/src/kudu/integration-tests/master_hms-itest.cc@112
PS2, Line 112: 
> warning: passing result of std::move() as a const reference argument; no mo
???


http://gerrit.cloudera.org:8080/#/c/8312/2/src/kudu/master/hms_catalog.cc
File src/kudu/master/hms_catalog.cc:

http://gerrit.cloudera.org:8080/#/c/8312/2/src/kudu/master/hms_catalog.cc@43
PS2, Line 43: using strings::CharSet;
> warning: using decl 'function' is unused [misc-unused-using-decls]
Done


http://gerrit.cloudera.org:8080/#/c/8312/2/src/kudu/master/hms_catalog.cc@47
PS2, Line 47: 
> warning: using decl 'Substitute' is unused [misc-unused-using-decls]
Done


http://gerrit.cloudera.org:8080/#/c/8312/2/src/kudu/master/hms_catalog.cc@54
PS2, Line 54: 
> warning: initializer for member 'lock_' is redundant [readability-redundant
Done


http://gerrit.cloudera.org:8080/#/c/8312/2/src/kudu/master/hms_catalog.cc@161
PS2, Line 161:   rpc.callback = s.AsStdStatusCallback();
> warning: missing username/bug in TODO [google-readability-todo]
Done


http://gerrit.cloudera.org:8080/#/c/8312/2/src/kudu/master/hms_catalog.cc@182
PS2, Line 182:   table->parameters = {
> warning: parameter 'schema' is unused [misc-unused-parameters]
this will be used in a later rev once were handling columns correctly.


http://gerrit.cloudera.org:8080/#/c/8312/2/src/kudu/master/hms_catalog.cc@250
PS2, Line 250:   }
> warning: the parameter 's' is copied for each invocation but only used as a
Done


http://gerrit.cloudera.org:8080/#/c/8312/3/src/kudu/master/hms_catalog.cc
File src/kudu/master/hms_catalog.cc:

http://gerrit.cloudera.org:8080/#/c/8312/3/src/kudu/master/hms_catalog.cc@203
PS3, Line 203:   for (int idx = 0; idx < table.size(); idx++) {
> I feel like this code could be written more concisely using some of the stu
perhaps we should just do the split on '.', and send the two parts to hive to validate.  We would probably just screw it up if we tried to recreate their rules.


http://gerrit.cloudera.org:8080/#/c/8312/2/src/kudu/util/net/net_util.h
File src/kudu/util/net/net_util.h:

http://gerrit.cloudera.org:8080/#/c/8312/2/src/kudu/util/net/net_util.h@133
PS2, Line 133: bool ValidateAddressListFlag(const char* flag_name, const std::string& addr_list);
> warning: function 'kudu::ValidateAddressListFlag' has a definition with dif
Done



-- 
To view, visit http://gerrit.cloudera.org:8080/8312
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1b8f360408b05d6ac4f7359a8ac0dd889b196baf
Gerrit-Change-Number: 8312
Gerrit-PatchSet: 3
Gerrit-Owner: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Mon, 13 Nov 2017 22:54:33 +0000
Gerrit-HasComments: Yes

[kudu-CR] KUDU-2191 (4/n): WIP: Hive Metastore catalog manager integration

Posted by "Todd Lipcon (Code Review)" <ge...@cloudera.org>.
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/8312 )

Change subject: KUDU-2191 (4/n): WIP: Hive Metastore catalog manager integration
......................................................................


Patch Set 3:

(11 comments)

curious about timeouts - I dont see any timeouts set here but I'm guessing they're necessary (what if you kill -STOP the minihms, what happens to our calls?)

http://gerrit.cloudera.org:8080/#/c/8312/3//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/8312/3//COMMIT_MSG@23
PS3, Line 23: master_hms-itest before this is committed:
any plan for fault tests? eg crashing either the HMS or the master at some point in the middle of these sequences?


http://gerrit.cloudera.org:8080/#/c/8312/3/src/kudu/master/catalog_manager.cc
File src/kudu/master/catalog_manager.cc:

http://gerrit.cloudera.org:8080/#/c/8312/3/src/kudu/master/catalog_manager.cc@231
PS3, Line 231:               "Addresses of Hive Metastore");
a bit more docs here, eg what the multiple addrs mean, etc. Also does the HMS expose more than one port? (eg an HTTP interface?) If so we should clarify that it's the Thrift addr


http://gerrit.cloudera.org:8080/#/c/8312/3/src/kudu/master/catalog_manager.cc@1419
PS3, Line 1419:       SetupError(resp->mutable_error(), MasterErrorPB::HIVE_METASTORE_ERROR, s);
I think it's worth logging such errors as well, since it may be an end user that gets the response,  and then they'll probably complain to the admin who will look in the logs for more details.

We should probably be especially verbose in these logs because some failures at this point can cause orphaned pointers in the HMS (eg a timeout may be thrown even if the op succeeded)

(same below)


http://gerrit.cloudera.org:8080/#/c/8312/3/src/kudu/master/hms_catalog.cc
File src/kudu/master/hms_catalog.cc:

http://gerrit.cloudera.org:8080/#/c/8312/3/src/kudu/master/hms_catalog.cc@88
PS3, Line 88:                                const Schema& schema) {
worth a CHECK(running_) in these functions? otherwise it seems like they would just hang forever


http://gerrit.cloudera.org:8080/#/c/8312/3/src/kudu/master/hms_catalog.cc@198
PS3, Line 198:             "Kudu table name must contain only lower-case ASCII characters, "
no digits allowed? that's surprising


http://gerrit.cloudera.org:8080/#/c/8312/3/src/kudu/master/hms_catalog.cc@203
PS3, Line 203:   for (int idx = 0; idx < table.size(); idx++) {
I feel like this code could be written more concisely using some of the stuff from gutil/strings/util.h.

For example, you could use Split() on '.' and then ensure that it has 1 or 2 results, and IsIdentifier() returns true for all parts. Or use AdvanceIdentifier up to twice? Or match the whole string against a class that includes '.' and then use strcount('.') to  see that there is at most one dot?

I think the 'split' is probably most straight forward


http://gerrit.cloudera.org:8080/#/c/8312/3/src/kudu/master/hms_catalog.cc@228
PS3, Line 228:   for (const HostPort& address : addresses_) {
I'd think we might need to store an index in the class so that we swap to the new HMS when one has an issue. Otherwise what if an HMS is "up" (ie accepts thrift connections) but times out all calls, or "up" but for some reason is unable to talk to its backing RDBMS or whatever? I'd guess we'd want to reconnect to the other one.


http://gerrit.cloudera.org:8080/#/c/8312/3/src/kudu/master/hms_catalog.cc@289
PS3, Line 289:         cond_.TimedWait(MonoDelta::FromMilliseconds(wait));
worth some logging here?


http://gerrit.cloudera.org:8080/#/c/8312/3/src/kudu/master/hms_catalog.cc@304
PS3, Line 304:       Status s = rpc.run(&client_.get());
I think wrapping this  in a TRACE_EVENT scope and a   LOG_IF_SLOW and such would  be helpful. Perhaps a TRACE message and/or a   TRACE_COUNTER_INCREMENT too to keep track of how  much time is spent in calls to HMS for a given CreateTable call. That will also require propagating the current Trace object into the 'Rpc' object.

If you'd rather defer this work please add a TODO. I'm happy to do it if you want to assign me the todo


http://gerrit.cloudera.org:8080/#/c/8312/3/src/kudu/master/hms_catalog.cc@308
PS3, Line 308:         client_ = boost::none;
is  there a more explicit Shutdown we should do on the client? or does calling the dtor like this suffice?


http://gerrit.cloudera.org:8080/#/c/8312/3/src/kudu/util/net/net_util.h
File src/kudu/util/net/net_util.h:

http://gerrit.cloudera.org:8080/#/c/8312/3/src/kudu/util/net/net_util.h@133
PS3, Line 133: bool ValidateAddressListFlag(const char* flag_name, const std::string& addr_list);
doc this? specifically I think this is meant to be used as a gflag VALIDATOR and thus the somewhat odd signature



-- 
To view, visit http://gerrit.cloudera.org:8080/8312
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1b8f360408b05d6ac4f7359a8ac0dd889b196baf
Gerrit-Change-Number: 8312
Gerrit-PatchSet: 3
Gerrit-Owner: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Tue, 24 Oct 2017 21:20:47 +0000
Gerrit-HasComments: Yes

[kudu-CR] KUDU-2191 (6/n): Hive Metastore catalog manager integration

Posted by "Adar Dembo (Code Review)" <ge...@cloudera.org>.
Adar Dembo has posted comments on this change. ( http://gerrit.cloudera.org:8080/8312 )

Change subject: KUDU-2191 (6/n): Hive Metastore catalog manager integration
......................................................................


Patch Set 9:

(55 comments)

http://gerrit.cloudera.org:8080/#/c/8312/8/src/kudu/hms/hms_client.cc
File src/kudu/hms/hms_client.cc:

http://gerrit.cloudera.org:8080/#/c/8312/8/src/kudu/hms/hms_client.cc@193
PS8, Line 193:   // Also check that the HMS is configured to allow changing the type of
             :   // columns, which is required to support dropping columns.
Could you elaborate on this a bit? It's not intuitive why it needs to be legal to change a column's type in order to drop it.


http://gerrit.cloudera.org:8080/#/c/8312/8/src/kudu/hms/hms_client.cc@202
PS8, Line 202:   if (boost::iequals(disallow_incompatible_column_type_changes, "true")) {
Hmm, is it really possible for the config's value to have a mixed case? Asking because L185 doesn't do a case-insensitive comparison when verifying that the required listeners are in hive.metastore.transactional.event.listeners.


http://gerrit.cloudera.org:8080/#/c/8312/8/src/kudu/hms/hms_client.cc@218
PS8, Line 218: bool HmsClient::IsConnected() {
Maybe add a bit of test coverage for this new method?


http://gerrit.cloudera.org:8080/#/c/8312/8/src/kudu/hms/mini_hms.cc
File src/kudu/hms/mini_hms.cc:

http://gerrit.cloudera.org:8080/#/c/8312/8/src/kudu/hms/mini_hms.cc@186
PS8, Line 186:   //    Configures the HMS to allow altering and dropping columns.
Nit: indentation is off by one character.


http://gerrit.cloudera.org:8080/#/c/8312/8/src/kudu/hms/mini_hms.cc@214
PS8, Line 214:     <value>jdbc:derby:$1/metadb;create=true</value>
Why the change from 'memory' back to 'directory'?


http://gerrit.cloudera.org:8080/#/c/8312/8/src/kudu/integration-tests/CMakeLists.txt
File src/kudu/integration-tests/CMakeLists.txt:

http://gerrit.cloudera.org:8080/#/c/8312/8/src/kudu/integration-tests/CMakeLists.txt@87
PS8, Line 87: ADD_KUDU_TEST(master_hms-itest)
Could you run this new test with KUDU_MEASURE_TEST_CPU_CONSUMPTION=1 (see run-test.sh) to figure out whether to set RUN_SERIAL or to enumerate the number of PROCESSORS?


http://gerrit.cloudera.org:8080/#/c/8312/9/src/kudu/integration-tests/master-stress-test.cc
File src/kudu/integration-tests/master-stress-test.cc:

http://gerrit.cloudera.org:8080/#/c/8312/9/src/kudu/integration-tests/master-stress-test.cc@343
PS9, Line 343:     return Substitute("default.table_$0", oid_generator_.Next());
Not necessary yet?


http://gerrit.cloudera.org:8080/#/c/8312/8/src/kudu/integration-tests/master_hms-itest.cc
File src/kudu/integration-tests/master_hms-itest.cc:

http://gerrit.cloudera.org:8080/#/c/8312/8/src/kudu/integration-tests/master_hms-itest.cc@122
PS8, Line 122: 
Could RETURN_NOT_OK instead.


http://gerrit.cloudera.org:8080/#/c/8312/8/src/kudu/integration-tests/master_hms-itest.cc@162
PS8, Line 162:   string table_name = Substitute("$0.$1", hms_database_name, hms_table_name);
             : 
             :   // Attempt to create the table before the database is created.
             :   Status s = CreateTable(hms_database_name, hms_table
Could be its own test (i.e. TestCreateTableWithoutDatabase)


http://gerrit.cloudera.org:8080/#/c/8312/8/src/kudu/integration-tests/master_hms-itest.cc@190
PS8, Line 190: 
             :   // Shutdown the HMS and try to create a table.
             :   ASSERT_OK(StopHms());
             : 
             :   s = CreateTable(hms_database_name, "foo");
             :   ASSERT_TRUE(s.IsNetworkError()) << s.ToString();
             : 
             :   // Start the HMS and try again.
             :   ASSERT_OK(StartHms());
             :   ASSERT_OK(CreateTable(hms_database_name, "foo"));
Could break out into a separate test since it only shares HMS database state with the previous test.


http://gerrit.cloudera.org:8080/#/c/8312/8/src/kudu/integration-tests/master_hms-itest.cc@202
PS8, Line 202: 
Maybe add a test that if the HMS is down renaming fails?


http://gerrit.cloudera.org:8080/#/c/8312/8/src/kudu/integration-tests/master_hms-itest.cc@207
PS8, Line 207: 
             :   // Create the database.
             :   hive::Database db;
             :   db.name = hms_database_name;
             :   ASSERT_OK(hms_client_->CreateDatabase(db));
Every test does this; consider doing it in the test fixture.


http://gerrit.cloudera.org:8080/#/c/8312/8/src/kudu/integration-tests/master_hms-itest.cc@243
PS8, Line 243:   // TODO(HIVE-18852): match on the error message.
             : 
             :   // Attempt to rename the Kudu table to an invalid table name.
             :   table_alterer.reset(client_->NewTableAlterer(table_name));
             :   s = table_alterer->RenameTo(Substitute("$0.☃", hms_database_name))->Alter();
             :   ASSERT_TRUE(s.IsIllegalState()) << s.ToString();
Would be a good negative test for creation too, right?


http://gerrit.cloudera.org:8080/#/c/8312/8/src/kudu/integration-tests/master_hms-itest.cc@255
PS8, Line 255:   hive::EnvironmentContext env_ctx;
Why is it necessary to provide the table's ID here, on L270, and on L371, but not when dropping on L331?


http://gerrit.cloudera.org:8080/#/c/8312/8/src/kudu/integration-tests/master_hms-itest.cc@268
PS8, Line 268: 
             :   // Drop the HMS table entry, then c
I was confused when I first read this, but now I think I understand: it's legal to post facto create an HMS external table for an existing Kudu table, and doing so causes the two to become connected such that future operations will affect both. Is that right? May want to elaborate the comment a bit to make that behavior clear.


http://gerrit.cloudera.org:8080/#/c/8312/8/src/kudu/integration-tests/master_hms-itest.cc@285
PS8, Line 285:       hms_external_table_name,
Why does this one exist? It was an external table created on L274, but didn't it get renamed to hms_rename_table_name on L276?


http://gerrit.cloudera.org:8080/#/c/8312/8/src/kudu/integration-tests/master_hms-itest.cc@309
PS8, Line 309:   hms_table.sd.cols.clear();
After this step could you use CheckCatalogs (or a variant thereof) to assert that the column entries do NOT match across HMS and Kudu?


http://gerrit.cloudera.org:8080/#/c/8312/8/src/kudu/integration-tests/master_hms-itest.cc@339
PS8, Line 339:   s = table_alterer->DropColumn("int32_val")->Alter();
Why does this fail? The non-Kudu HMS table entry has the same database and name as the Kudu table, so shouldn't they be "connected" to one another?


http://gerrit.cloudera.org:8080/#/c/8312/8/src/kudu/integration-tests/master_hms-itest.cc@354
PS8, Line 354: 
Maybe ensure it's gone from Kudu too, just to make sure that the HMS integration doesn't do something silly like drop the table from HMS but not Kudu?


http://gerrit.cloudera.org:8080/#/c/8312/8/src/kudu/integration-tests/master_hms-itest.cc@374
PS8, Line 374:   ASSERT_TRUE(s.IsNotFound()) << s.ToString();
Likewise, make sure it was really dropped from Kudu.


http://gerrit.cloudera.org:8080/#/c/8312/8/src/kudu/integration-tests/master_hms-itest.cc@384
PS8, Line 384:   NO_FATALS(CheckCatalogs(hms_database_name, hms_table_name));
Check that it's gone from both Kudu and HMS?


http://gerrit.cloudera.org:8080/#/c/8312/9/src/kudu/master/CMakeLists.txt
File src/kudu/master/CMakeLists.txt:

http://gerrit.cloudera.org:8080/#/c/8312/9/src/kudu/master/CMakeLists.txt@76
PS9, Line 76: ADD_KUDU_TEST(hms_catalog-test)
Likewise, please assess the perf impact of this test and annotate it accordingly.


http://gerrit.cloudera.org:8080/#/c/8312/9/src/kudu/master/catalog_manager.cc
File src/kudu/master/catalog_manager.cc:

http://gerrit.cloudera.org:8080/#/c/8312/9/src/kudu/master/catalog_manager.cc@697
PS9, Line 697:     CHECK_OK(HostPort::ParseStrings(FLAGS_hive_metastore_addresses,
Since you're inside Init, may as well convert these CHECK_OKs to RETURN_NOT_OK. The failure will be fatal either way; it'll just be more graceful.


http://gerrit.cloudera.org:8080/#/c/8312/9/src/kudu/master/catalog_manager.cc@1548
PS9, Line 1548:   // Create the table in the HMS.
Can you expand this comment a bit to explain why this should happen before step e and not after?

This is a meaty enough operation that I'd also give it its own step letter and reletter the others.


http://gerrit.cloudera.org:8080/#/c/8312/9/src/kudu/master/catalog_manager.cc@1743
PS9, Line 1743:   // Drop the HMS table entry.
As in Create, could you expand this to explain why this should happen here, and not after step 3?

Also, please give it its own number and renumber the others.


http://gerrit.cloudera.org:8080/#/c/8312/9/src/kudu/master/catalog_manager.cc@1755
PS9, Line 1755:   auto abort_hms = MakeScopedCleanup([&] {
What does it mean for a ScopedCleanup to return something? It's run on scope exit, so where is it returning that value to exactly?


http://gerrit.cloudera.org:8080/#/c/8312/9/src/kudu/master/catalog_manager.cc@2254
PS9, Line 2254:   if (hms_catalog_ != nullptr && has_metadata_changes) {
Add a comment explaining why we do this here and not after the catalog write, give it a number, renumber the rest, etc.


http://gerrit.cloudera.org:8080/#/c/8312/9/src/kudu/master/catalog_manager.cc@2272
PS9, Line 2272:         RETURN_NOT_OK_LOG(SchemaFromPB(l.data().pb.schema(), &schema), WARNING,
See above comment re: returning a value from a ScopedCleanup.


http://gerrit.cloudera.org:8080/#/c/8312/9/src/kudu/master/hms_catalog-test.cc
File src/kudu/master/hms_catalog-test.cc:

http://gerrit.cloudera.org:8080/#/c/8312/9/src/kudu/master/hms_catalog-test.cc@38
PS9, Line 38: 
Even though master_hms-test provides integration testing between CatalogManager and HmsCatalog, I think it'd still be worthwhile to have standalone HmsCatalog tests here to verify basic functionality. It'd certainly make it easier to test future changes to HmsCatalog.


http://gerrit.cloudera.org:8080/#/c/8312/9/src/kudu/master/hms_catalog.h
File src/kudu/master/hms_catalog.h:

http://gerrit.cloudera.org:8080/#/c/8312/9/src/kudu/master/hms_catalog.h@61
PS9, Line 61:   // Create a new table entry in the HMS.
Doc that this method fails if HMS is unreachable?


http://gerrit.cloudera.org:8080/#/c/8312/9/src/kudu/master/hms_catalog.h@66
PS9, Line 66:   // Drops a table entry from the HMS, if it exists.
Nit: it would be good to use the same tense when describing the functionality of each method. Here you're using present tense "[DropTable()] drops a table entry from the HMS..." but elsewhere you use future tense "[Start()] will start the HmsCatalog instance." I happen to prefer present tense (I think that's the Javadoc standard); could you apply it consistently in your method documentation?


http://gerrit.cloudera.org:8080/#/c/8312/9/src/kudu/master/hms_catalog.h@91
PS9, Line 91:   Status Reconnect();
Doc this?


http://gerrit.cloudera.org:8080/#/c/8312/9/src/kudu/master/hms_catalog.h@103
PS9, Line 103:   static Status ParseTableName(const std::string& table,
And this?


http://gerrit.cloudera.org:8080/#/c/8312/9/src/kudu/master/hms_catalog.h@110
PS9, Line 110:   gscoped_ptr<ThreadPool> threadpool_;
unique_ptr


http://gerrit.cloudera.org:8080/#/c/8312/9/src/kudu/master/hms_catalog.cc
File src/kudu/master/hms_catalog.cc:

http://gerrit.cloudera.org:8080/#/c/8312/9/src/kudu/master/hms_catalog.cc@52
PS9, Line 52: const int HmsCatalog::kNumRetries = 1;
Do you think it would it be useful to convert this into an advanced gflag?


http://gerrit.cloudera.org:8080/#/c/8312/9/src/kudu/master/hms_catalog.cc@76
PS9, Line 76:       .set_min_threads(0)
This is the default; it can be omitted.


http://gerrit.cloudera.org:8080/#/c/8312/9/src/kudu/master/hms_catalog.cc@77
PS9, Line 77:       .set_max_threads(1)
Doc why capping at one thread is important.


http://gerrit.cloudera.org:8080/#/c/8312/9/src/kudu/master/hms_catalog.cc@95
PS9, Line 95:   return Execute([=] (hms::HmsClient* client) {
Execute() waits for the provided functor to run, so why not capture variables by reference in all of these calls?


http://gerrit.cloudera.org:8080/#/c/8312/9/src/kudu/master/hms_catalog.cc@162
PS9, Line 162:       if (!s.ok()) {
I'm going to claim laziness and ask whether all of these edge cases are covered by hms_catalog-test rather than checking myself.


http://gerrit.cloudera.org:8080/#/c/8312/9/src/kudu/master/hms_catalog.cc@180
PS9, Line 180:       // Handle all other errors.
Nit: not really "handling" per se; it's just that all other errors are fatal.


http://gerrit.cloudera.org:8080/#/c/8312/9/src/kudu/master/hms_catalog.cc@201
PS9, Line 201:  If
             :       // the original table object and the new table object match exactly then
             :       // we don't need to alter the table in the HMS.
Does this short circuit make us vulnerable to a TOCTOU race, where we return OK to the client because we think orig_table and table are identical even though someone else may have altered the Hive table in the meantime?


http://gerrit.cloudera.org:8080/#/c/8312/9/src/kudu/master/hms_catalog.cc@222
PS9, Line 222:   // TODO(Todd): wrapping this in a TRACE_EVENT scope and a LOG_IF_SLOW and such
Nit: 

  $ git grep TODO\([a-z] | wc -l
  278
  $ git grep TODO\([A-Z] | grep -v TODO\(KUDU- | wc -l
  5


http://gerrit.cloudera.org:8080/#/c/8312/9/src/kudu/master/hms_catalog.cc@238
PS9, Line 238:     // Since every task submitted to the (single thread) pool runs this, it's
             :     // essentially a single iteration of a run loop which handles HMS client
             :     // reconnection and task processing.
Do you think the pool should have a min_thread count of 1 so that we don't pay the thread startup cost for HMS operations that are spaced out by the pool's idle timeout?


http://gerrit.cloudera.org:8080/#/c/8312/9/src/kudu/master/hms_catalog.cc@253
PS9, Line 253:     // network or IO fault (not an applicaiton level failure). The HMS client
application


http://gerrit.cloudera.org:8080/#/c/8312/9/src/kudu/master/hms_catalog.cc@258
PS9, Line 258:     for (int i = 0; i <= kNumRetries; i++) {
Seems like kNumRetries should depend at least somewhat on the number of HMSes. If we have three HMS instances and the first two are down, the very first submitted task will fail while the rest will succeed.


http://gerrit.cloudera.org:8080/#/c/8312/9/src/kudu/master/hms_catalog.cc@291
PS9, Line 291:       // A fatal error occurred. Tear down the connection, and try again. We
Is it possible for reconnect to succeed, for the task to exhibit a fatal error, and for this to repeat over and over? If so, we'll perform the retries in a tight loop; shouldn't we wait in between? I can see the tight loop being a "feature" rather than a bug if we're retrying the task on a different HMS each time, but not if it's on the same HMS over and over.


http://gerrit.cloudera.org:8080/#/c/8312/9/src/kudu/master/hms_catalog.cc@300
PS9, Line 300:     return callback(s);
This passes in the last failure; should we instead pass in the first?


http://gerrit.cloudera.org:8080/#/c/8312/9/src/kudu/master/hms_catalog.cc@316
PS9, Line 316:     hms_client_ = hms::HmsClient(address, hms::HmsClientOptions());
Nit: might consider making HmsClientOptions() the default value of the constructor's second argument, for better ergonomics.


http://gerrit.cloudera.org:8080/#/c/8312/9/src/kudu/master/hms_catalog.cc@324
PS9, Line 324:     LOG(WARNING) << "Failed to connect to Hive Metastore ("
             :                  << address.ToString() << "): " << s.ToString();
More terse:

  WARN_NOT_OK(s, Substitute("Failed to connect to Hive Metastore ($0)", address.ToString());


http://gerrit.cloudera.org:8080/#/c/8312/9/src/kudu/master/hms_catalog.cc@330
PS9, Line 330:   return s;
If we can't connect to any of the HMSs, should we return the first failure? Currently we're returning the last one.


http://gerrit.cloudera.org:8080/#/c/8312/9/src/kudu/master/hms_catalog.cc@368
PS9, Line 368:       return string();
Hmm, so what would happen at runtime if we returned an empty string here? Any reason not to promote the DFATAL above into a FATAL?


http://gerrit.cloudera.org:8080/#/c/8312/9/src/kudu/master/hms_catalog.cc@387
PS9, Line 387:   table->tableType = "MANAGED_TABLE";
Should this constant be given a little more visibility? What's its significance?


http://gerrit.cloudera.org:8080/#/c/8312/9/src/kudu/master/hms_catalog.cc@420
PS9, Line 420:         "When the Hive Metastore integration is enabled, Kudu table names must be a "
Nit: as a convention we should start a status message with a lower case letter, since they're almost always prepended with other colon-delimited messages on the return path.


http://gerrit.cloudera.org:8080/#/c/8312/9/src/kudu/mini-cluster/external_mini_cluster-test.cc
File src/kudu/mini-cluster/external_mini_cluster-test.cc:

http://gerrit.cloudera.org:8080/#/c/8312/9/src/kudu/mini-cluster/external_mini_cluster-test.cc@19
PS9, Line 19: #include <ostream>
Revert?


http://gerrit.cloudera.org:8080/#/c/8312/9/src/kudu/util/net/net_util.cc
File src/kudu/util/net/net_util.cc:

http://gerrit.cloudera.org:8080/#/c/8312/9/src/kudu/util/net/net_util.cc@18
PS9, Line 18: #include <cstdint>
Nit: this belongs in the next group of includes.



-- 
To view, visit http://gerrit.cloudera.org:8080/8312
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1b8f360408b05d6ac4f7359a8ac0dd889b196baf
Gerrit-Change-Number: 8312
Gerrit-PatchSet: 9
Gerrit-Owner: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Fri, 09 Mar 2018 00:36:36 +0000
Gerrit-HasComments: Yes

[kudu-CR] KUDU-2191 (6/n): Hive Metastore catalog manager integration

Posted by "Adar Dembo (Code Review)" <ge...@cloudera.org>.
Adar Dembo has posted comments on this change. ( http://gerrit.cloudera.org:8080/8312 )

Change subject: KUDU-2191 (6/n): Hive Metastore catalog manager integration
......................................................................


Patch Set 10:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/8312/8/src/kudu/integration-tests/master_hms-itest.cc
File src/kudu/integration-tests/master_hms-itest.cc:

http://gerrit.cloudera.org:8080/#/c/8312/8/src/kudu/integration-tests/master_hms-itest.cc@268
PS8, Line 268:   ASSERT_TRUE(s.IsIllegalState()) << s.ToString();
             :   ASSERT_STR_CONTAINS(s.ToString(), "
> 1. Done
I don't think swapping variables for literals will necessarily change anything. I mean, if you use literals with the same names as the variables, that just means you're adding double quotes, right? So I think improving the names is what's more important.



-- 
To view, visit http://gerrit.cloudera.org:8080/8312
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1b8f360408b05d6ac4f7359a8ac0dd889b196baf
Gerrit-Change-Number: 8312
Gerrit-PatchSet: 10
Gerrit-Owner: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Tue, 03 Apr 2018 17:23:00 +0000
Gerrit-HasComments: Yes

[kudu-CR] KUDU-2191 (6/n): Hive Metastore catalog manager integration

Posted by "Dan Burkert (Code Review)" <ge...@cloudera.org>.
Dan Burkert has posted comments on this change. ( http://gerrit.cloudera.org:8080/8312 )

Change subject: KUDU-2191 (6/n): Hive Metastore catalog manager integration
......................................................................


Patch Set 10:

> Patch Set 10:
> 
> (6 comments)

This has been split into the following reviews:
https://gerrit.cloudera.org/c/9861/
https://gerrit.cloudera.org/c/9862/
https://gerrit.cloudera.org/c/9863/


-- 
To view, visit http://gerrit.cloudera.org:8080/8312
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1b8f360408b05d6ac4f7359a8ac0dd889b196baf
Gerrit-Change-Number: 8312
Gerrit-PatchSet: 10
Gerrit-Owner: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Fri, 30 Mar 2018 01:26:26 +0000
Gerrit-HasComments: No

[kudu-CR] KUDU-2191 (4/n): WIP: Hive Metastore catalog manager integration

Posted by "Dan Burkert (Code Review)" <ge...@cloudera.org>.
Hello Tidy Bot, Kudu Jenkins, Adar Dembo, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/8312

to look at the new patch set (#3).

Change subject: KUDU-2191 (4/n): WIP: Hive Metastore catalog manager integration
......................................................................

KUDU-2191 (4/n): WIP: Hive Metastore catalog manager integration

This commit adds a basic integration with the Hive Metastore, which can
be enabled with a new flag: 'hive-metastore-addresses'. When the flag is
provided, the Master will push DDL changes in its catalog to the Hive
Metastore. In particular:

- Create table, delete table, and alter table operations in the Kudu
  master will synchronously update the corresponding entry in the HMS.

- New table and column names are required to be valid according to the
  Hive identifier rules, which are much stricter than Kudu's native
  identifier rules.

WIP: This patch doesn't yet have sufficient error handling logic, nor
test coverage of errors.  Here are the test cases I intend to add to
master_hms-itest before this is committed:

- Rename a Kudu table to a new name which already has an entry in the HMS
  (should fail).
- Rename a Kudu table when the original name doesn't have an entry in
  the HMS (should create an entry with the new name).
- Rename a Kudu table when the HMS is unreachable (should fail).
- Create a Kudu table with a name which already exists in the HMS
  (should fail).
- Create a Kudu table when the HMS is unreachable (should fail).
- Drop a Kudu table when the corresponding entry in the HMS does not
  exist (should succeed).
- Drop a Kudu table when the HMS is unreachable (should fail?).

Additionally, this patch as written overwrites the entire HMS table
entry during table rename operations. This is bad because it will wipe
out any stats that query engines like Impala have attached to the table.

Change-Id: I1b8f360408b05d6ac4f7359a8ac0dd889b196baf
---
M src/kudu/integration-tests/CMakeLists.txt
A src/kudu/integration-tests/master_hms-itest.cc
M src/kudu/master/CMakeLists.txt
M src/kudu/master/catalog_manager.cc
M src/kudu/master/catalog_manager.h
A src/kudu/master/hms_catalog.cc
A src/kudu/master/hms_catalog.h
M src/kudu/master/master.proto
M src/kudu/mini-cluster/external_mini_cluster.cc
M src/kudu/util/net/net_util.cc
M src/kudu/util/net/net_util.h
11 files changed, 649 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/12/8312/3
-- 
To view, visit http://gerrit.cloudera.org:8080/8312
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I1b8f360408b05d6ac4f7359a8ac0dd889b196baf
Gerrit-Change-Number: 8312
Gerrit-PatchSet: 3
Gerrit-Owner: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot

[kudu-CR] KUDU-2191 (6/n): Hive Metastore catalog manager integration

Posted by "Hao Hao (Code Review)" <ge...@cloudera.org>.
Hao Hao has posted comments on this change. ( http://gerrit.cloudera.org:8080/8312 )

Change subject: KUDU-2191 (6/n): Hive Metastore catalog manager integration
......................................................................


Patch Set 7:

(7 comments)

http://gerrit.cloudera.org:8080/#/c/8312/5/src/kudu/hms/hms_client.cc
File src/kudu/hms/hms_client.cc:

http://gerrit.cloudera.org:8080/#/c/8312/5/src/kudu/hms/hms_client.cc@204
PS5, Line 204:           "Hive Metastore configuration is invalid: $0 must be set to false",
Add "to support dropping columns" to specify the reason.


http://gerrit.cloudera.org:8080/#/c/8312/5/src/kudu/integration-tests/alter_table-randomized-test.cc
File src/kudu/integration-tests/alter_table-randomized-test.cc:

http://gerrit.cloudera.org:8080/#/c/8312/5/src/kudu/integration-tests/alter_table-randomized-test.cc@79
PS5, Line 79: const char* kTableName = "default.test_table";
Do we need to add a comment that when enable_hive_metastore is true, the table name has to have databasename.tablename pattern?


http://gerrit.cloudera.org:8080/#/c/8312/6/src/kudu/integration-tests/master_hms-itest.cc
File src/kudu/integration-tests/master_hms-itest.cc:

http://gerrit.cloudera.org:8080/#/c/8312/6/src/kudu/integration-tests/master_hms-itest.cc@176
PS6, Line 176:   ASSERT_EQ(table->id(), hms_table.parameters[hms::HmsClient::kKuduTableIdKey]);
Assert master addresses as well?


http://gerrit.cloudera.org:8080/#/c/8312/6/src/kudu/integration-tests/master_hms-itest.cc@236
PS6, Line 236:   // Drop the HMS table entry and rename the table.
I do not quite follow why do we need to drop the hms table entry? Will the drop table in HMS also trigger drop table in Kudu? If you are testing the corner case when the entry is not present in HMS, maybe add more comment here to be more clear.

Can we have a most common rename table test case without any corner cases?


http://gerrit.cloudera.org:8080/#/c/8312/6/src/kudu/integration-tests/master_hms-itest.cc@245
PS6, Line 245:   ASSERT_OK(table_alterer->RenameTo(renamed_table_name)->Alter());
Assert table id/master addresses/table schema.


http://gerrit.cloudera.org:8080/#/c/8312/6/src/kudu/integration-tests/master_hms-itest.cc@285
PS6, Line 285:   ASSERT_OK(hms_client_->AlterTable(hms_database_name, hms_table_name, hms_table));
So alter column through HMS would not affect the table schema stored in Kudu?


http://gerrit.cloudera.org:8080/#/c/8312/7/src/kudu/master/hms_catalog.cc
File src/kudu/master/hms_catalog.cc:

http://gerrit.cloudera.org:8080/#/c/8312/7/src/kudu/master/hms_catalog.cc@164
PS7, Line 164: create it in the HMS
I am not sure it is good to create a HMS entry if not present. What if the users make some mistakes when specifying the original table name?



-- 
To view, visit http://gerrit.cloudera.org:8080/8312
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1b8f360408b05d6ac4f7359a8ac0dd889b196baf
Gerrit-Change-Number: 8312
Gerrit-PatchSet: 7
Gerrit-Owner: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Thu, 08 Mar 2018 06:40:50 +0000
Gerrit-HasComments: Yes

[kudu-CR] KUDU-2191 (6/n): Hive Metastore catalog manager integration

Posted by "Dan Burkert (Code Review)" <ge...@cloudera.org>.
Dan Burkert has posted comments on this change. ( http://gerrit.cloudera.org:8080/8312 )

Change subject: KUDU-2191 (6/n): Hive Metastore catalog manager integration
......................................................................


Patch Set 7:

(7 comments)

http://gerrit.cloudera.org:8080/#/c/8312/5/src/kudu/hms/hms_client.cc
File src/kudu/hms/hms_client.cc:

http://gerrit.cloudera.org:8080/#/c/8312/5/src/kudu/hms/hms_client.cc@204
PS5, Line 204:           "Hive Metastore configuration is invalid: $0 must be set to false",
> Add "to support dropping columns" to specify the reason.
I'm not against adding that, but consider that this error will occur for every operation that hits the HMS (create table/all alter tables/drop table), not just ALTER TABLE DROP COLUMN ops.  In that context I think it makes sense to just have a blanket statement 'this config is required', and not get in to too much detail.  This is the same reason I didn't include the reasoning in the kudu metastore plugin check above (plus that one is way more nuanced :).


http://gerrit.cloudera.org:8080/#/c/8312/5/src/kudu/integration-tests/alter_table-randomized-test.cc
File src/kudu/integration-tests/alter_table-randomized-test.cc:

http://gerrit.cloudera.org:8080/#/c/8312/5/src/kudu/integration-tests/alter_table-randomized-test.cc@79
PS5, Line 79: const char* kTableName = "default.test_table";
> Do we need to add a comment that when enable_hive_metastore is true, the ta
We're going to have to document this new behavior very prominently, because it will surely trip up pretty much every current Kudu user that goes to turn on the HMS integration.  Given that it will hopefully be documented widely in other public facing locations, I'm not sure it's worth calling out in specific unit tests.


http://gerrit.cloudera.org:8080/#/c/8312/6/src/kudu/integration-tests/master_hms-itest.cc
File src/kudu/integration-tests/master_hms-itest.cc:

http://gerrit.cloudera.org:8080/#/c/8312/6/src/kudu/integration-tests/master_hms-itest.cc@176
PS6, Line 176:   ASSERT_EQ(table->id(), hms_table.parameters[hms::HmsClient::kKuduTableIdKey]);
> Assert master addresses as well?
Done


http://gerrit.cloudera.org:8080/#/c/8312/6/src/kudu/integration-tests/master_hms-itest.cc@236
PS6, Line 236:   // Drop the HMS table entry and rename the table.
> I do not quite follow why do we need to drop the hms table entry? Will the 
I've added the comment, and added the 'happy' path, I was so focused on the edge cases I forgot about that!


http://gerrit.cloudera.org:8080/#/c/8312/6/src/kudu/integration-tests/master_hms-itest.cc@245
PS6, Line 245:   ASSERT_OK(table_alterer->RenameTo(renamed_table_name)->Alter());
> Assert table id/master addresses/table schema.
Done


http://gerrit.cloudera.org:8080/#/c/8312/6/src/kudu/integration-tests/master_hms-itest.cc@285
PS6, Line 285:   ASSERT_OK(hms_client_->AlterTable(hms_database_name, hms_table_name, hms_table));
> So alter column through HMS would not affect the table schema stored in Kud
Correct.


http://gerrit.cloudera.org:8080/#/c/8312/7/src/kudu/master/hms_catalog.cc
File src/kudu/master/hms_catalog.cc:

http://gerrit.cloudera.org:8080/#/c/8312/7/src/kudu/master/hms_catalog.cc@164
PS7, Line 164: create it in the HMS
> I am not sure it is good to create a HMS entry if not present. What if the 
Users can't directly call this method, its called by the catalog manager, who checks that the Kudu table does exist before proceeding with an alter operation.

It's important to support this usecase so that users can rename 'legacy' Kudu tables that don't have an entry in the HMS, and have that create the new entry with the corrected name.



-- 
To view, visit http://gerrit.cloudera.org:8080/8312
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1b8f360408b05d6ac4f7359a8ac0dd889b196baf
Gerrit-Change-Number: 8312
Gerrit-PatchSet: 7
Gerrit-Owner: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Thu, 08 Mar 2018 18:57:57 +0000
Gerrit-HasComments: Yes

[kudu-CR] KUDU-2191 (6/n): Hive Metastore catalog manager integration

Posted by "Dan Burkert (Code Review)" <ge...@cloudera.org>.
Dan Burkert has posted comments on this change. ( http://gerrit.cloudera.org:8080/8312 )

Change subject: KUDU-2191 (6/n): Hive Metastore catalog manager integration
......................................................................


Patch Set 10:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/8312/8/src/kudu/integration-tests/master_hms-itest.cc
File src/kudu/integration-tests/master_hms-itest.cc:

http://gerrit.cloudera.org:8080/#/c/8312/8/src/kudu/integration-tests/master_hms-itest.cc@268
PS8, Line 268:   ASSERT_TRUE(s.IsIllegalState()) << s.ToString();
             :   ASSERT_STR_CONTAINS(s.ToString(), "
> I don't think swapping variables for literals will necessarily change anyth
Do you have a suggestion for how to rename?  We have three table name locals that are being used here:

table_name
external_table_name
renamed_table_name

It's additionally confusing because there are additional locals which are bound to just the table portion of then names (as opposed to database.table).



-- 
To view, visit http://gerrit.cloudera.org:8080/8312
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1b8f360408b05d6ac4f7359a8ac0dd889b196baf
Gerrit-Change-Number: 8312
Gerrit-PatchSet: 10
Gerrit-Owner: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Tue, 03 Apr 2018 17:56:50 +0000
Gerrit-HasComments: Yes

[kudu-CR] KUDU-2191 (6/n): Hive Metastore catalog manager integration

Posted by "Dan Burkert (Code Review)" <ge...@cloudera.org>.
Hello Tidy Bot, Kudu Jenkins, Adar Dembo, Hao Hao, Todd Lipcon, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/8312

to look at the new patch set (#6).

Change subject: KUDU-2191 (6/n): Hive Metastore catalog manager integration
......................................................................

KUDU-2191 (6/n): Hive Metastore catalog manager integration

This commit adds integration with the Hive Metastore to CatalogManager,
which can be enabled with a new flag: 'hive-metastore-addresses'. When
the flag is provided, the Master will push DDL changes in its catalog to
the Hive Metastore. In particular:

- Create table, delete table, and alter table operations in the Kudu
  master will synchronously update the corresponding entry in the HMS.

- New table and column names are required to be valid according to the
  Hive identifier rules, which are much stricter than Kudu's existing
  identifier rules.

I think the code-paths in hms_catalog are pretty well covered by the
tests added in this commit, however there is a dearth of stress and
chaos tests covering the actual CatalogManager integration. In
particular I've left some TODOs of roll-back code that could benefit
from more coverage, as well as a TODO in master-stress-test about
enabling HMS integration.

Change-Id: I1b8f360408b05d6ac4f7359a8ac0dd889b196baf
---
M src/kudu/hms/hms_client-test.cc
M src/kudu/hms/hms_client.cc
M src/kudu/hms/hms_client.h
M src/kudu/hms/mini_hms.cc
M src/kudu/integration-tests/CMakeLists.txt
M src/kudu/integration-tests/alter_table-randomized-test.cc
M src/kudu/integration-tests/master-stress-test.cc
M src/kudu/integration-tests/master_failover-itest.cc
A src/kudu/integration-tests/master_hms-itest.cc
M src/kudu/master/CMakeLists.txt
M src/kudu/master/catalog_manager.cc
M src/kudu/master/catalog_manager.h
A src/kudu/master/hms_catalog-test.cc
A src/kudu/master/hms_catalog.cc
A src/kudu/master/hms_catalog.h
M src/kudu/master/master.proto
M src/kudu/mini-cluster/external_mini_cluster-test.cc
M src/kudu/mini-cluster/external_mini_cluster.cc
M src/kudu/util/net/net_util.cc
M src/kudu/util/net/net_util.h
20 files changed, 1,263 insertions(+), 24 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/12/8312/6
-- 
To view, visit http://gerrit.cloudera.org:8080/8312
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I1b8f360408b05d6ac4f7359a8ac0dd889b196baf
Gerrit-Change-Number: 8312
Gerrit-PatchSet: 6
Gerrit-Owner: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>

[kudu-CR] KUDU-2191 (6/n): Hive Metastore catalog manager integration

Posted by "Dan Burkert (Code Review)" <ge...@cloudera.org>.
Dan Burkert has posted comments on this change. ( http://gerrit.cloudera.org:8080/8312 )

Change subject: KUDU-2191 (6/n): Hive Metastore catalog manager integration
......................................................................


Patch Set 9:

(52 comments)

http://gerrit.cloudera.org:8080/#/c/8312/8/src/kudu/hms/hms_client.cc
File src/kudu/hms/hms_client.cc:

http://gerrit.cloudera.org:8080/#/c/8312/8/src/kudu/hms/hms_client.cc@193
PS8, Line 193:   // Also check that the HMS is configured to allow changing the type of
             :   // columns, which is required to support dropping columns.
> Could you elaborate on this a bit? It's not intuitive why it needs to be le
Added more elaboration.   tldr; hive is a real :dumpster_fire2:


http://gerrit.cloudera.org:8080/#/c/8312/8/src/kudu/hms/hms_client.cc@202
PS8, Line 202:   if (boost::iequals(disallow_incompatible_column_type_changes, "true")) {
> Hmm, is it really possible for the config's value to have a mixed case? Ask
Yes, I believe it's taken directly from the hive-site.xml so it's possible to have mixed case.  Line 185 is matching on a class name, and Java is case sensitive so it's not a valid configuration to screw that up.


http://gerrit.cloudera.org:8080/#/c/8312/8/src/kudu/hms/hms_client.cc@218
PS8, Line 218: bool HmsClient::IsConnected() {
> Maybe add a bit of test coverage for this new method?
Done


http://gerrit.cloudera.org:8080/#/c/8312/8/src/kudu/hms/mini_hms.cc
File src/kudu/hms/mini_hms.cc:

http://gerrit.cloudera.org:8080/#/c/8312/8/src/kudu/hms/mini_hms.cc@214
PS8, Line 214:     <value>jdbc:derby:$1/metadb;create=true</value>
> Why the change from 'memory' back to 'directory'?
I changed it originally in the hope that it would make things a bit faster if derby didn't have to write to disk.  It's being changed back because I've added tests that stop and restart the HMS, and if it's not on disk the HMS will come back with an empty DB.


http://gerrit.cloudera.org:8080/#/c/8312/8/src/kudu/integration-tests/CMakeLists.txt
File src/kudu/integration-tests/CMakeLists.txt:

http://gerrit.cloudera.org:8080/#/c/8312/8/src/kudu/integration-tests/CMakeLists.txt@87
PS8, Line 87: ADD_KUDU_TEST(master_hms-itest)
> Could you run this new test with KUDU_MEASURE_TEST_CPU_CONSUMPTION=1 (see r
Working on this.


http://gerrit.cloudera.org:8080/#/c/8312/9/src/kudu/integration-tests/master-stress-test.cc
File src/kudu/integration-tests/master-stress-test.cc:

http://gerrit.cloudera.org:8080/#/c/8312/9/src/kudu/integration-tests/master-stress-test.cc@343
PS9, Line 343:     return Substitute("default.table_$0", oid_generator_.Next());
> Not necessary yet?
No, it's not but it doesn't hurt anything so I figured I may as well.


http://gerrit.cloudera.org:8080/#/c/8312/8/src/kudu/integration-tests/master_hms-itest.cc
File src/kudu/integration-tests/master_hms-itest.cc:

http://gerrit.cloudera.org:8080/#/c/8312/8/src/kudu/integration-tests/master_hms-itest.cc@122
PS8, Line 122: 
> Could RETURN_NOT_OK instead.
Done


http://gerrit.cloudera.org:8080/#/c/8312/8/src/kudu/integration-tests/master_hms-itest.cc@162
PS8, Line 162:   string table_name = Substitute("$0.$1", hms_database_name, hms_table_name);
             : 
             :   // Attempt to create the table before the database is created.
             :   Status s = CreateTable(hms_database_name, hms_table
> Could be its own test (i.e. TestCreateTableWithoutDatabase)
I've tried to reduce the number of tests cases since spinning up an HMS is very expensive (~10 seconds), so having many short tests becomes really expensive.  If you have an idea for how to fix that, or still feel like they should be split up I can go ahead, though.


http://gerrit.cloudera.org:8080/#/c/8312/8/src/kudu/integration-tests/master_hms-itest.cc@190
PS8, Line 190: 
             :   // Shutdown the HMS and try to create a table.
             :   ASSERT_OK(StopHms());
             : 
             :   s = CreateTable(hms_database_name, "foo");
             :   ASSERT_TRUE(s.IsNetworkError()) << s.ToString();
             : 
             :   // Start the HMS and try again.
             :   ASSERT_OK(StartHms());
             :   ASSERT_OK(CreateTable(hms_database_name, "foo"));
> Could break out into a separate test since it only shares HMS database stat
Likewise.


http://gerrit.cloudera.org:8080/#/c/8312/8/src/kudu/integration-tests/master_hms-itest.cc@207
PS8, Line 207: 
             :   // Create the database.
             :   hive::Database db;
             :   db.name = hms_database_name;
             :   ASSERT_OK(hms_client_->CreateDatabase(db));
> Every test does this; consider doing it in the test fixture.
Done


http://gerrit.cloudera.org:8080/#/c/8312/8/src/kudu/integration-tests/master_hms-itest.cc@255
PS8, Line 255:   hive::EnvironmentContext env_ctx;
> Why is it necessary to provide the table's ID here, on L270, and on L371, b
This wasn't really necessary.  If you do provide the table ID when dropping a Kudu table, the Kudu HMS plugin will ensure the expected table ID matches the actual, but if you don't provide it the drop will still occur.


http://gerrit.cloudera.org:8080/#/c/8312/8/src/kudu/integration-tests/master_hms-itest.cc@268
PS8, Line 268: 
             :   // Drop the HMS table entry, then c
> I was confused when I first read this, but now I think I understand: it's l
You may be reading too much into this.  This is just setting up a hypothetical scenario in which the Kudu/HMS catalogs may become unsychronized.  In this case we have a Kudu table, and an HMS table entry with the same name, but the HMS table entry isn't a Kudu table entry.  If we rename the Kudu table, it shouldn't attempt to rename the HMS entry (since it's not for the Kudu table being renamed).


http://gerrit.cloudera.org:8080/#/c/8312/8/src/kudu/integration-tests/master_hms-itest.cc@285
PS8, Line 285:       hms_external_table_name,
> It was an external table created on L274

correct.

> but didn't it get renamed to hms_rename_table_name on L276?

No, because the HMS catalog entry wasn't a Kudu table entry matching the Kudu table being renamed, Kudu simply created a new HMS entry instead of altering the old one.  This is important to support so that Kudu can synchronize the HMS and Kudu catalogs by renaming Kudu tables to avoid collisions.


http://gerrit.cloudera.org:8080/#/c/8312/8/src/kudu/integration-tests/master_hms-itest.cc@309
PS8, Line 309:   hms_table.sd.cols.clear();
> After this step could you use CheckCatalogs (or a variant thereof) to asser
I've done a one-off check since I don't think we check anything similar elsewhere.


http://gerrit.cloudera.org:8080/#/c/8312/8/src/kudu/integration-tests/master_hms-itest.cc@339
PS8, Line 339:   s = table_alterer->DropColumn("int32_val")->Alter();
> Why does this fail? The non-Kudu HMS table entry has the same database and 
No, one of the tenets of this integration is that we should _never_ modify a non-Kudu HMS entry.  If we did that it could really really screw things up.  This situation can be 'resolved' either using a manual tool which will offer to delete the HMS entry, or by renaming the Kudu table to avoid the name collision.


http://gerrit.cloudera.org:8080/#/c/8312/8/src/kudu/integration-tests/master_hms-itest.cc@354
PS8, Line 354: 
> Maybe ensure it's gone from Kudu too, just to make sure that the HMS integr
Done


http://gerrit.cloudera.org:8080/#/c/8312/8/src/kudu/integration-tests/master_hms-itest.cc@374
PS8, Line 374:   ASSERT_TRUE(s.IsNotFound()) << s.ToString();
> Likewise, make sure it was really dropped from Kudu.
Done


http://gerrit.cloudera.org:8080/#/c/8312/8/src/kudu/integration-tests/master_hms-itest.cc@384
PS8, Line 384:   NO_FATALS(CheckCatalogs(hms_database_name, hms_table_name));
> Check that it's gone from both Kudu and HMS?
Done


http://gerrit.cloudera.org:8080/#/c/8312/9/src/kudu/master/CMakeLists.txt
File src/kudu/master/CMakeLists.txt:

http://gerrit.cloudera.org:8080/#/c/8312/9/src/kudu/master/CMakeLists.txt@76
PS9, Line 76: ADD_KUDU_TEST(hms_catalog-test)
> Likewise, please assess the perf impact of this test and annotate it accord
Ack


http://gerrit.cloudera.org:8080/#/c/8312/9/src/kudu/master/catalog_manager.cc
File src/kudu/master/catalog_manager.cc:

http://gerrit.cloudera.org:8080/#/c/8312/9/src/kudu/master/catalog_manager.cc@697
PS9, Line 697:     CHECK_OK(HostPort::ParseStrings(FLAGS_hive_metastore_addresses,
> Since you're inside Init, may as well convert these CHECK_OKs to RETURN_NOT
Done


http://gerrit.cloudera.org:8080/#/c/8312/9/src/kudu/master/catalog_manager.cc@1548
PS9, Line 1548:   // Create the table in the HMS.
> Can you expand this comment a bit to explain why this should happen before 
Done


http://gerrit.cloudera.org:8080/#/c/8312/9/src/kudu/master/catalog_manager.cc@1743
PS9, Line 1743:   // Drop the HMS table entry.
> As in Create, could you expand this to explain why this should happen here,
Done


http://gerrit.cloudera.org:8080/#/c/8312/9/src/kudu/master/catalog_manager.cc@1755
PS9, Line 1755:   auto abort_hms = MakeScopedCleanup([&] {
> What does it mean for a ScopedCleanup to return something? It's run on scop
It's a hack that allows using the RETURN_NOT_OK_LOG macro inside the lambda body.  The returned value is never actually used.  If you think this too clever I can write it out manually.


http://gerrit.cloudera.org:8080/#/c/8312/9/src/kudu/master/catalog_manager.cc@2254
PS9, Line 2254:   if (hms_catalog_ != nullptr && has_metadata_changes) {
> Add a comment explaining why we do this here and not after the catalog writ
Done


http://gerrit.cloudera.org:8080/#/c/8312/9/src/kudu/master/catalog_manager.cc@2272
PS9, Line 2272:         RETURN_NOT_OK_LOG(SchemaFromPB(l.data().pb.schema(), &schema), WARNING,
> See above comment re: returning a value from a ScopedCleanup.
Same thing.


http://gerrit.cloudera.org:8080/#/c/8312/9/src/kudu/master/hms_catalog-test.cc
File src/kudu/master/hms_catalog-test.cc:

http://gerrit.cloudera.org:8080/#/c/8312/9/src/kudu/master/hms_catalog-test.cc@38
PS9, Line 38: 
> Even though master_hms-test provides integration testing between CatalogMan
I agree.  Will work on that in future revision.


http://gerrit.cloudera.org:8080/#/c/8312/9/src/kudu/master/hms_catalog.h
File src/kudu/master/hms_catalog.h:

http://gerrit.cloudera.org:8080/#/c/8312/9/src/kudu/master/hms_catalog.h@61
PS9, Line 61:   // Create a new table entry in the HMS.
> Doc that this method fails if HMS is unreachable?
Done


http://gerrit.cloudera.org:8080/#/c/8312/9/src/kudu/master/hms_catalog.h@66
PS9, Line 66:   // Drops a table entry from the HMS, if it exists.
> Nit: it would be good to use the same tense when describing the functionali
Done


http://gerrit.cloudera.org:8080/#/c/8312/9/src/kudu/master/hms_catalog.h@91
PS9, Line 91:   Status Reconnect();
> Doc this?
Done


http://gerrit.cloudera.org:8080/#/c/8312/9/src/kudu/master/hms_catalog.h@103
PS9, Line 103:   static Status ParseTableName(const std::string& table,
> And this?
Done


http://gerrit.cloudera.org:8080/#/c/8312/9/src/kudu/master/hms_catalog.h@110
PS9, Line 110:   gscoped_ptr<ThreadPool> threadpool_;
> unique_ptr
This doesn't appear to be possible due to the API of ThreadPoolBuilder.


http://gerrit.cloudera.org:8080/#/c/8312/9/src/kudu/master/hms_catalog.cc
File src/kudu/master/hms_catalog.cc:

http://gerrit.cloudera.org:8080/#/c/8312/9/src/kudu/master/hms_catalog.cc@52
PS9, Line 52: const int HmsCatalog::kNumRetries = 1;
> Do you think it would it be useful to convert this into an advanced gflag?
Done


http://gerrit.cloudera.org:8080/#/c/8312/9/src/kudu/master/hms_catalog.cc@76
PS9, Line 76:       .set_min_threads(0)
> This is the default; it can be omitted.
Done


http://gerrit.cloudera.org:8080/#/c/8312/9/src/kudu/master/hms_catalog.cc@77
PS9, Line 77:       .set_max_threads(1)
> Doc why capping at one thread is important.
Done


http://gerrit.cloudera.org:8080/#/c/8312/9/src/kudu/master/hms_catalog.cc@95
PS9, Line 95:   return Execute([=] (hms::HmsClient* client) {
> Execute() waits for the provided functor to run, so why not capture variabl
Done


http://gerrit.cloudera.org:8080/#/c/8312/9/src/kudu/master/hms_catalog.cc@162
PS9, Line 162:       if (!s.ok()) {
> I'm going to claim laziness and ask whether all of these edge cases are cov
I believe that all edge-cases are covered by a combination of master_hms-itest and hms_catalog-test.


http://gerrit.cloudera.org:8080/#/c/8312/9/src/kudu/master/hms_catalog.cc@180
PS9, Line 180:       // Handle all other errors.
> Nit: not really "handling" per se; it's just that all other errors are fata
Done


http://gerrit.cloudera.org:8080/#/c/8312/9/src/kudu/master/hms_catalog.cc@201
PS9, Line 201:  If
             :       // the original table object and the new table object match exactly then
             :       // we don't need to alter the table in the HMS.
> Does this short circuit make us vulnerable to a TOCTOU race, where we retur
Well you're right to expect there are TOCTOU races going on all over the place here, but this isn't one of them.  Since we're only accessing the HMS once during this short-circuit, there's no TOCTOU.

However in the non-short-circuited case there are absolutely TOCTOU issues where we can overwrite a table entry in the HMS which has been altered in the meantime; the way we mostly kind of solve this is to have the Kudu HMS plugin check that the Kudu table ID matches when altering a Kudu table entry.  That provides a defense against the master accidentally altering a non-Kudu table entry, which would be catastrophic.


http://gerrit.cloudera.org:8080/#/c/8312/9/src/kudu/master/hms_catalog.cc@222
PS9, Line 222:   // TODO(Todd): wrapping this in a TRACE_EVENT scope and a LOG_IF_SLOW and such
> Nit: 
Done


http://gerrit.cloudera.org:8080/#/c/8312/9/src/kudu/master/hms_catalog.cc@238
PS9, Line 238:     // Since every task submitted to the (single thread) pool runs this, it's
             :     // essentially a single iteration of a run loop which handles HMS client
             :     // reconnection and task processing.
> Do you think the pool should have a min_thread count of 1 so that we don't 
Done


http://gerrit.cloudera.org:8080/#/c/8312/9/src/kudu/master/hms_catalog.cc@253
PS9, Line 253:     // network or IO fault (not an applicaiton level failure). The HMS client
> application
Done


http://gerrit.cloudera.org:8080/#/c/8312/9/src/kudu/master/hms_catalog.cc@258
PS9, Line 258:     for (int i = 0; i <= kNumRetries; i++) {
> Seems like kNumRetries should depend at least somewhat on the number of HMS
Reconnect() handles connecting to the live HMS, if others are down.  Note that the HMS is active/active, it's not like our master where you have to go to a certain one.


http://gerrit.cloudera.org:8080/#/c/8312/9/src/kudu/master/hms_catalog.cc@291
PS9, Line 291:       // A fatal error occurred. Tear down the connection, and try again. We
> Is it possible for reconnect to succeed, for the task to exhibit a fatal er
Yes, this is possible because we can't really guarantee that we bucket errors 100% accurately into application (non-retriable) vs fatal (retriable).  To make sure this doesn't loop forever we break after kNumRetries, which defaults to the low value of 1.


http://gerrit.cloudera.org:8080/#/c/8312/9/src/kudu/master/hms_catalog.cc@300
PS9, Line 300:     return callback(s);
> This passes in the last failure; should we instead pass in the first?
I can't think of a reason it would make a difference, unless you are suggesting they somehow be combined?


http://gerrit.cloudera.org:8080/#/c/8312/9/src/kudu/master/hms_catalog.cc@316
PS9, Line 316:     hms_client_ = hms::HmsClient(address, hms::HmsClientOptions());
> Nit: might consider making HmsClientOptions() the default value of the cons
This is here because we're missing a bunch of flags that set HmsClient options, like enabling kerberos etc.  Going to fix in next revision.


http://gerrit.cloudera.org:8080/#/c/8312/9/src/kudu/master/hms_catalog.cc@324
PS9, Line 324:     LOG(WARNING) << "Failed to connect to Hive Metastore ("
             :                  << address.ToString() << "): " << s.ToString();
> More terse:
Done


http://gerrit.cloudera.org:8080/#/c/8312/9/src/kudu/master/hms_catalog.cc@330
PS9, Line 330:   return s;
> If we can't connect to any of the HMSs, should we return the first failure?
I can't think of a reason it would make a difference.


http://gerrit.cloudera.org:8080/#/c/8312/9/src/kudu/master/hms_catalog.cc@368
PS9, Line 368:       return string();
> Hmm, so what would happen at runtime if we returned an empty string here? A
Done


http://gerrit.cloudera.org:8080/#/c/8312/9/src/kudu/master/hms_catalog.cc@387
PS9, Line 387:   table->tableType = "MANAGED_TABLE";
> Should this constant be given a little more visibility? What's its signific
Done


http://gerrit.cloudera.org:8080/#/c/8312/9/src/kudu/master/hms_catalog.cc@420
PS9, Line 420:         "When the Hive Metastore integration is enabled, Kudu table names must be a "
> Nit: as a convention we should start a status message with a lower case let
Done


http://gerrit.cloudera.org:8080/#/c/8312/9/src/kudu/mini-cluster/external_mini_cluster-test.cc
File src/kudu/mini-cluster/external_mini_cluster-test.cc:

http://gerrit.cloudera.org:8080/#/c/8312/9/src/kudu/mini-cluster/external_mini_cluster-test.cc@19
PS9, Line 19: #include <ostream>
> Revert?
Done


http://gerrit.cloudera.org:8080/#/c/8312/9/src/kudu/util/net/net_util.cc
File src/kudu/util/net/net_util.cc:

http://gerrit.cloudera.org:8080/#/c/8312/9/src/kudu/util/net/net_util.cc@18
PS9, Line 18: #include <cstdint>
> Nit: this belongs in the next group of includes.
Done



-- 
To view, visit http://gerrit.cloudera.org:8080/8312
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1b8f360408b05d6ac4f7359a8ac0dd889b196baf
Gerrit-Change-Number: 8312
Gerrit-PatchSet: 9
Gerrit-Owner: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Wed, 28 Mar 2018 23:23:45 +0000
Gerrit-HasComments: Yes

[kudu-CR] KUDU-2191 (6/n): Hive Metastore catalog manager integration

Posted by "Adar Dembo (Code Review)" <ge...@cloudera.org>.
Adar Dembo has posted comments on this change. ( http://gerrit.cloudera.org:8080/8312 )

Change subject: KUDU-2191 (6/n): Hive Metastore catalog manager integration
......................................................................


Patch Set 10:

(7 comments)

http://gerrit.cloudera.org:8080/#/c/8312/8/src/kudu/hms/hms_client.cc
File src/kudu/hms/hms_client.cc:

http://gerrit.cloudera.org:8080/#/c/8312/8/src/kudu/hms/hms_client.cc@202
PS8, Line 202:   // operations properly.
> Yes, I believe it's taken directly from the hive-site.xml so it's possible 
Gotcha. Thanks for the explanation.


http://gerrit.cloudera.org:8080/#/c/8312/8/src/kudu/integration-tests/master_hms-itest.cc
File src/kudu/integration-tests/master_hms-itest.cc:

http://gerrit.cloudera.org:8080/#/c/8312/8/src/kudu/integration-tests/master_hms-itest.cc@162
PS8, Line 162:   }
             : 
             :   // Checks that a table does not exist in the Kudu and HMS catalogs.
             :   void CheckCatalogsNegative(const string& database_n
> I've tried to reduce the number of tests cases since spinning up an HMS is 
No good ideas here short of changing the MiniHMS lifecycle to outlive its minicluster (so that it's reset once per test suite rather than once per test). But that's a really bad idea for other reasons.

If you have the time, it might be fruitful to profile the HMS startup and figure out why it takes so long. If that time is spent searching an enormous classpath, we may be able to prune it to improve startup time.


http://gerrit.cloudera.org:8080/#/c/8312/8/src/kudu/integration-tests/master_hms-itest.cc@268
PS8, Line 268:   ASSERT_TRUE(s.IsIllegalState()) << s.ToString();
             :   ASSERT_STR_CONTAINS(s.ToString(), "
> You may be reading too much into this.  This is just setting up a hypotheti
But isn't the CheckCatalogs() call on L277 ensuring that the Kudu table (renamed_table_name) and the HMS table (hms_renamed_table_name) have the same names?

I was still confused so I went through the entire test and wrote down the actions taking place:

  // Create the Kudu table.
  Create Kudu table rename_db.kudu_table
  - Implicit create of same HMS table

  // Create a non-Kudu HMS table entry.
  Create non-Kudu HMS table rename_db.external_table

  // Drop the HMS table entry and rename the table. This tests that the
  // HmsCatalog will create a new entry when necessary.
  Drop HMS table rename_db.kudu_table
  Rename Kudu table from rename_db.kudu_table to rename_db.renamed_table
  - Implicit create of Kudu HMS table rename_db.renamed_table

  // Rename the table back to the original name. This tests the happy path.
  Rename Kudu table from rename_db.renamed_table to rename_db.kudu_table
  - Implicit rename of Kudu HMS table rename_db.renamed_table to rename_db.kudu_table

  // Drop the HMS table entry, then create a non-Kudu table entry in it's place,
  // and attempt to rename the table.
  Drop Kudu HMS table rename_db.kudu_table
  Create non-Kudu HMS table rename_db.kudu_table
  Rename Kudu table from rename_db.kudu_table to rename_db.renamed_table
  - No implicit rename because rename_db.kudu_table is non-Kudu HMS entry
  - Implicit create of Kudu HMS table rename_db.renamed_table

What I was missing was that in the last Kudu rename, not only does Kudu NOT rename the existing HMS entry (because it's a non-Kudu entry), but it'll implicitly create a new HMS entry for the destination table name.

A couple things that would make this easier to understand:
1. hms_client_->CreateTable() and CreateTable() are similar looking; rename CreateTable() to something else.
2. The table name variables are also pretty similar looking; rename them so that they stand out from one another more.


http://gerrit.cloudera.org:8080/#/c/8312/9/src/kudu/master/catalog_manager.cc
File src/kudu/master/catalog_manager.cc:

http://gerrit.cloudera.org:8080/#/c/8312/9/src/kudu/master/catalog_manager.cc@1755
PS9, Line 1755:         Schema schema;
> It's a hack that allows using the RETURN_NOT_OK_LOG macro inside the lambda
Hmm, I do think it's somewhat unintuitive; I could see someone else reading this and thinking that they've misunderstood how ScopedCleanup works.

I wouldn't mind it if there was a comment saying the return value is ignored. Or if you wrote it out manually.


http://gerrit.cloudera.org:8080/#/c/8312/9/src/kudu/master/hms_catalog.cc
File src/kudu/master/hms_catalog.cc:

http://gerrit.cloudera.org:8080/#/c/8312/9/src/kudu/master/hms_catalog.cc@201
PS9, Line 201: HMS table entry: "
             :                 << s.ToString();
             :         return create_table();
> Well you're right to expect there are TOCTOU races going on all over the pl
I don't understand your explanation.

On L172 we retrieve the original HMS table entry. On L207, if that entry's name matches the new name, we short-circuit. What's stopping another HMS client (perhaps another Kudu cluster?) from renaming the entry from the original name to some other name in between L172 and L207? If that happened, we'd tell our client that the rename went through just fine when in fact it didn't happen at all.

If instead we just tried the rename anyway, it would fail because the original table entry is now gone, and we'd be able to communicate that to our client properly.


http://gerrit.cloudera.org:8080/#/c/8312/9/src/kudu/master/hms_catalog.cc@258
PS9, Line 258:   // would be helpful. Perhaps a TRACE message and/or a TRACE_COUNTER_INCREMENT
> Reconnect() handles connecting to the live HMS, if others are down.  Note t
It's still an issue if Reconnect() succeeds (i.e. hms_client_->Start() succeeds) but then the task fails anyway. But you addressed this below when you talked about the difficulty of bucketing errors into retriable vs. non-retriable buckets.


http://gerrit.cloudera.org:8080/#/c/8312/9/src/kudu/master/hms_catalog.cc@300
PS9, Line 300:         // Attempt to reconnect.
> I can't think of a reason it would make a difference, unless you are sugges
No, just that in my experience, failures tend to cascade, so the first failure in a system is usually the most interesting.



-- 
To view, visit http://gerrit.cloudera.org:8080/8312
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1b8f360408b05d6ac4f7359a8ac0dd889b196baf
Gerrit-Change-Number: 8312
Gerrit-PatchSet: 10
Gerrit-Owner: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Fri, 30 Mar 2018 16:47:21 +0000
Gerrit-HasComments: Yes

[kudu-CR] KUDU-2191 (6/n): Hive Metastore catalog manager integration

Posted by "Dan Burkert (Code Review)" <ge...@cloudera.org>.
Hello Tidy Bot, Kudu Jenkins, Adar Dembo, Todd Lipcon, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/8312

to look at the new patch set (#4).

Change subject: KUDU-2191 (6/n): Hive Metastore catalog manager integration
......................................................................

KUDU-2191 (6/n): Hive Metastore catalog manager integration

This commit adds integration with the Hive Metastore to CatalogManager,
which can be enabled with a new flag: 'hive-metastore-addresses'. When
the flag is provided, the Master will push DDL changes in its catalog to
the Hive Metastore. In particular:

- Create table, delete table, and alter table operations in the Kudu
  master will synchronously update the corresponding entry in the HMS.

- New table and column names are required to be valid according to the
  Hive identifier rules, which are much stricter than Kudu's existing
  identifier rules.

I think the code-paths in hms_catalog are pretty well covered by the
tests added in this commit, however there is a dearth of stress and
chaos tests covering the actual CatalogManager integration. In
particular I've left some TODOs of roll-back code that could benefit
from more coverage, as well as a TODO in master-stress-test about
enabling HMS integration.

Change-Id: I1b8f360408b05d6ac4f7359a8ac0dd889b196baf
---
M src/kudu/hms/hms_client-test.cc
M src/kudu/hms/hms_client.cc
M src/kudu/hms/hms_client.h
M src/kudu/hms/mini_hms.cc
M src/kudu/integration-tests/CMakeLists.txt
M src/kudu/integration-tests/alter_table-randomized-test.cc
M src/kudu/integration-tests/master-stress-test.cc
A src/kudu/integration-tests/master_hms-itest.cc
M src/kudu/master/CMakeLists.txt
M src/kudu/master/catalog_manager.cc
M src/kudu/master/catalog_manager.h
A src/kudu/master/hms_catalog-test.cc
A src/kudu/master/hms_catalog.cc
A src/kudu/master/hms_catalog.h
M src/kudu/master/master.proto
M src/kudu/mini-cluster/external_mini_cluster-test.cc
M src/kudu/mini-cluster/external_mini_cluster.cc
M src/kudu/util/net/net_util.cc
M src/kudu/util/net/net_util.h
19 files changed, 1,214 insertions(+), 10 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/12/8312/4
-- 
To view, visit http://gerrit.cloudera.org:8080/8312
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I1b8f360408b05d6ac4f7359a8ac0dd889b196baf
Gerrit-Change-Number: 8312
Gerrit-PatchSet: 4
Gerrit-Owner: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>

[kudu-CR] KUDU-2191 (4/n): WIP: Hive Metastore catalog manager integration

Posted by "Dan Burkert (Code Review)" <ge...@cloudera.org>.
Hello Tidy Bot, Kudu Jenkins, Adar Dembo, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/8312

to look at the new patch set (#2).

Change subject: KUDU-2191 (4/n): WIP: Hive Metastore catalog manager integration
......................................................................

KUDU-2191 (4/n): WIP: Hive Metastore catalog manager integration

This commit adds a basic integration with the Hive Metastore, which can
be enabled with a new flag: 'hive-metastore-addresses'. When the flag is
provided, the Master will push DDL changes in its catalog to the Hive
Metastore. In particular:

- Create table, delete table, and alter table operations in the Kudu
  master will synchronously update the corresponding entry in the HMS.

- New table and column names are required to be valid according to the
  Hive identifier rules, which are much stricter than Kudu's native
  identifier rules.

WIP: This patch doesn't yet have sufficient error handling logic, nor
test coverage of errors.  Here are the test cases I intend to add to
master_hms-itest before this is committed:

- Rename a Kudu table to a new name which already has an entry in the HMS
  (should fail).
- Rename a Kudu table when the original name doesn't have an entry in
  the HMS (should create an entry with the new name).
- Rename a Kudu table when the HMS is unreachable (should fail).
- Create a Kudu table with a name which already exists in the HMS
  (should fail).
- Create a Kudu table when the HMS is unreachable (should fail).
- Drop a Kudu table when the corresponding entry in the HMS does not
  exist (should succeed).
- Drop a Kudu table when the HMS is unreachable (should fail?).

Additionally, this patch as written overwrites the entire HMS table
entry during table rename operations. This is bad because it will wipe
out any stats that query engines like Impala have attached to the table.

Change-Id: I1b8f360408b05d6ac4f7359a8ac0dd889b196baf
---
M src/kudu/integration-tests/CMakeLists.txt
A src/kudu/integration-tests/master_hms-itest.cc
M src/kudu/master/CMakeLists.txt
M src/kudu/master/catalog_manager.cc
M src/kudu/master/catalog_manager.h
A src/kudu/master/hms_catalog.cc
A src/kudu/master/hms_catalog.h
M src/kudu/master/master.proto
M src/kudu/mini-cluster/external_mini_cluster.cc
M src/kudu/util/net/net_util.cc
M src/kudu/util/net/net_util.h
11 files changed, 664 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/12/8312/2
-- 
To view, visit http://gerrit.cloudera.org:8080/8312
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I1b8f360408b05d6ac4f7359a8ac0dd889b196baf
Gerrit-Change-Number: 8312
Gerrit-PatchSet: 2
Gerrit-Owner: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot

[kudu-CR] KUDU-2191 (6/n): Hive Metastore catalog manager integration

Posted by "Dan Burkert (Code Review)" <ge...@cloudera.org>.
Hello Tidy Bot, Kudu Jenkins, Adar Dembo, Hao Hao, Todd Lipcon, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/8312

to look at the new patch set (#5).

Change subject: KUDU-2191 (6/n): Hive Metastore catalog manager integration
......................................................................

KUDU-2191 (6/n): Hive Metastore catalog manager integration

This commit adds integration with the Hive Metastore to CatalogManager,
which can be enabled with a new flag: 'hive-metastore-addresses'. When
the flag is provided, the Master will push DDL changes in its catalog to
the Hive Metastore. In particular:

- Create table, delete table, and alter table operations in the Kudu
  master will synchronously update the corresponding entry in the HMS.

- New table and column names are required to be valid according to the
  Hive identifier rules, which are much stricter than Kudu's existing
  identifier rules.

I think the code-paths in hms_catalog are pretty well covered by the
tests added in this commit, however there is a dearth of stress and
chaos tests covering the actual CatalogManager integration. In
particular I've left some TODOs of roll-back code that could benefit
from more coverage, as well as a TODO in master-stress-test about
enabling HMS integration.

Change-Id: I1b8f360408b05d6ac4f7359a8ac0dd889b196baf
---
M src/kudu/hms/hms_client-test.cc
M src/kudu/hms/hms_client.cc
M src/kudu/hms/hms_client.h
M src/kudu/hms/mini_hms.cc
M src/kudu/integration-tests/CMakeLists.txt
M src/kudu/integration-tests/alter_table-randomized-test.cc
M src/kudu/integration-tests/master-stress-test.cc
M src/kudu/integration-tests/master_failover-itest.cc
A src/kudu/integration-tests/master_hms-itest.cc
M src/kudu/master/CMakeLists.txt
M src/kudu/master/catalog_manager.cc
M src/kudu/master/catalog_manager.h
A src/kudu/master/hms_catalog-test.cc
A src/kudu/master/hms_catalog.cc
A src/kudu/master/hms_catalog.h
M src/kudu/master/master.proto
M src/kudu/mini-cluster/external_mini_cluster-test.cc
M src/kudu/mini-cluster/external_mini_cluster.cc
M src/kudu/util/net/net_util.cc
M src/kudu/util/net/net_util.h
20 files changed, 1,263 insertions(+), 22 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/12/8312/5
-- 
To view, visit http://gerrit.cloudera.org:8080/8312
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I1b8f360408b05d6ac4f7359a8ac0dd889b196baf
Gerrit-Change-Number: 8312
Gerrit-PatchSet: 5
Gerrit-Owner: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>

[kudu-CR] KUDU-2191 (6/n): Hive Metastore catalog manager integration

Posted by "Dan Burkert (Code Review)" <ge...@cloudera.org>.
Hello Tidy Bot, Kudu Jenkins, Adar Dembo, Hao Hao, Todd Lipcon, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/8312

to look at the new patch set (#8).

Change subject: KUDU-2191 (6/n): Hive Metastore catalog manager integration
......................................................................

KUDU-2191 (6/n): Hive Metastore catalog manager integration

This commit adds integration with the Hive Metastore to CatalogManager,
which can be enabled with a new flag: 'hive-metastore-addresses'. When
the flag is provided, the Master will push DDL changes in its catalog to
the Hive Metastore. In particular:

- Create table, delete table, and alter table operations in the Kudu
  master will synchronously update the corresponding entry in the HMS.

- New table and column names are required to be valid according to the
  Hive identifier rules, which are much stricter than Kudu's existing
  identifier rules.

I think the code-paths in hms_catalog are pretty well covered by the
tests added in this commit, however there is a dearth of stress and
chaos tests covering the actual CatalogManager integration. In
particular I've left some TODOs of roll-back code that could benefit
from more coverage, as well as a TODO in master-stress-test about
enabling HMS integration.

Change-Id: I1b8f360408b05d6ac4f7359a8ac0dd889b196baf
---
M src/kudu/hms/hms_client-test.cc
M src/kudu/hms/hms_client.cc
M src/kudu/hms/hms_client.h
M src/kudu/hms/mini_hms.cc
M src/kudu/integration-tests/CMakeLists.txt
M src/kudu/integration-tests/alter_table-randomized-test.cc
M src/kudu/integration-tests/master-stress-test.cc
M src/kudu/integration-tests/master_failover-itest.cc
A src/kudu/integration-tests/master_hms-itest.cc
M src/kudu/master/CMakeLists.txt
M src/kudu/master/catalog_manager.cc
M src/kudu/master/catalog_manager.h
A src/kudu/master/hms_catalog-test.cc
A src/kudu/master/hms_catalog.cc
A src/kudu/master/hms_catalog.h
M src/kudu/master/master.proto
M src/kudu/mini-cluster/external_mini_cluster-test.cc
M src/kudu/mini-cluster/external_mini_cluster.cc
M src/kudu/util/net/net_util.cc
M src/kudu/util/net/net_util.h
20 files changed, 1,300 insertions(+), 41 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/12/8312/8
-- 
To view, visit http://gerrit.cloudera.org:8080/8312
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I1b8f360408b05d6ac4f7359a8ac0dd889b196baf
Gerrit-Change-Number: 8312
Gerrit-PatchSet: 8
Gerrit-Owner: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>