[kudu] branch master updated: [hms] allow hive.metastore.disallow.incompatible.col.type.changes to be true

[aw...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

awong pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kudu.git


The following commit(s) were added to refs/heads/master by this push:
     new ea370f1  [hms] allow hive.metastore.disallow.incompatible.col.type.changes to be true
ea370f1 is described below

commit ea370f12d6c3f9fe8de9d3e7e7e8fda1d3cad7ca
Author: Andrew Wong <aw...@cloudera.com>
AuthorDate: Fri Apr 9 12:20:18 2021 -0700

    [hms] allow hive.metastore.disallow.incompatible.col.type.changes to be true
    
    A major distribution of Hive and Kudu (CDP) has changed its default
    behavior to set hive.metastore.disallow.incompatible.col.type.changes to
    true. This is a safer behavior for Hive, but breaks the Kudu-HMS
    integration, since we require the configuration for DROP COLUMN
    operations, and check the Hive configuration as such at startup.
    
    HIVE-24987 tracks the work on the Hive-side to permit Kudu operations to
    alter tables in a seemingly type-incompatible way that is actually safe
    for Kudu.
    
    Until then, this adds an option to disable this Kudu-side check (kept
    the current behavior by default, but added a gflag to make it more
    easily configurable).
    
    Change-Id: Idc9777e134e243abfe768d631592541720e101a8
    Reviewed-on: http://gerrit.cloudera.org:8080/17296
    Reviewed-by: Andrew Wong <aw...@cloudera.com>
    Tested-by: Andrew Wong <aw...@cloudera.com>
---
 src/kudu/hms/hms_client.cc | 36 +++++++++++++++++++++++++-----------
 1 file changed, 25 insertions(+), 11 deletions(-)

diff --git a/src/kudu/hms/hms_client.cc b/src/kudu/hms/hms_client.cc
index 440dde4..63499d3 100644
--- a/src/kudu/hms/hms_client.cc
+++ b/src/kudu/hms/hms_client.cc
@@ -25,6 +25,7 @@
 #include <string>
 #include <vector>
 
+#include <gflags/gflags.h>
 #include <glog/logging.h>
 #include <thrift/TApplicationException.h>
 #include <thrift/Thrift.h>
@@ -34,6 +35,7 @@
 #include <thrift/transport/TTransport.h>
 #include <thrift/transport/TTransportException.h>
 
+#include "kudu/gutil/macros.h"
 #include "kudu/gutil/map-util.h"
 #include "kudu/gutil/strings/split.h"
 #include "kudu/gutil/strings/strip.h"
@@ -42,10 +44,20 @@
 #include "kudu/hms/hive_metastore_types.h"
 #include "kudu/thrift/client.h"
 #include "kudu/thrift/sasl_client_transport.h"
+#include "kudu/util/flag_tags.h"
 #include "kudu/util/status.h"
 #include "kudu/util/stopwatch.h"
 #include "kudu/util/string_case.h"
 
+DEFINE_bool(disable_hms_incompatible_column_type_check, false,
+            "Whether or not to disable the check for the "
+            "'hive.metastore.disallow.incompatible.col.type.changes' "
+            "configuration in the Hive Metastore. This check is important to "
+            "ensure Hive will allow dropping columns, but may not be necessary "
+            "in some versions of Hive. See HIVE-24987 for more details.");
+TAG_FLAG(disable_hms_incompatible_column_type_check, advanced);
+TAG_FLAG(disable_hms_incompatible_column_type_check, hidden);
+
 using apache::thrift::TApplicationException;
 using apache::thrift::TException;
 using apache::thrift::protocol::TJSONProtocol;
@@ -190,17 +202,19 @@ Status HmsClient::Start() {
   // operations properly.
   //
   // See org.apache.hadoop.hive.metastore.MetaStoreUtils.throwExceptionIfIncompatibleColTypeChange.
-  string disallow_incompatible_column_type_changes;
-  HMS_RET_NOT_OK(client_.get_config_value(disallow_incompatible_column_type_changes,
-                                          kDisallowIncompatibleColTypeChanges,
-                                          "false"),
-                 Substitute("failed to get Hive Metastore $0 configuration",
-                            kDisallowIncompatibleColTypeChanges));
-
-  if (iequals(disallow_incompatible_column_type_changes, "true")) {
-    return Status::IllegalState(Substitute(
-        "Hive Metastore configuration is invalid: $0 must be set to false",
-        kDisallowIncompatibleColTypeChanges));
+  if (!FLAGS_disable_hms_incompatible_column_type_check) {
+    string disallow_incompatible_column_type_changes;
+    HMS_RET_NOT_OK(client_.get_config_value(disallow_incompatible_column_type_changes,
+                                            kDisallowIncompatibleColTypeChanges,
+                                            "false"),
+                  Substitute("failed to get Hive Metastore $0 configuration",
+                              kDisallowIncompatibleColTypeChanges));
+
+    if (iequals(disallow_incompatible_column_type_changes, "true")) {
+      return Status::IllegalState(Substitute(
+          "Hive Metastore configuration is invalid: $0 must be set to false",
+          kDisallowIncompatibleColTypeChanges));
+    }
   }
 
   // Check that the HMS is configured to add the entire thrift Table/Partition