[jira] [Work logged] (HIVE-26509) Introduce dynamic leader election in HMS

["ASF GitHub Bot (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HIVE-26509?focusedWorklogId=818591&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-818591 ]

ASF GitHub Bot logged work on HIVE-26509:
-----------------------------------------

                Author: ASF GitHub Bot
            Created on: 19/Oct/22 20:00
            Start Date: 19/Oct/22 20:00
    Worklog Time Spent: 10m 
      Work Description: saihemanth-cloudera commented on code in PR #3567:
URL: https://github.com/apache/hive/pull/3567#discussion_r999881750


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/leader/AuditLeaderListener.java:
##########
@@ -0,0 +1,124 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore.leader;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hive.common.TableName;
+import org.apache.hadoop.hive.metastore.ColumnType;
+import org.apache.hadoop.hive.metastore.HiveMetaStore;
+import org.apache.hadoop.hive.metastore.IHMSHandler;
+import org.apache.hadoop.hive.metastore.Warehouse;
+import org.apache.hadoop.hive.metastore.api.AlreadyExistsException;
+import org.apache.hadoop.hive.metastore.api.PrincipalType;
+import org.apache.hadoop.hive.metastore.api.Table;
+import org.apache.hadoop.hive.metastore.client.builder.TableBuilder;
+import org.apache.hadoop.hive.metastore.utils.SecurityUtils;
+
+import java.io.OutputStream;
+import java.net.InetAddress;
+import java.net.UnknownHostException;
+import java.nio.charset.StandardCharsets;
+import java.time.LocalDateTime;
+import java.time.format.DateTimeFormatter;
+
+import static java.util.Objects.requireNonNull;
+
+public class AuditLeaderListener implements LeaderElection.LeadershipStateListener {
+  private final Configuration configuration;
+
+  private final Path tableLocation;
+
+  private final static String SERDE = "org.apache.hadoop.hive.serde2.JsonSerDe";
+  private final static String INPUTFORMAT = "org.apache.hadoop.mapred.TextInputFormat";
+  private final static String OUTPUTFORMAT = "org.apache.hadoop.hive.ql.io.IgnoreKeyTextOutputFormat";
+
+  public AuditLeaderListener(TableName tableName, IHMSHandler handler) throws Exception {
+    requireNonNull(tableName, "tableName is null");
+    requireNonNull(handler, "handler is null");
+    this.configuration = handler.getConf();
+    try {
+      // Store the leader info as json + text for human-readable
+      Table table = new TableBuilder()
+          .setCatName(tableName.getCat())

Review Comment:
   Where is this audit leader listener called? Just want to understand if catalog will be set in the TableName or ifllll we need to set it if catlogname is null. 





Issue Time Tracking
-------------------

    Worklog Id:     (was: 818591)
    Time Spent: 5h  (was: 4h 50m)

> Introduce dynamic leader election in HMS
> ----------------------------------------
>
>                 Key: HIVE-26509
>                 URL: https://issues.apache.org/jira/browse/HIVE-26509
>             Project: Hive
>          Issue Type: New Feature
>          Components: Standalone Metastore
>            Reporter: Zhihua Deng
>            Assignee: Zhihua Deng
>            Priority: Major
>              Labels: pull-request-available
>          Time Spent: 5h
>  Remaining Estimate: 0h
>
> From HIVE-21841 we have a leader HMS selected by configuring metastore.housekeeping.leader.hostname on startup. This approach saves us from running duplicated HMS's housekeeping tasks cluster-wide. 
> In this jira, we introduce another dynamic leader election: adopt hive lock to implement the leader election. Once a HMS owns the lock, then it becomes the leader, carries out the housekeeping tasks, and sends heartbeats to renew the lock before timeout. If the leader fails to reclaim the lock, then stops the already started tasks if it has, the electing event is audited. We can achieve a more dynamic leader when the original goes down or in the public cloud without well configured property, and reduce the leader’s burdens by running these tasks among different leaders.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)