[jira] [Commented] (AURORA-1785) Populate curator latches with scheduler information

["John Sirois (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/AURORA-1785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15536188#comment-15536188 ] 

John Sirois commented on AURORA-1785:
-------------------------------------

This can be done using the [{{LeaderLatch.id}}|https://curator.apache.org/apidocs/org/apache/curator/framework/recipes/leader/LeaderLatch.html#LeaderLatch-org.apache.curator.framework.CuratorFramework-java.lang.String-java.lang.String-].  A quick sketch:
{noformat}
jsirois@gill ~/dev/apache/jsirois-aurora (jsirois/issues/AURORA-1784 *) $ git diff
diff --git a/src/main/java/org/apache/aurora/scheduler/discovery/CuratorServiceDiscoveryModule.java b/src/main/java/org/apache/aurora/scheduler/discovery/CuratorServiceDiscoveryModule.java
index 6ccfef4..67b750a 100644
--- a/src/main/java/org/apache/aurora/scheduler/discovery/CuratorServiceDiscoveryModule.java
+++ b/src/main/java/org/apache/aurora/scheduler/discovery/CuratorServiceDiscoveryModule.java
@@ -14,6 +14,7 @@
 package org.apache.aurora.scheduler.discovery;
 
 import java.net.InetSocketAddress;
+import java.net.UnknownHostException;
 import java.util.List;
 import java.util.function.Predicate;
 
@@ -179,7 +180,9 @@ class CuratorServiceDiscoveryModule extends PrivateModule {
   @Provides
   @Singleton
   @Exposed
-  SingletonService provideSingletonService(CuratorFramework client, Codec<ServiceInstance> codec) {
+  SingletonService provideSingletonService(CuratorFramework client, Codec<ServiceInstance> codec)
+      throws UnknownHostException {
+
     return new CuratorSingletonService(client, discoveryPath, MEMBER_TOKEN, codec);
   }
 }
diff --git a/src/main/java/org/apache/aurora/scheduler/discovery/CuratorSingletonService.java b/src/main/java/org/apache/aurora/scheduler/discovery/CuratorSingletonService.java
index c9bd1eb..5b0b1d5 100644
--- a/src/main/java/org/apache/aurora/scheduler/discovery/CuratorSingletonService.java
+++ b/src/main/java/org/apache/aurora/scheduler/discovery/CuratorSingletonService.java
@@ -15,7 +15,9 @@ package org.apache.aurora.scheduler.discovery;
 
 import java.io.ByteArrayOutputStream;
 import java.io.IOException;
+import java.net.InetAddress;
 import java.net.InetSocketAddress;
+import java.net.UnknownHostException;
 import java.util.Map;
 import java.util.concurrent.TimeUnit;
 
@@ -129,14 +131,16 @@ class CuratorSingletonService implements SingletonService {
    * @param groupPath The root ZooKeeper path service members advertise their presence under.
    * @param memberToken A token used to form service member node names.
    * @param codec A codec that can be used to deserialize group member {@link ServiceInstance} data.
+   * @throws UnknownHostException if there is a problem preparing the leader proposal.
    */
   CuratorSingletonService(
       CuratorFramework client,
       String groupPath,
       String memberToken,
-      Codec<ServiceInstance> codec) {
+      Codec<ServiceInstance> codec)
+      throws UnknownHostException {
 
-    leaderLatch = new LeaderLatch(client, groupPath);
+    leaderLatch = new LeaderLatch(client, groupPath, InetAddress.getLocalHost().getHostAddress());
     advertiser = new Advertiser(client, groupPath, memberToken, codec);
     this.groupPath = PathUtils.validatePath(groupPath);
   }
{noformat}

That gives:
{noformat}
vagrant@aurora:~$ zk_shell/bin/zk-shell localhost --run-once "ls aurora/scheduler"
_c_88e33306-6efe-4202-8065-0fa991eb67c7-latch-0000000097
member_0000000098
vagrant@aurora:~$ zk_shell/bin/zk-shell localhost --run-once "get aurora/scheduler/_c_88e33306-6efe-4202-8065-0fa991eb67c7-latch-0000000097"
127.0.0.1
{noformat}

Back to you Zameer.

> Populate curator latches with scheduler information
> ---------------------------------------------------
>
>                 Key: AURORA-1785
>                 URL: https://issues.apache.org/jira/browse/AURORA-1785
>             Project: Aurora
>          Issue Type: Task
>            Reporter: Zameer Manji
>            Assignee: John Sirois
>            Priority: Minor
>
> (Assigning this to John, who is our curator expert for triage/feasibility)
> If you look at the mesos ZK node for leader election you see something like this:
> {noformat}
>  u'json.info_0000000104',
>  u'json.info_0000000102',
>  u'json.info_0000000101',
>  u'json.info_0000000098',
>  u'json.info_0000000097'
> {noformat}
> Each of these nodes contains data about the machine contending for leadership. It is a JSON serialized {{MasterInfo}} protobuf. This means an operator can inspect who is contending for leadership by checking the content of the nodes.
> When you check the aurora ZK node you see something like this:
> {noformat}
>  u'_c_2884a0d3-b5b0-4445-b8d6-b271a6df6220-latch-0000000774',
>  u'_c_86a21335-c5a2-4bcb-b471-4ce128b67616-latch-0000000776',
>  u'_c_a4f8b0f7-d063-4df2-958b-7b3e6f666a95-latch-0000000775',
>  u'_c_120cd9da-3bc1-495b-b02f-2142fb22c0a0-latch-0000000784',
>  u'_c_46547c31-c5c2-4fb1-8a53-237e3cb0292f-latch-0000000780',
>  u'member_0000000781'
> {noformat}
> Only the leader node contains information. The curator latches contain no information. It is not possible to figure out which machines are contending for leadership purely from ZK.
> I think we should attach data to the latches like mesos.
> Being able to do this is invaluable to debug issues if an extra master is added to the cluster.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)