[jira] [Created] (MESOS-7648) Mesos master should not return `/state` before finishing recovering agents from registry

["Zhitao Li (JIRA)" <ji...@apache.org>]

Zhitao Li created MESOS-7648:
--------------------------------

             Summary: Mesos master should not return `/state` before finishing recovering agents from registry
                 Key: MESOS-7648
                 URL: https://issues.apache.org/jira/browse/MESOS-7648
             Project: Mesos
          Issue Type: Bug
            Reporter: Zhitao Li


We are working on relying on {{recovered_agents}} in MESOS-6177. However, we discovered that master could start to respond to {{/state.json}} endpoint before it finishes processing result from registry::recover.

The sequence seems to be registry was recovered -> /state query comes in -> recovered agents from registry.

See the following logs:

{noformat}
I0608 22:29:57.147212  6407 master.cpp:2124] Elected as the leading master!
I0608 22:29:57.147274  6407 master.cpp:1646] Recovering from registrar
I0608 22:29:57.148114  6412 log.cpp:553] Attempting to start the writer
I0608 22:29:57.149339  6411 replica.cpp:495] Replica received implicit promise request from __req_res__(2)@10.162.9.54:5050 with proposal 105
I0608 22:29:57.149860  6411 replica.cpp:344] Persisted promised to 105
I0608 22:29:57.151495  6410 coordinator.cpp:238] Coordinator attempting to fill missing positions
I0608 22:29:57.151595  6412 log.cpp:569] Writer started with ending position 36816
I0608 22:29:58.111565  6423 registrar.cpp:362] Successfully fetched the registry (1200222B) in 934048us
I0608 22:29:58.214422  6423 registrar.cpp:461] Applied 1 operations in 25.893664ms; attempting to update the registry
I0608 22:29:58.300578  6421 coordinator.cpp:348] Coordinator attempting to write APPEND action at position 36817
I0608 22:29:58.307567  6410 replica.cpp:539] Replica received write request for position 36817 from __req_res__(7)@10.162.9.54:5050
I0608 22:29:58.344857  6421 replica.cpp:693] Replica received learned notice for position 36817 from @0.0.0.0:0
I0608 22:29:58.378731  6408 coordinator.cpp:348] Coordinator attempting to write TRUNCATE action at position 36818
I0608 22:29:58.382043  6416 replica.cpp:539] Replica received write request for position 36818 from __req_res__(12)@10.162.9.54:5050
I0608 22:29:58.384946  6410 replica.cpp:693] Replica received learned notice for position 36818 from @0.0.0.0:0
I0608 22:29:59.507297  6423 registrar.cpp:506] Successfully updated the registry in 1.282937088secs
I0608 22:29:59.580960  6423 registrar.cpp:392] Successfully recovered registrar
I0608 22:29:59.940066  6415 http.cpp:420] HTTP GET for /master/state from 10.67.139.161:57197 with User-Agent='mesos-uns-bridge'
I0608 22:30:00.342932  6425 master.cpp:1762] Recovered 3549 agents from the registry (1200220B); allowing 15mins for agents to re-register
{noformat}

We found that the request corresponding to second to last line above returns 0 registered or recovered agents, thus incorrectly rendered its client to think it's an empty cluster.

[~anandmazumdar] [~vinodkone]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)