You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@mesos.apache.org by "Alex Kaplan (JIRA)" <ji...@apache.org> on 2016/09/22 21:34:20 UTC
[jira] [Comment Edited] (MESOS-6233) Unable to start Mesos Master
[ https://issues.apache.org/jira/browse/MESOS-6233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15514527#comment-15514527 ]
Alex Kaplan edited comment on MESOS-6233 at 9/22/16 9:33 PM:
-------------------------------------------------------------
Additional logs:
{code}
Sep 22 20:03:25 <agent_id> mesos-master[x.x78]: I0922 20:03:25.194582 104504 replica.cpp:673] Replica in EMPTY status received a broadcasted recover request from (20)@x.x.9.1:5050
Sep 22 20:03:25 <agent_id> mesos-master[x.x78]: I0922 20:03:25.194675 104520 recover.cpp:197] Received a recover response from a replica in EMPTY status
Sep 22 20:03:25 <agent_id> mesos-master[x.x78]: I0922 20:03:25.349588 104504 replica.cpp:673] Replica in EMPTY status received a broadcasted recover request from (23)@x.x.9.1:5050
Sep 22 20:03:25 <agent_id> mesos-master[x.x78]: I0922 20:03:25.349642 104512 recover.cpp:197] Received a recover response from a replica in EMPTY status
Sep 22 20:03:41 <agent_id> mesos-master[x.x78]: I0922 20:03:41.352486 104529 recover.cpp:110] Unable to finish the recover protocol in 10secs, retrying
Sep 22 20:03:41 <agent_id> mesos-master[x.x78]: I0922 20:03:41.352586 104529 recover.cpp:110] Unable to finish the recover protocol in 10secs, retrying
Sep 22 20:03:41 <agent_id> mesos-master[x.x78]: I0922 20:03:41.352995 104510 replica.cpp:673] Replica in EMPTY status received a broadcasted recover request from (26)@x.x.9.1:5050
Sep 22 20:03:41 <agent_id> mesos-master[x.x78]: I0922 20:03:41.353251 104525 recover.cpp:197] Received a recover response from a replica in EMPTY status
Sep 22 20:03:41 <agent_id> mesos-master[x.x78]: I0922 20:03:41.353310 104510 replica.cpp:673] Replica in EMPTY status received a broadcasted recover request from (29)@x.x.9.1:5050
Sep 22 20:03:41 <agent_id> mesos-master[x.x78]: I0922 20:03:41.353536 x.x94 recover.cpp:197] Received a recover response from a replica in EMPTY status
Sep 22 20:03:57 <agent_id> mesos-master[x.x78]: I0922 20:03:57.914526 104529 recover.cpp:110] Unable to finish the recover protocol in 10secs, retrying
Sep 22 20:03:57 <agent_id> mesos-master[x.x78]: I0922 20:03:57.915133 104529 recover.cpp:110] Unable to finish the recover protocol in 10secs, retrying
Sep 22 20:03:57 <agent_id> mesos-master[x.x78]: I0922 20:03:57.915585 104502 replica.cpp:673] Replica in EMPTY status received a broadcasted recover request from (32)@x.x.9.1:5050
Sep 22 20:03:57 <agent_id> mesos-master[x.x78]: I0922 20:03:57.915774 104525 recover.cpp:197] Received a recover response from a replica in EMPTY status
Sep 22 20:03:57 <agent_id> mesos-master[x.x78]: I0922 20:03:57.915876 104514 replica.cpp:673] Replica in EMPTY status received a broadcasted recover request from (35)@x.x.9.1:5050
Sep 22 20:03:57 <agent_id> mesos-master[x.x78]: I0922 20:03:57.916045 104510 recover.cpp:197] Received a recover response from a replica in EMPTY status
Sep 22 20:04:21 <agent_id> mesos-master[x.x78]: I0922 20:04:21.397429 104529 recover.cpp:110] Unable to finish the recover protocol in 10secs, retrying
Sep 22 20:04:21 <agent_id> mesos-master[x.x78]: I0922 20:04:21.397532 104529 recover.cpp:110] Unable to finish the recover protocol in 10secs, retrying
Sep 22 20:04:21 <agent_id> mesos-master[x.x78]: I0922 20:04:21.397794 104523 replica.cpp:673] Replica in EMPTY status received a broadcasted recover request from (38)@x.x.9.1:5050
Sep 22 20:04:21 <agent_id> mesos-master[x.x78]: I0922 20:04:21.397994 x.x82 replica.cpp:673] Replica in EMPTY status received a broadcasted recover request from (13)@x.x.11.1:5050
Sep 22 20:04:21 <agent_id> mesos-master[x.x78]: I0922 20:04:21.398031 x.x82 replica.cpp:673] Replica in EMPTY status received a broadcasted recover request from (41)@x.x.9.1:5050
Sep 22 20:04:21 <agent_id> mesos-master[x.x78]: I0922 20:04:21.398175 x.x87 recover.cpp:197] Received a recover response from a replica in EMPTY status
Sep 22 20:04:21 <agent_id> mesos-master[x.x78]: I0922 20:04:21.398236 104516 recover.cpp:197] Received a recover response from a replica in EMPTY status
Sep 22 20:04:29 <agent_id> mesos-master[x.x78]: I0922 20:04:29.4x.x6 x.x97 replica.cpp:673] Replica in EMPTY status received a broadcasted recover request from (12)@x.x.11.1:5050
Sep 22 20:04:29 <agent_id> mesos-master[x.x78]: I0922 20:04:29.449352 x.x90 replica.cpp:673] Replica in EMPTY status received a broadcasted recover request from (14)@x.x.11.1:5050
Sep 22 20:04:29 <agent_id> mesos-master[x.x78]: I0922 20:04:29.449399 x.x93 replica.cpp:673] Replica in EMPTY status received a broadcasted recover request from (22)@x.x.11.1:5050
Sep 22 20:04:29 <agent_id> mesos-master[x.x78]: I0922 20:04:29.872488 x.x84 manager.cpp:201] Overlay master overlay-master@x.x.11.1:5050 has exited
Sep 22 20:04:29 <agent_id> mesos-master[x.x78]: W0922 20:04:29.872529 x.x84 manager.cpp:204] Overlay master disconnected! Waiting for a new overlay master to be detected
Sep 22 20:04:29 <agent_id> mesos-master[x.x78]: I0922 20:04:29.872550 x.x84 manager.cpp:208] Moving overlay-master@x.x.11.1:5050 to `REGISTERING` state.
Sep 22 20:04:37 <agent_id> mesos-master[x.x78]: I0922 20:04:37.898635 104507 replica.cpp:673] Replica in EMPTY status received a broadcasted recover request from (16)@x.x.11.1:5050
Sep 22 20:04:37 <agent_id> mesos-master[x.x78]: I0922 20:04:37.899080 104529 recover.cpp:110] Unable to finish the recover protocol in 10secs, retrying
Sep 22 20:04:37 <agent_id> mesos-master[x.x78]: I0922 20:04:37.899253 104529 recover.cpp:110] Unable to finish the recover protocol in 10secs, retrying
Sep 22 20:04:37 <agent_id> mesos-master[x.x78]: I0922 20:04:37.899842 x.x90 replica.cpp:673] Replica in EMPTY status received a broadcasted recover request from (48)@x.x.9.1:5050
Sep 22 20:04:37 <agent_id> mesos-master[x.x78]: I0922 20:04:37.900182 104526 recover.cpp:197] Received a recover response from a replica in EMPTY status
Sep 22 20:04:37 <agent_id> mesos-master[x.x78]: I0922 20:04:37.900218 x.x84 replica.cpp:673] Replica in EMPTY status received a broadcasted recover request from (50)@x.x.9.1:5050
Sep 22 20:04:37 <agent_id> mesos-master[x.x78]: I0922 20:04:37.900280 x.x98 recover.cpp:197] Received a recover response from a replica in EMPTY status
Sep 22 20:04:37 <agent_id> mesos-master[x.x78]: E0922 20:04:37.900727 104529 process.cpp:2104] Failed to shutdown socket with fd 46: Transport endpoint is not connected
Sep 22 20:04:40 <agent_id> mesos-master[x.x78]: I0922 20:04:40.013473 104525 network.hpp:430] ZooKeeper group memberships changed
Sep 22 20:04:40 <agent_id> mesos-master[x.x78]: I0922 20:04:40.013525 104505 group.cpp:706] Trying to get '/mesos/overlay_log_replicas/0000003289' in ZooKeeper
Sep 22 20:04:40 <agent_id> mesos-master[x.x78]: I0922 20:04:40.013557 104520 detector.cpp:152] Detected a new leader: (id='2760')
Sep 22 20:04:40 <agent_id> mesos-master[x.x78]: I0922 20:04:40.013662 104523 network.hpp:430] ZooKeeper group memberships changed
Sep 22 20:04:40 <agent_id> mesos-master[x.x78]: I0922 20:04:40.013720 104528 detector.cpp:152] Detected a new leader: (id='2760')
Sep 22 20:04:40 <agent_id> mesos-master[x.x78]: I0922 20:04:40.013746 104503 group.cpp:706] Trying to get '/mesos/json.info_0000002760' in ZooKeeper
Sep 22 20:04:40 <agent_id> mesos-master[x.x78]: I0922 20:04:40.013777 x.x95 group.cpp:706] Trying to get '/mesos/json.info_0000002760' in ZooKeeper
Sep 22 20:04:40 <agent_id> mesos-master[x.x78]: I0922 20:04:40.013813 x.x86 group.cpp:706] Trying to get '/mesos/log_replicas/0000003284' in ZooKeeper
Sep 22 20:04:40 <agent_id> mesos-master[x.x78]: I0922 20:04:40.014015 104528 network.hpp:478] ZooKeeper group PIDs: { log-replica(2)@x.x.9.1:5050 }
Sep 22 20:04:40 <agent_id> mesos-master[x.x78]: I0922 20:04:40.014298 x.x97 network.hpp:478] ZooKeeper group PIDs: { log-replica(1)@x.x.9.1:5050 }
Sep 22 20:04:40 <agent_id> mesos-master[x.x78]: I0922 20:04:40.014318 x.x99 zookeeper.cpp:259] A new leading master (UPID=master@x.x.9.1:5050) is detected
Sep 22 20:04:40 <agent_id> mesos-master[x.x78]: I0922 20:04:40.014444 104526 zookeeper.cpp:259] A new leading master (UPID=master@x.x.9.1:5050) is detected
Sep 22 20:04:40 <agent_id> mesos-master[x.x78]: I0922 20:04:40.014487 x.x90 master.cpp:1847] The newly elected leader is master@x.x.9.1:5050 with id 184cd1f8-d3ff-4c90-8559-0ac41ea81616
Sep 22 20:04:40 <agent_id> mesos-master[x.x78]: I0922 20:04:40.014503 x.x90 master.cpp:1860] Elected as the leading master!
Sep 22 20:04:40 <agent_id> mesos-master[x.x78]: I0922 20:04:40.014504 x.x87 manager.cpp:397] Detected new overlay master at overlay-master@x.x.9.1:5050
Sep 22 20:04:40 <agent_id> mesos-master[x.x78]: I0922 20:04:40.014511 x.x90 master.cpp:1547] Recovering from registrar
Sep 22 20:04:40 <agent_id> mesos-master[x.x78]: I0922 20:04:40.014555 x.x86 manager.cpp:782] Got registration from pid: overlay-agent@x.x.9.1:5050
Sep 22 20:04:40 <agent_id> mesos-master[x.x78]: I0922 20:04:40.014582 x.x86 manager.cpp:787] overlay-master moving to `RECOVERING` state . Hence, not sending an update to agentoverlay-agent@x.x.9.1:5050
Sep 22 20:04:40 <agent_id> mesos-master[x.x78]: I0922 20:04:40.014590 104522 registrar.cpp:332] Recovering registrar
Sep 22 20:04:45 <agent_id> mesos-master[x.x78]: I0922 20:04:45.911886 104525 logfmt.cpp:164] dstip=x.x.9.1 type=audit timestamp=2016-09-22 20:04:45.911847936+00:00 reason="Valid authorization token" uid="dcos_mesos_dns" object="/master/state.json" authorizer="mesos-master" action="GET" result=allow srcip=x.x.11.1 dstport=5050 srcport=51602
Sep 22 20:04:45 <agent_id> mesos-master[x.x78]: I0922 20:04:45.912039 x.x84 http.cpp:381] HTTP GET for /master/state.json from x.x.11.1:51602 with User-Agent='Mesos-DNS'
Sep 22 20:04:53 <agent_id> mesos-master[x.x78]: I0922 20:04:53.121378 104522 network.hpp:430] ZooKeeper group memberships changed
Sep 22 20:04:53 <agent_id> mesos-master[x.x78]: I0922 20:04:53.121448 104523 group.cpp:706] Trying to get '/mesos/overlay_log_replicas/0000003289' in ZooKeeper
Sep 22 20:04:53 <agent_id> mesos-master[x.x78]: I0922 20:04:53.121789 104523 group.cpp:706] Trying to get '/mesos/overlay_log_replicas/0000003291' in ZooKeeper
Sep 22 20:04:53 <agent_id> mesos-master[x.x78]: I0922 20:04:53.122149 104502 network.hpp:478] ZooKeeper group PIDs: { log-replica(1)@x.x.11.1:5050, log-replica(1)@x.x.9.1:5050 }
Sep 22 20:04:53 <agent_id> mesos-master[x.x78]: I0922 20:04:53.309017 104509 network.hpp:430] ZooKeeper group memberships changed
Sep 22 20:04:53 <agent_id> mesos-master[x.x78]: I0922 20:04:53.309114 104502 group.cpp:706] Trying to get '/mesos/log_replicas/0000003284' in ZooKeeper
Sep 22 20:04:53 <agent_id> mesos-master[x.x78]: I0922 20:04:53.309340 104502 group.cpp:706] Trying to get '/mesos/log_replicas/0000003286' in ZooKeeper
Sep 22 20:04:53 <agent_id> mesos-master[x.x78]: I0922 20:04:53.309658 x.x81 network.hpp:478] ZooKeeper group PIDs: { log-replica(2)@x.x.11.1:5050, log-replica(2)@x.x.9.1:5050 }
Sep 22 20:04:53 <agent_id> mesos-master[x.x78]: I0922 20:04:53.732125 x.x89 network.hpp:430] ZooKeeper group memberships changed
Sep 22 20:04:53 <agent_id> mesos-master[x.x78]: I0922 20:04:53.732303 104507 group.cpp:706] Trying to get '/mesos/overlay_log_replicas/0000003289' in ZooKeeper
Sep 22 20:04:53 <agent_id> mesos-master[x.x78]: I0922 20:04:53.732770 104507 group.cpp:706] Trying to get '/mesos/overlay_log_replicas/0000003291' in ZooKeeper
Sep 22 20:04:53 <agent_id> mesos-master[x.x78]: I0922 20:04:53.733115 104507 group.cpp:706] Trying to get '/mesos/overlay_log_replicas/0000003292' in ZooKeeper
Sep 22 20:04:53 <agent_id> mesos-master[x.x78]: I0922 20:04:53.733594 x.x88 network.hpp:478] ZooKeeper group PIDs: { log-replica(1)@x.x.11.1:5050, log-replica(1)@x.x.9.1:5050, log-replica(1)@x.x.11.1:5050 }
Sep 22 20:04:53 <agent_id> mesos-master[x.x78]: I0922 20:04:53.886611 104503 network.hpp:430] ZooKeeper group memberships changed
Sep 22 20:04:53 <agent_id> mesos-master[x.x78]: I0922 20:04:53.886776 x.x99 group.cpp:706] Trying to get '/mesos/log_replicas/0000003284' in ZooKeeper
Sep 22 20:04:53 <agent_id> mesos-master[x.x78]: I0922 20:04:53.887078 x.x99 group.cpp:706] Trying to get '/mesos/log_replicas/0000003286' in ZooKeeper
Sep 22 20:04:53 <agent_id> mesos-master[x.x78]: I0922 20:04:53.887272 x.x99 group.cpp:706] Trying to get '/mesos/log_replicas/0000003287' in ZooKeeper
Sep 22 20:04:53 <agent_id> mesos-master[x.x78]: I0922 20:04:53.887572 104500 network.hpp:478] ZooKeeper group PIDs: { log-replica(2)@x.x.11.1:5050, log-replica(2)@x.x.9.1:5050, log-replica(2)@x.x.11.1:5050 }
Sep 22 20:04:53 <agent_id> mesos-master[x.x78]: I0922 20:04:53.919270 104529 recover.cpp:110] Unable to finish the recover protocol in 10secs, retrying
Sep 22 20:04:53 <agent_id> mesos-master[x.x78]: I0922 20:04:53.919317 104528 manager.cpp:782] Got registration from pid: overlay-agent@x.x.9.1:5050
Sep 22 20:04:53 <agent_id> mesos-master[x.x78]: I0922 20:04:53.919333 104529 recover.cpp:110] Unable to finish the recover protocol in 10secs, retrying
Sep 22 20:04:53 <agent_id> mesos-master[x.x78]: I0922 20:04:53.919342 104528 manager.cpp:795] overlay-master in `RECOVERING` state . Hence, not sending an update to agentoverlay-agent@x.x.9.1:5050
Sep 22 20:04:53 <agent_id> mesos-master[x.x78]: I0922 20:04:53.919360 104528 manager.cpp:782] Got registration from pid: overlay-agent@x.x.9.1:5050
Sep 22 20:04:53 <agent_id> mesos-master[x.x78]: I0922 20:04:53.919368 104528 manager.cpp:795] overlay-master in `RECOVERING` state . Hence, not sending an update to agentoverlay-agent@x.x.9.1:5050
Sep 22 20:04:53 <agent_id> mesos-master[x.x78]: I0922 20:04:53.921273 104523 logfmt.cpp:164] dstip=x.x.9.1 type=audit timestamp=2016-09-22 20:04:53.921229056+00:00 reason="Valid authorization token" uid="dcos_mesos_dns" object="/master/state.json" authorizer="mesos-master" action="GET" result=allow srcip=x.x.11.1 dstport=5050 srcport=34076
Sep 22 20:04:53 <agent_id> mesos-master[x.x78]: I0922 20:04:53.921572 104523 logfmt.cpp:164] dstip=x.x.9.1 type=audit timestamp=2016-09-22 20:04:53.921534976+00:00 reason="Valid authorization token" uid="dcos_mesos_dns" object="/master/state.json" authorizer="mesos-master" action="GET" result=allow srcip=x.x.9.1 dstport=5050 srcport=35372
Sep 22 20:04:53 <agent_id> mesos-master[x.x78]: I0922 20:04:53.921723 x.x86 http.cpp:381] HTTP GET for /master/state.json from x.x.9.1:35372 with User-Agent='Mesos-DNS'
Sep 22 20:04:53 <agent_id> mesos-master[x.x78]: I0922 20:04:53.921826 x.x86 http.cpp:381] HTTP GET for /master/state.json from x.x.11.1:34076 with User-Agent='Mesos-DNS'
Sep 22 20:05:08 <agent_id> mesos-master[x.x78]: I0922 20:05:08.023877 104522 network.hpp:430] ZooKeeper group memberships changed
Sep 22 20:05:08 <agent_id> mesos-master[x.x78]: I0922 20:05:08.023988 x.x82 network.hpp:430] ZooKeeper group memberships changed
Sep 22 20:05:08 <agent_id> mesos-master[x.x78]: I0922 20:05:08.024041 x.x88 group.cpp:706] Trying to get '/mesos/overlay_log_replicas/0000003289' in ZooKeeper
Sep 22 20:05:08 <agent_id> mesos-master[x.x78]: I0922 20:05:08.024093 x.x87 group.cpp:706] Trying to get '/mesos/log_replicas/0000003284' in ZooKeeper
Sep 22 20:05:08 <agent_id> mesos-master[x.x78]: I0922 20:05:08.024291 x.x88 group.cpp:706] Trying to get '/mesos/overlay_log_replicas/0000003292' in ZooKeeper
Sep 22 20:05:08 <agent_id> mesos-master[x.x78]: I0922 20:05:08.024492 x.x87 group.cpp:706] Trying to get '/mesos/log_replicas/0000003287' in ZooKeeper
Sep 22 20:05:08 <agent_id> mesos-master[x.x78]: I0922 20:05:08.024902 104511 network.hpp:478] ZooKeeper group PIDs: { log-replica(1)@x.x.9.1:5050, log-replica(1)@x.x.11.1:5050 }
Sep 22 20:05:08 <agent_id> mesos-master[x.x78]: I0922 20:05:08.025151 104506 network.hpp:478] ZooKeeper group PIDs: { log-replica(2)@x.x.9.1:5050, log-replica(2)@x.x.11.1:5050 }
Sep 22 20:05:12 <agent_id> mesos-master[x.x78]: I0922 20:05:12.016031 104522 network.hpp:430] ZooKeeper group memberships changed
Sep 22 20:05:12 <agent_id> mesos-master[x.x78]: I0922 20:05:12.016280 104527 network.hpp:430] ZooKeeper group memberships changed
Sep 22 20:05:12 <agent_id> mesos-master[x.x78]: I0922 20:05:12.016337 x.x98 group.cpp:706] Trying to get '/mesos/log_replicas/0000003284' in ZooKeeper
Sep 22 20:05:12 <agent_id> mesos-master[x.x78]: I0922 20:05:12.016345 x.x82 group.cpp:706] Trying to get '/mesos/overlay_log_replicas/0000003289' in ZooKeeper
Sep 22 20:05:12 <agent_id> mesos-master[x.x78]: I0922 20:05:12.016741 104522 network.hpp:478] ZooKeeper group PIDs: { log-replica(1)@x.x.9.1:5050 }
Sep 22 20:05:12 <agent_id> mesos-master[x.x78]: I0922 20:05:12.016783 104510 network.hpp:478] ZooKeeper group PIDs: { log-replica(2)@x.x.9.1:5050 }
Sep 22 20:05:17 <agent_id> mesos-master[x.x78]: I0922 20:05:17.947398 104529 recover.cpp:110] Unable to finish the recover protocol in 10secs, retrying
Sep 22 20:05:17 <agent_id> mesos-master[x.x78]: I0922 20:05:17.947479 104529 recover.cpp:110] Unable to finish the recover protocol in 10secs, retrying
Sep 22 20:05:17 <agent_id> mesos-master[x.x78]: I0922 20:05:17.947873 104511 manager.cpp:782] Got registration from pid: overlay-agent@x.x.9.1:5050
Sep 22 20:05:17 <agent_id> mesos-master[x.x78]: I0922 20:05:17.947899 104511 manager.cpp:795] overlay-master in `RECOVERING` state . Hence, not sending an update to agentoverlay-agent@x.x.9.1:5050
Sep 22 20:05:17 <agent_id> mesos-master[x.x78]: I0922 20:05:17.947917 104511 manager.cpp:782] Got registration from pid: overlay-agent@x.x.9.1:5050
Sep 22 20:05:17 <agent_id> mesos-master[x.x78]: I0922 20:05:17.947926 104511 manager.cpp:795] overlay-master in `RECOVERING` state . Hence, not sending an update to agentoverlay-agent@x.x.9.1:5050
Sep 22 20:05:17 <agent_id> mesos-master[x.x78]: I0922 20:05:17.947978 x.x95 manager.cpp:782] Got registration from pid: overlay-agent@x.x.4.1:5051
Sep 22 20:05:17 <agent_id> mesos-master[x.x78]: I0922 20:05:17.948004 x.x95 manager.cpp:795] overlay-master in `RECOVERING` state . Hence, not sending an update to agentoverlay-agent@x.x.4.1:5051
Sep 22 20:05:17 <agent_id> mesos-master[x.x78]: F0922 20:05:17.948120 104529 process.cpp:2243] Check failed: sockets.count(from_fd) > 0
Sep 22 20:05:17 <agent_id> mesos-master[x.x78]: *** Check failure stack trace: ***
Sep 22 20:05:17 <agent_id> mesos-master[x.x78]: @ 0x7fc1908829fd google::LogMessage::Fail()
Sep 22 20:05:17 <agent_id> mesos-master[x.x78]: @ 0x7fc19088482d google::LogMessage::SendToLog()
Sep 22 20:05:17 <agent_id> mesos-master[x.x78]: @ 0x7fc1908825ec google::LogMessage::Flush()
Sep 22 20:05:17 <agent_id> mesos-master[x.x78]: @ 0x7fc190885129 google::LogMessageFatal::~LogMessageFatal()
Sep 22 20:05:17 <agent_id> mesos-master[x.x78]: @ 0x7fc1908171dd process::SocketManager::swap_implementing_socket()
Sep 22 20:05:17 <agent_id> mesos-master[x.x78]: @ 0x7fc19081aa90 process::SocketManager::link_connect()
Sep 22 20:05:17 <agent_id> mesos-master[x.x78]: @ 0x7fc1908227f9 _ZNSt17_Function_handlerIFvRKN7process6FutureI7NothingEEEZNKS3_5onAnyISt5_BindIFSt7_Mem_fnIMNS0_13SocketManagerEFvS5_NS0_7network6SocketERKNS0_4UPIDEEEPSA_St12_PlaceholderILi1EESC_SD_EEvEES5_OT_NS3_6PreferEEUlS5_E_E9_M_invokeERKSt9_Any_dataS5_
Sep 22 20:05:17 <agent_id> mesos-master[x.x78]: @ 0x41eb26 _ZN7process8internal3runISt8functionIFvRKNS_6FutureI7NothingEEEEJRS5_EEEvRKSt6vectorIT_SaISC_EEDpOT0_
Sep 22 20:05:17 <agent_id> mesos-master[x.x78]: @ 0x42a36f process::Future<>::fail()
Sep 22 20:05:17 <agent_id> mesos-master[x.x78]: @ 0x7fc19085283c process::network::LibeventSSLSocketImpl::event_callback()
Sep 22 20:05:17 <agent_id> mesos-master[x.x78]: @ 0x7fc190852f17 process::network::LibeventSSLSocketImpl::event_callback()
Sep 22 20:05:17 <agent_id> mesos-master[x.x78]: @ 0x7fc18d616631 bufferevent_run_deferred_callbacks_locked
Sep 22 20:05:17 <agent_id> mesos-master[x.x78]: @ 0x7fc18d60cc5d event_base_loop
Sep 22 20:05:17 <agent_id> mesos-master[x.x78]: @ 0x7fc190865a1d process::EventLoop::run()
Sep 22 20:05:17 <agent_id> mesos-master[x.x78]: @ 0x7fc18eeabd73 (unknown)
Sep 22 20:05:17 <agent_id> mesos-master[x.x78]: @ 0x7fc18e6a852c (unknown)
Sep 22 20:05:17 <agent_id> mesos-master[x.x78]: @ 0x7fc18e3e61dd (unknown)
Sep 22 20:05:18 <agent_id> systemd[1]: [0;1;39mdcos-mesos-master.service: Main process exited, code=killed, status=6/ABRT[0m
Sep 22 20:05:18 <agent_id> systemd[1]: [0;1;39mdcos-mesos-master.service: Unit entered failed state.[0m
Sep 22 20:05:18 <agent_id> systemd[1]: [0;1;39mdcos-mesos-master.service: Failed with result 'signal'.[0m
Sep 22 20:05:33 <agent_id> systemd[1]: dcos-mesos-master.service: Service hold-off time over, scheduling restart.
Sep 22 20:05:33 <agent_id> systemd[1]: Stopped Mesos Master: DC/OS Mesos Master Service.
Sep 22 20:05:33 <agent_id> systemd[1]: Starting Mesos Master: DC/OS Mesos Master Service...
Sep 22 20:05:33 <agent_id> ping[104808]: PING ready.spartan (127.0.0.1) 56(84) bytes of data.
Sep 22 20:05:33 <agent_id> ping[104808]: 64 bytes from <agent_id> (127.0.0.1): icmp_seq=1 ttl=64 time=0.025 ms
Sep 22 20:05:33 <agent_id> ping[104808]: --- ready.spartan ping statistics ---
Sep 22 20:05:33 <agent_id> ping[104808]: 1 packets transmitted, 1 received, 0% packet loss, time 0ms
Sep 22 20:05:33 <agent_id> ping[104808]: rtt min/avg/max/mdev = 0.025/0.025/0.025/0.000 ms
Sep 22 20:05:33 <agent_id> bash[104810]: /proc/sys/net/ipv4/conf/all/rp_filter: 2
Sep 22 20:05:33 <agent_id> bash[104810]: /proc/sys/net/ipv4/conf/cbr0/rp_filter: 2
Sep 22 20:05:33 <agent_id> bash[104810]: /proc/sys/net/ipv4/conf/default/rp_filter: 2
Sep 22 20:05:33 <agent_id> bash[104810]: /proc/sys/net/ipv4/conf/dummy0/rp_filter: 2
Sep 22 20:05:33 <agent_id> bash[104810]: /proc/sys/net/ipv4/conf/eno1/rp_filter: 2
Sep 22 20:05:33 <agent_id> bash[104810]: /proc/sys/net/ipv4/conf/eno2/rp_filter: 2
Sep 22 20:05:33 <agent_id> bash[104810]: /proc/sys/net/ipv4/conf/eno3/rp_filter: 2
Sep 22 20:05:33 <agent_id> bash[104810]: /proc/sys/net/ipv4/conf/eno4/rp_filter: 2
Sep 22 20:05:33 <agent_id> bash[104810]: /proc/sys/net/ipv4/conf/lo/rp_filter: 2
Sep 22 20:05:33 <agent_id> bash[104810]: /proc/sys/net/ipv4/conf/spartan/rp_filter: 2
Sep 22 20:05:33 <agent_id> bootstrap[104823]: [INFO] Clearing proxy environment variables
Sep 22 20:05:33 <agent_id> bootstrap[104823]: [INFO] PID 3794 has command line [b'/opt/mesosphere/active/java/usr/java/bin/java', b'-Dzookeeper.log.dir=/var/lib/dcos/exhibitor/zookeeper', b'-Dzookeeper.root.logger=INFO,CONSOLE', b'-cp', b'/opt/mesosphere/active/exhibitor/usr/zookeeper/bin/../build/classes:/opt/mesosphere/active/exhibitor/usr/zookeeper/bin/../build/lib/*.jar:/opt/mesosphere/active/exhibitor/usr/zookeeper/bin/../lib/slf4j-log4j12-1.6.1.jar:/opt/mesosphere/active/exhibitor/usr/zookeeper/bin/../lib/slf4j-api-1.6.1.jar:/opt/mesosphere/active/exhibitor/usr/zookeeper/bin/../lib/netty-3.7.0.Final.jar:/opt/mesosphere/active/exhibitor/usr/zookeeper/bin/../lib/log4j-systemd-journal-appender-1.3.2.jar:/opt/mesosphere/active/exhibitor/usr/zookeeper/bin/../lib/log4j-1.2.16.jar:/opt/mesosphere/active/exhibitor/usr/zookeeper/bin/../lib/jna-4.2.2.jar:/opt/mesosphere/active/exhibitor/usr/zookeeper/bin/../lib/jline-0.9.94.jar:/opt/mesosphere/active/exhibitor/usr/zookeeper/bin/../zookeeper-3.4.8.jar:/opt/mesosphere/active/exhibitor/usr/zookeeper/bin/../src/java/lib/*.jar:/var/lib/dcos/exhibitor/conf:', b'-Dzookeeper.DigestAuthenticationProvider.superDigest=super:lK75jTNcA+U9vtVEw5vB51mj/w4=', b'-Dcom.sun.management.jmxremote', b'-Dcom.sun.management.jmxremote.local.only=false', b'org.apache.zookeeper.server.quorum.QuorumPeerMain', b'/var/lib/dcos/exhibitor/conf/zoo.cfg']
Sep 22 20:05:33 <agent_id> bootstrap[104823]: [INFO] PID file hasn't been modified. ZK still seems to be at that PID.
Sep 22 20:05:33 <agent_id> bootstrap[104823]: [INFO] Shortcut succeeeded, assuming local zk is in good config state, not waiting for quorum.
Sep 22 20:05:33 <agent_id> bootstrap[104823]: [INFO] Preparing directory /run/dcos
Sep 22 20:05:33 <agent_id> bootstrap[104823]: [INFO] Preparing directory /run/dcos/etc
Sep 22 20:05:33 <agent_id> bootstrap[104823]: [INFO] Preparing directory /run/dcos/etc/3dt
Sep 22 20:05:33 <agent_id> bootstrap[104823]: [INFO] Preparing directory /run/dcos/etc/marathon
Sep 22 20:05:33 <agent_id> bootstrap[104823]: [INFO] Preparing directory /run/dcos/etc/mesos
Sep 22 20:05:33 <agent_id> bootstrap[104823]: [INFO] Preparing directory /run/dcos/etc/mesos-dns
Sep 22 20:05:33 <agent_id> bootstrap[104823]: [INFO] Preparing directory /run/dcos/etc/dcos-ca
Sep 22 20:05:33 <agent_id> bootstrap[104823]: [INFO] Preparing directory /run/dcos/etc/metronome
Sep 22 20:05:33 <agent_id> bootstrap[104823]: [INFO] Preparing directory /run/dcos/etc/history-service
Sep 22 20:05:33 <agent_id> bootstrap[104823]: [INFO] Preparing directory /run/dcos/etc/signal-service
Sep 22 20:05:33 <agent_id> bootstrap[104823]: [INFO] Preparing directory /run/dcos/pki/tls/private
Sep 22 20:05:33 <agent_id> bootstrap[104823]: [INFO] Preparing directory /run/dcos/pki/tls/certs
Sep 22 20:05:33 <agent_id> bootstrap[104823]: [INFO] Preparing directory /run/dcos/pki/CA/certs
Sep 22 20:05:33 <agent_id> bootstrap[104823]: [INFO] Preparing directory /run/dcos/pki/CA/private
Sep 22 20:05:33 <agent_id> bootstrap[104823]: [INFO] Using super credentials for Zookeeper
Sep 22 20:05:33 <agent_id> bootstrap[104823]: [INFO] Connecting to 127.0.0.1:2181
Sep 22 20:05:33 <agent_id> bootstrap[104823]: [INFO] Zookeeper connection established, state: CONNECTED
Sep 22 20:05:33 <agent_id> bootstrap[104823]: [DEBUG] bootstrapping dcos-mesos-master
Sep 22 20:05:33 <agent_id> bootstrap[104823]: [INFO] Initializing ACLs for znode /
Sep 22 20:05:33 <agent_id> bootstrap[104823]: [INFO] ensure_zk_path(/, [ACL(perms=5, acl_list=['READ', 'CREATE'], id=Id(scheme='world', id='anyone')), ACL(perms=31, acl_list=['ALL'], id=Id(scheme='ip', id='127.0.0.1'))])
Sep 22 20:05:33 <agent_id> bootstrap[104823]: [INFO] Initializing ACLs for znode /cosmos
Sep 22 20:05:33 <agent_id> bootstrap[104823]: [INFO] ensure_zk_path(/cosmos, [ACL(perms=31, acl_list=['ALL'], id=Id(scheme='world', id='anyone'))])
Sep 22 20:05:33 <agent_id> bootstrap[104823]: [INFO] Initializing ACLs for znode /dcos
Sep 22 20:05:33 <agent_id> bootstrap[104823]: [INFO] ensure_zk_path(/dcos, [ACL(perms=1, acl_list=['READ'], id=Id(scheme='world', id='anyone'))])
Sep 22 20:05:33 <agent_id> bootstrap[104823]: [INFO] Initializing ACLs for znode /dcos/vault
Sep 22 20:05:33 <agent_id> bootstrap[104823]: [INFO] ensure_zk_path(/dcos/vault, [ACL(perms=1, acl_list=['READ'], id=Id(scheme='world', id='anyone'))])
Sep 22 20:05:33 <agent_id> bootstrap[104823]: [INFO] Initializing ACLs for znode /zookeeper
Sep 22 20:05:33 <agent_id> bootstrap[104823]: [INFO] ensure_zk_path(/zookeeper, [ACL(perms=1, acl_list=['READ'], id=Id(scheme='world', id='anyone'))])
Sep 22 20:05:33 <agent_id> bootstrap[104823]: [INFO] Initializing ACLs for znode /zookeeper/quotas
Sep 22 20:05:33 <agent_id> bootstrap[104823]: [INFO] ensure_zk_path(/zookeeper/quotas, [ACL(perms=1, acl_list=['READ'], id=Id(scheme='world', id='anyone'))])
Sep 22 20:05:33 <agent_id> bootstrap[104823]: [INFO] Creating master secrets with user dcos-master
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] ensure_zk_path(/dcos/master/secrets, [ACL(perms=1, acl_list=['READ'], id=Id(scheme='digest', id='dcos-master:x'))])
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] Reaching consensus about znode /dcos/master/secrets/private_keys
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] Consensus znode /dcos/master/secrets/private_keys already exists
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] ensure_zk_path(/dcos/master/secrets/zk, [ACL(perms=1, acl_list=['READ'], id=Id(scheme='digest', id='dcos-master:x'))])
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] Reaching consensus about znode /dcos/master/secrets/zk/dcos_bouncer
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] Consensus znode /dcos/master/secrets/zk/dcos_bouncer already exists
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] ensure_zk_path(/dcos/master/secrets/zk, [ACL(perms=1, acl_list=['READ'], id=Id(scheme='digest', id='dcos-master:x'))])
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] Reaching consensus about znode /dcos/master/secrets/zk/dcos_mesos_master
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] Consensus znode /dcos/master/secrets/zk/dcos_mesos_master already exists
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] ensure_zk_path(/dcos/master/secrets/zk, [ACL(perms=1, acl_list=['READ'], id=Id(scheme='digest', id='dcos-master:x'))])
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] Reaching consensus about znode /dcos/master/secrets/zk/dcos_secrets
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] Consensus znode /dcos/master/secrets/zk/dcos_secrets already exists
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] ensure_zk_path(/dcos/master/secrets/zk, [ACL(perms=1, acl_list=['READ'], id=Id(scheme='digest', id='dcos-master:x'))])
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] Reaching consensus about znode /dcos/master/secrets/zk/dcos_cosmos
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] Consensus znode /dcos/master/secrets/zk/dcos_cosmos already exists
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] ensure_zk_path(/dcos/master/secrets/zk, [ACL(perms=1, acl_list=['READ'], id=Id(scheme='digest', id='dcos-master:x'))])
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] Reaching consensus about znode /dcos/master/secrets/zk/dcos_ca
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] Consensus znode /dcos/master/secrets/zk/dcos_ca already exists
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] ensure_zk_path(/dcos/master/secrets/zk, [ACL(perms=1, acl_list=['READ'], id=Id(scheme='digest', id='dcos-master:x'))])
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] Reaching consensus about znode /dcos/master/secrets/zk/dcos_marathon
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] Consensus znode /dcos/master/secrets/zk/dcos_marathon already exists
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] ensure_zk_path(/dcos/master/secrets/zk, [ACL(perms=1, acl_list=['READ'], id=Id(scheme='digest', id='dcos-master:x'))])
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] Reaching consensus about znode /dcos/master/secrets/zk/dcos_vault_default
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] Consensus znode /dcos/master/secrets/zk/dcos_vault_default already exists
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] ensure_zk_path(/dcos/master/secrets/zk, [ACL(perms=1, acl_list=['READ'], id=Id(scheme='digest', id='dcos-master:x'))])
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] Reaching consensus about znode /dcos/master/secrets/zk/dcos_metronome
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] Consensus znode /dcos/master/secrets/zk/dcos_metronome already exists
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] ensure_zk_path(/dcos/master/secrets/CA, [ACL(perms=1, acl_list=['READ'], id=Id(scheme='digest', id='dcos-master:x'))])
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] Reaching consensus about znode /dcos/master/secrets/CA/RootCA
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] Consensus znode /dcos/master/secrets/CA/RootCA already exists
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] ensure_zk_path(/dcos/master/secrets/services, [ACL(perms=1, acl_list=['READ'], id=Id(scheme='digest', id='dcos-master:x'))])
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] Reaching consensus about znode /dcos/master/secrets/services/dcos_spartan_master
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] Consensus znode /dcos/master/secrets/services/dcos_spartan_master already exists
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] ensure_zk_path(/dcos/master/secrets/services, [ACL(perms=1, acl_list=['READ'], id=Id(scheme='digest', id='dcos-master:x'))])
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] Reaching consensus about znode /dcos/master/secrets/services/dcos_navstar_master
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] Consensus znode /dcos/master/secrets/services/dcos_navstar_master already exists
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] ensure_zk_path(/dcos/master/secrets/services, [ACL(perms=1, acl_list=['READ'], id=Id(scheme='digest', id='dcos-master:x'))])
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] Reaching consensus about znode /dcos/master/secrets/services/dcos_history_service
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] Consensus znode /dcos/master/secrets/services/dcos_history_service already exists
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] ensure_zk_path(/dcos/master/secrets/services, [ACL(perms=1, acl_list=['READ'], id=Id(scheme='digest', id='dcos-master:x'))])
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] Reaching consensus about znode /dcos/master/secrets/services/dcos_3dt_master
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] Consensus znode /dcos/master/secrets/services/dcos_3dt_master already exists
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] ensure_zk_path(/dcos/master/secrets/services, [ACL(perms=1, acl_list=['READ'], id=Id(scheme='digest', id='dcos-master:x'))])
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] Reaching consensus about znode /dcos/master/secrets/services/dcos_marathon
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] Consensus znode /dcos/master/secrets/services/dcos_marathon already exists
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] ensure_zk_path(/dcos/master/secrets/services, [ACL(perms=1, acl_list=['READ'], id=Id(scheme='digest', id='dcos-master:x'))])
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] Reaching consensus about znode /dcos/master/secrets/services/dcos_networking_api_master
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] Consensus znode /dcos/master/secrets/services/dcos_networking_api_master already exists
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] ensure_zk_path(/dcos/master/secrets/services, [ACL(perms=1, acl_list=['READ'], id=Id(scheme='digest', id='dcos-master:x'))])
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] Reaching consensus about znode /dcos/master/secrets/services/dcos_minuteman_master
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] Consensus znode /dcos/master/secrets/services/dcos_minuteman_master already exists
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] ensure_zk_path(/dcos/master/secrets/services, [ACL(perms=1, acl_list=['READ'], id=Id(scheme='digest', id='dcos-master:x'))])
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] Reaching consensus about znode /dcos/master/secrets/services/dcos_mesos_dns
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] Consensus znode /dcos/master/secrets/services/dcos_mesos_dns already exists
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] ensure_zk_path(/dcos/master/secrets/services, [ACL(perms=1, acl_list=['READ'], id=Id(scheme='digest', id='dcos-master:x'))])
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] Reaching consensus about znode /dcos/master/secrets/services/dcos_signal_service
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] Consensus znode /dcos/master/secrets/services/dcos_signal_service already exists
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] ensure_zk_path(/dcos/master/secrets/services, [ACL(perms=1, acl_list=['READ'], id=Id(scheme='digest', id='dcos-master:x'))])
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] Reaching consensus about znode /dcos/master/secrets/services/dcos_adminrouter
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] Consensus znode /dcos/master/secrets/services/dcos_adminrouter already exists
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] ensure_zk_path(/dcos/master/secrets/services, [ACL(perms=1, acl_list=['READ'], id=Id(scheme='digest', id='dcos-master:x'))])
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] Reaching consensus about znode /dcos/master/secrets/services/dcos_metronome
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] Consensus znode /dcos/master/secrets/services/dcos_metronome already exists
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] ensure_zk_path(/mesos, [ACL(perms=1, acl_list=['READ'], id=Id(scheme='world', id='anyone')), ACL(perms=31, acl_list=['ALL'], id=Id(scheme='ip', id='127.0.0.1')), ACL(perms=31, acl_list=['ALL'], id=Id(scheme='digest', id='dcos_mesos_master:IeBqBqLqnGlC3FoMAKYACqaZlIQ='))])
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] Writing Mesos Master ZK credentials to /run/dcos/etc/mesos-master
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] Opening /run/dcos/etc
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] Opened /run/dcos/etc with fd 8
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] Locking fd 8
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] Locked fd 8
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] Writing /run/dcos/etc/mesos-master with mode 600
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] Unlocked fd 8
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] Closing /run/dcos/etc with fd 8
Sep 22 20:05:36 <agent_id> bootstrap[104823]: [INFO] ensure_zk_path(/dcos/agent/secrets/services, [ACL(perms=1, acl_list=['READ'], id=Id(scheme='digest', id='dcos-agent:ZT4/4Ug5cTT0jcU4PP/2D9W6zOs='))])
Sep 22 20:05:36 <agent_id> bootstrap[104823]: [INFO] Reaching consensus about znode /dcos/agent/secrets/services/dcos_navstar_agent
Sep 22 20:05:36 <agent_id> bootstrap[104823]: [INFO] Consensus znode /dcos/agent/secrets/services/dcos_navstar_agent already exists
Sep 22 20:05:36 <agent_id> bootstrap[104823]: [INFO] ensure_zk_path(/dcos/agent/secrets/services, [ACL(perms=1, acl_list=['READ'], id=Id(scheme='digest', id='dcos-agent:ZT4/4Ug5cTT0jcU4PP/2D9W6zOs='))])
Sep 22 20:05:36 <agent_id> bootstrap[104823]: [INFO] Reaching consensus about znode /dcos/agent/secrets/services/dcos_spartan_agent
Sep 22 20:05:36 <agent_id> bootstrap[104823]: [INFO] Consensus znode /dcos/agent/secrets/services/dcos_spartan_agent already exists
Sep 22 20:05:36 <agent_id> bootstrap[104823]: [INFO] ensure_zk_path(/dcos/agent/secrets/services, [ACL(perms=1, acl_list=['READ'], id=Id(scheme='digest', id='dcos-agent:ZT4/4Ug5cTT0jcU4PP/2D9W6zOs='))])
Sep 22 20:05:36 <agent_id> bootstrap[104823]: [INFO] Reaching consensus about znode /dcos/agent/secrets/services/dcos_mesos_agent_public
Sep 22 20:05:36 <agent_id> bootstrap[104823]: [INFO] Consensus znode /dcos/agent/secrets/services/dcos_mesos_agent_public already exists
Sep 22 20:05:36 <agent_id> bootstrap[104823]: [INFO] ensure_zk_path(/dcos/agent/secrets/services, [ACL(perms=1, acl_list=['READ'], id=Id(scheme='digest', id='dcos-agent:ZT4/4Ug5cTT0jcU4PP/2D9W6zOs='))])
Sep 22 20:05:36 <agent_id> bootstrap[104823]: [INFO] Reaching consensus about znode /dcos/agent/secrets/services/dcos_minuteman_agent
Sep 22 20:05:36 <agent_id> bootstrap[104823]: [INFO] Consensus znode /dcos/agent/secrets/services/dcos_minuteman_agent already exists
Sep 22 20:05:36 <agent_id> bootstrap[104823]: [INFO] ensure_zk_path(/dcos/agent/secrets/services, [ACL(perms=1, acl_list=['READ'], id=Id(scheme='digest', id='dcos-agent:ZT4/4Ug5cTT0jcU4PP/2D9W6zOs='))])
Sep 22 20:05:36 <agent_id> bootstrap[104823]: [INFO] Reaching consensus about znode /dcos/agent/secrets/services/dcos_mesos_agent
Sep 22 20:05:36 <agent_id> bootstrap[104823]: [INFO] Consensus znode /dcos/agent/secrets/services/dcos_mesos_agent already exists
Sep 22 20:05:36 <agent_id> bootstrap[104823]: [INFO] ensure_zk_path(/dcos/agent/secrets/services, [ACL(perms=1, acl_list=['READ'], id=Id(scheme='digest', id='dcos-agent:ZT4/4Ug5cTT0jcU4PP/2D9W6zOs='))])
Sep 22 20:05:36 <agent_id> bootstrap[104823]: [INFO] Reaching consensus about znode /dcos/agent/secrets/services/dcos_agent
Sep 22 20:05:36 <agent_id> bootstrap[104823]: [INFO] Consensus znode /dcos/agent/secrets/services/dcos_agent already exists
Sep 22 20:05:36 <agent_id> bootstrap[104823]: [INFO] ensure_zk_path(/dcos/agent/secrets/services, [ACL(perms=1, acl_list=['READ'], id=Id(scheme='digest', id='dcos-agent:ZT4/4Ug5cTT0jcU4PP/2D9W6zOs='))])
Sep 22 20:05:36 <agent_id> bootstrap[104823]: [INFO] Reaching consensus about znode /dcos/agent/secrets/services/dcos_adminrouter_agent
Sep 22 20:05:36 <agent_id> bootstrap[104823]: [INFO] Consensus znode /dcos/agent/secrets/services/dcos_adminrouter_agent already exists
Sep 22 20:05:36 <agent_id> bootstrap[104823]: [INFO] ensure_zk_path(/dcos/agent/secrets/services, [ACL(perms=1, acl_list=['READ'], id=Id(scheme='digest', id='dcos-agent:ZT4/4Ug5cTT0jcU4PP/2D9W6zOs='))])
Sep 22 20:05:36 <agent_id> bootstrap[104823]: [INFO] Reaching consensus about znode /dcos/agent/secrets/services/dcos_3dt_agent
Sep 22 20:05:36 <agent_id> bootstrap[104823]: [INFO] Consensus znode /dcos/agent/secrets/services/dcos_3dt_agent already exists
Sep 22 20:05:36 <agent_id> bootstrap[104823]: [INFO] Creating service account dcos_agent
Sep 22 20:05:36 <agent_id> bootstrap[104823]: [INFO] Starting new HTTP connection (1): 127.0.0.1
Sep 22 20:05:37 <agent_id> bootstrap[104823]: [INFO] status_code=409
Sep 22 20:05:37 <agent_id> bootstrap[104823]: [INFO] Service account dcos_agent already exists
Sep 22 20:05:37 <agent_id> bootstrap[104823]: [INFO] Starting new HTTP connection (1): 127.0.0.1
Sep 22 20:05:37 <agent_id> bootstrap[104823]: [INFO] Creating service account dcos_mesos_agent
Sep 22 20:05:37 <agent_id> bootstrap[104823]: [INFO] Starting new HTTP connection (1): 127.0.0.1
Sep 22 20:05:37 <agent_id> bootstrap[104823]: [INFO] status_code=409
Sep 22 20:05:37 <agent_id> bootstrap[104823]: [INFO] Service account dcos_mesos_agent already exists
Sep 22 20:05:37 <agent_id> bootstrap[104823]: [INFO] Starting new HTTP connection (1): 127.0.0.1
Sep 22 20:05:37 <agent_id> bootstrap[104823]: [INFO] Creating service account dcos_mesos_agent_public
Sep 22 20:05:37 <agent_id> bootstrap[104823]: [INFO] Starting new HTTP connection (1): 127.0.0.1
Sep 22 20:05:37 <agent_id> bootstrap[104823]: [INFO] status_code=409
Sep 22 20:05:37 <agent_id> bootstrap[104823]: [INFO] Service account dcos_mesos_agent_public already exists
Sep 22 20:05:37 <agent_id> bootstrap[104823]: [INFO] Starting new HTTP connection (1): 127.0.0.1
Sep 22 20:05:37 <agent_id> bootstrap[104823]: [INFO] Creating service account dcos_adminrouter_agent
Sep 22 20:05:37 <agent_id> bootstrap[104823]: [INFO] Starting new HTTP connection (1): 127.0.0.1
Sep 22 20:05:37 <agent_id> bootstrap[104823]: [INFO] status_code=409
Sep 22 20:05:37 <agent_id> bootstrap[104823]: [INFO] Service account dcos_adminrouter_agent already exists
Sep 22 20:05:37 <agent_id> bootstrap[104823]: [INFO] Starting new HTTP connection (1): 127.0.0.1
Sep 22 20:05:37 <agent_id> bootstrap[104823]: [INFO] Creating service account dcos_3dt_agent
Sep 22 20:05:37 <agent_id> bootstrap[104823]: [INFO] Starting new HTTP connection (1): 127.0.0.1
Sep 22 20:05:37 <agent_id> bootstrap[104823]: [INFO] status_code=409
Sep 22 20:05:37 <agent_id> bootstrap[104823]: [INFO] Service account dcos_3dt_agent already exists
Sep 22 20:05:37 <agent_id> bootstrap[104823]: [INFO] Starting new HTTP connection (1): 127.0.0.1
Sep 22 20:05:37 <agent_id> bootstrap[104823]: [INFO] Creating service account dcos_minuteman_agent
Sep 22 20:05:37 <agent_id> bootstrap[104823]: [INFO] Starting new HTTP connection (1): 127.0.0.1
Sep 22 20:05:37 <agent_id> bootstrap[104823]: [INFO] status_code=409
Sep 22 20:05:37 <agent_id> bootstrap[104823]: [INFO] Service account dcos_minuteman_agent already exists
Sep 22 20:05:37 <agent_id> bootstrap[104823]: [INFO] Starting new HTTP connection (1): 127.0.0.1
Sep 22 20:05:37 <agent_id> bootstrap[104823]: [INFO] Creating service account dcos_navstar_agent
Sep 22 20:05:37 <agent_id> bootstrap[104823]: [INFO] Starting new HTTP connection (1): 127.0.0.1
Sep 22 20:05:37 <agent_id> bootstrap[104823]: [INFO] status_code=409
Sep 22 20:05:37 <agent_id> bootstrap[104823]: [INFO] Service account dcos_navstar_agent already exists
Sep 22 20:05:37 <agent_id> bootstrap[104823]: [INFO] Starting new HTTP connection (1): 127.0.0.1
Sep 22 20:05:37 <agent_id> bootstrap[104823]: [INFO] Creating service account dcos_spartan_agent
Sep 22 20:05:37 <agent_id> bootstrap[104823]: [INFO] Starting new HTTP connection (1): 127.0.0.1
Sep 22 20:05:37 <agent_id> bootstrap[104823]: [INFO] status_code=409
Sep 22 20:05:37 <agent_id> bootstrap[104823]: [INFO] Service account dcos_spartan_agent already exists
Sep 22 20:05:37 <agent_id> bootstrap[104823]: [INFO] Starting new HTTP connection (1): 127.0.0.1
Sep 22 20:05:37 <agent_id> systemd[1]: Started Mesos Master: DC/OS Mesos Master Service.
Sep 22 20:05:37 <agent_id> mesos-master[104837]: WARNING: Logging before InitGoogleLogging() is written to STDERR
Sep 22 20:05:37 <agent_id> mesos-master[104837]: I0922 20:05:37.972128 104837 main.cpp:263] Build: 2016-09-13 x.x2:04 by
Sep 22 20:05:37 <agent_id> mesos-master[104837]: I0922 20:05:37.972210 104837 main.cpp:264] Version: 1.0.1
Sep 22 20:05:37 <agent_id> mesos-master[104837]: I0922 20:05:37.972216 104837 main.cpp:271] Git SHA: a22b5dd552168a8dd925682f3b349059e10fc3d3
Sep 22 20:05:37 <agent_id> mesos-master[104837]: W0922 20:05:37.973675 104837 openssl.cpp:399] Failed SSL connections will be downgraded to a non-SSL socket
Sep 22 20:05:37 <agent_id> mesos-master[104837]: W0922 20:05:37.973877 104837 process.cpp:880] Failed SSL connections will be downgraded to a non-SSL socket
Sep 22 20:05:37 <agent_id> mesos-master[104837]: I0922 20:05:37.975872 104837 logging.cpp:194] INFO level logging started!
Sep 22 20:05:37 <agent_id> mesos-master[104837]: W0922 20:05:37.976052 104837 main.cpp:290] Loaded deprecated flag 'max_slave_ping_timeouts'
Sep 22 20:05:37 <agent_id> mesos-master[104837]: W0922 20:05:37.976161 104837 main.cpp:290] Loaded deprecated flag 'slave_removal_rate_limit'
Sep 22 20:05:37 <agent_id> mesos-master[104837]: I0922 20:05:37.992913 104837 manager.cpp:1422] Overlay master parameter 'master_config=/opt/mesosphere/etc/overlay/config/master.json'
Sep 22 20:05:37 <agent_id> mesos-master[104837]: I0922 20:05:37.993281 104837 manager.cpp:606] Configuring overlay network:dcos
Sep 22 20:05:37 <agent_id> mesos-master[104837]: I0922 20:05:37.993336 104837 manager.cpp:650] Initializing the replicated log.
Sep 22 20:05:37 <agent_id> mesos-master[104837]: I0922 20:05:37.993355 104837 manager.cpp:686] Using replicated log with zookeeper URL zk://dcos_mesos_master:6EIjIPV6aX3U8M8kTpSkqXw4e5bZBGZizVV02rdBQdjOv9TzvH4BimMFmmVLe8Kv@127.0.0.1:2181/mesos with a quorum of 2 for leader election.
Sep 22 20:05:38 <agent_id> mesos-master[104837]: I0922 20:05:38.094033 104837 leveldb.cpp:174] Opened db in 100.622609ms
Sep 22 20:05:38 <agent_id> mesos-master[104837]: I0922 20:05:38.x.x81 104837 leveldb.cpp:181] Compacted db in 14.315233ms
Sep 22 20:05:38 <agent_id> mesos-master[104837]: I0922 20:05:38.x.x58 104837 leveldb.cpp:196] Created db iterator in 6554ns
Sep 22 20:05:38 <agent_id> mesos-master[104837]: I0922 20:05:38.x.x67 104837 leveldb.cpp:202] Seeked to beginning of db in 672ns
Sep 22 20:05:38 <agent_id> mesos-master[104837]: I0922 20:05:38.x.x72 104837 leveldb.cpp:271] Iterated through 0 keys in the db in 288ns
Sep 22 20:05:38 <agent_id> mesos-master[104837]: I0922 20:05:38.x.x97 104837 replica.cpp:779] Replica recovered with log positions 0 -> 0 with 1 holes and 0 unlearned
Sep 22 20:05:38 <agent_id> mesos-master[104837]: 2016-09-22 20:05:38,108:104837(0x7f7a49d4d700):ZOO_INFO@log_env@726: Client environment:zookeeper.version=zookeeper C client 3.4.8
Sep 22 20:05:38 <agent_id> mesos-master[104837]: 2016-09-22 20:05:38,108:104837(0x7f7a49d4d700):ZOO_INFO@log_env@730: Client environment:host.name=<agent_id>
Sep 22 20:05:38 <agent_id> mesos-master[104837]: 2016-09-22 20:05:38,108:104837(0x7f7a49d4d700):ZOO_INFO@log_env@737: Client environment:os.name=Linux
Sep 22 20:05:38 <agent_id> mesos-master[104837]: 2016-09-22 20:05:38,108:104837(0x7f7a49d4d700):ZOO_INFO@log_env@738: Client environment:os.arch=4.6.3-coreos
Sep 22 20:05:38 <agent_id> mesos-master[104837]: 2016-09-22 20:05:38,108:104837(0x7f7a49d4d700):ZOO_INFO@log_env@739: Client environment:os.version=#2 SMP Tue Aug 23 02:14:16 UTC 2016
Sep 22 20:05:38 <agent_id> mesos-master[104837]: 2016-09-22 20:05:38,108:104837(0x7f7a4ed57700):ZOO_INFO@log_env@726: Client environment:zookeeper.version=zookeeper C client 3.4.8
Sep 22 20:05:38 <agent_id> mesos-master[104837]: 2016-09-22 20:05:38,108:104837(0x7f7a4ed57700):ZOO_INFO@log_env@730: Client environment:host.name=<agent_id>
Sep 22 20:05:38 <agent_id> mesos-master[104837]: 2016-09-22 20:05:38,108:104837(0x7f7a4ed57700):ZOO_INFO@log_env@737: Client environment:os.name=Linux
Sep 22 20:05:38 <agent_id> mesos-master[104837]: 2016-09-22 20:05:38,108:104837(0x7f7a4ed57700):ZOO_INFO@log_env@738: Client environment:os.arch=4.6.3-coreos
Sep 22 20:05:38 <agent_id> mesos-master[104837]: 2016-09-22 20:05:38,108:104837(0x7f7a4ed57700):ZOO_INFO@log_env@739: Client environment:os.version=#2 SMP Tue Aug 23 02:14:16 UTC 2016
Sep 22 20:05:38 <agent_id> mesos-master[104837]: 2016-09-22 20:05:38,109:104837(0x7f7a4ed57700):ZOO_INFO@log_env@747: Client environment:user.name=(null)
Sep 22 20:05:38 <agent_id> mesos-master[104837]: 2016-09-22 20:05:38,109:104837(0x7f7a49d4d700):ZOO_INFO@log_env@747: Client environment:user.name=(null)
Sep 22 20:05:38 <agent_id> mesos-master[104837]: 2016-09-22 20:05:38,109:104837(0x7f7a4ed57700):ZOO_INFO@log_env@755: Client environment:user.home=/root
Sep 22 20:05:38 <agent_id> mesos-master[104837]: 2016-09-22 20:05:38,109:104837(0x7f7a4ed57700):ZOO_INFO@log_env@767: Client environment:user.dir=/
Sep 22 20:05:38 <agent_id> mesos-master[104837]: 2016-09-22 20:05:38,109:104837(0x7f7a4ed57700):ZOO_INFO@zookeeper_init@800: Initiating client connection, host=127.0.0.1:2181 sessionTimeout=10000 watcher=0x7f7a5e765d00 sessionId=0 sessionPasswd=<null> context=0x7f7a38000ae0 flags=0
Sep 22 20:05:38 <agent_id> mesos-master[104837]: I0922 20:05:38.x.x31 104887 log.cpp:107] Attempting to join replica to ZooKeeper group
Sep 22 20:05:38 <agent_id> mesos-master[104837]: 2016-09-22 20:05:38,109:104837(0x7f7a49d4d700):ZOO_INFO@log_env@755: Client environment:user.home=/root
Sep 22 20:05:38 <agent_id> mesos-master[104837]: I0922 20:05:38.x.x78 104837 manager.cpp:871] Overlay agent parameter 'agent_config=/opt/mesosphere/etc/overlay/config/agent-master.json'
Sep 22 20:05:38 <agent_id> mesos-master[104837]: 2016-09-22 20:05:38,109:104837(0x7f7a49d4d700):ZOO_INFO@log_env@767: Client environment:user.dir=/
Sep 22 20:05:38 <agent_id> mesos-master[104837]: 2016-09-22 20:05:38,109:104837(0x7f7a49d4d700):ZOO_INFO@zookeeper_init@800: Initiating client connection, host=127.0.0.1:2181 sessionTimeout=10000 watcher=0x7f7a5e765d00 sessionId=0 sessionPasswd=<null> context=0x7f79f8000930 flags=0
Sep 22 20:05:38 <agent_id> mesos-master[104837]: I0922 20:05:38.x.x73 104876 manager.cpp:759] Adding route for 'overlay-master/state'
Sep 22 20:05:38 <agent_id> mesos-master[104837]: I0922 20:05:38.110025 104870 recover.cpp:451] Starting replica recovery
Sep 22 20:05:38 <agent_id> mesos-master[104837]: 2016-09-22 20:05:38,110:104837(0x7f7a47d49700):ZOO_INFO@log_env@726: Client environment:zookeeper.version=zookeeper C client 3.4.8
Sep 22 20:05:38 <agent_id> mesos-master[104837]: 2016-09-22 20:05:38,110:104837(0x7f7a47d49700):ZOO_INFO@log_env@730: Client environment:host.name=<agent_id>
Sep 22 20:05:38 <agent_id> mesos-master[104837]: 2016-09-22 20:05:38,110:104837(0x7f7a47d49700):ZOO_INFO@log_env@737: Client environment:os.name=Linux
Sep 22 20:05:38 <agent_id> mesos-master[104837]: 2016-09-22 20:05:38,110:104837(0x7f7a47d49700):ZOO_INFO@log_env@738: Client environment:os.arch=4.6.3-coreos
Sep 22 20:05:38 <agent_id> mesos-master[104837]: 2016-09-22 20:05:38,110:104837(0x7f7a47d49700):ZOO_INFO@log_env@739: Client environment:os.version=#2 SMP Tue Aug 23 02:14:16 UTC 2016
Sep 22 20:05:38 <agent_id> mesos-master[104837]: I0922 20:05:38.110203 104870 recover.cpp:477] Replica is in EMPTY status
Sep 22 20:05:38 <agent_id> mesos-master[104837]: 2016-09-22 20:05:38,110:104837(0x7f7a265f2700):ZOO_INFO@check_events@1728: initiated connection to server [127.0.0.1:2181]
Sep 22 20:05:38 <agent_id> mesos-master[104837]: 2016-09-22 20:05:38,110:104837(0x7f7a275f4700):ZOO_INFO@check_events@1728: initiated connection to server [127.0.0.1:2181]
Sep 22 20:05:38 <agent_id> mesos-master[104837]: 2016-09-22 20:05:38,110:104837(0x7f7a47d49700):ZOO_INFO@log_env@747: Client environment:user.name=(null)
Sep 22 20:05:38 <agent_id> mesos-master[104837]: 2016-09-22 20:05:38,111:104837(0x7f7a47d49700):ZOO_INFO@log_env@755: Client environment:user.home=/root
Sep 22 20:05:38 <agent_id> mesos-master[104837]: 2016-09-22 20:05:38,111:104837(0x7f7a47d49700):ZOO_INFO@log_env@767: Client environment:user.dir=/
Sep 22 20:05:38 <agent_id> mesos-master[104837]: 2016-09-22 20:05:38,111:104837(0x7f7a47d49700):ZOO_INFO@zookeeper_init@800: Initiating client connection, host=127.0.0.1:2181 sessionTimeout=10000 watcher=0x7f7a5e765d00 sessionId=0 sessionPasswd=<null> context=0x7f79e8001f30 flags=0
Sep 22 20:05:38 <agent_id> mesos-master[104837]: 2016-09-22 20:05:38,111:104837(0x7f7a255f0700):ZOO_INFO@check_events@1728: initiated connection to server [127.0.0.1:2181]
Sep 22 20:05:38 <agent_id> mesos-master[104837]: 2016-09-22 20:05:38,123:104837(0x7f7a265f2700):ZOO_INFO@check_events@1775: session establishment complete on server [127.0.0.1:2181], sessionId=0x3574e2d05adb704, negotiated timeout=10000
Sep 22 20:05:38 <agent_id> mesos-master[104837]: I0922 20:05:38.124027 104867 group.cpp:349] Group process (group(1)@x.x.9.1:5050) connected to ZooKeeper
Sep 22 20:05:38 <agent_id> mesos-master[104837]: I0922 20:05:38.124076 104867 group.cpp:837] Syncing group operations: queue size (joins, cancels, datas) = (0, 0, 0)
Sep 22 20:05:38 <agent_id> mesos-master[104837]: I0922 20:05:38.124088 104867 group.cpp:403] Authenticating with ZooKeeper using digest
Sep 22 20:05:38 <agent_id> mesos-master[104837]: 2016-09-22 20:05:38,129:104837(0x7f7a275f4700):ZOO_INFO@check_events@1775: session establishment complete on server [127.0.0.1:2181], sessionId=0x3574e2d05adb705, negotiated timeout=10000
Sep 22 20:05:38 <agent_id> mesos-master[104837]: I0922 20:05:38.129184 104871 group.cpp:349] Group process (group(2)@x.x.9.1:5050) connected to ZooKeeper
Sep 22 20:05:38 <agent_id> mesos-master[104837]: I0922 20:05:38.129216 104871 group.cpp:837] Syncing group operations: queue size (joins, cancels, datas) = (1, 0, 0)
Sep 22 20:05:38 <agent_id> mesos-master[104837]: I0922 20:05:38.129225 104871 group.cpp:403] Authenticating with ZooKeeper using digest
Sep 22 20:05:38 <agent_id> mesos-master[104837]: 2016-09-22 20:05:38,129:104837(0x7f7a255f0700):ZOO_INFO@check_events@1775: session establishment complete on server [127.0.0.1:2181], sessionId=0x3574e2d05adb706, negotiated timeout=10000
Sep 22 20:05:38 <agent_id> mesos-master[104837]: I0922 20:05:38.129473 104865 group.cpp:349] Group process (group(3)@x.x.9.1:5050) connected to ZooKeeper
Sep 22 20:05:38 <agent_id> mesos-master[104837]: I0922 20:05:38.129508 104865 group.cpp:837] Syncing group operations: queue size (joins, cancels, datas) = (0, 0, 0)
Sep 22 20:05:38 <agent_id> mesos-master[104837]: I0922 20:05:38.129521 104865 group.cpp:403] Authenticating with ZooKeeper using digest
Sep 22 20:05:41 <agent_id> mesos-master[104837]: 2016-09-22 20:05:41,446:104837(0x7f7a265f2700):ZOO_INFO@auth_completion_func@1300: Authentication scheme digest succeeded
Sep 22 20:05:41 <agent_id> mesos-master[104837]: 2016-09-22 20:05:41,446:104837(0x7f7a275f4700):ZOO_INFO@auth_completion_func@1300: Authentication scheme digest succeeded
Sep 22 20:05:41 <agent_id> mesos-master[104837]: I0922 20:05:41.446926 104867 group.cpp:427] Trying to create path '/mesos/overlay_log_replicas' in ZooKeeper
Sep 22 20:05:41 <agent_id> mesos-master[104837]: I0922 20:05:41.447043 104871 group.cpp:427] Trying to create path '/mesos/overlay_log_replicas' in ZooKeeper
Sep 22 20:05:41 <agent_id> mesos-master[104837]: 2016-09-22 20:05:41,449:104837(0x7f7a255f0700):ZOO_INFO@auth_completion_func@1300: Authentication scheme digest succeeded
Sep 22 20:05:41 <agent_id> mesos-master[104837]: I0922 20:05:41.449318 104865 group.cpp:427] Trying to create path '/mesos' in ZooKeeper
Sep 22 20:05:41 <agent_id> mesos-master[104837]: I0922 20:05:41.454278 104860 network.hpp:430] ZooKeeper group memberships changed
Sep 22 20:05:41 <agent_id> mesos-master[104837]: I0922 20:05:41.454349 104854 group.cpp:706] Trying to get '/mesos/overlay_log_replicas/0000003293' in ZooKeeper
Sep 22 20:05:41 <agent_id> mesos-master[104837]: I0922 20:05:41.454555 104854 group.cpp:706] Trying to get '/mesos/overlay_log_replicas/0000003294' in ZooKeeper
Sep 22 20:05:41 <agent_id> mesos-master[104837]: I0922 20:05:41.454640 104875 detector.cpp:152] Detected a new leader: (id='2763')
Sep 22 20:05:41 <agent_id> mesos-master[104837]: I0922 20:05:41.463635 104858 group.cpp:706] Trying to get '/mesos/json.info_0000002763' in ZooKeeper
Sep 22 20:05:41 <agent_id> mesos-master[104837]: I0922 20:05:41.464010 104854 group.cpp:706] Trying to get '/mesos/overlay_log_replicas/0000003295' in ZooKeeper
Sep 22 20:05:41 <agent_id> mesos-master[104837]: I0922 20:05:41.464095 104871 zookeeper.cpp:259] A new leading master (UPID=master@x.x.11.1:5050) is detected
Sep 22 20:05:41 <agent_id> mesos-master[104837]: I0922 20:05:41.464279 104868 network.hpp:478] ZooKeeper group PIDs: { log-replica(1)@x.x.11.1:5050, log-replica(1)@x.x.9.1:5050, log-replica(1)@x.x.11.1:5050 }
Sep 22 20:05:46 <agent_id> mesos-master[104837]: I0922 20:05:46.001029 104851 manager.cpp:180] Initializing overlay agent manager
Sep 22 20:05:46 <agent_id> mesos-master[104837]: I0922 20:05:46.001037 104837 manager.cpp:850] Overlay agent is ready
Sep 22 20:05:46 <agent_id> mesos-master[104837]: I0922 20:05:46.001206 104837 main.cpp:370] Using 'HierarchicalDRF' allocator
Sep 22 20:05:46 <agent_id> mesos-master[104837]: I0922 20:05:46.001278 104847 manager.cpp:397] Detected new overlay master at overlay-master@x.x.11.1:5050
Sep 22 20:05:46 <agent_id> mesos-master[104837]: I0922 20:05:46.042414 104837 leveldb.cpp:174] Opened db in 41.142916ms
Sep 22 20:05:46 <agent_id> mesos-master[104837]: I0922 20:05:46.054388 104837 leveldb.cpp:181] Compacted db in 11.932746ms
Sep 22 20:05:46 <agent_id> mesos-master[104837]: I0922 20:05:46.054451 104837 leveldb.cpp:196] Created db iterator in 3474ns
Sep 22 20:05:46 <agent_id> mesos-master[104837]: I0922 20:05:46.054461 104837 leveldb.cpp:202] Seeked to beginning of db in 687ns
Sep 22 20:05:46 <agent_id> mesos-master[104837]: I0922 20:05:46.054466 104837 leveldb.cpp:271] Iterated through 0 keys in the db in 251ns
Sep 22 20:05:46 <agent_id> mesos-master[104837]: I0922 20:05:46.054479 104837 replica.cpp:779] Replica recovered with log positions 0 -> 0 with 1 holes and 0 unlearned
Sep 22 20:05:46 <agent_id> mesos-master[104837]: 2016-09-22 20:05:46,054:104837(0x7f7a53d61700):ZOO_INFO@log_env@726: Client environment:zookeeper.version=zookeeper C client 3.4.8
Sep 22 20:05:46 <agent_id> mesos-master[104837]: 2016-09-22 20:05:46,054:104837(0x7f7a53d61700):ZOO_INFO@log_env@730: Client environment:host.name=<agent_id>
Sep 22 20:05:46 <agent_id> mesos-master[104837]: 2016-09-22 20:05:46,054:104837(0x7f7a53d61700):ZOO_INFO@log_env@737: Client environment:os.name=Linux
Sep 22 20:05:46 <agent_id> mesos-master[104837]: 2016-09-22 20:05:46,054:104837(0x7f7a53d61700):ZOO_INFO@log_env@738: Client environment:os.arch=4.6.3-coreos
Sep 22 20:05:46 <agent_id> mesos-master[104837]: I0922 20:05:46.054653 104867 log.cpp:107] Attempting to join replica to ZooKeeper group
Sep 22 20:05:46 <agent_id> mesos-master[104837]: 2016-09-22 20:05:46,054:104837(0x7f7a4253e700):ZOO_INFO@log_env@726: Client environment:zookeeper.version=zookeeper C client 3.4.8
Sep 22 20:05:46 <agent_id> mesos-master[104837]: 2016-09-22 20:05:46,054:104837(0x7f7a53d61700):ZOO_INFO@log_env@739: Client environment:os.version=#2 SMP Tue Aug 23 02:14:16 UTC 2016
Sep 22 20:05:46 <agent_id> mesos-master[104837]: 2016-09-22 20:05:46,054:104837(0x7f7a4253e700):ZOO_INFO@log_env@730: Client environment:host.name=<agent_id>
Sep 22 20:05:46 <agent_id> mesos-master[104837]: 2016-09-22 20:05:46,054:104837(0x7f7a4253e700):ZOO_INFO@log_env@737: Client environment:os.name=Linux
Sep 22 20:05:46 <agent_id> mesos-master[104837]: 2016-09-22 20:05:46,054:104837(0x7f7a4253e700):ZOO_INFO@log_env@738: Client environment:os.arch=4.6.3-coreos
Sep 22 20:05:46 <agent_id> mesos-master[104837]: 2016-09-22 20:05:46,054:104837(0x7f7a4253e700):ZOO_INFO@log_env@739: Client environment:os.version=#2 SMP Tue Aug 23 02:14:16 UTC 2016
Sep 22 20:05:46 <agent_id> mesos-master[104837]: I0922 20:05:46.054760 104868 recover.cpp:451] Starting replica recovery
Sep 22 20:05:46 <agent_id> mesos-master[104837]: I0922 20:05:46.054831 104868 recover.cpp:477] Replica is in EMPTY status
Sep 22 20:05:46 <agent_id> mesos-master[104837]: 2016-09-22 20:05:46,054:104837(0x7f7a53d61700):ZOO_INFO@log_env@747: Client environment:user.name=(null)
Sep 22 20:05:46 <agent_id> mesos-master[104837]: 2016-09-22 20:05:46,054:104837(0x7f7a53d61700):ZOO_INFO@log_env@755: Client environment:user.home=/root
Sep 22 20:05:46 <agent_id> mesos-master[104837]: 2016-09-22 20:05:46,054:104837(0x7f7a53d61700):ZOO_INFO@log_env@767: Client environment:user.dir=/
Sep 22 20:05:46 <agent_id> mesos-master[104837]: 2016-09-22 20:05:46,054:104837(0x7f7a53d61700):ZOO_INFO@zookeeper_init@800: Initiating client connection, host=127.0.0.1:2181 sessionTimeout=10000 watcher=0x7f7a5e765d00 sessionId=0 sessionPasswd=<null> context=0x7f7964000950 flags=0
Sep 22 20:05:46 <agent_id> mesos-master[104837]: 2016-09-22 20:05:46,054:104837(0x7f7a4c552700):ZOO_INFO@log_env@726: Client environment:zookeeper.version=zookeeper C client 3.4.8
Sep 22 20:05:46 <agent_id> mesos-master[104837]: 2016-09-22 20:05:46,054:104837(0x7f7a4c552700):ZOO_INFO@log_env@730: Client environment:host.name=<agent_id>
Sep 22 20:05:46 <agent_id> mesos-master[104837]: 2016-09-22 20:05:46,054:104837(0x7f7a4c552700):ZOO_INFO@log_env@737: Client environment:os.name=Linux
Sep 22 20:05:46 <agent_id> mesos-master[104837]: 2016-09-22 20:05:46,054:104837(0x7f7a4c552700):ZOO_INFO@log_env@738: Client environment:os.arch=4.6.3-coreos
Sep 22 20:05:46 <agent_id> mesos-master[104837]: 2016-09-22 20:05:46,054:104837(0x7f7a4c552700):ZOO_INFO@log_env@739: Client environment:os.version=#2 SMP Tue Aug 23 02:14:16 UTC 2016
Sep 22 20:05:46 <agent_id> mesos-master[104837]: 2016-09-22 20:05:46,055:104837(0x7f7a59d6d700):ZOO_INFO@log_env@726: Client environment:zookeeper.version=zookeeper C client 3.4.8
Sep 22 20:05:46 <agent_id> mesos-master[104837]: 2016-09-22 20:05:46,055:104837(0x7f7a59d6d700):ZOO_INFO@log_env@730: Client environment:host.name=<agent_id>
Sep 22 20:05:46 <agent_id> mesos-master[104837]: 2016-09-22 20:05:46,055:104837(0x7f7a59d6d700):ZOO_INFO@log_env@737: Client environment:os.name=Linux
Sep 22 20:05:46 <agent_id> mesos-master[104837]: 2016-09-22 20:05:46,055:104837(0x7f7a59d6d700):ZOO_INFO@log_env@738: Client environment:os.arch=4.6.3-coreos
Sep 22 20:05:46 <agent_id> mesos-master[104837]: 2016-09-22 20:05:46,055:104837(0x7f7a4c552700):ZOO_INFO@log_env@747: Client environment:user.name=(null)
Sep 22 20:05:46 <agent_id> mesos-master[104837]: 2016-09-22 20:05:46,055:104837(0x7f7a59d6d700):ZOO_INFO@log_env@739: Client environment:os.version=#2 SMP Tue Aug 23 02:14:16 UTC 2016
Sep 22 20:05:46 <agent_id> mesos-master[104837]: I0922 20:05:46.055543 104837 main.cpp:543] Starting Mesos master
Sep 22 20:05:46 <agent_id> mesos-master[104837]: 2016-09-22 20:05:46,055:104837(0x7f7a4253e700):ZOO_INFO@log_env@747: Client environment:user.name=(null)
Sep 22 20:05:46 <agent_id> mesos-master[104837]: 2016-09-22 20:05:46,055:104837(0x7f7a4c552700):ZOO_INFO@log_env@755: Client environment:user.home=/root
Sep 22 20:05:46 <agent_id> mesos-master[104837]: 2016-09-22 20:05:46,055:104837(0x7f7a4c552700):ZOO_INFO@log_env@767: Client environment:user.dir=/
Sep 22 20:05:46 <agent_id> mesos-master[104837]: 2016-09-22 20:05:46,055:104837(0x7f7a4c552700):ZOO_INFO@zookeeper_init@800: Initiating client connection, host=127.0.0.1:2181 sessionTimeout=10000 watcher=0x7f7a5e765d00 sessionId=0 sessionPasswd=<null> context=0x7f7a28000f50 flags=0
Sep 22 20:05:46 <agent_id> mesos-master[104837]: 2016-09-22 20:05:46,055:104837(0x7f7a4253e700):ZOO_INFO@log_env@755: Client environment:user.home=/root
Sep 22 20:05:46 <agent_id> mesos-master[104837]: 2016-09-22 20:05:46,055:104837(0x7f7a0ffff700):ZOO_INFO@check_events@1728: initiated connection to server [127.0.0.1:2181]
Sep 22 20:05:46 <agent_id> mesos-master[104837]: 2016-09-22 20:05:46,055:104837(0x7f7a4253e700):ZOO_INFO@log_env@767: Client environment:user.dir=/
Sep 22 20:05:46 <agent_id> mesos-master[104837]: 2016-09-22 20:05:46,055:104837(0x7f7a4253e700):ZOO_INFO@zookeeper_init@800: Initiating client connection, host=127.0.0.1:2181 sessionTimeout=10000 watcher=0x7f7a5e765d00 sessionId=0 sessionPasswd=<null> context=0x7f7980001030 flags=0
Sep 22 20:05:46 <agent_id> mesos-master[104837]: 2016-09-22 20:05:46,055:104837(0x7f7a59d6d700):ZOO_INFO@log_env@747: Client environment:user.name=(null)
Sep 22 20:05:46 <agent_id> mesos-master[104837]: 2016-09-22 20:05:46,055:104837(0x7f7a59d6d700):ZOO_INFO@log_env@755: Client environment:user.home=/root
Sep 22 20:05:46 <agent_id> mesos-master[104837]: 2016-09-22 20:05:46,055:104837(0x7f7a59d6d700):ZOO_INFO@log_env@767: Client environment:user.dir=/
Sep 22 20:05:46 <agent_id> mesos-master[104837]: 2016-09-22 20:05:46,055:104837(0x7f7a59d6d700):ZOO_INFO@zookeeper_init@800: Initiating client connection, host=127.0.0.1:2181 sessionTimeout=10000 watcher=0x7f7a5e765d00 sessionId=0 sessionPasswd=<null> context=0x7f79e0002170 flags=0
Sep 22 20:05:46 <agent_id> mesos-master[104837]: 2016-09-22 20:05:46,056:104837(0x7f7a0effd700):ZOO_INFO@check_events@1728: initiated connection to server [127.0.0.1:2181]
Sep 22 20:05:46 <agent_id> mesos-master[104837]: 2016-09-22 20:05:46,056:104837(0x7f7a0dffb700):ZOO_INFO@check_events@1728: initiated connection to server [127.0.0.1:2181]
Sep 22 20:05:46 <agent_id> mesos-master[104837]: 2016-09-22 20:05:46,056:104837(0x7f7a0cff9700):ZOO_INFO@check_events@1728: initiated connection to server [127.0.0.1:2181]
Sep 22 20:05:46 <agent_id> mesos-master[104837]: I0922 20:05:46.057487 104869 master.cpp:375] Master 4be01ec7-9b37-43be-ba3e-d8f70db0a3fe (x.x.9.1) started on x.x.9.1:5050
Sep 22 20:05:46 <agent_id> mesos-master[104837]: I0922 20:05:46.057510 104869 master.cpp:377] Flags at startup: --agent_ping_timeout="15secs" --slave_removal_rate_limit="1/20mins" --agent_reregister_timeout="10mins" --allocation_interval="1secs" --allocator="HierarchicalDRF" --authenticate_agents="false" --authenticate_frameworks="false" --authenticate_http_frameworks="false" --authenticate_http_readonly="true" --authenticate_http_readwrite="true" --authenticators="com_mesosphere_dcos_ClassicRPCAuthenticator" --authorizers="local" --cluster="walthamDCOS1.8" --framework_sorter="drf" --help="false" --hostname_lookup="false" --http_authenticators="com_mesosphere_dcos_http_Authenticator" --initialize_driver_logging="true" --ip_discovery_command="/opt/mesosphere/bin/detect_ip" --log_auto_initialize="true" --log_dir="/var/lib/dcos/mesos/log" --logbufsecs="0" --logging_level="INFO" --max_slave_ping_timeouts="20" --max_completed_frameworks="50" --max_completed_tasks_per_framework="1000" --modules_dir="/opt/mesosphere/etc/mesos-master-modules" --offer_timeout="2mins" --port="5050" --quiet="false" --quorum="2" --recovery_agent_removal_limit="100%" --registry="replicated_log" --registry_fetch_timeout="1mins" --registry_store_timeout="1mins" --registry_strict="false" --root_submissions="true" --user_sorter="drf" --version="false" --webui_dir="/opt/mesosphere/packages/mesos--2b6cac8d498de56ee4c2eb9a89900b5ca06e270b/share/mesos/webui" --weights="" --work_dir="/var/lib/dcos/mesos/master" --zk="zk://dcos_mesos_master:6EIjIPV6aX3U8M8kTpSkqXw4e5bZBGZizVV02rdBQdjOv9TzvH4BimMFmmVLe8Kv@127.0.0.1:2181/mesos" --zk_session_timeout="10secs"
Sep 22 20:05:46 <agent_id> mesos-master[104837]: I0922 20:05:46.057643 104869 master.cpp:429] Master allowing unauthenticated frameworks to register
Sep 22 20:05:46 <agent_id> mesos-master[104837]: I0922 20:05:46.057651 104869 master.cpp:443] Master allowing unauthenticated agents to register
Sep 22 20:05:46 <agent_id> mesos-master[104837]: I0922 20:05:46.057657 104869 master.cpp:457] Master allowing HTTP frameworks to register without authentication
Sep 22 20:05:46 <agent_id> mesos-master[104837]: 2016-09-22 20:05:46,060:104837(0x7f7a0ffff700):ZOO_INFO@check_events@1775: session establishment complete on server [127.0.0.1:2181], sessionId=0x3574e2d05adb710, negotiated timeout=10000
Sep 22 20:05:46 <agent_id> mesos-master[104837]: I0922 20:05:46.060330 104852 group.cpp:349] Group process (group(4)@x.x.9.1:5050) connected to ZooKeeper
Sep 22 20:05:46 <agent_id> mesos-master[104837]: I0922 20:05:46.060361 104852 group.cpp:837] Syncing group operations: queue size (joins, cancels, datas) = (0, 0, 0)
Sep 22 20:05:46 <agent_id> mesos-master[104837]: I0922 20:05:46.060374 104852 group.cpp:403] Authenticating with ZooKeeper using digest
Sep 22 20:05:46 <agent_id> mesos-master[104837]: 2016-09-22 20:05:46,065:104837(0x7f7a0effd700):ZOO_INFO@check_events@1775: session establishment complete on server [127.0.0.1:2181], sessionId=0x3574e2d05adb711, negotiated timeout=10000
Sep 22 20:05:46 <agent_id> mesos-master[104837]: I0922 20:05:46.066030 104856 group.cpp:349] Group process (group(6)@x.x.9.1:5050) connected to ZooKeeper
Sep 22 20:05:46 <agent_id> mesos-master[104837]: I0922 20:05:46.066056 104856 group.cpp:837] Syncing group operations: queue size (joins, cancels, datas) = (0, 0, 0)
Sep 22 20:05:46 <agent_id> mesos-master[104837]: I0922 20:05:46.066063 104856 group.cpp:403] Authenticating with ZooKeeper using digest
Sep 22 20:05:46 <agent_id> mesos-master[104837]: 2016-09-22 20:05:46,066:104837(0x7f7a0dffb700):ZOO_INFO@check_events@1775: session establishment complete on server [127.0.0.1:2181], sessionId=0x3574e2d05adb712, negotiated timeout=10000
Sep 22 20:05:46 <agent_id> mesos-master[104837]: 2016-09-22 20:05:46,066:104837(0x7f7a0cff9700):ZOO_INFO@check_events@1775: session establishment complete on server [127.0.0.1:2181], sessionId=0x3574e2d05adb713, negotiated timeout=10000
Sep 22 20:05:46 <agent_id> mesos-master[104837]: I0922 20:05:46.066226 104883 group.cpp:349] Group process (group(5)@x.x.9.1:5050) connected to ZooKeeper
Sep 22 20:05:46 <agent_id> mesos-master[104837]: I0922 20:05:46.066254 104883 group.cpp:837] Syncing group operations: queue size (joins, cancels, datas) = (1, 0, 0)
Sep 22 20:05:46 <agent_id> mesos-master[104837]: I0922 20:05:46.066264 104883 group.cpp:403] Authenticating with ZooKeeper using digest
Sep 22 20:05:46 <agent_id> mesos-master[104837]: I0922 20:05:46.066596 104843 group.cpp:349] Group process (group(7)@x.x.9.1:5050) connected to ZooKeeper
Sep 22 20:05:46 <agent_id> mesos-master[104837]: I0922 20:05:46.066627 104843 group.cpp:837] Syncing group operations: queue size (joins, cancels, datas) = (0, 0, 0)
Sep 22 20:05:46 <agent_id> mesos-master[104837]: I0922 20:05:46.066637 104843 group.cpp:403] Authenticating with ZooKeeper using digest
Sep 22 20:05:49 <agent_id> mesos-master[104837]: 2016-09-22 20:05:49,392:104837(0x7f7a0ffff700):ZOO_INFO@auth_completion_func@1300: Authentication scheme digest succeeded
Sep 22 20:05:49 <agent_id> mesos-master[104837]: I0922 20:05:49.392606 104852 group.cpp:427] Trying to create path '/mesos/log_replicas' in ZooKeeper
Sep 22 20:05:49 <agent_id> mesos-master[104837]: 2016-09-22 20:05:49,392:104837(0x7f7a0effd700):ZOO_INFO@auth_completion_func@1300: Authentication scheme digest succeeded
Sep 22 20:05:49 <agent_id> mesos-master[104837]: I0922 20:05:49.392740 104856 group.cpp:427] Trying to create path '/mesos' in ZooKeeper
Sep 22 20:05:49 <agent_id> mesos-master[104837]: 2016-09-22 20:05:49,393:104837(0x7f7a0dffb700):ZOO_INFO@auth_completion_func@1300: Authentication scheme digest succeeded
Sep 22 20:05:49 <agent_id> mesos-master[104837]: I0922 20:05:49.393430 104883 group.cpp:427] Trying to create path '/mesos/log_replicas' in ZooKeeper
Sep 22 20:05:49 <agent_id> mesos-master[104837]: 2016-09-22 20:05:49,393:104837(0x7f7a0cff9700):ZOO_INFO@auth_completion_func@1300: Authentication scheme digest succeeded
Sep 22 20:05:49 <agent_id> mesos-master[104837]: I0922 20:05:49.393985 104881 network.hpp:430] ZooKeeper group memberships changed
Sep 22 20:05:49 <agent_id> mesos-master[104837]: I0922 20:05:49.394183 104887 group.cpp:706] Trying to get '/mesos/log_replicas/0000003288' in ZooKeeper
Sep 22 20:05:49 <agent_id> mesos-master[104837]: I0922 20:05:49.394270 104843 group.cpp:427] Trying to create path '/mesos' in ZooKeeper
Sep 22 20:05:49 <agent_id> mesos-master[104837]: I0922 20:05:49.403239 104887 group.cpp:706] Trying to get '/mesos/log_replicas/0000003289' in ZooKeeper
Sep 22 20:05:49 <agent_id> mesos-master[104837]: I0922 20:05:49.403499 104859 detector.cpp:152] Detected a new leader: (id='2763')
Sep 22 20:05:49 <agent_id> mesos-master[104837]: I0922 20:05:49.403614 104849 group.cpp:706] Trying to get '/mesos/json.info_0000002763' in ZooKeeper
Sep 22 20:05:49 <agent_id> mesos-master[104837]: I0922 20:05:49.403832 104866 network.hpp:478] ZooKeeper group PIDs: { log-replica(2)@x.x.11.1:5050, log-replica(2)@x.x.11.1:5050 }
Sep 22 20:05:49 <agent_id> mesos-master[104837]: I0922 20:05:49.403944 104842 zookeeper.cpp:259] A new leading master (UPID=master@x.x.11.1:5050) is detected
Sep 22 20:05:49 <agent_id> mesos-master[104837]: I0922 20:05:49.404078 104875 network.hpp:430] ZooKeeper group memberships changed
Sep 22 20:05:49 <agent_id> mesos-master[104837]: I0922 20:05:49.404206 104884 group.cpp:706] Trying to get '/mesos/log_replicas/0000003288' in ZooKeeper
Sep 22 20:05:49 <agent_id> mesos-master[104837]: I0922 20:05:49.404521 104884 group.cpp:706] Trying to get '/mesos/log_replicas/0000003289' in ZooKeeper
Sep 22 20:05:49 <agent_id> mesos-master[104837]: I0922 20:05:49.404736 104884 group.cpp:706] Trying to get '/mesos/log_replicas/0000003290' in ZooKeeper
Sep 22 20:05:49 <agent_id> mesos-master[104837]: I0922 20:05:49.404980 104884 network.hpp:478] ZooKeeper group PIDs: { log-replica(2)@x.x.11.1:5050, log-replica(2)@x.x.9.1:5050, log-replica(2)@x.x.11.1:5050 }
Sep 22 20:05:57 <agent_id> mesos-master[104837]: I0922 20:05:57.483232 104867 replica.cpp:673] Replica in EMPTY status received a broadcasted recover request from (13)@x.x.9.1:5050
Sep 22 20:05:57 <agent_id> mesos-master[104837]: I0922 20:05:57.483387 104850 recover.cpp:197] Received a recover response from a replica in EMPTY status
{code}
was (Author: akaplan-mesosphere):
Additional logs:
{code}
Sep 22 20:03:25 <agent_id> mesos-master[x.x78]: I0922 20:03:25.194582 104504 replica.cpp:673] Replica in EMPTY status received a broadcasted recover request from (20)@x.x.9.1:5050
Sep 22 20:03:25 <agent_id> mesos-master[x.x78]: I0922 20:03:25.194675 104520 recover.cpp:197] Received a recover response from a replica in EMPTY status
Sep 22 20:03:25 <agent_id> mesos-master[x.x78]: I0922 20:03:25.349588 104504 replica.cpp:673] Replica in EMPTY status received a broadcasted recover request from (23)@x.x.9.1:5050
Sep 22 20:03:25 <agent_id> mesos-master[x.x78]: I0922 20:03:25.349642 104512 recover.cpp:197] Received a recover response from a replica in EMPTY status
Sep 22 20:03:41 <agent_id> mesos-master[x.x78]: I0922 20:03:41.352486 104529 recover.cpp:110] Unable to finish the recover protocol in 10secs, retrying
Sep 22 20:03:41 <agent_id> mesos-master[x.x78]: I0922 20:03:41.352586 104529 recover.cpp:110] Unable to finish the recover protocol in 10secs, retrying
Sep 22 20:03:41 <agent_id> mesos-master[x.x78]: I0922 20:03:41.352995 104510 replica.cpp:673] Replica in EMPTY status received a broadcasted recover request from (26)@x.x.9.1:5050
Sep 22 20:03:41 <agent_id> mesos-master[x.x78]: I0922 20:03:41.353251 104525 recover.cpp:197] Received a recover response from a replica in EMPTY status
Sep 22 20:03:41 <agent_id> mesos-master[x.x78]: I0922 20:03:41.353310 104510 replica.cpp:673] Replica in EMPTY status received a broadcasted recover request from (29)@x.x.9.1:5050
Sep 22 20:03:41 <agent_id> mesos-master[x.x78]: I0922 20:03:41.353536 x.x94 recover.cpp:197] Received a recover response from a replica in EMPTY status
Sep 22 20:03:57 <agent_id> mesos-master[x.x78]: I0922 20:03:57.914526 104529 recover.cpp:110] Unable to finish the recover protocol in 10secs, retrying
Sep 22 20:03:57 <agent_id> mesos-master[x.x78]: I0922 20:03:57.915133 104529 recover.cpp:110] Unable to finish the recover protocol in 10secs, retrying
Sep 22 20:03:57 <agent_id> mesos-master[x.x78]: I0922 20:03:57.915585 104502 replica.cpp:673] Replica in EMPTY status received a broadcasted recover request from (32)@x.x.9.1:5050
Sep 22 20:03:57 <agent_id> mesos-master[x.x78]: I0922 20:03:57.915774 104525 recover.cpp:197] Received a recover response from a replica in EMPTY status
Sep 22 20:03:57 <agent_id> mesos-master[x.x78]: I0922 20:03:57.915876 104514 replica.cpp:673] Replica in EMPTY status received a broadcasted recover request from (35)@x.x.9.1:5050
Sep 22 20:03:57 <agent_id> mesos-master[x.x78]: I0922 20:03:57.916045 104510 recover.cpp:197] Received a recover response from a replica in EMPTY status
Sep 22 20:04:21 <agent_id> mesos-master[x.x78]: I0922 20:04:21.397429 104529 recover.cpp:110] Unable to finish the recover protocol in 10secs, retrying
Sep 22 20:04:21 <agent_id> mesos-master[x.x78]: I0922 20:04:21.397532 104529 recover.cpp:110] Unable to finish the recover protocol in 10secs, retrying
Sep 22 20:04:21 <agent_id> mesos-master[x.x78]: I0922 20:04:21.397794 104523 replica.cpp:673] Replica in EMPTY status received a broadcasted recover request from (38)@x.x.9.1:5050
Sep 22 20:04:21 <agent_id> mesos-master[x.x78]: I0922 20:04:21.397994 x.x82 replica.cpp:673] Replica in EMPTY status received a broadcasted recover request from (13)@x.x.11.1:5050
Sep 22 20:04:21 <agent_id> mesos-master[x.x78]: I0922 20:04:21.398031 x.x82 replica.cpp:673] Replica in EMPTY status received a broadcasted recover request from (41)@x.x.9.1:5050
Sep 22 20:04:21 <agent_id> mesos-master[x.x78]: I0922 20:04:21.398175 x.x87 recover.cpp:197] Received a recover response from a replica in EMPTY status
Sep 22 20:04:21 <agent_id> mesos-master[x.x78]: I0922 20:04:21.398236 104516 recover.cpp:197] Received a recover response from a replica in EMPTY status
Sep 22 20:04:29 <agent_id> mesos-master[x.x78]: I0922 20:04:29.4x.x6 x.x97 replica.cpp:673] Replica in EMPTY status received a broadcasted recover request from (12)@x.x.11.1:5050
Sep 22 20:04:29 <agent_id> mesos-master[x.x78]: I0922 20:04:29.449352 x.x90 replica.cpp:673] Replica in EMPTY status received a broadcasted recover request from (14)@x.x.11.1:5050
Sep 22 20:04:29 <agent_id> mesos-master[x.x78]: I0922 20:04:29.449399 x.x93 replica.cpp:673] Replica in EMPTY status received a broadcasted recover request from (22)@x.x.11.1:5050
Sep 22 20:04:29 <agent_id> mesos-master[x.x78]: I0922 20:04:29.872488 x.x84 manager.cpp:201] Overlay master overlay-master@x.x.11.1:5050 has exited
Sep 22 20:04:29 <agent_id> mesos-master[x.x78]: W0922 20:04:29.872529 x.x84 manager.cpp:204] Overlay master disconnected! Waiting for a new overlay master to be detected
Sep 22 20:04:29 <agent_id> mesos-master[x.x78]: I0922 20:04:29.872550 x.x84 manager.cpp:208] Moving overlay-master@x.x.11.1:5050 to `REGISTERING` state.
Sep 22 20:04:37 <agent_id> mesos-master[x.x78]: I0922 20:04:37.898635 104507 replica.cpp:673] Replica in EMPTY status received a broadcasted recover request from (16)@x.x.11.1:5050
Sep 22 20:04:37 <agent_id> mesos-master[x.x78]: I0922 20:04:37.899080 104529 recover.cpp:110] Unable to finish the recover protocol in 10secs, retrying
Sep 22 20:04:37 <agent_id> mesos-master[x.x78]: I0922 20:04:37.899253 104529 recover.cpp:110] Unable to finish the recover protocol in 10secs, retrying
Sep 22 20:04:37 <agent_id> mesos-master[x.x78]: I0922 20:04:37.899842 x.x90 replica.cpp:673] Replica in EMPTY status received a broadcasted recover request from (48)@x.x.9.1:5050
Sep 22 20:04:37 <agent_id> mesos-master[x.x78]: I0922 20:04:37.900182 104526 recover.cpp:197] Received a recover response from a replica in EMPTY status
Sep 22 20:04:37 <agent_id> mesos-master[x.x78]: I0922 20:04:37.900218 x.x84 replica.cpp:673] Replica in EMPTY status received a broadcasted recover request from (50)@x.x.9.1:5050
Sep 22 20:04:37 <agent_id> mesos-master[x.x78]: I0922 20:04:37.900280 x.x98 recover.cpp:197] Received a recover response from a replica in EMPTY status
Sep 22 20:04:37 <agent_id> mesos-master[x.x78]: E0922 20:04:37.900727 104529 process.cpp:2104] Failed to shutdown socket with fd 46: Transport endpoint is not connected
Sep 22 20:04:40 <agent_id> mesos-master[x.x78]: I0922 20:04:40.013473 104525 network.hpp:430] ZooKeeper group memberships changed
Sep 22 20:04:40 <agent_id> mesos-master[x.x78]: I0922 20:04:40.013525 104505 group.cpp:706] Trying to get '/mesos/overlay_log_replicas/0000003289' in ZooKeeper
Sep 22 20:04:40 <agent_id> mesos-master[x.x78]: I0922 20:04:40.013557 104520 detector.cpp:152] Detected a new leader: (id='2760')
Sep 22 20:04:40 <agent_id> mesos-master[x.x78]: I0922 20:04:40.013662 104523 network.hpp:430] ZooKeeper group memberships changed
Sep 22 20:04:40 <agent_id> mesos-master[x.x78]: I0922 20:04:40.013720 104528 detector.cpp:152] Detected a new leader: (id='2760')
Sep 22 20:04:40 <agent_id> mesos-master[x.x78]: I0922 20:04:40.013746 104503 group.cpp:706] Trying to get '/mesos/json.info_0000002760' in ZooKeeper
Sep 22 20:04:40 <agent_id> mesos-master[x.x78]: I0922 20:04:40.013777 x.x95 group.cpp:706] Trying to get '/mesos/json.info_0000002760' in ZooKeeper
Sep 22 20:04:40 <agent_id> mesos-master[x.x78]: I0922 20:04:40.013813 x.x86 group.cpp:706] Trying to get '/mesos/log_replicas/0000003284' in ZooKeeper
Sep 22 20:04:40 <agent_id> mesos-master[x.x78]: I0922 20:04:40.014015 104528 network.hpp:478] ZooKeeper group PIDs: { log-replica(2)@x.x.9.1:5050 }
Sep 22 20:04:40 <agent_id> mesos-master[x.x78]: I0922 20:04:40.014298 x.x97 network.hpp:478] ZooKeeper group PIDs: { log-replica(1)@x.x.9.1:5050 }
Sep 22 20:04:40 <agent_id> mesos-master[x.x78]: I0922 20:04:40.014318 x.x99 zookeeper.cpp:259] A new leading master (UPID=master@x.x.9.1:5050) is detected
Sep 22 20:04:40 <agent_id> mesos-master[x.x78]: I0922 20:04:40.014444 104526 zookeeper.cpp:259] A new leading master (UPID=master@x.x.9.1:5050) is detected
Sep 22 20:04:40 <agent_id> mesos-master[x.x78]: I0922 20:04:40.014487 x.x90 master.cpp:1847] The newly elected leader is master@x.x.9.1:5050 with id 184cd1f8-d3ff-4c90-8559-0ac41ea81616
Sep 22 20:04:40 <agent_id> mesos-master[x.x78]: I0922 20:04:40.014503 x.x90 master.cpp:1860] Elected as the leading master!
Sep 22 20:04:40 <agent_id> mesos-master[x.x78]: I0922 20:04:40.014504 x.x87 manager.cpp:397] Detected new overlay master at overlay-master@x.x.9.1:5050
Sep 22 20:04:40 <agent_id> mesos-master[x.x78]: I0922 20:04:40.014511 x.x90 master.cpp:1547] Recovering from registrar
Sep 22 20:04:40 <agent_id> mesos-master[x.x78]: I0922 20:04:40.014555 x.x86 manager.cpp:782] Got registration from pid: overlay-agent@x.x.9.1:5050
Sep 22 20:04:40 <agent_id> mesos-master[x.x78]: I0922 20:04:40.014582 x.x86 manager.cpp:787] overlay-master moving to `RECOVERING` state . Hence, not sending an update to agentoverlay-agent@x.x.9.1:5050
Sep 22 20:04:40 <agent_id> mesos-master[x.x78]: I0922 20:04:40.014590 104522 registrar.cpp:332] Recovering registrar
Sep 22 20:04:45 <agent_id> mesos-master[x.x78]: I0922 20:04:45.911886 104525 logfmt.cpp:164] dstip=x.x.9.1 type=audit timestamp=2016-09-22 20:04:45.911847936+00:00 reason="Valid authorization token" uid="dcos_mesos_dns" object="/master/state.json" authorizer="mesos-master" action="GET" result=allow srcip=x.x.11.1 dstport=5050 srcport=51602
Sep 22 20:04:45 <agent_id> mesos-master[x.x78]: I0922 20:04:45.912039 x.x84 http.cpp:381] HTTP GET for /master/state.json from x.x.11.1:51602 with User-Agent='Mesos-DNS'
Sep 22 20:04:53 <agent_id> mesos-master[x.x78]: I0922 20:04:53.121378 104522 network.hpp:430] ZooKeeper group memberships changed
Sep 22 20:04:53 <agent_id> mesos-master[x.x78]: I0922 20:04:53.121448 104523 group.cpp:706] Trying to get '/mesos/overlay_log_replicas/0000003289' in ZooKeeper
Sep 22 20:04:53 <agent_id> mesos-master[x.x78]: I0922 20:04:53.121789 104523 group.cpp:706] Trying to get '/mesos/overlay_log_replicas/0000003291' in ZooKeeper
Sep 22 20:04:53 <agent_id> mesos-master[x.x78]: I0922 20:04:53.122149 104502 network.hpp:478] ZooKeeper group PIDs: { log-replica(1)@x.x.11.1:5050, log-replica(1)@x.x.9.1:5050 }
Sep 22 20:04:53 <agent_id> mesos-master[x.x78]: I0922 20:04:53.309017 104509 network.hpp:430] ZooKeeper group memberships changed
Sep 22 20:04:53 <agent_id> mesos-master[x.x78]: I0922 20:04:53.309114 104502 group.cpp:706] Trying to get '/mesos/log_replicas/0000003284' in ZooKeeper
Sep 22 20:04:53 <agent_id> mesos-master[x.x78]: I0922 20:04:53.309340 104502 group.cpp:706] Trying to get '/mesos/log_replicas/0000003286' in ZooKeeper
Sep 22 20:04:53 <agent_id> mesos-master[x.x78]: I0922 20:04:53.309658 x.x81 network.hpp:478] ZooKeeper group PIDs: { log-replica(2)@x.x.11.1:5050, log-replica(2)@x.x.9.1:5050 }
Sep 22 20:04:53 <agent_id> mesos-master[x.x78]: I0922 20:04:53.732125 x.x89 network.hpp:430] ZooKeeper group memberships changed
Sep 22 20:04:53 <agent_id> mesos-master[x.x78]: I0922 20:04:53.732303 104507 group.cpp:706] Trying to get '/mesos/overlay_log_replicas/0000003289' in ZooKeeper
Sep 22 20:04:53 <agent_id> mesos-master[x.x78]: I0922 20:04:53.732770 104507 group.cpp:706] Trying to get '/mesos/overlay_log_replicas/0000003291' in ZooKeeper
Sep 22 20:04:53 <agent_id> mesos-master[x.x78]: I0922 20:04:53.733115 104507 group.cpp:706] Trying to get '/mesos/overlay_log_replicas/0000003292' in ZooKeeper
Sep 22 20:04:53 <agent_id> mesos-master[x.x78]: I0922 20:04:53.733594 x.x88 network.hpp:478] ZooKeeper group PIDs: { log-replica(1)@x.x.11.1:5050, log-replica(1)@x.x.9.1:5050, log-replica(1)@x.x.11.1:5050 }
Sep 22 20:04:53 <agent_id> mesos-master[x.x78]: I0922 20:04:53.886611 104503 network.hpp:430] ZooKeeper group memberships changed
Sep 22 20:04:53 <agent_id> mesos-master[x.x78]: I0922 20:04:53.886776 x.x99 group.cpp:706] Trying to get '/mesos/log_replicas/0000003284' in ZooKeeper
Sep 22 20:04:53 <agent_id> mesos-master[x.x78]: I0922 20:04:53.887078 x.x99 group.cpp:706] Trying to get '/mesos/log_replicas/0000003286' in ZooKeeper
Sep 22 20:04:53 <agent_id> mesos-master[x.x78]: I0922 20:04:53.887272 x.x99 group.cpp:706] Trying to get '/mesos/log_replicas/0000003287' in ZooKeeper
Sep 22 20:04:53 <agent_id> mesos-master[x.x78]: I0922 20:04:53.887572 104500 network.hpp:478] ZooKeeper group PIDs: { log-replica(2)@x.x.11.1:5050, log-replica(2)@x.x.9.1:5050, log-replica(2)@x.x.11.1:5050 }
Sep 22 20:04:53 <agent_id> mesos-master[x.x78]: I0922 20:04:53.919270 104529 recover.cpp:110] Unable to finish the recover protocol in 10secs, retrying
Sep 22 20:04:53 <agent_id> mesos-master[x.x78]: I0922 20:04:53.919317 104528 manager.cpp:782] Got registration from pid: overlay-agent@x.x.9.1:5050
Sep 22 20:04:53 <agent_id> mesos-master[x.x78]: I0922 20:04:53.919333 104529 recover.cpp:110] Unable to finish the recover protocol in 10secs, retrying
Sep 22 20:04:53 <agent_id> mesos-master[x.x78]: I0922 20:04:53.919342 104528 manager.cpp:795] overlay-master in `RECOVERING` state . Hence, not sending an update to agentoverlay-agent@x.x.9.1:5050
Sep 22 20:04:53 <agent_id> mesos-master[x.x78]: I0922 20:04:53.919360 104528 manager.cpp:782] Got registration from pid: overlay-agent@x.x.9.1:5050
Sep 22 20:04:53 <agent_id> mesos-master[x.x78]: I0922 20:04:53.919368 104528 manager.cpp:795] overlay-master in `RECOVERING` state . Hence, not sending an update to agentoverlay-agent@x.x.9.1:5050
Sep 22 20:04:53 <agent_id> mesos-master[x.x78]: I0922 20:04:53.921273 104523 logfmt.cpp:164] dstip=x.x.9.1 type=audit timestamp=2016-09-22 20:04:53.921229056+00:00 reason="Valid authorization token" uid="dcos_mesos_dns" object="/master/state.json" authorizer="mesos-master" action="GET" result=allow srcip=x.x.11.1 dstport=5050 srcport=34076
Sep 22 20:04:53 <agent_id> mesos-master[x.x78]: I0922 20:04:53.921572 104523 logfmt.cpp:164] dstip=x.x.9.1 type=audit timestamp=2016-09-22 20:04:53.921534976+00:00 reason="Valid authorization token" uid="dcos_mesos_dns" object="/master/state.json" authorizer="mesos-master" action="GET" result=allow srcip=x.x.9.1 dstport=5050 srcport=35372
Sep 22 20:04:53 <agent_id> mesos-master[x.x78]: I0922 20:04:53.921723 x.x86 http.cpp:381] HTTP GET for /master/state.json from x.x.9.1:35372 with User-Agent='Mesos-DNS'
Sep 22 20:04:53 <agent_id> mesos-master[x.x78]: I0922 20:04:53.921826 x.x86 http.cpp:381] HTTP GET for /master/state.json from x.x.11.1:34076 with User-Agent='Mesos-DNS'
Sep 22 20:05:08 <agent_id> mesos-master[x.x78]: I0922 20:05:08.023877 104522 network.hpp:430] ZooKeeper group memberships changed
Sep 22 20:05:08 <agent_id> mesos-master[x.x78]: I0922 20:05:08.023988 x.x82 network.hpp:430] ZooKeeper group memberships changed
Sep 22 20:05:08 <agent_id> mesos-master[x.x78]: I0922 20:05:08.024041 x.x88 group.cpp:706] Trying to get '/mesos/overlay_log_replicas/0000003289' in ZooKeeper
Sep 22 20:05:08 <agent_id> mesos-master[x.x78]: I0922 20:05:08.024093 x.x87 group.cpp:706] Trying to get '/mesos/log_replicas/0000003284' in ZooKeeper
Sep 22 20:05:08 <agent_id> mesos-master[x.x78]: I0922 20:05:08.024291 x.x88 group.cpp:706] Trying to get '/mesos/overlay_log_replicas/0000003292' in ZooKeeper
Sep 22 20:05:08 <agent_id> mesos-master[x.x78]: I0922 20:05:08.024492 x.x87 group.cpp:706] Trying to get '/mesos/log_replicas/0000003287' in ZooKeeper
Sep 22 20:05:08 <agent_id> mesos-master[x.x78]: I0922 20:05:08.024902 104511 network.hpp:478] ZooKeeper group PIDs: { log-replica(1)@x.x.9.1:5050, log-replica(1)@x.x.11.1:5050 }
Sep 22 20:05:08 <agent_id> mesos-master[x.x78]: I0922 20:05:08.025151 104506 network.hpp:478] ZooKeeper group PIDs: { log-replica(2)@x.x.9.1:5050, log-replica(2)@x.x.11.1:5050 }
Sep 22 20:05:12 <agent_id> mesos-master[x.x78]: I0922 20:05:12.016031 104522 network.hpp:430] ZooKeeper group memberships changed
Sep 22 20:05:12 <agent_id> mesos-master[x.x78]: I0922 20:05:12.016280 104527 network.hpp:430] ZooKeeper group memberships changed
Sep 22 20:05:12 <agent_id> mesos-master[x.x78]: I0922 20:05:12.016337 x.x98 group.cpp:706] Trying to get '/mesos/log_replicas/0000003284' in ZooKeeper
Sep 22 20:05:12 <agent_id> mesos-master[x.x78]: I0922 20:05:12.016345 x.x82 group.cpp:706] Trying to get '/mesos/overlay_log_replicas/0000003289' in ZooKeeper
Sep 22 20:05:12 <agent_id> mesos-master[x.x78]: I0922 20:05:12.016741 104522 network.hpp:478] ZooKeeper group PIDs: { log-replica(1)@x.x.9.1:5050 }
Sep 22 20:05:12 <agent_id> mesos-master[x.x78]: I0922 20:05:12.016783 104510 network.hpp:478] ZooKeeper group PIDs: { log-replica(2)@x.x.9.1:5050 }
Sep 22 20:05:17 <agent_id> mesos-master[x.x78]: I0922 20:05:17.947398 104529 recover.cpp:110] Unable to finish the recover protocol in 10secs, retrying
Sep 22 20:05:17 <agent_id> mesos-master[x.x78]: I0922 20:05:17.947479 104529 recover.cpp:110] Unable to finish the recover protocol in 10secs, retrying
Sep 22 20:05:17 <agent_id> mesos-master[x.x78]: I0922 20:05:17.947873 104511 manager.cpp:782] Got registration from pid: overlay-agent@x.x.9.1:5050
Sep 22 20:05:17 <agent_id> mesos-master[x.x78]: I0922 20:05:17.947899 104511 manager.cpp:795] overlay-master in `RECOVERING` state . Hence, not sending an update to agentoverlay-agent@x.x.9.1:5050
Sep 22 20:05:17 <agent_id> mesos-master[x.x78]: I0922 20:05:17.947917 104511 manager.cpp:782] Got registration from pid: overlay-agent@x.x.9.1:5050
Sep 22 20:05:17 <agent_id> mesos-master[x.x78]: I0922 20:05:17.947926 104511 manager.cpp:795] overlay-master in `RECOVERING` state . Hence, not sending an update to agentoverlay-agent@x.x.9.1:5050
Sep 22 20:05:17 <agent_id> mesos-master[x.x78]: I0922 20:05:17.947978 x.x95 manager.cpp:782] Got registration from pid: overlay-agent@x.x.4.1:5051
Sep 22 20:05:17 <agent_id> mesos-master[x.x78]: I0922 20:05:17.948004 x.x95 manager.cpp:795] overlay-master in `RECOVERING` state . Hence, not sending an update to agentoverlay-agent@x.x.4.1:5051
Sep 22 20:05:17 <agent_id> mesos-master[x.x78]: F0922 20:05:17.948120 104529 process.cpp:2243] Check failed: sockets.count(from_fd) > 0
Sep 22 20:05:17 <agent_id> mesos-master[x.x78]: *** Check failure stack trace: ***
Sep 22 20:05:17 <agent_id> mesos-master[x.x78]: @ 0x7fc1908829fd google::LogMessage::Fail()
Sep 22 20:05:17 <agent_id> mesos-master[x.x78]: @ 0x7fc19088482d google::LogMessage::SendToLog()
Sep 22 20:05:17 <agent_id> mesos-master[x.x78]: @ 0x7fc1908825ec google::LogMessage::Flush()
Sep 22 20:05:17 <agent_id> mesos-master[x.x78]: @ 0x7fc190885129 google::LogMessageFatal::~LogMessageFatal()
Sep 22 20:05:17 <agent_id> mesos-master[x.x78]: @ 0x7fc1908171dd process::SocketManager::swap_implementing_socket()
Sep 22 20:05:17 <agent_id> mesos-master[x.x78]: @ 0x7fc19081aa90 process::SocketManager::link_connect()
Sep 22 20:05:17 <agent_id> mesos-master[x.x78]: @ 0x7fc1908227f9 _ZNSt17_Function_handlerIFvRKN7process6FutureI7NothingEEEZNKS3_5onAnyISt5_BindIFSt7_Mem_fnIMNS0_13SocketManagerEFvS5_NS0_7network6SocketERKNS0_4UPIDEEEPSA_St12_PlaceholderILi1EESC_SD_EEvEES5_OT_NS3_6PreferEEUlS5_E_E9_M_invokeERKSt9_Any_dataS5_
Sep 22 20:05:17 <agent_id> mesos-master[x.x78]: @ 0x41eb26 _ZN7process8internal3runISt8functionIFvRKNS_6FutureI7NothingEEEEJRS5_EEEvRKSt6vectorIT_SaISC_EEDpOT0_
Sep 22 20:05:17 <agent_id> mesos-master[x.x78]: @ 0x42a36f process::Future<>::fail()
Sep 22 20:05:17 <agent_id> mesos-master[x.x78]: @ 0x7fc19085283c process::network::LibeventSSLSocketImpl::event_callback()
Sep 22 20:05:17 <agent_id> mesos-master[x.x78]: @ 0x7fc190852f17 process::network::LibeventSSLSocketImpl::event_callback()
Sep 22 20:05:17 <agent_id> mesos-master[x.x78]: @ 0x7fc18d616631 bufferevent_run_deferred_callbacks_locked
Sep 22 20:05:17 <agent_id> mesos-master[x.x78]: @ 0x7fc18d60cc5d event_base_loop
Sep 22 20:05:17 <agent_id> mesos-master[x.x78]: @ 0x7fc190865a1d process::EventLoop::run()
Sep 22 20:05:17 <agent_id> mesos-master[x.x78]: @ 0x7fc18eeabd73 (unknown)
Sep 22 20:05:17 <agent_id> mesos-master[x.x78]: @ 0x7fc18e6a852c (unknown)
Sep 22 20:05:17 <agent_id> mesos-master[x.x78]: @ 0x7fc18e3e61dd (unknown)
Sep 22 20:05:18 <agent_id> systemd[1]: [0;1;39mdcos-mesos-master.service: Main process exited, code=killed, status=6/ABRT[0m
Sep 22 20:05:18 <agent_id> systemd[1]: [0;1;39mdcos-mesos-master.service: Unit entered failed state.[0m
Sep 22 20:05:18 <agent_id> systemd[1]: [0;1;39mdcos-mesos-master.service: Failed with result 'signal'.[0m
Sep 22 20:05:33 <agent_id> systemd[1]: dcos-mesos-master.service: Service hold-off time over, scheduling restart.
Sep 22 20:05:33 <agent_id> systemd[1]: Stopped Mesos Master: DC/OS Mesos Master Service.
Sep 22 20:05:33 <agent_id> systemd[1]: Starting Mesos Master: DC/OS Mesos Master Service...
Sep 22 20:05:33 <agent_id> ping[104808]: PING ready.spartan (127.0.0.1) 56(84) bytes of data.
Sep 22 20:05:33 <agent_id> ping[104808]: 64 bytes from <agent_id> (127.0.0.1): icmp_seq=1 ttl=64 time=0.025 ms
Sep 22 20:05:33 <agent_id> ping[104808]: --- ready.spartan ping statistics ---
Sep 22 20:05:33 <agent_id> ping[104808]: 1 packets transmitted, 1 received, 0% packet loss, time 0ms
Sep 22 20:05:33 <agent_id> ping[104808]: rtt min/avg/max/mdev = 0.025/0.025/0.025/0.000 ms
Sep 22 20:05:33 <agent_id> bash[104810]: /proc/sys/net/ipv4/conf/all/rp_filter: 2
Sep 22 20:05:33 <agent_id> bash[104810]: /proc/sys/net/ipv4/conf/cbr0/rp_filter: 2
Sep 22 20:05:33 <agent_id> bash[104810]: /proc/sys/net/ipv4/conf/default/rp_filter: 2
Sep 22 20:05:33 <agent_id> bash[104810]: /proc/sys/net/ipv4/conf/dummy0/rp_filter: 2
Sep 22 20:05:33 <agent_id> bash[104810]: /proc/sys/net/ipv4/conf/eno1/rp_filter: 2
Sep 22 20:05:33 <agent_id> bash[104810]: /proc/sys/net/ipv4/conf/eno2/rp_filter: 2
Sep 22 20:05:33 <agent_id> bash[104810]: /proc/sys/net/ipv4/conf/eno3/rp_filter: 2
Sep 22 20:05:33 <agent_id> bash[104810]: /proc/sys/net/ipv4/conf/eno4/rp_filter: 2
Sep 22 20:05:33 <agent_id> bash[104810]: /proc/sys/net/ipv4/conf/lo/rp_filter: 2
Sep 22 20:05:33 <agent_id> bash[104810]: /proc/sys/net/ipv4/conf/spartan/rp_filter: 2
Sep 22 20:05:33 <agent_id> bootstrap[104823]: [INFO] Clearing proxy environment variables
Sep 22 20:05:33 <agent_id> bootstrap[104823]: [INFO] PID 3794 has command line [b'/opt/mesosphere/active/java/usr/java/bin/java', b'-Dzookeeper.log.dir=/var/lib/dcos/exhibitor/zookeeper', b'-Dzookeeper.root.logger=INFO,CONSOLE', b'-cp', b'/opt/mesosphere/active/exhibitor/usr/zookeeper/bin/../build/classes:/opt/mesosphere/active/exhibitor/usr/zookeeper/bin/../build/lib/*.jar:/opt/mesosphere/active/exhibitor/usr/zookeeper/bin/../lib/slf4j-log4j12-1.6.1.jar:/opt/mesosphere/active/exhibitor/usr/zookeeper/bin/../lib/slf4j-api-1.6.1.jar:/opt/mesosphere/active/exhibitor/usr/zookeeper/bin/../lib/netty-3.7.0.Final.jar:/opt/mesosphere/active/exhibitor/usr/zookeeper/bin/../lib/log4j-systemd-journal-appender-1.3.2.jar:/opt/mesosphere/active/exhibitor/usr/zookeeper/bin/../lib/log4j-1.2.16.jar:/opt/mesosphere/active/exhibitor/usr/zookeeper/bin/../lib/jna-4.2.2.jar:/opt/mesosphere/active/exhibitor/usr/zookeeper/bin/../lib/jline-0.9.94.jar:/opt/mesosphere/active/exhibitor/usr/zookeeper/bin/../zookeeper-3.4.8.jar:/opt/mesosphere/active/exhibitor/usr/zookeeper/bin/../src/java/lib/*.jar:/var/lib/dcos/exhibitor/conf:', b'-Dzookeeper.DigestAuthenticationProvider.superDigest=super:lK75jTNcA+U9vtVEw5vB51mj/w4=', b'-Dcom.sun.management.jmxremote', b'-Dcom.sun.management.jmxremote.local.only=false', b'org.apache.zookeeper.server.quorum.QuorumPeerMain', b'/var/lib/dcos/exhibitor/conf/zoo.cfg']
Sep 22 20:05:33 <agent_id> bootstrap[104823]: [INFO] PID file hasn't been modified. ZK still seems to be at that PID.
Sep 22 20:05:33 <agent_id> bootstrap[104823]: [INFO] Shortcut succeeeded, assuming local zk is in good config state, not waiting for quorum.
Sep 22 20:05:33 <agent_id> bootstrap[104823]: [INFO] Preparing directory /run/dcos
Sep 22 20:05:33 <agent_id> bootstrap[104823]: [INFO] Preparing directory /run/dcos/etc
Sep 22 20:05:33 <agent_id> bootstrap[104823]: [INFO] Preparing directory /run/dcos/etc/3dt
Sep 22 20:05:33 <agent_id> bootstrap[104823]: [INFO] Preparing directory /run/dcos/etc/marathon
Sep 22 20:05:33 <agent_id> bootstrap[104823]: [INFO] Preparing directory /run/dcos/etc/mesos
Sep 22 20:05:33 <agent_id> bootstrap[104823]: [INFO] Preparing directory /run/dcos/etc/mesos-dns
Sep 22 20:05:33 <agent_id> bootstrap[104823]: [INFO] Preparing directory /run/dcos/etc/dcos-ca
Sep 22 20:05:33 <agent_id> bootstrap[104823]: [INFO] Preparing directory /run/dcos/etc/metronome
Sep 22 20:05:33 <agent_id> bootstrap[104823]: [INFO] Preparing directory /run/dcos/etc/history-service
Sep 22 20:05:33 <agent_id> bootstrap[104823]: [INFO] Preparing directory /run/dcos/etc/signal-service
Sep 22 20:05:33 <agent_id> bootstrap[104823]: [INFO] Preparing directory /run/dcos/pki/tls/private
Sep 22 20:05:33 <agent_id> bootstrap[104823]: [INFO] Preparing directory /run/dcos/pki/tls/certs
Sep 22 20:05:33 <agent_id> bootstrap[104823]: [INFO] Preparing directory /run/dcos/pki/CA/certs
Sep 22 20:05:33 <agent_id> bootstrap[104823]: [INFO] Preparing directory /run/dcos/pki/CA/private
Sep 22 20:05:33 <agent_id> bootstrap[104823]: [INFO] Using super credentials for Zookeeper
Sep 22 20:05:33 <agent_id> bootstrap[104823]: [INFO] Connecting to 127.0.0.1:2181
Sep 22 20:05:33 <agent_id> bootstrap[104823]: [INFO] Zookeeper connection established, state: CONNECTED
Sep 22 20:05:33 <agent_id> bootstrap[104823]: [DEBUG] bootstrapping dcos-mesos-master
Sep 22 20:05:33 <agent_id> bootstrap[104823]: [INFO] Initializing ACLs for znode /
Sep 22 20:05:33 <agent_id> bootstrap[104823]: [INFO] ensure_zk_path(/, [ACL(perms=5, acl_list=['READ', 'CREATE'], id=Id(scheme='world', id='anyone')), ACL(perms=31, acl_list=['ALL'], id=Id(scheme='ip', id='127.0.0.1'))])
Sep 22 20:05:33 <agent_id> bootstrap[104823]: [INFO] Initializing ACLs for znode /cosmos
Sep 22 20:05:33 <agent_id> bootstrap[104823]: [INFO] ensure_zk_path(/cosmos, [ACL(perms=31, acl_list=['ALL'], id=Id(scheme='world', id='anyone'))])
Sep 22 20:05:33 <agent_id> bootstrap[104823]: [INFO] Initializing ACLs for znode /dcos
Sep 22 20:05:33 <agent_id> bootstrap[104823]: [INFO] ensure_zk_path(/dcos, [ACL(perms=1, acl_list=['READ'], id=Id(scheme='world', id='anyone'))])
Sep 22 20:05:33 <agent_id> bootstrap[104823]: [INFO] Initializing ACLs for znode /dcos/vault
Sep 22 20:05:33 <agent_id> bootstrap[104823]: [INFO] ensure_zk_path(/dcos/vault, [ACL(perms=1, acl_list=['READ'], id=Id(scheme='world', id='anyone'))])
Sep 22 20:05:33 <agent_id> bootstrap[104823]: [INFO] Initializing ACLs for znode /zookeeper
Sep 22 20:05:33 <agent_id> bootstrap[104823]: [INFO] ensure_zk_path(/zookeeper, [ACL(perms=1, acl_list=['READ'], id=Id(scheme='world', id='anyone'))])
Sep 22 20:05:33 <agent_id> bootstrap[104823]: [INFO] Initializing ACLs for znode /zookeeper/quotas
Sep 22 20:05:33 <agent_id> bootstrap[104823]: [INFO] ensure_zk_path(/zookeeper/quotas, [ACL(perms=1, acl_list=['READ'], id=Id(scheme='world', id='anyone'))])
Sep 22 20:05:33 <agent_id> bootstrap[104823]: [INFO] Creating master secrets with user dcos-master
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] ensure_zk_path(/dcos/master/secrets, [ACL(perms=1, acl_list=['READ'], id=Id(scheme='digest', id='dcos-master:fMkMgKtR6Fl+wYKfdJg75Th6Vsc='))])
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] Reaching consensus about znode /dcos/master/secrets/private_keys
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] Consensus znode /dcos/master/secrets/private_keys already exists
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] ensure_zk_path(/dcos/master/secrets/zk, [ACL(perms=1, acl_list=['READ'], id=Id(scheme='digest', id='dcos-master:fMkMgKtR6Fl+wYKfdJg75Th6Vsc='))])
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] Reaching consensus about znode /dcos/master/secrets/zk/dcos_bouncer
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] Consensus znode /dcos/master/secrets/zk/dcos_bouncer already exists
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] ensure_zk_path(/dcos/master/secrets/zk, [ACL(perms=1, acl_list=['READ'], id=Id(scheme='digest', id='dcos-master:fMkMgKtR6Fl+wYKfdJg75Th6Vsc='))])
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] Reaching consensus about znode /dcos/master/secrets/zk/dcos_mesos_master
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] Consensus znode /dcos/master/secrets/zk/dcos_mesos_master already exists
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] ensure_zk_path(/dcos/master/secrets/zk, [ACL(perms=1, acl_list=['READ'], id=Id(scheme='digest', id='dcos-master:fMkMgKtR6Fl+wYKfdJg75Th6Vsc='))])
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] Reaching consensus about znode /dcos/master/secrets/zk/dcos_secrets
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] Consensus znode /dcos/master/secrets/zk/dcos_secrets already exists
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] ensure_zk_path(/dcos/master/secrets/zk, [ACL(perms=1, acl_list=['READ'], id=Id(scheme='digest', id='dcos-master:fMkMgKtR6Fl+wYKfdJg75Th6Vsc='))])
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] Reaching consensus about znode /dcos/master/secrets/zk/dcos_cosmos
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] Consensus znode /dcos/master/secrets/zk/dcos_cosmos already exists
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] ensure_zk_path(/dcos/master/secrets/zk, [ACL(perms=1, acl_list=['READ'], id=Id(scheme='digest', id='dcos-master:fMkMgKtR6Fl+wYKfdJg75Th6Vsc='))])
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] Reaching consensus about znode /dcos/master/secrets/zk/dcos_ca
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] Consensus znode /dcos/master/secrets/zk/dcos_ca already exists
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] ensure_zk_path(/dcos/master/secrets/zk, [ACL(perms=1, acl_list=['READ'], id=Id(scheme='digest', id='dcos-master:fMkMgKtR6Fl+wYKfdJg75Th6Vsc='))])
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] Reaching consensus about znode /dcos/master/secrets/zk/dcos_marathon
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] Consensus znode /dcos/master/secrets/zk/dcos_marathon already exists
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] ensure_zk_path(/dcos/master/secrets/zk, [ACL(perms=1, acl_list=['READ'], id=Id(scheme='digest', id='dcos-master:fMkMgKtR6Fl+wYKfdJg75Th6Vsc='))])
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] Reaching consensus about znode /dcos/master/secrets/zk/dcos_vault_default
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] Consensus znode /dcos/master/secrets/zk/dcos_vault_default already exists
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] ensure_zk_path(/dcos/master/secrets/zk, [ACL(perms=1, acl_list=['READ'], id=Id(scheme='digest', id='dcos-master:fMkMgKtR6Fl+wYKfdJg75Th6Vsc='))])
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] Reaching consensus about znode /dcos/master/secrets/zk/dcos_metronome
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] Consensus znode /dcos/master/secrets/zk/dcos_metronome already exists
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] ensure_zk_path(/dcos/master/secrets/CA, [ACL(perms=1, acl_list=['READ'], id=Id(scheme='digest', id='dcos-master:fMkMgKtR6Fl+wYKfdJg75Th6Vsc='))])
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] Reaching consensus about znode /dcos/master/secrets/CA/RootCA
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] Consensus znode /dcos/master/secrets/CA/RootCA already exists
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] ensure_zk_path(/dcos/master/secrets/services, [ACL(perms=1, acl_list=['READ'], id=Id(scheme='digest', id='dcos-master:fMkMgKtR6Fl+wYKfdJg75Th6Vsc='))])
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] Reaching consensus about znode /dcos/master/secrets/services/dcos_spartan_master
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] Consensus znode /dcos/master/secrets/services/dcos_spartan_master already exists
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] ensure_zk_path(/dcos/master/secrets/services, [ACL(perms=1, acl_list=['READ'], id=Id(scheme='digest', id='dcos-master:fMkMgKtR6Fl+wYKfdJg75Th6Vsc='))])
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] Reaching consensus about znode /dcos/master/secrets/services/dcos_navstar_master
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] Consensus znode /dcos/master/secrets/services/dcos_navstar_master already exists
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] ensure_zk_path(/dcos/master/secrets/services, [ACL(perms=1, acl_list=['READ'], id=Id(scheme='digest', id='dcos-master:fMkMgKtR6Fl+wYKfdJg75Th6Vsc='))])
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] Reaching consensus about znode /dcos/master/secrets/services/dcos_history_service
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] Consensus znode /dcos/master/secrets/services/dcos_history_service already exists
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] ensure_zk_path(/dcos/master/secrets/services, [ACL(perms=1, acl_list=['READ'], id=Id(scheme='digest', id='dcos-master:fMkMgKtR6Fl+wYKfdJg75Th6Vsc='))])
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] Reaching consensus about znode /dcos/master/secrets/services/dcos_3dt_master
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] Consensus znode /dcos/master/secrets/services/dcos_3dt_master already exists
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] ensure_zk_path(/dcos/master/secrets/services, [ACL(perms=1, acl_list=['READ'], id=Id(scheme='digest', id='dcos-master:fMkMgKtR6Fl+wYKfdJg75Th6Vsc='))])
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] Reaching consensus about znode /dcos/master/secrets/services/dcos_marathon
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] Consensus znode /dcos/master/secrets/services/dcos_marathon already exists
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] ensure_zk_path(/dcos/master/secrets/services, [ACL(perms=1, acl_list=['READ'], id=Id(scheme='digest', id='dcos-master:fMkMgKtR6Fl+wYKfdJg75Th6Vsc='))])
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] Reaching consensus about znode /dcos/master/secrets/services/dcos_networking_api_master
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] Consensus znode /dcos/master/secrets/services/dcos_networking_api_master already exists
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] ensure_zk_path(/dcos/master/secrets/services, [ACL(perms=1, acl_list=['READ'], id=Id(scheme='digest', id='dcos-master:fMkMgKtR6Fl+wYKfdJg75Th6Vsc='))])
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] Reaching consensus about znode /dcos/master/secrets/services/dcos_minuteman_master
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] Consensus znode /dcos/master/secrets/services/dcos_minuteman_master already exists
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] ensure_zk_path(/dcos/master/secrets/services, [ACL(perms=1, acl_list=['READ'], id=Id(scheme='digest', id='dcos-master:fMkMgKtR6Fl+wYKfdJg75Th6Vsc='))])
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] Reaching consensus about znode /dcos/master/secrets/services/dcos_mesos_dns
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] Consensus znode /dcos/master/secrets/services/dcos_mesos_dns already exists
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] ensure_zk_path(/dcos/master/secrets/services, [ACL(perms=1, acl_list=['READ'], id=Id(scheme='digest', id='dcos-master:fMkMgKtR6Fl+wYKfdJg75Th6Vsc='))])
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] Reaching consensus about znode /dcos/master/secrets/services/dcos_signal_service
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] Consensus znode /dcos/master/secrets/services/dcos_signal_service already exists
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] ensure_zk_path(/dcos/master/secrets/services, [ACL(perms=1, acl_list=['READ'], id=Id(scheme='digest', id='dcos-master:fMkMgKtR6Fl+wYKfdJg75Th6Vsc='))])
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] Reaching consensus about znode /dcos/master/secrets/services/dcos_adminrouter
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] Consensus znode /dcos/master/secrets/services/dcos_adminrouter already exists
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] ensure_zk_path(/dcos/master/secrets/services, [ACL(perms=1, acl_list=['READ'], id=Id(scheme='digest', id='dcos-master:fMkMgKtR6Fl+wYKfdJg75Th6Vsc='))])
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] Reaching consensus about znode /dcos/master/secrets/services/dcos_metronome
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] Consensus znode /dcos/master/secrets/services/dcos_metronome already exists
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] ensure_zk_path(/mesos, [ACL(perms=1, acl_list=['READ'], id=Id(scheme='world', id='anyone')), ACL(perms=31, acl_list=['ALL'], id=Id(scheme='ip', id='127.0.0.1')), ACL(perms=31, acl_list=['ALL'], id=Id(scheme='digest', id='dcos_mesos_master:IeBqBqLqnGlC3FoMAKYACqaZlIQ='))])
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] Writing Mesos Master ZK credentials to /run/dcos/etc/mesos-master
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] Opening /run/dcos/etc
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] Opened /run/dcos/etc with fd 8
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] Locking fd 8
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] Locked fd 8
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] Writing /run/dcos/etc/mesos-master with mode 600
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] Unlocked fd 8
Sep 22 20:05:35 <agent_id> bootstrap[104823]: [INFO] Closing /run/dcos/etc with fd 8
Sep 22 20:05:36 <agent_id> bootstrap[104823]: [INFO] ensure_zk_path(/dcos/agent/secrets/services, [ACL(perms=1, acl_list=['READ'], id=Id(scheme='digest', id='dcos-agent:ZT4/4Ug5cTT0jcU4PP/2D9W6zOs='))])
Sep 22 20:05:36 <agent_id> bootstrap[104823]: [INFO] Reaching consensus about znode /dcos/agent/secrets/services/dcos_navstar_agent
Sep 22 20:05:36 <agent_id> bootstrap[104823]: [INFO] Consensus znode /dcos/agent/secrets/services/dcos_navstar_agent already exists
Sep 22 20:05:36 <agent_id> bootstrap[104823]: [INFO] ensure_zk_path(/dcos/agent/secrets/services, [ACL(perms=1, acl_list=['READ'], id=Id(scheme='digest', id='dcos-agent:ZT4/4Ug5cTT0jcU4PP/2D9W6zOs='))])
Sep 22 20:05:36 <agent_id> bootstrap[104823]: [INFO] Reaching consensus about znode /dcos/agent/secrets/services/dcos_spartan_agent
Sep 22 20:05:36 <agent_id> bootstrap[104823]: [INFO] Consensus znode /dcos/agent/secrets/services/dcos_spartan_agent already exists
Sep 22 20:05:36 <agent_id> bootstrap[104823]: [INFO] ensure_zk_path(/dcos/agent/secrets/services, [ACL(perms=1, acl_list=['READ'], id=Id(scheme='digest', id='dcos-agent:ZT4/4Ug5cTT0jcU4PP/2D9W6zOs='))])
Sep 22 20:05:36 <agent_id> bootstrap[104823]: [INFO] Reaching consensus about znode /dcos/agent/secrets/services/dcos_mesos_agent_public
Sep 22 20:05:36 <agent_id> bootstrap[104823]: [INFO] Consensus znode /dcos/agent/secrets/services/dcos_mesos_agent_public already exists
Sep 22 20:05:36 <agent_id> bootstrap[104823]: [INFO] ensure_zk_path(/dcos/agent/secrets/services, [ACL(perms=1, acl_list=['READ'], id=Id(scheme='digest', id='dcos-agent:ZT4/4Ug5cTT0jcU4PP/2D9W6zOs='))])
Sep 22 20:05:36 <agent_id> bootstrap[104823]: [INFO] Reaching consensus about znode /dcos/agent/secrets/services/dcos_minuteman_agent
Sep 22 20:05:36 <agent_id> bootstrap[104823]: [INFO] Consensus znode /dcos/agent/secrets/services/dcos_minuteman_agent already exists
Sep 22 20:05:36 <agent_id> bootstrap[104823]: [INFO] ensure_zk_path(/dcos/agent/secrets/services, [ACL(perms=1, acl_list=['READ'], id=Id(scheme='digest', id='dcos-agent:ZT4/4Ug5cTT0jcU4PP/2D9W6zOs='))])
Sep 22 20:05:36 <agent_id> bootstrap[104823]: [INFO] Reaching consensus about znode /dcos/agent/secrets/services/dcos_mesos_agent
Sep 22 20:05:36 <agent_id> bootstrap[104823]: [INFO] Consensus znode /dcos/agent/secrets/services/dcos_mesos_agent already exists
Sep 22 20:05:36 <agent_id> bootstrap[104823]: [INFO] ensure_zk_path(/dcos/agent/secrets/services, [ACL(perms=1, acl_list=['READ'], id=Id(scheme='digest', id='dcos-agent:ZT4/4Ug5cTT0jcU4PP/2D9W6zOs='))])
Sep 22 20:05:36 <agent_id> bootstrap[104823]: [INFO] Reaching consensus about znode /dcos/agent/secrets/services/dcos_agent
Sep 22 20:05:36 <agent_id> bootstrap[104823]: [INFO] Consensus znode /dcos/agent/secrets/services/dcos_agent already exists
Sep 22 20:05:36 <agent_id> bootstrap[104823]: [INFO] ensure_zk_path(/dcos/agent/secrets/services, [ACL(perms=1, acl_list=['READ'], id=Id(scheme='digest', id='dcos-agent:ZT4/4Ug5cTT0jcU4PP/2D9W6zOs='))])
Sep 22 20:05:36 <agent_id> bootstrap[104823]: [INFO] Reaching consensus about znode /dcos/agent/secrets/services/dcos_adminrouter_agent
Sep 22 20:05:36 <agent_id> bootstrap[104823]: [INFO] Consensus znode /dcos/agent/secrets/services/dcos_adminrouter_agent already exists
Sep 22 20:05:36 <agent_id> bootstrap[104823]: [INFO] ensure_zk_path(/dcos/agent/secrets/services, [ACL(perms=1, acl_list=['READ'], id=Id(scheme='digest', id='dcos-agent:ZT4/4Ug5cTT0jcU4PP/2D9W6zOs='))])
Sep 22 20:05:36 <agent_id> bootstrap[104823]: [INFO] Reaching consensus about znode /dcos/agent/secrets/services/dcos_3dt_agent
Sep 22 20:05:36 <agent_id> bootstrap[104823]: [INFO] Consensus znode /dcos/agent/secrets/services/dcos_3dt_agent already exists
Sep 22 20:05:36 <agent_id> bootstrap[104823]: [INFO] Creating service account dcos_agent
Sep 22 20:05:36 <agent_id> bootstrap[104823]: [INFO] Starting new HTTP connection (1): 127.0.0.1
Sep 22 20:05:37 <agent_id> bootstrap[104823]: [INFO] status_code=409
Sep 22 20:05:37 <agent_id> bootstrap[104823]: [INFO] Service account dcos_agent already exists
Sep 22 20:05:37 <agent_id> bootstrap[104823]: [INFO] Starting new HTTP connection (1): 127.0.0.1
Sep 22 20:05:37 <agent_id> bootstrap[104823]: [INFO] Creating service account dcos_mesos_agent
Sep 22 20:05:37 <agent_id> bootstrap[104823]: [INFO] Starting new HTTP connection (1): 127.0.0.1
Sep 22 20:05:37 <agent_id> bootstrap[104823]: [INFO] status_code=409
Sep 22 20:05:37 <agent_id> bootstrap[104823]: [INFO] Service account dcos_mesos_agent already exists
Sep 22 20:05:37 <agent_id> bootstrap[104823]: [INFO] Starting new HTTP connection (1): 127.0.0.1
Sep 22 20:05:37 <agent_id> bootstrap[104823]: [INFO] Creating service account dcos_mesos_agent_public
Sep 22 20:05:37 <agent_id> bootstrap[104823]: [INFO] Starting new HTTP connection (1): 127.0.0.1
Sep 22 20:05:37 <agent_id> bootstrap[104823]: [INFO] status_code=409
Sep 22 20:05:37 <agent_id> bootstrap[104823]: [INFO] Service account dcos_mesos_agent_public already exists
Sep 22 20:05:37 <agent_id> bootstrap[104823]: [INFO] Starting new HTTP connection (1): 127.0.0.1
Sep 22 20:05:37 <agent_id> bootstrap[104823]: [INFO] Creating service account dcos_adminrouter_agent
Sep 22 20:05:37 <agent_id> bootstrap[104823]: [INFO] Starting new HTTP connection (1): 127.0.0.1
Sep 22 20:05:37 <agent_id> bootstrap[104823]: [INFO] status_code=409
Sep 22 20:05:37 <agent_id> bootstrap[104823]: [INFO] Service account dcos_adminrouter_agent already exists
Sep 22 20:05:37 <agent_id> bootstrap[104823]: [INFO] Starting new HTTP connection (1): 127.0.0.1
Sep 22 20:05:37 <agent_id> bootstrap[104823]: [INFO] Creating service account dcos_3dt_agent
Sep 22 20:05:37 <agent_id> bootstrap[104823]: [INFO] Starting new HTTP connection (1): 127.0.0.1
Sep 22 20:05:37 <agent_id> bootstrap[104823]: [INFO] status_code=409
Sep 22 20:05:37 <agent_id> bootstrap[104823]: [INFO] Service account dcos_3dt_agent already exists
Sep 22 20:05:37 <agent_id> bootstrap[104823]: [INFO] Starting new HTTP connection (1): 127.0.0.1
Sep 22 20:05:37 <agent_id> bootstrap[104823]: [INFO] Creating service account dcos_minuteman_agent
Sep 22 20:05:37 <agent_id> bootstrap[104823]: [INFO] Starting new HTTP connection (1): 127.0.0.1
Sep 22 20:05:37 <agent_id> bootstrap[104823]: [INFO] status_code=409
Sep 22 20:05:37 <agent_id> bootstrap[104823]: [INFO] Service account dcos_minuteman_agent already exists
Sep 22 20:05:37 <agent_id> bootstrap[104823]: [INFO] Starting new HTTP connection (1): 127.0.0.1
Sep 22 20:05:37 <agent_id> bootstrap[104823]: [INFO] Creating service account dcos_navstar_agent
Sep 22 20:05:37 <agent_id> bootstrap[104823]: [INFO] Starting new HTTP connection (1): 127.0.0.1
Sep 22 20:05:37 <agent_id> bootstrap[104823]: [INFO] status_code=409
Sep 22 20:05:37 <agent_id> bootstrap[104823]: [INFO] Service account dcos_navstar_agent already exists
Sep 22 20:05:37 <agent_id> bootstrap[104823]: [INFO] Starting new HTTP connection (1): 127.0.0.1
Sep 22 20:05:37 <agent_id> bootstrap[104823]: [INFO] Creating service account dcos_spartan_agent
Sep 22 20:05:37 <agent_id> bootstrap[104823]: [INFO] Starting new HTTP connection (1): 127.0.0.1
Sep 22 20:05:37 <agent_id> bootstrap[104823]: [INFO] status_code=409
Sep 22 20:05:37 <agent_id> bootstrap[104823]: [INFO] Service account dcos_spartan_agent already exists
Sep 22 20:05:37 <agent_id> bootstrap[104823]: [INFO] Starting new HTTP connection (1): 127.0.0.1
Sep 22 20:05:37 <agent_id> systemd[1]: Started Mesos Master: DC/OS Mesos Master Service.
Sep 22 20:05:37 <agent_id> mesos-master[104837]: WARNING: Logging before InitGoogleLogging() is written to STDERR
Sep 22 20:05:37 <agent_id> mesos-master[104837]: I0922 20:05:37.972128 104837 main.cpp:263] Build: 2016-09-13 x.x2:04 by
Sep 22 20:05:37 <agent_id> mesos-master[104837]: I0922 20:05:37.972210 104837 main.cpp:264] Version: 1.0.1
Sep 22 20:05:37 <agent_id> mesos-master[104837]: I0922 20:05:37.972216 104837 main.cpp:271] Git SHA: a22b5dd552168a8dd925682f3b349059e10fc3d3
Sep 22 20:05:37 <agent_id> mesos-master[104837]: W0922 20:05:37.973675 104837 openssl.cpp:399] Failed SSL connections will be downgraded to a non-SSL socket
Sep 22 20:05:37 <agent_id> mesos-master[104837]: W0922 20:05:37.973877 104837 process.cpp:880] Failed SSL connections will be downgraded to a non-SSL socket
Sep 22 20:05:37 <agent_id> mesos-master[104837]: I0922 20:05:37.975872 104837 logging.cpp:194] INFO level logging started!
Sep 22 20:05:37 <agent_id> mesos-master[104837]: W0922 20:05:37.976052 104837 main.cpp:290] Loaded deprecated flag 'max_slave_ping_timeouts'
Sep 22 20:05:37 <agent_id> mesos-master[104837]: W0922 20:05:37.976161 104837 main.cpp:290] Loaded deprecated flag 'slave_removal_rate_limit'
Sep 22 20:05:37 <agent_id> mesos-master[104837]: I0922 20:05:37.992913 104837 manager.cpp:1422] Overlay master parameter 'master_config=/opt/mesosphere/etc/overlay/config/master.json'
Sep 22 20:05:37 <agent_id> mesos-master[104837]: I0922 20:05:37.993281 104837 manager.cpp:606] Configuring overlay network:dcos
Sep 22 20:05:37 <agent_id> mesos-master[104837]: I0922 20:05:37.993336 104837 manager.cpp:650] Initializing the replicated log.
Sep 22 20:05:37 <agent_id> mesos-master[104837]: I0922 20:05:37.993355 104837 manager.cpp:686] Using replicated log with zookeeper URL zk://dcos_mesos_master:6EIjIPV6aX3U8M8kTpSkqXw4e5bZBGZizVV02rdBQdjOv9TzvH4BimMFmmVLe8Kv@127.0.0.1:2181/mesos with a quorum of 2 for leader election.
Sep 22 20:05:38 <agent_id> mesos-master[104837]: I0922 20:05:38.094033 104837 leveldb.cpp:174] Opened db in 100.622609ms
Sep 22 20:05:38 <agent_id> mesos-master[104837]: I0922 20:05:38.x.x81 104837 leveldb.cpp:181] Compacted db in 14.315233ms
Sep 22 20:05:38 <agent_id> mesos-master[104837]: I0922 20:05:38.x.x58 104837 leveldb.cpp:196] Created db iterator in 6554ns
Sep 22 20:05:38 <agent_id> mesos-master[104837]: I0922 20:05:38.x.x67 104837 leveldb.cpp:202] Seeked to beginning of db in 672ns
Sep 22 20:05:38 <agent_id> mesos-master[104837]: I0922 20:05:38.x.x72 104837 leveldb.cpp:271] Iterated through 0 keys in the db in 288ns
Sep 22 20:05:38 <agent_id> mesos-master[104837]: I0922 20:05:38.x.x97 104837 replica.cpp:779] Replica recovered with log positions 0 -> 0 with 1 holes and 0 unlearned
Sep 22 20:05:38 <agent_id> mesos-master[104837]: 2016-09-22 20:05:38,108:104837(0x7f7a49d4d700):ZOO_INFO@log_env@726: Client environment:zookeeper.version=zookeeper C client 3.4.8
Sep 22 20:05:38 <agent_id> mesos-master[104837]: 2016-09-22 20:05:38,108:104837(0x7f7a49d4d700):ZOO_INFO@log_env@730: Client environment:host.name=<agent_id>
Sep 22 20:05:38 <agent_id> mesos-master[104837]: 2016-09-22 20:05:38,108:104837(0x7f7a49d4d700):ZOO_INFO@log_env@737: Client environment:os.name=Linux
Sep 22 20:05:38 <agent_id> mesos-master[104837]: 2016-09-22 20:05:38,108:104837(0x7f7a49d4d700):ZOO_INFO@log_env@738: Client environment:os.arch=4.6.3-coreos
Sep 22 20:05:38 <agent_id> mesos-master[104837]: 2016-09-22 20:05:38,108:104837(0x7f7a49d4d700):ZOO_INFO@log_env@739: Client environment:os.version=#2 SMP Tue Aug 23 02:14:16 UTC 2016
Sep 22 20:05:38 <agent_id> mesos-master[104837]: 2016-09-22 20:05:38,108:104837(0x7f7a4ed57700):ZOO_INFO@log_env@726: Client environment:zookeeper.version=zookeeper C client 3.4.8
Sep 22 20:05:38 <agent_id> mesos-master[104837]: 2016-09-22 20:05:38,108:104837(0x7f7a4ed57700):ZOO_INFO@log_env@730: Client environment:host.name=<agent_id>
Sep 22 20:05:38 <agent_id> mesos-master[104837]: 2016-09-22 20:05:38,108:104837(0x7f7a4ed57700):ZOO_INFO@log_env@737: Client environment:os.name=Linux
Sep 22 20:05:38 <agent_id> mesos-master[104837]: 2016-09-22 20:05:38,108:104837(0x7f7a4ed57700):ZOO_INFO@log_env@738: Client environment:os.arch=4.6.3-coreos
Sep 22 20:05:38 <agent_id> mesos-master[104837]: 2016-09-22 20:05:38,108:104837(0x7f7a4ed57700):ZOO_INFO@log_env@739: Client environment:os.version=#2 SMP Tue Aug 23 02:14:16 UTC 2016
Sep 22 20:05:38 <agent_id> mesos-master[104837]: 2016-09-22 20:05:38,109:104837(0x7f7a4ed57700):ZOO_INFO@log_env@747: Client environment:user.name=(null)
Sep 22 20:05:38 <agent_id> mesos-master[104837]: 2016-09-22 20:05:38,109:104837(0x7f7a49d4d700):ZOO_INFO@log_env@747: Client environment:user.name=(null)
Sep 22 20:05:38 <agent_id> mesos-master[104837]: 2016-09-22 20:05:38,109:104837(0x7f7a4ed57700):ZOO_INFO@log_env@755: Client environment:user.home=/root
Sep 22 20:05:38 <agent_id> mesos-master[104837]: 2016-09-22 20:05:38,109:104837(0x7f7a4ed57700):ZOO_INFO@log_env@767: Client environment:user.dir=/
Sep 22 20:05:38 <agent_id> mesos-master[104837]: 2016-09-22 20:05:38,109:104837(0x7f7a4ed57700):ZOO_INFO@zookeeper_init@800: Initiating client connection, host=127.0.0.1:2181 sessionTimeout=10000 watcher=0x7f7a5e765d00 sessionId=0 sessionPasswd=<null> context=0x7f7a38000ae0 flags=0
Sep 22 20:05:38 <agent_id> mesos-master[104837]: I0922 20:05:38.x.x31 104887 log.cpp:107] Attempting to join replica to ZooKeeper group
Sep 22 20:05:38 <agent_id> mesos-master[104837]: 2016-09-22 20:05:38,109:104837(0x7f7a49d4d700):ZOO_INFO@log_env@755: Client environment:user.home=/root
Sep 22 20:05:38 <agent_id> mesos-master[104837]: I0922 20:05:38.x.x78 104837 manager.cpp:871] Overlay agent parameter 'agent_config=/opt/mesosphere/etc/overlay/config/agent-master.json'
Sep 22 20:05:38 <agent_id> mesos-master[104837]: 2016-09-22 20:05:38,109:104837(0x7f7a49d4d700):ZOO_INFO@log_env@767: Client environment:user.dir=/
Sep 22 20:05:38 <agent_id> mesos-master[104837]: 2016-09-22 20:05:38,109:104837(0x7f7a49d4d700):ZOO_INFO@zookeeper_init@800: Initiating client connection, host=127.0.0.1:2181 sessionTimeout=10000 watcher=0x7f7a5e765d00 sessionId=0 sessionPasswd=<null> context=0x7f79f8000930 flags=0
Sep 22 20:05:38 <agent_id> mesos-master[104837]: I0922 20:05:38.x.x73 104876 manager.cpp:759] Adding route for 'overlay-master/state'
Sep 22 20:05:38 <agent_id> mesos-master[104837]: I0922 20:05:38.110025 104870 recover.cpp:451] Starting replica recovery
Sep 22 20:05:38 <agent_id> mesos-master[104837]: 2016-09-22 20:05:38,110:104837(0x7f7a47d49700):ZOO_INFO@log_env@726: Client environment:zookeeper.version=zookeeper C client 3.4.8
Sep 22 20:05:38 <agent_id> mesos-master[104837]: 2016-09-22 20:05:38,110:104837(0x7f7a47d49700):ZOO_INFO@log_env@730: Client environment:host.name=<agent_id>
Sep 22 20:05:38 <agent_id> mesos-master[104837]: 2016-09-22 20:05:38,110:104837(0x7f7a47d49700):ZOO_INFO@log_env@737: Client environment:os.name=Linux
Sep 22 20:05:38 <agent_id> mesos-master[104837]: 2016-09-22 20:05:38,110:104837(0x7f7a47d49700):ZOO_INFO@log_env@738: Client environment:os.arch=4.6.3-coreos
Sep 22 20:05:38 <agent_id> mesos-master[104837]: 2016-09-22 20:05:38,110:104837(0x7f7a47d49700):ZOO_INFO@log_env@739: Client environment:os.version=#2 SMP Tue Aug 23 02:14:16 UTC 2016
Sep 22 20:05:38 <agent_id> mesos-master[104837]: I0922 20:05:38.110203 104870 recover.cpp:477] Replica is in EMPTY status
Sep 22 20:05:38 <agent_id> mesos-master[104837]: 2016-09-22 20:05:38,110:104837(0x7f7a265f2700):ZOO_INFO@check_events@1728: initiated connection to server [127.0.0.1:2181]
Sep 22 20:05:38 <agent_id> mesos-master[104837]: 2016-09-22 20:05:38,110:104837(0x7f7a275f4700):ZOO_INFO@check_events@1728: initiated connection to server [127.0.0.1:2181]
Sep 22 20:05:38 <agent_id> mesos-master[104837]: 2016-09-22 20:05:38,110:104837(0x7f7a47d49700):ZOO_INFO@log_env@747: Client environment:user.name=(null)
Sep 22 20:05:38 <agent_id> mesos-master[104837]: 2016-09-22 20:05:38,111:104837(0x7f7a47d49700):ZOO_INFO@log_env@755: Client environment:user.home=/root
Sep 22 20:05:38 <agent_id> mesos-master[104837]: 2016-09-22 20:05:38,111:104837(0x7f7a47d49700):ZOO_INFO@log_env@767: Client environment:user.dir=/
Sep 22 20:05:38 <agent_id> mesos-master[104837]: 2016-09-22 20:05:38,111:104837(0x7f7a47d49700):ZOO_INFO@zookeeper_init@800: Initiating client connection, host=127.0.0.1:2181 sessionTimeout=10000 watcher=0x7f7a5e765d00 sessionId=0 sessionPasswd=<null> context=0x7f79e8001f30 flags=0
Sep 22 20:05:38 <agent_id> mesos-master[104837]: 2016-09-22 20:05:38,111:104837(0x7f7a255f0700):ZOO_INFO@check_events@1728: initiated connection to server [127.0.0.1:2181]
Sep 22 20:05:38 <agent_id> mesos-master[104837]: 2016-09-22 20:05:38,123:104837(0x7f7a265f2700):ZOO_INFO@check_events@1775: session establishment complete on server [127.0.0.1:2181], sessionId=0x3574e2d05adb704, negotiated timeout=10000
Sep 22 20:05:38 <agent_id> mesos-master[104837]: I0922 20:05:38.124027 104867 group.cpp:349] Group process (group(1)@x.x.9.1:5050) connected to ZooKeeper
Sep 22 20:05:38 <agent_id> mesos-master[104837]: I0922 20:05:38.124076 104867 group.cpp:837] Syncing group operations: queue size (joins, cancels, datas) = (0, 0, 0)
Sep 22 20:05:38 <agent_id> mesos-master[104837]: I0922 20:05:38.124088 104867 group.cpp:403] Authenticating with ZooKeeper using digest
Sep 22 20:05:38 <agent_id> mesos-master[104837]: 2016-09-22 20:05:38,129:104837(0x7f7a275f4700):ZOO_INFO@check_events@1775: session establishment complete on server [127.0.0.1:2181], sessionId=0x3574e2d05adb705, negotiated timeout=10000
Sep 22 20:05:38 <agent_id> mesos-master[104837]: I0922 20:05:38.129184 104871 group.cpp:349] Group process (group(2)@x.x.9.1:5050) connected to ZooKeeper
Sep 22 20:05:38 <agent_id> mesos-master[104837]: I0922 20:05:38.129216 104871 group.cpp:837] Syncing group operations: queue size (joins, cancels, datas) = (1, 0, 0)
Sep 22 20:05:38 <agent_id> mesos-master[104837]: I0922 20:05:38.129225 104871 group.cpp:403] Authenticating with ZooKeeper using digest
Sep 22 20:05:38 <agent_id> mesos-master[104837]: 2016-09-22 20:05:38,129:104837(0x7f7a255f0700):ZOO_INFO@check_events@1775: session establishment complete on server [127.0.0.1:2181], sessionId=0x3574e2d05adb706, negotiated timeout=10000
Sep 22 20:05:38 <agent_id> mesos-master[104837]: I0922 20:05:38.129473 104865 group.cpp:349] Group process (group(3)@x.x.9.1:5050) connected to ZooKeeper
Sep 22 20:05:38 <agent_id> mesos-master[104837]: I0922 20:05:38.129508 104865 group.cpp:837] Syncing group operations: queue size (joins, cancels, datas) = (0, 0, 0)
Sep 22 20:05:38 <agent_id> mesos-master[104837]: I0922 20:05:38.129521 104865 group.cpp:403] Authenticating with ZooKeeper using digest
Sep 22 20:05:41 <agent_id> mesos-master[104837]: 2016-09-22 20:05:41,446:104837(0x7f7a265f2700):ZOO_INFO@auth_completion_func@1300: Authentication scheme digest succeeded
Sep 22 20:05:41 <agent_id> mesos-master[104837]: 2016-09-22 20:05:41,446:104837(0x7f7a275f4700):ZOO_INFO@auth_completion_func@1300: Authentication scheme digest succeeded
Sep 22 20:05:41 <agent_id> mesos-master[104837]: I0922 20:05:41.446926 104867 group.cpp:427] Trying to create path '/mesos/overlay_log_replicas' in ZooKeeper
Sep 22 20:05:41 <agent_id> mesos-master[104837]: I0922 20:05:41.447043 104871 group.cpp:427] Trying to create path '/mesos/overlay_log_replicas' in ZooKeeper
Sep 22 20:05:41 <agent_id> mesos-master[104837]: 2016-09-22 20:05:41,449:104837(0x7f7a255f0700):ZOO_INFO@auth_completion_func@1300: Authentication scheme digest succeeded
Sep 22 20:05:41 <agent_id> mesos-master[104837]: I0922 20:05:41.449318 104865 group.cpp:427] Trying to create path '/mesos' in ZooKeeper
Sep 22 20:05:41 <agent_id> mesos-master[104837]: I0922 20:05:41.454278 104860 network.hpp:430] ZooKeeper group memberships changed
Sep 22 20:05:41 <agent_id> mesos-master[104837]: I0922 20:05:41.454349 104854 group.cpp:706] Trying to get '/mesos/overlay_log_replicas/0000003293' in ZooKeeper
Sep 22 20:05:41 <agent_id> mesos-master[104837]: I0922 20:05:41.454555 104854 group.cpp:706] Trying to get '/mesos/overlay_log_replicas/0000003294' in ZooKeeper
Sep 22 20:05:41 <agent_id> mesos-master[104837]: I0922 20:05:41.454640 104875 detector.cpp:152] Detected a new leader: (id='2763')
Sep 22 20:05:41 <agent_id> mesos-master[104837]: I0922 20:05:41.463635 104858 group.cpp:706] Trying to get '/mesos/json.info_0000002763' in ZooKeeper
Sep 22 20:05:41 <agent_id> mesos-master[104837]: I0922 20:05:41.464010 104854 group.cpp:706] Trying to get '/mesos/overlay_log_replicas/0000003295' in ZooKeeper
Sep 22 20:05:41 <agent_id> mesos-master[104837]: I0922 20:05:41.464095 104871 zookeeper.cpp:259] A new leading master (UPID=master@x.x.11.1:5050) is detected
Sep 22 20:05:41 <agent_id> mesos-master[104837]: I0922 20:05:41.464279 104868 network.hpp:478] ZooKeeper group PIDs: { log-replica(1)@x.x.11.1:5050, log-replica(1)@x.x.9.1:5050, log-replica(1)@x.x.11.1:5050 }
Sep 22 20:05:46 <agent_id> mesos-master[104837]: I0922 20:05:46.001029 104851 manager.cpp:180] Initializing overlay agent manager
Sep 22 20:05:46 <agent_id> mesos-master[104837]: I0922 20:05:46.001037 104837 manager.cpp:850] Overlay agent is ready
Sep 22 20:05:46 <agent_id> mesos-master[104837]: I0922 20:05:46.001206 104837 main.cpp:370] Using 'HierarchicalDRF' allocator
Sep 22 20:05:46 <agent_id> mesos-master[104837]: I0922 20:05:46.001278 104847 manager.cpp:397] Detected new overlay master at overlay-master@x.x.11.1:5050
Sep 22 20:05:46 <agent_id> mesos-master[104837]: I0922 20:05:46.042414 104837 leveldb.cpp:174] Opened db in 41.142916ms
Sep 22 20:05:46 <agent_id> mesos-master[104837]: I0922 20:05:46.054388 104837 leveldb.cpp:181] Compacted db in 11.932746ms
Sep 22 20:05:46 <agent_id> mesos-master[104837]: I0922 20:05:46.054451 104837 leveldb.cpp:196] Created db iterator in 3474ns
Sep 22 20:05:46 <agent_id> mesos-master[104837]: I0922 20:05:46.054461 104837 leveldb.cpp:202] Seeked to beginning of db in 687ns
Sep 22 20:05:46 <agent_id> mesos-master[104837]: I0922 20:05:46.054466 104837 leveldb.cpp:271] Iterated through 0 keys in the db in 251ns
Sep 22 20:05:46 <agent_id> mesos-master[104837]: I0922 20:05:46.054479 104837 replica.cpp:779] Replica recovered with log positions 0 -> 0 with 1 holes and 0 unlearned
Sep 22 20:05:46 <agent_id> mesos-master[104837]: 2016-09-22 20:05:46,054:104837(0x7f7a53d61700):ZOO_INFO@log_env@726: Client environment:zookeeper.version=zookeeper C client 3.4.8
Sep 22 20:05:46 <agent_id> mesos-master[104837]: 2016-09-22 20:05:46,054:104837(0x7f7a53d61700):ZOO_INFO@log_env@730: Client environment:host.name=<agent_id>
Sep 22 20:05:46 <agent_id> mesos-master[104837]: 2016-09-22 20:05:46,054:104837(0x7f7a53d61700):ZOO_INFO@log_env@737: Client environment:os.name=Linux
Sep 22 20:05:46 <agent_id> mesos-master[104837]: 2016-09-22 20:05:46,054:104837(0x7f7a53d61700):ZOO_INFO@log_env@738: Client environment:os.arch=4.6.3-coreos
Sep 22 20:05:46 <agent_id> mesos-master[104837]: I0922 20:05:46.054653 104867 log.cpp:107] Attempting to join replica to ZooKeeper group
Sep 22 20:05:46 <agent_id> mesos-master[104837]: 2016-09-22 20:05:46,054:104837(0x7f7a4253e700):ZOO_INFO@log_env@726: Client environment:zookeeper.version=zookeeper C client 3.4.8
Sep 22 20:05:46 <agent_id> mesos-master[104837]: 2016-09-22 20:05:46,054:104837(0x7f7a53d61700):ZOO_INFO@log_env@739: Client environment:os.version=#2 SMP Tue Aug 23 02:14:16 UTC 2016
Sep 22 20:05:46 <agent_id> mesos-master[104837]: 2016-09-22 20:05:46,054:104837(0x7f7a4253e700):ZOO_INFO@log_env@730: Client environment:host.name=<agent_id>
Sep 22 20:05:46 <agent_id> mesos-master[104837]: 2016-09-22 20:05:46,054:104837(0x7f7a4253e700):ZOO_INFO@log_env@737: Client environment:os.name=Linux
Sep 22 20:05:46 <agent_id> mesos-master[104837]: 2016-09-22 20:05:46,054:104837(0x7f7a4253e700):ZOO_INFO@log_env@738: Client environment:os.arch=4.6.3-coreos
Sep 22 20:05:46 <agent_id> mesos-master[104837]: 2016-09-22 20:05:46,054:104837(0x7f7a4253e700):ZOO_INFO@log_env@739: Client environment:os.version=#2 SMP Tue Aug 23 02:14:16 UTC 2016
Sep 22 20:05:46 <agent_id> mesos-master[104837]: I0922 20:05:46.054760 104868 recover.cpp:451] Starting replica recovery
Sep 22 20:05:46 <agent_id> mesos-master[104837]: I0922 20:05:46.054831 104868 recover.cpp:477] Replica is in EMPTY status
Sep 22 20:05:46 <agent_id> mesos-master[104837]: 2016-09-22 20:05:46,054:104837(0x7f7a53d61700):ZOO_INFO@log_env@747: Client environment:user.name=(null)
Sep 22 20:05:46 <agent_id> mesos-master[104837]: 2016-09-22 20:05:46,054:104837(0x7f7a53d61700):ZOO_INFO@log_env@755: Client environment:user.home=/root
Sep 22 20:05:46 <agent_id> mesos-master[104837]: 2016-09-22 20:05:46,054:104837(0x7f7a53d61700):ZOO_INFO@log_env@767: Client environment:user.dir=/
Sep 22 20:05:46 <agent_id> mesos-master[104837]: 2016-09-22 20:05:46,054:104837(0x7f7a53d61700):ZOO_INFO@zookeeper_init@800: Initiating client connection, host=127.0.0.1:2181 sessionTimeout=10000 watcher=0x7f7a5e765d00 sessionId=0 sessionPasswd=<null> context=0x7f7964000950 flags=0
Sep 22 20:05:46 <agent_id> mesos-master[104837]: 2016-09-22 20:05:46,054:104837(0x7f7a4c552700):ZOO_INFO@log_env@726: Client environment:zookeeper.version=zookeeper C client 3.4.8
Sep 22 20:05:46 <agent_id> mesos-master[104837]: 2016-09-22 20:05:46,054:104837(0x7f7a4c552700):ZOO_INFO@log_env@730: Client environment:host.name=<agent_id>
Sep 22 20:05:46 <agent_id> mesos-master[104837]: 2016-09-22 20:05:46,054:104837(0x7f7a4c552700):ZOO_INFO@log_env@737: Client environment:os.name=Linux
Sep 22 20:05:46 <agent_id> mesos-master[104837]: 2016-09-22 20:05:46,054:104837(0x7f7a4c552700):ZOO_INFO@log_env@738: Client environment:os.arch=4.6.3-coreos
Sep 22 20:05:46 <agent_id> mesos-master[104837]: 2016-09-22 20:05:46,054:104837(0x7f7a4c552700):ZOO_INFO@log_env@739: Client environment:os.version=#2 SMP Tue Aug 23 02:14:16 UTC 2016
Sep 22 20:05:46 <agent_id> mesos-master[104837]: 2016-09-22 20:05:46,055:104837(0x7f7a59d6d700):ZOO_INFO@log_env@726: Client environment:zookeeper.version=zookeeper C client 3.4.8
Sep 22 20:05:46 <agent_id> mesos-master[104837]: 2016-09-22 20:05:46,055:104837(0x7f7a59d6d700):ZOO_INFO@log_env@730: Client environment:host.name=<agent_id>
Sep 22 20:05:46 <agent_id> mesos-master[104837]: 2016-09-22 20:05:46,055:104837(0x7f7a59d6d700):ZOO_INFO@log_env@737: Client environment:os.name=Linux
Sep 22 20:05:46 <agent_id> mesos-master[104837]: 2016-09-22 20:05:46,055:104837(0x7f7a59d6d700):ZOO_INFO@log_env@738: Client environment:os.arch=4.6.3-coreos
Sep 22 20:05:46 <agent_id> mesos-master[104837]: 2016-09-22 20:05:46,055:104837(0x7f7a4c552700):ZOO_INFO@log_env@747: Client environment:user.name=(null)
Sep 22 20:05:46 <agent_id> mesos-master[104837]: 2016-09-22 20:05:46,055:104837(0x7f7a59d6d700):ZOO_INFO@log_env@739: Client environment:os.version=#2 SMP Tue Aug 23 02:14:16 UTC 2016
Sep 22 20:05:46 <agent_id> mesos-master[104837]: I0922 20:05:46.055543 104837 main.cpp:543] Starting Mesos master
Sep 22 20:05:46 <agent_id> mesos-master[104837]: 2016-09-22 20:05:46,055:104837(0x7f7a4253e700):ZOO_INFO@log_env@747: Client environment:user.name=(null)
Sep 22 20:05:46 <agent_id> mesos-master[104837]: 2016-09-22 20:05:46,055:104837(0x7f7a4c552700):ZOO_INFO@log_env@755: Client environment:user.home=/root
Sep 22 20:05:46 <agent_id> mesos-master[104837]: 2016-09-22 20:05:46,055:104837(0x7f7a4c552700):ZOO_INFO@log_env@767: Client environment:user.dir=/
Sep 22 20:05:46 <agent_id> mesos-master[104837]: 2016-09-22 20:05:46,055:104837(0x7f7a4c552700):ZOO_INFO@zookeeper_init@800: Initiating client connection, host=127.0.0.1:2181 sessionTimeout=10000 watcher=0x7f7a5e765d00 sessionId=0 sessionPasswd=<null> context=0x7f7a28000f50 flags=0
Sep 22 20:05:46 <agent_id> mesos-master[104837]: 2016-09-22 20:05:46,055:104837(0x7f7a4253e700):ZOO_INFO@log_env@755: Client environment:user.home=/root
Sep 22 20:05:46 <agent_id> mesos-master[104837]: 2016-09-22 20:05:46,055:104837(0x7f7a0ffff700):ZOO_INFO@check_events@1728: initiated connection to server [127.0.0.1:2181]
Sep 22 20:05:46 <agent_id> mesos-master[104837]: 2016-09-22 20:05:46,055:104837(0x7f7a4253e700):ZOO_INFO@log_env@767: Client environment:user.dir=/
Sep 22 20:05:46 <agent_id> mesos-master[104837]: 2016-09-22 20:05:46,055:104837(0x7f7a4253e700):ZOO_INFO@zookeeper_init@800: Initiating client connection, host=127.0.0.1:2181 sessionTimeout=10000 watcher=0x7f7a5e765d00 sessionId=0 sessionPasswd=<null> context=0x7f7980001030 flags=0
Sep 22 20:05:46 <agent_id> mesos-master[104837]: 2016-09-22 20:05:46,055:104837(0x7f7a59d6d700):ZOO_INFO@log_env@747: Client environment:user.name=(null)
Sep 22 20:05:46 <agent_id> mesos-master[104837]: 2016-09-22 20:05:46,055:104837(0x7f7a59d6d700):ZOO_INFO@log_env@755: Client environment:user.home=/root
Sep 22 20:05:46 <agent_id> mesos-master[104837]: 2016-09-22 20:05:46,055:104837(0x7f7a59d6d700):ZOO_INFO@log_env@767: Client environment:user.dir=/
Sep 22 20:05:46 <agent_id> mesos-master[104837]: 2016-09-22 20:05:46,055:104837(0x7f7a59d6d700):ZOO_INFO@zookeeper_init@800: Initiating client connection, host=127.0.0.1:2181 sessionTimeout=10000 watcher=0x7f7a5e765d00 sessionId=0 sessionPasswd=<null> context=0x7f79e0002170 flags=0
Sep 22 20:05:46 <agent_id> mesos-master[104837]: 2016-09-22 20:05:46,056:104837(0x7f7a0effd700):ZOO_INFO@check_events@1728: initiated connection to server [127.0.0.1:2181]
Sep 22 20:05:46 <agent_id> mesos-master[104837]: 2016-09-22 20:05:46,056:104837(0x7f7a0dffb700):ZOO_INFO@check_events@1728: initiated connection to server [127.0.0.1:2181]
Sep 22 20:05:46 <agent_id> mesos-master[104837]: 2016-09-22 20:05:46,056:104837(0x7f7a0cff9700):ZOO_INFO@check_events@1728: initiated connection to server [127.0.0.1:2181]
Sep 22 20:05:46 <agent_id> mesos-master[104837]: I0922 20:05:46.057487 104869 master.cpp:375] Master 4be01ec7-9b37-43be-ba3e-d8f70db0a3fe (x.x.9.1) started on x.x.9.1:5050
Sep 22 20:05:46 <agent_id> mesos-master[104837]: I0922 20:05:46.057510 104869 master.cpp:377] Flags at startup: --agent_ping_timeout="15secs" --slave_removal_rate_limit="1/20mins" --agent_reregister_timeout="10mins" --allocation_interval="1secs" --allocator="HierarchicalDRF" --authenticate_agents="false" --authenticate_frameworks="false" --authenticate_http_frameworks="false" --authenticate_http_readonly="true" --authenticate_http_readwrite="true" --authenticators="com_mesosphere_dcos_ClassicRPCAuthenticator" --authorizers="local" --cluster="walthamDCOS1.8" --framework_sorter="drf" --help="false" --hostname_lookup="false" --http_authenticators="com_mesosphere_dcos_http_Authenticator" --initialize_driver_logging="true" --ip_discovery_command="/opt/mesosphere/bin/detect_ip" --log_auto_initialize="true" --log_dir="/var/lib/dcos/mesos/log" --logbufsecs="0" --logging_level="INFO" --max_slave_ping_timeouts="20" --max_completed_frameworks="50" --max_completed_tasks_per_framework="1000" --modules_dir="/opt/mesosphere/etc/mesos-master-modules" --offer_timeout="2mins" --port="5050" --quiet="false" --quorum="2" --recovery_agent_removal_limit="100%" --registry="replicated_log" --registry_fetch_timeout="1mins" --registry_store_timeout="1mins" --registry_strict="false" --root_submissions="true" --user_sorter="drf" --version="false" --webui_dir="/opt/mesosphere/packages/mesos--2b6cac8d498de56ee4c2eb9a89900b5ca06e270b/share/mesos/webui" --weights="" --work_dir="/var/lib/dcos/mesos/master" --zk="zk://dcos_mesos_master:6EIjIPV6aX3U8M8kTpSkqXw4e5bZBGZizVV02rdBQdjOv9TzvH4BimMFmmVLe8Kv@127.0.0.1:2181/mesos" --zk_session_timeout="10secs"
Sep 22 20:05:46 <agent_id> mesos-master[104837]: I0922 20:05:46.057643 104869 master.cpp:429] Master allowing unauthenticated frameworks to register
Sep 22 20:05:46 <agent_id> mesos-master[104837]: I0922 20:05:46.057651 104869 master.cpp:443] Master allowing unauthenticated agents to register
Sep 22 20:05:46 <agent_id> mesos-master[104837]: I0922 20:05:46.057657 104869 master.cpp:457] Master allowing HTTP frameworks to register without authentication
Sep 22 20:05:46 <agent_id> mesos-master[104837]: 2016-09-22 20:05:46,060:104837(0x7f7a0ffff700):ZOO_INFO@check_events@1775: session establishment complete on server [127.0.0.1:2181], sessionId=0x3574e2d05adb710, negotiated timeout=10000
Sep 22 20:05:46 <agent_id> mesos-master[104837]: I0922 20:05:46.060330 104852 group.cpp:349] Group process (group(4)@x.x.9.1:5050) connected to ZooKeeper
Sep 22 20:05:46 <agent_id> mesos-master[104837]: I0922 20:05:46.060361 104852 group.cpp:837] Syncing group operations: queue size (joins, cancels, datas) = (0, 0, 0)
Sep 22 20:05:46 <agent_id> mesos-master[104837]: I0922 20:05:46.060374 104852 group.cpp:403] Authenticating with ZooKeeper using digest
Sep 22 20:05:46 <agent_id> mesos-master[104837]: 2016-09-22 20:05:46,065:104837(0x7f7a0effd700):ZOO_INFO@check_events@1775: session establishment complete on server [127.0.0.1:2181], sessionId=0x3574e2d05adb711, negotiated timeout=10000
Sep 22 20:05:46 <agent_id> mesos-master[104837]: I0922 20:05:46.066030 104856 group.cpp:349] Group process (group(6)@x.x.9.1:5050) connected to ZooKeeper
Sep 22 20:05:46 <agent_id> mesos-master[104837]: I0922 20:05:46.066056 104856 group.cpp:837] Syncing group operations: queue size (joins, cancels, datas) = (0, 0, 0)
Sep 22 20:05:46 <agent_id> mesos-master[104837]: I0922 20:05:46.066063 104856 group.cpp:403] Authenticating with ZooKeeper using digest
Sep 22 20:05:46 <agent_id> mesos-master[104837]: 2016-09-22 20:05:46,066:104837(0x7f7a0dffb700):ZOO_INFO@check_events@1775: session establishment complete on server [127.0.0.1:2181], sessionId=0x3574e2d05adb712, negotiated timeout=10000
Sep 22 20:05:46 <agent_id> mesos-master[104837]: 2016-09-22 20:05:46,066:104837(0x7f7a0cff9700):ZOO_INFO@check_events@1775: session establishment complete on server [127.0.0.1:2181], sessionId=0x3574e2d05adb713, negotiated timeout=10000
Sep 22 20:05:46 <agent_id> mesos-master[104837]: I0922 20:05:46.066226 104883 group.cpp:349] Group process (group(5)@x.x.9.1:5050) connected to ZooKeeper
Sep 22 20:05:46 <agent_id> mesos-master[104837]: I0922 20:05:46.066254 104883 group.cpp:837] Syncing group operations: queue size (joins, cancels, datas) = (1, 0, 0)
Sep 22 20:05:46 <agent_id> mesos-master[104837]: I0922 20:05:46.066264 104883 group.cpp:403] Authenticating with ZooKeeper using digest
Sep 22 20:05:46 <agent_id> mesos-master[104837]: I0922 20:05:46.066596 104843 group.cpp:349] Group process (group(7)@x.x.9.1:5050) connected to ZooKeeper
Sep 22 20:05:46 <agent_id> mesos-master[104837]: I0922 20:05:46.066627 104843 group.cpp:837] Syncing group operations: queue size (joins, cancels, datas) = (0, 0, 0)
Sep 22 20:05:46 <agent_id> mesos-master[104837]: I0922 20:05:46.066637 104843 group.cpp:403] Authenticating with ZooKeeper using digest
Sep 22 20:05:49 <agent_id> mesos-master[104837]: 2016-09-22 20:05:49,392:104837(0x7f7a0ffff700):ZOO_INFO@auth_completion_func@1300: Authentication scheme digest succeeded
Sep 22 20:05:49 <agent_id> mesos-master[104837]: I0922 20:05:49.392606 104852 group.cpp:427] Trying to create path '/mesos/log_replicas' in ZooKeeper
Sep 22 20:05:49 <agent_id> mesos-master[104837]: 2016-09-22 20:05:49,392:104837(0x7f7a0effd700):ZOO_INFO@auth_completion_func@1300: Authentication scheme digest succeeded
Sep 22 20:05:49 <agent_id> mesos-master[104837]: I0922 20:05:49.392740 104856 group.cpp:427] Trying to create path '/mesos' in ZooKeeper
Sep 22 20:05:49 <agent_id> mesos-master[104837]: 2016-09-22 20:05:49,393:104837(0x7f7a0dffb700):ZOO_INFO@auth_completion_func@1300: Authentication scheme digest succeeded
Sep 22 20:05:49 <agent_id> mesos-master[104837]: I0922 20:05:49.393430 104883 group.cpp:427] Trying to create path '/mesos/log_replicas' in ZooKeeper
Sep 22 20:05:49 <agent_id> mesos-master[104837]: 2016-09-22 20:05:49,393:104837(0x7f7a0cff9700):ZOO_INFO@auth_completion_func@1300: Authentication scheme digest succeeded
Sep 22 20:05:49 <agent_id> mesos-master[104837]: I0922 20:05:49.393985 104881 network.hpp:430] ZooKeeper group memberships changed
Sep 22 20:05:49 <agent_id> mesos-master[104837]: I0922 20:05:49.394183 104887 group.cpp:706] Trying to get '/mesos/log_replicas/0000003288' in ZooKeeper
Sep 22 20:05:49 <agent_id> mesos-master[104837]: I0922 20:05:49.394270 104843 group.cpp:427] Trying to create path '/mesos' in ZooKeeper
Sep 22 20:05:49 <agent_id> mesos-master[104837]: I0922 20:05:49.403239 104887 group.cpp:706] Trying to get '/mesos/log_replicas/0000003289' in ZooKeeper
Sep 22 20:05:49 <agent_id> mesos-master[104837]: I0922 20:05:49.403499 104859 detector.cpp:152] Detected a new leader: (id='2763')
Sep 22 20:05:49 <agent_id> mesos-master[104837]: I0922 20:05:49.403614 104849 group.cpp:706] Trying to get '/mesos/json.info_0000002763' in ZooKeeper
Sep 22 20:05:49 <agent_id> mesos-master[104837]: I0922 20:05:49.403832 104866 network.hpp:478] ZooKeeper group PIDs: { log-replica(2)@x.x.11.1:5050, log-replica(2)@x.x.11.1:5050 }
Sep 22 20:05:49 <agent_id> mesos-master[104837]: I0922 20:05:49.403944 104842 zookeeper.cpp:259] A new leading master (UPID=master@x.x.11.1:5050) is detected
Sep 22 20:05:49 <agent_id> mesos-master[104837]: I0922 20:05:49.404078 104875 network.hpp:430] ZooKeeper group memberships changed
Sep 22 20:05:49 <agent_id> mesos-master[104837]: I0922 20:05:49.404206 104884 group.cpp:706] Trying to get '/mesos/log_replicas/0000003288' in ZooKeeper
Sep 22 20:05:49 <agent_id> mesos-master[104837]: I0922 20:05:49.404521 104884 group.cpp:706] Trying to get '/mesos/log_replicas/0000003289' in ZooKeeper
Sep 22 20:05:49 <agent_id> mesos-master[104837]: I0922 20:05:49.404736 104884 group.cpp:706] Trying to get '/mesos/log_replicas/0000003290' in ZooKeeper
Sep 22 20:05:49 <agent_id> mesos-master[104837]: I0922 20:05:49.404980 104884 network.hpp:478] ZooKeeper group PIDs: { log-replica(2)@x.x.11.1:5050, log-replica(2)@x.x.9.1:5050, log-replica(2)@x.x.11.1:5050 }
Sep 22 20:05:57 <agent_id> mesos-master[104837]: I0922 20:05:57.483232 104867 replica.cpp:673] Replica in EMPTY status received a broadcasted recover request from (13)@x.x.9.1:5050
Sep 22 20:05:57 <agent_id> mesos-master[104837]: I0922 20:05:57.483387 104850 recover.cpp:197] Received a recover response from a replica in EMPTY status
{code}
> Unable to start Mesos Master
> ----------------------------
>
> Key: MESOS-6233
> URL: https://issues.apache.org/jira/browse/MESOS-6233
> Project: Mesos
> Issue Type: Bug
> Components: general, master
> Affects Versions: 1.0.1
> Reporter: Alex Kaplan
> Priority: Blocker
> Labels: mesosphere
> Fix For: 1.0.2
>
>
> Mesos Version: 1.0.1
> OS: CoreOS 1068
> {code}
> Sep 22 20:05:17 node-44a84215535c mesos-master[104478]: I0922 20:05:17.948004 104495 manager.cpp:795] overlay-master in `RECOVERING` state . Hence, not sending an update to agentoverlay-agent@10.4.4.1:5051
> Sep 22 20:05:17 node-44a84215535c mesos-master[104478]: F0922 20:05:17.948120 104529 process.cpp:2243] Check failed: sockets.count(from_fd) > 0
> Sep 22 20:05:17 node-44a84215535c mesos-master[104478]: *** Check failure stack trace: ***
> Sep 22 20:05:17 node-44a84215535c mesos-master[104478]: @ 0x7fc1908829fd google::LogMessage::Fail()
> Sep 22 20:05:17 node-44a84215535c mesos-master[104478]: @ 0x7fc19088482d google::LogMessage::SendToLog()
> Sep 22 20:05:17 node-44a84215535c mesos-master[104478]: @ 0x7fc1908825ec google::LogMessage::Flush()
> Sep 22 20:05:17 node-44a84215535c mesos-master[104478]: @ 0x7fc190885129 google::LogMessageFatal::~LogMessageFatal()
> Sep 22 20:05:17 node-44a84215535c mesos-master[104478]: @ 0x7fc1908171dd process::SocketManager::swap_implementing_socket()
> Sep 22 20:05:17 node-44a84215535c mesos-master[104478]: @ 0x7fc19081aa90 process::SocketManager::link_connect()
> Sep 22 20:05:17 node-44a84215535c mesos-master[104478]: @ 0x7fc1908227f9 _ZNSt17_Function_handlerIFvRKN7process6FutureI7NothingEEEZNKS3_5onAnyISt5_BindIFSt7_Mem_fnIMNS0_13SocketManagerEFvS5_NS0_7network6SocketERKNS0_4UPIDEEEPSA_St12_PlaceholderILi1EESC_SD_EEvEES5_OT_NS3_6PreferEEUlS5_E_E9_M_invokeERKSt9_Any_dataS5_
> Sep 22 20:05:17 node-44a84215535c mesos-master[104478]: @ 0x41eb26 _ZN7process8internal3runISt8functionIFvRKNS_6FutureI7NothingEEEEJRS5_EEEvRKSt6vectorIT_SaISC_EEDpOT0_
> Sep 22 20:05:17 node-44a84215535c mesos-master[104478]: @ 0x42a36f process::Future<>::fail()
> Sep 22 20:05:17 node-44a84215535c mesos-master[104478]: @ 0x7fc19085283c process::network::LibeventSSLSocketImpl::event_callback()
> Sep 22 20:05:17 node-44a84215535c mesos-master[104478]: @ 0x7fc190852f17 process::network::LibeventSSLSocketImpl::event_callback()
> Sep 22 20:05:17 node-44a84215535c mesos-master[104478]: @ 0x7fc18d616631 bufferevent_run_deferred_callbacks_locked
> Sep 22 20:05:17 node-44a84215535c mesos-master[104478]: @ 0x7fc18d60cc5d event_base_loop
> Sep 22 20:05:17 node-44a84215535c mesos-master[104478]: @ 0x7fc190865a1d process::EventLoop::run()
> Sep 22 20:05:17 node-44a84215535c mesos-master[104478]: @ 0x7fc18eeabd73 (unknown)
> Sep 22 20:05:17 node-44a84215535c mesos-master[104478]: @ 0x7fc18e6a852c (unknown)
> Sep 22 20:05:17 node-44a84215535c mesos-master[104478]: @ 0x7fc18e3e61dd (unknown)
> Sep 22 20:05:18 node-44a84215535c systemd[1]: [0;1;39mdcos-mesos-master.service: Main process exited, code=killed, status=6/ABRT
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)