You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@mesos.apache.org by "kasim (JIRA)" <ji...@apache.org> on 2016/09/19 08:23:20 UTC
[jira] [Updated] (MESOS-6205) mesos-master can not found
mesos-slave, and elect a new leader in a short interval
[ https://issues.apache.org/jira/browse/MESOS-6205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
kasim updated MESOS-6205:
-------------------------
Description:
I follow this [doc][https://open.mesosphere.com/getting-started/install/#verifying-installation] to setup mesos cluster.
There are three vm(ubuntu 12, centos 6.5, centos 7.2).
$ cat /etc/hosts
10.142.55.190 zk1
10.142.55.196 zk2
10.142.55.202 zk3
config in each mathine:
$ cat /etc/mesos/zk
zk://10.142.55.190:2181,10.142.55.196:2181,10.142.55.202:2181/mesos
After I start zookeeper, mesos-master and mesos-slave in three vm, I can view the mesos webui(10.142.55.190:5050). I found agents count is 0.
After a little time, mesos page get error:
Failed to connect to 10.142.55.190:5050!
Retrying in 16 seconds...
(I found that zookeeper would elect a new leader in a short interval)
master info log:
I0919 15:54:59.677438 13281 http.cpp:2022] Redirecting request for /master/state?jsonp=angular.callbacks._1x to the leading master zk3
I0919 15:55:00.098667 13281 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (768)@10.142.55.202:5050
I0919 15:55:00.385279 13281 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (185)@10.142.55.196:5050
I0919 15:55:00.711119 13281 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (771)@10.142.55.202:5050
I0919 15:55:01.347291 13284 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (188)@10.142.55.196:5050
I0919 15:55:01.597682 13284 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (774)@10.142.55.202:5050
I0919 15:55:02.257159 13282 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (191)@10.142.55.196:5050
I0919 15:55:02.370692 13287 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (777)@10.142.55.202:5050
I0919 15:55:03.205920 13285 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (780)@10.142.55.202:5050
I0919 15:55:03.260007 13281 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (194)@10.142.55.196:5050
I0919 15:55:03.929611 13283 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (783)@10.142.55.202:5050
I0919 15:55:04.033308 13287 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (197)@10.142.55.196:5050
I0919 15:55:04.591275 13284 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (200)@10.142.55.196:5050
I0919 15:55:04.608211 13283 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (786)@10.142.55.202:5050
I0919 15:55:05.184682 13280 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (789)@10.142.55.202:5050
I0919 15:55:05.268277 13280 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (203)@10.142.55.196:5050
I0919 15:55:05.775377 13281 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (206)@10.142.55.196:5050
I0919 15:55:05.916445 13285 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (792)@10.142.55.202:5050
I0919 15:55:06.744927 13280 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (209)@10.142.55.196:5050
I0919 15:55:07.378521 13283 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (5)@10.142.55.202:5050
I0919 15:55:07.393311 13285 network.hpp:430] ZooKeeper group memberships changed
I0919 15:55:07.393427 13285 group.cpp:706] Trying to get '/mesos/log_replicas/0000000709' in ZooKeeper
I0919 15:55:07.393985 13285 group.cpp:706] Trying to get '/mesos/log_replicas/0000000711' in ZooKeeper
I0919 15:55:07.394394 13285 group.cpp:706] Trying to get '/mesos/log_replicas/0000000714' in ZooKeeper
I0919 15:55:07.394843 13285 group.cpp:706] Trying to get '/mesos/log_replicas/0000000715' in ZooKeeper
I0919 15:55:07.395418 13285 network.hpp:478] ZooKeeper group PIDs: { log-replica(1)@10.142.55.190:5050, log-replica(1)@10.142.55.196:5050, log-replica(1)@10.142.55.202:5050 }
I0919 15:55:08.178272 13280 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (14)@10.142.55.202:5050
I0919 15:55:09.059562 13282 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (21)@10.142.55.202:5050
I0919 15:55:09.700711 13286 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (24)@10.142.55.202:5050
I0919 15:55:09.742185 13287 http.cpp:381] HTTP GET for /master/state from 10.142.50.94:59987 with User-Agent='Mozilla/5.0 (Windows NT 6.2; WOW64; rv:47.0) Gecko/20100101 Firefox/47.0'
I0919 15:55:09.742359 13287 http.cpp:2022] Redirecting request for /master/state?jsonp=angular.callbacks._1y to the leading master zk3
I0919 15:55:10.660789 13280 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (30)@10.142.55.202:5050
I0919 15:55:11.480326 13281 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (34)@10.142.55.202:5050
I0919 15:55:12.386256 13286 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (37)@10.142.55.202:5050
I0919 15:55:12.975137 13287 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (42)@10.142.55.202:5050
I0919 15:55:13.843091 13285 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (47)@10.142.55.202:5050
I0919 15:55:14.373478 13281 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (51)@10.142.55.202:5050
I0919 15:55:14.937181 13280 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (54)@10.142.55.202:5050
I0919 15:55:15.658219 13283 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (58)@10.142.55.202:5050
I0919 15:55:16.007822 13286 network.hpp:430] ZooKeeper group memberships changed
I0919 15:55:16.007972 13286 group.cpp:706] Trying to get '/mesos/log_replicas/0000000711' in ZooKeeper
I0919 15:55:16.010170 13286 group.cpp:706] Trying to get '/mesos/log_replicas/0000000714' in ZooKeeper
I0919 15:55:16.011462 13284 detector.cpp:152] Detected a new leader: (id='702')
I0919 15:55:16.011556 13284 group.cpp:706] Trying to get '/mesos/json.info_0000000702' in ZooKeeper
I0919 15:55:16.011968 13286 group.cpp:706] Trying to get '/mesos/log_replicas/0000000715' in ZooKeeper
I0919 15:55:16.012526 13286 network.hpp:478] ZooKeeper group PIDs: { log-replica(1)@10.142.55.190:5050, log-replica(1)@10.142.55.196:5050, log-replica(1)@10.142.55.202:5050 }
I0919 15:55:16.013156 13284 zookeeper.cpp:259] A new leading master (UPID=master@10.142.55.190:5050) is detected
I0919 15:55:16.013222 13284 master.cpp:1847] The newly elected leader is master@10.142.55.190:5050 with id 677967bc-f6f0-46b3-a44e-72eed1befd60
I0919 15:55:16.013244 13284 master.cpp:1860] Elected as the leading master!
I0919 15:55:16.013273 13284 master.cpp:1547] Recovering from registrar
I0919 15:55:16.013352 13284 registrar.cpp:332] Recovering registrar
I0919 15:55:16.014081 13280 log.cpp:553] Attempting to start the writer
I0919 15:55:16.014515 13280 replica.cpp:493] Replica received implicit promise request from (211)@10.142.55.190:5050 with proposal 1204590
I0919 15:55:16.018023 13282 consensus.cpp:360] Aborting implicit promise request because 2 ignores received
I0919 15:55:16.018028 13280 leveldb.cpp:304] Persisting metadata (10 bytes) to leveldb took 3.469479ms
I0919 15:55:16.018338 13280 replica.cpp:342] Persisted promised to 1204590
I0919 15:55:16.018508 13282 log.cpp:565] Could not start the writer, but can be retried
I0919 15:55:16.018645 13282 log.cpp:553] Attempting to start the writer
I0919 15:55:16.018899 13282 replica.cpp:493] Replica received implicit promise request from (215)@10.142.55.190:5050 with proposal 1204591
I0919 15:55:16.022183 13287 consensus.cpp:360] Aborting implicit promise request because 2 ignores received
I0919 15:55:16.022367 13280 log.cpp:565] Could not start the writer, but can be retried
I0919 15:55:16.022510 13280 log.cpp:553] Attempting to start the writer
I0919 15:55:16.028880 13282 leveldb.cpp:304] Persisting metadata (10 bytes) to leveldb took 9.870818ms
I0919 15:55:16.029024 13282 replica.cpp:342] Persisted promised to 1204591
I0919 15:55:16.029428 13286 replica.cpp:493] Replica received implicit promise request from (219)@10.142.55.190:5050 with proposal 1204592
I0919 15:55:16.031600 13280 consensus.cpp:360] Aborting implicit promise request because 2 ignores received
I0919 15:55:16.036208 13283 log.cpp:565] Could not start the writer, but can be retried
I0919 15:55:16.036454 13283 log.cpp:553] Attempting to start the writer
I0919 15:55:16.040256 13286 leveldb.cpp:304] Persisting metadata (10 bytes) to leveldb took 10.783237ms
I0919 15:55:16.040339 13286 replica.cpp:342] Persisted promised to 1204592
I0919 15:55:16.040712 13286 replica.cpp:493] Replica received implicit promise request from (222)@10.142.55.190:5050 with proposal 1204593
I0919 15:55:16.042196 13286 leveldb.cpp:304] Persisting metadata (10 bytes) to leveldb took 1.435071ms
I0919 15:55:16.042250 13286 replica.cpp:342] Persisted promised to 1204593
I0919 15:55:16.042981 13286 consensus.cpp:360] Aborting implicit promise request because 2 ignores received
I0919 15:55:16.043099 13286 log.cpp:565] Could not start the writer, but can be retried
I0919 15:55:16.043303 13283 log.cpp:553] Attempting to start the writer
All later logs are looping
I0919 15:55:16.043676 13286 replica.cpp:493] Replica received implicit promise request from (225)@10.142.55.190:5050 with proposal 1204594
I0919 15:55:16.044122 13286 leveldb.cpp:304] Persisting metadata (10 bytes) to leveldb took 404769ns
I0919 15:55:16.044209 13286 replica.cpp:342] Persisted promised to 1204594
I0919 15:55:16.044837 13281 consensus.cpp:360] Aborting implicit promise request because 2 ignores received
I0919 15:55:16.044926 13281 log.cpp:565] Could not start the writer, but can be retried
I0919 15:55:16.045038 13281 log.cpp:553] Attempting to start the writer
slave info log:
Log file created at: 2016/09/19 15:41:16
Running on machine: ubuntu12
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
I0919 15:41:16.346844 12986 logging.cpp:194] INFO level logging started!
I0919 15:41:16.363313 12986 containerizer.cpp:196] Using isolation: posix/cpu,posix/mem,filesystem/posix,network/cni
I0919 15:41:16.370334 12986 main.cpp:434] Starting Mesos agent
I0919 15:41:16.371184 12986 slave.cpp:198] Agent started on 1)@127.0.1.1:5051
I0919 15:41:16.371636 12986 slave.cpp:199] Flags at startup: --appc_simple_discovery_uri_prefix="http://" --appc_store_dir="/tmp/mesos/store/appc" --authenticate_http_readonly="false" --authenticate_http_readwrite="false" --authenticatee="crammd5" --authentication_backoff_factor="1secs" --authorizer="local" --cgroups_cpu_enable_pids_and_tids_count="false" --cgroups_enable_cfs="false" --cgroups_hierarchy="/sys/fs/cgroup" --cgroups_limit_swap="false" --cgroups_root="mesos" --container_disk_watch_interval="15secs" --containerizers="mesos" --default_role="*" --disk_watch_interval="1mins" --docker="docker" --docker_kill_orphans="true" --docker_registry="https://registry-1.docker.io" --docker_remove_delay="6hrs" --docker_socket="/var/run/docker.sock" --docker_stop_timeout="0ns" --docker_store_dir="/tmp/mesos/store/docker" --docker_volume_checkpoint_dir="/var/run/mesos/isolators/docker/volume" --enforce_container_disk_quota="false" --executor_registration_timeout="1mins" --executor_shutdown_grace_period="5secs" --fetcher_cache_dir="/tmp/mesos/fetch" --fetcher_cache_size="2GB" --frameworks_home="" --gc_delay="1weeks" --gc_disk_headroom="0.1" --hadoop_home="" --help="false" --hostname_lookup="true" --http_authenticators="basic" --http_command_executor="false" --image_provisioner_backend="copy" --initialize_driver_logging="true" --isolation="posix/cpu,posix/mem" --launcher="posix" --launcher_dir="/usr/libexec/mesos" --log_dir="/var/log/mesos" --logbufsecs="0" --logging_level="INFO" --master="zk://10.142.55.190:2181,10.142.55.196:2181,10.142.55.202:2181/mesos" --oversubscribed_resources_interval="15secs" --perf_duration="10secs" --perf_interval="1mins" --port="5051" --qos_correction_interval_min="0ns" --quiet="false" --recover="reconnect" --recovery_timeout="15mins" --registration_backoff_factor="1secs" --revocable_cpu_low_priority="true" --sandbox_directory="/mnt/mesos/sandbox" --strict="true" --switch_user="true" --systemd_enable_support="true" --systemd_runtime_directory="/run/systemd/system" --version="false" --work_dir="/var/lib/mesos"
I0919 15:41:16.373072 12986 slave.cpp:519] Agent resources: cpus(*):2; mem(*):2930; disk(*):4469; ports(*):[31000-32000]
I0919 15:41:16.373291 12986 slave.cpp:527] Agent attributes: [ ]
I0919 15:41:16.373347 12986 slave.cpp:532] Agent hostname: ubuntu12
I0919 15:41:16.379895 13005 state.cpp:57] Recovering state from '/var/lib/mesos/meta'
I0919 15:41:16.382519 13005 group.cpp:349] Group process (group(1)@127.0.1.1:5051) connected to ZooKeeper
I0919 15:41:16.382593 13005 group.cpp:837] Syncing group operations: queue size (joins, cancels, datas) = (0, 0, 0)
I0919 15:41:16.382663 13005 group.cpp:427] Trying to create path '/mesos' in ZooKeeper
I0919 15:41:16.382910 13009 status_update_manager.cpp:200] Recovering status update manager
I0919 15:41:16.383419 13009 containerizer.cpp:522] Recovering containerizer
I0919 15:41:16.392206 13004 provisioner.cpp:253] Provisioner recovery complete
I0919 15:41:16.392354 13004 slave.cpp:4782] Finished recovery
I0919 15:41:16.405709 13004 detector.cpp:152] Detected a new leader: (id='678')
I0919 15:41:16.406067 13005 group.cpp:706] Trying to get '/mesos/json.info_0000000678' in ZooKeeper
I0919 15:41:16.407572 13002 zookeeper.cpp:259] A new leading master (UPID=master@10.142.55.190:5050) is detected
I0919 15:41:16.407977 13002 slave.cpp:895] New master detected at master@10.142.55.190:5050
I0919 15:41:16.408043 13002 slave.cpp:916] No credentials provided. Attempting to register without authentication
I0919 15:41:16.408140 13002 slave.cpp:927] Detecting new master
I0919 15:41:16.408223 13005 status_update_manager.cpp:174] Pausing sending status updates
I0919 15:42:08.418956 13006 slave.cpp:3732] master@10.142.55.190:5050 exited
W0919 15:42:08.419035 13006 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
I0919 15:42:16.374977 13007 slave.cpp:4591] Current disk usage 72.41%. Max allowed age: 1.231186482451933days
I0919 15:42:20.007169 13007 detector.cpp:152] Detected a new leader: (id='679')
I0919 15:42:20.007297 13007 group.cpp:706] Trying to get '/mesos/json.info_0000000679' in ZooKeeper
I0919 15:42:20.008503 13007 zookeeper.cpp:259] A new leading master (UPID=master@10.142.55.196:5050) is detected
I0919 15:42:20.008587 13007 slave.cpp:895] New master detected at master@10.142.55.196:5050
I0919 15:42:20.008610 13007 slave.cpp:916] No credentials provided. Attempting to register without authentication
I0919 15:42:20.008703 13007 slave.cpp:927] Detecting new master
I0919 15:42:20.008750 13007 status_update_manager.cpp:174] Pausing sending status updates
I0919 15:43:16.387984 13005 slave.cpp:4591] Current disk usage 72.41%. Max allowed age: 1.231162010606794days
I0919 15:43:20.081272 13005 slave.cpp:3732] master@10.142.55.196:5050 exited
W0919 15:43:20.081374 13005 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
I0919 15:43:26.855154 13005 slave.cpp:3732] master@10.142.55.196:5050 exited
W0919 15:43:26.855315 13005 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
E0919 15:43:26.855159 13010 process.cpp:2105] Failed to shutdown socket with fd 12: Transport endpoint is not connected
I0919 15:43:32.020196 13002 detector.cpp:152] Detected a new leader: (id='682')
I0919 15:43:32.020300 13002 group.cpp:706] Trying to get '/mesos/json.info_0000000682' in ZooKeeper
I0919 15:43:32.022203 13002 zookeeper.cpp:259] A new leading master (UPID=master@10.142.55.202:5050) is detected
I0919 15:43:32.022302 13002 slave.cpp:895] New master detected at master@10.142.55.202:5050
I0919 15:43:32.022328 13002 slave.cpp:916] No credentials provided. Attempting to register without authentication
I0919 15:43:32.022382 13002 slave.cpp:927] Detecting new master
I0919 15:43:32.022423 13002 status_update_manager.cpp:174] Pausing sending status updates
I0919 15:44:16.389369 13003 slave.cpp:4591] Current disk usage 72.41%. Max allowed age: 1.231119184877789days
I0919 15:44:32.535347 13003 slave.cpp:3732] master@10.142.55.202:5050 exited
W0919 15:44:32.535522 13003 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
I0919 15:44:42.005375 13002 detector.cpp:152] Detected a new leader: (id='684')
I0919 15:44:42.005496 13002 group.cpp:706] Trying to get '/mesos/json.info_0000000684' in ZooKeeper
I0919 15:44:42.006367 13002 zookeeper.cpp:259] A new leading master (UPID=master@10.142.55.190:5050) is detected
I0919 15:44:42.006492 13002 slave.cpp:895] New master detected at master@10.142.55.190:5050
I0919 15:44:42.006597 13002 slave.cpp:916] No credentials provided. Attempting to register without authentication
I0919 15:44:42.006675 13002 slave.cpp:927] Detecting new master
I0919 15:44:42.006577 13008 status_update_manager.cpp:174] Pausing sending status updates
I0919 15:45:16.400794 13006 slave.cpp:4591] Current disk usage 72.48%. Max allowed age: 1.226390000804074days
I0919 15:45:42.354790 13005 slave.cpp:3732] master@10.142.55.190:5050 exited
W0919 15:45:42.354857 13005 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
I0919 15:45:54.020563 13002 detector.cpp:152] Detected a new leader: (id='687')
I0919 15:45:54.020756 13002 group.cpp:706] Trying to get '/mesos/json.info_0000000687' in ZooKeeper
I0919 15:45:54.023296 13002 zookeeper.cpp:259] A new leading master (UPID=master@10.142.55.196:5050) is detected
I0919 15:45:54.023455 13002 slave.cpp:895] New master detected at master@10.142.55.196:5050
I0919 15:45:54.023558 13002 slave.cpp:916] No credentials provided. Attempting to register without authentication
I0919 15:45:54.023526 13008 status_update_manager.cpp:174] Pausing sending status updates
I0919 15:45:54.023669 13002 slave.cpp:927] Detecting new master
I0919 15:46:16.402402 13003 slave.cpp:4591] Current disk usage 72.53%. Max allowed age: 1.223205601954942days
I0919 15:46:54.075505 13007 slave.cpp:3732] master@10.142.55.196:5050 exited
W0919 15:46:54.075592 13007 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
E0919 15:46:56.098012 13010 process.cpp:2105] Failed to shutdown socket with fd 14: Transport endpoint is not connected
I0919 15:46:56.098016 13007 slave.cpp:3732] master@10.142.55.196:5050 exited
W0919 15:46:56.098253 13007 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
E0919 15:46:56.462254 13010 process.cpp:2105] Failed to shutdown socket with fd 14: Transport endpoint is not connected
I0919 15:46:56.462260 13005 slave.cpp:3732] master@10.142.55.196:5050 exited
W0919 15:46:56.462540 13005 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
I0919 15:47:02.005637 13009 detector.cpp:152] Detected a new leader: (id='688')
I0919 15:47:02.005765 13009 group.cpp:706] Trying to get '/mesos/json.info_0000000688' in ZooKeeper
I0919 15:47:02.006853 13009 zookeeper.cpp:259] A new leading master (UPID=master@10.142.55.202:5050) is detected
I0919 15:47:02.006959 13009 slave.cpp:895] New master detected at master@10.142.55.202:5050
I0919 15:47:02.006986 13009 slave.cpp:916] No credentials provided. Attempting to register without authentication
I0919 15:47:02.007025 13009 slave.cpp:927] Detecting new master
I0919 15:47:02.007061 13009 status_update_manager.cpp:174] Pausing sending status updates
I0919 15:47:16.406669 13008 slave.cpp:4591] Current disk usage 72.53%. Max allowed age: 1.223184189090440days
I0919 15:48:02.950891 13005 slave.cpp:3732] master@10.142.55.202:5050 exited
W0919 15:48:02.950994 13005 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
I0919 15:48:12.006634 13005 detector.cpp:152] Detected a new leader: (id='690')
I0919 15:48:12.006817 13003 group.cpp:706] Trying to get '/mesos/json.info_0000000690' in ZooKeeper
I0919 15:48:12.007987 13003 zookeeper.cpp:259] A new leading master (UPID=master@10.142.55.190:5050) is detected
I0919 15:48:12.008126 13003 slave.cpp:895] New master detected at master@10.142.55.190:5050
I0919 15:48:12.008210 13003 slave.cpp:916] No credentials provided. Attempting to register without authentication
I0919 15:48:12.008280 13003 slave.cpp:927] Detecting new master
I0919 15:48:12.008191 13008 status_update_manager.cpp:174] Pausing sending status updates
I0919 15:48:16.409266 13003 slave.cpp:4591] Current disk usage 72.54%. Max allowed age: 1.222480623542604days
I0919 15:49:12.379010 13009 slave.cpp:3732] master@10.142.55.190:5050 exited
W0919 15:49:12.379149 13009 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
E0919 15:49:12.379233 13010 process.cpp:2105] Failed to shutdown socket with fd 12: Transport endpoint is not connected
I0919 15:49:16.413767 13007 slave.cpp:4591] Current disk usage 72.64%. Max allowed age: 1.215032005677465days
I0919 15:49:24.016290 13007 detector.cpp:152] Detected a new leader: (id='693')
I0919 15:49:24.016417 13007 group.cpp:706] Trying to get '/mesos/json.info_0000000693' in ZooKeeper
I0919 15:49:24.018273 13007 zookeeper.cpp:259] A new leading master (UPID=master@10.142.55.196:5050) is detected
I0919 15:49:24.018437 13007 slave.cpp:895] New master detected at master@10.142.55.196:5050
I0919 15:49:24.018523 13007 slave.cpp:916] No credentials provided. Attempting to register without authentication
I0919 15:49:24.018604 13007 slave.cpp:927] Detecting new master
I0919 15:49:24.018496 13008 status_update_manager.cpp:174] Pausing sending status updates
I0919 15:50:16.416391 13008 slave.cpp:4591] Current disk usage 72.64%. Max allowed age: 1.215016710774248days
I0919 15:50:24.065268 13003 slave.cpp:3732] master@10.142.55.196:5050 exited
W0919 15:50:24.065342 13003 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
I0919 15:50:24.485752 13004 slave.cpp:3732] master@10.142.55.196:5050 exited
W0919 15:50:24.485839 13004 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
E0919 15:50:24.485977 13010 process.cpp:2105] Failed to shutdown socket with fd 14: Transport endpoint is not connected
I0919 15:50:28.343647 13003 slave.cpp:3732] master@10.142.55.196:5050 exited
W0919 15:50:28.343719 13003 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
E0919 15:50:28.343819 13010 process.cpp:2105] Failed to shutdown socket with fd 14: Transport endpoint is not connected
I0919 15:50:31.545099 13005 slave.cpp:3732] master@10.142.55.196:5050 exited
W0919 15:50:31.545171 13005 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
E0919 15:50:31.545284 13010 process.cpp:2105] Failed to shutdown socket with fd 14: Transport endpoint is not connected
I0919 15:50:32.007096 13008 detector.cpp:152] Detected a new leader: (id='694')
I0919 15:50:32.007195 13008 group.cpp:706] Trying to get '/mesos/json.info_0000000694' in ZooKeeper
I0919 15:50:32.009881 13008 zookeeper.cpp:259] A new leading master (UPID=master@10.142.55.202:5050) is detected
I0919 15:50:32.009970 13008 slave.cpp:895] New master detected at master@10.142.55.202:5050
I0919 15:50:32.009994 13008 slave.cpp:916] No credentials provided. Attempting to register without authentication
I0919 15:50:32.010030 13008 slave.cpp:927] Detecting new master
I0919 15:50:32.010079 13008 status_update_manager.cpp:174] Pausing sending status updates
I0919 15:51:16.417846 13006 slave.cpp:4591] Current disk usage 72.64%. Max allowed age: 1.214964708103322days
I0919 15:51:32.560317 13003 slave.cpp:3732] master@10.142.55.202:5050 exited
W0919 15:51:32.560410 13003 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
I0919 15:51:42.005147 13009 detector.cpp:152] Detected a new leader: (id='696')
I0919 15:51:42.005265 13009 group.cpp:706] Trying to get '/mesos/json.info_0000000696' in ZooKeeper
I0919 15:51:42.006824 13009 zookeeper.cpp:259] A new leading master (UPID=master@10.142.55.190:5050) is detected
I0919 15:51:42.006904 13009 slave.cpp:895] New master detected at master@10.142.55.190:5050
I0919 15:51:42.006928 13009 slave.cpp:916] No credentials provided. Attempting to register without authentication
I0919 15:51:42.006963 13009 slave.cpp:927] Detecting new master
I0919 15:51:42.006999 13009 status_update_manager.cpp:174] Pausing sending status updates
I0919 15:52:16.419373 13003 slave.cpp:4591] Current disk usage 72.71%. Max allowed age: 1.209981628636250days
I0919 15:52:42.336305 13002 slave.cpp:3732] master@10.142.55.190:5050 exited
W0919 15:52:42.336426 13002 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
I0919 15:52:54.005267 13005 detector.cpp:152] Detected a new leader: (id='699')
I0919 15:52:54.005408 13005 group.cpp:706] Trying to get '/mesos/json.info_0000000699' in ZooKeeper
I0919 15:52:54.006206 13005 zookeeper.cpp:259] A new leading master (UPID=master@10.142.55.196:5050) is detected
I0919 15:52:54.006285 13005 slave.cpp:895] New master detected at master@10.142.55.196:5050
I0919 15:52:54.006309 13005 slave.cpp:916] No credentials provided. Attempting to register without authentication
I0919 15:52:54.006398 13005 slave.cpp:927] Detecting new master
I0919 15:52:54.006451 13005 status_update_manager.cpp:174] Pausing sending status updates
I0919 15:53:16.420258 13005 slave.cpp:4591] Current disk usage 72.76%. Max allowed age: 1.206748286096840days
I0919 15:53:54.071012 13005 slave.cpp:3732] master@10.142.55.196:5050 exited
W0919 15:53:54.071143 13005 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
I0919 15:54:01.105780 13002 slave.cpp:3732] master@10.142.55.196:5050 exited
W0919 15:54:01.105854 13002 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
E0919 15:54:01.105970 13010 process.cpp:2105] Failed to shutdown socket with fd 15: Transport endpoint is not connected
I0919 15:54:05.733837 13007 slave.cpp:3732] master@10.142.55.196:5050 exited
W0919 15:54:05.733932 13007 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
E0919 15:54:05.734071 13010 process.cpp:2105] Failed to shutdown socket with fd 15: Transport endpoint is not connected
E0919 15:54:05.818560 13010 process.cpp:2105] Failed to shutdown socket with fd 15: Transport endpoint is not connected
I0919 15:54:05.818583 13003 slave.cpp:3732] master@10.142.55.196:5050 exited
W0919 15:54:05.818758 13003 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
I0919 15:54:06.004385 13009 detector.cpp:152] Detected a new leader: (id='700')
I0919 15:54:06.004494 13009 group.cpp:706] Trying to get '/mesos/json.info_0000000700' in ZooKeeper
I0919 15:54:06.005511 13009 zookeeper.cpp:259] A new leading master (UPID=master@10.142.55.202:5050) is detected
I0919 15:54:06.005586 13009 slave.cpp:895] New master detected at master@10.142.55.202:5050
I0919 15:54:06.005609 13009 slave.cpp:916] No credentials provided. Attempting to register without authentication
I0919 15:54:06.005676 13009 slave.cpp:927] Detecting new master
I0919 15:54:06.005720 13009 status_update_manager.cpp:174] Pausing sending status updates
I0919 15:54:16.423193 13002 slave.cpp:4591] Current disk usage 72.76%. Max allowed age: 1.206699342406551days
slave warn log
Log file created at: 2016/09/19 15:42:08
Running on machine: ubuntu12
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
W0919 15:42:08.419035 13006 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
W0919 15:43:20.081374 13005 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
W0919 15:43:26.855315 13005 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
E0919 15:43:26.855159 13010 process.cpp:2105] Failed to shutdown socket with fd 12: Transport endpoint is not connected
W0919 15:44:32.535522 13003 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
W0919 15:45:42.354857 13005 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
W0919 15:46:54.075592 13007 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
E0919 15:46:56.098012 13010 process.cpp:2105] Failed to shutdown socket with fd 14: Transport endpoint is not connected
W0919 15:46:56.098253 13007 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
E0919 15:46:56.462254 13010 process.cpp:2105] Failed to shutdown socket with fd 14: Transport endpoint is not connected
W0919 15:46:56.462540 13005 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
W0919 15:48:02.950994 13005 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
W0919 15:49:12.379149 13009 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
E0919 15:49:12.379233 13010 process.cpp:2105] Failed to shutdown socket with fd 12: Transport endpoint is not connected
W0919 15:50:24.065342 13003 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
W0919 15:50:24.485839 13004 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
E0919 15:50:24.485977 13010 process.cpp:2105] Failed to shutdown socket with fd 14: Transport endpoint is not connected
W0919 15:50:28.343719 13003 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
E0919 15:50:28.343819 13010 process.cpp:2105] Failed to shutdown socket with fd 14: Transport endpoint is not connected
W0919 15:50:31.545171 13005 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
E0919 15:50:31.545284 13010 process.cpp:2105] Failed to shutdown socket with fd 14: Transport endpoint is not connected
W0919 15:51:32.560410 13003 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
W0919 15:52:42.336426 13002 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
W0919 15:53:54.071143 13005 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
W0919 15:54:01.105854 13002 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
E0919 15:54:01.105970 13010 process.cpp:2105] Failed to shutdown socket with fd 15: Transport endpoint is not connected
W0919 15:54:05.733932 13007 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
E0919 15:54:05.734071 13010 process.cpp:2105] Failed to shutdown socket with fd 15: Transport endpoint is not connected
E0919 15:54:05.818560 13010 process.cpp:2105] Failed to shutdown socket with fd 15: Transport endpoint is not connected
W0919 15:54:05.818758 13003 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
W0919 15:55:06.821486 13009 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
was:
I follow this [doc][1] to setup mesos cluster.
There are three vm(ubuntu 12, centos 6.5, centos 7.2).
$ cat /etc/hosts
10.142.55.190 zk1
10.142.55.196 zk2
10.142.55.202 zk3
config in each mathine:
$ cat /etc/mesos/zk
zk://10.142.55.190:2181,10.142.55.196:2181,10.142.55.202:2181/mesos
After I start mesos-master in three vm, I can view the mesos webui(10.142.55.190:5050), but after a little time, mesos page get error:
Failed to connect to 10.142.55.190:5050!
Retrying in 16 seconds...
(I found that zookeeper would elect a new leader in a short interval)
master info log:
I0919 15:54:59.677438 13281 http.cpp:2022] Redirecting request for /master/state?jsonp=angular.callbacks._1x to the leading master zk3
I0919 15:55:00.098667 13281 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (768)@10.142.55.202:5050
I0919 15:55:00.385279 13281 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (185)@10.142.55.196:5050
I0919 15:55:00.711119 13281 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (771)@10.142.55.202:5050
I0919 15:55:01.347291 13284 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (188)@10.142.55.196:5050
I0919 15:55:01.597682 13284 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (774)@10.142.55.202:5050
I0919 15:55:02.257159 13282 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (191)@10.142.55.196:5050
I0919 15:55:02.370692 13287 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (777)@10.142.55.202:5050
I0919 15:55:03.205920 13285 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (780)@10.142.55.202:5050
I0919 15:55:03.260007 13281 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (194)@10.142.55.196:5050
I0919 15:55:03.929611 13283 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (783)@10.142.55.202:5050
I0919 15:55:04.033308 13287 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (197)@10.142.55.196:5050
I0919 15:55:04.591275 13284 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (200)@10.142.55.196:5050
I0919 15:55:04.608211 13283 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (786)@10.142.55.202:5050
I0919 15:55:05.184682 13280 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (789)@10.142.55.202:5050
I0919 15:55:05.268277 13280 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (203)@10.142.55.196:5050
I0919 15:55:05.775377 13281 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (206)@10.142.55.196:5050
I0919 15:55:05.916445 13285 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (792)@10.142.55.202:5050
I0919 15:55:06.744927 13280 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (209)@10.142.55.196:5050
I0919 15:55:07.378521 13283 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (5)@10.142.55.202:5050
I0919 15:55:07.393311 13285 network.hpp:430] ZooKeeper group memberships changed
I0919 15:55:07.393427 13285 group.cpp:706] Trying to get '/mesos/log_replicas/0000000709' in ZooKeeper
I0919 15:55:07.393985 13285 group.cpp:706] Trying to get '/mesos/log_replicas/0000000711' in ZooKeeper
I0919 15:55:07.394394 13285 group.cpp:706] Trying to get '/mesos/log_replicas/0000000714' in ZooKeeper
I0919 15:55:07.394843 13285 group.cpp:706] Trying to get '/mesos/log_replicas/0000000715' in ZooKeeper
I0919 15:55:07.395418 13285 network.hpp:478] ZooKeeper group PIDs: { log-replica(1)@10.142.55.190:5050, log-replica(1)@10.142.55.196:5050, log-replica(1)@10.142.55.202:5050 }
I0919 15:55:08.178272 13280 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (14)@10.142.55.202:5050
I0919 15:55:09.059562 13282 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (21)@10.142.55.202:5050
I0919 15:55:09.700711 13286 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (24)@10.142.55.202:5050
I0919 15:55:09.742185 13287 http.cpp:381] HTTP GET for /master/state from 10.142.50.94:59987 with User-Agent='Mozilla/5.0 (Windows NT 6.2; WOW64; rv:47.0) Gecko/20100101 Firefox/47.0'
I0919 15:55:09.742359 13287 http.cpp:2022] Redirecting request for /master/state?jsonp=angular.callbacks._1y to the leading master zk3
I0919 15:55:10.660789 13280 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (30)@10.142.55.202:5050
I0919 15:55:11.480326 13281 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (34)@10.142.55.202:5050
I0919 15:55:12.386256 13286 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (37)@10.142.55.202:5050
I0919 15:55:12.975137 13287 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (42)@10.142.55.202:5050
I0919 15:55:13.843091 13285 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (47)@10.142.55.202:5050
I0919 15:55:14.373478 13281 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (51)@10.142.55.202:5050
I0919 15:55:14.937181 13280 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (54)@10.142.55.202:5050
I0919 15:55:15.658219 13283 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (58)@10.142.55.202:5050
I0919 15:55:16.007822 13286 network.hpp:430] ZooKeeper group memberships changed
I0919 15:55:16.007972 13286 group.cpp:706] Trying to get '/mesos/log_replicas/0000000711' in ZooKeeper
I0919 15:55:16.010170 13286 group.cpp:706] Trying to get '/mesos/log_replicas/0000000714' in ZooKeeper
I0919 15:55:16.011462 13284 detector.cpp:152] Detected a new leader: (id='702')
I0919 15:55:16.011556 13284 group.cpp:706] Trying to get '/mesos/json.info_0000000702' in ZooKeeper
I0919 15:55:16.011968 13286 group.cpp:706] Trying to get '/mesos/log_replicas/0000000715' in ZooKeeper
I0919 15:55:16.012526 13286 network.hpp:478] ZooKeeper group PIDs: { log-replica(1)@10.142.55.190:5050, log-replica(1)@10.142.55.196:5050, log-replica(1)@10.142.55.202:5050 }
I0919 15:55:16.013156 13284 zookeeper.cpp:259] A new leading master (UPID=master@10.142.55.190:5050) is detected
I0919 15:55:16.013222 13284 master.cpp:1847] The newly elected leader is master@10.142.55.190:5050 with id 677967bc-f6f0-46b3-a44e-72eed1befd60
I0919 15:55:16.013244 13284 master.cpp:1860] Elected as the leading master!
I0919 15:55:16.013273 13284 master.cpp:1547] Recovering from registrar
I0919 15:55:16.013352 13284 registrar.cpp:332] Recovering registrar
I0919 15:55:16.014081 13280 log.cpp:553] Attempting to start the writer
I0919 15:55:16.014515 13280 replica.cpp:493] Replica received implicit promise request from (211)@10.142.55.190:5050 with proposal 1204590
I0919 15:55:16.018023 13282 consensus.cpp:360] Aborting implicit promise request because 2 ignores received
I0919 15:55:16.018028 13280 leveldb.cpp:304] Persisting metadata (10 bytes) to leveldb took 3.469479ms
I0919 15:55:16.018338 13280 replica.cpp:342] Persisted promised to 1204590
I0919 15:55:16.018508 13282 log.cpp:565] Could not start the writer, but can be retried
I0919 15:55:16.018645 13282 log.cpp:553] Attempting to start the writer
I0919 15:55:16.018899 13282 replica.cpp:493] Replica received implicit promise request from (215)@10.142.55.190:5050 with proposal 1204591
I0919 15:55:16.022183 13287 consensus.cpp:360] Aborting implicit promise request because 2 ignores received
I0919 15:55:16.022367 13280 log.cpp:565] Could not start the writer, but can be retried
I0919 15:55:16.022510 13280 log.cpp:553] Attempting to start the writer
I0919 15:55:16.028880 13282 leveldb.cpp:304] Persisting metadata (10 bytes) to leveldb took 9.870818ms
I0919 15:55:16.029024 13282 replica.cpp:342] Persisted promised to 1204591
I0919 15:55:16.029428 13286 replica.cpp:493] Replica received implicit promise request from (219)@10.142.55.190:5050 with proposal 1204592
I0919 15:55:16.031600 13280 consensus.cpp:360] Aborting implicit promise request because 2 ignores received
I0919 15:55:16.036208 13283 log.cpp:565] Could not start the writer, but can be retried
I0919 15:55:16.036454 13283 log.cpp:553] Attempting to start the writer
I0919 15:55:16.040256 13286 leveldb.cpp:304] Persisting metadata (10 bytes) to leveldb took 10.783237ms
I0919 15:55:16.040339 13286 replica.cpp:342] Persisted promised to 1204592
I0919 15:55:16.040712 13286 replica.cpp:493] Replica received implicit promise request from (222)@10.142.55.190:5050 with proposal 1204593
I0919 15:55:16.042196 13286 leveldb.cpp:304] Persisting metadata (10 bytes) to leveldb took 1.435071ms
I0919 15:55:16.042250 13286 replica.cpp:342] Persisted promised to 1204593
I0919 15:55:16.042981 13286 consensus.cpp:360] Aborting implicit promise request because 2 ignores received
I0919 15:55:16.043099 13286 log.cpp:565] Could not start the writer, but can be retried
I0919 15:55:16.043303 13283 log.cpp:553] Attempting to start the writer
All later logs are looping
I0919 15:55:16.043676 13286 replica.cpp:493] Replica received implicit promise request from (225)@10.142.55.190:5050 with proposal 1204594
I0919 15:55:16.044122 13286 leveldb.cpp:304] Persisting metadata (10 bytes) to leveldb took 404769ns
I0919 15:55:16.044209 13286 replica.cpp:342] Persisted promised to 1204594
I0919 15:55:16.044837 13281 consensus.cpp:360] Aborting implicit promise request because 2 ignores received
I0919 15:55:16.044926 13281 log.cpp:565] Could not start the writer, but can be retried
I0919 15:55:16.045038 13281 log.cpp:553] Attempting to start the writer
slave info log:
Log file created at: 2016/09/19 15:41:16
Running on machine: ubuntu12
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
I0919 15:41:16.346844 12986 logging.cpp:194] INFO level logging started!
I0919 15:41:16.363313 12986 containerizer.cpp:196] Using isolation: posix/cpu,posix/mem,filesystem/posix,network/cni
I0919 15:41:16.370334 12986 main.cpp:434] Starting Mesos agent
I0919 15:41:16.371184 12986 slave.cpp:198] Agent started on 1)@127.0.1.1:5051
I0919 15:41:16.371636 12986 slave.cpp:199] Flags at startup: --appc_simple_discovery_uri_prefix="http://" --appc_store_dir="/tmp/mesos/store/appc" --authenticate_http_readonly="false" --authenticate_http_readwrite="false" --authenticatee="crammd5" --authentication_backoff_factor="1secs" --authorizer="local" --cgroups_cpu_enable_pids_and_tids_count="false" --cgroups_enable_cfs="false" --cgroups_hierarchy="/sys/fs/cgroup" --cgroups_limit_swap="false" --cgroups_root="mesos" --container_disk_watch_interval="15secs" --containerizers="mesos" --default_role="*" --disk_watch_interval="1mins" --docker="docker" --docker_kill_orphans="true" --docker_registry="https://registry-1.docker.io" --docker_remove_delay="6hrs" --docker_socket="/var/run/docker.sock" --docker_stop_timeout="0ns" --docker_store_dir="/tmp/mesos/store/docker" --docker_volume_checkpoint_dir="/var/run/mesos/isolators/docker/volume" --enforce_container_disk_quota="false" --executor_registration_timeout="1mins" --executor_shutdown_grace_period="5secs" --fetcher_cache_dir="/tmp/mesos/fetch" --fetcher_cache_size="2GB" --frameworks_home="" --gc_delay="1weeks" --gc_disk_headroom="0.1" --hadoop_home="" --help="false" --hostname_lookup="true" --http_authenticators="basic" --http_command_executor="false" --image_provisioner_backend="copy" --initialize_driver_logging="true" --isolation="posix/cpu,posix/mem" --launcher="posix" --launcher_dir="/usr/libexec/mesos" --log_dir="/var/log/mesos" --logbufsecs="0" --logging_level="INFO" --master="zk://10.142.55.190:2181,10.142.55.196:2181,10.142.55.202:2181/mesos" --oversubscribed_resources_interval="15secs" --perf_duration="10secs" --perf_interval="1mins" --port="5051" --qos_correction_interval_min="0ns" --quiet="false" --recover="reconnect" --recovery_timeout="15mins" --registration_backoff_factor="1secs" --revocable_cpu_low_priority="true" --sandbox_directory="/mnt/mesos/sandbox" --strict="true" --switch_user="true" --systemd_enable_support="true" --systemd_runtime_directory="/run/systemd/system" --version="false" --work_dir="/var/lib/mesos"
I0919 15:41:16.373072 12986 slave.cpp:519] Agent resources: cpus(*):2; mem(*):2930; disk(*):4469; ports(*):[31000-32000]
I0919 15:41:16.373291 12986 slave.cpp:527] Agent attributes: [ ]
I0919 15:41:16.373347 12986 slave.cpp:532] Agent hostname: ubuntu12
I0919 15:41:16.379895 13005 state.cpp:57] Recovering state from '/var/lib/mesos/meta'
I0919 15:41:16.382519 13005 group.cpp:349] Group process (group(1)@127.0.1.1:5051) connected to ZooKeeper
I0919 15:41:16.382593 13005 group.cpp:837] Syncing group operations: queue size (joins, cancels, datas) = (0, 0, 0)
I0919 15:41:16.382663 13005 group.cpp:427] Trying to create path '/mesos' in ZooKeeper
I0919 15:41:16.382910 13009 status_update_manager.cpp:200] Recovering status update manager
I0919 15:41:16.383419 13009 containerizer.cpp:522] Recovering containerizer
I0919 15:41:16.392206 13004 provisioner.cpp:253] Provisioner recovery complete
I0919 15:41:16.392354 13004 slave.cpp:4782] Finished recovery
I0919 15:41:16.405709 13004 detector.cpp:152] Detected a new leader: (id='678')
I0919 15:41:16.406067 13005 group.cpp:706] Trying to get '/mesos/json.info_0000000678' in ZooKeeper
I0919 15:41:16.407572 13002 zookeeper.cpp:259] A new leading master (UPID=master@10.142.55.190:5050) is detected
I0919 15:41:16.407977 13002 slave.cpp:895] New master detected at master@10.142.55.190:5050
I0919 15:41:16.408043 13002 slave.cpp:916] No credentials provided. Attempting to register without authentication
I0919 15:41:16.408140 13002 slave.cpp:927] Detecting new master
I0919 15:41:16.408223 13005 status_update_manager.cpp:174] Pausing sending status updates
I0919 15:42:08.418956 13006 slave.cpp:3732] master@10.142.55.190:5050 exited
W0919 15:42:08.419035 13006 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
I0919 15:42:16.374977 13007 slave.cpp:4591] Current disk usage 72.41%. Max allowed age: 1.231186482451933days
I0919 15:42:20.007169 13007 detector.cpp:152] Detected a new leader: (id='679')
I0919 15:42:20.007297 13007 group.cpp:706] Trying to get '/mesos/json.info_0000000679' in ZooKeeper
I0919 15:42:20.008503 13007 zookeeper.cpp:259] A new leading master (UPID=master@10.142.55.196:5050) is detected
I0919 15:42:20.008587 13007 slave.cpp:895] New master detected at master@10.142.55.196:5050
I0919 15:42:20.008610 13007 slave.cpp:916] No credentials provided. Attempting to register without authentication
I0919 15:42:20.008703 13007 slave.cpp:927] Detecting new master
I0919 15:42:20.008750 13007 status_update_manager.cpp:174] Pausing sending status updates
I0919 15:43:16.387984 13005 slave.cpp:4591] Current disk usage 72.41%. Max allowed age: 1.231162010606794days
I0919 15:43:20.081272 13005 slave.cpp:3732] master@10.142.55.196:5050 exited
W0919 15:43:20.081374 13005 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
I0919 15:43:26.855154 13005 slave.cpp:3732] master@10.142.55.196:5050 exited
W0919 15:43:26.855315 13005 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
E0919 15:43:26.855159 13010 process.cpp:2105] Failed to shutdown socket with fd 12: Transport endpoint is not connected
I0919 15:43:32.020196 13002 detector.cpp:152] Detected a new leader: (id='682')
I0919 15:43:32.020300 13002 group.cpp:706] Trying to get '/mesos/json.info_0000000682' in ZooKeeper
I0919 15:43:32.022203 13002 zookeeper.cpp:259] A new leading master (UPID=master@10.142.55.202:5050) is detected
I0919 15:43:32.022302 13002 slave.cpp:895] New master detected at master@10.142.55.202:5050
I0919 15:43:32.022328 13002 slave.cpp:916] No credentials provided. Attempting to register without authentication
I0919 15:43:32.022382 13002 slave.cpp:927] Detecting new master
I0919 15:43:32.022423 13002 status_update_manager.cpp:174] Pausing sending status updates
I0919 15:44:16.389369 13003 slave.cpp:4591] Current disk usage 72.41%. Max allowed age: 1.231119184877789days
I0919 15:44:32.535347 13003 slave.cpp:3732] master@10.142.55.202:5050 exited
W0919 15:44:32.535522 13003 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
I0919 15:44:42.005375 13002 detector.cpp:152] Detected a new leader: (id='684')
I0919 15:44:42.005496 13002 group.cpp:706] Trying to get '/mesos/json.info_0000000684' in ZooKeeper
I0919 15:44:42.006367 13002 zookeeper.cpp:259] A new leading master (UPID=master@10.142.55.190:5050) is detected
I0919 15:44:42.006492 13002 slave.cpp:895] New master detected at master@10.142.55.190:5050
I0919 15:44:42.006597 13002 slave.cpp:916] No credentials provided. Attempting to register without authentication
I0919 15:44:42.006675 13002 slave.cpp:927] Detecting new master
I0919 15:44:42.006577 13008 status_update_manager.cpp:174] Pausing sending status updates
I0919 15:45:16.400794 13006 slave.cpp:4591] Current disk usage 72.48%. Max allowed age: 1.226390000804074days
I0919 15:45:42.354790 13005 slave.cpp:3732] master@10.142.55.190:5050 exited
W0919 15:45:42.354857 13005 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
I0919 15:45:54.020563 13002 detector.cpp:152] Detected a new leader: (id='687')
I0919 15:45:54.020756 13002 group.cpp:706] Trying to get '/mesos/json.info_0000000687' in ZooKeeper
I0919 15:45:54.023296 13002 zookeeper.cpp:259] A new leading master (UPID=master@10.142.55.196:5050) is detected
I0919 15:45:54.023455 13002 slave.cpp:895] New master detected at master@10.142.55.196:5050
I0919 15:45:54.023558 13002 slave.cpp:916] No credentials provided. Attempting to register without authentication
I0919 15:45:54.023526 13008 status_update_manager.cpp:174] Pausing sending status updates
I0919 15:45:54.023669 13002 slave.cpp:927] Detecting new master
I0919 15:46:16.402402 13003 slave.cpp:4591] Current disk usage 72.53%. Max allowed age: 1.223205601954942days
I0919 15:46:54.075505 13007 slave.cpp:3732] master@10.142.55.196:5050 exited
W0919 15:46:54.075592 13007 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
E0919 15:46:56.098012 13010 process.cpp:2105] Failed to shutdown socket with fd 14: Transport endpoint is not connected
I0919 15:46:56.098016 13007 slave.cpp:3732] master@10.142.55.196:5050 exited
W0919 15:46:56.098253 13007 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
E0919 15:46:56.462254 13010 process.cpp:2105] Failed to shutdown socket with fd 14: Transport endpoint is not connected
I0919 15:46:56.462260 13005 slave.cpp:3732] master@10.142.55.196:5050 exited
W0919 15:46:56.462540 13005 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
I0919 15:47:02.005637 13009 detector.cpp:152] Detected a new leader: (id='688')
I0919 15:47:02.005765 13009 group.cpp:706] Trying to get '/mesos/json.info_0000000688' in ZooKeeper
I0919 15:47:02.006853 13009 zookeeper.cpp:259] A new leading master (UPID=master@10.142.55.202:5050) is detected
I0919 15:47:02.006959 13009 slave.cpp:895] New master detected at master@10.142.55.202:5050
I0919 15:47:02.006986 13009 slave.cpp:916] No credentials provided. Attempting to register without authentication
I0919 15:47:02.007025 13009 slave.cpp:927] Detecting new master
I0919 15:47:02.007061 13009 status_update_manager.cpp:174] Pausing sending status updates
I0919 15:47:16.406669 13008 slave.cpp:4591] Current disk usage 72.53%. Max allowed age: 1.223184189090440days
I0919 15:48:02.950891 13005 slave.cpp:3732] master@10.142.55.202:5050 exited
W0919 15:48:02.950994 13005 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
I0919 15:48:12.006634 13005 detector.cpp:152] Detected a new leader: (id='690')
I0919 15:48:12.006817 13003 group.cpp:706] Trying to get '/mesos/json.info_0000000690' in ZooKeeper
I0919 15:48:12.007987 13003 zookeeper.cpp:259] A new leading master (UPID=master@10.142.55.190:5050) is detected
I0919 15:48:12.008126 13003 slave.cpp:895] New master detected at master@10.142.55.190:5050
I0919 15:48:12.008210 13003 slave.cpp:916] No credentials provided. Attempting to register without authentication
I0919 15:48:12.008280 13003 slave.cpp:927] Detecting new master
I0919 15:48:12.008191 13008 status_update_manager.cpp:174] Pausing sending status updates
I0919 15:48:16.409266 13003 slave.cpp:4591] Current disk usage 72.54%. Max allowed age: 1.222480623542604days
I0919 15:49:12.379010 13009 slave.cpp:3732] master@10.142.55.190:5050 exited
W0919 15:49:12.379149 13009 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
E0919 15:49:12.379233 13010 process.cpp:2105] Failed to shutdown socket with fd 12: Transport endpoint is not connected
I0919 15:49:16.413767 13007 slave.cpp:4591] Current disk usage 72.64%. Max allowed age: 1.215032005677465days
I0919 15:49:24.016290 13007 detector.cpp:152] Detected a new leader: (id='693')
I0919 15:49:24.016417 13007 group.cpp:706] Trying to get '/mesos/json.info_0000000693' in ZooKeeper
I0919 15:49:24.018273 13007 zookeeper.cpp:259] A new leading master (UPID=master@10.142.55.196:5050) is detected
I0919 15:49:24.018437 13007 slave.cpp:895] New master detected at master@10.142.55.196:5050
I0919 15:49:24.018523 13007 slave.cpp:916] No credentials provided. Attempting to register without authentication
I0919 15:49:24.018604 13007 slave.cpp:927] Detecting new master
I0919 15:49:24.018496 13008 status_update_manager.cpp:174] Pausing sending status updates
I0919 15:50:16.416391 13008 slave.cpp:4591] Current disk usage 72.64%. Max allowed age: 1.215016710774248days
I0919 15:50:24.065268 13003 slave.cpp:3732] master@10.142.55.196:5050 exited
W0919 15:50:24.065342 13003 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
I0919 15:50:24.485752 13004 slave.cpp:3732] master@10.142.55.196:5050 exited
W0919 15:50:24.485839 13004 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
E0919 15:50:24.485977 13010 process.cpp:2105] Failed to shutdown socket with fd 14: Transport endpoint is not connected
I0919 15:50:28.343647 13003 slave.cpp:3732] master@10.142.55.196:5050 exited
W0919 15:50:28.343719 13003 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
E0919 15:50:28.343819 13010 process.cpp:2105] Failed to shutdown socket with fd 14: Transport endpoint is not connected
I0919 15:50:31.545099 13005 slave.cpp:3732] master@10.142.55.196:5050 exited
W0919 15:50:31.545171 13005 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
E0919 15:50:31.545284 13010 process.cpp:2105] Failed to shutdown socket with fd 14: Transport endpoint is not connected
I0919 15:50:32.007096 13008 detector.cpp:152] Detected a new leader: (id='694')
I0919 15:50:32.007195 13008 group.cpp:706] Trying to get '/mesos/json.info_0000000694' in ZooKeeper
I0919 15:50:32.009881 13008 zookeeper.cpp:259] A new leading master (UPID=master@10.142.55.202:5050) is detected
I0919 15:50:32.009970 13008 slave.cpp:895] New master detected at master@10.142.55.202:5050
I0919 15:50:32.009994 13008 slave.cpp:916] No credentials provided. Attempting to register without authentication
I0919 15:50:32.010030 13008 slave.cpp:927] Detecting new master
I0919 15:50:32.010079 13008 status_update_manager.cpp:174] Pausing sending status updates
I0919 15:51:16.417846 13006 slave.cpp:4591] Current disk usage 72.64%. Max allowed age: 1.214964708103322days
I0919 15:51:32.560317 13003 slave.cpp:3732] master@10.142.55.202:5050 exited
W0919 15:51:32.560410 13003 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
I0919 15:51:42.005147 13009 detector.cpp:152] Detected a new leader: (id='696')
I0919 15:51:42.005265 13009 group.cpp:706] Trying to get '/mesos/json.info_0000000696' in ZooKeeper
I0919 15:51:42.006824 13009 zookeeper.cpp:259] A new leading master (UPID=master@10.142.55.190:5050) is detected
I0919 15:51:42.006904 13009 slave.cpp:895] New master detected at master@10.142.55.190:5050
I0919 15:51:42.006928 13009 slave.cpp:916] No credentials provided. Attempting to register without authentication
I0919 15:51:42.006963 13009 slave.cpp:927] Detecting new master
I0919 15:51:42.006999 13009 status_update_manager.cpp:174] Pausing sending status updates
I0919 15:52:16.419373 13003 slave.cpp:4591] Current disk usage 72.71%. Max allowed age: 1.209981628636250days
I0919 15:52:42.336305 13002 slave.cpp:3732] master@10.142.55.190:5050 exited
W0919 15:52:42.336426 13002 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
I0919 15:52:54.005267 13005 detector.cpp:152] Detected a new leader: (id='699')
I0919 15:52:54.005408 13005 group.cpp:706] Trying to get '/mesos/json.info_0000000699' in ZooKeeper
I0919 15:52:54.006206 13005 zookeeper.cpp:259] A new leading master (UPID=master@10.142.55.196:5050) is detected
I0919 15:52:54.006285 13005 slave.cpp:895] New master detected at master@10.142.55.196:5050
I0919 15:52:54.006309 13005 slave.cpp:916] No credentials provided. Attempting to register without authentication
I0919 15:52:54.006398 13005 slave.cpp:927] Detecting new master
I0919 15:52:54.006451 13005 status_update_manager.cpp:174] Pausing sending status updates
I0919 15:53:16.420258 13005 slave.cpp:4591] Current disk usage 72.76%. Max allowed age: 1.206748286096840days
I0919 15:53:54.071012 13005 slave.cpp:3732] master@10.142.55.196:5050 exited
W0919 15:53:54.071143 13005 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
I0919 15:54:01.105780 13002 slave.cpp:3732] master@10.142.55.196:5050 exited
W0919 15:54:01.105854 13002 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
E0919 15:54:01.105970 13010 process.cpp:2105] Failed to shutdown socket with fd 15: Transport endpoint is not connected
I0919 15:54:05.733837 13007 slave.cpp:3732] master@10.142.55.196:5050 exited
W0919 15:54:05.733932 13007 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
E0919 15:54:05.734071 13010 process.cpp:2105] Failed to shutdown socket with fd 15: Transport endpoint is not connected
E0919 15:54:05.818560 13010 process.cpp:2105] Failed to shutdown socket with fd 15: Transport endpoint is not connected
I0919 15:54:05.818583 13003 slave.cpp:3732] master@10.142.55.196:5050 exited
W0919 15:54:05.818758 13003 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
I0919 15:54:06.004385 13009 detector.cpp:152] Detected a new leader: (id='700')
I0919 15:54:06.004494 13009 group.cpp:706] Trying to get '/mesos/json.info_0000000700' in ZooKeeper
I0919 15:54:06.005511 13009 zookeeper.cpp:259] A new leading master (UPID=master@10.142.55.202:5050) is detected
I0919 15:54:06.005586 13009 slave.cpp:895] New master detected at master@10.142.55.202:5050
I0919 15:54:06.005609 13009 slave.cpp:916] No credentials provided. Attempting to register without authentication
I0919 15:54:06.005676 13009 slave.cpp:927] Detecting new master
I0919 15:54:06.005720 13009 status_update_manager.cpp:174] Pausing sending status updates
I0919 15:54:16.423193 13002 slave.cpp:4591] Current disk usage 72.76%. Max allowed age: 1.206699342406551days
slave warn log
Log file created at: 2016/09/19 15:42:08
Running on machine: ubuntu12
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
W0919 15:42:08.419035 13006 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
W0919 15:43:20.081374 13005 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
W0919 15:43:26.855315 13005 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
E0919 15:43:26.855159 13010 process.cpp:2105] Failed to shutdown socket with fd 12: Transport endpoint is not connected
W0919 15:44:32.535522 13003 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
W0919 15:45:42.354857 13005 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
W0919 15:46:54.075592 13007 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
E0919 15:46:56.098012 13010 process.cpp:2105] Failed to shutdown socket with fd 14: Transport endpoint is not connected
W0919 15:46:56.098253 13007 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
E0919 15:46:56.462254 13010 process.cpp:2105] Failed to shutdown socket with fd 14: Transport endpoint is not connected
W0919 15:46:56.462540 13005 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
W0919 15:48:02.950994 13005 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
W0919 15:49:12.379149 13009 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
E0919 15:49:12.379233 13010 process.cpp:2105] Failed to shutdown socket with fd 12: Transport endpoint is not connected
W0919 15:50:24.065342 13003 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
W0919 15:50:24.485839 13004 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
E0919 15:50:24.485977 13010 process.cpp:2105] Failed to shutdown socket with fd 14: Transport endpoint is not connected
W0919 15:50:28.343719 13003 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
E0919 15:50:28.343819 13010 process.cpp:2105] Failed to shutdown socket with fd 14: Transport endpoint is not connected
W0919 15:50:31.545171 13005 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
E0919 15:50:31.545284 13010 process.cpp:2105] Failed to shutdown socket with fd 14: Transport endpoint is not connected
W0919 15:51:32.560410 13003 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
W0919 15:52:42.336426 13002 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
W0919 15:53:54.071143 13005 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
W0919 15:54:01.105854 13002 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
E0919 15:54:01.105970 13010 process.cpp:2105] Failed to shutdown socket with fd 15: Transport endpoint is not connected
W0919 15:54:05.733932 13007 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
E0919 15:54:05.734071 13010 process.cpp:2105] Failed to shutdown socket with fd 15: Transport endpoint is not connected
E0919 15:54:05.818560 13010 process.cpp:2105] Failed to shutdown socket with fd 15: Transport endpoint is not connected
W0919 15:54:05.818758 13003 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
W0919 15:55:06.821486 13009 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
[1]: https://open.mesosphere.com/getting-started/install/#verifying-installation
> mesos-master can not found mesos-slave, and elect a new leader in a short interval
> ----------------------------------------------------------------------------------
>
> Key: MESOS-6205
> URL: https://issues.apache.org/jira/browse/MESOS-6205
> Project: Mesos
> Issue Type: Bug
> Components: master
> Environment: ubuntu 12 x64, centos 6.5 x64, centos 7.2 x64
> Reporter: kasim
>
> I follow this [doc][https://open.mesosphere.com/getting-started/install/#verifying-installation] to setup mesos cluster.
> There are three vm(ubuntu 12, centos 6.5, centos 7.2).
> $ cat /etc/hosts
> 10.142.55.190 zk1
> 10.142.55.196 zk2
> 10.142.55.202 zk3
> config in each mathine:
> $ cat /etc/mesos/zk
> zk://10.142.55.190:2181,10.142.55.196:2181,10.142.55.202:2181/mesos
> After I start zookeeper, mesos-master and mesos-slave in three vm, I can view the mesos webui(10.142.55.190:5050). I found agents count is 0.
> After a little time, mesos page get error:
> Failed to connect to 10.142.55.190:5050!
> Retrying in 16 seconds...
> (I found that zookeeper would elect a new leader in a short interval)
> master info log:
> I0919 15:54:59.677438 13281 http.cpp:2022] Redirecting request for /master/state?jsonp=angular.callbacks._1x to the leading master zk3
> I0919 15:55:00.098667 13281 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (768)@10.142.55.202:5050
> I0919 15:55:00.385279 13281 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (185)@10.142.55.196:5050
> I0919 15:55:00.711119 13281 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (771)@10.142.55.202:5050
> I0919 15:55:01.347291 13284 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (188)@10.142.55.196:5050
> I0919 15:55:01.597682 13284 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (774)@10.142.55.202:5050
> I0919 15:55:02.257159 13282 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (191)@10.142.55.196:5050
> I0919 15:55:02.370692 13287 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (777)@10.142.55.202:5050
> I0919 15:55:03.205920 13285 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (780)@10.142.55.202:5050
> I0919 15:55:03.260007 13281 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (194)@10.142.55.196:5050
> I0919 15:55:03.929611 13283 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (783)@10.142.55.202:5050
> I0919 15:55:04.033308 13287 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (197)@10.142.55.196:5050
> I0919 15:55:04.591275 13284 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (200)@10.142.55.196:5050
> I0919 15:55:04.608211 13283 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (786)@10.142.55.202:5050
> I0919 15:55:05.184682 13280 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (789)@10.142.55.202:5050
> I0919 15:55:05.268277 13280 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (203)@10.142.55.196:5050
> I0919 15:55:05.775377 13281 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (206)@10.142.55.196:5050
> I0919 15:55:05.916445 13285 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (792)@10.142.55.202:5050
> I0919 15:55:06.744927 13280 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (209)@10.142.55.196:5050
> I0919 15:55:07.378521 13283 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (5)@10.142.55.202:5050
> I0919 15:55:07.393311 13285 network.hpp:430] ZooKeeper group memberships changed
> I0919 15:55:07.393427 13285 group.cpp:706] Trying to get '/mesos/log_replicas/0000000709' in ZooKeeper
> I0919 15:55:07.393985 13285 group.cpp:706] Trying to get '/mesos/log_replicas/0000000711' in ZooKeeper
> I0919 15:55:07.394394 13285 group.cpp:706] Trying to get '/mesos/log_replicas/0000000714' in ZooKeeper
> I0919 15:55:07.394843 13285 group.cpp:706] Trying to get '/mesos/log_replicas/0000000715' in ZooKeeper
> I0919 15:55:07.395418 13285 network.hpp:478] ZooKeeper group PIDs: { log-replica(1)@10.142.55.190:5050, log-replica(1)@10.142.55.196:5050, log-replica(1)@10.142.55.202:5050 }
> I0919 15:55:08.178272 13280 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (14)@10.142.55.202:5050
> I0919 15:55:09.059562 13282 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (21)@10.142.55.202:5050
> I0919 15:55:09.700711 13286 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (24)@10.142.55.202:5050
> I0919 15:55:09.742185 13287 http.cpp:381] HTTP GET for /master/state from 10.142.50.94:59987 with User-Agent='Mozilla/5.0 (Windows NT 6.2; WOW64; rv:47.0) Gecko/20100101 Firefox/47.0'
> I0919 15:55:09.742359 13287 http.cpp:2022] Redirecting request for /master/state?jsonp=angular.callbacks._1y to the leading master zk3
> I0919 15:55:10.660789 13280 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (30)@10.142.55.202:5050
> I0919 15:55:11.480326 13281 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (34)@10.142.55.202:5050
> I0919 15:55:12.386256 13286 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (37)@10.142.55.202:5050
> I0919 15:55:12.975137 13287 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (42)@10.142.55.202:5050
> I0919 15:55:13.843091 13285 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (47)@10.142.55.202:5050
> I0919 15:55:14.373478 13281 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (51)@10.142.55.202:5050
> I0919 15:55:14.937181 13280 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (54)@10.142.55.202:5050
> I0919 15:55:15.658219 13283 replica.cpp:673] Replica in VOTING status received a broadcasted recover request from (58)@10.142.55.202:5050
> I0919 15:55:16.007822 13286 network.hpp:430] ZooKeeper group memberships changed
> I0919 15:55:16.007972 13286 group.cpp:706] Trying to get '/mesos/log_replicas/0000000711' in ZooKeeper
> I0919 15:55:16.010170 13286 group.cpp:706] Trying to get '/mesos/log_replicas/0000000714' in ZooKeeper
> I0919 15:55:16.011462 13284 detector.cpp:152] Detected a new leader: (id='702')
> I0919 15:55:16.011556 13284 group.cpp:706] Trying to get '/mesos/json.info_0000000702' in ZooKeeper
> I0919 15:55:16.011968 13286 group.cpp:706] Trying to get '/mesos/log_replicas/0000000715' in ZooKeeper
> I0919 15:55:16.012526 13286 network.hpp:478] ZooKeeper group PIDs: { log-replica(1)@10.142.55.190:5050, log-replica(1)@10.142.55.196:5050, log-replica(1)@10.142.55.202:5050 }
> I0919 15:55:16.013156 13284 zookeeper.cpp:259] A new leading master (UPID=master@10.142.55.190:5050) is detected
> I0919 15:55:16.013222 13284 master.cpp:1847] The newly elected leader is master@10.142.55.190:5050 with id 677967bc-f6f0-46b3-a44e-72eed1befd60
> I0919 15:55:16.013244 13284 master.cpp:1860] Elected as the leading master!
> I0919 15:55:16.013273 13284 master.cpp:1547] Recovering from registrar
> I0919 15:55:16.013352 13284 registrar.cpp:332] Recovering registrar
> I0919 15:55:16.014081 13280 log.cpp:553] Attempting to start the writer
> I0919 15:55:16.014515 13280 replica.cpp:493] Replica received implicit promise request from (211)@10.142.55.190:5050 with proposal 1204590
> I0919 15:55:16.018023 13282 consensus.cpp:360] Aborting implicit promise request because 2 ignores received
> I0919 15:55:16.018028 13280 leveldb.cpp:304] Persisting metadata (10 bytes) to leveldb took 3.469479ms
> I0919 15:55:16.018338 13280 replica.cpp:342] Persisted promised to 1204590
> I0919 15:55:16.018508 13282 log.cpp:565] Could not start the writer, but can be retried
> I0919 15:55:16.018645 13282 log.cpp:553] Attempting to start the writer
> I0919 15:55:16.018899 13282 replica.cpp:493] Replica received implicit promise request from (215)@10.142.55.190:5050 with proposal 1204591
> I0919 15:55:16.022183 13287 consensus.cpp:360] Aborting implicit promise request because 2 ignores received
> I0919 15:55:16.022367 13280 log.cpp:565] Could not start the writer, but can be retried
> I0919 15:55:16.022510 13280 log.cpp:553] Attempting to start the writer
> I0919 15:55:16.028880 13282 leveldb.cpp:304] Persisting metadata (10 bytes) to leveldb took 9.870818ms
> I0919 15:55:16.029024 13282 replica.cpp:342] Persisted promised to 1204591
> I0919 15:55:16.029428 13286 replica.cpp:493] Replica received implicit promise request from (219)@10.142.55.190:5050 with proposal 1204592
> I0919 15:55:16.031600 13280 consensus.cpp:360] Aborting implicit promise request because 2 ignores received
> I0919 15:55:16.036208 13283 log.cpp:565] Could not start the writer, but can be retried
> I0919 15:55:16.036454 13283 log.cpp:553] Attempting to start the writer
> I0919 15:55:16.040256 13286 leveldb.cpp:304] Persisting metadata (10 bytes) to leveldb took 10.783237ms
> I0919 15:55:16.040339 13286 replica.cpp:342] Persisted promised to 1204592
> I0919 15:55:16.040712 13286 replica.cpp:493] Replica received implicit promise request from (222)@10.142.55.190:5050 with proposal 1204593
> I0919 15:55:16.042196 13286 leveldb.cpp:304] Persisting metadata (10 bytes) to leveldb took 1.435071ms
> I0919 15:55:16.042250 13286 replica.cpp:342] Persisted promised to 1204593
> I0919 15:55:16.042981 13286 consensus.cpp:360] Aborting implicit promise request because 2 ignores received
> I0919 15:55:16.043099 13286 log.cpp:565] Could not start the writer, but can be retried
> I0919 15:55:16.043303 13283 log.cpp:553] Attempting to start the writer
> All later logs are looping
> I0919 15:55:16.043676 13286 replica.cpp:493] Replica received implicit promise request from (225)@10.142.55.190:5050 with proposal 1204594
> I0919 15:55:16.044122 13286 leveldb.cpp:304] Persisting metadata (10 bytes) to leveldb took 404769ns
> I0919 15:55:16.044209 13286 replica.cpp:342] Persisted promised to 1204594
> I0919 15:55:16.044837 13281 consensus.cpp:360] Aborting implicit promise request because 2 ignores received
> I0919 15:55:16.044926 13281 log.cpp:565] Could not start the writer, but can be retried
> I0919 15:55:16.045038 13281 log.cpp:553] Attempting to start the writer
> slave info log:
> Log file created at: 2016/09/19 15:41:16
> Running on machine: ubuntu12
> Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
> I0919 15:41:16.346844 12986 logging.cpp:194] INFO level logging started!
> I0919 15:41:16.363313 12986 containerizer.cpp:196] Using isolation: posix/cpu,posix/mem,filesystem/posix,network/cni
> I0919 15:41:16.370334 12986 main.cpp:434] Starting Mesos agent
> I0919 15:41:16.371184 12986 slave.cpp:198] Agent started on 1)@127.0.1.1:5051
> I0919 15:41:16.371636 12986 slave.cpp:199] Flags at startup: --appc_simple_discovery_uri_prefix="http://" --appc_store_dir="/tmp/mesos/store/appc" --authenticate_http_readonly="false" --authenticate_http_readwrite="false" --authenticatee="crammd5" --authentication_backoff_factor="1secs" --authorizer="local" --cgroups_cpu_enable_pids_and_tids_count="false" --cgroups_enable_cfs="false" --cgroups_hierarchy="/sys/fs/cgroup" --cgroups_limit_swap="false" --cgroups_root="mesos" --container_disk_watch_interval="15secs" --containerizers="mesos" --default_role="*" --disk_watch_interval="1mins" --docker="docker" --docker_kill_orphans="true" --docker_registry="https://registry-1.docker.io" --docker_remove_delay="6hrs" --docker_socket="/var/run/docker.sock" --docker_stop_timeout="0ns" --docker_store_dir="/tmp/mesos/store/docker" --docker_volume_checkpoint_dir="/var/run/mesos/isolators/docker/volume" --enforce_container_disk_quota="false" --executor_registration_timeout="1mins" --executor_shutdown_grace_period="5secs" --fetcher_cache_dir="/tmp/mesos/fetch" --fetcher_cache_size="2GB" --frameworks_home="" --gc_delay="1weeks" --gc_disk_headroom="0.1" --hadoop_home="" --help="false" --hostname_lookup="true" --http_authenticators="basic" --http_command_executor="false" --image_provisioner_backend="copy" --initialize_driver_logging="true" --isolation="posix/cpu,posix/mem" --launcher="posix" --launcher_dir="/usr/libexec/mesos" --log_dir="/var/log/mesos" --logbufsecs="0" --logging_level="INFO" --master="zk://10.142.55.190:2181,10.142.55.196:2181,10.142.55.202:2181/mesos" --oversubscribed_resources_interval="15secs" --perf_duration="10secs" --perf_interval="1mins" --port="5051" --qos_correction_interval_min="0ns" --quiet="false" --recover="reconnect" --recovery_timeout="15mins" --registration_backoff_factor="1secs" --revocable_cpu_low_priority="true" --sandbox_directory="/mnt/mesos/sandbox" --strict="true" --switch_user="true" --systemd_enable_support="true" --systemd_runtime_directory="/run/systemd/system" --version="false" --work_dir="/var/lib/mesos"
> I0919 15:41:16.373072 12986 slave.cpp:519] Agent resources: cpus(*):2; mem(*):2930; disk(*):4469; ports(*):[31000-32000]
> I0919 15:41:16.373291 12986 slave.cpp:527] Agent attributes: [ ]
> I0919 15:41:16.373347 12986 slave.cpp:532] Agent hostname: ubuntu12
> I0919 15:41:16.379895 13005 state.cpp:57] Recovering state from '/var/lib/mesos/meta'
> I0919 15:41:16.382519 13005 group.cpp:349] Group process (group(1)@127.0.1.1:5051) connected to ZooKeeper
> I0919 15:41:16.382593 13005 group.cpp:837] Syncing group operations: queue size (joins, cancels, datas) = (0, 0, 0)
> I0919 15:41:16.382663 13005 group.cpp:427] Trying to create path '/mesos' in ZooKeeper
> I0919 15:41:16.382910 13009 status_update_manager.cpp:200] Recovering status update manager
> I0919 15:41:16.383419 13009 containerizer.cpp:522] Recovering containerizer
> I0919 15:41:16.392206 13004 provisioner.cpp:253] Provisioner recovery complete
> I0919 15:41:16.392354 13004 slave.cpp:4782] Finished recovery
> I0919 15:41:16.405709 13004 detector.cpp:152] Detected a new leader: (id='678')
> I0919 15:41:16.406067 13005 group.cpp:706] Trying to get '/mesos/json.info_0000000678' in ZooKeeper
> I0919 15:41:16.407572 13002 zookeeper.cpp:259] A new leading master (UPID=master@10.142.55.190:5050) is detected
> I0919 15:41:16.407977 13002 slave.cpp:895] New master detected at master@10.142.55.190:5050
> I0919 15:41:16.408043 13002 slave.cpp:916] No credentials provided. Attempting to register without authentication
> I0919 15:41:16.408140 13002 slave.cpp:927] Detecting new master
> I0919 15:41:16.408223 13005 status_update_manager.cpp:174] Pausing sending status updates
> I0919 15:42:08.418956 13006 slave.cpp:3732] master@10.142.55.190:5050 exited
> W0919 15:42:08.419035 13006 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
> I0919 15:42:16.374977 13007 slave.cpp:4591] Current disk usage 72.41%. Max allowed age: 1.231186482451933days
> I0919 15:42:20.007169 13007 detector.cpp:152] Detected a new leader: (id='679')
> I0919 15:42:20.007297 13007 group.cpp:706] Trying to get '/mesos/json.info_0000000679' in ZooKeeper
> I0919 15:42:20.008503 13007 zookeeper.cpp:259] A new leading master (UPID=master@10.142.55.196:5050) is detected
> I0919 15:42:20.008587 13007 slave.cpp:895] New master detected at master@10.142.55.196:5050
> I0919 15:42:20.008610 13007 slave.cpp:916] No credentials provided. Attempting to register without authentication
> I0919 15:42:20.008703 13007 slave.cpp:927] Detecting new master
> I0919 15:42:20.008750 13007 status_update_manager.cpp:174] Pausing sending status updates
> I0919 15:43:16.387984 13005 slave.cpp:4591] Current disk usage 72.41%. Max allowed age: 1.231162010606794days
> I0919 15:43:20.081272 13005 slave.cpp:3732] master@10.142.55.196:5050 exited
> W0919 15:43:20.081374 13005 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
> I0919 15:43:26.855154 13005 slave.cpp:3732] master@10.142.55.196:5050 exited
> W0919 15:43:26.855315 13005 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
> E0919 15:43:26.855159 13010 process.cpp:2105] Failed to shutdown socket with fd 12: Transport endpoint is not connected
> I0919 15:43:32.020196 13002 detector.cpp:152] Detected a new leader: (id='682')
> I0919 15:43:32.020300 13002 group.cpp:706] Trying to get '/mesos/json.info_0000000682' in ZooKeeper
> I0919 15:43:32.022203 13002 zookeeper.cpp:259] A new leading master (UPID=master@10.142.55.202:5050) is detected
> I0919 15:43:32.022302 13002 slave.cpp:895] New master detected at master@10.142.55.202:5050
> I0919 15:43:32.022328 13002 slave.cpp:916] No credentials provided. Attempting to register without authentication
> I0919 15:43:32.022382 13002 slave.cpp:927] Detecting new master
> I0919 15:43:32.022423 13002 status_update_manager.cpp:174] Pausing sending status updates
> I0919 15:44:16.389369 13003 slave.cpp:4591] Current disk usage 72.41%. Max allowed age: 1.231119184877789days
> I0919 15:44:32.535347 13003 slave.cpp:3732] master@10.142.55.202:5050 exited
> W0919 15:44:32.535522 13003 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
> I0919 15:44:42.005375 13002 detector.cpp:152] Detected a new leader: (id='684')
> I0919 15:44:42.005496 13002 group.cpp:706] Trying to get '/mesos/json.info_0000000684' in ZooKeeper
> I0919 15:44:42.006367 13002 zookeeper.cpp:259] A new leading master (UPID=master@10.142.55.190:5050) is detected
> I0919 15:44:42.006492 13002 slave.cpp:895] New master detected at master@10.142.55.190:5050
> I0919 15:44:42.006597 13002 slave.cpp:916] No credentials provided. Attempting to register without authentication
> I0919 15:44:42.006675 13002 slave.cpp:927] Detecting new master
> I0919 15:44:42.006577 13008 status_update_manager.cpp:174] Pausing sending status updates
> I0919 15:45:16.400794 13006 slave.cpp:4591] Current disk usage 72.48%. Max allowed age: 1.226390000804074days
> I0919 15:45:42.354790 13005 slave.cpp:3732] master@10.142.55.190:5050 exited
> W0919 15:45:42.354857 13005 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
> I0919 15:45:54.020563 13002 detector.cpp:152] Detected a new leader: (id='687')
> I0919 15:45:54.020756 13002 group.cpp:706] Trying to get '/mesos/json.info_0000000687' in ZooKeeper
> I0919 15:45:54.023296 13002 zookeeper.cpp:259] A new leading master (UPID=master@10.142.55.196:5050) is detected
> I0919 15:45:54.023455 13002 slave.cpp:895] New master detected at master@10.142.55.196:5050
> I0919 15:45:54.023558 13002 slave.cpp:916] No credentials provided. Attempting to register without authentication
> I0919 15:45:54.023526 13008 status_update_manager.cpp:174] Pausing sending status updates
> I0919 15:45:54.023669 13002 slave.cpp:927] Detecting new master
> I0919 15:46:16.402402 13003 slave.cpp:4591] Current disk usage 72.53%. Max allowed age: 1.223205601954942days
> I0919 15:46:54.075505 13007 slave.cpp:3732] master@10.142.55.196:5050 exited
> W0919 15:46:54.075592 13007 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
> E0919 15:46:56.098012 13010 process.cpp:2105] Failed to shutdown socket with fd 14: Transport endpoint is not connected
> I0919 15:46:56.098016 13007 slave.cpp:3732] master@10.142.55.196:5050 exited
> W0919 15:46:56.098253 13007 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
> E0919 15:46:56.462254 13010 process.cpp:2105] Failed to shutdown socket with fd 14: Transport endpoint is not connected
> I0919 15:46:56.462260 13005 slave.cpp:3732] master@10.142.55.196:5050 exited
> W0919 15:46:56.462540 13005 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
> I0919 15:47:02.005637 13009 detector.cpp:152] Detected a new leader: (id='688')
> I0919 15:47:02.005765 13009 group.cpp:706] Trying to get '/mesos/json.info_0000000688' in ZooKeeper
> I0919 15:47:02.006853 13009 zookeeper.cpp:259] A new leading master (UPID=master@10.142.55.202:5050) is detected
> I0919 15:47:02.006959 13009 slave.cpp:895] New master detected at master@10.142.55.202:5050
> I0919 15:47:02.006986 13009 slave.cpp:916] No credentials provided. Attempting to register without authentication
> I0919 15:47:02.007025 13009 slave.cpp:927] Detecting new master
> I0919 15:47:02.007061 13009 status_update_manager.cpp:174] Pausing sending status updates
> I0919 15:47:16.406669 13008 slave.cpp:4591] Current disk usage 72.53%. Max allowed age: 1.223184189090440days
> I0919 15:48:02.950891 13005 slave.cpp:3732] master@10.142.55.202:5050 exited
> W0919 15:48:02.950994 13005 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
> I0919 15:48:12.006634 13005 detector.cpp:152] Detected a new leader: (id='690')
> I0919 15:48:12.006817 13003 group.cpp:706] Trying to get '/mesos/json.info_0000000690' in ZooKeeper
> I0919 15:48:12.007987 13003 zookeeper.cpp:259] A new leading master (UPID=master@10.142.55.190:5050) is detected
> I0919 15:48:12.008126 13003 slave.cpp:895] New master detected at master@10.142.55.190:5050
> I0919 15:48:12.008210 13003 slave.cpp:916] No credentials provided. Attempting to register without authentication
> I0919 15:48:12.008280 13003 slave.cpp:927] Detecting new master
> I0919 15:48:12.008191 13008 status_update_manager.cpp:174] Pausing sending status updates
> I0919 15:48:16.409266 13003 slave.cpp:4591] Current disk usage 72.54%. Max allowed age: 1.222480623542604days
> I0919 15:49:12.379010 13009 slave.cpp:3732] master@10.142.55.190:5050 exited
> W0919 15:49:12.379149 13009 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
> E0919 15:49:12.379233 13010 process.cpp:2105] Failed to shutdown socket with fd 12: Transport endpoint is not connected
> I0919 15:49:16.413767 13007 slave.cpp:4591] Current disk usage 72.64%. Max allowed age: 1.215032005677465days
> I0919 15:49:24.016290 13007 detector.cpp:152] Detected a new leader: (id='693')
> I0919 15:49:24.016417 13007 group.cpp:706] Trying to get '/mesos/json.info_0000000693' in ZooKeeper
> I0919 15:49:24.018273 13007 zookeeper.cpp:259] A new leading master (UPID=master@10.142.55.196:5050) is detected
> I0919 15:49:24.018437 13007 slave.cpp:895] New master detected at master@10.142.55.196:5050
> I0919 15:49:24.018523 13007 slave.cpp:916] No credentials provided. Attempting to register without authentication
> I0919 15:49:24.018604 13007 slave.cpp:927] Detecting new master
> I0919 15:49:24.018496 13008 status_update_manager.cpp:174] Pausing sending status updates
> I0919 15:50:16.416391 13008 slave.cpp:4591] Current disk usage 72.64%. Max allowed age: 1.215016710774248days
> I0919 15:50:24.065268 13003 slave.cpp:3732] master@10.142.55.196:5050 exited
> W0919 15:50:24.065342 13003 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
> I0919 15:50:24.485752 13004 slave.cpp:3732] master@10.142.55.196:5050 exited
> W0919 15:50:24.485839 13004 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
> E0919 15:50:24.485977 13010 process.cpp:2105] Failed to shutdown socket with fd 14: Transport endpoint is not connected
> I0919 15:50:28.343647 13003 slave.cpp:3732] master@10.142.55.196:5050 exited
> W0919 15:50:28.343719 13003 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
> E0919 15:50:28.343819 13010 process.cpp:2105] Failed to shutdown socket with fd 14: Transport endpoint is not connected
> I0919 15:50:31.545099 13005 slave.cpp:3732] master@10.142.55.196:5050 exited
> W0919 15:50:31.545171 13005 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
> E0919 15:50:31.545284 13010 process.cpp:2105] Failed to shutdown socket with fd 14: Transport endpoint is not connected
> I0919 15:50:32.007096 13008 detector.cpp:152] Detected a new leader: (id='694')
> I0919 15:50:32.007195 13008 group.cpp:706] Trying to get '/mesos/json.info_0000000694' in ZooKeeper
> I0919 15:50:32.009881 13008 zookeeper.cpp:259] A new leading master (UPID=master@10.142.55.202:5050) is detected
> I0919 15:50:32.009970 13008 slave.cpp:895] New master detected at master@10.142.55.202:5050
> I0919 15:50:32.009994 13008 slave.cpp:916] No credentials provided. Attempting to register without authentication
> I0919 15:50:32.010030 13008 slave.cpp:927] Detecting new master
> I0919 15:50:32.010079 13008 status_update_manager.cpp:174] Pausing sending status updates
> I0919 15:51:16.417846 13006 slave.cpp:4591] Current disk usage 72.64%. Max allowed age: 1.214964708103322days
> I0919 15:51:32.560317 13003 slave.cpp:3732] master@10.142.55.202:5050 exited
> W0919 15:51:32.560410 13003 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
> I0919 15:51:42.005147 13009 detector.cpp:152] Detected a new leader: (id='696')
> I0919 15:51:42.005265 13009 group.cpp:706] Trying to get '/mesos/json.info_0000000696' in ZooKeeper
> I0919 15:51:42.006824 13009 zookeeper.cpp:259] A new leading master (UPID=master@10.142.55.190:5050) is detected
> I0919 15:51:42.006904 13009 slave.cpp:895] New master detected at master@10.142.55.190:5050
> I0919 15:51:42.006928 13009 slave.cpp:916] No credentials provided. Attempting to register without authentication
> I0919 15:51:42.006963 13009 slave.cpp:927] Detecting new master
> I0919 15:51:42.006999 13009 status_update_manager.cpp:174] Pausing sending status updates
> I0919 15:52:16.419373 13003 slave.cpp:4591] Current disk usage 72.71%. Max allowed age: 1.209981628636250days
> I0919 15:52:42.336305 13002 slave.cpp:3732] master@10.142.55.190:5050 exited
> W0919 15:52:42.336426 13002 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
> I0919 15:52:54.005267 13005 detector.cpp:152] Detected a new leader: (id='699')
> I0919 15:52:54.005408 13005 group.cpp:706] Trying to get '/mesos/json.info_0000000699' in ZooKeeper
> I0919 15:52:54.006206 13005 zookeeper.cpp:259] A new leading master (UPID=master@10.142.55.196:5050) is detected
> I0919 15:52:54.006285 13005 slave.cpp:895] New master detected at master@10.142.55.196:5050
> I0919 15:52:54.006309 13005 slave.cpp:916] No credentials provided. Attempting to register without authentication
> I0919 15:52:54.006398 13005 slave.cpp:927] Detecting new master
> I0919 15:52:54.006451 13005 status_update_manager.cpp:174] Pausing sending status updates
> I0919 15:53:16.420258 13005 slave.cpp:4591] Current disk usage 72.76%. Max allowed age: 1.206748286096840days
> I0919 15:53:54.071012 13005 slave.cpp:3732] master@10.142.55.196:5050 exited
> W0919 15:53:54.071143 13005 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
> I0919 15:54:01.105780 13002 slave.cpp:3732] master@10.142.55.196:5050 exited
> W0919 15:54:01.105854 13002 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
> E0919 15:54:01.105970 13010 process.cpp:2105] Failed to shutdown socket with fd 15: Transport endpoint is not connected
> I0919 15:54:05.733837 13007 slave.cpp:3732] master@10.142.55.196:5050 exited
> W0919 15:54:05.733932 13007 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
> E0919 15:54:05.734071 13010 process.cpp:2105] Failed to shutdown socket with fd 15: Transport endpoint is not connected
> E0919 15:54:05.818560 13010 process.cpp:2105] Failed to shutdown socket with fd 15: Transport endpoint is not connected
> I0919 15:54:05.818583 13003 slave.cpp:3732] master@10.142.55.196:5050 exited
> W0919 15:54:05.818758 13003 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
> I0919 15:54:06.004385 13009 detector.cpp:152] Detected a new leader: (id='700')
> I0919 15:54:06.004494 13009 group.cpp:706] Trying to get '/mesos/json.info_0000000700' in ZooKeeper
> I0919 15:54:06.005511 13009 zookeeper.cpp:259] A new leading master (UPID=master@10.142.55.202:5050) is detected
> I0919 15:54:06.005586 13009 slave.cpp:895] New master detected at master@10.142.55.202:5050
> I0919 15:54:06.005609 13009 slave.cpp:916] No credentials provided. Attempting to register without authentication
> I0919 15:54:06.005676 13009 slave.cpp:927] Detecting new master
> I0919 15:54:06.005720 13009 status_update_manager.cpp:174] Pausing sending status updates
> I0919 15:54:16.423193 13002 slave.cpp:4591] Current disk usage 72.76%. Max allowed age: 1.206699342406551days
> slave warn log
> Log file created at: 2016/09/19 15:42:08
> Running on machine: ubuntu12
> Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
> W0919 15:42:08.419035 13006 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
> W0919 15:43:20.081374 13005 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
> W0919 15:43:26.855315 13005 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
> E0919 15:43:26.855159 13010 process.cpp:2105] Failed to shutdown socket with fd 12: Transport endpoint is not connected
> W0919 15:44:32.535522 13003 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
> W0919 15:45:42.354857 13005 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
> W0919 15:46:54.075592 13007 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
> E0919 15:46:56.098012 13010 process.cpp:2105] Failed to shutdown socket with fd 14: Transport endpoint is not connected
> W0919 15:46:56.098253 13007 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
> E0919 15:46:56.462254 13010 process.cpp:2105] Failed to shutdown socket with fd 14: Transport endpoint is not connected
> W0919 15:46:56.462540 13005 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
> W0919 15:48:02.950994 13005 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
> W0919 15:49:12.379149 13009 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
> E0919 15:49:12.379233 13010 process.cpp:2105] Failed to shutdown socket with fd 12: Transport endpoint is not connected
> W0919 15:50:24.065342 13003 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
> W0919 15:50:24.485839 13004 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
> E0919 15:50:24.485977 13010 process.cpp:2105] Failed to shutdown socket with fd 14: Transport endpoint is not connected
> W0919 15:50:28.343719 13003 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
> E0919 15:50:28.343819 13010 process.cpp:2105] Failed to shutdown socket with fd 14: Transport endpoint is not connected
> W0919 15:50:31.545171 13005 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
> E0919 15:50:31.545284 13010 process.cpp:2105] Failed to shutdown socket with fd 14: Transport endpoint is not connected
> W0919 15:51:32.560410 13003 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
> W0919 15:52:42.336426 13002 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
> W0919 15:53:54.071143 13005 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
> W0919 15:54:01.105854 13002 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
> E0919 15:54:01.105970 13010 process.cpp:2105] Failed to shutdown socket with fd 15: Transport endpoint is not connected
> W0919 15:54:05.733932 13007 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
> E0919 15:54:05.734071 13010 process.cpp:2105] Failed to shutdown socket with fd 15: Transport endpoint is not connected
> E0919 15:54:05.818560 13010 process.cpp:2105] Failed to shutdown socket with fd 15: Transport endpoint is not connected
> W0919 15:54:05.818758 13003 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
> W0919 15:55:06.821486 13009 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)